Quality control rules

Quality control rules help regulate access to a pool: you can filter out users who give incorrect responses in control tasks, skip too many tasks in a row, and so on.

A rule consists of two parts: the condition (“If”), and the action to perform when the condition is met (“then”). If there are multiple conditions, they are combined using AND.

To define quality control rules, add an array of quality_control.configs JSON objects to the pool. A sample quality control rule in JSON with key descriptions is shown below.

Sample quality control rule in JSON

This example shows settings for a quality control rule based on skipped tasks. A user who skips 10 task suites in a row is banned from accessing project tasks for one day.

{
    "configs": [
        {
            "collector_config": {
                "type": "SKIPPED_IN_ROW_ASSIGNMENTS"
                },
            "rules": [
                {"conditions": [
                        {
                            "key": "skipped_in_row_count",
                            "operator": "GTE",
                            "value": 10
                        }
                    ],
                  "action": {
                        "parameters": {
                            "duration_days": 1,
                            "scope": "PROJECT"
                        },
                        "type": "RESTRICTION"
                    }

                }
            ]
        }
    ]
}

Keys

Key

Value

Data type

Required during creation

configs.collector_сonfig

Parameters for collecting statistics (for example, the number of tasks skipped in the pool).

JSON

Yes

configs.rules.conditions

Conditions (for example, 10 task suites skipped in a row). Multiple conditions are combined with the “OR” operator.

JSON

Yes

configs.rules.action

The action to perform if conditions are met (for example, block access to the project).

JSON

Yes

configs.collector_сonfig key

Key

Value

Data type

Required during creation

type

Criteria for the quality control rule:

  • GOLDEN_SET — The number of correct and incorrect responses in the golden set tasks.

  • MAJORITY_VOTE — The percentage of responses that matched the majority vote.

  • CAPTCHA — The number of captchas entered successfully and unsuccessfully.

  • INCOME — Payment for tasks completed by the user over the past 24 hours.

  • SKIPPED_IN_ROW_ASSIGNMENTS — The number of task suites skipped in a row.

  • ANSWER_COUNT — The number of task suites completed by the user in the pool.

  • ASSIGNMENT_SUBMIT_TIME — The number of “hasty” responses (the minimum response speed is set in the parameters).

  • ACCEPTANCE_RATE — The percentage of user responses that were rejected during offline acceptance.

String

Yes

parameters

Parameters for collecting data (depends on the quality control rule specified in the type key).

JSON

For rules of type
GOLDEN_SET, MAJORITY_VOTE, CAPTCHA, ASSIGNMENT_SUBMIT_TIME.

configs.rules.conditions key

Key

Value

Data type

Required during creation

key

The value that is checked in the condition (for example, the number of task suites skipped).

Multiple conditions with the same key value are combined using the AND operator.

String

Yes

operator

Comparison operator (the key data is compared with the threshold value from value):

  • EQ (“Equal”)

  • NE (“Not equal to”)

  • GT (“Greater than”)

  • LT (“Less than”)

  • GTE (“Greater than or equal to”)

  • LTE (“Less than or equal to”)

String

Yes

value

The threshold value of the variable specified in key.

Number

Yes

action key

Key

Value

Data type

Required during creation

type

Type of action:

  • RESTRICTION — Block access to projects or pools.

  • SET_SKILL_FROM_OUTPUT_FIELD — Set the “percentage of correct responses” as the skill value (used in the golden set and majority vote).

    You can use the skill value for filtering users.

String

Yes

parameters

Action parameters.

JSON

Yes

parameters.scope

Scope:

  • POOL — pool. Affects the user's rating.

  • PROJECT — The project. Affects the user's rating.

  • ALL_PROJECTS — All the requester's projects.

String

Yes

parameters.duration_days

How many days the ban will last. By default, the ban is indefinite.

Integer

No

parameters.private_comment

Comments (the reason for blocking access). Visible only to the requester.

String

No

parameters.skill_id

ID of the skill to update as tasks are completed.

String

Yes, if the type of action is selected
type=SET_SKILL_FROM_OUTPUT_FIELD
parameters.from_field

The value to assign to the skill:

  • correct_answers _rate — The percentage of correct responses.

  • wrong_answers_rate — The percentage of incorrect responses.

String

Yes, if the type of action is selected
type=SET_SKILL_FROM_OUTPUT_FIELD
parameters.skill_value

A fixed value to assign to the skill (a number from 0 to 100).

Number

Yes, if the type of action is selected
type=SET_SKILL