Quality control rules

Quality control rules help regulate access to a pool: you can filter out users who give incorrect responses in control tasks, skip too many tasks in a row, and so on.

A rule consists of two parts: the condition (“If”), and the action to perform when the condition is met (“then”). If there are multiple conditions, they are combined using AND.

To define quality control rules, add an array of quality_control.configs JSON objects to the pool.

Example of a quality control rule in JSON format

This example shows settings for a quality control rule used to prevent skipping tasks. If a user skips 10 task suites in a row, you can ban them from the project tasks (for example, for 10 days).

Ban for 10 days
{
    "configs": [{
        "collector_config": {
            "type": "SKIPPED_IN_ROW_ASSIGNMENTS"
        },
        "rules": [{
            "conditions": [{
                "key": "skipped_in_row_count",
                "operator": "GTE",
                "value": 10
            }],
            "action": {
                "type": "RESTRICTION_V2",
                "parameters": {
                    "scope": "PROJECT",
                    "duration_unit": "DAYS",
                    "duration": 10,
                    "private_comment": "Skipped more than 10 task suites in a row"
                }
             }
        }]
    }]
}
Copied to clipboard

Keys

Key

Value

Data type

Required during creation

configs.collector_сonfig

Parameters for collecting statistics (for example, the number of tasks skipped in the pool).

JSON

Yes

configs.rules.conditions

Conditions (for example, 10 task suites skipped in a row). Multiple conditions are combined with the “OR” operator.

JSON

Yes

configs.rules.action

The action to perform if conditions are met (for example, block access to the project).

JSON

Yes

Key

Value

Data type

Required during creation

configs.collector_сonfig

Parameters for collecting statistics (for example, the number of tasks skipped in the pool).

JSON

Yes

configs.rules.conditions

Conditions (for example, 10 task suites skipped in a row). Multiple conditions are combined with the “OR” operator.

JSON

Yes

configs.rules.action

The action to perform if conditions are met (for example, block access to the project).

JSON

Yes

configs.collector_сonfig key

Key

Value

Data type

Required during creation

type

Criteria for the quality control rule:

  • GOLDEN_SET — The number of correct and incorrect responses in the control tasks.

  • MAJORITY_VOTE — The percentage of responses that matched the majority vote.

  • CAPTCHA — The number of captchas entered successfully and unsuccessfully.

  • INCOME — Payment for tasks completed by the user over the past 24 hours.

  • SKIPPED_IN_ROW_ASSIGNMENTS — The number of task suites skipped in a row.

  • ANSWER_COUNT — The number of task suites completed by the user in the pool.

  • ASSIGNMENT_SUBMIT_TIME — The number of “fast” responses (the minimum response speed is set in the parameters).

  • ACCEPTANCE_RATE — The percentage of user responses that were rejected during non-automatic acceptance.

  • ASSIGNMENTS_ASSESSMENT — The number of assignments accepted or rejected with non-automatic acceptance enabled.

  • USERS_ASSESSMENT — The performer's skill value and their bans.

String

Yes

parameters

Parameters for collecting data (depends on the quality control rule specified in the type key).

JSON

For rules of type
GOLDEN_SET, MAJORITY_VOTE, CAPTCHA, ASSIGNMENT_SUBMIT_TIME.

Key

Value

Data type

Required during creation

type

Criteria for the quality control rule:

  • GOLDEN_SET — The number of correct and incorrect responses in the control tasks.

  • MAJORITY_VOTE — The percentage of responses that matched the majority vote.

  • CAPTCHA — The number of captchas entered successfully and unsuccessfully.

  • INCOME — Payment for tasks completed by the user over the past 24 hours.

  • SKIPPED_IN_ROW_ASSIGNMENTS — The number of task suites skipped in a row.

  • ANSWER_COUNT — The number of task suites completed by the user in the pool.

  • ASSIGNMENT_SUBMIT_TIME — The number of “fast” responses (the minimum response speed is set in the parameters).

  • ACCEPTANCE_RATE — The percentage of user responses that were rejected during non-automatic acceptance.

  • ASSIGNMENTS_ASSESSMENT — The number of assignments accepted or rejected with non-automatic acceptance enabled.

  • USERS_ASSESSMENT — The performer's skill value and their bans.

String

Yes

parameters

Parameters for collecting data (depends on the quality control rule specified in the type key).

JSON

For rules of type
GOLDEN_SET, MAJORITY_VOTE, CAPTCHA, ASSIGNMENT_SUBMIT_TIME.

configs.rules.conditions key

Key

Value

Data type

Required during creation

key

The value that is checked in the condition (for example, the number of task suites skipped).

Multiple conditions with the same key value are combined using the AND operator.

String

Yes

operator

Comparison operator (the key data is compared with the threshold value from value):

  • EQ (“Equal”)

  • NE (“Not equal to”)

  • GT (“Greater than”)

  • LT (“Less than”)

  • GTE (“Greater than or equal to”)

  • LTE (“Less than or equal to”)

String

Yes

value

The threshold value of the variable specified in key.

Number

Yes

Key

Value

Data type

Required during creation

key

The value that is checked in the condition (for example, the number of task suites skipped).

Multiple conditions with the same key value are combined using the AND operator.

String

Yes

operator

Comparison operator (the key data is compared with the threshold value from value):

  • EQ (“Equal”)

  • NE (“Not equal to”)

  • GT (“Greater than”)

  • LT (“Less than”)

  • GTE (“Greater than or equal to”)

  • LTE (“Less than or equal to”)

String

Yes

value

The threshold value of the variable specified in key.

Number

Yes

action key

Key

Value

Data type

Required during creation

type

Type of action:

  • RESTRICTION — Ban access to projects or pools.

  • SET_SKILL_FROM_OUTPUT_FIELD — Set the “percentage of correct responses” as the skill value (used in control tasks and majority vote rules).

    You can use the skill value for filtering users.

  • CHANGE_OVERLAP — Change the overlap. For example, to re-assign a task suite to other performers or cancel the recompletion of already accepted assignments.

  • REJECT_ALL_ASSIGNMENTS — Reject all performer responses. For example, after a certain number of performer responses, it became clear that the performer completed tasks poorly.

  • APPROVE_ALL_ASSIGNMENTS — Accept all performer responses. For example, if the performer completes most tasks well and you are satisfied with this result.

  • SET_SKILL — Assign the specified constant value to the skill.

String

Yes

parameters

Action parameters.

JSON

Yes

parameters.scope

Scope:

  • POOL — pool. Affects the user's rating.

  • PROJECT — The project. Affects the user's rating.

  • ALL_PROJECTS — All the requester's projects.

String

Yes

parameters.duration_unit

Ban duration unit:

  • MINUTES — Minutes

  • HOURS — Hours

  • DAYS — Days

  • PERMANENT — Permanent ban

String

No

parameters.duration

Ban duration.

Integer

No

parameters.private_comment

Comments (the reason for blocking access). Visible only to the requester.

String

No

parameters.skill_id

ID of the skill to update as tasks are completed.

String

Yes, if the type of action is selected
type=SET_SKILL_FROM_OUTPUT_FIELD
parameters.from_field

The value to assign to the skill:

  • correct_answers_rate — The percentage of correct responses.

  • wrong_answers_rate — The percentage of incorrect responses.

String

Yes, if the type of action is selected
type=SET_SKILL_FROM_OUTPUT_FIELD
parameters.skill_value

A fixed value to assign to the skill (a number from 0 to 100).

Number

Yes, if the type of action is selected
type=SET_SKILL_FROM_OUTPUT_FIELD
parameters.delta
The value determines the amount to change the overlap by. Number Yes, if the type of action is selected
type=CHANGE_OVERLAP
parameters.open_pool
Determines whether to re-open a closed pool:
  • true — Open the pool after making changes if it is closed.
  • false — Don't open the pool after making changes, if it is closed.
Boolean No
parameters.public_comment
Comments (the reason for rejecting responses). Available to the requester and the performer. String Yes, if the type of action is selected
type=REJECT_ALL_ASSIGNMENTS

Key

Value

Data type

Required during creation

type

Type of action:

  • RESTRICTION — Ban access to projects or pools.

  • SET_SKILL_FROM_OUTPUT_FIELD — Set the “percentage of correct responses” as the skill value (used in control tasks and majority vote rules).

    You can use the skill value for filtering users.

  • CHANGE_OVERLAP — Change the overlap. For example, to re-assign a task suite to other performers or cancel the recompletion of already accepted assignments.

  • REJECT_ALL_ASSIGNMENTS — Reject all performer responses. For example, after a certain number of performer responses, it became clear that the performer completed tasks poorly.

  • APPROVE_ALL_ASSIGNMENTS — Accept all performer responses. For example, if the performer completes most tasks well and you are satisfied with this result.

  • SET_SKILL — Assign the specified constant value to the skill.

String

Yes

parameters

Action parameters.

JSON

Yes

parameters.scope

Scope:

  • POOL — pool. Affects the user's rating.

  • PROJECT — The project. Affects the user's rating.

  • ALL_PROJECTS — All the requester's projects.

String

Yes

parameters.duration_unit

Ban duration unit:

  • MINUTES — Minutes

  • HOURS — Hours

  • DAYS — Days

  • PERMANENT — Permanent ban

String

No

parameters.duration

Ban duration.

Integer

No

parameters.private_comment

Comments (the reason for blocking access). Visible only to the requester.

String

No

parameters.skill_id

ID of the skill to update as tasks are completed.

String

Yes, if the type of action is selected
type=SET_SKILL_FROM_OUTPUT_FIELD
parameters.from_field

The value to assign to the skill:

  • correct_answers_rate — The percentage of correct responses.

  • wrong_answers_rate — The percentage of incorrect responses.

String

Yes, if the type of action is selected
type=SET_SKILL_FROM_OUTPUT_FIELD
parameters.skill_value

A fixed value to assign to the skill (a number from 0 to 100).

Number

Yes, if the type of action is selected
type=SET_SKILL_FROM_OUTPUT_FIELD
parameters.delta
The value determines the amount to change the overlap by. Number Yes, if the type of action is selected
type=CHANGE_OVERLAP
parameters.open_pool
Determines whether to re-open a closed pool:
  • true — Open the pool after making changes if it is closed.
  • false — Don't open the pool after making changes, if it is closed.
Boolean No
parameters.public_comment
Comments (the reason for rejecting responses). Available to the requester and the performer. String Yes, if the type of action is selected
type=REJECT_ALL_ASSIGNMENTS