Quality control rules

To get more accurate responses, set up quality control rules. Quality control rules help regulate access to a pool: you can filter out users who give incorrect responses in control tasks, skip too many tasks in a row, and so on. Tasks that were completed by such users can be sent for completion again. For this, define overlap recovery settings.

Quality control can be set in both a pool and a project. If rules are set in the project, they apply to all pools in the project.

  • To add quality control to a pool, switch to edit mode and click Add Quality Control Rule.
  • To add quality control to a project, open the project settings page, click Project actions, and then select Set quality control.

Designing a quality control rule

A rule consists of two parts: the condition (If), and the action to perform when the condition is met (then). If there are multiple conditions, they must all be met.

After checking the condition, you can perform one of the actions:

  • “Set skill from "If"” — Save the user's percentage of correct responses in the control tasks or tasks with majority vote. To save the percentage (a number between 1 and 100), use a skill — an assessment of the user's performance that has a name and value. For example, you can create the skill “Completed control tasks in my pool” and use it for saving the percentage of correct responses for each user. Then you can set a filter using this skill to control users' access to the pool.

  • “Set skill” — Assign the skill a fixed value of N (from 1 to 100).

  • “Pause” — Suspend the user's access to the pool for the specified number of days. The reason for banning is only shown to the requester.

  • “Ban” — Block access to the project or all of the requester's projects for the specified number of days. The reason for banning is only shown to the requester.

Rules

Golden set
Correct and incorrect responses in control tasks. For the rule settings, see Golden set section.
Majority vote

The user's responses matching responses from other users. For the rule settings, see Majority vote.

Quick answers

The response time in tasks. Example:

Fields:

  • History size — The number of user's recent responses in the project used for the calculation. If the field is empty, the calculation is based on all user's responses in the pool.
  • Quick answers limit — What answers are considered as “fast” (time in seconds).
  • Quick answers count — allowed number of quick answers.
  • Answers collected — The number of the user's recent responses (less or equal to the number in the History size field).
Skipped assignments

The number of task suites skipped in a row. Example:

Income

The user's earnings in the pool or in the project over the past 24 hours.

Captcha

Correct and incorrect responses to a captcha.

Fields:

  • History size — The number of user's recently completed tasks in the project used for the calculation. If the field is empty, the calculation is based on all user's responses in the pool.
  • % correct answers — proportion of correct answers.
  • % incorrect answers — proportion of incorrect answers.

You should also specify the captcha frequency in the pool settings.

Submitted answers

The number of task suites in the pool completed by the user.

Reviewing assignments

The proportion of assignments accepted or rejected during assignments review.

Fields:

  • History size — The number of user's recent responses in the project used for the calculation. If the field is empty, the calculation is based on all user's responses in the pool.

  • Reviewed assignments — The number of user's assignments that were reviewed.

  • % accepted assignments — Proportion of user's assignments that were accepted.

  • % rejected assignments — Proportion of user's assignments that were rejected.

There are also sets of predefined rules.