Quality control

To get more accurate responses, set up quality control rules. Quality control rules help regulate access to a pool: you can filter out performers who give incorrect responses in control tasks, skip too many tasks in a row, and so on. The tasks completed by such users can be given out again. To do this, define overlap recovery settings.

You can configure quality control in the pool and in the project. In the latter case, the rules will apply to all pools in the project.

  • To set up quality control in the pool, switch to the pool editing mode and click the Add Quality Control Rule button.
  • To set up quality control in the project, open the project properties page and click the Project actions button at the top, then choose Set quality control.

Block structure

A rule consists of two parts: the condition (If), and the action to perform when the condition is met (then). If there are several conditions, all of them must be met.

After checking the condition, you can perform one of the actions:

  • “Set skill from "If"” — Save the performer's percentage of correct responses in the control tasks or tasks with majority vote. To save the percentage (a number between 1 and 100), use a skill — an assessment of the performer's work that has a name and value. For example, you can create a skill called “Completing control tasks in my pool” and use it for saving the percentage of correct responses for each performer. Then you can set a filter using this skill to control performers' access to the pool.

  • “Set skill” — Assign the skill a fixed value of N (from 1 to 100).

  • “Pause” — Suspend the performer's access to the pool for the specified number of days. The reason for banning is only shown to the requester.

  • “Ban” — Block access to the project or all of the requester's projects for the specified number of days. The reason for banning is only shown to the requester.

Rules

Golden set
Correct and incorrect responses in control tasks. For the rule settings, see the Golden set section.
Majority vote

The performer's responses matching responses from other performers. For the rule settings, see Majority vote.

Quick answers

The response time for a task page. Example:

Fields:

  • History size — The number of the performer's recently completed task pages in the project used for the calculation. If the field is not filled in, the calculation takes into account all pages completed by the performer in the pool.
  • Quick answers limit — What responses are considered “too fast” (time in seconds).
  • Quick answers count — Allowed number of quick answers.
  • Answers collected — The number of the performer's recent responses (less than or equal to the number in the History size field).
Skipped assignments

The number of task pages skipped in a row. Example:

Income

The performer's earnings in the pool or the project over the past 24 hours.

Captcha

Correct and incorrect responses to a captcha.

Fields:

  • History size — The number of the performer's recently completed task pages in the project used for the calculation. If the field is empty, the calculation is based on all the user's responses in the pool.
  • % correct answers — percentage of correct answers.
  • % incorrect answers — percentage of incorrect answers.

You must also choose captcha frequency in the pool settings.

Submitted answers

The number of task pages in the pool completed by the performer.

Reviewing assignments

The percentage of assignments accepted or rejected during assignments review.

Fields:

  • History size — The number of the user's recent responses in the project used for the calculation. If the field is empty, the calculation is based on all the performer's responses in the pool.

  • Reviewed assignments — The number of the performer's assignments that were reviewed.

  • % accepted assignments — Percentage of the performer's assignments that were accepted.

  • % rejected assignments — Percentage of the performer's assignments that were rejected.

In addition to the individual rules, you can use prepared rule sets.