Control tasks contain correct responses and are used to calculate the accuracy of the user responses.
Add control tasks to the pool and set up a quality control rule, for example: “If the user answered correctly, award skill points”. Awarded skill points can be used for selecting users: for example, only issue tasks to users who have at least 60% correct responses. The skill filter begins working after a user has completed N control tasks in the pool (the number of tasks is defined by the requester, see below). As tasks are completed, the skill level is recalculated and re-compared with the selection value.
For reliable quality control, the control tasks should make up at least 10% of the volume of the pool. You can create control tasks using completed tasks from closed pools or tasks received by the absolute majority.
The skill level is calculated using the formula:
is the number of correct responses on completed control tasks.
is the number of completed control tasks.
If task access is blocked temporarily (for example, for 7 days), the history of responses is deleted after the ban is lifted. The skill is calculated based in the new responses.
To use control tasks for checking:
Create a skill on the Skills page in the production version of Toloka.
Set up a quality control rule on the pool page (click Add Quality Control Rule → Golden set). Fill in the fields:
Field or group of fields
The number of recent responses in the project's control tasks used for calculating skill.
If this field is empty, skill is calculated from responses in the pool's control tasks.
A condition for performing the action in the then field. For example: “Number of responses ≥ 10”.
Action to perform for the condition:
Add a skill filter on the pool page (in the Users filter section). For example: “The myskill skill is missing or ≥ 60”.