Golden set

Control tasks contain correct responses and are used to calculate the accuracy of the user responses.

Add control tasks to the pool and set up a quality control rule, for example: “If the user answered correctly, award skill points”. Awarded skill points can be used for selecting users: for example, only issue tasks to users who have at least 60% correct responses. The skill filter begins working after a user has completed N control tasks in the pool (the number of tasks is defined by the requester, see below). As tasks are completed, the skill level is recalculated and re-compared with the selection value.

For reliable quality control, the control tasks should make up at least 10% of the volume of the pool. You can create control tasks using completed tasks from closed pools or tasks received by the absolute majority.

The skill level is calculated using the formula:

,

where:

is the number of correct responses on completed control tasks.

is the number of completed control tasks.

If task access is blocked temporarily (for example, for 7 days), the history of responses is deleted after the ban is lifted. The skill is calculated based in the new responses.

Settings for quality control rules.

To use control tasks for checking:

  1. Create a skill on the Skills page in the production version of Toloka.

  2. Set up a quality control rule on the pool page (click Add Quality Control Rule → Golden set). Fill in the fields:

    Field or group of fields

    Description

    History size

    The number of user's recent responses in the project's control tasks used for the calculation.

    If the field is empty, the calculation is based on all user's responses in the pool.

    If

    A condition for performing the action in the then field. For example, “Number of responses ≥ 10”.

    then

    Action to perform for the condition:

    • “Set skill from "If"” — Save the user's percentage of correct responses in the control tasks or tasks with majority vote. To save the percentage (a number between 1 and 100), use a skill — an assessment of the user's performance that has a name and value. For example, you can create the skill “Completed control tasks in my pool” and use it for saving the percentage of correct responses for each user. Then you can set a filter using this skill to control users' access to the pool.

    • “Set skill” — Assign the skill a fixed value of N (from 1 to 100).

    • “Pause” — Suspend the user's access to the pool for the specified number of days. The reason for banning is only shown to the requester.

    • “Ban” — Block access to the project or all of the requester's projects for the specified number of days. The reason for banning is only shown to the requester.

  3. Add a skill filter on the pool page (in the Users filter section). For example, “The myskill skill is missing or ≥ 60”.