Golden set

Control tasks contain correct responses and are used to calculate the accuracy of the performer's responses.

Add control tasks to the pool and set up a quality control rule, for example: “If the performer answered correctly, award skill points”. Awarded skill points can be used for selecting performers: for example, you can issue tasks only to performers who have at least 60% correct responses. The skill filter begins working after the performer completes $\text{[math]}$ control tasks in the pool (the number is defined by the requester, see below). As tasks are completed, the skill level is recalculated and re-compared with the selection value.

For reliable quality control, the control tasks should make up at least 10% of the volume of the pool. You can create control tasks using completed tasks from closed pools or tasks received by the absolute majority.

The skill level is calculated using the formula:

$\text{[math]}$ ,

where:

$\text{[math]}$ is the number of correct responses in the completed control tasks.

$\text{[math]}$ is the number of completed control tasks.

If access to tasks is blocked temporarily (for example, for 7 days), the history of the performer's responses is not saved after the ban is lifted. The skill is calculated based on the new responses.

Settings for quality control rules

To use control tasks for checking:

Create a skill on the Skills page.

Set up a quality control rule on the pool page (click Add Quality Control Rule → Golden set). Fill in the fields:


Field or group of fields	Description
History size	The number of the performer's recent responses in the project's control tasks used for the calculation. If the field is empty, the calculation is based on all the performer's responses in the pool.
If	A condition for performing the action in the then field: Answer count — The number of completed training and control tasks. correct_answers — The percentage of correct responses in training and control tasks (from 0 to 100). %incorrect answers — The percentage of incorrect responses in training and control tasks (from 0 to 100). golden answers count — The number of completed control tasks. % correct golden answers — The percentage of correct responses in control tasks (from 0 to 100). % incorrect golden answers — The percentage of incorrect responses in control tasks (from 0 to 100).
then	Action to perform for the condition: “Set skill from "If"” — Save the performer's percentage of correct responses in the control tasks or tasks with majority vote. To save the percentage (a number between 1 and 100), use a skill — an assessment of the performer's work that has a name and value. For example, you can create a skill called “Completing control tasks in my pool” and use it for saving the percentage of correct responses for each performer. Then you can set a filter using this skill to control performers' access to the pool. “Set skill” — Assign the skill a fixed value of N (from 1 to 100). “Pause” — Suspend the performer's access to the pool for the specified number of days. The reason for banning is only shown to the requester. “Ban” — Block access to the project or all of the requester's projects for the specified number of days. The reason for banning is only shown to the requester.


Field or group of fields	Description
History size	The number of the performer's recent responses in the project's control tasks used for the calculation. If the field is empty, the calculation is based on all the performer's responses in the pool.
If	A condition for performing the action in the then field: Answer count — The number of completed training and control tasks. correct_answers — The percentage of correct responses in training and control tasks (from 0 to 100). %incorrect answers — The percentage of incorrect responses in training and control tasks (from 0 to 100). golden answers count — The number of completed control tasks. % correct golden answers — The percentage of correct responses in control tasks (from 0 to 100). % incorrect golden answers — The percentage of incorrect responses in control tasks (from 0 to 100).
then	Action to perform for the condition: “Set skill from "If"” — Save the performer's percentage of correct responses in the control tasks or tasks with majority vote. To save the percentage (a number between 1 and 100), use a skill — an assessment of the performer's work that has a name and value. For example, you can create a skill called “Completing control tasks in my pool” and use it for saving the percentage of correct responses for each performer. Then you can set a filter using this skill to control performers' access to the pool. “Set skill” — Assign the skill a fixed value of N (from 1 to 100). “Pause” — Suspend the performer's access to the pool for the specified number of days. The reason for banning is only shown to the requester. “Ban” — Block access to the project or all of the requester's projects for the specified number of days. The reason for banning is only shown to the requester.

Add a skill filter on the pool page (in the Users filter section). For example, “The myskill skill is missing or ≥ 60”.