Control tasks

Add a rule to assign a skill to the performer based on their responses to control tasks and ban the performers who give incorrect responses.

Attention.

The If and then fields in this rule are mandatory. If you don't fill in at least one field, you won't be able to save the rule.

Rule settings


Field	Description
Recent values to use	The number of the performer's last responses to control tasks. To calculate control task responses from all project pools, fill in the field in the rule for each pool. If this field is not filled in, the calculation includes responses in control tasks only in the pool to which the rule applies.
If	A condition for performing the action in the then field: number of responses — The number of control and training tasks completed. correct responses (%) — The percentage of correct responses in training and control tasks (from 0 to 100). incorrect responses (%) — The percentage of incorrect responses in training and control tasks (from 0 to 100). number of control responses — The number of completed control tasks. correct control responses (%) — The percentage of correct responses in control tasks (from 0 to 100). incorrect control responses (%) — The percentage of incorrect responses in control tasks (from 0 to 100). To add multiple conditions, click .
then	Action to perform for the condition: assign skill from the field — Save the percentage of the performer's correct responses in control tasks as a skill value. Assign skill value — Assign a fixed value to the skill. accept user's answers — Requires the non-automatic acceptance option to be set. Useful if the performer completes most tasks well. Example: the toloker completed more than 80% of the tasks correctly and you are satisfied with this result. The rule will work automatically — all responses in the pool will be accepted. decline user's answers — Requires the non-automatic acceptance option to be set. Useful if you need several responses from the performer to evaluate the quality of the results. Example: after ten responses, it became clear that the performer completes tasks poorly. The configured action will work — all responses in the pool will be rejected. Suspend — Suspend the performer's access to the pool for the specified number of days. Only the requester can view the reason for the ban. Ban — Block access to the project or all requester's projects for the specified number of days. Only the requester can view the reason for the ban. If access to tasks is blocked temporarily (for example, for 7 days), the history of the performer's responses is not saved after the ban is lifted. The skill value is based on new responses.


Field	Description
Recent values to use	The number of the performer's last responses to control tasks. To calculate control task responses from all project pools, fill in the field in the rule for each pool. If this field is not filled in, the calculation includes responses in control tasks only in the pool to which the rule applies.
If	A condition for performing the action in the then field: number of responses — The number of control and training tasks completed. correct responses (%) — The percentage of correct responses in training and control tasks (from 0 to 100). incorrect responses (%) — The percentage of incorrect responses in training and control tasks (from 0 to 100). number of control responses — The number of completed control tasks. correct control responses (%) — The percentage of correct responses in control tasks (from 0 to 100). incorrect control responses (%) — The percentage of incorrect responses in control tasks (from 0 to 100). To add multiple conditions, click .
then	Action to perform for the condition: assign skill from the field — Save the percentage of the performer's correct responses in control tasks as a skill value. Assign skill value — Assign a fixed value to the skill. accept user's answers — Requires the non-automatic acceptance option to be set. Useful if the performer completes most tasks well. Example: the toloker completed more than 80% of the tasks correctly and you are satisfied with this result. The rule will work automatically — all responses in the pool will be accepted. decline user's answers — Requires the non-automatic acceptance option to be set. Useful if you need several responses from the performer to evaluate the quality of the results. Example: after ten responses, it became clear that the performer completes tasks poorly. The configured action will work — all responses in the pool will be rejected. Suspend — Suspend the performer's access to the pool for the specified number of days. Only the requester can view the reason for the ban. Ban — Block access to the project or all requester's projects for the specified number of days. Only the requester can view the reason for the ban. If access to tasks is blocked temporarily (for example, for 7 days), the history of the performer's responses is not saved after the ban is lifted. The skill value is based on new responses.

Examples of rules

Calculating a skill and banning for incorrect responses

Both rules work independently:

If the performer gives at least 3 responses to the control tasks, the percentage of correct answers is assigned as the skill value.
If the performer gives at least 3 answers to the control tasks and the percentage of correct answers is less than 60%, they lose access to the project.

The calculation uses up to 10 of the performer's recent responses to the project's control tasks.

The performer is banned after the first incorrect response to the first, second or third control task. The skill is not assigned. Since the ban reason is not specified, there is no way to find out why the performer is banned.

All rules are applied independently:

If the performer gives at least 3 responses to the control and training tasks, the percentage of correct answers is assigned as the skill value.
If the performer gives 2 incorrect answers to 3 control tasks, they lose access to the pool for 10 days.
If the performer gives 2 incorrect responses to 4 control tasks, they lose access to the pool for 10 days.
If the performer gives 5 or more responses to the control tasks and the percentage of correct responses is less than 80%, they lose access to the pool for 10 days.

A set of rules like this prevents performers from being banned for one incorrect response and lets you maintain high accuracy.

Setting the skill

If the performer completes 3 control or training tasks, they get a skill. Use the skill value to set access to other pools with filters.

Example of filter settings

This rule will never take effect because the number of responses counted (Recent values to use) is less than the number of responses in the rule (number of control responses).

Blocking for incorrect responses

If the percentage of correct responses in the control tasks is less than 40%, the performer loses access to the project for 30 days.

This rule doesn't take into account responses in the training tasks for banning.

If the percentage of correct responses in the control tasks is less than 40%, the performer loses access to the project for 30 days. The rule will be applied once — after the fifth response in the control task.

Troubleshooting

How many control tasks do I need to add?

We recommend adding at least 1% of control tasks in the pool. To filter out performers, use the Control tasks quality control rule. To rank performers by the quality of responses in control tasks, use a skill.

How are the correct responses to control questions counted?

The Control tasks rule starts working after the performer completes the number of control tasks you specified. If your pool contains both training and control tasks, you can take into account the responses in both of them (the Number of responses parameter) or only in control tasks (the Number of control responses parameter).

As soon as the needed number of responses is collected, Toloka calculates the percentage of correct and incorrect responses and performs an action (assigns a skill, or blocks the user in the pool or in the project). Then this percentage is updated as the tasks are completed by the performer. The number of the performer's last responses used for the calculation is set in the Recent values to use field. If you leave it empty, all the responses from the performer in the pool are counted.

Should I create a skill for every pool?

It is better to use one skill in a project. You can choose the way to calculate the skill:

Calculate the skill for each pool separately. The current skill value is the value of the skill in the pool the user completed last. This option is convenient if:
- The pools are intended for different groups of performers (for example, there are filters by city or country).
- Pools are started one by one and you don't want to take into account the responses in the previous pools to calculate the skill in the current pool.
This calculation method is used by default when adding a quality control rule to a pool. For the control tasks block, leave the Recent values to use field empty.
Calculate skill based on all tasks in a project This option is good if the pools are small and you don't need to have skill calculated for each pool.

This option is available only for skills on control tasks. To use it, fill in the Recent values to use field in quality control rules in pools.

Can I use a skill beyond a particular pool or project and apply it to other projects as well?

Yes, of course — you can use the same skill for different projects. But most often, a skill is intended for a specific project. If the performer completes a certain task well, this doesn't mean that they will complete other ones successfully. Another disadvantage is that if you filter by skills that were set long ago, you will artificially limit the number of available performers.

Contact support