Setting up quality control
Quality control lets you get more accurate responses and restrict access to tasks for cheating performers. Quality control consists of rules. All rules work independently.
Quality control rules that you set in the project are applied to all project pools, so you can't change them in one pool.
- Go to the pool editing page
- If you already have a pool with the appropriate quality control settings, you can copy it along with the audience settings. To do this, go to Users filter and click Copy settings from... and then Add Quality Control Rule.
Under Quality Control, choose the rules you want to use.
If you aren't sure what quality control rules you need, select a quality control preset with default settings.- Make settings for the rules you added. Below is a list of rules with links to detailed information about the rule settings.
- Save the pool.
List of rules
- To keep track of how often performers make mistakes:
- Control tasks: Use them to assign a skill to performers based on their responses to control tasks and ban performers who submit incorrect responses.
- Majority vote: Quality is based on matching the response from the majority of performers who complete the same task.
- Results of checking: Evaluate performers based on the number of accepted and rejected responses.
- To protect your project from fake users (robots) and cheaters:
- Fast responses: Control the minimum time that must be spent on a task page.
- Captcha: Show a captcha from time to time to make sure tasks aren't completed by robots.
- Skipped assignments: Restrict access to your pool tasks for performers who skip several task pages in a row.
- To attract a variety of performers:
- Submitted assignments: Limit how many assignments each performer can submit in the pool per day.
- To allow recompletion of certain assignments:
- Recompletion of assignments from banned users: Send completed assignments to other performers to redo them if the performer was banned.
- Processing of rejected and accepted assignments: Send rejected assignments to other performers to redo them.
What's next
- Add tasks to the pool.
- Learn more about how to set up a pool:
- Selective majority vote control.
Troubleshooting
Always use one or more ways to control quality of answers.
Counting fast responses makes sense for most tasks.
If the user has to choose between options (for example, by selecting checkboxes), check the answers using majority vote or control tasks.
If the user has to provide a response as a text or link or upload a photo, the best way to control quality is by reviewing assignments. You can outsource task acceptance to performers. Create tasks with a question (for example, "Is the phrase translated correctly?") and response options (for example, "Yes"/"No"). Set up overlap and majority vote check.
If a task is more like an opinion poll (for example, choosing nice pictures from a set), majority vote is not a good way to control quality. Make control tasks with artificial examples where the choice is evident.
Calculate the skill for each pool separately. The current skill value is the value of the skill in the pool the user completed last. This option is convenient if:
The pools are intended for different groups of performers (for example, there are filters by city or country).
Pools are started one by one and you don't want to take into account the responses in the previous pools to calculate the skill in the current pool.
This calculation method is used by default when adding a quality control rule to a pool. For the control tasks block, leave the Recent values to use field empty.
Calculate skill based on all tasks in a project This option is good if the pools are small and you don't need to have skill calculated for each pool.
This option is available only for skills on control tasks. To use it, fill in the Recent values to use field in quality control rules in pools.
The settings for quality control rules depend on the type of tasks. General recommendations:
We recommend adding at least 1% of control tasks in the pool. To filter out performers, use the Control tasks quality control rule. To rank performers by the quality of responses in control tasks, use a skill.
The Control tasks rule starts working after the performer completes the number of control tasks you specified. If your pool contains both training and control tasks, you can take into account the responses in both of them (the Number of responses parameter) or only in control tasks (the Number of control responses parameter).
As soon as the needed number of responses is collected, Toloka calculates the percentage of correct and incorrect responses and performs an action (assigns a skill, or blocks the user in the pool or in the project). Then this percentage is updated as the tasks are completed by the performer. The number of the performer's last responses used for the calculation is set in the Recent values to use field. If you leave it empty, all the responses from the performer in the pool are counted.
It is better to use one skill in a project. You can choose the way to calculate the skill:
Yes, of course — you can use the same skill for different projects. But most often, a skill is intended for a specific project. If the performer completes a certain task well, this doesn't mean that they will complete other ones successfully. Another disadvantage is that if you filter by skills that were set long ago, you will artificially limit the number of available performers.
You can deny access to the pool if the performer's responses are too fast, if they don't match the majority opinion or if the performer makes many mistakes in control tasks. Tasks completed by such performers can be given to other performers.
