How to check performers: quality control rules

Quality control rules regulate task completion and performer access. They are set up both for a project as a whole on the Quality control tab and for each pool individually.

Monitoring performer actions

If your project has a lot of similar tasks, you'll need to protect it from fake users (robots) and cheaters.

Limiting fast responses

Configure the Fast responses rule to filter out cheating performers who don't handle tasks carefully and complete them too quickly. You've tried doing tasks and you know that each task takes at least 15 seconds. You should suspend performers who submit three out of ten assignments in a row faster than the control time.

Limit on skipped assignments

Another “optimization” technique that cheaters use is skipping certain assignments. They pick out the ones that seem easier to them, while other performers reliably complete all the assignments they receive. However, both get paid in the same way.

Configure the Skipped assignments rule to avoid this situation. It will restrict access for those performers who skip too many assignments, like more than three in a row.

Captcha

To protect against robots, use the Captcha rule. In Toloka, a captcha is shown every 10 task pages. If assignments are completed quickly and captchas get in the way, they can be shown every 20 task pages: specify low captcha frequency in the pool settings.

Limit on available tasks

Limits are also useful when you need as many different performers as possible, like to collect statistics for a survey or choosing the best version of a logo. Another case is personal descriptions for a dating service. If texts are all written by the same person, they are likely to turn out monotonous and boring.

You can engage more participants if you limit the number of assignments “per performer”. You can do this by setting a limit on the number of tasks available to each performer or a limit on daily earnings in a single project. Use the Submitted responses and Earnings rules.

Task completion quality control

Results from conscientious performers should be checked, too. In Toloka, this can be done automatically.

Control tasks

In some tasks, performers are given a choice of two or more response options. To check the quality in this type of project, use the Control tasks rule. This involves adding questions where you define the correct response.

Mix the control questions in with the main tasks. They should make up at least 1% of the total tasks. Based on the number of errors, you'll evaluate performers and assign them a skill level from 0 to 100.

Those who make a critical number of errors will be automatically excluded from your tasks by skill filters. Pay more to highly skilled performers (who score 90 and higher) as an incentive. This encourages performers to try harder on tasks.

Majority vote

Sometimes requesters aren't able to regularly add control tasks and keep them updated.

In this case, the Majority vote rule is helpful. The platform assigns the same task to multiple performers. For example, five people get the exact same task. If three of them choose the same answer, we assume it is correct.

Use the percentage of matching responses to assign skill levels, control access to tasks, and reward performers.

Review results

Tasks where each performer has to give a unique response, like recording audio, taking a photo, or writing a text, are reviewed by requesters themselves or by other performers. Poor-quality responses are rejected.

Use the Review results rule to control performers' access to tasks. The platform calculates the percentage of responses accepted and rejected for each performer. Those who often made mistakes will stop getting tasks.

Recompletion of rejected assignments

Quality control rules let you minimize the consequences of errors. In Toloka, you can choose to resend for recompletion just individual rejected assignments, or all responses from performers who made too many mistakes.

Rejected tasks

For example, you can use Toloka to improve speech recognition algorithms. Performers listen to a series of 5-10-second recordings and make a transcription of each of them in order to convert speech to text.

Some of the recordings are transcribed incorrectly. Don't accept incorrect responses. Set up the Processing rejected and accepted assignments rule. The platform will automatically send incorrect transcriptions to other performers for revision.

Responses from banned performers

If a user makes mistakes too many times, the platform bans them from taking more tasks. But the responses that they have already provided remain in the database and may get into the final dataset. Send these assignments for recompletion.

The Recompletion of assignments from banned users rule is helpful here. It increases the cost of labeling, but protects the dataset from possible errors.

As soon as a performer loses access to a project, like due to a low skill level, Toloka automatically sends all their completed assignments to other performers for re-labeling.

General recommendations

Each project requires its own approach. Methods that are effective in one case may be pointless in another. But there are some general recommendations:

  1. Analyze the task and select the appropriate quality control rules.
  2. Write clear instructions. If you create control tasks, make sure they don't have any outdated, ambiguous, or incorrect responses.
  3. Test your project in the testing environment, the Sandbox. Running and completing tasks in it will help you understand the rules and estimate how much time it will take to complete a single task.
  4. Launch the first real project with a small number of tasks. Analyze the results and adjust the settings if necessary.
  5. Monitor the labeling results and listen to feedback from performers.
  6. Contact support If something isn't working.