Tips for designing tasks

To get more accurate responses, think through the structure and wording of the tasks and the settings for responses:

Make a task as simple as possible. Divide complex tasks into several projects.

Performers do better with short, uniform tasks. Therefore, if your task is large or consists of several stages, divide it in several simple tasks and create a project for each task or each stage.

For example, if you need to have road signs recognized, make a separate project for each of the stages: filtering out photos without road signs, selecting road signs in a photo, recognizing road signs, and reviewing the results.

Write good instructions

Instructions for completing a task should be:

  • Thorough. Describe all possible situations the performer may face.

  • Concise. Use concise wording and simple sentences.

  • Structured. Give step-by-step instructions (if possible). Use lists and formatting to make the text easier to read.

  • Visual. Provide examples. Add images to them (if the tasks have them).

    For example, if a task requires evaluating the quality of an image, put both high-quality and low-quality images in the instructions. If the task is to identify the type of clothing in a picture, describe the possible options in the instructions and illustrate the text with images.

If you review completed assignments using non-automatic acceptance, list the evaluation criteria in the instructions.

Create a simple and user-friendly task interface

Make the task interface effective using settings.

  • To help performers work faster, add hotkeys to the task interface.
  • To ensure that performers make fewer mistakes, add response validation.
  • Don't overload the interface. If there are too many elements in the interface, you might want to divide the project into several smaller projects.
  • Check the performance of projects and their display on computer and mobile device screens. Use the sandbox to tune a task.
Allow enough time for completing tasks.

Haste lowers the quality of responses, so you should allow a little extra time to complete the task.

Make training tasks.

Invite performers to pass the training before they start the main pool with tasks. Include disputable cases in the training tasks.

Select performers

Make sure to specify the language of the task instruction and text using filters. Add a skill filter and set the Speed/quality balance slider so that only performers with a high rating can participate in the project. For tasks to be completed in mobile apps, set the Client = Mobile Toloka filter.

Add an exam based on the training task responses to select performers. To do this, assign skills depending on whether their responses are correct.

Choose the appropriate type of quality control.

If the task contains a simple question with a multiple-choice response and is completed fairly quickly (1-10 minutes), it is best to run the task with overlap and use majority vote check, control tasks, captcha, and restriction of fast responses.

If the task doesn't have clearly defined response options (for example, it requires creating or translating a text or transcribing an audio recording), you can use the following review methods:

  • Check the responses yourself using the non-automatic acceptance option.

  • Submit the responses to Yandex.Toloka performers for checking. Create a task with a question (for example, “Is this phrase translated correctly?”) and possible responses (for example, “yes”/“no”). Set up overlap and majority vote checking.

Set a fair price and issue rewards
Set a fair price: the more complicated the task is and the more time the performer spends, the higher the price should be. Use dynamic pricing to motivate diligent performers.
You can issue rewards to performers for completing tasks well. This improves motivation and encourages performers to approach your tasks with care.
Keep in touch with performers
Proper communication with performers keeps them interested in the project and lets you quickly detect any issues with the tasks. Answer users' questions through private messages.

Troubleshooting

What is the right time limit for the task completion?
Try completing the tasks yourself. Ask your colleagues and friends to complete them. Find out average completion time and add 50% to it.
How many tasks should be on the page?

The number of tasks depends on how difficult and time-consuming the tasks are. Don't make task pages too large. They are unpopular, partly because they are inconvenient for performers (for example, if the internet connection is unstable).

How do I set quality control in a pool correctly?

The settings for quality control rules depend on the type of tasks. General recommendations:

  • Always use one or more ways to control quality of answers.

  • Counting fast responses makes sense for most tasks.

  • If the user has to choose between options (for example, by selecting checkboxes), check the answers using majority vote or control tasks.

  • If the user has to provide a response as a text or link or upload a photo, the best way to control quality is by reviewing assignments. You can outsource task acceptance to performers. Create tasks with a question (for example, "Is the phrase translated correctly?") and response options (for example, "Yes"/"No"). Set up overlap and majority vote check.

  • If a task is more like an opinion poll (for example, choosing nice pictures from a set), majority vote is not a good way to control quality. Make control tasks with artificial examples where the choice is evident.

Can I disable tasks for performers who complete the tasks poorly?

You can deny access to the pool if the performer's responses are too fast, if they don't match the majority opinion or if the performer makes many mistakes in control tasks. Tasks completed by such performers can be given to other performers.