Tips for designing tasks

To get more accurate responses, think through the structure and wording of the tasks and the settings for responses:

Make tasks as simple as possible. A good task contains a single question (or several questions of the same type).

This helps reduce the number of errors, since performers do better with short, uniform tasks. For example, you need to get product information: the name, manufacturer, price, photo, and description. It's best to create a separate project for each of these points.

Write instructions.

The instructions for completing a task should be:

  • Complete.

    Describe all possible situations the performer may face.

  • Concise.

    Use concise wording and simple sentences.

  • Structured.

    Give step-by-step instructions (if possible). Use lists and formatting to make the text easier to read.

  • Visual.

    Provide examples. Add images to them (if the tasks have them). For example, if a task requires evaluating the quality of an image, put both high-quality and low-quality images in the instructions. If the task is to identify the type of clothing in a picture, describe the possible options in the instructions and illustrate the text with images.

If you are reviewing assignments (the offline accept option), the instructions should briefly and clearly list the acceptance criteria.

Allow enough time for completing tasks.

Haste lowers the quality of responses, so you should allow a little extra time to complete the task.

Make training tasks.

Ask performers to go through the training tasks before beginning the pool tasks. Include disputable cases in the training tasks.

Choose the appropriate type of quality control.

If the task contains a simple question with a multiple-choice response and is completed fairly quickly (1-10 minutes), it is best to run the task with overlap and use majority vote checking, a golden set, a captcha, and restriction of quick responses.

If the task doesn't have clearly defined response options (for example, it requires creating or translating a text or transcribing an audio recording), you can use the following verification methods:

  • Check the responses yourself using the option to review assignments.

  • Submit the responses to Toloka performers for checking. Create a task with a question (for example, “Is this phrase translated correctly?”) and possible responses (for example, “yes”/“no”). Set up overlap and majority vote checking.

Award bonuses
You can issue bonuses to performers for completing tasks well. This improves motivation and makes performers approach your tasks with more care.
Keep in touch with performers
Proper communication with performers keeps them interested in the project and allows you to detect any problems in the tasks quickly. Answer users' questions in personal messages and on the forum.