Toloka

Workflow

To publish tasks in Toloka, first you need to create them and debug them in the testing environment, the sandbox. If all the settings are working correctly, you can move them to the production version of Toloka and start distributing tasks to users.

To start tasks and get responses:

  1. Register in the sandbox as a requester and set up tasks:

    1. Create a project and a pool with several tasks.

    2. Set up quality control rules, user selection, and offline acceptance of tasks.

  2. Try completing your tasks in the sandbox as a user:

    1. Create user accounts (see the instructions in the user documentation).

    2. Log in to the sandbox with the requester username and click Add trusted users on the Users page.

    3. Click Add user and enter the usernames of the accounts you created.

      Only trusted users can access your tasks.

    4. Check the tasks in the sandbox with the username of a trusted user. Make sure that the buttons and response validation are set up correctly, and assess how long it takes to complete the tasks. Check how user selection and quality control work.

  3. Register in the production version and add funds to your Toloka account.

  4. Move the project, pool, and settings to the production version of Toloka.

    1. Add OAuth-token from the production version to the sandbox profile page (External services Integration tab).

    2. Go to the project page in the sandbox and click the EditExport button.

    3. Select pools for export.

      When you select a pool with main tasks, pool with training tasks is exported automatically (if they are linked in the settings).

  5. Upload the tasks in a TSV file into the production version of Toloka and start the pool.

    Before starting all the tasks, send out a small trial pool for completion. The trial pool can contain from 10 to 100 tasks. Start the pool, get responses, and analyze them. If the responses aren't satisfactory, try to figure out why. Perhaps you need to edit the instructions, add training tasks, or change something else.

    After starting the tasks, watch for incoming messages from users during the first hour. Users normally react quickly to problems and ask questions if something isn't right.

  6. When the tasks are completed, download the received responses.

  7. Check the responses if you set up offline acceptance when creating the pool.

Tips for designing tasks

To get more accurate responses, think through the structure and wording of the tasks and the settings for responses:

Make tasks as simple as possible. A good task contains a single question (or several questions of the same type).

This helps reduce the number of errors, since users do better with short, uniform tasks. For example, you need to get product information: the name, manufacturer, price, photo, and description. It's best to create a separate project for each of these points.

Write instructions.

The instructions for completing a task should be:

  • Complete.

    Explain all the possible situations the user might encounter.

  • Concise.

    Use concise wording and simple sentences.

  • Structured.

    Give step-by-step instructions (if possible). Use lists and formatting to make the text easier to read.

  • Visual.

    Provide examples. Add images to them (if the tasks have them). For example, if a task requires evaluating the quality of an image, put both high-quality and low-quality images in the instructions. If the task is to identify the type of clothing in a picture, describe the possible options in the instructions and illustrate the text with images.

If you are using offline acceptance, the instructions should briefly and clearly list the acceptance criteria.

Allow enough time for completing tasks.

Haste lowers the quality of responses, so you should allow a little extra time to complete the task.

Make training tasks.

Ask users to go through the training tasks before beginning the pool tasks. Include disputable cases in the training tasks.

Choose the appropriate type of quality control.

If the task contains a simple question with a multiple-choice response and is completed fairly quickly (1-10 minutes), it is best to run the task with overlap and use majority vote checking, a golden set, a captcha, and restriction of fast answers.

If the task doesn't have clearly defined response options (for example, it requires creating or translating a text or transcribing an audio recording), you can use the following verification methods:

  • Check the responses yourself using offline acceptance.

  • Submit the responses to Toloka users for checking. Create a task with a question (for example, “Is this phrase translated correctly?”) and possible responses (for example, “yes”/“no”). Set up overlap and majority vote checking.

Give out bonuses
You can give users bonuses for good performance on tasks. This improves motivation and makes users approach your tasks with more care.
Stay in touch with users
Proper communication with users keeps them interested in the project and allows you to quickly detect any problems in the tasks. Answer users' questions in personal messages and on the forum.