Create a pool

You should test your pool and task suite settings in the sandbox before transferring them to the Toloka production version (see Workflow).

To create a pool with main tasks and test tasks:

  1. Prepare the data for tasks.

    If the tasks have images, they need to be available on the internet. For example, they can be stored on Yandex.Disk.

  2. Create a TSV file with input data for tasks.

    You can include control task answers in the TSV file or enter them later (at step 6) in task editing mode (to do this, you have to use “smart mixing”).

  3. Click Add pool on the project page.

  4. Set the pool parameters and click Save.

  5. Load a TSV file with tasks.

    Click Upload on the pool page and choose a method to split the pool into task suites:

    • By empty row.

      Set task suites in the TSV file. After each task suite in TSV file, add an empty line.

    • Set manually.

      Define the number of tasks in a task suite. Task suites will be formed from the tasks in the order they are placed in the TSV file.

    • Smart mixing.

      Specify how many tasks of each type should be in a task suite. For example, ten main tasks and one control task. Tasks will be distributed in the specified proportion. Users will not get repeated tasks. After uploading tasks with the “smart mixing” option, you can edit tasks and set up selective majority vote.

    To see what the tasks look like, click Preview.

To send tasks out for completion, click on the pool page.

Task edit mode

Task edit mode allows you to add correct answers and hints to the tasks, as well as edit them. For instance, you can turn a main task into a control task by specifying the correct answer.

Edit mode is only available for training pools or pools uploaded with the “smart mixing” option.

To edit tasks:

  1. Stop the pool.
  2. Click Edit in the Pool tasks section.
  3. Add or edit the correct answer and hint.
  4. Select the answers that should be checked. (The list of the output data fields is on the right.)
Example of task edit

The example shows adding the correct answer. Only the result field will be checked (Good / Bad / Failed to load).