How does Toloka work?

To place a task in Toloka, create three different entities: a project, a pool, and the tasks to be uploaded. Each of these entities is responsible for its own task settings block. This section explains why Toloka works this way and what its advantages are.

Project

A project contains instructions, task interface settings, and a specification for input and output data. You can also set up quality control rules that apply to the entire project.

Projects make it easier for you to post similar tasks in the future, because you don't have to re-configure the interface. You can also view aggregate statistics for all tasks in the project.

Pool

A pool is a set of tasks to be assigned to performers simultaneously. Main pool settings: pricing, selection of performers, overlap, and quality control.

Dividing tasks into pools lets you:
  • Select various performers for tasks.

    For example, text in tasks for comment moderation can be in different languages: some comments in Russian and others in English. You can split these comments into different pools and use filters to specify the language that performers should know.

  • Set a price, the overlap, and configure quality control based on the complexity of tasks.

    For example, tasks for transcribing audio files that are 10-20 seconds and 50-60 seconds long should have a different price.

  • Manage task priority by assigning one pool before another.

  • Group tasks to view separate statistics for each group.

Task

A task is the data you need to mark up. Each image or comment is a separate task.

Tasks can be of different types: regular, control (with responses), and training (with hints).

You upload tasks to Toloka as a TSV file. The file structure must match the input and output data specification that you set up in the project.

Tip. If you want multiple performers to complete a task, you don't need to include it in the input data multiple times. Instead, set the overlap in the pool settings.

task suite

Tasks are assigned to performers page by page. A page may contain one or several tasks. You specify their number when uploading tasks in the pool.

Important.
  • A performer's response will only be registered after they complete all the tasks on the page.
  • You specify the price per task suite. Performers are only paid for the entire page of completed tasks.

Some performers have a slow internet connection, so adding multiple tasks lets them update a page less often, which means they'll complete your tasks faster.

However, don't include too many tasks on a page: quality will be worse and performers will get tired. Place 5-10 simple tasks or 1-4 complex ones on a single page.

Tip. You can separate tasks on a page yourself: this lets you group tasks, such as by simultaneously assigning all similar photos to performers for annotation. In this case, the number of tasks on each page may differ.

What does it look like for performers?

When you start a pool, performers can see it in the Tasks section. They select the pool, read the instructions, and complete tasks.

Each task on a task suite is part of a web page that can include various objects, such as images, text, and input fields.

Note. Performers don't use divisions of pools, tasks, or task suites. They might call all this a “task”.

For more information, see How performers see pools.

Payment

For each accepted task, an amount of money is deducted from your account: payment for the task, plus the fee for using Toloka. The amount to pay for the task is set when creating the pool.

You can also give a reward to one or more performers.

Toloka

If you need to frequently post similar tasks, automate this process with the Toloka API.

If you want more flexibility when posting tasks, the Toloka API has additional features that are not yet available in the interface. For example, use the API to specify overlap for individual tasks and add multiple correct responses to control tasks.