Toloka

Main and control tasks

To create main and control tasks, add a pool to the project and set pool parameters, then load a TSV file with tasks data. You can track statistics on task completion on the pool page.

Creating a pool

It is better to test the pool and task suites settings in the sandbox, and then transfer them to the Toloka production version (see Workflow).

To create a pool with main tasks and control tasks:

  1. Prepare the data for tasks.

    If the tasks have images, they need to be available in the internet. For example, they can be stored on Yandex.Disk.

  2. Click Add pool on the project page.

  3. Set the pool parameters and click Save.

  4. Load a TSV file with tasks.

    Click Upload on the pool page and choose a method to split the pool into task suites:

    • By empty row.

      Set task suites in the TSV file. After each taskk suite in TSV file, add an empty line.

    • Set manually.

      Define the number of tasks in a task suite. Task suites will be formed from the tasks in the order they are placed in the TSV file.

    • Smart mixing.

      Specify how many tasks of each type should be in a task suite. For example, 10 main tasks and one control task. Tasks will be distributed in the specified proportion. Users will not get repeated tasks.

    To see what the tasks look like, click Preview.

To send tasks out for completion, click on the pool page.

Pool parameters

Parameter

Description

Name

The pool name and description (only accessible to the requester).

Description

Training

The name of the training pool.

Level required

Proportion of correct answers in training tasks (from 0 to 100) required to be admitted to the main tasks. Only first answer of the user in each task is taken into account.

Price

Payment per task suite in U.S. dollars. For cents, use the dot (".") as the separator. The minimum price is $0.01.

Time on task

The time allowed for completing a task suite, in seconds. Tasks not completed within this time are reassigned to other users.

We recommend spending no more than 60 seconds per task (including the time for page loading and sending responses).

Overlap

The number of users who should complete each task in the pool.

Expires

The date the pool closes. The pool closes on this date, even if the tasks aren't completed.

Adult content

Whether the tasks have porn content. Tasks with porn content are only issued to users who are prepared to perform these tasks.

If you didn't check whether the tasks have porn content, select "Yes".

Mobile only

Tasks are only available for performing in the mobile interface. In the desktop version, the user will see them in the list of tasks.

Keep task order

Assign tasks in the order they are listed in the TSV file. If the pool has an overlap, the next task is distributed only when the previous task is completed by the neccessary number of users. For example, you can order the tasks in the list by priority, so that more important tasks are completed faster with the necessary overlap.

By default, tasks are issued in random order.

Captcha frequency

The frequency of showing captchas:

  • “No” — Don't show captchas.

  • “Low” — Show a captcha after every 20 tasks.

  • “Medium”/ “High” — Show a captcha after every 10 tasks.

To use captchas for selecting users, set up quality control rules. For example:

Offline accept

Check the completed tasks manually.

Deadline

The number of days for accepting the tasks (max — 21).

Users filter

Settings for selecting users: filters and quality control rules.

TSV file with tasks

Input data is uploaded to Toloka in a TSV file. You can download a sample file for your project by clicking File example for task uploading (tsv) on the pool page. Each row contains data for a single task. Column headings:

To form task suites yoursef, add empty line after each task suite.

The example file below contains two task suites. Data for the task: Image URL and webpage URL ( INPUT columns) and correct answers (GOLDEN:result column).

Additional fields for tasks selected on the map
If a user chooses tasks on the map, add the coordinates of the map center after each task suite in columns:
  • Al:latitude — Latitude.

  • Al:longitude — Longitude.

Rules for populating the TSV file:

  • Field names must have the same case as the input data ids.

  • URLs must have the www or http:// prefix.

  • Quotation marks in rows must be paired.

  • Multiple paragraphs and tab characters can be included in a field. To do this, enclose the text with paragraphs in quotation marks.

The maximum file size is 100 MB.

Pool statistics

You can track the completion of tasks on the pool page (see the table). You can also get email notifications about pool completion. Click Edit profile in the requester profile and select the options Enable notifications and Send copies of notifications by email.

Field

Description

Task suites

The number of task suites in the pool.

Tasks

The total number of tasks in the pool.

Training tasks

The number of training taks in the pool.

Golden tasks

The number of control tasks.

Average submit time

The average time for completing a task (or task suite) in the pool.

Approximate finish time

The anticipated time for finishing tasks, in days and hours. Calculated from the average time per task.

Formula: ,

where:

is the number of tasks remaining uncompleted or completed with insufficient overlap.

is the average time per task (calculated from the amount of time that has passed since the pool opened and the number of tasks completed).

Budget spent (+ markup)

The total amount spent (not counting tasks waiting for offline acceptance). The amount of the markup is shown in parentheses.

Approximate budget (+ markup)

The amount of expenditures and anticipated costs (if all the tasks will be completed and approved). The amount of the markup is shown in parentheses.

Active users having access to pool

The number of users selected using filters and by their agreement to complete tasks with “adult content”. The number includes only users who viewed and and completed tasks in Toloka in the recent month.

Interested in pool

The number of users who have accepted at least one task.

Submitted in pool

The number of users who completed at least one task.

Submitted assignments per 1 user

The average number of tasks completed per user in the pool (including tasks rejected by the requester).

Expired tasks

The number of tasks that have expired.

Skipped tasks

The number of tasks skipped by users.