Terms and definitions

Toloka

The Toloka API, Toloka application programming interface, describes methods (a set of classes, procedures, functions, structures, or constants) used by the Toloka service to interact with another computer program and allows you to create tasks and receive user responses (see documentation).

TSV file with tasks

TSV (tab-separated values) is a text file format in which data is separated by a tab, and lines are separated by a newline symbol.

Active performers

Performers who viewed or completed tasks during the last hour and match filters specified in the pools settings. This is the number of performers who can access your task now, if it is published.

Archive pool

Pool, sent to the archive. Such pool can't be started, edite, and isn't available for the assignment review. The pool is automatically archived if none of the following actions is performed in the pool for a month:

Blocking a performer

'You can block a performer's access to one or more projects. This lets you control manually which performers will complete tasks. For example, you can choose all performers with a skill value lower than N and block their access to tasks. You can also unblock access.

Bonus

You can give rewards to performers for completing tasks well. This improves motivation and encourages performers to approach your tasks with care. You can give rewards to one or more performers (in addition to payment for completed tasks). The reward amount can be from 0.01 to 100 dollars per performer per time.

Fast responses

Restricting access to tasks for performers who respond too quickly. Used to:

  • Suspend access for performers who cheat in their responses (in this case, set the time required to complete a task suite giving random responses).
  • Provide protection from robots (in this case, the time for completing the task suite should be 2 times less).
Submitted in pool

The number of performers who completed at least one task suite.

Input and output data

In the Specification field, you describe the format of input and output data.

Input data is the data you need for your tasks. For instance, this could be links to images that you want to show to performers.

Output is the data you receive after the task is completed, like the performer's answers to your questions.

Selective majority vote check

Majority vote checks only some tasks. This allows you to save money and speeds up the pool completion.

Completed tasks (quality control rule)

Number of completed task suitess (taking overlap into account).

Ready-made audience sets (filters)

Ready set of filters for choosing performers. For example, mobile users.

Ready protection sets

Several quality control rules with default settings.

Incremental relabeling

Incremental relabeling allows you to change the number of performers to complete each task in the pool depending on how well the performers cope with the task or how consistent their responses are.

Dynamic pricing

Dynamic pricing: Allows you to additionally set other prices that depend on the performer's skill. For example, to pay more to those who have a higher skill. Learn more.

Task (Main task)

A task that doesn't provide hints or the correct response. The responses are provided by the performers. These are the main tasks you will upload to the pool to give to the performers.

In addition, there are control and training tasks.

Interested users

The number of performers who started at least one task suite.

Earnings (quality control rule)

Restriction of the performer's earnings in pool during the day, to get responses from as many performers as possible and provide protection from robots.

Instructions

Task instructions the performer sees when choosing and completing the task. The clarity and completeness of the instructions affect the response quality and the project rating. Good instructions help the performer complete the task correctly.

Task interface

The task interface defines the task appearance for the performer and the logic for processing responses.

Captcha (quality control rule)

Restricting access to tasks for the performers who fail to enter captcha several times in a row. This helps protect against robots.

Fees

The requester is charged a commission fee for using Toloka — a percentage of the cost of tasks. It is 20%, but not less than $0.005.

Control task

A task that contains the correct response. Control tasks are needed to monitor the quality of the performer's responses.

For example, you can track the percentage of correct responses and block performers who respond poorly. To do this, use the Control tasks rule.

Note.
  • Add at least 1% of control tasks to the pool.
  • Design control tasks so that different versions of correct responses are used with the same frequency. Learn more.

In addition to control tasks, there are main and training tasks.

Quality control

Quality control lets you get more accurate responses and restrict access to tasks for cheating performers. Quality control consists of rules that are independent from each other. The quality control block allows you to set rules and configure them. Learn more.

Majority vote (quality control rule)

A quality control method based on matching responses from the majority of performers who complete the same task.

Skill

Assessment of the performer's quality of task completion (a number from 0 to 100). The skill can be calculated automatically (for example, based on correct answers to control tasks), or assigned manually. Learn more.

Incomplete page

The last page with an insufficient number of main tasks. It's generated if less than the minimum remaining number of main tasks are not completed. In this case, the number of control and training tasks must be complete.

Training task

A task that contains the correct response and a hint. If the performer answers incorrectly, they will see a hint and won't be able to go to the next task suite until they complete all tasks correctly. Only the first response to the task is considered for quality control and skill calculation.

Training tasks can be used in free training pools to choose performers who passed training.

They can also be used in regular pools as control tasks with hints. For example, you can track the percentage of correct responses and block performers who respond poorly. To do this, use the Control tasks rule.

Training

Training pool with free tasks for the performers to train. Training tasks contain correct responses and a hint shown if the performer answers incorrectly.

To improve the quality of results, you can add a training pool and choose only those performers who successfully completed the training. Learn more.

Reviewing assignments

A pool setting that allows you to check responses and not to pay for poorly completed tasks.

Overlap

The number of performers who should complete each task in the pool.

Sandbox

Toloka testing environment. This is where you can test your project settings as a performer before moving them to the Toloka production version and running the project for real performers. This helps you avoid making mistakes and spending money on a task that isn't working right.

Recompletion of assignments from banned users (quality control rule)
A rule that can be configured to send tasks to other performers for re-execution if:All pool tasks completed by this performer will be resent to other performers. Tasks completed by banned performers are reassigned, but tasks completed by paused performers are not.
Field task

Task to be performed in the mobile app. It usually contains a point on the map where the performer needs to come, check for something, and take photos.

Quality control rule
Independent rules that make up the quality control. The quality control block allows you to set rules and configure them.
Project

The project consists of pools with tasks. Project settings define the task interface and the input and output data.

Skipping tasks (quality control rule)

Restricting access to the pool tasks for performers who skip several task suitess in a row.

Pool

A set of paid tasks sent out for completion at the same time.

Task markup

Task markup lets you add correct responses and hints to the tasks, as well as change task types. You can transform a main task in a control task by adding the correct response, or in a training task by specifying the correct response and a hint. Task markup is available only for training pools and pools uploaded with smart mixing.

Learn more about the task markup.

Retry pool

A "retry pool" will help performers who make mistakes improve their skills and get a se chance to complete tasks.

Performer rating

The performer's rating reflects the quality of the performer's responses to tasks. It takes into account responses to control tasks, bans and periods without work. The rating is calculated as the weighted average value for the last few weeks. So the rating can “suddenly” decrease or increase.

Certified partners

Companies that take on the launch and implementation of your project and help you process the results. These companies have already integrated crowdsourcing into their business processes and they offer excellent solutions.

Keeping the task order

task suitess in pool are distributed in the same order as they are in the uploaded TSV file. The tasks inside these pages are randomly mixed. Learn more.

Crowdsourcing experts

Professionals who help create and configure projects, post tasks and much more. They have passed a certification exam to prove their knowledge of crowdsourcing and earned a registered e-certificate from Yandex. To learn about terms of partnership, contact these experts directly.

Review period

Number of days to review the task.

Task suite

A page can contain one or several tasks. If the tasks are simple, you can add 10-20 tasks per page. Don't make pages too long because it slows down loading speed for performers.

Training

The main pool that consists only of training tasks and can have a zero cost.

Use it to train performers before performing real tasks if the functionality of the training pool doesn't suit you.

Training skill

A skill that is automatically created when the first training pool is added to the project. The skill name is <project name> - training. The value is the percentage of correct responses in the training pool. The skill is assigned to the performer after they pass the training pool.

Training skill is one for the entire project. If you create new training pools, they will use the same skill.

Smart mixing

Smart mixing randomly generates pages with tasks so that tasks aren't repeated for each performer.

Filters

Criteria for choosing performers. For example, to choose only Russian-speaking performers or performers with some skill.

Filters are used in the pool settings on the Users page and for sending messages to a group of users. Learn more.

Project template

The template contains pre-configured input and output data fields and the task interface, which you can edit.

Exam

Pool with control tasks for the choosing performers. Assign skills to performers depending on the correctness of responses, identify those who do better than others.