Additional criteria for various types of tasks

The following types of tasks have additional quality criteria:

Field tasks and selecting areas in images

Field tasks are completed in the Toloka mobile app. Usually the performer should go to a specified address, take photos or record a video, and answer certain questions. In tasks for selecting image areas, the performer should outline an area as a polygon or rectangle in an image.

Checking tasks

These types of tasks usually have non-automatic acceptance, so the performers' responses need to be checked. There are three ways to do this:

  • Auto checking. For example, you wrote a script that retrieves the results via the API, sends the responses for recognition and verification, checks the coordinates, and then returns the task status (accept or reject), and uploads them to Toloka. If you created a script like that or automated verification in another way, show us how it works.

  • Verification task. You can assign other performers a Toloka task for reviewing completed assignments. It's important that the verification task isn't available to performers who participated in the field task.

  • Smart template. For example, a field task runs JavaScript that prevents users from sending responses if they're more than 1 km from a certain placemark on the map.

Task acceptance

We expect your project to be configured as follows:

  • The average task review period is no more than 7 calendar days. Preferably it should be 2-3 days.

  • The task acceptance rate is at least 75%. Our experience shows that a high percentage of rejected tasks is an indicator of poor process management and this should be fixed. Most often, a performer is assigned a very large and complex task that can and should be divided into several simple ones.

Classification

Classification is a multiple choice task. Examples are moderating comments or grouping images by category.

Quality control

Rules help you measure the quality of performers' responses in real time: set a higher price for those who do well and restrict access to tasks for cheaters.

  • Use at least three different quality control rules in a project. One of these must be the Golden Set.

  • Add at least 1% of control tasks to a pool. Keep them up to date: add new ones at least once every 30 days. Web pages are very dynamic. A link that was working recently may now show error 404 and the page information may become outdated.

Training
None of us like to read long instructions. It's much easier to learn something from examples. Training helps users figure out how to do the task and you can choose the ones who are good at it. Update your training pool at least once every three months. Otherwise, it becomes outdated and performers will be able to learn it by heart.