How to save your budget

Every requester wants to optimize expenses. This section describes the main ways to save money without compromising the quality of labeling.

Decompose tasks

Use decomposition. Simpler tasks cost less. If a task is complex, you won't get good results immediately. Instead, you'll have to resend tasks to the pool so that other performers can redo them.

Use incremental relabeling.

Incremental relabeling lets you change overlap depending on:

  • How well the performers do on the task.
  • How well performers' responses match each other.

With this overlap setting, you can save on the number of performers who complete your tasks, which means the total cost of pool labeling will be lower.

Review assignments.

Assignment review allows you to check responses so you don't have to pay for poorly completed tasks.

Use assignment review if:

  • The task requires that users send free-format responses or data files.
  • You are sure that you have enough time to review the performers' responses. Otherwise, when the period ends, responses that haven't been reviewed are accepted and paid for automatically.

Set up a random majority vote check.

You can set up a random majority vote check for the main tasks. This helps you save money and speed up pool completion. You can set the period during which tasks with increased overlap will be displayed to the performer, depending on the number of completed tasks.

For example, let's say your pool's overlap is set to 3. You don't trust the responses of new performers and want to check them further. In this case, you set up a random majority vote check. If the performer completed 10 tasks in the last 14 days, set overlap to 5 for every third task, and if the performer completed 20 tasks, set overlap to 5 for every fifth task.

Without this setting, you would be forced to set overlap to 5 for each task in the pool. And it would be more expensive to label all tasks with overlap = 5, as opposed to just some of the tasks.

Set up dynamic pricing

Dynamic pricing lets you increase the price per task page for performers with a higher skill level. This way you can encourage performers to do a better job.

In some cases, this setting makes labeling more expensive, like for tasks that don't require performers to have high skills. In this case, there is no need to use dynamic pricing.

Use fractional payment on incomplete pages.

Use the Smart mixing and Assign partial page options. This way you can save on paying for an incomplete task page.

Compress images.

Compress images so that a task opens and displays correctly. Performers will spend less time and traffic uploading images.

Configure keyboard shortcuts

Create keyboard shortcuts. This will make it easier for performers to complete your tasks. Remember: the less time it takes to complete a task, the less you pay for it.

Save money using the Toloka API.

Merge tasks.

You can merge tasks that have identical input data. This allows you to save money if the same task was uploaded to different pools.

More.

Set the frequency of assigning control tasks.

Using the Toloka API, you can set the frequency of assigning control tasks. For example, make every fifth task a control task on the first task page and every 10th task on the following pages. This setting lets you maintain the quality of the pool and speed up the completion of tasks.

To do this, use the golden_task_distribution_function key.

More.

Set up preliminary responses.

In addition to Incremental relabeling, use baseline_solutions to speed up the overlap. In baseline_solutions, preliminary responses are used, that is, data that simulates performer responses. For example, you need to determine what an image shows: a cat or a dog. Suppose your neural network determined that the image may show a dog with a probability of 80% and a cat with a probability of 40%. You can use the neural network's responses. Depending on the responses of performers, the overlap will increase or remain unchanged.

More.