Task markup

Task markup lets you add correct responses and hints to the tasks, as well as change task types. You can turn a main task into a control task by adding the correct answer, or into a training task by adding the correct answer and a hint.

What makes a good hint?

Avoid wordings like: “You answered incorrectly, please provide the correct response”. The performer learns when the hint explains the essence of their mistake.

Make the hints clear. Explain which response should be chosen and why.

Restriction. Task markup is available only for training pools and pools uploaded with “smart mixing”.
  1. How do I mark up tasks?
  2. Improve the reliability of control tasks
  3. What's next
  4. Troubleshooting

How do I mark up tasks?

  1. Make sure the pool isn't running.
  2. Click Edit in the Pool tasks block.
  3. Open the Main, Control tasks or Training tasks tab. Create a control or a training task from another type of task.
  4. Select the responses that should be checked. (The list of the output data fields is on the right.)
  5. Click Save and go to next.
Example of task markup

In this example, the correct response is added for the control task. Only the choice in the result field (Good/Bad/Loading error) is checked.

Improve the reliability of control tasks

After you have created the control tasks, make sure that different variations of correct responses occur with the same frequency. This will help avoid random guessing in responses.

  1. Go to the task markup page.
  2. Open the Training tasks → Distribution of correct responses for control tasks tab.

    The distribution of responses is shown as a percentage.

Tip. When creating control tasks, enter only correct responses that answer the question. So for an image classification task, Image loading error is not a correct response that answers the task question.
Example of response distribution

Troubleshooting

How many tasks should be on the page?

The number of tasks depends on how difficult and time-consuming the tasks are. Don't make task pages too large. They are unpopular, partly because they are inconvenient for performers (for example, if the internet connection is unstable).

How many control tasks do I need to add?

We recommend adding at least 1% of control tasks in the pool. To filter out performers, use the Control tasks quality control rule. To rank performers by the quality of responses in control tasks, use a skill.

How are the correct responses to control questions counted?

The Control tasks rule starts working after the performer completes the number of control tasks you specified. If your pool contains both training and control tasks, you can take into account the responses in both of them (the Number of responses parameter) or only in control tasks (the Number of control responses parameter).

As soon as the needed number of responses is collected, Toloka calculates the percentage of correct and incorrect responses and performs an action (assigns a skill, or blocks the user in the pool or in the project). Then this percentage is updated as the tasks are completed by the performer. The number of the performer's last responses used for the calculation is set in the Recent values to use field. If you leave it empty, all the responses from the performer in the pool are counted.