Tips for designing tasks
To get more accurate responses, think through the structure and wording of the tasks and the settings for responses:
- Make a task as simple as possible. Divide complex tasks into several projects.
Performers do better with short tasks of the same type. Therefore try to divide a large task into several simple ones, creating a separate project for each task. Example
If your task consists of several stages, create a separate project for each stage.
For example, if you need to have road signs recognized, make a separate project for each of the stages: filtering out photos without road signs, selecting road signs in a photo, recognizing road signs, and reviewing the results.
- Prepare good instructions.
Instructions for completing a task should be:
Thorough. Describe all possible situations the performer may face.
Concise. Use concise wording and simple sentences.
Structured. Give step-by-step instructions (if possible). Use lists and formatting to make the text easier to read.
Visual. Provide examples. Add images to them (if the tasks have them).
For example, if a task requires evaluating the quality of an image, put both high-quality and low-quality images in the instructions. If the task is to identify the type of clothing in a picture, describe the possible options in the instructions and illustrate the text with images.
If you review completed assignments using non-automatic acceptance, list the evaluation criteria in the instructions.
- Create a simple and user-friendly task interface
- To help performers work faster, add hotkeys to the task interface.
- To ensure that performers make fewer mistakes, add response validation.
- Don't overload the interface. If there are too many elements in the interface, you might want to divide the project into several smaller projects.
- Check the performance of projects and their display on computer and mobile device screens. Use the sandbox to tune a task.
- Allow enough time for completing tasks.
Haste lowers the quality of responses, so you should allow a little extra time to complete the task.
- Make training tasks.
Ask performers to go through the training tasks before beginning the pool tasks. Include disputable cases in the training tasks.
- Select performers
Make sure to specify the language of the task instruction and text using filters. Add a filter by skill and the Speed/quality balance slider so that only performers with a high rating can participate in the project. For tasks to be completed in mobile apps, set the Client = Mobile Toloka filter.
Add a test based on the results of the training task to select performers. To do this, assign skills depending on whether their responses are correct.
- Choose the appropriate type of quality control.
If the task contains a simple question with a multiple-choice response and is completed fairly quickly (1-10 minutes), it is best to run the task with overlap and use majority vote checking, a golden set, a captcha, and restriction of quick responses.
If the task doesn't have clearly defined response options (for example, it requires creating or translating a text or transcribing an audio recording), you can use the following verification methods:
Check the responses yourself using the non-automatic acceptance option.
- Set a fair price and award rewards
- Set a fair price: the more complicated the task is and the more time the performer spends, the higher the price should be. Use dynamic pricing to motivate diligent performers.
- You can give rewards to performers for completing tasks well. This improves motivation and makes performers approach your tasks with more care.
- Keep in touch with performers
- Proper communication with performers keeps them interested in the project and lets you quickly detect any issues with the tasks. Answer users' questions through private messages.