Classification

Projects of the classification type are intended for tasks with a finite number of possible responses. Examples are moderating content or grouping images by category.

Suppose you have a set of cat photos and want them split into several groups according to the cat's mood. You should create a task where a performer sees a photo and has to choose one of three responses. The performer can also mark if they like a photo.

Example of a prepared task

To release tasks for completion and get responses:

Create a project

A project describes the task design for a performer.

  1. Click the + Create a project button and choose the Image categorization template.

  2. Enter a clear name and a short description for the project. Performers will see them in the task list.

  3. Write short and comprehensive guidelines (see the recommendations).
  4. Define the objects a performer will get and return. To do this, add input and output fields in the Specifications block.
    What are input and output data?

    Input data are types of objects that are passed to the performer for completing the task. For example, this could be a text, an image, or geographic coordinates.

    Output data are types of objects that you receive after the task is completed. For example, this could be one of several response options, typed text, or an uploaded file.

    More on input and output data fields.

    In this case they are:

    • The image input data field for a link to an image.
    • Output data fields:
      • like, a boolean for the answer to the checkbox question.
      • result, a string for the answer to the radio button question.
  5. Create the task interface in the HTML block. It describes how the task elements should be arranged in the task.

    You can use standard HTML tags and special expressions in double curly brackets for input and output data fields.

    {{img src=image width="100%" height="400px"}}
    
    {{field type="radio" name="result" value="OK" label="Good" hotkey="1"}}
    {{field type="radio" name="result" value="BAD" label="Bad" hotkey="2"}}
    {{field type="radio" name="result" value="404" label="Download error" hotkey="3"}}
    <br>
    {{field type="checkbox" name="like" label="I like the photo" hotkey="q"}}
    This notation describes the following task design:
    • A picture at the image link.
    • Three radio buttons, and the chosen option is output to the result field.
    • A checkbox, with the value (true or false) output to the like field.

    Leave the CSS and JavaScript blocks unchanged.

  6. Click the Preview button to see the task as a performer.
    Note. The preview shows four tasks with standard data. You can define the number of tasks to show on the page later.
  7. Save the project.

Add a task pool

A pool is a set of paid tasks sent out for completion at the same time.

  1. Open the project and click Add pool → Add pool.
  2. Give any name and description to the pool. The pool info is only available to you. Performers can view only the project name and description.
  3. Set the price per task page (for instance, $0.02).
    What is a task page?

    A page can contain one or several tasks. If the tasks are simple, you can add 10-20 tasks per page. Don't make pages too long, as it slows down the performer's download speed.

    The performer gets paid for completing the whole page.

    The number of tasks on the page is set when uploading tasks.

    What is the fair price for a task page?

    The general rule of pricing is the more time the performer spends to complete the task, the higher the price is.

    You can register in Toloka as a performer and find out how much other requesters pay for tasks.

  4. Set the Time allowed for completing a task page. It should be enough to read the guidelines and wait for task data to download. For example, 600 seconds.
  5. Set an Overlap, meaning the number of performers to complete the same task. For classification tasks, 3 is enough.
  6. Add Filters to choose performers. To make your task available only to English-speaking users, set filters by language and country detected by the phone number.
  7. Save the pool.

Upload tasks

  1. Download the File example for task uploading (tsv) in the pool.
    What is TSV?
    A TSV file presents a table as a text file in which columns are separated by tabs.
    You can work with it both in a table editor and a text editor, and then save it to the desired format. More about working with a TSV file.
    Note. Before uploading the file, make sure it is saved in UTF-8 encoding.
  2. Add input data in it. The header of the input data column contains the word INPUT. Leave the other columns empty.
  3. Upload the tasks using Smart mixing and enter the number of tasks per page. For example: 9 main tasks and 1 control task.
    What is smart mixing?
    Smart mixing randomly generates pages with tasks so that tasks are not repeated for each performer.
  4. Add control tasks. To do this, click the Edit button and give the correct responses for several tasks.
    What are the control tasks?

    In control tasks, the correct response is known in advance. They are used to track the performer's quality of responses. The response you provided is compared to the performer's response. If they match, it means the performer answered correctly.

    Control tasks should make up at least 1% of the total number of tasks. This means that for 1000 tasks you should add at least 20 control tasks.

    More about control tasks.

Set up quality control

Quality control rules allow you to filter out inattentive performers. You can configure quality control both in the project and in the pool.

  1. Go to the project and click Project actionsSet up quality control.
  2. Add the Golden set rule and specify the following values:

    This means that a performer who gives more than 40% of incorrect responses will be blocked for five days and won't be able to complete tasks in this project.

  3. Add a Quick answers limit and specify the following values:

    This means that a user who completes a task page in less than 20 seconds will be blocked for ten days and won't be able to complete your tasks.

Add training

A training pool is a set of unpaid training tasks where the performer learns to answer correctly. Training tasks contain the correct answer and a hint shown if the performer gives the wrong answer.

  1. Open the project and click Add poolAdd pool.
  2. Give a name to the training pool and set the time for task page completion.
  3. Save the pool.
  4. Download File example for task uploading (tsv) or edit the one you used for uploading the main tasks.
    Note. TSV files for all project pools have the same structure.
  5. Add links to images for the training tasks in the TSV file.
  6. Upload the file and specify the number of tasks on the page. For example, 10. This number must not exceed the number of tasks per page in the main pool.
  7. Click Download and enter the number of training tasks on the page.
  8. Click Edit and add correct answers and hints for the uploaded tasks.
  9. After the file is uploaded, open the Preview and check that the tasks are displayed correctly.
  10. Open the main tasks pool, link Training to it and set the Level required to 55. This means that the main pool will be available for users who made no more than 45% of mistakes in the training pool.

Start the pool and get the results

  1. Start the pool by clicking .
  2. Track the completion of tasks in the Pool statistics section.
  3. When the pool is completed, launch results aggregation. To do this, click  → Responses aggregation next to the Download results button.
  4. Track the aggregation progress on the Operations page. When the process is completed, click Download.