Object classification

Classification projects are well-suited for tasks with set answers. For instance, content moderation or distribution of images into several categories.

Suppose you have a set of cat photos and want them split into several groups according to the cat's mood. To have this done, you should create a task as follows: a user sees a photo and has to choose one of three answers. Besides, a user can like a photo.

Task example

To release tasks for completion and get responses:

Create a project

A project describes the task design for a user.

  1. Click the + Create project button and select the Image categorization template.

  2. Give a clear name and a short description to the project. Users will see them in the task list.

  3. Write short and comprehensive guidelines.
  4. Define the objects a user will get and return. To do this, add input and output fields in the Specifications block.
    What are input and output data?

    Input data are types of objects shown in the task. For example, text, image, earth coordinates. For more information refer to Input data.

    Output date are types of objects a user returns. For example, one of several response options, typed text, uploaded file. For more information refer to Output data.

    In the present case they are:

    • Input field image, a link to an image.
    • Ouput fields:
      • like, a boolean for the answer to the checkbox question;
      • result, a string for the answer to the radio button question.
  5. Create the task interface in the HTML block to design how objects are arranged in the task.

    You can use standard HTML tags and special components, enclosed in double curly brackets, for input and output data fields.

    {{img src=image width="100%" height="400px"}}
    {{field type="radio" name="result" value="OK" label="Good" hotkey="1"}}
    {{field type="radio" name="result" value="BAD" label="Bad" hotkey="2"}}
    {{field type="radio" name="result" value="404" label="Failed to load" hotkey="3"}}
    {{field type="checkbox" name="like" label="Do you like the photo?" hotkey="q"}}
    This notation describes the task design
    • an image by link image;
    • three radio buttons, the choice will be written in the result field;
    • a checkbox, the value (true or false) will be written in the like field.

    Leave the CSS and JavaScript blocks unchanged.

  6. Click the Preview button to see the task as a user.
    Примечание. In the projects preview, there are four tasks with standard input data. The number of tasks on page will be set further.
  7. Save the project.

Add a task pool

A pool is a set of paid tasks sent out for completion at the same time.

  1. Open the project and click Add pool → Add pool.
  2. Give any name and description to the pool. They are available to you and a user will see only the project name and description.
  3. Set the price per a task page, for instance $0.02.
    What is a task page?

    A page contains one or several tasks. For simple tasks, put 10–20 tasks on a page. Do not make page too long, because a user will have a low download rate. A user gets a reward for completing the whole page.

    The number of tasks on page will be set during tasks uploading.

    What is the fair price for a task page?

    The principle of the fair price is that the more time a user spends for tasks completing, the higher the price.

    Register as a user and learn how much other requesters pay.

  4. Set the Time a user will have to complete all task on the page. It should be enough to read the guidelines and wait for task data dowloaded. For example, 600 seconds.
  5. Set the Overlap, the number of users to complete the same task. For classification tasks 3 is enough.
  6. Add Filters to select performers. To make your task available for English-speaking users, set Languages = English.
  7. Save the pool.

Upload tasks

  1. Download File example for task uploading (tsv) in the pool.
    What is TSV?
    A TSV (tab-separated values) is the text format of spreadsheet data separated by the tab character.

    You can open it in a spreadsheet or text editor and then save as TSV with the UTF-8 encoding.

  2. Add input data to the file. The headings of input columns contain INPUT. Leave other columns unchanged.
  3. Upload tasks using the Smart mixing option and set the number of tasks on the page. For example: 9 main and 1 control task.
    What is the smart mixing?
    The Smart mixing randomizes tasks and generates task pages so that a task is not repeated for a user.
  4. Add control tasks. To do this, click the Edit button and give the correct responses for several tasks.
    What are control tasks?

    Control tasks contain the correct answer. They are used for checking and filtering out users who give incorrect responses too often.

    A pool should contain no less than 1% of control tasks. For example, in a 1000-task pool you should add at least 10 control tasks.

Add quality control

Quality control rules allows to filter out inefficient users. Quality control can be set in both a pool and a project.

  1. Go to the project and click Project actionsSet quality control.
  2. Add the Golden set rule and specify the following values:

    It means that a user who gives more than 40% of incorrect responses will be blocked for five days and will not be able to complete tasks of this project.

  3. Add the Quick answers rule and specify the following values:

    It means that a user who completes a task page quicker than 20 seconds will be blocked for ten days and will not be able to complete your tasks.

Add a training pool

A training pool is a set of unpaid training tasks. Training tasks contain correct responses and hints. If a user responds incorrectly, a hint is displayed on the screen. The user must enter the correct response before continuing to the next task.

  1. Open the project and click Add poolAdd training.
  2. Give a name to the training pool and set the time for task page completion .
  3. Save the pool.
  4. Download File example for task uploading (tsv) or edit the one you used for main tasks uploading.
    Примечание. TSV files of a project have the same columns.
  5. Add links to images in the TSV file.
  6. Upload the file and specify the number of tasks on the page. For example, 10. It should not be greater than in the main pool.
  7. Click Upload and specify the number of training tasks on the page.
  8. Click Edit and add correct answers and hints.
  9. Click Preview to see what the tasks look like.
  10. Open the main pool and link the training pool to it. Specify 55 in the Level required field. So the main pool will be available for users who made no more than 45% of mistakes in the training pool.

Launch the pool and get results

  1. Click to send tasks out for completion.
  2. Track the completion of tasks in the Pool statistics block.
  3. When the pool is completed, launch data analysis. To do this, click  → Responses aggregation.
  4. Track the aggregation in the Operations page. When the process is completed, click Download.