Project 1. Does a photo contain an object?

  1. Project creation
  2. Pool creation
  3. Preparing and uploading the file with tasks
  4. Receiving responses
Attention.

Before you start using Toloka, make sure that the English language is selected.

Project creation

  1. Click the button + Create project.
  2. Choose the Image classification template.
  3. Enter a clear project name and description.
    Attention.

    Before you start using Toloka, make sure that the English language is selected.

  4. Write short and simple instructions.
    Attention. To include an image in the instruction just paste the link from the dataset provided by pressing button.
  5. Define parameters for the input and output data:
    • The image input data field with the link type will be used to pass the image links to the performers.

      You will be able to upload the file with links to the pool later.

    • The result field will be used to receive performer's responses.
    • The like field in the template is used to pass the response to the question "Do you like the photo?".
      Attention.

      Our project doesn't require this checkbox, so you don't need an output field for it. Let's remove it.

  6. Create the task interface in the HTML block.
    • Delete the line with the checkbox component:
      {{field type="checkbox" name="like" label="Do you like the photo?" hotkey="q"}}
    • Add a question: does the image include a certain object? Example:
      <div>Are there traffic lights in the picture?<div>
    • Replace label with your response options. Example:
      {{field type="radio" name="result" value="OK" label="Yes" hotkey="1"}}
      {{field type="radio" name="result" value="BAD" label="No" hotkey="2"}}
      {{field type="radio" name="result" value="404" label="Failed to load" hotkey="3"}}
  7. Leave the CSS and JavaScript blocks unchanged.
  8. Click the Preview button to see the performer's view of the task.

    You will see four tasks with standard pictures on the page. You will set the number of tasks per page when configuring a pool.

  9. Select the radio buttons in the preview and make sure that the task can be completed.
  10. Click Save button to save the project.
    Note.

    To edit project parameters, click the button in the list of projects or Project actionsEdit on the project page.

Pool creation

  1. Click Add pool.
  2. Give the pool any convenient name and description. You are the only one who can see them.
  3. Specify the pool parameters:
    • Set the price per task page (for example, $0.01).
    • Overlap. This is the number of users who will complete the same task (for example, 3).
    • Time allowed for completing a task page (for example, 300 seconds).
  4. Set up user filters.
    • Select English-speaking performers using the Language = English filter.
    • Optionally, specify the percentage of top-rated performers in the Speed/Quality ratio.
      Attention.

      This can slow down pool completion.

  5. Set up quality control: Control tasks. Ban performers who give incorrect responses to control tasks. Example:

    This rule will be triggered when the performer completes 3 control tasks in the pool. If the performer gives at least 3 responses to the control tasks and the percentage of correct responses is less than 60%, they lose access to the project for 10 days. If the percentage of correct responses is over 60%, the performer can pass to the next task page. The rule will be triggered after the next control task. Optionally, add other quality control rules.

  6. Save the pool.

Preparing and uploading the file with tasks

  1. Download TSV file with images by link or QR code on the card with your login/password you have got in the beginning of the practice session.
  2. Upload pool tasks from this file.
    Attention.

    If you changed the name of the input field, change it in the file as well.

    Select Smart mixing and specify the number of tasks per page. For example: 9 main tasks and 1 control task.

  3. Create control tasks.
    Note.

    Control tasks are tasks with the correct response known in advance. They are used to track the performer's quality of responses. The performer's response is compared to the response you provided. If they match, it means the performer answered correctly.

    • Click EditCreate control tasks.
    • Check the result output field that is used to match the user response to the control one, select the response and click Save and go to next.
      Enter correct responses for 10 tasks. In small pools, control tasks should account for less than 10% of all tasks.
      Tip.

      Make sure to include different variations of correct responses in equal amounts. Open the ControlDistribution of known answers for control tasks tab.

    • Save the markup and check the number of control tasks.
  4. Start the pool.
    Attention.

    Remember that real Toloka performers will complete the tasks. Double check that everything is correct with configuration of your project before you start the pool.

Receiving responses

Restriction.

Aggregation takes from 5 to 20 minutes. During this time, you can start configuring your next project. Refresh the Operations page to check progress.

  1. Wait until the pool is completed. Refresh the pool page to check progress.
  2. Click the arrow next to the Download results button and run aggregation using the Dawid-Skene model.
  3. Go to the operations list and wait until aggregation finishes.
    Note.

    Aggregation takes from 5 to 20 minutes. During this time, you can start working on your next project. Refresh the Operations page to check progress.

  4. When aggregation is complete, download the TSV file with the results.