Project 3. Are the bounding boxes correct?

  1. Project creation
  2. Pool creation
  3. Preparing and uploading the file with tasks
  4. Receiving responses
  5. Upload review results
  6. Review assignments online (the way to see the final results of your pipeline)

Before you start using Toloka, make sure that the English language is selected.

Project creation

  1. Click the + Create project button and choose the Selecting a region in an image template. This template contains the editor for image area selection, which help you to show performers images with bounding boxes.
  2. Enter a short project name and description, and write simple and clear instructions.

    It will be visible for real people.

  3. Define parameters for the input and output data:
    • The image input data field with the link type will be used to pass the image links to the performers.
    • The selection field with the “json” type, will be used to pass the coordinates of the objects selected in the previous task. Don't make this field required.
    • The assignment_id field with the “line” type, will be used to pass the number of the completed task.
    • The result field with the “string” type will be used to write the result of performer's selection: correct or incorrect.
  4. Create the task interface.
    • Add the annotations=selection parameter to the area selection editor to show the selected object to the performer.
    • Chose another name for image area selection editor. For example, object.
    • Add two radio buttons:
      {{field type="radio" name="result" value="OK" label="Correct" hotkey="1"}}
      {{field type="radio" name="result" value="BAD" label="Incorrect" hotkey="2"}}
    • Remove the area selection tool from the interface: in this task, you don't need to correct or add markup.
    • Add code to the CSS block:
      .image-annotation-editor__shape-rectangle {display: none;}
    • Add code to set the height of the image markup interface that matches the image size:
      .image-annotation-editor__annotation-layer {height: max-content;}
  5. Click the Preview button to see the performer's view of the task.

    You will see four tasks with standard pictures on the page. You can set the number of tasks per page in the pool.

  6. Select the radio buttons in the preview and make sure that the task can be completed.
  7. Give the pool any convenient name and description.
  8. Click Save button to save the project.

Pool creation

  1. Click Add pool.
  2. Give the pool any convenient name and description.

    You are the only one who can see them.

  3. Specify the pool parameters:
    • Set the price per task page (for example, $0.01).
    • Overlap. Specify 3.
    • Time allowed for completing a task page (for example, 600 seconds).
    • Keep task order. Activate the radio button to get the number of responses considered the majority vote faster.
  4. Set up user filters.
    • Select English-speaking performers using the Language = English filter. Prevent performers who completed previous tasks from checking this one. To do this, set a filter with the worked_on_bounding skill:

      The worked_on_bounding skill = Absent (empty field).

      Optionally, specify the percentage of top-rated performers in the Speed/Quality ratio.


      This can slow down pool completion.

  5. Set up quality control:
    • Majority vote. Add the majority vote rule. Specify how many responses

      you accept as majority. For example, 3 out of 5 or 2 out of 3.


      The rule takes effect when the number of responses for the task is equal to the overlap. To get the required number of responses faster, turn on the Keep task order option in the pool parameters.

      Optionally, add other quality control rules.
  6. Save the pool.

Preparing and uploading the file with tasks

  1. Wait until the pool of Project 2. Highlight each object by a bounding box is completed.
  2. Open the pool page in Project 2. Highlight each object by a bounding box and click the Download results button.
    • Clear the Accepted checkbox and select Submitted.
    • Clear link, user ID, status, start time and Separate assignments with empty row checkboxes. This will give you a list of unreviewed tasks.

  3. Once you download the file just change the headings in the file.
    • Change the name of the OUTPUT:result column to INPUT:selection.
    • Change the ASSIGNMENT:assignment_id column name to INPUT:assignment_id.
    • Leave the INPUT:image column unchanged.
    • You can keep GOLDEN:result, HINT:text, ACCEPT:verdict and ACCEPT:comment columns unchanged.
  4. Save the file in TSV format.

    You can use a simple notepad for this action.

  5. Upload the file to the pool by selecting Set manually. Set the number of tasks per page (for example, 10).
  6. Start the pool.

Receiving responses


Aggregation takes from 5 to 20 minutes. During this time, you can start configuring your next project. Refresh the Operations page to check progress.

  1. Wait until the pool is completed.
  2. Click the arrow next to the Download results button and run aggregation using the Dawid-Skene model.
  3. Go to the operations list and wait until aggregation finishes.
  4. Download the responses.

Upload review results

As you set post acceptance in the pool settings in Project 2. Highlight each object by a bounding box, you need to check the performers' responses within the time limit set in the Deadline field.
  1. Open the file with aggregated results in a spreadsheet editor.
  2. Add the following columns:
    • ACCEPT:verdict — The result of verification.
    • ACCEPT:comment — Comments for performers if responses were rejected (for example, which part of the instructions wasn't followed).
  3. Change the name of the INPUT:assignment_id column to ASSIGNMENT:assignment_id.
  4. Fill in the ACCEPT:verdict and ACCEPT:comment columns:
    • If the aggregated result for the task is OK, put + then the task will be accepted.
    • If the result is BAD, put - then the task will be rejected. Enter the reason for rejection in the ACCEPT:comment field . For example: The object is not selected or selected incorrectly.
  5. Now you can delete the other columns. Save the edited TSV file.

    Open the pool page in Project 2. Highlight each object by a bounding box.

  6. Click Review assignments on the pool page above the progress bar.
  7. Click Upload review results.
  8. Select the file and upload it to Toloka.
  9. Check that all tasks have changed their status to accepted or rejected.
  10. If you rejected tasks and set up the rule to send them for re-completion, the pool will open again, and these tasks will be resent to other performers. After the pool is marked up, download the new results and submit them for review. Download the results. Repeat these steps until all the images from the second project are correctly marked up.

Review assignments online (the way to see the final results of your pipeline)

You can also review assignments by yourself and see the results of the crowdsourcing pipeline that you have created.

  1. Click the Review assignments button on the pool page.
    • Choose an assignment then click Accept or Reject.
    • For rejected assignments, enter a comment (explain why you decline it).
  2. Click the Download results button and select Accepted.
  3. Here you are photos with traffic lights bounded by a box.