Reviewing performers' responses

You started a pool with non-automatic acceptance, and the performers completed all your tasks. What next?

  • Review the performers' responses before the end of the Review period you specified. At the end of the period, unchecked responses will be accepted automatically.
  • If a performer didn't complete the task well, reject their response.
  • The performer can challenge your decision by submitting an appeal.

How do I review the tasks?

In the Yandex.Toloka interface. This option is suitable for a small number of tasks.

To accept or reject responses:

If you set non-automatic acceptance in the pool settings, you need to review the performers' responses within the time limit set in the Deadline field. Performers may challenge a rejected response by submitting an appeal.

You can review assignments online or upload review results in a TSV file.

Note. Note that you can't change the task status if the task pool was archived.
Online review of one task.

To accept or reject one task:

  1. Click the Review assignments button on the pool page.

  2. Choose an assignment.

    If responses include files uploaded by the performers, click the Actions → Download attachments button to download them. To download all files from submitted responses in a ZIP archive, click Download results → Download attachments on the assignments review page.

  3. Check the responses, click Accept or Decline. For rejected responses, enter a comment (specify the reason).

In the TSV file with the results. You will need the file to filter the results and process them programmatically.
Note.

This review method is also suitable when the result is not displayed in the interface.

Example 1: The task uses JS code, and the response contains initially hidden fields.

Example 2: The task where you need to download files attached by the toloker (for example, video or audio recordings) to check the responses.

To accept or reject responses:

  1. Click the Review assignments button on the pool page.

  2. To download responses that you didn't review yet: on the pool page, click Download results. In the window that opens:

    1. In Status leave only the Submitted option enabled.

    2. In Columns leave only the assignment ID option enabled.

    3. Disable the Separate assignments with empty row option.

    4. Click Download results.

  3. If your task requires tolokers to submit pictures, videos or audio files, download them for review in an archive.

    Click Download results → Download attachments.

    Note.

    What does the response archive contain?

    The system automatically assigns names to files when they are uploaded by the performer. You can rename the files manually or with a script if necessary.

    • Archive with responses

      • Folder1_ID_ of_responses

        • File1_attachment_ID

        • File2_attachment_ID

      • Folder2_ID_of_responses

    You can match responses and attachments by their IDs. Attachment ID is included in the TSV file, in the column with the output field where the file was uploaded.

    • TSV file

      • Column1_with_output_field

        • Attachment_ID

      • Column2_with_output_field

        • Attachment_ID

  4. Open the file with responses.

    For a single response ID, you need to fill in the verdict and comment only once. The rest should be left empty or deleted.

    • If there is one task on the page, add a verdict and a comment to each response ID (if there is a deviation).
    • If there is more than one task on the page, then different tasks on the page will have the same response ID.

    Explanation

    • ASSIGNMENT:assignment_id — ID of the assigned task page.

    • ACCEPT:verdict — Review result:

      • “+” if you accept the responses.

      • “-” if you reject the responses.

    • ACCEPT:comment — Comments for performers if responses were rejected (for example, specify which part of the instructions wasn't followed).

  5. Upload the edited TSV file to Yandex.Toloka (Import/ Export → Upload results).

Delegate assignment review to tolokers.

Place a separate task for performers to review the responses. To learn how to do it, see the last project in the Selecting an image area. It implements assignment review with the help of performers.

Note. Note that you can't change the task status if the task pool was archived.

Appeal

Within 7 days after the review, the performer can challenge the result by filing an appeal.

You will receive a message from that contains:

  • Toloker ID (assignment_id) in the subject.
  • Explanation of why responses should be accepted (optional).

To consider an appeal, you need the response ID. If the performer wrote to you from the rejected response page, the response ID is automatically attached to the toloker's message as a link. If there is no ID, request it from the toloker.

Do the following:

  • If the responses were mistakenly rejected, accept them.
  • Consider all appeals and respond on time — within 9 days (from the moment when the task was rejected).

Troubleshooting

Can I ask a performer to redo the task if they made mistakes in it?

No. After sending a task, the performer can't make any changes to it. You can add tasks that were completed incorrectly to a new pool.

Can I fix something in a completed task myself?

No, you can't fix anything in the task itself. However, you can do this manually in the results file.

What should I do if I want to accept a completed task but the pool is already archived or the performer wrote to me after the allowed time?

Simply give the performer a separate reward without changing the task status. You can't change the task status in the pool in this case.

What should I do if I rejected a task for a reason that isn't specified in the instructions?

Accept the task and update the instructions. Otherwise, you violate the requirement to clearly state the task requirements and the results expected from the user.

Can I reject part of the responses on the page and accept part of them?

No. For example, there are 10 tasks on the page that cost 0.1$, and the performer did only 2 of them incorrectly.

You can't accept the correct answers and pay him part of the cost - $0.08. Response pages are accepted or rejected only as a whole.