Toloka

Reviewing assignments

You can review assignments manually and reject them if they weren't completed satisfactorily. Rejected assignments are not paid.

Reviewing assignments may take from 1 to 21 days (this period is set by the requester). When the period ends, assignments that haven't been reviewed are automatically approved. The user can challenge the results by writing to the requester. The decision can be changed within 30 days from when the assignment was sent for review.

Setting up assignments review for a pool

To be able to review assignments when they are completed, make the settings before starting the pool:

  1. Choose the Offline accept parameter on the pool page in edit mode.

  2. In Deadline, enter the number of days for reviewing (from 1 to 21 days). The user will see the deadline in the task information on the main Toloka page.

  3. Save your settings.

Users' responses will become available for reviewing as assignments are completed.

Note. 

Explain the acceptance criteria in the instructions for users.

Resending assignments for completion

If you reject an assignment, Toloka can resend it for completion to other users. To set up automated resending of assignments, add After review re-assessment quality control rule to the pool.

Performing the review

You can review assignments online or upload review results in TSV file.

To accept or reject one task:

  1. Click the Review assignments button on the pool page.

  2. Choose an assignment.

    If responses include files uploaded by the users, click the ActionsDownload attachments button to download them. To download all files in submitted assignments in a ZIP archive, click Download resultsDownload attachments on the assignments review page.

  3. Review the assignment, click Accept or Decline. For rejected assignements, enter a comment (explain why you declined it).

To accept or reject multiple assignments at once, select the tasks in the list and click Accept or Decline.

To accept or reject assignments:

  1. Click the Review assignments button on the pool page.

  2. Click the Import/ Export button and choose “Submitted” to download responses that you haven't reviewed.

  3. Open the file with responses. Fill in the fields (the content of the task fields in each task suite should be the same):

    • ASSIGNMENT:assignment_id — ID of the assigned task suite.

    • ACCEPT:verdict — Review result:

      • “+” if you accept the responses.

      • “-” if you reject the responses.

    • ACCEPT:comment — Comments for users if responses were rejected (for example, specify which part of the instructions wasn't followed).

  4. Upload the edited TSV file to Toloka (Import/ ExportUpload results).