Aggregation of results

  1. Dawid-Skene aggregation model
  2. Aggregation by skill
If tasks were issued with an overlap of 2 or higher, run aggregation of results. Toloka will process all performers' responses for the task and issue the resulting response and its confidence level.
Note. If you run the pool with the assignment review, make sure that all responses are accepted.
  1. Open the pool.
  2. Click next to the Download results button.
  3. Choose the aggregation method:

Aggregation takes from several minutes to several hours. Track the progress on the Operations page. When aggregation is complete, download the TSV file with the results.

To receive notifications and emails when results aggregation is completed, set up notifications:

  1. Log in to your account.
  2. Go to Profile → Notifications  → Pool or aggregation completed.
  3. Choose the notification method:
    • Email: Messages will be sent to your email address.
    • Messages: Notifications will be displayed under Messages in your account. Apart from you, those who set up shared access to your account can see them.
    • Browser: Notifications will be sent to the devices that you logged in to your account from.

Dawid-Skene aggregation model

Analyzes all performers' responses and returns the final response and its statistical significance.
Note. Description of the Dawid-Skene method.

Aggregation can be applied only to certain kinds of output data fields:

Fields that can be aggregated
  • Fields with allowed values.
    How do I add allowed values?
    1. Go to project editing and scroll to the Specifications section.
    2. Hover the mouse over the output data field and click .
    3. Add allowed values.
      Example
    4. Save the field.

    5. Save the project.

      Attention. If you edit a required field, the changes apply only to new pools. Existing pools will continue using the previous version of the project.
  • Fields with a set of values in the task interface.
    Example

    The result field has the string type.

    Task interface:

    {{field type="radio" name="result" value="OK" label="Good" hotkey="1"}}
{{field type="radio" name="result" value="BAD" label="Bad" hotkey="2"}}
{{field type="radio" name="result" value="404" label="Loading error" hotkey="3"}}
    Copied to clipboard
Fields that can't be aggregated
  • Array.
  • File.
  • Coordinates.
  • JSON object.

TSV file with aggregated responses contains fields CONFIDENCE: <field name output> - the response significance as a percentage.

Aggregation by skill

Aggregates responses based on the level of trust in the performer. The confidence level is determined by the performer's skill value.

Use this aggregation method for:

Pools with dynamic overlap

Choose the fields and the skill that are set for dynamic overlap.

Pools without dynamic overlap

You can run skill-based aggregation if:

  • You set a skill that defines the confidence level for the performer's responses.
    Example
    For example, a skill calculated as a percentage of:
  • The project output data fields that you want to aggregate have allowed values.
    Output data fields that can be aggregated:
    • Strings and numbers with allowed values.
    • Boolean.
    • Integers with minimum and maximum values. The maximum difference between them is 32.
    How do I add allowed values?
    1. Go to project editing and scroll to the Specifications section.
    2. Hover the mouse over the output data field and click .
    3. Add allowed values.
      Example
    4. Save the field.

    5. Save the project.

      Attention. If you edit a required field, the changes apply only to new pools. Existing pools will continue using the previous version of the project.
  • The tasks were uploaded in the pool with “smart mixing”.

The TSV file with aggregated responses contains CONFIDENCE: <output data field name> fields, which indicate the confidence in the aggregated response.

Troubleshooting

How does the Dawid-Skene aggregation model work?
The Dawid-Skene aggregation model analyzes the performer responses and creates an error matrix for each performer. This lets us evaluate the statistical significance of the performer in the context of each assignment. Learn more about the model.
Where do I see the aggregation progress?

The pool page contains the List of Operations button.

Why might aggregation by performer skill be unavailable?

You cannot aggregate by project fields that have no valid values. Specify the possible values for all the fields of all types.

You can't aggregate by skill. When running via the API, I get the error code ONLY_FOR_POOL_WITH_MIXER. Why?

You need to use smart mixing.