Aggregation of results

  1. Dawid-Skene aggregation model
  2. Aggregation by skill
If tasks were issued with an overlap more than 2, run aggregation of results. Yandex.Toloka will process all performers' responses for the task and issue the resulting response and its confidence level.
Note. If you run the pool with the assignment review, make sure that all responses are accepted.
  1. Open the pool.
  2. Click next to the Download results button.
  3. Choose the aggregation method:

Aggregation takes from several minutes to several hours. Track the process on the Operations page. When aggregation is complete, download the TSV file with the results.

Dawid-Skene aggregation model

Analyzes all responses from performers and returns the final response and its statistical significance.
Note. Description of the Dawid-Skene method.

Aggregation can be applied only to certain kinds of output data fields:

Fields that can be aggregated
  • Fields with allowed values.
    How do I add allowed values?
    1. Go to project editing and scroll to the Specifications section.
    2. Hover the mouse over the output data field and click .
    3. Add allowed values.
      Example
    4. Save the field.

    5. Save the project.

      Attention. If you edit a required field, the changes apply only to new pools. Existing pools will continue using the previous version of the project.
  • Fields with a set of values in the task interface.
    Example

    The result field has the string type.

    Task interface:

    {{field type="radio" name="result" value="OK" label="Хорошее" hotkey="1"}}
{{field type="radio" name="result" value="BAD" label="Плохое" hotkey="2"}}
{{field type="radio" name="result" value="404" label="Ошибка загрузки" hotkey="3"}}
Fields that can't be aggregated
  • Array.
  • File.
  • Coordinates.
  • JSON object.

TSV file with aggregated responses contains fields CONFIDENCE: <field name output> - the response significance as a percentage.

Aggregation by skill

Aggregates responses based on the level of trust in the performer. The confidence level is determined by the performer's skill value.

Use this aggregation method for:

Pools with incremental relabeling

Choose the fields and the skill that are set for incremental relabeling.

Pools without incremental relabeling

You can run the aggregation by skill if:

  • You set a skill that defines the confidence level for the performer's responses.
    Example
    For example, a skill calculated as a percentage of:
  • The project output data fields that you want to aggregate have allowed values.
    Output data fields that can be aggregated:
    • Strings and numbers with allowed values.
    • Boolean.
    • Integers with minimum and maximum values. The maximum difference between them is 32.
    How do I add allowed values?
    1. Go to project editing and scroll to the Specifications section.
    2. Hover the mouse over the output data field and click .
    3. Add allowed values.
      Example
    4. Save the field.

    5. Save the project.

      Attention. If you edit a required field, the changes apply only to new pools. Existing pools will continue using the previous version of the project.
  • The tasks were uploaded in the pool with “smart mixing”.

The TSV file with aggregated responses contains CONFIDENCE: <output data field name> fields, which indicate the confidence in the aggregated response.