Aggregation of results

If tasks were issued with an overlap more than 2, run aggregation of results. Toloka will process all performers' responses for the task and issue the resulting response and its confidence level.
Note. If the pool has reviewing assignments, make sure that all responses are accepted.
  1. Open the pool.
  2. Click next to the Download results button.
  3. Choose the aggregation method:

Aggregation takes from several minutes to several hours. Track the process on the Operations page. When aggregation is complete, download the TSV file with the results.

Result aggregation based on the Dawid-Skene model

Analyzes all performers' responses and returns the final response and its statistical significance.
Note. Description of the David-Skin method.

Aggregation can be applied only to certain kinds of output data fields:

Fields that can be aggregated
  • Fields with allowed values.
    How do I add allowed values?
    1. Go to project editing and scroll to the Specifications section.
    2. Hover the mouse over the output data field and click .
    3. Add allowed values.
      Example
    4. Save the field.
    5. Save the project.
      Attention. If you edit a required field, the changes apply only to new pools. Existing pools will work according to the previous version of the project.
  • Fields with a set of values in the task interface.
    Example

    The result field has the string type.

    Task interface:

    {{field type="radio" name="result" value="OK" label="Good" hotkey="1"}}
{{field type="radio" name="result" value="BAD" label="Bad" hotkey="2"}}
{{field type="radio" name="result" value="404" label="Download error" hotkey="3"}}
Fields that can't be aggregated
  • Array.
  • File.
  • Coordinates.
  • JSON object.

TSV file with aggregated responses contains fields CONFIDENCE: <field name output> - the response significance as a percentage.

Aggregation of results by skill

Aggregates responses based on the level of trust in the performer. The confidence level is determined by the performer's skill value.

Use this aggregation method for:

Pools with dynamic overlap

Choose the fields and the skill set in the dynamic overlap.

Pools without dynamic overlap

You can run the aggregation based on majority vote if:

  • You set a skill that defines the confidence level for the performer's responses.
    Example
    For example, a skill calculated as a percentage of:
  • The output data fields that you want to aggregate have allowed values.
    Output data fields that can be aggregated:
    • Strings and numbers with allowed values.
    • Logical type.
    • Integers with minimum and maximum values. The gap between them can be up to 32.
    How do I add allowed values?
    1. Go to project editing and scroll to the Specifications section.
    2. Hover the mouse over the output data field and click .
    3. Add allowed values.
      Example
    4. Save the field.
    5. Save the project.
      Attention. If you edit a required field, the changes apply only to new pools. Existing pools will work according to the previous version of the project.
  • The tasks were uploaded with “smart mixing”.

TSV file with aggregated responses contains fields CONFIDENCE: <output data field name> confidence in the aggregated response.