Aggregation of results

If tasks were issued with an overlap more than 2, run aggregation of results. Toloka will process all performers' responses for the task and issue the resulting response and its confidence level.
Note. If you run the pool using non-automatic acceptance, make sure that all responses are accepted.
  1. Open the pool.
  2. Click next to the Download results button.
  3. Choose the aggregation method:

Aggregation takes from several minutes to several hours. Track the process on the Operations page. When aggregation is complete, download the TSV file with the results.

Result aggregation according to David-Skin

Analyzes all performers' responses and returns the final response and its statistical significance.
Note. Description of the David-Skin method.

Aggregation can be applied only to certain kinds of output data fields:

Fields that can be aggregated
  • Fields with allowed values.
    How do I add allowed values?
    1. Go to project editing and scroll to the Specifications section.
    2. Hover the mouse over the output data field and click .
    3. Add allowed values.
      Example
    4. Save the field.

    5. Save the project.

      Attention. If you edit a required field, the changes apply only to new pools. Existing pools will continue using the previous version of the project.
  • Fields with a set of values in the task interface.
    Example

    The result field has the string type.

    Task interface:

    {{field type="radio" name="result" value="OK" label="Good" hotkey="1"}}
{{field type="radio" name="result" value="BAD" label="Bad" hotkey="2"}}
{{field type="radio" name="result" value="404" label="Loading error" hotkey="3"}}
Fields that can't be aggregated
  • Array.
  • File.
  • Coordinates.
  • JSON object.

TSV file with aggregated responses contains fields CONFIDENCE: <field name output> - the response significance as a percentage.

Aggregation of results by skill

Aggregates responses based on the level of trust in the performer. The confidence level is determined by the performer's skill value.

Use this aggregation method for:

Pools with dynamic overlap

Choose the fields and the skill set in the dynamic overlap.

Pools without dynamic overlap

You can run the aggregation based on majority vote if:

  • You set a skill that defines the confidence level for the performer's responses.
    Example
    For example, a skill calculated as a percentage of:
  • The output data fields that you want to aggregate have allowed values.
    Output data fields that can be aggregated:
    • Strings and numbers with allowed values.
    • Boolean.
    • Integers with minimum and maximum values. The maximum difference between them is 32.
    How do I add allowed values?
    1. Go to project editing and scroll to the Specifications section.
    2. Hover the mouse over the output data field and click .
    3. Add allowed values.
      Example
    4. Save the field.

    5. Save the project.

      Attention. If you edit a required field, the changes apply only to new pools. Existing pools will continue using the previous version of the project.
  • The tasks were uploaded with “smart mixing”.

The TSV file with aggregated responses contains CONFIDENCE: <output data field name> fields, which indicate the confidence in the aggregated response.