Aggregation of results
- Open the pool.
- Click
next to the Download results button.
- Choose the aggregation method:
Aggregation takes from several minutes to several hours. Track the process on the Operations page. When aggregation is complete, download the TSV file with the results.
Dawid-Skene aggregation model
Aggregation can be applied only to certain kinds of output data fields:
- Fields that can be aggregated
-
- Fields with allowed values.
- Go to project editing and scroll to the Specifications section.
- Hover the mouse over the output data field and click
.
- Add allowed values.Example
- Save the field.
Save the project.
Attention. If you edit a required field, the changes apply only to new pools. Existing pools will continue using the previous version of the project.
How do I add allowed values? - Fields with a set of values in the task interface.Example
The
result
field has the string type.{{field type="radio" name="result" value="OK" label="Хорошее" hotkey="1"}} {{field type="radio" name="result" value="BAD" label="Плохое" hotkey="2"}} {{field type="radio" name="result" value="404" label="Ошибка загрузки" hotkey="3"}}
- Fields that can't be aggregated
-
- Array.
- File.
- Coordinates.
- JSON object.
TSV file with aggregated responses contains fields CONFIDENCE: <field name output>
- the response significance as a percentage.
Aggregation by skill
Aggregates responses based on the level of trust in the performer. The confidence level is determined by the performer's skill value.
Use this aggregation method for:
- Pools with incremental relabeling
-
Choose the fields and the skill that are set for incremental relabeling.
- Pools without incremental relabeling
-
You can run the aggregation by skill if:
- You set a skill that defines the confidence level for the performer's responses.ExampleFor example, a skill calculated as a percentage of:
- The project output data fields that you want to aggregate have allowed values.
- Strings and numbers with allowed values.
- Boolean.
- Integers with minimum and maximum values. The maximum difference between them is 32.
- Go to project editing and scroll to the Specifications section.
- Hover the mouse over the output data field and click
.
- Add allowed values.Example
- Save the field.
Save the project.
Attention. If you edit a required field, the changes apply only to new pools. Existing pools will continue using the previous version of the project.
Output data fields that can be aggregated:How do I add allowed values? - The tasks were uploaded in the pool with “smart mixing”.
The TSV file with aggregated responses contains CONFIDENCE: <output data field name>
fields, which indicate the confidence in the aggregated response.