Incremental relabeling

Overlap is the number of performers who should complete each task in the pool. To save money, you can set up incremental relabeling.

Incremental relabeling lets you change overlap depending on:
  • How well the performers cope with the task.
  • How well performers' responses match.
Tip. Don't use the Recompletion of assignments from banned users quality control rule with incremental relabeling. This increases the maximum overlap value.

How incremental relabeling works

To set up incremental relabeling, use:
Maximum overlap
The maximum number of performers to assign the same task to.
Skill

Skill reflects how much you trust the performer's responses. To configure incremental relabeling correctly, use the skill calculated as the percentage of correct responses in control tasks.

Confidence in response

The confidence in the performer's response depends on the skill value.

Confidence in an aggregated response is calculated based on the confidence in each response using a probability algorithm.

Output data for aggregation
Output data fields to use for aggregating responses. These fields must have fixed values.

First the task is distributed with minimal overlap. If, after this, the confidence in the aggregated response doesn't reach the specified value, the overlap is increased until:

  • The overlap reaches the maximum or
  • The confidence in the aggregated response reaches the specified value.

Preparing output data fields

To compare the performers' responses, specify fixed values for the output data fields to use for aggregating responses.

Output data fields that can be aggregated:
  • Strings and numbers with allowed values.
  • Boolean.
  • Integers with minimum and maximum values. The maximum difference between them is 32.
Note. Arrays can't be aggregated.

To add a set of fixed values for output data fields:

  1. Go to project editing and scroll to the Specifications section.
  2. Hover the mouse over the output data field and click .
  3. Add allowed values.
    Example
  4. Save the field.
  5. Save the project.

    Attention. If you edit a required field, the changes apply only to new pools. Existing pools will continue using the previous version of the project.

Setting up incremental relabeling

  1. Create a pool and fill in the parameters.
  2. Set the minimum overlap in the Overlap field.
  3. Turn on the Incremental relabeling option.
  4. Set Max overlap.
  5. Specify Confidence of aggregated answer. The higher the value, the more matching performers' responses are needed for the aggregated response. Set this value to at least 70.
  6. Choose Skill for users' scoring. This defines the weight of the performer's response.
  7. Check Output fields for answers agreement.
  8. Save the pool.
  9. Upload tasks via smart mixing, otherwise the pool tasks are assigned with minimal overlap.
  10. After the pool is completed, start Aggregation of results by skill.
Note. Approximate budget in the pool statistics is calculated based on the minimal overlap.

Example

Configuring incremental relabeling for simple classification.

The result has three allowed values: OK, BAD, and 404.

Task 1

Performer A with a skill value of 70 responded with OK.

Performer B with a skill value of 90 also responded with OK.

The confidence in the aggregated OK response is 97.7. The task is considered completed.

Task 2

Performer A with a skill value of 70 responded with OK.

Performer B with a skill value of 90 responded with BAD.

The confidence in the aggregated BAD response is 76.1. The overlap is increased by 1.

Performer C with a skill value of 80 responded with BAD.

The confidence in the aggregated OK response is 96.2. The task is considered completed.

Troubleshooting

What overlap should I set?

Overlap defines how many performers complete the same pool task.

The best overlap is an overlap that provides satisfying quality of results. For most tasks that are not reviewed, overlap from “3” to “5” is enough. If the tasks are simple, overlap of “3” is likely to be enough. For tasks that are reviewed, set overlap to “1”.

Can I change overlap after the pool is started?

Yes. Open edit mode for the pool and set a new overlap value. You don't need to restart the pool. Updating the settings is usually fast, but if there are many tasks, it may take several minutes.