Incremental relabeling

Overlap is the number of performers who should complete each task in the pool. To save money, you can set up incremental relabeling.

Incremental relabeling lets you change overlap depending on:
  • How well the performers do on the task.
  • How well performers' responses match each other.
Tip. Don't use the Recompletion of assignments from banned users quality control rule with incremental relabeling. This increases the maximum overlap value.

How incremental relabeling works

To set up incremental relabeling, use:
Maximum overlap
The maximum number of performers to assign the same task to.
Skill

Skill reflects how much you trust the performer's responses. To configure incremental relabeling correctly, use the skill calculated as the percentage of correct responses in control tasks.

Confidence in response

The confidence in the performer's response depends on the skill value.

Confidence in an aggregated response is calculated based on the confidence in each response using a probability algorithm.

Output data for aggregation
Output data fields to use for aggregating responses. These fields must have fixed values.

First the task is distributed with minimal overlap. If, after this, the confidence in the aggregated response doesn't reach the specified value, the overlap is increased until:

  • The overlap reaches the maximum or
  • The confidence in the aggregated response reaches the specified value.

Preparing output data fields

To compare the performers' responses, specify fixed values for the output data fields to use for aggregating responses.

Output data fields that can be aggregated:
  • Strings and numbers with allowed values.
  • Boolean.
  • Integers with minimum and maximum values. The maximum difference between them is 32.
Note. Arrays can't be aggregated.

To add a set of fixed values for output data fields:

  1. Go to project editing and scroll to the Specifications section.
  2. Hover the mouse over the output data field and click .
  3. Add allowed values.
    Example
  4. Save the field.
  5. Save the project.

    Attention. If you edit a required field, the changes apply only to new pools. Existing pools will continue using the previous version of the project.

Setting up incremental relabeling

  1. Create a pool and fill in the parameters.
  2. Set the minimum overlap in the Overlap field.
  3. Turn on the Incremental relabeling option.
  4. Set Max overlap.
  5. Specify Confidence of aggregated answer. The higher the value, the more matching performers' responses are needed for the aggregated response. Set this value to at least 70.
  6. Choose Skill for users' scoring. This defines the weight of the performer's response.
  7. Check Output fields for answers agreement.
  8. Save the pool.
  9. Upload tasks via smart mixing, otherwise the pool tasks are assigned with minimal overlap.
  10. After the pool is completed, start Aggregation of results by skill.
Note. Approximate budget in the pool statistics is calculated based on the minimal overlap.

Example

Configuring incremental relabeling for simple classification.

The result has three allowed values: OK, BAD, and 404.

Task 1

Performer A with a skill value of 70 responded with OK.

Performer B with a skill value of 90 also responded with OK.

The confidence in the aggregated OK response is 97.7. The task is considered completed.

Task 2

Performer A with a skill value of 70 responded with OK.

Performer B with a skill value of 90 responded with BAD.

The confidence in the aggregated BAD response is 76.1. The overlap is increased by 1.

Performer C with a skill value of 80 responded with BAD.

The confidence in the aggregated OK response is 96.2. The task is considered completed.

Troubleshooting

What overlap should I set?

Overlap defines how many performers complete the same pool task.

The best overlap is an overlap that provides satisfying quality of results. For most tasks that are not reviewed, overlap from “3” to “5” is enough. If the tasks are simple, overlap of “3” is likely to be enough. For tasks that are reviewed, set overlap to “1”.

Can I change overlap after the pool is started?

Yes. Open edit mode for the pool and set a new overlap value. You don't need to restart the pool. Updating the settings is usually fast, but if there are many tasks, it may take several minutes.

Can it happen with incremental relabeling that the pool closes before the tasks for minimal overlap run out? The overlap increased, and the pool closed, and I need to start it manually.

Yes, this might happen. You must set an adequate pool closing interval.

How does counting work if I set overlap = 3 in the pool and response threshold = 3 in the majority vote?

In this case, if you don't have 3 identical responses for your task (response threshold), no user would be considered a good or poor performer, because the system can't see which of the users made an error.

But if you set response threshold = 2 with overlap = 3, then two users with the same responses are considered good performers, but the third user, who gives a different response, is a poor performer.

Can I do it like this: set a basic overlap of 2 users, then, if both performers select the same response, close the pool, but if they give different responses, show the task to one more user?

Yes, you can do that. Set up incremental relabeling.

Is there a cross-check feature for tasks?

You can use overlap to let multiple performers do the same task. The overlap value is set up in the pool settings.

Why is the maximum number of completed tasks in the progress bar less than the total number of uploaded tasks?

The progress bar shows the number of task suites including the overlap. If the overlap is greater than one, the number of task suites is different from the total number of tasks.