Dynamic overlap

Dynamic overlap lets you change overlap depending on:
  • How well the performers cope with the task.
  • How well performers' responses match.
Dynamic overlap allows you to save money without setting a fixed overlap for all tasks in the pool. It works for tasks with response options and overlap of 2-5 (for example, classification).
  1. Dynamic overlap mechanism
  2. Preparing output data fields
  3. Setting up dynamic overlap
  4. Example

Dynamic overlap mechanism

To set up dynamic overlap, use:
Minimum overlap
The minimum number of performers to assign the task to.
Maximum overlap
The maximum number of performers to assign the task to.
Skill
Skill that sets the level of trust in the performer's responses. For example, the skill can be calculated as a percentage of:
Confidence in response

The confidence the performer's response depends on the skill value.

Confidence in an aggregate response is calculated based on the confidence in each response using a probability algorithm.

Output fields for aggregation
Output data fields, responses to which will be aggregated. These fields must have fixed values.

At first, the task is distributed with minimal overlap. If, after this, the confidence in the aggregated response doesn't reach the specified value, the overlap is increased until:

  • The overlap reaches the maximum or
  • The confidence in the aggregated response reaches the specified value.

Preparing output data fields

To compare the performers' responses, specify fixed values for the output data fields, the responses to which should be aggregated.

Output data fields that can be aggregated:
  • Strings and numbers with allowed values.
  • Logical type.
  • Integers with minimum and maximum values. The gap between them can be up to 32.
Note. Arrays can't be aggregated.

To add a set of fixed values for output data fields:

  1. Go to project editing and scroll to the Specifications section.
  2. Hover the mouse over the output data field and click .
  3. Add allowed values.
    Example
  4. Save the field.
  5. Save the project.
    Attention. If you edit a required field, the changes apply only to new pools. Existing pools will work according to the previous version of the project.

Setting up dynamic overlap

  1. Create a pool and fill the parameters.
  2. Set the minimum overlap in the Overlap field.
  3. Turn on the Dynamic overlap option.
  4. Set the Max overlap.
  5. Specify the Confidence of aggregated answer. The higher the value, the more matching performers' responses are needed for the aggregated response. Set this value at least to 70.
  6. Choose Skill for users' scoring. It determines the weight of the performer's response.
  7. Check Output fields for answers agreement.
  8. Save the pool.
  9. Upload tasks via smart mixing, otherwise the pool tasks are assigned with minimal overlap.
  10. After the pool is completed, start Aggregation of results by skill.
Note. Approximate budget in the pool statistics is calculated based on the minimal overlap.

Example

Configuring dynamic overlap for simple classification.

The result has 3 allowed values — OK, BAD and 404.

Task 1

Performer A with a skill value 70 responded with OK.

Performer B with skill value 90, also responded with OK.

The confidence in the aggregated response OK is 97.7. The task is considered completed.

Task 2

Performer A with a skill value 70 responded with OK.

Performer B with a skill value 90 responded BAD.

The confidence in the aggregated response BAD is 76.1. The overlap is increased by 1.

Performer B with a skill value 80 responded with BAD.

The confidence in the aggregated response OK is 96.2. The task is considered completed.