Pools

A pool consists of task suites that are sent out for completion at the same time. In the pool properties, you set the task price, overlap, user selection filters, quality control rules, and so on.

The pool properties should be sent in JSON format. A sample pool in JSON with key descriptions is shown below.

If the project has multiple pools, the order for completing them depends on the parameters:

  • Pools with identical filter settings and price per task are assigned to users in the order in which they were started. The pool that was started earlier will be completed sooner. You can change the order for completing the pools.

  • Pools with different filter settings and/or a different price per task are sent out for completion when the pool opens.

You can use the API for performing the following operations with pools:

Example of a pool in JSON

{
  "id": "197",
  "project_id": "1",
  "private_name": "My first pool",
  "private_comment": "This is my first pool",
  "public_description": "Pool's description ",
  "may_contain_adult_content": false,
  "will_expire": "2020-09-01T13:00",
  "reward_per_assignment": 0.02,
  "dynamic_pricing_config": {
    "type": "SKILL",
    "skill_id": "1289",
    "intervals": [
      {
        "from": 0,
        "to": 60,
        "reward_per_assignment": 0.03
      },
      {
        "from": 61,
        "to": 100,
        "reward_per_assignment": 0.04
      }
    ]
  },
  "assignment_max_duration_seconds": 300,
  "auto_accept_solutions": true,
  "auto_accept_period_day": 7,
  "auto_close_after_complete_delay_seconds": 60,
  "assignments_issuing_config": {
    "issue_task_suites_in_creation_order": false
  },
  "filter": {<settings for filtering users>},
  "quality_control": {
    "training_requirement": {
      "training_pool_id": "21",
      "training_passing_skill_value": 70
    },
    "captcha_frequency": "LOW",
    "configs": {<quality control rule settings>},
    "checkpoints_config": {
      "real_settings": {
        "target_overlap": 5,
        "task_distribution_function": {
          "scope": "PROJECT",
          "distribution": "UNIFORM",
          "window_days": 7,
          "intervals": [
            {
              "from": 1,
              "to": 100,
              "frequency": 5
            },
            {
              "from": 101,
              "to": 1000,
              "frequency": 25
            }
          ]
        }
      },
      "golden_settings": {
        "target_overlap": 5,
        "task_distribution_function": {
          "scope": "PROJECT",
          "distribution": "UNIFORM",
          "window_days": 7,
          "intervals": [
            {
              "from": 1,
              "to": 100,
              "frequency": 5
            },
            {
              "from": 101,
              "to": 1000,
              "frequency": 25
            }
          ]
        }
      },
      "training_settings": {
        "target_overlap": 5,
        "task_distribution_function": {
          "scope": "PROJECT",
          "distribution": "UNIFORM",
          "window_days": 7,
          "intervals": [
            {
              "from": 1,
              "to": 100,
              "frequency": 5
            },
            {
              "from": 101,
              "to": 1000,
              "frequency": 25
            }
          ]
        }
      }
    }
  },
  "dynamic_overlap_config": {
    "type": "BASIC",
    "max_overlap": 5,
    "min_confidence": 0.9,
    "answer_weight_skill_id": "1289",
    "fields": [
      {
        "name": "result"
      }
    ]
  },
  "defaults" : {
    "default_overlap_for_new_task_suites" : 3,
    "default_overlap_for_new_tasks": 3
  },
  "mixer_config": {
    "real_tasks_count": 6,
    "golden_tasks_count": 1,
    "training_tasks_count": 1,
    "min_real_tasks_count": 2,
    "min_golden_tasks_count": 0,
    "min_training_tasks_count": 0,
    "force_last_assignment": true,
    "force_last_assignment_delay_seconds": 10,
    "mix_tasks_in_creation_order": true,
    "shuffle_tasks_in_task_suite": false,
    "golden_task_distribution_function": {
      "scope": "PROJECT",
      "distribution": "UNIFORM",
      "window_days": 7 ,
      "intervals": [
        {
          "from": 1,
          "to": 25,
          "frequency": 5
        },
        {
          "from": 26,
          "to": 1000,
          "frequency": 25
        }
      ]
    },
    "training_task_distribution_function": {
      "scope": "PROJECT",
      "distribution": "UNIFORM",
      "window_days": 7,
      "intervals": [
        {
          "from": 1,
          "to": 25,
          "frequency": 5
        },
        {
          "from": 26,
          "to": 1000,
          "frequency": 25
        }
      ]
    }
  },
  "priority": 10,
  "type": "REGULAR",
  "status" : "CLOSED",
  "last_close_reason": "EXPIRED",
  "created" : "2016-04-08T14:23:04.785"
}
Copied to clipboard

Keys

Key

Value

Data type

Required during creation

project_id

ID of the project that the pool was created for.

String

Yes

private_name

Name of the pool (only visible to the requester).

String

Yes

private_comment

Comments on the pool (only visible to the requester).

String

No

public_description
Description for users. If it is filled in, the text will be displayed instead of the project's public_description in the list of tasks for performers. String No
may_contain_adult_content

Whether the tasks contain adult content.

Boolean

Yes

will_expire

The date and time in UTC when the pool should be closed (even if all the task suites haven't been completed). It uses ISO 8601 format: YYYY-MM-DDThh:mm:ss[.sss].

String

Yes

reward_per_assignment

Payment per task suite in U.S. dollars. For cents, use the dot (".") as the separator. The minimum payment is $0.01.

Only training and control tasks can be uploaded to zero-price pools.

Number

Yes

dynamic_pricing_config

Dynamic pricing settings.

JSON

No

dynamic_pricing_config.type

Parameter type for calculating dynamic pricing. The SKILL value.

String

Yes, if dynamic pricing is used.

dynamic_pricing_config.skill_id

ID of the skill that the task price is based on

String

Yes, if dynamic pricing is used

dynamic_pricing_config.intervals

Skill level intervals. Must not overlap.

A performer with a skill level that is not included in any interval will receive the basic price for a task suite.

JSON

Yes, if dynamic pricing is used

dynamic_pricing_config.intervals.from

Lower bound of the interval. By default 0.

Number from 0 to 100

Yes, if dynamic pricing is used.

dynamic_pricing_config.intervals.to

Upper bound of the interval. By default 100.

Number from 0 to 100

Yes, if dynamic pricing is used.

dynamic_pricing_config.intervals.reward_per_assignment

The price per task suite for a performer with the specified skill level.

Number

Yes, if dynamic pricing is used.

assignment_max_duration_seconds

The time allowed for completing a task suite, in seconds. Tasks not completed within this time are reassigned to other users.

We recommend allowing no more than 60 seconds per task suite (including the time for loading the page and submitting responses).

Number

Yes

auto_accept_solutions

Whether tasks must be checked manually:

  • true — Automatic task acceptance (manual checking isn't necessary).

  • false— The requester will check the tasks.

The default value is true.

Boolean

No

auto_accept_period_day

Time (number of days) for the requester to review the task. If the requester doesn't accept or reject the task within this period, the decision will be made automatically. May take a value from 1 to 21.

Number

No

auto_close_after_complete_delay_seconds

Waiting time (in seconds) before automatic closure of the pool after all tasks are completed. Minimum — 0, maximum — 259200 seconds (three days). By default, the value is 0.

Use it if:
  • Your data processing is close to real time.
  • You need an open pool where you upload tasks.
  • Dynamic overlap is enabled in the pool (dynamic_overlap_config).

Number

No

assignments_issuing_config

Settings for assigning tasks in the pool.

JSON

No

assignments_issuing_config.issue_task_suites_in_creation_order

For pools that don't use “smart mixing” .

Assign task suites in the order in which they were uploaded. For example, for a pool with an overlap of 5, the first task suite is assigned to five users, then the second task suite, and so on.

This parameter is available when the project has "assignments_issuing_type": "AUTOMATED".

Boolean

No

priority

The priority of the pool in relation to other pools in the project with the same task price and set of filters. Users are assigned tasks with a higher priority first.

Possible values: from -100 to 100.

By default the value is 0 .

Number

No

filter

Settings for user selection filters.

JSON

No

quality_control

Settings for quality control rules and the ID of the pool with training tasks.

JSON

No

quality_control.training_requirement

Parameters of the training pool that is linked to the main task pool.

JSON

No

quality_control.training_requirement.training_pool_id

ID of the training pool that is linked to the main task pool.

String

No

quality_control.training_requirement.training_passing_skill_value

Percentage of correct answers in training tasks (from 0 to 100) required in order to access the main tasks. Only the first answer of the user in each task is taken into account.

Number

No

quality_control.captcha_frequency

The frequency of showing captchas:

  • LOW — Show one for every 20 tasks.

  • MEDIUM, HIGH — Show one for every 10 tasks.

By default, captchas aren't displayed.

String

No

quality_control.configs

Quality control rules.

JSON

No

quality_control.checkpoints_config

Selective task review. If you want to use the majority vote quality control method, specify MAJORITY_VOTE in the quality_control parameter. Some tasks are assigned with higher overlap (for example, “5”) and checked. The rest of the tasks are assigned with the overlap set for the pool (for example, “1”) and they are not checked. Selective review lets you improve pool completion quality.

You can reduce the frequency of checking tasks over time.

Example of settings: in the first 100 tasks completed by the user in the pool, assign every 5th task with an overlap of “5” to check matching responses. In the following tasks, every 25th task is issued with an overlap of “5”.

JSON

No

quality_control.checkpoints_config.real_settings

Selective review of main tasks. To make sure selective review is enabled, don't forget to set up task display in mixer_config.

JSON

No

quality_control.checkpoints_config.real_settings.target_overlap

Overlap in tasks with selective review.

Number

Yes, if selective review is used
quality_control.checkpoints_config.real_settings.task_distribution_function
Distribution of tasks with selective review.

JSON

Yes, if selective review is used

quality_control.checkpoints_config.real_settings.task_distribution_function.scope
How to count tasks completed by the user:
  • POOL — Count completed pool tasks.

  • PROJECT — Count completed project tasks.

String

Yes, if selective review is used

quality_control.checkpoints_config.real_settings.task_distribution_function.distribution
Distribution of tasks with selective review within an interval. This parameter has only one possible value — UNIFORM.

String

Yes, if selective review is used

quality_control.checkpoints_config.real_settings.task_distribution_function.window_days

Period in which completed tasks are counted (number of days).

Number

Yes, if selective review is used

quality_control.checkpoints_config.real_settings.task_distribution_function.intervals[]

Interval limits and frequency of tasks with selective review.

Array of JSON objects

Yes, if selective review is used

quality_control.checkpoints_config.real_settings.task_distribution_function.intervals.from

Start of the interval (number of tasks completed by the user in the project or in the pool).

Number

Yes, if selective review is used

quality_control.checkpoints_config.real_settings.task_distribution_function.intervals.to
End of the interval (number of tasks completed by the user in the project or in the pool).

Number

Yes, if selective review is used

quality_control.checkpoints_config.real_settings.task_distribution_function.intervals.frequency

Frequency of tasks with selective review within an interval. The first task in an interval is a majority vote task. For example, if you set frequency: 3, then tasks number 1, 4, 7, and so on, will be majority vote tasks.

Number

Yes, if selective review is used

quality_control.checkpoints_config.golden_settings

Selective review of control tasks. To make sure selective review is enabled, don't forget to set up display of this type of task in mixer_config.

JSON

No

quality_control.checkpoints_config.golden_settings.target_overlap

Overlap in control tasks with selective review.

Number

Yes, if selective review is used
quality_control.checkpoints_config.golden_settings.task_distribution_function
Distribution of control tasks with selective review.

JSON

Yes, if selective review is used

quality_control.checkpoints_config.golden_settings.task_distribution_function.scope
How to count tasks completed by the user:
  • POOL — Count completed pool tasks.

  • PROJECT — Count completed project tasks.

String

Yes, if selective review is used

quality_control.checkpoints_config.golden_settings.task_distribution_function.distribution
Distribution of control tasks with selective review within an interval. This parameter has only one possible value — UNIFORM.

String

Yes, if selective review is used

quality_control.checkpoints_config.golden_settings.task_distribution_function.window_days

Period in which completed tasks are counted (number of days).

Number

Yes, if selective review is used

quality_control.checkpoints_config.golden_settings.task_distribution_function.intervals[]

Interval limits and frequency of control tasks with selective review.

Array of JSON objects

Yes, if selective review is used

quality_control.checkpoints_config.golden_settings.task_distribution_function.intervals.from

Start of the interval (number of tasks completed by the user in the project or in the pool).

Number

Yes, if selective review is used

quality_control.checkpoints_config.golden_settings.task_distribution_function.intervals.to
End of the interval (number of tasks completed by the user in the project or in the pool).

Number

Yes, if selective review is used

quality_control.checkpoints_config.golden_settings.task_distribution_function.intervals.frequency

Frequency of control tasks with selective review within an interval. The first task in an interval is a majority vote task. For example, if you set frequency: 3, then tasks number 1, 4, 7, and so on, will be majority vote tasks.

Number

Yes, if selective review is used

quality_control.checkpoints_config.training_settings

Selective review of training tasks. To make sure selective review is enabled, don't forget to set up display of this type of task in mixer_config.

JSON

No

quality_control.checkpoints_config.training_settings.target_overlap

Overlap in training tasks with selective review.

Number

Yes, if selective review is used
quality_control.checkpoints_config.training_settings.task_distribution_function
Distribution of training tasks with selective review.

JSON

Yes, if selective review is used

quality_control.checkpoints_config.training_settings.task_distribution_function.scope
How to count tasks completed by the user:
  • POOL — Count completed pool tasks.

  • PROJECT — Count completed project tasks.

String

Yes, if selective review is used

quality_control.checkpoints_config.training_settings.task_distribution_function.distribution
Distribution of training tasks with selective review within an interval. This parameter has only one possible value — UNIFORM.

String

Yes, if selective review is used

quality_control.checkpoints_config.training_settings.task_distribution_function.window_days

Period in which completed tasks are counted (number of days).

Number

Yes, if selective review is used

quality_control.checkpoints_config.training_settings.task_distribution_function.intervals[]

Interval limits and frequency of training tasks with selective review.

Array of JSON objects

Yes, if selective review is used

quality_control.checkpoints_config.training_settings.task_distribution_function.intervals.from

Start of the interval (number of tasks completed by the user in the project or in the pool).

Number

Yes, if selective review is used

quality_control.checkpoints_config.training_settings.task_distribution_function.intervals.to
End of the interval (number of tasks completed by the user in the project or in the pool).

Number

Yes, if selective review is used

quality_control.checkpoints_config.training_settings.task_distribution_function.intervals.frequency

Frequency of training tasks with selective review within an interval. The first task in an interval is a majority vote task. For example, if you set frequency: 3, then tasks number 1, 4, 7, and so on, will be majority vote tasks.

Number

Yes, if selective review is used

dynamic_overlap_config

Setting up dynamic overlap (also known as incremental relabeling or IRL). Allows you to change the overlap depending on how well the performers handle the task.

Set the closing interval (auto_close_after_complete_delay_seconds). It should be enough to complete tasks with an overlap higher than the minimum.

When all pool tasks are completed, aggregate the responses.

JSON No
dynamic_overlap_config.type

The algorithm for dynamic overlap.

BASIC — Each response is assigned a weight depending on the performer's skill value. The aggregated response confidence is calculated based on the probability algorithm. The task overlap increases until it reaches max_overlap or until the confidence of the aggregated response exceeds min_confidence.

You have to specify max_overlap, min_confidence, answer_weight_skill_id and fields.

String Yes, if dynamic overlap is used
dynamic_overlap_config.max_overlap

Maximum overlap. Must be higher than the values in defaults. Minimum — 1. Maximum — 30000.

Number Yes, if dynamic overlap is used
dynamic_overlap_config.min_confidence

Minimum confidence of the aggregated response. Values from 0 to 1.

Floating-point number Yes, if dynamic overlap is used
dynamic_overlap_config.answer_weight_skill_id
A skill that determines the weight of the performer's response. For best results, use a skill calculated as the percentage of correct responses in control tasks. String Yes, if dynamic overlap is used
dynamic_overlap_config.fields

Output data fields to use for aggregating responses. For best results, each of these fields must have a limited number of response options.

Don't specify several fields if their values depend on each other.

JSON Yes, if dynamic overlap is used
dynamic_overlap_config.fields.name
The output data field name. String Yes, if dynamic overlap is used
defaults

Settings that are applied by default when uploading new task suites to a pool.

JSON

Yes

defaults.default_overlap_for_new_task_suites

The overlap for task suites that are uploaded to the pool (used if the allow_defaults=true parameter is set at upload).

Number

Yes

defaults.default_overlap_for_new_tasks

The overlap for tasks that are uploaded to the pool (used if the allow_defaults=true parameter is set at upload).

Number

No

mixer_config

Parameters for automatically creating a task suite (“smart mixing”). For more information about creating task suites, see the Requester's guide.

JSON

Yes, if “smart mixing” is used

mixer_config.real_tasks_count

Number of main tasks per suite.

The maximum number of tasks per task suite if golden_task_distribution_function or training_task_distribution_function is used.

Number

Yes, if “smart mixing” is used

mixer_config.golden_tasks_count

Number of control tasks per suite.

Number

Yes, if “smart mixing” is used

mixer_config.training_tasks_count

Number of training tasks per suite.

Number

Yes, if “smart mixing” is used

mixer_config.min_real_tasks_count

Minimum number of main tasks in a task suite (if the number of assignments left is less than the number specified in mixer_config.real_tasks_count). Minimum — 0. By default, the value is the same as in mixer_config.real_tasks_count.

Number

No

mixer_config.min_golden_tasks_count

Minimum number of control tasks per task suite (if the number of assignments left is less than the number specified in mixer_config.golden_tasks_count). Minimum — 0. By default, the value is the same as in mixer_config.golden_tasks_count.

Number

No

mixer_config.min_training_tasks_count

Minimum number of training tasks per task suite (if the number of assignments left is less than the number specified in mixer_config.golden_tasks_count). Minimum — 0. By default, the value is the same as in mixer_config.training_tasks_count.

Number

No

mixer_config.force_last_assignment

Setting for the last task suite in the pool, if less than the minimum remaining number of tasks are not completed (mixer_config.min_real_tasks_count). Values:

  • true — Assign an incomplete task suite.

  • false — Don't assign tasks. This option can be used if you are adding tasks after the pool is started.

The default value is true.

This parameter only applies to main tasks. The number of control and training tasks in the last suite must be complete (mixer_config.golden_tasks_count, mixer_config.training_tasks_count).

Boolean

No

mixer_config.force_last_assignment_delay_seconds

Waiting time (in seconds) after adding a task or increasing the overlap before assigning the last task suite in the pool. The minimum is 0 and the maximum is 86,400 seconds (one day).

This parameter can be used if the pool has force_last_assignment: true.

Number

No

mixer_config.mix_tasks_in_creation_order

The order used to add tasks to suites:

  • true — Add tasks to suites in the order in which they were uploaded. For example, in a pool with an overlap of 5, the first uploaded task will be added to the first 5 task suites. They will be assigned to 5 users.

  • false — Add tasks to suites in random order.

The default value is false.

Boolean

No

mixer_config.shuffle_tasks_in_task_suite

The order of tasks within a suite:

  • true — Random.

  • false — The order in which tasks were uploaded.

The default value is true.

Boolean

No

mixer_config.golden_task_distribution_function

Vary the frequency of control tasks. The option allows you to change the frequency of checking as the user completes more tasks.

If it is filled in, use real_task_count to specify the maximum number of tasks per task suite. To use the option, set the golden_task_count:0 and min_golden_tasks_count:null parameter values.

The interval bound is the number of tasks in the pool. If the task suite has several intervals, the settings for the first interval are applied.

Example:
In the first task suite (the range from 1 to 25), every fifth task is a control task, and in the following suites, every 25th task is a control task.

JSON

No

mixer_config.golden_task_distribution_function.scope
How to count tasks completed by the user:
  • POOL — Count completed pool tasks.

  • PROJECT — Count completed project tasks.

String

Yes, if you use uneven distribution of control tasks

mixer_config.golden_task_distribution_function.distribution

Distribution of control tasks within an interval. This parameter has only one possible value — UNIFORM.

String

Yes, if you use uneven distribution of control tasks

mixer_config.golden_task_distribution_function.window_days

Period in which completed tasks are counted (number of days).

Number

Yes, if you use uneven distribution of control tasks

mixer_config.golden_task_distribution_function.intervals[]

Interval borders and number of control tasks in an interval.

Array of JSON objects

Yes, if you use uneven distribution of control tasks

mixer_config.golden_task_distribution_function.intervals.from

Start of the interval (number of tasks completed by the user in the project or in the pool).

Number

Yes, if you use uneven distribution of control tasks

mixer_config.golden_task_distribution_function.intervals.to

End of the interval (number of tasks completed by the user in the project or in the pool).

Number

Yes, if you use uneven distribution of control tasks

mixer_config.golden_task_distribution_function.intervals.frequency

Frequency of control tasks in an interval. The first task in an interval is a control task. For example, if you set frequency: 3 tasks number 1, 4, 7 and so on will be control tasks.

Number

Yes, if you use uneven distribution of control tasks

mixer_config.training_task_distribution_function

Issue of training tasks with uneven frequency. This option allows you to change the frequency of training tasks as the user completes more tasks.

If it is filled in, use real_task_count to specify the maximum number of tasks per task suite. To use the option, set parameter values training_task_count:0 and min_training_tasks_count:null.

Example:
In the first task suite (the range from 1 to 25), every fifth task is a training task, and in the following suites, every 25th task is a training task.

JSON

No

mixer_config.training_task_distribution_function.scope
How to count tasks completed by the user:
  • POOL — Count completed pool tasks.

  • PROJECT — Count completed project tasks.

String

Yes, if you use uneven distribution of training tasks

mixer_config.training_task_distribution_function.distribution

Distribution of training tasks within an interval. This parameter has only one possible value — UNIFORM.

String

Yes, if you use uneven distribution of training tasks

mixer_config.training_task_distribution_function.window_days

Period in which completed tasks are counted (number of days).

Number

Yes, if you use uneven distribution of training tasks

mixer_config.training_task_distribution_function.intervals[]

Interval borders and number of control tasks in an interval.

Array of JSON objects

Yes, if you use uneven distribution of training tasks

mixer_config.training_task_distribution_function.intervals.from

Start of the interval (number of tasks completed by the user in the project or in the pool).

Number

Yes, if you use uneven distribution of training tasks

mixer_config.training_task_distribution_function.intervals.to

End of the interval (number of tasks completed by the user in the project or in the pool).

Number

Yes, if you use uneven distribution of training tasks

mixer_config.training_task_distribution_function.intervals.frequency

Frequency of training tasks in an interval. The first task in an interval is a training task. For example, if you set frequency: 3 tasks number 1, 4, 7 and so on will be training tasks.

Number

Yes, if you use uneven distribution of training tasks

id

Pool ID.

String

Not used

status

Status of the pool:

  • OPEN

  • CLOSED

  • ARCHIVED

String

Not used

last_close_reason

The reason for closing the pool the last time:

  • MANUAL — Closed by the requester.

  • EXPIRED — Reached the date and time set in will_expire.

  • COMPLETED — Closed automatically because all the pool tasks were completed.

  • NOT_ENOUGH_BALANCE — Closed automatically because the Toloka account ran out of funds.

  • ASSIGNMENTS_LIMIT_EXCEEDED — Closed automatically because it exceeded the limit on assigned task suites (2 million maximum).

  • BLOCKED — Closed automatically because the requester's account was blocked by a Toloka administrator.

String

Not used

created

The UTC date and time when the pool was created, in ISO 8601 format: YYYY-MM-DDThh:mm:ss[.sss].

String

Not used

last_started

The date and time when the pool was last started, in UTC in ISO 8601 format: YYYY-MM-DDThh:mm:ss[.sss].

String

Not used

last_stopped

The date and time when the pool was last stopped, in UTC in ISO 8601 format: YYYY-MM-DDThh:mm:ss[.sss].

String

Not used

type
Types of pool:
  • REGULAR - normal pool;

  • TRAINING - training pool.

String

Not used

Key

Value

Data type

Required during creation

project_id

ID of the project that the pool was created for.

String

Yes

private_name

Name of the pool (only visible to the requester).

String

Yes

private_comment

Comments on the pool (only visible to the requester).

String

No

public_description
Description for users. If it is filled in, the text will be displayed instead of the project's public_description in the list of tasks for performers. String No
may_contain_adult_content

Whether the tasks contain adult content.

Boolean

Yes

will_expire

The date and time in UTC when the pool should be closed (even if all the task suites haven't been completed). It uses ISO 8601 format: YYYY-MM-DDThh:mm:ss[.sss].

String

Yes

reward_per_assignment

Payment per task suite in U.S. dollars. For cents, use the dot (".") as the separator. The minimum payment is $0.01.

Only training and control tasks can be uploaded to zero-price pools.

Number

Yes

dynamic_pricing_config

Dynamic pricing settings.

JSON

No

dynamic_pricing_config.type

Parameter type for calculating dynamic pricing. The SKILL value.

String

Yes, if dynamic pricing is used.

dynamic_pricing_config.skill_id

ID of the skill that the task price is based on

String

Yes, if dynamic pricing is used

dynamic_pricing_config.intervals

Skill level intervals. Must not overlap.

A performer with a skill level that is not included in any interval will receive the basic price for a task suite.

JSON

Yes, if dynamic pricing is used

dynamic_pricing_config.intervals.from

Lower bound of the interval. By default 0.

Number from 0 to 100

Yes, if dynamic pricing is used.

dynamic_pricing_config.intervals.to

Upper bound of the interval. By default 100.

Number from 0 to 100

Yes, if dynamic pricing is used.

dynamic_pricing_config.intervals.reward_per_assignment

The price per task suite for a performer with the specified skill level.

Number

Yes, if dynamic pricing is used.

assignment_max_duration_seconds

The time allowed for completing a task suite, in seconds. Tasks not completed within this time are reassigned to other users.

We recommend allowing no more than 60 seconds per task suite (including the time for loading the page and submitting responses).

Number

Yes

auto_accept_solutions

Whether tasks must be checked manually:

  • true — Automatic task acceptance (manual checking isn't necessary).

  • false— The requester will check the tasks.

The default value is true.

Boolean

No

auto_accept_period_day

Time (number of days) for the requester to review the task. If the requester doesn't accept or reject the task within this period, the decision will be made automatically. May take a value from 1 to 21.

Number

No

auto_close_after_complete_delay_seconds

Waiting time (in seconds) before automatic closure of the pool after all tasks are completed. Minimum — 0, maximum — 259200 seconds (three days). By default, the value is 0.

Use it if:
  • Your data processing is close to real time.
  • You need an open pool where you upload tasks.
  • Dynamic overlap is enabled in the pool (dynamic_overlap_config).

Number

No

assignments_issuing_config

Settings for assigning tasks in the pool.

JSON

No

assignments_issuing_config.issue_task_suites_in_creation_order

For pools that don't use “smart mixing” .

Assign task suites in the order in which they were uploaded. For example, for a pool with an overlap of 5, the first task suite is assigned to five users, then the second task suite, and so on.

This parameter is available when the project has "assignments_issuing_type": "AUTOMATED".

Boolean

No

priority

The priority of the pool in relation to other pools in the project with the same task price and set of filters. Users are assigned tasks with a higher priority first.

Possible values: from -100 to 100.

By default the value is 0 .

Number

No

filter

Settings for user selection filters.

JSON

No

quality_control

Settings for quality control rules and the ID of the pool with training tasks.

JSON

No

quality_control.training_requirement

Parameters of the training pool that is linked to the main task pool.

JSON

No

quality_control.training_requirement.training_pool_id

ID of the training pool that is linked to the main task pool.

String

No

quality_control.training_requirement.training_passing_skill_value

Percentage of correct answers in training tasks (from 0 to 100) required in order to access the main tasks. Only the first answer of the user in each task is taken into account.

Number

No

quality_control.captcha_frequency

The frequency of showing captchas:

  • LOW — Show one for every 20 tasks.

  • MEDIUM, HIGH — Show one for every 10 tasks.

By default, captchas aren't displayed.

String

No

quality_control.configs

Quality control rules.

JSON

No

quality_control.checkpoints_config

Selective task review. If you want to use the majority vote quality control method, specify MAJORITY_VOTE in the quality_control parameter. Some tasks are assigned with higher overlap (for example, “5”) and checked. The rest of the tasks are assigned with the overlap set for the pool (for example, “1”) and they are not checked. Selective review lets you improve pool completion quality.

You can reduce the frequency of checking tasks over time.

Example of settings: in the first 100 tasks completed by the user in the pool, assign every 5th task with an overlap of “5” to check matching responses. In the following tasks, every 25th task is issued with an overlap of “5”.

JSON

No

quality_control.checkpoints_config.real_settings

Selective review of main tasks. To make sure selective review is enabled, don't forget to set up task display in mixer_config.

JSON

No

quality_control.checkpoints_config.real_settings.target_overlap

Overlap in tasks with selective review.

Number

Yes, if selective review is used
quality_control.checkpoints_config.real_settings.task_distribution_function
Distribution of tasks with selective review.

JSON

Yes, if selective review is used

quality_control.checkpoints_config.real_settings.task_distribution_function.scope
How to count tasks completed by the user:
  • POOL — Count completed pool tasks.

  • PROJECT — Count completed project tasks.

String

Yes, if selective review is used

quality_control.checkpoints_config.real_settings.task_distribution_function.distribution
Distribution of tasks with selective review within an interval. This parameter has only one possible value — UNIFORM.

String

Yes, if selective review is used

quality_control.checkpoints_config.real_settings.task_distribution_function.window_days

Period in which completed tasks are counted (number of days).

Number

Yes, if selective review is used

quality_control.checkpoints_config.real_settings.task_distribution_function.intervals[]

Interval limits and frequency of tasks with selective review.

Array of JSON objects

Yes, if selective review is used

quality_control.checkpoints_config.real_settings.task_distribution_function.intervals.from

Start of the interval (number of tasks completed by the user in the project or in the pool).

Number

Yes, if selective review is used

quality_control.checkpoints_config.real_settings.task_distribution_function.intervals.to
End of the interval (number of tasks completed by the user in the project or in the pool).

Number

Yes, if selective review is used

quality_control.checkpoints_config.real_settings.task_distribution_function.intervals.frequency

Frequency of tasks with selective review within an interval. The first task in an interval is a majority vote task. For example, if you set frequency: 3, then tasks number 1, 4, 7, and so on, will be majority vote tasks.

Number

Yes, if selective review is used

quality_control.checkpoints_config.golden_settings

Selective review of control tasks. To make sure selective review is enabled, don't forget to set up display of this type of task in mixer_config.

JSON

No

quality_control.checkpoints_config.golden_settings.target_overlap

Overlap in control tasks with selective review.

Number

Yes, if selective review is used
quality_control.checkpoints_config.golden_settings.task_distribution_function
Distribution of control tasks with selective review.

JSON

Yes, if selective review is used

quality_control.checkpoints_config.golden_settings.task_distribution_function.scope
How to count tasks completed by the user:
  • POOL — Count completed pool tasks.

  • PROJECT — Count completed project tasks.

String

Yes, if selective review is used

quality_control.checkpoints_config.golden_settings.task_distribution_function.distribution
Distribution of control tasks with selective review within an interval. This parameter has only one possible value — UNIFORM.

String

Yes, if selective review is used

quality_control.checkpoints_config.golden_settings.task_distribution_function.window_days

Period in which completed tasks are counted (number of days).

Number

Yes, if selective review is used

quality_control.checkpoints_config.golden_settings.task_distribution_function.intervals[]

Interval limits and frequency of control tasks with selective review.

Array of JSON objects

Yes, if selective review is used

quality_control.checkpoints_config.golden_settings.task_distribution_function.intervals.from

Start of the interval (number of tasks completed by the user in the project or in the pool).

Number

Yes, if selective review is used

quality_control.checkpoints_config.golden_settings.task_distribution_function.intervals.to
End of the interval (number of tasks completed by the user in the project or in the pool).

Number

Yes, if selective review is used

quality_control.checkpoints_config.golden_settings.task_distribution_function.intervals.frequency

Frequency of control tasks with selective review within an interval. The first task in an interval is a majority vote task. For example, if you set frequency: 3, then tasks number 1, 4, 7, and so on, will be majority vote tasks.

Number

Yes, if selective review is used

quality_control.checkpoints_config.training_settings

Selective review of training tasks. To make sure selective review is enabled, don't forget to set up display of this type of task in mixer_config.

JSON

No

quality_control.checkpoints_config.training_settings.target_overlap

Overlap in training tasks with selective review.

Number

Yes, if selective review is used
quality_control.checkpoints_config.training_settings.task_distribution_function
Distribution of training tasks with selective review.

JSON

Yes, if selective review is used

quality_control.checkpoints_config.training_settings.task_distribution_function.scope
How to count tasks completed by the user:
  • POOL — Count completed pool tasks.

  • PROJECT — Count completed project tasks.

String

Yes, if selective review is used

quality_control.checkpoints_config.training_settings.task_distribution_function.distribution
Distribution of training tasks with selective review within an interval. This parameter has only one possible value — UNIFORM.

String

Yes, if selective review is used

quality_control.checkpoints_config.training_settings.task_distribution_function.window_days

Period in which completed tasks are counted (number of days).

Number

Yes, if selective review is used

quality_control.checkpoints_config.training_settings.task_distribution_function.intervals[]

Interval limits and frequency of training tasks with selective review.

Array of JSON objects

Yes, if selective review is used

quality_control.checkpoints_config.training_settings.task_distribution_function.intervals.from

Start of the interval (number of tasks completed by the user in the project or in the pool).

Number

Yes, if selective review is used

quality_control.checkpoints_config.training_settings.task_distribution_function.intervals.to
End of the interval (number of tasks completed by the user in the project or in the pool).

Number

Yes, if selective review is used

quality_control.checkpoints_config.training_settings.task_distribution_function.intervals.frequency

Frequency of training tasks with selective review within an interval. The first task in an interval is a majority vote task. For example, if you set frequency: 3, then tasks number 1, 4, 7, and so on, will be majority vote tasks.

Number

Yes, if selective review is used

dynamic_overlap_config

Setting up dynamic overlap (also known as incremental relabeling or IRL). Allows you to change the overlap depending on how well the performers handle the task.

Set the closing interval (auto_close_after_complete_delay_seconds). It should be enough to complete tasks with an overlap higher than the minimum.

When all pool tasks are completed, aggregate the responses.

JSON No
dynamic_overlap_config.type

The algorithm for dynamic overlap.

BASIC — Each response is assigned a weight depending on the performer's skill value. The aggregated response confidence is calculated based on the probability algorithm. The task overlap increases until it reaches max_overlap or until the confidence of the aggregated response exceeds min_confidence.

You have to specify max_overlap, min_confidence, answer_weight_skill_id and fields.

String Yes, if dynamic overlap is used
dynamic_overlap_config.max_overlap

Maximum overlap. Must be higher than the values in defaults. Minimum — 1. Maximum — 30000.

Number Yes, if dynamic overlap is used
dynamic_overlap_config.min_confidence

Minimum confidence of the aggregated response. Values from 0 to 1.

Floating-point number Yes, if dynamic overlap is used
dynamic_overlap_config.answer_weight_skill_id
A skill that determines the weight of the performer's response. For best results, use a skill calculated as the