Tasks

In order to upload tasks to a pool, you need to create JSON objects that contain:

  • The input data for the tasks (such as text or image URLs).

  • Correct responses (for control tasks).

  • Hints (for training tasks).

Toloka automatically combines tasks into suites. You only need to specify the number of tasks to include in each suite (the mixer_config key in the pool).

If you want to create the task suites yourself, follow the instructions in Task suites. For more information about creating task suites, see the Requester's guide.

A sample task in JSON with key descriptions is shown below.

You can use the API for performing the following operations with tasks:

Sample task in JSON

{
   "id":"95199da8-77f4-4ceb-8316-2916a84b4641",
   "origin_task_id": "e3da7fe1-828d-4d9c-b49d-42c0eb5fcfde",
   "pool_id": "1",
   "input_values": {
      "image_url": "www.images/image1.ru"
   },
   "known_solutions": [
      {
         "output_values": {
            "result": "OK",
            "like": false
         },
         "correctness_weight": 0.8
      },
      {
         "output_values": {
            "result": "OK",
            "like": true
         },
         "correctness_weight": 1
      },
    ],
   "baseline_solutions": [
      {
         "output_values": {
            "result": "OK",
            "like": false
         },
         "confidence_weight": 0.8
      },
      {
         "output_values": {
            "result": "OK",
            "like": true
         },
         "confidence_weight": 1
      } 
   ], 
   "message_on_unknown_solution": "The cat is in a good mood",
   "overlap": 3,
   "infinite_overlap": false,
   "reserved_for": [], 
   "unavailable_for": [],
   "created": "2016-09-07T08:49:27.272"
}

Keys

Key

Value

Data type

Required during creation

id

Task ID.

String

Not used

origin_task_id
The ID of a task from another pool from which this task was copied for majority vote verification. String Not used
pool_id

The ID of the pool that the task is uploaded to.

String

Yes

input_values

Input data for a task. List of pairs:

"<ID of field 1>": "<value of field 1>",
"<ID of field 2>": "<value of field 2>",
...
"<ID of field N>": "<value of field N>"

JSON

Yes

known_solutions[]

Correct responses for control and training tasks.

You can specify several options for a correct task response.

If one option is more correct than another, you can assign different weights to the response options. To do this, use the correctness_weight key.

Array

No

known_solutions[].output_values

Output data values to check. You should specify values for all required output data fields.

"<ID of field 1>": "<correct response>",
"<ID of field 2>": "<correct response>",
...         

JSON

Yes

known_solutions[].correctness_weight

The weight of a correct response in the range from 0 to 1.

Lets you count a response as partially correct. This is convenient when there is no single right response to the task.

This works like awarding points: if you need to complete one control task correctly to get a skill (receive 1 point), you may complete one task with a weight of 1 or two tasks with a weight of 0.5 or higher.

The default value is 1.

Floating-point number

No

baseline_solutions[]

Preliminary responses. This data simulates performer responses when calculating confidence in a response. It's used in dynamic overlap (also known as incremental relabeling or IRL) and aggregation of results by skill.

Define output_values and confidence_weight for each preliminary response.

For example, you ask performers to label whether an image shows a cat or a dog. Suppose your neural network already determined that the image might show a dog with a probability of 80% and a cat with a probability of 40%. Let's say you set dynamic overlap from 1 to 3 and the minimum response confidence at 85%.

If the performer answers "Dog" and the confidence in their response is high, the overlap most likely won't increase because one response is enough. If the performer answers "Cat", the confidence is most likely not high enough and the overlap will increase further.

Can't be used when creating a task suite: an error with code 400 saying VALUE_NOT_ALLOWED will be returned to your request.

Array

No

baseline_solutions[].output_values

Output data values for preliminary responses.

"<ID of field 1>": "<preliminary response>",
"<ID of field 2>": "<preliminary response>",
...

JSON

Yes

baseline_solutions[].confidence_weight

Confidence in a response, from 0 to 1.

The default value is 1.

Floating-point number

Yes

message_on_unknown_solution

Hint for the task (for training tasks).

String

No

overlap

Task suite overlap.

Number

No, if the parameter is used when creating tasks
allow_defaults=true
and the overlap is specified in the pool parameters (key defaults.default_overlap_for_new_tasks)
infinite_overlap
Assigns tasks with infinite overlap. For instance, you can use this for training tasks when you want to assign them to all users:
  • true — Use infinite overlap.

  • false — Use the overlap that is set for the task or pool.

The default value is false.

Boolean

No

origin_task_id
ID of the task it was copied from String No
reserved_for[]

IDs of users who will have access to the task.

Array

No

unavailable_for[]

IDs of users who shouldn't have access to the task.

Array

No

created

The UTC date and time when the task suite was created, in ISO 8601 format: YYYY-MM-DDThh:mm:ss[.sss].

String

Not used

Key

Value

Data type

Required during creation

id

Task ID.

String

Not used

origin_task_id
The ID of a task from another pool from which this task was copied for majority vote verification. String Not used
pool_id

The ID of the pool that the task is uploaded to.

String

Yes

input_values

Input data for a task. List of pairs:

"<ID of field 1>": "<value of field 1>",
"<ID of field 2>": "<value of field 2>",
...
"<ID of field N>": "<value of field N>"

JSON

Yes

known_solutions[]

Correct responses for control and training tasks.

You can specify several options for a correct task response.

If one option is more correct than another, you can assign different weights to the response options. To do this, use the correctness_weight key.

Array

No

known_solutions[].output_values

Output data values to check. You should specify values for all required output data fields.

"<ID of field 1>": "<correct response>",
"<ID of field 2>": "<correct response>",
...         

JSON

Yes

known_solutions[].correctness_weight

The weight of a correct response in the range from 0 to 1.

Lets you count a response as partially correct. This is convenient when there is no single right response to the task.

This works like awarding points: if you need to complete one control task correctly to get a skill (receive 1 point), you may complete one task with a weight of 1 or two tasks with a weight of 0.5 or higher.

The default value is 1.

Floating-point number

No

baseline_solutions[]

Preliminary responses. This data simulates performer responses when calculating confidence in a response. It's used in dynamic overlap (also known as incremental relabeling or IRL) and aggregation of results by skill.

Define output_values and confidence_weight for each preliminary response.

For example, you ask performers to label whether an image shows a cat or a dog. Suppose your neural network already determined that the image might show a dog with a probability of 80% and a cat with a probability of 40%. Let's say you set dynamic overlap from 1 to 3 and the minimum response confidence at 85%.

If the performer answers "Dog" and the confidence in their response is high, the overlap most likely won't increase because one response is enough. If the performer answers "Cat", the confidence is most likely not high enough and the overlap will increase further.

Can't be used when creating a task suite: an error with code 400 saying VALUE_NOT_ALLOWED will be returned to your request.

Array

No

baseline_solutions[].output_values

Output data values for preliminary responses.

"<ID of field 1>": "<preliminary response>",
"<ID of field 2>": "<preliminary response>",
...

JSON

Yes

baseline_solutions[].confidence_weight

Confidence in a response, from 0 to 1.

The default value is 1.

Floating-point number

Yes

message_on_unknown_solution

Hint for the task (for training tasks).

String

No

overlap

Task suite overlap.

Number

No, if the parameter is used when creating tasks
allow_defaults=true
and the overlap is specified in the pool parameters (key defaults.default_overlap_for_new_tasks)
infinite_overlap
Assigns tasks with infinite overlap. For instance, you can use this for training tasks when you want to assign them to all users:
  • true — Use infinite overlap.

  • false — Use the overlap that is set for the task or pool.

The default value is false.

Boolean

No

origin_task_id
ID of the task it was copied from String No
reserved_for[]

IDs of users who will have access to the task.

Array

No

unavailable_for[]

IDs of users who shouldn't have access to the task.

Array

No

created

The UTC date and time when the task suite was created, in ISO 8601 format: YYYY-MM-DDThh:mm:ss[.sss].

String

Not used