Uploading tasks to a pool

How do I save the task order?

Keeping the task order while ignoring overlap

If you need the performers to receive task pages in the same order as they are in the uploaded TSV file, set it up with the Keep task order option. The Keep task order option works differently depending on the method for distributing tasks on pages. If the by empty row and set manually methods are used, performers will get task pages one after another: page 1 first, then pages 2 and 3, and so on. Tasks within pages will also go one after another and all performers will see the same sequence. For smart mixing, the algorithm generates pages so that performers get tasks in the order they are listed in the TSV file. Note that only task pages will be distributed in order, while the tasks within the pages will be mixed.

To use this option in your project, turn on the Keep task order option in the Parameters settings when creating a new pool.

Note. Keeping the order of tasks is useful if you need to quickly reach the overlap to monitor the majority vote or maintain the sequence of questions in a survey or training.

• By default, this option is disabled (set to No). In this case, both the task pages and the tasks inside the pages will be given to the performers in random order.

  For example, if you upload 20 tasks in the TSV file to the pool (in order from the 1st to the 20th) and set four tasks per page, the tasks will be distributed to the performers in the following way:

  Performers	Task page number	Order of tasks on the page:
  1	1	3, 2, 4, 1
  2	5	17, 20, 18, 19
  1	3	12, 9, 11, 10
  3	2	7, 8, 6, 5
  2	4	16, 13, 15, 14
  3	3	11, 12, 10, 9
  ...	...	...

  Performers	Task page number	Order of tasks on the page:
  1	1	3, 2, 4, 1
  2	5	17, 20, 18, 19
  1	3	12, 9, 11, 10
  3	2	7, 8, 6, 5
  2	4	16, 13, 15, 14
  3	3	11, 12, 10, 9
  ...	...	...

  Example

  
  
  

• If the option is enabled (set to Yes), the tasks are given to the performer page by page in the same order as they are in the TSV file. The tasks within the page are shuffled.

  For example, like in the previous case, tasks are loaded in the pool in order (from the 1st to the 20th), four tasks per page. But in this case, the performers will get pages in the same order as in the upload file, with tasks shuffled inside each page:

  Performers	Task page number	Order of tasks on the page:
  1	1	1, 4, 3, 2
  2	1	3, 4, 1, 2
  1	2	6, 5, 7, 8
  3	1	2, 1, 4, 3
  2	2	8, 5, 7, 6
  3	2	5, 8, 6, 7
  ...	...	...

  Performers	Task page number	Order of tasks on the page:
  1	1	1, 4, 3, 2
  2	1	3, 4, 1, 2
  1	2	6, 5, 7, 8
  3	1	2, 1, 4, 3
  2	2	8, 5, 7, 6
  3	2	5, 8, 6, 7
  ...	...	...

Note. In the pool preview, pages and tasks are shuffled because the task order isn't preserved in the preview. However, when you start the pool, task pages will be issued to each performer in the specified order.

Task order accounting for overlap

If you set an overlap to more than one and turn on the Keep task order option, each subsequent page is distributed to interested users only after there are enough users who submitted the page that was already assigned (in other words, after it reaches full overlap).

In this case, if the user already completed one pool page or there is a new interested user, they will get the next page that isn't in progress yet, even if the previous one didn't reach full overlap.

If a user refuses the issued task page, it will be given to another user — either someone else who is interested in the pool, or an available user who accepts the task.

For example, if overlap is set to 3:

Performers	Task page number	The overlap value achieved	Note
1	1	1	Interested users received page 1
2	1	2
1	2	1	A performer completed page 1 and got page 2, although page 1 didn't reach full overlap yet
3	1	3	Full overlap of page 1
3	2	1	The user who took the task refused to complete page 2
4	2	2	The interested user received page 2 straight away, since there is already a full overlap for page 1, and the user who took it refused to perform page 2
1	3	1	A performer completed page 2 and got page 2, although page 2 didn't reach full overlap yet
2	2	3	Full overlap of page 2
5	3	1	Interested user refused to complete page 3
2	3	2	The user who submitted to the pool before received page 3, since the interested user refused to complete it
3	3	3	Full overlap of page 3
...	...	...	...

Performers	Task page number	The overlap value achieved	Note
1	1	1	Interested users received page 1
2	1	2
1	2	1	A performer completed page 1 and got page 2, although page 1 didn't reach full overlap yet
3	1	3	Full overlap of page 1
3	2	1	The user who took the task refused to complete page 2
4	2	2	The interested user received page 2 straight away, since there is already a full overlap for page 1, and the user who took it refused to perform page 2
1	3	1	A performer completed page 2 and got page 2, although page 2 didn't reach full overlap yet
2	2	3	Full overlap of page 2
5	3	1	Interested user refused to complete page 3
2	3	2	The user who submitted to the pool before received page 3, since the interested user refused to complete it
3	3	3	Full overlap of page 3
...	...	...	...

You can also set the order of tasks in the Yandex.Toloka API. To do this, use the function shuffle_tasks_in_task_suite: If true, the task order within a page is random. If false, the order in which tasks were uploaded is kept. The default is true, meaning that tasks are shuffled within the page.

Skill

If you added the majority vote quality control rule, once all completed pages have reached full overlap, a performer will be assigned a skill by majority vote. For example, if overlap 3 is set in the pool settings, the skill is calculated after each of these pages reaches overlap 3, not after the performer completes 3 pages.