Project 2. Select an object in the image
In this project performers will select image areas that contain a traffic sign. Use the results from project 1 as source images.
Create a project
Choose a template:
Click + Create project.
- Select the Object selection in an image template.
Provide general information:
- In Name to show performers, enter
Outline traffic signs in the image
. - In Description for performers, enter
Draw a box around all traffic signs in the image
. - Optionally add a Private comment.
- Click Save.
Edit the task interface:
Note. This tutorial shows how to create a task interface in the HTML/JS/CSS editor. You can also try creating a task interface in the Template Builder.In the Data specification section, you can configure the input and output fields.
What are input and output data?Input data is types of objects that are passed to the performer for completing the task. For example, this could be a text, an image, or geographic coordinates.
Output data is types of objects that you receive after the task is completed. For example, this could be one of several response options, typed text, or an uploaded file.
Learn more about input and output data fields.
In this project, you can leave the Data specification section unchanged.
In Task interface, leave the HTML block unchanged.
Edit the CSS block:
Set up the area selection tools. This template uses Editor for image area selection. It features rectangle and polygon tools (by default).
To set up rectangle selection, replace the code in the CSS block with the following:.image-annotation-editor__shape-polygon { display: none; }
Enter the code to adjust the interface height to the image size:
.image-annotation-editor__annotation-layer { height: max-content; }
(optional) You can ask performers to enter an annotation for the selected area or select one from a list. To do this, add an interface element in the JS block. For example, a text field or drop-down list.
Learn more about annotation.
Click
to see the performer's view of the task.
Tip.If the task isn't displayed in the preview mode, disable extensions and ad blockers in the browser.
In the window that opens, check if the task options work correctly:
Answer the question.
In the lower-right corner, click Submit.
Exit preview mode:
In the lower-left corner, click Exit.
In the list that opens, click Exit.
- Click Save.
Write instructions for performers:
- Instruction text:
- Click
and draw a rectangle around all traffic signs in the image.
Tip.If you want to add a task examples in the instruction, complete it yourself in the preview mode. Take screenshots, upload them to photo hosting or to your Yandex.Disk and insert image links in the instructions by clicking the
button on the toolbar.
Click Finish.
Open Toloka for requesters.
Click the + Create project button.
In the window that opens, find the Selecting a region in an image template and click Select.
In the window that opens, fill in the project parameters:
In the Project name field, enter
Outline the traffic signs in the image
.In the Description field, enter
Outline all traffic signs in the image with a rectangle
.In the Instructions field, enter the instruction.
- Instruction text:
- Click
and draw a rectangle around all traffic signs in the image.
Tip.If you want to add a task examples in the instruction, complete it yourself in the preview mode. Take screenshots, upload them to photo hosting or to your Yandex.Disk and insert image links in the instructions by clicking the
button on the toolbar.
- Note. This tutorial shows how to create a task interface in the HTML/JS/CSS editor. You can also try creating a task interface in the Template Builder.
In the Specifications section, you can configure the input and output fields.
What are input and output data?Input data is the data you need for your tasks. For instance, this could be links to images that you want to show to performers.
Output is the data you receive after the task is completed, like the performer's answers to your questions.
Learn more about input and output data fields.
In this project you can leave Specifications unchanged.
Prepare the Task interface: Leave the HTML block unchanged.
(optional) You can ask performers to enter an annotation for the selected area or select it from the list. To do this, add an interface element in the JS block. For example, a text field or drop-down list.
Learn more about annotation.
Edit the CSS block:
Set up the area selection tools. This template uses Editor for image area selection. It features rectangle and polygon tools (by default).
To set up rectangle selection, replace the code in the CSS block with the following:.image-annotation-editor__shape-polygon { display: none; }
Enter the code to adjust the interface height to the image size:
.image-annotation-editor__annotation-layer { height: max-content; }
Click the Preview button.
Tip.If the task isn't displayed in the preview mode, disable extensions and ad blockers in the browser.
In the window that opens, check if the task options work correctly:
Answer the question.
In the lower-right corner, click Submit.
Exit preview mode:
In the lower-left corner, click Exit.
In the list that opens, click Exit.
In the window that opens, click Finish editing.
Learn more about working with the project in Project.
Create a pool
A pool is a set of paid tasks sent out for completion at the same time.
Open the project page Outline the traffic signs in the image.
Click the Add a pool button.
Specify the Pool name.
(optional) Enter a private description.
Turn on the Add a private description option.
In the Private description field, enter the pool description. This information is available only to you.
In the Price per task suite section, find Price in US dollars and specify the price. For example,
0.01
. You can see more examples of pricing here.What is a task suite?A page can contain one or several tasks. If the tasks are simple, you can add 10-20 tasks per page. Don't make pages too long because it slows down loading speed for performers.
Performers get paid for completing the whole page.
The number of tasks on the page is set when uploading tasks.
What is the fair price for a task suite?The general rule of pricing is the more time the performer spends to complete the task, the higher the price is.
You can register in Toloka as a performer and find out how much other requesters pay for tasks, or see examples of cost for different types of tasks.
In the Performers section, filter performers:
Click Add filter.
Find the Audience presets block in the list and choose the Russian speaking users skill.
Add a new skill. Click Create skill.
In the window that opens, enter
Area selection
in the Title field.Click Add.
Quality control rules allow you to filter out careless performers. In the Quality control section, set the rules for the pool:
Turn on the Assignment review option.
What is non-automatic acceptance (assignment review)?The non-automatic acceptance option allows you to review completed task suites before accepting them and paying for them. If the performer didn't follow instructions, you can reject the task. The maximum allowed period for the review is set in the Deadline field.
In the Review period field, specify the number of days for checking the task.
- Add the following quality control blocks:
Recompletion of rejected assignments — sends the tasks you rejected to other performers according to the specified rules.
Click Add Quality Control Rule.
Find the Rules block in the list and choose Recompletion of rejected assignments.
Set a rule for a rejected task: if assignment becomes rejected, then extend overlap by 1. Turn on the Open pool if closed option.
This means that the rejected task will be returned to the pool and shown to another performer.
Submitted responses — allows you to limit the number of pool tasks available for the performer per day.
Click Add Quality Control Rule.
Find Rules block in the list and choose Submitted responses.
Set a rule for the completed task: if submitted assignments ≥ 1, then assign skill value Area selection equal to 1.
These parameters allow you to mark the performer who completed at least one task in the pool.
Tip.If the Area selection skill isn't displayed in the list, save the pool and reopen it for editing.
Fast responses — filters out performers who respond too quickly.
Click Add Quality Control Rule.
Find the Rules block in the list and choose Fast responses.
- In the Recent values to use field, enter the number of the performer's recently completed task suites. For example,
5
. - In the Minimum time per task suite field, enter the time in seconds. For example,
20
. Set a rule for fast responses: if the number of fast responses≥ 1, then ban on requester for 10 days. In the Reason field, enter Fast responses.
This means that a user who completes a task suite in less than 20 seconds will be blocked for 10 days and won't be able to complete your tasks.
Results of assignment review — restricts the pool access for performers who often make mistakes.
Click Add Quality Control Rule.
Find the Rules block in the list and choose Results of assignment review.
Set a rule for rejected tasks: if total reviewed responses ≥ 3 and rejected responses (%) > 35, then ban on requester for 15 days.
This means that if 35% or more of a performer's responses are rejected, the performer is banned and can't access your tasks for 15 days. The rule takes effect after 3 responses of the performer are reviewed.
Control tasks and majority vote aren't used for this project type, since the area markup provided by the performers must match (which is almost impossible). Learn more in Quality control.
Note.You can copy quality control settings from another pool. To do this, click Copy settings from in the Users filter section.
Set overlap, which is the number of performers to complete the same task. In the Overlap section, specify the Overlap field value. For image area selection tasks, it is usually
1
.(optional) In the Speed/quality ratio section, specify the percentage of top-rated users who will get access to the pool. Learn more in the Speed/quality balance section.
- In the Parameters section, specify the Time for completing a task suite in seconds filed value. The time should be long enough to read the instructions and wait for task data to load. For example,
1200
seconds. Click Save.
Prepare and upload a TSV file with tasks
Check out the example in a demo TSV file. You can find it on the pool page. At the top-left of the page, there are links to TSV files with regular, control, and training tasks.
Prepare a TSV file with tasks
In the text or spreadsheet editor, open the TSV file that you received after aggregating the results in project 1.
Choose images appropriate for the current project (the OK value).
Tip. You can use the awk commands to outline images on Linux and MacOS devices:awk 'BEGIN {OFS = FS = "\t";} $2=/OK/ {print $1}' <aggregated_res>.tsv > <filtered_res>.tsv
Copy the column with the selected links to a new sheet or document.
Name it
INPUT:image
. If you want to set a different name, rename the column in the source file with the results, too.Save the file in
TSV
format.
Upload the resulting TSV file.
Click Upload. In the window that opens, configure the file upload settings.
Choose Set manually.
In the Tasks per page field, specify
1
.Click Upload.
In the window that opens, choose the TSV file with tasks to upload and click Open.
In the window that opens, check the number of tasks and click Add.
Click
to start the pool.
Attention.The tasks will be completed by real performers in Toloka. Recheck your project setup before you start the pool.
Get the results
On the pool page, click Download results. In the window that opens:
In Status leave only the Submitted option enabled.
In Columns leave only the assignment ID option enabled.
Disable the Separate assignments with empty row option.
Click Download results.
Use the TSV file with results in the third project.
What's next
- Create Project 3 to review assignments.