Project 3. Are the bounding boxes correct?
In this project, performers will determine if traffic signs were selected correctly on the images in project 1, and submit the responses in project 2.
- Create a project
- Create a pool
- Prepare and upload a TSV file with the results
- Download the reviewed results
- Check the completed tasks
Create a project
Click the + Create project button.
In the window that opens, find the Selecting a region in an image template and click Select.
In the window that opens, fill in the project parameters:
In the Project name field, enter
Are the traffic signs outlined correctly?
.In the Description field, enter
Look at the image and decide whether the traffic signs are outlined correctly or not.
In the Instructions field, enter the instruction.
Instruction text:
Look at the image and answer the question: Are all traffic signs outlined correctly? If there are, click Yes. If there isn's, click No. For example, road signs are outlined correctly, so the correct answer is Yes.
Tip. If you want to add a task examples in the instruction, complete it yourself in the preview mode. Take screenshots, upload them to photo hosting or to your Yandex.Disk and insert image links in the instructions by clicking thebutton on the toolbar.
- Note. This tutorial shows how to create a task interface in Yandex.Toloka. You can also try creating a task interface in the Template Builder.
Set up Specifications:
Click
to switch graphic mode to JSON format.
Remove the template code from the Input data and Output data fields.
In the Input data field, enter the following code:
{ "image": { "type": "url", "hidden": false, "required": true }, "selection": { "type": "json", "hidden": false, "required": false }, "assignment_id": { "type": "string", "hidden": true, "required": true } }
In the Output data field, enter the following code:
{ "result": { "type": "string", "hidden": false, "required": true } }
Learn more about Specifications in Input and output data.
Prepare Task interface:
Connect the $TOLOKA_ASSETS/js/image-annotation.js library (click
in the Task interface block on the project page).
In the html block, replace the current code with the following:
<!-- editor for selecting areas that lets you add an area in advance --> {{field type="image-annotation" name="object" src=image annotations=selection}} <!-- buttons for responses --> {{field type="radio" name="result" value="OK" label="Correct" hotkey="1"}} {{field type="radio" name="result" value="BAD" label="Incorrect" hotkey="2"}}
In the css block, replace the current code with the following:
/* hide the button for polygon selection */ .image-annotation-editor__shape-polygon { display: none; } /* adjust the interface height */ .image-annotation-editor__annotation-layer { height: max-content; }
Click the Preview button. You should see the task interface.
To test the interface:- Click Change input data.
- In the selection field, add an example of input data:
[{"data":{"p1":{"x":0.472,"y":0.413},"p2":{"x":0.932,"y":0.877}},"type":"rectangle"}, {"data":[{"x":0.143,"y":0.807},{"x":0.317,"y":0.87},{"x":0.511,"y":0.145},{"x":0.328,"y":0.096},{"x":0.096,"y":0.554}],"type":"polygon"}]
Answer the question.
In the lower-right corner, click Submit.
If everything is OK, close the preview tab. If not, check the data you inserted in the code blocks.
On the project settings tab, click Finish editing.
If you still have questions about setting up projects, you can search for the answer in the common guidelines for Creating a project.
Create a pool
Open the project page titled Are the traffic signs outlined correctly?.
Click the Add a pool button.
Specify the Pool name.
(optional) Enter a private description.
Turn on the Add a private description option.
In the Private description field, enter the pool description. This information is available only to you.
In the Price per task page section, the Price in US dollars field, specify the price. For example,
0.01
. You can see more examples of pricing here.In the Performers section, filter performers:
Click Add filter.
Find the Audience presets block in the list and choose the Russian speaking users skill.
Add another filter. Click Add filter.
Find Skills in the list and choose Choose skill.
In the Choose a skill field, choose Object selection.
In the ? field specify
=
.Leave the Missing field blank.
In the Quality control section, specify quality control settings for the pool:
Click Add Quality Control Rule.
Find the Rules block in the list and choose Majority vote.
In the Accept as majority field, soecify
2
.Set the rule: if number of responses ≥ 10 and correct responses (%) < 50, then ban the performer on project for 10 days. Specify the reason Doesn't match the majority.
Note.The rule takes effect when the number of responses for the task is equal to the overlap.
Learn more in Quality control.
In the Overlap section, specify
3
in the Overlap field.(optional) In the Speed/quality ratio section, specify the percentage of top-rated users who will get access to the pool. Read more about this in Speed/quality balance.
In the Parameters section:
In the Time for completing a task page in seconds field, specify
600
.Turn on the Keep task order option.
Click Save.
Prepare and upload a TSV file with the results
Check out the example in a demo TSV file. You can find it on the pool page. At the top-left of the page, there are links to TSV files with regular, control, and training tasks.
Prepare a TSV file with tasks.
Leave the
INPUT:image
column unchanged.Change the
OUTPUT:result
column name toINPUT:selection
.Rename the
ASSIGNMENT:assignment_id
column toINPUT:assignment_id
.Delete the
ACCEPT:verdict:
andACCEPT:comment:
columns:Save the file in
TSV
format.
Tip.You can prepare TSV file in the Notepad program. To transfer data to Microsoft Excel, use Text Import Wizard and disable the Text qualifier option.
Upload the resulting TSV file to the Are the traffic signs outlined correctly? pool:
Open the Are the traffic signs outlined correctly? pool.
Click Upload. In the window that opens, configure the file upload settings.
Choose Set manually.
In the Tasks per page field, specify
10
.Click Upload.
In the window that opens, choose the TSV file with tasks to upload and click Open.
In the window that opens, check the number of tasks and click Add.
- On the pool page, click Preview. Make sure that the task displays images with the selected objects.Tip.
If there are no selected objects, check if the TSV file has quotation marks inserted correctly in the INPUT:selection column. Make sure that the interface parameters are set correctly.
Click
to start the pool.
Attention.The tasks will be completed by real performers in Yandex.Toloka. Recheck your project setup before you start the pool.
Download the reviewed results
Next to the Download results button, click
.
Choose Dawid-Skene aggregation model from the list. Read more about Result aggregation based on the Dawid-Skene model.
In the window that opens, click Yes.
At the top of the page, click View the list of operations.
Note.To track the progress, refresh the page from time to time. Aggregation takes from 5 to 20 minutes. You can start designing another project meanwhile.
When the operation is complete, download the TSV file with the results. To do this, click Download in the Files column.
Use the TSV file with the results in project 2.
Check the completed tasks
Since the Non-automatic acceptance option is enabled in the pool settings for project 2, you should check the performers' responses within the time limit set in the Review period field.
You can check the results in two ways:
- In the TSV file with the results.
- In the pool interface.
In the text or spreadsheet editor, open the TSV file that you received after aggregating the results.
Tip. You can use the online spreadsheet editor. Go to Yandex. Disk linked to your customer account. Click . Copy the aggregated data from the TSV file to the table.Prepare the TSV file:
Add a column named
ACCEPT:verdict
with the review results.Add a column named
ACCEPT:comment
with comments for performers if responses were rejected. For example, comment on which part of the instructions wasn't followed.Rename the
INPUT:assignment_id
column toASSIGNMENT:assignment_id
.
Fill in the
ACCEPT:verdict:
andACCEPT:comment:
columns:- If the aggregate result of the assignment is OK, put
+
to accept it. - If the aggregate result of the assignment is incorrect or it doesn't open, put
-
to reject it. Enter the reason for rejecting the task in theACCEPT:comment:
field, for example,The object isn't selected or is selected incorrectly.
Tip. You can use the awk commands to outline images on Linux and MacOS devices:awk 'BEGIN {FS=OFS="\t";} NR>1 {if($4~"OK"){ print $1, "+", ""; }else{ print $1, "-", "The object isn't selected or is selected incorrectly.";}}' <post_accept_res>.tsv > <review_res>.tsv
Delete all the other columns.
Save the file in
TSV
format.Open the pool page in project 2.
Click Review assignments.
Click Upload results.
In the window that opens, choose the TSV file with the results to upload and click Open.
In the window that opens, compare the number of tasks in the Processed successfully and Total submitted fields on the pool page.
Click Add.
In the window that opens, click Close.
When setting up a pool in the second project you turned on the Recompletion of the rejected tasks option.
In this case, the pool automatically reopens and the assignments are reassigned to other performers. When they're completed, send the results for review. Then download the results, check them, and upload the reviewed results. You can reject assignments as many times as you want to get more accurate results.
Open the pool page in project 2.
Click View assignments.
Hover over the line of the assignment you want to check.
In the Status column, you will see the buttons for accepting (
) or rejecting (
) the completed assignment. If you reject the assignment, enter a comment in the window that opens and click Done.
What's next
- Learn more about decomposition.