Project 3. Does the item found look similar to the original?
In this project, performers will compare the shoes from the original image (in project 1) with the shoes found in project 2.
- Create a project
- Create a pool
- Prepare and upload a TSV file with the results
- Download the reviewed results
- Check the completed tasks
Create a project
To create a project, open Toloka for requesters.
Click + Create project.
Choose a template:
Choose the Image classification template.
Provide general information:
In the Project title field, enter
Are these shoes similar to each other?
.In the Description field, enter
Take a look at two pairs of shoes and decide whether they look similar or not.
.Optionally add a Private comment.
Click Save.
Prepare the Task interface and Specifications:
Note. This tutorial shows how to create a task interface in the HTML/JS/CSS editor. You can also try creating a task interface in the Template Builder.Edit the Task interface: delete the template code from html and paste the following code into the html block:
{{img src=image height="400px"}} {{iframe src= found_link height="600px"}} <p>Make sure that the uploaded image matches the product from the store.</p> {{button label="Check the product" href=found_link action=true}} <p>This <b>shoes</b> similar?</p> <p>Shoes must be the same color, fabric, length, and style.</p> {{field type="radio" name="result" value="Yes "label="Yes"}} {{field type="radio" name="result" value="No" label="No"}}
Leave the JS block unchanged. In the css block, insert the following code that is responsible for setting the proportional image size:
.task { display: block; } .img { float: left; width: 50%; } .iframe { float: left; width: 50%; }
Edit input and output data in the Specifications block: click
to switch the graphic mode to JSON format and in the Input data field, enter:
{ "image": { "type": "url", "hidden": false, "required": true }, "found_link": { "type": "url", "hidden": false, "required": true }, "assignment_id": { "type": "string", "hidden": true, "required": true } }
In the Output data field, enter:
{ "result": { "type": "string", "hidden": false, "required": true } }
Learn more about the Specification parameters in Input and output data.
Click the Preview button.
In the window that opens, check if the task options work correctly: answer the question and click Submit in the lower-right corner.
Answer the question.
In the lower right corner, click Submit.
Exit the preview mode.
In the lower left corner, click Exit.
In the window that opens, click Exit.
In the Instructions field, enter the instructions.
Instructions:
Take a look at the pictures that show two pairs of shoes. Decide whether they look similar or not. Shoes are considered similar if they have a similar color, fabric, length, and style. If you don't see a pair of shoes in any of the pictures, click Yes.
Click Finish.
Learn more about working with the project in Project.
Click + Create project.
In the window that opens, select the Image categorization template. To do this, hover over the template and click Choose.
In the window that opens, fill in the project parameters.
In the Project title field, enter
Are these shoes similar to each other?
.In the Description field, enter
Take a look at two pairs of shoes and decide whether they look similar or not.
.In the Instructions field, enter the instructions.
Instructions:
Take a look at the pictures that show two pairs of shoes. Decide whether they look similar or not. Shoes are considered similar if they have a similar color, fabric, length, and style. If you don't see a pair of shoes in any of the pictures, click Yes.
- Note. This tutorial shows how to create a task interface in the HTML/JS/CSS editor. You can also try creating a task interface in the Template Builder.
Edit the input and output data in the Specifications block.
Click
to switch graphic mode to JSON format.
In the Input data field, enter:
{ "image": { "type": "url", "hidden": false, "required": true }, "found_link": { "type": "url", "hidden": false, "required": true }, "assignment_id": { "type": "string", "hidden": true, "required": true } }
In the Output data field, enter:
{ "result": { "type": "string", "hidden": false, "required": true } }
Learn more about the Specification parameters in Input and output data.
Prepare the Task interface.
Delete the template code from the html block.
Insert the following code in the html block:
{{img src=image height="400px"}} {{iframe src= found_link height="600px"}} <p>Make sure that the uploaded image matches the product from the store.</p> {{button label="Check the product" href=found_link action=true}} <p>This <b>shoes</b> similar?</p> <p>Shoes must be the same color, fabric, length, and style.</p> {{field type="radio" name="result" value="Yes" label="Yes"}} {{field type="radio" name="result" value="No" label="No"}}
Leave the JS block unchanged.
- In the css block, insert the following code that is responsible for setting the proportional image size:
.task { display: block; } .img { float: left; width: 50%; } .iframe { float: left; width: 50%; }
Click the Preview button.
In the window that opens, check if the task options work correctly.
Answer the question.
In the lower right corner, click Submit.
Exit the preview mode.
In the lower left corner, click Exit.
In the window that opens, click Exit.
In the window that opens, click Finish editing.
Learn more about working with the project in Project.
Create a pool
To create a pool:
Open the page of the project titled Are these shoes similar to each other?.
Click the Add pool button.
Specify the pool name.
(optional) Enter the Private description.
Turn on the Add a private description option.
In the Private description field, enter the pool description. This information is available only to you.
In the Price per task suite section, go to the Price in US dollars field and specify
0.01
. You can see more examples of pricing here.In the Performers section, filter performers:
Click Add filter.
Find the Performer profile block in the list and choose the Languages skill.
In the ? field specify
=
.Add another filter. Click Add filter.
Find the Skills block in the list and select a skill named Choose skill.
In the Choose skill field, select Found_shoes.
In the ? field specify
=
.Leave the Absent field empty.
In the Quality control section, specify quality control settings for the pool:
Click Add quality control rule.
Find the Rules block in the list and choose Control tasks.
Set a rule for control tasks: If the number of responses to control questions is greater than or equal to 3 and the percentage of correct responses to control questions is less than 60, then restrict the performer's access to the project for 10 days. Specify the Control task as a reason.
(optional) Add another quality control rule. Click Add quality control rule.
Find the Rules block in the list and choose Fast responses.
In the Minimum time per task suite field, specify
60
.Set a rule for fast responses: if the number of fast responses is more than 1, then restrict the performer's access to the project for 10 days. Specify Fast responses as the reason.
Learn more in Quality control.
In the Overlap section, go to the Overlap field and enter
3
.(optional) In the Speed/quality section, specify the percentage of top-rated users who will get access to the pool. Read more about this in Speed/quality balance.
In the Parameters section:
In the Time per task suite in seconds field, specify
300
.Turn on the Keep task order option.
Click Save.
Prepare and upload a TSV file with the results
Check out the example in a demo TSV file. You can find it on the pool page. At the top-left of the page, there are links to TSV files with regular, control, and training tasks.
Wait for the pool from project 2 to complete.
Open the pool page from project 2.
Click Download results.
In the Status block, only leave the Not checked option enabled.
In the Fields block, only leave the response ID option enabled.
Disable the Separate assignments with empty row option.
Click Download results.
Prepare a TSV file with tasks.
Leave the
INPUT:image
column unchanged.Rename the
OUTPUT:found_link
column toINPUT:found_link
.Rename the
ASSIGNMENT:assignment_id
column toINPUT:assignment_id
.Delete all the other columns.
Save the file in
TSV
format.
Upload the resulting TSV file to the Are these shoes similar to each other? pool.
Open the Are these shoes similar to each other? pool.
Click Upload. In the window that opens, configure the file upload settings.
Choose Smart mixing.
In the Main tasks field, specify
9
.In the Training tasks field, specify
0
.In the Control tasks field, specify
1
.Click Upload.
In the window that opens, choose the TSV file with tasks to upload and click Open.
In the window that opens, check the number of tasks and click Add.
Create a control task.
Click Edit.
Note. If you selected something else instead of smart mixing, click Edit. If this button is missing, delete the file and upload it again.In the window that opens, click Create control tasks.
In the window that opens, in the column on the left, turn on the result option.
Choose the correct answer to the question.
Click Save and go to next.
Click Are these shoes similar to each other? to exit task markup mode.
Note.In small pools, control tasks should be 10-20% of all tasks. Include different versions of correct responses in equal amounts. See the distribution of responses on the Edit tasks page, Control tasks tab.
Download the reviewed results
To get the results:
Next to the Download results button, click
.
Choose the Dawid-Skene aggregation model. Read more about Result aggregation based on the Dawid-Skene model.
In the window that opens, click Yes.
At the top of the page, click View operations list.
Note.To track the progress, refresh the page from time to time. Aggregation takes from 5 to 20 minutes. You can start designing another project meanwhile.
When the operation is complete, download the TSV file with the results. To do this, click Download in the Files column.
Use the TSV file with the results from project 2.
Check the completed tasks
You can check the results in two ways:
- In the TSV file with the results.
- In the pool interface.
- Review assignments in the TSV file with the results
-
To review the submitted assignments:
- Review assignments in the pool interface
-
To check submitted assignments:
What's next
- Create Project 4 to compare matching images.