Project 4. Which of the found items is most similar to the original?

In this project, performers compare images of shoes with the original image and decide which pair of images is most similar. To make a comparison in the project, add the results from the second project. To learn how, see Prepare and upload a TSV file with tasks.

  1. Create a project
  2. Create a pool
  3. Prepare and upload a TSV file with tasks
  4. Get the results

Create a project

To create a project, open Toloka for requesters.

  1. Choose a template:

    1. Click + Create project.

    2. Select the Side-by-side image comparison template.

  2. Provide general information:

    1. In the Project name field, enter Which of the found objects is similar to the original one?.

    2. In the Description field, enter Decide which pair of shoes is more similar to the original one..

    3. Optionally add a Private comment.

    4. Click Save.

  3. Edit Task interface and Specifications.

    Note. This tutorial shows how to create a task interface in the HTML/JS/CSS editor. You can also try creating a task interface in the Template Builder.
    1. Prepare the Task interface.

      1. Delete the template code from the html block.

      2. Insert the following code in the html block:

        <div class="header">
            <div class="left caption">
                {{button label="Go to site" href=uploaded_link_left size="L"}}
                <p class="url">{{uploaded_link_left}}</p>
            </div>
            <div class="right caption">
                <p class="url">{{uploaded_link_right}}</p>
                {{button label="Go to site" href=uploaded_link_right  size="L"}}
            </div>
        </div>
        
        {{img src=image}}
        
        <div class="content clearfix">
            <div class="left page">
                {{iframe src=uploaded_link_left width="100%" height="700px" real-size=true screenshot=true}}
            </div>
            <div class="right page">
                {{iframe src=uploaded_link_right width="100%" height="700px" real-size=true screenshot=true}}
            </div>
        </div>
        
        <div class="footer">
            {{field type="radio" name="result" label="Left image is better" value=result_left hotkey="1"}}
            {{field type="radio" name="result" label="Right image is better" value=result_right hotkey="2"}}
        </div>
      3. In the js block, edit the code by adding the following lines before OnRender.
        Attention.

        Don't remove the existing code.

        getTemplateData: function() {
                var data = TolokaHandlebarsTask.prototype.getTemplateData.apply(this, arguments),
                input = this.getTask().input_values;
                var left_link = input.left_link;
                var right_link = input.right_link;
                var uploaded_link_left = '',
                    uploaded_link_right = ''
                if (Math.floor(Math.random() * 2)) {
                    uploaded_link_left = left_link
                    uploaded_link_right = right_link
                } else {
                    uploaded_link_left = right_link
                    uploaded_link_right = left_link
                }
          data.uploaded_link_left = uploaded_link_left;
                data.uploaded_link_right = uploaded_link_right;
                data.result_left = uploaded_link_left;
                data.result_right = uploaded_link_right;
        
                return data;
        
            },
      4. Delete the template code from the css block.

      5. In the css block, insert the following code that is responsible for setting the proportional image size:
        .task {
          display: block;
          text-align:center;
        }
        
        .header {
          overflow: hidden;
          background-color: #FFCC00;
        }
        
        .caption {
          width: 50%;
        }
        
        .url {
          white-space: nowrap;
          overflow: hidden;
          text-overflow: ellipsis;
        
          max-width: calc(100% - 182px);
        
          display: inline-block;
          vertical-align: bottom;
        }
        
        .button {
          margin: 10px;
          max-width: 182px;
        }
        
        .button__label {
          white-space: nowrap;
          overflow: hidden;
          text-overflow: ellipsis;
          max-width: 150px;
        }
        
        .content {
          margin: 10px 0;
        }
        
        .page {
          display: inline-block;
          width: 50%;
        }
        
        .left {
          float: left;
          text-align: left;
        }
        
        .right {
          float: right;
          text-align: right;
        }
        
        .clearfix {
          overflow: hidden;
          width: 100%;
        }
    2. Edit the input and output data in the Specifications block.

      1. Click to switch graphic mode to JSON format.

      2. In the Input data field, enter:

        {
          "image": {
            "type": "url",
            "hidden": false,
            "required": true
          },
          "left_link": {
            "type": "url",
            "hidden": false,
            "required": true
          },
          "right_link": {
            "type": "url",
            "hidden": false,
            "required": true
          }
        }
      3. In the Output data field, enter:

        {
          "result": {
            "type": "url",
            "hidden": false,
            "required": true
          }
        }

      Learn more about the Specification parameters in Input and output data.

  4. Click the Preview button.

  5. In the window that opens, check if the task options work correctly.

    1. Answer the question.

    2. In the lower right corner, click Submit.

  6. Exit the preview mode.

    1. In the lower left corner, click Exit.

    2. In the window that opens, click Exit.

  7. In the Instructions field, enter the instructions.

    1. Instructions:

      Look at 2 pictures with different shoes and decide which pair of shoes looks most similar to the original pair. 
      Rely on your own sense of style, but also remember that the shoes look alike if 
      they are the same color, fabric, length, and style.
    2. Click Finish.

Learn more about working with the project in Project.

Create a pool

To create a pool:

  1. Open the page of the project titled Which shoes look more alike.

  2. Click the Add pool button.

  3. Specify the pool name.

  4. (optional) Enter the Private description.

    1. Turn on the Add a private description option.

    2. In the Private description field, enter the pool description. This information is available only to you.

  5. In the Price per task suite section, go to the Price in US dollars field and specify 0.01. You can see more examples of pricing here.

  6. In the Performers section, filter performers:

    1. Click Add filter.

    2. Find the Performer profile block in the list and choose the Languages skill.

    3. In the ? field specify =.

    4. Add another filter. Click Add filter.

    5. Find the Skills block in the list and select a skill named Choose skill.

    6. In the Choose skill field, select Found_shoes.

    7. In the ? field specify =.

    8. Leave the Absent field empty.

  7. In the Quality control section, specify quality control settings for the pool:

    1. Click Add quality control rule.

    2. Find the Rules block in the list and choose Fast responses.

    3. Set a rule for fast responses: if the number of fast responses is more than 3, then restrict the performer's access to the project for 10 days. Specify Fast responses as the reason.

    Learn more in Quality control.

  8. In the Overlap section, go to the Overlap field and specify 10.

  9. (optional) In the Speed/quality section, specify the percentage of top-rated users who will get access to the pool. Read more about this in Speed/quality balance.

  10. In the Parameters section:

    1. In the Time per task suite in seconds field, specify 300.

    2. Turn on the Keep task order option.

  11. Click Save.

Prepare and upload a TSV file with tasks

Check out the example in a demo TSV file. You can find it on the pool page. At the top-left of the page, there are links to TSV files with regular, control, and training tasks.

  1. Download the TSV file received in the third project.

  2. Create pairs for each input image so that you can compare the two found images with the original one and decide which one is more similar.

    You can create pairs manually using a text editor or automatically, like using Python or Jupyter Lab.

    As an example, you can use our results https://tlk.s3.yandex.net/wsdm2020/SbS_Toloka_prep&aggr_data.ipynb.

  3. Open the Which item is more similar? pool.

  4. Click Upload. In the window that opens, configure the file upload settings.

  5. Choose Set manually.

  6. In the Tasks per page field, specify1. You can experiment with the number of tasks per page.

  7. Click Upload.

  8. In the window that opens, choose the TSV file with tasks to upload and click Open.

  9. Click to start the pool.

    Attention.

    The tasks will be completed by real performers in Toloka. Recheck your project setup before you start the pool.

Get the results

To get the results:

  1. Click Download results.

    1. In the Status block, only leave the Accepted option enabled.

    2. In the Fields block, leave the link, response ID, and user ID options enabled.

  2. Disable the Separate assignments with empty row option.

  3. Try to use the Bradley-Terry model based on the results. As an example, you can use our https://tlk.s3.yandex.net/wsdm2020/SbS_Toloka_prep&aggr_data.ipynbresults.