Side-by-side image comparison

  1. Before you start
  2. Creating a project
  3. Adding a task pool
  4. Uploading tasks
  5. Starting a pool and getting results
Tip.

Run the project in the Sandbox first. This helps you avoid making mistakes and spending money on a task that isn't working right.

This project template it is suitable for those cases when you need to compare similar or different images, illustrations, and interfaces in pairs (side-by-side, SbS).

Use this template when you need to:

  • You need to understand which interface design users like best.
  • Testing different images in targeted ads.
  • Choose the best images for publications, illustrations, or desktop backgrounds.

Let's say you have several photos with cats and you want to figure out which one will be nicer in the ad for the user. To do this, create a task: the performer will see two options and will have to choose one of them.

Example of a prepared task

Before you start

To use your images in the project, you will need to upload them to a storage from which you can get file links. For example, your server, photo hosting, or cloud storage:

If you have a complex project, register in the sandbox and create a project there. There you can:

  1. Test the project settings as a performer.
  2. Then transfer them to the production version.
  3. Launch the project for real performers.

This helps you avoid making mistakes and spending unnecessary money on a task that doesn't work.

Creating a project

In the project, you define what the task will look like for the performer.

  1. Choose a template:

    1. Click + Create project.

    2. Select the Side-by-side image comparison template.
  2. Provide general information:

    1. Enter a clear name and a short description for the project. Performers will see this in the task list.

    2. Optionally add a Private comment.

    3. Click Save.

  3. Edit the task interface:

    Note. This tutorial shows how to create a task interface in the HTML/JS/CSS editor. You can also try creating a task interface in Template Builder.
    1. Define which objects you are going to pass to the performers and receive from them in response. To do this, add input and output fields in the Data specification section.

      What are input and output data?

      Input data is types of objects that are passed to the performer for completing the task. For example, this could be a text, an image, or geographic coordinates.

      Output data is types of objects that you receive after the task is completed. For example, this could be one of several response options, typed text, or an uploaded file.

      Learn more about input and output data fields.

      In this case they are:

      • The image input data field for a link to an image.
      • Output data fields:
        • Boolean like for a checkbox answer.
        • The result string with a radio button response.

      Create the task interface in the HTML block. It describes how the task elements should be arranged in the task.

      You can use standard HTML tags and special expressions in double curly brackets for input and output data fields.

      
      {{img src=image_left width="100%" real-size=true}}
      {{img src=image_right width="100%" real-size=true}}
      
      {{field type="radio" name="result" label="Left" value="LFT" hotkey="1"}}
      {{field type="radio" name="result" label="Right" value="RGHT" hotkey="2"}}
      {{field type="radio" name="result" label="They are the same" value="404" hotkey="3"}}
      
      Copied to clipboard

      This notation describes the following task design:

      • Two images by the image_left and image_right link.
      • Three radio buttons, and the chosen option is output to the result field.
    2. Most people view images from left to right, so the first image can affect the perception of the second. If you shuffle images in tasks, the reliability of the survey results increases. Images in pairs will be displayed each time in a new order for different performers.

      How do I make a task if I need to mix images?

      If you need to compare:

      Only two images.
      • You can shuffle the images manually when you create TSV file with links. Make two tasks with different image order. In the TSV file (using Excel, for example), add another line with two links that will be placed differently from the first line. One line is one page with one task for the performer. This will be the case after you apply the settings described below.

      • In the next step Adding a task pool, you need to configure quality control rule Submitted responses so that one performer gets only one task.

        Add a block and set the following parameters:

        This means that after the performer completes one task, they aren't offered the next one. The response will be accepted and the task completed.

      Many images.

      You can display them in random order in the interface. You need to edit the specification as well as the HTML and JS blocks. We recommend creating a test project following the instruction in the Which of the found items is more similar to the initial one? section.

    3. Edit Common interface elements.
    4. Click to see the performer's view of the task.

      Note. The project preview shows one task with standard data. You can define the number of tasks to show on the page later.
    5. Save the changes.

  4. Write instructions for performers:

    1. Write short and clear guidelines (see the recommendations). Describe what needs to be done and give examples in them.

      You can prepare instructions in HTML format, then copy and paste into the editor. Click <> to switch to HTML mode.

    2. Click Finish.

Adding a task pool

A pool is a set of paid tasks sent out for completion at the same time.

  1. Open the project and click Add pool.
  2. Give the pool any convenient name and description. The pool info is only available to you. Performers can view only the project name and description.
  3. Set the price per task suite (for instance, $0.01).
    What is a task suite?

    A task suite can contain one or several tasks that are shown on the same page. If the tasks are simple, you can add 10-20 tasks per suite. Don't make task suites too long because it slows down loading speed for performers.

    Performers get paid for completing the entire task suite.

    The number of tasks per suite is set when uploading tasks.

    What is a fair price for a task suite?

    The general rule of pricing is the more time the performer spends to complete the task, the higher the price is.

    You can register in Toloka as a performer and find out how much other requesters pay for tasks, or see examples of cost for different types of tasks.

  4. Add Filters to select performers. To make your task available only to English-speaking users, set filters by language and country detected by the phone number.
  5. Set up Quality control. Quality control rules allow you to filter out inattentive performers. You can also set up quality control in the project.

    • Typical quality control setting for such tasks is Fast responses. Add a block and specify the following values:

      This means that a performer who completes a task suite in less than 5 seconds will be suspended for 10 days and won't be able to access your tasks.

  6. Set Overlap, which is the number of performers to complete the same task. For image comparison tasks, 20.
  7. In the Speed/quality ratio block, you can leave the settings unchanged. For the image comparison, the performer's rating isn't too important.
  8. Set the Time allowed for completing a task suite. It should be long enough to read the guidelines and wait for task data to download (for example, 600 seconds).
  9. Save the pool.

Uploading tasks

Prepare your own task file. Check out the example in the demo TSV file . In the Toloka interface under the name of the pool there are links to TSV file templates for regular, control, and training tasks.

  1. Click Upload. In the window that opens, you can also download a sample TSV file by clicking Sample file for uploading tasks.
    What is TSV?
    A TSV file presents a table as a text file in which columns are separated by tabs.
    You can work with it in a table editor or in a text editor, and then save it to the desired format. Learn more about working with a TSV file. There is a CSV format that is similar to TSV, but you should use a TSV file for uploading.
    Note. The file must be saved in UTF-8 encoding.
  2. Add input data in it. The input data column headings that contain the word INPUT. Leave the other columns empty.
  3. Load the tasks by choosing Specify manually. Set the value to 1.
    What does "Set manually" mean?
    Set manually is a setting for forming task suites. It means that a page will contain the number of tasks that you specify in this setting.

Starting a pool and getting results

  1. Start the pool by clicking .
  2. Track the completion of tasks in the Pool statistics section.
  3. When the pool is completed, launch aggregation of results. To do this, find the Download results button and click  → Dawid-Skene aggregation model next to it.

    In this case, the TSV file with aggregated responses contains the response significance as a percentage — the CONFIDENCE field. The results show how much we can trust the response that the right (or left) picture is better. Learn more about aggregation.

  4. Track the aggregation progress on the Operations page. When the process is completed, click Download.

Troubleshooting

Uploading files from Yandex.Disk
I can't upload files from Yandex.Disk

If images, audio or video files from Yandex.Disk don't appear in the instructions or on the task suite, make sure you connected Yandex.Disk correctly and uploaded the files.

How to create a task where the performer has to view a video from Yandex.Disk

To create such a task, take the video markup template as a basis.

To host your videos on Yandex.Disk, connect Yandex.Disk and set up the project.

Why can't my task for selecting objects in an image display images from Yandex.Disk?
The problem is in your task template. Make sure that:
  • In the project, the input field where you pass the file link has the “string” type.
  • The component in the task template uses the "proxy" expression.
  • The format of relative links in the TSV file with tasks is correct: <unique name>/<file path and name>.
For detailed instructions and videos, see the page Using files from Yandex.Disk.
Frequent mistakes when connecting to Yandex.Disk and uploading files
  • The Input data field in the project settings has the link type. You should choose the string type.
  • The TSV file contains absolute references to the task files. You need to insert a link <unique name>/<path and file name>. For example: yadisk/image1.jpg or yadisk/photos/image1.png.
  • Photos from Yandex.Disk are used in the task instructions in the mobile app. To display the photos in the instructions, use only direct links.
  • Files are deleted or aren't located in the Yandex.Disk folder that the link leads to.
  • The OAuth token isn't active. Update the token on the External Services Integration page.
To display files from Yandex.Disk (images, audio files, videos) to the performer:
  1. Link Yandex.Disk in your profile.
  2. Set the string type for the input data field.
  3. Insert a file link using the proxy component.

Detailed instructions

Files load too slowly from Yandex.Disk. How do I speed up the loading process?

Try the recommendations on this page or contact Yandex.Disk support.

Frequent mistakes when connecting to Yandex.Disk and uploading files
  • The Input data field in the project settings has the link type. You should choose the string type.
  • The TSV file contains absolute references to the task files. You need to insert a link <unique name>/<path and file name>. For example: yadisk/image1.jpg or yadisk/photos/image1.png.
  • Photos from Yandex.Disk are used in the task instructions in the mobile app. To display the photos in the instructions, use only direct links.
  • Files are deleted or aren't located in the Yandex.Disk folder that the link leads to.
  • The OAuth token isn't active. Update the token on the External Services Integration page.
To display files from Yandex.Disk (images, audio files, videos) to the performer:
  1. Link Yandex.Disk in your profile.
  2. Set the string type for the input data field.
  3. Insert a file link using the proxy component.

Detailed instructions

Why doesn't the task preview show my images from Yandex.Disk?
The problem is in your task template. Make sure that:
  • In the project, the input field where you pass the file link has the “string” type.
  • The component in the task template uses the "proxy" expression.
  • The format of relative links in the TSV file with tasks is correct: <unique name>/<file path and name>.

Detailed instructions.

How do I embed multiple images using links to Yandex.Disk?
To add images using links to Yandex.Disk:
  1. Use a link, for example: /api/proxy/yadisk/image1.jpg .
  2. In the requester profile settings, go to External Services Integration → Proxy settings.
  3. Set up integration with external services.

    Learn more about using files from Yandex.Disk.

How do I create a task with a selection out of three image options with the paired image comparison?
  1. Use the “Side-by-side image comparison” template.
  2. In the TSV file, specify the links to the compared images.
  3. In the file, create the tasks where all the images will be compared in pairs:
    • Image 1 and Image 2

    • Image 1 and Image 3

    • Image 2 and Image 3
  4. Process the results.

You can also edit the project so that the user sees 3 images at once and selects one of them.

How do I make an image expand to its maximum size on click?

To the component that inserts the image, add the parameters: real-size=true and screenshot=true.

Do I need to convert all the images in the task to the same size or can they be different?
You can use different image sizes.