Rate video (classification)

Tip.

Run the project in the Sandbox first. This helps you avoid making mistakes and spending money on a task that isn't working right.

Projects of this type are intended for video classification. You can use them for:
  • Content moderation.
  • Sorting video clips by specified categories.
  • Rating video clips based on how much the performers like them.
  • Detecting noise and other visual imperfections in the video.

For example, you have a lot of video clips, and you need to determine which ones are of higher quality. To do this, create a task in which the performers will watch the video clips and evaluate their quality.

  1. Before you start
  2. Create a project
  3. Create a pool
  4. Upload a TSV file with tasks.
  5. Get the results

Before you start

  • To use your video clips in the project, you need to upload them to a storage from which you can get file links. For example, your server, video hosting, or cloud storage:

    • Instructions on how to use files from Yandex.Cloud.
    • Instructions on how to use files from Yandex.Disk.
    • If you use video clips from YouTube, replace the https://www.youtube.com/watch?v=example links with https://www.youtube.com/embed/example.
  • Make sure that the video clips look the same in the desktop and mobile versions of Yandex.Toloka. A video clip may not play or play incorrectly in the mobile version. In this case, when creating a pool, restrict access to the pool for the performers with mobile devices.

    Learn more about configuring tasks for mobile devices.

    Note.

    The video player supports only MP4 format with H.264 or H.265 encoding.

  • If you have a complex project, register in the sandbox and create a project there. There you can:

    1. Test the project settings as a performer.
    2. Then transfer them to the production version.
    3. Launch the project for real performers.

    This helps you avoid making mistakes and spending unnecessary money on a task that doesn't work.

Create a project

  1. Click the + Create project button.

  2. In the window that opens, find the Video moderation template and click Select.

  3. In the window that opens, fill in the project parameters:

    1. Fill in the Project name and Description fields. The name and description should be short and clear. Performers will see this in the task list.

    2. In the Instructions field, briefly describe what the performer should do.

  4. Note. This tutorial shows how to create a task interface in Yandex.Toloka. You can also try creating a task interface in the Template Builder.

    In the Specifications section, you can configure the input and output fields.

    What are input and output data?

    Input data is the data you need for your tasks. For example, links to images that you want to show to performers.

    Output is the data you receive after the task is completed. For example, the performer's answers to your questions.

    Learn more about input and output data fields.

    In this project you can leave Specifications unchanged.

    If you use a video from YouTube
    For the video input field, specify the string data type.
  5. In the Task interface section, you can customize the appearance of tasks for performers.

    Learn more about task interface.

    In this project you can leave Task interface unchanged.

    If you use a video from YouTube
    Add the iFrame element to the interface:
    <iframe src={{video}} width="560" height="315" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
  6. Click the Preview button.

    Tip.

    If the task isn't displayed in the preview mode, disable extensions and ad blockers in the browser.

  7. In the window that opens, check if the task options work correctly:

    1. Answer the question.

    2. In the lower-right corner, click Submit.

    3. Exit the preview mode.

  8. In the window that opens, click Finish editing.

Learn more about working with the project in Project.

Create a pool

A pool is a set of paid tasks sent out for completion at the same time.

  1. Open the page of the project that evaluates the video quality.

  2. Click the Add a pool button.

  3. Specify the Pool name.

  4. (optional) Enter a private description.

    1. Turn on the Add a private description option.

    2. In the Private description field, enter the pool description. This information is available only to you.

  5. In the Price per task page section, the Price in US dollars field, specify the price. For example, 0.01.

    What is a task page?

    A page can contain one or several tasks. If the tasks are simple, you can add 10-20 tasks per page. Don't make pages too long because it slows down loading speed for performers.

    Performers get paid for completing the whole page.

    The number of tasks on the page is set when uploading tasks.

    What is the fair price for a task page?

    The general rule of pricing is the more time the performer spends to complete the task, the higher the price is.

    You can register in Yandex.Toloka as a performer and find out how much other requesters pay for tasks, or see examples of cost for different types of tasks.

  6. In the Performers section, filter performers.

    • To make the task available only to performers who speak Russian:

      1. Click Add filter.

      2. Find the Audience presets block in the list and choose the Russian speaking users filter.

    • If the task isn't suitable for performers with mobile devices:

      1. Restrict access from the mobile browser. Add the User agent type filter.

      2. Set the following values: User agent type Mobile browser.

      3. Restrict access from smartphones and tablets. Add the Device type filter.

      4. Set the following values: Device type Smartphone or Tablet.

  7. Quality control rules allow you to filter out careless performers. In the Quality control section, add the following quality control blocks:

    • Control tasks — filters out performers who often make mistakes in the control tasks.

      1. Click Add Quality Control Rule.

      2. Find the Rules block in the list and choose Control tasks.

      3. 'Set a rule for control task: if the number of responses to the control questions is ≥ 3 and correct responses (%) to the control questions is < 60, then ban the performer on project for 10 days. Specify Control task as a reason.

        This means that if a performer completed more than three control tasks and gave incorrect answers in more than 60% of them, they will be blocked and won't be able to complete tasks on this project for 10 days.

    • Fast responses — filters out performers who respond too quickly.

      1. In the Recent values to use field, enter the number of the performer's recently completed task pages. For example, 5.
      2. In the Minimum time per task page field, enter the time in seconds. For example, 20.
      3. Set a rule for a fast response: if the number of fast responses ≥ 1, then ban on requester for 10 days. In the Reason field, enter Fast responses.

        This means that a user who completes a task page in less than 20 seconds won't be able to take your tasks for 10 days.

    • Majority vote — a check based on matching responses from the majority of performers.

      1. In the Accept as majority field, soecify 2.

      2. Set the rule: if number of responses ≥ 5 and correct responses (%) < 50, then ban the performer on project for 10 days. Specify the reason Doesn't match the majority.

        This means that if a performer completed more than ten tasks and their responses don't match the opinions of most of performers, they won't be able to complete project tasks for 10 days.

        Note.

        The rule takes effect when the number of responses for the task is equal to the overlap. To get the required number of responses faster, enable the Keep task order option in the pool settings.

    • Captcha — prevents the task completion by robots.

      1. In the Recent values to use field, enter the number of the performer's recently completed task pages. For example, 10.
      2. Set a captcha rule: if the number of responses ≥ 5 and correct responses (%) < 65, then ban on project for 10 days. In the Reason field, enter Captcha.

        This means that if a performer enters captcha correctly in less than 65% of cases, they won't be able to complete tasks on the project for 10 days.

    Note.

    You can copy quality control settings from another pool. To do this, click Copy settings from in the Users filter section.

  8. Set overlap, which is the number of performers to complete the same task. In the Overlap section, specify the Overlap field value. For tasks of this type it is usually 3-5.

  9. (optional) In the Speed/quality ratio section, specify the percentage of top-rated performers who will get access to the pool. Learn more in the Speed/quality balance section.

  10. Adjust the pool parameters:

    1. Specify the Time for completing a task page in seconds field value. There should be enough time to read the instructions, load the task, watch the video, and respond. For example, 1200 seconds.
    2. Turn on the Keep task order option.
  11. Click Save.

Upload a TSV file with tasks.

Prepare your own task file. Check out the example in demo TSV file. You can also find it on the pool page. At the top left of the page, there are links to the TSV file with regular, control, and training tasks.

  1. On the pool page, click Upload. In the window that opens, you can also download a sample TSV file by clicking Sample file for uploading tasks.

  2. In the window that opens, configure the file upload settings:

    1. Choose Smart mixing.

    2. In the Main tasks field, specify 9.

    3. In the Training tasks field, specify 0.

    4. In the Control tasks field, specify 1.

    5. Click Upload.

    6. In the window that opens, choose the TSV file with tasks to upload and click Open.

    7. In the window that opens, check the number of tasks and click Add.

  3. Create a control task.

    1. Click Edit.

      Note. If you selected something else instead of smart mixing, click Edit. If this button is missing, delete the file and upload it again.
    2. In the window that opens, click Create control tasks.

    3. In the window that opens, in the Create control task section, mark the result item on the left.

    4. Choose the correct answer to the question.

    5. Click Save and go to next.

    6. Click Rate the video quality to exit task markup mode.

    Note.

    In small pools, control tasks should be 10% of all tasks. Include different versions of correct responses in equal amounts. View the distribution of responses on the Edit tasks page, Control tasks tab.

  4. Click to start the pool.

Get the results

  1. Next to the Download results button, click .

  2. Choose Dawid-Skene aggregation model from the list. Read more about Result aggregation based on the Dawid-Skene model.

  3. In the window that opens, click Yes.

  4. At the top of the page, click View the list of operations.

  5. When the operation is complete, download the TSV file with the results. To do this, click Download in the Files column.

Tip.

To avoid unwanted errors, we recommend that you first run your project in the Sandbox.