Majority vote

  1. Majority vote verification
  2. Selective majority vote

Verification using the opinion of the majority (majority vote) defines how well the user's responses match the responses of other users. For example, if a task has an overlap of “5” and three users selected the same answer, this is considered indirect confirmation of the correct response.

The skill level is calculated using the formula:

,

where:

is the number of the user's responses that matched the majority.

is the number of the user's tasks where the majority all answered identically.

Majority vote verification

To use verification by the majority:

  1. Create a skill on the Skills page in the production version of Toloka.

  2. Set up a quality control rule on the pool page (click Add Quality Control Rule → Majority vote).

    Field or group of fields

    Description

    Answer threshold

    The number of matching responses that is considered the “majority vote” (for example, 3). The threshold should be less than or equal to the overlap.

    History size

    The number of user's recent responses in the project used for the calculation.

    If the field is empty, the calculation is based on all user's responses in the pool.

    then

    Action to perform for the condition:

    • “Set skill from "If"” — Save the user's percentage of correct responses in the control tasks or tasks with majority vote. To save the percentage (a number between 1 and 100), use a skill — an assessment of the user's performance that has a name and value. For example, you can create the skill “Completed control tasks in my pool” and use it for saving the percentage of correct responses for each user. Then you can set a filter using this skill to control users' access to the pool.

    • “Set skill” — Assign the skill a fixed value of N (from 1 to 100).

    • “Pause” — Suspend the user's access to the pool for the specified number of days. The reason for banning is only shown to the requester.

    • “Ban” — Block access to the project or all of the requester's projects for the specified number of days. The reason for banning is only shown to the requester.

  3. Add a skill filter on the pool page (in the Users filter section). For example, “The myskill skill is missing or ≥ 60”.

Selective majority vote

You can apply selective majority vote for a limited number of tasks. It allows you to save money and speed up pool completion. The frequency of checking can be reduced as the user completes more tasks. For that to happen, you should define ranges in the pool settings.

To add selective majority vote:

  1. Upload tasks with the smart mixing option to the pool.

  2. Switch to the edit mode.

  3. Click in the Tasks per page section.

  4. Specify the number of days to store answers and the overlap for selective majority vote tasks.

  5. Define the ranges and the number of control tasks for each.

Example of ranges for selective majority vote