Majority vote

Verification using the opinion of the majority (Majority vote) defines how well the user's responses correspond to the responses of other users. For example, if a task has an overlap of “5” and three users selected the same answer, this is considered indirect confirmation of the correct response.

The skill level is calculated using the formula:

,

where:

is the number of the user's responses that matched the majority.

is the number of the user's tasks where the majority all answered identically.

Setting up majority opinion verification

To use verification by the majority:

  1. Create a skill on the Skills page in the production version of Toloka.

  2. Set up a quality control rule on the pool page (click Add Quality Control Rule → Majority vote).

    Field or group of fields

    Description

    Answer threshold

    The number of matching responses that is considered the “majority vote” (for example, 3). The threshold should be less than or equal to the overlap.

    History size

    The number of user's recent responses in the project used for the calculation.

    If the field is empty, the calculation is based on all user's responses in the pool.

    then

    Action to perform for the condition:

    • “Set skill from "If"” — Save the user's percentage of correct responses in the control tasks or tasks with majority vote. To save the percentage (a number between 1 and 100), use a skill — an assessment of the user's performance that has a name and value. For example, you can create the skill “Completed control tasks in my pool” and use it for saving the percentage of correct responses for each user. Then you can set a filter using this skill to control users' access to the pool.

    • “Set skill” — Assign the skill a fixed value of N (from 1 to 100).

    • “Pause” — Suspend the user's access to the pool for the specified number of days. The reason for banning is only shown to the requester.

    • “Ban” — Block access to the project or all of the requester's projects for the specified number of days. The reason for banning is only shown to the requester.

  3. Add a skill filter on the pool page (in the Users filter section). For example, “The myskill skill is missing or ≥ 60”.