Majority vote

Majority vote is a quality control method based on matching responses from the majority of performers who complete the same task. For example, if a task has an overlap of “5” and three performers selected the same answer, this is considered indirect confirmation of the correct response. Based on the percentage of correct responses, you can change the value of the performer's skill or block their access to tasks.

The rule is activated when the task overlap is complete.
Example

Let's say we set up this rule in a pool with overlap=5.



It is activated when at least three of the tasks completed by the performer reach full overlap. When responses have been received from all five task performers, the system determines which response was selected by the majority. If the performer's responses to two out of three tasks differ from the responses of the majority, the performer is banned from the project for 10 days. To collect all the responses needed to trigger the rule, enable Keep task order in the pool settings.



The number of tasks per page doesn't affect how the rule works. Responses are evaluated separately for each task. Only the speed of collecting results can be affected, because the more tasks you put on each page, the longer it takes to do them.

When to use

Use the Majority vote rule if:
  • You can't regularly create and add control tasks.
  • You want to make an additional review of the performer's responses.
Don't use it if:
  • You have a lot of response options.
  • Users need to attach a file to their assignment.
  • Users need to transcribe text.
  • Users need to select objects in a photo.
  • Tasks don't have a correct or incorrect response. For example: “Which image do you like best?” or “Choose the page design option that you like best”.

Usage recommendations

Don't use too many response options or too many questions in the task.

The more options you have, the lower the probability that majority vote will be calculated correctly.

To calculate the majority vote for the task, all the responses to all the questions inside the task must match. If a task has multiple questions or the questions have more than just a few response options, it will be unlikely to have enough exactly matching task responses from different performers in order to calculate the skill. As a result, the task won't be used for skill calculation. If the majority vote hasn't been calculated for the task, the performer isn't penalized for mismatching the majority vote.

Example
Performer 1 Performer 2 Performer 3 Performer 4 Performer 5
Task 1 OK BAD OK BAD OK
Task 2 BAD OK OK BAD 404
Task 3 404 OK 404 OK OK
Task 4 OK BAD OK 404 404
Task 5 OK BAD OK 404 OK

The requester has five tasks, each completed by five performers. Each task has one output field with three response options. Overlap = 3. In this case, tasks 1, 3, and 5 match the threshold, but the other tasks are not included in the majority vote calculation.

The performers will be assigned the following skills based on majority vote:
Performer 1 (1+0+1)/3=66% correct
Performer 2 (0+1+0)/3=33% correct
Performer 3 (1+0+1)/3= 66% correct
Performer 4 (0+1+0)/3=33% correct
Performer 5 (1+1+1)/3=100% correct

Let's say the task performer has to select at least three matching categories out of twenty or report that the image is not displayed. Since there can be multiple combinations of categories, the rule calculation will mostly include tasks where all the performers responded Not displayed. Those might be either correct or “fraudulent” responses from performers who just clicked through the tasks in a hurry. In any case, it would be unreliable to base the skill on these responses.

Try using decomposition or control tasks. This way you can better assess the quality of your performers.

Don't set the threshold "Accept as majority" below 50% of the overlap or equal to the overlap.

If the response threshold is equal to the overlap, the rule will only use the tasks where all the performers gave the same responses, ignoring tasks with mismatching votes.

Example
Let's say the pool has overlap=5 and Accept as majority=2. The performers completed the task.
Task response
Performer 1 OK
Performer 2 OK
Performer 3 BAD
Performer 4 BAD
Performer 5 404

In this case, performers 1, 2, 3, and 4 responded correctly, but performer 5 failed. However, you don't have any way to know which response is correct.

Don't use the majority vote if:
  • You have a lot of response options.
  • Users need to attach a file to their assignment.
  • Users need to transcribe text or select objects in a photo, or other tasks where the performers can't possibly provide the same responses.

Rule settings

Attention. In this rule, all fields are required except Recent values to use. If you don't fill in at least one of them, you won't be able to save the rule.

Field

Overview

Accept as majority

The number of matching responses that is considered the “majority vote” (for example, 3).

Recent values to use

The number of the performer's recent responses.

To calculate task responses from all project pools, fill in the field in the rule for each pool.

If this field is not filled in, the calculation includes only task responses in the pool to which the rule applies.

If

A condition for performing the action in the then field:
  • number of responses — The number of completed tasks.

  • % correct answers — The percentage of correct responses, meaning responses that matched the majority opinion (from 0 to 100).

  • % incorrect answers — The percentage of incorrect responses, meaning responses that didn't match the majority opinion (from 0 to 100).

To add multiple conditions, click .

then

Action to perform for the condition:

  • assign skill value — Assign a fixed value to the skill.

  • accept user's answers — Requires the non-automatic acceptance option to be set.

    Useful if the performer completes most tasks well. Example: The Toloker completed more than 80% of the tasks correctly and you are satisfied with this result. The rule will work automatically and accept all responses in the pool.

  • suspend — Suspend the performer's access to the pool for the specified number of days. Only the requester can view the reason.

  • assign skill from the field — Save the percentage of the performer's correct responses in tasks as a skill value.

  • ban — Block access to the project or all of the requester's projects for the specified number of days. Only the requester can view the reason.

    If access to tasks is blocked temporarily (for example, for 7 days), the history of the performer's responses is not saved after the ban is lifted. The skill value is based on new responses.

Examples of rules

Examples are provided for simple classification. There are 10 tasks per page.

Calculating a skill and banning for incorrect responses




Both rules work independently:

  1. If the performer gives at least 3 responses to the tasks, the percentage of correct answers is written as the skill value.
  2. If the performer gives at least 3 answers to tasks and the percentage of correct answers is less than 65%, they are banned on the project.

The calculation uses up to 10 performer's responses to the project tasks.

Setting the skill


If the performer completes 3 tasks, the skill is set to the percentage of correct answers.

Use the skill value to set access to other pools with filters.

Example of filter settings


Blocking for incorrect responses


If the percentage of correct responses is less than 40%, the performer is blocked on the project for 30 days.

Troubleshooting

Should I create a skill for every pool?

It is better to use one skill in a project. You can choose the way to calculate the skill:

  • Calculate the skill for each pool separately. The current skill value is the value of the skill in the pool the user completed last. This option is convenient if:

    • The pools are intended for different groups of performers (for example, there are filters by city or country).

    • Pools are started one by one and you don't want to take into account the responses in the previous pools to calculate the skill in the current pool.

    This calculation method is used by default when adding a quality control rule to a pool. For the control tasks block, leave the Recent values to use field empty.

  • Calculate skill based on all tasks in a project This option is good if the pools are small and you don't need to have skill calculated for each pool.

    This option is available only for skills on control tasks. To use it, fill in the Recent values to use field in quality control rules in pools.

Can I use a skill beyond a particular pool or project and apply it to other projects as well?

Yes, of course — you can use the same skill for different projects. But most often, a skill is intended for a specific project. If the performer completes a certain task well, this doesn't mean that they will complete other ones successfully. Another disadvantage is that if you filter by skills that were set long ago, you will artificially limit the number of available performers.

What output format do I use for the review results to filter out mismatching users based on the “Majority vote”?

To perform actions with users (assign a skill or ban them) based on the majority vote, add a relevant rule to the pool.

Don't forget to enable Keep task order in the pool parameters. Majority vote is used in the projects with preset options (radio buttons or checkboxes). This rule won't apply to the text entry or file upload fields.

The pool has an overlap and majority vote set up, but some fraudulent performer opens the task suites, does nothing, and submits empty assignments. Could this cheater get more tasks from the pool before the results of other performers are known? Could the cheater quickly click through a lot of task suites before the majority vote is accumulated to ban the cheater?

Yes, unfortunately, this can happen. This is why we recommend that you offer a training task or exam before the main task. In this case, only those people who showed good performance at the previous stage are selected for the main pool.

My task uses a form with multiple fields. When there is an overlap and “Majority vote” is used for quality control, is each field taken into account, or if one field mismatches the majority vote, are the task results considered incorrect?

All responses to the task are taken into account. If one response differs from the majority vote, the whole task is counted as mismatching the responses of other performers.