Majority vote
Majority vote is a quality control method based on matching responses from the majority of performers who complete the same task. For example, if a task has an overlap of “5” and three performers selected the same answer, this is considered indirect confirmation of the correct response. Based on the percentage of correct responses, you can change the value of the performer's skill or block their access to tasks.
Let's say we set up this rule in a pool with overlap=5.
It is activated when at least three of the tasks completed by the performer reach full overlap. When responses have been received from all five task performers, the system determines which response was selected by the majority. If the performer's responses to two out of three tasks differ from the responses of the majority, the performer is banned from the project for 10 days. To collect all the responses needed to trigger the rule, enable Keep task order in the pool settings.
The number of tasks per page doesn't affect how the rule works. Responses are evaluated separately for each task. Only the speed of collecting results can be affected, because the more tasks you put on each page, the longer it takes to do them.
Usage recommendations
- Don't use too many response options or too many questions in the task.
-
The more options you have, the lower the probability that majority vote will be calculated correctly.
To calculate the majority vote for the task, all the responses to all the questions inside the task must match. If a task has multiple questions or the questions have more than just a few response options, it will be unlikely to have enough exactly matching task responses from different performers in order to calculate the skill. As a result, the task won't be used for skill calculation. If the majority vote hasn't been calculated for the task, the performer isn't penalized for mismatching the majority vote.
ExamplePerformer 1 Performer 2 Performer 3 Performer 4 Performer 5 Task 1 OK BAD OK BAD OK Task 2 BAD OK OK BAD 404 Task 3 404 OK 404 OK OK Task 4 OK BAD OK 404 404 Task 5 OK BAD OK 404 OK Performer 1 Performer 2 Performer 3 Performer 4 Performer 5 Task 1 OK BAD OK BAD OK Task 2 BAD OK OK BAD 404 Task 3 404 OK 404 OK OK Task 4 OK BAD OK 404 404 Task 5 OK BAD OK 404 OK The requester has five tasks, each completed by five performers. Each task has one output field with three response options.
Overlap = 3. In this case, tasks 1, 3, and 5 match the threshold, but the other tasks are not included in the majority vote calculation.The performers will be assigned the following skills based on majority vote:Performer 1 (1+0+1)/3=66% correct Performer 2 (0+1+0)/3=33% correct Performer 3 (1+0+1)/3= 66% correct Performer 4 (0+1+0)/3=33% correct Performer 5 (1+1+1)/3=100% correct Performer 1 (1+0+1)/3=66% correct Performer 2 (0+1+0)/3=33% correct Performer 3 (1+0+1)/3= 66% correct Performer 4 (0+1+0)/3=33% correct Performer 5 (1+1+1)/3=100% correct Let's say the task performer has to select at least three matching categories out of twenty or report that the image is not displayed. Since there can be multiple combinations of categories, the rule calculation will mostly include tasks where all the performers responded Not displayed. Those might be either correct or “fraudulent” responses from performers who just clicked through the tasks in a hurry. In any case, it would be unreliable to base the skill on these responses.
Try using decomposition or control tasks. This way you can better assess the quality of your performers.
- Don't set the threshold "Accept as majority" below 50% of the overlap or equal to the overlap.
-
If the response threshold is equal to the overlap, the rule will only use the tasks where all the performers gave the same responses, ignoring tasks with mismatching votes.
ExampleLet's say the pool hasoverlap=5andAccept as majority=2. The performers completed the task.Task response Performer 1 OK Performer 2 OK Performer 3 BAD Performer 4 BAD Performer 5 404 Task response Performer 1 OK Performer 2 OK Performer 3 BAD Performer 4 BAD Performer 5 404 In this case, performers 1, 2, 3, and 4 responded correctly, but performer 5 failed. However, you don't have any way to know which response is correct.
- Don't use the majority vote if:
-
- You have a lot of response options.
- Users need to attach a file to their assignment.
- Users need to transcribe text or select objects in a photo, or other tasks where the performers can't possibly provide the same responses.
Rule settings
| Field | Description |
|---|---|
| Accept as majority | The number of matching responses that is considered the “majority vote” (for example, 3). |
| History size | The number of the performer's recent responses. To calculate task responses from all project pools, fill in the field in the rule for each pool. If this field is not filled in, the calculation includes only task responses in the pool to which the rule applies. |
| If | A condition for performing the action in the then field:
To add multiple conditions, click |
| then | Action to perform for the condition:
|
.| Field | Description |
|---|---|
| Accept as majority | The number of matching responses that is considered the “majority vote” (for example, 3). |
| History size | The number of the performer's recent responses. To calculate task responses from all project pools, fill in the field in the rule for each pool. If this field is not filled in, the calculation includes only task responses in the pool to which the rule applies. |
| If | A condition for performing the action in the then field:
To add multiple conditions, click |
| then | Action to perform for the condition:
|
Examples of rules
Examples are provided for simple classification. There are 10 tasks per page.
- Calculating a skill and banning for incorrect responses
-
Both rules work independently:
- If the performer gives at least 3 responses to the tasks, the percentage of correct answers is written as the skill value.
- If the performer gives at least 3 answers to tasks and the percentage of correct answers is less than 65%, they are banned on the project.
The calculation uses up to 10 performer's responses to the project tasks.
The performer is blocked after the first incorrect response to the first, second or third task. The skill is not assigned. Since the ban reason is not specified, there is no way to find out why the performer is banned.
All rules are applied independently:- If the performer gives at least 3 responses to the tasks, the percentage of correct answers is written as the skill value.
- If the performer gives 2 incorrect responses to 3 tasks, they are blocked in the pool for 10 days.
- If the performer gives 2 incorrect responses to 4 tasks, they are blocked in the pool for 10 days.
- If the performer gives 5 or more responses to tasks and the percentage of correct responses is less than 80%, they are blocked in the pool for 10 days.
A set of rules like this prevents performers from being banned for one incorrect response and lets you maintain high accuracy.
-
If the performer completes 3 tasks, the skill is set to the percentage of correct answers.
Use the skill value to set access to other pools with filters.
Example of filter settings
This rule will never take effect because the number of responses counted (Recent values to use) is less than the number of responses in the rule (number of responses).
- Blocking for incorrect responses
-
If the percentage of correct responses is less than 40%, the performer is blocked on the project for 30 days.
If the percentage of correct responses is less than 40%, the performer is blocked on the project for 30 days. The rule will be applied once, after the fifth response.
Troubleshooting
Calculate the skill for each pool separately. The current skill value is the value of the skill in the pool the user completed last. This option is convenient if:
The pools are intended for different groups of performers (for example, there are filters by city or country).
Pools are started one by one and you don't want to take into account the responses in the previous pools to calculate the skill in the current pool.
This calculation method is used by default when adding a quality control rule to a pool. For the control tasks block, leave the Recent values to use field empty.
Calculate skill based on all tasks in a project This option is good if the pools are small and you don't need to have skill calculated for each pool.
This option is available only for skills on control tasks. To use it, fill in the Recent values to use field in quality control rules in pools.
It is better to use one skill in a project. You can choose the way to calculate the skill:
Yes, of course — you can use the same skill for different projects. But most often, a skill is intended for a specific project. If the performer completes a certain task well, this doesn't mean that they will complete other ones successfully. Another disadvantage is that if you filter by skills that were set long ago, you will artificially limit the number of available performers.
