Majority vote

Majority vote is a quality control method based on coinciding responses from the majority of users who are performing the same task. The response chosen by the majority is considered correct, and other responses are considered incorrect. Depending on the percentage of correct responses, you can either increase the user's skill value, or ban the user from tasks.

The majority vote algorithm only works with the required fields of the output specification.

For majority vote verification based on performer responses, you can set up the following:

  • Count of skill values.

  • Skill value updates.

  • User account blocking.

To set up the verification, define key values in the quality_control.configs array (pool parameters).

Example for a JSON block

The pool is set up to use verification with an overlap of “5”. The calculation uses tasks in which three or more performers submitted the same response. The percentage of correct responses (those that match the majority vote) is stored as skill level 43. You can apply the skill for filtering users. The skill is first calculated after two tasks that coincide with the majority vote have been completed.

You can ban a performer from accessing the tasks in the project for a given number of days, hours, minutes (at a time or in total) or forever.

Ban for 10 days
{
   "configs": [
      {
         "collector_config": {
            "type": "MAJORITY_VOTE",
            "parameters": {
               "answer_threshold": 3,
               "history_size": 10
            }
         },
         "rules": [
            {
               "conditions": [
                  {
                     "key": "total_answers_count",
                     "operator": "GT",
                     "value": 2
                  }
               ],
               "action": {
                  "type":"SET_SKILL_FROM_OUTPUT_FIELD",
                  "parameters": {
                     "skill_id": "43",
                     "from_field": "correct_answers_rate"
                  }
               }
            },
            {
               "conditions": [
                  {
                     "key": "total_answers_count",
                     "operator": "GTE",
                     "value": 5
                  },
                  {
                     "key": "incorrect_answers_rate",
                     "operator": "GT",
                     "value": 3
                  }
               ],
               "action": {
                  "type":"RESTRICTION_V2",
                  "parameters": {
                     "scope": "PROJECT",
                     "duration_unit": "DAYS",
                     "duration": 10,
                     "private_comment": "Doesn't match the majority vote"
                  }
               }
            }
         ]
      }
   ]
}

To set a different ban period, change the applicable parameter for the action key:

{
   ...
               "action": {
                  "type":"RESTRICTION_V2",
                  "parameters": {
                     "scope": "PROJECT",
                     "duration_unit": "HOURS",
                     "duration": 12,
                     "private_comment": "Doesn't match the majority vote"
                  }
               }
   ...
}

Keys

Key

Value

Data type

Required during creation

collector_сonfig.parameters.answer_threshold

The number of users considered the majority (for example, 3 out of 5).

Integer

Yes

collector_сonfig.parameters.history_size

The maximum number of the user's recent responses in the project to use for calculating the percentage of correct responses.

If this field is omitted, the calculation is based on all the user's responses in the pool.

Integer No
rules.conditions.key

Parameters that are checked in the condition:

  • total_answers_count — The number of completed tasks that had a majority vote (specify the majority in answer_threshold).

  • correct_answers_rate — The percentage of correct responses, meaning responses that matched the majority vote (from 0 to 100).

  • incorrect_answers_rate — The percentage of incorrect responses, meaning responses that didn't match the majority vote (from 0 to 100).

String

Yes

Key

Value

Data type

Required during creation

collector_сonfig.parameters.answer_threshold

The number of users considered the majority (for example, 3 out of 5).

Integer

Yes

collector_сonfig.parameters.history_size

The maximum number of the user's recent responses in the project to use for calculating the percentage of correct responses.

If this field is omitted, the calculation is based on all the user's responses in the pool.

Integer No
rules.conditions.key

Parameters that are checked in the condition:

  • total_answers_count — The number of completed tasks that had a majority vote (specify the majority in answer_threshold).

  • correct_answers_rate — The percentage of correct responses, meaning responses that matched the majority vote (from 0 to 100).

  • incorrect_answers_rate — The percentage of incorrect responses, meaning responses that didn't match the majority vote (from 0 to 100).

String

Yes