General criteria

  1. Incremental relabeling
  2. Performer selection: training — exam — retry
  3. Reward program and dynamic pricing

Incremental relabeling

incremental relabeling saves your budget by not setting a fixed overlap for all tasks in the pool. An algorithm determines how many performers should complete the same task in order to get a reliable result, based on the performer skill levels and the consistency of their responses. It works well for tasks with response options and an overlap of 2-5 (like classification tasks). Learn more about incremental relabeling.

Performer selection: training — exam — retry

Each performer has their own strengths. Some people are good at finding typos, some like selecting objects in images, others do a great job transcribing audio, and some people are talented secret shoppers. But they all need to practice before completing a new task. Your project should have a training pool and exam (the control tasks pool) to choose the best performers. A "retry pool" will help performers who make mistakes improve their skills and get a second chance to complete tasks.

Reward program and dynamic pricing

You can issue rewards to performers for completing tasks well or increase their payment depending on the skill level. This encourages performers to try harder on tasks.

If your projects meet these requirements, we'll be happy to partner with you and recommend your services to requesters.