Questions about templates
If you are looking for an answer to a specific question, use Ctrl+F to search the page (Cmd+F on MacOS).
Selecting a template
You can structure your text classification task using the source text and radio buttons. It can also be text and checkboxes, if you use multiple subjects.
We recommend that you take the "Text classification with additional options" template as a basis.
You can also create your own interface. All the available tools are described in the Requester's guide.
You can see how templates are implemented in the "Transcription" section: a string-type output field is used there. In the "Photo transcription" template, you can see how to define the text input fields. Please note that if you later want to use the Dawid-Skene aggregation method, you must use acceptable values.
You can create such an assignment based on the classification template. Show a product image and ask the question: "Does the product match the query?" Add two radio buttons for the responses: "Yes" and "No".
If your task contains many objects of different types, you should break it down. For example, you can ask users to select numbers in the first project, doorways and windows in the second project, walls in the third project, and plumbing in the fourth project.
The simpler the task, the cheaper it is and the better the quality of the final result. Set the cost of labeling a single class of objects in photos at about $0.01.
Use the Object selection in an image template. You can open this template in the editor and add a drop-down list for labeling the selected object. See how to do this in the editor description (Dropdown list tab).
You can use JavaScript to add assignment validation depending on a checkbox. An example is provided in the "Find information online" template.
Area selection
- Sorting images containing an object.
- Create a task using the "Image classification" template.
- Sort the images containing the object you are looking for.
- Show the image to the performer and ask if the object is in the image. Response options: Yes/No.
- Selecting objects in images.
- Select the object in the images you obtained after the previous project. You already have such a project. Run the task with non-automatic acceptance.
- Use the quality control rules: fast responses, non-automatic acceptance, and post-review re-assessment. Description of rules with examples.
- Reviewing object selection assignments.
- Create a task using the object selection template.
- Hide the editor and ask whether the object is selected correctly. Response options: Yes/No.
- In the input data, pass the images and coordinates of the labeled objects from the previous task.
To prevent the users who worked on the second project from doing the review, assign a skill to them. Use this skill as a filter in the pools of the third project.
- Select an arbitrary area in the image (for example, put a square in the upper-right corner). In this case, the project instructions for reviewers should also reflect this.
- Ask the performer to skip the task and report it in a personal message. Messages are reviewed by the requester. If the object is truly missing, the task is deleted from the pool by resetting the overlap.
- Add an additional "No object" checkbox to the interface. Make sure that your JS checks that either the object is selected or the checkbox is enabled. In this case, add information about the checkbox value in the review task interface.
- In the project, the input field where you pass the file link has the "string" type.
- The component in the task template uses the "proxy" expression.
- The format of relative links in the TSV file with tasks is correct: <unique name>/<file path and name>.
onKey: function(key) {
var el = this.getDOMElement().querySelector(".image-annotation-editor__shape-polygon");
if (key === 'D') {
el.click();
el.classList.add('image-annotation-editor__shape_active')
}
Use the C shortcut for closing areas.
You can also use the library to customize the keyboard shortcuts for your tasks.
You can use JavaScript to add assignment validation depending on a checkbox. An example is provided in the "Find information online" template.
In the case of crowdsourcing, it's better to break down this task. The simpler the task, the cheaper it is and the better the quality of the final result. The cost of labeling a single class of objects in photos might be about $0.01.
Use the "Object selection in an image" template as a basis. See the step-by-step guide for creating this type of project on this page.
The editor used in the template lets you add a drop-down list for labeling the selected object. See how to do this in the editor description (Dropdown list tab).
You can create a selection + drop-down list with category selection. See how to do this in the editor description (Dropdown list tab).
The coordinates are relative to the image.
In the standard template with an area selection editor, you can't use the control tasks, because in order for the assignment to be accepted by the system as correct, the object selected by the user must exactly match the control object. This is almost impossible. Therefore, you can leave the GOLDEN field empty in the task file or simply delete all the columns except INPUT.
In the standard template with an area selection editor, you can't use the control tasks, because in order for the assignment to be accepted by the system as correct, the object selected by the user must exactly match the control object. This is almost impossible. Therefore, you can leave the GOLDEN field empty in the task file or simply delete all the columns except INPUT.
Using training pools in an area selection project doesn't make sense, because for the response to be correct the user's selection must completely match the control object. This is almost impossible.
Such tasks are usually run with non-automatic acceptance: the performer submits an assignment, and then the assignment is rejected or accepted after the review.
For pre-selection of users you can use "examination tasks". Review the assignments and assign skills based on the percentage of accepted assignments. For this purpose, add the "Results of assignment review" rule to the pool. To make sure that only the good performers are admitted to the main pool, put a skill-based filter to the pool.
Field task
To open the camera instead of the gallery when the user taps the image upload button, in the Image upload button component specify camera=true
.
In the mobile apps, performers can add photos from the default gallery (iOS) or Google photo (Android). To limit the capacity of adding online images, specify in the component requiredCoordinates=true
. In this case, the system won't let the user add images without geotags.
Side-by-side comparison
- Use the "Side-by-side comparison" template.
- In the TSV file, specify the links to the compared images.
- In the file, create the tasks where all the images will be compared in pairs:
Image 1 and Image 2
Image 1 and Image 3
- Image 2 and Image 3
- Process the results.
You can also edit the project so that the user sees 3 images at once and selects one of them.
Survey
The survey includes an auxiliary input field. You can use it to pass any information, and it won't be visible to performers.
Upload one task and use the overlap parameter to enter the number of people you want to survey.