How to Write Data Annotation Guidelines

In short: a labeling guideline is the contract for what "correct" means. It has to settle eight things: the purpose, the classes, what to label and what to skip, how to draw, what to do with hidden objects, what to do when unsure, which extra flags to set, and how to deliver. Write a first version, try it on two annotators, and rewrite every rule they read differently.

Why the guideline decides the quality

Many label errors are not carelessness. Two careful people read the same vague rule in two ways. The model then learns both, and no amount of checking afterwards tells you which was meant.

A measurement of quality is a measurement against the guideline. Without one there is nothing to measure: see how to measure data annotation quality.

The eight things a guideline must settle

1. The purpose. One sentence on what the model will do with the labels. An annotator who knows the purpose resolves small doubts the right way.
2. The classes. For each: a name, a definition, what it includes, what it excludes, and example images. Add the near-misses, the things that look like the class and are not.
3. What to label and what to skip. The smallest size worth labeling. How much of an object must be visible. What to do with reflections, pictures of objects and toys.
4. How to draw. How tight a box or outline must be. Whether it covers what is visible or the object's full extent.
5. Hidden and cut-off objects. When an object counts as hidden, when it counts as cut off by the image edge, and whether that is recorded.
6. When unsure. The default choice, and a way to flag an item for a second opinion instead of guessing.
7. Flags and attributes. Anything recorded beside the label: hidden, cut off, blurred, colour, direction.
8. Delivery. The file format, the names of the classes exactly as they must appear, and how files are named.

A real example: the PASCAL VOC guidelines

The guidelines of the PASCAL VOC dataset are public, and they are a model of the form: short rules, each with a number or an example (VOC2011 Annotation Guidelines). Some of them, word for word:

• "Mark the bounding box of the visible area of the object (not the estimated total extent of the object)."
• "If more than 15-20% of the object lies outside the bounding box mark as Truncated."
• "If more than 5% of the object is occluded within the bounding box, mark as Occluded."
• "Do label objects in mirrors."
• "Label objects in pictures/posters/signs only if they are photorealistic but not if cartoons, symbols etc."

Each rule answers a question that a careful annotator would otherwise have to guess. It also defines each class by what is in and what is out: a car "includes cars, vans, large family cars" and excludes go-carts, tractors and trucks.

A one-page template

1. Task: one sentence. What is labeled, and what the model will do with it.
2. Classes: for each one, the name, a definition, "includes", "excludes", and two example images.
3. Label when: the conditions. Skip when: the conditions, with a size in pixels and a share that must be visible.
4. Drawing rule: tightness in pixels; visible area or full extent.
5. Hidden and cut-off objects: the thresholds, and the flag to set.
6. When unsure: the default, and how to flag.
7. Flags: the list, with a definition for each.
8. Delivery: format, class names, file names.
9. Change log: version, date, what changed.

Keep it to what fits on a page, and put the example images on a second one.

A vague rule and a rule that works

• Vague: "Label all cars." Works: "Label every car, van and pickup of which at least a fifth is visible. Skip toy cars and cars in posters."
• Vague: "Draw tight boxes." Works: "The box touches the outermost visible pixels of the object, within three pixels on each side."
• Vague: "Mark unclear images." Works: "If you cannot name the class within ten seconds, set the flag 'unsure' and move on."

The test for a rule is simple: could two people who have never met apply it and get the same answer?

Test it before you trust it

1. Give the same 50 items to two annotators who have only the guideline.
2. Compare their labels, item by item.
3. For every disagreement, find the rule that was missing or unclear, and rewrite it.
4. Repeat with new items until the disagreements are rare and random.

This is the cheapest test in the whole project, and it belongs in the pilot.

Keep it alive

• One list of questions. Every question an annotator asks goes in, with its answer. The list becomes the next version.
• Versions and dates. Everyone works from the same one.
• Announce changes. A silent change gives you a dataset labeled under two sets of rules.
• Decide about the past. When a rule changes what counts as correct, the items already labeled have to be redone or set aside.

Where Pripton fits

Pripton Innovations labels to your written guideline, and a small first batch is the quickest way to find its gaps. If you have data and no guideline yet, send a sample and the model's task through the contact page. Choosing the kind of label comes first: see bounding boxes, polygons or segmentation.