In short: twenty-five questions in five groups: the people, the quality, the security, the process and the terms. Good answers are specific. They name a number, a method or a person. Take the list into the first call, and ask for a small batch on your own data before you sign for the volume.
The people
- Who will label our data? Employees, contractors or an open crowd. It changes training, confidentiality and consistency.
- How are they trained for our task? Ask what happens between "we received your guideline" and "production starts".
- Have they worked in our field and language? Medical, legal and technical data need people who understand it.
- Who is our one point of contact? A named person who knows the project, not a ticket queue.
- What happens when our volume doubles, or stops for a month? How fast the team grows, and what you owe when it idles.
The quality
- How do you measure accuracy? Against which reference, on which sample, how often.
- Is every item reviewed, or a sample? And by whom: a second annotator, a senior reviewer, an expert.
- What happens when two annotators disagree? Who decides, and whether the rule is then written down.
- What accuracy do you commit to, and what happens below it? Rework, by when.
- Can we see results class by class? An average hides the rare class that is wrong half the time.
- How does a change to the guideline reach every annotator? And what happens to the items already labeled.
How each of these should work is in how to measure data annotation quality.
The security
- Where is our data stored, and who can open it? Country, system, and the number of people.
- Are the people who see it under a confidentiality agreement? Each of them, not only the company.
- Can they copy it? Whether work happens on managed machines, and what is blocked.
- When is it deleted, and how are we told? A date and a confirmation.
- Which standards or audits can you show? Ask for the document, not the logo.
What to do on your side first: how to prepare data for annotation.
The process
- Whose labeling tool is used? Yours, theirs, or one you both agree on.
- Which formats do the labels come back in? Down to the exact class names.
- How are annotators' questions raised and answered? One list, with answers everyone sees.
- How often do we get a delivery and a report? And what the report contains.
- Can we start with a small batch? See how to run a data annotation pilot.
The terms
- How do you charge? By unit of work, by time or by project. See how data annotation projects are quoted.
- What is included? Review, rework, project management, the tool.
- Who owns the labels? The answer should be: you do, all of them.
- How do we scale down or leave? The notice period, and what is handed back.
How to read the answers
- Specific beats confident. "Two annotators on a tenth of the items, differences settled by a reviewer, reported weekly" is an answer. "We have rigorous quality processes" is not.
- Ask to see one. A sample report, a sample guideline, the question list from a past project with the client's name removed.
- Watch for "100%". No labeling process is free of error. A vendor who says so is not measuring.
- Notice the questions you are asked. A good vendor asks about your model, your edge cases and your guideline before quoting.
- Then test. Answers describe a process. A small batch on your own data shows it.
Still deciding whether to hand the work out at all? Read in-house or outsourced data annotation.
How Pripton answers some of these
What we publish, so you can check it:
- Quality (6 to 9). We label to your written guideline, and labels are checked in a second pass before delivery. Our AWS Marketplace listings report 98.5% average accuracy. If a project falls below 96.5% accuracy, we rework it.
- People (1 and 5). More than 500 trained annotators, working in more than 15 languages.
- Confidentiality (13). Our people work under non-disclosure agreements and data protection protocols.
- Terms (22). Each project is quoted, because the cost depends on the data, the volume and the detail of the labels.
For the other questions, ask us: they deserve an answer that fits your project. Use the contact page.
