In short: seven steps before your data goes to a labeling team. Take stock of what you have. Remove what must not leave your systems. Clean out duplicates and broken files. Pick a sample that looks like the real thing. Fix the formats and the file names. Keep a reference set back. And agree in writing how the data travels and where it ends up.
1. Take stock
Before anything is sent, know what you hold:
- how many items, of which types and sizes
- where they came from
- whether you have the right to share them with a third party
- what they contain that is personal, confidential or licensed
A one-page inventory answers most of a vendor's first questions and shortens the quote: see how data annotation projects are quoted.
2. Remove what must not leave
Send what the task needs and nothing else.
- In text: names, addresses, phone numbers, email addresses, account and identity numbers.
- In photos: faces and number plates, if the task is not about them. Blur them.
- Inside photo files: the hidden data a camera writes, which can include the place the picture was taken. Strip it.
- In medical images: the patient details stored in the file header, and any burned into the picture itself.
- In file names: a file called "john-smith-invoice.pdf" has leaked a name before anyone opens it.
For health data in the United States, the HIPAA Privacy Rule describes two ways to de-identify: Expert Determination, and Safe Harbor, which lists 18 kinds of identifier to remove (US Department of Health and Human Services).
Other countries have their own rules. This guide is not legal advice; ask your own counsel which rules apply to your data.
3. Clean
- Duplicates and near-duplicates. Labeling the same picture twice wastes the work, and it can leak test data into training data.
- Broken and empty files. One corrupt file can stall a batch.
- Out of scope. Items the model will never see in use do not belong in its training set.
- Too poor to label. If you cannot tell what a picture shows, neither can an annotator. Decide the cut-off now.
4. Pick a sample that looks like the real thing
The first batch decides the guideline, the quote and the pace, so it must represent the whole.
- Keep the proportions of real use: the same mix of classes, sources and conditions.
- Add the rare classes and the hard cases on purpose. A random sample holds few of them.
- Choose at random within each group. A hand-picked sample is always too clean.
5. Fix formats and names
- One format per type of data, agreed with the vendor. Raise unusual formats early, with a sample file.
- Stable, unique file names. An identifier, not a description. The same name must mean the same item for the whole project.
- A manifest. A plain list of every file, with any facts that travel with it: source, date, group.
- A flat, predictable folder layout.
- The return format. State how the labels must come back, down to the exact class names. For image labels see bounding boxes, polygons or segmentation.
6. Keep a reference set back
Before the data goes out, have your own experts label a small part of it and agree the answers. This reference set is how you will measure what comes back.
Do not hand it over marked as such. Mix it into the batches. See how to measure data annotation quality.
7. Agree how the data travels
Put it in writing before the first file moves:
- how the data is transferred, and that it is encrypted on the way
- where it is stored, and who can open it
- that it is used for this project only
- when and how it is deleted at the end, and how you are told
- the confidentiality agreement that covers the people who see it
The hand-over checklist
- An inventory: counts, types, sources, rights
- Personal and confidential details removed, including inside files and file names
- Duplicates, broken files and out-of-scope items removed
- A representative first batch, with the hard cases added
- Agreed formats, stable file names, a manifest
- A reference set labeled by your own experts, held back
- The labeling guideline: see how to write data annotation guidelines
- Transfer, storage, access and deletion agreed in writing
Where Pripton fits
Pripton Innovations asks for de-identified data, and its people work under non-disclosure agreements and data protection protocols. Tell us what your data is and what may and may not leave your systems through the contact page. A small first batch is the usual way to start: see how to run a data annotation pilot.
