Human Feedback for Generative AI: Rating, Ranking and Red-Teaming

In short: generative models learn from people in three ways. People write examples of the ideal answer. People judge the model's own answers by rating or ranking them. And people try to make the model fail, which is called red-teaming. Unlike classic labeling there is rarely one right answer, so the quality of the data depends on the rubric and on who the raters are.

How this differs from classic annotation

A box around a car is right or wrong. "Which of these two answers is more helpful?" is a judgement.

That changes the work in three ways:

• A rubric replaces the rule. Raters need a written scale with an example for every score.
• The rater matters. A question about tax law needs someone who knows tax law. A general annotator can only judge whether the answer reads well.
• Disagreement is expected. Two good raters will sometimes differ. The aim is to measure and narrow the difference, not to pretend it away.

The tasks

1. Writing prompts and ideal answers. People write the question and the answer the model should have given. Also called demonstrations.
2. Rating. One answer is scored on a scale for qualities such as helpfulness, accuracy, tone and safety.
3. Ranking and comparison. Two or more answers to the same prompt are put in order.
4. Fact-checking and grounding. The rater checks each claim in an answer against a source, and marks what is unsupported.
5. Safety review. Answers are checked against a policy: harmful advice, private data, bias.
6. Red-teaming. People deliberately look for prompts that make the model misbehave.
7. Building evaluation sets. A fixed set of prompts with agreed good answers, used to test every new version of the model.

Where the method comes from

The paper behind much of this practice describes it plainly. OpenAI's researchers wrote that "we collect a dataset of labeler demonstrations of the desired model behavior", used it to train the model, and then: "We then collect a dataset of rankings of model outputs, which we use to further fine-tune this supervised model using reinforcement learning from human feedback" (Ouyang et al., 2022).

Demonstrations first, rankings second. The people doing both are doing annotation, with judgement in place of a drawing tool.

Red-teaming has a definition too. The US National Institute of Standards and Technology calls it "a structured testing effort to find flaws and vulnerabilities in an AI system" (NIST glossary). The word to notice is structured: a list of what to probe, a record of every attempt, and a way to report what was found.

What makes the data good

• A rubric with examples. For each score, a sample answer and the reason it earns that score. "4 = good" is not a rubric.
• Calibration before production. Every rater scores the same small set first. Where they differ, the rubric is discussed and tightened.
• More than one rater per item, at least on a sample, with agreement measured. See how to measure data annotation quality.
• The right raters. Subject experts for expert subjects, native speakers for each language.
• The reason, not only the score. A one-line reason makes a rating checkable and far more useful for training.
• Varied prompts. A thousand versions of the same question teach the model one thing.

The rubric is this field's guideline. The same discipline applies: see how to write data annotation guidelines.

The risks to manage

• Tired or hurried raters drift towards the middle of the scale. Keep sessions short and watch the spread of scores.
• A leading rubric produces the answers its author expected. Have someone outside the team read it.
• Disturbing content. Safety review and red-teaming expose people to harmful material. They need the right to step away, rotation between tasks, and support.
• Private data in prompts. Real user conversations may hold names and details. Remove them before they reach raters: see how to prepare data for annotation.

What to ask before you outsource it

• Who are the raters, and how were they chosen for this subject and language?
• How are they calibrated, and how often is that repeated?
• How many raters see each item, and what happens when they disagree?
• How are raters protected when the content is harmful?

More in our list of questions to ask a data annotation vendor.

Where Pripton fits

Pripton Innovations provides prompt and response writing, response rating, fact checking and safety review, in more than 15 languages: see generative AI data services. To describe your model and the feedback it needs, use the contact page.