Preference data for LLMs is a set of human comparisons between two or more model responses to the same prompt, where annotators indicate which response is better (and sometimes by how much). It is the training signal behind RLHF and DPO: the data that teaches a model not just to predict text, but to produce answers humans actually prefer. The quality of your model is gated by the quality of these comparisons, which is why most teams collect them through trained annotators or a managed annotation pod rather than untrained crowdsourcing. Managed pods start from around £6,000 per month with a single UK contract.
Below we explain what preference data and pairwise comparison are, their role in RLHF and DPO, how to collect data that actually improves a model, and what annotator skill the work demands.
What is preference data?
Definition (preference data): Preference data consists of human comparisons between two or more model outputs for the same prompt, where annotators indicate which response is better, and sometimes by how much. These judgments are the training signal for a reward model or for direct preference optimisation.
A base language model predicts the next token well, but it has no inherent sense of what makes one answer better than another. Preference data supplies that sense. Instead of asking a person to write the perfect answer from scratch (expensive and slow), you show them candidate responses the model already produced and ask which is better. Aggregated across thousands of prompts, those choices become a precise signal for "what humans want".
The approach was popularised by OpenAI's InstructGPT work; see OpenAI's research on aligning language models to follow instructions for the foundational idea that consistent human preference judgments can steer a model toward better behaviour.
What is pairwise comparison?
Pairwise comparison is the most common way to collect preference data. The annotator sees one prompt and two responses (A and B) and chooses the better one. Sometimes they also rate the strength of the preference, or flag a tie.
Why pairs rather than absolute scores? Because people are far more reliable at relative judgments than absolute ones. Asking "rate this answer 1 to 10" produces noisy, drifting numbers; asking "is A better than B?" produces stable, comparable signal. Pairwise data can then be aggregated into a ranking even when each annotator only ever sees two options at a time.
Common preference-collection formats:
- Binary pairwise: choose A or B. Simple, fast, the workhorse format.
- Pairwise with margin: choose A or B and rate how much better (slightly, clearly, much). Adds signal at some cost in speed.
- Listwise ranking: order three or more responses best to worst. Richer, but harder to keep consistent.
- Pairwise with rationale: choose and briefly justify. Slower, but the rationale is gold for refining the rubric and catching disagreement.
The role of preference data in RLHF and DPO
Preference data is the bridge between human judgment and model optimisation. It sits at the core of two related training approaches.
| Stage | What preference data does | Method |
|---|---|---|
| Reward modelling | Trains a model to predict a human-preference score for any response | RLHF (reward model + PPO) |
| Direct optimisation | Optimises the model directly on preference pairs, no separate reward model | DPO |
| Evaluation | Provides ground-truth pairs to measure whether a new model is preferred | Human eval |
In classic RLHF, pairwise comparisons train a reward model, which then guides reinforcement-learning optimisation (PPO). The reward model is a learned proxy for human judgment. In DPO, described in the DPO paper on arXiv, you skip the separate reward model and PPO loop and optimise the language model directly on the preference pairs, which is simpler to run. Either way, the human comparisons are the irreducible input. For a full walkthrough of how these stages connect, see what RLHF is and who provides RLHF teams.
How do you collect high-quality preference data?
The mechanics of showing two responses and recording a click are trivial. Getting comparisons that actually improve a model is not. Quality comes from process, not volume.
The controls that matter:
- A clear rubric. Define what "better" means for your model: more accurate, more helpful, safer, better formatted, and how to weigh them when they conflict. Without this, every annotator invents their own criteria.
- Calibration before live work. Annotators practise on gold examples with known answers until their judgments line up. This is OSCABE's "Trained First" model: the pod is trained on your rubric before it produces a single live label.
- Inter-annotator agreement. Measure how often annotators agree on the same pair. Low agreement signals an unclear rubric or insufficient training, not just "hard prompts".
- Adjudication of disagreements. A senior reviewer resolves conflicts and feeds the resolution back into the rubric, so the standard sharpens over time.
- A stable team. The same annotators over months, so calibration compounds instead of resetting with each new freelancer.
- Tie and "both bad" handling. Forcing a choice on genuine ties injects noise. Let annotators flag ties and cases where neither response is acceptable.
Skipping these is the most common reason a preference-data programme produces a reward model that rewards the wrong things. For the broader annotation discipline this sits inside, see our guide to data annotation and labelling for AI.
What annotator skill does preference data require?
Preference annotation is often mislabelled as low-skill clicking. For general-purpose models, strong generalist annotators handle tone, helpfulness and format well. But the moment your prompts touch a specialist domain, the skill bar rises sharply, because the better answer is only obvious to someone who knows the field.
| Prompt type | Risk if annotator is unskilled | Suitable annotator |
|---|---|---|
| General chat, tone, format | Low | Trained generalist annotators |
| Coding completions | Picks code that looks right but is wrong/insecure | CE-verified engineers |
| Legal / finance reasoning | Rewards confident but incorrect answers | Lawyers, ICAI chartered accountants |
| Medical / clinical | Rewards plausibly dangerous answers | Clinicians, medical professionals |
| Multilingual responses | Misjudges fluency or meaning | Native-speaker linguists |
The failure mode is the same across domains: if the annotator cannot tell a subtly wrong answer from a correct one, your reward model learns to prefer the wrong answer, and that error gets baked into the trained model. This is why domain fit matters as much as rubric clarity, the same principle behind hiring domain experts for AI model evaluation.
How OSCABE staffs preference-data pods
OSCABE's AI Training Teams include managed pods built for preference and comparison work. A dedicated team is recruited, trained on your rubric, and run for you under one contract, so the comparisons behind your reward model stay consistent. The talent pool spans India and the Middle East and includes:
- Trained, calibrated generalist annotators for high-volume pairwise work
- CE-verified engineers for coding preference data
- ICAI chartered accountants and IIT/NIT-trained experts for domain reasoning
- Native-speaker linguists for multilingual comparison
Pricing is a transparent monthly fee:
| OSCABE managed pod | From (per month) | Focus |
|---|---|---|
| Coding RLHF Team | £6,000 | Coding preference data and review |
| Training Data Pipeline Team | £8,000 | Annotation and preference pipelines |
| Domain Expert AI Team | £9,000 | Legal, medical, finance, STEM preference data |
| RLHF Evaluation Team | £10,000 | Preference data, model eval, red-teaming |
That is roughly 75 to 80% below the effective cost of sourcing equivalent expert hours on per-hour gig platforms, with management and a UK contract included. Because the same dedicated people work your project month after month, calibration compounds rather than resetting. The economics of building this yourself are covered in the true cost of an AI training data team. See also how it works and pricing.
Frequently asked questions
What is preference data in LLM training?
Preference data is a collection of human comparisons between model responses to the same prompt, where annotators indicate which response is better. It is the training signal that teaches a model to produce answers humans prefer, used to train a reward model in RLHF or to optimise the model directly in DPO. The quality of your model depends heavily on the quality and consistency of these comparisons.
Why use pairwise comparison instead of rating each answer?
People are far more reliable at relative judgments ("is A better than B?") than absolute ones ("rate this 1 to 10"). Absolute scores drift between annotators and over time; pairwise choices stay stable and comparable. Pairwise data can still be aggregated into a full ranking, so you get reliable ordering without the noise of absolute scoring.
Do I need domain experts to collect preference data?
It depends on the prompts. General chat, tone and formatting can be judged by trained generalist annotators. But coding, legal, medical, finance and multilingual prompts need annotators qualified in the domain, because only they can reliably tell a subtly wrong answer from a correct one. OSCABE staffs both generalist and domain-expert pods for exactly this reason.
How much does preference-data annotation cost?
On per-hour platforms, skilled comparison work is expensive once you add management and the cost of rework from inconsistent labels. OSCABE managed pods start from £6,000 per month for a Coding RLHF Team and £10,000 per month for an RLHF Evaluation Team, roughly 75 to 80% cheaper than gig-platform equivalents, with management and a UK contract included. See pricing for current figures.
Put consistent judgment behind your reward model
Preference data is the most leveraged data in your pipeline: every comparison either sharpens your reward signal or quietly corrupts it. Volume without calibration produces a model that confidently prefers the wrong answer; calibrated comparisons by people who understand your rubric and domain produce a model people actually prefer.
To put trained, consistent annotators behind your preference data without the cost and overhead of building the operation yourself, explore OSCABE's AI Training Teams or contact us. We will scope a managed pod trained on your rubric, with transparent monthly pricing and a UK contract, so the comparisons that shape your model stay consistent from the first label.