Definition
Preference data records which of two or more model responses a person judged better, and often why. It is used to train models to follow instructions and match human judgment, most famously through reinforcement learning from human feedback (RLHF).
Why it matters when licensing company data
Most preference data is produced on demand by paid reviewers and domain experts rather than licensed from existing records. Company archives can still feed it: resolved tickets, approved and rejected drafts, and code review comments all contain human judgments about better and worse answers.
Source
Ouyang et al., Training language models to follow instructions with human feedback (arXiv:2203.02155). Definition reviewed November 7, 2026 by Origin Data Partners.