Human comparisons can provide a practical way to express judgments that are difficult to capture in a single rule. An expert can recognize that one explanation handles an assumption correctly or that one plan responds more sensibly to uncertainty. The comparison makes a distinction visible.
But a preferred response is not a complete definition of the desired behavior. It is a choice between particular options under a particular brief. Converting that choice into training information requires care about the assumptions, criteria, and alternatives that shaped it.
This article offers a conceptual framework rather than an algorithm or an implementation specification. Its examples are illustrative. The aim is to sharpen the questions a research team asks when connecting expert feedback to reward.
1. Establish what a preference means.
A preference is relational: response A was favored over response B in a stated context. That does not establish that A is correct, that it is acceptable, or that it would be preferred over every alternative. Two inadequate responses can still produce a decisive comparison.
Preserve the task with the judgment. A reviewer choosing the better summary for a specialist may prioritize details that a general audience does not need. A reviewer asked to assess source fidelity may reach a different conclusion from one asked to assess independent factual truth.
Also preserve the available evidence. If the reviewer lacked information necessary to assess a claim, the resulting preference should not be interpreted as verification of that claim. The meaning of the label depends on what the task allowed the expert to know and do.
2. Separate the criteria before combining them.
Responses can differ in correctness, completeness, clarity, constraint adherence, and handling of uncertainty. An overall preference combines those dimensions, sometimes implicitly. If a research team needs to understand what drives the choice, it should make the important dimensions explicit.
This does not mean every task needs a large rubric. Too many overlapping categories can make review cumbersome without improving interpretation. Select dimensions that can vary independently and that matter to the intended behavior. Define them with boundary cases rather than only positive examples.
When dimensions conflict, specify priorities. A concise response containing a serious error should not win merely because brevity is valued. If factual correctness is essential, say how an error affects the overall judgment. Do not leave experts to invent a different weighting scheme for each comparison.
3. Follow an illustrative comparison.
Imagine a task asking for a recommendation based on a supplied set of observations. Response A makes a clear recommendation but treats an uncertain observation as established. Response B identifies the uncertainty, proposes a conditional recommendation, and explains which missing fact would change the decision.
An expert prefers B because the task requires evidence-sensitive reasoning. The useful signal is not simply “conditional language is good.” B is preferred because its level of certainty matches the evidence and because its condition is relevant to the decision. A vague disclaimer appended to an unsupported recommendation would not demonstrate the same behavior.
Now include response C: it refuses to make any recommendation even though the evidence supports a bounded conclusion. If B is also preferred over C, the comparison helps separate calibrated uncertainty from blanket avoidance. Carefully chosen alternatives make the intended distinction easier to study.
The rationale should identify these reasons explicitly. It can point to the uncertain observation, the consequence of assuming it, and the condition that makes B’s recommendation defensible. That explanation provides a way to inspect whether later favorable assessments still track the intended criterion.
4. Represent ambiguity rather than erasing it.
Some comparisons are tied or underdetermined. Responses may differ only in a dimension outside the brief, or the brief may not define which tradeoff matters. Forcing a preference can add apparent precision while weakening the interpretation of the label.
Distinguish types of uncertainty. A tie means the available criteria do not favor one option. Insufficient evidence means a necessary fact is missing. Reviewer uncertainty may mean the task requires different expertise. These cases suggest different next steps and should not automatically become the same training example.
Disagreement deserves similar care. Read the explanations before combining judgments. If one expert values completeness and another values concision because the brief omitted the audience, the appropriate fix may be better instructions. If they disagree about a substantive domain question, further review may be more useful than a forced consensus.
5. Test the proxy for shortcuts.
A reward signal stands in for a richer objective. The danger is that favorable assessments can correlate with features that are easy to recognize but not sufficient for the desired behavior. Length, confidence, formatting, and familiar phrasing can all become misleading cues.
Create counterexamples deliberately. Compare a short, accurate response with a long response containing a consequential error. Contrast a correct unfamiliar method with a polished invalid one. In the uncertainty example, compare calibrated reasoning with both overconfidence and unnecessary refusal.
Inspect whether the signal rewards the right distinction across these cases. A model might learn to add hedging language without changing its reasoning, or produce a detailed explanation after making an unsupported choice. Favorable style should not substitute for evidence-sensitive decisions.
For an episode, also test whether a positive outcome conceals poor process. A plan can succeed because an unjustified assumption happened to hold. The review should identify the decision quality separately from the eventual result when that distinction matters to the research objective.
6. Evaluate beyond the average.
Set aside evaluation material that is not used to shape the signal. Repeatedly inspecting the same cases while making changes can narrow attention to familiar examples. New tasks help test whether the intended distinction survives changes in wording, context, and difficulty.
Break results down by meaningful task groups. An overall score can conceal a weakness in rare but important cases, such as recognizing insufficient evidence or handling an unusual constraint. Choose groups based on the decisions the task requires rather than superficial categories alone.
Review failures directly. A changed average does not explain which behavior improved or deteriorated. Read representative outputs and rationales, including cases where experts and the signal disagree. Those disagreements can reveal a flawed criterion, limited example coverage, or a surface feature that has become an unintended shortcut.
Keep task and rubric versions identifiable. If evaluation examples become harder or instructions change, a shift in assessments may reflect that change rather than model quality. Interpretation depends on knowing what stayed comparable.
7. Keep human judgment in the learning loop.
Expert review remains useful after initial collection. Specialists can propose consequential counterexamples, identify subtle proxy failures, and explain whether an apparent improvement corresponds to the intended capability. The role of human judgment is not exhausted when a label becomes a number.
- Can you state what each preference establishes and what it does not?
- Are the brief, evidence boundary, and important criteria preserved?
- Can reviewers express ties and distinct forms of uncertainty?
- Have alternatives been chosen to separate substance from style?
- Do counterexamples expose ways to earn approval without meeting the goal?
- Is evaluation separate and broken down by meaningful task demands?
- Can changes be traced to task, rubric, or example differences?
The objective is an interpretable relationship between expert judgment and the behavior being encouraged. That relationship is easier to maintain when feedback quality, task design, and evaluation are considered together.