For AI labs
Build the signals
your models need.
Connect the capability you want to study with the expertise, tasks, and feedback that can make progress observable.
Start a program conversationBegin with a research question.
AI labs use Datlore to source high-quality human feedback and reinforcement-learning episodes at scale. Vetted domain experts, task environments, and reward-model tooling sit within one platform, connecting parts of the training process that need a shared understanding of the objective.
A productive starting question is concrete: can the model detect an unsupported scientific claim, repair a reasoning error, or choose a defensible next action under uncertainty? Those are different capabilities, even if each is described broadly as reasoning. The task must expose the distinction you care about.
Before deciding on volume or feedback format, identify what would count as useful evidence. A final answer may suffice for one question; another may require the full episode and an explanation of the decisive step. Clarifying that need helps the team avoid collecting data that is easy to count but difficult to interpret.
Illustrative program directions
Different questions.
Different review designs.
Deep response review
A team studying technical accuracy could ask domain experts to compare explanations against supplied evidence. The brief would separate correctness, completeness, and uncertainty, with rationales for consequential errors.
The key scoping question is which domain knowledge is required to notice those errors. The review should also allow the conclusion that neither response satisfies the task.
Plan expert feedbackSequential decision review
A team studying planning could define a bounded environment where information arrives across several steps. Expert review would assess whether actions followed from the evidence available at the time.
The key scoping question is how to distinguish a sound decision from a lucky outcome. That requires attention to observations, action boundaries, and termination conditions.
Explore episode anatomyThese are research-planning examples, not a list of preconfigured programs or guaranteed deliverables.
Before the first program brief
Four choices
that shape the data.
Program scope should establish the relationship between the brief, expert contribution, and evaluation.
- Expertise & evidence
- Describe what a qualified reviewer must recognize. Include the subject, task difficulty, permitted references, and any practical context. If the relevant evidence is unavailable to the reviewer, a more experienced expert cannot necessarily compensate for that gap.
- Output & interpretation
- Choose between comparisons, rubric assessments, corrections, or episode review based on the research decision. Define how uncertainty and disagreement should be represented. A forced preference can obscure a meaningful tie or a task that lacks enough context.
- Evaluation & separation
- Decide which examples will inform training and which will be reserved for evaluation. Inspect important task groups separately. A program should make it possible to ask whether apparent improvement reflects the intended capability rather than a surface feature shared by the collected examples.
- Data & responsibility
- Discuss what material is necessary, who may provide it, and any restrictions on use. Do not include confidential datasets or personal information in an initial website inquiry. Detailed program materials and service terms should be handled through an appropriate, separately agreed process.
Start narrow enough
to learn from the work.
A focused initial task set can expose ambiguity before it becomes systematic. Review the brief, example outputs, and rationales together. When experts disagree, ask whether the cause is missing context, a conflicting criterion, or a substantive difference in judgment. Each calls for a different response.
Keep changes interpretable. If a rubric is revised, distinguish feedback collected under the old instructions from feedback collected under the new ones. If tasks become harder, avoid interpreting a change in scores as a change in model quality without examining the task mix.
For a first conversation, bring the target capability, one representative task, an initial review criterion, and the question your current data does not answer. You do not need to arrive with a complete specification. A concrete example is often more informative than a long list of desired features.
Prepare a short program brief.
- What behavior are you studying, and why does it matter?
- What should an expert assess or contribute?
- Does the review concern an answer or an episode?
- What constraints and uncertainties should shape the discussion?