“Better planning” is an understandable research goal, but it is not yet a task. A task needs a situation, a goal, available information, permitted choices, and conditions under which the work is assessed. Each of those decisions changes what a model can demonstrate.
A well-scoped environment helps a team distinguish the intended capability from incidental success. Without that scope, a strong final result may reflect a shortcut, hidden information, or a favorable accident. The episode needs enough structure to make the relevant decisions visible.
This article follows an illustrative resource-planning scenario. It is a design exercise rather than a specification for a particular platform implementation. The same questions can be adapted to other domains, including reasoning and software work.
1. Name an observable capability.
Start by translating the broad objective into behavior a reviewer can identify. In the planning example, the capability might be choosing a feasible sequence of actions while respecting a fixed resource limit and revising assumptions when new evidence arrives. That is more specific than producing an impressive-looking plan.
Write a short description of both a successful decision and a tempting failure. Success might involve checking an uncertain prerequisite before committing resources. Failure might involve selecting a high-value action that cannot be completed within the limit. These examples clarify what the task should expose.
Ask whether the capability can actually be observed in the proposed task. If all candidate plans are equally feasible, the task cannot reveal constraint handling. If no new evidence appears, it cannot reveal adaptation. Task difficulty should come from the intended problem, rather than confusing instructions.
2. Set the environment boundary.
Define where the episode begins and what belongs inside it. The initial state might include a goal, a resource budget, several candidate activities, and incomplete information about one dependency. The model should know which parts are fixed and which can change.
Keep external consequences separate from the exercise. A conceptual planning task can operate on supplied materials without authorizing real-world actions. If a program requires tools, external systems, or sensitive data, those requirements need explicit scoping and an appropriate agreement rather than assumptions embedded in a sample brief.
Describe the boundary from the reviewer’s perspective too. Can the expert see the whole episode, or only a final submission? Is the reference material complete? An assessment that depends on invisible state will be difficult to reproduce or explain.
3. Define observations and actions.
An observation is the information available to the model at a particular point. An action is something it is allowed to do in response. In the planning example, actions might include inspecting a dependency, selecting an activity, revising a plan, or submitting a final proposal.
Each action needs a clear meaning. “Ask for more information” is underspecified unless the task defines what can be asked, how a response is supplied, and whether there is a cost or limit. Otherwise, the model may be rewarded for behavior that cannot be compared across episodes.
Record information timing. A choice that looks poor after the outcome may have been reasonable before a missing dependency was revealed. Expert review should use the evidence available when the choice was made, while separately considering whether the model should have sought more information.
Avoid leaking the answer through incidental formatting. If feasible plans always share a label, length, or position, the task may reward pattern recognition unrelated to planning. Vary superficial presentation while preserving the underlying decision structure.
4. Decide when the episode ends.
Termination is part of the task. An episode might end when a final plan is submitted, when a defined action limit is reached, or when a rule violation makes further progress invalid. These outcomes should remain distinguishable rather than being combined into an unexplained failure label.
Specify what an incomplete episode means. Did the model run out of permitted steps, fail to identify a valid action, or encounter a task that lacked enough information? The answer affects interpretation. A refusal to invent missing information can be better judgment than an unsupported final plan.
Outcome and process can also diverge. A model may submit a feasible plan after making an unjustified assumption that happened to be correct. Preserve that distinction in review so an accidental success does not erase the behavior the program was intended to study.
5. Design for expert review.
Define review criteria before collecting a large set of episodes. For the planning example, useful dimensions could include feasibility, evidence use, response to uncertainty, and adherence to the action boundary. Each dimension should refer to observable behavior.
Use domain expertise where it matters. A specialist may recognize that a prerequisite is not merely inconvenient but logically necessary, or that a proposed sequence is impossible under the stated conditions. The task should supply enough context for that judgment and invite an explanation of consequential errors.
Keep the review output focused. An assessment, a reference to the decisive step, and a short rationale may be more useful than a long unstructured essay. Provide a way to mark ambiguous or invalid tasks so review does not silently convert missing context into confident judgments.
Read What makes expert feedback useful? for a deeper discussion of evidence, formats, and disagreement. The key connection is that a review format should answer the question the environment was designed to expose.
6. Look for shortcuts before optimizing a signal.
Try to satisfy the task’s apparent success condition without demonstrating the capability. Could a model repeatedly choose the first feasible action? Could it copy language from a reference? Could it earn a favorable assessment by sounding cautious while ignoring a constraint?
Construct counterexamples that reveal these routes. In the planning exercise, include a superficially attractive option that violates the resource limit, a valid plan expressed briefly, and a case where requesting clarification is the sensible next step. Avoid making every difficult case follow the same pattern.
Do not confuse variety with coverage. Many differently worded tasks can still test the same narrow decision. Group examples by the reasoning demand they impose, then inspect whether important failure modes are represented. That makes the task set easier to interpret when a model’s behavior changes.
7. Pilot, revise, and separate evaluation.
Run a focused initial review and inspect episodes individually. Look for missing observations, inconsistent action meanings, unclear endings, and criteria that experts interpret differently. Revise the task before expanding a pattern that does not reveal what you intended.
Keep versions understandable. A change to the resource limit, action set, or rubric can alter the meaning of success. Distinguish examples collected under different conditions, and avoid treating changes in the task mix as direct evidence of changes in model capability.
- Can the target capability be observed in this environment?
- Are the initial state and information boundary explicit?
- Does every action have a stable meaning?
- Are completed, incomplete, and invalid episodes distinguishable?
- Can an expert justify the review using the episode record?
- Have obvious shortcuts and misleading proxies been tested?
- Is evaluation material kept separate from signal development?
The result is not a guarantee that a task measures everything you care about. It is a clearer basis for learning what it does measure, where it fails, and how to connect expert review to a defensible reward signal.