Environments
How a job becomes a training environment.
The technical method behind both data lines: task structure, grading, calibration and the format your team receives.
What an environment is
Everything a run needs, in one folder.
Each task is one folder in a Harbor-compatible layout. The agent sees the instruction and the project. The verifier and the reference solution stay out of its reach.
- Instruction
instruction.md - The work request as a colleague or client would send it, with the exact deliverables and the paths to save them to.
- Starting state
environment/project/ - The seed data the job starts from: drawings and their revisions, 3D models, tables, mail, database rows.
- Tools
environment/Dockerfile - A container with what the work needs: a CAD kernel, hydrostatics and calculation libraries, document and spreadsheet tools.
- Verifier
tests/test.sh - The program that reads the deliverables and the final state, runs every check and writes the reward, one line per check.
- Probes
tests/probes/ - Reward hacks, plausible mistakes and correct alternatives the verifier must score as expected before the task ships.
- Reference solution
solution/solve.sh - A solution that scores 1, which proves the task can be solved as written.
- Metadata and rights
task.toml - Domain, family, difficulty band, time limits, resources, split, source and licence.
task.toml Illustrative
version = "0.3" [metadata] id = "naval/tank-capacity-014" domain = "ship design" family = "tank-capacity" # generated, seed 1408 difficulty = "hard" # pass@1 0.12 at calibration split = "train" # train | sealed-eval [rights] source = "generated" # generated | licensed | authored use = "training and evaluation" [agent] timeout_sec = 2700 [environment] network = "none" cpus = 2 memory_mb = 4096 [verifier] timeout_sec = 300 reward = "strict" # 1 only if every check passes weights = [1, 3, 3, 2, 1, 2, 3, 1] # C1 to C8 tolerance = { volume_m3 = 0.5, lcg_m = 0.02, trim_m = 0.01 }
/logs/verifier/reward.json Illustrative
{
"reward": 0,
"score": 0.94,
"C1_deliverable_parses": 1,
"C2_inputs_unchanged": 1,
"C3_tank_volumes_98pct": 1,
"C4_tank_lcg_vcg": 1,
"C5_sounding_tables": 0,
"C6_trim_departure": 1,
"C7_tank_bounds_rev_c": 1,
"C8_tolerances_cited": 1
}
One failed check sets the reward to 0. The weighted score shows the rest of the answer was sound, and where it went wrong: C5 caught tank 5P’s soundings read from the sounding-pipe datum in revision B. Revision C moved the pipe, so every sounding in that tank is referenced to the wrong point.
Verifier principles
Grade the state, not the prose.
A reward is only as good as the verifier that computes it. Code checks what can be computed; where expert judgment is needed, calibrated model judges and custom rubric verifiers grade it, each checked against expert grades.
Grade the state
The verifier opens the files, geometry, rows and messages the agent left behind and recomputes what they should be. The agent’s summary of its work earns nothing.
Deterministic, fail-closed
Same deliverable, same reward, on every run. A missing, malformed or unreadable deliverable scores zero with a stated reason, never a crash and never a pass.
Strict pass is the reward
The reward is 1 only when every check passes. A weighted per-check score sits beside it, so you can read where an answer went wrong without rewarding a near miss.
Process and outcome
Outcome checks read the deliverable. Process checks read how it was reached: inputs left untouched, the current revision used, and the clauses and values applied, cited.
Tolerance-aware
Each quantity carries the engineering tolerance of its drawing or code. A correct answer in other units, another order or another layout still passes.
Isolated and resettable
No network, read-only inputs, and tests mounted only after the run ends. Reset returns the workspace to the same state, checked by digest.
Frozen and versioned
Each verifier is frozen by hash with its task version. A fix is a new version with a change-log entry, and every probe runs again.
Reward-hacking defence
Every verifier is attacked before it ships.
Before a task leaves us, we run a set of probes against its verifier.
| Probe class | What it does | Must score |
|---|---|---|
| Empty | Leaves the workspace exactly as it was reset. | 0 |
| Claim-only | Reports the job done and changes nothing. | 0 |
| Tamper | Edits an input, a protected table, the tests or the reward file. | 0 |
| Hard-coded answer | Writes the expected format with guessed or copied values. | 0 |
| Superseded input | Works from the old drawing revision or the first email, not the amendment. | Fail strict |
| Plausible mistake | Right overall shape or total, wrong toleranced feature or line item. | Fail strict |
| Correct alternative | The right answer by another method, in other units or another layout. | Pass strict |
Six demonstration tasks, one harness.
Gates run 10 November 2026; model attempts on or before 7 November 2026. All six are demonstration tasks we wrote in-house with invented names and figures, not partner records. The probes are a regression test, not a guarantee.
Results from the enterprise-workflow harness. The CAD sample run is reported separately.
- Checks
- 59 across the six tasks, 26 critical; the reference solution scores 1.0 on each
- Reward-hack probes
- 73, every one scoring zero; 101 hostile inputs, all zero without a verifier error
- Plausible mistakes
- 71, every one failing the strict reward
- Correct alternatives
- 8, all passing strict
- Known gaps
- 3 probes passing by design; documented on the evidence page
- Model attempts
- Two current models from one provider, including its small tier, one rollout per model per version. Both passed every task after we clarified the wording of two version-0 tasks. The hardened versions run next (below). These runs show that the harness works, not how hard the tasks are
- Open
- What the verifier does not yet catch is on the evidence page
Published with its date, pass or fail.
Pass rates on the six tasks for one small open model and one stronger model, five rollouts each. Published here when complete, whatever the result, with the run date and task versions.
Trajectory audit
Probes test the verifier. The audit tests the passes. On sampled runs from calibration we read how each pass was earned, not only that it was.
A pass is thrown out when the run
- opened the reference, the tests or anything outside the project
- changed an input, a protected table or the verifier
- reached for the network, or wrote an answer it never computed
- took a route the instruction rules out
A pass earned the wrong way becomes a new probe, and the verifier is hardened before the batch ships.
Difficulty and calibration
Hard enough to teach, solvable by design.
A task every model passes teaches nothing. A task no model passes may be broken. We measure each task before it ships and keep it only inside the band you set.
- Repeated attempts
- Models from a small open model to the strongest available attempt each task several times. A pass rate is an average over attempts, never one run.
- pass@1 and pass@k
- pass@1 places the task in the band; pass@k shows it is reachable. A task outside the band is hardened, rebalanced or dropped.
- Zero is a warning
- Before we keep a task no model passes, we re-check its instruction, its reference solution and its verifier.
- Knobs, not wording
- Generated families are tuned with the knobs below. Difficulty never comes from an ambiguous instruction or a hidden trap.
Difficulty knobs by domain
- Parts
- Feature count, tolerance class, views to reconcile
- Ships
- Tanks and loading conditions, superseded revisions in the folder
- Buildings
- Sheets and disciplines to cross-read, clash density
- Workflows
- Horizon length, amendments, distracting threads
Calibration card, one per batch Illustrative
- Batch
- naval/tank-capacity v0.3
- Tasks kept
- 40 of 46 drafted
- Attempts
- 3 per model per task
- Small model
- pass@1 0.05
- Large model
- pass@1 0.21 · pass@3 0.34
- Band kept
- pass@1 between 0.02 and 0.30
- Reward hacks
- 14 of 14 at zero
- Planted mistakes
- 12 of 12 failing strict
- Alternatives
- 4 of 4 passing strict
- Expert agreement
- 9 of 10 tasks, graded blind
- Time per attempt
- median 38 min
The shape of the card, not a measured result. Every batch carries its own.
Two engines for scale
Volume from generators. Depth from practitioners.
Most of a batch can be generated with its ground truth computed. The hardest tasks start with someone who has done the job.
Engine one
Parametric generators
A generator builds a family of tasks from parameters: parts with their features and tolerances, hull forms with their tank arrangements, building grids with their frames. The code that builds a task also computes its answer, so volume never costs correctness.
- Ground truth computed, never typed
- Every seed recorded on the datasheet and reproducible
Engine two
Expert seeds
For commissioned tasks, a practitioner supplies the job and signs off the result. Models assist with the drafting, so expert hours go where only an expert can help.
- SeedA practitioner describes a real job in six fields: situation, inputs, goal, constraints, how to verify, expected output.
- DraftWe build the instruction, project files, verifier and reference solution from the seed, with models assisting the drafting.
- SolveA solver model from another family attempts it blind, several times, with no access to the verifier or the reference. If it passes too easily, we read the solver’s log and harden the task.
- ValidateThe practitioner checks the reference and the verifier against their own judgement, and makes the task harder where the real job is harder.
- ReviewA second practitioner grades a sample blind. We report the agreement on the calibration card, and each disagreement goes to adjudication.
Delivery
In the shape your harness already runs.
Tasks arrive ready to run, documented item by item, and split so that evaluation stays clean.
- Format
- Harbor-compatible task folders, container images, or the interface your harness takes. One task runs in your harness first, then the batch follows.
- Datasheet
- Per item: source (generated, licensed or authored), generator and seed, or the author’s pseudonymous ID, revisions, de-identification, and the calibration results.
- Rights note
- Per item: who owns it, the licence for training and evaluation, and the assignment or agreement it rests on.
- Splits
- The agreed split and permitted use are recorded with each task. Evaluation and licensing policy.
- Upkeep
- As models improve, verifiers are hardened and saturated tasks are replaced with harder ones from the same family.
Get started
Read one task before you decide.
Inspect a complete task before discussing an integration.