SteelEnv H3+H4 Leaderboard

Agent benchmark on the two hardest SteelEnv tiers. 3 frontier models × 2 tiers × 2 repeats, 32-call budget.

SteelEnv tests whether an agent can turn a fabrication shop drawing into a buildable CadQuery solid, graded against the shop’s NC1 file. The main-set tiers (L0–L5) are published elsewhere; this page reports the H3 and H4 hard tiers only. H1/H2 and L0–L5 numbers are pending and will be added to this page as they land.

Headline: on H3 and H4 every frontier model we tried scored 0% full credit and 0% builds-ok across both repeats. These two tiers are open headroom; no model in this batch cleared the gate. Numbers below are reported as measured.

H3 — 119 tasks

Each cell is 119 tasks × 2 repeats. pass@1 is the mean full-credit rate over the two repeats; pass@2 counts a task as solved if either repeat earned full credit (reward = 1). No cell cleared either threshold on H3.
model	pass@1	pass@2	builds-ok	avg API calls	avg seconds	$ / task
Claude Opus 5.5	0.0%	0.0%	0.0%	6.5	19.7	$0.0865
GPT-6 Astra	0.0%	0.0%	0.0%	3.8	14.6	$0.0185
Qwen3.8-27B	0.0%	0.0%	0.0%	25.9	250.5	$0.1146

H4 — 96 tasks

H4 has 99 tasks in the frozen list but only 96 rows per repeat here: 3 tasks are silently skipped by all three models because of a data-bundle gap (asset files missing from this build); they will return once the bundle is rebuilt. Reporting /96 rather than /99 avoids inflating or deflating any model.
model	pass@1	pass@2	builds-ok	avg API calls	avg seconds	$ / task
Claude Opus 5.5	0.0%	0.0%	0.0%	6.8	20.6	$0.0902
GPT-6 Astra	0.0%	0.0%	0.0%	3.9	15.2	$0.0192
Qwen3.8-27B	0.0%	0.0%	0.0%	24.7	234.4	$0.1036

Method

Scoring. A task earns full credit only when the submitted solid passes all seven SteelEnv gates (thickness, outline bbox, outline shape, hole count, hole diameters, hole positions, volume). pass@1 is the mean full-credit rate across the two repeats; pass@2 counts a task as solved if at least one of the two repeats earned full credit. builds-ok is the share of rollouts where the submitted CadQuery program produced a single valid solid (the gate before any check runs).
Budget. b=32 (up to 32 API tool calls per task), toolset v2, two independent repeats (r0, r1) per model × tier.
Cost. Mean input×p_in + output×p_out per task, using Claude Opus 5.5 at $15 / $75 per 1M tokens, GPT-6 Astra at $3 / $15 per 1M, and Qwen3.8-27B via OpenRouter at $0.60 / $3 per 1M. Reasoning / cached tokens are not re-priced here.
Why zeros. H3 and H4 ship as the headroom tiers for the next model generation; these models were allowed to try the batch end to end, and the result is the honest zero above. Finer-grained diagnostics (which gate they fail, where the reasoning breaks down) are tracked in the paper, not on this page.

Submit your model

To submit your model’s results, open a PR on github.com/deccanai-org/steelenv-examples. Include per-task JSONL rows in the same shape as the files behind this page (task_id, reward, gates.builds, turns, api_calls, seconds, usage).

← back to the SteelEnv examples · Generated 2026-10-05 from gs://mlproject-501205-gym-runs/eval/h3h4/.