claude-opus-5
agent-swarm-optimization — graded by the task's verifier against a measured baseline.
- backend
- modal
- variant
- claude claude-opus-5
- metric
- sealed AIME avg accuracy
- actions
- 163
- duration
- 5h 16m
- recorded
- 2026-08-05
run idswarm-aime-optimization__kfV24B9
Everything claude-opus-5 did on this task, in order.
download JSON ↗encrypted chain-of-thought — recorded, not exposed
args
{
"taskId": "4",
"status": "completed",
"description": "DONE. Frozen 3c/resolve-rf-alt: 5 nodes, 10 active edges, 3 active LLM nodes, all caps validated. Verified end-to-end with the grader's own selfcheck 6x: 57.8% (52/90) at 3.00 calls/problem, 0 budget violations. Per-edge attribution run on the final graph."
}TaskUpdate result
Updated task #4 description, status
[metadata] {"success": true, "taskId": "4", "updatedFields": ["description", "status"], "statusChange": {"from": "in_progress", "to": "completed"}}Done. Final state of the submission in /app/methods/main/:
The frozen graph (graph.json)
5 nodes, 10 active edges, 3 LLM calls/problem (cost reference is 4.0):
problem ─┬─> solver_pot2 (llm) ──> code_exec (free) ─┐
└─> solver_alt (llm) ─┐ │
├─> verify (free) ───┤
│ [alt, code, pot2]│
└────────────────────┴─> decision (llm)The decision node is a conditional re-solver, not an aggregator: if the free verifier reports unanimity it restates that integer and stops; otherwise it ignores every candidate and solves from scratch.
Verified results
| train (n=90) | val fresh (n=90) | |
|---|---|---|
| frozen graph | 0.822 | 0.578 @ 3.00 calls |
| best single LLM call | 0.633 | 0.467 |
Six independent selfcheck.py passes: 53.3 / 66.7 / 53.3 / 66.7 / 53.3 / 53.3, zero budget violations. The cached-replay val figure was 0.644, but the design was partly selected on it — 0.578 is the honest fresh-sample number and it's what the write-up reports.
What the optimizer does differently
Trace caching keyed by (draw, prompt) gave an 87% hit rate and made every comparison paired on identical completions — the whole aggregator design space is then enumerable offline at zero LLM cost, which is the part REINFORCE spends its entire sample budget learning. Credit assignment is paired leave-one-edge-out rather than one scalar smeared over 16 edges.
Three findings that drove the design
- Free nodes carry the most credit. The highest-attribution edge in the final graph is
pot2 → code_exec(+0.100) — a non-LLM edge.code_execreads raw predecessor text, so it recovers a program even from a<think>block truncated before any prose answer existed. - Edge order is a decision rule.
symbolic_verifybreaks ties by earliest predecessor andcode_execdouble-votes, so ordering the reasoner first makes two agreeing reasoners override the program while a lone one doesn't. Fixing this ordering was worth +0.016 for free. - Aggregation can't arbitrate. When candidates split, the tally is right 0–43% of the time; when they agree, 88–96%. So the mandatory final call is worth much more as a fresh attempt than as a copyist (+0.033).
Negative results are recorded too: prompt-level control of thinking length (terse) lost more conditional accuracy than it gained closure, and pot's apparent advantage over pot2 was an artifact of a biased cached subset.
One deliberate refusal
My own cost-aware objective prefers deleting a solver — dropping to 2 LLM calls raises the efficiency multiplier ~28% while costing only 0.067 accuracy, so Δobj = −0.193 for the edge feeding solver_alt. I did not take it. Gate 1 pays exactly 0 for using too few LLM nodes, and "more than a minimum number" is ambiguous enough that 2 nodes is a real risk; trading a hard gate for a multiplier is a bad bet at any plausible odds. Three LLM nodes is the leanest count that doesn't gamble on the gate, and the graph sits at that corner on purpose.