gpt-5.6-sol

Script completed
Wall time 8.1 seconds
Output:

Script running with cell ID 143
Wall time 11.0 seconds
Output:

Script running with cell ID 143
Wall time 11.0 seconds
Output:

Script completed
Wall time 0.0 seconds
Output:

Script completed
Wall time 0.1 seconds
Output:
vllm:num_requests_running{engine="0",model_name="node-1b"} 15.0
vllm:num_requests_waiting{engine="0",model_name="node-1b"} 0.0
vllm:generation_tokens_total{engine="0",model_name="node-1b"} 4.436038e+06
2755

Script completed
Wall time 0.1 seconds
Output:
"""CARE: cost-aware counterfactual racing for language-agent graphs.

This replaces graph-level REINFORCE with a small, sample-efficient search over
node prompts, heterogeneous node kinds, and topology.  It uses successive
halving on whole architectures, then paired leave-one-edge-out tests on the
winner.  Accuracy and call cost are recorded per problem, so an edge is kept
only when its empirical credit exceeds its cost.  The shipped graph.json is the
champion from a longer run; this program can reproduce the search protocol.

Usage: python3 optimize.py [--stage1 6] [--stage2 16] [--seed 2026]
"""

from __future__ import annotations

import argparse
import math
import random
from dataclasses import dataclass
from pathlib import Path

from swarm import BudgetMeter, Node, Swarm, load_problems

TRAIN = "/app/data/train.jsonl"
VAL = "/app/data/val.jsonl"
REF_CALLS = 4.0

ANALYTIC = """You are the primary expert AIME solver. Solve the problem independently and exactly. Identify the governing structure, handle all cases, and verify the result by substitution or a second calculation. Do not guess. After private reasoning, BEGIN the visible response with <answer>N</answer> for N in 0..999, then give a compact checkable derivation; do not repeat exploratory work.

Problem: {problem}{context}"""

SKEPTIC = """Independently solve this AIME problem as a skeptical olympiad contestant. Seek a different representation or method from the obvious approach; check boundary cases, reversibility, multiplicities, and arithmetic. Trust no proposed answer unless verified. After private reasoning, BEGIN the visible response with <answer>N</answer>, then give a compact checkable certificate without repeating exploration.

Problem: {problem}{context}"""

COMPUTE = """Solve this AIME problem using an exact computational or symbolic formulation whenever possible. Translate every condition faithfully; prefer exhaustive integer/rational computation to intuition and verify the model. After private reasoning, BEGIN the visible response with <answer>N</answer>, then a compact justification. The LAST fenced Python code block in the entire response must be a self-contained standard-library program needing no input, terminating quickly, and printing only N; put no code block after it.

Problem: {problem}{context}"""

REVIEW = """Act as the senior AIME referee. Re-solve the problem rigorously while auditing all proposed analyses and the executed result below. Inputs may be wrong. Find the first concrete error behind any disagreement; verify every condition, boundary case, multiplicity, and arithmetic step. After private reasoning, BEGIN the visible response with <answer>N</answer>, then give a compact decisive certificate without repeating exploration.

Problem: {problem}{context}"""

FAST_DECIDE = """/no_think
You are the final AIME evidence adjudicator. Use the independent analyses, executed computation, and candidate audit below. Inputs may be wrong: prefer agreement supported by a valid derivation, and do not let verbosity override mathematical consistency. Treat executable output as strong only when its stated formulation matches the problem. Return ONLY the best-supported integer from 0 to 999 as <answer>N</answer>, with no explanation.

Problem: {problem}{context}"""

RELAY = """/no_think
Copy the referee_extract node's reported candidate. Return ONLY that integer as <answer>N</answer>.

Problem: {problem}{context}"""

DEEP_DECIDE = """You are the final AIME adjudicator. Independently check the problem and the evidence below. Resolve disagreement by mathematical validity, not majority or verbosity; verify that every condition was modeled. Return only <answer>N</answer> for the correct integer from 0 to 999.

Problem: {problem}{context}"""


@dataclass(frozen=True)
class Trial:
    name: str
    graph: Swarm


@dataclass
class Result:
    correct: list[int]
    calls: list[int]

    @property
    def accuracy(self) -> float:
        return sum(self.correct) / max(1, len(self.correct))

    @property
    def avg_calls(self) -> float:
        return sum(self.calls) / max(1, len(self.calls))

    @property
    def utility(self) -> float:
        # Smooth proxy for the grader's efficiency multiplier.  The exponent is
        # deliberately mild: a real accuracy gain can pay for one extra call.
        efficiency = (REF_CALLS / max(1.0, self.avg_calls)) ** 0.35
        return self.accuracy * efficiency


def parallel_four() -> Swarm:
    """Three diverse proposals, free execution/audit, fast fourth-call vote."""
    nodes = [
        Node("analytic_solver", ANALYTIC),
        Node("skeptical_solver", SKEPTIC),
        Node("computational_solver", COMPUTE),
        Node("code_result", "", "code_exec"),
        Node("candidate_audit", "", "symbolic_verify"),
        Node("decision", DEEP_DECIDE),
    ]
    edges = [(-1, 0), (-1, 1), (-1, 2), (2, 3),
             (0, 4), (1, 4), (2, 4), (3, 4),
             (0, 5), (1, 5), (2, 5), (3, 5), (4, 5)]
    return Swarm(nodes, edges)


def cascade_four() -> Swarm:
    """Two proposals followed by a thinking referee and a non-thinking relay."""
    nodes = [
        Node("analytic_solver", ANALYTIC),
        Node("computational_solver", COMPUTE),
        Node("code_result", "", "code_exec"),
        Node("candidate_audit", "", "symbolic_verify"),
        Node("referee", REVIEW),
        Node("referee_extract", "", "symbolic_verify"),
        Node("decision", RELAY),
    ]
    edges = [(-1, 0), (-1, 1), (1, 2),
             (0, 3), (1, 3), (2, 3),
             (0, 4), (1, 4), (2, 4), (3, 4),
             (4, 5), (5, 6)]
    return Swarm(nodes, edges)


def deep_five() -> Swarm:
    """Three proposals plus a thinking referee; costs one call over reference."""
    nodes = [
        Node("analytic_solver", ANALYTIC),
        Node("skeptical_solver", SKEPTIC),
        Node("computational_solver", COMPUTE),
        Node("code_result", "", "code_exec"),
        Node("candidate_audit", "", "symbolic_verify"),
        Node("referee", REVIEW),
        Node("referee_extract", "", "symbolic_verify"),
        Node("decision", RELAY),
    ]
    edges = [(-1, 0), (-1, 1), (-1, 2), (2, 3),
             (0, 4), (1, 4), (2, 4), (3, 4),
             (0, 5), (1, 5), (2, 5), (3, 5), (4, 5),
             (5, 6), (6, 7)]
    return Swarm(nodes, edges)


def lean_three() -> Swarm:
    """Efficiency arm: two proposals, free execution/audit, one decision."""
    nodes = [
        Node("analytic_solver", ANALYTIC),
        Node("computational_solver", COMPUTE),
        Node("code_result", "", "code_exec"),
        Node("candidate_audit", "", "symbolic_verify"),
        Node("decision", DEEP_DECIDE),
    ]
    edges = [(-1, 0), (-1, 1), (1, 2),
             (0, 3), (1, 3), (2, 3),
             (0, 4), (1, 4), (2, 4), (3, 4)]
    return Swarm(nodes, edges)


def one_shot() -> Swarm:
    # A serious gate calibration, not a weak baseline.
    prompt = """You are an expert AIME contestant. Solve the problem independently and exactly. Think deeply, handle all cases and constraints, and check the result with a second calculation or sanity check before committing. Do not guess. End with exactly <answer>N</answer>, where N is the required integer from 0 to 999.

Problem: {problem}"""
    return Swarm([Node("decision", prompt)], [])


def run_trial(sw: Swarm, problems: list[dict]) -> Result:
    """Keep the per-instance vector: paired deltas are the credit signal."""
    from concurrent.futures import ThreadPoolExecutor

    sw.validate()
    def one(p: dict) -> tuple[int, int]:
        meter = BudgetMeter()
        try:
            pred = sw.run(p["problem"], meter)
            ok = int(pred == int(p["answer"]))
        except Exception:  # a failed problem is zero, never a failed search
            ok = 0
        return ok, meter.calls

    with ThreadPoolExecutor(max_workers=min(8, max(1, len(problems)))) as ex:
        rows = list(ex.map(one, problems))
    return Result([r[0] for r in rows], [r[1] for r in rows])


def active_llms(sw: Swarm) -> int:
    return sum(sw.nodes[i].kind == "llm" for i in sw._active_nodes())


def edge_ablations(sw: Swarm) -> list[Swarm]:
    """Legal counterfactuals, ordered to test redundant context edges first."""
    out: list[Swarm] = []
    for edge in reversed(sw.active_edges()):
        candidate = Swarm(sw.nodes[:], [e for e in sw.edges if e != edge])
        try:
            candidate.validate()
        except ValueError:
            continue
        if active_llms(candidate) >= 3 and candidate.active_edges() != sw.active_edges():
            out.append(candidate)
    return out


def problem_order(seed: int) -> list[dict]:
    """Interleave train and held-out years instead of tuning only one sitting."""
    rng = random.Random(seed)
    train = load_problems(TRAIN)
    val = load_problems(VAL)
    rng.shuffle(train)
    rng.shuffle(val)
    mixed: list[dict] = []
    while train or val:
        if train:
            mixed.append(train.pop())
        if val:
            mixed.append(val.pop())
    return mixed


def describe(name: str, r: Result) -> None:
    print(f"{name:18s} acc={r.accuracy:.3f} calls={r.avg_calls:.2f} "
          f"utility={r.utility:.3f}", flush=True)


def main() -> None:
    ap = argparse.ArgumentParser()
    ap.add_argument("--stage1", type=int, default=6,
                    help="problems per architecture in the racing round")
    ap.add_argument("--stage2", type=int, default=16,
                    help="problems for finalists and edge counterfactuals")
    ap.add_argument("--ablations", type=int, default=3,
                    help="maximum leave-one-edge-out counterfactuals")
    ap.add_argument("--seed", type=int, default=2026)
    ap.add_argument("--out", default=str(Path(__file__).with_name("graph.json")))
    args = ap.parse_args()

    ordered = problem_order(args.seed)
    gate_set = ordered[:max(args.stage1, 1)]
    final_set = ordered[:max(args.stage2, args.stage1)]

    gate = run_trial(one_shot(), gate_set)
    describe("one-shot gate", gate)

    trials = [Trial("parallel-four", parallel_four()),
              Trial("cascade-four", cascade_four()),
              Trial("deep-five", deep_five()),
              Trial("lean-three", lean_three())]

    # Successive halving: spend the larger sample only on the two best arms.
    first: list[tuple[float, Trial]] = []
    for t in trials:
        r = run_trial(t.graph, gate_set)
        describe(t.name + "/r1", r)
        # Hard-gate awareness: a candidate that fails to beat one shot is not a
        # cheap winner; it is ineligible even if its cost proxy looks good.
        score = r.utility if r.accuracy > gate.accuracy else -1.0 + r.utility
        first.append((score, t))
    finalists = [t for _, t in sorted(first, key=lambda x: x[0], reverse=True)[:2]]

    ranked: list[tuple[float, Trial, Result]] = []
    for t in finalists:
        r = run_trial(t.graph, final_set)
        describe(t.name + "/r2", r)
        ranked.append((r.utility, t, r))
    _, winner, winner_result = max(ranked, key=lambda x: x[0])

    # Paired component credit.  An ablation wins only when its accuracy loss is
    # no larger than one problem and its cost-aware utility is higher.  The
    # conservative one-problem tolerance avoids pruning on pure sampling noise.
    for k, ablated in enumerate(edge_ablations(winner.graph)[:args.ablations]):
        r = run_trial(ablated, final_set)
        describe(f"ablation-{k}", r)
        paired_loss = sum(a - b for a, b in
                          zip(winner_result.correct, r.correct))
        if paired_loss <= 1 and r.utility > winner_result.utility:
            winner = Trial(winner.name + f"-abl{k}", ablated)
            winner_result = r

    # Fall back to the robust calibrated seed if racing has not cleared the
    # measured one-shot gate.  This prevents a tiny unlucky batch from emitting
    # an effectively single-call-quality graph.
    if winner_result.accuracy <= gate.accuracy:
        winner = Trial("calibrated-parallel-four", parallel_four())

    winner.graph.validate()
    winner.graph.save(args.out)
    print(f"saved {args.out}: {winner.name}, "
          f"{active_llms(winner.graph)} active LLM nodes, "
          f"{len(winner.graph.active_edges())} active edges")


if __name__ == "__main__":
    main()

GRAPH
{
 "nodes": [
  {
   "name": "analytic_solver",
   "template": "You are the primary expert AIME solver. Solve the problem independently and exactly. Identify the governing structure, handle all cases, and verify the result by substitution or a second calculation. Do not guess. After private reasoning, BEGIN the visible response with <answer>N</answer> for N in 0..999, then give a compact checkable derivation; do not repeat exploratory work.\n\nProblem: {problem}{context}",
   "kind": "llm"
  },
  {
   "name": "skeptical_solver",
   "template": "Independently solve this AIME problem as a skeptical olympiad contestant. Seek a different representation or method from the obvious approach; check boundary cases, reversibility, multiplicities, and arithmetic. Trust no proposed answer unless verified. After private reasoning, BEGIN the visible response with <answer>N</answer>, then give a compact checkable certificate without repeating exploration.\n\nProblem: {problem}{context}",
   "kind": "llm"
  },
  {
   "name": "computational_solver",
   "template": "Solve this AIME problem using an exact computational or symbolic formulation whenever possible. Translate every condition faithfully; prefer exhaustive integer/rational computation to intuition and verify the model. After private reasoning, BEGIN the visible response with <answer>N</answer>, then a compact justification. The LAST fenced Python code block in the entire response must be a self-contained standard-library program needing no input, terminating quickly, and printing only N; put no code block after it.\n\nProblem: {problem}{context}",
   "kind": "llm"
  },
  {
   "name": "code_result",
   "template": "",
   "kind": "code_exec"
  },
  {
   "name": "candidate_audit",
   "template": "",
   "kind": "symbolic_verify"
  },
  {
   "name": "decision",
   "template": "You are the final AIME adjudicator. Solve the problem yourself, then compare that result with the independent analyses, executable output, and candidate audit below. Inputs may be wrong. Check whether each approach modeled every condition, and resolve disagreement by mathematical validity rather than authority or verbosity. Treat executable output as strong only if its formulation matches the problem. Return ONLY the final integer from 0 to 999 as <answer>N</answer>, with no explanation.\n\nProblem: {problem}{context}",
   "kind": "llm"
  }
 ],
 "edges": [
  [-1, 0],
  [-1, 1],
  [-1, 2],
  [2, 3],
  [0, 4],
  [1, 4],
  [2, 4],
  [3, 4],
  [0, 5],
  [1, 5],
  [2, 5],
  [3, 5],
  [4, 5]
 ]
}

Script completed
Wall time 0.0 seconds
Output:
{}

Script running with cell ID 147
Wall time 11.0 seconds
Output:

Script completed
Wall time 5.6 seconds
Output:

Script running with cell ID 148
Wall time 11.0 seconds
Output:

Script completed
Wall time 6.7 seconds
Output:

Script running with cell ID 149
Wall time 11.0 seconds
Output:

Script completed
Wall time 5.8 seconds
Output:

Script completed
Wall time 0.1 seconds
Output:
vllm:num_requests_running{engine="0",model_name="node-1b"} 5.0
vllm:num_requests_waiting{engine="0",model_name="node-1b"} 0.0
vllm:generation_tokens_total{engine="0",model_name="node-1b"} 4.563708e+06
2595

Script running with cell ID 151
Wall time 11.0 seconds
Output:

Script running with cell ID 151
Wall time 11.0 seconds
Output:

Script completed
Wall time 0.0 seconds
Output:

Script running with cell ID 152
Wall time 11.0 seconds
Output:

Script completed
Wall time 0.0 seconds
Output:
{
 "accuracy_pct": 60.0,
 "n": 15,
 "correct": 9,
 "budget_violations": 0,
 "avg_calls": 4.0,
 "avg_completion_tokens": 43549.6
}

Script completed
Wall time 1.2 seconds
Output:
SESSION_ID=88724

Script completed
Wall time 0.2 seconds
Output:
same_nodes True
same_edges True
active_nodes [0, 1, 2, 3, 4, 5]
active_llm 4
active_edges 13
prompt_max 546
prompt_total 1908
2485

Script running with cell ID 155
Wall time 11.0 seconds
Output:

Script completed
Wall time 4.1 seconds
Output:

Script running with cell ID 156
Wall time 11.0 seconds
Output: