gpt-5.6-sol

Script running with cell ID 180
Wall time 11.0 seconds
Output:

Script running with cell ID 180
Wall time 11.0 seconds
Output:

Script completed
Wall time 0.0 seconds
Output:

Script running with cell ID 181
Wall time 11.0 seconds
Output:

Script completed
Wall time 7.2 seconds
Output:

Script completed
Wall time 0.1 seconds
Output:
vllm:num_requests_running{engine="0",model_name="node-1b"} 4.0
vllm:num_requests_waiting{engine="0",model_name="node-1b"} 0.0
vllm:generation_tokens_total{engine="0",model_name="node-1b"} 5.729348e+06
1454

Script running with cell ID 183
Wall time 11.0 seconds
Output:

Script running with cell ID 183
Wall time 11.0 seconds
Output:

Script completed
Wall time 0.0 seconds
Output:

Script running with cell ID 184
Wall time 11.0 seconds
Output:

Script completed
Wall time 6.2 seconds
Output:
{
 "accuracy_pct": 53.33,
 "n": 15,
 "correct": 8,
 "budget_violations": 0,
 "avg_calls": 4.0,
 "avg_completion_tokens": 32612.8
}

Script completed
Wall time 0.0 seconds
Output:
{}

Script completed
Wall time 0.2 seconds
Output:
exact_optimizer_match True
nodes 7 active_nodes [0, 1, 2, 3, 4, 5, 6] active_llms 4
active_edges 12 [(-1, 0), (-1, 1), (1, 2), (0, 3), (1, 3), (2, 3), (0, 4), (1, 4), (2, 4), (3, 4), (4, 5), (5, 6)]
prompt_max 546 prompt_total 1552
/app/methods/main/__pycache__/optimize.cpython-310.pyc
/app/methods/main/__pycache__/swarm.cpython-310.pyc
/app/methods/main/graph.json
/app/methods/main/optimize.py
/app/methods/main/swarm.py
1334

Script completed
Wall time 0.1 seconds
Output:
diff --git a/app/methods/main/optimize.py b/app/methods/main/graph.json
index 05df52d..3f615ab 100644
--- a/app/methods/main/optimize.py
+++ b/app/methods/main/graph.json
@@ -1,287 +1,53 @@
-"""CARE: cost-aware counterfactual racing for language-agent graphs.
-
-This replaces graph-level REINFORCE with a small, sample-efficient search over
-node prompts, heterogeneous node kinds, and topology.  It uses successive
-halving on whole architectures, then paired leave-one-edge-out tests on the
-winner.  Accuracy and call cost are recorded per problem, so an edge is kept
-only when its empirical credit exceeds its cost.  The shipped graph.json is the
-champion from a longer run; this program can reproduce the search protocol.
-
-Usage: python3 optimize.py [--stage1 6] [--stage2 16] [--seed 2026]
-"""
-
-from __future__ import annotations
-
-import argparse
-import random
-from dataclasses import dataclass
-from pathlib import Path
-
-from swarm import BudgetMeter, Node, Swarm, load_problems
-
-TRAIN = "/app/data/train.jsonl"
-VAL = "/app/data/val.jsonl"
-REF_CALLS = 4.0
-
-ANALYTIC = """You are the primary expert AIME solver. Solve the problem independently and exactly. Identify the governing structure, handle all cases, and verify the result by substitution or a second calculation. Do not guess. After private reasoning, BEGIN the visible response with <answer>N</answer> for N in 0..999, then give a compact checkable derivation; do not repeat exploratory work.
-
-Problem: {problem}{context}"""
-
-SKEPTIC = """Independently solve this AIME problem as a skeptical olympiad contestant. Seek a different representation or method from the obvious approach; check boundary cases, reversibility, multiplicities, and arithmetic. Trust no proposed answer unless verified. After private reasoning, BEGIN the visible response with <answer>N</answer>, then give a compact checkable certificate without repeating exploration.
-
-Problem: {problem}{context}"""
-
-COMPUTE = """Solve this AIME problem using an exact computational or symbolic formulation whenever possible. Translate every condition faithfully; prefer exhaustive integer/rational computation to intuition and verify the model. After private reasoning, BEGIN the visible response with <answer>N</answer>, then a compact justification. The LAST fenced Python code block in the entire response must be a self-contained standard-library program needing no input, terminating quickly, and printing only N; put no code block after it.
-
-Problem: {problem}{context}"""
-
-REVIEW = """Act as the senior AIME referee. Re-solve the problem rigorously while auditing all proposed analyses and the executed result below. Inputs may be wrong. Find the first concrete error behind any disagreement; verify every condition, boundary case, multiplicity, and arithmetic step. After private reasoning, BEGIN the visible response with <answer>N</answer>, then give a compact decisive certificate without repeating exploration.
-
-Problem: {problem}{context}"""
-
-FAST_DECIDE = """/no_think
-You are the final AIME evidence adjudicator. Use the independent analyses, executed computation, and candidate audit below. Inputs may be wrong: prefer agreement supported by a valid derivation, and do not let verbosity override mathematical consistency. Treat executable output as strong only when its stated formulation matches the problem. Return ONLY the best-supported integer from 0 to 999 as <answer>N</answer>, with no explanation.
-
-Problem: {problem}{context}"""
-
-RELAY = """/no_think
-Copy the referee_extract node's reported candidate. Return ONLY that integer as <answer>N</answer>.
-
-Problem: {problem}{context}"""
-
-DEEP_DECIDE = """You are the final AIME adjudicator. Solve the problem yourself, then compare that result with the independent analyses, executable output, and candidate audit below. Inputs may be wrong. Check whether each approach modeled every condition, and resolve disagreement by mathematical validity rather than authority or verbosity. Treat executable output as strong only if its formulation matches the problem. Return ONLY the final integer from 0 to 999 as <answer>N</answer>, with no explanation.
-
-Problem: {problem}{context}"""
-
-
-@dataclass(frozen=True)
-class Trial:
-    name: str
-    graph: Swarm
-
-
-@dataclass
-class Result:
-    correct: list[int]
-    calls: list[int]
-
-    @property
-    def accuracy(self) -> float:
-        return sum(self.correct) / max(1, len(self.correct))
-
-    @property
-    def avg_calls(self) -> float:
-        return sum(self.calls) / max(1, len(self.calls))
-
-    @property
-    def utility(self) -> float:
-        # Smooth proxy for the grader's efficiency multiplier.  The exponent is
-        # deliberately mild: a real accuracy gain can pay for one extra call.
-        efficiency = (REF_CALLS / max(1.0, self.avg_calls)) ** 0.35
-        return self.accuracy * efficiency
-
-
-def parallel_four() -> Swarm:
-    """Three diverse proposals, free execution/audit, deep fourth-call vote."""
-    nodes = [
-        Node("analytic_solver", ANALYTIC),
-        Node("skeptical_solver", SKEPTIC),
-        Node("computational_solver", COMPUTE),
-        Node("code_result", "", "code_exec"),
-        Node("candidate_audit", "", "symbolic_verify"),
-        Node("decision", DEEP_DECIDE),
-    ]
-    edges = [(-1, 0), (-1, 1), (-1, 2), (2, 3),
-             (0, 4), (1, 4), (2, 4), (3, 4),
-             (0, 5), (1, 5), (2, 5), (3, 5), (4, 5)]
-    return Swarm(nodes, edges)
-
-
-def cascade_four() -> Swarm:
-    """Two proposals followed by a thinking referee and a non-thinking relay."""
-    nodes = [
-        Node("analytic_solver", ANALYTIC),
-        Node("computational_solver", COMPUTE),
-        Node("code_result", "", "code_exec"),
-        Node("candidate_audit", "", "symbolic_verify"),
-        Node("referee", REVIEW),
-        Node("referee_extract", "", "symbolic_verify"),
-        Node("decision", RELAY),
-    ]
-    edges = [(-1, 0), (-1, 1), (1, 2),
-             (0, 3), (1, 3), (2, 3),
-             (0, 4), (1, 4), (2, 4), (3, 4),
-             (4, 5), (5, 6)]
-    return Swarm(nodes, edges)
-
-
-def deep_five() -> Swarm:
-    """Three proposals plus a thinking referee; costs one call over reference."""
-    nodes = [
-        Node("analytic_solver", ANALYTIC),
-        Node("skeptical_solver", SKEPTIC),
-        Node("computational_solver", COMPUTE),
-        Node("code_result", "", "code_exec"),
-        Node("candidate_audit", "", "symbolic_verify"),
-        Node("referee", REVIEW),
-        Node("referee_extract", "", "symbolic_verify"),
-        Node("decision", RELAY),
-    ]
-    edges = [(-1, 0), (-1, 1), (-1, 2), (2, 3),
-             (0, 4), (1, 4), (2, 4), (3, 4),
-             (0, 5), (1, 5), (2, 5), (3, 5), (4, 5),
-             (5, 6), (6, 7)]
-    return Swarm(nodes, edges)
-
-
-def lean_three() -> Swarm:
-    """Efficiency arm: two proposals, free execution/audit, one decision."""
-    nodes = [
-        Node("analytic_solver", ANALYTIC),
-        Node("computational_solver", COMPUTE),
-        Node("code_result", "", "code_exec"),
-        Node("candidate_audit", "", "symbolic_verify"),
-        Node("decision", DEEP_DECIDE),
-    ]
-    edges = [(-1, 0), (-1, 1), (1, 2),
-             (0, 3), (1, 3), (2, 3),
-             (0, 4), (1, 4), (2, 4), (3, 4)]
-    return Swarm(nodes, edges)
-
-
-def one_shot() -> Swarm:
-    # A serious gate calibration, not a weak baseline.
-    prompt = """You are an expert AIME contestant. Solve the problem independently and exactly. Think deeply, handle all cases and constraints, and check the result with a second calculation or sanity check before committing. Do not guess. End with exactly <answer>N</answer>, where N is the required integer from 0 to 999.
-
-Problem: {problem}"""
-    return Swarm([Node("decision", prompt)], [])
-
-
-def run_trial(sw: Swarm, problems: list[dict]) -> Result:
-    """Keep the per-instance vector: paired deltas are the credit signal."""
-    from concurrent.futures import ThreadPoolExecutor
-
-    sw.validate()
-
-    def one(p: dict) -> tuple[int, int]:
-        meter = BudgetMeter()
-        try:
-            pred = sw.run(p["problem"], meter)
-            ok = int(pred == int(p["answer"]))
-        except Exception:  # a failed problem is zero, never a failed search
-            ok = 0
-        return ok, meter.calls
-
-    with ThreadPoolExecutor(max_workers=min(8, max(1, len(problems)))) as ex:
-        rows = list(ex.map(one, problems))
-    return Result([r[0] for r in rows], [r[1] for r in rows])
-
-
-def active_llms(sw: Swarm) -> int:
-    return sum(sw.nodes[i].kind == "llm" for i in sw._active_nodes())
-
-
-def edge_ablations(sw: Swarm) -> list[Swarm]:
-    """Legal counterfactuals, ordered to test redundant context edges first."""
-    out: list[Swarm] = []
-    for edge in reversed(sw.active_edges()):
-        candidate = Swarm(sw.nodes[:], [e for e in sw.edges if e != edge])
-        try:
-            candidate.validate()
-        except ValueError:
-            continue
-        if active_llms(candidate) >= 3 and candidate.active_edges() != sw.active_edges():
-            out.append(candidate)
-    return out
-
-
-def problem_order(seed: int) -> list[dict]:
-    """Interleave train and held-out years instead of tuning only one sitting."""
-    rng = random.Random(seed)
-    train = load_problems(TRAIN)
-    val = load_problems(VAL)
-    rng.shuffle(train)
-    rng.shuffle(val)
-    mixed: list[dict] = []
-    while train or val:
-        if train:
-            mixed.append(train.pop())
-        if val:
-            mixed.append(val.pop())
-    return mixed
-
-
-def describe(name: str, r: Result) -> None:
-    print(f"{name:18s} acc={r.accuracy:.3f} calls={r.avg_calls:.2f} "
-          f"utility={r.utility:.3f}", flush=True)
-
-
-def main() -> None:
-    ap = argparse.ArgumentParser()
-    ap.add_argument("--stage1", type=int, default=6,
-                    help="problems per architecture in the racing round")
-    ap.add_argument("--stage2", type=int, default=16,
-                    help="problems for finalists and edge counterfactuals")
-    ap.add_argument("--ablations", type=int, default=3,
-                    help="maximum leave-one-edge-out counterfactuals")
-    ap.add_argument("--seed", type=int, default=2026)
-    ap.add_argument("--out", default=str(Path(__file__).with_name("graph.json")))
-    args = ap.parse_args()
-
-    ordered = problem_order(args.seed)
-    gate_set = ordered[:max(args.stage1, 1)]
-    final_set = ordered[:max(args.stage2, args.stage1)]
-
-    gate = run_trial(one_shot(), gate_set)
-    describe("one-shot gate", gate)
-
-    trials = [Trial("parallel-four", parallel_four()),
-              Trial("cascade-four", cascade_four()),
-              Trial("deep-five", deep_five()),
-              Trial("lean-three", lean_three())]
-
-    # Successive halving: spend the larger sample only on the two best arms.
-    first: list[tuple[float, Trial]] = []
-    for t in trials:
-        r = run_trial(t.graph, gate_set)
-        describe(t.name + "/r1", r)
-        # Hard-gate awareness: a candidate that fails to beat one shot is not a
-        # cheap winner; it is ineligible even if its cost proxy looks good.
-        score = r.utility if r.accuracy > gate.accuracy else -1.0 + r.utility
-        first.append((score, t))
-    finalists = [t for _, t in sorted(first, key=lambda x: x[0], reverse=True)[:2]]
-
-    ranked: list[tuple[float, Trial, Result]] = []
-    for t in finalists:

Script completed
Wall time 0.0 seconds
Output:
{}

Script completed
Wall time 0.2 seconds
Output:
usage: optimize.py [-h] [--stage1 STAGE1] [--stage2 STAGE2]
                   [--ablations ABLATIONS] [--repeats REPEATS] [--seed SEED]
                   [--out OUT]

options:
  -h, --help            show this help message and exit
  --stage1 STAGE1       problems per architecture in the racing round
  --stage2 STAGE2       problems for finalists and edge counterfactuals
  --ablations ABLATIONS
                        maximum leave-one-edge-out counterfactuals
  --repeats REPEATS     stochastic repeats per problem before ranking
  --seed SEED
  --out OUT
FINAL_OK 7 12 4
1283