on-policy-self-distillation

gpt-5.6-sol

on-policy-self-distillation — graded by the task's verifier against a measured baseline.

backend: modal
variant: codex azure_ai/gpt-5.6-sol
metric: AIME24 avg@12 accuracy
actions: 428
duration: 9h 21m
recorded: 2026-08-02

run idopsd-aime24-4gpus__eNDoWrD

Everything gpt-5.6-sol did on this task, in order.

download JSON ↗

codexgpt-5.6-sol444 turns16 narration428 reasoning428 action428 observation1 final

#301

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait

args

{
  "cell_id": "103",
  "yield_time_ms": 60000,
  "max_tokens": 80
}

wait result

Script running with cell ID 103
Wall time 61.0 seconds
Output:

#302

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait

args

{
  "cell_id": "103",
  "yield_time_ms": 60000,
  "max_tokens": 80
}

wait result

Script running with cell ID 103
Wall time 61.0 seconds
Output:

#303

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait

args

{
  "cell_id": "103",
  "yield_time_ms": 60000,
  "max_tokens": 80
}

wait result

Script running with cell ID 103
Wall time 61.0 seconds
Output:

#304

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait

args

{
  "cell_id": "103",
  "yield_time_ms": 60000,
  "max_tokens": 80
}

wait result

Script running with cell ID 103
Wall time 61.0 seconds
Output:

#305

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait

args

{
  "cell_id": "103",
  "yield_time_ms": 60000,
  "max_tokens": 80
}

wait result

Script running with cell ID 103
Wall time 61.0 seconds
Output:

#306

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait

args

{
  "cell_id": "103",
  "yield_time_ms": 60000,
  "max_tokens": 80
}

wait result

Script running with cell ID 103
Wall time 61.0 seconds
Output:

#307

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait

args

{
  "cell_id": "103",
  "yield_time_ms": 60000,
  "max_tokens": 80
}

wait result

Script running with cell ID 103
Wall time 61.0 seconds
Output:

#308

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait

args

{
  "cell_id": "103",
  "yield_time_ms": 60000,
  "max_tokens": 80
}

wait result

Script running with cell ID 103
Wall time 61.0 seconds
Output:

#309

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait

args

{
  "cell_id": "103",
  "yield_time_ms": 60000,
  "max_tokens": 80
}

wait result

Script running with cell ID 103
Wall time 61.0 seconds
Output:

#310

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait

args

{
  "cell_id": "103",
  "yield_time_ms": 60000,
  "max_tokens": 80
}

wait result

Script running with cell ID 103
Wall time 61.0 seconds
Output:

#311

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait

args

{
  "cell_id": "103",
  "yield_time_ms": 60000,
  "max_tokens": 80
}

wait result

Script running with cell ID 103
Wall time 61.0 seconds
Output:

#312

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait

args

{
  "cell_id": "103",
  "yield_time_ms": 60000,
  "max_tokens": 80
}

wait result

Script running with cell ID 103
Wall time 61.0 seconds
Output:

#313

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait

args

{
  "cell_id": "103",
  "yield_time_ms": 60000,
  "max_tokens": 80
}

wait result

Script running with cell ID 103
Wall time 61.0 seconds
Output:

#314

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait

args

{
  "cell_id": "103",
  "yield_time_ms": 60000,
  "max_tokens": 80
}

wait result

Script completed
Wall time 57.7 seconds
Output:
Warning: truncated output (original token count: 246)
Total output lines: 19

     | 0/96 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]

 [02:28<17:16, 12.34s/it, est. speed input: 12.64 toks/s, output: 520.17 toks/…166 tokens truncated…1<14:21, 17.95s/it, est. speed input: 4.00 toks/s, output: 778.95 toks/s]
0 problems)
Results written to /workspace/eval_results/dev_eval_1785707360.json

EXIT0

#315

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolexec

args

{
  "input": "const r = await tools.exec_command({cmd:\"python - <<'PY'\\nimport json\\nf='/workspace/eval_results/dev_eval_1785707360.json';d=json.load(open(f));print(d['average_at_n_pct'],sum(x['num_correct'] for x in d['results']));print([(x['problem_id'],x['num_correct']) for x in d['results']])\\nPY\\ncat /workspace/.timer/remaining_secs\",\"workdir\":\"/workspace\",\"yield_time_ms\":10000,\"max_output_tokens\":3000});\ntext(r.output);"
}

exec result

Script completed
Wall time 0.1 seconds
Output:
54.44444444444444 196
[(60, 12), (61, 10), (62, 0), (63, 0), (64, 4), (65, 7), (66, 7), (67, 12), (68, 11), (69, 12), (70, 6), (71, 12), (72, 12), (73, 0), (74, 10), (75, 5), (76, 5), (77, 6), (78, 6), (79, 12), (80, 6), (81, 0), (82, 6), (83, 10), (84, 12), (85, 0), (86, 9), (87, 4), (88, 0), (89, 0)]
8542

#316

narration

Rank 128 again lands at 54.44%; extra adapter capacity redistributes successes but does not raise the total. The remaining high-value hypothesis is a milder curriculum than the failed source-only replay: use the dataset’s own generation-token count as a difficulty signal, keep all contest sources, and spend the fixed 3,200-example budget on problems that required long verified reasoning. This avoids both easy-item dilution and narrow AIME overfitting.

#317

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolexec

args

{
  "input": "const patch = \"*** Begin Patch\\n*** Update File: /workspace/repo/opsd_train.py\\n@@\\n     dataset = load_dataset(\\\"siyanzhao/Openthoughts_math_30k_opsd\\\")\\n     train_dataset = dataset[\\\"train\\\"]\\n+\\n+    # Reasoning-effort curriculum.  `generated_token_count` is metadata shipped\\n+    # with the released dataset and measures how much reasoning was needed to\\n+    # obtain its verified solution.  Under a 100-step budget, uniformly drawing\\n+    # many short arithmetic items wastes privileged-teacher calls.  Retain the\\n+    # broad source mix but train on examples requiring at least 3k reasoning\\n+    # tokens, a model-native difficulty signal rather than a benchmark label.\\n+    train_dataset = train_dataset.filter(\\n+        lambda example: example[\\\"generated_token_count\\\"] >= 3000,\\n+        desc=\\\"Selecting high-reasoning-effort OPSD examples\\\",\\n+    )\\n+    if os.environ.get(\\\"LOCAL_RANK\\\", \\\"0\\\") == \\\"0\\\":\\n+        print(f\\\"Reasoning-effort curriculum: {len(train_dataset)} training rows\\\")\\n*** Update File: /workspace/submission/recipe.env\\n@@\\n-# Higher-capacity OPSD adapter with the released update scale and schedule.\\n+# Difficulty-curriculum OPSD with the released optimizer/model settings.\\n lmbda=0\\n top_k_loss=0\\n-lora_r=128\\n-lora_alpha=256\\n+lora_r=64\\n+lora_alpha=128\\n*** End Patch\";\ntext(await tools.apply_patch(patch));"
}

exec result

Script completed
Wall time 0.0 seconds
Output:
{}

#318

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolexec

args

{
  "input": "const r = await tools.exec_command({cmd:\"SEED=42 OUT_DIR=/workspace/runs/hard3000 PORT=12959 bash /workspace/train_capped.sh\",\"workdir\":\"/workspace\",\"yield_time_ms\":1000,\"max_output_tokens\":300});\ntext(r.output);if(r.session_id)text(`SESSION_ID=${r.session_id}`);"
}

exec result

Script completed
Wall time 1.2 seconds
Output:
SESSION_ID=94888

#319

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolexec

args

{
  "input": "let out=\"\";for(let i=0;i<12;i++){const r=await tools.write_stdin({session_id:94888,chars:\"\",yield_time_ms:60000,max_output_tokens:50});out+=r.output.slice(-100)+\"\\n\";if(r.exit_code!==undefined){out+=`EXIT${r.exit_code}`;break;}}text(out);\n"
}

exec result

Script running with cell ID 107
Wall time 11.0 seconds
Output:

#320

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait

args

{
  "cell_id": "107",
  "yield_time_ms": 60000,
  "max_tokens": 80
}

wait result

Script running with cell ID 107
Wall time 61.0 seconds
Output:

#321

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait

args

{
  "cell_id": "107",
  "yield_time_ms": 60000,
  "max_tokens": 80
}

wait result

Script running with cell ID 107
Wall time 61.0 seconds
Output:

#322

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait

args

{
  "cell_id": "107",
  "yield_time_ms": 60000,
  "max_tokens": 80
}

wait result

Script running with cell ID 107
Wall time 61.0 seconds
Output:

#323

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait

args

{
  "cell_id": "107",
  "yield_time_ms": 60000,
  "max_tokens": 80
}

wait result

Script running with cell ID 107
Wall time 61.0 seconds
Output:

#324

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait

args

{
  "cell_id": "107",
  "yield_time_ms": 60000,
  "max_tokens": 80
}

wait result

Script running with cell ID 107
Wall time 61.0 seconds
Output:

#325

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait

args

{
  "cell_id": "107",
  "yield_time_ms": 60000,
  "max_tokens": 80
}

wait result

Script running with cell ID 107
Wall time 61.0 seconds
Output: