on-policy-self-distillation

gpt-5.6-sol

on-policy-self-distillation — graded by the task's verifier against a measured baseline.

backend: modal
variant: codex azure_ai/gpt-5.6-sol
metric: AIME24 avg@12 accuracy
actions: 627
duration: 6h 30m
recorded: 2026-08-02

run idopsd-aime24-4gpus__RqGcDty

Everything gpt-5.6-sol did on this task, in order.

download JSON ↗

codexgpt-5.6-sol684 turns57 narration627 reasoning627 action627 observation1 final

#101

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolexec

args

{
  "input": "const r = await tools.write_stdin({session_id:99639,chars:\"\",yield_time_ms:30000,max_output_tokens:300});\ntext(r.output);\n"
}

exec result

Script running with cell ID 64
Wall time 11.0 seconds
Output:

#102

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait

args

{
  "cell_id": "64",
  "yield_time_ms": 30000,
  "max_tokens": 300
}

wait result

Script completed
Wall time 9.8 seconds
Output:

Processed prompts:  25%|██▌       | 24/96 [03:47<11:08,  9.28s/it, est. speed input: 17.02 toks/s, output: 759.57 toks/s]

#103

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolexec

args

{
  "input": "const r = await tools.write_stdin({session_id:99639,chars:\"\",yield_time_ms:30000,max_output_tokens:300});\ntext(r.output);\n"
}

exec result

Script running with cell ID 65
Wall time 11.0 seconds
Output:

#104

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait

args

{
  "cell_id": "65",
  "yield_time_ms": 30000,
  "max_tokens": 300
}

wait result

Script completed
Wall time 12.9 seconds
Output:

#105

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolexec

args

{
  "input": "const r = await tools.exec_command({cmd:\"python - <<'PY'\\nimport json\\nD=json.load(open('/workspace/baseline/logs/official_eval_result.json'))\\nfor x in D['results']:\\n print(x['id'], x.get('correct_count'), x.get('accuracy'), x.keys())\\nPY\",\"workdir\":\"/workspace\",\"yield_time_ms\":10000,\"max_output_tokens\":5000});\ntext(r.output);\n"
}

exec result

Script completed
Wall time 0.1 seconds
Output:
Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
KeyError: 'id'

#106

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolexec

args

{
  "input": "const r = await tools.exec_command({cmd:\"python - <<'PY'\\nimport json,pprint\\nD=json.load(open('/workspace/baseline/logs/official_eval_result.json'))\\npprint.pp(D['results'][0])\\nPY\",\"workdir\":\"/workspace\",\"yield_time_ms\":10000,\"max_output_tokens\":4000});\ntext(r.output);\n"
}

exec result

Script completed
Wall time 0.1 seconds
Output:
{'problem_id': 60,
 'ground_truth': '204',
 'num_correct': 12,
 'n': 12,
 'majority_vote_correct': True,
 'generations': [{'predicted_answer': '204',
                  'correct': True,
                  'formatted': True},
                 {'predicted_answer': '204',
                  'correct': True,
                  'formatted': True},
                 {'predicted_answer': '204',
                  'correct': True,
                  'formatted': True},
                 {'predicted_answer': '204',
                  'correct': True,
                  'formatted': True},
                 {'predicted_answer': '204',
                  'correct': True,
                  'formatted': True},
                 {'predicted_answer': '204',
                  'correct': True,
                  'formatted': True},
                 {'predicted_answer': '204',
                  'correct': True,
                  'formatted': True},
                 {'predicted_answer': '204',
                  'correct': True,
                  'formatted': True},
                 {'predicted_answer': '204',
                  'correct': True,
                  'formatted': True},
                 {'predicted_answer': '204',
                  'correct': True,
                  'formatted': True},
                 {'predicted_answer': '204',
                  'correct': True,
                  'formatted': True},
                 {'predicted_answer': '204',
                  'correct': True,
                  'formatted': True}]}

#107

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolexec

args

{
  "input": "const r = await tools.write_stdin({session_id:99639,chars:\"\",yield_time_ms:30000,max_output_tokens:300});\ntext(r.output);\n"
}

exec result

Script running with cell ID 68
Wall time 11.0 seconds
Output:

#108

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait

args

{
  "cell_id": "68",
  "yield_time_ms": 30000,
  "max_tokens": 300
}

wait result

Script completed
Wall time 15.6 seconds
Output:

#109

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolexec

args

{
  "input": "const r = await tools.write_stdin({session_id:99639,chars:\"\",yield_time_ms:30000,max_output_tokens:300});\ntext(r.output);\n"
}

exec result

Script running with cell ID 69
Wall time 11.0 seconds
Output:

#110

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait

args

{
  "cell_id": "69",
  "yield_time_ms": 30000,
  "max_tokens": 300
}

wait result

Script completed
Wall time 15.1 seconds
Output:

#111

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolexec

args

{
  "input": "const r = await tools.write_stdin({session_id:99639,chars:\"\",yield_time_ms:30000,max_output_tokens:300});\ntext(r.output);\n"
}

exec result

Script running with cell ID 70
Wall time 11.0 seconds
Output:

#112

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait

args

{
  "cell_id": "70",
  "yield_time_ms": 30000,
  "max_tokens": 300
}

wait result

Script completed
Wall time 11.5 seconds
Output:

#113

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolexec

args

{
  "input": "const r = await tools.write_stdin({session_id:99639,chars:\"\",yield_time_ms:30000,max_output_tokens:500});\ntext(r.output);\n"
}

exec result

Script running with cell ID 71
Wall time 11.0 seconds
Output:

#114

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait

args

{
  "cell_id": "71",
  "yield_time_ms": 30000,
  "max_tokens": 500
}

wait result

Script completed
Wall time 13.7 seconds
Output:

Processed prompts:  14%|█▍        | 12/84 [07:28<44:51, 37.38s/it, est. speed input: 2.06 toks/s, output: 303.32 toks/s]

#115

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolexec

args

{
  "input": "const patch = \"*** Begin Patch\\n*** Update File: /workspace/repo/data_collator.py\\n@@\\n-            student_user_message = f\\\"Problem: {problem}\\\\n\\\\nPlease reason step by step, and put your final answer within \\\\\\\\boxed{{}}.\\\"\\n+            # Match the frozen AIME evaluator exactly.  The released collator\\n+            # prepended \\\"Problem: \\\" even though evaluation supplies the raw\\n+            # problem, creating a needless train/inference context shift.\\n+            student_user_message = f\\\"{problem}\\\\n\\\\nPlease reason step by step, and put your final answer within \\\\\\\\boxed{{}}.\\\"\\n*** End Patch\";\ntext(await tools.apply_patch(patch));\n"
}

exec result

Script completed
Wall time 0.0 seconds
Output:
{}

#116

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolexec

args

{
  "input": "const patch = \"*** Begin Patch\\n*** Update File: /workspace/repo/opsd_train.py\\n@@\\n import os\\n+import re\\n import wandb\\n@@\\n-    # Target-domain replay.  The released pool is dominated by broad olympiad\\n-    # problems (72%) and contains only 1.6% AMC/AIME examples.  With a\\n-    # 3,200-example training budget this gives the policy very little exposure\\n-    # to the short, integer-answer competition problems used by AIME.  Build a\\n-    # deterministic mixture from the two closest in-domain sources and replay\\n-    # the scarce AMC/AIME partition.  Trainer still shuffles the resulting\\n-    # dataset with the configured seed; this changes neither optimizer steps\\n-    # nor global batch size.\\n-    aime = train_dataset.filter(lambda example: example[\\\"source\\\"] == \\\"amc_aime\\\")\\n-    aops = train_dataset.filter(lambda example: example[\\\"source\\\"] == \\\"aops_forum\\\")\\n-    train_dataset = concatenate_datasets([aime, aime, aime, aime, aops])\\n+    # Difficulty/format-matched replay.  AIME uses hard free-response problems\\n+    # whose answers are small integers, while the released pool is 72% broad\\n+    # olympiad data and its `amc_aime` source is mostly easy multiple choice.\\n+    # Select examples using only frozen dataset metadata, then replay the scarce\\n+    # past-AIME slice.  The mix remains diverse (AoPS, MATH, and olympiad) and is\\n+    # shuffled by Trainer with the scoring seed.  This changes neither the\\n+    # number of optimizer steps nor the global batch size.\\n+    def has_integer_answer(example):\\n+        answer = (example.get(\\\"Answer\\\") or \\\"\\\").strip()\\n+        return re.fullmatch(r\\\"(?:\\\\\\\\boxed\\\\{)?-?\\\\d{1,3}(?:\\\\})?\\\", answer) is not None\\n+\\n+    past_aime = train_dataset.filter(\\n+        lambda example: example[\\\"source\\\"] == \\\"amc_aime\\\" and has_integer_answer(example)\\n+    )\\n+    hard_aops = train_dataset.filter(\\n+        lambda example: example[\\\"source\\\"] == \\\"aops_forum\\\"\\n+        and has_integer_answer(example)\\n+        and example[\\\"generated_token_count\\\"] >= 2500\\n+    )\\n+    hard_math = train_dataset.filter(\\n+        lambda example: example[\\\"source\\\"] == \\\"math\\\"\\n+        and has_integer_answer(example)\\n+        and example[\\\"generated_token_count\\\"] >= 2500\\n+    )\\n+    hard_olympiad = train_dataset.filter(\\n+        lambda example: example[\\\"source\\\"] == \\\"olympiads\\\"\\n+        and has_integer_answer(example)\\n+        and example[\\\"generated_token_count\\\"] >= 3000\\n+    )\\n+    hard_olympiad = hard_olympiad.select(range(min(1000, len(hard_olympiad))))\\n+    train_dataset = concatenate_datasets(\\n+        [past_aime] * 8 + [hard_aops, hard_math, hard_olympiad]\\n+    )\\n     print(\\n-        f\\\"Target-domain OPSD mixture: {len(train_dataset)} rows \\\"\\n-        f\\\"({4 * len(aime)} AMC/AIME replay rows, {len(aops)} AoPS rows)\\\"\\n+        f\\\"AIME-matched OPSD mixture: {len(train_dataset)} rows \\\"\\n+        f\\\"({8 * len(past_aime)} past-AIME replay, {len(hard_aops)} AoPS, \\\"\\n+        f\\\"{len(hard_math)} MATH, {len(hard_olympiad)} olympiad)\\\"\\n     )\\n*** End Patch\";\ntext(await tools.apply_patch(patch));\n"
}

exec result

Script completed
Wall time 0.0 seconds
Output:
{}

#117

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolexec

args

{
  "input": "const r = await tools.exec_command({cmd:\"python -m py_compile opsd_train.py data_collator.py opsd_trainer.py && git diff --stat && git diff -- data_collator.py opsd_train.py | sed -n '1,240p'\",\"workdir\":\"/workspace/repo\",\"yield_time_ms\":10000,\"max_output_tokens\":15000});\ntext(r.output);\n"
}

exec result

Script completed
Wall time 0.2 seconds
Output:
 README.md        |  4 +---
 data_collator.py |  5 ++++-
 opsd_train.py    | 42 +++++++++++++++++++++++++++++++++++++++++-
 3 files changed, 46 insertions(+), 5 deletions(-)
diff --git a/data_collator.py b/data_collator.py
index 0829108..d596a51 100644
--- a/data_collator.py
+++ b/data_collator.py
@@ -64,7 +64,10 @@ class SelfDistillationDataCollator:
             solution = feature["solution"]
 
             # Student prompt: just the problem with instruction (matching evaluation format)
-            student_user_message = f"Problem: {problem}\n\nPlease reason step by step, and put your final answer within \\boxed{{}}."
+            # Match the frozen AIME evaluator exactly.  The released collator
+            # prepended "Problem: " even though evaluation supplies the raw
+            # problem, creating a needless train/inference context shift.
+            student_user_message = f"{problem}\n\nPlease reason step by step, and put your final answer within \\boxed{{}}."
             student_messages = [{"role": "user", "content": student_user_message}]
 
             # Apply chat template for student (matching evaluation)
diff --git a/opsd_train.py b/opsd_train.py
index 8526a68..86a3494 100644
--- a/opsd_train.py
+++ b/opsd_train.py
@@ -1,7 +1,8 @@
 import os
+import re
 import wandb
 
-from datasets import load_dataset
+from datasets import concatenate_datasets, load_dataset
 from transformers import AutoTokenizer, GenerationConfig
 
 from trl import (
@@ -266,6 +267,45 @@ if __name__ == "__main__":
     dataset = load_dataset("siyanzhao/Openthoughts_math_30k_opsd")
     train_dataset = dataset["train"]
 
+    # Difficulty/format-matched replay.  AIME uses hard free-response problems
+    # whose answers are small integers, while the released pool is 72% broad
+    # olympiad data and its `amc_aime` source is mostly easy multiple choice.
+    # Select examples using only frozen dataset metadata, then replay the scarce
+    # past-AIME slice.  The mix remains diverse (AoPS, MATH, and olympiad) and is
+    # shuffled by Trainer with the scoring seed.  This changes neither the
+    # number of optimizer steps nor the global batch size.
+    def has_integer_answer(example):
+        answer = (example.get("Answer") or "").strip()
+        return re.fullmatch(r"(?:\\boxed\{)?-?\d{1,3}(?:\})?", answer) is not None
+
+    past_aime = train_dataset.filter(
+        lambda example: example["source"] == "amc_aime" and has_integer_answer(example)
+    )
+    hard_aops = train_dataset.filter(
+        lambda example: example["source"] == "aops_forum"
+        and has_integer_answer(example)
+        and example["generated_token_count"] >= 2500
+    )
+    hard_math = train_dataset.filter(
+        lambda example: example["source"] == "math"
+        and has_integer_answer(example)
+        and example["generated_token_count"] >= 2500
+    )
+    hard_olympiad = train_dataset.filter(
+        lambda example: example["source"] == "olympiads"
+        and has_integer_answer(example)
+        and example["generated_token_count"] >= 3000
+    )
+    hard_olympiad = hard_olympiad.select(range(min(1000, len(hard_olympiad))))
+    train_dataset = concatenate_datasets(
+        [past_aime] * 8 + [hard_aops, hard_math, hard_olympiad]
+    )
+    print(
+        f"AIME-matched OPSD mixture: {len(train_dataset)} rows "
+        f"({8 * len(past_aime)} past-AIME replay, {len(hard_aops)} AoPS, "
+        f"{len(hard_math)} MATH, {len(hard_olympiad)} olympiad)"
+    )
+
     trainer = OPSDTrainer(
         model=model_args.model_name_or_path,
         args=training_args,

#118

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolexec

args

{
  "input": "const r = await tools.write_stdin({session_id:99639,chars:\"\",yield_time_ms:30000,max_output_tokens:300});\ntext(r.output);\n"
}

exec result

Script running with cell ID 75
Wall time 11.0 seconds
Output:

#119

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait

args

{
  "cell_id": "75",
  "yield_time_ms": 30000,
  "max_tokens": 300
}

wait result

Script completed
Wall time 15.6 seconds
Output:

#120

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolexec

args

{
  "input": "const r = await tools.write_stdin({session_id:99639,chars:\"\",yield_time_ms:30000,max_output_tokens:300});\ntext(r.output);\n"
}

exec result

Script running with cell ID 76
Wall time 11.0 seconds
Output:

#121

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait

args

{
  "cell_id": "76",
  "yield_time_ms": 30000,
  "max_tokens": 300
}

wait result

Script completed
Wall time 14.2 seconds
Output:

#122

narration

The refined sampler is now implemented and syntax-checked. It produces a roughly one-epoch-sized pool for the 3,200-example budget, so replay is deliberate but bounded rather than cycling a tiny set for many epochs. I’ve also applied the exact evaluator-side student message format for the next training run.

#123

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolexec

args

{
  "input": "const r = await tools.write_stdin({session_id:99639,chars:\"\",yield_time_ms:30000,max_output_tokens:300});\ntext(r.output);\n"
}

exec result

Script running with cell ID 77
Wall time 11.0 seconds
Output:

#124

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait

args

{
  "cell_id": "77",
  "yield_time_ms": 30000,
  "max_tokens": 300
}

wait result

Script completed
Wall time 12.2 seconds
Output:

#125

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolexec

args

{
  "input": "const r = await tools.write_stdin({session_id:99639,chars:\"\",yield_time_ms:30000,max_output_tokens:300});\ntext(r.output);\n"
}

exec result

Script running with cell ID 78
Wall time 11.0 seconds
Output: