on-policy-self-distillation

gpt-5.6-sol

on-policy-self-distillation — graded by the task's verifier against a measured baseline.

backend: modal
variant: codex azure_ai/gpt-5.6-sol
metric: AIME24 avg@12 accuracy
actions: 627
duration: 6h 30m
recorded: 2026-08-02

run idopsd-aime24-4gpus__RqGcDty

Everything gpt-5.6-sol did on this task, in order.

download JSON ↗

codexgpt-5.6-sol684 turns57 narration627 reasoning627 action627 observation1 final

#626

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolexec

args

{
  "input": "const r = await tools.exec_command({cmd:\"bash /workspace/dev_eval.sh /workspace/runs/refined_repeat3/capped_seed42/checkpoint-100 /workspace/eval_results/refined_repeat3_full.json\",\"workdir\":\"/workspace\",\"yield_time_ms\":1000,\"max_output_tokens\":300});\ntext(r.output); if(r.session_id) text(`SESSION_ID=${r.session_id}`);\n"
}

exec result

Script completed
Wall time 1.2 seconds
Output:
SESSION_ID=32207

#627

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolexec

args

{
  "input": "const r = await tools.write_stdin({session_id:32207,chars:\"\",yield_time_ms:30000,max_output_tokens:150});\ntext(r.output);\n"
}

exec result

Script running with cell ID 321
Wall time 11.0 seconds
Output:

#628

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait

args

{
  "cell_id": "321",
  "yield_time_ms": 30000,
  "max_tokens": 150
}

wait result

Script completed
Wall time 11.3 seconds
Output:
Warning: truncated output (original token count: 177)
Total output lines: 10

Warning: truncated output (original token count: 5933)
Total output lines: 154

`torch_dtype` is deprecated! Use `dtype` instead!
Checkpoint kind: lora
Merging LoRA adapter into base weights for evaluation (CPU, once)...

Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]
Loading check…27 tokens truncated…:26:33 [llm.py:306] Supported_tasks: ['generate']

Adding requests:   0%|          | 0/8 [00:00<?, ?it/s]
Adding requests: 100%|██████████| 8/8 [00:00<00:00, 324.05it/s]

Processed prompts:   0%|          | 0/96 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]

#629

narration

The third checkpoint is now in the final 360-sample evaluation. After it completes, I’ll compute the replicated mean and run final syntax/diff/recipe checks; no additional experimental changes are planned.

#630

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolexec

args

{
  "input": "const r = await tools.write_stdin({session_id:32207,chars:\"\",yield_time_ms:30000,max_output_tokens:300});\ntext(r.output);\n"
}

exec result

Script running with cell ID 322
Wall time 11.0 seconds
Output:

#631

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait

args

{
  "cell_id": "322",
  "yield_time_ms": 30000,
  "max_tokens": 300
}

wait result

Script completed
Wall time 10.4 seconds
Output:

#632

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolexec

args

{
  "input": "const r = await tools.write_stdin({session_id:32207,chars:\"\",yield_time_ms:30000,max_output_tokens:300});\ntext(r.output);\n"
}

exec result

Script running with cell ID 323
Wall time 11.0 seconds
Output:

#633

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait

args

{
  "cell_id": "323",
  "yield_time_ms": 30000,
  "max_tokens": 300
}

wait result

Script completed
Wall time 8.0 seconds
Output:

#634

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolexec

args

{
  "input": "const r = await tools.write_stdin({session_id:32207,chars:\"\",yield_time_ms:30000,max_output_tokens:300});\ntext(r.output);\n"
}

exec result

Script running with cell ID 324
Wall time 11.0 seconds
Output:

#635

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait

args

{
  "cell_id": "324",
  "yield_time_ms": 30000,
  "max_tokens": 300
}

wait result

Script completed
Wall time 11.5 seconds
Output:

#636

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolexec

args

{
  "input": "const r = await tools.write_stdin({session_id:32207,chars:\"\",yield_time_ms:30000,max_output_tokens:300});\ntext(r.output);\n"
}

exec result

Script running with cell ID 325
Wall time 11.0 seconds
Output:

#637

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait

args

{
  "cell_id": "325",
  "yield_time_ms": 30000,
  "max_tokens": 300
}

wait result

Script completed
Wall time 10.0 seconds
Output:

#638

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolexec

args

{
  "input": "const r = await tools.write_stdin({session_id:32207,chars:\"\",yield_time_ms:30000,max_output_tokens:300});\ntext(r.output);\n"
}

exec result

Script running with cell ID 326
Wall time 11.0 seconds
Output:

#639

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait

args

{
  "cell_id": "326",
  "yield_time_ms": 30000,
  "max_tokens": 300
}

wait result

Script completed
Wall time 8.2 seconds
Output:

Processed prompts:  12%|█▎        | 12/96 [03:12<22:29, 16.07s/it, est. speed input: 9.71 toks/s, output: 436.71 toks/s]
Processed prompts:  25%|██▌       | 24/96 [03:20<08:22,  6.98s/it, est. speed input: 19.36 toks/s, output: 892.11 toks/s]

#640

narration

The final evaluation has begun returning shard progress. As with earlier runs, I’m waiting for all 360 samples rather than extrapolating from early shards.

#641

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolexec

args

{
  "input": "const r = await tools.write_stdin({session_id:32207,chars:\"\",yield_time_ms:30000,max_output_tokens:300});\ntext(r.output);\n"
}

exec result

Script running with cell ID 327
Wall time 11.0 seconds
Output:

#642

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait

args

{
  "cell_id": "327",
  "yield_time_ms": 30000,
  "max_tokens": 300
}

wait result

Script completed
Wall time 10.0 seconds
Output:

#643

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolexec

args

{
  "input": "const r = await tools.write_stdin({session_id:32207,chars:\"\",yield_time_ms:30000,max_output_tokens:300});\ntext(r.output);\n"
}

exec result

Script running with cell ID 328
Wall time 11.0 seconds
Output:

#644

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait

args

{
  "cell_id": "328",
  "yield_time_ms": 30000,
  "max_tokens": 300
}

wait result

Script completed
Wall time 12.0 seconds
Output:

Processed prompts:  14%|█▍        | 12/84 [04:32<27:16, 22.73s/it, est. speed input: 3.39 toks/s, output: 493.64 toks/s]

#645

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolexec

args

{
  "input": "const r = await tools.write_stdin({session_id:32207,chars:\"\",yield_time_ms:30000,max_output_tokens:300});\ntext(r.output);\n"
}

exec result

Script running with cell ID 329
Wall time 11.0 seconds
Output:

#646

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait

args

{
  "cell_id": "329",
  "yield_time_ms": 30000,
  "max_tokens": 300
}

wait result

Script completed
Wall time 4.0 seconds
Output:

#647

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolexec

args

{
  "input": "const r = await tools.write_stdin({session_id:32207,chars:\"\",yield_time_ms:30000,max_output_tokens:300});\ntext(r.output);\n"
}

exec result

Script running with cell ID 330
Wall time 11.0 seconds
Output:

#648

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait

args

{
  "cell_id": "330",
  "yield_time_ms": 30000,
  "max_tokens": 300
}

wait result

Script completed
Wall time 10.2 seconds
Output:

#649

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolexec

args

{
  "input": "const r = await tools.write_stdin({session_id:32207,chars:\"\",yield_time_ms:30000,max_output_tokens:300});\ntext(r.output);\n"
}

exec result

Script running with cell ID 331
Wall time 11.0 seconds
Output:

#650

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait

args

{
  "cell_id": "331",
  "yield_time_ms": 30000,
  "max_tokens": 300
}

wait result

Script completed
Wall time 12.8 seconds
Output:

Processed prompts:  14%|█▍        | 12/84 [06:39<39:58, 33.32s/it, est. speed input: 2.52 toks/s, output: 362.77 toks/s]
Processed prompts:  12%|█▎        | 12/96 [06:47<47:30, 33.94s/it, est. speed input: 4.36 toks/s, output: 319.96 toks/s]