Skip to content
RSI Bench
on-policy-self-distillation

gpt-5.6-sol

on-policy-self-distillation — graded by the task's verifier against a measured baseline.

backend
modal
variant
codex azure_ai/gpt-5.6-sol
metric
AIME24 avg@12 accuracy
actions
596
duration
7h 59m
recorded
2026-08-03

run idopsd-aime24-4gpus__kuc95M7

Everything gpt-5.6-sol did on this task, in order.

download JSON ↗
codexgpt-5.6-sol669 turns73 narration596 reasoning596 action596 observation1 final
#101
reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolexec
args
{
  "input": "const r = await tools.write_stdin({session_id:31496,chars:\"\",yield_time_ms:60000,max_output_tokens:500});\ntext(JSON.stringify(r));\n"
}
exec result
Script running with cell ID 56
Wall time 11.0 seconds
Output:
#102
reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait
args
{
  "cell_id": "56",
  "yield_time_ms": 60000,
  "max_tokens": 500
}
wait result
Script completed
Wall time 46.2 seconds
Output:
{"chunk_id":"69ac06","wall_time_seconds":60.000835875,"session_id":31496,"original_token_count":0,"output":""}
#103
reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolexec
args
{
  "input": "const r = await tools.exec_command({cmd:\"python - <<'PY'\\nimport json\\nx=json.load(open('/opt/eval_assets/aime24.json'))\\nprint(type(x),len(x),x[0].keys())\\nPY\\ncat /workspace/.timer/remaining_secs\",\"workdir\":\"/workspace\",\"yield_time_ms\":10000,\"max_output_tokens\":1000});\ntext(r.output);\n"
}
exec result
Script completed
Wall time 0.1 seconds
Output:
<class 'list'> 30 dict_keys(['id', 'problem', 'answer'])
19046
#104
reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolexec
args
{
  "input": "const r = await tools.write_stdin({session_id:31496,chars:\"\",yield_time_ms:60000,max_output_tokens:500});\ntext(JSON.stringify(r));\n"
}
exec result
Script running with cell ID 58
Wall time 11.0 seconds
Output:
#105
reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait
args
{
  "cell_id": "58",
  "yield_time_ms": 60000,
  "max_tokens": 500
}
wait result
Script completed
Wall time 41.1 seconds
Output:
{"chunk_id":"6c88c5","wall_time_seconds":60.001941212,"session_id":31496,"original_token_count":63,"output":"\rProcessed prompts:  14%|█▍        | 12/84 [06:37<39:47, 33.16s/it, est. speed input: 2.20 toks/s, output: 340.86 toks/s]\rProcessed prompts:  12%|█▎        | 12/96 [06:39<46:33, 33.25s/it, est. speed input: 4.45 toks/s, output: 300.63 toks/s]"}
#106
narration

Generation is progressing normally but unevenly because some AIME samples use very long thinking traces (two shards are around 12–14% after six minutes). This is exactly why training-time response length must be judged by final accuracy rather than completion-speed or length statistics.

#107
reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolexec
args
{
  "input": "const r = await tools.write_stdin({session_id:31496,chars:\"\",yield_time_ms:60000,max_output_tokens:500});\ntext(JSON.stringify(r));\n"
}
exec result
Script running with cell ID 59
Wall time 11.0 seconds
Output:
#108
reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait
args
{
  "cell_id": "59",
  "yield_time_ms": 60000,
  "max_tokens": 500
}
wait result
Script completed
Wall time 34.8 seconds
Output:
{"chunk_id":"a2c0fc","wall_time_seconds":60.001492089,"session_id":31496,"original_token_count":64,"output":"\rProcessed prompts:  25%|██▌       | 24/96 [07:39<23:41, 19.75s/it, est. speed input: 8.44 toks/s, output: 416.38 toks/s]\rProcessed prompts:  29%|██▊       | 24/84 [08:00<17:41, 17.70s/it, est. speed input: 3.75 toks/s, output: 561.94 toks/s]"}
#109
reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolexec
args
{
  "input": "const r = await tools.write_stdin({session_id:31496,chars:\"\",yield_time_ms:60000,max_output_tokens:500});\ntext(JSON.stringify(r));\n"
}
exec result
Script running with cell ID 60
Wall time 11.0 seconds
Output:
#110
reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait
args
{
  "cell_id": "60",
  "yield_time_ms": 60000,
  "max_tokens": 500
}
wait result
Script completed
Wall time 45.6 seconds
Output:
{"chunk_id":"4af0f7","wall_time_seconds":60.00125116,"session_id":31496,"original_token_count":33,"output":"\rProcessed prompts:  43%|████▎     | 36/84 [09:18<10:04, 12.59s/it, est. speed input: 5.01 toks/s, output: 804.32 toks/s]"}
#111
reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolexec
args
{
  "input": "const r = await tools.write_stdin({session_id:31496,chars:\"\",yield_time_ms:60000,max_output_tokens:500});\ntext(JSON.stringify(r));\n"
}
exec result
Script running with cell ID 61
Wall time 11.0 seconds
Output:
#112
reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait
args
{
  "cell_id": "61",
  "yield_time_ms": 60000,
  "max_tokens": 500
}
wait result
Script completed
Wall time 45.8 seconds
Output:
{"chunk_id":"b5e747","wall_time_seconds":60.001787115,"session_id":31496,"original_token_count":65,"output":"\rProcessed prompts:  14%|█▍        | 12/84 [09:37<57:46, 48.14s/it, est. speed input: 1.74 toks/s, output: 311.61 toks/s]\rProcessed prompts:  57%|█████▋    | 48/84 [10:20<05:47,  9.65s/it, est. speed input: 6.52 toks/s, output: 1196.51 toks/s]"}
#113
reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolexec
args
{
  "input": "const r = await tools.write_stdin({session_id:31496,chars:\"\",yield_time_ms:60000,max_output_tokens:400});\ntext(JSON.stringify(r));\n"
}
exec result
Script running with cell ID 62
Wall time 11.0 seconds
Output:
#114
reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait
args
{
  "cell_id": "62",
  "yield_time_ms": 60000,
  "max_tokens": 400
}
wait result
Script completed
Wall time 45.3 seconds
Output:
{"chunk_id":"e45304","wall_time_seconds":60.001487819,"session_id":31496,"original_token_count":33,"output":"\rProcessed prompts:  38%|███▊      | 36/96 [10:59<18:21, 18.35s/it, est. speed input: 8.08 toks/s, output: 577.45 toks/s]"}
#115
narration

The evaluation remains healthy; shard imbalance is from the fixed round-robin problem assignment, with one difficult shard generating much longer traces. No failures or format errors have appeared.

#116
reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolexec
args
{
  "input": "const r = await tools.write_stdin({session_id:31496,chars:\"\",yield_time_ms:60000,max_output_tokens:400});\ntext(JSON.stringify(r));\n"
}
exec result
Script running with cell ID 63
Wall time 11.0 seconds
Output:
#117
reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait
args
{
  "cell_id": "63",
  "yield_time_ms": 60000,
  "max_tokens": 400
}
wait result
Script completed
Wall time 43.4 seconds
Output:
{"chunk_id":"423812","wall_time_seconds":60.001493151,"session_id":31496,"original_token_count":311,"output":"\rProcessed prompts:  29%|██▊       | 24/84 [11:42<25:55, 25.93s/it, est. speed input: 3.26 toks/s, output: 531.57 toks/s]\rProcessed prompts:  50%|█████     | 48/96 [11:57<10:24, 13.02s/it, est. speed input: 8.61 toks/s, output: 766.55 toks/s]\rProcessed prompts:  71%|███████▏  | 60/84 [11:58<03:38,  9.11s/it, est. speed input: 8.34 toks/s, output: 1319.84 toks/s]\rProcessed prompts:  62%|██████▎   | 60/96 [12:00<05:02,  8.41s/it, est. speed input: 10.04 toks/s, output: 1115.56 toks/s]\rProcessed prompts:  86%|████████▌ | 72/84 [12:06<01:15,  6.26s/it, est. speed input: 9.79 toks/s, output: 1778.70 toks/s]\rProcessed prompts: 100%|██████████| 84/84 [12:35<00:00,  4.99s/it, est. speed input: 11.56 toks/s, output: 2030.19 toks/s]\rProcessed prompts: 100%|██████████| 84/84 [12:35<00:00,  4.99s/it, est. speed input: 11.56 toks/s, output: 2030.19 toks/s]\rProcessed prompts: 100%|██████████| 84/84 [12:35<00:00,  8.99s/it, est. speed input: 11.56 toks/s, output: 2030.19 toks/s]\n\rProcessed prompts:  75%|███████▌  | 72/96 [12:36<02:37,  6.57s/it, est. speed input: 10.77 toks/s, output: 1375.18 toks/s]"}
#118
reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolexec
args
{
  "input": "const r = await tools.write_stdin({session_id:31496,chars:\"\",yield_time_ms:60000,max_output_tokens:400});\ntext(JSON.stringify(r));\n"
}
exec result
Script running with cell ID 64
Wall time 11.0 seconds
Output:
#119
reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait
args
{
  "cell_id": "64",
  "yield_time_ms": 60000,
  "max_tokens": 400
}
wait result
Script completed
Wall time 43.6 seconds
Output:
{"chunk_id":"d5e433","wall_time_seconds":60.002187089,"session_id":31496,"original_token_count":273,"output":"\rProcessed prompts:  25%|██▌       | 24/96 [12:47<38:06, 31.75s/it, est. speed input: 5.47 toks/s, output: 490.57 toks/s]\rProcessed prompts:  38%|███▊      | 36/96 [12:52<17:26, 17.44s/it, est. speed input: 6.76 toks/s, output: 811.38 toks/s]\rProcessed prompts:  88%|████████▊ | 84/96 [13:00<01:00,  5.06s/it, est. speed input: 13.06 toks/s, output: 1760.98 toks/s]\rProcessed prompts: 100%|██████████| 96/96 [13:05<00:00,  3.60s/it, est. speed input: 19.32 toks/s, output: 2172.88 toks/s]\rProcessed prompts: 100%|██████████| 96/96 [13:05<00:00,  3.60s/it, est. speed input: 19.32 toks/s, output: 2172.88 toks/s]\rProcessed prompts: 100%|██████████| 96/96 [13:05<00:00,  8.19s/it, est. speed input: 19.32 toks/s, output: 2172.88 toks/s]\n\rProcessed prompts:  43%|████▎     | 36/84 [13:28<14:30, 18.14s/it, est. speed input: 4.62 toks/s, output: 805.80 toks/s]\rProcessed prompts:  50%|█████     | 48/96 [13:43<09:48, 12.25s/it, est. speed input: 8.04 toks/s, output: 1031.99 toks/s]"}
#120
reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolexec
args
{
  "input": "const r = await tools.write_stdin({session_id:31496,chars:\"\",yield_time_ms:60000,max_output_tokens:400});\ntext(JSON.stringify(r));\n"
}
exec result
Script running with cell ID 65
Wall time 11.0 seconds
Output:
#121
reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait
args
{
  "cell_id": "65",
  "yield_time_ms": 60000,
  "max_tokens": 400
}
wait result
Script completed
Wall time 45.9 seconds
Output:
{"chunk_id":"674eaf","wall_time_seconds":60.002520125,"session_id":31496,"original_token_count":175,"output":"\rProcessed prompts:  57%|█████▋    | 48/84 [13:57<07:09, 11.92s/it, est. speed input: 5.89 toks/s, output: 1115.66 toks/s]\rProcessed prompts:  71%|███████▏  | 60/84 [14:12<03:13,  8.08s/it, est. speed input: 7.18 toks/s, output: 1488.93 toks/s]\rProcessed prompts:  86%|████████▌ | 72/84 [14:34<01:11,  5.97s/it, est. speed input: 10.01 toks/s, output: 1779.78 toks/s]\rProcessed prompts: 100%|██████████| 84/84 [14:35<00:00,  5.97s/it, est. speed input: 11.37 toks/s, output: 2133.02 toks/s]\rProcessed prompts: 100%|██████████| 84/84 [14:35<00:00, 10.42s/it, est. speed input: 11.37 toks/s, output: 2133.02 toks/s]\n"}
#122
reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolexec
args
{
  "input": "const r = await tools.write_stdin({session_id:31496,chars:\"\",yield_time_ms:60000,max_output_tokens:400});\ntext(JSON.stringify(r));\n"
}
exec result
Script running with cell ID 66
Wall time 11.0 seconds
Output:
#123
reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolwait
args
{
  "cell_id": "66",
  "yield_time_ms": 60000,
  "max_tokens": 400
}
wait result
Script completed
Wall time 44.4 seconds
Output:
{"chunk_id":"304134","wall_time_seconds":60.000555634,"session_id":31496,"original_token_count":34,"output":"\rProcessed prompts:  62%|██████▎   | 60/96 [15:11<06:16, 10.46s/it, est. speed input: 8.90 toks/s, output: 1211.64 toks/s]"}
#124
narration

Three of four evaluation shards have finished; the final shard is past 60%. I’ll report the exact avg@12 as soon as scoring/merging completes, then immediately launch the next 100-step candidate based on that evidence.

#125
reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolexec
args
{
  "input": "const r = await tools.write_stdin({session_id:31496,chars:\"\",yield_time_ms:60000,max_output_tokens:400});\ntext(JSON.stringify(r));\n"
}
exec result
Script running with cell ID 67
Wall time 11.0 seconds
Output: