nano-gpt-data-curation

claude-opus-5

nano-gpt-data-curation — graded by the task's verifier against a measured baseline.

backend: modal
variant: claude claude-opus-5
metric: held-out multi-domain perplexity
actions: 71
duration: 1h 2m
recorded: 2026-08-04

run iddclm-nano-curation__nEEkAJz

Everything claude-opus-5 did on this task, in order.

download JSON ↗

claude-codeclaude-opus-562 turns40 narration71 action71 observation22 reasoning1 final

#51

narration

Gated HTML docs barely exist (126k tokens), but ungated has 2.7M — and unlike ellipsis spam, raw-web HTML markup is genuinely structurally similar to the target's. Let me add an HTML lane, defaulting to off so the current submission stays reproducible.

toolBashTest HTML surface lane at three settings

args

{
  "command": "python - <<'EOF'\np='/workspace/submission/curate.py'; s=open(p).read()\ns=s.replace('PROSE_RANK = os.environ.get(\"PROSE_RANK\", \"general\")',\n'''PROSE_RANK = os.environ.get(\"PROSE_RANK\", \"general\")\n\n# Second surface lane, for the technical-Q&A quarter: that quarter is raw StackExchange\n# HTML, where markup tokens (\"</\", \"><\", \"&gt;\", ' href', \"<pre><code>\") are 11.7% of\n# tokens against 0.081% of the pool. Almost no pool document that passes the prose\n# quality gate contains markup, so this lane is drawn ungated.\nHTML = float(os.environ.get(\"HTML\", \"0.0\"))\nHTML_MIN = float(os.environ.get(\"HTML_MIN\", \"0.02\"))''')\ns=s.replace('def build_selection(lo, keep, ntok, ids, sprate=None):',\n            'def build_selection(lo, keep, ntok, ids, sprate=None, htmlrate=None):')\ns=s.replace('''    quotas = np.array(weights) * BUDGET * OVERFILL''',\n'''    html_sel = np.zeros(len(ntok), dtype=bool)\n    if HTML > 0 and htmlrate is not None:\n        cand = np.nonzero((ntok >= 128) & (htmlrate >= HTML_MIN) & ~style_sel)[0]\n        cand = cand[np.argsort(-htmlrate[cand])]\n        cum = np.cumsum(ntok[cand] + 1)\n        take = cand[:int(np.searchsorted(cum, HTML * BUDGET * OVERFILL) + 1)]\n        html_sel[take] = True\n        html_order = cand\n        weights = [w * (1.0 - HTML) for w in weights] + [HTML]\n        print(f\"  {'surface_html':14s} pool_cands={len(cand):6d} taken={len(take):6d} \"\n              f\"tokens={int(cum[min(len(take), len(cum)) - 1]):,}\")\n    quotas = np.array(weights) * BUDGET * OVERFILL''')\n# style lane must be appended before the html lane to match weight order\ns=s.replace('''    if style_sel.any():\n        lanes.append([int(i) for i in lane_order[:int(style_sel.sum())]])''',\n'''    if style_sel.any():\n        lanes.append([int(i) for i in lane_order[:int(style_sel.sum())]])\n    if html_sel.any():\n        lanes.append([int(i) for i in html_order[:int(html_sel.sum())]])''')\ns=s.replace('    cand = np.nonzero(keep & (best == d) & ~style_sel)[0]',\n            '    cand = np.nonzero(keep & (best == d) & ~style_sel & ~html_sel)[0]')\ns=s.replace('    sel = build_selection(lo, keep, ntok, ids, sprate)',\n            '    htmlrate = np.load(f\"{W}/htmlrate.npy\")\\n    sel = build_selection(lo, keep, ntok, ids, sprate, htmlrate)')\nopen(p,'w').write(s)\nEOF\n# quotas for 4 register lanes must be computed before html reweighting -> verify ordering is consistent\npython -c \"\nimport re\ns=open('/workspace/submission/curate.py').read()\ni=s.index('html_sel = np.zeros'); j=s.index('quotas = np.array(weights)')\nprint('html block precedes quotas:', i<j)\nprint('lane count logic ok:', s.count('lanes.append')==3)\n\"\nfor cfg in \"0.04 0.02\" \"0.08 0.02\" \"0.04 0.01\"; do set -- $cfg\n  HTML=$1 HTML_MIN=$2 OUT_SEL=/tmp/sel_H$1_$2.json python submission/curate.py | tail -1; done\n( bash work/ev.sh H0.04_0.02 /tmp/sel_H0.04_0.02.json & bash work/ev.sh H0.08_0.02 /tmp/sel_H0.08_0.02.json & bash work/ev.sh H0.04_0.01 /tmp/sel_H0.04_0.01.json & wait )",
  "description": "Test HTML surface lane at three settings",
  "timeout": 900000
}

Bash result

html block precedes quotas: True
lane count logic ok: True
wrote /tmp/sel_H0.04_0.02.json
wrote /tmp/sel_H0.08_0.02.json
wrote /tmp/sel_H0.04_0.01.json
[H0.04_0.01] QUARTER 0 loss 6.4556 ppl 636.27 QUARTER 1 loss 5.5829 ppl 265.85 QUARTER 2 loss 5.6618 ppl 287.67 QUARTER 3 loss 5.4932 ppl 243.04 HELDOUT_PPL 330.328  loss 5.8001  train_s 161  /tmp/tr_H0.04_0.01.npy [H0.04_0.02] QUARTER 0 loss 6.4291 ppl 619.63 QUARTER 1 loss 5.5756 ppl 263.91 QUARTER 2 loss 5.6553 ppl 285.79 QUARTER 3 loss 5.5124 ppl 247.74 HELDOUT_PPL 328.617  loss 5.7949  train_s 161  /tmp/tr_H0.04_0.02.npy 
[H0.08_0.02] QUARTER 0 loss 6.4003 ppl 602.04 QUARTER 1 loss 5.6021 ppl 270.98 QUARTER 2 loss 5.6859 ppl 294.67 QUARTER 3 loss 5.4953 ppl 243.55 HELDOUT_PPL 329.272  loss 5.7969  train_s 161  /tmp/tr_H0.08_0.02.npy

[stdout]
html block precedes quotas: True
lane count logic ok: True
wrote /tmp/sel_H0.04_0.02.json
wrote /tmp/sel_H0.08_0.02.json
wrote /tmp/sel_H0.04_0.01.json
[H0.04_0.01] QUARTER 0 loss 6.4556 ppl 636.27 QUARTER 1 loss 5.5829 ppl 265.85 QUARTER 2 loss 5.6618 ppl 287.67 QUARTER 3 loss 5.4932 ppl 243.04 HELDOUT_PPL 330.328  loss 5.8001  train_s 161  /tmp/tr_H0.04_0.01.npy [H0.04_0.02] QUARTER 0 loss 6.4291 ppl 619.63 QUARTER 1 loss 5.5756 ppl 263.91 QUARTER 2 loss 5.6553 ppl 285.79 QUARTER 3 loss 5.5124 ppl 247.74 HELDOUT_PPL 328.617  loss 5.7949  train_s 161  /tmp/tr_H0.04_0.02.npy 
[H0.08_0.02] QUARTER 0 loss 6.4003 ppl 602.04 QUARTER 1 loss 5.6021 ppl 270.98 QUARTER 2 loss 5.6859 ppl 294.67 QUARTER 3 loss 5.4953 ppl 243.55 HELDOUT_PPL 329.272  loss 5.7969  train_s 161  /tmp/tr_H0.08_0.02.npy
[metadata] {"noOutputExpected": false}

#52

narration

The HTML lane works: Q3 loss 5.639 → 5.512, total 328.62. Let me lock that in and run two more variants in the background.

toolBashSet HTML default and regenerate submission

args

{
  "command": "python - <<'EOF'\np='/workspace/submission/curate.py'; s=open(p).read()\ns=s.replace('HTML = float(os.environ.get(\"HTML\", \"0.0\"))','HTML = float(os.environ.get(\"HTML\", \"0.04\"))')\ns=s.replace('''# quality gate contains markup, so this lane is drawn ungated.''',\n'''# quality gate contains markup, so this lane is drawn ungated. Tuned on the disclosed\n# dev target: 0.04 at >=0.02 markup rate beat 0 (333.2), 0.08 (329.3) and >=0.01 (330.3),\n# and it is the one lane where ungated documents help - raw web pages carry markup in\n# genuine prose context, unlike the boilerplate that dominates the spacing signal.''')\ns=s.replace('''  3. SURFACE-FORM LANE (`STYLE`, see below)''','''  3. SURFACE-FORM LANES (`STYLE`, `HTML`, see below)''')\ns=s.replace('''     three, and ~0.5 nat of that excess is surface form rather than content.''',\n'''     three, and ~0.5 nat of that excess is surface form rather than content. The\n     technical-Q&A quarter has the same problem in a different alphabet - it is raw\n     StackExchange HTML, whose markup tokens are 11.7% of that quarter against 0.081%\n     of the pool - so a second, smaller lane supplies pool documents carrying markup.''')\ns=s.replace('''  qstats.npy (step 1). Run `python work/tok_pool.py && python work/score.py &&\n  python work/quality.py` first; see REPRODUCE.md.''',\n'''  qstats.npy (step 1), sprate.npy + htmlrate.npy (step 3). Run\n  `python work/tok_pool.py && python work/quality.py && python work/score.py &&\n  python work/sprate.py && python work/htmlrate.py` first; see REPRODUCE.md.''')\nopen(p,'w').write(s)\nEOF\npython submission/curate.py\nTOKENIZERS_PARALLELISM=false python pack_selection.py submission/selection.json /tmp/tr_sub.npy 2>/dev/null | tail -1\npython -c \"\nimport json,numpy as np\ns=json.load(open('submission/selection.json')); ids=set(int(x) for x in np.load('work/tok_ids.npy'))\na=np.load('/tmp/tr_sub.npy'); b=np.load('/tmp/tr_H0.04_0.02.npy')\nprint('ids',len(s),'| unique',len(set(s))==len(s),'| in-range',set(s)<=ids,'| tokens',len(a),'| bitmatch H0.04_0.02:',(a==b).all())\n\"; cat .timer/remaining_secs",
  "description": "Set HTML default and regenerate submission",
  "timeout": 900000
}

Bash result

quality gate: 88,490 / 182,016 docs survive (48.6%)
mixture: encyclopedic=0.25, web_prose=0.25, news=0.25, technical_qa=0.25, style=0.16
  surface_style  pool_cands=  3548 taken=  3548 tokens=3,529,081
  surface_html   pool_cands=   775 taken=   657 tokens=1,440,335
  encyclopedic   pool_cands= 14164 taken=  6605 tokens=7,258,225
  web_prose      pool_cands= 50842 taken=  7387 tokens=7,258,728
  news           pool_cands= 12820 taken= 11186 tokens=7,269,873
  technical_qa   pool_cands=  7076 taken=  7076 tokens=4,931,629
selection: 36,459 ids, 31,687,871 tokens (2.64x the 12,000,000 budget)
wrote /workspace/submission/selection.json
packed 12000000 tokens from 36459 ids -> /tmp/tr_sub.npy
ids 36459 | unique True | in-range True | tokens 12000000 | bitmatch H0.04_0.02: True
2416

[stdout]
quality gate: 88,490 / 182,016 docs survive (48.6%)
mixture: encyclopedic=0.25, web_prose=0.25, news=0.25, technical_qa=0.25, style=0.16
  surface_style  pool_cands=  3548 taken=  3548 tokens=3,529,081
  surface_html   pool_cands=   775 taken=   657 tokens=1,440,335
  encyclopedic   pool_cands= 14164 taken=  6605 tokens=7,258,225
  web_prose      pool_cands= 50842 taken=  7387 tokens=7,258,728
  news           pool_cands= 12820 taken= 11186 tokens=7,269,873
  technical_qa   pool_cands=  7076 taken=  7076 tokens=4,931,629
selection: 36,459 ids, 31,687,871 tokens (2.64x the 12,000,000 budget)
wrote /workspace/submission/selection.json
packed 12000000 tokens from 36459 ids -> /tmp/tr_sub.npy
ids 36459 | unique True | in-range True | tokens 12000000 | bitmatch H0.04_0.02: True
2416
[metadata] {"noOutputExpected": false}

#53

narration

Submission locked and bit-verified. Let me create the htmlrate.py script I referenced, and launch two more variants in the background.

toolWrite

args

{
  "file_path": "/workspace/work/htmlrate.py",
  "content": "\"\"\"Per-document rate of HTML-markup tokens in the pool.\n\nThe technical-Q&A quarter of the target is raw StackExchange HTML: the markup tokens\nbelow are 11.7% of that quarter's tokens against 0.081% of the pool. Those tokens are\nwhere a model trained on markup-free prose spends its nats on that quarter.\n\"\"\"\nimport numpy as np\nfrom transformers import AutoTokenizer\n\nW = \"/workspace/work\"\nFRAGS = [\"</\", \"><\", \"></\", \".</\", \"gt\", \"lt\", \" href\", '=\"', \"pre\", \"code\", \"<\", \">\"]\n\ntok = AutoTokenizer.from_pretrained(\"gpt2\")\nids = []\nfor f in FRAGS:\n    e = tok(f, add_special_tokens=False)[\"input_ids\"]\n    if len(e) == 1:\n        ids.append(e[0])\n\nflat = np.load(f\"{W}/tok_flat.npy\")\noffs = np.load(f\"{W}/tok_offs.npy\")\nhit = np.isin(flat, ids).astype(np.int32)\nper_doc = np.add.reduceat(hit, offs[:-1])\nrate = (per_doc / np.maximum(np.diff(offs), 1)).astype(np.float32)\nnp.save(f\"{W}/htmlrate.npy\", rate)\nprint(f\"markup ids {ids}\\npool overall rate {100 * hit.mean():.4f}%   \"\n      f\"docs >=2%: {(rate >= 0.02).sum()}\")\n"
}

Write result

File created successfully at: /workspace/work/htmlrate.py (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/workspace/work/htmlrate.py", "content": "\"\"\"Per-document rate of HTML-markup tokens in the pool.\n\nThe technical-Q&A quarter of the target is raw StackExchange HTML: the markup tokens\nbelow are 11.7% of that quarter's tokens against 0.081% of the pool. Those tokens are\nwhere a model trained on markup-free prose spends its nats on that quarter.\n\"\"\"\nimport numpy as np\nfrom transformers import AutoTokenizer\n\nW = \"/workspace/work\"\nFRAGS = [\"</\", \"><\", \"></\", \".</\", \"gt\", \"lt\", \" href\", '=\"', \"pre\", \"code\", \"<\", \">\"]\n\ntok = AutoTokenizer.from_pretrained(\"gpt2\")\nids = []\nfor f in FRAGS:\n    e = tok(f, add_special_tokens=False)[\"input_ids\"]\n    if len(e) == 1:\n        ids.append(e[0])\n\nflat = np.load(f\"{W}/tok_flat.npy\")\noffs = np.load(f\"{W}/tok_offs.npy\")\nhit = np.isin(flat, ids).astype(np.int32)\nper_doc = np.add.reduceat(hit, offs[:-1])\nrate = (per_doc / np.maximum(np.diff(offs), 1)).astype(np.float32)\nnp.save(f\"{W}/htmlrate.npy\", rate)\nprint(f\"markup ids {ids}\\npool overall rate {100 * hit.mean():.4f}%   \"\n      f\"docs >=2%: {(rate >= 0.02).sum()}\")\n", "structuredPatch": [], "originalFile": null, "userModified": false}

#54

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolBashVerify htmlrate script, launch two extra configs

args

{
  "command": "cp work/htmlrate.npy /tmp/hr_backup.npy && python work/htmlrate.py && python -c \"\nimport numpy as np; print('htmlrate.py reproduces submission array:', np.allclose(np.load('work/htmlrate.npy'), np.load('/tmp/hr_backup.npy')))\n\"\nHTML=0.06 HTML_MIN=0.02 OUT_SEL=/tmp/sel_H6.json python submission/curate.py >/dev/null\nHTML=0.04 HTML_MIN=0.02 STYLE=0.18 OUT_SEL=/tmp/sel_H7.json python submission/curate.py >/dev/null\nnohup bash -c '( bash work/ev.sh H6 /tmp/sel_H6.json & bash work/ev.sh H7 /tmp/sel_H7.json & wait )' > /tmp/extra.log 2>&1 &\necho \"launched extras\"",
  "description": "Verify htmlrate script, launch two extra configs",
  "timeout": 300000
}

Bash result

markup ids [3556, 6927, 12240, 25970, 13655, 2528, 13291, 2625, 3866, 8189, 27, 29]
pool overall rate 0.0812%   docs >=2%: 826
htmlrate.py reproduces submission array: True
launched extras

[stdout]
markup ids [3556, 6927, 12240, 25970, 13655, 2528, 13291, 2625, 3866, 8189, 27, 29]
pool overall rate 0.0812%   docs >=2%: 826
htmlrate.py reproduces submission array: True
launched extras
[metadata] {"noOutputExpected": false}

#55

narration

Now updating the writeup with the HTML-lane result while those run.

toolBashUpdate claim.md and REPRODUCE.md with HTML lane results

args

{
  "command": "python - <<'PYEOF'\np='/workspace/claim.md'; s=open(p).read()\ns=s.replace(\"\"\"**Result.** Dev perplexity **333.2** vs **470.1** for a random selection of the same\npool at the same 12M-token budget (−29%).\"\"\",\n\"\"\"**Result.** Dev perplexity **328.6** vs **470.1** for a random selection of the same\npool at the same 12M-token budget (−30%).\"\"\")\ns=s.replace(\"\"\"| gate + classifier, uniform register mixture | 355.2 |\n| **+ surface-form lane (final submission)** | **333.2** |\"\"\",\n\"\"\"| gate + classifier, uniform register mixture | 355.2 |\n| + spacing surface lane (`STYLE=0.16`) | 333.2 |\n| **+ HTML surface lane (`HTML=0.04`) — final submission** | **328.6** |\"\"\")\ns=s.replace(\"\"\"The strong claim is that term 2 is not a rounding error but the single largest\nremaining component of the loss here, and that generic \"quality filtering\" (the usual\nmove) captures almost none of either term: it is worth only 11 PPL of the 137 recovered.\"\"\",\n\"\"\"The strong claim is that term 2 is not a rounding error but the single largest\nremaining component of the loss here, and that generic \"quality filtering\" (the usual\nmove) captures almost none of either term: it is worth only 11 PPL of the 142 recovered.\nTerm 2 is also *general*, not a one-off: it was predicted and then confirmed a second\ntime, in a different alphabet, on a different quarter of the target.\"\"\")\n# strengthen the mechanism section with the HTML replication\ns=s.replace(\"\"\"A fourth observable falls out of the register model\"\"\",\n\"\"\"- **P3b (the replication) — the same term should exist wherever the target's surface\n  convention differs, and should be findable *a priori*.** Ranking tokens by excess\n  probability mass `p_target(t)·log[p_target(t)/p_pool(t)]` is a purely\n  distributional screen that needs no training run. Applied to the technical-Q&A\n  quarter it flags HTML markup — `</` at 929× the pool rate, `code` 435×, `.</`\n  4307× — totalling **1.27 nats** over the top 22 tokens, *larger* than the\n  encyclopedic quarter's spacing signature (0.72 nats). Prediction: a small lane of\n  markup-bearing pool documents lowers that quarter's loss specifically.\n  *Observed*: quarter-3 loss **5.639 → 5.512**, total **333.2 → 328.6**, with the\n  other three quarters unmoved. The screen located a second, independent instance of\n  the mechanism before any training run was spent on it.\n\n  Note this lane is drawn **ungated**, the exact opposite of P3's conclusion for\n  spacing, and that difference is itself predicted by the \"context, not frequency\"\n  mechanism: raw web pages carry markup wrapped around ordinary prose, which is the\n  target's context, whereas the densest spaced-punctuation documents are `. . .`\n  boilerplate, which is not.\n\nA fourth observable falls out of the register model\"\"\")\ns=s.replace(\"\"\"- **Check the target's tokenization before curating for it.** A single decode of the\n  held-out set and a histogram of its token ids exposed a 140× discrepancy in ` ,`\n  frequency that no quality filter, classifier, or dedup pass would ever surface.\"\"\",\n\"\"\"- **Screen the target's token histogram against the pool's before curating.** Ranking\n  by excess probability mass costs one `bincount` per corpus and no GPU time, and it\n  found both wins here: a 140× discrepancy in ` ,` and a 929× one in `</`, neither of\n  which any quality filter, classifier, or dedup pass would ever surface. Do this\n  first; it tells you which term dominates before you spend a training run.\"\"\")\ns=s.replace(\"\"\"  properties of the domain rather than of the sample, so they should carry to the hidden\n  set, but the specific optima (STYLE=0.16, uniform mixture) are fitted to ±5 PPL of\n  run-to-run spread and should not be read as precise.\"\"\",\n\"\"\"  properties of the domain rather than of the sample, so they should carry to the hidden\n  set, but the specific optima (STYLE=0.16, HTML=0.04, uniform mixture) are fitted to\n  ±5 PPL of run-to-run spread and should not be read as precise.\"\"\")\ns=s.replace(\"\"\"  and the gain tracks the spacing rate, peaking\n  and then reversing (0 → 355.2, 0.08 → 343.4, **0.16 → 333.2**, 0.20 → 340.4,\n  0.24 → 342.2) — a purity/volume trade-off, not a monotone topic effect.\"\"\",\n\"\"\"  and the gain tracks the spacing rate, peaking\n  and then reversing (0 → 355.2, 0.08 → 343.4, **0.16 → 333.2**, 0.20 → 340.4,\n  0.24 → 342.2) — a purity/volume trade-off, not a monotone topic effect. The HTML lane\n  behaves the same way (0 → 333.2, **0.04 → 328.6**, 0.08 → 329.3) and moves only the\n  quarter it targets, which a topic-confound story does not predict.\"\"\")\nopen(p,'w').write(s)\nprint('claim.md updated')\nPYEOF\npython - <<'PYEOF'\np='/workspace/REPRODUCE.md'; s=open(p).read()\ns=s.replace(\"\"\"python work/sprate.py       # per-doc rate of space-prefixed punctuation -> sprate.npy\"\"\",\n\"\"\"python work/sprate.py       # per-doc rate of space-prefixed punctuation -> sprate.npy\npython work/htmlrate.py     # per-doc rate of HTML markup tokens -> htmlrate.npy\"\"\")\ns=s.replace(\"\"\"`STYLE_MIN`, `STYLE_RANK`, `STYLE_GATE`, `PROSE_RANK`, `SCORES`)\"\"\",\n\"\"\"`STYLE_MIN`, `STYLE_RANK`, `STYLE_GATE`, `HTML`, `HTML_MIN`, `PROSE_RANK`, `SCORES`)\"\"\")\ns=s.replace(\"\"\"| gate + classifier, uniform mixture | 355.2 |\n| **+ surface-form lane (submitted)** | **333.2** |\"\"\",\n\"\"\"| gate + classifier, uniform mixture | 355.2 |\n| + spacing surface lane (`STYLE=0.16`) | 333.2 |\n| **+ HTML surface lane (`HTML=0.04`), submitted** | **328.6** |\"\"\")\nopen(p,'w').write(s)\nprint('REPRODUCE.md updated')\nPYEOF\nsleep 100; cat /tmp/extra.log | grep -E \"HELDOUT|QUARTER\"; cat .timer/remaining_secs",
  "description": "Update claim.md and REPRODUCE.md with HTML lane results",
  "timeout": 600000
}

Bash result

claim.md updated
REPRODUCE.md updated
[H7] QUARTER 0 loss 6.4255 ppl 617.39 QUARTER 1 loss 5.5867 ppl 266.86 QUARTER 2 loss 5.6730 ppl 290.90 QUARTER 3 loss 5.5536 ppl 258.16 HELDOUT_PPL 334.054  loss 5.8113  train_s 109  /tmp/tr_H7.npy [H6] QUARTER 0 loss 6.4212 ppl 614.74 QUARTER 1 loss 5.5683 ppl 262.00 QUARTER 2 loss 5.6440 ppl 282.60 QUARTER 3 loss 5.4891 ppl 242.03 HELDOUT_PPL 324.403  loss 5.7820  train_s 109  /tmp/tr_H6.npy 
2266

[stdout]
claim.md updated
REPRODUCE.md updated
[H7] QUARTER 0 loss 6.4255 ppl 617.39 QUARTER 1 loss 5.5867 ppl 266.86 QUARTER 2 loss 5.6730 ppl 290.90 QUARTER 3 loss 5.5536 ppl 258.16 HELDOUT_PPL 334.054  loss 5.8113  train_s 109  /tmp/tr_H7.npy [H6] QUARTER 0 loss 6.4212 ppl 614.74 QUARTER 1 loss 5.5683 ppl 262.00 QUARTER 2 loss 5.6440 ppl 282.60 QUARTER 3 loss 5.4891 ppl 242.03 HELDOUT_PPL 324.403  loss 5.7820  train_s 109  /tmp/tr_H6.npy 
2266
[metadata] {"noOutputExpected": false}

#56

narration

H6 (HTML=0.06) is better still: 324.40. Let me switch to it and confirm via the official path.

toolBashSwitch to HTML=0.06 and verify

args

{
  "command": "python - <<'EOF'\np='/workspace/submission/curate.py'; s=open(p).read()\ns=s.replace('HTML = float(os.environ.get(\"HTML\", \"0.04\"))','HTML = float(os.environ.get(\"HTML\", \"0.06\"))')\ns=s.replace('# dev target: 0.04 at >=0.02 markup rate beat 0 (333.2), 0.08 (329.3) and >=0.01 (330.3),',\n            '# dev target: 0.06 at >=0.02 markup rate beat 0 (333.2), 0.04 (328.6), 0.08 (329.3) and\\n# >=0.01 (330.3),')\nopen(p,'w').write(s)\nEOF\npython submission/curate.py | tail -8\nTOKENIZERS_PARALLELISM=false python pack_selection.py submission/selection.json /tmp/tr_sub2.npy 2>/dev/null | tail -1\npython -c \"\nimport json,numpy as np\ns=json.load(open('submission/selection.json')); ids=set(int(x) for x in np.load('work/tok_ids.npy'))\na=np.load('/tmp/tr_sub2.npy'); b=np.load('/tmp/tr_H6.npy')\nprint('ids',len(s),'| unique',len(set(s))==len(s),'| in-range',set(s)<=ids,'| tokens',len(a),'| bitmatch H6:',(a==b).all())\n\"\nnohup bash dev_eval.sh submission/selection.json > /tmp/official2.log 2>&1 &\necho \"official confirm launched\"",
  "description": "Switch to HTML=0.06 and verify",
  "timeout": 600000
}

Bash result

surface_style  pool_cands=  3548 taken=  3548 tokens=3,529,081
  surface_html   pool_cands=   775 taken=   775 tokens=1,560,374
  encyclopedic   pool_cands= 14164 taken=  6425 tokens=7,106,631
  web_prose      pool_cands= 50838 taken=  7247 tokens=7,113,209
  news           pool_cands= 12820 taken= 10916 tokens=7,106,600
  technical_qa   pool_cands=  7056 taken=  7056 tokens=4,920,505
selection: 35,967 ids, 31,336,400 tokens (2.61x the 12,000,000 budget)
wrote /workspace/submission/selection.json
packed 12000000 tokens from 35967 ids -> /tmp/tr_sub2.npy
ids 35967 | unique True | in-range True | tokens 12000000 | bitmatch H6: True
official confirm launched

[stdout]
  surface_style  pool_cands=  3548 taken=  3548 tokens=3,529,081
  surface_html   pool_cands=   775 taken=   775 tokens=1,560,374
  encyclopedic   pool_cands= 14164 taken=  6425 tokens=7,106,631
  web_prose      pool_cands= 50838 taken=  7247 tokens=7,113,209
  news           pool_cands= 12820 taken= 10916 tokens=7,106,600
  technical_qa   pool_cands=  7056 taken=  7056 tokens=4,920,505
selection: 35,967 ids, 31,336,400 tokens (2.61x the 12,000,000 budget)
wrote /workspace/submission/selection.json
packed 12000000 tokens from 35967 ids -> /tmp/tr_sub2.npy
ids 35967 | unique True | in-range True | tokens 12000000 | bitmatch H6: True
official confirm launched
[metadata] {"noOutputExpected": false}

#57

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolRead

args

{
  "file_path": "/workspace/claim.md",
  "offset": 1,
  "limit": 14
}

Read result

1	# Curating a 12M-token subset of raw web for a broad high-quality target
2	
3	**Result.** Dev perplexity **328.6** vs **470.1** for a random selection of the same
4	pool at the same 12M-token budget (−30%). All numbers below are the frozen
5	`train_nano.py` recipe (30M GPT, 3000 iters, seed 1337) on `data/multi_dev.npy`.
6	
7	| selection (12M tokens each) | dev PPL |
8	|---|---|
9	| random (do-nothing baseline) | 470.1 |
10	| quality heuristics only, no target matching | 459.0 |
11	| target-register classifier only, no quality gate | 361.4 |
12	| quality gate + classifier, uniform register mixture | 355.2 |
13	| **+ surface-form lane (final submission)** | **333.2** |
14

[metadata] {"type": "text", "file": {"filePath": "/workspace/claim.md", "content": "# Curating a 12M-token subset of raw web for a broad high-quality target\n\n**Result.** Dev perplexity **328.6** vs **470.1** for a random selection of the same\npool at the same 12M-token budget (−30%). All numbers below are the frozen\n`train_nano.py` recipe (30M GPT, 3000 iters, seed 1337) on `data/multi_dev.npy`.\n\n| selection (12M tokens each) | dev PPL |\n|---|---|\n| random (do-nothing baseline) | 470.1 |\n| quality heuristics only, no target matching | 459.0 |\n| target-register classifier only, no quality gate | 361.4 |\n| quality gate + classifier, uniform register mixture | 355.2 |\n| **+ surface-form lane (final submission)** | **333.2** |\n", "numLines": 14, "startLine": 1, "totalLines": 146}}

#58

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolEdit

args

{
  "replace_all": false,
  "file_path": "/workspace/claim.md",
  "old_string": "**Result.** Dev perplexity **328.6** vs **470.1** for a random selection of the same\npool at the same 12M-token budget (−30%). All numbers below are the frozen\n`train_nano.py` recipe (30M GPT, 3000 iters, seed 1337) on `data/multi_dev.npy`.\n\n| selection (12M tokens each) | dev PPL |\n|---|---|\n| random (do-nothing baseline) | 470.1 |\n| quality heuristics only, no target matching | 459.0 |\n| target-register classifier only, no quality gate | 361.4 |\n| quality gate + classifier, uniform register mixture | 355.2 |\n| **+ surface-form lane (final submission)** | **333.2** |",
  "new_string": "**Result.** Dev perplexity **324.4** vs **470.1** for a random selection of the same\npool at the same 12M-token budget (−31%). All numbers below are the frozen\n`train_nano.py` recipe (30M GPT, 3000 iters, seed 1337) on `data/multi_dev.npy`.\n\n| selection (12M tokens each) | dev PPL |\n|---|---|\n| random (do-nothing baseline) | 470.1 |\n| quality heuristics only, no target matching | 459.0 |\n| target-register classifier only, no quality gate | 361.4 |\n| quality gate + classifier, uniform register mixture | 355.2 |\n| + spacing surface lane (`STYLE=0.16`) | 333.2 |\n| **+ HTML surface lane (`HTML=0.06`) — final submission** | **324.4** |\n\nThe two surface-form lanes together are worth **30.8 PPL**, against **11.1** for the\ngeneric quality heuristics that a standard pipeline would stop at."
}

Edit result

The file /workspace/claim.md has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/claim.md", "oldString": "**Result.** Dev perplexity **328.6** vs **470.1** for a random selection of the same\npool at the same 12M-token budget (−30%). All numbers below are the frozen\n`train_nano.py` recipe (30M GPT, 3000 iters, seed 1337) on `data/multi_dev.npy`.\n\n| selection (12M tokens each) | dev PPL |\n|---|---|\n| random (do-nothing baseline) | 470.1 |\n| quality heuristics only, no target matching | 459.0 |\n| target-register classifier only, no quality gate | 361.4 |\n| quality gate + classifier, uniform register mixture | 355.2 |\n| **+ surface-form lane (final submission)** | **333.2** |", "newString": "**Result.** Dev perplexity **324.4** vs **470.1** for a random selection of the same\npool at the same 12M-token budget (−31%). All numbers below are the frozen\n`train_nano.py` recipe (30M GPT, 3000 iters, seed 1337) on `data/multi_dev.npy`.\n\n| selection (12M tokens each) | dev PPL |\n|---|---|\n| random (do-nothing baseline) | 470.1 |\n| quality heuristics only, no target matching | 459.0 |\n| target-register classifier only, no quality gate | 361.4 |\n| quality gate + classifier, uniform register mixture | 355.2 |\n| + spacing surface lane (`STYLE=0.16`) | 333.2 |\n| **+ HTML surface lane (`HTML=0.06`) — final submission** | **324.4** |\n\nThe two surface-form lanes together are worth **30.8 PPL**, against **11.1** for the\ngeneric quality heuristics that a standard pipeline would stop at.", "originalFile": "# Curating a 12M-token subset of raw web for a broad high-quality target\n\n**Result.** Dev perplexity **328.6** vs **470.1** for a random selection of the same\npool at the same 12M-token budget (−30%). All numbers below are the frozen\n`train_nano.py` recipe (30M GPT, 3000 iters, seed 1337) on `data/multi_dev.npy`.\n\n| selection (12M tokens each) | dev PPL |\n|---|---|\n| random (do-nothing baseline) | 470.1 |\n| quality heuristics only, no target matching | 459.0 |\n| target-register classifier only, no quality gate | 361.4 |\n| quality gate + classifier, uniform register mixture | 355.2 |\n| **+ surface-form lane (final submission)** | **333.2** |\n\n## Hypothesis\n\nAt a fixed, small token budget, held-out perplexity on a multi-domain target is\ngoverned by **two separable terms**, and the second is much larger than it looks:\n\n1. **Register match** — how much of the budget is spent on text drawn from the same\n   registers as the target (encyclopedic / web prose / news / technical Q&A).\n2. **Surface-form match** — whether the training data shares the target's\n   *tokenization conventions*, independent of topic or quality.\n\nThe strong claim is that term 2 is not a rounding error but the single largest\nremaining component of the loss here, and that generic \"quality filtering\" (the usual\nmove) captures almost none of either term: it is worth only 11 PPL of the 142 recovered.\nTerm 2 is also *general*, not a one-off: it was predicted and then confirmed a second\ntime, in a different alphabet, on a different quarter of the target.\n\n## Mechanism, with predictions on observables other than the final perplexity\n\nThe target's encyclopedic quarter is WikiText-style: **detokenized**, so punctuation is\nspace-prefixed (`Australia , China`). This makes ` ,` and ` .` (GPT-2 ids 837, 764)\n**7.9% of that quarter's tokens** (4.63% + 3.27%), against **0.055%** across the raw\npool. A model that has effectively never seen those tokens pays a large penalty on\nroughly one token in thirteen. Mechanism → three falsifiable predictions, all of which\nI measured *before* looking at any final score:\n\n- **P1 — the loss is concentrated, not diffuse.** Per-quarter decomposition should show\n  the encyclopedic quarter far above the other three, by an amount comparable to\n  (token rate) × (nats per surprise token), not spread evenly.\n  *Observed* (uniform mixture, no style lane): quarter losses **6.611 / 5.576 / 5.623 /\n  5.677** — the encyclopedic quarter sits ~1.0 nat above the others, while the other\n  three lie within 0.10 nat of each other.\n\n- **P2 — the excess is surface, not content.** Re-tokenizing that same held-out\n  quarter with the spacing artifacts normalized away (`\" ,\"→\",\"`, `@-@`, clitics),\n  changing *no* content, should remove a large part of the excess.\n  *Observed* on the final model: **6.369 nats raw → 5.878 normalized**. So **0.49 nats\n  of the encyclopedic quarter is pure surface form** — at 25% of the eval windows,\n  ~0.12 nats of the mean, i.e. ~37 PPL. This is the term the style lane attacks, and it\n  bounds how much is left to win.\n\n- **P3 — marginal token frequency is not the mechanism; context is.** If the model\n  merely needed the *unigram* statistics of ` ,`, then the densest spaced-punctuation\n  documents in the pool would be the most valuable per token. They are boilerplate\n  (nav menus, `. . .` spam) with rates up to 30%, so a small ungated slice raises the\n  training rate ~10× more cheaply than gated prose does. Prediction: this **fails** —\n  the token must be learned in ordinary prose context.\n  *Observed*: ungated style lanes score **342.1 / 344.9 / 355.3**, all worse than the\n  gated lane's **333.2**, despite far higher marginal rates. Confirmed: the final\n  selection reaches only **0.143%** ` ,`+` .` (target 2.0% overall) yet beats every\n  higher-rate variant. Frequency alone buys nothing.\n\n- **P3b (the replication) — the same term should exist wherever the target's surface\n  convention differs, and should be findable *a priori*.** Ranking tokens by excess\n  probability mass `p_target(t)·log[p_target(t)/p_pool(t)]` is a purely\n  distributional screen that needs no training run. Applied to the technical-Q&A\n  quarter it flags HTML markup — `</` at 929× the pool rate, `code` 435×, `.</`\n  4307× — totalling **1.27 nats** over the top 22 tokens, *larger* than the\n  encyclopedic quarter's spacing signature (0.72 nats). Prediction: a small lane of\n  markup-bearing pool documents lowers that quarter's loss specifically.\n  *Observed*: quarter-3 loss **5.639 → 5.512**, total **333.2 → 328.6**, with the\n  other three quarters unmoved. The screen located a second, independent instance of\n  the mechanism before any training run was spent on it.\n\n  Note this lane is drawn **ungated**, the exact opposite of P3's conclusion for\n  spacing, and that difference is itself predicted by the \"context, not frequency\"\n  mechanism: raw web pages carry markup wrapped around ordinary prose, which is the\n  target's context, whereas the densest spaced-punctuation documents are `. . .`\n  boilerplate, which is not.\n\nA fourth observable falls out of the register model and is worth stating because it is\nthe one prediction that came out *negative*:\n\n- **P4 — \"high-quality web prose\" is not a learnable register against this pool.**\n  With a doc-level held-out split, the five-class scorer reaches macro accuracy 0.94\n  (encyclopedic), 0.72 (news), 0.94 (technical Q&A) — but **~0.00 for web prose**,\n  which is classified as pool background. The pool *is* general web, so that class has\n  no discriminative content. Consequently that lane's own score is noise (its top-ranked\n  documents were literally nav-bar dumps), and the submission instead ranks it by mean\n  affinity to all four registers.\n\n## Falsification\n\nThe hypothesis is wrong, or the mechanism is misattributed, if:\n\n- **Format is incidental.** Under P2, evaluating the trained model on a\n  format-normalized encyclopedic quarter should have closed much of the gap. If it had\n  come back at ~6.3 nats (instead of 5.878), the excess would be *content* difficulty\n  and the entire surface-form story would be dead. Re-running `work/probe.py` on any\n  selection tests this directly.\n- **The style lane is a proxy for something else.** If the gain came from those\n  documents' topics rather than their spacing, then matching them on register while\n  *lacking* the spacing should reproduce the gain. It does not: `STYLE=0` with the same\n  quality gate and mixture gives 355.2, and the gain tracks the spacing rate, peaking\n  and then reversing (0 → 355.2, 0.08 → 343.4, **0.16 → 333.2**, 0.20 → 340.4,\n  0.24 → 342.2) — a purity/volume trade-off, not a monotone topic effect.\n- **Register balance is doing the work instead.** If so, mixture weights should matter\n  as much as the style lane. They do not: skewing to 0.40/0.15/0.20/0.25, or deleting\n  the web-prose lane (0.34/0/0.33/0.33), moved the score by 2–4 PPL (335.6, 345.3)\n  against the style lane's 22.\n- **Aggressive fitting of the scorer is harmful.** I predicted the opposite of the\n  textbook answer here: at 6% selectivity you *want* a scorer that keys on rare,\n  register-specific markers. A properly regularized scorer (wd swept, early-stopped on\n  a doc-level holdout, macro acc 0.725) selected **worse** data — 342.2/347.1 vs 333.2.\n  If the regularized scorer had won, the \"rare-marker\" reading would be falsified.\n\n## Transfer\n\n- **Screen the target's token histogram against the pool's before curating.** Ranking\n  by excess probability mass costs one `bincount` per corpus and no GPU time, and it\n  found both wins here: a 140× discrepancy in ` ,` and a 929× one in `</`, neither of\n  which any quality filter, classifier, or dedup pass would ever surface. Do this\n  first; it tells you which term dominates before you spend a training run. Any\n  pipeline mixing corpora with different detokenization conventions (WikiText,\n  Moses-tokenized MT corpora, OCR text) carries this term.\n- **Decompose held-out loss per domain, always.** The aggregate PPL said \"355, keep\n  filtering harder\"; the per-quarter split said \"one quarter is 1 nat worse and it is\n  not a content problem.\" That reallocated the remaining effort and produced the whole\n  gain. Cost: ~10 lines in a diagnostic copy of the trainer.\n- **Validate each class of a domain classifier separately.** A pooled accuracy number\n  would have hidden that one of four lanes was ranking on noise.\n- **Selection pressure and regularization interact.** The right amount of scorer\n  overfitting depends on how selective you are being; don't import a default from\n  classification settings.\n- **Limits.** The style term is specific to a target whose surface form the pool can\n  partly supply — here only ~3.5M gate-passing tokens existed, which caps the\n  recoverable 37 PPL at the ~22 actually realized. With no such documents the correct\n  conclusion would be that this quarter is simply out of reach by selection alone. All\n  tuning used the disclosed dev sample; register and surface-form structure are\n  properties of the domain rather than of the sample, so they should carry to the hidden\n  set, but the specific optima (STYLE=0.16, HTML=0.04, uniform mixture) are fitted to\n  ±5 PPL of run-to-run spread and should not be read as precise.\n", "structuredPatch": [{"oldStart": 1, "oldLines": 7, "newStart": 1, "newLines": 7, "lines": [" # Curating a 12M-token subset of raw web for a broad high-quality target", " ", "-**Result.** Dev perplexity **328.6** vs **470.1** for a random selection of the same", "-pool at the same 12M-token budget (−30%). All numbers below are the frozen", "+**Result.** Dev perplexity **324.4** vs **470.1** for a random selection of the same", "+pool at the same 12M-token budget (−31%). All numbers below are the frozen", " `train_nano.py` recipe (30M GPT, 3000 iters, seed 1337) on `data/multi_dev.npy`.", " ", " | selection (12M tokens each) | dev PPL |"]}, {"oldStart": 10, "oldLines": 8, "newStart": 10, "newLines": 12, "lines": [" | quality heuristics only, no target matching | 459.0 |", " | target-register classifier only, no quality gate | 361.4 |", " | quality gate + classifier, uniform register mixture | 355.2 |", "-| **+ surface-form lane (final submission)** | **333.2** |", "+| + spacing surface lane (`STYLE=0.16`) | 333.2 |", "+| **+ HTML surface lane (`HTML=0.06`) — final submission** | **324.4** |", " ", "+The two surface-form lanes together are worth **30.8 PPL**, against **11.1** for the", "+generic quality heuristics that a standard pipeline would stop at.", "+", " ## Hypothesis", " ", " At a fixed, small token budget, held-out perplexity on a multi-domain target is"]}], "userModified": false, "replaceAll": false}

#59

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolBashUpdate claim numbers and check official confirmation

args

{
  "command": "python - <<'PYEOF'\np='/workspace/claim.md'; s=open(p).read()\ns=s.replace(\"\"\"move) captures almost none of either term: it is worth only 11 PPL of the 142 recovered.\"\"\",\n\"\"\"move) captures almost none of either term: it is worth only 11 PPL of the 146 recovered.\"\"\")\ns=s.replace(\"\"\"  *Observed*: quarter-3 loss **5.639 → 5.512**, total **333.2 → 328.6**, with the\n  other three quarters unmoved. The screen located a second, independent instance of\n  the mechanism before any training run was spent on it.\"\"\",\n\"\"\"  *Observed*: quarter-3 loss **5.639 → 5.489**, total **333.2 → 324.4**\n  (sweep: 0.04 → 328.6, **0.06 → 324.4**, 0.08 → 329.3), with the other three quarters\n  unmoved to within run-to-run spread. The screen located a second, independent instance\n  of the mechanism before any training run was spent on it.\"\"\")\ns=s.replace(\"\"\"  behaves the same way (0 → 333.2, **0.04 → 328.6**, 0.08 → 329.3) and moves only the\n  quarter it targets, which a topic-confound story does not predict.\"\"\",\n\"\"\"  behaves the same way (0 → 333.2, 0.04 → 328.6, **0.06 → 324.4**, 0.08 → 329.3) and\n  moves only the quarter it targets, which a topic-confound story does not predict.\"\"\")\ns=s.replace(\"\"\"  against the style lane's 22.\"\"\",\"\"\"  against the surface lanes' 31 combined.\"\"\")\ns=s.replace(\"\"\"  a doc-level holdout, macro acc 0.725) selected **worse** data — 342.2/347.1 vs 333.2.\"\"\",\n\"\"\"  a doc-level holdout, macro acc 0.725) selected **worse** data — 342.2/347.1 vs 333.2\n  at the then-current configuration.\"\"\")\ns=s.replace(\"\"\"  set, but the specific optima (STYLE=0.16, HTML=0.04, uniform mixture) are fitted to\"\"\",\n\"\"\"  set, but the specific optima (STYLE=0.16, HTML=0.06, uniform mixture) are fitted to\"\"\")\ns=s.replace(\"\"\"  recoverable 37 PPL at the ~22 actually realized.\"\"\",\n\"\"\"  recoverable 37 PPL at the ~22 actually realized on that quarter.\"\"\")\nopen(p,'w').write(s)\nprint(\"ok\")\nPYEOF\nsleep 75; grep -vE \"Token indices\" /tmp/official2.log; grep -c 328.6 claim.md; cat .timer/remaining_secs",
  "description": "Update claim numbers and check official confirmation",
  "timeout": 600000
}

Bash result

ok
packed 12000000 tokens from 35967 ids -> /tmp/dev_train.npy
HELDOUT_PPL 324.403  loss 5.7820  train_s 51  /tmp/dev_train.npy
(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)
1
2095

[stdout]
ok
packed 12000000 tokens from 35967 ids -> /tmp/dev_train.npy
HELDOUT_PPL 324.403  loss 5.7820  train_s 51  /tmp/dev_train.npy
(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)
1
2095
[metadata] {"noOutputExpected": false}

#60

narration

Official path confirms 324.403 on the submitted file. With ~34 min left, let me re-tune the spacing lane jointly now that the HTML lane also draws from the budget.

toolBashJoint re-tune of style share with HTML lane active

args

{
  "command": "STYLE=0.12 OUT_SEL=/tmp/sel_J1.json python submission/curate.py >/dev/null\nSTYLE=0.20 OUT_SEL=/tmp/sel_J2.json python submission/curate.py >/dev/null\n( bash work/ev.sh J1 /tmp/sel_J1.json & bash work/ev.sh J2 /tmp/sel_J2.json & wait ); cat .timer/remaining_secs",
  "description": "Joint re-tune of style share with HTML lane active",
  "timeout": 600000
}

Bash result

[J1] QUARTER 0 loss 6.4628 ppl 640.88 QUARTER 1 loss 5.5842 ppl 266.20 QUARTER 2 loss 5.6802 ppl 293.00 QUARTER 3 loss 5.5653 ppl 261.21 HELDOUT_PPL 338.569  loss 5.8247  train_s 108  /tmp/tr_J1.npy [J2] QUARTER 0 loss 6.4097 ppl 607.71 QUARTER 1 loss 5.5905 ppl 267.87 QUARTER 2 loss 5.6792 ppl 292.72 QUARTER 3 loss 5.4582 ppl 234.67 HELDOUT_PPL 325.553  loss 5.7855  train_s 108  /tmp/tr_J2.npy 

1965

[stdout]
[J1] QUARTER 0 loss 6.4628 ppl 640.88 QUARTER 1 loss 5.5842 ppl 266.20 QUARTER 2 loss 5.6802 ppl 293.00 QUARTER 3 loss 5.5653 ppl 261.21 HELDOUT_PPL 338.569  loss 5.8247  train_s 108  /tmp/tr_J1.npy [J2] QUARTER 0 loss 6.4097 ppl 607.71 QUARTER 1 loss 5.5905 ppl 267.87 QUARTER 2 loss 5.6792 ppl 292.72 QUARTER 3 loss 5.4582 ppl 234.67 HELDOUT_PPL 325.553  loss 5.7855  train_s 108  /tmp/tr_J2.npy 

1965
[metadata] {"noOutputExpected": false}

#61

narration

STYLE=0.16 holds (0.12→338.6, 0.20→325.6 tie). I'm at a plateau, so let me stop tuning and do final verification of all deliverables.

toolBashFinal verification of reproducibility and deliverables

args

{
  "command": "cd /workspace\ncp submission/selection.json /tmp/sel_before.json\nenv -u STYLE -u HTML -u MIX -u SCORES -u PROSE_RANK -u STYLE_MIN -u STYLE_RANK -u STYLE_GATE -u HTML_MIN -u OUT_SEL python submission/curate.py | tail -3\npython -c \"\nimport json,hashlib\na=open('/tmp/sel_before.json','rb').read(); b=open('/workspace/submission/selection.json','rb').read()\nprint('curate.py reproduces submitted selection byte-identically:', a==b)\ns=json.load(open('/workspace/submission/selection.json'))\nprint('ids:',len(s),'unique:',len(set(s))==len(s),'min/max:',min(s),max(s))\nprint('sha256:',hashlib.sha256(b).hexdigest()[:16])\n\"\necho \"--- deliverables ---\"; ls -la submission/ claim.md REPRODUCE.md\necho \"--- frozen files untouched ---\"; md5sum train_nano.py model.py pack_selection.py",
  "description": "Final verification of reproducibility and deliverables",
  "timeout": 600000
}

Bash result

technical_qa   pool_cands=  7056 taken=  7056 tokens=4,920,505
selection: 35,967 ids, 31,336,400 tokens (2.61x the 12,000,000 budget)
wrote /workspace/submission/selection.json
curate.py reproduces submitted selection byte-identically: True
ids: 35967 unique: True min/max: 1 181942
sha256: 259a885b933a4bc7
--- deliverables ---
-rw-r--r-- 1 root root 2424 Aug  4 02:00 REPRODUCE.md
-rw-r--r-- 1 root root 9721 Aug  4 02:03 claim.md

submission/:
total 262
drwxr-xr-x 1 root root    200 Aug  4 01:54 .
drwxr-xr-x 1 root root    260 Aug  4 02:03 ..
drwxr-xr-x 1 root root    140 Aug  4 01:55 __pycache__
-rw-r--r-- 1 root root  13303 Aug  4 02:02 curate.py
-rw-r--r-- 1 root root 254267 Aug  4 02:07 selection.json
--- frozen files untouched ---
69750565b7b1a4134b66f571260d6445  train_nano.py
53731e75616616267fc67ff6c16a4ef1  model.py
f34aa5a206cf76ba6153586333098607  pack_selection.py

[stdout]
  technical_qa   pool_cands=  7056 taken=  7056 tokens=4,920,505
selection: 35,967 ids, 31,336,400 tokens (2.61x the 12,000,000 budget)
wrote /workspace/submission/selection.json
curate.py reproduces submitted selection byte-identically: True
ids: 35967 unique: True min/max: 1 181942
sha256: 259a885b933a4bc7
--- deliverables ---
-rw-r--r-- 1 root root 2424 Aug  4 02:00 REPRODUCE.md
-rw-r--r-- 1 root root 9721 Aug  4 02:03 claim.md

submission/:
total 262
drwxr-xr-x 1 root root    200 Aug  4 01:54 .
drwxr-xr-x 1 root root    260 Aug  4 02:03 ..
drwxr-xr-x 1 root root    140 Aug  4 01:55 __pycache__
-rw-r--r-- 1 root root  13303 Aug  4 02:02 curate.py
-rw-r--r-- 1 root root 254267 Aug  4 02:07 selection.json
--- frozen files untouched ---
69750565b7b1a4134b66f571260d6445  train_nano.py
53731e75616616267fc67ff6c16a4ef1  model.py
f34aa5a206cf76ba6153586333098607  pack_selection.py
[metadata] {"noOutputExpected": false}

#62

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolBashClean pycache and review Transfer section

args

{
  "command": "rm -rf /workspace/submission/__pycache__; cd /workspace; sed -n '126,152p' claim.md",
  "description": "Clean pycache and review Transfer section"
}

Bash result

## Transfer

- **Screen the target's token histogram against the pool's before curating.** Ranking
  by excess probability mass costs one `bincount` per corpus and no GPU time, and it
  found both wins here: a 140× discrepancy in ` ,` and a 929× one in `</`, neither of
  which any quality filter, classifier, or dedup pass would ever surface. Do this
  first; it tells you which term dominates before you spend a training run. Any
  pipeline mixing corpora with different detokenization conventions (WikiText,
  Moses-tokenized MT corpora, OCR text) carries this term.
- **Decompose held-out loss per domain, always.** The aggregate PPL said "355, keep
  filtering harder"; the per-quarter split said "one quarter is 1 nat worse and it is
  not a content problem." That reallocated the remaining effort and produced the whole
  gain. Cost: ~10 lines in a diagnostic copy of the trainer.
- **Validate each class of a domain classifier separately.** A pooled accuracy number
  would have hidden that one of four lanes was ranking on noise.
- **Selection pressure and regularization interact.** The right amount of scorer
  overfitting depends on how selective you are being; don't import a default from
  classification settings.
- **Limits.** The style term is specific to a target whose surface form the pool can
  partly supply — here only ~3.5M gate-passing tokens existed, which caps the
  recoverable 37 PPL at the ~22 actually realized on that quarter. With no such documents the correct
  conclusion would be that this quarter is simply out of reach by selection alone. All
  tuning used the disclosed dev sample; register and surface-form structure are
  properties of the domain rather than of the sample, so they should carry to the hidden
  set, but the specific optima (STYLE=0.16, HTML=0.06, uniform mixture) are fitted to
  ±5 PPL of run-to-run spread and should not be read as precise.

[stdout]
## Transfer

- **Screen the target's token histogram against the pool's before curating.** Ranking
  by excess probability mass costs one `bincount` per corpus and no GPU time, and it
  found both wins here: a 140× discrepancy in ` ,` and a 929× one in `</`, neither of
  which any quality filter, classifier, or dedup pass would ever surface. Do this
  first; it tells you which term dominates before you spend a training run. Any
  pipeline mixing corpora with different detokenization conventions (WikiText,
  Moses-tokenized MT corpora, OCR text) carries this term.
- **Decompose held-out loss per domain, always.** The aggregate PPL said "355, keep
  filtering harder"; the per-quarter split said "one quarter is 1 nat worse and it is
  not a content problem." That reallocated the remaining effort and produced the whole
  gain. Cost: ~10 lines in a diagnostic copy of the trainer.
- **Validate each class of a domain classifier separately.** A pooled accuracy number
  would have hidden that one of four lanes was ranking on noise.
- **Selection pressure and regularization interact.** The right amount of scorer
  overfitting depends on how selective you are being; don't import a default from
  classification settings.
- **Limits.** The style term is specific to a target whose surface form the pool can
  partly supply — here only ~3.5M gate-passing tokens existed, which caps the
  recoverable 37 PPL at the ~22 actually realized on that quarter. With no such documents the correct
  conclusion would be that this quarter is simply out of reach by selection alone. All
  tuning used the disclosed dev sample; register and surface-form structure are
  properties of the domain rather than of the sample, so they should carry to the hidden
  set, but the specific optima (STYLE=0.16, HTML=0.06, uniform mixture) are fitted to
  ±5 PPL of run-to-run spread and should not be read as precise.
[metadata] {"noOutputExpected": false}

final summary

All deliverables are verified. Final state:

Submission: dev perplexity 324.4 vs 470.1 random baseline (−31%), confirmed end-to-end through the official dev_eval.sh path (pack_selection.py + frozen train_nano.py).

selection (12M tokens each)	dev PPL
random baseline	470.1
quality heuristics only	459.0
register classifier only	361.4
gate + classifier, uniform mixture	355.2
+ spacing surface lane	333.2
+ HTML surface lane (submitted)	324.4

What drove the result. Standard curation (Gopher-style quality filters) was worth only 11 of the 146 PPL recovered. Two things mattered more:

A register classifier — 5-class logistic regression on hashed GPT-2 n-grams, trained with the disclosed dev target as positives and random pool docs as background. This alone took 470 → 361.
Surface-form matching, which I found by decomposing held-out loss per quarter. The Wikipedia quarter sat ~1 nat above the other three, and a normalization probe showed 0.49 nats of it was pure tokenization convention, not content: the target is detokenized WikiText where ,/ . are 7.9% of tokens vs 0.055% of the pool. A lane of pool docs sharing that spacing took 355 → 333. Screening token histograms by excess probability mass then predicted a second instance — HTML markup in the technical Q&A quarter (</ at 929× the pool rate) — which took 333 → 324.

Two results worth flagging because they cut against expectations:

Frequency isn't the mechanism, context is. The densest spaced-punctuation docs (up to 30%) are . . . boilerplate; using them ungated hurt (342–355) despite 10× the marginal rate. The winning selection reaches only 0.143% , against a 2.0% target. The HTML lane is the reverse — ungated works there, because raw web pages carry markup around genuine prose.
A properly regularized scorer selected worse data (342/347 vs 333). At 6% selectivity you want the scorer keying on rare register-specific markers. Relatedly, held-out validation revealed the "high-quality web prose" class has ~0 accuracy — it's indistinguishable from raw-web background, so that lane was ranking on noise and now ranks by mean affinity to all four registers.

Files: submission/selection.json (35,967 ids, 2.6× the budget, verified unique/in-range and byte-reproducible from curate.py defaults), submission/curate.py (stated criterion in its docstring, all knobs defaulted), claim.md, plus REPRODUCE.md and the work/ pipeline. train_nano.py, model.py, and pack_selection.py are unmodified — diagnostics live in separate copies.

One caveat on the tuned optima: neighbouring configurations vary by roughly ±5 PPL, so STYLE=0.16 / HTML=0.06 sit on a plateau rather than a sharp peak. The mechanism should transfer to the hidden sample since register and surface form are domain properties, but I wouldn't read those specific shares as precise.