claude-opus-5

HELDOUT_PPL 317.036  loss 5.7590  train_s 51  /tmp/tr_final.npy
{"train_npy": "/tmp/tr_final.npy", "heldout_ppl": 317.03587576609254, "mean_loss": 5.759014940237425, "max_iters": 3000, "seed": 1337, "train_s": 51.1}

[stdout]
HELDOUT_PPL 317.036  loss 5.7590  train_s 51  /tmp/tr_final.npy
{"train_npy": "/tmp/tr_final.npy", "heldout_ppl": 317.03587576609254, "mean_loss": 5.759014940237425, "max_iters": 3000, "seed": 1337, "train_s": 51.1}
[metadata] {"noOutputExpected": false}

The file /workspace/claim.md has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/claim.md", "oldString": "| A | 3-register 25/50/25, argmax, gate | 323.22 |\n| F | A + non-English filter | 322.09 |\n| **I** | **F + mixture tilt 0.22 / 0.43 / 0.35** | **317.04** |\n\nReproduce: `python3 submission/curate.py` → `submission/selection.json`;\nevaluate with `bash dev_eval.sh`.", "newString": "| A | 3-register 25/50/25, argmax, gate | 323.22 |\n| F | A + non-English filter | 322.09 |\n| K | tilt further to Q&A (0.20 / 0.38 / 0.42) | 325.90 |\n| M | tilt off wiki (0.18 / 0.47 / 0.35) | 324.30 |\n| L | tilt 0.25 / 0.40 / 0.35 | 323.12 |\n| O | I + stricter quality thresholds | 322.35 |\n| N | I + cap documents at 3k tokens | 321.76 |\n| P | I + raise minimum length to 256 tokens | 320.99 |\n| **I** | **F + mixture tilt 0.22 / 0.43 / 0.35 — submitted** | **317.04** |\n\nThree further tweaks on top of I (stricter gate, length cap, higher minimum length) each came\nout 4–5 PPL *worse*, and the neighbouring mixtures cluster at 321–326. So the recipe is\nsaturated at this budget: the large, robust effects are register coverage (+129 PPL vs\nwiki-only) and the junk gate (+24 PPL); everything after that is inside a few-PPL band.\n\n**Reproduction check.** `python3 submission/curate.py` was run end-to-end from the raw pool and\nemitted a selection *bit-identical* to the tuned candidate (18,543 ids / 13,500,216 tokens;\nclassifier losses 0.2672 / 0.3683 / 0.1454), which retrains to the same **317.04**. The 12M\nbudget cuts at the 16,872nd id, leaving 12.5% headroom, and the realised register mixture is\n0.22 / 0.43 / 0.35 at every prefix from 500 documents onward.\n\nReproduce: `python3 submission/curate.py` → `submission/selection.json`;\nevaluate with `bash dev_eval.sh`.", "originalFile": "# Curating a raw web pool for a fixed 12M-token budget\n\n**Setup.** 182,016 raw web documents (199.9M GPT-2 tokens). The trainer consumes an\nordered id list until 12M tokens are packed, then trains a frozen 30M GPT and reports\nheld-out perplexity on a broad, equal-parts high-quality English target\n(encyclopedic / general web prose / news / technical Q&A). Only the selection varies.\n\n---\n\n## Hypothesis\n\nAt a budget that is ~6% of the pool, held-out perplexity on a *broad multi-domain* target\nis set mainly by **distributional match to the target**, not by any single scalar notion of\n\"document quality\". Concretely, three things in decreasing order of effect:\n\n1. **Register coverage.** The training set must contain all registers the target contains.\n   Selecting only the register that looks most \"clean and encyclopedic\" should fail badly,\n   even though every selected document is individually high quality.\n2. **Junk removal.** Boilerplate / navigation dumps / duplicate-line spam / non-English\n   pages consume budget while contributing almost no target-relevant probability mass.\n3. **Target-similarity ranking.** The disclosed dev target (`multi_dev.npy`, 1M tokens) is a\n   sample of the target distribution itself; a bag-of-hashed-n-grams classifier separating\n   it from a random pool sample recovers enough of the importance ratio to rank documents\n   usefully, with no labels and no internet.\n\nI further predicted that the loss-minimising *training* mixture is **not** the target's own\nmixture, because registers differ in marginal return per token spent.\n\n## Mechanism — and an observable that is not the final perplexity\n\nThe mechanism is mass placement: selection wins by moving the training set's empirical\nn-gram distribution onto the target's. That is measurable **with no training at all**.\n\n*Pre-registered observable:* the unigram/bigram cross-entropy of the target token stream\nunder a candidate training set's own empirical n-gram distribution,\n`H(target | train)` (add-1 smoothed; `work/ngram_close.py`). Prediction: candidates that\nlower held-out PPL also lower `H(target | train)`, and — the sharp part — a\nsingle-register selection will **not** lower it, landing near random.\n\n| selection | unigram `H` | bigram `H` | trained PPL |\n|---|---|---|---|\n| random (do-nothing baseline) | 8.1343 | 13.6374 | 485.46 |\n| curated, no quality gate (D) | 7.9662 | 13.3881 | 357.61 |\n| curated, full recipe (F) | **7.9273** | **13.3064** | 322.09 |\n| curated but **encyclopedic register only** (J) | 8.1299 | 13.6209 | 445.88 |\n\nConfirmed, including the sharp prediction: the wiki-only selection is a *cleaner* corpus by\nevery intrinsic quality heuristic, yet its n-gram mass sits essentially as far from the broad\ntarget as random text does (8.1299 vs 8.1343), and its trained perplexity is correspondingly\nclose to random (445.9 vs 485.5) and far worse than the mixture-matched recipe (317–322).\nQuality-without-coverage does not work; the probe sees this before a single training step.\n\n*Second observable:* per-register `H(target_register | train)` should be **unequal**, which is\nwhy the optimal training mixture is not the eval mixture. Measured under the final selection:\n**wiki 8.31, qa 7.94, web 7.71**. The encyclopedic register is the hardest, and partly\n*irreducibly* so — the target renders Wikipedia with WikiText detokenisation artefacts\n(`@-@`, `@,@`, spaced ` , `) that occur in **no** pool document, and selection cannot create\nsurface form it does not have. Budget shifted off wiki toward the registers whose loss is\nactually reducible measured better (317.0 tilted vs 322.1 at the target's own 25/50/25).\n\n## Falsification\n\nThe hypothesis would be falsified by any of:\n\n- **Coverage is not what matters** → the single-register selection (J) would have matched the\n  mixture-matched selection. *Tested: 445.9 vs 317.0. Not falsified.*\n- **Junk removal is cosmetic** → dropping the quality gate would not hurt. *Tested: 357.6 vs\n  333.8 with the gate held otherwise identical — the gate is worth ~24 PPL. Not falsified.*\n- **The probe is not the mechanism** → `H(target | train)` would move independently of PPL.\n  *Tested: the four selections above rank identically on both. Not falsified.*\n- **Still open / weakly supported:** the mixture *tilt* is the least robust claim. Neighbouring\n  mixtures scored 317.0 / 323.1 / 324.3 / 325.9, a non-smooth ~5-point spread, so run-level\n  variance between selections is a few PPL and the tilt's true effect is small. It should be\n  treated as a marginal gain, not an established law. A clean falsification test would be\n  three seeds per mixture; that did not fit the compute budget.\n- One guess **was** falsified: I expected stripping HTML markup from the technical-Q&A\n  positives to help (the markup cannot transfer to plain-text pool documents). It measurably\n  hurt — 329.7 vs 322.1 — so the markup functions as a useful register indicator for the\n  classifier. The recipe keeps it, against my prior.\n\n## Transfer\n\nThe recipe needs no labels, no internet, and no target-specific engineering, so it transfers\nto any (unlabeled pool, small disclosed target sample) pair:\n\n1. Decode/collect a small sample of the target distribution; split it into registers.\n2. Fit one hashed-n-gram logistic regression per register, target-vs-random-pool.\n3. Veto junk with cheap surface heuristics (looser thresholds for code-bearing registers,\n   whose low alpha/stopword ratios are legitimate).\n4. Emit a weighted round-robin over registers so that **every prefix** of the list is\n   mixture-matched — which makes the output robust to *any* budget, since wherever the\n   trainer stops, the realised mixture is still right.\n\nCost is minutes of CPU plus a few seconds of GPU for the whole 200M-token pool; nothing\nscales worse than linearly, so the same script applies to much larger pools and budgets.\nThe parts I would expect to transfer least: the specific tilt weights, and the\ngate thresholds, both of which are tuned against this pool's junk profile. The parts I\nwould expect to transfer most: register coverage dominating single-axis quality, and\n`H(target | train)` as a zero-training proxy for ranking candidate selections — that probe\nis what makes the approach cheap to re-tune on a new corpus.\n\n---\n\n### Ablation ladder (dev PPL, frozen trainer, identical 12M budget)\n\n| # | selection | dev PPL |\n|---|---|---|\n| – | random baseline | 485.46 |\n| J | encyclopedic register only (falsification) | 445.88 |\n| D | 4-register, no quality gate | 357.61 |\n| B | 4-register, global (non-specialised) ranking | 333.83 |\n| E | merged web signal, markup stripped | 329.92 |\n| H | markup stripped from Q&A positives only | 329.73 |\n| C | 4-register (news/prose split), argmax assign | 327.85 |\n| A | 3-register 25/50/25, argmax, gate | 323.22 |\n| F | A + non-English filter | 322.09 |\n| **I** | **F + mixture tilt 0.22 / 0.43 / 0.35** | **317.04** |\n\nReproduce: `python3 submission/curate.py` → `submission/selection.json`;\nevaluate with `bash dev_eval.sh`.\n", "structuredPatch": [{"oldStart": 114, "oldLines": 7, "newStart": 114, "newLines": 24, "lines": [" | C | 4-register (news/prose split), argmax assign | 327.85 |", " | A | 3-register 25/50/25, argmax, gate | 323.22 |", " | F | A + non-English filter | 322.09 |", "-| **I** | **F + mixture tilt 0.22 / 0.43 / 0.35** | **317.04** |", "+| K | tilt further to Q&A (0.20 / 0.38 / 0.42) | 325.90 |", "+| M | tilt off wiki (0.18 / 0.47 / 0.35) | 324.30 |", "+| L | tilt 0.25 / 0.40 / 0.35 | 323.12 |", "+| O | I + stricter quality thresholds | 322.35 |", "+| N | I + cap documents at 3k tokens | 321.76 |", "+| P | I + raise minimum length to 256 tokens | 320.99 |", "+| **I** | **F + mixture tilt 0.22 / 0.43 / 0.35 — submitted** | **317.04** |", " ", "+Three further tweaks on top of I (stricter gate, length cap, higher minimum length) each came", "+out 4–5 PPL *worse*, and the neighbouring mixtures cluster at 321–326. So the recipe is", "+saturated at this budget: the large, robust effects are register coverage (+129 PPL vs", "+wiki-only) and the junk gate (+24 PPL); everything after that is inside a few-PPL band.", "+", "+**Reproduction check.** `python3 submission/curate.py` was run end-to-end from the raw pool and", "+emitted a selection *bit-identical* to the tuned candidate (18,543 ids / 13,500,216 tokens;", "+classifier losses 0.2672 / 0.3683 / 0.1454), which retrains to the same **317.04**. The 12M", "+budget cuts at the 16,872nd id, leaving 12.5% headroom, and the realised register mixture is", "+0.22 / 0.43 / 0.35 at every prefix from 500 documents onward.", "+", " Reproduce: `python3 submission/curate.py` → `submission/selection.json`;", " evaluate with `bash dev_eval.sh`."]}], "userModified": false, "replaceAll": false}

ids: 18543 | unique: True | all in pool: True
tokens provided: 13.50M (budget 12.00M) -> covers budget: True
--- deliverables ---
-rw-r--r-- 1 root root 8172 Aug  4 02:01 claim.md

submission/:
total 140
drwxr-xr-x 1 root root     80 Aug  4 01:50 .
drwxr-xr-x 1 root root    220 Aug  4 02:01 ..
-rw-r--r-- 1 root root  14474 Aug  4 01:55 curate.py
-rw-r--r-- 1 root root 128151 Aug  4 01:59 selection.json

2295

[stdout]
ids: 18543 | unique: True | all in pool: True
tokens provided: 13.50M (budget 12.00M) -> covers budget: True
--- deliverables ---
-rw-r--r-- 1 root root 8172 Aug  4 02:01 claim.md

submission/:
total 140
drwxr-xr-x 1 root root     80 Aug  4 01:50 .
drwxr-xr-x 1 root root    220 Aug  4 02:01 ..
-rw-r--r-- 1 root root  14474 Aug  4 01:55 curate.py
-rw-r--r-- 1 root root 128151 Aug  4 01:59 selection.json

2295
[metadata] {"noOutputExpected": false}