claude-opus-5

The file /workspace/claim.md has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"type": "update", "filePath": "/workspace/claim.md", "content": "# Claim: at a 12M-token budget, *register-typicality ranking* beats quality filtering\n\nAll numbers are dev perplexity on `data/multi_dev.npy`, produced by the frozen trainer at\nits default settings (seed 1337, 12M training tokens). The do-nothing reference — a random\nselection of pool ids — measures **469.9**. The submitted selection measures **341.3**\n(verified end-to-end through `pack_selection.py` + `train_nano.py` on the exact submitted\n`selection.json`). Retraining the *same* token set with a different seed moved PPL by ~12\n(449.5 → 437.8), so only gaps larger than ~25 PPL are treated as real below.\n\n## Hypothesis\n\nThe disclosed target is an equal-parts mixture of four registers (encyclopedic /\nhigh-quality web prose / news / technical Q&A). I claim that held-out perplexity at this\nbudget is governed by **selecting the most *typical* members of each target register and\nspending an equal token share on each**, and that **hand-written quality filters are\nactively harmful** because they silently delete whole registers of the target.\n\nTwo sub-claims:\n\n1. **Typicality ranking is the lever.** Train a bag-of-hashed-character-4-grams\n   classifier to separate the four registers of the decoded, format-stripped target from\n   a random-pool background class. Assign every document to its argmax register and rank\n   it by log-odds against the background class; fill the budget from the four ranked lists\n   interleaved *by token count*. Predicted: large gain over random. Measured: **341.3 vs\n   469.9 (-27%)**.\n2. **Filtering costs more than it buys.** Every heuristic gate I added — a Gopher/C4-style\n   boilerplate-and-fluency gate, and a Mahalanobis gate on the target's own character\n   statistics — *raised* perplexity, monotonically in strictness. Measured: 341.3 (no\n   gate) → 404.6 (junk gate) → 449.5 (junk + surface gate @0.70) → 497.3 (@0.40).\n\n## Mechanism\n\nThe model sees 24.6M training tokens (3000 x 32 x 256) drawn from a 12M-token set, ~2\nepochs, at 30M parameters. In that regime the model is still acquiring generic English\ninside the target's registers, so what matters is that each register is (a) present in\nproportion and (b) represented by its most prototypical documents — those are the ones\nwhose n-gram statistics the eval windows actually re-use.\n\nThe filters fail for a concrete, checkable reason: **they are register-selective**. My junk\ngate rejects symbols, digits, code punctuation and short lines, which is exactly what\ntechnical Q&A looks like. Under the gate, the Q&A list could supply only 1.6M tokens, so\nthe mixture came out 31/31/31/6 instead of 25/25/25/25 — a quarter of the target was\nstarved to ~6% of the budget. Removing the gate raised the Q&A candidate pool from 1,675\ndocuments to 17,893 and let the mixture reach exactly 25/25/25/25; that single change is\nworth 63 PPL (404.6 → 341.3).\n\n### Observables predicted, other than the final perplexity\n\n1. **The pool *is* mostly \"general high-quality web prose.\"** A 5-way classifier should\n   separate encyclopedic, news and Q&A text from the raw pool, but should *fail* on the\n   general-web-prose register, since that register and the pool are the same\n   distribution. *Observed held-out accuracy: wiki 0.97, qa 0.94, news 0.79–0.84,\n   pool 0.97, general-web-prose 0.11–0.35 (at/below chance).*\n2. **Register-selective filter damage.** The junk gate should cut the Q&A register far\n   harder than the others. *Observed: qa candidates 17,893 → 1,675 (-91%) vs wiki\n   33,346 → 9,133 (-73%) and news 40,343 → 15,044 (-63%); resulting token mix\n   25/25/25/25 → 31/31/31/6.*\n3. **Top-ranked documents are recognisably the right register.** *Observed (`peek.py`):\n   wiki list = Britannica/Wikipedia-style entries (\"Anglo-Dutch Wars, also called Dutch\n   Wars, …\"), news list = wire copy (\"LUCKNOW, India (Reuters) — …\"), qa list =\n   programming help threads (servlet `getParameter()` quoting, `.sql` deploy scripts).*\n4. **Monotone strictness curve** (observable as a curve, not a single number):\n   *keep-fraction 1.00 → 341.3/404.6, 0.70 → 449.5, 0.40 → 497.3.*\n5. **Typicality, not balance, does the work.** Keeping the same four balanced buckets but\n   shuffling *within* each bucket should erase most of the gain. *Observed: 445.0 with\n   in-bucket shuffling vs 341.3 ranked — and 437.1 for a plain random order over the same\n   length-floor pool, i.e. balance without ranking buys ~nothing.*\n6. **Format artifacts must be stripped from the target before training the ranker.** The\n   target's WikiText ` @,@ ` escapes / spaced punctuation and StackExchange HTML tags do\n   not exist in the pool (pool documents containing `<p>`: 96 of 182,016; containing\n   WikiText escapes: 0), so an un-normalised classifier would key on formatting that no\n   pool document can match. *Observed: the 313 pool documents with spaced-punctuation\n   statistics and the 205 with HTML markup are spam/product-listing junk, i.e. exactly the\n   documents a format-matching ranker would promote.*\n\n## Falsification\n\nFalsified if any of the following is observed:\n\n* Random order over the same length-floor candidate pool matches the register-ranked order\n  (would mean the classifier contributes nothing). *Tested: 437.1 vs 341.3.*\n* Tightening either gate lowers perplexity, or the strictly filtered selection beats the\n  unfiltered-but-ranked one. *Tested: monotone the other way.*\n* The gain survives removing the ranking but keeping the mixture. *Tested: it does not\n  (445.0).*\n* Over-weighting the register whose surface form the pool cannot match (Q&A, 40% instead\n  of 25%) improves perplexity, i.e. the equal mixture is not the right mixture.\n  *Tested: 354.8 vs 341.3 — no gain, consistent with matching the target's own mixture.*\n* The classifier separates general-web-prose from the pool as well as it separates\n  Wikipedia (would refute observable 1 and the \"pool is the residual register\" argument).\n\n## Full ablation table\n\n| selection rule | docs in 12M | mean doc (tok) | dev PPL |\n|---|---|---|---|\n| random ids (do-nothing) | 12938 | 928 | 469.9 |\n| length floor >=1000 chars, random order | 8767 | 1369 | 437.1 |\n| junk gate, random order | 12738 | 942 | 488.7 |\n| **length floor + register-balanced typicality ranking (submitted)** | **10029** | **1197** | **341.3** |\n| submitted, but shuffled within register buckets | 8284 | 1449 | 445.0 |\n| submitted, but unbalanced (single global ranking) | 9787 | 1226 | see note |\n| submitted + min 400 chars | 11925 | 1006 | 347.2 |\n| submitted + min 2000 chars | 7715 | 1555 | 353.0 |\n| submitted + 40% Q&A mixture | 9418 | 1274 | 354.8 |\n| submitted + junk gate inside the web bucket only | 10063 | 1192 | 348.8 |\n| submitted, ranked on 3 chunks/doc instead of the head | 9990 | 1201 | 352.0 |\n| junk gate + ranking | 12442 | 964 | 404.6 |\n| junk gate (min 500) + ranking | 14377 | 835 | 416.4 |\n| junk + surface gate @0.70 + ranking | 12678 | 947 | 449.5 |\n| junk + surface gate @0.40 + ranking | 12956 | 926 | 497.3 |\n| junk + surface gate, no balance | 12676 | 947 | 453.6 |\n| junk + surface gate, wiki-heavy mixture | 12551 | 956 | 451.4 |\n| junk + surface gate, 800–4500 chars only | 24205 | 496 | 540.2 |\n| junk + surface gate, 1200–12000 chars only | 15387 | 780 | 493.8 |\n| junk + surface gate, >=4000 chars | 6404 | 1874 | 469.2 |\n| junk + surface gate, >=8000 chars | 3313 | 3622 | 478.5 |\n\nDocument length is only weakly important once the gates are gone (347/341/353 for a 400 /\n1000 / 2000-character floor); the strong length effects in the lower block of the table are\ngate artifacts — restricting length inside an already-narrowed pool compounds the coverage\nloss.\n\n## Transfer\n\nThe recipe needs no labels, no reference LM and no tokenizer for scoring, so it transfers\nto any budgeted pretraining or fine-tuning selection where the eval target is disclosed as\na mixture:\n\n1. Decode the target and **strip corpus-specific surface artifacts** before using it as a\n   reference, or the ranker keys on formatting the pool cannot supply (observable 6).\n2. Train a cheap bag-of-hashed-4-grams classifier of *target register vs. raw pool* (~12 s\n   on one GPU) and read off log-odds against the pool class as a typicality score.\n3. Rank inside each register and interleave **by token count**, so that every prefix of the\n   priority list — including the one the budget happens to cut at — carries the target\n   mixture.\n4. Add filters only with an ablation in hand. The generic advice \"filter web data hard\"\n   inverts at small budgets: check, per filter, which register it deletes.\n", "structuredPatch": [{"oldStart": 1, "oldLines": 101, "newStart": 1, "newLines": 137, "lines": ["-# Claim: register-balanced coverage beats quality filtering at a 12M-token budget", "+# Claim: at a 12M-token budget, *register-typicality ranking* beats quality filtering", " ", "-*(numbers are dev perplexity on `data/multi_dev.npy` with the frozen trainer, seed 1337", "-unless stated; the random-selection reference measured here is **469.9**)*", "+All numbers are dev perplexity on `data/multi_dev.npy`, produced by the frozen trainer at", "+its default settings (seed 1337, 12M training tokens). The do-nothing reference — a random", "+selection of pool ids — measures **469.9**. The submitted selection measures **341.3**", "+(verified end-to-end through `pack_selection.py` + `train_nano.py` on the exact submitted", "+`selection.json`). Retraining the *same* token set with a different seed moved PPL by ~12", "+(449.5 → 437.8), so only gaps larger than ~25 PPL are treated as real below.", " ", " ## Hypothesis", " ", "-For a 30M-parameter LM trained from scratch on only 12M tokens, held-out perplexity on a", "-broad four-register target (encyclopedic / high-quality web prose / news / technical Q&A)", "-is governed by **how completely and how evenly the training set covers those four", "-registers**, and *not* by how aggressively low-quality documents are filtered out.", "+The disclosed target is an equal-parts mixture of four registers (encyclopedic /", "+high-quality web prose / news / technical Q&A). I claim that held-out perplexity at this", "+budget is governed by **selecting the most *typical* members of each target register and", "+spending an equal token share on each**, and that **hand-written quality filters are", "+actively harmful** because they silently delete whole registers of the target.", " ", "-Concretely I claim two things:", "+Two sub-claims:", " ", "-1. **Register balance is the effective lever.** Assigning every pool document to the", "-   target register it most resembles (log-odds against a raw-pool background class, from", "-   a bag-of-hashed-character-4-grams classifier trained on the *decoded, format-stripped*", "-   disclosed target) and then interleaving the four ranked lists **by token count**, so", "-   that the 12M-token prefix holds ~25% of each register, lowers perplexity substantially", "-   below a random selection.", "-2. **Extra \"quality\" gating is actively harmful past a very low bar.** Beyond a minimum", "-   document length (~1000 characters), each additional narrowing of the candidate pool", "-   *raises* perplexity, because at this budget the model is still learning generic", "-   English and a narrower pool means fewer distinct topics and phrasings per token spent.", "+1. **Typicality ranking is the lever.** Train a bag-of-hashed-character-4-grams", "+   classifier to separate the four registers of the decoded, format-stripped target from", "+   a random-pool background class. Assign every document to its argmax register and rank", "+   it by log-odds against the background class; fill the budget from the four ranked lists", "+   interleaved *by token count*. Predicted: large gain over random. Measured: **341.3 vs", "+   469.9 (-27%)**.", "+2. **Filtering costs more than it buys.** Every heuristic gate I added — a Gopher/C4-style", "+   boilerplate-and-fluency gate, and a Mahalanobis gate on the target's own character", "+   statistics — *raised* perplexity, monotonically in strictness. Measured: 341.3 (no", "+   gate) → 404.6 (junk gate) → 449.5 (junk + surface gate @0.70) → 497.3 (@0.40).", " ", " ## Mechanism", " ", "-The model sees 24.6M training tokens (3000 iters x 32 x 256) drawn from a 12M-token set —", "-about two epochs. In that regime almost all of the achievable loss reduction comes from", "-generic English statistics *in the target's registers*; capacity spent on a register the", "-target does not contain is wasted, but so is coverage lost to over-filtering. A quality", "-filter with a target-shaped acceptance region (my Mahalanobis \"surface-distribution\" gate)", "-removes documents that are perfectly good English simply because their character", "-statistics sit in the tail, and what it removes is disproportionately the *diversity* the", "-model needs.", "+The model sees 24.6M training tokens (3000 x 32 x 256) drawn from a 12M-token set, ~2", "+epochs, at 30M parameters. In that regime the model is still acquiring generic English", "+inside the target's registers, so what matters is that each register is (a) present in", "+proportion and (b) represented by its most prototypical documents — those are the ones", "+whose n-gram statistics the eval windows actually re-use.", " ", "+The filters fail for a concrete, checkable reason: **they are register-selective**. My junk", "+gate rejects symbols, digits, code punctuation and short lines, which is exactly what", "+technical Q&A looks like. Under the gate, the Q&A list could supply only 1.6M tokens, so", "+the mixture came out 31/31/31/6 instead of 25/25/25/25 — a quarter of the target was", "+starved to ~6% of the budget. Removing the gate raised the Q&A candidate pool from 1,675", "+documents to 17,893 and let the mixture reach exactly 25/25/25/25; that single change is", "+worth 63 PPL (404.6 → 341.3).", "+", " ### Observables predicted, other than the final perplexity", " ", "-1. **The pool is mostly \"general high-quality web prose.\"** A 5-way classifier", "-   (4 target registers + raw-pool background) should separate the encyclopedic, news and", "-   Q&A registers from the pool but should *fail* on the general-web-prose register,", "-   because that register and the pool are the same distribution.", "-   *Observed held-out accuracy: wiki 0.97, qa 0.94, news 0.79–0.84, pool 0.97,", "-   general-web-prose 0.11–0.35 (i.e. at/below chance vs. the pool class).*", "-2. **Top-ranked documents are recognisably the right register on inspection.**", "-   *Observed (`peek.py`): the wiki list returns Britannica/Wikipedia-style entries", "-   (\"Anglo-Dutch Wars, also called Dutch Wars…\"), the news list returns wire copy", "-   (\"LUCKNOW, India (Reuters) —…\"), the Q&A list returns programming help threads", "-   (servlet `getParameter()` quoting, `.sql` deployment scripts).*", "-3. **Monotone filter-strictness curve.** Perplexity should increase monotonically as the", "-   surface-distribution gate is tightened, all else equal.", "-   *Observed, at min-length 1000 chars with identical register balancing:", "-   keep-fraction 1.00 → **404.6**, 0.70 → 449.5, 0.40 → 497.3.*", "-4. **Non-monotone document-length curve with an interior optimum.** Forcing very short", "-   documents starves each 256-token window of coherent context; forcing very long ones", "-   buys fewer distinct topics per token.", "-   *Observed mean selected document length vs. PPL: 496 tok → 540.2, 780 → 493.8,", "-   947 → 449.5, 1874 → 469.2, 3622 → 478.5.*", "-5. **Balance-vs-order ablation.** Replacing the register-balanced order with a random", "-   order over the *same* candidate pool should lose most of the gain.", "-   *Observed: see the ablation table below.*", "+1. **The pool *is* mostly \"general high-quality web prose.\"** A 5-way classifier should", "+   separate encyclopedic, news and Q&A text from the raw pool, but should *fail* on the", "+   general-web-prose register, since that register and the pool are the same", "+   distribution. *Observed held-out accuracy: wiki 0.97, qa 0.94, news 0.79–0.84,", "+   pool 0.97, general-web-prose 0.11–0.35 (at/below chance).*", "+2. **Register-selective filter damage.** The junk gate should cut the Q&A register far", "+   harder than the others. *Observed: qa candidates 17,893 → 1,675 (-91%) vs wiki", "+   33,346 → 9,133 (-73%) and news 40,343 → 15,044 (-63%); resulting token mix", "+   25/25/25/25 → 31/31/31/6.*", "+3. **Top-ranked documents are recognisably the right register.** *Observed (`peek.py`):", "+   wiki list = Britannica/Wikipedia-style entries (\"Anglo-Dutch Wars, also called Dutch", "+   Wars, …\"), news list = wire copy (\"LUCKNOW, India (Reuters) — …\"), qa list =", "+   programming help threads (servlet `getParameter()` quoting, `.sql` deploy scripts).*", "+4. **Monotone strictness curve** (observable as a curve, not a single number):", "+   *keep-fraction 1.00 → 341.3/404.6, 0.70 → 449.5, 0.40 → 497.3.*", "+5. **Typicality, not balance, does the work.** Keeping the same four balanced buckets but", "+   shuffling *within* each bucket should erase most of the gain. *Observed: 445.0 with", "+   in-bucket shuffling vs 341.3 ranked — and 437.1 for a plain random order over the same", "+   length-floor pool, i.e. balance without ranking buys ~nothing.*", "+6. **Format artifacts must be stripped from the target before training the ranker.** The", "+   target's WikiText ` @,@ ` escapes / spaced punctuation and StackExchange HTML tags do", "+   not exist in the pool (pool documents containing `<p>`: 96 of 182,016; containing", "+   WikiText escapes: 0), so an un-normalised classifier would key on formatting that no", "+   pool document can match. *Observed: the 313 pool documents with spaced-punctuation", "+   statistics and the 205 with HTML markup are spam/product-listing junk, i.e. exactly the", "+   documents a format-matching ranker would promote.*", " ", " ## Falsification", " ", "-The claim is falsified if any of these hold:", "+Falsified if any of the following is observed:", " ", "-* A random order over the same minimum-length candidate pool matches the register-balanced", "-  order (would mean the classifier and the mixture contribute nothing, and the entire", "-  effect is the length floor).", "-* Tightening the surface-distribution gate *lowers* perplexity, or a strict", "-  quality-filtered selection beats the unfiltered-but-balanced one.", "-* Perplexity is monotone in mean document length (would refute the interior optimum, and", "-  the \"diversity per token\" mechanism behind it).", "-* The 5-way classifier separates general-web-prose from the pool as well as it separates", "-  Wikipedia (would refute observable 1, i.e. the pool is *not* mostly that register, and", "-  the balance argument would need re-deriving).", "+* Random order over the same length-floor candidate pool matches the register-ranked order", "+  (would mean the classifier contributes nothing). *Tested: 437.1 vs 341.3.*", "+* Tightening either gate lowers perplexity, or the strictly filtered selection beats the", "+  unfiltered-but-ranked one. *Tested: monotone the other way.*", "+* The gain survives removing the ranking but keeping the mixture. *Tested: it does not", "+  (445.0).*", "+* Over-weighting the register whose surface form the pool cannot match (Q&A, 40% instead", "+  of 25%) improves perplexity, i.e. the equal mixture is not the right mixture.", "+  *Tested: 354.8 vs 341.3 — no gain, consistent with matching the target's own mixture.*", "+* The classifier separates general-web-prose from the pool as well as it separates", "+  Wikipedia (would refute observable 1 and the \"pool is the residual register\" argument).", " ", "-Note on measurement precision: retraining the *same* 12M-token set with a different seed", "-moved dev PPL by ~12 (449.5 → 437.8), so only differences of roughly >25 PPL are treated", "-as real above; the strictness and length effects cited are 45–90 PPL.", "+## Full ablation table", " ", "-## Transfer", "+| selection rule | docs in 12M | mean doc (tok) | dev PPL |", "+|---|---|---|---|", "+| random ids (do-nothing) | 12938 | 928 | 469.9 |", "+| length floor >=1000 chars, random order | 8767 | 1369 | 437.1 |", "+| junk gate, random order | 12738 | 942 | 488.7 |", "+| **length floor + register-balanced typicality ranking (submitted)** | **10029** | **1197** | **341.3** |", "+| submitted, but shuffled within register buckets | 8284 | 1449 | 445.0 |", "+| submitted, but unbalanced (single global ranking) | 9787 | 1226 | see note |", "+| submitted + min 400 chars | 11925 | 1006 | 347.2 |", "+| submitted + min 2000 chars | 7715 | 1555 | 353.0 |", "+| submitted + 40% Q&A mixture | 9418 | 1274 | 354.8 |", "+| submitted + junk gate inside the web bucket only | 10063 | 1192 | 348.8 |", "+| submitted, ranked on 3 chunks/doc instead of the head | 9990 | 1201 | 352.0 |", "+| junk gate + ranking | 12442 | 964 | 404.6 |", "+| junk gate (min 500) + ranking | 14377 | 835 | 416.4 |", "+| junk + surface gate @0.70 + ranking | 12678 | 947 | 449.5 |", "+| junk + surface gate @0.40 + ranking | 12956 | 926 | 497.3 |", "+| junk + surface gate, no balance | 12676 | 947 | 453.6 |", "+| junk + surface gate, wiki-heavy mixture | 12551 | 956 | 451.4 |", "+| junk + surface gate, 800–4500 chars only | 24205 | 496 | 540.2 |", "+| junk + surface gate, 1200–12000 chars only | 15387 | 780 | 493.8 |", "+| junk + surface gate, >=4000 chars | 6404 | 1874 | 469.2 |", "+| junk + surface gate, >=8000 chars | 3313 | 3622 | 478.5 |", " ", "-The recipe needs no labels and no reference LM, so it transfers to any budgeted", "-pretraining-data curation where the evaluation target is disclosed as a mixture:", "+Document length is only weakly important once the gates are gone (347/341/353 for a 400 /", "+1000 / 2000-character floor); the strong length effects in the lower block of the table are", "+gate artifacts — restricting length inside an already-narrowed pool compounds the coverage", "+loss.", " ", "-1. Decode the target, **strip corpus-specific surface artifacts** (here WikiText's", "-   ` @,@ ` escapes and spaced punctuation, StackExchange's HTML tags and entities) —", "-   otherwise the classifier keys on formatting that does not exist in the pool and the", "-   register signal is lost;", "-2. train a cheap bag-of-hashed-4-grams classifier of *target register vs. raw pool*", "-   (seconds on one GPU, no tokenizer needed);", "-3. rank inside each register, then interleave **by token count** so that every prefix of", "-   the priority list — including whatever prefix the budget happens to cut at — carries", "-   the target's mixture;", "-4. filter only for junk you can defend (length floor, non-English, duplicated lines), and", "-   verify with an ablation that each filter you add is not costing you coverage.", "+## Transfer", " ", "-The same \"balance by token share, filter minimally\" conclusion should hold for any", "-compute-starved run in this regime, and the classifier is reusable as a router for", "-mixing weights when the target mixture is known but unbalanced.", "+The recipe needs no labels, no reference LM and no tokenizer for scoring, so it transfers", "+to any budgeted pretraining or fine-tuning selection where the eval target is disclosed as", "+a mixture:", "+", "+1. Decode the target and **strip corpus-specific surface artifacts** before using it as a", "+   reference, or the ranker keys on formatting the pool cannot supply (observable 6).", "+2. Train a cheap bag-of-hashed-4-grams classifier of *target register vs. raw pool* (~12 s", "+   on one GPU) and read off log-odds against the pool class as a typicality score.", "+3. Rank inside each register and interleave **by token count**, so that every prefix of the", "+   priority list — including the one the budget happens to cut at — carries the target", "+   mixture.", "+4. Add filters only with an ablation in hand. The generic advice \"filter web data hard\"", "+   inverts at small budgets: check, per filter, which register it deletes."]}], "originalFile": "# Claim: register-balanced coverage beats quality filtering at a 12M-token budget\n\n*(numbers are dev perplexity on `data/multi_dev.npy` with the frozen trainer, seed 1337\nunless stated; the random-selection reference measured here is **469.9**)*\n\n## Hypothesis\n\nFor a 30M-parameter LM trained from scratch on only 12M tokens, held-out perplexity on a\nbroad four-register target (encyclopedic / high-quality web prose / news / technical Q&A)\nis governed by **how completely and how evenly the training set covers those four\nregisters**, and *not* by how aggressively low-quality documents are filtered out.\n\nConcretely I claim two things:\n\n1. **Register balance is the effective lever.** Assigning every pool document to the\n   target register it most resembles (log-odds against a raw-pool background class, from\n   a bag-of-hashed-character-4-grams classifier trained on the *decoded, format-stripped*\n   disclosed target) and then interleaving the four ranked lists **by token count**, so\n   that the 12M-token prefix holds ~25% of each register, lowers perplexity substantially\n   below a random selection.\n2. **Extra \"quality\" gating is actively harmful past a very low bar.** Beyond a minimum\n   document length (~1000 characters), each additional narrowing of the candidate pool\n   *raises* perplexity, because at this budget the model is still learning generic\n   English and a narrower pool means fewer distinct topics and phrasings per token spent.\n\n## Mechanism\n\nThe model sees 24.6M training tokens (3000 iters x 32 x 256) drawn from a 12M-token set —\nabout two epochs. In that regime almost all of the achievable loss reduction comes from\ngeneric English statistics *in the target's registers*; capacity spent on a register the\ntarget does not contain is wasted, but so is coverage lost to over-filtering. A quality\nfilter with a target-shaped acceptance region (my Mahalanobis \"surface-distribution\" gate)\nremoves documents that are perfectly good English simply because their character\nstatistics sit in the tail, and what it removes is disproportionately the *diversity* the\nmodel needs.\n\n### Observables predicted, other than the final perplexity\n\n1. **The pool is mostly \"general high-quality web prose.\"** A 5-way classifier\n   (4 target registers + raw-pool background) should separate the encyclopedic, news and\n   Q&A registers from the pool but should *fail* on the general-web-prose register,\n   because that register and the pool are the same distribution.\n   *Observed held-out accuracy: wiki 0.97, qa 0.94, news 0.79–0.84, pool 0.97,\n   general-web-prose 0.11–0.35 (i.e. at/below chance vs. the pool class).*\n2. **Top-ranked documents are recognisably the right register on inspection.**\n   *Observed (`peek.py`): the wiki list returns Britannica/Wikipedia-style entries\n   (\"Anglo-Dutch Wars, also called Dutch Wars…\"), the news list returns wire copy\n   (\"LUCKNOW, India (Reuters) —…\"), the Q&A list returns programming help threads\n   (servlet `getParameter()` quoting, `.sql` deployment scripts).*\n3. **Monotone filter-strictness curve.** Perplexity should increase monotonically as the\n   surface-distribution gate is tightened, all else equal.\n   *Observed, at min-length 1000 chars with identical register balancing:\n   keep-fraction 1.00 → **404.6**, 0.70 → 449.5, 0.40 → 497.3.*\n4. **Non-monotone document-length curve with an interior optimum.** Forcing very short\n   documents starves each 256-token window of coherent context; forcing very long ones\n   buys fewer distinct topics per token.\n   *Observed mean selected document length vs. PPL: 496 tok → 540.2, 780 → 493.8,\n   947 → 449.5, 1874 → 469.2, 3622 → 478.5.*\n5. **Balance-vs-order ablation.** Replacing the register-balanced order with a random\n   order over the *same* candidate pool should lose most of the gain.\n   *Observed: see the ablation table below.*\n\n## Falsification\n\nThe claim is falsified if any of these hold:\n\n* A random order over the same minimum-length candidate pool matches the register-balanced\n  order (would mean the classifier and the mixture contribute nothing, and the entire\n  effect is the length floor).\n* Tightening the surface-distribution gate *lowers* perplexity, or a strict\n  quality-filtered selection beats the unfiltered-but-balanced one.\n* Perplexity is monotone in mean document length (would refute the interior optimum, and\n  the \"diversity per token\" mechanism behind it).\n* The 5-way classifier separates general-web-prose from the pool as well as it separates\n  Wikipedia (would refute observable 1, i.e. the pool is *not* mostly that register, and\n  the balance argument would need re-deriving).\n\nNote on measurement precision: retraining the *same* 12M-token set with a different seed\nmoved dev PPL by ~12 (449.5 → 437.8), so only differences of roughly >25 PPL are treated\nas real above; the strictness and length effects cited are 45–90 PPL.\n\n## Transfer\n\nThe recipe needs no labels and no reference LM, so it transfers to any budgeted\npretraining-data curation where the evaluation target is disclosed as a mixture:\n\n1. Decode the target, **strip corpus-specific surface artifacts** (here WikiText's\n   ` @,@ ` escapes and spaced punctuation, StackExchange's HTML tags and entities) —\n   otherwise the classifier keys on formatting that does not exist in the pool and the\n   register signal is lost;\n2. train a cheap bag-of-hashed-4-grams classifier of *target register vs. raw pool*\n   (seconds on one GPU, no tokenizer needed);\n3. rank inside each register, then interleave **by token count** so that every prefix of\n   the priority list — including whatever prefix the budget happens to cut at — carries\n   the target's mixture;\n4. filter only for junk you can defend (length floor, non-English, duplicated lines), and\n   verify with an ablation that each filter you add is not costing you coverage.\n\nThe same \"balance by token share, filter minimally\" conclusion should hold for any\ncompute-starved run in this regime, and the classifier is reusable as a router for\nmixing weights when the target mixture is known but unbalanced.\n", "userModified": false}

2787

[stdout]
2787
[metadata] {"noOutputExpected": false}

(Bash completed with no output)

[metadata] {"returnCodeInterpretation": "No matches found", "noOutputExpected": false}

154 bash -c  rm -f /tmp/ri-exec-1e6a5466a5e54f48a655f70cf0d6a6a4.pid setsid bash -c 'set -o pipefail; export PATH="$HOME/.local/bin:/usr/local/bin:$PATH"; claude --verbose --output-format=stream-json --permission-mode=bypassPermissions --disallowedTools WebSearch WebFetch --print -- '"'"'# Task: Curate a raw web corpus to train the best small language model  ## Context You are assembling the pretraining data for a small language model. You have a large pool of **raw web documents** at `/workspace/data/pool.jsonl` — one JSON object per line: `{"id": <int>, "text": "..."}`. The documents are unlabeled. The pool holds **182,016 documents (~200M GPT-2 tokens total)**.  Compute is fixed: the model is trained on a **fixed budget of 12,000,000 tokens** — only ~6% of the pool, far smaller than the whole thing. So you cannot train on everything: you must decide *which* documents are worth training on. The training pipeline tokenizes your selection (GPT-2 BPE) in priority order and stops once it has accumulated 12M tokens.  ## Goal Curate the pool: choose the documents that, trained on under the fixed budget, produce the **best language model**. Quality is measured by **held-out perplexity on a BROAD, multi-domain high-quality English target** — equal parts encyclopedic (Wikipedia), general high-quality web prose, news, and technical Q&A. This is your disclosed target: select data that makes the model good across *all* of these registers, not just one. Lower perplexity is better.  ## Deliverables 1. `/workspace/submission/selection.json` — a JSON list of pool `id`s to train    on, in **priority order** (best first). The training pipeline consumes your    list in order until the **12M-token** budget is filled, then trains. Provide    enough ids to cover at least 12M tokens (a few hundred MB of text is ample);    ids beyond the budget are simply unused. 2. `/workspace/submission/curate.py` — the reproducible script that produced    your selection from a stated criterion (not a hand-picked id list). 3. `/workspace/claim.md` — Hypothesis / Mechanism (predict an observable other    than the final perplexity) / Falsification / Transfer.  ## Resources - The **frozen training script** is provided (`/workspace/train_nano.py`, with   `model.py`). It trains a ~30M-parameter GPT from scratch on a token budget with   fixed hyperparameters. **You may run it to evaluate a candidate selection**   against a dev target (`/workspace/data/multi_dev.npy`) — one run is ~60s on the   GPU — but you **cannot change it**; the official run uses an identical frozen   copy. The only thing you control is the data selection. - Standard GPU sandbox (Python, PyTorch, transformers). How you assess document   quality is entirely up to you (filters, classifiers, heuristics, a model, etc.). - Your session ends when the wall-clock budget runs out — check   `/workspace/.timer/remaining_secs` at any point for the authoritative time left   (don'"'"'"'"'"'"'"'"'t assume a fixed number of hours). Every training run you do comes out of   that budget. - No internet. The pool'"'"'"'"'"'"'"'"'s documents are freshly assembled; you cannot look up   any quality labels — any quality signal must come from your own analysis.  ## How you are scored Official: your selection fills the fixed token budget, the frozen script trains the 30M GPT from scratch, and the verifier measures the **absolute held-out perplexity on the hidden high-quality target** (a different sample from the disclosed domain than the dev set). That perplexity is your score. **Lower is better** — the goal is simply to push held-out perplexity as low as you can.  The verifier reports the raw perplexity and does no normalization. For reference and for a separate downstream normalization step, the benchmark records two fixed anchors: a **baseline** of held-out perplexity from a random selection (the do-nothing reference), and a **theoretical best** of perplexity 1 (the perplexity floor, unreachable in practice). Aim to get your held-out perplexity well below the random baseline.  ## Constraints - `selection.json` must be produced by `curate.py` from a stated criterion; an   opaque hand-picked id list fails the publishability check even if it scores well. - A selection with duplicate or out-of-range ids, or too few tokens to fill the   budget, is marked invalid (no score recorded) and excluded from ranking, so it   can'"'"'"'"'"'"'"'"'t win and doesn'"'"'"'"'"'"'"'"'t distort averages. '"'"' 2>&1 </dev/null | tee /logs/agent/claude-code.txt' & child="$!" echo "$child" > /tmp/ri-exec-1e6a5466a5e54f48a655f70cf0d6a6a4.pid wait "$child" rc="$?" rm -f /tmp/ri-exec-1e6a5466a5e54f48a655f70cf0d6a6a4.pid exit "$rc" 
156 bash -c set -o pipefail; export PATH="$HOME/.local/bin:/usr/local/bin:$PATH"; claude --verbose --output-format=stream-json --permission-mode=bypassPermissions --disallowedTools WebSearch WebFetch --print -- '# Task: Curate a raw web corpus to train the best small language model  ## Context You are assembling the pretraining data for a small language model. You have a large pool of **raw web documents** at `/workspace/data/pool.jsonl` — one JSON object per line: `{"id": <int>, "text": "..."}`. The documents are unlabeled. The pool holds **182,016 documents (~200M GPT-2 tokens total)**.  Compute is fixed: the model is trained on a **fixed budget of 12,000,000 tokens** — only ~6% of the pool, far smaller than the whole thing. So you cannot train on everything: you must decide *which* documents are worth training on. The training pipeline tokenizes your selection (GPT-2 BPE) in priority order and stops once it has accumulated 12M tokens.  ## Goal Curate the pool: choose the documents that, trained on under the fixed budget, produce the **best language model**. Quality is measured by **held-out perplexity on a BROAD, multi-domain high-quality English target** — equal parts encyclopedic (Wikipedia), general high-quality web prose, news, and technical Q&A. This is your disclosed target: select data that makes the model good across *all* of these registers, not just one. Lower perplexity is better.  ## Deliverables 1. `/workspace/submission/selection.json` — a JSON list of pool `id`s to train    on, in **priority order** (best first). The training pipeline consumes your    list in order until the **12M-token** budget is filled, then trains. Provide    enough ids to cover at least 12M tokens (a few hundred MB of text is ample);    ids beyond the budget are simply unused. 2. `/workspace/submission/curate.py` — the reproducible script that produced    your selection from a stated criterion (not a hand-picked id list). 3. `/workspace/claim.md` — Hypothesis / Mechanism (predict an observable other    than the final perplexity) / Falsification / Transfer.  ## Resources - The **frozen training script** is provided (`/workspace/train_nano.py`, with   `model.py`). It trains a ~30M-parameter GPT from scratch on a token budget with   fixed hyperparameters. **You may run it to evaluate a candidate selection**   against a dev target (`/workspace/data/multi_dev.npy`) — one run is ~60s on the   GPU — but you **cannot change it**; the official run uses an identical frozen   copy. The only thing you control is the data selection. - Standard GPU sandbox (Python, PyTorch, transformers). How you assess document   quality is entirely up to you (filters, classifiers, heuristics, a model, etc.). - Your session ends when the wall-clock budget runs out — check   `/workspace/.timer/remaining_secs` at any point for the authoritative time left   (don'"'"'t assume a fixed number of hours). Every training run you do comes out of   that budget. - No internet. The pool'"'"'s documents are freshly assembled; you cannot look up   any quality labels — any quality signal must come from your own analysis.  ## How you are scored Official: your selection fills the fixed token budget, the frozen script trains the 30M GPT from scratch, and the verifier measures the **absolute held-out perplexity on the hidden high-quality target** (a different sample from the disclosed domain than the dev set). That perplexity is your score. **Lower is better** — the goal is simply to push held-out perplexity as low as you can.  The verifier reports the raw perplexity and does no normalization. For reference and for a separate downstream normalization step, the benchmark records two fixed anchors: a **baseline** of held-out perplexity from a random selection (the do-nothing reference), and a **theoretical best** of perplexity 1 (the perplexity floor, unreachable in practice). Aim to get your held-out perplexity well below the random baseline.  ## Constraints - `selection.json` must be produced by `curate.py` from a stated criterion; an   opaque hand-picked id list fails the publishability check even if it scores well. - A selection with duplicate or out-of-range ids, or too few tokens to fill the   budget, is marked invalid (no score recorded) and excluded from ranking, so it   can'"'"'t win and doesn'"'"'t distort averages. ' 2>&1 </dev/null | tee /logs/agent/claude-code.txt
157 claude --verbose --output-format=stream-json --permission-mode=bypassPermissions --disallowedTools WebSearch WebFetch --print -- # Task: Curate a raw web corpus to train the best small language model  ## Context You are assembling the pretraining data for a small language model. You have a large pool of **raw web documents** at `/workspace/data/pool.jsonl` — one JSON object per line: `{"id": <int>, "text": "..."}`. The documents are unlabeled. The pool holds **182,016 documents (~200M GPT-2 tokens total)**.  Compute is fixed: the model is trained on a **fixed budget of 12,000,000 tokens** — only ~6% of the pool, far smaller than the whole thing. So you cannot train on everything: you must decide *which* documents are worth training on. The training pipeline tokenizes your selection (GPT-2 BPE) in priority order and stops once it has accumulated 12M tokens.  ## Goal Curate the pool: choose the documents that, trained on under the fixed budget, produce the **best language model**. Quality is measured by **held-out perplexity on a BROAD, multi-domain high-quality English target** — equal parts encyclopedic (Wikipedia), general high-quality web prose, news, and technical Q&A. This is your disclosed target: select data that makes the model good across *all* of these registers, not just one. Lower perplexity is better.  ## Deliverables 1. `/workspace/submission/selection.json` — a JSON list of pool `id`s to train    on, in **priority order** (best first). The training pipeline consumes your    list in order until the **12M-token** budget is filled, then trains. Provide    enough ids to cover at least 12M tokens (a few hundred MB of text is ample);    ids beyond the budget are simply unused. 2. `/workspace/submission/curate.py` — the reproducible script that produced    your selection from a stated criterion (not a hand-picked id list). 3. `/workspace/claim.md` — Hypothesis / Mechanism (predict an observable other    than the final perplexity) / Falsification / Transfer.  ## Resources - The **frozen training script** is provided (`/workspace/train_nano.py`, with   `model.py`). It trains a ~30M-parameter GPT from scratch on a token budget with   fixed hyperparameters. **You may run it to evaluate a candidate selection**   against a dev target (`/workspace/data/multi_dev.npy`) — one run is ~60s on the   GPU — but you **cannot change it**; the official run uses an identical frozen   copy. The only thing you control is the data selection. - Standard GPU sandbox (Python, PyTorch, transformers). How you assess document   quality is entirely up to you (filters, classifiers, heuristics, a model, etc.). - Your session ends when the wall-clock budget runs out — check   `/workspace/.timer/remaining_secs` at any point for the authoritative time left   (don't assume a fixed number of hours). Every training run you do comes out of   that budget. - No internet. The pool's documents are freshly assembled; you cannot look up   any quality labels — any quality signal must come from your own analysis.  ## How you are scored Official: your selection fills the fixed token budget, the frozen script trains the 30M GPT from scratch, and the verifier measures the **absolute held-out perplexity on the hidden high-quality target** (a different sample from the disclosed domain than the dev set). That perplexity is your score. **Lower is better** — the goal is simply to push held-out perplexity as low as you can.  The verifier reports the raw perplexity and does no normalization. For reference and for a separate downstream normalization step, the benchmark records two fixed anchors: a **baseline** of held-out perplexity from a random selection (the do-nothing reference), and a **theoretical best** of perplexity 1 (the perplexity floor, unreachable in practice). Aim to get your held-out perplexity well below the random baseline.  ## Constraints - `selection.json` must be produced by `curate.py` from a stated criterion; an   opaque hand-picked id list fails the publishability check even if it scores well. - A selection with duplicate or out-of-range ids, or too few tokens to fill the   budget, is marked invalid (no score recorded) and excluded from ranking, so it   can't win and doesn't distort averages.

[stdout]
154 bash -c  rm -f /tmp/ri-exec-1e6a5466a5e54f48a655f70cf0d6a6a4.pid setsid bash -c 'set -o pipefail; export PATH="$HOME/.local/bin:/usr/local/bin:$PATH"; claude --verbose --output-format=stream-json --permission-mode=bypassPermissions --disallowedTools WebSearch WebFetch --print -- '"'"'# Task: Curate a raw web corpus to train the best small language model  ## Context You are assembling the pretraining data for a small language model. You have a large pool of **raw web documents** at `/workspace/data/pool.jsonl` — one JSON object per line: `{"id": <int>, "text": "..."}`. The documents are unlabeled. The pool holds **182,016 documents (~200M GPT-2 tokens total)**.  Compute is fixed: the model is trained on a **fixed budget of 12,000,000 tokens** — only ~6% of the pool, far smaller than the whole thing. So you cannot train on everything: you must decide *which* documents are worth training on. The training pipeline tokenizes your selection (GPT-2 BPE) in priority order and stops once it has accumulated 12M tokens.  ## Goal Curate the pool: choose the documents that, trained on under the fixed budget, produce the **best language model**. Quality is measured by **held-out perplexity on a BROAD, multi-domain high-quality English target** — equal parts encyclopedic (Wikipedia), general high-quality web prose, news, and technical Q&A. This is your disclosed target: select data that makes the model good across *all* of these registers, not just one. Lower perplexity is better.  ## Deliverables 1. `/workspace/submission/selection.json` — a JSON list of pool `id`s to train    on, in **priority order** (best first). The training pipeline consumes your    list in order until the **12M-token** budget is filled, then trains. Provide    enough ids to cover at least 12M tokens (a few hundred MB of text is ample);    ids beyond the budget are simply unused. 2. `/workspace/submission/curate.py` — the reproducible script that produced    your selection from a stated criterion (not a hand-picked id list). 3. `/workspace/claim.md` — Hypothesis / Mechanism (predict an observable other    than the final perplexity) / Falsification / Transfer.  ## Resources - The **frozen training script** is provided (`/workspace/train_nano.py`, with   `model.py`). It trains a ~30M-parameter GPT from scratch on a token budget with   fixed hyperparameters. **You may run it to evaluate a candidate selection**   against a dev target (`/workspace/data/multi_dev.npy`) — one run is ~60s on the   GPU — but you **cannot change it**; the official run uses an identical frozen   copy. The only thing you control is the data selection. - Standard GPU sandbox (Python, PyTorch, transformers). How you assess document   quality is entirely up to you (filters, classifiers, heuristics, a model, etc.). - Your session ends when the wall-clock budget runs out — check   `/workspace/.timer/remaining_secs` at any point for the authoritative time left   (don'"'"'"'"'"'"'"'"'t assume a fixed number of hours). Every training run you do comes out of   that budget. - No internet. The pool'"'"'"'"'"'"'"'"'s documents are freshly assembled; you cannot look up   any quality labels — any quality signal must come from your own analysis.  ## How you are scored Official: your selection fills the fixed token budget, the frozen script trains the 30M GPT from scratch, and the verifier measures the **absolute held-out perplexity on the hidden high-quality target** (a different sample from the disclosed domain than the dev set). That perplexity is your score. **Lower is better** — the goal is simply to push held-out perplexity as low as you can.  The verifier reports the raw perplexity and does no normalization. For reference and for a separate downstream normalization step, the benchmark records two fixed anchors: a **baseline** of held-out perplexity from a random selection (the do-nothing reference), and a **theoretical best** of perplexity 1 (the perplexity floor, unreachable in practice). Aim to get your held-out perplexity well below the random baseline.  ## Constraints - `selection.json` must be produced by `curate.py` from a stated criterion; an   opaque hand-picked id list fails the publishability check even if it scores well. - A selection with duplicate or out-of-range ids, or too few tokens to fill the   budget, is marked invalid (no score recorded) and excluded from ranking, so it   can'"'"'"'"'"'"'"'"'t win and doesn'"'"'"'"'"'"'"'"'t distort averages. '"'"' 2>&1 </dev/null | tee /logs/agent/claude-code.txt' & child="$!" echo "$child" > /tmp/ri-exec-1e6a5466a5e54f48a655f70cf0d6a6a4.pid wait "$child" rc="$?" rm -f /tmp/ri-exec-1e6a5466a5e54f48a655f70cf0d6a6a4.pid exit "$rc" 
156 bash -c set -o pipefail; export PATH="$HOME/.local/bin:/usr/local/bin:$PATH"; claude --verbose --output-format=stream-json --permission-mode=bypassPermissions --disallowedTools WebSearch WebFetch --print -- '# Task: Curate a raw web corpus to train the best small language model  ## Context You are assembling the pretraining data for a small language model. You have a large pool of **raw web documents** at `/workspace/data/pool.jsonl` — one JSON object per line: `{"id": <int>, "text": "..."}`. The documents are unlabeled. The pool holds **182,016 documents (~200M GPT-2 tokens total)**.  Compute is fixed: the model is trained on a **fixed budget of 12,000,000 tokens** — only ~6% of the pool, far smaller than the whole thing. So you cannot train on everything: you must decide *which* documents are worth training on. The training pipeline tokenizes your selection (GPT-2 BPE) in priority order and stops once it has accumulated 12M tokens.  ## Goal Curate the pool: choose the documents that, trained on under the fixed budget, produce the **best language model**. Quality is measured by **held-out perplexity on a BROAD, multi-domain high-quality English target** — equal parts encyclopedic (Wikipedia), general high-quality web prose, news, and technical Q&A. This is your disclosed target: select data that makes the model good across *all* of these registers, not just one. Lower perplexity is better.  ## Deliverables 1. `/workspace/submission/selection.json` — a JSON list of pool `id`s to train    on, in **priority order** (best first). The training pipeline consumes your    list in order until the **12M-token** budget is filled, then trains. Provide    enough ids to cover at least 12M tokens (a few hundred MB of text is ample);    ids beyond the budget are simply unused. 2. `/workspace/submission/curate.py` — the reproducible script that produced    your selection from a stated criterion (not a hand-picked id list). 3. `/workspace/claim.md` — Hypothesis / Mechanism (predict an observable other    than the final perplexity) / Falsification / Transfer.  ## Resources - The **frozen training script** is provided (`/workspace/train_nano.py`, with   `model.py`). It trains a ~30M-parameter GPT from scratch on a token budget with   fixed hyperparameters. **You may run it to evaluate a candidate selection**   against a dev target (`/workspace/data/multi_dev.npy`) — one run is ~60s on the   GPU — but you **cannot change it**; the official run uses an identical frozen   copy. The only thing you control is the data selection. - Standard GPU sandbox (Python, PyTorch, transformers). How you assess document   quality is entirely up to you (filters, classifiers, heuristics, a model, etc.). - Your session ends when the wall-clock budget runs out — check   `/workspace/.timer/remaining_secs` at any point for the authoritative time left   (don'"'"'t assume a fixed number of hours). Every training run you do comes out of   that budget. - No internet. The pool'"'"'s documents are freshly assembled; you cannot look up   any quality labels — any quality signal must come from your own analysis.  ## How you are scored Official: your selection fills the fixed token budget, the frozen script trains the 30M GPT from scratch, and the verifier measures the **absolute held-out perplexity on the hidden high-quality target** (a different sample from the disclosed domain than the dev set). That perplexity is your score. **Lower is better** — the goal is simply to push held-out perplexity as low as you can.  The verifier reports the raw perplexity and does no normalization. For reference and for a separate downstream normalization step, the benchmark records two fixed anchors: a **baseline** of held-out perplexity from a random selection (the do-nothing reference), and a **theoretical best** of perplexity 1 (the perplexity floor, unreachable in practice). Aim to get your held-out perplexity well below the random baseline.  ## Constraints - `selection.json` must be produced by `curate.py` from a stated criterion; an   opaque hand-picked id list fails the publishability check even if it scores well. - A selection with duplicate or out-of-range ids, or too few tokens to fill the   budget, is marked invalid (no score recorded) and excluded from ranking, so it   can'"'"'t win and doesn'"'"'t distort averages. ' 2>&1 </dev/null | tee /logs/agent/claude-code.txt
157 claude --verbose --output-format=stream-json --permission-mode=bypassPermissions --disallowedTools WebSearch WebFetch --print -- # Task: Curate a raw web corpus to train the best small language model  ## Context You are assembling the pretraining data for a small language model. You have a large pool of **raw web documents** at `/workspace/data/pool.jsonl` — one JSON object per line: `{"id": <int>, "text": "..."}`. The documents are unlabeled. The pool holds **182,016 documents (~200M GPT-2 tokens total)**.  Compute is fixed: the model is trained on a **fixed budget of 12,000,000 tokens** — only ~6% of the pool, far smaller than the whole thing. So you cannot train on everything: you must decide *which* documents are worth training on. The training pipeline tokenizes your selection (GPT-2 BPE) in priority order and stops once it has accumulated 12M tokens.  ## Goal Curate the pool: choose the documents that, trained on under the fixed budget, produce the **best language model**. Quality is measured by **held-out perplexity on a BROAD, multi-domain high-quality English target** — equal parts encyclopedic (Wikipedia), general high-quality web prose, news, and technical Q&A. This is your disclosed target: select data that makes the model good across *all* of these registers, not just one. Lower perplexity is better.  ## Deliverables 1. `/workspace/submission/selection.json` — a JSON list of pool `id`s to train    on, in **priority order** (best first). The training pipeline consumes your    list in order until the **12M-token** budget is filled, then trains. Provide    enough ids to cover at least 12M tokens (a few hundred MB of text is ample);    ids beyond the budget are simply unused. 2. `/workspace/submission/curate.py` — the reproducible script that produced    your selection from a stated criterion (not a hand-picked id list). 3. `/workspace/claim.md` — Hypothesis / Mechanism (predict an observable other    than the final perplexity) / Falsification / Transfer.  ## Resources - The **frozen training script** is provided (`/workspace/train_nano.py`, with   `model.py`). It trains a ~30M-parameter GPT from scratch on a token budget with   fixed hyperparameters. **You may run it to evaluate a candidate selection**   against a dev target (`/workspace/data/multi_dev.npy`) — one run is ~60s on the   GPU — but you **cannot change it**; the official run uses an identical frozen   copy. The only thing you control is the data selection. - Standard GPU sandbox (Python, PyTorch, transformers). How you assess document   quality is entirely up to you (filters, classifiers, heuristics, a model, etc.). - Your session ends when the wall-clock budget runs out — check   `/workspace/.timer/remaining_secs` at any point for the authoritative time left   (don't assume a fixed number of hours). Every training run you do comes out of   that budget. - No internet. The pool's documents are freshly assembled; you cannot look up   any quality labels — any quality signal must come from your own analysis.  ## How you are scored Official: your selection fills the fixed token budget, the frozen script trains the 30M GPT from scratch, and the verifier measures the **absolute held-out perplexity on the hidden high-quality target** (a different sample from the disclosed domain than the dev set). That perplexity is your score. **Lower is better** — the goal is simply to push held-out perplexity as low as you can.  The verifier reports the raw perplexity and does no normalization. For reference and for a separate downstream normalization step, the benchmark records two fixed anchors: a **baseline** of held-out perplexity from a random selection (the do-nothing reference), and a **theoretical best** of perplexity 1 (the perplexity floor, unreachable in practice). Aim to get your held-out perplexity well below the random baseline.  ## Constraints - `selection.json` must be produced by `curate.py` from a stated criterion; an   opaque hand-picked id list fails the publishability check even if it scores well. - A selection with duplicate or out-of-range ids, or too few tokens to fill the   budget, is marked invalid (no score recorded) and excluded from ranking, so it   can't win and doesn't distort averages.
[metadata] {"noOutputExpected": false}

ok

[stdout]
ok
[metadata] {"noOutputExpected": false}

/tmp/njpure.json: packed 12000000 tok, 10952 docs, mean doc 1096 tok -> /tmp/njpure_train.npy
HELDOUT_PPL 379.470  loss 5.9388  train_s 56  /tmp/njpure_train.npy
2467

[stdout]
/tmp/njpure.json: packed 12000000 tok, 10952 docs, mean doc 1096 tok -> /tmp/njpure_train.npy
HELDOUT_PPL 379.470  loss 5.9388  train_s 56  /tmp/njpure_train.npy
2467
[metadata] {"noOutputExpected": false}

qa     17893 docs   26.3M est tokens
selected 22747 docs, ~26.0M est tokens (834 near-duplicates dropped); register mix [0.25 0.25 0.25 0.25]
wrote /tmp/repro.json
identical: True 22747 22747

[stdout]
  qa     17893 docs   26.3M est tokens
selected 22747 docs, ~26.0M est tokens (834 near-duplicates dropped); register mix [0.25 0.25 0.25 0.25]
wrote /tmp/repro.json
identical: True 22747 22747
[metadata] {"noOutputExpected": false}

The file /workspace/claim.md has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/claim.md", "oldString": "| submitted, but unbalanced (single global ranking) | 9787 | 1226 | see note |", "newString": "| submitted, but unbalanced (single global ranking) | 10952 | 1096 | 379.5 |", "originalFile": "# Claim: at a 12M-token budget, *register-typicality ranking* beats quality filtering\n\nAll numbers are dev perplexity on `data/multi_dev.npy`, produced by the frozen trainer at\nits default settings (seed 1337, 12M training tokens). The do-nothing reference — a random\nselection of pool ids — measures **469.9**. The submitted selection measures **341.3**\n(verified end-to-end through `pack_selection.py` + `train_nano.py` on the exact submitted\n`selection.json`). Retraining the *same* token set with a different seed moved PPL by ~12\n(449.5 → 437.8), so only gaps larger than ~25 PPL are treated as real below.\n\n## Hypothesis\n\nThe disclosed target is an equal-parts mixture of four registers (encyclopedic /\nhigh-quality web prose / news / technical Q&A). I claim that held-out perplexity at this\nbudget is governed by **selecting the most *typical* members of each target register and\nspending an equal token share on each**, and that **hand-written quality filters are\nactively harmful** because they silently delete whole registers of the target.\n\nTwo sub-claims:\n\n1. **Typicality ranking is the lever.** Train a bag-of-hashed-character-4-grams\n   classifier to separate the four registers of the decoded, format-stripped target from\n   a random-pool background class. Assign every document to its argmax register and rank\n   it by log-odds against the background class; fill the budget from the four ranked lists\n   interleaved *by token count*. Predicted: large gain over random. Measured: **341.3 vs\n   469.9 (-27%)**.\n2. **Filtering costs more than it buys.** Every heuristic gate I added — a Gopher/C4-style\n   boilerplate-and-fluency gate, and a Mahalanobis gate on the target's own character\n   statistics — *raised* perplexity, monotonically in strictness. Measured: 341.3 (no\n   gate) → 404.6 (junk gate) → 449.5 (junk + surface gate @0.70) → 497.3 (@0.40).\n\n## Mechanism\n\nThe model sees 24.6M training tokens (3000 x 32 x 256) drawn from a 12M-token set, ~2\nepochs, at 30M parameters. In that regime the model is still acquiring generic English\ninside the target's registers, so what matters is that each register is (a) present in\nproportion and (b) represented by its most prototypical documents — those are the ones\nwhose n-gram statistics the eval windows actually re-use.\n\nThe filters fail for a concrete, checkable reason: **they are register-selective**. My junk\ngate rejects symbols, digits, code punctuation and short lines, which is exactly what\ntechnical Q&A looks like. Under the gate, the Q&A list could supply only 1.6M tokens, so\nthe mixture came out 31/31/31/6 instead of 25/25/25/25 — a quarter of the target was\nstarved to ~6% of the budget. Removing the gate raised the Q&A candidate pool from 1,675\ndocuments to 17,893 and let the mixture reach exactly 25/25/25/25; that single change is\nworth 63 PPL (404.6 → 341.3).\n\n### Observables predicted, other than the final perplexity\n\n1. **The pool *is* mostly \"general high-quality web prose.\"** A 5-way classifier should\n   separate encyclopedic, news and Q&A text from the raw pool, but should *fail* on the\n   general-web-prose register, since that register and the pool are the same\n   distribution. *Observed held-out accuracy: wiki 0.97, qa 0.94, news 0.79–0.84,\n   pool 0.97, general-web-prose 0.11–0.35 (at/below chance).*\n2. **Register-selective filter damage.** The junk gate should cut the Q&A register far\n   harder than the others. *Observed: qa candidates 17,893 → 1,675 (-91%) vs wiki\n   33,346 → 9,133 (-73%) and news 40,343 → 15,044 (-63%); resulting token mix\n   25/25/25/25 → 31/31/31/6.*\n3. **Top-ranked documents are recognisably the right register.** *Observed (`peek.py`):\n   wiki list = Britannica/Wikipedia-style entries (\"Anglo-Dutch Wars, also called Dutch\n   Wars, …\"), news list = wire copy (\"LUCKNOW, India (Reuters) — …\"), qa list =\n   programming help threads (servlet `getParameter()` quoting, `.sql` deploy scripts).*\n4. **Monotone strictness curve** (observable as a curve, not a single number):\n   *keep-fraction 1.00 → 341.3/404.6, 0.70 → 449.5, 0.40 → 497.3.*\n5. **Typicality, not balance, does the work.** Keeping the same four balanced buckets but\n   shuffling *within* each bucket should erase most of the gain. *Observed: 445.0 with\n   in-bucket shuffling vs 341.3 ranked — and 437.1 for a plain random order over the same\n   length-floor pool, i.e. balance without ranking buys ~nothing.*\n6. **Format artifacts must be stripped from the target before training the ranker.** The\n   target's WikiText ` @,@ ` escapes / spaced punctuation and StackExchange HTML tags do\n   not exist in the pool (pool documents containing `<p>`: 96 of 182,016; containing\n   WikiText escapes: 0), so an un-normalised classifier would key on formatting that no\n   pool document can match. *Observed: the 313 pool documents with spaced-punctuation\n   statistics and the 205 with HTML markup are spam/product-listing junk, i.e. exactly the\n   documents a format-matching ranker would promote.*\n\n## Falsification\n\nFalsified if any of the following is observed:\n\n* Random order over the same length-floor candidate pool matches the register-ranked order\n  (would mean the classifier contributes nothing). *Tested: 437.1 vs 341.3.*\n* Tightening either gate lowers perplexity, or the strictly filtered selection beats the\n  unfiltered-but-ranked one. *Tested: monotone the other way.*\n* The gain survives removing the ranking but keeping the mixture. *Tested: it does not\n  (445.0).*\n* Over-weighting the register whose surface form the pool cannot match (Q&A, 40% instead\n  of 25%) improves perplexity, i.e. the equal mixture is not the right mixture.\n  *Tested: 354.8 vs 341.3 — no gain, consistent with matching the target's own mixture.*\n* The classifier separates general-web-prose from the pool as well as it separates\n  Wikipedia (would refute observable 1 and the \"pool is the residual register\" argument).\n\n## Full ablation table\n\n| selection rule | docs in 12M | mean doc (tok) | dev PPL |\n|---|---|---|---|\n| random ids (do-nothing) | 12938 | 928 | 469.9 |\n| length floor >=1000 chars, random order | 8767 | 1369 | 437.1 |\n| junk gate, random order | 12738 | 942 | 488.7 |\n| **length floor + register-balanced typicality ranking (submitted)** | **10029** | **1197** | **341.3** |\n| submitted, but shuffled within register buckets | 8284 | 1449 | 445.0 |\n| submitted, but unbalanced (single global ranking) | 9787 | 1226 | see note |\n| submitted + min 400 chars | 11925 | 1006 | 347.2 |\n| submitted + min 2000 chars | 7715 | 1555 | 353.0 |\n| submitted + 40% Q&A mixture | 9418 | 1274 | 354.8 |\n| submitted + junk gate inside the web bucket only | 10063 | 1192 | 348.8 |\n| submitted, ranked on 3 chunks/doc instead of the head | 9990 | 1201 | 352.0 |\n| junk gate + ranking | 12442 | 964 | 404.6 |\n| junk gate (min 500) + ranking | 14377 | 835 | 416.4 |\n| junk + surface gate @0.70 + ranking | 12678 | 947 | 449.5 |\n| junk + surface gate @0.40 + ranking | 12956 | 926 | 497.3 |\n| junk + surface gate, no balance | 12676 | 947 | 453.6 |\n| junk + surface gate, wiki-heavy mixture | 12551 | 956 | 451.4 |\n| junk + surface gate, 800–4500 chars only | 24205 | 496 | 540.2 |\n| junk + surface gate, 1200–12000 chars only | 15387 | 780 | 493.8 |\n| junk + surface gate, >=4000 chars | 6404 | 1874 | 469.2 |\n| junk + surface gate, >=8000 chars | 3313 | 3622 | 478.5 |\n\nDocument length is only weakly important once the gates are gone (347/341/353 for a 400 /\n1000 / 2000-character floor); the strong length effects in the lower block of the table are\ngate artifacts — restricting length inside an already-narrowed pool compounds the coverage\nloss.\n\n## Transfer\n\nThe recipe needs no labels, no reference LM and no tokenizer for scoring, so it transfers\nto any budgeted pretraining or fine-tuning selection where the eval target is disclosed as\na mixture:\n\n1. Decode the target and **strip corpus-specific surface artifacts** before using it as a\n   reference, or the ranker keys on formatting the pool cannot supply (observable 6).\n2. Train a cheap bag-of-hashed-4-grams classifier of *target register vs. raw pool* (~12 s\n   on one GPU) and read off log-odds against the pool class as a typicality score.\n3. Rank inside each register and interleave **by token count**, so that every prefix of the\n   priority list — including the one the budget happens to cut at — carries the target\n   mixture.\n4. Add filters only with an ablation in hand. The generic advice \"filter web data hard\"\n   inverts at small budgets: check, per filter, which register it deletes.\n", "structuredPatch": [{"oldStart": 98, "oldLines": 7, "newStart": 98, "newLines": 7, "lines": [" | junk gate, random order | 12738 | 942 | 488.7 |", " | **length floor + register-balanced typicality ranking (submitted)** | **10029** | **1197** | **341.3** |", " | submitted, but shuffled within register buckets | 8284 | 1449 | 445.0 |", "-| submitted, but unbalanced (single global ranking) | 9787 | 1226 | see note |", "+| submitted, but unbalanced (single global ranking) | 10952 | 1096 | 379.5 |", " | submitted + min 400 chars | 11925 | 1006 | 347.2 |", " | submitted + min 2000 chars | 7715 | 1555 | 353.0 |", " | submitted + 40% Q&A mixture | 9418 | 1274 | 354.8 |"]}], "userModified": false, "replaceAll": false}

The file /workspace/claim.md has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/claim.md", "oldString": "5. **Typicality, not balance, does the work.** Keeping the same four balanced buckets but\n   shuffling *within* each bucket should erase most of the gain. *Observed: 445.0 with\n   in-bucket shuffling vs 341.3 ranked — and 437.1 for a plain random order over the same\n   length-floor pool, i.e. balance without ranking buys ~nothing.*", "newString": "5. **Typicality is necessary, balance is only useful on top of it.** Keeping the same four\n   balanced buckets but shuffling *within* each bucket should erase most of the gain, while\n   ranking without balancing should keep most of it. *Observed: in-bucket shuffling 445.0\n   ~ plain random order over the same length-floor pool 437.1 (balance alone buys nothing);\n   single global ranking without balancing 379.5; ranking + balance 341.3. Decomposition:\n   ranking 437 -> 379 (-58), then balance 379 -> 341 (-38).*", "originalFile": "# Claim: at a 12M-token budget, *register-typicality ranking* beats quality filtering\n\nAll numbers are dev perplexity on `data/multi_dev.npy`, produced by the frozen trainer at\nits default settings (seed 1337, 12M training tokens). The do-nothing reference — a random\nselection of pool ids — measures **469.9**. The submitted selection measures **341.3**\n(verified end-to-end through `pack_selection.py` + `train_nano.py` on the exact submitted\n`selection.json`). Retraining the *same* token set with a different seed moved PPL by ~12\n(449.5 → 437.8), so only gaps larger than ~25 PPL are treated as real below.\n\n## Hypothesis\n\nThe disclosed target is an equal-parts mixture of four registers (encyclopedic /\nhigh-quality web prose / news / technical Q&A). I claim that held-out perplexity at this\nbudget is governed by **selecting the most *typical* members of each target register and\nspending an equal token share on each**, and that **hand-written quality filters are\nactively harmful** because they silently delete whole registers of the target.\n\nTwo sub-claims:\n\n1. **Typicality ranking is the lever.** Train a bag-of-hashed-character-4-grams\n   classifier to separate the four registers of the decoded, format-stripped target from\n   a random-pool background class. Assign every document to its argmax register and rank\n   it by log-odds against the background class; fill the budget from the four ranked lists\n   interleaved *by token count*. Predicted: large gain over random. Measured: **341.3 vs\n   469.9 (-27%)**.\n2. **Filtering costs more than it buys.** Every heuristic gate I added — a Gopher/C4-style\n   boilerplate-and-fluency gate, and a Mahalanobis gate on the target's own character\n   statistics — *raised* perplexity, monotonically in strictness. Measured: 341.3 (no\n   gate) → 404.6 (junk gate) → 449.5 (junk + surface gate @0.70) → 497.3 (@0.40).\n\n## Mechanism\n\nThe model sees 24.6M training tokens (3000 x 32 x 256) drawn from a 12M-token set, ~2\nepochs, at 30M parameters. In that regime the model is still acquiring generic English\ninside the target's registers, so what matters is that each register is (a) present in\nproportion and (b) represented by its most prototypical documents — those are the ones\nwhose n-gram statistics the eval windows actually re-use.\n\nThe filters fail for a concrete, checkable reason: **they are register-selective**. My junk\ngate rejects symbols, digits, code punctuation and short lines, which is exactly what\ntechnical Q&A looks like. Under the gate, the Q&A list could supply only 1.6M tokens, so\nthe mixture came out 31/31/31/6 instead of 25/25/25/25 — a quarter of the target was\nstarved to ~6% of the budget. Removing the gate raised the Q&A candidate pool from 1,675\ndocuments to 17,893 and let the mixture reach exactly 25/25/25/25; that single change is\nworth 63 PPL (404.6 → 341.3).\n\n### Observables predicted, other than the final perplexity\n\n1. **The pool *is* mostly \"general high-quality web prose.\"** A 5-way classifier should\n   separate encyclopedic, news and Q&A text from the raw pool, but should *fail* on the\n   general-web-prose register, since that register and the pool are the same\n   distribution. *Observed held-out accuracy: wiki 0.97, qa 0.94, news 0.79–0.84,\n   pool 0.97, general-web-prose 0.11–0.35 (at/below chance).*\n2. **Register-selective filter damage.** The junk gate should cut the Q&A register far\n   harder than the others. *Observed: qa candidates 17,893 → 1,675 (-91%) vs wiki\n   33,346 → 9,133 (-73%) and news 40,343 → 15,044 (-63%); resulting token mix\n   25/25/25/25 → 31/31/31/6.*\n3. **Top-ranked documents are recognisably the right register.** *Observed (`peek.py`):\n   wiki list = Britannica/Wikipedia-style entries (\"Anglo-Dutch Wars, also called Dutch\n   Wars, …\"), news list = wire copy (\"LUCKNOW, India (Reuters) — …\"), qa list =\n   programming help threads (servlet `getParameter()` quoting, `.sql` deploy scripts).*\n4. **Monotone strictness curve** (observable as a curve, not a single number):\n   *keep-fraction 1.00 → 341.3/404.6, 0.70 → 449.5, 0.40 → 497.3.*\n5. **Typicality, not balance, does the work.** Keeping the same four balanced buckets but\n   shuffling *within* each bucket should erase most of the gain. *Observed: 445.0 with\n   in-bucket shuffling vs 341.3 ranked — and 437.1 for a plain random order over the same\n   length-floor pool, i.e. balance without ranking buys ~nothing.*\n6. **Format artifacts must be stripped from the target before training the ranker.** The\n   target's WikiText ` @,@ ` escapes / spaced punctuation and StackExchange HTML tags do\n   not exist in the pool (pool documents containing `<p>`: 96 of 182,016; containing\n   WikiText escapes: 0), so an un-normalised classifier would key on formatting that no\n   pool document can match. *Observed: the 313 pool documents with spaced-punctuation\n   statistics and the 205 with HTML markup are spam/product-listing junk, i.e. exactly the\n   documents a format-matching ranker would promote.*\n\n## Falsification\n\nFalsified if any of the following is observed:\n\n* Random order over the same length-floor candidate pool matches the register-ranked order\n  (would mean the classifier contributes nothing). *Tested: 437.1 vs 341.3.*\n* Tightening either gate lowers perplexity, or the strictly filtered selection beats the\n  unfiltered-but-ranked one. *Tested: monotone the other way.*\n* The gain survives removing the ranking but keeping the mixture. *Tested: it does not\n  (445.0).*\n* Over-weighting the register whose surface form the pool cannot match (Q&A, 40% instead\n  of 25%) improves perplexity, i.e. the equal mixture is not the right mixture.\n  *Tested: 354.8 vs 341.3 — no gain, consistent with matching the target's own mixture.*\n* The classifier separates general-web-prose from the pool as well as it separates\n  Wikipedia (would refute observable 1 and the \"pool is the residual register\" argument).\n\n## Full ablation table\n\n| selection rule | docs in 12M | mean doc (tok) | dev PPL |\n|---|---|---|---|\n| random ids (do-nothing) | 12938 | 928 | 469.9 |\n| length floor >=1000 chars, random order | 8767 | 1369 | 437.1 |\n| junk gate, random order | 12738 | 942 | 488.7 |\n| **length floor + register-balanced typicality ranking (submitted)** | **10029** | **1197** | **341.3** |\n| submitted, but shuffled within register buckets | 8284 | 1449 | 445.0 |\n| submitted, but unbalanced (single global ranking) | 10952 | 1096 | 379.5 |\n| submitted + min 400 chars | 11925 | 1006 | 347.2 |\n| submitted + min 2000 chars | 7715 | 1555 | 353.0 |\n| submitted + 40% Q&A mixture | 9418 | 1274 | 354.8 |\n| submitted + junk gate inside the web bucket only | 10063 | 1192 | 348.8 |\n| submitted, ranked on 3 chunks/doc instead of the head | 9990 | 1201 | 352.0 |\n| junk gate + ranking | 12442 | 964 | 404.6 |\n| junk gate (min 500) + ranking | 14377 | 835 | 416.4 |\n| junk + surface gate @0.70 + ranking | 12678 | 947 | 449.5 |\n| junk + surface gate @0.40 + ranking | 12956 | 926 | 497.3 |\n| junk + surface gate, no balance | 12676 | 947 | 453.6 |\n| junk + surface gate, wiki-heavy mixture | 12551 | 956 | 451.4 |\n| junk + surface gate, 800–4500 chars only | 24205 | 496 | 540.2 |\n| junk + surface gate, 1200–12000 chars only | 15387 | 780 | 493.8 |\n| junk + surface gate, >=4000 chars | 6404 | 1874 | 469.2 |\n| junk + surface gate, >=8000 chars | 3313 | 3622 | 478.5 |\n\nDocument length is only weakly important once the gates are gone (347/341/353 for a 400 /\n1000 / 2000-character floor); the strong length effects in the lower block of the table are\ngate artifacts — restricting length inside an already-narrowed pool compounds the coverage\nloss.\n\n## Transfer\n\nThe recipe needs no labels, no reference LM and no tokenizer for scoring, so it transfers\nto any budgeted pretraining or fine-tuning selection where the eval target is disclosed as\na mixture:\n\n1. Decode the target and **strip corpus-specific surface artifacts** before using it as a\n   reference, or the ranker keys on formatting the pool cannot supply (observable 6).\n2. Train a cheap bag-of-hashed-4-grams classifier of *target register vs. raw pool* (~12 s\n   on one GPU) and read off log-odds against the pool class as a typicality score.\n3. Rank inside each register and interleave **by token count**, so that every prefix of the\n   priority list — including the one the budget happens to cut at — carries the target\n   mixture.\n4. Add filters only with an ablation in hand. The generic advice \"filter web data hard\"\n   inverts at small budgets: check, per filter, which register it deletes.\n", "structuredPatch": [{"oldStart": 61, "oldLines": 10, "newStart": 61, "newLines": 12, "lines": ["    programming help threads (servlet `getParameter()` quoting, `.sql` deploy scripts).*", " 4. **Monotone strictness curve** (observable as a curve, not a single number):", "    *keep-fraction 1.00 → 341.3/404.6, 0.70 → 449.5, 0.40 → 497.3.*", "-5. **Typicality, not balance, does the work.** Keeping the same four balanced buckets but", "-   shuffling *within* each bucket should erase most of the gain. *Observed: 445.0 with", "-   in-bucket shuffling vs 341.3 ranked — and 437.1 for a plain random order over the same", "-   length-floor pool, i.e. balance without ranking buys ~nothing.*", "+5. **Typicality is necessary, balance is only useful on top of it.** Keeping the same four", "+   balanced buckets but shuffling *within* each bucket should erase most of the gain, while", "+   ranking without balancing should keep most of it. *Observed: in-bucket shuffling 445.0", "+   ~ plain random order over the same length-floor pool 437.1 (balance alone buys nothing);", "+   single global ranking without balancing 379.5; ranking + balance 341.3. Decomposition:", "+   ranking 437 -> 379 (-58), then balance 379 -> 341 (-38).*", " 6. **Format artifacts must be stripped from the target before training the ranker.** The", "    target's WikiText ` @,@ ` escapes / spaced punctuation and StackExchange HTML tags do", "    not exist in the pool (pool documents containing `<p>`: 96 of 182,016; containing"]}], "userModified": false, "replaceAll": false}

SUBMITTED s= HELDOUT_PPL 341.268  loss 5.8327  train_s 61  /tmp/dev_train.npy
SUBMITTED s= HELDOUT_PPL 341.804  loss 5.8342  train_s 56  /tmp/dev_train.npy
2296

[stdout]
SUBMITTED s= HELDOUT_PPL 341.268  loss 5.8327  train_s 61  /tmp/dev_train.npy
SUBMITTED s= HELDOUT_PPL 341.804  loss 5.8342  train_s 56  /tmp/dev_train.npy
2296
[metadata] {"noOutputExpected": false}

launched

[stdout]
launched
[metadata] {"noOutputExpected": false}

The file /workspace/claim.md has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/claim.md", "oldString": "`selection.json`). Retraining the *same* token set with a different seed moved PPL by ~12\n(449.5 → 437.8), so only gaps larger than ~25 PPL are treated as real below.", "newString": "`selection.json`). Seed sensitivity was measured directly by retraining the *same* token\nset: the submitted set gives 341.26 / 341.27 / 341.80 at seeds 1337 / 555 / 999, but one of\nthe filtered sets moved by 12 (449.5 → 437.8), so gaps under ~15 PPL are not treated as\nmeaningful below. Every conclusion drawn rests on a gap of 25 PPL or more.", "originalFile": "# Claim: at a 12M-token budget, *register-typicality ranking* beats quality filtering\n\nAll numbers are dev perplexity on `data/multi_dev.npy`, produced by the frozen trainer at\nits default settings (seed 1337, 12M training tokens). The do-nothing reference — a random\nselection of pool ids — measures **469.9**. The submitted selection measures **341.3**\n(verified end-to-end through `pack_selection.py` + `train_nano.py` on the exact submitted\n`selection.json`). Retraining the *same* token set with a different seed moved PPL by ~12\n(449.5 → 437.8), so only gaps larger than ~25 PPL are treated as real below.\n\n## Hypothesis\n\nThe disclosed target is an equal-parts mixture of four registers (encyclopedic /\nhigh-quality web prose / news / technical Q&A). I claim that held-out perplexity at this\nbudget is governed by **selecting the most *typical* members of each target register and\nspending an equal token share on each**, and that **hand-written quality filters are\nactively harmful** because they silently delete whole registers of the target.\n\nTwo sub-claims:\n\n1. **Typicality ranking is the lever.** Train a bag-of-hashed-character-4-grams\n   classifier to separate the four registers of the decoded, format-stripped target from\n   a random-pool background class. Assign every document to its argmax register and rank\n   it by log-odds against the background class; fill the budget from the four ranked lists\n   interleaved *by token count*. Predicted: large gain over random. Measured: **341.3 vs\n   469.9 (-27%)**.\n2. **Filtering costs more than it buys.** Every heuristic gate I added — a Gopher/C4-style\n   boilerplate-and-fluency gate, and a Mahalanobis gate on the target's own character\n   statistics — *raised* perplexity, monotonically in strictness. Measured: 341.3 (no\n   gate) → 404.6 (junk gate) → 449.5 (junk + surface gate @0.70) → 497.3 (@0.40).\n\n## Mechanism\n\nThe model sees 24.6M training tokens (3000 x 32 x 256) drawn from a 12M-token set, ~2\nepochs, at 30M parameters. In that regime the model is still acquiring generic English\ninside the target's registers, so what matters is that each register is (a) present in\nproportion and (b) represented by its most prototypical documents — those are the ones\nwhose n-gram statistics the eval windows actually re-use.\n\nThe filters fail for a concrete, checkable reason: **they are register-selective**. My junk\ngate rejects symbols, digits, code punctuation and short lines, which is exactly what\ntechnical Q&A looks like. Under the gate, the Q&A list could supply only 1.6M tokens, so\nthe mixture came out 31/31/31/6 instead of 25/25/25/25 — a quarter of the target was\nstarved to ~6% of the budget. Removing the gate raised the Q&A candidate pool from 1,675\ndocuments to 17,893 and let the mixture reach exactly 25/25/25/25; that single change is\nworth 63 PPL (404.6 → 341.3).\n\n### Observables predicted, other than the final perplexity\n\n1. **The pool *is* mostly \"general high-quality web prose.\"** A 5-way classifier should\n   separate encyclopedic, news and Q&A text from the raw pool, but should *fail* on the\n   general-web-prose register, since that register and the pool are the same\n   distribution. *Observed held-out accuracy: wiki 0.97, qa 0.94, news 0.79–0.84,\n   pool 0.97, general-web-prose 0.11–0.35 (at/below chance).*\n2. **Register-selective filter damage.** The junk gate should cut the Q&A register far\n   harder than the others. *Observed: qa candidates 17,893 → 1,675 (-91%) vs wiki\n   33,346 → 9,133 (-73%) and news 40,343 → 15,044 (-63%); resulting token mix\n   25/25/25/25 → 31/31/31/6.*\n3. **Top-ranked documents are recognisably the right register.** *Observed (`peek.py`):\n   wiki list = Britannica/Wikipedia-style entries (\"Anglo-Dutch Wars, also called Dutch\n   Wars, …\"), news list = wire copy (\"LUCKNOW, India (Reuters) — …\"), qa list =\n   programming help threads (servlet `getParameter()` quoting, `.sql` deploy scripts).*\n4. **Monotone strictness curve** (observable as a curve, not a single number):\n   *keep-fraction 1.00 → 341.3/404.6, 0.70 → 449.5, 0.40 → 497.3.*\n5. **Typicality is necessary, balance is only useful on top of it.** Keeping the same four\n   balanced buckets but shuffling *within* each bucket should erase most of the gain, while\n   ranking without balancing should keep most of it. *Observed: in-bucket shuffling 445.0\n   ~ plain random order over the same length-floor pool 437.1 (balance alone buys nothing);\n   single global ranking without balancing 379.5; ranking + balance 341.3. Decomposition:\n   ranking 437 -> 379 (-58), then balance 379 -> 341 (-38).*\n6. **Format artifacts must be stripped from the target before training the ranker.** The\n   target's WikiText ` @,@ ` escapes / spaced punctuation and StackExchange HTML tags do\n   not exist in the pool (pool documents containing `<p>`: 96 of 182,016; containing\n   WikiText escapes: 0), so an un-normalised classifier would key on formatting that no\n   pool document can match. *Observed: the 313 pool documents with spaced-punctuation\n   statistics and the 205 with HTML markup are spam/product-listing junk, i.e. exactly the\n   documents a format-matching ranker would promote.*\n\n## Falsification\n\nFalsified if any of the following is observed:\n\n* Random order over the same length-floor candidate pool matches the register-ranked order\n  (would mean the classifier contributes nothing). *Tested: 437.1 vs 341.3.*\n* Tightening either gate lowers perplexity, or the strictly filtered selection beats the\n  unfiltered-but-ranked one. *Tested: monotone the other way.*\n* The gain survives removing the ranking but keeping the mixture. *Tested: it does not\n  (445.0).*\n* Over-weighting the register whose surface form the pool cannot match (Q&A, 40% instead\n  of 25%) improves perplexity, i.e. the equal mixture is not the right mixture.\n  *Tested: 354.8 vs 341.3 — no gain, consistent with matching the target's own mixture.*\n* The classifier separates general-web-prose from the pool as well as it separates\n  Wikipedia (would refute observable 1 and the \"pool is the residual register\" argument).\n\n## Full ablation table\n\n| selection rule | docs in 12M | mean doc (tok) | dev PPL |\n|---|---|---|---|\n| random ids (do-nothing) | 12938 | 928 | 469.9 |\n| length floor >=1000 chars, random order | 8767 | 1369 | 437.1 |\n| junk gate, random order | 12738 | 942 | 488.7 |\n| **length floor + register-balanced typicality ranking (submitted)** | **10029** | **1197** | **341.3** |\n| submitted, but shuffled within register buckets | 8284 | 1449 | 445.0 |\n| submitted, but unbalanced (single global ranking) | 10952 | 1096 | 379.5 |\n| submitted + min 400 chars | 11925 | 1006 | 347.2 |\n| submitted + min 2000 chars | 7715 | 1555 | 353.0 |\n| submitted + 40% Q&A mixture | 9418 | 1274 | 354.8 |\n| submitted + junk gate inside the web bucket only | 10063 | 1192 | 348.8 |\n| submitted, ranked on 3 chunks/doc instead of the head | 9990 | 1201 | 352.0 |\n| junk gate + ranking | 12442 | 964 | 404.6 |\n| junk gate (min 500) + ranking | 14377 | 835 | 416.4 |\n| junk + surface gate @0.70 + ranking | 12678 | 947 | 449.5 |\n| junk + surface gate @0.40 + ranking | 12956 | 926 | 497.3 |\n| junk + surface gate, no balance | 12676 | 947 | 453.6 |\n| junk + surface gate, wiki-heavy mixture | 12551 | 956 | 451.4 |\n| junk + surface gate, 800–4500 chars only | 24205 | 496 | 540.2 |\n| junk + surface gate, 1200–12000 chars only | 15387 | 780 | 493.8 |\n| junk + surface gate, >=4000 chars | 6404 | 1874 | 469.2 |\n| junk + surface gate, >=8000 chars | 3313 | 3622 | 478.5 |\n\nDocument length is only weakly important once the gates are gone (347/341/353 for a 400 /\n1000 / 2000-character floor); the strong length effects in the lower block of the table are\ngate artifacts — restricting length inside an already-narrowed pool compounds the coverage\nloss.\n\n## Transfer\n\nThe recipe needs no labels, no reference LM and no tokenizer for scoring, so it transfers\nto any budgeted pretraining or fine-tuning selection where the eval target is disclosed as\na mixture:\n\n1. Decode the target and **strip corpus-specific surface artifacts** before using it as a\n   reference, or the ranker keys on formatting the pool cannot supply (observable 6).\n2. Train a cheap bag-of-hashed-4-grams classifier of *target register vs. raw pool* (~12 s\n   on one GPU) and read off log-odds against the pool class as a typicality score.\n3. Rank inside each register and interleave **by token count**, so that every prefix of the\n   priority list — including the one the budget happens to cut at — carries the target\n   mixture.\n4. Add filters only with an ablation in hand. The generic advice \"filter web data hard\"\n   inverts at small budgets: check, per filter, which register it deletes.\n", "structuredPatch": [{"oldStart": 4, "oldLines": 8, "newStart": 4, "newLines": 10, "lines": [" its default settings (seed 1337, 12M training tokens). The do-nothing reference — a random", " selection of pool ids — measures **469.9**. The submitted selection measures **341.3**", " (verified end-to-end through `pack_selection.py` + `train_nano.py` on the exact submitted", "-`selection.json`). Retraining the *same* token set with a different seed moved PPL by ~12", "-(449.5 → 437.8), so only gaps larger than ~25 PPL are treated as real below.", "+`selection.json`). Seed sensitivity was measured directly by retraining the *same* token", "+set: the submitted set gives 341.26 / 341.27 / 341.80 at seeds 1337 / 555 / 999, but one of", "+the filtered sets moved by 12 (449.5 → 437.8), so gaps under ~15 PPL are not treated as", "+meaningful below. Every conclusion drawn rests on a gap of 25 PPL or more.", " ", " ## Hypothesis", " "]}], "userModified": false, "replaceAll": false}

/tmp/n3000.json: packed 12000000 tok, 10114 docs, mean doc 1186 tok -> /tmp/n3000_train.npy
/tmp/mc1400.json: packed 12000000 tok, 8948 docs, mean doc 1341 tok -> /tmp/mc1400_train.npy
n3000 HELDOUT_PPL 340.467  loss 5.8303  train_s 56  /tmp/n3000_train.npy
mc1400 HELDOUT_PPL 345.262  loss 5.8443  train_s 56  /tmp/mc1400_train.npy
2066

[stdout]
/tmp/n3000.json: packed 12000000 tok, 10114 docs, mean doc 1186 tok -> /tmp/n3000_train.npy
/tmp/mc1400.json: packed 12000000 tok, 8948 docs, mean doc 1341 tok -> /tmp/mc1400_train.npy
n3000 HELDOUT_PPL 340.467  loss 5.8303  train_s 56  /tmp/n3000_train.npy
mc1400 HELDOUT_PPL 345.262  loss 5.8443  train_s 56  /tmp/mc1400_train.npy
2066
[metadata] {"noOutputExpected": false}

--- deliverables:
-rw-r--r-- 1 root root 9654 Aug  4 00:21 claim.md

submission/:
total 174
drwxr-xr-x 1 root root    160 Aug  4 00:05 .
drwxr-xr-x 1 root root    660 Aug  4 00:18 ..
drwxr-xr-x 1 root root     80 Aug  4 00:05 __pycache__
-rw-r--r-- 1 root root  16154 Aug  3 23:58 curate.py
-rw-r--r-- 1 root root 161023 Aug  4 00:03 selection.json
ids: 22747 unique: True min 6 max 181997
132

[stdout]
--- deliverables:
-rw-r--r-- 1 root root 9654 Aug  4 00:21 claim.md

submission/:
total 174
drwxr-xr-x 1 root root    160 Aug  4 00:05 .
drwxr-xr-x 1 root root    660 Aug  4 00:18 ..
drwxr-xr-x 1 root root     80 Aug  4 00:05 __pycache__
-rw-r--r-- 1 root root  16154 Aug  3 23:58 curate.py
-rw-r--r-- 1 root root 161023 Aug  4 00:03 selection.json
ids: 22747 unique: True min 6 max 181997
132
[metadata] {"noOutputExpected": false}

1956
---
__pycache__
curate.py
selection.json
type ok: True
all in pool: True | n: 22747 | dupes: 0

[stdout]
1956
---
__pycache__
curate.py
selection.json
type ok: True
all in pool: True | n: 22747 | dupes: 0
[metadata] {"noOutputExpected": false}

66:ap.add_argument("--out", default="/workspace/submission/selection.json")

[stdout]
66:ap.add_argument("--out", default="/workspace/submission/selection.json")
[metadata] {"noOutputExpected": false}

target reference docs: {'wiki': 1546, 'web': 181, 'news': 444, 'qa': 358}
classifier held-out acc: wiki=0.97 web=0.11 news=0.79 qa=0.94 __pool__=0.97
length floor keeps 139147 / 182016
  wiki   33346 docs   44.7M est tokens
  web    47565 docs   56.6M est tokens
  news   40343 docs   42.4M est tokens
  qa     17893 docs   26.3M est tokens
selected 22747 docs, ~26.0M est tokens (834 near-duplicates dropped); register mix [0.25 0.25 0.25 0.25]
wrote /tmp/rerun.json
=== identical to submitted:
YES

[stdout]
target reference docs: {'wiki': 1546, 'web': 181, 'news': 444, 'qa': 358}
classifier held-out acc: wiki=0.97 web=0.11 news=0.79 qa=0.94 __pool__=0.97
length floor keeps 139147 / 182016
  wiki   33346 docs   44.7M est tokens
  web    47565 docs   56.6M est tokens
  news   40343 docs   42.4M est tokens
  qa     17893 docs   26.3M est tokens
selected 22747 docs, ~26.0M est tokens (834 near-duplicates dropped); register mix [0.25 0.25 0.25 0.25]
wrote /tmp/rerun.json
=== identical to submitted:
YES
[metadata] {"noOutputExpected": false}

File created successfully at: /logs/agent/sessions/projects/-workspace/memory/small-budget-filters-hurt.md

[metadata] {"type": "create", "filePath": "/logs/agent/sessions/projects/-workspace/memory/small-budget-filters-hurt.md", "content": "---\nname: small-budget-filters-hurt\ndescription: Measured on the /workspace 12M-token curation benchmark — heuristic junk filters raise held-out perplexity because they delete a whole target register\nmetadata: \n  node_type: memory\n  type: project\n  originSessionId: f21eea37-10b1-429e-b8f9-c24d864d7c2c\n  modified: 2026-08-04T00:24:15.464Z\n---\n\nOn the 12M-token data-curation benchmark in `/workspace` (frozen nanoGPT trainer, dev\ntarget = equal parts Wikipedia / web prose / news / technical Q&A), Gopher/C4-style\nquality filters made perplexity *worse*, monotonically in strictness: 341 (no gate) →\n405 (junk gate) → 450 → 497. Ranking documents by register typicality (bag-of-hashed\nchar-4-grams classifier, target register vs. raw pool, log-odds against the pool class)\nand interleaving the four registers *by token count* gave 341.3 vs 469.9 random.\nDecomposition: ranking −58 PPL, then token-share balance −38 PPL.\n\n**Why:** symbol/digit/short-line heuristics are register-selective — they read technical\nQ&A as junk, so the Q&A candidate pool fell 17,893 → 1,675 docs and the mixture skewed to\n31/31/31/6. At a small budget, coverage of the target mixture dominates per-document\ncleanliness.\n\n**How to apply:** for budgeted selection against a disclosed target mixture, rank by\ntypicality and balance token shares before reaching for filters; add each filter only\nwith an ablation showing which register it deletes. Also strip corpus-specific surface\nartifacts (WikiText ` @,@ `, StackExchange HTML) from the target before using it as a\nclassifier reference — otherwise the ranker promotes the pool's spam, which is the only\nthing carrying that formatting.\n", "structuredPatch": [], "originalFile": null, "userModified": false, "memdirStamped": true}

(no MEMORY.md)
Shell cwd was reset to /workspace

[stdout]
(no MEMORY.md)
[stderr]

Shell cwd was reset to /workspace
[metadata] {"noOutputExpected": false}

File created successfully at: /logs/agent/sessions/projects/-workspace/memory/MEMORY.md (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/logs/agent/sessions/projects/-workspace/memory/MEMORY.md", "content": "- [Small-budget filters hurt](small-budget-filters-hurt.md) — on the 12M-token curation benchmark, junk filters raised perplexity by starving a target register; typicality ranking + token-balanced registers won.\n", "structuredPatch": [], "originalFile": null, "userModified": false}