claude-opus-5

lives only in documents that any quality filter throws away.

## Mechanism (observables other than the final perplexity)

**M1 — unigram CE is the mediating quantity.** It is measurable on the packed
12M-token array before any training, and it should order the selections the same
way perplexity does:

| selection | unigram CE ↓ | dev PPL ↓ |
|---|---|---|
| random (do-nothing baseline) | 8.129 | 477.8 |
| 4-register quality/domain classifier + hard prose gate | 8.039 | 365.0 |
| greedy CE matching, hard prose gate | 7.907 | 296.1 |
| greedy CE matching + capped non-prose allowance | 7.741 | **272.6** |

Across all 18 selections trained in this study, Pearson `r`(unigram CE of the
packed array, log dev PPL) = **0.925**. So a quantity computable in seconds from
the packed tokens, with no gradient steps, explains ~86% of the variance in the
log perplexity of a 3000-iteration training run. That is the mechanism claim:
the selector is not "finding good documents", it is moving `q` toward `p`, and
moving `q` toward `p` is what moves the loss.

**M2 — the avoidable loss is concentrated on identifiable token types, and it is
a *surface-form* deficit, not a topical one.** Decomposing the target by register
and comparing each block's token mass `p(t)` with a random web sample's `q(t)`:

- encyclopedic block: `" ,"` 4.63%, `" ."` 3.27%, `" @"` 0.96% of its tokens
  (WikiText detokenisation escapes) — each ≤0.04% in a random pool sample;
- Q&A block: `" "` 9.31% (HTML indentation), `">"` 4.07%, `"</"` 1.50%, `"code"`
  1.13% — each ≤0.15% in a random pool sample.

[stdout]
lives only in documents that any quality filter throws away.

## Mechanism (observables other than the final perplexity)

**M1 — unigram CE is the mediating quantity.** It is measurable on the packed
12M-token array before any training, and it should order the selections the same
way perplexity does:

| selection | unigram CE ↓ | dev PPL ↓ |
|---|---|---|
| random (do-nothing baseline) | 8.129 | 477.8 |
| 4-register quality/domain classifier + hard prose gate | 8.039 | 365.0 |
| greedy CE matching, hard prose gate | 7.907 | 296.1 |
| greedy CE matching + capped non-prose allowance | 7.741 | **272.6** |

Across all 18 selections trained in this study, Pearson `r`(unigram CE of the
packed array, log dev PPL) = **0.925**. So a quantity computable in seconds from
the packed tokens, with no gradient steps, explains ~86% of the variance in the
log perplexity of a 3000-iteration training run. That is the mechanism claim:
the selector is not "finding good documents", it is moving `q` toward `p`, and
moving `q` toward `p` is what moves the loss.

**M2 — the avoidable loss is concentrated on identifiable token types, and it is
a *surface-form* deficit, not a topical one.** Decomposing the target by register
and comparing each block's token mass `p(t)` with a random web sample's `q(t)`:

- encyclopedic block: `" ,"` 4.63%, `" ."` 3.27%, `" @"` 0.96% of its tokens
  (WikiText detokenisation escapes) — each ≤0.04% in a random pool sample;
- Q&A block: `" "` 9.31% (HTML indentation), `">"` 4.07%, `"</"` 1.50%, `"code"`
  1.13% — each ≤0.15% in a random pool sample.
[metadata] {"noOutputExpected": false}

The file /workspace/claim.md has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/claim.md", "oldString": "30M-parameter GPT, seed 1337.\n\n## Hypothesis", "newString": "30M-parameter GPT, seed 1337.\n\nResult on our own held-out split of the disclosed target: random selection 477.8\n→ **279.3** (−42%), with a fastText-style quality/domain classifier at 365.0 in\nbetween. The selector never sees a quality label; it minimises one scalar.\n\n## Hypothesis", "originalFile": "# Curating a 12M-token pretraining set by token-distribution coverage\n\nPool: 182,016 raw web documents / 199.9M GPT-2 tokens. Budget: 12M tokens (6% of\nthe pool). Target: held-out perplexity on an equal-parts mixture of encyclopedic\nprose, general high-quality web prose, news, and technical Q&A. Frozen trainer,\n30M-parameter GPT, seed 1337.\n\n## Hypothesis\n\nAt a small fixed token budget with a *disclosed* target mixture, held-out\nperplexity is governed primarily by **how completely the selected data's BPE\ntoken distribution covers the target's**, and only secondarily by document-level\n\"quality\". Consequently, greedily choosing documents to minimise the unigram\ncross-entropy\n\n    CE(S) = - Σ_t p_target(t) · log q_S(t)\n\nshould beat both random selection and a strong quality/domain classifier — and,\ncrucially, it should beat a *pure* quality filter, because some target token mass\nlives only in documents that any quality filter throws away.\n\n## Mechanism (observables other than the final perplexity)\n\n**M1 — unigram CE is the mediating quantity.** It is measurable on the packed\n12M-token array before any training, and it should order the selections the same\nway perplexity does:\n\n| selection | unigram CE ↓ | dev PPL ↓ |\n|---|---|---|\n| random (do-nothing baseline) | 8.129 | 477.8 |\n| 4-register quality/domain classifier + hard prose gate | 8.039 | 365.0 |\n| greedy CE matching, hard prose gate | 7.907 | 296.1 |\n| greedy CE matching + capped non-prose allowance | 7.741 | **272.6** |\n\nAcross all 18 selections trained in this study, Pearson `r`(unigram CE of the\npacked array, log dev PPL) = **0.925**. So a quantity computable in seconds from\nthe packed tokens, with no gradient steps, explains ~86% of the variance in the\nlog perplexity of a 3000-iteration training run. That is the mechanism claim:\nthe selector is not \"finding good documents\", it is moving `q` toward `p`, and\nmoving `q` toward `p` is what moves the loss.\n\n**M2 — the avoidable loss is concentrated on identifiable token types, and it is\na *surface-form* deficit, not a topical one.** Decomposing the target by register\nand comparing each block's token mass `p(t)` with a random web sample's `q(t)`:\n\n- encyclopedic block: `\" ,\"` 4.63%, `\" .\"` 3.27%, `\" @\"` 0.96% of its tokens\n  (WikiText detokenisation escapes) — each ≤0.04% in a random pool sample;\n- Q&A block: `\" \"` 9.31% (HTML indentation), `\">\"` 4.07%, `\"</\"` 1.50%, `\"code\"`\n  1.13% — each ≤0.15% in a random pool sample.\n\nPrediction: an intervention that raises `q` for *one* block's scarce token types\nmoves *that block's* perplexity and leaves the others roughly alone. Observed —\nrelaxing the prose gate (which had been excluding markup-heavy documents) moved\nthe Q&A block 229.9 → **132.0** (−43%) while wiki/web/news moved by ≤ +12%.\n\n**M2b — that scarce mass is unreachable by any quality-first pipeline.** Where the\nneeded token types actually live, by tier (share of each type's total pool count):\n\n| token type | passes prose gate | English but not prose-layout | fails even \"is this English?\" |\n|---|---|---|---|\n| `\" \"` (indentation) | 2.7% | 1.3% | **96.1%** |\n| `\">\"` | 6.8% | 2.2% | **91.0%** |\n| `\"</\"` | 10.9% | 4.9% | 84.1% |\n| `\" ,\"` | 18.2% | 4.9% | 76.9% |\n\nSo the mass cannot be recovered by *loosening* a quality filter along some\nlinguistic axis — we checked a middle tier that keeps the language tests\n(stopword rate, word length, non-ASCII) and drops the layout tests, and it holds\nonly 1.3% of the indentation mass. A hard budget allowance for documents that are\nnot prose at all is the only route to it, which is why the method needs one.\n\n**M3 — coverage saturates, so the non-prose allowance must be U-shaped.** The\ngreedy weight `w = p/q` falls as `q` catches up with `p`, so once markup coverage\nis bought, further non-prose documents are pure dilution. Predicted and observed\nminimum at ≈25% of the budget:\n\n| non-prose allowance | 0% | 12% | 20% | **25%** | 30% | 40% | uncapped |\n|---|---|---|---|---|---|---|---|\n| dev PPL | 296.1 | 288.6 | 278.7 | **272.6** | 279.5 | 280.3 | 281.0 |\n\n`curate.py` does not hard-code that allowance. It sweeps it and picks by unigram\nCE, which is free to evaluate — no training required, which is the point of M1:\n\n| allowance | 0% | 10% | 16% | 20% | 24% | **28%** | 32% | 40% | uncapped |\n|---|---|---|---|---|---|---|---|---|---|\n| budget CE ↓ | 7.9062 | 7.7668 | 7.7538 | 7.7440 | 7.7389 | 7.7368 | 7.7359 | 7.7369 | 7.7369 |\n\nThe curve is steep below ~0.2 and then flat to <0.002 nats. Its exact argmin\n(32%) is inside the noise, so the script takes the *smallest* allowance within\n`CE_TOL = 0.001` nats of the best — 28%. The tie-break is deliberately toward\nless non-prose: CE is a first-order objective that cannot see the cost of\nmarkup, so where CE is indifferent, prefer prose. The submitted selection is\nthat config (41,285 ids / 36.1M tokens of priority-ordered candidates for a 12M\nbudget), dev PPL **279.3** — per register: wiki 599.8, web 264.8, news 280.5,\nQ&A 136.1.\n\n## Falsification\n\n- **F1 (quality-first).** If the driver were generic quality, tightening the\n  prose gate would help monotonically. It does not: the hard gate is *worse* than\n  the capped-allowance gate (296.1 vs 272.6), and a 4-register quality/domain\n  classifier with a hard gate is far worse (365.0) despite 97.5% held-out\n  register accuracy. Selecting the documents a quality filter *rejects* is what\n  bought the largest single win (Q&A −43%).\n- **F2 (the markup documents are doing the work).** Hold the token count fixed\n  and replace every gate-failing document with the next-best prose document\n  (`--junk_frac 0`). The Q&A block must revert toward ~230 while the other three\n  stay within a few percent. If Q&A stayed near 140, coverage would not be the\n  mechanism. Observed: Q&A 132.0 → 229.9 (+74%), others within 8%. Confirmed.\n- **F3 (a pool-imposed floor).** The pool holds only 0.018% `\" ,\"` tokens, so the\n  maximum attainable `q(\" ,\")` inside a 12M-token budget is ≈0.30% against a\n  4.63% target share — the encyclopedic deficit is *unbuyable* from this pool.\n  Prediction: no selection from this pool takes the encyclopedic block below\n  ~450 PPL. Any selection reaching <400 there falsifies the claim that its\n  residual loss is dominated by unattainable surface tokens. (Best seen: 490.9,\n  by explicitly over-weighting that block — which cost more elsewhere than it\n  gained, net 313.6.)\n- **F4 (mediation).** A selection with materially lower unigram CE but higher\n  perplexity (beyond ~±2%) falsifies M1.\n\nHonest limit on resolution: two selections that differ by ~6% of their documents\ndiffer by ~±7 PPL (e.g. changing only how the smoothing prior is estimated moved\n272.6 → 279.5). Everything inside the 272–281 band is one band, not a ranking;\nonly the gaps to 296 / 365 / 478 are resolved by this experiment.\n\n## Transfer\n\nRequirements are only (i) a sample of the target distribution and (ii) the\ntraining tokenizer — no labels, no reference LM, no proxy model. Cost is one\ntokenisation pass over the pool plus one sparse mat-vec per greedy round.\n\nWhere it should transfer:\n\n- Any fixed-budget pretraining or continued-pretraining selection where the\n  evaluation mixture can be sampled: domain adaptation, multilingual mixes,\n  code+prose mixes. The greedy self-balances across sub-domains without being\n  told the mixture weights, because `w = p/q` automatically stops rewarding\n  token types that are already covered.\n- Best where the target has surface/format idiosyncrasies — markup, detokenised\n  punctuation, indentation, transliteration. That is exactly where quality-first\n  pipelines fail, because the documents carrying those forms *look* like junk.\n\nWhere it should weaken:\n\n- As the budget approaches the pool size, coverage saturates and the method\n  becomes a no-op (at 100% selection it is the identity).\n- When pool and target are already distributionally matched, there is nothing to\n  buy; expect gains to shrink toward zero rather than reverse.\n- The objective is first-order (unigram). It cannot create structure the pool\n  lacks — hence the F3 floor — and left uncapped it is gameable by token-dense\n  degenerate documents. The single tuned quantity in the whole method is the\n  non-prose allowance that guards against this (~25% here).\n\n## Reproduction\n\n`submission/curate.py` produces `submission/selection.json` from the stated\ncriterion alone — no randomness, no seeds, no hand-picked ids, no workspace\nimports. One tokenisation pass over the pool (cached), then the greedy. Its\noutput packed by the official `pack_selection.py` is bit-identical to the array\nthese numbers were measured on.\n\nDeterminism note: the ranking mat-vec runs in float64. In float32 the cuSPARSE\nreduction order made ties resolve differently between runs — the selected\ndocument *set* was identical at every prefix (100% overlap at 5k / 20k / all),\nbut documents permuted within a greedy batch, which matters at the 12M truncation\nboundary. float64 removes that.\n\nValidity: 41,285 unique ids, all present in the pool, 36.1M tokens — 3.0× the\nbudget, so the packer fills 12,000,000 tokens exactly (from the first 16,620\ndocuments) with no risk of a short pack.\n", "structuredPatch": [{"oldStart": 5, "oldLines": 6, "newStart": 5, "newLines": 10, "lines": [" prose, general high-quality web prose, news, and technical Q&A. Frozen trainer,", " 30M-parameter GPT, seed 1337.", " ", "+Result on our own held-out split of the disclosed target: random selection 477.8", "+→ **279.3** (−42%), with a fastText-style quality/domain classifier at 365.0 in", "+between. The selector never sees a quality label; it minimises one scalar.", "+", " ## Hypothesis", " ", " At a small fixed token budget with a *disclosed* target mixture, held-out"]}], "userModified": false, "replaceAll": false}

wrote /workspace/submission/selection.json: 41285 ids, 36.1M tokens (3.0x the 12M budget)
wrote /tmp/repro2.json: 41285 ids, 36.1M tokens (3.0x the 12M budget)
REPRO bit-identical: True 41285
packed 12000000 tokens from 41285 ids -> /tmp/tr_S2.npy
PER_REGISTER wiki=591.9  web=264.7  news=282.3  qa=135.3
HELDOUT_PPL 278.349  loss 5.6289  train_s 54  /tmp/tr_S2.npy
---
1715

[stdout]
wrote /workspace/submission/selection.json: 41285 ids, 36.1M tokens (3.0x the 12M budget)
wrote /tmp/repro2.json: 41285 ids, 36.1M tokens (3.0x the 12M budget)
REPRO bit-identical: True 41285
packed 12000000 tokens from 41285 ids -> /tmp/tr_S2.npy
PER_REGISTER wiki=591.9  web=264.7  news=282.3  qa=135.3
HELDOUT_PPL 278.349  loss 5.6289  train_s 54  /tmp/tr_S2.npy
---
1715
[metadata] {"noOutputExpected": false}

The file /workspace/claim.md has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/claim.md", "oldString": "→ **279.3** (−42%), with a fastText-style quality/domain classifier at 365.0 in", "newString": "→ **278.3** (−42%), with a fastText-style quality/domain classifier at 365.0 in", "originalFile": "# Curating a 12M-token pretraining set by token-distribution coverage\n\nPool: 182,016 raw web documents / 199.9M GPT-2 tokens. Budget: 12M tokens (6% of\nthe pool). Target: held-out perplexity on an equal-parts mixture of encyclopedic\nprose, general high-quality web prose, news, and technical Q&A. Frozen trainer,\n30M-parameter GPT, seed 1337.\n\nResult on our own held-out split of the disclosed target: random selection 477.8\n→ **279.3** (−42%), with a fastText-style quality/domain classifier at 365.0 in\nbetween. The selector never sees a quality label; it minimises one scalar.\n\n## Hypothesis\n\nAt a small fixed token budget with a *disclosed* target mixture, held-out\nperplexity is governed primarily by **how completely the selected data's BPE\ntoken distribution covers the target's**, and only secondarily by document-level\n\"quality\". Consequently, greedily choosing documents to minimise the unigram\ncross-entropy\n\n    CE(S) = - Σ_t p_target(t) · log q_S(t)\n\nshould beat both random selection and a strong quality/domain classifier — and,\ncrucially, it should beat a *pure* quality filter, because some target token mass\nlives only in documents that any quality filter throws away.\n\n## Mechanism (observables other than the final perplexity)\n\n**M1 — unigram CE is the mediating quantity.** It is measurable on the packed\n12M-token array before any training, and it should order the selections the same\nway perplexity does:\n\n| selection | unigram CE ↓ | dev PPL ↓ |\n|---|---|---|\n| random (do-nothing baseline) | 8.129 | 477.8 |\n| 4-register quality/domain classifier + hard prose gate | 8.039 | 365.0 |\n| greedy CE matching, hard prose gate | 7.907 | 296.1 |\n| greedy CE matching + capped non-prose allowance | 7.741 | **272.6** |\n\nAcross all 18 selections trained in this study, Pearson `r`(unigram CE of the\npacked array, log dev PPL) = **0.925**. So a quantity computable in seconds from\nthe packed tokens, with no gradient steps, explains ~86% of the variance in the\nlog perplexity of a 3000-iteration training run. That is the mechanism claim:\nthe selector is not \"finding good documents\", it is moving `q` toward `p`, and\nmoving `q` toward `p` is what moves the loss.\n\n**M2 — the avoidable loss is concentrated on identifiable token types, and it is\na *surface-form* deficit, not a topical one.** Decomposing the target by register\nand comparing each block's token mass `p(t)` with a random web sample's `q(t)`:\n\n- encyclopedic block: `\" ,\"` 4.63%, `\" .\"` 3.27%, `\" @\"` 0.96% of its tokens\n  (WikiText detokenisation escapes) — each ≤0.04% in a random pool sample;\n- Q&A block: `\" \"` 9.31% (HTML indentation), `\">\"` 4.07%, `\"</\"` 1.50%, `\"code\"`\n  1.13% — each ≤0.15% in a random pool sample.\n\nPrediction: an intervention that raises `q` for *one* block's scarce token types\nmoves *that block's* perplexity and leaves the others roughly alone. Observed —\nrelaxing the prose gate (which had been excluding markup-heavy documents) moved\nthe Q&A block 229.9 → **132.0** (−43%) while wiki/web/news moved by ≤ +12%.\n\n**M2b — that scarce mass is unreachable by any quality-first pipeline.** Where the\nneeded token types actually live, by tier (share of each type's total pool count):\n\n| token type | passes prose gate | English but not prose-layout | fails even \"is this English?\" |\n|---|---|---|---|\n| `\" \"` (indentation) | 2.7% | 1.3% | **96.1%** |\n| `\">\"` | 6.8% | 2.2% | **91.0%** |\n| `\"</\"` | 10.9% | 4.9% | 84.1% |\n| `\" ,\"` | 18.2% | 4.9% | 76.9% |\n\nSo the mass cannot be recovered by *loosening* a quality filter along some\nlinguistic axis — we checked a middle tier that keeps the language tests\n(stopword rate, word length, non-ASCII) and drops the layout tests, and it holds\nonly 1.3% of the indentation mass. A hard budget allowance for documents that are\nnot prose at all is the only route to it, which is why the method needs one.\n\n**M3 — coverage saturates, so the non-prose allowance must be U-shaped.** The\ngreedy weight `w = p/q` falls as `q` catches up with `p`, so once markup coverage\nis bought, further non-prose documents are pure dilution. Predicted and observed\nminimum at ≈25% of the budget:\n\n| non-prose allowance | 0% | 12% | 20% | **25%** | 30% | 40% | uncapped |\n|---|---|---|---|---|---|---|---|\n| dev PPL | 296.1 | 288.6 | 278.7 | **272.6** | 279.5 | 280.3 | 281.0 |\n\n`curate.py` does not hard-code that allowance. It sweeps it and picks by unigram\nCE, which is free to evaluate — no training required, which is the point of M1:\n\n| allowance | 0% | 10% | 16% | 20% | 24% | **28%** | 32% | 40% | uncapped |\n|---|---|---|---|---|---|---|---|---|---|\n| budget CE ↓ | 7.9062 | 7.7668 | 7.7538 | 7.7440 | 7.7389 | 7.7368 | 7.7359 | 7.7369 | 7.7369 |\n\nThe curve is steep below ~0.2 and then flat to <0.002 nats. Its exact argmin\n(32%) is inside the noise, so the script takes the *smallest* allowance within\n`CE_TOL = 0.001` nats of the best — 28%. The tie-break is deliberately toward\nless non-prose: CE is a first-order objective that cannot see the cost of\nmarkup, so where CE is indifferent, prefer prose. The submitted selection is\nthat config (41,285 ids / 36.1M tokens of priority-ordered candidates for a 12M\nbudget), dev PPL **279.3** — per register: wiki 599.8, web 264.8, news 280.5,\nQ&A 136.1.\n\n## Falsification\n\n- **F1 (quality-first).** If the driver were generic quality, tightening the\n  prose gate would help monotonically. It does not: the hard gate is *worse* than\n  the capped-allowance gate (296.1 vs 272.6), and a 4-register quality/domain\n  classifier with a hard gate is far worse (365.0) despite 97.5% held-out\n  register accuracy. Selecting the documents a quality filter *rejects* is what\n  bought the largest single win (Q&A −43%).\n- **F2 (the markup documents are doing the work).** Hold the token count fixed\n  and replace every gate-failing document with the next-best prose document\n  (`--junk_frac 0`). The Q&A block must revert toward ~230 while the other three\n  stay within a few percent. If Q&A stayed near 140, coverage would not be the\n  mechanism. Observed: Q&A 132.0 → 229.9 (+74%), others within 8%. Confirmed.\n- **F3 (a pool-imposed floor).** The pool holds only 0.018% `\" ,\"` tokens, so the\n  maximum attainable `q(\" ,\")` inside a 12M-token budget is ≈0.30% against a\n  4.63% target share — the encyclopedic deficit is *unbuyable* from this pool.\n  Prediction: no selection from this pool takes the encyclopedic block below\n  ~450 PPL. Any selection reaching <400 there falsifies the claim that its\n  residual loss is dominated by unattainable surface tokens. (Best seen: 490.9,\n  by explicitly over-weighting that block — which cost more elsewhere than it\n  gained, net 313.6.)\n- **F4 (mediation).** A selection with materially lower unigram CE but higher\n  perplexity (beyond ~±2%) falsifies M1.\n\nHonest limit on resolution: two selections that differ by ~6% of their documents\ndiffer by ~±7 PPL (e.g. changing only how the smoothing prior is estimated moved\n272.6 → 279.5). Everything inside the 272–281 band is one band, not a ranking;\nonly the gaps to 296 / 365 / 478 are resolved by this experiment.\n\n## Transfer\n\nRequirements are only (i) a sample of the target distribution and (ii) the\ntraining tokenizer — no labels, no reference LM, no proxy model. Cost is one\ntokenisation pass over the pool plus one sparse mat-vec per greedy round.\n\nWhere it should transfer:\n\n- Any fixed-budget pretraining or continued-pretraining selection where the\n  evaluation mixture can be sampled: domain adaptation, multilingual mixes,\n  code+prose mixes. The greedy self-balances across sub-domains without being\n  told the mixture weights, because `w = p/q` automatically stops rewarding\n  token types that are already covered.\n- Best where the target has surface/format idiosyncrasies — markup, detokenised\n  punctuation, indentation, transliteration. That is exactly where quality-first\n  pipelines fail, because the documents carrying those forms *look* like junk.\n\nWhere it should weaken:\n\n- As the budget approaches the pool size, coverage saturates and the method\n  becomes a no-op (at 100% selection it is the identity).\n- When pool and target are already distributionally matched, there is nothing to\n  buy; expect gains to shrink toward zero rather than reverse.\n- The objective is first-order (unigram). It cannot create structure the pool\n  lacks — hence the F3 floor — and left uncapped it is gameable by token-dense\n  degenerate documents. The single tuned quantity in the whole method is the\n  non-prose allowance that guards against this (~25% here).\n\n## Reproduction\n\n`submission/curate.py` produces `submission/selection.json` from the stated\ncriterion alone — no randomness, no seeds, no hand-picked ids, no workspace\nimports. One tokenisation pass over the pool (cached), then the greedy. Its\noutput packed by the official `pack_selection.py` is bit-identical to the array\nthese numbers were measured on.\n\nDeterminism note: the ranking mat-vec runs in float64. In float32 the cuSPARSE\nreduction order made ties resolve differently between runs — the selected\ndocument *set* was identical at every prefix (100% overlap at 5k / 20k / all),\nbut documents permuted within a greedy batch, which matters at the 12M truncation\nboundary. float64 removes that.\n\nValidity: 41,285 unique ids, all present in the pool, 36.1M tokens — 3.0× the\nbudget, so the packer fills 12,000,000 tokens exactly (from the first 16,620\ndocuments) with no risk of a short pack.\n", "structuredPatch": [{"oldStart": 6, "oldLines": 7, "newStart": 6, "newLines": 7, "lines": [" 30M-parameter GPT, seed 1337.", " ", " Result on our own held-out split of the disclosed target: random selection 477.8", "-→ **279.3** (−42%), with a fastText-style quality/domain classifier at 365.0 in", "+→ **278.3** (−42%), with a fastText-style quality/domain classifier at 365.0 in", " between. The selector never sees a quality label; it minimises one scalar.", " ", " ## Hypothesis"]}], "userModified": false, "replaceAll": false}

The file /workspace/claim.md has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/claim.md", "oldString": "budget), dev PPL **279.3** — per register: wiki 599.8, web 264.8, news 280.5,\nQ&A 136.1.", "newString": "budget), dev PPL **278.3** — per register: wiki 591.9, web 264.7, news 282.3,\nQ&A 135.3.", "originalFile": "# Curating a 12M-token pretraining set by token-distribution coverage\n\nPool: 182,016 raw web documents / 199.9M GPT-2 tokens. Budget: 12M tokens (6% of\nthe pool). Target: held-out perplexity on an equal-parts mixture of encyclopedic\nprose, general high-quality web prose, news, and technical Q&A. Frozen trainer,\n30M-parameter GPT, seed 1337.\n\nResult on our own held-out split of the disclosed target: random selection 477.8\n→ **278.3** (−42%), with a fastText-style quality/domain classifier at 365.0 in\nbetween. The selector never sees a quality label; it minimises one scalar.\n\n## Hypothesis\n\nAt a small fixed token budget with a *disclosed* target mixture, held-out\nperplexity is governed primarily by **how completely the selected data's BPE\ntoken distribution covers the target's**, and only secondarily by document-level\n\"quality\". Consequently, greedily choosing documents to minimise the unigram\ncross-entropy\n\n    CE(S) = - Σ_t p_target(t) · log q_S(t)\n\nshould beat both random selection and a strong quality/domain classifier — and,\ncrucially, it should beat a *pure* quality filter, because some target token mass\nlives only in documents that any quality filter throws away.\n\n## Mechanism (observables other than the final perplexity)\n\n**M1 — unigram CE is the mediating quantity.** It is measurable on the packed\n12M-token array before any training, and it should order the selections the same\nway perplexity does:\n\n| selection | unigram CE ↓ | dev PPL ↓ |\n|---|---|---|\n| random (do-nothing baseline) | 8.129 | 477.8 |\n| 4-register quality/domain classifier + hard prose gate | 8.039 | 365.0 |\n| greedy CE matching, hard prose gate | 7.907 | 296.1 |\n| greedy CE matching + capped non-prose allowance | 7.741 | **272.6** |\n\nAcross all 18 selections trained in this study, Pearson `r`(unigram CE of the\npacked array, log dev PPL) = **0.925**. So a quantity computable in seconds from\nthe packed tokens, with no gradient steps, explains ~86% of the variance in the\nlog perplexity of a 3000-iteration training run. That is the mechanism claim:\nthe selector is not \"finding good documents\", it is moving `q` toward `p`, and\nmoving `q` toward `p` is what moves the loss.\n\n**M2 — the avoidable loss is concentrated on identifiable token types, and it is\na *surface-form* deficit, not a topical one.** Decomposing the target by register\nand comparing each block's token mass `p(t)` with a random web sample's `q(t)`:\n\n- encyclopedic block: `\" ,\"` 4.63%, `\" .\"` 3.27%, `\" @\"` 0.96% of its tokens\n  (WikiText detokenisation escapes) — each ≤0.04% in a random pool sample;\n- Q&A block: `\" \"` 9.31% (HTML indentation), `\">\"` 4.07%, `\"</\"` 1.50%, `\"code\"`\n  1.13% — each ≤0.15% in a random pool sample.\n\nPrediction: an intervention that raises `q` for *one* block's scarce token types\nmoves *that block's* perplexity and leaves the others roughly alone. Observed —\nrelaxing the prose gate (which had been excluding markup-heavy documents) moved\nthe Q&A block 229.9 → **132.0** (−43%) while wiki/web/news moved by ≤ +12%.\n\n**M2b — that scarce mass is unreachable by any quality-first pipeline.** Where the\nneeded token types actually live, by tier (share of each type's total pool count):\n\n| token type | passes prose gate | English but not prose-layout | fails even \"is this English?\" |\n|---|---|---|---|\n| `\" \"` (indentation) | 2.7% | 1.3% | **96.1%** |\n| `\">\"` | 6.8% | 2.2% | **91.0%** |\n| `\"</\"` | 10.9% | 4.9% | 84.1% |\n| `\" ,\"` | 18.2% | 4.9% | 76.9% |\n\nSo the mass cannot be recovered by *loosening* a quality filter along some\nlinguistic axis — we checked a middle tier that keeps the language tests\n(stopword rate, word length, non-ASCII) and drops the layout tests, and it holds\nonly 1.3% of the indentation mass. A hard budget allowance for documents that are\nnot prose at all is the only route to it, which is why the method needs one.\n\n**M3 — coverage saturates, so the non-prose allowance must be U-shaped.** The\ngreedy weight `w = p/q` falls as `q` catches up with `p`, so once markup coverage\nis bought, further non-prose documents are pure dilution. Predicted and observed\nminimum at ≈25% of the budget:\n\n| non-prose allowance | 0% | 12% | 20% | **25%** | 30% | 40% | uncapped |\n|---|---|---|---|---|---|---|---|\n| dev PPL | 296.1 | 288.6 | 278.7 | **272.6** | 279.5 | 280.3 | 281.0 |\n\n`curate.py` does not hard-code that allowance. It sweeps it and picks by unigram\nCE, which is free to evaluate — no training required, which is the point of M1:\n\n| allowance | 0% | 10% | 16% | 20% | 24% | **28%** | 32% | 40% | uncapped |\n|---|---|---|---|---|---|---|---|---|---|\n| budget CE ↓ | 7.9062 | 7.7668 | 7.7538 | 7.7440 | 7.7389 | 7.7368 | 7.7359 | 7.7369 | 7.7369 |\n\nThe curve is steep below ~0.2 and then flat to <0.002 nats. Its exact argmin\n(32%) is inside the noise, so the script takes the *smallest* allowance within\n`CE_TOL = 0.001` nats of the best — 28%. The tie-break is deliberately toward\nless non-prose: CE is a first-order objective that cannot see the cost of\nmarkup, so where CE is indifferent, prefer prose. The submitted selection is\nthat config (41,285 ids / 36.1M tokens of priority-ordered candidates for a 12M\nbudget), dev PPL **279.3** — per register: wiki 599.8, web 264.8, news 280.5,\nQ&A 136.1.\n\n## Falsification\n\n- **F1 (quality-first).** If the driver were generic quality, tightening the\n  prose gate would help monotonically. It does not: the hard gate is *worse* than\n  the capped-allowance gate (296.1 vs 272.6), and a 4-register quality/domain\n  classifier with a hard gate is far worse (365.0) despite 97.5% held-out\n  register accuracy. Selecting the documents a quality filter *rejects* is what\n  bought the largest single win (Q&A −43%).\n- **F2 (the markup documents are doing the work).** Hold the token count fixed\n  and replace every gate-failing document with the next-best prose document\n  (`--junk_frac 0`). The Q&A block must revert toward ~230 while the other three\n  stay within a few percent. If Q&A stayed near 140, coverage would not be the\n  mechanism. Observed: Q&A 132.0 → 229.9 (+74%), others within 8%. Confirmed.\n- **F3 (a pool-imposed floor).** The pool holds only 0.018% `\" ,\"` tokens, so the\n  maximum attainable `q(\" ,\")` inside a 12M-token budget is ≈0.30% against a\n  4.63% target share — the encyclopedic deficit is *unbuyable* from this pool.\n  Prediction: no selection from this pool takes the encyclopedic block below\n  ~450 PPL. Any selection reaching <400 there falsifies the claim that its\n  residual loss is dominated by unattainable surface tokens. (Best seen: 490.9,\n  by explicitly over-weighting that block — which cost more elsewhere than it\n  gained, net 313.6.)\n- **F4 (mediation).** A selection with materially lower unigram CE but higher\n  perplexity (beyond ~±2%) falsifies M1.\n\nHonest limit on resolution: two selections that differ by ~6% of their documents\ndiffer by ~±7 PPL (e.g. changing only how the smoothing prior is estimated moved\n272.6 → 279.5). Everything inside the 272–281 band is one band, not a ranking;\nonly the gaps to 296 / 365 / 478 are resolved by this experiment.\n\n## Transfer\n\nRequirements are only (i) a sample of the target distribution and (ii) the\ntraining tokenizer — no labels, no reference LM, no proxy model. Cost is one\ntokenisation pass over the pool plus one sparse mat-vec per greedy round.\n\nWhere it should transfer:\n\n- Any fixed-budget pretraining or continued-pretraining selection where the\n  evaluation mixture can be sampled: domain adaptation, multilingual mixes,\n  code+prose mixes. The greedy self-balances across sub-domains without being\n  told the mixture weights, because `w = p/q` automatically stops rewarding\n  token types that are already covered.\n- Best where the target has surface/format idiosyncrasies — markup, detokenised\n  punctuation, indentation, transliteration. That is exactly where quality-first\n  pipelines fail, because the documents carrying those forms *look* like junk.\n\nWhere it should weaken:\n\n- As the budget approaches the pool size, coverage saturates and the method\n  becomes a no-op (at 100% selection it is the identity).\n- When pool and target are already distributionally matched, there is nothing to\n  buy; expect gains to shrink toward zero rather than reverse.\n- The objective is first-order (unigram). It cannot create structure the pool\n  lacks — hence the F3 floor — and left uncapped it is gameable by token-dense\n  degenerate documents. The single tuned quantity in the whole method is the\n  non-prose allowance that guards against this (~25% here).\n\n## Reproduction\n\n`submission/curate.py` produces `submission/selection.json` from the stated\ncriterion alone — no randomness, no seeds, no hand-picked ids, no workspace\nimports. One tokenisation pass over the pool (cached), then the greedy. Its\noutput packed by the official `pack_selection.py` is bit-identical to the array\nthese numbers were measured on.\n\nDeterminism note: the ranking mat-vec runs in float64. In float32 the cuSPARSE\nreduction order made ties resolve differently between runs — the selected\ndocument *set* was identical at every prefix (100% overlap at 5k / 20k / all),\nbut documents permuted within a greedy batch, which matters at the 12M truncation\nboundary. float64 removes that.\n\nValidity: 41,285 unique ids, all present in the pool, 36.1M tokens — 3.0× the\nbudget, so the packer fills 12,000,000 tokens exactly (from the first 16,620\ndocuments) with no risk of a short pack.\n", "structuredPatch": [{"oldStart": 95, "oldLines": 8, "newStart": 95, "newLines": 8, "lines": [" less non-prose: CE is a first-order objective that cannot see the cost of", " markup, so where CE is indifferent, prefer prose. The submitted selection is", " that config (41,285 ids / 36.1M tokens of priority-ordered candidates for a 12M", "-budget), dev PPL **279.3** — per register: wiki 599.8, web 264.8, news 280.5,", "-Q&A 136.1.", "+budget), dev PPL **278.3** — per register: wiki 591.9, web 264.7, news 282.3,", "+Q&A 135.3.", " ", " ## Falsification", " "]}], "userModified": false, "replaceAll": false}

The file /workspace/claim.md has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/claim.md", "oldString": "Determinism note: the ranking mat-vec runs in float64. In float32 the cuSPARSE\nreduction order made ties resolve differently between runs — the selected\ndocument *set* was identical at every prefix (100% overlap at 5k / 20k / all),\nbut documents permuted within a greedy batch, which matters at the 12M truncation\nboundary. float64 removes that.", "newString": "Determinism note: verified by running the curator twice and comparing — the id\nlists are bit-identical. Getting there took two fixes, both worth recording\nbecause they are invisible in the final score. The ranking mat-vec is sparse and\nreduces in a nondeterministic order on GPU, so gains wobble in their last bits\nand near-ties permute *within* a greedy batch. The selected document set was\nnever affected (100% prefix overlap at 5k / 20k / all, across float32 runs), but\nthe emitted priority order was, and order is what decides which document the\npacker cuts at 12M. Fix: run the mat-vec in float64, and sort each batch on a\nfloat32-rounded gain with the document index as final tie-break, so the order is\na pure function of the (stable) selected set.", "originalFile": "# Curating a 12M-token pretraining set by token-distribution coverage\n\nPool: 182,016 raw web documents / 199.9M GPT-2 tokens. Budget: 12M tokens (6% of\nthe pool). Target: held-out perplexity on an equal-parts mixture of encyclopedic\nprose, general high-quality web prose, news, and technical Q&A. Frozen trainer,\n30M-parameter GPT, seed 1337.\n\nResult on our own held-out split of the disclosed target: random selection 477.8\n→ **278.3** (−42%), with a fastText-style quality/domain classifier at 365.0 in\nbetween. The selector never sees a quality label; it minimises one scalar.\n\n## Hypothesis\n\nAt a small fixed token budget with a *disclosed* target mixture, held-out\nperplexity is governed primarily by **how completely the selected data's BPE\ntoken distribution covers the target's**, and only secondarily by document-level\n\"quality\". Consequently, greedily choosing documents to minimise the unigram\ncross-entropy\n\n    CE(S) = - Σ_t p_target(t) · log q_S(t)\n\nshould beat both random selection and a strong quality/domain classifier — and,\ncrucially, it should beat a *pure* quality filter, because some target token mass\nlives only in documents that any quality filter throws away.\n\n## Mechanism (observables other than the final perplexity)\n\n**M1 — unigram CE is the mediating quantity.** It is measurable on the packed\n12M-token array before any training, and it should order the selections the same\nway perplexity does:\n\n| selection | unigram CE ↓ | dev PPL ↓ |\n|---|---|---|\n| random (do-nothing baseline) | 8.129 | 477.8 |\n| 4-register quality/domain classifier + hard prose gate | 8.039 | 365.0 |\n| greedy CE matching, hard prose gate | 7.907 | 296.1 |\n| greedy CE matching + capped non-prose allowance | 7.741 | **272.6** |\n\nAcross all 18 selections trained in this study, Pearson `r`(unigram CE of the\npacked array, log dev PPL) = **0.925**. So a quantity computable in seconds from\nthe packed tokens, with no gradient steps, explains ~86% of the variance in the\nlog perplexity of a 3000-iteration training run. That is the mechanism claim:\nthe selector is not \"finding good documents\", it is moving `q` toward `p`, and\nmoving `q` toward `p` is what moves the loss.\n\n**M2 — the avoidable loss is concentrated on identifiable token types, and it is\na *surface-form* deficit, not a topical one.** Decomposing the target by register\nand comparing each block's token mass `p(t)` with a random web sample's `q(t)`:\n\n- encyclopedic block: `\" ,\"` 4.63%, `\" .\"` 3.27%, `\" @\"` 0.96% of its tokens\n  (WikiText detokenisation escapes) — each ≤0.04% in a random pool sample;\n- Q&A block: `\" \"` 9.31% (HTML indentation), `\">\"` 4.07%, `\"</\"` 1.50%, `\"code\"`\n  1.13% — each ≤0.15% in a random pool sample.\n\nPrediction: an intervention that raises `q` for *one* block's scarce token types\nmoves *that block's* perplexity and leaves the others roughly alone. Observed —\nrelaxing the prose gate (which had been excluding markup-heavy documents) moved\nthe Q&A block 229.9 → **132.0** (−43%) while wiki/web/news moved by ≤ +12%.\n\n**M2b — that scarce mass is unreachable by any quality-first pipeline.** Where the\nneeded token types actually live, by tier (share of each type's total pool count):\n\n| token type | passes prose gate | English but not prose-layout | fails even \"is this English?\" |\n|---|---|---|---|\n| `\" \"` (indentation) | 2.7% | 1.3% | **96.1%** |\n| `\">\"` | 6.8% | 2.2% | **91.0%** |\n| `\"</\"` | 10.9% | 4.9% | 84.1% |\n| `\" ,\"` | 18.2% | 4.9% | 76.9% |\n\nSo the mass cannot be recovered by *loosening* a quality filter along some\nlinguistic axis — we checked a middle tier that keeps the language tests\n(stopword rate, word length, non-ASCII) and drops the layout tests, and it holds\nonly 1.3% of the indentation mass. A hard budget allowance for documents that are\nnot prose at all is the only route to it, which is why the method needs one.\n\n**M3 — coverage saturates, so the non-prose allowance must be U-shaped.** The\ngreedy weight `w = p/q` falls as `q` catches up with `p`, so once markup coverage\nis bought, further non-prose documents are pure dilution. Predicted and observed\nminimum at ≈25% of the budget:\n\n| non-prose allowance | 0% | 12% | 20% | **25%** | 30% | 40% | uncapped |\n|---|---|---|---|---|---|---|---|\n| dev PPL | 296.1 | 288.6 | 278.7 | **272.6** | 279.5 | 280.3 | 281.0 |\n\n`curate.py` does not hard-code that allowance. It sweeps it and picks by unigram\nCE, which is free to evaluate — no training required, which is the point of M1:\n\n| allowance | 0% | 10% | 16% | 20% | 24% | **28%** | 32% | 40% | uncapped |\n|---|---|---|---|---|---|---|---|---|---|\n| budget CE ↓ | 7.9062 | 7.7668 | 7.7538 | 7.7440 | 7.7389 | 7.7368 | 7.7359 | 7.7369 | 7.7369 |\n\nThe curve is steep below ~0.2 and then flat to <0.002 nats. Its exact argmin\n(32%) is inside the noise, so the script takes the *smallest* allowance within\n`CE_TOL = 0.001` nats of the best — 28%. The tie-break is deliberately toward\nless non-prose: CE is a first-order objective that cannot see the cost of\nmarkup, so where CE is indifferent, prefer prose. The submitted selection is\nthat config (41,285 ids / 36.1M tokens of priority-ordered candidates for a 12M\nbudget), dev PPL **278.3** — per register: wiki 591.9, web 264.7, news 282.3,\nQ&A 135.3.\n\n## Falsification\n\n- **F1 (quality-first).** If the driver were generic quality, tightening the\n  prose gate would help monotonically. It does not: the hard gate is *worse* than\n  the capped-allowance gate (296.1 vs 272.6), and a 4-register quality/domain\n  classifier with a hard gate is far worse (365.0) despite 97.5% held-out\n  register accuracy. Selecting the documents a quality filter *rejects* is what\n  bought the largest single win (Q&A −43%).\n- **F2 (the markup documents are doing the work).** Hold the token count fixed\n  and replace every gate-failing document with the next-best prose document\n  (`--junk_frac 0`). The Q&A block must revert toward ~230 while the other three\n  stay within a few percent. If Q&A stayed near 140, coverage would not be the\n  mechanism. Observed: Q&A 132.0 → 229.9 (+74%), others within 8%. Confirmed.\n- **F3 (a pool-imposed floor).** The pool holds only 0.018% `\" ,\"` tokens, so the\n  maximum attainable `q(\" ,\")` inside a 12M-token budget is ≈0.30% against a\n  4.63% target share — the encyclopedic deficit is *unbuyable* from this pool.\n  Prediction: no selection from this pool takes the encyclopedic block below\n  ~450 PPL. Any selection reaching <400 there falsifies the claim that its\n  residual loss is dominated by unattainable surface tokens. (Best seen: 490.9,\n  by explicitly over-weighting that block — which cost more elsewhere than it\n  gained, net 313.6.)\n- **F4 (mediation).** A selection with materially lower unigram CE but higher\n  perplexity (beyond ~±2%) falsifies M1.\n\nHonest limit on resolution: two selections that differ by ~6% of their documents\ndiffer by ~±7 PPL (e.g. changing only how the smoothing prior is estimated moved\n272.6 → 279.5). Everything inside the 272–281 band is one band, not a ranking;\nonly the gaps to 296 / 365 / 478 are resolved by this experiment.\n\n## Transfer\n\nRequirements are only (i) a sample of the target distribution and (ii) the\ntraining tokenizer — no labels, no reference LM, no proxy model. Cost is one\ntokenisation pass over the pool plus one sparse mat-vec per greedy round.\n\nWhere it should transfer:\n\n- Any fixed-budget pretraining or continued-pretraining selection where the\n  evaluation mixture can be sampled: domain adaptation, multilingual mixes,\n  code+prose mixes. The greedy self-balances across sub-domains without being\n  told the mixture weights, because `w = p/q` automatically stops rewarding\n  token types that are already covered.\n- Best where the target has surface/format idiosyncrasies — markup, detokenised\n  punctuation, indentation, transliteration. That is exactly where quality-first\n  pipelines fail, because the documents carrying those forms *look* like junk.\n\nWhere it should weaken:\n\n- As the budget approaches the pool size, coverage saturates and the method\n  becomes a no-op (at 100% selection it is the identity).\n- When pool and target are already distributionally matched, there is nothing to\n  buy; expect gains to shrink toward zero rather than reverse.\n- The objective is first-order (unigram). It cannot create structure the pool\n  lacks — hence the F3 floor — and left uncapped it is gameable by token-dense\n  degenerate documents. The single tuned quantity in the whole method is the\n  non-prose allowance that guards against this (~25% here).\n\n## Reproduction\n\n`submission/curate.py` produces `submission/selection.json` from the stated\ncriterion alone — no randomness, no seeds, no hand-picked ids, no workspace\nimports. One tokenisation pass over the pool (cached), then the greedy. Its\noutput packed by the official `pack_selection.py` is bit-identical to the array\nthese numbers were measured on.\n\nDeterminism note: the ranking mat-vec runs in float64. In float32 the cuSPARSE\nreduction order made ties resolve differently between runs — the selected\ndocument *set* was identical at every prefix (100% overlap at 5k / 20k / all),\nbut documents permuted within a greedy batch, which matters at the 12M truncation\nboundary. float64 removes that.\n\nValidity: 41,285 unique ids, all present in the pool, 36.1M tokens — 3.0× the\nbudget, so the packer fills 12,000,000 tokens exactly (from the first 16,620\ndocuments) with no risk of a short pack.\n", "structuredPatch": [{"oldStart": 163, "oldLines": 11, "newStart": 163, "newLines": 16, "lines": [" output packed by the official `pack_selection.py` is bit-identical to the array", " these numbers were measured on.", " ", "-Determinism note: the ranking mat-vec runs in float64. In float32 the cuSPARSE", "-reduction order made ties resolve differently between runs — the selected", "-document *set* was identical at every prefix (100% overlap at 5k / 20k / all),", "-but documents permuted within a greedy batch, which matters at the 12M truncation", "-boundary. float64 removes that.", "+Determinism note: verified by running the curator twice and comparing — the id", "+lists are bit-identical. Getting there took two fixes, both worth recording", "+because they are invisible in the final score. The ranking mat-vec is sparse and", "+reduces in a nondeterministic order on GPU, so gains wobble in their last bits", "+and near-ties permute *within* a greedy batch. The selected document set was", "+never affected (100% prefix overlap at 5k / 20k / all, across float32 runs), but", "+the emitted priority order was, and order is what decides which document the", "+packer cuts at 12M. Fix: run the mat-vec in float64, and sort each batch on a", "+float32-rounded gain with the document index as final tie-break, so the order is", "+a pure function of the (stable) selected set.", " ", " Validity: 41,285 unique ids, all present in the pool, 36.1M tokens — 3.0× the", " budget, so the packer fills 12,000,000 tokens exactly (from the first 16,620"]}], "userModified": false, "replaceAll": false}

list of ints: True | n: 41285 | unique: True | in-range: True
pack: (12000000,) uint16 | max id: 50256
--- claim.md ---
179 claim.md
1685

[stdout]
list of ints: True | n: 41285 | unique: True | in-range: True
pack: (12000000,) uint16 | max id: 50256
--- claim.md ---
179 claim.md
1685
[metadata] {"noOutputExpected": false}

78	is bought, further non-prose documents are pure dilution. Predicted and observed
79	minimum at ≈25% of the budget:
80	
81	| non-prose allowance | 0% | 12% | 20% | **25%** | 30% | 40% | uncapped |
82	|---|---|---|---|---|---|---|---|
83	| dev PPL | 296.1 | 288.6 | 278.7 | **272.6** | 279.5 | 280.3 | 281.0 |
84	
85	`curate.py` does not hard-code that allowance. It sweeps it and picks by unigram
86	CE, which is free to evaluate — no training required, which is the point of M1:
87	
88	| allowance | 0% | 10% | 16% | 20% | 24% | **28%** | 32% | 40% | uncapped |
89	|---|---|---|---|---|---|---|---|---|---|
90	| budget CE ↓ | 7.9062 | 7.7668 | 7.7538 | 7.7440 | 7.7389 | 7.7368 | 7.7359 | 7.7369 | 7.7369 |
91	
92	The curve is steep below ~0.2 and then flat to <0.002 nats. Its exact argmin
93	(32%) is inside the noise, so the script takes the *smallest* allowance within
94	`CE_TOL = 0.001` nats of the best — 28%. The tie-break is deliberately toward
95	less non-prose: CE is a first-order objective that cannot see the cost of
96	markup, so where CE is indifferent, prefer prose. The submitted selection is
97	that config (41,285 ids / 36.1M tokens of priority-ordered candidates for a 12M
98	budget), dev PPL **278.3** — per register: wiki 591.9, web 264.7, news 282.3,
99	Q&A 135.3.
100	
101	## Falsification
102	
103	- **F1 (quality-first).** If the driver were generic quality, tightening the
104	  prose gate would help monotonically. It does not: the hard gate is *worse* than
105	  the capped-allowance gate (296.1 vs 272.6), and a 4-register quality/domain
106	  classifier with a hard gate is far worse (365.0) despite 97.5% held-out
107	  register accuracy. Selecting the documents a quality filter *rejects* is what
108	  bought the largest single win (Q&A −43%).
109	- **F2 (the markup documents are doing the work).** Hold the token count fixed
110	  and replace every gate-failing document with the next-best prose document
111	  (`--junk_frac 0`). The Q&A block must revert toward ~230 while the other three
112	  stay within a few percent. If Q&A stayed near 140, coverage would not be the
113	  mechanism. Observed: Q&A 132.0 → 229.9 (+74%), others within 8%. Confirmed.
114	- **F3 (a pool-imposed floor).** The pool holds only 0.018% `" ,"` tokens, so the
115	  maximum attainable `q(" ,")` inside a 12M-token budget is ≈0.30% against a
116	  4.63% target share — the encyclopedic deficit is *unbuyable* from this pool.
117	  Prediction: no selection from this pool takes the encyclopedic block below

[metadata] {"type": "text", "file": {"filePath": "/workspace/claim.md", "content": "is bought, further non-prose documents are pure dilution. Predicted and observed\nminimum at ≈25% of the budget:\n\n| non-prose allowance | 0% | 12% | 20% | **25%** | 30% | 40% | uncapped |\n|---|---|---|---|---|---|---|---|\n| dev PPL | 296.1 | 288.6 | 278.7 | **272.6** | 279.5 | 280.3 | 281.0 |\n\n`curate.py` does not hard-code that allowance. It sweeps it and picks by unigram\nCE, which is free to evaluate — no training required, which is the point of M1:\n\n| allowance | 0% | 10% | 16% | 20% | 24% | **28%** | 32% | 40% | uncapped |\n|---|---|---|---|---|---|---|---|---|---|\n| budget CE ↓ | 7.9062 | 7.7668 | 7.7538 | 7.7440 | 7.7389 | 7.7368 | 7.7359 | 7.7369 | 7.7369 |\n\nThe curve is steep below ~0.2 and then flat to <0.002 nats. Its exact argmin\n(32%) is inside the noise, so the script takes the *smallest* allowance within\n`CE_TOL = 0.001` nats of the best — 28%. The tie-break is deliberately toward\nless non-prose: CE is a first-order objective that cannot see the cost of\nmarkup, so where CE is indifferent, prefer prose. The submitted selection is\nthat config (41,285 ids / 36.1M tokens of priority-ordered candidates for a 12M\nbudget), dev PPL **278.3** — per register: wiki 591.9, web 264.7, news 282.3,\nQ&A 135.3.\n\n## Falsification\n\n- **F1 (quality-first).** If the driver were generic quality, tightening the\n  prose gate would help monotonically. It does not: the hard gate is *worse* than\n  the capped-allowance gate (296.1 vs 272.6), and a 4-register quality/domain\n  classifier with a hard gate is far worse (365.0) despite 97.5% held-out\n  register accuracy. Selecting the documents a quality filter *rejects* is what\n  bought the largest single win (Q&A −43%).\n- **F2 (the markup documents are doing the work).** Hold the token count fixed\n  and replace every gate-failing document with the next-best prose document\n  (`--junk_frac 0`). The Q&A block must revert toward ~230 while the other three\n  stay within a few percent. If Q&A stayed near 140, coverage would not be the\n  mechanism. Observed: Q&A 132.0 → 229.9 (+74%), others within 8%. Confirmed.\n- **F3 (a pool-imposed floor).** The pool holds only 0.018% `\" ,\"` tokens, so the\n  maximum attainable `q(\" ,\")` inside a 12M-token budget is ≈0.30% against a\n  4.63% target share — the encyclopedic deficit is *unbuyable* from this pool.\n  Prediction: no selection from this pool takes the encyclopedic block below", "numLines": 40, "startLine": 78, "totalLines": 180}}