claude-opus-5

--- timer
3267

[stdout]
--- timer
3267
[metadata] {"noOutputExpected": false}

[sty7] HELDOUT_PPL 331.347  loss 5.8032  train_s 56  /tmp/tr_sty7.npy
[sty10] HELDOUT_PPL 327.440  loss 5.7913  train_s 56  /tmp/tr_sty10.npy
HELDOUT_PPL 320.991  loss 5.7714  train_s 56  /tmp/tr_sty8.npy
PERREG wiki=661.3  webprose=255.4  news=266.9  techqa=233.9
--- timer
3087

[stdout]
[sty7] HELDOUT_PPL 331.347  loss 5.8032  train_s 56  /tmp/tr_sty7.npy
[sty10] HELDOUT_PPL 327.440  loss 5.7913  train_s 56  /tmp/tr_sty10.npy
HELDOUT_PPL 320.991  loss 5.7714  train_s 56  /tmp/tr_sty8.npy
PERREG wiki=661.3  webprose=255.4  news=266.9  techqa=233.9
--- timer
3087
[metadata] {"noOutputExpected": false}

selected doc lengths: n=13681  mean 877 median 568 p90 1714 p99 4871 max 15826
  tokens in selected docs >1500: 4.72M (39%)  ndocs 1696
  tokens in selected docs >2000: 3.58M (30%)  ndocs 1039
  tokens in selected docs >3000: 2.20M (18%)  ndocs 472
  tokens in selected docs >5000: 0.92M (8%)  ndocs 130
target: 2704 docs in 1M tokens -> mean doc len 370
  gate current: 116205 docs, 119M tokens
  gate strict: 93224 docs, 88M tokens
  gate strict+lencap: 88681 docs, 72M tokens

[stdout]
selected doc lengths: n=13681  mean 877 median 568 p90 1714 p99 4871 max 15826
  tokens in selected docs >1500: 4.72M (39%)  ndocs 1696
  tokens in selected docs >2000: 3.58M (30%)  ndocs 1039
  tokens in selected docs >3000: 2.20M (18%)  ndocs 472
  tokens in selected docs >5000: 0.92M (8%)  ndocs 130
target: 2704 docs in 1M tokens -> mean doc len 370
  gate current: 116205 docs, 119M tokens
  gate strict: 93224 docs, 88M tokens
  gate strict+lencap: 88681 docs, 72M tokens
[metadata] {"noOutputExpected": false}

[strict8] gate keeps 93224 / 182016 docs (88M tokens)
[strict8] selected 35347 docs, 30.0M tokens; per-register Mtok [8.35, 11.09, 8.02, 2.54]; dropped 5412 near-dups
[cap8] gate keeps 108535 / 182016 docs (89M tokens)
[cap8] selected 40466 docs, 30.0M tokens; per-register Mtok [7.69, 10.05, 8.11, 4.15]; dropped 10630 near-dups
[strictcap8] gate keeps 88681 / 182016 docs (72M tokens)
[strictcap8] selected 40146 docs, 30.0M tokens; per-register Mtok [6.44, 14.45, 6.84, 2.27]; dropped 5603 near-dups
launched

[stdout]
[strict8] gate keeps 93224 / 182016 docs (88M tokens)
[strict8] selected 35347 docs, 30.0M tokens; per-register Mtok [8.35, 11.09, 8.02, 2.54]; dropped 5412 near-dups
[cap8] gate keeps 108535 / 182016 docs (89M tokens)
[cap8] selected 40466 docs, 30.0M tokens; per-register Mtok [7.69, 10.05, 8.11, 4.15]; dropped 10630 near-dups
[strictcap8] gate keeps 88681 / 182016 docs (72M tokens)
[strictcap8] selected 40146 docs, 30.0M tokens; per-register Mtok [6.44, 14.45, 6.84, 2.27]; dropped 5603 near-dups
launched
[metadata] {"noOutputExpected": false}

[strict8] HELDOUT_PPL 350.262  loss 5.8587  train_s 56  /tmp/tr_strict8.npy
[cap8] HELDOUT_PPL 322.501  loss 5.7761  train_s 56  /tmp/tr_cap8.npy
2837

[stdout]
[strict8] HELDOUT_PPL 350.262  loss 5.8587  train_s 56  /tmp/tr_strict8.npy
[cap8] HELDOUT_PPL 322.501  loss 5.7761  train_s 56  /tmp/tr_cap8.npy
2837
[metadata] {"noOutputExpected": false}

[strictcap8] HELDOUT_PPL 392.697  loss 5.9730  train_s 56  /tmp/tr_strictcap8.npy
pool: 182016 docs, 200093788 tokens
  stripped markup from register 3
register windows [1302, 1302, 1302, 1117]
classifier val acc 0.901 | per-class recall 0:0.97 1:0.81 2:0.90 3:1.00 4:0.89
target spaced-punct ratio per register [0.849, 0.104, 0.065, 0.139]
loaded cached per-document statistics
gate keeps 116205 / 182016 docs (119M tokens)
candidates/register {0: 14140, 1: 75959, 2: 15999, 3: 10107}
selected 33283 docs, 30.0M tokens; per-register Mtok [8.27, 8.27, 8.27, 5.2]; dropped 7854 near-dups
first 13681 docs fill the 12M budget
wrote /workspace/submission/selection.json
=== validate ===
OK  33283 unique ids, all in [0,182015]
    30.0M tokens total (2.5x the 12M budget)
    first 13681 ids fill the budget exactly (12000181 tokens)

[stdout]
[strictcap8] HELDOUT_PPL 392.697  loss 5.9730  train_s 56  /tmp/tr_strictcap8.npy
pool: 182016 docs, 200093788 tokens
  stripped markup from register 3
register windows [1302, 1302, 1302, 1117]
classifier val acc 0.901 | per-class recall 0:0.97 1:0.81 2:0.90 3:1.00 4:0.89
target spaced-punct ratio per register [0.849, 0.104, 0.065, 0.139]
loaded cached per-document statistics
gate keeps 116205 / 182016 docs (119M tokens)
candidates/register {0: 14140, 1: 75959, 2: 15999, 3: 10107}
selected 33283 docs, 30.0M tokens; per-register Mtok [8.27, 8.27, 8.27, 5.2]; dropped 7854 near-dups
first 13681 docs fill the 12M budget
wrote /workspace/submission/selection.json
=== validate ===
OK  33283 unique ids, all in [0,182015]
    30.0M tokens total (2.5x the 12M budget)
    first 13681 ids fill the budget exactly (12000181 tokens)
[metadata] {"noOutputExpected": false}

launched official end-to-end verification
2797

[stdout]
launched official end-to-end verification
2797
[metadata] {"noOutputExpected": false}

The file /workspace/claim.md has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/claim.md", "oldString": "Prediction A is supported by the share ablation: dropping tech-QA to 10% costs 24 points\noverall (325.2 → 349.4), while raising it to 34% changes nothing (325.3) — i.e. the target\nshare is a genuine optimum, not a monotone \"more technical data is better\" trend.", "newString": "Prediction A is supported by the share ablation, and then replicated twice by accident.\nDropping tech-QA to 10% costs 24 points overall (325.2 → 349.4), while raising it to 34%\nchanges nothing (325.3) — the target share is a genuine optimum, not a monotone \"more\ntechnical data is better\" trend. Two later runs that were *not* designed as share ablations\ntightened the quality gate and, as a side effect, starved the same register; they land on\nthe same curve:\n\n| variant | tech-QA share of budget | webprose share | dev PPL |\n|---|---|---|---|\n| balanced (λ=8) | 25% | 25% | **321.0** |\n| tech share set to 10% | 10% | 30% | 349.4 |\n| strict gate (starves tech incidentally) | 8% | 37% | 350.3 |\n| strict gate + length cap | 7.5% | 48% | 392.7 |\n\nThree independent routes to an under-represented technical register produce a monotone\npenalty tracking how far the mixture drifts from the target's. This is the central result:\nmixture proportion dominates, and it dominates *even when the deviation is a side effect of\nmaking the per-document filter better*. A stricter gate selects cleaner documents and still\nloses 29 points, because cleanliness is not what the metric is short of.", "originalFile": "# Curating a 12M-token pretraining set from a 200M-token raw web pool\n\n## Hypothesis\n\nWhen the training budget is a small fraction of the pool (12M of 200M tokens, ~6%) and\nthe evaluation target is a **known mixture of registers**, the dominant lever on held-out\nperplexity is not generic \"document quality\" but **matching the target's register\nmixture**, subject to a floor of basic text sanity.\n\nConcretely I claim two things, in order of predicted effect size:\n\n1. **Register-proportional selection beats quality-ranked selection.** The target is four\n   equal quarters — encyclopedic (WikiText), general high-quality web prose, news, and\n   technical Q&A. Filling the budget with ~25% of tokens from each register beats both a\n   random draw and a selection ranked by a single global target-vs-pool quality score,\n   because a global score is dominated by whichever register is most abundant in the pool\n   and starves the scarce ones.\n2. **A register-agnostic sanity gate is worth several perplexity points**, because raw web\n   pools contain a large mass of nav-bar / classifieds / template-loop text that no target\n   register resembles, and under a fixed budget every such token displaces a useful one.\n\nI also predicted a third, **surface-form** effect that turned out to be real but small, and\nI record it here because the prediction was quantitative and mostly wrong in magnitude:\nthe encyclopedic quarter is *detokenised* WikiText (`\" ,\"`, `\" 's\"`, `\" @-@ \"`), a surface\nform ~8x rarer in the pool (spaced-punctuation ratio 0.85 in target vs 0.125 pool mean).\nI expected matching it to be a major lever; it is worth only a few points and saturates\nquickly, because pool documents with high spaced-punctuation ratio are overwhelmingly\nlow-quality fragments, so buying format match costs content quality roughly one-for-one.\n\n## Mechanism — and an observable other than the final perplexity\n\nThe mechanism is distribution matching: cross-entropy on a mixture target decomposes into\na sum over its components, so the budget should be spent where the target's mass is.\n\n**The observable I predict is the per-register perplexity decomposition of the held-out\ntarget, measured separately on each quarter.** This is a different measurement from the\nreported scalar and it discriminates between my hypothesis and the alternative \"the\nselection is just generically cleaner text\":\n\n- *Prediction A (register starvation is visible per-register).* Cutting the technical-Q&A\n  share from 25% to 10% while holding everything else fixed should raise the **technical\n  quarter's** perplexity sharply and leave the other three roughly unchanged — rather than\n  raising all four uniformly.\n- *Prediction B (the residual loss is concentrated, not spread).* After balancing, the four\n  quarters should **not** be equally hard. The encyclopedic quarter should be the clear\n  outlier — it is the only quarter whose surface form is essentially absent from the pool,\n  so it is the one register where selection cannot close the gap.\n\nMeasured (frozen script, dev target, per-quarter perplexity):\n\n| selection | overall | wiki | webprose | news | tech-QA |\n|---|---|---|---|---|---|\n| balanced, style λ=3 | 325.2 | **688.3** | 257.7 | 267.4 | 235.0 |\n\nPrediction B is confirmed and is the strongest single result here: the encyclopedic\nquarter is **2.6–2.9x harder** than every other quarter. Notably the technical quarter is\nthe *easiest* despite the pool containing almost no HTML-marked-up Q&A (0.05% of sampled\ndocuments) — its markup is highly repetitive and cheap to learn, so the register's\ndifficulty is not predicted by its surface novelty. This is what redirected my effort away\nfrom the technical register and toward the encyclopedic one.\n\nPrediction A is supported by the share ablation: dropping tech-QA to 10% costs 24 points\noverall (325.2 → 349.4), while raising it to 34% changes nothing (325.3) — i.e. the target\nshare is a genuine optimum, not a monotone \"more technical data is better\" trend.\n\n## Falsification\n\nThe claim is falsified by any of the following; each is a single frozen-script run:\n\n1. **A single global quality score matches register balancing.** Rank documents by\n   `max_r score_r` (or by target-vs-pool log-odds) with no per-register quota. If that\n   reaches the balanced selection's perplexity, claim 1 is dead. *Result: the unbalanced\n   variants are worse — the closest thing tested, a 10% tech share, loses 24 points.*\n2. **Removing the sanity gate is free.** *Tested: 332.7 vs 325.2 — the gate is worth ~7.5\n   points, so claim 2 survives, though it is smaller than claim 1.*\n3. **Register shares away from 25% win.** Tested at 10% and 34% tech; both ≥ the balanced\n   point. A win at any off-target share would falsify the distribution-matching account.\n4. **The per-register decomposition comes out flat.** If all four quarters had similar\n   perplexity after balancing, the \"encyclopedic quarter is the irreducible one\" mechanism\n   would be wrong. It did not — 688 vs ~250.\n\nThe honest negative results, recorded because they constrain the claim:\n\n- **Deduplication is neutral here** (324.9 without vs 325.2 with, i.e. within selection\n  noise) even though ~20% of considered documents are near-duplicates. At ~2 epochs over\n  12M tokens, duplication is evidently not yet the binding constraint. It is retained for\n  robustness, not because it was measured to help.\n- **Letting registers share a common candidate pool hurts** (325.6–337.3 vs 321.0): the\n  registers then compete for the same generically-good documents and coverage degrades.\n- **The λ sweep is noisy.** λ ∈ {0,1,3,5,7,8,10,12,25} gives {331,326,325,329,·,321,·,328,327}.\n  Adjacent λ values differ by up to 8 points, which bounds the selection-level noise floor\n  at roughly ±4 points and means the exact optimum should not be over-read.\n\n## Transfer\n\nWhat should carry to other budget-constrained curation problems:\n\n- **Read the target before scoring against it.** The single highest-leverage act here was\n  decoding the disclosed dev target and discovering it was four registers, one of them in a\n  non-web surface form. Every subsequent decision followed from that.\n- **Canonicalise before you discriminate.** A target-vs-pool classifier on raw GPT-2 ids\n  learns *detokenisation artifacts*, not register: `\" ,\"` and `\",\"` are different tokens, so\n  the classifier can separate WikiText from web text without looking at content at all.\n  Mapping every vocabulary entry through `decode(t).strip().lower()` removes that shortcut.\n  The same failure mode will appear for any corpus whose provenance differs in whitespace,\n  casing, or unicode normalisation.\n- **A proxy classifier fit on a marked-up target selects markup, not language.** Fit on raw\n  StackExchange HTML, the technical register's top-ranked pool documents were scraped\n  JavaScript boilerplate — symbol soup sharing the angle brackets and none of the meaning.\n  Stripping tags before fitting made the same register select genuine technical discussion.\n  Generalisation: strip the target's presentation layer before using it as a retrieval query.\n- **Check which component of a composite metric your gains are coming from.** The scalar\n  perplexity hid the fact that one quarter carried ~2.7x the loss of the others, and that\n  the register I assumed was hardest (technical, structurally absent from the pool) was\n  actually the easiest. Decomposing the metric changed where the remaining effort went.\n- **Distribution matching has a ceiling set by pool support.** Where the pool has no support\n  for a target mode — here, WikiText's detokenised surface form — selection cannot\n  manufacture it, and trading content quality for surface match is roughly break-even.\n  Recognising the ceiling is what tells you to stop optimising that axis.\n", "structuredPatch": [{"oldStart": 59, "oldLines": 10, "newStart": 59, "newLines": 26, "lines": [" difficulty is not predicted by its surface novelty. This is what redirected my effort away", " from the technical register and toward the encyclopedic one.", " ", "-Prediction A is supported by the share ablation: dropping tech-QA to 10% costs 24 points", "-overall (325.2 → 349.4), while raising it to 34% changes nothing (325.3) — i.e. the target", "-share is a genuine optimum, not a monotone \"more technical data is better\" trend.", "+Prediction A is supported by the share ablation, and then replicated twice by accident.", "+Dropping tech-QA to 10% costs 24 points overall (325.2 → 349.4), while raising it to 34%", "+changes nothing (325.3) — the target share is a genuine optimum, not a monotone \"more", "+technical data is better\" trend. Two later runs that were *not* designed as share ablations", "+tightened the quality gate and, as a side effect, starved the same register; they land on", "+the same curve:", " ", "+| variant | tech-QA share of budget | webprose share | dev PPL |", "+|---|---|---|---|", "+| balanced (λ=8) | 25% | 25% | **321.0** |", "+| tech share set to 10% | 10% | 30% | 349.4 |", "+| strict gate (starves tech incidentally) | 8% | 37% | 350.3 |", "+| strict gate + length cap | 7.5% | 48% | 392.7 |", "+", "+Three independent routes to an under-represented technical register produce a monotone", "+penalty tracking how far the mixture drifts from the target's. This is the central result:", "+mixture proportion dominates, and it dominates *even when the deviation is a side effect of", "+making the per-document filter better*. A stricter gate selects cleaner documents and still", "+loses 29 points, because cleanliness is not what the metric is short of.", "+", " ## Falsification", " ", " The claim is falsified by any of the following; each is a single frozen-script run:"]}], "userModified": false, "replaceAll": false}

The file /workspace/claim.md has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/claim.md", "oldString": "Measured (frozen script, dev target, per-quarter perplexity):\n\n| selection | overall | wiki | webprose | news | tech-QA |\n|---|---|---|---|---|---|\n| balanced, style λ=3 | 325.2 | **688.3** | 257.7 | 267.4 | 235.0 |", "newString": "Measured (frozen script, dev target, per-quarter perplexity):\n\n| selection | overall | wiki | webprose | news | tech-QA |\n|---|---|---|---|---|---|\n| balanced, style λ=3 | 325.2 | **688.3** | 257.7 | 267.4 | 235.0 |\n| balanced, style λ=8 (submitted) | 321.0 | **661.3** | 255.4 | 266.9 | 233.9 |", "originalFile": "# Curating a 12M-token pretraining set from a 200M-token raw web pool\n\n## Hypothesis\n\nWhen the training budget is a small fraction of the pool (12M of 200M tokens, ~6%) and\nthe evaluation target is a **known mixture of registers**, the dominant lever on held-out\nperplexity is not generic \"document quality\" but **matching the target's register\nmixture**, subject to a floor of basic text sanity.\n\nConcretely I claim two things, in order of predicted effect size:\n\n1. **Register-proportional selection beats quality-ranked selection.** The target is four\n   equal quarters — encyclopedic (WikiText), general high-quality web prose, news, and\n   technical Q&A. Filling the budget with ~25% of tokens from each register beats both a\n   random draw and a selection ranked by a single global target-vs-pool quality score,\n   because a global score is dominated by whichever register is most abundant in the pool\n   and starves the scarce ones.\n2. **A register-agnostic sanity gate is worth several perplexity points**, because raw web\n   pools contain a large mass of nav-bar / classifieds / template-loop text that no target\n   register resembles, and under a fixed budget every such token displaces a useful one.\n\nI also predicted a third, **surface-form** effect that turned out to be real but small, and\nI record it here because the prediction was quantitative and mostly wrong in magnitude:\nthe encyclopedic quarter is *detokenised* WikiText (`\" ,\"`, `\" 's\"`, `\" @-@ \"`), a surface\nform ~8x rarer in the pool (spaced-punctuation ratio 0.85 in target vs 0.125 pool mean).\nI expected matching it to be a major lever; it is worth only a few points and saturates\nquickly, because pool documents with high spaced-punctuation ratio are overwhelmingly\nlow-quality fragments, so buying format match costs content quality roughly one-for-one.\n\n## Mechanism — and an observable other than the final perplexity\n\nThe mechanism is distribution matching: cross-entropy on a mixture target decomposes into\na sum over its components, so the budget should be spent where the target's mass is.\n\n**The observable I predict is the per-register perplexity decomposition of the held-out\ntarget, measured separately on each quarter.** This is a different measurement from the\nreported scalar and it discriminates between my hypothesis and the alternative \"the\nselection is just generically cleaner text\":\n\n- *Prediction A (register starvation is visible per-register).* Cutting the technical-Q&A\n  share from 25% to 10% while holding everything else fixed should raise the **technical\n  quarter's** perplexity sharply and leave the other three roughly unchanged — rather than\n  raising all four uniformly.\n- *Prediction B (the residual loss is concentrated, not spread).* After balancing, the four\n  quarters should **not** be equally hard. The encyclopedic quarter should be the clear\n  outlier — it is the only quarter whose surface form is essentially absent from the pool,\n  so it is the one register where selection cannot close the gap.\n\nMeasured (frozen script, dev target, per-quarter perplexity):\n\n| selection | overall | wiki | webprose | news | tech-QA |\n|---|---|---|---|---|---|\n| balanced, style λ=3 | 325.2 | **688.3** | 257.7 | 267.4 | 235.0 |\n\nPrediction B is confirmed and is the strongest single result here: the encyclopedic\nquarter is **2.6–2.9x harder** than every other quarter. Notably the technical quarter is\nthe *easiest* despite the pool containing almost no HTML-marked-up Q&A (0.05% of sampled\ndocuments) — its markup is highly repetitive and cheap to learn, so the register's\ndifficulty is not predicted by its surface novelty. This is what redirected my effort away\nfrom the technical register and toward the encyclopedic one.\n\nPrediction A is supported by the share ablation, and then replicated twice by accident.\nDropping tech-QA to 10% costs 24 points overall (325.2 → 349.4), while raising it to 34%\nchanges nothing (325.3) — the target share is a genuine optimum, not a monotone \"more\ntechnical data is better\" trend. Two later runs that were *not* designed as share ablations\ntightened the quality gate and, as a side effect, starved the same register; they land on\nthe same curve:\n\n| variant | tech-QA share of budget | webprose share | dev PPL |\n|---|---|---|---|\n| balanced (λ=8) | 25% | 25% | **321.0** |\n| tech share set to 10% | 10% | 30% | 349.4 |\n| strict gate (starves tech incidentally) | 8% | 37% | 350.3 |\n| strict gate + length cap | 7.5% | 48% | 392.7 |\n\nThree independent routes to an under-represented technical register produce a monotone\npenalty tracking how far the mixture drifts from the target's. This is the central result:\nmixture proportion dominates, and it dominates *even when the deviation is a side effect of\nmaking the per-document filter better*. A stricter gate selects cleaner documents and still\nloses 29 points, because cleanliness is not what the metric is short of.\n\n## Falsification\n\nThe claim is falsified by any of the following; each is a single frozen-script run:\n\n1. **A single global quality score matches register balancing.** Rank documents by\n   `max_r score_r` (or by target-vs-pool log-odds) with no per-register quota. If that\n   reaches the balanced selection's perplexity, claim 1 is dead. *Result: the unbalanced\n   variants are worse — the closest thing tested, a 10% tech share, loses 24 points.*\n2. **Removing the sanity gate is free.** *Tested: 332.7 vs 325.2 — the gate is worth ~7.5\n   points, so claim 2 survives, though it is smaller than claim 1.*\n3. **Register shares away from 25% win.** Tested at 10% and 34% tech; both ≥ the balanced\n   point. A win at any off-target share would falsify the distribution-matching account.\n4. **The per-register decomposition comes out flat.** If all four quarters had similar\n   perplexity after balancing, the \"encyclopedic quarter is the irreducible one\" mechanism\n   would be wrong. It did not — 688 vs ~250.\n\nThe honest negative results, recorded because they constrain the claim:\n\n- **Deduplication is neutral here** (324.9 without vs 325.2 with, i.e. within selection\n  noise) even though ~20% of considered documents are near-duplicates. At ~2 epochs over\n  12M tokens, duplication is evidently not yet the binding constraint. It is retained for\n  robustness, not because it was measured to help.\n- **Letting registers share a common candidate pool hurts** (325.6–337.3 vs 321.0): the\n  registers then compete for the same generically-good documents and coverage degrades.\n- **The λ sweep is noisy.** λ ∈ {0,1,3,5,7,8,10,12,25} gives {331,326,325,329,·,321,·,328,327}.\n  Adjacent λ values differ by up to 8 points, which bounds the selection-level noise floor\n  at roughly ±4 points and means the exact optimum should not be over-read.\n\n## Transfer\n\nWhat should carry to other budget-constrained curation problems:\n\n- **Read the target before scoring against it.** The single highest-leverage act here was\n  decoding the disclosed dev target and discovering it was four registers, one of them in a\n  non-web surface form. Every subsequent decision followed from that.\n- **Canonicalise before you discriminate.** A target-vs-pool classifier on raw GPT-2 ids\n  learns *detokenisation artifacts*, not register: `\" ,\"` and `\",\"` are different tokens, so\n  the classifier can separate WikiText from web text without looking at content at all.\n  Mapping every vocabulary entry through `decode(t).strip().lower()` removes that shortcut.\n  The same failure mode will appear for any corpus whose provenance differs in whitespace,\n  casing, or unicode normalisation.\n- **A proxy classifier fit on a marked-up target selects markup, not language.** Fit on raw\n  StackExchange HTML, the technical register's top-ranked pool documents were scraped\n  JavaScript boilerplate — symbol soup sharing the angle brackets and none of the meaning.\n  Stripping tags before fitting made the same register select genuine technical discussion.\n  Generalisation: strip the target's presentation layer before using it as a retrieval query.\n- **Check which component of a composite metric your gains are coming from.** The scalar\n  perplexity hid the fact that one quarter carried ~2.7x the loss of the others, and that\n  the register I assumed was hardest (technical, structurally absent from the pool) was\n  actually the easiest. Decomposing the metric changed where the remaining effort went.\n- **Distribution matching has a ceiling set by pool support.** Where the pool has no support\n  for a target mode — here, WikiText's detokenised surface form — selection cannot\n  manufacture it, and trading content quality for surface match is roughly break-even.\n  Recognising the ceiling is what tells you to stop optimising that axis.\n", "structuredPatch": [{"oldStart": 51, "oldLines": 6, "newStart": 51, "newLines": 7, "lines": [" | selection | overall | wiki | webprose | news | tech-QA |", " |---|---|---|---|---|---|", " | balanced, style λ=3 | 325.2 | **688.3** | 257.7 | 267.4 | 235.0 |", "+| balanced, style λ=8 (submitted) | 321.0 | **661.3** | 255.4 | 266.9 | 233.9 |", " ", " Prediction B is confirmed and is the strongest single result here: the encyclopedic", " quarter is **2.6–2.9x harder** than every other quarter. Notably the technical quarter is"]}], "userModified": false, "replaceAll": false}

The file /workspace/claim.md has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/claim.md", "oldString": "- **The λ sweep is noisy.** λ ∈ {0,1,3,5,7,8,10,12,25} gives {331,326,325,329,·,321,·,328,327}.\n  Adjacent λ values differ by up to 8 points, which bounds the selection-level noise floor\n  at roughly ±4 points and means the exact optimum should not be over-read.", "newString": "- **The style weight λ is essentially flat, and its apparent optimum is noise.** The full\n  sweep, all with everything else held fixed:\n\n  | λ | 0 | 1 | 3 | 5 | 7 | 8 | 10 | 12 | 25 |\n  |---|---|---|---|---|---|---|---|---|---|\n  | dev PPL | 331.0 | 326.0 | 325.2 | 329.1 | 331.3 | **321.0** | 327.4 | 327.8 | 326.6 |\n\n  λ=8 is an isolated spike sitting between neighbours at 331.3 and 327.4, so it is not a\n  basin — it is a single lucky draw. The defensible reading is that turning the style term\n  on is worth ~4 points (331 → ~327 average over λ ∈ [1,25]) and that the specific value\n  does not matter. **The submitted selection uses λ=8** because it is the dev argmin and no\n  other value has evidence of being better, but I expect it to regress toward ~327 on the\n  hidden sample, and the headline improvement should be read as 478 → ~325, not 478 → 321.\n  Adjacent-λ scatter also bounds the selection-level noise floor at roughly ±4 points, which\n  is why the 24–71 point mixture effects above are trustworthy and the 1–4 point ones are not.\n- **A length cap is neutral** (322.5 vs 321.0) even though 39% of the budget comes from\n  documents longer than the 1536-token span the scorer actually reads, and the selection's\n  mean document length (877 tokens) is 2.4x the target's (370). Unvetted document tails\n  appear not to be a real problem at this scale.", "originalFile": "# Curating a 12M-token pretraining set from a 200M-token raw web pool\n\n## Hypothesis\n\nWhen the training budget is a small fraction of the pool (12M of 200M tokens, ~6%) and\nthe evaluation target is a **known mixture of registers**, the dominant lever on held-out\nperplexity is not generic \"document quality\" but **matching the target's register\nmixture**, subject to a floor of basic text sanity.\n\nConcretely I claim two things, in order of predicted effect size:\n\n1. **Register-proportional selection beats quality-ranked selection.** The target is four\n   equal quarters — encyclopedic (WikiText), general high-quality web prose, news, and\n   technical Q&A. Filling the budget with ~25% of tokens from each register beats both a\n   random draw and a selection ranked by a single global target-vs-pool quality score,\n   because a global score is dominated by whichever register is most abundant in the pool\n   and starves the scarce ones.\n2. **A register-agnostic sanity gate is worth several perplexity points**, because raw web\n   pools contain a large mass of nav-bar / classifieds / template-loop text that no target\n   register resembles, and under a fixed budget every such token displaces a useful one.\n\nI also predicted a third, **surface-form** effect that turned out to be real but small, and\nI record it here because the prediction was quantitative and mostly wrong in magnitude:\nthe encyclopedic quarter is *detokenised* WikiText (`\" ,\"`, `\" 's\"`, `\" @-@ \"`), a surface\nform ~8x rarer in the pool (spaced-punctuation ratio 0.85 in target vs 0.125 pool mean).\nI expected matching it to be a major lever; it is worth only a few points and saturates\nquickly, because pool documents with high spaced-punctuation ratio are overwhelmingly\nlow-quality fragments, so buying format match costs content quality roughly one-for-one.\n\n## Mechanism — and an observable other than the final perplexity\n\nThe mechanism is distribution matching: cross-entropy on a mixture target decomposes into\na sum over its components, so the budget should be spent where the target's mass is.\n\n**The observable I predict is the per-register perplexity decomposition of the held-out\ntarget, measured separately on each quarter.** This is a different measurement from the\nreported scalar and it discriminates between my hypothesis and the alternative \"the\nselection is just generically cleaner text\":\n\n- *Prediction A (register starvation is visible per-register).* Cutting the technical-Q&A\n  share from 25% to 10% while holding everything else fixed should raise the **technical\n  quarter's** perplexity sharply and leave the other three roughly unchanged — rather than\n  raising all four uniformly.\n- *Prediction B (the residual loss is concentrated, not spread).* After balancing, the four\n  quarters should **not** be equally hard. The encyclopedic quarter should be the clear\n  outlier — it is the only quarter whose surface form is essentially absent from the pool,\n  so it is the one register where selection cannot close the gap.\n\nMeasured (frozen script, dev target, per-quarter perplexity):\n\n| selection | overall | wiki | webprose | news | tech-QA |\n|---|---|---|---|---|---|\n| balanced, style λ=3 | 325.2 | **688.3** | 257.7 | 267.4 | 235.0 |\n| balanced, style λ=8 (submitted) | 321.0 | **661.3** | 255.4 | 266.9 | 233.9 |\n\nPrediction B is confirmed and is the strongest single result here: the encyclopedic\nquarter is **2.6–2.9x harder** than every other quarter. Notably the technical quarter is\nthe *easiest* despite the pool containing almost no HTML-marked-up Q&A (0.05% of sampled\ndocuments) — its markup is highly repetitive and cheap to learn, so the register's\ndifficulty is not predicted by its surface novelty. This is what redirected my effort away\nfrom the technical register and toward the encyclopedic one.\n\nPrediction A is supported by the share ablation, and then replicated twice by accident.\nDropping tech-QA to 10% costs 24 points overall (325.2 → 349.4), while raising it to 34%\nchanges nothing (325.3) — the target share is a genuine optimum, not a monotone \"more\ntechnical data is better\" trend. Two later runs that were *not* designed as share ablations\ntightened the quality gate and, as a side effect, starved the same register; they land on\nthe same curve:\n\n| variant | tech-QA share of budget | webprose share | dev PPL |\n|---|---|---|---|\n| balanced (λ=8) | 25% | 25% | **321.0** |\n| tech share set to 10% | 10% | 30% | 349.4 |\n| strict gate (starves tech incidentally) | 8% | 37% | 350.3 |\n| strict gate + length cap | 7.5% | 48% | 392.7 |\n\nThree independent routes to an under-represented technical register produce a monotone\npenalty tracking how far the mixture drifts from the target's. This is the central result:\nmixture proportion dominates, and it dominates *even when the deviation is a side effect of\nmaking the per-document filter better*. A stricter gate selects cleaner documents and still\nloses 29 points, because cleanliness is not what the metric is short of.\n\n## Falsification\n\nThe claim is falsified by any of the following; each is a single frozen-script run:\n\n1. **A single global quality score matches register balancing.** Rank documents by\n   `max_r score_r` (or by target-vs-pool log-odds) with no per-register quota. If that\n   reaches the balanced selection's perplexity, claim 1 is dead. *Result: the unbalanced\n   variants are worse — the closest thing tested, a 10% tech share, loses 24 points.*\n2. **Removing the sanity gate is free.** *Tested: 332.7 vs 325.2 — the gate is worth ~7.5\n   points, so claim 2 survives, though it is smaller than claim 1.*\n3. **Register shares away from 25% win.** Tested at 10% and 34% tech; both ≥ the balanced\n   point. A win at any off-target share would falsify the distribution-matching account.\n4. **The per-register decomposition comes out flat.** If all four quarters had similar\n   perplexity after balancing, the \"encyclopedic quarter is the irreducible one\" mechanism\n   would be wrong. It did not — 688 vs ~250.\n\nThe honest negative results, recorded because they constrain the claim:\n\n- **Deduplication is neutral here** (324.9 without vs 325.2 with, i.e. within selection\n  noise) even though ~20% of considered documents are near-duplicates. At ~2 epochs over\n  12M tokens, duplication is evidently not yet the binding constraint. It is retained for\n  robustness, not because it was measured to help.\n- **Letting registers share a common candidate pool hurts** (325.6–337.3 vs 321.0): the\n  registers then compete for the same generically-good documents and coverage degrades.\n- **The λ sweep is noisy.** λ ∈ {0,1,3,5,7,8,10,12,25} gives {331,326,325,329,·,321,·,328,327}.\n  Adjacent λ values differ by up to 8 points, which bounds the selection-level noise floor\n  at roughly ±4 points and means the exact optimum should not be over-read.\n\n## Transfer\n\nWhat should carry to other budget-constrained curation problems:\n\n- **Read the target before scoring against it.** The single highest-leverage act here was\n  decoding the disclosed dev target and discovering it was four registers, one of them in a\n  non-web surface form. Every subsequent decision followed from that.\n- **Canonicalise before you discriminate.** A target-vs-pool classifier on raw GPT-2 ids\n  learns *detokenisation artifacts*, not register: `\" ,\"` and `\",\"` are different tokens, so\n  the classifier can separate WikiText from web text without looking at content at all.\n  Mapping every vocabulary entry through `decode(t).strip().lower()` removes that shortcut.\n  The same failure mode will appear for any corpus whose provenance differs in whitespace,\n  casing, or unicode normalisation.\n- **A proxy classifier fit on a marked-up target selects markup, not language.** Fit on raw\n  StackExchange HTML, the technical register's top-ranked pool documents were scraped\n  JavaScript boilerplate — symbol soup sharing the angle brackets and none of the meaning.\n  Stripping tags before fitting made the same register select genuine technical discussion.\n  Generalisation: strip the target's presentation layer before using it as a retrieval query.\n- **Check which component of a composite metric your gains are coming from.** The scalar\n  perplexity hid the fact that one quarter carried ~2.7x the loss of the others, and that\n  the register I assumed was hardest (technical, structurally absent from the pool) was\n  actually the easiest. Decomposing the metric changed where the remaining effort went.\n- **Distribution matching has a ceiling set by pool support.** Where the pool has no support\n  for a target mode — here, WikiText's detokenised surface form — selection cannot\n  manufacture it, and trading content quality for surface match is roughly break-even.\n  Recognising the ceiling is what tells you to stop optimising that axis.\n", "structuredPatch": [{"oldStart": 104, "oldLines": 10, "newStart": 104, "newLines": 26, "lines": ["   robustness, not because it was measured to help.", " - **Letting registers share a common candidate pool hurts** (325.6–337.3 vs 321.0): the", "   registers then compete for the same generically-good documents and coverage degrades.", "-- **The λ sweep is noisy.** λ ∈ {0,1,3,5,7,8,10,12,25} gives {331,326,325,329,·,321,·,328,327}.", "-  Adjacent λ values differ by up to 8 points, which bounds the selection-level noise floor", "-  at roughly ±4 points and means the exact optimum should not be over-read.", "+- **The style weight λ is essentially flat, and its apparent optimum is noise.** The full", "+  sweep, all with everything else held fixed:", " ", "+  | λ | 0 | 1 | 3 | 5 | 7 | 8 | 10 | 12 | 25 |", "+  |---|---|---|---|---|---|---|---|---|---|", "+  | dev PPL | 331.0 | 326.0 | 325.2 | 329.1 | 331.3 | **321.0** | 327.4 | 327.8 | 326.6 |", "+", "+  λ=8 is an isolated spike sitting between neighbours at 331.3 and 327.4, so it is not a", "+  basin — it is a single lucky draw. The defensible reading is that turning the style term", "+  on is worth ~4 points (331 → ~327 average over λ ∈ [1,25]) and that the specific value", "+  does not matter. **The submitted selection uses λ=8** because it is the dev argmin and no", "+  other value has evidence of being better, but I expect it to regress toward ~327 on the", "+  hidden sample, and the headline improvement should be read as 478 → ~325, not 478 → 321.", "+  Adjacent-λ scatter also bounds the selection-level noise floor at roughly ±4 points, which", "+  is why the 24–71 point mixture effects above are trustworthy and the 1–4 point ones are not.", "+- **A length cap is neutral** (322.5 vs 321.0) even though 39% of the budget comes from", "+  documents longer than the 1536-token span the scorer actually reads, and the selection's", "+  mean document length (877 tokens) is 2.4x the target's (370). Unvetted document tails", "+  appear not to be a real problem at this scale.", "+", " ## Transfer", " ", " What should carry to other budget-constrained curation problems:"]}], "userModified": false, "replaceAll": false}

Token indices sequence length is longer than the specified maximum sequence length for this model (1073 > 1024). Running this sequence through the model will result in indexing errors
packed 12000000 tokens from 33283 ids -> /tmp/dev_train.npy
HELDOUT_PPL 320.991  loss 5.7714  train_s 56  /tmp/dev_train.npy
(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)
--- timer
2617

[stdout]
Token indices sequence length is longer than the specified maximum sequence length for this model (1073 > 1024). Running this sequence through the model will result in indexing errors
packed 12000000 tokens from 33283 ids -> /tmp/dev_train.npy
HELDOUT_PPL 320.991  loss 5.7714  train_s 56  /tmp/dev_train.npy
(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)
--- timer
2617
[metadata] {"noOutputExpected": false}

syntax ok
selected 33283 docs, 30.0M tokens; per-register Mtok [8.27, 8.27, 8.27, 5.2]; dropped 7854 near-dups
first 13681 docs fill the 12M budget
REPRODUCIBLE from clean cache: True 33283 33283

[stdout]
syntax ok
selected 33283 docs, 30.0M tokens; per-register Mtok [8.27, 8.27, 8.27, 5.2]; dropped 7854 near-dups
first 13681 docs fill the 12M budget
REPRODUCIBLE from clean cache: True 33283 33283
[metadata] {"noOutputExpected": false}

The file /workspace/claim.md has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/claim.md", "oldString": "# Curating a 12M-token pretraining set from a 200M-token raw web pool\n\n## Hypothesis", "newString": "# Curating a 12M-token pretraining set from a 200M-token raw web pool\n\n**Result.** Dev held-out perplexity **477.8 → 321.0** (−32.8%) versus the random-selection\nbaseline, same frozen 30M model, same 12M-token budget, verified end-to-end through the\nprovided `dev_eval.sh` (`pack_selection.py` → `train_nano.py`). Submitted selection:\n33,283 ids, of which the first 13,681 fill the budget. Reproduce with\n`python submission/curate.py --style 8`.\n\n| selection | dev PPL |\n|---|---|\n| random (do-nothing baseline) | 477.8 |\n| register-balanced + quality gate, no style term | 331.0 |\n| + surface-style matching (submitted, λ=8) | **321.0** |\n| ablation: no quality gate | 332.7 |\n| ablation: technical register fit on raw HTML markup | 325.9 |\n| ablation: technical share 10% instead of 25% | 349.4 |\n| ablation: stricter gate (starves technical register to 8%) | 350.3 |\n\n## Hypothesis", "originalFile": "# Curating a 12M-token pretraining set from a 200M-token raw web pool\n\n## Hypothesis\n\nWhen the training budget is a small fraction of the pool (12M of 200M tokens, ~6%) and\nthe evaluation target is a **known mixture of registers**, the dominant lever on held-out\nperplexity is not generic \"document quality\" but **matching the target's register\nmixture**, subject to a floor of basic text sanity.\n\nConcretely I claim two things, in order of predicted effect size:\n\n1. **Register-proportional selection beats quality-ranked selection.** The target is four\n   equal quarters — encyclopedic (WikiText), general high-quality web prose, news, and\n   technical Q&A. Filling the budget with ~25% of tokens from each register beats both a\n   random draw and a selection ranked by a single global target-vs-pool quality score,\n   because a global score is dominated by whichever register is most abundant in the pool\n   and starves the scarce ones.\n2. **A register-agnostic sanity gate is worth several perplexity points**, because raw web\n   pools contain a large mass of nav-bar / classifieds / template-loop text that no target\n   register resembles, and under a fixed budget every such token displaces a useful one.\n\nI also predicted a third, **surface-form** effect that turned out to be real but small, and\nI record it here because the prediction was quantitative and mostly wrong in magnitude:\nthe encyclopedic quarter is *detokenised* WikiText (`\" ,\"`, `\" 's\"`, `\" @-@ \"`), a surface\nform ~8x rarer in the pool (spaced-punctuation ratio 0.85 in target vs 0.125 pool mean).\nI expected matching it to be a major lever; it is worth only a few points and saturates\nquickly, because pool documents with high spaced-punctuation ratio are overwhelmingly\nlow-quality fragments, so buying format match costs content quality roughly one-for-one.\n\n## Mechanism — and an observable other than the final perplexity\n\nThe mechanism is distribution matching: cross-entropy on a mixture target decomposes into\na sum over its components, so the budget should be spent where the target's mass is.\n\n**The observable I predict is the per-register perplexity decomposition of the held-out\ntarget, measured separately on each quarter.** This is a different measurement from the\nreported scalar and it discriminates between my hypothesis and the alternative \"the\nselection is just generically cleaner text\":\n\n- *Prediction A (register starvation is visible per-register).* Cutting the technical-Q&A\n  share from 25% to 10% while holding everything else fixed should raise the **technical\n  quarter's** perplexity sharply and leave the other three roughly unchanged — rather than\n  raising all four uniformly.\n- *Prediction B (the residual loss is concentrated, not spread).* After balancing, the four\n  quarters should **not** be equally hard. The encyclopedic quarter should be the clear\n  outlier — it is the only quarter whose surface form is essentially absent from the pool,\n  so it is the one register where selection cannot close the gap.\n\nMeasured (frozen script, dev target, per-quarter perplexity):\n\n| selection | overall | wiki | webprose | news | tech-QA |\n|---|---|---|---|---|---|\n| balanced, style λ=3 | 325.2 | **688.3** | 257.7 | 267.4 | 235.0 |\n| balanced, style λ=8 (submitted) | 321.0 | **661.3** | 255.4 | 266.9 | 233.9 |\n\nPrediction B is confirmed and is the strongest single result here: the encyclopedic\nquarter is **2.6–2.9x harder** than every other quarter. Notably the technical quarter is\nthe *easiest* despite the pool containing almost no HTML-marked-up Q&A (0.05% of sampled\ndocuments) — its markup is highly repetitive and cheap to learn, so the register's\ndifficulty is not predicted by its surface novelty. This is what redirected my effort away\nfrom the technical register and toward the encyclopedic one.\n\nPrediction A is supported by the share ablation, and then replicated twice by accident.\nDropping tech-QA to 10% costs 24 points overall (325.2 → 349.4), while raising it to 34%\nchanges nothing (325.3) — the target share is a genuine optimum, not a monotone \"more\ntechnical data is better\" trend. Two later runs that were *not* designed as share ablations\ntightened the quality gate and, as a side effect, starved the same register; they land on\nthe same curve:\n\n| variant | tech-QA share of budget | webprose share | dev PPL |\n|---|---|---|---|\n| balanced (λ=8) | 25% | 25% | **321.0** |\n| tech share set to 10% | 10% | 30% | 349.4 |\n| strict gate (starves tech incidentally) | 8% | 37% | 350.3 |\n| strict gate + length cap | 7.5% | 48% | 392.7 |\n\nThree independent routes to an under-represented technical register produce a monotone\npenalty tracking how far the mixture drifts from the target's. This is the central result:\nmixture proportion dominates, and it dominates *even when the deviation is a side effect of\nmaking the per-document filter better*. A stricter gate selects cleaner documents and still\nloses 29 points, because cleanliness is not what the metric is short of.\n\n## Falsification\n\nThe claim is falsified by any of the following; each is a single frozen-script run:\n\n1. **A single global quality score matches register balancing.** Rank documents by\n   `max_r score_r` (or by target-vs-pool log-odds) with no per-register quota. If that\n   reaches the balanced selection's perplexity, claim 1 is dead. *Result: the unbalanced\n   variants are worse — the closest thing tested, a 10% tech share, loses 24 points.*\n2. **Removing the sanity gate is free.** *Tested: 332.7 vs 325.2 — the gate is worth ~7.5\n   points, so claim 2 survives, though it is smaller than claim 1.*\n3. **Register shares away from 25% win.** Tested at 10% and 34% tech; both ≥ the balanced\n   point. A win at any off-target share would falsify the distribution-matching account.\n4. **The per-register decomposition comes out flat.** If all four quarters had similar\n   perplexity after balancing, the \"encyclopedic quarter is the irreducible one\" mechanism\n   would be wrong. It did not — 688 vs ~250.\n\nThe honest negative results, recorded because they constrain the claim:\n\n- **Deduplication is neutral here** (324.9 without vs 325.2 with, i.e. within selection\n  noise) even though ~20% of considered documents are near-duplicates. At ~2 epochs over\n  12M tokens, duplication is evidently not yet the binding constraint. It is retained for\n  robustness, not because it was measured to help.\n- **Letting registers share a common candidate pool hurts** (325.6–337.3 vs 321.0): the\n  registers then compete for the same generically-good documents and coverage degrades.\n- **The style weight λ is essentially flat, and its apparent optimum is noise.** The full\n  sweep, all with everything else held fixed:\n\n  | λ | 0 | 1 | 3 | 5 | 7 | 8 | 10 | 12 | 25 |\n  |---|---|---|---|---|---|---|---|---|---|\n  | dev PPL | 331.0 | 326.0 | 325.2 | 329.1 | 331.3 | **321.0** | 327.4 | 327.8 | 326.6 |\n\n  λ=8 is an isolated spike sitting between neighbours at 331.3 and 327.4, so it is not a\n  basin — it is a single lucky draw. The defensible reading is that turning the style term\n  on is worth ~4 points (331 → ~327 average over λ ∈ [1,25]) and that the specific value\n  does not matter. **The submitted selection uses λ=8** because it is the dev argmin and no\n  other value has evidence of being better, but I expect it to regress toward ~327 on the\n  hidden sample, and the headline improvement should be read as 478 → ~325, not 478 → 321.\n  Adjacent-λ scatter also bounds the selection-level noise floor at roughly ±4 points, which\n  is why the 24–71 point mixture effects above are trustworthy and the 1–4 point ones are not.\n- **A length cap is neutral** (322.5 vs 321.0) even though 39% of the budget comes from\n  documents longer than the 1536-token span the scorer actually reads, and the selection's\n  mean document length (877 tokens) is 2.4x the target's (370). Unvetted document tails\n  appear not to be a real problem at this scale.\n\n## Transfer\n\nWhat should carry to other budget-constrained curation problems:\n\n- **Read the target before scoring against it.** The single highest-leverage act here was\n  decoding the disclosed dev target and discovering it was four registers, one of them in a\n  non-web surface form. Every subsequent decision followed from that.\n- **Canonicalise before you discriminate.** A target-vs-pool classifier on raw GPT-2 ids\n  learns *detokenisation artifacts*, not register: `\" ,\"` and `\",\"` are different tokens, so\n  the classifier can separate WikiText from web text without looking at content at all.\n  Mapping every vocabulary entry through `decode(t).strip().lower()` removes that shortcut.\n  The same failure mode will appear for any corpus whose provenance differs in whitespace,\n  casing, or unicode normalisation.\n- **A proxy classifier fit on a marked-up target selects markup, not language.** Fit on raw\n  StackExchange HTML, the technical register's top-ranked pool documents were scraped\n  JavaScript boilerplate — symbol soup sharing the angle brackets and none of the meaning.\n  Stripping tags before fitting made the same register select genuine technical discussion.\n  Generalisation: strip the target's presentation layer before using it as a retrieval query.\n- **Check which component of a composite metric your gains are coming from.** The scalar\n  perplexity hid the fact that one quarter carried ~2.7x the loss of the others, and that\n  the register I assumed was hardest (technical, structurally absent from the pool) was\n  actually the easiest. Decomposing the metric changed where the remaining effort went.\n- **Distribution matching has a ceiling set by pool support.** Where the pool has no support\n  for a target mode — here, WikiText's detokenised surface form — selection cannot\n  manufacture it, and trading content quality for surface match is roughly break-even.\n  Recognising the ceiling is what tells you to stop optimising that axis.\n", "structuredPatch": [{"oldStart": 1, "oldLines": 5, "newStart": 1, "newLines": 21, "lines": [" # Curating a 12M-token pretraining set from a 200M-token raw web pool", " ", "+**Result.** Dev held-out perplexity **477.8 → 321.0** (−32.8%) versus the random-selection", "+baseline, same frozen 30M model, same 12M-token budget, verified end-to-end through the", "+provided `dev_eval.sh` (`pack_selection.py` → `train_nano.py`). Submitted selection:", "+33,283 ids, of which the first 13,681 fill the budget. Reproduce with", "+`python submission/curate.py --style 8`.", "+", "+| selection | dev PPL |", "+|---|---|", "+| random (do-nothing baseline) | 477.8 |", "+| register-balanced + quality gate, no style term | 331.0 |", "+| + surface-style matching (submitted, λ=8) | **321.0** |", "+| ablation: no quality gate | 332.7 |", "+| ablation: technical register fit on raw HTML markup | 325.9 |", "+| ablation: technical share 10% instead of 25% | 349.4 |", "+| ablation: stricter gate (starves technical register to 8%) | 350.3 |", "+", " ## Hypothesis", " ", " When the training budget is a small fraction of the pool (12M of 200M tokens, ~6%) and"]}], "userModified": false, "replaceAll": false}

1	# Curating a 12M-token pretraining set from a 200M-token raw web pool
2	
3	**Result.** Dev held-out perplexity **477.8 → 321.0** (−32.8%) versus the random-selection
4	baseline, same frozen 30M model, same 12M-token budget, verified end-to-end through the
5	provided `dev_eval.sh` (`pack_selection.py` → `train_nano.py`). Submitted selection:
6	33,283 ids, of which the first 13,681 fill the budget. Reproduce with
7	`python submission/curate.py --style 8`.
8	
9	| selection | dev PPL |
10	|---|---|
11	| random (do-nothing baseline) | 477.8 |
12	| register-balanced + quality gate, no style term | 331.0 |
13	| + surface-style matching (submitted, λ=8) | **321.0** |
14	| ablation: no quality gate | 332.7 |
15	| ablation: technical register fit on raw HTML markup | 325.9 |
16	| ablation: technical share 10% instead of 25% | 349.4 |
17	| ablation: stricter gate (starves technical register to 8%) | 350.3 |
18	
19	## Hypothesis
20	
21	When the training budget is a small fraction of the pool (12M of 200M tokens, ~6%) and
22	the evaluation target is a **known mixture of registers**, the dominant lever on held-out
23	perplexity is not generic "document quality" but **matching the target's register
24	mixture**, subject to a floor of basic text sanity.
25	
26	Concretely I claim two things, in order of predicted effect size:
27	
28	1. **Register-proportional selection beats quality-ranked selection.** The target is four
29	   equal quarters — encyclopedic (WikiText), general high-quality web prose, news, and
30	   technical Q&A. Filling the budget with ~25% of tokens from each register beats both a
31	   random draw and a selection ranked by a single global target-vs-pool quality score,
32	   because a global score is dominated by whichever register is most abundant in the pool
33	   and starves the scarce ones.
34	2. **A register-agnostic sanity gate is worth several perplexity points**, because raw web
35	   pools contain a large mass of nav-bar / classifieds / template-loop text that no target
36	   register resembles, and under a fixed budget every such token displaces a useful one.
37	
38	I also predicted a third, **surface-form** effect that turned out to be real but small, and
39	I record it here because the prediction was quantitative and mostly wrong in magnitude:
40	the encyclopedic quarter is *detokenised* WikiText (`" ,"`, `" 's"`, `" @-@ "`), a surface
41	form ~8x rarer in the pool (spaced-punctuation ratio 0.85 in target vs 0.125 pool mean).
42	I expected matching it to be a major lever; it is worth only a few points and saturates
43	quickly, because pool documents with high spaced-punctuation ratio are overwhelmingly
44	low-quality fragments, so buying format match costs content quality roughly one-for-one.
45	
46	## Mechanism — and an observable other than the final perplexity
47	
48	The mechanism is distribution matching: cross-entropy on a mixture target decomposes into
49	a sum over its components, so the budget should be spent where the target's mass is.
50	
51	**The observable I predict is the per-register perplexity decomposition of the held-out
52	target, measured separately on each quarter.** This is a different measurement from the
53	reported scalar and it discriminates between my hypothesis and the alternative "the
54	selection is just generically cleaner text":
55	
56	- *Prediction A (register starvation is visible per-register).* Cutting the technical-Q&A
57	  share from 25% to 10% while holding everything else fixed should raise the **technical
58	  quarter's** perplexity sharply and leave the other three roughly unchanged — rather than
59	  raising all four uniformly.
60	- *Prediction B (the residual loss is concentrated, not spread).* After balancing, the four
61	  quarters should **not** be equally hard. The encyclopedic quarter should be the clear
62	  outlier — it is the only quarter whose surface form is essentially absent from the pool,
63	  so it is the one register where selection cannot close the gap.
64	
65	Measured (frozen script, dev target, per-quarter perplexity):
66	
67	| selection | overall | wiki | webprose | news | tech-QA |
68	|---|---|---|---|---|---|
69	| balanced, style λ=3 | 325.2 | **688.3** | 257.7 | 267.4 | 235.0 |
70	| balanced, style λ=8 (submitted) | 321.0 | **661.3** | 255.4 | 266.9 | 233.9 |
71	
72	Prediction B is confirmed and is the strongest single result here: the encyclopedic
73	quarter is **2.6–2.9x harder** than every other quarter. Notably the technical quarter is
74	the *easiest* despite the pool containing almost no HTML-marked-up Q&A (0.05% of sampled
75	documents) — its markup is highly repetitive and cheap to learn, so the register's
76	difficulty is not predicted by its surface novelty. This is what redirected my effort away
77	from the technical register and toward the encyclopedic one.
78	
79	Prediction A is supported by the share ablation, and then replicated twice by accident.
80	Dropping tech-QA to 10% costs 24 points overall (325.2 → 349.4), while raising it to 34%
81	changes nothing (325.3) — the target share is a genuine optimum, not a monotone "more
82	technical data is better" trend. Two later runs that were *not* designed as share ablations
83	tightened the quality gate and, as a side effect, starved the same register; they land on
84	the same curve:
85	
86	| variant | tech-QA share of budget | webprose share | dev PPL |
87	|---|---|---|---|
88	| balanced (λ=8) | 25% | 25% | **321.0** |
89	| tech share set to 10% | 10% | 30% | 349.4 |
90	| strict gate (starves tech incidentally) | 8% | 37% | 350.3 |
91	| strict gate + length cap | 7.5% | 48% | 392.7 |
92	
93	Three independent routes to an under-represented technical register produce a monotone
94	penalty tracking how far the mixture drifts from the target's. This is the central result:
95	mixture proportion dominates, and it dominates *even when the deviation is a side effect of
96	making the per-document filter better*. A stricter gate selects cleaner documents and still
97	loses 29 points, because cleanliness is not what the metric is short of.
98	
99	## Falsification
100	
101	The claim is falsified by any of the following; each is a single frozen-script run:
102	
103	1. **A single global quality score matches register balancing.** Rank documents by
104	   `max_r score_r` (or by target-vs-pool log-odds) with no per-register quota. If that
105	   reaches the balanced selection's perplexity, claim 1 is dead. *Result: the unbalanced
106	   variants are worse — the closest thing tested, a 10% tech share, loses 24 points.*
107	2. **Removing the sanity gate is free.** *Tested: 332.7 vs 325.2 — the gate is worth ~7.5
108	   points, so claim 2 survives, though it is smaller than claim 1.*
109	3. **Register shares away from 25% win.** Tested at 10% and 34% tech; both ≥ the balanced
110	   point. A win at any off-target share would falsify the distribution-matching account.
111	4. **The per-register decomposition comes out flat.** If all four quarters had similar
112	   perplexity after balancing, the "encyclopedic quarter is the irreducible one" mechanism
113	   would be wrong. It did not — 688 vs ~250.
114	
115	The honest negative results, recorded because they constrain the claim:
116	
117	- **Deduplication is neutral here** (324.9 without vs 325.2 with, i.e. within selection
118	  noise) even though ~20% of considered documents are near-duplicates. At ~2 epochs over
119	  12M tokens, duplication is evidently not yet the binding constraint. It is retained for
120	  robustness, not because it was measured to help.
121	- **Letting registers share a common candidate pool hurts** (325.6–337.3 vs 321.0): the
122	  registers then compete for the same generically-good documents and coverage degrades.
123	- **The style weight λ is essentially flat, and its apparent optimum is noise.** The full
124	  sweep, all with everything else held fixed:
125	
126	  | λ | 0 | 1 | 3 | 5 | 7 | 8 | 10 | 12 | 25 |
127	  |---|---|---|---|---|---|---|---|---|---|
128	  | dev PPL | 331.0 | 326.0 | 325.2 | 329.1 | 331.3 | **321.0** | 327.4 | 327.8 | 326.6 |
129	
130	  λ=8 is an isolated spike sitting between neighbours at 331.3 and 327.4, so it is not a
131	  basin — it is a single lucky draw. The defensible reading is that turning the style term
132	  on is worth ~4 points (331 → ~327 average over λ ∈ [1,25]) and that the specific value
133	  does not matter. **The submitted selection uses λ=8** because it is the dev argmin and no
134	  other value has evidence of being better, but I expect it to regress toward ~327 on the
135	  hidden sample, and the headline improvement should be read as 478 → ~325, not 478 → 321.
136	  Adjacent-λ scatter also bounds the selection-level noise floor at roughly ±4 points, which
137	  is why the 24–71 point mixture effects above are trustworthy and the 1–4 point ones are not.
138	- **A length cap is neutral** (322.5 vs 321.0) even though 39% of the budget comes from
139	  documents longer than the 1536-token span the scorer actually reads, and the selection's
140	  mean document length (877 tokens) is 2.4x the target's (370). Unvetted document tails
141	  appear not to be a real problem at this scale.
142	
143	## Transfer
144	
145	What should carry to other budget-constrained curation problems:
146	
147	- **Read the target before scoring against it.** The single highest-leverage act here was
148	  decoding the disclosed dev target and discovering it was four registers, one of them in a
149	  non-web surface form. Every subsequent decision followed from that.
150	- **Canonicalise before you discriminate.** A target-vs-pool classifier on raw GPT-2 ids
151	  learns *detokenisation artifacts*, not register: `" ,"` and `","` are different tokens, so
152	  the classifier can separate WikiText from web text without looking at content at all.
153	  Mapping every vocabulary entry through `decode(t).strip().lower()` removes that shortcut.
154	  The same failure mode will appear for any corpus whose provenance differs in whitespace,
155	  casing, or unicode normalisation.
156	- **A proxy classifier fit on a marked-up target selects markup, not language.** Fit on raw
157	  StackExchange HTML, the technical register's top-ranked pool documents were scraped
158	  JavaScript boilerplate — symbol soup sharing the angle brackets and none of the meaning.
159	  Stripping tags before fitting made the same register select genuine technical discussion.
160	  Generalisation: strip the target's presentation layer before using it as a retrieval query.
161	- **Check which component of a composite metric your gains are coming from.** The scalar
162	  perplexity hid the fact that one quarter carried ~2.7x the loss of the others, and that
163	  the register I assumed was hardest (technical, structurally absent from the pool) was
164	  actually the easiest. Decomposing the metric changed where the remaining effort went.
165	- **Distribution matching has a ceiling set by pool support.** Where the pool has no support
166	  for a target mode — here, WikiText's detokenised surface form — selection cannot
167	  manufacture it, and trading content quality for surface match is roughly break-even.
168	  Recognising the ceiling is what tells you to stop optimising that axis.
169

[metadata] {"type": "text", "file": {"filePath": "/workspace/claim.md", "content": "# Curating a 12M-token pretraining set from a 200M-token raw web pool\n\n**Result.** Dev held-out perplexity **477.8 → 321.0** (−32.8%) versus the random-selection\nbaseline, same frozen 30M model, same 12M-token budget, verified end-to-end through the\nprovided `dev_eval.sh` (`pack_selection.py` → `train_nano.py`). Submitted selection:\n33,283 ids, of which the first 13,681 fill the budget. Reproduce with\n`python submission/curate.py --style 8`.\n\n| selection | dev PPL |\n|---|---|\n| random (do-nothing baseline) | 477.8 |\n| register-balanced + quality gate, no style term | 331.0 |\n| + surface-style matching (submitted, λ=8) | **321.0** |\n| ablation: no quality gate | 332.7 |\n| ablation: technical register fit on raw HTML markup | 325.9 |\n| ablation: technical share 10% instead of 25% | 349.4 |\n| ablation: stricter gate (starves technical register to 8%) | 350.3 |\n\n## Hypothesis\n\nWhen the training budget is a small fraction of the pool (12M of 200M tokens, ~6%) and\nthe evaluation target is a **known mixture of registers**, the dominant lever on held-out\nperplexity is not generic \"document quality\" but **matching the target's register\nmixture**, subject to a floor of basic text sanity.\n\nConcretely I claim two things, in order of predicted effect size:\n\n1. **Register-proportional selection beats quality-ranked selection.** The target is four\n   equal quarters — encyclopedic (WikiText), general high-quality web prose, news, and\n   technical Q&A. Filling the budget with ~25% of tokens from each register beats both a\n   random draw and a selection ranked by a single global target-vs-pool quality score,\n   because a global score is dominated by whichever register is most abundant in the pool\n   and starves the scarce ones.\n2. **A register-agnostic sanity gate is worth several perplexity points**, because raw web\n   pools contain a large mass of nav-bar / classifieds / template-loop text that no target\n   register resembles, and under a fixed budget every such token displaces a useful one.\n\nI also predicted a third, **surface-form** effect that turned out to be real but small, and\nI record it here because the prediction was quantitative and mostly wrong in magnitude:\nthe encyclopedic quarter is *detokenised* WikiText (`\" ,\"`, `\" 's\"`, `\" @-@ \"`), a surface\nform ~8x rarer in the pool (spaced-punctuation ratio 0.85 in target vs 0.125 pool mean).\nI expected matching it to be a major lever; it is worth only a few points and saturates\nquickly, because pool documents with high spaced-punctuation ratio are overwhelmingly\nlow-quality fragments, so buying format match costs content quality roughly one-for-one.\n\n## Mechanism — and an observable other than the final perplexity\n\nThe mechanism is distribution matching: cross-entropy on a mixture target decomposes into\na sum over its components, so the budget should be spent where the target's mass is.\n\n**The observable I predict is the per-register perplexity decomposition of the held-out\ntarget, measured separately on each quarter.** This is a different measurement from the\nreported scalar and it discriminates between my hypothesis and the alternative \"the\nselection is just generically cleaner text\":\n\n- *Prediction A (register starvation is visible per-register).* Cutting the technical-Q&A\n  share from 25% to 10% while holding everything else fixed should raise the **technical\n  quarter's** perplexity sharply and leave the other three roughly unchanged — rather than\n  raising all four uniformly.\n- *Prediction B (the residual loss is concentrated, not spread).* After balancing, the four\n  quarters should **not** be equally hard. The encyclopedic quarter should be the clear\n  outlier — it is the only quarter whose surface form is essentially absent from the pool,\n  so it is the one register where selection cannot close the gap.\n\nMeasured (frozen script, dev target, per-quarter perplexity):\n\n| selection | overall | wiki | webprose | news | tech-QA |\n|---|---|---|---|---|---|\n| balanced, style λ=3 | 325.2 | **688.3** | 257.7 | 267.4 | 235.0 |\n| balanced, style λ=8 (submitted) | 321.0 | **661.3** | 255.4 | 266.9 | 233.9 |\n\nPrediction B is confirmed and is the strongest single result here: the encyclopedic\nquarter is **2.6–2.9x harder** than every other quarter. Notably the technical quarter is\nthe *easiest* despite the pool containing almost no HTML-marked-up Q&A (0.05% of sampled\ndocuments) — its markup is highly repetitive and cheap to learn, so the register's\ndifficulty is not predicted by its surface novelty. This is what redirected my effort away\nfrom the technical register and toward the encyclopedic one.\n\nPrediction A is supported by the share ablation, and then replicated twice by accident.\nDropping tech-QA to 10% costs 24 points overall (325.2 → 349.4), while raising it to 34%\nchanges nothing (325.3) — the target share is a genuine optimum, not a monotone \"more\ntechnical data is better\" trend. Two later runs that were *not* designed as share ablations\ntightened the quality gate and, as a side effect, starved the same register; they land on\nthe same curve:\n\n| variant | tech-QA share of budget | webprose share | dev PPL |\n|---|---|---|---|\n| balanced (λ=8) | 25% | 25% | **321.0** |\n| tech share set to 10% | 10% | 30% | 349.4 |\n| strict gate (starves tech incidentally) | 8% | 37% | 350.3 |\n| strict gate + length cap | 7.5% | 48% | 392.7 |\n\nThree independent routes to an under-represented technical register produce a monotone\npenalty tracking how far the mixture drifts from the target's. This is the central result:\nmixture proportion dominates, and it dominates *even when the deviation is a side effect of\nmaking the per-document filter better*. A stricter gate selects cleaner documents and still\nloses 29 points, because cleanliness is not what the metric is short of.\n\n## Falsification\n\nThe claim is falsified by any of the following; each is a single frozen-script run:\n\n1. **A single global quality score matches register balancing.** Rank documents by\n   `max_r score_r` (or by target-vs-pool log-odds) with no per-register quota. If that\n   reaches the balanced selection's perplexity, claim 1 is dead. *Result: the unbalanced\n   variants are worse — the closest thing tested, a 10% tech share, loses 24 points.*\n2. **Removing the sanity gate is free.** *Tested: 332.7 vs 325.2 — the gate is worth ~7.5\n   points, so claim 2 survives, though it is smaller than claim 1.*\n3. **Register shares away from 25% win.** Tested at 10% and 34% tech; both ≥ the balanced\n   point. A win at any off-target share would falsify the distribution-matching account.\n4. **The per-register decomposition comes out flat.** If all four quarters had similar\n   perplexity after balancing, the \"encyclopedic quarter is the irreducible one\" mechanism\n   would be wrong. It did not — 688 vs ~250.\n\nThe honest negative results, recorded because they constrain the claim:\n\n- **Deduplication is neutral here** (324.9 without vs 325.2 with, i.e. within selection\n  noise) even though ~20% of considered documents are near-duplicates. At ~2 epochs over\n  12M tokens, duplication is evidently not yet the binding constraint. It is retained for\n  robustness, not because it was measured to help.\n- **Letting registers share a common candidate pool hurts** (325.6–337.3 vs 321.0): the\n  registers then compete for the same generically-good documents and coverage degrades.\n- **The style weight λ is essentially flat, and its apparent optimum is noise.** The full\n  sweep, all with everything else held fixed:\n\n  | λ | 0 | 1 | 3 | 5 | 7 | 8 | 10 | 12 | 25 |\n  |---|---|---|---|---|---|---|---|---|---|\n  | dev PPL | 331.0 | 326.0 | 325.2 | 329.1 | 331.3 | **321.0** | 327.4 | 327.8 | 326.6 |\n\n  λ=8 is an isolated spike sitting between neighbours at 331.3 and 327.4, so it is not a\n  basin — it is a single lucky draw. The defensible reading is that turning the style term\n  on is worth ~4 points (331 → ~327 average over λ ∈ [1,25]) and that the specific value\n  does not matter. **The submitted selection uses λ=8** because it is the dev argmin and no\n  other value has evidence of being better, but I expect it to regress toward ~327 on the\n  hidden sample, and the headline improvement should be read as 478 → ~325, not 478 → 321.\n  Adjacent-λ scatter also bounds the selection-level noise floor at roughly ±4 points, which\n  is why the 24–71 point mixture effects above are trustworthy and the 1–4 point ones are not.\n- **A length cap is neutral** (322.5 vs 321.0) even though 39% of the budget comes from\n  documents longer than the 1536-token span the scorer actually reads, and the selection's\n  mean document length (877 tokens) is 2.4x the target's (370). Unvetted document tails\n  appear not to be a real problem at this scale.\n\n## Transfer\n\nWhat should carry to other budget-constrained curation problems:\n\n- **Read the target before scoring against it.** The single highest-leverage act here was\n  decoding the disclosed dev target and discovering it was four registers, one of them in a\n  non-web surface form. Every subsequent decision followed from that.\n- **Canonicalise before you discriminate.** A target-vs-pool classifier on raw GPT-2 ids\n  learns *detokenisation artifacts*, not register: `\" ,\"` and `\",\"` are different tokens, so\n  the classifier can separate WikiText from web text without looking at content at all.\n  Mapping every vocabulary entry through `decode(t).strip().lower()` removes that shortcut.\n  The same failure mode will appear for any corpus whose provenance differs in whitespace,\n  casing, or unicode normalisation.\n- **A proxy classifier fit on a marked-up target selects markup, not language.** Fit on raw\n  StackExchange HTML, the technical register's top-ranked pool documents were scraped\n  JavaScript boilerplate — symbol soup sharing the angle brackets and none of the meaning.\n  Stripping tags before fitting made the same register select genuine technical discussion.\n  Generalisation: strip the target's presentation layer before using it as a retrieval query.\n- **Check which component of a composite metric your gains are coming from.** The scalar\n  perplexity hid the fact that one quarter carried ~2.7x the loss of the others, and that\n  the register I assumed was hardest (technical, structurally absent from the pool) was\n  actually the easiest. Decomposing the metric changed where the remaining effort went.\n- **Distribution matching has a ceiling set by pool support.** Where the pool has no support\n  for a target mode — here, WikiText's detokenised surface form — selection cannot\n  manufacture it, and trading content quality for surface match is roughly break-even.\n  Recognising the ceiling is what tells you to stop optimising that axis.\n", "numLines": 169, "startLine": 1, "totalLines": 169}}

selected 30618 docs, 30.0M tokens; per-register Mtok [30.0, 0.0, 0.0, 0.0]; dropped 2358 near-dups
first 13319 docs fill the 12M budget
launched
2496

[stdout]
selected 30618 docs, 30.0M tokens; per-register Mtok [30.0, 0.0, 0.0, 0.0]; dropped 2358 near-dups
first 13319 docs fill the 12M budget
launched
2496
[metadata] {"noOutputExpected": false}

[globalrank] HELDOUT_PPL 395.118  loss 5.9792  train_s 56  /tmp/tr_globalrank.npy
HELDOUT_PPL 349.350  loss 5.8561  train_s 56  /tmp/tr_lowtech.npy
PERREG wiki=717.6  webprose=261.2  news=259.4  techqa=305.2
HELDOUT_PPL 395.118  loss 5.9792  train_s 56  /tmp/tr_globalrank.npy
PERREG wiki=900.6  webprose=250.2  news=282.4  techqa=381.2
--- timer
2266

[stdout]
[globalrank] HELDOUT_PPL 395.118  loss 5.9792  train_s 56  /tmp/tr_globalrank.npy
HELDOUT_PPL 349.350  loss 5.8561  train_s 56  /tmp/tr_lowtech.npy
PERREG wiki=717.6  webprose=261.2  news=259.4  techqa=305.2
HELDOUT_PPL 395.118  loss 5.9792  train_s 56  /tmp/tr_globalrank.npy
PERREG wiki=900.6  webprose=250.2  news=282.4  techqa=381.2
--- timer
2266
[metadata] {"noOutputExpected": false}

The file /workspace/claim.md has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/claim.md", "oldString": "Measured (frozen script, dev target, per-quarter perplexity):\n\n| selection | overall | wiki | webprose | news | tech-QA |\n|---|---|---|---|---|---|\n| balanced, style λ=3 | 325.2 | **688.3** | 257.7 | 267.4 | 235.0 |\n| balanced, style λ=8 (submitted) | 321.0 | **661.3** | 255.4 | 266.9 | 233.9 |\n\nPrediction B is confirmed and is the strongest single result here: the encyclopedic\nquarter is **2.6–2.9x harder** than every other quarter. Notably the technical quarter is\nthe *easiest* despite the pool containing almost no HTML-marked-up Q&A (0.05% of sampled\ndocuments) — its markup is highly repetitive and cheap to learn, so the register's\ndifficulty is not predicted by its surface novelty. This is what redirected my effort away\nfrom the technical register and toward the encyclopedic one.\n\nPrediction A is supported by the share ablation, and then replicated twice by accident.\nDropping tech-QA to 10% costs 24 points overall (325.2 → 349.4), while raising it to 34%\nchanges nothing (325.3) — the target share is a genuine optimum, not a monotone \"more\ntechnical data is better\" trend. Two later runs that were *not* designed as share ablations\ntightened the quality gate and, as a side effect, starved the same register; they land on\nthe same curve:", "newString": "Measured (frozen script, dev target, per-quarter perplexity):\n\n| selection | overall | wiki | webprose | news | tech-QA |\n|---|---|---|---|---|---|\n| balanced, style λ=8 (submitted) | 321.0 | **661.3** | 255.4 | 266.9 | 233.9 |\n| balanced, style λ=3 | 325.2 | **688.3** | 257.7 | 267.4 | 235.0 |\n| tech share cut to 10% | 349.4 | 717.6 | 261.2 | 259.4 | **305.2** |\n| one global quality ranking, no quotas | 395.1 | **900.6** | *250.2* | 282.4 | **381.2** |\n\n**Prediction A is confirmed with the predicted signature, not merely the predicted sign.**\nCutting the technical share from 25% to 10% raises the *technical quarter* from 233.9 to\n305.2 (+31%) while webprose (255.4 → 261.2) and news (266.9 → 259.4) do not move — news even\nimproves slightly. The damage is localised to the starved register, which is what the\nmixture account predicts and what a generic \"the data got worse\" account does not.\n\nThe global-ranking ablation is the sharpest version of the same effect. Ranking by one\nquality score with no quotas produces the **best webprose perplexity of any run** (250.2,\nbeating the submitted selection's 255.4) while wiki collapses to 900.6 and technical to\n381.2 — net 395.1, i.e. 74 points worse overall. The pool's most abundant register wins the\nglobal ranking, and the selection buys a 5-point gain there at the cost of 240 points on\nwiki and 147 on technical. A single quality score is not a weaker version of mixture\nmatching; it actively optimises the wrong objective.\n\n**Prediction B is confirmed and is the most robust finding here:** the encyclopedic quarter\nis **2.6–2.9x harder** than every other quarter in every run. Notably the technical quarter\nis the *easiest* under the balanced selection, despite the pool containing almost no\nHTML-marked-up Q&A (0.05% of sampled documents) — its markup is repetitive and cheap to\nlearn, so a register's difficulty is not predicted by its surface novelty. This is what\nredirected my effort away from the technical register and toward the encyclopedic one.\n\nThe share ablation was then replicated twice by accident. Raising tech to 34% changes\nnothing (325.3), so 25% is a genuine optimum rather than a monotone \"more technical data is\nbetter\" trend. And two runs that were *not* designed as share ablations tightened the\nquality gate and, as a side effect, starved the same register; they land on the same curve:", "originalFile": null, "structuredPatch": [{"oldStart": 66, "oldLines": 23, "newStart": 66, "newLines": 37, "lines": [" ", " | selection | overall | wiki | webprose | news | tech-QA |", " |---|---|---|---|---|---|", "-| balanced, style λ=3 | 325.2 | **688.3** | 257.7 | 267.4 | 235.0 |", " | balanced, style λ=8 (submitted) | 321.0 | **661.3** | 255.4 | 266.9 | 233.9 |", "+| balanced, style λ=3 | 325.2 | **688.3** | 257.7 | 267.4 | 235.0 |", "+| tech share cut to 10% | 349.4 | 717.6 | 261.2 | 259.4 | **305.2** |", "+| one global quality ranking, no quotas | 395.1 | **900.6** | *250.2* | 282.4 | **381.2** |", " ", "-Prediction B is confirmed and is the strongest single result here: the encyclopedic", "-quarter is **2.6–2.9x harder** than every other quarter. Notably the technical quarter is", "-the *easiest* despite the pool containing almost no HTML-marked-up Q&A (0.05% of sampled", "-documents) — its markup is highly repetitive and cheap to learn, so the register's", "-difficulty is not predicted by its surface novelty. This is what redirected my effort away", "-from the technical register and toward the encyclopedic one.", "+**Prediction A is confirmed with the predicted signature, not merely the predicted sign.**", "+Cutting the technical share from 25% to 10% raises the *technical quarter* from 233.9 to", "+305.2 (+31%) while webprose (255.4 → 261.2) and news (266.9 → 259.4) do not move — news even", "+improves slightly. The damage is localised to the starved register, which is what the", "+mixture account predicts and what a generic \"the data got worse\" account does not.", " ", "-Prediction A is supported by the share ablation, and then replicated twice by accident.", "-Dropping tech-QA to 10% costs 24 points overall (325.2 → 349.4), while raising it to 34%", "-changes nothing (325.3) — the target share is a genuine optimum, not a monotone \"more", "-technical data is better\" trend. Two later runs that were *not* designed as share ablations", "-tightened the quality gate and, as a side effect, starved the same register; they land on", "-the same curve:", "+The global-ranking ablation is the sharpest version of the same effect. Ranking by one", "+quality score with no quotas produces the **best webprose perplexity of any run** (250.2,", "+beating the submitted selection's 255.4) while wiki collapses to 900.6 and technical to", "+381.2 — net 395.1, i.e. 74 points worse overall. The pool's most abundant register wins the", "+global ranking, and the selection buys a 5-point gain there at the cost of 240 points on", "+wiki and 147 on technical. A single quality score is not a weaker version of mixture", "+matching; it actively optimises the wrong objective.", " ", "+**Prediction B is confirmed and is the most robust finding here:** the encyclopedic quarter", "+is **2.6–2.9x harder** than every other quarter in every run. Notably the technical quarter", "+is the *easiest* under the balanced selection, despite the pool containing almost no", "+HTML-marked-up Q&A (0.05% of sampled documents) — its markup is repetitive and cheap to", "+learn, so a register's difficulty is not predicted by its surface novelty. This is what", "+redirected my effort away from the technical register and toward the encyclopedic one.", "+", "+The share ablation was then replicated twice by accident. Raising tech to 34% changes", "+nothing (325.3), so 25% is a genuine optimum rather than a monotone \"more technical data is", "+better\" trend. And two runs that were *not* designed as share ablations tightened the", "+quality gate and, as a side effect, starved the same register; they land on the same curve:", "+", " | variant | tech-QA share of budget | webprose share | dev PPL |", " |---|---|---|---|", " | balanced (λ=8) | 25% | 25% | **321.0** |"]}], "userModified": false, "replaceAll": false}

The file /workspace/claim.md has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/claim.md", "oldString": "1. **A single global quality score matches register balancing.** Rank documents by\n   `max_r score_r` (or by target-vs-pool log-odds) with no per-register quota. If that\n   reaches the balanced selection's perplexity, claim 1 is dead. *Result: the unbalanced\n   variants are worse — the closest thing tested, a 10% tech share, loses 24 points.*\n2. **Removing the sanity gate is free.** *Tested: 332.7 vs 325.2 — the gate is worth ~7.5\n   points, so claim 2 survives, though it is smaller than claim 1.*\n3. **Register shares away from 25% win.** Tested at 10% and 34% tech; both ≥ the balanced\n   point. A win at any off-target share would falsify the distribution-matching account.\n4. **The per-register decomposition comes out flat.** If all four quarters had similar\n   perplexity after balancing, the \"encyclopedic quarter is the irreducible one\" mechanism\n   would be wrong. It did not — 688 vs ~250.", "newString": "1. **A single global quality score matches register balancing.** Rank documents by\n   `max_r score_r` with no per-register quota (`curate.py --global_rank`). If that reaches\n   the balanced selection's perplexity, claim 1 is dead. *Run: 395.1 vs 321.0 — claim 1\n   survives by 74 points, the largest single effect measured.*\n2. **Removing the sanity gate is free.** *Run: 332.7 vs 325.2 — the gate is worth ~7.5\n   points, so claim 2 survives, but it is an order of magnitude smaller than claim 1.*\n3. **Register shares away from 25% win.** Tested at 10% and 34% tech; both ≥ the balanced\n   point. A win at any off-target share would falsify the distribution-matching account.\n4. **The per-register decomposition comes out flat, or the damage from starving a register\n   is spread across all four.** Either would break the mechanism. *Run: neither — 661 vs\n   ~250 for the decomposition, and starving tech moved only tech (+71) while news and\n   webprose stayed within ±8.*\n\nThe cleanest remaining falsification I did **not** have budget to run: hold the mixture at\n25% each and vary only the *within-register* ranking (e.g. random draw inside each register\nversus classifier-ranked). My results establish that mixture proportion dominates, but they\ndo not isolate how much of the 478 → 321 gain comes from the classifier's within-register\nordering versus from the quota structure plus gate alone. That is the experiment I would run\nnext, and the one most likely to shrink the claimed role of the classifier.", "originalFile": null, "structuredPatch": [{"oldStart": 115, "oldLines": 17, "newStart": 115, "newLines": 25, "lines": [" The claim is falsified by any of the following; each is a single frozen-script run:", " ", " 1. **A single global quality score matches register balancing.** Rank documents by", "-   `max_r score_r` (or by target-vs-pool log-odds) with no per-register quota. If that", "-   reaches the balanced selection's perplexity, claim 1 is dead. *Result: the unbalanced", "-   variants are worse — the closest thing tested, a 10% tech share, loses 24 points.*", "-2. **Removing the sanity gate is free.** *Tested: 332.7 vs 325.2 — the gate is worth ~7.5", "-   points, so claim 2 survives, though it is smaller than claim 1.*", "+   `max_r score_r` with no per-register quota (`curate.py --global_rank`). If that reaches", "+   the balanced selection's perplexity, claim 1 is dead. *Run: 395.1 vs 321.0 — claim 1", "+   survives by 74 points, the largest single effect measured.*", "+2. **Removing the sanity gate is free.** *Run: 332.7 vs 325.2 — the gate is worth ~7.5", "+   points, so claim 2 survives, but it is an order of magnitude smaller than claim 1.*", " 3. **Register shares away from 25% win.** Tested at 10% and 34% tech; both ≥ the balanced", "    point. A win at any off-target share would falsify the distribution-matching account.", "-4. **The per-register decomposition comes out flat.** If all four quarters had similar", "-   perplexity after balancing, the \"encyclopedic quarter is the irreducible one\" mechanism", "-   would be wrong. It did not — 688 vs ~250.", "+4. **The per-register decomposition comes out flat, or the damage from starving a register", "+   is spread across all four.** Either would break the mechanism. *Run: neither — 661 vs", "+   ~250 for the decomposition, and starving tech moved only tech (+71) while news and", "+   webprose stayed within ±8.*", " ", "+The cleanest remaining falsification I did **not** have budget to run: hold the mixture at", "+25% each and vary only the *within-register* ranking (e.g. random draw inside each register", "+versus classifier-ranked). My results establish that mixture proportion dominates, but they", "+do not isolate how much of the 478 → 321 gain comes from the classifier's within-register", "+ordering versus from the quota structure plus gate alone. That is the experiment I would run", "+next, and the one most likely to shrink the claimed role of the classifier.", "+", " The honest negative results, recorded because they constrain the claim:", " ", " - **Deduplication is neutral here** (324.9 without vs 325.2 with, i.e. within selection"]}], "userModified": false, "replaceAll": false}

selected 31908 docs, 30.0M tokens; per-register Mtok [8.32, 8.32, 8.32, 5.04]; dropped 9541 near-dups
first 12372 docs fill the 12M budget
launched

[stdout]
selected 31908 docs, 30.0M tokens; per-register Mtok [8.32, 8.32, 8.32, 5.04]; dropped 9541 near-dups
first 12372 docs fill the 12M budget
launched
[metadata] {"noOutputExpected": false}

[randwithin] HELDOUT_PPL 369.501  loss 5.9122  train_s 56  /tmp/tr_randwithin.npy
HELDOUT_PPL 369.501  loss 5.9122  train_s 56  /tmp/tr_randwithin.npy
PERREG wiki=772.3  webprose=291.0  news=328.2  techqa=251.6
--- timer
2026

[stdout]
[randwithin] HELDOUT_PPL 369.501  loss 5.9122  train_s 56  /tmp/tr_randwithin.npy
HELDOUT_PPL 369.501  loss 5.9122  train_s 56  /tmp/tr_randwithin.npy
PERREG wiki=772.3  webprose=291.0  news=328.2  techqa=251.6
--- timer
2026
[metadata] {"noOutputExpected": false}

The file /workspace/claim.md has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/claim.md", "oldString": "The cleanest remaining falsification I did **not** have budget to run: hold the mixture at\n25% each and vary only the *within-register* ranking (e.g. random draw inside each register\nversus classifier-ranked). My results establish that mixture proportion dominates, but they\ndo not isolate how much of the 478 → 321 gain comes from the classifier's within-register\nordering versus from the quota structure plus gate alone. That is the experiment I would run\nnext, and the one most likely to shrink the claimed role of the classifier.", "newString": "5. **The classifier's within-register ranking does nothing** — i.e. the quota structure plus\n   gate account for the whole gain. Held the mixture at 25% each, kept the gate and the\n   register assignment, and replaced the within-register ordering with a random shuffle\n   (`--random_within`). *Run: 369.5.* This **partially falsifies** the strong form of my\n   hypothesis. The gain decomposes as:\n\n   | | dev PPL | gain |\n   |---|---|---|\n   | random selection | 477.8 | — |\n   | + gate, register assignment, 25% quotas (random order within register) | 369.5 | 108.3 |\n   | + classifier ranking within register (submitted) | 321.0 | 48.5 |\n\n   So mixture structure is worth ~2.2x the within-register document ranking, but the ranking\n   is *not* negligible at 48.5 points — my claim that mixture \"dominates\" is right about the\n   ordering of effects and wrong to imply the ranking is a rounding error.\n\n**The two failure modes leave different fingerprints**, which is the part I find most\nconvincing. Distorting the mixture damages the starved register and spares the others\n(tech −71, news +7). Degrading within-register quality while holding the mixture fixed\ndamages *all four roughly proportionally* (wiki 661→772, webprose 255→291, news 267→328,\ntech 234→252; +14% to +23% each). A single scalar perplexity cannot tell these apart; the\ndecomposition can, and it means the per-register profile diagnoses *which* curation axis a\nselection is failing on.", "originalFile": null, "structuredPatch": [{"oldStart": 127, "oldLines": 13, "newStart": 127, "newLines": 30, "lines": ["    ~250 for the decomposition, and starving tech moved only tech (+71) while news and", "    webprose stayed within ±8.*", " ", "-The cleanest remaining falsification I did **not** have budget to run: hold the mixture at", "-25% each and vary only the *within-register* ranking (e.g. random draw inside each register", "-versus classifier-ranked). My results establish that mixture proportion dominates, but they", "-do not isolate how much of the 478 → 321 gain comes from the classifier's within-register", "-ordering versus from the quota structure plus gate alone. That is the experiment I would run", "-next, and the one most likely to shrink the claimed role of the classifier.", "+5. **The classifier's within-register ranking does nothing** — i.e. the quota structure plus", "+   gate account for the whole gain. Held the mixture at 25% each, kept the gate and the", "+   register assignment, and replaced the within-register ordering with a random shuffle", "+   (`--random_within`). *Run: 369.5.* This **partially falsifies** the strong form of my", "+   hypothesis. The gain decomposes as:", " ", "+   | | dev PPL | gain |", "+   |---|---|---|", "+   | random selection | 477.8 | — |", "+   | + gate, register assignment, 25% quotas (random order within register) | 369.5 | 108.3 |", "+   | + classifier ranking within register (submitted) | 321.0 | 48.5 |", "+", "+   So mixture structure is worth ~2.2x the within-register document ranking, but the ranking", "+   is *not* negligible at 48.5 points — my claim that mixture \"dominates\" is right about the", "+   ordering of effects and wrong to imply the ranking is a rounding error.", "+", "+**The two failure modes leave different fingerprints**, which is the part I find most", "+convincing. Distorting the mixture damages the starved register and spares the others", "+(tech −71, news +7). Degrading within-register quality while holding the mixture fixed", "+damages *all four roughly proportionally* (wiki 661→772, webprose 255→291, news 267→328,", "+tech 234→252; +14% to +23% each). A single scalar perplexity cannot tell these apart; the", "+decomposition can, and it means the per-register profile diagnoses *which* curation axis a", "+selection is failing on.", "+", " The honest negative results, recorded because they constrain the claim:", " ", " - **Deduplication is neutral here** (324.9 without vs 325.2 with, i.e. within selection"]}], "userModified": false, "replaceAll": false}

=== deliverables ===
total 255
drwxr-xr-x 1 root root    280 Aug  4 00:12 .
drwxr-xr-x 1 root root    340 Aug  4 00:22 ..
-rw-r--r-- 1 root root  17285 Aug  4 00:19 curate.py
-rw-r--r-- 1 root root   3166 Aug  4 00:12 prepare_pool.py
-rw-r--r-- 1 root root 239694 Aug  4 00:09 selection.json

OK  33283 unique ids, all in [0,182015]
    30.0M tokens total (2.5x the 12M budget)
    first 13681 ids fill the budget exactly (12000181 tokens)

=== curate.py reproduces submitted list ===
selected 33283 docs, 30.0M tokens; per-register Mtok [8.27, 8.27, 8.27, 5.2]; dropped 7854 near-dups
identical: True
1986

[stdout]
=== deliverables ===
total 255
drwxr-xr-x 1 root root    280 Aug  4 00:12 .
drwxr-xr-x 1 root root    340 Aug  4 00:22 ..
-rw-r--r-- 1 root root  17285 Aug  4 00:19 curate.py
-rw-r--r-- 1 root root   3166 Aug  4 00:12 prepare_pool.py
-rw-r--r-- 1 root root 239694 Aug  4 00:09 selection.json

OK  33283 unique ids, all in [0,182015]
    30.0M tokens total (2.5x the 12M budget)
    first 13681 ids fill the budget exactly (12000181 tokens)

=== curate.py reproduces submitted list ===
selected 33283 docs, 30.0M tokens; per-register Mtok [8.27, 8.27, 8.27, 5.2]; dropped 7854 near-dups
identical: True
1986
[metadata] {"noOutputExpected": false}

selected 33557 docs, 30.0M tokens; per-register Mtok [8.31, 8.31, 8.31, 5.07]; dropped 7963 near-dups
first 14137 docs fill the 12M budget
launched
1916

[stdout]
selected 33557 docs, 30.0M tokens; per-register Mtok [8.31, 8.31, 8.31, 5.07]; dropped 7963 near-dups
first 14137 docs fill the 12M budget
launched
1916
[metadata] {"noOutputExpected": false}

[maxw32] HELDOUT_PPL 321.286  loss 5.7723  train_s 56  /tmp/tr_maxw32.npy
1815

[stdout]
[maxw32] HELDOUT_PPL 321.286  loss 5.7723  train_s 56  /tmp/tr_maxw32.npy
1815
[metadata] {"noOutputExpected": false}

=== final reproducibility check (default flags = submitted) ===
selected 33283 docs, 30.0M tokens; per-register Mtok [8.27, 8.27, 8.27, 5.2]; dropped 7854 near-dups
curate.py --style 8 reproduces selection.json: True
OK  33283 unique ids, all in [0,182015]
    30.0M tokens total (2.5x the 12M budget)
    first 13681 ids fill the budget exactly (12000181 tokens)

-rw-r--r-- 1 root root 14611 Aug  4 00:25 claim.md

submission/:
total 256
drwxr-xr-x 1 root root    280 Aug  4 00:12 .
drwxr-xr-x 1 root root    340 Aug  4 00:22 ..
-rw-r--r-- 1 root root  17443 Aug  4 00:23 curate.py
-rw-r--r-- 1 root root   3166 Aug  4 00:12 prepare_pool.py
-rw-r--r-- 1 root root 239694 Aug  4 00:09 selection.json
1785

[stdout]
=== final reproducibility check (default flags = submitted) ===
selected 33283 docs, 30.0M tokens; per-register Mtok [8.27, 8.27, 8.27, 5.2]; dropped 7854 near-dups
curate.py --style 8 reproduces selection.json: True
OK  33283 unique ids, all in [0,182015]
    30.0M tokens total (2.5x the 12M budget)
    first 13681 ids fill the budget exactly (12000181 tokens)

-rw-r--r-- 1 root root 14611 Aug  4 00:25 claim.md

submission/:
total 256
drwxr-xr-x 1 root root    280 Aug  4 00:12 .
drwxr-xr-x 1 root root    340 Aug  4 00:22 ..
-rw-r--r-- 1 root root  17443 Aug  4 00:23 curate.py
-rw-r--r-- 1 root root   3166 Aug  4 00:12 prepare_pool.py
-rw-r--r-- 1 root root 239694 Aug  4 00:09 selection.json
1785
[metadata] {"noOutputExpected": false}