claude-opus-5

The file /workspace/claim.md has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/claim.md", "oldString": "Measured noise: the same 12M-token set retrained at seed 4242 gives 307.9 vs 296.5 at seed\n1337, so single-run differences below ~10 ppl are not meaningful.", "newString": "The shipped row is the exact `submission/selection.json`, packed by the official\n`pack_selection.py` (12,000,000 tokens from 34,584 ids) — not a variant.\n\n**Robustness to the eval sample.** The official target is a *different* sample of the same\nfour registers, so the gain must not be an artifact of this particular dev sample. Splitting\neach register block into disjoint halves and scoring them separately: the shipped selection\ngives 299.7 / 301.4 and the uniform 25/25/25/25 mixture gives 314.6 / 316.1 — the same\n~15-point gap on each independent half.\n\nMeasured noise: the same 12M-token set retrained at seed 4242 gives 307.9 vs 296.5 at seed\n1337, so single-run differences below ~10 ppl are not meaningful.", "originalFile": "# Claim: the winning lever is *mixture matching*, not document quality\n\n## Hypothesis\n\nThe target is disclosed as \"equal parts encyclopedic / HQ web prose / news / technical Q&A\".\nDecoding `multi_dev.npy` shows this literally: it is **four contiguous 250k-token blocks**,\none per register (WikiText-103-style encyclopedic text with `@,@`/spaced-punctuation\ndetokenization artifacts; OpenWebText-style prose; a news wire; HTML-marked-up\nStackExchange Q&A with `<p>`, `<pre><code>`, `&gt;`).\n\n**Claim:** under a fixed 12M-token budget, held-out perplexity on such a target is governed\nfirst by *how the budget is divided across the target's registers*, and only second by\nper-document quality. A single global \"quality\" ranking — however good — spends the whole\nbudget on whichever register dominates the pool (here: generic web prose) and is therefore\nbadly mispriced against a 4-register target.\n\nConcretely: selecting documents by per-register affinity and emitting them so that **every\nprefix** of the list matches a target token mixture beats both a random selection and a\njunk-filtered global-quality ranking by a wide margin.\n\n## Mechanism — and an observable that is not the final perplexity\n\nMechanism: each register is a separate slice of the eval loss. Loss on a slice falls only if\nthe training set contains that register's surface form and vocabulary. Sorting the pool by\none global quality score leaves some slices essentially untrained, and their loss dominates\nthe mean. Balancing by token share trains every slice.\n\n**Predicted observable (measured with a diagnostic copy of the frozen trainer that reports\nper-quarter loss instead of the pooled number):** the per-register loss decomposition is\nstrongly *ordered and asymmetric*, and the ordering is predictable in advance from a\npool-side statistic — how much of the register's surface form exists in the pool at all:\n\n| register | pool documents matching its surface form | measured loss (nats) | ppl |\n|---|---|---|---|\n| W encyclopedic (WikiText) | **0** of 182,016 have `@,@`/`@-@`; only 129 have spaced punctuation | 6.42 | 616 |\n| N news | abundant | 5.70 | 299 |\n| P HQ web prose | abundant | 5.56 | 261 |\n| S technical Q&A (HTML) | **43** of 182,016 contain `<p>`/`<pre><code>` | **5.08** | **161** |\n\nTwo non-obvious predictions follow, both confirmed:\n\n1. **W is the loss floor-setter and is nearly irreducible.** The pool contains no\n   encyclopedic register to retrieve, so no selection rule can fix it. Prediction: attempts\n   to sharpen encyclopedic retrieval yield ~0 gain. Confirmed — a discriminative\n   register-vs-siblings profile plus a third-person/date-dense/no-promo style gate for W\n   changed dev ppl from 301.9 to 305.2 (i.e. nothing, at the noise level), and boosting W's\n   share 1.5x *hurt* (304.6 vs 296.5).\n2. **S is the cheapest slice despite being the rarest surface form**, because markup and code\n   are low-entropy once any structural text is seen. Prediction: its share should be pushed\n   *above* its 25% weight in the target. Confirmed: S-share 45% beats 25% by ~19 ppl.\n\n## Evidence\n\nAll numbers are dev perplexity from the frozen recipe, seed 1337 (train set = 12,000,000\ntokens in every row).\n\n| selection rule | dev ppl |\n|---|---|\n| random selection (do-nothing baseline) | 485.5 |\n| junk gate only, no register targeting | 430.9 |\n| single blended NB \"quality\" score, best-first | 429.7 |\n| register-balanced 25/25/25/25 | 315.7 |\n| register-balanced, S share 40% | 301.4 |\n| **shipped: register-balanced, weights 1:1:1:2.5 (S share 45%)** | **300.7** (293.0 at seed 4242; 2-seed mean 296.8) |\n| register-balanced, S share 50% | 296.5 |\n| register-balanced, S share 71% | 319.8 |\n| shipped rule but documents capped at 500 tokens | 415.9 |\n\nThe decisive comparison is row 3 vs row 4: **the same junk gate and the same NB machinery,\ndiffering only in whether the budget is divided by register (315.7) or spent best-first on a\nblended score (429.7)** — a 114-point swing with document-level quality held fixed.\n\nMeasured noise: the same 12M-token set retrained at seed 4242 gives 307.9 vs 296.5 at seed\n1337, so single-run differences below ~10 ppl are not meaningful. The shipped weighting was\nchosen for being the *stable* interior of the plateau, not the single-seed winner:\n2-seed means are 297.0 (S=2.5, spread 0.5), 299.8 (S=2), 302.2 (S=3, spread 11.4).\n\n## Falsification\n\nThe claim is wrong, or the mechanism is misattributed, if any of these hold:\n\n- **Mixture vs quality.** If a global-quality ranking (no register balancing) trained on the\n  same budget comes within ~10 ppl of the register-balanced mixture, the mechanism is not\n  mixture matching. *Tested: it loses by 114 ppl.*\n- **Register-share sensitivity.** If perplexity were flat in the register shares, \"matching\n  the mixture\" would be vacuous. *Tested: shares of 25%/45%/71% for S give 315.7/297.2/319.8\n  — a real, non-monotone optimum.*\n- **Prefix property.** If the pipeline's truncation point changed (say 6M instead of 12M) and\n  the mixture broke, the round-robin construction would be pointless. By construction every\n  prefix holds the target shares; a run at a different budget should show the same ranking.\n- **Sub-hypotheses that were refuted (reported as they came out):**\n  - *\"The S gain comes from genuine code/technical content.\"* Refuted. Requiring explicit\n    code/markup evidence for the S bucket (code-marker or tech-keyword rate above threshold,\n    2.4M–12.7M tokens available) gave 305.2/305.7, no better than the unrestricted S bucket\n    (301.9/296.5). The gain comes from broad structural/symbolic text, not from code per se.\n  - *\"Match the target's document-boundary density.\"* The dev target has an `<|endoftext|>`\n    every ~370 tokens; the selection averages ~700. Capping documents at 500 tokens to match\n    it gave **415.9** — catastrophically worse. Document coherence dominates boundary\n    statistics; short pool documents are fragments and boilerplate.\n  - *\"Refine the register profiles with pseudo-relevance feedback\"* (re-estimate each profile\n    from its own top-ranked pool documents, 2.5M tokens each, then re-score). Gave 319.1 vs\n    315.7 — no gain. The dev block vs pool background unigram is already sufficient.\n- **Falsifier for the shipped weights specifically:** evaluate on a *differently mixed*\n  target (e.g. all-encyclopedic). The S-upweighted selection must then lose to a W-heavy one.\n  If it still wins, the effect is not mixture matching but some generic property of the\n  S-bucket text, and the shipped weights are right for the wrong reason.\n\n## Transfer\n\n- **What transfers is the procedure, not the weights.** Segment a sample of the target into\n  registers; fit a cheap GPT-2-token unigram per register against a pool background; score\n  every pool document by mean clipped log-ratio; junk-gate *register-awarely* (prose gates\n  delete real code, so the technical bucket needs its own gate); dedup with word-5-gram\n  MinHash; then emit a **token-balanced round-robin** so the mixture is preserved at any\n  truncation point. No labels, no trained classifier, one tokenization pass (~1 min on 16\n  cores for 200M tokens), and it is budget-agnostic.\n- **The round-robin is the reusable trick.** When a downstream pipeline consumes a priority\n  list until a budget is met, ordering by \"whichever stratum is furthest behind its target\n  share\" makes the selection correct for every budget simultaneously — worth using anywhere a\n  cut point is out of the curator's control.\n- **Expect the optimal shares to differ from the target's shares.** They tilt toward registers\n  whose loss is most reducible per token (here: +20 points of share to the low-entropy\n  technical register) and away from registers with no matching surface form in the pool. That\n  is a small sweep, and it needs a noise estimate first: with a 30M model on 12M tokens,\n  seed-to-seed spread was ~10 ppl, which is larger than most of the weight-tuning effects.\n- **Limits.** Single pool, single target mixture, one model scale, and register profiles built\n  from a 1M-token target sample. The unigram profile is a weak retriever for registers that\n  are absent from the pool (the W quarter here); with a target register that the pool actually\n  contains, a stronger retriever should pay off more than it did here.\n", "structuredPatch": [{"oldStart": 70, "oldLines": 6, "newStart": 70, "newLines": 15, "lines": [" differing only in whether the budget is divided by register (315.7) or spent best-first on a", " blended score (429.7)** — a 114-point swing with document-level quality held fixed.", " ", "+The shipped row is the exact `submission/selection.json`, packed by the official", "+`pack_selection.py` (12,000,000 tokens from 34,584 ids) — not a variant.", "+", "+**Robustness to the eval sample.** The official target is a *different* sample of the same", "+four registers, so the gain must not be an artifact of this particular dev sample. Splitting", "+each register block into disjoint halves and scoring them separately: the shipped selection", "+gives 299.7 / 301.4 and the uniform 25/25/25/25 mixture gives 314.6 / 316.1 — the same", "+~15-point gap on each independent half.", "+", " Measured noise: the same 12M-token set retrained at seed 4242 gives 307.9 vs 296.5 at seed", " 1337, so single-run differences below ~10 ppl are not meaningful. The shipped weighting was", " chosen for being the *stable* interior of the plateau, not the single-seed winner:"]}], "userModified": false, "replaceAll": false}

-rw-r--r-- 1 root root 8913 Aug  4 00:24 claim.md

submission/:
total 252
drwxr-xr-x 1 root root     80 Aug  4 00:18 .
drwxr-xr-x 1 root root    240 Aug  4 00:24 ..
-rw-r--r-- 1 root root  10297 Aug  4 00:17 curate.py
-rw-r--r-- 1 root root 247073 Aug  4 00:18 selection.json
selection.json: 34584 unique ids, range [1,181996], first 8: [112271, 7641, 41186, 6138, 9544, 119803, 58452, 51268]
--- curate.py reproduces the shipped file:
  register token shares: [0.182 0.182 0.182 0.455] total 24.0M tokens
wrote /tmp/repro.json: 34584 ids
identical: True 34584 34584
1875

[stdout]
-rw-r--r-- 1 root root 8913 Aug  4 00:24 claim.md

submission/:
total 252
drwxr-xr-x 1 root root     80 Aug  4 00:18 .
drwxr-xr-x 1 root root    240 Aug  4 00:24 ..
-rw-r--r-- 1 root root  10297 Aug  4 00:17 curate.py
-rw-r--r-- 1 root root 247073 Aug  4 00:18 selection.json
selection.json: 34584 unique ids, range [1,181996], first 8: [112271, 7641, 41186, 6138, 9544, 119803, 58452, 51268]
--- curate.py reproduces the shipped file:
  register token shares: [0.182 0.182 0.182 0.455] total 24.0M tokens
wrote /tmp/repro.json: 34584 ids
identical: True 34584 34584
1875
[metadata] {"noOutputExpected": false}