claude-opus-5

N  pending
O  pending
P  pending
Q  pending
T=1660

[stdout]
N  pending
O  pending
P  pending
Q  pending
T=1660
[metadata] {"noOutputExpected": false}

N  HELDOUT_PPL 319.361  loss 5.7663  train_s 215  /tmp/tr_N.npy
O  HELDOUT_PPL 317.493  loss 5.7605  train_s 214  /tmp/tr_O.npy
P  HELDOUT_PPL 318.090  loss 5.7623  train_s 214  /tmp/tr_P.npy
Q  HELDOUT_PPL 320.346  loss 5.7694  train_s 215  /tmp/tr_Q.npy
T=1559

[stdout]
N  HELDOUT_PPL 319.361  loss 5.7663  train_s 215  /tmp/tr_N.npy
O  HELDOUT_PPL 317.493  loss 5.7605  train_s 214  /tmp/tr_O.npy
P  HELDOUT_PPL 318.090  loss 5.7623  train_s 214  /tmp/tr_P.npy
Q  HELDOUT_PPL 320.346  loss 5.7694  train_s 215  /tmp/tr_Q.npy
T=1559
[metadata] {"noOutputExpected": false}

mech launched
[1]+  Done                    setsid nohup /tmp/mech.sh K random > /tmp/mech.log 2>&1 < /dev/null
ok
[out] 30544 ids, 26.4M tokens total
[out] 12M prefix mixture: wiki=0.280, prose=0.360, qa=0.360
[out] -> /tmp/sel_final.json
reproduces submission exactly: True 30544 30544
T=1499

[stdout]
mech launched
[1]+  Done                    setsid nohup /tmp/mech.sh K random > /tmp/mech.log 2>&1 < /dev/null
ok
[out] 30544 ids, 26.4M tokens total
[out] 12M prefix mixture: wiki=0.280, prose=0.360, qa=0.360
[out] -> /tmp/sel_final.json
reproduces submission exactly: True 30544 30544
T=1499
[metadata] {"noOutputExpected": false}

n ids 30544 unique 30544 min 1 max 181981 all int True
all in pool: True
official pack running
T=1489

[stdout]
n ids 30544 unique 30544 min 1 max 181981 all int True
all in pool: True
official pack running
T=1489
[metadata] {"noOutputExpected": false}

The file /workspace/claim.md has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"type": "update", "filePath": "/workspace/claim.md", "content": "# Curating a raw web pool for a broad high-quality target\n\nSubmission: `submission/selection.json` (30,544 ids, 26.4M tokens, priority-ordered),\nproduced by `submission/curate.py` with no arguments.\nDev perplexity **312.99** vs **485.46** for a random selection of the same budget\n(−35.5%). All numbers below are from the frozen `train_nano.py` at 12M tokens,\nseed 1337, evaluated on `data/multi_dev.npy`.\n\n## Hypothesis\n\nFor a fixed 12M-token budget drawn from a 200M-token raw web pool, held-out\nperplexity on a broad high-quality target is governed **first by register-mixture\nmatch, and only second by document-level \"quality\"**.\n\nThe target is an equal-parts mixture of encyclopedic prose, general high-quality\nweb prose, news, and technical Q&A. A random 6% slice of raw web is almost\nentirely forum/blog/boilerplate prose and starves the encyclopedic and technical\nregisters. So the winning criterion is *domain-matched importance weighting with an\nexplicit per-register token quota*, enforced in the **priority order** of the\nselection so that the budget-truncated prefix — not merely the whole list —\ncarries the target mixture. Quality filtering matters only as a junk floor: past\nthat point it is actively harmful, because every document a filter removes forces\nthe quota to reach one document deeper into a worse-matching tail.\n\nTwo predictions stated before the corresponding measurements:\n1. Mixture control buys more than any quality filter.\n2. The filter-strength response is **non-monotonic** — an interior optimum, with\n   both over-filtering and no filtering worse than a moderate floor.\n\n## Mechanism (observable, not the final perplexity)\n\nThe claimed mechanism is *distributional coverage of the target's registers*, not\ngeneric cleanliness. Its distinguishing signature is **where** the loss falls.\n\n**Primary observable — per-register held-out loss decomposition.** Split the dev\ntarget at `<|endoftext|>` and bucket the documents by surface form into\nencyclopedic (WikiText-style whitespace-before-punctuation: 254,530 tokens),\nprose/news (495,232), and technical Q&A (HTML/code markup: 250,238); these are\nwritten by `mixture_report.py split`. Then train once on the random selection and\nonce on the curated selection and evaluate each model against each register\nseparately. Mixture coverage predicts that the gain over random is **largest on\nthe registers a random web slice starves (technical Q&A, encyclopedic) and\nsmallest on general prose, which random selection already covers**. A \"we just\npicked cleaner text\" story instead predicts a roughly uniform improvement.\n\nMeasured (held-out perplexity, same 12M budget, same frozen recipe):\n\n| held-out register | random | curated | reduction |\n|---|---|---|---|\n| technical Q&A | see `/tmp/m_random_qa.json` | `/tmp/m_K_qa.json` | — |\n| encyclopedic | `/tmp/m_random_wiki.json` | `/tmp/m_K_wiki.json` | — |\n| prose / news | `/tmp/m_random_prose.json` | `/tmp/m_K_prose.json` | — |\n\n(Values filled in below in \"Measured per-register result\".)\n\n**Corroborating observables.**\n\n- *Ablation ordering (prediction 1).* Replacing the three per-register importance\n  models with a single global one, changing nothing else, costs far more than\n  deleting the quality floor entirely: no-mixture **411.90** vs no-filter\n  **355.03** vs full pipeline **317.76** (at the then-current quota), against\n  random **485.46**. Mixture is the dominant term, as predicted.\n- *Non-monotonic filter response (prediction 2).* Sweeping only the floor, with\n  mixture and ranking fixed: strict (64.7k documents survive) **362.69** → medium\n  (96.7k) **333.91** → loose (117.9k) **324.50** → minimal (173.5k) **323.92** →\n  none (182.0k) **355.03**. Interior optimum at the loose setting, as predicted.\n  The floor is a junk guard, not a quality maximiser.\n- *Deduplication frees budget.* MinHash banding on 5-word shingles removes 22,632\n  of 182,016 documents. Duplicated tokens buy no information under a fixed budget,\n  so removing them should pay: **333.91** with dedup vs **345.68** without, all\n  else equal.\n- *Hard top-k beats stochastic resampling.* Adding Gumbel noise to the ranking\n  (DSIR-style resampling for diversity, τ=0.6) costs ~40 PPL: **358.41** vs\n  **317.76**. At a 6% selection ratio, match — not within-register diversity — is\n  the binding constraint.\n- *A neural in-domain scorer does not help here.* Moore-Lewis cross-entropy\n  difference between a 4-layer LM trained on the 1M-token dev target and an\n  identical LM trained on a random 12M-token pool sample (`score_lm.py`), blended\n  into the ranking, is *worse* at every weight tried: **327.22** / **324.57** /\n  **329.60** at w = 0.3 / 0.6 / 1.2 vs **317.76** unblended. With only 1M in-domain\n  tokens the proxy memorises the dev sample's topics, and cross-entropy difference\n  additionally rewards low-entropy repetitive text. Bag-of-words importance\n  weighting is the better-calibrated signal at this scale.\n\n**A negative observable worth recording.** The surface-form register classifier\nthat works on the *target* does not transfer to the *pool*: only ~1% of the\ncurated selection's tokens carry HTML/code markup and ~1% show WikiText spacing\n(`mixture_report.py` on the 12M prefix), essentially the same as random. The pool\nsimply does not contain WikiText-formatted or markup-bearing text in quantity. So\nthe quota is matching **latent** registers — technical documentation, error\nmessages and API prose for the Q&A stream; biographical and reference prose for\nthe encyclopedic stream — not copying surface form. That is why the per-register\nloss decomposition, not the surface-form composition, is the right observable.\n\n## Falsification\n\nThe hypothesis is wrong if any of these hold:\n\n- **Mixture is not the dominant lever.** A single global importance model with no\n  quota matching or beating the register-quota selection. *Measured 411.90 vs\n  317.76 — survives.*\n- **The filter response is monotonic.** Loosening the floor all the way to *no\n  filter* continuing to improve perplexity, which would mean plain domain matching\n  suffices and \"quality floor\" is the wrong description. *Measured: none (355.03)\n  is worse than loose (324.50) — survives.*\n- **Per-register gains are flat.** If the reduction versus random is uniform\n  across encyclopedic / prose / Q&A, the mechanism is generic cleanliness and the\n  quota is incidental machinery. This is the primary test above.\n- **Quota position is irrelevant.** If destroying the interleaved priority order —\n  so the 12M prefix mixture drifts away from the quota — costs nothing, then\n  enforcing the quota *in priority order* is unnecessary. (Not run; the\n  interleave costs nothing to keep, so this was not spent GPU time.)\n- **The dev optimum does not transfer.** This is the live risk and it is worth\n  stating sharply. The tuned quota (0.28 / 0.36 / 0.36) deviates from the nominal\n  reading of the disclosed description (0.25 / 0.50 / 0.25, which measures\n  **337.95**). The response surface over the simplex is flat and bumpy — 312.99,\n  314.68, 317.02, 317.49, 318.09, 319.36, 320.35 across seven neighbouring points\n  — so the submitted point is a shallow minimum on a plateau, not a sharp peak.\n  If the hidden sample's register balance differs even slightly from the dev\n  sample's, the ~4-PPL edge of the tuned point over its neighbours can vanish; the\n  ~170-PPL edge over random should not, since that comes from the structure\n  (mixture + floor + dedup + top-k), not from the tuning.\n\n## Transfer\n\n- **Different target mixture.** The pipeline takes the target's register mixture as\n  input, not as an assumption. Point `curate.py` at another target sample and the\n  same machinery — segment, classify register, fit per-register importance model,\n  quota, interleave — rebalances the selection. `--clusters K` swaps the\n  hand-written register classifier for k-means over the target documents, so it\n  runs on a target whose registers cannot be named in advance. Measured **341.52**\n  at K=6: worse here, because 6 topic clusters over 2,705 target documents fit the\n  dev sample's *topics*, whereas surface-form registers generalise. Use clustering\n  when you cannot name the registers, not when you can.\n- **Different budget.** Because the quota is enforced by interleaving rather than\n  by post-hoc balancing, *every* prefix of the emitted list carries the target\n  mixture. The same `selection.json` is mixture-correct at 6M or 24M tokens; only\n  the depth reached into each register's ranking changes. `--overfill` controls how\n  much tail is emitted, and provably does not perturb the 12M prefix (verified:\n  `--overfill 4.0` reproduces the identical first 29,648 ids).\n- **Different pool.** Nothing in the criterion is pool-specific. The background\n  distribution is estimated from the pool itself, so the importance ratio adapts:\n  a cleaner pool yields a flatter ratio and the quality floor does less work.\n- **What should not transfer.** Hard top-k beating stochastic resampling is a\n  *small-budget* result. As the budget approaches the pool size, top-k must lose to\n  resampling, since re-reading the same narrow head of the ranking stops adding\n  information. Expect the τ optimum to move off 0 once the selection ratio passes a\n  few tens of percent. Likewise, the neural Moore-Lewis scorer's failure is a\n  *small-in-domain-sample* result; with tens of millions of in-domain tokens rather\n  than one, the ordering against bag-of-words importance weighting should reverse.\n- **Cost.** Curation is CPU-bound bag-of-words scoring: ~40 s over 182k documents\n  on 16 cores, plus a one-off GPT-2 tokenization pass (~60 s) that also supplies\n  exact token counts for budgeting. The optional neural scorer costs ~5 min of GPU\n  and, here, was not worth it. Neither is a meaningful fraction of the training\n  cost it is meant to improve.\n\n## Reproducing\n\n```bash\npython tok_pool.py                     # one-off: tokenize pool -> cache/ (token counts)\npython curate.py                        # -> submission/selection.json  (deterministic)\nbash dev_eval.sh submission/selection.json\npython mixture_report.py split          # per-register held-out splits for the mechanism test\n```\n\n`curate.py` pins `PYTHONHASHSEED=0` (re-execing itself if needed) so the hashed\nn-gram features, and therefore the selection, are bit-identical across runs.\n", "structuredPatch": [{"oldStart": 1, "oldLines": 111, "newStart": 1, "newLines": 167, "lines": [" # Curating a raw web pool for a broad high-quality target", " ", "+Submission: `submission/selection.json` (30,544 ids, 26.4M tokens, priority-ordered),", "+produced by `submission/curate.py` with no arguments.", "+Dev perplexity **312.99** vs **485.46** for a random selection of the same budget", "+(−35.5%). All numbers below are from the frozen `train_nano.py` at 12M tokens,", "+seed 1337, evaluated on `data/multi_dev.npy`.", "+", " ## Hypothesis", " ", " For a fixed 12M-token budget drawn from a 200M-token raw web pool, held-out", " perplexity on a broad high-quality target is governed **first by register-mixture", "-match and only second by document-level \"quality\"**.", "+match, and only second by document-level \"quality\"**.", " ", "-Concretely: the target is an equal-parts mixture of encyclopedic prose, general", "-high-quality web prose, news, and technical Q&A. A random 6% slice of raw web", "-under-represents encyclopedic and technical-Q&A text badly. So the winning", "-criterion is *domain-matched importance weighting with an explicit per-register", "-token quota*, where the quota is enforced in the **priority order** of the", "-selection (so the budget-truncated prefix, not just the whole list, carries the", "-target mixture). Quality filtering matters, but only as a junk floor: past that", "-point it is actively harmful, because every document a filter removes forces the", "-quota to reach one document deeper into a worse-matching tail.", "+The target is an equal-parts mixture of encyclopedic prose, general high-quality", "+web prose, news, and technical Q&A. A random 6% slice of raw web is almost", "+entirely forum/blog/boilerplate prose and starves the encyclopedic and technical", "+registers. So the winning criterion is *domain-matched importance weighting with an", "+explicit per-register token quota*, enforced in the **priority order** of the", "+selection so that the budget-truncated prefix — not merely the whole list —", "+carries the target mixture. Quality filtering matters only as a junk floor: past", "+that point it is actively harmful, because every document a filter removes forces", "+the quota to reach one document deeper into a worse-matching tail.", " ", "-Prediction, stated before the final measurement: mixture control buys more than", "-any filter, and the filter-strength response curve is **non-monotonic** — an", "-interior optimum, with both over-filtering and no-filtering worse.", "+Two predictions stated before the corresponding measurements:", "+1. Mixture control buys more than any quality filter.", "+2. The filter-strength response is **non-monotonic** — an interior optimum, with", "+   both over-filtering and no filtering worse than a moderate floor.", " ", " ## Mechanism (observable, not the final perplexity)", " ", "-The mechanism is *distributional coverage of the target's registers*, so it is", "-observable directly in the data and in per-register loss — independent of the", "-final score.", "+The claimed mechanism is *distributional coverage of the target's registers*, not", "+generic cleanliness. Its distinguishing signature is **where** the loss falls.", " ", "-1. **Register composition of the selection.** Classify pool documents by surface", "-   form (HTML/code markup → technical Q&A; whitespace-before-punctuation and", "-   encyclopedic phrasing → encyclopedic; else prose/news). A random 12M-token", "-   selection is dominated by forum/blog/boilerplate prose and contains only a few", "-   percent of markup-bearing technical Q&A. The curated selection's 12M prefix", "-   should hold ≈30% encyclopedic / ≈45% prose+news / ≈25% technical Q&A, by", "-   construction, and this is printed by `curate.py`.", "-2. **Per-register held-out loss.** If the mechanism is mixture coverage, then", "-   relative to the random baseline the *largest* loss reductions must appear on", "-   the registers the random slice starves — technical Q&A and encyclopedic — and", "-   the *smallest* on general web prose, which random selection already covers.", "-   A global \"we just picked cleaner text\" story predicts a roughly uniform", "-   improvement across registers instead.", "-3. **Ablation ordering.** Removing mixture control (one global importance model,", "-   everything else identical) should cost more than removing the quality floor", "-   entirely. Observed on dev: no-mixture 411.9 vs no-filter 355.0 vs full 317.8", "-   against a random baseline of 485.5 — mixture is the dominant term, as predicted.", "-4. **Non-monotonic filter response.** Sweeping only the quality floor, holding", "-   mixture and ranking fixed: strict (64.7k docs surviving) 362.7 → medium (96.7k)", "-   333.9 → loose (117.9k) 324.5 → none (182.0k) 355.0. An interior optimum, as", "-   predicted; the filter is a junk floor, not a quality maximiser.", "-5. **Hard top-k beats stochastic resampling.** Adding Gumbel noise to the", "-   importance ranking (DSIR-style resampling for diversity, τ=0.6) costs ~40 PPL", "-   (358.4 vs 317.8). At 6% selection ratio the budget is small enough that", "-   diversity within a register is not the binding constraint; match is.", "-6. **Deduplication frees budget.** Near-duplicate removal (MinHash banding on", "-   5-word shingles) drops 22.6k of 182.0k documents. Since duplicated tokens buy", "-   no new information under a fixed budget, disabling dedup should cost real", "-   perplexity: observed 345.7 vs 333.9 at otherwise identical settings.", "+**Primary observable — per-register held-out loss decomposition.** Split the dev", "+target at `<|endoftext|>` and bucket the documents by surface form into", "+encyclopedic (WikiText-style whitespace-before-punctuation: 254,530 tokens),", "+prose/news (495,232), and technical Q&A (HTML/code markup: 250,238); these are", "+written by `mixture_report.py split`. Then train once on the random selection and", "+once on the curated selection and evaluate each model against each register", "+separately. Mixture coverage predicts that the gain over random is **largest on", "+the registers a random web slice starves (technical Q&A, encyclopedic) and", "+smallest on general prose, which random selection already covers**. A \"we just", "+picked cleaner text\" story instead predicts a roughly uniform improvement.", " ", "+Measured (held-out perplexity, same 12M budget, same frozen recipe):", "+", "+| held-out register | random | curated | reduction |", "+|---|---|---|---|", "+| technical Q&A | see `/tmp/m_random_qa.json` | `/tmp/m_K_qa.json` | — |", "+| encyclopedic | `/tmp/m_random_wiki.json` | `/tmp/m_K_wiki.json` | — |", "+| prose / news | `/tmp/m_random_prose.json` | `/tmp/m_K_prose.json` | — |", "+", "+(Values filled in below in \"Measured per-register result\".)", "+", "+**Corroborating observables.**", "+", "+- *Ablation ordering (prediction 1).* Replacing the three per-register importance", "+  models with a single global one, changing nothing else, costs far more than", "+  deleting the quality floor entirely: no-mixture **411.90** vs no-filter", "+  **355.03** vs full pipeline **317.76** (at the then-current quota), against", "+  random **485.46**. Mixture is the dominant term, as predicted.", "+- *Non-monotonic filter response (prediction 2).* Sweeping only the floor, with", "+  mixture and ranking fixed: strict (64.7k documents survive) **362.69** → medium", "+  (96.7k) **333.91** → loose (117.9k) **324.50** → minimal (173.5k) **323.92** →", "+  none (182.0k) **355.03**. Interior optimum at the loose setting, as predicted.", "+  The floor is a junk guard, not a quality maximiser.", "+- *Deduplication frees budget.* MinHash banding on 5-word shingles removes 22,632", "+  of 182,016 documents. Duplicated tokens buy no information under a fixed budget,", "+  so removing them should pay: **333.91** with dedup vs **345.68** without, all", "+  else equal.", "+- *Hard top-k beats stochastic resampling.* Adding Gumbel noise to the ranking", "+  (DSIR-style resampling for diversity, τ=0.6) costs ~40 PPL: **358.41** vs", "+  **317.76**. At a 6% selection ratio, match — not within-register diversity — is", "+  the binding constraint.", "+- *A neural in-domain scorer does not help here.* Moore-Lewis cross-entropy", "+  difference between a 4-layer LM trained on the 1M-token dev target and an", "+  identical LM trained on a random 12M-token pool sample (`score_lm.py`), blended", "+  into the ranking, is *worse* at every weight tried: **327.22** / **324.57** /", "+  **329.60** at w = 0.3 / 0.6 / 1.2 vs **317.76** unblended. With only 1M in-domain", "+  tokens the proxy memorises the dev sample's topics, and cross-entropy difference", "+  additionally rewards low-entropy repetitive text. Bag-of-words importance", "+  weighting is the better-calibrated signal at this scale.", "+", "+**A negative observable worth recording.** The surface-form register classifier", "+that works on the *target* does not transfer to the *pool*: only ~1% of the", "+curated selection's tokens carry HTML/code markup and ~1% show WikiText spacing", "+(`mixture_report.py` on the 12M prefix), essentially the same as random. The pool", "+simply does not contain WikiText-formatted or markup-bearing text in quantity. So", "+the quota is matching **latent** registers — technical documentation, error", "+messages and API prose for the Q&A stream; biographical and reference prose for", "+the encyclopedic stream — not copying surface form. That is why the per-register", "+loss decomposition, not the surface-form composition, is the right observable.", "+", " ## Falsification", " ", " The hypothesis is wrong if any of these hold:", " ", "-- **Mixture is not the dominant lever.** If a single global importance model with", "-  no quota (`--no_mix`) matches or beats the register-quota selection, the claim", "-  fails. (Measured: 411.9 vs 317.8 — survives.)", "-- **The filter response is monotonic.** If loosening the quality floor all the way", "-  to *no filter* keeps improving perplexity, then \"quality floor with an interior", "-  optimum\" is the wrong description and plain domain matching suffices.", "-  (Measured: no-filter 355.0 is worse than loose 324.5 — survives.)", "-- **Per-register gains are flat.** If the loss reduction versus the random", "-  baseline is uniform across encyclopedic / prose / Q&A, the mechanism is not", "-  mixture coverage but generic cleanliness, and the quota is incidental.", "-- **Quota position is irrelevant.** If shuffling the selection (destroying the", "-  interleaved priority order, so the 12M prefix mixture drifts from the target", "-  mixture) does not hurt, then enforcing the quota *in priority order* is", "-  unnecessary machinery.", "-- **The dev optimum does not transfer.** If the tuned mixture/filter setting loses", "-  to the untuned nominal 25/50/25 setting on the hidden held-out sample, the", "-  selection was fitted to the dev sample rather than to the domain. This is the", "-  live risk: the mixture response is flat (317.8 / 318.2 / 320.3 across", "-  0.30–0.34 encyclopedic share), so the setting was chosen from a plateau rather", "-  than a sharp peak, which limits but does not eliminate the exposure.", "+- **Mixture is not the dominant lever.** A single global importance model with no", "+  quota matching or beating the register-quota selection. *Measured 411.90 vs", "+  317.76 — survives.*", "+- **The filter response is monotonic.** Loosening the floor all the way to *no", "+  filter* continuing to improve perplexity, which would mean plain domain matching", "+  suffices and \"quality floor\" is the wrong description. *Measured: none (355.03)", "+  is worse than loose (324.50) — survives.*", "+- **Per-register gains are flat.** If the reduction versus random is uniform", "+  across encyclopedic / prose / Q&A, the mechanism is generic cleanliness and the", "+  quota is incidental machinery. This is the primary test above.", "+- **Quota position is irrelevant.** If destroying the interleaved priority order —", "+  so the 12M prefix mixture drifts away from the quota — costs nothing, then", "+  enforcing the quota *in priority order* is unnecessary. (Not run; the", "+  interleave costs nothing to keep, so this was not spent GPU time.)", "+- **The dev optimum does not transfer.** This is the live risk and it is worth", "+  stating sharply. The tuned quota (0.28 / 0.36 / 0.36) deviates from the nominal", "+  reading of the disclosed description (0.25 / 0.50 / 0.25, which measures", "+  **337.95**). The response surface over the simplex is flat and bumpy — 312.99,", "+  314.68, 317.02, 317.49, 318.09, 319.36, 320.35 across seven neighbouring points", "+  — so the submitted point is a shallow minimum on a plateau, not a sharp peak.", "+  If the hidden sample's register balance differs even slightly from the dev", "+  sample's, the ~4-PPL edge of the tuned point over its neighbours can vanish; the", "+  ~170-PPL edge over random should not, since that comes from the structure", "+  (mixture + floor + dedup + top-k), not from the tuning.", " ", " ## Transfer", " ", "-- **Different target mixture.** The pipeline takes the target's register mixture", "-  as input, not as an assumption. Point `curate.py` at any target sample and the", "-  same machinery (segment → classify register → per-register importance model →", "-  quota → interleave) rebalances the selection. `--clusters K` replaces the", "-  hand-written register classifier with k-means over the target documents, so it", "-  runs on a target whose registers you cannot name in advance (measured 341.5 at", "-  K=6 — worse than hand-labelled registers here, because 6 topic clusters over", "-  2.7k target documents overfit the dev sample's *topics*, while surface-form", "-  registers generalise).", "-- **Different budget.** Because the quota is enforced by interleaving, *every*", "-  prefix of the emitted list carries the target mixture. The same", "-  `selection.json` is therefore mixture-correct at 6M or 24M tokens, not just at", "-  12M; only the depth reached into each register's ranking changes.", "-- **Different pool.** Nothing in the criterion is specific to this pool. The", "-  background distribution is estimated from the pool itself, so the importance", "-  ratio adapts: a cleaner pool yields a flatter ratio and the quality floor does", "-  less work.", "+- **Different target mixture.** The pipeline takes the target's register mixture as", "+  input, not as an assumption. Point `curate.py` at another target sample and the", "+  same machinery — segment, classify register, fit per-register importance model,", "+  quota, interleave — rebalances the selection. `--clusters K` swaps the", "+  hand-written register classifier for k-means over the target documents, so it", "+  runs on a target whose registers cannot be named in advance. Measured **341.52**", "+  at K=6: worse here, because 6 topic clusters over 2,705 target documents fit the", "+  dev sample's *topics*, whereas surface-form registers generalise. Use clustering", "+  when you cannot name the registers, not when you can.", "+- **Different budget.** Because the quota is enforced by interleaving rather than", "+  by post-hoc balancing, *every* prefix of the emitted list carries the target", "+  mixture. The same `selection.json` is mixture-correct at 6M or 24M tokens; only", "+  the depth reached into each register's ranking changes. `--overfill` controls how", "+  much tail is emitted, and provably does not perturb the 12M prefix (verified:", "+  `--overfill 4.0` reproduces the identical first 29,648 ids).", "+- **Different pool.** Nothing in the criterion is pool-specific. The background", "+  distribution is estimated from the pool itself, so the importance ratio adapts:", "+  a cleaner pool yields a flatter ratio and the quality floor does less work.", " - **What should not transfer.** Hard top-k beating stochastic resampling is a", "-  *small-budget* result. As the budget approaches the pool size, top-k must lose", "-  to resampling, because repeatedly reading the same narrow head of the ranking", "-  stops adding information. Expect the τ optimum to move away from 0 as the", "-  selection ratio grows past a few tens of percent.", "-- **Cost.** End-to-end curation is CPU-bound bag-of-words scoring: ~40 s over", "-  182k documents on 16 cores, plus a one-off tokenization pass. The optional", "-  neural Moore-Lewis cross-entropy-difference scorer (`score_lm.py`) costs ~5 min", "-  of GPU. Neither is a meaningful fraction of the training cost it improves.", "+  *small-budget* result. As the budget approaches the pool size, top-k must lose to", "+  resampling, since re-reading the same narrow head of the ranking stops adding", "+  information. Expect the τ optimum to move off 0 once the selection ratio passes a", "+  few tens of percent. Likewise, the neural Moore-Lewis scorer's failure is a", "+  *small-in-domain-sample* result; with tens of millions of in-domain tokens rather", "+  than one, the ordering against bag-of-words importance weighting should reverse.", "+- **Cost.** Curation is CPU-bound bag-of-words scoring: ~40 s over 182k documents", "+  on 16 cores, plus a one-off GPT-2 tokenization pass (~60 s) that also supplies", "+  exact token counts for budgeting. The optional neural scorer costs ~5 min of GPU", "+  and, here, was not worth it. Neither is a meaningful fraction of the training", "+  cost it is meant to improve.", "+", "+## Reproducing", "+", "+```bash", "+python tok_pool.py                     # one-off: tokenize pool -> cache/ (token counts)", "+python curate.py                        # -> submission/selection.json  (deterministic)", "+bash dev_eval.sh submission/selection.json", "+python mixture_report.py split          # per-register held-out splits for the mechanism test", "+```", "+", "+`curate.py` pins `PYTHONHASHSEED=0` (re-execing itself if needed) so the hashed", "+n-gram features, and therefore the selection, are bit-identical across runs."]}], "originalFile": "# Curating a raw web pool for a broad high-quality target\n\n## Hypothesis\n\nFor a fixed 12M-token budget drawn from a 200M-token raw web pool, held-out\nperplexity on a broad high-quality target is governed **first by register-mixture\nmatch and only second by document-level \"quality\"**.\n\nConcretely: the target is an equal-parts mixture of encyclopedic prose, general\nhigh-quality web prose, news, and technical Q&A. A random 6% slice of raw web\nunder-represents encyclopedic and technical-Q&A text badly. So the winning\ncriterion is *domain-matched importance weighting with an explicit per-register\ntoken quota*, where the quota is enforced in the **priority order** of the\nselection (so the budget-truncated prefix, not just the whole list, carries the\ntarget mixture). Quality filtering matters, but only as a junk floor: past that\npoint it is actively harmful, because every document a filter removes forces the\nquota to reach one document deeper into a worse-matching tail.\n\nPrediction, stated before the final measurement: mixture control buys more than\nany filter, and the filter-strength response curve is **non-monotonic** — an\ninterior optimum, with both over-filtering and no-filtering worse.\n\n## Mechanism (observable, not the final perplexity)\n\nThe mechanism is *distributional coverage of the target's registers*, so it is\nobservable directly in the data and in per-register loss — independent of the\nfinal score.\n\n1. **Register composition of the selection.** Classify pool documents by surface\n   form (HTML/code markup → technical Q&A; whitespace-before-punctuation and\n   encyclopedic phrasing → encyclopedic; else prose/news). A random 12M-token\n   selection is dominated by forum/blog/boilerplate prose and contains only a few\n   percent of markup-bearing technical Q&A. The curated selection's 12M prefix\n   should hold ≈30% encyclopedic / ≈45% prose+news / ≈25% technical Q&A, by\n   construction, and this is printed by `curate.py`.\n2. **Per-register held-out loss.** If the mechanism is mixture coverage, then\n   relative to the random baseline the *largest* loss reductions must appear on\n   the registers the random slice starves — technical Q&A and encyclopedic — and\n   the *smallest* on general web prose, which random selection already covers.\n   A global \"we just picked cleaner text\" story predicts a roughly uniform\n   improvement across registers instead.\n3. **Ablation ordering.** Removing mixture control (one global importance model,\n   everything else identical) should cost more than removing the quality floor\n   entirely. Observed on dev: no-mixture 411.9 vs no-filter 355.0 vs full 317.8\n   against a random baseline of 485.5 — mixture is the dominant term, as predicted.\n4. **Non-monotonic filter response.** Sweeping only the quality floor, holding\n   mixture and ranking fixed: strict (64.7k docs surviving) 362.7 → medium (96.7k)\n   333.9 → loose (117.9k) 324.5 → none (182.0k) 355.0. An interior optimum, as\n   predicted; the filter is a junk floor, not a quality maximiser.\n5. **Hard top-k beats stochastic resampling.** Adding Gumbel noise to the\n   importance ranking (DSIR-style resampling for diversity, τ=0.6) costs ~40 PPL\n   (358.4 vs 317.8). At 6% selection ratio the budget is small enough that\n   diversity within a register is not the binding constraint; match is.\n6. **Deduplication frees budget.** Near-duplicate removal (MinHash banding on\n   5-word shingles) drops 22.6k of 182.0k documents. Since duplicated tokens buy\n   no new information under a fixed budget, disabling dedup should cost real\n   perplexity: observed 345.7 vs 333.9 at otherwise identical settings.\n\n## Falsification\n\nThe hypothesis is wrong if any of these hold:\n\n- **Mixture is not the dominant lever.** If a single global importance model with\n  no quota (`--no_mix`) matches or beats the register-quota selection, the claim\n  fails. (Measured: 411.9 vs 317.8 — survives.)\n- **The filter response is monotonic.** If loosening the quality floor all the way\n  to *no filter* keeps improving perplexity, then \"quality floor with an interior\n  optimum\" is the wrong description and plain domain matching suffices.\n  (Measured: no-filter 355.0 is worse than loose 324.5 — survives.)\n- **Per-register gains are flat.** If the loss reduction versus the random\n  baseline is uniform across encyclopedic / prose / Q&A, the mechanism is not\n  mixture coverage but generic cleanliness, and the quota is incidental.\n- **Quota position is irrelevant.** If shuffling the selection (destroying the\n  interleaved priority order, so the 12M prefix mixture drifts from the target\n  mixture) does not hurt, then enforcing the quota *in priority order* is\n  unnecessary machinery.\n- **The dev optimum does not transfer.** If the tuned mixture/filter setting loses\n  to the untuned nominal 25/50/25 setting on the hidden held-out sample, the\n  selection was fitted to the dev sample rather than to the domain. This is the\n  live risk: the mixture response is flat (317.8 / 318.2 / 320.3 across\n  0.30–0.34 encyclopedic share), so the setting was chosen from a plateau rather\n  than a sharp peak, which limits but does not eliminate the exposure.\n\n## Transfer\n\n- **Different target mixture.** The pipeline takes the target's register mixture\n  as input, not as an assumption. Point `curate.py` at any target sample and the\n  same machinery (segment → classify register → per-register importance model →\n  quota → interleave) rebalances the selection. `--clusters K` replaces the\n  hand-written register classifier with k-means over the target documents, so it\n  runs on a target whose registers you cannot name in advance (measured 341.5 at\n  K=6 — worse than hand-labelled registers here, because 6 topic clusters over\n  2.7k target documents overfit the dev sample's *topics*, while surface-form\n  registers generalise).\n- **Different budget.** Because the quota is enforced by interleaving, *every*\n  prefix of the emitted list carries the target mixture. The same\n  `selection.json` is therefore mixture-correct at 6M or 24M tokens, not just at\n  12M; only the depth reached into each register's ranking changes.\n- **Different pool.** Nothing in the criterion is specific to this pool. The\n  background distribution is estimated from the pool itself, so the importance\n  ratio adapts: a cleaner pool yields a flatter ratio and the quality floor does\n  less work.\n- **What should not transfer.** Hard top-k beating stochastic resampling is a\n  *small-budget* result. As the budget approaches the pool size, top-k must lose\n  to resampling, because repeatedly reading the same narrow head of the ranking\n  stops adding information. Expect the τ optimum to move away from 0 as the\n  selection ratio grows past a few tens of percent.\n- **Cost.** End-to-end curation is CPU-bound bag-of-words scoring: ~40 s over\n  182k documents on 16 cores, plus a one-off tokenization pass. The optional\n  neural Moore-Lewis cross-entropy-difference scorer (`score_lm.py`) costs ~5 min\n  of GPU. Neither is a meaningful fraction of the training cost it improves.\n", "userModified": false}

Token indices sequence length is longer than the specified maximum sequence length for this model (9362 > 1024). Running this sequence through the model will result in indexing errors
packed 12000000 tokens from 30544 ids -> /tmp/official.npy
mlog_K_prose.txt  pending
mlog_K_qa.txt  pending
mlog_K_wiki.txt  pending
mlog_random_prose.txt  pending
mlog_random_qa.txt  pending
mlog_random_wiki.txt  pending
T=1429

[stdout]
Token indices sequence length is longer than the specified maximum sequence length for this model (9362 > 1024). Running this sequence through the model will result in indexing errors
packed 12000000 tokens from 30544 ids -> /tmp/official.npy
mlog_K_prose.txt  pending
mlog_K_qa.txt  pending
mlog_K_wiki.txt  pending
mlog_random_prose.txt  pending
mlog_random_qa.txt  pending
mlog_random_wiki.txt  pending
T=1429
[metadata] {"noOutputExpected": false}

official (12000000,) mine (12000000,) identical: True
T=1429

[stdout]
official (12000000,) mine (12000000,) identical: True
T=1429
[metadata] {"noOutputExpected": false}

ok
still reproduces exactly: True
total 234
drwxr-xr-x 1 root root    100 Aug  4 02:17 .
drwxr-xr-x 1 root root    420 Aug  4 02:16 ..
-rw-r--r-- 1 root root  21486 Aug  4 02:17 curate.py
-rw-r--r-- 1 root root 215308 Aug  4 02:07 selection.json
-rw-r--r-- 1 root root   1831 Aug  4 02:17 tok_pool.py
T=1369

[stdout]
ok
still reproduces exactly: True
total 234
drwxr-xr-x 1 root root    100 Aug  4 02:17 .
drwxr-xr-x 1 root root    420 Aug  4 02:16 ..
-rw-r--r-- 1 root root  21486 Aug  4 02:17 curate.py
-rw-r--r-- 1 root root 215308 Aug  4 02:07 selection.json
-rw-r--r-- 1 root root   1831 Aug  4 02:17 tok_pool.py
T=1369
[metadata] {"noOutputExpected": false}

mlog_K_prose  pending
mlog_K_qa  pending
mlog_K_wiki  pending
mlog_random_prose  pending
mlog_random_qa  pending
mlog_random_wiki  pending
T=1249

[stdout]
mlog_K_prose  pending
mlog_K_qa  pending
mlog_K_wiki  pending
mlog_random_prose  pending
mlog_random_qa  pending
mlog_random_wiki  pending
T=1249
[metadata] {"noOutputExpected": false}

/tmp/mech.done
MECHDONE
mlog_K_prose  HELDOUT_PPL 276.329
mlog_K_qa  HELDOUT_PPL 189.331
mlog_K_wiki  HELDOUT_PPL 654.212
mlog_random_prose  HELDOUT_PPL 386.612
mlog_random_qa  HELDOUT_PPL 362.979
mlog_random_wiki  HELDOUT_PPL 1007.672
T=1129

[stdout]
/tmp/mech.done
MECHDONE
mlog_K_prose  HELDOUT_PPL 276.329
mlog_K_qa  HELDOUT_PPL 189.331
mlog_K_wiki  HELDOUT_PPL 654.212
mlog_random_prose  HELDOUT_PPL 386.612
mlog_random_qa  HELDOUT_PPL 362.979
mlog_random_wiki  HELDOUT_PPL 1007.672
T=1129
[metadata] {"noOutputExpected": false}

46:**Measured** (held-out perplexity per register, same 12M budget, same frozen
T=1109

[stdout]
46:**Measured** (held-out perplexity per register, same 12M budget, same frozen
T=1109
[metadata] {"noOutputExpected": false}

[1]+  Done                    setsid nohup /tmp/runall.sh nom > /tmp/runall.log 2>&1 < /dev/null
patched
T=1069

[stdout]
[1]+  Done                    setsid nohup /tmp/runall.sh nom > /tmp/runall.log 2>&1 < /dev/null
patched
T=1069
[metadata] {"noOutputExpected": false}

HELDOUT_PPL 326.680
T=949

[stdout]
HELDOUT_PPL 326.680
T=949
[metadata] {"noOutputExpected": false}

NOMINAL_PPL left: False
len 12287
T=929

[stdout]
NOMINAL_PPL left: False
len 12287
T=929
[metadata] {"noOutputExpected": false}

=== deliverables ===
-rw-r--r-- 1 root root 12394 Aug  4 02:24 claim.md

submission/:
total 234
drwxr-xr-x 1 root root    100 Aug  4 02:17 .
drwxr-xr-x 1 root root    420 Aug  4 02:16 ..
-rw-r--r-- 1 root root  21486 Aug  4 02:17 curate.py
-rw-r--r-- 1 root root 215308 Aug  4 02:07 selection.json
-rw-r--r-- 1 root root   1831 Aug  4 02:17 tok_pool.py

=== selection validity ===
ids: 30544 | duplicates: 0 | all in pool: True
tokens available: 26.43M  (budget 12.00M, headroom 2.20x)
budget filled after 14582 ids

=== curate.py reproduces selection.json bit-exactly ===
[out] 12M prefix mixture: wiki=0.280, prose=0.360, qa=0.360
[out] -> /tmp/final_verify.json
identical: True
T=889

[stdout]
=== deliverables ===
-rw-r--r-- 1 root root 12394 Aug  4 02:24 claim.md

submission/:
total 234
drwxr-xr-x 1 root root    100 Aug  4 02:17 .
drwxr-xr-x 1 root root    420 Aug  4 02:16 ..
-rw-r--r-- 1 root root  21486 Aug  4 02:17 curate.py
-rw-r--r-- 1 root root 215308 Aug  4 02:07 selection.json
-rw-r--r-- 1 root root   1831 Aug  4 02:17 tok_pool.py

=== selection validity ===
ids: 30544 | duplicates: 0 | all in pool: True
tokens available: 26.43M  (budget 12.00M, headroom 2.20x)
budget filled after 14582 ids

=== curate.py reproduces selection.json bit-exactly ===
[out] 12M prefix mixture: wiki=0.280, prose=0.360, qa=0.360
[out] -> /tmp/final_verify.json
identical: True
T=889
[metadata] {"noOutputExpected": false}