claude-opus-5

3047

[stdout]
3047
[metadata] {"noOutputExpected": false}

The file /workspace/claim.md has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"type": "update", "filePath": "/workspace/claim.md", "content": "# Mixture quotas, not filter strength, decide small-budget data selection\n\n## Hypothesis\n\nFor a fixed, small token budget evaluated against a **mixture** target, held-out\nperplexity is governed not by *how on-target the average selected token is* but\nby *whether every component of the mixture is represented at all*. Concretely:\n\n> Ranking pool documents by a target/pool n-gram density ratio and taking the\n> top-k is **worse than random selection**, because top-k silently collapses the\n> budget onto whichever single register the pool holds the most high-ratio\n> material in. The *same* density ratio, applied per-register with a token quota\n> proportional to each register's share of the target, is **far better** than\n> random — and once the quota is enforced, *more* aggressive selection helps\n> monotonically.\n\nThe prediction is a non-monotonicity that a \"stronger filter = better data\" view\ndoes not anticipate: selectivity is harmful without mixture control and strongly\nhelpful with it. Selectivity and balance are not independent knobs — balance is\nwhat licenses selectivity.\n\n## Mechanism (and the observables it predicts, which are not the final score)\n\nThe budget is 12M tokens from a ~200M-token pool (~6%). A global ranking is\ntherefore a hard *subset* decision, not a reweighting: the trainer consumes a\nprefix and never sees the rest. A single per-token log density ratio is maximised\nby documents from whichever register is lexically closest to the pool-vs-target\ncontrast — here, contemporary news/narrative prose — so the global top-6% is\nnearly one register.\n\nCross-entropy on a mixture is the *average* of per-register cross-entropies, and\nthat average is dominated by its worst term: a register the model never saw costs\nfar more than an extra-well-served register saves. The mechanism is therefore\n**per-register starvation**, which predicts two observables independent of the\naggregate perplexity:\n\n**P1 — per-register cross-entropy spread.** The unbalanced selection should show\na *larger spread* across held-out registers than the balanced one: competitive or\neven *better* on the register it over-served, and catastrophically worse on the\nstarved ones. If it were uniformly worse on every register, the mechanism would\nbe wrong and plain text quality would be the explanation instead.\n\n**P2 — the objective anti-correlates with the outcome.** The unbalanced top-k\nselection should achieve a *higher* mean per-token target density ratio than the\nbalanced selection while scoring worse, and should *cover fewer* distinct target\nvocabulary types. Selection objective and outcome move in opposite directions.\n\n### P1: measured — confirmed\n\nHeld-out target split into 4 registers by clustering its own 256-token windows\n(independently of the K used for selection, so the split is not circular).\nPer-register perplexity, same frozen recipe, same 12M budget:\n\n| held-out register | global top-k (unbalanced) | mixture quota | ratio |\n|---|---|---|---|\n| reg2 — technical / web-dev prose | **1331.0** | **213.7** | 6.2× worse |\n| reg0 — encyclopedic (Wikipedia) | 785.5 | 696.8 | 1.13× worse |\n| reg1 — literary / quoted prose | 624.7 | 446.3 | 1.40× worse |\n| reg3 — narrative / entertainment news | **266.4** | 268.7 | **0.99× — better** |\n| aggregate | 544.9 | 329.7 | |\n\nExactly the predicted signature: the unbalanced selection *wins* on the single\nregister it over-served (266.4 vs 268.7) and loses by 6.2× on the register it\nstarved. Its spread is 266→1331 (5.0×) against 214→697 (3.3×) for the balanced\nselection. The diagnostic reproduces the frozen script's aggregate numbers to\nthree decimals (544.877 / 329.706), so it is the same training recipe.\n\n### P2: measured — confirmed\n\nMean per-token *global* target density ratio over the 12M-token prefix each\nselection actually trains on, and the fraction of target vocabulary types covered:\n\n| selection | mean log-ratio / token | target types covered | dev PPL |\n|---|---|---|---|\n| global top-k (unbalanced) | **−0.006 (best)** | **0.846 (worst)** | **544.9 (worst)** |\n| gate + dedup, shuffled | −0.181 | 0.856 | 422.9 |\n| random pool sample | −0.299 (worst) | 0.861 | 458.0 |\n| mixture quota, K=12 | −0.030 | **0.877 (best)** | 329.7 |\n| mixture quota, K=16 | −0.033 | **0.877 (best)** | **328.5 (best)** |\n\nThe selection that maximises the density-ratio objective is the worst model, and\nthe ranking by *coverage* separates the extremes that the ranking by *mean ratio*\ninverts.\n\n## Main result\n\nDev perplexity, frozen `train_nano.py`, 12M tokens, seed 1337, `multi_dev.npy`:\n\n| selection | mixture quota? | selectivity | dev PPL |\n|---|---|---|---|\n| random pool sample (baseline) | – | 100% | 458.0 |\n| global top-k by density ratio | **no** | ~6% | **544.9** |\n| per-register top-35%, shuffled | yes | 35% | 461.7 |\n| quality gate + dedup, shuffled | no | 63% | 422.9 / 399.3 |\n| per-register top-12%, shuffled | yes | 12% | 405.8 |\n| **per-register strict rank + quota, K=16 (submitted)** | **yes** | ~6% | **328.5** |\n\nThe two ~6%-selectivity rows differ *only* in whether the register quota is\nenforced: **544.9 vs 328.5**. Within the quota-enforced rows perplexity falls\nmonotonically as selectivity rises (461.7 → 405.8 → 328.5); without the quota,\nselectivity is catastrophic (458.0 → 544.9).\n\n**Noise floor.** Re-shuffling one fixed document set (gate + dedup) gives 422.9\nand 399.3 — the same data, different order. So run-to-run noise is ≈ ±12 PPL and\ngaps below ~25 PPL are not interpretable. The quota effect (216 PPL) and the\ngain over baseline (130 PPL) are far above it; the K sweep is not.\n\n**Flat and negative directions** (all within or near noise of the submitted\nconfig, so *not* claimed as improvements): K = 3/6/12/16/20/30 → 341.7 / 338.4 /\n329.7 / **328.5** / 325.9 / 332.6. K=16 is the plateau midpoint, chosen over the\nnoisy argmin (K=20) deliberately — the hidden target is a different sample, and\npicking the dev argmin off a flat, noisy curve is fitting the dev set.\nClearly *worse*: strict C4-style prose gates (385.7 at K=12; 488.5 without the\nquota), no gate beyond a length floor (340.5), and length shrinkage in the\nper-token normaliser (347.2). The permissive gate is load-bearing in both\ndirections: tightening it discards number-dense news and markup-bearing\ntechnical text the target contains, loosening it lets navigation boilerplate —\nwhich scores *well* on a density ratio — into the budget.\n\n## Falsification\n\nThe claim is wrong if any of these hold:\n\n1. **Balance is not the active ingredient.** Re-order the submitted selection so\n   one register fills the whole 12M prefix, holding gate, dedup and scoring\n   fixed. If perplexity does not degrade toward ~545, the win came from\n   something else. (The 544.9 vs 328.5 pair is exactly this experiment run in\n   the opposite direction, so this is already largely discharged.)\n2. **P1 fails**: the unbalanced selection is worse on *every* register instead of\n   winning on one and collapsing on another. Refuted above (reg3: 266.4 vs 268.7).\n3. **P2 fails**: the better model also has the higher mean density ratio, so no\n   objective/outcome divergence exists. Refuted above.\n4. **Selectivity is not monotone under balance**: pushing selectivity further\n   under a quota reverses and raises perplexity, implying an interior optimum\n   rather than \"balance licenses selectivity\". Only tested down to ~6%; a much\n   smaller budget could still expose an optimum.\n5. **The clusters carry no register information.** Replace the k-means registers\n   with K *random* partitions of the target and re-run. If random partitions\n   reproduce the gain, the effect is an ensembling artifact of averaging K scores\n   rather than mixture matching. This is the strongest untested falsifier — the\n   flatness of the K sweep (K=3 already gets 341.7) is weak evidence that coarse\n   balance is most of the effect, and a random-partition control would separate\n   \"balance across real registers\" from \"any K-way score averaging\".\n\n## Transfer\n\n- **Any budget-constrained pretraining mix against a known multi-domain\n  evaluation.** The recipe — estimate a density ratio per evaluation component,\n  then fill the budget by component quota rather than global rank — applies\n  directly to the DSIR/DoReMi setting and needs only a decodable sample of the\n  target, no labels and no trained scorer.\n- **The failure mode transfers further than the fix.** Any pipeline that ranks\n  documents by a single scalar \"quality\" or \"similarity\" score and keeps a top\n  fraction is exposed: the tighter the budget, the more a scalar ranking\n  silently becomes a *domain* filter. Retrieval corpus construction, fine-tuning\n  set selection, and active-learning pools all have this shape.\n- **Practical rule.** When the keep-fraction is small relative to the pool,\n  report the *composition* of what was kept, not just the mean score of what was\n  kept. Mean score improving while evaluation degrades is the signature of this\n  bug, and it is invisible unless the evaluation is broken out per component.\n- **Also transferable: measure the noise floor before tuning.** Re-shuffling one\n  fixed selection moved perplexity by 23 points here, which is larger than the\n  entire K sweep. Without that measurement, six of these ablations would have\n  been over-read as real effects.\n- **Boundaries.** The effect should weaken as the budget approaches the pool size\n  (top-k stops being a subset decision) and as the target becomes\n  single-register (nothing left to starve), and strengthen for smaller budgets\n  and more heterogeneous targets. It also assumes the pool actually *contains*\n  each target register: the pool here held almost no markup-bearing Q&A (203\n  docs) or spaced-punctuation Wikipedia (764 docs), which is why reg0 stays at\n  ~697 perplexity for every selection tried — a quota cannot buy coverage the\n  pool does not have.\n\n## Reproducing\n\n```bash\npython /workspace/submission/curate.py        # defaults = submitted config\nbash /workspace/dev_eval.sh                   # frozen trainer, dev target\n```\nEvery ablation above is one env var on `curate.py`: `GATE={min,loose,strict}`,\n`ORDER={mixture,shuffle}`, `K=<int>`, `KEEP_FRAC=<float>`, `W_BONUS=<float>`,\n`SHRINK=<float>`. Diagnostics: `diag_registers.py` (P1), `diag_ratio.py` (P2).\n", "structuredPatch": [{"oldStart": 1, "oldLines": 111, "newStart": 1, "newLines": 182, "lines": ["-# Mixture-matched importance selection beats both random data and greedy on-target data", "+# Mixture quotas, not filter strength, decide small-budget data selection", " ", " ## Hypothesis", " ", "-For a fixed, small token budget against a **mixture** target, the quantity that", "-determines held-out perplexity is not *how on-target the average selected token", "-is*, but *whether every component of the mixture is represented*. Concretely:", "+For a fixed, small token budget evaluated against a **mixture** target, held-out", "+perplexity is governed not by *how on-target the average selected token is* but", "+by *whether every component of the mixture is represented at all*. Concretely:", " ", " > Ranking pool documents by a target/pool n-gram density ratio and taking the", "-> top-k is **worse than random selection**, because top-k collapses the budget", "-> onto whichever single register the pool holds the most high-ratio material in.", "-> The *same* density ratio, applied per-register with a token quota proportional", "-> to each register's share of the target, is **much better** than random — and", "-> once the quota is enforced, *more* aggressive selection monotonically helps.", "+> top-k is **worse than random selection**, because top-k silently collapses the", "+> budget onto whichever single register the pool holds the most high-ratio", "+> material in. The *same* density ratio, applied per-register with a token quota", "+> proportional to each register's share of the target, is **far better** than", "+> random — and once the quota is enforced, *more* aggressive selection helps", "+> monotonically.", " ", "-The prediction is a non-monotonicity that a \"better filter = better data\" view", "-does not anticipate: selectivity is harmful without mixture control and", "-strongly helpful with it. Selectivity and balance are not independent knobs;", "-balance is what licenses selectivity.", "+The prediction is a non-monotonicity that a \"stronger filter = better data\" view", "+does not anticipate: selectivity is harmful without mixture control and strongly", "+helpful with it. Selectivity and balance are not independent knobs — balance is", "+what licenses selectivity.", " ", "-## Mechanism (and the observable it predicts — not the final perplexity)", "+## Mechanism (and the observables it predicts, which are not the final score)", " ", "-The budget is 12M tokens out of a ~200M-token pool, i.e. only ~6%. A global", "-ranking is therefore a *hard subset* decision, not a reweighting: the trainer", "-consumes a prefix of the list and never sees the rest. A single per-token log", "-density ratio is maximised by documents from whichever register happens to be", "-lexically closest to the pool-vs-target contrast — here, contemporary news", "-prose. So the global top-6% is nearly a single register.", "+The budget is 12M tokens from a ~200M-token pool (~6%). A global ranking is", "+therefore a hard *subset* decision, not a reweighting: the trainer consumes a", "+prefix and never sees the rest. A single per-token log density ratio is maximised", "+by documents from whichever register is lexically closest to the pool-vs-target", "+contrast — here, contemporary news/narrative prose — so the global top-6% is", "+nearly one register.", " ", "-Cross-entropy on an equal-parts mixture is the *average* of per-register", "-cross-entropies, and that average is dominated by its worst term: a register the", "-model has never seen costs unboundedly more than an extra-well-served register", "-saves. Hence the mechanism is **per-register starvation**, and it predicts an", "-observable that is not the aggregate score:", "+Cross-entropy on a mixture is the *average* of per-register cross-entropies, and", "+that average is dominated by its worst term: a register the model never saw costs", "+far more than an extra-well-served register saves. The mechanism is therefore", "+**per-register starvation**, which predicts two observables independent of the", "+aggregate perplexity:", " ", "-> **Observable P1 (per-register cross-entropy spread).** Split the held-out", "-> target into its registers. The globally-ranked (unbalanced) selection should", "-> show a *larger spread* across registers than the quota-balanced selection —", "-> competitive or even better CE on the register it over-served, and markedly", "-> worse CE on the starved registers. The quota-balanced selection should lose", "-> slightly on the over-served register and win by much more on the starved ones.", "-> If the unbalanced selection is uniformly worse on every register, the", "-> mechanism is wrong and something else (e.g. plain text quality) explains it.", "+**P1 — per-register cross-entropy spread.** The unbalanced selection should show", "+a *larger spread* across held-out registers than the balanced one: competitive or", "+even *better* on the register it over-served, and catastrophically worse on the", "+starved ones. If it were uniformly worse on every register, the mechanism would", "+be wrong and plain text quality would be the explanation instead.", " ", "-> **Observable P2 (average density ratio anti-correlates with score).** The", "-> unbalanced top-k selection should achieve a *higher* mean per-token target", "-> density ratio than the quota-balanced selection, while scoring worse. The", "-> selection objective and the outcome move in opposite directions.", "+**P2 — the objective anti-correlates with the outcome.** The unbalanced top-k", "+selection should achieve a *higher* mean per-token target density ratio than the", "+balanced selection while scoring worse, and should *cover fewer* distinct target", "+vocabulary types. Selection objective and outcome move in opposite directions.", " ", "-### Measured", "+### P1: measured — confirmed", " ", "-Dev perplexity (frozen `train_nano.py`, 12M tokens, seed 1337, `multi_dev.npy`):", "+Held-out target split into 4 registers by clustering its own 256-token windows", "+(independently of the K used for selection, so the split is not circular).", "+Per-register perplexity, same frozen recipe, same 12M budget:", " ", "+| held-out register | global top-k (unbalanced) | mixture quota | ratio |", "+|---|---|---|---|", "+| reg2 — technical / web-dev prose | **1331.0** | **213.7** | 6.2× worse |", "+| reg0 — encyclopedic (Wikipedia) | 785.5 | 696.8 | 1.13× worse |", "+| reg1 — literary / quoted prose | 624.7 | 446.3 | 1.40× worse |", "+| reg3 — narrative / entertainment news | **266.4** | 268.7 | **0.99× — better** |", "+| aggregate | 544.9 | 329.7 | |", "+", "+Exactly the predicted signature: the unbalanced selection *wins* on the single", "+register it over-served (266.4 vs 268.7) and loses by 6.2× on the register it", "+starved. Its spread is 266→1331 (5.0×) against 214→697 (3.3×) for the balanced", "+selection. The diagnostic reproduces the frozen script's aggregate numbers to", "+three decimals (544.877 / 329.706), so it is the same training recipe.", "+", "+### P2: measured — confirmed", "+", "+Mean per-token *global* target density ratio over the 12M-token prefix each", "+selection actually trains on, and the fraction of target vocabulary types covered:", "+", "+| selection | mean log-ratio / token | target types covered | dev PPL |", "+|---|---|---|---|", "+| global top-k (unbalanced) | **−0.006 (best)** | **0.846 (worst)** | **544.9 (worst)** |", "+| gate + dedup, shuffled | −0.181 | 0.856 | 422.9 |", "+| random pool sample | −0.299 (worst) | 0.861 | 458.0 |", "+| mixture quota, K=12 | −0.030 | **0.877 (best)** | 329.7 |", "+| mixture quota, K=16 | −0.033 | **0.877 (best)** | **328.5 (best)** |", "+", "+The selection that maximises the density-ratio objective is the worst model, and", "+the ranking by *coverage* separates the extremes that the ranking by *mean ratio*", "+inverts.", "+", "+## Main result", "+", "+Dev perplexity, frozen `train_nano.py`, 12M tokens, seed 1337, `multi_dev.npy`:", "+", " | selection | mixture quota? | selectivity | dev PPL |", " |---|---|---|---|", "-| random pool sample (baseline) | – | none (100%) | **458.0** |", "-| quality gate + dedup, shuffled | no | 63% | 422.9 |", "+| random pool sample (baseline) | – | 100% | 458.0 |", " | global top-k by density ratio | **no** | ~6% | **544.9** |", " | per-register top-35%, shuffled | yes | 35% | 461.7 |", "+| quality gate + dedup, shuffled | no | 63% | 422.9 / 399.3 |", " | per-register top-12%, shuffled | yes | 12% | 405.8 |", "-| per-register strict rank + quota | **yes** | ~6% | **338.4** |", "+| **per-register strict rank + quota, K=16 (submitted)** | **yes** | ~6% | **328.5** |", " ", "-The two 6%-selectivity rows differ *only* in whether the register quota is", "-enforced: 544.9 vs 338.4. That is the claim. And within the quota-enforced", "-rows, PPL falls monotonically as selectivity rises (461.7 → 405.8 → 338.4),", "-whereas without the quota, selectivity is catastrophic (458.0 → 544.9).", "+The two ~6%-selectivity rows differ *only* in whether the register quota is", "+enforced: **544.9 vs 328.5**. Within the quota-enforced rows perplexity falls", "+monotonically as selectivity rises (461.7 → 405.8 → 328.5); without the quota,", "+selectivity is catastrophic (458.0 → 544.9).", " ", "-P1/P2 results are reported in `## Mechanism check` below.", "+**Noise floor.** Re-shuffling one fixed document set (gate + dedup) gives 422.9", "+and 399.3 — the same data, different order. So run-to-run noise is ≈ ±12 PPL and", "+gaps below ~25 PPL are not interpretable. The quota effect (216 PPL) and the", "+gain over baseline (130 PPL) are far above it; the K sweep is not.", " ", "+**Flat and negative directions** (all within or near noise of the submitted", "+config, so *not* claimed as improvements): K = 3/6/12/16/20/30 → 341.7 / 338.4 /", "+329.7 / **328.5** / 325.9 / 332.6. K=16 is the plateau midpoint, chosen over the", "+noisy argmin (K=20) deliberately — the hidden target is a different sample, and", "+picking the dev argmin off a flat, noisy curve is fitting the dev set.", "+Clearly *worse*: strict C4-style prose gates (385.7 at K=12; 488.5 without the", "+quota), no gate beyond a length floor (340.5), and length shrinkage in the", "+per-token normaliser (347.2). The permissive gate is load-bearing in both", "+directions: tightening it discards number-dense news and markup-bearing", "+technical text the target contains, loosening it lets navigation boilerplate —", "+which scores *well* on a density ratio — into the budget.", "+", " ## Falsification", " ", " The claim is wrong if any of these hold:", " ", "-1. **Balance is not the active ingredient.** Take the quota-balanced selection", "-   and re-order it so one register fills the whole 12M prefix, holding the gate,", "-   dedup, and scoring function fixed. If perplexity does *not* degrade toward", "-   the ~545 range, the win came from something other than mixture matching.", "-2. **P1 fails.** If the unbalanced selection is worse on *every* register rather", "-   than showing a large spread, starvation is not the mechanism.", "-3. **P2 fails.** If the better-scoring selection also has the higher mean", "-   density ratio, then the ratio is simply a good quality score and no", "-   \"objective vs outcome divergence\" exists.", "-4. **Selectivity is not monotone under balance.** If pushing selectivity further", "-   under a quota (finer registers, smaller kept fraction) reverses and *raises*", "-   perplexity, then the \"balance licenses selectivity\" statement is too strong", "-   and there is an interior optimum instead.", "-5. **Cluster count is doing the work.** If replacing the discovered k-means", "-   registers with K arbitrary random partitions of the target reproduces the", "-   gain, then the clusters carry no register information and the effect is just", "-   an ensembling/averaging artifact of using K scores instead of one.", "+1. **Balance is not the active ingredient.** Re-order the submitted selection so", "+   one register fills the whole 12M prefix, holding gate, dedup and scoring", "+   fixed. If perplexity does not degrade toward ~545, the win came from", "+   something else. (The 544.9 vs 328.5 pair is exactly this experiment run in", "+   the opposite direction, so this is already largely discharged.)", "+2. **P1 fails**: the unbalanced selection is worse on *every* register instead of", "+   winning on one and collapsing on another. Refuted above (reg3: 266.4 vs 268.7).", "+3. **P2 fails**: the better model also has the higher mean density ratio, so no", "+   objective/outcome divergence exists. Refuted above.", "+4. **Selectivity is not monotone under balance**: pushing selectivity further", "+   under a quota reverses and raises perplexity, implying an interior optimum", "+   rather than \"balance licenses selectivity\". Only tested down to ~6%; a much", "+   smaller budget could still expose an optimum.", "+5. **The clusters carry no register information.** Replace the k-means registers", "+   with K *random* partitions of the target and re-run. If random partitions", "+   reproduce the gain, the effect is an ensembling artifact of averaging K scores", "+   rather than mixture matching. This is the strongest untested falsifier — the", "+   flatness of the K sweep (K=3 already gets 341.7) is weak evidence that coarse", "+   balance is most of the effect, and a random-partition control would separate", "+   \"balance across real registers\" from \"any K-way score averaging\".", " ", " ## Transfer", " ", " - **Any budget-constrained pretraining mix against a known multi-domain", "   evaluation.** The recipe — estimate a density ratio per evaluation component,", "-  then fill the budget by component quota rather than by global rank — applies", "-  directly to domain-mixture selection (the DoReMi/DSIR setting) and needs only", "-  a decodable sample of the target, no labels.", "-- **The failure mode transfers more broadly than the fix.** Any pipeline that", "-  ranks documents by a single scalar \"quality\" or \"similarity\" score and then", "-  keeps a top fraction is exposed to this: the tighter the budget, the more a", "-  scalar ranking silently becomes a domain filter. Retrieval corpora, fine-tune", "-  set construction, and active-learning pools all have this shape.", "+  then fill the budget by component quota rather than global rank — applies", "+  directly to the DSIR/DoReMi setting and needs only a decodable sample of the", "+  target, no labels and no trained scorer.", "+- **The failure mode transfers further than the fix.** Any pipeline that ranks", "+  documents by a single scalar \"quality\" or \"similarity\" score and keeps a top", "+  fraction is exposed: the tighter the budget, the more a scalar ranking", "+  silently becomes a *domain* filter. Retrieval corpus construction, fine-tuning", "+  set selection, and active-learning pools all have this shape.", " - **Practical rule.** When the keep-fraction is small relative to the pool,", "-  always report the *composition* of what was kept, not just the mean score of", "-  what was kept. A monotone improvement in mean score with a degrading", "-  evaluation is the signature of this bug.", "-- **Boundary.** This should weaken as the budget approaches the pool size (top-k", "-  stops being a subset decision) and as the target becomes single-register", "-  (there is nothing to starve). It should strengthen for smaller budgets and", "-  more heterogeneous targets.", "+  report the *composition* of what was kept, not just the mean score of what was", "+  kept. Mean score improving while evaluation degrades is the signature of this", "+  bug, and it is invisible unless the evaluation is broken out per component.", "+- **Also transferable: measure the noise floor before tuning.** Re-shuffling one", "+  fixed selection moved perplexity by 23 points here, which is larger than the", "+  entire K sweep. Without that measurement, six of these ablations would have", "+  been over-read as real effects.", "+- **Boundaries.** The effect should weaken as the budget approaches the pool size", "+  (top-k stops being a subset decision) and as the target becomes", "+  single-register (nothing left to starve), and strengthen for smaller budgets", "+  and more heterogeneous targets. It also assumes the pool actually *contains*", "+  each target register: the pool here held almost no markup-bearing Q&A (203", "+  docs) or spaced-punctuation Wikipedia (764 docs), which is why reg0 stays at", "+  ~697 perplexity for every selection tried — a quota cannot buy coverage the", "+  pool does not have.", "+", "+## Reproducing", "+", "+```bash", "+python /workspace/submission/curate.py        # defaults = submitted config", "+bash /workspace/dev_eval.sh                   # frozen trainer, dev target", "+```", "+Every ablation above is one env var on `curate.py`: `GATE={min,loose,strict}`,", "+`ORDER={mixture,shuffle}`, `K=<int>`, `KEEP_FRAC=<float>`, `W_BONUS=<float>`,", "+`SHRINK=<float>`. Diagnostics: `diag_registers.py` (P1), `diag_ratio.py` (P2)."]}], "originalFile": "# Mixture-matched importance selection beats both random data and greedy on-target data\n\n## Hypothesis\n\nFor a fixed, small token budget against a **mixture** target, the quantity that\ndetermines held-out perplexity is not *how on-target the average selected token\nis*, but *whether every component of the mixture is represented*. Concretely:\n\n> Ranking pool documents by a target/pool n-gram density ratio and taking the\n> top-k is **worse than random selection**, because top-k collapses the budget\n> onto whichever single register the pool holds the most high-ratio material in.\n> The *same* density ratio, applied per-register with a token quota proportional\n> to each register's share of the target, is **much better** than random — and\n> once the quota is enforced, *more* aggressive selection monotonically helps.\n\nThe prediction is a non-monotonicity that a \"better filter = better data\" view\ndoes not anticipate: selectivity is harmful without mixture control and\nstrongly helpful with it. Selectivity and balance are not independent knobs;\nbalance is what licenses selectivity.\n\n## Mechanism (and the observable it predicts — not the final perplexity)\n\nThe budget is 12M tokens out of a ~200M-token pool, i.e. only ~6%. A global\nranking is therefore a *hard subset* decision, not a reweighting: the trainer\nconsumes a prefix of the list and never sees the rest. A single per-token log\ndensity ratio is maximised by documents from whichever register happens to be\nlexically closest to the pool-vs-target contrast — here, contemporary news\nprose. So the global top-6% is nearly a single register.\n\nCross-entropy on an equal-parts mixture is the *average* of per-register\ncross-entropies, and that average is dominated by its worst term: a register the\nmodel has never seen costs unboundedly more than an extra-well-served register\nsaves. Hence the mechanism is **per-register starvation**, and it predicts an\nobservable that is not the aggregate score:\n\n> **Observable P1 (per-register cross-entropy spread).** Split the held-out\n> target into its registers. The globally-ranked (unbalanced) selection should\n> show a *larger spread* across registers than the quota-balanced selection —\n> competitive or even better CE on the register it over-served, and markedly\n> worse CE on the starved registers. The quota-balanced selection should lose\n> slightly on the over-served register and win by much more on the starved ones.\n> If the unbalanced selection is uniformly worse on every register, the\n> mechanism is wrong and something else (e.g. plain text quality) explains it.\n\n> **Observable P2 (average density ratio anti-correlates with score).** The\n> unbalanced top-k selection should achieve a *higher* mean per-token target\n> density ratio than the quota-balanced selection, while scoring worse. The\n> selection objective and the outcome move in opposite directions.\n\n### Measured\n\nDev perplexity (frozen `train_nano.py`, 12M tokens, seed 1337, `multi_dev.npy`):\n\n| selection | mixture quota? | selectivity | dev PPL |\n|---|---|---|---|\n| random pool sample (baseline) | – | none (100%) | **458.0** |\n| quality gate + dedup, shuffled | no | 63% | 422.9 |\n| global top-k by density ratio | **no** | ~6% | **544.9** |\n| per-register top-35%, shuffled | yes | 35% | 461.7 |\n| per-register top-12%, shuffled | yes | 12% | 405.8 |\n| per-register strict rank + quota | **yes** | ~6% | **338.4** |\n\nThe two 6%-selectivity rows differ *only* in whether the register quota is\nenforced: 544.9 vs 338.4. That is the claim. And within the quota-enforced\nrows, PPL falls monotonically as selectivity rises (461.7 → 405.8 → 338.4),\nwhereas without the quota, selectivity is catastrophic (458.0 → 544.9).\n\nP1/P2 results are reported in `## Mechanism check` below.\n\n## Falsification\n\nThe claim is wrong if any of these hold:\n\n1. **Balance is not the active ingredient.** Take the quota-balanced selection\n   and re-order it so one register fills the whole 12M prefix, holding the gate,\n   dedup, and scoring function fixed. If perplexity does *not* degrade toward\n   the ~545 range, the win came from something other than mixture matching.\n2. **P1 fails.** If the unbalanced selection is worse on *every* register rather\n   than showing a large spread, starvation is not the mechanism.\n3. **P2 fails.** If the better-scoring selection also has the higher mean\n   density ratio, then the ratio is simply a good quality score and no\n   \"objective vs outcome divergence\" exists.\n4. **Selectivity is not monotone under balance.** If pushing selectivity further\n   under a quota (finer registers, smaller kept fraction) reverses and *raises*\n   perplexity, then the \"balance licenses selectivity\" statement is too strong\n   and there is an interior optimum instead.\n5. **Cluster count is doing the work.** If replacing the discovered k-means\n   registers with K arbitrary random partitions of the target reproduces the\n   gain, then the clusters carry no register information and the effect is just\n   an ensembling/averaging artifact of using K scores instead of one.\n\n## Transfer\n\n- **Any budget-constrained pretraining mix against a known multi-domain\n  evaluation.** The recipe — estimate a density ratio per evaluation component,\n  then fill the budget by component quota rather than by global rank — applies\n  directly to domain-mixture selection (the DoReMi/DSIR setting) and needs only\n  a decodable sample of the target, no labels.\n- **The failure mode transfers more broadly than the fix.** Any pipeline that\n  ranks documents by a single scalar \"quality\" or \"similarity\" score and then\n  keeps a top fraction is exposed to this: the tighter the budget, the more a\n  scalar ranking silently becomes a domain filter. Retrieval corpora, fine-tune\n  set construction, and active-learning pools all have this shape.\n- **Practical rule.** When the keep-fraction is small relative to the pool,\n  always report the *composition* of what was kept, not just the mean score of\n  what was kept. A monotone improvement in mean score with a degrading\n  evaluation is the signature of this bug.\n- **Boundary.** This should weaken as the budget approaches the pool size (top-k\n  stops being a subset decision) and as the target becomes single-register\n  (there is nothing to starve). It should strengthen for smaller budgets and\n  more heterogeneous targets.\n", "userModified": false}

Token indices sequence length is longer than the specified maximum sequence length for this model (4313 > 1024). Running this sequence through the model will result in indexing errors
packed 12000000 tokens from 44986 ids -> /tmp/dev_train.npy
--- t:
2997
total 330
drwxr-xr-x 1 root root    200 Aug  4 01:48 .
drwxr-xr-x 1 root root    360 Aug  4 01:50 ..
-rw-r--r-- 1 root root  19451 Aug  4 01:47 curate.py
-rw-r--r-- 1 root root 317582 Aug  4 01:48 selection.json

[stdout]
Token indices sequence length is longer than the specified maximum sequence length for this model (4313 > 1024). Running this sequence through the model will result in indexing errors
packed 12000000 tokens from 44986 ids -> /tmp/dev_train.npy
--- t:
2997
total 330
drwxr-xr-x 1 root root    200 Aug  4 01:48 .
drwxr-xr-x 1 root root    360 Aug  4 01:50 ..
-rw-r--r-- 1 root root  19451 Aug  4 01:47 curate.py
-rw-r--r-- 1 root root 317582 Aug  4 01:48 selection.json
[metadata] {"noOutputExpected": false}

patched
      cluster shares: [0.063, 0.065, 0.05, 0.05, 0.065, 0.053, 0.065, 0.05, 0.069, 0.066, 0.066, 0.063, 0.066, 0.065, 0.071, 0.074]
      emitted 43941 ids (~40.0M est tokens)

[stdout]
patched
      cluster shares: [0.063, 0.065, 0.05, 0.05, 0.065, 0.053, 0.065, 0.05, 0.069, 0.066, 0.066, 0.063, 0.066, 0.065, 0.071, 0.074]
      emitted 43941 ids (~40.0M est tokens)
[metadata] {"noOutputExpected": false}

[1]+  Done                    TOKENIZERS_PARALLELISM=false nohup python batch_eval.py /tmp/sel_rp16.json > /tmp/batch6.log 2>&1
HELDOUT_PPL 328.457  loss 5.7944  train_s 56  /tmp/dev_train.npy
2786

[stdout]
[1]+  Done                    TOKENIZERS_PARALLELISM=false nohup python batch_eval.py /tmp/sel_rp16.json > /tmp/batch6.log 2>&1
HELDOUT_PPL 328.457  loss 5.7944  train_s 56  /tmp/dev_train.npy
2786
[metadata] {"noOutputExpected": false}

pool loaded
Token indices sequence length is longer than the specified maximum sequence length for this model (1113 > 1024). Running this sequence through the model will result in indexing errors
packed sel_rp16 12000000
RESULT sel_rp16 HELDOUT_PPL 381.519  loss 5.9442  train_s 55  /tmp/tr_sel_rp16.npy

[stdout]
pool loaded
Token indices sequence length is longer than the specified maximum sequence length for this model (1113 > 1024). Running this sequence through the model will result in indexing errors
packed sel_rp16 12000000
RESULT sel_rp16 HELDOUT_PPL 381.519  loss 5.9442  train_s 55  /tmp/tr_sel_rp16.npy
[metadata] {"noOutputExpected": false}

updated
167

[stdout]
updated
167
[metadata] {"noOutputExpected": false}

ok
=== deliverables ===
selection.json : 44986 ids | unique: True | in-range: True | type ok: True
177
460 /workspace/submission/curate.py
2746

[stdout]
ok
=== deliverables ===
selection.json : 44986 ids | unique: True | in-range: True | type ok: True
177
460 /workspace/submission/curate.py
2746
[metadata] {"noOutputExpected": false}

emitted 44986 ids (~40.0M est tokens)
wrote /workspace/submission/selection.json
final curate.py reproduces submitted selection exactly: True

[stdout]
      emitted 44986 ids (~40.0M est tokens)
wrote /workspace/submission/selection.json
final curate.py reproduces submitted selection exactly: True
[metadata] {"noOutputExpected": false}