claude-opus-5

training runs completed: 30
distinct candidate selections: 26 (+1 submission)
per-register diagnostics: 5
work/run_dgreedy.log
work/run_final.log
work/run_diag.log
work/run_sub.log
work/run_qg.log

[stdout]
training runs completed: 30
distinct candidate selections: 26 (+1 submission)
per-register diagnostics: 5
work/run_dgreedy.log
work/run_final.log
work/run_diag.log
work/run_sub.log
work/run_qg.log
[metadata] {"noOutputExpected": false}

The file /workspace/claim.md has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/claim.md", "oldString": "4. **The encyclopedic register stays the worst one no matter what.** Its\n   convention is nearly absent from the pool (0 documents contain ` @-@ `), so it\n   remained the highest-loss quarter in every one of the 20 selections I trained.\n   That is a ceiling imposed by the pool, not by the selector.", "newString": "4. **The encyclopedic register stays the worst one no matter what.** Its\n   convention is nearly absent from the pool (0 documents contain ` @-@ `), so it\n   remained the highest-loss quarter in all five selections where I measured\n   per-register loss (quality-gated random, NB-balanced, two greedy variants, and\n   the submitted one), never dropping below 6.26 nats while the other three\n   registers reached 4.97–5.59. That is a ceiling imposed by the pool, not the\n   selector: no selection can teach a convention the pool does not contain.", "originalFile": "# Curating a raw web pool by target-distribution matching\n\nSubmitted selection: `submission/selection.json` (18,418 ids, 15.0M tokens; the\nfirst 15,285 fill the 12M budget), produced by `submission/curate.py`.\n**Dev perplexity 267.7 vs 485.5 for a random selection** (frozen recipe, seed 1337).\n\n## Hypothesis\n\nUnder a token budget far smaller than the pool (12M of 200M), held-out perplexity\non a broad multi-domain target is governed mainly by **how closely the training\nsubset's token distribution matches the target's**, and only secondarily by\ngeneric \"document quality\". Concretely:\n\n> Greedily choosing documents to minimise the unigram cross-entropy\n> `H(p_target, p_selection)` — the same functional the metric evaluates, at the\n> highest order estimable from the 1M-token disclosed target — beats random\n> selection, beats quality-filtered random selection, and beats single-axis\n> \"pick the best-looking documents\" ranking.\n\nThe corollary that makes this non-trivial: the pool is *not* short of readable\nprose. It is short of prose in the target's **registers and surface conventions**,\nand a budget-constrained selector must buy those deliberately.\n\n## Mechanism\n\nHeld-out loss is a mean over windows of four equal registers (recovered by\ndecoding `multi_dev.npy`: WikiText-style encyclopedic / clean web prose /\nnewswire / StackExchange HTML Q&A). Mean loss is additive across registers, so\nthe **highest-loss register dominates the score**, and a selection that leaves one\nregister unrepresented pays disproportionately.\n\nMatching `p_target` allocates budget by each register's share of the *target*\nrather than its share of the *pool*. The importance ratio\n`r(v) = p_target(v)/p_selection(v)` is recomputed as the selection grows, so once\na register is covered its tokens stop being attractive and the selector moves on —\ncoverage is self-balancing rather than hand-weighted.\n\n**Predicted observables (not the final perplexity), all measured:**\n\n1. **The proxy objective tracks the real one — inside a fixed quality gate.** The\n   selection reduces unigram `H(p_target, p_selection)` from 8.078 nats (pool\n   background = do nothing) to **7.854 nats**. Across six selections that share\n   the same target estimate and the same quality gate, the proxy predicts dev\n   perplexity at **Pearson 0.96** (Spearman 0.77, range 267.7–281.8 PPL).\n   I used this prospectively: two variants (a 10× stronger background prior and a\n   tighter quality gate) were built, their proxies read off *before* training, and\n   both were predicted to lose. They did, monotonically —\n   proxy 7.854 → 7.868 → 7.888 gave PPL 267.7 → 275.9 → 281.8.\n   Also, smoothing `p_target` toward the pool background achieves a larger proxy\n   reduction than smoothing toward uniform (0.224 vs 0.195 nats) and a lower PPL\n   (272.9 vs 279.3, both at min-length 300); uniform smoothing spreads ~8% of\n   target mass over ~25k token types the target never emits, and greedy burns\n   budget chasing that noise.\n\n   **Where the proxy fails, and why the gate is not optional:** relaxing the\n   quality gate produced the *lowest* proxy of all runs (7.820) but a *worse* PPL\n   (269.2 vs 267.7). Boilerplate can match unigram frequencies cheaply while being\n   poor training data, so the proxy is only a valid predictor within a fixed\n   candidate set. Neither component substitutes for the other: the gate defines\n   what is admissible, the proxy orders what is admitted. The proxy also does not\n   resolve differences inside a ~0.005-nat cluster (min-length 100/150/200 span\n   0.004 nats and 2 PPL, out of order).\n2. **Gains land on the register that was starved, not uniformly.** Per-register\n   held-out loss (nats), quality-filtered random → submitted selection:\n\n   | register | qgate-random | submitted | Δ |\n   |---|---|---|---|\n   | technical Q&A | 5.777 | **4.973** | −0.804 |\n   | encyclopedic | 6.724 | **6.260** | −0.464 |\n   | newswire | 5.870 | **5.585** | −0.285 |\n   | web prose | 5.627 | **5.539** | −0.088 |\n\n   The register the raw pool is emptiest of (technical Q&A: **zero** `<pre><code>`\n   documents pool-wide) improves ~9× more than web prose, the register the pool\n   most resembles. This is the signature of register rebalancing, and it is not\n   what a uniform \"better data\" story predicts.\n3. **The selector buys the target's surface conventions at a measurable rate.**\n   The encyclopedic quarter is WikiText-formatted: GPT-2 tokens `' ,'` and `' .'`\n   (space-before-punctuation) are **7.9%** of its tokens but **0.055%** of pool\n   tokens. The submitted selection puts **5.20%** of its budget in documents\n   carrying that convention — 5.3× the random rate (623k vs 119k tokens) — and its\n   first 400 documents are 38× denser in those tokens than random (1962 vs 51 per\n   100k).\n4. **The encyclopedic register stays the worst one no matter what.** Its\n   convention is nearly absent from the pool (0 documents contain ` @-@ `), so it\n   remained the highest-loss quarter in every one of the 20 selections I trained.\n   That is a ceiling imposed by the pool, not by the selector.\n\n## Falsification\n\nThe claim predicts specific *failures*. Each was run as its own training run\n(frozen recipe, seed 1337, dev target). All would have falsified it; none did.\n\n| Selection criterion | dev PPL |\n|---|---|\n| Random (do-nothing baseline) | 485.5 |\n| Quality gate + dedup only, random within it | 403.6 |\n| Single global quality score (LLR vs. target mixture), top-k | 387.0 |\n| Balanced per-register Naive-Bayes round-robin | 312.5 |\n| Greedy match, uniform-smoothed target, min-len 300 | 279.3 |\n| Greedy match, background-smoothed target, min-len 300 | 272.9 |\n| **Greedy match, background-smoothed, min-len 150 (submitted)** | **267.7** |\n\n- *If the junk filter were doing the work*, quality-gate-only random selection\n  would approach the matched selection. It does not: the gate buys 485.5 → 403.6,\n  matching buys 403.6 → 267.7. **Matching contributes ~1.7× more than filtering**,\n  and the two are separable because both selections pass the identical gate.\n- *If a single quality axis sufficed*, ranking documents by their likelihood ratio\n  against the pooled target mixture would match register-aware selection. It does\n  not (387.0 vs 312.5): a single global ranking collapses onto whichever register\n  the pool most resembles.\n- *If the hardest register's surface tokens were the bottleneck*, force-feeding\n  1.5M tokens (12.5% of budget) of the documents richest in `' ,'`/`' .'` would\n  help. It hurt: 279.3 → 290.5 at matched settings. Those documents are nav-bar\n  and sports-ellipsis boilerplate; the objective's own 5.2% allocation is already\n  the right dose, and buying more trades prose for junk.\n- *If the hardest register deserved extra weight*, doubling the encyclopedic\n  register's weight in `p_target` would help. It hurt (288.8 vs 279.3): the\n  equal-parts weighting is already correct, as the metric's additive structure implies.\n- *If longer documents were simply better*, raising the minimum length would help.\n  It hurt monotonically above the optimum: 267.7 (150) → 269.5 (200) → 279.3 (300)\n  → 295.4 (900). The optimum is a bracketed interior minimum (267.7 at both 100 and\n  150), not a monotone preference — past ~150 tokens, extra length costs coverage.\n- *If a more faithful greedy were better*, shrinking the batch from 250 to 80\n  documents per re-scoring step would help. It hurt (273.4): finer steps overfit\n  the unigram proxy, which is a means, not the objective.\n- Also tested and rejected against 267.7: a looser ratio clip (1000 → 269.0), a\n  looser quality gate (269.2), a tighter quality gate (281.8), weaker target\n  smoothing (268.1), and a 10× stronger background prior (275.9). The gate results\n  bracket the submitted setting on both sides, so it is an interior optimum rather\n  than the endpoint of a one-sided sweep.\n\n**What would still falsify the claim:** a selection with a *higher* proxy\ncross-entropy that trains to lower perplexity; or a quality-only criterion using\nno target sample reaching ~268 at this budget.\n\n**Honest limits.** `p_target` is estimated from the disclosed dev sample, so the\nmatch is only as good as that sample's coverage of the hidden one (stated to be a\ndisjoint sample of the same domain). Five scalar hyperparameters were tuned\nagainst dev, which bounds but does not eliminate selection-on-dev risk; the\nflatness of the optimum (267.7 across min-length 100–150, 268.1 at a 5× different\nsmoothing mass) suggests the setting is not knife-edge. Differences below ~1.5 PPL\nhere are within run-to-run noise and I do not claim them. All runs share one seed,\nso I have not measured seed variance directly.\n\n## Transfer\n\nThe method needs only (a) a sample of target-domain text, (b) the tokenizer, and\n(c) one counting pass — no labels, no reference model, no GPU. The greedy step is\na gather plus a segment-sum, so scoring the full 200M-token pool costs ~1 s and\nthe entire selection ~80 s on CPU; cost is linear in pool tokens and independent\nof how much is finally kept, so it scales to far larger pools.\n\nIt transfers to any budget-constrained data decision where the evaluation\ndistribution is known or samplable: choosing pretraining mixtures, picking\nfine-tuning subsets for a known deployment domain, or setting domain weights\nwithout hand-tuned ratios. Three lessons generalise beyond this task: **smooth the\ntarget toward the achievable background, not toward uniform** (else the selector\nchases unreachable rare tokens); **clip the importance ratio** (some target tokens\ncannot be bought at any price, and an unclipped objective spends the budget\ntrying); and **keep greedy steps coarse** (chasing the proxy too precisely\ndegrades the true metric). The natural extension is the same greedy on\nhigher-order statistics once more target text is available — the unigram order is\na consequence of having only 1M target tokens, not a limit of the criterion.\n", "structuredPatch": [{"oldStart": 83, "oldLines": 8, "newStart": 83, "newLines": 11, "lines": ["    100k).", " 4. **The encyclopedic register stays the worst one no matter what.** Its", "    convention is nearly absent from the pool (0 documents contain ` @-@ `), so it", "-   remained the highest-loss quarter in every one of the 20 selections I trained.", "-   That is a ceiling imposed by the pool, not by the selector.", "+   remained the highest-loss quarter in all five selections where I measured", "+   per-register loss (quality-gated random, NB-balanced, two greedy variants, and", "+   the submitted one), never dropping below 6.26 nats while the other three", "+   registers reached 4.97–5.59. That is a ceiling imposed by the pool, not the", "+   selector: no selection can teach a convention the pool does not contain.", " ", " ## Falsification", " "]}], "userModified": false, "replaceAll": false}

=== FINAL SUBMISSION CHECKS ===
selection.json == curate.py output == evaluated winner: True
ids: 18418 | unique: True | all ints in [0,182016): True
Traceback (most recent call last):
  File "<stdin>", line 9, in <module>
ValueError: unsupported format character ',' (0x2c) at index 19
-rw-r--r-- 1 root root 10290 Aug  4 02:07 claim.md

submission/:
total 138
drwxr-xr-x 1 root root    140 Aug  4 01:58 .
drwxr-xr-x 1 root root    320 Aug  4 02:07 ..
-rw-r--r-- 1 root root   9432 Aug  4 02:02 curate.py
-rw-r--r-- 1 root root 131344 Aug  4 01:59 selection.json
168

[stdout]
=== FINAL SUBMISSION CHECKS ===
selection.json == curate.py output == evaluated winner: True
ids: 18418 | unique: True | all ints in [0,182016): True
Traceback (most recent call last):
  File "<stdin>", line 9, in <module>
ValueError: unsupported format character ',' (0x2c) at index 19
-rw-r--r-- 1 root root 10290 Aug  4 02:07 claim.md

submission/:
total 138
drwxr-xr-x 1 root root    140 Aug  4 01:58 .
drwxr-xr-x 1 root root    320 Aug  4 02:07 ..
-rw-r--r-- 1 root root   9432 Aug  4 02:02 curate.py
-rw-r--r-- 1 root root 131344 Aug  4 01:59 selection.json
168
[metadata] {"noOutputExpected": false}

fixed: True
tokens available: 15,000,209 (budget 12,000,000) -> covered: True
1926

[stdout]
fixed: True
tokens available: 15,000,209 (budget 12,000,000) -> covered: True
1926
[metadata] {"noOutputExpected": false}

patched
building

[stdout]
patched
building
[metadata] {"noOutputExpected": false}

docs filling 12M budget: full-dev 15285, half-A 15324, half-B 15188
half-A vs half-B (two DISJOINT 500k-token target samples):
   shared docs: 8076  = 52.7% of A / Jaccard 0.360
   token overlap: 55.6% of A's budget
half-A vs full-dev: 74.2% shared
launched
1826

[stdout]
docs filling 12M budget: full-dev 15285, half-A 15324, half-B 15188
half-A vs half-B (two DISJOINT 500k-token target samples):
   shared docs: 8076  = 52.7% of A / Jaccard 0.360
   token overlap: 55.6% of A's budget
half-A vs full-dev: 74.2% shared
launched
1826
[metadata] {"noOutputExpected": false}

half-A (first 500k of each register):
half-B (disjoint second 500k):
full-dev (submitted): 267.721
1715

[stdout]
half-A (first 500k of each register):
half-B (disjoint second 500k):
full-dev (submitted): 267.721
1715
[metadata] {"noOutputExpected": false}

half-A:
HELDOUT_PPL 275.107  loss 5.6172  train_s 108  /tmp/tr_g_half1.npy
half-B:
HELDOUT_PPL 274.966  loss 5.6166  train_s 108  /tmp/tr_g_half2.npy
1635

[stdout]
half-A:
HELDOUT_PPL 275.107  loss 5.6172  train_s 108  /tmp/tr_g_half1.npy
half-B:
HELDOUT_PPL 274.966  loss 5.6166  train_s 108  /tmp/tr_g_half2.npy
1635
[metadata] {"noOutputExpected": false}

The file /workspace/claim.md has been updated successfully. (note: the file had been modified on disk since you last read it — the edit applied cleanly, but the file contains other changes not in your context. Read it before edits that depend on surrounding content.)

[metadata] {"filePath": "/workspace/claim.md", "oldString": "## Falsification", "newString": "5. **The criterion is stable across target samples, though the document set is\n   not.** Splitting the dev target into two disjoint 500k-token halves (half of\n   each register) and running the identical pipeline on each gives selections that\n   share only **52.7% of their documents** (Jaccard 0.36) yet train to\n   **275.11 and 274.97** dev PPL — a 0.14 difference, i.e. the two disjoint target\n   samples are interchangeable for the purpose of ranking documents even though\n   they disagree about which documents to pick. Many pool documents are\n   substitutable carriers of the same distributional signal; what the method\n   extracts is the register/vocabulary structure, not particular documents.\n   Separately, estimation quality does scale with target sample size: 500k target\n   tokens give ~275 and the full 1M give 267.7, so ~7 PPL of the result comes from\n   how well `p_target` is estimated. This is the observable most directly relevant\n   to the hidden target, and it is the reason I expect the hidden-sample score to\n   track the dev score rather than diverge from it.\n\n## Falsification", "originalFile": null, "structuredPatch": [{"oldStart": 89, "oldLines": 6, "newStart": 89, "newLines": 21, "lines": ["    registers reached 4.97–5.59. That is a ceiling imposed by the pool, not the", "    selector: no selection can teach a convention the pool does not contain.", " ", "+5. **The criterion is stable across target samples, though the document set is", "+   not.** Splitting the dev target into two disjoint 500k-token halves (half of", "+   each register) and running the identical pipeline on each gives selections that", "+   share only **52.7% of their documents** (Jaccard 0.36) yet train to", "+   **275.11 and 274.97** dev PPL — a 0.14 difference, i.e. the two disjoint target", "+   samples are interchangeable for the purpose of ranking documents even though", "+   they disagree about which documents to pick. Many pool documents are", "+   substitutable carriers of the same distributional signal; what the method", "+   extracts is the register/vocabulary structure, not particular documents.", "+   Separately, estimation quality does scale with target sample size: 500k target", "+   tokens give ~275 and the full 1M give 267.7, so ~7 PPL of the result comes from", "+   how well `p_target` is estimated. This is the observable most directly relevant", "+   to the hidden target, and it is the reason I expect the hidden-sample score to", "+   track the dev score rather than diverge from it.", "+", " ## Falsification", " ", " The claim predicts specific *failures*. Each was run as its own training run"]}], "userModified": false, "replaceAll": false, "staleRecovered": true}

ok
186
--- deliverables:
-rw-r--r-- 1 root root  11620 Aug  4 02:13 claim.md
-rw-r--r-- 1 root root   9432 Aug  4 02:02 submission/curate.py
-rw-r--r-- 1 root root 131344 Aug  4 01:59 submission/selection.json

[stdout]
ok
186
--- deliverables:
-rw-r--r-- 1 root root  11620 Aug  4 02:13 claim.md
-rw-r--r-- 1 root root   9432 Aug  4 02:02 submission/curate.py
-rw-r--r-- 1 root root 131344 Aug  4 01:59 submission/selection.json
[metadata] {"noOutputExpected": false}

[1]+  Done                    nohup bash dev_eval.sh /workspace/submission/selection.json > work/official_eval.log 2>&1
Token indices sequence length is longer than the specified maximum sequence length for this model (1348 > 1024). Running this sequence through the model will result in indexing errors
packed 12000000 tokens from 18418 ids -> /tmp/dev_train.npy
HELDOUT_PPL 267.721  loss 5.5899  train_s 51  /tmp/dev_train.npy
(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)
1485

[stdout]
[1]+  Done                    nohup bash dev_eval.sh /workspace/submission/selection.json > work/official_eval.log 2>&1
Token indices sequence length is longer than the specified maximum sequence length for this model (1348 > 1024). Running this sequence through the model will result in indexing errors
packed 12000000 tokens from 18418 ids -> /tmp/dev_train.npy
HELDOUT_PPL 267.721  loss 5.5899  train_s 51  /tmp/dev_train.npy
(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)
1485
[metadata] {"noOutputExpected": false}