claude-opus-5

All deliverables verified. Final summary:

Result

Dev perplexity 302.4 vs the random-selection baseline of 477.8 — a 37% reduction under the identical frozen 12M-token recipe. The official packer produces an array bit-identical to the one I measured.

Method

The key realization: the dev target is the only ground truth available, and it's enough. Decoding multi_dev.npy showed it's four equal contiguous quarters — WikiText-style encyclopedic, news, general web prose, and StackExchange-HTML technical Q&A. So curate.py:

1. Splits the decoded target into its four register blocks → positive documents per register
2. Trains a 5-way softmax classifier (hashed unigram+bigram features, 2^17 buckets) on GPU: each register vs. a random 40k pool sample as the "do-nothing" background. Held-out register accuracy 0.867 vs 0.20 chance.
3. Filters for corpus hygiene and removes near-duplicates (MinHash over word 5-grams)
4. Fills the budget by equal document count per register, emitted round-robin, with a reserve tail so the budget cannot underfill

What the experiments actually showed

• Per-register allocation is most of the win: pooled top-k scored 444 vs 333 for per-register, same scores and filters.
• *Equal document count beat equal token share (302–334 vs 339–347). Reproducing the winner's token mix while choosing documents differently scored 353 — so the gain is which* documents, not the ratio.
• Strictness dominated: prefix depth swept monotonically 302.4 → 333.7 over a 31-ppl range.
• A diagnostic saved me from the obvious wrong move. Register 0 was the loss bottleneck (849 ppl vs 161–336), because 12.7% of its tokens are WikiText detokenization artifacts absent from web text. I checked whether the pool could fix this — the 1,658 spaced-punctuation documents are SEO keyword spam, so it's irreducible by selection. Pouring budget into that register does lose (346.6).

Two corrections worth flagging

I initially read the winning run as "69% news" and built a sweep around it; that was an artifact of attributing documents to their argmax register rather than the register that selected them. Reproducing that mix explicitly scored 353, not 333, which disproved it.

More importantly, my first noise estimate (0.45 ppl) was badly wrong. Training is bit-deterministic — the same array twice gives identical ppl — but two cold runs of the final script produced the identical id set in different order and scored 315.6 vs 304.3. Selection jitter is ±6 ppl, from GPU sparse-matmul nondeterminism perturbing near-tied scores. So the sweep's direction is real but its fine ranking is not, and I expect the hidden score to regress somewhat toward ~310. claim.md documents this as the headline caveat, along with the root cause (the prefix was sized in estimated tokens at 4.05 chars/token when the true ratio is 4.248, pulling the last 3% of documents from the jitter-sensitive reserve).

I did not achieve bit-exact reproducibility — stable sorts plus score quantization reduced but didn't eliminate the jitter, and I stopped there rather than rewrite the classifier onto CPU with the clock running.