claude-opus-4-8

I curated the pretraining pool via DSIR-style distribution-matching resampling and validated it end-to-end on the frozen pipeline.

What I found (each measured on the real 30M-GPT / 12M-token eval):

The key insight: for a broad, multi-register target (Wikipedia + web prose + news + technical Q&A), the intuitive move — rank docs by quality/target-likeness and take the best — actively hurts, because it collapses diversity onto generic high-frequency prose and drops 30% of the target's informative vocabulary. The winning move is to (1) gate out non-prose junk (non-English, boilerplate, repetitive, number-lists — exactly the artifacts that fool naive relevance scorers), then (2) importance-resample toward the disclosed target with a temperature that preserves within-register diversity.

Mechanism, confirmed with a training-free observable: KL(target‖selection) over words is monotone with perplexity (mine 0.462 < random 0.558 < top-k 0.659), and target-vocabulary coverage moves inversely (0.827 > 0.780 > 0.695). The tilt temperature has a genuine interior optimum (T≈0.4 beats both top-k collapse and pure random), showing the win is a match-vs-diversity trade-off, not just junk removal.

All three deliverables are in place, validity-checked (unique/in-range ids, fills the full 12M budget), and the submitted selection.json was regenerated by curate.py and re-verified at PPL 365.9. ~47 minutes of budget remain unused, deliberately — the curve is flat around the optimum and further tuning would risk chasing dev-set noise, since the official target is a disjoint sample.