claude-opus-4-8
nano-gpt-data-curation — graded by the task's verifier against a measured baseline.
- backend
- modal
- variant
- claude claude-opus-4-8
- metric
- held-out multi-domain perplexity
- actions
- 87
- duration
- 1h 21m
- recorded
- 2026-08-04
run iddclm-nano-curation__7umcSHc
Score
Summary
Done. All three deliverables are complete, validated, and the result is certified end-to-end.
Result
| held-out ppl | vs random | |
|---|---|---|
| random baseline (do-nothing) | 457.2 | — |
| delivered selection | 343.9 | −24.8% |
Certified via dev_eval.sh on the exact deliverable: HELDOUT_PPL 343.857, packing exactly 12,000,000 tokens from 17,860 ids.
Deliverables
- `submission/selection.json` — 17,860 unique ids, all in range, list of ints, fills the 12M budget. Reproduced byte-identically by the script.
- `submission/curate.py` — produces the selection from a stated criterion (not a hand-picked list): DSIR importance scoring → English-prose quality gates → register-mixture stratification → MinHash/LSH dedup. Deterministic (crc32 hashing, fixed k-means seed).
- `claim.md` — Hypothesis / Mechanism / Falsification / Transfer.
The core finding
The winning lever was *matching the target's register mixture, not maximizing average affinity. Plain DSIR top-k over-selects the single highest-scoring register and starves the target's second-largest one (share 0.253 → 0.073; register-mix L1 = 0.43). Clustering the target into 8 registers and round-robin quota-filling to each register's target share drives that L1 to 0.002 and ppl from 355 → 344 — even though it slightly lowers global affinity* (−0.498 → −0.530).
That's the sharp, pre-registered observable: register-mixture distance predicts ppl against the affinity signal, so affinity alone isn't the mechanism — distributional balance is. Two sub-findings confirmed it: (1) sealing the last ~2.5% of the budget so even the tail is on-mixture gave a further −4.4 ppl; (2) the naive-DSIR rarity-reward bug (smoothing rewards unseen gibberish n-grams) had to be corrected by clamping weak-target-count weights to ≤0, or garbage ranks first.
Each component's contribution decomposes cleanly (gates 15%, target-matching 68%, dedup 7%, register-balance 10%), all verified by deterministic ablation.
I stopped here rather than hyperparameter-tuning further — the method is principled and the scoring target is a hidden disjoint sample, so chasing another 1–2 dev ppl by tuning K risks overfitting the dev set against exactly what generalizes.