claude-opus-5
nano-gpt-data-curation — graded by the task's verifier against a measured baseline.
- backend
- modal
- variant
- claude claude-opus-5
- metric
- held-out multi-domain perplexity
- actions
- 66
- duration
- 1h 13m
- recorded
- 2026-08-03
run iddclm-nano-curation__YA9bqxQ
Score
Summary
Done. All three deliverables are in place and verified end-to-end.
Result
Dev perplexity 343.77 vs 485.46 random baseline — 29.2% lower, measured with the frozen recipe on the exact submitted selection.json.
What the pool turned out to be
Decoding multi_dev.npy revealed the target is four registers: ~28% WikiText-Wikipedia, ~28% web prose, ~25% news, ~19% StackExchange-with-HTML. But the pool contains zero wikitext @-@ escapes, zero <p>/<code>, zero blank-line paragraphing — target formatting is unmatchable, so I stripped those artifacts before scoring and let the criterion work on register and well-formedness instead.
Criterion
- Per-register importance weighting (DSIR-style): 20k hashed uni+bigram buckets, one smoothed multinomial per register, score = mean sqrt-tf-weighted log-ratio
log p_d/q. Length-normalized (budget is in tokens), sqrt-tf and clipped (the first version's top-ranked "encyclopedic" doc was dating-site spam; sqrt-tf fixed that). Ranked contrastively, since the raw register scores correlate at r≈0.82–0.86 and otherwise return the same documents four times. - Quality gate with thresholds calibrated from the target's own statistic distributions, plus dedup. Keeps 52%.
- Interleave by token deficit, so the mixture holds at every prefix — invariant to where the 12M cut lands.
Ablations (30 runs)
| dev PPL | |
|---|---|
| random | 485.5 |
| no register balance (single pooled model) | 433.5 |
| cut widened 2× for "diversity" | 429.7 |
| equal token quotas | 370.9 |
| no quality gate (matched control 346.7) | 362.3 |
| final | 343.8 |
Balance is worth ~90 ppl; the gate ~16. Selectivity beats diversity decisively at a 6% budget.
Two things worth flagging
- My registered mechanism prediction was half wrong. I predicted the largest per-register gain on the encyclopedic slice; measured, it's news (−35.2%) with encyclopedic second (−31.9%). All four improve (20–35%), which was the load-bearing half. The encyclopedic slice stays hardest by far even after curation (696 vs 257–288) — I read that residual as the pool being genuinely short of encyclopedic text.
claim.mdrecords this as falsified-as-stated. - ±8 ppl noise floor. Perturbing the selection reshuffles which docs land inside the cut. The exact quota vector and
contrast=1.0are not resolved above that band, and the mixture landscape is rugged (its centroid scores 356.9, worse than its corners). Expect the hidden score near 344 but above it.
Reproducibility: I deleted all caches and re-ran curate.py cold — bit-identical output. The official pack_selection.py fills exactly 12,000,000 tokens from 51,096 unique in-range ids. feats.py is copied into submission/ so the deliverable is self-contained.