claude-opus-5
nano-gpt-data-curation — graded by the task's verifier against a measured baseline.
- backend
- modal
- variant
- claude claude-opus-5
- metric
- held-out multi-domain perplexity
- actions
- 46
- duration
- 1h 20m
- recorded
- 2026-08-04
run iddclm-nano-curation__twR79Yh
Score
Summary
All three deliverables are in place and verified end-to-end.
Result
Dev held-out perplexity 322.8 vs 459.8 random baseline — a 30% reduction.
Method (submission/curate.py, runs in ~80s, CPU only)
The dev target turned out to be GPT-2 tokens I could detokenize, revealing four contiguous quarters — WikiText, general web prose, news, and StackExchange HTML Q&A. That gave me labeled positives for free.
- Coarse prose filter — drops nav menus, word lists, key blocks, boilerplate, exact dupes (17% of pool).
- Frequent-vocabulary log-odds — per register,
mean[log P_register(w) − log P_pool(w)]over the ~16k words appearing ≥3× in the target. - Balanced round-robin — equal token shares per register, interleaved so every prefix of the list is balanced.
What the ablations showed
Three of my first four attempts were worse than random, and the failures were the informative part:
| variant | dev PPL |
|---|---|
| NB log-count-ratio, unbounded hashed n-grams | 1124.7 |
| global ranking, no register balance | 665.3 |
| generative target-LM likelihood ranking | 554.8 |
| strict quality filter, no ranking | 446.9 |
| submitted | 322.8 |
- Unbounded rare features put plant-genus lists and PGP key blocks at rank 1 — a log-count ratio rewards pool-rarity.
- Ranking by likelihood is worse than random: the most probable text under a smoothed word LM is bland SEO filler. The signal has to be relative, not absolute.
- Strict "quality" filtering actively hurt (335 → 323 when loosened). At 6% of the pool, coverage binds and diversity beats purity.
Mechanism, verified separately from the score
I predicted that dropping the balance constraint would collapse the least-web-like register rather than degrade everything uniformly, and measured per-quarter PPL to check: technical Q&A degraded 7.4× (149 → 1099) while web prose moved 1.05× (289 → 304). Confirmed in the predicted shape.
One honest limit I found: wiki stays the worst quarter (804) under every selection, because the target is detokenized WikiText-103 with @,@ / @-@ conventions that no pool document has. That's a tokenization floor curation can't cross — claim.md records it, along with the caveat that these are single-seed runs so gaps under ~15 PPL shouldn't be trusted.