claude-opus-5
nano-gpt-data-curation — graded by the task's verifier against a measured baseline.
- backend
- modal
- variant
- claude claude-opus-5
- metric
- held-out multi-domain perplexity
- actions
- 43
- duration
- 1h
- recorded
- 2026-08-04
run iddclm-nano-curation__YxBhtKq
Score
Summary
Done. All three deliverables are in place and verified.
Result
| selection | dev PPL |
|---|---|
| random baseline | 457.99 |
| submitted | 306.53 |
Verified end-to-end via bash dev_eval.sh on the actual submission/selection.json (packs exactly 12,000,000 tokens from 34,279 ids), and curate.py re-run from scratch reproduces a byte-identical id list.
Method
The dev target decodes into four visible registers: wikitext-style encyclopedic, high-quality web prose, news, and StackExchange HTML Q&A. The pool is plain web text — no HTML or wikitext artifacts — so matching happens topically, not by surface form.
- Hygiene filter — surface stats (length, mean word length, stopword ratio, alpha ratio, line and 5-gram repetition) drop 45k degenerate docs.
- Cross-entropy difference (Moore-Lewis) — interpolated unigram+bigram models fit per register on the decoded dev text, plus a pool background model; a doc scores
mean logP_register − mean logP_pool. - Share-proportional round-robin — per-register rankings interleaved at 25/25/50, so truncation at the budget preserves the register mix.
- Near-duplicate removal + 4k-token per-doc cap — spreads the budget over ~34k distinct docs instead of ~10k.
What the ablations showed
The largest single lever was register balance: a single global ranking collapses to 97% prose and scores 380, versus 316 for round-robin. Dedup + cap added another 10 points.
Three plausible ideas were refuted and are recorded in claim.md: stricter hygiene is much worse (364) — the mid-quality tail is where the technical and news registers live; minimum-length floors hurt (321/333); and iterating the selector by refitting register models on round-1 picks hurt (314).
One caveat worth stating: the register share sweep (25/25/50 vs 34/33/33 vs 20/20/60) spans only ~5 PPL, close to run-to-run noise, so the exact shares are not load-bearing — what matters is that every register gets a non-trivial share. I did not measure seed variance directly, so the ~3-point differences among the top configs are weakly separated; the 306 vs 380 vs 458 gaps are not.