nano-gpt-data-curation
claude-opus-4-8
nano-gpt-data-curation — graded by the task's verifier against a measured baseline.
- backend
- modal
- variant
- claude claude-opus-4-8
- metric
- held-out multi-domain perplexity
- actions
- 65
- duration
- 59m
- recorded
- 2026-08-04
run iddclm-nano-curation__SWfSgxK
Score
reward
336.180
Summary
All deliverables are complete and validated. Final summary:
Result
| Selection | Dev held-out PPL |
|---|---|
| Random baseline (reference) | 485.4 |
| Single global top-N quality filter | 380.7 |
| Equal 3M/register split | 366.1 |
| My submission (register-allocated) | 336.8 |
30.6% below the random baseline, validated end-to-end through the official frozen pack_selection.py + train_nano.py.
What I did
The target is disclosed as equal parts of 4 registers (Wikipedia / high-quality web prose / news / technical Q&A), and the dev reference turned out to be exactly 4 ordered quarters. My criterion:
- Prose-quality prefilter (Gopher/C4-style): token-length window, mean word length, stopword fraction, alpha/digit ratios, unique-line and type/token ratios — removes lists, boilerplate, prime-number tables, galleries.
- Per-register match: a hashed uni+bigram Naive-Bayes log-likelihood ratio of each dev quarter vs the pool background; each doc's register = argmax score.
- Anti-spam via cross-register spread: SEO word-salad matches every register weakly and uniformly (spread ≈ 0.2–0.35) while genuine prose scores high on its own register and very negative on Q&A (spread ≈ 1.0–1.6). Dropping
spread < 0.38cleanly kills the spun text that fooled the vocabulary filter. - Difficulty-and-scarcity-weighted allocation: per-quarter perplexity is very uneven (wiki ≈ 688 ≫ qa 376 > web 268 ≳ news 258) and wiki is the most data-hungry, so I allocate
[4.9, 4.7, 2.2, 0.4]Mtokens instead of equal splits, and take all the scarce Q&A (removing it alone cost +48 PPL). Priority order is a weighted round-robin so the first 12M tokens stay balanced.
Deliverables
/workspace/submission/selection.json— 60,001 unique valid ids, priority-ordered (first ~18.5k fill the 12M budget)./workspace/submission/curate.py— self-contained, deterministic (re-execs withPYTHONHASHSEED=0); reproduces the selection frompool.jsonl+ the dev reference./workspace/claim.md— Hypothesis / Mechanism (with two non-perplexity predictions: spread separates spam, and per-quarter difficulty ordering) / Falsification (each backed by a measured run) / Transfer.