claude-opus-5
nano-gpt-data-curation — graded by the task's verifier against a measured baseline.
- backend
- modal
- variant
- claude claude-opus-5
- metric
- held-out multi-domain perplexity
- actions
- 78
- duration
- 1h 4m
- recorded
- 2026-08-03
run iddclm-nano-curation__q2CFtsE
Score
Summary
All three deliverables are in place and verified.
Result
Held-out dev perplexity 320.32 vs 457.50 for the random baseline — 30.0% lower, verified end-to-end through the unmodified official path (pack_selection.py → train_nano.py): 12,000,000 tokens packed from 24,728 ordered ids.
What drove it
I decoded multi_dev.npy and found the target is four equal 250k-token blocks: WikiText-formatted Wikipedia, high-quality web prose, newswire, StackExchange Q&A. Two early checks shaped everything: the pool contains 0 of 182,016 docs with the target's WikiText surface form ( @-@ , spaced punctuation) and only 69 with </p> — so surface matching is impossible and content/register match is the only lever.
The criterion: per-register Naive-Bayes importance weights (log p_register/p_pool over unigram+bigram features), then fill the budget by interleaving the four rankings one doc at a time. Measured contributions:
| dev PPL | |
|---|---|
| random baseline | 457.5 |
| quality gates only, random within | 464.0 — no better than random |
| global top-k, no register balance | 388.6 |
| balanced match, moderate gates | 354.8 |
| submitted (balanced, very light gates) | 320.3 |
Two findings worth flagging:
Quality filtering did nothing; register match did everything. Gates-only scored 464.0 — indistinguishable from random. And my own moderate gates were actively harmful: the per-register diagnostic showed loosening them cut the Q&A block from 303 → 165 (−46%) while barely moving the rest. The gates (frac_alpha ≥ 0.68, frac_digit ≤ 0.16, pipe ≤ 3) encoded a flowing-prose prior that excluded every document containing a code block — silently starving 25% of the target. A control (prose-strict + Q&A-loose gates, 324.1) rules out the alternative reading.
I tested transfer rather than assuming it. Since the profile is fit on the disclosed dev sample, its edge could have been topical leakage that dies on the hidden set. Refitting on the first half of each block and evaluating on the unseen second half: 362.5 unrestricted vs 363.9 with topical vocabulary stripped — indistinguishable, so the signal is register-level and should carry to the hidden sample.
Four hypotheses I tested and rejected, all recorded in claim.md: mixture-pushing toward the highest-loss register (up to 380.3), register-specific gates (324.1), equal-token instead of equal-rank-depth allocation (361.2), and feature normalization to bridge the surface-form gap (321.9, neutral).
Files: /workspace/submission/selection.json (24,728 ids, unique and in range), /workspace/submission/curate.py (re-run from a cold cache reproduces the selection exactly, ~60s, no GPU), /workspace/claim.md.
One caveat on the headline margin: the 457.5 baseline is my own random-selection run, not the benchmark's recorded anchor, so the official normalized gap may differ slightly even though the absolute 320.32 is what gets scored.