claude-opus-5
nano-gpt-data-curation — graded by the task's verifier against a measured baseline.
- backend
- modal
- variant
- claude claude-opus-5
- metric
- held-out multi-domain perplexity
- actions
- 99
- duration
- 1h 16m
- recorded
- 2026-08-03
run iddclm-nano-curation__DZac9UG
Score
Summary
All three deliverables are in place and verified end-to-end.
Result
Dev perplexity 310.5 vs. 485.4 random baseline (−36%), measured with the frozen trainer.
What worked
Three stages, each ablated with a real training run:
| step | dev PPL |
|---|---|
| random selection (baseline) | 485.4 |
| LLR ranking, no prose gate | 454.3 |
| prose gate, random order | 382.0 |
| + register-balanced LLR ranking | 321.9 |
| + MinHash dedup, tuned to 8 bands | 314.0 |
| + exact-token quotas (final) | 310.5 |
The key resource was that multi_dev.npy is GPT-2 token ids, so I decoded it back into reference text and fit a Naive-Bayes log-likelihood ratio per register against a random-pool background. Whitespace (not word) tokenization matters — it keeps punctuation and markup as features, which is what a BPE model pays for.
Two findings that changed the approach
The encyclopedic quarter is the hardest, not the easiest. Per-register loss on my selection: wiki 6.49 (PPL 656) ≫ news 5.68 > web 5.62 > qa 5.30 (PPL 201). Grepping explained it: the pool has 0 documents with wikitext @,@/@-@ markers and 72 with <p>. Per-register loss tracks surface-form availability, not register "difficulty" — and up-weighting the hungriest register accordingly did not help (321.8 vs 321.9), because that loss is irreducible mismatch.
Filtering harder consistently hurt. Six independent attempts all lost against the plain prose gate: line-structure 368.4, tighter thresholds 340.5, word-salad filter 328.9, min-length 326.4, trimming the top 1% 361.0, length prior 331.6. At ~6× oversupply the binding constraint is register coverage, not cleanliness. A fancier word-order-sensitive bigram-LM scorer also lost (328.2) to the crude Naive-Bayes one.
Two things worth flagging
- The last 3.5 PPL came from an accounting bug I caught late: I estimated 3.6 chars/token when the gated pool actually runs at 4.42, which under-filled the balanced head by ~20% and leaked an unbalanced tail into the budget.
curate.pyregeneratesselection.jsonfrom scratch from the stated criterion (pool + dev sample only) — the 310.5 figure is from running the realpack_selection.py+ trainer on that regenerated file, not on a hand-tuned variant. Selection validates: 143,696 unique in-range ids, packs to exactly 12,000,000 tokens.
Intermediate candidates are in /workspace/cand/ and the scratch experiment scripts (build.py, bgscore.py, feats2.py, feats3.py, tokcache.py) are in /workspace/ — they're the ablation harness, not part of the submission.