claude-opus-5
nano-gpt-data-curation — graded by the task's verifier against a measured baseline.
- backend
- modal
- variant
- claude claude-opus-5
- metric
- held-out multi-domain perplexity
- actions
- 51
- duration
- 60m
- recorded
- 2026-08-03
run iddclm-nano-curation__BebKMFd
Score
Summary
All three deliverables are in place and verified end-to-end.
Result
Dev perplexity 308.18 vs. random-selection baseline 469.9 — a 34% reduction, verified through the official dev_eval.sh path (which re-tokenizes from pool.jsonl with the frozen packer and trainer, seed 1337).
Approach
The dev target decodes into four visible registers: WikiText-style encyclopedic, news/sports, general web prose, and HTML-formatted StackExchange Q&A. curate.py implements one stated criterion:
- Well-formedness gate — cheap surface heuristics for running English prose (stopword rate, alpha/digit/uppercase ratios, line-duplication, sentence punctuation). Keeps 136,895/182,016 docs (117M tokens).
- Per-register target-similarity classifiers — the dev target is decoded, split at
<|endoftext|>, WikiText detokenization artifacts undone, and weakly labeled into 3 registers. One logistic regression per register (GPT-2 unigrams + hashed bigrams, fit on GPU as anEmbeddingBaglinear model) discriminates target from a random pool sample. - Near-duplicate removal — single-band 5-gram MinHash, drops 31k docs (~23%).
- Quota-filled, interleaved emission — each doc goes to its best-matching register; top-ranked docs fill a 40/35/25 token quota, emitted round-robin so any prefix keeps the mixture (the trainer truncates at 12M).
What the ablations showed (all in claim.md)
| variant | dev PPL |
|---|---|
| random baseline | 469.9 |
| classifier + quota, no dedup | 322–326 |
| 3×-wide top band, random within | 385.9 — selectivity matters a lot |
| + dedup | 317.6 |
| shipped: quota 40/35/25 + dedup | 308.2 |
| quota skewed 50/30/20 | 332.1 |
| ≥512-token length floor | 328.4 |
Two things I'd flag honestly: seed noise is ±5–9 ppl (measured by re-running two configs at a second seed), so the fine quota shape (40/35/25 vs 34/33/33) is not resolved — only the coarse "mirror the mixture, don't skew hard" claim is supported. And I did not run the ablation that separates the heuristic filter's contribution from the classifier's, so that decomposition remains untested; it's noted as the weakest part of the claim.