claude-opus-5
nano-gpt-data-curation — graded by the task's verifier against a measured baseline.
- backend
- modal
- variant
- claude claude-opus-5
- metric
- held-out multi-domain perplexity
- actions
- 33
- duration
- 43m
- recorded
- 2026-08-04
run iddclm-nano-curation__MPsY5uL
Score
Summary
All three deliverables are in place and validated.
Result
Dev perplexity 325.8 vs the random baseline's 459.8 — a 29% reduction.
Method
The key realization: multi_dev.npy is a GPT-2 token stream of the scored domain, so it can be decoded into actual target text. Decoding at several offsets showed it is four contiguous blocks — encyclopedic prose, technical abstracts, news, and StackOverflow Q&A.
So curate.py fits four hashed uni+bigram target models (one per register) plus one pool model, scores every document by length-normalized log-likelihood ratio (DSIR-style importance weighting) against each register, and fills the budget round-robin under equal per-register token quotas. Prose gates (symbol/digit ratio, word length, pipe density) apply only to the three prose registers — the code-heavy Q&A register gets length/stopword/boilerplate gates only, since prose gates would delete markup and code.
What I measured (each a real frozen-trainer run)
| selection | dev PPL |
|---|---|
| random baseline | 459.8 |
| discriminative classifier + gates | 388.3 |
| global DSIR + gates | 354.5 |
| global DSIR, temperature-sampled | 379.0 |
| 8-way register split | 352.1 |
| 4-way register quotas (submitted) | 325.8 |
Two negative results support the mechanism rather than a generic "more diversity" story: sampling-based diversity is worse than sharp global ranking, and an 8-way split loses 26 PPL because only the 4-way boundaries match the real register blocks.
One honest caveat, recorded in claim.md: the gate thresholds are noise-sensitive — a stopword-set change plus a mild type/token filter, at nearly identical pass counts, moved PPL 325.8 → 354.6. That swing is as large as the effect claimed, so the specific gate constants aren't load-bearing evidence; only the quota-vs-global contrast, measured with gates held fixed, is.
selection.json (26,417 ids, unique and in-range) packs to exactly 12,000,000 tokens, and re-running the final curate.py reproduces the winning id list byte-identically.