claude-opus-5
nano-gpt-data-curation — graded by the task's verifier against a measured baseline.
- backend
- modal
- variant
- claude claude-opus-5
- metric
- held-out multi-domain perplexity
- actions
- 49
- duration
- 1h 10m
- recorded
- 2026-08-04
run iddclm-nano-curation__HzsPDMT
Score
Summary
Done. All three deliverables are in place and verified.
Result
| selection | dev PPL |
|---|---|
| random baseline (do-nothing) | 459.8 |
| pooled single quality score | 333.9 |
| + register-balanced quotas | 317.3 |
| + English/non-Latin penalty + length floor | 304.1 |
34% below the random baseline.
Approach
The disclosed dev target ships as GPT-2 token ids — decoding it back to text yields 2,645 labeled positives for free. I split those into the target's constituent registers (encyclopedic / technical Q&A / news+web prose, identifiable from wikitext @-@ artifacts and StackExchange HTML tags), trained one hashed bag-of-words logistic regression per register against a random pool sample, and filled the 12M-token budget in fixed per-register shares.
Two things mattered more than the classifier itself:
- Register quotas, not a pooled score. A single target-likeness score collapsed onto news, which is the most lexically separable slice from generic web text — the top-ranked documents were near-uniformly South-Asian politics. Splitting and quota-filling was worth 17 perplexity on its own.
- Normalizing away unreachable cues. The target's Wikipedia slice carries wikitext formatting that zero of 182,016 pool documents have. That formatting is the single most discriminative feature available and is entirely useless, so both sides go through the same aggressive normalizer.
Things that did not work (all in claim.md)
- Bigrams raised classifier accuracy on every register yet made perplexity worse (304.1 → 308.2). Better discrimination ≠ better selection.
- Matching the dev mixture exactly (.308/.229/.463) scored 314.0, worse than the flatter .25/.25/.50 — going deeper into the wiki ranking costs more in document quality than mixture fidelity buys.
- Gumbel-top-k sampling to broaden topics scored 317.1; the ranking head is genuinely informative, not a narrow-topic artifact.
Mechanism check
claim.md predicts an observable other than the final perplexity: unigram KL to the target, measurable on the packed corpus before any training. It falls 0.4515 → 0.2864 (−37%), tracking the −34% perplexity drop. Target token coverage barely moves (0.9753 → 0.9795) — the gain is mass on common words, not rarer vocabulary, which rules out a coverage story.
Two caveats I've recorded rather than smoothed over: the length-floor optimum is sharp (±50 tokens costs ~9 perplexity), and every configuration was trained once with the frozen script's fixed seed, so some of the margin between 304 and the 313–317 cluster may be selection luck on this dev sample rather than something that transfers to the hidden one. The gap to the 459.8 baseline is far too large to be explained that way.