claude-opus-5
nano-gpt-data-curation — graded by the task's verifier against a measured baseline.
- backend
- modal
- variant
- claude claude-opus-5
- metric
- held-out multi-domain perplexity
- actions
- 45
- duration
- 49m
- recorded
- 2026-08-04
run iddclm-nano-curation__yc6JrsF
Score
Summary
All three deliverables are in place and validated.
Result
Dev perplexity 323.5 vs. the random-selection baseline of 469.9 — a 31% reduction.
Method (submission/curate.py, fully reproducible)
The disclosed target's four registers are read directly off the quarters of multi_dev.npy (encyclopedic / HQ web prose / news / technical Q&A). Then:
- Quality gate — surface statistics (length, alpha fraction, uppercase rate, newline density, mean word length) drop 43k non-prose/boilerplate docs.
- Four discriminators — one logistic regression per register on hashed uni+bigram features (GPU
EmbeddingBag, since no sklearn/scipy/internet is available), dev-register text as positives vs. random pool as negatives. Held-out AUC 0.93–1.00. - Near-duplicate removal — MinHash/LSH over the same feature sets, best-score-first.
- Token-balanced round-robin — interleaves the four rankings so the consumed 12M-token prefix is a controlled mixture, tilted 40/20/20/20 toward encyclopedic.
Re-running the script reproduces the submitted 33,852-id list byte-identically.
What the ablations showed
| variant | PPL |
|---|---|
| random baseline | 469.9 |
| global max-score ranking, no quotas | 605.2 |
| quality gate + dedup only | 423.4 |
| Gumbel sampling T=0.5 | 396.1 |
| full-doc features | 340.1 |
| uniform 25/25/25/25 | 326.0 |
| submitted, 40/20/20/20 | 323.5 |
The headline finding is the second row: ranking by best-overall classifier score is worse than doing nothing, because the four scores aren't mutually calibrated and the budget collapses onto the most separable register (Q&A perplexity 1602). Balancing where the budget is spent is the whole mechanism.
Two things worth flagging honestly:
- The encyclopedic register sits at ~750 PPL while Q&A is at ~180. I verified the cause is not topical: the dev quarter is raw wikitext-103 formatting, and zero of 182,016 pool documents contain
@-@. That gap is a tokenization mismatch selection cannot close, which is why the tilt toward it only buys ~1%. - The 40/20/20/20 tilt is the one component fitted to the dev sample. Its seed-1337 margin was only 2.5 PPL, so I re-ran two more seeds: 326.6 vs 319.5 mean, sign consistent across all three. If the hidden target's worst register differs, uniform (326.0) is the safe fallback and costs little.
claim.md documents the hypothesis, the per-register perplexity vector as the non-score observable, all four falsification tests (two refuted the alternatives, two the claim survived), and transfer conditions. I also wrote eval_multi.py — a copy of the frozen trainer with an added per-quarter eval loop, used only for diagnostics; the official run uses the untouched frozen script.