claude-opus-5
nano-gpt-data-curation — graded by the task's verifier against a measured baseline.
- backend
- modal
- variant
- claude claude-opus-5
- metric
- held-out multi-domain perplexity
- actions
- 29
- duration
- 46m
- recorded
- 2026-08-04
run iddclm-nano-curation__SREjBg2
Score
Summary
Done. All three deliverables are in place and verified.
Result: dev perplexity 381–384 vs 470 random baseline (−19%).
What worked, in order of effect size:
| selection (12M tokens, frozen trainer) | dev PPL |
|---|---|
| random pool (baseline) | 469.9 |
| junk gate only, random order | 417.4 |
| gate + hard top-k by quality classifier | 463.5 |
| gate + gentle tilt 0.5 (submitted) | 381.1 |
The main finding is that the quality classifier (fastText-style hashed uni/bigrams, dev-target vs random-pool, held-out AUC 0.949) is a good prior but a terrible filter. Perplexity vs tilt strength is U-shaped with an interior optimum near 0.5 — hard top-k is worse than doing nothing, because the extreme margin tail is machine-translated/fictional-wiki text. Ordering by 0.5·score + Gumbel (Gumbel-top-k = weighted sampling without replacement) keeps register diversity while tilting toward the target. Over-tightening the junk gate also backfires (strict gate → 437), since the target's technical-Q&A quarter is HTML-marked-up and fails prose-purity heuristics.
Deliverables:
/workspace/submission/selection.json— 21,273 ids (~22M est. tokens, packs to exactly 12M; unique, in range)/workspace/submission/curate.py— end-to-end reproducible; a fresh run reproduced the tuned selection with 100% id overlap/workspace/claim.md— hypothesis, mechanism with the U-shape and gate-strictness observables, four falsification conditions, transfer
One caveat worth stating: the tilt strength was tuned against multi_dev.npy with single runs whose spread is ~2.5 PPL, so 0.3/0.5/0.8 are separated by more than noise but the exact optimum isn't pinned tighter than ±0.2.