claude-opus-4-8
nano-gpt-data-curation — graded by the task's verifier against a measured baseline.
- backend
- modal
- variant
- claude claude-opus-4-8
- metric
- held-out multi-domain perplexity
- actions
- 45
- duration
- 43m
- recorded
- 2026-08-04
run iddclm-nano-curation__NFLvnqE
Score
Summary
All deliverables are complete. Here's a summary of what I did and found.
Result
| Selection method | Dev held-out PPL |
|---|---|
| Random baseline (do-nothing) | 438 |
| Quality classifier (encyclopedic-dominated) | 445 |
| Register-balanced classifier | 447 |
| Clean-broad junk filter (submitted) | 396 (~10% below random) |
What I built
- `submission/selection.json` — 40,000 unique, in-range pool ids (covers >12M tokens; packer confirmed it fills the budget).
- `submission/curate.py` — reproducible criterion: a lenient English-prose junk gate (min length, ≥60% letters, ≤25% symbols, ≥6% English function-word density, no single token >12%, ≤50% repeated lines) that keeps 166,180/182,016 docs (~9% removed) and emits them in a deterministic breadth-preserving shuffle.
- `claim.md` — Hypothesis / Mechanism / Falsification / Transfer.
Key insight
I first tried the standard "train a quality classifier against the target and pick the top docs" approach — and it lost to random (445 vs 438), because the target is broad and equal-parts (Wikipedia + web prose + news + technical Q&A), and the classifier collapsed onto a single register, starving the model of the others.
The real lever turned out to be the opposite: the raw pool is already broadly on-domain, so the win comes from removing genuine junk (non-English, boilerplate/nav spam, markup dumps, fragments) while preserving breadth, not from re-weighting toward "premium" prose. Two grep-level observations backed this up and shaped the claim:
- Over-filtering reverses the gain (strict gate went 396 → 425) — narrowing hurts.
- The pool has ~2.7% code vs the target's ~25% technical register (
pre><code: 0 docs) — that register is essentially unlearnable from this pool, so forcing scarce JS-junk in made things worse. Gains necessarily come from the three prose registers.
I also verified the eval is noisy (a two-stopword filter tweak swung 396↔416), so I deliberately did not seed-shop — the official target is a disjoint sample, so only the robust junk-removal effect transfers, and every junk-filtered variant (396–425) beat random (438).
Time remaining: ~50 min of budget unused — I stopped once the method was proven and the deliverables were validated, rather than gamble further evals against eval noise.