claude-opus-4-8
nano-gpt-data-curation — graded by the task's verifier against a measured baseline.
- backend
- modal
- variant
- claude claude-opus-4-8
- metric
- held-out multi-domain perplexity
- actions
- 97
- duration
- 1h 16m
- recorded
- 2026-08-04
run iddclm-nano-curation__xJxXGaB
Score
Summary
All deliverables are complete and validated. Here's the final summary.
Result
| selection (equal 12M-token budget) | dev PPL |
|---|---|
| random baseline (do-nothing) | 438.5 |
| single target-vs-pool classifier | 343.7 |
| + register-balanced (1:1:1:1) | 321.9 |
| + fluency gate | 322.0 |
| + 3:1:1:1 register weighting (submitted) | 313.7 |
Final held-out dev perplexity: 313.7 — 28.5% below the random baseline.
What I built
`/workspace/submission/curate.py` — a reproducible, pure-numpy pipeline (verified to regenerate selection.json byte-identically):
- Clean-prose gates — length, alphabetic ratio, digit ratio, function-word (stopword) ratio, and type-token/dominant-token repetition checks strip the bulk of raw-web junk (nav menus, non-English, tables, code dumps).
- Bigram-fluency gate — rejects "spun" word-salad SEO spam that passes vocabulary checks but has nonsensical word order (a real failure I caught: it was polluting the whole encyclopedic slot).
- Per-register targeting — four Naive-Bayes log-odds models (one per target register: wiki / web-prose / news / Q&A), positives taken from the four positional blocks of the disclosed dev target.
- Weighted round-robin fill (3:1:1:1) — over-samples the encyclopedic register, which a per-register diagnostic showed is ~2× harder than the others (a WikiText format the pool can't match) and dominates the equal-parts average.
`/workspace/submission/selection.json` — 40,000 unique, in-range ids in priority order (fills 12M tokens with margin).
`/workspace/claim.md` — Hypothesis / Mechanism / Falsification / Transfer. The mechanism prediction is the observable per-register perplexity profile (wiki 612 vs ~230–280 for the rest, and its non-monotone response to over-sampling: 612→569→565→595 as the mix goes 1→2→3→4×), independent of the final aggregate score.
The key empirical discoveries that drove the score down: (a) mean-per-token classifier scoring rewards rare-token junk — fixed by centering + total-log-odds; (b) word-salad spam needs a fluency (bigram) signal, not a vocabulary one; and (c) the equal-parts target is bottlenecked by the encyclopedic register, so the optimal training mix is deliberately unequal (3:1:1:1), not balanced.