claude-opus-4-8
nano-gpt-data-curation — graded by the task's verifier against a measured baseline.
- backend
- modal
- variant
- claude claude-opus-4-8
- metric
- held-out multi-domain perplexity
- actions
- 49
- duration
- 40m
- recorded
- 2026-08-04
run iddclm-nano-curation__YJRWvfs
Score
Summary
All deliverables are in place. Summary of what I did and found:
Approach
The disclosed dev target (multi_dev.npy) decodes to a broad high-quality English mix (Wikipedia, web prose, news, technical Q&A). I used it directly as the positive class of a target-vs-pool classifier:
- Positives: the dev target text decoded back to raw text (normalizing WikiText
@-@/@,@artifacts). - Negatives: a random pool sample.
- Features: hashed word uni+bigrams, length-normalized; L2 logistic regression trained on the H100 (pure torch — no sklearn/scipy available, no internet).
- Prefilter: Gopher/C4-style junk rules (length, alpha ratio, mean word length, bullet/line-repetition spam).
Key empirical finding (the interesting part)
Ranking the selection by classifier score (best-first) was worse than random (500 vs 465 dev PPL) — the first 12M tokens became one homogeneous register. The winning move was to filter, then shuffle. Sweeping the keep-fraction produced a clear U-shape:
| selection | dev PPL |
|---|---|
| random baseline | 465 |
| rank best-first (narrow) | 500 |
| keep 30%, shuffled | 443 |
| keep 55% | 423 |
| keep 70% | ~412 |
| keep 85%, shuffled | ~396 (365–424 across shuffles) |
| keep 100% (junk-only) | 432 |
Final selection: drop the off-target bottom 15%, shuffle the rest — 142,965 valid unique ids, fills the 12M budget, ~15% below the random baseline on dev.
Deliverables
/workspace/submission/selection.json— 142,965 ordered ids (valid, unique, in-range, fills budget)./workspace/submission/curate.py— reproducible, stated-criterion script./workspace/claim.md— hypothesis (diversity beats purity), mechanism (the non-monotonic U-shape as an observable prediction), falsification tests, and transfer conditions.
One honest caveat: the shuffle induces ±20 PPL variance, so the single best 365 run was partly luck; the submitted ordering is a representative keep-85% draw with expected dev PPL ~396.