claude-opus-4-8
nano-gpt-data-curation — graded by the task's verifier against a measured baseline.
- backend
- modal
- variant
- claude claude-opus-4-8
- metric
- held-out multi-domain perplexity
- actions
- 49
- duration
- 32m
- recorded
- 2026-08-04
run iddclm-nano-curation__2yV7BuE
Score
Summary
All deliverables are complete and validated. Here's the summary.
What I did
I curated the 12M-token training set by register-stratified DSIR (importance ranking), after establishing the key structural facts about the data:
- The target is an equal-parts, 4-register mixture (Wikipedia, high-quality web prose, news, technical Q&A). Held-out loss is the mean across registers, so the optimal training mixture is the target's own mixture — and the register whose absence hurts most is technical Q&A/code (its tokens are otherwise unseen).
- The target's WikiText surface formatting (
@,@,@-@, spaced punctuation) appears in zero pool docs, so it's unreproducible by selection — I match on content/register (word n-grams), not formatting. - The pool is ~40% code-dense but the target is ~25% — so both random selection and naive quality ranking miss the mixture.
Pipeline (curate.py, fully reproducible, no hand-picking): split pool into TECH vs PROSE by code-density (threshold auto-calibrated to the target's 75th percentile → 25% tech); estimate per-stratum DSIR weights log p_target_s − log p_rawpool from hashed uni/bi-grams; rank docs within each stratum by average per-token log-ratio (with a light quality prefilter, length cap, and dedup); interleave best-first at 3 prose : 1 tech so the used 12M tokens are register-balanced.
Measured result (frozen recipe, dev target)
| Selection | Tech fraction | Held-out PPL |
|---|---|---|
| Random baseline | 0.40 | 485.4 |
| Combined-DSIR top-k (ablation) | 0.16 | 402.1 |
| Register-stratified (submitted) | 0.25 | 331.5 |
Importance ranking beats random (485→402); matching the register mixture adds the larger gain (402→331) — a 32% perplexity reduction vs. the random baseline. I also tested a finer 4-way prose split but the diagnostic showed degenerate clusters (the target's Wikipedia portion is a few long articles), so I kept the robust 2-way split rather than overfit.
Deliverables
/workspace/submission/selection.json— 41,768 unique in-range ids in priority order (packs to exactly 12M tokens)./workspace/submission/curate.py— the reproducible criterion./workspace/claim.md— Hypothesis / Mechanism (predicts the technical-fraction observable and the run ordering, not just the final number) / Falsification / Transfer.
~62 min of budget remain; the submission is finalized.