nano-gpt-data-curation
claude-opus-4-8
nano-gpt-data-curation — graded by the task's verifier against a measured baseline.
- backend
- modal
- variant
- claude claude-opus-4-8
- metric
- held-out multi-domain perplexity
- actions
- 82
- duration
- 1h 1m
- recorded
- 2026-08-04
run iddclm-nano-curation__gSSiCuY
Score
reward
348.860
Summary
Done. All three deliverables are complete, validated, and mutually consistent.
Result
| Selection (12M-token budget) | Dev held-out PPL |
|---|---|
| Bottom-scored anti-selection (falsification) | 1045.0 |
| Random baseline (do-nothing reference) | 462.5 |
| Quality-ranked, prose-only (no register balancing) | 400.4 |
| Submitted: quality-ranked + register-balanced | 356.6 |
356.6 is 23% below the random baseline, confirmed by an authoritative eval on the exact submitted file.
Method (in curate.py)
- Recover the target as text: GPT-2-decode
multi_dev.npyand de-normalize the WikiText@-@/spaced-punctuation artifacts → ~2,700 positive examples of the disclosed domain. - Domain classifier: a bag-of-words (word uni+bigram) logistic regression, trained target-vs-random-pool in torch on the H100 (no sklearn available). Scores all 182k docs by target-likeness.
- Quality gates + dedup: keep 200–60k-char docs, drop near-duplicates.
- Register balancing: interleave two score-sorted streams to hold ~20% of the budget as technical-QA/code (the register a single classifier under-ranks), matching the target's equal-parts mixture across any budget prefix.
What the experiments established
- The classifier score is a causal, monotone predictor of training value: inverting it (bottom-scored docs, same quality gate) nearly triples perplexity (1045 vs 357) — the gain is not a length/dedup artifact.
- The win decomposes cleanly: quality ranking (462→400) + register balancing (400→357). A qa_frac sweep {0.00:400, 0.20:357, 0.25:370, 0.28:364} shows including QA is the large robust effect; the exact fraction is budget-tail noise, so I took the best-validated 0.20.
Deliverables
/workspace/submission/selection.json— 27,608 priority-ordered, unique, in-range ids (~20M tokens; fills the 12M budget with margin)./workspace/curate.py— reproducible end-to-end from the stated criterion (fixed seed;python3 curate.pyregenerates it)./workspace/claim.md— Hypothesis / Mechanism (predicts the score→PPL ordering, an observable other than final PPL) / Falsification / Transfer.
~33 min of budget remain, but the method is at its ceiling — further config tweaks land inside the 357–370 dev-noise band and would only overfit the dev set, so I'm stopping here rather than chase noise.