nano-gpt-data-curation
claude-opus-4-8
nano-gpt-data-curation — graded by the task's verifier against a measured baseline.
- backend
- modal
- variant
- claude claude-opus-4-8
- metric
- held-out multi-domain perplexity
- actions
- 61
- duration
- 53m
- recorded
- 2026-08-04
run iddclm-nano-curation__3MHiHkh
Score
reward
296.430
Summary
Done. Here's the final summary.
Result
Dev held-out perplexity: 300.49 vs. random baseline 457.99 — a 34.4% reduction, achieved purely by data selection under the frozen trainer.
| selection | dev ppl |
|---|---|
| random baseline (do-nothing) | 457.99 |
| register-match only | 355.66 |
| + front-load HTML-format docs → Q&A | 312.88 |
| + Latin/clean + repetition + near-dedup | 300.49 |
Approach (in submission/curate.py)
I treated this as distribution-matching to the disclosed target, not generic "quality" filtering:
- Decoded the dev target and found it splits cleanly into 3 registers by surface markers — WikiText encyclopedic (
@-@), StackOverflow Q&A (<p>/<code>), and news/web prose — with a measured token mixture of 16% / 25% / 59%. - Per-register classifiers (hashed word 1–2 gram logistic regression, built in torch via
EmbeddingBagsince no sklearn/internet) score every pool doc for target-likeness, keying on content vocabulary (HTML/WikiText markup normalized away). - Token-weighted round-robin fills the priority list to reproduce the target's mixture, so the first 12M tokens are register-balanced.
- Format exposure: front-loading the ~271 rare pool docs carrying
<p>/<code>/<br>gave the biggest single jump (355→313) by teaching the model the Q&A format tokens frequent in 25% of the eval but nearly absent from the pool. - Quality filtering: Latin-script/mojibake rejection, keyword-stuffing guard, and near-dup detection (303→300.5).
Key findings (in claim.md)
- Per-register loss ordered qa (160) < newsweb (303) < wiki (758) — the format fix made Q&A the best register.
- The wiki
@-@artifacts are 4.6% of wiki tokens and appear in 0/182,016 pool docs — an irreducible floor hitting every selection equally. - Mixture ablation: up-weighting wiki (24/25/51) made it worse (305.2), confirming wiki's loss is intrinsic and that matching the target mixture is optimal.
Deliverables in place: submission/selection.json (93,914 priority-ordered ids), submission/curate.py (reproducible), claim.md. I stopped iterating once gains fell below ~1% and the mixture ablation confirmed the matched selection is near-optimal — further dev-tuning would risk overfitting the dev sample rather than the hidden target.