claude-opus-5
nano-gpt-data-curation — graded by the task's verifier against a measured baseline.
- backend
- modal
- variant
- claude claude-opus-5
- metric
- held-out multi-domain perplexity
- actions
- 106
- duration
- 41m
- recorded
- 2026-08-04
run iddclm-nano-curation__PCUF55U
Score
Summary
All three deliverables are in place and validated.
What I did
Approach. The dev target (multi_dev.npy) is GPT-2 token ids, so I decoded it back to text and used it as the positive class for target-likeness classifiers. Inspecting the decoded target showed it is four visibly distinct registers — wikitext (@-@ markers), StackExchange-style HTML Q&A (<p>, <code>), newswire, and general web prose.
submission/curate.py (self-contained, deterministic, seeded):
- Hard filters on surface stats (length, ASCII/alpha fraction, stopword profile, duplicate lines, digit density, boilerplate) → keeps 73.2% of the pool.
- Per-register classifiers: spherical k-means (K=10) over hashed word 1–2 grams clusters the decoded target into registers; one L2-regularised logistic regression per cluster (target cluster vs random pool sample), all in torch on GPU — no sklearn available offline.
- Quota-balanced merge: each of the four registers gets 25% of the budget, split within a register by dev mass; the list is emitted by always serving the cluster furthest behind quota, so the ordering is quota-correct at every prefix.
Results (frozen trainer, dev target):
| selection | dev PPL |
|---|---|
| random baseline | 488.5 |
| single global classifier | 389.5 |
| equal-per-cluster quota, K=6 | 342.0 |
| dev-proportional quota, K=10 | 335.5 |
| submitted (25%-per-register, K=10) | 335.0 |
| K=16 | 338.2 |
31% below the random baseline. The bulk of the gain past simple filtering came from register balancing, not filter tuning — a single global classifier over-selects news, the register the raw pool is richest in.
One prediction failed. I predicted the curated set would consist of longer documents; measured, it is shorter (mean 535 vs 644 words, 14.6k vs 11.7k docs to fill the budget) because the Q&A register is intrinsically short and the quota forces it in. I recorded that as falsified in claim.md. The n-gram mechanism check did hold: unigram cross-entropy of the target dropped 8.117 → 7.904 nats vs random, with no model trained.
Caveat: all numbers are single-seed, so the ~1% spread among the top three variants is within noise and I don't claim an ordering there; the 488 → 389 → 335 steps are far larger than noise. The classifiers' positives are the dev sample while the official score is a disjoint hidden sample — hashed 1–2 grams with L2 is the guard against sample-level overfitting, but that exposure is real and is stated in the claim.