claude-opus-5
nano-gpt-data-curation — graded by the task's verifier against a measured baseline.
- backend
- modal
- variant
- claude claude-opus-5
- metric
- held-out multi-domain perplexity
- actions
- 57
- duration
- 1h 16m
- recorded
- 2026-08-03
run iddclm-nano-curation__AoiWXUF
Score
Summary
All three deliverables are in place and verified end-to-end.
Result
| Selection | Dev PPL |
|---|---|
| Random baseline (do-nothing) | 469.9 |
Shipped selection.json | 368.4 (−22%) |
Best draws of the same rule hit 359–362; draw-to-draw spread is ~±5, so I shipped the verbatim output of curate.py rather than cherry-picking a lucky draw, and said so in claim.md.
Method (submission/curate.py, one stated criterion)
- Target-likeness classifier — logistic regression over hashed word uni+bigrams (2^18 buckets, crc32 so it's seed-independent), positives = ~700-token chunks decoded from the disclosed dev target, negatives = 40k random 3000-char windows of pool docs. Fit full-batch on GPU. No labels, no external model.
- Whole-document scoring — score up to 4 evenly spaced 3000-char windows per doc; document score =
0.5·mean + 0.5·worst. The trainer eats the whole document, so one junk segment is penalized even when the opening looks clean. - Temperature sampling — draw 25k docs without replacement from the top 20% of the ranking with
p ∝ softmax(score), rather than strict top-K.
What the ablations showed (all measured with the frozen trainer)
- Whole-document scoring beats first-3000-chars scoring: 389.5 → 375.4.
- Sampling beats strict top-K despite lowering mean selection quality: 375.4 → 367.3; the knob is non-monotone (12% → 376.0, 20% → 367.3, 35% → 370.4).
- Hand-written junk/length/dedup filters on top of the classifier made things worse (389.5 → 396.4) — they cut register coverage the classifier already handled.
- Over-weighting the worst window (0.8) hurts: 373.4.
One thing to flag honestly: I initially split the dev target into quartiles assuming it was ordered by domain and trained per-domain classifiers — the quartiles weren't clean domains and that variant produced junk rankings, so I dropped it. A properly domain-balanced selection (cluster the dev chunks, balance per cluster) is the most promising untested direction, and I ran out of wall-clock before I could test it.
selection.json validity re-checked: 25,000 ids, all unique, all in range, packs to the full 12M tokens.