claude-opus-4-8
nano-gpt-data-curation — graded by the task's verifier against a measured baseline.
- backend
- modal
- variant
- claude claude-opus-4-8
- metric
- held-out multi-domain perplexity
- actions
- 37
- duration
- 59m
- recorded
- 2026-08-04
run iddclm-nano-curation__LYgVBSB
Score
Summary
All deliverables are complete and verified. Summary:
Result
Held-out perplexity on the dev target: 413.0, vs 459.8 for a random selection — ~10% lower (lower is better).
What I found
The disclosed target is broad (Wikipedia + HQ web + news + technical Q&A) and the budget is tiny (12M tokens, ~6% of the pool). I tested the intuitive approach first — rank documents by resemblance to the target — and every ranked variant was worse than random:
| approach | PPL |
|---|---|
| Fluency (target-LM) ranking | 579 |
| Naive-Bayes target-vs-pool ranking | 528 |
| random baseline | 460 |
| junk-filter + dedup + uniform sample (submission) | 413 |
Quality-ranking is self-defeating under a broad target + small budget: it concentrates the budget on the single most "fluent" slice and collapses register coverage (vocabulary shrank 63%; the code-bearing technical-Q&A register dropped from 5% → 1%). Coverage, not resemblance, is what a broad target rewards.
The winning recipe (curate.py)
- Loose junk filter — drop only clear garbage (non-English by function-word ratio, symbol-soup, single-token spam, numeric dumps, too-short). Thresholds deliberately loose so every genuine register survives.
- Near-dup removal — normalized head signature; templated web dupes waste budget.
- Uniform (seeded) sampling of survivors to fill the budget — preserves the pool's natural register mix.
Self-contained, runs in ~67s, emits 23,473 valid unique ids (~24M tokens).
Deliverables
/workspace/submission/selection.json— 23,473 priority-ordered ids (validated: unique, in-range, >12M tokens)/workspace/submission/curate.py— reproducible, stated-criterion script (reproduces PPL 412.98 exactly)/workspace/claim.md— Hypothesis / Mechanism (predicts diversity & code-register fraction track PPL, confirmed before measuring PPL) / Falsification / Transfer