nano-gpt-data-curation
claude-opus-4-8
nano-gpt-data-curation — graded by the task's verifier against a measured baseline.
- backend
- modal
- variant
- claude claude-opus-4-8
- metric
- held-out multi-domain perplexity
- actions
- 36
- duration
- 24m
- recorded
- 2026-08-04
run iddclm-nano-curation__Z4r6k8s
Score
reward
337.550
Summary
All deliverables are complete and validated.
I curated the pool using DSIR-style importance selection toward the disclosed high-quality target, with fluency gating.
Results (frozen trainer, 12M-token budget, dev target):
| Selection | Held-out PPL |
|---|---|
| Random baseline | 458.0 |
| Pure DSIR (no gates) | surfaced junk (taxonomic lists, tweets) |
| Final: DSIR + prose gates + dedup | 358.5 (~22% below baseline) |
Method (`curate.py`):
- Decode the dev target tokens → clean WikiText artifacts (
@-@,@,@) → build a hashed unigram+bigram language model of the target (q) and of the raw pool (p), using deterministiccrc32hashing (reproducible). - Score each doc by length-normalized log-likelihood ratio
mean_ngram log(q/p)— how much it "looks like" the HQ target vs generic web. - Fluency gates that fixed the key failure mode (pure DSIR ranked keyword-dense lists to the top): min length, alpha-char ratio ≥0.60, stop-word fraction in the fluent-prose band (0.20–0.60, kills lists/code), and ≥8 words/line.
- Emit best-first, dropping near-duplicate reposts so the budget spends on distinct docs.
Top-ranked docs are now clean news/encyclopedic prose (Reuters, wire copy) rather than junk.
Deliverables:
/workspace/submission/selection.json— 45,000 unique, in-range ids in priority order (covers 12M tokens; packs cleanly)./workspace/submission/curate.py— reproducible, criterion-driven./workspace/claim.md— hypothesis, mechanism (predicts higher stop-word fraction & lower target-KL in the selection vs random, an observable other than final PPL), falsification, and transfer.