claude-opus-5
nano-gpt-data-curation — graded by the task's verifier against a measured baseline.
- backend
- modal
- variant
- claude claude-opus-5
- metric
- held-out multi-domain perplexity
- actions
- 34
- duration
- 38m
- recorded
- 2026-08-03
run iddclm-nano-curation__HBfU9js
Score
Summary
Done. All three deliverables are in place and the final selection is the best-measured configuration.
Result: dev perplexity 350.27 vs 457.99 for the random baseline (−24%).
Method (submission/curate.py, one 40s CPU pass, no GPU):
- Decode
multi_dev.npy, split into its four equal register blocks (wiki / general web prose / news / technical Q&A) → four positive unigram+bigram distributions. - Score every pool doc by length-normalized interpolated log-likelihood ratio
mean_t log(λ·p_d(t)/p_0(t) + 1−λ), λ=0.7, against a pool backgroundp_0. - Gate boilerplate (length, non-ASCII/markup fraction, type-token repetition), dedupe near-duplicate page families on head/tail word sequences.
- Assign each doc to its argmax register, greedily fill each register, emit token-balanced interleaved so every prefix of the list is an equal-parts mixture.
Ablations run (each a full frozen training run, recorded in claim.md): the add-1-smoothed variant of the same scorer scores 463 — worse than random, because unbounded per-token weights select vocabulary-novel nav-bar/link-farm pages; skewing the mixture (360), loosening the Q&A gate (365), and scarcity-first register ownership (359) all lost to the equal-parts argmax version.
Validity checks: 28,813 unique ids in range, packs to the full 12,000,000 tokens.