claude-opus-4-8
nano-gpt-data-curation — graded by the task's verifier against a measured baseline.
- backend
- modal
- variant
- claude claude-opus-4-8
- metric
- held-out multi-domain perplexity
- actions
- 36
- duration
- 42m
- recorded
- 2026-08-04
run iddclm-nano-curation__UFqhk3Z
Score
Summary
Everything is complete and validated. Here's the summary.
What I did
Task: Pick ~6% of a 182K-doc raw web pool (fixed 12M-token budget) to minimize held-out perplexity on a broad, multi-domain HQ English target.
Key discovery: Decoding the target showed it is an equal mix of 4 registers — Wikipedia, high-quality web prose, news, and technical Q&A (StackOverflow). The pool is raw web junk (forum posts, spam, boilerplate).
Criterion (in `curate.py`): Rank every doc by a DSIR-style bigram log-likelihood ratio in GPT-2 token space — mean[ log p_target(a,b) − log p_pool(a,b) ] — plus junk/length gates and exact dedup. This joint-bigram ratio decomposes into a conditional (fluency) term + a unigram (domain-vocabulary) term, so one score rewards fluent, in-domain English and starves the budget of pool junk.
How I validated (each = one 60s frozen train run):
| Selection | Held-out PPL |
|---|---|
| random baseline | 485.4 |
| unigram DSIR | 406.6 |
| target-likelihood / blend (rejected) | 618 / 567 |
| bigram DSIR — submitted | 337.8 |
- −30% perplexity vs. random, confirmed through the official
pack_selection.pypacker. - Generalization test (the crucial one): fitting the scorer on one interleaved half of the target and evaluating the model on the disjoint other half, bigram still beat unigram (329.5 vs 368.1) and random (~485) — proving it's domain matching, not memorization of the eval sample's bigrams (which the task explicitly warns about).
- Mechanism observable: the selected 12M tokens are distributionally closer to the target — KL(target‖selected)=0.477 vs KL(target‖random)=0.570.
Deliverables
/workspace/submission/selection.json— 41,829 unique valid ids in priority order (~36M tokens, comfortably fills the 12M budget)./workspace/submission/curate.py— reproducible bigram-DSIR criterion (tokenizes pool → builds target/pool bigram tables → scores → gates+dedup → ranks)./workspace/claim.md— Hypothesis / Mechanism (with the KL observable) / Falsification (4 concrete tests, all passed) / Transfer.