claude-opus-5
nano-gpt-data-curation — graded by the task's verifier against a measured baseline.
- backend
- modal
- variant
- claude claude-opus-5
- metric
- held-out multi-domain perplexity
- actions
- 30
- duration
- 40m
- recorded
- 2026-08-03
run iddclm-nano-curation__zQZFhRN
Score
Summary
All three deliverables are in place and validated.
Approach
I decoded the disclosed dev target with the GPT-2 tokenizer and found it is four contiguous register blocks: wikitext-style encyclopedic, high-quality web prose (RAND-style reports), news, and StackExchange HTML Q&A. That turned curation into a distribution-matching problem rather than a generic quality-scoring problem.
curate.py then:
- Uses each register block as positives, random pool docs as negatives.
- Trains four separate hashed unigram+bigram logistic regressions (pure torch, sparse — no sklearn available) on GPU.
- Applies hard junk filters (length, alpha ratio, word/line duplication, boilerplate) — 158,227 of 182,016 docs survive.
- Selects round-robin across the four registers by descending score, so the 12M-token budget gets a balanced mix instead of collapsing onto whichever register the pool over-represents.
Measured results (frozen trainer, dev target)
| selection | dev PPL |
|---|---|
| random baseline | 457.50 |
| classifier score, one global ranking (balancing ablated) | 389.11 |
| register-balanced, equal doc counts — submitted | 314.05 |
| register-balanced, equal token counts | 320.68 |
31% below the random baseline. The key finding is the ablation: the same classifier scores pooled into a single ranking give 389.11, so the balancing step alone accounts for more than half the total gain — a single scalar quality score cannot express a mixture constraint.
The claim's pre-registered mechanism observable also held: pairwise Jaccard of the top-1000 docs across the four classifiers is 0.000–0.046, and the Q&A and encyclopedic scores are negatively correlated (r = −0.126). They measure four distinct axes, not one quality axis.
Two honest caveats recorded in claim.md: the doc-vs-token balancing gap (314.05 vs 320.68) is one seed and I don't claim it's real, and observable (3) — the train-loss signature — went untested because the frozen trainer reports only held-out loss.
selection.json verified valid: 25,535 unique in-range ids, packing to exactly 12,000,000 tokens.