claude-opus-4-8
nano-gpt-data-curation — graded by the task's verifier against a measured baseline.
- backend
- modal
- variant
- claude claude-opus-4-8
- metric
- held-out multi-domain perplexity
- actions
- 25
- duration
- 21m
- recorded
- 2026-08-04
run iddclm-nano-curation__YiLXzw2
Score
Summary
All deliverables are in place and validated. Summary:
Result
Held-out dev perplexity: 421.5 vs 485.5 random baseline (−13.2%).
Approach
The key insight: I have no quality labels, but multi_dev.npy decodes to a sample of the disclosed target distribution. I turned that into a supervised signal:
- Decode the target — the dev tokens decode to genuinely multi-register text (Wikipedia highway articles, casual web prose, blog essays, StackOverflow Q&A), confirming the broad target.
- Train a quality classifier — POSITIVES = decoded target docs, NEGATIVES = random pool sample. A hashed word unigram+bigram logistic regression written in pure numpy (no internet → sklearn/pip unavailable), the classic CCNet/GPT-3 quality-classifier pattern.
- Rank the whole pool by P(target), with a length floor pushing degenerate stubs to the back.
- Output all 182,016 ids in priority order — the packer fills 12M tokens from the best-scoring head.
Verified the ranking is interpretable (top = clean news/encyclopedic prose; bottom = promo/subscription stubs), near-duplicate waste is negligible (7/4000 in the head), and the selection is valid (182,016 unique in-range ids, fills the budget).
Deliverables
/workspace/submission/selection.json— ordered ids/workspace/submission/curate.py— reproducible, criterion-driven/workspace/claim.md— hypothesis / mechanism (score-separation + head/tail register, both observed pre-training) / falsification (not falsified) / transfer
One caveat noted in the claim: the pool is news-heavy, so the head skews toward news; the main transfer risk is register imbalance if the hidden target's mix differs sharply from dev's.