claude-opus-4-8
nano-gpt-data-curation — graded by the task's verifier against a measured baseline.
- backend
- modal
- variant
- claude claude-opus-4-8
- metric
- held-out multi-domain perplexity
- actions
- 36
- duration
- 39m
- recorded
- 2026-08-04
run iddclm-nano-curation__FwSgZtE
Score
Summary
All deliverables are complete and validated. Here's the summary.
Result
| Selection | Held-out dev PPL | vs. random |
|---|---|---|
| Random baseline | 485.4 | — |
| Blind top-k similarity (DSIR) | 441.7 | −9% |
| Register-balanced (final) | 337.0 | −30% |
What I did
Diagnosis. Decoding the disclosed target (multi_dev.npy) showed it's a broad mix — WikiText-style Wikipedia, news, general web prose, and StackOverflow-style technical Q&A (~25% of tokens). A standard DSIR n-gram importance-weighting selection scored well but collapsed onto clean prose: only 51/11000 budget-filling docs contained any markup/code. The technical quarter of the target was essentially untrained.
Fix — register-balanced curation (curate.py):
- Quality gate: length / word-count / alpha-ratio / mean-word-length + exact-dedup (drops 17k junk docs).
- DSIR domain model, split by register: build hashed unigram+bigram Naive-Bayes models of the target's technical vs prose segments against a pool background; score every doc by mean per-token log-likelihood ratio.
- Reserve budget for the hard register: interleave prose and technical docs in priority order at a swept-optimal 15% technical quota.
Tuning. Sweeping the technical fraction gave a clean U-shape (0%→441, 15%→337, 30%→353, 40%→355), minimum at 15% — below the target's 25% share because raw-web technical docs are noisier than curated Q&A.
Mechanism, verified (in claim.md). Training prose-only vs 15%-technical and evaluating on per-register held-out splits: adding 15% technical data cut technical-segment perplexity 6.5× (1430→219) while prose barely moved (384→414). The improvement is concentrated entirely in the underrepresented register — confirming the predicted observable and satisfying the falsification test.
Deliverables: /workspace/submission/selection.json (40k valid, unique, priority-ordered ids), /workspace/submission/curate.py (self-contained, reproduces the selection exactly), and /workspace/claim.md. ~3300s of budget remain unused.