nano-gpt-data-curation
claude-opus-4-8
nano-gpt-data-curation — graded by the task's verifier against a measured baseline.
- backend
- modal
- variant
- claude claude-opus-4-8
- metric
- held-out multi-domain perplexity
- actions
- 111
- duration
- 1h 9m
- recorded
- 2026-08-04
run iddclm-nano-curation__4uHWTbn
Score
reward
415.520
Summary
All deliverables are complete, consistent, and validated. Here's the summary.
Result
| Selection | Dev held-out PPL |
|---|---|
| Random baseline (do-nothing) | 474.86 |
| Classifier only | 412.28 |
| + English gate | 410.76 |
| + 25% tech quota | 407.53 |
| + 35% tech quota (final submission) | 404.59 |
Final selection is ~14.8% below the random baseline on the dev target (the official target is a disjoint sample of the same domain).
Method (submission/curate.py, reproducible & deterministic)
- Quality/domain classifier — a logistic regression trained on L2-normalized hashed unigram+bigram features to separate the disclosed HQ target (
multi_dev.npy, decoded) from random pool docs. Each pool doc is scored by "target-likeness." L2 normalization makes the score length/repetition-robust, avoiding the failure mode I first hit with raw DSIR log-ratio ranking (which floated license-plate/list spam to the top). - Light gates — length floor, alpha-fraction, English-only (drops Cyrillic junk that
isalpha()was leaking through). - Register balancing (the key second-order lever) — the target is equal parts four registers, but a plain top-by-score selection was ~48% news / ~51% prose and only 15 of ~11,000 technical docs — the technical-Q&A quarter was absent. Detecting code/tech docs and guaranteeing them ~35% of the budget recovered that register and lowered perplexity.
What I verified along the way
- A stronger stopword "fluency" gate regressed (430) — it was stripping the code/tech register. Removed it.
- Capping the news register (three-bucket 25/25/50) regressed (415) — news prose is genuinely valuable here. So the mechanism is "fill starved registers," not "mechanically equalize all four." Both findings are documented in
claim.md.
Deliverables
submission/selection.json— 27,817 priority-ordered ids, unique & in-range, packs to exactly 12M tokens.submission/curate.py(=/workspace/curate.py) — deterministic (stable crc32 hashing), reproduces the selection from the stated criterion.claim.md— Hypothesis / Mechanism (predicts register composition of the selected head, checkable without training) / Falsification / Transfer.
~1,440s of budget remain, but returns had flattened to sub-1-ppl noise, so I locked in the best measured configuration.