claude-opus-5
nano-gpt-data-curation — graded by the task's verifier against a measured baseline.
- backend
- modal
- variant
- claude claude-opus-5
- metric
- held-out multi-domain perplexity
- actions
- 121
- duration
- 1h 17m
- recorded
- 2026-08-04
run iddclm-nano-curation__r4AKe3x
Score
Summary
Done. Final state of the three deliverables:
`/workspace/submission/selection.json` — 48,280 pool ids in priority order, 48,000,870 GPT-2 tokens (4.00× the budget). All ids unique, all present in the pool, range [1, 181978]. Valid.
`/workspace/submission/curate.py` — reruns end-to-end from pool.jsonl + multi_dev.npy and reproduced the submitted selection.json byte-identically this session. Criterion: per-register Jelinek-Mercer importance ratio over whitespace-aware word/bigram features, MinHash dedup, token-quota round-robin with QUOTA = [3,1,1,1]. No prose filter (measured to hurt).
`/workspace/claim.md` — hypothesis (5 claims), three per-register mechanism predictions with measurements, a no-training composition read-out, ablation tables, transfer.
Verified chain: pack_selection.py on the submitted selection.json produces a train array bit-identical to the one I trained on, which scored HELDOUT_PPL 313.161 on multi_dev.npy vs 485.4 random baseline (−35.5%).
The last stretch added two real gains beyond where the summary left off:
| step | dev PPL |
|---|---|
| whitespace-aware features, equal quota (prior best) | 316.7 |
| GPT-2 token-id features (checked, not adopted) | 316.2–316.6 |
| 3× quota on the encyclopedic register | 313.2 |
The quota tilt came from the mechanism rather than a sweep: the per-register decomposition showed encyclopedic text stuck at 743 ppl while technical Q&A sat at 144, so tokens should flow from the well-served register to the starved one. Measured result matched the prediction — Q0 −0.080, Q1 −0.031, Q2 −0.050 nats, paid for by Q3 +0.119 — and the sweep confirmed an interior optimum (1×/2×/3×/4×/6× → 316.7/314.6/313.2/316.6/319.8), so over-tilting costs as much as not tilting.
Two things I'd flag as honest limits: the 3× tilt is a ~3.5-point gain chosen on the dev sample, so some of it may not transfer to the hidden sample (the composition effect it exploits is a mixture-level property, which is why I expect most of it to survive); and the argmax token-share read-out shows the tilt barely raises the encyclopedic share (3.3% → 3.8%) — it works by drawing deeper into that register's ranking, and claim.md says so rather than papering over it.