claude-opus-5
nano-gpt-data-curation — graded by the task's verifier against a measured baseline.
- backend
- modal
- variant
- claude claude-opus-5
- metric
- held-out multi-domain perplexity
- actions
- 53
- duration
- 1h 5m
- recorded
- 2026-08-03
run iddclm-nano-curation__Wz5kwvV
Score
Summary
All deliverables are in place and verified. Final state:
`/workspace/submission/selection.json` — 34,584 unique in-range ids (~24M tokens, 2× budget); the official pack_selection.py fills exactly 12,000,000 tokens from it. `/workspace/submission/curate.py` — reproduces that file byte-identically from a stated criterion (I switched the MinHash to CRC32 so it's not dependent on Python's randomized hash()). `/workspace/claim.md` — hypothesis, mechanism with a non-perplexity observable, falsification (including refuted sub-hypotheses), transfer.
What I found
Decoding the dev target showed it is literally four contiguous 250K-token blocks: WikiText-style encyclopedic text (with @,@ detokenization artifacts), HQ web prose, a news wire, and HTML-marked-up StackExchange Q&A. That turned the problem from "rank documents by quality" into "divide the budget across four registers".
The criterion: register-aware junk gates → per-register Naive-Bayes affinity (GPT-2-token unigram from each dev block vs. pool background) → 5-gram MinHash dedup → token-balanced round-robin, so every prefix of the list matches the target mixture regardless of where the pipeline truncates.
| selection | dev ppl (seed 1337) |
|---|---|
| random baseline | 485.5 |
| junk gate only | 430.9 |
| single blended "quality" score, best-first | 429.7 |
| register-balanced 25/25/25/25 | 315.7 |
| shipped (1:1:1:2.5) | 300.7 (293.0 at seed 4242) |
The decisive comparison is rows 3 vs 4 — identical gates and identical NB machinery, differing only in whether the budget is split by register: a 114-point swing with document quality held fixed.
Caveats I'd flag
- Seed noise is ~±10 ppl (same data: 296.5 vs 307.9 across seeds). I chose the weights by 2-seed means and picked the stable plateau interior (297.0, spread 0.5) rather than the single-seed winner, so anything below ~10 points in my sweep should be treated as tied.
- Three sub-hypotheses were refuted, and I've reported them as such: pseudo-relevance feedback on the profiles (319.1, no gain), restricting the technical bucket to genuine code/tech evidence (305.2, no gain — so the gain is broad structural text, not code), and matching the target's document-boundary density by capping doc length (415.9, much worse).
- The encyclopedic quarter is largely irreducible here: 0 of 182,016 pool docs carry WikiText's surface form, and its loss stays at ppl ~616 while the technical quarter reaches ~161. Attempts to sharpen encyclopedic retrieval bought nothing.
- Robustness to the hidden sample: on disjoint halves of each register the shipped selection gives 299.7 / 301.4 vs 314.6 / 316.1 for the uniform mixture — the same gap on independent samples.