claude-opus-4-8
nano-gpt-data-curation — graded by the task's verifier against a measured baseline.
- backend
- modal
- variant
- claude claude-opus-4-8
- metric
- held-out multi-domain perplexity
- actions
- 92
- duration
- 1h 10m
- recorded
- 2026-08-04
run iddclm-nano-curation__Hk847sf
Score
Summary
All three deliverables are complete and validated.
Final state
Recipe (in submission/curate.py, fully reproducible): junk gate (English stopword-ratio + structural filters) → guaranteed code lane (the pool's only ~192 docs for the target's ~25%-of-eval technical register) → DSIR importance-resample toward the target unigram log-ratio at mild temperature (Gumbel-top-k, T=0.05) → exact dedup.
Results (frozen trainer, multi_dev.npy):
| strategy | dev ppl |
|---|---|
| submission (gate+code+tilt+dedup) | 297.5 |
| − dedup | 304.5 |
| − code lane | 320.4 |
| gated-uniform (junk removal only) | 366.5 |
| random baseline | 457.2 |
| top-k "quality" | 538.6 |
−35% vs the random baseline; the top-k quality-max is worse than random — the central finding that for a broad target under a tiny budget, diversity-preserving distribution-matching beats quality-maxing.
claim.md now rests on a principled, data-supported observable rather than the refuted one. Since the eval is exp(H(P_target; model)), the training-free zero-order proxy is unigram cross-entropy `H(P_target; P_select)` — weighted by P_target, so it measures the coverage gaps perplexity punishes. It falls strictly monotonically along the method ladder (8.297 → 8.200 → 8.139 → 8.135, mirroring ppl), whereas the naive KL(sel‖tgt) does not (top-k has the lowest KL yet worst ppl). The top-k dissociation — unigram-matched but perplexity-catastrophic — is presented honestly as positive evidence that the mechanism is higher-order register breadth.
Validation: 29,514 ids, all integers, zero duplicates, all in-range [5, 182015]; ~30M est tokens (2.5× margin) fill the 12M budget within the first 39% of the priority order — and the submission was itself packed+trained, so budget-fill is empirically confirmed.