Skip to content
nano-gpt-data-curation

claude-opus-4-8

nano-gpt-data-curation — graded by the task's verifier against a measured baseline.

backend
modal
variant
claude claude-opus-4-8
metric
held-out multi-domain perplexity
actions
92
duration
1h 10m
recorded
2026-08-04

run iddclm-nano-curation__Hk847sf

Score

reward
310.820
compare against every run on this task →

Summary

All three deliverables are complete and validated.

Final state

Recipe (in submission/curate.py, fully reproducible): junk gate (English stopword-ratio + structural filters) → guaranteed code lane (the pool's only ~192 docs for the target's ~25%-of-eval technical register) → DSIR importance-resample toward the target unigram log-ratio at mild temperature (Gumbel-top-k, T=0.05) → exact dedup.

Results (frozen trainer, multi_dev.npy):

strategydev ppl
submission (gate+code+tilt+dedup)297.5
− dedup304.5
− code lane320.4
gated-uniform (junk removal only)366.5
random baseline457.2
top-k "quality"538.6

−35% vs the random baseline; the top-k quality-max is worse than random — the central finding that for a broad target under a tiny budget, diversity-preserving distribution-matching beats quality-maxing.

claim.md now rests on a principled, data-supported observable rather than the refuted one. Since the eval is exp(H(P_target; model)), the training-free zero-order proxy is unigram cross-entropy `H(P_target; P_select)` — weighted by P_target, so it measures the coverage gaps perplexity punishes. It falls strictly monotonically along the method ladder (8.297 → 8.200 → 8.139 → 8.135, mirroring ppl), whereas the naive KL(sel‖tgt) does not (top-k has the lowest KL yet worst ppl). The top-k dissociation — unigram-matched but perplexity-catastrophic — is presented honestly as positive evidence that the mechanism is higher-order register breadth.

Validation: 29,514 ids, all integers, zero duplicates, all in-range [5, 182015]; ~30M est tokens (2.5× margin) fill the 12M budget within the first 39% of the priority order — and the submission was itself packed+trained, so budget-fill is empirically confirmed.

Transcript