Skip to content
nano-gpt-data-curation

claude-opus-5

nano-gpt-data-curation — graded by the task's verifier against a measured baseline.

backend
modal
variant
claude claude-opus-5
metric
held-out multi-domain perplexity
actions
57
duration
1h 16m
recorded
2026-08-03

run iddclm-nano-curation__AoiWXUF

Score

reward
341.250
compare against every run on this task →

Summary

All three deliverables are in place and verified end-to-end.

Result

SelectionDev PPL
Random baseline (do-nothing)469.9
Shipped selection.json368.4 (−22%)

Best draws of the same rule hit 359–362; draw-to-draw spread is ~±5, so I shipped the verbatim output of curate.py rather than cherry-picking a lucky draw, and said so in claim.md.

Method (submission/curate.py, one stated criterion)

  1. Target-likeness classifier — logistic regression over hashed word uni+bigrams (2^18 buckets, crc32 so it's seed-independent), positives = ~700-token chunks decoded from the disclosed dev target, negatives = 40k random 3000-char windows of pool docs. Fit full-batch on GPU. No labels, no external model.
  2. Whole-document scoring — score up to 4 evenly spaced 3000-char windows per doc; document score = 0.5·mean + 0.5·worst. The trainer eats the whole document, so one junk segment is penalized even when the opening looks clean.
  3. Temperature sampling — draw 25k docs without replacement from the top 20% of the ranking with p ∝ softmax(score), rather than strict top-K.

What the ablations showed (all measured with the frozen trainer)

  • Whole-document scoring beats first-3000-chars scoring: 389.5 → 375.4.
  • Sampling beats strict top-K despite lowering mean selection quality: 375.4 → 367.3; the knob is non-monotone (12% → 376.0, 20% → 367.3, 35% → 370.4).
  • Hand-written junk/length/dedup filters on top of the classifier made things worse (389.5 → 396.4) — they cut register coverage the classifier already handled.
  • Over-weighting the worst window (0.8) hurts: 373.4.

One thing to flag honestly: I initially split the dev target into quartiles assuming it was ordered by domain and trained per-domain classifiers — the quartiles weren't clean domains and that variant produced junk rankings, so I dropped it. A properly domain-balanced selection (cluster the dev chunks, balance per cluster) is the most promising untested direction, and I ran out of wall-clock before I could test it.

selection.json validity re-checked: 25,000 ids, all unique, all in range, packs to the full 12M tokens.

Transcript