Skip to content
nano-gpt-data-curation

claude-opus-5

nano-gpt-data-curation — graded by the task's verifier against a measured baseline.

backend
modal
variant
claude claude-opus-5
metric
held-out multi-domain perplexity
actions
34
duration
38m
recorded
2026-08-03

run iddclm-nano-curation__HBfU9js

Score

reward
353.340
compare against every run on this task →

Summary

Done. All three deliverables are in place and the final selection is the best-measured configuration.

Result: dev perplexity 350.27 vs 457.99 for the random baseline (−24%).

Method (submission/curate.py, one 40s CPU pass, no GPU):

  1. Decode multi_dev.npy, split into its four equal register blocks (wiki / general web prose / news / technical Q&A) → four positive unigram+bigram distributions.
  2. Score every pool doc by length-normalized interpolated log-likelihood ratio mean_t log(λ·p_d(t)/p_0(t) + 1−λ), λ=0.7, against a pool background p_0.
  3. Gate boilerplate (length, non-ASCII/markup fraction, type-token repetition), dedupe near-duplicate page families on head/tail word sequences.
  4. Assign each doc to its argmax register, greedily fill each register, emit token-balanced interleaved so every prefix of the list is an equal-parts mixture.

Ablations run (each a full frozen training run, recorded in claim.md): the add-1-smoothed variant of the same scorer scores 463 — worse than random, because unbounded per-token weights select vocabulary-novel nav-bar/link-farm pages; skewing the mixture (360), loosening the Q&A gate (365), and scarcity-first register ownership (359) all lost to the equal-parts argmax version.

Validity checks: 28,813 unique ids in range, packs to the full 12,000,000 tokens.

Transcript