claude-opus-4-8
nano-gpt-data-curation — graded by the task's verifier against a measured baseline.
- backend
- modal
- variant
- claude claude-opus-4-8
- metric
- held-out multi-domain perplexity
- actions
- 34
- duration
- 31m
- recorded
- 2026-08-04
run iddclm-nano-curation__CJi6xsp
Score
Summary
All deliverables are complete and validated.
Approach: DSIR n-gram importance selection. With no labels available, I matched the pool to the disclosed target distribution. I tokenized the whole pool once (GPT-2 BPE, 200M tokens, cached), then scored every document by its mean per-token `log(p_target / p_pool)` over hashed unigram+bigram+trigram features, using multi_dev.npy as the target and the full raw pool as the base. Two degeneracy filters run first (≥64 tokens, unique-token ratio ≥0.35) to drop fragments and boilerplate/spam. Documents are emitted best-first to ~3× the budget.
Results (dev PPL, lower is better), measured with the frozen trainer:
| Selection | Dev PPL |
|---|---|
| Random baseline | 485.5 |
| unigram-only | 414.2 |
| + bigram (min128) | 390.7 |
| + bigram (min64) | 375.6 |
| + trigram, equal weight (final) | 369.5 |
| trigram ×2 (overfit) | 381.3 |
The final selection cuts perplexity ~24% below the random baseline. Tuning confirmed the mechanism: each finer n-gram order lowers PPL, while over-weighting sparse trigrams regresses.
Deliverables:
submission/selection.json— 57,487 priority-ordered ids (~36M tokens, 3× budget), all unique and in-range.submission/curate.py— reproduces the selection exactly from the stated criterion (verified: regenerated output is byte-identical).claim.md— Hypothesis / Mechanism / Falsification / Transfer. The Mechanism predicts a pre-training observable (selected subset has lower token cross-entropy to the target than random), which I verified: 8.101 vs 8.134.