claude-opus-4-8
nano-gpt-data-curation — graded by the task's verifier against a measured baseline.
- backend
- modal
- variant
- claude claude-opus-4-8
- metric
- held-out multi-domain perplexity
- actions
- 50
- duration
- 45m
- recorded
- 2026-08-04
run iddclm-nano-curation__aQTkr5P
Score
Summary
All deliverables are finalized and validated. Here's the summary.
Result
I curated the pool with a DSIR-style n-gram importance filter and drove held-out dev perplexity to 356.5, a ~27% reduction below the random baseline (485.5).
Approach
The disclosed dev target (multi_dev.npy) is GPT-2 tokens, so I decoded it back to text and used it as the positive reference for the target domain (broad HQ English — encyclopedic/news/prose/Q&A). Then:
- Estimate distributions: smoothed unigram+bigram counts from the decoded target vs. the raw pool (background).
- Rank each document by mean per-token log-likelihood ratio
log P_target(w) − log P_background(w)(bigram term weighted 0.5). - Filter out ill-formed docs (too short, symbol soup, menu/list boilerplate, repetitive) — drops ~45% of the pool.
- Emit pool ids in descending score, best-first.
What I tried (all evaluated on the frozen trainer)
| Method | Dev PPL |
|---|---|
| Random baseline | 485.5 |
| Hashed logistic-regression classifier | 462–493 (noisy) |
| Coverage-weighted / clipped DSIR (v3) | 372.6 |
| DSIR bigram-weight 1.0 (v4) | 366.8 |
| DSIR bigram-weight 0.5 (submitted) | 356.5 |
Key finding: signal quality matters — collision-based hashed features were within noise of random, while explicit n-gram statistics delivered the large, robust gain.
Deliverables
/workspace/submission/selection.json— 100,720 priority-ordered ids (validated: unique, in range, covers 12M tokens)./workspace/submission/curate.py— deterministic, reproduces the selection from the stated criterion./workspace/claim.md— hypothesis / mechanism / falsification / transfer, with confirmed numbers.
~48 min of budget remain, but variants have plateaued above the winning 356.5, so I'm finalizing here.