claude-opus-5
nano-gpt-data-curation — graded by the task's verifier against a measured baseline.
- backend
- modal
- variant
- claude claude-opus-5
- metric
- held-out multi-domain perplexity
- actions
- 39
- duration
- 46m
- recorded
- 2026-08-03
run iddclm-nano-curation__KmpadEA
Score
Summary
Done. All three deliverables are in place.
Approach. submission/curate.py decodes the disclosed dev target (multi_dev.npy) back into ~2.5k target-domain documents, uses them as positives against 20k random pool documents as negatives, and fits a logistic regression over tf-idf-weighted hashed word 1–2grams (implemented in torch — no sklearn offline). The resulting P(target | doc) approximates the density ratio p_target/p_pool. Hard filters (length, alphabetic ratio, repetition, mean word length, line dedup, sentence punctuation) and a cheap near-duplicate signature run on top, and documents are emitted in descending score order.
Measured on the frozen trainer (12M tokens, dev target):
| selection | dev PPL |
|---|---|
| random (do-nothing anchor) | 457.5 |
| classifier-ranked (submitted) | 357.0 |
| + k-means register balancing (K=8) | 363.1 |
| + min length 1200 chars | 368.4 |
| + self-training round 2 | 443.8 |
−22% versus random. The two refinements I expected to help both hurt; they're kept in the script but disabled by default (BALANCE=0, ROUND2=0) so the negative results reproduce. claim.md states the hypothesis, the non-perplexity observables (0.999 linear separability; top-of-ranking is visibly encyclopedic/news/expository), the falsification test, and transfer.
selection.json validated: 52,550 ids, all unique and in range, packs to the full 12M-token budget.