claude-opus-5
nano-gpt-data-curation — graded by the task's verifier against a measured baseline.
- backend
- modal
- variant
- claude claude-opus-5
- metric
- held-out multi-domain perplexity
- actions
- 30
- duration
- 40m
- recorded
- 2026-08-03
run iddclm-nano-curation__zQZFhRN
Everything claude-opus-5 did on this task, in order.
download JSON ↗encrypted chain-of-thought — recorded, not exposed
args
{
"command": "sleep 200; grep -h HELD /tmp/e_tok.log /tmp/e_glob.log 2>/dev/null; tail -1 /tmp/e_glob.log; cat /workspace/.timer/remaining_secs"
}Bash result · error
Exit code 143
Command timed out after 2m 0s
[error] tool reported failureencrypted chain-of-thought — recorded, not exposed
args
{
"command": "sleep 90; grep -h HELD /tmp/e_tok.log /tmp/e_glob.log 2>/dev/null; tail -2 /tmp/e_glob.log; cat /workspace/.timer/remaining_secs"
}Bash result
HELDOUT_PPL 320.680 loss 5.7704 train_s 52 /tmp/dev_train.npy
HELDOUT_PPL 389.113 loss 5.9639 train_s 52 /tmp/dev_train.npy
(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)
DONE
3480
[stdout]
HELDOUT_PPL 320.680 loss 5.7704 train_s 52 /tmp/dev_train.npy
HELDOUT_PPL 389.113 loss 5.9639 train_s 52 /tmp/dev_train.npy
(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)
DONE
3480
[metadata] {"noOutputExpected": false}Results in. Finalizing with the best variant and recording the evidence:
args
{
"command": "cd /workspace && python -c \"\nimport json\ns=json.load(open('submission/selection.json'))\nids={json.loads(l)['id'] for l in open('data/pool.jsonl')}\nprint('n',len(s),'unique',len(set(s)),'all in pool',set(s)<=ids,'type',type(s[0]))\n\"; grep -c . /tmp/e_tok.log"
}Bash result
n 25535 unique 25535 all in pool True type <class 'int'>
4
[stdout]
n 25535 unique 25535 all in pool True type <class 'int'>
4
[metadata] {"noOutputExpected": false}encrypted chain-of-thought — recorded, not exposed
args
{
"command": "cd /workspace/submission && python - <<'EOF'\ns=open('curate.py').read()\ns=s.replace(''' for mode, path in [(\"doc\", OUT), (\"token\", \"/tmp/sel_token.json\"),\n (\"global\", \"/tmp/sel_global.json\")]:''',\n''' # \"doc\" (equal document counts per register) is the submitted selection:\n # measured dev PPL 314.05, vs 320.68 for token-balancing, 389.11 for the\n # unbalanced global ranking, and 457.50 for a random selection.\n for mode, path in [(\"doc\", OUT), (\"token\", \"/tmp/sel_token.json\"),\n (\"global\", \"/tmp/sel_global.json\")]:''')\nopen('curate.py','w').write(s)\n\nc=open('/workspace/claim.md').read()\nc += \"\"\"\n# Results (dev target, frozen trainer, seed 1337)\n\n| selection | dev PPL | final train loss signature |\n|---|---|---|\n| random pool sample (baseline) | 457.50 | — |\n| classifier score, single global ranking (no balancing) | 389.11 | 5.9639 heldout loss |\n| **register-balanced round-robin, equal doc counts (submitted)** | **314.05** | 5.7496 heldout loss |\n| register-balanced round-robin, equal token counts | 320.68 | 5.7704 heldout loss |\n\n**Verdict: hypothesis supported, not falsified.**\n- Curation beats random by 31% (457.50 -> 314.05), so falsification test 1 fails to fire.\n- Balancing is *not* inert: the same classifier scores pooled into one global\n ranking give 389.11 vs 314.05 balanced. The balancing step, which is the\n actual content of the claim, is worth 19% of perplexity on its own —\n more than half the total gain over random. Falsification test 2 fails to fire.\n- Observable (1) confirmed quantitatively: pairwise Jaccard of the top-1000\n documents of the four register classifiers is 0.000-0.046 (all < 0.05), and\n the Q&A classifier is *negatively* correlated with the encyclopedic one\n (r = -0.126) across the filtered pool. The four classifiers are measuring\n four different axes, not one shared quality axis.\n\nUnresolved: balancing by equal document counts slightly beat balancing by equal\ntoken counts (314.05 vs 320.68). The claim as stated does not predict which,\nand one seed cannot separate a 2% gap from run-to-run noise; the honest reading\nis that the two balancing rules are equivalent and both clearly beat no\nbalancing. Observable (3) (the train-loss signature) was not measured — the\nfrozen trainer reports only held-out loss, not final training loss, so that\nprediction stands untested.\n\"\"\"\nopen('/workspace/claim.md','w').write(c)\nEOF\nls -l /workspace/submission /workspace/claim.md"
}Bash result
-rw-r--r-- 1 root root 5474 Aug 3 23:58 /workspace/claim.md
/workspace/submission:
total 187
-rw-r--r-- 1 root root 8194 Aug 3 23:58 curate.py
-rw-r--r-- 1 root root 182051 Aug 3 23:48 selection.json
[stdout]
-rw-r--r-- 1 root root 5474 Aug 3 23:58 /workspace/claim.md
/workspace/submission:
total 187
-rw-r--r-- 1 root root 8194 Aug 3 23:58 curate.py
-rw-r--r-- 1 root root 182051 Aug 3 23:48 selection.json
[metadata] {"noOutputExpected": false}All three deliverables are in place and validated.
Approach
I decoded the disclosed dev target with the GPT-2 tokenizer and found it is four contiguous register blocks: wikitext-style encyclopedic, high-quality web prose (RAND-style reports), news, and StackExchange HTML Q&A. That turned curation into a distribution-matching problem rather than a generic quality-scoring problem.
curate.py then:
- Uses each register block as positives, random pool docs as negatives.
- Trains four separate hashed unigram+bigram logistic regressions (pure torch, sparse — no sklearn available) on GPU.
- Applies hard junk filters (length, alpha ratio, word/line duplication, boilerplate) — 158,227 of 182,016 docs survive.
- Selects round-robin across the four registers by descending score, so the 12M-token budget gets a balanced mix instead of collapsing onto whichever register the pool over-represents.
Measured results (frozen trainer, dev target)
| selection | dev PPL |
|---|---|
| random baseline | 457.50 |
| classifier score, one global ranking (balancing ablated) | 389.11 |
| register-balanced, equal doc counts — submitted | 314.05 |
| register-balanced, equal token counts | 320.68 |
31% below the random baseline. The key finding is the ablation: the same classifier scores pooled into a single ranking give 389.11, so the balancing step alone accounts for more than half the total gain — a single scalar quality score cannot express a mixture constraint.
The claim's pre-registered mechanism observable also held: pairwise Jaccard of the top-1000 docs across the four classifiers is 0.000–0.046, and the Q&A and encyclopedic scores are negatively correlated (r = −0.126). They measure four distinct axes, not one quality axis.
Two honest caveats recorded in claim.md: the doc-vs-token balancing gap (314.05 vs 320.68) is one seed and I don't claim it's real, and observable (3) — the train-loss signature — went untested because the frozen trainer reports only held-out loss.
selection.json verified valid: 25,535 unique in-range ids, packing to exactly 12,000,000 tokens.