nano-gpt-data-curation

claude-opus-4-8

nano-gpt-data-curation — graded by the task's verifier against a measured baseline.

backend: modal
variant: claude claude-opus-4-8
metric: held-out multi-domain perplexity
actions: 84
duration: 1h 22m
recorded: 2026-08-04

run iddclm-nano-curation__ohgR2nu

Everything claude-opus-4-8 did on this task, in order.

download JSON ↗

claude-codeclaude-opus-4-884 turns46 narration84 action84 observation38 reasoning1 final

#76

narration

Now let me write claim.md with the mechanism data while the official verification runs.

toolWrite

args

{
  "file_path": "/workspace/claim.md",
  "content": "# Claim: register-mixture-matched curation beats global quality ranking\n\n## Setup / what was curated\nPool = 182,016 raw web docs (~200M GPT-2 tokens). Budget = 12M tokens (~6%).\nDisclosed target = a broad, *equal-parts* multi-domain HQ English sample\n(encyclopedic / HQ web prose / news / technical Q&A), given as `multi_dev.npy`.\n\n`curate.py` (stated criterion): (1) a cheap byte-level **English-prose quality\ngate** drops non-English / symbol- or digit-heavy / repetitive / too-short junk;\n(2) the target is clustered into **K=4 registers** (spherical k-means on GPT-2\ncontent-token TF); (3) each register gets a unigram importance profile\n`L_c = log p_c − log p_pool`, and the budget is filled by **weighted round-robin\nretrieval** — each register pulls its best-matching gated pool docs, in proportion\nto the register's token mass in the target. I.e. *train on the mixture you are\nevaluated on*, instead of taking the globally top-scoring docs.\n\n## Hypothesis\nUnder a scarce token budget and a *broad* target, aggregate held-out perplexity is\nminimized not by \"take the globally most target-like documents\" but by **matching\nthe target's register mixture** (after junk filtering). A single global importance\nscore concentrates the budget on whichever register is largest/loudest in the pool\n(here: news), and the aggregate is then **bottlenecked by the under-funded target\nregisters**.\n\n## Mechanism → predicted observable (NOT the final perplexity)\nDecompose held-out loss **by target register**. Prediction, made before measuring\nthe split: a global-importance selection is *imbalanced* — the dominant register\nhas low loss while ≥1 other target register has far higher loss — and\nmixture-matching **collapses the worst-covered register's loss at only marginal\ncost to the dominant one**; the aggregate gain is attributable to a specific\nbottleneck register, not a uniform shift.\n\nMeasured (same frozen recipe, per-register dev PPL; OVERALL reproduces the official\naggregate exactly, validating the probe):\n\n| selection            | reg0 | reg1 | reg2 (dominant) | reg3 | OVERALL |\n|----------------------|-----:|-----:|----------------:|-----:|--------:|\n| random               |  —   |  —   |       —         |  —   |  485.4  |\n| global-DSIR + gate   | 755  | 546  |   **280**       | 459  |  389.7  |\n| **balanced K=4**     | 754  | 590  |     282         |**178**| **310.6**|\n\ntarget register token mixture (K=4) = `[0.096, 0.15, 0.496, 0.258]`.\nThe 389.7→310.6 aggregate drop is driven almost entirely by **reg3: 459→178**\n(25.8% of target tokens, under-funded by global-DSIR), while the saturated\ndominant register reg2 is essentially unchanged (280→282). This is precisely the\npredicted \"fix the bottleneck register\" signature.\n\n## Falsification\n- If per-register losses under global-DSIR were already even, there is no\n  bottleneck to fix and mixture-matching should not help. (It helped by 79 ppl.)\n- The extra gain is **not** from the quality gate: global-DSIR uses the *same*\n  gate and still sits at 389.7; only re-allocating the mixture buys 389.7→310.6.\n- Sharpest test: a selection that *over*-weights the already-dominant register\n  (reg2) should **raise** aggregate PPL even though reg2's own loss falls. If\n  instead more-news kept lowering aggregate PPL, the mechanism is wrong.\n- If training on the single best-matching register alone (news-heavy top-DSIR)\n  matched the mixture-matched selection on this broad target, the claim is false\n  (observed opposite: 389.7 vs 310.6).\n\n## Transfer\n- **Hidden official target (disjoint sample, same domain):** registers and their\n  mixture are properties of the *domain*, not the dev draw; re-clustering the\n  official sample recovers the same structure. Predict the ordering\n  `random > global-DSIR > balanced` holds and balanced lands near ~310 (± noise).\n- **Other target mixtures / new domains:** re-cluster the new target and re-balance;\n  no pool labels required.\n- **Larger budget or model:** the advantage is largest in the scarce-budget regime\n  (6% of pool); as budget grows the two selections converge (budget eventually\n  covers every register), so mixture-matching is a small-data curation lever.\n- **Boundary (honest caveat):** balancing only helps where the *pool contains*\n  documents for a register. reg0 (an encyclopedic-like cluster) stays hard under\n  both (~754) because the raw web pool has few strong matches — you can match the\n  *achievable* mixture, not an arbitrary one.\n"
}

Write result

File created successfully at: /workspace/claim.md (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/workspace/claim.md", "content": "# Claim: register-mixture-matched curation beats global quality ranking\n\n## Setup / what was curated\nPool = 182,016 raw web docs (~200M GPT-2 tokens). Budget = 12M tokens (~6%).\nDisclosed target = a broad, *equal-parts* multi-domain HQ English sample\n(encyclopedic / HQ web prose / news / technical Q&A), given as `multi_dev.npy`.\n\n`curate.py` (stated criterion): (1) a cheap byte-level **English-prose quality\ngate** drops non-English / symbol- or digit-heavy / repetitive / too-short junk;\n(2) the target is clustered into **K=4 registers** (spherical k-means on GPT-2\ncontent-token TF); (3) each register gets a unigram importance profile\n`L_c = log p_c − log p_pool`, and the budget is filled by **weighted round-robin\nretrieval** — each register pulls its best-matching gated pool docs, in proportion\nto the register's token mass in the target. I.e. *train on the mixture you are\nevaluated on*, instead of taking the globally top-scoring docs.\n\n## Hypothesis\nUnder a scarce token budget and a *broad* target, aggregate held-out perplexity is\nminimized not by \"take the globally most target-like documents\" but by **matching\nthe target's register mixture** (after junk filtering). A single global importance\nscore concentrates the budget on whichever register is largest/loudest in the pool\n(here: news), and the aggregate is then **bottlenecked by the under-funded target\nregisters**.\n\n## Mechanism → predicted observable (NOT the final perplexity)\nDecompose held-out loss **by target register**. Prediction, made before measuring\nthe split: a global-importance selection is *imbalanced* — the dominant register\nhas low loss while ≥1 other target register has far higher loss — and\nmixture-matching **collapses the worst-covered register's loss at only marginal\ncost to the dominant one**; the aggregate gain is attributable to a specific\nbottleneck register, not a uniform shift.\n\nMeasured (same frozen recipe, per-register dev PPL; OVERALL reproduces the official\naggregate exactly, validating the probe):\n\n| selection            | reg0 | reg1 | reg2 (dominant) | reg3 | OVERALL |\n|----------------------|-----:|-----:|----------------:|-----:|--------:|\n| random               |  —   |  —   |       —         |  —   |  485.4  |\n| global-DSIR + gate   | 755  | 546  |   **280**       | 459  |  389.7  |\n| **balanced K=4**     | 754  | 590  |     282         |**178**| **310.6**|\n\ntarget register token mixture (K=4) = `[0.096, 0.15, 0.496, 0.258]`.\nThe 389.7→310.6 aggregate drop is driven almost entirely by **reg3: 459→178**\n(25.8% of target tokens, under-funded by global-DSIR), while the saturated\ndominant register reg2 is essentially unchanged (280→282). This is precisely the\npredicted \"fix the bottleneck register\" signature.\n\n## Falsification\n- If per-register losses under global-DSIR were already even, there is no\n  bottleneck to fix and mixture-matching should not help. (It helped by 79 ppl.)\n- The extra gain is **not** from the quality gate: global-DSIR uses the *same*\n  gate and still sits at 389.7; only re-allocating the mixture buys 389.7→310.6.\n- Sharpest test: a selection that *over*-weights the already-dominant register\n  (reg2) should **raise** aggregate PPL even though reg2's own loss falls. If\n  instead more-news kept lowering aggregate PPL, the mechanism is wrong.\n- If training on the single best-matching register alone (news-heavy top-DSIR)\n  matched the mixture-matched selection on this broad target, the claim is false\n  (observed opposite: 389.7 vs 310.6).\n\n## Transfer\n- **Hidden official target (disjoint sample, same domain):** registers and their\n  mixture are properties of the *domain*, not the dev draw; re-clustering the\n  official sample recovers the same structure. Predict the ordering\n  `random > global-DSIR > balanced` holds and balanced lands near ~310 (± noise).\n- **Other target mixtures / new domains:** re-cluster the new target and re-balance;\n  no pool labels required.\n- **Larger budget or model:** the advantage is largest in the scarce-budget regime\n  (6% of pool); as budget grows the two selections converge (budget eventually\n  covers every register), so mixture-matching is a small-data curation lever.\n- **Boundary (honest caveat):** balancing only helps where the *pool contains*\n  documents for a register. reg0 (an encyclopedic-like cluster) stays hard under\n  both (~754) because the raw web pool has few strong matches — you can match the\n  *achievable* mixture, not an arbitrary one.\n", "structuredPatch": [], "originalFile": null, "userModified": false}

#77

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolBashPoll official verification

args

{
  "command": "sleep 40; cat /tmp/final_verify.log 2>/dev/null | grep -vE \"Warning|Token indices\"; echo \"---timer---\"; cat /workspace/.timer/remaining_secs",
  "description": "Poll official verification"
}

Bash result

packed 12000000 tokens from 22284 ids -> /tmp/official_final.npy
packed 12000000 tokens from up to 22284 ids -> /tmp/fast_final.npy
official tokens 12000000 fast tokens 12000000 identical True
HELDOUT_PPL 310.611  loss 5.7385  train_s 54  /tmp/official_final.npy
FINAL_VERIFY_DONE
---timer---
2468

[stdout]
packed 12000000 tokens from 22284 ids -> /tmp/official_final.npy
packed 12000000 tokens from up to 22284 ids -> /tmp/fast_final.npy
official tokens 12000000 fast tokens 12000000 identical True
HELDOUT_PPL 310.611  loss 5.7385  train_s 54  /tmp/official_final.npy
FINAL_VERIFY_DONE
---timer---
2468
[metadata] {"noOutputExpected": false}

#78

narration

Official pipeline fully verified: pack_selection.py fills exactly 12,000,000 tokens from the deliverable, fast_pack is byte-identical to the official packer (so all my dev numbers are valid), and the authoritative dev PPL is 310.6 (vs random 485.4 — 36% lower). Let me run final validity guards on selection.json.

toolBashValidate selection.json and confirm deliverables

args

{
  "command": "cd /workspace; python3 -c \"\nimport json, numpy as np\nsel=json.load(open('/workspace/submission/selection.json'))\npoolids=set(np.load('/tmp/pool_ids.npy').tolist())\nprint('n ids           :', len(sel))\nprint('all unique      :', len(sel)==len(set(sel)))\nprint('all in pool      :', all(i in poolids for i in sel))\nprint('all python int   :', all(isinstance(i,int) for i in sel))\n# coverage in tokens\noff=np.load('/tmp/pool_offsets.npy'); ids=np.load('/tmp/pool_ids.npy'); pos={int(v):k for k,v in enumerate(ids)}\ncov=sum(int(off[pos[i]+1]-off[pos[i]])+1 for i in sel)\nprint('token coverage  :', cov, '(>=12M:', cov>=12_000_000, ')')\n\"\necho \"=== deliverables ===\"; ls -la /workspace/submission/selection.json /workspace/submission/curate.py /workspace/claim.md",
  "description": "Validate selection.json and confirm deliverables"
}

Bash result

n ids           : 22284
all unique      : True
all in pool      : True
all python int   : True
token coverage  : 14502146 (>=12M: True )
=== deliverables ===
-rw-r--r-- 1 root root   4496 Aug  4 04:11 /workspace/claim.md
-rw-r--r-- 1 root root   7498 Aug  4 03:56 /workspace/submission/curate.py
-rw-r--r-- 1 root root 157340 Aug  4 04:08 /workspace/submission/selection.json

[stdout]
n ids           : 22284
all unique      : True
all in pool      : True
all python int   : True
token coverage  : 14502146 (>=12M: True )
=== deliverables ===
-rw-r--r-- 1 root root   4496 Aug  4 04:11 /workspace/claim.md
-rw-r--r-- 1 root root   7498 Aug  4 03:56 /workspace/submission/curate.py
-rw-r--r-- 1 root root 157340 Aug  4 04:08 /workspace/submission/selection.json
[metadata] {"noOutputExpected": false}

#79

narration

All validity guards pass: 22,284 unique in-range ids, 14.5M-token coverage (fills the 12M budget), and all three deliverables are in place. The result is fully validated end-to-end. Let me finalize tasks and save a memory of the non-obvious findings.

toolTaskUpdate

args

{
  "taskId": "3",
  "status": "completed"
}

TaskUpdate result

Updated task #3 status

[metadata] {"success": true, "taskId": "3", "updatedFields": ["status"], "statusChange": {"from": "in_progress", "to": "completed"}}

#80

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolTaskUpdate

args

{
  "taskId": "4",
  "status": "completed"
}

TaskUpdate result

Updated task #4 status

[metadata] {"success": true, "taskId": "4", "updatedFields": ["status"], "statusChange": {"from": "pending", "to": "completed"}}

#81

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolBashCheck existing memory

args

{
  "command": "ls /logs/agent/sessions/projects/-workspace/memory/ 2>/dev/null; echo \"---\"; cat /logs/agent/sessions/projects/-workspace/memory/MEMORY.md 2>/dev/null | head",
  "description": "Check existing memory"
}

Bash result

---

[stdout]
---
[metadata] {"noOutputExpected": false}

#82

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolWrite

args

{
  "file_path": "/logs/agent/sessions/projects/-workspace/memory/curate-corpus-small-lm.md",
  "content": "---\nname: curate-corpus-small-lm\ndescription: Winning method + gotchas for the /workspace data-curation benchmark (pick pool docs to train a 30M GPT under a 12M-token budget, min held-out PPL on a multi-domain target)\nmetadata:\n  type: project\n---\n\nTask: choose pool ids (`/workspace/data/pool.jsonl`, 182,016 docs / ~200M GPT-2 tokens)\nto fill a 12M-token budget; frozen `train_nano.py` trains a 30M GPT and scores held-out\nPPL on a BROAD multi-domain target (equal parts Wikipedia / HQ web / news / technical Q&A),\ndisclosed as `data/multi_dev.npy` (1M GPT-2 tokens, EOS=50256 separated).\n\n**Winning recipe (dev PPL, lower=better): random 485 → global-DSIR+gate 390 → register-balanced K=4 311.**\n`curate.py`: (1) byte-level English-prose quality gate; (2) k-means the target into K=4\nregisters on content-token TF; (3) per-register unigram importance `L_c=log p_c−log p_pool`;\n(4) weighted round-robin retrieval fills the budget so the selection matches the target's\nregister token-mixture `[0.096,0.15,0.496,0.258]`. Mechanism confirmed via per-register dev\nloss: global-DSIR starves under-funded registers (reg3 PPL 459); balancing fixes the\nbottleneck (reg3 459→178) → drives the whole aggregate gain. K sweep flat 309–312 for K≥4;\nK=4 chosen (matches the 4 disclosed registers, defensible). **Why:** for a broad target the\naggregate is bottlenecked by the worst-covered register, so match-the-mixture > take-the-top.\n\nGOTCHAS (cost real time):\n- Text-mode `for line in open(big.jsonl)` throws `OSError [Errno 22]` under gVisor on the\n  800MB pool. Fix: **open in binary mode** (`open(path,'rb')`) — 1.8s for full pass.\n- `bytes.translate(None, DEL_NONX)` LEAVES X (deletes the complement); do NOT subtract from\n  len — that gives the complement count. Bug made alpha/mean_wlen look impossibly low.\n- ~69% of pool docs contain literal `<|endoftext|>` (token 50256) as internal separators —\n  harmless for training; do NOT gate them out (dropping them guts most clusters).\n- `eval.sh` name collision: `train_nano --out_json /tmp/X.json` will clobber a selection file\n  named `/tmp/X.json` AFTER packing (numbers still valid). Use distinct out_json paths.\n- 1TB RAM, H100. Full GPU-tokenize of pool ~100s (TOKENIZERS_PARALLELISM=true; ~350s if false).\n  One frozen train run ~55s. fast_pack (from cached tokens) is byte-identical to pack_selection.py.\n"
}

Write result

File created successfully at: /logs/agent/sessions/projects/-workspace/memory/curate-corpus-small-lm.md

[metadata] {"type": "create", "filePath": "/logs/agent/sessions/projects/-workspace/memory/curate-corpus-small-lm.md", "content": "---\nname: curate-corpus-small-lm\ndescription: \"Winning method + gotchas for the /workspace data-curation benchmark (pick pool docs to train a 30M GPT under a 12M-token budget, min held-out PPL on a multi-domain target)\"\nmetadata: \n  node_type: memory\n  type: project\n  originSessionId: fd821a6f-a260-47da-a909-e4a257c63ee5\n  modified: 2026-08-04T04:14:20.827Z\n---\n\nTask: choose pool ids (`/workspace/data/pool.jsonl`, 182,016 docs / ~200M GPT-2 tokens)\nto fill a 12M-token budget; frozen `train_nano.py` trains a 30M GPT and scores held-out\nPPL on a BROAD multi-domain target (equal parts Wikipedia / HQ web / news / technical Q&A),\ndisclosed as `data/multi_dev.npy` (1M GPT-2 tokens, EOS=50256 separated).\n\n**Winning recipe (dev PPL, lower=better): random 485 → global-DSIR+gate 390 → register-balanced K=4 311.**\n`curate.py`: (1) byte-level English-prose quality gate; (2) k-means the target into K=4\nregisters on content-token TF; (3) per-register unigram importance `L_c=log p_c−log p_pool`;\n(4) weighted round-robin retrieval fills the budget so the selection matches the target's\nregister token-mixture `[0.096,0.15,0.496,0.258]`. Mechanism confirmed via per-register dev\nloss: global-DSIR starves under-funded registers (reg3 PPL 459); balancing fixes the\nbottleneck (reg3 459→178) → drives the whole aggregate gain. K sweep flat 309–312 for K≥4;\nK=4 chosen (matches the 4 disclosed registers, defensible). **Why:** for a broad target the\naggregate is bottlenecked by the worst-covered register, so match-the-mixture > take-the-top.\n\nGOTCHAS (cost real time):\n- Text-mode `for line in open(big.jsonl)` throws `OSError [Errno 22]` under gVisor on the\n  800MB pool. Fix: **open in binary mode** (`open(path,'rb')`) — 1.8s for full pass.\n- `bytes.translate(None, DEL_NONX)` LEAVES X (deletes the complement); do NOT subtract from\n  len — that gives the complement count. Bug made alpha/mean_wlen look impossibly low.\n- ~69% of pool docs contain literal `<|endoftext|>` (token 50256) as internal separators —\n  harmless for training; do NOT gate them out (dropping them guts most clusters).\n- `eval.sh` name collision: `train_nano --out_json /tmp/X.json` will clobber a selection file\n  named `/tmp/X.json` AFTER packing (numbers still valid). Use distinct out_json paths.\n- 1TB RAM, H100. Full GPU-tokenize of pool ~100s (TOKENIZERS_PARALLELISM=true; ~350s if false).\n  One frozen train run ~55s. fast_pack (from cached tokens) is byte-identical to pack_selection.py.\n", "structuredPatch": [], "originalFile": null, "userModified": false, "memdirStamped": true}

#83

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolWrite

args

{
  "file_path": "/logs/agent/sessions/projects/-workspace/memory/MEMORY.md",
  "content": "- [Curate corpus for small LM](curate-corpus-small-lm.md) — /workspace data-selection benchmark: register-balanced curation (dev PPL 311 vs 485 random) + gVisor/byte-translate gotchas\n"
}

Write result

File created successfully at: /logs/agent/sessions/projects/-workspace/memory/MEMORY.md (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/logs/agent/sessions/projects/-workspace/memory/MEMORY.md", "content": "- [Curate corpus for small LM](curate-corpus-small-lm.md) — /workspace data-selection benchmark: register-balanced curation (dev PPL 311 vs 485 random) + gVisor/byte-translate gotchas\n", "structuredPatch": [], "originalFile": null, "userModified": false}

#84

narration

All deliverables are complete and validated end-to-end. Final status:

toolBashConfirm timer and deliverables

args

{
  "command": "echo \"Time remaining: $(cat /workspace/.timer/remaining_secs)s\"; echo \"=== Deliverables ===\"; ls -la /workspace/submission/selection.json /workspace/submission/curate.py /workspace/claim.md",
  "description": "Confirm timer and deliverables"
}

Bash result

Time remaining: 2018s
=== Deliverables ===
-rw-r--r-- 1 root root   4496 Aug  4 04:11 /workspace/claim.md
-rw-r--r-- 1 root root   7498 Aug  4 03:56 /workspace/submission/curate.py
-rw-r--r-- 1 root root 157340 Aug  4 04:08 /workspace/submission/selection.json

[stdout]
Time remaining: 2018s
=== Deliverables ===
-rw-r--r-- 1 root root   4496 Aug  4 04:11 /workspace/claim.md
-rw-r--r-- 1 root root   7498 Aug  4 03:56 /workspace/submission/curate.py
-rw-r--r-- 1 root root 157340 Aug  4 04:08 /workspace/submission/selection.json
[metadata] {"noOutputExpected": false}

final summary

Done. All three deliverables are in place and verified against the official frozen pipeline.

Result

selection	dev held-out PPL
random baseline	485.4
global quality/importance ranking + gate	389.7
register-balanced (shipped)	310.6

36% below the random baseline. Verified with the official pack_selection.py + train_nano.py (not just my fast packer, which I confirmed is byte-identical): the selection packs to exactly 12,000,000 tokens and scores 310.611. Validity guards pass — 22,284 ids, all unique, all in-range, 14.5M-token coverage.

Method (`curate.py`, self-contained & reproducible)

Quality gate — cheap byte-level features drop non-English / symbol- or digit-heavy / repetitive / too-short junk (keeps 85%).
Register model — k-means the disclosed target into K=4 registers (matching the 4 disclosed domains); each gets a unigram importance profile L_c = log p_c − log p_pool.
Balanced retrieval — weighted round-robin fills the budget so the selection's register mixture matches the target's mixture ([0.10, 0.15, 0.50, 0.26]), instead of letting one loud register (news) dominate.

Why it works (the claim's observable, measured — not the final number)

I decomposed held-out loss by register. A global importance score over-serves the dominant register (reg2 PPL 280) but starves the others (reg3 PPL 459). Balancing collapses the bottleneck register — reg3: 459 → 178 — at ~zero cost to the saturated one (280 → 282), and that single fix accounts for essentially the entire 389.7 → 310.6 aggregate drop. The aggregate on a broad target is bottlenecked by its worst-covered register; matching the mixture beats taking the top.

Honest boundary noted in claim.md: balancing only helps where the pool has documents for a register — reg0 (encyclopedic-like) stays hard (~754) under both because the raw web pool has few strong matches.

A K sweep (3–20) was flat at 309–312 for K≥4; I chose the principled K=4. ~34 min of budget remain, but further tuning would chase noise-level differences and risk overfitting the disjoint official target, so I'm locking in the validated, well-understood result.

Result

Method (curate.py, self-contained & reproducible)

Why it works (the claim's observable, measured — not the final number)

Method (`curate.py`, self-contained & reproducible)