nano-gpt-data-curation

claude-opus-4-8

nano-gpt-data-curation — graded by the task's verifier against a measured baseline.

backend: modal
variant: claude claude-opus-4-8
metric: held-out multi-domain perplexity
actions: 79
duration: 1h 5m
recorded: 2026-08-04

run iddclm-nano-curation__ETg2M2Y

Everything claude-opus-4-8 did on this task, in order.

download JSON ↗

claude-codeclaude-opus-4-879 turns39 narration79 action79 observation40 reasoning1 final

#76

narration

Confirmed end-to-end: the shipped submission/selection.json packs exactly 12,000,000 tokens and trains to HELDOUT_PPL 302.46, matching the reproducible pipeline. Let me close out tasks and record the reusable finding.

toolTaskUpdate

args

{
  "taskId": "2",
  "status": "completed"
}

TaskUpdate result

Updated task #2 status

[metadata] {"success": true, "taskId": "2", "updatedFields": ["status"], "statusChange": {"from": "pending", "to": "completed"}}

#77

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolTaskUpdate

args

{
  "taskId": "3",
  "status": "completed"
}

TaskUpdate result

Updated task #3 status

[metadata] {"success": true, "taskId": "3", "updatedFields": ["status"], "statusChange": {"from": "pending", "to": "completed"}}

#78

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolWrite

args

{
  "file_path": "/logs/agent/sessions/projects/-workspace/memory/pretrain-curation-length-cap.md",
  "content": "---\nname: pretrain-curation-length-cap\ndescription: Under a tight pretraining token budget, capping document length beats no cap — diversity lever\nmetadata:\n  type: reference\n---\n\nFixed-budget LM pretraining-data curation (select ~6% of a raw web pool to hit a\nbroad HQ multi-domain target, minimise held-out perplexity). What worked, in\norder of impact, on a frozen 30M-GPT / 12M-token eval:\n\n1. **DSIR domain matching + coherence gate** (random 485 → 324). Decode the\n   disclosed target tokens to text, fit target vs pool n-gram models, rank docs\n   by per-token log(p_target/p_pool) (unigram + 2×bigram). Gate out spam/foreign\n   with: min words, function-word frac, and especially **fraction of bigrams\n   attested in the target model** (word-salad SEO spam ~0.05-0.11 vs real text\n   0.44-0.70 — clean separator).\n2. **Document length cap** (324 → 302, the biggest single win). A few mega-docs\n   (up to ~130k tokens) waste a tiny budget on narrow topics. Cap doc length →\n   budget buys topical diversity. U-shaped: no-cap 324 / 36k 314 / **24k 302** /\n   18k 309 / 12k 309. Optimum ~24k chars.\n3. **Do NOT force equal-per-register balance** (hurt: 330-333 > 324). Held-out\n   perplexity is an *average* loss dominated by the hardest register (technical/\n   code); the importance ratio already invests budget there. Balancing dilutes it.\n\nUnsupervised k-means clustering of the target to recover its 4 registers FAILED\n(over-split the large Wikipedia block into 3 clusters). If you need registers,\nuse contiguous token slices of the target, not clustering.\n"
}

Write result

File created successfully at: /logs/agent/sessions/projects/-workspace/memory/pretrain-curation-length-cap.md

[metadata] {"type": "create", "filePath": "/logs/agent/sessions/projects/-workspace/memory/pretrain-curation-length-cap.md", "content": "---\nname: pretrain-curation-length-cap\ndescription: \"Under a tight pretraining token budget, capping document length beats no cap — diversity lever\"\nmetadata: \n  node_type: memory\n  type: reference\n  originSessionId: 8d3f2e42-7a38-4920-b1b1-3dc7521e7d7f\n  modified: 2026-08-04T04:59:38.971Z\n---\n\nFixed-budget LM pretraining-data curation (select ~6% of a raw web pool to hit a\nbroad HQ multi-domain target, minimise held-out perplexity). What worked, in\norder of impact, on a frozen 30M-GPT / 12M-token eval:\n\n1. **DSIR domain matching + coherence gate** (random 485 → 324). Decode the\n   disclosed target tokens to text, fit target vs pool n-gram models, rank docs\n   by per-token log(p_target/p_pool) (unigram + 2×bigram). Gate out spam/foreign\n   with: min words, function-word frac, and especially **fraction of bigrams\n   attested in the target model** (word-salad SEO spam ~0.05-0.11 vs real text\n   0.44-0.70 — clean separator).\n2. **Document length cap** (324 → 302, the biggest single win). A few mega-docs\n   (up to ~130k tokens) waste a tiny budget on narrow topics. Cap doc length →\n   budget buys topical diversity. U-shaped: no-cap 324 / 36k 314 / **24k 302** /\n   18k 309 / 12k 309. Optimum ~24k chars.\n3. **Do NOT force equal-per-register balance** (hurt: 330-333 > 324). Held-out\n   perplexity is an *average* loss dominated by the hardest register (technical/\n   code); the importance ratio already invests budget there. Balancing dilutes it.\n\nUnsupervised k-means clustering of the target to recover its 4 registers FAILED\n(over-split the large Wikipedia block into 3 clusters). If you need registers,\nuse contiguous token slices of the target, not clustering.\n", "structuredPatch": [], "originalFile": null, "userModified": false, "memdirStamped": true}

#79

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolBashUpdate MEMORY.md index, verify deliverables

args

{
  "command": "cd /logs/agent/sessions/projects/-workspace/memory/; if [ -f MEMORY.md ]; then echo \"exists\"; else echo \"creating\"; fi\npython3 - <<'PY'\nimport os\np=\"MEMORY.md\"\nline=\"- [Pretraining data curation](pretrain-curation-length-cap.md) — DSIR domain-match + doc-length cap for diversity; don't force register balance\\n\"\nif os.path.exists(p):\n    c=open(p).read()\n    if \"pretrain-curation-length-cap\" not in c:\n        open(p,\"a\").write(line)\nelse:\n    open(p,\"w\").write(\"# Memory index\\n\\n\"+line)\nprint(\"done\")\nPY\nls -la /workspace/submission/ /workspace/claim.md",
  "description": "Update MEMORY.md index, verify deliverables"
}

Bash result

creating
done
-rw-r--r-- 1 root root 3836 Aug  4 04:57 /workspace/claim.md

/workspace/submission/:
total 1067
drwxr-xr-x 1 root root     380 Aug  4 04:51 .
drwxr-xr-x 1 root root     200 Aug  4 04:57 ..
drwxr-xr-x 1 root root      80 Aug  4 04:38 __pycache__
-rw-r--r-- 1 root root    6639 Aug  4 04:51 curate.py
-rw-r--r-- 1 root root 1085680 Aug  4 04:53 selection.json
Shell cwd was reset to /workspace

[stdout]
creating
done
-rw-r--r-- 1 root root 3836 Aug  4 04:57 /workspace/claim.md

/workspace/submission/:
total 1067
drwxr-xr-x 1 root root     380 Aug  4 04:51 .
drwxr-xr-x 1 root root     200 Aug  4 04:57 ..
drwxr-xr-x 1 root root      80 Aug  4 04:38 __pycache__
-rw-r--r-- 1 root root    6639 Aug  4 04:51 curate.py
-rw-r--r-- 1 root root 1085680 Aug  4 04:53 selection.json
[stderr]

Shell cwd was reset to /workspace
[metadata] {"noOutputExpected": false}

final summary

All deliverables are in place and validated. Done.

Summary

I curated the pretraining pool by treating the disclosed dev target as a distribution to match, then optimizing for diversity under the tight budget.

Approach (in `submission/curate.py`, fully reproducible):

Domain matching (DSIR-style): decoded the dev target tokens to text, fit target vs. pool n-gram models, and ranked every doc by per-token log(p_target/p_pool) (unigrams + 2×bigrams).
Coherence gate: dropped junk using a fraction-of-bigrams-attested-in-target filter — this cleanly separates multilingual SEO word-salad (0.05–0.11) from all four real registers (0.44–0.70), which a pure unigram score otherwise ranks at the very top.
Length cap (24k chars): the single biggest win — under a 12M-token budget, a few sprawling mega-documents (up to ~130k tokens each) waste the budget on narrow topics; capping length buys topical diversity.

Validated results (frozen 30M-GPT, dev perplexity, lower is better):

Selection	Dev PPL
Random baseline	485.4
Domain-matched, quality-first	324.3
Round-robin register balance	333.1
Capped register balance	329.6
+ 24k-char length cap (shipped)	302.5

Key finding, against my initial intuition: forcing equal-per-register balance hurt (330–333 vs 324). Held-out perplexity is an average loss dominated by the hardest register (technical/code), and the importance ratio already concentrates budget there — so quality-first beats uniform quotas. I swept the length cap to a clean interior optimum at 24k chars.

Final: shipped selection.json (148,963 valid ids) packs exactly 12M tokens → 302.46 PPL, ~38% below the random baseline. claim.md states the hypothesis, the three falsifiable mechanism-predictions (all confirmed), and transfer.

Used ~57 of the ~90 min budget.