claude-opus-4-8

{
  "command": "cd /workspace; export TOKENIZERS_PARALLELISM=false; python3 submission/curate.py 2>&1 | tail -8",
  "description": "Run final curate.py to produce selection.json"
}

{
  "command": "cd /workspace; python3 -c \"\nimport json\na=json.load(open('submission/selection.json')); b=json.load(open('/tmp/sel_t15.json'))\nprint('lengths', len(a), len(b), 'identical:', a==b)\nprint('unique ids:', len(set(a)), 'in range:', min(a)>=0 and max(a)<182016)\n\"",
  "description": "Verify reproducibility vs winner"
}

{
  "command": "cd /workspace; export TOKENIZERS_PARALLELISM=false; python3 <<'PY'\nimport numpy as np\nfrom transformers import AutoTokenizer\ntok=AutoTokenizer.from_pretrained(\"gpt2\")\nho=np.load(\"/workspace/data/multi_dev.npy\")\nsegs,prev=[],0\nfor j in np.where(ho==50256)[0]:\n    segs.append(ho[prev:j+1]); prev=j+1   # keep EOS\nCODE=[\"{\",\"}\",\";\",\"def \",\"function\",\"return\",\"import \",\"var \",\"==\",\"() \"]\ntech,prose=[],[]\nfor s in segs:\n    txt=tok.decode([int(x) for x in s])\n    is_tech=(\"</\" in txt) or (\"<p\" in txt) or (\"/>\" in txt) or (sum(txt.count(k) for k in CODE)>4)\n    (tech if is_tech else prose).append(s)\ntech=np.concatenate(tech).astype(np.uint16); prose=np.concatenate(prose).astype(np.uint16)\nnp.save(\"/tmp/ho_tech.npy\",tech); np.save(\"/tmp/ho_prose.npy\",prose)\nprint(\"tech heldout tokens\",len(tech),\"prose heldout tokens\",len(prose))\nPY",
  "description": "Split dev target into per-register heldout files"
}

{
  "command": "cd /workspace; export TOKENIZERS_PARALLELISM=false\npython3 /tmp/emit2.py 0 20 /tmp/sel_prose_only.json\npython3 pack_selection.py /tmp/sel_prose_only.json /tmp/tr_prose.npy 2>&1 | grep packed\npython3 pack_selection.py /workspace/submission/selection.json /tmp/tr_final.npy 2>&1 | grep packed\necho \"packed both\"",
  "description": "Generate and pack prose-only and final training sets"
}

{
  "command": "cd /workspace; export TOKENIZERS_PARALLELISM=false\nfor tr in prose final; do\n  for ho in tech prose; do\n    python3 train_nano.py --train_npy /tmp/tr_$tr.npy --heldout_npy /tmp/ho_$ho.npy \\\n      --out_json /tmp/o.json --seed 1337 2>&1 | grep HELDOUT_PPL | sed \"s#^#TRAIN=$tr HO=$ho #\"\n  done\ndone",
  "description": "Run 4 register-decomposed training evals",
  "run_in_background": true
}

{
  "command": "sleep 180; cat /tmp/claude-0/-workspace/b4066b14-4abe-4b47-8865-a86bf0e3f675/tasks/bt24hkgs7.output 2>/dev/null | grep TRAIN; echo \"remaining: $(cat /workspace/.timer/remaining_secs)\"",
  "description": "Check register-decomposition results"
}

{
  "command": "sleep 120; cat /tmp/claude-0/-workspace/b4066b14-4abe-4b47-8865-a86bf0e3f675/tasks/bt24hkgs7.output 2>/dev/null | grep TRAIN; echo \"remaining: $(cat /workspace/.timer/remaining_secs)\"",
  "description": "Check register-decomposition results again"
}

{
  "command": "cat /tmp/claude-0/-workspace/b4066b14-4abe-4b47-8865-a86bf0e3f675/tasks/bt24hkgs7.output | grep TRAIN; echo \"remaining: $(cat /workspace/.timer/remaining_secs)\"",
  "description": "Read final register-decomposition results"
}

{
  "file_path": "/workspace/claim.md",
  "content": "# Curation claim: register coverage beats mean similarity\n\n## Hypothesis\nUnder a tiny fixed token budget (12M ≈ 6% of the pool), the dominant lever for\nheld-out perplexity on a **broad, multi-register** target (equal parts\nWikipedia / general web prose / news / technical Q&A) is **register coverage**,\nnot per-document quality or aggregate similarity to the target.\n\nConcretely: ranking pool docs by how target-like they are on average and taking\nthe top ~11k docs to fill the budget **collapses onto clean prose** and trains on\n≈0% technical/Q&A text — because that register is only ~25% of the target, so the\nblended similarity model always prefers the prose majority. The model is then\ncatastrophically bad on the technical quarter of the target, and that dominates\naggregate perplexity. Deliberately reserving a slice of the budget for the\nunderrepresented technical register — even a smaller slice than its target share —\nwins.\n\n## Mechanism / predicted observable (not the final perplexity)\nThe selection criterion changes *which registers appear in training*, and the\neffect should be **concentrated in the technical register, not spread uniformly**.\nTwo observables were predicted before the confirming run:\n\n1. **Register composition of the budget-filling docs.** A blind top-k similarity\n   selection puts almost no markup/code docs in-budget; a register-balanced one\n   raises it to the reserved quota.\n   - Measured: blind top-k → **51 / 11000** budget docs contain markup (0.5%);\n     register-balanced → the reserved **15%**.\n\n2. **Register-decomposed held-out loss.** Splitting the dev target into technical\n   vs prose segments and training on prose-only vs the 15%-tech mixture, the loss\n   improvement must land almost entirely on the technical segments:\n\n   | train set        | tech-heldout loss (ppl) | prose-heldout loss (ppl) |\n   |------------------|-------------------------|--------------------------|\n   | prose only (0%)  | 7.27 (1430)             | 5.95 (384)               |\n   | 15% technical    | **5.39 (219)**          | 6.02 (414)               |\n\n   Adding 15% technical data cuts technical-segment perplexity **6.5×**\n   (Δloss −1.88 nats) while prose loss barely moves (+0.07). The aggregate\n   token-weighted loss (0.32·5.39 + 0.68·6.02 = 5.82 → **ppl 337**) reproduces the\n   measured full-dev result, and its drop from the blind selection is almost\n   entirely the technical term.\n\n## Falsification\n- If register coverage were *not* the mechanism, sweeping the technical quota\n  0→40% would leave aggregate perplexity flat or monotone. **Observed instead a\n  clear U-shape** with a minimum at ~15%: 0%→441, 10%→348, **15%→337**, 20%→340,\n  30%→353, 40%→355. (Full-pipeline dev ppl; random baseline 485.)\n- If the mechanism were generic \"more/better data,\" the improvement in the\n  decomposition above would be spread across *both* registers. It is not — prose\n  loss actually rises slightly (budget reallocated away from it) while technical\n  loss collapses. A uniform improvement across registers would falsify the claim.\n- The optimum (15%) sitting **below** the target's ~25% technical share is itself\n  a prediction: raw-web technical docs are noisier/higher-entropy than the\n  target's curated Q&A, so a little goes a long way. If the optimum matched or\n  exceeded 25%, the \"noisy-raw-technical\" reasoning would be wrong.\n\n## Transfer\n- **Method.** \"Decompose the target into registers, model each, and reserve\n  budget for hard/underrepresented registers\" should transfer to any multi-domain\n  target and any budget — the tighter the budget, the larger the effect, because a\n  missing register can't be amortized away.\n- **The 15% number is pool-specific**, not the principle. It reflects this pool's\n  technical-doc noise; a cleaner technical source would push the optimum up toward\n  the target's 25% share, a dirtier one down. The **direction** (reserve *some*,\n  and typically *less than the raw target share*) is the transferable part.\n- **Scale.** A larger model with more capacity for the high-entropy technical\n  register should tolerate (and want) a technical fraction closer to the true 25%,\n  since it is less penalized by that register's noise — a testable prediction on\n  the same data with a bigger student.\n"
}

{
  "command": "cd /workspace; export TOKENIZERS_PARALLELISM=false; python3 <<'PY'\nimport json, numpy as np, zlib, re\nsel=json.load(open('submission/selection.json'))\ntext={}\nfor line in open('data/pool.jsonl'):\n    r=json.loads(line); text[r['id']]=r['text']\n# tokenize in priority order until ~12M gpt2 tokens to find the in-budget prefix\nfrom transformers import AutoTokenizer\ntok=AutoTokenizer.from_pretrained('gpt2')\nWORD=re.compile(r\"[a-z][a-z']+\")\ntot=0; inbudget=[]\nfor i in sel:\n    n=len(tok(text[i],add_special_tokens=False).input_ids)+1\n    tot+=n; inbudget.append(i)\n    if tot>=12_000_000: break\nprint(\"in-budget docs:\", len(inbudget))\n# near-dup proxy: 8-shingle set jaccard is costly; use cheap 5-word shingle minhash (bottom-4)\ndef sig(t):\n    w=WORD.findall(t.lower())\n    if len(w)<5: return None\n    sh=set()\n    for k in range(0,len(w)-4, 3):\n        sh.add(zlib.crc32((' '.join(w[k:k+5])).encode()))\n    if not sh: return None\n    return tuple(sorted(sh)[:4])\nfrom collections import Counter\nsigs=Counter()\ndupdocs=0\nseen=set()\nfor i in inbudget:\n    s=sig(text[i])\n    if s is None: continue\n    if s in seen: dupdocs+=1\n    else: seen.add(s)\nprint(\"near-dup (shared bottom-4 5gram sig) in-budget:\", dupdocs, \"/\", len(inbudget))\n# also exact prefix-120 collisions\npref=Counter(text[i][:120] for i in inbudget)\nprint(\"docs sharing a 120-char prefix:\", sum(c-1 for c in pref.values() if c>1))\nPY",
  "description": "Measure near-duplicate prevalence in-budget"
}

{
  "command": "cd /workspace; ls -la submission/ claim.md; echo \"---\"; python3 -c \"\nimport json\ns=json.load(open('submission/selection.json'))\nprint('selection.json:', len(s), 'ids | unique:', len(set(s))==len(s), '| in-range:', min(s)>=0 and max(s)<182016, '| priority-ordered list of ints:', all(isinstance(x,int) for x in s))\n\"; echo \"remaining_secs: $(cat .timer/remaining_secs)\"",
  "description": "Final deliverable verification"
}