claude-opus-4-8

{
  "command": "cd /workspace; python3 -c \"\nimport json,re\ntext={}; \nfor line in open('data/pool.jsonl'):\n    r=json.loads(line); text[r['id']]=r['text']\norig=json.load(open('submission/selection.json'))\n# strong code/QA structural markers\nqa_pat=re.compile(r'</?(p|code|pre|div|li|ul)>|def \\w+\\(|import \\w|public\\s+\\w+\\s+\\w+\\(|#include|console\\.|\\bvar\\s+\\w+\\s*=|\\}\\s*else|System\\.out')\nwp=re.compile(r\\\"[a-z0-9']+\\\")\nqa=[]\nfor i,t in text.items():\n    if qa_pat.search(t) and len(wp.findall(t.lower()))>=40:\n        qa.append(i)\nprint('QA docs found:',len(qa))\n# force QA first (~4M tokens worth), then original prose ordering\nseen=set(qa); merged=list(qa)+[i for i in orig if i not in seen]\njson.dump(merged, open('/tmp/sel_qa.json','w'))\nprint('merged selection',len(merged),'qa-front tokens approx', sum(len(wp.findall(text[i].lower())) for i in qa))\n\"",
  "description": "Build QA-forced selection"
}

{
  "command": "cd /workspace; python3 -c \"\nimport json,re\ntext={}\nfor line in open('data/pool.jsonl'):\n    r=json.loads(line); text[r['id']]=r['text']\norig=json.load(open('submission/selection.json'))\nqa_pat=re.compile(r'</?(p|code|pre|div|li|ul)>|def \\w+\\(|import \\w|public\\s+\\w+\\s+\\w+\\(|#include|console\\.|\\bvar\\s+\\w+\\s*=|\\}\\s*else|System\\.out')\nwp=re.compile(r\\\"[a-z0-9']+\\\")\nqa=[i for i,t in text.items() if qa_pat.search(t) and len(wp.findall(t.lower()))>=40]\n# cap to ~2.2M words (~30% of budget)\ncap=2_200_000; cum=0; qsel=[]\nfor i in qa:\n    cum+=len(wp.findall(text[i].lower())); qsel.append(i)\n    if cum>=cap: break\nseen=set(qsel); merged=qsel+[i for i in orig if i not in seen]\njson.dump(merged, open('/tmp/sel_qa.json','w'))\nprint('QA front docs',len(qsel),'~words',cum,'total merged',len(merged))\n\"\nbash dev_eval.sh /tmp/sel_qa.json 2>&1 | grep -v -i warning | grep -E \"packed|HELDOUT\"",
  "description": "Eval capped QA-forced selection"
}

{
  "command": "sleep 60; cat /tmp/claude-0/-workspace/*/tasks/bvk0dck7a.output 2>/dev/null | grep -v -i warning | grep -E \"QA front|packed|HELDOUT\"",
  "description": "Read QA-forced eval result"
}

{
  "command": "cd /workspace; python3 -c \"\nimport json\nsel=json.load(open('submission/selection.json'))\nids=set(json.loads(l)['id'] for l in open('data/pool.jsonl'))\nassert len(sel)==len(set(sel)), 'dupes!'\nassert all(i in ids for i in sel), 'out of range!'\nprint('VALID: ',len(sel),'unique in-range ids; first5',sel[:5])\n\"; cat /workspace/.timer/remaining_secs",
  "description": "Validate final selection"
}

{
  "file_path": "/workspace/claim.md",
  "content": "# Curation claim\n\n## Hypothesis\nHeld-out perplexity on the disclosed broad-HQ-English target (Wikipedia + news +\nhigh-quality web prose + technical Q&A) is minimized under a fixed 12M-token\nbudget by selecting the raw-web documents whose **word n-gram distribution most\nresembles the target domain rather than the generic web background**, after first\nremoving non-prose (number/name lists, nav menus, boilerplate, spam) with cheap\nstructural gates. A domain-importance ranking (DSIR-style) plus a prose gate beats\na random selection by a large margin, and it already finds the right *mixture* of\nregisters on its own — hand-forcing register balance does not help.\n\n## Mechanism (prediction of an observable other than final perplexity)\nThe selection works by shifting the **training token distribution toward the\ntarget's** and away from web boilerplate. Concretely, independent of the final\nperplexity number, I predict and observe:\n\n1. **Composition shift.** Ranking by `mean_ngram[ log P_target − log P_raw ]` with\n   no gate puts *degenerate low-diversity pages at the very top* (pure prime-number\n   listing pages, alphabetical plant/name galleries) because the target is\n   number/name-rich — a directly visible failure. Adding the prose gate\n   (stopword-fraction in [0.20,0.60], digit-fraction < 0.15, alpha-fraction > 0.60,\n   type/token < 0.85) removes exactly those pages and the top of the ranking becomes\n   fluent multi-register prose. This is observable by eye and was the deciding\n   diagnostic.\n2. **Monotone dev signal from the mechanism, not the label.** Because the target\n   profile is estimated purely from the disclosed dev tokens decoded back to text,\n   the mechanism predicts the *direction* of every ablation before training:\n   - gated domain-match selection → **dev PPL 315.4**\n   - random selection (do-nothing baseline) → **458.0** (−31% from the method)\n   - relaxing the prose gate (admits mediocre prose) → **320.5** (worse)\n   - force-injecting ~30% technical-QA/code docs to \"balance\" registers → **325.8**\n     (worse: the pool holds only ~2% QA docs and the unranked ones dilute quality).\n   The ordering 315.4 < 320.5 < 325.8 < 458.0 is the mechanism's fingerprint:\n   purity of domain match dominates, forced balance hurts.\n\n## Falsification\nThe claim is false if any of the following held:\n- A random selection reached comparable held-out perplexity (it did not: 458 vs 315).\n- Removing the prose gate left the ranking unchanged or improved it (it degraded\n  it, both visibly — junk pages resurfaced at the top — and numerically 315→320).\n- Explicitly matching the target's 4-way register mix (25% technical-QA) lowered\n  perplexity below the unconstrained domain-match ranking (it raised it, 315→326).\nAny of these would show that \"look like the target, drop non-prose\" is not the\noperative lever.\n\n## Transfer\nThe criterion is target-agnostic: swap the disclosed dev sample for any other\ntarget corpus and the same script re-estimates `P_target`, re-ranks the pool, and\nselects the budget-fitting best-first ids — no target-specific tuning, no labels.\nIt should transfer to (a) a different held-out domain (e.g. pure code, or legal\ntext) by re-profiling, and (b) larger pools/budgets, where the prose gate + DSIR\nweight scale linearly (single streaming pass, ~75 s for 182k docs / 200M tokens).\nThe prediction that transfers: for any target, the unconstrained domain-match\nranking on prose-gated documents will beat both random selection and a\nhand-balanced register mix drawn from the same pool.\n"
}

{
  "command": "cd /workspace; ls -la submission/ claim.md; echo \"---\"; grep 'OUT  =' submission/curate.py; echo \"--- selection size ---\"; python3 -c \"import json;print(len(json.load(open('submission/selection.json'))),'ids')\"",
  "description": "Final deliverable check"
}