claude-opus-4-8

Command running in background with ID: bl83pjg2i. Output is being written to: /tmp/claude-0/-workspace/06cb3775-ded4-4ad5-84cf-ed7eaa26a5bd/tasks/bl83pjg2i.output. You will be notified when it completes. To check interim output, use Read on that file path.
Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "bl83pjg2i", "backgroundCwdHint": "Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands."}

Command running in background with ID: bvrr598n2. Output is being written to: /tmp/claude-0/-workspace/06cb3775-ded4-4ad5-84cf-ed7eaa26a5bd/tasks/bvrr598n2.output. You will be notified when it completes. To check interim output, use Read on that file path.
Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "bvrr598n2", "backgroundCwdHint": "Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands."}

=== v3 curate ===
  dev register shares: {'tech': '26%', 'wiki': '25%', 'news': '34%', 'prose': '15%'}
=== v2t (techboost) eval ===
packed 12000000 tokens from 40122 ids -> /tmp/dev_train.npy
HELDOUT_PPL 354.426  loss 5.8705  train_s 56  /tmp/dev_train.npy
remaining: 3226s

[stdout]
=== v3 curate ===
  dev register shares: {'tech': '26%', 'wiki': '25%', 'news': '34%', 'prose': '15%'}
=== v2t (techboost) eval ===
packed 12000000 tokens from 40122 ids -> /tmp/dev_train.npy
HELDOUT_PPL 354.426  loss 5.8705  train_s 56  /tmp/dev_train.npy
remaining: 3226s
[metadata] {"noOutputExpected": false}

Command did not complete within its 120s timeout and was moved to the background (ID: bw1bzj135). Output is being written to: /tmp/claude-0/-workspace/06cb3775-ded4-4ad5-84cf-ed7eaa26a5bd/tasks/bw1bzj135.output. You will be notified when it completes. To check interim output, use Read on that file path.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "bw1bzj135", "timedOutAfterMs": 120000}

STILL RUNNING
  dev register shares: {'tech': '26%', 'wiki': '25%', 'news': '34%', 'prose': '15%'}
  139592/182016 eligible for >=1 register
remaining: 3085s

[stdout]
STILL RUNNING
  dev register shares: {'tech': '26%', 'wiki': '25%', 'news': '34%', 'prose': '15%'}
  139592/182016 eligible for >=1 register
remaining: 3085s
[metadata] {"noOutputExpected": false}

register est-token shares: {'tech': '25%', 'wiki': '25%', 'news': '25%', 'prose': '25%'}
  register doc supply: {'tech': 134545, 'wiki': 128158, 'news': 128158, 'prose': 128158}
wrote 39914 ids -> /tmp/sel_v3.json
remaining: 3065s

[stdout]
  register est-token shares: {'tech': '25%', 'wiki': '25%', 'news': '25%', 'prose': '25%'}
  register doc supply: {'tech': 134545, 'wiki': 128158, 'news': 128158, 'prose': 128158}
wrote 39914 ids -> /tmp/sel_v3.json
remaining: 3065s
[metadata] {"noOutputExpected": false}

Command running in background with ID: brxnu0iw0. Output is being written to: /tmp/claude-0/-workspace/06cb3775-ded4-4ad5-84cf-ed7eaa26a5bd/tasks/brxnu0iw0.output. You will be notified when it completes. To check interim output, use Read on that file path.
Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "brxnu0iw0", "backgroundCwdHint": "Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands."}

First 20 of v3 (round-robin tech,wiki,news,prose):
tech [162538]: int Up Module Stdint module Stdint: sig .. end Standard integer types type int8 Signed 8-bit integer type int16 Sig
wiki [76107]: Major Bhupinder Singh In sept. 1965, Major Bhupinder Singh commanded the "B" squadron of the 4 Horse against Pakist
news [58452]: <|endoftext|>New Delhi: Five new Supreme Court judges were today sworn in, taking the top court’s strength to 28, i
prose[103958]: last wicket pair of Fidel Edwards and Corey Collymore battled grimly for the last 19 deliveries to earn the West In
tech [87261]: for her role as Brittany on the BET comedy-drama series The Game. She appeared in 16 episodes of the series between
wiki [81859]: |Rediff India Abroad Home | All the sections| Bihar: Vigilante justice resurfaces, three people lynched February 18
news [12743]: ZF-5830: Zend_Db_Table_Select doesn't allow use of $select->columns('..') Zend_Db_Table_Select doesn't allow use of
prose[3475]: Anglo-Dutch Wars, also called Dutch Wars, Dutch Engelse Oorlogen, four 17th- and 18th-century naval conflicts betwe
tech [180946]: <|endoftext|>bdlymmgs.com - Database Error Discuz! Database Error (1040) notconnect PHP Debug No. File Line Code 1 
wiki [138968]: .<|endoftext|>Boletos en para el Excite Tickets Deportes WWE Royal Rumble Boston Bruins Brooklyn Nets Dallas Cowboy
news [27085]: <|endoftext|>News on : Jagan Mohan The Atmakur Civil Judge on Tuesday sent TDP MLA Erra Shekhar to 14-day judicial 
prose[103973]: pur: Prime Minister Manmohan Singh on Friday kicked off the Congress campaign from Kanpur on Friday. Addressing the
tech [12364]: The attack was launched at 0730hrs on the 1st July 1916. Along a twenty mile Front 200,000 British and French troop
wiki [37522]: Prime Minister Imran Khan on Thursday congratulated India’s premier Narendra Modi on his victory in the country's n
news [165621]: Sitemap ↑<|endoftext|>Pkcs11Interop: Net.Pkcs11Interop.HighLevelAPI40.Mechanism Class Reference Pkcs11Interop 4.1.1
prose[39794]: Fabian of Rome The Hieromartyr Fabian II of Rome was the bishop of Rome of the Church of Rome from 236 to 250. The 
tech [55624]: to local government officials, the incident took place around 9:30 pm local time leaving the district police chief 
wiki [36908]: Thomas Robert Gore Browne |Birthplace:||Aylesbury, Buckinghamshire, England| |Death:||Died in London, England| |Man
news [91191]: <|endoftext|>BJP-led Goa government has decided to subsidise three more LPG cylinders for the people in addition to
prose[41186]: Amarnath Yatra 2017 Amarnath Yatra 2017 News Jammu and Kashmir police said three people, alleged conspirators in th

overlap v2∩v3 in first 12k: 11809 / 12000

[stdout]
First 20 of v3 (round-robin tech,wiki,news,prose):
tech [162538]: int Up Module Stdint module Stdint: sig .. end Standard integer types type int8 Signed 8-bit integer type int16 Sig
wiki [76107]: Major Bhupinder Singh In sept. 1965, Major Bhupinder Singh commanded the "B" squadron of the 4 Horse against Pakist
news [58452]: <|endoftext|>New Delhi: Five new Supreme Court judges were today sworn in, taking the top court’s strength to 28, i
prose[103958]: last wicket pair of Fidel Edwards and Corey Collymore battled grimly for the last 19 deliveries to earn the West In
tech [87261]: for her role as Brittany on the BET comedy-drama series The Game. She appeared in 16 episodes of the series between
wiki [81859]: |Rediff India Abroad Home | All the sections| Bihar: Vigilante justice resurfaces, three people lynched February 18
news [12743]: ZF-5830: Zend_Db_Table_Select doesn't allow use of $select->columns('..') Zend_Db_Table_Select doesn't allow use of
prose[3475]: Anglo-Dutch Wars, also called Dutch Wars, Dutch Engelse Oorlogen, four 17th- and 18th-century naval conflicts betwe
tech [180946]: <|endoftext|>bdlymmgs.com - Database Error Discuz! Database Error (1040) notconnect PHP Debug No. File Line Code 1 
wiki [138968]: .<|endoftext|>Boletos en para el Excite Tickets Deportes WWE Royal Rumble Boston Bruins Brooklyn Nets Dallas Cowboy
news [27085]: <|endoftext|>News on : Jagan Mohan The Atmakur Civil Judge on Tuesday sent TDP MLA Erra Shekhar to 14-day judicial 
prose[103973]: pur: Prime Minister Manmohan Singh on Friday kicked off the Congress campaign from Kanpur on Friday. Addressing the
tech [12364]: The attack was launched at 0730hrs on the 1st July 1916. Along a twenty mile Front 200,000 British and French troop
wiki [37522]: Prime Minister Imran Khan on Thursday congratulated India’s premier Narendra Modi on his victory in the country's n
news [165621]: Sitemap ↑<|endoftext|>Pkcs11Interop: Net.Pkcs11Interop.HighLevelAPI40.Mechanism Class Reference Pkcs11Interop 4.1.1
prose[39794]: Fabian of Rome The Hieromartyr Fabian II of Rome was the bishop of Rome of the Church of Rome from 236 to 250. The 
tech [55624]: to local government officials, the incident took place around 9:30 pm local time leaving the district police chief 
wiki [36908]: Thomas Robert Gore Browne |Birthplace:||Aylesbury, Buckinghamshire, England| |Death:||Died in London, England| |Man
news [91191]: <|endoftext|>BJP-led Goa government has decided to subsidise three more LPG cylinders for the people in addition to
prose[41186]: Amarnath Yatra 2017 Amarnath Yatra 2017 News Jammu and Kashmir police said three people, alleged conspirators in th

overlap v2∩v3 in first 12k: 11809 / 12000
[metadata] {"noOutputExpected": false}

Command did not complete within its 120s timeout and was moved to the background (ID: bcn2ntwdk). Output is being written to: /tmp/claude-0/-workspace/06cb3775-ded4-4ad5-84cf-ed7eaa26a5bd/tasks/bcn2ntwdk.output. You will be notified when it completes. To check interim output, use Read on that file path.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "bcn2ntwdk", "timedOutAfterMs": 120000}

v3 bal+fluency  : HELDOUT_PPL 337.320
remaining: 2845s

[stdout]
v3 bal+fluency  : HELDOUT_PPL 337.320
remaining: 2845s
[metadata] {"noOutputExpected": false}

v4 patched + syntax OK
remaining: 2805s

[stdout]
v4 patched + syntax OK
remaining: 2805s
[metadata] {"noOutputExpected": false}

Command running in background with ID: bmzt6g8w7. Output is being written to: /tmp/claude-0/-workspace/06cb3775-ded4-4ad5-84cf-ed7eaa26a5bd/tasks/bmzt6g8w7.output. You will be notified when it completes. To check interim output, use Read on that file path.
Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "bmzt6g8w7", "backgroundCwdHint": "Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands."}

File created successfully at: /workspace/claim.md (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/workspace/claim.md", "content": "# Curation claim\n\n## Hypothesis\nThe pool is mostly off-target web junk (shopping-cart / SEO / forum / navigation\nboilerplate, foreign-language and keyword-salad spam, duplicated pages). Under the\nfixed 12M-token budget, held-out perplexity on the *broad, equal-parts* HQ target\n(encyclopedic + general web prose + news + technical Q&A) is minimized by two\ncompounding moves:\n\n1. **Target-matched selection** — keep documents whose *content* distribution\n   resembles the disclosed target, via DSIR-style importance weighting (per-feature\n   `log P_target − log P_pool` on a common word uni/bi-gram vocabulary), and drop\n   boilerplate/duplicates.\n2. **Register balancing** — allocate the budget *equally* across the target's four\n   registers using one importance classifier per register, instead of ranking by a\n   single global target score. A single global score collapses the budget onto the\n   one register that is easiest to separate from pool junk (news), starving the\n   others; because the target is equal-parts, the starved registers dominate the\n   average loss.\n\n## Mechanism / prediction (observable *other* than final perplexity)\nMechanism: perplexity = exp(mean next-token loss) averaged over four equal target\nquarters. A tiny 30M model on 12M tokens is capacity/data limited, so each quarter's\nloss falls with the share of *in-register* training tokens and saturates; the mean\nis dominated by whichever quarter is most starved. Matching the training register\nmix to the target's equal mix therefore minimizes the mean.\n\nConcretely predicted, measurable **without** looking at the final score:\n\n- **Budget-slice composition (measured, no training needed).** Ranking by a single\n  global target classifier fills the 12M budget **~77% news / ~20% web-prose /\n  ~2% technical / ~1% encyclopedic**. The per-register method fills it **25/25/25/25**.\n  This composition is computable directly from `selection.json` + the pool.\n- **Allocation sweep is convex with its minimum at equal shares.** Pushing the\n  technical share *above* 25% must *raise* perplexity (the target is equal-parts, so\n  over-serving one register starves others). Verified: tech-boosted (32.5% technical)\n  scored **354.4** vs **347.5** for equal — worse, as predicted.\n- **Learned feature weights separate boilerplate from prose.** The classifier's most\n  pool-typical (negative) unigrams are commerce/spam tokens (`cart`, `sitemap`,\n  `accessories`, `clomid`, `towbars`); its most target-typical (positive) tokens are\n  register-content words (`wicket`/`blockquote`/`bjp`/`regiment`). A curation that\n  merely maximized length or raw token count would not show this separation.\n- **Fluency gate removes proper-noun keyword-salad.** Requiring a minimum function-\n  word ratio on the prose registers drops spam like \"Boletos Excite Tickets Royal\n  Rumble Boston Bruins …\" (≈0% stopwords) that a mean-log-odds score otherwise ranks\n  high because proper nouns look target-like. Observable: mean stopword ratio of the\n  selected prose docs rises well above that of the discarded high-score salad.\n\n## Falsification\nThe claim is false if:\n- A **single global target ranking** (no register balancing) matches or beats the\n  balanced selection at equal token budget. (Measured: global **388.2** vs balanced\n  **337.3** — balancing wins by ~13%.)\n- A **random** selection matches the curated one. (Measured: random **457.2** vs\n  curated **337.3**.)\n- **Over-weighting** one register (e.g. technical > 25%) *lowers* perplexity — it\n  raised it (354.4), so the equal-mix mechanism holds.\nIf, instead, the win came only from generic \"clean-text\" filtering, then shuffling\nregister shares would not matter — but it does (77/20/2/1 → 388, 25/25/25/25 → 337).\n\n## Transfer\n- **To the hidden target:** the criterion keys on *content* word n-grams, not on the\n  dev sample's surface formatting (WikiText spacing / `@.@`), so it transfers to the\n  disjoint hidden sample of the *same* four-register domain rather than memorizing\n  the dev set.\n- **To other targets:** the method is target-agnostic — swap `multi_dev.npy` for any\n  disclosed target sample and re-run; the per-register split + equal-allocation\n  round-robin generalizes to any known target mixture (change the register detector\n  and the allocation weights).\n- **To larger budgets/models:** balancing matters *most* when the budget is far\n  smaller than the pool (starvation is acute). As the budget grows toward the pool\n  size the balanced and global selections converge, so the gap should shrink with a\n  larger token budget — a further falsifiable prediction.\n", "structuredPatch": [], "originalFile": null, "userModified": false}

Command did not complete within its 120s timeout and was moved to the background (ID: ba5jwzfdo). Output is being written to: /tmp/claude-0/-workspace/06cb3775-ded4-4ad5-84cf-ed7eaa26a5bd/tasks/ba5jwzfdo.output. You will be notified when it completes. To check interim output, use Read on that file path.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "ba5jwzfdo", "timedOutAfterMs": 120000}

Command running in background with ID: bl0ixolzc. Output is being written to: /tmp/claude-0/-workspace/06cb3775-ded4-4ad5-84cf-ed7eaa26a5bd/tasks/bl0ixolzc.output. You will be notified when it completes. To check interim output, use Read on that file path.
Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "bl0ixolzc", "backgroundCwdHint": "Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands."}

Command did not complete within its 120s timeout and was moved to the background (ID: bg05zhe11). Output is being written to: /tmp/claude-0/-workspace/06cb3775-ded4-4ad5-84cf-ed7eaa26a5bd/tasks/bg05zhe11.output. You will be notified when it completes. To check interim output, use Read on that file path.
Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "bg05zhe11", "timedOutAfterMs": 120000, "backgroundCwdHint": "Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands."}

=== v4 (tech floor 0.10) eval ===
packed 12000000 tokens from 40472 ids -> /tmp/dev_train.npy
--- v4 curate supply ---
  137994/182016 eligible for >=1 register
  register doc supply: {'tech': 133233, 'wiki': 128158, 'news': 128158, 'prose': 128158}
wrote 40472 ids -> /tmp/sel_v4.json
remaining: 2444s

[stdout]
=== v4 (tech floor 0.10) eval ===
packed 12000000 tokens from 40472 ids -> /tmp/dev_train.npy
--- v4 curate supply ---
  137994/182016 eligible for >=1 register
  register doc supply: {'tech': 133233, 'wiki': 128158, 'news': 128158, 'prose': 128158}
wrote 40472 ids -> /tmp/sel_v4.json
remaining: 2444s
[metadata] {"noOutputExpected": false}

Command did not complete within its 120s timeout and was moved to the background (ID: b4302wxjj). Output is being written to: /tmp/claude-0/-workspace/06cb3775-ded4-4ad5-84cf-ed7eaa26a5bd/tasks/b4302wxjj.output. You will be notified when it completes. To check interim output, use Read on that file path.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "b4302wxjj", "timedOutAfterMs": 120000}

HELDOUT_PPL 337.800
remaining: 2304s

[stdout]
HELDOUT_PPL 337.800
remaining: 2304s
[metadata] {"noOutputExpected": false}

The file /workspace/submission/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"type": "update", "filePath": "/workspace/submission/curate.py", "content": "#!/usr/bin/env python3\n\"\"\"Curate the raw web pool for from-scratch LM pretraining under a fixed token budget.\n\nSTATED CRITERION\n================\nThe disclosed evaluation target is a BROAD, *equal-parts* multi-domain high-quality\nEnglish distribution: encyclopedic (Wikipedia), general high-quality web prose,\nnews, and technical Q&A.  We select pool documents so that the 12M-token training\nbudget (a) contains documents whose *content* matches the target and (b) is split\nEQUALLY across the four target registers.\n\nTwo ideas, both necessary (see /workspace/claim.md for the ablation):\n\n1. DSIR-style importance weighting (Xie et al. 2023).  On a fixed common vocabulary\n   (most-frequent word unigrams+bigrams, lower-cased [a-z']+), score a feature f by\n   `weight(f) = clip(log P_target(f) - log P_pool(f))`.  P_target is estimated from\n   the decoded disclosed target sample (`multi_dev.npy`); P_pool from a random pool\n   background sample.  Restricting to *common* features keeps rare near-unique tokens\n   (proper-name lists / table cells) from dominating with sampling-noise weights.\n\n2. Register balancing.  A single global target classifier collapses the budget onto\n   the register that is easiest to separate from pool junk (news: ~77% of the budget)\n   and starves the technical and encyclopedic registers, which then dominate the\n   average loss.  Instead we split the target sample into the four registers, train\n   one importance classifier per register, and fill the budget by round-robin over\n   registers (always extend the register with the fewest tokens so far).  The budget\n   is thus 25/25/25/25 by construction.\n\nQuality gates (drop): fragments (too short / too few in-vocab features), digit/table\ndumps, navigation-menu boilerplate (mostly tiny lines), repeated-line pages, largely\nnon-ASCII (foreign/CJK) pages, and — for the three PROSE registers — documents below\na function-word (stopword) ratio floor, which removes proper-noun keyword-salad SEO\nspam that a mean-log-odds score otherwise ranks high.  The code register is exempt\nfrom the stopword floor (code legitimately has few function words, and markup/code\ntoken exposure helps the target's HTML/code quarter).  Exact and near-duplicate\ndocuments are removed, keeping the single highest-scoring copy.  Hashing is crc32\n(deterministic) so the script reproduces its `selection.json` exactly.\n\nOutput: `selection.json` — surviving ids sorted best-first (register-interleaved).\nThe frozen training pipeline consumes this order until the 12M-token budget is full.\n\nReproduce:  python3 curate.py            # writes /workspace/submission/selection.json\n\"\"\"\nimport argparse, json, re, sys, math, zlib\nfrom collections import Counter\nimport numpy as np\n\nPOOL = \"/workspace/data/pool.jsonl\"\nDEV_NPY = \"/workspace/data/multi_dev.npy\"          # disclosed target sample (GPT-2 ids)\nOUT = \"/workspace/submission/selection.json\"\n\n# ---- hyper-parameters (fixed) -----------------------------------------------\nN_UNI, N_BI = 40000, 40000                          # common vocabulary size\nBG_SAMPLE = 60000                                   # pool docs for background + vocab\nALPHA, CLIP, MIN_FEATS = 1.0, 4.0, 20               # smoothing / weight clip / min feats\nREGISTERS = [\"tech\", \"wiki\", \"news\", \"prose\"]\n# per-register function-word floor: prose registers require fluent English; code exempt\nSTOP_FLOOR = {\"tech\": 0.0, \"wiki\": 0.20, \"news\": 0.20, \"prose\": 0.20}\nALLOC_DEFAULT = \"1,1,1,1\"                            # equal token budget per register\nWORD_RE = re.compile(r\"[a-z']+\")\nWS_RE = re.compile(r\"\\s+\")\n# general quality gates\nMIN_CHARS, MIN_WORDS = 200, 60\nMAX_DIGIT_FRAC, MAX_SHORT_LINE_FRAC, MAX_DUP_LINE_FRAC, MAX_NONASCII_FRAC = 0.15, 0.66, 0.50, 0.10\nTARGET_TOTAL = 40_000_000                            # emit ~40M est-tokens (budget is 12M)\n\nSTOP = set((\n    \"the of and to a in is that it for on with as was were be been being by this these those are am \"\n    \"at from or an which not no but have has had they you we he she his her their its will would can could \"\n    \"i my me your our us them do does did so if then than out up down about into over under after before \"\n    \"all any more most some such only own same other new one two first last time year people\"\n).split())\n\n\ndef words_of(t):\n    return WORD_RE.findall(t.lower())\n\n\ndef dkey(s):\n    return zlib.crc32(s.encode(\"utf-8\", \"ignore\")) & 0xFFFFFFFF\n\n\ndef general_ok(text, words):\n    if len(text) < MIN_CHARS or len(words) < MIN_WORDS:\n        return False\n    if sum(1 for c in text if ord(c) > 127) / len(text) > MAX_NONASCII_FRAC:\n        return False\n    if sum(c.isdigit() for c in text) / len(text) > MAX_DIGIT_FRAC:\n        return False\n    lines = [ln for ln in text.split(\"\\n\") if ln.strip()]\n    if len(lines) >= 8:\n        if sum(1 for ln in lines if len(ln.split()) <= 3) / len(lines) > MAX_SHORT_LINE_FRAC:\n            return False\n        if 1 - len(set(lines)) / len(lines) > MAX_DUP_LINE_FRAC:\n            return False\n    return True\n\n\ndef register_of(txt):\n    \"\"\"Assign a decoded target segment to one of four registers by surface markers.\"\"\"\n    n = max(1, len(txt))\n    sym = sum(c in \"<>{}=;/&|_#\\\\`$\" for c in txt)\n    if sym / n > 0.02 or \"&lt;\" in txt or \"&gt;\" in txt or (txt.count(\"{\") + txt.count(\"}\") >= 4):\n        return \"tech\"\n    if \" @.@ \" in txt or \" @-@ \" in txt or (txt.count(\" .\") + txt.count(\" ,\")) > n / 250:\n        return \"wiki\"                                # WikiText-103 spaced punctuation\n    tl = txt.lower()\n    if re.search(r\"\\((reuters|ians|ani|ap|afp|pti|xinhua|dw)\\)\", tl) or \\\n       re.search(r\"\\b(said|told reporters|according to|reported|correspondent|\"\n                 r\"on (monday|tuesday|wednesday|thursday|friday|saturday|sunday))\\b\", tl):\n        return \"news\"\n    return \"prose\"\n\n\ndef doc_features(text, uni_ix, bi_ix):\n    w = words_of(text)\n    feats = [uni_ix[x] for x in w if x in uni_ix]\n    prev = None\n    for x in w:\n        if prev is not None:\n            b = bi_ix.get((prev, x))\n            if b is not None:\n                feats.append(b)\n        prev = x\n    stop_frac = (sum(1 for x in w if x in STOP) / len(w)) if w else 0.0\n    return w, feats, stop_frac\n\n\ndef main():\n    ap = argparse.ArgumentParser()\n    ap.add_argument(\"--out\", default=OUT)\n    ap.add_argument(\"--seed\", type=int, default=0)\n    ap.add_argument(\"--alloc\", default=ALLOC_DEFAULT, help=\"token weights tech,wiki,news,prose\")\n    args = ap.parse_args()\n    rng = np.random.default_rng(args.seed)\n    alloc = np.array([float(x) for x in args.alloc.split(\",\")], dtype=float)\n    alloc = alloc / alloc.sum()\n\n    # ---- 1. load pool -------------------------------------------------------\n    print(\"loading pool...\", file=sys.stderr)\n    ids, texts = [], []\n    with open(POOL) as f:\n        for line in f:\n            r = json.loads(line)\n            ids.append(r[\"id\"]); texts.append(r[\"text\"])\n    ids = np.asarray(ids)\n    N = len(ids)\n\n    # ---- 2. common vocabulary + pool background counts ----------------------\n    print(\"building vocabulary + background counts...\", file=sys.stderr)\n    bg_idx = rng.choice(N, size=min(BG_SAMPLE, N), replace=False)\n    uni_bg, bi_bg = Counter(), Counter()\n    for j in bg_idx:\n        w = words_of(texts[j]); uni_bg.update(w); bi_bg.update(zip(w, w[1:]))\n    uni_ix = {w: k for k, (w, _) in enumerate(uni_bg.most_common(N_UNI))}\n    bi_ix = {b: N_UNI + k for k, (b, _) in enumerate(bi_bg.most_common(N_BI))}\n    Vf = N_UNI + N_BI\n    bg_counts = np.zeros(Vf)\n    for w, c in uni_bg.items():\n        k = uni_ix.get(w)\n        if k is not None: bg_counts[k] = c\n    for b, c in bi_bg.items():\n        k = bi_ix.get(b)\n        if k is not None: bg_counts[k] = c\n    log_bg = np.log(bg_counts + ALPHA) - math.log(bg_counts.sum() + ALPHA * Vf)\n\n    # ---- 3. per-register target counts from decoded target sample -----------\n    print(\"decoding target + per-register counts...\", file=sys.stderr)\n    from transformers import AutoTokenizer\n    tok = AutoTokenizer.from_pretrained(\"gpt2\")\n    dev = np.load(DEV_NPY).astype(np.int64); EOS = 50256\n    cuts = np.where(dev == EOS)[0]\n    segs, prev = [], 0\n    for i in cuts:\n        if i > prev: segs.append((prev, i))\n        prev = i + 1\n    if prev < len(dev): segs.append((prev, len(dev)))\n    reg_counts = {r: np.zeros(Vf) for r in REGISTERS}\n    reg_toks = {r: 0 for r in REGISTERS}\n    for s, e in segs:\n        txt = tok.decode(dev[s:e].tolist()); r = register_of(txt); reg_toks[r] += (e - s)\n        w = words_of(txt); cc = reg_counts[r]\n        for x in w:\n            k = uni_ix.get(x)\n            if k is not None: cc[k] += 1\n        for x in zip(w, w[1:]):\n            k = bi_ix.get(x)\n            if k is not None: cc[k] += 1\n    tot = sum(reg_toks.values())\n    print(\"  dev register shares:\", {r: f\"{100*reg_toks[r]/tot:.0f}%\" for r in REGISTERS}, file=sys.stderr)\n\n    weight = {}\n    for r in REGISTERS:\n        log_t = np.log(reg_counts[r] + ALPHA) - math.log(reg_counts[r].sum() + ALPHA * Vf)\n        weight[r] = np.clip(log_t - log_bg, -CLIP, CLIP)\n\n    # ---- 4. score + gate every document under all four registers ------------\n    print(\"scoring all documents...\", file=sys.stderr)\n    score = {r: np.full(N, -1e9) for r in REGISTERS}\n    est_tok = np.zeros(N, dtype=np.int64)\n    ok = np.zeros(N, dtype=bool)\n    for k in range(N):\n        text = texts[k]\n        est_tok[k] = len(text) // 4\n        w, feats, stop_frac = doc_features(text, uni_ix, bi_ix)\n        if not general_ok(text, w) or len(feats) < MIN_FEATS:\n            continue\n        fa = np.asarray(feats)\n        any_ok = False\n        for r in REGISTERS:\n            if stop_frac < STOP_FLOOR[r]:\n                continue\n            score[r][k] = weight[r][fa].mean(); any_ok = True\n        ok[k] = any_ok\n        if (k + 1) % 40000 == 0:\n            print(f\"  {k+1}/{N}\", file=sys.stderr)\n    print(f\"  {int(ok.sum())}/{N} eligible for >=1 register\", file=sys.stderr)\n\n    # ---- 5. deterministic (near-)dedup, keeping highest-scoring copy --------\n    best = np.max(np.stack([score[r] for r in REGISTERS]), axis=0)\n    seen_full, seen_sig = set(), set()\n    keep = np.zeros(N, dtype=bool)\n    for k in np.argsort(-best, kind=\"stable\"):\n        if not ok[k]:\n            continue\n        norm = WS_RE.sub(\" \", texts[k]).strip()\n        full = dkey(norm)\n        core = norm[120:920]\n        sig = dkey(core) if len(core) > 200 else full\n        if full in seen_full or sig in seen_sig:\n            continue\n        seen_full.add(full); seen_sig.add(sig); keep[k] = True\n\n    # ---- 6. register-balanced round-robin fill ------------------------------\n    ranked = {}\n    for r in REGISTERS:\n        sc = score[r].copy(); sc[~keep] = -1e9\n        o = np.argsort(-sc, kind=\"stable\")\n        ranked[r] = o[keep[o] & (sc[o] > -1e8)]\n    ptr = {r: 0 for r in REGISTERS}\n    used = np.zeros(N, dtype=bool)\n    got = {r: 0 for r in REGISTERS}\n    sel = []\n    while True:\n        cand = [r for r in REGISTERS if ptr[r] < len(ranked[r])]\n        if not cand:\n            break\n        r = min(cand, key=lambda r: got[r] / alloc[REGISTERS.index(r)]\n                if alloc[REGISTERS.index(r)] > 0 else 1e18)\n        while ptr[r] < len(ranked[r]) and used[ranked[r][ptr[r]]]:\n            ptr[r] += 1\n        if ptr[r] >= len(ranked[r]):\n            continue\n        k = ranked[r][ptr[r]]; ptr[r] += 1\n        used[k] = True; sel.append(int(ids[k])); got[r] += int(est_tok[k])\n        if sum(got.values()) >= TARGET_TOTAL:\n            break\n\n    print(\"  emitted register est-token shares:\",\n          {r: f\"{100*got[r]/max(1, sum(got.values())):.0f}%\" for r in REGISTERS}, file=sys.stderr)\n    json.dump(sel, open(args.out, \"w\"))\n    print(f\"wrote {len(sel)} ids -> {args.out}\", file=sys.stderr)\n\n\nif __name__ == \"__main__\":\n    main()\n", "structuredPatch": [{"oldStart": 2, "oldLines": 91, "newStart": 2, "newLines": 139, "lines": [" \"\"\"Curate the raw web pool for from-scratch LM pretraining under a fixed token budget.", " ", " STATED CRITERION", "-----------------", "-The disclosed evaluation target is a BROAD, multi-domain high-quality English", "-distribution (equal parts Wikipedia/encyclopedic, general HQ web prose, news, and", "-technical Q&A).  We rank every pool document by how much its *content* resembles", "-that target distribution, drop near-duplicates, and drop non-prose boilerplate.", "+================", "+The disclosed evaluation target is a BROAD, *equal-parts* multi-domain high-quality", "+English distribution: encyclopedic (Wikipedia), general high-quality web prose,", "+news, and technical Q&A.  We select pool documents so that the 12M-token training", "+budget (a) contains documents whose *content* matches the target and (b) is split", "+EQUALLY across the four target registers.", " ", "-Scoring is DSIR-style importance weighting (Xie et al. 2023) on a FIXED, common", "-bag-of-words vocabulary:", "+Two ideas, both necessary (see /workspace/claim.md for the ablation):", " ", "-  * Vocabulary = the most frequent word unigrams and bigrams in the pool", "-    (lower-cased [a-z']+ runs).  Restricting to *common* features is deliberate:", "-    it makes every feature's target/pool frequency ratio well-estimated and keeps", "-    rare, near-unique tokens (proper-name lists, table cells) from dominating the", "-    score with sampling-noise weights.", "-  * For each feature f, weight(f) = clip( log P_target(f) - log P_pool(f) ), where", "-    P_target is estimated from the decoded target sample (`multi_dev.npy`, the", "-    disclosed target) and P_pool from a random pool background sample.", "-  * score(doc) = mean_{f in vocab-features(doc)} weight(f).", "+1. DSIR-style importance weighting (Xie et al. 2023).  On a fixed common vocabulary", "+   (most-frequent word unigrams+bigrams, lower-cased [a-z']+), score a feature f by", "+   `weight(f) = clip(log P_target(f) - log P_pool(f))`.  P_target is estimated from", "+   the decoded disclosed target sample (`multi_dev.npy`); P_pool from a random pool", "+   background sample.  Restricting to *common* features keeps rare near-unique tokens", "+   (proper-name lists / table cells) from dominating with sampling-noise weights.", " ", "-High score  = vocabulary/register of the multi-domain HQ target (news, encyclopedic", "-prose, technical Q&A).  Low score = pool-typical junk vocabulary (shopping-cart /", "-sign-in / navigation / SEO boilerplate).  Because features are common content words", "-and phrases, the criterion keys on register, not on the target sample's surface", "-formatting, so it transfers to the disjoint hidden target of the same domain.", "+2. Register balancing.  A single global target classifier collapses the budget onto", "+   the register that is easiest to separate from pool junk (news: ~77% of the budget)", "+   and starves the technical and encyclopedic registers, which then dominate the", "+   average loss.  Instead we split the target sample into the four registers, train", "+   one importance classifier per register, and fill the budget by round-robin over", "+   registers (always extend the register with the fewest tokens so far).  The budget", "+   is thus 25/25/25/25 by construction.", " ", "-Quality gate (drop): fragments (too short / too few in-vocab features), digit/table", "-dumps (high digit ratio), navigation-menu boilerplate (mostly tiny lines), and", "-heavily repeated-line documents.  Exact and near-duplicate documents are removed,", "-keeping the single highest-scoring copy.", "+Quality gates (drop): fragments (too short / too few in-vocab features), digit/table", "+dumps, navigation-menu boilerplate (mostly tiny lines), repeated-line pages, largely", "+non-ASCII (foreign/CJK) pages, and — for the three PROSE registers — documents below", "+a function-word (stopword) ratio floor, which removes proper-noun keyword-salad SEO", "+spam that a mean-log-odds score otherwise ranks high.  The code register is exempt", "+from the stopword floor (code legitimately has few function words, and markup/code", "+token exposure helps the target's HTML/code quarter).  Exact and near-duplicate", "+documents are removed, keeping the single highest-scoring copy.  Hashing is crc32", "+(deterministic) so the script reproduces its `selection.json` exactly.", " ", "-Output: `selection.json` — surviving ids sorted best-first.  The frozen training", "-pipeline consumes this order until the 12M-token budget is filled.", "+Output: `selection.json` — surviving ids sorted best-first (register-interleaved).", "+The frozen training pipeline consumes this order until the 12M-token budget is full.", "+", "+Reproduce:  python3 curate.py            # writes /workspace/submission/selection.json", " \"\"\"", "-import argparse, json, re, sys, math", "+import argparse, json, re, sys, math, zlib", " from collections import Counter", " import numpy as np", " ", " POOL = \"/workspace/data/pool.jsonl\"", "-DEV_NPY = \"/workspace/data/multi_dev.npy\"     # disclosed target sample (GPT-2 token ids)", "+DEV_NPY = \"/workspace/data/multi_dev.npy\"          # disclosed target sample (GPT-2 ids)", " OUT = \"/workspace/submission/selection.json\"", " ", "-# ------------------------------ config ---------------------------------------", "-N_UNI = 40000                 # keep this many most-frequent unigrams", "-N_BI = 40000                  # keep this many most-frequent bigrams", "-BG_SAMPLE = 60000             # pool docs for background LM + vocab frequencies", "-ALPHA = 1.0                   # add-alpha smoothing", "-CLIP = 4.0                    # clip per-feature log-weight to [-CLIP, +CLIP]", "-MIN_FEATS = 20                # doc must have >= this many in-vocab features to score", "+# ---- hyper-parameters (fixed) -----------------------------------------------", "+N_UNI, N_BI = 40000, 40000                          # common vocabulary size", "+BG_SAMPLE = 60000                                   # pool docs for background + vocab", "+ALPHA, CLIP, MIN_FEATS = 1.0, 4.0, 20               # smoothing / weight clip / min feats", "+REGISTERS = [\"tech\", \"wiki\", \"news\", \"prose\"]", "+# per-register function-word floor: prose registers require fluent English; code exempt", "+STOP_FLOOR = {\"tech\": 0.0, \"wiki\": 0.20, \"news\": 0.20, \"prose\": 0.20}", "+ALLOC_DEFAULT = \"1,1,1,1\"                            # equal token budget per register", " WORD_RE = re.compile(r\"[a-z']+\")", "+WS_RE = re.compile(r\"\\s+\")", "+# general quality gates", "+MIN_CHARS, MIN_WORDS = 200, 60", "+MAX_DIGIT_FRAC, MAX_SHORT_LINE_FRAC, MAX_DUP_LINE_FRAC, MAX_NONASCII_FRAC = 0.15, 0.66, 0.50, 0.10", "+TARGET_TOTAL = 40_000_000                            # emit ~40M est-tokens (budget is 12M)", " ", "-# quality gate thresholds", "-MIN_CHARS = 200", "-MIN_WORDS = 60", "-MAX_DIGIT_FRAC = 0.15         # table / census / listing dumps", "-MAX_SHORT_LINE_FRAC = 0.66    # navigation-menu / listing boilerplate", "-MAX_DUP_LINE_FRAC = 0.50      # heavily repeated lines", "+STOP = set((", "+    \"the of and to a in is that it for on with as was were be been being by this these those are am \"", "+    \"at from or an which not no but have has had they you we he she his her their its will would can could \"", "+    \"i my me your our us them do does did so if then than out up down about into over under after before \"", "+    \"all any more most some such only own same other new one two first last time year people\"", "+).split())", " ", " ", "-def words_of(text):", "-    return WORD_RE.findall(text.lower())", "+def words_of(t):", "+    return WORD_RE.findall(t.lower())", " ", " ", "-def quality_ok(text, words):", "+def dkey(s):", "+    return zlib.crc32(s.encode(\"utf-8\", \"ignore\")) & 0xFFFFFFFF", "+", "+", "+def general_ok(text, words):", "     if len(text) < MIN_CHARS or len(words) < MIN_WORDS:", "         return False", "-    ndig = sum(c.isdigit() for c in text)", "-    if ndig / len(text) > MAX_DIGIT_FRAC:", "+    if sum(1 for c in text if ord(c) > 127) / len(text) > MAX_NONASCII_FRAC:", "         return False", "+    if sum(c.isdigit() for c in text) / len(text) > MAX_DIGIT_FRAC:", "+        return False", "     lines = [ln for ln in text.split(\"\\n\") if ln.strip()]", "     if len(lines) >= 8:", "-        short = sum(1 for ln in lines if len(ln.split()) <= 3)", "-        if short / len(lines) > MAX_SHORT_LINE_FRAC:", "+        if sum(1 for ln in lines if len(ln.split()) <= 3) / len(lines) > MAX_SHORT_LINE_FRAC:", "             return False", "         if 1 - len(set(lines)) / len(lines) > MAX_DUP_LINE_FRAC:", "             return False", "     return True", " ", " ", "+def register_of(txt):", "+    \"\"\"Assign a decoded target segment to one of four registers by surface markers.\"\"\"", "+    n = max(1, len(txt))", "+    sym = sum(c in \"<>{}=;/&|_#\\\\`$\" for c in txt)", "+    if sym / n > 0.02 or \"&lt;\" in txt or \"&gt;\" in txt or (txt.count(\"{\") + txt.count(\"}\") >= 4):", "+        return \"tech\"", "+    if \" @.@ \" in txt or \" @-@ \" in txt or (txt.count(\" .\") + txt.count(\" ,\")) > n / 250:", "+        return \"wiki\"                                # WikiText-103 spaced punctuation", "+    tl = txt.lower()", "+    if re.search(r\"\\((reuters|ians|ani|ap|afp|pti|xinhua|dw)\\)\", tl) or \\", "+       re.search(r\"\\b(said|told reporters|according to|reported|correspondent|\"", "+                 r\"on (monday|tuesday|wednesday|thursday|friday|saturday|sunday))\\b\", tl):", "+        return \"news\"", "+    return \"prose\"", "+", "+", "+def doc_features(text, uni_ix, bi_ix):", "+    w = words_of(text)", "+    feats = [uni_ix[x] for x in w if x in uni_ix]", "+    prev = None", "+    for x in w:", "+        if prev is not None:", "+            b = bi_ix.get((prev, x))", "+            if b is not None:", "+                feats.append(b)", "+        prev = x", "+    stop_frac = (sum(1 for x in w if x in STOP) / len(w)) if w else 0.0", "+    return w, feats, stop_frac", "+", "+", " def main():", "     ap = argparse.ArgumentParser()", "     ap.add_argument(\"--out\", default=OUT)", "     ap.add_argument(\"--seed\", type=int, default=0)", "-    ap.add_argument(\"--min_score\", type=float, default=None)", "+    ap.add_argument(\"--alloc\", default=ALLOC_DEFAULT, help=\"token weights tech,wiki,news,prose\")", "     args = ap.parse_args()", "     rng = np.random.default_rng(args.seed)", "+    alloc = np.array([float(x) for x in args.alloc.split(\",\")], dtype=float)", "+    alloc = alloc / alloc.sum()", " ", "     # ---- 1. load pool -------------------------------------------------------", "     print(\"loading pool...\", file=sys.stderr)"]}, {"oldStart": 95, "oldLines": 120, "newStart": 143, "newLines": 123, "lines": ["         for line in f:", "             r = json.loads(line)", "             ids.append(r[\"id\"]); texts.append(r[\"text\"])", "+    ids = np.asarray(ids)", "     N = len(ids)", "-    print(f\"  {N} docs\", file=sys.stderr)", " ", "-    # ---- 2. background sample: build common vocab + pool counts -------------", "+    # ---- 2. common vocabulary + pool background counts ----------------------", "     print(\"building vocabulary + background counts...\", file=sys.stderr)", "     bg_idx = rng.choice(N, size=min(BG_SAMPLE, N), replace=False)", "     uni_bg, bi_bg = Counter(), Counter()", "     for j in bg_idx:", "-        w = words_of(texts[j])", "-        uni_bg.update(w)", "-        bi_bg.update(zip(w, w[1:]))", "-    uni_vocab = [w for w, _ in uni_bg.most_common(N_UNI)]", "-    bi_vocab = [b for b, _ in bi_bg.most_common(N_BI)]", "-    uni_ix = {w: k for k, w in enumerate(uni_vocab)}", "-    bi_ix = {b: N_UNI + k for k, b in enumerate(bi_vocab)}", "+        w = words_of(texts[j]); uni_bg.update(w); bi_bg.update(zip(w, w[1:]))", "+    uni_ix = {w: k for k, (w, _) in enumerate(uni_bg.most_common(N_UNI))}", "+    bi_ix = {b: N_UNI + k for k, (b, _) in enumerate(bi_bg.most_common(N_BI))}", "     Vf = N_UNI + N_BI", "-    print(f\"  vocab: {len(uni_ix)} unigrams + {len(bi_ix)} bigrams\", file=sys.stderr)", "-", "-    bg_counts = np.zeros(Vf, dtype=np.float64)", "+    bg_counts = np.zeros(Vf)", "     for w, c in uni_bg.items():", "         k = uni_ix.get(w)", "         if k is not None: bg_counts[k] = c", "     for b, c in bi_bg.items():", "         k = bi_ix.get(b)", "         if k is not None: bg_counts[k] = c", "+    log_bg = np.log(bg_counts + ALPHA) - math.log(bg_counts.sum() + ALPHA * Vf)", " ", "-    # ---- 3. target counts on the same vocab --------------------------------", "-    print(\"decoding target sample + counting...\", file=sys.stderr)", "+    # ---- 3. per-register target counts from decoded target sample -----------", "+    print(\"decoding target + per-register counts...\", file=sys.stderr)", "     from transformers import AutoTokenizer", "     tok = AutoTokenizer.from_pretrained(\"gpt2\")", "-    dev = np.load(DEV_NPY).astype(np.int64)", "-    EOS = 50256", "-    idx = np.where(dev == EOS)[0]", "+    dev = np.load(DEV_NPY).astype(np.int64); EOS = 50256", "+    cuts = np.where(dev == EOS)[0]", "     segs, prev = [], 0", "-    for i in idx:", "+    for i in cuts:", "         if i > prev: segs.append((prev, i))", "         prev = i + 1", "     if prev < len(dev): segs.append((prev, len(dev)))", "-    tgt_counts = np.zeros(Vf, dtype=np.float64)", "+    reg_counts = {r: np.zeros(Vf) for r in REGISTERS}", "+    reg_toks = {r: 0 for r in REGISTERS}", "     for s, e in segs:", "-        w = words_of(tok.decode(dev[s:e].tolist()))", "+        txt = tok.decode(dev[s:e].tolist()); r = register_of(txt); reg_toks[r] += (e - s)", "+        w = words_of(txt); cc = reg_counts[r]", "         for x in w:", "             k = uni_ix.get(x)", "-            if k is not None: tgt_counts[k] += 1", "+            if k is not None: cc[k] += 1", "         for x in zip(w, w[1:]):", "             k = bi_ix.get(x)", "-            if k is not None: tgt_counts[k] += 1", "-    print(f\"  target: {len(segs)} segments\", file=sys.stderr)", "+            if k is not None: cc[k] += 1", "+    tot = sum(reg_toks.values())", "+    print(\"  dev register shares:\", {r: f\"{100*reg_toks[r]/tot:.0f}%\" for r in REGISTERS}, file=sys.stderr)", " ", "-    # ---- 4. per-feature clipped log-likelihood-ratio weights ---------------", "-    T = tgt_counts.sum(); B = bg_counts.sum()", "-    log_tgt = np.log(tgt_counts + ALPHA) - math.log(T + ALPHA * Vf)", "-    log_bg = np.log(bg_counts + ALPHA) - math.log(B + ALPHA * Vf)", "-    weight = np.clip(log_tgt - log_bg, -CLIP, CLIP)", "+    weight = {}", "+    for r in REGISTERS:", "+        log_t = np.log(reg_counts[r] + ALPHA) - math.log(reg_counts[r].sum() + ALPHA * Vf)", "+        weight[r] = np.clip(log_t - log_bg, -CLIP, CLIP)", " ", "-    # interpretability: most target-ish / most junk-ish words", "-    inv_uni = {k: w for w, k in uni_ix.items()}", "-    uni_w = [(weight[k], inv_uni[k]) for k in range(N_UNI)]", "-    uni_w.sort()", "-    print(\"  most JUNK unigrams:\", [w for _, w in uni_w[:15]], file=sys.stderr)", "-    print(\"  most TARGET unigrams:\", [w for _, w in uni_w[-15:]], file=sys.stderr)", "-", "-    # ---- 5. score + gate every document ------------------------------------", "+    # ---- 4. score + gate every document under all four registers ------------", "     print(\"scoring all documents...\", file=sys.stderr)", "-    scores = np.full(N, -1e9, dtype=np.float64)", "+    score = {r: np.full(N, -1e9) for r in REGISTERS}", "+    est_tok = np.zeros(N, dtype=np.int64)", "+    ok = np.zeros(N, dtype=bool)", "     for k in range(N):", "         text = texts[k]", "-        w = words_of(text)", "-        if not quality_ok(text, w):", "+        est_tok[k] = len(text) // 4", "+        w, feats, stop_frac = doc_features(text, uni_ix, bi_ix)", "+        if not general_ok(text, w) or len(feats) < MIN_FEATS:", "             continue", "-        feats = [uni_ix[x] for x in w if x in uni_ix]", "-        bg = bi_ix", "-        prev = None", "-        for x in w:", "-            if prev is not None:", "-                bi = bg.get((prev, x))", "-                if bi is not None: feats.append(bi)", "-            prev = x", "-        if len(feats) < MIN_FEATS:", "-            continue", "-        scores[k] = weight[np.asarray(feats)].mean()", "+        fa = np.asarray(feats)", "+        any_ok = False", "+        for r in REGISTERS:", "+            if stop_frac < STOP_FLOOR[r]:", "+                continue", "+            score[r][k] = weight[r][fa].mean(); any_ok = True", "+        ok[k] = any_ok", "         if (k + 1) % 40000 == 0:", "             print(f\"  {k+1}/{N}\", file=sys.stderr)", "-    ok_mask = scores > -1e8", "-    print(f\"  {int(ok_mask.sum())}/{N} docs passed the quality gate\", file=sys.stderr)", "+    print(f\"  {int(ok.sum())}/{N} eligible for >=1 register\", file=sys.stderr)", " ", "-    # ---- 6. rank, dedup, emit ----------------------------------------------", "-    order = np.argsort(-scores)", "-    order = order[ok_mask[order]]", "-    if args.min_score is not None:", "-        order = order[scores[order] >= args.min_score]", "-", "-    seen_full, seen_sig, sel_ids = set(), set(), []", "-    ws = re.compile(r\"\\s+\")", "-    dropped_dup = 0", "-    for k in order:", "-        t = texts[k]", "-        full = hash(ws.sub(\" \", t).strip())", "-        # near-dup signature: interior slice, robust to differing nav prefixes", "-        core = ws.sub(\" \", t)[120:920]", "-        sig = hash(core) if len(core) > 200 else full", "+    # ---- 5. deterministic (near-)dedup, keeping highest-scoring copy --------", "+    best = np.max(np.stack([score[r] for r in REGISTERS]), axis=0)", "+    seen_full, seen_sig = set(), set()", "+    keep = np.zeros(N, dtype=bool)", "+    for k in np.argsort(-best, kind=\"stable\"):", "+        if not ok[k]:", "+            continue", "+        norm = WS_RE.sub(\" \", texts[k]).strip()", "+        full = dkey(norm)", "+        core = norm[120:920]", "+        sig = dkey(core) if len(core) > 200 else full", "         if full in seen_full or sig in seen_sig:", "-            dropped_dup += 1", "             continue", "-        seen_full.add(full); seen_sig.add(sig)", "-        sel_ids.append(int(ids[k]))", "-    print(f\"  dropped {dropped_dup} (near-)duplicate docs\", file=sys.stderr)", "+        seen_full.add(full); seen_sig.add(sig); keep[k] = True", " ", "-    ss = np.sort(scores[ok_mask])", "-    for p in [50, 75, 90, 95, 99, 100]:", "-        print(f\"  score p{p}: {ss[min(len(ss)-1, int(p/100*len(ss)))]:.4f}\", file=sys.stderr)", "+    # ---- 6. register-balanced round-robin fill ------------------------------", "+    ranked = {}", "+    for r in REGISTERS:", "+        sc = score[r].copy(); sc[~keep] = -1e9", "+        o = np.argsort(-sc, kind=\"stable\")", "+        ranked[r] = o[keep[o] & (sc[o] > -1e8)]", "+    ptr = {r: 0 for r in REGISTERS}", "+    used = np.zeros(N, dtype=bool)", "+    got = {r: 0 for r in REGISTERS}", "+    sel = []", "+    while True:", "+        cand = [r for r in REGISTERS if ptr[r] < len(ranked[r])]", "+        if not cand:", "+            break", "+        r = min(cand, key=lambda r: got[r] / alloc[REGISTERS.index(r)]", "+                if alloc[REGISTERS.index(r)] > 0 else 1e18)", "+        while ptr[r] < len(ranked[r]) and used[ranked[r][ptr[r]]]:", "+            ptr[r] += 1", "+        if ptr[r] >= len(ranked[r]):", "+            continue", "+        k = ranked[r][ptr[r]]; ptr[r] += 1", "+        used[k] = True; sel.append(int(ids[k])); got[r] += int(est_tok[k])", "+        if sum(got.values()) >= TARGET_TOTAL:", "+            break", " ", "-    json.dump(sel_ids, open(args.out, \"w\"))", "-    print(f\"wrote {len(sel_ids)} ids -> {args.out}\", file=sys.stderr)", "+    print(\"  emitted register est-token shares:\",", "+          {r: f\"{100*got[r]/max(1, sum(got.values())):.0f}%\" for r in REGISTERS}, file=sys.stderr)", "+    json.dump(sel, open(args.out, \"w\"))", "+    print(f\"wrote {len(sel)} ids -> {args.out}\", file=sys.stderr)", " ", " ", " if __name__ == \"__main__\":"]}], "originalFile": "#!/usr/bin/env python3\n\"\"\"Curate the raw web pool for from-scratch LM pretraining under a fixed token budget.\n\nSTATED CRITERION\n----------------\nThe disclosed evaluation target is a BROAD, multi-domain high-quality English\ndistribution (equal parts Wikipedia/encyclopedic, general HQ web prose, news, and\ntechnical Q&A).  We rank every pool document by how much its *content* resembles\nthat target distribution, drop near-duplicates, and drop non-prose boilerplate.\n\nScoring is DSIR-style importance weighting (Xie et al. 2023) on a FIXED, common\nbag-of-words vocabulary:\n\n  * Vocabulary = the most frequent word unigrams and bigrams in the pool\n    (lower-cased [a-z']+ runs).  Restricting to *common* features is deliberate:\n    it makes every feature's target/pool frequency ratio well-estimated and keeps\n    rare, near-unique tokens (proper-name lists, table cells) from dominating the\n    score with sampling-noise weights.\n  * For each feature f, weight(f) = clip( log P_target(f) - log P_pool(f) ), where\n    P_target is estimated from the decoded target sample (`multi_dev.npy`, the\n    disclosed target) and P_pool from a random pool background sample.\n  * score(doc) = mean_{f in vocab-features(doc)} weight(f).\n\nHigh score  = vocabulary/register of the multi-domain HQ target (news, encyclopedic\nprose, technical Q&A).  Low score = pool-typical junk vocabulary (shopping-cart /\nsign-in / navigation / SEO boilerplate).  Because features are common content words\nand phrases, the criterion keys on register, not on the target sample's surface\nformatting, so it transfers to the disjoint hidden target of the same domain.\n\nQuality gate (drop): fragments (too short / too few in-vocab features), digit/table\ndumps (high digit ratio), navigation-menu boilerplate (mostly tiny lines), and\nheavily repeated-line documents.  Exact and near-duplicate documents are removed,\nkeeping the single highest-scoring copy.\n\nOutput: `selection.json` — surviving ids sorted best-first.  The frozen training\npipeline consumes this order until the 12M-token budget is filled.\n\"\"\"\nimport argparse, json, re, sys, math\nfrom collections import Counter\nimport numpy as np\n\nPOOL = \"/workspace/data/pool.jsonl\"\nDEV_NPY = \"/workspace/data/multi_dev.npy\"     # disclosed target sample (GPT-2 token ids)\nOUT = \"/workspace/submission/selection.json\"\n\n# ------------------------------ config ---------------------------------------\nN_UNI = 40000                 # keep this many most-frequent unigrams\nN_BI = 40000                  # keep this many most-frequent bigrams\nBG_SAMPLE = 60000             # pool docs for background LM + vocab frequencies\nALPHA = 1.0                   # add-alpha smoothing\nCLIP = 4.0                    # clip per-feature log-weight to [-CLIP, +CLIP]\nMIN_FEATS = 20                # doc must have >= this many in-vocab features to score\nWORD_RE = re.compile(r\"[a-z']+\")\n\n# quality gate thresholds\nMIN_CHARS = 200\nMIN_WORDS = 60\nMAX_DIGIT_FRAC = 0.15         # table / census / listing dumps\nMAX_SHORT_LINE_FRAC = 0.66    # navigation-menu / listing boilerplate\nMAX_DUP_LINE_FRAC = 0.50      # heavily repeated lines\n\n\ndef words_of(text):\n    return WORD_RE.findall(text.lower())\n\n\ndef quality_ok(text, words):\n    if len(text) < MIN_CHARS or len(words) < MIN_WORDS:\n        return False\n    ndig = sum(c.isdigit() for c in text)\n    if ndig / len(text) > MAX_DIGIT_FRAC:\n        return False\n    lines = [ln for ln in text.split(\"\\n\") if ln.strip()]\n    if len(lines) >= 8:\n        short = sum(1 for ln in lines if len(ln.split()) <= 3)\n        if short / len(lines) > MAX_SHORT_LINE_FRAC:\n            return False\n        if 1 - len(set(lines)) / len(lines) > MAX_DUP_LINE_FRAC:\n            return False\n    return True\n\n\ndef main():\n    ap = argparse.ArgumentParser()\n    ap.add_argument(\"--out\", default=OUT)\n    ap.add_argument(\"--seed\", type=int, default=0)\n    ap.add_argument(\"--min_score\", type=float, default=None)\n    args = ap.parse_args()\n    rng = np.random.default_rng(args.seed)\n\n    # ---- 1. load pool -------------------------------------------------------\n    print(\"loading pool...\", file=sys.stderr)\n    ids, texts = [], []\n    with open(POOL) as f:\n        for line in f:\n            r = json.loads(line)\n            ids.append(r[\"id\"]); texts.append(r[\"text\"])\n    N = len(ids)\n    print(f\"  {N} docs\", file=sys.stderr)\n\n    # ---- 2. background sample: build common vocab + pool counts -------------\n    print(\"building vocabulary + background counts...\", file=sys.stderr)\n    bg_idx = rng.choice(N, size=min(BG_SAMPLE, N), replace=False)\n    uni_bg, bi_bg = Counter(), Counter()\n    for j in bg_idx:\n        w = words_of(texts[j])\n        uni_bg.update(w)\n        bi_bg.update(zip(w, w[1:]))\n    uni_vocab = [w for w, _ in uni_bg.most_common(N_UNI)]\n    bi_vocab = [b for b, _ in bi_bg.most_common(N_BI)]\n    uni_ix = {w: k for k, w in enumerate(uni_vocab)}\n    bi_ix = {b: N_UNI + k for k, b in enumerate(bi_vocab)}\n    Vf = N_UNI + N_BI\n    print(f\"  vocab: {len(uni_ix)} unigrams + {len(bi_ix)} bigrams\", file=sys.stderr)\n\n    bg_counts = np.zeros(Vf, dtype=np.float64)\n    for w, c in uni_bg.items():\n        k = uni_ix.get(w)\n        if k is not None: bg_counts[k] = c\n    for b, c in bi_bg.items():\n        k = bi_ix.get(b)\n        if k is not None: bg_counts[k] = c\n\n    # ---- 3. target counts on the same vocab --------------------------------\n    print(\"decoding target sample + counting...\", file=sys.stderr)\n    from transformers import AutoTokenizer\n    tok = AutoTokenizer.from_pretrained(\"gpt2\")\n    dev = np.load(DEV_NPY).astype(np.int64)\n    EOS = 50256\n    idx = np.where(dev == EOS)[0]\n    segs, prev = [], 0\n    for i in idx:\n        if i > prev: segs.append((prev, i))\n        prev = i + 1\n    if prev < len(dev): segs.append((prev, len(dev)))\n    tgt_counts = np.zeros(Vf, dtype=np.float64)\n    for s, e in segs:\n        w = words_of(tok.decode(dev[s:e].tolist()))\n        for x in w:\n            k = uni_ix.get(x)\n            if k is not None: tgt_counts[k] += 1\n        for x in zip(w, w[1:]):\n            k = bi_ix.get(x)\n            if k is not None: tgt_counts[k] += 1\n    print(f\"  target: {len(segs)} segments\", file=sys.stderr)\n\n    # ---- 4. per-feature clipped log-likelihood-ratio weights ---------------\n    T = tgt_counts.sum(); B = bg_counts.sum()\n    log_tgt = np.log(tgt_counts + ALPHA) - math.log(T + ALPHA * Vf)\n    log_bg = np.log(bg_counts + ALPHA) - math.log(B + ALPHA * Vf)\n    weight = np.clip(log_tgt - log_bg, -CLIP, CLIP)\n\n    # interpretability: most target-ish / most junk-ish words\n    inv_uni = {k: w for w, k in uni_ix.items()}\n    uni_w = [(weight[k], inv_uni[k]) for k in range(N_UNI)]\n    uni_w.sort()\n    print(\"  most JUNK unigrams:\", [w for _, w in uni_w[:15]], file=sys.stderr)\n    print(\"  most TARGET unigrams:\", [w for _, w in uni_w[-15:]], file=sys.stderr)\n\n    # ---- 5. score + gate every document ------------------------------------\n    print(\"scoring all documents...\", file=sys.stderr)\n    scores = np.full(N, -1e9, dtype=np.float64)\n    for k in range(N):\n        text = texts[k]\n        w = words_of(text)\n        if not quality_ok(text, w):\n            continue\n        feats = [uni_ix[x] for x in w if x in uni_ix]\n        bg = bi_ix\n        prev = None\n        for x in w:\n            if prev is not None:\n                bi = bg.get((prev, x))\n                if bi is not None: feats.append(bi)\n            prev = x\n        if len(feats) < MIN_FEATS:\n            continue\n        scores[k] = weight[np.asarray(feats)].mean()\n        if (k + 1) % 40000 == 0:\n            print(f\"  {k+1}/{N}\", file=sys.stderr)\n    ok_mask = scores > -1e8\n    print(f\"  {int(ok_mask.sum())}/{N} docs passed the quality gate\", file=sys.stderr)\n\n    # ---- 6. rank, dedup, emit ----------------------------------------------\n    order = np.argsort(-scores)\n    order = order[ok_mask[order]]\n    if args.min_score is not None:\n        order = order[scores[order] >= args.min_score]\n\n    seen_full, seen_sig, sel_ids = set(), set(), []\n    ws = re.compile(r\"\\s+\")\n    dropped_dup = 0\n    for k in order:\n        t = texts[k]\n        full = hash(ws.sub(\" \", t).strip())\n        # near-dup signature: interior slice, robust to differing nav prefixes\n        core = ws.sub(\" \", t)[120:920]\n        sig = hash(core) if len(core) > 200 else full\n        if full in seen_full or sig in seen_sig:\n            dropped_dup += 1\n            continue\n        seen_full.add(full); seen_sig.add(sig)\n        sel_ids.append(int(ids[k]))\n    print(f\"  dropped {dropped_dup} (near-)duplicate docs\", file=sys.stderr)\n\n    ss = np.sort(scores[ok_mask])\n    for p in [50, 75, 90, 95, 99, 100]:\n        print(f\"  score p{p}: {ss[min(len(ss)-1, int(p/100*len(ss)))]:.4f}\", file=sys.stderr)\n\n    json.dump(sel_ids, open(args.out, \"w\"))\n    print(f\"wrote {len(sel_ids)} ids -> {args.out}\", file=sys.stderr)\n\n\nif __name__ == \"__main__\":\n    main()\n", "userModified": false}

2133
---
total 288
drwxr-xr-x 1 root root    140 Aug  4 05:26 .
drwxr-xr-x 1 root root    180 Aug  4 05:17 ..
-rw-r--r-- 1 root root  11816 Aug  4 05:26 curate.py
-rw-r--r-- 1 root root 282549 Aug  4 05:16 selection.json

[stdout]
2133
---
total 288
drwxr-xr-x 1 root root    140 Aug  4 05:26 .
drwxr-xr-x 1 root root    180 Aug  4 05:17 ..
-rw-r--r-- 1 root root  11816 Aug  4 05:26 curate.py
-rw-r--r-- 1 root root 282549 Aug  4 05:16 selection.json
[metadata] {"noOutputExpected": false}

Command running in background with ID: b6h6h5fls. Output is being written to: /tmp/claude-0/-workspace/06cb3775-ded4-4ad5-84cf-ed7eaa26a5bd/tasks/b6h6h5fls.output. You will be notified when it completes. To check interim output, use Read on that file path.
Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "b6h6h5fls", "backgroundCwdHint": "Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands."}

File created successfully at: /tmp/validate.py (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/tmp/validate.py", "content": "#!/usr/bin/env python3\nimport json, sys, numpy as np\nPOOL=\"/workspace/data/pool.jsonl\"; SEL=\"/workspace/submission/selection.json\"\nsel=json.load(open(SEL))\nprint(\"type:\", type(sel).__name__, \"len:\", len(sel))\nassert isinstance(sel,list) and all(isinstance(x,int) for x in sel), \"must be list[int]\"\n# duplicates\nif len(sel)!=len(set(sel)):\n    from collections import Counter\n    d=[k for k,c in Counter(sel).items() if c>1]\n    print(\"DUPLICATE ids:\",d[:10]); sys.exit(1)\nprint(\"no duplicates: OK\")\n# load pool ids+texts\nids=[]; texts={}\nwith open(POOL) as f:\n    for line in f:\n        r=json.loads(line); ids.append(r[\"id\"]); texts[r[\"id\"]]=r[\"text\"]\nidset=set(ids)\nprint(\"pool size:\",len(ids),\"id range:\",min(ids),\"..\",max(ids))\nbad=[x for x in sel if x not in idset]\nprint(\"out-of-pool ids:\",len(bad), bad[:10])\nassert not bad, \"selection has ids not in pool\"\n# real gpt2 token count (matches training pipeline: concat doc tokens; +1 EOS/doc)\nfrom transformers import AutoTokenizer\ntok=AutoTokenizer.from_pretrained(\"gpt2\")\ntotal=0; n=0\nBUDGET=12_000_000\nfor x in sel:\n    t=texts[x]\n    total+=len(tok(t)[\"input_ids\"])+1   # +EOS separator\n    n+=1\n    if total>=BUDGET and n%2000==0:\n        pass\nprint(f\"selected docs: {len(sel)}   GPT-2 tokens (with EOS/doc): {total:,}\")\nprint(f\"budget 12,000,000 -> covered: {total>=BUDGET}  margin x{total/BUDGET:.2f}\")\n", "structuredPatch": [], "originalFile": null, "userModified": false}

<system-reminder>Warning: the file exists but the contents are empty.</system-reminder>

[metadata] {"type": "text", "file": {"filePath": "/tmp/claude-0/-workspace/06cb3775-ded4-4ad5-84cf-ed7eaa26a5bd/tasks/b6h6h5fls.output", "content": "", "numLines": 1, "startLine": 1, "totalLines": 1}}

=== curate_run.log ===
loading pool...
building vocabulary + background counts...
decoding target + per-register counts...
  dev register shares: {'tech': '26%', 'wiki': '25%', 'news': '34%', 'prose': '15%'}
scoring all documents...
=== remaining ===
2053

[stdout]
=== curate_run.log ===
loading pool...
building vocabulary + background counts...
decoding target + per-register counts...
  dev register shares: {'tech': '26%', 'wiki': '25%', 'news': '34%', 'prose': '15%'}
scoring all documents...
=== remaining ===
2053
[metadata] {"noOutputExpected": false}