claude-opus-4-8

The file /workspace/submission/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/submission/curate.py", "oldString": "pos_uni = {r: np.zeros(V, np.float64) for r in REGISTERS}\npos_big = {r: np.zeros(DB, np.float64) for r in REGISTERS}\nfor k in range(len(b) - 1):\n    d = dev[b[k]:b[k+1]]\n    d = d[d != EOS]\n    if len(d) < 20:\n        continue\n    raw = tok.decode(d.tolist())\n    r = register_of(raw)", "newString": "pos_uni = {r: np.zeros(V, np.float64) for r in REGISTERS}\npos_big = {r: np.zeros(DB, np.float64) for r in REGISTERS}\nraw_tok = {r: 0 for r in REGISTERS}             # RAW token share per register (for quota)\nfor k in range(len(b) - 1):\n    d = dev[b[k]:b[k+1]]\n    d = d[d != EOS]\n    if len(d) < 20:\n        continue\n    raw = tok.decode(d.tolist())\n    r = register_of(raw)\n    raw_tok[r] += len(d)", "originalFile": "\"\"\"Curate a pretraining subset from a raw web pool by register-balanced DSIR.\n\nThe disclosed evaluation target is a BROAD, multi-domain HQ English mix in equal\nparts: encyclopedic (Wikipedia), general web prose, news, and technical Q&A.\nHeld-out perplexity averages over all four registers, so a training set that\nover-weights one register (e.g. news, which is easiest to match) leaves the\nothers under-trained and drags the average up. We therefore select in two steps:\n\n  (1) DSIR importance scoring, PER REGISTER.  We split the disclosed target\n      (data/multi_dev.npy, GPT-2 tokens) into four register buckets by surface\n      signatures, and for each register r estimate a bag-of-n-grams importance\n      weight\n          w_r(gram) = log p_target_r(gram) - log p_pool(gram)\n      (unigrams + hashed bigrams; p_pool from the whole pool as background).\n      Each pool document is scored against every register using a PRESENCE-based\n      mean over its unique n-gram types (repetition-robust), behind a length\n      floor and a letter-fraction gate that reject fragments and non-prose\n      (directory listings, tables).  A document is routed to its best-fit\n      register (argmax score).\n\n  (2) Balanced quota fill.  Within each register we rank routed documents by\n      score, then interleave the four ranked lists by EQUAL token quota, so the\n      first 12M tokens the trainer consumes are ~25% from each register --\n      matching the equal-parts evaluation mixture.\n\nThis is Data Selection via Importance Resampling (Xie et al. 2023) with a\nper-domain target, plus classic quality gating.  No labels, no internet: the only\nsupervision is the disclosed target token stream.\n\"\"\"\nimport json, time, re, numpy as np\nfrom transformers import AutoTokenizer\n\nPOOL = \"/workspace/data/pool.jsonl\"\nDEV  = \"/workspace/data/multi_dev.npy\"\nOUT  = \"/workspace/submission/selection.json\"\n\nEOS   = 50256\nV     = 50257            # GPT-2 vocab\nDB    = 1 << 20          # hashed bigram buckets\nALPHA = 0.1             # additive smoothing\nMIN_TOK   = 100         # length floor: drop fragments\nMIN_ALPHA = 0.55        # letter-fraction floor: reject non-prose\nLAMBDA    = 0.5         # weight of bigram term relative to unigram term\nCOVER_TOKENS = 30_000_000   # emit ids covering ~2.5x the 12M budget\nREGISTERS = [\"wiki\", \"web\", \"news\", \"code_qa\"]\n# QUOTA (target training mixture) is set below to the target's own per-register\n# TOKEN proportions: held-out PPL is window-weighted over the concatenated target,\n# so the eval effectively weights each register by its token share -- matching that\n# share in training minimizes the token-weighted average loss.\n\n_NONALPHA = re.compile(r\"[^A-Za-z]\")\n_NEWS = re.compile(r\"\\b(reuters|said|told|according to|\"\n                   r\"on (monday|tuesday|wednesday|thursday|friday|saturday|sunday))\\b\")\n\ndef normalize_target(t):\n    \"\"\"Strip WikiText-103 surface formatting so the wiki target matches on CONTENT\n    vocabulary, not on markup absent from the raw pool (which would otherwise make\n    every pool doc look un-wiki-like and starve the encyclopedic quota).\"\"\"\n    t = t.replace(\"@-@\", \"-\").replace(\"@,@\", \",\").replace(\"@.@\", \".\")\n    t = re.sub(r\"\\s+([,.;:!?)])\", r\"\\1\", t)     # drop space before punctuation\n    t = re.sub(r\"([(\\[])\\s+\", r\"\\1\", t)          # drop space after open bracket\n    t = re.sub(r\"\\s+\", \" \", t)                   # collapse whitespace\n    return t\n\ndef register_of(t):\n    \"\"\"Surface-signature register label for a piece of text.\"\"\"\n    if \"@-@\" in t or \"@,@\" in t or \"@.@\" in t:\n        return \"wiki\"\n    if (\"<p>\" in t or \"<code>\" in t or \"</\" in t or \"def \" in t or \"import \" in t\n            or (\"{\" in t and \"}\" in t and \";\" in t)):\n        return \"code_qa\"\n    if _NEWS.search(t.lower()):\n        return \"news\"\n    return \"web\"\n\ndef bigrams(a):\n    return ((a[:-1] * np.int64(1000003) + a[1:]) & np.int64(DB - 1))\n\nt0 = time.time()\ntok = AutoTokenizer.from_pretrained(\"gpt2\")\n\n# ------------------------------------------------------------------ load pool\nids, texts = [], []\nwith open(POOL) as f:\n    for line in f:\n        r = json.loads(line)\n        ids.append(r[\"id\"]); texts.append(r[\"text\"])\nids = np.asarray(ids, dtype=np.int64)\nndoc = len(ids)\nprint(f\"[{time.time()-t0:.0f}s] loaded pool: {ndoc} docs\", flush=True)\n\n# ------------------------------------------------------ tokenize pool (batched)\ndocs = []\nfor i in range(0, ndoc, 4000):\n    enc = tok(texts[i:i+4000], add_special_tokens=False).input_ids\n    docs.extend(np.asarray(x, dtype=np.int32) for x in enc)\nlens = np.fromiter((len(d) for d in docs), dtype=np.int64, count=ndoc)\nalltok = np.concatenate([d for d in docs]).astype(np.int64)\nprint(f\"[{time.time()-t0:.0f}s] tokenized pool: {lens.sum()/1e6:.1f}M tokens\", flush=True)\n\n# --------------------------------- pool background counts (shared denominator)\nneg_uni = np.bincount(alltok, minlength=V).astype(np.float64)\nneg_big = np.bincount(bigrams(alltok), minlength=DB).astype(np.float64)\nneg_uni_ln = np.log(neg_uni + ALPHA) - np.log(neg_uni.sum() + ALPHA * V)\nneg_big_ln = np.log(neg_big + ALPHA) - np.log(neg_big.sum() + ALPHA * DB)\ndel alltok\nprint(f\"[{time.time()-t0:.0f}s] built pool background\", flush=True)\n\n# ---------------------------- per-register target counts from disclosed target\ndev = np.load(DEV).astype(np.int64)\nsplits = np.where(dev == EOS)[0]\nb = [0] + (splits + 1).tolist() + [len(dev)]\nuni_w = {r: None for r in REGISTERS}\nbig_w = {r: None for r in REGISTERS}\npos_uni = {r: np.zeros(V, np.float64) for r in REGISTERS}\npos_big = {r: np.zeros(DB, np.float64) for r in REGISTERS}\nfor k in range(len(b) - 1):\n    d = dev[b[k]:b[k+1]]\n    d = d[d != EOS]\n    if len(d) < 20:\n        continue\n    raw = tok.decode(d.tolist())\n    r = register_of(raw)\n    if r == \"wiki\":                              # de-format encyclopedic text\n        d = np.asarray(tok(normalize_target(raw), add_special_tokens=False).input_ids,\n                       dtype=np.int64)\n    pos_uni[r] += np.bincount(d, minlength=V)\n    if len(d) > 1:\n        pos_big[r] += np.bincount(bigrams(d), minlength=DB)\nfor r in REGISTERS:\n    uni_w[r] = (np.log(pos_uni[r] + ALPHA) - np.log(pos_uni[r].sum() + ALPHA * V)) - neg_uni_ln\n    big_w[r] = (np.log(pos_big[r] + ALPHA) - np.log(pos_big[r].sum() + ALPHA * DB)) - neg_big_ln\nprint(f\"[{time.time()-t0:.0f}s] built {len(REGISTERS)} register targets \"\n      f\"(tok: {{{', '.join(f'{r}:{int(pos_uni[r].sum())}' for r in REGISTERS)}}})\", flush=True)\n\n# ------------------------ presence-based scoring of every doc vs every register\nNR = len(REGISTERS)\nscoremat = np.full((ndoc, NR), -1e30, dtype=np.float64)\ngate_ok = np.zeros(ndoc, dtype=bool)\nfor i in range(ndoc):\n    d = docs[i]; n = d.shape[0]\n    if n < MIN_TOK:\n        continue\n    t = texts[i]\n    if len(t) == 0 or len(_NONALPHA.sub(\"\", t)) / len(t) < MIN_ALPHA:\n        continue\n    di = d.astype(np.int64)\n    u = np.unique(di)\n    ub = np.unique(bigrams(di)) if n > 1 else np.array([0], dtype=np.int64)\n    for ri, r in enumerate(REGISTERS):\n        scoremat[i, ri] = uni_w[r][u].mean() + LAMBDA * big_w[r][ub].mean()\n    gate_ok[i] = True\n    if (i & 0x3FFF) == 0:\n        print(f\"[{time.time()-t0:.0f}s] scoring {i}/{ndoc}\", flush=True)\nprint(f\"[{time.time()-t0:.0f}s] scored docs ({gate_ok.sum()} passed gates)\", flush=True)\n\n# ------------- independent per-register ranked lists, interleaved by equal quota\nok = np.where(gate_ok)[0]\nranked = {r: ok[np.argsort(-scoremat[ok, ri])] for ri, r in enumerate(REGISTERS)}\n\nptr = {r: 0 for r in REGISTERS}\ncum = {r: 0 for r in REGISTERS}\nselection, seen, chosen, tot = [], set(), set(), 0\nwhile tot < COVER_TOKENS:\n    cand = [r for r in REGISTERS if ptr[r] < len(ranked[r])]\n    if not cand:\n        break\n    r = min(cand, key=lambda r: cum[r] / QUOTA[r])\n    # advance past docs already claimed by another register\n    while ptr[r] < len(ranked[r]) and ranked[r][ptr[r]] in chosen:\n        ptr[r] += 1\n    if ptr[r] >= len(ranked[r]):\n        continue\n    j = int(ranked[r][ptr[r]]); ptr[r] += 1\n    chosen.add(j)\n    h = hash(texts[j])\n    if h in seen:                                 # exact-duplicate guard\n        continue\n    seen.add(h)\n    selection.append(int(ids[j]))\n    cum[r] += int(lens[j]); tot += int(lens[j])\n\njson.dump(selection, open(OUT, \"w\"))\nmix = {r: round(cum[r] / max(1, tot), 3) for r in REGISTERS}\nprint(f\"[{time.time()-t0:.0f}s] wrote {len(selection)} ids covering ~{tot/1e6:.1f}M tokens\", flush=True)\nprint(f\"register token mix of emitted list: {mix}\", flush=True)\n", "structuredPatch": [{"oldStart": 114, "oldLines": 6, "newStart": 114, "newLines": 7, "lines": [" big_w = {r: None for r in REGISTERS}", " pos_uni = {r: np.zeros(V, np.float64) for r in REGISTERS}", " pos_big = {r: np.zeros(DB, np.float64) for r in REGISTERS}", "+raw_tok = {r: 0 for r in REGISTERS}             # RAW token share per register (for quota)", " for k in range(len(b) - 1):", "     d = dev[b[k]:b[k+1]]", "     d = d[d != EOS]"]}, {"oldStart": 121, "oldLines": 6, "newStart": 122, "newLines": 7, "lines": ["         continue", "     raw = tok.decode(d.tolist())", "     r = register_of(raw)", "+    raw_tok[r] += len(d)", "     if r == \"wiki\":                              # de-format encyclopedic text", "         d = np.asarray(tok(normalize_target(raw), add_special_tokens=False).input_ids,", "                        dtype=np.int64)"]}], "userModified": false, "replaceAll": false}

The file /workspace/submission/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/submission/curate.py", "oldString": "for r in REGISTERS:\n    uni_w[r] = (np.log(pos_uni[r] + ALPHA) - np.log(pos_uni[r].sum() + ALPHA * V)) - neg_uni_ln\n    big_w[r] = (np.log(pos_big[r] + ALPHA) - np.log(pos_big[r].sum() + ALPHA * DB)) - neg_big_ln\nprint(f\"[{time.time()-t0:.0f}s] built {len(REGISTERS)} register targets \"", "newString": "for r in REGISTERS:\n    uni_w[r] = (np.log(pos_uni[r] + ALPHA) - np.log(pos_uni[r].sum() + ALPHA * V)) - neg_uni_ln\n    big_w[r] = (np.log(pos_big[r] + ALPHA) - np.log(pos_big[r].sum() + ALPHA * DB)) - neg_big_ln\n_rawtot = sum(raw_tok.values())\nQUOTA = {r: raw_tok[r] / _rawtot for r in REGISTERS}     # match target token shares\nprint(f\"[{time.time()-t0:.0f}s] target token-share quota: \"\n      f\"{{{', '.join(f'{r}:{QUOTA[r]:.3f}' for r in REGISTERS)}}}\", flush=True)\nprint(f\"[{time.time()-t0:.0f}s] built {len(REGISTERS)} register targets \"", "originalFile": "\"\"\"Curate a pretraining subset from a raw web pool by register-balanced DSIR.\n\nThe disclosed evaluation target is a BROAD, multi-domain HQ English mix in equal\nparts: encyclopedic (Wikipedia), general web prose, news, and technical Q&A.\nHeld-out perplexity averages over all four registers, so a training set that\nover-weights one register (e.g. news, which is easiest to match) leaves the\nothers under-trained and drags the average up. We therefore select in two steps:\n\n  (1) DSIR importance scoring, PER REGISTER.  We split the disclosed target\n      (data/multi_dev.npy, GPT-2 tokens) into four register buckets by surface\n      signatures, and for each register r estimate a bag-of-n-grams importance\n      weight\n          w_r(gram) = log p_target_r(gram) - log p_pool(gram)\n      (unigrams + hashed bigrams; p_pool from the whole pool as background).\n      Each pool document is scored against every register using a PRESENCE-based\n      mean over its unique n-gram types (repetition-robust), behind a length\n      floor and a letter-fraction gate that reject fragments and non-prose\n      (directory listings, tables).  A document is routed to its best-fit\n      register (argmax score).\n\n  (2) Balanced quota fill.  Within each register we rank routed documents by\n      score, then interleave the four ranked lists by EQUAL token quota, so the\n      first 12M tokens the trainer consumes are ~25% from each register --\n      matching the equal-parts evaluation mixture.\n\nThis is Data Selection via Importance Resampling (Xie et al. 2023) with a\nper-domain target, plus classic quality gating.  No labels, no internet: the only\nsupervision is the disclosed target token stream.\n\"\"\"\nimport json, time, re, numpy as np\nfrom transformers import AutoTokenizer\n\nPOOL = \"/workspace/data/pool.jsonl\"\nDEV  = \"/workspace/data/multi_dev.npy\"\nOUT  = \"/workspace/submission/selection.json\"\n\nEOS   = 50256\nV     = 50257            # GPT-2 vocab\nDB    = 1 << 20          # hashed bigram buckets\nALPHA = 0.1             # additive smoothing\nMIN_TOK   = 100         # length floor: drop fragments\nMIN_ALPHA = 0.55        # letter-fraction floor: reject non-prose\nLAMBDA    = 0.5         # weight of bigram term relative to unigram term\nCOVER_TOKENS = 30_000_000   # emit ids covering ~2.5x the 12M budget\nREGISTERS = [\"wiki\", \"web\", \"news\", \"code_qa\"]\n# QUOTA (target training mixture) is set below to the target's own per-register\n# TOKEN proportions: held-out PPL is window-weighted over the concatenated target,\n# so the eval effectively weights each register by its token share -- matching that\n# share in training minimizes the token-weighted average loss.\n\n_NONALPHA = re.compile(r\"[^A-Za-z]\")\n_NEWS = re.compile(r\"\\b(reuters|said|told|according to|\"\n                   r\"on (monday|tuesday|wednesday|thursday|friday|saturday|sunday))\\b\")\n\ndef normalize_target(t):\n    \"\"\"Strip WikiText-103 surface formatting so the wiki target matches on CONTENT\n    vocabulary, not on markup absent from the raw pool (which would otherwise make\n    every pool doc look un-wiki-like and starve the encyclopedic quota).\"\"\"\n    t = t.replace(\"@-@\", \"-\").replace(\"@,@\", \",\").replace(\"@.@\", \".\")\n    t = re.sub(r\"\\s+([,.;:!?)])\", r\"\\1\", t)     # drop space before punctuation\n    t = re.sub(r\"([(\\[])\\s+\", r\"\\1\", t)          # drop space after open bracket\n    t = re.sub(r\"\\s+\", \" \", t)                   # collapse whitespace\n    return t\n\ndef register_of(t):\n    \"\"\"Surface-signature register label for a piece of text.\"\"\"\n    if \"@-@\" in t or \"@,@\" in t or \"@.@\" in t:\n        return \"wiki\"\n    if (\"<p>\" in t or \"<code>\" in t or \"</\" in t or \"def \" in t or \"import \" in t\n            or (\"{\" in t and \"}\" in t and \";\" in t)):\n        return \"code_qa\"\n    if _NEWS.search(t.lower()):\n        return \"news\"\n    return \"web\"\n\ndef bigrams(a):\n    return ((a[:-1] * np.int64(1000003) + a[1:]) & np.int64(DB - 1))\n\nt0 = time.time()\ntok = AutoTokenizer.from_pretrained(\"gpt2\")\n\n# ------------------------------------------------------------------ load pool\nids, texts = [], []\nwith open(POOL) as f:\n    for line in f:\n        r = json.loads(line)\n        ids.append(r[\"id\"]); texts.append(r[\"text\"])\nids = np.asarray(ids, dtype=np.int64)\nndoc = len(ids)\nprint(f\"[{time.time()-t0:.0f}s] loaded pool: {ndoc} docs\", flush=True)\n\n# ------------------------------------------------------ tokenize pool (batched)\ndocs = []\nfor i in range(0, ndoc, 4000):\n    enc = tok(texts[i:i+4000], add_special_tokens=False).input_ids\n    docs.extend(np.asarray(x, dtype=np.int32) for x in enc)\nlens = np.fromiter((len(d) for d in docs), dtype=np.int64, count=ndoc)\nalltok = np.concatenate([d for d in docs]).astype(np.int64)\nprint(f\"[{time.time()-t0:.0f}s] tokenized pool: {lens.sum()/1e6:.1f}M tokens\", flush=True)\n\n# --------------------------------- pool background counts (shared denominator)\nneg_uni = np.bincount(alltok, minlength=V).astype(np.float64)\nneg_big = np.bincount(bigrams(alltok), minlength=DB).astype(np.float64)\nneg_uni_ln = np.log(neg_uni + ALPHA) - np.log(neg_uni.sum() + ALPHA * V)\nneg_big_ln = np.log(neg_big + ALPHA) - np.log(neg_big.sum() + ALPHA * DB)\ndel alltok\nprint(f\"[{time.time()-t0:.0f}s] built pool background\", flush=True)\n\n# ---------------------------- per-register target counts from disclosed target\ndev = np.load(DEV).astype(np.int64)\nsplits = np.where(dev == EOS)[0]\nb = [0] + (splits + 1).tolist() + [len(dev)]\nuni_w = {r: None for r in REGISTERS}\nbig_w = {r: None for r in REGISTERS}\npos_uni = {r: np.zeros(V, np.float64) for r in REGISTERS}\npos_big = {r: np.zeros(DB, np.float64) for r in REGISTERS}\nraw_tok = {r: 0 for r in REGISTERS}             # RAW token share per register (for quota)\nfor k in range(len(b) - 1):\n    d = dev[b[k]:b[k+1]]\n    d = d[d != EOS]\n    if len(d) < 20:\n        continue\n    raw = tok.decode(d.tolist())\n    r = register_of(raw)\n    raw_tok[r] += len(d)\n    if r == \"wiki\":                              # de-format encyclopedic text\n        d = np.asarray(tok(normalize_target(raw), add_special_tokens=False).input_ids,\n                       dtype=np.int64)\n    pos_uni[r] += np.bincount(d, minlength=V)\n    if len(d) > 1:\n        pos_big[r] += np.bincount(bigrams(d), minlength=DB)\nfor r in REGISTERS:\n    uni_w[r] = (np.log(pos_uni[r] + ALPHA) - np.log(pos_uni[r].sum() + ALPHA * V)) - neg_uni_ln\n    big_w[r] = (np.log(pos_big[r] + ALPHA) - np.log(pos_big[r].sum() + ALPHA * DB)) - neg_big_ln\nprint(f\"[{time.time()-t0:.0f}s] built {len(REGISTERS)} register targets \"\n      f\"(tok: {{{', '.join(f'{r}:{int(pos_uni[r].sum())}' for r in REGISTERS)}}})\", flush=True)\n\n# ------------------------ presence-based scoring of every doc vs every register\nNR = len(REGISTERS)\nscoremat = np.full((ndoc, NR), -1e30, dtype=np.float64)\ngate_ok = np.zeros(ndoc, dtype=bool)\nfor i in range(ndoc):\n    d = docs[i]; n = d.shape[0]\n    if n < MIN_TOK:\n        continue\n    t = texts[i]\n    if len(t) == 0 or len(_NONALPHA.sub(\"\", t)) / len(t) < MIN_ALPHA:\n        continue\n    di = d.astype(np.int64)\n    u = np.unique(di)\n    ub = np.unique(bigrams(di)) if n > 1 else np.array([0], dtype=np.int64)\n    for ri, r in enumerate(REGISTERS):\n        scoremat[i, ri] = uni_w[r][u].mean() + LAMBDA * big_w[r][ub].mean()\n    gate_ok[i] = True\n    if (i & 0x3FFF) == 0:\n        print(f\"[{time.time()-t0:.0f}s] scoring {i}/{ndoc}\", flush=True)\nprint(f\"[{time.time()-t0:.0f}s] scored docs ({gate_ok.sum()} passed gates)\", flush=True)\n\n# ------------- independent per-register ranked lists, interleaved by equal quota\nok = np.where(gate_ok)[0]\nranked = {r: ok[np.argsort(-scoremat[ok, ri])] for ri, r in enumerate(REGISTERS)}\n\nptr = {r: 0 for r in REGISTERS}\ncum = {r: 0 for r in REGISTERS}\nselection, seen, chosen, tot = [], set(), set(), 0\nwhile tot < COVER_TOKENS:\n    cand = [r for r in REGISTERS if ptr[r] < len(ranked[r])]\n    if not cand:\n        break\n    r = min(cand, key=lambda r: cum[r] / QUOTA[r])\n    # advance past docs already claimed by another register\n    while ptr[r] < len(ranked[r]) and ranked[r][ptr[r]] in chosen:\n        ptr[r] += 1\n    if ptr[r] >= len(ranked[r]):\n        continue\n    j = int(ranked[r][ptr[r]]); ptr[r] += 1\n    chosen.add(j)\n    h = hash(texts[j])\n    if h in seen:                                 # exact-duplicate guard\n        continue\n    seen.add(h)\n    selection.append(int(ids[j]))\n    cum[r] += int(lens[j]); tot += int(lens[j])\n\njson.dump(selection, open(OUT, \"w\"))\nmix = {r: round(cum[r] / max(1, tot), 3) for r in REGISTERS}\nprint(f\"[{time.time()-t0:.0f}s] wrote {len(selection)} ids covering ~{tot/1e6:.1f}M tokens\", flush=True)\nprint(f\"register token mix of emitted list: {mix}\", flush=True)\n", "structuredPatch": [{"oldStart": 132, "oldLines": 6, "newStart": 132, "newLines": 10, "lines": [" for r in REGISTERS:", "     uni_w[r] = (np.log(pos_uni[r] + ALPHA) - np.log(pos_uni[r].sum() + ALPHA * V)) - neg_uni_ln", "     big_w[r] = (np.log(pos_big[r] + ALPHA) - np.log(pos_big[r].sum() + ALPHA * DB)) - neg_big_ln", "+_rawtot = sum(raw_tok.values())", "+QUOTA = {r: raw_tok[r] / _rawtot for r in REGISTERS}     # match target token shares", "+print(f\"[{time.time()-t0:.0f}s] target token-share quota: \"", "+      f\"{{{', '.join(f'{r}:{QUOTA[r]:.3f}' for r in REGISTERS)}}}\", flush=True)", " print(f\"[{time.time()-t0:.0f}s] built {len(REGISTERS)} register targets \"", "       f\"(tok: {{{', '.join(f'{r}:{int(pos_uni[r].sum())}' for r in REGISTERS)}}})\", flush=True)", " "]}], "userModified": false, "replaceAll": false}

Command did not complete within its 120s timeout and was moved to the background (ID: bhvd2z9l2). Output is being written to: /tmp/claude-0/-workspace/4bb4d0fa-198e-4747-8902-b67961f6b67a/tasks/bhvd2z9l2.output. You will be notified when it completes. To check interim output, use Read on that file path.
Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "bhvd2z9l2", "timedOutAfterMs": 120000, "backgroundCwdHint": "Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands."}

[162s] scoring 180224/182016
[163s] scored docs (171150 passed gates)
[163s] wrote 36419 ids covering ~30.0M tokens
register token mix of emitted list: {'wiki': 0.164, 'web': 0.207, 'news': 0.355, 'code_qa': 0.274}

real	2m48.232s
user	17m52.540s
sys	0m3.110s

[stdout]
[162s] scoring 180224/182016
[163s] scored docs (171150 passed gates)
[163s] wrote 36419 ids covering ~30.0M tokens
register token mix of emitted list: {'wiki': 0.164, 'web': 0.207, 'news': 0.355, 'code_qa': 0.274}

real	2m48.232s
user	17m52.540s
sys	0m3.110s
[metadata] {"noOutputExpected": false}

2786
=== dev_eval on TOKEN-PROPORTION balanced selection ===
packed 12000000 tokens from 36419 ids -> /tmp/dev_train.npy
HELDOUT_PPL 330.567  loss 5.8008  train_s 56  /tmp/dev_train.npy
(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)

real	1m41.449s
user	1m35.730s
sys	0m2.980s

[stdout]
2786
=== dev_eval on TOKEN-PROPORTION balanced selection ===
packed 12000000 tokens from 36419 ids -> /tmp/dev_train.npy
HELDOUT_PPL 330.567  loss 5.8008  train_s 56  /tmp/dev_train.npy
(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)

real	1m41.449s
user	1m35.730s
sys	0m2.980s
[metadata] {"noOutputExpected": false}

Updated task #2 status

[metadata] {"success": true, "taskId": "2", "updatedFields": ["status"], "statusChange": {"from": "in_progress", "to": "completed"}}

count 36419
all ints: True
unique: True
in-range 0..182015: True
first 8 ids: [50793, 44450, 105412, 108566, 6138, 67417, 37064, 75295]
--- packer already reported: packed 12000000 tokens from 36419 ids (budget filled) ---

[stdout]
count 36419
all ints: True
unique: True
in-range 0..182015: True
first 8 ids: [50793, 44450, 105412, 108566, 6138, 67417, 37064, 75295]
--- packer already reported: packed 12000000 tokens from 36419 ids (budget filled) ---
[metadata] {"noOutputExpected": false}

2586
wiki 164456 tokens
web 207370 tokens
news 354588 tokens
code_qa 273586 tokens

[stdout]
2586
wiki 164456 tokens
web 207370 tokens
news 354588 tokens
code_qa 273586 tokens
[metadata] {"noOutputExpected": false}

Command did not complete within its 120s timeout and was moved to the background (ID: bvjbf1g3l). Output is being written to: /tmp/claude-0/-workspace/4bb4d0fa-198e-4747-8902-b67961f6b67a/tasks/bvjbf1g3l.output. You will be notified when it completes. To check interim output, use Read on that file path.
Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "bvjbf1g3l", "timedOutAfterMs": 120000, "backgroundCwdHint": "Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands."}

REG=code_qa  SEL=unbal  HELDOUT_PPL 603.176

[stdout]
REG=code_qa  SEL=unbal  HELDOUT_PPL 603.176
[metadata] {"noOutputExpected": false}

REG=code_qa  SEL=unbal  HELDOUT_PPL 603.176
REG=code_qa  SEL=bal  HELDOUT_PPL 195.310
REG=news  SEL=unbal  HELDOUT_PPL 235.175
REG=news  SEL=bal  HELDOUT_PPL 262.053
--- time ---
2185

[stdout]
REG=code_qa  SEL=unbal  HELDOUT_PPL 603.176
REG=code_qa  SEL=bal  HELDOUT_PPL 195.310
REG=news  SEL=unbal  HELDOUT_PPL 235.175
REG=news  SEL=bal  HELDOUT_PPL 262.053
--- time ---
2185
[metadata] {"noOutputExpected": false}

REG=code_qa  SEL=unbal  HELDOUT_PPL 603.176
REG=code_qa  SEL=bal  HELDOUT_PPL 195.310
REG=news  SEL=unbal  HELDOUT_PPL 235.175
REG=news  SEL=bal  HELDOUT_PPL 262.053
REG=wiki  SEL=unbal  HELDOUT_PPL 867.222
REG=wiki  SEL=bal  HELDOUT_PPL 790.765
DONE_MECH

[stdout]
REG=code_qa  SEL=unbal  HELDOUT_PPL 603.176
REG=code_qa  SEL=bal  HELDOUT_PPL 195.310
REG=news  SEL=unbal  HELDOUT_PPL 235.175
REG=news  SEL=bal  HELDOUT_PPL 262.053
REG=wiki  SEL=unbal  HELDOUT_PPL 867.222
REG=wiki  SEL=bal  HELDOUT_PPL 790.765
DONE_MECH
[metadata] {"noOutputExpected": false}

Updated task #3 status

[metadata] {"success": true, "taskId": "3", "updatedFields": ["status"], "statusChange": {"from": "pending", "to": "in_progress"}}

File created successfully at: /workspace/claim.md (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/workspace/claim.md", "content": "# Claim: register-balance dominates per-document quality for broad-target curation\n\n## Hypothesis\nFor a **fixed, small token budget** (12M) trained from scratch and evaluated by\nheld-out perplexity on a **broad, equal-parts multi-domain target** (encyclopedic /\nweb prose / news / technical Q&A), the controlling variable is not per-document\n\"quality\" but the **register mixture** of the selected data. An unconstrained\nimportance/quality ranking collapses the budget onto the single register whose\nvocabulary is easiest to distinguish from generic web (news), starving the others\nand inflating the *average* perplexity. Explicitly matching the training mixture\nto the target's **token-weighted register composition** (register-balanced DSIR)\nbeats importance ranking alone by a wide margin.\n\nMeasured on the dev target (`data/multi_dev.npy`, frozen 30M GPT, 12M tokens):\n\n| selection | held-out PPL | selected-set composition |\n|---|---|---|\n| random (do-nothing baseline)     | 485.4 | pool average |\n| global importance ranking (DSIR) | 441.0 | **63% news, 2% technical Q&A**, ~0% encyclopedic |\n| **register-balanced DSIR (submitted)** | **330.6** | matched to target token shares (wiki .16 / web .21 / news .35 / code .27) |\n\nBalancing captures **~32% below random** and **~25% below** an already-nontrivial\nquality ranking, from the *same* importance signal — only the mixture changed.\n\n## Mechanism (prediction of an observable other than the final PPL)\nThe importance weight `log p_target(gram) − p_pool(gram)` is largest for the\nregister whose n-grams are most distinctive vs. generic web (news datelines, named\nentities, reporting verbs). A pure top-k ranking therefore over-selects news and\ndrives technical Q&A to ~2% and encyclopedic to ~0%. Because per-register held-out\nloss is convex in how well that register is covered, the *average* is dominated by\nthe **starved** registers, not by the well-covered one.\n\n**Predicted observable:** decompose held-out loss by register. Moving from the\nunbalanced to the balanced selection will (a) sharply lower loss on the\n**under-represented** registers, and (b) leave — or even slightly *worsen* — the\n**over-served** register (news), because news was already saturated and merely\nloses budget. Measured on per-register slices of `multi_dev`:\n\n| register slice | unbalanced PPL | balanced PPL | effect |\n|---|---|---|---|\n| technical Q&A / code | 603.2 | **195.3** | −68%  (was starved at 2% → 27%) |\n| encyclopedic (wiki)  | 867.2 | **790.8** | −9%   (was ~0% → 16%) |\n| news                 | 235.2 | 262.1 | **+11%** (over-served 63% → 35%) |\n\nThe improvement is entirely concentrated in the starved registers and the\nsaturated register regresses — exactly as the starvation mechanism predicts, and\nnot explainable by \"the balanced set is simply higher quality.\"\n\nA second, pre-training observable: the selected-set register histogram shifts from\n`{news .63, web .35, code .02, wiki ~0}` to `{news .35, web .21, code .27, wiki .16}`.\n\n(Residual note: even balanced, the encyclopedic slice stays high (~791). Its\nWikipedia text is in WikiText-103 surface format — spaces around punctuation,\n`@-@` — absent from the raw pool, so its *formatting* is largely unlearnable here;\nbalancing recovers the *content* vocabulary, not the markup.)\n\n## Falsification\nThe claim is refuted if any of the following hold:\n1. **Balancing gives no aggregate gain** — forcing the target token-share mixture\n   fails to beat the unconstrained importance ranking (441.0). *Observed: 330.6 < 441.0 — not refuted.*\n2. **The gain is uniform across registers** rather than concentrated in the\n   starved ones. *Observed: code −68% / wiki −9% vs news +11% — not refuted.*\n3. **News improves as much as code_qa under balancing**, i.e. the saturation\n   premise is false. *Observed: news got worse (+11%) — not refuted.*\n4. A finer sweep of the mixture around the target token shares should be flat near\n   the optimum; a large asymmetric gain from *unbalancing further* toward one\n   register would refute \"match-the-eval-mixture.\" *Observed: equal-parts 331.7 ≈\n   token-share 330.6 — flat, consistent.*\n\n## Transfer\n- **Curation against a known multi-domain eval:** when the downstream evaluation\n  mixture is disclosed, allocate the budget to match its *token-weighted* domain\n  composition instead of pure quality ranking. Directly applicable to\n  FineWeb/DataComp-LM-style pipelines whose eval is multi-register.\n- **Regime dependence:** the effect is largest in the *data-scarce, small-budget*\n  regime, where over-selecting one register genuinely starves others. Prediction:\n  as the budget grows relative to per-register supply, registers saturate and the\n  balanced−unbalanced gap shrinks (at 10× budget it should narrow markedly).\n- **Model/tokenizer-agnostic:** the mechanism is about mixture, not the specific\n  30M GPT. A larger student trained on the identical selection should preserve the\n  ordering random > importance-only > register-balanced, and the per-register\n  decomposition (starved registers move most) should reproduce.\n- **Signal-agnostic:** replacing the n-gram importance score with a neural quality\n  classifier would *not* remove the failure — any single \"target-likeness\" score\n  concentrates on the most-separable register, so explicit mixture control is still\n  required.\n", "structuredPatch": [], "originalFile": null, "userModified": false}

=== deliverables ===
-rw-r--r-- 1 root root   5381 Aug  4 04:18 claim.md
-rw-r--r-- 1 root root   8912 Aug  4 04:02 submission/curate.py
-rw-r--r-- 1 root root 256458 Aug  4 04:05 submission/selection.json

=== selection.json validity ===
ids: 36419  unique+in-range: OK
first 5 (priority order): [50793, 44450, 105412, 108566, 6138]

=== curate.py header ===
"""Curate a pretraining subset from a raw web pool by register-balanced DSIR.

The disclosed evaluation target is a BROAD, multi-domain HQ English mix in equal
parts: encyclopedic (Wikipedia), general web prose, news, and technical Q&A.
Held-out perplexity averages over all four registers, so a training set that
over-weights one register (e.g. news, which is easiest to match) leaves the
others under-trained and drags the average up. We therefore select in two steps:

  (1) DSIR importance scoring, PER REGISTER.  We split the disclosed target
      (data/multi_dev.npy, GPT-2 tokens) into four register buckets by surface
      signatures, and for each register r estimate a bag-of-n-grams importance
      weight
...
=== time left ===
2015

[stdout]
=== deliverables ===
-rw-r--r-- 1 root root   5381 Aug  4 04:18 claim.md
-rw-r--r-- 1 root root   8912 Aug  4 04:02 submission/curate.py
-rw-r--r-- 1 root root 256458 Aug  4 04:05 submission/selection.json

=== selection.json validity ===
ids: 36419  unique+in-range: OK
first 5 (priority order): [50793, 44450, 105412, 108566, 6138]

=== curate.py header ===
"""Curate a pretraining subset from a raw web pool by register-balanced DSIR.

The disclosed evaluation target is a BROAD, multi-domain HQ English mix in equal
parts: encyclopedic (Wikipedia), general web prose, news, and technical Q&A.
Held-out perplexity averages over all four registers, so a training set that
over-weights one register (e.g. news, which is easiest to match) leaves the
others under-trained and drags the average up. We therefore select in two steps:

  (1) DSIR importance scoring, PER REGISTER.  We split the disclosed target
      (data/multi_dev.npy, GPT-2 tokens) into four register buckets by surface
      signatures, and for each register r estimate a bag-of-n-grams importance
      weight
...
=== time left ===
2015
[metadata] {"noOutputExpected": false}

Updated task #3 status

[metadata] {"success": true, "taskId": "3", "updatedFields": ["status"], "statusChange": {"from": "in_progress", "to": "completed"}}