claude-opus-4-8

The file /workspace/submission/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/submission/curate.py", "oldString": "MIN_WORDS = 50            # too short to teach anything\nMIN_ASCII_RATIO = 0.90    # guard against binary / non-English gibberish\nMAX_TOP_WORD_FRAC = 0.28  # a single word dominating -> spam / repetition\n\nrandom.seed(SEED)\n# HTML tags, HTML entities, and alnum words are all tokens (keeps the Q&A/code register)\nTOK_RE = re.compile(r\"</?[a-z][a-z0-9]*/?>|&[a-z]+;|[a-z0-9]+\")\nQA_MARK = (\"<p>\", \"<code>\", \"<pre>\", \"</\", \"&quot;\", \"&lt;\", \"&gt;\", \"&amp;\",\n           \"def \", \"import \", \"function \", \"public static\", \"#include\")\n\n\ndef toks(text):\n    return TOK_RE.findall(text.lower())\n\n\ndef is_qa(text):\n    return any(m in text for m in QA_MARK)\n\n\ndef passes(text):\n    if len(text) < 200:\n        return False, None\n    w = toks(text)\n    if len(w) < MIN_WORDS:\n        return False, None\n    ascii_ok = sum(32 <= ord(c) < 127 or c in \"\\n\\t\\r\" for c in text)\n    if ascii_ok / len(text) < MIN_ASCII_RATIO:\n        return False, None\n    c = Counter(w)\n    if c.most_common(1)[0][1] / len(w) > MAX_TOP_WORD_FRAC:\n        return False, None\n    return True, w", "newString": "MIN_WORDS = 60            # too short to teach anything\nMIN_ASCII_RATIO = 0.90    # guard against binary / non-English gibberish\nMAX_TOP_WORD_FRAC = 0.28  # a single word dominating -> spam / repetition\nMIN_STOP_RATIO = 0.12     # real prose is stopword-rich; menus/lists/code-dumps aren't\n\nrandom.seed(SEED)\n# HTML tags, HTML entities, and alnum words are all tokens (keeps the Q&A/code register)\nTOK_RE = re.compile(r\"</?[a-z][a-z0-9]*/?>|&[a-z]+;|[a-z0-9]+\")\nSTOP = set(\"the of and to in a is that for it as was on are with be by this \"\n           \"an at or from his he she they you not have has had but which we \"\n           \"were their its been would their about into more can\".split())\n\n\ndef toks(text):\n    return TOK_RE.findall(text.lower())\n\n\ndef passes(text):\n    if len(text) < 200:\n        return False, None\n    w = toks(text)\n    if len(w) < MIN_WORDS:\n        return False, None\n    ascii_ok = sum(32 <= ord(c) < 127 or c in \"\\n\\t\\r\" for c in text)\n    if ascii_ok / len(text) < MIN_ASCII_RATIO:\n        return False, None\n    c = Counter(w)\n    if c.most_common(1)[0][1] / len(w) > MAX_TOP_WORD_FRAC:\n        return False, None\n    stop = sum(c[s] for s in STOP)\n    if stop / len(w) < MIN_STOP_RATIO:   # navigation menus / list junk / raw code\n        return False, None\n    return True, w", "originalFile": "#!/usr/bin/env python3\n\"\"\"Curate the raw web pool for training a small LM whose held-out target is a\nBROAD, multi-domain high-quality English distribution (Wikipedia + high-quality\nweb prose + news + technical Q&A), disclosed via /workspace/data/multi_dev.npy.\n\nCriterion (stated, reproducible): DSIR-style importance scoring with\nregister-aware coverage.\n\n  1. Decode the disclosed target token array back to text -> a sample of the\n     target domain.\n  2. Fit a unigram+bigram bag-of-features distribution for the TARGET and for a\n     random BACKGROUND sample of the raw pool.  Features are HTML/entity aware\n     (`<code>`, `<p>`, `&quot;` are tokens) so the technical-Q&A register is\n     represented, not stripped.\n  3. Score every pool document by its mean per-token log-likelihood ratio\n     log p_target(f) - log p_background(f).  Clean multi-domain prose scores\n     high; boilerplate / forum / spam scores low.\n  4. Light hard filters drop only degenerate docs (too short, gibberish/binary,\n     single word repeated).  Exact-duplicate texts are removed.\n  5. The target is ~1/4 technical Q&A, but such HTML/code docs are extremely\n     rare in this raw pool.  To avoid starving that register we surface EVERY\n     genuine Q&A/code document (ranked by score) ahead of the prose tail, then\n     fill the remaining budget with the highest-scoring prose.\n\nEmits pool ids best-first; the trainer consumes them until the 12M-token budget\nis full.  This is a pure ranking function over a stated feature, not a\nhand-picked list.\n\"\"\"\nimport json, re, time, random\nimport numpy as np\nfrom collections import Counter\n\nPOOL = \"/workspace/data/pool.jsonl\"\nTARGET_NPY = \"/workspace/data/multi_dev.npy\"\nOUT = \"/workspace/submission/selection.json\"\n\nN_UNI = 60_000\nN_BI = 60_000\nBG_SAMPLE_FRAC = 0.40\nALPHA = 1.0\nN_OUT = 40_000\nSEED = 0\n\nMIN_WORDS = 50            # too short to teach anything\nMIN_ASCII_RATIO = 0.90    # guard against binary / non-English gibberish\nMAX_TOP_WORD_FRAC = 0.28  # a single word dominating -> spam / repetition\n\nrandom.seed(SEED)\n# HTML tags, HTML entities, and alnum words are all tokens (keeps the Q&A/code register)\nTOK_RE = re.compile(r\"</?[a-z][a-z0-9]*/?>|&[a-z]+;|[a-z0-9]+\")\nQA_MARK = (\"<p>\", \"<code>\", \"<pre>\", \"</\", \"&quot;\", \"&lt;\", \"&gt;\", \"&amp;\",\n           \"def \", \"import \", \"function \", \"public static\", \"#include\")\n\n\ndef toks(text):\n    return TOK_RE.findall(text.lower())\n\n\ndef is_qa(text):\n    return any(m in text for m in QA_MARK)\n\n\ndef passes(text):\n    if len(text) < 200:\n        return False, None\n    w = toks(text)\n    if len(w) < MIN_WORDS:\n        return False, None\n    ascii_ok = sum(32 <= ord(c) < 127 or c in \"\\n\\t\\r\" for c in text)\n    if ascii_ok / len(text) < MIN_ASCII_RATIO:\n        return False, None\n    c = Counter(w)\n    if c.most_common(1)[0][1] / len(w) > MAX_TOP_WORD_FRAC:\n        return False, None\n    return True, w\n\n\ndef main():\n    t0 = time.time()\n    print(\"loading pool...\", flush=True)\n    ids, texts = [], []\n    with open(POOL) as f:\n        for line in f:\n            r = json.loads(line)\n            ids.append(r[\"id\"]); texts.append(r[\"text\"])\n    N = len(ids)\n    print(f\"  {N} docs loaded in {time.time()-t0:.0f}s\", flush=True)\n\n    # ---- target distribution ----\n    from transformers import AutoTokenizer\n    tok = AutoTokenizer.from_pretrained(\"gpt2\")\n    tgt_tokens = np.load(TARGET_NPY)\n    parts = []\n    for s in range(0, len(tgt_tokens), 4096):\n        parts.append(tok.decode(tgt_tokens[s:s+4096].tolist()))\n    tw = toks(\" \".join(parts))\n    tgt_uni = Counter(tw)\n    tgt_bi = Counter(a + \" \" + b for a, b in zip(tw, tw[1:]))\n    print(f\"  target: {len(tw)} tokens, {len(tgt_uni)} uni, {len(tgt_bi)} bi \"\n          f\"({time.time()-t0:.0f}s)\", flush=True)\n\n    vocab_uni = {w: i for i, (w, _) in enumerate(\n        [x for x in tgt_uni.most_common(N_UNI) if x[1] >= 2])}\n    vocab_bi = {w: i for i, (w, _) in enumerate(\n        [x for x in tgt_bi.most_common(N_BI) if x[1] >= 2])}\n    VU, VB = len(vocab_uni), len(vocab_bi)\n    print(f\"  vocab kept: {VU} uni, {VB} bi\", flush=True)\n\n    tgt_uc = np.zeros(VU); tgt_bc = np.zeros(VB)\n    for w, i in vocab_uni.items():\n        tgt_uc[i] = tgt_uni[w]\n    for w, i in vocab_bi.items():\n        tgt_bc[i] = tgt_bi[w]\n\n    # ---- background distribution (random pool sample) ----\n    bg_uc = np.zeros(VU); bg_bc = np.zeros(VB)\n    bg_idx = [j for j in range(N) if random.random() < BG_SAMPLE_FRAC]\n    for j in bg_idx:\n        w = toks(texts[j])\n        for x in w:\n            i = vocab_uni.get(x)\n            if i is not None:\n                bg_uc[i] += 1\n        for a, b in zip(w, w[1:]):\n            i = vocab_bi.get(a + \" \" + b)\n            if i is not None:\n                bg_bc[i] += 1\n    print(f\"  background from {len(bg_idx)} docs ({time.time()-t0:.0f}s)\", flush=True)\n\n    Tu, Bu = tgt_uc.sum(), bg_uc.sum()\n    Tb, Bb = tgt_bc.sum(), bg_bc.sum()\n    wu = np.log((tgt_uc + ALPHA) / (Tu + ALPHA * VU)) - \\\n         np.log((bg_uc + ALPHA) / (Bu + ALPHA * VU))\n    wb = np.log((tgt_bc + ALPHA) / (Tb + ALPHA * VB)) - \\\n         np.log((bg_bc + ALPHA) / (Bb + ALPHA * VB))\n\n    # ---- score every doc; track register + dedup ----\n    scores = np.full(N, -1e9)\n    qa_flag = np.zeros(N, dtype=bool)\n    seen = set()\n    kept = 0\n    for j in range(N):\n        h = hash(texts[j])\n        if h in seen:\n            continue          # exact-duplicate text\n        ok, w = passes(texts[j])\n        if not ok:\n            continue\n        seen.add(h)\n        kept += 1\n        qa_flag[j] = is_qa(texts[j])\n        su = 0.0\n        for x in w:\n            i = vocab_uni.get(x)\n            if i is not None:\n                su += wu[i]\n        sb = 0.0\n        for a, b in zip(w, w[1:]):\n            i = vocab_bi.get(a + \" \" + b)\n            if i is not None:\n                sb += wb[i]\n        scores[j] = (su + 0.5 * sb) / len(w)\n    print(f\"  scored; {kept} unique docs pass ({time.time()-t0:.0f}s)\", flush=True)\n\n    valid = scores > -1e8\n    order = np.argsort(-scores)\n    order = [j for j in order if valid[j]]\n    # surface all genuine Q&A/code docs (best-first) ahead of the prose tail\n    qa_order = [j for j in order if qa_flag[j]]\n    prose_order = [j for j in order if not qa_flag[j]]\n    final = qa_order + prose_order\n    sel = [int(ids[j]) for j in final[:N_OUT]]\n    json.dump(sel, open(OUT, \"w\"))\n    print(f\"  qa docs surfaced: {len(qa_order)}\", flush=True)\n    print(f\"wrote {len(sel)} ids -> {OUT} ({time.time()-t0:.0f}s)\", flush=True)\n    sv = scores[valid]\n    print(\"score quantiles:\", np.round(np.quantile(sv, [0, .5, .9, 1]), 3))\n\n\nif __name__ == \"__main__\":\n    main()\n", "structuredPatch": [{"oldStart": 41, "oldLines": 25, "newStart": 41, "newLines": 23, "lines": [" N_OUT = 40_000", " SEED = 0", " ", "-MIN_WORDS = 50            # too short to teach anything", "+MIN_WORDS = 60            # too short to teach anything", " MIN_ASCII_RATIO = 0.90    # guard against binary / non-English gibberish", " MAX_TOP_WORD_FRAC = 0.28  # a single word dominating -> spam / repetition", "+MIN_STOP_RATIO = 0.12     # real prose is stopword-rich; menus/lists/code-dumps aren't", " ", " random.seed(SEED)", " # HTML tags, HTML entities, and alnum words are all tokens (keeps the Q&A/code register)", " TOK_RE = re.compile(r\"</?[a-z][a-z0-9]*/?>|&[a-z]+;|[a-z0-9]+\")", "-QA_MARK = (\"<p>\", \"<code>\", \"<pre>\", \"</\", \"&quot;\", \"&lt;\", \"&gt;\", \"&amp;\",", "-           \"def \", \"import \", \"function \", \"public static\", \"#include\")", "+STOP = set(\"the of and to in a is that for it as was on are with be by this \"", "+           \"an at or from his he she they you not have has had but which we \"", "+           \"were their its been would their about into more can\".split())", " ", " ", " def toks(text):", "     return TOK_RE.findall(text.lower())", " ", " ", "-def is_qa(text):", "-    return any(m in text for m in QA_MARK)", "-", "-", " def passes(text):", "     if len(text) < 200:", "         return False, None"]}, {"oldStart": 72, "oldLines": 6, "newStart": 70, "newLines": 9, "lines": ["     c = Counter(w)", "     if c.most_common(1)[0][1] / len(w) > MAX_TOP_WORD_FRAC:", "         return False, None", "+    stop = sum(c[s] for s in STOP)", "+    if stop / len(w) < MIN_STOP_RATIO:   # navigation menus / list junk / raw code", "+        return False, None", "     return True, w", " ", " "]}], "userModified": false, "replaceAll": false}

The file /workspace/submission/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/submission/curate.py", "oldString": "    # ---- score every doc; track register + dedup ----\n    scores = np.full(N, -1e9)\n    qa_flag = np.zeros(N, dtype=bool)\n    seen = set()\n    kept = 0\n    for j in range(N):\n        h = hash(texts[j])\n        if h in seen:\n            continue          # exact-duplicate text\n        ok, w = passes(texts[j])\n        if not ok:\n            continue\n        seen.add(h)\n        kept += 1\n        qa_flag[j] = is_qa(texts[j])\n        su = 0.0\n        for x in w:\n            i = vocab_uni.get(x)\n            if i is not None:\n                su += wu[i]\n        sb = 0.0\n        for a, b in zip(w, w[1:]):\n            i = vocab_bi.get(a + \" \" + b)\n            if i is not None:\n                sb += wb[i]\n        scores[j] = (su + 0.5 * sb) / len(w)\n    print(f\"  scored; {kept} unique docs pass ({time.time()-t0:.0f}s)\", flush=True)\n\n    valid = scores > -1e8\n    order = np.argsort(-scores)\n    order = [j for j in order if valid[j]]\n    # surface all genuine Q&A/code docs (best-first) ahead of the prose tail\n    qa_order = [j for j in order if qa_flag[j]]\n    prose_order = [j for j in order if not qa_flag[j]]\n    final = qa_order + prose_order\n    sel = [int(ids[j]) for j in final[:N_OUT]]\n    json.dump(sel, open(OUT, \"w\"))\n    print(f\"  qa docs surfaced: {len(qa_order)}\", flush=True)\n    print(f\"wrote {len(sel)} ids -> {OUT} ({time.time()-t0:.0f}s)\", flush=True)\n    sv = scores[valid]\n    print(\"score quantiles:\", np.round(np.quantile(sv, [0, .5, .9, 1]), 3))", "newString": "    # ---- score every doc; dedup exact-duplicate texts ----\n    scores = np.full(N, -1e9)\n    seen = set()\n    kept = 0\n    for j in range(N):\n        h = hash(texts[j])\n        if h in seen:\n            continue          # exact-duplicate text\n        ok, w = passes(texts[j])\n        if not ok:\n            continue\n        seen.add(h)\n        kept += 1\n        su = 0.0\n        for x in w:\n            i = vocab_uni.get(x)\n            if i is not None:\n                su += wu[i]\n        sb = 0.0\n        for a, b in zip(w, w[1:]):\n            i = vocab_bi.get(a + \" \" + b)\n            if i is not None:\n                sb += wb[i]\n        scores[j] = (su + 0.5 * sb) / len(w)\n    print(f\"  scored; {kept} unique docs pass ({time.time()-t0:.0f}s)\", flush=True)\n\n    # pure importance ranking: the LLR against the full 4-register target already\n    # reflects the target's register mix, so no forced per-register quotas.\n    valid = scores > -1e8\n    order = [j for j in np.argsort(-scores) if valid[j]]\n    sel = [int(ids[j]) for j in order[:N_OUT]]\n    json.dump(sel, open(OUT, \"w\"))\n    print(f\"wrote {len(sel)} ids -> {OUT} ({time.time()-t0:.0f}s)\", flush=True)\n    print(\"score quantiles:\", np.round(np.quantile(scores[valid], [0, .5, .9, 1]), 3))", "originalFile": "#!/usr/bin/env python3\n\"\"\"Curate the raw web pool for training a small LM whose held-out target is a\nBROAD, multi-domain high-quality English distribution (Wikipedia + high-quality\nweb prose + news + technical Q&A), disclosed via /workspace/data/multi_dev.npy.\n\nCriterion (stated, reproducible): DSIR-style importance scoring with\nregister-aware coverage.\n\n  1. Decode the disclosed target token array back to text -> a sample of the\n     target domain.\n  2. Fit a unigram+bigram bag-of-features distribution for the TARGET and for a\n     random BACKGROUND sample of the raw pool.  Features are HTML/entity aware\n     (`<code>`, `<p>`, `&quot;` are tokens) so the technical-Q&A register is\n     represented, not stripped.\n  3. Score every pool document by its mean per-token log-likelihood ratio\n     log p_target(f) - log p_background(f).  Clean multi-domain prose scores\n     high; boilerplate / forum / spam scores low.\n  4. Light hard filters drop only degenerate docs (too short, gibberish/binary,\n     single word repeated).  Exact-duplicate texts are removed.\n  5. The target is ~1/4 technical Q&A, but such HTML/code docs are extremely\n     rare in this raw pool.  To avoid starving that register we surface EVERY\n     genuine Q&A/code document (ranked by score) ahead of the prose tail, then\n     fill the remaining budget with the highest-scoring prose.\n\nEmits pool ids best-first; the trainer consumes them until the 12M-token budget\nis full.  This is a pure ranking function over a stated feature, not a\nhand-picked list.\n\"\"\"\nimport json, re, time, random\nimport numpy as np\nfrom collections import Counter\n\nPOOL = \"/workspace/data/pool.jsonl\"\nTARGET_NPY = \"/workspace/data/multi_dev.npy\"\nOUT = \"/workspace/submission/selection.json\"\n\nN_UNI = 60_000\nN_BI = 60_000\nBG_SAMPLE_FRAC = 0.40\nALPHA = 1.0\nN_OUT = 40_000\nSEED = 0\n\nMIN_WORDS = 60            # too short to teach anything\nMIN_ASCII_RATIO = 0.90    # guard against binary / non-English gibberish\nMAX_TOP_WORD_FRAC = 0.28  # a single word dominating -> spam / repetition\nMIN_STOP_RATIO = 0.12     # real prose is stopword-rich; menus/lists/code-dumps aren't\n\nrandom.seed(SEED)\n# HTML tags, HTML entities, and alnum words are all tokens (keeps the Q&A/code register)\nTOK_RE = re.compile(r\"</?[a-z][a-z0-9]*/?>|&[a-z]+;|[a-z0-9]+\")\nSTOP = set(\"the of and to in a is that for it as was on are with be by this \"\n           \"an at or from his he she they you not have has had but which we \"\n           \"were their its been would their about into more can\".split())\n\n\ndef toks(text):\n    return TOK_RE.findall(text.lower())\n\n\ndef passes(text):\n    if len(text) < 200:\n        return False, None\n    w = toks(text)\n    if len(w) < MIN_WORDS:\n        return False, None\n    ascii_ok = sum(32 <= ord(c) < 127 or c in \"\\n\\t\\r\" for c in text)\n    if ascii_ok / len(text) < MIN_ASCII_RATIO:\n        return False, None\n    c = Counter(w)\n    if c.most_common(1)[0][1] / len(w) > MAX_TOP_WORD_FRAC:\n        return False, None\n    stop = sum(c[s] for s in STOP)\n    if stop / len(w) < MIN_STOP_RATIO:   # navigation menus / list junk / raw code\n        return False, None\n    return True, w\n\n\ndef main():\n    t0 = time.time()\n    print(\"loading pool...\", flush=True)\n    ids, texts = [], []\n    with open(POOL) as f:\n        for line in f:\n            r = json.loads(line)\n            ids.append(r[\"id\"]); texts.append(r[\"text\"])\n    N = len(ids)\n    print(f\"  {N} docs loaded in {time.time()-t0:.0f}s\", flush=True)\n\n    # ---- target distribution ----\n    from transformers import AutoTokenizer\n    tok = AutoTokenizer.from_pretrained(\"gpt2\")\n    tgt_tokens = np.load(TARGET_NPY)\n    parts = []\n    for s in range(0, len(tgt_tokens), 4096):\n        parts.append(tok.decode(tgt_tokens[s:s+4096].tolist()))\n    tw = toks(\" \".join(parts))\n    tgt_uni = Counter(tw)\n    tgt_bi = Counter(a + \" \" + b for a, b in zip(tw, tw[1:]))\n    print(f\"  target: {len(tw)} tokens, {len(tgt_uni)} uni, {len(tgt_bi)} bi \"\n          f\"({time.time()-t0:.0f}s)\", flush=True)\n\n    vocab_uni = {w: i for i, (w, _) in enumerate(\n        [x for x in tgt_uni.most_common(N_UNI) if x[1] >= 2])}\n    vocab_bi = {w: i for i, (w, _) in enumerate(\n        [x for x in tgt_bi.most_common(N_BI) if x[1] >= 2])}\n    VU, VB = len(vocab_uni), len(vocab_bi)\n    print(f\"  vocab kept: {VU} uni, {VB} bi\", flush=True)\n\n    tgt_uc = np.zeros(VU); tgt_bc = np.zeros(VB)\n    for w, i in vocab_uni.items():\n        tgt_uc[i] = tgt_uni[w]\n    for w, i in vocab_bi.items():\n        tgt_bc[i] = tgt_bi[w]\n\n    # ---- background distribution (random pool sample) ----\n    bg_uc = np.zeros(VU); bg_bc = np.zeros(VB)\n    bg_idx = [j for j in range(N) if random.random() < BG_SAMPLE_FRAC]\n    for j in bg_idx:\n        w = toks(texts[j])\n        for x in w:\n            i = vocab_uni.get(x)\n            if i is not None:\n                bg_uc[i] += 1\n        for a, b in zip(w, w[1:]):\n            i = vocab_bi.get(a + \" \" + b)\n            if i is not None:\n                bg_bc[i] += 1\n    print(f\"  background from {len(bg_idx)} docs ({time.time()-t0:.0f}s)\", flush=True)\n\n    Tu, Bu = tgt_uc.sum(), bg_uc.sum()\n    Tb, Bb = tgt_bc.sum(), bg_bc.sum()\n    wu = np.log((tgt_uc + ALPHA) / (Tu + ALPHA * VU)) - \\\n         np.log((bg_uc + ALPHA) / (Bu + ALPHA * VU))\n    wb = np.log((tgt_bc + ALPHA) / (Tb + ALPHA * VB)) - \\\n         np.log((bg_bc + ALPHA) / (Bb + ALPHA * VB))\n\n    # ---- score every doc; track register + dedup ----\n    scores = np.full(N, -1e9)\n    qa_flag = np.zeros(N, dtype=bool)\n    seen = set()\n    kept = 0\n    for j in range(N):\n        h = hash(texts[j])\n        if h in seen:\n            continue          # exact-duplicate text\n        ok, w = passes(texts[j])\n        if not ok:\n            continue\n        seen.add(h)\n        kept += 1\n        qa_flag[j] = is_qa(texts[j])\n        su = 0.0\n        for x in w:\n            i = vocab_uni.get(x)\n            if i is not None:\n                su += wu[i]\n        sb = 0.0\n        for a, b in zip(w, w[1:]):\n            i = vocab_bi.get(a + \" \" + b)\n            if i is not None:\n                sb += wb[i]\n        scores[j] = (su + 0.5 * sb) / len(w)\n    print(f\"  scored; {kept} unique docs pass ({time.time()-t0:.0f}s)\", flush=True)\n\n    valid = scores > -1e8\n    order = np.argsort(-scores)\n    order = [j for j in order if valid[j]]\n    # surface all genuine Q&A/code docs (best-first) ahead of the prose tail\n    qa_order = [j for j in order if qa_flag[j]]\n    prose_order = [j for j in order if not qa_flag[j]]\n    final = qa_order + prose_order\n    sel = [int(ids[j]) for j in final[:N_OUT]]\n    json.dump(sel, open(OUT, \"w\"))\n    print(f\"  qa docs surfaced: {len(qa_order)}\", flush=True)\n    print(f\"wrote {len(sel)} ids -> {OUT} ({time.time()-t0:.0f}s)\", flush=True)\n    sv = scores[valid]\n    print(\"score quantiles:\", np.round(np.quantile(sv, [0, .5, .9, 1]), 3))\n\n\nif __name__ == \"__main__\":\n    main()\n", "structuredPatch": [{"oldStart": 135, "oldLines": 9, "newStart": 135, "newLines": 8, "lines": ["     wb = np.log((tgt_bc + ALPHA) / (Tb + ALPHA * VB)) - \\", "          np.log((bg_bc + ALPHA) / (Bb + ALPHA * VB))", " ", "-    # ---- score every doc; track register + dedup ----", "+    # ---- score every doc; dedup exact-duplicate texts ----", "     scores = np.full(N, -1e9)", "-    qa_flag = np.zeros(N, dtype=bool)", "     seen = set()", "     kept = 0", "     for j in range(N):"]}, {"oldStart": 149, "oldLines": 7, "newStart": 148, "newLines": 6, "lines": ["             continue", "         seen.add(h)", "         kept += 1", "-        qa_flag[j] = is_qa(texts[j])", "         su = 0.0", "         for x in w:", "             i = vocab_uni.get(x)"]}, {"oldStart": 163, "oldLines": 19, "newStart": 161, "newLines": 14, "lines": ["         scores[j] = (su + 0.5 * sb) / len(w)", "     print(f\"  scored; {kept} unique docs pass ({time.time()-t0:.0f}s)\", flush=True)", " ", "+    # pure importance ranking: the LLR against the full 4-register target already", "+    # reflects the target's register mix, so no forced per-register quotas.", "     valid = scores > -1e8", "-    order = np.argsort(-scores)", "-    order = [j for j in order if valid[j]]", "-    # surface all genuine Q&A/code docs (best-first) ahead of the prose tail", "-    qa_order = [j for j in order if qa_flag[j]]", "-    prose_order = [j for j in order if not qa_flag[j]]", "-    final = qa_order + prose_order", "-    sel = [int(ids[j]) for j in final[:N_OUT]]", "+    order = [j for j in np.argsort(-scores) if valid[j]]", "+    sel = [int(ids[j]) for j in order[:N_OUT]]", "     json.dump(sel, open(OUT, \"w\"))", "-    print(f\"  qa docs surfaced: {len(qa_order)}\", flush=True)", "     print(f\"wrote {len(sel)} ids -> {OUT} ({time.time()-t0:.0f}s)\", flush=True)", "-    sv = scores[valid]", "-    print(\"score quantiles:\", np.round(np.quantile(sv, [0, .5, .9, 1]), 3))", "+    print(\"score quantiles:\", np.round(np.quantile(scores[valid], [0, .5, .9, 1]), 3))", " ", " ", " if __name__ == \"__main__\":"]}], "userModified": false, "replaceAll": false}

Command running in background with ID: bvm8edx6a. Output is being written to: /tmp/claude-0/-workspace/2eda86eb-7dd5-4994-9ab5-ba9a2e6aa267/tasks/bvm8edx6a.output. You will be notified when it completes. To check interim output, use Read on that file path.
Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "bvm8edx6a", "backgroundCwdHint": "Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands."}

loading pool...
  182016 docs loaded in 6s
  target: 699682 tokens, 38561 uni, 326803 bi (11s)
  vocab kept: 22629 uni, 60000 bi

[stdout]
loading pool...
  182016 docs loaded in 6s
  target: 699682 tokens, 38561 uni, 326803 bi (11s)
  vocab kept: 22629 uni, 60000 bi
[metadata] {"noOutputExpected": false}

background from 73259 docs (44s)
  scored; 147493 unique docs pass (146s)
wrote 40000 ids -> /workspace/submission/selection.json (146s)
score quantiles: [-1.241 -0.283 -0.051  0.652]

[stdout]
  background from 73259 docs (44s)
  scored; 147493 unique docs pass (146s)
wrote 40000 ids -> /workspace/submission/selection.json (146s)
score quantiles: [-1.241 -0.283 -0.051  0.652]
[metadata] {"noOutputExpected": false}

top-12k register: Counter({'prose': 11953, 'entity': 27, 'SO_html': 20})
=== TOP 4 docs ===
[58452] '<|endoftext|>New Delhi: Five new Supreme Court judges were today sworn in, taking the top court’s strength to 28, including the Chief Justice of India. Justices Sanjay Ki'

[37064] 'The Bharatiya Janata Party (BJP) leaders of the West Bengal unit will meet Union Home Minister Rajnath Singh on Thursday to lodge protest against the vandalising of its p'

[79735] '<|endoftext|>India’s PM Narendra Modi has tweeted pictures of his mother’s first visit to his official residence in Delhi.\nTwitter users have praised Narendra Modi’s deci'

[63278] 'leging that Prime Minister Narendra Modi was trying to see there won’t be any Opposition in the country, AICC secretary V Hanumanth Rao said the people were watching Modi'

[stdout]
top-12k register: Counter({'prose': 11953, 'entity': 27, 'SO_html': 20})
=== TOP 4 docs ===
[58452] '<|endoftext|>New Delhi: Five new Supreme Court judges were today sworn in, taking the top court’s strength to 28, including the Chief Justice of India. Justices Sanjay Ki'

[37064] 'The Bharatiya Janata Party (BJP) leaders of the West Bengal unit will meet Union Home Minister Rajnath Singh on Thursday to lodge protest against the vandalising of its p'

[79735] '<|endoftext|>India’s PM Narendra Modi has tweeted pictures of his mother’s first visit to his official residence in Delhi.\nTwitter users have praised Narendra Modi’s deci'

[63278] 'leging that Prime Minister Narendra Modi was trying to see there won’t be any Opposition in the country, AICC secretary V Hanumanth Rao said the people were watching Modi'
[metadata] {"noOutputExpected": false}

Command running in background with ID: b71r2gy1x. Output is being written to: /tmp/claude-0/-workspace/2eda86eb-7dd5-4994-9ab5-ba9a2e6aa267/tasks/b71r2gy1x.output. You will be notified when it completes. To check interim output, use Read on that file path.
Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "b71r2gy1x", "backgroundCwdHint": "Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands."}

HELDOUT_PPL 353.634  loss 5.8683  train_s 55  /tmp/dev_train.npy

[stdout]
HELDOUT_PPL 353.634  loss 5.8683  train_s 55  /tmp/dev_train.npy
[metadata] {"noOutputExpected": false}

3869
--- duplicate rate in pool ---
exact-dup docs: 11753 of 182016 (6.5%)

[stdout]
3869
--- duplicate rate in pool ---
exact-dup docs: 11753 of 182016 (6.5%)
[metadata] {"noOutputExpected": false}

The file /workspace/submission/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"type": "update", "filePath": "/workspace/submission/curate.py", "content": "#!/usr/bin/env python3\n\"\"\"Curate the raw web pool for training a small LM whose held-out target is a\nBROAD, multi-domain high-quality English distribution (Wikipedia + high-quality\nweb prose + news + technical Q&A), disclosed via /workspace/data/multi_dev.npy.\n\nCriterion (stated, reproducible) — DSIR-style importance ranking:\n\n  1. Decode the disclosed target token array back to text -> a sample of the\n     target domain.\n  2. Fit a unigram+bigram bag-of-words distribution for the TARGET and for a\n     random BACKGROUND sample of the raw pool.\n  3. Score every pool document by its mean per-token log-likelihood ratio\n     log p_target(w) - log p_background(w).  Documents whose vocabulary looks\n     like the clean multi-domain target score high; boilerplate / forum / spam\n     (whose n-grams are common in the raw pool but rare in the target) score low.\n  4. Light hard filters drop only degenerate documents (too short, gibberish /\n     binary, a single word repeated).  Exact-duplicate texts are removed so the\n     token budget is not spent re-reading the same document (the pool is ~6.5%\n     exact duplicates).\n  5. Emit pool ids in descending score order (best first).  The trainer consumes\n     this priority list until the 12M-token budget is full.\n\nThe importance ratio is computed against the FULL multi-register target, so the\nranking already reflects the target's register mix; no hand-tuned per-domain\nquotas are applied.  This is a pure ranking function over a stated feature, not\na hand-picked id list.\n\"\"\"\nimport json, re, time, random\nimport numpy as np\nfrom collections import Counter\n\nPOOL = \"/workspace/data/pool.jsonl\"\nTARGET_NPY = \"/workspace/data/multi_dev.npy\"\nOUT = \"/workspace/submission/selection.json\"\n\nN_UNI = 60_000          # top target unigrams kept in the model vocabulary\nN_BI = 60_000           # top target bigrams kept\nBG_SAMPLE_FRAC = 0.40   # fraction of pool docs used to estimate the background\nALPHA = 1.0             # additive smoothing\nN_OUT = 40_000          # ids to emit (>> enough to cover 12M tokens)\nSEED = 0\nDEDUP = True            # drop exact-duplicate documents\n\nMIN_WORDS = 40\nMIN_ALPHA_RATIO = 0.55\nMAX_TOP_WORD_FRAC = 0.30\n\nrandom.seed(SEED)\nWORD_RE = re.compile(r\"[a-z0-9]+\")\n\n\ndef words(text):\n    return WORD_RE.findall(text.lower())\n\n\ndef passes(text):\n    if len(text) < 200:\n        return False, None\n    w = words(text)\n    if len(w) < MIN_WORDS:\n        return False, None\n    good = sum(c.isalpha() or c.isspace() or c in \",.;:'\\\"?!()-\" for c in text)\n    if good / max(1, len(text)) < MIN_ALPHA_RATIO:\n        return False, None\n    c = Counter(w)\n    if c.most_common(1)[0][1] / len(w) > MAX_TOP_WORD_FRAC:\n        return False, None\n    return True, w\n\n\ndef main():\n    t0 = time.time()\n    print(\"loading pool...\", flush=True)\n    ids, texts = [], []\n    with open(POOL) as f:\n        for line in f:\n            r = json.loads(line)\n            ids.append(r[\"id\"]); texts.append(r[\"text\"])\n    N = len(ids)\n    print(f\"  {N} docs loaded in {time.time()-t0:.0f}s\", flush=True)\n\n    # ---- target distribution ----\n    from transformers import AutoTokenizer\n    tok = AutoTokenizer.from_pretrained(\"gpt2\")\n    tgt_tokens = np.load(TARGET_NPY)\n    parts = []\n    for s in range(0, len(tgt_tokens), 4096):\n        parts.append(tok.decode(tgt_tokens[s:s+4096].tolist()))\n    tw = words(\" \".join(parts))\n    tgt_uni = Counter(tw)\n    tgt_bi = Counter(a + \" \" + b for a, b in zip(tw, tw[1:]))\n    print(f\"  target: {len(tw)} words, {len(tgt_uni)} uni, {len(tgt_bi)} bi \"\n          f\"({time.time()-t0:.0f}s)\", flush=True)\n\n    vocab_uni = {w: i for i, (w, _) in enumerate(\n        [x for x in tgt_uni.most_common(N_UNI) if x[1] >= 2])}\n    vocab_bi = {w: i for i, (w, _) in enumerate(\n        [x for x in tgt_bi.most_common(N_BI) if x[1] >= 2])}\n    VU, VB = len(vocab_uni), len(vocab_bi)\n    print(f\"  vocab kept: {VU} uni, {VB} bi\", flush=True)\n\n    tgt_uc = np.zeros(VU); tgt_bc = np.zeros(VB)\n    for w, i in vocab_uni.items():\n        tgt_uc[i] = tgt_uni[w]\n    for w, i in vocab_bi.items():\n        tgt_bc[i] = tgt_bi[w]\n\n    # ---- background distribution (random pool sample) ----\n    bg_uc = np.zeros(VU); bg_bc = np.zeros(VB)\n    bg_idx = [j for j in range(N) if random.random() < BG_SAMPLE_FRAC]\n    for j in bg_idx:\n        w = words(texts[j])\n        for x in w:\n            i = vocab_uni.get(x)\n            if i is not None:\n                bg_uc[i] += 1\n        for a, b in zip(w, w[1:]):\n            i = vocab_bi.get(a + \" \" + b)\n            if i is not None:\n                bg_bc[i] += 1\n    print(f\"  background from {len(bg_idx)} docs ({time.time()-t0:.0f}s)\", flush=True)\n\n    Tu, Bu = tgt_uc.sum(), bg_uc.sum()\n    Tb, Bb = tgt_bc.sum(), bg_bc.sum()\n    wu = np.log((tgt_uc + ALPHA) / (Tu + ALPHA * VU)) - \\\n         np.log((bg_uc + ALPHA) / (Bu + ALPHA * VU))\n    wb = np.log((tgt_bc + ALPHA) / (Tb + ALPHA * VB)) - \\\n         np.log((bg_bc + ALPHA) / (Bb + ALPHA * VB))\n\n    # ---- score every doc ----\n    scores = np.full(N, -1e9)\n    seen = set()\n    kept = 0\n    for j in range(N):\n        if DEDUP:\n            h = hash(texts[j])\n            if h in seen:\n                continue\n            seen.add(h)\n        ok, w = passes(texts[j])\n        if not ok:\n            continue\n        kept += 1\n        su = 0.0\n        for x in w:\n            i = vocab_uni.get(x)\n            if i is not None:\n                su += wu[i]\n        sb = 0.0\n        for a, b in zip(w, w[1:]):\n            i = vocab_bi.get(a + \" \" + b)\n            if i is not None:\n                sb += wb[i]\n        scores[j] = (su + 0.5 * sb) / len(w)\n    print(f\"  scored; {kept} docs pass ({time.time()-t0:.0f}s)\", flush=True)\n\n    valid = scores > -1e8\n    order = [j for j in np.argsort(-scores) if valid[j]]\n    sel = [int(ids[j]) for j in order[:N_OUT]]\n    json.dump(sel, open(OUT, \"w\"))\n    print(f\"wrote {len(sel)} ids -> {OUT} ({time.time()-t0:.0f}s)\", flush=True)\n    print(\"score quantiles:\", np.round(np.quantile(scores[valid], [0, .5, .9, 1]), 3))\n\n\nif __name__ == \"__main__\":\n    main()\n", "structuredPatch": [{"oldStart": 3, "oldLines": 28, "newStart": 3, "newLines": 27, "lines": [" BROAD, multi-domain high-quality English distribution (Wikipedia + high-quality", " web prose + news + technical Q&A), disclosed via /workspace/data/multi_dev.npy.", " ", "-Criterion (stated, reproducible): DSIR-style importance scoring with", "-register-aware coverage.", "+Criterion (stated, reproducible) — DSIR-style importance ranking:", " ", "   1. Decode the disclosed target token array back to text -> a sample of the", "      target domain.", "-  2. Fit a unigram+bigram bag-of-features distribution for the TARGET and for a", "-     random BACKGROUND sample of the raw pool.  Features are HTML/entity aware", "-     (`<code>`, `<p>`, `&quot;` are tokens) so the technical-Q&A register is", "-     represented, not stripped.", "+  2. Fit a unigram+bigram bag-of-words distribution for the TARGET and for a", "+     random BACKGROUND sample of the raw pool.", "   3. Score every pool document by its mean per-token log-likelihood ratio", "-     log p_target(f) - log p_background(f).  Clean multi-domain prose scores", "-     high; boilerplate / forum / spam scores low.", "-  4. Light hard filters drop only degenerate docs (too short, gibberish/binary,", "-     single word repeated).  Exact-duplicate texts are removed.", "-  5. The target is ~1/4 technical Q&A, but such HTML/code docs are extremely", "-     rare in this raw pool.  To avoid starving that register we surface EVERY", "-     genuine Q&A/code document (ranked by score) ahead of the prose tail, then", "-     fill the remaining budget with the highest-scoring prose.", "+     log p_target(w) - log p_background(w).  Documents whose vocabulary looks", "+     like the clean multi-domain target score high; boilerplate / forum / spam", "+     (whose n-grams are common in the raw pool but rare in the target) score low.", "+  4. Light hard filters drop only degenerate documents (too short, gibberish /", "+     binary, a single word repeated).  Exact-duplicate texts are removed so the", "+     token budget is not spent re-reading the same document (the pool is ~6.5%", "+     exact duplicates).", "+  5. Emit pool ids in descending score order (best first).  The trainer consumes", "+     this priority list until the 12M-token budget is full.", " ", "-Emits pool ids best-first; the trainer consumes them until the 12M-token budget", "-is full.  This is a pure ranking function over a stated feature, not a", "-hand-picked list.", "+The importance ratio is computed against the FULL multi-register target, so the", "+ranking already reflects the target's register mix; no hand-tuned per-domain", "+quotas are applied.  This is a pure ranking function over a stated feature, not", "+a hand-picked id list.", " \"\"\"", " import json, re, time, random", " import numpy as np"]}, {"oldStart": 34, "oldLines": 45, "newStart": 33, "newLines": 38, "lines": [" TARGET_NPY = \"/workspace/data/multi_dev.npy\"", " OUT = \"/workspace/submission/selection.json\"", " ", "-N_UNI = 60_000", "-N_BI = 60_000", "-BG_SAMPLE_FRAC = 0.40", "-ALPHA = 1.0", "-N_OUT = 40_000", "+N_UNI = 60_000          # top target unigrams kept in the model vocabulary", "+N_BI = 60_000           # top target bigrams kept", "+BG_SAMPLE_FRAC = 0.40   # fraction of pool docs used to estimate the background", "+ALPHA = 1.0             # additive smoothing", "+N_OUT = 40_000          # ids to emit (>> enough to cover 12M tokens)", " SEED = 0", "+DEDUP = True            # drop exact-duplicate documents", " ", "-MIN_WORDS = 60            # too short to teach anything", "-MIN_ASCII_RATIO = 0.90    # guard against binary / non-English gibberish", "-MAX_TOP_WORD_FRAC = 0.28  # a single word dominating -> spam / repetition", "-MIN_STOP_RATIO = 0.12     # real prose is stopword-rich; menus/lists/code-dumps aren't", "+MIN_WORDS = 40", "+MIN_ALPHA_RATIO = 0.55", "+MAX_TOP_WORD_FRAC = 0.30", " ", " random.seed(SEED)", "-# HTML tags, HTML entities, and alnum words are all tokens (keeps the Q&A/code register)", "-TOK_RE = re.compile(r\"</?[a-z][a-z0-9]*/?>|&[a-z]+;|[a-z0-9]+\")", "-STOP = set(\"the of and to in a is that for it as was on are with be by this \"", "-           \"an at or from his he she they you not have has had but which we \"", "-           \"were their its been would their about into more can\".split())", "+WORD_RE = re.compile(r\"[a-z0-9]+\")", " ", " ", "-def toks(text):", "-    return TOK_RE.findall(text.lower())", "+def words(text):", "+    return WORD_RE.findall(text.lower())", " ", " ", " def passes(text):", "     if len(text) < 200:", "         return False, None", "-    w = toks(text)", "+    w = words(text)", "     if len(w) < MIN_WORDS:", "         return False, None", "-    ascii_ok = sum(32 <= ord(c) < 127 or c in \"\\n\\t\\r\" for c in text)", "-    if ascii_ok / len(text) < MIN_ASCII_RATIO:", "+    good = sum(c.isalpha() or c.isspace() or c in \",.;:'\\\"?!()-\" for c in text)", "+    if good / max(1, len(text)) < MIN_ALPHA_RATIO:", "         return False, None", "     c = Counter(w)", "     if c.most_common(1)[0][1] / len(w) > MAX_TOP_WORD_FRAC:", "         return False, None", "-    stop = sum(c[s] for s in STOP)", "-    if stop / len(w) < MIN_STOP_RATIO:   # navigation menus / list junk / raw code", "-        return False, None", "     return True, w", " ", " "]}, {"oldStart": 94, "oldLines": 10, "newStart": 86, "newLines": 10, "lines": ["     parts = []", "     for s in range(0, len(tgt_tokens), 4096):", "         parts.append(tok.decode(tgt_tokens[s:s+4096].tolist()))", "-    tw = toks(\" \".join(parts))", "+    tw = words(\" \".join(parts))", "     tgt_uni = Counter(tw)", "     tgt_bi = Counter(a + \" \" + b for a, b in zip(tw, tw[1:]))", "-    print(f\"  target: {len(tw)} tokens, {len(tgt_uni)} uni, {len(tgt_bi)} bi \"", "+    print(f\"  target: {len(tw)} words, {len(tgt_uni)} uni, {len(tgt_bi)} bi \"", "           f\"({time.time()-t0:.0f}s)\", flush=True)", " ", "     vocab_uni = {w: i for i, (w, _) in enumerate("]}, {"oldStart": 117, "oldLines": 7, "newStart": 109, "newLines": 7, "lines": ["     bg_uc = np.zeros(VU); bg_bc = np.zeros(VB)", "     bg_idx = [j for j in range(N) if random.random() < BG_SAMPLE_FRAC]", "     for j in bg_idx:", "-        w = toks(texts[j])", "+        w = words(texts[j])", "         for x in w:", "             i = vocab_uni.get(x)", "             if i is not None:"]}, {"oldStart": 135, "oldLines": 18, "newStart": 127, "newLines": 19, "lines": ["     wb = np.log((tgt_bc + ALPHA) / (Tb + ALPHA * VB)) - \\", "          np.log((bg_bc + ALPHA) / (Bb + ALPHA * VB))", " ", "-    # ---- score every doc; dedup exact-duplicate texts ----", "+    # ---- score every doc ----", "     scores = np.full(N, -1e9)", "     seen = set()", "     kept = 0", "     for j in range(N):", "-        h = hash(texts[j])", "-        if h in seen:", "-            continue          # exact-duplicate text", "+        if DEDUP:", "+            h = hash(texts[j])", "+            if h in seen:", "+                continue", "+            seen.add(h)", "         ok, w = passes(texts[j])", "         if not ok:", "             continue", "-        seen.add(h)", "         kept += 1", "         su = 0.0", "         for x in w:"]}, {"oldStart": 159, "oldLines": 10, "newStart": 152, "newLines": 8, "lines": ["             if i is not None:", "                 sb += wb[i]", "         scores[j] = (su + 0.5 * sb) / len(w)", "-    print(f\"  scored; {kept} unique docs pass ({time.time()-t0:.0f}s)\", flush=True)", "+    print(f\"  scored; {kept} docs pass ({time.time()-t0:.0f}s)\", flush=True)", " ", "-    # pure importance ranking: the LLR against the full 4-register target already", "-    # reflects the target's register mix, so no forced per-register quotas.", "     valid = scores > -1e8", "     order = [j for j in np.argsort(-scores) if valid[j]]", "     sel = [int(ids[j]) for j in order[:N_OUT]]"]}], "originalFile": "#!/usr/bin/env python3\n\"\"\"Curate the raw web pool for training a small LM whose held-out target is a\nBROAD, multi-domain high-quality English distribution (Wikipedia + high-quality\nweb prose + news + technical Q&A), disclosed via /workspace/data/multi_dev.npy.\n\nCriterion (stated, reproducible): DSIR-style importance scoring with\nregister-aware coverage.\n\n  1. Decode the disclosed target token array back to text -> a sample of the\n     target domain.\n  2. Fit a unigram+bigram bag-of-features distribution for the TARGET and for a\n     random BACKGROUND sample of the raw pool.  Features are HTML/entity aware\n     (`<code>`, `<p>`, `&quot;` are tokens) so the technical-Q&A register is\n     represented, not stripped.\n  3. Score every pool document by its mean per-token log-likelihood ratio\n     log p_target(f) - log p_background(f).  Clean multi-domain prose scores\n     high; boilerplate / forum / spam scores low.\n  4. Light hard filters drop only degenerate docs (too short, gibberish/binary,\n     single word repeated).  Exact-duplicate texts are removed.\n  5. The target is ~1/4 technical Q&A, but such HTML/code docs are extremely\n     rare in this raw pool.  To avoid starving that register we surface EVERY\n     genuine Q&A/code document (ranked by score) ahead of the prose tail, then\n     fill the remaining budget with the highest-scoring prose.\n\nEmits pool ids best-first; the trainer consumes them until the 12M-token budget\nis full.  This is a pure ranking function over a stated feature, not a\nhand-picked list.\n\"\"\"\nimport json, re, time, random\nimport numpy as np\nfrom collections import Counter\n\nPOOL = \"/workspace/data/pool.jsonl\"\nTARGET_NPY = \"/workspace/data/multi_dev.npy\"\nOUT = \"/workspace/submission/selection.json\"\n\nN_UNI = 60_000\nN_BI = 60_000\nBG_SAMPLE_FRAC = 0.40\nALPHA = 1.0\nN_OUT = 40_000\nSEED = 0\n\nMIN_WORDS = 60            # too short to teach anything\nMIN_ASCII_RATIO = 0.90    # guard against binary / non-English gibberish\nMAX_TOP_WORD_FRAC = 0.28  # a single word dominating -> spam / repetition\nMIN_STOP_RATIO = 0.12     # real prose is stopword-rich; menus/lists/code-dumps aren't\n\nrandom.seed(SEED)\n# HTML tags, HTML entities, and alnum words are all tokens (keeps the Q&A/code register)\nTOK_RE = re.compile(r\"</?[a-z][a-z0-9]*/?>|&[a-z]+;|[a-z0-9]+\")\nSTOP = set(\"the of and to in a is that for it as was on are with be by this \"\n           \"an at or from his he she they you not have has had but which we \"\n           \"were their its been would their about into more can\".split())\n\n\ndef toks(text):\n    return TOK_RE.findall(text.lower())\n\n\ndef passes(text):\n    if len(text) < 200:\n        return False, None\n    w = toks(text)\n    if len(w) < MIN_WORDS:\n        return False, None\n    ascii_ok = sum(32 <= ord(c) < 127 or c in \"\\n\\t\\r\" for c in text)\n    if ascii_ok / len(text) < MIN_ASCII_RATIO:\n        return False, None\n    c = Counter(w)\n    if c.most_common(1)[0][1] / len(w) > MAX_TOP_WORD_FRAC:\n        return False, None\n    stop = sum(c[s] for s in STOP)\n    if stop / len(w) < MIN_STOP_RATIO:   # navigation menus / list junk / raw code\n        return False, None\n    return True, w\n\n\ndef main():\n    t0 = time.time()\n    print(\"loading pool...\", flush=True)\n    ids, texts = [], []\n    with open(POOL) as f:\n        for line in f:\n            r = json.loads(line)\n            ids.append(r[\"id\"]); texts.append(r[\"text\"])\n    N = len(ids)\n    print(f\"  {N} docs loaded in {time.time()-t0:.0f}s\", flush=True)\n\n    # ---- target distribution ----\n    from transformers import AutoTokenizer\n    tok = AutoTokenizer.from_pretrained(\"gpt2\")\n    tgt_tokens = np.load(TARGET_NPY)\n    parts = []\n    for s in range(0, len(tgt_tokens), 4096):\n        parts.append(tok.decode(tgt_tokens[s:s+4096].tolist()))\n    tw = toks(\" \".join(parts))\n    tgt_uni = Counter(tw)\n    tgt_bi = Counter(a + \" \" + b for a, b in zip(tw, tw[1:]))\n    print(f\"  target: {len(tw)} tokens, {len(tgt_uni)} uni, {len(tgt_bi)} bi \"\n          f\"({time.time()-t0:.0f}s)\", flush=True)\n\n    vocab_uni = {w: i for i, (w, _) in enumerate(\n        [x for x in tgt_uni.most_common(N_UNI) if x[1] >= 2])}\n    vocab_bi = {w: i for i, (w, _) in enumerate(\n        [x for x in tgt_bi.most_common(N_BI) if x[1] >= 2])}\n    VU, VB = len(vocab_uni), len(vocab_bi)\n    print(f\"  vocab kept: {VU} uni, {VB} bi\", flush=True)\n\n    tgt_uc = np.zeros(VU); tgt_bc = np.zeros(VB)\n    for w, i in vocab_uni.items():\n        tgt_uc[i] = tgt_uni[w]\n    for w, i in vocab_bi.items():\n        tgt_bc[i] = tgt_bi[w]\n\n    # ---- background distribution (random pool sample) ----\n    bg_uc = np.zeros(VU); bg_bc = np.zeros(VB)\n    bg_idx = [j for j in range(N) if random.random() < BG_SAMPLE_FRAC]\n    for j in bg_idx:\n        w = toks(texts[j])\n        for x in w:\n            i = vocab_uni.get(x)\n            if i is not None:\n                bg_uc[i] += 1\n        for a, b in zip(w, w[1:]):\n            i = vocab_bi.get(a + \" \" + b)\n            if i is not None:\n                bg_bc[i] += 1\n    print(f\"  background from {len(bg_idx)} docs ({time.time()-t0:.0f}s)\", flush=True)\n\n    Tu, Bu = tgt_uc.sum(), bg_uc.sum()\n    Tb, Bb = tgt_bc.sum(), bg_bc.sum()\n    wu = np.log((tgt_uc + ALPHA) / (Tu + ALPHA * VU)) - \\\n         np.log((bg_uc + ALPHA) / (Bu + ALPHA * VU))\n    wb = np.log((tgt_bc + ALPHA) / (Tb + ALPHA * VB)) - \\\n         np.log((bg_bc + ALPHA) / (Bb + ALPHA * VB))\n\n    # ---- score every doc; dedup exact-duplicate texts ----\n    scores = np.full(N, -1e9)\n    seen = set()\n    kept = 0\n    for j in range(N):\n        h = hash(texts[j])\n        if h in seen:\n            continue          # exact-duplicate text\n        ok, w = passes(texts[j])\n        if not ok:\n            continue\n        seen.add(h)\n        kept += 1\n        su = 0.0\n        for x in w:\n            i = vocab_uni.get(x)\n            if i is not None:\n                su += wu[i]\n        sb = 0.0\n        for a, b in zip(w, w[1:]):\n            i = vocab_bi.get(a + \" \" + b)\n            if i is not None:\n                sb += wb[i]\n        scores[j] = (su + 0.5 * sb) / len(w)\n    print(f\"  scored; {kept} unique docs pass ({time.time()-t0:.0f}s)\", flush=True)\n\n    # pure importance ranking: the LLR against the full 4-register target already\n    # reflects the target's register mix, so no forced per-register quotas.\n    valid = scores > -1e8\n    order = [j for j in np.argsort(-scores) if valid[j]]\n    sel = [int(ids[j]) for j in order[:N_OUT]]\n    json.dump(sel, open(OUT, \"w\"))\n    print(f\"wrote {len(sel)} ids -> {OUT} ({time.time()-t0:.0f}s)\", flush=True)\n    print(\"score quantiles:\", np.round(np.quantile(scores[valid], [0, .5, .9, 1]), 3))\n\n\nif __name__ == \"__main__\":\n    main()\n", "userModified": false}

Command running in background with ID: biex4090z. Output is being written to: /tmp/claude-0/-workspace/2eda86eb-7dd5-4994-9ab5-ba9a2e6aa267/tasks/biex4090z.output. You will be notified when it completes. To check interim output, use Read on that file path.
Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "biex4090z", "backgroundCwdHint": "Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands."}

background from 73259 docs (40s)
  scored; 167870 docs pass (134s)
wrote 40000 ids -> /workspace/submission/selection.json (134s)
score quantiles: [-1.556 -0.303 -0.059  0.707]

[stdout]
  background from 73259 docs (40s)
  scored; 167870 docs pass (134s)
wrote 40000 ids -> /workspace/submission/selection.json (134s)
score quantiles: [-1.556 -0.303 -0.059  0.707]
[metadata] {"noOutputExpected": false}

Command running in background with ID: b3xeq1ul9. Output is being written to: /tmp/claude-0/-workspace/2eda86eb-7dd5-4994-9ab5-ba9a2e6aa267/tasks/b3xeq1ul9.output. You will be notified when it completes. To check interim output, use Read on that file path.
Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "b3xeq1ul9", "backgroundCwdHint": "Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands."}

(Bash completed with no output)

[metadata] {"returnCodeInterpretation": "No matches found", "noOutputExpected": false}

HELDOUT_PPL 347.445  loss 5.8506  train_s 55  /tmp/dev_train.npy

[stdout]
HELDOUT_PPL 347.445  loss 5.8506  train_s 55  /tmp/dev_train.npy
[metadata] {"noOutputExpected": false}

The file /workspace/submission/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/submission/curate.py", "oldString": "DEDUP = True            # drop exact-duplicate documents", "newString": "DEDUP = False           # keep exact duplicates: duplicated high-LLR on-target docs\n                        # act as an implicit up-weighting of the best content under\n                        # the trainer's random-window sampling (measured: dedup RAISED\n                        # dev perplexity 335.6 -> 347.4, so it is disabled).", "originalFile": "#!/usr/bin/env python3\n\"\"\"Curate the raw web pool for training a small LM whose held-out target is a\nBROAD, multi-domain high-quality English distribution (Wikipedia + high-quality\nweb prose + news + technical Q&A), disclosed via /workspace/data/multi_dev.npy.\n\nCriterion (stated, reproducible) — DSIR-style importance ranking:\n\n  1. Decode the disclosed target token array back to text -> a sample of the\n     target domain.\n  2. Fit a unigram+bigram bag-of-words distribution for the TARGET and for a\n     random BACKGROUND sample of the raw pool.\n  3. Score every pool document by its mean per-token log-likelihood ratio\n     log p_target(w) - log p_background(w).  Documents whose vocabulary looks\n     like the clean multi-domain target score high; boilerplate / forum / spam\n     (whose n-grams are common in the raw pool but rare in the target) score low.\n  4. Light hard filters drop only degenerate documents (too short, gibberish /\n     binary, a single word repeated).  Exact-duplicate texts are removed so the\n     token budget is not spent re-reading the same document (the pool is ~6.5%\n     exact duplicates).\n  5. Emit pool ids in descending score order (best first).  The trainer consumes\n     this priority list until the 12M-token budget is full.\n\nThe importance ratio is computed against the FULL multi-register target, so the\nranking already reflects the target's register mix; no hand-tuned per-domain\nquotas are applied.  This is a pure ranking function over a stated feature, not\na hand-picked id list.\n\"\"\"\nimport json, re, time, random\nimport numpy as np\nfrom collections import Counter\n\nPOOL = \"/workspace/data/pool.jsonl\"\nTARGET_NPY = \"/workspace/data/multi_dev.npy\"\nOUT = \"/workspace/submission/selection.json\"\n\nN_UNI = 60_000          # top target unigrams kept in the model vocabulary\nN_BI = 60_000           # top target bigrams kept\nBG_SAMPLE_FRAC = 0.40   # fraction of pool docs used to estimate the background\nALPHA = 1.0             # additive smoothing\nN_OUT = 40_000          # ids to emit (>> enough to cover 12M tokens)\nSEED = 0\nDEDUP = True            # drop exact-duplicate documents\n\nMIN_WORDS = 40\nMIN_ALPHA_RATIO = 0.55\nMAX_TOP_WORD_FRAC = 0.30\n\nrandom.seed(SEED)\nWORD_RE = re.compile(r\"[a-z0-9]+\")\n\n\ndef words(text):\n    return WORD_RE.findall(text.lower())\n\n\ndef passes(text):\n    if len(text) < 200:\n        return False, None\n    w = words(text)\n    if len(w) < MIN_WORDS:\n        return False, None\n    good = sum(c.isalpha() or c.isspace() or c in \",.;:'\\\"?!()-\" for c in text)\n    if good / max(1, len(text)) < MIN_ALPHA_RATIO:\n        return False, None\n    c = Counter(w)\n    if c.most_common(1)[0][1] / len(w) > MAX_TOP_WORD_FRAC:\n        return False, None\n    return True, w\n\n\ndef main():\n    t0 = time.time()\n    print(\"loading pool...\", flush=True)\n    ids, texts = [], []\n    with open(POOL) as f:\n        for line in f:\n            r = json.loads(line)\n            ids.append(r[\"id\"]); texts.append(r[\"text\"])\n    N = len(ids)\n    print(f\"  {N} docs loaded in {time.time()-t0:.0f}s\", flush=True)\n\n    # ---- target distribution ----\n    from transformers import AutoTokenizer\n    tok = AutoTokenizer.from_pretrained(\"gpt2\")\n    tgt_tokens = np.load(TARGET_NPY)\n    parts = []\n    for s in range(0, len(tgt_tokens), 4096):\n        parts.append(tok.decode(tgt_tokens[s:s+4096].tolist()))\n    tw = words(\" \".join(parts))\n    tgt_uni = Counter(tw)\n    tgt_bi = Counter(a + \" \" + b for a, b in zip(tw, tw[1:]))\n    print(f\"  target: {len(tw)} words, {len(tgt_uni)} uni, {len(tgt_bi)} bi \"\n          f\"({time.time()-t0:.0f}s)\", flush=True)\n\n    vocab_uni = {w: i for i, (w, _) in enumerate(\n        [x for x in tgt_uni.most_common(N_UNI) if x[1] >= 2])}\n    vocab_bi = {w: i for i, (w, _) in enumerate(\n        [x for x in tgt_bi.most_common(N_BI) if x[1] >= 2])}\n    VU, VB = len(vocab_uni), len(vocab_bi)\n    print(f\"  vocab kept: {VU} uni, {VB} bi\", flush=True)\n\n    tgt_uc = np.zeros(VU); tgt_bc = np.zeros(VB)\n    for w, i in vocab_uni.items():\n        tgt_uc[i] = tgt_uni[w]\n    for w, i in vocab_bi.items():\n        tgt_bc[i] = tgt_bi[w]\n\n    # ---- background distribution (random pool sample) ----\n    bg_uc = np.zeros(VU); bg_bc = np.zeros(VB)\n    bg_idx = [j for j in range(N) if random.random() < BG_SAMPLE_FRAC]\n    for j in bg_idx:\n        w = words(texts[j])\n        for x in w:\n            i = vocab_uni.get(x)\n            if i is not None:\n                bg_uc[i] += 1\n        for a, b in zip(w, w[1:]):\n            i = vocab_bi.get(a + \" \" + b)\n            if i is not None:\n                bg_bc[i] += 1\n    print(f\"  background from {len(bg_idx)} docs ({time.time()-t0:.0f}s)\", flush=True)\n\n    Tu, Bu = tgt_uc.sum(), bg_uc.sum()\n    Tb, Bb = tgt_bc.sum(), bg_bc.sum()\n    wu = np.log((tgt_uc + ALPHA) / (Tu + ALPHA * VU)) - \\\n         np.log((bg_uc + ALPHA) / (Bu + ALPHA * VU))\n    wb = np.log((tgt_bc + ALPHA) / (Tb + ALPHA * VB)) - \\\n         np.log((bg_bc + ALPHA) / (Bb + ALPHA * VB))\n\n    # ---- score every doc ----\n    scores = np.full(N, -1e9)\n    seen = set()\n    kept = 0\n    for j in range(N):\n        if DEDUP:\n            h = hash(texts[j])\n            if h in seen:\n                continue\n            seen.add(h)\n        ok, w = passes(texts[j])\n        if not ok:\n            continue\n        kept += 1\n        su = 0.0\n        for x in w:\n            i = vocab_uni.get(x)\n            if i is not None:\n                su += wu[i]\n        sb = 0.0\n        for a, b in zip(w, w[1:]):\n            i = vocab_bi.get(a + \" \" + b)\n            if i is not None:\n                sb += wb[i]\n        scores[j] = (su + 0.5 * sb) / len(w)\n    print(f\"  scored; {kept} docs pass ({time.time()-t0:.0f}s)\", flush=True)\n\n    valid = scores > -1e8\n    order = [j for j in np.argsort(-scores) if valid[j]]\n    sel = [int(ids[j]) for j in order[:N_OUT]]\n    json.dump(sel, open(OUT, \"w\"))\n    print(f\"wrote {len(sel)} ids -> {OUT} ({time.time()-t0:.0f}s)\", flush=True)\n    print(\"score quantiles:\", np.round(np.quantile(scores[valid], [0, .5, .9, 1]), 3))\n\n\nif __name__ == \"__main__\":\n    main()\n", "structuredPatch": [{"oldStart": 39, "oldLines": 7, "newStart": 39, "newLines": 10, "lines": [" ALPHA = 1.0             # additive smoothing", " N_OUT = 40_000          # ids to emit (>> enough to cover 12M tokens)", " SEED = 0", "-DEDUP = True            # drop exact-duplicate documents", "+DEDUP = False           # keep exact duplicates: duplicated high-LLR on-target docs", "+                        # act as an implicit up-weighting of the best content under", "+                        # the trainer's random-window sampling (measured: dedup RAISED", "+                        # dev perplexity 335.6 -> 347.4, so it is disabled).", " ", " MIN_WORDS = 40", " MIN_ALPHA_RATIO = 0.55"]}], "userModified": false, "replaceAll": false}

Command running in background with ID: bajhvseto. Output is being written to: /tmp/claude-0/-workspace/2eda86eb-7dd5-4994-9ab5-ba9a2e6aa267/tasks/bajhvseto.output. You will be notified when it completes. To check interim output, use Read on that file path.
Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "bajhvseto", "backgroundCwdHint": "Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands."}

wrote 40000 ids -> /workspace/submission/selection.json (141s)
score quantiles: [-1.556 -0.314 -0.064  0.707]
--- compare to original v1 selection (335.6) ---
identical to v1 selection: True
len 40000 unique 40000

[stdout]
wrote 40000 ids -> /workspace/submission/selection.json (141s)
score quantiles: [-1.556 -0.314 -0.064  0.707]
--- compare to original v1 selection (335.6) ---
identical to v1 selection: True
len 40000 unique 40000
[metadata] {"noOutputExpected": false}

Command running in background with ID: b60814acf. Output is being written to: /tmp/claude-0/-workspace/2eda86eb-7dd5-4994-9ab5-ba9a2e6aa267/tasks/b60814acf.output. You will be notified when it completes. To check interim output, use Read on that file path.
Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "b60814acf", "backgroundCwdHint": "Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands."}

Task #2 created successfully: Write claim.md (hypothesis/mechanism/falsification/transfer)

[metadata] {"task": {"id": "2", "subject": "Write claim.md (hypothesis/mechanism/falsification/transfer)"}}

34:OUT = "/tmp/sel_worst.json"
40:N_OUT = 60_000          # ids to emit (>> enough to cover 12M tokens)
161:    order = [j for j in np.argsort(scores) if valid[j]]  # ASCENDING: worst LLR first

[stdout]
34:OUT = "/tmp/sel_worst.json"
40:N_OUT = 60_000          # ids to emit (>> enough to cover 12M tokens)
161:    order = [j for j in np.argsort(scores) if valid[j]]  # ASCENDING: worst LLR first
[metadata] {"noOutputExpected": false}

Command running in background with ID: bowr4rz8b. Output is being written to: /tmp/claude-0/-workspace/2eda86eb-7dd5-4994-9ab5-ba9a2e6aa267/tasks/bowr4rz8b.output. You will be notified when it completes. To check interim output, use Read on that file path.
Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "bowr4rz8b", "backgroundCwdHint": "Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands."}