claude-opus-4-8

bir04 : HELDOUT_PPL 428.102  loss 6.0594  train_s 55  /tmp/dev_train.npy
---timer---
3546

[stdout]
bir04 : HELDOUT_PPL 428.102  loss 6.0594  train_s 55  /tmp/dev_train.npy
---timer---
3546
[metadata] {"noOutputExpected": false}

bir05 : HELDOUT_PPL 437.328  loss 6.0807  train_s 55  /tmp/dev_train.npy
ALLDONE

[stdout]
bir05 : HELDOUT_PPL 437.328  loss 6.0807  train_s 55  /tmp/dev_train.npy
ALLDONE
[metadata] {"noOutputExpected": false}

Command running in background with ID: btc3jmi26. Output is being written to: /tmp/claude-0/-workspace/15a2b8ef-4272-42e8-a818-f020ab1f9107/tasks/btc3jmi26.output. You will be notified when it completes. To check interim output, use Read on that file path.
Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "btc3jmi26", "backgroundCwdHint": "Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands."}

The file /workspace/submission/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"type": "update", "filePath": "/workspace/submission/curate.py", "content": "\"\"\"Curate the raw web pool to train a small LM for a BROAD, multi-domain\nhigh-quality English target (Wikipedia + high-quality web prose + news +\ntechnical Q&A).\n\nCRITERION (stated, reproducible — DSIR-style distribution matching,\narXiv:2302.03169):\n\n  1. VALIDITY GATE. Drop documents that are not usable English prose: non-Latin /\n     non-ASCII heavy, boilerplate / navigation (low alpha-word ratio, degenerate\n     mean word length), highly repetitive (low unique-word ratio), digit-dominated\n     (number/price/id lists), or too short. These are exactly the artifacts that\n     fool a naive relevance scorer.\n\n  2. DOMAIN SCORE. For every surviving document compute a per-token\n     Naive-Bayes log-likelihood ratio between the disclosed target domain and the\n     raw-pool background, using UNIGRAM word features:\n\n         score(doc) = mean_{w in doc} [ log P_target(w) - log P_pool(w) ]\n\n     P_target is estimated from the decoded dev target; P_pool from a random pool\n     sample. (Empirically, adding bigram features to this ratio HURT held-out\n     perplexity — unigram vocabulary matching is the robust signal — and ranking\n     the raw score top-k COLLAPSES diversity and also hurts. Both were measured.)\n\n  3. IMPORTANCE RESAMPLING (not top-k). Select via Gumbel-perturbed keys\n         key(doc) = z(doc)/T + Gumbel\n     where z is the standardized domain score and T is a temperature. This shifts\n     the SELECTED SET's word distribution toward the target while PRESERVING\n     within-register diversity (essential for a broad target: pure top-k selection\n     of the \"most target-like\" docs was measured to be worse than random). T was\n     tuned on the dev target; the interior optimum is ~0.4.\n\nWe emit pool ids in descending key order (priority order) covering well beyond\nthe 12M-token budget.\n\"\"\"\nimport json, re, sys\nimport numpy as np\nfrom math import log\n\nPOOL = \"/workspace/data/pool.jsonl\"\nDEV  = \"/workspace/data/multi_dev.npy\"\nOUT  = \"/workspace/submission/selection.json\"\n\nSEED        = 1337\nBG_SAMPLE   = 25000\nWORD_CAP    = 1200\nSMOOTH      = 2.0\nMIN_WORDS   = 25\nT           = 0.40          # resampling temperature (tuned on dev)\nTARGET_TOK  = 20_000_000    # emit ids well beyond the 12M budget\nCHARS_PER_TOK = 4.0\n\nWORD_RE = re.compile(r\"[a-z0-9]+\")\nALPHA_RE = re.compile(r\"[A-Za-z]+\")\nALL_RE   = re.compile(r\"\\S+\")\n\n\ndef gate(text):\n    \"\"\"Return True if the document is usable English prose.\"\"\"\n    n = len(text)\n    if n < 200:\n        return False\n    ascii_ok = ascii_letters = letters = digits = 0\n    for c in text:\n        o = ord(c)\n        if o < 128:\n            ascii_ok += 1\n            if c.isalpha():\n                ascii_letters += 1; letters += 1\n        elif c.isalpha():\n            letters += 1\n        if c.isdigit():\n            digits += 1\n    if letters == 0:\n        return False\n    if ascii_ok / n < 0.85:            # non-ascii heavy\n        return False\n    if 1 - ascii_letters / letters > 0.15:   # non-Latin script heavy\n        return False\n    toks = ALL_RE.findall(text)\n    if len(toks) < 40:\n        return False\n    words = ALPHA_RE.findall(text)\n    if len(words) / len(toks) < 0.6:   # boilerplate / symbol soup\n        return False\n    mwl = sum(len(w) for w in words) / len(words)\n    if mwl < 3.2 or mwl > 9:           # menu tokens / gibberish\n        return False\n    if len(set(w.lower() for w in words)) / len(words) < 0.28:  # repetitive\n        return False\n    if digits / n > 0.15:              # number / price / id lists\n        return False\n    return True\n\n\ndef main():\n    rng = np.random.default_rng(SEED)\n\n    # ---- target unigram counts from the disclosed dev sample ----\n    from transformers import AutoTokenizer\n    tok = AutoTokenizer.from_pretrained(\"gpt2\")\n    dev = np.load(DEV)\n    tgt_docs = [d for d in tok.decode(dev.tolist()).split(\"<|endoftext|>\") if d.strip()]\n    print(f\"target docs: {len(tgt_docs)}\", file=sys.stderr)\n\n    tc = {}; tt = 0\n    for d in tgt_docs:\n        for w in WORD_RE.findall(d.lower()):\n            tc[w] = tc.get(w, 0) + 1; tt += 1\n\n    # ---- load pool ----\n    ids = []; texts = []\n    for line in open(POOL):\n        r = json.loads(line); ids.append(r[\"id\"]); texts.append(r[\"text\"])\n    ids = np.array(ids); N = len(ids)\n    print(f\"pool docs: {N}\", file=sys.stderr)\n\n    # ---- background unigram counts from a random pool sample ----\n    bc = {}; bt = 0\n    for j in rng.choice(N, size=min(BG_SAMPLE, N), replace=False):\n        w = WORD_RE.findall(texts[j].lower())\n        for x in (w[:WORD_CAP] if len(w) > WORD_CAP else w):\n            bc[x] = bc.get(x, 0) + 1; bt += 1\n\n    # ---- NB log-likelihood-ratio weights per word ----\n    V = len(set(tc) | set(bc))\n    dt = tt + SMOOTH * V; db = bt + SMOOTH * V\n    weight = {}\n    for f in set(tc) | set(bc):\n        weight[f] = log((tc.get(f, 0) + SMOOTH) / dt) - log((bc.get(f, 0) + SMOOTH) / db)\n    UNSEEN = log(SMOOTH / dt) - log(SMOOTH / db)\n    get = weight.get\n    print(f\"vocab: {len(weight)}\", file=sys.stderr)\n\n    # ---- gate + score every pool doc ----\n    scores = np.full(N, np.nan)\n    est_tok = np.empty(N, dtype=np.int64)\n    for k in range(N):\n        t = texts[k]\n        est_tok[k] = int(len(t) / CHARS_PER_TOK)\n        if not gate(t):\n            continue\n        w = WORD_RE.findall(t.lower())\n        if len(w) < MIN_WORDS:\n            continue\n        wc = w[:WORD_CAP] if len(w) > WORD_CAP else w\n        s = 0.0\n        for x in wc:\n            s += get(x, UNSEEN)\n        scores[k] = s / len(wc)\n        if k % 40000 == 0:\n            print(f\"scored {k}/{N}\", file=sys.stderr)\n\n    valid = ~np.isnan(scores)\n    idx = np.where(valid)[0]\n    print(f\"gated survivors: {len(idx)}\", file=sys.stderr)\n\n    # ---- DSIR importance resampling via Gumbel-perturbed keys ----\n    z = scores[idx]\n    z = (z - z.mean()) / (z.std() + 1e-9)\n    g = rng.gumbel(size=len(idx))\n    key = z / T + g\n    order = idx[np.argsort(-key)]\n\n    # ---- emit ids in priority order until we cover >> budget ----\n    sel = []; cum = 0\n    for k in order:\n        sel.append(int(ids[k])); cum += int(est_tok[k])\n        if cum >= TARGET_TOK:\n            break\n    print(f\"selected {len(sel)} ids, est tokens ~{cum}\", file=sys.stderr)\n    json.dump(sel, open(OUT, \"w\"))\n\n\nif __name__ == \"__main__\":\n    main()\n", "structuredPatch": [{"oldStart": 1, "oldLines": 23, "newStart": 1, "newLines": 37, "lines": ["-\"\"\"Curate the raw web pool for training a small LM whose target is a BROAD,", "-multi-domain high-quality English distribution (Wikipedia + high-quality web", "-prose + news + technical Q&A).", "+\"\"\"Curate the raw web pool to train a small LM for a BROAD, multi-domain", "+high-quality English target (Wikipedia + high-quality web prose + news +", "+technical Q&A).", " ", "-Criterion (DSIR-style domain matching, arXiv:2302.03169, plus a light validity", "-filter): score every pool document by how much its word distribution looks like", "-the *disclosed target domain* relative to the raw-pool background, using a", "-Naive-Bayes log-likelihood-ratio linear scorer over word uni+bigram features:", "+CRITERION (stated, reproducible — DSIR-style distribution matching,", "+arXiv:2302.03169):", " ", "-    weight[f] = log( (P_target[f] + smoothing) / (P_pool[f] + smoothing) )", "-    score(doc) = mean_{f in doc} weight[f]     (out-of-vocab f -> weight 0)", "+  1. VALIDITY GATE. Drop documents that are not usable English prose: non-Latin /", "+     non-ASCII heavy, boilerplate / navigation (low alpha-word ratio, degenerate", "+     mean word length), highly repetitive (low unique-word ratio), digit-dominated", "+     (number/price/id lists), or too short. These are exactly the artifacts that", "+     fool a naive relevance scorer.", " ", "-Positives = the decoded dev target (its own text, split into per-register docs).", "-Background = a random sample of the raw pool. Documents whose vocabulary/phrasing", "-resembles ANY of the four target registers (encyclopedic, high-quality web,", "-news, technical Q&A) score high; forum shorthand, SEO/boilerplate and garbage", "-score low. We emit pool ids in descending score order (priority order), after", "-dropping trivially-invalid docs, covering well over the 12M-token budget.", "+  2. DOMAIN SCORE. For every surviving document compute a per-token", "+     Naive-Bayes log-likelihood ratio between the disclosed target domain and the", "+     raw-pool background, using UNIGRAM word features:", " ", "-This is a stated, reproducible criterion — not a hand-picked id list.", "+         score(doc) = mean_{w in doc} [ log P_target(w) - log P_pool(w) ]", "+", "+     P_target is estimated from the decoded dev target; P_pool from a random pool", "+     sample. (Empirically, adding bigram features to this ratio HURT held-out", "+     perplexity — unigram vocabulary matching is the robust signal — and ranking", "+     the raw score top-k COLLAPSES diversity and also hurts. Both were measured.)", "+", "+  3. IMPORTANCE RESAMPLING (not top-k). Select via Gumbel-perturbed keys", "+         key(doc) = z(doc)/T + Gumbel", "+     where z is the standardized domain score and T is a temperature. This shifts", "+     the SELECTED SET's word distribution toward the target while PRESERVING", "+     within-register diversity (essential for a broad target: pure top-k selection", "+     of the \"most target-like\" docs was measured to be worse than random). T was", "+     tuned on the dev target; the interior optimum is ~0.4.", "+", "+We emit pool ids in descending key order (priority order) covering well beyond", "+the 12M-token budget.", " \"\"\"", " import json, re, sys", " import numpy as np"]}, {"oldStart": 27, "oldLines": 109, "newStart": 41, "newLines": 136, "lines": [" DEV  = \"/workspace/data/multi_dev.npy\"", " OUT  = \"/workspace/submission/selection.json\"", " ", "-SMOOTH     = 2.0          # additive smoothing (pseudo-count) on feature probs", "-VOCAB_MIN  = 2            # keep features seen >= this many times overall", "-BG_SAMPLE  = 25000        # background docs sampled from pool", "-WORD_CAP   = 1200         # cap words scored per doc (domain signal saturates)", "-MIN_WORDS  = 25           # drop trivially short docs", "-TARGET_TOK = 20_000_000   # emit enough ids to well exceed the 12M budget", "-CHARS_PER_TOK = 4.0       # rough token estimate for coverage only", "+SEED        = 1337", "+BG_SAMPLE   = 25000", "+WORD_CAP    = 1200", "+SMOOTH      = 2.0", "+MIN_WORDS   = 25", "+T           = 0.40          # resampling temperature (tuned on dev)", "+TARGET_TOK  = 20_000_000    # emit ids well beyond the 12M budget", "+CHARS_PER_TOK = 4.0", " ", " WORD_RE = re.compile(r\"[a-z0-9]+\")", "+ALPHA_RE = re.compile(r\"[A-Za-z]+\")", "+ALL_RE   = re.compile(r\"\\S+\")", " ", "-def tokens(text, cap=WORD_CAP):", "-    w = WORD_RE.findall(text.lower())", "-    return w[:cap] if len(w) > cap else w", " ", "-def add_counts(words, d):", "-    for t in words:", "-        d[t] = d.get(t, 0) + 1", "-    for i in range(len(words) - 1):", "-        b = words[i] + \" \" + words[i + 1]", "-        d[b] = d.get(b, 0) + 1", "+def gate(text):", "+    \"\"\"Return True if the document is usable English prose.\"\"\"", "+    n = len(text)", "+    if n < 200:", "+        return False", "+    ascii_ok = ascii_letters = letters = digits = 0", "+    for c in text:", "+        o = ord(c)", "+        if o < 128:", "+            ascii_ok += 1", "+            if c.isalpha():", "+                ascii_letters += 1; letters += 1", "+        elif c.isalpha():", "+            letters += 1", "+        if c.isdigit():", "+            digits += 1", "+    if letters == 0:", "+        return False", "+    if ascii_ok / n < 0.85:            # non-ascii heavy", "+        return False", "+    if 1 - ascii_letters / letters > 0.15:   # non-Latin script heavy", "+        return False", "+    toks = ALL_RE.findall(text)", "+    if len(toks) < 40:", "+        return False", "+    words = ALPHA_RE.findall(text)", "+    if len(words) / len(toks) < 0.6:   # boilerplate / symbol soup", "+        return False", "+    mwl = sum(len(w) for w in words) / len(words)", "+    if mwl < 3.2 or mwl > 9:           # menu tokens / gibberish", "+        return False", "+    if len(set(w.lower() for w in words)) / len(words) < 0.28:  # repetitive", "+        return False", "+    if digits / n > 0.15:              # number / price / id lists", "+        return False", "+    return True", " ", "+", " def main():", "-    rng = np.random.default_rng(1337)", "+    rng = np.random.default_rng(SEED)", " ", "-    # ---- target counts from disclosed dev sample ----", "+    # ---- target unigram counts from the disclosed dev sample ----", "     from transformers import AutoTokenizer", "     tok = AutoTokenizer.from_pretrained(\"gpt2\")", "     dev = np.load(DEV)", "-    dev_text = tok.decode(dev.tolist())", "-    tgt_docs = [d for d in dev_text.split(\"<|endoftext|>\") if d.strip()]", "+    tgt_docs = [d for d in tok.decode(dev.tolist()).split(\"<|endoftext|>\") if d.strip()]", "     print(f\"target docs: {len(tgt_docs)}\", file=sys.stderr)", " ", "-    tc = {}", "-    tt = 0", "+    tc = {}; tt = 0", "     for d in tgt_docs:", "-        w = tokens(d)", "-        add_counts(w, tc)", "-        tt += 2 * len(w) - 1 if len(w) else 0", "+        for w in WORD_RE.findall(d.lower()):", "+            tc[w] = tc.get(w, 0) + 1; tt += 1", " ", "     # ---- load pool ----", "-    ids, texts = [], []", "+    ids = []; texts = []", "     for line in open(POOL):", "-        r = json.loads(line)", "-        ids.append(r[\"id\"]); texts.append(r[\"text\"])", "-    N = len(ids)", "+        r = json.loads(line); ids.append(r[\"id\"]); texts.append(r[\"text\"])", "+    ids = np.array(ids); N = len(ids)", "     print(f\"pool docs: {N}\", file=sys.stderr)", " ", "-    # ---- background counts from random pool sample ----", "-    bc = {}", "-    bt = 0", "+    # ---- background unigram counts from a random pool sample ----", "+    bc = {}; bt = 0", "     for j in rng.choice(N, size=min(BG_SAMPLE, N), replace=False):", "-        w = tokens(texts[j])", "-        add_counts(w, bc)", "-        bt += 2 * len(w) - 1 if len(w) else 0", "+        w = WORD_RE.findall(texts[j].lower())", "+        for x in (w[:WORD_CAP] if len(w) > WORD_CAP else w):", "+            bc[x] = bc.get(x, 0) + 1; bt += 1", " ", "-    # ---- NB log-likelihood-ratio weights (only for features in target vocab) ----", "-    # A feature absent from target gets negative weight via smoothing; we only need", "-    # to store weights for target features (others contribute ~log(smoothing/...) ~", "-    # small & roughly constant, so treat out-of-vocab as neutral 0).", "+    # ---- NB log-likelihood-ratio weights per word ----", "     V = len(set(tc) | set(bc))", "-    denom_t = tt + SMOOTH * V", "-    denom_b = bt + SMOOTH * V", "+    dt = tt + SMOOTH * V; db = bt + SMOOTH * V", "     weight = {}", "-    for f, c in tc.items():", "-        if c < VOCAB_MIN:", "-            continue", "-        pt = (c + SMOOTH) / denom_t", "-        pb = (bc.get(f, 0) + SMOOTH) / denom_b", "-        weight[f] = log(pt) - log(pb)", "-    print(f\"vocab weights: {len(weight)}\", file=sys.stderr)", "+    for f in set(tc) | set(bc):", "+        weight[f] = log((tc.get(f, 0) + SMOOTH) / dt) - log((bc.get(f, 0) + SMOOTH) / db)", "+    UNSEEN = log(SMOOTH / dt) - log(SMOOTH / db)", "     get = weight.get", "+    print(f\"vocab: {len(weight)}\", file=sys.stderr)", " ", "-    # ---- score every pool doc: mean feature weight (domain-match density) ----", "-    scores = np.full(N, -1e9, dtype=np.float64)", "+    # ---- gate + score every pool doc ----", "+    scores = np.full(N, np.nan)", "     est_tok = np.empty(N, dtype=np.int64)", "     for k in range(N):", "         t = texts[k]", "         est_tok[k] = int(len(t) / CHARS_PER_TOK)", "-        w = tokens(t)", "+        if not gate(t):", "+            continue", "+        w = WORD_RE.findall(t.lower())", "         if len(w) < MIN_WORDS:", "             continue", "+        wc = w[:WORD_CAP] if len(w) > WORD_CAP else w", "         s = 0.0", "-        for x in w:", "-            s += get(x, 0.0)", "-        for i in range(len(w) - 1):", "-            s += get(w[i] + \" \" + w[i + 1], 0.0)", "-        scores[k] = s / (2 * len(w) - 1)", "+        for x in wc:", "+            s += get(x, UNSEEN)", "+        scores[k] = s / len(wc)", "         if k % 40000 == 0:", "             print(f\"scored {k}/{N}\", file=sys.stderr)", " ", "-    order = np.argsort(-scores)", "+    valid = ~np.isnan(scores)", "+    idx = np.where(valid)[0]", "+    print(f\"gated survivors: {len(idx)}\", file=sys.stderr)", " ", "+    # ---- DSIR importance resampling via Gumbel-perturbed keys ----", "+    z = scores[idx]", "+    z = (z - z.mean()) / (z.std() + 1e-9)", "+    g = rng.gumbel(size=len(idx))", "+    key = z / T + g", "+    order = idx[np.argsort(-key)]", "+", "     # ---- emit ids in priority order until we cover >> budget ----", "-    sel, cum = [], 0", "+    sel = []; cum = 0", "     for k in order:", "-        if scores[k] <= -1e8:", "-            continue", "-        sel.append(int(ids[k]))", "-        cum += int(est_tok[k])", "+        sel.append(int(ids[k])); cum += int(est_tok[k])", "         if cum >= TARGET_TOK:", "             break", "     print(f\"selected {len(sel)} ids, est tokens ~{cum}\", file=sys.stderr)", "     json.dump(sel, open(OUT, \"w\"))", " ", "+", " if __name__ == \"__main__\":", "     main()"]}], "originalFile": "\"\"\"Curate the raw web pool for training a small LM whose target is a BROAD,\nmulti-domain high-quality English distribution (Wikipedia + high-quality web\nprose + news + technical Q&A).\n\nCriterion (DSIR-style domain matching, arXiv:2302.03169, plus a light validity\nfilter): score every pool document by how much its word distribution looks like\nthe *disclosed target domain* relative to the raw-pool background, using a\nNaive-Bayes log-likelihood-ratio linear scorer over word uni+bigram features:\n\n    weight[f] = log( (P_target[f] + smoothing) / (P_pool[f] + smoothing) )\n    score(doc) = mean_{f in doc} weight[f]     (out-of-vocab f -> weight 0)\n\nPositives = the decoded dev target (its own text, split into per-register docs).\nBackground = a random sample of the raw pool. Documents whose vocabulary/phrasing\nresembles ANY of the four target registers (encyclopedic, high-quality web,\nnews, technical Q&A) score high; forum shorthand, SEO/boilerplate and garbage\nscore low. We emit pool ids in descending score order (priority order), after\ndropping trivially-invalid docs, covering well over the 12M-token budget.\n\nThis is a stated, reproducible criterion — not a hand-picked id list.\n\"\"\"\nimport json, re, sys\nimport numpy as np\nfrom math import log\n\nPOOL = \"/workspace/data/pool.jsonl\"\nDEV  = \"/workspace/data/multi_dev.npy\"\nOUT  = \"/workspace/submission/selection.json\"\n\nSMOOTH     = 2.0          # additive smoothing (pseudo-count) on feature probs\nVOCAB_MIN  = 2            # keep features seen >= this many times overall\nBG_SAMPLE  = 25000        # background docs sampled from pool\nWORD_CAP   = 1200         # cap words scored per doc (domain signal saturates)\nMIN_WORDS  = 25           # drop trivially short docs\nTARGET_TOK = 20_000_000   # emit enough ids to well exceed the 12M budget\nCHARS_PER_TOK = 4.0       # rough token estimate for coverage only\n\nWORD_RE = re.compile(r\"[a-z0-9]+\")\n\ndef tokens(text, cap=WORD_CAP):\n    w = WORD_RE.findall(text.lower())\n    return w[:cap] if len(w) > cap else w\n\ndef add_counts(words, d):\n    for t in words:\n        d[t] = d.get(t, 0) + 1\n    for i in range(len(words) - 1):\n        b = words[i] + \" \" + words[i + 1]\n        d[b] = d.get(b, 0) + 1\n\ndef main():\n    rng = np.random.default_rng(1337)\n\n    # ---- target counts from disclosed dev sample ----\n    from transformers import AutoTokenizer\n    tok = AutoTokenizer.from_pretrained(\"gpt2\")\n    dev = np.load(DEV)\n    dev_text = tok.decode(dev.tolist())\n    tgt_docs = [d for d in dev_text.split(\"<|endoftext|>\") if d.strip()]\n    print(f\"target docs: {len(tgt_docs)}\", file=sys.stderr)\n\n    tc = {}\n    tt = 0\n    for d in tgt_docs:\n        w = tokens(d)\n        add_counts(w, tc)\n        tt += 2 * len(w) - 1 if len(w) else 0\n\n    # ---- load pool ----\n    ids, texts = [], []\n    for line in open(POOL):\n        r = json.loads(line)\n        ids.append(r[\"id\"]); texts.append(r[\"text\"])\n    N = len(ids)\n    print(f\"pool docs: {N}\", file=sys.stderr)\n\n    # ---- background counts from random pool sample ----\n    bc = {}\n    bt = 0\n    for j in rng.choice(N, size=min(BG_SAMPLE, N), replace=False):\n        w = tokens(texts[j])\n        add_counts(w, bc)\n        bt += 2 * len(w) - 1 if len(w) else 0\n\n    # ---- NB log-likelihood-ratio weights (only for features in target vocab) ----\n    # A feature absent from target gets negative weight via smoothing; we only need\n    # to store weights for target features (others contribute ~log(smoothing/...) ~\n    # small & roughly constant, so treat out-of-vocab as neutral 0).\n    V = len(set(tc) | set(bc))\n    denom_t = tt + SMOOTH * V\n    denom_b = bt + SMOOTH * V\n    weight = {}\n    for f, c in tc.items():\n        if c < VOCAB_MIN:\n            continue\n        pt = (c + SMOOTH) / denom_t\n        pb = (bc.get(f, 0) + SMOOTH) / denom_b\n        weight[f] = log(pt) - log(pb)\n    print(f\"vocab weights: {len(weight)}\", file=sys.stderr)\n    get = weight.get\n\n    # ---- score every pool doc: mean feature weight (domain-match density) ----\n    scores = np.full(N, -1e9, dtype=np.float64)\n    est_tok = np.empty(N, dtype=np.int64)\n    for k in range(N):\n        t = texts[k]\n        est_tok[k] = int(len(t) / CHARS_PER_TOK)\n        w = tokens(t)\n        if len(w) < MIN_WORDS:\n            continue\n        s = 0.0\n        for x in w:\n            s += get(x, 0.0)\n        for i in range(len(w) - 1):\n            s += get(w[i] + \" \" + w[i + 1], 0.0)\n        scores[k] = s / (2 * len(w) - 1)\n        if k % 40000 == 0:\n            print(f\"scored {k}/{N}\", file=sys.stderr)\n\n    order = np.argsort(-scores)\n\n    # ---- emit ids in priority order until we cover >> budget ----\n    sel, cum = [], 0\n    for k in order:\n        if scores[k] <= -1e8:\n            continue\n        sel.append(int(ids[k]))\n        cum += int(est_tok[k])\n        if cum >= TARGET_TOK:\n            break\n    print(f\"selected {len(sel)} ids, est tokens ~{cum}\", file=sys.stderr)\n    json.dump(sel, open(OUT, \"w\"))\n\nif __name__ == \"__main__\":\n    main()\n", "userModified": false}

T=035 : HELDOUT_PPL 373.125  loss 5.9219  train_s 55  /tmp/dev_train.npy
---timer---
3316

[stdout]
T=035 : HELDOUT_PPL 373.125  loss 5.9219  train_s 55  /tmp/dev_train.npy
---timer---
3316
[metadata] {"noOutputExpected": false}

T=045 : HELDOUT_PPL 369.556  loss 5.9123  train_s 55  /tmp/dev_train.npy
ALLDONE

[stdout]
T=045 : HELDOUT_PPL 369.556  loss 5.9123  train_s 55  /tmp/dev_train.npy
ALLDONE
[metadata] {"noOutputExpected": false}

Updated task #2 status

[metadata] {"success": true, "taskId": "2", "updatedFields": ["status"], "statusChange": {"from": "in_progress", "to": "completed"}}

Updated task #3 status

[metadata] {"success": true, "taskId": "3", "updatedFields": ["status"], "statusChange": {"from": "pending", "to": "in_progress"}}

target docs: 2705
pool docs: 182016
vocab: 294561
scored 0/182016
scored 40000/182016
scored 80000/182016
gated survivors: 169670
selected 19995 ids, est tokens ~20000020

real	2m42.779s
user	2m32.870s
sys	0m2.030s

[stdout]
target docs: 2705
pool docs: 182016
vocab: 294561
scored 0/182016
scored 40000/182016
scored 80000/182016
gated survivors: 169670
selected 19995 ids, est tokens ~20000020

real	2m42.779s
user	2m32.870s
sys	0m2.030s
[metadata] {"noOutputExpected": false}

Command running in background with ID: b7b1nhjbw. Output is being written to: /tmp/claude-0/-workspace/15a2b8ef-4272-42e8-a818-f020ab1f9107/tasks/b7b1nhjbw.output. You will be notified when it completes. To check interim output, use Read on that file path.
Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "b7b1nhjbw", "backgroundCwdHint": "Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands."}

Token indices sequence length is longer than the specified maximum sequence length for this model (31886 > 1024). Running this sequence through the model will result in indexing errors
packed 12000000 tokens from 19995 ids -> /tmp/dev_train.npy
HELDOUT_PPL 365.919  loss 5.9024  train_s 55  /tmp/dev_train.npy
(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)
DONE

[stdout]
Token indices sequence length is longer than the specified maximum sequence length for this model (31886 > 1024). Running this sequence through the model will result in indexing errors
packed 12000000 tokens from 19995 ids -> /tmp/dev_train.npy
HELDOUT_PPL 365.919  loss 5.9024  train_s 55  /tmp/dev_train.npy
(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)
DONE
[metadata] {"noOutputExpected": false}

File created successfully at: /tmp/kl.py (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/tmp/kl.py", "content": "import json, re, numpy as np\nfrom math import log\nWORD_RE=re.compile(r\"[a-z0-9]+\")\nfrom transformers import AutoTokenizer\ntok=AutoTokenizer.from_pretrained(\"gpt2\")\ndev=np.load(\"/workspace/data/multi_dev.npy\")\ntgt_text=tok.decode(dev.tolist())\ndef dist(words):\n    d={}\n    for w in words: d[w]=d.get(w,0)+1\n    return d\ntgt=dist(WORD_RE.findall(tgt_text.lower()))\nTtot=sum(tgt.values())\ntext={}\nfor line in open(\"/workspace/data/pool.jsonl\"):\n    r=json.loads(line); text[r[\"id\"]]=r[\"text\"]\ndef sel_dist(path,cap_tokens=12_000_000):\n    sel=json.load(open(path)); d={}; approx=0\n    for i in sel:\n        for w in WORD_RE.findall(text[i].lower()):\n            d[w]=d.get(w,0)+1; approx+=1\n        if approx>=cap_tokens*0.75:  # ~12M gpt2 tok ~ 0.75x words\n            break\n    return d\ndef kl_and_cov(sd):\n    Stot=sum(sd.values()); V=len(set(tgt)|set(sd)); SM=0.1\n    dt=Ttot+SM*V; ds=Stot+SM*V\n    kl=0.0; cov=0\n    for w,c in tgt.items():\n        pt=c/Ttot\n        ps=(sd.get(w,0)+SM)/ds\n        kl+=pt*log(pt/ps)\n        if sd.get(w,0)>0: cov+=1\n    return kl, cov/len(tgt)\nfor name,path in [(\"random\",\"/tmp/random_sel.json\"),\n                  (\"fluency_max\",\"/tmp/sel_tlm.json\"),\n                  (\"my_selection\",\"/workspace/submission/selection.json\")]:\n    sd=sel_dist(path)\n    kl,cov=kl_and_cov(sd)\n    print(f\"{name:14s} KL(target||sel)={kl:.4f}  target-vocab-coverage={cov:.4f}\")\n", "structuredPatch": [], "originalFile": null, "userModified": false}

random         KL(target||sel)=0.5584  target-vocab-coverage=0.7804
fluency_max    KL(target||sel)=0.6588  target-vocab-coverage=0.6950
my_selection   KL(target||sel)=0.4619  target-vocab-coverage=0.8270

[stdout]
random         KL(target||sel)=0.5584  target-vocab-coverage=0.7804
fluency_max    KL(target||sel)=0.6588  target-vocab-coverage=0.6950
my_selection   KL(target||sel)=0.4619  target-vocab-coverage=0.8270
[metadata] {"noOutputExpected": false}

File created successfully at: /workspace/claim.md (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/workspace/claim.md", "content": "# Claim: distribution-matching resampling beats quality-ranking for a broad target\n\n## Hypothesis\nFor a **broad, multi-register** high-quality target, held-out perplexity of a\nfrom-scratch small LM is minimized by selecting a training set whose **word\ndistribution matches the target's**, *while preserving within-register\ndiversity* — **not** by ranking documents by \"quality\"/target-likeness and taking\nthe top. Concretely: (1) drop unusable non-prose (non-English, boilerplate,\nrepetitive, digit-list) documents, then (2) **importance-resample** the rest\ntoward the disclosed target with a temperature that keeps diversity (DSIR-style,\narXiv:2302.03169). This beats both the random baseline and greedy quality\nranking.\n\nMeasured on the frozen 30M-GPT / 12M-token dev pipeline:\nrandom **458** → my selection **366** (−20%). Greedy \"most-target-like\" top-k is\n**610 (worse than random)**.\n\n## Mechanism (predicts an observable *other than* final perplexity)\nThe eval loss is a cross-entropy against the target's token distribution, so what\nmatters is how close the *training* unigram distribution is to the *target*\nunigram distribution — specifically whether the training data covers the target's\n**mid-frequency, informative vocabulary** (named entities, technical terms, code\ntokens), not just high-frequency function words.\n\n**Prediction, computable with no training:** rank selections by\n`KL(P_target ‖ P_selection)` over words and by target-vocabulary coverage; this\nordering will match the held-out-perplexity ordering. Greedy top-k on a fluency /\nlikelihood score maximizes high-frequency-word density, which *raises* KL and\n*lowers* target-vocab coverage — so it must do **worse**, despite selecting\n\"cleaner\" text.\n\nConfirmed (no training used to compute these):\n\n| selection            | KL(target‖sel) ↓ | target-vocab coverage ↑ | held-out PPL ↓ |\n|----------------------|:---:|:---:|:---:|\n| random               | 0.558 | 0.780 | 458 |\n| fluency-max (top-k)  | 0.659 | 0.695 | 610 |\n| **mine (DSIR T=0.4)**| **0.462** | **0.827** | **366** |\n\nKL is monotone with PPL; the top-k \"quality\" set is the *worst* on both the\nobservable and PPL. A second observable: the tilt temperature has an **interior\noptimum** (T≈0.4 beats both T→0, top-k collapse, and T→∞, random), i.e. the win\nis a match–diversity trade-off, not merely \"avoid junk.\"\n\n## Falsification\nThe hypothesis is false if any of:\n- A selection with **lower** unigram `KL(target‖sel)` yields **higher** held-out\n  PPL than one with higher KL (mechanism decoupled from score). Not observed.\n- Greedy top-k by the domain score **beats** resampling at equal budget (diversity\n  irrelevant). Observed opposite: top-k 610 vs resample 366.\n- PPL is **monotone decreasing in diversity** (temperature), i.e. plain random\n  ties or beats the matched selection. Observed opposite: interior optimum at\n  T≈0.4 (random 458, T=0.15→369, T=0.4→366, T=0.6→375).\n- Adding richer features (bigrams) to the ratio, which sharpens\n  \"target-likeness,\" should help if the driver is relevance rather than\n  distribution match. Observed opposite: uni+bigram ratio *hurt* (428/437),\n  because it over-tilts toward distinctive phrasing and loses coverage.\n\n## Transfer\n- **New target domain:** swap the target sample fed to the scorer; the same\n  gate + unigram-ratio resample selects for that domain. No retraining of the\n  pipeline.\n- **Larger budget / bigger model:** the criterion is budget-agnostic (emit ids\n  in priority order; the trainer consumes to its budget). Distribution matching\n  should help more as the pool/target mismatch dominates.\n- **General principle:** for any *broad* held-out target, prefer\n  distribution-matching resampling over quality top-k ranking; report\n  `KL(target‖selection)` and target-vocab coverage as cheap, training-free proxies\n  to tune selection before spending a training run.\n", "structuredPatch": [], "originalFile": null, "userModified": false}

n ids: 19995
unique: True
in range: True
all ints: True
--- files ---
-rw-r--r-- 1 root root 3913 Aug  4 05:21 claim.md

submission/:
total 148
drwxr-xr-x 1 root root    140 Aug  4 05:12 .
drwxr-xr-x 1 root root    180 Aug  4 05:21 ..
-rw-r--r-- 1 root root   6305 Aug  4 05:12 curate.py
-rw-r--r-- 1 root root 144005 Aug  4 05:18 selection.json
--- pack confirms 12M ---
packed 12000000 tokens
--- timer ---
2855

[stdout]
n ids: 19995
unique: True
in range: True
all ints: True
--- files ---
-rw-r--r-- 1 root root 3913 Aug  4 05:21 claim.md

submission/:
total 148
drwxr-xr-x 1 root root    140 Aug  4 05:12 .
drwxr-xr-x 1 root root    180 Aug  4 05:21 ..
-rw-r--r-- 1 root root   6305 Aug  4 05:12 curate.py
-rw-r--r-- 1 root root 144005 Aug  4 05:18 selection.json
--- pack confirms 12M ---
packed 12000000 tokens
--- timer ---
2855
[metadata] {"noOutputExpected": false}

Updated task #3 status

[metadata] {"success": true, "taskId": "3", "updatedFields": ["status"], "statusChange": {"from": "in_progress", "to": "completed"}}