claude-opus-5

passed quality gates: 110356 / 182016
wiki: 54294 candidates, top score 0.981, est tokens available 41.1M
webnews: 45118 candidates, top score 0.934, est tokens available 40.1M
qa: 10944 candidates, top score 0.948, est tokens available 7.3M
after register fill: 32937 docs {'wiki': 7.18, 'webnews': 12.7, 'qa': 6.52}
final: 32937 docs, est tokens 26.4 M
wrote /workspace/submission/selection.json
4619

[stdout]
passed quality gates: 110356 / 182016
wiki: 54294 candidates, top score 0.981, est tokens available 41.1M
webnews: 45118 candidates, top score 0.934, est tokens available 40.1M
qa: 10944 candidates, top score 0.948, est tokens available 7.3M
after register fill: 32937 docs {'wiki': 7.18, 'webnews': 12.7, 'qa': 6.52}
final: 32937 docs, est tokens 26.4 M
wrote /workspace/submission/selection.json
4619
[metadata] {"noOutputExpected": false}

n 32937
--- 'ergio Aguero has confirmed he broke a rib in his car crash in Amsterdam.\nThe Manchester City striker could be out for two months following the accident on Thursday night.\nThe taxi in which he was trav'
--- '<|endoftext|>Referring to her remarks in a press conference in New Delhi [ Images ] on the issue, he said, "She knows that her candidate Rajakannappan has filed an election petition in the Madras high'
--- '.<|endoftext|>Black Hawk War facts\nThe Black Hawk War was a war between the Indians under command of Chief Black Hawk and American settlers on the frontier in 1832. It took place mostly in Illinois an'
--- '<|endoftext|>String Method problem.\nHey guys, I am new to Java and I need some help trying to solve this problem.\npublic class Tester\npublic static void main(String args)\nScanner kb = new Scanner(Syst'
===== late in list
--- ' President Donald Trump has begun his first address to a joint session of Congress by condemning anti-Semitism and racism following a series of attacks in recent weeks.\n"Recent threats targeting Jewis'
--- 'usalem (AFP) - Hundreds of Israelis gathered on a grassy hill overlooking parliament on Thursday to smoke marijuana in an act of communal defiance of current laws.\nSitting in small groups on mats shad'
--- ' Formula 1 driver Lance Stroll has been cleared of rejoining the track unsafely in Mexican Grand Prix qualifying, the second such incident at the same corner this weekend.\nFIA race director Charlie Wh'
--- 'I am running a server application on a PC which receives UDP packets from a client. The UDP packets are received (verified in WireShark), but include the wrong checksum. I have no control over the cli'

[stdout]
n 32937
--- 'ergio Aguero has confirmed he broke a rib in his car crash in Amsterdam.\nThe Manchester City striker could be out for two months following the accident on Thursday night.\nThe taxi in which he was trav'
--- '<|endoftext|>Referring to her remarks in a press conference in New Delhi [ Images ] on the issue, he said, "She knows that her candidate Rajakannappan has filed an election petition in the Madras high'
--- '.<|endoftext|>Black Hawk War facts\nThe Black Hawk War was a war between the Indians under command of Chief Black Hawk and American settlers on the frontier in 1832. It took place mostly in Illinois an'
--- '<|endoftext|>String Method problem.\nHey guys, I am new to Java and I need some help trying to solve this problem.\npublic class Tester\npublic static void main(String args)\nScanner kb = new Scanner(Syst'
===== late in list
--- ' President Donald Trump has begun his first address to a joint session of Congress by condemning anti-Semitism and racism following a series of attacks in recent weeks.\n"Recent threats targeting Jewis'
--- 'usalem (AFP) - Hundreds of Israelis gathered on a grassy hill overlooking parliament on Thursday to smoke marijuana in an act of communal defiance of current laws.\nSitting in small groups on mats shad'
--- ' Formula 1 driver Lance Stroll has been cleared of rejoining the track unsafely in Mexican Grand Prix qualifying, the second such incident at the same corner this weekend.\nFIA race director Charlie Wh'
--- 'I am running a server application on a PC which receives UDP packets from a client. The UDP packets are received (verified in WireShark), but include the wrong checksum. I have no control over the cli'
[metadata] {"noOutputExpected": false}

started
random baseline selection written

[stdout]
started
random baseline selection written
[metadata] {"noOutputExpected": false}

File created successfully at: /workspace/submission/curate.py (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/workspace/submission/curate.py", "content": "#!/usr/bin/env python3\n\"\"\"Reproducible curation of /workspace/data/pool.jsonl -> /workspace/submission/selection.json\n\nSTATED CRITERION\n================\nThe scoring target is disclosed: held-out perplexity on a broad high-quality English\nmixture (encyclopedic / general high-quality web prose / news / technical Q&A).  A dev\nsample of exactly that distribution is provided as GPT-2 token ids in\n`/workspace/data/multi_dev.npy`.  So the criterion is:\n\n  1. DOMAIN AFFINITY.  Decode the dev target back to text, split it into its three\n     observable registers, and train a hashed-ngram multinomial logistic regression to\n     separate each register from random pool documents.  Every pool document is scored\n     by P(register | doc), averaged over up to three 1200-character windows.\n     Surface-form artifacts are normalised away on both sides (wikitext-103\n     detokenisation markers ` @-@ `/` @,@ `, HTML tags and entities) so the classifier\n     keys on *content and register*, not on formatting that the raw pool never contains.\n\n  2. STRUCTURAL QUALITY.  Independently of the classifier, gate out documents that are\n     not usable running prose: too short, low alphabetic fraction, non-ASCII-heavy,\n     implausible word/sentence length, too few sentences, navigation/boilerplate line\n     structure, duplicated lines, low type-token ratio, low trigram diversity,\n     cookie/subscribe/ToS boilerplate, or SHOUTING case.\n\n  3. MIXTURE MATCHING.  The dev target's token shares are encyclopedic 27.2%,\n     news+general-web 48.1%, technical Q&A 24.7%.  Each surviving document competes\n     only inside the register it most resembles, and the 12M-token budget is filled\n     round-robin across registers in those proportions -- so any prefix of the emitted\n     list already has the target mixture, which matters because the packer truncates.\n\n  4. DEDUPLICATION.  Greedy banded min-hash over 5-word shingles drops near-duplicates,\n     so the budget buys distinct tokens rather than repeated ones.\n\nDocuments are emitted best-first within register.  ~2.2x the budget is emitted so the\npacker never runs dry; ids past 12M tokens are simply unused.\n\nUsage:  python3 curate.py            (writes /workspace/submission/selection.json)\n\"\"\"\nimport json, os, pickle, re, sys, time\nimport numpy as np\nimport torch\n\nPOOL = '/workspace/data/pool.jsonl'\nDEV_NPY = '/workspace/data/multi_dev.npy'\nOUT = '/workspace/submission/selection.json'\nCACHE = '/dev/shm'\n\nBUDGET = 12_000_000\nOVERFILL = 2.2            # emit this multiple of the budget\nCHARS_PER_TOK = 4.0       # pool-wide GPT-2 chars/token, for budget bookkeeping only\nEOS = 50256\n\n# Register segments of the decoded dev target, established by inspecting the documents:\n# indices [0,1730) are wikitext-style encyclopedic, [1730,2350) news + general web prose,\n# [2350,end) StackExchange-style technical Q&A.\nSEG = [('wiki', 0, 1730), ('webnews', 1730, 2350), ('qa', 2350, None)]\nCL = ['neg', 'wiki', 'webnews', 'qa']\nSHARE = {'wiki': 0.272, 'webnews': 0.481, 'qa': 0.247}   # dev-target token shares\nNEG_DOCS = 30000          # random pool documents used as the negative class\nSEED = 0\n\n# ------------------------------------------------------------------ featurisation\nNBITS = 20\nNFEAT = 1 << NBITS\nW = 1200                  # scoring window in characters (~ dev document length scale)\n_tokre = re.compile(r\"[a-z]+'?[a-z]*|[0-9]+|[^\\sa-z0-9]\")\nimport zlib\n_crc = zlib.crc32\n\n\ndef featurize(win):\n    \"\"\"l2-normalised log-count hashed uni+bigram features for one window.\"\"\"\n    tk = _tokre.findall(win.lower())\n    idx = [_crc(w.encode()) & (NFEAT - 1) for w in tk]\n    for a, b in zip(tk, tk[1:]):\n        idx.append(_crc((a + '\\x00' + b).encode()) & (NFEAT - 1))\n    if not idx:\n        return np.zeros(0, np.int32), np.zeros(0, np.float32)\n    u, c = np.unique(np.asarray(idx, np.int64), return_counts=True)\n    v = np.log1p(c).astype(np.float32)\n    v /= (np.linalg.norm(v) + 1e-8)\n    return u.astype(np.int32), v\n\n\ndef featurize_many(wins):\n    ii, vv, ptr = [], [], [0]\n    for w in wins:\n        a, b = featurize(w)\n        ii.append(a); vv.append(b); ptr.append(ptr[-1] + len(a))\n    return (np.concatenate(ii) if ii else np.zeros(0, np.int32),\n            np.concatenate(vv) if vv else np.zeros(0, np.float32),\n            np.asarray(ptr, np.int64))\n\n\ndef windows(s, nw):\n    \"\"\"Up to nw evenly spaced W-char windows covering the document.\"\"\"\n    s = s.strip()\n    if not s:\n        return []\n    if len(s) <= W:\n        return [s]\n    step = max(W, (len(s) - W) // max(1, nw - 1)) if nw > 1 else len(s)\n    return [s[i:i + W] for i in range(0, len(s), step)][:nw]\n\n\n_TAG = re.compile(r'<[^>\\n]{1,40}>')\n\n\ndef strip_html(s):\n    \"\"\"Raw pool text carries no markup; drop tags/entities so both sides match.\"\"\"\n    s = _TAG.sub(' ', s)\n    for a, b in (('&gt;', '>'), ('&lt;', '<'), ('&amp;', '&'), ('&quot;', '\"'), ('&#39;', \"'\")):\n        s = s.replace(a, b)\n    return s\n\n\ndef denorm(s):\n    \"\"\"Undo wikitext-103 detokenisation so the target's surface form matches raw web text.\"\"\"\n    s = s.replace(' @-@ ', '-').replace(' @,@ ', ',').replace(' @.@ ', '.')\n    s = re.sub(r' ([,.;:!?%\\)\\]])', r'\\1', s)\n    s = re.sub(r'([\\(\\[]) ', r'\\1', s)\n    s = s.replace(\" 's\", \"'s\").replace(\" n't\", \"n't\")\n    return strip_html(s)\n\n\n# ------------------------------------------------------------------ quality gates\n_WORD = re.compile(r\"[A-Za-z']+\")\n_SENT_END = re.compile(r'[.!?][\"\\')\\]]?(\\s|$)')\nBAD = ('cookie', 'javascript', 'add to cart', 'all rights reserved', 'sign up',\n       'subscribe', 'click here', 'log in', 'terms of service', 'privacy policy')\n\n\ndef quality(t):\n    \"\"\"Structural gates for usable running prose. Returns (ok, prose_score).\"\"\"\n    n = len(t)\n    if n < 400:\n        return False, 0.0\n    words = _WORD.findall(t)\n    nw = len(words)\n    if nw < 60:\n        return False, 0.0\n    if sum(c.isalpha() or c.isspace() for c in t) / n < 0.80:\n        return False, 0.0\n    if sum(c < '\\x80' for c in t) / n < 0.95:\n        return False, 0.0\n    if not 3.0 <= sum(len(w) for w in words) / nw <= 9.0:\n        return False, 0.0\n    nsent = len(_SENT_END.findall(t))\n    if nsent < 3:\n        return False, 0.0\n    if not 5.0 <= nw / nsent <= 100.0:\n        return False, 0.0\n    lines = [l.strip() for l in t.split('\\n') if l.strip()]\n    if lines:\n        if sum(len(l) < 40 for l in lines) / len(lines) > 0.6 and len(lines) > 8:\n            return False, 0.0\n        if len(set(lines)) / len(lines) < 0.7:\n            return False, 0.0\n    lw = [w.lower() for w in words]\n    uniq = len(set(lw)) / nw\n    if uniq < 0.25:\n        return False, 0.0\n    tri = set(zip(lw, lw[1:], lw[2:]))\n    if nw > 30 and len(tri) / max(1, nw - 2) < 0.55:\n        return False, 0.0\n    if sum(b in t[:600].lower() for b in BAD) >= 3:\n        return False, 0.0\n    if sum(c.isupper() for c in t) / max(1, sum(c.isalpha() for c in t)) > 0.25:\n        return False, 0.0\n    return True, min(1.0, nsent / 8.0) * min(1.0, uniq / 0.5)\n\n\n# ------------------------------------------------------------------ near-dup sketch\ndef sig(t):\n    \"\"\"4 banded min-hashes over 5-word shingles. Collision in >=2 bands => near-dup.\"\"\"\n    w = [x.lower() for x in _WORD.findall(t)][:400]\n    if len(w) < 8:\n        return None\n    sh = np.array([_crc(' '.join(w[i:i + 5]).encode()) for i in range(len(w) - 4)],\n                  dtype=np.uint64)\n    if len(sh) == 0:\n        return None\n    return tuple(int(((sh * np.uint64(2654435761 + 7919 * k)) & np.uint64(0xFFFFFFFF)).min())\n                 for k in range(4))\n\n\ndef main():\n    t0 = time.time()\n\n    # ---- load pool\n    ids, texts = [], []\n    with open(POOL) as f:\n        for line in f:\n            r = json.loads(line)\n            ids.append(r['id']); texts.append(r['text'])\n    ids = np.asarray(ids)\n    print(f'pool: {len(ids)} docs ({time.time()-t0:.0f}s)', flush=True)\n\n    # ---- decode the dev target into documents (the positive class)\n    from transformers import AutoTokenizer\n    tok = AutoTokenizer.from_pretrained('gpt2')\n    a = np.load(DEV_NPY).astype(np.int64)\n    dev, prev = [], 0\n    for i in np.nonzero(a == EOS)[0]:\n        if i > prev:\n            dev.append(a[prev:i])\n        prev = i + 1\n    if len(a) > prev:\n        dev.append(a[prev:])\n    dev = [tok.decode(d.tolist()) for d in dev]\n    print(f'dev target: {len(dev)} docs ({time.time()-t0:.0f}s)', flush=True)\n\n    pos_txt, pos_lab = [], []\n    for name, lo, hi in SEG:\n        for d in dev[lo:(hi if hi is not None else len(dev))]:\n            pos_txt.append(denorm(d)); pos_lab.append(CL.index(name))\n\n    rng = np.random.default_rng(SEED)\n    neg_txt = []\n    for i in rng.choice(len(texts), NEG_DOCS, replace=False):\n        neg_txt.extend(strip_html(w) for w in windows(texts[i], 2))\n\n    # ---- fit the register classifier on GPU\n    tr = pos_txt + neg_txt\n    y = np.array(pos_lab + [0] * len(neg_txt))\n    ii, vv, ptr = featurize_many(tr)\n    dv = 'cuda' if torch.cuda.is_available() else 'cpu'\n    X = torch.sparse_csr_tensor(torch.from_numpy(ptr), torch.from_numpy(ii.astype(np.int64)),\n                                torch.from_numpy(vv), size=(len(tr), NFEAT)).to(dv)\n    Y = torch.from_numpy(y).to(dv)\n    cnt = np.bincount(y, minlength=len(CL)).astype(np.float32)\n    cw = torch.tensor(cnt.sum() / (len(CL) * cnt), device=dv)\n    Wt = torch.zeros(NFEAT, len(CL), device=dv, requires_grad=True)\n    bt = torch.zeros(len(CL), device=dv, requires_grad=True)\n    opt = torch.optim.Adam([Wt, bt], lr=0.5)\n    for _ in range(400):\n        loss = torch.nn.functional.cross_entropy(torch.sparse.mm(X, Wt) + bt, Y, weight=cw)\n        opt.zero_grad(set_to_none=True)\n        (loss + 1e-5 * (Wt * Wt).sum()).backward()\n        opt.step()\n    Wd, bd = Wt.detach(), bt.detach()\n    print(f'classifier fitted ({time.time()-t0:.0f}s)', flush=True)\n\n    # ---- score every pool document\n    S = np.zeros((len(texts), len(CL)), np.float32)\n    B = 6000\n    for s0 in range(0, len(texts), B):\n        chunk = texts[s0:s0 + B]\n        flat, owner = [], []\n        for j, tx in enumerate(chunk):\n            ws = [strip_html(w) for w in windows(tx, 3)]\n            flat.extend(ws); owner.extend([j] * len(ws))\n        ii, vv, ptr = featurize_many(flat)\n        Xb = torch.sparse_csr_tensor(torch.from_numpy(ptr), torch.from_numpy(ii.astype(np.int64)),\n                                     torch.from_numpy(vv), size=(len(flat), NFEAT)).to(dv)\n        with torch.no_grad():\n            P = torch.softmax(torch.sparse.mm(Xb, Wd) + bd, dim=1).cpu().numpy()\n        ow = np.asarray(owner)\n        cw2 = np.bincount(ow, minlength=len(chunk)).clip(1)\n        for c in range(len(CL)):\n            S[s0:s0 + len(chunk), c] = np.bincount(ow, weights=P[:, c], minlength=len(chunk)) / cw2\n    print(f'pool scored ({time.time()-t0:.0f}s)', flush=True)\n\n    # ---- structural quality\n    ok = np.zeros(len(texts), bool)\n    prose = np.zeros(len(texts), np.float32)\n    for i, t in enumerate(texts):\n        ok[i], prose[i] = quality(t)\n    print(f'quality gates: {int(ok.sum())}/{len(texts)} kept ({time.time()-t0:.0f}s)', flush=True)\n\n    # ---- per-register candidate rankings\n    est_tok = np.array([len(t) for t in texts], np.float64) / CHARS_PER_TOK\n    tgt = S[:, 1:].sum(1)\n    arg = 1 + S[:, 1:].argmax(1)\n    cands = {}\n    for name in SHARE:\n        col = CL.index(name)\n        idx = np.nonzero(ok & (arg == col))[0]\n        key = S[idx, col] * (0.7 + 0.3 * prose[idx])\n        cands[name] = idx[np.argsort(-key)]\n        print(f'  {name}: {len(idx)} candidates, {est_tok[idx].sum()/1e6:.1f}M est tokens')\n    topup = np.nonzero(ok)[0]\n    topup = topup[np.argsort(-tgt[topup])]\n\n    # ---- greedy dedup + mixture-balanced round-robin fill\n    bands = [set() for _ in range(4)]\n\n    def is_dup(i):\n        s = sig(texts[i])\n        if s is None:\n            return True\n        if sum(s[k] in bands[k] for k in range(4)) >= 2:\n            return True\n        for k in range(4):\n            bands[k].add(s[k])\n        return False\n\n    target_tok = {n: SHARE[n] * BUDGET * OVERFILL for n in SHARE}\n    ptr_r = {n: 0 for n in SHARE}\n    got = {n: 0.0 for n in SHARE}\n    order = ['webnews', 'wiki', 'qa']\n    STRIDE = {'webnews': 2, 'wiki': 1, 'qa': 1}\n    sel, used = [], set()\n    progress = True\n    while progress:\n        progress = False\n        for name in order:\n            for _ in range(STRIDE[name]):\n                if got[name] >= target_tok[name]:\n                    continue\n                arr = cands[name]\n                while ptr_r[name] < len(arr):\n                    i = int(arr[ptr_r[name]]); ptr_r[name] += 1\n                    if i in used or is_dup(i):\n                        continue\n                    used.add(i); sel.append(i); got[name] += est_tok[i]; progress = True\n                    break\n    print('register fill:', len(sel), 'docs',\n          {k: round(v / 1e6, 2) for k, v in got.items()}, flush=True)\n\n    total = sum(got.values())\n    tp = 0\n    while total < BUDGET * OVERFILL and tp < len(topup):\n        i = int(topup[tp]); tp += 1\n        if i in used or is_dup(i):\n            continue\n        used.add(i); sel.append(i); total += est_tok[i]\n\n    out = [int(ids[i]) for i in sel]\n    assert len(set(out)) == len(out), 'duplicate ids'\n    os.makedirs(os.path.dirname(OUT), exist_ok=True)\n    json.dump(out, open(OUT, 'w'))\n    print(f'wrote {OUT}: {len(out)} ids, ~{total/1e6:.1f}M est tokens ({time.time()-t0:.0f}s)')\n\n\nif __name__ == '__main__':\n    main()\n", "structuredPatch": [], "originalFile": null, "userModified": false}

Token indices sequence length is longer than the specified maximum sequence length for this model (1028 > 1024). Running this sequence through the model will result in indexing errors
packed 12000000 tokens from 32937 ids -> /tmp/dev_train.npy
--- time:
4539

[stdout]
Token indices sequence length is longer than the specified maximum sequence length for this model (1028 > 1024). Running this sequence through the model will result in indexing errors
packed 12000000 tokens from 32937 ids -> /tmp/dev_train.npy
--- time:
4539
[metadata] {"noOutputExpected": false}

packed 12000000 tokens from 32937 ids -> /tmp/dev_train.npy
HELDOUT_PPL 351.488  loss 5.8622  train_s 50  /tmp/dev_train.npy
(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)

[stdout]
packed 12000000 tokens from 32937 ids -> /tmp/dev_train.npy
HELDOUT_PPL 351.488  loss 5.8622  train_s 50  /tmp/dev_train.npy
(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)
[metadata] {"noOutputExpected": false}

baseline_started
docs with wikitext @-@ markers: 0
docs with detok space-before-punct style: 414
docs with <p>: 96 with <code>: 118 both: 7
--- SBP "I Love You Like A Fat Lady Loves Apples\nHappy Valentine's Day my love. I brought you some apples, eat until your heart's content. A collaboration between Geoffrey Lillemon & Random Studio.\nActivating Your Intuition\nThere are 40+ hours of FREE audio m"
--- SBP 'Thursday, December 17, 2009\nThis old man was the star of many news . I remember seeing him in quite a few places. Now another newsclip has surfaced on the British Pathe Website - titled "Nehru In Kashmir", it shows the visit of PM Nehru and various o'
--- SBP 'To "Die Empty " is to die with no regrets of unfulfilled agenda, with no pending plans. This is possible when , after careful and deep evaluation , wehave arrived at our main purpose in life and when wehave fully spent all my energy, time and focus i'
--- PCODE "++ I'm commenting mostly just to bump this excellent piece of advice.\nSince port is rarely important and I like to use this idiom in addition to running a traditional webserver on port 80, I'd shorten it to use the default port 5000–\nplackup -L Shotg"
--- PCODE '<|endoftext|>OO in the real world....\n- Wrench set\n- Socket Set\n- Screwdriver set\n- Pots and pans\nOO exists in many place in many ways. It is great in situations where there are large unknowns in the future. You build code, test it and maintain it se'
--- PCODE "Sorry that this reply is almost boiler-plate, but...\n- Does that site's usage guidance permit scraping?\n- Do you have authority/permission to extract data?\n- Does the site publish an API you could use rather than rolling your own?\n- Assuming that by "

[stdout]
baseline_started
docs with wikitext @-@ markers: 0
docs with detok space-before-punct style: 414
docs with <p>: 96 with <code>: 118 both: 7
--- SBP "I Love You Like A Fat Lady Loves Apples\nHappy Valentine's Day my love. I brought you some apples, eat until your heart's content. A collaboration between Geoffrey Lillemon & Random Studio.\nActivating Your Intuition\nThere are 40+ hours of FREE audio m"
--- SBP 'Thursday, December 17, 2009\nThis old man was the star of many news . I remember seeing him in quite a few places. Now another newsclip has surfaced on the British Pathe Website - titled "Nehru In Kashmir", it shows the visit of PM Nehru and various o'
--- SBP 'To "Die Empty " is to die with no regrets of unfulfilled agenda, with no pending plans. This is possible when , after careful and deep evaluation , wehave arrived at our main purpose in life and when wehave fully spent all my energy, time and focus i'
--- PCODE "++ I'm commenting mostly just to bump this excellent piece of advice.\nSince port is rarely important and I like to use this idiom in addition to running a traditional webserver on port 80, I'd shorten it to use the default port 5000–\nplackup -L Shotg"
--- PCODE '<|endoftext|>OO in the real world....\n- Wrench set\n- Socket Set\n- Screwdriver set\n- Pots and pans\nOO exists in many place in many ways. It is great in situations where there are large unknowns in the future. You build code, test it and maintain it se'
--- PCODE "Sorry that this reply is almost boiler-plate, but...\n- Does that site's usage guidance permit scraping?\n- Do you have authority/permission to extract data?\n- Does the site publish an API you could use rather than rolling your own?\n- Assuming that by "
[metadata] {"noOutputExpected": false}

File created successfully at: /workspace/work/variants.py (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/workspace/work/variants.py", "content": "\"\"\"Build selection variants from the cached scores, to A/B on the dev target.\n\nV2 = quality-gated, ranked by total target affinity, no register balancing.\nV3 = register-balanced but *sampled* from a wider high-affinity band (DSIR-style\n     importance resampling) instead of strict top-k -> more topical diversity.\n\"\"\"\nimport json, pickle, sys\nimport numpy as np\n\nsys.path.insert(0, '/workspace/work')\nfrom select_lib import load_all, is_dup_factory, EST, BUDGET, SHARE, CL\n\nMODE = sys.argv[1]\nOUT = sys.argv[2]\ntexts, ids, S, ok, prose, est_tok = load_all()\ntgt = S[:, 1:].sum(1)\narg = 1 + S[:, 1:].argmax(1)\nis_dup = is_dup_factory(texts)\nOVERFILL = 2.2\n\nsel, used, total = [], set(), 0.0\n\nif MODE == 'V2':\n    order = np.nonzero(ok)[0]\n    order = order[np.argsort(-tgt[order])]\n    for i in order:\n        i = int(i)\n        if total >= BUDGET * OVERFILL:\n            break\n        if is_dup(i):\n            continue\n        used.add(i); sel.append(i); total += est_tok[i]\n\nelif MODE == 'V3':\n    rng = np.random.default_rng(11)\n    BAND = 4.0          # draw from a band holding ~4x the tokens we need\n    per_reg = []\n    for name in SHARE:\n        col = CL.index(name)\n        idx = np.nonzero(ok & (arg == col))[0]\n        idx = idx[np.argsort(-S[idx, col])]\n        need = SHARE[name] * BUDGET * OVERFILL\n        cum = np.cumsum(est_tok[idx])\n        cut = int(np.searchsorted(cum, need * BAND)) + 1\n        band = idx[:min(cut, len(idx))]\n        # sample without replacement, probability proportional to affinity\n        w = S[band, col].astype(np.float64) ** 2\n        w /= w.sum()\n        perm = rng.choice(len(band), size=len(band), replace=False, p=w)\n        per_reg.append((name, band[perm], need))\n        print(name, 'band', len(band), 'est tokens', round(est_tok[band].sum() / 1e6, 1), 'M')\n    ptr = {n: 0 for n, _, _ in per_reg}\n    got = {n: 0.0 for n, _, _ in per_reg}\n    progress = True\n    while progress:\n        progress = False\n        for name, arr, need in per_reg:\n            reps = 2 if name == 'webnews' else 1\n            for _ in range(reps):\n                if got[name] >= need:\n                    continue\n                while ptr[name] < len(arr):\n                    i = int(arr[ptr[name]]); ptr[name] += 1\n                    if i in used or is_dup(i):\n                        continue\n                    used.add(i); sel.append(i); got[name] += est_tok[i]\n                    progress = True\n                    break\n    total = sum(got.values())\n    print({k: round(v / 1e6, 2) for k, v in got.items()})\n\njson.dump([int(ids[i]) for i in sel], open(OUT, 'w'))\nprint(MODE, len(sel), 'docs, est', round(total / 1e6, 2), 'M ->', OUT)\n", "structuredPatch": [], "originalFile": null, "userModified": false}

File created successfully at: /workspace/work/select_lib.py (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/workspace/work/select_lib.py", "content": "\"\"\"Shared loading / quality / dedup helpers for the selection variants.\"\"\"\nimport pickle, re\nimport numpy as np\nimport zlib\n\nBUDGET = 12_000_000\nEST = 4.0\nSHARE = {'wiki': 0.272, 'webnews': 0.481, 'qa': 0.247}\nCL = ['neg', 'wiki', 'webnews', 'qa']\n\n_WORD = re.compile(r\"[A-Za-z']+\")\n_SENT_END = re.compile(r'[.!?][\"\\')\\]]?(\\s|$)')\nBAD = ('cookie', 'javascript', 'add to cart', 'all rights reserved', 'sign up',\n       'subscribe', 'click here', 'log in', 'terms of service', 'privacy policy')\n\n\ndef quality(t):\n    n = len(t)\n    if n < 400:\n        return False, 0.0\n    words = _WORD.findall(t)\n    nw = len(words)\n    if nw < 60:\n        return False, 0.0\n    if sum(c.isalpha() or c.isspace() for c in t) / n < 0.80:\n        return False, 0.0\n    if sum(c < '\\x80' for c in t) / n < 0.95:\n        return False, 0.0\n    if not 3.0 <= sum(len(w) for w in words) / nw <= 9.0:\n        return False, 0.0\n    nsent = len(_SENT_END.findall(t))\n    if nsent < 3:\n        return False, 0.0\n    if not 5.0 <= nw / nsent <= 100.0:\n        return False, 0.0\n    lines = [l.strip() for l in t.split('\\n') if l.strip()]\n    if lines:\n        if sum(len(l) < 40 for l in lines) / len(lines) > 0.6 and len(lines) > 8:\n            return False, 0.0\n        if len(set(lines)) / len(lines) < 0.7:\n            return False, 0.0\n    lw = [w.lower() for w in words]\n    uniq = len(set(lw)) / nw\n    if uniq < 0.25:\n        return False, 0.0\n    tri = set(zip(lw, lw[1:], lw[2:]))\n    if nw > 30 and len(tri) / max(1, nw - 2) < 0.55:\n        return False, 0.0\n    if sum(b in t[:600].lower() for b in BAD) >= 3:\n        return False, 0.0\n    if sum(c.isupper() for c in t) / max(1, sum(c.isalpha() for c in t)) > 0.25:\n        return False, 0.0\n    return True, min(1.0, nsent / 8.0) * min(1.0, uniq / 0.5)\n\n\ndef load_all():\n    ids, texts = pickle.load(open('/dev/shm/pool.pkl', 'rb'))\n    ids = np.asarray(ids)\n    S = np.load('/workspace/work/scores.npy')\n    try:\n        ok = np.load('/workspace/work/ok.npy')\n        prose = np.load('/workspace/work/prose.npy')\n    except FileNotFoundError:\n        ok = np.zeros(len(texts), bool)\n        prose = np.zeros(len(texts), np.float32)\n        for i, t in enumerate(texts):\n            ok[i], prose[i] = quality(t)\n        np.save('/workspace/work/ok.npy', ok)\n        np.save('/workspace/work/prose.npy', prose)\n    est_tok = np.array([len(t) for t in texts], np.float64) / EST\n    return texts, ids, S, ok, prose, est_tok\n\n\ndef sig(t):\n    w = [x.lower() for x in _WORD.findall(t)][:400]\n    if len(w) < 8:\n        return None\n    sh = np.array([zlib.crc32(' '.join(w[i:i + 5]).encode()) for i in range(len(w) - 4)],\n                  dtype=np.uint64)\n    if len(sh) == 0:\n        return None\n    return tuple(int(((sh * np.uint64(2654435761 + 7919 * k)) & np.uint64(0xFFFFFFFF)).min())\n                 for k in range(4))\n\n\ndef is_dup_factory(texts):\n    bands = [set() for _ in range(4)]\n\n    def is_dup(i):\n        s = sig(texts[i])\n        if s is None:\n            return True\n        if sum(s[k] in bands[k] for k in range(4)) >= 2:\n            return True\n        for k in range(4):\n            bands[k].add(s[k])\n        return False\n    return is_dup\n", "structuredPatch": [], "originalFile": null, "userModified": false}

Command did not complete within its 120s timeout and was moved to the background (ID: beyus5add). Output is being written to: /tmp/claude-0/-workspace/8a292ca2-99c0-45e1-8f12-d316d2eed732/tasks/beyus5add.output. You will be notified when it completes. To check interim output, use Read on that file path.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "beyus5add", "timedOutAfterMs": 120000}

Token indices sequence length is longer than the specified maximum sequence length for this model (2709 > 1024). Running this sequence through the model will result in indexing errors
packed 12000000 tokens from 60000 ids -> /tmp/dev_train.npy
V2 31107 docs, est 26.4 M -> /workspace/work/sel_v2.json
wiki band 34569 est tokens 28.7 M
webnews band 45118 est tokens 40.1 M
qa band 10944 est tokens 7.3 M
{'wiki': 7.18, 'webnews': 12.7, 'qa': 6.52}
V3 32137 docs, est 26.4 M -> /workspace/work/sel_v3.json
4228
=== time
4178

[stdout]
Token indices sequence length is longer than the specified maximum sequence length for this model (2709 > 1024). Running this sequence through the model will result in indexing errors
packed 12000000 tokens from 60000 ids -> /tmp/dev_train.npy
V2 31107 docs, est 26.4 M -> /workspace/work/sel_v2.json
wiki band 34569 est tokens 28.7 M
webnews band 45118 est tokens 40.1 M
qa band 10944 est tokens 7.3 M
{'wiki': 7.18, 'webnews': 12.7, 'qa': 6.52}
V3 32137 docs, est 26.4 M -> /workspace/work/sel_v3.json
4228
=== time
4178
[metadata] {"noOutputExpected": false}

HELDOUT_PPL 470.092  loss 6.1529  train_s 51  /tmp/dev_train.npy
(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)

[stdout]
HELDOUT_PPL 470.092  loss 6.1529  train_s 51  /tmp/dev_train.npy
(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)
[metadata] {"noOutputExpected": false}

1730 "Some observers say that Alabama's move to close dozens of drivers license offices is a discriminatory move tha"
1770 'Introduction\n\nThe Templar Assassin is a hero who seems counter-intuitive at first glance, having neither the a'
1810 'Many of the tennis WAGs will be seen at this year’s U.S. Open in New York, scheduled to be played through Sept'
1850 '“Game of Thrones” and “The Hunger Games” actress Natalie Dormer is to star as English headmistress Hester Appl'
1890 'We are living through the Summer of Sanders.\n\nThe senator from Vermont is drawing the biggest crowds of the 20'
1930 ': An Egyptian prosecutor has ordered the detention of the train drivers and their assistants involved in the f'
1970 'Anti-H1B posters have come up in local metro stations and commuter trains in San Francisco and California, ahe'
2010 'Beijing: Prime Minister Narendra Modi and Chinese President Xi Jinping will not sign any agreements or issue a'
2050 ': The Supreme Court on Friday granted time till November 10 to the Election Commission to decide the claims ov'
2090 'The Pakistani opposition party led by former cricket star Imran Khan is gaining ground in opinion polls ahead '
2130 "DON'T MISS: From his match-winning century, him going through tough times to coming up with new celebration st"
2170 'Can a splash of grey pavement paint help combat global warming?In Los Angeles, where summer temperatures regul'
2210 'New Zealand came from a goal down to draw 1-1 against Turkey in the opening Group B match of the FIFA U-17 Wor'
2250 'Bigg Boss Marathi 23rd April 2018 Episode 9 begins with talks about elimination between Resham Tipnis, Rajesh,'
2290 'The Congress on Friday night termed the reduction in GST rates as "too little, too late" and said a lot more n'
2330 'Related Story GST Return - Last Week to File Your 1st Ever GSTR before Aug 20\nNearly 45 days have passed since'

[stdout]
1730 "Some observers say that Alabama's move to close dozens of drivers license offices is a discriminatory move tha"
1770 'Introduction\n\nThe Templar Assassin is a hero who seems counter-intuitive at first glance, having neither the a'
1810 'Many of the tennis WAGs will be seen at this year’s U.S. Open in New York, scheduled to be played through Sept'
1850 '“Game of Thrones” and “The Hunger Games” actress Natalie Dormer is to star as English headmistress Hester Appl'
1890 'We are living through the Summer of Sanders.\n\nThe senator from Vermont is drawing the biggest crowds of the 20'
1930 ': An Egyptian prosecutor has ordered the detention of the train drivers and their assistants involved in the f'
1970 'Anti-H1B posters have come up in local metro stations and commuter trains in San Francisco and California, ahe'
2010 'Beijing: Prime Minister Narendra Modi and Chinese President Xi Jinping will not sign any agreements or issue a'
2050 ': The Supreme Court on Friday granted time till November 10 to the Election Commission to decide the claims ov'
2090 'The Pakistani opposition party led by former cricket star Imran Khan is gaining ground in opinion polls ahead '
2130 "DON'T MISS: From his match-winning century, him going through tough times to coming up with new celebration st"
2170 'Can a splash of grey pavement paint help combat global warming?In Los Angeles, where summer temperatures regul'
2210 'New Zealand came from a goal down to draw 1-1 against Turkey in the opening Group B match of the FIFA U-17 Wor'
2250 'Bigg Boss Marathi 23rd April 2018 Episode 9 begins with talks about elimination between Resham Tipnis, Rajesh,'
2290 'The Congress on Friday night termed the reduction in GST rates as "too little, too late" and said a lot more n'
2330 'Related Story GST Return - Last Week to File Your 1st Ever GSTR before Aug 20\nNearly 45 days have passed since'
[metadata] {"noOutputExpected": false}

HELDOUT_PPL 363.574  loss 5.8960  train_s 50  /tmp/dev_train.npy
(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)
v2 started

[stdout]
HELDOUT_PPL 363.574  loss 5.8960  train_s 50  /tmp/dev_train.npy
(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)
v2 started
[metadata] {"noOutputExpected": false}

File created successfully at: /workspace/work/mixvar.py (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/workspace/work/mixvar.py", "content": "\"\"\"Build a V1-style (sharp, register-balanced) selection with an arbitrary mixture.\n\nusage: python3 mixvar.py OUT.json w_wiki w_webnews w_qa\n\"\"\"\nimport json, sys\nimport numpy as np\n\nsys.path.insert(0, '/workspace/work')\nfrom select_lib import load_all, is_dup_factory, BUDGET, CL\n\nOUT = sys.argv[1]\nw = np.array([float(x) for x in sys.argv[2:5]])\nw = w / w.sum()\nSHARE = dict(zip(['wiki', 'webnews', 'qa'], w))\nOVERFILL = 2.2\n\ntexts, ids, S, ok, prose, est_tok = load_all()\ntgt = S[:, 1:].sum(1)\narg = 1 + S[:, 1:].argmax(1)\nis_dup = is_dup_factory(texts)\n\ncands = {}\nfor name in SHARE:\n    col = CL.index(name)\n    idx = np.nonzero(ok & (arg == col))[0]\n    key = S[idx, col] * (0.7 + 0.3 * prose[idx])\n    cands[name] = idx[np.argsort(-key)]\n\n# emissions per round proportional to share, so any prefix is mixture-balanced\nbase = min(SHARE.values())\nSTRIDE = {n: max(1, int(round(SHARE[n] / base))) for n in SHARE}\ntarget_tok = {n: SHARE[n] * BUDGET * OVERFILL for n in SHARE}\nptr = {n: 0 for n in SHARE}\ngot = {n: 0.0 for n in SHARE}\nsel, used = [], set()\nprogress = True\nwhile progress:\n    progress = False\n    for name in ['webnews', 'wiki', 'qa']:\n        for _ in range(STRIDE[name]):\n            if got[name] >= target_tok[name]:\n                continue\n            arr = cands[name]\n            while ptr[name] < len(arr):\n                i = int(arr[ptr[name]]); ptr[name] += 1\n                if i in used or is_dup(i):\n                    continue\n                used.add(i); sel.append(i); got[name] += est_tok[i]; progress = True\n                break\n\ntotal = sum(got.values())\ntopup = np.nonzero(ok)[0]\ntopup = topup[np.argsort(-tgt[topup])]\ntp = 0\nwhile total < BUDGET * OVERFILL and tp < len(topup):\n    i = int(topup[tp]); tp += 1\n    if i in used or is_dup(i):\n        continue\n    used.add(i); sel.append(i); total += est_tok[i]\n\njson.dump([int(ids[i]) for i in sel], open(OUT, 'w'))\nprint(OUT, 'shares', {k: round(v, 3) for k, v in SHARE.items()}, 'stride', STRIDE,\n      '->', len(sel), 'docs', round(total / 1e6, 2), 'M est',\n      {k: round(v / 1e6, 2) for k, v in got.items()})\n", "structuredPatch": [], "originalFile": null, "userModified": false}

work/sel_v4.json shares {'wiki': 0.4, 'webnews': 0.4, 'qa': 0.2} stride {'wiki': 2, 'webnews': 2, 'qa': 1} -> 32926 docs 26.4 M est {'wiki': 10.56, 'webnews': 10.56, 'qa': 5.28}
work/sel_v5.json shares {'wiki': 0.2, 'webnews': 0.55, 'qa': 0.25} stride {'wiki': 1, 'webnews': 3, 'qa': 1} -> 32602 docs 26.4 M est {'wiki': 5.28, 'webnews': 14.52, 'qa': 6.6}
packed 12000000 tokens from 31107 ids -> /tmp/dev_train.npy
3958

[stdout]
work/sel_v4.json shares {'wiki': 0.4, 'webnews': 0.4, 'qa': 0.2} stride {'wiki': 2, 'webnews': 2, 'qa': 1} -> 32926 docs 26.4 M est {'wiki': 10.56, 'webnews': 10.56, 'qa': 5.28}
work/sel_v5.json shares {'wiki': 0.2, 'webnews': 0.55, 'qa': 0.25} stride {'wiki': 1, 'webnews': 3, 'qa': 1} -> 32602 docs 26.4 M est {'wiki': 5.28, 'webnews': 14.52, 'qa': 6.6}
packed 12000000 tokens from 31107 ids -> /tmp/dev_train.npy
3958
[metadata] {"noOutputExpected": false}

packing v4,v5 in parallel
[1]-  Done                    nohup python3 pack_selection.py work/sel_$v.json /tmp/tr_$v.npy > work/pack_$v.log 2>&1
[2]+  Done                    nohup python3 pack_selection.py work/sel_$v.json /tmp/tr_$v.npy > work/pack_$v.log 2>&1
(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)

[stdout]
packing v4,v5 in parallel
[1]-  Done                    nohup python3 pack_selection.py work/sel_$v.json /tmp/tr_$v.npy > work/pack_$v.log 2>&1
[2]+  Done                    nohup python3 pack_selection.py work/sel_$v.json /tmp/tr_$v.npy > work/pack_$v.log 2>&1
(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)
[metadata] {"noOutputExpected": false}

HELDOUT_PPL 383.099  loss 5.9483  train_s 50  /tmp/dev_train.npy
-rw-r--r-- 1 root root 24000128 Aug  3 23:07 /tmp/tr_v4.npy
-rw-r--r-- 1 root root 24000128 Aug  3 23:07 /tmp/tr_v5.npy
packed 12000000 tokens from 32926 ids -> /tmp/tr_v4.npy

[stdout]
HELDOUT_PPL 383.099  loss 5.9483  train_s 50  /tmp/dev_train.npy
-rw-r--r-- 1 root root 24000128 Aug  3 23:07 /tmp/tr_v4.npy
-rw-r--r-- 1 root root 24000128 Aug  3 23:07 /tmp/tr_v5.npy
packed 12000000 tokens from 32926 ids -> /tmp/tr_v4.npy
[metadata] {"noOutputExpected": false}

v4 HELDOUT_PPL 357.273  loss 5.8785  train_s 50  /tmp/tr_v4.npy
v5 HELDOUT_PPL 367.248  loss 5.9060  train_s 50  /tmp/tr_v5.npy
3778

[stdout]
v4 HELDOUT_PPL 357.273  loss 5.8785  train_s 50  /tmp/tr_v4.npy
v5 HELDOUT_PPL 367.248  loss 5.9060  train_s 50  /tmp/tr_v5.npy
3778
[metadata] {"noOutputExpected": false}

File created successfully at: /workspace/work/round2.py (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/workspace/work/round2.py", "content": "\"\"\"Round-2 (pseudo-labelled) register classifier.\n\nRound 1 trains target-vs-pool, but the target's surface form (wikitext detokenisation,\nStackExchange HTML) never occurs in the raw pool, so part of the round-1 decision\nboundary is spent on formatting rather than register.  Round 2 fixes that: take round-1's\nhighest-scoring *pool* documents per register as positives (same surface distribution as\nevery other pool document) and re-fit against random pool negatives.  The resulting\nranking is driven purely by content/register.\n\"\"\"\nimport pickle, sys, time\nimport numpy as np\nimport torch\n\nsys.path.insert(0, '/workspace/work')\nfrom feats import featurize_many, windows, NFEAT\nfrom select_lib import load_all, CL\n\nTOPK = 4000          # round-1 pool positives per register\nt0 = time.time()\ntexts, ids, S, ok, prose, est_tok = load_all()\narg = 1 + S[:, 1:].argmax(1)\n\npos_txt, pos_lab = [], []\npos_ids = set()\nfor name in ['wiki', 'webnews', 'qa']:\n    col = CL.index(name)\n    idx = np.nonzero(ok & (arg == col))[0]\n    idx = idx[np.argsort(-S[idx, col])][:TOPK]\n    for i in idx:\n        pos_ids.add(int(i))\n        for w in windows(texts[i], 2):\n            pos_txt.append(w)\n            pos_lab.append(col)\n    print(name, len(idx), 'pool positives', flush=True)\n\nrng = np.random.default_rng(1)\nneg_txt = []\nfor i in rng.choice(len(texts), 40000, replace=False):\n    if int(i) in pos_ids:\n        continue\n    neg_txt.extend(windows(texts[i], 2))\nprint('windows: pos', len(pos_txt), 'neg', len(neg_txt), round(time.time() - t0, 1), flush=True)\n\ntr = pos_txt + neg_txt\ny = np.array(pos_lab + [0] * len(neg_txt))\nii, vv, ptr = featurize_many(tr)\ndv = 'cuda'\nX = torch.sparse_csr_tensor(torch.from_numpy(ptr), torch.from_numpy(ii.astype(np.int64)),\n                            torch.from_numpy(vv), size=(len(tr), NFEAT)).to(dv)\nY = torch.from_numpy(y).to(dv)\ncnt = np.bincount(y, minlength=len(CL)).astype(np.float32)\ncw = torch.tensor(cnt.sum() / (len(CL) * cnt), device=dv)\nWt = torch.zeros(NFEAT, len(CL), device=dv, requires_grad=True)\nbt = torch.zeros(len(CL), device=dv, requires_grad=True)\nopt = torch.optim.Adam([Wt, bt], lr=0.5)\nfor step in range(400):\n    logits = torch.sparse.mm(X, Wt) + bt\n    loss = torch.nn.functional.cross_entropy(logits, Y, weight=cw)\n    opt.zero_grad(set_to_none=True)\n    (loss + 1e-5 * (Wt * Wt).sum()).backward()\n    opt.step()\n    if step % 200 == 0:\n        print('  step', step, round(loss.item(), 4),\n              round((logits.argmax(1) == Y).float().mean().item(), 4), flush=True)\nWd, bd = Wt.detach(), bt.detach()\n\nS2 = np.zeros((len(texts), len(CL)), np.float32)\nB = 6000\nfor s0 in range(0, len(texts), B):\n    chunk = texts[s0:s0 + B]\n    flat, owner = [], []\n    for j, tx in enumerate(chunk):\n        ws = windows(tx, 3)\n        flat.extend(ws); owner.extend([j] * len(ws))\n    ii, vv, ptr = featurize_many(flat)\n    Xb = torch.sparse_csr_tensor(torch.from_numpy(ptr), torch.from_numpy(ii.astype(np.int64)),\n                                 torch.from_numpy(vv), size=(len(flat), NFEAT)).to(dv)\n    with torch.no_grad():\n        P = torch.softmax(torch.sparse.mm(Xb, Wd) + bd, dim=1).cpu().numpy()\n    ow = np.asarray(owner)\n    cw2 = np.bincount(ow, minlength=len(chunk)).clip(1)\n    for c in range(len(CL)):\n        S2[s0:s0 + len(chunk), c] = np.bincount(ow, weights=P[:, c], minlength=len(chunk)) / cw2\n\n# average the two rounds: round 1 anchors on the real target, round 2 on pool content\nnp.save('/workspace/work/scores_r2.npy', S2)\nnp.save('/workspace/work/scores_blend.npy', (0.5 * S + 0.5 * S2).astype(np.float32))\nprint('DONE', round(time.time() - t0, 1))\n", "structuredPatch": [], "originalFile": null, "userModified": false}

wiki 4000 pool positives
webnews 4000 pool positives
qa 4000 pool positives
windows: pos 20213 neg 63895 2.7
  step 0 1.3863 0.7597
  step 200 0.2608 0.8594
DONE 68.0

[stdout]
wiki 4000 pool positives
webnews 4000 pool positives
qa 4000 pool positives
windows: pos 20213 neg 63895 2.7
  step 0 1.3863 0.7597
  step 200 0.2608 0.8594
DONE 68.0
[metadata] {"noOutputExpected": false}

work/sel_v6.json shares {'wiki': 0.272, 'webnews': 0.481, 'qa': 0.247} stride {'wiki': 1, 'webnews': 2, 'qa': 1} -> 34913 docs 26.4 M est {'wiki': 7.18, 'webnews': 12.7, 'qa': 6.52}
work/sel_v7.json shares {'wiki': 0.272, 'webnews': 0.481, 'qa': 0.247} stride {'wiki': 1, 'webnews': 2, 'qa': 1} -> 35491 docs 26.4 M est {'wiki': 7.18, 'webnews': 12.7, 'qa': 6.52}
packing

[stdout]
work/sel_v6.json shares {'wiki': 0.272, 'webnews': 0.481, 'qa': 0.247} stride {'wiki': 1, 'webnews': 2, 'qa': 1} -> 34913 docs 26.4 M est {'wiki': 7.18, 'webnews': 12.7, 'qa': 6.52}
work/sel_v7.json shares {'wiki': 0.272, 'webnews': 0.481, 'qa': 0.247} stride {'wiki': 1, 'webnews': 2, 'qa': 1} -> 35491 docs 26.4 M est {'wiki': 7.18, 'webnews': 12.7, 'qa': 6.52}
packing
[metadata] {"noOutputExpected": false}

[1]+  Done                    ( for v in v6 v7;
do
    python3 train_nano.py --train_npy /tmp/tr_$v.npy --heldout_npy data/multi_dev.npy --out_json /tmp/$v.json --max_iters 3000 --seed 1337 2>&1 | grep HELDOUT_PPL | sed "s|^|$v |";
done ) > work/eval67.log 2>&1
v6 HELDOUT_PPL 373.865  loss 5.9239  train_s 50  /tmp/tr_v6.npy
v7 HELDOUT_PPL 381.611  loss 5.9444  train_s 51  /tmp/tr_v7.npy
3397

[stdout]
[1]+  Done                    ( for v in v6 v7;
do
    python3 train_nano.py --train_npy /tmp/tr_$v.npy --heldout_npy data/multi_dev.npy --out_json /tmp/$v.json --max_iters 3000 --seed 1337 2>&1 | grep HELDOUT_PPL | sed "s|^|$v |";
done ) > work/eval67.log 2>&1
v6 HELDOUT_PPL 373.865  loss 5.9239  train_s 50  /tmp/tr_v6.npy
v7 HELDOUT_PPL 381.611  loss 5.9444  train_s 51  /tmp/tr_v7.npy
3397
[metadata] {"noOutputExpected": false}

pool: 182016 docs (7s)
dev target: 2705 docs (10s)
classifier fitted (23s)
pool scored (88s)

[stdout]
pool: 182016 docs (7s)
dev target: 2705 docs (10s)
classifier fitted (23s)
pool scored (88s)
[metadata] {"noOutputExpected": false}