claude-opus-4-8

packed 12000000 tokens from 45000 ids -> /tmp/dev_train.npy
HELDOUT_PPL 457.994  loss 6.1269  train_s 55  /tmp/dev_train.npy
4548

[stdout]
packed 12000000 tokens from 45000 ids -> /tmp/dev_train.npy
HELDOUT_PPL 457.994  loss 6.1269  train_s 55  /tmp/dev_train.npy
4548
[metadata] {"noOutputExpected": false}

n 45000 unique 45000 min 1 max 181948
VALID: unique + in range

[stdout]
n 45000 unique 45000 min 1 max 181948
VALID: unique + in range
[metadata] {"noOutputExpected": false}

after dedup 44629 removed 371

[stdout]
after dedup 44629 removed 371
[metadata] {"noOutputExpected": false}

The file /workspace/submission/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/submission/curate.py", "oldString": "    order = np.argsort(-scores)\n    sel = [int(ids[i]) for i in order[:N_OUT] if scores[i] > -1e8]\n    json.dump(sel, open(OUT, \"w\"))", "newString": "    order = np.argsort(-scores)\n    # emit best-first, dropping near-duplicates (normalised 120-char prefix) so the\n    # 12M-token budget spends on distinct documents rather than reposts/spam.\n    sel, seen = [], set()\n    for i in order:\n        if scores[i] <= -1e8:\n            break\n        key = re.sub(r\"[^a-z0-9]\", \"\", texts[i].lower())[:120]\n        if key in seen:\n            continue\n        seen.add(key)\n        sel.append(int(ids[i]))\n        if len(sel) >= N_OUT:\n            break\n    json.dump(sel, open(OUT, \"w\"))", "originalFile": "\"\"\"Curate the raw web pool for pretraining a small LM whose target is a BROAD\nhigh-quality English distribution (Wikipedia / clean web prose / news / technical Q&A).\n\nCriterion: DSIR-style importance selection (Xie et al. 2023). We build a hashed\nunigram+bigram language model of the disclosed HQ TARGET (decoded from the dev\ntarget tokens, with WikiText tokenization artifacts normalised away) and of the\nRAW POOL. Each pool document is scored by its length-normalised log-likelihood\nratio  mean_ngram log( q_target / p_pool ). Documents whose n-gram statistics\nlook more like the high-quality target than like generic web get higher scores.\nA light quality gate removes tiny/degenerate docs. Output = pool ids ordered by\nscore (best first); the trainer consumes them in order until the token budget.\n\"\"\"\nimport json, re, math, zlib\nimport numpy as np\nfrom transformers import AutoTokenizer\n\nPOOL = \"/workspace/data/pool.jsonl\"\nDEV  = \"/workspace/data/multi_dev.npy\"\nOUT  = \"/workspace/submission/selection.json\"\nB    = 1 << 20            # hash buckets\nALPHA = 1.0              # additive smoothing on probabilities\nN_OUT = 45000            # ids to emit (>> enough to cover 12M tokens)\n\n_word = re.compile(r\"[a-z0-9']+\")\nSTOP = set(\"the of and to a in is that it for as was on with by are be this an at \"\n           \"from or which but not have has had were their they he she his her its \"\n           \"we you i been would will can there more one all\".split())\n\ndef clean_wikitext(s):\n    s = s.replace(\"@-@\", \"-\").replace(\"@,@\", \",\").replace(\"@.@\", \".\")\n    s = re.sub(r\"\\s+([,.;:!?])\", r\"\\1\", s)   # drop space before punctuation\n    return s\n\ndef tokens(s):\n    return _word.findall(s.lower())\n\ndef hashes(words):\n    # unigram + bigram hashed indices\n    out = []\n    prev = None\n    for w in words:\n        out.append(zlib.crc32((\"u\\t\" + w).encode()) & (B - 1))\n        if prev is not None:\n            out.append(zlib.crc32((\"b\\t\" + prev + \"\\t\" + w).encode()) & (B - 1))\n        prev = w\n    return out\n\ndef counts_from_iter(texts):\n    buf, c = [], np.zeros(B, dtype=np.float64)\n    for t in texts:\n        buf.extend(hashes(tokens(t)))\n        if len(buf) > 4_000_000:\n            c += np.bincount(np.array(buf, dtype=np.int64), minlength=B)\n            buf = []\n    if buf:\n        c += np.bincount(np.array(buf, dtype=np.int64), minlength=B)\n    return c\n\ndef main():\n    # ---- TARGET distribution from decoded dev tokens ----\n    tok = AutoTokenizer.from_pretrained(\"gpt2\")\n    dev = np.load(DEV)\n    dev_txt = tok.decode(dev.tolist())\n    dev_docs = [clean_wikitext(d) for d in dev_txt.split(\"<|endoftext|>\") if d.strip()]\n    q = counts_from_iter(dev_docs)\n\n    # ---- POOL distribution + per-doc score ----\n    ids, texts = [], []\n    for line in open(POOL):\n        r = json.loads(line)\n        ids.append(r[\"id\"]); texts.append(r[\"text\"])\n\n    p = np.zeros(B, dtype=np.float64)\n    doc_idx, doc_nw, buf = [], [], []\n    for t in texts:\n        w = tokens(t)\n        idx = hashes(w)\n        a = np.array(idx, dtype=np.uint32)\n        doc_idx.append(a); doc_nw.append(len(w))\n        buf.extend(idx)\n        if len(buf) > 4_000_000:\n            p += np.bincount(np.array(buf, dtype=np.int64), minlength=B); buf = []\n    if buf:\n        p += np.bincount(np.array(buf, dtype=np.int64), minlength=B)\n\n    qp = (q + ALPHA) / (q.sum() + ALPHA * B)\n    pp = (p + ALPHA) / (p.sum() + ALPHA * B)\n    logratio = np.log(qp) - np.log(pp)\n\n    scores = np.full(len(ids), -1e9)\n    for k, a in enumerate(doc_idx):\n        if doc_nw[k] < 50 or len(a) == 0:          # gate: too short\n            continue\n        txt = texts[k]\n        alpha_ratio = sum(c.isalpha() for c in txt) / max(1, len(txt))\n        if alpha_ratio < 0.60:                      # gate: too much markup/symbols\n            continue\n        w = tokens(txt)\n        stop_frac = sum(x in STOP for x in w) / len(w)\n        if stop_frac < 0.20 or stop_frac > 0.60:    # gate: fluent prose band (kills lists/code)\n            continue\n        lines = [ln for ln in txt.split(\"\\n\") if ln.strip()]\n        wpl = len(w) / max(1, len(lines))\n        if wpl < 8:                                  # gate: list-like (few words per line)\n            continue\n        scores[k] = logratio[a.astype(np.int64)].mean()\n\n    order = np.argsort(-scores)\n    sel = [int(ids[i]) for i in order[:N_OUT] if scores[i] > -1e8]\n    json.dump(sel, open(OUT, \"w\"))\n    print(f\"wrote {len(sel)} ids -> {OUT}\")\n\nif __name__ == \"__main__\":\n    main()\n", "structuredPatch": [{"oldStart": 106, "oldLines": 7, "newStart": 106, "newLines": 19, "lines": ["         scores[k] = logratio[a.astype(np.int64)].mean()", " ", "     order = np.argsort(-scores)", "-    sel = [int(ids[i]) for i in order[:N_OUT] if scores[i] > -1e8]", "+    # emit best-first, dropping near-duplicates (normalised 120-char prefix) so the", "+    # 12M-token budget spends on distinct documents rather than reposts/spam.", "+    sel, seen = [], set()", "+    for i in order:", "+        if scores[i] <= -1e8:", "+            break", "+        key = re.sub(r\"[^a-z0-9]\", \"\", texts[i].lower())[:120]", "+        if key in seen:", "+            continue", "+        seen.add(key)", "+        sel.append(int(ids[i]))", "+        if len(sel) >= N_OUT:", "+            break", "     json.dump(sel, open(OUT, \"w\"))", "     print(f\"wrote {len(sel)} ids -> {OUT}\")", " "]}], "userModified": false, "replaceAll": false}

Command running in background with ID: b21ipv6bs. Output is being written to: /tmp/claude-0/-workspace/7dbcc54a-5cf4-4a86-97de-2df43702e8dc/tasks/b21ipv6bs.output. You will be notified when it completes. To check interim output, use Read on that file path.
Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "b21ipv6bs", "backgroundCwdHint": "Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands."}

===
n 45000 uniq 45000 inrange True

[stdout]
===
n 45000 uniq 45000 inrange True
[metadata] {"noOutputExpected": false}

wrote 45000 ids -> /workspace/submission/selection.json

[stdout]
wrote 45000 ids -> /workspace/submission/selection.json
[metadata] {"noOutputExpected": false}

Command running in background with ID: bflabm7pg. Output is being written to: /tmp/claude-0/-workspace/7dbcc54a-5cf4-4a86-97de-2df43702e8dc/tasks/bflabm7pg.output. You will be notified when it completes. To check interim output, use Read on that file path.
Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "bflabm7pg", "backgroundCwdHint": "Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands."}

packed 12000000 tokens from 45000 ids -> /tmp/dev_train.npy
HELDOUT_PPL 358.515  loss 5.8820  train_s 55  /tmp/dev_train.npy

[stdout]
packed 12000000 tokens from 45000 ids -> /tmp/dev_train.npy
HELDOUT_PPL 358.515  loss 5.8820  train_s 55  /tmp/dev_train.npy
[metadata] {"noOutputExpected": false}