claude-opus-4-8

2

[stdout]
2
[metadata] {"noOutputExpected": false}

ranking + balancing + dedup ...
wrote 29826 ids (~30000165 est tokens) -> /workspace/submission/selection.json
bucket token shares: {0: 0.5, 1: 0.25, 2: 0.25} (0=other 1=tech 2=news)
0

[stdout]
ranking + balancing + dedup ...
wrote 29826 ids (~30000165 est tokens) -> /workspace/submission/selection.json
bucket token shares: {0: 0.5, 1: 0.25, 2: 0.25} (0=other 1=tech 2=news)
0
[metadata] {"noOutputExpected": false}

Command running in background with ID: b4gopsmq8. Output is being written to: /tmp/claude-0/-workspace/b3699ac4-1c9f-4c54-b65c-10b5011c1085/tasks/b4gopsmq8.output. You will be notified when it completes. To check interim output, use Read on that file path.
Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "b4gopsmq8", "backgroundCwdHint": "Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands."}

packed 12000000 tokens from 29826 ids -> /tmp/dev_train.npy
HELDOUT_PPL 415.056  loss 6.0284  train_s 56  /tmp/dev_train.npy
2315
2295

[stdout]
packed 12000000 tokens from 29826 ids -> /tmp/dev_train.npy
HELDOUT_PPL 415.056  loss 6.0284  train_s 56  /tmp/dev_train.npy
2315
2295
[metadata] {"noOutputExpected": false}

The file /workspace/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/curate.py", "oldString": "        bucket[j] = 2 if news_score(tl) >= 2 else 0", "newString": "        bucket[j] = 0     # non-tech prose (news + encyclopedic + general web)", "originalFile": "\"\"\"Curate the raw web pool for a small LM whose target is a BROAD, multi-domain\nhigh-quality English distribution (Wikipedia + high-quality web prose + news +\ntechnical Q&A).\n\nStated, reproducible criterion: a QUALITY/DOMAIN classifier.\nWe train a logistic-regression classifier on L2-normalized hashed unigram+bigram\nword features to separate\n  positive = the disclosed HQ target (multi_dev.npy decoded into documents), from\n  negative = random raw pool documents.\nEvery pool document is scored by the classifier's decision value = \"how\ntarget-like\". Documents are ranked highest-first, subject to light quality gates\n(length floor, alpha-fraction, English-only) and near-duplicate removal.\n\nREGISTER BALANCING (key step): the target is EQUAL PARTS four registers, one of\nwhich is technical Q&A / code. A plain top-by-score selection floats news/prose\nto the top and starves the technical register (<0.2% of the budget), leaving a\nwhole quarter of the target badly modeled. So we detect technical documents with\na code-signal heuristic and interleave the two ranked lists to guarantee the\ntechnical register ~TECH_SHARE of the 12M-token budget, emitting the merged\npriority order until well past budget.\n\nLength-robustness: features are L2-normalized, so score reflects the *direction*\nof a document's n-gram profile, not its length or raw repetition — this avoids\nthe failure mode of raw log-likelihood-ratio ranking, which floats generic\nnavigation/list spam to the top.\n\nDev perplexity (frozen train_nano.py, multi_dev target): random selection 474.9\n-> classifier 412.3 -> +English gate 410.8 -> +25% tech 407.5 -> +35% tech 406.8.\n\"\"\"\nimport json, re, zlib, numpy as np\nfrom transformers import AutoTokenizer\n\nPOOL = \"/workspace/data/pool.jsonl\"\nDEV  = \"/workspace/data/multi_dev.npy\"\nOUT  = \"/workspace/submission/selection.json\"\n\nB = 1 << 20                 # hash buckets\nNEG_SAMPLE = 24000          # random pool docs used as negatives\nEPOCHS = 6\nLR = 0.5\nL2 = 1e-6\nSEED = 0\n\n_word = re.compile(r\"[a-z0-9]+\")\ndef words_of(s):\n    return _word.findall(s.lower())\n\n# small English function-word set — fluent prose is rich in these; navigation\n# menus / name lists / license-plate spam are not. A cheap, robust fluency gate.\nSTOP = set(\"the of and to a in is that it for was on as with by at be this from or \"\n           \"an are not but he she they we you his her their which have has had were \"\n           \"been i who what when where how all would there been more one about\".split())\n\ndef english_frac(s):\n    \"\"\"Fraction of ASCII latin letters among all alphabetic characters (0..1).\"\"\"\n    a = ascii_ = 0\n    for c in s:\n        if c.isalpha():\n            a += 1\n            if 'a' <= c <= 'z' or 'A' <= c <= 'Z':\n                ascii_ += 1\n    return ascii_ / max(1, a)\n\ndef crc(b):\n    return zlib.crc32(b) & (B - 1)\n\ndef featvec(words):\n    \"\"\"Return (unique_bucket_ids int32, L2-normalized float32 values) for\n    unigram+bigram hashed features of a word list.\"\"\"\n    if not words:\n        return np.empty(0, np.int32), np.empty(0, np.float32)\n    h = [crc(w.encode()) for w in words]\n    for i in range(len(words) - 1):\n        h.append(crc((words[i] + \"\\x00\" + words[i+1]).encode()))\n    h = np.asarray(h, dtype=np.int64)\n    idx, cnt = np.unique(h, return_counts=True)\n    v = cnt.astype(np.float32)\n    v /= np.sqrt((v * v).sum())\n    return idx.astype(np.int32), v\n\n# ---------- 1. build training features ----------\nprint(\"decoding target ...\")\ntok = AutoTokenizer.from_pretrained(\"gpt2\")\ndev = np.load(DEV)\neos = tok.eos_token_id\narr = dev.tolist()\n# split target into documents on EOS\npos = [i for i, t in enumerate(arr) if t == eos]\nsegs, prev = [], 0\nfor p in pos:\n    if p - prev > 5:\n        segs.append(arr[prev:p])\n    prev = p + 1\nif len(arr) - prev > 5:\n    segs.append(arr[prev:])\npos_feats = [featvec(words_of(tok.decode(s))) for s in segs]\nprint(\"target docs:\", len(pos_feats))\n\nprint(\"loading pool ...\")\nids, texts = [], []\nfor line in open(POOL):\n    r = json.loads(line)\n    ids.append(r[\"id\"]); texts.append(r[\"text\"])\nN = len(ids)\nprint(\"pool docs:\", N)\n\nrng = np.random.default_rng(SEED)\nneg_idx = rng.choice(N, size=min(NEG_SAMPLE, N), replace=False)\nprint(\"building negative features ...\")\nneg_feats = [featvec(words_of(texts[j])) for j in neg_idx]\n\n# ---------- 2. train logistic regression (sparse SGD) ----------\nprint(\"training classifier ...\")\nw = np.zeros(B, dtype=np.float32)\nb = 0.0\ntrain = [(f, 1.0) for f in pos_feats] + [(f, 0.0) for f in neg_feats]\norder = np.arange(len(train))\nfor ep in range(EPOCHS):\n    rng.shuffle(order)\n    lr = LR * (1.0 - ep / (EPOCHS + 1))\n    loss = 0.0\n    for oi in order:\n        (idx, v), y = train[oi]\n        if idx.size == 0:\n            continue\n        z = float(w[idx] @ v) + b\n        p = 1.0 / (1.0 + np.exp(-z))\n        g = p - y\n        w[idx] -= lr * (g * v + L2 * w[idx])\n        b -= lr * g\n        loss += -(y*np.log(p+1e-9) + (1-y)*np.log(1-p+1e-9))\n    print(f\"  epoch {ep} loss {loss/len(train):.4f}\")\n\n# The target is EQUAL PARTS four registers incl. technical Q&A / code. Detect the\n# technical register so we can guarantee it a token quota — otherwise the plain\n# classifier floats news/prose to the top and code is starved (<0.2% of budget),\n# leaving a whole quarter of the target badly modeled.\nTECH_SIG = ['<pre', '<code', 'stack overflow', 'stackoverflow', 'function(', 'var ',\n            'import ', 'public static', '#include', 'console.log', 'println', '</code',\n            '&lt;', 'def ', 'npm ', 'printf', 'const ', 'return ', ');']\ndef tech_score(tl):\n    return sum(1 for m in TECH_SIG if m in tl)\n\n# News cue heuristic — the pool is heavy on news portals, so a plain ranking\n# over-weights news (~48% of the head) vs. the target's ~25%, starving the\n# encyclopedic/prose registers. Detect news so we can cap its budget share too.\nNEWS_SIG = [' said ', ' told reporters', 'reuters', '(ap)', 'associated press',\n            'on monday', 'on tuesday', 'on wednesday', 'on thursday', 'on friday',\n            'on saturday', 'on sunday', 'the minister', 'prime minister', 'the police',\n            'according to', 'officials said', 'spokesman', 'spokesperson']\ndef news_score(tl):\n    return sum(1 for m in NEWS_SIG if m in tl)\n\n# ---------- 3. score every document, with register-aware gates ----------\nprint(\"scoring documents ...\")\nscores = np.full(N, -1e9, dtype=np.float64)\nntok = np.zeros(N, dtype=np.int32)\nbucket = np.zeros(N, dtype=np.int8)               # 0=other, 1=tech, 2=news\nfor j in range(N):\n    t = texts[j]\n    tl = t.lower()\n    wds = words_of(t)\n    n = len(wds)\n    ntok[j] = n\n    if n < 50:                                    # length floor\n        continue\n    head = t[:3000]\n    tech = tech_score(tl) >= 2\n    alpha = sum(c.isalpha() for c in head) / max(1, len(head))\n    if tech:\n        if alpha < 0.30:                          # code is symbol-heavy: relax gate\n            continue\n        bucket[j] = 1\n    else:\n        if alpha < 0.55 or english_frac(head) < 0.85:\n            continue                              # prose gates: clean English only\n        bucket[j] = 2 if news_score(tl) >= 2 else 0\n    idx, v = featvec(wds)\n    scores[j] = float(w[idx] @ v) + b\n    if j % 40000 == 0:\n        print(f\"  {j}/{N}\")\n\n# ---------- 4. rank each register, merge to a balanced priority order ----------\n# Target is ~equal parts {encyclopedic, prose, news, tech}. We can cleanly detect\n# tech and news; \"other\" holds encyclopedic+prose (two registers). Quotas below\n# cap news/tech near their 1/4 shares and let \"other\" carry the remaining half.\nprint(\"ranking + balancing + dedup ...\")\nBUDGET = 12_000_000\nQUOTA = {1: 0.25, 2: 0.25, 0: 0.50}   # bucket -> token share of budget\norder = np.argsort(-scores)\nqueues = {b: [j for j in order if scores[j] > -1e8 and bucket[j] == b] for b in (0, 1, 2)}\nptr = {0: 0, 1: 0, 2: 0}\nused = {0: 0, 1: 0, 2: 0}\n\ndef est(j):\n    return int(ntok[j] * 1.3) + 1\n\nseen = set()\ndef fresh(j):\n    wl = words_of(texts[j])\n    key = (hash(\" \".join(wl[:60])), len(wl) // 40)\n    if key in seen:\n        return False\n    seen.add(key)\n    return True\n\nselection, tot = [], 0\nwhile tot < BUDGET * 2.5:\n    # pick the bucket that is furthest *below* its quota and still has documents\n    cand = [b for b in (0, 1, 2) if ptr[b] < len(queues[b])]\n    if not cand:\n        break\n    b = min(cand, key=lambda b: used[b] - QUOTA[b] * max(1, tot))\n    j = queues[b][ptr[b]]; ptr[b] += 1\n    if not fresh(j):\n        continue\n    e = est(j)\n    selection.append(int(ids[j])); tot += e; used[b] += e\n\njson.dump(selection, open(OUT, \"w\"))\nprint(f\"wrote {len(selection)} ids (~{tot} est tokens) -> {OUT}\")\nprint(\"bucket token shares:\", {b: round(used[b] / max(1, tot), 3) for b in (0, 1, 2)},\n      \"(0=other 1=tech 2=news)\")\n", "structuredPatch": [{"oldStart": 174, "oldLines": 7, "newStart": 174, "newLines": 7, "lines": ["     else:", "         if alpha < 0.55 or english_frac(head) < 0.85:", "             continue                              # prose gates: clean English only", "-        bucket[j] = 2 if news_score(tl) >= 2 else 0", "+        bucket[j] = 0     # non-tech prose (news + encyclopedic + general web)", "     idx, v = featvec(wds)", "     scores[j] = float(w[idx] @ v) + b", "     if j % 40000 == 0:"]}], "userModified": false, "replaceAll": false}

The file /workspace/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/curate.py", "oldString": "# Target is ~equal parts {encyclopedic, prose, news, tech}. We can cleanly detect\n# tech and news; \"other\" holds encyclopedic+prose (two registers). Quotas below\n# cap news/tech near their 1/4 shares and let \"other\" carry the remaining half.\nprint(\"ranking + balancing + dedup ...\")\nBUDGET = 12_000_000\nQUOTA = {1: 0.25, 2: 0.25, 0: 0.50}   # bucket -> token share of budget\norder = np.argsort(-scores)\nqueues = {b: [j for j in order if scores[j] > -1e8 and bucket[j] == b] for b in (0, 1, 2)}\nptr = {0: 0, 1: 0, 2: 0}\nused = {0: 0, 1: 0, 2: 0}", "newString": "# Target is ~equal parts {encyclopedic, prose, news, tech}. The plain classifier\n# starves the technical register, so we give it a fixed token quota; the rest of\n# the budget (news + encyclopedic + general prose) is filled by pure classifier\n# score. A NEWS cap was tried and REGRESSED (dev 406.8 -> 415.1): news-register\n# prose is genuinely valuable for this target, so we do not cap it.\nprint(\"ranking + balancing + dedup ...\")\nBUDGET = 12_000_000\nQUOTA = {1: 0.35, 0: 0.65}             # bucket -> token share (1=tech, 0=rest)\nBUCKETS = (0, 1)\norder = np.argsort(-scores)\nqueues = {b: [j for j in order if scores[j] > -1e8 and bucket[j] == b] for b in BUCKETS}\nptr = {b: 0 for b in BUCKETS}\nused = {b: 0 for b in BUCKETS}", "originalFile": "\"\"\"Curate the raw web pool for a small LM whose target is a BROAD, multi-domain\nhigh-quality English distribution (Wikipedia + high-quality web prose + news +\ntechnical Q&A).\n\nStated, reproducible criterion: a QUALITY/DOMAIN classifier.\nWe train a logistic-regression classifier on L2-normalized hashed unigram+bigram\nword features to separate\n  positive = the disclosed HQ target (multi_dev.npy decoded into documents), from\n  negative = random raw pool documents.\nEvery pool document is scored by the classifier's decision value = \"how\ntarget-like\". Documents are ranked highest-first, subject to light quality gates\n(length floor, alpha-fraction, English-only) and near-duplicate removal.\n\nREGISTER BALANCING (key step): the target is EQUAL PARTS four registers, one of\nwhich is technical Q&A / code. A plain top-by-score selection floats news/prose\nto the top and starves the technical register (<0.2% of the budget), leaving a\nwhole quarter of the target badly modeled. So we detect technical documents with\na code-signal heuristic and interleave the two ranked lists to guarantee the\ntechnical register ~TECH_SHARE of the 12M-token budget, emitting the merged\npriority order until well past budget.\n\nLength-robustness: features are L2-normalized, so score reflects the *direction*\nof a document's n-gram profile, not its length or raw repetition — this avoids\nthe failure mode of raw log-likelihood-ratio ranking, which floats generic\nnavigation/list spam to the top.\n\nDev perplexity (frozen train_nano.py, multi_dev target): random selection 474.9\n-> classifier 412.3 -> +English gate 410.8 -> +25% tech 407.5 -> +35% tech 406.8.\n\"\"\"\nimport json, re, zlib, numpy as np\nfrom transformers import AutoTokenizer\n\nPOOL = \"/workspace/data/pool.jsonl\"\nDEV  = \"/workspace/data/multi_dev.npy\"\nOUT  = \"/workspace/submission/selection.json\"\n\nB = 1 << 20                 # hash buckets\nNEG_SAMPLE = 24000          # random pool docs used as negatives\nEPOCHS = 6\nLR = 0.5\nL2 = 1e-6\nSEED = 0\n\n_word = re.compile(r\"[a-z0-9]+\")\ndef words_of(s):\n    return _word.findall(s.lower())\n\n# small English function-word set — fluent prose is rich in these; navigation\n# menus / name lists / license-plate spam are not. A cheap, robust fluency gate.\nSTOP = set(\"the of and to a in is that it for was on as with by at be this from or \"\n           \"an are not but he she they we you his her their which have has had were \"\n           \"been i who what when where how all would there been more one about\".split())\n\ndef english_frac(s):\n    \"\"\"Fraction of ASCII latin letters among all alphabetic characters (0..1).\"\"\"\n    a = ascii_ = 0\n    for c in s:\n        if c.isalpha():\n            a += 1\n            if 'a' <= c <= 'z' or 'A' <= c <= 'Z':\n                ascii_ += 1\n    return ascii_ / max(1, a)\n\ndef crc(b):\n    return zlib.crc32(b) & (B - 1)\n\ndef featvec(words):\n    \"\"\"Return (unique_bucket_ids int32, L2-normalized float32 values) for\n    unigram+bigram hashed features of a word list.\"\"\"\n    if not words:\n        return np.empty(0, np.int32), np.empty(0, np.float32)\n    h = [crc(w.encode()) for w in words]\n    for i in range(len(words) - 1):\n        h.append(crc((words[i] + \"\\x00\" + words[i+1]).encode()))\n    h = np.asarray(h, dtype=np.int64)\n    idx, cnt = np.unique(h, return_counts=True)\n    v = cnt.astype(np.float32)\n    v /= np.sqrt((v * v).sum())\n    return idx.astype(np.int32), v\n\n# ---------- 1. build training features ----------\nprint(\"decoding target ...\")\ntok = AutoTokenizer.from_pretrained(\"gpt2\")\ndev = np.load(DEV)\neos = tok.eos_token_id\narr = dev.tolist()\n# split target into documents on EOS\npos = [i for i, t in enumerate(arr) if t == eos]\nsegs, prev = [], 0\nfor p in pos:\n    if p - prev > 5:\n        segs.append(arr[prev:p])\n    prev = p + 1\nif len(arr) - prev > 5:\n    segs.append(arr[prev:])\npos_feats = [featvec(words_of(tok.decode(s))) for s in segs]\nprint(\"target docs:\", len(pos_feats))\n\nprint(\"loading pool ...\")\nids, texts = [], []\nfor line in open(POOL):\n    r = json.loads(line)\n    ids.append(r[\"id\"]); texts.append(r[\"text\"])\nN = len(ids)\nprint(\"pool docs:\", N)\n\nrng = np.random.default_rng(SEED)\nneg_idx = rng.choice(N, size=min(NEG_SAMPLE, N), replace=False)\nprint(\"building negative features ...\")\nneg_feats = [featvec(words_of(texts[j])) for j in neg_idx]\n\n# ---------- 2. train logistic regression (sparse SGD) ----------\nprint(\"training classifier ...\")\nw = np.zeros(B, dtype=np.float32)\nb = 0.0\ntrain = [(f, 1.0) for f in pos_feats] + [(f, 0.0) for f in neg_feats]\norder = np.arange(len(train))\nfor ep in range(EPOCHS):\n    rng.shuffle(order)\n    lr = LR * (1.0 - ep / (EPOCHS + 1))\n    loss = 0.0\n    for oi in order:\n        (idx, v), y = train[oi]\n        if idx.size == 0:\n            continue\n        z = float(w[idx] @ v) + b\n        p = 1.0 / (1.0 + np.exp(-z))\n        g = p - y\n        w[idx] -= lr * (g * v + L2 * w[idx])\n        b -= lr * g\n        loss += -(y*np.log(p+1e-9) + (1-y)*np.log(1-p+1e-9))\n    print(f\"  epoch {ep} loss {loss/len(train):.4f}\")\n\n# The target is EQUAL PARTS four registers incl. technical Q&A / code. Detect the\n# technical register so we can guarantee it a token quota — otherwise the plain\n# classifier floats news/prose to the top and code is starved (<0.2% of budget),\n# leaving a whole quarter of the target badly modeled.\nTECH_SIG = ['<pre', '<code', 'stack overflow', 'stackoverflow', 'function(', 'var ',\n            'import ', 'public static', '#include', 'console.log', 'println', '</code',\n            '&lt;', 'def ', 'npm ', 'printf', 'const ', 'return ', ');']\ndef tech_score(tl):\n    return sum(1 for m in TECH_SIG if m in tl)\n\n# News cue heuristic — the pool is heavy on news portals, so a plain ranking\n# over-weights news (~48% of the head) vs. the target's ~25%, starving the\n# encyclopedic/prose registers. Detect news so we can cap its budget share too.\nNEWS_SIG = [' said ', ' told reporters', 'reuters', '(ap)', 'associated press',\n            'on monday', 'on tuesday', 'on wednesday', 'on thursday', 'on friday',\n            'on saturday', 'on sunday', 'the minister', 'prime minister', 'the police',\n            'according to', 'officials said', 'spokesman', 'spokesperson']\ndef news_score(tl):\n    return sum(1 for m in NEWS_SIG if m in tl)\n\n# ---------- 3. score every document, with register-aware gates ----------\nprint(\"scoring documents ...\")\nscores = np.full(N, -1e9, dtype=np.float64)\nntok = np.zeros(N, dtype=np.int32)\nbucket = np.zeros(N, dtype=np.int8)               # 0=other, 1=tech, 2=news\nfor j in range(N):\n    t = texts[j]\n    tl = t.lower()\n    wds = words_of(t)\n    n = len(wds)\n    ntok[j] = n\n    if n < 50:                                    # length floor\n        continue\n    head = t[:3000]\n    tech = tech_score(tl) >= 2\n    alpha = sum(c.isalpha() for c in head) / max(1, len(head))\n    if tech:\n        if alpha < 0.30:                          # code is symbol-heavy: relax gate\n            continue\n        bucket[j] = 1\n    else:\n        if alpha < 0.55 or english_frac(head) < 0.85:\n            continue                              # prose gates: clean English only\n        bucket[j] = 0     # non-tech prose (news + encyclopedic + general web)\n    idx, v = featvec(wds)\n    scores[j] = float(w[idx] @ v) + b\n    if j % 40000 == 0:\n        print(f\"  {j}/{N}\")\n\n# ---------- 4. rank each register, merge to a balanced priority order ----------\n# Target is ~equal parts {encyclopedic, prose, news, tech}. We can cleanly detect\n# tech and news; \"other\" holds encyclopedic+prose (two registers). Quotas below\n# cap news/tech near their 1/4 shares and let \"other\" carry the remaining half.\nprint(\"ranking + balancing + dedup ...\")\nBUDGET = 12_000_000\nQUOTA = {1: 0.25, 2: 0.25, 0: 0.50}   # bucket -> token share of budget\norder = np.argsort(-scores)\nqueues = {b: [j for j in order if scores[j] > -1e8 and bucket[j] == b] for b in (0, 1, 2)}\nptr = {0: 0, 1: 0, 2: 0}\nused = {0: 0, 1: 0, 2: 0}\n\ndef est(j):\n    return int(ntok[j] * 1.3) + 1\n\nseen = set()\ndef fresh(j):\n    wl = words_of(texts[j])\n    key = (hash(\" \".join(wl[:60])), len(wl) // 40)\n    if key in seen:\n        return False\n    seen.add(key)\n    return True\n\nselection, tot = [], 0\nwhile tot < BUDGET * 2.5:\n    # pick the bucket that is furthest *below* its quota and still has documents\n    cand = [b for b in (0, 1, 2) if ptr[b] < len(queues[b])]\n    if not cand:\n        break\n    b = min(cand, key=lambda b: used[b] - QUOTA[b] * max(1, tot))\n    j = queues[b][ptr[b]]; ptr[b] += 1\n    if not fresh(j):\n        continue\n    e = est(j)\n    selection.append(int(ids[j])); tot += e; used[b] += e\n\njson.dump(selection, open(OUT, \"w\"))\nprint(f\"wrote {len(selection)} ids (~{tot} est tokens) -> {OUT}\")\nprint(\"bucket token shares:\", {b: round(used[b] / max(1, tot), 3) for b in (0, 1, 2)},\n      \"(0=other 1=tech 2=news)\")\n", "structuredPatch": [{"oldStart": 181, "oldLines": 16, "newStart": 181, "newLines": 19, "lines": ["         print(f\"  {j}/{N}\")", " ", " # ---------- 4. rank each register, merge to a balanced priority order ----------", "-# Target is ~equal parts {encyclopedic, prose, news, tech}. We can cleanly detect", "-# tech and news; \"other\" holds encyclopedic+prose (two registers). Quotas below", "-# cap news/tech near their 1/4 shares and let \"other\" carry the remaining half.", "+# Target is ~equal parts {encyclopedic, prose, news, tech}. The plain classifier", "+# starves the technical register, so we give it a fixed token quota; the rest of", "+# the budget (news + encyclopedic + general prose) is filled by pure classifier", "+# score. A NEWS cap was tried and REGRESSED (dev 406.8 -> 415.1): news-register", "+# prose is genuinely valuable for this target, so we do not cap it.", " print(\"ranking + balancing + dedup ...\")", " BUDGET = 12_000_000", "-QUOTA = {1: 0.25, 2: 0.25, 0: 0.50}   # bucket -> token share of budget", "+QUOTA = {1: 0.35, 0: 0.65}             # bucket -> token share (1=tech, 0=rest)", "+BUCKETS = (0, 1)", " order = np.argsort(-scores)", "-queues = {b: [j for j in order if scores[j] > -1e8 and bucket[j] == b] for b in (0, 1, 2)}", "-ptr = {0: 0, 1: 0, 2: 0}", "-used = {0: 0, 1: 0, 2: 0}", "+queues = {b: [j for j in order if scores[j] > -1e8 and bucket[j] == b] for b in BUCKETS}", "+ptr = {b: 0 for b in BUCKETS}", "+used = {b: 0 for b in BUCKETS}", " ", " def est(j):", "     return int(ntok[j] * 1.3) + 1"]}], "userModified": false, "replaceAll": false}

The file /workspace/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/curate.py", "oldString": "    cand = [b for b in (0, 1, 2) if ptr[b] < len(queues[b])]\n    if not cand:\n        break\n    b = min(cand, key=lambda b: used[b] - QUOTA[b] * max(1, tot))", "newString": "    cand = [b for b in BUCKETS if ptr[b] < len(queues[b])]\n    if not cand:\n        break\n    b = min(cand, key=lambda b: used[b] - QUOTA[b] * max(1, tot))", "originalFile": "\"\"\"Curate the raw web pool for a small LM whose target is a BROAD, multi-domain\nhigh-quality English distribution (Wikipedia + high-quality web prose + news +\ntechnical Q&A).\n\nStated, reproducible criterion: a QUALITY/DOMAIN classifier.\nWe train a logistic-regression classifier on L2-normalized hashed unigram+bigram\nword features to separate\n  positive = the disclosed HQ target (multi_dev.npy decoded into documents), from\n  negative = random raw pool documents.\nEvery pool document is scored by the classifier's decision value = \"how\ntarget-like\". Documents are ranked highest-first, subject to light quality gates\n(length floor, alpha-fraction, English-only) and near-duplicate removal.\n\nREGISTER BALANCING (key step): the target is EQUAL PARTS four registers, one of\nwhich is technical Q&A / code. A plain top-by-score selection floats news/prose\nto the top and starves the technical register (<0.2% of the budget), leaving a\nwhole quarter of the target badly modeled. So we detect technical documents with\na code-signal heuristic and interleave the two ranked lists to guarantee the\ntechnical register ~TECH_SHARE of the 12M-token budget, emitting the merged\npriority order until well past budget.\n\nLength-robustness: features are L2-normalized, so score reflects the *direction*\nof a document's n-gram profile, not its length or raw repetition — this avoids\nthe failure mode of raw log-likelihood-ratio ranking, which floats generic\nnavigation/list spam to the top.\n\nDev perplexity (frozen train_nano.py, multi_dev target): random selection 474.9\n-> classifier 412.3 -> +English gate 410.8 -> +25% tech 407.5 -> +35% tech 406.8.\n\"\"\"\nimport json, re, zlib, numpy as np\nfrom transformers import AutoTokenizer\n\nPOOL = \"/workspace/data/pool.jsonl\"\nDEV  = \"/workspace/data/multi_dev.npy\"\nOUT  = \"/workspace/submission/selection.json\"\n\nB = 1 << 20                 # hash buckets\nNEG_SAMPLE = 24000          # random pool docs used as negatives\nEPOCHS = 6\nLR = 0.5\nL2 = 1e-6\nSEED = 0\n\n_word = re.compile(r\"[a-z0-9]+\")\ndef words_of(s):\n    return _word.findall(s.lower())\n\n# small English function-word set — fluent prose is rich in these; navigation\n# menus / name lists / license-plate spam are not. A cheap, robust fluency gate.\nSTOP = set(\"the of and to a in is that it for was on as with by at be this from or \"\n           \"an are not but he she they we you his her their which have has had were \"\n           \"been i who what when where how all would there been more one about\".split())\n\ndef english_frac(s):\n    \"\"\"Fraction of ASCII latin letters among all alphabetic characters (0..1).\"\"\"\n    a = ascii_ = 0\n    for c in s:\n        if c.isalpha():\n            a += 1\n            if 'a' <= c <= 'z' or 'A' <= c <= 'Z':\n                ascii_ += 1\n    return ascii_ / max(1, a)\n\ndef crc(b):\n    return zlib.crc32(b) & (B - 1)\n\ndef featvec(words):\n    \"\"\"Return (unique_bucket_ids int32, L2-normalized float32 values) for\n    unigram+bigram hashed features of a word list.\"\"\"\n    if not words:\n        return np.empty(0, np.int32), np.empty(0, np.float32)\n    h = [crc(w.encode()) for w in words]\n    for i in range(len(words) - 1):\n        h.append(crc((words[i] + \"\\x00\" + words[i+1]).encode()))\n    h = np.asarray(h, dtype=np.int64)\n    idx, cnt = np.unique(h, return_counts=True)\n    v = cnt.astype(np.float32)\n    v /= np.sqrt((v * v).sum())\n    return idx.astype(np.int32), v\n\n# ---------- 1. build training features ----------\nprint(\"decoding target ...\")\ntok = AutoTokenizer.from_pretrained(\"gpt2\")\ndev = np.load(DEV)\neos = tok.eos_token_id\narr = dev.tolist()\n# split target into documents on EOS\npos = [i for i, t in enumerate(arr) if t == eos]\nsegs, prev = [], 0\nfor p in pos:\n    if p - prev > 5:\n        segs.append(arr[prev:p])\n    prev = p + 1\nif len(arr) - prev > 5:\n    segs.append(arr[prev:])\npos_feats = [featvec(words_of(tok.decode(s))) for s in segs]\nprint(\"target docs:\", len(pos_feats))\n\nprint(\"loading pool ...\")\nids, texts = [], []\nfor line in open(POOL):\n    r = json.loads(line)\n    ids.append(r[\"id\"]); texts.append(r[\"text\"])\nN = len(ids)\nprint(\"pool docs:\", N)\n\nrng = np.random.default_rng(SEED)\nneg_idx = rng.choice(N, size=min(NEG_SAMPLE, N), replace=False)\nprint(\"building negative features ...\")\nneg_feats = [featvec(words_of(texts[j])) for j in neg_idx]\n\n# ---------- 2. train logistic regression (sparse SGD) ----------\nprint(\"training classifier ...\")\nw = np.zeros(B, dtype=np.float32)\nb = 0.0\ntrain = [(f, 1.0) for f in pos_feats] + [(f, 0.0) for f in neg_feats]\norder = np.arange(len(train))\nfor ep in range(EPOCHS):\n    rng.shuffle(order)\n    lr = LR * (1.0 - ep / (EPOCHS + 1))\n    loss = 0.0\n    for oi in order:\n        (idx, v), y = train[oi]\n        if idx.size == 0:\n            continue\n        z = float(w[idx] @ v) + b\n        p = 1.0 / (1.0 + np.exp(-z))\n        g = p - y\n        w[idx] -= lr * (g * v + L2 * w[idx])\n        b -= lr * g\n        loss += -(y*np.log(p+1e-9) + (1-y)*np.log(1-p+1e-9))\n    print(f\"  epoch {ep} loss {loss/len(train):.4f}\")\n\n# The target is EQUAL PARTS four registers incl. technical Q&A / code. Detect the\n# technical register so we can guarantee it a token quota — otherwise the plain\n# classifier floats news/prose to the top and code is starved (<0.2% of budget),\n# leaving a whole quarter of the target badly modeled.\nTECH_SIG = ['<pre', '<code', 'stack overflow', 'stackoverflow', 'function(', 'var ',\n            'import ', 'public static', '#include', 'console.log', 'println', '</code',\n            '&lt;', 'def ', 'npm ', 'printf', 'const ', 'return ', ');']\ndef tech_score(tl):\n    return sum(1 for m in TECH_SIG if m in tl)\n\n# News cue heuristic — the pool is heavy on news portals, so a plain ranking\n# over-weights news (~48% of the head) vs. the target's ~25%, starving the\n# encyclopedic/prose registers. Detect news so we can cap its budget share too.\nNEWS_SIG = [' said ', ' told reporters', 'reuters', '(ap)', 'associated press',\n            'on monday', 'on tuesday', 'on wednesday', 'on thursday', 'on friday',\n            'on saturday', 'on sunday', 'the minister', 'prime minister', 'the police',\n            'according to', 'officials said', 'spokesman', 'spokesperson']\ndef news_score(tl):\n    return sum(1 for m in NEWS_SIG if m in tl)\n\n# ---------- 3. score every document, with register-aware gates ----------\nprint(\"scoring documents ...\")\nscores = np.full(N, -1e9, dtype=np.float64)\nntok = np.zeros(N, dtype=np.int32)\nbucket = np.zeros(N, dtype=np.int8)               # 0=other, 1=tech, 2=news\nfor j in range(N):\n    t = texts[j]\n    tl = t.lower()\n    wds = words_of(t)\n    n = len(wds)\n    ntok[j] = n\n    if n < 50:                                    # length floor\n        continue\n    head = t[:3000]\n    tech = tech_score(tl) >= 2\n    alpha = sum(c.isalpha() for c in head) / max(1, len(head))\n    if tech:\n        if alpha < 0.30:                          # code is symbol-heavy: relax gate\n            continue\n        bucket[j] = 1\n    else:\n        if alpha < 0.55 or english_frac(head) < 0.85:\n            continue                              # prose gates: clean English only\n        bucket[j] = 0     # non-tech prose (news + encyclopedic + general web)\n    idx, v = featvec(wds)\n    scores[j] = float(w[idx] @ v) + b\n    if j % 40000 == 0:\n        print(f\"  {j}/{N}\")\n\n# ---------- 4. rank each register, merge to a balanced priority order ----------\n# Target is ~equal parts {encyclopedic, prose, news, tech}. The plain classifier\n# starves the technical register, so we give it a fixed token quota; the rest of\n# the budget (news + encyclopedic + general prose) is filled by pure classifier\n# score. A NEWS cap was tried and REGRESSED (dev 406.8 -> 415.1): news-register\n# prose is genuinely valuable for this target, so we do not cap it.\nprint(\"ranking + balancing + dedup ...\")\nBUDGET = 12_000_000\nQUOTA = {1: 0.35, 0: 0.65}             # bucket -> token share (1=tech, 0=rest)\nBUCKETS = (0, 1)\norder = np.argsort(-scores)\nqueues = {b: [j for j in order if scores[j] > -1e8 and bucket[j] == b] for b in BUCKETS}\nptr = {b: 0 for b in BUCKETS}\nused = {b: 0 for b in BUCKETS}\n\ndef est(j):\n    return int(ntok[j] * 1.3) + 1\n\nseen = set()\ndef fresh(j):\n    wl = words_of(texts[j])\n    key = (hash(\" \".join(wl[:60])), len(wl) // 40)\n    if key in seen:\n        return False\n    seen.add(key)\n    return True\n\nselection, tot = [], 0\nwhile tot < BUDGET * 2.5:\n    # pick the bucket that is furthest *below* its quota and still has documents\n    cand = [b for b in (0, 1, 2) if ptr[b] < len(queues[b])]\n    if not cand:\n        break\n    b = min(cand, key=lambda b: used[b] - QUOTA[b] * max(1, tot))\n    j = queues[b][ptr[b]]; ptr[b] += 1\n    if not fresh(j):\n        continue\n    e = est(j)\n    selection.append(int(ids[j])); tot += e; used[b] += e\n\njson.dump(selection, open(OUT, \"w\"))\nprint(f\"wrote {len(selection)} ids (~{tot} est tokens) -> {OUT}\")\nprint(\"bucket token shares:\", {b: round(used[b] / max(1, tot), 3) for b in (0, 1, 2)},\n      \"(0=other 1=tech 2=news)\")\n", "structuredPatch": [{"oldStart": 210, "oldLines": 7, "newStart": 210, "newLines": 7, "lines": [" selection, tot = [], 0", " while tot < BUDGET * 2.5:", "     # pick the bucket that is furthest *below* its quota and still has documents", "-    cand = [b for b in (0, 1, 2) if ptr[b] < len(queues[b])]", "+    cand = [b for b in BUCKETS if ptr[b] < len(queues[b])]", "     if not cand:", "         break", "     b = min(cand, key=lambda b: used[b] - QUOTA[b] * max(1, tot))"]}], "userModified": false, "replaceAll": false}

The file /workspace/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/curate.py", "oldString": "print(\"bucket token shares:\", {b: round(used[b] / max(1, tot), 3) for b in (0, 1, 2)},\n      \"(0=other 1=tech 2=news)\")", "newString": "print(\"bucket token shares:\", {b: round(used[b] / max(1, tot), 3) for b in BUCKETS},\n      \"(0=rest 1=tech)\")", "originalFile": "\"\"\"Curate the raw web pool for a small LM whose target is a BROAD, multi-domain\nhigh-quality English distribution (Wikipedia + high-quality web prose + news +\ntechnical Q&A).\n\nStated, reproducible criterion: a QUALITY/DOMAIN classifier.\nWe train a logistic-regression classifier on L2-normalized hashed unigram+bigram\nword features to separate\n  positive = the disclosed HQ target (multi_dev.npy decoded into documents), from\n  negative = random raw pool documents.\nEvery pool document is scored by the classifier's decision value = \"how\ntarget-like\". Documents are ranked highest-first, subject to light quality gates\n(length floor, alpha-fraction, English-only) and near-duplicate removal.\n\nREGISTER BALANCING (key step): the target is EQUAL PARTS four registers, one of\nwhich is technical Q&A / code. A plain top-by-score selection floats news/prose\nto the top and starves the technical register (<0.2% of the budget), leaving a\nwhole quarter of the target badly modeled. So we detect technical documents with\na code-signal heuristic and interleave the two ranked lists to guarantee the\ntechnical register ~TECH_SHARE of the 12M-token budget, emitting the merged\npriority order until well past budget.\n\nLength-robustness: features are L2-normalized, so score reflects the *direction*\nof a document's n-gram profile, not its length or raw repetition — this avoids\nthe failure mode of raw log-likelihood-ratio ranking, which floats generic\nnavigation/list spam to the top.\n\nDev perplexity (frozen train_nano.py, multi_dev target): random selection 474.9\n-> classifier 412.3 -> +English gate 410.8 -> +25% tech 407.5 -> +35% tech 406.8.\n\"\"\"\nimport json, re, zlib, numpy as np\nfrom transformers import AutoTokenizer\n\nPOOL = \"/workspace/data/pool.jsonl\"\nDEV  = \"/workspace/data/multi_dev.npy\"\nOUT  = \"/workspace/submission/selection.json\"\n\nB = 1 << 20                 # hash buckets\nNEG_SAMPLE = 24000          # random pool docs used as negatives\nEPOCHS = 6\nLR = 0.5\nL2 = 1e-6\nSEED = 0\n\n_word = re.compile(r\"[a-z0-9]+\")\ndef words_of(s):\n    return _word.findall(s.lower())\n\n# small English function-word set — fluent prose is rich in these; navigation\n# menus / name lists / license-plate spam are not. A cheap, robust fluency gate.\nSTOP = set(\"the of and to a in is that it for was on as with by at be this from or \"\n           \"an are not but he she they we you his her their which have has had were \"\n           \"been i who what when where how all would there been more one about\".split())\n\ndef english_frac(s):\n    \"\"\"Fraction of ASCII latin letters among all alphabetic characters (0..1).\"\"\"\n    a = ascii_ = 0\n    for c in s:\n        if c.isalpha():\n            a += 1\n            if 'a' <= c <= 'z' or 'A' <= c <= 'Z':\n                ascii_ += 1\n    return ascii_ / max(1, a)\n\ndef crc(b):\n    return zlib.crc32(b) & (B - 1)\n\ndef featvec(words):\n    \"\"\"Return (unique_bucket_ids int32, L2-normalized float32 values) for\n    unigram+bigram hashed features of a word list.\"\"\"\n    if not words:\n        return np.empty(0, np.int32), np.empty(0, np.float32)\n    h = [crc(w.encode()) for w in words]\n    for i in range(len(words) - 1):\n        h.append(crc((words[i] + \"\\x00\" + words[i+1]).encode()))\n    h = np.asarray(h, dtype=np.int64)\n    idx, cnt = np.unique(h, return_counts=True)\n    v = cnt.astype(np.float32)\n    v /= np.sqrt((v * v).sum())\n    return idx.astype(np.int32), v\n\n# ---------- 1. build training features ----------\nprint(\"decoding target ...\")\ntok = AutoTokenizer.from_pretrained(\"gpt2\")\ndev = np.load(DEV)\neos = tok.eos_token_id\narr = dev.tolist()\n# split target into documents on EOS\npos = [i for i, t in enumerate(arr) if t == eos]\nsegs, prev = [], 0\nfor p in pos:\n    if p - prev > 5:\n        segs.append(arr[prev:p])\n    prev = p + 1\nif len(arr) - prev > 5:\n    segs.append(arr[prev:])\npos_feats = [featvec(words_of(tok.decode(s))) for s in segs]\nprint(\"target docs:\", len(pos_feats))\n\nprint(\"loading pool ...\")\nids, texts = [], []\nfor line in open(POOL):\n    r = json.loads(line)\n    ids.append(r[\"id\"]); texts.append(r[\"text\"])\nN = len(ids)\nprint(\"pool docs:\", N)\n\nrng = np.random.default_rng(SEED)\nneg_idx = rng.choice(N, size=min(NEG_SAMPLE, N), replace=False)\nprint(\"building negative features ...\")\nneg_feats = [featvec(words_of(texts[j])) for j in neg_idx]\n\n# ---------- 2. train logistic regression (sparse SGD) ----------\nprint(\"training classifier ...\")\nw = np.zeros(B, dtype=np.float32)\nb = 0.0\ntrain = [(f, 1.0) for f in pos_feats] + [(f, 0.0) for f in neg_feats]\norder = np.arange(len(train))\nfor ep in range(EPOCHS):\n    rng.shuffle(order)\n    lr = LR * (1.0 - ep / (EPOCHS + 1))\n    loss = 0.0\n    for oi in order:\n        (idx, v), y = train[oi]\n        if idx.size == 0:\n            continue\n        z = float(w[idx] @ v) + b\n        p = 1.0 / (1.0 + np.exp(-z))\n        g = p - y\n        w[idx] -= lr * (g * v + L2 * w[idx])\n        b -= lr * g\n        loss += -(y*np.log(p+1e-9) + (1-y)*np.log(1-p+1e-9))\n    print(f\"  epoch {ep} loss {loss/len(train):.4f}\")\n\n# The target is EQUAL PARTS four registers incl. technical Q&A / code. Detect the\n# technical register so we can guarantee it a token quota — otherwise the plain\n# classifier floats news/prose to the top and code is starved (<0.2% of budget),\n# leaving a whole quarter of the target badly modeled.\nTECH_SIG = ['<pre', '<code', 'stack overflow', 'stackoverflow', 'function(', 'var ',\n            'import ', 'public static', '#include', 'console.log', 'println', '</code',\n            '&lt;', 'def ', 'npm ', 'printf', 'const ', 'return ', ');']\ndef tech_score(tl):\n    return sum(1 for m in TECH_SIG if m in tl)\n\n# News cue heuristic — the pool is heavy on news portals, so a plain ranking\n# over-weights news (~48% of the head) vs. the target's ~25%, starving the\n# encyclopedic/prose registers. Detect news so we can cap its budget share too.\nNEWS_SIG = [' said ', ' told reporters', 'reuters', '(ap)', 'associated press',\n            'on monday', 'on tuesday', 'on wednesday', 'on thursday', 'on friday',\n            'on saturday', 'on sunday', 'the minister', 'prime minister', 'the police',\n            'according to', 'officials said', 'spokesman', 'spokesperson']\ndef news_score(tl):\n    return sum(1 for m in NEWS_SIG if m in tl)\n\n# ---------- 3. score every document, with register-aware gates ----------\nprint(\"scoring documents ...\")\nscores = np.full(N, -1e9, dtype=np.float64)\nntok = np.zeros(N, dtype=np.int32)\nbucket = np.zeros(N, dtype=np.int8)               # 0=other, 1=tech, 2=news\nfor j in range(N):\n    t = texts[j]\n    tl = t.lower()\n    wds = words_of(t)\n    n = len(wds)\n    ntok[j] = n\n    if n < 50:                                    # length floor\n        continue\n    head = t[:3000]\n    tech = tech_score(tl) >= 2\n    alpha = sum(c.isalpha() for c in head) / max(1, len(head))\n    if tech:\n        if alpha < 0.30:                          # code is symbol-heavy: relax gate\n            continue\n        bucket[j] = 1\n    else:\n        if alpha < 0.55 or english_frac(head) < 0.85:\n            continue                              # prose gates: clean English only\n        bucket[j] = 0     # non-tech prose (news + encyclopedic + general web)\n    idx, v = featvec(wds)\n    scores[j] = float(w[idx] @ v) + b\n    if j % 40000 == 0:\n        print(f\"  {j}/{N}\")\n\n# ---------- 4. rank each register, merge to a balanced priority order ----------\n# Target is ~equal parts {encyclopedic, prose, news, tech}. The plain classifier\n# starves the technical register, so we give it a fixed token quota; the rest of\n# the budget (news + encyclopedic + general prose) is filled by pure classifier\n# score. A NEWS cap was tried and REGRESSED (dev 406.8 -> 415.1): news-register\n# prose is genuinely valuable for this target, so we do not cap it.\nprint(\"ranking + balancing + dedup ...\")\nBUDGET = 12_000_000\nQUOTA = {1: 0.35, 0: 0.65}             # bucket -> token share (1=tech, 0=rest)\nBUCKETS = (0, 1)\norder = np.argsort(-scores)\nqueues = {b: [j for j in order if scores[j] > -1e8 and bucket[j] == b] for b in BUCKETS}\nptr = {b: 0 for b in BUCKETS}\nused = {b: 0 for b in BUCKETS}\n\ndef est(j):\n    return int(ntok[j] * 1.3) + 1\n\nseen = set()\ndef fresh(j):\n    wl = words_of(texts[j])\n    key = (hash(\" \".join(wl[:60])), len(wl) // 40)\n    if key in seen:\n        return False\n    seen.add(key)\n    return True\n\nselection, tot = [], 0\nwhile tot < BUDGET * 2.5:\n    # pick the bucket that is furthest *below* its quota and still has documents\n    cand = [b for b in BUCKETS if ptr[b] < len(queues[b])]\n    if not cand:\n        break\n    b = min(cand, key=lambda b: used[b] - QUOTA[b] * max(1, tot))\n    j = queues[b][ptr[b]]; ptr[b] += 1\n    if not fresh(j):\n        continue\n    e = est(j)\n    selection.append(int(ids[j])); tot += e; used[b] += e\n\njson.dump(selection, open(OUT, \"w\"))\nprint(f\"wrote {len(selection)} ids (~{tot} est tokens) -> {OUT}\")\nprint(\"bucket token shares:\", {b: round(used[b] / max(1, tot), 3) for b in (0, 1, 2)},\n      \"(0=other 1=tech 2=news)\")\n", "structuredPatch": [{"oldStart": 222, "oldLines": 5, "newStart": 222, "newLines": 5, "lines": [" ", " json.dump(selection, open(OUT, \"w\"))", " print(f\"wrote {len(selection)} ids (~{tot} est tokens) -> {OUT}\")", "-print(\"bucket token shares:\", {b: round(used[b] / max(1, tot), 3) for b in (0, 1, 2)},", "-      \"(0=other 1=tech 2=news)\")", "+print(\"bucket token shares:\", {b: round(used[b] / max(1, tot), 3) for b in BUCKETS},", "+      \"(0=rest 1=tech)\")"]}], "userModified": false, "replaceAll": false}

Command running in background with ID: b5hu3apjm. Output is being written to: /tmp/claude-0/-workspace/b3699ac4-1c9f-4c54-b65c-10b5011c1085/tasks/b5hu3apjm.output. You will be notified when it completes. To check interim output, use Read on that file path.
Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "b5hu3apjm", "backgroundCwdHint": "Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands."}

2

[stdout]
2
[metadata] {"noOutputExpected": false}

(Bash completed with no output)

[metadata] {"noOutputExpected": false}

ranking + balancing + dedup ...
wrote 27817 ids (~30002008 est tokens) -> /workspace/submission/selection.json
bucket token shares: {0: 0.653, 1: 0.347} (0=rest 1=tech)

[stdout]
ranking + balancing + dedup ...
wrote 27817 ids (~30002008 est tokens) -> /workspace/submission/selection.json
bucket token shares: {0: 0.653, 1: 0.347} (0=rest 1=tech)
[metadata] {"noOutputExpected": false}

identical to v5 (406.77): False
valid: unique + in-range, n= 27817
synced submission/curate.py
2015

[stdout]
identical to v5 (406.77): False
valid: unique + in-range, n= 27817
synced submission/curate.py
2015
[metadata] {"noOutputExpected": false}

Command running in background with ID: bmxftgkun. Output is being written to: /tmp/claude-0/-workspace/b3699ac4-1c9f-4c54-b65c-10b5011c1085/tasks/bmxftgkun.output. You will be notified when it completes. To check interim output, use Read on that file path.
Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "bmxftgkun", "backgroundCwdHint": "Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands."}

packed 12000000 tokens from 27817 ids -> /tmp/dev_train.npy
HELDOUT_PPL 404.594  loss 6.0029  train_s 56  /tmp/dev_train.npy
1895
1875

[stdout]
packed 12000000 tokens from 27817 ids -> /tmp/dev_train.npy
HELDOUT_PPL 404.594  loss 6.0029  train_s 56  /tmp/dev_train.npy
1895
1875
[metadata] {"noOutputExpected": false}

28:-> classifier 412.3 -> +English gate 410.8 -> +25% tech 407.5 -> +35% tech 404.6.

[stdout]
28:-> classifier 412.3 -> +English gate 410.8 -> +25% tech 407.5 -> +35% tech 404.6.
[metadata] {"noOutputExpected": false}

The file /workspace/curate.py has been updated. All occurrences were successfully replaced. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/curate.py", "oldString": "# small English function-word set — fluent prose is rich in these; navigation\n# menus / name lists / license-plate spam are not. A cheap, robust fluency gate.\nSTOP = set(\"the of and to a in is that it for was on as with by at be this from or \"\n           \"an are not but he she they we you his her their which have has had were \"\n           \"been i who what when where how all would there been more one about\".split())\n\ndef english_frac(s):", "newString": "def english_frac(s):", "originalFile": "\"\"\"Curate the raw web pool for a small LM whose target is a BROAD, multi-domain\nhigh-quality English distribution (Wikipedia + high-quality web prose + news +\ntechnical Q&A).\n\nStated, reproducible criterion: a QUALITY/DOMAIN classifier.\nWe train a logistic-regression classifier on L2-normalized hashed unigram+bigram\nword features to separate\n  positive = the disclosed HQ target (multi_dev.npy decoded into documents), from\n  negative = random raw pool documents.\nEvery pool document is scored by the classifier's decision value = \"how\ntarget-like\". Documents are ranked highest-first, subject to light quality gates\n(length floor, alpha-fraction, English-only) and near-duplicate removal.\n\nREGISTER BALANCING (key step): the target is EQUAL PARTS four registers, one of\nwhich is technical Q&A / code. A plain top-by-score selection floats news/prose\nto the top and starves the technical register (<0.2% of the budget), leaving a\nwhole quarter of the target badly modeled. So we detect technical documents with\na code-signal heuristic and interleave the two ranked lists to guarantee the\ntechnical register ~TECH_SHARE of the 12M-token budget, emitting the merged\npriority order until well past budget.\n\nLength-robustness: features are L2-normalized, so score reflects the *direction*\nof a document's n-gram profile, not its length or raw repetition — this avoids\nthe failure mode of raw log-likelihood-ratio ranking, which floats generic\nnavigation/list spam to the top.\n\nDev perplexity (frozen train_nano.py, multi_dev target): random selection 474.9\n-> classifier 412.3 -> +English gate 410.8 -> +25% tech 407.5 -> +35% tech 404.6.\n\"\"\"\nimport json, re, zlib, numpy as np\nfrom transformers import AutoTokenizer\n\nPOOL = \"/workspace/data/pool.jsonl\"\nDEV  = \"/workspace/data/multi_dev.npy\"\nOUT  = \"/workspace/submission/selection.json\"\n\nB = 1 << 20                 # hash buckets\nNEG_SAMPLE = 24000          # random pool docs used as negatives\nEPOCHS = 6\nLR = 0.5\nL2 = 1e-6\nSEED = 0\n\n_word = re.compile(r\"[a-z0-9]+\")\ndef words_of(s):\n    return _word.findall(s.lower())\n\n# small English function-word set — fluent prose is rich in these; navigation\n# menus / name lists / license-plate spam are not. A cheap, robust fluency gate.\nSTOP = set(\"the of and to a in is that it for was on as with by at be this from or \"\n           \"an are not but he she they we you his her their which have has had were \"\n           \"been i who what when where how all would there been more one about\".split())\n\ndef english_frac(s):\n    \"\"\"Fraction of ASCII latin letters among all alphabetic characters (0..1).\"\"\"\n    a = ascii_ = 0\n    for c in s:\n        if c.isalpha():\n            a += 1\n            if 'a' <= c <= 'z' or 'A' <= c <= 'Z':\n                ascii_ += 1\n    return ascii_ / max(1, a)\n\ndef crc(b):\n    return zlib.crc32(b) & (B - 1)\n\ndef featvec(words):\n    \"\"\"Return (unique_bucket_ids int32, L2-normalized float32 values) for\n    unigram+bigram hashed features of a word list.\"\"\"\n    if not words:\n        return np.empty(0, np.int32), np.empty(0, np.float32)\n    h = [crc(w.encode()) for w in words]\n    for i in range(len(words) - 1):\n        h.append(crc((words[i] + \"\\x00\" + words[i+1]).encode()))\n    h = np.asarray(h, dtype=np.int64)\n    idx, cnt = np.unique(h, return_counts=True)\n    v = cnt.astype(np.float32)\n    v /= np.sqrt((v * v).sum())\n    return idx.astype(np.int32), v\n\n# ---------- 1. build training features ----------\nprint(\"decoding target ...\")\ntok = AutoTokenizer.from_pretrained(\"gpt2\")\ndev = np.load(DEV)\neos = tok.eos_token_id\narr = dev.tolist()\n# split target into documents on EOS\npos = [i for i, t in enumerate(arr) if t == eos]\nsegs, prev = [], 0\nfor p in pos:\n    if p - prev > 5:\n        segs.append(arr[prev:p])\n    prev = p + 1\nif len(arr) - prev > 5:\n    segs.append(arr[prev:])\npos_feats = [featvec(words_of(tok.decode(s))) for s in segs]\nprint(\"target docs:\", len(pos_feats))\n\nprint(\"loading pool ...\")\nids, texts = [], []\nfor line in open(POOL):\n    r = json.loads(line)\n    ids.append(r[\"id\"]); texts.append(r[\"text\"])\nN = len(ids)\nprint(\"pool docs:\", N)\n\nrng = np.random.default_rng(SEED)\nneg_idx = rng.choice(N, size=min(NEG_SAMPLE, N), replace=False)\nprint(\"building negative features ...\")\nneg_feats = [featvec(words_of(texts[j])) for j in neg_idx]\n\n# ---------- 2. train logistic regression (sparse SGD) ----------\nprint(\"training classifier ...\")\nw = np.zeros(B, dtype=np.float32)\nb = 0.0\ntrain = [(f, 1.0) for f in pos_feats] + [(f, 0.0) for f in neg_feats]\norder = np.arange(len(train))\nfor ep in range(EPOCHS):\n    rng.shuffle(order)\n    lr = LR * (1.0 - ep / (EPOCHS + 1))\n    loss = 0.0\n    for oi in order:\n        (idx, v), y = train[oi]\n        if idx.size == 0:\n            continue\n        z = float(w[idx] @ v) + b\n        p = 1.0 / (1.0 + np.exp(-z))\n        g = p - y\n        w[idx] -= lr * (g * v + L2 * w[idx])\n        b -= lr * g\n        loss += -(y*np.log(p+1e-9) + (1-y)*np.log(1-p+1e-9))\n    print(f\"  epoch {ep} loss {loss/len(train):.4f}\")\n\n# The target is EQUAL PARTS four registers incl. technical Q&A / code. Detect the\n# technical register so we can guarantee it a token quota — otherwise the plain\n# classifier floats news/prose to the top and code is starved (<0.2% of budget),\n# leaving a whole quarter of the target badly modeled.\nTECH_SIG = ['<pre', '<code', 'stack overflow', 'stackoverflow', 'function(', 'var ',\n            'import ', 'public static', '#include', 'console.log', 'println', '</code',\n            '&lt;', 'def ', 'npm ', 'printf', 'const ', 'return ', ');']\ndef tech_score(tl):\n    return sum(1 for m in TECH_SIG if m in tl)\n\n# News cue heuristic — the pool is heavy on news portals, so a plain ranking\n# over-weights news (~48% of the head) vs. the target's ~25%, starving the\n# encyclopedic/prose registers. Detect news so we can cap its budget share too.\nNEWS_SIG = [' said ', ' told reporters', 'reuters', '(ap)', 'associated press',\n            'on monday', 'on tuesday', 'on wednesday', 'on thursday', 'on friday',\n            'on saturday', 'on sunday', 'the minister', 'prime minister', 'the police',\n            'according to', 'officials said', 'spokesman', 'spokesperson']\ndef news_score(tl):\n    return sum(1 for m in NEWS_SIG if m in tl)\n\n# ---------- 3. score every document, with register-aware gates ----------\nprint(\"scoring documents ...\")\nscores = np.full(N, -1e9, dtype=np.float64)\nntok = np.zeros(N, dtype=np.int32)\nbucket = np.zeros(N, dtype=np.int8)               # 0=other, 1=tech, 2=news\nfor j in range(N):\n    t = texts[j]\n    tl = t.lower()\n    wds = words_of(t)\n    n = len(wds)\n    ntok[j] = n\n    if n < 50:                                    # length floor\n        continue\n    head = t[:3000]\n    tech = tech_score(tl) >= 2\n    alpha = sum(c.isalpha() for c in head) / max(1, len(head))\n    if tech:\n        if alpha < 0.30:                          # code is symbol-heavy: relax gate\n            continue\n        bucket[j] = 1\n    else:\n        if alpha < 0.55 or english_frac(head) < 0.85:\n            continue                              # prose gates: clean English only\n        bucket[j] = 0     # non-tech prose (news + encyclopedic + general web)\n    idx, v = featvec(wds)\n    scores[j] = float(w[idx] @ v) + b\n    if j % 40000 == 0:\n        print(f\"  {j}/{N}\")\n\n# ---------- 4. rank each register, merge to a balanced priority order ----------\n# Target is ~equal parts {encyclopedic, prose, news, tech}. The plain classifier\n# starves the technical register, so we give it a fixed token quota; the rest of\n# the budget (news + encyclopedic + general prose) is filled by pure classifier\n# score. A NEWS cap was tried and REGRESSED (dev 406.8 -> 415.1): news-register\n# prose is genuinely valuable for this target, so we do not cap it.\nprint(\"ranking + balancing + dedup ...\")\nBUDGET = 12_000_000\nQUOTA = {1: 0.35, 0: 0.65}             # bucket -> token share (1=tech, 0=rest)\nBUCKETS = (0, 1)\norder = np.argsort(-scores)\nqueues = {b: [j for j in order if scores[j] > -1e8 and bucket[j] == b] for b in BUCKETS}\nptr = {b: 0 for b in BUCKETS}\nused = {b: 0 for b in BUCKETS}\n\ndef est(j):\n    return int(ntok[j] * 1.3) + 1\n\nseen = set()\ndef fresh(j):\n    wl = words_of(texts[j])\n    key = (hash(\" \".join(wl[:60])), len(wl) // 40)\n    if key in seen:\n        return False\n    seen.add(key)\n    return True\n\nselection, tot = [], 0\nwhile tot < BUDGET * 2.5:\n    # pick the bucket that is furthest *below* its quota and still has documents\n    cand = [b for b in BUCKETS if ptr[b] < len(queues[b])]\n    if not cand:\n        break\n    b = min(cand, key=lambda b: used[b] - QUOTA[b] * max(1, tot))\n    j = queues[b][ptr[b]]; ptr[b] += 1\n    if not fresh(j):\n        continue\n    e = est(j)\n    selection.append(int(ids[j])); tot += e; used[b] += e\n\njson.dump(selection, open(OUT, \"w\"))\nprint(f\"wrote {len(selection)} ids (~{tot} est tokens) -> {OUT}\")\nprint(\"bucket token shares:\", {b: round(used[b] / max(1, tot), 3) for b in BUCKETS},\n      \"(0=rest 1=tech)\")\n", "structuredPatch": [{"oldStart": 45, "oldLines": 12, "newStart": 45, "newLines": 6, "lines": [" def words_of(s):", "     return _word.findall(s.lower())", " ", "-# small English function-word set — fluent prose is rich in these; navigation", "-# menus / name lists / license-plate spam are not. A cheap, robust fluency gate.", "-STOP = set(\"the of and to a in is that it for was on as with by at be this from or \"", "-           \"an are not but he she they we you his her their which have has had were \"", "-           \"been i who what when where how all would there been more one about\".split())", "-", " def english_frac(s):", "     \"\"\"Fraction of ASCII latin letters among all alphabetic characters (0..1).\"\"\"", "     a = ascii_ = 0"]}], "userModified": false, "replaceAll": true}

The file /workspace/curate.py has been updated. All occurrences were successfully replaced. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/curate.py", "oldString": "def tech_score(tl):\n    return sum(1 for m in TECH_SIG if m in tl)\n\n# News cue heuristic — the pool is heavy on news portals, so a plain ranking\n# over-weights news (~48% of the head) vs. the target's ~25%, starving the\n# encyclopedic/prose registers. Detect news so we can cap its budget share too.\nNEWS_SIG = [' said ', ' told reporters', 'reuters', '(ap)', 'associated press',\n            'on monday', 'on tuesday', 'on wednesday', 'on thursday', 'on friday',\n            'on saturday', 'on sunday', 'the minister', 'prime minister', 'the police',\n            'according to', 'officials said', 'spokesman', 'spokesperson']\ndef news_score(tl):\n    return sum(1 for m in NEWS_SIG if m in tl)\n\n# ---------- 3. score every document, with register-aware gates ----------", "newString": "def tech_score(tl):\n    return sum(1 for m in TECH_SIG if m in tl)\n\n# ---------- 3. score every document, with register-aware gates ----------", "originalFile": "\"\"\"Curate the raw web pool for a small LM whose target is a BROAD, multi-domain\nhigh-quality English distribution (Wikipedia + high-quality web prose + news +\ntechnical Q&A).\n\nStated, reproducible criterion: a QUALITY/DOMAIN classifier.\nWe train a logistic-regression classifier on L2-normalized hashed unigram+bigram\nword features to separate\n  positive = the disclosed HQ target (multi_dev.npy decoded into documents), from\n  negative = random raw pool documents.\nEvery pool document is scored by the classifier's decision value = \"how\ntarget-like\". Documents are ranked highest-first, subject to light quality gates\n(length floor, alpha-fraction, English-only) and near-duplicate removal.\n\nREGISTER BALANCING (key step): the target is EQUAL PARTS four registers, one of\nwhich is technical Q&A / code. A plain top-by-score selection floats news/prose\nto the top and starves the technical register (<0.2% of the budget), leaving a\nwhole quarter of the target badly modeled. So we detect technical documents with\na code-signal heuristic and interleave the two ranked lists to guarantee the\ntechnical register ~TECH_SHARE of the 12M-token budget, emitting the merged\npriority order until well past budget.\n\nLength-robustness: features are L2-normalized, so score reflects the *direction*\nof a document's n-gram profile, not its length or raw repetition — this avoids\nthe failure mode of raw log-likelihood-ratio ranking, which floats generic\nnavigation/list spam to the top.\n\nDev perplexity (frozen train_nano.py, multi_dev target): random selection 474.9\n-> classifier 412.3 -> +English gate 410.8 -> +25% tech 407.5 -> +35% tech 404.6.\n\"\"\"\nimport json, re, zlib, numpy as np\nfrom transformers import AutoTokenizer\n\nPOOL = \"/workspace/data/pool.jsonl\"\nDEV  = \"/workspace/data/multi_dev.npy\"\nOUT  = \"/workspace/submission/selection.json\"\n\nB = 1 << 20                 # hash buckets\nNEG_SAMPLE = 24000          # random pool docs used as negatives\nEPOCHS = 6\nLR = 0.5\nL2 = 1e-6\nSEED = 0\n\n_word = re.compile(r\"[a-z0-9]+\")\ndef words_of(s):\n    return _word.findall(s.lower())\n\ndef english_frac(s):\n    \"\"\"Fraction of ASCII latin letters among all alphabetic characters (0..1).\"\"\"\n    a = ascii_ = 0\n    for c in s:\n        if c.isalpha():\n            a += 1\n            if 'a' <= c <= 'z' or 'A' <= c <= 'Z':\n                ascii_ += 1\n    return ascii_ / max(1, a)\n\ndef crc(b):\n    return zlib.crc32(b) & (B - 1)\n\ndef featvec(words):\n    \"\"\"Return (unique_bucket_ids int32, L2-normalized float32 values) for\n    unigram+bigram hashed features of a word list.\"\"\"\n    if not words:\n        return np.empty(0, np.int32), np.empty(0, np.float32)\n    h = [crc(w.encode()) for w in words]\n    for i in range(len(words) - 1):\n        h.append(crc((words[i] + \"\\x00\" + words[i+1]).encode()))\n    h = np.asarray(h, dtype=np.int64)\n    idx, cnt = np.unique(h, return_counts=True)\n    v = cnt.astype(np.float32)\n    v /= np.sqrt((v * v).sum())\n    return idx.astype(np.int32), v\n\n# ---------- 1. build training features ----------\nprint(\"decoding target ...\")\ntok = AutoTokenizer.from_pretrained(\"gpt2\")\ndev = np.load(DEV)\neos = tok.eos_token_id\narr = dev.tolist()\n# split target into documents on EOS\npos = [i for i, t in enumerate(arr) if t == eos]\nsegs, prev = [], 0\nfor p in pos:\n    if p - prev > 5:\n        segs.append(arr[prev:p])\n    prev = p + 1\nif len(arr) - prev > 5:\n    segs.append(arr[prev:])\npos_feats = [featvec(words_of(tok.decode(s))) for s in segs]\nprint(\"target docs:\", len(pos_feats))\n\nprint(\"loading pool ...\")\nids, texts = [], []\nfor line in open(POOL):\n    r = json.loads(line)\n    ids.append(r[\"id\"]); texts.append(r[\"text\"])\nN = len(ids)\nprint(\"pool docs:\", N)\n\nrng = np.random.default_rng(SEED)\nneg_idx = rng.choice(N, size=min(NEG_SAMPLE, N), replace=False)\nprint(\"building negative features ...\")\nneg_feats = [featvec(words_of(texts[j])) for j in neg_idx]\n\n# ---------- 2. train logistic regression (sparse SGD) ----------\nprint(\"training classifier ...\")\nw = np.zeros(B, dtype=np.float32)\nb = 0.0\ntrain = [(f, 1.0) for f in pos_feats] + [(f, 0.0) for f in neg_feats]\norder = np.arange(len(train))\nfor ep in range(EPOCHS):\n    rng.shuffle(order)\n    lr = LR * (1.0 - ep / (EPOCHS + 1))\n    loss = 0.0\n    for oi in order:\n        (idx, v), y = train[oi]\n        if idx.size == 0:\n            continue\n        z = float(w[idx] @ v) + b\n        p = 1.0 / (1.0 + np.exp(-z))\n        g = p - y\n        w[idx] -= lr * (g * v + L2 * w[idx])\n        b -= lr * g\n        loss += -(y*np.log(p+1e-9) + (1-y)*np.log(1-p+1e-9))\n    print(f\"  epoch {ep} loss {loss/len(train):.4f}\")\n\n# The target is EQUAL PARTS four registers incl. technical Q&A / code. Detect the\n# technical register so we can guarantee it a token quota — otherwise the plain\n# classifier floats news/prose to the top and code is starved (<0.2% of budget),\n# leaving a whole quarter of the target badly modeled.\nTECH_SIG = ['<pre', '<code', 'stack overflow', 'stackoverflow', 'function(', 'var ',\n            'import ', 'public static', '#include', 'console.log', 'println', '</code',\n            '&lt;', 'def ', 'npm ', 'printf', 'const ', 'return ', ');']\ndef tech_score(tl):\n    return sum(1 for m in TECH_SIG if m in tl)\n\n# News cue heuristic — the pool is heavy on news portals, so a plain ranking\n# over-weights news (~48% of the head) vs. the target's ~25%, starving the\n# encyclopedic/prose registers. Detect news so we can cap its budget share too.\nNEWS_SIG = [' said ', ' told reporters', 'reuters', '(ap)', 'associated press',\n            'on monday', 'on tuesday', 'on wednesday', 'on thursday', 'on friday',\n            'on saturday', 'on sunday', 'the minister', 'prime minister', 'the police',\n            'according to', 'officials said', 'spokesman', 'spokesperson']\ndef news_score(tl):\n    return sum(1 for m in NEWS_SIG if m in tl)\n\n# ---------- 3. score every document, with register-aware gates ----------\nprint(\"scoring documents ...\")\nscores = np.full(N, -1e9, dtype=np.float64)\nntok = np.zeros(N, dtype=np.int32)\nbucket = np.zeros(N, dtype=np.int8)               # 0=other, 1=tech, 2=news\nfor j in range(N):\n    t = texts[j]\n    tl = t.lower()\n    wds = words_of(t)\n    n = len(wds)\n    ntok[j] = n\n    if n < 50:                                    # length floor\n        continue\n    head = t[:3000]\n    tech = tech_score(tl) >= 2\n    alpha = sum(c.isalpha() for c in head) / max(1, len(head))\n    if tech:\n        if alpha < 0.30:                          # code is symbol-heavy: relax gate\n            continue\n        bucket[j] = 1\n    else:\n        if alpha < 0.55 or english_frac(head) < 0.85:\n            continue                              # prose gates: clean English only\n        bucket[j] = 0     # non-tech prose (news + encyclopedic + general web)\n    idx, v = featvec(wds)\n    scores[j] = float(w[idx] @ v) + b\n    if j % 40000 == 0:\n        print(f\"  {j}/{N}\")\n\n# ---------- 4. rank each register, merge to a balanced priority order ----------\n# Target is ~equal parts {encyclopedic, prose, news, tech}. The plain classifier\n# starves the technical register, so we give it a fixed token quota; the rest of\n# the budget (news + encyclopedic + general prose) is filled by pure classifier\n# score. A NEWS cap was tried and REGRESSED (dev 406.8 -> 415.1): news-register\n# prose is genuinely valuable for this target, so we do not cap it.\nprint(\"ranking + balancing + dedup ...\")\nBUDGET = 12_000_000\nQUOTA = {1: 0.35, 0: 0.65}             # bucket -> token share (1=tech, 0=rest)\nBUCKETS = (0, 1)\norder = np.argsort(-scores)\nqueues = {b: [j for j in order if scores[j] > -1e8 and bucket[j] == b] for b in BUCKETS}\nptr = {b: 0 for b in BUCKETS}\nused = {b: 0 for b in BUCKETS}\n\ndef est(j):\n    return int(ntok[j] * 1.3) + 1\n\nseen = set()\ndef fresh(j):\n    wl = words_of(texts[j])\n    key = (hash(\" \".join(wl[:60])), len(wl) // 40)\n    if key in seen:\n        return False\n    seen.add(key)\n    return True\n\nselection, tot = [], 0\nwhile tot < BUDGET * 2.5:\n    # pick the bucket that is furthest *below* its quota and still has documents\n    cand = [b for b in BUCKETS if ptr[b] < len(queues[b])]\n    if not cand:\n        break\n    b = min(cand, key=lambda b: used[b] - QUOTA[b] * max(1, tot))\n    j = queues[b][ptr[b]]; ptr[b] += 1\n    if not fresh(j):\n        continue\n    e = est(j)\n    selection.append(int(ids[j])); tot += e; used[b] += e\n\njson.dump(selection, open(OUT, \"w\"))\nprint(f\"wrote {len(selection)} ids (~{tot} est tokens) -> {OUT}\")\nprint(\"bucket token shares:\", {b: round(used[b] / max(1, tot), 3) for b in BUCKETS},\n      \"(0=rest 1=tech)\")\n", "structuredPatch": [{"oldStart": 135, "oldLines": 16, "newStart": 135, "newLines": 6, "lines": [" def tech_score(tl):", "     return sum(1 for m in TECH_SIG if m in tl)", " ", "-# News cue heuristic — the pool is heavy on news portals, so a plain ranking", "-# over-weights news (~48% of the head) vs. the target's ~25%, starving the", "-# encyclopedic/prose registers. Detect news so we can cap its budget share too.", "-NEWS_SIG = [' said ', ' told reporters', 'reuters', '(ap)', 'associated press',", "-            'on monday', 'on tuesday', 'on wednesday', 'on thursday', 'on friday',", "-            'on saturday', 'on sunday', 'the minister', 'prime minister', 'the police',", "-            'according to', 'officials said', 'spokesman', 'spokesperson']", "-def news_score(tl):", "-    return sum(1 for m in NEWS_SIG if m in tl)", "-", " # ---------- 3. score every document, with register-aware gates ----------", " print(\"scoring documents ...\")", " scores = np.full(N, -1e9, dtype=np.float64)"]}], "userModified": false, "replaceAll": true}

--- syntax check + reproduce head ---
curate.py parses OK

[stdout]
--- syntax check + reproduce head ---
curate.py parses OK
[metadata] {"noOutputExpected": false}

The file /workspace/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/curate.py", "oldString": "    wl = words_of(texts[j])\n    key = (hash(\" \".join(wl[:60])), len(wl) // 40)", "newString": "    wl = words_of(texts[j])\n    key = (zlib.crc32(\" \".join(wl[:60]).encode()), len(wl) // 40)  # stable hash (reproducible)", "originalFile": "\"\"\"Curate the raw web pool for a small LM whose target is a BROAD, multi-domain\nhigh-quality English distribution (Wikipedia + high-quality web prose + news +\ntechnical Q&A).\n\nStated, reproducible criterion: a QUALITY/DOMAIN classifier.\nWe train a logistic-regression classifier on L2-normalized hashed unigram+bigram\nword features to separate\n  positive = the disclosed HQ target (multi_dev.npy decoded into documents), from\n  negative = random raw pool documents.\nEvery pool document is scored by the classifier's decision value = \"how\ntarget-like\". Documents are ranked highest-first, subject to light quality gates\n(length floor, alpha-fraction, English-only) and near-duplicate removal.\n\nREGISTER BALANCING (key step): the target is EQUAL PARTS four registers, one of\nwhich is technical Q&A / code. A plain top-by-score selection floats news/prose\nto the top and starves the technical register (<0.2% of the budget), leaving a\nwhole quarter of the target badly modeled. So we detect technical documents with\na code-signal heuristic and interleave the two ranked lists to guarantee the\ntechnical register ~TECH_SHARE of the 12M-token budget, emitting the merged\npriority order until well past budget.\n\nLength-robustness: features are L2-normalized, so score reflects the *direction*\nof a document's n-gram profile, not its length or raw repetition — this avoids\nthe failure mode of raw log-likelihood-ratio ranking, which floats generic\nnavigation/list spam to the top.\n\nDev perplexity (frozen train_nano.py, multi_dev target): random selection 474.9\n-> classifier 412.3 -> +English gate 410.8 -> +25% tech 407.5 -> +35% tech 404.6.\n\"\"\"\nimport json, re, zlib, numpy as np\nfrom transformers import AutoTokenizer\n\nPOOL = \"/workspace/data/pool.jsonl\"\nDEV  = \"/workspace/data/multi_dev.npy\"\nOUT  = \"/workspace/submission/selection.json\"\n\nB = 1 << 20                 # hash buckets\nNEG_SAMPLE = 24000          # random pool docs used as negatives\nEPOCHS = 6\nLR = 0.5\nL2 = 1e-6\nSEED = 0\n\n_word = re.compile(r\"[a-z0-9]+\")\ndef words_of(s):\n    return _word.findall(s.lower())\n\ndef english_frac(s):\n    \"\"\"Fraction of ASCII latin letters among all alphabetic characters (0..1).\"\"\"\n    a = ascii_ = 0\n    for c in s:\n        if c.isalpha():\n            a += 1\n            if 'a' <= c <= 'z' or 'A' <= c <= 'Z':\n                ascii_ += 1\n    return ascii_ / max(1, a)\n\ndef crc(b):\n    return zlib.crc32(b) & (B - 1)\n\ndef featvec(words):\n    \"\"\"Return (unique_bucket_ids int32, L2-normalized float32 values) for\n    unigram+bigram hashed features of a word list.\"\"\"\n    if not words:\n        return np.empty(0, np.int32), np.empty(0, np.float32)\n    h = [crc(w.encode()) for w in words]\n    for i in range(len(words) - 1):\n        h.append(crc((words[i] + \"\\x00\" + words[i+1]).encode()))\n    h = np.asarray(h, dtype=np.int64)\n    idx, cnt = np.unique(h, return_counts=True)\n    v = cnt.astype(np.float32)\n    v /= np.sqrt((v * v).sum())\n    return idx.astype(np.int32), v\n\n# ---------- 1. build training features ----------\nprint(\"decoding target ...\")\ntok = AutoTokenizer.from_pretrained(\"gpt2\")\ndev = np.load(DEV)\neos = tok.eos_token_id\narr = dev.tolist()\n# split target into documents on EOS\npos = [i for i, t in enumerate(arr) if t == eos]\nsegs, prev = [], 0\nfor p in pos:\n    if p - prev > 5:\n        segs.append(arr[prev:p])\n    prev = p + 1\nif len(arr) - prev > 5:\n    segs.append(arr[prev:])\npos_feats = [featvec(words_of(tok.decode(s))) for s in segs]\nprint(\"target docs:\", len(pos_feats))\n\nprint(\"loading pool ...\")\nids, texts = [], []\nfor line in open(POOL):\n    r = json.loads(line)\n    ids.append(r[\"id\"]); texts.append(r[\"text\"])\nN = len(ids)\nprint(\"pool docs:\", N)\n\nrng = np.random.default_rng(SEED)\nneg_idx = rng.choice(N, size=min(NEG_SAMPLE, N), replace=False)\nprint(\"building negative features ...\")\nneg_feats = [featvec(words_of(texts[j])) for j in neg_idx]\n\n# ---------- 2. train logistic regression (sparse SGD) ----------\nprint(\"training classifier ...\")\nw = np.zeros(B, dtype=np.float32)\nb = 0.0\ntrain = [(f, 1.0) for f in pos_feats] + [(f, 0.0) for f in neg_feats]\norder = np.arange(len(train))\nfor ep in range(EPOCHS):\n    rng.shuffle(order)\n    lr = LR * (1.0 - ep / (EPOCHS + 1))\n    loss = 0.0\n    for oi in order:\n        (idx, v), y = train[oi]\n        if idx.size == 0:\n            continue\n        z = float(w[idx] @ v) + b\n        p = 1.0 / (1.0 + np.exp(-z))\n        g = p - y\n        w[idx] -= lr * (g * v + L2 * w[idx])\n        b -= lr * g\n        loss += -(y*np.log(p+1e-9) + (1-y)*np.log(1-p+1e-9))\n    print(f\"  epoch {ep} loss {loss/len(train):.4f}\")\n\n# The target is EQUAL PARTS four registers incl. technical Q&A / code. Detect the\n# technical register so we can guarantee it a token quota — otherwise the plain\n# classifier floats news/prose to the top and code is starved (<0.2% of budget),\n# leaving a whole quarter of the target badly modeled.\nTECH_SIG = ['<pre', '<code', 'stack overflow', 'stackoverflow', 'function(', 'var ',\n            'import ', 'public static', '#include', 'console.log', 'println', '</code',\n            '&lt;', 'def ', 'npm ', 'printf', 'const ', 'return ', ');']\ndef tech_score(tl):\n    return sum(1 for m in TECH_SIG if m in tl)\n\n# ---------- 3. score every document, with register-aware gates ----------\nprint(\"scoring documents ...\")\nscores = np.full(N, -1e9, dtype=np.float64)\nntok = np.zeros(N, dtype=np.int32)\nbucket = np.zeros(N, dtype=np.int8)               # 0=rest (news+encyclopedic+prose), 1=tech\nfor j in range(N):\n    t = texts[j]\n    tl = t.lower()\n    wds = words_of(t)\n    n = len(wds)\n    ntok[j] = n\n    if n < 50:                                    # length floor\n        continue\n    head = t[:3000]\n    tech = tech_score(tl) >= 2\n    alpha = sum(c.isalpha() for c in head) / max(1, len(head))\n    if tech:\n        if alpha < 0.30:                          # code is symbol-heavy: relax gate\n            continue\n        bucket[j] = 1\n    else:\n        if alpha < 0.55 or english_frac(head) < 0.85:\n            continue                              # prose gates: clean English only\n        bucket[j] = 0     # non-tech prose (news + encyclopedic + general web)\n    idx, v = featvec(wds)\n    scores[j] = float(w[idx] @ v) + b\n    if j % 40000 == 0:\n        print(f\"  {j}/{N}\")\n\n# ---------- 4. rank each register, merge to a balanced priority order ----------\n# Target is ~equal parts {encyclopedic, prose, news, tech}. The plain classifier\n# starves the technical register, so we give it a fixed token quota; the rest of\n# the budget (news + encyclopedic + general prose) is filled by pure classifier\n# score. A NEWS cap was tried and REGRESSED (dev 406.8 -> 415.1): news-register\n# prose is genuinely valuable for this target, so we do not cap it.\nprint(\"ranking + balancing + dedup ...\")\nBUDGET = 12_000_000\nQUOTA = {1: 0.35, 0: 0.65}             # bucket -> token share (1=tech, 0=rest)\nBUCKETS = (0, 1)\norder = np.argsort(-scores)\nqueues = {b: [j for j in order if scores[j] > -1e8 and bucket[j] == b] for b in BUCKETS}\nptr = {b: 0 for b in BUCKETS}\nused = {b: 0 for b in BUCKETS}\n\ndef est(j):\n    return int(ntok[j] * 1.3) + 1\n\nseen = set()\ndef fresh(j):\n    wl = words_of(texts[j])\n    key = (hash(\" \".join(wl[:60])), len(wl) // 40)\n    if key in seen:\n        return False\n    seen.add(key)\n    return True\n\nselection, tot = [], 0\nwhile tot < BUDGET * 2.5:\n    # pick the bucket that is furthest *below* its quota and still has documents\n    cand = [b for b in BUCKETS if ptr[b] < len(queues[b])]\n    if not cand:\n        break\n    b = min(cand, key=lambda b: used[b] - QUOTA[b] * max(1, tot))\n    j = queues[b][ptr[b]]; ptr[b] += 1\n    if not fresh(j):\n        continue\n    e = est(j)\n    selection.append(int(ids[j])); tot += e; used[b] += e\n\njson.dump(selection, open(OUT, \"w\"))\nprint(f\"wrote {len(selection)} ids (~{tot} est tokens) -> {OUT}\")\nprint(\"bucket token shares:\", {b: round(used[b] / max(1, tot), 3) for b in BUCKETS},\n      \"(0=rest 1=tech)\")\n", "structuredPatch": [{"oldStart": 185, "oldLines": 7, "newStart": 185, "newLines": 7, "lines": [" seen = set()", " def fresh(j):", "     wl = words_of(texts[j])", "-    key = (hash(\" \".join(wl[:60])), len(wl) // 40)", "+    key = (zlib.crc32(\" \".join(wl[:60]).encode()), len(wl) // 40)  # stable hash (reproducible)", "     if key in seen:", "         return False", "     seen.add(key)"]}], "userModified": false, "replaceAll": false}

Command running in background with ID: bk1s6tjoq. Output is being written to: /tmp/claude-0/-workspace/b3699ac4-1c9f-4c54-b65c-10b5011c1085/tasks/bk1s6tjoq.output. You will be notified when it completes. To check interim output, use Read on that file path.
Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "bk1s6tjoq", "backgroundCwdHint": "Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands."}