claude-opus-4-8

2

[stdout]
2
[metadata] {"noOutputExpected": false}

160000/182016
ranking + balancing + dedup ...
wrote 30522 ids (~30000217 est tokens, tech~7501253) -> /workspace/submission/selection.json
score p99/p50/min-selected: 4.30266970889042 -1.8804948876671972

[stdout]
  160000/182016
ranking + balancing + dedup ...
wrote 30522 ids (~30000217 est tokens, tech~7501253) -> /workspace/submission/selection.json
score p99/p50/min-selected: 4.30266970889042 -1.8804948876671972
[metadata] {"noOutputExpected": false}

Command running in background with ID: b39d7447u. Output is being written to: /tmp/claude-0/-workspace/b3699ac4-1c9f-4c54-b65c-10b5011c1085/tasks/b39d7447u.output. You will be notified when it completes. To check interim output, use Read on that file path.
Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "b39d7447u", "backgroundCwdHint": "Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands."}

packed 12000000 tokens from 30522 ids -> /tmp/dev_train.npy
HELDOUT_PPL 407.533  loss 6.0101  train_s 56  /tmp/dev_train.npy
3277
3257

[stdout]
packed 12000000 tokens from 30522 ids -> /tmp/dev_train.npy
HELDOUT_PPL 407.533  loss 6.0101  train_s 56  /tmp/dev_train.npy
3277
3257
[metadata] {"noOutputExpected": false}

Command running in background with ID: b8jbrmq8p. Output is being written to: /tmp/claude-0/-workspace/b3699ac4-1c9f-4c54-b65c-10b5011c1085/tasks/b8jbrmq8p.output. You will be notified when it completes. To check interim output, use Read on that file path.
Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "b8jbrmq8p", "backgroundCwdHint": "Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands."}

(Bash completed with no output)

[metadata] {"noOutputExpected": false}

-rw-r--r-- 1 root root 215961 Aug  4 04:39 submission/selection.json

[stdout]
-rw-r--r-- 1 root root 215961 Aug  4 04:39 submission/selection.json
[metadata] {"noOutputExpected": false}

wrote 27817 ids (~30002008 est tokens, tech~10400462) -> /workspace/submission/selection.json
score p99/p50/min-selected: 4.30266970889042 -1.790838296251792

[stdout]
wrote 27817 ids (~30002008 est tokens, tech~10400462) -> /workspace/submission/selection.json
score p99/p50/min-selected: 4.30266970889042 -1.790838296251792
[metadata] {"noOutputExpected": false}

Command running in background with ID: bgg5bw1x3. Output is being written to: /tmp/claude-0/-workspace/b3699ac4-1c9f-4c54-b65c-10b5011c1085/tasks/bgg5bw1x3.output. You will be notified when it completes. To check interim output, use Read on that file path.
Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "bgg5bw1x3", "backgroundCwdHint": "Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands."}

packed 12000000 tokens from 27817 ids -> /tmp/dev_train.npy
HELDOUT_PPL 406.771  loss 6.0082  train_s 56  /tmp/dev_train.npy
2906
2886

[stdout]
packed 12000000 tokens from 27817 ids -> /tmp/dev_train.npy
HELDOUT_PPL 406.771  loss 6.0082  train_s 56  /tmp/dev_train.npy
2906
2886
[metadata] {"noOutputExpected": false}

ids 27817 unique & in-range OK
curate.py TECH_SHARE matches submission (v5)

[stdout]
ids 27817 unique & in-range OK
curate.py TECH_SHARE matches submission (v5)
[metadata] {"noOutputExpected": false}

The file /workspace/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/curate.py", "oldString": "Every pool document is scored by the classifier's decision value = \"how\ntarget-like\". Documents are ranked highest-first, subject to light quality gates\n(length floor, alpha-fraction) and near-duplicate removal, then emitted in\npriority order until well past the 12M-token training budget.\n\nLength-robustness: features are L2-normalized, so score reflects the *direction*\nof a document's n-gram profile, not its length or raw repetition — this avoids\nthe failure mode of raw log-likelihood-ratio ranking, which floats generic\nnavigation/list spam to the top.\n\"\"\"", "newString": "Every pool document is scored by the classifier's decision value = \"how\ntarget-like\". Documents are ranked highest-first, subject to light quality gates\n(length floor, alpha-fraction, English-only) and near-duplicate removal.\n\nREGISTER BALANCING (key step): the target is EQUAL PARTS four registers, one of\nwhich is technical Q&A / code. A plain top-by-score selection floats news/prose\nto the top and starves the technical register (<0.2% of the budget), leaving a\nwhole quarter of the target badly modeled. So we detect technical documents with\na code-signal heuristic and interleave the two ranked lists to guarantee the\ntechnical register ~TECH_SHARE of the 12M-token budget, emitting the merged\npriority order until well past budget.\n\nLength-robustness: features are L2-normalized, so score reflects the *direction*\nof a document's n-gram profile, not its length or raw repetition — this avoids\nthe failure mode of raw log-likelihood-ratio ranking, which floats generic\nnavigation/list spam to the top.\n\nDev perplexity (frozen train_nano.py, multi_dev target): random selection 474.9\n-> classifier 412.3 -> +English gate 410.8 -> +25% tech 407.5 -> +35% tech 406.8.\n\"\"\"", "originalFile": "\"\"\"Curate the raw web pool for a small LM whose target is a BROAD, multi-domain\nhigh-quality English distribution (Wikipedia + high-quality web prose + news +\ntechnical Q&A).\n\nStated, reproducible criterion: a QUALITY/DOMAIN classifier.\nWe train a logistic-regression classifier on L2-normalized hashed unigram+bigram\nword features to separate\n  positive = the disclosed HQ target (multi_dev.npy decoded into documents), from\n  negative = random raw pool documents.\nEvery pool document is scored by the classifier's decision value = \"how\ntarget-like\". Documents are ranked highest-first, subject to light quality gates\n(length floor, alpha-fraction) and near-duplicate removal, then emitted in\npriority order until well past the 12M-token training budget.\n\nLength-robustness: features are L2-normalized, so score reflects the *direction*\nof a document's n-gram profile, not its length or raw repetition — this avoids\nthe failure mode of raw log-likelihood-ratio ranking, which floats generic\nnavigation/list spam to the top.\n\"\"\"\nimport json, re, zlib, numpy as np\nfrom transformers import AutoTokenizer\n\nPOOL = \"/workspace/data/pool.jsonl\"\nDEV  = \"/workspace/data/multi_dev.npy\"\nOUT  = \"/workspace/submission/selection.json\"\n\nB = 1 << 20                 # hash buckets\nNEG_SAMPLE = 24000          # random pool docs used as negatives\nEPOCHS = 6\nLR = 0.5\nL2 = 1e-6\nSEED = 0\n\n_word = re.compile(r\"[a-z0-9]+\")\ndef words_of(s):\n    return _word.findall(s.lower())\n\n# small English function-word set — fluent prose is rich in these; navigation\n# menus / name lists / license-plate spam are not. A cheap, robust fluency gate.\nSTOP = set(\"the of and to a in is that it for was on as with by at be this from or \"\n           \"an are not but he she they we you his her their which have has had were \"\n           \"been i who what when where how all would there been more one about\".split())\n\ndef english_frac(s):\n    \"\"\"Fraction of ASCII latin letters among all alphabetic characters (0..1).\"\"\"\n    a = ascii_ = 0\n    for c in s:\n        if c.isalpha():\n            a += 1\n            if 'a' <= c <= 'z' or 'A' <= c <= 'Z':\n                ascii_ += 1\n    return ascii_ / max(1, a)\n\ndef crc(b):\n    return zlib.crc32(b) & (B - 1)\n\ndef featvec(words):\n    \"\"\"Return (unique_bucket_ids int32, L2-normalized float32 values) for\n    unigram+bigram hashed features of a word list.\"\"\"\n    if not words:\n        return np.empty(0, np.int32), np.empty(0, np.float32)\n    h = [crc(w.encode()) for w in words]\n    for i in range(len(words) - 1):\n        h.append(crc((words[i] + \"\\x00\" + words[i+1]).encode()))\n    h = np.asarray(h, dtype=np.int64)\n    idx, cnt = np.unique(h, return_counts=True)\n    v = cnt.astype(np.float32)\n    v /= np.sqrt((v * v).sum())\n    return idx.astype(np.int32), v\n\n# ---------- 1. build training features ----------\nprint(\"decoding target ...\")\ntok = AutoTokenizer.from_pretrained(\"gpt2\")\ndev = np.load(DEV)\neos = tok.eos_token_id\narr = dev.tolist()\n# split target into documents on EOS\npos = [i for i, t in enumerate(arr) if t == eos]\nsegs, prev = [], 0\nfor p in pos:\n    if p - prev > 5:\n        segs.append(arr[prev:p])\n    prev = p + 1\nif len(arr) - prev > 5:\n    segs.append(arr[prev:])\npos_feats = [featvec(words_of(tok.decode(s))) for s in segs]\nprint(\"target docs:\", len(pos_feats))\n\nprint(\"loading pool ...\")\nids, texts = [], []\nfor line in open(POOL):\n    r = json.loads(line)\n    ids.append(r[\"id\"]); texts.append(r[\"text\"])\nN = len(ids)\nprint(\"pool docs:\", N)\n\nrng = np.random.default_rng(SEED)\nneg_idx = rng.choice(N, size=min(NEG_SAMPLE, N), replace=False)\nprint(\"building negative features ...\")\nneg_feats = [featvec(words_of(texts[j])) for j in neg_idx]\n\n# ---------- 2. train logistic regression (sparse SGD) ----------\nprint(\"training classifier ...\")\nw = np.zeros(B, dtype=np.float32)\nb = 0.0\ntrain = [(f, 1.0) for f in pos_feats] + [(f, 0.0) for f in neg_feats]\norder = np.arange(len(train))\nfor ep in range(EPOCHS):\n    rng.shuffle(order)\n    lr = LR * (1.0 - ep / (EPOCHS + 1))\n    loss = 0.0\n    for oi in order:\n        (idx, v), y = train[oi]\n        if idx.size == 0:\n            continue\n        z = float(w[idx] @ v) + b\n        p = 1.0 / (1.0 + np.exp(-z))\n        g = p - y\n        w[idx] -= lr * (g * v + L2 * w[idx])\n        b -= lr * g\n        loss += -(y*np.log(p+1e-9) + (1-y)*np.log(1-p+1e-9))\n    print(f\"  epoch {ep} loss {loss/len(train):.4f}\")\n\n# The target is EQUAL PARTS four registers incl. technical Q&A / code. Detect the\n# technical register so we can guarantee it a token quota — otherwise the plain\n# classifier floats news/prose to the top and code is starved (<0.2% of budget),\n# leaving a whole quarter of the target badly modeled.\nTECH_SIG = ['<pre', '<code', 'stack overflow', 'stackoverflow', 'function(', 'var ',\n            'import ', 'public static', '#include', 'console.log', 'println', '</code',\n            '&lt;', 'def ', 'npm ', 'printf', 'const ', 'return ', ');']\ndef tech_score(tl):\n    return sum(1 for m in TECH_SIG if m in tl)\n\n# ---------- 3. score every document, with register-aware gates ----------\nprint(\"scoring documents ...\")\nscores = np.full(N, -1e9, dtype=np.float64)\nntok = np.zeros(N, dtype=np.int32)\nis_tech = np.zeros(N, dtype=bool)\nfor j in range(N):\n    t = texts[j]\n    wds = words_of(t)\n    n = len(wds)\n    ntok[j] = n\n    if n < 50:                                    # length floor\n        continue\n    head = t[:3000]\n    tech = tech_score(t.lower()) >= 2\n    is_tech[j] = tech\n    alpha = sum(c.isalpha() for c in head) / max(1, len(head))\n    if tech:\n        if alpha < 0.30:                          # code is symbol-heavy: relax gate\n            continue\n    else:\n        if alpha < 0.55 or english_frac(head) < 0.85:\n            continue                              # prose gates: clean English only\n    idx, v = featvec(wds)\n    scores[j] = float(w[idx] @ v) + b\n    if j % 40000 == 0:\n        print(f\"  {j}/{N}\")\n\n# ---------- 4. rank each register, merge to a balanced priority order ----------\nprint(\"ranking + balancing + dedup ...\")\nBUDGET = 12_000_000\nTECH_SHARE = 0.35            # technical register's share of the 12M budget\norder = np.argsort(-scores)\ntech_q = [j for j in order if scores[j] > -1e8 and is_tech[j]]\nrest_q = [j for j in order if scores[j] > -1e8 and not is_tech[j]]\n\ndef est(j):\n    return int(ntok[j] * 1.3) + 1\n\nseen = set()\ndef fresh(j):\n    wl = words_of(texts[j])\n    key = (hash(\" \".join(wl[:60])), len(wl) // 40)\n    if key in seen:\n        return False\n    seen.add(key)\n    return True\n\nselection, tot, tech_tok = [], 0, 0\nti = ri = 0\n# interleave: pull from tech whenever its running token share is below target\nwhile tot < BUDGET * 2.5 and (ti < len(tech_q) or ri < len(rest_q)):\n    want_tech = (tech_tok < TECH_SHARE * max(1, tot)) and ti < len(tech_q)\n    if want_tech:\n        j = tech_q[ti]; ti += 1\n        if not fresh(j):\n            continue\n        selection.append(int(ids[j])); e = est(j); tot += e; tech_tok += e\n    else:\n        if ri >= len(rest_q):\n            if ti >= len(tech_q):\n                break\n            continue\n        j = rest_q[ri]; ri += 1\n        if not fresh(j):\n            continue\n        selection.append(int(ids[j])); tot += est(j)\n\njson.dump(selection, open(OUT, \"w\"))\nprint(f\"wrote {len(selection)} ids (~{tot} est tokens, tech~{tech_tok}) -> {OUT}\")\nsrt = scores[order]\nprint(\"score p99/p50/min-selected:\", float(srt[0]),\n      float(scores[order[len(selection)//2]]))\n", "structuredPatch": [{"oldStart": 9, "oldLines": 13, "newStart": 9, "newLines": 23, "lines": ["   negative = random raw pool documents.", " Every pool document is scored by the classifier's decision value = \"how", " target-like\". Documents are ranked highest-first, subject to light quality gates", "-(length floor, alpha-fraction) and near-duplicate removal, then emitted in", "-priority order until well past the 12M-token training budget.", "+(length floor, alpha-fraction, English-only) and near-duplicate removal.", " ", "+REGISTER BALANCING (key step): the target is EQUAL PARTS four registers, one of", "+which is technical Q&A / code. A plain top-by-score selection floats news/prose", "+to the top and starves the technical register (<0.2% of the budget), leaving a", "+whole quarter of the target badly modeled. So we detect technical documents with", "+a code-signal heuristic and interleave the two ranked lists to guarantee the", "+technical register ~TECH_SHARE of the 12M-token budget, emitting the merged", "+priority order until well past budget.", "+", " Length-robustness: features are L2-normalized, so score reflects the *direction*", " of a document's n-gram profile, not its length or raw repetition — this avoids", " the failure mode of raw log-likelihood-ratio ranking, which floats generic", " navigation/list spam to the top.", "+", "+Dev perplexity (frozen train_nano.py, multi_dev target): random selection 474.9", "+-> classifier 412.3 -> +English gate 410.8 -> +25% tech 407.5 -> +35% tech 406.8.", " \"\"\"", " import json, re, zlib, numpy as np", " from transformers import AutoTokenizer"]}], "userModified": false, "replaceAll": false}

File created successfully at: /workspace/claim.md (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/workspace/claim.md", "content": "# Curation claim\n\n## Hypothesis\nFor a fixed 12M-token budget, held-out perplexity on a *broad, equal-parts*\nmulti-domain target (Wikipedia / high-quality web prose / news / technical Q&A)\nis minimized not merely by selecting the documents most similar to the target,\nbut by selecting target-like documents **while enforcing coverage of every\ntarget register** — in particular the technical Q&A / code register, which a\nnaive similarity ranking starves. Both quality *and* register balance matter.\n\n## Mechanism (observable other than final perplexity)\nA logistic-regression classifier (target vs. random pool, on L2-normalized\nhashed unigram+bigram features) ranks documents by \"target-likeness.\" The\n**observable prediction** is a *compositional* one, checkable without training:\n\n- When the top-N documents that fill the 12M budget are bucketed by register,\n  a pure top-by-score selection is **dominated by news + general prose and\n  contains almost no code/tech** (measured: ~48% news, ~51% prose, **15 of\n  ~11,000** documents technical). The technical quarter of the target is\n  therefore absent from training.\n- Injecting a technical-register quota changes the training mix's composition in\n  a directly measurable way (tech token share 0.2% → ~25–35%), and this is the\n  step that moves dev perplexity.\n\nPrediction: register composition of the selected head, not raw average\nclassifier score, tracks perplexity. Concretely, at fixed classifier quality,\nraising the technical share from ~0% toward ~25% **lowers** dev perplexity, and\nthe gain saturates once the register is adequately represented (diminishing\nreturns beyond ~25–35%).\n\nThis was borne out on the dev target (frozen `train_nano.py`):\nrandom 474.9 → classifier-only 412.3 → +English gate 410.8 →\n+25% tech quota 407.5 → +35% tech quota 406.8. The classifier captures the large\nfirst-order gain (quality/domain match); the register quota captures a real\nsecond-order gain (coverage) that pure ranking leaves on the table, and it\nsaturates — exactly the predicted shape.\n\n## Falsification\nThe claim is false if either holds:\n1. **Balance doesn't help.** If, holding the classifier fixed, adding a\n   technical-register quota (0% → 25%) does *not* reduce held-out perplexity (or\n   increases it), then coverage is not a distinct lever beyond similarity.\n2. **Similarity doesn't help.** If the classifier-ranked selection scores no\n   better than a random selection at equal token budget, the target-likeness\n   signal is worthless. (Observed: 412 vs 475 — clearly rejected.)\nA further falsifier: if perplexity kept dropping monotonically as tech share → 1,\nthe \"equal-parts coverage\" story would be wrong and it would just be \"more code\nis better.\" Observed saturation (407.5 → 406.8 from 25%→35%) rejects that.\n\n## Transfer\nThe recipe transfers to any fixed-budget pretraining-data curation against a\n*known heterogeneous* target: (1) train a cheap linear classifier target-vs-pool\non hashed n-grams for a length-robust quality/domain score; (2) explicitly\ndetect each target register and enforce per-register token quotas so no\ncomponent of a multi-domain target is starved. It requires only a sample of the\ntarget distribution and needs no labels on the pool. The specific technical-\nsignal heuristic is domain-specific, but the two-part principle — *rank for\nquality, allocate for coverage* — is general, and is exactly what single-score\nselection methods (DSIR, perplexity filtering, one classifier) miss on\nmulti-domain targets.\n```\n```\n", "structuredPatch": [], "originalFile": null, "userModified": false}

The file /workspace/claim.md has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/claim.md", "oldString": "single-score\nselection methods (DSIR, perplexity filtering, one classifier) miss on\nmulti-domain targets.\n```\n```", "newString": "single-score\nselection methods (DSIR, perplexity filtering, one classifier) miss on\nmulti-domain targets.", "originalFile": "# Curation claim\n\n## Hypothesis\nFor a fixed 12M-token budget, held-out perplexity on a *broad, equal-parts*\nmulti-domain target (Wikipedia / high-quality web prose / news / technical Q&A)\nis minimized not merely by selecting the documents most similar to the target,\nbut by selecting target-like documents **while enforcing coverage of every\ntarget register** — in particular the technical Q&A / code register, which a\nnaive similarity ranking starves. Both quality *and* register balance matter.\n\n## Mechanism (observable other than final perplexity)\nA logistic-regression classifier (target vs. random pool, on L2-normalized\nhashed unigram+bigram features) ranks documents by \"target-likeness.\" The\n**observable prediction** is a *compositional* one, checkable without training:\n\n- When the top-N documents that fill the 12M budget are bucketed by register,\n  a pure top-by-score selection is **dominated by news + general prose and\n  contains almost no code/tech** (measured: ~48% news, ~51% prose, **15 of\n  ~11,000** documents technical). The technical quarter of the target is\n  therefore absent from training.\n- Injecting a technical-register quota changes the training mix's composition in\n  a directly measurable way (tech token share 0.2% → ~25–35%), and this is the\n  step that moves dev perplexity.\n\nPrediction: register composition of the selected head, not raw average\nclassifier score, tracks perplexity. Concretely, at fixed classifier quality,\nraising the technical share from ~0% toward ~25% **lowers** dev perplexity, and\nthe gain saturates once the register is adequately represented (diminishing\nreturns beyond ~25–35%).\n\nThis was borne out on the dev target (frozen `train_nano.py`):\nrandom 474.9 → classifier-only 412.3 → +English gate 410.8 →\n+25% tech quota 407.5 → +35% tech quota 406.8. The classifier captures the large\nfirst-order gain (quality/domain match); the register quota captures a real\nsecond-order gain (coverage) that pure ranking leaves on the table, and it\nsaturates — exactly the predicted shape.\n\n## Falsification\nThe claim is false if either holds:\n1. **Balance doesn't help.** If, holding the classifier fixed, adding a\n   technical-register quota (0% → 25%) does *not* reduce held-out perplexity (or\n   increases it), then coverage is not a distinct lever beyond similarity.\n2. **Similarity doesn't help.** If the classifier-ranked selection scores no\n   better than a random selection at equal token budget, the target-likeness\n   signal is worthless. (Observed: 412 vs 475 — clearly rejected.)\nA further falsifier: if perplexity kept dropping monotonically as tech share → 1,\nthe \"equal-parts coverage\" story would be wrong and it would just be \"more code\nis better.\" Observed saturation (407.5 → 406.8 from 25%→35%) rejects that.\n\n## Transfer\nThe recipe transfers to any fixed-budget pretraining-data curation against a\n*known heterogeneous* target: (1) train a cheap linear classifier target-vs-pool\non hashed n-grams for a length-robust quality/domain score; (2) explicitly\ndetect each target register and enforce per-register token quotas so no\ncomponent of a multi-domain target is starved. It requires only a sample of the\ntarget distribution and needs no labels on the pool. The specific technical-\nsignal heuristic is domain-specific, but the two-part principle — *rank for\nquality, allocate for coverage* — is general, and is exactly what single-score\nselection methods (DSIR, perplexity filtering, one classifier) miss on\nmulti-domain targets.\n```\n```\n", "structuredPatch": [{"oldStart": 58, "oldLines": 5, "newStart": 58, "newLines": 3, "lines": [" quality, allocate for coverage* — is general, and is exactly what single-score", " selection methods (DSIR, perplexity filtering, one classifier) miss on", " multi-domain targets.", "-```", "-```"]}], "userModified": false, "replaceAll": false}

total 201
drwxr-xr-x 1 root root     80 Aug  4 04:49 .
drwxr-xr-x 1 root root    380 Aug  4 04:49 ..
-rw-r--r-- 1 root root   8079 Aug  4 04:49 curate.py
-rw-r--r-- 1 root root 197529 Aug  4 04:45 selection.json
--- final files ---
[13746, 127454, 57336, 7849, 79653, 162401, 138557, 49061, 109550, 3196, 63484, 76107, 100430, 82269, 67453, 2403, 72960, 37522, 42070, 26975, 3393, 108673, 11714, 2186, 44750, 46689, 23727, 126683, 9
--- confirm pack fills 12M ---
packed 12000000 tokens from 27817 ids -> /tmp/final_check.npy
2746

[stdout]
total 201
drwxr-xr-x 1 root root     80 Aug  4 04:49 .
drwxr-xr-x 1 root root    380 Aug  4 04:49 ..
-rw-r--r-- 1 root root   8079 Aug  4 04:49 curate.py
-rw-r--r-- 1 root root 197529 Aug  4 04:45 selection.json
--- final files ---
[13746, 127454, 57336, 7849, 79653, 162401, 138557, 49061, 109550, 3196, 63484, 76107, 100430, 82269, 67453, 2403, 72960, 37522, 42070, 26975, 3393, 108673, 11714, 2186, 44750, 46689, 23727, 126683, 9
--- confirm pack fills 12M ---
packed 12000000 tokens from 27817 ids -> /tmp/final_check.npy
2746
[metadata] {"noOutputExpected": false}

The file /workspace/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/curate.py", "oldString": "TECH_SIG = ['<pre', '<code', 'stack overflow', 'stackoverflow', 'function(', 'var ',\n            'import ', 'public static', '#include', 'console.log', 'println', '</code',\n            '&lt;', 'def ', 'npm ', 'printf', 'const ', 'return ', ');']\ndef tech_score(tl):\n    return sum(1 for m in TECH_SIG if m in tl)", "newString": "TECH_SIG = ['<pre', '<code', 'stack overflow', 'stackoverflow', 'function(', 'var ',\n            'import ', 'public static', '#include', 'console.log', 'println', '</code',\n            '&lt;', 'def ', 'npm ', 'printf', 'const ', 'return ', ');']\ndef tech_score(tl):\n    return sum(1 for m in TECH_SIG if m in tl)\n\n# News cue heuristic — the pool is heavy on news portals, so a plain ranking\n# over-weights news (~48% of the head) vs. the target's ~25%, starving the\n# encyclopedic/prose registers. Detect news so we can cap its budget share too.\nNEWS_SIG = [' said ', ' told reporters', 'reuters', '(ap)', 'associated press',\n            'on monday', 'on tuesday', 'on wednesday', 'on thursday', 'on friday',\n            'on saturday', 'on sunday', 'the minister', 'prime minister', 'the police',\n            'according to', 'officials said', 'spokesman', 'spokesperson']\ndef news_score(tl):\n    return sum(1 for m in NEWS_SIG if m in tl)", "originalFile": "\"\"\"Curate the raw web pool for a small LM whose target is a BROAD, multi-domain\nhigh-quality English distribution (Wikipedia + high-quality web prose + news +\ntechnical Q&A).\n\nStated, reproducible criterion: a QUALITY/DOMAIN classifier.\nWe train a logistic-regression classifier on L2-normalized hashed unigram+bigram\nword features to separate\n  positive = the disclosed HQ target (multi_dev.npy decoded into documents), from\n  negative = random raw pool documents.\nEvery pool document is scored by the classifier's decision value = \"how\ntarget-like\". Documents are ranked highest-first, subject to light quality gates\n(length floor, alpha-fraction, English-only) and near-duplicate removal.\n\nREGISTER BALANCING (key step): the target is EQUAL PARTS four registers, one of\nwhich is technical Q&A / code. A plain top-by-score selection floats news/prose\nto the top and starves the technical register (<0.2% of the budget), leaving a\nwhole quarter of the target badly modeled. So we detect technical documents with\na code-signal heuristic and interleave the two ranked lists to guarantee the\ntechnical register ~TECH_SHARE of the 12M-token budget, emitting the merged\npriority order until well past budget.\n\nLength-robustness: features are L2-normalized, so score reflects the *direction*\nof a document's n-gram profile, not its length or raw repetition — this avoids\nthe failure mode of raw log-likelihood-ratio ranking, which floats generic\nnavigation/list spam to the top.\n\nDev perplexity (frozen train_nano.py, multi_dev target): random selection 474.9\n-> classifier 412.3 -> +English gate 410.8 -> +25% tech 407.5 -> +35% tech 406.8.\n\"\"\"\nimport json, re, zlib, numpy as np\nfrom transformers import AutoTokenizer\n\nPOOL = \"/workspace/data/pool.jsonl\"\nDEV  = \"/workspace/data/multi_dev.npy\"\nOUT  = \"/workspace/submission/selection.json\"\n\nB = 1 << 20                 # hash buckets\nNEG_SAMPLE = 24000          # random pool docs used as negatives\nEPOCHS = 6\nLR = 0.5\nL2 = 1e-6\nSEED = 0\n\n_word = re.compile(r\"[a-z0-9]+\")\ndef words_of(s):\n    return _word.findall(s.lower())\n\n# small English function-word set — fluent prose is rich in these; navigation\n# menus / name lists / license-plate spam are not. A cheap, robust fluency gate.\nSTOP = set(\"the of and to a in is that it for was on as with by at be this from or \"\n           \"an are not but he she they we you his her their which have has had were \"\n           \"been i who what when where how all would there been more one about\".split())\n\ndef english_frac(s):\n    \"\"\"Fraction of ASCII latin letters among all alphabetic characters (0..1).\"\"\"\n    a = ascii_ = 0\n    for c in s:\n        if c.isalpha():\n            a += 1\n            if 'a' <= c <= 'z' or 'A' <= c <= 'Z':\n                ascii_ += 1\n    return ascii_ / max(1, a)\n\ndef crc(b):\n    return zlib.crc32(b) & (B - 1)\n\ndef featvec(words):\n    \"\"\"Return (unique_bucket_ids int32, L2-normalized float32 values) for\n    unigram+bigram hashed features of a word list.\"\"\"\n    if not words:\n        return np.empty(0, np.int32), np.empty(0, np.float32)\n    h = [crc(w.encode()) for w in words]\n    for i in range(len(words) - 1):\n        h.append(crc((words[i] + \"\\x00\" + words[i+1]).encode()))\n    h = np.asarray(h, dtype=np.int64)\n    idx, cnt = np.unique(h, return_counts=True)\n    v = cnt.astype(np.float32)\n    v /= np.sqrt((v * v).sum())\n    return idx.astype(np.int32), v\n\n# ---------- 1. build training features ----------\nprint(\"decoding target ...\")\ntok = AutoTokenizer.from_pretrained(\"gpt2\")\ndev = np.load(DEV)\neos = tok.eos_token_id\narr = dev.tolist()\n# split target into documents on EOS\npos = [i for i, t in enumerate(arr) if t == eos]\nsegs, prev = [], 0\nfor p in pos:\n    if p - prev > 5:\n        segs.append(arr[prev:p])\n    prev = p + 1\nif len(arr) - prev > 5:\n    segs.append(arr[prev:])\npos_feats = [featvec(words_of(tok.decode(s))) for s in segs]\nprint(\"target docs:\", len(pos_feats))\n\nprint(\"loading pool ...\")\nids, texts = [], []\nfor line in open(POOL):\n    r = json.loads(line)\n    ids.append(r[\"id\"]); texts.append(r[\"text\"])\nN = len(ids)\nprint(\"pool docs:\", N)\n\nrng = np.random.default_rng(SEED)\nneg_idx = rng.choice(N, size=min(NEG_SAMPLE, N), replace=False)\nprint(\"building negative features ...\")\nneg_feats = [featvec(words_of(texts[j])) for j in neg_idx]\n\n# ---------- 2. train logistic regression (sparse SGD) ----------\nprint(\"training classifier ...\")\nw = np.zeros(B, dtype=np.float32)\nb = 0.0\ntrain = [(f, 1.0) for f in pos_feats] + [(f, 0.0) for f in neg_feats]\norder = np.arange(len(train))\nfor ep in range(EPOCHS):\n    rng.shuffle(order)\n    lr = LR * (1.0 - ep / (EPOCHS + 1))\n    loss = 0.0\n    for oi in order:\n        (idx, v), y = train[oi]\n        if idx.size == 0:\n            continue\n        z = float(w[idx] @ v) + b\n        p = 1.0 / (1.0 + np.exp(-z))\n        g = p - y\n        w[idx] -= lr * (g * v + L2 * w[idx])\n        b -= lr * g\n        loss += -(y*np.log(p+1e-9) + (1-y)*np.log(1-p+1e-9))\n    print(f\"  epoch {ep} loss {loss/len(train):.4f}\")\n\n# The target is EQUAL PARTS four registers incl. technical Q&A / code. Detect the\n# technical register so we can guarantee it a token quota — otherwise the plain\n# classifier floats news/prose to the top and code is starved (<0.2% of budget),\n# leaving a whole quarter of the target badly modeled.\nTECH_SIG = ['<pre', '<code', 'stack overflow', 'stackoverflow', 'function(', 'var ',\n            'import ', 'public static', '#include', 'console.log', 'println', '</code',\n            '&lt;', 'def ', 'npm ', 'printf', 'const ', 'return ', ');']\ndef tech_score(tl):\n    return sum(1 for m in TECH_SIG if m in tl)\n\n# ---------- 3. score every document, with register-aware gates ----------\nprint(\"scoring documents ...\")\nscores = np.full(N, -1e9, dtype=np.float64)\nntok = np.zeros(N, dtype=np.int32)\nis_tech = np.zeros(N, dtype=bool)\nfor j in range(N):\n    t = texts[j]\n    wds = words_of(t)\n    n = len(wds)\n    ntok[j] = n\n    if n < 50:                                    # length floor\n        continue\n    head = t[:3000]\n    tech = tech_score(t.lower()) >= 2\n    is_tech[j] = tech\n    alpha = sum(c.isalpha() for c in head) / max(1, len(head))\n    if tech:\n        if alpha < 0.30:                          # code is symbol-heavy: relax gate\n            continue\n    else:\n        if alpha < 0.55 or english_frac(head) < 0.85:\n            continue                              # prose gates: clean English only\n    idx, v = featvec(wds)\n    scores[j] = float(w[idx] @ v) + b\n    if j % 40000 == 0:\n        print(f\"  {j}/{N}\")\n\n# ---------- 4. rank each register, merge to a balanced priority order ----------\nprint(\"ranking + balancing + dedup ...\")\nBUDGET = 12_000_000\nTECH_SHARE = 0.35            # technical register's share of the 12M budget\norder = np.argsort(-scores)\ntech_q = [j for j in order if scores[j] > -1e8 and is_tech[j]]\nrest_q = [j for j in order if scores[j] > -1e8 and not is_tech[j]]\n\ndef est(j):\n    return int(ntok[j] * 1.3) + 1\n\nseen = set()\ndef fresh(j):\n    wl = words_of(texts[j])\n    key = (hash(\" \".join(wl[:60])), len(wl) // 40)\n    if key in seen:\n        return False\n    seen.add(key)\n    return True\n\nselection, tot, tech_tok = [], 0, 0\nti = ri = 0\n# interleave: pull from tech whenever its running token share is below target\nwhile tot < BUDGET * 2.5 and (ti < len(tech_q) or ri < len(rest_q)):\n    want_tech = (tech_tok < TECH_SHARE * max(1, tot)) and ti < len(tech_q)\n    if want_tech:\n        j = tech_q[ti]; ti += 1\n        if not fresh(j):\n            continue\n        selection.append(int(ids[j])); e = est(j); tot += e; tech_tok += e\n    else:\n        if ri >= len(rest_q):\n            if ti >= len(tech_q):\n                break\n            continue\n        j = rest_q[ri]; ri += 1\n        if not fresh(j):\n            continue\n        selection.append(int(ids[j])); tot += est(j)\n\njson.dump(selection, open(OUT, \"w\"))\nprint(f\"wrote {len(selection)} ids (~{tot} est tokens, tech~{tech_tok}) -> {OUT}\")\nsrt = scores[order]\nprint(\"score p99/p50/min-selected:\", float(srt[0]),\n      float(scores[order[len(selection)//2]]))\n", "structuredPatch": [{"oldStart": 141, "oldLines": 6, "newStart": 141, "newLines": 16, "lines": [" def tech_score(tl):", "     return sum(1 for m in TECH_SIG if m in tl)", " ", "+# News cue heuristic — the pool is heavy on news portals, so a plain ranking", "+# over-weights news (~48% of the head) vs. the target's ~25%, starving the", "+# encyclopedic/prose registers. Detect news so we can cap its budget share too.", "+NEWS_SIG = [' said ', ' told reporters', 'reuters', '(ap)', 'associated press',", "+            'on monday', 'on tuesday', 'on wednesday', 'on thursday', 'on friday',", "+            'on saturday', 'on sunday', 'the minister', 'prime minister', 'the police',", "+            'according to', 'officials said', 'spokesman', 'spokesperson']", "+def news_score(tl):", "+    return sum(1 for m in NEWS_SIG if m in tl)", "+", " # ---------- 3. score every document, with register-aware gates ----------", " print(\"scoring documents ...\")", " scores = np.full(N, -1e9, dtype=np.float64)"]}], "userModified": false, "replaceAll": false}

The file /workspace/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/curate.py", "oldString": "scores = np.full(N, -1e9, dtype=np.float64)\nntok = np.zeros(N, dtype=np.int32)\nis_tech = np.zeros(N, dtype=bool)\nfor j in range(N):\n    t = texts[j]\n    wds = words_of(t)\n    n = len(wds)\n    ntok[j] = n\n    if n < 50:                                    # length floor\n        continue\n    head = t[:3000]\n    tech = tech_score(t.lower()) >= 2\n    is_tech[j] = tech\n    alpha = sum(c.isalpha() for c in head) / max(1, len(head))\n    if tech:\n        if alpha < 0.30:                          # code is symbol-heavy: relax gate\n            continue\n    else:\n        if alpha < 0.55 or english_frac(head) < 0.85:\n            continue                              # prose gates: clean English only\n    idx, v = featvec(wds)\n    scores[j] = float(w[idx] @ v) + b\n    if j % 40000 == 0:\n        print(f\"  {j}/{N}\")", "newString": "scores = np.full(N, -1e9, dtype=np.float64)\nntok = np.zeros(N, dtype=np.int32)\nbucket = np.zeros(N, dtype=np.int8)               # 0=other, 1=tech, 2=news\nfor j in range(N):\n    t = texts[j]\n    tl = t.lower()\n    wds = words_of(t)\n    n = len(wds)\n    ntok[j] = n\n    if n < 50:                                    # length floor\n        continue\n    head = t[:3000]\n    tech = tech_score(tl) >= 2\n    alpha = sum(c.isalpha() for c in head) / max(1, len(head))\n    if tech:\n        if alpha < 0.30:                          # code is symbol-heavy: relax gate\n            continue\n        bucket[j] = 1\n    else:\n        if alpha < 0.55 or english_frac(head) < 0.85:\n            continue                              # prose gates: clean English only\n        bucket[j] = 2 if news_score(tl) >= 2 else 0\n    idx, v = featvec(wds)\n    scores[j] = float(w[idx] @ v) + b\n    if j % 40000 == 0:\n        print(f\"  {j}/{N}\")", "originalFile": "\"\"\"Curate the raw web pool for a small LM whose target is a BROAD, multi-domain\nhigh-quality English distribution (Wikipedia + high-quality web prose + news +\ntechnical Q&A).\n\nStated, reproducible criterion: a QUALITY/DOMAIN classifier.\nWe train a logistic-regression classifier on L2-normalized hashed unigram+bigram\nword features to separate\n  positive = the disclosed HQ target (multi_dev.npy decoded into documents), from\n  negative = random raw pool documents.\nEvery pool document is scored by the classifier's decision value = \"how\ntarget-like\". Documents are ranked highest-first, subject to light quality gates\n(length floor, alpha-fraction, English-only) and near-duplicate removal.\n\nREGISTER BALANCING (key step): the target is EQUAL PARTS four registers, one of\nwhich is technical Q&A / code. A plain top-by-score selection floats news/prose\nto the top and starves the technical register (<0.2% of the budget), leaving a\nwhole quarter of the target badly modeled. So we detect technical documents with\na code-signal heuristic and interleave the two ranked lists to guarantee the\ntechnical register ~TECH_SHARE of the 12M-token budget, emitting the merged\npriority order until well past budget.\n\nLength-robustness: features are L2-normalized, so score reflects the *direction*\nof a document's n-gram profile, not its length or raw repetition — this avoids\nthe failure mode of raw log-likelihood-ratio ranking, which floats generic\nnavigation/list spam to the top.\n\nDev perplexity (frozen train_nano.py, multi_dev target): random selection 474.9\n-> classifier 412.3 -> +English gate 410.8 -> +25% tech 407.5 -> +35% tech 406.8.\n\"\"\"\nimport json, re, zlib, numpy as np\nfrom transformers import AutoTokenizer\n\nPOOL = \"/workspace/data/pool.jsonl\"\nDEV  = \"/workspace/data/multi_dev.npy\"\nOUT  = \"/workspace/submission/selection.json\"\n\nB = 1 << 20                 # hash buckets\nNEG_SAMPLE = 24000          # random pool docs used as negatives\nEPOCHS = 6\nLR = 0.5\nL2 = 1e-6\nSEED = 0\n\n_word = re.compile(r\"[a-z0-9]+\")\ndef words_of(s):\n    return _word.findall(s.lower())\n\n# small English function-word set — fluent prose is rich in these; navigation\n# menus / name lists / license-plate spam are not. A cheap, robust fluency gate.\nSTOP = set(\"the of and to a in is that it for was on as with by at be this from or \"\n           \"an are not but he she they we you his her their which have has had were \"\n           \"been i who what when where how all would there been more one about\".split())\n\ndef english_frac(s):\n    \"\"\"Fraction of ASCII latin letters among all alphabetic characters (0..1).\"\"\"\n    a = ascii_ = 0\n    for c in s:\n        if c.isalpha():\n            a += 1\n            if 'a' <= c <= 'z' or 'A' <= c <= 'Z':\n                ascii_ += 1\n    return ascii_ / max(1, a)\n\ndef crc(b):\n    return zlib.crc32(b) & (B - 1)\n\ndef featvec(words):\n    \"\"\"Return (unique_bucket_ids int32, L2-normalized float32 values) for\n    unigram+bigram hashed features of a word list.\"\"\"\n    if not words:\n        return np.empty(0, np.int32), np.empty(0, np.float32)\n    h = [crc(w.encode()) for w in words]\n    for i in range(len(words) - 1):\n        h.append(crc((words[i] + \"\\x00\" + words[i+1]).encode()))\n    h = np.asarray(h, dtype=np.int64)\n    idx, cnt = np.unique(h, return_counts=True)\n    v = cnt.astype(np.float32)\n    v /= np.sqrt((v * v).sum())\n    return idx.astype(np.int32), v\n\n# ---------- 1. build training features ----------\nprint(\"decoding target ...\")\ntok = AutoTokenizer.from_pretrained(\"gpt2\")\ndev = np.load(DEV)\neos = tok.eos_token_id\narr = dev.tolist()\n# split target into documents on EOS\npos = [i for i, t in enumerate(arr) if t == eos]\nsegs, prev = [], 0\nfor p in pos:\n    if p - prev > 5:\n        segs.append(arr[prev:p])\n    prev = p + 1\nif len(arr) - prev > 5:\n    segs.append(arr[prev:])\npos_feats = [featvec(words_of(tok.decode(s))) for s in segs]\nprint(\"target docs:\", len(pos_feats))\n\nprint(\"loading pool ...\")\nids, texts = [], []\nfor line in open(POOL):\n    r = json.loads(line)\n    ids.append(r[\"id\"]); texts.append(r[\"text\"])\nN = len(ids)\nprint(\"pool docs:\", N)\n\nrng = np.random.default_rng(SEED)\nneg_idx = rng.choice(N, size=min(NEG_SAMPLE, N), replace=False)\nprint(\"building negative features ...\")\nneg_feats = [featvec(words_of(texts[j])) for j in neg_idx]\n\n# ---------- 2. train logistic regression (sparse SGD) ----------\nprint(\"training classifier ...\")\nw = np.zeros(B, dtype=np.float32)\nb = 0.0\ntrain = [(f, 1.0) for f in pos_feats] + [(f, 0.0) for f in neg_feats]\norder = np.arange(len(train))\nfor ep in range(EPOCHS):\n    rng.shuffle(order)\n    lr = LR * (1.0 - ep / (EPOCHS + 1))\n    loss = 0.0\n    for oi in order:\n        (idx, v), y = train[oi]\n        if idx.size == 0:\n            continue\n        z = float(w[idx] @ v) + b\n        p = 1.0 / (1.0 + np.exp(-z))\n        g = p - y\n        w[idx] -= lr * (g * v + L2 * w[idx])\n        b -= lr * g\n        loss += -(y*np.log(p+1e-9) + (1-y)*np.log(1-p+1e-9))\n    print(f\"  epoch {ep} loss {loss/len(train):.4f}\")\n\n# The target is EQUAL PARTS four registers incl. technical Q&A / code. Detect the\n# technical register so we can guarantee it a token quota — otherwise the plain\n# classifier floats news/prose to the top and code is starved (<0.2% of budget),\n# leaving a whole quarter of the target badly modeled.\nTECH_SIG = ['<pre', '<code', 'stack overflow', 'stackoverflow', 'function(', 'var ',\n            'import ', 'public static', '#include', 'console.log', 'println', '</code',\n            '&lt;', 'def ', 'npm ', 'printf', 'const ', 'return ', ');']\ndef tech_score(tl):\n    return sum(1 for m in TECH_SIG if m in tl)\n\n# News cue heuristic — the pool is heavy on news portals, so a plain ranking\n# over-weights news (~48% of the head) vs. the target's ~25%, starving the\n# encyclopedic/prose registers. Detect news so we can cap its budget share too.\nNEWS_SIG = [' said ', ' told reporters', 'reuters', '(ap)', 'associated press',\n            'on monday', 'on tuesday', 'on wednesday', 'on thursday', 'on friday',\n            'on saturday', 'on sunday', 'the minister', 'prime minister', 'the police',\n            'according to', 'officials said', 'spokesman', 'spokesperson']\ndef news_score(tl):\n    return sum(1 for m in NEWS_SIG if m in tl)\n\n# ---------- 3. score every document, with register-aware gates ----------\nprint(\"scoring documents ...\")\nscores = np.full(N, -1e9, dtype=np.float64)\nntok = np.zeros(N, dtype=np.int32)\nis_tech = np.zeros(N, dtype=bool)\nfor j in range(N):\n    t = texts[j]\n    wds = words_of(t)\n    n = len(wds)\n    ntok[j] = n\n    if n < 50:                                    # length floor\n        continue\n    head = t[:3000]\n    tech = tech_score(t.lower()) >= 2\n    is_tech[j] = tech\n    alpha = sum(c.isalpha() for c in head) / max(1, len(head))\n    if tech:\n        if alpha < 0.30:                          # code is symbol-heavy: relax gate\n            continue\n    else:\n        if alpha < 0.55 or english_frac(head) < 0.85:\n            continue                              # prose gates: clean English only\n    idx, v = featvec(wds)\n    scores[j] = float(w[idx] @ v) + b\n    if j % 40000 == 0:\n        print(f\"  {j}/{N}\")\n\n# ---------- 4. rank each register, merge to a balanced priority order ----------\nprint(\"ranking + balancing + dedup ...\")\nBUDGET = 12_000_000\nTECH_SHARE = 0.35            # technical register's share of the 12M budget\norder = np.argsort(-scores)\ntech_q = [j for j in order if scores[j] > -1e8 and is_tech[j]]\nrest_q = [j for j in order if scores[j] > -1e8 and not is_tech[j]]\n\ndef est(j):\n    return int(ntok[j] * 1.3) + 1\n\nseen = set()\ndef fresh(j):\n    wl = words_of(texts[j])\n    key = (hash(\" \".join(wl[:60])), len(wl) // 40)\n    if key in seen:\n        return False\n    seen.add(key)\n    return True\n\nselection, tot, tech_tok = [], 0, 0\nti = ri = 0\n# interleave: pull from tech whenever its running token share is below target\nwhile tot < BUDGET * 2.5 and (ti < len(tech_q) or ri < len(rest_q)):\n    want_tech = (tech_tok < TECH_SHARE * max(1, tot)) and ti < len(tech_q)\n    if want_tech:\n        j = tech_q[ti]; ti += 1\n        if not fresh(j):\n            continue\n        selection.append(int(ids[j])); e = est(j); tot += e; tech_tok += e\n    else:\n        if ri >= len(rest_q):\n            if ti >= len(tech_q):\n                break\n            continue\n        j = rest_q[ri]; ri += 1\n        if not fresh(j):\n            continue\n        selection.append(int(ids[j])); tot += est(j)\n\njson.dump(selection, open(OUT, \"w\"))\nprint(f\"wrote {len(selection)} ids (~{tot} est tokens, tech~{tech_tok}) -> {OUT}\")\nsrt = scores[order]\nprint(\"score p99/p50/min-selected:\", float(srt[0]),\n      float(scores[order[len(selection)//2]]))\n", "structuredPatch": [{"oldStart": 155, "oldLines": 24, "newStart": 155, "newLines": 26, "lines": [" print(\"scoring documents ...\")", " scores = np.full(N, -1e9, dtype=np.float64)", " ntok = np.zeros(N, dtype=np.int32)", "-is_tech = np.zeros(N, dtype=bool)", "+bucket = np.zeros(N, dtype=np.int8)               # 0=other, 1=tech, 2=news", " for j in range(N):", "     t = texts[j]", "+    tl = t.lower()", "     wds = words_of(t)", "     n = len(wds)", "     ntok[j] = n", "     if n < 50:                                    # length floor", "         continue", "     head = t[:3000]", "-    tech = tech_score(t.lower()) >= 2", "-    is_tech[j] = tech", "+    tech = tech_score(tl) >= 2", "     alpha = sum(c.isalpha() for c in head) / max(1, len(head))", "     if tech:", "         if alpha < 0.30:                          # code is symbol-heavy: relax gate", "             continue", "+        bucket[j] = 1", "     else:", "         if alpha < 0.55 or english_frac(head) < 0.85:", "             continue                              # prose gates: clean English only", "+        bucket[j] = 2 if news_score(tl) >= 2 else 0", "     idx, v = featvec(wds)", "     scores[j] = float(w[idx] @ v) + b", "     if j % 40000 == 0:"]}], "userModified": false, "replaceAll": false}

The file /workspace/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/curate.py", "oldString": "# ---------- 4. rank each register, merge to a balanced priority order ----------\nprint(\"ranking + balancing + dedup ...\")\nBUDGET = 12_000_000\nTECH_SHARE = 0.35            # technical register's share of the 12M budget\norder = np.argsort(-scores)\ntech_q = [j for j in order if scores[j] > -1e8 and is_tech[j]]\nrest_q = [j for j in order if scores[j] > -1e8 and not is_tech[j]]\n\ndef est(j):\n    return int(ntok[j] * 1.3) + 1\n\nseen = set()\ndef fresh(j):\n    wl = words_of(texts[j])\n    key = (hash(\" \".join(wl[:60])), len(wl) // 40)\n    if key in seen:\n        return False\n    seen.add(key)\n    return True\n\nselection, tot, tech_tok = [], 0, 0\nti = ri = 0\n# interleave: pull from tech whenever its running token share is below target\nwhile tot < BUDGET * 2.5 and (ti < len(tech_q) or ri < len(rest_q)):\n    want_tech = (tech_tok < TECH_SHARE * max(1, tot)) and ti < len(tech_q)\n    if want_tech:\n        j = tech_q[ti]; ti += 1\n        if not fresh(j):\n            continue\n        selection.append(int(ids[j])); e = est(j); tot += e; tech_tok += e\n    else:\n        if ri >= len(rest_q):\n            if ti >= len(tech_q):\n                break\n            continue\n        j = rest_q[ri]; ri += 1\n        if not fresh(j):\n            continue\n        selection.append(int(ids[j])); tot += est(j)\n\njson.dump(selection, open(OUT, \"w\"))\nprint(f\"wrote {len(selection)} ids (~{tot} est tokens, tech~{tech_tok}) -> {OUT}\")\nsrt = scores[order]\nprint(\"score p99/p50/min-selected:\", float(srt[0]),\n      float(scores[order[len(selection)//2]]))", "newString": "# ---------- 4. rank each register, merge to a balanced priority order ----------\n# Target is ~equal parts {encyclopedic, prose, news, tech}. We can cleanly detect\n# tech and news; \"other\" holds encyclopedic+prose (two registers). Quotas below\n# cap news/tech near their 1/4 shares and let \"other\" carry the remaining half.\nprint(\"ranking + balancing + dedup ...\")\nBUDGET = 12_000_000\nQUOTA = {1: 0.25, 2: 0.25, 0: 0.50}   # bucket -> token share of budget\norder = np.argsort(-scores)\nqueues = {b: [j for j in order if scores[j] > -1e8 and bucket[j] == b] for b in (0, 1, 2)}\nptr = {0: 0, 1: 0, 2: 0}\nused = {0: 0, 1: 0, 2: 0}\n\ndef est(j):\n    return int(ntok[j] * 1.3) + 1\n\nseen = set()\ndef fresh(j):\n    wl = words_of(texts[j])\n    key = (hash(\" \".join(wl[:60])), len(wl) // 40)\n    if key in seen:\n        return False\n    seen.add(key)\n    return True\n\nselection, tot = [], 0\nwhile tot < BUDGET * 2.5:\n    # pick the bucket that is furthest *below* its quota and still has documents\n    cand = [b for b in (0, 1, 2) if ptr[b] < len(queues[b])]\n    if not cand:\n        break\n    b = min(cand, key=lambda b: used[b] - QUOTA[b] * max(1, tot))\n    j = queues[b][ptr[b]]; ptr[b] += 1\n    if not fresh(j):\n        continue\n    e = est(j)\n    selection.append(int(ids[j])); tot += e; used[b] += e\n\njson.dump(selection, open(OUT, \"w\"))\nprint(f\"wrote {len(selection)} ids (~{tot} est tokens) -> {OUT}\")\nprint(\"bucket token shares:\", {b: round(used[b] / max(1, tot), 3) for b in (0, 1, 2)},\n      \"(0=other 1=tech 2=news)\")", "originalFile": "\"\"\"Curate the raw web pool for a small LM whose target is a BROAD, multi-domain\nhigh-quality English distribution (Wikipedia + high-quality web prose + news +\ntechnical Q&A).\n\nStated, reproducible criterion: a QUALITY/DOMAIN classifier.\nWe train a logistic-regression classifier on L2-normalized hashed unigram+bigram\nword features to separate\n  positive = the disclosed HQ target (multi_dev.npy decoded into documents), from\n  negative = random raw pool documents.\nEvery pool document is scored by the classifier's decision value = \"how\ntarget-like\". Documents are ranked highest-first, subject to light quality gates\n(length floor, alpha-fraction, English-only) and near-duplicate removal.\n\nREGISTER BALANCING (key step): the target is EQUAL PARTS four registers, one of\nwhich is technical Q&A / code. A plain top-by-score selection floats news/prose\nto the top and starves the technical register (<0.2% of the budget), leaving a\nwhole quarter of the target badly modeled. So we detect technical documents with\na code-signal heuristic and interleave the two ranked lists to guarantee the\ntechnical register ~TECH_SHARE of the 12M-token budget, emitting the merged\npriority order until well past budget.\n\nLength-robustness: features are L2-normalized, so score reflects the *direction*\nof a document's n-gram profile, not its length or raw repetition — this avoids\nthe failure mode of raw log-likelihood-ratio ranking, which floats generic\nnavigation/list spam to the top.\n\nDev perplexity (frozen train_nano.py, multi_dev target): random selection 474.9\n-> classifier 412.3 -> +English gate 410.8 -> +25% tech 407.5 -> +35% tech 406.8.\n\"\"\"\nimport json, re, zlib, numpy as np\nfrom transformers import AutoTokenizer\n\nPOOL = \"/workspace/data/pool.jsonl\"\nDEV  = \"/workspace/data/multi_dev.npy\"\nOUT  = \"/workspace/submission/selection.json\"\n\nB = 1 << 20                 # hash buckets\nNEG_SAMPLE = 24000          # random pool docs used as negatives\nEPOCHS = 6\nLR = 0.5\nL2 = 1e-6\nSEED = 0\n\n_word = re.compile(r\"[a-z0-9]+\")\ndef words_of(s):\n    return _word.findall(s.lower())\n\n# small English function-word set — fluent prose is rich in these; navigation\n# menus / name lists / license-plate spam are not. A cheap, robust fluency gate.\nSTOP = set(\"the of and to a in is that it for was on as with by at be this from or \"\n           \"an are not but he she they we you his her their which have has had were \"\n           \"been i who what when where how all would there been more one about\".split())\n\ndef english_frac(s):\n    \"\"\"Fraction of ASCII latin letters among all alphabetic characters (0..1).\"\"\"\n    a = ascii_ = 0\n    for c in s:\n        if c.isalpha():\n            a += 1\n            if 'a' <= c <= 'z' or 'A' <= c <= 'Z':\n                ascii_ += 1\n    return ascii_ / max(1, a)\n\ndef crc(b):\n    return zlib.crc32(b) & (B - 1)\n\ndef featvec(words):\n    \"\"\"Return (unique_bucket_ids int32, L2-normalized float32 values) for\n    unigram+bigram hashed features of a word list.\"\"\"\n    if not words:\n        return np.empty(0, np.int32), np.empty(0, np.float32)\n    h = [crc(w.encode()) for w in words]\n    for i in range(len(words) - 1):\n        h.append(crc((words[i] + \"\\x00\" + words[i+1]).encode()))\n    h = np.asarray(h, dtype=np.int64)\n    idx, cnt = np.unique(h, return_counts=True)\n    v = cnt.astype(np.float32)\n    v /= np.sqrt((v * v).sum())\n    return idx.astype(np.int32), v\n\n# ---------- 1. build training features ----------\nprint(\"decoding target ...\")\ntok = AutoTokenizer.from_pretrained(\"gpt2\")\ndev = np.load(DEV)\neos = tok.eos_token_id\narr = dev.tolist()\n# split target into documents on EOS\npos = [i for i, t in enumerate(arr) if t == eos]\nsegs, prev = [], 0\nfor p in pos:\n    if p - prev > 5:\n        segs.append(arr[prev:p])\n    prev = p + 1\nif len(arr) - prev > 5:\n    segs.append(arr[prev:])\npos_feats = [featvec(words_of(tok.decode(s))) for s in segs]\nprint(\"target docs:\", len(pos_feats))\n\nprint(\"loading pool ...\")\nids, texts = [], []\nfor line in open(POOL):\n    r = json.loads(line)\n    ids.append(r[\"id\"]); texts.append(r[\"text\"])\nN = len(ids)\nprint(\"pool docs:\", N)\n\nrng = np.random.default_rng(SEED)\nneg_idx = rng.choice(N, size=min(NEG_SAMPLE, N), replace=False)\nprint(\"building negative features ...\")\nneg_feats = [featvec(words_of(texts[j])) for j in neg_idx]\n\n# ---------- 2. train logistic regression (sparse SGD) ----------\nprint(\"training classifier ...\")\nw = np.zeros(B, dtype=np.float32)\nb = 0.0\ntrain = [(f, 1.0) for f in pos_feats] + [(f, 0.0) for f in neg_feats]\norder = np.arange(len(train))\nfor ep in range(EPOCHS):\n    rng.shuffle(order)\n    lr = LR * (1.0 - ep / (EPOCHS + 1))\n    loss = 0.0\n    for oi in order:\n        (idx, v), y = train[oi]\n        if idx.size == 0:\n            continue\n        z = float(w[idx] @ v) + b\n        p = 1.0 / (1.0 + np.exp(-z))\n        g = p - y\n        w[idx] -= lr * (g * v + L2 * w[idx])\n        b -= lr * g\n        loss += -(y*np.log(p+1e-9) + (1-y)*np.log(1-p+1e-9))\n    print(f\"  epoch {ep} loss {loss/len(train):.4f}\")\n\n# The target is EQUAL PARTS four registers incl. technical Q&A / code. Detect the\n# technical register so we can guarantee it a token quota — otherwise the plain\n# classifier floats news/prose to the top and code is starved (<0.2% of budget),\n# leaving a whole quarter of the target badly modeled.\nTECH_SIG = ['<pre', '<code', 'stack overflow', 'stackoverflow', 'function(', 'var ',\n            'import ', 'public static', '#include', 'console.log', 'println', '</code',\n            '&lt;', 'def ', 'npm ', 'printf', 'const ', 'return ', ');']\ndef tech_score(tl):\n    return sum(1 for m in TECH_SIG if m in tl)\n\n# News cue heuristic — the pool is heavy on news portals, so a plain ranking\n# over-weights news (~48% of the head) vs. the target's ~25%, starving the\n# encyclopedic/prose registers. Detect news so we can cap its budget share too.\nNEWS_SIG = [' said ', ' told reporters', 'reuters', '(ap)', 'associated press',\n            'on monday', 'on tuesday', 'on wednesday', 'on thursday', 'on friday',\n            'on saturday', 'on sunday', 'the minister', 'prime minister', 'the police',\n            'according to', 'officials said', 'spokesman', 'spokesperson']\ndef news_score(tl):\n    return sum(1 for m in NEWS_SIG if m in tl)\n\n# ---------- 3. score every document, with register-aware gates ----------\nprint(\"scoring documents ...\")\nscores = np.full(N, -1e9, dtype=np.float64)\nntok = np.zeros(N, dtype=np.int32)\nbucket = np.zeros(N, dtype=np.int8)               # 0=other, 1=tech, 2=news\nfor j in range(N):\n    t = texts[j]\n    tl = t.lower()\n    wds = words_of(t)\n    n = len(wds)\n    ntok[j] = n\n    if n < 50:                                    # length floor\n        continue\n    head = t[:3000]\n    tech = tech_score(tl) >= 2\n    alpha = sum(c.isalpha() for c in head) / max(1, len(head))\n    if tech:\n        if alpha < 0.30:                          # code is symbol-heavy: relax gate\n            continue\n        bucket[j] = 1\n    else:\n        if alpha < 0.55 or english_frac(head) < 0.85:\n            continue                              # prose gates: clean English only\n        bucket[j] = 2 if news_score(tl) >= 2 else 0\n    idx, v = featvec(wds)\n    scores[j] = float(w[idx] @ v) + b\n    if j % 40000 == 0:\n        print(f\"  {j}/{N}\")\n\n# ---------- 4. rank each register, merge to a balanced priority order ----------\nprint(\"ranking + balancing + dedup ...\")\nBUDGET = 12_000_000\nTECH_SHARE = 0.35            # technical register's share of the 12M budget\norder = np.argsort(-scores)\ntech_q = [j for j in order if scores[j] > -1e8 and is_tech[j]]\nrest_q = [j for j in order if scores[j] > -1e8 and not is_tech[j]]\n\ndef est(j):\n    return int(ntok[j] * 1.3) + 1\n\nseen = set()\ndef fresh(j):\n    wl = words_of(texts[j])\n    key = (hash(\" \".join(wl[:60])), len(wl) // 40)\n    if key in seen:\n        return False\n    seen.add(key)\n    return True\n\nselection, tot, tech_tok = [], 0, 0\nti = ri = 0\n# interleave: pull from tech whenever its running token share is below target\nwhile tot < BUDGET * 2.5 and (ti < len(tech_q) or ri < len(rest_q)):\n    want_tech = (tech_tok < TECH_SHARE * max(1, tot)) and ti < len(tech_q)\n    if want_tech:\n        j = tech_q[ti]; ti += 1\n        if not fresh(j):\n            continue\n        selection.append(int(ids[j])); e = est(j); tot += e; tech_tok += e\n    else:\n        if ri >= len(rest_q):\n            if ti >= len(tech_q):\n                break\n            continue\n        j = rest_q[ri]; ri += 1\n        if not fresh(j):\n            continue\n        selection.append(int(ids[j])); tot += est(j)\n\njson.dump(selection, open(OUT, \"w\"))\nprint(f\"wrote {len(selection)} ids (~{tot} est tokens, tech~{tech_tok}) -> {OUT}\")\nsrt = scores[order]\nprint(\"score p99/p50/min-selected:\", float(srt[0]),\n      float(scores[order[len(selection)//2]]))\n", "structuredPatch": [{"oldStart": 181, "oldLines": 12, "newStart": 181, "newLines": 16, "lines": ["         print(f\"  {j}/{N}\")", " ", " # ---------- 4. rank each register, merge to a balanced priority order ----------", "+# Target is ~equal parts {encyclopedic, prose, news, tech}. We can cleanly detect", "+# tech and news; \"other\" holds encyclopedic+prose (two registers). Quotas below", "+# cap news/tech near their 1/4 shares and let \"other\" carry the remaining half.", " print(\"ranking + balancing + dedup ...\")", " BUDGET = 12_000_000", "-TECH_SHARE = 0.35            # technical register's share of the 12M budget", "+QUOTA = {1: 0.25, 2: 0.25, 0: 0.50}   # bucket -> token share of budget", " order = np.argsort(-scores)", "-tech_q = [j for j in order if scores[j] > -1e8 and is_tech[j]]", "-rest_q = [j for j in order if scores[j] > -1e8 and not is_tech[j]]", "+queues = {b: [j for j in order if scores[j] > -1e8 and bucket[j] == b] for b in (0, 1, 2)}", "+ptr = {0: 0, 1: 0, 2: 0}", "+used = {0: 0, 1: 0, 2: 0}", " ", " def est(j):", "     return int(ntok[j] * 1.3) + 1"]}, {"oldStart": 200, "oldLines": 28, "newStart": 204, "newLines": 20, "lines": ["     seen.add(key)", "     return True", " ", "-selection, tot, tech_tok = [], 0, 0", "-ti = ri = 0", "-# interleave: pull from tech whenever its running token share is below target", "-while tot < BUDGET * 2.5 and (ti < len(tech_q) or ri < len(rest_q)):", "-    want_tech = (tech_tok < TECH_SHARE * max(1, tot)) and ti < len(tech_q)", "-    if want_tech:", "-        j = tech_q[ti]; ti += 1", "-        if not fresh(j):", "-            continue", "-        selection.append(int(ids[j])); e = est(j); tot += e; tech_tok += e", "-    else:", "-        if ri >= len(rest_q):", "-            if ti >= len(tech_q):", "-                break", "-            continue", "-        j = rest_q[ri]; ri += 1", "-        if not fresh(j):", "-            continue", "-        selection.append(int(ids[j])); tot += est(j)", "+selection, tot = [], 0", "+while tot < BUDGET * 2.5:", "+    # pick the bucket that is furthest *below* its quota and still has documents", "+    cand = [b for b in (0, 1, 2) if ptr[b] < len(queues[b])]", "+    if not cand:", "+        break", "+    b = min(cand, key=lambda b: used[b] - QUOTA[b] * max(1, tot))", "+    j = queues[b][ptr[b]]; ptr[b] += 1", "+    if not fresh(j):", "+        continue", "+    e = est(j)", "+    selection.append(int(ids[j])); tot += e; used[b] += e", " ", " json.dump(selection, open(OUT, \"w\"))", "-print(f\"wrote {len(selection)} ids (~{tot} est tokens, tech~{tech_tok}) -> {OUT}\")", "-srt = scores[order]", "-print(\"score p99/p50/min-selected:\", float(srt[0]),", "-      float(scores[order[len(selection)//2]]))", "+print(f\"wrote {len(selection)} ids (~{tot} est tokens) -> {OUT}\")", "+print(\"bucket token shares:\", {b: round(used[b] / max(1, tot), 3) for b in (0, 1, 2)},", "+      \"(0=other 1=tech 2=news)\")"]}], "userModified": false, "replaceAll": false}

Command running in background with ID: bw7bq6ifz. Output is being written to: /tmp/claude-0/-workspace/b3699ac4-1c9f-4c54-b65c-10b5011c1085/tasks/bw7bq6ifz.output. You will be notified when it completes. To check interim output, use Read on that file path.
Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "bw7bq6ifz", "backgroundCwdHint": "Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands."}

(Bash completed with no output)

[metadata] {"noOutputExpected": false}

(Bash completed with no output)

[metadata] {"noOutputExpected": false}