nano-gpt-data-curation

claude-opus-4-8

nano-gpt-data-curation — graded by the task's verifier against a measured baseline.

backend: modal
variant: claude claude-opus-4-8
metric: held-out multi-domain perplexity
actions: 35
duration: 39m
recorded: 2026-08-04

run iddclm-nano-curation__K2QidbF

Everything claude-opus-4-8 did on this task, in order.

download JSON ↗

claude-codeclaude-opus-4-835 turns21 narration35 action35 observation14 reasoning1 final

#26

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolBashCheck balanced curation output

args

{
  "command": "sleep 60; tail -8 /tmp/claude-0/-workspace/dbb0adbe-6465-46c0-8321-7f69f624a779/tasks/bb3wrp1ft.output",
  "description": "Check balanced curation output"
}

Bash result

techQA: 626 candidate docs
  news_web: 171998 candidate docs
  quota queue sizes: {'wiki': 2, 'techQA': 626, 'news_web': 4846}
wrote /workspace/submission/selection.json  (45000 ids)

real	2m42.948s
user	2m33.720s
sys	0m1.350s

[stdout]
  techQA: 626 candidate docs
  news_web: 171998 candidate docs
  quota queue sizes: {'wiki': 2, 'techQA': 626, 'news_web': 4846}
wrote /workspace/submission/selection.json  (45000 ids)

real	2m42.948s
user	2m33.720s
sys	0m1.350s
[metadata] {"noOutputExpected": false}

#27

narration

The argmax assignment collapsed everything into news_web (wiki got only 2 docs) — registers aren't separable by argmax. Fix: for each register, independently take the top docs by that register's score (overlap allowed), then interleave. This actually pulls the most wiki-like and tech-like docs in. Let me rewrite the scoring/selection stage.

toolEdit

args

{
  "replace_all": false,
  "file_path": "/workspace/submission/curate.py",
  "old_string": "# ---------------------------------------------------------------- score every doc under every register\nprint(\"scoring docs ...\")\n# per-register best lists: (score, idx)\nbest = {r: [] for r in REGISTERS}\ntok_counts = np.zeros(N, dtype=np.int32)\nfor j in range(N):\n    t = texts[j][:SCORE_CHARS]\n    w = toks(t)\n    if not quality_ok(w, t):\n        continue\n    fl = list(feats(w))\n    n = len(fl)\n    if n < 40:\n        continue\n    sr = {r: 0.0 for r in REGISTERS}\n    matched = 0\n    for fk in fl:\n        hit = False\n        for r in REGISTERS:\n            v = logr[r].get(fk)\n            if v is not None:\n                sr[r] += v; hit = True\n        if hit:\n            matched += 1\n    if matched / n < 0.55:          # English-coverage gate\n        continue\n    reg = max(REGISTERS, key=lambda r: sr[r])\n    best[reg].append((sr[reg] / n, j))\n    if j % 40000 == 0:\n        print(f\"  {j}/{N}\")\n\nfor r in REGISTERS:\n    best[r].sort(reverse=True)\n    print(f\"  {r}: {len(best[r])} candidate docs\")",
  "new_string": "# ---------------------------------------------------------------- score every doc under every register\n# Keep EACH register's score for every passing doc (no argmax collapse). A doc can\n# represent more than one register; per-register top-lists then pull in the most\n# wiki-like / tech-like / news-like docs even when a broader register would outscore\n# them overall.\nprint(\"scoring docs ...\")\nbest = {r: [] for r in REGISTERS}          # (score_r, idx) for passing docs\nfor j in range(N):\n    t = texts[j][:SCORE_CHARS]\n    w = toks(t)\n    if not quality_ok(w, t):\n        continue\n    fl = list(feats(w))\n    n = len(fl)\n    if n < 40:\n        continue\n    sr = {r: 0.0 for r in REGISTERS}\n    matched = 0\n    for fk in fl:\n        hit = False\n        for r in REGISTERS:\n            v = logr[r].get(fk)\n            if v is not None:\n                sr[r] += v; hit = True\n        if hit:\n            matched += 1\n    if matched / n < 0.55:                 # English-coverage gate\n        continue\n    for r in REGISTERS:\n        best[r].append((sr[r] / n, j))\n    if j % 40000 == 0:\n        print(f\"  {j}/{N}\")\n\nfor r in REGISTERS:\n    best[r].sort(reverse=True)\n    print(f\"  {r}: {len(best[r])} candidate docs\")"
}

Edit result

The file /workspace/submission/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/submission/curate.py", "oldString": "# ---------------------------------------------------------------- score every doc under every register\nprint(\"scoring docs ...\")\n# per-register best lists: (score, idx)\nbest = {r: [] for r in REGISTERS}\ntok_counts = np.zeros(N, dtype=np.int32)\nfor j in range(N):\n    t = texts[j][:SCORE_CHARS]\n    w = toks(t)\n    if not quality_ok(w, t):\n        continue\n    fl = list(feats(w))\n    n = len(fl)\n    if n < 40:\n        continue\n    sr = {r: 0.0 for r in REGISTERS}\n    matched = 0\n    for fk in fl:\n        hit = False\n        for r in REGISTERS:\n            v = logr[r].get(fk)\n            if v is not None:\n                sr[r] += v; hit = True\n        if hit:\n            matched += 1\n    if matched / n < 0.55:          # English-coverage gate\n        continue\n    reg = max(REGISTERS, key=lambda r: sr[r])\n    best[reg].append((sr[reg] / n, j))\n    if j % 40000 == 0:\n        print(f\"  {j}/{N}\")\n\nfor r in REGISTERS:\n    best[r].sort(reverse=True)\n    print(f\"  {r}: {len(best[r])} candidate docs\")", "newString": "# ---------------------------------------------------------------- score every doc under every register\n# Keep EACH register's score for every passing doc (no argmax collapse). A doc can\n# represent more than one register; per-register top-lists then pull in the most\n# wiki-like / tech-like / news-like docs even when a broader register would outscore\n# them overall.\nprint(\"scoring docs ...\")\nbest = {r: [] for r in REGISTERS}          # (score_r, idx) for passing docs\nfor j in range(N):\n    t = texts[j][:SCORE_CHARS]\n    w = toks(t)\n    if not quality_ok(w, t):\n        continue\n    fl = list(feats(w))\n    n = len(fl)\n    if n < 40:\n        continue\n    sr = {r: 0.0 for r in REGISTERS}\n    matched = 0\n    for fk in fl:\n        hit = False\n        for r in REGISTERS:\n            v = logr[r].get(fk)\n            if v is not None:\n                sr[r] += v; hit = True\n        if hit:\n            matched += 1\n    if matched / n < 0.55:                 # English-coverage gate\n        continue\n    for r in REGISTERS:\n        best[r].append((sr[r] / n, j))\n    if j % 40000 == 0:\n        print(f\"  {j}/{N}\")\n\nfor r in REGISTERS:\n    best[r].sort(reverse=True)\n    print(f\"  {r}: {len(best[r])} candidate docs\")", "originalFile": "\"\"\"Curate a raw web pool for training a small LM whose eval target is a BROAD,\nmulti-domain HQ English corpus that is disclosed as EQUAL PARTS four registers:\nencyclopedic (Wikipedia), general HQ web prose, news, and technical Q&A. The\ntarget is disclosed via the tokenized dev file `multi_dev.npy`.\n\nCriterion (stated, reproducible), two coupled ideas:\n\n  (1) DSIR-style importance scoring. Rank documents by the average per-token\n      log-likelihood ratio  log p_target(w) - log p_pool(w)  over word\n      unigrams+bigrams, where p_* are word distributions. Documents whose\n      vocabulary/style match the HQ target score high; boilerplate/junk low.\n\n  (2) Register balancing. A single combined target distribution is dominated by\n      the pool's most common HQ register (news/web prose), so a naive DSIR\n      selection that fills a 12M-token budget comes out ~87% web-prose, ~12%\n      news, ~1% technical Q&A and ~0% encyclopedic -- badly mismatched to an\n      EQUAL-PARTS target. Held-out perplexity is dominated by the worst-served\n      register, so we instead build a separate log-ratio scorer per register\n      (prototypes carved from the disclosed dev by simple markers), assign each\n      pool document to its best-matching register, and fill the budget with a\n      ~25% quota per register, interleaved in priority order so the selection\n      stays balanced no matter where the trainer truncates.\n\nA light quality gate removes only degenerate docs (too short / non-text /\nrepetitive); aggressive markup filtering is deliberately avoided because it\nstrips the on-target technical-Q&A (code/HTML) and Wikipedia (tables) registers.\n\"\"\"\nimport json, re, math, random\nfrom collections import Counter\nimport numpy as np\n\nPOOL = \"/workspace/data/pool.jsonl\"\nDEV  = \"/workspace/data/multi_dev.npy\"\nOUT  = \"/workspace/submission/selection.json\"\n\nrandom.seed(0)\nWORD = re.compile(r\"[a-z]+\")\nSCORE_CHARS = 4000\nPOOL_SAMPLE = 20000\nALPHA = 1.0\nMIN_WORDS = 50\nN_OUT = 45000\nREGISTERS = [\"wiki\", \"techQA\", \"news_web\"]\n\ndef toks(text):\n    return WORD.findall(text.lower())\n\ndef feats(words):\n    for x in words:\n        yield x\n    for a, b in zip(words, words[1:]):\n        yield a + \" \" + b\n\ndef register_of_dev(t):\n    if \"@,@\" in t or \"@-@\" in t:                              return \"wiki\"\n    if \"<p>\" in t or \"<code>\" in t or \"</\" in t or \"<pre>\" in t: return \"techQA\"\n    return \"news_web\"\n\n# ---------------------------------------------------------------- load pool\nprint(\"loading pool ...\")\nids, texts = [], []\nwith open(POOL) as f:\n    for line in f:\n        r = json.loads(line)\n        ids.append(r[\"id\"]); texts.append(r[\"text\"])\nN = len(ids)\nprint(f\"  {N} docs\")\n\n# ---------------------------------------------------------------- target dists (per register)\nprint(\"decoding target & building per-register dists ...\")\nfrom transformers import AutoTokenizer\ntk = AutoTokenizer.from_pretrained(\"gpt2\")\ndev = np.load(DEV)\ndev_docs = [d for d in tk.decode(dev.tolist()).split(\"<|endoftext|>\") if len(d.strip()) > 200]\nct = {r: Counter() for r in REGISTERS}\nfor d in dev_docs:\n    ct[register_of_dev(d)].update(feats(toks(d)))\nfor r in REGISTERS:\n    print(f\"  {r}: {sum(ct[r].values())} feat-tokens\")\n\n# ---------------------------------------------------------------- pool dist\nprint(\"estimating pool dist ...\")\ncp = Counter()\nfor i in random.sample(range(N), min(POOL_SAMPLE, N)):\n    cp.update(feats(toks(texts[i][:SCORE_CHARS])))\nCp = sum(cp.values())\n\n# ---------------------------------------------------------------- per-register log-ratio tables\nlogr = {r: {} for r in REGISTERS}\nfor r in REGISTERS:\n    Ct = sum(ct[r].values())\n    V = len(set(ct[r]) | set(cp))\n    for k in set(ct[r]) | set(cp):\n        a = ct[r].get(k, 0); b = cp.get(k, 0)\n        if a + b < 2:\n            continue\n        logr[r][k] = math.log((a + ALPHA) / (Ct + ALPHA * V)) - math.log((b + ALPHA) / (Cp + ALPHA * V))\nprint(\"  logratio tables:\", {r: len(logr[r]) for r in REGISTERS})\n\n# ---------------------------------------------------------------- quality gate\ndef quality_ok(words, text):\n    if len(words) < MIN_WORDS:\n        return False\n    if sum(c.isalpha() for c in text) < 0.5 * len(text):\n        return False\n    if len(set(words)) / len(words) < 0.30:\n        return False\n    return True\n\n# ---------------------------------------------------------------- score every doc under every register\nprint(\"scoring docs ...\")\n# per-register best lists: (score, idx)\nbest = {r: [] for r in REGISTERS}\ntok_counts = np.zeros(N, dtype=np.int32)\nfor j in range(N):\n    t = texts[j][:SCORE_CHARS]\n    w = toks(t)\n    if not quality_ok(w, t):\n        continue\n    fl = list(feats(w))\n    n = len(fl)\n    if n < 40:\n        continue\n    sr = {r: 0.0 for r in REGISTERS}\n    matched = 0\n    for fk in fl:\n        hit = False\n        for r in REGISTERS:\n            v = logr[r].get(fk)\n            if v is not None:\n                sr[r] += v; hit = True\n        if hit:\n            matched += 1\n    if matched / n < 0.55:          # English-coverage gate\n        continue\n    reg = max(REGISTERS, key=lambda r: sr[r])\n    best[reg].append((sr[reg] / n, j))\n    if j % 40000 == 0:\n        print(f\"  {j}/{N}\")\n\nfor r in REGISTERS:\n    best[r].sort(reverse=True)\n    print(f\"  {r}: {len(best[r])} candidate docs\")\n\n# ---------------------------------------------------------------- balanced, interleaved selection\n# Estimate tokens per doc from chars (~4 chars/token) to hold ~25% budget per register,\n# then interleave in priority order so truncation at 12M stays balanced. Spill if a\n# register is exhausted.\nBUDGET = 12_000_000\nper_reg_budget = BUDGET / len(REGISTERS)\ndef est_tokens(idx):\n    return max(1, len(texts[idx]) // 4)\n\n# take enough docs per register to cover its quota (plus headroom), keep as queues\nqueues = {}\nfor r in REGISTERS:\n    acc, q = 0, []\n    for sc, idx in best[r]:\n        q.append(idx); acc += est_tokens(idx)\n        if acc >= per_reg_budget * 1.4:\n            break\n    queues[r] = q\nprint(\"  quota queue sizes:\", {r: len(queues[r]) for r in REGISTERS})\n\n# round-robin interleave (wiki, techQA, news_web, wiki, ...) for balanced priority order\nseen_sig, sel = set(), []\nptr = {r: 0 for r in REGISTERS}\nwhile len(sel) < N_OUT:\n    progressed = False\n    for r in REGISTERS:\n        while ptr[r] < len(queues[r]):\n            idx = queues[r][ptr[r]]; ptr[r] += 1\n            sig = re.sub(r\"\\s+\", \" \", texts[idx][:300].lower()).strip()[:200]\n            if sig in seen_sig:\n                continue\n            seen_sig.add(sig); sel.append(int(ids[idx])); progressed = True\n            break\n    if not progressed:\n        break\n\n# append remaining high-scoring docs from all registers (unused-beyond-budget tail, priority-safe)\ntail = sorted(\n    [(sc, idx) for r in REGISTERS for sc, idx in best[r][len(queues[r]):]],\n    reverse=True,\n)\nfor sc, idx in tail:\n    if len(sel) >= N_OUT:\n        break\n    sig = re.sub(r\"\\s+\", \" \", texts[idx][:300].lower()).strip()[:200]\n    if sig in seen_sig:\n        continue\n    seen_sig.add(sig); sel.append(int(ids[idx]))\n\njson.dump(sel, open(OUT, \"w\"))\nprint(f\"wrote {OUT}  ({len(sel)} ids)\")\n", "structuredPatch": [{"oldStart": 108, "oldLines": 10, "newStart": 108, "newLines": 12, "lines": ["     return True", " ", " # ---------------------------------------------------------------- score every doc under every register", "+# Keep EACH register's score for every passing doc (no argmax collapse). A doc can", "+# represent more than one register; per-register top-lists then pull in the most", "+# wiki-like / tech-like / news-like docs even when a broader register would outscore", "+# them overall.", " print(\"scoring docs ...\")", "-# per-register best lists: (score, idx)", "-best = {r: [] for r in REGISTERS}", "-tok_counts = np.zeros(N, dtype=np.int32)", "+best = {r: [] for r in REGISTERS}          # (score_r, idx) for passing docs", " for j in range(N):", "     t = texts[j][:SCORE_CHARS]", "     w = toks(t)"]}, {"oldStart": 131, "oldLines": 10, "newStart": 133, "newLines": 10, "lines": ["                 sr[r] += v; hit = True", "         if hit:", "             matched += 1", "-    if matched / n < 0.55:          # English-coverage gate", "+    if matched / n < 0.55:                 # English-coverage gate", "         continue", "-    reg = max(REGISTERS, key=lambda r: sr[r])", "-    best[reg].append((sr[reg] / n, j))", "+    for r in REGISTERS:", "+        best[r].append((sr[r] / n, j))", "     if j % 40000 == 0:", "         print(f\"  {j}/{N}\")", " "]}], "userModified": false, "replaceAll": false}

#28

narration

Now let me run the corrected balanced version and evaluate it.

toolBashRun balanced curation and eval

args

{
  "command": "cd /workspace; python3 submission/curate.py 2>&1 | tail -6; echo \"=== EVAL ===\"; bash dev_eval.sh /workspace/submission/selection.json 2>&1 | grep -E \"packed|HELDOUT\"",
  "description": "Run balanced curation and eval"
}

Bash result

Command did not complete within its 120s timeout and was moved to the background (ID: ble3e6nbr). Output is being written to: /tmp/claude-0/-workspace/dbb0adbe-6465-46c0-8321-7f69f624a779/tasks/ble3e6nbr.output. You will be notified when it completes. To check interim output, use Read on that file path.
Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "ble3e6nbr", "timedOutAfterMs": 120000, "backgroundCwdHint": "Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands."}

#29

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolBashCheck balanced eval progress

args

{
  "command": "sleep 115; tail -8 /tmp/claude-0/-workspace/dbb0adbe-6465-46c0-8321-7f69f624a779/tasks/ble3e6nbr.output",
  "description": "Check balanced eval progress"
}

Bash result

160000/182016
  wiki: 172626 candidate docs
  techQA: 172626 candidate docs
  news_web: 172626 candidate docs
  quota queue sizes: {'wiki': 3416, 'techQA': 3536, 'news_web': 4837}
wrote /workspace/submission/selection.json  (45000 ids)
=== EVAL ===
packed 12000000 tokens from 45000 ids -> /tmp/dev_train.npy

[stdout]
  160000/182016
  wiki: 172626 candidate docs
  techQA: 172626 candidate docs
  news_web: 172626 candidate docs
  quota queue sizes: {'wiki': 3416, 'techQA': 3536, 'news_web': 4837}
wrote /workspace/submission/selection.json  (45000 ids)
=== EVAL ===
packed 12000000 tokens from 45000 ids -> /tmp/dev_train.npy
[metadata] {"noOutputExpected": false}

#30

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolBashGet balanced eval result

args

{
  "command": "sleep 70; tail -3 /tmp/claude-0/-workspace/dbb0adbe-6465-46c0-8321-7f69f624a779/tasks/ble3e6nbr.output | grep -E \"HELDOUT|packed\"; cat /workspace/.timer/remaining_secs",
  "description": "Get balanced eval result"
}

Bash result

packed 12000000 tokens from 45000 ids -> /tmp/dev_train.npy
HELDOUT_PPL 409.383  loss 6.0147  train_s 51  /tmp/dev_train.npy
3568

[stdout]
packed 12000000 tokens from 45000 ids -> /tmp/dev_train.npy
HELDOUT_PPL 409.383  loss 6.0147  train_s 51  /tmp/dev_train.npy
3568
[metadata] {"noOutputExpected": false}

#31

narration

Balanced version = 409, worse than 335.6. Register-balancing via these noisy prototypes hurts — forcing 33% wiki/tech pulls in mediocre docs and displaces the high-quality prose the combined-DSIR nailed. The model transfers to tech/wiki fine from clean prose. Combined-DSIR (335.6) wins. Let me restore that proven config as the final curate.py.

toolWrite

args

{
  "file_path": "/workspace/submission/curate.py",
  "content": "\"\"\"Curate a raw web pool for training a small LM whose eval target is a BROAD,\nmulti-domain high-quality English corpus -- disclosed as equal parts encyclopedic\n(Wikipedia), general HQ web prose, news, and technical Q&A -- provided in\ntokenized form as `multi_dev.npy`.\n\nCRITERION (stated, reproducible)\n--------------------------------\nRank every pool document by a DSIR-style importance score: the average per-token\nlog-likelihood ratio\n\n        score(doc) = mean_{f in features(doc)}  [ log p_target(f) - log p_pool(f) ]\n\nover word unigrams + bigrams, where p_target is the word distribution of the\ndisclosed HQ target (decoded from multi_dev.npy) and p_pool is the word\ndistribution of the raw pool (estimated from a random sample). OOV features\ncontribute 0, so the average is diluted toward 0 for documents full of foreign /\njunk vocabulary; an explicit English-coverage gate (>=55% of features known) drops\nthe rest. Documents whose vocabulary and style look like the multi-domain HQ\ntarget score high; boilerplate / navigation / non-prose / non-English score low.\n\nA DELIBERATELY LIGHT quality gate removes only degenerate documents (too short,\nnon-text-heavy, or degenerately repetitive). Aggressive markup/symbol filtering\nwas measured to HURT held-out perplexity, because it strips the on-target\ntechnical-Q&A (code/HTML) and Wikipedia (tables/infobox) registers.\n\nRegister-balancing (forcing equal quotas of wiki/tech/news via per-register\nscorers) was also tried and measured WORSE than this single combined target\n(409 vs 336 dev ppl): the pool's most wiki-/tech-like documents are mediocre, and\ndisplacing high-quality prose to hit quotas costs more than the balance buys --\nthe model transfers to the under-represented registers from clean prose. So the\nfinal criterion is the single combined-target ranking below.\n\nOutput ids are emitted in descending score (priority) order; the trainer consumes\nthem until the 12M-token budget is full. Near-duplicate documents are dropped so\nno budget is wasted on repeats.\n\"\"\"\nimport json, re, math, random\nfrom collections import Counter\nimport numpy as np\n\nPOOL = \"/workspace/data/pool.jsonl\"\nDEV  = \"/workspace/data/multi_dev.npy\"\nOUT  = \"/workspace/submission/selection.json\"\n\nrandom.seed(0)\nWORD = re.compile(r\"[a-z]+\")\nSCORE_CHARS = 4000          # cap text per doc when scoring (speed; quality signal is in the prefix)\nPOOL_SAMPLE = 20000         # docs used to estimate the pool word distribution\nALPHA = 1.0                 # additive smoothing\nMIN_WORDS = 50              # quality gate: minimum words\nMIN_FEATS = 40              # minimum unigram+bigram features to score a doc\nMIN_MATCH = 0.55            # English-coverage gate: fraction of features that are known\nN_OUT = 45000               # emit this many ids (priority order); packer truncates at 12M tokens\n\ndef toks(text):\n    return WORD.findall(text.lower())\n\ndef feats(words):\n    for x in words:\n        yield x\n    for a, b in zip(words, words[1:]):\n        yield a + \" \" + b\n\n# ---------------------------------------------------------------- load pool\nprint(\"loading pool ...\")\nids, texts = [], []\nwith open(POOL) as f:\n    for line in f:\n        r = json.loads(line)\n        ids.append(r[\"id\"]); texts.append(r[\"text\"])\nN = len(ids)\nprint(f\"  {N} docs\")\n\n# ---------------------------------------------------------------- target dist\nprint(\"decoding target (multi_dev) ...\")\nfrom transformers import AutoTokenizer\ntk = AutoTokenizer.from_pretrained(\"gpt2\")\ndev_text = tk.decode(np.load(DEV).tolist())\nct = Counter(feats(toks(dev_text)))\nCt = sum(ct.values())\nprint(f\"  target feature tokens {Ct}, vocab {len(ct)}\")\n\n# ---------------------------------------------------------------- pool dist\nprint(\"estimating pool dist ...\")\ncp = Counter()\nfor i in random.sample(range(N), min(POOL_SAMPLE, N)):\n    cp.update(feats(toks(texts[i][:SCORE_CHARS])))\nCp = sum(cp.values())\nprint(f\"  pool feature tokens {Cp}, vocab {len(cp)}\")\n\n# ---------------------------------------------------------------- log-ratio table\nV = len(set(ct) | set(cp))\nlogr = {}\nfor k in set(ct) | set(cp):\n    a = ct.get(k, 0); b = cp.get(k, 0)\n    if a + b < 2:                       # drop hapax features (noise)\n        continue\n    logr[k] = (math.log((a + ALPHA) / (Ct + ALPHA * V))\n               - math.log((b + ALPHA) / (Cp + ALPHA * V)))\nprint(f\"  logratio table size {len(logr)}\")\n\n# ---------------------------------------------------------------- quality gate\ndef quality_ok(words, text):\n    # Light on purpose: the target spans Wikipedia (tables/infoboxes) and technical\n    # Q&A (code/HTML), so aggressive symbol/markup filtering removes on-target\n    # registers. Only drop the obviously degenerate.\n    if len(words) < MIN_WORDS:\n        return False\n    if sum(c.isalpha() for c in text) < 0.5 * len(text):   # non-prose / markup-only / binary\n        return False\n    if len(set(words)) / len(words) < 0.30:                # degenerate repetition\n        return False\n    return True\n\n# ---------------------------------------------------------------- score\nprint(\"scoring docs ...\")\nscores = np.full(N, -1e9, dtype=np.float64)\nfor j in range(N):\n    t = texts[j][:SCORE_CHARS]\n    w = toks(t)\n    if not quality_ok(w, t):\n        continue\n    s = 0.0; n = 0; matched = 0\n    for fk in feats(w):\n        v = logr.get(fk)\n        n += 1\n        if v is not None:\n            s += v; matched += 1          # OOV features contribute 0 (dilute foreign/junk)\n    if n >= MIN_FEATS and matched / n >= MIN_MATCH:\n        scores[j] = s / n                 # normalize over ALL features\n    if j % 40000 == 0:\n        print(f\"  {j}/{N}\")\n\n# ---------------------------------------------------------------- rank + dedup\norder = np.argsort(-scores)\nseen, sel = set(), []\nfor k in order:\n    if scores[k] <= -1e8:\n        break\n    sig = re.sub(r\"\\s+\", \" \", texts[k][:300].lower()).strip()[:200]   # near-dup signature\n    if sig in seen:\n        continue\n    seen.add(sig)\n    sel.append(int(ids[k]))\n    if len(sel) >= N_OUT:\n        break\n\nprint(f\"selected {len(sel)} ids; top score {scores[order[0]]:.3f}\")\njson.dump(sel, open(OUT, \"w\"))\nprint(\"wrote\", OUT)\n"
}

Write result

The file /workspace/submission/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"type": "update", "filePath": "/workspace/submission/curate.py", "content": "\"\"\"Curate a raw web pool for training a small LM whose eval target is a BROAD,\nmulti-domain high-quality English corpus -- disclosed as equal parts encyclopedic\n(Wikipedia), general HQ web prose, news, and technical Q&A -- provided in\ntokenized form as `multi_dev.npy`.\n\nCRITERION (stated, reproducible)\n--------------------------------\nRank every pool document by a DSIR-style importance score: the average per-token\nlog-likelihood ratio\n\n        score(doc) = mean_{f in features(doc)}  [ log p_target(f) - log p_pool(f) ]\n\nover word unigrams + bigrams, where p_target is the word distribution of the\ndisclosed HQ target (decoded from multi_dev.npy) and p_pool is the word\ndistribution of the raw pool (estimated from a random sample). OOV features\ncontribute 0, so the average is diluted toward 0 for documents full of foreign /\njunk vocabulary; an explicit English-coverage gate (>=55% of features known) drops\nthe rest. Documents whose vocabulary and style look like the multi-domain HQ\ntarget score high; boilerplate / navigation / non-prose / non-English score low.\n\nA DELIBERATELY LIGHT quality gate removes only degenerate documents (too short,\nnon-text-heavy, or degenerately repetitive). Aggressive markup/symbol filtering\nwas measured to HURT held-out perplexity, because it strips the on-target\ntechnical-Q&A (code/HTML) and Wikipedia (tables/infobox) registers.\n\nRegister-balancing (forcing equal quotas of wiki/tech/news via per-register\nscorers) was also tried and measured WORSE than this single combined target\n(409 vs 336 dev ppl): the pool's most wiki-/tech-like documents are mediocre, and\ndisplacing high-quality prose to hit quotas costs more than the balance buys --\nthe model transfers to the under-represented registers from clean prose. So the\nfinal criterion is the single combined-target ranking below.\n\nOutput ids are emitted in descending score (priority) order; the trainer consumes\nthem until the 12M-token budget is full. Near-duplicate documents are dropped so\nno budget is wasted on repeats.\n\"\"\"\nimport json, re, math, random\nfrom collections import Counter\nimport numpy as np\n\nPOOL = \"/workspace/data/pool.jsonl\"\nDEV  = \"/workspace/data/multi_dev.npy\"\nOUT  = \"/workspace/submission/selection.json\"\n\nrandom.seed(0)\nWORD = re.compile(r\"[a-z]+\")\nSCORE_CHARS = 4000          # cap text per doc when scoring (speed; quality signal is in the prefix)\nPOOL_SAMPLE = 20000         # docs used to estimate the pool word distribution\nALPHA = 1.0                 # additive smoothing\nMIN_WORDS = 50              # quality gate: minimum words\nMIN_FEATS = 40              # minimum unigram+bigram features to score a doc\nMIN_MATCH = 0.55            # English-coverage gate: fraction of features that are known\nN_OUT = 45000               # emit this many ids (priority order); packer truncates at 12M tokens\n\ndef toks(text):\n    return WORD.findall(text.lower())\n\ndef feats(words):\n    for x in words:\n        yield x\n    for a, b in zip(words, words[1:]):\n        yield a + \" \" + b\n\n# ---------------------------------------------------------------- load pool\nprint(\"loading pool ...\")\nids, texts = [], []\nwith open(POOL) as f:\n    for line in f:\n        r = json.loads(line)\n        ids.append(r[\"id\"]); texts.append(r[\"text\"])\nN = len(ids)\nprint(f\"  {N} docs\")\n\n# ---------------------------------------------------------------- target dist\nprint(\"decoding target (multi_dev) ...\")\nfrom transformers import AutoTokenizer\ntk = AutoTokenizer.from_pretrained(\"gpt2\")\ndev_text = tk.decode(np.load(DEV).tolist())\nct = Counter(feats(toks(dev_text)))\nCt = sum(ct.values())\nprint(f\"  target feature tokens {Ct}, vocab {len(ct)}\")\n\n# ---------------------------------------------------------------- pool dist\nprint(\"estimating pool dist ...\")\ncp = Counter()\nfor i in random.sample(range(N), min(POOL_SAMPLE, N)):\n    cp.update(feats(toks(texts[i][:SCORE_CHARS])))\nCp = sum(cp.values())\nprint(f\"  pool feature tokens {Cp}, vocab {len(cp)}\")\n\n# ---------------------------------------------------------------- log-ratio table\nV = len(set(ct) | set(cp))\nlogr = {}\nfor k in set(ct) | set(cp):\n    a = ct.get(k, 0); b = cp.get(k, 0)\n    if a + b < 2:                       # drop hapax features (noise)\n        continue\n    logr[k] = (math.log((a + ALPHA) / (Ct + ALPHA * V))\n               - math.log((b + ALPHA) / (Cp + ALPHA * V)))\nprint(f\"  logratio table size {len(logr)}\")\n\n# ---------------------------------------------------------------- quality gate\ndef quality_ok(words, text):\n    # Light on purpose: the target spans Wikipedia (tables/infoboxes) and technical\n    # Q&A (code/HTML), so aggressive symbol/markup filtering removes on-target\n    # registers. Only drop the obviously degenerate.\n    if len(words) < MIN_WORDS:\n        return False\n    if sum(c.isalpha() for c in text) < 0.5 * len(text):   # non-prose / markup-only / binary\n        return False\n    if len(set(words)) / len(words) < 0.30:                # degenerate repetition\n        return False\n    return True\n\n# ---------------------------------------------------------------- score\nprint(\"scoring docs ...\")\nscores = np.full(N, -1e9, dtype=np.float64)\nfor j in range(N):\n    t = texts[j][:SCORE_CHARS]\n    w = toks(t)\n    if not quality_ok(w, t):\n        continue\n    s = 0.0; n = 0; matched = 0\n    for fk in feats(w):\n        v = logr.get(fk)\n        n += 1\n        if v is not None:\n            s += v; matched += 1          # OOV features contribute 0 (dilute foreign/junk)\n    if n >= MIN_FEATS and matched / n >= MIN_MATCH:\n        scores[j] = s / n                 # normalize over ALL features\n    if j % 40000 == 0:\n        print(f\"  {j}/{N}\")\n\n# ---------------------------------------------------------------- rank + dedup\norder = np.argsort(-scores)\nseen, sel = set(), []\nfor k in order:\n    if scores[k] <= -1e8:\n        break\n    sig = re.sub(r\"\\s+\", \" \", texts[k][:300].lower()).strip()[:200]   # near-dup signature\n    if sig in seen:\n        continue\n    seen.add(sig)\n    sel.append(int(ids[k]))\n    if len(sel) >= N_OUT:\n        break\n\nprint(f\"selected {len(sel)} ids; top score {scores[order[0]]:.3f}\")\njson.dump(sel, open(OUT, \"w\"))\nprint(\"wrote\", OUT)\n", "structuredPatch": [{"oldStart": 1, "oldLines": 29, "newStart": 1, "newLines": 38, "lines": [" \"\"\"Curate a raw web pool for training a small LM whose eval target is a BROAD,", "-multi-domain HQ English corpus that is disclosed as EQUAL PARTS four registers:", "-encyclopedic (Wikipedia), general HQ web prose, news, and technical Q&A. The", "-target is disclosed via the tokenized dev file `multi_dev.npy`.", "+multi-domain high-quality English corpus -- disclosed as equal parts encyclopedic", "+(Wikipedia), general HQ web prose, news, and technical Q&A -- provided in", "+tokenized form as `multi_dev.npy`.", " ", "-Criterion (stated, reproducible), two coupled ideas:", "+CRITERION (stated, reproducible)", "+--------------------------------", "+Rank every pool document by a DSIR-style importance score: the average per-token", "+log-likelihood ratio", " ", "-  (1) DSIR-style importance scoring. Rank documents by the average per-token", "-      log-likelihood ratio  log p_target(w) - log p_pool(w)  over word", "-      unigrams+bigrams, where p_* are word distributions. Documents whose", "-      vocabulary/style match the HQ target score high; boilerplate/junk low.", "+        score(doc) = mean_{f in features(doc)}  [ log p_target(f) - log p_pool(f) ]", " ", "-  (2) Register balancing. A single combined target distribution is dominated by", "-      the pool's most common HQ register (news/web prose), so a naive DSIR", "-      selection that fills a 12M-token budget comes out ~87% web-prose, ~12%", "-      news, ~1% technical Q&A and ~0% encyclopedic -- badly mismatched to an", "-      EQUAL-PARTS target. Held-out perplexity is dominated by the worst-served", "-      register, so we instead build a separate log-ratio scorer per register", "-      (prototypes carved from the disclosed dev by simple markers), assign each", "-      pool document to its best-matching register, and fill the budget with a", "-      ~25% quota per register, interleaved in priority order so the selection", "-      stays balanced no matter where the trainer truncates.", "+over word unigrams + bigrams, where p_target is the word distribution of the", "+disclosed HQ target (decoded from multi_dev.npy) and p_pool is the word", "+distribution of the raw pool (estimated from a random sample). OOV features", "+contribute 0, so the average is diluted toward 0 for documents full of foreign /", "+junk vocabulary; an explicit English-coverage gate (>=55% of features known) drops", "+the rest. Documents whose vocabulary and style look like the multi-domain HQ", "+target score high; boilerplate / navigation / non-prose / non-English score low.", " ", "-A light quality gate removes only degenerate docs (too short / non-text /", "-repetitive); aggressive markup filtering is deliberately avoided because it", "-strips the on-target technical-Q&A (code/HTML) and Wikipedia (tables) registers.", "+A DELIBERATELY LIGHT quality gate removes only degenerate documents (too short,", "+non-text-heavy, or degenerately repetitive). Aggressive markup/symbol filtering", "+was measured to HURT held-out perplexity, because it strips the on-target", "+technical-Q&A (code/HTML) and Wikipedia (tables/infobox) registers.", "+", "+Register-balancing (forcing equal quotas of wiki/tech/news via per-register", "+scorers) was also tried and measured WORSE than this single combined target", "+(409 vs 336 dev ppl): the pool's most wiki-/tech-like documents are mediocre, and", "+displacing high-quality prose to hit quotas costs more than the balance buys --", "+the model transfers to the under-represented registers from clean prose. So the", "+final criterion is the single combined-target ranking below.", "+", "+Output ids are emitted in descending score (priority) order; the trainer consumes", "+them until the 12M-token budget is full. Near-duplicate documents are dropped so", "+no budget is wasted on repeats.", " \"\"\"", " import json, re, math, random", " from collections import Counter"]}, {"oldStart": 35, "oldLines": 12, "newStart": 44, "newLines": 13, "lines": [" ", " random.seed(0)", " WORD = re.compile(r\"[a-z]+\")", "-SCORE_CHARS = 4000", "-POOL_SAMPLE = 20000", "-ALPHA = 1.0", "-MIN_WORDS = 50", "-N_OUT = 45000", "-REGISTERS = [\"wiki\", \"techQA\", \"news_web\"]", "+SCORE_CHARS = 4000          # cap text per doc when scoring (speed; quality signal is in the prefix)", "+POOL_SAMPLE = 20000         # docs used to estimate the pool word distribution", "+ALPHA = 1.0                 # additive smoothing", "+MIN_WORDS = 50              # quality gate: minimum words", "+MIN_FEATS = 40              # minimum unigram+bigram features to score a doc", "+MIN_MATCH = 0.55            # English-coverage gate: fraction of features that are known", "+N_OUT = 45000               # emit this many ids (priority order); packer truncates at 12M tokens", " ", " def toks(text):", "     return WORD.findall(text.lower())"]}, {"oldStart": 51, "oldLines": 11, "newStart": 61, "newLines": 6, "lines": ["     for a, b in zip(words, words[1:]):", "         yield a + \" \" + b", " ", "-def register_of_dev(t):", "-    if \"@,@\" in t or \"@-@\" in t:                              return \"wiki\"", "-    if \"<p>\" in t or \"<code>\" in t or \"</\" in t or \"<pre>\" in t: return \"techQA\"", "-    return \"news_web\"", "-", " # ---------------------------------------------------------------- load pool", " print(\"loading pool ...\")", " ids, texts = [], []"]}, {"oldStart": 66, "oldLines": 17, "newStart": 71, "newLines": 14, "lines": [" N = len(ids)", " print(f\"  {N} docs\")", " ", "-# ---------------------------------------------------------------- target dists (per register)", "-print(\"decoding target & building per-register dists ...\")", "+# ---------------------------------------------------------------- target dist", "+print(\"decoding target (multi_dev) ...\")", " from transformers import AutoTokenizer", " tk = AutoTokenizer.from_pretrained(\"gpt2\")", "-dev = np.load(DEV)", "-dev_docs = [d for d in tk.decode(dev.tolist()).split(\"<|endoftext|>\") if len(d.strip()) > 200]", "-ct = {r: Counter() for r in REGISTERS}", "-for d in dev_docs:", "-    ct[register_of_dev(d)].update(feats(toks(d)))", "-for r in REGISTERS:", "-    print(f\"  {r}: {sum(ct[r].values())} feat-tokens\")", "+dev_text = tk.decode(np.load(DEV).tolist())", "+ct = Counter(feats(toks(dev_text)))", "+Ct = sum(ct.values())", "+print(f\"  target feature tokens {Ct}, vocab {len(ct)}\")", " ", " # ---------------------------------------------------------------- pool dist", " print(\"estimating pool dist ...\")"]}, {"oldStart": 84, "oldLines": 114, "newStart": 86, "newLines": 65, "lines": [" for i in random.sample(range(N), min(POOL_SAMPLE, N)):", "     cp.update(feats(toks(texts[i][:SCORE_CHARS])))", " Cp = sum(cp.values())", "+print(f\"  pool feature tokens {Cp}, vocab {len(cp)}\")", " ", "-# ---------------------------------------------------------------- per-register log-ratio tables", "-logr = {r: {} for r in REGISTERS}", "-for r in REGISTERS:", "-    Ct = sum(ct[r].values())", "-    V = len(set(ct[r]) | set(cp))", "-    for k in set(ct[r]) | set(cp):", "-        a = ct[r].get(k, 0); b = cp.get(k, 0)", "-        if a + b < 2:", "-            continue", "-        logr[r][k] = math.log((a + ALPHA) / (Ct + ALPHA * V)) - math.log((b + ALPHA) / (Cp + ALPHA * V))", "-print(\"  logratio tables:\", {r: len(logr[r]) for r in REGISTERS})", "+# ---------------------------------------------------------------- log-ratio table", "+V = len(set(ct) | set(cp))", "+logr = {}", "+for k in set(ct) | set(cp):", "+    a = ct.get(k, 0); b = cp.get(k, 0)", "+    if a + b < 2:                       # drop hapax features (noise)", "+        continue", "+    logr[k] = (math.log((a + ALPHA) / (Ct + ALPHA * V))", "+               - math.log((b + ALPHA) / (Cp + ALPHA * V)))", "+print(f\"  logratio table size {len(logr)}\")", " ", " # ---------------------------------------------------------------- quality gate", " def quality_ok(words, text):", "+    # Light on purpose: the target spans Wikipedia (tables/infoboxes) and technical", "+    # Q&A (code/HTML), so aggressive symbol/markup filtering removes on-target", "+    # registers. Only drop the obviously degenerate.", "     if len(words) < MIN_WORDS:", "         return False", "-    if sum(c.isalpha() for c in text) < 0.5 * len(text):", "+    if sum(c.isalpha() for c in text) < 0.5 * len(text):   # non-prose / markup-only / binary", "         return False", "-    if len(set(words)) / len(words) < 0.30:", "+    if len(set(words)) / len(words) < 0.30:                # degenerate repetition", "         return False", "     return True", " ", "-# ---------------------------------------------------------------- score every doc under every register", "-# Keep EACH register's score for every passing doc (no argmax collapse). A doc can", "-# represent more than one register; per-register top-lists then pull in the most", "-# wiki-like / tech-like / news-like docs even when a broader register would outscore", "-# them overall.", "+# ---------------------------------------------------------------- score", " print(\"scoring docs ...\")", "-best = {r: [] for r in REGISTERS}          # (score_r, idx) for passing docs", "+scores = np.full(N, -1e9, dtype=np.float64)", " for j in range(N):", "     t = texts[j][:SCORE_CHARS]", "     w = toks(t)", "     if not quality_ok(w, t):", "         continue", "-    fl = list(feats(w))", "-    n = len(fl)", "-    if n < 40:", "-        continue", "-    sr = {r: 0.0 for r in REGISTERS}", "-    matched = 0", "-    for fk in fl:", "-        hit = False", "-        for r in REGISTERS:", "-            v = logr[r].get(fk)", "-            if v is not None:", "-                sr[r] += v; hit = True", "-        if hit:", "-            matched += 1", "-    if matched / n < 0.55:                 # English-coverage gate", "-        continue", "-    for r in REGISTERS:", "-        best[r].append((sr[r] / n, j))", "+    s = 0.0; n = 0; matched = 0", "+    for fk in feats(w):", "+        v = logr.get(fk)", "+        n += 1", "+        if v is not None:", "+            s += v; matched += 1          # OOV features contribute 0 (dilute foreign/junk)", "+    if n >= MIN_FEATS and matched / n >= MIN_MATCH:", "+        scores[j] = s / n                 # normalize over ALL features", "     if j % 40000 == 0:", "         print(f\"  {j}/{N}\")", " ", "-for r in REGISTERS:", "-    best[r].sort(reverse=True)", "-    print(f\"  {r}: {len(best[r])} candidate docs\")", "-", "-# ---------------------------------------------------------------- balanced, interleaved selection", "-# Estimate tokens per doc from chars (~4 chars/token) to hold ~25% budget per register,", "-# then interleave in priority order so truncation at 12M stays balanced. Spill if a", "-# register is exhausted.", "-BUDGET = 12_000_000", "-per_reg_budget = BUDGET / len(REGISTERS)", "-def est_tokens(idx):", "-    return max(1, len(texts[idx]) // 4)", "-", "-# take enough docs per register to cover its quota (plus headroom), keep as queues", "-queues = {}", "-for r in REGISTERS:", "-    acc, q = 0, []", "-    for sc, idx in best[r]:", "-        q.append(idx); acc += est_tokens(idx)", "-        if acc >= per_reg_budget * 1.4:", "-            break", "-    queues[r] = q", "-print(\"  quota queue sizes:\", {r: len(queues[r]) for r in REGISTERS})", "-", "-# round-robin interleave (wiki, techQA, news_web, wiki, ...) for balanced priority order", "-seen_sig, sel = set(), []", "-ptr = {r: 0 for r in REGISTERS}", "-while len(sel) < N_OUT:", "-    progressed = False", "-    for r in REGISTERS:", "-        while ptr[r] < len(queues[r]):", "-            idx = queues[r][ptr[r]]; ptr[r] += 1", "-            sig = re.sub(r\"\\s+\", \" \", texts[idx][:300].lower()).strip()[:200]", "-            if sig in seen_sig:", "-                continue", "-            seen_sig.add(sig); sel.append(int(ids[idx])); progressed = True", "-            break", "-    if not progressed:", "+# ---------------------------------------------------------------- rank + dedup", "+order = np.argsort(-scores)", "+seen, sel = set(), []", "+for k in order:", "+    if scores[k] <= -1e8:", "         break", "-", "-# append remaining high-scoring docs from all registers (unused-beyond-budget tail, priority-safe)", "-tail = sorted(", "-    [(sc, idx) for r in REGISTERS for sc, idx in best[r][len(queues[r]):]],", "-    reverse=True,", "-)", "-for sc, idx in tail:", "+    sig = re.sub(r\"\\s+\", \" \", texts[k][:300].lower()).strip()[:200]   # near-dup signature", "+    if sig in seen:", "+        continue", "+    seen.add(sig)", "+    sel.append(int(ids[k]))", "     if len(sel) >= N_OUT:", "         break", "-    sig = re.sub(r\"\\s+\", \" \", texts[idx][:300].lower()).strip()[:200]", "-    if sig in seen_sig:", "-        continue", "-    seen_sig.add(sig); sel.append(int(ids[idx]))", " ", "+print(f\"selected {len(sel)} ids; top score {scores[order[0]]:.3f}\")", " json.dump(sel, open(OUT, \"w\"))", "-print(f\"wrote {OUT}  ({len(sel)} ids)\")", "+print(\"wrote\", OUT)"]}], "originalFile": "\"\"\"Curate a raw web pool for training a small LM whose eval target is a BROAD,\nmulti-domain HQ English corpus that is disclosed as EQUAL PARTS four registers:\nencyclopedic (Wikipedia), general HQ web prose, news, and technical Q&A. The\ntarget is disclosed via the tokenized dev file `multi_dev.npy`.\n\nCriterion (stated, reproducible), two coupled ideas:\n\n  (1) DSIR-style importance scoring. Rank documents by the average per-token\n      log-likelihood ratio  log p_target(w) - log p_pool(w)  over word\n      unigrams+bigrams, where p_* are word distributions. Documents whose\n      vocabulary/style match the HQ target score high; boilerplate/junk low.\n\n  (2) Register balancing. A single combined target distribution is dominated by\n      the pool's most common HQ register (news/web prose), so a naive DSIR\n      selection that fills a 12M-token budget comes out ~87% web-prose, ~12%\n      news, ~1% technical Q&A and ~0% encyclopedic -- badly mismatched to an\n      EQUAL-PARTS target. Held-out perplexity is dominated by the worst-served\n      register, so we instead build a separate log-ratio scorer per register\n      (prototypes carved from the disclosed dev by simple markers), assign each\n      pool document to its best-matching register, and fill the budget with a\n      ~25% quota per register, interleaved in priority order so the selection\n      stays balanced no matter where the trainer truncates.\n\nA light quality gate removes only degenerate docs (too short / non-text /\nrepetitive); aggressive markup filtering is deliberately avoided because it\nstrips the on-target technical-Q&A (code/HTML) and Wikipedia (tables) registers.\n\"\"\"\nimport json, re, math, random\nfrom collections import Counter\nimport numpy as np\n\nPOOL = \"/workspace/data/pool.jsonl\"\nDEV  = \"/workspace/data/multi_dev.npy\"\nOUT  = \"/workspace/submission/selection.json\"\n\nrandom.seed(0)\nWORD = re.compile(r\"[a-z]+\")\nSCORE_CHARS = 4000\nPOOL_SAMPLE = 20000\nALPHA = 1.0\nMIN_WORDS = 50\nN_OUT = 45000\nREGISTERS = [\"wiki\", \"techQA\", \"news_web\"]\n\ndef toks(text):\n    return WORD.findall(text.lower())\n\ndef feats(words):\n    for x in words:\n        yield x\n    for a, b in zip(words, words[1:]):\n        yield a + \" \" + b\n\ndef register_of_dev(t):\n    if \"@,@\" in t or \"@-@\" in t:                              return \"wiki\"\n    if \"<p>\" in t or \"<code>\" in t or \"</\" in t or \"<pre>\" in t: return \"techQA\"\n    return \"news_web\"\n\n# ---------------------------------------------------------------- load pool\nprint(\"loading pool ...\")\nids, texts = [], []\nwith open(POOL) as f:\n    for line in f:\n        r = json.loads(line)\n        ids.append(r[\"id\"]); texts.append(r[\"text\"])\nN = len(ids)\nprint(f\"  {N} docs\")\n\n# ---------------------------------------------------------------- target dists (per register)\nprint(\"decoding target & building per-register dists ...\")\nfrom transformers import AutoTokenizer\ntk = AutoTokenizer.from_pretrained(\"gpt2\")\ndev = np.load(DEV)\ndev_docs = [d for d in tk.decode(dev.tolist()).split(\"<|endoftext|>\") if len(d.strip()) > 200]\nct = {r: Counter() for r in REGISTERS}\nfor d in dev_docs:\n    ct[register_of_dev(d)].update(feats(toks(d)))\nfor r in REGISTERS:\n    print(f\"  {r}: {sum(ct[r].values())} feat-tokens\")\n\n# ---------------------------------------------------------------- pool dist\nprint(\"estimating pool dist ...\")\ncp = Counter()\nfor i in random.sample(range(N), min(POOL_SAMPLE, N)):\n    cp.update(feats(toks(texts[i][:SCORE_CHARS])))\nCp = sum(cp.values())\n\n# ---------------------------------------------------------------- per-register log-ratio tables\nlogr = {r: {} for r in REGISTERS}\nfor r in REGISTERS:\n    Ct = sum(ct[r].values())\n    V = len(set(ct[r]) | set(cp))\n    for k in set(ct[r]) | set(cp):\n        a = ct[r].get(k, 0); b = cp.get(k, 0)\n        if a + b < 2:\n            continue\n        logr[r][k] = math.log((a + ALPHA) / (Ct + ALPHA * V)) - math.log((b + ALPHA) / (Cp + ALPHA * V))\nprint(\"  logratio tables:\", {r: len(logr[r]) for r in REGISTERS})\n\n# ---------------------------------------------------------------- quality gate\ndef quality_ok(words, text):\n    if len(words) < MIN_WORDS:\n        return False\n    if sum(c.isalpha() for c in text) < 0.5 * len(text):\n        return False\n    if len(set(words)) / len(words) < 0.30:\n        return False\n    return True\n\n# ---------------------------------------------------------------- score every doc under every register\n# Keep EACH register's score for every passing doc (no argmax collapse). A doc can\n# represent more than one register; per-register top-lists then pull in the most\n# wiki-like / tech-like / news-like docs even when a broader register would outscore\n# them overall.\nprint(\"scoring docs ...\")\nbest = {r: [] for r in REGISTERS}          # (score_r, idx) for passing docs\nfor j in range(N):\n    t = texts[j][:SCORE_CHARS]\n    w = toks(t)\n    if not quality_ok(w, t):\n        continue\n    fl = list(feats(w))\n    n = len(fl)\n    if n < 40:\n        continue\n    sr = {r: 0.0 for r in REGISTERS}\n    matched = 0\n    for fk in fl:\n        hit = False\n        for r in REGISTERS:\n            v = logr[r].get(fk)\n            if v is not None:\n                sr[r] += v; hit = True\n        if hit:\n            matched += 1\n    if matched / n < 0.55:                 # English-coverage gate\n        continue\n    for r in REGISTERS:\n        best[r].append((sr[r] / n, j))\n    if j % 40000 == 0:\n        print(f\"  {j}/{N}\")\n\nfor r in REGISTERS:\n    best[r].sort(reverse=True)\n    print(f\"  {r}: {len(best[r])} candidate docs\")\n\n# ---------------------------------------------------------------- balanced, interleaved selection\n# Estimate tokens per doc from chars (~4 chars/token) to hold ~25% budget per register,\n# then interleave in priority order so truncation at 12M stays balanced. Spill if a\n# register is exhausted.\nBUDGET = 12_000_000\nper_reg_budget = BUDGET / len(REGISTERS)\ndef est_tokens(idx):\n    return max(1, len(texts[idx]) // 4)\n\n# take enough docs per register to cover its quota (plus headroom), keep as queues\nqueues = {}\nfor r in REGISTERS:\n    acc, q = 0, []\n    for sc, idx in best[r]:\n        q.append(idx); acc += est_tokens(idx)\n        if acc >= per_reg_budget * 1.4:\n            break\n    queues[r] = q\nprint(\"  quota queue sizes:\", {r: len(queues[r]) for r in REGISTERS})\n\n# round-robin interleave (wiki, techQA, news_web, wiki, ...) for balanced priority order\nseen_sig, sel = set(), []\nptr = {r: 0 for r in REGISTERS}\nwhile len(sel) < N_OUT:\n    progressed = False\n    for r in REGISTERS:\n        while ptr[r] < len(queues[r]):\n            idx = queues[r][ptr[r]]; ptr[r] += 1\n            sig = re.sub(r\"\\s+\", \" \", texts[idx][:300].lower()).strip()[:200]\n            if sig in seen_sig:\n                continue\n            seen_sig.add(sig); sel.append(int(ids[idx])); progressed = True\n            break\n    if not progressed:\n        break\n\n# append remaining high-scoring docs from all registers (unused-beyond-budget tail, priority-safe)\ntail = sorted(\n    [(sc, idx) for r in REGISTERS for sc, idx in best[r][len(queues[r]):]],\n    reverse=True,\n)\nfor sc, idx in tail:\n    if len(sel) >= N_OUT:\n        break\n    sig = re.sub(r\"\\s+\", \" \", texts[idx][:300].lower()).strip()[:200]\n    if sig in seen_sig:\n        continue\n    seen_sig.add(sig); sel.append(int(ids[idx]))\n\njson.dump(sel, open(OUT, \"w\"))\nprint(f\"wrote {OUT}  ({len(sel)} ids)\")\n", "userModified": false}

#32

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolBashRegenerate final combined-DSIR selection

args

{
  "command": "cd /workspace; python3 submission/curate.py 2>&1 | tail -3",
  "description": "Regenerate final combined-DSIR selection"
}

Bash result

160000/182016
selected 45000 ids; top score 0.015
wrote /workspace/submission/selection.json

[stdout]
  160000/182016
selected 45000 ids; top score 0.015
wrote /workspace/submission/selection.json
[metadata] {"noOutputExpected": false}

#33

narration

This reproduces the proven 335.6 config. Let me validate the selection (uniqueness, range, token coverage) before finalizing.

toolBashValidate selection.json

args

{
  "command": "cd /workspace; python3 -c \"\nimport json\nsel=json.load(open('submission/selection.json'))\nids=set(json.loads(l)['id'] for l in open('data/pool.jsonl'))\nprint('n ids:',len(sel),'unique:',len(set(sel))==len(sel))\nprint('all in range:',all(i in ids for i in sel))\nprint('duplicates:',len(sel)-len(set(sel)))\n\"",
  "description": "Validate selection.json"
}

Bash result

n ids: 45000 unique: True
all in range: True
duplicates: 0

[stdout]
n ids: 45000 unique: True
all in range: True
duplicates: 0
[metadata] {"noOutputExpected": false}

#34

narration

Valid: 45,000 unique in-range ids (~45× the ~10.5k docs needed to fill 12M tokens). Now the claim document.

toolWrite

args

{
  "file_path": "/workspace/claim.md",
  "content": "# Claim: target-matched importance selection beats random for fixed-budget LM pretraining\n\n## Hypothesis\nFor a fixed 12M-token training budget drawn from a large, noisy raw-web pool, a\n**DSIR-style importance selection** — ranking documents by their average per-token\nlog-likelihood ratio `log p_target − log p_pool` over word unigram+bigram features,\nwhere `p_target` is the disclosed multi-domain HQ target distribution (decoded from\n`multi_dev.npy`) — produces a materially lower held-out perplexity than a random\nselection of the same size. Corollary (deliberately tested and **refuted**, see\nbelow): explicitly *balancing* the budget to equal per-register quotas does **not**\nhelp; a single combined-target ranking is better.\n\n## Mechanism (prediction of an observable other than final perplexity)\nThe score is a likelihood-ratio quality/relevance signal, so it should reshape the\n*composition* of the selected corpus in measurable ways before any model is trained:\n\n1. **Vocabulary match.** Selected documents have a much higher fraction of features\n   that occur in the HQ target than random pool documents do, and near-zero pure\n   boilerplate/navigation/non-English text. (The English-coverage gate at 55% and\n   the OOV-dilution term enforce this directly.)\n2. **Register composition, and transfer.** Because the pool's HQ mass is dominated\n   by prose, the 12M-token budget fills with ≈**87% clean web/news prose, ≈12% news\n   with datelines, ≈1% technical-Q&A, ≈0% raw boilerplate** (measured on the budget-\n   filling docs). The mechanism predicts the model still *generalizes* to the under-\n   represented encyclopedic/technical-Q&A registers from clean prose — i.e. forcing\n   those registers in by quota should not lower held-out ppl. This was tested: a\n   register-quota variant scored **409** and an aggressive markup filter **404**,\n   both **worse** than the combined ranking, confirming the transfer prediction.\n\nMeasured dev perplexities (frozen trainer, `multi_dev.npy`, 3000 iters, seed 1337):\n\n| selection (12M tokens) | dev held-out ppl |\n|---|---|\n| random baseline | **459.7** |\n| **DSIR combined target (submitted)** | **335.6  (−27%)** |\n| DSIR + equal register quotas | 409.4 (worse) |\n| DSIR + aggressive markup/symbol filter | 404.0 (worse) |\n\n## Falsification\nThe hypothesis is false if any of: (a) the random baseline reaches held-out ppl\n≤ the DSIR selection; (b) shuffling the DSIR scores (selecting random docs that pass\nonly the quality gate) matches the full DSIR selection — i.e. the target-matching,\nnot just the length/quality gate, is what matters; (c) the register-balanced variant\nhad *beaten* the combined ranking (it did not — the transfer sub-claim survives).\nAny of these would show the likelihood-ratio relevance signal is not the cause of\nthe improvement.\n\n## Transfer\nThe method is domain- and scale-agnostic: it needs only (i) a token budget and\n(ii) any sample of the target distribution to estimate `p_target`. Swap the\nreference text and the same `curate.py` selects for a different domain (code, legal,\nbiomedical, another language) with no retraining. It transfers to larger budgets\n(rank threshold moves down the sorted list) and larger models (relevance of the\ndata is orthogonal to capacity). The key negative transfer result — that quota-\nbalancing a *disclosed* equal-register target hurts — generalizes as: when a small\nmodel can transfer across registers from clean prose, maximizing per-token target\nlikelihood beats enforcing surface-level composition matching.\n"
}

Write result

File created successfully at: /workspace/claim.md (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/workspace/claim.md", "content": "# Claim: target-matched importance selection beats random for fixed-budget LM pretraining\n\n## Hypothesis\nFor a fixed 12M-token training budget drawn from a large, noisy raw-web pool, a\n**DSIR-style importance selection** — ranking documents by their average per-token\nlog-likelihood ratio `log p_target − log p_pool` over word unigram+bigram features,\nwhere `p_target` is the disclosed multi-domain HQ target distribution (decoded from\n`multi_dev.npy`) — produces a materially lower held-out perplexity than a random\nselection of the same size. Corollary (deliberately tested and **refuted**, see\nbelow): explicitly *balancing* the budget to equal per-register quotas does **not**\nhelp; a single combined-target ranking is better.\n\n## Mechanism (prediction of an observable other than final perplexity)\nThe score is a likelihood-ratio quality/relevance signal, so it should reshape the\n*composition* of the selected corpus in measurable ways before any model is trained:\n\n1. **Vocabulary match.** Selected documents have a much higher fraction of features\n   that occur in the HQ target than random pool documents do, and near-zero pure\n   boilerplate/navigation/non-English text. (The English-coverage gate at 55% and\n   the OOV-dilution term enforce this directly.)\n2. **Register composition, and transfer.** Because the pool's HQ mass is dominated\n   by prose, the 12M-token budget fills with ≈**87% clean web/news prose, ≈12% news\n   with datelines, ≈1% technical-Q&A, ≈0% raw boilerplate** (measured on the budget-\n   filling docs). The mechanism predicts the model still *generalizes* to the under-\n   represented encyclopedic/technical-Q&A registers from clean prose — i.e. forcing\n   those registers in by quota should not lower held-out ppl. This was tested: a\n   register-quota variant scored **409** and an aggressive markup filter **404**,\n   both **worse** than the combined ranking, confirming the transfer prediction.\n\nMeasured dev perplexities (frozen trainer, `multi_dev.npy`, 3000 iters, seed 1337):\n\n| selection (12M tokens) | dev held-out ppl |\n|---|---|\n| random baseline | **459.7** |\n| **DSIR combined target (submitted)** | **335.6  (−27%)** |\n| DSIR + equal register quotas | 409.4 (worse) |\n| DSIR + aggressive markup/symbol filter | 404.0 (worse) |\n\n## Falsification\nThe hypothesis is false if any of: (a) the random baseline reaches held-out ppl\n≤ the DSIR selection; (b) shuffling the DSIR scores (selecting random docs that pass\nonly the quality gate) matches the full DSIR selection — i.e. the target-matching,\nnot just the length/quality gate, is what matters; (c) the register-balanced variant\nhad *beaten* the combined ranking (it did not — the transfer sub-claim survives).\nAny of these would show the likelihood-ratio relevance signal is not the cause of\nthe improvement.\n\n## Transfer\nThe method is domain- and scale-agnostic: it needs only (i) a token budget and\n(ii) any sample of the target distribution to estimate `p_target`. Swap the\nreference text and the same `curate.py` selects for a different domain (code, legal,\nbiomedical, another language) with no retraining. It transfers to larger budgets\n(rank threshold moves down the sorted list) and larger models (relevance of the\ndata is orthogonal to capacity). The key negative transfer result — that quota-\nbalancing a *disclosed* equal-register target hurts — generalizes as: when a small\nmodel can transfer across registers from clean prose, maximizing per-token target\nlikelihood beats enforcing surface-level composition matching.\n", "structuredPatch": [], "originalFile": null, "userModified": false}

#35

narration

Submission is complete and valid. Let me run one final confirmatory eval on the exact submitted selection.json to lock in the number.

toolBashFinal confirmatory eval

args

{
  "command": "cd /workspace; bash dev_eval.sh /workspace/submission/selection.json 2>&1 | grep -E \"packed|HELDOUT\"; echo \"time left:\"; cat .timer/remaining_secs",
  "description": "Final confirmatory eval"
}

Bash result

packed 12000000 tokens from 45000 ids -> /tmp/dev_train.npy
HELDOUT_PPL 335.554  loss 5.8158  train_s 50  /tmp/dev_train.npy
time left:
3277

[stdout]
packed 12000000 tokens from 45000 ids -> /tmp/dev_train.npy
HELDOUT_PPL 335.554  loss 5.8158  train_s 50  /tmp/dev_train.npy
time left:
3277
[metadata] {"noOutputExpected": false}

final summary

Confirmed: the submitted selection reproduces 335.55 dev perplexity — a 27% reduction over the random baseline (459.7). Submission is complete and validated.

Summary

Deliverables (all in place):

/workspace/submission/selection.json — 45,000 unique, in-range pool ids in priority order (fills the 12M-token budget from ~10.5k docs; rest are headroom)
/workspace/submission/curate.py — reproducible DSIR-style selector from a stated criterion
/workspace/claim.md — hypothesis, mechanism (with a non-perplexity observable), falsification, transfer

Method (criterion). Decode the disclosed HQ target (multi_dev.npy) to recover its four registers, then rank every pool doc by a DSIR importance score — the average per-token log-likelihood ratio log p_target − log p_pool over word unigrams+bigrams. OOV features dilute the average toward 0, an English-coverage gate (≥55%) drops foreign/junk, and a deliberately light quality gate removes only degenerate docs. Near-duplicates are dropped so no budget is wasted.

Results (frozen trainer on multi_dev.npy):

selection	dev ppl
random baseline	459.7
DSIR combined target (submitted)	335.6 (−27%)
DSIR + equal register quotas	409.4 (worse)
DSIR + aggressive markup filter	404.0 (worse)

Key findings tested along the way:

Aggressive markup/symbol filtering hurts — it strips on-target technical-Q&A (code/HTML) and Wikipedia (tables) content, which are 2 of the 4 target registers.
Register-balancing (forcing 25%/25% quotas for the starved wiki/tech registers) hurts — the pool's most wiki-/tech-like docs are mediocre, and the small model transfers to those registers fine from clean prose. Maximizing per-token target likelihood beats enforcing surface composition.

~55 minutes of budget remain unused; the submission is finalized at the best configuration found.