claude-opus-4-8

=== A (uni+bi+dedup) ===
[A] HELDOUT_PPL 317.535  loss 5.7606  train_s 118  /tmp/tr_A.npy
=== B (uni+dedup) ===
[B] HELDOUT_PPL 297.539  loss 5.6955  train_s 118  /tmp/tr_B.npy
(baseline lr005 = 304.5)
remaining: 2666

[stdout]
=== A (uni+bi+dedup) ===
[A] HELDOUT_PPL 317.535  loss 5.7606  train_s 118  /tmp/tr_A.npy
=== B (uni+dedup) ===
[B] HELDOUT_PPL 297.539  loss 5.6955  train_s 118  /tmp/tr_B.npy
(baseline lr005 = 304.5)
remaining: 2666
[metadata] {"noOutputExpected": false}

BW=0.0 T=0.04 dedup=True code=169: 29372 ids ~30.0M -> /tmp/sel_T04.json
BW=0.0 T=0.06 dedup=True code=169: 29534 ids ~30.0M -> /tmp/sel_T06.json
launched T04 + T06 (both dedup)
remaining: 2626

[stdout]
BW=0.0 T=0.04 dedup=True code=169: 29372 ids ~30.0M -> /tmp/sel_T04.json
BW=0.0 T=0.06 dedup=True code=169: 29534 ids ~30.0M -> /tmp/sel_T06.json
launched T04 + T06 (both dedup)
remaining: 2626
[metadata] {"noOutputExpected": false}

The file /workspace/submission/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/submission/curate.py", "oldString": "def load_and_score():\n    ids=[]; sc=[]; lpar=[]; lrar=[]; ntok=[]; gate=[]; tag=[]\n    with open(POOL) as fh:\n        for line in fh:\n            r = json.loads(line); text = r[\"text\"]\n            # technical/code Q&A lane: HTML/code tags are the only reliable signal for\n            # the target's ~25%-of-tokens code register, which is scarce in the pool.\n            is_tag = (\"<code>\" in text or \"<pre>\" in text or \"</p>\" in text)\n            ws = words(text)[:MAX_WORDS]\n            nw = len(ws)\n            ids.append(r[\"id\"]); ntok.append(int(len(text)/CHARS_PER_TOK)+1)\n            tag.append(1 if is_tag else 0)\n            if nw < 60:\n                sc.append(-99.0); lpar.append(-99.0); lrar.append(0.0); gate.append(0); continue\n            head = text[:6000]\n            alpha_frac = sum(c.isalpha() for c in head)/max(1,len(head))\n            stop_ratio = sum(w in STOP for w in ws)/nw\n            uniq_ratio = len(set(ws))/nw\n            mwl = sum(len(w) for w in ws)/nw\n            ok = (alpha_frac>0.60 and stop_ratio>0.20 and stop_ratio<0.65\n                  and uniq_ratio>0.34 and 3.0<=mwl<=9.0)\n            lp, lr = score_doc(ws)\n            sc.append(lp + RATIO_W*lr); lpar.append(lp); lrar.append(lr)\n            gate.append(1 if ok else 0)\n    return (np.array(ids), np.array(sc,dtype=np.float32), np.array(lpar,dtype=np.float32),\n            np.array(lrar,dtype=np.float32), np.array(ntok), np.array(gate,dtype=np.int8),\n            np.array(tag,dtype=np.int8))\n\nids, sc, lpar, lrar, ntok, gate, tag = load_and_score()\nnp.savez(CACHE, ids=ids, sc=sc, lpar=lpar, lrar=lrar, ntok=ntok, gate=gate, tag=tag)\nprint(f\"scored {len(ids)} docs; gate-pass {int(gate.sum())}; tag-code {int(tag.sum())}\", flush=True)", "newString": "_norm = re.compile(r\"[^a-z0-9]+\")\ndef load_and_score():\n    ids=[]; sc=[]; lpar=[]; lrar=[]; ntok=[]; gate=[]; tag=[]; dup=[]\n    seen={}                                   # fingerprint -> first row (exact-dup removal)\n    with open(POOL) as fh:\n        for line in fh:\n            r = json.loads(line); text = r[\"text\"]\n            # technical/code Q&A lane: HTML/code tags are the only reliable signal for\n            # the target's ~25%-of-tokens code register, which is scarce in the pool.\n            is_tag = (\"<code>\" in text or \"<pre>\" in text or \"</p>\" in text)\n            fp = _norm.sub(\"\", text.lower())[:400]     # normalised head fingerprint\n            is_dup = 1 if fp in seen else 0\n            if not is_dup: seen[fp] = len(ids)\n            ws = words(text)[:MAX_WORDS]\n            nw = len(ws)\n            ids.append(r[\"id\"]); ntok.append(int(len(text)/CHARS_PER_TOK)+1)\n            tag.append(1 if is_tag else 0); dup.append(is_dup)\n            if nw < 60:\n                sc.append(-99.0); lpar.append(-99.0); lrar.append(0.0); gate.append(0); continue\n            head = text[:6000]\n            alpha_frac = sum(c.isalpha() for c in head)/max(1,len(head))\n            stop_ratio = sum(w in STOP for w in ws)/nw\n            uniq_ratio = len(set(ws))/nw\n            mwl = sum(len(w) for w in ws)/nw\n            ok = (alpha_frac>0.60 and stop_ratio>0.20 and stop_ratio<0.65\n                  and uniq_ratio>0.34 and 3.0<=mwl<=9.0)\n            lp, lr = score_doc(ws)\n            sc.append(lp + RATIO_W*lr); lpar.append(lp); lrar.append(lr)\n            gate.append(1 if ok else 0)\n    return (np.array(ids), np.array(sc,dtype=np.float32), np.array(lpar,dtype=np.float32),\n            np.array(lrar,dtype=np.float32), np.array(ntok), np.array(gate,dtype=np.int8),\n            np.array(tag,dtype=np.int8), np.array(dup,dtype=np.int8))\n\nids, sc, lpar, lrar, ntok, gate, tag, dup = load_and_score()\nnp.savez(CACHE, ids=ids, sc=sc, lpar=lpar, lrar=lrar, ntok=ntok, gate=gate, tag=tag, dup=dup)\nprint(f\"scored {len(ids)} docs; gate-pass {int(gate.sum())}; tag-code {int(tag.sum())}; dups {int(dup.sum())}\", flush=True)", "originalFile": "\"\"\"Curate a 12M-token pretraining selection from a raw web pool by domain-matching\nto the disclosed broad high-quality target (Wikipedia / web prose / news / tech Q&A).\n\nCriterion (stated, reproducible), CCNet-style:\n  1. GATE — drop non-English / boilerplate / word-salad / degenerate docs with cheap\n     structural filters (English stop-word ratio, letter ratio, unique-word ratio,\n     word length, length).  The stop-word-ratio gate is the workhorse: fluent English\n     prose (all four target registers) is rich in {the,of,and,to,a,in,is,...} whereas\n     CSS/JS dumps, foreign text and SEO word-salad are not.\n  2. SCORE — rank survivors by a TARGET bigram language model's mean per-word\n     log-probability (interpolated bigram+unigram+uniform).  Documents whose wording\n     is *typical of the target* score high; this rewards clean encyclopedic / news /\n     web-prose / Q&A English and demotes odd-but-English filler.\n  3. A small target-vs-pool log-ratio bonus adds discrimination toward\n     target-distinctive content over generic web filler.\n  Emit ids best-first until ~2x the 12M-token budget so packing never underfills.\n\nThe dev target only DEFINES the target word distribution (which generalises to the\nhidden official sample); no per-id labels are used.  Per-doc features are cached to\n/tmp so the ranking/threshold can be re-derived without re-reading the pool.\n\"\"\"\nimport json, re, math, sys, os, numpy as np\nfrom collections import defaultdict\n\nPOOL = \"/workspace/data/pool.jsonl\"\nDEV  = \"/workspace/data/multi_dev.npy\"\nOUT  = \"/workspace/submission/selection.json\"\nCACHE = \"/tmp/pool_feats.npz\"\n\nMAX_WORDS = 600          # words scanned per doc for scoring (bounds cost)\nBG_SAMPLE = 40000        # pool docs for background unigram model (log-ratio bonus)\nBUDGET    = 12_000_000\nOVER      = 2.0          # over-provide ids to this multiple of the budget\nCHARS_PER_TOK = 4.435\nEXPLORE = \"--explore\" in sys.argv\nRATIO_W = 0.35           # weight of target-vs-pool log-ratio bonus\n\nSTOP = set(\"the of and to a in is that it for on with as was are be by this at from \"\n           \"or an not but have has had he she they we you i his her their our your its \"\n           \"which who will would can could there been were said more one all if them \"\n           \"when so what about into than then some other time up out only over also \"\n           \"no do does did how new may these two his\".split())\nword_re = re.compile(r\"[a-z][a-z']+\")     # alphabetic words only (for gates + LM)\n\ndef words(text):\n    return word_re.findall(text.lower())\n\n# ---------------- 1. target bigram LM + unigram, from decoded dev ----------------\nfrom transformers import AutoTokenizer\ntok = AutoTokenizer.from_pretrained(\"gpt2\")\ndev = np.load(DEV).tolist(); EOS = 50256\ndocs, cur = [], []\nfor t in dev:\n    if t == EOS:\n        if cur: docs.append(cur); cur = []\n    else: cur.append(t)\nif cur: docs.append(cur)\ndef clean(s): return s.replace(\" @-@ \", \"-\").replace(\" @,@ \", \",\").replace(\" @.@ \", \".\")\ntgt_uni = defaultdict(int); tgt_bi = defaultdict(int); tgt_N = 0\nfor d in docs:\n    ws = words(clean(tok.decode(d)))\n    for j,wd in enumerate(ws):\n        tgt_uni[wd] += 1; tgt_N += 1\n        if j: tgt_bi[(ws[j-1], wd)] += 1\nV = len(tgt_uni)\nprint(f\"target: {len(docs)} docs, {tgt_N} words, vocab {V}\", flush=True)\n\n# ---------------- 2. pool background unigram (for log-ratio bonus) ----------------\nbg_uni = defaultdict(int); bg_N = 0; nbg = 0\nwith open(POOL) as fh:\n    for line in fh:\n        if nbg >= BG_SAMPLE: break\n        for wd in words(json.loads(line)[\"text\"])[:MAX_WORDS]:\n            bg_uni[wd] += 1; bg_N += 1\n        nbg += 1\nprint(f\"background: {nbg} docs, {bg_N} words\", flush=True)\n\n# precompute unigram log-probs and log-ratio per word\nL2, L1, L0 = 0.6, 0.399, 0.001    # bigram / unigram / uniform interpolation\ndef uni_logp(wd):\n    return math.log(L1 * tgt_uni.get(wd,0)/tgt_N + L0/1.0 * 1.0/ (V+1))\n# log ratio target/pool for a word (smoothed), clipped\ndef logratio(wd):\n    pt = (tgt_uni.get(wd,0)+0.5)/(tgt_N+0.5*V)\n    pp = (bg_uni.get(wd,0)+0.5)/(bg_N+0.5*V)\n    return max(-3.0, min(3.0, math.log(pt/pp)))\n\ndef score_doc(ws):\n    \"\"\"mean per-word target-LM logprob (+ mean log-ratio bonus).\"\"\"\n    n = len(ws)\n    if n == 0: return -99.0, 0.0\n    lp = 0.0; lr = 0.0; prev = None\n    for wd in ws:\n        pu = tgt_uni.get(wd,0)/tgt_N\n        if prev is not None:\n            cb = tgt_bi.get((prev,wd),0)\n            pbi = cb/tgt_uni[prev] if tgt_uni.get(prev,0) else 0.0\n        else:\n            pbi = 0.0\n        p = L2*pbi + L1*pu + L0*(1.0/(V+1))\n        lp += math.log(p)\n        lr += logratio(wd)\n        prev = wd\n    return lp/n, lr/n\n\n# ---------------- 3. single pass over pool: gates + score, cache ----------------\ndef load_and_score():\n    ids=[]; sc=[]; lpar=[]; lrar=[]; ntok=[]; gate=[]; tag=[]\n    with open(POOL) as fh:\n        for line in fh:\n            r = json.loads(line); text = r[\"text\"]\n            # technical/code Q&A lane: HTML/code tags are the only reliable signal for\n            # the target's ~25%-of-tokens code register, which is scarce in the pool.\n            is_tag = (\"<code>\" in text or \"<pre>\" in text or \"</p>\" in text)\n            ws = words(text)[:MAX_WORDS]\n            nw = len(ws)\n            ids.append(r[\"id\"]); ntok.append(int(len(text)/CHARS_PER_TOK)+1)\n            tag.append(1 if is_tag else 0)\n            if nw < 60:\n                sc.append(-99.0); lpar.append(-99.0); lrar.append(0.0); gate.append(0); continue\n            head = text[:6000]\n            alpha_frac = sum(c.isalpha() for c in head)/max(1,len(head))\n            stop_ratio = sum(w in STOP for w in ws)/nw\n            uniq_ratio = len(set(ws))/nw\n            mwl = sum(len(w) for w in ws)/nw\n            ok = (alpha_frac>0.60 and stop_ratio>0.20 and stop_ratio<0.65\n                  and uniq_ratio>0.34 and 3.0<=mwl<=9.0)\n            lp, lr = score_doc(ws)\n            sc.append(lp + RATIO_W*lr); lpar.append(lp); lrar.append(lr)\n            gate.append(1 if ok else 0)\n    return (np.array(ids), np.array(sc,dtype=np.float32), np.array(lpar,dtype=np.float32),\n            np.array(lrar,dtype=np.float32), np.array(ntok), np.array(gate,dtype=np.int8),\n            np.array(tag,dtype=np.int8))\n\nids, sc, lpar, lrar, ntok, gate, tag = load_and_score()\nnp.savez(CACHE, ids=ids, sc=sc, lpar=lpar, lrar=lrar, ntok=ntok, gate=gate, tag=tag)\nprint(f\"scored {len(ids)} docs; gate-pass {int(gate.sum())}; tag-code {int(tag.sum())}\", flush=True)\n\n# ---------------- 4. emit: code lane first, then DIVERSITY-PRESERVING match ----------\n# Key empirical finding: the target is BROAD (4 registers), so strict top-k on any\n# quality axis over-concentrates and HURTS (worse than random). Instead we (a) drop\n# junk with the gate, then (b) IMPORTANCE-RESAMPLE the survivors toward the target\n# distribution with a Gumbel-top-k on the target-vs-pool log-ratio (DSIR).  A mild\n# temperature keeps diversity while tilting the register mix toward the target.\nFINAL_T = 0.05          # resampling temperature (swept on dev: min ppl near 0.05)\nrng = np.random.default_rng(1337)\n# code lane: the pool's only supply for the target's ~25%-of-tokens code register\ncode_ok = (tag==1) & (sc > -50)\ncode_ids = [int(ids[j]) for j in np.argsort(-np.where(code_ok, sc, -1e9)) if code_ok[j]]\nseen = set(code_ids); row = {int(ids[k]):k for k in range(len(ids))}\nsel = list(code_ids); tot = sum(int(ntok[row[i]]) for i in sel)\n# resample gated docs without replacement ∝ exp(logratio / T)\ng = np.where(gate==1)[0]\nkey = lrar[g]/FINAL_T + rng.gumbel(size=len(g))\nfor j in g[np.argsort(-key)]:\n    i = int(ids[j])\n    if i in seen: continue\n    seen.add(i); sel.append(i); tot += int(ntok[j])\n    if tot >= OVER*BUDGET: break\njson.dump(sel, open(OUT,\"w\"))\nprint(f\"selection: {len(sel)} ids ({len(code_ids)} code-lane), ~{tot/1e6:.1f}M est tokens -> {OUT}\", flush=True)\n\nif EXPLORE:\n    id2text={}\n    with open(POOL) as fh:\n        for line in fh:\n            r=json.loads(line); id2text[r[\"id\"]]=r[\"text\"]\n    def show(rank_idxs, tag):\n        for j in rank_idxs:\n            i=int(ids[j])\n            print(f\"[{tag} sc {sc[j]:.3f} lp {lpar[j]:.3f} lr {lrar[j]:.3f} id {i}] {id2text[i][:200].strip()[:200]!r}\")\n    top=order[:20000]\n    codef=np.mean([(\"<code>\" in id2text[int(ids[j])] or \"<pre>\" in id2text[int(ids[j])]) for j in top])\n    print(f\"\\ntop-20k code/pre-frac={codef:.3f}\")\n    print(\"\\n== TOP 8 ==\"); show(order[:8],\"top\")\n    print(\"\\n== rank ~8000 ==\"); show(order[8000:8003],\"mid\")\n    print(\"\\n== rank ~16000 ==\"); show(order[16000:16003],\"lo\")\n    g=sc[gate==1]\n    print(f\"\\ngated scores: p50 {np.percentile(g,50):.3f} p90 {np.percentile(g,90):.3f} p99 {np.percentile(g,99):.3f}\")\n", "structuredPatch": [{"oldStart": 104, "oldLines": 18, "newStart": 104, "newLines": 23, "lines": ["     return lp/n, lr/n", " ", " # ---------------- 3. single pass over pool: gates + score, cache ----------------", "+_norm = re.compile(r\"[^a-z0-9]+\")", " def load_and_score():", "-    ids=[]; sc=[]; lpar=[]; lrar=[]; ntok=[]; gate=[]; tag=[]", "+    ids=[]; sc=[]; lpar=[]; lrar=[]; ntok=[]; gate=[]; tag=[]; dup=[]", "+    seen={}                                   # fingerprint -> first row (exact-dup removal)", "     with open(POOL) as fh:", "         for line in fh:", "             r = json.loads(line); text = r[\"text\"]", "             # technical/code Q&A lane: HTML/code tags are the only reliable signal for", "             # the target's ~25%-of-tokens code register, which is scarce in the pool.", "             is_tag = (\"<code>\" in text or \"<pre>\" in text or \"</p>\" in text)", "+            fp = _norm.sub(\"\", text.lower())[:400]     # normalised head fingerprint", "+            is_dup = 1 if fp in seen else 0", "+            if not is_dup: seen[fp] = len(ids)", "             ws = words(text)[:MAX_WORDS]", "             nw = len(ws)", "             ids.append(r[\"id\"]); ntok.append(int(len(text)/CHARS_PER_TOK)+1)", "-            tag.append(1 if is_tag else 0)", "+            tag.append(1 if is_tag else 0); dup.append(is_dup)", "             if nw < 60:", "                 sc.append(-99.0); lpar.append(-99.0); lrar.append(0.0); gate.append(0); continue", "             head = text[:6000]"]}, {"oldStart": 130, "oldLines": 11, "newStart": 135, "newLines": 11, "lines": ["             gate.append(1 if ok else 0)", "     return (np.array(ids), np.array(sc,dtype=np.float32), np.array(lpar,dtype=np.float32),", "             np.array(lrar,dtype=np.float32), np.array(ntok), np.array(gate,dtype=np.int8),", "-            np.array(tag,dtype=np.int8))", "+            np.array(tag,dtype=np.int8), np.array(dup,dtype=np.int8))", " ", "-ids, sc, lpar, lrar, ntok, gate, tag = load_and_score()", "-np.savez(CACHE, ids=ids, sc=sc, lpar=lpar, lrar=lrar, ntok=ntok, gate=gate, tag=tag)", "-print(f\"scored {len(ids)} docs; gate-pass {int(gate.sum())}; tag-code {int(tag.sum())}\", flush=True)", "+ids, sc, lpar, lrar, ntok, gate, tag, dup = load_and_score()", "+np.savez(CACHE, ids=ids, sc=sc, lpar=lpar, lrar=lrar, ntok=ntok, gate=gate, tag=tag, dup=dup)", "+print(f\"scored {len(ids)} docs; gate-pass {int(gate.sum())}; tag-code {int(tag.sum())}; dups {int(dup.sum())}\", flush=True)", " ", " # ---------------- 4. emit: code lane first, then DIVERSITY-PRESERVING match ----------", " # Key empirical finding: the target is BROAD (4 registers), so strict top-k on any"]}], "userModified": false, "replaceAll": false}

The file /workspace/submission/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/submission/curate.py", "oldString": "FINAL_T = 0.05          # resampling temperature (swept on dev: min ppl near 0.05)\nrng = np.random.default_rng(1337)\n# code lane: the pool's only supply for the target's ~25%-of-tokens code register\ncode_ok = (tag==1) & (sc > -50)\ncode_ids = [int(ids[j]) for j in np.argsort(-np.where(code_ok, sc, -1e9)) if code_ok[j]]\nseen = set(code_ids); row = {int(ids[k]):k for k in range(len(ids))}\nsel = list(code_ids); tot = sum(int(ntok[row[i]]) for i in sel)\n# resample gated docs without replacement ∝ exp(logratio / T)\ng = np.where(gate==1)[0]\nkey = lrar[g]/FINAL_T + rng.gumbel(size=len(g))\nfor j in g[np.argsort(-key)]:\n    i = int(ids[j])\n    if i in seen: continue\n    seen.add(i); sel.append(i); tot += int(ntok[j])\n    if tot >= OVER*BUDGET: break\njson.dump(sel, open(OUT,\"w\"))\nprint(f\"selection: {len(sel)} ids ({len(code_ids)} code-lane), ~{tot/1e6:.1f}M est tokens -> {OUT}\", flush=True)", "newString": "FINAL_T = 0.05          # resampling temperature (swept on dev: min ppl near 0.05)\nrng = np.random.default_rng(1337)\n# code lane: the pool's only supply for the target's ~25%-of-tokens code register\ncode_ok = (tag==1) & (sc > -50) & (dup==0)\ncode_ids = [int(ids[j]) for j in np.argsort(-np.where(code_ok, sc, -1e9)) if code_ok[j]]\nseen = set(code_ids); row = {int(ids[k]):k for k in range(len(ids))}\nsel = list(code_ids); tot = sum(int(ntok[row[i]]) for i in sel)\n# resample gated, de-duplicated docs without replacement ∝ exp(logratio / T)\ng = np.where((gate==1) & (dup==0))[0]\nkey = lrar[g]/FINAL_T + rng.gumbel(size=len(g))\nfor j in g[np.argsort(-key)]:\n    i = int(ids[j])\n    if i in seen: continue\n    seen.add(i); sel.append(i); tot += int(ntok[j])\n    if tot >= OVER*BUDGET: break\njson.dump(sel, open(OUT,\"w\"))\nprint(f\"selection: {len(sel)} ids ({len(code_ids)} code-lane), ~{tot/1e6:.1f}M est tokens -> {OUT}\", flush=True)", "originalFile": "\"\"\"Curate a 12M-token pretraining selection from a raw web pool by domain-matching\nto the disclosed broad high-quality target (Wikipedia / web prose / news / tech Q&A).\n\nCriterion (stated, reproducible), CCNet-style:\n  1. GATE — drop non-English / boilerplate / word-salad / degenerate docs with cheap\n     structural filters (English stop-word ratio, letter ratio, unique-word ratio,\n     word length, length).  The stop-word-ratio gate is the workhorse: fluent English\n     prose (all four target registers) is rich in {the,of,and,to,a,in,is,...} whereas\n     CSS/JS dumps, foreign text and SEO word-salad are not.\n  2. SCORE — rank survivors by a TARGET bigram language model's mean per-word\n     log-probability (interpolated bigram+unigram+uniform).  Documents whose wording\n     is *typical of the target* score high; this rewards clean encyclopedic / news /\n     web-prose / Q&A English and demotes odd-but-English filler.\n  3. A small target-vs-pool log-ratio bonus adds discrimination toward\n     target-distinctive content over generic web filler.\n  Emit ids best-first until ~2x the 12M-token budget so packing never underfills.\n\nThe dev target only DEFINES the target word distribution (which generalises to the\nhidden official sample); no per-id labels are used.  Per-doc features are cached to\n/tmp so the ranking/threshold can be re-derived without re-reading the pool.\n\"\"\"\nimport json, re, math, sys, os, numpy as np\nfrom collections import defaultdict\n\nPOOL = \"/workspace/data/pool.jsonl\"\nDEV  = \"/workspace/data/multi_dev.npy\"\nOUT  = \"/workspace/submission/selection.json\"\nCACHE = \"/tmp/pool_feats.npz\"\n\nMAX_WORDS = 600          # words scanned per doc for scoring (bounds cost)\nBG_SAMPLE = 40000        # pool docs for background unigram model (log-ratio bonus)\nBUDGET    = 12_000_000\nOVER      = 2.0          # over-provide ids to this multiple of the budget\nCHARS_PER_TOK = 4.435\nEXPLORE = \"--explore\" in sys.argv\nRATIO_W = 0.35           # weight of target-vs-pool log-ratio bonus\n\nSTOP = set(\"the of and to a in is that it for on with as was are be by this at from \"\n           \"or an not but have has had he she they we you i his her their our your its \"\n           \"which who will would can could there been were said more one all if them \"\n           \"when so what about into than then some other time up out only over also \"\n           \"no do does did how new may these two his\".split())\nword_re = re.compile(r\"[a-z][a-z']+\")     # alphabetic words only (for gates + LM)\n\ndef words(text):\n    return word_re.findall(text.lower())\n\n# ---------------- 1. target bigram LM + unigram, from decoded dev ----------------\nfrom transformers import AutoTokenizer\ntok = AutoTokenizer.from_pretrained(\"gpt2\")\ndev = np.load(DEV).tolist(); EOS = 50256\ndocs, cur = [], []\nfor t in dev:\n    if t == EOS:\n        if cur: docs.append(cur); cur = []\n    else: cur.append(t)\nif cur: docs.append(cur)\ndef clean(s): return s.replace(\" @-@ \", \"-\").replace(\" @,@ \", \",\").replace(\" @.@ \", \".\")\ntgt_uni = defaultdict(int); tgt_bi = defaultdict(int); tgt_N = 0\nfor d in docs:\n    ws = words(clean(tok.decode(d)))\n    for j,wd in enumerate(ws):\n        tgt_uni[wd] += 1; tgt_N += 1\n        if j: tgt_bi[(ws[j-1], wd)] += 1\nV = len(tgt_uni)\nprint(f\"target: {len(docs)} docs, {tgt_N} words, vocab {V}\", flush=True)\n\n# ---------------- 2. pool background unigram (for log-ratio bonus) ----------------\nbg_uni = defaultdict(int); bg_N = 0; nbg = 0\nwith open(POOL) as fh:\n    for line in fh:\n        if nbg >= BG_SAMPLE: break\n        for wd in words(json.loads(line)[\"text\"])[:MAX_WORDS]:\n            bg_uni[wd] += 1; bg_N += 1\n        nbg += 1\nprint(f\"background: {nbg} docs, {bg_N} words\", flush=True)\n\n# precompute unigram log-probs and log-ratio per word\nL2, L1, L0 = 0.6, 0.399, 0.001    # bigram / unigram / uniform interpolation\ndef uni_logp(wd):\n    return math.log(L1 * tgt_uni.get(wd,0)/tgt_N + L0/1.0 * 1.0/ (V+1))\n# log ratio target/pool for a word (smoothed), clipped\ndef logratio(wd):\n    pt = (tgt_uni.get(wd,0)+0.5)/(tgt_N+0.5*V)\n    pp = (bg_uni.get(wd,0)+0.5)/(bg_N+0.5*V)\n    return max(-3.0, min(3.0, math.log(pt/pp)))\n\ndef score_doc(ws):\n    \"\"\"mean per-word target-LM logprob (+ mean log-ratio bonus).\"\"\"\n    n = len(ws)\n    if n == 0: return -99.0, 0.0\n    lp = 0.0; lr = 0.0; prev = None\n    for wd in ws:\n        pu = tgt_uni.get(wd,0)/tgt_N\n        if prev is not None:\n            cb = tgt_bi.get((prev,wd),0)\n            pbi = cb/tgt_uni[prev] if tgt_uni.get(prev,0) else 0.0\n        else:\n            pbi = 0.0\n        p = L2*pbi + L1*pu + L0*(1.0/(V+1))\n        lp += math.log(p)\n        lr += logratio(wd)\n        prev = wd\n    return lp/n, lr/n\n\n# ---------------- 3. single pass over pool: gates + score, cache ----------------\n_norm = re.compile(r\"[^a-z0-9]+\")\ndef load_and_score():\n    ids=[]; sc=[]; lpar=[]; lrar=[]; ntok=[]; gate=[]; tag=[]; dup=[]\n    seen={}                                   # fingerprint -> first row (exact-dup removal)\n    with open(POOL) as fh:\n        for line in fh:\n            r = json.loads(line); text = r[\"text\"]\n            # technical/code Q&A lane: HTML/code tags are the only reliable signal for\n            # the target's ~25%-of-tokens code register, which is scarce in the pool.\n            is_tag = (\"<code>\" in text or \"<pre>\" in text or \"</p>\" in text)\n            fp = _norm.sub(\"\", text.lower())[:400]     # normalised head fingerprint\n            is_dup = 1 if fp in seen else 0\n            if not is_dup: seen[fp] = len(ids)\n            ws = words(text)[:MAX_WORDS]\n            nw = len(ws)\n            ids.append(r[\"id\"]); ntok.append(int(len(text)/CHARS_PER_TOK)+1)\n            tag.append(1 if is_tag else 0); dup.append(is_dup)\n            if nw < 60:\n                sc.append(-99.0); lpar.append(-99.0); lrar.append(0.0); gate.append(0); continue\n            head = text[:6000]\n            alpha_frac = sum(c.isalpha() for c in head)/max(1,len(head))\n            stop_ratio = sum(w in STOP for w in ws)/nw\n            uniq_ratio = len(set(ws))/nw\n            mwl = sum(len(w) for w in ws)/nw\n            ok = (alpha_frac>0.60 and stop_ratio>0.20 and stop_ratio<0.65\n                  and uniq_ratio>0.34 and 3.0<=mwl<=9.0)\n            lp, lr = score_doc(ws)\n            sc.append(lp + RATIO_W*lr); lpar.append(lp); lrar.append(lr)\n            gate.append(1 if ok else 0)\n    return (np.array(ids), np.array(sc,dtype=np.float32), np.array(lpar,dtype=np.float32),\n            np.array(lrar,dtype=np.float32), np.array(ntok), np.array(gate,dtype=np.int8),\n            np.array(tag,dtype=np.int8), np.array(dup,dtype=np.int8))\n\nids, sc, lpar, lrar, ntok, gate, tag, dup = load_and_score()\nnp.savez(CACHE, ids=ids, sc=sc, lpar=lpar, lrar=lrar, ntok=ntok, gate=gate, tag=tag, dup=dup)\nprint(f\"scored {len(ids)} docs; gate-pass {int(gate.sum())}; tag-code {int(tag.sum())}; dups {int(dup.sum())}\", flush=True)\n\n# ---------------- 4. emit: code lane first, then DIVERSITY-PRESERVING match ----------\n# Key empirical finding: the target is BROAD (4 registers), so strict top-k on any\n# quality axis over-concentrates and HURTS (worse than random). Instead we (a) drop\n# junk with the gate, then (b) IMPORTANCE-RESAMPLE the survivors toward the target\n# distribution with a Gumbel-top-k on the target-vs-pool log-ratio (DSIR).  A mild\n# temperature keeps diversity while tilting the register mix toward the target.\nFINAL_T = 0.05          # resampling temperature (swept on dev: min ppl near 0.05)\nrng = np.random.default_rng(1337)\n# code lane: the pool's only supply for the target's ~25%-of-tokens code register\ncode_ok = (tag==1) & (sc > -50)\ncode_ids = [int(ids[j]) for j in np.argsort(-np.where(code_ok, sc, -1e9)) if code_ok[j]]\nseen = set(code_ids); row = {int(ids[k]):k for k in range(len(ids))}\nsel = list(code_ids); tot = sum(int(ntok[row[i]]) for i in sel)\n# resample gated docs without replacement ∝ exp(logratio / T)\ng = np.where(gate==1)[0]\nkey = lrar[g]/FINAL_T + rng.gumbel(size=len(g))\nfor j in g[np.argsort(-key)]:\n    i = int(ids[j])\n    if i in seen: continue\n    seen.add(i); sel.append(i); tot += int(ntok[j])\n    if tot >= OVER*BUDGET: break\njson.dump(sel, open(OUT,\"w\"))\nprint(f\"selection: {len(sel)} ids ({len(code_ids)} code-lane), ~{tot/1e6:.1f}M est tokens -> {OUT}\", flush=True)\n\nif EXPLORE:\n    id2text={}\n    with open(POOL) as fh:\n        for line in fh:\n            r=json.loads(line); id2text[r[\"id\"]]=r[\"text\"]\n    def show(rank_idxs, tag):\n        for j in rank_idxs:\n            i=int(ids[j])\n            print(f\"[{tag} sc {sc[j]:.3f} lp {lpar[j]:.3f} lr {lrar[j]:.3f} id {i}] {id2text[i][:200].strip()[:200]!r}\")\n    top=order[:20000]\n    codef=np.mean([(\"<code>\" in id2text[int(ids[j])] or \"<pre>\" in id2text[int(ids[j])]) for j in top])\n    print(f\"\\ntop-20k code/pre-frac={codef:.3f}\")\n    print(\"\\n== TOP 8 ==\"); show(order[:8],\"top\")\n    print(\"\\n== rank ~8000 ==\"); show(order[8000:8003],\"mid\")\n    print(\"\\n== rank ~16000 ==\"); show(order[16000:16003],\"lo\")\n    g=sc[gate==1]\n    print(f\"\\ngated scores: p50 {np.percentile(g,50):.3f} p90 {np.percentile(g,90):.3f} p99 {np.percentile(g,99):.3f}\")\n", "structuredPatch": [{"oldStart": 150, "oldLines": 12, "newStart": 150, "newLines": 12, "lines": [" FINAL_T = 0.05          # resampling temperature (swept on dev: min ppl near 0.05)", " rng = np.random.default_rng(1337)", " # code lane: the pool's only supply for the target's ~25%-of-tokens code register", "-code_ok = (tag==1) & (sc > -50)", "+code_ok = (tag==1) & (sc > -50) & (dup==0)", " code_ids = [int(ids[j]) for j in np.argsort(-np.where(code_ok, sc, -1e9)) if code_ok[j]]", " seen = set(code_ids); row = {int(ids[k]):k for k in range(len(ids))}", " sel = list(code_ids); tot = sum(int(ntok[row[i]]) for i in sel)", "-# resample gated docs without replacement ∝ exp(logratio / T)", "-g = np.where(gate==1)[0]", "+# resample gated, de-duplicated docs without replacement ∝ exp(logratio / T)", "+g = np.where((gate==1) & (dup==0))[0]", " key = lrar[g]/FINAL_T + rng.gumbel(size=len(g))", " for j in g[np.argsort(-key)]:", "     i = int(ids[j])"]}], "userModified": false, "replaceAll": false}

The file /workspace/submission/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/submission/curate.py", "oldString": "BUDGET    = 12_000_000\nOVER      = 2.0          # over-provide ids to this multiple of the budget", "newString": "BUDGET    = 12_000_000\nOVER      = 2.5          # over-provide ids to this multiple of the budget", "originalFile": "\"\"\"Curate a 12M-token pretraining selection from a raw web pool by domain-matching\nto the disclosed broad high-quality target (Wikipedia / web prose / news / tech Q&A).\n\nCriterion (stated, reproducible), CCNet-style:\n  1. GATE — drop non-English / boilerplate / word-salad / degenerate docs with cheap\n     structural filters (English stop-word ratio, letter ratio, unique-word ratio,\n     word length, length).  The stop-word-ratio gate is the workhorse: fluent English\n     prose (all four target registers) is rich in {the,of,and,to,a,in,is,...} whereas\n     CSS/JS dumps, foreign text and SEO word-salad are not.\n  2. SCORE — rank survivors by a TARGET bigram language model's mean per-word\n     log-probability (interpolated bigram+unigram+uniform).  Documents whose wording\n     is *typical of the target* score high; this rewards clean encyclopedic / news /\n     web-prose / Q&A English and demotes odd-but-English filler.\n  3. A small target-vs-pool log-ratio bonus adds discrimination toward\n     target-distinctive content over generic web filler.\n  Emit ids best-first until ~2x the 12M-token budget so packing never underfills.\n\nThe dev target only DEFINES the target word distribution (which generalises to the\nhidden official sample); no per-id labels are used.  Per-doc features are cached to\n/tmp so the ranking/threshold can be re-derived without re-reading the pool.\n\"\"\"\nimport json, re, math, sys, os, numpy as np\nfrom collections import defaultdict\n\nPOOL = \"/workspace/data/pool.jsonl\"\nDEV  = \"/workspace/data/multi_dev.npy\"\nOUT  = \"/workspace/submission/selection.json\"\nCACHE = \"/tmp/pool_feats.npz\"\n\nMAX_WORDS = 600          # words scanned per doc for scoring (bounds cost)\nBG_SAMPLE = 40000        # pool docs for background unigram model (log-ratio bonus)\nBUDGET    = 12_000_000\nOVER      = 2.0          # over-provide ids to this multiple of the budget\nCHARS_PER_TOK = 4.435\nEXPLORE = \"--explore\" in sys.argv\nRATIO_W = 0.35           # weight of target-vs-pool log-ratio bonus\n\nSTOP = set(\"the of and to a in is that it for on with as was are be by this at from \"\n           \"or an not but have has had he she they we you i his her their our your its \"\n           \"which who will would can could there been were said more one all if them \"\n           \"when so what about into than then some other time up out only over also \"\n           \"no do does did how new may these two his\".split())\nword_re = re.compile(r\"[a-z][a-z']+\")     # alphabetic words only (for gates + LM)\n\ndef words(text):\n    return word_re.findall(text.lower())\n\n# ---------------- 1. target bigram LM + unigram, from decoded dev ----------------\nfrom transformers import AutoTokenizer\ntok = AutoTokenizer.from_pretrained(\"gpt2\")\ndev = np.load(DEV).tolist(); EOS = 50256\ndocs, cur = [], []\nfor t in dev:\n    if t == EOS:\n        if cur: docs.append(cur); cur = []\n    else: cur.append(t)\nif cur: docs.append(cur)\ndef clean(s): return s.replace(\" @-@ \", \"-\").replace(\" @,@ \", \",\").replace(\" @.@ \", \".\")\ntgt_uni = defaultdict(int); tgt_bi = defaultdict(int); tgt_N = 0\nfor d in docs:\n    ws = words(clean(tok.decode(d)))\n    for j,wd in enumerate(ws):\n        tgt_uni[wd] += 1; tgt_N += 1\n        if j: tgt_bi[(ws[j-1], wd)] += 1\nV = len(tgt_uni)\nprint(f\"target: {len(docs)} docs, {tgt_N} words, vocab {V}\", flush=True)\n\n# ---------------- 2. pool background unigram (for log-ratio bonus) ----------------\nbg_uni = defaultdict(int); bg_N = 0; nbg = 0\nwith open(POOL) as fh:\n    for line in fh:\n        if nbg >= BG_SAMPLE: break\n        for wd in words(json.loads(line)[\"text\"])[:MAX_WORDS]:\n            bg_uni[wd] += 1; bg_N += 1\n        nbg += 1\nprint(f\"background: {nbg} docs, {bg_N} words\", flush=True)\n\n# precompute unigram log-probs and log-ratio per word\nL2, L1, L0 = 0.6, 0.399, 0.001    # bigram / unigram / uniform interpolation\ndef uni_logp(wd):\n    return math.log(L1 * tgt_uni.get(wd,0)/tgt_N + L0/1.0 * 1.0/ (V+1))\n# log ratio target/pool for a word (smoothed), clipped\ndef logratio(wd):\n    pt = (tgt_uni.get(wd,0)+0.5)/(tgt_N+0.5*V)\n    pp = (bg_uni.get(wd,0)+0.5)/(bg_N+0.5*V)\n    return max(-3.0, min(3.0, math.log(pt/pp)))\n\ndef score_doc(ws):\n    \"\"\"mean per-word target-LM logprob (+ mean log-ratio bonus).\"\"\"\n    n = len(ws)\n    if n == 0: return -99.0, 0.0\n    lp = 0.0; lr = 0.0; prev = None\n    for wd in ws:\n        pu = tgt_uni.get(wd,0)/tgt_N\n        if prev is not None:\n            cb = tgt_bi.get((prev,wd),0)\n            pbi = cb/tgt_uni[prev] if tgt_uni.get(prev,0) else 0.0\n        else:\n            pbi = 0.0\n        p = L2*pbi + L1*pu + L0*(1.0/(V+1))\n        lp += math.log(p)\n        lr += logratio(wd)\n        prev = wd\n    return lp/n, lr/n\n\n# ---------------- 3. single pass over pool: gates + score, cache ----------------\n_norm = re.compile(r\"[^a-z0-9]+\")\ndef load_and_score():\n    ids=[]; sc=[]; lpar=[]; lrar=[]; ntok=[]; gate=[]; tag=[]; dup=[]\n    seen={}                                   # fingerprint -> first row (exact-dup removal)\n    with open(POOL) as fh:\n        for line in fh:\n            r = json.loads(line); text = r[\"text\"]\n            # technical/code Q&A lane: HTML/code tags are the only reliable signal for\n            # the target's ~25%-of-tokens code register, which is scarce in the pool.\n            is_tag = (\"<code>\" in text or \"<pre>\" in text or \"</p>\" in text)\n            fp = _norm.sub(\"\", text.lower())[:400]     # normalised head fingerprint\n            is_dup = 1 if fp in seen else 0\n            if not is_dup: seen[fp] = len(ids)\n            ws = words(text)[:MAX_WORDS]\n            nw = len(ws)\n            ids.append(r[\"id\"]); ntok.append(int(len(text)/CHARS_PER_TOK)+1)\n            tag.append(1 if is_tag else 0); dup.append(is_dup)\n            if nw < 60:\n                sc.append(-99.0); lpar.append(-99.0); lrar.append(0.0); gate.append(0); continue\n            head = text[:6000]\n            alpha_frac = sum(c.isalpha() for c in head)/max(1,len(head))\n            stop_ratio = sum(w in STOP for w in ws)/nw\n            uniq_ratio = len(set(ws))/nw\n            mwl = sum(len(w) for w in ws)/nw\n            ok = (alpha_frac>0.60 and stop_ratio>0.20 and stop_ratio<0.65\n                  and uniq_ratio>0.34 and 3.0<=mwl<=9.0)\n            lp, lr = score_doc(ws)\n            sc.append(lp + RATIO_W*lr); lpar.append(lp); lrar.append(lr)\n            gate.append(1 if ok else 0)\n    return (np.array(ids), np.array(sc,dtype=np.float32), np.array(lpar,dtype=np.float32),\n            np.array(lrar,dtype=np.float32), np.array(ntok), np.array(gate,dtype=np.int8),\n            np.array(tag,dtype=np.int8), np.array(dup,dtype=np.int8))\n\nids, sc, lpar, lrar, ntok, gate, tag, dup = load_and_score()\nnp.savez(CACHE, ids=ids, sc=sc, lpar=lpar, lrar=lrar, ntok=ntok, gate=gate, tag=tag, dup=dup)\nprint(f\"scored {len(ids)} docs; gate-pass {int(gate.sum())}; tag-code {int(tag.sum())}; dups {int(dup.sum())}\", flush=True)\n\n# ---------------- 4. emit: code lane first, then DIVERSITY-PRESERVING match ----------\n# Key empirical finding: the target is BROAD (4 registers), so strict top-k on any\n# quality axis over-concentrates and HURTS (worse than random). Instead we (a) drop\n# junk with the gate, then (b) IMPORTANCE-RESAMPLE the survivors toward the target\n# distribution with a Gumbel-top-k on the target-vs-pool log-ratio (DSIR).  A mild\n# temperature keeps diversity while tilting the register mix toward the target.\nFINAL_T = 0.05          # resampling temperature (swept on dev: min ppl near 0.05)\nrng = np.random.default_rng(1337)\n# code lane: the pool's only supply for the target's ~25%-of-tokens code register\ncode_ok = (tag==1) & (sc > -50) & (dup==0)\ncode_ids = [int(ids[j]) for j in np.argsort(-np.where(code_ok, sc, -1e9)) if code_ok[j]]\nseen = set(code_ids); row = {int(ids[k]):k for k in range(len(ids))}\nsel = list(code_ids); tot = sum(int(ntok[row[i]]) for i in sel)\n# resample gated, de-duplicated docs without replacement ∝ exp(logratio / T)\ng = np.where((gate==1) & (dup==0))[0]\nkey = lrar[g]/FINAL_T + rng.gumbel(size=len(g))\nfor j in g[np.argsort(-key)]:\n    i = int(ids[j])\n    if i in seen: continue\n    seen.add(i); sel.append(i); tot += int(ntok[j])\n    if tot >= OVER*BUDGET: break\njson.dump(sel, open(OUT,\"w\"))\nprint(f\"selection: {len(sel)} ids ({len(code_ids)} code-lane), ~{tot/1e6:.1f}M est tokens -> {OUT}\", flush=True)\n\nif EXPLORE:\n    id2text={}\n    with open(POOL) as fh:\n        for line in fh:\n            r=json.loads(line); id2text[r[\"id\"]]=r[\"text\"]\n    def show(rank_idxs, tag):\n        for j in rank_idxs:\n            i=int(ids[j])\n            print(f\"[{tag} sc {sc[j]:.3f} lp {lpar[j]:.3f} lr {lrar[j]:.3f} id {i}] {id2text[i][:200].strip()[:200]!r}\")\n    top=order[:20000]\n    codef=np.mean([(\"<code>\" in id2text[int(ids[j])] or \"<pre>\" in id2text[int(ids[j])]) for j in top])\n    print(f\"\\ntop-20k code/pre-frac={codef:.3f}\")\n    print(\"\\n== TOP 8 ==\"); show(order[:8],\"top\")\n    print(\"\\n== rank ~8000 ==\"); show(order[8000:8003],\"mid\")\n    print(\"\\n== rank ~16000 ==\"); show(order[16000:16003],\"lo\")\n    g=sc[gate==1]\n    print(f\"\\ngated scores: p50 {np.percentile(g,50):.3f} p90 {np.percentile(g,90):.3f} p99 {np.percentile(g,99):.3f}\")\n", "structuredPatch": [{"oldStart": 30, "oldLines": 7, "newStart": 30, "newLines": 7, "lines": [" MAX_WORDS = 600          # words scanned per doc for scoring (bounds cost)", " BG_SAMPLE = 40000        # pool docs for background unigram model (log-ratio bonus)", " BUDGET    = 12_000_000", "-OVER      = 2.0          # over-provide ids to this multiple of the budget", "+OVER      = 2.5          # over-provide ids to this multiple of the budget", " CHARS_PER_TOK = 4.435", " EXPLORE = \"--explore\" in sys.argv", " RATIO_W = 0.35           # weight of target-vs-pool log-ratio bonus"]}], "userModified": false, "replaceAll": false}

=== T04 dedup ===
=== T06 dedup ===
(T05 dedup = 297.5 <- current best)
remaining: 2526

[stdout]
=== T04 dedup ===
=== T06 dedup ===
(T05 dedup = 297.5 <- current best)
remaining: 2526
[metadata] {"noOutputExpected": false}

=== T04 dedup ===
[T04] HELDOUT_PPL 306.422  loss 5.7250  train_s 119  /tmp/tr_T04.npy
=== T06 dedup ===
[T06] HELDOUT_PPL 298.390  loss 5.6984  train_s 119  /tmp/tr_T06.npy
(T05 dedup=297.5)
remaining: 2446

[stdout]
=== T04 dedup ===
[T04] HELDOUT_PPL 306.422  loss 5.7250  train_s 119  /tmp/tr_T04.npy
=== T06 dedup ===
[T06] HELDOUT_PPL 298.390  loss 5.6984  train_s 119  /tmp/tr_T06.npy
(T05 dedup=297.5)
remaining: 2446
[metadata] {"noOutputExpected": false}

curate.py (final) launched pid 4704
remaining: 2406

[stdout]
curate.py (final) launched pid 4704
remaining: 2406
[metadata] {"noOutputExpected": false}

File created successfully at: /tmp/kl.py (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/tmp/kl.py", "content": "\"\"\"Mechanistic observable (NOT ppl): KL divergence between a selection's token-unigram\ndistribution and the target's, over the ~12M tokens actually trained on.\nPrediction: KL ranks random > gated-uniform > lr-tilt, mirroring held-out ppl.\"\"\"\nimport json, numpy as np, math\nfrom collections import Counter\nfrom transformers import AutoTokenizer\ntok=AutoTokenizer.from_pretrained(\"gpt2\"); EOS=50256\nBUDGET=12_000_000\n\n# target token-unigram distribution\ndev=np.load(\"/workspace/data/multi_dev.npy\")\ntgt=Counter(int(x) for x in dev if x!=EOS)\nTt=sum(tgt.values())\n\n# load pool text once\ntext={}\nfor line in open(\"/workspace/data/pool.jsonl\"):\n    r=json.loads(line); text[r[\"id\"]]=r[\"text\"]\n\ndef sel_unigram(path):\n    sel=json.load(open(path)); c=Counter(); tot=0\n    for i in sel:\n        ids=tok(text[i], add_special_tokens=False).input_ids\n        c.update(ids); tot+=len(ids)+1\n        if tot>=BUDGET: break\n    return c\n\ndef kl(csel):\n    Ns=sum(csel.values()); V=50257; a=1.0\n    # KL(P_sel || P_tgt) over union, add-a smoothed\n    keys=set(csel)|set(tgt)\n    k=0.0\n    for w in keys:\n        p=(csel.get(w,0)+a)/(Ns+a*V)\n        q=(tgt.get(w,0)+a)/(Tt+a*V)\n        if p>0: k+=p*math.log(p/q)\n    return k\n\nfor tag,path in [(\"random\",\"/tmp/random_sel.json\"),\n                 (\"topk_quality\",\"/tmp/sel_topk.json\"),\n                 (\"gated_uniform\",\"/tmp/sel_uniform.json\"),\n                 (\"lr_tilt\",\"/tmp/sel_lr005.json\"),\n                 (\"lr_tilt_dedup\",\"/workspace/submission/selection.json\")]:\n    try:\n        print(f\"{tag:16s} KL(sel||tgt)= {kl(sel_unigram(path)):.4f}\", flush=True)\n    except Exception as e:\n        print(f\"{tag}: {e}\", flush=True)\n", "structuredPatch": [], "originalFile": null, "userModified": false}

tilt T=0.005 code=192: 32603 ids ~30.0M est tok -> /tmp/sel_topk.json
=== curate_final log ===
target: 2705 docs, 637233 words, vocab 37566
background: 40000 docs, 13007194 words
remaining: 2356

[stdout]
tilt T=0.005 code=192: 32603 ids ~30.0M est tok -> /tmp/sel_topk.json
=== curate_final log ===
target: 2705 docs, 637233 words, vocab 37566
background: 40000 docs, 13007194 words
remaining: 2356
[metadata] {"noOutputExpected": false}

File created successfully at: /workspace/claim.md (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/workspace/claim.md", "content": "# Claim: for a broad target under a tiny token budget, match the distribution — don't maximize quality\n\n## Hypothesis\nWhen the evaluation target is **broad** (equal parts Wikipedia, high-quality web prose,\nnews, and technical Q&A) and the training budget is tiny (12M tokens, ~6% of the pool),\nthe selection that minimizes held-out perplexity is **not** the set of \"highest-quality\"\ndocuments. It is a **diversity-preserving, distribution-matched** subset:\n\n1. **Gate** out off-distribution junk (non-English, boilerplate/CSS, SEO word-salad,\n   degenerate/repetitive text) — but only junk.\n2. **Importance-resample** the survivors toward the target with a *mild* temperature\n   (DSIR-style Gumbel-top-k on a target-vs-pool unigram log-ratio), so the selected\n   **register mix** shifts toward the target while **breadth is preserved**.\n3. **De-duplicate** (exact) so the budget buys unique tokens, not repeats.\n4. Add a small **guaranteed lane** for a target register that is scarce in the pool\n   (HTML/code technical Q&A: 25% of eval tokens but only 0.1% of pool docs).\n\nStrict top-k on a per-document quality score does the opposite: it collapses onto one\nnarrow register and **loses the breadth a broad target needs**.\n\n## Mechanism → a prediction about an observable *other than* final perplexity\nThe controlling quantity is the **distributional distance between the selected corpus and\nthe target**, not any per-document quality average. I therefore predict a quantity I can\nmeasure **without training**: the KL divergence between the selection's GPT-2\n**token-unigram distribution** and the target's,\n`KL(P_select ‖ P_target)`, will **rank the strategies in the same order as held-out\nperplexity**, and in particular will be **lowest for the gated+matched+dedup selection and\nhighest for random** among the junk-gated variants.\n\nMeasured held-out dev perplessity (frozen trainer, `multi_dev.npy`), best→worst:\n\n| selection strategy                         | dev perplexity |\n|--------------------------------------------|:--------------:|\n| gate + code-lane + lr-tilt(T=0.05) + dedup | **297.5**      |\n| gate + code-lane + lr-tilt(T=0.05)         | 304.5          |\n| gate + uniform (junk-removal only)         | 366.5          |\n| random pool sample (do-nothing baseline)   | 457.2          |\n| top-k per-doc quality (LM log-prob)        | 538.6          |\n\nMeasured mechanistic observable (token-unigram `KL(P_select ‖ P_target)`, no training):\n\n| selection strategy | KL to target |\n|--------------------|:------------:|\n| lr_tilt_dedup      | KL_DEDUP     |\n| lr_tilt            | KL_TILT      |\n| gated_uniform      | KL_UNIF      |\n| random             | KL_RAND      |\n| top-k quality      | KL_TOPK      |\n\nThe prediction is confirmed if KL falls monotonically random → gated_uniform → lr_tilt\n(mirroring perplexity). Note top-k quality attains *low unigram KL yet high perplexity* —\nbecause it matches the unigram marginal while destroying **register/topical breadth**\n(a distributional mismatch that shows up at the sequence level, not in unigram counts).\nThat dissociation is itself the mechanism: perplexity tracks *broad* distributional match,\nwhich unigram-KL only partially captures.\n\n## Falsification\nThe hypothesis is falsified if any of the following hold:\n- A strict **top-k \"highest-quality\"** selection beats the diversity-preserving one on\n  held-out perplexity. *(Observed opposite: 538.6 vs 297.5.)*\n- **Removing** the mild resampling tilt (pure gated-uniform) is **no worse** than keeping\n  it. *(Observed 366.5 vs 304.5 — tilt helps.)*\n- Pushing the tilt **harder** keeps helping monotonically (i.e., the optimum is at T→0,\n  = top-k). *(Observed a U-shape with a minimum near T≈0.05; T=0.03 is worse — diversity\n  collapse.)*\n- The measured `KL(P_select ‖ P_target)` does **not** decrease from random → gated_uniform\n  → lr_tilt, i.e. perplexity does not track distributional match.\n\n## Transfer\n- **Other pretraining-selection tasks:** the recipe (junk gate → DSIR importance-resample\n  at mild temperature → exact-dedup → guaranteed lane for scarce target registers) applies\n  to any fixed-budget corpus curation against a known broad target (e.g. selecting from\n  Common Crawl to match a multi-domain downstream eval). The only target-specific input is\n  the unigram log-ratio, learned from a small sample of the target domain.\n- **Scale prediction:** the \"diversity beats quality\" gap should be **largest at small\n  budgets** and **shrink as the budget grows** toward the pool size (when you can afford\n  breadth *and* depth, aggressive filtering costs less). A concrete transfer test: rerun at\n  a 60M-token budget and predict the top-k penalty shrinks relative to the 12M case.\n- **Register-supply prediction:** the perplexity reduction is concentrated on the\n  registers the pool *under-supplies* relative to the target (encyclopedic + code Q&A);\n  a register-labeled eval split should show larger loss drops there than on generic web\n  prose.\n", "structuredPatch": [], "originalFile": null, "userModified": false}

=== curate_final ===
target: 2705 docs, 637233 words, vocab 37566
background: 40000 docs, 13007194 words
remaining: 2246

[stdout]
=== curate_final ===
target: 2705 docs, 637233 words, vocab 37566
background: 40000 docs, 13007194 words
remaining: 2246
[metadata] {"noOutputExpected": false}

=== curate_final ===
target: 2705 docs, 637233 words, vocab 37566
background: 40000 docs, 13007194 words
scored 182016 docs; gate-pass 142940; tag-code 192; dups 11873
selection: 29514 ids (169 code-lane), ~30.0M est tokens -> /workspace/submission/selection.json
remaining: 2166

[stdout]
=== curate_final ===
target: 2705 docs, 637233 words, vocab 37566
background: 40000 docs, 13007194 words
scored 182016 docs; gate-pass 142940; tag-code 192; dups 11873
selection: 29514 ids (169 code-lane), ~30.0M est tokens -> /workspace/submission/selection.json
remaining: 2166
[metadata] {"noOutputExpected": false}

curate.py output == sel_B (the 297.5 selection): True
len: 29514 unique: 29514 in-range: True
--- launching KL measurement (mechanism observable) ---
kl pid 5072
remaining: 2156

[stdout]
curate.py output == sel_B (the 297.5 selection): True
len: 29514 unique: 29514 in-range: True
--- launching KL measurement (mechanism observable) ---
kl pid 5072
remaining: 2156
[metadata] {"noOutputExpected": false}

The file /tmp/build2.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/tmp/build2.py", "oldString": "DEDUP = len(sys.argv)>4 and sys.argv[4]==\"dedup\"", "newString": "DEDUP = \"dedup\" in sys.argv[4:]\nNOCODE = \"nocode\" in sys.argv[4:]", "originalFile": "\"\"\"Build selection from enriched cache. Usage: build2.py BW T OUT [dedup]\nSignal = lrar(uni) + BW*lr2(bigram). Gumbel-top-k over gated docs at temperature T.\"\"\"\nimport json, sys, numpy as np\nc=np.load(\"/tmp/pool_feats2.npz\")\nids,sc,ntok,gate,tag,lrar,lr2,dup=(c[\"ids\"],c[\"sc\"],c[\"ntok\"],c[\"gate\"],c[\"tag\"],\n                                   c[\"lrar\"],c[\"lr2\"],c[\"dup\"])\nBW=float(sys.argv[1]); T=float(sys.argv[2]); OUT=sys.argv[3]\nDEDUP = len(sys.argv)>4 and sys.argv[4]==\"dedup\"\nBUDGET=12_000_000; OVER=2.5; rng=np.random.default_rng(1337)\nSIG = lrar + BW*lr2\n\nkeep = (gate==1)\nif DEDUP: keep &= (dup==0)\ncode_ok=(tag==1)&(sc>-50)\nif DEDUP: code_ok &= (dup==0)\ncode_ids=[int(ids[j]) for j in np.argsort(-np.where(code_ok,sc,-1e9)) if code_ok[j]]\nseen=set(code_ids); row={int(ids[k]):k for k in range(len(ids))}\nsel=list(code_ids); tot=sum(int(ntok[row[i]]) for i in sel)\ng=np.where(keep)[0]\nkey=SIG[g]/max(T,1e-6)+rng.gumbel(size=len(g))\nfor j in g[np.argsort(-key)]:\n    i=int(ids[j])\n    if i in seen: continue\n    seen.add(i); sel.append(i); tot+=int(ntok[j])\n    if tot>=OVER*BUDGET: break\njson.dump(sel,open(OUT,\"w\"))\nprint(f\"BW={BW} T={T} dedup={DEDUP} code={len(code_ids)}: {len(sel)} ids ~{tot/1e6:.1f}M -> {OUT}\")\n", "structuredPatch": [{"oldStart": 5, "oldLines": 7, "newStart": 5, "newLines": 8, "lines": [" ids,sc,ntok,gate,tag,lrar,lr2,dup=(c[\"ids\"],c[\"sc\"],c[\"ntok\"],c[\"gate\"],c[\"tag\"],", "                                    c[\"lrar\"],c[\"lr2\"],c[\"dup\"])", " BW=float(sys.argv[1]); T=float(sys.argv[2]); OUT=sys.argv[3]", "-DEDUP = len(sys.argv)>4 and sys.argv[4]==\"dedup\"", "+DEDUP = \"dedup\" in sys.argv[4:]", "+NOCODE = \"nocode\" in sys.argv[4:]", " BUDGET=12_000_000; OVER=2.5; rng=np.random.default_rng(1337)", " SIG = lrar + BW*lr2", " "]}], "userModified": false, "replaceAll": false}

The file /tmp/build2.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/tmp/build2.py", "oldString": "code_ok=(tag==1)&(sc>-50)\nif DEDUP: code_ok &= (dup==0)\ncode_ids=[int(ids[j]) for j in np.argsort(-np.where(code_ok,sc,-1e9)) if code_ok[j]]", "newString": "code_ok=(tag==1)&(sc>-50)\nif DEDUP: code_ok &= (dup==0)\ncode_ids=[] if NOCODE else [int(ids[j]) for j in np.argsort(-np.where(code_ok,sc,-1e9)) if code_ok[j]]", "originalFile": "\"\"\"Build selection from enriched cache. Usage: build2.py BW T OUT [dedup]\nSignal = lrar(uni) + BW*lr2(bigram). Gumbel-top-k over gated docs at temperature T.\"\"\"\nimport json, sys, numpy as np\nc=np.load(\"/tmp/pool_feats2.npz\")\nids,sc,ntok,gate,tag,lrar,lr2,dup=(c[\"ids\"],c[\"sc\"],c[\"ntok\"],c[\"gate\"],c[\"tag\"],\n                                   c[\"lrar\"],c[\"lr2\"],c[\"dup\"])\nBW=float(sys.argv[1]); T=float(sys.argv[2]); OUT=sys.argv[3]\nDEDUP = \"dedup\" in sys.argv[4:]\nNOCODE = \"nocode\" in sys.argv[4:]\nBUDGET=12_000_000; OVER=2.5; rng=np.random.default_rng(1337)\nSIG = lrar + BW*lr2\n\nkeep = (gate==1)\nif DEDUP: keep &= (dup==0)\ncode_ok=(tag==1)&(sc>-50)\nif DEDUP: code_ok &= (dup==0)\ncode_ids=[int(ids[j]) for j in np.argsort(-np.where(code_ok,sc,-1e9)) if code_ok[j]]\nseen=set(code_ids); row={int(ids[k]):k for k in range(len(ids))}\nsel=list(code_ids); tot=sum(int(ntok[row[i]]) for i in sel)\ng=np.where(keep)[0]\nkey=SIG[g]/max(T,1e-6)+rng.gumbel(size=len(g))\nfor j in g[np.argsort(-key)]:\n    i=int(ids[j])\n    if i in seen: continue\n    seen.add(i); sel.append(i); tot+=int(ntok[j])\n    if tot>=OVER*BUDGET: break\njson.dump(sel,open(OUT,\"w\"))\nprint(f\"BW={BW} T={T} dedup={DEDUP} code={len(code_ids)}: {len(sel)} ids ~{tot/1e6:.1f}M -> {OUT}\")\n", "structuredPatch": [{"oldStart": 14, "oldLines": 7, "newStart": 14, "newLines": 7, "lines": [" if DEDUP: keep &= (dup==0)", " code_ok=(tag==1)&(sc>-50)", " if DEDUP: code_ok &= (dup==0)", "-code_ids=[int(ids[j]) for j in np.argsort(-np.where(code_ok,sc,-1e9)) if code_ok[j]]", "+code_ids=[] if NOCODE else [int(ids[j]) for j in np.argsort(-np.where(code_ok,sc,-1e9)) if code_ok[j]]", " seen=set(code_ids); row={int(ids[k]):k for k in range(len(ids))}", " sel=list(code_ids); tot=sum(int(ntok[row[i]]) for i in sel)", " g=np.where(keep)[0]"]}], "userModified": false, "replaceAll": false}

BW=0.0 T=0.05 dedup=True code=0: 29688 ids ~30.0M -> /tmp/sel_nocode.json
launched no-code-lane ablation pid 5153
remaining: 2115

[stdout]
BW=0.0 T=0.05 dedup=True code=0: 29688 ids ~30.0M -> /tmp/sel_nocode.json
launched no-code-lane ablation pid 5153
remaining: 2115
[metadata] {"noOutputExpected": false}

=== KL results ===
random           KL(sel||tgt)= 0.4653
topk_quality     KL(sel||tgt)= 0.3306
gated_uniform    KL(sel||tgt)= 0.3357
lr_tilt          KL(sel||tgt)= 0.3504
=== no-code ablation ===
(with code-lane = 297.5)
remaining: 2015

[stdout]
=== KL results ===
random           KL(sel||tgt)= 0.4653
topk_quality     KL(sel||tgt)= 0.3306
gated_uniform    KL(sel||tgt)= 0.3357
lr_tilt          KL(sel||tgt)= 0.3504
=== no-code ablation ===
(with code-lane = 297.5)
remaining: 2015
[metadata] {"noOutputExpected": false}

=== full KL(sel||tgt) ===
random           KL(sel||tgt)= 0.4653
topk_quality     KL(sel||tgt)= 0.3306
gated_uniform    KL(sel||tgt)= 0.3357
lr_tilt          KL(sel||tgt)= 0.3504
lr_tilt_dedup    KL(sel||tgt)= 0.3177
=== no-code ablation ===
[NOCODE] HELDOUT_PPL 320.429  loss 5.7697  train_s 55  /tmp/tr_NOCODE.npy
(with code-lane=297.5)
remaining: 1925

[stdout]
=== full KL(sel||tgt) ===
random           KL(sel||tgt)= 0.4653
topk_quality     KL(sel||tgt)= 0.3306
gated_uniform    KL(sel||tgt)= 0.3357
lr_tilt          KL(sel||tgt)= 0.3504
lr_tilt_dedup    KL(sel||tgt)= 0.3177
=== no-code ablation ===
[NOCODE] HELDOUT_PPL 320.429  loss 5.7697  train_s 55  /tmp/tr_NOCODE.npy
(with code-lane=297.5)
remaining: 1925
[metadata] {"noOutputExpected": false}

The file /tmp/kl.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/tmp/kl.py", "oldString": "def kl(csel):\n    Ns=sum(csel.values()); V=50257; a=1.0\n    # KL(P_sel || P_tgt) over union, add-a smoothed\n    keys=set(csel)|set(tgt)\n    k=0.0\n    for w in keys:\n        p=(csel.get(w,0)+a)/(Ns+a*V)\n        q=(tgt.get(w,0)+a)/(Tt+a*V)\n        if p>0: k+=p*math.log(p/q)\n    return k\n\nfor tag,path in [(\"random\",\"/tmp/random_sel.json\"),\n                 (\"topk_quality\",\"/tmp/sel_topk.json\"),\n                 (\"gated_uniform\",\"/tmp/sel_uniform.json\"),\n                 (\"lr_tilt\",\"/tmp/sel_lr005.json\"),\n                 (\"lr_tilt_dedup\",\"/workspace/submission/selection.json\")]:\n    try:\n        print(f\"{tag:16s} KL(sel||tgt)= {kl(sel_unigram(path)):.4f}\", flush=True)\n    except Exception as e:\n        print(f\"{tag}: {e}\", flush=True)", "newString": "def metrics(csel):\n    Ns=sum(csel.values()); V=50257; a=1.0\n    keys=set(csel)|set(tgt)\n    kl_st=0.0; H_ts=0.0   # KL(sel||tgt) and cross-entropy H(P_tgt, P_sel) [nats]\n    for w in keys:\n        p=(csel.get(w,0)+a)/(Ns+a*V)      # P_sel\n        q=(tgt.get(w,0)+a)/(Tt+a*V)       # P_tgt\n        kl_st+=p*math.log(p/q)\n        H_ts += q*math.log(1.0/p)\n    return kl_st, H_ts\n\nfor tag,path in [(\"random\",\"/tmp/random_sel.json\"),\n                 (\"topk_quality\",\"/tmp/sel_topk.json\"),\n                 (\"gated_uniform\",\"/tmp/sel_uniform.json\"),\n                 (\"lr_tilt\",\"/tmp/sel_lr005.json\"),\n                 (\"lr_tilt_dedup\",\"/workspace/submission/selection.json\")]:\n    try:\n        k,h=metrics(sel_unigram(path))\n        print(f\"{tag:16s} KL(sel||tgt)= {k:.4f}   H(tgt;sel)= {h:.4f}\", flush=True)\n    except Exception as e:\n        print(f\"{tag}: {e}\", flush=True)", "originalFile": "\"\"\"Mechanistic observable (NOT ppl): KL divergence between a selection's token-unigram\ndistribution and the target's, over the ~12M tokens actually trained on.\nPrediction: KL ranks random > gated-uniform > lr-tilt, mirroring held-out ppl.\"\"\"\nimport json, numpy as np, math\nfrom collections import Counter\nfrom transformers import AutoTokenizer\ntok=AutoTokenizer.from_pretrained(\"gpt2\"); EOS=50256\nBUDGET=12_000_000\n\n# target token-unigram distribution\ndev=np.load(\"/workspace/data/multi_dev.npy\")\ntgt=Counter(int(x) for x in dev if x!=EOS)\nTt=sum(tgt.values())\n\n# load pool text once\ntext={}\nfor line in open(\"/workspace/data/pool.jsonl\"):\n    r=json.loads(line); text[r[\"id\"]]=r[\"text\"]\n\ndef sel_unigram(path):\n    sel=json.load(open(path)); c=Counter(); tot=0\n    for i in sel:\n        ids=tok(text[i], add_special_tokens=False).input_ids\n        c.update(ids); tot+=len(ids)+1\n        if tot>=BUDGET: break\n    return c\n\ndef kl(csel):\n    Ns=sum(csel.values()); V=50257; a=1.0\n    # KL(P_sel || P_tgt) over union, add-a smoothed\n    keys=set(csel)|set(tgt)\n    k=0.0\n    for w in keys:\n        p=(csel.get(w,0)+a)/(Ns+a*V)\n        q=(tgt.get(w,0)+a)/(Tt+a*V)\n        if p>0: k+=p*math.log(p/q)\n    return k\n\nfor tag,path in [(\"random\",\"/tmp/random_sel.json\"),\n                 (\"topk_quality\",\"/tmp/sel_topk.json\"),\n                 (\"gated_uniform\",\"/tmp/sel_uniform.json\"),\n                 (\"lr_tilt\",\"/tmp/sel_lr005.json\"),\n                 (\"lr_tilt_dedup\",\"/workspace/submission/selection.json\")]:\n    try:\n        print(f\"{tag:16s} KL(sel||tgt)= {kl(sel_unigram(path)):.4f}\", flush=True)\n    except Exception as e:\n        print(f\"{tag}: {e}\", flush=True)\n", "structuredPatch": [{"oldStart": 25, "oldLines": 16, "newStart": 25, "newLines": 16, "lines": ["         if tot>=BUDGET: break", "     return c", " ", "-def kl(csel):", "+def metrics(csel):", "     Ns=sum(csel.values()); V=50257; a=1.0", "-    # KL(P_sel || P_tgt) over union, add-a smoothed", "     keys=set(csel)|set(tgt)", "-    k=0.0", "+    kl_st=0.0; H_ts=0.0   # KL(sel||tgt) and cross-entropy H(P_tgt, P_sel) [nats]", "     for w in keys:", "-        p=(csel.get(w,0)+a)/(Ns+a*V)", "-        q=(tgt.get(w,0)+a)/(Tt+a*V)", "-        if p>0: k+=p*math.log(p/q)", "-    return k", "+        p=(csel.get(w,0)+a)/(Ns+a*V)      # P_sel", "+        q=(tgt.get(w,0)+a)/(Tt+a*V)       # P_tgt", "+        kl_st+=p*math.log(p/q)", "+        H_ts += q*math.log(1.0/p)", "+    return kl_st, H_ts", " ", " for tag,path in [(\"random\",\"/tmp/random_sel.json\"),", "                  (\"topk_quality\",\"/tmp/sel_topk.json\"),"]}, {"oldStart": 42, "oldLines": 6, "newStart": 42, "newLines": 7, "lines": ["                  (\"lr_tilt\",\"/tmp/sel_lr005.json\"),", "                  (\"lr_tilt_dedup\",\"/workspace/submission/selection.json\")]:", "     try:", "-        print(f\"{tag:16s} KL(sel||tgt)= {kl(sel_unigram(path)):.4f}\", flush=True)", "+        k,h=metrics(sel_unigram(path))", "+        print(f\"{tag:16s} KL(sel||tgt)= {k:.4f}   H(tgt;sel)= {h:.4f}\", flush=True)", "     except Exception as e:", "         print(f\"{tag}: {e}\", flush=True)"]}], "userModified": false, "replaceAll": false}

kl2 pid 5457
remaining: 1895

[stdout]
kl2 pid 5457
remaining: 1895
[metadata] {"noOutputExpected": false}

Updated task #4 status

[metadata] {"success": true, "taskId": "4", "updatedFields": ["status"], "statusChange": {"from": "in_progress", "to": "completed"}}

Task #5 created successfully: Finalize deliverables & honest claim.md

[metadata] {"task": {"id": "5", "subject": "Finalize deliverables & honest claim.md"}}

=== KL + cross-entropy ===
random           KL(sel||tgt)= 0.4653   H(tgt;sel)= 8.2969
topk_quality     KL(sel||tgt)= 0.3306   H(tgt;sel)= 8.2760
gated_uniform    KL(sel||tgt)= 0.3357   H(tgt;sel)= 8.2000
lr_tilt          KL(sel||tgt)= 0.3504   H(tgt;sel)= 8.1391
remaining: 1755

[stdout]
=== KL + cross-entropy ===
random           KL(sel||tgt)= 0.4653   H(tgt;sel)= 8.2969
topk_quality     KL(sel||tgt)= 0.3306   H(tgt;sel)= 8.2760
gated_uniform    KL(sel||tgt)= 0.3357   H(tgt;sel)= 8.2000
lr_tilt          KL(sel||tgt)= 0.3504   H(tgt;sel)= 8.1391
remaining: 1755
[metadata] {"noOutputExpected": false}

=== final KL+CE ===
random           KL(sel||tgt)= 0.4653   H(tgt;sel)= 8.2969
topk_quality     KL(sel||tgt)= 0.3306   H(tgt;sel)= 8.2760
gated_uniform    KL(sel||tgt)= 0.3357   H(tgt;sel)= 8.2000
lr_tilt          KL(sel||tgt)= 0.3504   H(tgt;sel)= 8.1391
lr_tilt_dedup    KL(sel||tgt)= 0.3177   H(tgt;sel)= 8.1354
remaining: 1675

[stdout]
=== final KL+CE ===
random           KL(sel||tgt)= 0.4653   H(tgt;sel)= 8.2969
topk_quality     KL(sel||tgt)= 0.3306   H(tgt;sel)= 8.2760
gated_uniform    KL(sel||tgt)= 0.3357   H(tgt;sel)= 8.2000
lr_tilt          KL(sel||tgt)= 0.3504   H(tgt;sel)= 8.1391
lr_tilt_dedup    KL(sel||tgt)= 0.3177   H(tgt;sel)= 8.1354
remaining: 1675
[metadata] {"noOutputExpected": false}