claude-opus-5

noise replicate 113970
[4/7] pool background LM
[5/7] per-cluster weights
[6/7] scoring pool
      scored=78355
[7/7] mixture-matched ordering
      gate+dedup=78355 keep_frac=1.0 kept=78355
      emitted 78355 ids (~75.0M est tokens)
wrote /tmp/sel_sg.json

[stdout]
noise replicate 113970
[4/7] pool background LM
[5/7] per-cluster weights
[6/7] scoring pool
      scored=78355
[7/7] mixture-matched ordering
      gate+dedup=78355 keep_frac=1.0 kept=78355
      emitted 78355 ids (~75.0M est tokens)
wrote /tmp/sel_sg.json
[metadata] {"noOutputExpected": false}

best_frac -> /tmp/sel_sg35.json: 27424 ids ~26.1M est tokens
best_frac -> /tmp/sel_sg60.json: 47013 ids ~46.3M est tokens
RESULT sel_rf12 HELDOUT_PPL 405.817  loss 6.0059  train_s 54  /tmp/tr_sel_rf12.npy
RESULT sel_v3 HELDOUT_PPL 338.417  loss 5.8243  train_s 55  /tmp/tr_sel_v3.npy
4509

[stdout]
best_frac -> /tmp/sel_sg35.json: 27424 ids ~26.1M est tokens
best_frac -> /tmp/sel_sg60.json: 47013 ids ~46.3M est tokens
RESULT sel_rf12 HELDOUT_PPL 405.817  loss 6.0059  train_s 54  /tmp/tr_sel_rf12.npy
RESULT sel_v3 HELDOUT_PPL 338.417  loss 5.8243  train_s 55  /tmp/tr_sel_v3.npy
4509
[metadata] {"noOutputExpected": false}

pool=182016 passed_gate=78747
      after dedup=78355
      emitted 41083 ids (~40.0M est tokens)
[1]+  Done                    GATE=strict ORDER=mixture K=6 python submission/curate.py /tmp/sel_sgm.json 2>&1 | grep -E "emitted|passed_gate|dedup"
      cluster 9: share=0.063 tok=2.5M
      cluster 10: share=0.068 tok=2.7M
      cluster 11: share=0.040 tok=1.6M
      emitted 41782 ids (~40.0M est tokens)
      emitted 44020 ids (~40.0M est tokens)
4348

[stdout]
      pool=182016 passed_gate=78747
      after dedup=78355
      emitted 41083 ids (~40.0M est tokens)
[1]+  Done                    GATE=strict ORDER=mixture K=6 python submission/curate.py /tmp/sel_sgm.json 2>&1 | grep -E "emitted|passed_gate|dedup"
      cluster 9: share=0.063 tok=2.5M
      cluster 10: share=0.068 tok=2.7M
      cluster 11: share=0.040 tok=1.6M
      emitted 41782 ids (~40.0M est tokens)
      emitted 44020 ids (~40.0M est tokens)
4348
[metadata] {"noOutputExpected": false}

launched

[stdout]
launched
[metadata] {"noOutputExpected": false}

safety submission installed: 43028 ids, unique+in-range OK

[stdout]
safety submission installed: 43028 ids, unique+in-range OK
[metadata] {"noOutputExpected": false}

File created successfully at: /workspace/claim.md (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/workspace/claim.md", "content": "# Mixture-matched importance selection beats both random data and greedy on-target data\n\n## Hypothesis\n\nFor a fixed, small token budget against a **mixture** target, the quantity that\ndetermines held-out perplexity is not *how on-target the average selected token\nis*, but *whether every component of the mixture is represented*. Concretely:\n\n> Ranking pool documents by a target/pool n-gram density ratio and taking the\n> top-k is **worse than random selection**, because top-k collapses the budget\n> onto whichever single register the pool holds the most high-ratio material in.\n> The *same* density ratio, applied per-register with a token quota proportional\n> to each register's share of the target, is **much better** than random — and\n> once the quota is enforced, *more* aggressive selection monotonically helps.\n\nThe prediction is a non-monotonicity that a \"better filter = better data\" view\ndoes not anticipate: selectivity is harmful without mixture control and\nstrongly helpful with it. Selectivity and balance are not independent knobs;\nbalance is what licenses selectivity.\n\n## Mechanism (and the observable it predicts — not the final perplexity)\n\nThe budget is 12M tokens out of a ~200M-token pool, i.e. only ~6%. A global\nranking is therefore a *hard subset* decision, not a reweighting: the trainer\nconsumes a prefix of the list and never sees the rest. A single per-token log\ndensity ratio is maximised by documents from whichever register happens to be\nlexically closest to the pool-vs-target contrast — here, contemporary news\nprose. So the global top-6% is nearly a single register.\n\nCross-entropy on an equal-parts mixture is the *average* of per-register\ncross-entropies, and that average is dominated by its worst term: a register the\nmodel has never seen costs unboundedly more than an extra-well-served register\nsaves. Hence the mechanism is **per-register starvation**, and it predicts an\nobservable that is not the aggregate score:\n\n> **Observable P1 (per-register cross-entropy spread).** Split the held-out\n> target into its registers. The globally-ranked (unbalanced) selection should\n> show a *larger spread* across registers than the quota-balanced selection —\n> competitive or even better CE on the register it over-served, and markedly\n> worse CE on the starved registers. The quota-balanced selection should lose\n> slightly on the over-served register and win by much more on the starved ones.\n> If the unbalanced selection is uniformly worse on every register, the\n> mechanism is wrong and something else (e.g. plain text quality) explains it.\n\n> **Observable P2 (average density ratio anti-correlates with score).** The\n> unbalanced top-k selection should achieve a *higher* mean per-token target\n> density ratio than the quota-balanced selection, while scoring worse. The\n> selection objective and the outcome move in opposite directions.\n\n### Measured\n\nDev perplexity (frozen `train_nano.py`, 12M tokens, seed 1337, `multi_dev.npy`):\n\n| selection | mixture quota? | selectivity | dev PPL |\n|---|---|---|---|\n| random pool sample (baseline) | – | none (100%) | **458.0** |\n| quality gate + dedup, shuffled | no | 63% | 422.9 |\n| global top-k by density ratio | **no** | ~6% | **544.9** |\n| per-register top-35%, shuffled | yes | 35% | 461.7 |\n| per-register top-12%, shuffled | yes | 12% | 405.8 |\n| per-register strict rank + quota | **yes** | ~6% | **338.4** |\n\nThe two 6%-selectivity rows differ *only* in whether the register quota is\nenforced: 544.9 vs 338.4. That is the claim. And within the quota-enforced\nrows, PPL falls monotonically as selectivity rises (461.7 → 405.8 → 338.4),\nwhereas without the quota, selectivity is catastrophic (458.0 → 544.9).\n\nP1/P2 results are reported in `## Mechanism check` below.\n\n## Falsification\n\nThe claim is wrong if any of these hold:\n\n1. **Balance is not the active ingredient.** Take the quota-balanced selection\n   and re-order it so one register fills the whole 12M prefix, holding the gate,\n   dedup, and scoring function fixed. If perplexity does *not* degrade toward\n   the ~545 range, the win came from something other than mixture matching.\n2. **P1 fails.** If the unbalanced selection is worse on *every* register rather\n   than showing a large spread, starvation is not the mechanism.\n3. **P2 fails.** If the better-scoring selection also has the higher mean\n   density ratio, then the ratio is simply a good quality score and no\n   \"objective vs outcome divergence\" exists.\n4. **Selectivity is not monotone under balance.** If pushing selectivity further\n   under a quota (finer registers, smaller kept fraction) reverses and *raises*\n   perplexity, then the \"balance licenses selectivity\" statement is too strong\n   and there is an interior optimum instead.\n5. **Cluster count is doing the work.** If replacing the discovered k-means\n   registers with K arbitrary random partitions of the target reproduces the\n   gain, then the clusters carry no register information and the effect is just\n   an ensembling/averaging artifact of using K scores instead of one.\n\n## Transfer\n\n- **Any budget-constrained pretraining mix against a known multi-domain\n  evaluation.** The recipe — estimate a density ratio per evaluation component,\n  then fill the budget by component quota rather than by global rank — applies\n  directly to domain-mixture selection (the DoReMi/DSIR setting) and needs only\n  a decodable sample of the target, no labels.\n- **The failure mode transfers more broadly than the fix.** Any pipeline that\n  ranks documents by a single scalar \"quality\" or \"similarity\" score and then\n  keeps a top fraction is exposed to this: the tighter the budget, the more a\n  scalar ranking silently becomes a domain filter. Retrieval corpora, fine-tune\n  set construction, and active-learning pools all have this shape.\n- **Practical rule.** When the keep-fraction is small relative to the pool,\n  always report the *composition* of what was kept, not just the mean score of\n  what was kept. A monotone improvement in mean score with a degrading\n  evaluation is the signature of this bug.\n- **Boundary.** This should weaken as the budget approaches the pool size (top-k\n  stops being a subset decision) and as the target becomes single-register\n  (there is nothing to starve). It should strengthen for smaller budgets and\n  more heterogeneous targets.\n", "structuredPatch": [], "originalFile": null, "userModified": false}

4268

[stdout]
4268
[metadata] {"noOutputExpected": false}

File created successfully at: /workspace/diag_registers.py (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/workspace/diag_registers.py", "content": "\"\"\"Mechanism check P1: per-register held-out cross-entropy.\n\nReplicates the FROZEN train_nano.py recipe exactly (same model, seed, schedule,\niters, batching) but evaluates the resulting model on several held-out slices in\none run instead of one. Nothing about training differs -- this only avoids\nretraining once per slice. The aggregate number it prints is verified against\nthe frozen script's number for the same selection.\n\nRegisters are discovered by clustering the held-out target's own 256-token\nwindows, independently of the K used for selection (so the split is not\ncircular).\n\"\"\"\nimport json, math, os, re, sys, time\nimport numpy as np, torch\nfrom collections import Counter\nfrom model import GPT, GPTConfig\n\nBLOCK, BATCH, ITERS, WARMUP, LR, SEED = 256, 32, 3000, 150, 6e-4, 1337\nKR = 4\nWORD_RE = re.compile(r\"[a-z0-9']+\")\n\n\ndef register_split():\n    \"\"\"Cluster the held-out target's 256-token windows into KR registers.\"\"\"\n    from transformers import AutoTokenizer\n    tok = AutoTokenizer.from_pretrained(\"gpt2\")\n    ho = np.load(\"/workspace/data/multi_dev.npy\").astype(np.int64)\n    nwin = (len(ho) - 1) // BLOCK\n    texts = tok.batch_decode([ho[j * BLOCK:(j + 1) * BLOCK].tolist() for j in range(nwin)])\n    docs = [WORD_RE.findall(t.lower()) for t in texts]\n    df = Counter()\n    for d in docs:\n        df.update(set(d))\n    vocab = [w for w, c in df.most_common(3000) if c >= 3]\n    vi = {w: j for j, w in enumerate(vocab)}\n    X = np.zeros((nwin, len(vocab)), dtype=np.float32)\n    for r, d in enumerate(docs):\n        for w, c in Counter(d).items():\n            j = vi.get(w)\n            if j is not None:\n                X[r, j] = 1.0 + math.log(c)\n    X *= np.log(nwin / (1.0 + np.array([df[w] for w in vocab], dtype=np.float32)))\n    X /= np.linalg.norm(X, axis=1, keepdims=True) + 1e-9\n    rng = np.random.default_rng(0)\n    C = X[rng.choice(nwin, KR, replace=False)].copy()\n    for _ in range(60):\n        lab = (X @ C.T).argmax(1)\n        for c in range(KR):\n            m = lab == c\n            C[c] = X[rng.integers(nwin)] if m.sum() == 0 else \\\n                X[m].mean(0) / (np.linalg.norm(X[m].mean(0)) + 1e-9)\n    # label each window; keep a few sample snippets to name the registers\n    samples = {c: [texts[j][:110] for j in np.where(lab == c)[0][:2]] for c in range(KR)}\n    return ho, lab, nwin, samples\n\n\ndef evaluate(model, ho, starts, dev=\"cuda\"):\n    model.eval()\n    tot, n = 0.0, 0\n    with torch.no_grad():\n        for i in range(0, len(starts), BATCH):\n            bs = starts[i:i + BATCH]\n            x = torch.stack([ho[s:s + BLOCK] for s in bs]).to(dev)\n            y = torch.stack([ho[s + 1:s + 1 + BLOCK] for s in bs]).to(dev)\n            with torch.autocast(\"cuda\", dtype=torch.bfloat16):\n                _, l = model(x, y)\n            tot += l.item() * len(bs); n += len(bs)\n    return tot / max(1, n)\n\n\ndef run(train_npy, ho, lab, nwin):\n    torch.manual_seed(SEED); np.random.seed(SEED)\n    rng = np.random.default_rng(SEED)\n    tr = torch.from_numpy(np.load(train_npy).astype(np.int64))\n    hot = torch.from_numpy(ho)\n    model = GPT(GPTConfig(block_size=BLOCK, vocab_size=50257, n_layer=6, n_head=6,\n                          n_embd=384, dropout=0.0, bias=False)).to(\"cuda\")\n    opt = model.configure_optimizers(0.1, LR, (0.9, 0.95), \"cuda\")\n\n    def lr_at(it):\n        if it < WARMUP: return LR * (it + 1) / (WARMUP + 1)\n        r = (it - WARMUP) / max(1, ITERS - WARMUP)\n        return 0.1 * LR + 0.5 * (1 + math.cos(math.pi * r)) * (LR - 0.1 * LR)\n\n    model.train()\n    for it in range(ITERS):\n        for g in opt.param_groups: g[\"lr\"] = lr_at(it)\n        ix = rng.integers(0, len(tr) - BLOCK - 1, size=BATCH)\n        x = torch.stack([tr[i:i + BLOCK] for i in ix]).to(\"cuda\")\n        y = torch.stack([tr[i + 1:i + 1 + BLOCK] for i in ix]).to(\"cuda\")\n        with torch.autocast(\"cuda\", dtype=torch.bfloat16):\n            _, loss = model(x, y)\n        opt.zero_grad(set_to_none=True); loss.backward()\n        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0); opt.step()\n\n    allstarts = [j * BLOCK for j in range(nwin)]\n    out = {\"all\": evaluate(model, hot, allstarts)}\n    for c in range(KR):\n        st = [j * BLOCK for j in range(nwin) if lab[j] == c]\n        out[f\"reg{c}\"] = evaluate(model, hot, st)\n        out[f\"reg{c}_n\"] = len(st)\n    return out\n\n\nif __name__ == \"__main__\":\n    ho, lab, nwin, samples = register_split()\n    print(\"register sizes:\", np.bincount(lab, minlength=KR).tolist(), flush=True)\n    for c, s in samples.items():\n        print(f\"  reg{c}: {s[0]!r}\", flush=True)\n    res = {}\n    for npy in sys.argv[1:]:\n        name = os.path.basename(npy).replace(\"tr_\", \"\").replace(\".npy\", \"\")\n        r = run(npy, ho, lab, nwin)\n        res[name] = r\n        ppl = {k: round(math.exp(v), 1) for k, v in r.items() if not k.endswith(\"_n\")}\n        print(\"PPL\", name, ppl, flush=True)\n    json.dump(res, open(\"/tmp/diag_registers.json\", \"w\"), indent=1)\n", "structuredPatch": [], "originalFile": null, "userModified": false}

RESULT sel_sgm HELDOUT_PPL 406.521  loss 6.0076  train_s 54  /tmp/tr_sel_sgm.npy
4228

[stdout]
RESULT sel_sgm HELDOUT_PPL 406.521  loss 6.0076  train_s 54  /tmp/tr_sel_sgm.npy
4228
[metadata] {"noOutputExpected": false}

k3       emitted 41524 ids (~40.0M est tokens)
k20       emitted 44795 ids (~40.0M est tokens)
b0       emitted 44045 ids (~40.0M est tokens)
b2       emitted 40314 ids (~40.0M est tokens)
4008

[stdout]
k3       emitted 41524 ids (~40.0M est tokens)
k20       emitted 44795 ids (~40.0M est tokens)
b0       emitted 44045 ids (~40.0M est tokens)
b2       emitted 40314 ids (~40.0M est tokens)
4008
[metadata] {"noOutputExpected": false}

RESULT sel_sgm HELDOUT_PPL 406.521  loss 6.0076  train_s 54  /tmp/tr_sel_sgm.npy
RESULT sel_sgm12 HELDOUT_PPL 385.724  loss 5.9551  train_s 55  /tmp/tr_sel_sgm12.npy
RESULT sel_v3k12 HELDOUT_PPL 329.706  loss 5.7982  train_s 56  /tmp/tr_sel_v3k12.npy
--- t:
4008

[stdout]
RESULT sel_sgm HELDOUT_PPL 406.521  loss 6.0076  train_s 54  /tmp/tr_sel_sgm.npy
RESULT sel_sgm12 HELDOUT_PPL 385.724  loss 5.9551  train_s 55  /tmp/tr_sel_sgm12.npy
RESULT sel_v3k12 HELDOUT_PPL 329.706  loss 5.7982  train_s 56  /tmp/tr_sel_v3k12.npy
--- t:
4008
[metadata] {"noOutputExpected": false}

k30       emitted 45568 ids (~40.0M est tokens)
k16       emitted 44986 ids (~40.0M est tokens)

[stdout]
k30       emitted 45568 ids (~40.0M est tokens)
k16       emitted 44986 ids (~40.0M est tokens)
[metadata] {"noOutputExpected": false}

/tmp/batch2.log:RESULT sel_sgm HELDOUT_PPL 406.521  loss 6.0076  train_s 54  /tmp/tr_sel_sgm.npy
/tmp/batch2.log:RESULT sel_sgm12 HELDOUT_PPL 385.724  loss 5.9551  train_s 55  /tmp/tr_sel_sgm12.npy
/tmp/batch2.log:RESULT sel_v3k12 HELDOUT_PPL 329.706  loss 5.7982  train_s 56  /tmp/tr_sel_v3k12.npy
/tmp/batch2.log:RESULT sel_sg HELDOUT_PPL 488.535  loss 6.1914  train_s 55  /tmp/tr_sel_sg.npy
--- t:
3858

[stdout]
/tmp/batch2.log:RESULT sel_sgm HELDOUT_PPL 406.521  loss 6.0076  train_s 54  /tmp/tr_sel_sgm.npy
/tmp/batch2.log:RESULT sel_sgm12 HELDOUT_PPL 385.724  loss 5.9551  train_s 55  /tmp/tr_sel_sgm12.npy
/tmp/batch2.log:RESULT sel_v3k12 HELDOUT_PPL 329.706  loss 5.7982  train_s 56  /tmp/tr_sel_v3k12.npy
/tmp/batch2.log:RESULT sel_sg HELDOUT_PPL 488.535  loss 6.1914  train_s 55  /tmp/tr_sel_sg.npy
--- t:
3858
[metadata] {"noOutputExpected": false}

installed v3k12 (329.7)
utilization.gpu [%], memory.used [MiB]
97 %, 9537 MiB

[stdout]
installed v3k12 (329.7)
utilization.gpu [%], memory.used [MiB]
97 %, 9537 MiB
[metadata] {"noOutputExpected": false}

launched
3828

[stdout]
launched
3828
[metadata] {"noOutputExpected": false}

RESULT sel_k30 HELDOUT_PPL 332.614  loss 5.8070  train_s 97  /tmp/tr_sel_k30.npy
RESULT sel_gate HELDOUT_PPL 422.876  loss 6.0471  train_s 54  /tmp/tr_sel_gate.npy
RESULT sel_rf35 HELDOUT_PPL 461.740  loss 6.1350  train_s 54  /tmp/tr_sel_rf35.npy
RESULT sel_rf12 HELDOUT_PPL 405.817  loss 6.0059  train_s 54  /tmp/tr_sel_rf12.npy
RESULT sel_v3 HELDOUT_PPL 338.417  loss 5.8243  train_s 55  /tmp/tr_sel_v3.npy
RESULT sel_k20 HELDOUT_PPL 325.883  loss 5.7865  train_s 107  /tmp/tr_sel_k20.npy
RESULT sel_k3 HELDOUT_PPL 341.663  loss 5.8338  train_s 96  /tmp/tr_sel_k3.npy
RESULT sel_sgm HELDOUT_PPL 406.521  loss 6.0076  train_s 54  /tmp/tr_sel_sgm.npy
RESULT sel_sgm12 HELDOUT_PPL 385.724  loss 5.9551  train_s 55  /tmp/tr_sel_sgm12.npy
RESULT sel_v3k12 HELDOUT_PPL 329.706  loss 5.7982  train_s 56  /tmp/tr_sel_v3k12.npy
RESULT sel_sg HELDOUT_PPL 488.535  loss 6.1914  train_s 55  /tmp/tr_sel_sg.npy
RESULT sel_gateB HELDOUT_PPL 399.280  loss 5.9897  train_s 106  /tmp/tr_sel_gateB.npy
--- t:
3607
packed 12000000 tokens from 29315 ids -> /tmp/tr_sel_v1.npy

[stdout]
RESULT sel_k30 HELDOUT_PPL 332.614  loss 5.8070  train_s 97  /tmp/tr_sel_k30.npy
RESULT sel_gate HELDOUT_PPL 422.876  loss 6.0471  train_s 54  /tmp/tr_sel_gate.npy
RESULT sel_rf35 HELDOUT_PPL 461.740  loss 6.1350  train_s 54  /tmp/tr_sel_rf35.npy
RESULT sel_rf12 HELDOUT_PPL 405.817  loss 6.0059  train_s 54  /tmp/tr_sel_rf12.npy
RESULT sel_v3 HELDOUT_PPL 338.417  loss 5.8243  train_s 55  /tmp/tr_sel_v3.npy
RESULT sel_k20 HELDOUT_PPL 325.883  loss 5.7865  train_s 107  /tmp/tr_sel_k20.npy
RESULT sel_k3 HELDOUT_PPL 341.663  loss 5.8338  train_s 96  /tmp/tr_sel_k3.npy
RESULT sel_sgm HELDOUT_PPL 406.521  loss 6.0076  train_s 54  /tmp/tr_sel_sgm.npy
RESULT sel_sgm12 HELDOUT_PPL 385.724  loss 5.9551  train_s 55  /tmp/tr_sel_sgm12.npy
RESULT sel_v3k12 HELDOUT_PPL 329.706  loss 5.7982  train_s 56  /tmp/tr_sel_v3k12.npy
RESULT sel_sg HELDOUT_PPL 488.535  loss 6.1914  train_s 55  /tmp/tr_sel_sg.npy
RESULT sel_gateB HELDOUT_PPL 399.280  loss 5.9897  train_s 106  /tmp/tr_sel_gateB.npy
--- t:
3607
packed 12000000 tokens from 29315 ids -> /tmp/tr_sel_v1.npy
[metadata] {"noOutputExpected": false}

RESULT sel_k30 HELDOUT_PPL 332.614  loss 5.8070  train_s 97  /tmp/tr_sel_k30.npy

[stdout]
RESULT sel_k30 HELDOUT_PPL 332.614  loss 5.8070  train_s 97  /tmp/tr_sel_k30.npy
[metadata] {"noOutputExpected": false}

File created successfully at: /workspace/diag_ratio.py (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/workspace/diag_ratio.py", "content": "\"\"\"Mechanism check P2: mean per-token GLOBAL target density ratio of the 12M\nprefix each selection actually trains on, plus its register composition.\n\nP2 predicts the selection that scores WORSE (global top-k) achieves the HIGHER\nmean density ratio -- i.e. the objective and the outcome diverge.\n\"\"\"\nimport json, math, re, sys\nfrom collections import Counter\nfrom multiprocessing import Pool as MPPool\nimport numpy as np\n\nWORD_RE = re.compile(r\"[a-z0-9']+|[.,;:!?()\\\"\\-]\")\nwords = lambda t: WORD_RE.findall(t.lower())\nBUDGET = 12_000_000\n\n\ndef bgcount(lines):\n    cu, cb = Counter(), Counter()\n    for ln in lines:\n        ws = words(json.loads(ln)[\"text\"])\n        cu.update(ws); cb.update(zip(ws, ws[1:]))\n    return cu, cb\n\n\nif __name__ == \"__main__\":\n    from transformers import AutoTokenizer\n    tok = AutoTokenizer.from_pretrained(\"gpt2\")\n    tgt = tok.decode(np.load(\"/workspace/data/multi_dev.npy\").astype(np.int64).tolist())\n    tws = words(tgt)\n    tu, tb = Counter(tws), Counter(zip(tws, tws[1:]))\n    uni = {k: 1 for k, c in tu.items() if c >= 2}\n    bi = {k: 1 for k, c in tb.items() if c >= 3}\n\n    lines = open(\"/workspace/data/pool.jsonl\").readlines()\n    per = (len(lines) + 63) // 64\n    parts = [lines[i:i + per] for i in range(0, len(lines), per)]\n    with MPPool(16) as p:\n        bgs = p.map(bgcount, [c[::5] for c in parts])\n    pu, pb = Counter(), Counter()\n    for a, b in bgs:\n        pu.update(a); pb.update(b)\n\n    Tu, Tb = sum(tu.values()), sum(tb.values())\n    Pu, Pb = sum(pu.values()), sum(pb.values())\n    WU = {k: math.log(tu[k] / Tu) - math.log((pu.get(k, 0) + 1) / (Pu + len(uni)))\n          for k in uni}\n    WB = {k: math.log(tb[k] / Tb) - math.log((pb.get(k, 0) + 1) / (Pb + len(bi)))\n          for k in bi}\n    t_ou = max(1, sum(c for k, c in tu.items() if k not in uni))\n    t_ob = max(1, sum(c for k, c in tb.items() if k not in bi))\n    OU = math.log(t_ou / Tu) - math.log(max(1, Pu - sum(pu.get(k, 0) for k in uni)) / (Pu + len(uni)))\n    OB = math.log(t_ob / Tb) - math.log(max(1, Pb - sum(pb.get(k, 0) for k in bi)) / (Pb + len(bi)))\n\n    text = {}\n    for ln in lines:\n        r = json.loads(ln); text[r[\"id\"]] = r[\"text\"]\n\n    print(f\"{'selection':16s} {'mean_ratio/tok':>14s} {'docs':>7s} {'distinct_tgt_types':>19s}\")\n    for sp in sys.argv[1:]:\n        sel = json.load(open(sp))\n        tot, s_sum, n_sum, ndoc = 0, 0.0, 0, 0\n        seen = set()\n        for i in sel:\n            t = text[i]\n            est = len(t) / 4.0\n            ws = words(t)\n            n = len(ws)\n            s = sum(WU.get(x, OU) for x in ws) + sum(WB.get(x, OB) for x in zip(ws, ws[1:]))\n            s_sum += s; n_sum += 2 * n; ndoc += 1\n            seen.update(x for x in ws if x in uni)\n            tot += est\n            if tot >= BUDGET:\n                break\n        print(f\"{sp.split('sel_')[-1][:-5]:16s} {s_sum/n_sum:14.4f} {ndoc:7d} \"\n              f\"{len(seen)/len(uni):18.3f}\")\n", "structuredPatch": [], "originalFile": null, "userModified": false}

register sizes: [969, 198, 981, 1758]
  reg0: ' Beardmore was sufficiently impressed with Shackleton to offer financial support , but other donations proved '
  reg1: ' and dreams " , " the veils of history " , " the Kingdom of the Spirit " , " the nymphs of the forest , the sp'
  reg2: ' Explorer in quirks mode, where width and height determine the width and height including padding and borders\n'
  reg3: ' until she finally trips and vomits on the floor at which Peter gets mad with her . When Peter also bullies hi'
RESULT sel_k30 HELDOUT_PPL 332.614  loss 5.8070  train_s 97  /tmp/tr_sel_k30.npy
RESULT sel_k16 HELDOUT_PPL 328.457  loss 5.7944  train_s 96  /tmp/tr_sel_k16.npy
--- t:
3547

[stdout]
register sizes: [969, 198, 981, 1758]
  reg0: ' Beardmore was sufficiently impressed with Shackleton to offer financial support , but other donations proved '
  reg1: ' and dreams " , " the veils of history " , " the Kingdom of the Spirit " , " the nymphs of the forest , the sp'
  reg2: ' Explorer in quirks mode, where width and height determine the width and height including padding and borders\n'
  reg3: ' until she finally trips and vomits on the floor at which Peter gets mad with her . When Peter also bullies hi'
RESULT sel_k30 HELDOUT_PPL 332.614  loss 5.8070  train_s 97  /tmp/tr_sel_k30.npy
RESULT sel_k16 HELDOUT_PPL 328.457  loss 5.7944  train_s 96  /tmp/tr_sel_k16.npy
--- t:
3547
[metadata] {"noOutputExpected": false}

patched
RESULT sel_k20 HELDOUT_PPL 325.883  loss 5.7865  train_s 107  /tmp/tr_sel_k20.npy
RESULT sel_k3 HELDOUT_PPL 341.663  loss 5.8338  train_s 96  /tmp/tr_sel_k3.npy
RESULT sel_b0 HELDOUT_PPL 340.993  loss 5.8319  train_s 93  /tmp/tr_sel_b0.npy

[stdout]
patched
RESULT sel_k20 HELDOUT_PPL 325.883  loss 5.7865  train_s 107  /tmp/tr_sel_k20.npy
RESULT sel_k3 HELDOUT_PPL 341.663  loss 5.8338  train_s 96  /tmp/tr_sel_k3.npy
RESULT sel_b0 HELDOUT_PPL 340.993  loss 5.8319  train_s 93  /tmp/tr_sel_b0.npy
[metadata] {"noOutputExpected": false}

pool=182016 passed_gate=166049
      after dedup=154821
      emitted 40371 ids (~40.0M est tokens)
      emitted 70831 ids (~40.0M est tokens)
3387

[stdout]
      pool=182016 passed_gate=166049
      after dedup=154821
      emitted 40371 ids (~40.0M est tokens)
      emitted 70831 ids (~40.0M est tokens)
3387
[metadata] {"noOutputExpected": false}

PPL sel_v1 {'all': 544.9, 'reg0': 785.5, 'reg1': 624.7, 'reg2': 1331.0, 'reg3': 266.4}
PPL sel_v3k12 {'all': 329.7, 'reg0': 696.8, 'reg1': 446.3, 'reg2': 213.7, 'reg3': 268.7}

[stdout]
PPL sel_v1 {'all': 544.9, 'reg0': 785.5, 'reg1': 624.7, 'reg2': 1331.0, 'reg3': 266.4}
PPL sel_v3k12 {'all': 329.7, 'reg0': 696.8, 'reg1': 446.3, 'reg2': 213.7, 'reg3': 268.7}
[metadata] {"noOutputExpected": false}

selection        mean_ratio/tok    docs  distinct_tgt_types
rand                    -0.2994   11420              0.861
v1                      -0.0060    8671              0.846
gate                    -0.1814   12415              0.856
v3k12                   -0.0303   14630              0.877
k16                     -0.0327   15028              0.877
=== batch5:
RESULT sel_mg16 HELDOUT_PPL 340.488  loss 5.8304  train_s 54  /tmp/tr_sel_mg16.npy
RESULT sel_sh400 HELDOUT_PPL 347.233  loss 5.8500  train_s 55  /tmp/tr_sel_sh400.npy
--- t:
3187

[stdout]
selection        mean_ratio/tok    docs  distinct_tgt_types
rand                    -0.2994   11420              0.861
v1                      -0.0060    8671              0.846
gate                    -0.1814   12415              0.856
v3k12                   -0.0303   14630              0.877
k16                     -0.0327   15028              0.877
=== batch5:
RESULT sel_mg16 HELDOUT_PPL 340.488  loss 5.8304  train_s 54  /tmp/tr_sel_mg16.npy
RESULT sel_sh400 HELDOUT_PPL 347.233  loss 5.8500  train_s 55  /tmp/tr_sel_sh400.npy
--- t:
3187
[metadata] {"noOutputExpected": false}

defaults: K=16 ORDER=mixture GATE=loose
69:K = int(os.environ.get("K", 16))              # target registers to discover
71:W_BONUS = float(os.environ.get("W_BONUS", "0.5"))
72:ORDER = os.environ.get("ORDER", "mixture")   # shuffle | mixture
73:KEEP_FRAC = float(os.environ.get("KEEP_FRAC", "1.0"))
78:SHRINK = float(os.environ.get("SHRINK", "0.0"))
91:GATE = os.environ.get("GATE", "loose")

[stdout]
defaults: K=16 ORDER=mixture GATE=loose
69:K = int(os.environ.get("K", 16))              # target registers to discover
71:W_BONUS = float(os.environ.get("W_BONUS", "0.5"))
72:ORDER = os.environ.get("ORDER", "mixture")   # shuffle | mixture
73:KEEP_FRAC = float(os.environ.get("KEEP_FRAC", "1.0"))
78:SHRINK = float(os.environ.get("SHRINK", "0.0"))
91:GATE = os.environ.get("GATE", "loose")
[metadata] {"noOutputExpected": false}

cluster 6: share=0.034 tok=1.4M
      cluster 7: share=0.054 tok=2.2M
      cluster 8: share=0.044 tok=1.8M
      cluster 9: share=0.041 tok=1.6M
      cluster 10: share=0.063 tok=2.5M
      cluster 11: share=0.060 tok=2.4M
      cluster 12: share=0.043 tok=1.7M
      cluster 13: share=0.075 tok=3.0M
      cluster 14: share=0.057 tok=2.3M
      cluster 15: share=0.094 tok=3.8M
      emitted 44986 ids (~40.0M est tokens)
wrote /workspace/submission/selection.json
reproduces tested k16 selection exactly: True
n= 44986 unique: True in range: True

[stdout]
      cluster 6: share=0.034 tok=1.4M
      cluster 7: share=0.054 tok=2.2M
      cluster 8: share=0.044 tok=1.8M
      cluster 9: share=0.041 tok=1.6M
      cluster 10: share=0.063 tok=2.5M
      cluster 11: share=0.060 tok=2.4M
      cluster 12: share=0.043 tok=1.7M
      cluster 13: share=0.075 tok=3.0M
      cluster 14: share=0.057 tok=2.3M
      cluster 15: share=0.094 tok=3.8M
      emitted 44986 ids (~40.0M est tokens)
wrote /workspace/submission/selection.json
reproduces tested k16 selection exactly: True
n= 44986 unique: True in range: True
[metadata] {"noOutputExpected": false}