claude-opus-5

patched
gate v1: 82137 docs, 77.3M tokens
per-register tokens: {0: 3.0, 1: 3.0, 2: 3.01, 3: 3.0}  head total 12.01M
gate v1s: 78877 docs, 71.0M tokens
per-register tokens: {0: 3.02, 1: 3.0, 2: 3.0, 3: 3.0}  head total 12.02M
gate v1: 82137 docs, 77.3M tokens
per-register tokens: {0: 3.0, 1: 3.0, 2: 3.01, 3: 3.0}  head total 12.01M

[stdout]
patched
gate v1: 82137 docs, 77.3M tokens
per-register tokens: {0: 3.0, 1: 3.0, 2: 3.01, 3: 3.0}  head total 12.01M
gate v1s: 78877 docs, 71.0M tokens
per-register tokens: {0: 3.02, 1: 3.0, 2: 3.0, 3: 3.0}  head total 12.02M
gate v1: 82137 docs, 77.3M tokens
per-register tokens: {0: 3.0, 1: 3.0, 2: 3.01, 3: 3.0}  head total 12.01M
[metadata] {"noOutputExpected": false}

v1eq 12000000
F_dedup 12000000
D_rand 12000000
ok

[stdout]
v1eq 12000000
F_dedup 12000000
D_rand 12000000
ok
[metadata] {"noOutputExpected": false}

D_rand   HELDOUT_PPL 382.040  loss 5.9455

E_gate2  HELDOUT_PPL 340.500  loss 5.8304

F_dedup  HELDOUT_PPL 316.657  loss 5.7578

2897

[stdout]
D_rand   HELDOUT_PPL 382.040  loss 5.9455

E_gate2  HELDOUT_PPL 340.500  loss 5.8304

F_dedup  HELDOUT_PPL 316.657  loss 5.7578

2897
[metadata] {"noOutputExpected": false}

82137 82137 1 182015

[stdout]
82137 82137 1 182015
[metadata] {"noOutputExpected": false}

ok
per-register tokens: {0: 3.0, 1: 3.0, 2: 3.01, 3: 3.0}  head total 12.01M
per-register tokens: {0: 4.2, 1: 2.64, 2: 2.64, 3: 2.52}  head total 12.00M
gate v1d: 82281 docs, 77.5M tokens
per-register tokens: {0: 3.0, 1: 3.0, 2: 3.01, 3: 3.0}  head total 12.01M

[stdout]
ok
per-register tokens: {0: 3.0, 1: 3.0, 2: 3.01, 3: 3.0}  head total 12.01M
per-register tokens: {0: 4.2, 1: 2.64, 2: 2.64, 3: 2.52}  head total 12.00M
gate v1d: 82281 docs, 77.5M tokens
per-register tokens: {0: 3.0, 1: 3.0, 2: 3.01, 3: 3.0}  head total 12.01M
[metadata] {"noOutputExpected": false}

launched
2807

[stdout]
launched
2807
[metadata] {"noOutputExpected": false}

File created successfully at: /workspace/bgscore.py (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/workspace/bgscore.py", "content": "\"\"\"Bigram-LM likelihood-ratio scorer.\n\nThe Naive-Bayes score ignores word order, which is exactly the blind spot SEO\nword-salad exploits. Here we score each document by its average per-token\nlog-ratio under an interpolated *bigram* LM fitted on register k versus one\nfitted on the raw pool:\n\n    s_k(doc) = mean_i [ log P_k(w_i | w_{i-1}) - log P_pool(w_i | w_{i-1}) ]\n\nPunctuation is kept as its own token so surface style still counts.\n\"\"\"\nimport pickle, re, numpy as np, math\nfrom collections import Counter\nfrom multiprocessing import Pool\nfrom transformers import AutoTokenizer\n\nTOKRE = re.compile(r\"[a-z']+|[0-9]+|[^\\sa-z0-9]\")\nDOMAINS = [\"wiki\", \"web\", \"news\", \"qa\"]\nVMAX = 60000\nFEAT_CHARS = 2500\n\nids, texts = pickle.load(open(\"/tmp/pool.pkl\", \"rb\"))\nN = len(texts)\n\ntk = AutoTokenizer.from_pretrained(\"gpt2\")\na = np.load(\"/workspace/data/multi_dev.npy\")\nq = len(a) // 4\ndev_tok = {d: TOKRE.findall(tk.decode(a[k * q:(k + 1) * q].tolist()).lower())\n           for k, d in enumerate(DOMAINS)}\nprint({d: len(v) for d, v in dev_tok.items()}, flush=True)\n\nrng = np.random.default_rng(0)\nneg_idx = rng.permutation(N)[:12000]\nneg_tok = []\nfor i in neg_idx:\n    neg_tok.append(TOKRE.findall(texts[i][:FEAT_CHARS].lower()))\n\n# shared vocabulary\ncnt = Counter()\nfor v in dev_tok.values():\n    cnt.update(v)\nfor v in neg_tok:\n    cnt.update(v)\nvocab = {w: j for j, (w, _) in enumerate(cnt.most_common(VMAX))}\nV = len(vocab) + 1\nUNK = V - 1\nprint(\"vocab\", V, flush=True)\n\n\ndef enc(toks):\n    g = vocab.get\n    return np.array([g(t, UNK) for t in toks], dtype=np.int64)\n\n\ndef fit(seqs):\n    \"\"\"Interpolated bigram: 0.7 * c(v,w)/c(v) + 0.3 * unigram, as log dicts.\"\"\"\n    uni = np.ones(V)\n    bi = Counter()\n    ctx = np.zeros(V)\n    for s in seqs:\n        if len(s) < 2:\n            continue\n        np.add.at(uni, s, 1)\n        key = s[:-1] * V + s[1:]\n        bi.update(key.tolist())\n        np.add.at(ctx, s[:-1], 1)\n    p_uni = uni / uni.sum()\n    return bi, ctx, p_uni\n\n\ndef logp(bi, ctx, p_uni, s):\n    \"\"\"Vectorised: 0.7*bigram MLE + 0.3*unigram.\"\"\"\n    key = (s[:-1] * V + s[1:]).tolist()\n    g = bi.get\n    c = np.fromiter((g(k, 0) for k in key), dtype=np.float64, count=len(key))\n    denom = ctx[s[:-1]]\n    pb = np.divide(c, denom, out=np.zeros_like(c), where=denom > 0)\n    return np.log(0.7 * pb + 0.3 * p_uni[s[1:]])\n\n\nMODELS = {}\nMODELS[\"pool\"] = fit([enc(t) for t in neg_tok])\nfor d in DOMAINS:\n    s = enc(dev_tok[d])\n    MODELS[d] = fit([s])\nprint(\"fitted\", flush=True)\n\n\ndef work(rng_idx):\n    out = np.zeros((len(rng_idx), 4), dtype=np.float32)\n    pb, pc, pu = MODELS[\"pool\"]\n    for r, i in enumerate(rng_idx):\n        s = enc(TOKRE.findall(texts[i][:FEAT_CHARS].lower()))\n        if len(s) < 20:\n            continue\n        base = logp(pb, pc, pu, s)\n        for k, d in enumerate(DOMAINS):\n            b, c, u = MODELS[d]\n            out[r, k] = (logp(b, c, u, s) - base).mean()\n    return rng_idx, out\n\n\nA = np.zeros((N, 4), dtype=np.float32)\nchunks = [list(range(N))[i::16] for i in range(16)]\nwith Pool(16) as p:\n    for rr, out in p.imap_unordered(work, chunks):\n        A[rr] = out\nnp.save(\"/tmp/bgscores.npy\", A)\nprint(\"done\", A.mean(0).round(3), np.percentile(A.max(1), [50, 90, 99]).round(3))\n", "structuredPatch": [], "originalFile": null, "userModified": false}

G_dd8      
H_ddwiki   
I_ddloose  
--- bg:
vocab 60001
fitted
done [-1.788 -1.275 -1.53  -2.217] [-0.983 -0.481 -0.016]

[stdout]
G_dd8      
H_ddwiki   
I_ddloose  
--- bg:
vocab 60001
fitted
done [-1.788 -1.275 -1.53  -2.217] [-0.983 -0.481 -0.016]
[metadata] {"noOutputExpected": false}

G_dd8      HELDOUT_PPL 314.042  loss 5.7495

H_ddwiki   HELDOUT_PPL 320.953  loss 5.7713

I_ddloose  HELDOUT_PPL 322.428  loss 5.7759

2557

[stdout]
G_dd8      HELDOUT_PPL 314.042  loss 5.7495

H_ddwiki   HELDOUT_PPL 320.953  loss 5.7713

I_ddloose  HELDOUT_PPL 322.428  loss 5.7759

2557
[metadata] {"noOutputExpected": false}

ok
per-register tokens: {0: 3.0, 1: 3.0, 2: 3.0, 3: 3.0}  head total 12.00M
per-register tokens: {0: 3.0, 1: 3.0, 2: 3.0, 3: 3.0}  head total 12.00M
per-register tokens: {0: 3.0, 1: 3.0, 2: 3.0, 3: 3.0}  head total 12.00M
launched

[stdout]
ok
per-register tokens: {0: 3.0, 1: 3.0, 2: 3.0, 3: 3.0}  head total 12.00M
per-register tokens: {0: 3.0, 1: 3.0, 2: 3.0, 3: 3.0}  head total 12.00M
per-register tokens: {0: 3.0, 1: 3.0, 2: 3.0, 3: 3.0}  head total 12.00M
launched
[metadata] {"noOutputExpected": false}

waiting
2397

[stdout]
waiting
2397
[metadata] {"noOutputExpected": false}

J_dd16   HELDOUT_PPL 320.398  loss 5.7696

K_bg     HELDOUT_PPL 328.219  loss 5.7937

L_comb   HELDOUT_PPL 316.975  loss 5.7588

2287

[stdout]
J_dd16   HELDOUT_PPL 320.398  loss 5.7696

K_bg     HELDOUT_PPL 328.219  loss 5.7937

L_comb   HELDOUT_PPL 316.975  loss 5.7588

2287
[metadata] {"noOutputExpected": false}

ok
per-register tokens: {0: 3.0, 1: 3.0, 2: 3.0, 3: 3.0}  head total 12.00M
per-register tokens: {0: 3.0, 1: 3.01, 2: 3.0, 3: 3.0}  head total 12.01M
per-register tokens: {0: 3.0, 1: 3.0, 2: 3.0, 3: 3.0}  head total 12.00M
launched

[stdout]
ok
per-register tokens: {0: 3.0, 1: 3.0, 2: 3.0, 3: 3.0}  head total 12.00M
per-register tokens: {0: 3.0, 1: 3.01, 2: 3.0, 3: 3.0}  head total 12.01M
per-register tokens: {0: 3.0, 1: 3.0, 2: 3.0, 3: 3.0}  head total 12.00M
launched
[metadata] {"noOutputExpected": false}

The file /workspace/submission/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"type": "update", "filePath": "/workspace/submission/curate.py", "content": "#!/usr/bin/env python3\n\"\"\"Curate /workspace/data/pool.jsonl into a priority-ordered selection for a\n12M-GPT-2-token pretraining budget.\n\nSTATED CRITERION\n================\nThe disclosed target is a BROAD high-quality English mixture in four equal parts\n(encyclopedic / general HQ web prose / news / technical Q&A). Its dev sample,\n`data/multi_dev.npy`, is GPT-2 token ids, so it can be decoded back to reference\ntext and used as the *only* source of supervision. Three stages, in the order\ntheir measured contribution demands:\n\n1. PROSE GATE (largest effect: 485 -> 382 dev PPL).\n   Drop everything that is not connected English prose -- nav menus, link farms,\n   repeated-line templates, keyword-stuffed pages, non-Latin fragments, log and\n   API dumps. Cheap surface statistics only. Deliberately *not* stricter than\n   this: tightening it (extra line-structure, sentence-length, or bigram-\n   plausibility constraints) was measured to make perplexity WORSE, because at\n   ~6x oversupply the binding constraint is register coverage, not cleanliness.\n\n2. REGISTER-BALANCED RANKING (382 -> 314 dev PPL, together with stage 3).\n   For each target register d, fit a Naive-Bayes log-likelihood ratio over\n   whitespace n-grams,\n\n       s_d(doc) = (1/|doc|) * sum_t [ log P_d(t) - log P_pool(t) ]\n\n   the average per-token evidence that a document came from register d rather\n   than from the raw web pool. Whitespace (not word) tokenisation is the point:\n   it keeps punctuation, markup and spacing artefacts as first-class features, so\n   the score is sensitive to surface form -- which is what a BPE LM actually pays\n   for. Each register then ranks the WHOLE gated pool by its own score (soft\n   assignment) and the budget is filled round-robin with an equal token quota per\n   register. Soft assignment matters: hard argmax assignment starves the scarce\n   registers because the generic-web direction wins the argmax for most of the\n   pool.\n\n3. NEAR-DUPLICATE REMOVAL (~ -8 dev PPL).\n   MinHash LSH over word 5-grams, 8 bands x 2 hashes, applied while filling, so\n   the budget is not spent re-reading the same page. 8 bands was tuned: 4 bands\n   is too permissive (317), 16 too aggressive (320).\n\nOutput is priority order (best first); the pipeline consumes it until 12M tokens.\n\nMeasured on the frozen trainer against data/multi_dev.npy:\n    random selection ................ 485.4\n    this criterion .................. 314.0\n\"\"\"\nimport json, re, hashlib, sys\nfrom collections import Counter\nimport numpy as np\n\nPOOL = \"/workspace/data/pool.jsonl\"\nDEV = \"/workspace/data/multi_dev.npy\"\nOUT = \"/workspace/submission/selection.json\"\n\nDOMAINS = [\"wiki\", \"web\", \"news\", \"qa\"]   # the four equal parts of the target\nFEAT_CHARS = 2500      # chars per doc used for scoring (makes docs comparable)\nN_NEG = 20000          # random pool docs standing in for the \"raw web\" background\nMIN_TOKCOUNT = 5       # n-gram support needed to estimate a ratio\nBUDGET = 12_000_000\nTOK_PER_CHAR = 1 / 3.6  # GPT-2 rate, for quota bookkeeping only\nN_BANDS = 8            # MinHash LSH bands (of 16 hashes total)\nSHINGLE = 5\nSEED = 0\n\n_WS = re.compile(r\"\\S+\")\n_W = re.compile(r\"[a-z']+\")\nSTOP = set(\"the of and to in a is that it for was on as with by are be this from \"\n           \"at or an have has not but they you we he she his her their its will \"\n           \"can which more when there all would about were been if what who\".split())\n\n\n# ----------------------------------------------------------------- featurising\ndef ngrams(text):\n    \"\"\"Whitespace unigrams + adjacent bigrams, lowercased.\"\"\"\n    u = _WS.findall(text.lower())\n    return u + [a + \"\\x00\" + b for a, b in zip(u, u[1:])]\n\n\ndef load_pool():\n    ids, texts = [], []\n    with open(POOL) as f:\n        for line in f:\n            r = json.loads(line)\n            ids.append(r[\"id\"])\n            texts.append(r[\"text\"])\n    return np.array(ids), texts\n\n\ndef load_registers():\n    \"\"\"Decode the dev target and cut it into its four equal registers.\"\"\"\n    from transformers import AutoTokenizer\n    tk = AutoTokenizer.from_pretrained(\"gpt2\")\n    a = np.load(DEV)\n    q = len(a) // len(DOMAINS)\n    out = {}\n    for k, name in enumerate(DOMAINS):\n        txt = tk.decode(a[k * q:(k + 1) * q].tolist())\n        ch = [txt[i:i + FEAT_CHARS] for i in range(0, len(txt), FEAT_CHARS)]\n        out[name] = [c for c in ch if len(c) > 800]\n    return out\n\n\n# ------------------------------------------------- stage 2: register LLR scorer\ndef fit_scorer(registers, neg_texts):\n    neg = Counter()\n    for t in neg_texts:\n        neg.update(ngrams(t[:FEAT_CHARS]))\n    pos = {d: Counter() for d in DOMAINS}\n    for d in DOMAINS:\n        for c in registers[d]:\n            pos[d].update(ngrams(c))\n\n    keep = {t for t, c in neg.items() if c >= MIN_TOKCOUNT}\n    for d in DOMAINS:\n        keep |= {t for t, c in pos[d].items() if c >= MIN_TOKCOUNT}\n    vocab = {t: i for i, t in enumerate(sorted(keep))}\n    V = len(vocab)\n\n    nv = np.ones(V)                       # add-one smoothing\n    for t, c in neg.items():\n        j = vocab.get(t)\n        if j is not None:\n            nv[j] += c\n    logp_neg = np.log(nv / nv.sum())\n\n    W = np.zeros((V, len(DOMAINS)), dtype=np.float32)\n    for k, d in enumerate(DOMAINS):\n        pv = np.ones(V)\n        for t, c in pos[d].items():\n            j = vocab.get(t)\n            if j is not None:\n                pv[j] += c\n        W[:, k] = (np.log(pv / pv.sum()) - logp_neg).astype(np.float32)\n    return vocab, W\n\n\ndef score_all(texts, vocab, W):\n    S = np.zeros((len(texts), len(DOMAINS)), dtype=np.float32)\n    get = vocab.get\n    for i, t in enumerate(texts):\n        idx = [j for j in map(get, ngrams(t[:FEAT_CHARS])) if j is not None]\n        if idx:\n            S[i] = W[idx].mean(0)\n    return S\n\n\n# ------------------------------------------------------- stage 1: prose gate\ndef doc_stats(text):\n    n = len(text)\n    if n == 0:\n        return dict(n=0, stop=0.0, dig=0.0, nonascii=1.0, duplines=1.0, uniq=0.0, alpha=0.0)\n    words = _W.findall(text.lower())\n    nw = max(1, len(words))\n    lines = [l.strip() for l in text.split(\"\\n\") if l.strip()]\n    dup = 1.0 - len(Counter(lines)) / len(lines) if lines else 0.0\n    return dict(\n        n=n,\n        stop=sum(w in STOP for w in words) / nw,\n        dig=sum(ch.isdigit() for ch in text) / n,\n        nonascii=sum(ord(ch) > 127 for ch in text) / n,\n        duplines=dup,\n        uniq=len(set(words)) / nw,\n        alpha=sum(ch.isalpha() for ch in text) / n,\n    )\n\n\ndef is_prose(s):\n    return (s[\"n\"] >= 500 and          # long enough to carry context\n            s[\"stop\"] >= 0.12 and      # real English function-word density\n            s[\"alpha\"] >= 0.55 and     # not a table / link farm / encoded blob\n            s[\"dig\"] <= 0.15 and       # not a statistics dump\n            s[\"nonascii\"] <= 0.10 and  # predominantly English\n            s[\"duplines\"] <= 0.30 and  # not a repeated-line template\n            s[\"uniq\"] >= 0.20)         # not keyword-stuffed\n\n\n# --------------------------------------------- stage 3: MinHash LSH near-dupes\n_MA = np.array([2654435761, 40503, 2246822519, 3266489917, 668265263, 374761393,\n                2147483647, 1000003, 999983, 65537, 257, 131071, 524287, 8191, 61, 97],\n               dtype=np.uint64)\n_MB = np.arange(1, 17, dtype=np.uint64) * np.uint64(2654435769)\n_MP = np.uint64((1 << 61) - 1)\n_WCACHE = {}\n\n\ndef _wh(x):\n    v = _WCACHE.get(x)\n    if v is None:\n        v = _WCACHE[x] = np.uint64(\n            int(hashlib.blake2b(x.encode(), digest_size=8).hexdigest(), 16) % (1 << 61))\n    return v\n\n\ndef lsh_bands(text, k=SHINGLE, nb=N_BANDS):\n    \"\"\"Signature bands over word k-grams; a shared band means near-duplicate.\"\"\"\n    w = _W.findall(text[:20000].lower())\n    if len(w) < k + 2:\n        return []\n    h = np.array([_wh(x) for x in w], dtype=np.uint64)\n    g = np.zeros(len(w) - k + 1, dtype=np.uint64)\n    for j in range(k):\n        g = g * np.uint64(1000003) + h[j:len(h) - k + 1 + j]\n    sig = [int(((_MA[m] * g + _MB[m]) % _MP).min()) for m in range(16)]\n    wid = 16 // nb\n    return [(m, tuple(sig[m * wid:(m + 1) * wid])) for m in range(nb)]\n\n\n# ------------------------------------------------------------------------ main\ndef main():\n    rng = np.random.default_rng(SEED)\n    print(\"loading pool ...\", flush=True)\n    ids, texts = load_pool()\n    N = len(ids)\n\n    print(\"decoding target registers ...\", flush=True)\n    registers = load_registers()\n\n    print(\"fitting register scorers ...\", flush=True)\n    vocab, W = fit_scorer(registers, [texts[i] for i in rng.permutation(N)[:N_NEG]])\n    print(f\"  vocab={len(vocab)}\", flush=True)\n\n    print(\"scoring pool ...\", flush=True)\n    S = score_all(texts, vocab, W)\n\n    print(\"prose gate ...\", flush=True)\n    ok = np.zeros(N, dtype=bool)\n    est_tok = np.zeros(N, dtype=np.int64)\n    for i, t in enumerate(texts):\n        ok[i] = is_prose(doc_stats(t[:20000]))\n        est_tok[i] = max(1, int(len(t) * TOK_PER_CHAR))\n    print(f\"  kept {ok.sum()} / {N} documents\", flush=True)\n\n    # ---- fill an equal token quota per register, best first, skipping near-dupes\n    gated = np.where(ok)[0]\n    ranked = {k: gated[np.argsort(-S[gated, k])] for k in range(len(DOMAINS))}\n    quota = BUDGET / len(DOMAINS)\n    ptr = {k: 0 for k in ranked}\n    per = {k: 0 for k in ranked}\n    order, taken, seen_bands = [], set(), set()\n    print(\"filling register quotas ...\", flush=True)\n    while True:\n        moved = False\n        for k in ranked:\n            if per[k] >= quota:\n                continue\n            r = ranked[k]\n            while ptr[k] < len(r):\n                i = int(r[ptr[k]]); ptr[k] += 1\n                if i in taken:\n                    continue\n                b = lsh_bands(texts[i])\n                if b and any(x in seen_bands for x in b):\n                    taken.add(i)          # near-duplicate of something already taken\n                    continue\n                seen_bands.update(b)\n                taken.add(i); order.append(i)\n                per[k] += int(est_tok[i]); moved = True\n                break\n        if not moved:\n            break\n    print(\"  per-register est tokens:\",\n          {DOMAINS[k]: round(v / 1e6, 2) for k, v in per.items()}, flush=True)\n\n    # ---- tail: next best gated documents overall, so the list comfortably\n    # ---- covers 12M tokens even if the char->token estimate ran optimistic.\n    head = set(order)\n    smax = S.max(1)\n    order += [int(i) for i in np.argsort(-smax) if ok[i] and int(i) not in head]\n\n    sel = [int(ids[i]) for i in order]\n    assert len(sel) == len(set(sel)), \"duplicate ids\"\n    assert all(0 <= x < N for x in sel), \"id out of range\"\n    json.dump(sel, open(OUT, \"w\"))\n    print(f\"wrote {OUT}: {len(sel)} ids, \"\n          f\"{sum(int(est_tok[i]) for i in order) / 1e6:.0f}M est tokens \"\n          f\"({sum(int(est_tok[i]) for i in head) / 1e6:.1f}M in the quota-filled head)\")\n\n\nif __name__ == \"__main__\":\n    main()\n", "structuredPatch": [{"oldStart": 1, "oldLines": 62, "newStart": 1, "newLines": 82, "lines": [" #!/usr/bin/env python3", "-\"\"\"Curate /workspace/data/pool.jsonl for a 12M-token pretraining budget.", "+\"\"\"Curate /workspace/data/pool.jsonl into a priority-ordered selection for a", "+12M-GPT-2-token pretraining budget.", " ", " STATED CRITERION", "-----------------", "-The disclosed evaluation target is a BROAD high-quality English mixture in four", "-equal parts (encyclopedic / general HQ web prose / news / technical Q&A). The dev", "-sample of that target, `data/multi_dev.npy`, is GPT-2 token ids, so it can be", "-decoded back into reference text. We therefore score every pool document by how", "-well its *token distribution matches each target register*, and then fill the", "-budget with an equal token share per register.", "+================", "+The disclosed target is a BROAD high-quality English mixture in four equal parts", "+(encyclopedic / general HQ web prose / news / technical Q&A). Its dev sample,", "+`data/multi_dev.npy`, is GPT-2 token ids, so it can be decoded back to reference", "+text and used as the *only* source of supervision. Three stages, in the order", "+their measured contribution demands:", " ", "-Per register d we fit a Naive-Bayes log-likelihood ratio over whitespace n-grams", "+1. PROSE GATE (largest effect: 485 -> 382 dev PPL).", "+   Drop everything that is not connected English prose -- nav menus, link farms,", "+   repeated-line templates, keyword-stuffed pages, non-Latin fragments, log and", "+   API dumps. Cheap surface statistics only. Deliberately *not* stricter than", "+   this: tightening it (extra line-structure, sentence-length, or bigram-", "+   plausibility constraints) was measured to make perplexity WORSE, because at", "+   ~6x oversupply the binding constraint is register coverage, not cleanliness.", " ", "-    s_d(doc) = (1/|doc|) * sum_t  log P_d(t) - log P_pool(t)", "+2. REGISTER-BALANCED RANKING (382 -> 314 dev PPL, together with stage 3).", "+   For each target register d, fit a Naive-Bayes log-likelihood ratio over", "+   whitespace n-grams,", " ", "-i.e. the average per-token evidence that a document was drawn from register d", "-rather than from the raw web pool. Whitespace tokenisation (rather than word", "-tokenisation) is deliberate: it keeps punctuation, HTML tags (`<p>`, `&quot;`)", "-and wikitext artefacts (`@,@`, `@-@`) as first-class features, so the score is", "-sensitive to surface form as well as topic -- and surface form is a large part", "-of held-out perplexity.", "+       s_d(doc) = (1/|doc|) * sum_t [ log P_d(t) - log P_pool(t) ]", " ", "-Two filters run before ranking:", "-  * a junk gate (length, English-ness, digit/symbol load, line duplication,", "-    token diversity) to drop boilerplate, link farms and gibberish;", "-  * a near-duplicate gate on a 5-gram MinHash sketch, so the budget is not", "-    spent re-reading the same page.", "+   the average per-token evidence that a document came from register d rather", "+   than from the raw web pool. Whitespace (not word) tokenisation is the point:", "+   it keeps punctuation, markup and spacing artefacts as first-class features, so", "+   the score is sensitive to surface form -- which is what a BPE LM actually pays", "+   for. Each register then ranks the WHOLE gated pool by its own score (soft", "+   assignment) and the budget is filled round-robin with an equal token quota per", "+   register. Soft assignment matters: hard argmax assignment starves the scarce", "+   registers because the generic-web direction wins the argmax for most of the", "+   pool.", " ", "-Selection is round-robin across the four registers by *estimated token count*,", "-best-scoring document first, so the training mixture reproduces the target", "-mixture instead of collapsing onto whichever register is easiest to match.", "-Output ordering is priority order: the pipeline consumes it until 12M tokens.", "+3. NEAR-DUPLICATE REMOVAL (~ -8 dev PPL).", "+   MinHash LSH over word 5-grams, 8 bands x 2 hashes, applied while filling, so", "+   the budget is not spent re-reading the same page. 8 bands was tuned: 4 bands", "+   is too permissive (317), 16 too aggressive (320).", "+", "+Output is priority order (best first); the pipeline consumes it until 12M tokens.", "+", "+Measured on the frozen trainer against data/multi_dev.npy:", "+    random selection ................ 485.4", "+    this criterion .................. 314.0", " \"\"\"", "-import json, re, pickle, hashlib, math, os, sys", "+import json, re, hashlib, sys", " from collections import Counter", " import numpy as np", " ", " POOL = \"/workspace/data/pool.jsonl\"", " DEV = \"/workspace/data/multi_dev.npy\"", " OUT = \"/workspace/submission/selection.json\"", "-DOMAINS = [\"wiki\", \"web\", \"news\", \"qa\"]", "-FEAT_CHARS = 2500        # chars of each doc used for scoring (docs are comparable)", "-N_NEG = 20000            # random pool docs acting as the \"raw web\" background", "+", "+DOMAINS = [\"wiki\", \"web\", \"news\", \"qa\"]   # the four equal parts of the target", "+FEAT_CHARS = 2500      # chars per doc used for scoring (makes docs comparable)", "+N_NEG = 20000          # random pool docs standing in for the \"raw web\" background", "+MIN_TOKCOUNT = 5       # n-gram support needed to estimate a ratio", " BUDGET = 12_000_000", "-TOK_PER_CHAR = 1 / 3.6   # rough GPT-2 rate, only used for budget bookkeeping", "+TOK_PER_CHAR = 1 / 3.6  # GPT-2 rate, for quota bookkeeping only", "+N_BANDS = 8            # MinHash LSH bands (of 16 hashes total)", "+SHINGLE = 5", " SEED = 0", " ", "-# ---------------------------------------------------------------- tokenisation", " _WS = re.compile(r\"\\S+\")", "+_W = re.compile(r\"[a-z']+\")", "+STOP = set(\"the of and to in a is that it for was on as with by are be this from \"", "+           \"at or an have has not but they you we he she his her their its will \"", "+           \"can which more when there all would about were been if what who\".split())", " ", " ", "-def toks(text):", "-    \"\"\"Whitespace tokens, lowercased, plus adjacent bigrams.\"\"\"", "+# ----------------------------------------------------------------- featurising", "+def ngrams(text):", "+    \"\"\"Whitespace unigrams + adjacent bigrams, lowercased.\"\"\"", "     u = _WS.findall(text.lower())", "     return u + [a + \"\\x00\" + b for a, b in zip(u, u[1:])]", " ", " ", "-# ------------------------------------------------------------------- load data", " def load_pool():", "     ids, texts = [], []", "     with open(POOL) as f:"]}, {"oldStart": 67, "oldLines": 8, "newStart": 87, "newLines": 8, "lines": ["     return np.array(ids), texts", " ", " ", "-def load_targets():", "-    \"\"\"Decode the dev target and cut it into four equal registers of chunks.\"\"\"", "+def load_registers():", "+    \"\"\"Decode the dev target and cut it into its four equal registers.\"\"\"", "     from transformers import AutoTokenizer", "     tk = AutoTokenizer.from_pretrained(\"gpt2\")", "     a = np.load(DEV)"]}, {"oldStart": 81, "oldLines": 25, "newStart": 101, "newLines": 23, "lines": ["     return out", " ", " ", "-# --------------------------------------------------------- naive-bayes scoring", "-def fit_scorer(dev_doms, neg_texts):", "-    \"\"\"Return (vocab dict, weight matrix (V,4)) of per-token log-likelihood ratios.\"\"\"", "+# ------------------------------------------------- stage 2: register LLR scorer", "+def fit_scorer(registers, neg_texts):", "     neg = Counter()", "     for t in neg_texts:", "-        neg.update(toks(t[:FEAT_CHARS]))", "+        neg.update(ngrams(t[:FEAT_CHARS]))", "     pos = {d: Counter() for d in DOMAINS}", "     for d in DOMAINS:", "-        for c in dev_doms[d]:", "-            pos[d].update(toks(c))", "+        for c in registers[d]:", "+            pos[d].update(ngrams(c))", " ", "-    # vocabulary: tokens with enough support to estimate a ratio at all", "-    keep = {t for t, c in neg.items() if c >= 5}", "+    keep = {t for t, c in neg.items() if c >= MIN_TOKCOUNT}", "     for d in DOMAINS:", "-        keep |= {t for t, c in pos[d].items() if c >= 5}", "+        keep |= {t for t, c in pos[d].items() if c >= MIN_TOKCOUNT}", "     vocab = {t: i for i, t in enumerate(sorted(keep))}", "     V = len(vocab)", " ", "-    nv = np.ones(V)  # add-one smoothing", "+    nv = np.ones(V)                       # add-one smoothing", "     for t, c in neg.items():", "         j = vocab.get(t)", "         if j is not None:"]}, {"oldStart": 118, "oldLines": 23, "newStart": 136, "newLines": 16, "lines": [" ", " ", " def score_all(texts, vocab, W):", "-    \"\"\"Average per-token LLR of every document against each register.\"\"\"", "     S = np.zeros((len(texts), len(DOMAINS)), dtype=np.float32)", "     get = vocab.get", "     for i, t in enumerate(texts):", "-        idx = [j for j in map(get, toks(t[:FEAT_CHARS])) if j is not None]", "+        idx = [j for j in map(get, ngrams(t[:FEAT_CHARS])) if j is not None]", "         if idx:", "             S[i] = W[idx].mean(0)", "     return S", " ", " ", "-# ------------------------------------------------------------------ junk gate", "-STOP = set(\"the of and to in a is that it for was on as with by are be this from \"", "-           \"at or an have has not but they you we he she his her their its will \"", "-           \"can which more when there all would about were been if what who\".split())", "-_W = re.compile(r\"[a-z']+\")", "-", "-", "+# ------------------------------------------------------- stage 1: prose gate", " def doc_stats(text):", "     n = len(text)", "     if n == 0:"]}, {"oldStart": 142, "oldLines": 10, "newStart": 153, "newLines": 7, "lines": ["     words = _W.findall(text.lower())", "     nw = max(1, len(words))", "     lines = [l.strip() for l in text.split(\"\\n\") if l.strip()]", "-    dup = 0.0", "-    if lines:", "-        c = Counter(lines)", "-        dup = 1.0 - len(c) / len(lines)", "+    dup = 1.0 - len(Counter(lines)) / len(lines) if lines else 0.0", "     return dict(", "         n=n,", "         stop=sum(w in STOP for w in words) / nw,"]}, {"oldStart": 157, "oldLines": 109, "newStart": 165, "newLines": 116, "lines": ["     )", " ", " ", "-def passes(s):", "-    \"\"\"Keep documents that look like connected, English, non-boilerplate prose.\"\"\"", "-    return (s[\"n\"] >= 500 and", "-            s[\"stop\"] >= 0.12 and          # real English function-word density", "-            s[\"alpha\"] >= 0.55 and         # not a table / link farm / base64 blob", "-            s[\"dig\"] <= 0.15 and", "-            s[\"nonascii\"] <= 0.10 and", "-            s[\"duplines\"] <= 0.30 and      # not a repeated-line template", "-            s[\"uniq\"] >= 0.20)             # not a keyword-stuffed loop", "+def is_prose(s):", "+    return (s[\"n\"] >= 500 and          # long enough to carry context", "+            s[\"stop\"] >= 0.12 and      # real English function-word density", "+            s[\"alpha\"] >= 0.55 and     # not a table / link farm / encoded blob", "+            s[\"dig\"] <= 0.15 and       # not a statistics dump", "+            s[\"nonascii\"] <= 0.10 and  # predominantly English", "+            s[\"duplines\"] <= 0.30 and  # not a repeated-line template", "+            s[\"uniq\"] >= 0.20)         # not keyword-stuffed", " ", " ", "-# ------------------------------------------------------------------ near-dupes", "-def sketch(text, k=5, nperm=8):", "-    \"\"\"Tiny MinHash over word 5-grams; equal sketches => near-duplicate.\"\"\"", "-    w = _W.findall(text.lower())[:400]", "-    if len(w) < k:", "-        return None", "-    grams = {\" \".join(w[i:i + k]) for i in range(len(w) - k + 1)}", "-    hs = np.array([int(hashlib.blake2b(g.encode(), digest_size=8).hexdigest(), 16)", "-                   for g in grams], dtype=np.uint64)", "-    if len(hs) == 0:", "-        return None", "-    step = max(1, len(hs) // nperm)", "-    return tuple(np.sort(hs)[:nperm:step][:4].tolist())", "+# --------------------------------------------- stage 3: MinHash LSH near-dupes", "+_MA = np.array([2654435761, 40503, 2246822519, 3266489917, 668265263, 374761393,", "+                2147483647, 1000003, 999983, 65537, 257, 131071, 524287, 8191, 61, 97],", "+               dtype=np.uint64)", "+_MB = np.arange(1, 17, dtype=np.uint64) * np.uint64(2654435769)", "+_MP = np.uint64((1 << 61) - 1)", "+_WCACHE = {}", " ", " ", "-# ----------------------------------------------------------------------- main", "+def _wh(x):", "+    v = _WCACHE.get(x)", "+    if v is None:", "+        v = _WCACHE[x] = np.uint64(", "+            int(hashlib.blake2b(x.encode(), digest_size=8).hexdigest(), 16) % (1 << 61))", "+    return v", "+", "+", "+def lsh_bands(text, k=SHINGLE, nb=N_BANDS):", "+    \"\"\"Signature bands over word k-grams; a shared band means near-duplicate.\"\"\"", "+    w = _W.findall(text[:20000].lower())", "+    if len(w) < k + 2:", "+        return []", "+    h = np.array([_wh(x) for x in w], dtype=np.uint64)", "+    g = np.zeros(len(w) - k + 1, dtype=np.uint64)", "+    for j in range(k):", "+        g = g * np.uint64(1000003) + h[j:len(h) - k + 1 + j]", "+    sig = [int(((_MA[m] * g + _MB[m]) % _MP).min()) for m in range(16)]", "+    wid = 16 // nb", "+    return [(m, tuple(sig[m * wid:(m + 1) * wid])) for m in range(nb)]", "+", "+", "+# ------------------------------------------------------------------------ main", " def main():", "     rng = np.random.default_rng(SEED)", "     print(\"loading pool ...\", flush=True)", "     ids, texts = load_pool()", "     N = len(ids)", " ", "-    print(\"decoding target ...\", flush=True)", "-    dev_doms = load_targets()", "+    print(\"decoding target registers ...\", flush=True)", "+    registers = load_registers()", " ", "-    neg_idx = rng.permutation(N)[:N_NEG]", "     print(\"fitting register scorers ...\", flush=True)", "-    vocab, W = fit_scorer(dev_doms, [texts[i] for i in neg_idx])", "+    vocab, W = fit_scorer(registers, [texts[i] for i in rng.permutation(N)[:N_NEG]])", "     print(f\"  vocab={len(vocab)}\", flush=True)", " ", "     print(\"scoring pool ...\", flush=True)", "     S = score_all(texts, vocab, W)", " ", "-    print(\"computing quality stats ...\", flush=True)", "+    print(\"prose gate ...\", flush=True)", "     ok = np.zeros(N, dtype=bool)", "     est_tok = np.zeros(N, dtype=np.int64)", "     for i, t in enumerate(texts):", "-        st = doc_stats(t[:20000])", "-        ok[i] = passes(st)", "+        ok[i] = is_prose(doc_stats(t[:20000]))", "         est_tok[i] = max(1, int(len(t) * TOK_PER_CHAR))", "-    print(f\"  passed junk gate: {ok.sum()} / {N}\", flush=True)", "+    print(f\"  kept {ok.sum()} / {N} documents\", flush=True)", " ", "-    np.save(\"/tmp/scores.npy\", S)", "-    np.save(\"/tmp/ok.npy\", ok)", "-    np.save(\"/tmp/est_tok.npy\", est_tok)", "-", "-    # ---- assign each document to the register it matches best, then take the", "-    # ---- best documents of each register until its token share is filled.", "-    best = S.argmax(1)", "-    share = BUDGET / len(DOMAINS)", "-    seen, order = set(), []", "-    per = {d: 0 for d in DOMAINS}", "-    ranked = {}", "-    for k, d in enumerate(DOMAINS):", "-        cand = np.where(ok & (best == k))[0]", "-        ranked[d] = cand[np.argsort(-S[cand, k])]", "-", "-    ptr = {d: 0 for d in DOMAINS}", "-    # round-robin so the head of the list is already domain-balanced", "+    # ---- fill an equal token quota per register, best first, skipping near-dupes", "+    gated = np.where(ok)[0]", "+    ranked = {k: gated[np.argsort(-S[gated, k])] for k in range(len(DOMAINS))}", "+    quota = BUDGET / len(DOMAINS)", "+    ptr = {k: 0 for k in ranked}", "+    per = {k: 0 for k in ranked}", "+    order, taken, seen_bands = [], set(), set()", "+    print(\"filling register quotas ...\", flush=True)", "     while True:", "         moved = False", "-        for k, d in enumerate(DOMAINS):", "-            if per[d] >= share:", "+        for k in ranked:", "+            if per[k] >= quota:", "                 continue", "-            r = ranked[d]", "-            while ptr[d] < len(r):", "-                i = int(r[ptr[d]]); ptr[d] += 1", "-                sk = sketch(texts[i])", "-                if sk is not None and sk in seen:", "+            r = ranked[k]", "+            while ptr[k] < len(r):", "+                i = int(r[ptr[k]]); ptr[k] += 1", "+                if i in taken:", "                     continue", "-                if sk is not None:", "-                    seen.add(sk)", "-                order.append(i)", "-                per[d] += int(est_tok[i])", "-                moved = True", "+                b = lsh_bands(texts[i])", "+                if b and any(x in seen_bands for x in b):", "+                    taken.add(i)          # near-duplicate of something already taken", "+                    continue", "+                seen_bands.update(b)", "+                taken.add(i); order.append(i)", "+                per[k] += int(est_tok[i]); moved = True", "                 break", "         if not moved:", "             break", "-    print(\"per-register est tokens:\", per, flush=True)", "+    print(\"  per-register est tokens:\",", "+          {DOMAINS[k]: round(v / 1e6, 2) for k, v in per.items()}, flush=True)", " ", "-    # ---- tail: keep filling with the next best documents overall so the list", "-    # ---- comfortably covers the budget even if token estimates were optimistic", "+    # ---- tail: next best gated documents overall, so the list comfortably", "+    # ---- covers 12M tokens even if the char->token estimate ran optimistic.", "     head = set(order)", "     smax = S.max(1)", "-    rest = [i for i in np.argsort(-smax) if ok[i] and i not in head]", "-    order += rest[:60000]", "+    order += [int(i) for i in np.argsort(-smax) if ok[i] and int(i) not in head]", " ", "     sel = [int(ids[i]) for i in order]", "-    assert len(sel) == len(set(sel))", "+    assert len(sel) == len(set(sel)), \"duplicate ids\"", "+    assert all(0 <= x < N for x in sel), \"id out of range\"", "     json.dump(sel, open(OUT, \"w\"))", "     print(f\"wrote {OUT}: {len(sel)} ids, \"", "-          f\"est {sum(int(est_tok[i]) for i in order) / 1e6:.1f}M tokens\")", "+          f\"{sum(int(est_tok[i]) for i in order) / 1e6:.0f}M est tokens \"", "+          f\"({sum(int(est_tok[i]) for i in head) / 1e6:.1f}M in the quota-filled head)\")", " ", " ", " if __name__ == \"__main__\":"]}], "originalFile": "#!/usr/bin/env python3\n\"\"\"Curate /workspace/data/pool.jsonl for a 12M-token pretraining budget.\n\nSTATED CRITERION\n----------------\nThe disclosed evaluation target is a BROAD high-quality English mixture in four\nequal parts (encyclopedic / general HQ web prose / news / technical Q&A). The dev\nsample of that target, `data/multi_dev.npy`, is GPT-2 token ids, so it can be\ndecoded back into reference text. We therefore score every pool document by how\nwell its *token distribution matches each target register*, and then fill the\nbudget with an equal token share per register.\n\nPer register d we fit a Naive-Bayes log-likelihood ratio over whitespace n-grams\n\n    s_d(doc) = (1/|doc|) * sum_t  log P_d(t) - log P_pool(t)\n\ni.e. the average per-token evidence that a document was drawn from register d\nrather than from the raw web pool. Whitespace tokenisation (rather than word\ntokenisation) is deliberate: it keeps punctuation, HTML tags (`<p>`, `&quot;`)\nand wikitext artefacts (`@,@`, `@-@`) as first-class features, so the score is\nsensitive to surface form as well as topic -- and surface form is a large part\nof held-out perplexity.\n\nTwo filters run before ranking:\n  * a junk gate (length, English-ness, digit/symbol load, line duplication,\n    token diversity) to drop boilerplate, link farms and gibberish;\n  * a near-duplicate gate on a 5-gram MinHash sketch, so the budget is not\n    spent re-reading the same page.\n\nSelection is round-robin across the four registers by *estimated token count*,\nbest-scoring document first, so the training mixture reproduces the target\nmixture instead of collapsing onto whichever register is easiest to match.\nOutput ordering is priority order: the pipeline consumes it until 12M tokens.\n\"\"\"\nimport json, re, pickle, hashlib, math, os, sys\nfrom collections import Counter\nimport numpy as np\n\nPOOL = \"/workspace/data/pool.jsonl\"\nDEV = \"/workspace/data/multi_dev.npy\"\nOUT = \"/workspace/submission/selection.json\"\nDOMAINS = [\"wiki\", \"web\", \"news\", \"qa\"]\nFEAT_CHARS = 2500        # chars of each doc used for scoring (docs are comparable)\nN_NEG = 20000            # random pool docs acting as the \"raw web\" background\nBUDGET = 12_000_000\nTOK_PER_CHAR = 1 / 3.6   # rough GPT-2 rate, only used for budget bookkeeping\nSEED = 0\n\n# ---------------------------------------------------------------- tokenisation\n_WS = re.compile(r\"\\S+\")\n\n\ndef toks(text):\n    \"\"\"Whitespace tokens, lowercased, plus adjacent bigrams.\"\"\"\n    u = _WS.findall(text.lower())\n    return u + [a + \"\\x00\" + b for a, b in zip(u, u[1:])]\n\n\n# ------------------------------------------------------------------- load data\ndef load_pool():\n    ids, texts = [], []\n    with open(POOL) as f:\n        for line in f:\n            r = json.loads(line)\n            ids.append(r[\"id\"])\n            texts.append(r[\"text\"])\n    return np.array(ids), texts\n\n\ndef load_targets():\n    \"\"\"Decode the dev target and cut it into four equal registers of chunks.\"\"\"\n    from transformers import AutoTokenizer\n    tk = AutoTokenizer.from_pretrained(\"gpt2\")\n    a = np.load(DEV)\n    q = len(a) // len(DOMAINS)\n    out = {}\n    for k, name in enumerate(DOMAINS):\n        txt = tk.decode(a[k * q:(k + 1) * q].tolist())\n        ch = [txt[i:i + FEAT_CHARS] for i in range(0, len(txt), FEAT_CHARS)]\n        out[name] = [c for c in ch if len(c) > 800]\n    return out\n\n\n# --------------------------------------------------------- naive-bayes scoring\ndef fit_scorer(dev_doms, neg_texts):\n    \"\"\"Return (vocab dict, weight matrix (V,4)) of per-token log-likelihood ratios.\"\"\"\n    neg = Counter()\n    for t in neg_texts:\n        neg.update(toks(t[:FEAT_CHARS]))\n    pos = {d: Counter() for d in DOMAINS}\n    for d in DOMAINS:\n        for c in dev_doms[d]:\n            pos[d].update(toks(c))\n\n    # vocabulary: tokens with enough support to estimate a ratio at all\n    keep = {t for t, c in neg.items() if c >= 5}\n    for d in DOMAINS:\n        keep |= {t for t, c in pos[d].items() if c >= 5}\n    vocab = {t: i for i, t in enumerate(sorted(keep))}\n    V = len(vocab)\n\n    nv = np.ones(V)  # add-one smoothing\n    for t, c in neg.items():\n        j = vocab.get(t)\n        if j is not None:\n            nv[j] += c\n    logp_neg = np.log(nv / nv.sum())\n\n    W = np.zeros((V, len(DOMAINS)), dtype=np.float32)\n    for k, d in enumerate(DOMAINS):\n        pv = np.ones(V)\n        for t, c in pos[d].items():\n            j = vocab.get(t)\n            if j is not None:\n                pv[j] += c\n        W[:, k] = (np.log(pv / pv.sum()) - logp_neg).astype(np.float32)\n    return vocab, W\n\n\ndef score_all(texts, vocab, W):\n    \"\"\"Average per-token LLR of every document against each register.\"\"\"\n    S = np.zeros((len(texts), len(DOMAINS)), dtype=np.float32)\n    get = vocab.get\n    for i, t in enumerate(texts):\n        idx = [j for j in map(get, toks(t[:FEAT_CHARS])) if j is not None]\n        if idx:\n            S[i] = W[idx].mean(0)\n    return S\n\n\n# ------------------------------------------------------------------ junk gate\nSTOP = set(\"the of and to in a is that it for was on as with by are be this from \"\n           \"at or an have has not but they you we he she his her their its will \"\n           \"can which more when there all would about were been if what who\".split())\n_W = re.compile(r\"[a-z']+\")\n\n\ndef doc_stats(text):\n    n = len(text)\n    if n == 0:\n        return dict(n=0, stop=0.0, dig=0.0, nonascii=1.0, duplines=1.0, uniq=0.0, alpha=0.0)\n    words = _W.findall(text.lower())\n    nw = max(1, len(words))\n    lines = [l.strip() for l in text.split(\"\\n\") if l.strip()]\n    dup = 0.0\n    if lines:\n        c = Counter(lines)\n        dup = 1.0 - len(c) / len(lines)\n    return dict(\n        n=n,\n        stop=sum(w in STOP for w in words) / nw,\n        dig=sum(ch.isdigit() for ch in text) / n,\n        nonascii=sum(ord(ch) > 127 for ch in text) / n,\n        duplines=dup,\n        uniq=len(set(words)) / nw,\n        alpha=sum(ch.isalpha() for ch in text) / n,\n    )\n\n\ndef passes(s):\n    \"\"\"Keep documents that look like connected, English, non-boilerplate prose.\"\"\"\n    return (s[\"n\"] >= 500 and\n            s[\"stop\"] >= 0.12 and          # real English function-word density\n            s[\"alpha\"] >= 0.55 and         # not a table / link farm / base64 blob\n            s[\"dig\"] <= 0.15 and\n            s[\"nonascii\"] <= 0.10 and\n            s[\"duplines\"] <= 0.30 and      # not a repeated-line template\n            s[\"uniq\"] >= 0.20)             # not a keyword-stuffed loop\n\n\n# ------------------------------------------------------------------ near-dupes\ndef sketch(text, k=5, nperm=8):\n    \"\"\"Tiny MinHash over word 5-grams; equal sketches => near-duplicate.\"\"\"\n    w = _W.findall(text.lower())[:400]\n    if len(w) < k:\n        return None\n    grams = {\" \".join(w[i:i + k]) for i in range(len(w) - k + 1)}\n    hs = np.array([int(hashlib.blake2b(g.encode(), digest_size=8).hexdigest(), 16)\n                   for g in grams], dtype=np.uint64)\n    if len(hs) == 0:\n        return None\n    step = max(1, len(hs) // nperm)\n    return tuple(np.sort(hs)[:nperm:step][:4].tolist())\n\n\n# ----------------------------------------------------------------------- main\ndef main():\n    rng = np.random.default_rng(SEED)\n    print(\"loading pool ...\", flush=True)\n    ids, texts = load_pool()\n    N = len(ids)\n\n    print(\"decoding target ...\", flush=True)\n    dev_doms = load_targets()\n\n    neg_idx = rng.permutation(N)[:N_NEG]\n    print(\"fitting register scorers ...\", flush=True)\n    vocab, W = fit_scorer(dev_doms, [texts[i] for i in neg_idx])\n    print(f\"  vocab={len(vocab)}\", flush=True)\n\n    print(\"scoring pool ...\", flush=True)\n    S = score_all(texts, vocab, W)\n\n    print(\"computing quality stats ...\", flush=True)\n    ok = np.zeros(N, dtype=bool)\n    est_tok = np.zeros(N, dtype=np.int64)\n    for i, t in enumerate(texts):\n        st = doc_stats(t[:20000])\n        ok[i] = passes(st)\n        est_tok[i] = max(1, int(len(t) * TOK_PER_CHAR))\n    print(f\"  passed junk gate: {ok.sum()} / {N}\", flush=True)\n\n    np.save(\"/tmp/scores.npy\", S)\n    np.save(\"/tmp/ok.npy\", ok)\n    np.save(\"/tmp/est_tok.npy\", est_tok)\n\n    # ---- assign each document to the register it matches best, then take the\n    # ---- best documents of each register until its token share is filled.\n    best = S.argmax(1)\n    share = BUDGET / len(DOMAINS)\n    seen, order = set(), []\n    per = {d: 0 for d in DOMAINS}\n    ranked = {}\n    for k, d in enumerate(DOMAINS):\n        cand = np.where(ok & (best == k))[0]\n        ranked[d] = cand[np.argsort(-S[cand, k])]\n\n    ptr = {d: 0 for d in DOMAINS}\n    # round-robin so the head of the list is already domain-balanced\n    while True:\n        moved = False\n        for k, d in enumerate(DOMAINS):\n            if per[d] >= share:\n                continue\n            r = ranked[d]\n            while ptr[d] < len(r):\n                i = int(r[ptr[d]]); ptr[d] += 1\n                sk = sketch(texts[i])\n                if sk is not None and sk in seen:\n                    continue\n                if sk is not None:\n                    seen.add(sk)\n                order.append(i)\n                per[d] += int(est_tok[i])\n                moved = True\n                break\n        if not moved:\n            break\n    print(\"per-register est tokens:\", per, flush=True)\n\n    # ---- tail: keep filling with the next best documents overall so the list\n    # ---- comfortably covers the budget even if token estimates were optimistic\n    head = set(order)\n    smax = S.max(1)\n    rest = [i for i in np.argsort(-smax) if ok[i] and i not in head]\n    order += rest[:60000]\n\n    sel = [int(ids[i]) for i in order]\n    assert len(sel) == len(set(sel))\n    json.dump(sel, open(OUT, \"w\"))\n    print(f\"wrote {OUT}: {len(sel)} ids, \"\n          f\"est {sum(int(est_tok[i]) for i in order) / 1e6:.1f}M tokens\")\n\n\nif __name__ == \"__main__\":\n    main()\n", "userModified": false}

M_trim    
N_lenpri  
P_pure    
2166

[stdout]
M_trim    
N_lenpri  
P_pure    
2166
[metadata] {"noOutputExpected": false}

M_trim    
N_lenpri  
P_pure

[stdout]
M_trim    
N_lenpri  
P_pure
[metadata] {"noOutputExpected": false}

M_trim    HELDOUT_PPL 361.046  loss 5.8890

N_lenpri  HELDOUT_PPL 331.612  loss 5.8040

P_pure    HELDOUT_PPL 319.206  loss 5.7658

1966

[stdout]
M_trim    HELDOUT_PPL 361.046  loss 5.8890

N_lenpri  HELDOUT_PPL 331.612  loss 5.8040

P_pure    HELDOUT_PPL 319.206  loss 5.7658

1966
[metadata] {"noOutputExpected": false}

chars/token over gated docs: median 4.42 mean 4.41 p10 3.84 p90 4.98

[stdout]
chars/token over gated docs: median 4.42 mean 4.41 p10 3.84 p90 4.98
[metadata] {"noOutputExpected": false}

The file /workspace/submission/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/submission/curate.py", "oldString": "TOK_PER_CHAR = 1 / 3.6  # GPT-2 rate, for quota bookkeeping only", "newString": "TOK_PER_CHAR = 1 / 4.42  # measured GPT-2 rate on the gated pool (median chars/token)", "originalFile": null, "structuredPatch": [{"oldStart": 58, "oldLines": 7, "newStart": 58, "newLines": 7, "lines": [" N_NEG = 20000          # random pool docs standing in for the \"raw web\" background", " MIN_TOKCOUNT = 5       # n-gram support needed to estimate a ratio", " BUDGET = 12_000_000", "-TOK_PER_CHAR = 1 / 3.6  # GPT-2 rate, for quota bookkeeping only", "+TOK_PER_CHAR = 1 / 4.42  # measured GPT-2 rate on the gated pool (median chars/token)", " N_BANDS = 8            # MinHash LSH bands (of 16 hashes total)", " SHINGLE = 5", " SEED = 0"]}], "userModified": false, "replaceAll": false}

loading pool ...
decoding target registers ...
fitting register scorers ...
  vocab=202108
scoring pool ...
prose gate ...

[stdout]
loading pool ...
decoding target registers ...
fitting register scorers ...
  vocab=202108
scoring pool ...
prose gate ...
[metadata] {"noOutputExpected": false}

loading pool ...
decoding target registers ...
fitting register scorers ...
  vocab=202108
scoring pool ...
prose gate ...
  kept 143696 / 182016 documents
filling register quotas ...
  per-register est tokens: {'wiki': 3.0, 'web': 3.0, 'news': 3.0, 'qa': 3.0}
wrote /workspace/submission/selection.json: 143696 ids, 130M est tokens (12.0M in the quota-filled head)
1696

[stdout]
loading pool ...
decoding target registers ...
fitting register scorers ...
  vocab=202108
scoring pool ...
prose gate ...
  kept 143696 / 182016 documents
filling register quotas ...
  per-register est tokens: {'wiki': 3.0, 'web': 3.0, 'news': 3.0, 'qa': 3.0}
wrote /workspace/submission/selection.json: 143696 ids, 130M est tokens (12.0M in the quota-filled head)
1696
[metadata] {"noOutputExpected": false}

packed 12000000 tokens from 143696 ids -> /tmp/dev_train.npy
HELDOUT_PPL 318.907  loss 5.7649  train_s 54  /tmp/dev_train.npy
(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)

[stdout]
packed 12000000 tokens from 143696 ids -> /tmp/dev_train.npy
HELDOUT_PPL 318.907  loss 5.7649  train_s 54  /tmp/dev_train.npy
(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)
[metadata] {"noOutputExpected": false}

backed up

[stdout]
backed up
[metadata] {"noOutputExpected": false}

The file /workspace/submission/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/submission/curate.py", "oldString": "    # ---- fill an equal token quota per register, best first, skipping near-dupes\n    gated = np.where(ok)[0]\n    ranked = {k: gated[np.argsort(-S[gated, k])] for k in range(len(DOMAINS))}", "newString": "    # ---- exact GPT-2 token counts for the documents that can actually be\n    # ---- selected, so the quotas are in real tokens rather than estimated ones\n    gated = np.where(ok)[0]\n    ranked = {k: gated[np.argsort(-S[gated, k])] for k in range(len(DOMAINS))}\n    cand = sorted({int(i) for k in ranked for i in ranked[k][:TOPK_EXACT]})\n    print(f\"exact token counts for {len(cand)} candidates ...\", flush=True)\n    from transformers import AutoTokenizer\n    _tk = AutoTokenizer.from_pretrained(\"gpt2\")\n    for s in range(0, len(cand), 256):\n        blk = cand[s:s + 256]\n        for i, e in zip(blk, _tk([texts[j][:200000] for j in blk],\n                                 add_special_tokens=False)[\"input_ids\"]):\n            est_tok[i] = len(e) + 1        # +1 for the EOS the packer appends", "originalFile": null, "structuredPatch": [{"oldStart": 231, "oldLines": 9, "newStart": 231, "newLines": 19, "lines": ["         est_tok[i] = max(1, int(len(t) * TOK_PER_CHAR))", "     print(f\"  kept {ok.sum()} / {N} documents\", flush=True)", " ", "-    # ---- fill an equal token quota per register, best first, skipping near-dupes", "+    # ---- exact GPT-2 token counts for the documents that can actually be", "+    # ---- selected, so the quotas are in real tokens rather than estimated ones", "     gated = np.where(ok)[0]", "     ranked = {k: gated[np.argsort(-S[gated, k])] for k in range(len(DOMAINS))}", "+    cand = sorted({int(i) for k in ranked for i in ranked[k][:TOPK_EXACT]})", "+    print(f\"exact token counts for {len(cand)} candidates ...\", flush=True)", "+    from transformers import AutoTokenizer", "+    _tk = AutoTokenizer.from_pretrained(\"gpt2\")", "+    for s in range(0, len(cand), 256):", "+        blk = cand[s:s + 256]", "+        for i, e in zip(blk, _tk([texts[j][:200000] for j in blk],", "+                                 add_special_tokens=False)[\"input_ids\"]):", "+            est_tok[i] = len(e) + 1        # +1 for the EOS the packer appends", "     quota = BUDGET / len(DOMAINS)", "     ptr = {k: 0 for k in ranked}", "     per = {k: 0 for k in ranked}"]}], "userModified": false, "replaceAll": false}

loading pool ...
decoding target registers ...
fitting register scorers ...
  vocab=202108
scoring pool ...
prose gate ...
1446

[stdout]
loading pool ...
decoding target registers ...
fitting register scorers ...
  vocab=202108
scoring pool ...
prose gate ...
1446
[metadata] {"noOutputExpected": false}

loading pool ...
decoding target registers ...
fitting register scorers ...
  vocab=202108
scoring pool ...
prose gate ...
  kept 143696 / 182016 documents
exact token counts for 33404 candidates ...
Token indices sequence length is longer than the specified maximum sequence length for this model (1048 > 1024). Running this sequence through the model will result in indexing errors
filling register quotas ...
  per-register est tokens: {'wiki': 3.0, 'web': 3.0, 'news': 3.01, 'qa': 3.0}
wrote /workspace/submission/selection.json: 143696 ids, 131M est tokens (12.0M in the quota-filled head)
1325

[stdout]
loading pool ...
decoding target registers ...
fitting register scorers ...
  vocab=202108
scoring pool ...
prose gate ...
  kept 143696 / 182016 documents
exact token counts for 33404 candidates ...
Token indices sequence length is longer than the specified maximum sequence length for this model (1048 > 1024). Running this sequence through the model will result in indexing errors
filling register quotas ...
  per-register est tokens: {'wiki': 3.0, 'web': 3.0, 'news': 3.01, 'qa': 3.0}
wrote /workspace/submission/selection.json: 143696 ids, 131M est tokens (12.0M in the quota-filled head)
1325
[metadata] {"noOutputExpected": false}

packed 12000000 tokens from 143696 ids -> /tmp/dev_train.npy
HELDOUT_PPL 310.521  loss 5.7383  train_s 54  /tmp/dev_train.npy
(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)

[stdout]
packed 12000000 tokens from 143696 ids -> /tmp/dev_train.npy
HELDOUT_PPL 310.521  loss 5.7383  train_s 54  /tmp/dev_train.npy
(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)
[metadata] {"noOutputExpected": false}