claude-opus-5

[1]   Done                    nohup python work/greedy.py --out cand/g_bg200.json --clip 100 --bgsm 1000 --minlen 200 > work/g_bg200.log 2>&1
[3]+  Done                    nohup python work/greedy.py --out cand/g_bg150s.json --clip 100 --bgsm 200 --minlen 150 > work/g_bg150s.log 2>&1
[2]+  Done                    nohup python work/greedy.py --out cand/g_bg100.json --clip 100 --bgsm 1000 --minlen 100 > work/g_bg100.log 2>&1
launched
3167

[stdout]
[1]   Done                    nohup python work/greedy.py --out cand/g_bg200.json --clip 100 --bgsm 1000 --minlen 200 > work/g_bg200.log 2>&1
[3]+  Done                    nohup python work/greedy.py --out cand/g_bg150s.json --clip 100 --bgsm 200 --minlen 150 > work/g_bg150s.log 2>&1
[2]+  Done                    nohup python work/greedy.py --out cand/g_bg100.json --clip 100 --bgsm 1000 --minlen 100 > work/g_bg100.log 2>&1
launched
3167
[metadata] {"noOutputExpected": false}

File created successfully at: /workspace/claim.md (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/workspace/claim.md", "content": "# Curating a raw web pool by target-distribution matching\n\n## Hypothesis\n\nUnder a token budget far smaller than the pool (12M of 200M), held-out perplexity\non a broad multi-domain target is governed mainly by **how closely the training\nsubset's token distribution matches the target's**, and only secondarily by\ngeneric \"document quality\". Concretely:\n\n> Greedily choosing documents to minimise the unigram cross-entropy\n> `H(p_target, p_selection)` — the same functional the metric evaluates, at the\n> highest order estimable from the 1M-token disclosed target — beats both random\n> selection and quality-filtered random selection by a wide margin, and beats\n> single-axis \"pick the best-looking documents\" ranking.\n\nThe corollary that makes this non-trivial: the pool is *not* short of readable\nprose. It is short of prose in the target's **registers and surface\nconventions**, and a budget-constrained selector must buy those deliberately.\n\n## Mechanism\n\nHeld-out loss is a mean over windows of four equal registers (I recovered them by\ndecoding `multi_dev.npy`: WikiText-style encyclopedic / clean web prose /\nnewswire / StackExchange HTML Q&A). Mean loss is therefore additive across\nregisters, so the **highest-loss register dominates the gradient of the score**,\nand a selection that leaves one register unrepresented pays for it disproportionately.\n\nMatching `p_target` allocates budget to registers in proportion to their share of\nthe target rather than their share of the pool. The importance ratio\n`r(v) = p_target(v)/p_selection(v)` is recomputed as the selection grows, so once\na register is covered its tokens stop being attractive and the selector moves on —\ncoverage is self-balancing rather than hand-weighted.\n\n**Predicted observables (not the final perplexity), all measured:**\n\n1. **The proxy objective tracks the real one.** Unigram\n   `H(p_target, p_selection)` falls 8.825 nats (pool background) → 7.854 nats\n   (final selection). Variants with a lower proxy score train to lower dev\n   perplexity; e.g. smoothing `p_target` toward the pool background instead of\n   toward uniform lowers the proxy 8.630 → 7.854 nats and dev PPL 279.3 → 272.9,\n   because uniform smoothing puts ~8% of target mass on ~25k token types the\n   target never emits, and greedy then spends budget chasing that noise.\n2. **Gains land on the register that was starved, not uniformly.** Per-register\n   held-out loss, quality-filtered random vs. matched selection:\n   Q&A **5.777 → 5.038** nats, news 5.870 → 5.630, web prose 5.627 → 5.559,\n   encyclopedic 6.724 → 6.308. The technical-Q&A register — the one the raw pool\n   is emptiest of — improves ~3× more than the register it is richest in.\n3. **The selector buys the target's surface conventions at a measurable rate.**\n   The encyclopedic quarter is WikiText-formatted: GPT-2 tokens `' ,'` and `' .'`\n   (space-before-punctuation) are 7.9% of its tokens but 0.05% of pool tokens.\n   The matched selection puts ~5% of its budget in documents carrying that\n   convention — ~5× the random rate (592k vs 119k tokens) — and its head is 25×\n   denser in those tokens (1276 vs 51 per 100k).\n4. **The encyclopedic register stays the worst one no matter what.** Its\n   convention is nearly absent from the pool, so it remains the highest-loss\n   quarter (6.31 nats vs 5.04–5.66) in every selection I trained. This is a\n   ceiling imposed by the pool, not by the selector.\n\n## Falsification\n\nThe claim predicts specific *failures*, and I ran each as its own training run\n(frozen recipe, seed 1337, dev target). All would have falsified it; none did.\n\n| Selection criterion | dev PPL |\n|---|---|\n| Random (do-nothing baseline) | 485.5 |\n| Quality gate + dedup only, random within it | 403.6 |\n| Single global quality score (LLR vs. target mixture), top-k | 387.0 |\n| Balanced per-register Naive-Bayes round-robin | 312.5 |\n| **Greedy `H(p_target,p_selection)` matching (submitted)** | **267.7** |\n\nFalsifiers that did not fire:\n\n- *If the junk filter were doing the work*, quality-gate-only random selection\n  would approach the matched selection. It does not: the gate buys 485.5 → 403.6,\n  matching buys 403.6 → 267.7. Matching contributes ~1.7× more than filtering.\n- *If a single quality axis sufficed*, ranking every document by its likelihood\n  ratio against the pooled target mixture would match register-aware selection.\n  It does not (387.0 vs 312.5): a single global ranking collapses onto whichever\n  register the pool most resembles.\n- *If the hardest register's surface tokens were the bottleneck*, force-feeding\n  1.5M tokens (12.5% of budget) of the documents richest in `' ,'`/`' .'` would\n  help. It hurt (267.7 → 290.5): those documents are nav-bar and sports-ellipsis\n  boilerplate, and the objective's own ~5% allocation is already the right dose.\n- *If the hardest register deserved extra weight*, doubling the encyclopedic\n  register's weight in `p_target` would help. It hurt (288.8 vs 279.3 at matched\n  settings): the equal-parts target weighting is already correct, as the additive\n  structure of the metric implies.\n- *If longer documents were simply better*, raising the minimum length would\n  help. It hurt (295.4 at 900 tokens vs 267.7 at 150): past ~150 tokens, extra\n  length costs distributional coverage. The optimum is a bracketed interior\n  minimum, not a monotone preference.\n\n**What would still falsify the claim:** a selection with a *higher* proxy\ncross-entropy that trains to lower perplexity; or a quality-only criterion\n(no target sample used) reaching ~268 on this budget. Note the honest limit of\nthe evidence: `p_target` is estimated from the disclosed dev sample, so the\nmatching is only as good as that sample's coverage of the hidden one — the two\nare stated to be disjoint samples of the same domain, and I tuned only five\nscalar hyperparameters against dev, which bounds but does not eliminate\nselection-on-dev risk.\n\n## Transfer\n\nThe method needs only (a) a sample of target-domain text, (b) the tokenizer, and\n(c) one pass to count tokens — no labels, no reference model, no GPU. The greedy\nstep is a gather plus a segment-sum, so scoring the full 200M-token pool costs\n~1 s and the whole selection ~1 min on CPU; it scales to far larger pools because\ncost is linear in tokens and independent of how much is finally kept.\n\nIt transfers to any budget-constrained data decision where the evaluation\ndistribution is known or samplable: choosing pretraining mixtures, picking\nfine-tuning subsets for a known deployment domain, or deciding domain weights\nwithout hand-set ratios. Two transferable lessons generalise beyond this task:\n**smooth the target distribution toward the achievable background rather than\ntoward uniform** (otherwise the selector chases unreachable rare tokens), and\n**clip the importance ratio** (some target tokens cannot be bought at any price,\nand an unclipped objective will spend the whole budget trying). The natural\nextension is to run the same greedy on higher-order statistics as more target\ntext becomes available; the unigram order is a consequence of having only 1M\ntarget tokens, not a limitation of the criterion.\n", "structuredPatch": [], "originalFile": null, "userModified": false}

g_bg200: 
g_bg100: 
g_bg150s: 
3117

[stdout]
g_bg200: 
g_bg100: 
g_bg150s: 
3117
[metadata] {"noOutputExpected": false}

g_bg200: 
g_bg100: 
g_bg150s: 
proxy H(p_t,p_sel) = 7.852730316473987
proxy H(p_t,p_sel) = 7.85631344865526
proxy H(p_t,p_sel) = 7.853342037362712
2997

[stdout]
g_bg200: 
g_bg100: 
g_bg150s: 
proxy H(p_t,p_sel) = 7.852730316473987
proxy H(p_t,p_sel) = 7.85631344865526
proxy H(p_t,p_sel) = 7.853342037362712
2997
[metadata] {"noOutputExpected": false}

g_bg200: HELDOUT_PPL 269.531  loss 5.5967  train_s 161  /tmp/tr_g_bg200.npy

g_bg100: HELDOUT_PPL 267.693  loss 5.5898  train_s 162  /tmp/tr_g_bg100.npy

g_bg150s: HELDOUT_PPL 268.063  loss 5.5912  train_s 162  /tmp/tr_g_bg150s.npy

2947

[stdout]
g_bg200: HELDOUT_PPL 269.531  loss 5.5967  train_s 161  /tmp/tr_g_bg200.npy

g_bg100: HELDOUT_PPL 267.693  loss 5.5898  train_s 162  /tmp/tr_g_bg100.npy

g_bg150s: HELDOUT_PPL 268.063  loss 5.5912  train_s 162  /tmp/tr_g_bg150s.npy

2947
[metadata] {"noOutputExpected": false}

patched
building
2927

[stdout]
patched
building
2927
[metadata] {"noOutputExpected": false}

ls: cannot access 'cand/g_fine.json': No such file or directory
cand/g_cl1k.json
cand/g_loose.json
140 docs 11200 tok 8270088 147.0
selected 17733 docs 15000826 tokens 75.5
selected 18404 docs 15000412 tokens 77.6
2777

[stdout]
ls: cannot access 'cand/g_fine.json': No such file or directory
cand/g_cl1k.json
cand/g_loose.json
140 docs 11200 tok 8270088 147.0
selected 17733 docs 15000826 tokens 75.5
selected 18404 docs 15000412 tokens 77.6
2777
[metadata] {"noOutputExpected": false}

launched2
2767

[stdout]
launched2
2767
[metadata] {"noOutputExpected": false}

Command did not complete within its 120s timeout and was moved to the background (ID: b2dryf5zg). Output is being written to: /tmp/claude-0/-workspace/21ef8abe-2df0-457e-9c7a-a72b69647ba6/tasks/b2dryf5zg.output. You will be notified when it completes. To check interim output, use Read on that file path.
Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "b2dryf5zg", "timedOutAfterMs": 120000, "backgroundCwdHint": "Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands."}

loose:
HELDOUT_PPL 269.228  loss 5.5956  train_s 119  /tmp/tr_g_loose.npy
clip1k:
HELDOUT_PPL 269.020  loss 5.5948  train_s 120  /tmp/tr_g_cl1k.npy
fine:
HELDOUT_PPL 273.415  loss 5.6110  train_s 77  /tmp/tr_g_fine.npy
2537

[stdout]
loose:
HELDOUT_PPL 269.228  loss 5.5956  train_s 119  /tmp/tr_g_loose.npy
clip1k:
HELDOUT_PPL 269.020  loss 5.5948  train_s 120  /tmp/tr_g_cl1k.npy
fine:
HELDOUT_PPL 273.415  loss 5.6110  train_s 77  /tmp/tr_g_fine.npy
2537
[metadata] {"noOutputExpected": false}

The file /workspace/submission/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/submission/curate.py", "oldString": "CLIP     = 100.0   # cap on the per-token importance ratio\nBATCH    = 250     # documents admitted per greedy re-scoring step\nPRIOR    = 3e5     # pseudo-tokens of pool background seeding p_selection\nMIN_LEN  = 300     # tokens; >= block_size(256) so a window can sit inside a doc", "newString": "CLIP     = 100.0   # cap on the per-token importance ratio\nBATCH    = 250     # documents admitted per greedy re-scoring step\nPRIOR    = 3e5     # pseudo-tokens of pool background seeding p_selection\nBGSM     = 1000.0  # pseudo-tokens smoothing p_target toward the pool background\nMIN_LEN  = 150     # min document length in tokens", "originalFile": "#!/usr/bin/env python3\n\"\"\"\nCurate /workspace/data/pool.jsonl down to a 12M-token pretraining set for the\nfrozen 30M-param GPT, targeting low perplexity on a broad multi-domain\nhigh-quality English target (encyclopedic / web prose / news / technical Q&A).\n\nSTATED CRITERION\n----------------\nHeld-out perplexity is exp of the mean next-token cross-entropy on the target\nstream.  We therefore select the subset of the pool whose *own* token\ndistribution is closest to the target's, measuring closeness with the same\nfunctional the metric uses -- cross-entropy -- at unigram order, which is the\nhighest order estimable from the 1M-token disclosed dev target:\n\n    minimise   H(p_target, p_selection) = - sum_v p_target(v) log p_selection(v)\n    over subsets S of the pool with  sum_{d in S} |d| <= budget.\n\nAdding document d moves p_selection along the direction (c_d/|d| - p_selection),\nso the first-order decrease of H from adding d is proportional to\n\n    gain(d) = (1/|d|) sum_{t in d} min( p_target(t) / p_selection(t), CLIP )\n\ni.e. the mean *importance ratio* of d's tokens under the current selection.\nWe run batched greedy on this gain: score every eligible document, admit the\nbest BATCH of them, update p_selection, repeat until the budget is covered.\nThe ratio is clipped because some target tokens are unreachable at any price\n(the encyclopedic quarter is WikiText-formatted: ' ,' / ' .' / ' @-@ ' occur at\n~9% of its tokens and are essentially absent from raw web text); without a clip\nthe objective would spend unbounded budget chasing them.\n\nTwo guards keep the objective honest, since a pure frequency-matching score is\nhappy to buy rare tokens from boilerplate:\n  * quality gate -- minimum length, and caps on 5-gram repetition / top-token\n    share / minimum type-token ratio, which reject nav-bars, link farms and\n    directory listings;\n  * streaming near-duplicate filter -- 4-permutation MinHash over 5-gram\n    shingles; a document sharing >=2 of 4 sketch minima with an already-selected\n    document is dropped, so the budget is not spent re-reading the same page.\n\nDocuments are emitted in the order greedy admitted them, so truncating the list\nat the 12M-token budget preserves the matched mixture.\n\nRun:  python curate.py            (~3 min: tokenise pool, featurise, select)\n\"\"\"\nimport json, os, time\nimport numpy as np\n\nPOOL     = \"/workspace/data/pool.jsonl\"\nDEV      = \"/workspace/data/multi_dev.npy\"      # disclosed dev target\nOUT      = \"/workspace/submission/selection.json\"\nCACHE    = \"/workspace/work\"                    # reused if already present\nV        = 50257\nBUDGET   = 12_000_000\nTARGET   = 15_000_000        # emit ~25% past the budget so truncation is safe\n\n# --- selection hyperparameters (chosen on the disclosed dev target) ---\nCLIP     = 100.0   # cap on the per-token importance ratio\nBATCH    = 250     # documents admitted per greedy re-scoring step\nPRIOR    = 3e5     # pseudo-tokens of pool background seeding p_selection\nMIN_LEN  = 300     # tokens; >= block_size(256) so a window can sit inside a doc\nMAX_REP5 = 0.35    # max fraction of duplicated 5-gram shingles\nMAX_TOP1 = 0.12    # max share of the single most frequent token\nMIN_UNIQ = 0.18    # min type-token ratio\nTWEIGHTS = (1., 1., 1., 1.)   # relative weight of the 4 target registers\n\nK = 5              # shingle length\nP = np.uint64(1099511628211)\n\n\ndef tokenize_pool():\n    \"\"\"GPT-2 BPE over the whole pool -> flat token stream + document offsets.\"\"\"\n    os.environ[\"TOKENIZERS_PARALLELISM\"] = \"true\"\n    from tokenizers import Tokenizer\n    tok = Tokenizer.from_pretrained(\"gpt2\")\n    arrs, lens, ids, bt, bi = [], [], [], [], []\n\n    def flush():\n        if not bt:\n            return\n        for i, e in zip(bi, tok.encode_batch(bt)):\n            arrs.append(np.asarray(e.ids, dtype=np.uint16))\n            lens.append(len(e.ids)); ids.append(i)\n        bt.clear(); bi.clear()\n\n    for line in open(POOL):\n        r = json.loads(line); bt.append(r[\"text\"]); bi.append(r[\"id\"])\n        if len(bt) >= 4096:\n            flush()\n    flush()\n    lens = np.array(lens, dtype=np.int64)\n    off = np.zeros(len(lens) + 1, dtype=np.int64); np.cumsum(lens, out=off[1:])\n    return np.concatenate(arrs), off, lens, np.array(ids, dtype=np.int64)\n\n\ndef features(flat, off, lens):\n    \"\"\"Per-document junk features and MinHash sketches (5-gram shingles).\"\"\"\n    N = len(lens)\n    f = flat.astype(np.uint64)\n    h = np.zeros(len(f) - K + 1, dtype=np.uint64)       # rolling poly hash mod 2^64\n    for k in range(K):\n        h = h * P + f[k:len(f) - K + 1 + k]\n    mixers = [np.uint64(x) for x in (0x9E3779B97F4A7C15, 0xC2B2AE3D27D4EB4F,\n                                     0x165667B19E3779F9, 0x27D4EB2F165667C5)]\n    rep5 = np.zeros(N, np.float32); uniq = np.zeros(N, np.float32)\n    top1 = np.zeros(N, np.float32); sketch = np.zeros((N, 4), np.uint64)\n    dochash = np.zeros(N, np.uint64)\n    for i in range(N):\n        s, e = off[i], off[i + 1]\n        a = f[s:e]\n        if len(a) == 0:\n            continue\n        dh = np.uint64(len(a))\n        for chunk in (a[:32], a[-32:]):\n            for v in chunk:\n                dh = dh * P + v\n        dochash[i] = dh\n        u, c = np.unique(a, return_counts=True)\n        uniq[i] = len(u) / len(a); top1[i] = c.max() / len(a)\n        if e - s >= K + 1:\n            g = np.unique(h[s:e - K + 1])\n            rep5[i] = 1.0 - len(g) / (e - s - K + 1)\n            for j, m in enumerate(mixers):\n                sketch[i, j] = (g * m).min()\n    return rep5, uniq, top1, sketch.astype(np.int64), dochash.astype(np.int64)\n\n\ndef cached():\n    \"\"\"Load cached tokenisation/features if this script already ran once.\"\"\"\n    need = [\"flat\", \"off\", \"lens\", \"docids\", \"rep5\", \"uniq\", \"top1\", \"sketch\", \"dochash\"]\n    if all(os.path.exists(f\"{CACHE}/{n}.npy\") for n in need):\n        return [np.load(f\"{CACHE}/{n}.npy\") for n in need]\n    return None\n\n\ndef main():\n    t0 = time.time()\n    c = cached()\n    if c is None:\n        flat, off, lens, docids = tokenize_pool()\n        print(f\"tokenised {len(lens)} docs, {int(lens.sum())} tokens  {time.time()-t0:.0f}s\")\n        rep5, uniq, top1, sketch, dochash = features(flat, off, lens)\n        os.makedirs(CACHE, exist_ok=True)\n        for n, v in zip([\"flat\", \"off\", \"lens\", \"docids\", \"rep5\", \"uniq\", \"top1\",\n                         \"sketch\", \"dochash\"],\n                        [flat, off, lens, docids, rep5, uniq, top1, sketch, dochash]):\n            np.save(f\"{CACHE}/{n}.npy\", v)\n    else:\n        flat, off, lens, docids, rep5, uniq, top1, sketch, dochash = c\n        print(f\"loaded cached features  {time.time()-t0:.0f}s\")\n    f64 = flat.astype(np.int64)\n\n    # pool background distribution (the \"do nothing\" reference distribution)\n    bg = np.bincount(f64, minlength=V).astype(np.float64)\n    p_bg = (bg + 0.5) / (bg.sum() + 0.5 * V)\n\n    # target unigram: equal-parts mixture of the 4 disclosed registers\n    dev = np.load(DEV).astype(np.int64)\n    Q = len(dev) // 4\n    tw = np.array(TWEIGHTS, dtype=np.float64); tw /= tw.sum()\n    p_t = np.zeros(V)\n    for k in range(4):\n        ck = np.bincount(dev[k * Q:(k + 1) * Q], minlength=V).astype(np.float64)\n        p_t += tw[k] * (ck + 1.0) / (ck.sum() + V)\n\n    ok = ((lens >= MIN_LEN) & (rep5 <= MAX_REP5) &\n          (top1 <= MAX_TOP1) & (uniq >= MIN_UNIQ))\n    print(f\"eligible after quality gate: {int(ok.sum())} docs, \"\n          f\"{int(lens[ok].sum())} tokens\")\n\n    alive = ok.copy()\n    c_sel = p_bg * PRIOR; n_sel = PRIOR\n    sel, tot = [], 0\n    seen_doc, seen_sk = set(), set()\n    while tot < TARGET:\n        r = np.clip(p_t / (c_sel / n_sel), 0, CLIP).astype(np.float32)\n        gain = np.add.reduceat(r[f64].astype(np.float64), off[:-1]) / lens\n        gain[~alive] = -1e9\n        cand = np.argpartition(-gain, BATCH * 3)[:BATCH * 3]\n        cand = cand[np.argsort(-gain[cand])]\n        added = 0\n        for j in cand:\n            j = int(j)\n            if not alive[j]:\n                continue\n            if int(dochash[j]) in seen_doc:\n                alive[j] = False; continue\n            sk = [int(x) for x in sketch[j] if x != 0]\n            if sum(x in seen_sk for x in sk) >= 2:      # near-duplicate\n                alive[j] = False; continue\n            seen_doc.add(int(dochash[j])); seen_sk.update(sk)\n            alive[j] = False\n            sel.append(j); tot += int(lens[j]) + 1\n            c_sel += np.bincount(f64[off[j]:off[j + 1]], minlength=V)\n            n_sel += int(lens[j])\n            added += 1\n            if added >= BATCH or tot >= TARGET:\n                break\n        if added == 0:\n            break\n\n    p_sel = c_sel / n_sel\n    print(f\"proxy cross-entropy H(p_t,p_sel) = {-(p_t*np.log(p_sel)).sum():.4f} nats \"\n          f\"(pool background: {-(p_t*np.log(p_bg)).sum():.4f})\")\n    ids = [int(docids[j]) for j in sel]\n    assert len(ids) == len(set(ids)), \"duplicate ids\"\n    json.dump(ids, open(OUT, \"w\"))\n    print(f\"wrote {len(ids)} ids ({tot} tokens, budget {BUDGET}) -> {OUT}  \"\n          f\"{time.time()-t0:.0f}s\")\n\n\nif __name__ == \"__main__\":\n    main()\n", "structuredPatch": [{"oldStart": 57, "oldLines": 7, "newStart": 57, "newLines": 8, "lines": [" CLIP     = 100.0   # cap on the per-token importance ratio", " BATCH    = 250     # documents admitted per greedy re-scoring step", " PRIOR    = 3e5     # pseudo-tokens of pool background seeding p_selection", "-MIN_LEN  = 300     # tokens; >= block_size(256) so a window can sit inside a doc", "+BGSM     = 1000.0  # pseudo-tokens smoothing p_target toward the pool background", "+MIN_LEN  = 150     # min document length in tokens", " MAX_REP5 = 0.35    # max fraction of duplicated 5-gram shingles", " MAX_TOP1 = 0.12    # max share of the single most frequent token", " MIN_UNIQ = 0.18    # min type-token ratio"]}], "userModified": false, "replaceAll": false}

The file /workspace/submission/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/submission/curate.py", "oldString": "    tw = np.array(TWEIGHTS, dtype=np.float64); tw /= tw.sum()\n    p_t = np.zeros(V)\n    for k in range(4):\n        ck = np.bincount(dev[k * Q:(k + 1) * Q], minlength=V).astype(np.float64)\n        p_t += tw[k] * (ck + 1.0) / (ck.sum() + V)", "newString": "    # Smoothing is toward p_bg, NOT toward uniform: uniform smoothing spreads\n    # ~8% of the target mass over ~25k token types the target never emits, and\n    # greedy then burns budget chasing that noise (costs ~6 dev PPL).\n    tw = np.array(TWEIGHTS, dtype=np.float64); tw /= tw.sum()\n    p_t = np.zeros(V)\n    for k in range(4):\n        ck = np.bincount(dev[k * Q:(k + 1) * Q], minlength=V).astype(np.float64)\n        p_t += tw[k] * (ck + BGSM * p_bg) / (ck.sum() + BGSM)", "originalFile": "#!/usr/bin/env python3\n\"\"\"\nCurate /workspace/data/pool.jsonl down to a 12M-token pretraining set for the\nfrozen 30M-param GPT, targeting low perplexity on a broad multi-domain\nhigh-quality English target (encyclopedic / web prose / news / technical Q&A).\n\nSTATED CRITERION\n----------------\nHeld-out perplexity is exp of the mean next-token cross-entropy on the target\nstream.  We therefore select the subset of the pool whose *own* token\ndistribution is closest to the target's, measuring closeness with the same\nfunctional the metric uses -- cross-entropy -- at unigram order, which is the\nhighest order estimable from the 1M-token disclosed dev target:\n\n    minimise   H(p_target, p_selection) = - sum_v p_target(v) log p_selection(v)\n    over subsets S of the pool with  sum_{d in S} |d| <= budget.\n\nAdding document d moves p_selection along the direction (c_d/|d| - p_selection),\nso the first-order decrease of H from adding d is proportional to\n\n    gain(d) = (1/|d|) sum_{t in d} min( p_target(t) / p_selection(t), CLIP )\n\ni.e. the mean *importance ratio* of d's tokens under the current selection.\nWe run batched greedy on this gain: score every eligible document, admit the\nbest BATCH of them, update p_selection, repeat until the budget is covered.\nThe ratio is clipped because some target tokens are unreachable at any price\n(the encyclopedic quarter is WikiText-formatted: ' ,' / ' .' / ' @-@ ' occur at\n~9% of its tokens and are essentially absent from raw web text); without a clip\nthe objective would spend unbounded budget chasing them.\n\nTwo guards keep the objective honest, since a pure frequency-matching score is\nhappy to buy rare tokens from boilerplate:\n  * quality gate -- minimum length, and caps on 5-gram repetition / top-token\n    share / minimum type-token ratio, which reject nav-bars, link farms and\n    directory listings;\n  * streaming near-duplicate filter -- 4-permutation MinHash over 5-gram\n    shingles; a document sharing >=2 of 4 sketch minima with an already-selected\n    document is dropped, so the budget is not spent re-reading the same page.\n\nDocuments are emitted in the order greedy admitted them, so truncating the list\nat the 12M-token budget preserves the matched mixture.\n\nRun:  python curate.py            (~3 min: tokenise pool, featurise, select)\n\"\"\"\nimport json, os, time\nimport numpy as np\n\nPOOL     = \"/workspace/data/pool.jsonl\"\nDEV      = \"/workspace/data/multi_dev.npy\"      # disclosed dev target\nOUT      = \"/workspace/submission/selection.json\"\nCACHE    = \"/workspace/work\"                    # reused if already present\nV        = 50257\nBUDGET   = 12_000_000\nTARGET   = 15_000_000        # emit ~25% past the budget so truncation is safe\n\n# --- selection hyperparameters (chosen on the disclosed dev target) ---\nCLIP     = 100.0   # cap on the per-token importance ratio\nBATCH    = 250     # documents admitted per greedy re-scoring step\nPRIOR    = 3e5     # pseudo-tokens of pool background seeding p_selection\nBGSM     = 1000.0  # pseudo-tokens smoothing p_target toward the pool background\nMIN_LEN  = 150     # min document length in tokens\nMAX_REP5 = 0.35    # max fraction of duplicated 5-gram shingles\nMAX_TOP1 = 0.12    # max share of the single most frequent token\nMIN_UNIQ = 0.18    # min type-token ratio\nTWEIGHTS = (1., 1., 1., 1.)   # relative weight of the 4 target registers\n\nK = 5              # shingle length\nP = np.uint64(1099511628211)\n\n\ndef tokenize_pool():\n    \"\"\"GPT-2 BPE over the whole pool -> flat token stream + document offsets.\"\"\"\n    os.environ[\"TOKENIZERS_PARALLELISM\"] = \"true\"\n    from tokenizers import Tokenizer\n    tok = Tokenizer.from_pretrained(\"gpt2\")\n    arrs, lens, ids, bt, bi = [], [], [], [], []\n\n    def flush():\n        if not bt:\n            return\n        for i, e in zip(bi, tok.encode_batch(bt)):\n            arrs.append(np.asarray(e.ids, dtype=np.uint16))\n            lens.append(len(e.ids)); ids.append(i)\n        bt.clear(); bi.clear()\n\n    for line in open(POOL):\n        r = json.loads(line); bt.append(r[\"text\"]); bi.append(r[\"id\"])\n        if len(bt) >= 4096:\n            flush()\n    flush()\n    lens = np.array(lens, dtype=np.int64)\n    off = np.zeros(len(lens) + 1, dtype=np.int64); np.cumsum(lens, out=off[1:])\n    return np.concatenate(arrs), off, lens, np.array(ids, dtype=np.int64)\n\n\ndef features(flat, off, lens):\n    \"\"\"Per-document junk features and MinHash sketches (5-gram shingles).\"\"\"\n    N = len(lens)\n    f = flat.astype(np.uint64)\n    h = np.zeros(len(f) - K + 1, dtype=np.uint64)       # rolling poly hash mod 2^64\n    for k in range(K):\n        h = h * P + f[k:len(f) - K + 1 + k]\n    mixers = [np.uint64(x) for x in (0x9E3779B97F4A7C15, 0xC2B2AE3D27D4EB4F,\n                                     0x165667B19E3779F9, 0x27D4EB2F165667C5)]\n    rep5 = np.zeros(N, np.float32); uniq = np.zeros(N, np.float32)\n    top1 = np.zeros(N, np.float32); sketch = np.zeros((N, 4), np.uint64)\n    dochash = np.zeros(N, np.uint64)\n    for i in range(N):\n        s, e = off[i], off[i + 1]\n        a = f[s:e]\n        if len(a) == 0:\n            continue\n        dh = np.uint64(len(a))\n        for chunk in (a[:32], a[-32:]):\n            for v in chunk:\n                dh = dh * P + v\n        dochash[i] = dh\n        u, c = np.unique(a, return_counts=True)\n        uniq[i] = len(u) / len(a); top1[i] = c.max() / len(a)\n        if e - s >= K + 1:\n            g = np.unique(h[s:e - K + 1])\n            rep5[i] = 1.0 - len(g) / (e - s - K + 1)\n            for j, m in enumerate(mixers):\n                sketch[i, j] = (g * m).min()\n    return rep5, uniq, top1, sketch.astype(np.int64), dochash.astype(np.int64)\n\n\ndef cached():\n    \"\"\"Load cached tokenisation/features if this script already ran once.\"\"\"\n    need = [\"flat\", \"off\", \"lens\", \"docids\", \"rep5\", \"uniq\", \"top1\", \"sketch\", \"dochash\"]\n    if all(os.path.exists(f\"{CACHE}/{n}.npy\") for n in need):\n        return [np.load(f\"{CACHE}/{n}.npy\") for n in need]\n    return None\n\n\ndef main():\n    t0 = time.time()\n    c = cached()\n    if c is None:\n        flat, off, lens, docids = tokenize_pool()\n        print(f\"tokenised {len(lens)} docs, {int(lens.sum())} tokens  {time.time()-t0:.0f}s\")\n        rep5, uniq, top1, sketch, dochash = features(flat, off, lens)\n        os.makedirs(CACHE, exist_ok=True)\n        for n, v in zip([\"flat\", \"off\", \"lens\", \"docids\", \"rep5\", \"uniq\", \"top1\",\n                         \"sketch\", \"dochash\"],\n                        [flat, off, lens, docids, rep5, uniq, top1, sketch, dochash]):\n            np.save(f\"{CACHE}/{n}.npy\", v)\n    else:\n        flat, off, lens, docids, rep5, uniq, top1, sketch, dochash = c\n        print(f\"loaded cached features  {time.time()-t0:.0f}s\")\n    f64 = flat.astype(np.int64)\n\n    # pool background distribution (the \"do nothing\" reference distribution)\n    bg = np.bincount(f64, minlength=V).astype(np.float64)\n    p_bg = (bg + 0.5) / (bg.sum() + 0.5 * V)\n\n    # target unigram: equal-parts mixture of the 4 disclosed registers\n    dev = np.load(DEV).astype(np.int64)\n    Q = len(dev) // 4\n    tw = np.array(TWEIGHTS, dtype=np.float64); tw /= tw.sum()\n    p_t = np.zeros(V)\n    for k in range(4):\n        ck = np.bincount(dev[k * Q:(k + 1) * Q], minlength=V).astype(np.float64)\n        p_t += tw[k] * (ck + 1.0) / (ck.sum() + V)\n\n    ok = ((lens >= MIN_LEN) & (rep5 <= MAX_REP5) &\n          (top1 <= MAX_TOP1) & (uniq >= MIN_UNIQ))\n    print(f\"eligible after quality gate: {int(ok.sum())} docs, \"\n          f\"{int(lens[ok].sum())} tokens\")\n\n    alive = ok.copy()\n    c_sel = p_bg * PRIOR; n_sel = PRIOR\n    sel, tot = [], 0\n    seen_doc, seen_sk = set(), set()\n    while tot < TARGET:\n        r = np.clip(p_t / (c_sel / n_sel), 0, CLIP).astype(np.float32)\n        gain = np.add.reduceat(r[f64].astype(np.float64), off[:-1]) / lens\n        gain[~alive] = -1e9\n        cand = np.argpartition(-gain, BATCH * 3)[:BATCH * 3]\n        cand = cand[np.argsort(-gain[cand])]\n        added = 0\n        for j in cand:\n            j = int(j)\n            if not alive[j]:\n                continue\n            if int(dochash[j]) in seen_doc:\n                alive[j] = False; continue\n            sk = [int(x) for x in sketch[j] if x != 0]\n            if sum(x in seen_sk for x in sk) >= 2:      # near-duplicate\n                alive[j] = False; continue\n            seen_doc.add(int(dochash[j])); seen_sk.update(sk)\n            alive[j] = False\n            sel.append(j); tot += int(lens[j]) + 1\n            c_sel += np.bincount(f64[off[j]:off[j + 1]], minlength=V)\n            n_sel += int(lens[j])\n            added += 1\n            if added >= BATCH or tot >= TARGET:\n                break\n        if added == 0:\n            break\n\n    p_sel = c_sel / n_sel\n    print(f\"proxy cross-entropy H(p_t,p_sel) = {-(p_t*np.log(p_sel)).sum():.4f} nats \"\n          f\"(pool background: {-(p_t*np.log(p_bg)).sum():.4f})\")\n    ids = [int(docids[j]) for j in sel]\n    assert len(ids) == len(set(ids)), \"duplicate ids\"\n    json.dump(ids, open(OUT, \"w\"))\n    print(f\"wrote {len(ids)} ids ({tot} tokens, budget {BUDGET}) -> {OUT}  \"\n          f\"{time.time()-t0:.0f}s\")\n\n\nif __name__ == \"__main__\":\n    main()\n", "structuredPatch": [{"oldStart": 157, "oldLines": 11, "newStart": 157, "newLines": 14, "lines": ["     # target unigram: equal-parts mixture of the 4 disclosed registers", "     dev = np.load(DEV).astype(np.int64)", "     Q = len(dev) // 4", "+    # Smoothing is toward p_bg, NOT toward uniform: uniform smoothing spreads", "+    # ~8% of the target mass over ~25k token types the target never emits, and", "+    # greedy then burns budget chasing that noise (costs ~6 dev PPL).", "     tw = np.array(TWEIGHTS, dtype=np.float64); tw /= tw.sum()", "     p_t = np.zeros(V)", "     for k in range(4):", "         ck = np.bincount(dev[k * Q:(k + 1) * Q], minlength=V).astype(np.float64)", "-        p_t += tw[k] * (ck + 1.0) / (ck.sum() + V)", "+        p_t += tw[k] * (ck + BGSM * p_bg) / (ck.sum() + BGSM)", " ", "     ok = ((lens >= MIN_LEN) & (rep5 <= MAX_REP5) &", "           (top1 <= MAX_TOP1) & (uniq >= MIN_UNIQ))"]}], "userModified": false, "replaceAll": false}

loaded cached features  0s
eligible after quality gate: 128837 docs, 128245057 tokens
proxy cross-entropy H(p_t,p_sel) = 7.8543 nats (pool background: 8.0783)
wrote 18418 ids (15000209 tokens, budget 12000000) -> /workspace/submission/selection.json  78s
reproduces evaluated winner g_bg150 exactly: True 18418 18418
2426

[stdout]
loaded cached features  0s
eligible after quality gate: 128837 docs, 128245057 tokens
proxy cross-entropy H(p_t,p_sel) = 7.8543 nats (pool background: 8.0783)
wrote 18418 ids (15000209 tokens, budget 12000000) -> /workspace/submission/selection.json  78s
reproduces evaluated winner g_bg150 exactly: True 18418 18418
2426
[metadata] {"noOutputExpected": false}

packed 12000000 tokens from 18418 ids -> /tmp/tr_final.npy
ids unique: True | min 24 max 181971 < 182016: True
total tokens incl EOS: 15000209 budget 12,000,000 covered: True
docs actually trained on (first 12M): 15285
  FINAL         artifact-rich(>1%) tokens   623140 = 5.20% of budget
  qgate_random  artifact-rich(>1%) tokens    90530 = 0.60% of budget
  head-400 ' ,'+' .' per-100k: 1962.0
2396

[stdout]
packed 12000000 tokens from 18418 ids -> /tmp/tr_final.npy
ids unique: True | min 24 max 181971 < 182016: True
total tokens incl EOS: 15000209 budget 12,000,000 covered: True
docs actually trained on (first 12M): 15285
  FINAL         artifact-rich(>1%) tokens   623140 = 5.20% of budget
  qgate_random  artifact-rich(>1%) tokens    90530 = 0.60% of budget
  head-400 ' ,'+' .' per-100k: 1962.0
2396
[metadata] {"noOutputExpected": false}

SEG 0 wiki loss 6.2604 ppl 523.43
SEG 1 webprose loss 5.5391 ppl 254.44
SEG 2 news loss 5.5853 ppl 266.48
SEG 3 qa loss 4.9727 ppl 144.42
HELDOUT_PPL 267.721  loss 5.5899  train_s 51  /tmp/tr_final.npy
2326

[stdout]
SEG 0 wiki loss 6.2604 ppl 523.43
SEG 1 webprose loss 5.5391 ppl 254.44
SEG 2 news loss 5.5853 ppl 266.48
SEG 3 qa loss 4.9727 ppl 144.42
HELDOUT_PPL 267.721  loss 5.5899  train_s 51  /tmp/tr_final.npy
2326
[metadata] {"noOutputExpected": false}

The file /workspace/claim.md has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"type": "update", "filePath": "/workspace/claim.md", "content": "# Curating a raw web pool by target-distribution matching\n\nSubmitted selection: `submission/selection.json` (18,418 ids, 15.0M tokens; the\nfirst 15,285 fill the 12M budget), produced by `submission/curate.py`.\n**Dev perplexity 267.7 vs 485.5 for a random selection** (frozen recipe, seed 1337).\n\n## Hypothesis\n\nUnder a token budget far smaller than the pool (12M of 200M), held-out perplexity\non a broad multi-domain target is governed mainly by **how closely the training\nsubset's token distribution matches the target's**, and only secondarily by\ngeneric \"document quality\". Concretely:\n\n> Greedily choosing documents to minimise the unigram cross-entropy\n> `H(p_target, p_selection)` — the same functional the metric evaluates, at the\n> highest order estimable from the 1M-token disclosed target — beats random\n> selection, beats quality-filtered random selection, and beats single-axis\n> \"pick the best-looking documents\" ranking.\n\nThe corollary that makes this non-trivial: the pool is *not* short of readable\nprose. It is short of prose in the target's **registers and surface conventions**,\nand a budget-constrained selector must buy those deliberately.\n\n## Mechanism\n\nHeld-out loss is a mean over windows of four equal registers (recovered by\ndecoding `multi_dev.npy`: WikiText-style encyclopedic / clean web prose /\nnewswire / StackExchange HTML Q&A). Mean loss is additive across registers, so\nthe **highest-loss register dominates the score**, and a selection that leaves one\nregister unrepresented pays disproportionately.\n\nMatching `p_target` allocates budget by each register's share of the *target*\nrather than its share of the *pool*. The importance ratio\n`r(v) = p_target(v)/p_selection(v)` is recomputed as the selection grows, so once\na register is covered its tokens stop being attractive and the selector moves on —\ncoverage is self-balancing rather than hand-weighted.\n\n**Predicted observables (not the final perplexity), all measured:**\n\n1. **The proxy objective tracks the real one.** The selection reduces unigram\n   `H(p_target, p_selection)` from 8.078 nats (pool background = do nothing) to\n   **7.854 nats**. Across the smoothing choice, the variant achieving the larger\n   proxy *reduction* also trains to lower perplexity: smoothing `p_target` toward\n   the pool background reduces the proxy by 0.224 nats and gives dev PPL 272.9,\n   while smoothing toward uniform reduces its own proxy by only 0.195 nats and\n   gives 279.3 (both at min-length 300). Uniform smoothing spreads ~8% of target\n   mass over ~25k token types the target never emits, and greedy burns budget\n   chasing that noise.\n2. **Gains land on the register that was starved, not uniformly.** Per-register\n   held-out loss (nats), quality-filtered random → submitted selection:\n\n   | register | qgate-random | submitted | Δ |\n   |---|---|---|---|\n   | technical Q&A | 5.777 | **4.973** | −0.804 |\n   | encyclopedic | 6.724 | **6.260** | −0.464 |\n   | newswire | 5.870 | **5.585** | −0.285 |\n   | web prose | 5.627 | **5.539** | −0.088 |\n\n   The register the raw pool is emptiest of (technical Q&A: **zero** `<pre><code>`\n   documents pool-wide) improves ~9× more than web prose, the register the pool\n   most resembles. This is the signature of register rebalancing, and it is not\n   what a uniform \"better data\" story predicts.\n3. **The selector buys the target's surface conventions at a measurable rate.**\n   The encyclopedic quarter is WikiText-formatted: GPT-2 tokens `' ,'` and `' .'`\n   (space-before-punctuation) are **7.9%** of its tokens but **0.055%** of pool\n   tokens. The submitted selection puts **5.20%** of its budget in documents\n   carrying that convention — 5.3× the random rate (623k vs 119k tokens) — and its\n   first 400 documents are 38× denser in those tokens than random (1962 vs 51 per\n   100k).\n4. **The encyclopedic register stays the worst one no matter what.** Its\n   convention is nearly absent from the pool (0 documents contain ` @-@ `), so it\n   remained the highest-loss quarter in every one of the 20 selections I trained.\n   That is a ceiling imposed by the pool, not by the selector.\n\n## Falsification\n\nThe claim predicts specific *failures*. Each was run as its own training run\n(frozen recipe, seed 1337, dev target). All would have falsified it; none did.\n\n| Selection criterion | dev PPL |\n|---|---|\n| Random (do-nothing baseline) | 485.5 |\n| Quality gate + dedup only, random within it | 403.6 |\n| Single global quality score (LLR vs. target mixture), top-k | 387.0 |\n| Balanced per-register Naive-Bayes round-robin | 312.5 |\n| Greedy match, uniform-smoothed target, min-len 300 | 279.3 |\n| Greedy match, background-smoothed target, min-len 300 | 272.9 |\n| **Greedy match, background-smoothed, min-len 150 (submitted)** | **267.7** |\n\n- *If the junk filter were doing the work*, quality-gate-only random selection\n  would approach the matched selection. It does not: the gate buys 485.5 → 403.6,\n  matching buys 403.6 → 267.7. **Matching contributes ~1.7× more than filtering**,\n  and the two are separable because both selections pass the identical gate.\n- *If a single quality axis sufficed*, ranking documents by their likelihood ratio\n  against the pooled target mixture would match register-aware selection. It does\n  not (387.0 vs 312.5): a single global ranking collapses onto whichever register\n  the pool most resembles.\n- *If the hardest register's surface tokens were the bottleneck*, force-feeding\n  1.5M tokens (12.5% of budget) of the documents richest in `' ,'`/`' .'` would\n  help. It hurt: 279.3 → 290.5 at matched settings. Those documents are nav-bar\n  and sports-ellipsis boilerplate; the objective's own 5.2% allocation is already\n  the right dose, and buying more trades prose for junk.\n- *If the hardest register deserved extra weight*, doubling the encyclopedic\n  register's weight in `p_target` would help. It hurt (288.8 vs 279.3): the\n  equal-parts weighting is already correct, as the metric's additive structure implies.\n- *If longer documents were simply better*, raising the minimum length would help.\n  It hurt monotonically above the optimum: 267.7 (150) → 269.5 (200) → 279.3 (300)\n  → 295.4 (900). The optimum is a bracketed interior minimum (267.7 at both 100 and\n  150), not a monotone preference — past ~150 tokens, extra length costs coverage.\n- *If a more faithful greedy were better*, shrinking the batch from 250 to 80\n  documents per re-scoring step would help. It hurt (273.4): finer steps overfit\n  the unigram proxy, which is a means, not the objective.\n- Also tested and rejected against 267.7: unclipped-er ratio (clip 1000 → 269.0),\n  looser quality gate (269.2), weaker target smoothing (268.1).\n\n**What would still falsify the claim:** a selection with a *higher* proxy\ncross-entropy that trains to lower perplexity; or a quality-only criterion using\nno target sample reaching ~268 at this budget.\n\n**Honest limits.** `p_target` is estimated from the disclosed dev sample, so the\nmatch is only as good as that sample's coverage of the hidden one (stated to be a\ndisjoint sample of the same domain). Five scalar hyperparameters were tuned\nagainst dev, which bounds but does not eliminate selection-on-dev risk; the\nflatness of the optimum (267.7 across min-length 100–150, 268.1 at a 5× different\nsmoothing mass) suggests the setting is not knife-edge. Differences below ~1.5 PPL\nhere are within run-to-run noise and I do not claim them. All runs share one seed,\nso I have not measured seed variance directly.\n\n## Transfer\n\nThe method needs only (a) a sample of target-domain text, (b) the tokenizer, and\n(c) one counting pass — no labels, no reference model, no GPU. The greedy step is\na gather plus a segment-sum, so scoring the full 200M-token pool costs ~1 s and\nthe entire selection ~80 s on CPU; cost is linear in pool tokens and independent\nof how much is finally kept, so it scales to far larger pools.\n\nIt transfers to any budget-constrained data decision where the evaluation\ndistribution is known or samplable: choosing pretraining mixtures, picking\nfine-tuning subsets for a known deployment domain, or setting domain weights\nwithout hand-tuned ratios. Three lessons generalise beyond this task: **smooth the\ntarget toward the achievable background, not toward uniform** (else the selector\nchases unreachable rare tokens); **clip the importance ratio** (some target tokens\ncannot be bought at any price, and an unclipped objective spends the budget\ntrying); and **keep greedy steps coarse** (chasing the proxy too precisely\ndegrades the true metric). The natural extension is the same greedy on\nhigher-order statistics once more target text is available — the unigram order is\na consequence of having only 1M target tokens, not a limit of the criterion.\n", "structuredPatch": [{"oldStart": 1, "oldLines": 5, "newStart": 1, "newLines": 9, "lines": [" # Curating a raw web pool by target-distribution matching", " ", "+Submitted selection: `submission/selection.json` (18,418 ids, 15.0M tokens; the", "+first 15,285 fill the 12M budget), produced by `submission/curate.py`.", "+**Dev perplexity 267.7 vs 485.5 for a random selection** (frozen recipe, seed 1337).", "+", " ## Hypothesis", " ", " Under a token budget far smaller than the pool (12M of 200M), held-out perplexity"]}, {"oldStart": 9, "oldLines": 56, "newStart": 13, "newLines": 68, "lines": [" ", " > Greedily choosing documents to minimise the unigram cross-entropy", " > `H(p_target, p_selection)` — the same functional the metric evaluates, at the", "-> highest order estimable from the 1M-token disclosed target — beats both random", "-> selection and quality-filtered random selection by a wide margin, and beats", "-> single-axis \"pick the best-looking documents\" ranking.", "+> highest order estimable from the 1M-token disclosed target — beats random", "+> selection, beats quality-filtered random selection, and beats single-axis", "+> \"pick the best-looking documents\" ranking.", " ", " The corollary that makes this non-trivial: the pool is *not* short of readable", "-prose. It is short of prose in the target's **registers and surface", "-conventions**, and a budget-constrained selector must buy those deliberately.", "+prose. It is short of prose in the target's **registers and surface conventions**,", "+and a budget-constrained selector must buy those deliberately.", " ", " ## Mechanism", " ", "-Held-out loss is a mean over windows of four equal registers (I recovered them by", "+Held-out loss is a mean over windows of four equal registers (recovered by", " decoding `multi_dev.npy`: WikiText-style encyclopedic / clean web prose /", "-newswire / StackExchange HTML Q&A). Mean loss is therefore additive across", "-registers, so the **highest-loss register dominates the gradient of the score**,", "-and a selection that leaves one register unrepresented pays for it disproportionately.", "+newswire / StackExchange HTML Q&A). Mean loss is additive across registers, so", "+the **highest-loss register dominates the score**, and a selection that leaves one", "+register unrepresented pays disproportionately.", " ", "-Matching `p_target` allocates budget to registers in proportion to their share of", "-the target rather than their share of the pool. The importance ratio", "+Matching `p_target` allocates budget by each register's share of the *target*", "+rather than its share of the *pool*. The importance ratio", " `r(v) = p_target(v)/p_selection(v)` is recomputed as the selection grows, so once", " a register is covered its tokens stop being attractive and the selector moves on —", " coverage is self-balancing rather than hand-weighted.", " ", " **Predicted observables (not the final perplexity), all measured:**", " ", "-1. **The proxy objective tracks the real one.** Unigram", "-   `H(p_target, p_selection)` falls 8.825 nats (pool background) → 7.854 nats", "-   (final selection). Variants with a lower proxy score train to lower dev", "-   perplexity; e.g. smoothing `p_target` toward the pool background instead of", "-   toward uniform lowers the proxy 8.630 → 7.854 nats and dev PPL 279.3 → 272.9,", "-   because uniform smoothing puts ~8% of target mass on ~25k token types the", "-   target never emits, and greedy then spends budget chasing that noise.", "+1. **The proxy objective tracks the real one.** The selection reduces unigram", "+   `H(p_target, p_selection)` from 8.078 nats (pool background = do nothing) to", "+   **7.854 nats**. Across the smoothing choice, the variant achieving the larger", "+   proxy *reduction* also trains to lower perplexity: smoothing `p_target` toward", "+   the pool background reduces the proxy by 0.224 nats and gives dev PPL 272.9,", "+   while smoothing toward uniform reduces its own proxy by only 0.195 nats and", "+   gives 279.3 (both at min-length 300). Uniform smoothing spreads ~8% of target", "+   mass over ~25k token types the target never emits, and greedy burns budget", "+   chasing that noise.", " 2. **Gains land on the register that was starved, not uniformly.** Per-register", "-   held-out loss, quality-filtered random vs. matched selection:", "-   Q&A **5.777 → 5.038** nats, news 5.870 → 5.630, web prose 5.627 → 5.559,", "-   encyclopedic 6.724 → 6.308. The technical-Q&A register — the one the raw pool", "-   is emptiest of — improves ~3× more than the register it is richest in.", "+   held-out loss (nats), quality-filtered random → submitted selection:", "+", "+   | register | qgate-random | submitted | Δ |", "+   |---|---|---|---|", "+   | technical Q&A | 5.777 | **4.973** | −0.804 |", "+   | encyclopedic | 6.724 | **6.260** | −0.464 |", "+   | newswire | 5.870 | **5.585** | −0.285 |", "+   | web prose | 5.627 | **5.539** | −0.088 |", "+", "+   The register the raw pool is emptiest of (technical Q&A: **zero** `<pre><code>`", "+   documents pool-wide) improves ~9× more than web prose, the register the pool", "+   most resembles. This is the signature of register rebalancing, and it is not", "+   what a uniform \"better data\" story predicts.", " 3. **The selector buys the target's surface conventions at a measurable rate.**", "    The encyclopedic quarter is WikiText-formatted: GPT-2 tokens `' ,'` and `' .'`", "-   (space-before-punctuation) are 7.9% of its tokens but 0.05% of pool tokens.", "-   The matched selection puts ~5% of its budget in documents carrying that", "-   convention — ~5× the random rate (592k vs 119k tokens) — and its head is 25×", "-   denser in those tokens (1276 vs 51 per 100k).", "+   (space-before-punctuation) are **7.9%** of its tokens but **0.055%** of pool", "+   tokens. The submitted selection puts **5.20%** of its budget in documents", "+   carrying that convention — 5.3× the random rate (623k vs 119k tokens) — and its", "+   first 400 documents are 38× denser in those tokens than random (1962 vs 51 per", "+   100k).", " 4. **The encyclopedic register stays the worst one no matter what.** Its", "-   convention is nearly absent from the pool, so it remains the highest-loss", "-   quarter (6.31 nats vs 5.04–5.66) in every selection I trained. This is a", "-   ceiling imposed by the pool, not by the selector.", "+   convention is nearly absent from the pool (0 documents contain ` @-@ `), so it", "+   remained the highest-loss quarter in every one of the 20 selections I trained.", "+   That is a ceiling imposed by the pool, not by the selector.", " ", " ## Falsification", " ", "-The claim predicts specific *failures*, and I ran each as its own training run", "+The claim predicts specific *failures*. Each was run as its own training run", " (frozen recipe, seed 1337, dev target). All would have falsified it; none did.", " ", " | Selection criterion | dev PPL |"]}, {"oldStart": 67, "oldLines": 55, "newStart": 83, "newLines": 65, "lines": [" | Quality gate + dedup only, random within it | 403.6 |", " | Single global quality score (LLR vs. target mixture), top-k | 387.0 |", " | Balanced per-register Naive-Bayes round-robin | 312.5 |", "-| **Greedy `H(p_target,p_selection)` matching (submitted)** | **267.7** |", "+| Greedy match, uniform-smoothed target, min-len 300 | 279.3 |", "+| Greedy match, background-smoothed target, min-len 300 | 272.9 |", "+| **Greedy match, background-smoothed, min-len 150 (submitted)** | **267.7** |", " ", "-Falsifiers that did not fire:", "-", " - *If the junk filter were doing the work*, quality-gate-only random selection", "   would approach the matched selection. It does not: the gate buys 485.5 → 403.6,", "-  matching buys 403.6 → 267.7. Matching contributes ~1.7× more than filtering.", "-- *If a single quality axis sufficed*, ranking every document by its likelihood", "-  ratio against the pooled target mixture would match register-aware selection.", "-  It does not (387.0 vs 312.5): a single global ranking collapses onto whichever", "-  register the pool most resembles.", "+  matching buys 403.6 → 267.7. **Matching contributes ~1.7× more than filtering**,", "+  and the two are separable because both selections pass the identical gate.", "+- *If a single quality axis sufficed*, ranking documents by their likelihood ratio", "+  against the pooled target mixture would match register-aware selection. It does", "+  not (387.0 vs 312.5): a single global ranking collapses onto whichever register", "+  the pool most resembles.", " - *If the hardest register's surface tokens were the bottleneck*, force-feeding", "   1.5M tokens (12.5% of budget) of the documents richest in `' ,'`/`' .'` would", "-  help. It hurt (267.7 → 290.5): those documents are nav-bar and sports-ellipsis", "-  boilerplate, and the objective's own ~5% allocation is already the right dose.", "+  help. It hurt: 279.3 → 290.5 at matched settings. Those documents are nav-bar", "+  and sports-ellipsis boilerplate; the objective's own 5.2% allocation is already", "+  the right dose, and buying more trades prose for junk.", " - *If the hardest register deserved extra weight*, doubling the encyclopedic", "-  register's weight in `p_target` would help. It hurt (288.8 vs 279.3 at matched", "-  settings): the equal-parts target weighting is already correct, as the additive", "-  structure of the metric implies.", "-- *If longer documents were simply better*, raising the minimum length would", "-  help. It hurt (295.4 at 900 tokens vs 267.7 at 150): past ~150 tokens, extra", "-  length costs distributional coverage. The optimum is a bracketed interior", "-  minimum, not a monotone preference.", "+  register's weight in `p_target` would help. It hurt (288.8 vs 279.3): the", "+  equal-parts weighting is already correct, as the metric's additive structure implies.", "+- *If longer documents were simply better*, raising the minimum length would help.", "+  It hurt monotonically above the optimum: 267.7 (150) → 269.5 (200) → 279.3 (300)", "+  → 295.4 (900). The optimum is a bracketed interior minimum (267.7 at both 100 and", "+  150), not a monotone preference — past ~150 tokens, extra length costs coverage.", "+- *If a more faithful greedy were better*, shrinking the batch from 250 to 80", "+  documents per re-scoring step would help. It hurt (273.4): finer steps overfit", "+  the unigram proxy, which is a means, not the objective.", "+- Also tested and rejected against 267.7: unclipped-er ratio (clip 1000 → 269.0),", "+  looser quality gate (269.2), weaker target smoothing (268.1).", " ", " **What would still falsify the claim:** a selection with a *higher* proxy", "-cross-entropy that trains to lower perplexity; or a quality-only criterion", "-(no target sample used) reaching ~268 on this budget. Note the honest limit of", "-the evidence: `p_target` is estimated from the disclosed dev sample, so the", "-matching is only as good as that sample's coverage of the hidden one — the two", "-are stated to be disjoint samples of the same domain, and I tuned only five", "-scalar hyperparameters against dev, which bounds but does not eliminate", "-selection-on-dev risk.", "+cross-entropy that trains to lower perplexity; or a quality-only criterion using", "+no target sample reaching ~268 at this budget.", " ", "+**Honest limits.** `p_target` is estimated from the disclosed dev sample, so the", "+match is only as good as that sample's coverage of the hidden one (stated to be a", "+disjoint sample of the same domain). Five scalar hyperparameters were tuned", "+against dev, which bounds but does not eliminate selection-on-dev risk; the", "+flatness of the optimum (267.7 across min-length 100–150, 268.1 at a 5× different", "+smoothing mass) suggests the setting is not knife-edge. Differences below ~1.5 PPL", "+here are within run-to-run noise and I do not claim them. All runs share one seed,", "+so I have not measured seed variance directly.", "+", " ## Transfer", " ", " The method needs only (a) a sample of target-domain text, (b) the tokenizer, and", "-(c) one pass to count tokens — no labels, no reference model, no GPU. The greedy", "-step is a gather plus a segment-sum, so scoring the full 200M-token pool costs", "-~1 s and the whole selection ~1 min on CPU; it scales to far larger pools because", "-cost is linear in tokens and independent of how much is finally kept.", "+(c) one counting pass — no labels, no reference model, no GPU. The greedy step is", "+a gather plus a segment-sum, so scoring the full 200M-token pool costs ~1 s and", "+the entire selection ~80 s on CPU; cost is linear in pool tokens and independent", "+of how much is finally kept, so it scales to far larger pools.", " ", " It transfers to any budget-constrained data decision where the evaluation", " distribution is known or samplable: choosing pretraining mixtures, picking", "-fine-tuning subsets for a known deployment domain, or deciding domain weights", "-without hand-set ratios. Two transferable lessons generalise beyond this task:", "-**smooth the target distribution toward the achievable background rather than", "-toward uniform** (otherwise the selector chases unreachable rare tokens), and", "-**clip the importance ratio** (some target tokens cannot be bought at any price,", "-and an unclipped objective will spend the whole budget trying). The natural", "-extension is to run the same greedy on higher-order statistics as more target", "-text becomes available; the unigram order is a consequence of having only 1M", "-target tokens, not a limitation of the criterion.", "+fine-tuning subsets for a known deployment domain, or setting domain weights", "+without hand-tuned ratios. Three lessons generalise beyond this task: **smooth the", "+target toward the achievable background, not toward uniform** (else the selector", "+chases unreachable rare tokens); **clip the importance ratio** (some target tokens", "+cannot be bought at any price, and an unclipped objective spends the budget", "+trying); and **keep greedy steps coarse** (chasing the proxy too precisely", "+degrades the true metric). The natural extension is the same greedy on", "+higher-order statistics once more target text is available — the unigram order is", "+a consequence of having only 1M target tokens, not a limit of the criterion."]}], "originalFile": "# Curating a raw web pool by target-distribution matching\n\n## Hypothesis\n\nUnder a token budget far smaller than the pool (12M of 200M), held-out perplexity\non a broad multi-domain target is governed mainly by **how closely the training\nsubset's token distribution matches the target's**, and only secondarily by\ngeneric \"document quality\". Concretely:\n\n> Greedily choosing documents to minimise the unigram cross-entropy\n> `H(p_target, p_selection)` — the same functional the metric evaluates, at the\n> highest order estimable from the 1M-token disclosed target — beats both random\n> selection and quality-filtered random selection by a wide margin, and beats\n> single-axis \"pick the best-looking documents\" ranking.\n\nThe corollary that makes this non-trivial: the pool is *not* short of readable\nprose. It is short of prose in the target's **registers and surface\nconventions**, and a budget-constrained selector must buy those deliberately.\n\n## Mechanism\n\nHeld-out loss is a mean over windows of four equal registers (I recovered them by\ndecoding `multi_dev.npy`: WikiText-style encyclopedic / clean web prose /\nnewswire / StackExchange HTML Q&A). Mean loss is therefore additive across\nregisters, so the **highest-loss register dominates the gradient of the score**,\nand a selection that leaves one register unrepresented pays for it disproportionately.\n\nMatching `p_target` allocates budget to registers in proportion to their share of\nthe target rather than their share of the pool. The importance ratio\n`r(v) = p_target(v)/p_selection(v)` is recomputed as the selection grows, so once\na register is covered its tokens stop being attractive and the selector moves on —\ncoverage is self-balancing rather than hand-weighted.\n\n**Predicted observables (not the final perplexity), all measured:**\n\n1. **The proxy objective tracks the real one.** Unigram\n   `H(p_target, p_selection)` falls 8.825 nats (pool background) → 7.854 nats\n   (final selection). Variants with a lower proxy score train to lower dev\n   perplexity; e.g. smoothing `p_target` toward the pool background instead of\n   toward uniform lowers the proxy 8.630 → 7.854 nats and dev PPL 279.3 → 272.9,\n   because uniform smoothing puts ~8% of target mass on ~25k token types the\n   target never emits, and greedy then spends budget chasing that noise.\n2. **Gains land on the register that was starved, not uniformly.** Per-register\n   held-out loss, quality-filtered random vs. matched selection:\n   Q&A **5.777 → 5.038** nats, news 5.870 → 5.630, web prose 5.627 → 5.559,\n   encyclopedic 6.724 → 6.308. The technical-Q&A register — the one the raw pool\n   is emptiest of — improves ~3× more than the register it is richest in.\n3. **The selector buys the target's surface conventions at a measurable rate.**\n   The encyclopedic quarter is WikiText-formatted: GPT-2 tokens `' ,'` and `' .'`\n   (space-before-punctuation) are 7.9% of its tokens but 0.05% of pool tokens.\n   The matched selection puts ~5% of its budget in documents carrying that\n   convention — ~5× the random rate (592k vs 119k tokens) — and its head is 25×\n   denser in those tokens (1276 vs 51 per 100k).\n4. **The encyclopedic register stays the worst one no matter what.** Its\n   convention is nearly absent from the pool, so it remains the highest-loss\n   quarter (6.31 nats vs 5.04–5.66) in every selection I trained. This is a\n   ceiling imposed by the pool, not by the selector.\n\n## Falsification\n\nThe claim predicts specific *failures*, and I ran each as its own training run\n(frozen recipe, seed 1337, dev target). All would have falsified it; none did.\n\n| Selection criterion | dev PPL |\n|---|---|\n| Random (do-nothing baseline) | 485.5 |\n| Quality gate + dedup only, random within it | 403.6 |\n| Single global quality score (LLR vs. target mixture), top-k | 387.0 |\n| Balanced per-register Naive-Bayes round-robin | 312.5 |\n| **Greedy `H(p_target,p_selection)` matching (submitted)** | **267.7** |\n\nFalsifiers that did not fire:\n\n- *If the junk filter were doing the work*, quality-gate-only random selection\n  would approach the matched selection. It does not: the gate buys 485.5 → 403.6,\n  matching buys 403.6 → 267.7. Matching contributes ~1.7× more than filtering.\n- *If a single quality axis sufficed*, ranking every document by its likelihood\n  ratio against the pooled target mixture would match register-aware selection.\n  It does not (387.0 vs 312.5): a single global ranking collapses onto whichever\n  register the pool most resembles.\n- *If the hardest register's surface tokens were the bottleneck*, force-feeding\n  1.5M tokens (12.5% of budget) of the documents richest in `' ,'`/`' .'` would\n  help. It hurt (267.7 → 290.5): those documents are nav-bar and sports-ellipsis\n  boilerplate, and the objective's own ~5% allocation is already the right dose.\n- *If the hardest register deserved extra weight*, doubling the encyclopedic\n  register's weight in `p_target` would help. It hurt (288.8 vs 279.3 at matched\n  settings): the equal-parts target weighting is already correct, as the additive\n  structure of the metric implies.\n- *If longer documents were simply better*, raising the minimum length would\n  help. It hurt (295.4 at 900 tokens vs 267.7 at 150): past ~150 tokens, extra\n  length costs distributional coverage. The optimum is a bracketed interior\n  minimum, not a monotone preference.\n\n**What would still falsify the claim:** a selection with a *higher* proxy\ncross-entropy that trains to lower perplexity; or a quality-only criterion\n(no target sample used) reaching ~268 on this budget. Note the honest limit of\nthe evidence: `p_target` is estimated from the disclosed dev sample, so the\nmatching is only as good as that sample's coverage of the hidden one — the two\nare stated to be disjoint samples of the same domain, and I tuned only five\nscalar hyperparameters against dev, which bounds but does not eliminate\nselection-on-dev risk.\n\n## Transfer\n\nThe method needs only (a) a sample of target-domain text, (b) the tokenizer, and\n(c) one pass to count tokens — no labels, no reference model, no GPU. The greedy\nstep is a gather plus a segment-sum, so scoring the full 200M-token pool costs\n~1 s and the whole selection ~1 min on CPU; it scales to far larger pools because\ncost is linear in tokens and independent of how much is finally kept.\n\nIt transfers to any budget-constrained data decision where the evaluation\ndistribution is known or samplable: choosing pretraining mixtures, picking\nfine-tuning subsets for a known deployment domain, or deciding domain weights\nwithout hand-set ratios. Two transferable lessons generalise beyond this task:\n**smooth the target distribution toward the achievable background rather than\ntoward uniform** (otherwise the selector chases unreachable rare tokens), and\n**clip the importance ratio** (some target tokens cannot be bought at any price,\nand an unclipped objective will spend the whole budget trying). The natural\nextension is to run the same greedy on higher-order statistics as more target\ntext becomes available; the unigram order is a consequence of having only 1M\ntarget tokens, not a limitation of the criterion.\n", "userModified": false}

/tmp/ctest/curate_test.py:116: RuntimeWarning: overflow encountered in scalar multiply
  dh = dh * P + v
tokenised 4000 docs, 2768019 tokens  2s
eligible after quality gate: 3323 docs, 2588922 tokens
proxy cross-entropy H(p_t,p_sel) = 8.1457 nats (pool background: 8.1595)
wrote 2393 ids (2000104 tokens, budget 12000000) -> /tmp/ctest/sel.json  2s
  rep5: fresh-vs-cached identical = True
  uniq: fresh-vs-cached identical = True
  top1: fresh-vs-cached identical = True
  dochash: fresh-vs-cached identical = True
Shell cwd was reset to /workspace

[stdout]
/tmp/ctest/curate_test.py:116: RuntimeWarning: overflow encountered in scalar multiply
  dh = dh * P + v
tokenised 4000 docs, 2768019 tokens  2s
eligible after quality gate: 3323 docs, 2588922 tokens
proxy cross-entropy H(p_t,p_sel) = 8.1457 nats (pool background: 8.1595)
wrote 2393 ids (2000104 tokens, budget 12000000) -> /tmp/ctest/sel.json  2s
  rep5: fresh-vs-cached identical = True
  uniq: fresh-vs-cached identical = True
  top1: fresh-vs-cached identical = True
  dochash: fresh-vs-cached identical = True
[stderr]

Shell cwd was reset to /workspace
[metadata] {"noOutputExpected": false}

patched: True
building
2236

[stdout]
patched: True
building
2236
[metadata] {"noOutputExpected": false}

eligible 107319
proxy H(p_t,p_sel) = 7.888082093965288
eligible 128837
proxy H(p_t,p_sel) = 7.86794081280648
launched
2156

[stdout]
eligible 107319
proxy H(p_t,p_sel) = 7.888082093965288
eligible 128837
proxy H(p_t,p_sel) = 7.86794081280648
launched
2156
[metadata] {"noOutputExpected": false}

31:ok=(lens>=a.minlen)&(rep5<=a.maxrep5)&(top1<=a.maxtop1)&(uniq>=a.minuniq)
--- g_pri.log:
eligible 128837
10 docs 2500 tok 1676319 10.8
20 docs 5000 tok 3602901 21.1
30 docs 7500 tok 5555724 31.3
40 docs 10000 tok 7571016 41.5
50 docs 12500 tok 9745071 51.8
60 docs 15000 tok 12097541 62.1
70 docs 17500 tok 14430381 72.6
proxy H(p_t,p_sel) = 7.86794081280648
selected 18068 docs 15009328 tokens 75.7
--- g_tight.log:
eligible 107319
10 docs 2500 tok 1508743 10.9
20 docs 5000 tok 3241971 21.4
30 docs 7500 tok 5100621 31.9
40 docs 10000 tok 7295678 42.0
50 docs 12500 tok 9466973 52.4
60 docs 15000 tok 11677353 62.8
70 docs 17500 tok 14085825 73.2
proxy H(p_t,p_sel) = 7.888082093965288
selected 18361 docs 15017808 tokens 77.4

[stdout]
31:ok=(lens>=a.minlen)&(rep5<=a.maxrep5)&(top1<=a.maxtop1)&(uniq>=a.minuniq)
--- g_pri.log:
eligible 128837
10 docs 2500 tok 1676319 10.8
20 docs 5000 tok 3602901 21.1
30 docs 7500 tok 5555724 31.3
40 docs 10000 tok 7571016 41.5
50 docs 12500 tok 9745071 51.8
60 docs 15000 tok 12097541 62.1
70 docs 17500 tok 14430381 72.6
proxy H(p_t,p_sel) = 7.86794081280648
selected 18068 docs 15009328 tokens 75.7
--- g_tight.log:
eligible 107319
10 docs 2500 tok 1508743 10.9
20 docs 5000 tok 3241971 21.4
30 docs 7500 tok 5100621 31.9
40 docs 10000 tok 7295678 42.0
50 docs 12500 tok 9466973 52.4
60 docs 15000 tok 11677353 62.8
70 docs 17500 tok 14085825 73.2
proxy H(p_t,p_sel) = 7.888082093965288
selected 18361 docs 15017808 tokens 77.4
[metadata] {"noOutputExpected": false}

prior 3e6 (proxy 7.868):
HELDOUT_PPL 275.874  loss 5.6199  train_s 108  /tmp/tr_g_pri.npy
tight gate (proxy 7.888):
HELDOUT_PPL 281.775  loss 5.6411  train_s 108  /tmp/tr_g_tight.npy
2026

[stdout]
prior 3e6 (proxy 7.868):
HELDOUT_PPL 275.874  loss 5.6199  train_s 108  /tmp/tr_g_pri.npy
tight gate (proxy 7.888):
HELDOUT_PPL 281.775  loss 5.6411  train_s 108  /tmp/tr_g_tight.npy
2026
[metadata] {"noOutputExpected": false}

g_fine proxy H(p_t,p_sel) = 7.854384065328796
g_loose proxy H(p_t,p_sel) = 7.819665179215654
g_cl1k proxy H(p_t,p_sel) = 7.854213848527171
g_bg1k proxy H(p_t,p_sel) = 7.861199568088065
proxy vs dev PPL:  Pearson 0.962  Spearman 0.771  (n=6)
excluding the tight <0.005-nat cluster (min-len 100/150/200):
   coarse-scale runs only: Pearson 0.974
1996

[stdout]
g_fine proxy H(p_t,p_sel) = 7.854384065328796
g_loose proxy H(p_t,p_sel) = 7.819665179215654
g_cl1k proxy H(p_t,p_sel) = 7.854213848527171
g_bg1k proxy H(p_t,p_sel) = 7.861199568088065
proxy vs dev PPL:  Pearson 0.962  Spearman 0.771  (n=6)
excluding the tight <0.005-nat cluster (min-len 100/150/200):
   coarse-scale runs only: Pearson 0.974
1996
[metadata] {"noOutputExpected": false}

The file /workspace/claim.md has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/claim.md", "oldString": "1. **The proxy objective tracks the real one.** The selection reduces unigram\n   `H(p_target, p_selection)` from 8.078 nats (pool background = do nothing) to\n   **7.854 nats**. Across the smoothing choice, the variant achieving the larger\n   proxy *reduction* also trains to lower perplexity: smoothing `p_target` toward\n   the pool background reduces the proxy by 0.224 nats and gives dev PPL 272.9,\n   while smoothing toward uniform reduces its own proxy by only 0.195 nats and\n   gives 279.3 (both at min-length 300). Uniform smoothing spreads ~8% of target\n   mass over ~25k token types the target never emits, and greedy burns budget\n   chasing that noise.", "newString": "1. **The proxy objective tracks the real one — inside a fixed quality gate.** The\n   selection reduces unigram `H(p_target, p_selection)` from 8.078 nats (pool\n   background = do nothing) to **7.854 nats**. Across six selections that share\n   the same target estimate and the same quality gate, the proxy predicts dev\n   perplexity at **Pearson 0.96** (Spearman 0.77, range 267.7–281.8 PPL).\n   I used this prospectively: two variants (a 10× stronger background prior and a\n   tighter quality gate) were built, their proxies read off *before* training, and\n   both were predicted to lose. They did, monotonically —\n   proxy 7.854 → 7.868 → 7.888 gave PPL 267.7 → 275.9 → 281.8.\n   Also, smoothing `p_target` toward the pool background achieves a larger proxy\n   reduction than smoothing toward uniform (0.224 vs 0.195 nats) and a lower PPL\n   (272.9 vs 279.3, both at min-length 300); uniform smoothing spreads ~8% of\n   target mass over ~25k token types the target never emits, and greedy burns\n   budget chasing that noise.\n\n   **Where the proxy fails, and why the gate is not optional:** relaxing the\n   quality gate produced the *lowest* proxy of all runs (7.820) but a *worse* PPL\n   (269.2 vs 267.7). Boilerplate can match unigram frequencies cheaply while being\n   poor training data, so the proxy is only a valid predictor within a fixed\n   candidate set. Neither component substitutes for the other: the gate defines\n   what is admissible, the proxy orders what is admitted. The proxy also does not\n   resolve differences inside a ~0.005-nat cluster (min-length 100/150/200 span\n   0.004 nats and 2 PPL, out of order).", "originalFile": "# Curating a raw web pool by target-distribution matching\n\nSubmitted selection: `submission/selection.json` (18,418 ids, 15.0M tokens; the\nfirst 15,285 fill the 12M budget), produced by `submission/curate.py`.\n**Dev perplexity 267.7 vs 485.5 for a random selection** (frozen recipe, seed 1337).\n\n## Hypothesis\n\nUnder a token budget far smaller than the pool (12M of 200M), held-out perplexity\non a broad multi-domain target is governed mainly by **how closely the training\nsubset's token distribution matches the target's**, and only secondarily by\ngeneric \"document quality\". Concretely:\n\n> Greedily choosing documents to minimise the unigram cross-entropy\n> `H(p_target, p_selection)` — the same functional the metric evaluates, at the\n> highest order estimable from the 1M-token disclosed target — beats random\n> selection, beats quality-filtered random selection, and beats single-axis\n> \"pick the best-looking documents\" ranking.\n\nThe corollary that makes this non-trivial: the pool is *not* short of readable\nprose. It is short of prose in the target's **registers and surface conventions**,\nand a budget-constrained selector must buy those deliberately.\n\n## Mechanism\n\nHeld-out loss is a mean over windows of four equal registers (recovered by\ndecoding `multi_dev.npy`: WikiText-style encyclopedic / clean web prose /\nnewswire / StackExchange HTML Q&A). Mean loss is additive across registers, so\nthe **highest-loss register dominates the score**, and a selection that leaves one\nregister unrepresented pays disproportionately.\n\nMatching `p_target` allocates budget by each register's share of the *target*\nrather than its share of the *pool*. The importance ratio\n`r(v) = p_target(v)/p_selection(v)` is recomputed as the selection grows, so once\na register is covered its tokens stop being attractive and the selector moves on —\ncoverage is self-balancing rather than hand-weighted.\n\n**Predicted observables (not the final perplexity), all measured:**\n\n1. **The proxy objective tracks the real one.** The selection reduces unigram\n   `H(p_target, p_selection)` from 8.078 nats (pool background = do nothing) to\n   **7.854 nats**. Across the smoothing choice, the variant achieving the larger\n   proxy *reduction* also trains to lower perplexity: smoothing `p_target` toward\n   the pool background reduces the proxy by 0.224 nats and gives dev PPL 272.9,\n   while smoothing toward uniform reduces its own proxy by only 0.195 nats and\n   gives 279.3 (both at min-length 300). Uniform smoothing spreads ~8% of target\n   mass over ~25k token types the target never emits, and greedy burns budget\n   chasing that noise.\n2. **Gains land on the register that was starved, not uniformly.** Per-register\n   held-out loss (nats), quality-filtered random → submitted selection:\n\n   | register | qgate-random | submitted | Δ |\n   |---|---|---|---|\n   | technical Q&A | 5.777 | **4.973** | −0.804 |\n   | encyclopedic | 6.724 | **6.260** | −0.464 |\n   | newswire | 5.870 | **5.585** | −0.285 |\n   | web prose | 5.627 | **5.539** | −0.088 |\n\n   The register the raw pool is emptiest of (technical Q&A: **zero** `<pre><code>`\n   documents pool-wide) improves ~9× more than web prose, the register the pool\n   most resembles. This is the signature of register rebalancing, and it is not\n   what a uniform \"better data\" story predicts.\n3. **The selector buys the target's surface conventions at a measurable rate.**\n   The encyclopedic quarter is WikiText-formatted: GPT-2 tokens `' ,'` and `' .'`\n   (space-before-punctuation) are **7.9%** of its tokens but **0.055%** of pool\n   tokens. The submitted selection puts **5.20%** of its budget in documents\n   carrying that convention — 5.3× the random rate (623k vs 119k tokens) — and its\n   first 400 documents are 38× denser in those tokens than random (1962 vs 51 per\n   100k).\n4. **The encyclopedic register stays the worst one no matter what.** Its\n   convention is nearly absent from the pool (0 documents contain ` @-@ `), so it\n   remained the highest-loss quarter in every one of the 20 selections I trained.\n   That is a ceiling imposed by the pool, not by the selector.\n\n## Falsification\n\nThe claim predicts specific *failures*. Each was run as its own training run\n(frozen recipe, seed 1337, dev target). All would have falsified it; none did.\n\n| Selection criterion | dev PPL |\n|---|---|\n| Random (do-nothing baseline) | 485.5 |\n| Quality gate + dedup only, random within it | 403.6 |\n| Single global quality score (LLR vs. target mixture), top-k | 387.0 |\n| Balanced per-register Naive-Bayes round-robin | 312.5 |\n| Greedy match, uniform-smoothed target, min-len 300 | 279.3 |\n| Greedy match, background-smoothed target, min-len 300 | 272.9 |\n| **Greedy match, background-smoothed, min-len 150 (submitted)** | **267.7** |\n\n- *If the junk filter were doing the work*, quality-gate-only random selection\n  would approach the matched selection. It does not: the gate buys 485.5 → 403.6,\n  matching buys 403.6 → 267.7. **Matching contributes ~1.7× more than filtering**,\n  and the two are separable because both selections pass the identical gate.\n- *If a single quality axis sufficed*, ranking documents by their likelihood ratio\n  against the pooled target mixture would match register-aware selection. It does\n  not (387.0 vs 312.5): a single global ranking collapses onto whichever register\n  the pool most resembles.\n- *If the hardest register's surface tokens were the bottleneck*, force-feeding\n  1.5M tokens (12.5% of budget) of the documents richest in `' ,'`/`' .'` would\n  help. It hurt: 279.3 → 290.5 at matched settings. Those documents are nav-bar\n  and sports-ellipsis boilerplate; the objective's own 5.2% allocation is already\n  the right dose, and buying more trades prose for junk.\n- *If the hardest register deserved extra weight*, doubling the encyclopedic\n  register's weight in `p_target` would help. It hurt (288.8 vs 279.3): the\n  equal-parts weighting is already correct, as the metric's additive structure implies.\n- *If longer documents were simply better*, raising the minimum length would help.\n  It hurt monotonically above the optimum: 267.7 (150) → 269.5 (200) → 279.3 (300)\n  → 295.4 (900). The optimum is a bracketed interior minimum (267.7 at both 100 and\n  150), not a monotone preference — past ~150 tokens, extra length costs coverage.\n- *If a more faithful greedy were better*, shrinking the batch from 250 to 80\n  documents per re-scoring step would help. It hurt (273.4): finer steps overfit\n  the unigram proxy, which is a means, not the objective.\n- Also tested and rejected against 267.7: unclipped-er ratio (clip 1000 → 269.0),\n  looser quality gate (269.2), weaker target smoothing (268.1).\n\n**What would still falsify the claim:** a selection with a *higher* proxy\ncross-entropy that trains to lower perplexity; or a quality-only criterion using\nno target sample reaching ~268 at this budget.\n\n**Honest limits.** `p_target` is estimated from the disclosed dev sample, so the\nmatch is only as good as that sample's coverage of the hidden one (stated to be a\ndisjoint sample of the same domain). Five scalar hyperparameters were tuned\nagainst dev, which bounds but does not eliminate selection-on-dev risk; the\nflatness of the optimum (267.7 across min-length 100–150, 268.1 at a 5× different\nsmoothing mass) suggests the setting is not knife-edge. Differences below ~1.5 PPL\nhere are within run-to-run noise and I do not claim them. All runs share one seed,\nso I have not measured seed variance directly.\n\n## Transfer\n\nThe method needs only (a) a sample of target-domain text, (b) the tokenizer, and\n(c) one counting pass — no labels, no reference model, no GPU. The greedy step is\na gather plus a segment-sum, so scoring the full 200M-token pool costs ~1 s and\nthe entire selection ~80 s on CPU; cost is linear in pool tokens and independent\nof how much is finally kept, so it scales to far larger pools.\n\nIt transfers to any budget-constrained data decision where the evaluation\ndistribution is known or samplable: choosing pretraining mixtures, picking\nfine-tuning subsets for a known deployment domain, or setting domain weights\nwithout hand-tuned ratios. Three lessons generalise beyond this task: **smooth the\ntarget toward the achievable background, not toward uniform** (else the selector\nchases unreachable rare tokens); **clip the importance ratio** (some target tokens\ncannot be bought at any price, and an unclipped objective spends the budget\ntrying); and **keep greedy steps coarse** (chasing the proxy too precisely\ndegrades the true metric). The natural extension is the same greedy on\nhigher-order statistics once more target text is available — the unigram order is\na consequence of having only 1M target tokens, not a limit of the criterion.\n", "structuredPatch": [{"oldStart": 37, "oldLines": 15, "newStart": 37, "newLines": 29, "lines": [" ", " **Predicted observables (not the final perplexity), all measured:**", " ", "-1. **The proxy objective tracks the real one.** The selection reduces unigram", "-   `H(p_target, p_selection)` from 8.078 nats (pool background = do nothing) to", "-   **7.854 nats**. Across the smoothing choice, the variant achieving the larger", "-   proxy *reduction* also trains to lower perplexity: smoothing `p_target` toward", "-   the pool background reduces the proxy by 0.224 nats and gives dev PPL 272.9,", "-   while smoothing toward uniform reduces its own proxy by only 0.195 nats and", "-   gives 279.3 (both at min-length 300). Uniform smoothing spreads ~8% of target", "-   mass over ~25k token types the target never emits, and greedy burns budget", "-   chasing that noise.", "+1. **The proxy objective tracks the real one — inside a fixed quality gate.** The", "+   selection reduces unigram `H(p_target, p_selection)` from 8.078 nats (pool", "+   background = do nothing) to **7.854 nats**. Across six selections that share", "+   the same target estimate and the same quality gate, the proxy predicts dev", "+   perplexity at **Pearson 0.96** (Spearman 0.77, range 267.7–281.8 PPL).", "+   I used this prospectively: two variants (a 10× stronger background prior and a", "+   tighter quality gate) were built, their proxies read off *before* training, and", "+   both were predicted to lose. They did, monotonically —", "+   proxy 7.854 → 7.868 → 7.888 gave PPL 267.7 → 275.9 → 281.8.", "+   Also, smoothing `p_target` toward the pool background achieves a larger proxy", "+   reduction than smoothing toward uniform (0.224 vs 0.195 nats) and a lower PPL", "+   (272.9 vs 279.3, both at min-length 300); uniform smoothing spreads ~8% of", "+   target mass over ~25k token types the target never emits, and greedy burns", "+   budget chasing that noise.", "+", "+   **Where the proxy fails, and why the gate is not optional:** relaxing the", "+   quality gate produced the *lowest* proxy of all runs (7.820) but a *worse* PPL", "+   (269.2 vs 267.7). Boilerplate can match unigram frequencies cheaply while being", "+   poor training data, so the proxy is only a valid predictor within a fixed", "+   candidate set. Neither component substitutes for the other: the gate defines", "+   what is admissible, the proxy orders what is admitted. The proxy also does not", "+   resolve differences inside a ~0.005-nat cluster (min-length 100/150/200 span", "+   0.004 nats and 2 PPL, out of order).", " 2. **Gains land on the register that was starved, not uniformly.** Per-register", "    held-out loss (nats), quality-filtered random → submitted selection:", " "]}], "userModified": false, "replaceAll": false}

The file /workspace/claim.md has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/claim.md", "oldString": "- Also tested and rejected against 267.7: unclipped-er ratio (clip 1000 → 269.0),\n  looser quality gate (269.2), weaker target smoothing (268.1).", "newString": "- Also tested and rejected against 267.7: a looser ratio clip (1000 → 269.0), a\n  looser quality gate (269.2), a tighter quality gate (281.8), weaker target\n  smoothing (268.1), and a 10× stronger background prior (275.9). The gate results\n  bracket the submitted setting on both sides, so it is an interior optimum rather\n  than the endpoint of a one-sided sweep.", "originalFile": "# Curating a raw web pool by target-distribution matching\n\nSubmitted selection: `submission/selection.json` (18,418 ids, 15.0M tokens; the\nfirst 15,285 fill the 12M budget), produced by `submission/curate.py`.\n**Dev perplexity 267.7 vs 485.5 for a random selection** (frozen recipe, seed 1337).\n\n## Hypothesis\n\nUnder a token budget far smaller than the pool (12M of 200M), held-out perplexity\non a broad multi-domain target is governed mainly by **how closely the training\nsubset's token distribution matches the target's**, and only secondarily by\ngeneric \"document quality\". Concretely:\n\n> Greedily choosing documents to minimise the unigram cross-entropy\n> `H(p_target, p_selection)` — the same functional the metric evaluates, at the\n> highest order estimable from the 1M-token disclosed target — beats random\n> selection, beats quality-filtered random selection, and beats single-axis\n> \"pick the best-looking documents\" ranking.\n\nThe corollary that makes this non-trivial: the pool is *not* short of readable\nprose. It is short of prose in the target's **registers and surface conventions**,\nand a budget-constrained selector must buy those deliberately.\n\n## Mechanism\n\nHeld-out loss is a mean over windows of four equal registers (recovered by\ndecoding `multi_dev.npy`: WikiText-style encyclopedic / clean web prose /\nnewswire / StackExchange HTML Q&A). Mean loss is additive across registers, so\nthe **highest-loss register dominates the score**, and a selection that leaves one\nregister unrepresented pays disproportionately.\n\nMatching `p_target` allocates budget by each register's share of the *target*\nrather than its share of the *pool*. The importance ratio\n`r(v) = p_target(v)/p_selection(v)` is recomputed as the selection grows, so once\na register is covered its tokens stop being attractive and the selector moves on —\ncoverage is self-balancing rather than hand-weighted.\n\n**Predicted observables (not the final perplexity), all measured:**\n\n1. **The proxy objective tracks the real one — inside a fixed quality gate.** The\n   selection reduces unigram `H(p_target, p_selection)` from 8.078 nats (pool\n   background = do nothing) to **7.854 nats**. Across six selections that share\n   the same target estimate and the same quality gate, the proxy predicts dev\n   perplexity at **Pearson 0.96** (Spearman 0.77, range 267.7–281.8 PPL).\n   I used this prospectively: two variants (a 10× stronger background prior and a\n   tighter quality gate) were built, their proxies read off *before* training, and\n   both were predicted to lose. They did, monotonically —\n   proxy 7.854 → 7.868 → 7.888 gave PPL 267.7 → 275.9 → 281.8.\n   Also, smoothing `p_target` toward the pool background achieves a larger proxy\n   reduction than smoothing toward uniform (0.224 vs 0.195 nats) and a lower PPL\n   (272.9 vs 279.3, both at min-length 300); uniform smoothing spreads ~8% of\n   target mass over ~25k token types the target never emits, and greedy burns\n   budget chasing that noise.\n\n   **Where the proxy fails, and why the gate is not optional:** relaxing the\n   quality gate produced the *lowest* proxy of all runs (7.820) but a *worse* PPL\n   (269.2 vs 267.7). Boilerplate can match unigram frequencies cheaply while being\n   poor training data, so the proxy is only a valid predictor within a fixed\n   candidate set. Neither component substitutes for the other: the gate defines\n   what is admissible, the proxy orders what is admitted. The proxy also does not\n   resolve differences inside a ~0.005-nat cluster (min-length 100/150/200 span\n   0.004 nats and 2 PPL, out of order).\n2. **Gains land on the register that was starved, not uniformly.** Per-register\n   held-out loss (nats), quality-filtered random → submitted selection:\n\n   | register | qgate-random | submitted | Δ |\n   |---|---|---|---|\n   | technical Q&A | 5.777 | **4.973** | −0.804 |\n   | encyclopedic | 6.724 | **6.260** | −0.464 |\n   | newswire | 5.870 | **5.585** | −0.285 |\n   | web prose | 5.627 | **5.539** | −0.088 |\n\n   The register the raw pool is emptiest of (technical Q&A: **zero** `<pre><code>`\n   documents pool-wide) improves ~9× more than web prose, the register the pool\n   most resembles. This is the signature of register rebalancing, and it is not\n   what a uniform \"better data\" story predicts.\n3. **The selector buys the target's surface conventions at a measurable rate.**\n   The encyclopedic quarter is WikiText-formatted: GPT-2 tokens `' ,'` and `' .'`\n   (space-before-punctuation) are **7.9%** of its tokens but **0.055%** of pool\n   tokens. The submitted selection puts **5.20%** of its budget in documents\n   carrying that convention — 5.3× the random rate (623k vs 119k tokens) — and its\n   first 400 documents are 38× denser in those tokens than random (1962 vs 51 per\n   100k).\n4. **The encyclopedic register stays the worst one no matter what.** Its\n   convention is nearly absent from the pool (0 documents contain ` @-@ `), so it\n   remained the highest-loss quarter in every one of the 20 selections I trained.\n   That is a ceiling imposed by the pool, not by the selector.\n\n## Falsification\n\nThe claim predicts specific *failures*. Each was run as its own training run\n(frozen recipe, seed 1337, dev target). All would have falsified it; none did.\n\n| Selection criterion | dev PPL |\n|---|---|\n| Random (do-nothing baseline) | 485.5 |\n| Quality gate + dedup only, random within it | 403.6 |\n| Single global quality score (LLR vs. target mixture), top-k | 387.0 |\n| Balanced per-register Naive-Bayes round-robin | 312.5 |\n| Greedy match, uniform-smoothed target, min-len 300 | 279.3 |\n| Greedy match, background-smoothed target, min-len 300 | 272.9 |\n| **Greedy match, background-smoothed, min-len 150 (submitted)** | **267.7** |\n\n- *If the junk filter were doing the work*, quality-gate-only random selection\n  would approach the matched selection. It does not: the gate buys 485.5 → 403.6,\n  matching buys 403.6 → 267.7. **Matching contributes ~1.7× more than filtering**,\n  and the two are separable because both selections pass the identical gate.\n- *If a single quality axis sufficed*, ranking documents by their likelihood ratio\n  against the pooled target mixture would match register-aware selection. It does\n  not (387.0 vs 312.5): a single global ranking collapses onto whichever register\n  the pool most resembles.\n- *If the hardest register's surface tokens were the bottleneck*, force-feeding\n  1.5M tokens (12.5% of budget) of the documents richest in `' ,'`/`' .'` would\n  help. It hurt: 279.3 → 290.5 at matched settings. Those documents are nav-bar\n  and sports-ellipsis boilerplate; the objective's own 5.2% allocation is already\n  the right dose, and buying more trades prose for junk.\n- *If the hardest register deserved extra weight*, doubling the encyclopedic\n  register's weight in `p_target` would help. It hurt (288.8 vs 279.3): the\n  equal-parts weighting is already correct, as the metric's additive structure implies.\n- *If longer documents were simply better*, raising the minimum length would help.\n  It hurt monotonically above the optimum: 267.7 (150) → 269.5 (200) → 279.3 (300)\n  → 295.4 (900). The optimum is a bracketed interior minimum (267.7 at both 100 and\n  150), not a monotone preference — past ~150 tokens, extra length costs coverage.\n- *If a more faithful greedy were better*, shrinking the batch from 250 to 80\n  documents per re-scoring step would help. It hurt (273.4): finer steps overfit\n  the unigram proxy, which is a means, not the objective.\n- Also tested and rejected against 267.7: unclipped-er ratio (clip 1000 → 269.0),\n  looser quality gate (269.2), weaker target smoothing (268.1).\n\n**What would still falsify the claim:** a selection with a *higher* proxy\ncross-entropy that trains to lower perplexity; or a quality-only criterion using\nno target sample reaching ~268 at this budget.\n\n**Honest limits.** `p_target` is estimated from the disclosed dev sample, so the\nmatch is only as good as that sample's coverage of the hidden one (stated to be a\ndisjoint sample of the same domain). Five scalar hyperparameters were tuned\nagainst dev, which bounds but does not eliminate selection-on-dev risk; the\nflatness of the optimum (267.7 across min-length 100–150, 268.1 at a 5× different\nsmoothing mass) suggests the setting is not knife-edge. Differences below ~1.5 PPL\nhere are within run-to-run noise and I do not claim them. All runs share one seed,\nso I have not measured seed variance directly.\n\n## Transfer\n\nThe method needs only (a) a sample of target-domain text, (b) the tokenizer, and\n(c) one counting pass — no labels, no reference model, no GPU. The greedy step is\na gather plus a segment-sum, so scoring the full 200M-token pool costs ~1 s and\nthe entire selection ~80 s on CPU; cost is linear in pool tokens and independent\nof how much is finally kept, so it scales to far larger pools.\n\nIt transfers to any budget-constrained data decision where the evaluation\ndistribution is known or samplable: choosing pretraining mixtures, picking\nfine-tuning subsets for a known deployment domain, or setting domain weights\nwithout hand-tuned ratios. Three lessons generalise beyond this task: **smooth the\ntarget toward the achievable background, not toward uniform** (else the selector\nchases unreachable rare tokens); **clip the importance ratio** (some target tokens\ncannot be bought at any price, and an unclipped objective spends the budget\ntrying); and **keep greedy steps coarse** (chasing the proxy too precisely\ndegrades the true metric). The natural extension is the same greedy on\nhigher-order statistics once more target text is available — the unigram order is\na consequence of having only 1M target tokens, not a limit of the criterion.\n", "structuredPatch": [{"oldStart": 124, "oldLines": 8, "newStart": 124, "newLines": 11, "lines": [" - *If a more faithful greedy were better*, shrinking the batch from 250 to 80", "   documents per re-scoring step would help. It hurt (273.4): finer steps overfit", "   the unigram proxy, which is a means, not the objective.", "-- Also tested and rejected against 267.7: unclipped-er ratio (clip 1000 → 269.0),", "-  looser quality gate (269.2), weaker target smoothing (268.1).", "+- Also tested and rejected against 267.7: a looser ratio clip (1000 → 269.0), a", "+  looser quality gate (269.2), a tighter quality gate (281.8), weaker target", "+  smoothing (268.1), and a 10× stronger background prior (275.9). The gate results", "+  bracket the submitted setting on both sides, so it is an interior optimum rather", "+  than the endpoint of a one-sided sweep.", " ", " **What would still falsify the claim:** a selection with a *higher* proxy", " cross-entropy that trains to lower perplexity; or a quality-only criterion using"]}], "userModified": false, "replaceAll": false}