claude-opus-4-8

The file /workspace/submission/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"type": "update", "filePath": "/workspace/submission/curate.py", "content": "\"\"\"Curate the raw web pool for pretraining a small LM.\n\nCriterion: DSIR-style importance selection against the disclosed target.\n\nThe disclosed evaluation target is a BROAD, equal-parts mixture of clean English\nregisters: encyclopedic (Wikipedia), high-quality web prose, news, and technical\nQ&A. We model that target as a reference n-gram distribution (decoded from the\nprovided multi_dev sample) and the raw pool as a background distribution. Each pool\ndocument is scored by its average log-likelihood ratio\n\n    score(d) = mean_{w in features(d)} log p_target(w) / p_pool(w)\n\nover unigram+bigram word features. Documents whose vocabulary matches the target's\nclean multi-domain prose score high; forum chatter, spam, boilerplate and\nlink-farms score low. A light quality gate removes degenerate docs (too short,\nsymbol-dominated, or highly repetitive) before ranking.\n\nEmpirically this global ranking beats both a random baseline and a register-\nstratified variant: the pool contains little genuine code-Q&A, so quota-balancing\nmerely dilutes the selection with weaker matches, whereas ranking by target-\nlikeness surfaces the cleanest prose, which transfers across all four registers.\n\nOutput: selection.json = pool ids ordered best-first. The training pipeline\nconsumes them in order until the 12M-token budget is filled (~top 13k docs).\n\"\"\"\nimport json, re, math, numpy as np\nfrom collections import Counter\n\nPOOL = \"/workspace/data/pool.jsonl\"\nTARGET = \"/workspace/data/multi_dev.npy\"\nOUT = \"/workspace/submission/selection.json\"\n\nWORD = re.compile(r\"[a-z0-9']+\")\ndef toks(s): return WORD.findall(s.lower())\ndef feats(ws):\n    f = list(ws)\n    for i in range(len(ws) - 1):\n        f.append(ws[i] + \" \" + ws[i + 1])\n    return f\n\n# ---- disclosed target distribution ----\nfrom transformers import AutoTokenizer\ntk = AutoTokenizer.from_pretrained(\"gpt2\")\ntgt_text = tk.decode(np.load(TARGET).tolist())\ntgt_c = Counter(feats(toks(tgt_text)))\nNt = sum(tgt_c.values())\n\n# ---- load pool + background distribution ----\nids, texts, tokd = [], [], []\npool_c = Counter()\nfor line in open(POOL):\n    r = json.loads(line)\n    ids.append(r[\"id\"]); texts.append(r[\"text\"])\n    ws = toks(r[\"text\"]); tokd.append(ws)\n    pool_c.update(feats(ws))\nNp = sum(pool_c.values())\n\nV = len(tgt_c); a = 1.0\nlogratio = {}\nfor w, ct in tgt_c.items():\n    pt = (ct + a) / (Nt + a * V)\n    pp = (pool_c.get(w, 0) + a) / (Np + a * V)\n    logratio[w] = math.log(pt / pp)\n\ndef quality_ok(ws, text):\n    n = len(ws)\n    if n < 50: return False\n    alpha = sum(c.isalpha() or c.isspace() for c in text)\n    if alpha / max(1, len(text)) < 0.45: return False\n    if len(set(ws)) / n < 0.25: return False\n    return True\n\nscores = []\nfor i, ws in enumerate(tokd):\n    if not quality_ok(ws, texts[i]): continue\n    fs = feats(ws)\n    s = 0.0\n    for w in fs:\n        r = logratio.get(w)\n        if r is not None: s += r\n    scores.append((s / len(fs), ids[i]))\n\nscores.sort(reverse=True)\nsel = [i for _, i in scores]\njson.dump(sel, open(OUT, \"w\"))\nprint(f\"scored {len(scores)}/{len(ids)} docs; wrote {len(sel)} ids -> {OUT}\")\nprint(\"top\", scores[0][0], \"median\", scores[len(scores)//2][0])\n", "structuredPatch": [{"oldStart": 1, "oldLines": 22, "newStart": 1, "newLines": 27, "lines": [" \"\"\"Curate the raw web pool for pretraining a small LM.", " ", "-Criterion: STRATIFIED DSIR-style importance selection. The disclosed target is a", "-BROAD, equal-parts mixture of four registers of clean English — encyclopedic,", "-high-quality web prose, news, and technical Q&A. A single importance ranking over", "-the whole target collapses onto whichever register is easiest to match (here: news", "-and general web), starving the others — empirically the naive ranking yields ~0%", "-technical Q&A, so the model never learns that quarter of the target and its", "-perplexity there explodes.", "+Criterion: DSIR-style importance selection against the disclosed target.", " ", "-Fix: partition the disclosed target (multi_dev) into four contiguous segments that", "-correspond to its four registers, and build a separate n-gram reference", "-distribution for each. Score every pool document against all four references", "-(average log-likelihood ratio log p_seg(w)/p_pool(w) over unigram+bigram word", "-features), assign it to its best-matching register, and select the top documents", "-per register in round-robin so the training mix is balanced ~25% across registers.", "-A light quality gate removes degenerate docs first.", "+The disclosed evaluation target is a BROAD, equal-parts mixture of clean English", "+registers: encyclopedic (Wikipedia), high-quality web prose, news, and technical", "+Q&A. We model that target as a reference n-gram distribution (decoded from the", "+provided multi_dev sample) and the raw pool as a background distribution. Each pool", "+document is scored by its average log-likelihood ratio", " ", "-Output: selection.json = pool ids ordered best-first (round-robin over registers).", "+    score(d) = mean_{w in features(d)} log p_target(w) / p_pool(w)", "+", "+over unigram+bigram word features. Documents whose vocabulary matches the target's", "+clean multi-domain prose score high; forum chatter, spam, boilerplate and", "+link-farms score low. A light quality gate removes degenerate docs (too short,", "+symbol-dominated, or highly repetitive) before ranking.", "+", "+Empirically this global ranking beats both a random baseline and a register-", "+stratified variant: the pool contains little genuine code-Q&A, so quota-balancing", "+merely dilutes the selection with weaker matches, whereas ranking by target-", "+likeness surfaces the cleanest prose, which transfers across all four registers.", "+", "+Output: selection.json = pool ids ordered best-first. The training pipeline", "+consumes them in order until the 12M-token budget is filled (~top 13k docs).", " \"\"\"", " import json, re, math, numpy as np", " from collections import Counter"]}, {"oldStart": 24, "oldLines": 7, "newStart": 29, "newLines": 6, "lines": [" POOL = \"/workspace/data/pool.jsonl\"", " TARGET = \"/workspace/data/multi_dev.npy\"", " OUT = \"/workspace/submission/selection.json\"", "-NSEG = 4", " ", " WORD = re.compile(r\"[a-z0-9']+\")", " def toks(s): return WORD.findall(s.lower())"]}, {"oldStart": 34, "oldLines": 16, "newStart": 38, "newLines": 12, "lines": ["         f.append(ws[i] + \" \" + ws[i + 1])", "     return f", " ", "-# ---- disclosed target, split into NSEG contiguous register-segments ----", "+# ---- disclosed target distribution ----", " from transformers import AutoTokenizer", " tk = AutoTokenizer.from_pretrained(\"gpt2\")", "-tgt_ids = np.load(TARGET).tolist()", "-seg_counts, seg_N = [], []", "-L = len(tgt_ids) // NSEG", "-for s in range(NSEG):", "-    txt = tk.decode(tgt_ids[s * L:(s + 1) * L])", "-    c = Counter(feats(toks(txt)))", "-    seg_counts.append(c); seg_N.append(sum(c.values()))", "+tgt_text = tk.decode(np.load(TARGET).tolist())", "+tgt_c = Counter(feats(toks(tgt_text)))", "+Nt = sum(tgt_c.values())", " ", " # ---- load pool + background distribution ----", " ids, texts, tokd = [], [], []"]}, {"oldStart": 55, "oldLines": 55, "newStart": 55, "newLines": 33, "lines": ["     pool_c.update(feats(ws))", " Np = sum(pool_c.values())", " ", "-# union vocab across segments (for smoothing denom)", "-V = len(set().union(*[set(c) for c in seg_counts]))", "-a = 1.0", "-seg_lr = []", "-for c, N in zip(seg_counts, seg_N):", "-    lr = {}", "-    for w, ct in c.items():", "-        pt = (ct + a) / (N + a * V)", "-        pp = (pool_c.get(w, 0) + a) / (Np + a * V)", "-        lr[w] = math.log(pt / pp)", "-    seg_lr.append(lr)", "+V = len(tgt_c); a = 1.0", "+logratio = {}", "+for w, ct in tgt_c.items():", "+    pt = (ct + a) / (Nt + a * V)", "+    pp = (pool_c.get(w, 0) + a) / (Np + a * V)", "+    logratio[w] = math.log(pt / pp)", " ", " def quality_ok(ws, text):", "     n = len(ws)", "     if n < 50: return False", "     alpha = sum(c.isalpha() or c.isspace() for c in text)", "-    if alpha / max(1, len(text)) < 0.45: return False   # keep code-bearing Q&A", "+    if alpha / max(1, len(text)) < 0.45: return False", "     if len(set(ws)) / n < 0.25: return False", "     return True", " ", "-# ---- score every doc against each register, assign to best register ----", "-buckets = [[] for _ in range(NSEG)]", "-for idx, ws in enumerate(tokd):", "-    if not quality_ok(ws, texts[idx]): continue", "-    fs = feats(ws); n = len(fs)", "-    best_s, best_seg = None, 0", "-    for s in range(NSEG):", "-        lr = seg_lr[s]", "-        sc = 0.0", "-        for w in fs:", "-            r = lr.get(w)", "-            if r is not None: sc += r", "-        sc /= n", "-        if best_s is None or sc > best_s:", "-            best_s, best_seg = sc, s", "-    buckets[best_seg].append((best_s, ids[idx]))", "+scores = []", "+for i, ws in enumerate(tokd):", "+    if not quality_ok(ws, texts[i]): continue", "+    fs = feats(ws)", "+    s = 0.0", "+    for w in fs:", "+        r = logratio.get(w)", "+        if r is not None: s += r", "+    scores.append((s / len(fs), ids[i]))", " ", "-for b in buckets: b.sort(reverse=True)", "-sizes = [len(b) for b in buckets]", "-print(\"register bucket sizes:\", sizes)", "-", "-# ---- round-robin interleave -> balanced ~25% mix in priority order ----", "-sel, ptr = [], [0] * NSEG", "-while True:", "-    progressed = False", "-    for s in range(NSEG):", "-        if ptr[s] < len(buckets[s]):", "-            sel.append(buckets[s][ptr[s]][1]); ptr[s] += 1; progressed = True", "-    if not progressed: break", "-", "+scores.sort(reverse=True)", "+sel = [i for _, i in scores]", " json.dump(sel, open(OUT, \"w\"))", "-print(f\"wrote {len(sel)} ids -> {OUT}\")", "+print(f\"scored {len(scores)}/{len(ids)} docs; wrote {len(sel)} ids -> {OUT}\")", "+print(\"top\", scores[0][0], \"median\", scores[len(scores)//2][0])"]}], "originalFile": "\"\"\"Curate the raw web pool for pretraining a small LM.\n\nCriterion: STRATIFIED DSIR-style importance selection. The disclosed target is a\nBROAD, equal-parts mixture of four registers of clean English — encyclopedic,\nhigh-quality web prose, news, and technical Q&A. A single importance ranking over\nthe whole target collapses onto whichever register is easiest to match (here: news\nand general web), starving the others — empirically the naive ranking yields ~0%\ntechnical Q&A, so the model never learns that quarter of the target and its\nperplexity there explodes.\n\nFix: partition the disclosed target (multi_dev) into four contiguous segments that\ncorrespond to its four registers, and build a separate n-gram reference\ndistribution for each. Score every pool document against all four references\n(average log-likelihood ratio log p_seg(w)/p_pool(w) over unigram+bigram word\nfeatures), assign it to its best-matching register, and select the top documents\nper register in round-robin so the training mix is balanced ~25% across registers.\nA light quality gate removes degenerate docs first.\n\nOutput: selection.json = pool ids ordered best-first (round-robin over registers).\n\"\"\"\nimport json, re, math, numpy as np\nfrom collections import Counter\n\nPOOL = \"/workspace/data/pool.jsonl\"\nTARGET = \"/workspace/data/multi_dev.npy\"\nOUT = \"/workspace/submission/selection.json\"\nNSEG = 4\n\nWORD = re.compile(r\"[a-z0-9']+\")\ndef toks(s): return WORD.findall(s.lower())\ndef feats(ws):\n    f = list(ws)\n    for i in range(len(ws) - 1):\n        f.append(ws[i] + \" \" + ws[i + 1])\n    return f\n\n# ---- disclosed target, split into NSEG contiguous register-segments ----\nfrom transformers import AutoTokenizer\ntk = AutoTokenizer.from_pretrained(\"gpt2\")\ntgt_ids = np.load(TARGET).tolist()\nseg_counts, seg_N = [], []\nL = len(tgt_ids) // NSEG\nfor s in range(NSEG):\n    txt = tk.decode(tgt_ids[s * L:(s + 1) * L])\n    c = Counter(feats(toks(txt)))\n    seg_counts.append(c); seg_N.append(sum(c.values()))\n\n# ---- load pool + background distribution ----\nids, texts, tokd = [], [], []\npool_c = Counter()\nfor line in open(POOL):\n    r = json.loads(line)\n    ids.append(r[\"id\"]); texts.append(r[\"text\"])\n    ws = toks(r[\"text\"]); tokd.append(ws)\n    pool_c.update(feats(ws))\nNp = sum(pool_c.values())\n\n# union vocab across segments (for smoothing denom)\nV = len(set().union(*[set(c) for c in seg_counts]))\na = 1.0\nseg_lr = []\nfor c, N in zip(seg_counts, seg_N):\n    lr = {}\n    for w, ct in c.items():\n        pt = (ct + a) / (N + a * V)\n        pp = (pool_c.get(w, 0) + a) / (Np + a * V)\n        lr[w] = math.log(pt / pp)\n    seg_lr.append(lr)\n\ndef quality_ok(ws, text):\n    n = len(ws)\n    if n < 50: return False\n    alpha = sum(c.isalpha() or c.isspace() for c in text)\n    if alpha / max(1, len(text)) < 0.45: return False   # keep code-bearing Q&A\n    if len(set(ws)) / n < 0.25: return False\n    return True\n\n# ---- score every doc against each register, assign to best register ----\nbuckets = [[] for _ in range(NSEG)]\nfor idx, ws in enumerate(tokd):\n    if not quality_ok(ws, texts[idx]): continue\n    fs = feats(ws); n = len(fs)\n    best_s, best_seg = None, 0\n    for s in range(NSEG):\n        lr = seg_lr[s]\n        sc = 0.0\n        for w in fs:\n            r = lr.get(w)\n            if r is not None: sc += r\n        sc /= n\n        if best_s is None or sc > best_s:\n            best_s, best_seg = sc, s\n    buckets[best_seg].append((best_s, ids[idx]))\n\nfor b in buckets: b.sort(reverse=True)\nsizes = [len(b) for b in buckets]\nprint(\"register bucket sizes:\", sizes)\n\n# ---- round-robin interleave -> balanced ~25% mix in priority order ----\nsel, ptr = [], [0] * NSEG\nwhile True:\n    progressed = False\n    for s in range(NSEG):\n        if ptr[s] < len(buckets[s]):\n            sel.append(buckets[s][ptr[s]][1]); ptr[s] += 1; progressed = True\n    if not progressed: break\n\njson.dump(sel, open(OUT, \"w\"))\nprint(f\"wrote {len(sel)} ids -> {OUT}\")\n", "userModified": false}

started

[stdout]
started
[metadata] {"noOutputExpected": false}

(Bash completed with no output)

[metadata] {"noOutputExpected": false}

scored 173880/182016 docs; wrote 173880 ids -> /workspace/submission/selection.json
top 0.6716788079331899 median -0.15164550234564164

[stdout]
scored 173880/182016 docs; wrote 173880 ids -> /workspace/submission/selection.json
top 0.6716788079331899 median -0.15164550234564164
[metadata] {"noOutputExpected": false}

ids 173880 unique OK, range 0 182015
HELDOUT_PPL 340.294  loss 5.8298  train_s 51  /tmp/dev_train.npy

[stdout]
ids 173880 unique OK, range 0 182015
HELDOUT_PPL 340.294  loss 5.8298  train_s 51  /tmp/dev_train.npy
[metadata] {"noOutputExpected": false}

if n < 50: return False
    if alpha / max(1, len(text)) < 0.6: return False
    if len(set(ws)) / n < 0.25: return False
started

[stdout]
    if n < 50: return False
    if alpha / max(1, len(text)) < 0.6: return False
    if len(set(ws)) / n < 0.25: return False
started
[metadata] {"noOutputExpected": false}

(Bash completed with no output)

[metadata] {"noOutputExpected": false}

scored 173739/182016 docs; wrote 173739 ids -> /workspace/submission/selection.json
top 0.6716788079331899 median -0.1515142580524862
TIMER
3397

[stdout]
scored 173739/182016 docs; wrote 173739 ids -> /workspace/submission/selection.json
top 0.6716788079331899 median -0.1515142580524862
TIMER
3397
[metadata] {"noOutputExpected": false}

HELDOUT_PPL 335.740  loss 5.8163  train_s 51  /tmp/dev_train.npy

[stdout]
HELDOUT_PPL 335.740  loss 5.8163  train_s 51  /tmp/dev_train.npy
[metadata] {"noOutputExpected": false}

File created successfully at: /workspace/claim.md (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/workspace/claim.md", "content": "# Curation claim: importance selection toward the disclosed broad target\n\n## Hypothesis\nThe pool is raw web text of wildly varying quality; only ~6% of it fits the 12M-token\nbudget. Held-out perplexity on the disclosed broad, multi-domain high-quality English\ntarget (encyclopedic + high-quality web + news + technical Q&A) is minimized by\ntraining on the pool documents whose word-distribution most resembles that target,\nrather than on a random draw. Concretely: rank every pool doc by its average\nlog-likelihood ratio `mean_w log p_target(w)/p_pool(w)` over unigram+bigram word\nfeatures (DSIR-style importance), gate out degenerate docs, and train on the top ranks\nuntil the budget fills (~13k docs).\n\n## Mechanism (prediction of an observable *other* than final perplexity)\nThe ranking acts as a domain/quality classifier. Predicted, verifiable observables:\n1. **Composition shift.** The top ~13k selected docs (the ones actually trained on)\n   are dominated by clean news + high-quality web prose, with forum chatter, spam,\n   boilerplate and link-farms pushed to the bottom of the ranking. *Observed:* top\n   docs are coherent news/encyclopedic/RAND-style prose; a forum post (\"ostriker\n   wrote…\") first appears only around rank 30k, far past the budget cutoff.\n2. **Training-loss gap.** On identical frozen hyperparameters, the curated selection\n   reaches a lower final train loss than a random selection because its tokens are\n   more self-consistent and on-distribution. *Observed:* train loss 5.82 (curated) vs\n   6.13 (random) at 3000 iters.\n3. **Register bias of a single ranking.** A global target ranking collapses onto the\n   easiest-to-match registers. *Observed:* the top 15k contains ~55% news / ~45%\n   general web and essentially no code-bearing technical Q&A — the pool simply\n   contains little genuine StackOverflow-style text.\n\n## Falsification\nThe claim is false if any of:\n- The curated selection does **not** beat a random selection of equal token budget on\n  held-out PPL. *Result: curated 335.7 vs random 457.5 — not falsified (−27%).*\n- Register-balancing the mix (quota 25% per register via stratified DSIR) beats the\n  global ranking. *Result: stratified variant scored 361.4 > 335.7 — balancing HURTS,\n  because the pool lacks true code-Q&A so quotas dilute the top with weaker matches.\n  Confirms the global ranking, not falsified.*\n- Loosening the quality gate to admit symbol-heavy docs improves PPL. *Result:\n  gate 0.45 → 340.3 vs gate 0.60 → 335.7; loosening hurts. Not falsified.*\n\n## Transfer\nThe method needs only a sample of the target distribution and the raw pool; it is\ndomain-agnostic. Given a different disclosed target (e.g. pure code, or biomedical),\nrebuilding `p_target` from a sample of that target and re-ranking would select the\nmatching slice of the same pool — no labels, classifier training, or manual lists\nrequired. It also scales: the score is a single streaming pass of n-gram counting,\nindependent of model size or token budget.\n", "structuredPatch": [], "originalFile": null, "userModified": false}

selection.json valid: 173739 unique ids in range
-rw-r--r-- 1 root root 3010 Aug  4 03:58 claim.md

submission/:
total 1254
drwxr-xr-x 1 root root     180 Aug  4 03:52 .
drwxr-xr-x 1 root root     180 Aug  4 03:58 ..
-rw-r--r-- 1 root root    3158 Aug  4 03:52 curate.py
-rw-r--r-- 1 root root 1280187 Aug  4 03:55 selection.json

[stdout]
selection.json valid: 173739 unique ids in range
-rw-r--r-- 1 root root 3010 Aug  4 03:58 claim.md

submission/:
total 1254
drwxr-xr-x 1 root root     180 Aug  4 03:52 .
drwxr-xr-x 1 root root     180 Aug  4 03:58 ..
-rw-r--r-- 1 root root    3158 Aug  4 03:52 curate.py
-rw-r--r-- 1 root root 1280187 Aug  4 03:55 selection.json
[metadata] {"noOutputExpected": false}