claude-opus-5

{
  "command": "python - <<'EOF'\ns=open('submission/curate.py').read()\n# two-pass: rescore the top candidates using the FULL document, not just the first 3000 chars\ns=s.replace('''np.save(\"/tmp/ml_scores.npy\", scores)\npickle.dump(rscores, open(\"/tmp/reg_scores.pkl\", \"wb\"))''',\n'''np.save(\"/tmp/ml_scores.npy\", scores)\npickle.dump(rscores, open(\"/tmp/reg_scores.pkl\", \"wb\"))\n\n# Pass 2: the pass-1 score only sees the first 3000 chars, which lets a document\n# with a clean lede but a junk body slip through. Rescore the leading candidates\n# on their FULL text (a doc must be good end-to-end, since training uses all of it).\ncand = [i for i in np.argsort(-scores)[:45000] if scores[i] > -1e8]\nfor k, i in enumerate(cand):\n    w = [vi.get(x, UNK) for x in toks(texts[i][:40000])]\n    if len(w) < 30: continue\n    scores[i] = xent(w, bg_lm) - xent(w, tgt_lm)\n    if k % 20000 == 0: print(\"  pass2\", k, time.time()-t0, flush=True)\nscores[[i for i in range(len(texts)) if scores[i] > -1e8 and i not in set(cand)]] = -1e8 + 1\nnp.save(\"/tmp/ml_scores2.npy\", scores)\nprint(\"pass2 done\", time.time()-t0, flush=True)''')\nopen('/tmp/curate_c.py','w').write(s)\nEOF\nnohup python /tmp/curate_c.py > /tmp/cur3.log 2>&1 &\nsleep 5; echo started"
}

{
  "file_path": "/workspace/claim.md",
  "content": "# Claim: target-matched Moore–Lewis selection beats random web at a fixed token budget\n\n## Hypothesis\n\nUnder a fixed 12M-token budget, held-out perplexity on a broad high-quality English\ntarget is governed mainly by **how closely the training distribution matches the target\nregister mix**, not by how much of the pool is covered. Concretely: ranking pool\ndocuments by a *contrastive* n-gram score — cross-entropy under a background LM fit to\nthe raw pool minus cross-entropy under an LM fit to the target register — and training\non the strict top of that ranking will substantially beat a random draw of the same\npool, and the advantage will grow monotonically as the selection is made *more*\nselective (a sharper cut of the ranking), not less.\n\nThe selection is a two-stage criterion (see `submission/curate.py`):\n1. **Hard quality filter** (Gopher/C4-style): length, mean word length, English\n   stop-word presence, alphabetic fraction, ASCII fraction, duplicate-line fraction,\n   short-line (nav-menu / link-farm) fraction, sentence-punctuation density, URL\n   density. This removes boilerplate, spam and non-English text: 87,776 of 182,016\n   documents survive.\n2. **Moore–Lewis domain score** on the survivors, `H_pool(d) − H_target(d)`, using\n   interpolated unigram+bigram word LMs. The target LM is fit to the decoded dev\n   target with its WikiText detokenization artifacts (` @-@ `, ` @,@ `, ` @.@ `,\n   spaced punctuation) stripped, so the score keys on *content register* rather than\n   on a formatting quirk no pool document could ever match.\n   Documents are then emitted in descending score order.\n\n## Mechanism — the observable prediction (not the final perplexity)\n\nThe mechanism is distribution matching, so the prediction is about the **shape of the\nselectivity curve**, which is observable independently of the final score:\n\n> Held-out perplexity should be *monotonically increasing* in the width of the score\n> band the budget is drawn from. If the same criterion is used to select a band of\n> `T` tokens which is then shuffled (so the 12M budget is a random draw from the top\n> `T`), perplexity should rise steadily as `T` grows from 12M toward the whole\n> filtered pool, with no diversity-driven optimum in between.\n\nThis is the discriminating prediction. A \"diversity/coverage\" account of small-model\npretraining predicts the opposite in this regime: with only ~2 epochs over 12M tokens,\ntoo narrow a slice should overfit a register and hurt broad-target perplexity, giving a\nU-shaped curve with a minimum at some `T > 12M`.\n\n**Measured** (frozen `train_nano.py`, `data/multi_dev.npy`, seed 1337):\n\n| selection | dev PPL |\n|---|---|\n| random draw from pool (do-nothing baseline) | 469.9 |\n| filtered pool, band `T=36M`, shuffled | 426.1 |\n| filtered pool, band `T=18M`, shuffled | 372.6 |\n| **strict top of the ranking (`T≈12M`)** | **359.3** |\n\nMonotone, no interior optimum: the prediction holds and the diversity account is\nrejected in this regime.\n\nA second observable, also confirmed: **register-balancing hurts**. The target is an\nequal-parts mix, so a \"match the mix explicitly\" account predicts that assigning each\npool document to its best-matching register and filling per-register quotas should beat\na single global ranking. It does not — 369.8 vs 359.3. The global contrastive score\nalready prices register match; forcing quotas only admits lower-ranked documents into\nthe budget. This is consistent with the mechanism being *per-document* target\nlikelihood rather than *aggregate* mixture proportions.\n\n## Falsification\n\nThe claim is wrong if any of these is observed:\n\n- The selectivity curve is U-shaped — i.e. some band `T` meaningfully wider than 12M\n  scores *below* the strict top-12M cut. (Tested at 18M and 36M; both worse.)\n- Removing the Moore–Lewis stage and keeping only the hard quality filter, then drawing\n  randomly from the survivors, recovers most of the gain. That would mean the win is\n  junk-removal, not target matching, and the \"domain score\" is decoration.\n- Fitting the target LM on the *raw* dev text (artifacts left in) scores the same. That\n  would mean the score is keying on formatting, not register, and the whole framing is\n  wrong.\n- The gain does not transfer: dev PPL improves but held-out PPL on a disjoint sample of\n  the same domain does not. That would indicate the score memorized the dev sample's\n  specific vocabulary rather than its register.\n\n## Transfer\n\nThe criterion needs only (a) a sample of the target register and (b) an unlabeled pool\n— no quality labels, no reference model, no GPU. It should transfer to any fixed-budget\npretraining or fine-tuning data selection where the evaluation distribution can be\nsampled, and the scoring cost is linear in pool size with a small constant (the whole\n182k-document pool is filtered and scored in under three minutes on one CPU core).\n\nTwo boundaries worth stating. First, the monotone-selectivity result is specific to the\nregime where the budget is a *small* fraction of the pool (~6% here) and the model makes\nonly ~2 passes; at a budget approaching the size of the high-scoring head, the top of the\nranking runs out and the curve must turn. Second, the score is an n-gram contrast, so it\ntransfers across *registers* (encyclopedic, news, prose, technical Q&A) but would need\nrefitting for a target in another language or a target defined by properties n-grams\ncannot see, such as reasoning depth or factual correctness.\n"
}

{
  "command": "cat /tmp/cur3.log; cat /workspace/.timer/remaining_secs"
}

{
  "command": "python - <<'EOF'\nimport numpy as np,pickle,json\ns=np.load('/tmp/ml_scores.npy'); ids=np.load('/tmp/ids.npy')\nsurv=np.flatnonzero(s>-1e8); r=np.random.default_rng(3)\nsel=[int(ids[i]) for i in r.permutation(surv)[:40000]]\njson.dump(sel,open('/tmp/filteronly.json','w')); print(len(sel))\nEOF\nbash dev_eval.sh /tmp/filteronly.json 2>&1 | grep PPL; echo ---; cat /tmp/cur3.log"
}

{
  "command": "sleep 70; cat /tmp/cur3.log"
}

{
  "command": "sleep 45; tail -3 /tmp/cur3.log\npython - <<'EOF'\nimport numpy as np,pickle,json,os\nif os.path.exists('/tmp/ml_scores2.npy'):\n    s=np.load('/tmp/ml_scores2.npy'); texts=pickle.load(open('/tmp/texts.pkl','rb')); ids=np.load('/tmp/ids.npy')\n    sel=[];tot=0;seen=set()\n    for i in np.argsort(-s):\n        if s[i]<=-1e8: break\n        k=texts[i][:120]\n        if k in seen: continue\n        seen.add(k); sel.append(int(ids[i])); tot+=len(texts[i])/3.9\n        if tot>=24e6: break\n    json.dump(sel,open('/tmp/sel_p2.json','w')); print('p2',len(sel),tot/1e6)\nEOF"
}

{
  "command": "sleep 60; tail -3 /tmp/cur3.log; ls -la /tmp/ml_scores2.npy 2>/dev/null"
}

{
  "command": "pkill -f curate_c.py; python - <<'EOF'\nimport numpy as np,pickle,json\ns=np.load('/tmp/ml_scores.npy'); texts=pickle.load(open('/tmp/texts.pkl','rb')); ids=np.load('/tmp/ids.npy')\nsel=[];tot=0;seen=set()\nfor i in np.argsort(-s):\n    if s[i]<=-1e8: break\n    k=texts[i][:120]\n    if k in seen: continue\n    seen.add(k); sel.append(int(ids[i])); tot+=len(texts[i])/3.9\n    if tot>=30e6: break\njson.dump(sel,open('/workspace/submission/selection.json','w')); print('restored',len(sel),tot/1e6)\nEOF\ncat /workspace/.timer/remaining_secs"
}

{
  "command": "python - <<'EOF'\nimport numpy as np,pickle,json\ns=np.load('/tmp/ml_scores.npy'); texts=pickle.load(open('/tmp/texts.pkl','rb')); ids=np.load('/tmp/ids.npy')\nsel=[];tot=0;seen=set()\nfor i in np.argsort(-s):\n    if s[i]<=-1e8: break\n    k=texts[i][:120]\n    if k in seen: continue\n    seen.add(k); sel.append(int(ids[i])); tot+=len(texts[i])/3.9\n    if tot>=30e6: break\njson.dump(sel,open('/workspace/submission/selection.json','w')); print('restored',len(sel),tot/1e6)\nEOF\ncat /workspace/.timer/remaining_secs"
}

{
  "command": "python - <<'EOF'\ns=open('/tmp/curate_c.py').read()\ns=s.replace('''scores[[i for i in range(len(texts)) if scores[i] > -1e8 and i not in set(cand)]] = -1e8 + 1\nnp.save(\"/tmp/ml_scores2.npy\", scores)\nprint(\"pass2 done\", time.time()-t0, flush=True)''',\n'''cs = set(cand)\ndrop = np.array([i for i in np.flatnonzero(scores > -1e8) if i not in cs])\nif len(drop): scores[drop] = -1e8 + 1\nnp.save(\"/tmp/ml_scores2.npy\", scores)\nprint(\"pass2 done\", time.time()-t0, flush=True)\n\n# variant: no hard quality filter, pure Moore-Lewis ranking over the whole pool\nnf = np.full(len(texts), -1e9, dtype=np.float32)\nfor i in range(len(texts)):\n    w = [vi.get(x, UNK) for x in toks(texts[i][:3000])]\n    if len(w) < 30: continue\n    nf[i] = xent(w, bg_lm) - xent(w, tgt_lm)\nnp.save(\"/tmp/ml_nofilter.npy\", nf)\nprint(\"nofilter done\", time.time()-t0, flush=True)\nimport sys; sys.exit(0)''')\nopen('/tmp/curate_c.py','w').write(s)\nEOF\nnohup python /tmp/curate_c.py > /tmp/cur4.log 2>&1 & sleep 5; echo ok"
}

{
  "command": "sleep 230; tail -4 /tmp/cur4.log"
}

{
  "command": "sleep 115; tail -4 /tmp/cur4.log"
}

{
  "command": "python - <<'EOF'\nimport numpy as np,pickle,json\ntexts=pickle.load(open('/tmp/texts.pkl','rb')); ids=np.load('/tmp/ids.npy')\ndef emit(sc,out,T=24e6):\n    sel=[];tot=0;seen=set()\n    for i in np.argsort(-sc):\n        if sc[i]<=-1e8: break\n        k=texts[i][:120]\n        if k in seen: continue\n        seen.add(k); sel.append(int(ids[i])); tot+=len(texts[i])/3.9\n        if tot>=T: break\n    json.dump(sel,open(out,'w')); print(out,len(sel),round(tot/1e6,1))\nemit(np.load('/tmp/ml_scores2.npy'),'/tmp/sel_p2.json')\nEOF\nbash dev_eval.sh /tmp/sel_p2.json 2>&1 | grep PPL; tail -2 /tmp/cur4.log; cat /workspace/.timer/remaining_secs"
}

{
  "command": "python - <<'EOF'\nimport numpy as np,pickle,json\ntexts=pickle.load(open('/tmp/texts.pkl','rb')); ids=np.load('/tmp/ids.npy')\nsc=np.load('/tmp/ml_nofilter.npy'); sel=[];tot=0;seen=set()\nfor i in np.argsort(-sc):\n    if sc[i]<=-1e8: break\n    k=texts[i][:120]\n    if k in seen: continue\n    seen.add(k); sel.append(int(ids[i])); tot+=len(texts[i])/3.9\n    if tot>=24e6: break\njson.dump(sel,open('/tmp/sel_nf.json','w')); print(len(sel),tot/1e6)\nEOF\nbash dev_eval.sh /tmp/sel_nf.json 2>&1 | grep PPL"
}

{
  "command": "python - <<'EOF'\ns=open('/tmp/curate_c.py').read()\n# drop the no-filter ablation block\ni=s.index('# variant: no hard quality filter'); j=s.index('import sys; sys.exit(0)')\ns=s[:i]+s[j+len('import sys; sys.exit(0)'):]\n# drop register-balanced emission (measured worse); emit strict global rank order\ni=s.index('# Each pool doc is assigned to the register')\nj=s.index('json.dump(sel, open(OUT, \"w\"))')\ns=s[:i]+'''# Emit in strict descending score order. Selectivity was measured to be monotonically\n# good in this regime (see claim.md): the budget should come from the very top of the\n# ranking, so priority order == score order, with a cheap near-duplicate guard.\norder = np.argsort(-scores)\nsel, tot, seen = [], 0, set()\nfor i in order:\n    if scores[i] <= -1e8: break\n    k = texts[i][:120]\n    if k in seen: continue\n    seen.add(k)\n    sel.append(int(ids[i])); tot += len(texts[i]) / 3.9   # ~chars per GPT-2 token\n    if tot >= TARGET_TOKENS: break\n'''+s[j:]\ns=s.replace('TARGET_TOKENS = 30_000_000        # emit ~2.5x the 12M budget',\n            'TARGET_TOKENS = 24_000_000        # emit 2x the 12M budget')\ns=s.replace('tot = TARGET_TOKENS','')\n# remove now-unused register machinery? keep reg scores off to save time\ns=s.replace('''    for r in REG: rscores[r][i] = hb - xent(w, reg_lm[r])\\n''','')\ns=s.replace('''reg_lm = {r: build(*reg_counts[r]) for r in REG}''','')\ns=s.replace('''reg_counts = {r: ngram_counts(reg_docs[r]) for r in REG}\\n''','')\ns=s.replace('''rscores = {r: np.full(len(texts), -1e9, dtype=np.float32) for r in REG}\\n''','')\ns=s.replace('''pickle.dump(rscores, open(\"/tmp/reg_scores.pkl\", \"wb\"))\\n''','')\ns=s.replace('''    hb = xent(w, bg_lm)\\n    scores[i] = hb - xent(w, tgt_lm)''','    scores[i] = xent(w, bg_lm) - xent(w, tgt_lm)')\nopen('/workspace/submission/curate.py','w').write(s)\nEOF\nrm -f /tmp/texts.pkl /tmp/ids.npy && nohup python submission/curate.py > /tmp/final.log 2>&1 & sleep 8; echo started"
}

{
  "command": "sleep 200; cat /tmp/final.log"
}

{
  "command": "sleep 110; cat /tmp/final.log; cat /workspace/.timer/remaining_secs"
}

{
  "command": "python - <<'EOF'\nimport json\na=json.load(open('/workspace/submission/selection.json')); b=json.load(open('/tmp/sel_p2.json'))\nprint('n',len(a),'identical_to_tested:',a==b,'dupes:',len(a)!=len(set(a)),'range ok:',min(a)>=0 and max(a)<182016)\nEOF\nbash dev_eval.sh 2>&1 | grep -E \"packed|PPL\""
}

{
  "command": "python - <<'EOF'\ns=open('/workspace/claim.md').read()\ns=s.replace('''   Documents are then emitted in descending score order.''',\n'''   Documents are then rescored in a second pass on their **full** text (pass 1 only\n   sees the first 3000 chars, which lets a document with a clean lede and a junk body\n   through) and emitted in descending score order.''')\ns=s.replace('''| **strict top of the ranking (`T≈12M`)** | **359.3** |''',\n'''| strict top of the ranking (`T≈12M`), pass-1 score | 359.3 |\n| **strict top of the ranking, full-document pass-2 score (submitted)** | **354.2** |''')\ns=s.replace('''- Removing the Moore–Lewis stage and keeping only the hard quality filter, then drawing\n  randomly from the survivors, recovers most of the gain. That would mean the win is\n  junk-removal, not target matching, and the \"domain score\" is decoration.''',\n'''- Removing the Moore–Lewis stage and keeping only the hard quality filter, then drawing\n  randomly from the survivors, recovers most of the gain. That would mean the win is\n  junk-removal, not target matching, and the \"domain score\" is decoration.\n  **Tested: it does not** — filter-only random scores 494.9, i.e. *worse* than the 469.9\n  do-nothing baseline. None of the gain is junk-removal.''')\ns=s.replace('''## Transfer''','''## The two stages are complementary, and neither works alone\n\nThe ablations are unusually clean, and they sharpen the mechanism:\n\n| variant | dev PPL |\n|---|---|\n| hard quality filter only, random draw from survivors | 494.9 |\n| Moore–Lewis score only, no hard filter, strict top | 547.4 |\n| random draw from raw pool | 469.9 |\n| **both stages (submitted)** | **354.2** |\n\n*Each stage alone is worse than doing nothing; together they cut perplexity by 25%.*\nThe reason is that the two stages fail in opposite directions. An n-gram contrast score\nhas no notion of degeneracy: the highest-scoring unfiltered documents are link farms,\ntelephone-prefix tables and SEO spam that happen to be dense in target-like vocabulary,\nso the score alone actively selects junk (547.4). The hard filter removes exactly that\nfailure mode but is register-blind, so on its own it just returns a slightly longer-\ndocument sample of the same generic web (494.9). Target matching only pays off once the\ndegenerate tail it cannot see has been removed. This predicts that the ordering is not\ninterchangeable — no reweighting of the two signals into a single additive score should\nmatch the two-stage pipeline, because the filter's contribution is a hard support\nrestriction, not a soft preference.\n\n## Transfer''')\nopen('/workspace/claim.md','w').write(s)\nEOF\ngrep -c . /workspace/claim.md; ls -la /workspace/submission/"
}