claude-opus-4-8

match-stat (mean per-token log-ratio) vs held-out ppl:
  random     -0.799   457.2
  gates-only -0.707   439.9
  curated V2 -0.498   355.0

[stdout]
match-stat (mean per-token log-ratio) vs held-out ppl:
  random     -0.799   457.2
  gates-only -0.707   439.9
  curated V2 -0.498   355.0
[metadata] {"noOutputExpected": false}

File created successfully at: /workspace/claim.md (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/workspace/claim.md", "content": "# Claim: distribution‑matched, quality‑filtered, deduplicated curation minimizes held‑out perplexity under a fixed token budget\n\n## Hypothesis\nUnder a fixed 12M‑token budget (~6% of the 200M‑token pool), held‑out perplexity on\nthe broad high‑quality target (Wikipedia / web prose / news / technical Q&A) is\nminimized **not by which documents are individually \"good\", but by making the\nselected token distribution match the target distribution while spending no budget\non web junk or redundant copies.** Concretely, ranking pool documents by their\n**per‑token target‑vs‑pool n‑gram log‑likelihood ratio** (DSIR‑style importance),\nbehind cheap English‑prose quality gates, then deduplicating, drives held‑out\nperplexity far below a random selection.\n\nMeasured (frozen 30M GPT, seed 1337, dev target):\n\n| selection | held‑out ppl |\n|---|---|\n| random pool sample (baseline) | 457.2 |\n| **curated (this method, `selection.json`)** | **355.0** |\n\n→ **22.4% lower perplexity**, from data selection alone.\n\n## Mechanism — predicted observable *other* than final perplexity\nThe effect is *distributional*, not per‑document \"quality\", and it is additively\ndecomposable. I verified two observables that are independent of the final score:\n\n**(1) A training‑free statistic — mean per‑token target/pool log‑ratio of the packed\n12M tokens — is a monotone predictor of held‑out perplexity.** Prediction: the more\ntarget‑matched the tokens, the lower the ppl, *before any training*. Verified across\nthree selections:\n\n| selection | match‑stat (↑ better) | ppl (↓ better) |\n|---|---|---|\n| random | −0.799 | 457.2 |\n| quality‑gates‑only (no matching) | −0.707 | 439.9 |\n| curated (matching + dedup) | −0.498 | 355.0 |\n\nStrictly monotone: closing the distributional gap tracks the perplexity drop.\n\n**(2) The gain decomposes, and junk‑removal is the *minor* term.** ~34% of pool\ndocuments fail basic English‑prose gates (non‑English, nav‑menu boilerplate,\nsymbol/number soup). A selection that *only* removes junk (gates, random order)\nwas predicted to land strictly between random and the full method — and does, but\nclose to random:\n\n| step added | ppl | Δ | share of the 102‑ppl gain |\n|---|---|---|---|\n| random → quality gates | 457.2 → 439.9 | −17.3 | 17% |\n| gates → target‑matching (DSIR rank) | 439.9 → 362.9 | −77.0 | **75%** |\n| matching → dedup | 362.9 → 355.0 | −7.9 | 8% |\n\nSo the dominant lever for a *broad* target is aligning the training distribution to\nit (75%), not merely filtering bad pages (17%); near‑dup removal adds a further 8%.\n\n## Falsification\nThe hypothesis is falsified if any of the following held (none did):\n- Curated ppl ≥ random ppl (observed 355.0 ≪ 457.2).\n- The match‑stat is **not** monotone with ppl across {random, gates‑only, curated}\n  — e.g. a more target‑matched selection yielding *higher* ppl would refute the\n  distribution‑matching mechanism (observed strictly monotone).\n- Quality‑gates‑only matches or beats the full method, i.e. target‑matching adds\n  nothing beyond junk removal (observed: matching adds 77 ppl, 4.5× the gate term).\n- Removing dedup does not raise ppl (observed +7.9 ppl without it).\nA further pre‑registered test: a selection built to *maximize* the match‑stat but\ncomposed of gibberish/rare‑word documents (the naive‑DSIR failure mode we corrected)\nshould **not** lower ppl — matching must be to genuine target n‑grams, not to\nsmoothing artifacts of rare tokens.\n\n## Transfer\n- **Other targets:** the criterion is target‑agnostic — rebuild the n‑gram profile\n  from any reference corpus and the same `curate.py` curates for that domain.\n- **Hidden official target:** it is a disjoint sample of the *same* HQ domain, and\n  our features are aggregated over 2,705 target segments (domain‑level, not\n  memorized documents), so we predict a comparable ~20% reduction vs random there.\n- **Budget / model scale:** the priority ranking is budget‑independent; a larger\n  budget simply consumes further down the same list, and a larger model still\n  benefits from on‑distribution, deduplicated tokens — with diminishing returns as\n  the budget approaches the pool size (less to gain from being selective).\n- **Recipe portability:** importance‑ratio scoring + Gopher/C4 gates + MinHash dedup\n  is a standard, reproducible pretraining‑curation pipeline, requiring no labels and\n  no target document memorization; `curate.py` regenerates `selection.json`\n  deterministically.\n", "structuredPatch": [], "originalFile": null, "userModified": false}

selection ids: 20896
all unique: True
all valid pool ids: True
id range: min 5 max 181924 (pool has 182016 docs)
type ok (list of int): True
=== deliverables present ===
-rw-r--r-- 1 root root   4574 Aug  4 05:09 /workspace/claim.md
-rw-r--r-- 1 root root  10135 Aug  4 05:01 /workspace/submission/curate.py
-rw-r--r-- 1 root root 147142 Aug  4 05:01 /workspace/submission/selection.json
=== certification recap ===
HELDOUT_PPL 354.963  loss 5.8720  train_s 56  /tmp/dev_train.npy

[stdout]
selection ids: 20896
all unique: True
all valid pool ids: True
id range: min 5 max 181924 (pool has 182016 docs)
type ok (list of int): True
=== deliverables present ===
-rw-r--r-- 1 root root   4574 Aug  4 05:09 /workspace/claim.md
-rw-r--r-- 1 root root  10135 Aug  4 05:01 /workspace/submission/curate.py
-rw-r--r-- 1 root root 147142 Aug  4 05:01 /workspace/submission/selection.json
=== certification recap ===
HELDOUT_PPL 354.963  loss 5.8720  train_s 56  /tmp/dev_train.npy
[metadata] {"noOutputExpected": false}

Command did not complete within its 120s timeout and was moved to the background (ID: bdmv181yy). Output is being written to: /tmp/claude-0/-workspace/d723e7a0-a155-47f3-b352-d47c686c4960/tasks/bdmv181yy.output. You will be notified when it completes. To check interim output, use Read on that file path.
Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "bdmv181yy", "timedOutAfterMs": 120000, "backgroundCwdHint": "Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands."}

Updated task #3 status

[metadata] {"success": true, "taskId": "3", "updatedFields": ["status"], "statusChange": {"from": "in_progress", "to": "completed"}}

=== rebuild repro output ===
running curate.py --rebuild from scratch (no cache)...
    featurized 182016 docs in 123s -> /tmp/feat_cache.npz
passing gates: 120480/182016 (66.2%)
dropped 1061 near-duplicates; wrote 20896 ids -> /tmp/sel_rebuild.json
selection covers ~22.3M est tokens (budget 12M)
=== diff against certified selection.json ===
certified ids: 20896  rebuilt ids: 20896
IDENTICAL: True
=== timer ===
2854

[stdout]
=== rebuild repro output ===
running curate.py --rebuild from scratch (no cache)...
    featurized 182016 docs in 123s -> /tmp/feat_cache.npz
passing gates: 120480/182016 (66.2%)
dropped 1061 near-duplicates; wrote 20896 ids -> /tmp/sel_rebuild.json
selection covers ~22.3M est tokens (budget 12M)
=== diff against certified selection.json ===
certified ids: 20896  rebuilt ids: 20896
IDENTICAL: True
=== timer ===
2854
[metadata] {"noOutputExpected": false}

frac: [0.465 0.069 0.06  0.034 0.049 0.011 0.059 0.253]
assigned per cluster: [31219, 4989, 5128, 4009, 9308, 14061, 14602, 37164]
V4 strat: 17987 ids, ~15.0M est tok, per-cluster tok: [5.86 0.87 0.77 0.43 0.62 0.13 0.75 3.18] 33s

[stdout]
frac: [0.465 0.069 0.06  0.034 0.049 0.011 0.059 0.253]
assigned per cluster: [31219, 4989, 5128, 4009, 9308, 14061, 14602, 37164]
V4 strat: 17987 ids, ~15.0M est tok, per-cluster tok: [5.86 0.87 0.77 0.43 0.62 0.13 0.75 3.18] 33s
[metadata] {"noOutputExpected": false}

started V4 strat eval pid 3960
2694

[stdout]
started V4 strat eval pid 3960
2694
[metadata] {"noOutputExpected": false}

V2 (top-k+dedup) global match-stat: -0.498  (ppl 355.0)
V4 (stratified)   global match-stat: -0.530  (ppl ?)

[stdout]
V2 (top-k+dedup) global match-stat: -0.498  (ppl 355.0)
V4 (stratified)   global match-stat: -0.530  (ppl ?)
[metadata] {"noOutputExpected": false}

still training...

[stdout]
still training...
[metadata] {"noOutputExpected": false}

still training...
2594

[stdout]
still training...
2594
[metadata] {"noOutputExpected": false}

=== full v4 log ===
packed 12000000 tokens from 17987 ids -> /tmp/dev_train.npy
HELDOUT_PPL 348.225  loss 5.8528  train_s 56  /tmp/dev_train.npy
(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)

[stdout]
=== full v4 log ===
packed 12000000 tokens from 17987 ids -> /tmp/dev_train.npy
HELDOUT_PPL 348.225  loss 5.8528  train_s 56  /tmp/dev_train.npy
(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)
[metadata] {"noOutputExpected": false}

backed up V2 deliverables
=== re-run v4 to check determinism ===
assigned per cluster: [31219, 4989, 5128, 4009, 9308, 14061, 14602, 37164]
V4 strat: 17987 ids, ~15.0M est tok, per-cluster tok: [5.86 0.87 0.77 0.43 0.62 0.13 0.75 3.18] 32s
v4 ids: 17987

[stdout]
backed up V2 deliverables
=== re-run v4 to check determinism ===
assigned per cluster: [31219, 4989, 5128, 4009, 9308, 14061, 14602, 37164]
V4 strat: 17987 ids, ~15.0M est tok, per-cluster tok: [5.86 0.87 0.77 0.43 0.62 0.13 0.75 3.18] 32s
v4 ids: 17987
[metadata] {"noOutputExpected": false}

run1==run2 byte-identical: True len 17987
unique: True min 17 max 181838

[stdout]
run1==run2 byte-identical: True len 17987
unique: True min 17 max 181838
[metadata] {"noOutputExpected": false}

1	import numpy as np, math, json, time, sys
2	sys.path.insert(0,'/workspace/submission')
3	from curate import doc_ngrams, WTOK, D_UNI, T, ALPHA, K_SKETCH, LSH_B, LSH_R
4	from transformers import AutoTokenizer
5	t0=time.time(); A=ALPHA; MINCT=2; K=8
6	z=np.load('/tmp/feat_cache.npz')
7	allf=z['allfeats']; off=z['off']; ids=z['ids']; keep=z['keep']; nchars=z['nchars']; Cr=z['Cr'].astype(np.float64)
8	Nr=Cr.sum(); est=nchars/4.0; counts=np.diff(off)
9	tok=AutoTokenizer.from_pretrained('gpt2'); d=np.load('/workspace/data/multi_dev.npy')
10	idx=np.where(d==50256)[0]; segs=[]; prev=0
11	for i in idx: segs.append((prev,i)); prev=i+1
12	segs.append((prev,len(d)))
13	seg_ng=[]; seg_len=[]; uni_glob=np.zeros(D_UNI,np.float64)
14	for a,b in segs:
15	    if b-a<5: seg_ng.append(np.empty(0,np.int32)); seg_len.append(0); continue
16	    g=doc_ngrams(WTOK.findall(tok.decode(d[a:b]).lower())); seg_ng.append(g); seg_len.append(b-a)
17	    u=g[g<D_UNI]
18	    if len(u): uni_glob+=np.bincount(u,minlength=D_UNI)
19	F=1500; topb=np.argsort(-uni_glob)[:F]; col={int(b):i for i,b in enumerate(topb)}
20	M=np.zeros((len(segs),F),np.float32)
21	for si,g in enumerate(seg_ng):
22	    for h in np.unique(g[g<D_UNI]):
23	        c=col.get(int(h));
24	        if c is not None: M[si,c]+=1
25	nz=M.sum(1)>0; M=M/np.maximum(np.linalg.norm(M,axis=1,keepdims=True),1e-9)
26	rng=np.random.default_rng(0); C=M[rng.choice(np.where(nz)[0],K,replace=False)].copy()
27	for it in range(30):
28	    lab=(M@C.T).argmax(1)
29	    for c in range(K):
30	        m=(lab==c)&nz
31	        if m.sum()>0: C[c]=M[m].mean(0); C[c]/=max(np.linalg.norm(C[c]),1e-9)
32	lab[~nz]=-1
33	Ct_c=[np.zeros(T,np.float64) for _ in range(K)]; tokc=np.zeros(K)
34	for si,g in enumerate(seg_ng):
35	    c=lab[si]
36	    if c<0 or len(g)==0: continue
37	    Ct_c[c]+=np.bincount(g,minlength=T); tokc[c]+=seg_len[si]
38	frac=tokc/tokc.sum()
39	# score docs per cluster
40	S=np.full((len(ids),K),-1e18,np.float64)
41	for c in range(K):
42	    lrc=(np.log(Ct_c[c]+A)-math.log(Ct_c[c].sum()+A*T))-(np.log(Cr+A)-math.log(Nr+A*T))
43	    w=Ct_c[c]<MINCT; lrc[w]=np.minimum(lrc[w],0.0)
44	    v=lrc[allf]; sm=np.add.reduceat(v,off[:-1]); sm[counts==0]=-1e18
45	    S[:,c]=sm/np.maximum(counts,1)
46	valid=keep&(S.max(1)>-1e17)
47	# z-score each column over valid docs -> comparable across clusters
48	Sz=S.copy()
49	for c in range(K):
50	    col_c=S[valid,c]; mu,sd=col_c.mean(),col_c.std()+1e-9
51	    Sz[:,c]=(S[:,c]-mu)/sd
52	Sz[~valid]=-1e18
53	best_c=Sz.argmax(1)
54	print('frac:',np.round(frac,3))
55	print('assigned per cluster:',[int(((best_c==c)&valid).sum()) for c in range(K)])
56	order_c=[]
57	for c in range(K):
58	    m=np.where(valid&(best_c==c))[0]; m=m[np.argsort(-Sz[m,c])]; order_c.append(list(m))
59	BUD=12_600_000; quota=BUD*frac
60	seen=set(); ptr=[0]*K; taken=np.zeros(K); final=[]; cum=0.0; chosen=set()
61	def isdup(j):
62	    sk=np.unique(allf[off[j]:off[j+1]][allf[off[j]:off[j+1]]<D_UNI])[:K_SKETCH]
63	    if len(sk)<LSH_B*LSH_R: return False
64	    bd=[hash(sk[i*LSH_R:(i+1)*LSH_R].tobytes()) for i in range(LSH_B)]
65	    if any(b in seen for b in bd): return True
66	    seen.update(bd); return False
67	active=set(c for c in range(K) if len(order_c[c])>0 and quota[c]>0)
68	while active and cum<BUD:
69	    c=min(active,key=lambda c:taken[c]/max(quota[c],1))
70	    adv=False
71	    while ptr[c]<len(order_c[c]):
72	        j=order_c[c][ptr[c]]; ptr[c]+=1
73	        if isdup(j): continue
74	        final.append(int(ids[j])); chosen.add(j); taken[c]+=est[j]; cum+=est[j]; adv=True; break
75	    if ptr[c]>=len(order_c[c]) or taken[c]>=quota[c]*1.02: active.discard(c)
76	# fallback: guarantee budget coverage with remaining best-global docs
77	if cum<15_000_000:
78	    gscore=Sz.max(1); go=np.argsort(-gscore)
79	    for j in go:
80	        if not valid[j] or j in chosen: continue
81	        if isdup(j): continue
82	        final.append(int(ids[j])); cum+=est[j]
83	        if cum>=15_000_000: break
84	json.dump(final,open('/tmp/sel_v4.json','w'))
85	print('V4 strat: %d ids, ~%.1fM est tok, per-cluster tok:'%(len(final),cum/1e6),np.round(taken/1e6,2),'%.0fs'%(time.time()-t0))
86

[metadata] {"type": "text", "file": {"filePath": "/tmp/v4_strat.py", "content": "import numpy as np, math, json, time, sys\nsys.path.insert(0,'/workspace/submission')\nfrom curate import doc_ngrams, WTOK, D_UNI, T, ALPHA, K_SKETCH, LSH_B, LSH_R\nfrom transformers import AutoTokenizer\nt0=time.time(); A=ALPHA; MINCT=2; K=8\nz=np.load('/tmp/feat_cache.npz')\nallf=z['allfeats']; off=z['off']; ids=z['ids']; keep=z['keep']; nchars=z['nchars']; Cr=z['Cr'].astype(np.float64)\nNr=Cr.sum(); est=nchars/4.0; counts=np.diff(off)\ntok=AutoTokenizer.from_pretrained('gpt2'); d=np.load('/workspace/data/multi_dev.npy')\nidx=np.where(d==50256)[0]; segs=[]; prev=0\nfor i in idx: segs.append((prev,i)); prev=i+1\nsegs.append((prev,len(d)))\nseg_ng=[]; seg_len=[]; uni_glob=np.zeros(D_UNI,np.float64)\nfor a,b in segs:\n    if b-a<5: seg_ng.append(np.empty(0,np.int32)); seg_len.append(0); continue\n    g=doc_ngrams(WTOK.findall(tok.decode(d[a:b]).lower())); seg_ng.append(g); seg_len.append(b-a)\n    u=g[g<D_UNI]\n    if len(u): uni_glob+=np.bincount(u,minlength=D_UNI)\nF=1500; topb=np.argsort(-uni_glob)[:F]; col={int(b):i for i,b in enumerate(topb)}\nM=np.zeros((len(segs),F),np.float32)\nfor si,g in enumerate(seg_ng):\n    for h in np.unique(g[g<D_UNI]):\n        c=col.get(int(h));\n        if c is not None: M[si,c]+=1\nnz=M.sum(1)>0; M=M/np.maximum(np.linalg.norm(M,axis=1,keepdims=True),1e-9)\nrng=np.random.default_rng(0); C=M[rng.choice(np.where(nz)[0],K,replace=False)].copy()\nfor it in range(30):\n    lab=(M@C.T).argmax(1)\n    for c in range(K):\n        m=(lab==c)&nz\n        if m.sum()>0: C[c]=M[m].mean(0); C[c]/=max(np.linalg.norm(C[c]),1e-9)\nlab[~nz]=-1\nCt_c=[np.zeros(T,np.float64) for _ in range(K)]; tokc=np.zeros(K)\nfor si,g in enumerate(seg_ng):\n    c=lab[si]\n    if c<0 or len(g)==0: continue\n    Ct_c[c]+=np.bincount(g,minlength=T); tokc[c]+=seg_len[si]\nfrac=tokc/tokc.sum()\n# score docs per cluster\nS=np.full((len(ids),K),-1e18,np.float64)\nfor c in range(K):\n    lrc=(np.log(Ct_c[c]+A)-math.log(Ct_c[c].sum()+A*T))-(np.log(Cr+A)-math.log(Nr+A*T))\n    w=Ct_c[c]<MINCT; lrc[w]=np.minimum(lrc[w],0.0)\n    v=lrc[allf]; sm=np.add.reduceat(v,off[:-1]); sm[counts==0]=-1e18\n    S[:,c]=sm/np.maximum(counts,1)\nvalid=keep&(S.max(1)>-1e17)\n# z-score each column over valid docs -> comparable across clusters\nSz=S.copy()\nfor c in range(K):\n    col_c=S[valid,c]; mu,sd=col_c.mean(),col_c.std()+1e-9\n    Sz[:,c]=(S[:,c]-mu)/sd\nSz[~valid]=-1e18\nbest_c=Sz.argmax(1)\nprint('frac:',np.round(frac,3))\nprint('assigned per cluster:',[int(((best_c==c)&valid).sum()) for c in range(K)])\norder_c=[]\nfor c in range(K):\n    m=np.where(valid&(best_c==c))[0]; m=m[np.argsort(-Sz[m,c])]; order_c.append(list(m))\nBUD=12_600_000; quota=BUD*frac\nseen=set(); ptr=[0]*K; taken=np.zeros(K); final=[]; cum=0.0; chosen=set()\ndef isdup(j):\n    sk=np.unique(allf[off[j]:off[j+1]][allf[off[j]:off[j+1]]<D_UNI])[:K_SKETCH]\n    if len(sk)<LSH_B*LSH_R: return False\n    bd=[hash(sk[i*LSH_R:(i+1)*LSH_R].tobytes()) for i in range(LSH_B)]\n    if any(b in seen for b in bd): return True\n    seen.update(bd); return False\nactive=set(c for c in range(K) if len(order_c[c])>0 and quota[c]>0)\nwhile active and cum<BUD:\n    c=min(active,key=lambda c:taken[c]/max(quota[c],1))\n    adv=False\n    while ptr[c]<len(order_c[c]):\n        j=order_c[c][ptr[c]]; ptr[c]+=1\n        if isdup(j): continue\n        final.append(int(ids[j])); chosen.add(j); taken[c]+=est[j]; cum+=est[j]; adv=True; break\n    if ptr[c]>=len(order_c[c]) or taken[c]>=quota[c]*1.02: active.discard(c)\n# fallback: guarantee budget coverage with remaining best-global docs\nif cum<15_000_000:\n    gscore=Sz.max(1); go=np.argsort(-gscore)\n    for j in go:\n        if not valid[j] or j in chosen: continue\n        if isdup(j): continue\n        final.append(int(ids[j])); cum+=est[j]\n        if cum>=15_000_000: break\njson.dump(final,open('/tmp/sel_v4.json','w'))\nprint('V4 strat: %d ids, ~%.1fM est tok, per-cluster tok:'%(len(final),cum/1e6),np.round(taken/1e6,2),'%.0fs'%(time.time()-t0))\n", "numLines": 86, "startLine": 1, "totalLines": 86}}

The file /workspace/submission/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"type": "update", "filePath": "/workspace/submission/curate.py", "content": "\"\"\"Curate a raw web pool into a priority-ordered training selection for a small LM.\n\nSTATED CRITERION\n----------------\nThe disclosed evaluation target is a BROAD, multi-domain, high-quality English\ndistribution (encyclopedic / general web prose / news / technical Q&A).  We do\nNOT merely pick the individually \"best\" documents; we select a set whose token\ndistribution MATCHES the target distribution -- including its mix of registers --\nwhile being clean, well-formed English prose and non-redundant.\n\nFour ingredients, combined:\n\n1. TARGET-MATCHING (DSIR-style importance score).  Model the target and the raw\n   pool each as a bag of hashed word n-grams (unigrams + bigrams).  Score a\n   document by its per-token average log-likelihood ratio\n        s(d) = mean_{g in d} w(g),   w(g) = log p_target(g) - log p_pool(g).\n   High s(d) => reads like the high-quality target domain; low s(d) => reads like\n   generic/boilerplate/off-domain web text (Naive-Bayes / DSIR log-odds, Xie 2023).\n\n   RARITY CORRECTION.  The target corpus (~2M n-grams) is ~100x smaller than the\n   pool (~250M).  Naive additive smoothing then gives *any* n-gram unseen in the\n   target (gibberish, foreign text, misspellings, rare jargon) a positive weight,\n   because the smoothing floor of p_target exceeds a genuinely-rare p_pool.  We\n   correct this: an n-gram may earn POSITIVE weight only if it is actually present\n   in the target (count >= MINCT); otherwise its weight is clamped to <= 0 (it may\n   still PENALISE a document when it is common in the pool but absent from target).\n\n2. REGISTER STRATIFICATION (distribution matching, not just affinity).  A broad\n   target is a MIXTURE of sub-registers (encyclopedic, news, Q&A, prose) in some\n   proportion.  Maximising average affinity (plain top-k) over-selects whichever\n   single register scores highest and starves the rest, so the training mix drifts\n   off the target mixture.  Instead we cluster the target's own segments into K\n   sub-registers (cosine k-means on their top unigrams), measure each register's\n   token SHARE of the target, score every pool document against EACH register's\n   n-gram profile, assign it to its best-matching register (scores z-scored per\n   register so they are comparable), and fill the budget by round-robin QUOTA so\n   each register contributes tokens in proportion to its target share.  This makes\n   the selected distribution match the target *mixture*, not just the target mean.\n\n3. QUALITY GATES.  Cheap Gopher/C4-style filters remove degenerate documents\n   (too short, non-English, symbol/number soup, navigation-menu boilerplate,\n   near-duplicate lines) so nothing ranks purely on n-gram artefacts.\n\n4. NEAR-DUP DEDUP.  Within the quota fill we drop documents whose unigram set\n   collides (MinHash + LSH) with an already-selected one, so the fixed budget is\n   spent on DIVERSE on-target tokens instead of many copies of the same syndicated\n   article / boilerplate page.\n\nThe selection is emitted best-first (highest per-register match first, registers\ninterleaved by quota); the training packer consumes this order until the 12M-token\nbudget is full.\n\nWord n-grams use a regex tokenizer over [a-z0-9]+, which also normalises the\ndecoded target's WikiText spacing artifacts (\"@-@\", \" , \") so they align with\nnormal raw-web punctuation.  Deterministic (crc32 hashing, fixed RNG seed); no\nexternal labels.\n\"\"\"\nimport argparse, json, re, math, os, time, zlib\nimport numpy as np\nfrom transformers import AutoTokenizer\n\nPOOL = \"/workspace/data/pool.jsonl\"\nDEV  = \"/workspace/data/multi_dev.npy\"\nOUT  = \"/workspace/submission/selection.json\"\nCACHE = \"/tmp/feat_cache.npz\"\n\nD_UNI = 1 << 20\nD_BI  = 1 << 20\nT     = D_UNI + D_BI\nALPHA = 1.0              # additive smoothing on n-gram counts\nMINCT = 2                # min target count for an n-gram to earn positive weight\nBUDGET = 12_000_000      # training token budget the packer will consume\n\n# register stratification\nK_CLUST     = 8          # number of target sub-registers (k-means clusters)\nCLUST_FEATS = 1500       # top unigram features used to cluster target segments\nCLUST_ITERS = 30         # k-means iterations\nCLUST_SEED  = 0          # k-means init RNG seed (determinism)\nSTRAT_BUD   = 12_600_000 # quota base + round-robin fill target (small margin over BUDGET)\nEMIT_MIN    = 15_000_000 # guarantee at least this many est tokens are emitted (headroom)\n\n# near-dup dedup\nK_SKETCH = 16            # MinHash sketch size (smallest-hash unigrams)\nLSH_B, LSH_R = 4, 4      # LSH banding: near-dup if any 4-row band collides (~Jaccard>=0.7)\n\nWTOK = re.compile(r\"[a-z0-9]+\")\n\nSTOP = set((\"the be to of and a in that have i it for not on with he as you do at \"\n    \"this but his by from they we say her she or an will my one all would there \"\n    \"their what so up out if about who get which go me when make can like time no \"\n    \"just him know take people into year your good some could them see other than \"\n    \"then now look only come it's over think also back after use two how our work \"\n    \"first well way even new want because any these give day most us is are was \"\n    \"were been has had did does\").split())\n\n_uni_cache = {}\ndef _hw(w):\n    h = _uni_cache.get(w)\n    if h is None:\n        h = zlib.crc32(w.encode()) & (D_UNI - 1)\n        _uni_cache[w] = h\n    return h\n\ndef doc_ngrams(words):\n    n = len(words)\n    if n == 0:\n        return np.empty(0, np.int32)\n    uni = np.fromiter((_hw(w) for w in words), np.int64, n)\n    if n == 1:\n        return uni.astype(np.int32)\n    bi = (((uni[:-1] * 1000003) ^ uni[1:]) & (D_BI - 1)) + D_UNI\n    return np.concatenate([uni, bi]).astype(np.int32)\n\ndef quality(text, words):\n    nw = len(words)\n    nchars = len(text)\n    alpha = sum(c.isalpha() for c in text)\n    frac_alpha = alpha / max(1, nchars)\n    sw_ratio = sum(w in STOP for w in words) / max(1, nw)\n    mwl = sum(len(w) for w in words) / max(1, nw)\n    lines = [ln for ln in text.split(\"\\n\") if ln.strip()]\n    if lines:\n        frac_short = sum(len(ln.split()) < 4 for ln in lines) / len(lines)\n        frac_dup = 1.0 - len(set(lines)) / len(lines)\n    else:\n        frac_short, frac_dup = 1.0, 0.0\n    ok = (50 <= nw <= 50000 and frac_alpha >= 0.60 and sw_ratio >= 0.22\n          and 3.0 <= mwl <= 9.0 and frac_short <= 0.5 and frac_dup <= 0.30)\n    return ok, nchars\n\ndef dev_segments():\n    \"\"\"Decode the dev target into per-segment word-ngram arrays + token lengths.\n    Segments are delimited by the GPT-2 <|endoftext|> (50256) separator.\"\"\"\n    tok = AutoTokenizer.from_pretrained(\"gpt2\")\n    d = np.load(DEV)\n    idx = np.where(d == 50256)[0]\n    bounds, prev = [], 0\n    for i in idx:\n        bounds.append((prev, i)); prev = i + 1\n    bounds.append((prev, len(d)))\n    seg_ng, seg_len = [], []\n    for a, b in bounds:\n        if b - a < 5:\n            seg_ng.append(np.empty(0, np.int32)); seg_len.append(0); continue\n        g = doc_ngrams(WTOK.findall(tok.decode(d[a:b]).lower()))\n        seg_ng.append(g); seg_len.append(b - a)\n    return seg_ng, seg_len\n\ndef build_target_counts(seg_ng):\n    Ct = np.zeros(T, np.float64)\n    for g in seg_ng:\n        if len(g):\n            Ct += np.bincount(g, minlength=T)\n    return Ct\n\ndef build_cache():\n    \"\"\"Featurize the whole pool once; cache concatenated n-grams + metadata.\"\"\"\n    t0 = time.time()\n    seg_ng, _ = dev_segments()\n    Ct = build_target_counts(seg_ng)\n    ids, keep, nchars, off = [], [], [], [0]\n    chunks = []\n    with open(POOL) as f:\n        for line in f:\n            r = json.loads(line)\n            t = r[\"text\"]\n            words = WTOK.findall(t.replace(\"<|endoftext|>\", \" \").lower())\n            ok, nc = quality(t, words)\n            g = doc_ngrams(words)\n            ids.append(r[\"id\"]); keep.append(ok); nchars.append(nc)\n            chunks.append(g); off.append(off[-1] + len(g))\n    allfeats = np.concatenate(chunks).astype(np.int32)\n    ids = np.array(ids, np.int64)\n    keep = np.array(keep, bool)\n    nchars = np.array(nchars, np.int64)\n    off = np.array(off, np.int64)\n    Cr = np.bincount(allfeats, minlength=T).astype(np.float64)\n    np.savez(CACHE, allfeats=allfeats, ids=ids, keep=keep, nchars=nchars,\n             off=off, Ct=Ct, Cr=Cr)\n    print(f\"    featurized {len(ids)} docs in {time.time()-t0:.0f}s -> {CACHE}\")\n    return dict(allfeats=allfeats, ids=ids, keep=keep, nchars=nchars,\n                off=off, Ct=Ct, Cr=Cr)\n\ndef load_cache(force=False):\n    if not force and os.path.exists(CACHE):\n        z = np.load(CACHE)\n        return {k: z[k] for k in z.files}\n    return build_cache()\n\ndef _lr(Ct, Cr, Nt, Nr):\n    \"\"\"DSIR log-odds weight per n-gram, rarity-corrected (only genuine target\n    n-grams may earn positive weight).\"\"\"\n    lr = (np.log(Ct + ALPHA) - math.log(Nt + ALPHA * T)) \\\n       - (np.log(Cr + ALPHA) - math.log(Nr + ALPHA * T))\n    weak = Ct < MINCT\n    lr[weak] = np.minimum(lr[weak], 0.0)\n    return lr\n\ndef _doc_scores(lr, allf, off, counts):\n    \"\"\"Per-document mean n-gram weight (empty docs -> -1e18).\"\"\"\n    sm = np.add.reduceat(lr[allf], off[:-1])\n    sm[counts == 0] = -1e18\n    return sm / np.maximum(counts, 1)\n\ndef score(cache):\n    \"\"\"Global (non-stratified) DSIR score, kept for diagnostics / ablation.\"\"\"\n    Ct, Cr = cache[\"Ct\"], cache[\"Cr\"]\n    lr = _lr(Ct, Cr, Ct.sum(), Cr.sum())\n    off, counts = cache[\"off\"], np.diff(cache[\"off\"])\n    s = _doc_scores(lr, cache[\"allfeats\"], off, counts)\n    s[~cache[\"keep\"]] = -1e18\n    return s\n\ndef cluster_target(seg_ng, k=K_CLUST, topf=CLUST_FEATS, iters=CLUST_ITERS):\n    \"\"\"Cosine k-means over the top-unigram profile of each target segment; returns\n    a cluster label per segment (-1 for empty segments).\"\"\"\n    uni_glob = np.zeros(D_UNI, np.float64)\n    for g in seg_ng:\n        u = g[g < D_UNI]\n        if len(u):\n            uni_glob += np.bincount(u, minlength=D_UNI)\n    topb = np.argsort(-uni_glob)[:topf]\n    col = {int(b): i for i, b in enumerate(topb)}\n    M = np.zeros((len(seg_ng), topf), np.float32)\n    for si, g in enumerate(seg_ng):\n        for h in np.unique(g[g < D_UNI]):\n            c = col.get(int(h))\n            if c is not None:\n                M[si, c] += 1\n    nz = M.sum(1) > 0\n    M = M / np.maximum(np.linalg.norm(M, axis=1, keepdims=True), 1e-9)\n    rng = np.random.default_rng(CLUST_SEED)\n    C = M[rng.choice(np.where(nz)[0], k, replace=False)].copy()\n    for _ in range(iters):\n        lab = (M @ C.T).argmax(1)\n        for c in range(k):\n            m = (lab == c) & nz\n            if m.sum() > 0:\n                C[c] = M[m].mean(0); C[c] /= max(np.linalg.norm(C[c]), 1e-9)\n    lab[~nz] = -1\n    return lab\n\ndef stratified_select(cache, seg_ng, seg_len, lab, k=K_CLUST):\n    \"\"\"Assign each pool doc to its best-matching target register and fill the\n    budget by round-robin quota proportional to each register's target token share,\n    deduping near-copies.  Returns (ids, target_frac, taken_tokens, est_total).\"\"\"\n    allf, off = cache[\"allfeats\"], cache[\"off\"]\n    ids, keep = cache[\"ids\"], cache[\"keep\"]\n    Cr = cache[\"Cr\"].astype(np.float64)\n    Nr = Cr.sum(); est = cache[\"nchars\"] / 4.0; counts = np.diff(off)\n\n    # per-register target counts and token mass (share of the target mixture)\n    Ct_c = [np.zeros(T, np.float64) for _ in range(k)]\n    tokc = np.zeros(k)\n    for si, g in enumerate(seg_ng):\n        c = lab[si]\n        if c < 0 or len(g) == 0:\n            continue\n        Ct_c[c] += np.bincount(g, minlength=T); tokc[c] += seg_len[si]\n    frac = tokc / tokc.sum()\n\n    # score every doc against every register profile\n    S = np.full((len(ids), k), -1e18, np.float64)\n    for c in range(k):\n        lrc = _lr(Ct_c[c], Cr, Ct_c[c].sum(), Nr)\n        S[:, c] = _doc_scores(lrc, allf, off, counts)\n    valid = keep & (S.max(1) > -1e17)\n\n    # z-score each register's scores across docs so assignment is comparable\n    Sz = S.copy()\n    for c in range(k):\n        col_c = S[valid, c]; mu, sd = col_c.mean(), col_c.std() + 1e-9\n        Sz[:, c] = (S[:, c] - mu) / sd\n    Sz[~valid] = -1e18\n    best_c = Sz.argmax(1)\n\n    order_c = []\n    for c in range(k):\n        m = np.where(valid & (best_c == c))[0]\n        order_c.append(list(m[np.argsort(-Sz[m, c])]))\n\n    quota = STRAT_BUD * frac\n    seen = set(); ptr = [0] * k; taken = np.zeros(k)\n    final = []; cum = 0.0; chosen = set()\n\n    def isdup(j):\n        f = allf[off[j]:off[j + 1]]\n        sk = np.unique(f[f < D_UNI])[:K_SKETCH]\n        if len(sk) < LSH_B * LSH_R:\n            return False\n        bd = [hash(sk[i * LSH_R:(i + 1) * LSH_R].tobytes()) for i in range(LSH_B)]\n        if any(b in seen for b in bd):\n            return True\n        seen.update(bd); return False\n\n    active = set(c for c in range(k) if order_c[c] and quota[c] > 0)\n    while active and cum < STRAT_BUD:\n        c = min(active, key=lambda c: taken[c] / max(quota[c], 1))\n        while ptr[c] < len(order_c[c]):\n            j = order_c[c][ptr[c]]; ptr[c] += 1\n            if isdup(j):\n                continue\n            final.append(int(ids[j])); chosen.add(j)\n            taken[c] += est[j]; cum += est[j]; break\n        if ptr[c] >= len(order_c[c]) or taken[c] >= quota[c] * 1.02:\n            active.discard(c)\n\n    # fallback: guarantee budget coverage with the remaining best-global docs\n    if cum < EMIT_MIN:\n        for j in np.argsort(-Sz.max(1)):\n            if not valid[j] or j in chosen or isdup(j):\n                continue\n            final.append(int(ids[j])); cum += est[j]\n            if cum >= EMIT_MIN:\n                break\n    return final, frac, taken, cum\n\ndef main():\n    ap = argparse.ArgumentParser()\n    ap.add_argument(\"--out\", default=OUT)\n    ap.add_argument(\"--rebuild\", action=\"store_true\")\n    ap.add_argument(\"--diag\", action=\"store_true\")\n    a = ap.parse_args()\n\n    cache = load_cache(force=a.rebuild)\n    seg_ng, seg_len = dev_segments()\n    lab = cluster_target(seg_ng)\n    sel, frac, taken, cum = stratified_select(cache, seg_ng, seg_len, lab)\n    json.dump(sel, open(a.out, \"w\"))\n\n    npass = int((cache[\"keep\"]).sum())\n    print(f\"passing gates: {npass}/{len(cache['ids'])} \"\n          f\"({100*npass/len(cache['ids']):.1f}%)\")\n    print(f\"target register token shares: {np.round(frac, 3)}\")\n    print(f\"selected per-register est tokens (M): {np.round(taken/1e6, 2)}\")\n    print(f\"wrote {len(sel)} ids -> {a.out}; ~{cum/1e6:.1f}M est tokens \"\n          f\"(budget {BUDGET/1e6:.0f}M)\")\n\n    if a.diag:\n        ids = cache[\"ids\"]\n        want = set(sel[:8]) | set(sel[len(sel)//2:len(sel)//2 + 3])\n        txt = {}\n        with open(POOL) as f:\n            for line in f:\n                r = json.loads(line)\n                if r[\"id\"] in want:\n                    txt[r[\"id\"]] = r[\"text\"]\n        print(\"\\n--- FIRST 8 SELECTED ---\")\n        for i in sel[:8]:\n            print(f\"id={i}: {txt[i][:170].strip().replace(chr(10),' | ')}\")\n        print(\"\\n--- MIDDLE 3 SELECTED ---\")\n        for i in sel[len(sel)//2:len(sel)//2 + 3]:\n            print(f\"id={i}: {txt[i][:170].strip().replace(chr(10),' | ')}\")\n\nif __name__ == \"__main__\":\n    main()\n", "structuredPatch": [{"oldStart": 3, "oldLines": 14, "newStart": 3, "newLines": 15, "lines": [" STATED CRITERION", " ----------------", " The disclosed evaluation target is a BROAD, multi-domain, high-quality English", "-distribution (encyclopedic / general web prose / news / technical Q&A). We select", "-the pool documents whose token distribution most looks like that target while", "-being clean, well-formed English prose.", "+distribution (encyclopedic / general web prose / news / technical Q&A).  We do", "+NOT merely pick the individually \"best\" documents; we select a set whose token", "+distribution MATCHES the target distribution -- including its mix of registers --", "+while being clean, well-formed English prose and non-redundant.", " ", "-Two ingredients, combined:", "+Four ingredients, combined:", " ", " 1. TARGET-MATCHING (DSIR-style importance score).  Model the target and the raw", "-   pool each as a bag of hashed word n-grams (unigrams + bigrams).  Score every", "+   pool each as a bag of hashed word n-grams (unigrams + bigrams).  Score a", "    document by its per-token average log-likelihood ratio", "         s(d) = mean_{g in d} w(g),   w(g) = log p_target(g) - log p_pool(g).", "    High s(d) => reads like the high-quality target domain; low s(d) => reads like"]}, {"oldStart": 23, "oldLines": 25, "newStart": 24, "newLines": 36, "lines": ["    correct this: an n-gram may earn POSITIVE weight only if it is actually present", "    in the target (count >= MINCT); otherwise its weight is clamped to <= 0 (it may", "    still PENALISE a document when it is common in the pool but absent from target).", "-   So a document is rewarded only for n-grams the target genuinely uses.", " ", "-2. QUALITY GATES.  Cheap Gopher/C4-style filters remove degenerate documents", "+2. REGISTER STRATIFICATION (distribution matching, not just affinity).  A broad", "+   target is a MIXTURE of sub-registers (encyclopedic, news, Q&A, prose) in some", "+   proportion.  Maximising average affinity (plain top-k) over-selects whichever", "+   single register scores highest and starves the rest, so the training mix drifts", "+   off the target mixture.  Instead we cluster the target's own segments into K", "+   sub-registers (cosine k-means on their top unigrams), measure each register's", "+   token SHARE of the target, score every pool document against EACH register's", "+   n-gram profile, assign it to its best-matching register (scores z-scored per", "+   register so they are comparable), and fill the budget by round-robin QUOTA so", "+   each register contributes tokens in proportion to its target share.  This makes", "+   the selected distribution match the target *mixture*, not just the target mean.", "+", "+3. QUALITY GATES.  Cheap Gopher/C4-style filters remove degenerate documents", "    (too short, non-English, symbol/number soup, navigation-menu boilerplate,", "    near-duplicate lines) so nothing ranks purely on n-gram artefacts.", " ", "-3. NEAR-DUP DEDUP.  Walking the ranked survivors best-first, we drop documents", "-   whose unigram set collides (MinHash + LSH) with an already-selected one, so", "-   the fixed budget is spent on DIVERSE on-target tokens instead of many copies", "-   of the same syndicated article / boilerplate page.", "+4. NEAR-DUP DEDUP.  Within the quota fill we drop documents whose unigram set", "+   collides (MinHash + LSH) with an already-selected one, so the fixed budget is", "+   spent on DIVERSE on-target tokens instead of many copies of the same syndicated", "+   article / boilerplate page.", " ", "-Documents that fail any gate are dropped; survivors are emitted DEDUPED in", "-DESCENDING score order (best first).  The training packer consumes this order", "-until the 12M-token budget is full, so the highest-quality, most on-target,", "-non-redundant tokens are used.", "+The selection is emitted best-first (highest per-register match first, registers", "+interleaved by quota); the training packer consumes this order until the 12M-token", "+budget is full.", " ", " Word n-grams use a regex tokenizer over [a-z0-9]+, which also normalises the", " decoded target's WikiText spacing artifacts (\"@-@\", \" , \") so they align with", "-normal raw-web punctuation.  Deterministic (crc32 hashing); no external labels.", "+normal raw-web punctuation.  Deterministic (crc32 hashing, fixed RNG seed); no", "+external labels.", " \"\"\"", " import argparse, json, re, math, os, time, zlib", " import numpy as np"]}, {"oldStart": 57, "oldLines": 8, "newStart": 69, "newLines": 17, "lines": [" T     = D_UNI + D_BI", " ALPHA = 1.0              # additive smoothing on n-gram counts", " MINCT = 2                # min target count for an n-gram to earn positive weight", "-BUDGET = 12_000_000", "-EMIT_TOKENS = 20_000_000 # emit deduped ids until this many est tokens (margin over budget)", "+BUDGET = 12_000_000      # training token budget the packer will consume", "+", "+# register stratification", "+K_CLUST     = 8          # number of target sub-registers (k-means clusters)", "+CLUST_FEATS = 1500       # top unigram features used to cluster target segments", "+CLUST_ITERS = 30         # k-means iterations", "+CLUST_SEED  = 0          # k-means init RNG seed (determinism)", "+STRAT_BUD   = 12_600_000 # quota base + round-robin fill target (small margin over BUDGET)", "+EMIT_MIN    = 15_000_000 # guarantee at least this many est tokens are emitted (headroom)", "+", "+# near-dup dedup", " K_SKETCH = 16            # MinHash sketch size (smallest-hash unigrams)", " LSH_B, LSH_R = 4, 4      # LSH banding: near-dup if any 4-row band collides (~Jaccard>=0.7)", " "]}, {"oldStart": 107, "oldLines": 20, "newStart": 128, "newLines": 27, "lines": ["           and 3.0 <= mwl <= 9.0 and frac_short <= 0.5 and frac_dup <= 0.30)", "     return ok, nchars", " ", "-def build_target_counts():", "+def dev_segments():", "+    \"\"\"Decode the dev target into per-segment word-ngram arrays + token lengths.", "+    Segments are delimited by the GPT-2 <|endoftext|> (50256) separator.\"\"\"", "     tok = AutoTokenizer.from_pretrained(\"gpt2\")", "     d = np.load(DEV)", "     idx = np.where(d == 50256)[0]", "-    segs, prev = [], 0", "+    bounds, prev = [], 0", "     for i in idx:", "-        segs.append((prev, i)); prev = i + 1", "-    segs.append((prev, len(d)))", "-    Ct = np.zeros(T, np.float64)", "-    for a, b in segs:", "+        bounds.append((prev, i)); prev = i + 1", "+    bounds.append((prev, len(d)))", "+    seg_ng, seg_len = [], []", "+    for a, b in bounds:", "         if b - a < 5:", "-            continue", "-        words = WTOK.findall(tok.decode(d[a:b]).lower())", "-        g = doc_ngrams(words)", "+            seg_ng.append(np.empty(0, np.int32)); seg_len.append(0); continue", "+        g = doc_ngrams(WTOK.findall(tok.decode(d[a:b]).lower()))", "+        seg_ng.append(g); seg_len.append(b - a)", "+    return seg_ng, seg_len", "+", "+def build_target_counts(seg_ng):", "+    Ct = np.zeros(T, np.float64)", "+    for g in seg_ng:", "         if len(g):", "             Ct += np.bincount(g, minlength=T)", "     return Ct"]}, {"oldStart": 128, "oldLines": 7, "newStart": 156, "newLines": 8, "lines": [" def build_cache():", "     \"\"\"Featurize the whole pool once; cache concatenated n-grams + metadata.\"\"\"", "     t0 = time.time()", "-    Ct = build_target_counts()", "+    seg_ng, _ = dev_segments()", "+    Ct = build_target_counts(seg_ng)", "     ids, keep, nchars, off = [], [], [], [0]", "     chunks = []", "     with open(POOL) as f:"]}, {"oldStart": 158, "oldLines": 83, "newStart": 187, "newLines": 170, "lines": ["         return {k: z[k] for k in z.files}", "     return build_cache()", " ", "-def score(cache, alpha=ALPHA, minct=MINCT):", "-    Ct, Cr = cache[\"Ct\"], cache[\"Cr\"]", "-    Nt, Nr = Ct.sum(), Cr.sum()", "-    lr = (np.log(Ct + alpha) - math.log(Nt + alpha * T)) \\", "-       - (np.log(Cr + alpha) - math.log(Nr + alpha * T))", "-    # rarity correction: n-grams not solidly in target may only penalise, not reward", "-    weak = Ct < minct", "+def _lr(Ct, Cr, Nt, Nr):", "+    \"\"\"DSIR log-odds weight per n-gram, rarity-corrected (only genuine target", "+    n-grams may earn positive weight).\"\"\"", "+    lr = (np.log(Ct + ALPHA) - math.log(Nt + ALPHA * T)) \\", "+       - (np.log(Cr + ALPHA) - math.log(Nr + ALPHA * T))", "+    weak = Ct < MINCT", "     lr[weak] = np.minimum(lr[weak], 0.0)", "-    off = cache[\"off\"]", "-    vals = lr[cache[\"allfeats\"]]", "-    counts = np.diff(off)", "-    sums = np.add.reduceat(vals, off[:-1])", "-    sums[counts == 0] = -1e18", "-    s = sums / np.maximum(counts, 1)", "+    return lr", "+", "+def _doc_scores(lr, allf, off, counts):", "+    \"\"\"Per-document mean n-gram weight (empty docs -> -1e18).\"\"\"", "+    sm = np.add.reduceat(lr[allf], off[:-1])", "+    sm[counts == 0] = -1e18", "+    return sm / np.maximum(counts, 1)", "+", "+def score(cache):", "+    \"\"\"Global (non-stratified) DSIR score, kept for diagnostics / ablation.\"\"\"", "+    Ct, Cr = cache[\"Ct\"], cache[\"Cr\"]", "+    lr = _lr(Ct, Cr, Ct.sum(), Cr.sum())", "+    off, counts = cache[\"off\"], np.diff(cache[\"off\"])", "+    s = _doc_scores(lr, cache[\"allfeats\"], off, counts)", "     s[~cache[\"keep\"]] = -1e18", "     return s", " ", "-def dedup_select(cache, order_passing, target_tokens=EMIT_TOKENS, cap=60000):", "-    \"\"\"Walk documents in score order, dropping near-duplicates (syndicated /", "-    boilerplate-repeated content) via MinHash+LSH over each doc's unigram set, so", "-    the budget is spent on DIVERSE on-target tokens.  Deterministic.\"\"\"", "-    allf, off, ids = cache[\"allfeats\"], cache[\"off\"], cache[\"ids\"]", "-    est = cache[\"nchars\"] / 4.0", "-    seen, out, cum, ndup = set(), [], 0.0, 0", "-    for j in order_passing:", "+def cluster_target(seg_ng, k=K_CLUST, topf=CLUST_FEATS, iters=CLUST_ITERS):", "+    \"\"\"Cosine k-means over the top-unigram profile of each target segment; returns", "+    a cluster label per segment (-1 for empty segments).\"\"\"", "+    uni_glob = np.zeros(D_UNI, np.float64)", "+    for g in seg_ng:", "+        u = g[g < D_UNI]", "+        if len(u):", "+            uni_glob += np.bincount(u, minlength=D_UNI)", "+    topb = np.argsort(-uni_glob)[:topf]", "+    col = {int(b): i for i, b in enumerate(topb)}", "+    M = np.zeros((len(seg_ng), topf), np.float32)", "+    for si, g in enumerate(seg_ng):", "+        for h in np.unique(g[g < D_UNI]):", "+            c = col.get(int(h))", "+            if c is not None:", "+                M[si, c] += 1", "+    nz = M.sum(1) > 0", "+    M = M / np.maximum(np.linalg.norm(M, axis=1, keepdims=True), 1e-9)", "+    rng = np.random.default_rng(CLUST_SEED)", "+    C = M[rng.choice(np.where(nz)[0], k, replace=False)].copy()", "+    for _ in range(iters):", "+        lab = (M @ C.T).argmax(1)", "+        for c in range(k):", "+            m = (lab == c) & nz", "+            if m.sum() > 0:", "+                C[c] = M[m].mean(0); C[c] /= max(np.linalg.norm(C[c]), 1e-9)", "+    lab[~nz] = -1", "+    return lab", "+", "+def stratified_select(cache, seg_ng, seg_len, lab, k=K_CLUST):", "+    \"\"\"Assign each pool doc to its best-matching target register and fill the", "+    budget by round-robin quota proportional to each register's target token share,", "+    deduping near-copies.  Returns (ids, target_frac, taken_tokens, est_total).\"\"\"", "+    allf, off = cache[\"allfeats\"], cache[\"off\"]", "+    ids, keep = cache[\"ids\"], cache[\"keep\"]", "+    Cr = cache[\"Cr\"].astype(np.float64)", "+    Nr = Cr.sum(); est = cache[\"nchars\"] / 4.0; counts = np.diff(off)", "+", "+    # per-register target counts and token mass (share of the target mixture)", "+    Ct_c = [np.zeros(T, np.float64) for _ in range(k)]", "+    tokc = np.zeros(k)", "+    for si, g in enumerate(seg_ng):", "+        c = lab[si]", "+        if c < 0 or len(g) == 0:", "+            continue", "+        Ct_c[c] += np.bincount(g, minlength=T); tokc[c] += seg_len[si]", "+    frac = tokc / tokc.sum()", "+", "+    # score every doc against every register profile", "+    S = np.full((len(ids), k), -1e18, np.float64)", "+    for c in range(k):", "+        lrc = _lr(Ct_c[c], Cr, Ct_c[c].sum(), Nr)", "+        S[:, c] = _doc_scores(lrc, allf, off, counts)", "+    valid = keep & (S.max(1) > -1e17)", "+", "+    # z-score each register's scores across docs so assignment is comparable", "+    Sz = S.copy()", "+    for c in range(k):", "+        col_c = S[valid, c]; mu, sd = col_c.mean(), col_c.std() + 1e-9", "+        Sz[:, c] = (S[:, c] - mu) / sd", "+    Sz[~valid] = -1e18", "+    best_c = Sz.argmax(1)", "+", "+    order_c = []", "+    for c in range(k):", "+        m = np.where(valid & (best_c == c))[0]", "+        order_c.append(list(m[np.argsort(-Sz[m, c])]))", "+", "+    quota = STRAT_BUD * frac", "+    seen = set(); ptr = [0] * k; taken = np.zeros(k)", "+    final = []; cum = 0.0; chosen = set()", "+", "+    def isdup(j):", "         f = allf[off[j]:off[j + 1]]", "-        sk = np.unique(f[f < D_UNI])[:K_SKETCH]          # smallest-hash unigrams", "-        if len(sk) >= LSH_B * LSH_R:", "-            bands = [hash(sk[b*LSH_R:(b+1)*LSH_R].tobytes()) for b in range(LSH_B)]", "-            if any(bd in seen for bd in bands):", "-                ndup += 1", "+        sk = np.unique(f[f < D_UNI])[:K_SKETCH]", "+        if len(sk) < LSH_B * LSH_R:", "+            return False", "+        bd = [hash(sk[i * LSH_R:(i + 1) * LSH_R].tobytes()) for i in range(LSH_B)]", "+        if any(b in seen for b in bd):", "+            return True", "+        seen.update(bd); return False", "+", "+    active = set(c for c in range(k) if order_c[c] and quota[c] > 0)", "+    while active and cum < STRAT_BUD:", "+        c = min(active, key=lambda c: taken[c] / max(quota[c], 1))", "+        while ptr[c] < len(order_c[c]):", "+            j = order_c[c][ptr[c]]; ptr[c] += 1", "+            if isdup(j):", "                 continue", "-            seen.update(bands)", "-        out.append(int(ids[j])); cum += est[j]", "-        if cum >= target_tokens or len(out) >= cap:", "-            break", "-    return out, ndup", "+            final.append(int(ids[j])); chosen.add(j)", "+            taken[c] += est[j]; cum += est[j]; break", "+        if ptr[c] >= len(order_c[c]) or taken[c] >= quota[c] * 1.02:", "+            active.discard(c)", " ", "+    # fallback: guarantee budget coverage with the remaining best-global docs", "+    if cum < EMIT_MIN:", "+        for j in np.argsort(-Sz.max(1)):", "+            if not valid[j] or j in chosen or isdup(j):", "+                continue", "+            final.append(int(ids[j])); cum += est[j]", "+            if cum >= EMIT_MIN:", "+                break", "+    return final, frac, taken, cum", "+", " def main():", "     ap = argparse.ArgumentParser()", "     ap.add_argument(\"--out\", default=OUT)", "-    ap.add_argument(\"--top\", type=int, default=60000)", "     ap.add_argument(\"--rebuild\", action=\"store_true\")", "     ap.add_argument(\"--diag\", action=\"store_true\")", "     a = ap.parse_args()", " ", "     cache = load_cache(force=a.rebuild)", "-    s = score(cache)", "-    ids = cache[\"ids\"]", "-    order = np.argsort(-s)", "-    passing = order[s[order] > -1e17]", "-    sel, ndup = dedup_select(cache, passing, cap=a.top)", "+    seg_ng, seg_len = dev_segments()", "+    lab = cluster_target(seg_ng)", "+    sel, frac, taken, cum = stratified_select(cache, seg_ng, seg_len, lab)", "     json.dump(sel, open(a.out, \"w\"))", " ", "-    est_sel = np.cumsum([cache[\"nchars\"][j] / 4.0 for j in passing[:len(sel)+ndup]])", "-    print(f\"passing gates: {len(passing)}/{len(ids)} \"", "-          f\"({100*len(passing)/len(ids):.1f}%)\")", "-    print(f\"dropped {ndup} near-duplicates; wrote {len(sel)} ids -> {a.out}\")", "-    print(f\"selection covers ~{est_sel[-1]/1e6:.1f}M est tokens (budget {BUDGET/1e6:.0f}M)\")", "+    npass = int((cache[\"keep\"]).sum())", "+    print(f\"passing gates: {npass}/{len(cache['ids'])} \"", "+          f\"({100*npass/len(cache['ids']):.1f}%)\")", "+    print(f\"target register token shares: {np.round(frac, 3)}\")", "+    print(f\"selected per-register est tokens (M): {np.round(taken/1e6, 2)}\")", "+    print(f\"wrote {len(sel)} ids -> {a.out}; ~{cum/1e6:.1f}M est tokens \"", "+          f\"(budget {BUDGET/1e6:.0f}M)\")", " ", "     if a.diag:", "-        want = set(int(ids[j]) for j in passing[:10]) | \\", "-               set(int(ids[j]) for j in passing[len(passing)//2:len(passing)//2+3])", "+        ids = cache[\"ids\"]", "+        want = set(sel[:8]) | set(sel[len(sel)//2:len(sel)//2 + 3])", "         txt = {}", "         with open(POOL) as f:", "             for line in f:", "                 r = json.loads(line)", "                 if r[\"id\"] in want:", "                     txt[r[\"id\"]] = r[\"text\"]", "-        print(\"\\n--- TOP 10 ---\")", "-        for j in passing[:10]:", "-            i = int(ids[j])", "-            print(f\"{s[j]:+.3f} id={i}: {txt[i][:170].strip().replace(chr(10),' | ')}\")", "-        print(\"\\n--- MEDIAN-RANK 3 ---\")", "-        for j in passing[len(passing)//2:len(passing)//2+3]:", "-            i = int(ids[j])", "-            print(f\"{s[j]:+.3f} id={i}: {txt[i][:170].strip().replace(chr(10),' | ')}\")", "+        print(\"\\n--- FIRST 8 SELECTED ---\")", "+        for i in sel[:8]:", "+            print(f\"id={i}: {txt[i][:170].strip().replace(chr(10),' | ')}\")", "+        print(\"\\n--- MIDDLE 3 SELECTED ---\")", "+        for i in sel[len(sel)//2:len(sel)//2 + 3]:", "+            print(f\"id={i}: {txt[i][:170].strip().replace(chr(10),' | ')}\")", " ", " if __name__ == \"__main__\":", "     main()"]}], "originalFile": null, "userModified": false}

passing gates: 120480/182016 (66.2%)
target register token shares: [0.465 0.069 0.06  0.034 0.049 0.011 0.059 0.253]
selected per-register est tokens (M): [5.86 0.87 0.77 0.43 0.62 0.13 0.75 3.18]
wrote 17987 ids -> /tmp/sel_from_curate.py.json; ~15.0M est tokens (budget 12M)
=== compare to eval'd sel_v4.json ===
curate.py output == eval'd V4: True | len 17987 vs 17987

[stdout]
passing gates: 120480/182016 (66.2%)
target register token shares: [0.465 0.069 0.06  0.034 0.049 0.011 0.059 0.253]
selected per-register est tokens (M): [5.86 0.87 0.77 0.43 0.62 0.13 0.75 3.18]
wrote 17987 ids -> /tmp/sel_from_curate.py.json; ~15.0M est tokens (budget 12M)
=== compare to eval'd sel_v4.json ===
curate.py output == eval'd V4: True | len 17987 vs 17987
[metadata] {"noOutputExpected": false}

selected per-register est tokens (M): [5.86 0.87 0.77 0.43 0.62 0.13 0.75 3.18]
wrote 17987 ids -> /workspace/submission/selection.json; ~15.0M est tokens (budget 12M)
=== validate deliverable ===
ids: 17987 | unique: True | min 17 max 181838 | in-range(0..182015): True
== eval'd V4: True
=== re-certify with dev_eval.sh ===
2063

[stdout]
selected per-register est tokens (M): [5.86 0.87 0.77 0.43 0.62 0.13 0.75 3.18]
wrote 17987 ids -> /workspace/submission/selection.json; ~15.0M est tokens (budget 12M)
=== validate deliverable ===
ids: 17987 | unique: True | min 17 max 181838 | in-range(0..182015): True
== eval'd V4: True
=== re-certify with dev_eval.sh ===
2063
[metadata] {"noOutputExpected": false}

Token indices sequence length is longer than the specified maximum sequence length for this model (2170 > 1024). Running this sequence through the model will result in indexing errors
packed 12000000 tokens from 17987 ids -> /tmp/dev_train.npy
HELDOUT_PPL 348.225  loss 5.8528  train_s 56  /tmp/dev_train.npy

[stdout]
Token indices sequence length is longer than the specified maximum sequence length for this model (2170 > 1024). Running this sequence through the model will result in indexing errors
packed 12000000 tokens from 17987 ids -> /tmp/dev_train.npy
HELDOUT_PPL 348.225  loss 5.8528  train_s 56  /tmp/dev_train.npy
[metadata] {"noOutputExpected": false}

target register shares: [0.465 0.069 0.06  0.034 0.049 0.011 0.059 0.253]
V2 top-k     shares   : [0.534 0.116 0.097 0.02  0.079 0.023 0.048 0.083]  L1 to target: 0.389
V4 stratified shares  : [0.391 0.058 0.051 0.029 0.078 0.114 0.05  0.23 ]  L1 to target: 0.264

[stdout]
target register shares: [0.465 0.069 0.06  0.034 0.049 0.011 0.059 0.253]
V2 top-k     shares   : [0.534 0.116 0.097 0.02  0.079 0.023 0.048 0.083]  L1 to target: 0.389
V4 stratified shares  : [0.391 0.058 0.051 0.029 0.078 0.114 0.05  0.23 ]  L1 to target: 0.264
[metadata] {"noOutputExpected": false}

target register shares  : [0.465 0.069 0.06  0.034 0.049 0.011 0.059 0.253]
V2 top-k    (first 12M) : [0.547 0.121 0.106 0.019 0.077 0.018 0.039 0.073]  L1: 0.429
V4 stratified(first 12M): [0.465 0.069 0.06  0.034 0.049 0.011 0.06  0.253]  L1: 0.002

[stdout]
target register shares  : [0.465 0.069 0.06  0.034 0.049 0.011 0.059 0.253]
V2 top-k    (first 12M) : [0.547 0.121 0.106 0.019 0.077 0.018 0.039 0.073]  L1: 0.429
V4 stratified(first 12M): [0.465 0.069 0.06  0.034 0.049 0.011 0.06  0.253]  L1: 0.002
[metadata] {"noOutputExpected": false}

random         global match-stat: -0.799
gates-only     global match-stat: -0.707
V2 top-k       global match-stat: -0.498
V4 stratified  global match-stat: -0.530
timer:
1753

[stdout]
random         global match-stat: -0.799
gates-only     global match-stat: -0.707
V2 top-k       global match-stat: -0.498
V4 stratified  global match-stat: -0.530
timer:
1753
[metadata] {"noOutputExpected": false}

The file /workspace/claim.md has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"type": "update", "filePath": "/workspace/claim.md", "content": "# Claim: matching the target's *register mixture* (not just its average n‑gram profile), behind quality gates and dedup, minimizes held‑out perplexity under a fixed token budget\n\n## Hypothesis\nUnder a fixed 12M‑token budget (~6% of the ~193M‑token pool), held‑out perplexity\non the broad high‑quality target (Wikipedia / web prose / news / technical Q&A) is\nminimized **not by picking the individually \"best\" documents, but by making the\nselected token distribution match the target distribution** — spending no budget on\nweb junk or redundant copies. The target is a *mixture* of sub‑registers, and the\ndecisive move is to reproduce that mixture in the training data, not to maximize\naverage target‑affinity (which over‑selects one register and starves the others).\n\nConcretely: rank pool documents by their **per‑token target‑vs‑pool n‑gram\nlog‑likelihood ratio** (DSIR‑style importance), behind cheap English‑prose quality\ngates; then, rather than taking the global top‑k, **cluster the target into K\nsub‑registers, and fill the budget by quota so each register contributes tokens in\nproportion to its share of the target**, deduplicating near‑copies as we go.\n\nMeasured (frozen 30M GPT, seed 1337, dev target):\n\n| selection | held‑out ppl |\n|---|---|\n| random pool sample (baseline) | 457.2 |\n| DSIR top‑k + gates + dedup | 355.0 |\n| **register‑stratified (this method, `selection.json`)** | **348.2** |\n\n→ **23.8% lower perplexity than random**, from data selection alone; the\nregister‑stratification step alone accounts for the final 6.7 ppl over plain top‑k.\n\n## Mechanism — predicted observables *other* than final perplexity\nPerplexity is governed by the *distributional distance* between the training tokens\nand the target, which has **two** components. Both are training‑free and were\nmeasured directly.\n\n**(1) Mean per‑token target/pool log‑ratio (\"global affinity\") explains the large\ngain.** Prediction: the more target‑matched the tokens, the lower the ppl, *before\nany training*. Across the affinity‑increasing ablation this holds monotonically:\n\n| selection | global affinity (↑) | ppl (↓) |\n|---|---|---|\n| random | −0.799 | 457.2 |\n| quality‑gates‑only | −0.707 | 439.9 |\n| DSIR top‑k + dedup | −0.498 | 355.0 |\n\n**(2) Register‑mixture L1 distance explains the *final* gain — and is the sharper,\npre‑registered test, because it moves ppl *against* the affinity signal.**\nPrediction (made before running the stratified selection): forcing the selected\nregister proportions to match the target's will lower ppl *even though it slightly\nlowers global affinity* (quota admits some lower‑affinity docs from starved\nregisters). Measured over the 12M tokens actually trained on:\n\n| selection | global affinity | register‑mix L1 to target (↓) | ppl (↓) |\n|---|---|---|---|\n| DSIR top‑k + dedup | −0.498 | **0.429** | 355.0 |\n| register‑stratified | −0.530 | **0.002** | **348.2** |\n\nTop‑k, chasing average affinity, crushes the target's second‑largest register\n(target share 0.253 → 0.073) and over‑fills the highest‑scoring one. Stratification\nrestores the mixture to L1 ≈ 0.002 and ppl drops 6.7 — **while global affinity gets\nslightly worse**. So affinity is *not* a strictly monotone predictor once\ncomposition changes; the missing variable is register‑mixture distance, exactly as\npredicted. Training is deterministic (seed 1337: identical data ⇒ identical ppl),\nso 6.7 ppl is real signal, not run‑to‑run noise.\n\n**(3) The gain decomposes; junk‑removal is the *minor* term.** ~34% of pool\ndocuments fail basic English‑prose gates (non‑English, nav‑menu boilerplate,\nsymbol/number soup).\n\n| step added | ppl | Δ | share of the 109‑ppl gain |\n|---|---|---|---|\n| random → quality gates | 457.2 → 439.9 | −17.3 | 16% |\n| gates → target‑matching (DSIR rank) | 439.9 → 362.9 | −77.0 | **71%** |\n| matching → near‑dup dedup | 362.9 → 355.0 | −7.9 | 7% |\n| top‑k dedup → register‑stratified | 355.0 → 348.2 | −6.7 | 6% |\n\nThe dominant lever for a *broad* target is aligning the training distribution to it\n(71%), not merely filtering bad pages (16%); dedup and register‑balance each add a\nfurther ~6–7%.\n\n## Falsification\nThe hypothesis is falsified if any of the following held (none did):\n- Curated ppl ≥ random ppl (observed 348.2 ≪ 457.2).\n- **Register‑stratification does not beat plain top‑k** — i.e. forcing the mixture\n  to match the target (L1 0.429 → 0.002) fails to lower ppl, or raises it. This is\n  the load‑bearing test, because stratification *lowers* global affinity\n  (−0.498 → −0.530): if ppl tracked affinity alone it would have gone *up*. Observed\n  −6.7 ppl, so mixture‑matching is a genuine, separate lever.\n- Quality‑gates‑only matches or beats target‑matching (observed: matching adds\n  77 ppl, 4.5× the gate term).\n- Removing dedup does not raise ppl (observed +7.9 ppl without it).\n- The gain is within training noise (impossible: training is seed‑fixed and\n  deterministic; identical selections reproduce ppl to the digit).\nA further pre‑registered test: a selection built to *maximize* affinity but composed\nof gibberish/rare‑word documents (the naive‑DSIR failure mode we corrected by\nclamping rare‑n‑gram weights to ≤0) should **not** lower ppl — matching must be to\ngenuine target n‑grams, not to smoothing artifacts of rare tokens.\n\n## Transfer\n- **Other targets:** the criterion is target‑agnostic — rebuild the n‑gram profile\n  and the register clustering from any reference corpus and the same `curate.py`\n  curates for that domain's mixture.\n- **Hidden official target:** it is a disjoint sample of the *same* HQ domain, and\n  our features are aggregated over ~2,700 target segments and 8 register clusters\n  (domain‑level, not memorized documents). We matched the dev register mixture to\n  L1 ≈ 0.002; the hidden sample shares the same registers, so we predict a\n  comparable ~24% reduction vs random there.\n- **Budget / model scale:** the priority ordering is budget‑independent (a larger\n  budget fills the same quotas deeper); a larger model still benefits from\n  on‑distribution, deduplicated, register‑balanced tokens — with diminishing returns\n  as the budget approaches the pool size (less to gain from being selective).\n- **Recipe portability:** importance‑ratio scoring + Gopher/C4 gates + register\n  stratification + MinHash dedup is a standard, reproducible pretraining‑curation\n  pipeline, requiring no labels and no target‑document memorization. `curate.py`\n  regenerates `selection.json` deterministically (crc32 hashing, fixed k‑means seed).\n", "structuredPatch": [{"oldStart": 1, "oldLines": 81, "newStart": 1, "newLines": 113, "lines": ["-# Claim: distribution‑matched, quality‑filtered, deduplicated curation minimizes held‑out perplexity under a fixed token budget", "+# Claim: matching the target's *register mixture* (not just its average n‑gram profile), behind quality gates and dedup, minimizes held‑out perplexity under a fixed token budget", " ", " ## Hypothesis", "-Under a fixed 12M‑token budget (~6% of the 200M‑token pool), held‑out perplexity on", "-the broad high‑quality target (Wikipedia / web prose / news / technical Q&A) is", "-minimized **not by which documents are individually \"good\", but by making the", "-selected token distribution match the target distribution while spending no budget", "-on web junk or redundant copies.** Concretely, ranking pool documents by their", "-**per‑token target‑vs‑pool n‑gram log‑likelihood ratio** (DSIR‑style importance),", "-behind cheap English‑prose quality gates, then deduplicating, drives held‑out", "-perplexity far below a random selection.", "+Under a fixed 12M‑token budget (~6% of the ~193M‑token pool), held‑out perplexity", "+on the broad high‑quality target (Wikipedia / web prose / news / technical Q&A) is", "+minimized **not by picking the individually \"best\" documents, but by making the", "+selected token distribution match the target distribution** — spending no budget on", "+web junk or redundant copies. The target is a *mixture* of sub‑registers, and the", "+decisive move is to reproduce that mixture in the training data, not to maximize", "+average target‑affinity (which over‑selects one register and starves the others).", " ", "+Concretely: rank pool documents by their **per‑token target‑vs‑pool n‑gram", "+log‑likelihood ratio** (DSIR‑style importance), behind cheap English‑prose quality", "+gates; then, rather than taking the global top‑k, **cluster the target into K", "+sub‑registers, and fill the budget by quota so each register contributes tokens in", "+proportion to its share of the target**, deduplicating near‑copies as we go.", "+", " Measured (frozen 30M GPT, seed 1337, dev target):", " ", " | selection | held‑out ppl |", " |---|---|", " | random pool sample (baseline) | 457.2 |", "-| **curated (this method, `selection.json`)** | **355.0** |", "+| DSIR top‑k + gates + dedup | 355.0 |", "+| **register‑stratified (this method, `selection.json`)** | **348.2** |", " ", "-→ **22.4% lower perplexity**, from data selection alone.", "+→ **23.8% lower perplexity than random**, from data selection alone; the", "+register‑stratification step alone accounts for the final 6.7 ppl over plain top‑k.", " ", "-## Mechanism — predicted observable *other* than final perplexity", "-The effect is *distributional*, not per‑document \"quality\", and it is additively", "-decomposable. I verified two observables that are independent of the final score:", "+## Mechanism — predicted observables *other* than final perplexity", "+Perplexity is governed by the *distributional distance* between the training tokens", "+and the target, which has **two** components. Both are training‑free and were", "+measured directly.", " ", "-**(1) A training‑free statistic — mean per‑token target/pool log‑ratio of the packed", "-12M tokens — is a monotone predictor of held‑out perplexity.** Prediction: the more", "-target‑matched the tokens, the lower the ppl, *before any training*. Verified across", "-three selections:", "+**(1) Mean per‑token target/pool log‑ratio (\"global affinity\") explains the large", "+gain.** Prediction: the more target‑matched the tokens, the lower the ppl, *before", "+any training*. Across the affinity‑increasing ablation this holds monotonically:", " ", "-| selection | match‑stat (↑ better) | ppl (↓ better) |", "+| selection | global affinity (↑) | ppl (↓) |", " |---|---|---|", " | random | −0.799 | 457.2 |", "-| quality‑gates‑only (no matching) | −0.707 | 439.9 |", "-| curated (matching + dedup) | −0.498 | 355.0 |", "+| quality‑gates‑only | −0.707 | 439.9 |", "+| DSIR top‑k + dedup | −0.498 | 355.0 |", " ", "-Strictly monotone: closing the distributional gap tracks the perplexity drop.", "+**(2) Register‑mixture L1 distance explains the *final* gain — and is the sharper,", "+pre‑registered test, because it moves ppl *against* the affinity signal.**", "+Prediction (made before running the stratified selection): forcing the selected", "+register proportions to match the target's will lower ppl *even though it slightly", "+lowers global affinity* (quota admits some lower‑affinity docs from starved", "+registers). Measured over the 12M tokens actually trained on:", " ", "-**(2) The gain decomposes, and junk‑removal is the *minor* term.** ~34% of pool", "+| selection | global affinity | register‑mix L1 to target (↓) | ppl (↓) |", "+|---|---|---|---|", "+| DSIR top‑k + dedup | −0.498 | **0.429** | 355.0 |", "+| register‑stratified | −0.530 | **0.002** | **348.2** |", "+", "+Top‑k, chasing average affinity, crushes the target's second‑largest register", "+(target share 0.253 → 0.073) and over‑fills the highest‑scoring one. Stratification", "+restores the mixture to L1 ≈ 0.002 and ppl drops 6.7 — **while global affinity gets", "+slightly worse**. So affinity is *not* a strictly monotone predictor once", "+composition changes; the missing variable is register‑mixture distance, exactly as", "+predicted. Training is deterministic (seed 1337: identical data ⇒ identical ppl),", "+so 6.7 ppl is real signal, not run‑to‑run noise.", "+", "+**(3) The gain decomposes; junk‑removal is the *minor* term.** ~34% of pool", " documents fail basic English‑prose gates (non‑English, nav‑menu boilerplate,", "-symbol/number soup). A selection that *only* removes junk (gates, random order)", "-was predicted to land strictly between random and the full method — and does, but", "-close to random:", "+symbol/number soup).", " ", "-| step added | ppl | Δ | share of the 102‑ppl gain |", "+| step added | ppl | Δ | share of the 109‑ppl gain |", " |---|---|---|---|", "-| random → quality gates | 457.2 → 439.9 | −17.3 | 17% |", "-| gates → target‑matching (DSIR rank) | 439.9 → 362.9 | −77.0 | **75%** |", "-| matching → dedup | 362.9 → 355.0 | −7.9 | 8% |", "+| random → quality gates | 457.2 → 439.9 | −17.3 | 16% |", "+| gates → target‑matching (DSIR rank) | 439.9 → 362.9 | −77.0 | **71%** |", "+| matching → near‑dup dedup | 362.9 → 355.0 | −7.9 | 7% |", "+| top‑k dedup → register‑stratified | 355.0 → 348.2 | −6.7 | 6% |", " ", "-So the dominant lever for a *broad* target is aligning the training distribution to", "-it (75%), not merely filtering bad pages (17%); near‑dup removal adds a further 8%.", "+The dominant lever for a *broad* target is aligning the training distribution to it", "+(71%), not merely filtering bad pages (16%); dedup and register‑balance each add a", "+further ~6–7%.", " ", " ## Falsification", " The hypothesis is falsified if any of the following held (none did):", "-- Curated ppl ≥ random ppl (observed 355.0 ≪ 457.2).", "-- The match‑stat is **not** monotone with ppl across {random, gates‑only, curated}", "-  — e.g. a more target‑matched selection yielding *higher* ppl would refute the", "-  distribution‑matching mechanism (observed strictly monotone).", "-- Quality‑gates‑only matches or beats the full method, i.e. target‑matching adds", "-  nothing beyond junk removal (observed: matching adds 77 ppl, 4.5× the gate term).", "+- Curated ppl ≥ random ppl (observed 348.2 ≪ 457.2).", "+- **Register‑stratification does not beat plain top‑k** — i.e. forcing the mixture", "+  to match the target (L1 0.429 → 0.002) fails to lower ppl, or raises it. This is", "+  the load‑bearing test, because stratification *lowers* global affinity", "+  (−0.498 → −0.530): if ppl tracked affinity alone it would have gone *up*. Observed", "+  −6.7 ppl, so mixture‑matching is a genuine, separate lever.", "+- Quality‑gates‑only matches or beats target‑matching (observed: matching adds", "+  77 ppl, 4.5× the gate term).", " - Removing dedup does not raise ppl (observed +7.9 ppl without it).", "-A further pre‑registered test: a selection built to *maximize* the match‑stat but", "-composed of gibberish/rare‑word documents (the naive‑DSIR failure mode we corrected)", "-should **not** lower ppl — matching must be to genuine target n‑grams, not to", "-smoothing artifacts of rare tokens.", "+- The gain is within training noise (impossible: training is seed‑fixed and", "+  deterministic; identical selections reproduce ppl to the digit).", "+A further pre‑registered test: a selection built to *maximize* affinity but composed", "+of gibberish/rare‑word documents (the naive‑DSIR failure mode we corrected by", "+clamping rare‑n‑gram weights to ≤0) should **not** lower ppl — matching must be to", "+genuine target n‑grams, not to smoothing artifacts of rare tokens.", " ", " ## Transfer", " - **Other targets:** the criterion is target‑agnostic — rebuild the n‑gram profile", "-  from any reference corpus and the same `curate.py` curates for that domain.", "+  and the register clustering from any reference corpus and the same `curate.py`", "+  curates for that domain's mixture.", " - **Hidden official target:** it is a disjoint sample of the *same* HQ domain, and", "-  our features are aggregated over 2,705 target segments (domain‑level, not", "-  memorized documents), so we predict a comparable ~20% reduction vs random there.", "-- **Budget / model scale:** the priority ranking is budget‑independent; a larger", "-  budget simply consumes further down the same list, and a larger model still", "-  benefits from on‑distribution, deduplicated tokens — with diminishing returns as", "-  the budget approaches the pool size (less to gain from being selective).", "-- **Recipe portability:** importance‑ratio scoring + Gopher/C4 gates + MinHash dedup", "-  is a standard, reproducible pretraining‑curation pipeline, requiring no labels and", "-  no target document memorization; `curate.py` regenerates `selection.json`", "-  deterministically.", "+  our features are aggregated over ~2,700 target segments and 8 register clusters", "+  (domain‑level, not memorized documents). We matched the dev register mixture to", "+  L1 ≈ 0.002; the hidden sample shares the same registers, so we predict a", "+  comparable ~24% reduction vs random there.", "+- **Budget / model scale:** the priority ordering is budget‑independent (a larger", "+  budget fills the same quotas deeper); a larger model still benefits from", "+  on‑distribution, deduplicated, register‑balanced tokens — with diminishing returns", "+  as the budget approaches the pool size (less to gain from being selective).", "+- **Recipe portability:** importance‑ratio scoring + Gopher/C4 gates + register", "+  stratification + MinHash dedup is a standard, reproducible pretraining‑curation", "+  pipeline, requiring no labels and no target‑document memorization. `curate.py`", "+  regenerates `selection.json` deterministically (crc32 hashing, fixed k‑means seed)."]}], "originalFile": "# Claim: distribution‑matched, quality‑filtered, deduplicated curation minimizes held‑out perplexity under a fixed token budget\n\n## Hypothesis\nUnder a fixed 12M‑token budget (~6% of the 200M‑token pool), held‑out perplexity on\nthe broad high‑quality target (Wikipedia / web prose / news / technical Q&A) is\nminimized **not by which documents are individually \"good\", but by making the\nselected token distribution match the target distribution while spending no budget\non web junk or redundant copies.** Concretely, ranking pool documents by their\n**per‑token target‑vs‑pool n‑gram log‑likelihood ratio** (DSIR‑style importance),\nbehind cheap English‑prose quality gates, then deduplicating, drives held‑out\nperplexity far below a random selection.\n\nMeasured (frozen 30M GPT, seed 1337, dev target):\n\n| selection | held‑out ppl |\n|---|---|\n| random pool sample (baseline) | 457.2 |\n| **curated (this method, `selection.json`)** | **355.0** |\n\n→ **22.4% lower perplexity**, from data selection alone.\n\n## Mechanism — predicted observable *other* than final perplexity\nThe effect is *distributional*, not per‑document \"quality\", and it is additively\ndecomposable. I verified two observables that are independent of the final score:\n\n**(1) A training‑free statistic — mean per‑token target/pool log‑ratio of the packed\n12M tokens — is a monotone predictor of held‑out perplexity.** Prediction: the more\ntarget‑matched the tokens, the lower the ppl, *before any training*. Verified across\nthree selections:\n\n| selection | match‑stat (↑ better) | ppl (↓ better) |\n|---|---|---|\n| random | −0.799 | 457.2 |\n| quality‑gates‑only (no matching) | −0.707 | 439.9 |\n| curated (matching + dedup) | −0.498 | 355.0 |\n\nStrictly monotone: closing the distributional gap tracks the perplexity drop.\n\n**(2) The gain decomposes, and junk‑removal is the *minor* term.** ~34% of pool\ndocuments fail basic English‑prose gates (non‑English, nav‑menu boilerplate,\nsymbol/number soup). A selection that *only* removes junk (gates, random order)\nwas predicted to land strictly between random and the full method — and does, but\nclose to random:\n\n| step added | ppl | Δ | share of the 102‑ppl gain |\n|---|---|---|---|\n| random → quality gates | 457.2 → 439.9 | −17.3 | 17% |\n| gates → target‑matching (DSIR rank) | 439.9 → 362.9 | −77.0 | **75%** |\n| matching → dedup | 362.9 → 355.0 | −7.9 | 8% |\n\nSo the dominant lever for a *broad* target is aligning the training distribution to\nit (75%), not merely filtering bad pages (17%); near‑dup removal adds a further 8%.\n\n## Falsification\nThe hypothesis is falsified if any of the following held (none did):\n- Curated ppl ≥ random ppl (observed 355.0 ≪ 457.2).\n- The match‑stat is **not** monotone with ppl across {random, gates‑only, curated}\n  — e.g. a more target‑matched selection yielding *higher* ppl would refute the\n  distribution‑matching mechanism (observed strictly monotone).\n- Quality‑gates‑only matches or beats the full method, i.e. target‑matching adds\n  nothing beyond junk removal (observed: matching adds 77 ppl, 4.5× the gate term).\n- Removing dedup does not raise ppl (observed +7.9 ppl without it).\nA further pre‑registered test: a selection built to *maximize* the match‑stat but\ncomposed of gibberish/rare‑word documents (the naive‑DSIR failure mode we corrected)\nshould **not** lower ppl — matching must be to genuine target n‑grams, not to\nsmoothing artifacts of rare tokens.\n\n## Transfer\n- **Other targets:** the criterion is target‑agnostic — rebuild the n‑gram profile\n  from any reference corpus and the same `curate.py` curates for that domain.\n- **Hidden official target:** it is a disjoint sample of the *same* HQ domain, and\n  our features are aggregated over 2,705 target segments (domain‑level, not\n  memorized documents), so we predict a comparable ~20% reduction vs random there.\n- **Budget / model scale:** the priority ranking is budget‑independent; a larger\n  budget simply consumes further down the same list, and a larger model still\n  benefits from on‑distribution, deduplicated tokens — with diminishing returns as\n  the budget approaches the pool size (less to gain from being selective).\n- **Recipe portability:** importance‑ratio scoring + Gopher/C4 gates + MinHash dedup\n  is a standard, reproducible pretraining‑curation pipeline, requiring no labels and\n  no target document memorization; `curate.py` regenerates `selection.json`\n  deterministically.\n", "userModified": false}

=== DELIVERABLES ===
-rw-r--r-- 1 root root   6749 Aug  4 05:33 claim.md
-rw-r--r-- 1 root root  15018 Aug  4 05:25 submission/curate.py
-rw-r--r-- 1 root root 127043 Aug  4 05:27 submission/selection.json

=== selection.json validity ===
type: list of int = True
count: 17987 | unique: True | range: [17,181838] OK

=== curate.py reproduces selection.json from criterion (no --rebuild) ===
wrote 17987 ids -> /tmp/repro_final.json; ~15.0M est tokens (budget 12M)
reproducible == deliverable: True

=== certified score ===
HELDOUT_PPL 348.225 (random baseline 457.226 -> 23.8% reduction)
timer remaining:
1643

[stdout]
=== DELIVERABLES ===
-rw-r--r-- 1 root root   6749 Aug  4 05:33 claim.md
-rw-r--r-- 1 root root  15018 Aug  4 05:25 submission/curate.py
-rw-r--r-- 1 root root 127043 Aug  4 05:27 submission/selection.json

=== selection.json validity ===
type: list of int = True
count: 17987 | unique: True | range: [17,181838] OK

=== curate.py reproduces selection.json from criterion (no --rebuild) ===
wrote 17987 ids -> /tmp/repro_final.json; ~15.0M est tokens (budget 12M)
reproducible == deliverable: True

=== certified score ===
HELDOUT_PPL 348.225 (random baseline 457.226 -> 23.8% reduction)
timer remaining:
1643
[metadata] {"noOutputExpected": false}

list: 17987 ids | est cum total 15.0M | quota boundary ~id#15371 (12.6M est)
ACTUAL tokens: total 14.10M | 12M budget reached at id#15598 of 17987
quota portion actual tokens (first 15371 ids): 11.70M
=> packer stays within quota portion: False
timer:
1512

[stdout]
list: 17987 ids | est cum total 15.0M | quota boundary ~id#15371 (12.6M est)
ACTUAL tokens: total 14.10M | 12M budget reached at id#15598 of 17987
quota portion actual tokens (first 15371 ids): 11.70M
=> packer stays within quota portion: False
timer:
1512
[metadata] {"noOutputExpected": false}

The file /workspace/submission/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/submission/curate.py", "oldString": "STRAT_BUD   = 12_600_000 # quota base + round-robin fill target (small margin over BUDGET)\nEMIT_MIN    = 15_000_000 # guarantee at least this many est tokens are emitted (headroom)", "newString": "STRAT_BUD   = 13_800_000 # quota base + round-robin fill target; sized so the quota-\n                         # matched portion alone covers the full 12M ACTUAL tokens\n                         # (est=nchars/4 runs ~7% high) -> packer never reaches fallback\nEMIT_MIN    = 15_000_000 # guarantee at least this many est tokens are emitted (headroom)", "originalFile": null, "structuredPatch": [{"oldStart": 76, "oldLines": 7, "newStart": 76, "newLines": 9, "lines": [" CLUST_FEATS = 1500       # top unigram features used to cluster target segments", " CLUST_ITERS = 30         # k-means iterations", " CLUST_SEED  = 0          # k-means init RNG seed (determinism)", "-STRAT_BUD   = 12_600_000 # quota base + round-robin fill target (small margin over BUDGET)", "+STRAT_BUD   = 13_800_000 # quota base + round-robin fill target; sized so the quota-", "+                         # matched portion alone covers the full 12M ACTUAL tokens", "+                         # (est=nchars/4 runs ~7% high) -> packer never reaches fallback", " EMIT_MIN    = 15_000_000 # guarantee at least this many est tokens are emitted (headroom)", " ", " # near-dup dedup"]}], "userModified": false, "replaceAll": false}