nano-gpt-data-curation

claude-opus-4-8

nano-gpt-data-curation — graded by the task's verifier against a measured baseline.

backend: modal
variant: claude claude-opus-4-8
metric: held-out multi-domain perplexity
actions: 87
duration: 1h 21m
recorded: 2026-08-04

run iddclm-nano-curation__7umcSHc

Everything claude-opus-4-8 did on this task, in order.

download JSON ↗

claude-codeclaude-opus-4-885 turns69 narration87 action87 observation16 reasoning1 final

#26

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolTaskUpdate

args

{
  "taskId": "2",
  "status": "in_progress"
}

TaskUpdate result

Updated task #2 status

[metadata] {"success": true, "taskId": "2", "updatedFields": ["status"], "statusChange": {"from": "pending", "to": "in_progress"}}

#27

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolBashCheck dedup analysis output

args

{
  "command": "sleep 30; echo \"=== dedup analysis ===\"; cat /tmp/claude-0/-workspace/d723e7a0-a155-47f3-b352-d47c686c4960/tasks/bwxgzhj46.output 2>/dev/null; echo \"(end)\"; cat /workspace/.timer/remaining_secs",
  "description": "Check dedup analysis output"
}

Bash result

=== dedup analysis ===
(end)
4058

[stdout]
=== dedup analysis ===
(end)
4058
[metadata] {"noOutputExpected": false}

#28

narration

Baseline confirms the win (457→363). The dedup analysis stalled on the 1GB cache load; rather than keep measuring, I'll build dedup directly and let dev_eval decide. My top picks had 4 near-duplicate Modi/BJP news items in the top-10, so syndicated redundancy is real. Let me generate a deduplicated V2 selection (MinHash+LSH over unigram sets, in score order) and evaluate it:

toolBashBuild deduplicated V2 selection

args

{
  "command": "cd /workspace; cat > /tmp/dedup_select.py <<'PY'\nimport numpy as np, math, json, time\nt0=time.time()\nz=np.load('/tmp/feat_cache.npz')\nCt,Cr=z['Ct'],z['Cr']; T=len(Ct); A=1.0\nlr=(np.log(Ct+A)-math.log(Ct.sum()+A*T))-(np.log(Cr+A)-math.log(Cr.sum()+A*T))\nweak=Ct<2; lr[weak]=np.minimum(lr[weak],0.0)\noff=z['off']; allf=z['allfeats']; vals=lr[allf]; counts=np.diff(off)\nsums=np.add.reduceat(vals,off[:-1]); sums[counts==0]=-1e18\ns=sums/np.maximum(counts,1); s[~z['keep']]=-1e18\norder=np.argsort(-s); ids=z['ids']; est=z['nchars']/4.0\npassing=order[s[order]>-1e17]\nD_UNI=1<<20; K=16; B=4; R=4\nseen_bands=set()\nsel=[]; cum=0; ndup=0\nfor j in passing:\n    a,b=off[j],off[j+1]\n    f=allf[a:b]; uni=np.unique(f[f<D_UNI])\n    sk=uni[:K]\n    if len(sk)>=B*R:\n        bands=[hash(sk[i*R:(i+1)*R].tobytes()) for i in range(B)]\n        if any(bd in seen_bands for bd in bands):\n            ndup+=1; continue\n        for bd in bands: seen_bands.add(bd)\n    sel.append(int(ids[j])); cum+=est[j]\n    if cum>=16_000_000: break\njson.dump(sel, open('/tmp/sel_v2_dedup.json','w'))\nprint('deduped selection: %d ids, ~%.1fM est tokens, skipped %d near-dups, %.0fs'%(len(sel),cum/1e6,ndup,time.time()-t0))\nPY\npython3 /tmp/dedup_select.py",
  "description": "Build deduplicated V2 selection"
}

Bash result

deduped selection: 16769 ids, ~16.0M est tokens, skipped 677 near-dups, 4s

[stdout]
deduped selection: 16769 ids, ~16.0M est tokens, skipped 677 near-dups, 4s
[metadata] {"noOutputExpected": false}

#29

narration

Only 677 near-dups (~4%) — the top-10 Modi items were distinct articles, not true duplicates, so duplication is modest. Note training is deterministic (fixed seed 1337), so even small ppl differences are real signal, not noise. Let me eval V2 (dedup) and, in parallel thinking, prepare a register-balanced variant. Launching V2:

toolBashLaunch V2 dedup eval

args

{
  "command": "cd /workspace; export TOKENIZERS_PARALLELISM=false; nohup bash dev_eval.sh /tmp/sel_v2_dedup.json > /tmp/eval_v2.log 2>&1 & echo \"started V2 dedup eval pid $!\"; cat /workspace/.timer/remaining_secs",
  "description": "Launch V2 dedup eval"
}

Bash result

started V2 dedup eval pid 2173
3968

[stdout]
started V2 dedup eval pid 2173
3968
[metadata] {"noOutputExpected": false}

#30

narration

V2 is training. Let me check its result, then build the more principled variant. The theory: pure top-k maximizes average target-likeness but can distort the target's register proportions; since the target is "equal parts," a stratified distribution-match (cluster the target, select pool docs proportional to each cluster's target mass) should match the target distribution better. Let me check V2 first:

toolBashCheck V2 result

args

{
  "command": "cd /workspace; sleep 25; echo \"=== V2 dedup result ===\"; grep HELDOUT_PPL /tmp/eval_v2.log 2>/dev/null || echo \"still training...\"; cat /workspace/.timer/remaining_secs",
  "description": "Check V2 result"
}

Bash result

=== V2 dedup result ===
HELDOUT_PPL 354.963  loss 5.8720  train_s 56  /tmp/dev_train.npy
3857

[stdout]
=== V2 dedup result ===
HELDOUT_PPL 354.963  loss 5.8720  train_s 56  /tmp/dev_train.npy
3857
[metadata] {"noOutputExpected": false}

#31

narration

V2 (dedup) = 354.96 — dedup helped ~8 ppl (2.2%) even at 4% docs changed. Keeping it. Scoreboard: baseline 457.2 → V1 363 → V2 355.

Now the stratified variant. Let me build V3: cluster the target into 8 sub-registers, compute a per-cluster DSIR weight, and fill the budget proportional to each cluster's target mass (round-robin so the 12M prefix stays balanced) + dedup. This matches the target distribution rather than just maximizing average affinity.

toolBashBuild V3 stratified balanced selection

args

{
  "command": "cd /workspace; cat > /tmp/v3_strat.py <<'PY'\nimport numpy as np, math, json, time, sys\nsys.path.insert(0,'/workspace/submission')\nfrom curate import doc_ngrams, WTOK, D_UNI, T, ALPHA\nfrom transformers import AutoTokenizer\nt0=time.time(); A=ALPHA; MINCT=2; K=8\nz=np.load('/tmp/feat_cache.npz')\nallf=z['allfeats']; off=z['off']; ids=z['ids']; keep=z['keep']; nchars=z['nchars']; Cr=z['Cr'].astype(np.float64)\nNr=Cr.sum(); est=nchars/4.0\n\n# ---- dev per-segment ngrams + clustering ----\ntok=AutoTokenizer.from_pretrained('gpt2'); d=np.load('/workspace/data/multi_dev.npy')\nidx=np.where(d==50256)[0]; segs=[]; prev=0\nfor i in idx: segs.append((prev,i)); prev=i+1\nsegs.append((prev,len(d)))\nseg_ng=[]; seg_len=[]\nuni_glob=np.zeros(D_UNI,np.float64)\nfor a,b in segs:\n    if b-a<5: seg_ng.append(np.empty(0,np.int32)); seg_len.append(0); continue\n    w=WTOK.findall(tok.decode(d[a:b]).lower()); g=doc_ngrams(w)\n    seg_ng.append(g); seg_len.append(b-a)\n    u=g[g<D_UNI]\n    if len(u): uni_glob+=np.bincount(u,minlength=D_UNI)\n# top-F unigram buckets as clustering features\nF=1500; topb=np.argsort(-uni_glob)[:F]; col={int(b):i for i,b in enumerate(topb)}\nM=np.zeros((len(segs),F),np.float32)\nfor si,g in enumerate(seg_ng):\n    u=g[g<D_UNI]\n    for h in u:\n        c=col.get(int(h));\n        if c is not None: M[si,c]+=1\nnz=M.sum(1)>0\nM=M/np.maximum(np.linalg.norm(M,axis=1,keepdims=True),1e-9)\n# kmeans\nrng=np.random.default_rng(0); init=rng.choice(np.where(nz)[0],K,replace=False)\nC=M[init].copy()\nfor it in range(25):\n    sim=M@C.T; lab=sim.argmax(1)\n    for c in range(K):\n        m=(lab==c)&nz\n        if m.sum()>0: C[c]=M[m].mean(0); C[c]/=max(np.linalg.norm(C[c]),1e-9)\nlab[~nz]=-1\n# per-cluster target counts + fraction of target tokens\nCt_c=[np.zeros(T,np.float64) for _ in range(K)]; tok_c=np.zeros(K)\nfor si,g in enumerate(seg_ng):\n    c=lab[si]\n    if c<0 or len(g)==0: continue\n    Ct_c[c]+=np.bincount(g,minlength=T); tok_c[c]+=seg_len[si]\nfrac=tok_c/tok_c.sum()\nprint('cluster target-token fractions:',np.round(frac,3),' sizes:',[int((lab==c).sum()) for c in range(K)])\n\n# ---- score pool docs under each cluster ----\ncounts=np.diff(off)\nS=np.full((len(ids),K),-1e18,np.float64)\nfor c in range(K):\n    Ntc=Ct_c[c].sum()\n    lrc=(np.log(Ct_c[c]+A)-math.log(Ntc+A*T))-(np.log(Cr+A)-math.log(Nr+A*T))\n    weak=Ct_c[c]<MINCT; lrc[weak]=np.minimum(lrc[weak],0.0)\n    v=lrc[allf]; sums=np.add.reduceat(v,off[:-1]); sums[counts==0]=-1e18\n    S[:,c]=sums/np.maximum(counts,1)\nS[~keep]=-1e18\nbest_c=S.argmax(1); best_s=S.max(1)\nvalid=best_s>-1e17\n\n# ---- per-cluster ranked lists ----\norder_c=[]\nfor c in range(K):\n    m=np.where(valid&(best_c==c))[0]\n    m=m[np.argsort(-best_s[m])]\n    order_c.append(list(m))\n    print('cluster %d: %d docs assigned'%(c,len(m)))\n\n# ---- round-robin quota fill with dedup ----\nBUD=12_600_000; quota=BUD*frac\nseen=set(); ptr=[0]*K; taken=np.zeros(K); final=[]; cumtok=0\ndef sketch(j):\n    f=allf[off[j]:off[j+1]]; u=np.unique(f[f<D_UNI]); return u[:16]\ndef isdup(j):\n    sk=sketch(j)\n    if len(sk)<16: return False\n    bands=[hash(sk[i*4:(i+1)*4].tobytes()) for i in range(4)]\n    if any(b in seen for b in bands): return True\n    for b in bands: seen.add(b); \n    return False\nactive=set(c for c in range(K) if len(order_c[c])>0 and quota[c]>0)\nwhile active and cumtok<BUD:\n    # cluster most behind its quota\n    c=min(active,key=lambda c:taken[c]/max(quota[c],1))\n    adv=False\n    while ptr[c]<len(order_c[c]):\n        j=order_c[c][ptr[c]]; ptr[c]+=1\n        if isdup(j): continue\n        final.append(int(ids[j])); taken[c]+=est[j]; cumtok+=est[j]; adv=True; break\n    if ptr[c]>=len(order_c[c]) or taken[c]>=quota[c]*1.05:\n        active.discard(c)\n    if not adv and ptr[c]>=len(order_c[c]):\n        active.discard(c)\njson.dump(final, open('/tmp/sel_v3_strat.json','w'))\nprint('V3 stratified: %d ids, ~%.1fM est tok, per-cluster tokens:'%(len(final),cumtok/1e6),np.round(taken/1e6,2),' %.0fs'%(time.time()-t0))\nPY\npython3 /tmp/v3_strat.py 2>&1 | grep -v Warning",
  "description": "Build V3 stratified balanced selection"
}

Bash result

cluster target-token fractions: [0.376 0.044 0.037 0.097 0.043 0.001 0.154 0.249]  sizes: [550, 247, 173, 459, 222, 22, 677, 355]
cluster 0: 120193 docs assigned
cluster 1: 0 docs assigned
cluster 2: 0 docs assigned
cluster 3: 2 docs assigned
cluster 4: 0 docs assigned
cluster 5: 0 docs assigned
cluster 6: 2 docs assigned
cluster 7: 283 docs assigned
V3 stratified: 6195 ids, ~5.1M est tok, per-cluster tokens: [4.97 0.   0.   0.   0.   0.   0.   0.12]  35s

[stdout]
cluster target-token fractions: [0.376 0.044 0.037 0.097 0.043 0.001 0.154 0.249]  sizes: [550, 247, 173, 459, 222, 22, 677, 355]
cluster 0: 120193 docs assigned
cluster 1: 0 docs assigned
cluster 2: 0 docs assigned
cluster 3: 2 docs assigned
cluster 4: 0 docs assigned
cluster 5: 0 docs assigned
cluster 6: 2 docs assigned
cluster 7: 283 docs assigned
V3 stratified: 6195 ids, ~5.1M est tok, per-cluster tokens: [4.97 0.   0.   0.   0.   0.   0.   0.12]  35s
[metadata] {"noOutputExpected": false}

#32

narration

The clustering collapsed — almost all docs fell into the largest cluster because per-cluster log-ratios aren't scale-comparable across clusters (argmax bias toward the cluster with most target mass). Stratification is fragile and I've spent one attempt on it; the top-k mix was already balanced, so I'll not sink more time here. V2 (DSIR + gates + dedup = 355) is my best, principled result.

Let me fold dedup into curate.py (the deliverable must reproduce selection.json), computing sketches from the cached n-grams so no rebuild is needed:

toolRead

args

{
  "file_path": "/workspace/submission/curate.py",
  "offset": 185,
  "limit": 60
}

Read result

185	    est = cache["nchars"][passing] / 4.0
186	    cum = np.cumsum(est)
187	    nb = int(np.searchsorted(cum, BUDGET)) + 1
188	    print(f"passing gates: {len(passing)}/{len(ids)} "
189	          f"({100*len(passing)/len(ids):.1f}%)")
190	    print(f"wrote {min(a.top,len(sel))} ids -> {a.out}")
191	    print(f"~{nb} docs (~{cum[min(nb,len(cum))-1]/1e6:.1f}M est tok) fill 12M budget")
192	
193	    if a.diag:
194	        want = set(int(ids[j]) for j in passing[:10]) | \
195	               set(int(ids[j]) for j in passing[len(passing)//2:len(passing)//2+3])
196	        txt = {}
197	        with open(POOL) as f:
198	            for line in f:
199	                r = json.loads(line)
200	                if r["id"] in want:
201	                    txt[r["id"]] = r["text"]
202	        print("\n--- TOP 10 ---")
203	        for j in passing[:10]:
204	            i = int(ids[j])
205	            print(f"{s[j]:+.3f} id={i}: {txt[i][:170].strip().replace(chr(10),' | ')}")
206	        print("\n--- MEDIAN-RANK 3 ---")
207	        for j in passing[len(passing)//2:len(passing)//2+3]:
208	            i = int(ids[j])
209	            print(f"{s[j]:+.3f} id={i}: {txt[i][:170].strip().replace(chr(10),' | ')}")
210	
211	if __name__ == "__main__":
212	    main()
213

[metadata] {"type": "text", "file": {"filePath": "/workspace/submission/curate.py", "content": "    est = cache[\"nchars\"][passing] / 4.0\n    cum = np.cumsum(est)\n    nb = int(np.searchsorted(cum, BUDGET)) + 1\n    print(f\"passing gates: {len(passing)}/{len(ids)} \"\n          f\"({100*len(passing)/len(ids):.1f}%)\")\n    print(f\"wrote {min(a.top,len(sel))} ids -> {a.out}\")\n    print(f\"~{nb} docs (~{cum[min(nb,len(cum))-1]/1e6:.1f}M est tok) fill 12M budget\")\n\n    if a.diag:\n        want = set(int(ids[j]) for j in passing[:10]) | \\\n               set(int(ids[j]) for j in passing[len(passing)//2:len(passing)//2+3])\n        txt = {}\n        with open(POOL) as f:\n            for line in f:\n                r = json.loads(line)\n                if r[\"id\"] in want:\n                    txt[r[\"id\"]] = r[\"text\"]\n        print(\"\\n--- TOP 10 ---\")\n        for j in passing[:10]:\n            i = int(ids[j])\n            print(f\"{s[j]:+.3f} id={i}: {txt[i][:170].strip().replace(chr(10),' | ')}\")\n        print(\"\\n--- MEDIAN-RANK 3 ---\")\n        for j in passing[len(passing)//2:len(passing)//2+3]:\n            i = int(ids[j])\n            print(f\"{s[j]:+.3f} id={i}: {txt[i][:170].strip().replace(chr(10),' | ')}\")\n\nif __name__ == \"__main__\":\n    main()\n", "numLines": 29, "startLine": 185, "totalLines": 213}}

#33

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolRead

args

{
  "file_path": "/workspace/submission/curate.py",
  "offset": 140,
  "limit": 52
}

Read result

140	    np.savez(CACHE, allfeats=allfeats, ids=ids, keep=keep, nchars=nchars,
141	             off=off, Ct=Ct, Cr=Cr)
142	    print(f"    featurized {len(ids)} docs in {time.time()-t0:.0f}s -> {CACHE}")
143	    return dict(allfeats=allfeats, ids=ids, keep=keep, nchars=nchars,
144	                off=off, Ct=Ct, Cr=Cr)
145	
146	def load_cache(force=False):
147	    if not force and os.path.exists(CACHE):
148	        z = np.load(CACHE)
149	        return {k: z[k] for k in z.files}
150	    return build_cache()
151	
152	def score(cache, alpha=ALPHA, minct=MINCT):
153	    Ct, Cr = cache["Ct"], cache["Cr"]
154	    Nt, Nr = Ct.sum(), Cr.sum()
155	    lr = (np.log(Ct + alpha) - math.log(Nt + alpha * T)) \
156	       - (np.log(Cr + alpha) - math.log(Nr + alpha * T))
157	    # rarity correction: n-grams not solidly in target may only penalise, not reward
158	    weak = Ct < minct
159	    lr[weak] = np.minimum(lr[weak], 0.0)
160	    off = cache["off"]
161	    vals = lr[cache["allfeats"]]
162	    counts = np.diff(off)
163	    sums = np.add.reduceat(vals, off[:-1])
164	    sums[counts == 0] = -1e18
165	    s = sums / np.maximum(counts, 1)
166	    s[~cache["keep"]] = -1e18
167	    return s
168	
169	def main():
170	    ap = argparse.ArgumentParser()
171	    ap.add_argument("--out", default=OUT)
172	    ap.add_argument("--top", type=int, default=60000)
173	    ap.add_argument("--rebuild", action="store_true")
174	    ap.add_argument("--diag", action="store_true")
175	    a = ap.parse_args()
176	
177	    cache = load_cache(force=a.rebuild)
178	    s = score(cache)
179	    ids = cache["ids"]
180	    order = np.argsort(-s)
181	    passing = order[s[order] > -1e17]
182	    sel = [int(ids[j]) for j in passing]
183	    json.dump(sel[: a.top], open(a.out, "w"))
184	
185	    est = cache["nchars"][passing] / 4.0
186	    cum = np.cumsum(est)
187	    nb = int(np.searchsorted(cum, BUDGET)) + 1
188	    print(f"passing gates: {len(passing)}/{len(ids)} "
189	          f"({100*len(passing)/len(ids):.1f}%)")
190	    print(f"wrote {min(a.top,len(sel))} ids -> {a.out}")
191	    print(f"~{nb} docs (~{cum[min(nb,len(cum))-1]/1e6:.1f}M est tok) fill 12M budget")

[metadata] {"type": "text", "file": {"filePath": "/workspace/submission/curate.py", "content": "    np.savez(CACHE, allfeats=allfeats, ids=ids, keep=keep, nchars=nchars,\n             off=off, Ct=Ct, Cr=Cr)\n    print(f\"    featurized {len(ids)} docs in {time.time()-t0:.0f}s -> {CACHE}\")\n    return dict(allfeats=allfeats, ids=ids, keep=keep, nchars=nchars,\n                off=off, Ct=Ct, Cr=Cr)\n\ndef load_cache(force=False):\n    if not force and os.path.exists(CACHE):\n        z = np.load(CACHE)\n        return {k: z[k] for k in z.files}\n    return build_cache()\n\ndef score(cache, alpha=ALPHA, minct=MINCT):\n    Ct, Cr = cache[\"Ct\"], cache[\"Cr\"]\n    Nt, Nr = Ct.sum(), Cr.sum()\n    lr = (np.log(Ct + alpha) - math.log(Nt + alpha * T)) \\\n       - (np.log(Cr + alpha) - math.log(Nr + alpha * T))\n    # rarity correction: n-grams not solidly in target may only penalise, not reward\n    weak = Ct < minct\n    lr[weak] = np.minimum(lr[weak], 0.0)\n    off = cache[\"off\"]\n    vals = lr[cache[\"allfeats\"]]\n    counts = np.diff(off)\n    sums = np.add.reduceat(vals, off[:-1])\n    sums[counts == 0] = -1e18\n    s = sums / np.maximum(counts, 1)\n    s[~cache[\"keep\"]] = -1e18\n    return s\n\ndef main():\n    ap = argparse.ArgumentParser()\n    ap.add_argument(\"--out\", default=OUT)\n    ap.add_argument(\"--top\", type=int, default=60000)\n    ap.add_argument(\"--rebuild\", action=\"store_true\")\n    ap.add_argument(\"--diag\", action=\"store_true\")\n    a = ap.parse_args()\n\n    cache = load_cache(force=a.rebuild)\n    s = score(cache)\n    ids = cache[\"ids\"]\n    order = np.argsort(-s)\n    passing = order[s[order] > -1e17]\n    sel = [int(ids[j]) for j in passing]\n    json.dump(sel[: a.top], open(a.out, \"w\"))\n\n    est = cache[\"nchars\"][passing] / 4.0\n    cum = np.cumsum(est)\n    nb = int(np.searchsorted(cum, BUDGET)) + 1\n    print(f\"passing gates: {len(passing)}/{len(ids)} \"\n          f\"({100*len(passing)/len(ids):.1f}%)\")\n    print(f\"wrote {min(a.top,len(sel))} ids -> {a.out}\")\n    print(f\"~{nb} docs (~{cum[min(nb,len(cum))-1]/1e6:.1f}M est tok) fill 12M budget\")", "numLines": 52, "startLine": 140, "totalLines": 213}}

#34

narration

I'll add deduplication to curate.py. First the constants and the dedup function:

toolEdit

args

{
  "replace_all": false,
  "file_path": "/workspace/submission/curate.py",
  "old_string": "ALPHA = 1.0              # additive smoothing on n-gram counts\nMINCT = 2                # min target count for an n-gram to earn positive weight\nBUDGET = 12_000_000",
  "new_string": "ALPHA = 1.0              # additive smoothing on n-gram counts\nMINCT = 2                # min target count for an n-gram to earn positive weight\nBUDGET = 12_000_000\nEMIT_TOKENS = 20_000_000 # emit deduped ids until this many est tokens (margin over budget)\nK_SKETCH = 16            # MinHash sketch size (smallest-hash unigrams)\nLSH_B, LSH_R = 4, 4      # LSH banding: near-dup if any 4-row band collides (~Jaccard>=0.7)"
}

Edit result

The file /workspace/submission/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/submission/curate.py", "oldString": "ALPHA = 1.0              # additive smoothing on n-gram counts\nMINCT = 2                # min target count for an n-gram to earn positive weight\nBUDGET = 12_000_000", "newString": "ALPHA = 1.0              # additive smoothing on n-gram counts\nMINCT = 2                # min target count for an n-gram to earn positive weight\nBUDGET = 12_000_000\nEMIT_TOKENS = 20_000_000 # emit deduped ids until this many est tokens (margin over budget)\nK_SKETCH = 16            # MinHash sketch size (smallest-hash unigrams)\nLSH_B, LSH_R = 4, 4      # LSH banding: near-dup if any 4-row band collides (~Jaccard>=0.7)", "originalFile": "\"\"\"Curate a raw web pool into a priority-ordered training selection for a small LM.\n\nSTATED CRITERION\n----------------\nThe disclosed evaluation target is a BROAD, multi-domain, high-quality English\ndistribution (encyclopedic / general web prose / news / technical Q&A). We select\nthe pool documents whose token distribution most looks like that target while\nbeing clean, well-formed English prose.\n\nTwo ingredients, combined:\n\n1. TARGET-MATCHING (DSIR-style importance score).  Model the target and the raw\n   pool each as a bag of hashed word n-grams (unigrams + bigrams).  Score every\n   document by its per-token average log-likelihood ratio\n        s(d) = mean_{g in d} w(g),   w(g) = log p_target(g) - log p_pool(g).\n   High s(d) => reads like the high-quality target domain; low s(d) => reads like\n   generic/boilerplate/off-domain web text (Naive-Bayes / DSIR log-odds, Xie 2023).\n\n   RARITY CORRECTION.  The target corpus (~2M n-grams) is ~100x smaller than the\n   pool (~250M).  Naive additive smoothing then gives *any* n-gram unseen in the\n   target (gibberish, foreign text, misspellings, rare jargon) a positive weight,\n   because the smoothing floor of p_target exceeds a genuinely-rare p_pool.  We\n   correct this: an n-gram may earn POSITIVE weight only if it is actually present\n   in the target (count >= MINCT); otherwise its weight is clamped to <= 0 (it may\n   still PENALISE a document when it is common in the pool but absent from target).\n   So a document is rewarded only for n-grams the target genuinely uses.\n\n2. QUALITY GATES.  Cheap Gopher/C4-style filters remove degenerate documents\n   (too short, non-English, symbol/number soup, navigation-menu boilerplate,\n   near-duplicate lines) so nothing ranks purely on n-gram artefacts.\n\nDocuments that fail any gate are dropped; survivors are emitted in DESCENDING\nscore order (best first).  The training packer consumes this order until the\n12M-token budget is full, so the highest-quality, most on-target tokens are used.\n\nWord n-grams use a regex tokenizer over [a-z0-9]+, which also normalises the\ndecoded target's WikiText spacing artifacts (\"@-@\", \" , \") so they align with\nnormal raw-web punctuation.  Deterministic (crc32 hashing); no external labels.\n\"\"\"\nimport argparse, json, re, math, os, time, zlib\nimport numpy as np\nfrom transformers import AutoTokenizer\n\nPOOL = \"/workspace/data/pool.jsonl\"\nDEV  = \"/workspace/data/multi_dev.npy\"\nOUT  = \"/workspace/submission/selection.json\"\nCACHE = \"/tmp/feat_cache.npz\"\n\nD_UNI = 1 << 20\nD_BI  = 1 << 20\nT     = D_UNI + D_BI\nALPHA = 1.0              # additive smoothing on n-gram counts\nMINCT = 2                # min target count for an n-gram to earn positive weight\nBUDGET = 12_000_000\n\nWTOK = re.compile(r\"[a-z0-9]+\")\n\nSTOP = set((\"the be to of and a in that have i it for not on with he as you do at \"\n    \"this but his by from they we say her she or an will my one all would there \"\n    \"their what so up out if about who get which go me when make can like time no \"\n    \"just him know take people into year your good some could them see other than \"\n    \"then now look only come it's over think also back after use two how our work \"\n    \"first well way even new want because any these give day most us is are was \"\n    \"were been has had did does\").split())\n\n_uni_cache = {}\ndef _hw(w):\n    h = _uni_cache.get(w)\n    if h is None:\n        h = zlib.crc32(w.encode()) & (D_UNI - 1)\n        _uni_cache[w] = h\n    return h\n\ndef doc_ngrams(words):\n    n = len(words)\n    if n == 0:\n        return np.empty(0, np.int32)\n    uni = np.fromiter((_hw(w) for w in words), np.int64, n)\n    if n == 1:\n        return uni.astype(np.int32)\n    bi = (((uni[:-1] * 1000003) ^ uni[1:]) & (D_BI - 1)) + D_UNI\n    return np.concatenate([uni, bi]).astype(np.int32)\n\ndef quality(text, words):\n    nw = len(words)\n    nchars = len(text)\n    alpha = sum(c.isalpha() for c in text)\n    frac_alpha = alpha / max(1, nchars)\n    sw_ratio = sum(w in STOP for w in words) / max(1, nw)\n    mwl = sum(len(w) for w in words) / max(1, nw)\n    lines = [ln for ln in text.split(\"\\n\") if ln.strip()]\n    if lines:\n        frac_short = sum(len(ln.split()) < 4 for ln in lines) / len(lines)\n        frac_dup = 1.0 - len(set(lines)) / len(lines)\n    else:\n        frac_short, frac_dup = 1.0, 0.0\n    ok = (50 <= nw <= 50000 and frac_alpha >= 0.60 and sw_ratio >= 0.22\n          and 3.0 <= mwl <= 9.0 and frac_short <= 0.5 and frac_dup <= 0.30)\n    return ok, nchars\n\ndef build_target_counts():\n    tok = AutoTokenizer.from_pretrained(\"gpt2\")\n    d = np.load(DEV)\n    idx = np.where(d == 50256)[0]\n    segs, prev = [], 0\n    for i in idx:\n        segs.append((prev, i)); prev = i + 1\n    segs.append((prev, len(d)))\n    Ct = np.zeros(T, np.float64)\n    for a, b in segs:\n        if b - a < 5:\n            continue\n        words = WTOK.findall(tok.decode(d[a:b]).lower())\n        g = doc_ngrams(words)\n        if len(g):\n            Ct += np.bincount(g, minlength=T)\n    return Ct\n\ndef build_cache():\n    \"\"\"Featurize the whole pool once; cache concatenated n-grams + metadata.\"\"\"\n    t0 = time.time()\n    Ct = build_target_counts()\n    ids, keep, nchars, off = [], [], [], [0]\n    chunks = []\n    with open(POOL) as f:\n        for line in f:\n            r = json.loads(line)\n            t = r[\"text\"]\n            words = WTOK.findall(t.replace(\"<|endoftext|>\", \" \").lower())\n            ok, nc = quality(t, words)\n            g = doc_ngrams(words)\n            ids.append(r[\"id\"]); keep.append(ok); nchars.append(nc)\n            chunks.append(g); off.append(off[-1] + len(g))\n    allfeats = np.concatenate(chunks).astype(np.int32)\n    ids = np.array(ids, np.int64)\n    keep = np.array(keep, bool)\n    nchars = np.array(nchars, np.int64)\n    off = np.array(off, np.int64)\n    Cr = np.bincount(allfeats, minlength=T).astype(np.float64)\n    np.savez(CACHE, allfeats=allfeats, ids=ids, keep=keep, nchars=nchars,\n             off=off, Ct=Ct, Cr=Cr)\n    print(f\"    featurized {len(ids)} docs in {time.time()-t0:.0f}s -> {CACHE}\")\n    return dict(allfeats=allfeats, ids=ids, keep=keep, nchars=nchars,\n                off=off, Ct=Ct, Cr=Cr)\n\ndef load_cache(force=False):\n    if not force and os.path.exists(CACHE):\n        z = np.load(CACHE)\n        return {k: z[k] for k in z.files}\n    return build_cache()\n\ndef score(cache, alpha=ALPHA, minct=MINCT):\n    Ct, Cr = cache[\"Ct\"], cache[\"Cr\"]\n    Nt, Nr = Ct.sum(), Cr.sum()\n    lr = (np.log(Ct + alpha) - math.log(Nt + alpha * T)) \\\n       - (np.log(Cr + alpha) - math.log(Nr + alpha * T))\n    # rarity correction: n-grams not solidly in target may only penalise, not reward\n    weak = Ct < minct\n    lr[weak] = np.minimum(lr[weak], 0.0)\n    off = cache[\"off\"]\n    vals = lr[cache[\"allfeats\"]]\n    counts = np.diff(off)\n    sums = np.add.reduceat(vals, off[:-1])\n    sums[counts == 0] = -1e18\n    s = sums / np.maximum(counts, 1)\n    s[~cache[\"keep\"]] = -1e18\n    return s\n\ndef main():\n    ap = argparse.ArgumentParser()\n    ap.add_argument(\"--out\", default=OUT)\n    ap.add_argument(\"--top\", type=int, default=60000)\n    ap.add_argument(\"--rebuild\", action=\"store_true\")\n    ap.add_argument(\"--diag\", action=\"store_true\")\n    a = ap.parse_args()\n\n    cache = load_cache(force=a.rebuild)\n    s = score(cache)\n    ids = cache[\"ids\"]\n    order = np.argsort(-s)\n    passing = order[s[order] > -1e17]\n    sel = [int(ids[j]) for j in passing]\n    json.dump(sel[: a.top], open(a.out, \"w\"))\n\n    est = cache[\"nchars\"][passing] / 4.0\n    cum = np.cumsum(est)\n    nb = int(np.searchsorted(cum, BUDGET)) + 1\n    print(f\"passing gates: {len(passing)}/{len(ids)} \"\n          f\"({100*len(passing)/len(ids):.1f}%)\")\n    print(f\"wrote {min(a.top,len(sel))} ids -> {a.out}\")\n    print(f\"~{nb} docs (~{cum[min(nb,len(cum))-1]/1e6:.1f}M est tok) fill 12M budget\")\n\n    if a.diag:\n        want = set(int(ids[j]) for j in passing[:10]) | \\\n               set(int(ids[j]) for j in passing[len(passing)//2:len(passing)//2+3])\n        txt = {}\n        with open(POOL) as f:\n            for line in f:\n                r = json.loads(line)\n                if r[\"id\"] in want:\n                    txt[r[\"id\"]] = r[\"text\"]\n        print(\"\\n--- TOP 10 ---\")\n        for j in passing[:10]:\n            i = int(ids[j])\n            print(f\"{s[j]:+.3f} id={i}: {txt[i][:170].strip().replace(chr(10),' | ')}\")\n        print(\"\\n--- MEDIAN-RANK 3 ---\")\n        for j in passing[len(passing)//2:len(passing)//2+3]:\n            i = int(ids[j])\n            print(f\"{s[j]:+.3f} id={i}: {txt[i][:170].strip().replace(chr(10),' | ')}\")\n\nif __name__ == \"__main__\":\n    main()\n", "structuredPatch": [{"oldStart": 52, "oldLines": 6, "newStart": 52, "newLines": 9, "lines": [" ALPHA = 1.0              # additive smoothing on n-gram counts", " MINCT = 2                # min target count for an n-gram to earn positive weight", " BUDGET = 12_000_000", "+EMIT_TOKENS = 20_000_000 # emit deduped ids until this many est tokens (margin over budget)", "+K_SKETCH = 16            # MinHash sketch size (smallest-hash unigrams)", "+LSH_B, LSH_R = 4, 4      # LSH banding: near-dup if any 4-row band collides (~Jaccard>=0.7)", " ", " WTOK = re.compile(r\"[a-z0-9]+\")", " "]}], "userModified": false, "replaceAll": false}

#35

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolEdit

args

{
  "replace_all": false,
  "file_path": "/workspace/submission/curate.py",
  "old_string": "    s = sums / np.maximum(counts, 1)\n    s[~cache[\"keep\"]] = -1e18\n    return s\n\ndef main():",
  "new_string": "    s = sums / np.maximum(counts, 1)\n    s[~cache[\"keep\"]] = -1e18\n    return s\n\ndef dedup_select(cache, order_passing, target_tokens=EMIT_TOKENS, cap=60000):\n    \"\"\"Walk documents in score order, dropping near-duplicates (syndicated /\n    boilerplate-repeated content) via MinHash+LSH over each doc's unigram set, so\n    the budget is spent on DIVERSE on-target tokens.  Deterministic.\"\"\"\n    allf, off, ids = cache[\"allfeats\"], cache[\"off\"], cache[\"ids\"]\n    est = cache[\"nchars\"] / 4.0\n    seen, out, cum, ndup = set(), [], 0.0, 0\n    for j in order_passing:\n        f = allf[off[j]:off[j + 1]]\n        sk = np.unique(f[f < D_UNI])[:K_SKETCH]          # smallest-hash unigrams\n        if len(sk) >= LSH_B * LSH_R:\n            bands = [hash(sk[b*LSH_R:(b+1)*LSH_R].tobytes()) for b in range(LSH_B)]\n            if any(bd in seen for bd in bands):\n                ndup += 1\n                continue\n            seen.update(bands)\n        out.append(int(ids[j])); cum += est[j]\n        if cum >= target_tokens or len(out) >= cap:\n            break\n    return out, ndup\n\ndef main():"
}

Edit result

The file /workspace/submission/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/submission/curate.py", "oldString": "    s = sums / np.maximum(counts, 1)\n    s[~cache[\"keep\"]] = -1e18\n    return s\n\ndef main():", "newString": "    s = sums / np.maximum(counts, 1)\n    s[~cache[\"keep\"]] = -1e18\n    return s\n\ndef dedup_select(cache, order_passing, target_tokens=EMIT_TOKENS, cap=60000):\n    \"\"\"Walk documents in score order, dropping near-duplicates (syndicated /\n    boilerplate-repeated content) via MinHash+LSH over each doc's unigram set, so\n    the budget is spent on DIVERSE on-target tokens.  Deterministic.\"\"\"\n    allf, off, ids = cache[\"allfeats\"], cache[\"off\"], cache[\"ids\"]\n    est = cache[\"nchars\"] / 4.0\n    seen, out, cum, ndup = set(), [], 0.0, 0\n    for j in order_passing:\n        f = allf[off[j]:off[j + 1]]\n        sk = np.unique(f[f < D_UNI])[:K_SKETCH]          # smallest-hash unigrams\n        if len(sk) >= LSH_B * LSH_R:\n            bands = [hash(sk[b*LSH_R:(b+1)*LSH_R].tobytes()) for b in range(LSH_B)]\n            if any(bd in seen for bd in bands):\n                ndup += 1\n                continue\n            seen.update(bands)\n        out.append(int(ids[j])); cum += est[j]\n        if cum >= target_tokens or len(out) >= cap:\n            break\n    return out, ndup\n\ndef main():", "originalFile": "\"\"\"Curate a raw web pool into a priority-ordered training selection for a small LM.\n\nSTATED CRITERION\n----------------\nThe disclosed evaluation target is a BROAD, multi-domain, high-quality English\ndistribution (encyclopedic / general web prose / news / technical Q&A). We select\nthe pool documents whose token distribution most looks like that target while\nbeing clean, well-formed English prose.\n\nTwo ingredients, combined:\n\n1. TARGET-MATCHING (DSIR-style importance score).  Model the target and the raw\n   pool each as a bag of hashed word n-grams (unigrams + bigrams).  Score every\n   document by its per-token average log-likelihood ratio\n        s(d) = mean_{g in d} w(g),   w(g) = log p_target(g) - log p_pool(g).\n   High s(d) => reads like the high-quality target domain; low s(d) => reads like\n   generic/boilerplate/off-domain web text (Naive-Bayes / DSIR log-odds, Xie 2023).\n\n   RARITY CORRECTION.  The target corpus (~2M n-grams) is ~100x smaller than the\n   pool (~250M).  Naive additive smoothing then gives *any* n-gram unseen in the\n   target (gibberish, foreign text, misspellings, rare jargon) a positive weight,\n   because the smoothing floor of p_target exceeds a genuinely-rare p_pool.  We\n   correct this: an n-gram may earn POSITIVE weight only if it is actually present\n   in the target (count >= MINCT); otherwise its weight is clamped to <= 0 (it may\n   still PENALISE a document when it is common in the pool but absent from target).\n   So a document is rewarded only for n-grams the target genuinely uses.\n\n2. QUALITY GATES.  Cheap Gopher/C4-style filters remove degenerate documents\n   (too short, non-English, symbol/number soup, navigation-menu boilerplate,\n   near-duplicate lines) so nothing ranks purely on n-gram artefacts.\n\nDocuments that fail any gate are dropped; survivors are emitted in DESCENDING\nscore order (best first).  The training packer consumes this order until the\n12M-token budget is full, so the highest-quality, most on-target tokens are used.\n\nWord n-grams use a regex tokenizer over [a-z0-9]+, which also normalises the\ndecoded target's WikiText spacing artifacts (\"@-@\", \" , \") so they align with\nnormal raw-web punctuation.  Deterministic (crc32 hashing); no external labels.\n\"\"\"\nimport argparse, json, re, math, os, time, zlib\nimport numpy as np\nfrom transformers import AutoTokenizer\n\nPOOL = \"/workspace/data/pool.jsonl\"\nDEV  = \"/workspace/data/multi_dev.npy\"\nOUT  = \"/workspace/submission/selection.json\"\nCACHE = \"/tmp/feat_cache.npz\"\n\nD_UNI = 1 << 20\nD_BI  = 1 << 20\nT     = D_UNI + D_BI\nALPHA = 1.0              # additive smoothing on n-gram counts\nMINCT = 2                # min target count for an n-gram to earn positive weight\nBUDGET = 12_000_000\nEMIT_TOKENS = 20_000_000 # emit deduped ids until this many est tokens (margin over budget)\nK_SKETCH = 16            # MinHash sketch size (smallest-hash unigrams)\nLSH_B, LSH_R = 4, 4      # LSH banding: near-dup if any 4-row band collides (~Jaccard>=0.7)\n\nWTOK = re.compile(r\"[a-z0-9]+\")\n\nSTOP = set((\"the be to of and a in that have i it for not on with he as you do at \"\n    \"this but his by from they we say her she or an will my one all would there \"\n    \"their what so up out if about who get which go me when make can like time no \"\n    \"just him know take people into year your good some could them see other than \"\n    \"then now look only come it's over think also back after use two how our work \"\n    \"first well way even new want because any these give day most us is are was \"\n    \"were been has had did does\").split())\n\n_uni_cache = {}\ndef _hw(w):\n    h = _uni_cache.get(w)\n    if h is None:\n        h = zlib.crc32(w.encode()) & (D_UNI - 1)\n        _uni_cache[w] = h\n    return h\n\ndef doc_ngrams(words):\n    n = len(words)\n    if n == 0:\n        return np.empty(0, np.int32)\n    uni = np.fromiter((_hw(w) for w in words), np.int64, n)\n    if n == 1:\n        return uni.astype(np.int32)\n    bi = (((uni[:-1] * 1000003) ^ uni[1:]) & (D_BI - 1)) + D_UNI\n    return np.concatenate([uni, bi]).astype(np.int32)\n\ndef quality(text, words):\n    nw = len(words)\n    nchars = len(text)\n    alpha = sum(c.isalpha() for c in text)\n    frac_alpha = alpha / max(1, nchars)\n    sw_ratio = sum(w in STOP for w in words) / max(1, nw)\n    mwl = sum(len(w) for w in words) / max(1, nw)\n    lines = [ln for ln in text.split(\"\\n\") if ln.strip()]\n    if lines:\n        frac_short = sum(len(ln.split()) < 4 for ln in lines) / len(lines)\n        frac_dup = 1.0 - len(set(lines)) / len(lines)\n    else:\n        frac_short, frac_dup = 1.0, 0.0\n    ok = (50 <= nw <= 50000 and frac_alpha >= 0.60 and sw_ratio >= 0.22\n          and 3.0 <= mwl <= 9.0 and frac_short <= 0.5 and frac_dup <= 0.30)\n    return ok, nchars\n\ndef build_target_counts():\n    tok = AutoTokenizer.from_pretrained(\"gpt2\")\n    d = np.load(DEV)\n    idx = np.where(d == 50256)[0]\n    segs, prev = [], 0\n    for i in idx:\n        segs.append((prev, i)); prev = i + 1\n    segs.append((prev, len(d)))\n    Ct = np.zeros(T, np.float64)\n    for a, b in segs:\n        if b - a < 5:\n            continue\n        words = WTOK.findall(tok.decode(d[a:b]).lower())\n        g = doc_ngrams(words)\n        if len(g):\n            Ct += np.bincount(g, minlength=T)\n    return Ct\n\ndef build_cache():\n    \"\"\"Featurize the whole pool once; cache concatenated n-grams + metadata.\"\"\"\n    t0 = time.time()\n    Ct = build_target_counts()\n    ids, keep, nchars, off = [], [], [], [0]\n    chunks = []\n    with open(POOL) as f:\n        for line in f:\n            r = json.loads(line)\n            t = r[\"text\"]\n            words = WTOK.findall(t.replace(\"<|endoftext|>\", \" \").lower())\n            ok, nc = quality(t, words)\n            g = doc_ngrams(words)\n            ids.append(r[\"id\"]); keep.append(ok); nchars.append(nc)\n            chunks.append(g); off.append(off[-1] + len(g))\n    allfeats = np.concatenate(chunks).astype(np.int32)\n    ids = np.array(ids, np.int64)\n    keep = np.array(keep, bool)\n    nchars = np.array(nchars, np.int64)\n    off = np.array(off, np.int64)\n    Cr = np.bincount(allfeats, minlength=T).astype(np.float64)\n    np.savez(CACHE, allfeats=allfeats, ids=ids, keep=keep, nchars=nchars,\n             off=off, Ct=Ct, Cr=Cr)\n    print(f\"    featurized {len(ids)} docs in {time.time()-t0:.0f}s -> {CACHE}\")\n    return dict(allfeats=allfeats, ids=ids, keep=keep, nchars=nchars,\n                off=off, Ct=Ct, Cr=Cr)\n\ndef load_cache(force=False):\n    if not force and os.path.exists(CACHE):\n        z = np.load(CACHE)\n        return {k: z[k] for k in z.files}\n    return build_cache()\n\ndef score(cache, alpha=ALPHA, minct=MINCT):\n    Ct, Cr = cache[\"Ct\"], cache[\"Cr\"]\n    Nt, Nr = Ct.sum(), Cr.sum()\n    lr = (np.log(Ct + alpha) - math.log(Nt + alpha * T)) \\\n       - (np.log(Cr + alpha) - math.log(Nr + alpha * T))\n    # rarity correction: n-grams not solidly in target may only penalise, not reward\n    weak = Ct < minct\n    lr[weak] = np.minimum(lr[weak], 0.0)\n    off = cache[\"off\"]\n    vals = lr[cache[\"allfeats\"]]\n    counts = np.diff(off)\n    sums = np.add.reduceat(vals, off[:-1])\n    sums[counts == 0] = -1e18\n    s = sums / np.maximum(counts, 1)\n    s[~cache[\"keep\"]] = -1e18\n    return s\n\ndef main():\n    ap = argparse.ArgumentParser()\n    ap.add_argument(\"--out\", default=OUT)\n    ap.add_argument(\"--top\", type=int, default=60000)\n    ap.add_argument(\"--rebuild\", action=\"store_true\")\n    ap.add_argument(\"--diag\", action=\"store_true\")\n    a = ap.parse_args()\n\n    cache = load_cache(force=a.rebuild)\n    s = score(cache)\n    ids = cache[\"ids\"]\n    order = np.argsort(-s)\n    passing = order[s[order] > -1e17]\n    sel = [int(ids[j]) for j in passing]\n    json.dump(sel[: a.top], open(a.out, \"w\"))\n\n    est = cache[\"nchars\"][passing] / 4.0\n    cum = np.cumsum(est)\n    nb = int(np.searchsorted(cum, BUDGET)) + 1\n    print(f\"passing gates: {len(passing)}/{len(ids)} \"\n          f\"({100*len(passing)/len(ids):.1f}%)\")\n    print(f\"wrote {min(a.top,len(sel))} ids -> {a.out}\")\n    print(f\"~{nb} docs (~{cum[min(nb,len(cum))-1]/1e6:.1f}M est tok) fill 12M budget\")\n\n    if a.diag:\n        want = set(int(ids[j]) for j in passing[:10]) | \\\n               set(int(ids[j]) for j in passing[len(passing)//2:len(passing)//2+3])\n        txt = {}\n        with open(POOL) as f:\n            for line in f:\n                r = json.loads(line)\n                if r[\"id\"] in want:\n                    txt[r[\"id\"]] = r[\"text\"]\n        print(\"\\n--- TOP 10 ---\")\n        for j in passing[:10]:\n            i = int(ids[j])\n            print(f\"{s[j]:+.3f} id={i}: {txt[i][:170].strip().replace(chr(10),' | ')}\")\n        print(\"\\n--- MEDIAN-RANK 3 ---\")\n        for j in passing[len(passing)//2:len(passing)//2+3]:\n            i = int(ids[j])\n            print(f\"{s[j]:+.3f} id={i}: {txt[i][:170].strip().replace(chr(10),' | ')}\")\n\nif __name__ == \"__main__\":\n    main()\n", "structuredPatch": [{"oldStart": 169, "oldLines": 6, "newStart": 169, "newLines": 27, "lines": ["     s[~cache[\"keep\"]] = -1e18", "     return s", " ", "+def dedup_select(cache, order_passing, target_tokens=EMIT_TOKENS, cap=60000):", "+    \"\"\"Walk documents in score order, dropping near-duplicates (syndicated /", "+    boilerplate-repeated content) via MinHash+LSH over each doc's unigram set, so", "+    the budget is spent on DIVERSE on-target tokens.  Deterministic.\"\"\"", "+    allf, off, ids = cache[\"allfeats\"], cache[\"off\"], cache[\"ids\"]", "+    est = cache[\"nchars\"] / 4.0", "+    seen, out, cum, ndup = set(), [], 0.0, 0", "+    for j in order_passing:", "+        f = allf[off[j]:off[j + 1]]", "+        sk = np.unique(f[f < D_UNI])[:K_SKETCH]          # smallest-hash unigrams", "+        if len(sk) >= LSH_B * LSH_R:", "+            bands = [hash(sk[b*LSH_R:(b+1)*LSH_R].tobytes()) for b in range(LSH_B)]", "+            if any(bd in seen for bd in bands):", "+                ndup += 1", "+                continue", "+            seen.update(bands)", "+        out.append(int(ids[j])); cum += est[j]", "+        if cum >= target_tokens or len(out) >= cap:", "+            break", "+    return out, ndup", "+", " def main():", "     ap = argparse.ArgumentParser()", "     ap.add_argument(\"--out\", default=OUT)"]}], "userModified": false, "replaceAll": false}

#36

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolEdit

args

{
  "replace_all": false,
  "file_path": "/workspace/submission/curate.py",
  "old_string": "    order = np.argsort(-s)\n    passing = order[s[order] > -1e17]\n    sel = [int(ids[j]) for j in passing]\n    json.dump(sel[: a.top], open(a.out, \"w\"))\n\n    est = cache[\"nchars\"][passing] / 4.0\n    cum = np.cumsum(est)\n    nb = int(np.searchsorted(cum, BUDGET)) + 1\n    print(f\"passing gates: {len(passing)}/{len(ids)} \"\n          f\"({100*len(passing)/len(ids):.1f}%)\")\n    print(f\"wrote {min(a.top,len(sel))} ids -> {a.out}\")\n    print(f\"~{nb} docs (~{cum[min(nb,len(cum))-1]/1e6:.1f}M est tok) fill 12M budget\")",
  "new_string": "    order = np.argsort(-s)\n    passing = order[s[order] > -1e17]\n    sel, ndup = dedup_select(cache, passing, cap=a.top)\n    json.dump(sel, open(a.out, \"w\"))\n\n    est_sel = np.cumsum([cache[\"nchars\"][j] / 4.0 for j in passing[:len(sel)+ndup]])\n    print(f\"passing gates: {len(passing)}/{len(ids)} \"\n          f\"({100*len(passing)/len(ids):.1f}%)\")\n    print(f\"dropped {ndup} near-duplicates; wrote {len(sel)} ids -> {a.out}\")\n    print(f\"selection covers ~{est_sel[-1]/1e6:.1f}M est tokens (budget {BUDGET/1e6:.0f}M)\")"
}

Edit result

The file /workspace/submission/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/submission/curate.py", "oldString": "    order = np.argsort(-s)\n    passing = order[s[order] > -1e17]\n    sel = [int(ids[j]) for j in passing]\n    json.dump(sel[: a.top], open(a.out, \"w\"))\n\n    est = cache[\"nchars\"][passing] / 4.0\n    cum = np.cumsum(est)\n    nb = int(np.searchsorted(cum, BUDGET)) + 1\n    print(f\"passing gates: {len(passing)}/{len(ids)} \"\n          f\"({100*len(passing)/len(ids):.1f}%)\")\n    print(f\"wrote {min(a.top,len(sel))} ids -> {a.out}\")\n    print(f\"~{nb} docs (~{cum[min(nb,len(cum))-1]/1e6:.1f}M est tok) fill 12M budget\")", "newString": "    order = np.argsort(-s)\n    passing = order[s[order] > -1e17]\n    sel, ndup = dedup_select(cache, passing, cap=a.top)\n    json.dump(sel, open(a.out, \"w\"))\n\n    est_sel = np.cumsum([cache[\"nchars\"][j] / 4.0 for j in passing[:len(sel)+ndup]])\n    print(f\"passing gates: {len(passing)}/{len(ids)} \"\n          f\"({100*len(passing)/len(ids):.1f}%)\")\n    print(f\"dropped {ndup} near-duplicates; wrote {len(sel)} ids -> {a.out}\")\n    print(f\"selection covers ~{est_sel[-1]/1e6:.1f}M est tokens (budget {BUDGET/1e6:.0f}M)\")", "originalFile": "\"\"\"Curate a raw web pool into a priority-ordered training selection for a small LM.\n\nSTATED CRITERION\n----------------\nThe disclosed evaluation target is a BROAD, multi-domain, high-quality English\ndistribution (encyclopedic / general web prose / news / technical Q&A). We select\nthe pool documents whose token distribution most looks like that target while\nbeing clean, well-formed English prose.\n\nTwo ingredients, combined:\n\n1. TARGET-MATCHING (DSIR-style importance score).  Model the target and the raw\n   pool each as a bag of hashed word n-grams (unigrams + bigrams).  Score every\n   document by its per-token average log-likelihood ratio\n        s(d) = mean_{g in d} w(g),   w(g) = log p_target(g) - log p_pool(g).\n   High s(d) => reads like the high-quality target domain; low s(d) => reads like\n   generic/boilerplate/off-domain web text (Naive-Bayes / DSIR log-odds, Xie 2023).\n\n   RARITY CORRECTION.  The target corpus (~2M n-grams) is ~100x smaller than the\n   pool (~250M).  Naive additive smoothing then gives *any* n-gram unseen in the\n   target (gibberish, foreign text, misspellings, rare jargon) a positive weight,\n   because the smoothing floor of p_target exceeds a genuinely-rare p_pool.  We\n   correct this: an n-gram may earn POSITIVE weight only if it is actually present\n   in the target (count >= MINCT); otherwise its weight is clamped to <= 0 (it may\n   still PENALISE a document when it is common in the pool but absent from target).\n   So a document is rewarded only for n-grams the target genuinely uses.\n\n2. QUALITY GATES.  Cheap Gopher/C4-style filters remove degenerate documents\n   (too short, non-English, symbol/number soup, navigation-menu boilerplate,\n   near-duplicate lines) so nothing ranks purely on n-gram artefacts.\n\nDocuments that fail any gate are dropped; survivors are emitted in DESCENDING\nscore order (best first).  The training packer consumes this order until the\n12M-token budget is full, so the highest-quality, most on-target tokens are used.\n\nWord n-grams use a regex tokenizer over [a-z0-9]+, which also normalises the\ndecoded target's WikiText spacing artifacts (\"@-@\", \" , \") so they align with\nnormal raw-web punctuation.  Deterministic (crc32 hashing); no external labels.\n\"\"\"\nimport argparse, json, re, math, os, time, zlib\nimport numpy as np\nfrom transformers import AutoTokenizer\n\nPOOL = \"/workspace/data/pool.jsonl\"\nDEV  = \"/workspace/data/multi_dev.npy\"\nOUT  = \"/workspace/submission/selection.json\"\nCACHE = \"/tmp/feat_cache.npz\"\n\nD_UNI = 1 << 20\nD_BI  = 1 << 20\nT     = D_UNI + D_BI\nALPHA = 1.0              # additive smoothing on n-gram counts\nMINCT = 2                # min target count for an n-gram to earn positive weight\nBUDGET = 12_000_000\nEMIT_TOKENS = 20_000_000 # emit deduped ids until this many est tokens (margin over budget)\nK_SKETCH = 16            # MinHash sketch size (smallest-hash unigrams)\nLSH_B, LSH_R = 4, 4      # LSH banding: near-dup if any 4-row band collides (~Jaccard>=0.7)\n\nWTOK = re.compile(r\"[a-z0-9]+\")\n\nSTOP = set((\"the be to of and a in that have i it for not on with he as you do at \"\n    \"this but his by from they we say her she or an will my one all would there \"\n    \"their what so up out if about who get which go me when make can like time no \"\n    \"just him know take people into year your good some could them see other than \"\n    \"then now look only come it's over think also back after use two how our work \"\n    \"first well way even new want because any these give day most us is are was \"\n    \"were been has had did does\").split())\n\n_uni_cache = {}\ndef _hw(w):\n    h = _uni_cache.get(w)\n    if h is None:\n        h = zlib.crc32(w.encode()) & (D_UNI - 1)\n        _uni_cache[w] = h\n    return h\n\ndef doc_ngrams(words):\n    n = len(words)\n    if n == 0:\n        return np.empty(0, np.int32)\n    uni = np.fromiter((_hw(w) for w in words), np.int64, n)\n    if n == 1:\n        return uni.astype(np.int32)\n    bi = (((uni[:-1] * 1000003) ^ uni[1:]) & (D_BI - 1)) + D_UNI\n    return np.concatenate([uni, bi]).astype(np.int32)\n\ndef quality(text, words):\n    nw = len(words)\n    nchars = len(text)\n    alpha = sum(c.isalpha() for c in text)\n    frac_alpha = alpha / max(1, nchars)\n    sw_ratio = sum(w in STOP for w in words) / max(1, nw)\n    mwl = sum(len(w) for w in words) / max(1, nw)\n    lines = [ln for ln in text.split(\"\\n\") if ln.strip()]\n    if lines:\n        frac_short = sum(len(ln.split()) < 4 for ln in lines) / len(lines)\n        frac_dup = 1.0 - len(set(lines)) / len(lines)\n    else:\n        frac_short, frac_dup = 1.0, 0.0\n    ok = (50 <= nw <= 50000 and frac_alpha >= 0.60 and sw_ratio >= 0.22\n          and 3.0 <= mwl <= 9.0 and frac_short <= 0.5 and frac_dup <= 0.30)\n    return ok, nchars\n\ndef build_target_counts():\n    tok = AutoTokenizer.from_pretrained(\"gpt2\")\n    d = np.load(DEV)\n    idx = np.where(d == 50256)[0]\n    segs, prev = [], 0\n    for i in idx:\n        segs.append((prev, i)); prev = i + 1\n    segs.append((prev, len(d)))\n    Ct = np.zeros(T, np.float64)\n    for a, b in segs:\n        if b - a < 5:\n            continue\n        words = WTOK.findall(tok.decode(d[a:b]).lower())\n        g = doc_ngrams(words)\n        if len(g):\n            Ct += np.bincount(g, minlength=T)\n    return Ct\n\ndef build_cache():\n    \"\"\"Featurize the whole pool once; cache concatenated n-grams + metadata.\"\"\"\n    t0 = time.time()\n    Ct = build_target_counts()\n    ids, keep, nchars, off = [], [], [], [0]\n    chunks = []\n    with open(POOL) as f:\n        for line in f:\n            r = json.loads(line)\n            t = r[\"text\"]\n            words = WTOK.findall(t.replace(\"<|endoftext|>\", \" \").lower())\n            ok, nc = quality(t, words)\n            g = doc_ngrams(words)\n            ids.append(r[\"id\"]); keep.append(ok); nchars.append(nc)\n            chunks.append(g); off.append(off[-1] + len(g))\n    allfeats = np.concatenate(chunks).astype(np.int32)\n    ids = np.array(ids, np.int64)\n    keep = np.array(keep, bool)\n    nchars = np.array(nchars, np.int64)\n    off = np.array(off, np.int64)\n    Cr = np.bincount(allfeats, minlength=T).astype(np.float64)\n    np.savez(CACHE, allfeats=allfeats, ids=ids, keep=keep, nchars=nchars,\n             off=off, Ct=Ct, Cr=Cr)\n    print(f\"    featurized {len(ids)} docs in {time.time()-t0:.0f}s -> {CACHE}\")\n    return dict(allfeats=allfeats, ids=ids, keep=keep, nchars=nchars,\n                off=off, Ct=Ct, Cr=Cr)\n\ndef load_cache(force=False):\n    if not force and os.path.exists(CACHE):\n        z = np.load(CACHE)\n        return {k: z[k] for k in z.files}\n    return build_cache()\n\ndef score(cache, alpha=ALPHA, minct=MINCT):\n    Ct, Cr = cache[\"Ct\"], cache[\"Cr\"]\n    Nt, Nr = Ct.sum(), Cr.sum()\n    lr = (np.log(Ct + alpha) - math.log(Nt + alpha * T)) \\\n       - (np.log(Cr + alpha) - math.log(Nr + alpha * T))\n    # rarity correction: n-grams not solidly in target may only penalise, not reward\n    weak = Ct < minct\n    lr[weak] = np.minimum(lr[weak], 0.0)\n    off = cache[\"off\"]\n    vals = lr[cache[\"allfeats\"]]\n    counts = np.diff(off)\n    sums = np.add.reduceat(vals, off[:-1])\n    sums[counts == 0] = -1e18\n    s = sums / np.maximum(counts, 1)\n    s[~cache[\"keep\"]] = -1e18\n    return s\n\ndef dedup_select(cache, order_passing, target_tokens=EMIT_TOKENS, cap=60000):\n    \"\"\"Walk documents in score order, dropping near-duplicates (syndicated /\n    boilerplate-repeated content) via MinHash+LSH over each doc's unigram set, so\n    the budget is spent on DIVERSE on-target tokens.  Deterministic.\"\"\"\n    allf, off, ids = cache[\"allfeats\"], cache[\"off\"], cache[\"ids\"]\n    est = cache[\"nchars\"] / 4.0\n    seen, out, cum, ndup = set(), [], 0.0, 0\n    for j in order_passing:\n        f = allf[off[j]:off[j + 1]]\n        sk = np.unique(f[f < D_UNI])[:K_SKETCH]          # smallest-hash unigrams\n        if len(sk) >= LSH_B * LSH_R:\n            bands = [hash(sk[b*LSH_R:(b+1)*LSH_R].tobytes()) for b in range(LSH_B)]\n            if any(bd in seen for bd in bands):\n                ndup += 1\n                continue\n            seen.update(bands)\n        out.append(int(ids[j])); cum += est[j]\n        if cum >= target_tokens or len(out) >= cap:\n            break\n    return out, ndup\n\ndef main():\n    ap = argparse.ArgumentParser()\n    ap.add_argument(\"--out\", default=OUT)\n    ap.add_argument(\"--top\", type=int, default=60000)\n    ap.add_argument(\"--rebuild\", action=\"store_true\")\n    ap.add_argument(\"--diag\", action=\"store_true\")\n    a = ap.parse_args()\n\n    cache = load_cache(force=a.rebuild)\n    s = score(cache)\n    ids = cache[\"ids\"]\n    order = np.argsort(-s)\n    passing = order[s[order] > -1e17]\n    sel = [int(ids[j]) for j in passing]\n    json.dump(sel[: a.top], open(a.out, \"w\"))\n\n    est = cache[\"nchars\"][passing] / 4.0\n    cum = np.cumsum(est)\n    nb = int(np.searchsorted(cum, BUDGET)) + 1\n    print(f\"passing gates: {len(passing)}/{len(ids)} \"\n          f\"({100*len(passing)/len(ids):.1f}%)\")\n    print(f\"wrote {min(a.top,len(sel))} ids -> {a.out}\")\n    print(f\"~{nb} docs (~{cum[min(nb,len(cum))-1]/1e6:.1f}M est tok) fill 12M budget\")\n\n    if a.diag:\n        want = set(int(ids[j]) for j in passing[:10]) | \\\n               set(int(ids[j]) for j in passing[len(passing)//2:len(passing)//2+3])\n        txt = {}\n        with open(POOL) as f:\n            for line in f:\n                r = json.loads(line)\n                if r[\"id\"] in want:\n                    txt[r[\"id\"]] = r[\"text\"]\n        print(\"\\n--- TOP 10 ---\")\n        for j in passing[:10]:\n            i = int(ids[j])\n            print(f\"{s[j]:+.3f} id={i}: {txt[i][:170].strip().replace(chr(10),' | ')}\")\n        print(\"\\n--- MEDIAN-RANK 3 ---\")\n        for j in passing[len(passing)//2:len(passing)//2+3]:\n            i = int(ids[j])\n            print(f\"{s[j]:+.3f} id={i}: {txt[i][:170].strip().replace(chr(10),' | ')}\")\n\nif __name__ == \"__main__\":\n    main()\n", "structuredPatch": [{"oldStart": 203, "oldLines": 16, "newStart": 203, "newLines": 14, "lines": ["     ids = cache[\"ids\"]", "     order = np.argsort(-s)", "     passing = order[s[order] > -1e17]", "-    sel = [int(ids[j]) for j in passing]", "-    json.dump(sel[: a.top], open(a.out, \"w\"))", "+    sel, ndup = dedup_select(cache, passing, cap=a.top)", "+    json.dump(sel, open(a.out, \"w\"))", " ", "-    est = cache[\"nchars\"][passing] / 4.0", "-    cum = np.cumsum(est)", "-    nb = int(np.searchsorted(cum, BUDGET)) + 1", "+    est_sel = np.cumsum([cache[\"nchars\"][j] / 4.0 for j in passing[:len(sel)+ndup]])", "     print(f\"passing gates: {len(passing)}/{len(ids)} \"", "           f\"({100*len(passing)/len(ids):.1f}%)\")", "-    print(f\"wrote {min(a.top,len(sel))} ids -> {a.out}\")", "-    print(f\"~{nb} docs (~{cum[min(nb,len(cum))-1]/1e6:.1f}M est tok) fill 12M budget\")", "+    print(f\"dropped {ndup} near-duplicates; wrote {len(sel)} ids -> {a.out}\")", "+    print(f\"selection covers ~{est_sel[-1]/1e6:.1f}M est tokens (budget {BUDGET/1e6:.0f}M)\")", " ", "     if a.diag:", "         want = set(int(ids[j]) for j in passing[:10]) | \\"]}], "userModified": false, "replaceAll": false}

#37

narration

Let me update the docstring to document the dedup step as part of the stated criterion:

toolEdit

args

{
  "replace_all": false,
  "file_path": "/workspace/submission/curate.py",
  "old_string": "2. QUALITY GATES.  Cheap Gopher/C4-style filters remove degenerate documents\n   (too short, non-English, symbol/number soup, navigation-menu boilerplate,\n   near-duplicate lines) so nothing ranks purely on n-gram artefacts.\n\nDocuments that fail any gate are dropped; survivors are emitted in DESCENDING\nscore order (best first).  The training packer consumes this order until the\n12M-token budget is full, so the highest-quality, most on-target tokens are used.",
  "new_string": "2. QUALITY GATES.  Cheap Gopher/C4-style filters remove degenerate documents\n   (too short, non-English, symbol/number soup, navigation-menu boilerplate,\n   near-duplicate lines) so nothing ranks purely on n-gram artefacts.\n\n3. NEAR-DUP DEDUP.  Walking the ranked survivors best-first, we drop documents\n   whose unigram set collides (MinHash + LSH) with an already-selected one, so\n   the fixed budget is spent on DIVERSE on-target tokens instead of many copies\n   of the same syndicated article / boilerplate page.\n\nDocuments that fail any gate are dropped; survivors are emitted DEDUPED in\nDESCENDING score order (best first).  The training packer consumes this order\nuntil the 12M-token budget is full, so the highest-quality, most on-target,\nnon-redundant tokens are used."
}

Edit result

The file /workspace/submission/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/submission/curate.py", "oldString": "2. QUALITY GATES.  Cheap Gopher/C4-style filters remove degenerate documents\n   (too short, non-English, symbol/number soup, navigation-menu boilerplate,\n   near-duplicate lines) so nothing ranks purely on n-gram artefacts.\n\nDocuments that fail any gate are dropped; survivors are emitted in DESCENDING\nscore order (best first).  The training packer consumes this order until the\n12M-token budget is full, so the highest-quality, most on-target tokens are used.", "newString": "2. QUALITY GATES.  Cheap Gopher/C4-style filters remove degenerate documents\n   (too short, non-English, symbol/number soup, navigation-menu boilerplate,\n   near-duplicate lines) so nothing ranks purely on n-gram artefacts.\n\n3. NEAR-DUP DEDUP.  Walking the ranked survivors best-first, we drop documents\n   whose unigram set collides (MinHash + LSH) with an already-selected one, so\n   the fixed budget is spent on DIVERSE on-target tokens instead of many copies\n   of the same syndicated article / boilerplate page.\n\nDocuments that fail any gate are dropped; survivors are emitted DEDUPED in\nDESCENDING score order (best first).  The training packer consumes this order\nuntil the 12M-token budget is full, so the highest-quality, most on-target,\nnon-redundant tokens are used.", "originalFile": "\"\"\"Curate a raw web pool into a priority-ordered training selection for a small LM.\n\nSTATED CRITERION\n----------------\nThe disclosed evaluation target is a BROAD, multi-domain, high-quality English\ndistribution (encyclopedic / general web prose / news / technical Q&A). We select\nthe pool documents whose token distribution most looks like that target while\nbeing clean, well-formed English prose.\n\nTwo ingredients, combined:\n\n1. TARGET-MATCHING (DSIR-style importance score).  Model the target and the raw\n   pool each as a bag of hashed word n-grams (unigrams + bigrams).  Score every\n   document by its per-token average log-likelihood ratio\n        s(d) = mean_{g in d} w(g),   w(g) = log p_target(g) - log p_pool(g).\n   High s(d) => reads like the high-quality target domain; low s(d) => reads like\n   generic/boilerplate/off-domain web text (Naive-Bayes / DSIR log-odds, Xie 2023).\n\n   RARITY CORRECTION.  The target corpus (~2M n-grams) is ~100x smaller than the\n   pool (~250M).  Naive additive smoothing then gives *any* n-gram unseen in the\n   target (gibberish, foreign text, misspellings, rare jargon) a positive weight,\n   because the smoothing floor of p_target exceeds a genuinely-rare p_pool.  We\n   correct this: an n-gram may earn POSITIVE weight only if it is actually present\n   in the target (count >= MINCT); otherwise its weight is clamped to <= 0 (it may\n   still PENALISE a document when it is common in the pool but absent from target).\n   So a document is rewarded only for n-grams the target genuinely uses.\n\n2. QUALITY GATES.  Cheap Gopher/C4-style filters remove degenerate documents\n   (too short, non-English, symbol/number soup, navigation-menu boilerplate,\n   near-duplicate lines) so nothing ranks purely on n-gram artefacts.\n\nDocuments that fail any gate are dropped; survivors are emitted in DESCENDING\nscore order (best first).  The training packer consumes this order until the\n12M-token budget is full, so the highest-quality, most on-target tokens are used.\n\nWord n-grams use a regex tokenizer over [a-z0-9]+, which also normalises the\ndecoded target's WikiText spacing artifacts (\"@-@\", \" , \") so they align with\nnormal raw-web punctuation.  Deterministic (crc32 hashing); no external labels.\n\"\"\"\nimport argparse, json, re, math, os, time, zlib\nimport numpy as np\nfrom transformers import AutoTokenizer\n\nPOOL = \"/workspace/data/pool.jsonl\"\nDEV  = \"/workspace/data/multi_dev.npy\"\nOUT  = \"/workspace/submission/selection.json\"\nCACHE = \"/tmp/feat_cache.npz\"\n\nD_UNI = 1 << 20\nD_BI  = 1 << 20\nT     = D_UNI + D_BI\nALPHA = 1.0              # additive smoothing on n-gram counts\nMINCT = 2                # min target count for an n-gram to earn positive weight\nBUDGET = 12_000_000\nEMIT_TOKENS = 20_000_000 # emit deduped ids until this many est tokens (margin over budget)\nK_SKETCH = 16            # MinHash sketch size (smallest-hash unigrams)\nLSH_B, LSH_R = 4, 4      # LSH banding: near-dup if any 4-row band collides (~Jaccard>=0.7)\n\nWTOK = re.compile(r\"[a-z0-9]+\")\n\nSTOP = set((\"the be to of and a in that have i it for not on with he as you do at \"\n    \"this but his by from they we say her she or an will my one all would there \"\n    \"their what so up out if about who get which go me when make can like time no \"\n    \"just him know take people into year your good some could them see other than \"\n    \"then now look only come it's over think also back after use two how our work \"\n    \"first well way even new want because any these give day most us is are was \"\n    \"were been has had did does\").split())\n\n_uni_cache = {}\ndef _hw(w):\n    h = _uni_cache.get(w)\n    if h is None:\n        h = zlib.crc32(w.encode()) & (D_UNI - 1)\n        _uni_cache[w] = h\n    return h\n\ndef doc_ngrams(words):\n    n = len(words)\n    if n == 0:\n        return np.empty(0, np.int32)\n    uni = np.fromiter((_hw(w) for w in words), np.int64, n)\n    if n == 1:\n        return uni.astype(np.int32)\n    bi = (((uni[:-1] * 1000003) ^ uni[1:]) & (D_BI - 1)) + D_UNI\n    return np.concatenate([uni, bi]).astype(np.int32)\n\ndef quality(text, words):\n    nw = len(words)\n    nchars = len(text)\n    alpha = sum(c.isalpha() for c in text)\n    frac_alpha = alpha / max(1, nchars)\n    sw_ratio = sum(w in STOP for w in words) / max(1, nw)\n    mwl = sum(len(w) for w in words) / max(1, nw)\n    lines = [ln for ln in text.split(\"\\n\") if ln.strip()]\n    if lines:\n        frac_short = sum(len(ln.split()) < 4 for ln in lines) / len(lines)\n        frac_dup = 1.0 - len(set(lines)) / len(lines)\n    else:\n        frac_short, frac_dup = 1.0, 0.0\n    ok = (50 <= nw <= 50000 and frac_alpha >= 0.60 and sw_ratio >= 0.22\n          and 3.0 <= mwl <= 9.0 and frac_short <= 0.5 and frac_dup <= 0.30)\n    return ok, nchars\n\ndef build_target_counts():\n    tok = AutoTokenizer.from_pretrained(\"gpt2\")\n    d = np.load(DEV)\n    idx = np.where(d == 50256)[0]\n    segs, prev = [], 0\n    for i in idx:\n        segs.append((prev, i)); prev = i + 1\n    segs.append((prev, len(d)))\n    Ct = np.zeros(T, np.float64)\n    for a, b in segs:\n        if b - a < 5:\n            continue\n        words = WTOK.findall(tok.decode(d[a:b]).lower())\n        g = doc_ngrams(words)\n        if len(g):\n            Ct += np.bincount(g, minlength=T)\n    return Ct\n\ndef build_cache():\n    \"\"\"Featurize the whole pool once; cache concatenated n-grams + metadata.\"\"\"\n    t0 = time.time()\n    Ct = build_target_counts()\n    ids, keep, nchars, off = [], [], [], [0]\n    chunks = []\n    with open(POOL) as f:\n        for line in f:\n            r = json.loads(line)\n            t = r[\"text\"]\n            words = WTOK.findall(t.replace(\"<|endoftext|>\", \" \").lower())\n            ok, nc = quality(t, words)\n            g = doc_ngrams(words)\n            ids.append(r[\"id\"]); keep.append(ok); nchars.append(nc)\n            chunks.append(g); off.append(off[-1] + len(g))\n    allfeats = np.concatenate(chunks).astype(np.int32)\n    ids = np.array(ids, np.int64)\n    keep = np.array(keep, bool)\n    nchars = np.array(nchars, np.int64)\n    off = np.array(off, np.int64)\n    Cr = np.bincount(allfeats, minlength=T).astype(np.float64)\n    np.savez(CACHE, allfeats=allfeats, ids=ids, keep=keep, nchars=nchars,\n             off=off, Ct=Ct, Cr=Cr)\n    print(f\"    featurized {len(ids)} docs in {time.time()-t0:.0f}s -> {CACHE}\")\n    return dict(allfeats=allfeats, ids=ids, keep=keep, nchars=nchars,\n                off=off, Ct=Ct, Cr=Cr)\n\ndef load_cache(force=False):\n    if not force and os.path.exists(CACHE):\n        z = np.load(CACHE)\n        return {k: z[k] for k in z.files}\n    return build_cache()\n\ndef score(cache, alpha=ALPHA, minct=MINCT):\n    Ct, Cr = cache[\"Ct\"], cache[\"Cr\"]\n    Nt, Nr = Ct.sum(), Cr.sum()\n    lr = (np.log(Ct + alpha) - math.log(Nt + alpha * T)) \\\n       - (np.log(Cr + alpha) - math.log(Nr + alpha * T))\n    # rarity correction: n-grams not solidly in target may only penalise, not reward\n    weak = Ct < minct\n    lr[weak] = np.minimum(lr[weak], 0.0)\n    off = cache[\"off\"]\n    vals = lr[cache[\"allfeats\"]]\n    counts = np.diff(off)\n    sums = np.add.reduceat(vals, off[:-1])\n    sums[counts == 0] = -1e18\n    s = sums / np.maximum(counts, 1)\n    s[~cache[\"keep\"]] = -1e18\n    return s\n\ndef dedup_select(cache, order_passing, target_tokens=EMIT_TOKENS, cap=60000):\n    \"\"\"Walk documents in score order, dropping near-duplicates (syndicated /\n    boilerplate-repeated content) via MinHash+LSH over each doc's unigram set, so\n    the budget is spent on DIVERSE on-target tokens.  Deterministic.\"\"\"\n    allf, off, ids = cache[\"allfeats\"], cache[\"off\"], cache[\"ids\"]\n    est = cache[\"nchars\"] / 4.0\n    seen, out, cum, ndup = set(), [], 0.0, 0\n    for j in order_passing:\n        f = allf[off[j]:off[j + 1]]\n        sk = np.unique(f[f < D_UNI])[:K_SKETCH]          # smallest-hash unigrams\n        if len(sk) >= LSH_B * LSH_R:\n            bands = [hash(sk[b*LSH_R:(b+1)*LSH_R].tobytes()) for b in range(LSH_B)]\n            if any(bd in seen for bd in bands):\n                ndup += 1\n                continue\n            seen.update(bands)\n        out.append(int(ids[j])); cum += est[j]\n        if cum >= target_tokens or len(out) >= cap:\n            break\n    return out, ndup\n\ndef main():\n    ap = argparse.ArgumentParser()\n    ap.add_argument(\"--out\", default=OUT)\n    ap.add_argument(\"--top\", type=int, default=60000)\n    ap.add_argument(\"--rebuild\", action=\"store_true\")\n    ap.add_argument(\"--diag\", action=\"store_true\")\n    a = ap.parse_args()\n\n    cache = load_cache(force=a.rebuild)\n    s = score(cache)\n    ids = cache[\"ids\"]\n    order = np.argsort(-s)\n    passing = order[s[order] > -1e17]\n    sel, ndup = dedup_select(cache, passing, cap=a.top)\n    json.dump(sel, open(a.out, \"w\"))\n\n    est_sel = np.cumsum([cache[\"nchars\"][j] / 4.0 for j in passing[:len(sel)+ndup]])\n    print(f\"passing gates: {len(passing)}/{len(ids)} \"\n          f\"({100*len(passing)/len(ids):.1f}%)\")\n    print(f\"dropped {ndup} near-duplicates; wrote {len(sel)} ids -> {a.out}\")\n    print(f\"selection covers ~{est_sel[-1]/1e6:.1f}M est tokens (budget {BUDGET/1e6:.0f}M)\")\n\n    if a.diag:\n        want = set(int(ids[j]) for j in passing[:10]) | \\\n               set(int(ids[j]) for j in passing[len(passing)//2:len(passing)//2+3])\n        txt = {}\n        with open(POOL) as f:\n            for line in f:\n                r = json.loads(line)\n                if r[\"id\"] in want:\n                    txt[r[\"id\"]] = r[\"text\"]\n        print(\"\\n--- TOP 10 ---\")\n        for j in passing[:10]:\n            i = int(ids[j])\n            print(f\"{s[j]:+.3f} id={i}: {txt[i][:170].strip().replace(chr(10),' | ')}\")\n        print(\"\\n--- MEDIAN-RANK 3 ---\")\n        for j in passing[len(passing)//2:len(passing)//2+3]:\n            i = int(ids[j])\n            print(f\"{s[j]:+.3f} id={i}: {txt[i][:170].strip().replace(chr(10),' | ')}\")\n\nif __name__ == \"__main__\":\n    main()\n", "structuredPatch": [{"oldStart": 29, "oldLines": 10, "newStart": 29, "newLines": 16, "lines": ["    (too short, non-English, symbol/number soup, navigation-menu boilerplate,", "    near-duplicate lines) so nothing ranks purely on n-gram artefacts.", " ", "-Documents that fail any gate are dropped; survivors are emitted in DESCENDING", "-score order (best first).  The training packer consumes this order until the", "-12M-token budget is full, so the highest-quality, most on-target tokens are used.", "+3. NEAR-DUP DEDUP.  Walking the ranked survivors best-first, we drop documents", "+   whose unigram set collides (MinHash + LSH) with an already-selected one, so", "+   the fixed budget is spent on DIVERSE on-target tokens instead of many copies", "+   of the same syndicated article / boilerplate page.", " ", "+Documents that fail any gate are dropped; survivors are emitted DEDUPED in", "+DESCENDING score order (best first).  The training packer consumes this order", "+until the 12M-token budget is full, so the highest-quality, most on-target,", "+non-redundant tokens are used.", "+", " Word n-grams use a regex tokenizer over [a-z0-9]+, which also normalises the", " decoded target's WikiText spacing artifacts (\"@-@\", \" , \") so they align with", " normal raw-web punctuation.  Deterministic (crc32 hashing); no external labels."]}], "userModified": false, "replaceAll": false}

#38

narration

Now let me run the finalized curate.py (uses existing cache, no rebuild) to regenerate selection.json, and verify it matches the V2 selection I evaluated:

toolBashRegenerate selection.json and verify match with V2

args

{
  "command": "cd /workspace; export TOKENIZERS_PARALLELISM=false; python3 submission/curate.py 2>&1 | grep -v -i warning; echo \"=== verify matches V2 eval selection (prefix that fills 12M) ===\"; python3 -c \"\nimport json\na=json.load(open('/workspace/submission/selection.json'))\nb=json.load(open('/tmp/sel_v2_dedup.json'))\nprint('curate.py selection ids:',len(a))\nprint('prev V2 ids:',len(b))\n# compare prefix\nn=min(len(a),len(b))\nsame=sum(1 for i in range(n) if a[i]==b[i])\nprint('prefix identical for first',n,'ids:',same==n,'(matching:',same,')')\nprint('first 5 ids:',a[:5])\n\"",
  "description": "Regenerate selection.json and verify match with V2"
}

Bash result

passing gates: 120480/182016 (66.2%)
dropped 1061 near-duplicates; wrote 20896 ids -> /workspace/submission/selection.json
selection covers ~22.3M est tokens (budget 12M)
=== verify matches V2 eval selection (prefix that fills 12M) ===
curate.py selection ids: 20896
prev V2 ids: 16769
prefix identical for first 16769 ids: True (matching: 16769 )
first 5 ids: [58452, 85430, 73993, 43758, 63278]

[stdout]
passing gates: 120480/182016 (66.2%)
dropped 1061 near-duplicates; wrote 20896 ids -> /workspace/submission/selection.json
selection covers ~22.3M est tokens (budget 12M)
=== verify matches V2 eval selection (prefix that fills 12M) ===
curate.py selection ids: 20896
prev V2 ids: 16769
prefix identical for first 16769 ids: True (matching: 16769 )
first 5 ids: [58452, 85430, 73993, 43758, 63278]
[metadata] {"noOutputExpected": false}

#39

narration

curate.py reproduces selection.json (20,896 ids, ~22M est tokens) and its first 16,769 ids are byte-identical to the evaluated V2 selection — so the deliverable reproduces the 354.96 result deterministically. Let me check remaining time to decide whether to attempt further gains:

toolBashCheck remaining time

args

{
  "command": "cat /workspace/.timer/remaining_secs; echo \"seconds remaining\"",
  "description": "Check remaining time"
}

Bash result

3595
seconds remaining

[stdout]
3595
seconds remaining
[metadata] {"noOutputExpected": false}

#40

narration

~60 min left — comfortable. The deliverable is locked at 355. I noticed the very top of the ranking skews toward one narrow topic (Indian politics), which could over-concentrate the training distribution. Let me quantify topic concentration in the budget-filling prefix before deciding whether to invest in topic-diversification:

toolBashMeasure topic concentration in budget prefix

args

{
  "command": "cd /workspace; python3 -c \"\nimport json, re\nsel=json.load(open('/workspace/submission/selection.json'))\n# budget-filling prefix ~ first 12437 by earlier est; use first 13000\nprefix=set(sel[:13000])\ntxt={}\nwith open('/workspace/data/pool.jsonl') as f:\n    for line in f:\n        r=json.loads(line)\n        if r['id'] in prefix: txt[r['id']]=r['text']\nindia=re.compile(r'\\b(Modi|BJP|Congress|Delhi|Mumbai|India|Indian|Pakistan|Bengal|Kerala|Rahul|Lok Sabha|crore|lakh|IANS|ANI|PTI)\\b')\nn_india=sum(1 for t in txt.values() if len(india.findall(t))>=3)\nn_any=sum(1 for t in txt.values() if india.search(t))\nprint('budget-prefix docs sampled:',len(txt))\nprint('docs with >=3 India/SouthAsia terms: %d (%.1f%%)'%(n_india,100*n_india/len(txt)))\nprint('docs with >=1 such term: %d (%.1f%%)'%(n_any,100*n_any/len(txt)))\n# generic news vs other\nsaid=sum(1 for t in txt.values() if t.lower().count(' said')>=3)\nprint('docs with >=3 \\\"said\\\" (news-like): %d (%.1f%%)'%(said,100*said/len(txt)))\n\"",
  "description": "Measure topic concentration in budget prefix"
}

Bash result

budget-prefix docs sampled: 13000
docs with >=3 India/SouthAsia terms: 766 (5.9%)
docs with >=1 such term: 1798 (13.8%)
docs with >=3 "said" (news-like): 2815 (21.7%)

[stdout]
budget-prefix docs sampled: 13000
docs with >=3 India/SouthAsia terms: 766 (5.9%)
docs with >=1 such term: 1798 (13.8%)
docs with >=3 "said" (news-like): 2815 (21.7%)
[metadata] {"noOutputExpected": false}

#41

narration

Topic concentration is low (Indian news only 6–14%, general news ~22%) — the selection is well-diversified across topics and registers, so aggressive diversification would likely remove legitimate variety. The evidence says V2 is well-balanced.

I'll make the disciplined call: certify the actual deliverable selection.json end-to-end (confirms file integrity with the full 20,896-id list), and write claim.md concurrently. Launching the certification eval:

toolBashCertify deliverable selection.json end-to-end

args

{
  "command": "cd /workspace; export TOKENIZERS_PARALLELISM=false; nohup bash dev_eval.sh /workspace/submission/selection.json > /tmp/eval_final.log 2>&1 & echo \"started final certification eval pid $!\"",
  "description": "Certify deliverable selection.json end-to-end"
}

Bash result

started final certification eval pid 2821

[stdout]
started final certification eval pid 2821
[metadata] {"noOutputExpected": false}

#42

narration

While it certifies, let me compute the training-free distributional-match statistic (mean per-token target/pool log-ratio) that my mechanism predicts should track perplexity — I'll cite real numbers in claim.md:

toolBashCompute distributional-match statistic for random vs curated

args

{
  "command": "cd /workspace; python3 -c \"\nimport numpy as np, math, json\nz=np.load('/tmp/feat_cache.npz')\nCt,Cr=z['Ct'],z['Cr']; T=len(Ct); A=1.0\nlr=(np.log(Ct+A)-math.log(Ct.sum()+A*T))-(np.log(Cr+A)-math.log(Cr.sum()+A*T))\nweak=Ct<2; lr[weak]=np.minimum(lr[weak],0.0)\noff=z['off']; allf=z['allfeats']; ids=z['ids']; est=z['nchars']/4.0\nid2idx={int(v):i for i,v in enumerate(ids)}\ndef stat(selfile, budget=12_000_000):\n    sel=json.load(open(selfile)); tot=0; ss=0.0; nn=0\n    for i in sel:\n        j=id2idx[i]; a,b=off[j],off[j+1]\n        ss+=lr[allf[a:b]].sum(); nn+=(b-a); tot+=est[j]\n        if tot>=budget: break\n    return ss/max(nn,1)\ndef stat_random(budget=12_000_000):\n    sel=json.load(open('/tmp/random_sel.json')); tot=0; ss=0.0; nn=0\n    for i in sel:\n        j=id2idx[i]; a,b=off[j],off[j+1]\n        ss+=lr[allf[a:b]].sum(); nn+=(b-a); tot+=est[j]\n        if tot>=budget: break\n    return ss/max(nn,1)\nprint('mean per-token target/pool log-ratio of the 12M training tokens:')\nprint('  random baseline : %.4f   (heldout ppl 457.2)'%stat_random())\nprint('  curated (V1)    : %.4f   (heldout ppl 362.9)'%stat('/tmp/random_sel.json' if False else '/workspace/submission/selection.json'))\n\"",
  "description": "Compute distributional-match statistic for random vs curated"
}

Bash result

mean per-token target/pool log-ratio of the 12M training tokens:
  random baseline : -0.7989   (heldout ppl 457.2)
  curated (V1)    : -0.4978   (heldout ppl 362.9)

[stdout]
mean per-token target/pool log-ratio of the 12M training tokens:
  random baseline : -0.7989   (heldout ppl 457.2)
  curated (V1)    : -0.4978   (heldout ppl 362.9)
[metadata] {"noOutputExpected": false}

#43

narration

The statistic separates cleanly and tracks perplexity: curated tokens score −0.498 vs random −0.799 (monotone with ppl 355 vs 457). Let me check the certification result, then I'll run one mechanism-probe (quality-gates-only, no target-matching) to decompose the gain for claim.md:

toolBashCheck certification eval result

args

{
  "command": "cd /workspace; sleep 20; echo \"=== certification (deliverable selection.json) ===\"; grep HELDOUT_PPL /tmp/eval_final.log 2>/dev/null || echo \"still training...\"; grep packed /tmp/eval_final.log 2>/dev/null; cat /workspace/.timer/remaining_secs",
  "description": "Check certification eval result"
}

Bash result

=== certification (deliverable selection.json) ===
still training...
packed 12000000 tokens from 20896 ids -> /tmp/dev_train.npy
3395

[stdout]
=== certification (deliverable selection.json) ===
still training...
packed 12000000 tokens from 20896 ids -> /tmp/dev_train.npy
3395
[metadata] {"noOutputExpected": false}

#44

narration

Packing works (12M from the 20,896-id deliverable). While it trains, let me build the quality-gates-only probe (gate-passing docs in random order — no target-matching, no dedup) to isolate how much of the gain comes from junk-removal vs. target-matching:

toolBashBuild gates-only mechanism probe selection

args

{
  "command": "cd /workspace; python3 -c \"\nimport numpy as np, json\nz=np.load('/tmp/feat_cache.npz')\nids=z['ids']; keep=z['keep']\npassing_ids=ids[keep]\nrng=np.random.default_rng(7)\nperm=rng.permutation(len(passing_ids))\nsel=[int(passing_ids[i]) for i in perm[:40000]]\njson.dump(sel, open('/tmp/sel_gatesonly.json','w'))\nprint('gates-only (quality filter, random order):',len(sel),'ids from',keep.sum(),'passing')\n\"",
  "description": "Build gates-only mechanism probe selection"
}

Bash result

gates-only (quality filter, random order): 40000 ids from 120480 passing

[stdout]
gates-only (quality filter, random order): 40000 ids from 120480 passing
[metadata] {"noOutputExpected": false}

#45

narration

Let me check the certification result and then launch the gates-only probe:

toolBashCheck certification result

args

{
  "command": "cd /workspace; sleep 30; echo \"=== certification result ===\"; grep HELDOUT_PPL /tmp/eval_final.log 2>/dev/null || echo \"still training...\"",
  "description": "Check certification result"
}

Bash result

=== certification result ===
HELDOUT_PPL 354.963  loss 5.8720  train_s 56  /tmp/dev_train.npy

[stdout]
=== certification result ===
HELDOUT_PPL 354.963  loss 5.8720  train_s 56  /tmp/dev_train.npy
[metadata] {"noOutputExpected": false}

#46

narration

Deliverable certified: `selection.json` → 354.963 ppl, exactly matching V2. Now the mechanism probe (gates-only):

toolBashLaunch gates-only mechanism probe

args

{
  "command": "cd /workspace; export TOKENIZERS_PARALLELISM=false; nohup bash dev_eval.sh /tmp/sel_gatesonly.json > /tmp/eval_gates.log 2>&1 & echo \"started gates-only probe pid $!\"",
  "description": "Launch gates-only mechanism probe"
}

Bash result

started gates-only probe pid 3077

[stdout]
started gates-only probe pid 3077
[metadata] {"noOutputExpected": false}

#47

narration

While the probe trains, let me update tasks and wait for its result to finalize claim.md:

toolTaskUpdate

args

{
  "taskId": "2",
  "status": "completed"
}

TaskUpdate result

Updated task #2 status

[metadata] {"success": true, "taskId": "2", "updatedFields": ["status"], "statusChange": {"from": "in_progress", "to": "completed"}}

#48

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolTaskUpdate

args

{
  "taskId": "3",
  "status": "in_progress"
}

TaskUpdate result

Updated task #3 status

[metadata] {"success": true, "taskId": "3", "updatedFields": ["status"], "statusChange": {"from": "pending", "to": "in_progress"}}

#49

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolBashCheck gates-only probe result

args

{
  "command": "cd /workspace; sleep 55; echo \"=== gates-only probe result ===\"; grep HELDOUT_PPL /tmp/eval_gates.log 2>/dev/null || echo \"still training...\"; echo \"=== timer ===\"; cat /workspace/.timer/remaining_secs",
  "description": "Check gates-only probe result"
}

Bash result

=== gates-only probe result ===
still training...
=== timer ===
3245

[stdout]
=== gates-only probe result ===
still training...
=== timer ===
3245
[metadata] {"noOutputExpected": false}

#50

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolBashCheck gates-only probe result again

args

{
  "command": "cd /workspace; sleep 35; grep HELDOUT_PPL /tmp/eval_gates.log 2>/dev/null || echo \"still training...\"",
  "description": "Check gates-only probe result again"
}

Bash result

HELDOUT_PPL 439.908  loss 6.0866  train_s 56  /tmp/dev_train.npy

[stdout]
HELDOUT_PPL 439.908  loss 6.0866  train_s 56  /tmp/dev_train.npy
[metadata] {"noOutputExpected": false}