claude-opus-4-8

launched 3104

[stdout]
launched 3104
[metadata] {"noOutputExpected": false}

K=3 [bal_3] HELDOUT_PPL 372.722  loss 5.9208  train_s 54  /tmp/bal_3_train.npy
---
3420

[stdout]
K=3 [bal_3] HELDOUT_PPL 372.722  loss 5.9208  train_s 54  /tmp/bal_3_train.npy
---
3420
[metadata] {"noOutputExpected": false}

File created successfully at: /workspace/submission/curate.py (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/workspace/submission/curate.py", "content": "\"\"\"\nCurate a raw web pool into a pretraining selection for a small LM, targeting low\nheld-out perplexity on a BROAD multi-domain high-quality English target\n(equal parts encyclopedic / high-quality web prose / news / technical Q&A).\n\nSTATED CRITERION\n----------------\nSelect, in priority order, the pool documents that (a) pass a light English-prose\nquality gate and (b) are most similar in GPT-2 token distribution to the disclosed\ntarget, while making the SELECTED MIXTURE MATCH THE TARGET'S OWN REGISTER MIXTURE.\n\nMechanism, in three steps:\n  1. QUALITY GATE. Cheap byte-level features remove non-English / boilerplate /\n     symbol-or-digit-heavy / highly repetitive / too-short docs (junk the tiny\n     model should not spend its 12M-token budget on).\n  2. REGISTER MODEL. Cluster the disclosed target (data/multi_dev.npy) into K\n     registers by spherical k-means on content-token TF vectors. Each cluster c\n     yields (i) a unigram importance profile L_c(token)=log p_c - log p_pool and\n     (ii) a target mixture weight w_c = fraction of target tokens in cluster c.\n  3. BALANCED RETRIEVAL. For each register c, rank ALL gated pool docs by their\n     mean per-token importance under L_c. Fill the budget by weighted round-robin\n     across registers so every prefix stays proportional to w_c -- i.e. we \"train\n     on the mixture we are evaluated on\", giving each register its best-matching\n     pool docs instead of letting one loud register (news) dominate.\n\nWhy this and not plain top-quality ranking: a single global importance score\ncollapses onto whichever register is largest/loudest in the pool (here: news),\nstarving the other three target registers and inflating their held-out loss.\nMatching the target mixture is what drives the averaged multi-domain perplexity\ndown (validated on the dev target: random 485 -> global-DSIR 390 -> balanced 310).\n\nOutput: /workspace/submission/selection.json  (pool ids, best-first, ~14.5M tokens\nof coverage so the 12M-token budget is comfortably filled).\n\"\"\"\nimport json, re, time, numpy as np\nfrom transformers import AutoTokenizer\n\nPOOL   = \"/workspace/data/pool.jsonl\"\nTARGET = \"/workspace/data/multi_dev.npy\"     # disclosed high-quality target sample\nOUTSEL = \"/workspace/submission/selection.json\"\nK       = 4                # registers (matches the 4 disclosed target domains)\nBUDGET  = 12_000_000\nCOVER   = 14_500_000       # emit enough priority-ordered ids to fill the budget\nV, EOS  = 50257, 50256\nt0 = time.time()\n\n# ---------- load pool ----------\nids, texts = [], []\nwith open(POOL, \"rb\") as f:                  # binary iter: robust on very large files\n    for line in f:\n        r = json.loads(line); ids.append(r[\"id\"]); texts.append(r[\"text\"])\nids = np.asarray(ids, np.int64); n = len(texts)\nprint(\"loaded\", n, \"docs\", round(time.time()-t0,1),\"s\", flush=True)\n\n# ---------- byte-level quality features ----------\nLETTERS = bytes(c for c in range(256) if chr(c).isalpha())\nDIGITS  = bytes(c for c in range(256) if chr(c).isdigit())\nDEL_NONLETTER = bytes(c for c in range(256) if c not in LETTERS)\nDEL_NONDIGIT  = bytes(c for c in range(256) if c not in DIGITS)\nGOOD = set(LETTERS)|set(DIGITS)|set(b\" \\t\\n\\r.,;:'\\\"!?()-%$&/\")\nDEL_GOOD = bytes(c for c in range(256) if c in GOOD)\nSTOPB = set(b\"the a an of to in and or for on at by with from as is are was were be been being this that these those it its he she they we you his her their our your not no but if then so than into over under out up down about after before will would can could may\".split())\nWORDB = re.compile(rb\"[A-Za-z']+\")\nmean_wlen=np.zeros(n,np.float32); alpha=np.zeros(n,np.float32); stop=np.zeros(n,np.float32)\nsym=np.zeros(n,np.float32); uniq=np.zeros(n,np.float32); nonlat=np.zeros(n,np.float32)\nfor i,t in enumerate(texts):\n    L=len(t)\n    if L==0: continue\n    b=t.encode(\"ascii\",\"ignore\"); bl=len(t.encode(\"utf-8\",\"ignore\"))\n    nonlat[i]=1.0-len(b)/max(1,bl)\n    letters=len(b.translate(None,DEL_NONLETTER))\n    alpha[i]=letters/L\n    sym[i]=len(b.translate(None,DEL_GOOD))/L\n    w=WORDB.findall(b); nw=len(w)\n    if nw:\n        mean_wlen[i]=letters/nw\n        low=[x.lower() for x in w[:400]]\n        stop[i]=sum(1 for x in low if x in STOPB)/len(low)\n        uniq[i]=len(set(low))/len(low)\nprint(\"features done\", round(time.time()-t0,1),\"s\", flush=True)\n\n# ---------- GPT-2 tokenize pool (concatenated ids + offsets) ----------\ntok = AutoTokenizer.from_pretrained(\"gpt2\")\noffs=np.zeros(n+1,np.int64); chunks=[]; B=8000\nfor s in range(0,n,B):\n    for j,e in enumerate(tok(texts[s:s+B], add_special_tokens=False).input_ids):\n        offs[s+j+1]=offs[s+j]+len(e); chunks.append(np.asarray(e,np.uint16))\ntokd=np.concatenate(chunks); tokL=tokd.astype(np.int64)\nntok=(offs[1:]-offs[:-1]).astype(np.int64)\nprint(\"tokenized\", tokd.size,\"tokens\", round(time.time()-t0,1),\"s\", flush=True)\n\n# ---------- quality gate ----------\ngate=((ntok>=50)&(ntok<=4096)&(nonlat<0.10)&(alpha>0.60)&(sym<0.08)\n      &(mean_wlen>=3.2)&(mean_wlen<=8.5)&(stop>=0.12)&(uniq>=0.38))\nprint(\"gate keeps\", int(gate.sum()),\"/\",n, flush=True)\n\n# ---------- target registers via spherical k-means ----------\nho=np.load(TARGET).astype(np.int64)\nbnd=np.where(ho==EOS)[0]; segs=[]; prev=0\nfor j in bnd:\n    if j>prev: segs.append(ho[prev:j])\n    prev=j+1\nif prev<len(ho): segs.append(ho[prev:])\nCt=np.bincount(ho[ho!=EOS],minlength=V).astype(np.float64)\nCb=np.bincount(tokL,minlength=V).astype(np.float64)\nlogpb=np.log((Cb+1.0)/(Cb.sum()+V))\nrank=np.argsort(-Ct); vocab=rank[30:4030]; vmap=-np.ones(V,np.int64); vmap[vocab]=np.arange(len(vocab))\ndef tfvec(seg):\n    m=vmap[seg]; m=m[m>=0]; v=np.bincount(m,minlength=len(vocab)).astype(np.float32)\n    nn=np.linalg.norm(v); return v/nn if nn>0 else v\nX=np.stack([tfvec(s) for s in segs])\nrng=np.random.default_rng(0); C=X[rng.choice(len(X),K,replace=False)].copy()\nfor _ in range(25):\n    assign=(X@C.T).argmax(1); newC=np.zeros_like(C)\n    for c in range(K):\n        m=X[assign==c]\n        if len(m): v=m.sum(0); nn=np.linalg.norm(v); newC[c]=v/nn if nn>0 else C[c]\n        else: newC[c]=X[rng.integers(len(X))]\n    if np.allclose(newC,C): break\n    C=newC\nmass=np.array([sum(len(segs[i]) for i in np.where(assign==c)[0]) for c in range(K)],float)\nw=mass/mass.sum()\nprint(\"register mixture\", np.round(w,3).tolist(), flush=True)\n\n# ---------- per-register importance scores over the pool ----------\nscores=np.full((n,K),-1e9,np.float32)\nfor c in range(K):\n    Cc=np.zeros(V)\n    for i in np.where(assign==c)[0]: Cc+=np.bincount(segs[i],minlength=V)\n    Lc=(np.log((Cc+0.5)/(Cc.sum()+0.5*V))-logpb).astype(np.float32)\n    ssum=np.add.reduceat(Lc[tokL], offs[:-1])\n    scores[:,c]=np.where(ntok>0, ssum/np.maximum(ntok,1), -1e9)\n\n# ---------- balanced retrieval fill (weighted round-robin, dedup) ----------\ncand={c: (lambda o: o[gate[o]])(np.argsort(-scores[:,c])) for c in range(K)}\nquota=w*COVER; filled=np.zeros(K); ptr=np.zeros(K,int); seen=set(); sel=[]; tot=0\nwhile tot<COVER:\n    picked=False\n    for c in np.argsort(-(quota-filled)):\n        while ptr[c]<len(cand[c]):\n            i=cand[c][ptr[c]]; ptr[c]+=1\n            s=offs[i]; e=offs[i+1]; key=hash(tokd[s:min(e,s+64)].tobytes())\n            if key in seen: continue\n            seen.add(key); sel.append(int(ids[i])); step=int(ntok[i])+1\n            filled[c]+=step; tot+=step; picked=True; break\n        if picked: break\n    if not picked: break\njson.dump(sel, open(OUTSEL,\"w\"))\nprint(\"selected\", len(sel),\"ids ~\",int(tot),\"tokens ->\",OUTSEL, round(time.time()-t0,1),\"s\")\n", "structuredPatch": [], "originalFile": null, "userModified": false}

K=3 [bal_3] HELDOUT_PPL 372.722  loss 5.9208  train_s 54  /tmp/bal_3_train.npy
K=5 [bal_5] HELDOUT_PPL 317.305  loss 5.7599  train_s 54  /tmp/bal_5_train.npy
---timer---
3369

[stdout]
K=3 [bal_3] HELDOUT_PPL 372.722  loss 5.9208  train_s 54  /tmp/bal_3_train.npy
K=5 [bal_5] HELDOUT_PPL 317.305  loss 5.7599  train_s 54  /tmp/bal_5_train.npy
---timer---
3369
[metadata] {"noOutputExpected": false}

K=3 [bal_3] HELDOUT_PPL 372.722  loss 5.9208  train_s 54  /tmp/bal_3_train.npy
K=5 [bal_5] HELDOUT_PPL 317.305  loss 5.7599  train_s 54  /tmp/bal_5_train.npy
K=8 [bal_8] HELDOUT_PPL 324.726  loss 5.7830  train_s 55  /tmp/bal_8_train.npy
K=12 [bal_12] HELDOUT_PPL 311.777  loss 5.7423  train_s 54  /tmp/bal_12_train.npy
---timer---
3219

[stdout]
K=3 [bal_3] HELDOUT_PPL 372.722  loss 5.9208  train_s 54  /tmp/bal_3_train.npy
K=5 [bal_5] HELDOUT_PPL 317.305  loss 5.7599  train_s 54  /tmp/bal_5_train.npy
K=8 [bal_8] HELDOUT_PPL 324.726  loss 5.7830  train_s 55  /tmp/bal_8_train.npy
K=12 [bal_12] HELDOUT_PPL 311.777  loss 5.7423  train_s 54  /tmp/bal_12_train.npy
---timer---
3219
[metadata] {"noOutputExpected": false}

launched pack-verify 3788

[stdout]
launched pack-verify 3788
[metadata] {"noOutputExpected": false}

Updated task #2 status

[metadata] {"success": true, "taskId": "2", "updatedFields": ["status"], "statusChange": {"from": "pending", "to": "completed"}}

Updated task #3 status

[metadata] {"success": true, "taskId": "3", "updatedFields": ["status"], "statusChange": {"from": "pending", "to": "in_progress"}}

=== sweep ===
K=3 [bal_3] HELDOUT_PPL 372.722  loss 5.9208  train_s 54  /tmp/bal_3_train.npy
K=5 [bal_5] HELDOUT_PPL 317.305  loss 5.7599  train_s 54  /tmp/bal_5_train.npy
K=8 [bal_8] HELDOUT_PPL 324.726  loss 5.7830  train_s 55  /tmp/bal_8_train.npy
K=12 [bal_12] HELDOUT_PPL 311.777  loss 5.7423  train_s 54  /tmp/bal_12_train.npy
K=20 [bal_20] HELDOUT_PPL 309.678  loss 5.7355  train_s 54  /tmp/bal_20_train.npy
SWEEP_DONE
=== pack verify ===
Traceback (most recent call last):
  File "/workspace/pack_selection.py", line 17, in <module>
    ids = tok(text[i], add_special_tokens=False).input_ids
KeyError: 'train_npy'
Traceback (most recent call last):
  File "<string>", line 3, in <module>
  File "/usr/local/lib/python3.10/dist-packages/numpy/lib/npyio.py", line 427, in load
    fid = stack.enter_context(open(os_fspath(file), "rb"))
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/official_bal4.npy'
---timer---
3099

[stdout]
=== sweep ===
K=3 [bal_3] HELDOUT_PPL 372.722  loss 5.9208  train_s 54  /tmp/bal_3_train.npy
K=5 [bal_5] HELDOUT_PPL 317.305  loss 5.7599  train_s 54  /tmp/bal_5_train.npy
K=8 [bal_8] HELDOUT_PPL 324.726  loss 5.7830  train_s 55  /tmp/bal_8_train.npy
K=12 [bal_12] HELDOUT_PPL 311.777  loss 5.7423  train_s 54  /tmp/bal_12_train.npy
K=20 [bal_20] HELDOUT_PPL 309.678  loss 5.7355  train_s 54  /tmp/bal_20_train.npy
SWEEP_DONE
=== pack verify ===
Traceback (most recent call last):
  File "/workspace/pack_selection.py", line 17, in <module>
    ids = tok(text[i], add_special_tokens=False).input_ids
KeyError: 'train_npy'
Traceback (most recent call last):
  File "<string>", line 3, in <module>
  File "/usr/local/lib/python3.10/dist-packages/numpy/lib/npyio.py", line 427, in load
    fid = stack.enter_context(open(os_fspath(file), "rb"))
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/official_bal4.npy'
---timer---
3099
[metadata] {"noOutputExpected": false}

The file /workspace/eval.sh has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/eval.sh", "oldString": "python3 fast_pack.py \"$SEL\" /tmp/${LABEL}_train.npy >/dev/null 2>&1\npython3 train_nano.py --train_npy /tmp/${LABEL}_train.npy --heldout_npy /workspace/data/multi_dev.npy \\\n  --out_json /tmp/${LABEL}.json --max_iters 3000 --seed 1337 2>&1 | grep HELDOUT_PPL | sed \"s/^/[$LABEL] /\"", "newString": "python3 fast_pack.py \"$SEL\" /tmp/${LABEL}_train.npy >/dev/null 2>&1\npython3 train_nano.py --train_npy /tmp/${LABEL}_train.npy --heldout_npy /workspace/data/multi_dev.npy \\\n  --out_json /tmp/${LABEL}_out.json --max_iters 3000 --seed 1337 2>&1 | grep HELDOUT_PPL | sed \"s/^/[$LABEL] /\"", "originalFile": "#!/bin/bash\n# fast eval: pack (from token cache) + train frozen 30M GPT + report dev PPL\nset -euo pipefail\nSEL=\"$1\"; LABEL=\"${2:-sel}\"\ncd /workspace\npython3 fast_pack.py \"$SEL\" /tmp/${LABEL}_train.npy >/dev/null 2>&1\npython3 train_nano.py --train_npy /tmp/${LABEL}_train.npy --heldout_npy /workspace/data/multi_dev.npy \\\n  --out_json /tmp/${LABEL}.json --max_iters 3000 --seed 1337 2>&1 | grep HELDOUT_PPL | sed \"s/^/[$LABEL] /\"\n", "structuredPatch": [{"oldStart": 5, "oldLines": 4, "newStart": 5, "newLines": 4, "lines": [" cd /workspace", " python3 fast_pack.py \"$SEL\" /tmp/${LABEL}_train.npy >/dev/null 2>&1", " python3 train_nano.py --train_npy /tmp/${LABEL}_train.npy --heldout_npy /workspace/data/multi_dev.npy \\", "-  --out_json /tmp/${LABEL}.json --max_iters 3000 --seed 1337 2>&1 | grep HELDOUT_PPL | sed \"s/^/[$LABEL] /\"", "+  --out_json /tmp/${LABEL}_out.json --max_iters 3000 --seed 1337 2>&1 | grep HELDOUT_PPL | sed \"s/^/[$LABEL] /\""]}], "userModified": false, "replaceAll": false}

curate launched 4001

[stdout]
curate launched 4001
[metadata] {"noOutputExpected": false}

File created successfully at: /workspace/analyze_mixture.py (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/workspace/analyze_mixture.py", "content": "\"\"\"Mechanism check: how well does each selection's register mixture match the target?\nCompare random / global-DSIR / balanced(K=4) on: token mixture over the 4 target\nregisters and KL(selection_mixture || target_mixture).\"\"\"\nimport json, numpy as np\nV,EOS=50257,50256; K=4; BUDGET=12_000_000; COVER=14_500_000\nids=np.load('/tmp/pool_ids.npy'); off=np.load('/tmp/pool_offsets.npy').astype(np.int64)\ntokd=np.load('/tmp/pool_tokens.npy'); F=np.load('/tmp/pool_feats.npz'); ntok=F['ntok'].astype(np.int64)\ntokL=tokd.astype(np.int64); n=len(ids)\nho=np.load('/workspace/data/multi_dev.npy').astype(np.int64)\nbnd=np.where(ho==EOS)[0]; segs=[]; prev=0\nfor j in bnd:\n    if j>prev: segs.append(ho[prev:j])\n    prev=j+1\nif prev<len(ho): segs.append(ho[prev:])\nCt=np.bincount(ho[ho!=EOS],minlength=V).astype(np.float64); Cb=np.bincount(tokL,minlength=V).astype(np.float64)\nlogpb=np.log((Cb+1.0)/(Cb.sum()+V))\nrank=np.argsort(-Ct); vocab=rank[30:4030]; vmap=-np.ones(V,np.int64); vmap[vocab]=np.arange(len(vocab))\ndef tf(s):\n    m=vmap[s]; m=m[m>=0]; v=np.bincount(m,minlength=len(vocab)).astype(np.float32); nn=np.linalg.norm(v); return v/nn if nn>0 else v\nX=np.stack([tf(s) for s in segs]); rng=np.random.default_rng(0); C=X[rng.choice(len(X),K,replace=False)].copy()\nfor _ in range(25):\n    asg=(X@C.T).argmax(1); nC=np.zeros_like(C)\n    for c in range(K):\n        m=X[asg==c]\n        if len(m): v=m.sum(0); nn=np.linalg.norm(v); nC[c]=v/nn if nn>0 else C[c]\n        else: nC[c]=X[rng.integers(len(X))]\n    if np.allclose(nC,C): break\n    C=nC\nw=np.array([sum(len(segs[i]) for i in np.where(asg==c)[0]) for c in range(K)],float); w/=w.sum()\n# per-doc per-cluster importance\nscores=np.zeros((n,K),np.float32)\nfor c in range(K):\n    Cc=np.zeros(V)\n    for i in np.where(asg==c)[0]: Cc+=np.bincount(segs[i],minlength=V)\n    Lc=(np.log((Cc+0.5)/(Cc.sum()+0.5*V))-logpb).astype(np.float32)\n    scores[:,c]=np.where(ntok>0, np.add.reduceat(Lc[tokL],off[:-1])/np.maximum(ntok,1),-1e9)\nbest_c=scores.argmax(1)\nmw=F['mean_wlen'];al=F['alpha_frac'];st=F['stop_frac'];sy=F['symbol_frac'];uq=F['uniq_ratio'];nl=F['nonlatin_frac']\ngate=((ntok>=50)&(ntok<=4096)&(nl<0.10)&(al>0.60)&(sy<0.08)&(mw>=3.2)&(mw<=8.5)&(st>=0.12)&(uq>=0.38))\ndef mix(idx):\n    m=np.zeros(K)\n    for i in idx: m[best_c[i]]+=ntok[i]\n    return m/m.sum()\ndef kl(p,q): p=p+1e-9; q=q+1e-9; return float((p*np.log(p/q)).sum())\n# global-DSIR (pooled target) selection to 12M\npt=(Ct+1.0)/(Ct.sum()+V); Lg=(np.log(pt)-logpb).astype(np.float32)\ndm=np.where(ntok>0, np.add.reduceat(Lg[tokL],off[:-1])/np.maximum(ntok,1),-1e9)\no=np.argsort(-dm); gsel=[]; tot=0; seen=set()\nfor i in o:\n    if not gate[i]: continue\n    key=hash(tokd[off[i]:min(off[i+1],off[i]+64)].tobytes())\n    if key in seen: continue\n    seen.add(key); gsel.append(i); tot+=ntok[i]+1\n    if tot>=BUDGET: break\n# balanced K=4 to 12M\ncand={c:(lambda oo:oo[gate[oo]])(np.argsort(-scores[:,c])) for c in range(K)}\nq=w*COVER; fl=np.zeros(K); pt2=np.zeros(K,int); seen=set(); bsel=[]; tot=0\nwhile tot<BUDGET:\n    pk=False\n    for c in np.argsort(-(q-fl)):\n        while pt2[c]<len(cand[c]):\n            i=cand[c][pt2[c]]; pt2[c]+=1\n            key=hash(tokd[off[i]:min(off[i+1],off[i]+64)].tobytes())\n            if key in seen: continue\n            seen.add(key); bsel.append(i); s=ntok[i]+1; fl[c]+=s; tot+=s; pk=True; break\n        if pk: break\n    if not pk: break\nrng2=np.random.default_rng(1); rperm=rng2.permutation(n); rsel=[]; tot=0\nfor i in rperm:\n    rsel.append(i); tot+=ntok[i]+1\n    if tot>=BUDGET: break\nprint('target register mixture w =', np.round(w,3).tolist())\nfor name,sl in [('random',rsel),('global-DSIR',gsel),('balanced-K4',bsel)]:\n    m=mix(sl); print(f'{name:12s} mixture {np.round(m,3).tolist()}  KL(sel||target)={kl(m,w):.3f}  ndocs {len(sl)}')\n", "structuredPatch": [], "originalFile": null, "userModified": false}

loaded 182016 docs 6.9 s
features done 27.1 s
Token indices sequence length is longer than the specified maximum sequence length for this model (1048 > 1024). Running this sequence through the model will result in indexing errors
---timer---
2959

[stdout]
loaded 182016 docs 6.9 s
features done 27.1 s
Token indices sequence length is longer than the specified maximum sequence length for this model (1048 > 1024). Running this sequence through the model will result in indexing errors
---timer---
2959
[metadata] {"noOutputExpected": false}

target register mixture w = [0.096, 0.15, 0.496, 0.258]
random       mixture [0.022, 0.003, 0.809, 0.165]  KL(sel||target)=0.277  ndocs 10805
global-DSIR  mixture [0.001, 0.001, 0.987, 0.01]  KL(sel||target)=0.636  ndocs 15938
balanced-K4  mixture [0.003, 0.003, 0.88, 0.115]  KL(sel||target)=0.392  ndocs 18409

[stdout]
target register mixture w = [0.096, 0.15, 0.496, 0.258]
random       mixture [0.022, 0.003, 0.809, 0.165]  KL(sel||target)=0.277  ndocs 10805
global-DSIR  mixture [0.001, 0.001, 0.987, 0.01]  KL(sel||target)=0.636  ndocs 15938
balanced-K4  mixture [0.003, 0.003, 0.88, 0.115]  KL(sel||target)=0.392  ndocs 18409
[metadata] {"noOutputExpected": false}

loaded 182016 docs 6.9 s
features done 27.1 s
---sel check---
selection.json ids 21097 first5 [105413, 181263, 37522, 102634, 66736] allint True
2849

[stdout]
loaded 182016 docs 6.9 s
features done 27.1 s
---sel check---
selection.json ids 21097 first5 [105413, 181263, 37522, 102634, 66736] allint True
2849
[metadata] {"noOutputExpected": false}

15:class GPTConfig:
25:class LayerNorm(nn.Module):
26:    def __init__(self, ndim, bias):
31:    def forward(self, x):
32:        return F.layer_norm(x, self.weight.shape, self.weight, self.bias, 1e-5)
35:class CausalSelfAttention(nn.Module):
36:    def __init__(self, cfg: GPTConfig):
45:    def forward(self, x):
55:        return self.c_proj(y)
58:class MLP(nn.Module):
59:    def __init__(self, cfg: GPTConfig):
65:    def forward(self, x):
66:        return self.dropout(self.c_proj(F.gelu(self.c_fc(x))))
69:class Block(nn.Module):
70:    def __init__(self, cfg: GPTConfig):
77:    def forward(self, x):
80:        return x
83:class GPT(nn.Module):
84:    def __init__(self, cfg: GPTConfig):
111:    def forward(self, idx, targets=None):
112:        """Return (logits, ce_loss). logits: [B, T, vocab]. ce_loss is the plain
114:        loss is computed separately in kd_trainer.kd_loss."""
121:        logits = self.lm_head(x)
122:        ce_loss = None
124:            ce_loss = F.cross_entropy(
125:                logits.view(-1, logits.size(-1)), targets.reshape(-1), ignore_index=-1
127:        return logits, ce_loss
140:        return torch.optim.AdamW(groups, lr=lr, betas=betas, fused=fused)
147:        losses = torch.zeros(n_batches)
150:            _, loss = self(x, y)
151:            losses[i] = loss.item()
154:        return math.exp(losses.mean().item())

[stdout]
15:class GPTConfig:
25:class LayerNorm(nn.Module):
26:    def __init__(self, ndim, bias):
31:    def forward(self, x):
32:        return F.layer_norm(x, self.weight.shape, self.weight, self.bias, 1e-5)
35:class CausalSelfAttention(nn.Module):
36:    def __init__(self, cfg: GPTConfig):
45:    def forward(self, x):
55:        return self.c_proj(y)
58:class MLP(nn.Module):
59:    def __init__(self, cfg: GPTConfig):
65:    def forward(self, x):
66:        return self.dropout(self.c_proj(F.gelu(self.c_fc(x))))
69:class Block(nn.Module):
70:    def __init__(self, cfg: GPTConfig):
77:    def forward(self, x):
80:        return x
83:class GPT(nn.Module):
84:    def __init__(self, cfg: GPTConfig):
111:    def forward(self, idx, targets=None):
112:        """Return (logits, ce_loss). logits: [B, T, vocab]. ce_loss is the plain
114:        loss is computed separately in kd_trainer.kd_loss."""
121:        logits = self.lm_head(x)
122:        ce_loss = None
124:            ce_loss = F.cross_entropy(
125:                logits.view(-1, logits.size(-1)), targets.reshape(-1), ignore_index=-1
127:        return logits, ce_loss
140:        return torch.optim.AdamW(groups, lr=lr, betas=betas, fused=fused)
147:        losses = torch.zeros(n_batches)
150:            _, loss = self(x, y)
151:            losses[i] = loss.item()
154:        return math.exp(losses.mean().item())
[metadata] {"noOutputExpected": false}

File created successfully at: /workspace/eval_perreg.py (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/workspace/eval_perreg.py", "content": "\"\"\"Diagnostic (NOT the official scorer): train the frozen recipe on a selection,\nthen report held-out dev loss SPLIT BY the 4 target registers. Used only to expose\nthe mechanism behind the aggregate perplexity.\"\"\"\nimport sys, math, time, numpy as np, torch, torch.nn.functional as Fn\nfrom model import GPT, GPTConfig\nTRAIN=sys.argv[1]; LABEL=sys.argv[2] if len(sys.argv)>2 else 'sel'\nV,EOS=50257,50256; K=4; block=256; batch=32; max_iters=3000; warmup=150; lr=6e-4; seed=1337\ntorch.manual_seed(seed); np.random.seed(seed); dev='cuda'; rng=np.random.default_rng(seed)\n\n# --- dev docs + K=4 clusters (deterministic, matches curate.py) ---\nho=np.load('/workspace/data/multi_dev.npy').astype(np.int64)\nbnd=np.where(ho==EOS)[0]; docs=[]; prev=0\nfor j in bnd:\n    if j>prev: docs.append((prev,j))\n    prev=j+1\nif prev<len(ho): docs.append((prev,len(ho)))\nsegs=[ho[a:b] for a,b in docs]\nCt=np.bincount(ho[ho!=EOS],minlength=V).astype(np.float64)\nrank=np.argsort(-Ct); vocab=rank[30:4030]; vmap=-np.ones(V,np.int64); vmap[vocab]=np.arange(len(vocab))\ndef tf(s):\n    m=vmap[s]; m=m[m>=0]; v=np.bincount(m,minlength=len(vocab)).astype(np.float32); nn=np.linalg.norm(v); return v/nn if nn>0 else v\nX=np.stack([tf(s) for s in segs]); r=np.random.default_rng(0); C=X[r.choice(len(X),K,replace=False)].copy()\nfor _ in range(25):\n    asg=(X@C.T).argmax(1); nC=np.zeros_like(C)\n    for c in range(K):\n        m=X[asg==c]\n        if len(m): v=m.sum(0); nn=np.linalg.norm(v); nC[c]=v/nn if nn>0 else C[c]\n        else: nC[c]=X[r.integers(len(X))]\n    if np.allclose(nC,C): break\n    C=nC\n# map each dev position -> cluster (by containing doc); positions in EOS gaps -> -1\npos_cluster=np.full(len(ho),-1,np.int64)\nfor (a,b),c in zip(docs,asg): pos_cluster[a:b]=c\n\n# --- train exactly like train_nano ---\ntr=torch.from_numpy(np.load(TRAIN).astype(np.int64))\nmodel=GPT(GPTConfig(block_size=block,vocab_size=V,n_layer=6,n_head=6,n_embd=384,dropout=0.0,bias=False)).to(dev)\nopt=model.configure_optimizers(0.1,lr,(0.9,0.95),'cuda')\ndef lr_at(it):\n    if it<warmup: return lr*(it+1)/(warmup+1)\n    rr=(it-warmup)/max(1,max_iters-warmup); return 0.1*lr+0.5*(1+math.cos(math.pi*rr))*(lr-0.1*lr)\ndef get_batch():\n    ix=rng.integers(0,len(tr)-block-1,size=batch)\n    x=torch.stack([tr[i:i+block] for i in ix]).to(dev); y=torch.stack([tr[i+1:i+1+block] for i in ix]).to(dev)\n    return x,y\nt0=time.time(); model.train()\nfor it in range(max_iters):\n    for g in opt.param_groups: g['lr']=lr_at(it)\n    x,y=get_batch()\n    with torch.autocast('cuda',dtype=torch.bfloat16): _,loss=model(x,y)\n    opt.zero_grad(set_to_none=True); loss.backward()\n    torch.nn.utils.clip_grad_norm_(model.parameters(),1.0); opt.step()\n\n# --- eval per-token CE over non-overlapping windows, bucket by cluster ---\nmodel.eval(); ho_t=torch.from_numpy(ho.astype(np.int64))\nnwin=(len(ho)-1)//block; sl=np.zeros(K+1); cnt=np.zeros(K+1)  # index K = overall\nwith torch.no_grad():\n    for i in range(0,nwin,batch):\n        bs=[j*block for j in range(i,min(i+batch,nwin))]\n        x=torch.stack([ho_t[s:s+block] for s in bs]).to(dev)\n        y=torch.stack([ho_t[s+1:s+1+block] for s in bs]).to(dev)\n        with torch.autocast('cuda',dtype=torch.bfloat16): logits,_=model(x,y)\n        ce=Fn.cross_entropy(logits.reshape(-1,V).float(),y.reshape(-1),reduction='none').reshape(len(bs),block).cpu().numpy()\n        for k,s in enumerate(bs):\n            cl=pos_cluster[s+1:s+1+block]  # target token clusters\n            for c in range(K):\n                mm=cl==c; sl[c]+=ce[k][mm].sum(); cnt[c]+=mm.sum()\n            sl[K]+=ce[k].sum(); cnt[K]+=block\nprint(f'[{LABEL}] train_s {time.time()-t0:.0f}')\nnames=['reg0','reg1','reg2','reg3','OVERALL']\nfor c in range(K+1):\n    if cnt[c]>0: print(f'  {names[c]:8s} ppl {math.exp(sl[c]/cnt[c]):8.2f}  toks {int(cnt[c])}')\n", "structuredPatch": [], "originalFile": null, "userModified": false}

loaded 182016 docs 6.9 s
features done 27.1 s
---sel now---
ids 21097 first5 [105413, 181263, 37522, 102634, 66736]
---timer---
2799

[stdout]
loaded 182016 docs 6.9 s
features done 27.1 s
---sel now---
ids 21097 first5 [105413, 181263, 37522, 102634, 66736]
---timer---
2799
[metadata] {"noOutputExpected": false}

154
156
157
4001
4376
RUNNING
---full log tail---
loaded 182016 docs 6.9 s
features done 27.1 s
Token indices sequence length is longer than the specified maximum sequence length for this model (1048 > 1024). Running this sequence through the model will result in indexing errors

[stdout]
154
156
157
4001
4376
RUNNING
---full log tail---
loaded 182016 docs 6.9 s
features done 27.1 s
Token indices sequence length is longer than the specified maximum sequence length for this model (1048 > 1024). Running this sequence through the model will result in indexing errors
[metadata] {"noOutputExpected": false}

packed 12000000 tokens from up to 21097 ids -> /tmp/global_train.npy
launched perreg 4426

[stdout]
packed 12000000 tokens from up to 21097 ids -> /tmp/global_train.npy
launched perreg 4426
[metadata] {"noOutputExpected": false}

=== curate ===
gate keeps 154451 / 182016
register mixture [0.096, 0.15, 0.496, 0.258]
selected 22284 ids ~ 14502146 tokens -> /workspace/submission/selection.json 356.5 s
=== perreg ===
[GLOBAL-DSIR] train_s 56
  reg0     ppl   755.10  toks 95748
  reg1     ppl   546.48  toks 150036
  reg2     ppl   280.20  toks 494453
  reg3     ppl   459.26  toks 256995
  OVERALL  ppl   389.70  toks 999936
---timer---
2669

[stdout]
=== curate ===
gate keeps 154451 / 182016
register mixture [0.096, 0.15, 0.496, 0.258]
selected 22284 ids ~ 14502146 tokens -> /workspace/submission/selection.json 356.5 s
=== perreg ===
[GLOBAL-DSIR] train_s 56
  reg0     ppl   755.10  toks 95748
  reg1     ppl   546.48  toks 150036
  reg2     ppl   280.20  toks 494453
  reg3     ppl   459.26  toks 256995
  OVERALL  ppl   389.70  toks 999936
---timer---
2669
[metadata] {"noOutputExpected": false}

[GLOBAL-DSIR] train_s 56
  reg0     ppl   755.10  toks 95748
  reg1     ppl   546.48  toks 150036
  reg2     ppl   280.20  toks 494453
  reg3     ppl   459.26  toks 256995
  OVERALL  ppl   389.70  toks 999936
[BALANCED-K4] train_s 56
  reg0     ppl   753.94  toks 95748
  reg1     ppl   589.75  toks 150036
  reg2     ppl   282.41  toks 494453
  reg3     ppl   178.13  toks 256995
  OVERALL  ppl   310.62  toks 999936
PERREG_DONE
---timer---
2599

[stdout]
[GLOBAL-DSIR] train_s 56
  reg0     ppl   755.10  toks 95748
  reg1     ppl   546.48  toks 150036
  reg2     ppl   280.20  toks 494453
  reg3     ppl   459.26  toks 256995
  OVERALL  ppl   389.70  toks 999936
[BALANCED-K4] train_s 56
  reg0     ppl   753.94  toks 95748
  reg1     ppl   589.75  toks 150036
  reg2     ppl   282.41  toks 494453
  reg3     ppl   178.13  toks 256995
  OVERALL  ppl   310.62  toks 999936
PERREG_DONE
---timer---
2599
[metadata] {"noOutputExpected": false}

launched final-verify 4712

[stdout]
launched final-verify 4712
[metadata] {"noOutputExpected": false}