nano-gpt-data-curation

claude-opus-5

nano-gpt-data-curation — graded by the task's verifier against a measured baseline.

backend: modal
variant: claude claude-opus-5
metric: held-out multi-domain perplexity
actions: 49
duration: 1h 10m
recorded: 2026-08-04

run iddclm-nano-curation__HzsPDMT

Everything claude-opus-5 did on this task, in order.

download JSON ↗

claude-codeclaude-opus-548 turns23 narration49 action49 observation25 reasoning1 final

#26

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolBash

args

{
  "command": "sed -i 's/if len(tk) < 40:/if len(tk) < 100:/' submission/curate.py && grep -n \"len(tk) < 100\" submission/curate.py; tail -2 /tmp/eval_rand.log; cat .timer/remaining_secs"
}

Bash result

307:            if len(tk) < 100:
Token indices sequence length is longer than the specified maximum sequence length for this model (1230 > 1024). Running this sequence through the model will result in indexing errors
4499

[stdout]
307:            if len(tk) < 100:
Token indices sequence length is longer than the specified maximum sequence length for this model (1230 > 1024). Running this sequence through the model will result in indexing errors
4499
[metadata] {"noOutputExpected": false}

#27

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolBashRandom baseline + curate v3

args

{
  "command": "grep -h HELDOUT /tmp/eval_rand.log; python submission/curate.py 2>&1 | grep -viE \"warning|futurew|detach|return float\" | tail -12",
  "description": "Random baseline + curate v3",
  "timeout": 1500000
}

Bash result

qa: 336 chunks
loading pool ...
  182016 pool docs
normalizing pool ...
precomputing pool features ...
penalties ...
  wiki: loss 0.1722 bal-acc 0.9924
  qa: loss 0.0632 bal-acc 0.9980
  webnews: loss 0.2719 bal-acc 0.9886
interleaving domains to budget shares ...
  est tokens per domain: {'wiki': 9008032, 'qa': 8998700, 'webnews': 17993891}
wrote 42287 ids -> /workspace/submission/selection.json

[stdout]
  qa: 336 chunks
loading pool ...
  182016 pool docs
normalizing pool ...
precomputing pool features ...
penalties ...
  wiki: loss 0.1722 bal-acc 0.9924
  qa: loss 0.0632 bal-acc 0.9980
  webnews: loss 0.2719 bal-acc 0.9886
interleaving domains to budget shares ...
  est tokens per domain: {'wiki': 9008032, 'qa': 8998700, 'webnews': 17993891}
wrote 42287 ids -> /workspace/submission/selection.json
[metadata] {"noOutputExpected": false}

#28

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolBashEval v3

args

{
  "command": "cp submission/selection.json /tmp/sel_v3.json; bash dev_eval.sh 2>&1 | grep HELDOUT; cat .timer/remaining_secs",
  "description": "Eval v3",
  "timeout": 1500000
}

Bash result

HELDOUT_PPL 304.127  loss 5.7174  train_s 53  /tmp/dev_train.npy
4238

[stdout]
HELDOUT_PPL 304.127  loss 5.7174  train_s 53  /tmp/dev_train.npy
4238
[metadata] {"noOutputExpected": false}

#29

narration

304.1 (333.9 → 317.3 → 304.1). Checking a potentially large lever — does the pool contain any wikitext-style detokenized text matching the target's surface form?

toolBashCheck for wikitext-format docs in pool

args

{
  "command": "python -c \"\nimport json,re,pickle\nDET=re.compile(r' [.,]')\nn=0; hits=0; at=0; ex=[]\nfor line in open('data/pool.jsonl'):\n    r=json.loads(line); t=r['text']; n+=1\n    if '@-@' in t or '@,@' in t: at+=1\n    d=len(DET.findall(t))/max(1,len(t))\n    if d>0.004 and len(t)>500:\n        hits+=1\n        if len(ex)<3: ex.append(t[:200])\nprint('pool docs',n,'with @-@:',at,'detok-spacing:',hits)\nfor e in ex: print('---',repr(e))\n\"",
  "description": "Check for wikitext-format docs in pool",
  "timeout": 1500000
}

Bash result

pool docs 182016 with @-@: 0 detok-spacing: 931
--- 'I called this on the 19th with this post but I guess it is now official. Boy am I good. Read below:\nJuly 25, 2011. MediaTakeOut.com has just received some EARTH SHATTERING NEWS . . . Jennifer Lopez an'
--- 'Borden lemon meringue pie\nRecipes / Borden lemon meringue pie (1000+)\nBordens Lemon Meringue Pie, ingredients: 3 x Large eggs, separated, 1 x (15-oz) can\nThe Best Lemon Meringue Pie, ingredients: 1 1/'
--- 'Riley on Cloyd: Our ability to serve receive and defend will be critical for the teams success ... expecting Amanda to take a large portion of the serve receive role in order to keep the team in syste'

[stdout]
pool docs 182016 with @-@: 0 detok-spacing: 931
--- 'I called this on the 19th with this post but I guess it is now official. Boy am I good. Read below:\nJuly 25, 2011. MediaTakeOut.com has just received some EARTH SHATTERING NEWS . . . Jennifer Lopez an'
--- 'Borden lemon meringue pie\nRecipes / Borden lemon meringue pie (1000+)\nBordens Lemon Meringue Pie, ingredients: 3 x Large eggs, separated, 1 x (15-oz) can\nThe Best Lemon Meringue Pie, ingredients: 1 1/'
--- 'Riley on Cloyd: Our ability to serve receive and defend will be critical for the teams success ... expecting Amanda to take a large portion of the serve receive role in order to keep the team in syste'
[metadata] {"noOutputExpected": false}

#30

narration

No wikitext-formatted docs in the pool — format matching is impossible, so normalizing it away was right. Now adding a score cache so I can sweep domain shares cheaply:

toolEdit

args

{
  "replace_all": false,
  "file_path": "/workspace/submission/curate.py",
  "old_string": "    dom_scores = {}\n    for dom in (\"wiki\", \"qa\", \"webnews\"):",
  "new_string": "    dom_scores = {}\n    sc_cache = \"/tmp/dom_scores.npz\"\n    if os.path.exists(sc_cache):\n        z = np.load(sc_cache)\n        dom_scores = {k: z[k] for k in z.files}\n    for dom in (\"wiki\", \"qa\", \"webnews\"):\n        if dom in dom_scores:\n            continue"
}

Edit result

The file /workspace/submission/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/submission/curate.py", "oldString": "    dom_scores = {}\n    for dom in (\"wiki\", \"qa\", \"webnews\"):", "newString": "    dom_scores = {}\n    sc_cache = \"/tmp/dom_scores.npz\"\n    if os.path.exists(sc_cache):\n        z = np.load(sc_cache)\n        dom_scores = {k: z[k] for k in z.files}\n    for dom in (\"wiki\", \"qa\", \"webnews\"):\n        if dom in dom_scores:\n            continue", "originalFile": null, "structuredPatch": [{"oldStart": 267, "oldLines": 7, "newStart": 267, "newLines": 13, "lines": ["     # One classifier per target register; each ranks the whole pool for", "     # \"looks like THIS register\".", "     dom_scores = {}", "+    sc_cache = \"/tmp/dom_scores.npz\"", "+    if os.path.exists(sc_cache):", "+        z = np.load(sc_cache)", "+        dom_scores = {k: z[k] for k in z.files}", "     for dom in (\"wiki\", \"qa\", \"webnews\"):", "+        if dom in dom_scores:", "+            continue", "         pos_rows = [featurize(norm_tokens(t), cap=400) for t in groups[dom]]", "         bag = rows_to_bag(pos_rows + neg_rows, device)", "         y = np.concatenate([np.ones(len(pos_rows)), np.zeros(len(neg_rows))])"]}], "userModified": false, "replaceAll": false}

#31

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolBashAdd share env override and score cache

args

{
  "command": "python - <<'EOF'\nimport re\np='submission/curate.py'; s=open(p).read()\ns=s.replace('''        print(f\"  {dom}: loss {loss:.4f} bal-acc {acc:.4f}\", flush=True)''',\n'''        print(f\"  {dom}: loss {loss:.4f} bal-acc {acc:.4f}\", flush=True)\n    np.savez(sc_cache, **dom_scores)''')\ns=s.replace('''    SHARE = {\"wiki\": 0.25, \"qa\": 0.25, \"webnews\": 0.50}''',\n'''    SHARE = json.loads(os.environ.get(\n        \"SHARES\", '{\"wiki\": 0.25, \"qa\": 0.25, \"webnews\": 0.50}'))''')\ns=s.replace('''import json, re, sys, hashlib''','''import json, os, re, sys, hashlib''')\ns=s.replace('''    import os, pickle\n    cache''','''    import pickle\n    cache''')\nopen(p,'w').write(s)\nEOF\ngrep -n \"SHARES\\|savez\\|^import\" submission/curate.py",
  "description": "Add share env override and score cache"
}

Bash result

31:import json, os, re, sys, hashlib
32:import numpy as np
289:    np.savez(sc_cache, **dom_scores)
297:        "SHARES", '{"wiki": 0.25, "qa": 0.25, "webnews": 0.50}'))

[stdout]
31:import json, os, re, sys, hashlib
32:import numpy as np
289:    np.savez(sc_cache, **dom_scores)
297:        "SHARES", '{"wiki": 0.25, "qa": 0.25, "webnews": 0.50}'))
[metadata] {"noOutputExpected": false}

#32

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolBashMeasure dev token shares

args

{
  "command": "python -c \"\nimport numpy as np,re,sys\nsys.path.insert(0,'submission')\nfrom curate import dev_domain\nfrom transformers import AutoTokenizer\ntok=AutoTokenizer.from_pretrained('gpt2')\nd=np.load('data/multi_dev.npy').astype(np.int64)\ncuts=np.where(d==50256)[0]; prev=0\nfrom collections import Counter\ntk=Counter()\nfor c in cuts:\n    if c-prev>32: tk[dev_domain(tok.decode(d[prev:c]))]+=int(c-prev)\n    prev=c+1\nT=sum(tk.values()); print({k:round(v/T,3) for k,v in tk.items()})\n\" 2>&1 | grep -v Warning",
  "description": "Measure dev token shares",
  "timeout": 1500000
}

Bash result

{'wiki': 0.308, 'webnews': 0.463, 'qa': 0.229}

[stdout]
{'wiki': 0.308, 'webnews': 0.463, 'qa': 0.229}
[metadata] {"noOutputExpected": false}

#33

narration

Measured dev shares: wiki 0.308 / webnews 0.463 / qa 0.229. Testing exact-match shares:

toolBashExperiment: measured shares

args

{
  "command": "SHARES='{\"wiki\":0.308,\"qa\":0.229,\"webnews\":0.463}' python submission/curate.py 2>&1 | tail -3\ncp submission/selection.json /tmp/sel_v4.json; bash dev_eval.sh 2>&1|grep HELDOUT; cat .timer/remaining_secs",
  "description": "Experiment: measured shares",
  "timeout": 1500000
}

Bash result

interleaving domains to budget shares ...
  est tokens per domain: {'wiki': 11093340, 'qa': 8242758, 'webnews': 16663931}
wrote 43255 ids -> /workspace/submission/selection.json
HELDOUT_PPL 314.035  loss 5.7495  train_s 54  /tmp/dev_train.npy
3938

[stdout]
interleaving domains to budget shares ...
  est tokens per domain: {'wiki': 11093340, 'qa': 8242758, 'webnews': 16663931}
wrote 43255 ids -> /workspace/submission/selection.json
HELDOUT_PPL 314.035  loss 5.7495  train_s 54  /tmp/dev_train.npy
3938
[metadata] {"noOutputExpected": false}

#34

narration

Exact-match shares are worse (314.0) — deeper into the wiki ranking pulls lower-quality docs. Probing the other direction:

toolBashExperiment: less wiki

args

{
  "command": "SHARES='{\"wiki\":0.15,\"qa\":0.25,\"webnews\":0.60}' python submission/curate.py 2>&1 | tail -2\ncp submission/selection.json /tmp/sel_v5.json; bash dev_eval.sh 2>&1|grep HELDOUT",
  "description": "Experiment: less wiki",
  "timeout": 1500000
}

Bash result

est tokens per domain: {'wiki': 5399652, 'qa': 9003389, 'webnews': 21598670}
wrote 40293 ids -> /workspace/submission/selection.json
HELDOUT_PPL 310.448  loss 5.7380  train_s 54  /tmp/dev_train.npy

[stdout]
  est tokens per domain: {'wiki': 5399652, 'qa': 9003389, 'webnews': 21598670}
wrote 40293 ids -> /workspace/submission/selection.json
HELDOUT_PPL 310.448  loss 5.7380  train_s 54  /tmp/dev_train.npy
[metadata] {"noOutputExpected": false}

#35

narration

Near-optimum on shares. Now testing whether strict top-k is over-concentrated, via temperature-based sampling within each domain ranking:

toolBashExperiment: temperature 1.0

args

{
  "command": "python - <<'EOF'\np='submission/curate.py'; s=open(p).read()\nold='''    orders = {d: np.argsort(-dom_scores[d]) for d in SHARE}'''\nnew='''    # Ranking sharpness. TEMP=0 takes the strict top-k of each domain ranking.\n    # TEMP>0 draws without replacement with probability rising in the score\n    # (Gumbel top-k), which keeps the quality bar high but broadens topical\n    # coverage -- a hedge against the classifier collapsing onto one narrow\n    # sub-register of a domain (e.g. all South-Asian politics inside \"news\").\n    TEMP = float(os.environ.get(\"TEMP\", \"0\"))\n    if TEMP > 0:\n        grng = np.random.default_rng(SEED)\n        orders = {}\n        for d in SHARE:\n            sc = dom_scores[d] / TEMP\n            g = grng.gumbel(size=sc.shape).astype(np.float32)\n            orders[d] = np.argsort(-(sc + g))\n    else:\n        orders = {d: np.argsort(-dom_scores[d]) for d in SHARE}'''\nassert old in s\ns=s.replace(old,new); open(p,'w').write(s)\nEOF\nTEMP=1.0 python submission/curate.py 2>&1|tail -2; cp submission/selection.json /tmp/sel_t1.json; bash dev_eval.sh 2>&1|grep HELDOUT; cat .timer/remaining_secs",
  "description": "Experiment: temperature 1.0",
  "timeout": 1500000
}

Bash result

est tokens per domain: {'wiki': 8998285, 'qa': 9004614, 'webnews': 17997118}
wrote 40511 ids -> /workspace/submission/selection.json
HELDOUT_PPL 317.054  loss 5.7591  train_s 54  /tmp/dev_train.npy
3387

[stdout]
  est tokens per domain: {'wiki': 8998285, 'qa': 9004614, 'webnews': 17997118}
wrote 40511 ids -> /workspace/submission/selection.json
HELDOUT_PPL 317.054  loss 5.7591  train_s 54  /tmp/dev_train.npy
3387
[metadata] {"noOutputExpected": false}

#36

narration

TEMP sampling is worse (317.1) — the strict ranking is genuinely informative. Best remains v3 at 304.1. One more improvement attempt: bigram features for a sharper classifier.

toolBashCurate with bigrams

args

{
  "command": "python - <<'EOF'\np='submission/curate.py'; s=open(p).read()\nold='''def featurize(toks, cap=3000):\n    \"\"\"L2-normalized sqrt-tf hashed bag of words over the first `cap` tokens.\"\"\"\n    cnt = Counter(toks[:cap])'''\nnew='''BIGRAM = os.environ.get(\"BIGRAM\", \"0\") == \"1\"\n\ndef featurize(toks, cap=3000):\n    \"\"\"L2-normalized sqrt-tf hashed bag of words over the first `cap` tokens.\n    With BIGRAM=1, adjacent word pairs are hashed in alongside unigrams, which\n    lets the classifier see register cues that single words miss (\"said the\",\n    \"was born in\", \"how do i\").\"\"\"\n    w = toks[:cap]\n    cnt = Counter(w)\n    if BIGRAM:\n        cnt.update(map(\"\\\\x00\".join, zip(w, w[1:])))'''\nassert old in s\ns=s.replace(old,new)\ns=s.replace('sc_cache = \"/tmp/dom_scores.npz\"',\n            'sc_cache = \"/tmp/dom_scores%s.npz\" % (\"_bg\" if BIGRAM else \"\")')\ns=s.replace('cache = \"/tmp/pool_toks.pkl\"','cache = \"/tmp/pool_toks.pkl\"')\nopen(p,'w').write(s)\nEOF\nBIGRAM=1 python submission/curate.py 2>&1|grep -viE \"warning|detach|return float\"|tail -7",
  "description": "Curate with bigrams",
  "timeout": 1500000
}

Bash result

penalties ...
  wiki: loss 0.1764 bal-acc 0.9955
  qa: loss 0.0678 bal-acc 0.9988
  webnews: loss 0.2323 bal-acc 0.9959
interleaving domains to budget shares ...
  est tokens per domain: {'wiki': 9010018, 'qa': 8997329, 'webnews': 17993366}
wrote 42356 ids -> /workspace/submission/selection.json

[stdout]
penalties ...
  wiki: loss 0.1764 bal-acc 0.9955
  qa: loss 0.0678 bal-acc 0.9988
  webnews: loss 0.2323 bal-acc 0.9959
interleaving domains to budget shares ...
  est tokens per domain: {'wiki': 9010018, 'qa': 8997329, 'webnews': 17993366}
wrote 42356 ids -> /workspace/submission/selection.json
[metadata] {"noOutputExpected": false}

#37

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolBashEval bigram selection

args

{
  "command": "cp submission/selection.json /tmp/sel_bg.json; bash dev_eval.sh 2>&1|grep HELDOUT; cat .timer/remaining_secs",
  "description": "Eval bigram selection",
  "timeout": 1500000
}

Bash result

HELDOUT_PPL 308.155  loss 5.7306  train_s 54  /tmp/dev_train.npy
3047

[stdout]
HELDOUT_PPL 308.155  loss 5.7306  train_s 54  /tmp/dev_train.npy
3047
[metadata] {"noOutputExpected": false}

#38

narration

Bigrams slightly worse (308.2). Reverting to unigrams. Next: document-length floor (longer docs = fewer EOS breaks in the 256-token training windows).

toolBashExperiment: min length 300

args

{
  "command": "python - <<'EOF'\np='submission/curate.py'; s=open(p).read()\ns=s.replace('''            if len(tk) < 100:''','''            if len(tk) < MINLEN:''')\ns=s.replace('''    GOAL = 12_000_000 * 3.0''','''    GOAL = 12_000_000 * 3.0\n    MINLEN = int(os.environ.get(\"MINLEN\", \"100\"))''')\nopen(p,'w').write(s)\nEOF\nMINLEN=300 python submission/curate.py 2>&1|tail -2; cp submission/selection.json /tmp/sel_ml300.json; bash dev_eval.sh 2>&1|grep HELDOUT",
  "description": "Experiment: min length 300",
  "timeout": 1500000
}

Bash result

est tokens per domain: {'wiki': 9032518, 'qa': 8989881, 'webnews': 17979622}
wrote 25413 ids -> /workspace/submission/selection.json
HELDOUT_PPL 316.565  loss 5.7575  train_s 54  /tmp/dev_train.npy

[stdout]
  est tokens per domain: {'wiki': 9032518, 'qa': 8989881, 'webnews': 17979622}
wrote 25413 ids -> /workspace/submission/selection.json
HELDOUT_PPL 316.565  loss 5.7575  train_s 54  /tmp/dev_train.npy
[metadata] {"noOutputExpected": false}

#39

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolBashExperiment: min length 40

args

{
  "command": "MINLEN=40 python submission/curate.py 2>&1|tail -1; cp submission/selection.json /tmp/sel_ml40.json; bash dev_eval.sh 2>&1|grep HELDOUT; cat .timer/remaining_secs",
  "description": "Experiment: min length 40",
  "timeout": 1500000
}

Bash result

wrote 48016 ids -> /workspace/submission/selection.json
HELDOUT_PPL 314.168  loss 5.7499  train_s 54  /tmp/dev_train.npy
2517

[stdout]
wrote 48016 ids -> /workspace/submission/selection.json
HELDOUT_PPL 314.168  loss 5.7499  train_s 54  /tmp/dev_train.npy
2517
[metadata] {"noOutputExpected": false}

#40

narration

Non-monotone optimum at MINLEN=100 (40→314, 100→304, 300→317) confirms these gaps are signal. Locking in the best config and regenerating cleanly:

toolBashRegenerate final selection with default config

args

{
  "command": "rm -f /tmp/dom_scores.npz /tmp/dom_scores_bg.npz && python submission/curate.py 2>&1|grep -viE \"warning|detach|return float\"|tail -8\npython -c \"\nimport json\na=json.load(open('/workspace/submission/selection.json')); b=json.load(open('/tmp/sel_v3.json'))\nprint('matches best-config run:', a==b)\nprint('n',len(a),'unique',len(set(a)),'in range',min(a)>=0 and max(a)<182016)\n\"",
  "description": "Regenerate final selection with default config",
  "timeout": 1500000
}

Bash result

precomputing pool features ...
penalties ...
  wiki: loss 0.1722 bal-acc 0.9924
  qa: loss 0.0632 bal-acc 0.9980
  webnews: loss 0.2719 bal-acc 0.9886
interleaving domains to budget shares ...
  est tokens per domain: {'wiki': 9008032, 'qa': 8998700, 'webnews': 17993891}
wrote 42287 ids -> /workspace/submission/selection.json
matches best-config run: True
n 42287 unique 42287 in range True

[stdout]
precomputing pool features ...
penalties ...
  wiki: loss 0.1722 bal-acc 0.9924
  qa: loss 0.0632 bal-acc 0.9980
  webnews: loss 0.2719 bal-acc 0.9886
interleaving domains to budget shares ...
  est tokens per domain: {'wiki': 9008032, 'qa': 8998700, 'webnews': 17993891}
wrote 42287 ids -> /workspace/submission/selection.json
matches best-config run: True
n 42287 unique 42287 in range True
[metadata] {"noOutputExpected": false}

#41

narration

Reproduced exactly. Running the random baseline for reference while I write the claim:

toolBashRandom baseline eval

args

{
  "command": "nohup bash dev_eval.sh /tmp/random_sel.json > /tmp/eval_rand.log 2>&1 & echo started",
  "description": "Random baseline eval"
}

Bash result

started

[stdout]
started
[metadata] {"noOutputExpected": false}

#42

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolBashGet random baseline result

args

{
  "command": "sleep 170; grep -h HELDOUT /tmp/eval_rand.log; cat .timer/remaining_secs",
  "description": "Get random baseline result",
  "timeout": 1500000
}

Bash result

HELDOUT_PPL 459.787  loss 6.1308  train_s 54  /tmp/dev_train.npy
2166

[stdout]
HELDOUT_PPL 459.787  loss 6.1308  train_s 54  /tmp/dev_train.npy
2166
[metadata] {"noOutputExpected": false}

#43

narration

Random baseline = 459.8 vs our 304.1. Let me measure the mechanism observable (vocabulary divergence) before writing the claim:

toolBashMeasure vocabulary divergence

args

{
  "command": "python -c \"\nimport json,numpy as np,pickle,sys\nsys.path.insert(0,'submission')\nfrom curate import norm_tokens, dev_domain\nfrom collections import Counter\nfrom transformers import AutoTokenizer\ntok=AutoTokenizer.from_pretrained('gpt2')\nd=np.load('data/multi_dev.npy').astype(np.int64)\ncuts=np.where(d==50256)[0]; prev=0; T=Counter()\nfor c in cuts:\n    if c-prev>32: T.update(norm_tokens(tok.decode(d[prev:c])))\n    prev=c+1\ntoks=pickle.load(open('/tmp/pool_toks.pkl','rb'))\nids=[]; \nfor line in open('data/pool.jsonl'): ids.append(json.loads(line)['id'])\npos={v:i for i,v in enumerate(ids)}\nsel=json.load(open('submission/selection.json'))\nrnd=json.load(open('/tmp/random_sel.json'))\ndef counts(idlist,budget=12_000_000):\n    C=Counter(); t=0\n    for i in idlist:\n        w=toks[pos[i]]; C.update(w); t+=len(w)*1.35\n        if t>=budget: break\n    return C\ndef xent(T,C):\n    V=set(T)|set(C); n=sum(C.values()); k=len(V)\n    tot=sum(T.values()); s=0.0\n    for w,c in T.items():\n        p=c/tot; q=(C.get(w,0)+1)/(n+k); s-=p*np.log(q)\n    return s\nS=counts(sel); R=counts(rnd)\nHt=xent(T,T)\nprint(f'target self-entropy      {Ht:.4f}')\nprint(f'xent(target||selected)   {xent(T,S):.4f}  KL={xent(T,S)-Ht:.4f}')\nprint(f'xent(target||random)     {xent(T,R):.4f}  KL={xent(T,R)-Ht:.4f}')\ncov=lambda C: sum(T[w] for w in T if C.get(w,0)>0)/sum(T.values())\nprint(f'target token coverage: selected {cov(S):.4f}  random {cov(R):.4f}')\n\" 2>&1 | grep -v Warning",
  "description": "Measure vocabulary divergence",
  "timeout": 1500000
}

Bash result

target self-entropy      7.4920
xent(target||selected)   7.7784  KL=0.2864
xent(target||random)     7.9435  KL=0.4515
target token coverage: selected 0.9795  random 0.9753

[stdout]
target self-entropy      7.4920
xent(target||selected)   7.7784  KL=0.2864
xent(target||random)     7.9435  KL=0.4515
target token coverage: selected 0.9795  random 0.9753
[metadata] {"noOutputExpected": false}

#44

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolWrite

args

{
  "file_path": "/workspace/claim.md",
  "content": "# Claim\n\n## Hypothesis\n\nUnder a token budget far smaller than the pool (12M of ~200M), held-out perplexity\non a broad multi-domain target is governed less by generic \"web page quality\" than\nby **distributional match to the target, held in the target's own domain proportions**.\n\nConcretely, I claim two things:\n\n1. **Direction.** Ranking pool documents by the log-odds that they came from the\n   target distribution rather than the raw pool — a bag-of-words discriminator whose\n   positives are recovered by decoding the disclosed dev target back to text — beats\n   random selection by a large margin.\n2. **Mixture.** Ranking by a *single* pooled target-likeness score is materially worse\n   than ranking within each target register separately and then filling the budget in\n   fixed per-register token shares. A single score collapses onto whichever register is\n   most lexically separable from generic web text (here: news), starving the others.\n\nThe second point is the substantive one. A pooled discriminator is a *quality* filter;\nper-register quotas make it a *coverage* filter, and coverage is what a broad target rewards.\n\n## Mechanism — and a prediction that is not the final perplexity\n\nThe mechanism is unigram/register distribution matching, not document-level \"goodness.\"\nIf that is what is happening, it should be visible in the **selected corpus itself, before\nany model is trained**. The falsifiable prediction:\n\n> The unigram KL divergence from the target to the selected 12M-token corpus will be\n> substantially lower than from the target to a random 12M-token corpus, and the\n> *relative* reduction in KL will be of the same order as the relative reduction in\n> held-out perplexity.\n\nMeasured, on the packed 12M-token corpora (add-1 smoothed, over the shared normalizer):\n\n| corpus | cross-entropy to target | KL to target |\n|---|---|---|\n| target itself (floor) | 7.4920 | 0 |\n| **curated selection** | 7.7784 | **0.2864** |\n| random selection | 7.9435 | 0.4515 |\n\nKL falls by **37%**. Held-out dev perplexity falls by **34%** (459.8 → 304.1). The two\ntrack each other closely, which is what the mechanism predicts and what a\n\"we just found generically better-written documents\" explanation does not require.\n\nNote that raw target-token *coverage* barely moves (0.9753 → 0.9795). The gain is not\nfrom seeing rarer words; it is from getting the **mass** on common words right. That\ndistinguishes this mechanism from a vocabulary-coverage story.\n\nA second, structural observable: no pool document carries the target's wikitext surface\nartifacts (`@-@`, detokenized `\" ,\"` spacing) — 0 of 182,016 documents contain `@-@`.\nSo none of the gain can be surface-format mimicry; the discriminator is forced onto\ncontent, which is why both sides are pushed through the same aggressive normalizer.\n\n## Falsification\n\nThe hypothesis is falsified if any of these had come out differently — each was run:\n\n- **Balancing does nothing.** Falsified if the pooled single-score ranking matched the\n  register-balanced one. It did not: **333.9 → 317.3** simply by splitting the target\n  into wiki / Q&A / news+web and filling proportional shares. (Remaining gain to 304.1\n  came from an English/non-Latin penalty and a document-length floor.)\n- **Selection is just noise.** Falsified if the metric were flat across configurations.\n  It is not, and it is *non-monotone* in the document-length floor — 40 → 314.2,\n  100 → **304.1**, 300 → 316.6 — an interior optimum, which random variation would not\n  produce.\n- **Sharper ranking is always better.** Falsified — and this one **did** partly falsify\n  a sub-claim of mine. Adding word bigrams raised balanced classifier accuracy on every\n  register (0.9924→0.9955 wiki, 0.9886→0.9959 news/web) yet made perplexity *worse*\n  (304.1 → 308.2). Better discrimination is not better selection.\n- **Match the target mixture exactly.** Falsified. Using the measured dev token shares\n  (wiki .308 / Q&A .229 / news+web .463) scored **314.0**, worse than the flatter\n  .25/.25/.50. Going deeper into the wiki ranking costs more in document quality than\n  proportional fidelity buys. The optimum is a *compromise* between mixture-match and\n  rank-depth, not mixture-match alone.\n- **Top-k is over-concentrated.** Not supported. Gumbel-top-k sampling at TEMP=1.0,\n  intended to broaden topical coverage, scored **317.1** — strictly worse. The ranking's\n  head is genuinely informative, not a narrow-topic artifact.\n\nThe cleanest single falsifier of the whole approach would be a curated selection whose\nunigram KL to the target is lower than random's while its perplexity is not — that would\nsever the claimed mechanism from the outcome. It did not occur here.\n\n## Transfer\n\n- **Recovering positives from the eval artifact generalizes.** The dev target was\n  supplied only as GPT-2 token ids; decoding it back to text turned the evaluation\n  artifact into 2,645 labeled positives at zero annotation cost. Any benchmark that\n  ships a tokenized dev set ships a training signal for its own data selector.\n- **Register quotas over pooled quality scores.** Whenever the evaluation target is a\n  mixture and the training budget is small, a single quality score will over-serve the\n  most lexically distinctive component. Split the target, rank within component, and\n  quota the budget. This is the transferable design rule; it needs no labels beyond the\n  target sample itself.\n- **Normalize away anything the pool cannot supply.** The wikitext artifacts were the\n  most discriminative features available and were entirely useless — worse than useless,\n  since they would have made the classifier rank by a cue with zero support in the pool.\n  Check whether a discriminative feature is *reachable* in the candidate set before\n  letting a selector use it.\n- **Scope / limits.** Everything here is measured at one model size (30M), one budget\n  (12M tokens), and one recipe. The interior optimum in length floor and the flat-vs-exact\n  mixture result are budget-dependent: as the budget approaches the pool size, rank depth\n  stops being scarce and exact mixture-matching should win. The bigram result is likewise\n  a small-budget phenomenon — with more data to select from, sharper discrimination should\n  pay off rather than cost. I did not test either boundary.\n- **Selection differences here are single-seed.** Each configuration was trained once with\n  the frozen script's fixed seed, so run-to-run variance is not separately estimated; the\n  ~10-point spreads are treated as signal on the strength of the non-monotone pattern, not\n  on replication.\n"
}

Write result

File created successfully at: /workspace/claim.md (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/workspace/claim.md", "content": "# Claim\n\n## Hypothesis\n\nUnder a token budget far smaller than the pool (12M of ~200M), held-out perplexity\non a broad multi-domain target is governed less by generic \"web page quality\" than\nby **distributional match to the target, held in the target's own domain proportions**.\n\nConcretely, I claim two things:\n\n1. **Direction.** Ranking pool documents by the log-odds that they came from the\n   target distribution rather than the raw pool — a bag-of-words discriminator whose\n   positives are recovered by decoding the disclosed dev target back to text — beats\n   random selection by a large margin.\n2. **Mixture.** Ranking by a *single* pooled target-likeness score is materially worse\n   than ranking within each target register separately and then filling the budget in\n   fixed per-register token shares. A single score collapses onto whichever register is\n   most lexically separable from generic web text (here: news), starving the others.\n\nThe second point is the substantive one. A pooled discriminator is a *quality* filter;\nper-register quotas make it a *coverage* filter, and coverage is what a broad target rewards.\n\n## Mechanism — and a prediction that is not the final perplexity\n\nThe mechanism is unigram/register distribution matching, not document-level \"goodness.\"\nIf that is what is happening, it should be visible in the **selected corpus itself, before\nany model is trained**. The falsifiable prediction:\n\n> The unigram KL divergence from the target to the selected 12M-token corpus will be\n> substantially lower than from the target to a random 12M-token corpus, and the\n> *relative* reduction in KL will be of the same order as the relative reduction in\n> held-out perplexity.\n\nMeasured, on the packed 12M-token corpora (add-1 smoothed, over the shared normalizer):\n\n| corpus | cross-entropy to target | KL to target |\n|---|---|---|\n| target itself (floor) | 7.4920 | 0 |\n| **curated selection** | 7.7784 | **0.2864** |\n| random selection | 7.9435 | 0.4515 |\n\nKL falls by **37%**. Held-out dev perplexity falls by **34%** (459.8 → 304.1). The two\ntrack each other closely, which is what the mechanism predicts and what a\n\"we just found generically better-written documents\" explanation does not require.\n\nNote that raw target-token *coverage* barely moves (0.9753 → 0.9795). The gain is not\nfrom seeing rarer words; it is from getting the **mass** on common words right. That\ndistinguishes this mechanism from a vocabulary-coverage story.\n\nA second, structural observable: no pool document carries the target's wikitext surface\nartifacts (`@-@`, detokenized `\" ,\"` spacing) — 0 of 182,016 documents contain `@-@`.\nSo none of the gain can be surface-format mimicry; the discriminator is forced onto\ncontent, which is why both sides are pushed through the same aggressive normalizer.\n\n## Falsification\n\nThe hypothesis is falsified if any of these had come out differently — each was run:\n\n- **Balancing does nothing.** Falsified if the pooled single-score ranking matched the\n  register-balanced one. It did not: **333.9 → 317.3** simply by splitting the target\n  into wiki / Q&A / news+web and filling proportional shares. (Remaining gain to 304.1\n  came from an English/non-Latin penalty and a document-length floor.)\n- **Selection is just noise.** Falsified if the metric were flat across configurations.\n  It is not, and it is *non-monotone* in the document-length floor — 40 → 314.2,\n  100 → **304.1**, 300 → 316.6 — an interior optimum, which random variation would not\n  produce.\n- **Sharper ranking is always better.** Falsified — and this one **did** partly falsify\n  a sub-claim of mine. Adding word bigrams raised balanced classifier accuracy on every\n  register (0.9924→0.9955 wiki, 0.9886→0.9959 news/web) yet made perplexity *worse*\n  (304.1 → 308.2). Better discrimination is not better selection.\n- **Match the target mixture exactly.** Falsified. Using the measured dev token shares\n  (wiki .308 / Q&A .229 / news+web .463) scored **314.0**, worse than the flatter\n  .25/.25/.50. Going deeper into the wiki ranking costs more in document quality than\n  proportional fidelity buys. The optimum is a *compromise* between mixture-match and\n  rank-depth, not mixture-match alone.\n- **Top-k is over-concentrated.** Not supported. Gumbel-top-k sampling at TEMP=1.0,\n  intended to broaden topical coverage, scored **317.1** — strictly worse. The ranking's\n  head is genuinely informative, not a narrow-topic artifact.\n\nThe cleanest single falsifier of the whole approach would be a curated selection whose\nunigram KL to the target is lower than random's while its perplexity is not — that would\nsever the claimed mechanism from the outcome. It did not occur here.\n\n## Transfer\n\n- **Recovering positives from the eval artifact generalizes.** The dev target was\n  supplied only as GPT-2 token ids; decoding it back to text turned the evaluation\n  artifact into 2,645 labeled positives at zero annotation cost. Any benchmark that\n  ships a tokenized dev set ships a training signal for its own data selector.\n- **Register quotas over pooled quality scores.** Whenever the evaluation target is a\n  mixture and the training budget is small, a single quality score will over-serve the\n  most lexically distinctive component. Split the target, rank within component, and\n  quota the budget. This is the transferable design rule; it needs no labels beyond the\n  target sample itself.\n- **Normalize away anything the pool cannot supply.** The wikitext artifacts were the\n  most discriminative features available and were entirely useless — worse than useless,\n  since they would have made the classifier rank by a cue with zero support in the pool.\n  Check whether a discriminative feature is *reachable* in the candidate set before\n  letting a selector use it.\n- **Scope / limits.** Everything here is measured at one model size (30M), one budget\n  (12M tokens), and one recipe. The interior optimum in length floor and the flat-vs-exact\n  mixture result are budget-dependent: as the budget approaches the pool size, rank depth\n  stops being scarce and exact mixture-matching should win. The bigram result is likewise\n  a small-budget phenomenon — with more data to select from, sharper discrimination should\n  pay off rather than cost. I did not test either boundary.\n- **Selection differences here are single-seed.** Each configuration was trained once with\n  the frozen script's fixed seed, so run-to-run variance is not separately estimated; the\n  ~10-point spreads are treated as signal on the strength of the non-monotone pattern, not\n  on replication.\n", "structuredPatch": [], "originalFile": null, "userModified": false}

#45

narration

Final confirmation run on the submitted selection:

toolBashFinal verification

args

{
  "command": "bash dev_eval.sh 2>&1 | grep -E \"packed|HELDOUT\"; echo \"---\"; python -c \"\nimport json; s=json.load(open('/workspace/submission/selection.json'))\nassert len(s)==len(set(s)) and all(isinstance(i,int) and 0<=i<182016 for i in s)\nprint('VALID: %d unique ids, all in range'%len(s))\n\"; ls -la /workspace/submission /workspace/claim.md; cat .timer/remaining_secs",
  "description": "Final verification",
  "timeout": 1500000
}

Bash result

packed 12000000 tokens from 42287 ids -> /tmp/dev_train.npy
HELDOUT_PPL 304.127  loss 5.7174  train_s 54  /tmp/dev_train.npy
---
VALID: 42287 unique ids, all in range
-rw-r--r-- 1 root root 6626 Aug  4 02:05 /workspace/claim.md

/workspace/submission:
total 307
drwxr-xr-x 1 root root    300 Aug  4 01:30 .
drwxr-xr-x 1 root root    180 Aug  4 02:05 ..
drwxr-xr-x 1 root root    100 Aug  4 02:04 __pycache__
-rw-r--r-- 1 root root  14557 Aug  4 01:49 curate.py
-rw-r--r-- 1 root root 299198 Aug  4 02:00 selection.json
1986

[stdout]
packed 12000000 tokens from 42287 ids -> /tmp/dev_train.npy
HELDOUT_PPL 304.127  loss 5.7174  train_s 54  /tmp/dev_train.npy
---
VALID: 42287 unique ids, all in range
-rw-r--r-- 1 root root 6626 Aug  4 02:05 /workspace/claim.md

/workspace/submission:
total 307
drwxr-xr-x 1 root root    300 Aug  4 01:30 .
drwxr-xr-x 1 root root    180 Aug  4 02:05 ..
drwxr-xr-x 1 root root    100 Aug  4 02:04 __pycache__
-rw-r--r-- 1 root root  14557 Aug  4 01:49 curate.py
-rw-r--r-- 1 root root 299198 Aug  4 02:00 selection.json
1986
[metadata] {"noOutputExpected": false}

#46

narration

Solid. I have ~30 min left — bracketing the length-floor optimum for a possible extra gain:

toolBashExperiment: min length 150

args

{
  "command": "rm -rf submission/__pycache__; MINLEN=150 python submission/curate.py 2>&1|tail -1; cp submission/selection.json /tmp/sel_ml150.json; bash dev_eval.sh 2>&1|grep HELDOUT",
  "description": "Experiment: min length 150",
  "timeout": 1500000
}

Bash result

wrote 36384 ids -> /workspace/submission/selection.json
HELDOUT_PPL 313.142  loss 5.7467  train_s 54  /tmp/dev_train.npy

[stdout]
wrote 36384 ids -> /workspace/submission/selection.json
HELDOUT_PPL 313.142  loss 5.7467  train_s 54  /tmp/dev_train.npy
[metadata] {"noOutputExpected": false}

#47

narration

MINLEN=150 → 313.1. The floor at 100 is a sharp optimum. Restoring the best configuration:

toolBashRestore and final verify

args

{
  "command": "python submission/curate.py 2>&1|tail -1\npython -c \"\nimport json\na=json.load(open('/workspace/submission/selection.json'))\nprint('reproduces best run:', a==json.load(open('/tmp/sel_v3.json')))\nprint('VALID:',len(a),'unique',len(set(a))==len(a),'range ok',min(a)>=0 and max(a)<182016)\n\"\nrm -rf submission/__pycache__; bash dev_eval.sh 2>&1|grep -E \"packed|HELDOUT\"; cat .timer/remaining_secs",
  "description": "Restore and final verify",
  "timeout": 1500000
}

Bash result

wrote 42287 ids -> /workspace/submission/selection.json
reproduces best run: True
VALID: 42287 unique True range ok True
packed 12000000 tokens from 42287 ids -> /tmp/dev_train.npy
HELDOUT_PPL 304.127  loss 5.7174  train_s 54  /tmp/dev_train.npy
1475

[stdout]
wrote 42287 ids -> /workspace/submission/selection.json
reproduces best run: True
VALID: 42287 unique True range ok True
packed 12000000 tokens from 42287 ids -> /tmp/dev_train.npy
HELDOUT_PPL 304.127  loss 5.7174  train_s 54  /tmp/dev_train.npy
1475
[metadata] {"noOutputExpected": false}

#48

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolEdit

args

{
  "replace_all": false,
  "file_path": "/workspace/claim.md",
  "old_string": "  It is not, and it is *non-monotone* in the document-length floor — 40 → 314.2,\n  100 → **304.1**, 300 → 316.6 — an interior optimum, which random variation would not\n  produce.",
  "new_string": "  It is not, and it is *non-monotone* in the document-length floor — 40 → 314.2,\n  100 → **304.1**, 150 → 313.1, 300 → 316.6 — an interior optimum, which random variation\n  would not produce. The optimum is however sharp (±50 tokens costs ~9 perplexity), which\n  is the main reason I flag the single-seed caveat below: some of the margin between 304\n  and the ~313–317 cluster may be selection luck on this particular dev sample rather than\n  a property that transfers to the hidden one."
}

Edit result

The file /workspace/claim.md has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/claim.md", "oldString": "  It is not, and it is *non-monotone* in the document-length floor — 40 → 314.2,\n  100 → **304.1**, 300 → 316.6 — an interior optimum, which random variation would not\n  produce.", "newString": "  It is not, and it is *non-monotone* in the document-length floor — 40 → 314.2,\n  100 → **304.1**, 150 → 313.1, 300 → 316.6 — an interior optimum, which random variation\n  would not produce. The optimum is however sharp (±50 tokens costs ~9 perplexity), which\n  is the main reason I flag the single-seed caveat below: some of the margin between 304\n  and the ~313–317 cluster may be selection luck on this particular dev sample rather than\n  a property that transfers to the hidden one.", "originalFile": "# Claim\n\n## Hypothesis\n\nUnder a token budget far smaller than the pool (12M of ~200M), held-out perplexity\non a broad multi-domain target is governed less by generic \"web page quality\" than\nby **distributional match to the target, held in the target's own domain proportions**.\n\nConcretely, I claim two things:\n\n1. **Direction.** Ranking pool documents by the log-odds that they came from the\n   target distribution rather than the raw pool — a bag-of-words discriminator whose\n   positives are recovered by decoding the disclosed dev target back to text — beats\n   random selection by a large margin.\n2. **Mixture.** Ranking by a *single* pooled target-likeness score is materially worse\n   than ranking within each target register separately and then filling the budget in\n   fixed per-register token shares. A single score collapses onto whichever register is\n   most lexically separable from generic web text (here: news), starving the others.\n\nThe second point is the substantive one. A pooled discriminator is a *quality* filter;\nper-register quotas make it a *coverage* filter, and coverage is what a broad target rewards.\n\n## Mechanism — and a prediction that is not the final perplexity\n\nThe mechanism is unigram/register distribution matching, not document-level \"goodness.\"\nIf that is what is happening, it should be visible in the **selected corpus itself, before\nany model is trained**. The falsifiable prediction:\n\n> The unigram KL divergence from the target to the selected 12M-token corpus will be\n> substantially lower than from the target to a random 12M-token corpus, and the\n> *relative* reduction in KL will be of the same order as the relative reduction in\n> held-out perplexity.\n\nMeasured, on the packed 12M-token corpora (add-1 smoothed, over the shared normalizer):\n\n| corpus | cross-entropy to target | KL to target |\n|---|---|---|\n| target itself (floor) | 7.4920 | 0 |\n| **curated selection** | 7.7784 | **0.2864** |\n| random selection | 7.9435 | 0.4515 |\n\nKL falls by **37%**. Held-out dev perplexity falls by **34%** (459.8 → 304.1). The two\ntrack each other closely, which is what the mechanism predicts and what a\n\"we just found generically better-written documents\" explanation does not require.\n\nNote that raw target-token *coverage* barely moves (0.9753 → 0.9795). The gain is not\nfrom seeing rarer words; it is from getting the **mass** on common words right. That\ndistinguishes this mechanism from a vocabulary-coverage story.\n\nA second, structural observable: no pool document carries the target's wikitext surface\nartifacts (`@-@`, detokenized `\" ,\"` spacing) — 0 of 182,016 documents contain `@-@`.\nSo none of the gain can be surface-format mimicry; the discriminator is forced onto\ncontent, which is why both sides are pushed through the same aggressive normalizer.\n\n## Falsification\n\nThe hypothesis is falsified if any of these had come out differently — each was run:\n\n- **Balancing does nothing.** Falsified if the pooled single-score ranking matched the\n  register-balanced one. It did not: **333.9 → 317.3** simply by splitting the target\n  into wiki / Q&A / news+web and filling proportional shares. (Remaining gain to 304.1\n  came from an English/non-Latin penalty and a document-length floor.)\n- **Selection is just noise.** Falsified if the metric were flat across configurations.\n  It is not, and it is *non-monotone* in the document-length floor — 40 → 314.2,\n  100 → **304.1**, 300 → 316.6 — an interior optimum, which random variation would not\n  produce.\n- **Sharper ranking is always better.** Falsified — and this one **did** partly falsify\n  a sub-claim of mine. Adding word bigrams raised balanced classifier accuracy on every\n  register (0.9924→0.9955 wiki, 0.9886→0.9959 news/web) yet made perplexity *worse*\n  (304.1 → 308.2). Better discrimination is not better selection.\n- **Match the target mixture exactly.** Falsified. Using the measured dev token shares\n  (wiki .308 / Q&A .229 / news+web .463) scored **314.0**, worse than the flatter\n  .25/.25/.50. Going deeper into the wiki ranking costs more in document quality than\n  proportional fidelity buys. The optimum is a *compromise* between mixture-match and\n  rank-depth, not mixture-match alone.\n- **Top-k is over-concentrated.** Not supported. Gumbel-top-k sampling at TEMP=1.0,\n  intended to broaden topical coverage, scored **317.1** — strictly worse. The ranking's\n  head is genuinely informative, not a narrow-topic artifact.\n\nThe cleanest single falsifier of the whole approach would be a curated selection whose\nunigram KL to the target is lower than random's while its perplexity is not — that would\nsever the claimed mechanism from the outcome. It did not occur here.\n\n## Transfer\n\n- **Recovering positives from the eval artifact generalizes.** The dev target was\n  supplied only as GPT-2 token ids; decoding it back to text turned the evaluation\n  artifact into 2,645 labeled positives at zero annotation cost. Any benchmark that\n  ships a tokenized dev set ships a training signal for its own data selector.\n- **Register quotas over pooled quality scores.** Whenever the evaluation target is a\n  mixture and the training budget is small, a single quality score will over-serve the\n  most lexically distinctive component. Split the target, rank within component, and\n  quota the budget. This is the transferable design rule; it needs no labels beyond the\n  target sample itself.\n- **Normalize away anything the pool cannot supply.** The wikitext artifacts were the\n  most discriminative features available and were entirely useless — worse than useless,\n  since they would have made the classifier rank by a cue with zero support in the pool.\n  Check whether a discriminative feature is *reachable* in the candidate set before\n  letting a selector use it.\n- **Scope / limits.** Everything here is measured at one model size (30M), one budget\n  (12M tokens), and one recipe. The interior optimum in length floor and the flat-vs-exact\n  mixture result are budget-dependent: as the budget approaches the pool size, rank depth\n  stops being scarce and exact mixture-matching should win. The bigram result is likewise\n  a small-budget phenomenon — with more data to select from, sharper discrimination should\n  pay off rather than cost. I did not test either boundary.\n- **Selection differences here are single-seed.** Each configuration was trained once with\n  the frozen script's fixed seed, so run-to-run variance is not separately estimated; the\n  ~10-point spreads are treated as signal on the strength of the non-monotone pattern, not\n  on replication.\n", "structuredPatch": [{"oldStart": 62, "oldLines": 8, "newStart": 62, "newLines": 11, "lines": ["   came from an English/non-Latin penalty and a document-length floor.)", " - **Selection is just noise.** Falsified if the metric were flat across configurations.", "   It is not, and it is *non-monotone* in the document-length floor — 40 → 314.2,", "-  100 → **304.1**, 300 → 316.6 — an interior optimum, which random variation would not", "-  produce.", "+  100 → **304.1**, 150 → 313.1, 300 → 316.6 — an interior optimum, which random variation", "+  would not produce. The optimum is however sharp (±50 tokens costs ~9 perplexity), which", "+  is the main reason I flag the single-seed caveat below: some of the margin between 304", "+  and the ~313–317 cluster may be selection luck on this particular dev sample rather than", "+  a property that transfers to the hidden one.", " - **Sharper ranking is always better.** Falsified — and this one **did** partly falsify", "   a sub-claim of mine. Adding word bigrams raised balanced classifier accuracy on every", "   register (0.9924→0.9955 wiki, 0.9886→0.9959 news/web) yet made perplexity *worse*"]}], "userModified": false, "replaceAll": false}

final summary

Done. All three deliverables are in place and verified.

Result

selection	dev PPL
random baseline (do-nothing)	459.8
pooled single quality score	333.9
+ register-balanced quotas	317.3
+ English/non-Latin penalty + length floor	304.1

34% below the random baseline.

Approach

The disclosed dev target ships as GPT-2 token ids — decoding it back to text yields 2,645 labeled positives for free. I split those into the target's constituent registers (encyclopedic / technical Q&A / news+web prose, identifiable from wikitext @-@ artifacts and StackExchange HTML tags), trained one hashed bag-of-words logistic regression per register against a random pool sample, and filled the 12M-token budget in fixed per-register shares.

Two things mattered more than the classifier itself:

Register quotas, not a pooled score. A single target-likeness score collapsed onto news, which is the most lexically separable slice from generic web text — the top-ranked documents were near-uniformly South-Asian politics. Splitting and quota-filling was worth 17 perplexity on its own.
Normalizing away unreachable cues. The target's Wikipedia slice carries wikitext formatting that zero of 182,016 pool documents have. That formatting is the single most discriminative feature available and is entirely useless, so both sides go through the same aggressive normalizer.

Things that did not work (all in `claim.md`)

Bigrams raised classifier accuracy on every register yet made perplexity worse (304.1 → 308.2). Better discrimination ≠ better selection.
Matching the dev mixture exactly (.308/.229/.463) scored 314.0, worse than the flatter .25/.25/.50 — going deeper into the wiki ranking costs more in document quality than mixture fidelity buys.
Gumbel-top-k sampling to broaden topics scored 317.1; the ranking head is genuinely informative, not a narrow-topic artifact.

Mechanism check

claim.md predicts an observable other than the final perplexity: unigram KL to the target, measurable on the packed corpus before any training. It falls 0.4515 → 0.2864 (−37%), tracking the −34% perplexity drop. Target token coverage barely moves (0.9753 → 0.9795) — the gain is mass on common words, not rarer vocabulary, which rules out a coverage story.

Two caveats I've recorded rather than smoothed over: the length-floor optimum is sharp (±50 tokens costs ~9 perplexity), and every configuration was trained once with the frozen script's fixed seed, so some of the margin between 304 and the 313–317 cluster may be selection luck on this dev sample rather than something that transfers to the hidden one. The gap to the 459.8 baseline is far too large to be explained that way.

Result

Approach

Things that did not work (all in claim.md)

Mechanism check

Things that did not work (all in `claim.md`)