claude-opus-4-8

Traceback (most recent call last):
  File "/tmp/make_ablation.py", line 19, in <module>
    with Pool(16) as p: rcount=sum(p.map(cur.count_chunk,[sample[i::16] for i in range(16)]))
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 367, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 774, in get
    raise self._value
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 540, in _handle_tasks
    put(task)
  File "/usr/lib/python3.10/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/usr/lib/python3.10/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function count_chunk at 0x2b59016ba320>: import of module 'cur' failed

[stdout]
Traceback (most recent call last):
  File "/tmp/make_ablation.py", line 19, in <module>
    with Pool(16) as p: rcount=sum(p.map(cur.count_chunk,[sample[i::16] for i in range(16)]))
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 367, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 774, in get
    raise self._value
  File "/usr/lib/python3.10/multiprocessing/pool.py", line 540, in _handle_tasks
    put(task)
  File "/usr/lib/python3.10/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/usr/lib/python3.10/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function count_chunk at 0x2b59016ba320>: import of module 'cur' failed
[metadata] {"noOutputExpected": false}

# Ablation: plain combined-DSIR top-k (NO register stratification). Same cap+dedup as the
# real pipeline; only difference is a single combined target distribution + global top-k.
import json, re, zlib, numpy as np
from multiprocessing import Pool
import sys; sys.path.insert(0,'/workspace/submission')
import curate as cur
from transformers import AutoTokenizer
tok=AutoTokenizer.from_pretrained('gpt2')
--- run ---
ablation selected 40321 est_tokens 32.0 M
top-12M-est portion: 15755 docs, tech_frac=0.159

[stdout]
# Ablation: plain combined-DSIR top-k (NO register stratification). Same cap+dedup as the
# real pipeline; only difference is a single combined target distribution + global top-k.
import json, re, zlib, numpy as np
from multiprocessing import Pool
import sys; sys.path.insert(0,'/workspace/submission')
import curate as cur
from transformers import AutoTokenizer
tok=AutoTokenizer.from_pretrained('gpt2')
--- run ---
ablation selected 40321 est_tokens 32.0 M
top-12M-est portion: 15755 docs, tech_frac=0.159
[metadata] {"noOutputExpected": false}

=== baseline ===
HELDOUT_PPL 485.420  loss 6.1850  train_s 54  /tmp/dev_train.npy
HELDOUT_PPL 485.420  loss 6.1850  train_s 54  /tmp/dev_train.npy
(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)
timer:
4123

[stdout]
=== baseline ===
HELDOUT_PPL 485.420  loss 6.1850  train_s 54  /tmp/dev_train.npy
HELDOUT_PPL 485.420  loss 6.1850  train_s 54  /tmp/dev_train.npy
(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)
timer:
4123
[metadata] {"noOutputExpected": false}

launched ablation PID 2293

[stdout]
launched ablation PID 2293
[metadata] {"noOutputExpected": false}

pool raw tech fraction: 0.396
submission/selection.json: top-12M-est docs=16401 tech_frac=0.249
/tmp/ablation_sel.json: top-12M-est docs=15755 tech_frac=0.159
/tmp/baseline_sel.json: top-12M-est docs=11410 tech_frac=0.396

[stdout]
pool raw tech fraction: 0.396
submission/selection.json: top-12M-est docs=16401 tech_frac=0.249
/tmp/ablation_sel.json: top-12M-est docs=15755 tech_frac=0.159
/tmp/baseline_sel.json: top-12M-est docs=11410 tech_frac=0.396
[metadata] {"noOutputExpected": false}

=== ablation ===
HELDOUT_PPL 402.112  loss 5.9967  train_s 54  /tmp/dev_train.npy
(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)
timer:
3972

[stdout]
=== ablation ===
HELDOUT_PPL 402.112  loss 5.9967  train_s 54  /tmp/dev_train.npy
(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)
timer:
3972
[metadata] {"noOutputExpected": false}

Updated task #2 status

[metadata] {"success": true, "taskId": "2", "updatedFields": ["status"], "statusChange": {"from": "in_progress", "to": "completed"}}

Updated task #3 status

[metadata] {"success": true, "taskId": "3", "updatedFields": ["status"], "statusChange": {"from": "pending", "to": "in_progress"}}

total ids: 41768
unique: 41768
all ints: True
id range: 0 181997 | in-range(0..182015): True

[stdout]
total ids: 41768
unique: 41768
all ints: True
id range: 0 181997 | in-range(0..182015): True
[metadata] {"noOutputExpected": false}

File created successfully at: /workspace/claim.md (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/workspace/claim.md", "content": "# Claim: register-mixture matching beats quality ranking for fixed-budget curation\n\n## Hypothesis\nWhen the disclosed target is an **equal-parts mixture of registers** (Wikipedia,\nhigh-quality web prose, news, technical Q&A) and the score is **mean held-out\nnext-token cross-entropy**, the 12M-token training budget that minimizes perplexity\nis the one whose **register mixture matches the target's**, with each register filled\nby the pool documents most similar to that register (DSIR importance weight = target-vs-\nraw-pool n-gram log-likelihood ratio). Matching the mixture should beat both (a) random\nselection and (b) ranking every document by a single combined quality/similarity score,\nbecause a single score collapses the budget onto whichever register has the largest\ntarget-vs-pool vocabulary ratio and starves the rest — and the mean loss is dominated by\nthe worst-covered register (here: technical Q&A / code, whose tokens are otherwise unseen).\n\n## Mechanism — observable predicted *before* the official perplexity\nThe lever is the **technical-register token fraction** of the 12M-token budget. The\ntarget is ~25% technical; the raw pool is ~40% technical; a single combined-DSIR ranking\ncollapses to ~16% technical (it favors news, whose vocabulary is rarer in the raw pool).\nPrediction: pushing that fraction toward the target's 25% — from *either* side — lowers\nheld-out loss, and a selection pinned at 25% beats both the pool's 40% and the collapsed\n16%. Measured on the frozen recipe (identical 30M GPT, 12M tokens, dev target):\n\n| Selection                         | technical fraction | held-out loss | perplexity |\n|-----------------------------------|:-----:|:-----:|:-----:|\n| Random pool sample (baseline)     | 0.40  | 6.185 | 485.4  |\n| Combined-DSIR top-k (no strata)   | 0.16  | 5.997 | 402.1  |\n| **Register-stratified (this run)**| **0.25** | **5.804** | **331.5** |\n\nOrdering confirmed: random > combined-top-k > stratified. Importance ranking alone buys\n485→402; **matching the register mixture buys a further 402→331** (−0.19 nats), the larger\nshare of the total −0.38-nat gain. The predicted observable (technical fraction → 0.25)\ntracks the loss, independent of the final number.\n\n## Falsification\nThe claim is falsified if, under the frozen recipe, **combined-DSIR top-k reaches\nheld-out perplexity ≤ the stratified selection** — i.e. mixture-matching adds nothing over\nplain quality ranking. It is further falsified if forcing the technical fraction to the\ntarget's 25% does *not* beat the pool-natural 40% and the collapsed 16% (e.g. if a\nprose-only or tech-heavy budget scored as low). Neither held: 402.1 and 485.4 are both\nclearly above 331.5, and 0.25 beats 0.40 and 0.16. A stronger single-run falsifier: a\nrandom selection matching the stratified perplexity would sink the entire importance-\nselection premise.\n\n## Transfer\nThe recipe needs only a small target sample and one CPU n-gram pass — **no labels, no\nreference LM, no GPU** — and generalizes to any fixed-budget pretraining curation against\na known or estimable multi-register target:\n1. estimate the target's register proportions (here: a code-density split auto-calibrated\n   to the target's own 75th percentile → ~25% technical);\n2. allocate the budget to those proportions;\n3. within each register, rank pool documents by the DSIR importance weight\n   (target-register vs raw-pool hashed uni/bi-gram log-likelihood ratio);\n4. interleave best-first (round-robin at the target ratio) so that truncating at the token\n   budget preserves the mixture.\nIt scales to web-scale pools (hashed n-grams, streaming) and to more registers or other\ntarget compositions by changing only the strata and their quotas. The one thing selection\n*cannot* fix is target-specific surface formatting absent from the pool (the target's\nWikiText spaced punctuation / `@,@` / `@-@` markers occur in zero pool documents), which\nsets a perplexity floor no data choice can lower — so matching is done on content/register,\nnot formatting.\n", "structuredPatch": [], "originalFile": null, "userModified": false}

1	"""Curate a 12M-token pretraining set from a raw web pool by register-stratified DSIR.
2	
3	Criterion (fully reproducible, no hand-picking)
4	-----------------------------------------------
5	The held-out score is perplexity on a BROAD target of *equal parts* four registers:
6	Wikipedia, high-quality web prose, news, and technical Q&A.  Held-out loss is the MEAN
7	next-token CE across those registers, so the training mixture that minimizes it is the
8	target's own mixture -- and the register whose absence hurts most is the distinctive one:
9	technical Q&A (code + HTML), whose tokens are otherwise almost never seen.
10	
11	A single combined importance score does NOT preserve the mixture: it collapses onto the
12	register with the largest target-vs-pool vocabulary ratio (empirically: news) and starves
13	the rest.  So we stratify by register and match the target's proportions:
14	
15	  1. Split docs into TECH (code/markup) vs PROSE by code-character density.  The threshold
16	     is auto-calibrated to the target itself (its 75th percentile), so ~1/4 of the target
17	     is TECH -- i.e. the disclosed "equal parts" fraction.
18	  2. Estimate a DSIR importance weight per stratum s:
19	        W_s[f] = log p_target_s(f) - log p_rawpool(f)
20	     (add-alpha-smoothed hashed uni/bi-gram distributions; raw from a pool sample).
21	  3. Rank pool docs WITHIN their stratum by average per-token log-ratio (a quality/domain
22	     score); a light language prefilter drops non-prose junk (relaxed for TECH so genuine
23	     code Q&A survives).
24	  4. Fill the budget by round-robin at the target ratio (3 PROSE : 1 TECH), best-first
25	     within each stratum, so the first 12M tokens the packer consumes are register-balanced.
26	
27	Matching is on word vocabulary (content/register); the target's WikiText surface
28	formatting (spaced punctuation, @,@ / @-@) never occurs in the pool and cannot be induced
29	by selection, so it is deliberately ignored.
30	
31	Output: /workspace/submission/selection.json -- pool ids, priority order (best first).
32	"""
33	import json, re, zlib, time, numpy as np
34	from multiprocessing import Pool
35	
36	POOL = "/workspace/data/pool.jsonl"
37	DEV  = "/workspace/data/multi_dev.npy"
38	OUT  = "/workspace/submission/selection.json"
39	
40	NBITS      = 18
41	NBUCKETS   = 1 << NBITS
42	MASK       = NBUCKETS - 1
43	ALPHA      = 0.1
44	RAW_SAMPLE = 40000
45	SEED       = 1337
46	COVER_TOK  = 32_000_000        # ~2.7x the 12M budget
47	TECH_RATIO = 0.25              # target's technical-Q&A share (equal-parts -> 1/4)
48	MAX_CHARS  = 15000             # target docs are short; longer pool docs are dumps/nav bloat
49	MIN_WORDS  = 50
50	MIN_ALPHA  = 0.50
51	MIN_STOP   = 0.10
52	# tech (code/Q&A) prefilter: relaxed, but must contain real English prose too
53	TECH_MINW  = 30
54	TECH_MINSTOP = 0.03
55	TECH_MAXCP = 0.45              # reject minified/base64 junk (too code-dense)
56	WORD_RE    = re.compile(r"[a-z]+")
57	CODE_CH    = set("{}[]<>;=()/$_|`\\\t")
58	
59	
60	def words(text):
61	    return WORD_RE.findall(text.lower())
62	
63	
64	def buckets(ws):
65	    b = [zlib.crc32(w.encode()) & MASK for w in ws]
66	    for i in range(len(ws) - 1):
67	        b.append(zlib.crc32((ws[i] + " " + ws[i + 1]).encode()) & MASK)
68	    return b
69	
70	
71	def code_density(text):
72	    if not text:
73	        return 0.0
74	    return sum(1 for ch in text if ch in CODE_CH) / len(text)
75	
76	
77	def count_chunk(texts):
78	    c = np.zeros(NBUCKETS, dtype=np.float64)
79	    for t in texts:
80	        for h in buckets(words(t)):
81	            c[h] += 1.0
82	    return c
83	
84	
85	# globals shared with workers via fork
86	_WT = None   # tech weights
87	_WP = None   # prose weights
88	_TH = None   # tech/prose code-density threshold
89	_DOCS = None
90	
91	
92	def _init(wt, wp, th, docs):
93	    global _WT, _WP, _TH, _DOCS
94	    _WT, _WP, _TH, _DOCS = wt, wp, th, docs
95	
96	
97	def score_range(rng):
98	    """Return (id, stratum, score, est_tokens); stratum: 0=prose,1=tech,-1=drop."""
99	    s, e = rng
100	    out = []
101	    for k in range(s, e):
102	        did, text = _DOCS[k]
103	        est = len(text) / 4.0
104	        if len(text) > MAX_CHARS:
105	            out.append((did, -1, -1e30, est)); continue
106	        ws = words(text)
107	        nw = len(ws)
108	        cp = code_density(text)
109	        stop_frac = (sum(1 for w in ws if w in STOP) / nw) if nw else 0.0
110	        if cp > _TH:  # TECH candidate
111	            ok = (nw >= TECH_MINW) and (stop_frac >= TECH_MINSTOP) and (cp <= TECH_MAXCP)
112	            if not ok:
113	                out.append((did, -1, -1e30, est)); continue
114	            bs = np.asarray(buckets(ws))
115	            out.append((did, 1, float(_WT[bs].mean()), est))
116	        else:         # PROSE candidate
117	            alpha = sum(ch.isalpha() for ch in text)
118	            ok = (nw >= MIN_WORDS) and (alpha >= MIN_ALPHA * max(1, len(text))) and (stop_frac >= MIN_STOP)
119	            if not ok:
120	                out.append((did, -1, -1e30, est)); continue
121	            bs = np.asarray(buckets(ws))
122	            out.append((did, 0, float(_WP[bs].mean()), est))
123	    return out
124	
125	
126	STOP = set("the a an and or but of to in on at for with by from as is are was were be been "
127	           "being this that these those it its he she they we you i his her their our your "
128	           "not no do does did have has had will would can could should may might must".split())
129	
130	
131	def main():
132	    t0 = time.time()
133	    from transformers import AutoTokenizer
134	    tok = AutoTokenizer.from_pretrained("gpt2")
135	    dev = np.load(DEV).astype(np.int64)
136	    tdocs = [d for d in tok.decode(dev.tolist()).split("<|endoftext|>") if len(d.split()) >= 10]
137	    cd = np.array([code_density(d) for d in tdocs])
138	    TH = float(np.quantile(cd, 1.0 - TECH_RATIO))       # auto-calibrated split
139	    t_tech = [d for d, c in zip(tdocs, cd) if c > TH]
140	    t_prose = [d for d, c in zip(tdocs, cd) if c <= TH]
141	    print(f"[t={time.time()-t0:.0f}s] target docs={len(tdocs)}  tech_thresh={TH:.4f}  "
142	          f"tech={len(t_tech)} prose={len(t_prose)}")
143	
144	    tcount = count_chunk(t_tech)
145	    pcount = count_chunk(t_prose)
146	
147	    docs = []
148	    with open(POOL) as f:
149	        for line in f:
150	            r = json.loads(line)
151	            docs.append((r["id"], r["text"]))
152	    print(f"[t={time.time()-t0:.0f}s] pool docs: {len(docs)}")
153	
154	    rng = np.random.default_rng(SEED)
155	    idx = rng.choice(len(docs), size=min(RAW_SAMPLE, len(docs)), replace=False)
156	    sample = [docs[i][1] for i in idx]
157	    nproc = 16
158	    with Pool(nproc) as p:
159	        rcount = sum(p.map(count_chunk, [sample[i::nproc] for i in range(nproc)]))
160	    R = rcount.sum()
161	    pr = (rcount + ALPHA) / (R + ALPHA * NBUCKETS)
162	
163	    def wof(counts):
164	        pt = (counts + ALPHA) / (counts.sum() + ALPHA * NBUCKETS)
165	        return np.log(pt) - np.log(pr)
166	    WT, WP = wof(tcount), wof(pcount)
167	    print(f"[t={time.time()-t0:.0f}s] weights ready")
168	
169	    N = len(docs)
170	    step = (N + nproc - 1) // nproc
171	    ranges = [(i, min(i + step, N)) for i in range(0, N, step)]
172	    with Pool(nproc, initializer=_init, initargs=(WT, WP, TH, docs)) as p:
173	        parts = p.map(score_range, ranges)
174	    rows = [r for part in parts for r in part]
175	    prose = sorted([(sc, did, est) for did, st, sc, est in rows if st == 0], reverse=True)
176	    tech  = sorted([(sc, did, est) for did, st, sc, est in rows if st == 1], reverse=True)
177	    print(f"[t={time.time()-t0:.0f}s] scored: prose_cand={len(prose)} tech_cand={len(tech)}")
178	
179	    # round-robin fill at target ratio (per 4 docs: 3 prose, 1 tech), skipping
180	    # near-exact duplicates (normalized-content hash keeps the first occurrence)
181	    id2text = {did: t for did, t in docs}
182	    seen = set()
183	    _ws = re.compile(r"\s+")
184	
185	    def take(pool_list, i):
186	        while i < len(pool_list):
187	            sc, did, est = pool_list[i]; i += 1
188	            key = zlib.crc32(_ws.sub("", id2text[did].lower())[:2000].encode())
189	            if key in seen:
190	                continue
191	            seen.add(key)
192	            return did, est, i
193	        return None, 0.0, i
194	
195	    sel, cum = [], 0.0
196	    pi = ti = 0
197	    ptok = ttok = 0.0
198	    while cum < COVER_TOK and (pi < len(prose) or ti < len(tech)):
199	        for _ in range(3):
200	            did, est, pi = take(prose, pi)
201	            if did is not None:
202	                sel.append(int(did)); ptok += est; cum += est
203	        did, est, ti = take(tech, ti)
204	        if did is not None:
205	            sel.append(int(did)); ttok += est; cum += est
206	    json.dump(sel, open(OUT, "w"))
207	    print(f"[t={time.time()-t0:.0f}s] selected={len(sel)} est_tokens~{cum/1e6:.1f}M "
208	          f"(prose~{ptok/1e6:.1f}M tech~{ttok/1e6:.1f}M)")
209	    print(f"wrote {OUT}")
210	
211	
212	if __name__ == "__main__":
213	    main()
214

[metadata] {"type": "text", "file": {"filePath": "/workspace/submission/curate.py", "content": "\"\"\"Curate a 12M-token pretraining set from a raw web pool by register-stratified DSIR.\n\nCriterion (fully reproducible, no hand-picking)\n-----------------------------------------------\nThe held-out score is perplexity on a BROAD target of *equal parts* four registers:\nWikipedia, high-quality web prose, news, and technical Q&A.  Held-out loss is the MEAN\nnext-token CE across those registers, so the training mixture that minimizes it is the\ntarget's own mixture -- and the register whose absence hurts most is the distinctive one:\ntechnical Q&A (code + HTML), whose tokens are otherwise almost never seen.\n\nA single combined importance score does NOT preserve the mixture: it collapses onto the\nregister with the largest target-vs-pool vocabulary ratio (empirically: news) and starves\nthe rest.  So we stratify by register and match the target's proportions:\n\n  1. Split docs into TECH (code/markup) vs PROSE by code-character density.  The threshold\n     is auto-calibrated to the target itself (its 75th percentile), so ~1/4 of the target\n     is TECH -- i.e. the disclosed \"equal parts\" fraction.\n  2. Estimate a DSIR importance weight per stratum s:\n        W_s[f] = log p_target_s(f) - log p_rawpool(f)\n     (add-alpha-smoothed hashed uni/bi-gram distributions; raw from a pool sample).\n  3. Rank pool docs WITHIN their stratum by average per-token log-ratio (a quality/domain\n     score); a light language prefilter drops non-prose junk (relaxed for TECH so genuine\n     code Q&A survives).\n  4. Fill the budget by round-robin at the target ratio (3 PROSE : 1 TECH), best-first\n     within each stratum, so the first 12M tokens the packer consumes are register-balanced.\n\nMatching is on word vocabulary (content/register); the target's WikiText surface\nformatting (spaced punctuation, @,@ / @-@) never occurs in the pool and cannot be induced\nby selection, so it is deliberately ignored.\n\nOutput: /workspace/submission/selection.json -- pool ids, priority order (best first).\n\"\"\"\nimport json, re, zlib, time, numpy as np\nfrom multiprocessing import Pool\n\nPOOL = \"/workspace/data/pool.jsonl\"\nDEV  = \"/workspace/data/multi_dev.npy\"\nOUT  = \"/workspace/submission/selection.json\"\n\nNBITS      = 18\nNBUCKETS   = 1 << NBITS\nMASK       = NBUCKETS - 1\nALPHA      = 0.1\nRAW_SAMPLE = 40000\nSEED       = 1337\nCOVER_TOK  = 32_000_000        # ~2.7x the 12M budget\nTECH_RATIO = 0.25              # target's technical-Q&A share (equal-parts -> 1/4)\nMAX_CHARS  = 15000             # target docs are short; longer pool docs are dumps/nav bloat\nMIN_WORDS  = 50\nMIN_ALPHA  = 0.50\nMIN_STOP   = 0.10\n# tech (code/Q&A) prefilter: relaxed, but must contain real English prose too\nTECH_MINW  = 30\nTECH_MINSTOP = 0.03\nTECH_MAXCP = 0.45              # reject minified/base64 junk (too code-dense)\nWORD_RE    = re.compile(r\"[a-z]+\")\nCODE_CH    = set(\"{}[]<>;=()/$_|`\\\\\\t\")\n\n\ndef words(text):\n    return WORD_RE.findall(text.lower())\n\n\ndef buckets(ws):\n    b = [zlib.crc32(w.encode()) & MASK for w in ws]\n    for i in range(len(ws) - 1):\n        b.append(zlib.crc32((ws[i] + \" \" + ws[i + 1]).encode()) & MASK)\n    return b\n\n\ndef code_density(text):\n    if not text:\n        return 0.0\n    return sum(1 for ch in text if ch in CODE_CH) / len(text)\n\n\ndef count_chunk(texts):\n    c = np.zeros(NBUCKETS, dtype=np.float64)\n    for t in texts:\n        for h in buckets(words(t)):\n            c[h] += 1.0\n    return c\n\n\n# globals shared with workers via fork\n_WT = None   # tech weights\n_WP = None   # prose weights\n_TH = None   # tech/prose code-density threshold\n_DOCS = None\n\n\ndef _init(wt, wp, th, docs):\n    global _WT, _WP, _TH, _DOCS\n    _WT, _WP, _TH, _DOCS = wt, wp, th, docs\n\n\ndef score_range(rng):\n    \"\"\"Return (id, stratum, score, est_tokens); stratum: 0=prose,1=tech,-1=drop.\"\"\"\n    s, e = rng\n    out = []\n    for k in range(s, e):\n        did, text = _DOCS[k]\n        est = len(text) / 4.0\n        if len(text) > MAX_CHARS:\n            out.append((did, -1, -1e30, est)); continue\n        ws = words(text)\n        nw = len(ws)\n        cp = code_density(text)\n        stop_frac = (sum(1 for w in ws if w in STOP) / nw) if nw else 0.0\n        if cp > _TH:  # TECH candidate\n            ok = (nw >= TECH_MINW) and (stop_frac >= TECH_MINSTOP) and (cp <= TECH_MAXCP)\n            if not ok:\n                out.append((did, -1, -1e30, est)); continue\n            bs = np.asarray(buckets(ws))\n            out.append((did, 1, float(_WT[bs].mean()), est))\n        else:         # PROSE candidate\n            alpha = sum(ch.isalpha() for ch in text)\n            ok = (nw >= MIN_WORDS) and (alpha >= MIN_ALPHA * max(1, len(text))) and (stop_frac >= MIN_STOP)\n            if not ok:\n                out.append((did, -1, -1e30, est)); continue\n            bs = np.asarray(buckets(ws))\n            out.append((did, 0, float(_WP[bs].mean()), est))\n    return out\n\n\nSTOP = set(\"the a an and or but of to in on at for with by from as is are was were be been \"\n           \"being this that these those it its he she they we you i his her their our your \"\n           \"not no do does did have has had will would can could should may might must\".split())\n\n\ndef main():\n    t0 = time.time()\n    from transformers import AutoTokenizer\n    tok = AutoTokenizer.from_pretrained(\"gpt2\")\n    dev = np.load(DEV).astype(np.int64)\n    tdocs = [d for d in tok.decode(dev.tolist()).split(\"<|endoftext|>\") if len(d.split()) >= 10]\n    cd = np.array([code_density(d) for d in tdocs])\n    TH = float(np.quantile(cd, 1.0 - TECH_RATIO))       # auto-calibrated split\n    t_tech = [d for d, c in zip(tdocs, cd) if c > TH]\n    t_prose = [d for d, c in zip(tdocs, cd) if c <= TH]\n    print(f\"[t={time.time()-t0:.0f}s] target docs={len(tdocs)}  tech_thresh={TH:.4f}  \"\n          f\"tech={len(t_tech)} prose={len(t_prose)}\")\n\n    tcount = count_chunk(t_tech)\n    pcount = count_chunk(t_prose)\n\n    docs = []\n    with open(POOL) as f:\n        for line in f:\n            r = json.loads(line)\n            docs.append((r[\"id\"], r[\"text\"]))\n    print(f\"[t={time.time()-t0:.0f}s] pool docs: {len(docs)}\")\n\n    rng = np.random.default_rng(SEED)\n    idx = rng.choice(len(docs), size=min(RAW_SAMPLE, len(docs)), replace=False)\n    sample = [docs[i][1] for i in idx]\n    nproc = 16\n    with Pool(nproc) as p:\n        rcount = sum(p.map(count_chunk, [sample[i::nproc] for i in range(nproc)]))\n    R = rcount.sum()\n    pr = (rcount + ALPHA) / (R + ALPHA * NBUCKETS)\n\n    def wof(counts):\n        pt = (counts + ALPHA) / (counts.sum() + ALPHA * NBUCKETS)\n        return np.log(pt) - np.log(pr)\n    WT, WP = wof(tcount), wof(pcount)\n    print(f\"[t={time.time()-t0:.0f}s] weights ready\")\n\n    N = len(docs)\n    step = (N + nproc - 1) // nproc\n    ranges = [(i, min(i + step, N)) for i in range(0, N, step)]\n    with Pool(nproc, initializer=_init, initargs=(WT, WP, TH, docs)) as p:\n        parts = p.map(score_range, ranges)\n    rows = [r for part in parts for r in part]\n    prose = sorted([(sc, did, est) for did, st, sc, est in rows if st == 0], reverse=True)\n    tech  = sorted([(sc, did, est) for did, st, sc, est in rows if st == 1], reverse=True)\n    print(f\"[t={time.time()-t0:.0f}s] scored: prose_cand={len(prose)} tech_cand={len(tech)}\")\n\n    # round-robin fill at target ratio (per 4 docs: 3 prose, 1 tech), skipping\n    # near-exact duplicates (normalized-content hash keeps the first occurrence)\n    id2text = {did: t for did, t in docs}\n    seen = set()\n    _ws = re.compile(r\"\\s+\")\n\n    def take(pool_list, i):\n        while i < len(pool_list):\n            sc, did, est = pool_list[i]; i += 1\n            key = zlib.crc32(_ws.sub(\"\", id2text[did].lower())[:2000].encode())\n            if key in seen:\n                continue\n            seen.add(key)\n            return did, est, i\n        return None, 0.0, i\n\n    sel, cum = [], 0.0\n    pi = ti = 0\n    ptok = ttok = 0.0\n    while cum < COVER_TOK and (pi < len(prose) or ti < len(tech)):\n        for _ in range(3):\n            did, est, pi = take(prose, pi)\n            if did is not None:\n                sel.append(int(did)); ptok += est; cum += est\n        did, est, ti = take(tech, ti)\n        if did is not None:\n            sel.append(int(did)); ttok += est; cum += est\n    json.dump(sel, open(OUT, \"w\"))\n    print(f\"[t={time.time()-t0:.0f}s] selected={len(sel)} est_tokens~{cum/1e6:.1f}M \"\n          f\"(prose~{ptok/1e6:.1f}M tech~{ttok/1e6:.1f}M)\")\n    print(f\"wrote {OUT}\")\n\n\nif __name__ == \"__main__\":\n    main()\n", "numLines": 214, "startLine": 1, "totalLines": 214}}

prose target clusters: [601, 1096, 332]
  c0: On 1 January 1908 , Nimrod sailed for the Antarctic from Lyttelton Harbour , New Zealand . Shackleton 's origi
  c1: To conserve coal , the ship was towed 1 @,@ 650 miles ( 2 @,@ 655 km ) by the steamer Koonya to the Antarctic 
  c2: In 1910 , Shackleton made a series of three recordings describing the expedition using an Edison Phonograph .
weights ready t=20
cand counts [82419, 13701, 5293, 66244]
selected 44775 est_tokens 32.0 M  by-stratum [11.6, 9.9, 3.6, 6.9]

[stdout]
prose target clusters: [601, 1096, 332]
  c0: On 1 January 1908 , Nimrod sailed for the Antarctic from Lyttelton Harbour , New Zealand . Shackleton 's origi
  c1: To conserve coal , the ship was towed 1 @,@ 650 miles ( 2 @,@ 655 km ) by the steamer Koonya to the Antarctic 
  c2: In 1910 , Shackleton made a series of three recordings describing the expedition using an Edison Phonograph .
weights ready t=20
cand counts [82419, 13701, 5293, 66244]
selected 44775 est_tokens 32.0 M  by-stratum [11.6, 9.9, 3.6, 6.9]
[metadata] {"noOutputExpected": false}

=== deliverables ===
-rw-r--r-- 1 root root 4044 Aug  4 04:32 claim.md

submission/:
total 298
drwxr-xr-x 1 root root    220 Aug  4 04:22 .
drwxr-xr-x 1 root root    180 Aug  4 04:32 ..
drwxr-xr-x 1 root root    100 Aug  4 04:27 __pycache__
-rw-r--r-- 1 root root   8416 Aug  4 04:22 curate.py
-rw-r--r-- 1 root root 295518 Aug  4 04:23 selection.json
=== selection.json is the stratified 2-way (unchanged) ===
ids: 41768 unique: 41768 range: 0 181997
first 8 (priority order): [8287, 91191, 58452, 45266, 93562, 37522, 6400, 162538]
=== timer remaining (s) ===
3731

[stdout]
=== deliverables ===
-rw-r--r-- 1 root root 4044 Aug  4 04:32 claim.md

submission/:
total 298
drwxr-xr-x 1 root root    220 Aug  4 04:22 .
drwxr-xr-x 1 root root    180 Aug  4 04:32 ..
drwxr-xr-x 1 root root    100 Aug  4 04:27 __pycache__
-rw-r--r-- 1 root root   8416 Aug  4 04:22 curate.py
-rw-r--r-- 1 root root 295518 Aug  4 04:23 selection.json
=== selection.json is the stratified 2-way (unchanged) ===
ids: 41768 unique: 41768 range: 0 181997
first 8 (priority order): [8287, 91191, 58452, 45266, 93562, 37522, 6400, 162538]
=== timer remaining (s) ===
3731
[metadata] {"noOutputExpected": false}

cleaned
curate.py
selection.json

[stdout]
cleaned
curate.py
selection.json
[metadata] {"noOutputExpected": false}

Updated task #3 status

[metadata] {"success": true, "taskId": "3", "updatedFields": ["status"], "statusChange": {"from": "in_progress", "to": "completed"}}