claude-opus-5

44:Stages are cached under /tmp/cc_cold so re-runs are cheap.
52:CACHE = "/tmp/cc_cold"
1305

[stdout]
44:Stages are cached under /tmp/cc_cold so re-runs are cheap.
52:CACHE = "/tmp/cc_cold"
1305
[metadata] {"noOutputExpected": false}

1	#!/usr/bin/env python3
2	"""
3	Curate a 12M-token pretraining selection from a raw, unlabeled web pool for a BROAD
4	multi-register high-quality English target (encyclopedic / general web prose / news /
5	technical Q&A).   No labels, no internet, no GPU: ~3 min end-to-end on 15 CPU cores.
6	
7	=============================== STATED CRITERION ===============================
8	Let the disclosed dev target be split into its R = 4 equally-weighted registers
9	(the target file is four contiguous equal blocks).  For each register r we fit a
10	Dirichlet-smoothed backoff bigram model over hashed word n-grams, plus (a) a
11	BACKGROUND model estimated from the pool itself and (b) a "not-r" model (equal
12	mixture of the other registers and the background).  A document d is scored by the
13	register-contrastive Moore-Lewis cross-entropy difference
14	
15	        s_r(d) = H_{not r}(d) - H_r(d)                                       (1)
16	
17	i.e. how much better register r explains d than everything else does.  Technical
18	recall is sharpened by additive bonuses for code/Q&A surface markers, and
19	encyclopedic recall by a bonus for wiki-derived markers (see BONUS below).
20	
21	A document is ELIGIBLE if it passes fixed mechanical junk filters (length,
22	English-ness via stopword rate, non-ASCII rate, mean word length, terminal-
23	punctuation rate, duplicate-line rate, caps/digit rate, boilerplate density,
24	adult/spam terms).  The digit/table/boilerplate rules are relaxed for the
25	encyclopedic register (dates and infobox rows are signal there, not noise) and the
26	prose-shape rules are relaxed for documents that carry >= 3 code markers.
27	
28	SURFACE FORM: the encyclopedic target is detokenized WikiText (94.5% of its
29	punctuation marks are space-separated, 2.8% of its tokens are " @-@ " artifacts) - a
30	style the content score cannot express because it is orthogonal to word identity.
31	SURF_TOKENS of the encyclopedic quota are therefore reserved for the pool documents
32	with the highest space-before-punctuation rate (>= SURF_MIN), highest first.
33	
34	SELECTION = fill a per-register TOKEN QUOTA equal to that register's weight in the
35	evaluation, taking documents in descending s_r(d) and skipping near-duplicates
36	(MinHash LSH over 8-token shingles).  The quota vector MIX is the only tuned knob;
37	it is near-uniform because the metric averages loss uniformly over the registers.
38	Documents are emitted round-robin across registers so that ANY prefix of the list
39	(the trainer consumes the list in order until the budget is full) is register-
40	balanced, followed by a tail buffer of next-best ids that guards the 12M floor.
41	================================================================================
42	
43	Usage:  python3 curate_final.py [--out selection.json] [--mix w,x,y,z]
44	Stages are cached under /tmp/curate_cache so re-runs are cheap.
45	"""
46	import argparse, html, json, os, re, sys, zlib
47	import multiprocessing as mp
48	import numpy as np
49	
50	POOL = "/workspace/data/pool.jsonl"
51	DEV = "/workspace/data/multi_dev.npy"
52	CACHE = "/tmp/curate_cache"
53	BUDGET = 12_000_000
54	N_REG = 4
55	REG_NAMES = ["wiki", "webprose", "news", "techqa"]
56	os.makedirs(CACHE, exist_ok=True)
57	
58	WORD_RE = re.compile(r"[a-z]+|[0-9]+|[^\sa-z0-9]")
59	V_UNI = 1 << 19            # hashed unigram table
60	V_BI = 1 << 21             # hashed bigram table
61	MAXW = 6000                # words scored per document
62	ALPHA = 6.0                # Dirichlet backoff strength (bigram -> unigram)
63	
64	# ---- tuned selection knobs (chosen on the dev target; see claim.md) -----------
65	MIX = (0.225, 0.25, 0.25, 0.275)   # token quota per register
66	CODE_BOOST = 1.2                   # additive bonus, techqa score, per code marker (capped)
67	QA_BOOST = 0.5                     # additive bonus, techqa score, per Q&A marker (capped)
68	WIKI_BOOST = 0.7                   # additive bonus, wiki score, per wiki marker (capped)
69	SURF_TOKENS = 900_000              # tokens of the wiki quota reserved for surface-form matches
70	SURF_MIN = 0.12                    # min space-before-punctuation rate to count as a match
71	
72	
73	# --------------------------------------------------------------------- hashing
74	_HMEMO = {}
75

[metadata] {"type": "text", "file": {"filePath": "/workspace/submission/curate.py", "content": "#!/usr/bin/env python3\n\"\"\"\nCurate a 12M-token pretraining selection from a raw, unlabeled web pool for a BROAD\nmulti-register high-quality English target (encyclopedic / general web prose / news /\ntechnical Q&A).   No labels, no internet, no GPU: ~3 min end-to-end on 15 CPU cores.\n\n=============================== STATED CRITERION ===============================\nLet the disclosed dev target be split into its R = 4 equally-weighted registers\n(the target file is four contiguous equal blocks).  For each register r we fit a\nDirichlet-smoothed backoff bigram model over hashed word n-grams, plus (a) a\nBACKGROUND model estimated from the pool itself and (b) a \"not-r\" model (equal\nmixture of the other registers and the background).  A document d is scored by the\nregister-contrastive Moore-Lewis cross-entropy difference\n\n        s_r(d) = H_{not r}(d) - H_r(d)                                       (1)\n\ni.e. how much better register r explains d than everything else does.  Technical\nrecall is sharpened by additive bonuses for code/Q&A surface markers, and\nencyclopedic recall by a bonus for wiki-derived markers (see BONUS below).\n\nA document is ELIGIBLE if it passes fixed mechanical junk filters (length,\nEnglish-ness via stopword rate, non-ASCII rate, mean word length, terminal-\npunctuation rate, duplicate-line rate, caps/digit rate, boilerplate density,\nadult/spam terms).  The digit/table/boilerplate rules are relaxed for the\nencyclopedic register (dates and infobox rows are signal there, not noise) and the\nprose-shape rules are relaxed for documents that carry >= 3 code markers.\n\nSURFACE FORM: the encyclopedic target is detokenized WikiText (94.5% of its\npunctuation marks are space-separated, 2.8% of its tokens are \" @-@ \" artifacts) - a\nstyle the content score cannot express because it is orthogonal to word identity.\nSURF_TOKENS of the encyclopedic quota are therefore reserved for the pool documents\nwith the highest space-before-punctuation rate (>= SURF_MIN), highest first.\n\nSELECTION = fill a per-register TOKEN QUOTA equal to that register's weight in the\nevaluation, taking documents in descending s_r(d) and skipping near-duplicates\n(MinHash LSH over 8-token shingles).  The quota vector MIX is the only tuned knob;\nit is near-uniform because the metric averages loss uniformly over the registers.\nDocuments are emitted round-robin across registers so that ANY prefix of the list\n(the trainer consumes the list in order until the budget is full) is register-\nbalanced, followed by a tail buffer of next-best ids that guards the 12M floor.\n================================================================================\n\nUsage:  python3 curate_final.py [--out selection.json] [--mix w,x,y,z]\nStages are cached under /tmp/curate_cache so re-runs are cheap.\n\"\"\"\nimport argparse, html, json, os, re, sys, zlib\nimport multiprocessing as mp\nimport numpy as np\n\nPOOL = \"/workspace/data/pool.jsonl\"\nDEV = \"/workspace/data/multi_dev.npy\"\nCACHE = \"/tmp/curate_cache\"\nBUDGET = 12_000_000\nN_REG = 4\nREG_NAMES = [\"wiki\", \"webprose\", \"news\", \"techqa\"]\nos.makedirs(CACHE, exist_ok=True)\n\nWORD_RE = re.compile(r\"[a-z]+|[0-9]+|[^\\sa-z0-9]\")\nV_UNI = 1 << 19            # hashed unigram table\nV_BI = 1 << 21             # hashed bigram table\nMAXW = 6000                # words scored per document\nALPHA = 6.0                # Dirichlet backoff strength (bigram -> unigram)\n\n# ---- tuned selection knobs (chosen on the dev target; see claim.md) -----------\nMIX = (0.225, 0.25, 0.25, 0.275)   # token quota per register\nCODE_BOOST = 1.2                   # additive bonus, techqa score, per code marker (capped)\nQA_BOOST = 0.5                     # additive bonus, techqa score, per Q&A marker (capped)\nWIKI_BOOST = 0.7                   # additive bonus, wiki score, per wiki marker (capped)\nSURF_TOKENS = 900_000              # tokens of the wiki quota reserved for surface-form matches\nSURF_MIN = 0.12                    # min space-before-punctuation rate to count as a match\n\n\n# --------------------------------------------------------------------- hashing\n_HMEMO = {}\n", "numLines": 75, "startLine": 1, "totalLines": 450}}

The file /workspace/submission/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/submission/curate.py", "oldString": "Usage:  python3 curate_final.py [--out selection.json] [--mix w,x,y,z]\nStages are cached under /tmp/curate_cache so re-runs are cheap.", "newString": "Usage:  python3 curate.py [--out selection.json] [--mix w,x,y,z]\n                          [--surf TOKENS] [--surf_min RATE]\nDeterministic (crc32 word hashing, seeded MinHash permutations, no PYTHONHASHSEED\ndependence): a cold run reproduces the shipped selection.json byte for byte.\nStages are cached under /tmp/curate_cache so re-runs are cheap; the token cache is\nchecked for coverage, not just presence, so a stale cache cannot shrink the pool.", "originalFile": null, "structuredPatch": [{"oldStart": 40, "oldLines": 8, "newStart": 40, "newLines": 12, "lines": [" balanced, followed by a tail buffer of next-best ids that guards the 12M floor.", " ================================================================================", " ", "-Usage:  python3 curate_final.py [--out selection.json] [--mix w,x,y,z]", "-Stages are cached under /tmp/curate_cache so re-runs are cheap.", "+Usage:  python3 curate.py [--out selection.json] [--mix w,x,y,z]", "+                          [--surf TOKENS] [--surf_min RATE]", "+Deterministic (crc32 word hashing, seeded MinHash permutations, no PYTHONHASHSEED", "+dependence): a cold run reproduces the shipped selection.json byte for byte.", "+Stages are cached under /tmp/curate_cache so re-runs are cheap; the token cache is", "+checked for coverage, not just presence, so a stale cache cannot shrink the pool.", " \"\"\"", " import argparse, html, json, os, re, sys, zlib", " import multiprocessing as mp"]}], "userModified": false, "replaceAll": false}

wiki       2150 docs  2.70M tok (quota 2.70M)
      webprose   3259 docs  3.00M tok (quota 3.00M)
      news       3998 docs  3.00M tok (quota 3.00M)
      techqa     2056 docs  3.30M tok (quota 3.30M)
      near-duplicates skipped: 282; surface-form tokens: 0.77M
wrote 17463 ids, 18.78M tokens available for a 12M budget -> /tmp/sel_cold.json
cold run identical to shipped selection: True 17463 17463
1195

[stdout]
      wiki       2150 docs  2.70M tok (quota 2.70M)
      webprose   3259 docs  3.00M tok (quota 3.00M)
      news       3998 docs  3.00M tok (quota 3.00M)
      techqa     2056 docs  3.30M tok (quota 3.30M)
      near-duplicates skipped: 282; surface-form tokens: 0.77M
wrote 17463 ids, 18.78M tokens available for a 12M budget -> /tmp/sel_cold.json
cold run identical to shipped selection: True 17463 17463
1195
[metadata] {"noOutputExpected": false}

ids                : 17463
unique             : True
in range [0,182015]  : True  (min 6, max 181894)
ints only          : True
official pack      : 12000000 tokens, dtype uint16, max id 50256
total 140
drwxr-xr-x 1 root root    140 Aug  3 23:43 __pycache__
-rw-r--r-- 1 root root  20289 Aug  3 23:51 curate.py
-rw-r--r-- 1 root root 122733 Aug  3 23:48 selection.json
-r--r--r-- 1 root root 809679384 08-03_22:35 data/pool.jsonl
-rw-rw-r-- 1 root root      5706 08-03_22:34 model.py
-rw-rw-r-- 1 root root      2962 08-03_22:34 train_nano.py

[stdout]
ids                : 17463
unique             : True
in range [0,182015]  : True  (min 6, max 181894)
ints only          : True
official pack      : 12000000 tokens, dtype uint16, max id 50256
total 140
drwxr-xr-x 1 root root    140 Aug  3 23:43 __pycache__
-rw-r--r-- 1 root root  20289 Aug  3 23:51 curate.py
-rw-r--r-- 1 root root 122733 Aug  3 23:48 selection.json
-r--r--r-- 1 root root 809679384 08-03_22:35 data/pool.jsonl
-rw-rw-r-- 1 root root      5706 08-03_22:34 model.py
-rw-rw-r-- 1 root root      2962 08-03_22:34 train_nano.py
[metadata] {"noOutputExpected": false}