claude-opus-4-8

HELDOUT_PPL 320.243  loss 5.7691  train_s 54  /tmp/dev_train.npy
---timer---
2807

[stdout]
HELDOUT_PPL 320.243  loss 5.7691  train_s 54  /tmp/dev_train.npy
---timer---
2807
[metadata] {"noOutputExpected": false}

consumed docs 11337 ~tokens 12000251
near-dup exact-6minhash collisions: 152 extra docs in 126 groups out of 11337 (1.3%)

[stdout]
consumed docs 11337 ~tokens 12000251
near-dup exact-6minhash collisions: 152 extra docs in 126 groups out of 11337 (1.3%)
[metadata] {"noOutputExpected": false}

=== gen v8: 0.33/0.67/0.0 (no tech) ===
---timer---
2626

[stdout]
=== gen v8: 0.33/0.67/0.0 (no tech) ===
---timer---
2626
[metadata] {"noOutputExpected": false}

loading pool ...
  182016 docs
  tokenized (6s)
  featurized (7s)
  scored register wiki (17s)
  scored register webnews (24s)
  scored register techqa (30s)
  common vocab 30767 words (30s)
  quality gate keeps 128437/182016  (prose 128364, html 171) (34s)
    wiki: 16930 docs
    webnews: 109469 docs
    techqa: 2038 docs
Traceback (most recent call last):
  File "/workspace/submission/curate.py", line 297, in <module>
    main()
  File "/workspace/submission/curate.py", line 268, in main
    r = min(cand, key=lambda r: tok_taken[r] / MIX[r])
  File "/workspace/submission/curate.py", line 268, in <lambda>
    r = min(cand, key=lambda r: tok_taken[r] / MIX[r])
ZeroDivisionError: float division by zero

[stdout]
loading pool ...
  182016 docs
  tokenized (6s)
  featurized (7s)
  scored register wiki (17s)
  scored register webnews (24s)
  scored register techqa (30s)
  common vocab 30767 words (30s)
  quality gate keeps 128437/182016  (prose 128364, html 171) (34s)
    wiki: 16930 docs
    webnews: 109469 docs
    techqa: 2038 docs
Traceback (most recent call last):
  File "/workspace/submission/curate.py", line 297, in <module>
    main()
  File "/workspace/submission/curate.py", line 268, in main
    r = min(cand, key=lambda r: tok_taken[r] / MIX[r])
  File "/workspace/submission/curate.py", line 268, in <lambda>
    r = min(cand, key=lambda r: tok_taken[r] / MIX[r])
ZeroDivisionError: float division by zero
[metadata] {"noOutputExpected": false}

The file /workspace/submission/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/submission/curate.py", "oldString": "        cand = [r for r in reg_names if pos[r] < len(queues[r])]\n        if not cand: break\n        r = min(cand, key=lambda r: tok_taken[r] / MIX[r])", "newString": "        cand = [r for r in reg_names if pos[r] < len(queues[r]) and MIX[r] > 0]\n        if not cand: break\n        r = min(cand, key=lambda r: tok_taken[r] / MIX[r])", "originalFile": null, "structuredPatch": [{"oldStart": 263, "oldLines": 7, "newStart": 263, "newLines": 7, "lines": ["     total = 0", "     while total < TARGET_TOK:", "         # pick the register furthest below its token quota that still has docs", "-        cand = [r for r in reg_names if pos[r] < len(queues[r])]", "+        cand = [r for r in reg_names if pos[r] < len(queues[r]) and MIX[r] > 0]", "         if not cand: break", "         r = min(cand, key=lambda r: tok_taken[r] / MIX[r])", "         di = queues[r][pos[r]]; pos[r] += 1"]}], "userModified": false, "replaceAll": false}

Command did not complete within its 120s timeout and was moved to the background (ID: b4db2v0ao). Output is being written to: /tmp/claude-0/-workspace/25296ac2-340c-4280-bfe7-333d9d814974/tasks/b4db2v0ao.output. You will be notified when it completes. To check interim output, use Read on that file path.
Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "b4db2v0ao", "timedOutAfterMs": 120000, "backgroundCwdHint": "Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands."}

pool: 132550/182016 docs pass gate = 72.8%
random baseline consumed 11022 docs / 12009746 tokens
  junk (gate-fail) tokens: 42.6%   junk docs: 27.7%

[stdout]
pool: 132550/182016 docs pass gate = 72.8%
random baseline consumed 11022 docs / 12009746 tokens
  junk (gate-fail) tokens: 42.6%   junk docs: 27.7%
[metadata] {"noOutputExpected": false}

1	  balanced-portion tokens by register: {'wiki': 5280689, 'webnews': 10719437, 'techqa': 0} sum 16000126
2	wrote 21986 ids (~24.0M tokens) -> /workspace/artifacts/sel_v8_notech.json (34s)
3	=== EVAL v8 (no tech) ===
4	HELDOUT_PPL 405.736  loss 6.0057  train_s 54  /tmp/dev_train.npy
5	---timer---
6	2436
7

[metadata] {"type": "text", "file": {"filePath": "/tmp/claude-0/-workspace/25296ac2-340c-4280-bfe7-333d9d814974/tasks/b4db2v0ao.output", "content": "  balanced-portion tokens by register: {'wiki': 5280689, 'webnews': 10719437, 'techqa': 0} sum 16000126\nwrote 21986 ids (~24.0M tokens) -> /workspace/artifacts/sel_v8_notech.json (34s)\n=== EVAL v8 (no tech) ===\nHELDOUT_PPL 405.736  loss 6.0057  train_s 54  /tmp/dev_train.npy\n---timer---\n2436\n", "numLines": 7, "startLine": 1, "totalLines": 7}}

=== MIX + BUDGET constants ===
41:BUDGET       = 12_000_000     # trainer stops here
46:SEED         = 0
48:MIX          = {"wiki": 0.25, "webnews": 0.50, "techqa": 0.25}
50:DEV_BLOCKS   = {"wiki": (0, 1713), "webnews": (1713, 2346), "techqa": (2346, 100000)}
51:TARGET_TOK   = 16_000_000     # build a balanced list up to here, then backfill
52:BACKFILL_TOK = 24_000_000     # total tokens of ids to emit (margin over budget)

=== __main__ block ===
if __name__ == "__main__":
    # optional CLI mix override for sweeping: curate.py [wiki web tech] [out.json]
    av = sys.argv[1:]
    if len(av) >= 3:
        MIX = {"wiki": float(av[0]), "webnews": float(av[1]), "techqa": float(av[2])}
        s = sum(MIX.values()); MIX = {k: v/s for k, v in MIX.items()}
    if len(av) >= 4:
        OUT = av[3]
    main()
=== timer ===
2276

[stdout]
=== MIX + BUDGET constants ===
41:BUDGET       = 12_000_000     # trainer stops here
46:SEED         = 0
48:MIX          = {"wiki": 0.25, "webnews": 0.50, "techqa": 0.25}
50:DEV_BLOCKS   = {"wiki": (0, 1713), "webnews": (1713, 2346), "techqa": (2346, 100000)}
51:TARGET_TOK   = 16_000_000     # build a balanced list up to here, then backfill
52:BACKFILL_TOK = 24_000_000     # total tokens of ids to emit (margin over budget)

=== __main__ block ===
if __name__ == "__main__":
    # optional CLI mix override for sweeping: curate.py [wiki web tech] [out.json]
    av = sys.argv[1:]
    if len(av) >= 3:
        MIX = {"wiki": float(av[0]), "webnews": float(av[1]), "techqa": float(av[2])}
        s = sum(MIX.values()); MIX = {k: v/s for k, v in MIX.items()}
    if len(av) >= 4:
        OUT = av[3]
    main()
=== timer ===
2276
[metadata] {"noOutputExpected": false}

37:OUT    = "/workspace/submission/selection.json"
116:def main():
194:        json.dump(sorted(common), open(cw_path, "w"))
286:    json.dump(order, open(OUT, "w"))
287:    print(f"wrote {len(order)} ids (~{bt/1e6:.1f}M tokens) -> {OUT} ({time.time()-t0:.0f}s)")
296:        OUT = av[3]

[stdout]
37:OUT    = "/workspace/submission/selection.json"
116:def main():
194:        json.dump(sorted(common), open(cw_path, "w"))
286:    json.dump(order, open(OUT, "w"))
287:    print(f"wrote {len(order)} ids (~{bt/1e6:.1f}M tokens) -> {OUT} ({time.time()-t0:.0f}s)")
296:        OUT = av[3]
[metadata] {"noOutputExpected": false}

Command running in background with ID: b2cg88nrf. Output is being written to: /tmp/claude-0/-workspace/25296ac2-340c-4280-bfe7-333d9d814974/tasks/b2cg88nrf.output. You will be notified when it completes. To check interim output, use Read on that file path.
Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "b2cg88nrf", "backgroundCwdHint": "Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands."}

File created successfully at: /workspace/claim.md (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/workspace/claim.md", "content": "# Claim\n\n## Hypothesis\n\nUnder a fixed 12M-token budget, held-out perplexity on a broad HQ-English target\nis limited primarily by the **fluency and register composition** of the training\ntokens — not by raw domain-keyword relevance, and not by reproducing the eval's\nregister proportions exactly. A selection that (a) filters the raw pool down to\nclean, fluent English prose and (b) weights each register by its *learnable*\nsupply will train a substantially better small LM than a random draw of the same\nsize.\n\nThe submitted `selection.json` is produced by `curate.py` from a single stated\ncriterion: **keep documents that pass an English-fluency gate, score each against\nthe target's three registers (encyclopedic / web+news / technical) with a\nNaive-Bayes log-count-ratio classifier, and interleave the top-scoring documents\nto a 0.25 / 0.50 / 0.25 register budget.** No document is hand-picked.\n\n## Mechanism (prediction distinct from final perplexity)\n\nThe gain comes from **removing wasted budget, not from clever ranking**. A random\n12M-token draw spends a large share of its budget on tokens a tiny model cannot\ncompress — non-English text, navigation/boilerplate, machine-generated/templated\nstrings, and raw code — which inflate cross-entropy. The fluency gate reclaims\nthat budget for learnable prose. Three observables follow that do **not** depend\non the final perplexity value:\n\n1. **Junk fraction of the random baseline.** ≥40% of the random baseline's 12M\n   training tokens come from documents that fail a simple fluency gate (ASCII\n   ratio, stopword density, common-word coverage, type-token ratio, long-segment\n   fraction).\n   → *Observed: 42.6% of tokens / 27.7% of documents.*\n\n2. **The gate, not the ranking, is the causal lever.** Domain-keyword scoring\n   *without* the fluency gate scores **worse than random**, because keyword\n   density rewards spam, boilerplate, and foreign keyword-stuffing.\n   → *Observed: 741.9 (no gate) vs 474.5 (random).*\n\n3. **Register weighting is non-monotone in the eval share.** The optimal weight on\n   the technical register is *below* its nominal eval share but *above zero*: the\n   pool contains almost no in-format technical Q&A, so both **forcing** the eval's\n   25% technical proportion with the available code-dense docs **and dropping the\n   register entirely** raise perplexity relative to the ~8–11% fluent-technical\n   fraction the classifier assigns on its own.\n   → *Observed: forced-25%-code ≈ 410, zero-tech ≈ 406, natural fraction ≈ FINAL_PPL.*\n\n## Falsification\n\nThe hypothesis is false if any of the following hold:\n\n- A random selection, or a gate-free domain-scored selection, matches or beats the\n  curated selection's held-out perplexity.\n- Removing the fluency gate from `curate.py` (keeping domain ranking + register\n  mix) does **not** worsen perplexity — i.e. the gate is not the causal lever.\n- Observable (1) is wrong: the random baseline's junk-token fraction is small\n  (<~15%), so wasted budget cannot account for the gap.\n- A selection that matches the eval's exact 25% technical proportion (with any\n  available pool docs) **beats** the natural-underfill selection.\n\n## Transfer\n\n- **Target-agnostic gate.** The fluency gate removes universal web junk, not\n  target-specific content, so it should lower perplexity on any broad HQ-English\n  target and on any raw-web pool.\n- **Scale.** The advantage should hold or grow with budget and model size until\n  the pool's clean-prose supply (~132k docs / ~130M+ tokens here) is exhausted;\n  past that point marginal tokens revert to pool average and the gap narrows.\n- **Register rule, not the constants.** The specific 0.25/0.50/0.25 numbers are\n  target-specific, but the transferable rule is \"weight each register by its\n  *learnable* (in-format, fluent) supply, not by its nominal eval share\" — which\n  is why under-weighting the code-scarce technical register wins here.\n", "structuredPatch": [], "originalFile": null, "userModified": false}

Updated task #4 status

[metadata] {"success": true, "taskId": "4", "updatedFields": ["status"], "statusChange": {"from": "pending", "to": "in_progress"}}

1	=== REGEN default selection.json (0.25/0.50/0.25, fluency gate) ===
2	loading pool ...
3	  182016 docs
4	  tokenized (6s)
5	  featurized (7s)
6	  scored register wiki (18s)
7	  scored register webnews (24s)
8	  scored register techqa (30s)
9	  common vocab 30767 words (30s)
10	  quality gate keeps 128437/182016  (prose 128364, html 171) (34s)
11	    wiki: 16930 docs
12	    webnews: 109469 docs
13	    techqa: 2038 docs
14	  balanced-portion tokens by register: {'wiki': 4951801, 'webnews': 9814098, 'techqa': 1235948} sum 16001847
15	wrote 22854 ids (~24.0M tokens) -> /workspace/submission/selection.json (34s)
16	=== EVAL final selection.json ===
17	Token indices sequence length is longer than the specified maximum sequence length for this model (2451 > 1024). Running this sequence through the model will result in indexing errors
18

[metadata] {"type": "text", "file": {"filePath": "/tmp/claude-0/-workspace/25296ac2-340c-4280-bfe7-333d9d814974/tasks/b2cg88nrf.output", "content": "=== REGEN default selection.json (0.25/0.50/0.25, fluency gate) ===\nloading pool ...\n  182016 docs\n  tokenized (6s)\n  featurized (7s)\n  scored register wiki (18s)\n  scored register webnews (24s)\n  scored register techqa (30s)\n  common vocab 30767 words (30s)\n  quality gate keeps 128437/182016  (prose 128364, html 171) (34s)\n    wiki: 16930 docs\n    webnews: 109469 docs\n    techqa: 2038 docs\n  balanced-portion tokens by register: {'wiki': 4951801, 'webnews': 9814098, 'techqa': 1235948} sum 16001847\nwrote 22854 ids (~24.0M tokens) -> /workspace/submission/selection.json (34s)\n=== EVAL final selection.json ===\nToken indices sequence length is longer than the specified maximum sequence length for this model (2451 > 1024). Running this sequence through the model will result in indexing errors\n", "numLines": 18, "startLine": 1, "totalLines": 18}}

<retrieval_status>success</retrieval_status>

<task_id>b2cg88nrf</task_id>

<task_type>local_bash</task_type>

<status>completed</status>

<exit_code>0</exit_code>

<output>
=== REGEN default selection.json (0.25/0.50/0.25, fluency gate) ===
loading pool ...
  182016 docs
  tokenized (6s)
  featurized (7s)
  scored register wiki (18s)
  scored register webnews (24s)
  scored register techqa (30s)
  common vocab 30767 words (30s)
  quality gate keeps 128437/182016  (prose 128364, html 171) (34s)
    wiki: 16930 docs
    webnews: 109469 docs
    techqa: 2038 docs
  balanced-portion tokens by register: {'wiki': 4951801, 'webnews': 9814098, 'techqa': 1235948} sum 16001847
wrote 22854 ids (~24.0M tokens) -> /workspace/submission/selection.json (34s)
=== EVAL final selection.json ===
Token indices sequence length is longer than the specified maximum sequence length for this model (2451 > 1024). Running this sequence through the model will result in indexing errors
packed 12000000 tokens from 22854 ids -> /tmp/dev_train.npy
HELDOUT_PPL 320.243  loss 5.7691  train_s 54  /tmp/dev_train.npy
(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)
---timer---
2126
</output>

[metadata] {"retrieval_status": "success", "task": {"task_id": "b2cg88nrf", "task_type": "local_bash", "status": "completed", "description": "Regenerate default selection.json and eval it", "output": "=== REGEN default selection.json (0.25/0.50/0.25, fluency gate) ===\nloading pool ...\n  182016 docs\n  tokenized (6s)\n  featurized (7s)\n  scored register wiki (18s)\n  scored register webnews (24s)\n  scored register techqa (30s)\n  common vocab 30767 words (30s)\n  quality gate keeps 128437/182016  (prose 128364, html 171) (34s)\n    wiki: 16930 docs\n    webnews: 109469 docs\n    techqa: 2038 docs\n  balanced-portion tokens by register: {'wiki': 4951801, 'webnews': 9814098, 'techqa': 1235948} sum 16001847\nwrote 22854 ids (~24.0M tokens) -> /workspace/submission/selection.json (34s)\n=== EVAL final selection.json ===\nToken indices sequence length is longer than the specified maximum sequence length for this model (2451 > 1024). Running this sequence through the model will result in indexing errors\npacked 12000000 tokens from 22854 ids -> /tmp/dev_train.npy\nHELDOUT_PPL 320.243  loss 5.7691  train_s 54  /tmp/dev_train.npy\n(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)\n---timer---\n2126\n", "exitCode": 0}}

The file /workspace/claim.md has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/claim.md", "oldString": "   → *Observed: forced-25%-code ≈ 410, zero-tech ≈ 406, natural fraction ≈ FINAL_PPL.*", "newString": "   → *Observed: forced-25%-code ≈ 410, zero-tech ≈ 406, natural ~8% fraction = 320.*", "originalFile": "# Claim\n\n## Hypothesis\n\nUnder a fixed 12M-token budget, held-out perplexity on a broad HQ-English target\nis limited primarily by the **fluency and register composition** of the training\ntokens — not by raw domain-keyword relevance, and not by reproducing the eval's\nregister proportions exactly. A selection that (a) filters the raw pool down to\nclean, fluent English prose and (b) weights each register by its *learnable*\nsupply will train a substantially better small LM than a random draw of the same\nsize.\n\nThe submitted `selection.json` is produced by `curate.py` from a single stated\ncriterion: **keep documents that pass an English-fluency gate, score each against\nthe target's three registers (encyclopedic / web+news / technical) with a\nNaive-Bayes log-count-ratio classifier, and interleave the top-scoring documents\nto a 0.25 / 0.50 / 0.25 register budget.** No document is hand-picked.\n\n## Mechanism (prediction distinct from final perplexity)\n\nThe gain comes from **removing wasted budget, not from clever ranking**. A random\n12M-token draw spends a large share of its budget on tokens a tiny model cannot\ncompress — non-English text, navigation/boilerplate, machine-generated/templated\nstrings, and raw code — which inflate cross-entropy. The fluency gate reclaims\nthat budget for learnable prose. Three observables follow that do **not** depend\non the final perplexity value:\n\n1. **Junk fraction of the random baseline.** ≥40% of the random baseline's 12M\n   training tokens come from documents that fail a simple fluency gate (ASCII\n   ratio, stopword density, common-word coverage, type-token ratio, long-segment\n   fraction).\n   → *Observed: 42.6% of tokens / 27.7% of documents.*\n\n2. **The gate, not the ranking, is the causal lever.** Domain-keyword scoring\n   *without* the fluency gate scores **worse than random**, because keyword\n   density rewards spam, boilerplate, and foreign keyword-stuffing.\n   → *Observed: 741.9 (no gate) vs 474.5 (random).*\n\n3. **Register weighting is non-monotone in the eval share.** The optimal weight on\n   the technical register is *below* its nominal eval share but *above zero*: the\n   pool contains almost no in-format technical Q&A, so both **forcing** the eval's\n   25% technical proportion with the available code-dense docs **and dropping the\n   register entirely** raise perplexity relative to the ~8–11% fluent-technical\n   fraction the classifier assigns on its own.\n   → *Observed: forced-25%-code ≈ 410, zero-tech ≈ 406, natural fraction ≈ FINAL_PPL.*\n\n## Falsification\n\nThe hypothesis is false if any of the following hold:\n\n- A random selection, or a gate-free domain-scored selection, matches or beats the\n  curated selection's held-out perplexity.\n- Removing the fluency gate from `curate.py` (keeping domain ranking + register\n  mix) does **not** worsen perplexity — i.e. the gate is not the causal lever.\n- Observable (1) is wrong: the random baseline's junk-token fraction is small\n  (<~15%), so wasted budget cannot account for the gap.\n- A selection that matches the eval's exact 25% technical proportion (with any\n  available pool docs) **beats** the natural-underfill selection.\n\n## Transfer\n\n- **Target-agnostic gate.** The fluency gate removes universal web junk, not\n  target-specific content, so it should lower perplexity on any broad HQ-English\n  target and on any raw-web pool.\n- **Scale.** The advantage should hold or grow with budget and model size until\n  the pool's clean-prose supply (~132k docs / ~130M+ tokens here) is exhausted;\n  past that point marginal tokens revert to pool average and the gap narrows.\n- **Register rule, not the constants.** The specific 0.25/0.50/0.25 numbers are\n  target-specific, but the transferable rule is \"weight each register by its\n  *learnable* (in-format, fluent) supply, not by its nominal eval share\" — which\n  is why under-weighting the code-scarce technical register wins here.\n", "structuredPatch": [{"oldStart": 42, "oldLines": 7, "newStart": 42, "newLines": 7, "lines": ["    25% technical proportion with the available code-dense docs **and dropping the", "    register entirely** raise perplexity relative to the ~8–11% fluent-technical", "    fraction the classifier assigns on its own.", "-   → *Observed: forced-25%-code ≈ 410, zero-tech ≈ 406, natural fraction ≈ FINAL_PPL.*", "+   → *Observed: forced-25%-code ≈ 410, zero-tech ≈ 406, natural ~8% fraction = 320.*", " ", " ## Falsification", " "]}], "userModified": false, "replaceAll": false}

The file /workspace/claim.md has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/claim.md", "oldString": "is why under-weighting the code-scarce technical register wins here.", "newString": "is why under-weighting the code-scarce technical register wins here.\n\n## Result\n\nHeld-out perplexity (dev, `multi_dev`), 12M-token budget, frozen `train_nano`:\n\n| Selection                              | HELDOUT_PPL |\n| -------------------------------------- | ----------: |\n| Theoretical best                       |         1.0 |\n| **Curated (this submission)**          |     **320.2** |\n| Random baseline (same budget)          |       474.5 |\n| Domain-score, fluency gate **removed** |       741.9 |\n| Forced 25% code-dense technical        |       410.0 |\n| Technical register dropped entirely    |       405.7 |\n\nThe curated selection cuts perplexity 32.5% below the random baseline; the\ngate-off and mis-weighted variants confirm the mechanism above.\n", "originalFile": "# Claim\n\n## Hypothesis\n\nUnder a fixed 12M-token budget, held-out perplexity on a broad HQ-English target\nis limited primarily by the **fluency and register composition** of the training\ntokens — not by raw domain-keyword relevance, and not by reproducing the eval's\nregister proportions exactly. A selection that (a) filters the raw pool down to\nclean, fluent English prose and (b) weights each register by its *learnable*\nsupply will train a substantially better small LM than a random draw of the same\nsize.\n\nThe submitted `selection.json` is produced by `curate.py` from a single stated\ncriterion: **keep documents that pass an English-fluency gate, score each against\nthe target's three registers (encyclopedic / web+news / technical) with a\nNaive-Bayes log-count-ratio classifier, and interleave the top-scoring documents\nto a 0.25 / 0.50 / 0.25 register budget.** No document is hand-picked.\n\n## Mechanism (prediction distinct from final perplexity)\n\nThe gain comes from **removing wasted budget, not from clever ranking**. A random\n12M-token draw spends a large share of its budget on tokens a tiny model cannot\ncompress — non-English text, navigation/boilerplate, machine-generated/templated\nstrings, and raw code — which inflate cross-entropy. The fluency gate reclaims\nthat budget for learnable prose. Three observables follow that do **not** depend\non the final perplexity value:\n\n1. **Junk fraction of the random baseline.** ≥40% of the random baseline's 12M\n   training tokens come from documents that fail a simple fluency gate (ASCII\n   ratio, stopword density, common-word coverage, type-token ratio, long-segment\n   fraction).\n   → *Observed: 42.6% of tokens / 27.7% of documents.*\n\n2. **The gate, not the ranking, is the causal lever.** Domain-keyword scoring\n   *without* the fluency gate scores **worse than random**, because keyword\n   density rewards spam, boilerplate, and foreign keyword-stuffing.\n   → *Observed: 741.9 (no gate) vs 474.5 (random).*\n\n3. **Register weighting is non-monotone in the eval share.** The optimal weight on\n   the technical register is *below* its nominal eval share but *above zero*: the\n   pool contains almost no in-format technical Q&A, so both **forcing** the eval's\n   25% technical proportion with the available code-dense docs **and dropping the\n   register entirely** raise perplexity relative to the ~8–11% fluent-technical\n   fraction the classifier assigns on its own.\n   → *Observed: forced-25%-code ≈ 410, zero-tech ≈ 406, natural ~8% fraction = 320.*\n\n## Falsification\n\nThe hypothesis is false if any of the following hold:\n\n- A random selection, or a gate-free domain-scored selection, matches or beats the\n  curated selection's held-out perplexity.\n- Removing the fluency gate from `curate.py` (keeping domain ranking + register\n  mix) does **not** worsen perplexity — i.e. the gate is not the causal lever.\n- Observable (1) is wrong: the random baseline's junk-token fraction is small\n  (<~15%), so wasted budget cannot account for the gap.\n- A selection that matches the eval's exact 25% technical proportion (with any\n  available pool docs) **beats** the natural-underfill selection.\n\n## Transfer\n\n- **Target-agnostic gate.** The fluency gate removes universal web junk, not\n  target-specific content, so it should lower perplexity on any broad HQ-English\n  target and on any raw-web pool.\n- **Scale.** The advantage should hold or grow with budget and model size until\n  the pool's clean-prose supply (~132k docs / ~130M+ tokens here) is exhausted;\n  past that point marginal tokens revert to pool average and the gap narrows.\n- **Register rule, not the constants.** The specific 0.25/0.50/0.25 numbers are\n  target-specific, but the transferable rule is \"weight each register by its\n  *learnable* (in-format, fluent) supply, not by its nominal eval share\" — which\n  is why under-weighting the code-scarce technical register wins here.\n", "structuredPatch": [{"oldStart": 69, "oldLines": 3, "newStart": 69, "newLines": 20, "lines": ["   target-specific, but the transferable rule is \"weight each register by its", "   *learnable* (in-format, fluent) supply, not by its nominal eval share\" — which", "   is why under-weighting the code-scarce technical register wins here.", "+", "+## Result", "+", "+Held-out perplexity (dev, `multi_dev`), 12M-token budget, frozen `train_nano`:", "+", "+| Selection                              | HELDOUT_PPL |", "+| -------------------------------------- | ----------: |", "+| Theoretical best                       |         1.0 |", "+| **Curated (this submission)**          |     **320.2** |", "+| Random baseline (same budget)          |       474.5 |", "+| Domain-score, fluency gate **removed** |       741.9 |", "+| Forced 25% code-dense technical        |       410.0 |", "+| Technical register dropped entirely    |       405.7 |", "+", "+The curated selection cuts perplexity 32.5% below the random baseline; the", "+gate-off and mis-weighted variants confirm the mechanism above.", "+"]}], "userModified": false, "replaceAll": false}

n ids           : 22854
all int         : True
in range 0..182015  : True
no duplicates   : True
ids to fill 12M : 11337 of 22854
tokens available: 24.0 M (>=12M required)
fills 12M budget: True

[stdout]
n ids           : 22854
all int         : True
in range 0..182015  : True
no duplicates   : True
ids to fill 12M : 11337 of 22854
tokens available: 24.0 M (>=12M required)
fills 12M budget: True
[metadata] {"noOutputExpected": false}

=== placeholder check (should be empty) ===
=== curate.py cache-regen + external deps ===
7:(measured on the disclosed dev target `multi_dev.npy`):
14:  1. Decode `multi_dev.npy` (the disclosed target, GPT-2 tokens) back to text and
36:DEVNPY = "/workspace/data/multi_dev.npy"
88:def dev_register_texts():
89:    a = np.load(DEVNPY).astype(np.int64)
126:    if os.path.exists(cnt_path):
127:        counts = np.load(cnt_path)
143:    if os.path.exists(fz_path):
144:        z = np.load(fz_path, allow_pickle=True)
186:    if os.path.exists(cw_path):
=== deliverables present ===
-rw-r--r-- 1 root root   4623 Aug  4 05:40 claim.md
-rw-r--r-- 1 root root  13166 Aug  4 05:32 submission/curate.py
-rw-r--r-- 1 root root 162412 Aug  4 05:38 submission/selection.json

[stdout]
=== placeholder check (should be empty) ===
=== curate.py cache-regen + external deps ===
7:(measured on the disclosed dev target `multi_dev.npy`):
14:  1. Decode `multi_dev.npy` (the disclosed target, GPT-2 tokens) back to text and
36:DEVNPY = "/workspace/data/multi_dev.npy"
88:def dev_register_texts():
89:    a = np.load(DEVNPY).astype(np.int64)
126:    if os.path.exists(cnt_path):
127:        counts = np.load(cnt_path)
143:    if os.path.exists(fz_path):
144:        z = np.load(fz_path, allow_pickle=True)
186:    if os.path.exists(cw_path):
=== deliverables present ===
-rw-r--r-- 1 root root   4623 Aug  4 05:40 claim.md
-rw-r--r-- 1 root root  13166 Aug  4 05:32 submission/curate.py
-rw-r--r-- 1 root root 162412 Aug  4 05:38 submission/selection.json
[metadata] {"noOutputExpected": false}

1	#!/usr/bin/env python3
2	"""Curate a raw web pool into a training selection for a small LM.
3	
4	STATED CRITERION
5	----------------
6	The scoring target is a BROAD high-quality English mix in three registers
7	(measured on the disclosed dev target `multi_dev.npy`):
8	    ~25% encyclopedic (Wikipedia)   ~50% general web + news prose   ~25% technical Q&A
9	
10	We therefore select pool documents by *domain match to that target*, register by
11	register, and assemble them so that the first 12M tokens the trainer consumes
12	reproduce the target's register mix. Concretely:
13	
14	  1. Decode `multi_dev.npy` (the disclosed target, GPT-2 tokens) back to text and
15	     split it into its three contiguous register blocks -> these are POSITIVES.
16	  2. Take a random sample of the pool as the background/NEGATIVE class.
17	  3. For each register, fit a Naive-Bayes log-count-ratio linear scorer over hashed
18	     word uni+bigram features (NBSVM-style). A pool doc's score under register R is
19	     the length-normalised sum of NB weights of its features = evidence that the
20	     doc reads like register R rather than like average web.
21	  4. Rank pool docs within each register; drop obvious junk (too short / low
22	     alpha-ratio / duplicate). Interleave the three ranked lists by *token* quota
23	     (25/50/25) so the consumed 12M-token budget matches the target mix, best
24	     documents first. Append the remaining ranked docs as backfill.
25	
26	Output: /workspace/submission/selection.json  (pool ids, priority order).
27	
28	This is a pure function of the pool + the disclosed dev target; nothing is
29	hand-picked. Re-running reproduces the same selection (deterministic hashing +
30	fixed seeds).
31	"""
32	import json, re, sys, zlib, time, os
33	import numpy as np
34	
35	POOL   = "/workspace/data/pool.jsonl"

[metadata] {"type": "text", "file": {"filePath": "/workspace/submission/curate.py", "content": "#!/usr/bin/env python3\n\"\"\"Curate a raw web pool into a training selection for a small LM.\n\nSTATED CRITERION\n----------------\nThe scoring target is a BROAD high-quality English mix in three registers\n(measured on the disclosed dev target `multi_dev.npy`):\n    ~25% encyclopedic (Wikipedia)   ~50% general web + news prose   ~25% technical Q&A\n\nWe therefore select pool documents by *domain match to that target*, register by\nregister, and assemble them so that the first 12M tokens the trainer consumes\nreproduce the target's register mix. Concretely:\n\n  1. Decode `multi_dev.npy` (the disclosed target, GPT-2 tokens) back to text and\n     split it into its three contiguous register blocks -> these are POSITIVES.\n  2. Take a random sample of the pool as the background/NEGATIVE class.\n  3. For each register, fit a Naive-Bayes log-count-ratio linear scorer over hashed\n     word uni+bigram features (NBSVM-style). A pool doc's score under register R is\n     the length-normalised sum of NB weights of its features = evidence that the\n     doc reads like register R rather than like average web.\n  4. Rank pool docs within each register; drop obvious junk (too short / low\n     alpha-ratio / duplicate). Interleave the three ranked lists by *token* quota\n     (25/50/25) so the consumed 12M-token budget matches the target mix, best\n     documents first. Append the remaining ranked docs as backfill.\n\nOutput: /workspace/submission/selection.json  (pool ids, priority order).\n\nThis is a pure function of the pool + the disclosed dev target; nothing is\nhand-picked. Re-running reproduces the same selection (deterministic hashing +\nfixed seeds).\n\"\"\"\nimport json, re, sys, zlib, time, os\nimport numpy as np\n\nPOOL   = \"/workspace/data/pool.jsonl\"", "numLines": 35, "startLine": 1, "totalLines": 298}}

The file /workspace/submission/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/submission/curate.py", "oldString": "  4. Rank pool docs within each register; drop obvious junk (too short / low\n     alpha-ratio / duplicate). Interleave the three ranked lists by *token* quota", "newString": "  4. Rank pool docs within each register, but first apply an English-FLUENCY GATE\n     (this is the dominant lever): keep a doc only if it reads as clean English\n     prose -- enough words, high ASCII ratio, adequate stopword density, most\n     tokens in the pool's common vocabulary, healthy type-token ratio, and a\n     majority of words inside long (>=8-word) segments; a lenient variant admits\n     genuine HTML/code Q&A. Then interleave the three ranked lists by *token* quota", "originalFile": null, "structuredPatch": [{"oldStart": 18, "oldLines": 8, "newStart": 18, "newLines": 12, "lines": ["      word uni+bigram features (NBSVM-style). A pool doc's score under register R is", "      the length-normalised sum of NB weights of its features = evidence that the", "      doc reads like register R rather than like average web.", "-  4. Rank pool docs within each register; drop obvious junk (too short / low", "-     alpha-ratio / duplicate). Interleave the three ranked lists by *token* quota", "+  4. Rank pool docs within each register, but first apply an English-FLUENCY GATE", "+     (this is the dominant lever): keep a doc only if it reads as clean English", "+     prose -- enough words, high ASCII ratio, adequate stopword density, most", "+     tokens in the pool's common vocabulary, healthy type-token ratio, and a", "+     majority of words inside long (>=8-word) segments; a lenient variant admits", "+     genuine HTML/code Q&A. Then interleave the three ranked lists by *token* quota", "      (25/50/25) so the consumed 12M-token budget matches the target mix, best", "      documents first. Append the remaining ranked docs as backfill.", " "]}], "userModified": false, "replaceAll": false}