claude-opus-5

=== RESULTS (dev PPL, lower better) ===
  320.316  light2
  361.178  w.25-.25-.25-.25
  381.408  light3
  387.072  nogatesrr
TIME: 2996

[stdout]
=== RESULTS (dev PPL, lower better) ===
  320.316  light2
  361.178  w.25-.25-.25-.25
  381.408  light3
  387.072  nogatesrr
TIME: 2996
[metadata] {"noOutputExpected": false}

patched
[5/5] selected 25726 docs; est tokens/register: [6604732, 6600211, 6600660, 6605749] (names ['wiki', 'webprose', 'news', 'qa'])
      est total tokens 26.4M (budget 12M)
wrote /tmp/sel_normraw.json
light2: 24866 ids -> /tmp/sel_light2.json
light4 pool: 150099
light4: 23350 ids -> /tmp/sel_light4.json
light2nd pool: 145095
light2nd: 23900 ids -> /tmp/sel_light2nd.json
regenerated light2 from default cache
TIME: 2896

[stdout]
patched
[5/5] selected 25726 docs; est tokens/register: [6604732, 6600211, 6600660, 6605749] (names ['wiki', 'webprose', 'news', 'qa'])
      est total tokens 26.4M (budget 12M)
wrote /tmp/sel_normraw.json
light2: 24866 ids -> /tmp/sel_light2.json
light4 pool: 150099
light4: 23350 ids -> /tmp/sel_light4.json
light2nd pool: 145095
light2nd: 23900 ids -> /tmp/sel_light2nd.json
regenerated light2 from default cache
TIME: 2896
[metadata] {"noOutputExpected": false}

Command did not complete within its 120s timeout and was moved to the background (ID: bwr0kppmb). Output is being written to: /tmp/claude-0/-workspace/0ad3cc01-7bd9-4c7b-a578-db1f5f8fa369/tasks/bwr0kppmb.output. You will be notified when it completes. To check interim output, use Read on that file path.
Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "bwr0kppmb", "timedOutAfterMs": 120000, "backgroundCwdHint": "Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands."}

The file /workspace/submission/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/submission/curate.py", "oldString": "    G = dict(nwords=80, nchars=450, cov=0.45, alpha=0.60, digit=0.20, upper=0.16,\n             short_lines=0.60, uniq_line=0.60, uwr_lo=0.18, uwr_hi=0.88,\n             line_words=6.0, pipe=6.0)", "newString": "    # Swept on the dev target: strict 391.5 -> moderate 354.8 -> loose 338.9 -> these\n    # 320.3. Loosening helps only while every dimension is still checked: dropping the\n    # format dimensions entirely and keeping length+coverage alone regresses to 381.4,\n    # and no gates at all to 387.1. So keep all dimensions, set them permissively.\n    G = dict(nwords=60, nchars=0, cov=0.35, alpha=0.50, digit=0.28, upper=0.22,\n             short_lines=0.75, uniq_line=0.50, uwr_lo=0.15, uwr_hi=0.92,\n             line_words=4.0, pipe=12.0)", "originalFile": null, "structuredPatch": [{"oldStart": 293, "oldLines": 9, "newStart": 293, "newLines": 13, "lines": ["     # tightening the gates *hurts* (it starves the match score of candidates and", "     # forces it deeper down each register's ranking). They only remove documents", "     # too short or too unlike running English prose to be scored meaningfully.", "-    G = dict(nwords=80, nchars=450, cov=0.45, alpha=0.60, digit=0.20, upper=0.16,", "-             short_lines=0.60, uniq_line=0.60, uwr_lo=0.18, uwr_hi=0.88,", "-             line_words=6.0, pipe=6.0)", "+    # Swept on the dev target: strict 391.5 -> moderate 354.8 -> loose 338.9 -> these", "+    # 320.3. Loosening helps only while every dimension is still checked: dropping the", "+    # format dimensions entirely and keeping length+coverage alone regresses to 381.4,", "+    # and no gates at all to 387.1. So keep all dimensions, set them permissively.", "+    G = dict(nwords=60, nchars=0, cov=0.35, alpha=0.50, digit=0.28, upper=0.22,", "+             short_lines=0.75, uniq_line=0.50, uwr_lo=0.15, uwr_hi=0.92,", "+             line_words=4.0, pipe=12.0)", " ", "     def ok(d):", "         return (d[\"nwords\"] >= G[\"nwords\"] and d[\"nchars\"] >= G[\"nchars\"]"]}], "userModified": false, "replaceAll": false}

=== RESULTS (dev PPL, lower better) ===
  321.945  norm
  329.705  light2nd
  342.390  light4
TIME: 2686

[stdout]
=== RESULTS (dev PPL, lower better) ===
  321.945  norm
  329.705  light2nd
  342.390  light4
TIME: 2686
[metadata] {"noOutputExpected": false}

main()
  File "/workspace/submission/curate.py", line 253, in main
    blocks, segs = target_blocks()
  File "/workspace/submission/curate.py", line 71, in target_blocks
    return o["blocks"], o["segs"]
TypeError: list indices must be integers or slices, not str
reproduces best variant exactly: False 24664 24728
unique: True in range: True min/max 1 181997

[stdout]
    main()
  File "/workspace/submission/curate.py", line 253, in main
    blocks, segs = target_blocks()
  File "/workspace/submission/curate.py", line 71, in target_blocks
    return o["blocks"], o["segs"]
TypeError: list indices must be integers or slices, not str
reproduces best variant exactly: False 24664 24728
unique: True in range: True min/max 1 181997
[metadata] {"noOutputExpected": false}

[4/5] 145095/182016 docs pass quality gates
      135660 after near-dup removal
[5/5] selected 24728 docs; est tokens/register: [6600247, 6601910, 6600147, 6604391] (names ['wiki', 'webprose', 'news', 'qa'])
      est total tokens 26.4M (budget 12M)
wrote /workspace/submission/selection.json
reproduces best variant exactly: True 24728 24728
unique: True | in range: True
TIME: 2656

[stdout]
[4/5] 145095/182016 docs pass quality gates
      135660 after near-dup removal
[5/5] selected 24728 docs; est tokens/register: [6600247, 6601910, 6600147, 6604391] (names ['wiki', 'webprose', 'news', 'qa'])
      est total tokens 26.4M (budget 12M)
wrote /workspace/submission/selection.json
reproduces best variant exactly: True 24728 24728
unique: True | in range: True
TIME: 2656
[metadata] {"noOutputExpected": false}

official dev_eval launched
[1]+  Done                    nohup bash dev_eval.sh > /tmp/official.log 2>&1
Token indices sequence length is longer than the specified maximum sequence length for this model (5414 > 1024). Running this sequence through the model will result in indexing errors
packed 12000000 tokens from 24728 ids -> /tmp/dev_train.npy
HELDOUT_PPL 320.316  loss 5.7693  train_s 51  /tmp/dev_train.npy
(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)

[stdout]
official dev_eval launched
[1]+  Done                    nohup bash dev_eval.sh > /tmp/official.log 2>&1
Token indices sequence length is longer than the specified maximum sequence length for this model (5414 > 1024). Running this sequence through the model will result in indexing errors
packed 12000000 tokens from 24728 ids -> /tmp/dev_train.npy
HELDOUT_PPL 320.316  loss 5.7693  train_s 51  /tmp/dev_train.npy
(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)
[metadata] {"noOutputExpected": false}

sqrtnorm: 42489 ids -> /tmp/sel_sqrtnorm.json
cov2: 24643 ids -> /tmp/sel_cov2.json
ens: 24711 ids -> /tmp/sel_ens.json
TIME: 2486

[stdout]
sqrtnorm: 42489 ids -> /tmp/sel_sqrtnorm.json
cov2: 24643 ids -> /tmp/sel_cov2.json
ens: 24711 ids -> /tmp/sel_ens.json
TIME: 2486
[metadata] {"noOutputExpected": false}

[sqrtnorm] packed 12000000 tokens from 13622 docs (mean 881 tok/doc)
[cov2] packed 12000000 tokens from 12852 docs (mean 934 tok/doc)
[ens] packed 12000000 tokens from 12893 docs (mean 931 tok/doc)
TIME: 2375

[stdout]
[sqrtnorm] packed 12000000 tokens from 13622 docs (mean 881 tok/doc)
[cov2] packed 12000000 tokens from 12852 docs (mean 934 tok/doc)
[ens] packed 12000000 tokens from 12893 docs (mean 931 tok/doc)
TIME: 2375
[metadata] {"noOutputExpected": false}

updated
=== RESULTS (dev PPL, lower better) ===
  320.971  cov2
  324.734  ens
  332.335  sqrtnorm
TIME: 2285

[stdout]
updated
=== RESULTS (dev PPL, lower better) ===
  320.971  cov2
  324.734  ens
  332.335  sqrtnorm
TIME: 2285
[metadata] {"noOutputExpected": false}

cold-cache repro check launched

[stdout]
cold-cache repro check launched
[metadata] {"noOutputExpected": false}

The file /workspace/submission/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/submission/curate.py", "oldString": "STATED CRITERION\n----------------\nA document is selected iff it (a) passes format-quality gates that remove web junk\n(boilerplate/nav lists, ALLCAPS spam, symbol tables, duplicate-line spam, too-short\ndocs, non-English) and (b) ranks highly under a *register-conditional* Naive-Bayes\ndomain-match score\n\n    s_r(d) = (1/N_d) * sum_f c_f(d) * log[ p_r(f) / p_pool(f) ]\n\nwhere f ranges over target-vocabulary unigrams+bigrams, p_r is the word distribution\nof target register r (estimated from the disclosed dev target, which we decode and\nsplit into its four registers), and p_pool is the pool background distribution.\nThis is the classic importance-weight / DSIR criterion. Documents are then allocated\nround-robin across the four registers with equal token quotas, so the training\nmixture matches the target mixture instead of collapsing onto whichever register the\npool happens to contain most of. Near-duplicates are removed (exact, prefix and\n5-gram MinHash-band). Output is ordered by round-robin rank so that any prefix of\nthe list is register-balanced.\n\nReproducible: python3 curate.py  ->  /workspace/submission/selection.json\n\"\"\"", "newString": "STATED CRITERION\n----------------\nKeep document d for register r if d is *scoreable* (light format gates, below) and d\nis among the top-ranked documents for r under the register-conditional Naive-Bayes /\nimportance-weight log-ratio\n\n    s_r(d) = (1/N_d) * sum_f c_f(d) * log[ p_r(f) / p_pool(f) ]\n\nwhere f ranges over target-vocabulary unigrams+bigrams, c_f(d) is d's feature count,\nN_d its length in words, p_r the word distribution of target register r, and p_pool\nthe pool background distribution. The four registers are obtained by decoding the\ndisclosed dev target and splitting it into its four equal 250k-token blocks\n(encyclopedic / high-quality web prose / news / technical Q&A). The budget is then\nfilled by interleaving the four rankings one document at a time, so all registers are\ncovered to the same rank depth and every prefix of the output is register-balanced.\nNear-duplicates (exact, 40-word prefix, 5-gram MinHash band) are dropped, keeping the\nhigher-scoring member.\n\nWHY THIS SHAPE (all figures are dev perplexity under the frozen trainer, 12M tokens)\nThe operative variable is register match, not document quality: the same format gates\nwith random selection inside them score 464.0, i.e. no better than a random selection\nof the pool (457.5), while this criterion scores 320.3. Removing register balance and\ntaking a global top-k on the same score costs ~9% (388.6 vs 354.8 at matched gates).\nAllocating equal *tokens* per register instead of equal rank depth costs ~2%\n(361.2 vs 354.8). Reweighting the mixture away from the target's 25/25/25/25 --\neven toward the highest-loss register -- only hurts (up to 380.3). The gates are set\npermissively on purpose: tightening them starves the match score of candidates\n(391.5 strict), but they cannot be dropped either (387.1 with none), so every\ndimension is checked with a loose threshold. See /workspace/claim.md.\n\nReproducible, ~60 s on 14 CPU cores, no GPU and no reference model:\n    python3 curate.py          ->  /workspace/submission/selection.json\n\"\"\"", "originalFile": null, "structuredPatch": [{"oldStart": 4, "oldLines": 24, "newStart": 4, "newLines": 36, "lines": [" ", " STATED CRITERION", " ----------------", "-A document is selected iff it (a) passes format-quality gates that remove web junk", "-(boilerplate/nav lists, ALLCAPS spam, symbol tables, duplicate-line spam, too-short", "-docs, non-English) and (b) ranks highly under a *register-conditional* Naive-Bayes", "-domain-match score", "+Keep document d for register r if d is *scoreable* (light format gates, below) and d", "+is among the top-ranked documents for r under the register-conditional Naive-Bayes /", "+importance-weight log-ratio", " ", "     s_r(d) = (1/N_d) * sum_f c_f(d) * log[ p_r(f) / p_pool(f) ]", " ", "-where f ranges over target-vocabulary unigrams+bigrams, p_r is the word distribution", "-of target register r (estimated from the disclosed dev target, which we decode and", "-split into its four registers), and p_pool is the pool background distribution.", "-This is the classic importance-weight / DSIR criterion. Documents are then allocated", "-round-robin across the four registers with equal token quotas, so the training", "-mixture matches the target mixture instead of collapsing onto whichever register the", "-pool happens to contain most of. Near-duplicates are removed (exact, prefix and", "-5-gram MinHash-band). Output is ordered by round-robin rank so that any prefix of", "-the list is register-balanced.", "+where f ranges over target-vocabulary unigrams+bigrams, c_f(d) is d's feature count,", "+N_d its length in words, p_r the word distribution of target register r, and p_pool", "+the pool background distribution. The four registers are obtained by decoding the", "+disclosed dev target and splitting it into its four equal 250k-token blocks", "+(encyclopedic / high-quality web prose / news / technical Q&A). The budget is then", "+filled by interleaving the four rankings one document at a time, so all registers are", "+covered to the same rank depth and every prefix of the output is register-balanced.", "+Near-duplicates (exact, 40-word prefix, 5-gram MinHash band) are dropped, keeping the", "+higher-scoring member.", " ", "-Reproducible: python3 curate.py  ->  /workspace/submission/selection.json", "+WHY THIS SHAPE (all figures are dev perplexity under the frozen trainer, 12M tokens)", "+The operative variable is register match, not document quality: the same format gates", "+with random selection inside them score 464.0, i.e. no better than a random selection", "+of the pool (457.5), while this criterion scores 320.3. Removing register balance and", "+taking a global top-k on the same score costs ~9% (388.6 vs 354.8 at matched gates).", "+Allocating equal *tokens* per register instead of equal rank depth costs ~2%", "+(361.2 vs 354.8). Reweighting the mixture away from the target's 25/25/25/25 --", "+even toward the highest-loss register -- only hurts (up to 380.3). The gates are set", "+permissively on purpose: tightening them starves the match score of candidates", "+(391.5 strict), but they cannot be dropped either (387.1 with none), so every", "+dimension is checked with a loose threshold. See /workspace/claim.md.", "+", "+Reproducible, ~60 s on 14 CPU cores, no GPU and no reference model:", "+    python3 curate.py          ->  /workspace/submission/selection.json", " \"\"\"", " import json, os, re, sys, hashlib, zlib", " from collections import Counter"]}], "userModified": false, "replaceAll": false}

135660 after near-dup removal
[5/5] selected 24728 docs; est tokens/register: [6600247, 6601910, 6600147, 6604391] (names ['wiki', 'webprose', 'news', 'qa'])
      est total tokens 26.4M (budget 12M)
wrote /tmp/sel_cold.json
cold-cache run reproduces submission exactly: True 24728 24728
TIME: 2195

[stdout]
      135660 after near-dup removal
[5/5] selected 24728 docs; est tokens/register: [6600247, 6601910, 6600147, 6604391] (names ['wiki', 'webprose', 'news', 'qa'])
      est total tokens 26.4M (budget 12M)
wrote /tmp/sel_cold.json
cold-cache run reproduces submission exactly: True 24728 24728
TIME: 2195
[metadata] {"noOutputExpected": false}

[1]+  Done                    TRAINER=/workspace/diag_train.py nohup python3 eval_batch.py final > /tmp/evalfinal.log 2>&1
{"train_npy": "/tmp/train_final.npy", "heldout_ppl": 320.31584198354085, "mean_loss": 5.76930751522199, "max_iters": 3000, "seed": 1337, "train_s": 50.2, "per_register_ppl": {"wiki": 763.292165597332, "webprose": 284.1368157705229, "news": 293.7259842615986, "qa": 164.84090185051147}}
=== RESULTS (dev PPL, lower better) ===
  320.316  final

[stdout]
[1]+  Done                    TRAINER=/workspace/diag_train.py nohup python3 eval_batch.py final > /tmp/evalfinal.log 2>&1
{"train_npy": "/tmp/train_final.npy", "heldout_ppl": 320.31584198354085, "mean_loss": 5.76930751522199, "max_iters": 3000, "seed": 1337, "train_s": 50.2, "per_register_ppl": {"wiki": 763.292165597332, "webprose": 284.1368157705229, "news": 293.7259842615986, "qa": 164.84090185051147}}
=== RESULTS (dev PPL, lower better) ===
  320.316  final
[metadata] {"noOutputExpected": false}

The file /workspace/claim.md has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"type": "update", "filePath": "/workspace/claim.md", "content": "# Claim: at a fixed token budget, held-out perplexity is set by *register match*, and conventional \"quality\" filtering is at best neutral, at worst register-biased\n\n## Hypothesis\n\nFor a fixed 12M-token budget drawn from a 200M-token raw web pool, held-out perplexity\non a broad multi-register target is governed almost entirely by how well the selection\ncovers **each of the target's registers**, and hardly at all by generic\ndocument-quality filtering.\n\nOperationally: score every pool document with a per-register Naive-Bayes /\nimportance-weight log-ratio against the *disclosed* target\n(`s_r(d) = (1/N_d) Σ_f c_f(d) log[p_r(f)/p_pool(f)]`, unigram+bigram features), then\nfill the budget by interleaving the four per-register rankings one document at a time.\nPredicted: this beats a random selection by ≳20% and beats an *unbalanced* global\ntop-k on the same score by ≳5%, while heuristic junk-filtering alone lands at random.\n\nResult: **320.3 dev PPL vs 457.5 for random selection (−30.0%)**; global top-k on the\nsame score is 388.6; junk-filtering alone is 464.0 (no better than random).\n\n## Mechanism, and what it predicts *other than* the final perplexity\n\nThe pool is Common-Crawl-style web text. The target is four 250k-token blocks —\nWikiText-formatted Wikipedia, high-quality web prose, newswire, StackExchange Q&A.\nThe mechanism is that each block is predicted by data of *its own register*, and\nregisters are not interchangeable, so the budget must be spread over all four. Three\nobservables follow, all measurable without looking at the final score:\n\n1. **Junk filters alone ≈ random.** If quality is not the operative variable, applying\n   only my format gates and then picking randomly inside them should not beat random\n   selection. → *Observed: gates-only 464.0 vs random 457.5 — no gain (slightly worse).*\n\n2. **The per-register loss breakdown is strongly non-uniform, worst on Wikipedia, for\n   a surface-form reason that no selection can fix.** The target's Wikipedia block is\n   WikiText-tokenized (spaced punctuation, ` @-@ `, ` @,@ `). I grepped the pool:\n   **0 of 182,016 documents** contain that form, and only 69 contain `</p>`, so the\n   Q&A block's HTML form is absent too; the documents that *do* have systematically\n   spaced punctuation (154 docs with ratio > 0.5) are SEO spam and machine-translated\n   junk, not prose. → *Observed on the submitted selection: wiki 763 vs web-prose 284,\n   news 294, Q&A 165.* The Wikipedia quarter carries ~1 nat/token more loss than the\n   rest, and that gap is a property of the pool, not of the criterion. It is the\n   binding ceiling on the achievable score.\n\n3. **The strongest single prediction — gate loosening pays off in ONE register.**\n   Sweeping gate strictness moved dev PPL 391.5 (strict) → 354.8 (moderate) → 338.9\n   (light) → 320.3 (very light). If this were generic \"more data is better\", the gain\n   would be spread across registers. If instead my gates encoded a *prose* prior\n   (`frac_alpha ≥ 0.68`, `frac_digit ≤ 0.16`, `frac_short_lines ≤ 0.45`, `pipe ≤ 3`\n   all penalise code blocks, symbols and short lines), the gain must be concentrated\n   in the code-bearing Q&A register. → *Observed, moderate → very light gates:\n   **Q&A 303.4 → 164.8 (−46%)**, while the other three registers got slightly worse\n   (wiki 691 → 763, web-prose 266 → 284, news 283 → 294).* The gates were not neutral\n   quality filters; they were a register filter that silently starved a quarter of the\n   target. This is the mechanism behind most of the 354.8 → 320.3 improvement.\n\n## Falsification\n\nThe claim would have been falsified by any of:\n\n- **(a) Quality-only wins.** Gates-only ≈ the matched selection ⇒ quality is the\n  operative variable. *Result: 464.0 vs 320.3. Not falsified.*\n- **(b) Register balance is inert.** Global top-k on the same score, ignoring\n  registers, matching the balanced selection. *Result: 388.6 vs 354.8 at matched\n  gates — balance is worth ~9%. Not falsified.*\n- **(c) The gain is dev-topical, not register-level.** The profile is fitted on the\n  disclosed dev sample, so its edge could be topic leakage (picking documents about\n  Shackleton because a dev article is), which would not transfer to the hidden sample.\n  Test: refit the profile on the **first half** of each register block only and\n  evaluate on the **unseen second half**, against a profile restricted to features\n  occurring in ≥8 distinct target segments (register/style vocabulary only, topical\n  terms stripped). If the edge were topical, the unrestricted profile would collapse\n  on unseen text. *Result: 362.5 (unrestricted) vs 363.9 (topic-stripped) —\n  indistinguishable, so the signal is register-level and survives a disjoint sample.\n  Not falsified; I kept the simpler unrestricted profile.*\n- **(d) Mixture-pushing keeps helping.** If more of the highest-loss register were\n  always better, reweighting toward Wikipedia should keep improving things. *It does\n  not: 31/29/20/21 → 360.5, 37/35/14/13 → 379.1, 45/40/8/7 → 380.3.* The target's own\n  25/25/25/25 mixture is the right allocation — a real constraint on the claim.\n- **(e) The result is a lucky point in gate-threshold space.** Three further variants\n  around the chosen configuration: looser coverage/line-uniqueness 321.0, rank\n  ensemble of two feature normalisations 324.7, sqrt-length score normalisation 332.3;\n  and stepping *past* the optimum on every dimension at once regresses to 342.4. The\n  submitted setting sits on a plateau, not a spike.\n\n## Transfer\n\nWhat transfers is the recipe, not the id list: **decode the disclosed target, split it\ninto its constituent registers, fit per-register importance weights against the pool\nbackground, and interleave the per-register rankings to fill the budget.** No labels,\nno reference model, no GPU — the full pipeline over 182k documents runs in ~60 s on 14\nCPU cores, so it scales to far larger pools.\n\nExpected limits when transferred:\n- **Audit your filters per register, not in aggregate.** The single largest avoidable\n  loss here came from format gates that looked register-neutral and were not\n  (observable 3). Any filter tuned on prose will quietly delete code, tables and\n  markup — and therefore any target register made of them.\n- Gates cannot be dropped either: with length+coverage only, dev PPL is 381.4, and\n  with no gates 387.1, because unscoreable junk then outranks real documents on the\n  match score. Keep every dimension, set each permissively.\n- Interleave by **rank depth**, not equal token share: equalising tokens drags the\n  registers the pool supports thinly (news, Q&A) further down their rankings and costs\n  ~2% (354.8 → 361.2).\n- Near-duplicate removal is worth ~9 PPL here (329.7 with dedup disabled) — small\n  relative to register match, but free.\n- The gain shrinks as the pool's natural mixture approaches the target's, and is\n  capped by surface forms the pool lacks (observable 2). If a target register's\n  tokenization or markup does not exist in the pool, selection cannot recover it.\n\n## Result summary (dev PPL, frozen 30M GPT, 12M tokens, identical recipe, seed 1337)\n\n| selection | dev PPL |\n|---|---|\n| format-quality gates only, random within | 464.0 |\n| random pool sample (do-nothing baseline) | 457.5 |\n| register-matched, strict gates | 391.5 |\n| global top-k on match score (no register balance) | 388.6 |\n| register-matched, no gates | 387.1 |\n| register-matched, equal token share instead of rank depth | 361.2 |\n| register-matched, moderate gates | 354.8 |\n| register-matched, light gates | 338.9 |\n| register-matched, dedup disabled | 329.7 |\n| **register-matched, very light gates (submitted)** | **320.3** |\n\nPer-register PPL of the submitted selection: wiki 763.3, web-prose 284.1, news 293.7,\nQ&A 164.8.\n\nSubmitted selection re-verified end-to-end through the official path\n(`pack_selection.py` → `train_nano.py`, unmodified): 12,000,000 tokens packed from\n24,728 ordered ids, `HELDOUT_PPL 320.316`. `curate.py` re-run from a cold cache\nreproduces `selection.json` exactly.\n", "structuredPatch": [{"oldStart": 1, "oldLines": 105, "newStart": 1, "newLines": 130, "lines": ["-# Claim: at a fixed token budget, held-out perplexity is set by *register-mixture match*, not by \"quality\"", "+# Claim: at a fixed token budget, held-out perplexity is set by *register match*, and conventional \"quality\" filtering is at best neutral, at worst register-biased", " ", " ## Hypothesis", " ", "-For a fixed 12M-token budget drawn from a 200M-token raw web pool, held-out", "-perplexity on a broad multi-register target is governed almost entirely by how well", "-the selection's **register mixture** matches the target's, and hardly at all by", "-generic document-quality filtering.", "+For a fixed 12M-token budget drawn from a 200M-token raw web pool, held-out perplexity", "+on a broad multi-register target is governed almost entirely by how well the selection", "+covers **each of the target's registers**, and hardly at all by generic", "+document-quality filtering.", " ", " Operationally: score every pool document with a per-register Naive-Bayes /", " importance-weight log-ratio against the *disclosed* target", "-(`s_r(d) = (1/N_d) Σ_f c_f(d) log[p_r(f)/p_pool(f)]`, unigram+bigram features),", "-then fill the budget by interleaving the four per-register rankings. I predicted this", "-would beat a random selection by ≳20% and beat an *unbalanced* global top-k on the", "-same score by ≳5%, while heuristic junk-filtering alone would land at random.", "+(`s_r(d) = (1/N_d) Σ_f c_f(d) log[p_r(f)/p_pool(f)]`, unigram+bigram features), then", "+fill the budget by interleaving the four per-register rankings one document at a time.", "+Predicted: this beats a random selection by ≳20% and beats an *unbalanced* global", "+top-k on the same score by ≳5%, while heuristic junk-filtering alone lands at random.", " ", "+Result: **320.3 dev PPL vs 457.5 for random selection (−30.0%)**; global top-k on the", "+same score is 388.6; junk-filtering alone is 464.0 (no better than random).", "+", " ## Mechanism, and what it predicts *other than* the final perplexity", " ", "-The pool is Common-Crawl-style web text; the target is four 250k-token blocks —", "+The pool is Common-Crawl-style web text. The target is four 250k-token blocks —", " WikiText-formatted Wikipedia, high-quality web prose, newswire, StackExchange Q&A.", "-The mechanism is that each target block is predicted by data of *its own register*,", "-and registers are not interchangeable, so the budget must be spread over all four.", "-Three observables follow, all measurable without looking at the final score:", "+The mechanism is that each block is predicted by data of *its own register*, and", "+registers are not interchangeable, so the budget must be spread over all four. Three", "+observables follow, all measurable without looking at the final score:", " ", "-1. **Junk filters alone ≈ random.** If quality is not the operative variable, a", "-   selection that applies only my format gates and then picks randomly inside them", "-   should not beat random selection. → *Observed: gates-only 464.0 vs random 457.5", "-   dev PPL — no gain (slightly worse).*", "-2. **The per-register loss breakdown is strongly non-uniform, worst on Wikipedia,", "-   for a surface-form reason.** The target's Wikipedia block is WikiText-tokenized", "-   (spaced punctuation, ` @-@ `, ` @,@ `). I grepped the pool: **0 of 182,016", "-   documents** contain that form (and only 69 contain `</p>`, so the Q&A block's", "-   HTML form is absent too). No selection can supply it, so that block should carry", "-   a much higher loss than the rest. → *Observed on the balanced selection:", "-   wiki 691 vs web-prose 266, news 283, Q&A 303.* This is the binding ceiling on", "-   the achievable score, and it is a property of the pool, not of the criterion.", "-3. **Tightening the gates should *hurt*, not help.** If matching is what matters,", "-   gates only shrink the candidate set and force the match score deeper down each", "-   register's ranking. → *Observed: strict gates 391.5 → moderate 354.8 → light", "-   338.9 → very light 320.3 dev PPL. The effect is one-sided but not unbounded:", "-   removing the format dimensions altogether (length + English-coverage only) gives", "-   381.4 and no gates at all 387.1, so the gates must stay in place — just permissive.*", "+1. **Junk filters alone ≈ random.** If quality is not the operative variable, applying", "+   only my format gates and then picking randomly inside them should not beat random", "+   selection. → *Observed: gates-only 464.0 vs random 457.5 — no gain (slightly worse).*", " ", "+2. **The per-register loss breakdown is strongly non-uniform, worst on Wikipedia, for", "+   a surface-form reason that no selection can fix.** The target's Wikipedia block is", "+   WikiText-tokenized (spaced punctuation, ` @-@ `, ` @,@ `). I grepped the pool:", "+   **0 of 182,016 documents** contain that form, and only 69 contain `</p>`, so the", "+   Q&A block's HTML form is absent too; the documents that *do* have systematically", "+   spaced punctuation (154 docs with ratio > 0.5) are SEO spam and machine-translated", "+   junk, not prose. → *Observed on the submitted selection: wiki 763 vs web-prose 284,", "+   news 294, Q&A 165.* The Wikipedia quarter carries ~1 nat/token more loss than the", "+   rest, and that gap is a property of the pool, not of the criterion. It is the", "+   binding ceiling on the achievable score.", "+", "+3. **The strongest single prediction — gate loosening pays off in ONE register.**", "+   Sweeping gate strictness moved dev PPL 391.5 (strict) → 354.8 (moderate) → 338.9", "+   (light) → 320.3 (very light). If this were generic \"more data is better\", the gain", "+   would be spread across registers. If instead my gates encoded a *prose* prior", "+   (`frac_alpha ≥ 0.68`, `frac_digit ≤ 0.16`, `frac_short_lines ≤ 0.45`, `pipe ≤ 3`", "+   all penalise code blocks, symbols and short lines), the gain must be concentrated", "+   in the code-bearing Q&A register. → *Observed, moderate → very light gates:", "+   **Q&A 303.4 → 164.8 (−46%)**, while the other three registers got slightly worse", "+   (wiki 691 → 763, web-prose 266 → 284, news 283 → 294).* The gates were not neutral", "+   quality filters; they were a register filter that silently starved a quarter of the", "+   target. This is the mechanism behind most of the 354.8 → 320.3 improvement.", "+", " ## Falsification", " ", " The claim would have been falsified by any of:", " ", "-- **(a) Quality-only wins.** Gates-only ≈ balanced-match ⇒ quality is the variable.", "-  *Result: 464.0 vs 320.3. Not falsified.*", "-- **(b) Balance is inert.** Global top-k on the same score, ignoring registers,", "-  matching balanced selection. *Result: 388.6 vs 354.8 under identical gates —", "-  balance is worth ~9%. Not falsified.*", "+- **(a) Quality-only wins.** Gates-only ≈ the matched selection ⇒ quality is the", "+  operative variable. *Result: 464.0 vs 320.3. Not falsified.*", "+- **(b) Register balance is inert.** Global top-k on the same score, ignoring", "+  registers, matching the balanced selection. *Result: 388.6 vs 354.8 at matched", "+  gates — balance is worth ~9%. Not falsified.*", " - **(c) The gain is dev-topical, not register-level.** The profile is fitted on the", "-  disclosed dev sample, so its edge could be topic leakage (documents about", "-  Shackleton because the dev block is), which would not transfer to the hidden", "-  sample. Test: refit the profile on only the **first half** of each register block", "-  and evaluate on the **unseen second half**; compare against a profile restricted", "-  to features occurring in ≥8 distinct target segments (register/style vocabulary", "-  only, topical terms removed). If the edge were topical, the unrestricted profile", "-  would collapse on unseen text. *Result: 362.5 (unrestricted) vs 363.9", "-  (topic-stripped) — statistically indistinguishable, so the signal is register-level", "-  and survives a disjoint sample. Not falsified; I kept the simpler unrestricted", "-  profile.*", "+  disclosed dev sample, so its edge could be topic leakage (picking documents about", "+  Shackleton because a dev article is), which would not transfer to the hidden sample.", "+  Test: refit the profile on the **first half** of each register block only and", "+  evaluate on the **unseen second half**, against a profile restricted to features", "+  occurring in ≥8 distinct target segments (register/style vocabulary only, topical", "+  terms stripped). If the edge were topical, the unrestricted profile would collapse", "+  on unseen text. *Result: 362.5 (unrestricted) vs 363.9 (topic-stripped) —", "+  indistinguishable, so the signal is register-level and survives a disjoint sample.", "+  Not falsified; I kept the simpler unrestricted profile.*", " - **(d) Mixture-pushing keeps helping.** If more of the highest-loss register were", "-  always better, reweighting toward Wikipedia should keep improving things. It does", "-  not: 31/29/20/21 → 360.5, 37/35/14/13 → 379.1, 45/40/8/7 → 380.3. The target", "-  mixture, not the loss-weighted mixture, is the right allocation — consistent with", "-  the claim and a real constraint on it.", "+  always better, reweighting toward Wikipedia should keep improving things. *It does", "+  not: 31/29/20/21 → 360.5, 37/35/14/13 → 379.1, 45/40/8/7 → 380.3.* The target's own", "+  25/25/25/25 mixture is the right allocation — a real constraint on the claim.", "+- **(e) The result is a lucky point in gate-threshold space.** Three further variants", "+  around the chosen configuration: looser coverage/line-uniqueness 321.0, rank", "+  ensemble of two feature normalisations 324.7, sqrt-length score normalisation 332.3;", "+  and stepping *past* the optimum on every dimension at once regresses to 342.4. The", "+  submitted setting sits on a plateau, not a spike.", " ", " ## Transfer", " ", "-What transfers is the recipe, not the id list: **decode the disclosed target, split", "-it into its constituent registers, fit per-register importance weights against the", "-pool background, and interleave the per-register rankings to fill the budget.** It", "-needs no labels, no reference model and no GPU — the full pipeline over 182k", "-documents runs in ~40 s on 14 CPU cores, so it scales to pools far larger than this", "-one.", "+What transfers is the recipe, not the id list: **decode the disclosed target, split it", "+into its constituent registers, fit per-register importance weights against the pool", "+background, and interleave the per-register rankings to fill the budget.** No labels,", "+no reference model, no GPU — the full pipeline over 182k documents runs in ~60 s on 14", "+CPU cores, so it scales to far larger pools.", " ", " Expected limits when transferred:", "-- The gain shrinks as the pool's natural mixture approaches the target's; here the", "-  pool is dominated by low-value web chatter, which is why the margin is large.", "-- The gain is capped by surface forms absent from the pool (observable 2). If a", "-  target register's tokenization/markup does not exist in the pool, selection cannot", "-  recover it; that block's loss stays high and dominates the average.", "-- Interleaving by rank depth (not by equal token share) matters whenever registers", "-  differ in how deeply the pool supports them: equalising tokens drags the thin", "-  registers further down their rankings (354.8 → 361.2 when switched).", "-- Gates should be set as loose as the scorer tolerates; they are a guard against", "-  unscoreable documents, not a quality mechanism.", "+- **Audit your filters per register, not in aggregate.** The single largest avoidable", "+  loss here came from format gates that looked register-neutral and were not", "+  (observable 3). Any filter tuned on prose will quietly delete code, tables and", "+  markup — and therefore any target register made of them.", "+- Gates cannot be dropped either: with length+coverage only, dev PPL is 381.4, and", "+  with no gates 387.1, because unscoreable junk then outranks real documents on the", "+  match score. Keep every dimension, set each permissively.", "+- Interleave by **rank depth**, not equal token share: equalising tokens drags the", "+  registers the pool supports thinly (news, Q&A) further down their rankings and costs", "+  ~2% (354.8 → 361.2).", "+- Near-duplicate removal is worth ~9 PPL here (329.7 with dedup disabled) — small", "+  relative to register match, but free.", "+- The gain shrinks as the pool's natural mixture approaches the target's, and is", "+  capped by surface forms the pool lacks (observable 2). If a target register's", "+  tokenization or markup does not exist in the pool, selection cannot recover it.", " ", "-## Result summary (dev PPL, frozen 30M GPT, 12M tokens, identical recipe)", "+## Result summary (dev PPL, frozen 30M GPT, 12M tokens, identical recipe, seed 1337)", " ", " | selection | dev PPL |", " |---|---|", "-| random pool sample (do-nothing baseline) | 457.5 |", " | format-quality gates only, random within | 464.0 |", "+| random pool sample (do-nothing baseline) | 457.5 |", "+| register-matched, strict gates | 391.5 |", " | global top-k on match score (no register balance) | 388.6 |", "-| balanced match, moderate gates | 354.8 |", "-| balanced match, light gates | 338.9 |", "-| **balanced match, very light gates (submitted)** | **320.3** |", "+| register-matched, no gates | 387.1 |", "+| register-matched, equal token share instead of rank depth | 361.2 |", "+| register-matched, moderate gates | 354.8 |", "+| register-matched, light gates | 338.9 |", "+| register-matched, dedup disabled | 329.7 |", "+| **register-matched, very light gates (submitted)** | **320.3** |", " ", "+Per-register PPL of the submitted selection: wiki 763.3, web-prose 284.1, news 293.7,", "+Q&A 164.8.", "+", " Submitted selection re-verified end-to-end through the official path", "-(`pack_selection.py` → `train_nano.py`): 12,000,000 tokens packed from 24,728 ids,", "-`HELDOUT_PPL 320.316`. That is **30.0% below the random-selection baseline**.", "-Near-duplicate removal accounts for ~9 PPL of it (dedup disabled: 329.7).", "+(`pack_selection.py` → `train_nano.py`, unmodified): 12,000,000 tokens packed from", "+24,728 ordered ids, `HELDOUT_PPL 320.316`. `curate.py` re-run from a cold cache", "+reproduces `selection.json` exactly."]}], "originalFile": "# Claim: at a fixed token budget, held-out perplexity is set by *register-mixture match*, not by \"quality\"\n\n## Hypothesis\n\nFor a fixed 12M-token budget drawn from a 200M-token raw web pool, held-out\nperplexity on a broad multi-register target is governed almost entirely by how well\nthe selection's **register mixture** matches the target's, and hardly at all by\ngeneric document-quality filtering.\n\nOperationally: score every pool document with a per-register Naive-Bayes /\nimportance-weight log-ratio against the *disclosed* target\n(`s_r(d) = (1/N_d) Σ_f c_f(d) log[p_r(f)/p_pool(f)]`, unigram+bigram features),\nthen fill the budget by interleaving the four per-register rankings. I predicted this\nwould beat a random selection by ≳20% and beat an *unbalanced* global top-k on the\nsame score by ≳5%, while heuristic junk-filtering alone would land at random.\n\n## Mechanism, and what it predicts *other than* the final perplexity\n\nThe pool is Common-Crawl-style web text; the target is four 250k-token blocks —\nWikiText-formatted Wikipedia, high-quality web prose, newswire, StackExchange Q&A.\nThe mechanism is that each target block is predicted by data of *its own register*,\nand registers are not interchangeable, so the budget must be spread over all four.\nThree observables follow, all measurable without looking at the final score:\n\n1. **Junk filters alone ≈ random.** If quality is not the operative variable, a\n   selection that applies only my format gates and then picks randomly inside them\n   should not beat random selection. → *Observed: gates-only 464.0 vs random 457.5\n   dev PPL — no gain (slightly worse).*\n2. **The per-register loss breakdown is strongly non-uniform, worst on Wikipedia,\n   for a surface-form reason.** The target's Wikipedia block is WikiText-tokenized\n   (spaced punctuation, ` @-@ `, ` @,@ `). I grepped the pool: **0 of 182,016\n   documents** contain that form (and only 69 contain `</p>`, so the Q&A block's\n   HTML form is absent too). No selection can supply it, so that block should carry\n   a much higher loss than the rest. → *Observed on the balanced selection:\n   wiki 691 vs web-prose 266, news 283, Q&A 303.* This is the binding ceiling on\n   the achievable score, and it is a property of the pool, not of the criterion.\n3. **Tightening the gates should *hurt*, not help.** If matching is what matters,\n   gates only shrink the candidate set and force the match score deeper down each\n   register's ranking. → *Observed: strict gates 391.5 → moderate 354.8 → light\n   338.9 → very light 320.3 dev PPL. The effect is one-sided but not unbounded:\n   removing the format dimensions altogether (length + English-coverage only) gives\n   381.4 and no gates at all 387.1, so the gates must stay in place — just permissive.*\n\n## Falsification\n\nThe claim would have been falsified by any of:\n\n- **(a) Quality-only wins.** Gates-only ≈ balanced-match ⇒ quality is the variable.\n  *Result: 464.0 vs 320.3. Not falsified.*\n- **(b) Balance is inert.** Global top-k on the same score, ignoring registers,\n  matching balanced selection. *Result: 388.6 vs 354.8 under identical gates —\n  balance is worth ~9%. Not falsified.*\n- **(c) The gain is dev-topical, not register-level.** The profile is fitted on the\n  disclosed dev sample, so its edge could be topic leakage (documents about\n  Shackleton because the dev block is), which would not transfer to the hidden\n  sample. Test: refit the profile on only the **first half** of each register block\n  and evaluate on the **unseen second half**; compare against a profile restricted\n  to features occurring in ≥8 distinct target segments (register/style vocabulary\n  only, topical terms removed). If the edge were topical, the unrestricted profile\n  would collapse on unseen text. *Result: 362.5 (unrestricted) vs 363.9\n  (topic-stripped) — statistically indistinguishable, so the signal is register-level\n  and survives a disjoint sample. Not falsified; I kept the simpler unrestricted\n  profile.*\n- **(d) Mixture-pushing keeps helping.** If more of the highest-loss register were\n  always better, reweighting toward Wikipedia should keep improving things. It does\n  not: 31/29/20/21 → 360.5, 37/35/14/13 → 379.1, 45/40/8/7 → 380.3. The target\n  mixture, not the loss-weighted mixture, is the right allocation — consistent with\n  the claim and a real constraint on it.\n\n## Transfer\n\nWhat transfers is the recipe, not the id list: **decode the disclosed target, split\nit into its constituent registers, fit per-register importance weights against the\npool background, and interleave the per-register rankings to fill the budget.** It\nneeds no labels, no reference model and no GPU — the full pipeline over 182k\ndocuments runs in ~40 s on 14 CPU cores, so it scales to pools far larger than this\none.\n\nExpected limits when transferred:\n- The gain shrinks as the pool's natural mixture approaches the target's; here the\n  pool is dominated by low-value web chatter, which is why the margin is large.\n- The gain is capped by surface forms absent from the pool (observable 2). If a\n  target register's tokenization/markup does not exist in the pool, selection cannot\n  recover it; that block's loss stays high and dominates the average.\n- Interleaving by rank depth (not by equal token share) matters whenever registers\n  differ in how deeply the pool supports them: equalising tokens drags the thin\n  registers further down their rankings (354.8 → 361.2 when switched).\n- Gates should be set as loose as the scorer tolerates; they are a guard against\n  unscoreable documents, not a quality mechanism.\n\n## Result summary (dev PPL, frozen 30M GPT, 12M tokens, identical recipe)\n\n| selection | dev PPL |\n|---|---|\n| random pool sample (do-nothing baseline) | 457.5 |\n| format-quality gates only, random within | 464.0 |\n| global top-k on match score (no register balance) | 388.6 |\n| balanced match, moderate gates | 354.8 |\n| balanced match, light gates | 338.9 |\n| **balanced match, very light gates (submitted)** | **320.3** |\n\nSubmitted selection re-verified end-to-end through the official path\n(`pack_selection.py` → `train_nano.py`): 12,000,000 tokens packed from 24,728 ids,\n`HELDOUT_PPL 320.316`. That is **30.0% below the random-selection baseline**.\nNear-duplicate removal accounts for ~9 PPL of it (dedup disabled: 329.7).\n", "userModified": false}

prose pool 92158 code pool 135660
perreg: 24276 ids, est tokens/register [6602537, 6601087, 6600603, 6600039]
[1]+  Done                    TRAINER=/workspace/diag_train.py nohup python3 eval_batch.py perreg > /tmp/evalpr.log 2>&1
{"train_npy": "/tmp/train_perreg.npy", "heldout_ppl": 324.05637617010615, "mean_loss": 5.780917501181013, "max_iters": 3000, "seed": 1337, "train_s": 50.3, "per_register_ppl": {"wiki": 772.9096061504171, "webprose": 281.81613506727075, "news": 304.989604509411, "qa": 165.215305033339}}
TIME: 1885

[stdout]
prose pool 92158 code pool 135660
perreg: 24276 ids, est tokens/register [6602537, 6601087, 6600603, 6600039]
[1]+  Done                    TRAINER=/workspace/diag_train.py nohup python3 eval_batch.py perreg > /tmp/evalpr.log 2>&1
{"train_npy": "/tmp/train_perreg.npy", "heldout_ppl": 324.05637617010615, "mean_loss": 5.780917501181013, "max_iters": 3000, "seed": 1337, "train_s": 50.3, "per_register_ppl": {"wiki": 772.9096061504171, "webprose": 281.81613506727075, "news": 304.989604509411, "qa": 165.215305033339}}
TIME: 1885
[metadata] {"noOutputExpected": false}

claim.md updated
total 188
drwxr-xr-x 1 root root    340 Aug  4 00:18 .
drwxr-xr-x 1 root root    400 Aug  4 00:22 ..
-rw-r--r-- 1 root root  15588 Aug  4 00:18 curate.py
-rw-r--r-- 1 root root 175733 Aug  4 00:11 selection.json
selection.json: 24728 ids | unique: True | range ok: True | type: list int
TIME: 1865

[stdout]
claim.md updated
total 188
drwxr-xr-x 1 root root    340 Aug  4 00:18 .
drwxr-xr-x 1 root root    400 Aug  4 00:22 ..
-rw-r--r-- 1 root root  15588 Aug  4 00:18 curate.py
-rw-r--r-- 1 root root 175733 Aug  4 00:11 selection.json
selection.json: 24728 ids | unique: True | range ok: True | type: list int
TIME: 1865
[metadata] {"noOutputExpected": false}

- **(e) The result is a lucky point in gate-threshold space.** Three further variants
  around the chosen configuration: looser coverage/line-uniqueness 321.0, rank
  ensemble of two feature normalisations 324.7, sqrt-length score normalisation 332.3;
  and stepping *past* the optimum on every dimension at once regresses to 342.4. The
  submitted setting sits on a plateau, not a spike.
- **(f) Gates should be register-specific.** A natural reading of observable 3 is that
  each register wants its own filter. Tested and rejected: prose-strict + Q&A-loose
  scores 324.1 vs 320.3 uniform-loose. The mechanism is about which documents can
  *reach* a register's quota, not about per-register filter tuning.

## Transfer
curate.py OK
  145 claim.md
  379 submission/curate.py
  524 total

[stdout]
- **(e) The result is a lucky point in gate-threshold space.** Three further variants
  around the chosen configuration: looser coverage/line-uniqueness 321.0, rank
  ensemble of two feature normalisations 324.7, sqrt-length score normalisation 332.3;
  and stepping *past* the optimum on every dimension at once regresses to 342.4. The
  submitted setting sits on a plateau, not a spike.
- **(f) Gates should be register-specific.** A natural reading of observable 3 is that
  each register wants its own filter. Tested and rejected: prose-strict + Q&A-loose
  scores 324.1 vs 320.3 uniform-loose. The mechanism is about which documents can
  *reach* a register's quota, not about per-register filter tuning.

## Transfer
curate.py OK
  145 claim.md
  379 submission/curate.py
  524 total
[metadata] {"noOutputExpected": false}