claude-opus-5

File created successfully at: /workspace/claim.md (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/workspace/claim.md", "content": "# Claim: register-balanced importance weighting beats a single global quality score\n\n## Hypothesis\n\nFor a fixed, small token budget (12M of a 200M-token web pool) evaluated by\nperplexity on a **multi-register** target, the binding constraint is not\ndocument \"quality\" in the abstract but **coverage of every register in the\ntarget mixture**. A single global quality/importance ranking silently\nreallocates the budget toward whichever target register the pool is richest in,\nstarving the others; perplexity is a mean over all registers, so the starved\nones dominate the loss.\n\nI therefore claim: estimating a *separate* importance weight per target register\nand filling the budget under **equal per-register token quotas** beats the same\nimportance estimator applied as one global ranking, which in turn beats random\nselection.\n\n## Mechanism (observable predictions other than final perplexity)\n\nThe mechanism is that the disclosed dev target is a *mixture of four contiguous\nregister blocks*, and DSIR-style log-likelihood ratios are register-specific.\nObservables:\n\n1. **The target is four blocks, not one distribution.** Decoding\n   `multi_dev.npy` at 0 / 25 / 50 / 75 / 90% yields, in order: encyclopedic\n   prose (Shackleton's Antarctic expedition), technical/scientific abstracts\n   (evacuated-tube transit), news (entertainment wire copy), and\n   StackOverflow-style Q&A with `<p>`/`<code>` markup at both 75% and 90%.\n   *Confirmed by inspection.*\n2. **Per-register weights disagree about which documents are good.** The four\n   score rows in `S` rank different documents first: register 0 puts an\n   encyclopedic battle article on top, register 2 an Indian-politics news wire,\n   register 3 an HTML/JS-laden page. If one global \"quality\" axis explained the\n   data, these argmaxes would coincide. *Confirmed.*\n3. **Prose-oriented gates would delete the Q&A register.** The code/markup\n   register fails alphabetic-ratio, digit-ratio and pipe-density gates that the\n   prose registers pass — which is why register 3 is gated by length/stopword/\n   boilerplate checks only. Applying prose gates to all four cuts the eligible\n   pool by ~5% overall but disproportionately on code-bearing documents.\n4. **A discriminative classifier latches onto topic, not register.** A hashed\n   uni+bigram logistic classifier (target vs. pool) reaches 100% train accuracy\n   in 3 epochs and its top-ranked documents collapse onto one topic (Indian\n   politics) plus an OCR'd 1900s newspaper. The smooth generative\n   log-ratio does not collapse this way. *Confirmed by inspection; the\n   classifier also scored worse (388 vs 354 dev PPL).*\n\n## Falsification\n\nThe claim is falsified if, holding the importance estimator, gates and\ndedup fixed, the **global** ranking matches or beats the **register-quota**\nranking. It is also falsified if diversity-by-noise works as well as\ndiversity-by-quota (i.e. if temperature-sampling the global score recovers the\nquota result), which would mean the win is generic entropy, not register\ncoverage.\n\nBoth were run with the frozen trainer on the dev target (12M tokens, 30M GPT):\n\n| selection | dev PPL |\n|---|---|\n| random (do-nothing baseline) | 459.8 |\n| global discriminative classifier + gates | 388.3 |\n| global DSIR log-ratio + gates | 354.5 |\n| global DSIR, temperature-sampled (T=0.5) for diversity | 379.0 |\n| global DSIR scored over 8k-char documents, stricter gates | 367.6 |\n| **per-register DSIR, equal 4x8M quotas (submitted)** | **325.8** |\n\nNot falsified: the quota version beats the best global version by 29 PPL, and\nsampling-based diversity is *worse* than sharp global ranking (379 > 354) — so\nthe gain is specifically register coverage, not added entropy.\n\nHonest limits: the gate thresholds are sensitive. Changing the stopword set and\nadding a mild type/token-ratio filter, at nearly identical pass counts\n(138.8k vs 138.2k documents), moved dev PPL from 325.8 to 354.6. That swing is\nas large as the effect I am claiming, so the *specific numbers* in the gates are\nnot load-bearing evidence — only the quota-vs-global contrast, which I measured\nwith gates held fixed, is. An aggressive TTR gate (pool halved to 60k) scored\n366.9, consistent with over-filtering costing more than it removes.\n\n## Transfer\n\n- **Any fixed-budget curation against a known-mixture evaluation.** Fit one\n  importance model per evaluation component and allocate the budget by quota\n  rather than by a single scalar \"quality\". This applies directly to\n  instruction-tuning mixes and to domain-balanced pretraining.\n- **Gate per register, not globally.** Filters calibrated on natural prose\n  (symbol ratios, word length, punctuation density) systematically delete code,\n  math and tabular data. If the target contains those registers, the gate must\n  be conditioned on the register the document is being selected for.\n- **Prefer generative log-ratios to discriminative classifiers** when the\n  positive set is small: the classifier saturates and encodes topic, and its\n  top-of-ranking is where the budget is actually spent, so its failure mode is\n  maximally costly.\n- **Does not transfer** to targets whose registers are unknown or unobservable;\n  the whole method rests on the evaluation target being samplable. Where it\n  isn't, the fallback is the global log-ratio, which still bought 105 PPL over\n  random here.\n", "structuredPatch": [], "originalFile": null, "userModified": false}

pool=182016 docs
gates: base=138241 prose=130040
selected 25517 docs; est tokens per register [4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0]M
reg8: HELDOUT_PPL 352.082  loss 5.8639  train_s 54  /tmp/tr_r8.npy
3087

[stdout]
pool=182016 docs
gates: base=138241 prose=130040
selected 25517 docs; est tokens per register [4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0]M
reg8: HELDOUT_PPL 352.082  loss 5.8639  train_s 54  /tmp/tr_r8.npy
3087
[metadata] {"noOutputExpected": false}

The file /workspace/claim.md has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/claim.md", "oldString": "| **per-register DSIR, equal 4x8M quotas (submitted)** | **325.8** |", "newString": "| per-register DSIR, 8-way split, equal 8x4M quotas | 352.1 |\n| **per-register DSIR, equal 4x8M quotas (submitted)** | **325.8** |", "originalFile": "# Claim: register-balanced importance weighting beats a single global quality score\n\n## Hypothesis\n\nFor a fixed, small token budget (12M of a 200M-token web pool) evaluated by\nperplexity on a **multi-register** target, the binding constraint is not\ndocument \"quality\" in the abstract but **coverage of every register in the\ntarget mixture**. A single global quality/importance ranking silently\nreallocates the budget toward whichever target register the pool is richest in,\nstarving the others; perplexity is a mean over all registers, so the starved\nones dominate the loss.\n\nI therefore claim: estimating a *separate* importance weight per target register\nand filling the budget under **equal per-register token quotas** beats the same\nimportance estimator applied as one global ranking, which in turn beats random\nselection.\n\n## Mechanism (observable predictions other than final perplexity)\n\nThe mechanism is that the disclosed dev target is a *mixture of four contiguous\nregister blocks*, and DSIR-style log-likelihood ratios are register-specific.\nObservables:\n\n1. **The target is four blocks, not one distribution.** Decoding\n   `multi_dev.npy` at 0 / 25 / 50 / 75 / 90% yields, in order: encyclopedic\n   prose (Shackleton's Antarctic expedition), technical/scientific abstracts\n   (evacuated-tube transit), news (entertainment wire copy), and\n   StackOverflow-style Q&A with `<p>`/`<code>` markup at both 75% and 90%.\n   *Confirmed by inspection.*\n2. **Per-register weights disagree about which documents are good.** The four\n   score rows in `S` rank different documents first: register 0 puts an\n   encyclopedic battle article on top, register 2 an Indian-politics news wire,\n   register 3 an HTML/JS-laden page. If one global \"quality\" axis explained the\n   data, these argmaxes would coincide. *Confirmed.*\n3. **Prose-oriented gates would delete the Q&A register.** The code/markup\n   register fails alphabetic-ratio, digit-ratio and pipe-density gates that the\n   prose registers pass — which is why register 3 is gated by length/stopword/\n   boilerplate checks only. Applying prose gates to all four cuts the eligible\n   pool by ~5% overall but disproportionately on code-bearing documents.\n4. **A discriminative classifier latches onto topic, not register.** A hashed\n   uni+bigram logistic classifier (target vs. pool) reaches 100% train accuracy\n   in 3 epochs and its top-ranked documents collapse onto one topic (Indian\n   politics) plus an OCR'd 1900s newspaper. The smooth generative\n   log-ratio does not collapse this way. *Confirmed by inspection; the\n   classifier also scored worse (388 vs 354 dev PPL).*\n\n## Falsification\n\nThe claim is falsified if, holding the importance estimator, gates and\ndedup fixed, the **global** ranking matches or beats the **register-quota**\nranking. It is also falsified if diversity-by-noise works as well as\ndiversity-by-quota (i.e. if temperature-sampling the global score recovers the\nquota result), which would mean the win is generic entropy, not register\ncoverage.\n\nBoth were run with the frozen trainer on the dev target (12M tokens, 30M GPT):\n\n| selection | dev PPL |\n|---|---|\n| random (do-nothing baseline) | 459.8 |\n| global discriminative classifier + gates | 388.3 |\n| global DSIR log-ratio + gates | 354.5 |\n| global DSIR, temperature-sampled (T=0.5) for diversity | 379.0 |\n| global DSIR scored over 8k-char documents, stricter gates | 367.6 |\n| **per-register DSIR, equal 4x8M quotas (submitted)** | **325.8** |\n\nNot falsified: the quota version beats the best global version by 29 PPL, and\nsampling-based diversity is *worse* than sharp global ranking (379 > 354) — so\nthe gain is specifically register coverage, not added entropy.\n\nHonest limits: the gate thresholds are sensitive. Changing the stopword set and\nadding a mild type/token-ratio filter, at nearly identical pass counts\n(138.8k vs 138.2k documents), moved dev PPL from 325.8 to 354.6. That swing is\nas large as the effect I am claiming, so the *specific numbers* in the gates are\nnot load-bearing evidence — only the quota-vs-global contrast, which I measured\nwith gates held fixed, is. An aggressive TTR gate (pool halved to 60k) scored\n366.9, consistent with over-filtering costing more than it removes.\n\n## Transfer\n\n- **Any fixed-budget curation against a known-mixture evaluation.** Fit one\n  importance model per evaluation component and allocate the budget by quota\n  rather than by a single scalar \"quality\". This applies directly to\n  instruction-tuning mixes and to domain-balanced pretraining.\n- **Gate per register, not globally.** Filters calibrated on natural prose\n  (symbol ratios, word length, punctuation density) systematically delete code,\n  math and tabular data. If the target contains those registers, the gate must\n  be conditioned on the register the document is being selected for.\n- **Prefer generative log-ratios to discriminative classifiers** when the\n  positive set is small: the classifier saturates and encodes topic, and its\n  top-of-ranking is where the budget is actually spent, so its failure mode is\n  maximally costly.\n- **Does not transfer** to targets whose registers are unknown or unobservable;\n  the whole method rests on the evaluation target being samplable. Where it\n  isn't, the fallback is the global log-ratio, which still bought 105 PPL over\n  random here.\n", "structuredPatch": [{"oldStart": 62, "oldLines": 6, "newStart": 62, "newLines": 7, "lines": [" | global DSIR log-ratio + gates | 354.5 |", " | global DSIR, temperature-sampled (T=0.5) for diversity | 379.0 |", " | global DSIR scored over 8k-char documents, stricter gates | 367.6 |", "+| per-register DSIR, 8-way split, equal 8x4M quotas | 352.1 |", " | **per-register DSIR, equal 4x8M quotas (submitted)** | **325.8** |", " ", " Not falsified: the quota version beats the best global version by 29 PPL, and"]}], "userModified": false, "replaceAll": false}

The file /workspace/claim.md has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/claim.md", "oldString": "Not falsified: the quota version beats", "newString": "The 8-way row is a further check on the mechanism: splitting the dev stream into\neight blocks instead of four *loses* 26 PPL. If the win came from generic\nfine-grained balancing, 8 would beat 4. It does not, because only the 4-way\nboundaries coincide with the actual register block boundaries — an 8-way split\ncuts each register in half and spends quota on halves of the same distribution.\n\nNot falsified: the quota version beats", "originalFile": "# Claim: register-balanced importance weighting beats a single global quality score\n\n## Hypothesis\n\nFor a fixed, small token budget (12M of a 200M-token web pool) evaluated by\nperplexity on a **multi-register** target, the binding constraint is not\ndocument \"quality\" in the abstract but **coverage of every register in the\ntarget mixture**. A single global quality/importance ranking silently\nreallocates the budget toward whichever target register the pool is richest in,\nstarving the others; perplexity is a mean over all registers, so the starved\nones dominate the loss.\n\nI therefore claim: estimating a *separate* importance weight per target register\nand filling the budget under **equal per-register token quotas** beats the same\nimportance estimator applied as one global ranking, which in turn beats random\nselection.\n\n## Mechanism (observable predictions other than final perplexity)\n\nThe mechanism is that the disclosed dev target is a *mixture of four contiguous\nregister blocks*, and DSIR-style log-likelihood ratios are register-specific.\nObservables:\n\n1. **The target is four blocks, not one distribution.** Decoding\n   `multi_dev.npy` at 0 / 25 / 50 / 75 / 90% yields, in order: encyclopedic\n   prose (Shackleton's Antarctic expedition), technical/scientific abstracts\n   (evacuated-tube transit), news (entertainment wire copy), and\n   StackOverflow-style Q&A with `<p>`/`<code>` markup at both 75% and 90%.\n   *Confirmed by inspection.*\n2. **Per-register weights disagree about which documents are good.** The four\n   score rows in `S` rank different documents first: register 0 puts an\n   encyclopedic battle article on top, register 2 an Indian-politics news wire,\n   register 3 an HTML/JS-laden page. If one global \"quality\" axis explained the\n   data, these argmaxes would coincide. *Confirmed.*\n3. **Prose-oriented gates would delete the Q&A register.** The code/markup\n   register fails alphabetic-ratio, digit-ratio and pipe-density gates that the\n   prose registers pass — which is why register 3 is gated by length/stopword/\n   boilerplate checks only. Applying prose gates to all four cuts the eligible\n   pool by ~5% overall but disproportionately on code-bearing documents.\n4. **A discriminative classifier latches onto topic, not register.** A hashed\n   uni+bigram logistic classifier (target vs. pool) reaches 100% train accuracy\n   in 3 epochs and its top-ranked documents collapse onto one topic (Indian\n   politics) plus an OCR'd 1900s newspaper. The smooth generative\n   log-ratio does not collapse this way. *Confirmed by inspection; the\n   classifier also scored worse (388 vs 354 dev PPL).*\n\n## Falsification\n\nThe claim is falsified if, holding the importance estimator, gates and\ndedup fixed, the **global** ranking matches or beats the **register-quota**\nranking. It is also falsified if diversity-by-noise works as well as\ndiversity-by-quota (i.e. if temperature-sampling the global score recovers the\nquota result), which would mean the win is generic entropy, not register\ncoverage.\n\nBoth were run with the frozen trainer on the dev target (12M tokens, 30M GPT):\n\n| selection | dev PPL |\n|---|---|\n| random (do-nothing baseline) | 459.8 |\n| global discriminative classifier + gates | 388.3 |\n| global DSIR log-ratio + gates | 354.5 |\n| global DSIR, temperature-sampled (T=0.5) for diversity | 379.0 |\n| global DSIR scored over 8k-char documents, stricter gates | 367.6 |\n| per-register DSIR, 8-way split, equal 8x4M quotas | 352.1 |\n| **per-register DSIR, equal 4x8M quotas (submitted)** | **325.8** |\n\nNot falsified: the quota version beats the best global version by 29 PPL, and\nsampling-based diversity is *worse* than sharp global ranking (379 > 354) — so\nthe gain is specifically register coverage, not added entropy.\n\nHonest limits: the gate thresholds are sensitive. Changing the stopword set and\nadding a mild type/token-ratio filter, at nearly identical pass counts\n(138.8k vs 138.2k documents), moved dev PPL from 325.8 to 354.6. That swing is\nas large as the effect I am claiming, so the *specific numbers* in the gates are\nnot load-bearing evidence — only the quota-vs-global contrast, which I measured\nwith gates held fixed, is. An aggressive TTR gate (pool halved to 60k) scored\n366.9, consistent with over-filtering costing more than it removes.\n\n## Transfer\n\n- **Any fixed-budget curation against a known-mixture evaluation.** Fit one\n  importance model per evaluation component and allocate the budget by quota\n  rather than by a single scalar \"quality\". This applies directly to\n  instruction-tuning mixes and to domain-balanced pretraining.\n- **Gate per register, not globally.** Filters calibrated on natural prose\n  (symbol ratios, word length, punctuation density) systematically delete code,\n  math and tabular data. If the target contains those registers, the gate must\n  be conditioned on the register the document is being selected for.\n- **Prefer generative log-ratios to discriminative classifiers** when the\n  positive set is small: the classifier saturates and encodes topic, and its\n  top-of-ranking is where the budget is actually spent, so its failure mode is\n  maximally costly.\n- **Does not transfer** to targets whose registers are unknown or unobservable;\n  the whole method rests on the evaluation target being samplable. Where it\n  isn't, the fallback is the global log-ratio, which still bought 105 PPL over\n  random here.\n", "structuredPatch": [{"oldStart": 65, "oldLines": 6, "newStart": 65, "newLines": 12, "lines": [" | per-register DSIR, 8-way split, equal 8x4M quotas | 352.1 |", " | **per-register DSIR, equal 4x8M quotas (submitted)** | **325.8** |", " ", "+The 8-way row is a further check on the mechanism: splitting the dev stream into", "+eight blocks instead of four *loses* 26 PPL. If the win came from generic", "+fine-grained balancing, 8 would beat 4. It does not, because only the 4-way", "+boundaries coincide with the actual register block boundaries — an 8-way split", "+cuts each register in half and spends quota on halves of the same distribution.", "+", " Not falsified: the quota version beats the best global version by 29 PPL, and", " sampling-based diversity is *worse* than sharp global ranking (379 > 354) — so", " the gain is specifically register coverage, not added entropy."]}], "userModified": false, "replaceAll": false}