claude-opus-5

Command running in background with ID: badfoqckg. Output is being written to: /tmp/claude-0/-workspace/2123b707-8a58-4911-b84d-2ce5f3aed78d/tasks/badfoqckg.output. You will be notified when it completes. To check interim output, use Read on that file path.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "badfoqckg"}

33852 33852 1 181981

[stdout]
33852 33852 1 181981
[metadata] {"noOutputExpected": false}

File created successfully at: /workspace/claim.md (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/workspace/claim.md", "content": "# Claim: register-conditional target likeness beats undifferentiated \"quality\"\n\n## Hypothesis\n\nWhen the evaluation target is a **broad, equal-weight mixture of registers**\n(encyclopedic / high-quality web prose / news / technical Q&A) and the training\nbudget is a small slice of the pool (12M of ~200M tokens), the selection rule that\nminimises held-out perplexity is **not** a single global quality score. It is:\n\n> *k* separate target-likeness discriminators, one per register of the disclosed\n> target, applied under a shared non-prose/near-duplicate filter, with the\n> **consumed token prefix** balanced across registers.\n\nTwo claims follow, and both are what the design is betting on:\n\n1. **Register balance in the consumed prefix matters more than raw score.** A\n   single global ranking concentrates the budget on whichever register is most\n   linearly separable from generic web text, starving the others.\n2. **The right balance is not uniform: it tilts toward the register the model is\n   worst at**, because the metric averages *cross-entropy* (not perplexity) over\n   registers, so absolute loss is dominated by the hardest register.\n\n## Mechanism — and an observable that is not the final perplexity\n\nThe mechanism is register-conditional loss reallocation. The predicted\n**observable is the per-register perplexity vector**, measured by evaluating the\nsame trained model separately on each quarter of `multi_dev.npy`. Predictions\nmade *before* the final run and confirmed:\n\n| selection | wiki | hq-web | news | Q&A | overall |\n|---|---|---|---|---|---|\n| random pool (baseline) | — | — | — | — | **469.9** |\n| classifier + gate, doc-balanced (v1) | — | — | — | — | **340.7** |\n| + near-dup removal, token-balanced 25/25/25/25 | 756.0 | 266.0 | 305.6 | 183.5 | **326.0** |\n| + tilt to encyclopedic 40/20/20/20 | 747.6 | 266.7 | 306.7 | 178.7 | **323.5** |\n| over-tilt 55/15/15/15 | 705.8 | 268.7 | 306.1 | 200.5 | **328.7** |\n\nThree mechanism-level observables, all checkable without looking at the score:\n\n- **The register spread is huge and encyclopedic is the outlier** (756 vs 184).\n  Predicted cause: the encyclopedic quarter is raw wikitext-103 formatting\n  (` @-@ `, ` @,@ `, spaces before punctuation). Directly verified: **zero**\n  documents in the 182,016-document pool contain `@-@`, and only 133 have\n  majority spaced punctuation. So a large part of that 756 is a *tokenisation*\n  gap that no data selection can close — which is exactly why tilting budget\n  toward it yields only ~1% and over-tilting backfires.\n- **Tilting trades registers monotonically.** 25→40→55% encyclopedic share\n  monotonically lowers the wiki column (756→748→706) and monotonically raises\n  the Q&A column (183→179→201 — the Q&A minimum sits between). The overall\n  optimum is interior, at 40%. That is a mechanism signature, not a fit artefact.\n- **The discriminators are genuinely separating register, not length or\n  boilerplate.** Held-out AUC against unseen random pool documents:\n  0.997 / 0.928 / 0.983 / 1.000 for the four registers. Negatives are truncated to\n  the same 3,000-character prefix as positives so length cannot be the signal.\n\n## Falsification\n\nThe claim is wrong if any of these hold:\n\n1. **Balance is irrelevant.** Take the same gated, deduplicated candidate set and\n   rank by `max_d score_d` (one global ranking, no quotas). If that matches or\n   beats the register-balanced prefix, claim 1 is false. Prediction: it is worse,\n   and its per-register vector is visibly skewed — the most separable register's\n   perplexity drops while at least one other rises.\n2. **The tilt is noise.** If re-running 40/20/20/20 vs 25/25/25/25 under different\n   training seeds does not preserve the ordering, claim 2 is false. The measured\n   gap is 2.5 PPL (0.008 nats) — small, and this is the weakest leg of the claim.\n   The monotone per-register trend across three weightings is the real evidence;\n   the overall-PPL ordering alone would not be.\n3. **The classifier adds nothing over the surface gate.** Run the quality gate +\n   near-dup removal alone, filling the budget by random draw from survivors. If\n   that reaches ~325, the discriminators are doing no work and the whole gain is\n   a boiler-plate filter.\n4. **Diversity beats fidelity.** If Gumbel-perturbed sampling at high temperature\n   (broad, low-fidelity draw from the gated pool) beats hard top-k, then\n   target-likeness is not the operative variable — coverage is.\n\n## Transfer\n\n- **Scales with disclosed-target structure, not with this pool.** Any setting\n  where the eval target is a *known mixture* — a domain-specific model, an\n  instruction-tuning mix, a multilingual budget split — can use the same recipe:\n  segment the target, fit one cheap hashed-n-gram discriminator per segment, and\n  balance the *consumed* prefix rather than the candidate pool. No labels, no\n  reference LM, no GPU-hours: the whole curation runs in ~2 minutes on CPU+1 GPU\n  for 182k documents.\n- **The correction generalises.** \"Weight the mixture toward the register with\n  the highest cross-entropy, but only until the interior optimum\" applies\n  whenever the metric averages loss over heterogeneous slices. The size of the\n  useful tilt is bounded by how much of the slice's loss is *reachable* — here,\n  a formatting gap with no pool support capped it at ~1%.\n- **Known limit.** The gain is bounded by pool support. If a target register has\n  no analogue in the pool (as encyclopedic wikitext formatting does not here),\n  selection cannot manufacture it; the answer is a different pool or a\n  tokenisation-level fix, not a better filter. Reporting the per-register vector\n  is what makes that limit visible — an aggregate perplexity would hide it.\n", "structuredPatch": [], "originalFile": null, "userModified": false}

selected 30885 docs ~30.0M tokens -> /tmp/sel_T0.5.json
== T=0.5
PER_REGISTER [740.58, 272.09, 329.13, 369.74]
HELDOUT_PPL 396.065  loss 5.9816  train_s 55  /tmp/e_T0.5.npy
rem:
3838

[stdout]
selected 30885 docs ~30.0M tokens -> /tmp/sel_T0.5.json
== T=0.5
PER_REGISTER [740.58, 272.09, 329.13, 369.74]
HELDOUT_PPL 396.065  loss 5.9816  train_s 55  /tmp/e_T0.5.npy
rem:
3838
[metadata] {"noOutputExpected": false}

{"message":"Successfully stopped task: badfoqckg (bash /tmp/sweep2.sh 2>&1)","task_id":"badfoqckg","task_type":"local_bash","command":"bash /tmp/sweep2.sh 2>&1"}

[metadata] {"message": "Successfully stopped task: badfoqckg (bash /tmp/sweep2.sh 2>&1)", "task_id": "badfoqckg", "task_type": "local_bash", "command": "bash /tmp/sweep2.sh 2>&1"}

Command running in background with ID: bux90o2tg. Output is being written to: /tmp/claude-0/-workspace/2123b707-8a58-4911-b84d-2ce5f3aed78d/tasks/bux90o2tg.output. You will be notified when it completes. To check interim output, use Read on that file path.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "bux90o2tg"}

"""
Curate ~12M+ tokens of pretraining data from a raw web pool so that a 30M-param
GPT trained on the selection minimises held-out perplexity on a BROAD
high-quality English target.

STATED CRITERION
================
The disclosed target is an equal-weight mixture of four registers:
  (0) encyclopedic  (1) high-quality general/technical web prose
  (2) news          (3) technical Q&A
(read directly off the four quarters of `data/multi_dev.npy`).

We select by **domain-conditional target likeness under a quality gate**:

 1. QUALITY GATE - drop documents that are too short, boiler-plate heavy, or
    non-prose (character-class and mean-word-length statistics). These raise
    perplexity on any prose target regardless of topic.

 2. FOUR DISCRIMINATORS - for each register d, fit an L2-regularised logistic
    regression  p_d(target | doc)  on hashed word uni+bi-gram features, with the
    dev-target text of register d as positives and a uniform random pool sample
    as negatives ("generic web"). Every surviving pool document receives four
    scores.

 3. BALANCED ROUND-ROBIN - the four ranked lists are interleaved one document at
    a time. Because the trainer consumes our list in order until 12M tokens, this
    makes *the consumed prefix itself* an equal-weight four-register mixture,
    instead of letting the most separable register dominate.

Everything below is a deterministic function of pool.jsonl + multi_dev.npy.
No document id is ever named by hand.
"""
import json, re, os, sys, math, pickle, zlib
import numpy as np
import torch
from multiprocessing import Pool as MPPool
from transformers import GPT2TokenizerFast

POOL_PATH = "/workspace/data/pool.jsonl"
DEV_PATH  = "/workspace/data/multi_dev.npy"
OUT_PATH  = os.environ.get("CURATE_OUT", "/workspace/submission/selection.json")

SEED          = 0
NBITS         = 20                 # 2^20 hashed feature bins
NFEAT         = 1 << NBITS
MAXCHARS      = 3000               # prefix of each doc used for featurisation
N_NEG         = 40_000             # random pool docs = negative class
TARGET_TOKENS = 30_000_000         # emit ~2.5x the 12M budget
QUARTERS      = 4
# Token share per register. The evaluation averages cross-entropy equally over the
# four registers, so absolute loss is dominated by the register the model does
# WORST on (encyclopedic prose, which no pool document imitates exactly).
# Shifting budget toward it buys more absolute loss than equalising doc counts.
# Selection temperature: 0 = hard top-k by score; >0 = Gumbel-perturbed ranking,
# i.e. sampling without replacement with prob proportional to exp(score/T).
# A little stochasticity trades a small amount of target-likeness for topical
# diversity, which matters when the budget is 12M tokens of a 200M-token pool.
TEMP = float(os.environ.get("CURATE_T", "0"))
WEIGHTS = [float(x) for x in os.environ.get("CURATE_W", "0.40,0.20,0.20,0.20").split(",")]
EPOCHS        = 400

[stdout]
"""
Curate ~12M+ tokens of pretraining data from a raw web pool so that a 30M-param
GPT trained on the selection minimises held-out perplexity on a BROAD
high-quality English target.

STATED CRITERION
================
The disclosed target is an equal-weight mixture of four registers:
  (0) encyclopedic  (1) high-quality general/technical web prose
  (2) news          (3) technical Q&A
(read directly off the four quarters of `data/multi_dev.npy`).

We select by **domain-conditional target likeness under a quality gate**:

 1. QUALITY GATE - drop documents that are too short, boiler-plate heavy, or
    non-prose (character-class and mean-word-length statistics). These raise
    perplexity on any prose target regardless of topic.

 2. FOUR DISCRIMINATORS - for each register d, fit an L2-regularised logistic
    regression  p_d(target | doc)  on hashed word uni+bi-gram features, with the
    dev-target text of register d as positives and a uniform random pool sample
    as negatives ("generic web"). Every surviving pool document receives four
    scores.

 3. BALANCED ROUND-ROBIN - the four ranked lists are interleaved one document at
    a time. Because the trainer consumes our list in order until 12M tokens, this
    makes *the consumed prefix itself* an equal-weight four-register mixture,
    instead of letting the most separable register dominate.

Everything below is a deterministic function of pool.jsonl + multi_dev.npy.
No document id is ever named by hand.
"""
import json, re, os, sys, math, pickle, zlib
import numpy as np
import torch
from multiprocessing import Pool as MPPool
from transformers import GPT2TokenizerFast

POOL_PATH = "/workspace/data/pool.jsonl"
DEV_PATH  = "/workspace/data/multi_dev.npy"
OUT_PATH  = os.environ.get("CURATE_OUT", "/workspace/submission/selection.json")

SEED          = 0
NBITS         = 20                 # 2^20 hashed feature bins
NFEAT         = 1 << NBITS
MAXCHARS      = 3000               # prefix of each doc used for featurisation
N_NEG         = 40_000             # random pool docs = negative class
TARGET_TOKENS = 30_000_000         # emit ~2.5x the 12M budget
QUARTERS      = 4
# Token share per register. The evaluation averages cross-entropy equally over the
# four registers, so absolute loss is dominated by the register the model does
# WORST on (encyclopedic prose, which no pool document imitates exactly).
# Shifting budget toward it buys more absolute loss than equalising doc counts.
# Selection temperature: 0 = hard top-k by score; >0 = Gumbel-perturbed ranking,
# i.e. sampling without replacement with prob proportional to exp(score/T).
# A little stochasticity trades a small amount of target-likeness for topical
# diversity, which matters when the budget is 12M tokens of a 200M-token pool.
TEMP = float(os.environ.get("CURATE_T", "0"))
WEIGHTS = [float(x) for x in os.environ.get("CURATE_W", "0.40,0.20,0.20,0.20").split(",")]
EPOCHS        = 400
[metadata] {"noOutputExpected": false}

maxrank 24821
gateonly 28495
== MAXRANK (no register quotas)
PER_REGISTER [899.22, 275.32, 337.01, 1602.09]
HELDOUT_PPL 605.165  loss 6.4055  train_s 56  /tmp/e_max.npy
== GATE+DEDUP ONLY (random fill)
rem:
3648

[stdout]
maxrank 24821
gateonly 28495
== MAXRANK (no register quotas)
PER_REGISTER [899.22, 275.32, 337.01, 1602.09]
HELDOUT_PPL 605.165  loss 6.4055  train_s 56  /tmp/e_max.npy
== GATE+DEDUP ONLY (random fill)
rem:
3648
[metadata] {"noOutputExpected": false}

HELDOUT_PPL 605.165  loss 6.4055  train_s 56  /tmp/e_max.npy
== GATE+DEDUP ONLY (random fill)
PER_REGISTER [895.64, 288.57, 378.8, 327.76]
HELDOUT_PPL 423.399  loss 6.0483  train_s 56  /tmp/e_gate.npy

[stdout]
HELDOUT_PPL 605.165  loss 6.4055  train_s 56  /tmp/e_max.npy
== GATE+DEDUP ONLY (random fill)
PER_REGISTER [895.64, 288.57, 378.8, 327.76]
HELDOUT_PPL 423.399  loss 6.0483  train_s 56  /tmp/e_gate.npy
[metadata] {"noOutputExpected": false}

Command running in background with ID: bchohwj57. Output is being written to: /tmp/claude-0/-workspace/2123b707-8a58-4911-b84d-2ce5f3aed78d/tasks/bchohwj57.output. You will be notified when it completes. To check interim output, use Read on that file path.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "bchohwj57"}

register token shares: [12.0, 6.0, 6.0, 6.0]
selected 54327 docs ~30.0M tokens -> /tmp/sel_full.json
== FULLDOC FEATS
PER_REGISTER [789.21, 261.78, 305.59, 211.4]
HELDOUT_PPL 340.061  loss 5.8291  train_s 56  /tmp/e_full.npy
rem:
3167

[stdout]
register token shares: [12.0, 6.0, 6.0, 6.0]
selected 54327 docs ~30.0M tokens -> /tmp/sel_full.json
== FULLDOC FEATS
PER_REGISTER [789.21, 261.78, 305.59, 211.4]
HELDOUT_PPL 340.061  loss 5.8291  train_s 56  /tmp/e_full.npy
rem:
3167
[metadata] {"noOutputExpected": false}

register token shares: [12.0, 6.1, 6.0, 6.0]
selected 33852 docs ~30.0M tokens -> /tmp/repro.json
reproduces submission exactly: True 33852

[stdout]
register token shares: [12.0, 6.1, 6.0, 6.0]
selected 33852 docs ~30.0M tokens -> /tmp/repro.json
reproduces submission exactly: True 33852
[metadata] {"noOutputExpected": false}

Command running in background with ID: bmfk74kw1. Output is being written to: /tmp/claude-0/-workspace/2123b707-8a58-4911-b84d-2ce5f3aed78d/tasks/bmfk74kw1.output. You will be notified when it completes. To check interim output, use Read on that file path.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "bmfk74kw1"}

seed=202 v2  HELDOUT_PPL 327.541  loss 5.7916  train_s 56  /tmp/v2.npy
seed=202 e_0.40_0.20_0.20_0.20  HELDOUT_PPL 317.766  loss 5.7613  train_s 56  /tmp/e_0.40_0.20_0.20_0.20.npy
seed=707 v2  HELDOUT_PPL 326.178  loss 5.7874  train_s 56  /tmp/v2.npy
seed=707 e_0.40_0.20_0.20_0.20  HELDOUT_PPL 317.051  loss 5.7591  train_s 55  /tmp/e_0.40_0.20_0.20_0.20.npy
rem:
2776

[stdout]
seed=202 v2  HELDOUT_PPL 327.541  loss 5.7916  train_s 56  /tmp/v2.npy
seed=202 e_0.40_0.20_0.20_0.20  HELDOUT_PPL 317.766  loss 5.7613  train_s 56  /tmp/e_0.40_0.20_0.20_0.20.npy
seed=707 v2  HELDOUT_PPL 326.178  loss 5.7874  train_s 56  /tmp/v2.npy
seed=707 e_0.40_0.20_0.20_0.20  HELDOUT_PPL 317.051  loss 5.7591  train_s 55  /tmp/e_0.40_0.20_0.20_0.20.npy
rem:
2776
[metadata] {"noOutputExpected": false}

The file /workspace/claim.md has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"type": "update", "filePath": "/workspace/claim.md", "content": "# Claim: register-conditional target likeness, balanced over the *consumed* prefix\n\n## Hypothesis\n\nWhen the evaluation target is a **broad, equal-weight mixture of registers**\n(encyclopedic / high-quality web prose / news / technical Q&A) and the training\nbudget is a small slice of the pool (12M of ~200M tokens), the selection rule that\nminimises held-out perplexity is **not** a single global quality score. It is:\n\n> *k* separate target-likeness discriminators, one per register of the disclosed\n> target, applied under a shared non-prose + near-duplicate filter, with the\n> **token prefix the trainer actually consumes** balanced across registers.\n\nThree sub-claims, each of which the design bets on:\n\n1. **Balance over the consumed prefix is load-bearing, not cosmetic.** A single\n   global ranking concentrates the budget on whichever register is most linearly\n   separable from generic web text, starving the others.\n2. **The optimal balance is not uniform: it tilts toward the register the model is\n   worst at**, because the metric averages *cross-entropy* (not perplexity) over\n   registers, so absolute loss is dominated by the hardest register. But the\n   optimum is interior — over-tilting loses more elsewhere than it gains.\n3. **Fidelity beats diversity at this budget.** Hard top-k by score beats\n   temperature-sampled draws from the same gated pool.\n\n## Mechanism — and the observable it predicts (not the final perplexity)\n\nThe mechanism is **register-conditional loss reallocation**. The predicted\nobservable is the **per-register perplexity vector**: the same trained model\nevaluated separately on each quarter of `multi_dev.npy` (`eval_multi.py`, a copy\nof the frozen trainer with an extra per-quarter eval loop and identical training).\n\n| selection (12M tokens, frozen trainer, seed 1337) | wiki | hq-web | news | Q&A | overall |\n|---|---|---|---|---|---|\n| random pool — do-nothing baseline | — | — | — | — | **469.9** |\n| global max-score ranking, **no** register quotas | 899.2 | 275.3 | 337.0 | **1602.1** | **605.2** |\n| quality gate + near-dup only, random fill | 895.6 | 288.6 | 378.8 | 327.8 | **423.4** |\n| 4 discriminators, doc-balanced, no dedup (v1) | — | — | — | — | **340.7** |\n| Gumbel sampling T=0.5 (diversity) over final rule | 740.6 | 272.1 | 329.1 | 369.7 | **396.1** |\n| full-document features instead of 3k-char prefix | 789.2 | 261.8 | 305.6 | 211.4 | **340.1** |\n| final rule, uniform 25/25/25/25 tilt | 756.0 | 266.0 | 305.6 | 183.5 | **326.0** |\n| final rule, over-tilt 55/15/15/15 | 705.8 | 268.7 | 306.1 | 200.5 | **328.7** |\n| **final rule, tilt 40/20/20/20 (submitted)** | **747.6** | **266.7** | **306.7** | **178.7** | **323.5** |\n\nFour mechanism-level observables, all checkable without reading the overall score:\n\n- **Unbalanced ranking collapses onto one register — visibly, in the vector.**\n  Global `max_d score_d` ranking sends Q&A to 1602 while hq-web *improves* to 275.\n  The four logistic scores are not mutually calibrated, so \"best overall score\"\n  silently means \"most separable register\". This selection is **worse than random\n  selection** (605 vs 470) despite every document in it being high-scoring: a\n  filter that is good on average can be worse than no filter if it is unbalanced.\n- **The tilt trades registers monotonically.** 25→40→55% encyclopedic share moves\n  the wiki column monotonically down (756→748→706) and the Q&A column\n  non-monotonically (183→179→201, minimum in the middle). The overall optimum is\n  interior at 40%. That interior structure is a mechanism signature, not a fit.\n- **The encyclopedic register is an outlier for a reason that is not topical.**\n  756 vs 184 for Q&A. Predicted cause: the encyclopedic quarter is raw\n  wikitext-103 formatting (` @-@ `, ` @,@ `, spaces before punctuation). Directly\n  verified against the pool: **zero** of 182,016 documents contain `@-@`, and only\n  133 have majority spaced punctuation. A large part of that 756 is a\n  *tokenisation* gap no data selection can close — which is exactly why tilting\n  toward it yields ~1% and over-tilting backfires.\n- **The discriminators separate register, not length or boiler-plate.** Held-out\n  AUC against unseen random pool documents: 0.997 / 0.928 / 0.983 / 1.000.\n  Negatives are truncated to the same 3,000-character prefix as positives, so\n  document length cannot be the signal.\n\n## Falsification\n\nEach sub-claim was given a way to die. All four tests were run.\n\n1. **\"Balance is irrelevant.\"** Rank the same gated, deduplicated candidates by\n   `max_d score_d`, no quotas. *Result: 605.2 vs 323.5 — refuted decisively, and\n   in the predicted shape (one register collapses).*\n2. **\"The tilt is seed noise.\"** The seed-1337 gap was only 2.5 PPL (0.008 nats),\n   the weakest leg. Re-ran 25/25/25/25 vs 40/20/20/20 at two further training\n   seeds: 326.0/327.5/326.2 vs 323.5/317.8/317.1 (means 326.6 vs 319.5). *Sign\n   consistent across all three seeds; survived, and the true gap is larger than\n   the first measurement suggested.*\n3. **\"The classifiers add nothing over the surface gate.\"** Gate + near-dup\n   removal alone, random fill: 423.4. *So the gate is worth ~47 PPL and the\n   register discriminators + balancing are worth a further ~100. Survived.*\n4. **\"Diversity beats fidelity.\"** Gumbel-perturbed sampling at T=0.5 over the\n   identical gated pool: 396.1, ~73 PPL worse. *Refuted — at a 6%-of-pool budget,\n   target fidelity dominates coverage.*\n\nA fifth test the claim would not survive: if per-register perplexity on the hidden\ntarget set showed a *different* worst register than encyclopedic, the 40/20/20/20\ntilt would be mis-aimed and should revert to uniform. The tilt is the one\ncomponent fitted to the dev sample rather than derived; uniform (326.0) is the\nsafe fallback and costs little.\n\n## Transfer\n\n- **Transfers with disclosed-target structure, not with this pool.** Any setting\n  where the eval target is a *known mixture* — a domain-specific model, an SFT\n  data mix, a multilingual budget split — can reuse the recipe: segment the\n  target, fit one cheap hashed-n-gram discriminator per segment against random\n  pool negatives, gate for prose, dedup, then balance **the consumed prefix**\n  rather than the candidate pool. No labels, no reference LM, no proxy-model\n  training: curation of 182k documents runs in ~2 minutes (CPU hashing + a\n  one-parameter-per-feature GPU logistic regression).\n- **The generalisable correction is \"balance where the budget is spent.\"** The\n  usual failure is balancing the candidate set and then truncating it by a global\n  score — which reintroduces exactly the skew the balancing removed. Because the\n  trainer here consumes an ordered list until a token budget, the ordering *is*\n  the mixture; a 605-vs-323 swing turns on that alone.\n- **Secondary transfer: weight toward the highest-cross-entropy slice, but expect\n  an interior optimum.** Applies to any metric that averages loss over\n  heterogeneous slices. The useful tilt is bounded by how much of that slice's\n  loss is *reachable* from the pool.\n- **Known limit.** Gains are bounded by pool support. If a target register has no\n  analogue in the pool — as wikitext-formatted encyclopedic prose does not here —\n  selection cannot manufacture it; the fix is a different pool or a normalisation\n  step, not a better filter. Reporting the per-register vector is what makes that\n  ceiling visible; an aggregate perplexity hides it.\n\n## Reproduction\n\n```\npython /workspace/submission/curate.py        # -> submission/selection.json (deterministic)\nbash   /workspace/dev_eval.sh                 # -> dev PPL with the frozen trainer\n```\n`curate.py` re-run from scratch reproduces the submitted 33,852-id list exactly.\n", "structuredPatch": [{"oldStart": 1, "oldLines": 4, "newStart": 1, "newLines": 4, "lines": ["-# Claim: register-conditional target likeness beats undifferentiated \"quality\"", "+# Claim: register-conditional target likeness, balanced over the *consumed* prefix", " ", " ## Hypothesis", " "]}, {"oldStart": 8, "oldLines": 89, "newStart": 8, "newLines": 118, "lines": [" minimises held-out perplexity is **not** a single global quality score. It is:", " ", " > *k* separate target-likeness discriminators, one per register of the disclosed", "-> target, applied under a shared non-prose/near-duplicate filter, with the", "-> **consumed token prefix** balanced across registers.", "+> target, applied under a shared non-prose + near-duplicate filter, with the", "+> **token prefix the trainer actually consumes** balanced across registers.", " ", "-Two claims follow, and both are what the design is betting on:", "+Three sub-claims, each of which the design bets on:", " ", "-1. **Register balance in the consumed prefix matters more than raw score.** A", "-   single global ranking concentrates the budget on whichever register is most", "-   linearly separable from generic web text, starving the others.", "-2. **The right balance is not uniform: it tilts toward the register the model is", "+1. **Balance over the consumed prefix is load-bearing, not cosmetic.** A single", "+   global ranking concentrates the budget on whichever register is most linearly", "+   separable from generic web text, starving the others.", "+2. **The optimal balance is not uniform: it tilts toward the register the model is", "    worst at**, because the metric averages *cross-entropy* (not perplexity) over", "-   registers, so absolute loss is dominated by the hardest register.", "+   registers, so absolute loss is dominated by the hardest register. But the", "+   optimum is interior — over-tilting loses more elsewhere than it gains.", "+3. **Fidelity beats diversity at this budget.** Hard top-k by score beats", "+   temperature-sampled draws from the same gated pool.", " ", "-## Mechanism — and an observable that is not the final perplexity", "+## Mechanism — and the observable it predicts (not the final perplexity)", " ", "-The mechanism is register-conditional loss reallocation. The predicted", "-**observable is the per-register perplexity vector**, measured by evaluating the", "-same trained model separately on each quarter of `multi_dev.npy`. Predictions", "-made *before* the final run and confirmed:", "+The mechanism is **register-conditional loss reallocation**. The predicted", "+observable is the **per-register perplexity vector**: the same trained model", "+evaluated separately on each quarter of `multi_dev.npy` (`eval_multi.py`, a copy", "+of the frozen trainer with an extra per-quarter eval loop and identical training).", " ", "-| selection | wiki | hq-web | news | Q&A | overall |", "+| selection (12M tokens, frozen trainer, seed 1337) | wiki | hq-web | news | Q&A | overall |", " |---|---|---|---|---|---|", "-| random pool (baseline) | — | — | — | — | **469.9** |", "-| classifier + gate, doc-balanced (v1) | — | — | — | — | **340.7** |", "-| + near-dup removal, token-balanced 25/25/25/25 | 756.0 | 266.0 | 305.6 | 183.5 | **326.0** |", "-| + tilt to encyclopedic 40/20/20/20 | 747.6 | 266.7 | 306.7 | 178.7 | **323.5** |", "-| over-tilt 55/15/15/15 | 705.8 | 268.7 | 306.1 | 200.5 | **328.7** |", "+| random pool — do-nothing baseline | — | — | — | — | **469.9** |", "+| global max-score ranking, **no** register quotas | 899.2 | 275.3 | 337.0 | **1602.1** | **605.2** |", "+| quality gate + near-dup only, random fill | 895.6 | 288.6 | 378.8 | 327.8 | **423.4** |", "+| 4 discriminators, doc-balanced, no dedup (v1) | — | — | — | — | **340.7** |", "+| Gumbel sampling T=0.5 (diversity) over final rule | 740.6 | 272.1 | 329.1 | 369.7 | **396.1** |", "+| full-document features instead of 3k-char prefix | 789.2 | 261.8 | 305.6 | 211.4 | **340.1** |", "+| final rule, uniform 25/25/25/25 tilt | 756.0 | 266.0 | 305.6 | 183.5 | **326.0** |", "+| final rule, over-tilt 55/15/15/15 | 705.8 | 268.7 | 306.1 | 200.5 | **328.7** |", "+| **final rule, tilt 40/20/20/20 (submitted)** | **747.6** | **266.7** | **306.7** | **178.7** | **323.5** |", " ", "-Three mechanism-level observables, all checkable without looking at the score:", "+Four mechanism-level observables, all checkable without reading the overall score:", " ", "-- **The register spread is huge and encyclopedic is the outlier** (756 vs 184).", "-  Predicted cause: the encyclopedic quarter is raw wikitext-103 formatting", "-  (` @-@ `, ` @,@ `, spaces before punctuation). Directly verified: **zero**", "-  documents in the 182,016-document pool contain `@-@`, and only 133 have", "-  majority spaced punctuation. So a large part of that 756 is a *tokenisation*", "-  gap that no data selection can close — which is exactly why tilting budget", "-  toward it yields only ~1% and over-tilting backfires.", "-- **Tilting trades registers monotonically.** 25→40→55% encyclopedic share", "-  monotonically lowers the wiki column (756→748→706) and monotonically raises", "-  the Q&A column (183→179→201 — the Q&A minimum sits between). The overall", "-  optimum is interior, at 40%. That is a mechanism signature, not a fit artefact.", "-- **The discriminators are genuinely separating register, not length or", "-  boilerplate.** Held-out AUC against unseen random pool documents:", "-  0.997 / 0.928 / 0.983 / 1.000 for the four registers. Negatives are truncated to", "-  the same 3,000-character prefix as positives so length cannot be the signal.", "+- **Unbalanced ranking collapses onto one register — visibly, in the vector.**", "+  Global `max_d score_d` ranking sends Q&A to 1602 while hq-web *improves* to 275.", "+  The four logistic scores are not mutually calibrated, so \"best overall score\"", "+  silently means \"most separable register\". This selection is **worse than random", "+  selection** (605 vs 470) despite every document in it being high-scoring: a", "+  filter that is good on average can be worse than no filter if it is unbalanced.", "+- **The tilt trades registers monotonically.** 25→40→55% encyclopedic share moves", "+  the wiki column monotonically down (756→748→706) and the Q&A column", "+  non-monotonically (183→179→201, minimum in the middle). The overall optimum is", "+  interior at 40%. That interior structure is a mechanism signature, not a fit.", "+- **The encyclopedic register is an outlier for a reason that is not topical.**", "+  756 vs 184 for Q&A. Predicted cause: the encyclopedic quarter is raw", "+  wikitext-103 formatting (` @-@ `, ` @,@ `, spaces before punctuation). Directly", "+  verified against the pool: **zero** of 182,016 documents contain `@-@`, and only", "+  133 have majority spaced punctuation. A large part of that 756 is a", "+  *tokenisation* gap no data selection can close — which is exactly why tilting", "+  toward it yields ~1% and over-tilting backfires.", "+- **The discriminators separate register, not length or boiler-plate.** Held-out", "+  AUC against unseen random pool documents: 0.997 / 0.928 / 0.983 / 1.000.", "+  Negatives are truncated to the same 3,000-character prefix as positives, so", "+  document length cannot be the signal.", " ", " ## Falsification", " ", "-The claim is wrong if any of these hold:", "+Each sub-claim was given a way to die. All four tests were run.", " ", "-1. **Balance is irrelevant.** Take the same gated, deduplicated candidate set and", "-   rank by `max_d score_d` (one global ranking, no quotas). If that matches or", "-   beats the register-balanced prefix, claim 1 is false. Prediction: it is worse,", "-   and its per-register vector is visibly skewed — the most separable register's", "-   perplexity drops while at least one other rises.", "-2. **The tilt is noise.** If re-running 40/20/20/20 vs 25/25/25/25 under different", "-   training seeds does not preserve the ordering, claim 2 is false. The measured", "-   gap is 2.5 PPL (0.008 nats) — small, and this is the weakest leg of the claim.", "-   The monotone per-register trend across three weightings is the real evidence;", "-   the overall-PPL ordering alone would not be.", "-3. **The classifier adds nothing over the surface gate.** Run the quality gate +", "-   near-dup removal alone, filling the budget by random draw from survivors. If", "-   that reaches ~325, the discriminators are doing no work and the whole gain is", "-   a boiler-plate filter.", "-4. **Diversity beats fidelity.** If Gumbel-perturbed sampling at high temperature", "-   (broad, low-fidelity draw from the gated pool) beats hard top-k, then", "-   target-likeness is not the operative variable — coverage is.", "+1. **\"Balance is irrelevant.\"** Rank the same gated, deduplicated candidates by", "+   `max_d score_d`, no quotas. *Result: 605.2 vs 323.5 — refuted decisively, and", "+   in the predicted shape (one register collapses).*", "+2. **\"The tilt is seed noise.\"** The seed-1337 gap was only 2.5 PPL (0.008 nats),", "+   the weakest leg. Re-ran 25/25/25/25 vs 40/20/20/20 at two further training", "+   seeds: 326.0/327.5/326.2 vs 323.5/317.8/317.1 (means 326.6 vs 319.5). *Sign", "+   consistent across all three seeds; survived, and the true gap is larger than", "+   the first measurement suggested.*", "+3. **\"The classifiers add nothing over the surface gate.\"** Gate + near-dup", "+   removal alone, random fill: 423.4. *So the gate is worth ~47 PPL and the", "+   register discriminators + balancing are worth a further ~100. Survived.*", "+4. **\"Diversity beats fidelity.\"** Gumbel-perturbed sampling at T=0.5 over the", "+   identical gated pool: 396.1, ~73 PPL worse. *Refuted — at a 6%-of-pool budget,", "+   target fidelity dominates coverage.*", " ", "+A fifth test the claim would not survive: if per-register perplexity on the hidden", "+target set showed a *different* worst register than encyclopedic, the 40/20/20/20", "+tilt would be mis-aimed and should revert to uniform. The tilt is the one", "+component fitted to the dev sample rather than derived; uniform (326.0) is the", "+safe fallback and costs little.", "+", " ## Transfer", " ", "-- **Scales with disclosed-target structure, not with this pool.** Any setting", "-  where the eval target is a *known mixture* — a domain-specific model, an", "-  instruction-tuning mix, a multilingual budget split — can use the same recipe:", "-  segment the target, fit one cheap hashed-n-gram discriminator per segment, and", "-  balance the *consumed* prefix rather than the candidate pool. No labels, no", "-  reference LM, no GPU-hours: the whole curation runs in ~2 minutes on CPU+1 GPU", "-  for 182k documents.", "-- **The correction generalises.** \"Weight the mixture toward the register with", "-  the highest cross-entropy, but only until the interior optimum\" applies", "-  whenever the metric averages loss over heterogeneous slices. The size of the", "-  useful tilt is bounded by how much of the slice's loss is *reachable* — here,", "-  a formatting gap with no pool support capped it at ~1%.", "-- **Known limit.** The gain is bounded by pool support. If a target register has", "-  no analogue in the pool (as encyclopedic wikitext formatting does not here),", "-  selection cannot manufacture it; the answer is a different pool or a", "-  tokenisation-level fix, not a better filter. Reporting the per-register vector", "-  is what makes that limit visible — an aggregate perplexity would hide it.", "+- **Transfers with disclosed-target structure, not with this pool.** Any setting", "+  where the eval target is a *known mixture* — a domain-specific model, an SFT", "+  data mix, a multilingual budget split — can reuse the recipe: segment the", "+  target, fit one cheap hashed-n-gram discriminator per segment against random", "+  pool negatives, gate for prose, dedup, then balance **the consumed prefix**", "+  rather than the candidate pool. No labels, no reference LM, no proxy-model", "+  training: curation of 182k documents runs in ~2 minutes (CPU hashing + a", "+  one-parameter-per-feature GPU logistic regression).", "+- **The generalisable correction is \"balance where the budget is spent.\"** The", "+  usual failure is balancing the candidate set and then truncating it by a global", "+  score — which reintroduces exactly the skew the balancing removed. Because the", "+  trainer here consumes an ordered list until a token budget, the ordering *is*", "+  the mixture; a 605-vs-323 swing turns on that alone.", "+- **Secondary transfer: weight toward the highest-cross-entropy slice, but expect", "+  an interior optimum.** Applies to any metric that averages loss over", "+  heterogeneous slices. The useful tilt is bounded by how much of that slice's", "+  loss is *reachable* from the pool.", "+- **Known limit.** Gains are bounded by pool support. If a target register has no", "+  analogue in the pool — as wikitext-formatted encyclopedic prose does not here —", "+  selection cannot manufacture it; the fix is a different pool or a normalisation", "+  step, not a better filter. Reporting the per-register vector is what makes that", "+  ceiling visible; an aggregate perplexity hides it.", "+", "+## Reproduction", "+", "+```", "+python /workspace/submission/curate.py        # -> submission/selection.json (deterministic)", "+bash   /workspace/dev_eval.sh                 # -> dev PPL with the frozen trainer", "+```", "+`curate.py` re-run from scratch reproduces the submitted 33,852-id list exactly."]}], "originalFile": "# Claim: register-conditional target likeness beats undifferentiated \"quality\"\n\n## Hypothesis\n\nWhen the evaluation target is a **broad, equal-weight mixture of registers**\n(encyclopedic / high-quality web prose / news / technical Q&A) and the training\nbudget is a small slice of the pool (12M of ~200M tokens), the selection rule that\nminimises held-out perplexity is **not** a single global quality score. It is:\n\n> *k* separate target-likeness discriminators, one per register of the disclosed\n> target, applied under a shared non-prose/near-duplicate filter, with the\n> **consumed token prefix** balanced across registers.\n\nTwo claims follow, and both are what the design is betting on:\n\n1. **Register balance in the consumed prefix matters more than raw score.** A\n   single global ranking concentrates the budget on whichever register is most\n   linearly separable from generic web text, starving the others.\n2. **The right balance is not uniform: it tilts toward the register the model is\n   worst at**, because the metric averages *cross-entropy* (not perplexity) over\n   registers, so absolute loss is dominated by the hardest register.\n\n## Mechanism — and an observable that is not the final perplexity\n\nThe mechanism is register-conditional loss reallocation. The predicted\n**observable is the per-register perplexity vector**, measured by evaluating the\nsame trained model separately on each quarter of `multi_dev.npy`. Predictions\nmade *before* the final run and confirmed:\n\n| selection | wiki | hq-web | news | Q&A | overall |\n|---|---|---|---|---|---|\n| random pool (baseline) | — | — | — | — | **469.9** |\n| classifier + gate, doc-balanced (v1) | — | — | — | — | **340.7** |\n| + near-dup removal, token-balanced 25/25/25/25 | 756.0 | 266.0 | 305.6 | 183.5 | **326.0** |\n| + tilt to encyclopedic 40/20/20/20 | 747.6 | 266.7 | 306.7 | 178.7 | **323.5** |\n| over-tilt 55/15/15/15 | 705.8 | 268.7 | 306.1 | 200.5 | **328.7** |\n\nThree mechanism-level observables, all checkable without looking at the score:\n\n- **The register spread is huge and encyclopedic is the outlier** (756 vs 184).\n  Predicted cause: the encyclopedic quarter is raw wikitext-103 formatting\n  (` @-@ `, ` @,@ `, spaces before punctuation). Directly verified: **zero**\n  documents in the 182,016-document pool contain `@-@`, and only 133 have\n  majority spaced punctuation. So a large part of that 756 is a *tokenisation*\n  gap that no data selection can close — which is exactly why tilting budget\n  toward it yields only ~1% and over-tilting backfires.\n- **Tilting trades registers monotonically.** 25→40→55% encyclopedic share\n  monotonically lowers the wiki column (756→748→706) and monotonically raises\n  the Q&A column (183→179→201 — the Q&A minimum sits between). The overall\n  optimum is interior, at 40%. That is a mechanism signature, not a fit artefact.\n- **The discriminators are genuinely separating register, not length or\n  boilerplate.** Held-out AUC against unseen random pool documents:\n  0.997 / 0.928 / 0.983 / 1.000 for the four registers. Negatives are truncated to\n  the same 3,000-character prefix as positives so length cannot be the signal.\n\n## Falsification\n\nThe claim is wrong if any of these hold:\n\n1. **Balance is irrelevant.** Take the same gated, deduplicated candidate set and\n   rank by `max_d score_d` (one global ranking, no quotas). If that matches or\n   beats the register-balanced prefix, claim 1 is false. Prediction: it is worse,\n   and its per-register vector is visibly skewed — the most separable register's\n   perplexity drops while at least one other rises.\n2. **The tilt is noise.** If re-running 40/20/20/20 vs 25/25/25/25 under different\n   training seeds does not preserve the ordering, claim 2 is false. The measured\n   gap is 2.5 PPL (0.008 nats) — small, and this is the weakest leg of the claim.\n   The monotone per-register trend across three weightings is the real evidence;\n   the overall-PPL ordering alone would not be.\n3. **The classifier adds nothing over the surface gate.** Run the quality gate +\n   near-dup removal alone, filling the budget by random draw from survivors. If\n   that reaches ~325, the discriminators are doing no work and the whole gain is\n   a boiler-plate filter.\n4. **Diversity beats fidelity.** If Gumbel-perturbed sampling at high temperature\n   (broad, low-fidelity draw from the gated pool) beats hard top-k, then\n   target-likeness is not the operative variable — coverage is.\n\n## Transfer\n\n- **Scales with disclosed-target structure, not with this pool.** Any setting\n  where the eval target is a *known mixture* — a domain-specific model, an\n  instruction-tuning mix, a multilingual budget split — can use the same recipe:\n  segment the target, fit one cheap hashed-n-gram discriminator per segment, and\n  balance the *consumed* prefix rather than the candidate pool. No labels, no\n  reference LM, no GPU-hours: the whole curation runs in ~2 minutes on CPU+1 GPU\n  for 182k documents.\n- **The correction generalises.** \"Weight the mixture toward the register with\n  the highest cross-entropy, but only until the interior optimum\" applies\n  whenever the metric averages loss over heterogeneous slices. The size of the\n  useful tilt is bounded by how much of the slice's loss is *reachable* — here,\n  a formatting gap with no pool support capped it at ~1%.\n- **Known limit.** The gain is bounded by pool support. If a target register has\n  no analogue in the pool (as encyclopedic wikitext formatting does not here),\n  selection cannot manufacture it; the answer is a different pool or a\n  tokenisation-level fix, not a better filter. Reporting the per-register vector\n  is what makes that limit visible — an aggregate perplexity would hide it.\n", "userModified": false}

valid: 33852 ids, range 1 181981
packed prefix tokens: 12000000
-rw-r--r-- 1 root root 7769 Aug  4 01:40 claim.md

submission/:
total 244
drwxr-xr-x 1 root root    100 Aug  4 01:02 .
drwxr-xr-x 1 root root    220 Aug  4 01:40 ..
drwxr-xr-x 1 root root     80 Aug  4 01:02 __pycache__
-rw-r--r-- 1 root root  12783 Aug  4 01:28 curate.py
-rw-r--r-- 1 root root 236263 Aug  4 01:18 selection.json

[stdout]
valid: 33852 ids, range 1 181981
packed prefix tokens: 12000000
-rw-r--r-- 1 root root 7769 Aug  4 01:40 claim.md

submission/:
total 244
drwxr-xr-x 1 root root    100 Aug  4 01:02 .
drwxr-xr-x 1 root root    220 Aug  4 01:40 ..
drwxr-xr-x 1 root root     80 Aug  4 01:02 __pycache__
-rw-r--r-- 1 root root  12783 Aug  4 01:28 curate.py
-rw-r--r-- 1 root root 236263 Aug  4 01:18 selection.json
[metadata] {"noOutputExpected": false}

done

[stdout]
done
[metadata] {"noOutputExpected": false}