claude-opus-5

HELDOUT_PPL 328.378  loss 5.7942  train_s 55  /tmp/t_long.npy
HELDOUT_PPL 317.607  loss 5.7608  train_s 55  /tmp/t_dd.npy
4208

[stdout]
HELDOUT_PPL 328.378  loss 5.7942  train_s 55  /tmp/t_long.npy
HELDOUT_PPL 317.607  loss 5.7608  train_s 55  /tmp/t_dd.npy
4208
[metadata] {"noOutputExpected": false}

[1]+  Done                    nohup python work/round2.py > work/r2.log 2>&1
done wiki
done web
done qa

[stdout]
[1]+  Done                    nohup python work/round2.py > work/r2.log 2>&1
done wiki
done web
done qa
[metadata] {"noOutputExpected": false}

dedup dropped 31017
work/sel_r2.json 28687 {'wiki': 8.2, 'web': 7.9, 'qa': 7.9}
HELDOUT_PPL 317.059  loss 5.7591  train_s 56  /tmp/t_r2.npy
4008

[stdout]
dedup dropped 31017
work/sel_r2.json 28687 {'wiki': 8.2, 'web': 7.9, 'qa': 7.9}
HELDOUT_PPL 317.059  loss 5.7591  train_s 56  /tmp/t_r2.npy
4008
[metadata] {"noOutputExpected": false}

+ python work/policy4.py '{'\''wiki'\'':0.34,'\''web'\'':0.33,'\''qa'\'':0.33}' work/sel_dd3.json 0 3
dedup dropped 56315
work/sel_dd3.json 28111 {'wiki': 8.2, 'web': 7.9, 'qa': 7.9}
+ bash work/run_eval.sh work/sel_dd3.json dd3
HELDOUT_PPL 323.244  loss 5.7784  train_s 56  /tmp/t_dd3.npy
+ python work/policy2.py '{'\''wiki'\'':0.40,'\''web'\'':0.35,'\''qa'\'':0.25}' work/sel_q2.json 0 1
dedup dropped 31017
work/sel_q2.json 27691 {'wiki': 9.6, 'web': 8.4, 'qa': 6.0}
+ bash work/run_eval.sh work/sel_q2.json q2

[stdout]
+ python work/policy4.py '{'\''wiki'\'':0.34,'\''web'\'':0.33,'\''qa'\'':0.33}' work/sel_dd3.json 0 3
dedup dropped 56315
work/sel_dd3.json 28111 {'wiki': 8.2, 'web': 7.9, 'qa': 7.9}
+ bash work/run_eval.sh work/sel_dd3.json dd3
HELDOUT_PPL 323.244  loss 5.7784  train_s 56  /tmp/t_dd3.npy
+ python work/policy2.py '{'\''wiki'\'':0.40,'\''web'\'':0.35,'\''qa'\'':0.25}' work/sel_q2.json 0 1
dedup dropped 31017
work/sel_q2.json 27691 {'wiki': 9.6, 'web': 8.4, 'qa': 6.0}
+ bash work/run_eval.sh work/sel_q2.json q2
[metadata] {"noOutputExpected": false}

HELDOUT_PPL 323.244  loss 5.7784  train_s 56  /tmp/t_dd3.npy
HELDOUT_PPL 308.182  loss 5.7307  train_s 56  /tmp/t_q2.npy
3808

[stdout]
HELDOUT_PPL 323.244  loss 5.7784  train_s 56  /tmp/t_dd3.npy
HELDOUT_PPL 308.182  loss 5.7307  train_s 56  /tmp/t_q2.npy
3808
[metadata] {"noOutputExpected": false}

HELDOUT_PPL 332.091  loss 5.8054  train_s 55  /tmp/t_q3.npy

[stdout]
HELDOUT_PPL 332.091  loss 5.8054  train_s 55  /tmp/t_q3.npy
[metadata] {"noOutputExpected": false}

HELDOUT_PPL 332.091  loss 5.8054  train_s 55  /tmp/t_q3.npy
HELDOUT_PPL 320.058  loss 5.7685  train_s 55  /tmp/t_q4.npy
3567

[stdout]
HELDOUT_PPL 332.091  loss 5.8054  train_s 55  /tmp/t_q3.npy
HELDOUT_PPL 320.058  loss 5.7685  train_s 55  /tmp/t_q4.npy
3567
[metadata] {"noOutputExpected": false}

HELDOUT_PPL 317.359  loss 5.7600  train_s 55  /tmp/t_q2s2.npy

[stdout]
HELDOUT_PPL 317.359  loss 5.7600  train_s 55  /tmp/t_q2s2.npy
[metadata] {"noOutputExpected": false}

HELDOUT_PPL 317.359  loss 5.7600  train_s 55  /tmp/t_q2s2.npy
HELDOUT_PPL 311.348  loss 5.7409  train_s 55  /tmp/t_dds2.npy
HELDOUT_PPL 316.052  loss 5.7559  train_s 55  /tmp/t_q5.npy

[stdout]
HELDOUT_PPL 317.359  loss 5.7600  train_s 55  /tmp/t_q2s2.npy
HELDOUT_PPL 311.348  loss 5.7409  train_s 55  /tmp/t_dds2.npy
HELDOUT_PPL 316.052  loss 5.7559  train_s 55  /tmp/t_q5.npy
[metadata] {"noOutputExpected": false}

started
3287

[stdout]
started
3287
[metadata] {"noOutputExpected": false}

File created successfully at: /workspace/claim.md (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/workspace/claim.md", "content": "# Claim: register-balanced target-similarity selection beats undirected quality filtering\n\n## Hypothesis\n\nFor a fixed 12M-token budget (~6% of a 200M-token raw web pool), held-out perplexity on a\n**broad, multi-register** high-quality target is minimised not by \"generic quality\" filtering\nbut by **selecting for similarity to each register of the target separately, under a token\nquota that mirrors the target's mixture**.\n\nConcretely, I claim two separable effects:\n\n1. **Direction** — documents ranked target-like by a cheap linear classifier (dev target as\n   positives, random pool as negatives, GPT-2 unigram+hashed-bigram features) train a much\n   better model than a random draw, and the effect is monotone in selectivity: taking the\n   strict top of the ranking beats randomly sampling a 3×-wider top band.\n2. **Coverage** — because the target is an equal-parts mixture (encyclopedic / general web\n   prose + news / technical Q&A), a *single* undifferentiated ranking under-serves whichever\n   registers the classifier finds least pool-like. Splitting the classifier per register and\n   filling a per-register token quota keeps all registers represented in every prefix of the\n   selection, which matters because the trainer truncates at 12M tokens.\n\nAdditionally, **near-duplicate removal** should help at this budget: duplicated pages spend\nbudget on text the model has already fit.\n\n## Mechanism (observable other than final perplexity)\n\nThe mechanism is that the classifier score is a proxy for *token-distribution overlap with the\ntarget*, and that overlap, not surface \"cleanliness\", is what drives loss. Predictions that are\nobservable without ever training a model:\n\n- **M1.** The unigram cross-entropy of the *target* under the selected corpus's empirical token\n  distribution should fall monotonically as selection tightens: random pool > heuristic-filtered\n  pool > wide-band classifier selection > strict-top classifier selection. This is measurable\n  directly from token counts, with no training run.\n- **M2.** Register assignment should be *sharply* trimodal on the pool, not diffuse: the\n  three per-register classifiers should disagree strongly (a large gap between the top and\n  second z-score for most documents), because the registers differ in surface form (HTML\n  entity/`<code>` markup for Q&A, encyclopedic sentence structure for wiki).\n- **M3.** Near-duplication is a substantial fraction of the *heuristically clean* pool — a\n  single-band 5-gram MinHash should drop a double-digit percentage of documents. (Observed:\n  31,017 of 136,895 kept docs, ~23%.)\n- **M4.** If the mechanism is coverage rather than raw purity, then the *marginal* value of\n  extra tokens of the register that is already over-represented in the selection should be\n  negative — i.e. quota curves should be unimodal, with a plateau near the target's mixture and\n  degradation as the quota is skewed toward any single register.\n\n## Falsification\n\nThe claim is falsified if, at the same 12M budget and identical frozen recipe:\n\n- **F1.** A random selection matched only on the heuristic prose filter (no classifier) reaches\n  perplexity within noise of the classifier-selected corpus → *direction* effect is not real.\n- **F2.** Randomly sampling a much wider top band (3× the needed tokens) does **not** hurt →\n  selectivity/purity does not matter, and any \"target-like\" set of clean prose would do.\n- **F3.** A single undifferentiated ranking (no register split, no quota) matches the\n  register-balanced selection → the *coverage* claim is unnecessary.\n- **F4.** Skewing the quota heavily toward one register (e.g. 50% encyclopedic) is neutral or\n  better → the mixture-matching claim is wrong.\n\nStatus against the dev target (`multi_dev.npy`, frozen recipe, dev perplexity; seed 1337 unless\nnoted). Run-to-run seed noise is ~±5–9 ppl, measured by re-running two configurations under a\nsecond seed, so differences below ~10 ppl are not resolved:\n\n| selection | dev PPL |\n|---|---|\n| random pool (do-nothing baseline) | 469.9 |\n| classifier + quota 30/45/25, no dedup | 325.8 |\n| classifier + equal quota, no dedup | 322.3 |\n| **F2 test**: 3×-wide top band, random within band | 385.9 |\n| classifier + equal quota + dedup | 317.6 (311.3 @seed 2024) |\n| **final**: quota 40/35/25 + dedup | **308.2** (317.4 @seed 2024) |\n| **F4 test**: quota 50/30/20 + dedup | 332.1 |\n| length floor ≥512 tokens (equal quota) | 328.4 |\n| 3-band (more aggressive) dedup | 323.2 |\n\nF2 and F4 are clearly **not** triggered: widening the band costs ~68 ppl and skewing the quota\nto 50% encyclopedic costs ~24 ppl, both far outside seed noise. The direction effect (470 → ~310,\na 34% reduction) is an order of magnitude larger than noise. F1 and F3 were not run for lack of\nwall-clock, so the *decomposition* between \"clean prose\" and \"target-like\" is the weakest part of\nthis claim — the heuristic filter alone might account for part of the gain. What is *not*\nresolved by these runs is the fine quota shape: 40/35/25, 34/33/33 and 37/34/29 all lie within\nnoise of each other (two-seed means 312.8 vs 314.5), so the final quota choice is a weak\npreference, and only the coarse claim \"roughly mirror the mixture, do not skew hard\" is supported.\n\n## Transfer\n\n- **Budget.** The mechanism is budget-relative: the tighter the budget as a fraction of the pool,\n  the more purity should dominate diversity. At a 10× larger budget the wide-band variant should\n  close most of its gap, and dedup should matter more (repeats become unavoidable).\n- **Target.** Nothing here is specific to *these* four registers. Given any disclosed target\n  sample, the same recipe applies: decode it, weakly segment it into registers by surface cues,\n  fit one cheap linear classifier per register against a random pool sample, fill quotas. Only\n  the segmentation heuristic (`register_of_dev`) is target-specific, and it is three lines.\n- **Model scale.** The selection is model-free — it never runs the trained model — so it should\n  transfer across model sizes, though the optimal selectivity should loosen with capacity: a\n  larger model benefits from more diverse data before saturating.\n- **Known leak-ish caveat.** The dev target is *disclosed* by the task, and the classifier uses\n  it as its positive class. The official target is a disjoint sample of the same domains, so this\n  is domain transfer, not memorisation — but the selection would not transfer to an *undisclosed*\n  target, where one would substitute a proxy corpus for the positive class.\n", "structuredPatch": [], "originalFile": null, "userModified": false}

Token indices sequence length is longer than the specified maximum sequence length for this model (1257 > 1024). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (4664 > 1024). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (1097 > 1024). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (1217 > 1024). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (1316 > 1024). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (3524 > 1024). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (1216 > 1024). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (1121 > 1024). Running this sequence through the model will result in indexing errors
3197

[stdout]
Token indices sequence length is longer than the specified maximum sequence length for this model (1257 > 1024). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (4664 > 1024). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (1097 > 1024). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (1217 > 1024). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (1316 > 1024). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (3524 > 1024). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (1216 > 1024). Running this sequence through the model will result in indexing errors
Token indices sequence length is longer than the specified maximum sequence length for this model (1121 > 1024). Running this sequence through the model will result in indexing errors
3197
[metadata] {"noOutputExpected": false}

heuristic filter keeps 136895/182016 docs (117.2M tokens)
fitting wiki: 920 positives
fitting web: 1275 positives
fitting qa: 359 positives
near-dup filter dropped 31017 docs, 84.8M tokens left
wiki: 31126 docs assigned, 23.9M tokens
web: 31627 docs assigned, 25.6M tokens
qa: 43125 docs assigned, 35.3M tokens
selected 27643 docs, 24.0M tokens; wiki=9.6M, web=8.4M, qa=6.0M
wrote /workspace/submission/selection.json

[stdout]
heuristic filter keeps 136895/182016 docs (117.2M tokens)
fitting wiki: 920 positives
fitting web: 1275 positives
fitting qa: 359 positives
near-dup filter dropped 31017 docs, 84.8M tokens left
wiki: 31126 docs assigned, 23.9M tokens
web: 31627 docs assigned, 25.6M tokens
qa: 43125 docs assigned, 35.3M tokens
selected 27643 docs, 24.0M tokens; wiki=9.6M, web=8.4M, qa=6.0M
wrote /workspace/submission/selection.json
[metadata] {"noOutputExpected": false}

n 27643 unique 27643 max 181996 min 4 identical_to_tested False
packed 12000000 tokens from 27643 ids -> /tmp/dev_train.npy
HELDOUT_PPL 318.750  loss 5.7644  train_s 55  /tmp/dev_train.npy
(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)
2896

[stdout]
n 27643 unique 27643 max 181996 min 4 identical_to_tested False
packed 12000000 tokens from 27643 ids -> /tmp/dev_train.npy
HELDOUT_PPL 318.750  loss 5.7644  train_s 55  /tmp/dev_train.npy
(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)
2896
[metadata] {"noOutputExpected": false}

187:def main(quota=None, seed=SEED, out=OUT):
229:    # near-duplicate removal (single-band MinHash over 5-grams of the first 512 tokens):
243:    print(f"near-dup filter dropped {dropped} docs, {lens[kept].sum()/1e6:.1f}M tokens left",
254:    want = {r: quota[r] * EMIT_TOKENS for r in regs}
263:            while block < max(1, int(quota[r] * 40)) and ptr[r] < len(order[r]) and got[r] < want[r]:

[stdout]
187:def main(quota=None, seed=SEED, out=OUT):
229:    # near-duplicate removal (single-band MinHash over 5-grams of the first 512 tokens):
243:    print(f"near-dup filter dropped {dropped} docs, {lens[kept].sum()/1e6:.1f}M tokens left",
254:    want = {r: quota[r] * EMIT_TOKENS for r in regs}
263:            while block < max(1, int(quota[r] * 40)) and ptr[r] < len(order[r]) and got[r] < want[r]:
[metadata] {"noOutputExpected": false}

S[k] = (S[k] - v.mean()) / (v.std() + 1e-6)
    assign = S[:, :].argmax(0)
    best = S.max(0)

    # near-duplicate removal (single-band MinHash over 5-grams of the first 512 tokens):
    # duplicated pages otherwise burn budget on text the model has already seen.
    seen, dropped = set(), 0
    for j in kept:
        a = np.asarray(flat[starts[j]:starts[j] + min(lens[j], 512)], dtype=np.int64)
        if len(a) < 6:
            continue
        g = (a[:-4]*1000003 + a[1:-3]*7919 + a[2:-2]*104729 + a[3:-1]*31 + a[4:]) % (1 << 61)
        h = int(g.min())
        if h in seen:
            keep[j] = False; dropped += 1
        else:
            seen.add(h)
    kept = np.flatnonzero(keep)
    print(f"near-dup filter dropped {dropped} docs, {lens[kept].sum()/1e6:.1f}M tokens left",
          flush=True)

    order = {}
    for k, r in enumerate(regs):
        idx = kept[assign[kept] == k]
        idx = idx[np.argsort(-best[idx])]
        order[r] = list(idx)
        print(f"{r}: {len(idx)} docs assigned, {lens[idx].sum()/1e6:.1f}M tokens", flush=True)

    # take top-of-list per register up to its token quota, round-robin interleaved
    want = {r: quota[r] * EMIT_TOKENS for r in regs}
    got = {r: 0 for r in regs}

[stdout]
        S[k] = (S[k] - v.mean()) / (v.std() + 1e-6)
    assign = S[:, :].argmax(0)
    best = S.max(0)

    # near-duplicate removal (single-band MinHash over 5-grams of the first 512 tokens):
    # duplicated pages otherwise burn budget on text the model has already seen.
    seen, dropped = set(), 0
    for j in kept:
        a = np.asarray(flat[starts[j]:starts[j] + min(lens[j], 512)], dtype=np.int64)
        if len(a) < 6:
            continue
        g = (a[:-4]*1000003 + a[1:-3]*7919 + a[2:-2]*104729 + a[3:-1]*31 + a[4:]) % (1 << 61)
        h = int(g.min())
        if h in seen:
            keep[j] = False; dropped += 1
        else:
            seen.add(h)
    kept = np.flatnonzero(keep)
    print(f"near-dup filter dropped {dropped} docs, {lens[kept].sum()/1e6:.1f}M tokens left",
          flush=True)

    order = {}
    for k, r in enumerate(regs):
        idx = kept[assign[kept] == k]
        idx = idx[np.argsort(-best[idx])]
        order[r] = list(idx)
        print(f"{r}: {len(idx)} docs assigned, {lens[idx].sum()/1e6:.1f}M tokens", flush=True)

    # take top-of-list per register up to its token quota, round-robin interleaved
    want = {r: quota[r] * EMIT_TOKENS for r in regs}
    got = {r: 0 for r in regs}
[metadata] {"noOutputExpected": false}

2726

[stdout]
2726
[metadata] {"noOutputExpected": false}

selected 27691 docs, 24.0M tokens; wiki=9.6M, web=8.4M, qa=6.0M
HELDOUT_PPL 308.182  loss 5.7307  train_s 56  /tmp/t_ca.npy
2556

[stdout]
selected 27691 docs, 24.0M tokens; wiki=9.6M, web=8.4M, qa=6.0M
HELDOUT_PPL 308.182  loss 5.7307  train_s 56  /tmp/t_ca.npy
2556
[metadata] {"noOutputExpected": false}

selected 27691 docs, 24.0M tokens; wiki=9.6M, web=8.4M, qa=6.0M
HELDOUT_PPL 308.182  loss 5.7307  train_s 56  /tmp/t_ca.npy
selected 27589 docs, 24.0M tokens; wiki=8.2M, web=7.9M, qa=7.9M
HELDOUT_PPL 317.607  loss 5.7608  train_s 55  /tmp/t_cb.npy
2386

[stdout]
selected 27691 docs, 24.0M tokens; wiki=9.6M, web=8.4M, qa=6.0M
HELDOUT_PPL 308.182  loss 5.7307  train_s 56  /tmp/t_ca.npy
selected 27589 docs, 24.0M tokens; wiki=8.2M, web=7.9M, qa=7.9M
HELDOUT_PPL 317.607  loss 5.7608  train_s 55  /tmp/t_cb.npy
2386
[metadata] {"noOutputExpected": false}

selected 27691 docs, 24.0M tokens; wiki=9.6M, web=8.4M, qa=6.0M
wrote /workspace/submission/selection.json
matches tested candidate: True n 27691 unique 27691 range 4 181996

[stdout]
selected 27691 docs, 24.0M tokens; wiki=9.6M, web=8.4M, qa=6.0M
wrote /workspace/submission/selection.json
matches tested candidate: True n 27691 unique 27691 range 4 181996
[metadata] {"noOutputExpected": false}

packed 12000000 tokens from 27691 ids -> /tmp/dev_train.npy
HELDOUT_PPL 308.182  loss 5.7307  train_s 55  /tmp/dev_train.npy
(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)
2115

[stdout]
packed 12000000 tokens from 27691 ids -> /tmp/dev_train.npy
HELDOUT_PPL 308.182  loss 5.7307  train_s 55  /tmp/dev_train.npy
(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)
2115
[metadata] {"noOutputExpected": false}

The file /workspace/claim.md has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/claim.md", "oldString": "| **final**: quota 40/35/25 + dedup | **308.2** (317.4 @seed 2024) |", "newString": "| **final (shipped)**: quota 40/35/25 + dedup | **308.2** (317.4 @seed 2024) |", "originalFile": "# Claim: register-balanced target-similarity selection beats undirected quality filtering\n\n## Hypothesis\n\nFor a fixed 12M-token budget (~6% of a 200M-token raw web pool), held-out perplexity on a\n**broad, multi-register** high-quality target is minimised not by \"generic quality\" filtering\nbut by **selecting for similarity to each register of the target separately, under a token\nquota that mirrors the target's mixture**.\n\nConcretely, I claim two separable effects:\n\n1. **Direction** — documents ranked target-like by a cheap linear classifier (dev target as\n   positives, random pool as negatives, GPT-2 unigram+hashed-bigram features) train a much\n   better model than a random draw, and the effect is monotone in selectivity: taking the\n   strict top of the ranking beats randomly sampling a 3×-wider top band.\n2. **Coverage** — because the target is an equal-parts mixture (encyclopedic / general web\n   prose + news / technical Q&A), a *single* undifferentiated ranking under-serves whichever\n   registers the classifier finds least pool-like. Splitting the classifier per register and\n   filling a per-register token quota keeps all registers represented in every prefix of the\n   selection, which matters because the trainer truncates at 12M tokens.\n\nAdditionally, **near-duplicate removal** should help at this budget: duplicated pages spend\nbudget on text the model has already fit.\n\n## Mechanism (observable other than final perplexity)\n\nThe mechanism is that the classifier score is a proxy for *token-distribution overlap with the\ntarget*, and that overlap, not surface \"cleanliness\", is what drives loss. Predictions that are\nobservable without ever training a model:\n\n- **M1.** The unigram cross-entropy of the *target* under the selected corpus's empirical token\n  distribution should fall monotonically as selection tightens: random pool > heuristic-filtered\n  pool > wide-band classifier selection > strict-top classifier selection. This is measurable\n  directly from token counts, with no training run.\n- **M2.** Register assignment should be *sharply* trimodal on the pool, not diffuse: the\n  three per-register classifiers should disagree strongly (a large gap between the top and\n  second z-score for most documents), because the registers differ in surface form (HTML\n  entity/`<code>` markup for Q&A, encyclopedic sentence structure for wiki).\n- **M3.** Near-duplication is a substantial fraction of the *heuristically clean* pool — a\n  single-band 5-gram MinHash should drop a double-digit percentage of documents. (Observed:\n  31,017 of 136,895 kept docs, ~23%.)\n- **M4.** If the mechanism is coverage rather than raw purity, then the *marginal* value of\n  extra tokens of the register that is already over-represented in the selection should be\n  negative — i.e. quota curves should be unimodal, with a plateau near the target's mixture and\n  degradation as the quota is skewed toward any single register.\n\n## Falsification\n\nThe claim is falsified if, at the same 12M budget and identical frozen recipe:\n\n- **F1.** A random selection matched only on the heuristic prose filter (no classifier) reaches\n  perplexity within noise of the classifier-selected corpus → *direction* effect is not real.\n- **F2.** Randomly sampling a much wider top band (3× the needed tokens) does **not** hurt →\n  selectivity/purity does not matter, and any \"target-like\" set of clean prose would do.\n- **F3.** A single undifferentiated ranking (no register split, no quota) matches the\n  register-balanced selection → the *coverage* claim is unnecessary.\n- **F4.** Skewing the quota heavily toward one register (e.g. 50% encyclopedic) is neutral or\n  better → the mixture-matching claim is wrong.\n\nStatus against the dev target (`multi_dev.npy`, frozen recipe, dev perplexity; seed 1337 unless\nnoted). Run-to-run seed noise is ~±5–9 ppl, measured by re-running two configurations under a\nsecond seed, so differences below ~10 ppl are not resolved:\n\n| selection | dev PPL |\n|---|---|\n| random pool (do-nothing baseline) | 469.9 |\n| classifier + quota 30/45/25, no dedup | 325.8 |\n| classifier + equal quota, no dedup | 322.3 |\n| **F2 test**: 3×-wide top band, random within band | 385.9 |\n| classifier + equal quota + dedup | 317.6 (311.3 @seed 2024) |\n| **final**: quota 40/35/25 + dedup | **308.2** (317.4 @seed 2024) |\n| **F4 test**: quota 50/30/20 + dedup | 332.1 |\n| length floor ≥512 tokens (equal quota) | 328.4 |\n| 3-band (more aggressive) dedup | 323.2 |\n\nF2 and F4 are clearly **not** triggered: widening the band costs ~68 ppl and skewing the quota\nto 50% encyclopedic costs ~24 ppl, both far outside seed noise. The direction effect (470 → ~310,\na 34% reduction) is an order of magnitude larger than noise. F1 and F3 were not run for lack of\nwall-clock, so the *decomposition* between \"clean prose\" and \"target-like\" is the weakest part of\nthis claim — the heuristic filter alone might account for part of the gain. What is *not*\nresolved by these runs is the fine quota shape: 40/35/25, 34/33/33 and 37/34/29 all lie within\nnoise of each other (two-seed means 312.8 vs 314.5), so the final quota choice is a weak\npreference, and only the coarse claim \"roughly mirror the mixture, do not skew hard\" is supported.\n\n## Transfer\n\n- **Budget.** The mechanism is budget-relative: the tighter the budget as a fraction of the pool,\n  the more purity should dominate diversity. At a 10× larger budget the wide-band variant should\n  close most of its gap, and dedup should matter more (repeats become unavoidable).\n- **Target.** Nothing here is specific to *these* four registers. Given any disclosed target\n  sample, the same recipe applies: decode it, weakly segment it into registers by surface cues,\n  fit one cheap linear classifier per register against a random pool sample, fill quotas. Only\n  the segmentation heuristic (`register_of_dev`) is target-specific, and it is three lines.\n- **Model scale.** The selection is model-free — it never runs the trained model — so it should\n  transfer across model sizes, though the optimal selectivity should loosen with capacity: a\n  larger model benefits from more diverse data before saturating.\n- **Known leak-ish caveat.** The dev target is *disclosed* by the task, and the classifier uses\n  it as its positive class. The official target is a disjoint sample of the same domains, so this\n  is domain transfer, not memorisation — but the selection would not transfer to an *undisclosed*\n  target, where one would substitute a proxy corpus for the positive class.\n", "structuredPatch": [{"oldStart": 68, "oldLines": 7, "newStart": 68, "newLines": 7, "lines": [" | classifier + equal quota, no dedup | 322.3 |", " | **F2 test**: 3×-wide top band, random within band | 385.9 |", " | classifier + equal quota + dedup | 317.6 (311.3 @seed 2024) |", "-| **final**: quota 40/35/25 + dedup | **308.2** (317.4 @seed 2024) |", "+| **final (shipped)**: quota 40/35/25 + dedup | **308.2** (317.4 @seed 2024) |", " | **F4 test**: quota 50/30/20 + dedup | 332.1 |", " | length floor ≥512 tokens (equal quota) | 328.4 |", " | 3-band (more aggressive) dedup | 323.2 |"]}], "userModified": false, "replaceAll": false}

The file /workspace/claim.md has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/claim.md", "oldString": "F2 and F4 are clearly **not** triggered:", "newString": "The shipped `selection.json` was regenerated by `submission/curate.py` with its default\nsettings and re-verified through the official `dev_eval.sh` path (which re-tokenises from\n`pool.jsonl` via `pack_selection.py`): **308.182**, bit-identical to the fast-packed run above.\nCandidate selection among quota settings used the trainer's fixed seed (1337): since the frozen\nrecipe is deterministic given the token array, the model measured here is the same model the\nverifier trains, and only the eval *sample* differs.\n\nF2 and F4 are clearly **not** triggered:", "originalFile": "# Claim: register-balanced target-similarity selection beats undirected quality filtering\n\n## Hypothesis\n\nFor a fixed 12M-token budget (~6% of a 200M-token raw web pool), held-out perplexity on a\n**broad, multi-register** high-quality target is minimised not by \"generic quality\" filtering\nbut by **selecting for similarity to each register of the target separately, under a token\nquota that mirrors the target's mixture**.\n\nConcretely, I claim two separable effects:\n\n1. **Direction** — documents ranked target-like by a cheap linear classifier (dev target as\n   positives, random pool as negatives, GPT-2 unigram+hashed-bigram features) train a much\n   better model than a random draw, and the effect is monotone in selectivity: taking the\n   strict top of the ranking beats randomly sampling a 3×-wider top band.\n2. **Coverage** — because the target is an equal-parts mixture (encyclopedic / general web\n   prose + news / technical Q&A), a *single* undifferentiated ranking under-serves whichever\n   registers the classifier finds least pool-like. Splitting the classifier per register and\n   filling a per-register token quota keeps all registers represented in every prefix of the\n   selection, which matters because the trainer truncates at 12M tokens.\n\nAdditionally, **near-duplicate removal** should help at this budget: duplicated pages spend\nbudget on text the model has already fit.\n\n## Mechanism (observable other than final perplexity)\n\nThe mechanism is that the classifier score is a proxy for *token-distribution overlap with the\ntarget*, and that overlap, not surface \"cleanliness\", is what drives loss. Predictions that are\nobservable without ever training a model:\n\n- **M1.** The unigram cross-entropy of the *target* under the selected corpus's empirical token\n  distribution should fall monotonically as selection tightens: random pool > heuristic-filtered\n  pool > wide-band classifier selection > strict-top classifier selection. This is measurable\n  directly from token counts, with no training run.\n- **M2.** Register assignment should be *sharply* trimodal on the pool, not diffuse: the\n  three per-register classifiers should disagree strongly (a large gap between the top and\n  second z-score for most documents), because the registers differ in surface form (HTML\n  entity/`<code>` markup for Q&A, encyclopedic sentence structure for wiki).\n- **M3.** Near-duplication is a substantial fraction of the *heuristically clean* pool — a\n  single-band 5-gram MinHash should drop a double-digit percentage of documents. (Observed:\n  31,017 of 136,895 kept docs, ~23%.)\n- **M4.** If the mechanism is coverage rather than raw purity, then the *marginal* value of\n  extra tokens of the register that is already over-represented in the selection should be\n  negative — i.e. quota curves should be unimodal, with a plateau near the target's mixture and\n  degradation as the quota is skewed toward any single register.\n\n## Falsification\n\nThe claim is falsified if, at the same 12M budget and identical frozen recipe:\n\n- **F1.** A random selection matched only on the heuristic prose filter (no classifier) reaches\n  perplexity within noise of the classifier-selected corpus → *direction* effect is not real.\n- **F2.** Randomly sampling a much wider top band (3× the needed tokens) does **not** hurt →\n  selectivity/purity does not matter, and any \"target-like\" set of clean prose would do.\n- **F3.** A single undifferentiated ranking (no register split, no quota) matches the\n  register-balanced selection → the *coverage* claim is unnecessary.\n- **F4.** Skewing the quota heavily toward one register (e.g. 50% encyclopedic) is neutral or\n  better → the mixture-matching claim is wrong.\n\nStatus against the dev target (`multi_dev.npy`, frozen recipe, dev perplexity; seed 1337 unless\nnoted). Run-to-run seed noise is ~±5–9 ppl, measured by re-running two configurations under a\nsecond seed, so differences below ~10 ppl are not resolved:\n\n| selection | dev PPL |\n|---|---|\n| random pool (do-nothing baseline) | 469.9 |\n| classifier + quota 30/45/25, no dedup | 325.8 |\n| classifier + equal quota, no dedup | 322.3 |\n| **F2 test**: 3×-wide top band, random within band | 385.9 |\n| classifier + equal quota + dedup | 317.6 (311.3 @seed 2024) |\n| **final (shipped)**: quota 40/35/25 + dedup | **308.2** (317.4 @seed 2024) |\n| **F4 test**: quota 50/30/20 + dedup | 332.1 |\n| length floor ≥512 tokens (equal quota) | 328.4 |\n| 3-band (more aggressive) dedup | 323.2 |\n\nF2 and F4 are clearly **not** triggered: widening the band costs ~68 ppl and skewing the quota\nto 50% encyclopedic costs ~24 ppl, both far outside seed noise. The direction effect (470 → ~310,\na 34% reduction) is an order of magnitude larger than noise. F1 and F3 were not run for lack of\nwall-clock, so the *decomposition* between \"clean prose\" and \"target-like\" is the weakest part of\nthis claim — the heuristic filter alone might account for part of the gain. What is *not*\nresolved by these runs is the fine quota shape: 40/35/25, 34/33/33 and 37/34/29 all lie within\nnoise of each other (two-seed means 312.8 vs 314.5), so the final quota choice is a weak\npreference, and only the coarse claim \"roughly mirror the mixture, do not skew hard\" is supported.\n\n## Transfer\n\n- **Budget.** The mechanism is budget-relative: the tighter the budget as a fraction of the pool,\n  the more purity should dominate diversity. At a 10× larger budget the wide-band variant should\n  close most of its gap, and dedup should matter more (repeats become unavoidable).\n- **Target.** Nothing here is specific to *these* four registers. Given any disclosed target\n  sample, the same recipe applies: decode it, weakly segment it into registers by surface cues,\n  fit one cheap linear classifier per register against a random pool sample, fill quotas. Only\n  the segmentation heuristic (`register_of_dev`) is target-specific, and it is three lines.\n- **Model scale.** The selection is model-free — it never runs the trained model — so it should\n  transfer across model sizes, though the optimal selectivity should loosen with capacity: a\n  larger model benefits from more diverse data before saturating.\n- **Known leak-ish caveat.** The dev target is *disclosed* by the task, and the classifier uses\n  it as its positive class. The official target is a disjoint sample of the same domains, so this\n  is domain transfer, not memorisation — but the selection would not transfer to an *undisclosed*\n  target, where one would substitute a proxy corpus for the positive class.\n", "structuredPatch": [{"oldStart": 73, "oldLines": 6, "newStart": 73, "newLines": 13, "lines": [" | length floor ≥512 tokens (equal quota) | 328.4 |", " | 3-band (more aggressive) dedup | 323.2 |", " ", "+The shipped `selection.json` was regenerated by `submission/curate.py` with its default", "+settings and re-verified through the official `dev_eval.sh` path (which re-tokenises from", "+`pool.jsonl` via `pack_selection.py`): **308.182**, bit-identical to the fast-packed run above.", "+Candidate selection among quota settings used the trainer's fixed seed (1337): since the frozen", "+recipe is deterministic given the token array, the model measured here is the same model the", "+verifier trains, and only the eval *sample* differs.", "+", " F2 and F4 are clearly **not** triggered: widening the band costs ~68 ppl and skewing the quota", " to 50% encyclopedic costs ~24 ppl, both far outside seed noise. The direction effect (470 → ~310,", " a 34% reduction) is an order of magnitude larger than noise. F1 and F3 were not run for lack of"]}], "userModified": false, "replaceAll": false}

-rw-r--r-- 1 root root 7083 Aug  4 00:21 /workspace/claim.md

/workspace/submission:
total 205
drwxr-xr-x 1 root root    140 Aug  3 23:46 .
drwxr-xr-x 1 root root    240 Aug  4 00:21 ..
drwxr-xr-x 1 root root     80 Aug  3 23:46 __pycache__
-rw-r--r-- 1 root root  11607 Aug  4 00:08 curate.py
-rw-r--r-- 1 root root 197477 Aug  4 00:19 selection.json
all ids in pool: True n= 27691
2095

[stdout]
-rw-r--r-- 1 root root 7083 Aug  4 00:21 /workspace/claim.md

/workspace/submission:
total 205
drwxr-xr-x 1 root root    140 Aug  3 23:46 .
drwxr-xr-x 1 root root    240 Aug  4 00:21 ..
drwxr-xr-x 1 root root     80 Aug  3 23:46 __pycache__
-rw-r--r-- 1 root root  11607 Aug  4 00:08 curate.py
-rw-r--r-- 1 root root 197477 Aug  4 00:19 selection.json
all ids in pool: True n= 27691
2095
[metadata] {"noOutputExpected": false}