claude-opus-5

packed 12000000 tokens from 21273 ids -> /tmp/dev_train.npy
n 21273 unique 21273 min 9 max 181984
-rw-r--r-- 1 root root 4758 Aug  4 01:49 /workspace/claim.md

/workspace/submission:
total 158
drwxr-xr-x 1 root root    140 Aug  4 01:46 .
drwxr-xr-x 1 root root    180 Aug  4 01:49 ..
-rw-r--r-- 1 root root   8307 Aug  4 01:46 curate.py
-rw-r--r-- 1 root root 152843 Aug  4 01:48 selection.json

[stdout]
packed 12000000 tokens from 21273 ids -> /tmp/dev_train.npy
n 21273 unique 21273 min 9 max 181984
-rw-r--r-- 1 root root 4758 Aug  4 01:49 /workspace/claim.md

/workspace/submission:
total 158
drwxr-xr-x 1 root root    140 Aug  4 01:46 .
drwxr-xr-x 1 root root    180 Aug  4 01:49 ..
-rw-r--r-- 1 root root   8307 Aug  4 01:46 curate.py
-rw-r--r-- 1 root root 152843 Aug  4 01:48 selection.json
[metadata] {"noOutputExpected": false}

HELDOUT_PPL 381.128  loss 5.9431  train_s 54  /tmp/dev_train.npy

[stdout]
HELDOUT_PPL 381.128  loss 5.9431  train_s 54  /tmp/dev_train.npy
[metadata] {"noOutputExpected": false}

The file /workspace/claim.md has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/claim.md", "oldString": "  | **gate + tilt 0.5 (submitted)** | **383.5** |", "newString": "  | **gate + tilt 0.5 (submitted)** | **383.5** (381.1 on a repeat run of the final file; run-to-run spread ~2.5) |", "originalFile": "# Claim: at fixed tiny compute, junk removal dominates and domain matching must be *gentle*\n\n## Hypothesis\n\nFor a 30M GPT trained on a 12M-token budget, held-out perplexity on a broad\nhigh-quality English target is driven by two separable effects, in this order:\n\n1. **Removing non-prose junk** from the raw web pool (link farms, navigation and\n   boilerplate dumps, numeric tables, encoding noise). This is the large effect.\n2. A **mild** tilt of the surviving distribution toward the target domain. The\n   optimum is a *soft* reweighting, not a hard top-k: because the model is far\n   from converged at this budget, it still needs broad coverage of the target's\n   four registers (encyclopedic, general web prose, news, technical Q&A), and\n   the extreme tail of any learned quality score is stylistically degenerate.\n\nConcretely: score every doc with a fastText-style classifier separating the\ndisclosed dev target from a random pool sample, then order docs by\n`0.5 * score + Gumbel` (exact weighted sampling-without-replacement with weight\n`exp(0.5 * score)`) over the gate survivors.\n\n## Mechanism — observable predictions other than final perplexity\n\n- **The classifier is genuinely informative, yet its argmax is bad.** Held-out\n  AUC separating dev-target text from pool text is high (~0.95), so the ranking\n  is not noise — but perplexity as a function of tilt strength is **U-shaped**,\n  with hard top-k *worse than doing nothing*. Measured on dev:\n\n  | selection (12M tokens, frozen trainer, seed 1337) | dev PPL |\n  |---|---|\n  | random pool sample (do-nothing baseline) | 469.9 |\n  | gate only, random order | 417.4 |\n  | gate + tilt 0.3 | 389.9 |\n  | **gate + tilt 0.5 (submitted)** | **383.5** |\n  | gate + tilt 0.8 | 395.0 |\n  | gate + tilt 2.0 | 416.5 |\n  | gate + hard top-k (tilt → ∞) | 463.5 |\n\n  The interior optimum at ≈0.5 is the signature of the mechanism: the score is\n  useful as a *prior*, harmful as a *filter*.\n- **Over-tightening the junk gate also hurts**, for the same coverage reason. A\n  strict gate (≥1200 chars, ≥70% alpha, ≤2% non-ASCII, ≥8 sentences, stopword\n  floor, sentence-length band) keeps only 89.9k of 182k docs and scores 437.1 —\n  worse than the permissive gate that keeps 152.5k (417.4). Filtering past the\n  junk boundary starts deleting legitimate registers (the target's technical Q&A\n  quarter is HTML-marked-up and short-lined, so strict prose gates cut it).\n- **Inspecting the score's extreme tail** shows why top-k fails: the highest-margin\n  documents are machine-translated encyclopedic fragments and fictional-wiki\n  articles — locally target-like in vocabulary, atypical in everything else.\n\n## Falsification\n\nThe claim is wrong if any of these hold:\n\n- Hard top-k by the classifier score **beats** gate+tilt-0.5 on the hidden target.\n  (Observed on dev: 463.5 vs 383.5 — falsified there.)\n- The gate contributes nothing once the tilt is applied: i.e. tilt-0.5 applied to\n  the *ungated* pool matches 383.5. My decomposition predicts it should land\n  meaningfully worse, since ~16% of the pool is junk the tilt only partially\n  suppresses.\n- Perplexity is **monotone decreasing** in tilt strength (no interior optimum) on\n  the hidden target. That would mean the selection is simply under-tilted rather\n  than optimally soft, and the mechanism story is wrong.\n- Stricter gating (the strict variant above) beats the permissive gate on the\n  hidden target, which would mean the loss came from noise tolerance rather than\n  register coverage.\n\n## Transfer\n\n- **Any fixed-compute pretraining run far from convergence** should use quality\n  scores as a soft reweighting, and tune the temperature — reporting only \"we\n  took the top-k% by a quality classifier\" leaves most of the gain, and can be\n  net-negative. One scalar (tilt strength) is worth more here than a better\n  classifier: going from AUC 0.79 to AUC 0.95 at fixed hard top-k moved dev PPL\n  only 472 → 463, while changing top-k to tilt-0.5 moved it 463 → 384.\n- **Multi-register targets penalize narrowing.** When the eval is a mixture,\n  filters must be calibrated to the junk boundary, not to the most prestigious\n  register; each additional tightening trades coverage of one eval quarter for\n  purity in another.\n- The Gumbel-top-k trick makes soft selection a drop-in replacement for ranking:\n  ordering by `TILT * score + Gumbel` needs no resampling machinery and still\n  produces a *priority-ordered* list, so it works with any pipeline that consumes\n  a ranked id list under a budget.\n- The gate/tilt split is cheap and label-free: gates are regex-level statistics,\n  and the classifier's positives come from the eval-domain sample you already\n  have. No pretrained scorer or external labels required.\n", "structuredPatch": [{"oldStart": 30, "oldLines": 7, "newStart": 30, "newLines": 7, "lines": ["   | random pool sample (do-nothing baseline) | 469.9 |", "   | gate only, random order | 417.4 |", "   | gate + tilt 0.3 | 389.9 |", "-  | **gate + tilt 0.5 (submitted)** | **383.5** |", "+  | **gate + tilt 0.5 (submitted)** | **383.5** (381.1 on a repeat run of the final file; run-to-run spread ~2.5) |", "   | gate + tilt 0.8 | 395.0 |", "   | gate + tilt 2.0 | 416.5 |", "   | gate + hard top-k (tilt → ∞) | 463.5 |"]}], "userModified": false, "replaceAll": false}