nano-gpt-data-curation

claude-opus-5

nano-gpt-data-curation — graded by the task's verifier against a measured baseline.

backend: modal
variant: claude claude-opus-5
metric: held-out multi-domain perplexity
actions: 34
duration: 38m
recorded: 2026-08-03

run iddclm-nano-curation__HBfU9js

Everything claude-opus-5 did on this task, in order.

download JSON ↗

claude-codeclaude-opus-533 turns3 narration34 action34 observation30 reasoning1 final

#26

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolBash

args

{
  "command": "python - <<'EOF'\ns=open('curate.py').read()\ns=s.replace('''    owner0 = scores.argmax(0)\n    strict = (alpha > 0.85) & (rep < 0.55) & (nwords >= 150)\n    loose = (alpha > 0.55) & (rep < 0.75) & (nwords >= 60)\n    keep = (nchars >= MIN_CHARS) & np.where(owner0 == 3, loose, strict)''','''    keep = (nchars >= MIN_CHARS) & (alpha > 0.85) & (rep < 0.55) & (nwords >= 150)''')\ns=s.replace('MIX = [0.25, 0.25, 0.25, 0.25]','MIX = [0.30, 0.30, 0.25, 0.15]')\nopen('curate.py','w').write(s)\nEOF\npython curate.py 2>&1 | tail -2 && bash dev_eval.sh 2>&1 | grep HELDOUT",
  "timeout": 900000
}

Bash result

qa avail 4163 picked 3735 est tokens 2252054
wrote 28813 ids -> /workspace/submission/selection.json est tokens 24768772
HELDOUT_PPL 360.215  loss 5.8867  train_s 54  /tmp/dev_train.npy

[stdout]
qa avail 4163 picked 3735 est tokens 2252054
wrote 28813 ids -> /workspace/submission/selection.json est tokens 24768772
HELDOUT_PPL 360.215  loss 5.8867  train_s 54  /tmp/dev_train.npy
[metadata] {"noOutputExpected": false}

#27

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolBash

args

{
  "command": "python - <<'EOF'\ns=open('curate.py').read()\ns=s.replace('MIX = [0.30, 0.30, 0.25, 0.15]','MIX = [0.25, 0.25, 0.25, 0.25]')\ns=s.replace('''    keep = (nchars >= MIN_CHARS) & (alpha > 0.85) & (rep < 0.55) & (nwords >= 150)''','''    # Register-aware gate: the boilerplate/repetition filter is identical for\n    # all registers, but technical Q&A documents are legitimately shorter, so\n    # the length floor is relaxed for that register only.\n    owner0 = scores.argmax(0)\n    base = (alpha > 0.85) & (rep < 0.55)\n    keep = (nchars >= MIN_CHARS) & base & np.where(owner0 == 3, nwords >= 70, nwords >= 150)''')\nopen('curate.py','w').write(s)\nEOF\npython curate.py 2>&1 | tail -3 && bash dev_eval.sh 2>&1 | grep HELDOUT",
  "timeout": 900000
}

Bash result

news avail 9926 picked 9349 est tokens 7506219
qa avail 6969 picked 6339 est tokens 2720493
wrote 31417 ids -> /workspace/submission/selection.json est tokens 25237212
HELDOUT_PPL 358.657  loss 5.8824  train_s 54  /tmp/dev_train.npy

[stdout]
news avail 9926 picked 9349 est tokens 7506219
qa avail 6969 picked 6339 est tokens 2720493
wrote 31417 ids -> /workspace/submission/selection.json est tokens 25237212
HELDOUT_PPL 358.657  loss 5.8824  train_s 54  /tmp/dev_train.npy
[metadata] {"noOutputExpected": false}

#28

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolBash

args

{
  "command": "python - <<'EOF'\ns=open('curate.py').read()\ns=s.replace('''    # Register-aware gate: the boilerplate/repetition filter is identical for\n    # all registers, but technical Q&A documents are legitimately shorter, so\n    # the length floor is relaxed for that register only.\n    owner0 = scores.argmax(0)\n    base = (alpha > 0.85) & (rep < 0.55)\n    keep = (nchars >= MIN_CHARS) & base & np.where(owner0 == 3, nwords >= 70, nwords >= 150)''','''    keep = (nchars >= MIN_CHARS) & (alpha > 0.85) & (rep < 0.55) & (nwords >= 150)''')\nopen('curate.py','w').write(s)\nEOF\npython curate.py 2>&1 | tail -6 && cp curate.py submission/curate.py && bash dev_eval.sh 2>&1 | grep HELDOUT",
  "timeout": 900000
}

Bash result

kept by gate: 101171 / 182016
wiki avail 17297 picked 6936 est tokens 7510129
web avail 69785 picked 8793 est tokens 7500369
news avail 9926 picked 9349 est tokens 7506219
qa avail 4163 picked 3735 est tokens 2252054
wrote 28813 ids -> /workspace/submission/selection.json est tokens 24768772
HELDOUT_PPL 350.272  loss 5.8587  train_s 54  /tmp/dev_train.npy

[stdout]
kept by gate: 101171 / 182016
wiki avail 17297 picked 6936 est tokens 7510129
web avail 69785 picked 8793 est tokens 7500369
news avail 9926 picked 9349 est tokens 7506219
qa avail 4163 picked 3735 est tokens 2252054
wrote 28813 ids -> /workspace/submission/selection.json est tokens 24768772
HELDOUT_PPL 350.272  loss 5.8587  train_s 54  /tmp/dev_train.npy
[metadata] {"noOutputExpected": false}

#29

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolWrite

args

{
  "file_path": "/workspace/claim.md",
  "content": "# Claim\n\n## Hypothesis\nUnder a token budget ~6% of the pool, held-out perplexity on a *multi-register*\ntarget is governed less by generic \"web quality\" than by **per-register\ndistributional match**. Selecting documents by a length-normalised\nlog-likelihood ratio against each of the four target registers separately, and\nthen emitting them so that every prefix of the priority list is an equal-token\nmixture of the four, beats both a random selection and a single global\nquality score.\n\nConcretely: score_d(doc) = (1/N) Σ_t log( λ·p_d(t)/p_0(t) + (1−λ) ), where p_d is\nthe unigram+bigram distribution of target register d (wiki / general web prose /\nnews / technical Q&A), p_0 is the raw-pool background, λ = 0.7. Each document is\nowned by its argmax register; each register greedily takes its top-scored,\nde-duplicated documents; output interleaves registers by accumulated tokens.\n\n## Mechanism (observable that is *not* the final perplexity)\nThe mechanism is that the LLR ranks documents by *expected cross-entropy\nreduction on the target*, so the selected set should differ from the pool in\nmeasurable surface statistics before any training happens:\n\n1. **The interpolation is what makes the score a match score, not a novelty\n   score.** With add-1 smoothing instead of Jelinek–Mercer, the same pipeline's\n   top-ranked documents are observably degenerate — non-English pages, nav-bar\n   link lists, brand indexes (verified by inspecting the top-12 documents). The\n   bounded form log(λ·p_d/p_0 + 1−λ) is what puts encyclopedic prose, wire-service\n   news and StackExchange threads at the top. Prediction: any scorer whose\n   per-token weight is unbounded below will preferentially select\n   vocabulary-novel boilerplate, regardless of downstream training.\n2. **Supply is register-asymmetric.** Prediction (confirmed on this pool): after\n   gating, \"general web prose\" is ~30× more abundant than \"technical Q&A\"\n   (68M vs 2M tokens). So an unconditional target-similarity filter must collapse\n   the mixture toward web prose; only per-register conditioning holds the\n   equal-parts mixture, which is why balancing helps.\n3. Duplicate structure: a plain top-k selection of this pool draws visible\n   near-duplicate page families (same body, different boilerplate); the head/tail\n   word-sequence dedup removes them and frees budget for unique text.\n\n## Falsification\nThe hypothesis is falsified if, holding everything else fixed:\n- a single global target classifier (positives = all four registers pooled)\n  matches or beats the per-register version; or\n- deliberately *unbalancing* the mixture improves dev perplexity.\n\nBoth were tested. Measured dev perplexity (frozen `train_nano.py`, seed 1337,\n12M tokens, `multi_dev.npy`):\n\n| selection | dev PPL |\n|---|---|\n| random baseline (do-nothing) | 457.99 |\n| add-1-smoothed LLR, doc-balanced (early version) | 463.51 |\n| Jelinek–Mercer LLR, per-register, doc round-robin | 356.26 |\n| **+ token-balanced interleave (final, with bigrams)** | **350.27** |\n| final, but mixture skewed to [.30 .30 .25 .15] | 360.22 |\n| final, but loosened gate for the Q&A register | 365.35 |\n\nSo the \"vocabulary-novelty\" variant is *worse than random* (falsifying the naive\nversion of the idea), and both perturbations of the balanced mixture are worse\nthan equal parts — consistent with the hypothesis. The remaining risk is dev\noverfitting: positives come from `multi_dev.npy`, and the official target is a\ndisjoint sample of the same domain. Because the scorer uses only unigram/bigram\n*register* statistics (not memorised content), the transfer loss should be small;\nif the hidden-target perplexity is far above 350 while dev is 350, that is the\nfalsifying observation.\n\n## Transfer\nThe recipe is domain-agnostic and needs only (a) a sample of the target\ndistribution and (b) knowledge of its register composition: estimate a background\ndistribution from the raw pool, build one interpolated LLR scorer per target\nregister, gate obvious boilerplate, de-duplicate, and emit prefix-balanced by\ntokens. It costs one CPU pass over the pool (~40 s for 182k documents here, no\nGPU, no neural scorer) and scales to larger pools linearly. It should transfer to\nany budgeted pretraining-data curation where the evaluation mixture is known, and\nthe prefix-balancing property means the same list stays valid if the token budget\nis later raised or lowered.\n"
}

Write result

File created successfully at: /workspace/claim.md (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/workspace/claim.md", "content": "# Claim\n\n## Hypothesis\nUnder a token budget ~6% of the pool, held-out perplexity on a *multi-register*\ntarget is governed less by generic \"web quality\" than by **per-register\ndistributional match**. Selecting documents by a length-normalised\nlog-likelihood ratio against each of the four target registers separately, and\nthen emitting them so that every prefix of the priority list is an equal-token\nmixture of the four, beats both a random selection and a single global\nquality score.\n\nConcretely: score_d(doc) = (1/N) Σ_t log( λ·p_d(t)/p_0(t) + (1−λ) ), where p_d is\nthe unigram+bigram distribution of target register d (wiki / general web prose /\nnews / technical Q&A), p_0 is the raw-pool background, λ = 0.7. Each document is\nowned by its argmax register; each register greedily takes its top-scored,\nde-duplicated documents; output interleaves registers by accumulated tokens.\n\n## Mechanism (observable that is *not* the final perplexity)\nThe mechanism is that the LLR ranks documents by *expected cross-entropy\nreduction on the target*, so the selected set should differ from the pool in\nmeasurable surface statistics before any training happens:\n\n1. **The interpolation is what makes the score a match score, not a novelty\n   score.** With add-1 smoothing instead of Jelinek–Mercer, the same pipeline's\n   top-ranked documents are observably degenerate — non-English pages, nav-bar\n   link lists, brand indexes (verified by inspecting the top-12 documents). The\n   bounded form log(λ·p_d/p_0 + 1−λ) is what puts encyclopedic prose, wire-service\n   news and StackExchange threads at the top. Prediction: any scorer whose\n   per-token weight is unbounded below will preferentially select\n   vocabulary-novel boilerplate, regardless of downstream training.\n2. **Supply is register-asymmetric.** Prediction (confirmed on this pool): after\n   gating, \"general web prose\" is ~30× more abundant than \"technical Q&A\"\n   (68M vs 2M tokens). So an unconditional target-similarity filter must collapse\n   the mixture toward web prose; only per-register conditioning holds the\n   equal-parts mixture, which is why balancing helps.\n3. Duplicate structure: a plain top-k selection of this pool draws visible\n   near-duplicate page families (same body, different boilerplate); the head/tail\n   word-sequence dedup removes them and frees budget for unique text.\n\n## Falsification\nThe hypothesis is falsified if, holding everything else fixed:\n- a single global target classifier (positives = all four registers pooled)\n  matches or beats the per-register version; or\n- deliberately *unbalancing* the mixture improves dev perplexity.\n\nBoth were tested. Measured dev perplexity (frozen `train_nano.py`, seed 1337,\n12M tokens, `multi_dev.npy`):\n\n| selection | dev PPL |\n|---|---|\n| random baseline (do-nothing) | 457.99 |\n| add-1-smoothed LLR, doc-balanced (early version) | 463.51 |\n| Jelinek–Mercer LLR, per-register, doc round-robin | 356.26 |\n| **+ token-balanced interleave (final, with bigrams)** | **350.27** |\n| final, but mixture skewed to [.30 .30 .25 .15] | 360.22 |\n| final, but loosened gate for the Q&A register | 365.35 |\n\nSo the \"vocabulary-novelty\" variant is *worse than random* (falsifying the naive\nversion of the idea), and both perturbations of the balanced mixture are worse\nthan equal parts — consistent with the hypothesis. The remaining risk is dev\noverfitting: positives come from `multi_dev.npy`, and the official target is a\ndisjoint sample of the same domain. Because the scorer uses only unigram/bigram\n*register* statistics (not memorised content), the transfer loss should be small;\nif the hidden-target perplexity is far above 350 while dev is 350, that is the\nfalsifying observation.\n\n## Transfer\nThe recipe is domain-agnostic and needs only (a) a sample of the target\ndistribution and (b) knowledge of its register composition: estimate a background\ndistribution from the raw pool, build one interpolated LLR scorer per target\nregister, gate obvious boilerplate, de-duplicate, and emit prefix-balanced by\ntokens. It costs one CPU pass over the pool (~40 s for 182k documents here, no\nGPU, no neural scorer) and scales to larger pools linearly. It should transfer to\nany budgeted pretraining-data curation where the evaluation mixture is known, and\nthe prefix-balancing property means the same list stays valid if the token budget\nis later raised or lowered.\n", "structuredPatch": [], "originalFile": null, "userModified": false}

#30

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolBash

args

{
  "command": "python -c \"\nimport json\ns=json.load(open('submission/selection.json'))\nprint(len(s), len(set(s)), min(s), max(s), type(s[0]))\n\" && ls -l submission && diff <(cat curate.py) submission/curate.py && echo SAME; cat .timer/remaining_secs"
}

Bash result

28813 28813 1 182015 <class 'int'>
total 211
-rw-r--r-- 1 root root   7915 Aug  3 23:52 curate.py
-rw-r--r-- 1 root root 207384 Aug  3 23:52 selection.json
SAME
3718

[stdout]
28813 28813 1 182015 <class 'int'>
total 211
-rw-r--r-- 1 root root   7915 Aug  3 23:52 curate.py
-rw-r--r-- 1 root root 207384 Aug  3 23:52 selection.json
SAME
3718
[metadata] {"noOutputExpected": false}

#31

narration

Deliverables are in place (dev PPL 350.3 vs 458 random baseline). Let me try one more improvement with the remaining time.

toolBash

args

{
  "command": "cp submission/selection.json /tmp/best350.json\npython - <<'EOF'\ns=open('curate.py').read()\ns=s.replace('''    owner = scores.argmax(0)''','''    # Ownership by scarcity-first claim order: the scarcest register picks its\n    # top documents from the whole gated pool first, so a genuinely Q&A-like\n    # document is not lost to a marginally higher \"general web\" score.\n    owner = np.full(len(ids), -1)\n    for k in [3, 0, 2, 1]:\n        cand = np.flatnonzero(keep & (owner < 0))\n        cand = cand[np.argsort(-scores[k][cand])]\n        cum = np.cumsum(est_tok[cand])\n        cut = min(len(cand), int(np.searchsorted(cum, BUDGET * OVERSHOOT / 4)) + 1)\n        owner[cand[:cut]] = k''')\nopen('curate.py','w').write(s)\nEOF\npython curate.py 2>&1 | tail -5 && bash dev_eval.sh 2>&1 | grep HELDOUT",
  "timeout": 900000
}

Bash result

wiki avail 7109 picked 7069 est tokens 7381798
web avail 7103 picked 7058 est tokens 7366381
news avail 8493 picked 8408 est tokens 7239544
qa avail 11531 picked 11263 est tokens 7246285
wrote 33798 ids -> /workspace/submission/selection.json est tokens 29234008
HELDOUT_PPL 359.375  loss 5.8844  train_s 54  /tmp/dev_train.npy

[stdout]
wiki avail 7109 picked 7069 est tokens 7381798
web avail 7103 picked 7058 est tokens 7366381
news avail 8493 picked 8408 est tokens 7239544
qa avail 11531 picked 11263 est tokens 7246285
wrote 33798 ids -> /workspace/submission/selection.json est tokens 29234008
HELDOUT_PPL 359.375  loss 5.8844  train_s 54  /tmp/dev_train.npy
[metadata] {"noOutputExpected": false}

#32

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolBash

args

{
  "command": "python - <<'EOF'\ns=open('curate.py').read()\ns=s.replace('''    # Ownership by scarcity-first claim order: the scarcest register picks its\n    # top documents from the whole gated pool first, so a genuinely Q&A-like\n    # document is not lost to a marginally higher \"general web\" score.\n    owner = np.full(len(ids), -1)\n    for k in [3, 0, 2, 1]:\n        cand = np.flatnonzero(keep & (owner < 0))\n        cand = cand[np.argsort(-scores[k][cand])]\n        cum = np.cumsum(est_tok[cand])\n        cut = min(len(cand), int(np.searchsorted(cum, BUDGET * OVERSHOOT / 4)) + 1)\n        owner[cand[:cut]] = k''','''    owner = scores.argmax(0)''')\nopen('curate.py','w').write(s)\nEOF\npython curate.py 2>&1 | tail -2 && cp curate.py submission/curate.py\npython -c \"\nimport json;a=json.load(open('submission/selection.json'));b=json.load(open('/tmp/best350.json'));print('matches best config:',a==b,len(a))\""
}

Bash result

qa avail 4163 picked 3735 est tokens 2252054
wrote 28813 ids -> /workspace/submission/selection.json est tokens 24768772
matches best config: True 28813

[stdout]
qa avail 4163 picked 3735 est tokens 2252054
wrote 28813 ids -> /workspace/submission/selection.json est tokens 24768772
matches best config: True 28813
[metadata] {"noOutputExpected": false}

#33

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolEdit

args

{
  "replace_all": false,
  "file_path": "/workspace/claim.md",
  "old_string": "| final, but loosened gate for the Q&A register | 365.35 |",
  "new_string": "| final, but loosened gate for the Q&A register | 365.35 |\n| final, but scarcity-first register ownership (instead of argmax) | 359.38 |"
}

Edit result

The file /workspace/claim.md has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/claim.md", "oldString": "| final, but loosened gate for the Q&A register | 365.35 |", "newString": "| final, but loosened gate for the Q&A register | 365.35 |\n| final, but scarcity-first register ownership (instead of argmax) | 359.38 |", "originalFile": "# Claim\n\n## Hypothesis\nUnder a token budget ~6% of the pool, held-out perplexity on a *multi-register*\ntarget is governed less by generic \"web quality\" than by **per-register\ndistributional match**. Selecting documents by a length-normalised\nlog-likelihood ratio against each of the four target registers separately, and\nthen emitting them so that every prefix of the priority list is an equal-token\nmixture of the four, beats both a random selection and a single global\nquality score.\n\nConcretely: score_d(doc) = (1/N) Σ_t log( λ·p_d(t)/p_0(t) + (1−λ) ), where p_d is\nthe unigram+bigram distribution of target register d (wiki / general web prose /\nnews / technical Q&A), p_0 is the raw-pool background, λ = 0.7. Each document is\nowned by its argmax register; each register greedily takes its top-scored,\nde-duplicated documents; output interleaves registers by accumulated tokens.\n\n## Mechanism (observable that is *not* the final perplexity)\nThe mechanism is that the LLR ranks documents by *expected cross-entropy\nreduction on the target*, so the selected set should differ from the pool in\nmeasurable surface statistics before any training happens:\n\n1. **The interpolation is what makes the score a match score, not a novelty\n   score.** With add-1 smoothing instead of Jelinek–Mercer, the same pipeline's\n   top-ranked documents are observably degenerate — non-English pages, nav-bar\n   link lists, brand indexes (verified by inspecting the top-12 documents). The\n   bounded form log(λ·p_d/p_0 + 1−λ) is what puts encyclopedic prose, wire-service\n   news and StackExchange threads at the top. Prediction: any scorer whose\n   per-token weight is unbounded below will preferentially select\n   vocabulary-novel boilerplate, regardless of downstream training.\n2. **Supply is register-asymmetric.** Prediction (confirmed on this pool): after\n   gating, \"general web prose\" is ~30× more abundant than \"technical Q&A\"\n   (68M vs 2M tokens). So an unconditional target-similarity filter must collapse\n   the mixture toward web prose; only per-register conditioning holds the\n   equal-parts mixture, which is why balancing helps.\n3. Duplicate structure: a plain top-k selection of this pool draws visible\n   near-duplicate page families (same body, different boilerplate); the head/tail\n   word-sequence dedup removes them and frees budget for unique text.\n\n## Falsification\nThe hypothesis is falsified if, holding everything else fixed:\n- a single global target classifier (positives = all four registers pooled)\n  matches or beats the per-register version; or\n- deliberately *unbalancing* the mixture improves dev perplexity.\n\nBoth were tested. Measured dev perplexity (frozen `train_nano.py`, seed 1337,\n12M tokens, `multi_dev.npy`):\n\n| selection | dev PPL |\n|---|---|\n| random baseline (do-nothing) | 457.99 |\n| add-1-smoothed LLR, doc-balanced (early version) | 463.51 |\n| Jelinek–Mercer LLR, per-register, doc round-robin | 356.26 |\n| **+ token-balanced interleave (final, with bigrams)** | **350.27** |\n| final, but mixture skewed to [.30 .30 .25 .15] | 360.22 |\n| final, but loosened gate for the Q&A register | 365.35 |\n\nSo the \"vocabulary-novelty\" variant is *worse than random* (falsifying the naive\nversion of the idea), and both perturbations of the balanced mixture are worse\nthan equal parts — consistent with the hypothesis. The remaining risk is dev\noverfitting: positives come from `multi_dev.npy`, and the official target is a\ndisjoint sample of the same domain. Because the scorer uses only unigram/bigram\n*register* statistics (not memorised content), the transfer loss should be small;\nif the hidden-target perplexity is far above 350 while dev is 350, that is the\nfalsifying observation.\n\n## Transfer\nThe recipe is domain-agnostic and needs only (a) a sample of the target\ndistribution and (b) knowledge of its register composition: estimate a background\ndistribution from the raw pool, build one interpolated LLR scorer per target\nregister, gate obvious boilerplate, de-duplicate, and emit prefix-balanced by\ntokens. It costs one CPU pass over the pool (~40 s for 182k documents here, no\nGPU, no neural scorer) and scales to larger pools linearly. It should transfer to\nany budgeted pretraining-data curation where the evaluation mixture is known, and\nthe prefix-balancing property means the same list stays valid if the token budget\nis later raised or lowered.\n", "structuredPatch": [{"oldStart": 54, "oldLines": 6, "newStart": 54, "newLines": 7, "lines": [" | **+ token-balanced interleave (final, with bigrams)** | **350.27** |", " | final, but mixture skewed to [.30 .30 .25 .15] | 360.22 |", " | final, but loosened gate for the Q&A register | 365.35 |", "+| final, but scarcity-first register ownership (instead of argmax) | 359.38 |", " ", " So the \"vocabulary-novelty\" variant is *worse than random* (falsifying the naive", " version of the idea), and both perturbations of the balanced mixture are worse"]}], "userModified": false, "replaceAll": false}

final summary

Done. All three deliverables are in place and the final selection is the best-measured configuration.

Result: dev perplexity 350.27 vs 457.99 for the random baseline (−24%).

Method (submission/curate.py, one 40s CPU pass, no GPU):

Decode multi_dev.npy, split into its four equal register blocks (wiki / general web prose / news / technical Q&A) → four positive unigram+bigram distributions.
Score every pool doc by length-normalized interpolated log-likelihood ratio mean_t log(λ·p_d(t)/p_0(t) + 1−λ), λ=0.7, against a pool background p_0.
Gate boilerplate (length, non-ASCII/markup fraction, type-token repetition), dedupe near-duplicate page families on head/tail word sequences.
Assign each doc to its argmax register, greedily fill each register, emit token-balanced interleaved so every prefix of the list is an equal-parts mixture.

Ablations run (each a full frozen training run, recorded in claim.md): the add-1-smoothed variant of the same scorer scores 463 — worse than random, because unbounded per-token weights select vocabulary-novel nav-bar/link-farm pages; skewing the mixture (360), loosening the Q&A gate (365), and scarcity-first register ownership (359) all lost to the equal-parts argmax version.

Validity checks: 28,813 unique ids in range, packs to the full 12,000,000 tokens.