claude-opus-5

All three deliverables are in place and verified.

Result: dev perplexity 354.2 vs 469.9 random baseline (−25%).

Approach

I decoded multi_dev.npy with the GPT-2 tokenizer to recover the target text itself — it turned out to be WikiText-style encyclopedic text, web prose, news, and HTML-formatted StackExchange Q&A. That gave me a positive sample to fit against, with no labels needed.

The criterion in curate.py is two stages:

1. Hard quality filter (Gopher/C4-style): length, mean word length, English stop-word presence, alphabetic/ASCII fraction, duplicate-line and short-line fraction, punctuation density, URL density. 87,776 of 182,016 docs survive.
2. Moore–Lewis score: interpolated unigram+bigram LMs fit to the target and to the pool background; rank by H_pool(d) − H_target(d). I strip the WikiText detokenization artifacts ( @-@ , spaced punctuation) from the target first, so the score keys on register rather than a formatting quirk no pool doc could match. A second pass rescores leading candidates on their full text (worth 5 ppl — pass 1 only saw the first 3000 chars, letting clean-lede/junk-body docs through).

What the experiments showed

The ablations are the interesting part, and they're in claim.md:

Each stage alone is worse than doing nothing. They fail in opposite directions — the n-gram score has no notion of degeneracy, so unfiltered it selects link farms and SEO spam dense in target-like vocabulary; the filter is register-blind, so alone it returns a generic web sample. Target matching only pays once the degenerate tail is gone.

I also tested selectivity breadth (12M strict 359 < 18M band 373 < 36M band 426) — monotone, no diversity optimum, so the budget should come from the very top of the ranking. And explicit register-balancing hurt (369.8 vs 359.3), so I dropped it: the global contrastive score already prices register match, and quotas only admit lower-ranked docs.

Two caveats worth flagging. The 354.2 figure is on multi_dev, which my target LM was fit on — the hidden set is a disjoint sample, so expect some regression, though the signal is register-level n-grams rather than memorized specifics. And a mid-session script bug (an accidental O(n²) set rebuild) cost me one experiment cycle, so I didn't get to test whether a soft blend of the two signals could beat the hard two-stage pipeline; claim.md records that as an open prediction rather than a measured result.