claude-opus-5

Official dev_eval.sh on the submitted file: 267.721 — matching my cached-pipeline measurement exactly. Everything is validated end-to-end.

Dev perplexity 267.7 vs 485.5 random baseline — a 44.9% reduction (verified by the official dev_eval.sh with none of my own tooling in the path).

What I found

Decoding multi_dev.npy revealed the target is four exact quarters: WikiText-formatted encyclopedic prose, clean web prose, newswire, and StackExchange HTML Q&A. Two facts drove everything:

• The pool contains zero <pre><code> documents and zero @-@ artifacts — it cannot match two of the four registers on surface form.
• A per-register loss breakdown (via a diagnostic copy of the trainer) showed the encyclopedic quarter is by far the hardest (ppl 709 vs 154 for Q&A). This reversed my initial guess: I had upweighted Q&A, which was already the easiest register.

The criterion

Rather than rank documents by "quality", I minimize the same functional the metric evaluates — greedily selecting documents to reduce unigram cross-entropy H(p_target, p_selection), with clipped importance ratios, a junk gate, and MinHash near-dup filtering. Progression: random 485.5 → global quality ranking 387.0 → per-register balancing 312.5 → distribution matching 267.7.

Evidence the mechanism is real, not just the number

• Quality-gate + dedup alone reaches only 403.6 — the distribution matching contributes ~1.7× more than the filtering, and the two are cleanly separable since both pass an identical gate.
• The proxy predicted two runs before I trained them. I read off the proxy for a stronger prior and a tighter gate, predicted both would lose, and they did monotonically (7.854→7.868→7.888 nats → 267.7→275.9→281.8 ppl).
• Five hypotheses I expected to win instead lost: forcing artifact-carrying documents (290.5), upweighting the hardest register (288.8), demanding longer documents (295.4), finer greedy steps (273.4), looser gate (269.2).
• Robustness to the hidden target: two disjoint 500k-token halves of the dev target yield selections sharing only 53% of documents yet landing within 0.14 ppl of each other — the criterion extracts stable structure, not particular documents.

I also found and reported an honest failure of my own proxy: the loose-gate run had the lowest proxy of all runs but a worse perplexity, so the proxy is only valid within a fixed quality gate. That's documented in claim.md rather than omitted.

Two caveats worth your attention: all runs share seed 1337, so I have not measured seed variance, and five scalar hyperparameters were tuned against dev — bounded by the optimum's flatness (267.7 across min-length 100–150) but not eliminated. Differences below ~1.5 ppl in my tables should be treated as noise.