claude-opus-5

{
  "command": "python -c \"\nimport sys; sys.argv=['x','roundrobin','/tmp/sel_ddcap.json']\nimport importlib.util as iu\nspec=iu.spec_from_file_location('c','/workspace/submission/curate.py'); m=iu.module_from_spec(spec); spec.loader.exec_module(m)\nm.MAX_TOK=4000; m.main()\" 2>&1|tail -1\nTOKENIZERS_PARALLELISM=false python /tmp/pack2.py dd ddcap 2>&1|grep -v Token\nfor n in dd ddcap; do python train_nano.py --train_npy /tmp/tr_$n.npy --heldout_npy data/multi_dev.npy --out_json /tmp/x.json 2>&1|grep HELDOUT; done; cat /workspace/.timer/remaining_secs",
  "timeout": 600000
}

{
  "command": "for c in 1800 2800 6000; do python -c \"\nimport sys; sys.argv=['x','roundrobin','/tmp/sel_c$c.json']\nimport importlib.util as iu\nspec=iu.spec_from_file_location('c','/workspace/submission/curate.py'); m=iu.module_from_spec(spec); spec.loader.exec_module(m)\nm.MAX_TOK=$c; m.main()\" 2>&1|tail -1; done\nTOKENIZERS_PARALLELISM=false python /tmp/pack2.py c1800 c2800 c6000 2>&1|grep -v Token\nfor n in c1800 c2800 c6000; do python train_nano.py --train_npy /tmp/tr_$n.npy --heldout_npy data/multi_dev.npy --out_json /tmp/x.json 2>&1|grep HELDOUT; done; cat /workspace/.timer/remaining_secs",
  "timeout": 900000
}

{
  "command": "sed -i 's/^MAX_TOK = 0  /MAX_TOK = 4000/' submission/curate.py && grep -n \"^MAX_TOK\" submission/curate.py && rm -f /tmp/curate_scores.npz && python submission/curate.py 2>&1 | tail -3",
  "timeout": 600000
}

{
  "command": "python -c \"\nimport json\ns=json.load(open('/workspace/submission/selection.json')); r=json.load(open('/tmp/sel_ddcap.json'))\nprint(len(s), len(set(s))==len(s), min(s), max(s), s==r)\""
}

{
  "file_path": "/workspace/claim.md",
  "content": "# Claim: register-balanced cross-entropy-difference selection\n\n## Hypothesis\nFor a fixed 12M-token budget on a *broad* multi-domain target, the binding\nconstraint is not \"how clean is each document\" but **how the budget is split\nacross the target's registers**. Concretely:\n\n1. Documents whose token distribution is closer to the target than to the raw\n   pool (positive Moore-Lewis cross-entropy difference) train a better model\n   than random pool documents.\n2. Scoring against a *single* pooled target model is worse than scoring against\n   *per-register* models and spending the budget on each register in proportion\n   to its share of the target — because a single global ranking is dominated by\n   the majority register and starves the others.\n3. At a fixed token budget, spending on many medium-length documents beats\n   spending on a few very long ones (a 30k-token document consumes 0.25% of the\n   budget on one topic/one writing style).\n\n## Mechanism (observable other than final perplexity)\nThe mechanism is *register coverage of the budget*, not per-document polish.\nPredictions that are observable without looking at the final score:\n\n- **Ranking collapse.** If documents are assigned to their argmax register\n  score without per-register normalisation, >99% of them land in the majority\n  (\"prose\") register — the wiki and technical-QA rankings are never consumed.\n  Observed: 136,158 / 136,682 kept docs assigned to `prose`, 130 to `wiki`,\n  394 to `qa`. This is the concrete failure mode round-robin fixes.\n- **Domain-token composition.** Round-robin selection at 25/25/50 yields a\n  training set whose token mix is 25% wiki-like / 25% technical-QA-like / 50%\n  news+web-prose, versus ~97% prose for a single global ranking.\n- **Documents per budget.** Under the same 12M tokens, the length cap raises\n  the number of distinct documents trained on from ~10k to ~34k (selection list\n  grew from 22,352 to 34,279 ids for the same estimated token mass), i.e. the\n  budget covers ~3x more distinct sources/topics.\n\n## Falsification\nThe claim is falsified if, at equal budget and identical frozen recipe:\n\n- a single global (mixture) ranking matches or beats the register-balanced\n  round-robin, or\n- shifting the register shares far from the target's true shares does not hurt.\n\nMeasured on the dev target (frozen script, seed 1337, dev PPL, lower better):\n\n| selection | dev PPL |\n|---|---|\n| random pool sample (baseline) | 457.99 |\n| global mixture ranking (`mix`) | 380.31 |\n| round-robin 25/25/50 | 316.30 |\n| round-robin 34/33/33 | 321.28 |\n| round-robin 20/20/60 | 318.29 |\n| round-robin + length cap 4k | 313.74 |\n| round-robin + near-dup removal | 315.83 |\n| **round-robin + dedup + cap 4k (submitted)** | **306.53** |\n| dedup + cap 1.8k / 2.8k / 6k | 325.26 / 317.01 / 310.19 |\n\nThe global-ranking prediction survives (380 vs 316, a 64-point gap). The\nshare-sensitivity prediction is only *weakly* supported: 25/25/50, 34/33/33 and\n20/20/60 differ by ~5 PPL, near run-to-run noise — so the strong form (\"shares\nmust match the target\") is **not** confirmed; what matters is that every\nregister gets a non-trivial share at all. The length-cap prediction survives\nwith an interior optimum near 4k tokens/document (both 1.8k and 6k are worse),\nwhich is the trade-off the hypothesis predicts: too small a cap discards\nlong-form coherent prose, too large a cap concentrates the budget.\n\n## Transfer\n- The method needs only (a) a sample of target text and (b) the raw pool; no\n  labels, no GPU, no external classifier. Fitting the n-gram models and scoring\n  182k documents takes ~2 minutes on CPU, so it scales to much larger pools and\n  re-runs cheaply when the target changes.\n- Register detection is the only target-specific part (here: HTML markup ->\n  technical Q&A, wikitext spacing artefacts -> encyclopedic, else prose). For a\n  new target one would substitute a clustering of the target sample; the rest\n  (CE-difference scoring, share-proportional round-robin, dedup, length cap) is\n  domain-agnostic.\n- Expected to transfer to any *budget-limited* pretraining setting with a\n  multi-domain evaluation. It should transfer *less* well when the budget is\n  large relative to the pool (then coverage, not selection, dominates) or when\n  the target is single-domain (then the global ranking is already correct and\n  round-robin adds nothing).\n- Caveat: the register models are fit on the disclosed dev sample, so some\n  adaptation to that specific sample is possible; the hidden target is a\n  disjoint sample of the same domains, and the selection uses only coarse\n  unigram/bigram statistics, which should limit sample-specific overfitting.\n"
}

{
  "command": "cat > /tmp/round2.py <<'EOF'\nimport json, sys, numpy as np, importlib.util as iu\nspec = iu.spec_from_file_location(\"c\", \"/workspace/submission/curate.py\")\nC = iu.module_from_spec(spec); spec.loader.exec_module(C)\n\nz = np.load(C.CACHE, allow_pickle=True)\nS, keep, est, ids, names, sig = z[\"S\"], z[\"keep\"], z[\"est\"], z[\"ids\"], list(z[\"names\"]), z[\"sig\"]\ntexts = {}\norder = []\nfor line in open(C.POOL):\n    r = json.loads(line); texts[r[\"id\"]] = r[\"text\"]; order.append(r[\"id\"])\npos_of = {d: i for i, d in enumerate(order)}\n\nkept = np.where(keep)[0]; kept = kept[est[kept] <= 4000]\n# round-2 seeds: top pool docs per register from round 1\nregs = {}\nSEED = 3000\nfor k, r in enumerate(names):\n    top = kept[np.argsort(-S[kept, k])][:SEED]\n    m = C.NGram()\n    for i in top:\n        m.add(C.hash_ids(C.toks(texts[int(ids[i])][:C.SCORE_CHARS])).astype(np.int64))\n    m.finish(); regs[r] = m\n    print(\"refit\", r, flush=True)\nbg = C.NGram()\nstep = max(1, len(order) // C.BG_DOCS)\nfor d in order[::step]:\n    bg.add(C.hash_ids(C.toks(texts[d][:C.SCORE_CHARS])).astype(np.int64))\nbg.finish()\n\nS2 = np.full_like(S, -1e9)\nfor n, i in enumerate(kept):\n    u, b = C.prep(C.toks(texts[int(ids[i])][:C.SCORE_CHARS]))\n    base = bg.mean_logp(u, b)\n    for k, r in enumerate(names):\n        S2[i, k] = regs[r].mean_logp(u, b) - base\n    if n % 40000 == 0: print(n, flush=True)\nBLEND = float(sys.argv[1]) if len(sys.argv) > 1 else 0.5\nSb = (1 - BLEND) * S + BLEND * S2\nC.MAX_TOK = 4000\nC.select(Sb, keep, est, ids, names, sig, mode=\"roundrobin\", out=\"/tmp/sel_r2.json\")\nEOF\nnohup python /tmp/round2.py 0.5 > /tmp/r2.log 2>&1 & sleep 3; echo started; cat /workspace/.timer/remaining_secs"
}

{
  "command": "sleep 170; tail -2 /tmp/r2.log"
}

{
  "command": "sleep 90; tail -2 /tmp/r2.log; cat /workspace/.timer/remaining_secs"
}

{
  "command": "TOKENIZERS_PARALLELISM=false python /tmp/pack2.py r2 2>&1|grep -v Token\npython train_nano.py --train_npy /tmp/tr_r2.npy --heldout_npy data/multi_dev.npy --out_json /tmp/x.json 2>&1|grep HELDOUT; cat /workspace/.timer/remaining_secs",
  "timeout": 600000
}

{
  "command": "python - <<'EOF'\np='/workspace/submission/curate.py'; s=open(p).read()\ns=s.replace('''    if MAX_TOK:                 # cap per-doc size so the budget spreads wider\n        kept = kept[est[kept] <= MAX_TOK]''','''    if MAX_TOK:                 # cap per-doc size so the budget spreads wider\n        kept = kept[est[kept] <= MAX_TOK]\n    if MIN_TOK:                 # floor out snippets too short to be coherent\n        kept = kept[est[kept] >= MIN_TOK]''')\ns=s.replace('MAX_TOK = 4000','MIN_TOK = 0                # 0 = no floor; else drop docs shorter than this\\nMAX_TOK = 4000')\nopen(p,'w').write(s)\nEOF\nfor f in 400 800; do python -c \"\nimport sys; sys.argv=['x','roundrobin','/tmp/sel_f$f.json']\nimport importlib.util as iu\nspec=iu.spec_from_file_location('c','/workspace/submission/curate.py'); m=iu.module_from_spec(spec); spec.loader.exec_module(m)\nm.MIN_TOK=$f; m.main()\" 2>&1|tail -1; done\nTOKENIZERS_PARALLELISM=false python /tmp/pack2.py f400 f800 2>&1|grep -v Token\nfor n in f400 f800; do python train_nano.py --train_npy /tmp/tr_$n.npy --heldout_npy data/multi_dev.npy --out_json /tmp/x.json 2>&1|grep HELDOUT; done; cat /workspace/.timer/remaining_secs",
  "timeout": 900000
}

{
  "command": "bash dev_eval.sh 2>&1 | tail -3; cat /workspace/.timer/remaining_secs",
  "timeout": 900000
}

{
  "command": "cat > /tmp/nohyg.py <<'EOF'\nimport importlib.util as iu, sys\nspec = iu.spec_from_file_location(\"c\", \"/workspace/submission/curate.py\")\nC = iu.module_from_spec(spec); spec.loader.exec_module(C)\nC.hygiene = lambda t: len(t) >= 600          # length-only gate\nC.CACHE = \"/tmp/nohyg.npz\"\nS, keep, est, ids, names, sig = C.score_pool()\nC.select(S, keep, est, ids, names, sig, mode=\"roundrobin\", out=\"/tmp/sel_nohyg.json\")\nEOF\nnohup python /tmp/nohyg.py > /tmp/nh.log 2>&1 & sleep 115; tail -2 /tmp/nh.log"
}

{
  "command": "TOKENIZERS_PARALLELISM=false python /tmp/pack2.py nohyg 2>&1|grep -v Token\npython train_nano.py --train_npy /tmp/tr_nohyg.npy --heldout_npy data/multi_dev.npy --out_json /tmp/x.json 2>&1|grep HELDOUT; cat /workspace/.timer/remaining_secs",
  "timeout": 600000
}

{
  "command": "cat > /tmp/strict.py <<'EOF'\nimport importlib.util as iu, re\nspec = iu.spec_from_file_location(\"c\", \"/workspace/submission/curate.py\")\nC = iu.module_from_spec(spec); spec.loader.exec_module(C)\n\ndef strict(t):\n    n = len(t)\n    if n < 600 or n > 400_000: return False\n    w = C.toks(t[:20000])\n    if len(w) < 100: return False\n    mwl = sum(map(len, w)) / len(w)\n    if mwl < 3.2 or mwl > 7.5: return False\n    if sum(x in C.STOP for x in w) / len(w) < 0.20: return False\n    head = t[:20000]\n    if sum(c.isalpha() or c.isspace() for c in head) / len(head) < 0.85: return False\n    lines = [l.strip() for l in head.split(\"\\n\") if l.strip()]\n    if lines and len(set(lines)) / len(lines) < 0.75: return False\n    g = [tuple(w[i:i+5]) for i in range(len(w)-4)]\n    if g and len(set(g)) / len(g) < 0.85: return False\n    # sentence-shaped: needs terminal punctuation density typical of prose\n    if head.count(\". \") / max(1, len(w)) < 0.02: return False\n    return True\n\nC.hygiene = strict\nC.CACHE = \"/tmp/strict.npz\"\nS, keep, est, ids, names, sig = C.score_pool()\nC.select(S, keep, est, ids, names, sig, mode=\"roundrobin\", out=\"/tmp/sel_strict.json\")\nEOF\npython /tmp/strict.py 2>&1 | grep -E \"kept|selected\"",
  "timeout": 600000
}

{
  "command": "TOKENIZERS_PARALLELISM=false python /tmp/pack2.py strict 2>&1|grep -v Token\npython train_nano.py --train_npy /tmp/tr_strict.npy --heldout_npy data/multi_dev.npy --out_json /tmp/x.json 2>&1|grep HELDOUT; cat /workspace/.timer/remaining_secs",
  "timeout": 600000
}

{
  "command": "python - <<'EOF'\np='/workspace/claim.md'; s=open(p).read()\ns=s.replace('''| dedup + cap 1.8k / 2.8k / 6k | 325.26 / 317.01 / 310.19 |''','''| dedup + cap 1.8k / 2.8k / 6k | 325.26 / 317.01 / 310.19 |\n| submitted config, but no hygiene filter (length gate only) | 333.53 |\n| submitted config, but much stricter hygiene (77.7k docs survive) | 364.04 |\n| + minimum doc length 400 / 800 tokens | 321.07 / 332.70 |\n| + round-2 refit of register models on round-1 picks (blend 0.5) | 314.22 |\n\nThe submitted number is an end-to-end verification: `bash dev_eval.sh` on the\nactual `submission/selection.json` packs 12,000,000 tokens from 34,279 ids and\nreports 306.527.''')\ns=s.replace('''## Transfer''','''Three predictions were **refuted**, and they sharpen the mechanism:\n\n- *\"Cleaner is better.\"* Tightening the hygiene thresholds (stopword ratio 0.20,\n  alpha 0.85, 5-gram uniqueness 0.85, sentence-punctuation density) cuts the\n  candidate set from 136.7k to 77.7k documents and makes the model **much\n  worse** (364 vs 307). Removing the filter entirely is also worse (334). The\n  filter's job is to delete degenerate text, not to select prestige text: the\n  mid-quality tail is where the technical and news registers live.\n- *\"Longer documents are more coherent, so impose a length floor.\"* Floors of\n  400 and 800 tokens both hurt (321, 333). Combined with the cap result, what\n  helps is the natural length mixture truncated from above, not shifted upward.\n- *\"Iterating the selector sharpens it.\"* Refitting the register models on the\n  round-1 picks (so the models describe pool-realizable text rather than the\n  dev sample's HTML/wikitext surface form) is worse (314 vs 307): the refit\n  models drift toward whatever round 1 already liked, narrowing coverage.\n\n## Transfer''')\nopen(p,'w').write(s)\nEOF\necho done; ls -la /workspace/submission /workspace/claim.md; cat /workspace/.timer/remaining_secs"
}

{
  "command": "rm -rf /workspace/submission/__pycache__ /tmp/curate_scores.npz\ncp /workspace/submission/selection.json /tmp/sel_final_ref.json\npython submission/curate.py 2>&1 | tail -2\npython -c \"\nimport json; a=json.load(open('/tmp/sel_final_ref.json')); b=json.load(open('/workspace/submission/selection.json'))\nprint('reproducible identical:', a==b, len(b), len(set(b))==len(b), min(b)>=0, max(b)<182016)\"\ncat /workspace/.timer/remaining_secs",
  "timeout": 600000
}