claude-opus-5

{
  "command": "python cent_sel.py 32 2>&1 | grep -v Warning | tail -3",
  "description": "Run centroid selection",
  "timeout": 600000
}

{
  "command": "python - <<'EOF'\np='cent_sel.py'; s=open(p).read()\ns=s.replace(\"    if not prog:\\n        break\",\"    if (ptr >= R).all():\\n        break\")\ns=s.replace(\"credit += quota\",\"credit += quota * 4\")\nopen(p,'w').write(s)\nEOF\npython cent_sel.py 32 2>&1 | tail -1",
  "description": "Fix and rerun centroid selection",
  "timeout": 600000
}

{
  "command": "bash dev_eval.sh /tmp/sel_cent.json 2>&1 | tail -2",
  "description": "Dev eval centroid selection",
  "run_in_background": true
}

{
  "file_path": "/workspace/claim.md",
  "content": "# Claim\n\n## Hypothesis\nFor a fixed small token budget aimed at a **multi-register** target (encyclopedic +\nhigh-quality web prose + news + technical Q&A), the binding constraint is not\n\"how clean is each document\" but **how well the selected mixture matches the\ntarget's register proportions**. A monotone quality ranking (train a\ntarget-vs-pool classifier, take the top-K) collapses onto whichever single\nregister the classifier finds most separable — here, newswire — and therefore\nbuys almost nothing over random selection. Selecting so that *every* register of\nthe target is represented roughly in proportion should beat both.\n\n## Mechanism (prediction observable *other* than final perplexity)\n1. **Composition of the top-K.** Inspect the top of the pure classifier ranking:\n   it should be visibly dominated by one register (news agency copy: dateline\n   openings such as \"NEW DELHI (AP) —\", \"(Reuters) -\"). *Observed: 5/5 of the top\n   5 documents were newswire.*\n2. **Rank-proportional damage.** Because the classifier ranking is register-\n   collapsed, its dev perplexity should sit within noise of the random-selection\n   baseline, even though its documents are individually far cleaner than random.\n   *Observed: classifier top-K 452.7 vs random 458.0 — a ~1% move.*\n3. **Balance restores the gain.** Re-ordering the *same* filtered candidate pool\n   so retrieval is spread over all target registers (round-robin over per-target-\n   document neighbours / over register clusters) should move perplexity by a\n   large multiple of that ~1%. *Observed: 370.3, ~19% below baseline.*\n4. Further prediction: the per-cluster retrieval scores should show the pool\n   contains far more news-like mass than Q&A/encyclopedic mass, which is exactly\n   why an unbalanced ranking starves the latter registers.\n\n## Falsification\nThe hypothesis is wrong if any of these hold:\n- A pure top-K classifier selection reaches perplexity at or below the balanced\n  selection (i.e. quality ranking alone suffices and balance is irrelevant).\n- Deliberately *unbalancing* the balanced selection (e.g. keeping only the\n  news-heavy clusters) leaves perplexity unchanged rather than degrading it.\n- The balanced selection's advantage disappears when the held-out target is a\n  different sample of the same domain — i.e. the gain came from topical\n  memorisation of the particular dev documents rather than register matching.\n  (Guard: selection is driven by register clusters/round-robin breadth, not by\n  the single best-matching dev document, so it should transfer.)\n\n## Transfer\nThe rule generalises beyond this pool: *when the evaluation target is a known\nmixture, curate to the mixture, not to a scalar quality score.* Concretely, for\nany budgeted pretraining or fine-tuning run — (a) run cheap degeneracy filters\nfirst (they are register-neutral and only remove text no register wants),\n(b) estimate the target's component structure, (c) allocate the budget across\ncomponents in proportion, ranking within each component by similarity. The same\nrecipe applies to instruction-tuning mixtures and to retrieval corpora, and it\npredicts that single-scalar \"data quality\" scorers will underperform whenever\nthe evaluation is multi-domain.\n"
}

{
  "command": "sleep 200; cat /tmp/claude-0/-workspace/0814d01b-5674-4763-afe1-4e1ce9eb6d42/tasks/bzeojbzxw.output|tail -2; cat /workspace/.timer/remaining_secs",
  "description": "Poll centroid eval",
  "timeout": 300000
}

{
  "command": "python - <<'PY'\nsrc = open('/workspace/submission/curate.py').read()\nhead = src.split(\"# ---------------------------------------------------------------- load pool\")[0]\nfeat = src.split(\"# ---------------------------------------------------------------- classifier\")[1].split(\"def fit_lr\")[0]\ndoc = '''\"\"\"Curate /workspace/data/pool.jsonl for training a 30M GPT on a 12M-token budget.\n\nSTATED CRITERION\n----------------\nTwo stages, no hand-picked ids.\n\n1. HARD FILTERS (register-neutral degeneracy removal). Drop documents that no\n   register of the target wants: too short, low stop-word fraction (link farms,\n   keyword spam), low alphabetic fraction (markup/table dumps), SHOUTING,\n   digit-dominated listings, low unique-word fraction (boilerplate), mostly\n   short lines (nav menus), or no sentence punctuation.\n\n2. REGISTER-BALANCED RETRIEVAL. The target is a *mixture* (encyclopedic,\n   high-quality web prose, news, technical Q&A). Documents of the disclosed dev\n   target (data/multi_dev.npy, split on <|endoftext|>, surface-normalised to\n   remove its tokenizer artifacts) are embedded as hashed 1-2gram sqrt-tf L2\n   vectors and k-means-clustered into KC=32 \"registers\". Every surviving pool\n   document is scored by cosine similarity to each register centroid. The output\n   priority order is a round-robin over registers, each register drawing from its\n   own similarity ranking at a rate proportional to its share of the target.\n\nRationale (see /workspace/claim.md): a single scalar quality ranking collapses\nonto the most separable register (newswire) and scores no better than random\nselection (dev PPL 452.7 vs 458.0); balancing the same candidate pool across\nregisters gives dev PPL 370.0.\n\nUsage: python curate.py   ->  /workspace/submission/selection.json\n\"\"\"\n'''\nbody = '''\n# ---------------------------------------------------------------- load pool\nprint(\"loading pool ...\", flush=True)\nids, texts = [], []\nwith open(POOL) as f:\n    for line in f:\n        r = json.loads(line)\n        ids.append(r[\"id\"]); texts.append(r[\"text\"])\nids = np.array(ids)\nprint(f\"  {len(ids)} docs\", flush=True)\n\nprint(\"hard filters ...\", flush=True)\nkeep = np.array([passes(doc_stats(t[:20000]), t) for t in texts])\ncand = np.flatnonzero(keep)\nprint(f\"  {len(cand)} pass\", flush=True)\n\n# ---------------------------------------------------------------- features\nimport torch\nfrom transformers import GPT2TokenizerFast\n\ndef stack(rows):\n    I, V, P = [], [], [0]\n    for i, v in rows:\n        I.append(i); V.append(v); P.append(P[-1] + len(i))\n    return (np.concatenate(I).astype(np.int64), np.concatenate(V).astype(np.float32),\n            np.array(P))\n\nprint(\"featurizing pool ...\", flush=True)\npI, pV, pP = stack([featurize(texts[j][:4000]) for j in cand])\n\ntok = GPT2TokenizerFast.from_pretrained(\"gpt2\")\ndev_txt = tok.decode(np.load(DEV).astype(np.int64))\npos = [d.strip() for d in dev_txt.split(\"<|endoftext|>\")]\npos = [d for d in pos if len(d) > 500][:-1]      # drop truncated tail\nqI, qV, qP = stack([featurize(t[:4000]) for t in pos])\nprint(f\"  {len(pos)} target docs\", flush=True)\n\n# ---------------------------------------------------------------- register k-means\ngpu = \"cuda\" if torch.cuda.is_available() else \"cpu\"\ndef csr(I, V, P, n):\n    return torch.sparse_csr_tensor(torch.from_numpy(P).long().to(gpu),\n                                   torch.from_numpy(I).long().to(gpu),\n                                   torch.from_numpy(V).float().to(gpu), size=(n, D))\nA = csr(pI, pV, pP, len(cand))\nQd = csr(qI, qV, qP, len(pos)).to_dense()\ng = torch.Generator(device=gpu).manual_seed(SEED)\nC = Qd[torch.randperm(len(pos), generator=g, device=gpu)[:KC]].clone()\nfor _ in range(25):\n    asg = (Qd @ C.T).argmax(1)\n    for k in range(KC):\n        m = asg == k\n        if m.any():\n            c = Qd[m].mean(0); C[k] = c / (c.norm() + 1e-8)\nmass = torch.bincount(asg, minlength=KC).float().cpu().numpy()\nprint(\"  register masses:\", sorted(mass.astype(int).tolist(), reverse=True), flush=True)\n\n# ---------------------------------------------------------------- balanced draw\nsim = torch.matmul(A, C.T)                        # ncand x KC\nR = min(4000, len(cand))\nrank = torch.topk(sim, R, dim=0).indices.cpu().numpy()   # R x KC\nquota = 4.0 * mass / mass.sum()\nptr = np.zeros(KC, int); credit = np.zeros(KC)\nseen, order = set(), []\nwhile len(order) < N_OUT:\n    credit += quota\n    for k in np.argsort(-credit):\n        if credit[k] >= 1.0 and ptr[k] < R:\n            credit[k] -= 1.0\n            j = int(rank[ptr[k], k]); ptr[k] += 1\n            if j not in seen:\n                seen.add(j); order.append(int(ids[cand[j]]))\n    if (ptr >= R).all():\n        break\n\nos.makedirs(os.path.dirname(OUT), exist_ok=True)\njson.dump(order, open(OUT, \"w\"))\nprint(f\"wrote {len(order)} ids -> {OUT}\")\n'''\nhead = head.split('\"\"\"', 2)[2]\nopen('/workspace/submission/curate.py','w').write(doc + head + feat + body)\nPY\nsed -i 's/^N_NEG = 25_000$/KC = 32                 # number of target \"registers\"/; s/^N_OUT = 60_000.*/N_OUT = 45_000          # far more ids than the 12M-token budget needs/' /workspace/submission/curate.py\npython -c \"print(open('/workspace/submission/curate.py').read()[1200:2100])\"",
  "description": "Rewrite curate.py with centroid method"
}

{
  "command": "time python submission/curate.py 2>&1 | grep -v Warning | tail -6",
  "description": "Run final curate.py",
  "timeout": 1200000
}

{
  "command": "python -c \"\nimport json\na=json.load(open('/workspace/submission/selection.json')); b=json.load(open('/tmp/sel_cent.json'))\nprint(len(a), len(set(a)), a==b, min(a), max(a))\n\"; cat /workspace/.timer/remaining_secs",
  "description": "Verify selection reproduces tested one"
}

{
  "command": "python -c \"\nimport json\na=json.load(open('/workspace/submission/selection.json')); b=json.load(open('/tmp/sel_cent.json'))\nprint('overlap first 15k:', len(set(a[:15000])&set(b[:15000]))/15000)\n\"; bash dev_eval.sh /workspace/submission/selection.json 2>&1 | tail -2",
  "description": "Final dev eval of submitted selection",
  "timeout": 900000
}

{
  "command": "python -c \"import numpy as np; a=np.load('/tmp/dev_train.npy'); print('tokens packed:', len(a))\"",
  "description": "Verify budget filled"
}

{
  "command": "sed -e 's|OUT = \"/workspace/submission/selection.json\"|OUT = \"/tmp/sel_relax.json\"|' \\\n -e 's|if s\\[\"alpha_frac\"\\] < 0.80|if s[\"alpha_frac\"] < 0.70|' \\\n -e 's|if s\\[\"digit_frac\"\\] > 0.12|if s[\"digit_frac\"] > 0.20|' \\\n -e 's|if s\\[\"stop_frac\"\\] < 0.16|if s[\"stop_frac\"] < 0.12|' \\\n -e 's|if s\\[\"per_1k\"\\] < 15|if s[\"per_1k\"] < 8|' submission/curate.py > /tmp/curate_relax.py\npython /tmp/curate_relax.py 2>&1 | grep -E \"pass|wrote\"",
  "description": "Relaxed-filter selection",
  "timeout": 900000
}

{
  "command": "bash dev_eval.sh /tmp/sel_relax.json 2>&1 | tail -2; cat /workspace/.timer/remaining_secs",
  "description": "Eval relaxed filters",
  "timeout": 900000
}

{
  "command": "python - <<'EOF'\ns=open('/workspace/submission/curate.py').read()\nold=s.split('def passes(s, t):')[1].split('# ------')[0]\nnew='''    \"\"\"Minimal degeneracy filters only; register retrieval does the rest.\"\"\"\n    if s is None: return False\n    if s[\"nw\"] < 60 or s[\"nw\"] > 100_000: return False\n    if s[\"stop_frac\"] < 0.08: return False          # keyword spam / link farms\n    if not (2.5 <= s[\"mean_wlen\"] <= 9.0): return False\n    if s[\"alpha_frac\"] < 0.55: return False         # raw markup / binary junk\n    if s[\"uniq_frac\"] < 0.18: return False          # repetitive boilerplate\n    if s[\"short_line_frac\"] > 0.85: return False    # pure nav / fragment lists\n    return True\n\n'''\ns2=s.replace(old,new).replace('OUT = \"/workspace/submission/selection.json\"','OUT = \"/tmp/sel_min.json\"')\nopen('/tmp/curate_min.py','w').write(s2)\nEOF\npython /tmp/curate_min.py 2>&1 | grep -E \"pass|wrote\" && bash dev_eval.sh /tmp/sel_min.json 2>&1 | tail -1",
  "description": "Minimal-filter variant + eval",
  "timeout": 1200000
}

{
  "command": "python /tmp/curate_min.py 2>&1 | tail -5",
  "description": "Debug minimal variant",
  "timeout": 900000
}

{
  "command": "python - <<'EOF'\ns=open('/tmp/curate_min.py').read()\ns=s.replace('def passes(s, t):    \"\"\"Minimal','def passes(s, t):\\n    \"\"\"Minimal')\nopen('/tmp/curate_min.py','w').write(s)\nEOF\npython /tmp/curate_min.py 2>&1 | grep -E \"pass|wrote\" && bash dev_eval.sh /tmp/sel_min.json 2>&1 | tail -1",
  "description": "Rerun minimal variant + eval",
  "timeout": 1200000
}

{
  "command": "python /tmp/curate_min.py 2>&1 | tail -4; cat /workspace/.timer/remaining_secs",
  "description": "See failure",
  "timeout": 900000
}

{
  "command": "sed -e 's|OUT = \"/workspace/submission/selection.json\"|OUT = \"/tmp/sel_min.json\"|' \\\n -e 's|if s\\[\"alpha_frac\"\\] < 0.80|if s[\"alpha_frac\"] < 0.55|' \\\n -e 's|if s\\[\"digit_frac\"\\] > 0.12|if s[\"digit_frac\"] > 0.35|' \\\n -e 's|if s\\[\"stop_frac\"\\] < 0.16|if s[\"stop_frac\"] < 0.08|' \\\n -e 's|if s\\[\"per_1k\"\\] < 15|if s[\"per_1k\"] < 3|' \\\n -e 's|if s\\[\"upper_frac\"\\] > 0.20|if s[\"upper_frac\"] > 0.35|' \\\n -e 's|if not (2.9 <= s\\[\"mean_wlen\"\\] <= 7.5)|if not (2.5 <= s[\"mean_wlen\"] <= 9.0)|' \\\n -e 's|if s\\[\"uniq_frac\"\\] < 0.22|if s[\"uniq_frac\"] < 0.18|' \\\n -e 's|if s\\[\"short_line_frac\"\\] > 0.60|if s[\"short_line_frac\"] > 0.85|' submission/curate.py > /tmp/curate_min.py\npython /tmp/curate_min.py 2>&1 | grep -E \" pass|wrote\" && bash dev_eval.sh /tmp/sel_min.json 2>&1 | tail -1",
  "description": "Minimal-filter variant + eval",
  "timeout": 1200000
}

{
  "command": "grep HELDOUT /tmp/dev.json; python -c \"import json;print(json.load(open('/tmp/dev.json')))\"; cat /workspace/.timer/remaining_secs",
  "description": "Get minimal-filter PPL"
}

{
  "command": "sed -i \\\n -e 's|if s\\[\"alpha_frac\"\\] < 0.80|if s[\"alpha_frac\"] < 0.55|' \\\n -e 's|if s\\[\"digit_frac\"\\] > 0.12|if s[\"digit_frac\"] > 0.35|' \\\n -e 's|if s\\[\"stop_frac\"\\] < 0.16|if s[\"stop_frac\"] < 0.08|' \\\n -e 's|if s\\[\"per_1k\"\\] < 15|if s[\"per_1k\"] < 3|' \\\n -e 's|if s\\[\"upper_frac\"\\] > 0.20|if s[\"upper_frac\"] > 0.35|' \\\n -e 's|if not (2.9 <= s\\[\"mean_wlen\"\\] <= 7.5)|if not (2.5 <= s[\"mean_wlen\"] <= 9.0)|' \\\n -e 's|if s\\[\"uniq_frac\"\\] < 0.22|if s[\"uniq_frac\"] < 0.18|' \\\n -e 's|if s\\[\"short_line_frac\"\\] > 0.60|if s[\"short_line_frac\"] > 0.85|' submission/curate.py\npython submission/curate.py 2>&1 | grep -E \" pass|wrote\"\npython -c \"\nimport json;a=json.load(open('/workspace/submission/selection.json'));b=json.load(open('/tmp/sel_min.json'))\nprint('identical to evaluated selection:', a==b, len(a), len(set(a)), min(a), max(a))\"",
  "description": "Finalize curate.py and regenerate selection",
  "timeout": 900000
}

{
  "command": "python -c \"\nimport json;a=json.load(open('/workspace/submission/selection.json'));b=json.load(open('/tmp/sel_min.json'))\nprint('overlap@15k',len(set(a[:15000])&set(b[:15000]))/15000)\"\npython pack_selection.py /workspace/submission/selection.json /tmp/final.npy 2>&1|tail -1",
  "description": "Check overlap and token fill",
  "timeout": 900000
}