claude-opus-5

5400
---
total 14
drwxr-xr-x 1 root root  100 Aug  3 23:25 .
drwxr-xr-x 1 root root  160 Aug  3 23:25 ..
drwxr-xr-x 1 root root   60 Aug  3 23:25 .live
drwxr-xr-x 1 root root  100 Aug  3 23:25 .metrics
drwxr-xr-x 1 root root  140 Aug  3 23:25 .timer
drwxr-xr-x 1 root root   57 Aug  3 22:37 data
-rwxrwxr-x 1 root root  714 Aug  3 22:34 dev_eval.sh
-rw-rw-r-- 1 root root 5706 Aug  3 22:34 model.py
-rw-rw-r-- 1 root root  869 Aug  3 22:34 pack_selection.py
drwxr-xr-x 1 root root   10 Aug  3 22:37 submission
-rwxrwxr-x 1 root root 1509 Aug  3 22:34 timer.sh
-rw-rw-r-- 1 root root 2962 Aug  3 22:34 train_nano.py
---
total 792657
drwxr-xr-x 1 root root        57 Aug  3 22:37 .
drwxr-xr-x 1 root root       100 Aug  3 23:25 ..
-r--r--r-- 1 root root   2000128 Aug  3 22:34 multi_dev.npy
-r--r--r-- 1 root root 809679384 Aug  3 22:35 pool.jsonl
---
Mon Aug  3 23:25:54 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H200                    On  |   00000000:CB:00.0 Off |                    0 |
| N/A   28C    P0             77W /  700W |       4MiB / 143771MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

[stdout]
5400
---
total 14
drwxr-xr-x 1 root root  100 Aug  3 23:25 .
drwxr-xr-x 1 root root  160 Aug  3 23:25 ..
drwxr-xr-x 1 root root   60 Aug  3 23:25 .live
drwxr-xr-x 1 root root  100 Aug  3 23:25 .metrics
drwxr-xr-x 1 root root  140 Aug  3 23:25 .timer
drwxr-xr-x 1 root root   57 Aug  3 22:37 data
-rwxrwxr-x 1 root root  714 Aug  3 22:34 dev_eval.sh
-rw-rw-r-- 1 root root 5706 Aug  3 22:34 model.py
-rw-rw-r-- 1 root root  869 Aug  3 22:34 pack_selection.py
drwxr-xr-x 1 root root   10 Aug  3 22:37 submission
-rwxrwxr-x 1 root root 1509 Aug  3 22:34 timer.sh
-rw-rw-r-- 1 root root 2962 Aug  3 22:34 train_nano.py
---
total 792657
drwxr-xr-x 1 root root        57 Aug  3 22:37 .
drwxr-xr-x 1 root root       100 Aug  3 23:25 ..
-r--r--r-- 1 root root   2000128 Aug  3 22:34 multi_dev.npy
-r--r--r-- 1 root root 809679384 Aug  3 22:35 pool.jsonl
---
Mon Aug  3 23:25:54 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H200                    On  |   00000000:CB:00.0 Off |                    0 |
| N/A   28C    P0             77W /  700W |       4MiB / 143771MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+
[metadata] {"noOutputExpected": false}

{"id": 0, "text": "|Viewing Single Post From: Spoilers for the Week of February 11th|\n|Lil||Feb 1 2013, 09:58 AM|\nDon't care about Chloe/Taniel/Jen-Jen. Don't care about Sami, really, but hoping that we get some good \"SAMANTHA GENE!!\" Marlena Death-Stares out of it. And \"newfound\" feelings. Please. If only.\nSTEFANO!! STEFANO, STEFANO, STEFANO!!!! :cheer:\n|Spoilers for the Week of February 11th \u00b7 DAYS: News, Spoilers & Discussion|"}
{"id": 1, "text": "*sigh* Fundamentalist community, let me pass on some advice to you I learned from the atheistic community:\nIf you have set yourself on fire, do not run.\nOkay? Okay?? Please?\nLook, D, you had two months to say to Harvard in private emails, \"Im sorry, I shouldnt have been using that animation in my paid presentations. I wont use it again. I really do like 'Inner Life', though, and would love to use it in classroom presentations, from the BioVisions site, if that is acceptable.\"\nI sat here, for two months, waiting for that to happen, anything to happen, and it didnt. Two months, on your own terms, you could have written a similar post to yesterdays. I would have given you the benefit of the doubt-- maybe you didnt know the credits werent visible to the audience, and I wouldnt have said a word beyond this, as its Harvards problem, not mine. This would have been a funny joke to those of us involved in dealing with you people, but it would have been a PR non-issue for you.\nBut after you set yourself on fire, you didnt douse it out with a bucket of ice cold reality and accountability. You ran. And youre still running.\nWhy not just state \"I screwed up. Sorry everyone.\" and move on? Why the excuses? Why the denial? Why the passive language? Why the vague words and cryptic capitalizations? Why the writes and rewrites of your 'press release'? We know it wasnt written of your own volition, or it would have been done *before* Harvard had to take action. And, your behavior before this, regarding this issue, is 
===
182016 /workspace/data/pool.jsonl

[stdout]
{"id": 0, "text": "|Viewing Single Post From: Spoilers for the Week of February 11th|\n|Lil||Feb 1 2013, 09:58 AM|\nDon't care about Chloe/Taniel/Jen-Jen. Don't care about Sami, really, but hoping that we get some good \"SAMANTHA GENE!!\" Marlena Death-Stares out of it. And \"newfound\" feelings. Please. If only.\nSTEFANO!! STEFANO, STEFANO, STEFANO!!!! :cheer:\n|Spoilers for the Week of February 11th \u00b7 DAYS: News, Spoilers & Discussion|"}
{"id": 1, "text": "*sigh* Fundamentalist community, let me pass on some advice to you I learned from the atheistic community:\nIf you have set yourself on fire, do not run.\nOkay? Okay?? Please?\nLook, D, you had two months to say to Harvard in private emails, \"Im sorry, I shouldnt have been using that animation in my paid presentations. I wont use it again. I really do like 'Inner Life', though, and would love to use it in classroom presentations, from the BioVisions site, if that is acceptable.\"\nI sat here, for two months, waiting for that to happen, anything to happen, and it didnt. Two months, on your own terms, you could have written a similar post to yesterdays. I would have given you the benefit of the doubt-- maybe you didnt know the credits werent visible to the audience, and I wouldnt have said a word beyond this, as its Harvards problem, not mine. This would have been a funny joke to those of us involved in dealing with you people, but it would have been a PR non-issue for you.\nBut after you set yourself on fire, you didnt douse it out with a bucket of ice cold reality and accountability. You ran. And youre still running.\nWhy not just state \"I screwed up. Sorry everyone.\" and move on? Why the excuses? Why the denial? Why the passive language? Why the vague words and cryptic capitalizations? Why the writes and rewrites of your 'press release'? We know it wasnt written of your own volition, or it would have been done *before* Harvard had to take action. And, your behavior before this, regarding this issue, is 
===
182016 /workspace/data/pool.jsonl
[metadata] {"noOutputExpected": false}

=== train_nano.py ===
"""From-scratch nano-GPT (30M) LM training on a 1D token .npy; report held-out perplexity.
Frozen recipe — the ONLY thing that varies between runs is the training token set."""
import argparse, math, time, numpy as np, torch
from model import GPT, GPTConfig

ap = argparse.ArgumentParser()
ap.add_argument("--train_npy", required=True)
ap.add_argument("--heldout_npy", required=True)
ap.add_argument("--out_json", required=True)
ap.add_argument("--max_iters", type=int, default=3000)
ap.add_argument("--warmup", type=int, default=150)
ap.add_argument("--batch", type=int, default=32)
ap.add_argument("--block", type=int, default=256)
ap.add_argument("--lr", type=float, default=6e-4)
ap.add_argument("--seed", type=int, default=1337)
a = ap.parse_args()

torch.manual_seed(a.seed); np.random.seed(a.seed)
dev = "cuda"
tr = np.load(a.train_npy); ho = np.load(a.heldout_npy)
tr = torch.from_numpy(tr.astype(np.int64)); ho = torch.from_numpy(ho.astype(np.int64))
rng = np.random.default_rng(a.seed)

model = GPT(GPTConfig(block_size=a.block, vocab_size=50257, n_layer=6, n_head=6,
                      n_embd=384, dropout=0.0, bias=False)).to(dev)
opt = model.configure_optimizers(0.1, a.lr, (0.9, 0.95), "cuda")

def lr_at(it):
    if it < a.warmup: return a.lr * (it + 1) / (a.warmup + 1)
    r = (it - a.warmup) / max(1, a.max_iters - a.warmup)
    return 0.1 * a.lr + 0.5 * (1 + math.cos(math.pi * r)) * (a.lr - 0.1 * a.lr)

def get_batch(src):
    ix = rng.integers(0, len(src) - a.block - 1, size=a.batch)
    x = torch.stack([src[i:i+a.block] for i in ix]).to(dev)
    y = torch.stack([src[i+1:i+1+a.block] for i in ix]).to(dev)
    return x, y

t0 = time.time()
model.train()
for it in range(a.max_iters):
    for g in opt.param_groups: g["lr"] = lr_at(it)
    x, y = get_batch(tr)
    with torch.autocast("cuda", dtype=torch.bfloat16):
        _, loss = model(x, y)
    opt.zero_grad(set_to_none=True); loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0); opt.step()
train_s = time.time() - t0

# held-out perplexity: mean next-token CE over non-overlapping windows (window-weighted)
import json
model.eval()
nwin = (len(ho) - 1) // a.block
starts = [j * a.block for j in range(nwin)]
sum_loss, nseen = 0.0, 0
with torch.no_grad():
    for i in range(0, nwin, a.batch):
        bs = starts[i:i+a.batch]
        x = torch.stack([ho[s:s+a.block] for s in bs]).to(dev)
        y = torch.stack([ho[s+1:s+1+a.block] for s in bs]).to(dev)
        with torch.autocast("cuda", dtype=torch.bfloat16):
            _, l = model(x, y)
        sum_loss += l.item() * len(bs); nseen += len(bs)
mean_loss = sum_loss / max(1, nseen)
ppl = math.exp(mean_loss)
out = {"train_npy": a.train_npy, "heldout_ppl": ppl, "mean_loss": mean_loss,
       "max_iters": a.max_iters, "seed": a.seed, "train_s": round(train_s, 1)}
json.dump(out, open(a.out_json, "w"))
print(f"HELDOUT_PPL {ppl:.3f}  loss {mean_loss:.4f}  train_s {train_s:.0f}  {a.train_npy}")
=== pack_selection.py ===
"""Solver-side: pack a selection.json (ordered ids) into a 12M-token train array from pool.jsonl."""
import json, sys, numpy as np
from transformers import AutoTokenizer

POOL = "/workspace/data/pool.jsonl"
SEL = sys.argv[1] if len(sys.argv) > 1 else "/workspace/submission/selection.json"
OUT = sys.argv[2] if len(sys.argv) > 2 else "/tmp/dev_train.npy"
BUDGET = 12_000_000

tok = AutoTokenizer.from_pretrained("gpt2"); EOS = tok.eos_token_id
text = {}
for line in open(POOL):
    r = json.loads(line); text[r["id"]] = r["text"]
sel = json.load(open(SEL))
parts, tot = [], 0
for i in sel:
    ids = tok(text[i], add_special_tokens=False).input_ids
    parts.extend(ids); parts.append(EOS); tot += len(ids) + 1
    if tot >= BUDGET: break
arr = np.array(parts[:BUDGET], dtype=np.uint16)
np.save(OUT, arr)
print(f"packed {len(arr)} tokens from {len(sel)} ids -> {OUT}")
=== dev_eval.sh ===
#!/bin/bash
# Diagnostic: train the frozen 30M GPT on your selection and report DEV perplexity.
# Usage: bash dev_eval.sh [selection.json]   (default /workspace/submission/selection.json)
# NOTE: the official target is a DIFFERENT (hidden) sample from the same HQ domain.
set -euo pipefail
export TOKENIZERS_PARALLELISM=false
SEL="${1:-/workspace/submission/selection.json}"
cd /workspace
python3 pack_selection.py "$SEL" /tmp/dev_train.npy
python3 train_nano.py --train_npy /tmp/dev_train.npy --heldout_npy /workspace/data/multi_dev.npy \
    --out_json /tmp/dev.json --max_iters 3000 --seed 1337 2>&1 | grep HELDOUT_PPL
echo "(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)"
=== timer.sh ===
#!/usr/bin/env bash
# Budget timer daemon — writes /workspace/.timer/remaining_secs every 10s so the
# solver can poll time left instead of doing wall-clock arithmetic.
#
# Budget (seconds) comes from $TASK_BUDGET_SECS (set by the runner to the agent
# timeout). Idempotent via a PID file. On ScaleTrain the trial runner starts its
# own timer; this file is the fallback for the Modal/local harbor backends, which
# bootstrap /workspace/timer.sh on environment start. The fallback default below
# (7200 = 2 h) matches this task's [agent] timeout_sec and is only used if
# $TASK_BUDGET_SECS is unset.
set -u
TIMER_DIR="/workspace/.timer"
PID_FILE="$TIMER_DIR/timer.pid"
mkdir -p "$TIMER_DIR"

if [ -s "$PID_FILE" ] && kill -0 "$(cat "$PID_FILE" 2>/dev/null)" 2>/dev/null; then
    exit 0   # already running
fi
echo $$ > "$PID_FILE"
trap 'rm -f "$PID_FILE"' EXIT INT TERM

START=$(date +%s)
BUDGET="${TASK_BUDGET_SECS:-7200}"
echo "$START" > "$TIMER_DIR/start_epoch"
echo "$BUDGET" > "$TIMER_DIR/budget_secs"

while true; do
    NOW=$(date +%s)
    ELAPSED=$((NOW - START))
    REMAINING=$((BUDGET - ELAPSED))
    [ "$REMAINING" -lt 0 ] && REMAINING=0
    echo "$REMAINING" > "$TIMER_DIR/remaining_secs"
    echo "$ELAPSED" > "$TIMER_DIR/elapsed_secs"
    [ "$REMAINING" -le 1800 ] && [ ! -f "$TIMER_DIR/alert_30min" ] && touch "$TIMER_DIR/alert_30min"
    [ "$REMAINING" -le 600 ]  && [ ! -f "$TIMER_DIR/alert_10min" ] && touch "$TIMER_DIR/alert_10min"
    [ "$REMAINING" -le 0 ] && break
    sleep 10
done

[stdout]
=== train_nano.py ===
"""From-scratch nano-GPT (30M) LM training on a 1D token .npy; report held-out perplexity.
Frozen recipe — the ONLY thing that varies between runs is the training token set."""
import argparse, math, time, numpy as np, torch
from model import GPT, GPTConfig

ap = argparse.ArgumentParser()
ap.add_argument("--train_npy", required=True)
ap.add_argument("--heldout_npy", required=True)
ap.add_argument("--out_json", required=True)
ap.add_argument("--max_iters", type=int, default=3000)
ap.add_argument("--warmup", type=int, default=150)
ap.add_argument("--batch", type=int, default=32)
ap.add_argument("--block", type=int, default=256)
ap.add_argument("--lr", type=float, default=6e-4)
ap.add_argument("--seed", type=int, default=1337)
a = ap.parse_args()

torch.manual_seed(a.seed); np.random.seed(a.seed)
dev = "cuda"
tr = np.load(a.train_npy); ho = np.load(a.heldout_npy)
tr = torch.from_numpy(tr.astype(np.int64)); ho = torch.from_numpy(ho.astype(np.int64))
rng = np.random.default_rng(a.seed)

model = GPT(GPTConfig(block_size=a.block, vocab_size=50257, n_layer=6, n_head=6,
                      n_embd=384, dropout=0.0, bias=False)).to(dev)
opt = model.configure_optimizers(0.1, a.lr, (0.9, 0.95), "cuda")

def lr_at(it):
    if it < a.warmup: return a.lr * (it + 1) / (a.warmup + 1)
    r = (it - a.warmup) / max(1, a.max_iters - a.warmup)
    return 0.1 * a.lr + 0.5 * (1 + math.cos(math.pi * r)) * (a.lr - 0.1 * a.lr)

def get_batch(src):
    ix = rng.integers(0, len(src) - a.block - 1, size=a.batch)
    x = torch.stack([src[i:i+a.block] for i in ix]).to(dev)
    y = torch.stack([src[i+1:i+1+a.block] for i in ix]).to(dev)
    return x, y

t0 = time.time()
model.train()
for it in range(a.max_iters):
    for g in opt.param_groups: g["lr"] = lr_at(it)
    x, y = get_batch(tr)
    with torch.autocast("cuda", dtype=torch.bfloat16):
        _, loss = model(x, y)
    opt.zero_grad(set_to_none=True); loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0); opt.step()
train_s = time.time() - t0

# held-out perplexity: mean next-token CE over non-overlapping windows (window-weighted)
import json
model.eval()
nwin = (len(ho) - 1) // a.block
starts = [j * a.block for j in range(nwin)]
sum_loss, nseen = 0.0, 0
with torch.no_grad():
    for i in range(0, nwin, a.batch):
        bs = starts[i:i+a.batch]
        x = torch.stack([ho[s:s+a.block] for s in bs]).to(dev)
        y = torch.stack([ho[s+1:s+1+a.block] for s in bs]).to(dev)
        with torch.autocast("cuda", dtype=torch.bfloat16):
            _, l = model(x, y)
        sum_loss += l.item() * len(bs); nseen += len(bs)
mean_loss = sum_loss / max(1, nseen)
ppl = math.exp(mean_loss)
out = {"train_npy": a.train_npy, "heldout_ppl": ppl, "mean_loss": mean_loss,
       "max_iters": a.max_iters, "seed": a.seed, "train_s": round(train_s, 1)}
json.dump(out, open(a.out_json, "w"))
print(f"HELDOUT_PPL {ppl:.3f}  loss {mean_loss:.4f}  train_s {train_s:.0f}  {a.train_npy}")
=== pack_selection.py ===
"""Solver-side: pack a selection.json (ordered ids) into a 12M-token train array from pool.jsonl."""
import json, sys, numpy as np
from transformers import AutoTokenizer

POOL = "/workspace/data/pool.jsonl"
SEL = sys.argv[1] if len(sys.argv) > 1 else "/workspace/submission/selection.json"
OUT = sys.argv[2] if len(sys.argv) > 2 else "/tmp/dev_train.npy"
BUDGET = 12_000_000

tok = AutoTokenizer.from_pretrained("gpt2"); EOS = tok.eos_token_id
text = {}
for line in open(POOL):
    r = json.loads(line); text[r["id"]] = r["text"]
sel = json.load(open(SEL))
parts, tot = [], 0
for i in sel:
    ids = tok(text[i], add_special_tokens=False).input_ids
    parts.extend(ids); parts.append(EOS); tot += len(ids) + 1
    if tot >= BUDGET: break
arr = np.array(parts[:BUDGET], dtype=np.uint16)
np.save(OUT, arr)
print(f"packed {len(arr)} tokens from {len(sel)} ids -> {OUT}")
=== dev_eval.sh ===
#!/bin/bash
# Diagnostic: train the frozen 30M GPT on your selection and report DEV perplexity.
# Usage: bash dev_eval.sh [selection.json]   (default /workspace/submission/selection.json)
# NOTE: the official target is a DIFFERENT (hidden) sample from the same HQ domain.
set -euo pipefail
export TOKENIZERS_PARALLELISM=false
SEL="${1:-/workspace/submission/selection.json}"
cd /workspace
python3 pack_selection.py "$SEL" /tmp/dev_train.npy
python3 train_nano.py --train_npy /tmp/dev_train.npy --heldout_npy /workspace/data/multi_dev.npy \
    --out_json /tmp/dev.json --max_iters 3000 --seed 1337 2>&1 | grep HELDOUT_PPL
echo "(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)"
=== timer.sh ===
#!/usr/bin/env bash
# Budget timer daemon — writes /workspace/.timer/remaining_secs every 10s so the
# solver can poll time left instead of doing wall-clock arithmetic.
#
# Budget (seconds) comes from $TASK_BUDGET_SECS (set by the runner to the agent
# timeout). Idempotent via a PID file. On ScaleTrain the trial runner starts its
# own timer; this file is the fallback for the Modal/local harbor backends, which
# bootstrap /workspace/timer.sh on environment start. The fallback default below
# (7200 = 2 h) matches this task's [agent] timeout_sec and is only used if
# $TASK_BUDGET_SECS is unset.
set -u
TIMER_DIR="/workspace/.timer"
PID_FILE="$TIMER_DIR/timer.pid"
mkdir -p "$TIMER_DIR"

if [ -s "$PID_FILE" ] && kill -0 "$(cat "$PID_FILE" 2>/dev/null)" 2>/dev/null; then
    exit 0   # already running
fi
echo $$ > "$PID_FILE"
trap 'rm -f "$PID_FILE"' EXIT INT TERM

START=$(date +%s)
BUDGET="${TASK_BUDGET_SECS:-7200}"
echo "$START" > "$TIMER_DIR/start_epoch"
echo "$BUDGET" > "$TIMER_DIR/budget_secs"

while true; do
    NOW=$(date +%s)
    ELAPSED=$((NOW - START))
    REMAINING=$((BUDGET - ELAPSED))
    [ "$REMAINING" -lt 0 ] && REMAINING=0
    echo "$REMAINING" > "$TIMER_DIR/remaining_secs"
    echo "$ELAPSED" > "$TIMER_DIR/elapsed_secs"
    [ "$REMAINING" -le 1800 ] && [ ! -f "$TIMER_DIR/alert_30min" ] && touch "$TIMER_DIR/alert_30min"
    [ "$REMAINING" -le 600 ]  && [ ! -f "$TIMER_DIR/alert_10min" ] && touch "$TIMER_DIR/alert_10min"
    [ "$REMAINING" -le 0 ] && break
    sleep 10
done
[metadata] {"noOutputExpected": false}

=== model.py ===
"""A small GPT (decoder-only Transformer) used as the distillation *student*.

Self-contained re-implementation in this repo's own idiom (no external GPT code).
GPT-2 byte-level BPE vocabulary (50257) so the student shares the teacher's
tokenizer. Standard pre-norm Transformer with weight-tied embeddings.
"""
from dataclasses import dataclass
import math
import torch
import torch.nn as nn
import torch.nn.functional as F


@dataclass
class GPTConfig:
    block_size: int = 256
    vocab_size: int = 50257
    n_layer: int = 6
    n_head: int = 6
    n_embd: int = 384
    dropout: float = 0.0
    bias: bool = False


class LayerNorm(nn.Module):
    def __init__(self, ndim, bias):
        super().__init__()
        self.weight = nn.Parameter(torch.ones(ndim))
        self.bias = nn.Parameter(torch.zeros(ndim)) if bias else None

    def forward(self, x):
        return F.layer_norm(x, self.weight.shape, self.weight, self.bias, 1e-5)


class CausalSelfAttention(nn.Module):
    def __init__(self, cfg: GPTConfig):
        super().__init__()
        assert cfg.n_embd % cfg.n_head == 0
        self.c_attn = nn.Linear(cfg.n_embd, 3 * cfg.n_embd, bias=cfg.bias)
        self.c_proj = nn.Linear(cfg.n_embd, cfg.n_embd, bias=cfg.bias)
        self.n_head = cfg.n_head
        self.n_embd = cfg.n_embd
        self.dropout = cfg.dropout

    def forward(self, x):
        B, T, C = x.size()
        q, k, v = self.c_attn(x).split(self.n_embd, dim=2)
        q = q.view(B, T, self.n_head, C // self.n_head).transpose(1, 2)
        k = k.view(B, T, self.n_head, C // self.n_head).transpose(1, 2)
        v = v.view(B, T, self.n_head, C // self.n_head).transpose(1, 2)
        y = F.scaled_dot_product_attention(
            q, k, v, dropout_p=self.dropout if self.training else 0.0, is_causal=True
        )
        y = y.transpose(1, 2).contiguous().view(B, T, C)
        return self.c_proj(y)


class MLP(nn.Module):
    def __init__(self, cfg: GPTConfig):
        super().__init__()
        self.c_fc = nn.Linear(cfg.n_embd, 4 * cfg.n_embd, bias=cfg.bias)
        self.c_proj = nn.Linear(4 * cfg.n_embd, cfg.n_embd, bias=cfg.bias)
        self.dropout = nn.Dropout(cfg.dropout)

    def forward(self, x):
        return self.dropout(self.c_proj(F.gelu(self.c_fc(x))))


class Block(nn.Module):
    def __init__(self, cfg: GPTConfig):
        super().__init__()
        self.ln_1 = LayerNorm(cfg.n_embd, cfg.bias)
        self.attn = CausalSelfAttention(cfg)
        self.ln_2 = LayerNorm(cfg.n_embd, cfg.bias)
        self.mlp = MLP(cfg)

    def forward(self, x):
        x = x + self.attn(self.ln_1(x))
        x = x + self.mlp(self.ln_2(x))
        return x


class GPT(nn.Module):
    def __init__(self, cfg: GPTConfig):
        super().__init__()
        self.cfg = cfg
        self.transformer = nn.ModuleDict(dict(
            wte=nn.Embedding(cfg.vocab_size, cfg.n_embd),
            wpe=nn.Embedding(cfg.block_size, cfg.n_embd),
            drop=nn.Dropout(cfg.dropout),
            h=nn.ModuleList([Block(cfg) for _ in range(cfg.n_layer)]),
            ln_f=LayerNorm(cfg.n_embd, cfg.bias),
        ))
        self.lm_head = nn.Linear(cfg.n_embd, cfg.vocab_size, bias=False)
        self.transformer.wte.weight = self.lm_head.weight  # weight tying

        self.apply(self._init_weights)
        # GPT-2 style scaled init on residual projections
        for pn, p in self.named_parameters():
            if pn.endswith('c_proj.weight'):
                nn.init.normal_(p, mean=0.0, std=0.02 / math.sqrt(2 * cfg.n_layer))

    def _init_weights(self, module):
        if isinstance(module, nn.Linear):
            nn.init.normal_(module.weight, mean=0.0, std=0.02)
            if module.bias is not None:
                nn.init.zeros_(module.bias)
        elif isinstance(module, nn.Embedding):
            nn.init.normal_(module.weight, mean=0.0, std=0.02)

    def forward(self, idx, targets=None):
        """Return (logits, ce_loss). logits: [B, T, vocab]. ce_loss is the plain
        next-token cross-entropy if `targets` given, else None. The distillation
        loss is computed separately in kd_trainer.kd_loss."""
        B, T = idx.size()
        pos = torch.arange(0, T, dtype=torch.long, device=idx.device)
        x = self.transformer.drop(self.transformer.wte(idx) + self.transformer.wpe(pos))
        for block in self.transformer.h:
            x = block(x)
        x = self.transformer.ln_f(x)
        logits = self.lm_head(x)
        ce_loss = None
        if targets is not None:
            ce_loss = F.cross_entropy(
                logits.view(-1, logits.size(-1)), targets.reshape(-1), ignore_index=-1
            )
        return logits, ce_loss

    def configure_optimizers(self, weight_decay, lr, betas, device_type):
        decay, no_decay = [], []
        for n, p in self.named_parameters():
            if not p.requires_grad:
                continue
            (decay if p.dim() >= 2 else no_decay).append(p)
        groups = [
            {'params': decay, 'weight_decay': weight_decay},
            {'params': no_decay, 'weight_decay': 0.0},
        ]
        fused = device_type == 'cuda'
        return torch.optim.AdamW(groups, lr=lr, betas=betas, fused=fused)

    @torch.no_grad()
    def estimate_ppl(self, get_eval_batch, n_batches):
        """Mean per-token NLL over `n_batches` eval batches -> perplexity."""
        was_training = self.training
        self.eval()
        losses = torch.zeros(n_batches)
        for i in range(n_batches):
            x, y = get_eval_batch()
            _, loss = self(x, y)
            losses[i] = loss.item()
        if was_training:
            self.train()
        return math.exp(losses.mean().item())
=== python env ===
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'sklearn'
16

[stdout]
=== model.py ===
"""A small GPT (decoder-only Transformer) used as the distillation *student*.

Self-contained re-implementation in this repo's own idiom (no external GPT code).
GPT-2 byte-level BPE vocabulary (50257) so the student shares the teacher's
tokenizer. Standard pre-norm Transformer with weight-tied embeddings.
"""
from dataclasses import dataclass
import math
import torch
import torch.nn as nn
import torch.nn.functional as F


@dataclass
class GPTConfig:
    block_size: int = 256
    vocab_size: int = 50257
    n_layer: int = 6
    n_head: int = 6
    n_embd: int = 384
    dropout: float = 0.0
    bias: bool = False


class LayerNorm(nn.Module):
    def __init__(self, ndim, bias):
        super().__init__()
        self.weight = nn.Parameter(torch.ones(ndim))
        self.bias = nn.Parameter(torch.zeros(ndim)) if bias else None

    def forward(self, x):
        return F.layer_norm(x, self.weight.shape, self.weight, self.bias, 1e-5)


class CausalSelfAttention(nn.Module):
    def __init__(self, cfg: GPTConfig):
        super().__init__()
        assert cfg.n_embd % cfg.n_head == 0
        self.c_attn = nn.Linear(cfg.n_embd, 3 * cfg.n_embd, bias=cfg.bias)
        self.c_proj = nn.Linear(cfg.n_embd, cfg.n_embd, bias=cfg.bias)
        self.n_head = cfg.n_head
        self.n_embd = cfg.n_embd
        self.dropout = cfg.dropout

    def forward(self, x):
        B, T, C = x.size()
        q, k, v = self.c_attn(x).split(self.n_embd, dim=2)
        q = q.view(B, T, self.n_head, C // self.n_head).transpose(1, 2)
        k = k.view(B, T, self.n_head, C // self.n_head).transpose(1, 2)
        v = v.view(B, T, self.n_head, C // self.n_head).transpose(1, 2)
        y = F.scaled_dot_product_attention(
            q, k, v, dropout_p=self.dropout if self.training else 0.0, is_causal=True
        )
        y = y.transpose(1, 2).contiguous().view(B, T, C)
        return self.c_proj(y)


class MLP(nn.Module):
    def __init__(self, cfg: GPTConfig):
        super().__init__()
        self.c_fc = nn.Linear(cfg.n_embd, 4 * cfg.n_embd, bias=cfg.bias)
        self.c_proj = nn.Linear(4 * cfg.n_embd, cfg.n_embd, bias=cfg.bias)
        self.dropout = nn.Dropout(cfg.dropout)

    def forward(self, x):
        return self.dropout(self.c_proj(F.gelu(self.c_fc(x))))


class Block(nn.Module):
    def __init__(self, cfg: GPTConfig):
        super().__init__()
        self.ln_1 = LayerNorm(cfg.n_embd, cfg.bias)
        self.attn = CausalSelfAttention(cfg)
        self.ln_2 = LayerNorm(cfg.n_embd, cfg.bias)
        self.mlp = MLP(cfg)

    def forward(self, x):
        x = x + self.attn(self.ln_1(x))
        x = x + self.mlp(self.ln_2(x))
        return x


class GPT(nn.Module):
    def __init__(self, cfg: GPTConfig):
        super().__init__()
        self.cfg = cfg
        self.transformer = nn.ModuleDict(dict(
            wte=nn.Embedding(cfg.vocab_size, cfg.n_embd),
            wpe=nn.Embedding(cfg.block_size, cfg.n_embd),
            drop=nn.Dropout(cfg.dropout),
            h=nn.ModuleList([Block(cfg) for _ in range(cfg.n_layer)]),
            ln_f=LayerNorm(cfg.n_embd, cfg.bias),
        ))
        self.lm_head = nn.Linear(cfg.n_embd, cfg.vocab_size, bias=False)
        self.transformer.wte.weight = self.lm_head.weight  # weight tying

        self.apply(self._init_weights)
        # GPT-2 style scaled init on residual projections
        for pn, p in self.named_parameters():
            if pn.endswith('c_proj.weight'):
                nn.init.normal_(p, mean=0.0, std=0.02 / math.sqrt(2 * cfg.n_layer))

    def _init_weights(self, module):
        if isinstance(module, nn.Linear):
            nn.init.normal_(module.weight, mean=0.0, std=0.02)
            if module.bias is not None:
                nn.init.zeros_(module.bias)
        elif isinstance(module, nn.Embedding):
            nn.init.normal_(module.weight, mean=0.0, std=0.02)

    def forward(self, idx, targets=None):
        """Return (logits, ce_loss). logits: [B, T, vocab]. ce_loss is the plain
        next-token cross-entropy if `targets` given, else None. The distillation
        loss is computed separately in kd_trainer.kd_loss."""
        B, T = idx.size()
        pos = torch.arange(0, T, dtype=torch.long, device=idx.device)
        x = self.transformer.drop(self.transformer.wte(idx) + self.transformer.wpe(pos))
        for block in self.transformer.h:
            x = block(x)
        x = self.transformer.ln_f(x)
        logits = self.lm_head(x)
        ce_loss = None
        if targets is not None:
            ce_loss = F.cross_entropy(
                logits.view(-1, logits.size(-1)), targets.reshape(-1), ignore_index=-1
            )
        return logits, ce_loss

    def configure_optimizers(self, weight_decay, lr, betas, device_type):
        decay, no_decay = [], []
        for n, p in self.named_parameters():
            if not p.requires_grad:
                continue
            (decay if p.dim() >= 2 else no_decay).append(p)
        groups = [
            {'params': decay, 'weight_decay': weight_decay},
            {'params': no_decay, 'weight_decay': 0.0},
        ]
        fused = device_type == 'cuda'
        return torch.optim.AdamW(groups, lr=lr, betas=betas, fused=fused)

    @torch.no_grad()
    def estimate_ppl(self, get_eval_batch, n_batches):
        """Mean per-token NLL over `n_batches` eval batches -> perplexity."""
        was_training = self.training
        self.eval()
        losses = torch.zeros(n_batches)
        for i in range(n_batches):
            x, y = get_eval_batch()
            _, loss = self(x, y)
            losses[i] = loss.item()
        if was_training:
            self.train()
        return math.exp(losses.mean().item())
=== python env ===
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'sklearn'
16
[metadata] {"noOutputExpected": false}

total 2
drwxr-xr-x 1 root root 52 Aug  3 22:37 .
drwxr-xr-x 1 root root 25 Aug  3 22:37 ..
drwxr-xr-x 1 root root 34 Aug  3 22:37 .locks
drwxr-xr-x 1 root root 85 Aug  3 22:37 models--gpt2
--- pip ---
Package                  Version
------------------------ ---------
certifi                  2026.7.22
charset-normalizer       3.4.9
filelock                 3.32.2
fsspec                   2026.7.0
hf-xet                   1.6.0
huggingface_hub          0.36.2
idna                     3.18
Jinja2                   3.1.6
MarkupSafe               3.0.3
mpmath                   1.3.0
networkx                 3.4.2
numpy                    1.26.4
nvidia-cublas-cu12       12.8.4.1
nvidia-cuda-cupti-cu12   12.8.90
nvidia-cuda-nvrtc-cu12   12.8.93
nvidia-cuda-runtime-cu12 12.8.90
nvidia-cudnn-cu12        9.10.2.21
nvidia-cufft-cu12        11.3.3.83
nvidia-cufile-cu12       1.13.1.3
nvidia-curand-cu12       10.3.9.90
nvidia-cusolver-cu12     11.7.3.90
nvidia-cusparse-cu12     12.5.8.93
nvidia-cusparselt-cu12   0.7.1
nvidia-nccl-cu12         2.27.3
nvidia-nvjitlink-cu12    12.8.93
nvidia-nvtx-cu12         12.8.90
packaging                26.2
pip                      26.2
PyYAML                   6.0.3
regex                    2026.7.19
requests                 2.34.2
safetensors              0.8.0
setuptools               83.0.0
sympy                    1.14.0
tokenizers               0.22.2
torch                    2.8.0
tqdm                     4.70.0
transformers             4.57.1

[stdout]
total 2
drwxr-xr-x 1 root root 52 Aug  3 22:37 .
drwxr-xr-x 1 root root 25 Aug  3 22:37 ..
drwxr-xr-x 1 root root 34 Aug  3 22:37 .locks
drwxr-xr-x 1 root root 85 Aug  3 22:37 models--gpt2
--- pip ---
Package                  Version
------------------------ ---------
certifi                  2026.7.22
charset-normalizer       3.4.9
filelock                 3.32.2
fsspec                   2026.7.0
hf-xet                   1.6.0
huggingface_hub          0.36.2
idna                     3.18
Jinja2                   3.1.6
MarkupSafe               3.0.3
mpmath                   1.3.0
networkx                 3.4.2
numpy                    1.26.4
nvidia-cublas-cu12       12.8.4.1
nvidia-cuda-cupti-cu12   12.8.90
nvidia-cuda-nvrtc-cu12   12.8.93
nvidia-cuda-runtime-cu12 12.8.90
nvidia-cudnn-cu12        9.10.2.21
nvidia-cufft-cu12        11.3.3.83
nvidia-cufile-cu12       1.13.1.3
nvidia-curand-cu12       10.3.9.90
nvidia-cusolver-cu12     11.7.3.90
nvidia-cusparse-cu12     12.5.8.93
nvidia-cusparselt-cu12   0.7.1
nvidia-nccl-cu12         2.27.3
nvidia-nvjitlink-cu12    12.8.93
nvidia-nvtx-cu12         12.8.90
packaging                26.2
pip                      26.2
PyYAML                   6.0.3
regex                    2026.7.19
requests                 2.34.2
safetensors              0.8.0
setuptools               83.0.0
sympy                    1.14.0
tokenizers               0.22.2
torch                    2.8.0
tqdm                     4.70.0
transformers             4.57.1
[metadata] {"noOutputExpected": false}

uint16 (1000000,) 0 50256
n_eos 2704 first [ 206  332  556  652  854  986 1010 1080 1231 1555 1789 2010]
n_segs 2705 median len 193 mean 369
~~~~ ' Beardmore was sufficiently impressed with Shackleton to offer financial support , but other donations proved hard to come by . Nevertheless , in February 1907 , Shackleton presented to the Royal Geographic Society his plans for an Antarctic expedition , the details of which , under the name British'
~~~~ " On 1 January 1908 , Nimrod sailed for the Antarctic from Lyttelton Harbour , New Zealand . Shackleton 's original plans had envisaged using the old Discovery base in McMurdo Sound to launch his attempts on the South Pole and South Magnetic Pole . However , before leaving England , he had been press"
~~~~ " To conserve coal , the ship was towed 1 @,@ 650 miles ( 2 @,@ 655 km ) by the steamer Koonya to the Antarctic ice , after Shackleton had persuaded the New Zealand government and the Union Steamship Company to share the cost . In accordance with Shackleton 's promise to Scott , the ship headed for t"
~~~~ " Nimrod arrived at McMurdo Sound on 29 January , but was stopped by ice 16 miles ( 26 km ) north of Discovery 's old base at Hut Point . After considerable weather delays , Shackleton 's base was eventually established at Cape Royds , about 24 miles ( 39 km ) north of Hut Point . The party was in hi"
~~~~ ' The " Great Southern Journey " , as Frank Wild called it , began on 29 October 1908 . On 9 January 1909 , Shackleton and three companions ( Wild , Eric Marshall and Jameson Adams ) reached a new Farthest South latitude of 88 ° 23 \' S , a point only 112 miles ( 180 km ) from the Pole . En route the '
~~~~ " The expedition 's other main accomplishments included the first ascent of Mount Erebus , and the discovery of the approximate location of the South Magnetic Pole , reached on 16 January 1909 by Edgeworth David , Douglas Mawson , and Alistair Mackay . Shackleton returned to the United Kingdom as a h"

[stdout]
uint16 (1000000,) 0 50256
n_eos 2704 first [ 206  332  556  652  854  986 1010 1080 1231 1555 1789 2010]
n_segs 2705 median len 193 mean 369
~~~~ ' Beardmore was sufficiently impressed with Shackleton to offer financial support , but other donations proved hard to come by . Nevertheless , in February 1907 , Shackleton presented to the Royal Geographic Society his plans for an Antarctic expedition , the details of which , under the name British'
~~~~ " On 1 January 1908 , Nimrod sailed for the Antarctic from Lyttelton Harbour , New Zealand . Shackleton 's original plans had envisaged using the old Discovery base in McMurdo Sound to launch his attempts on the South Pole and South Magnetic Pole . However , before leaving England , he had been press"
~~~~ " To conserve coal , the ship was towed 1 @,@ 650 miles ( 2 @,@ 655 km ) by the steamer Koonya to the Antarctic ice , after Shackleton had persuaded the New Zealand government and the Union Steamship Company to share the cost . In accordance with Shackleton 's promise to Scott , the ship headed for t"
~~~~ " Nimrod arrived at McMurdo Sound on 29 January , but was stopped by ice 16 miles ( 26 km ) north of Discovery 's old base at Hut Point . After considerable weather delays , Shackleton 's base was eventually established at Cape Royds , about 24 miles ( 39 km ) north of Hut Point . The party was in hi"
~~~~ ' The " Great Southern Journey " , as Frank Wild called it , began on 29 October 1908 . On 9 January 1909 , Shackleton and three companions ( Wild , Eric Marshall and Jameson Adams ) reached a new Farthest South latitude of 88 ° 23 \' S , a point only 112 miles ( 180 km ) from the Pole . En route the '
~~~~ " The expedition 's other main accomplishments included the first ascent of Mount Erebus , and the discovery of the approximate location of the South Magnetic Pole , reached on 16 January 1909 by Edgeworth David , Douglas Mawson , and Alistair Mackay . Shackleton returned to the United Kingdom as a h"
[metadata] {"noOutputExpected": false}

--- frac=0.0 seg=0 len=206
' Beardmore was sufficiently impressed with Shackleton to offer financial support , but other donations proved hard to come by . Nevertheless , in February 1907 , Shackleton presented to the Royal Geographic Society his plans for an Antarctic expedition , the d'

--- frac=0.2 seg=540 len=277
" As the 27th Infantry 's trucks rolled northward from Tabu @-@ dong and approached their Line of Departure , the men inside could see the North Koreans and South Koreans fighting on the ridges overlooking the road . The infantry dismounted and deployed an atta"

--- frac=0.26 seg=703 len=145
" After releasing the book , John Lennon , My Brother — written with Giuliano , and a foreword by McCartney — Baird travelled to New York during 1989 to appear at a Beatlefest convention , and was asked if she could prove she was really Lennon 's half @-@ siste"

--- frac=0.4 seg=1081 len=75
" In 1998 , Wilder collaborated on the book Gilda 's Disease with oncologist Steven Piver , sharing personal experiences of Radner 's struggle with ovarian cancer . Wilder himself was hospitalized with non @-@ Hodgkin lymphoma in 1999 , but confirmed in March 2"

--- frac=0.5 seg=1352 len=101
" Mining subsidence coupled with structural and political changes to the mining industry began the decline in Astley 's industrial activities during the mid @-@ 20th century ; its cotton mill closed in 1955 , and the last coal was brought to the surface in 1970"

--- frac=0.6 seg=1622 len=81
' The Desert Sunlight Solar Farm is a 550 MW power plant in Riverside County , California , that uses thin @-@ film CdTe @-@ modules made by First Solar . As of November 2014 , the 550 megawatt Topaz Solar Farm was the largest photovoltaic power plant in the wo'

--- frac=0.75 seg=2028 len=874
'When Congress president Rahul Gandhi enters the imposing corridors of the 1300-year-old Sharada Peeth on Wednesday, historians will remember the time his grandmother Indira Gandhi visited the spot 40 years ago in 1978.Like the Congress of today, the party was '

--- frac=0.8 seg=2163 len=296
"Barcelona host Real Madrid in the Spanish football season's opening act in the Super Cup, first leg on Sunday needing to dispel doubts they have fallen further behind their rivals following Neymar's departure.The Brazilian's world record 222 million euro ($261"

--- frac=0.9 seg=2433 len=480
"<p>I need to plot discrete X,Y data as a MAP in a PNG or GIF file colour-coded to indicate discrete values.</p>\n\n<p>All the Javascript Chart libraries I've seen do pie charts, line graphs, bar charts, etc, but NOT scatterplots.</p>\n\n<p>Does anyone know the nam"

--- frac=0.99 seg=2676 len=703
'<p>I\'m looking for a C# / ASP.net website that I can download the source code and practice on, ideally it will come with documentation and hopefully some "bugs" or feature requests to try fixing. </p>\n\n<p>Dose anyone know where to find such a website?</p>\n\n<p>'

[stdout]
--- frac=0.0 seg=0 len=206
' Beardmore was sufficiently impressed with Shackleton to offer financial support , but other donations proved hard to come by . Nevertheless , in February 1907 , Shackleton presented to the Royal Geographic Society his plans for an Antarctic expedition , the d'

--- frac=0.2 seg=540 len=277
" As the 27th Infantry 's trucks rolled northward from Tabu @-@ dong and approached their Line of Departure , the men inside could see the North Koreans and South Koreans fighting on the ridges overlooking the road . The infantry dismounted and deployed an atta"

--- frac=0.26 seg=703 len=145
" After releasing the book , John Lennon , My Brother — written with Giuliano , and a foreword by McCartney — Baird travelled to New York during 1989 to appear at a Beatlefest convention , and was asked if she could prove she was really Lennon 's half @-@ siste"

--- frac=0.4 seg=1081 len=75
" In 1998 , Wilder collaborated on the book Gilda 's Disease with oncologist Steven Piver , sharing personal experiences of Radner 's struggle with ovarian cancer . Wilder himself was hospitalized with non @-@ Hodgkin lymphoma in 1999 , but confirmed in March 2"

--- frac=0.5 seg=1352 len=101
" Mining subsidence coupled with structural and political changes to the mining industry began the decline in Astley 's industrial activities during the mid @-@ 20th century ; its cotton mill closed in 1955 , and the last coal was brought to the surface in 1970"

--- frac=0.6 seg=1622 len=81
' The Desert Sunlight Solar Farm is a 550 MW power plant in Riverside County , California , that uses thin @-@ film CdTe @-@ modules made by First Solar . As of November 2014 , the 550 megawatt Topaz Solar Farm was the largest photovoltaic power plant in the wo'

--- frac=0.75 seg=2028 len=874
'When Congress president Rahul Gandhi enters the imposing corridors of the 1300-year-old Sharada Peeth on Wednesday, historians will remember the time his grandmother Indira Gandhi visited the spot 40 years ago in 1978.Like the Congress of today, the party was '

--- frac=0.8 seg=2163 len=296
"Barcelona host Real Madrid in the Spanish football season's opening act in the Super Cup, first leg on Sunday needing to dispel doubts they have fallen further behind their rivals following Neymar's departure.The Brazilian's world record 222 million euro ($261"

--- frac=0.9 seg=2433 len=480
"<p>I need to plot discrete X,Y data as a MAP in a PNG or GIF file colour-coded to indicate discrete values.</p>\n\n<p>All the Javascript Chart libraries I've seen do pie charts, line graphs, bar charts, etc, but NOT scatterplots.</p>\n\n<p>Does anyone know the nam"

--- frac=0.99 seg=2676 len=703
'<p>I\'m looking for a C# / ASP.net website that I can download the source code and practice on, ideally it will come with documentation and hopefully some "bugs" or feature requests to try fixing. </p>\n\n<p>Dose anyone know where to find such a website?</p>\n\n<p>'
[metadata] {"noOutputExpected": false}

sampled 152
=== id 0 len 414
'|Viewing Single Post From: Spoilers for the Week of February 11th|\n|Lil||Feb 1 2013, 09:58 AM|\nDon\'t care about Chloe/Taniel/Jen-Jen. Don\'t care about Sami, really, but hoping that we get some good "SAMANTHA GENE!!" Marlena Death-'

=== id 1200 len 3344
'Camtasia Studio Crack:is an amazing software at that time in the market which permits you to create many good looking videos and many other applications for entertainment purposes. With the help of this application, you can make s'

=== id 2400 len 1458
'North Shore Farmhouse | What’s New Is Old AgainNorth Shore Farmhouse | A Race to the Punch ListNorth Shore Farmhouse | House Requires Some AssemblyNorth Shore Farmhouse | Making New Look OldThe North Shore Farmhouse | 400 Years of'

=== id 3600 len 417
"Actionable Intelligence for Today's Market\nPrecious Metals Futures\nAll other Futures\nM1904Trading Members Only\nThis page is a work in progress, I have to get the site making money before we do all the big brain stuff. Hold pls.\nBu"

=== id 4800 len 1848
'Spain: Casillas (Víctor Valdés 46’); Azpilicueta, Javi Martínez, Sergio Ramos (Raúl Albiol 65‘), Jordi Alba; Busquets (Xabi Alonso 46’), Thiago; Pedro (Cazorla 82’), Fàbregas (David Silva 46’), Iniesta (Jesús Navas 65‘); Diego Cos'

=== id 6000 len 4259
'Once a month, female students pack the cozy chapel at the Holy Spirit Friary that overlooks the Franciscan University of Steubenville, Ohio.\nThese gatherings are confidential, with no one discussing who is or who isn’t among the 5'

=== id 7200 len 1353
'Season 4 of Apex Legends titled Assimilation is finally here and to kick things off, Respawn Entertainment has introduced a ton of back story for Season 4, we were scheduled to get “Forge” a new legend but after his apparent murde'

=== id 8400 len 467
'Our return to Paris was all but a sure thing. We had our passports. We’d spent weeks brushing up our French in Duolingo. We were even arguing about the best apartments on HomeAway.\nAnd then life happened.\nIt wasn’t one thing that '

=== id 9600 len 1668
'Unlike the state of California, John Barrowman wasted no time.\nBarrowman, star of "Doctor Who" and "Torchwood," married Scott Gill, his partner of twenty years, following the Supreme Court\'s decision to overturn the Defense of Mar'

=== id 10800 len 670
"Love those items! I've 2/4. Except my Hunter's are never that shiny. I have the original hunter green- maybe I'll google how to make them pretty again!\nYou have such a cute blog, and I love these friday picks! What is that adorabl"

=== id 12000 len 5520
'Last Monday’s announcement of Glass Enterprise Edition 2 (EE2) didn’t receive as many headlines as I had expected. That’s probably due to several different factors – the fact that it wasn’t announced at a major event, that it look'

=== id 13200 len 986
'Three men have been jailed for a scam in which they stole credit cards from hotel conference goers and then tricked victims into parting with bank details.\nAnthony Brookes, John Lawlor and Michael McConnell travelled the country t'

=== id 14400 len 369
"Sign up for the mailing list and receive access to download all of the\nThe Consultative Real Estate Agent!\nOur Subscribers Get\nSpecial deals, discounts and gifts. Plus, get regular updates on what's going on in marketing, personal"

=== id 15600 len 3495
'The second installment of our series commemorating the 40th anniversary of the Arizona schools joining the conference — we’re taking this series into basketball season, folks — highlights the greatest players in a traditional form'

[stdout]
sampled 152
=== id 0 len 414
'|Viewing Single Post From: Spoilers for the Week of February 11th|\n|Lil||Feb 1 2013, 09:58 AM|\nDon\'t care about Chloe/Taniel/Jen-Jen. Don\'t care about Sami, really, but hoping that we get some good "SAMANTHA GENE!!" Marlena Death-'

=== id 1200 len 3344
'Camtasia Studio Crack:is an amazing software at that time in the market which permits you to create many good looking videos and many other applications for entertainment purposes. With the help of this application, you can make s'

=== id 2400 len 1458
'North Shore Farmhouse | What’s New Is Old AgainNorth Shore Farmhouse | A Race to the Punch ListNorth Shore Farmhouse | House Requires Some AssemblyNorth Shore Farmhouse | Making New Look OldThe North Shore Farmhouse | 400 Years of'

=== id 3600 len 417
"Actionable Intelligence for Today's Market\nPrecious Metals Futures\nAll other Futures\nM1904Trading Members Only\nThis page is a work in progress, I have to get the site making money before we do all the big brain stuff. Hold pls.\nBu"

=== id 4800 len 1848
'Spain: Casillas (Víctor Valdés 46’); Azpilicueta, Javi Martínez, Sergio Ramos (Raúl Albiol 65‘), Jordi Alba; Busquets (Xabi Alonso 46’), Thiago; Pedro (Cazorla 82’), Fàbregas (David Silva 46’), Iniesta (Jesús Navas 65‘); Diego Cos'

=== id 6000 len 4259
'Once a month, female students pack the cozy chapel at the Holy Spirit Friary that overlooks the Franciscan University of Steubenville, Ohio.\nThese gatherings are confidential, with no one discussing who is or who isn’t among the 5'

=== id 7200 len 1353
'Season 4 of Apex Legends titled Assimilation is finally here and to kick things off, Respawn Entertainment has introduced a ton of back story for Season 4, we were scheduled to get “Forge” a new legend but after his apparent murde'

=== id 8400 len 467
'Our return to Paris was all but a sure thing. We had our passports. We’d spent weeks brushing up our French in Duolingo. We were even arguing about the best apartments on HomeAway.\nAnd then life happened.\nIt wasn’t one thing that '

=== id 9600 len 1668
'Unlike the state of California, John Barrowman wasted no time.\nBarrowman, star of "Doctor Who" and "Torchwood," married Scott Gill, his partner of twenty years, following the Supreme Court\'s decision to overturn the Defense of Mar'

=== id 10800 len 670
"Love those items! I've 2/4. Except my Hunter's are never that shiny. I have the original hunter green- maybe I'll google how to make them pretty again!\nYou have such a cute blog, and I love these friday picks! What is that adorabl"

=== id 12000 len 5520
'Last Monday’s announcement of Glass Enterprise Edition 2 (EE2) didn’t receive as many headlines as I had expected. That’s probably due to several different factors – the fact that it wasn’t announced at a major event, that it look'

=== id 13200 len 986
'Three men have been jailed for a scam in which they stole credit cards from hotel conference goers and then tricked victims into parting with bank details.\nAnthony Brookes, John Lawlor and Michael McConnell travelled the country t'

=== id 14400 len 369
"Sign up for the mailing list and receive access to download all of the\nThe Consultative Real Estate Agent!\nOur Subscribers Get\nSpecial deals, discounts and gifts. Plus, get regular updates on what's going on in marketing, personal"

=== id 15600 len 3495
'The second installment of our series commemorating the 40th anniversary of the Arizona schools joining the conference — we’re taking this series into basketball season, folks — highlights the greatest players in a traditional form'
[metadata] {"noOutputExpected": false}

--- pool marker counts (docs containing) ---
 @-@       0
 @,@       0
<p>        96
</p>       69
--- space-before-comma heuristic ---
7110
--- unlink ---
0

[stdout]
--- pool marker counts (docs containing) ---
 @-@       0
 @,@       0
<p>        96
</p>       69
--- space-before-comma heuristic ---
7110
--- unlink ---
0
[metadata] {"returnCodeInterpretation": "No matches found", "noOutputExpected": false}

segcounts {'wiki': 1696, 'qa': 359, 'news': 0, 'other': 650}
tokshares {'wiki': 0.285, 'qa': 0.25, 'news': 0.0, 'other': 0.464}
~~~ ' The title of the episode " The Tan Aquatic with Steve Zissou " is a reference to the 2004 Wes Anderson movie " The Life Aquatic with Steve Zissou " . \n'
~~~ ' India has traditionally been the dominant country at the South Asian Games . An example of this dominance is the basketball competition where Team India won three out of four tournaments to date . Th'
~~~ ' " Jersey – My Island – History – German Occupation " . British Broadcasting Corporation . Retrieved 25 March 2015 . \n'
~~~ ' Lighting War ( television documentary ) . United States : World Media Rights . 2009 . Retrieved 22 March 2015 . \n'
~~~ ' The Great Landings ( television documentary ) . France : France 2 . 2009 . Retrieved 22 March 2015 . \n'
~~~ ' Overlord ( television documentary ) . United States : World Media Rights . 2009 . Retrieved 22 March 2015 . \n'
~~~ ' 20th Century Fox has purchased the rights to the book with the intent of creating a motion picture . \n'
~~~ ' Automation maximises throughput and reduces the possibility of error or contamination during manual handling . Single devices to do the assay from beginning to end are now available . \n'

[stdout]
segcounts {'wiki': 1696, 'qa': 359, 'news': 0, 'other': 650}
tokshares {'wiki': 0.285, 'qa': 0.25, 'news': 0.0, 'other': 0.464}
~~~ ' The title of the episode " The Tan Aquatic with Steve Zissou " is a reference to the 2004 Wes Anderson movie " The Life Aquatic with Steve Zissou " . \n'
~~~ ' India has traditionally been the dominant country at the South Asian Games . An example of this dominance is the basketball competition where Team India won three out of four tournaments to date . Th'
~~~ ' " Jersey – My Island – History – German Occupation " . British Broadcasting Corporation . Retrieved 25 March 2015 . \n'
~~~ ' Lighting War ( television documentary ) . United States : World Media Rights . 2009 . Retrieved 22 March 2015 . \n'
~~~ ' The Great Landings ( television documentary ) . France : France 2 . 2009 . Retrieved 22 March 2015 . \n'
~~~ ' Overlord ( television documentary ) . United States : World Media Rights . 2009 . Retrieved 22 March 2015 . \n'
~~~ ' 20th Century Fox has purchased the rights to the book with the intent of creating a motion picture . \n'
~~~ ' Automation maximises throughput and reduces the possibility of error or contamination during manual handling . Single devices to do the assay from beginning to end are now available . \n'
[metadata] {"noOutputExpected": false}

0k  sp_comma_ratio=0.98 at= 466 p_tags=   0 nl=  315  'raphic Society his plans for an Antarctic expedition , the details of which , under the na'
  50k  sp_comma_ratio=0.98 at= 373 p_tags=   0 nl=  326  'ed her to do dance training consisting of eight hours of rehearsals a day for 14 weeks . S'
 100k  sp_comma_ratio=0.99 at= 444 p_tags=   0 nl=  312  'o Wi @-@ Fi Connection to trade , battle , and interact with other players of the games , '
 150k  sp_comma_ratio=0.98 at= 413 p_tags=   0 nl=  346  'pots are anticyclonic . Smaller anticyclones tend to be white . Vortices are thought to be'
 200k  sp_comma_ratio=0.97 at= 402 p_tags=   0 nl=  414  'vensville ; By the end of the year , I @-@ 94 / US 12 extended all the way to New Buffalo '
 250k  sp_comma_ratio=0.01 at=   0 p_tags=   0 nl= 1780  "g., 21 min) than it takes an aircraft to climb to an efficient operating altitude. VHST's "
 300k  sp_comma_ratio=0.00 at=   0 p_tags=   0 nl= 2286  '” resumes are almost always the ones with all the critical details the employer desires. I'
 350k  sp_comma_ratio=0.00 at=   0 p_tags=   0 nl= 1304  "gh we invited and welcomed the DOJ investigation, the DOJ's investigation and findings rep"
 400k  sp_comma_ratio=0.00 at=   0 p_tags=   0 nl= 1976  'Had he stayed healthy, Walker might have been able to eclipse 2,500 hits, 450 homers and 3'
 450k  sp_comma_ratio=0.01 at=   0 p_tags=   0 nl= 1292  'the popular Slurpee flavors might get sold out, and you’ll have to settle for one of those'
 500k  sp_comma_ratio=0.00 at=   0 p_tags=   0 nl=  271  'am Hemsworth, on social media. She shared a video of them both in a car, which showed Hems'
 550k  sp_comma_ratio=0.00 at=   0 p_tags=   0 nl=   36  'anded Club World Cup but did not comment on the amount involved.FIFA said on Monday that t'
 600k  sp_comma_ratio=0.00 at=   0 p_tags=   0 nl=  129  'vides and resultantly deliver poor productivity. If you too are availing the Work from Hom'
 650k  sp_comma_ratio=0.00 at=   0 p_tags=   0 nl=  661  ' the excellent corner kick taken from the right by Kesgin to bulge the right corner of the'
 700k  sp_comma_ratio=0.00 at=   0 p_tags=   0 nl=  295  's away and a loss of our history as nearly all from migrant population burns to ashes.If o'
 750k  sp_comma_ratio=0.00 at=   0 p_tags= 692 nl= 2940  'the question is, how do implemement?</p>\n<pre><code>if is_windows():\n  ...\n</code></pre>\n<'
 800k  sp_comma_ratio=0.00 at=   0 p_tags= 656 nl= 2863  'PDATE</strong></p>\n<p>More details.</p>\n<p>Maybe I should say what other attempts I made t'
 850k  sp_comma_ratio=0.01 at=   0 p_tags= 554 nl= 2999  'ge the file associations manually, but there are quite a lot of file extensions to go thro'
 900k  sp_comma_ratio=0.00 at=   0 p_tags= 664 nl= 2814  'y server (actually Squid 3.0), which is set to require NTLM authentication.</p>\n\n<p>Runnin'
 950k  sp_comma_ratio=0.01 at=   0 p_tags= 613 nl= 2904  'ere is a space to the left of the instance so I would assume that there should be an arrow'

[stdout]
   0k  sp_comma_ratio=0.98 at= 466 p_tags=   0 nl=  315  'raphic Society his plans for an Antarctic expedition , the details of which , under the na'
  50k  sp_comma_ratio=0.98 at= 373 p_tags=   0 nl=  326  'ed her to do dance training consisting of eight hours of rehearsals a day for 14 weeks . S'
 100k  sp_comma_ratio=0.99 at= 444 p_tags=   0 nl=  312  'o Wi @-@ Fi Connection to trade , battle , and interact with other players of the games , '
 150k  sp_comma_ratio=0.98 at= 413 p_tags=   0 nl=  346  'pots are anticyclonic . Smaller anticyclones tend to be white . Vortices are thought to be'
 200k  sp_comma_ratio=0.97 at= 402 p_tags=   0 nl=  414  'vensville ; By the end of the year , I @-@ 94 / US 12 extended all the way to New Buffalo '
 250k  sp_comma_ratio=0.01 at=   0 p_tags=   0 nl= 1780  "g., 21 min) than it takes an aircraft to climb to an efficient operating altitude. VHST's "
 300k  sp_comma_ratio=0.00 at=   0 p_tags=   0 nl= 2286  '” resumes are almost always the ones with all the critical details the employer desires. I'
 350k  sp_comma_ratio=0.00 at=   0 p_tags=   0 nl= 1304  "gh we invited and welcomed the DOJ investigation, the DOJ's investigation and findings rep"
 400k  sp_comma_ratio=0.00 at=   0 p_tags=   0 nl= 1976  'Had he stayed healthy, Walker might have been able to eclipse 2,500 hits, 450 homers and 3'
 450k  sp_comma_ratio=0.01 at=   0 p_tags=   0 nl= 1292  'the popular Slurpee flavors might get sold out, and you’ll have to settle for one of those'
 500k  sp_comma_ratio=0.00 at=   0 p_tags=   0 nl=  271  'am Hemsworth, on social media. She shared a video of them both in a car, which showed Hems'
 550k  sp_comma_ratio=0.00 at=   0 p_tags=   0 nl=   36  'anded Club World Cup but did not comment on the amount involved.FIFA said on Monday that t'
 600k  sp_comma_ratio=0.00 at=   0 p_tags=   0 nl=  129  'vides and resultantly deliver poor productivity. If you too are availing the Work from Hom'
 650k  sp_comma_ratio=0.00 at=   0 p_tags=   0 nl=  661  ' the excellent corner kick taken from the right by Kesgin to bulge the right corner of the'
 700k  sp_comma_ratio=0.00 at=   0 p_tags=   0 nl=  295  's away and a loss of our history as nearly all from migrant population burns to ashes.If o'
 750k  sp_comma_ratio=0.00 at=   0 p_tags= 692 nl= 2940  'the question is, how do implemement?</p>\n<pre><code>if is_windows():\n  ...\n</code></pre>\n<'
 800k  sp_comma_ratio=0.00 at=   0 p_tags= 656 nl= 2863  'PDATE</strong></p>\n<p>More details.</p>\n<p>Maybe I should say what other attempts I made t'
 850k  sp_comma_ratio=0.01 at=   0 p_tags= 554 nl= 2999  'ge the file associations manually, but there are quite a lot of file extensions to go thro'
 900k  sp_comma_ratio=0.00 at=   0 p_tags= 664 nl= 2814  'y server (actually Squid 3.0), which is set to require NTLM authentication.</p>\n\n<p>Runnin'
 950k  sp_comma_ratio=0.01 at=   0 p_tags= 613 nl= 2904  'ere is a space to the left of the instance so I would assume that there should be an arrow'
[metadata] {"noOutputExpected": false}

File created successfully at: /workspace/submission/curate.py (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/workspace/submission/curate.py", "content": "#!/usr/bin/env python3\n\"\"\"Curate a 12M-token pretraining set from a raw web pool for a BROAD high-quality\nEnglish target (equal parts encyclopedic / HQ web prose / news / technical Q&A).\n\nSTATED CRITERION\n----------------\nA document is selected iff it (a) passes format-quality gates that remove web junk\n(boilerplate/nav lists, ALLCAPS spam, symbol tables, duplicate-line spam, too-short\ndocs, non-English) and (b) ranks highly under a *register-conditional* Naive-Bayes\ndomain-match score\n\n    s_r(d) = (1/N_d) * sum_f c_f(d) * log[ p_r(f) / p_pool(f) ]\n\nwhere f ranges over target-vocabulary unigrams+bigrams, p_r is the word distribution\nof target register r (estimated from the disclosed dev target, which we decode and\nsplit into its four registers), and p_pool is the pool background distribution.\nThis is the classic importance-weight / DSIR criterion. Documents are then allocated\nround-robin across the four registers with equal token quotas, so the training\nmixture matches the target mixture instead of collapsing onto whichever register the\npool happens to contain most of. Near-duplicates are removed (exact, prefix and\n5-gram MinHash-band). Output is ordered by round-robin rank so that any prefix of\nthe list is register-balanced.\n\nReproducible: python3 curate.py  ->  /workspace/submission/selection.json\n\"\"\"\nimport json, os, re, sys, hashlib, zlib\nfrom collections import Counter\nfrom multiprocessing import Pool as MPPool\nimport numpy as np\n\nPOOL = \"/workspace/data/pool.jsonl\"\nDEV = \"/workspace/data/multi_dev.npy\"\nOUT = \"/workspace/submission/selection.json\"\nCACHE = \"/tmp/curate_cache\"\nBUDGET = 12_000_000\nOVERSHOOT = 2.2          # emit ~2.2x the budget in ids so truncation is never binding\nNREG = 4\nREG_NAMES = [\"wiki\", \"webprose\", \"news\", \"qa\"]\nWORD_CAP = 1200          # words per doc used for the match score\nNPROC = max(1, min(16, os.cpu_count() - 2))\n\nWORD_RE = re.compile(r\"[a-z0-9']+\")\n\n\ndef words_of(text, cap=WORD_CAP):\n    return WORD_RE.findall(text.lower())[:cap]\n\n\n# ---------------------------------------------------------------- target profiles\ndef target_blocks():\n    \"\"\"Decode the disclosed dev target and split it into its four equal registers.\"\"\"\n    os.makedirs(CACHE, exist_ok=True)\n    cf = os.path.join(CACHE, \"target_blocks.json\")\n    if os.path.exists(cf):\n        return json.load(open(cf))\n    from transformers import AutoTokenizer\n    tok = AutoTokenizer.from_pretrained(\"gpt2\")\n    a = np.load(DEV)\n    n = len(a) // NREG\n    blocks = [tok.decode(a[i * n:(i + 1) * n]) for i in range(NREG)]\n    json.dump(blocks, open(cf, \"w\"))\n    return blocks\n\n\ndef build_vocab(blocks):\n    \"\"\"Unigram+bigram vocabulary from the target, with per-register counts.\"\"\"\n    uni, bi = Counter(), Counter()\n    per = []\n    for b in blocks:\n        w = words_of(b, cap=10 ** 9)\n        cu = Counter(w)\n        cb = Counter(zip(w, w[1:]))\n        per.append((cu, cb))\n        uni.update(cu)\n        bi.update(cb)\n    V_uni = {w: i for i, (w, c) in enumerate(uni.most_common(60000)) if c >= 3}\n    off = len(V_uni)\n    V_bi = {b: off + i for i, (b, c) in enumerate(bi.most_common(120000)) if c >= 4}\n    F = off + len(V_bi)\n    tgt = np.zeros((NREG, F), dtype=np.float64)\n    for r, (cu, cb) in enumerate(per):\n        for w, c in cu.items():\n            j = V_uni.get(w)\n            if j is not None:\n                tgt[r, j] = c\n        for b, c in cb.items():\n            j = V_bi.get(b)\n            if j is not None:\n                tgt[r, j] = c\n    return V_uni, V_bi, tgt, F\n\n\n# ---------------------------------------------------------------- pool workers\n_G = {}\n\n\ndef _init(V_uni, V_bi, F, W):\n    _G[\"u\"], _G[\"b\"], _G[\"F\"], _G[\"W\"] = V_uni, V_bi, F, W\n\n\ndef _featurize(text):\n    \"\"\"(feature ids, counts, n_words_used) over the target vocabulary.\"\"\"\n    u, b = _G[\"u\"], _G[\"b\"]\n    w = words_of(text)\n    c = Counter()\n    ug = u.get\n    for x in w:\n        j = ug(x)\n        if j is not None:\n            c[j] += 1\n    bg = b.get\n    prev = None\n    for x in w:\n        if prev is not None:\n            j = bg((prev, x))\n            if j is not None:\n                c[j] += 1\n        prev = x\n    return c, len(w)\n\n\ndef _stats(text, wlist):\n    \"\"\"Cheap format-quality statistics over the full document.\"\"\"\n    L = len(text)\n    lines = text.split(\"\\n\")\n    nl = len(lines)\n    alpha = sum(ch.isalpha() for ch in text[:6000])\n    digit = sum(ch.isdigit() for ch in text[:6000])\n    upper = sum(ch.isupper() for ch in text[:6000])\n    head = min(L, 6000)\n    short_lines = sum(1 for ln in lines if len(ln.split()) < 4)\n    uniq_lines = len(set(ln.strip() for ln in lines if ln.strip()))\n    nonblank = sum(1 for ln in lines if ln.strip())\n    nw = max(1, len(wlist))\n    return dict(\n        nchars=L,\n        nwords=len(wlist),\n        frac_alpha=alpha / max(1, head),\n        frac_digit=digit / max(1, head),\n        frac_upper=upper / max(1, head),\n        frac_short_lines=short_lines / max(1, nl),\n        uniq_line_ratio=uniq_lines / max(1, nonblank),\n        uniq_word_ratio=len(set(wlist)) / nw,\n        mean_line_words=len(wlist) / max(1, nonblank),\n        pipe=text.count(\"|\") / max(1, L / 1000.0),\n    )\n\n\ndef _sketch(wlist, text):\n    \"\"\"Dedup keys: exact-ish body hash, prefix hash, and two 5-gram MinHash bands.\"\"\"\n    norm = \" \".join(wlist)\n    exact = zlib.crc32(norm.encode()[:20000])\n    prefix = zlib.crc32(\" \".join(wlist[:40]).encode())\n    m1 = m2 = 0xFFFFFFFF\n    for i in range(0, max(1, len(wlist) - 5), 2):\n        g = \" \".join(wlist[i:i + 5]).encode()\n        h = zlib.crc32(g)\n        if h < m1:\n            m1 = h\n        h2 = zlib.adler32(g)\n        if h2 < m2:\n            m2 = h2\n    return exact, prefix, (m1 ^ (m2 << 1)) & 0xFFFFFFFF\n\n\ndef work_count(chunk):\n    \"\"\"Pass 1: pool background feature counts (on a subsample).\"\"\"\n    acc = np.zeros(_G[\"F\"], dtype=np.float64)\n    for line in chunk:\n        t = json.loads(line)[\"text\"]\n        c, _ = _featurize(t)\n        for j, v in c.items():\n            acc[j] += v\n    return acc\n\n\ndef work_score(chunk):\n    \"\"\"Pass 2: per-doc register scores, quality stats, dedup sketches.\"\"\"\n    W = _G[\"W\"]\n    out = []\n    for line in chunk:\n        r = json.loads(line)\n        t = r[\"text\"]\n        c, nw = _featurize(t)\n        if c:\n            idx = np.fromiter(c.keys(), dtype=np.int64, count=len(c))\n            cnt = np.fromiter(c.values(), dtype=np.float64, count=len(c))\n            s = (W[:, idx] * cnt).sum(axis=1) / max(1, nw)\n            cov = cnt[idx < _G[\"off\"]].sum() / max(1, nw)\n        else:\n            s = np.zeros(NREG)\n            cov = 0.0\n        wl = words_of(t, cap=4000)\n        st = _stats(t, wl)\n        st[\"cov\"] = cov\n        st[\"id\"] = r[\"id\"]\n        st[\"scores\"] = s.tolist()\n        st[\"sk\"] = _sketch(wl, t)\n        out.append(st)\n    return out\n\n\ndef chunks(path, size, every=1):\n    buf = []\n    with open(path) as f:\n        for i, line in enumerate(f):\n            if i % every:\n                continue\n            buf.append(line)\n            if len(buf) >= size:\n                yield buf\n                buf = []\n    if buf:\n        yield buf\n\n\ndef main():\n    os.makedirs(CACHE, exist_ok=True)\n    print(f\"[1/5] target profiles ({NPROC} procs)\", flush=True)\n    blocks = target_blocks()\n    V_uni, V_bi, tgt, F = build_vocab(blocks)\n    off = len(V_uni)\n    print(f\"      vocab: {off} unigrams + {len(V_bi)} bigrams = {F}\", flush=True)\n\n    # ---- pass 1: pool background over a 1/8 subsample\n    bg_f = os.path.join(CACHE, \"bg.npy\")\n    if os.path.exists(bg_f):\n        bg = np.load(bg_f)\n    else:\n        print(\"[2/5] pool background counts\", flush=True)\n        with MPPool(NPROC, initializer=_init, initargs=(V_uni, V_bi, F, None)) as p:\n            parts = p.map(work_count, chunks(POOL, 2000, every=8))\n        bg = np.sum(parts, axis=0)\n        np.save(bg_f, bg)\n\n    # ---- log-ratio weights per register (add-1 smoothed multinomials)\n    a = 1.0\n    lp_pool = np.log((bg + a) / (bg.sum() + a * F))\n    W = np.zeros((NREG, F))\n    for r in range(NREG):\n        lp_r = np.log((tgt[r] + a) / (tgt[r].sum() + a * F))\n        W[r] = lp_r - lp_pool\n\n    # ---- pass 2: score the whole pool\n    sc_f = os.path.join(CACHE, \"docs.json\")\n    if os.path.exists(sc_f):\n        docs = json.load(open(sc_f))\n    else:\n        print(\"[3/5] scoring pool\", flush=True)\n\n        def init2(*args):\n            _init(*args)\n            _G[\"off\"] = off\n\n        with MPPool(NPROC, initializer=init2, initargs=(V_uni, V_bi, F, W)) as p:\n            parts = p.map(work_score, chunks(POOL, 1500))\n        docs = [d for part in parts for d in part]\n        json.dump(docs, open(sc_f, \"w\"))\n    print(f\"      scored {len(docs)} docs\", flush=True)\n\n    # ---- quality gates: format junk removal (register-agnostic)\n    def ok(d):\n        return (d[\"nwords\"] >= 120 and d[\"nchars\"] >= 700\n                and d[\"cov\"] >= 0.50\n                and d[\"frac_alpha\"] >= 0.68\n                and d[\"frac_digit\"] <= 0.16\n                and d[\"frac_upper\"] <= 0.13\n                and d[\"frac_short_lines\"] <= 0.45\n                and d[\"uniq_line_ratio\"] >= 0.70\n                and 0.22 <= d[\"uniq_word_ratio\"] <= 0.85\n                and d[\"mean_line_words\"] >= 8.0\n                and d[\"pipe\"] <= 3.0)\n\n    elig = [d for d in docs if ok(d)]\n    print(f\"[4/5] {len(elig)}/{len(docs)} docs pass quality gates\", flush=True)\n\n    # ---- dedup (keep the higher-scoring member of each near-dup cluster)\n    elig.sort(key=lambda d: -max(d[\"scores\"]))\n    seen_e, seen_p, seen_m, kept = set(), set(), set(), []\n    for d in elig:\n        e, pfx, mh = d[\"sk\"]\n        if e in seen_e or pfx in seen_p or mh in seen_m:\n            continue\n        seen_e.add(e); seen_p.add(pfx); seen_m.add(mh)\n        kept.append(d)\n    print(f\"      {len(kept)} after near-dup removal\", flush=True)\n\n    # ---- estimated GPT-2 token count (calibrated chars->tokens, +1 for EOS)\n    for d in kept:\n        d[\"tok\"] = 0.2745 * d[\"nchars\"] + 1.0\n\n    # ---- register-balanced round-robin allocation with equal token quotas\n    order = [sorted(range(len(kept)), key=lambda i: -kept[i][\"scores\"][r]) for r in range(NREG)]\n    quota = BUDGET * OVERSHOOT / NREG\n    ptr = [0] * NREG\n    used = [0.0] * NREG\n    taken = set()\n    sel = []\n    active = list(range(NREG))\n    while active:\n        for r in list(active):\n            while ptr[r] < len(order[r]) and order[r][ptr[r]] in taken:\n                ptr[r] += 1\n            if ptr[r] >= len(order[r]) or used[r] >= quota:\n                active.remove(r)\n                continue\n            i = order[r][ptr[r]]; ptr[r] += 1\n            taken.add(i)\n            d = kept[i]\n            used[r] += d[\"tok\"]\n            sel.append(d[\"id\"])\n    print(f\"[5/5] selected {len(sel)} docs; est tokens/register: \"\n          f\"{[int(u) for u in used]} (names {REG_NAMES})\", flush=True)\n    est = sum(kept[i][\"tok\"] for i in taken)\n    print(f\"      est total tokens {est/1e6:.1f}M (budget {BUDGET/1e6:.0f}M)\", flush=True)\n    assert len(sel) == len(set(sel))\n    json.dump(sel, open(OUT, \"w\"))\n    print(\"wrote\", OUT, flush=True)\n\n\nif __name__ == \"__main__\":\n    main()\n", "structuredPatch": [], "originalFile": null, "userModified": false}

The file /workspace/submission/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/submission/curate.py", "oldString": "def _init(V_uni, V_bi, F, W):\n    _G[\"u\"], _G[\"b\"], _G[\"F\"], _G[\"W\"] = V_uni, V_bi, F, W", "newString": "def _init(V_uni, V_bi, F, W, off=0):\n    _G[\"u\"], _G[\"b\"], _G[\"F\"], _G[\"W\"], _G[\"off\"] = V_uni, V_bi, F, W, off", "originalFile": null, "structuredPatch": [{"oldStart": 94, "oldLines": 8, "newStart": 94, "newLines": 8, "lines": [" _G = {}", " ", " ", "-def _init(V_uni, V_bi, F, W):", "-    _G[\"u\"], _G[\"b\"], _G[\"F\"], _G[\"W\"] = V_uni, V_bi, F, W", "+def _init(V_uni, V_bi, F, W, off=0):", "+    _G[\"u\"], _G[\"b\"], _G[\"F\"], _G[\"W\"], _G[\"off\"] = V_uni, V_bi, F, W, off", " ", " ", " def _featurize(text):"]}], "userModified": false, "replaceAll": false}

The file /workspace/submission/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/submission/curate.py", "oldString": "        print(\"[3/5] scoring pool\", flush=True)\n\n        def init2(*args):\n            _init(*args)\n            _G[\"off\"] = off\n\n        with MPPool(NPROC, initializer=init2, initargs=(V_uni, V_bi, F, W)) as p:", "newString": "        print(\"[3/5] scoring pool\", flush=True)\n        with MPPool(NPROC, initializer=_init, initargs=(V_uni, V_bi, F, W, off)) as p:", "originalFile": null, "structuredPatch": [{"oldStart": 247, "oldLines": 12, "newStart": 247, "newLines": 7, "lines": ["         docs = json.load(open(sc_f))", "     else:", "         print(\"[3/5] scoring pool\", flush=True)", "-", "-        def init2(*args):", "-            _init(*args)", "-            _G[\"off\"] = off", "-", "-        with MPPool(NPROC, initializer=init2, initargs=(V_uni, V_bi, F, W)) as p:", "+        with MPPool(NPROC, initializer=_init, initargs=(V_uni, V_bi, F, W, off)) as p:", "             parts = p.map(work_score, chunks(POOL, 1500))", "         docs = [d for part in parts for d in part]", "         json.dump(docs, open(sc_f, \"w\"))"]}], "userModified": false, "replaceAll": false}

started
[1]+  Done                    nohup python3 submission/curate.py > /tmp/curate.log 2>&1
[1/5] target profiles (14 procs)
      vocab: 16787 unigrams + 26160 bigrams = 42947
[2/5] pool background counts
[3/5] scoring pool
      scored 182016 docs
[4/5] 93714/182016 docs pass quality gates
      92158 after near-dup removal
[5/5] selected 24664 docs; est tokens/register: [6600693, 6601243, 6600264, 6600217] (names ['wiki', 'webprose', 'news', 'qa'])
      est total tokens 26.4M (budget 12M)
wrote /workspace/submission/selection.json
5039

[stdout]
started
[1]+  Done                    nohup python3 submission/curate.py > /tmp/curate.log 2>&1
[1/5] target profiles (14 procs)
      vocab: 16787 unigrams + 26160 bigrams = 42947
[2/5] pool background counts
[3/5] scoring pool
      scored 182016 docs
[4/5] 93714/182016 docs pass quality gates
      92158 after near-dup removal
[5/5] selected 24664 docs; est tokens/register: [6600693, 6601243, 6600264, 6600217] (names ['wiki', 'webprose', 'news', 'qa'])
      est total tokens 26.4M (budget 12M)
wrote /workspace/submission/selection.json
5039
[metadata] {"noOutputExpected": false}

sel 24664 first ids [82269, 66305, 58452, 45266, 3475, 81027, 37522, 92766]
--- id 82269 scores [1.449, 0.292, 0.413, -0.936] nw 4000
'Hill 303 massacre\n|Hill 303 massacre|\nBodies of massacre victims gathered near Waegwan, South Korea, many with their hands still bound\n|Location||Hill 303, Waegwan, South Korea|\n|Date||August 17, 1950\n|Target||U.S. Army prisoners of war|\n|Deaths||42 prisoners '
--- id 66305 scores [-0.029, 0.601, 0.304, -1.034] nw 223
'Bradford West Respect MP George Galloway has defended his controversial claim that a sex assault allegation against WikiLeaks campaigner Julian Assange amounted to no more than bad "sexual etiquette".\nMr Galloway provoked a furious response from women\'s groups'
--- id 58452 scores [0.662, 0.281, 1.3, -1.171] nw 209
'<|endoftext|>New Delhi: Five new Supreme Court judges were today sworn in, taking the top court’s strength to 28, including the Chief Justice of India. Justices Sanjay Kishan Kaul, Navin Sinha, Mohan M Shantanagoudar, Deepak Gupta and S Abdul Nazeer were admin'
--- id 45266 scores [-0.738, 0.114, -0.683, 0.942] nw 146
"'m interested in approaches that avoids code in the code behind. In my opinion, there are some cases where code must be placed in the code behind. For example: I have a grid with an undefined count of columns. Columns can't be binded. So the easiest way would "
--- id 3475 scores [0.835, 0.093, 0.184, -1.119] nw 540
'Anglo-Dutch Wars, also called Dutch Wars, Dutch Engelse Oorlogen, four 17th- and 18th-century naval conflicts between England and the Dutch Republic. The first three wars, stemming from commercial rivalry, established England’s naval might, and the last, arisi'
--- id 81027 scores [-0.048, 0.465, 0.185, -1.288] nw 359
'uador’s president has lashed out at WikiLeaks founder Julian Assange even as he says his government is working behind the scenes to help him out of the Ecuadorean embassy in London.\nLenin Moreno said in a televised interview Sunday that Assange had become “mor'
--- id 37522 scores [0.357, 0.216, 1.072, -1.047] nw 190
'Prime Minister Imran Khan on Thursday congratulated India’s premier Narendra Modi on his victory in the country\'s national elections.\n"I congratulate Prime Minister Modi on the electoral victory of BJP and allies. Look forward to working with him for peace, pr'
--- id 92766 scores [-1.107, -0.069, -0.551, 0.774] nw 723
'<|endoftext|>While working with Silverlight applications, we may face some scenarios where we may need to embed Silverlight as a component, like for e.g in Sharepoint Webpars or simple we can have the same with ASP.NET. The biggest challenge comes when we have'

[stdout]
sel 24664 first ids [82269, 66305, 58452, 45266, 3475, 81027, 37522, 92766]
--- id 82269 scores [1.449, 0.292, 0.413, -0.936] nw 4000
'Hill 303 massacre\n|Hill 303 massacre|\nBodies of massacre victims gathered near Waegwan, South Korea, many with their hands still bound\n|Location||Hill 303, Waegwan, South Korea|\n|Date||August 17, 1950\n|Target||U.S. Army prisoners of war|\n|Deaths||42 prisoners '
--- id 66305 scores [-0.029, 0.601, 0.304, -1.034] nw 223
'Bradford West Respect MP George Galloway has defended his controversial claim that a sex assault allegation against WikiLeaks campaigner Julian Assange amounted to no more than bad "sexual etiquette".\nMr Galloway provoked a furious response from women\'s groups'
--- id 58452 scores [0.662, 0.281, 1.3, -1.171] nw 209
'<|endoftext|>New Delhi: Five new Supreme Court judges were today sworn in, taking the top court’s strength to 28, including the Chief Justice of India. Justices Sanjay Kishan Kaul, Navin Sinha, Mohan M Shantanagoudar, Deepak Gupta and S Abdul Nazeer were admin'
--- id 45266 scores [-0.738, 0.114, -0.683, 0.942] nw 146
"'m interested in approaches that avoids code in the code behind. In my opinion, there are some cases where code must be placed in the code behind. For example: I have a grid with an undefined count of columns. Columns can't be binded. So the easiest way would "
--- id 3475 scores [0.835, 0.093, 0.184, -1.119] nw 540
'Anglo-Dutch Wars, also called Dutch Wars, Dutch Engelse Oorlogen, four 17th- and 18th-century naval conflicts between England and the Dutch Republic. The first three wars, stemming from commercial rivalry, established England’s naval might, and the last, arisi'
--- id 81027 scores [-0.048, 0.465, 0.185, -1.288] nw 359
'uador’s president has lashed out at WikiLeaks founder Julian Assange even as he says his government is working behind the scenes to help him out of the Ecuadorean embassy in London.\nLenin Moreno said in a televised interview Sunday that Assange had become “mor'
--- id 37522 scores [0.357, 0.216, 1.072, -1.047] nw 190
'Prime Minister Imran Khan on Thursday congratulated India’s premier Narendra Modi on his victory in the country\'s national elections.\n"I congratulate Prime Minister Modi on the electoral victory of BJP and allies. Look forward to working with him for peace, pr'
--- id 92766 scores [-1.107, -0.069, -0.551, 0.774] nw 723
'<|endoftext|>While working with Silverlight applications, we may face some scenarios where we may need to embed Silverlight as a component, like for e.g in Sharepoint Webpars or simple we can have the same with ASP.NET. The biggest challenge comes when we have'
[metadata] {"noOutputExpected": false}

total        used        free      shared  buff/cache   available
Mem:            1024           0        1023           0           0        1023
none            8.0E   86M  8.0E   1% /

[stdout]
               total        used        free      shared  buff/cache   available
Mem:            1024           0        1023           0           0        1023
none            8.0E   86M  8.0E   1% /
[metadata] {"noOutputExpected": false}

File created successfully at: /workspace/variants.py (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/workspace/variants.py", "content": "#!/usr/bin/env python3\n\"\"\"Emit ablation selections from the cached per-doc scores (see submission/curate.py).\"\"\"\nimport json, sys, random\nimport numpy as np\n\ndocs = json.load(open(\"/tmp/curate_cache/docs.json\"))\nBUDGET = 12_000_000\nOVER = 2.2\nNREG = 4\nfor d in docs:\n    d[\"tok\"] = 0.2745 * d[\"nchars\"] + 1.0\n\n\ndef gates(d, on=True):\n    if not on:\n        return d[\"nwords\"] >= 50\n    return (d[\"nwords\"] >= 120 and d[\"nchars\"] >= 700 and d[\"cov\"] >= 0.50\n            and d[\"frac_alpha\"] >= 0.68 and d[\"frac_digit\"] <= 0.16\n            and d[\"frac_upper\"] <= 0.13 and d[\"frac_short_lines\"] <= 0.45\n            and d[\"uniq_line_ratio\"] >= 0.70\n            and 0.22 <= d[\"uniq_word_ratio\"] <= 0.85\n            and d[\"mean_line_words\"] >= 8.0 and d[\"pipe\"] <= 3.0)\n\n\ndef dedup(cand):\n    se, sp, sm, kept = set(), set(), set(), []\n    for d in cand:\n        e, p, m = d[\"sk\"]\n        if e in se or p in sp or m in sm:\n            continue\n        se.add(e); sp.add(p); sm.add(m); kept.append(d)\n    return kept\n\n\ndef balanced(kept, w=(1, 1, 1, 1)):\n    order = [sorted(range(len(kept)), key=lambda i: -kept[i][\"scores\"][r]) for r in range(NREG)]\n    wsum = sum(w)\n    quota = [BUDGET * OVER * wi / wsum for wi in w]\n    ptr, used, taken, sel = [0] * NREG, [0.0] * NREG, set(), []\n    active = [r for r in range(NREG) if w[r] > 0]\n    while active:\n        for r in list(active):\n            while ptr[r] < len(order[r]) and order[r][ptr[r]] in taken:\n                ptr[r] += 1\n            if ptr[r] >= len(order[r]) or used[r] >= quota[r]:\n                active.remove(r); continue\n            i = order[r][ptr[r]]; ptr[r] += 1\n            taken.add(i); used[r] += kept[i][\"tok\"]; sel.append(kept[i][\"id\"])\n    return sel\n\n\ndef topk_global(kept):\n    ranked = sorted(kept, key=lambda d: -max(d[\"scores\"]))\n    sel, tot = [], 0.0\n    for d in ranked:\n        sel.append(d[\"id\"]); tot += d[\"tok\"]\n        if tot >= BUDGET * OVER:\n            break\n    return sel\n\n\ndef gates_only(kept, seed=0):\n    r = random.Random(seed)\n    c = list(kept); r.shuffle(c)\n    sel, tot = [], 0.0\n    for d in c:\n        sel.append(d[\"id\"]); tot += d[\"tok\"]\n        if tot >= BUDGET * OVER:\n            break\n    return sel\n\n\ndef rand_all(seed=0):\n    r = random.Random(seed)\n    c = list(docs); r.shuffle(c)\n    sel, tot = [], 0.0\n    for d in c:\n        sel.append(d[\"id\"]); tot += d[\"tok\"]\n        if tot >= BUDGET * OVER:\n            break\n    return sel\n\n\nwhich = sys.argv[1]\nkept_g = dedup(sorted([d for d in docs if gates(d)], key=lambda d: -max(d[\"scores\"])))\nif which == \"balanced\":\n    sel = balanced(kept_g)\nelif which == \"topk\":\n    sel = topk_global(kept_g)\nelif which == \"gatesonly\":\n    sel = gates_only(kept_g)\nelif which == \"random\":\n    sel = rand_all()\nelif which == \"nogates\":\n    sel = balanced(dedup(sorted([d for d in docs if gates(d, on=False)], key=lambda d: -max(d[\"scores\"]))))\nelif which.startswith(\"w\"):                      # e.g. w1-1-1-1 custom register weights\n    w = tuple(float(x) for x in which[1:].split(\"-\"))\n    sel = balanced(kept_g, w)\nelse:\n    raise SystemExit(\"unknown variant \" + which)\nout = f\"/tmp/sel_{which}.json\"\njson.dump(sel, open(out, \"w\"))\nprint(f\"{which}: {len(sel)} ids -> {out}\")\n", "structuredPatch": [], "originalFile": null, "userModified": false}

File created successfully at: /workspace/eval_batch.py (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/workspace/eval_batch.py", "content": "#!/usr/bin/env python3\n\"\"\"Pack several selections and train the frozen 30M GPT on each, concurrently.\nPacking replicates pack_selection.py exactly (ordered docs, EOS after each, cut at 12M).\"\"\"\nimport json, os, subprocess, sys, time\nimport numpy as np\n\nos.environ[\"TOKENIZERS_PARALLELISM\"] = \"true\"\nfrom transformers import AutoTokenizer\n\nBUDGET = 12_000_000\nnames = sys.argv[1:]\ntok = AutoTokenizer.from_pretrained(\"gpt2\")\nEOS = tok.eos_token_id\n\nt0 = time.time()\ntext = {}\nfor line in open(\"/workspace/data/pool.jsonl\"):\n    r = json.loads(line)\n    text[r[\"id\"]] = r[\"text\"]\nprint(f\"pool loaded {len(text)} docs in {time.time()-t0:.0f}s\", flush=True)\n\nfor nm in names:\n    p = f\"/tmp/sel_{nm}.json\" if os.path.exists(f\"/tmp/sel_{nm}.json\") else nm\n    sel = json.load(open(p))\n    # take a generous prefix (by char-estimated tokens) then tokenize in order\n    pre, est = [], 0.0\n    for i in sel:\n        pre.append(i); est += 0.2745 * len(text[i]) + 1\n        if est >= BUDGET * 1.25:\n            break\n    enc = tok([text[i] for i in pre], add_special_tokens=False)[\"input_ids\"]\n    parts, tot = [], 0\n    for ids in enc:\n        parts.append(np.asarray(ids, dtype=np.uint16))\n        parts.append(np.array([EOS], dtype=np.uint16))\n        tot += len(ids) + 1\n        if tot >= BUDGET:\n            break\n    arr = np.concatenate(parts)[:BUDGET]\n    np.save(f\"/tmp/train_{nm}.npy\", arr)\n    ndoc = sum(1 for _ in parts) // 2\n    print(f\"[{nm}] packed {len(arr)} tokens from {ndoc} docs \"\n          f\"(mean {len(arr)/max(1,ndoc):.0f} tok/doc)\", flush=True)\n\nprocs = {}\nfor nm in names:\n    cmd = [\"python3\", \"/workspace/train_nano.py\", \"--train_npy\", f\"/tmp/train_{nm}.npy\",\n           \"--heldout_npy\", \"/workspace/data/multi_dev.npy\",\n           \"--out_json\", f\"/tmp/res_{nm}.json\"]\n    procs[nm] = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)\n    print(f\"launched {nm}\", flush=True)\nres = {}\nfor nm, p in procs.items():\n    out = p.communicate()[0]\n    line = [l for l in out.splitlines() if \"HELDOUT_PPL\" in l]\n    res[nm] = float(line[0].split()[1]) if line else float(\"nan\")\n    if not line:\n        print(f\"[{nm}] FAILED:\\n{out[-1500:]}\", flush=True)\nprint(\"\\n=== RESULTS (dev PPL, lower better) ===\", flush=True)\nfor nm, v in sorted(res.items(), key=lambda kv: kv[1]):\n    print(f\"{v:9.3f}  {nm}\", flush=True)\njson.dump(res, open(f\"/tmp/results_{'_'.join(names)}.json\", \"w\"))\n", "structuredPatch": [], "originalFile": null, "userModified": false}

balanced: 24664 ids -> /tmp/sel_balanced.json
topk: 21697 ids -> /tmp/sel_topk.json
gatesonly: 24883 ids -> /tmp/sel_gatesonly.json
random: 22503 ids -> /tmp/sel_random.json
4939

[stdout]
balanced: 24664 ids -> /tmp/sel_balanced.json
topk: 21697 ids -> /tmp/sel_topk.json
gatesonly: 24883 ids -> /tmp/sel_gatesonly.json
random: 22503 ids -> /tmp/sel_random.json
4939
[metadata] {"noOutputExpected": false}

launched
pool loaded 182016 docs in 7s
Token indices sequence length is longer than the specified maximum sequence length for this model (5414 > 1024). Running this sequence through the model will result in indexing errors
[balanced] packed 12000000 tokens from 14049 docs (mean 854 tok/doc)
[topk] packed 12000000 tokens from 12645 docs (mean 949 tok/doc)
[gatesonly] packed 12000000 tokens from 13748 docs (mean 873 tok/doc)
[random] packed 12000000 tokens from 11128 docs (mean 1078 tok/doc)
launched balanced
launched topk
launched gatesonly
launched random

[stdout]
launched
pool loaded 182016 docs in 7s
Token indices sequence length is longer than the specified maximum sequence length for this model (5414 > 1024). Running this sequence through the model will result in indexing errors
[balanced] packed 12000000 tokens from 14049 docs (mean 854 tok/doc)
[topk] packed 12000000 tokens from 12645 docs (mean 949 tok/doc)
[gatesonly] packed 12000000 tokens from 13748 docs (mean 873 tok/doc)
[random] packed 12000000 tokens from 11128 docs (mean 1078 tok/doc)
launched balanced
launched topk
launched gatesonly
launched random
[metadata] {"noOutputExpected": false}

docs w/ >=10 punct: 156065
  ratio>0.1: 2768
  ratio>0.2: 942
  ratio>0.3: 467
  ratio>0.5: 154
  ratio>0.7: 96
  ratio>0.9: 33
top ids [152178 164967 101361  67363  36051 176291] [1. 1. 1. 1. 1. 1.]

[stdout]
docs w/ >=10 punct: 156065
  ratio>0.1: 2768
  ratio>0.2: 942
  ratio>0.3: 467
  ratio>0.5: 154
  ratio>0.7: 96
  ratio>0.9: 33
top ids [152178 164967 101361  67363  36051 176291] [1. 1. 1. 1. 1. 1.]
[metadata] {"noOutputExpected": false}

[balanced] packed 12000000 tokens from 14049 docs (mean 854 tok/doc)
[topk] packed 12000000 tokens from 12645 docs (mean 949 tok/doc)
[gatesonly] packed 12000000 tokens from 13748 docs (mean 873 tok/doc)
[random] packed 12000000 tokens from 11128 docs (mean 1078 tok/doc)
launched balanced
launched topk
launched gatesonly
launched random
TIME: 4809

[stdout]
[balanced] packed 12000000 tokens from 14049 docs (mean 854 tok/doc)
[topk] packed 12000000 tokens from 12645 docs (mean 949 tok/doc)
[gatesonly] packed 12000000 tokens from 13748 docs (mean 873 tok/doc)
[random] packed 12000000 tokens from 11128 docs (mean 1078 tok/doc)
launched balanced
launched topk
launched gatesonly
launched random
TIME: 4809
[metadata] {"noOutputExpected": false}

=== id 15992 len 2687
'Cornering The Market\nThere is a lot of companies out there that like the gamer exclusion .\nWhat is gamer exclusion you ask ?\nIts wen a company makes a game or buys a game and makes it Exclusive to there gaming system or only play on there phone as an app .\nWho’s doing this ?\nWell just to name a few '
=== id 28967 len 1229
'umsafar Title Song Mp3 Free Download Audio\nHere is the track list for "Humsafar Song Mp3 Free Download" we may collect and you can listen to and download.\nHumsafar mp3 song download free . Play and Listen presenting humsafar audio song in the . Sun Re Humsafar Mp3 Download title/name of Song / Music'
=== id 30799 len 2207
'HomeKitchen Cabinet Drawer Slide Kitchen Cabinet Drawer Replacement Kitchen Cabinet Drawer Repair Kitchen Drawer Replacement Kitchen Inside Replacement Kitchen Cabinet Kitchen Cabinet Kitchen Cabinet Draw Kitchen Cabinet Drawer Slide Kitchen Cabinet Drawer Replacement Kitchen Cabinet Drawer Repair K'
=== id 36051 len 1633
'<|endoftext|>monsters inc t shirt as skirt of generation that eider down takes) of the leg ministry is gotten , O word model , the freeboard skill business can appear do not the stripe that benefit other coloring of Hermes(love horse) series : The water to is the place and tiger , red color of leath'
=== id 44076 len 713
' related to "Distribution switchboards"\nFrom Electrical Installation Guide\nThis is a list of changes made recently to pages linked from a specified page (or to members of a specified category). Pages on your watchlist are bold.\n15 August 2019\n- (diff | hist) . . m List of external influences; 19:33 '

[stdout]
=== id 15992 len 2687
'Cornering The Market\nThere is a lot of companies out there that like the gamer exclusion .\nWhat is gamer exclusion you ask ?\nIts wen a company makes a game or buys a game and makes it Exclusive to there gaming system or only play on there phone as an app .\nWho’s doing this ?\nWell just to name a few '
=== id 28967 len 1229
'umsafar Title Song Mp3 Free Download Audio\nHere is the track list for "Humsafar Song Mp3 Free Download" we may collect and you can listen to and download.\nHumsafar mp3 song download free . Play and Listen presenting humsafar audio song in the . Sun Re Humsafar Mp3 Download title/name of Song / Music'
=== id 30799 len 2207
'HomeKitchen Cabinet Drawer Slide Kitchen Cabinet Drawer Replacement Kitchen Cabinet Drawer Repair Kitchen Drawer Replacement Kitchen Inside Replacement Kitchen Cabinet Kitchen Cabinet Kitchen Cabinet Draw Kitchen Cabinet Drawer Slide Kitchen Cabinet Drawer Replacement Kitchen Cabinet Drawer Repair K'
=== id 36051 len 1633
'<|endoftext|>monsters inc t shirt as skirt of generation that eider down takes) of the leg ministry is gotten , O word model , the freeboard skill business can appear do not the stripe that benefit other coloring of Hermes(love horse) series : The water to is the place and tiger , red color of leath'
=== id 44076 len 713
' related to "Distribution switchboards"\nFrom Electrical Installation Guide\nThis is a list of changes made recently to pages linked from a specified page (or to members of a specified category). Pages on your watchlist are bold.\n15 August 2019\n- (diff | hist) . . m List of external influences; 19:33 '
[metadata] {"noOutputExpected": false}

The file /workspace/variants.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/variants.py", "oldString": "def gates(d, on=True):\n    if not on:\n        return d[\"nwords\"] >= 50\n    return (d[\"nwords\"] >= 120 and d[\"nchars\"] >= 700 and d[\"cov\"] >= 0.50\n            and d[\"frac_alpha\"] >= 0.68 and d[\"frac_digit\"] <= 0.16\n            and d[\"frac_upper\"] <= 0.13 and d[\"frac_short_lines\"] <= 0.45\n            and d[\"uniq_line_ratio\"] >= 0.70\n            and 0.22 <= d[\"uniq_word_ratio\"] <= 0.85\n            and d[\"mean_line_words\"] >= 8.0 and d[\"pipe\"] <= 3.0)", "newString": "def gates(d, on=True, strict=False):\n    if not on:\n        return d[\"nwords\"] >= 50\n    if strict:\n        return (d[\"nwords\"] >= 200 and d[\"nchars\"] >= 1200 and d[\"cov\"] >= 0.58\n                and d[\"frac_alpha\"] >= 0.72 and d[\"frac_digit\"] <= 0.12\n                and d[\"frac_upper\"] <= 0.10 and d[\"frac_short_lines\"] <= 0.28\n                and d[\"uniq_line_ratio\"] >= 0.85\n                and 0.25 <= d[\"uniq_word_ratio\"] <= 0.80\n                and d[\"mean_line_words\"] >= 12.0 and d[\"pipe\"] <= 1.0)\n    return (d[\"nwords\"] >= 120 and d[\"nchars\"] >= 700 and d[\"cov\"] >= 0.50\n            and d[\"frac_alpha\"] >= 0.68 and d[\"frac_digit\"] <= 0.16\n            and d[\"frac_upper\"] <= 0.13 and d[\"frac_short_lines\"] <= 0.45\n            and d[\"uniq_line_ratio\"] >= 0.70\n            and 0.22 <= d[\"uniq_word_ratio\"] <= 0.85\n            and d[\"mean_line_words\"] >= 8.0 and d[\"pipe\"] <= 3.0)\n\n\ndef alloc_greedy(kept, w=(1, 1, 1, 1)):\n    \"\"\"Token-exact register balance: always extend whichever register is furthest\n    below its token quota, so EVERY prefix of the list matches the target mixture.\"\"\"\n    order = [sorted(range(len(kept)), key=lambda i: -kept[i][\"scores\"][r]) for r in range(NREG)]\n    wsum = sum(w)\n    frac = [wi / wsum for wi in w]\n    ptr, used, taken, sel = [0] * NREG, [0.0] * NREG, set(), []\n    cap = BUDGET * OVER\n    while True:\n        cand = [r for r in range(NREG) if frac[r] > 0 and ptr[r] < len(order[r])\n                and used[r] < cap * frac[r]]\n        if not cand:\n            break\n        r = min(cand, key=lambda r: used[r] / frac[r])\n        while ptr[r] < len(order[r]) and order[r][ptr[r]] in taken:\n            ptr[r] += 1\n        if ptr[r] >= len(order[r]):\n            continue\n        i = order[r][ptr[r]]; ptr[r] += 1\n        taken.add(i); used[r] += kept[i][\"tok\"]; sel.append(kept[i][\"id\"])\n    return sel", "originalFile": "#!/usr/bin/env python3\n\"\"\"Emit ablation selections from the cached per-doc scores (see submission/curate.py).\"\"\"\nimport json, sys, random\nimport numpy as np\n\ndocs = json.load(open(\"/tmp/curate_cache/docs.json\"))\nBUDGET = 12_000_000\nOVER = 2.2\nNREG = 4\nfor d in docs:\n    d[\"tok\"] = 0.2745 * d[\"nchars\"] + 1.0\n\n\ndef gates(d, on=True):\n    if not on:\n        return d[\"nwords\"] >= 50\n    return (d[\"nwords\"] >= 120 and d[\"nchars\"] >= 700 and d[\"cov\"] >= 0.50\n            and d[\"frac_alpha\"] >= 0.68 and d[\"frac_digit\"] <= 0.16\n            and d[\"frac_upper\"] <= 0.13 and d[\"frac_short_lines\"] <= 0.45\n            and d[\"uniq_line_ratio\"] >= 0.70\n            and 0.22 <= d[\"uniq_word_ratio\"] <= 0.85\n            and d[\"mean_line_words\"] >= 8.0 and d[\"pipe\"] <= 3.0)\n\n\ndef dedup(cand):\n    se, sp, sm, kept = set(), set(), set(), []\n    for d in cand:\n        e, p, m = d[\"sk\"]\n        if e in se or p in sp or m in sm:\n            continue\n        se.add(e); sp.add(p); sm.add(m); kept.append(d)\n    return kept\n\n\ndef balanced(kept, w=(1, 1, 1, 1)):\n    order = [sorted(range(len(kept)), key=lambda i: -kept[i][\"scores\"][r]) for r in range(NREG)]\n    wsum = sum(w)\n    quota = [BUDGET * OVER * wi / wsum for wi in w]\n    ptr, used, taken, sel = [0] * NREG, [0.0] * NREG, set(), []\n    active = [r for r in range(NREG) if w[r] > 0]\n    while active:\n        for r in list(active):\n            while ptr[r] < len(order[r]) and order[r][ptr[r]] in taken:\n                ptr[r] += 1\n            if ptr[r] >= len(order[r]) or used[r] >= quota[r]:\n                active.remove(r); continue\n            i = order[r][ptr[r]]; ptr[r] += 1\n            taken.add(i); used[r] += kept[i][\"tok\"]; sel.append(kept[i][\"id\"])\n    return sel\n\n\ndef topk_global(kept):\n    ranked = sorted(kept, key=lambda d: -max(d[\"scores\"]))\n    sel, tot = [], 0.0\n    for d in ranked:\n        sel.append(d[\"id\"]); tot += d[\"tok\"]\n        if tot >= BUDGET * OVER:\n            break\n    return sel\n\n\ndef gates_only(kept, seed=0):\n    r = random.Random(seed)\n    c = list(kept); r.shuffle(c)\n    sel, tot = [], 0.0\n    for d in c:\n        sel.append(d[\"id\"]); tot += d[\"tok\"]\n        if tot >= BUDGET * OVER:\n            break\n    return sel\n\n\ndef rand_all(seed=0):\n    r = random.Random(seed)\n    c = list(docs); r.shuffle(c)\n    sel, tot = [], 0.0\n    for d in c:\n        sel.append(d[\"id\"]); tot += d[\"tok\"]\n        if tot >= BUDGET * OVER:\n            break\n    return sel\n\n\nwhich = sys.argv[1]\nkept_g = dedup(sorted([d for d in docs if gates(d)], key=lambda d: -max(d[\"scores\"])))\nif which == \"balanced\":\n    sel = balanced(kept_g)\nelif which == \"topk\":\n    sel = topk_global(kept_g)\nelif which == \"gatesonly\":\n    sel = gates_only(kept_g)\nelif which == \"random\":\n    sel = rand_all()\nelif which == \"nogates\":\n    sel = balanced(dedup(sorted([d for d in docs if gates(d, on=False)], key=lambda d: -max(d[\"scores\"]))))\nelif which.startswith(\"w\"):                      # e.g. w1-1-1-1 custom register weights\n    w = tuple(float(x) for x in which[1:].split(\"-\"))\n    sel = balanced(kept_g, w)\nelse:\n    raise SystemExit(\"unknown variant \" + which)\nout = f\"/tmp/sel_{which}.json\"\njson.dump(sel, open(out, \"w\"))\nprint(f\"{which}: {len(sel)} ids -> {out}\")\n", "structuredPatch": [{"oldStart": 11, "oldLines": 9, "newStart": 11, "newLines": 16, "lines": ["     d[\"tok\"] = 0.2745 * d[\"nchars\"] + 1.0", " ", " ", "-def gates(d, on=True):", "+def gates(d, on=True, strict=False):", "     if not on:", "         return d[\"nwords\"] >= 50", "+    if strict:", "+        return (d[\"nwords\"] >= 200 and d[\"nchars\"] >= 1200 and d[\"cov\"] >= 0.58", "+                and d[\"frac_alpha\"] >= 0.72 and d[\"frac_digit\"] <= 0.12", "+                and d[\"frac_upper\"] <= 0.10 and d[\"frac_short_lines\"] <= 0.28", "+                and d[\"uniq_line_ratio\"] >= 0.85", "+                and 0.25 <= d[\"uniq_word_ratio\"] <= 0.80", "+                and d[\"mean_line_words\"] >= 12.0 and d[\"pipe\"] <= 1.0)", "     return (d[\"nwords\"] >= 120 and d[\"nchars\"] >= 700 and d[\"cov\"] >= 0.50", "             and d[\"frac_alpha\"] >= 0.68 and d[\"frac_digit\"] <= 0.16", "             and d[\"frac_upper\"] <= 0.13 and d[\"frac_short_lines\"] <= 0.45"]}, {"oldStart": 22, "oldLines": 6, "newStart": 29, "newLines": 29, "lines": ["             and d[\"mean_line_words\"] >= 8.0 and d[\"pipe\"] <= 3.0)", " ", " ", "+def alloc_greedy(kept, w=(1, 1, 1, 1)):", "+    \"\"\"Token-exact register balance: always extend whichever register is furthest", "+    below its token quota, so EVERY prefix of the list matches the target mixture.\"\"\"", "+    order = [sorted(range(len(kept)), key=lambda i: -kept[i][\"scores\"][r]) for r in range(NREG)]", "+    wsum = sum(w)", "+    frac = [wi / wsum for wi in w]", "+    ptr, used, taken, sel = [0] * NREG, [0.0] * NREG, set(), []", "+    cap = BUDGET * OVER", "+    while True:", "+        cand = [r for r in range(NREG) if frac[r] > 0 and ptr[r] < len(order[r])", "+                and used[r] < cap * frac[r]]", "+        if not cand:", "+            break", "+        r = min(cand, key=lambda r: used[r] / frac[r])", "+        while ptr[r] < len(order[r]) and order[r][ptr[r]] in taken:", "+            ptr[r] += 1", "+        if ptr[r] >= len(order[r]):", "+            continue", "+        i = order[r][ptr[r]]; ptr[r] += 1", "+        taken.add(i); used[r] += kept[i][\"tok\"]; sel.append(kept[i][\"id\"])", "+    return sel", "+", "+", " def dedup(cand):", "     se, sp, sm, kept = set(), set(), set(), []", "     for d in cand:"]}], "userModified": false, "replaceAll": false}

The file /workspace/variants.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/variants.py", "oldString": "elif which.startswith(\"w\"):                      # e.g. w1-1-1-1 custom register weights\n    w = tuple(float(x) for x in which[1:].split(\"-\"))\n    sel = balanced(kept_g, w)", "newString": "elif which == \"bal2\":\n    sel = alloc_greedy(kept_g)\nelif which == \"strict\":\n    kept_s = dedup(sorted([d for d in docs if gates(d, strict=True)], key=lambda d: -max(d[\"scores\"])))\n    print(\"strict pool:\", len(kept_s))\n    sel = alloc_greedy(kept_s)\nelif which.startswith(\"w\"):                      # e.g. w1-1-1-1 custom register weights\n    w = tuple(float(x) for x in which[1:].split(\"-\"))\n    sel = alloc_greedy(kept_g, w)", "originalFile": "#!/usr/bin/env python3\n\"\"\"Emit ablation selections from the cached per-doc scores (see submission/curate.py).\"\"\"\nimport json, sys, random\nimport numpy as np\n\ndocs = json.load(open(\"/tmp/curate_cache/docs.json\"))\nBUDGET = 12_000_000\nOVER = 2.2\nNREG = 4\nfor d in docs:\n    d[\"tok\"] = 0.2745 * d[\"nchars\"] + 1.0\n\n\ndef gates(d, on=True, strict=False):\n    if not on:\n        return d[\"nwords\"] >= 50\n    if strict:\n        return (d[\"nwords\"] >= 200 and d[\"nchars\"] >= 1200 and d[\"cov\"] >= 0.58\n                and d[\"frac_alpha\"] >= 0.72 and d[\"frac_digit\"] <= 0.12\n                and d[\"frac_upper\"] <= 0.10 and d[\"frac_short_lines\"] <= 0.28\n                and d[\"uniq_line_ratio\"] >= 0.85\n                and 0.25 <= d[\"uniq_word_ratio\"] <= 0.80\n                and d[\"mean_line_words\"] >= 12.0 and d[\"pipe\"] <= 1.0)\n    return (d[\"nwords\"] >= 120 and d[\"nchars\"] >= 700 and d[\"cov\"] >= 0.50\n            and d[\"frac_alpha\"] >= 0.68 and d[\"frac_digit\"] <= 0.16\n            and d[\"frac_upper\"] <= 0.13 and d[\"frac_short_lines\"] <= 0.45\n            and d[\"uniq_line_ratio\"] >= 0.70\n            and 0.22 <= d[\"uniq_word_ratio\"] <= 0.85\n            and d[\"mean_line_words\"] >= 8.0 and d[\"pipe\"] <= 3.0)\n\n\ndef alloc_greedy(kept, w=(1, 1, 1, 1)):\n    \"\"\"Token-exact register balance: always extend whichever register is furthest\n    below its token quota, so EVERY prefix of the list matches the target mixture.\"\"\"\n    order = [sorted(range(len(kept)), key=lambda i: -kept[i][\"scores\"][r]) for r in range(NREG)]\n    wsum = sum(w)\n    frac = [wi / wsum for wi in w]\n    ptr, used, taken, sel = [0] * NREG, [0.0] * NREG, set(), []\n    cap = BUDGET * OVER\n    while True:\n        cand = [r for r in range(NREG) if frac[r] > 0 and ptr[r] < len(order[r])\n                and used[r] < cap * frac[r]]\n        if not cand:\n            break\n        r = min(cand, key=lambda r: used[r] / frac[r])\n        while ptr[r] < len(order[r]) and order[r][ptr[r]] in taken:\n            ptr[r] += 1\n        if ptr[r] >= len(order[r]):\n            continue\n        i = order[r][ptr[r]]; ptr[r] += 1\n        taken.add(i); used[r] += kept[i][\"tok\"]; sel.append(kept[i][\"id\"])\n    return sel\n\n\ndef dedup(cand):\n    se, sp, sm, kept = set(), set(), set(), []\n    for d in cand:\n        e, p, m = d[\"sk\"]\n        if e in se or p in sp or m in sm:\n            continue\n        se.add(e); sp.add(p); sm.add(m); kept.append(d)\n    return kept\n\n\ndef balanced(kept, w=(1, 1, 1, 1)):\n    order = [sorted(range(len(kept)), key=lambda i: -kept[i][\"scores\"][r]) for r in range(NREG)]\n    wsum = sum(w)\n    quota = [BUDGET * OVER * wi / wsum for wi in w]\n    ptr, used, taken, sel = [0] * NREG, [0.0] * NREG, set(), []\n    active = [r for r in range(NREG) if w[r] > 0]\n    while active:\n        for r in list(active):\n            while ptr[r] < len(order[r]) and order[r][ptr[r]] in taken:\n                ptr[r] += 1\n            if ptr[r] >= len(order[r]) or used[r] >= quota[r]:\n                active.remove(r); continue\n            i = order[r][ptr[r]]; ptr[r] += 1\n            taken.add(i); used[r] += kept[i][\"tok\"]; sel.append(kept[i][\"id\"])\n    return sel\n\n\ndef topk_global(kept):\n    ranked = sorted(kept, key=lambda d: -max(d[\"scores\"]))\n    sel, tot = [], 0.0\n    for d in ranked:\n        sel.append(d[\"id\"]); tot += d[\"tok\"]\n        if tot >= BUDGET * OVER:\n            break\n    return sel\n\n\ndef gates_only(kept, seed=0):\n    r = random.Random(seed)\n    c = list(kept); r.shuffle(c)\n    sel, tot = [], 0.0\n    for d in c:\n        sel.append(d[\"id\"]); tot += d[\"tok\"]\n        if tot >= BUDGET * OVER:\n            break\n    return sel\n\n\ndef rand_all(seed=0):\n    r = random.Random(seed)\n    c = list(docs); r.shuffle(c)\n    sel, tot = [], 0.0\n    for d in c:\n        sel.append(d[\"id\"]); tot += d[\"tok\"]\n        if tot >= BUDGET * OVER:\n            break\n    return sel\n\n\nwhich = sys.argv[1]\nkept_g = dedup(sorted([d for d in docs if gates(d)], key=lambda d: -max(d[\"scores\"])))\nif which == \"balanced\":\n    sel = balanced(kept_g)\nelif which == \"topk\":\n    sel = topk_global(kept_g)\nelif which == \"gatesonly\":\n    sel = gates_only(kept_g)\nelif which == \"random\":\n    sel = rand_all()\nelif which == \"nogates\":\n    sel = balanced(dedup(sorted([d for d in docs if gates(d, on=False)], key=lambda d: -max(d[\"scores\"]))))\nelif which.startswith(\"w\"):                      # e.g. w1-1-1-1 custom register weights\n    w = tuple(float(x) for x in which[1:].split(\"-\"))\n    sel = balanced(kept_g, w)\nelse:\n    raise SystemExit(\"unknown variant \" + which)\nout = f\"/tmp/sel_{which}.json\"\njson.dump(sel, open(out, \"w\"))\nprint(f\"{which}: {len(sel)} ids -> {out}\")\n", "structuredPatch": [{"oldStart": 123, "oldLines": 9, "newStart": 123, "newLines": 15, "lines": ["     sel = rand_all()", " elif which == \"nogates\":", "     sel = balanced(dedup(sorted([d for d in docs if gates(d, on=False)], key=lambda d: -max(d[\"scores\"]))))", "+elif which == \"bal2\":", "+    sel = alloc_greedy(kept_g)", "+elif which == \"strict\":", "+    kept_s = dedup(sorted([d for d in docs if gates(d, strict=True)], key=lambda d: -max(d[\"scores\"])))", "+    print(\"strict pool:\", len(kept_s))", "+    sel = alloc_greedy(kept_s)", " elif which.startswith(\"w\"):                      # e.g. w1-1-1-1 custom register weights", "     w = tuple(float(x) for x in which[1:].split(\"-\"))", "-    sel = balanced(kept_g, w)", "+    sel = alloc_greedy(kept_g, w)", " else:", "     raise SystemExit(\"unknown variant \" + which)", " out = f\"/tmp/sel_{which}.json\""]}], "userModified": false, "replaceAll": false}

[balanced] packed 12000000 tokens from 14049 docs (mean 854 tok/doc)
[topk] packed 12000000 tokens from 12645 docs (mean 949 tok/doc)
[gatesonly] packed 12000000 tokens from 13748 docs (mean 873 tok/doc)
[random] packed 12000000 tokens from 11128 docs (mean 1078 tok/doc)
launched balanced
launched topk
launched gatesonly
launched random
TIME: 4769

[stdout]
[balanced] packed 12000000 tokens from 14049 docs (mean 854 tok/doc)
[topk] packed 12000000 tokens from 12645 docs (mean 949 tok/doc)
[gatesonly] packed 12000000 tokens from 13748 docs (mean 873 tok/doc)
[random] packed 12000000 tokens from 11128 docs (mean 1078 tok/doc)
launched balanced
launched topk
launched gatesonly
launched random
TIME: 4769
[metadata] {"noOutputExpected": false}

bal2: 24488 ids -> /tmp/sel_bal2.json
strict pool: 54746
strict: 19313 ids -> /tmp/sel_strict.json
[balanced] packed 12000000 tokens from 14049 docs (mean 854 tok/doc)
[topk] packed 12000000 tokens from 12645 docs (mean 949 tok/doc)
[gatesonly] packed 12000000 tokens from 13748 docs (mean 873 tok/doc)
[random] packed 12000000 tokens from 11128 docs (mean 1078 tok/doc)
launched balanced
launched topk
launched gatesonly
launched random

[stdout]
bal2: 24488 ids -> /tmp/sel_bal2.json
strict pool: 54746
strict: 19313 ids -> /tmp/sel_strict.json
[balanced] packed 12000000 tokens from 14049 docs (mean 854 tok/doc)
[topk] packed 12000000 tokens from 12645 docs (mean 949 tok/doc)
[gatesonly] packed 12000000 tokens from 13748 docs (mean 873 tok/doc)
[random] packed 12000000 tokens from 11128 docs (mean 1078 tok/doc)
launched balanced
launched topk
launched gatesonly
launched random
[metadata] {"noOutputExpected": false}

launched random

=== RESULTS (dev PPL, lower better) ===
  354.828  balanced
  388.574  topk
  457.499  random
  464.036  gatesonly
TIME: 4629
0 %, 4 MiB

[stdout]
launched random

=== RESULTS (dev PPL, lower better) ===
  354.828  balanced
  388.574  topk
  457.499  random
  464.036  gatesonly
TIME: 4629
0 %, 4 MiB
[metadata] {"noOutputExpected": false}

The file /workspace/variants.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/variants.py", "oldString": "elif which == \"bal2\":\n    sel = alloc_greedy(kept_g)", "newString": "elif which == \"bal2\":\n    sel = alloc_greedy(kept_g)\nelif which == \"margin\":\n    # rank within register by margin over the other registers -> register-pure quarters\n    for d in kept_g:\n        s = d[\"scores\"]\n        d[\"scores\"] = [s[r] - (sum(s) - s[r]) / (NREG - 1) for r in range(NREG)]\n    sel = alloc_greedy(kept_g)", "originalFile": "#!/usr/bin/env python3\n\"\"\"Emit ablation selections from the cached per-doc scores (see submission/curate.py).\"\"\"\nimport json, sys, random\nimport numpy as np\n\ndocs = json.load(open(\"/tmp/curate_cache/docs.json\"))\nBUDGET = 12_000_000\nOVER = 2.2\nNREG = 4\nfor d in docs:\n    d[\"tok\"] = 0.2745 * d[\"nchars\"] + 1.0\n\n\ndef gates(d, on=True, strict=False):\n    if not on:\n        return d[\"nwords\"] >= 50\n    if strict:\n        return (d[\"nwords\"] >= 200 and d[\"nchars\"] >= 1200 and d[\"cov\"] >= 0.58\n                and d[\"frac_alpha\"] >= 0.72 and d[\"frac_digit\"] <= 0.12\n                and d[\"frac_upper\"] <= 0.10 and d[\"frac_short_lines\"] <= 0.28\n                and d[\"uniq_line_ratio\"] >= 0.85\n                and 0.25 <= d[\"uniq_word_ratio\"] <= 0.80\n                and d[\"mean_line_words\"] >= 12.0 and d[\"pipe\"] <= 1.0)\n    return (d[\"nwords\"] >= 120 and d[\"nchars\"] >= 700 and d[\"cov\"] >= 0.50\n            and d[\"frac_alpha\"] >= 0.68 and d[\"frac_digit\"] <= 0.16\n            and d[\"frac_upper\"] <= 0.13 and d[\"frac_short_lines\"] <= 0.45\n            and d[\"uniq_line_ratio\"] >= 0.70\n            and 0.22 <= d[\"uniq_word_ratio\"] <= 0.85\n            and d[\"mean_line_words\"] >= 8.0 and d[\"pipe\"] <= 3.0)\n\n\ndef alloc_greedy(kept, w=(1, 1, 1, 1)):\n    \"\"\"Token-exact register balance: always extend whichever register is furthest\n    below its token quota, so EVERY prefix of the list matches the target mixture.\"\"\"\n    order = [sorted(range(len(kept)), key=lambda i: -kept[i][\"scores\"][r]) for r in range(NREG)]\n    wsum = sum(w)\n    frac = [wi / wsum for wi in w]\n    ptr, used, taken, sel = [0] * NREG, [0.0] * NREG, set(), []\n    cap = BUDGET * OVER\n    while True:\n        cand = [r for r in range(NREG) if frac[r] > 0 and ptr[r] < len(order[r])\n                and used[r] < cap * frac[r]]\n        if not cand:\n            break\n        r = min(cand, key=lambda r: used[r] / frac[r])\n        while ptr[r] < len(order[r]) and order[r][ptr[r]] in taken:\n            ptr[r] += 1\n        if ptr[r] >= len(order[r]):\n            continue\n        i = order[r][ptr[r]]; ptr[r] += 1\n        taken.add(i); used[r] += kept[i][\"tok\"]; sel.append(kept[i][\"id\"])\n    return sel\n\n\ndef dedup(cand):\n    se, sp, sm, kept = set(), set(), set(), []\n    for d in cand:\n        e, p, m = d[\"sk\"]\n        if e in se or p in sp or m in sm:\n            continue\n        se.add(e); sp.add(p); sm.add(m); kept.append(d)\n    return kept\n\n\ndef balanced(kept, w=(1, 1, 1, 1)):\n    order = [sorted(range(len(kept)), key=lambda i: -kept[i][\"scores\"][r]) for r in range(NREG)]\n    wsum = sum(w)\n    quota = [BUDGET * OVER * wi / wsum for wi in w]\n    ptr, used, taken, sel = [0] * NREG, [0.0] * NREG, set(), []\n    active = [r for r in range(NREG) if w[r] > 0]\n    while active:\n        for r in list(active):\n            while ptr[r] < len(order[r]) and order[r][ptr[r]] in taken:\n                ptr[r] += 1\n            if ptr[r] >= len(order[r]) or used[r] >= quota[r]:\n                active.remove(r); continue\n            i = order[r][ptr[r]]; ptr[r] += 1\n            taken.add(i); used[r] += kept[i][\"tok\"]; sel.append(kept[i][\"id\"])\n    return sel\n\n\ndef topk_global(kept):\n    ranked = sorted(kept, key=lambda d: -max(d[\"scores\"]))\n    sel, tot = [], 0.0\n    for d in ranked:\n        sel.append(d[\"id\"]); tot += d[\"tok\"]\n        if tot >= BUDGET * OVER:\n            break\n    return sel\n\n\ndef gates_only(kept, seed=0):\n    r = random.Random(seed)\n    c = list(kept); r.shuffle(c)\n    sel, tot = [], 0.0\n    for d in c:\n        sel.append(d[\"id\"]); tot += d[\"tok\"]\n        if tot >= BUDGET * OVER:\n            break\n    return sel\n\n\ndef rand_all(seed=0):\n    r = random.Random(seed)\n    c = list(docs); r.shuffle(c)\n    sel, tot = [], 0.0\n    for d in c:\n        sel.append(d[\"id\"]); tot += d[\"tok\"]\n        if tot >= BUDGET * OVER:\n            break\n    return sel\n\n\nwhich = sys.argv[1]\nkept_g = dedup(sorted([d for d in docs if gates(d)], key=lambda d: -max(d[\"scores\"])))\nif which == \"balanced\":\n    sel = balanced(kept_g)\nelif which == \"topk\":\n    sel = topk_global(kept_g)\nelif which == \"gatesonly\":\n    sel = gates_only(kept_g)\nelif which == \"random\":\n    sel = rand_all()\nelif which == \"nogates\":\n    sel = balanced(dedup(sorted([d for d in docs if gates(d, on=False)], key=lambda d: -max(d[\"scores\"]))))\nelif which == \"bal2\":\n    sel = alloc_greedy(kept_g)\nelif which == \"strict\":\n    kept_s = dedup(sorted([d for d in docs if gates(d, strict=True)], key=lambda d: -max(d[\"scores\"])))\n    print(\"strict pool:\", len(kept_s))\n    sel = alloc_greedy(kept_s)\nelif which.startswith(\"w\"):                      # e.g. w1-1-1-1 custom register weights\n    w = tuple(float(x) for x in which[1:].split(\"-\"))\n    sel = alloc_greedy(kept_g, w)\nelse:\n    raise SystemExit(\"unknown variant \" + which)\nout = f\"/tmp/sel_{which}.json\"\njson.dump(sel, open(out, \"w\"))\nprint(f\"{which}: {len(sel)} ids -> {out}\")\n", "structuredPatch": [{"oldStart": 125, "oldLines": 6, "newStart": 125, "newLines": 12, "lines": ["     sel = balanced(dedup(sorted([d for d in docs if gates(d, on=False)], key=lambda d: -max(d[\"scores\"]))))", " elif which == \"bal2\":", "     sel = alloc_greedy(kept_g)", "+elif which == \"margin\":", "+    # rank within register by margin over the other registers -> register-pure quarters", "+    for d in kept_g:", "+        s = d[\"scores\"]", "+        d[\"scores\"] = [s[r] - (sum(s) - s[r]) / (NREG - 1) for r in range(NREG)]", "+    sel = alloc_greedy(kept_g)", " elif which == \"strict\":", "     kept_s = dedup(sorted([d for d in docs if gates(d, strict=True)], key=lambda d: -max(d[\"scores\"])))", "     print(\"strict pool:\", len(kept_s))"]}], "userModified": false, "replaceAll": false}

The file /workspace/submission/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/submission/curate.py", "oldString": "POOL = \"/workspace/data/pool.jsonl\"\nDEV = \"/workspace/data/multi_dev.npy\"\nOUT = \"/workspace/submission/selection.json\"\nCACHE = \"/tmp/curate_cache\"", "newString": "import argparse\n_ap = argparse.ArgumentParser()\n_ap.add_argument(\"--dfmin\", type=int, default=1,\n                 help=\"min DOCUMENT frequency of a feature in the target (kills topical terms)\")\n_ap.add_argument(\"--clip\", type=float, default=0.0, help=\"clip |log-ratio| weights (0 = off)\")\n_ap.add_argument(\"--cache\", default=\"/tmp/curate_cache\")\n_ap.add_argument(\"--out\", default=\"/workspace/submission/selection.json\")\n_A = _ap.parse_args()\n\nPOOL = \"/workspace/data/pool.jsonl\"\nDEV = \"/workspace/data/multi_dev.npy\"\nOUT = _A.out\nCACHE = _A.cache", "originalFile": null, "structuredPatch": [{"oldStart": 28, "oldLines": 10, "newStart": 28, "newLines": 19, "lines": [" from multiprocessing import Pool as MPPool", " import numpy as np", " ", "+import argparse", "+_ap = argparse.ArgumentParser()", "+_ap.add_argument(\"--dfmin\", type=int, default=1,", "+                 help=\"min DOCUMENT frequency of a feature in the target (kills topical terms)\")", "+_ap.add_argument(\"--clip\", type=float, default=0.0, help=\"clip |log-ratio| weights (0 = off)\")", "+_ap.add_argument(\"--cache\", default=\"/tmp/curate_cache\")", "+_ap.add_argument(\"--out\", default=\"/workspace/submission/selection.json\")", "+_A = _ap.parse_args()", "+", " POOL = \"/workspace/data/pool.jsonl\"", " DEV = \"/workspace/data/multi_dev.npy\"", "-OUT = \"/workspace/submission/selection.json\"", "-CACHE = \"/tmp/curate_cache\"", "+OUT = _A.out", "+CACHE = _A.cache", " BUDGET = 12_000_000", " OVERSHOOT = 2.2          # emit ~2.2x the budget in ids so truncation is never binding", " NREG = 4"]}], "userModified": false, "replaceAll": false}

The file /workspace/submission/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/submission/curate.py", "oldString": "def target_blocks():\n    \"\"\"Decode the disclosed dev target and split it into its four equal registers.\"\"\"\n    os.makedirs(CACHE, exist_ok=True)\n    cf = os.path.join(CACHE, \"target_blocks.json\")\n    if os.path.exists(cf):\n        return json.load(open(cf))\n    from transformers import AutoTokenizer\n    tok = AutoTokenizer.from_pretrained(\"gpt2\")\n    a = np.load(DEV)\n    n = len(a) // NREG\n    blocks = [tok.decode(a[i * n:(i + 1) * n]) for i in range(NREG)]\n    json.dump(blocks, open(cf, \"w\"))\n    return blocks\n\n\ndef build_vocab(blocks):\n    \"\"\"Unigram+bigram vocabulary from the target, with per-register counts.\"\"\"\n    uni, bi = Counter(), Counter()\n    per = []\n    for b in blocks:\n        w = words_of(b, cap=10 ** 9)\n        cu = Counter(w)\n        cb = Counter(zip(w, w[1:]))\n        per.append((cu, cb))\n        uni.update(cu)\n        bi.update(cb)\n    V_uni = {w: i for i, (w, c) in enumerate(uni.most_common(60000)) if c >= 3}\n    off = len(V_uni)\n    V_bi = {b: off + i for i, (b, c) in enumerate(bi.most_common(120000)) if c >= 4}\n    F = off + len(V_bi)", "newString": "def target_blocks():\n    \"\"\"Decode the disclosed dev target; split into its four equal registers, and\n    into EOS-delimited segments (used for feature document frequencies).\"\"\"\n    os.makedirs(CACHE, exist_ok=True)\n    cf = os.path.join(CACHE, \"target_blocks.json\")\n    if os.path.exists(cf):\n        o = json.load(open(cf))\n        return o[\"blocks\"], o[\"segs\"]\n    from transformers import AutoTokenizer\n    tok = AutoTokenizer.from_pretrained(\"gpt2\")\n    a = np.load(DEV)\n    n = len(a) // NREG\n    blocks = [tok.decode(a[i * n:(i + 1) * n]) for i in range(NREG)]\n    eos = tok.eos_token_id\n    idx = np.where(a == eos)[0]\n    segs = [tok.decode(s) for s in np.split(a, idx) if len(s) > 30]\n    json.dump({\"blocks\": blocks, \"segs\": segs}, open(cf, \"w\"))\n    return blocks, segs\n\n\ndef build_vocab(blocks, segs):\n    \"\"\"Unigram+bigram vocabulary from the target, with per-register counts.\n    With --dfmin>1, a feature is kept only if it occurs in >= dfmin distinct target\n    segments: register/style vocabulary survives, topic-specific terms (which are\n    frequent but concentrated in one or two dev articles) are dropped, so the\n    criterion transfers to a *different* sample of the same registers.\"\"\"\n    uni, bi = Counter(), Counter()\n    per = []\n    for b in blocks:\n        w = words_of(b, cap=10 ** 9)\n        cu = Counter(w)\n        cb = Counter(zip(w, w[1:]))\n        per.append((cu, cb))\n        uni.update(cu)\n        bi.update(cb)\n    if _A.dfmin > 1:\n        dfu, dfb = Counter(), Counter()\n        for s in segs:\n            w = words_of(s, cap=10 ** 9)\n            dfu.update(set(w))\n            dfb.update(set(zip(w, w[1:])))\n        uni = Counter({w: c for w, c in uni.items() if dfu[w] >= _A.dfmin})\n        bi = Counter({b: c for b, c in bi.items() if dfb[b] >= _A.dfmin})\n    V_uni = {w: i for i, (w, c) in enumerate(uni.most_common(60000)) if c >= 3}\n    off = len(V_uni)\n    V_bi = {b: off + i for i, (b, c) in enumerate(bi.most_common(120000)) if c >= 4}\n    F = off + len(V_bi)", "originalFile": null, "structuredPatch": [{"oldStart": 57, "oldLines": 22, "newStart": 57, "newLines": 31, "lines": [" ", " # ---------------------------------------------------------------- target profiles", " def target_blocks():", "-    \"\"\"Decode the disclosed dev target and split it into its four equal registers.\"\"\"", "+    \"\"\"Decode the disclosed dev target; split into its four equal registers, and", "+    into EOS-delimited segments (used for feature document frequencies).\"\"\"", "     os.makedirs(CACHE, exist_ok=True)", "     cf = os.path.join(CACHE, \"target_blocks.json\")", "     if os.path.exists(cf):", "-        return json.load(open(cf))", "+        o = json.load(open(cf))", "+        return o[\"blocks\"], o[\"segs\"]", "     from transformers import AutoTokenizer", "     tok = AutoTokenizer.from_pretrained(\"gpt2\")", "     a = np.load(DEV)", "     n = len(a) // NREG", "     blocks = [tok.decode(a[i * n:(i + 1) * n]) for i in range(NREG)]", "-    json.dump(blocks, open(cf, \"w\"))", "-    return blocks", "+    eos = tok.eos_token_id", "+    idx = np.where(a == eos)[0]", "+    segs = [tok.decode(s) for s in np.split(a, idx) if len(s) > 30]", "+    json.dump({\"blocks\": blocks, \"segs\": segs}, open(cf, \"w\"))", "+    return blocks, segs", " ", " ", "-def build_vocab(blocks):", "-    \"\"\"Unigram+bigram vocabulary from the target, with per-register counts.\"\"\"", "+def build_vocab(blocks, segs):", "+    \"\"\"Unigram+bigram vocabulary from the target, with per-register counts.", "+    With --dfmin>1, a feature is kept only if it occurs in >= dfmin distinct target", "+    segments: register/style vocabulary survives, topic-specific terms (which are", "+    frequent but concentrated in one or two dev articles) are dropped, so the", "+    criterion transfers to a *different* sample of the same registers.\"\"\"", "     uni, bi = Counter(), Counter()", "     per = []", "     for b in blocks:"]}, {"oldStart": 82, "oldLines": 6, "newStart": 91, "newLines": 14, "lines": ["         per.append((cu, cb))", "         uni.update(cu)", "         bi.update(cb)", "+    if _A.dfmin > 1:", "+        dfu, dfb = Counter(), Counter()", "+        for s in segs:", "+            w = words_of(s, cap=10 ** 9)", "+            dfu.update(set(w))", "+            dfb.update(set(zip(w, w[1:])))", "+        uni = Counter({w: c for w, c in uni.items() if dfu[w] >= _A.dfmin})", "+        bi = Counter({b: c for b, c in bi.items() if dfb[b] >= _A.dfmin})", "     V_uni = {w: i for i, (w, c) in enumerate(uni.most_common(60000)) if c >= 3}", "     off = len(V_uni)", "     V_bi = {b: off + i for i, (b, c) in enumerate(bi.most_common(120000)) if c >= 4}"]}], "userModified": false, "replaceAll": false}

The file /workspace/submission/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/submission/curate.py", "oldString": "    blocks = target_blocks()\n    V_uni, V_bi, tgt, F = build_vocab(blocks)", "newString": "    blocks, segs = target_blocks()\n    V_uni, V_bi, tgt, F = build_vocab(blocks, segs)", "originalFile": null, "structuredPatch": [{"oldStart": 243, "oldLines": 8, "newStart": 243, "newLines": 8, "lines": [" def main():", "     os.makedirs(CACHE, exist_ok=True)", "     print(f\"[1/5] target profiles ({NPROC} procs)\", flush=True)", "-    blocks = target_blocks()", "-    V_uni, V_bi, tgt, F = build_vocab(blocks)", "+    blocks, segs = target_blocks()", "+    V_uni, V_bi, tgt, F = build_vocab(blocks, segs)", "     off = len(V_uni)", "     print(f\"      vocab: {off} unigrams + {len(V_bi)} bigrams = {F}\", flush=True)", " "]}], "userModified": false, "replaceAll": false}

The file /workspace/submission/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/submission/curate.py", "oldString": "    for r in range(NREG):\n        lp_r = np.log((tgt[r] + a) / (tgt[r].sum() + a * F))\n        W[r] = lp_r - lp_pool", "newString": "    for r in range(NREG):\n        lp_r = np.log((tgt[r] + a) / (tgt[r].sum() + a * F))\n        W[r] = lp_r - lp_pool\n    if _A.clip > 0:\n        W = np.clip(W, -_A.clip, _A.clip)", "originalFile": null, "structuredPatch": [{"oldStart": 266, "oldLines": 6, "newStart": 266, "newLines": 8, "lines": ["     for r in range(NREG):", "         lp_r = np.log((tgt[r] + a) / (tgt[r].sum() + a * F))", "         W[r] = lp_r - lp_pool", "+    if _A.clip > 0:", "+        W = np.clip(W, -_A.clip, _A.clip)", " ", "     # ---- pass 2: score the whole pool", "     sc_f = os.path.join(CACHE, \"docs.json\")"]}], "userModified": false, "replaceAll": false}