nano-gpt-data-curation

claude-opus-4-8

nano-gpt-data-curation — graded by the task's verifier against a measured baseline.

backend: modal
variant: claude claude-opus-4-8
metric: held-out multi-domain perplexity
actions: 99
duration: 1h 15m
recorded: 2026-08-04

run iddclm-nano-curation__wasW5uR

Everything claude-opus-4-8 did on this task, in order.

download JSON ↗

claude-codeclaude-opus-4-899 turns62 narration99 action99 observation37 reasoning1 final

narration

I'll start by understanding the environment, the data, and the training setup.

toolBashCheck timer, list files, peek at pool

args

{
  "command": "cat /workspace/.timer/remaining_secs 2>/dev/null; echo \"---TIMER---\"; ls -la /workspace/; echo \"---DATA---\"; ls -la /workspace/data/; echo \"---SUBMISSION---\"; ls -la /workspace/submission/ 2>/dev/null; echo \"---HEAD POOL---\"; head -c 2000 /workspace/data/pool.jsonl",
  "description": "Check timer, list files, peek at pool"
}

Bash result

5390
---TIMER---
total 14
drwxr-xr-x 1 root root  100 Aug  4 04:33 .
drwxr-xr-x 1 root root  160 Aug  4 04:33 ..
drwxr-xr-x 1 root root   60 Aug  4 04:33 .live
drwxr-xr-x 1 root root  100 Aug  4 04:33 .metrics
drwxr-xr-x 1 root root  140 Aug  4 04:33 .timer
drwxr-xr-x 1 root root   57 Aug  3 22:37 data
-rwxrwxr-x 1 root root  714 Aug  3 22:34 dev_eval.sh
-rw-rw-r-- 1 root root 5706 Aug  3 22:34 model.py
-rw-rw-r-- 1 root root  869 Aug  3 22:34 pack_selection.py
drwxr-xr-x 1 root root   10 Aug  3 22:37 submission
-rwxrwxr-x 1 root root 1509 Aug  3 22:34 timer.sh
-rw-rw-r-- 1 root root 2962 Aug  3 22:34 train_nano.py
---DATA---
total 792657
drwxr-xr-x 1 root root        57 Aug  3 22:37 .
drwxr-xr-x 1 root root       100 Aug  4 04:33 ..
-r--r--r-- 1 root root   2000128 Aug  3 22:34 multi_dev.npy
-r--r--r-- 1 root root 809679384 Aug  3 22:35 pool.jsonl
---SUBMISSION---
total 1
drwxr-xr-x 1 root root  10 Aug  3 22:37 .
drwxr-xr-x 1 root root 100 Aug  4 04:33 ..
---HEAD POOL---
{"id": 0, "text": "|Viewing Single Post From: Spoilers for the Week of February 11th|\n|Lil||Feb 1 2013, 09:58 AM|\nDon't care about Chloe/Taniel/Jen-Jen. Don't care about Sami, really, but hoping that we get some good \"SAMANTHA GENE!!\" Marlena Death-Stares out of it. And \"newfound\" feelings. Please. If only.\nSTEFANO!! STEFANO, STEFANO, STEFANO!!!! :cheer:\n|Spoilers for the Week of February 11th \u00b7 DAYS: News, Spoilers & Discussion|"}
{"id": 1, "text": "*sigh* Fundamentalist community, let me pass on some advice to you I learned from the atheistic community:\nIf you have set yourself on fire, do not run.\nOkay? Okay?? Please?\nLook, D, you had two months to say to Harvard in private emails, \"Im sorry, I shouldnt have been using that animation in my paid presentations. I wont use it again. I really do like 'Inner Life', though, and would love to use it in classroom presentations, from the BioVisions site, if that is acceptable.\"\nI sat here, for two months, waiting for that to happen, anything to happen, and it didnt. Two months, on your own terms, you could have written a similar post to yesterdays. I would have given you the benefit of the doubt-- maybe you didnt know the credits werent visible to the audience, and I wouldnt have said a word beyond this, as its Harvards problem, not mine. This would have been a funny joke to those of us involved in dealing with you people, but it would have been a PR non-issue for you.\nBut after you set yourself on fire, you didnt douse it out with a bucket of ice cold reality and accountability. You ran. And youre still running.\nWhy not just state \"I screwed up. Sorry everyone.\" and move on? Why the excuses? Why the denial? Why the passive language? Why the vague words and cryptic capitalizations? Why the writes and rewrites of your 'press release'? We know it wasnt written of your own volition, or it would have been done *before* Harvard had to take action. And, your behavior before this, regarding this issue, is

[stdout]
5390
---TIMER---
total 14
drwxr-xr-x 1 root root  100 Aug  4 04:33 .
drwxr-xr-x 1 root root  160 Aug  4 04:33 ..
drwxr-xr-x 1 root root   60 Aug  4 04:33 .live
drwxr-xr-x 1 root root  100 Aug  4 04:33 .metrics
drwxr-xr-x 1 root root  140 Aug  4 04:33 .timer
drwxr-xr-x 1 root root   57 Aug  3 22:37 data
-rwxrwxr-x 1 root root  714 Aug  3 22:34 dev_eval.sh
-rw-rw-r-- 1 root root 5706 Aug  3 22:34 model.py
-rw-rw-r-- 1 root root  869 Aug  3 22:34 pack_selection.py
drwxr-xr-x 1 root root   10 Aug  3 22:37 submission
-rwxrwxr-x 1 root root 1509 Aug  3 22:34 timer.sh
-rw-rw-r-- 1 root root 2962 Aug  3 22:34 train_nano.py
---DATA---
total 792657
drwxr-xr-x 1 root root        57 Aug  3 22:37 .
drwxr-xr-x 1 root root       100 Aug  4 04:33 ..
-r--r--r-- 1 root root   2000128 Aug  3 22:34 multi_dev.npy
-r--r--r-- 1 root root 809679384 Aug  3 22:35 pool.jsonl
---SUBMISSION---
total 1
drwxr-xr-x 1 root root  10 Aug  3 22:37 .
drwxr-xr-x 1 root root 100 Aug  4 04:33 ..
---HEAD POOL---
{"id": 0, "text": "|Viewing Single Post From: Spoilers for the Week of February 11th|\n|Lil||Feb 1 2013, 09:58 AM|\nDon't care about Chloe/Taniel/Jen-Jen. Don't care about Sami, really, but hoping that we get some good \"SAMANTHA GENE!!\" Marlena Death-Stares out of it. And \"newfound\" feelings. Please. If only.\nSTEFANO!! STEFANO, STEFANO, STEFANO!!!! :cheer:\n|Spoilers for the Week of February 11th \u00b7 DAYS: News, Spoilers & Discussion|"}
{"id": 1, "text": "*sigh* Fundamentalist community, let me pass on some advice to you I learned from the atheistic community:\nIf you have set yourself on fire, do not run.\nOkay? Okay?? Please?\nLook, D, you had two months to say to Harvard in private emails, \"Im sorry, I shouldnt have been using that animation in my paid presentations. I wont use it again. I really do like 'Inner Life', though, and would love to use it in classroom presentations, from the BioVisions site, if that is acceptable.\"\nI sat here, for two months, waiting for that to happen, anything to happen, and it didnt. Two months, on your own terms, you could have written a similar post to yesterdays. I would have given you the benefit of the doubt-- maybe you didnt know the credits werent visible to the audience, and I wouldnt have said a word beyond this, as its Harvards problem, not mine. This would have been a funny joke to those of us involved in dealing with you people, but it would have been a PR non-issue for you.\nBut after you set yourself on fire, you didnt douse it out with a bucket of ice cold reality and accountability. You ran. And youre still running.\nWhy not just state \"I screwed up. Sorry everyone.\" and move on? Why the excuses? Why the denial? Why the passive language? Why the vague words and cryptic capitalizations? Why the writes and rewrites of your 'press release'? We know it wasnt written of your own volition, or it would have been done *before* Harvard had to take action. And, your behavior before this, regarding this issue, is
[metadata] {"noOutputExpected": false}

narration

I have ~90 minutes of wall-clock budget. Let me examine the training pipeline and supporting scripts.

toolBashRead training and packing scripts

args

{
  "command": "echo \"===== train_nano.py =====\"; cat /workspace/train_nano.py; echo; echo \"===== pack_selection.py =====\"; cat /workspace/pack_selection.py; echo; echo \"===== dev_eval.sh =====\"; cat /workspace/dev_eval.sh; echo; echo \"===== timer.sh =====\"; cat /workspace/timer.sh",
  "description": "Read training and packing scripts"
}

Bash result

===== train_nano.py =====
"""From-scratch nano-GPT (30M) LM training on a 1D token .npy; report held-out perplexity.
Frozen recipe — the ONLY thing that varies between runs is the training token set."""
import argparse, math, time, numpy as np, torch
from model import GPT, GPTConfig

ap = argparse.ArgumentParser()
ap.add_argument("--train_npy", required=True)
ap.add_argument("--heldout_npy", required=True)
ap.add_argument("--out_json", required=True)
ap.add_argument("--max_iters", type=int, default=3000)
ap.add_argument("--warmup", type=int, default=150)
ap.add_argument("--batch", type=int, default=32)
ap.add_argument("--block", type=int, default=256)
ap.add_argument("--lr", type=float, default=6e-4)
ap.add_argument("--seed", type=int, default=1337)
a = ap.parse_args()

torch.manual_seed(a.seed); np.random.seed(a.seed)
dev = "cuda"
tr = np.load(a.train_npy); ho = np.load(a.heldout_npy)
tr = torch.from_numpy(tr.astype(np.int64)); ho = torch.from_numpy(ho.astype(np.int64))
rng = np.random.default_rng(a.seed)

model = GPT(GPTConfig(block_size=a.block, vocab_size=50257, n_layer=6, n_head=6,
                      n_embd=384, dropout=0.0, bias=False)).to(dev)
opt = model.configure_optimizers(0.1, a.lr, (0.9, 0.95), "cuda")

def lr_at(it):
    if it < a.warmup: return a.lr * (it + 1) / (a.warmup + 1)
    r = (it - a.warmup) / max(1, a.max_iters - a.warmup)
    return 0.1 * a.lr + 0.5 * (1 + math.cos(math.pi * r)) * (a.lr - 0.1 * a.lr)

def get_batch(src):
    ix = rng.integers(0, len(src) - a.block - 1, size=a.batch)
    x = torch.stack([src[i:i+a.block] for i in ix]).to(dev)
    y = torch.stack([src[i+1:i+1+a.block] for i in ix]).to(dev)
    return x, y

t0 = time.time()
model.train()
for it in range(a.max_iters):
    for g in opt.param_groups: g["lr"] = lr_at(it)
    x, y = get_batch(tr)
    with torch.autocast("cuda", dtype=torch.bfloat16):
        _, loss = model(x, y)
    opt.zero_grad(set_to_none=True); loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0); opt.step()
train_s = time.time() - t0

# held-out perplexity: mean next-token CE over non-overlapping windows (window-weighted)
import json
model.eval()
nwin = (len(ho) - 1) // a.block
starts = [j * a.block for j in range(nwin)]
sum_loss, nseen = 0.0, 0
with torch.no_grad():
    for i in range(0, nwin, a.batch):
        bs = starts[i:i+a.batch]
        x = torch.stack([ho[s:s+a.block] for s in bs]).to(dev)
        y = torch.stack([ho[s+1:s+1+a.block] for s in bs]).to(dev)
        with torch.autocast("cuda", dtype=torch.bfloat16):
            _, l = model(x, y)
        sum_loss += l.item() * len(bs); nseen += len(bs)
mean_loss = sum_loss / max(1, nseen)
ppl = math.exp(mean_loss)
out = {"train_npy": a.train_npy, "heldout_ppl": ppl, "mean_loss": mean_loss,
       "max_iters": a.max_iters, "seed": a.seed, "train_s": round(train_s, 1)}
json.dump(out, open(a.out_json, "w"))
print(f"HELDOUT_PPL {ppl:.3f}  loss {mean_loss:.4f}  train_s {train_s:.0f}  {a.train_npy}")

===== pack_selection.py =====
"""Solver-side: pack a selection.json (ordered ids) into a 12M-token train array from pool.jsonl."""
import json, sys, numpy as np
from transformers import AutoTokenizer

POOL = "/workspace/data/pool.jsonl"
SEL = sys.argv[1] if len(sys.argv) > 1 else "/workspace/submission/selection.json"
OUT = sys.argv[2] if len(sys.argv) > 2 else "/tmp/dev_train.npy"
BUDGET = 12_000_000

tok = AutoTokenizer.from_pretrained("gpt2"); EOS = tok.eos_token_id
text = {}
for line in open(POOL):
    r = json.loads(line); text[r["id"]] = r["text"]
sel = json.load(open(SEL))
parts, tot = [], 0
for i in sel:
    ids = tok(text[i], add_special_tokens=False).input_ids
    parts.extend(ids); parts.append(EOS); tot += len(ids) + 1
    if tot >= BUDGET: break
arr = np.array(parts[:BUDGET], dtype=np.uint16)
np.save(OUT, arr)
print(f"packed {len(arr)} tokens from {len(sel)} ids -> {OUT}")

===== dev_eval.sh =====
#!/bin/bash
# Diagnostic: train the frozen 30M GPT on your selection and report DEV perplexity.
# Usage: bash dev_eval.sh [selection.json]   (default /workspace/submission/selection.json)
# NOTE: the official target is a DIFFERENT (hidden) sample from the same HQ domain.
set -euo pipefail
export TOKENIZERS_PARALLELISM=false
SEL="${1:-/workspace/submission/selection.json}"
cd /workspace
python3 pack_selection.py "$SEL" /tmp/dev_train.npy
python3 train_nano.py --train_npy /tmp/dev_train.npy --heldout_npy /workspace/data/multi_dev.npy \
    --out_json /tmp/dev.json --max_iters 3000 --seed 1337 2>&1 | grep HELDOUT_PPL
echo "(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)"

===== timer.sh =====
#!/usr/bin/env bash
# Budget timer daemon — writes /workspace/.timer/remaining_secs every 10s so the
# solver can poll time left instead of doing wall-clock arithmetic.
#
# Budget (seconds) comes from $TASK_BUDGET_SECS (set by the runner to the agent
# timeout). Idempotent via a PID file. On ScaleTrain the trial runner starts its
# own timer; this file is the fallback for the Modal/local harbor backends, which
# bootstrap /workspace/timer.sh on environment start. The fallback default below
# (7200 = 2 h) matches this task's [agent] timeout_sec and is only used if
# $TASK_BUDGET_SECS is unset.
set -u
TIMER_DIR="/workspace/.timer"
PID_FILE="$TIMER_DIR/timer.pid"
mkdir -p "$TIMER_DIR"

if [ -s "$PID_FILE" ] && kill -0 "$(cat "$PID_FILE" 2>/dev/null)" 2>/dev/null; then
    exit 0   # already running
fi
echo $$ > "$PID_FILE"
trap 'rm -f "$PID_FILE"' EXIT INT TERM

START=$(date +%s)
BUDGET="${TASK_BUDGET_SECS:-7200}"
echo "$START" > "$TIMER_DIR/start_epoch"
echo "$BUDGET" > "$TIMER_DIR/budget_secs"

while true; do
    NOW=$(date +%s)
    ELAPSED=$((NOW - START))
    REMAINING=$((BUDGET - ELAPSED))
    [ "$REMAINING" -lt 0 ] && REMAINING=0
    echo "$REMAINING" > "$TIMER_DIR/remaining_secs"
    echo "$ELAPSED" > "$TIMER_DIR/elapsed_secs"
    [ "$REMAINING" -le 1800 ] && [ ! -f "$TIMER_DIR/alert_30min" ] && touch "$TIMER_DIR/alert_30min"
    [ "$REMAINING" -le 600 ]  && [ ! -f "$TIMER_DIR/alert_10min" ] && touch "$TIMER_DIR/alert_10min"
    [ "$REMAINING" -le 0 ] && break
    sleep 10
done

[stdout]
===== train_nano.py =====
"""From-scratch nano-GPT (30M) LM training on a 1D token .npy; report held-out perplexity.
Frozen recipe — the ONLY thing that varies between runs is the training token set."""
import argparse, math, time, numpy as np, torch
from model import GPT, GPTConfig

ap = argparse.ArgumentParser()
ap.add_argument("--train_npy", required=True)
ap.add_argument("--heldout_npy", required=True)
ap.add_argument("--out_json", required=True)
ap.add_argument("--max_iters", type=int, default=3000)
ap.add_argument("--warmup", type=int, default=150)
ap.add_argument("--batch", type=int, default=32)
ap.add_argument("--block", type=int, default=256)
ap.add_argument("--lr", type=float, default=6e-4)
ap.add_argument("--seed", type=int, default=1337)
a = ap.parse_args()

torch.manual_seed(a.seed); np.random.seed(a.seed)
dev = "cuda"
tr = np.load(a.train_npy); ho = np.load(a.heldout_npy)
tr = torch.from_numpy(tr.astype(np.int64)); ho = torch.from_numpy(ho.astype(np.int64))
rng = np.random.default_rng(a.seed)

model = GPT(GPTConfig(block_size=a.block, vocab_size=50257, n_layer=6, n_head=6,
                      n_embd=384, dropout=0.0, bias=False)).to(dev)
opt = model.configure_optimizers(0.1, a.lr, (0.9, 0.95), "cuda")

def lr_at(it):
    if it < a.warmup: return a.lr * (it + 1) / (a.warmup + 1)
    r = (it - a.warmup) / max(1, a.max_iters - a.warmup)
    return 0.1 * a.lr + 0.5 * (1 + math.cos(math.pi * r)) * (a.lr - 0.1 * a.lr)

def get_batch(src):
    ix = rng.integers(0, len(src) - a.block - 1, size=a.batch)
    x = torch.stack([src[i:i+a.block] for i in ix]).to(dev)
    y = torch.stack([src[i+1:i+1+a.block] for i in ix]).to(dev)
    return x, y

t0 = time.time()
model.train()
for it in range(a.max_iters):
    for g in opt.param_groups: g["lr"] = lr_at(it)
    x, y = get_batch(tr)
    with torch.autocast("cuda", dtype=torch.bfloat16):
        _, loss = model(x, y)
    opt.zero_grad(set_to_none=True); loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0); opt.step()
train_s = time.time() - t0

# held-out perplexity: mean next-token CE over non-overlapping windows (window-weighted)
import json
model.eval()
nwin = (len(ho) - 1) // a.block
starts = [j * a.block for j in range(nwin)]
sum_loss, nseen = 0.0, 0
with torch.no_grad():
    for i in range(0, nwin, a.batch):
        bs = starts[i:i+a.batch]
        x = torch.stack([ho[s:s+a.block] for s in bs]).to(dev)
        y = torch.stack([ho[s+1:s+1+a.block] for s in bs]).to(dev)
        with torch.autocast("cuda", dtype=torch.bfloat16):
            _, l = model(x, y)
        sum_loss += l.item() * len(bs); nseen += len(bs)
mean_loss = sum_loss / max(1, nseen)
ppl = math.exp(mean_loss)
out = {"train_npy": a.train_npy, "heldout_ppl": ppl, "mean_loss": mean_loss,
       "max_iters": a.max_iters, "seed": a.seed, "train_s": round(train_s, 1)}
json.dump(out, open(a.out_json, "w"))
print(f"HELDOUT_PPL {ppl:.3f}  loss {mean_loss:.4f}  train_s {train_s:.0f}  {a.train_npy}")

===== pack_selection.py =====
"""Solver-side: pack a selection.json (ordered ids) into a 12M-token train array from pool.jsonl."""
import json, sys, numpy as np
from transformers import AutoTokenizer

POOL = "/workspace/data/pool.jsonl"
SEL = sys.argv[1] if len(sys.argv) > 1 else "/workspace/submission/selection.json"
OUT = sys.argv[2] if len(sys.argv) > 2 else "/tmp/dev_train.npy"
BUDGET = 12_000_000

tok = AutoTokenizer.from_pretrained("gpt2"); EOS = tok.eos_token_id
text = {}
for line in open(POOL):
    r = json.loads(line); text[r["id"]] = r["text"]
sel = json.load(open(SEL))
parts, tot = [], 0
for i in sel:
    ids = tok(text[i], add_special_tokens=False).input_ids
    parts.extend(ids); parts.append(EOS); tot += len(ids) + 1
    if tot >= BUDGET: break
arr = np.array(parts[:BUDGET], dtype=np.uint16)
np.save(OUT, arr)
print(f"packed {len(arr)} tokens from {len(sel)} ids -> {OUT}")

===== dev_eval.sh =====
#!/bin/bash
# Diagnostic: train the frozen 30M GPT on your selection and report DEV perplexity.
# Usage: bash dev_eval.sh [selection.json]   (default /workspace/submission/selection.json)
# NOTE: the official target is a DIFFERENT (hidden) sample from the same HQ domain.
set -euo pipefail
export TOKENIZERS_PARALLELISM=false
SEL="${1:-/workspace/submission/selection.json}"
cd /workspace
python3 pack_selection.py "$SEL" /tmp/dev_train.npy
python3 train_nano.py --train_npy /tmp/dev_train.npy --heldout_npy /workspace/data/multi_dev.npy \
    --out_json /tmp/dev.json --max_iters 3000 --seed 1337 2>&1 | grep HELDOUT_PPL
echo "(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)"

===== timer.sh =====
#!/usr/bin/env bash
# Budget timer daemon — writes /workspace/.timer/remaining_secs every 10s so the
# solver can poll time left instead of doing wall-clock arithmetic.
#
# Budget (seconds) comes from $TASK_BUDGET_SECS (set by the runner to the agent
# timeout). Idempotent via a PID file. On ScaleTrain the trial runner starts its
# own timer; this file is the fallback for the Modal/local harbor backends, which
# bootstrap /workspace/timer.sh on environment start. The fallback default below
# (7200 = 2 h) matches this task's [agent] timeout_sec and is only used if
# $TASK_BUDGET_SECS is unset.
set -u
TIMER_DIR="/workspace/.timer"
PID_FILE="$TIMER_DIR/timer.pid"
mkdir -p "$TIMER_DIR"

if [ -s "$PID_FILE" ] && kill -0 "$(cat "$PID_FILE" 2>/dev/null)" 2>/dev/null; then
    exit 0   # already running
fi
echo $$ > "$PID_FILE"
trap 'rm -f "$PID_FILE"' EXIT INT TERM

START=$(date +%s)
BUDGET="${TASK_BUDGET_SECS:-7200}"
echo "$START" > "$TIMER_DIR/start_epoch"
echo "$BUDGET" > "$TIMER_DIR/budget_secs"

while true; do
    NOW=$(date +%s)
    ELAPSED=$((NOW - START))
    REMAINING=$((BUDGET - ELAPSED))
    [ "$REMAINING" -lt 0 ] && REMAINING=0
    echo "$REMAINING" > "$TIMER_DIR/remaining_secs"
    echo "$ELAPSED" > "$TIMER_DIR/elapsed_secs"
    [ "$REMAINING" -le 1800 ] && [ ! -f "$TIMER_DIR/alert_30min" ] && touch "$TIMER_DIR/alert_30min"
    [ "$REMAINING" -le 600 ]  && [ ! -f "$TIMER_DIR/alert_10min" ] && touch "$TIMER_DIR/alert_10min"
    [ "$REMAINING" -le 0 ] && break
    sleep 10
done
[metadata] {"noOutputExpected": false}

narration

Now let me check the GPU, the model config, and decode the dev target to understand what we're optimizing for.

toolBashCheck model, GPU, dev npy shape, pool count

args

{
  "command": "cd /workspace\necho \"===== model.py =====\"; sed -n '1,60p' model.py\necho \"===== GPU =====\"; nvidia-smi --query-gpu=name,memory.total,memory.used --format=csv 2>/dev/null || echo \"no nvidia-smi\"\necho \"===== dev npy =====\"; python3 -c \"\nimport numpy as np\na = np.load('/workspace/data/multi_dev.npy')\nprint('shape', a.shape, 'dtype', a.dtype, 'min', a.min(), 'max', a.max())\nprint('n_tokens', a.size)\n\"\necho \"===== pool line count =====\"; wc -l /workspace/data/pool.jsonl",
  "description": "Check model, GPU, dev npy shape, pool count"
}

Bash result

===== model.py =====
"""A small GPT (decoder-only Transformer) used as the distillation *student*.

Self-contained re-implementation in this repo's own idiom (no external GPT code).
GPT-2 byte-level BPE vocabulary (50257) so the student shares the teacher's
tokenizer. Standard pre-norm Transformer with weight-tied embeddings.
"""
from dataclasses import dataclass
import math
import torch
import torch.nn as nn
import torch.nn.functional as F


@dataclass
class GPTConfig:
    block_size: int = 256
    vocab_size: int = 50257
    n_layer: int = 6
    n_head: int = 6
    n_embd: int = 384
    dropout: float = 0.0
    bias: bool = False


class LayerNorm(nn.Module):
    def __init__(self, ndim, bias):
        super().__init__()
        self.weight = nn.Parameter(torch.ones(ndim))
        self.bias = nn.Parameter(torch.zeros(ndim)) if bias else None

    def forward(self, x):
        return F.layer_norm(x, self.weight.shape, self.weight, self.bias, 1e-5)


class CausalSelfAttention(nn.Module):
    def __init__(self, cfg: GPTConfig):
        super().__init__()
        assert cfg.n_embd % cfg.n_head == 0
        self.c_attn = nn.Linear(cfg.n_embd, 3 * cfg.n_embd, bias=cfg.bias)
        self.c_proj = nn.Linear(cfg.n_embd, cfg.n_embd, bias=cfg.bias)
        self.n_head = cfg.n_head
        self.n_embd = cfg.n_embd
        self.dropout = cfg.dropout

    def forward(self, x):
        B, T, C = x.size()
        q, k, v = self.c_attn(x).split(self.n_embd, dim=2)
        q = q.view(B, T, self.n_head, C // self.n_head).transpose(1, 2)
        k = k.view(B, T, self.n_head, C // self.n_head).transpose(1, 2)
        v = v.view(B, T, self.n_head, C // self.n_head).transpose(1, 2)
        y = F.scaled_dot_product_attention(
            q, k, v, dropout_p=self.dropout if self.training else 0.0, is_causal=True
        )
        y = y.transpose(1, 2).contiguous().view(B, T, C)
        return self.c_proj(y)


class MLP(nn.Module):
    def __init__(self, cfg: GPTConfig):
        super().__init__()
===== GPU =====
name, memory.total [MiB], memory.used [MiB]
NVIDIA H100 80GB HBM3, 81559 MiB, 4 MiB
===== dev npy =====
shape (1000000,) dtype uint16 min 0 max 50256
n_tokens 1000000
===== pool line count =====
182016 /workspace/data/pool.jsonl

[stdout]
===== model.py =====
"""A small GPT (decoder-only Transformer) used as the distillation *student*.

Self-contained re-implementation in this repo's own idiom (no external GPT code).
GPT-2 byte-level BPE vocabulary (50257) so the student shares the teacher's
tokenizer. Standard pre-norm Transformer with weight-tied embeddings.
"""
from dataclasses import dataclass
import math
import torch
import torch.nn as nn
import torch.nn.functional as F


@dataclass
class GPTConfig:
    block_size: int = 256
    vocab_size: int = 50257
    n_layer: int = 6
    n_head: int = 6
    n_embd: int = 384
    dropout: float = 0.0
    bias: bool = False


class LayerNorm(nn.Module):
    def __init__(self, ndim, bias):
        super().__init__()
        self.weight = nn.Parameter(torch.ones(ndim))
        self.bias = nn.Parameter(torch.zeros(ndim)) if bias else None

    def forward(self, x):
        return F.layer_norm(x, self.weight.shape, self.weight, self.bias, 1e-5)


class CausalSelfAttention(nn.Module):
    def __init__(self, cfg: GPTConfig):
        super().__init__()
        assert cfg.n_embd % cfg.n_head == 0
        self.c_attn = nn.Linear(cfg.n_embd, 3 * cfg.n_embd, bias=cfg.bias)
        self.c_proj = nn.Linear(cfg.n_embd, cfg.n_embd, bias=cfg.bias)
        self.n_head = cfg.n_head
        self.n_embd = cfg.n_embd
        self.dropout = cfg.dropout

    def forward(self, x):
        B, T, C = x.size()
        q, k, v = self.c_attn(x).split(self.n_embd, dim=2)
        q = q.view(B, T, self.n_head, C // self.n_head).transpose(1, 2)
        k = k.view(B, T, self.n_head, C // self.n_head).transpose(1, 2)
        v = v.view(B, T, self.n_head, C // self.n_head).transpose(1, 2)
        y = F.scaled_dot_product_attention(
            q, k, v, dropout_p=self.dropout if self.training else 0.0, is_causal=True
        )
        y = y.transpose(1, 2).contiguous().view(B, T, C)
        return self.c_proj(y)


class MLP(nn.Module):
    def __init__(self, cfg: GPTConfig):
        super().__init__()
===== GPU =====
name, memory.total [MiB], memory.used [MiB]
NVIDIA H100 80GB HBM3, 81559 MiB, 4 MiB
===== dev npy =====
shape (1000000,) dtype uint16 min 0 max 50256
n_tokens 1000000
===== pool line count =====
182016 /workspace/data/pool.jsonl
[metadata] {"noOutputExpected": false}

narration

H100 GPU, 1M-token dev target. Let me decode the dev target to see the domains I'm optimizing for, and profile the pool.

toolBashDecode samples across dev target

args

{
  "command": "cd /workspace\nexport TOKENIZERS_PARALLELISM=false\npython3 -c \"\nimport numpy as np\nfrom transformers import AutoTokenizer\ntok = AutoTokenizer.from_pretrained('gpt2')\na = np.load('/workspace/data/multi_dev.npy')\n# The dev is 'equal parts' 4 domains. Sample from 5 evenly spaced regions.\nN = a.size\nfor frac in [0.0, 0.24, 0.26, 0.49, 0.51, 0.74, 0.76, 0.99]:\n    s = int(frac*N)\n    txt = tok.decode(a[s:s+120].tolist())\n    print(f'--- @ {frac:.2f} (tok {s}) ---')\n    print(repr(txt[:400]))\n    print()\n\"",
  "description": "Decode samples across dev target"
}

Bash result

--- @ 0.00 (tok 0) ---
" Beardmore was sufficiently impressed with Shackleton to offer financial support , but other donations proved hard to come by . Nevertheless , in February 1907 , Shackleton presented to the Royal Geographic Society his plans for an Antarctic expedition , the details of which , under the name British Antarctic Expedition , were published in the Royal Society 's newsletter , Geographic Journal . The"

--- @ 0.24 (tok 240000) ---
'aics differ across countries , including Australia , China , Germany , Israel , Japan , and the United States and even across states within the US . \n<|endoftext|> The Japanese government through its Ministry of International Trade and Industry ran a successful programme of subsidies from 1994 to 2003 . By the end of 2004 , Japan led the world in installed PV capacity with over 1 @.@ 1 GW . \n<|end'

--- @ 0.26 (tok 260000) ---
" tonnes shipped in 2015, further exacerbating a tightening rice market as drought has also hit top rice exporters India and Thailand.\n\nWatched by Cambodia's King Norodom Sihamoni, and a crowd of thousands in the ceremonial furrow in Siem Reap province, the two cows ate 90 percent of three out of seven snacks on offer in ornate bowls.\n\nEach year, based on the oxen's choice of crops and the amount t"

--- @ 0.49 (tok 490000) ---
' Life rally and to talk about improvements to mental health treatment in the province.\n\n"[People] can\'t be complacent, they can\'t hide behind their doors, they have to get involved," Bonnie Bricker said.\n\n"We can\'t afford to be lazy and not involved in this."\n\nThe rally honoured Bricker\'s son, Reid, who died after suffering from depression.\n\nReid disappeared following his release from the Health S'

--- @ 0.51 (tok 510000) ---
' playing a Test match or what! Such a confident shot against one of the better spinners going around. 124/1\n34.1 Y Shah to Samarawickrama, Loopy ball around off, defended off the front foot onto the ground. 118/1\n33.6 W Riaz to Karunaratne, Another delivery is kept out from within the crease. A maiden from Riaz! 118/1\n33.5 W Riaz to Karunaratne, Karunaratne blocks this ball from within the crease.'

--- @ 0.74 (tok 740000) ---
' the world."Her show Superstore follows the lives of different individuals working in the store. The second season is currently on air. Ferrera says diversity drew her to the project."The writer and creator of the show worked the pilot script first well before I was attached to it as an actor and producer. These elements (diversity and human interest issues) definitely made the project exciting to'

--- @ 0.76 (tok 760000) ---
' echo "Line 0: \'${LINES[0]}\'"\n    echo "Line 1: \'${LINES[1]}\'"\n    # Line 0: \'Hello\'\n    # Line 1: \'there\'\n);(\n    echo Test 10\n    local LINE_STR=$( emulateUnsafeInput )\n    eval declare -a LINES=( ${LINE_STR} )\n    echo "Line 0: \'${LINES[0]}\'"\n    echo "Line 1:'

--- @ 0.99 (tok 990000) ---
'"&gt;\n&lt;tr&gt;\n    &lt;td&gt;\n    1\n    &lt;/td&gt;\n    &lt;td&gt;\n    2\n    &lt;/td&gt;\n    &lt;td&gt;\n    3\n    &lt;/td&gt;\n    &lt;td class="dragMe"&gt;\n      '

[stdout]
--- @ 0.00 (tok 0) ---
" Beardmore was sufficiently impressed with Shackleton to offer financial support , but other donations proved hard to come by . Nevertheless , in February 1907 , Shackleton presented to the Royal Geographic Society his plans for an Antarctic expedition , the details of which , under the name British Antarctic Expedition , were published in the Royal Society 's newsletter , Geographic Journal . The"

--- @ 0.24 (tok 240000) ---
'aics differ across countries , including Australia , China , Germany , Israel , Japan , and the United States and even across states within the US . \n<|endoftext|> The Japanese government through its Ministry of International Trade and Industry ran a successful programme of subsidies from 1994 to 2003 . By the end of 2004 , Japan led the world in installed PV capacity with over 1 @.@ 1 GW . \n<|end'

--- @ 0.26 (tok 260000) ---
" tonnes shipped in 2015, further exacerbating a tightening rice market as drought has also hit top rice exporters India and Thailand.\n\nWatched by Cambodia's King Norodom Sihamoni, and a crowd of thousands in the ceremonial furrow in Siem Reap province, the two cows ate 90 percent of three out of seven snacks on offer in ornate bowls.\n\nEach year, based on the oxen's choice of crops and the amount t"

--- @ 0.49 (tok 490000) ---
' Life rally and to talk about improvements to mental health treatment in the province.\n\n"[People] can\'t be complacent, they can\'t hide behind their doors, they have to get involved," Bonnie Bricker said.\n\n"We can\'t afford to be lazy and not involved in this."\n\nThe rally honoured Bricker\'s son, Reid, who died after suffering from depression.\n\nReid disappeared following his release from the Health S'

--- @ 0.51 (tok 510000) ---
' playing a Test match or what! Such a confident shot against one of the better spinners going around. 124/1\n34.1 Y Shah to Samarawickrama, Loopy ball around off, defended off the front foot onto the ground. 118/1\n33.6 W Riaz to Karunaratne, Another delivery is kept out from within the crease. A maiden from Riaz! 118/1\n33.5 W Riaz to Karunaratne, Karunaratne blocks this ball from within the crease.'

--- @ 0.74 (tok 740000) ---
' the world."Her show Superstore follows the lives of different individuals working in the store. The second season is currently on air. Ferrera says diversity drew her to the project."The writer and creator of the show worked the pilot script first well before I was attached to it as an actor and producer. These elements (diversity and human interest issues) definitely made the project exciting to'

--- @ 0.76 (tok 760000) ---
' echo "Line 0: \'${LINES[0]}\'"\n    echo "Line 1: \'${LINES[1]}\'"\n    # Line 0: \'Hello\'\n    # Line 1: \'there\'\n);(\n    echo Test 10\n    local LINE_STR=$( emulateUnsafeInput )\n    eval declare -a LINES=( ${LINE_STR} )\n    echo "Line 0: \'${LINES[0]}\'"\n    echo "Line 1:'

--- @ 0.99 (tok 990000) ---
'"&gt;\n&lt;tr&gt;\n    &lt;td&gt;\n    1\n    &lt;/td&gt;\n    &lt;td&gt;\n    2\n    &lt;/td&gt;\n    &lt;td&gt;\n    3\n    &lt;/td&gt;\n    &lt;td class="dragMe"&gt;\n      '
[metadata] {"noOutputExpected": false}

narration

The target spans 4 registers: encyclopedic (Wikipedia/WikiText-style), news, general web prose, and technical Q&A (Stack Overflow with code). Now let me profile the pool's quality distribution.

toolBashProfile pool lengths and sample random docs

args

{
  "command": "cd /workspace\npython3 -c \"\nimport json, random\nrandom.seed(0)\ndocs=[]\nwith open('/workspace/data/pool.jsonl') as f:\n    for line in f:\n        docs.append(json.loads(line))\nprint('total docs', len(docs))\nimport statistics\nlens=[len(d['text']) for d in docs]\nprint('char len: min',min(lens),'med',statistics.median(lens),'mean',round(statistics.mean(lens)),'max',max(lens))\n# char length percentiles\nsl=sorted(lens)\nfor p in [1,5,10,25,50,75,90,95,99]:\n    print(f'  p{p}: {sl[int(p/100*len(sl))]}')\n# sample 8 random docs\nprint('===== RANDOM SAMPLES =====')\nfor d in random.sample(docs, 8):\n    print(f'--- id {d[\\\"id\\\"]} len {len(d[\\\"text\\\"])} ---')\n    print(repr(d['text'][:300]))\n    print()\n\"",
  "description": "Profile pool lengths and sample random docs"
}

Bash result

total docs 182016
char len: min 2 med 2246.0 mean 4233 max 522573
  p1: 160
  p5: 408
  p10: 561
  p25: 1050
  p50: 2246
  p75: 4500
  p90: 8458
  p95: 13065
  p99: 34874
===== RANDOM SAMPLES =====
--- id 100989 len 495 ---
" [Facebook]<|endoftext|>Your favorite Apple, iPhone, iPad, iOS, Jailbreak, and Cydia site.\nThread: Third party or apple cable?\n03-26-2012, 04:07 PM #1What's Jailbreak?\nThird party or apple cable?\nI have a jailbreak iPhone 3G on 4.2.1 with displayout sky sports tv app with third party cable, the qual"

--- id 110250 len 1875 ---
' 7:35 a.m. Tuesday in Barton County. Injuries were minor but one driver, Cindy Aracely Esquivel-Dominguez, was arrested for driving without a license, a traffic infraction and no insurance.\nAfter a Ferguson, Mo., police officer fatally shot Michael Brown on Aug. 9, and the subsequent rioting, the Ob'

--- id 10612 len 4160 ---
'Tasmania in the winter can be brutal, but considering the winters I have seen in the northeast US and the Shenandoah Valley in my lifetime, the cold was a tad inconvenient. But next time (if there is a next time), come in the summer. There’s just a lot more to enjoy when the weather is warm.\nWe land'

--- id 67873 len 15960 ---
"<|endoftext|>by Michael Doliner\nDrury, Shadia B.: Leo Strauss and the American Right, Palgrave Macmillan, February 1999, ISBN 0-31221-783-8, 256 pages, $29.95 (hardcover)\n(Swans - October 10, 2005) Will the backlash from Katrina's destruction and the Bush Administration's woeful response to it final"

--- id 134027 len 5078 ---
' - NX - Air & Fuel - VMX Unlimited\nLoading... Please wait...\nMy Account\nOrder Status\nWish Lists\nGift Certificates\nView Cart\nSign in or Create an account\nSearch\nAdvanced Search | Search Tips\nHome\nArticles\nMaico Stories\nMaico History through 1977\nVintage Plastics info\nExhausts info\nSeat cover info\nCZ '

--- id 127383 len 6155 ---
'<|endoftext|>online business systems that work Archives - VistaPrint - 250 Free Business Cards\nVistaPrint – 250 Free Business Cards\nFree Business Cards Online from VistaPrint\nMain navigation\nHome\nDisclaimer\nPrivacy\nContact Us\nNewsletter\nYou are here: Home › online business systems that work\nTag Arch'

--- id 106151 len 1184 ---
'The Lockpicking Corollary\nIn a game with optional lockpicking mechanics, designers must include chests and doors and such that contain treasure to justify the investment of (usually) finite skill points in an otherwise non-combat skill. This reward cannot be too generous however, as it otherwise mov'

--- id 79511 len 12527 ---
'Accreditation of Continuing Education Providers in Pharmacy\nBy Dimitra V. Travlos, PharmD, assistant executive director and director, Continuing Pharmacy Education (CPE) Provider Accreditation; Jennifer L. Baumgartner, PharmD, BCPP, assistant director, CPE Provider Accreditation; and Peter H. Vlasse'

[stdout]
total docs 182016
char len: min 2 med 2246.0 mean 4233 max 522573
  p1: 160
  p5: 408
  p10: 561
  p25: 1050
  p50: 2246
  p75: 4500
  p90: 8458
  p95: 13065
  p99: 34874
===== RANDOM SAMPLES =====
--- id 100989 len 495 ---
" [Facebook]<|endoftext|>Your favorite Apple, iPhone, iPad, iOS, Jailbreak, and Cydia site.\nThread: Third party or apple cable?\n03-26-2012, 04:07 PM #1What's Jailbreak?\nThird party or apple cable?\nI have a jailbreak iPhone 3G on 4.2.1 with displayout sky sports tv app with third party cable, the qual"

--- id 110250 len 1875 ---
' 7:35 a.m. Tuesday in Barton County. Injuries were minor but one driver, Cindy Aracely Esquivel-Dominguez, was arrested for driving without a license, a traffic infraction and no insurance.\nAfter a Ferguson, Mo., police officer fatally shot Michael Brown on Aug. 9, and the subsequent rioting, the Ob'

--- id 10612 len 4160 ---
'Tasmania in the winter can be brutal, but considering the winters I have seen in the northeast US and the Shenandoah Valley in my lifetime, the cold was a tad inconvenient. But next time (if there is a next time), come in the summer. There’s just a lot more to enjoy when the weather is warm.\nWe land'

--- id 67873 len 15960 ---
"<|endoftext|>by Michael Doliner\nDrury, Shadia B.: Leo Strauss and the American Right, Palgrave Macmillan, February 1999, ISBN 0-31221-783-8, 256 pages, $29.95 (hardcover)\n(Swans - October 10, 2005) Will the backlash from Katrina's destruction and the Bush Administration's woeful response to it final"

--- id 134027 len 5078 ---
' - NX - Air & Fuel - VMX Unlimited\nLoading... Please wait...\nMy Account\nOrder Status\nWish Lists\nGift Certificates\nView Cart\nSign in or Create an account\nSearch\nAdvanced Search | Search Tips\nHome\nArticles\nMaico Stories\nMaico History through 1977\nVintage Plastics info\nExhausts info\nSeat cover info\nCZ '

--- id 127383 len 6155 ---
'<|endoftext|>online business systems that work Archives - VistaPrint - 250 Free Business Cards\nVistaPrint – 250 Free Business Cards\nFree Business Cards Online from VistaPrint\nMain navigation\nHome\nDisclaimer\nPrivacy\nContact Us\nNewsletter\nYou are here: Home › online business systems that work\nTag Arch'

--- id 106151 len 1184 ---
'The Lockpicking Corollary\nIn a game with optional lockpicking mechanics, designers must include chests and doors and such that contain treasure to justify the investment of (usually) finite skill points in an otherwise non-combat skill. This reward cannot be too generous however, as it otherwise mov'

--- id 79511 len 12527 ---
'Accreditation of Continuing Education Providers in Pharmacy\nBy Dimitra V. Travlos, PharmD, assistant executive director and director, Continuing Pharmacy Education (CPE) Provider Accreditation; Jennifer L. Baumgartner, PharmD, BCPP, assistant director, CPE Provider Accreditation; and Peter H. Vlasse'
[metadata] {"noOutputExpected": false}

narration

The pool is classic raw web: a mix of clean prose (news, essays, travel) and junk (SEO spam, e-commerce boilerplate, forum snippets). My plan is a target-similarity classifier (DSIR-style) plus quality gates. Let me set up tracking and check the environment.

toolTaskCreateProfile data and target

args

{
  "description": "Profile data and target",
  "subject": "Understand pool + target distribution",
  "activeForm": "Profiling pool and target"
}

TaskCreate result

Task #1 created successfully: Understand pool + target distribution

[metadata] {"task": {"id": "1", "subject": "Understand pool + target distribution"}}

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolTaskCreateBuild curation script

args

{
  "description": "Build curation script",
  "subject": "Build target-similarity classifier + quality gates in curate.py",
  "activeForm": "Building curate.py"
}

TaskCreate result

Task #2 created successfully: Build target-similarity classifier + quality gates in curate.py

[metadata] {"task": {"id": "2", "subject": "Build target-similarity classifier + quality gates in curate.py"}}

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolTaskCreateEvaluate selection

args

{
  "description": "Evaluate selection",
  "subject": "Run dev_eval on candidate selections, compare vs random baseline",
  "activeForm": "Evaluating selections"
}

TaskCreate result

Task #3 created successfully: Run dev_eval on candidate selections, compare vs random baseline

[metadata] {"task": {"id": "3", "subject": "Run dev_eval on candidate selections, compare vs random baseline"}}

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolTaskCreateWrite deliverables

args

{
  "description": "Write deliverables",
  "subject": "Write selection.json, curate.py, claim.md deliverables",
  "activeForm": "Writing deliverables"
}

TaskCreate result

Task #4 created successfully: Write selection.json, curate.py, claim.md deliverables

[metadata] {"task": {"id": "4", "subject": "Write selection.json, curate.py, claim.md deliverables"}}

#10

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolBashCheck sklearn/scipy, split dev into docs

args

{
  "command": "cd /workspace\npython3 -c \"import sklearn; print('sklearn', sklearn.__version__)\" 2>&1 | head -1\npython3 -c \"import scipy; print('scipy', scipy.__version__)\" 2>&1 | head -1\n# decode dev target, split into docs by EOS, profile\npython3 -c \"\nimport numpy as np\nfrom transformers import AutoTokenizer\ntok = AutoTokenizer.from_pretrained('gpt2')\na = np.load('/workspace/data/multi_dev.npy').astype(np.int64)\nEOS=50256\n# split into docs on EOS\nidx = np.where(a==EOS)[0]\nprint('num EOS tokens in dev:', len(idx))\n# doc lengths between EOS\nsegs=[]\nprev=0\nfor i in idx:\n    if i>prev: segs.append((prev,i))\n    prev=i+1\nif prev<len(a): segs.append((prev,len(a)))\nprint('num segments:', len(segs))\nlens=[e-s for s,e in segs]\nimport statistics\nprint('seg token len: min',min(lens),'med',statistics.median(lens),'max',max(lens))\n\"",
  "description": "Check sklearn/scipy, split dev into docs"
}

Bash result

Traceback (most recent call last):
Traceback (most recent call last):
num EOS tokens in dev: 2704
num segments: 2705
seg token len: min 20 med 192 max 23346

[stdout]
Traceback (most recent call last):
Traceback (most recent call last):
num EOS tokens in dev: 2704
num segments: 2705
seg token len: min 20 med 192 max 23346
[metadata] {"noOutputExpected": false}

#11

narration

sklearn/scipy aren't available, so I'll implement the DSIR-style importance classifier in pure numpy. Let me confirm the environment and RAM, then write the curation script.

toolBashConfirm libs and RAM

args

{
  "command": "cd /workspace\npython3 -c \"import numpy, torch, transformers; print('numpy',numpy.__version__,'torch',torch.__version__,'tf',transformers.__version__)\"\nfree -g | head -2\necho \"remaining: $(cat /workspace/.timer/remaining_secs)s\"",
  "description": "Confirm libs and RAM"
}

Bash result

numpy 1.26.4 torch 2.8.0+cu128 tf 4.57.1
               total        used        free      shared  buff/cache   available
Mem:             944           0         944           0           0         944
remaining: 5150s

[stdout]
numpy 1.26.4 torch 2.8.0+cu128 tf 4.57.1
               total        used        free      shared  buff/cache   available
Mem:             944           0         944           0           0         944
remaining: 5150s
[metadata] {"noOutputExpected": false}

#12

narration

944 GB RAM — no constraints. Now I'll write the curation script: a DSIR-style importance classifier using hashed word n-grams to match the disclosed target, plus light boilerplate gates. Deterministic (crc32 hashing) for reproducibility.

toolWrite

args

{
  "file_path": "/workspace/submission/curate.py",
  "content": "#!/usr/bin/env python3\n\"\"\"Curate the raw web pool for from-scratch LM pretraining under a fixed token budget.\n\nSTATED CRITERION\n----------------\nThe disclosed evaluation target is a BROAD, multi-domain high-quality English\ndistribution (equal parts Wikipedia/encyclopedic, general HQ web prose, news, and\ntechnical Q&A).  We therefore rank every pool document by how much its *content*\nresembles that target distribution, and drop obvious non-prose boilerplate.\n\nConcretely we use DSIR-style importance weighting (Xie et al. 2023): estimate a\nbag-of-hashed-n-grams language model for the TARGET (decoded from the provided\n`multi_dev.npy`, the disclosed target sample) and for the POOL background, and\nscore each document by its mean per-token log-likelihood ratio\n\n        score(doc) = mean_{f in ngrams(doc)}  [ log P_target(f) - log P_pool(f) ].\n\nDocuments whose vocabulary looks like the multi-domain HQ target score high;\ndocuments dominated by pool-typical junk (SEO/e-commerce boilerplate, forum cruft,\nnavigation menus) score low.  Features are word unigrams+bigrams (lower-cased\n[a-z']+ runs), so the score keys on *content/register*, not on the target sample's\nsurface formatting — it transfers to the disjoint hidden target of the same domain.\n\nA light quality gate removes fragments and boilerplate that the ratio alone can\nrank too high (very short docs, navigation-menu pages with mostly tiny lines, and\nhighly repetitive documents).\n\nOutput: `selection.json` — every surviving id, sorted best-first.  The frozen\ntraining pipeline consumes this order until the 12M-token budget is filled.\n\"\"\"\nimport argparse, json, re, zlib, sys, math\nimport numpy as np\n\nPOOL = \"/workspace/data/pool.jsonl\"\nDEV_NPY = \"/workspace/data/multi_dev.npy\"   # disclosed target sample (GPT-2 token ids)\nOUT = \"/workspace/submission/selection.json\"\n\n# ------------------------------ config ---------------------------------------\nHASH_BITS = 20                 # 2^20 = ~1M feature buckets\nMASK = (1 << HASH_BITS) - 1\nBG_SAMPLE = 60000              # pool docs used to estimate the background n-gram LM\nALPHA = 1.0                    # add-alpha smoothing on bucket counts\nBIGRAM_MULT = 1000003          # mix constant for combining two unigram hashes\nWORD_RE = re.compile(r\"[a-z']+\")\nLINE_RE = re.compile(r\"[^\\n]*\\n?\")\n\n# quality gate thresholds\nMIN_CHARS = 200\nMIN_WORDS = 50\nMAX_SHORT_LINE_FRAC = 0.66     # navigation-menu / listing boilerplate\nMAX_DUP_LINE_FRAC = 0.50       # heavily repeated lines\n\nSTOPWORDS = set(\"\"\"the of and to a in is that it for on with as was be by this are at from or an\nwhich not but have has had they you we he she his her their its will would can could been being\ni my me your our us them do does did so if then than out up about into over after \"\"\".split())\n\n\ndef word_hashes(words, cache):\n    \"\"\"crc32-based deterministic hash per word (cached).\"\"\"\n    out = []\n    ap = out.append\n    for w in words:\n        h = cache.get(w)\n        if h is None:\n            h = zlib.crc32(w.encode()) & 0xffffffff\n            cache[w] = h\n        ap(h)\n    return out\n\n\ndef doc_feature_buckets(words, cache):\n    \"\"\"unigram + bigram hashed buckets for a document (as a python list of ints).\"\"\"\n    uh = word_hashes(words, cache)\n    buckets = [h & MASK for h in uh]\n    for i in range(len(uh) - 1):\n        buckets.append(((uh[i] * BIGRAM_MULT + uh[i + 1]) & MASK))\n    return buckets\n\n\ndef accumulate_counts(bucket_lists, counts):\n    for b in bucket_lists:\n        if b:\n            np.add.at(counts, np.asarray(b, dtype=np.int64), 1.0)\n\n\ndef quality_ok(text, words):\n    \"\"\"Light boilerplate/fragment gate. Returns (ok: bool).\"\"\"\n    if len(text) < MIN_CHARS or len(words) < MIN_WORDS:\n        return False\n    lines = [ln for ln in text.split(\"\\n\") if ln.strip()]\n    if len(lines) >= 8:\n        short = sum(1 for ln in lines if len(ln.split()) <= 3)\n        if short / len(lines) > MAX_SHORT_LINE_FRAC:\n            return False\n        uniq = len(set(lines))\n        if 1 - uniq / len(lines) > MAX_DUP_LINE_FRAC:\n            return False\n    return True\n\n\ndef main():\n    ap = argparse.ArgumentParser()\n    ap.add_argument(\"--out\", default=OUT)\n    ap.add_argument(\"--seed\", type=int, default=0)\n    ap.add_argument(\"--min_score\", type=float, default=None,\n                    help=\"optional hard score cutoff; default keeps all gated docs\")\n    args = ap.parse_args()\n    rng = np.random.default_rng(args.seed)\n\n    # ---- 1. load pool -------------------------------------------------------\n    print(\"loading pool...\", file=sys.stderr)\n    ids, texts = [], []\n    with open(POOL) as f:\n        for line in f:\n            r = json.loads(line)\n            ids.append(r[\"id\"]); texts.append(r[\"text\"])\n    ids = np.array(ids)\n    N = len(ids)\n    print(f\"  {N} docs\", file=sys.stderr)\n\n    # ---- 2. build TARGET n-gram LM from decoded dev sample ------------------\n    print(\"decoding target sample + building target LM...\", file=sys.stderr)\n    from transformers import AutoTokenizer\n    tok = AutoTokenizer.from_pretrained(\"gpt2\")\n    dev = np.load(DEV_NPY).astype(np.int64)\n    EOS = 50256\n    # split into documents on EOS, decode each, featurize\n    idx = np.where(dev == EOS)[0]\n    segs, prev = [], 0\n    for i in idx:\n        if i > prev: segs.append((prev, i))\n        prev = i + 1\n    if prev < len(dev): segs.append((prev, len(dev)))\n    tgt_counts = np.zeros(MASK + 1, dtype=np.float64)\n    cache = {}\n    tgt_bucket_lists = []\n    for s, e in segs:\n        txt = tok.decode(dev[s:e].tolist())\n        words = WORD_RE.findall(txt.lower())\n        tgt_bucket_lists.append(doc_feature_buckets(words, cache))\n    accumulate_counts(tgt_bucket_lists, tgt_counts)\n    print(f\"  target: {len(segs)} segments, {tgt_counts.sum():.0f} features\", file=sys.stderr)\n\n    # ---- 3. build POOL background n-gram LM from a random sample ------------\n    print(\"building pool background LM...\", file=sys.stderr)\n    bg_idx = rng.choice(N, size=min(BG_SAMPLE, N), replace=False)\n    bg_counts = np.zeros(MASK + 1, dtype=np.float64)\n    bg_lists = []\n    for j in bg_idx:\n        words = WORD_RE.findall(texts[j].lower())\n        bg_lists.append(doc_feature_buckets(words, cache))\n        if len(bg_lists) >= 4000:\n            accumulate_counts(bg_lists, bg_counts); bg_lists = []\n    accumulate_counts(bg_lists, bg_counts)\n    print(f\"  background: {len(bg_idx)} docs, {bg_counts.sum():.0f} features\", file=sys.stderr)\n\n    # ---- 4. per-bucket log-likelihood-ratio weights ------------------------\n    T = tgt_counts.sum(); B = bg_counts.sum(); V = MASK + 1\n    log_tgt = np.log(tgt_counts + ALPHA) - math.log(T + ALPHA * V)\n    log_bg = np.log(bg_counts + ALPHA) - math.log(B + ALPHA * V)\n    weight = log_tgt - log_bg          # importance log-weight per feature bucket\n\n    # ---- 5. score + gate every document ------------------------------------\n    print(\"scoring all documents...\", file=sys.stderr)\n    scores = np.full(N, -1e9, dtype=np.float64)\n    ok_mask = np.zeros(N, dtype=bool)\n    for k in range(N):\n        text = texts[k]\n        words = WORD_RE.findall(text.lower())\n        if not quality_ok(text, words):\n            continue\n        b = doc_feature_buckets(words, cache)\n        if not b:\n            continue\n        scores[k] = weight[np.asarray(b, dtype=np.int64)].mean()\n        ok_mask[k] = True\n        if (k + 1) % 20000 == 0:\n            print(f\"  {k+1}/{N}\", file=sys.stderr)\n\n    n_ok = int(ok_mask.sum())\n    print(f\"  {n_ok}/{N} docs passed the quality gate\", file=sys.stderr)\n\n    # ---- 6. rank + emit ----------------------------------------------------\n    order = np.argsort(-scores)                 # best first\n    order = order[ok_mask[order]]               # keep only gated docs\n    if args.min_score is not None:\n        order = order[scores[order] >= args.min_score]\n    sel_ids = ids[order].tolist()\n\n    # diagnostics: score distribution\n    ss = np.sort(scores[ok_mask])\n    for p in [50, 75, 90, 95, 99, 100]:\n        v = ss[min(len(ss) - 1, int(p / 100 * len(ss)))]\n        print(f\"  score p{p}: {v:.4f}\", file=sys.stderr)\n\n    json.dump(sel_ids, open(args.out, \"w\"))\n    print(f\"wrote {len(sel_ids)} ids -> {args.out}\", file=sys.stderr)\n\n\nif __name__ == \"__main__\":\n    main()\n"
}

Write result

File created successfully at: /workspace/submission/curate.py (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/workspace/submission/curate.py", "content": "#!/usr/bin/env python3\n\"\"\"Curate the raw web pool for from-scratch LM pretraining under a fixed token budget.\n\nSTATED CRITERION\n----------------\nThe disclosed evaluation target is a BROAD, multi-domain high-quality English\ndistribution (equal parts Wikipedia/encyclopedic, general HQ web prose, news, and\ntechnical Q&A).  We therefore rank every pool document by how much its *content*\nresembles that target distribution, and drop obvious non-prose boilerplate.\n\nConcretely we use DSIR-style importance weighting (Xie et al. 2023): estimate a\nbag-of-hashed-n-grams language model for the TARGET (decoded from the provided\n`multi_dev.npy`, the disclosed target sample) and for the POOL background, and\nscore each document by its mean per-token log-likelihood ratio\n\n        score(doc) = mean_{f in ngrams(doc)}  [ log P_target(f) - log P_pool(f) ].\n\nDocuments whose vocabulary looks like the multi-domain HQ target score high;\ndocuments dominated by pool-typical junk (SEO/e-commerce boilerplate, forum cruft,\nnavigation menus) score low.  Features are word unigrams+bigrams (lower-cased\n[a-z']+ runs), so the score keys on *content/register*, not on the target sample's\nsurface formatting — it transfers to the disjoint hidden target of the same domain.\n\nA light quality gate removes fragments and boilerplate that the ratio alone can\nrank too high (very short docs, navigation-menu pages with mostly tiny lines, and\nhighly repetitive documents).\n\nOutput: `selection.json` — every surviving id, sorted best-first.  The frozen\ntraining pipeline consumes this order until the 12M-token budget is filled.\n\"\"\"\nimport argparse, json, re, zlib, sys, math\nimport numpy as np\n\nPOOL = \"/workspace/data/pool.jsonl\"\nDEV_NPY = \"/workspace/data/multi_dev.npy\"   # disclosed target sample (GPT-2 token ids)\nOUT = \"/workspace/submission/selection.json\"\n\n# ------------------------------ config ---------------------------------------\nHASH_BITS = 20                 # 2^20 = ~1M feature buckets\nMASK = (1 << HASH_BITS) - 1\nBG_SAMPLE = 60000              # pool docs used to estimate the background n-gram LM\nALPHA = 1.0                    # add-alpha smoothing on bucket counts\nBIGRAM_MULT = 1000003          # mix constant for combining two unigram hashes\nWORD_RE = re.compile(r\"[a-z']+\")\nLINE_RE = re.compile(r\"[^\\n]*\\n?\")\n\n# quality gate thresholds\nMIN_CHARS = 200\nMIN_WORDS = 50\nMAX_SHORT_LINE_FRAC = 0.66     # navigation-menu / listing boilerplate\nMAX_DUP_LINE_FRAC = 0.50       # heavily repeated lines\n\nSTOPWORDS = set(\"\"\"the of and to a in is that it for on with as was be by this are at from or an\nwhich not but have has had they you we he she his her their its will would can could been being\ni my me your our us them do does did so if then than out up about into over after \"\"\".split())\n\n\ndef word_hashes(words, cache):\n    \"\"\"crc32-based deterministic hash per word (cached).\"\"\"\n    out = []\n    ap = out.append\n    for w in words:\n        h = cache.get(w)\n        if h is None:\n            h = zlib.crc32(w.encode()) & 0xffffffff\n            cache[w] = h\n        ap(h)\n    return out\n\n\ndef doc_feature_buckets(words, cache):\n    \"\"\"unigram + bigram hashed buckets for a document (as a python list of ints).\"\"\"\n    uh = word_hashes(words, cache)\n    buckets = [h & MASK for h in uh]\n    for i in range(len(uh) - 1):\n        buckets.append(((uh[i] * BIGRAM_MULT + uh[i + 1]) & MASK))\n    return buckets\n\n\ndef accumulate_counts(bucket_lists, counts):\n    for b in bucket_lists:\n        if b:\n            np.add.at(counts, np.asarray(b, dtype=np.int64), 1.0)\n\n\ndef quality_ok(text, words):\n    \"\"\"Light boilerplate/fragment gate. Returns (ok: bool).\"\"\"\n    if len(text) < MIN_CHARS or len(words) < MIN_WORDS:\n        return False\n    lines = [ln for ln in text.split(\"\\n\") if ln.strip()]\n    if len(lines) >= 8:\n        short = sum(1 for ln in lines if len(ln.split()) <= 3)\n        if short / len(lines) > MAX_SHORT_LINE_FRAC:\n            return False\n        uniq = len(set(lines))\n        if 1 - uniq / len(lines) > MAX_DUP_LINE_FRAC:\n            return False\n    return True\n\n\ndef main():\n    ap = argparse.ArgumentParser()\n    ap.add_argument(\"--out\", default=OUT)\n    ap.add_argument(\"--seed\", type=int, default=0)\n    ap.add_argument(\"--min_score\", type=float, default=None,\n                    help=\"optional hard score cutoff; default keeps all gated docs\")\n    args = ap.parse_args()\n    rng = np.random.default_rng(args.seed)\n\n    # ---- 1. load pool -------------------------------------------------------\n    print(\"loading pool...\", file=sys.stderr)\n    ids, texts = [], []\n    with open(POOL) as f:\n        for line in f:\n            r = json.loads(line)\n            ids.append(r[\"id\"]); texts.append(r[\"text\"])\n    ids = np.array(ids)\n    N = len(ids)\n    print(f\"  {N} docs\", file=sys.stderr)\n\n    # ---- 2. build TARGET n-gram LM from decoded dev sample ------------------\n    print(\"decoding target sample + building target LM...\", file=sys.stderr)\n    from transformers import AutoTokenizer\n    tok = AutoTokenizer.from_pretrained(\"gpt2\")\n    dev = np.load(DEV_NPY).astype(np.int64)\n    EOS = 50256\n    # split into documents on EOS, decode each, featurize\n    idx = np.where(dev == EOS)[0]\n    segs, prev = [], 0\n    for i in idx:\n        if i > prev: segs.append((prev, i))\n        prev = i + 1\n    if prev < len(dev): segs.append((prev, len(dev)))\n    tgt_counts = np.zeros(MASK + 1, dtype=np.float64)\n    cache = {}\n    tgt_bucket_lists = []\n    for s, e in segs:\n        txt = tok.decode(dev[s:e].tolist())\n        words = WORD_RE.findall(txt.lower())\n        tgt_bucket_lists.append(doc_feature_buckets(words, cache))\n    accumulate_counts(tgt_bucket_lists, tgt_counts)\n    print(f\"  target: {len(segs)} segments, {tgt_counts.sum():.0f} features\", file=sys.stderr)\n\n    # ---- 3. build POOL background n-gram LM from a random sample ------------\n    print(\"building pool background LM...\", file=sys.stderr)\n    bg_idx = rng.choice(N, size=min(BG_SAMPLE, N), replace=False)\n    bg_counts = np.zeros(MASK + 1, dtype=np.float64)\n    bg_lists = []\n    for j in bg_idx:\n        words = WORD_RE.findall(texts[j].lower())\n        bg_lists.append(doc_feature_buckets(words, cache))\n        if len(bg_lists) >= 4000:\n            accumulate_counts(bg_lists, bg_counts); bg_lists = []\n    accumulate_counts(bg_lists, bg_counts)\n    print(f\"  background: {len(bg_idx)} docs, {bg_counts.sum():.0f} features\", file=sys.stderr)\n\n    # ---- 4. per-bucket log-likelihood-ratio weights ------------------------\n    T = tgt_counts.sum(); B = bg_counts.sum(); V = MASK + 1\n    log_tgt = np.log(tgt_counts + ALPHA) - math.log(T + ALPHA * V)\n    log_bg = np.log(bg_counts + ALPHA) - math.log(B + ALPHA * V)\n    weight = log_tgt - log_bg          # importance log-weight per feature bucket\n\n    # ---- 5. score + gate every document ------------------------------------\n    print(\"scoring all documents...\", file=sys.stderr)\n    scores = np.full(N, -1e9, dtype=np.float64)\n    ok_mask = np.zeros(N, dtype=bool)\n    for k in range(N):\n        text = texts[k]\n        words = WORD_RE.findall(text.lower())\n        if not quality_ok(text, words):\n            continue\n        b = doc_feature_buckets(words, cache)\n        if not b:\n            continue\n        scores[k] = weight[np.asarray(b, dtype=np.int64)].mean()\n        ok_mask[k] = True\n        if (k + 1) % 20000 == 0:\n            print(f\"  {k+1}/{N}\", file=sys.stderr)\n\n    n_ok = int(ok_mask.sum())\n    print(f\"  {n_ok}/{N} docs passed the quality gate\", file=sys.stderr)\n\n    # ---- 6. rank + emit ----------------------------------------------------\n    order = np.argsort(-scores)                 # best first\n    order = order[ok_mask[order]]               # keep only gated docs\n    if args.min_score is not None:\n        order = order[scores[order] >= args.min_score]\n    sel_ids = ids[order].tolist()\n\n    # diagnostics: score distribution\n    ss = np.sort(scores[ok_mask])\n    for p in [50, 75, 90, 95, 99, 100]:\n        v = ss[min(len(ss) - 1, int(p / 100 * len(ss)))]\n        print(f\"  score p{p}: {v:.4f}\", file=sys.stderr)\n\n    json.dump(sel_ids, open(args.out, \"w\"))\n    print(f\"wrote {len(sel_ids)} ids -> {args.out}\", file=sys.stderr)\n\n\nif __name__ == \"__main__\":\n    main()\n", "structuredPatch": [], "originalFile": null, "userModified": false}

#13

narration

Now let me run the curation and time it.

toolBashRun curation script

args

{
  "command": "cd /workspace\nexport TOKENIZERS_PARALLELISM=false\ntime python3 submission/curate.py 2>&1 | tail -40\necho \"remaining: $(cat /workspace/.timer/remaining_secs)s\"",
  "description": "Run curation script"
}

Bash result

Command did not complete within its 120s timeout and was moved to the background (ID: b02zjvtx5). Output is being written to: /tmp/claude-0/-workspace/06cb3775-ded4-4ad5-84cf-ed7eaa26a5bd/tasks/b02zjvtx5.output. You will be notified when it completes. To check interim output, use Read on that file path.
Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "b02zjvtx5", "timedOutAfterMs": 120000, "backgroundCwdHint": "Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands."}

#14

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolBashCheck curation progress

args

{
  "command": "sleep 20; cat /tmp/claude-0/-workspace/06cb3775-ded4-4ad5-84cf-ed7eaa26a5bd/tasks/b02zjvtx5.output 2>/dev/null | tail -20",
  "description": "Check curation progress"
}

Bash result

40000/182016
  60000/182016
  80000/182016
  100000/182016
  120000/182016
  160000/182016
  180000/182016
  143210/182016 docs passed the quality gate
  score p50: -0.3691
  score p75: -0.2603
  score p90: -0.1719
  score p95: -0.1210
  score p99: -0.0196
  score p100: 0.8811
wrote 143210 ids -> /workspace/submission/selection.json

real	2m14.407s
user	2m5.130s
sys	0m2.130s
remaining: 4919s

[stdout]
  40000/182016
  60000/182016
  80000/182016
  100000/182016
  120000/182016
  160000/182016
  180000/182016
  143210/182016 docs passed the quality gate
  score p50: -0.3691
  score p75: -0.2603
  score p90: -0.1719
  score p95: -0.1210
  score p99: -0.0196
  score p100: 0.8811
wrote 143210 ids -> /workspace/submission/selection.json

real	2m14.407s
user	2m5.130s
sys	0m2.130s
remaining: 4919s
[metadata] {"noOutputExpected": false}

#15

narration

Curation done: 143,210 docs ranked. Let me sanity-check the top/bottom ranked docs before spending a training run.

toolBashInspect top/mid/bottom ranked docs

args

{
  "command": "cd /workspace\npython3 -c \"\nimport json\nsel = json.load(open('/workspace/submission/selection.json'))\ntexts={}\nwith open('/workspace/data/pool.jsonl') as f:\n    for line in f:\n        r=json.loads(line); texts[r['id']]=r['text']\nprint('=== TOP 6 ranked ===')\nfor i in sel[:6]:\n    print(f'--- id {i} len {len(texts[i])} ---'); print(repr(texts[i][:260])); print()\nprint('=== RANK ~5000 ===')\nfor i in sel[5000:5003]:\n    print(f'--- id {i} ---'); print(repr(texts[i][:220])); print()\nprint('=== BOTTOM 4 (worst that still passed) ===')\nfor i in sel[-4:]:\n    print(f'--- id {i} len {len(texts[i])} ---'); print(repr(texts[i][:220])); print()\n\"",
  "description": "Inspect top/mid/bottom ranked docs"
}

Bash result

=== TOP 6 ranked ===
--- id 164601 len 160132 ---
' Vehicle Hydraulic Off-Line Filtration\nWelcome to Rico Europe\nCREDIT ACCOUNTS\nMy Account\nSign In\nRegistration\nEnglish GB\nEspañol\nDeutsch\nItaliano\nFrançais\nCurrency\n€ EUR\n£ GBP\n$ USD\nCur\n€ EUR\n£ GBP\n$ USD\nshopping_cart 0 item(s)\n- US$0.00\n+44 (0) 1327 312838\nsa'

--- id 131290 len 13660 ---
' Railway Station\nCoffee Shop - Addlestone Railway Station\nCoffee Shop Home Page • About the Coffee Shop • Acronyms • Station Comparator • Calendar • New Topics • Popular Topics • MRUG • Coffee Shop Forum\nCampaigning Home Page • Campaigning - list of daily tips'

--- id 153946 len 13676 ---
' — Accessibility Notice<|endoftext|>Addlestone Railway Station\nCoffee Shop - Addlestone Railway Station\nCoffee Shop Home Page • About the Coffee Shop • Acronyms • Station Comparator • Calendar • New Topics • Popular Topics • MRUG • Coffee Shop Forum\nCampaignin'

--- id 108673 len 4060 ---
'Here are the list of players included in each of the eight playing team in IPL 2007-08. The tournament will begin on April 18, when Bangalore takes on Kolkata at the Chinnaswamy Stadium in Bangalore. The tournament will feature 59 matches in total, the teams p'

--- id 149512 len 178563 ---
"imbabwe<|endoftext|>HANSON - SHOWING ALL MATCHES :: Census Data Research Online\nCENSUS DATA\nRESEARCH ONLINE\nBROWSING LAST NAME = 'HANSON'\nType a First Name to Search\nHANSON,\nAll First Name Matches are Shown Below:\nAARON HANSON Found: 182, 0.133118% ABBEY HANSO"

--- id 126856 len 178563 ---
"imbabwe<|endoftext|>HANSON - SHOWING ALL MATCHES :: Census Data Research Online\nCENSUS DATA\nRESEARCH ONLINE\nBROWSING LAST NAME = 'HANSON'\nType a First Name to Search\nHANSON,\nAll First Name Matches are Shown Below:\nAARON HANSON Found: 182, 0.133118% ABBEY HANSO"

=== RANK ~5000 ===
--- id 64838 ---
' MOLD man was locked up for defrauding his elderly step-father out of £150,000 – after he was disinherited.\nJohn Whiteford, aged 62, had been entrusted to look after the financial affairs of 77-year-old William Search, w'

--- id 53915 ---
'Liverpool Pembroke Sefton went to the final match of Division 2 of the Northern League at Bebington with a reasonable chance of retaining membership of this highly competitive division but nothing was certain. Although n'

--- id 66087 ---
" Saxony state elections for the AfD, Germany's euroskeptic party, is unsettling for Merkel and her CDU in Berlin. However, the party isn't anything to be afraid of, says political scientist Oskar Niedermayer.\nDW: Despite"

=== BOTTOM 4 (worst that still passed) ===
--- id 156367 len 28587 ---
'.com Latest REAL UFO Sightings: New 2015 UFO Sighting| Latest UFO Sightings\nRead the Latest UFO Sightings\nSearch the Latest UFO Sightings\nTranslate\nWednesday, November 14, 2018\nNew 2015 UFO Sighting\nUFO Sighting in Gävle'

--- id 133711 len 28577 ---
' SkinPress.com<|endoftext|>4UFOS.com Latest REAL UFO Sightings: New 2015 UFO Sighting| Latest UFO Sightings\nRead the Latest UFO Sightings\nSearch the Latest UFO Sightings\nTranslate\nWednesday, November 14, 2018\nNew 2015 UF'

--- id 121310 len 15867 ---
'ancel<|endoftext|>01/08/13 | My Bubbletea Time\nskip to main | skip to sidebar\nPages\nHome\nProjects\nDownloads\nAnime\nMedia\nWallpapers\nThemes\nMy Bubbletea Time\nDaily thoughts of the otaku world, the internet and life in gene'

--- id 143966 len 15867 ---
'ancel<|endoftext|>01/08/13 | My Bubbletea Time\nskip to main | skip to sidebar\nPages\nHome\nProjects\nDownloads\nAnime\nMedia\nWallpapers\nThemes\nMy Bubbletea Time\nDaily thoughts of the otaku world, the internet and life in gene'

[stdout]
=== TOP 6 ranked ===
--- id 164601 len 160132 ---
' Vehicle Hydraulic Off-Line Filtration\nWelcome to Rico Europe\nCREDIT ACCOUNTS\nMy Account\nSign In\nRegistration\nEnglish GB\nEspañol\nDeutsch\nItaliano\nFrançais\nCurrency\n€ EUR\n£ GBP\n$ USD\nCur\n€ EUR\n£ GBP\n$ USD\nshopping_cart 0 item(s)\n- US$0.00\n+44 (0) 1327 312838\nsa'

--- id 131290 len 13660 ---
' Railway Station\nCoffee Shop - Addlestone Railway Station\nCoffee Shop Home Page • About the Coffee Shop • Acronyms • Station Comparator • Calendar • New Topics • Popular Topics • MRUG • Coffee Shop Forum\nCampaigning Home Page • Campaigning - list of daily tips'

--- id 153946 len 13676 ---
' — Accessibility Notice<|endoftext|>Addlestone Railway Station\nCoffee Shop - Addlestone Railway Station\nCoffee Shop Home Page • About the Coffee Shop • Acronyms • Station Comparator • Calendar • New Topics • Popular Topics • MRUG • Coffee Shop Forum\nCampaignin'

--- id 108673 len 4060 ---
'Here are the list of players included in each of the eight playing team in IPL 2007-08. The tournament will begin on April 18, when Bangalore takes on Kolkata at the Chinnaswamy Stadium in Bangalore. The tournament will feature 59 matches in total, the teams p'

--- id 149512 len 178563 ---
"imbabwe<|endoftext|>HANSON - SHOWING ALL MATCHES :: Census Data Research Online\nCENSUS DATA\nRESEARCH ONLINE\nBROWSING LAST NAME = 'HANSON'\nType a First Name to Search\nHANSON,\nAll First Name Matches are Shown Below:\nAARON HANSON Found: 182, 0.133118% ABBEY HANSO"

--- id 126856 len 178563 ---
"imbabwe<|endoftext|>HANSON - SHOWING ALL MATCHES :: Census Data Research Online\nCENSUS DATA\nRESEARCH ONLINE\nBROWSING LAST NAME = 'HANSON'\nType a First Name to Search\nHANSON,\nAll First Name Matches are Shown Below:\nAARON HANSON Found: 182, 0.133118% ABBEY HANSO"

=== RANK ~5000 ===
--- id 64838 ---
' MOLD man was locked up for defrauding his elderly step-father out of £150,000 – after he was disinherited.\nJohn Whiteford, aged 62, had been entrusted to look after the financial affairs of 77-year-old William Search, w'

--- id 53915 ---
'Liverpool Pembroke Sefton went to the final match of Division 2 of the Northern League at Bebington with a reasonable chance of retaining membership of this highly competitive division but nothing was certain. Although n'

--- id 66087 ---
" Saxony state elections for the AfD, Germany's euroskeptic party, is unsettling for Merkel and her CDU in Berlin. However, the party isn't anything to be afraid of, says political scientist Oskar Niedermayer.\nDW: Despite"

=== BOTTOM 4 (worst that still passed) ===
--- id 156367 len 28587 ---
'.com Latest REAL UFO Sightings: New 2015 UFO Sighting| Latest UFO Sightings\nRead the Latest UFO Sightings\nSearch the Latest UFO Sightings\nTranslate\nWednesday, November 14, 2018\nNew 2015 UFO Sighting\nUFO Sighting in Gävle'

--- id 133711 len 28577 ---
' SkinPress.com<|endoftext|>4UFOS.com Latest REAL UFO Sightings: New 2015 UFO Sighting| Latest UFO Sightings\nRead the Latest UFO Sightings\nSearch the Latest UFO Sightings\nTranslate\nWednesday, November 14, 2018\nNew 2015 UF'

--- id 121310 len 15867 ---
'ancel<|endoftext|>01/08/13 | My Bubbletea Time\nskip to main | skip to sidebar\nPages\nHome\nProjects\nDownloads\nAnime\nMedia\nWallpapers\nThemes\nMy Bubbletea Time\nDaily thoughts of the otaku world, the internet and life in gene'

--- id 143966 len 15867 ---
'ancel<|endoftext|>01/08/13 | My Bubbletea Time\nskip to main | skip to sidebar\nPages\nHome\nProjects\nDownloads\nAnime\nMedia\nWallpapers\nThemes\nMy Bubbletea Time\nDaily thoughts of the otaku world, the internet and life in gene'
[metadata] {"noOutputExpected": false}

#16

narration

The top-ranked docs are junk — long e-commerce boilerplate, forum nav, and repetitive census/name-list tables. Two problems: (1) rare-token gaming (unique name bigrams get huge importance weights), and (2) the pool has exact/near duplicates. Let me fix the scoring with a capped common-vocabulary classifier (interpretable, no rare-token gaming), add dedup, and a digit/table gate.

toolTaskUpdate

args

{
  "taskId": "1",
  "status": "completed"
}

TaskUpdate result

Updated task #1 status

[metadata] {"success": true, "taskId": "1", "updatedFields": ["status"], "statusChange": {"from": "pending", "to": "completed"}}

#17

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolTaskUpdate

args

{
  "taskId": "2",
  "status": "in_progress"
}

TaskUpdate result

Updated task #2 status

[metadata] {"success": true, "taskId": "2", "updatedFields": ["status"], "statusChange": {"from": "pending", "to": "in_progress"}}

#18

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolWrite

args

{
  "file_path": "/workspace/submission/curate.py",
  "content": "#!/usr/bin/env python3\n\"\"\"Curate the raw web pool for from-scratch LM pretraining under a fixed token budget.\n\nSTATED CRITERION\n----------------\nThe disclosed evaluation target is a BROAD, multi-domain high-quality English\ndistribution (equal parts Wikipedia/encyclopedic, general HQ web prose, news, and\ntechnical Q&A).  We rank every pool document by how much its *content* resembles\nthat target distribution, drop near-duplicates, and drop non-prose boilerplate.\n\nScoring is DSIR-style importance weighting (Xie et al. 2023) on a FIXED, common\nbag-of-words vocabulary:\n\n  * Vocabulary = the most frequent word unigrams and bigrams in the pool\n    (lower-cased [a-z']+ runs).  Restricting to *common* features is deliberate:\n    it makes every feature's target/pool frequency ratio well-estimated and keeps\n    rare, near-unique tokens (proper-name lists, table cells) from dominating the\n    score with sampling-noise weights.\n  * For each feature f, weight(f) = clip( log P_target(f) - log P_pool(f) ), where\n    P_target is estimated from the decoded target sample (`multi_dev.npy`, the\n    disclosed target) and P_pool from a random pool background sample.\n  * score(doc) = mean_{f in vocab-features(doc)} weight(f).\n\nHigh score  = vocabulary/register of the multi-domain HQ target (news, encyclopedic\nprose, technical Q&A).  Low score = pool-typical junk vocabulary (shopping-cart /\nsign-in / navigation / SEO boilerplate).  Because features are common content words\nand phrases, the criterion keys on register, not on the target sample's surface\nformatting, so it transfers to the disjoint hidden target of the same domain.\n\nQuality gate (drop): fragments (too short / too few in-vocab features), digit/table\ndumps (high digit ratio), navigation-menu boilerplate (mostly tiny lines), and\nheavily repeated-line documents.  Exact and near-duplicate documents are removed,\nkeeping the single highest-scoring copy.\n\nOutput: `selection.json` — surviving ids sorted best-first.  The frozen training\npipeline consumes this order until the 12M-token budget is filled.\n\"\"\"\nimport argparse, json, re, sys, math\nfrom collections import Counter\nimport numpy as np\n\nPOOL = \"/workspace/data/pool.jsonl\"\nDEV_NPY = \"/workspace/data/multi_dev.npy\"     # disclosed target sample (GPT-2 token ids)\nOUT = \"/workspace/submission/selection.json\"\n\n# ------------------------------ config ---------------------------------------\nN_UNI = 40000                 # keep this many most-frequent unigrams\nN_BI = 40000                  # keep this many most-frequent bigrams\nBG_SAMPLE = 60000             # pool docs for background LM + vocab frequencies\nALPHA = 1.0                   # add-alpha smoothing\nCLIP = 4.0                    # clip per-feature log-weight to [-CLIP, +CLIP]\nMIN_FEATS = 20                # doc must have >= this many in-vocab features to score\nWORD_RE = re.compile(r\"[a-z']+\")\n\n# quality gate thresholds\nMIN_CHARS = 200\nMIN_WORDS = 60\nMAX_DIGIT_FRAC = 0.15         # table / census / listing dumps\nMAX_SHORT_LINE_FRAC = 0.66    # navigation-menu / listing boilerplate\nMAX_DUP_LINE_FRAC = 0.50      # heavily repeated lines\n\n\ndef words_of(text):\n    return WORD_RE.findall(text.lower())\n\n\ndef quality_ok(text, words):\n    if len(text) < MIN_CHARS or len(words) < MIN_WORDS:\n        return False\n    ndig = sum(c.isdigit() for c in text)\n    if ndig / len(text) > MAX_DIGIT_FRAC:\n        return False\n    lines = [ln for ln in text.split(\"\\n\") if ln.strip()]\n    if len(lines) >= 8:\n        short = sum(1 for ln in lines if len(ln.split()) <= 3)\n        if short / len(lines) > MAX_SHORT_LINE_FRAC:\n            return False\n        if 1 - len(set(lines)) / len(lines) > MAX_DUP_LINE_FRAC:\n            return False\n    return True\n\n\ndef main():\n    ap = argparse.ArgumentParser()\n    ap.add_argument(\"--out\", default=OUT)\n    ap.add_argument(\"--seed\", type=int, default=0)\n    ap.add_argument(\"--min_score\", type=float, default=None)\n    args = ap.parse_args()\n    rng = np.random.default_rng(args.seed)\n\n    # ---- 1. load pool -------------------------------------------------------\n    print(\"loading pool...\", file=sys.stderr)\n    ids, texts = [], []\n    with open(POOL) as f:\n        for line in f:\n            r = json.loads(line)\n            ids.append(r[\"id\"]); texts.append(r[\"text\"])\n    N = len(ids)\n    print(f\"  {N} docs\", file=sys.stderr)\n\n    # ---- 2. background sample: build common vocab + pool counts -------------\n    print(\"building vocabulary + background counts...\", file=sys.stderr)\n    bg_idx = rng.choice(N, size=min(BG_SAMPLE, N), replace=False)\n    uni_bg, bi_bg = Counter(), Counter()\n    for j in bg_idx:\n        w = words_of(texts[j])\n        uni_bg.update(w)\n        bi_bg.update(zip(w, w[1:]))\n    uni_vocab = [w for w, _ in uni_bg.most_common(N_UNI)]\n    bi_vocab = [b for b, _ in bi_bg.most_common(N_BI)]\n    uni_ix = {w: k for k, w in enumerate(uni_vocab)}\n    bi_ix = {b: N_UNI + k for k, b in enumerate(bi_vocab)}\n    Vf = N_UNI + N_BI\n    print(f\"  vocab: {len(uni_ix)} unigrams + {len(bi_ix)} bigrams\", file=sys.stderr)\n\n    bg_counts = np.zeros(Vf, dtype=np.float64)\n    for w, c in uni_bg.items():\n        k = uni_ix.get(w)\n        if k is not None: bg_counts[k] = c\n    for b, c in bi_bg.items():\n        k = bi_ix.get(b)\n        if k is not None: bg_counts[k] = c\n\n    # ---- 3. target counts on the same vocab --------------------------------\n    print(\"decoding target sample + counting...\", file=sys.stderr)\n    from transformers import AutoTokenizer\n    tok = AutoTokenizer.from_pretrained(\"gpt2\")\n    dev = np.load(DEV_NPY).astype(np.int64)\n    EOS = 50256\n    idx = np.where(dev == EOS)[0]\n    segs, prev = [], 0\n    for i in idx:\n        if i > prev: segs.append((prev, i))\n        prev = i + 1\n    if prev < len(dev): segs.append((prev, len(dev)))\n    tgt_counts = np.zeros(Vf, dtype=np.float64)\n    for s, e in segs:\n        w = words_of(tok.decode(dev[s:e].tolist()))\n        for x in w:\n            k = uni_ix.get(x)\n            if k is not None: tgt_counts[k] += 1\n        for x in zip(w, w[1:]):\n            k = bi_ix.get(x)\n            if k is not None: tgt_counts[k] += 1\n    print(f\"  target: {len(segs)} segments\", file=sys.stderr)\n\n    # ---- 4. per-feature clipped log-likelihood-ratio weights ---------------\n    T = tgt_counts.sum(); B = bg_counts.sum()\n    log_tgt = np.log(tgt_counts + ALPHA) - math.log(T + ALPHA * Vf)\n    log_bg = np.log(bg_counts + ALPHA) - math.log(B + ALPHA * Vf)\n    weight = np.clip(log_tgt - log_bg, -CLIP, CLIP)\n\n    # interpretability: most target-ish / most junk-ish words\n    inv_uni = {k: w for w, k in uni_ix.items()}\n    uni_w = [(weight[k], inv_uni[k]) for k in range(N_UNI)]\n    uni_w.sort()\n    print(\"  most JUNK unigrams:\", [w for _, w in uni_w[:15]], file=sys.stderr)\n    print(\"  most TARGET unigrams:\", [w for _, w in uni_w[-15:]], file=sys.stderr)\n\n    # ---- 5. score + gate every document ------------------------------------\n    print(\"scoring all documents...\", file=sys.stderr)\n    scores = np.full(N, -1e9, dtype=np.float64)\n    for k in range(N):\n        text = texts[k]\n        w = words_of(text)\n        if not quality_ok(text, w):\n            continue\n        feats = [uni_ix[x] for x in w if x in uni_ix]\n        bg = bi_ix\n        prev = None\n        for x in w:\n            if prev is not None:\n                bi = bg.get((prev, x))\n                if bi is not None: feats.append(bi)\n            prev = x\n        if len(feats) < MIN_FEATS:\n            continue\n        scores[k] = weight[np.asarray(feats)].mean()\n        if (k + 1) % 40000 == 0:\n            print(f\"  {k+1}/{N}\", file=sys.stderr)\n    ok_mask = scores > -1e8\n    print(f\"  {int(ok_mask.sum())}/{N} docs passed the quality gate\", file=sys.stderr)\n\n    # ---- 6. rank, dedup, emit ----------------------------------------------\n    order = np.argsort(-scores)\n    order = order[ok_mask[order]]\n    if args.min_score is not None:\n        order = order[scores[order] >= args.min_score]\n\n    seen_full, seen_sig, sel_ids = set(), set(), []\n    ws = re.compile(r\"\\s+\")\n    dropped_dup = 0\n    for k in order:\n        t = texts[k]\n        full = hash(ws.sub(\" \", t).strip())\n        # near-dup signature: interior slice, robust to differing nav prefixes\n        core = ws.sub(\" \", t)[120:920]\n        sig = hash(core) if len(core) > 200 else full\n        if full in seen_full or sig in seen_sig:\n            dropped_dup += 1\n            continue\n        seen_full.add(full); seen_sig.add(sig)\n        sel_ids.append(int(ids[k]))\n    print(f\"  dropped {dropped_dup} (near-)duplicate docs\", file=sys.stderr)\n\n    ss = np.sort(scores[ok_mask])\n    for p in [50, 75, 90, 95, 99, 100]:\n        print(f\"  score p{p}: {ss[min(len(ss)-1, int(p/100*len(ss)))]:.4f}\", file=sys.stderr)\n\n    json.dump(sel_ids, open(args.out, \"w\"))\n    print(f\"wrote {len(sel_ids)} ids -> {args.out}\", file=sys.stderr)\n\n\nif __name__ == \"__main__\":\n    main()\n"
}

Write result

The file /workspace/submission/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"type": "update", "filePath": "/workspace/submission/curate.py", "content": "#!/usr/bin/env python3\n\"\"\"Curate the raw web pool for from-scratch LM pretraining under a fixed token budget.\n\nSTATED CRITERION\n----------------\nThe disclosed evaluation target is a BROAD, multi-domain high-quality English\ndistribution (equal parts Wikipedia/encyclopedic, general HQ web prose, news, and\ntechnical Q&A).  We rank every pool document by how much its *content* resembles\nthat target distribution, drop near-duplicates, and drop non-prose boilerplate.\n\nScoring is DSIR-style importance weighting (Xie et al. 2023) on a FIXED, common\nbag-of-words vocabulary:\n\n  * Vocabulary = the most frequent word unigrams and bigrams in the pool\n    (lower-cased [a-z']+ runs).  Restricting to *common* features is deliberate:\n    it makes every feature's target/pool frequency ratio well-estimated and keeps\n    rare, near-unique tokens (proper-name lists, table cells) from dominating the\n    score with sampling-noise weights.\n  * For each feature f, weight(f) = clip( log P_target(f) - log P_pool(f) ), where\n    P_target is estimated from the decoded target sample (`multi_dev.npy`, the\n    disclosed target) and P_pool from a random pool background sample.\n  * score(doc) = mean_{f in vocab-features(doc)} weight(f).\n\nHigh score  = vocabulary/register of the multi-domain HQ target (news, encyclopedic\nprose, technical Q&A).  Low score = pool-typical junk vocabulary (shopping-cart /\nsign-in / navigation / SEO boilerplate).  Because features are common content words\nand phrases, the criterion keys on register, not on the target sample's surface\nformatting, so it transfers to the disjoint hidden target of the same domain.\n\nQuality gate (drop): fragments (too short / too few in-vocab features), digit/table\ndumps (high digit ratio), navigation-menu boilerplate (mostly tiny lines), and\nheavily repeated-line documents.  Exact and near-duplicate documents are removed,\nkeeping the single highest-scoring copy.\n\nOutput: `selection.json` — surviving ids sorted best-first.  The frozen training\npipeline consumes this order until the 12M-token budget is filled.\n\"\"\"\nimport argparse, json, re, sys, math\nfrom collections import Counter\nimport numpy as np\n\nPOOL = \"/workspace/data/pool.jsonl\"\nDEV_NPY = \"/workspace/data/multi_dev.npy\"     # disclosed target sample (GPT-2 token ids)\nOUT = \"/workspace/submission/selection.json\"\n\n# ------------------------------ config ---------------------------------------\nN_UNI = 40000                 # keep this many most-frequent unigrams\nN_BI = 40000                  # keep this many most-frequent bigrams\nBG_SAMPLE = 60000             # pool docs for background LM + vocab frequencies\nALPHA = 1.0                   # add-alpha smoothing\nCLIP = 4.0                    # clip per-feature log-weight to [-CLIP, +CLIP]\nMIN_FEATS = 20                # doc must have >= this many in-vocab features to score\nWORD_RE = re.compile(r\"[a-z']+\")\n\n# quality gate thresholds\nMIN_CHARS = 200\nMIN_WORDS = 60\nMAX_DIGIT_FRAC = 0.15         # table / census / listing dumps\nMAX_SHORT_LINE_FRAC = 0.66    # navigation-menu / listing boilerplate\nMAX_DUP_LINE_FRAC = 0.50      # heavily repeated lines\n\n\ndef words_of(text):\n    return WORD_RE.findall(text.lower())\n\n\ndef quality_ok(text, words):\n    if len(text) < MIN_CHARS or len(words) < MIN_WORDS:\n        return False\n    ndig = sum(c.isdigit() for c in text)\n    if ndig / len(text) > MAX_DIGIT_FRAC:\n        return False\n    lines = [ln for ln in text.split(\"\\n\") if ln.strip()]\n    if len(lines) >= 8:\n        short = sum(1 for ln in lines if len(ln.split()) <= 3)\n        if short / len(lines) > MAX_SHORT_LINE_FRAC:\n            return False\n        if 1 - len(set(lines)) / len(lines) > MAX_DUP_LINE_FRAC:\n            return False\n    return True\n\n\ndef main():\n    ap = argparse.ArgumentParser()\n    ap.add_argument(\"--out\", default=OUT)\n    ap.add_argument(\"--seed\", type=int, default=0)\n    ap.add_argument(\"--min_score\", type=float, default=None)\n    args = ap.parse_args()\n    rng = np.random.default_rng(args.seed)\n\n    # ---- 1. load pool -------------------------------------------------------\n    print(\"loading pool...\", file=sys.stderr)\n    ids, texts = [], []\n    with open(POOL) as f:\n        for line in f:\n            r = json.loads(line)\n            ids.append(r[\"id\"]); texts.append(r[\"text\"])\n    N = len(ids)\n    print(f\"  {N} docs\", file=sys.stderr)\n\n    # ---- 2. background sample: build common vocab + pool counts -------------\n    print(\"building vocabulary + background counts...\", file=sys.stderr)\n    bg_idx = rng.choice(N, size=min(BG_SAMPLE, N), replace=False)\n    uni_bg, bi_bg = Counter(), Counter()\n    for j in bg_idx:\n        w = words_of(texts[j])\n        uni_bg.update(w)\n        bi_bg.update(zip(w, w[1:]))\n    uni_vocab = [w for w, _ in uni_bg.most_common(N_UNI)]\n    bi_vocab = [b for b, _ in bi_bg.most_common(N_BI)]\n    uni_ix = {w: k for k, w in enumerate(uni_vocab)}\n    bi_ix = {b: N_UNI + k for k, b in enumerate(bi_vocab)}\n    Vf = N_UNI + N_BI\n    print(f\"  vocab: {len(uni_ix)} unigrams + {len(bi_ix)} bigrams\", file=sys.stderr)\n\n    bg_counts = np.zeros(Vf, dtype=np.float64)\n    for w, c in uni_bg.items():\n        k = uni_ix.get(w)\n        if k is not None: bg_counts[k] = c\n    for b, c in bi_bg.items():\n        k = bi_ix.get(b)\n        if k is not None: bg_counts[k] = c\n\n    # ---- 3. target counts on the same vocab --------------------------------\n    print(\"decoding target sample + counting...\", file=sys.stderr)\n    from transformers import AutoTokenizer\n    tok = AutoTokenizer.from_pretrained(\"gpt2\")\n    dev = np.load(DEV_NPY).astype(np.int64)\n    EOS = 50256\n    idx = np.where(dev == EOS)[0]\n    segs, prev = [], 0\n    for i in idx:\n        if i > prev: segs.append((prev, i))\n        prev = i + 1\n    if prev < len(dev): segs.append((prev, len(dev)))\n    tgt_counts = np.zeros(Vf, dtype=np.float64)\n    for s, e in segs:\n        w = words_of(tok.decode(dev[s:e].tolist()))\n        for x in w:\n            k = uni_ix.get(x)\n            if k is not None: tgt_counts[k] += 1\n        for x in zip(w, w[1:]):\n            k = bi_ix.get(x)\n            if k is not None: tgt_counts[k] += 1\n    print(f\"  target: {len(segs)} segments\", file=sys.stderr)\n\n    # ---- 4. per-feature clipped log-likelihood-ratio weights ---------------\n    T = tgt_counts.sum(); B = bg_counts.sum()\n    log_tgt = np.log(tgt_counts + ALPHA) - math.log(T + ALPHA * Vf)\n    log_bg = np.log(bg_counts + ALPHA) - math.log(B + ALPHA * Vf)\n    weight = np.clip(log_tgt - log_bg, -CLIP, CLIP)\n\n    # interpretability: most target-ish / most junk-ish words\n    inv_uni = {k: w for w, k in uni_ix.items()}\n    uni_w = [(weight[k], inv_uni[k]) for k in range(N_UNI)]\n    uni_w.sort()\n    print(\"  most JUNK unigrams:\", [w for _, w in uni_w[:15]], file=sys.stderr)\n    print(\"  most TARGET unigrams:\", [w for _, w in uni_w[-15:]], file=sys.stderr)\n\n    # ---- 5. score + gate every document ------------------------------------\n    print(\"scoring all documents...\", file=sys.stderr)\n    scores = np.full(N, -1e9, dtype=np.float64)\n    for k in range(N):\n        text = texts[k]\n        w = words_of(text)\n        if not quality_ok(text, w):\n            continue\n        feats = [uni_ix[x] for x in w if x in uni_ix]\n        bg = bi_ix\n        prev = None\n        for x in w:\n            if prev is not None:\n                bi = bg.get((prev, x))\n                if bi is not None: feats.append(bi)\n            prev = x\n        if len(feats) < MIN_FEATS:\n            continue\n        scores[k] = weight[np.asarray(feats)].mean()\n        if (k + 1) % 40000 == 0:\n            print(f\"  {k+1}/{N}\", file=sys.stderr)\n    ok_mask = scores > -1e8\n    print(f\"  {int(ok_mask.sum())}/{N} docs passed the quality gate\", file=sys.stderr)\n\n    # ---- 6. rank, dedup, emit ----------------------------------------------\n    order = np.argsort(-scores)\n    order = order[ok_mask[order]]\n    if args.min_score is not None:\n        order = order[scores[order] >= args.min_score]\n\n    seen_full, seen_sig, sel_ids = set(), set(), []\n    ws = re.compile(r\"\\s+\")\n    dropped_dup = 0\n    for k in order:\n        t = texts[k]\n        full = hash(ws.sub(\" \", t).strip())\n        # near-dup signature: interior slice, robust to differing nav prefixes\n        core = ws.sub(\" \", t)[120:920]\n        sig = hash(core) if len(core) > 200 else full\n        if full in seen_full or sig in seen_sig:\n            dropped_dup += 1\n            continue\n        seen_full.add(full); seen_sig.add(sig)\n        sel_ids.append(int(ids[k]))\n    print(f\"  dropped {dropped_dup} (near-)duplicate docs\", file=sys.stderr)\n\n    ss = np.sort(scores[ok_mask])\n    for p in [50, 75, 90, 95, 99, 100]:\n        print(f\"  score p{p}: {ss[min(len(ss)-1, int(p/100*len(ss)))]:.4f}\", file=sys.stderr)\n\n    json.dump(sel_ids, open(args.out, \"w\"))\n    print(f\"wrote {len(sel_ids)} ids -> {args.out}\", file=sys.stderr)\n\n\nif __name__ == \"__main__\":\n    main()\n", "structuredPatch": [{"oldStart": 5, "oldLines": 95, "newStart": 5, "newLines": 77, "lines": [" ----------------", " The disclosed evaluation target is a BROAD, multi-domain high-quality English", " distribution (equal parts Wikipedia/encyclopedic, general HQ web prose, news, and", "-technical Q&A).  We therefore rank every pool document by how much its *content*", "-resembles that target distribution, and drop obvious non-prose boilerplate.", "+technical Q&A).  We rank every pool document by how much its *content* resembles", "+that target distribution, drop near-duplicates, and drop non-prose boilerplate.", " ", "-Concretely we use DSIR-style importance weighting (Xie et al. 2023): estimate a", "-bag-of-hashed-n-grams language model for the TARGET (decoded from the provided", "-`multi_dev.npy`, the disclosed target sample) and for the POOL background, and", "-score each document by its mean per-token log-likelihood ratio", "+Scoring is DSIR-style importance weighting (Xie et al. 2023) on a FIXED, common", "+bag-of-words vocabulary:", " ", "-        score(doc) = mean_{f in ngrams(doc)}  [ log P_target(f) - log P_pool(f) ].", "+  * Vocabulary = the most frequent word unigrams and bigrams in the pool", "+    (lower-cased [a-z']+ runs).  Restricting to *common* features is deliberate:", "+    it makes every feature's target/pool frequency ratio well-estimated and keeps", "+    rare, near-unique tokens (proper-name lists, table cells) from dominating the", "+    score with sampling-noise weights.", "+  * For each feature f, weight(f) = clip( log P_target(f) - log P_pool(f) ), where", "+    P_target is estimated from the decoded target sample (`multi_dev.npy`, the", "+    disclosed target) and P_pool from a random pool background sample.", "+  * score(doc) = mean_{f in vocab-features(doc)} weight(f).", " ", "-Documents whose vocabulary looks like the multi-domain HQ target score high;", "-documents dominated by pool-typical junk (SEO/e-commerce boilerplate, forum cruft,", "-navigation menus) score low.  Features are word unigrams+bigrams (lower-cased", "-[a-z']+ runs), so the score keys on *content/register*, not on the target sample's", "-surface formatting — it transfers to the disjoint hidden target of the same domain.", "+High score  = vocabulary/register of the multi-domain HQ target (news, encyclopedic", "+prose, technical Q&A).  Low score = pool-typical junk vocabulary (shopping-cart /", "+sign-in / navigation / SEO boilerplate).  Because features are common content words", "+and phrases, the criterion keys on register, not on the target sample's surface", "+formatting, so it transfers to the disjoint hidden target of the same domain.", " ", "-A light quality gate removes fragments and boilerplate that the ratio alone can", "-rank too high (very short docs, navigation-menu pages with mostly tiny lines, and", "-highly repetitive documents).", "+Quality gate (drop): fragments (too short / too few in-vocab features), digit/table", "+dumps (high digit ratio), navigation-menu boilerplate (mostly tiny lines), and", "+heavily repeated-line documents.  Exact and near-duplicate documents are removed,", "+keeping the single highest-scoring copy.", " ", "-Output: `selection.json` — every surviving id, sorted best-first.  The frozen", "-training pipeline consumes this order until the 12M-token budget is filled.", "+Output: `selection.json` — surviving ids sorted best-first.  The frozen training", "+pipeline consumes this order until the 12M-token budget is filled.", " \"\"\"", "-import argparse, json, re, zlib, sys, math", "+import argparse, json, re, sys, math", "+from collections import Counter", " import numpy as np", " ", " POOL = \"/workspace/data/pool.jsonl\"", "-DEV_NPY = \"/workspace/data/multi_dev.npy\"   # disclosed target sample (GPT-2 token ids)", "+DEV_NPY = \"/workspace/data/multi_dev.npy\"     # disclosed target sample (GPT-2 token ids)", " OUT = \"/workspace/submission/selection.json\"", " ", " # ------------------------------ config ---------------------------------------", "-HASH_BITS = 20                 # 2^20 = ~1M feature buckets", "-MASK = (1 << HASH_BITS) - 1", "-BG_SAMPLE = 60000              # pool docs used to estimate the background n-gram LM", "-ALPHA = 1.0                    # add-alpha smoothing on bucket counts", "-BIGRAM_MULT = 1000003          # mix constant for combining two unigram hashes", "+N_UNI = 40000                 # keep this many most-frequent unigrams", "+N_BI = 40000                  # keep this many most-frequent bigrams", "+BG_SAMPLE = 60000             # pool docs for background LM + vocab frequencies", "+ALPHA = 1.0                   # add-alpha smoothing", "+CLIP = 4.0                    # clip per-feature log-weight to [-CLIP, +CLIP]", "+MIN_FEATS = 20                # doc must have >= this many in-vocab features to score", " WORD_RE = re.compile(r\"[a-z']+\")", "-LINE_RE = re.compile(r\"[^\\n]*\\n?\")", " ", " # quality gate thresholds", " MIN_CHARS = 200", "-MIN_WORDS = 50", "-MAX_SHORT_LINE_FRAC = 0.66     # navigation-menu / listing boilerplate", "-MAX_DUP_LINE_FRAC = 0.50       # heavily repeated lines", "+MIN_WORDS = 60", "+MAX_DIGIT_FRAC = 0.15         # table / census / listing dumps", "+MAX_SHORT_LINE_FRAC = 0.66    # navigation-menu / listing boilerplate", "+MAX_DUP_LINE_FRAC = 0.50      # heavily repeated lines", " ", "-STOPWORDS = set(\"\"\"the of and to a in is that it for on with as was be by this are at from or an", "-which not but have has had they you we he she his her their its will would can could been being", "-i my me your our us them do does did so if then than out up about into over after \"\"\".split())", " ", "+def words_of(text):", "+    return WORD_RE.findall(text.lower())", " ", "-def word_hashes(words, cache):", "-    \"\"\"crc32-based deterministic hash per word (cached).\"\"\"", "-    out = []", "-    ap = out.append", "-    for w in words:", "-        h = cache.get(w)", "-        if h is None:", "-            h = zlib.crc32(w.encode()) & 0xffffffff", "-            cache[w] = h", "-        ap(h)", "-    return out", " ", "-", "-def doc_feature_buckets(words, cache):", "-    \"\"\"unigram + bigram hashed buckets for a document (as a python list of ints).\"\"\"", "-    uh = word_hashes(words, cache)", "-    buckets = [h & MASK for h in uh]", "-    for i in range(len(uh) - 1):", "-        buckets.append(((uh[i] * BIGRAM_MULT + uh[i + 1]) & MASK))", "-    return buckets", "-", "-", "-def accumulate_counts(bucket_lists, counts):", "-    for b in bucket_lists:", "-        if b:", "-            np.add.at(counts, np.asarray(b, dtype=np.int64), 1.0)", "-", "-", " def quality_ok(text, words):", "-    \"\"\"Light boilerplate/fragment gate. Returns (ok: bool).\"\"\"", "     if len(text) < MIN_CHARS or len(words) < MIN_WORDS:", "         return False", "+    ndig = sum(c.isdigit() for c in text)", "+    if ndig / len(text) > MAX_DIGIT_FRAC:", "+        return False", "     lines = [ln for ln in text.split(\"\\n\") if ln.strip()]", "     if len(lines) >= 8:", "         short = sum(1 for ln in lines if len(ln.split()) <= 3)", "         if short / len(lines) > MAX_SHORT_LINE_FRAC:", "             return False", "-        uniq = len(set(lines))", "-        if 1 - uniq / len(lines) > MAX_DUP_LINE_FRAC:", "+        if 1 - len(set(lines)) / len(lines) > MAX_DUP_LINE_FRAC:", "             return False", "     return True", " "]}, {"oldStart": 102, "oldLines": 8, "newStart": 84, "newLines": 7, "lines": ["     ap = argparse.ArgumentParser()", "     ap.add_argument(\"--out\", default=OUT)", "     ap.add_argument(\"--seed\", type=int, default=0)", "-    ap.add_argument(\"--min_score\", type=float, default=None,", "-                    help=\"optional hard score cutoff; default keeps all gated docs\")", "+    ap.add_argument(\"--min_score\", type=float, default=None)", "     args = ap.parse_args()", "     rng = np.random.default_rng(args.seed)", " "]}, {"oldStart": 114, "oldLines": 84, "newStart": 95, "newLines": 117, "lines": ["         for line in f:", "             r = json.loads(line)", "             ids.append(r[\"id\"]); texts.append(r[\"text\"])", "-    ids = np.array(ids)", "     N = len(ids)", "     print(f\"  {N} docs\", file=sys.stderr)", " ", "-    # ---- 2. build TARGET n-gram LM from decoded dev sample ------------------", "-    print(\"decoding target sample + building target LM...\", file=sys.stderr)", "+    # ---- 2. background sample: build common vocab + pool counts -------------", "+    print(\"building vocabulary + background counts...\", file=sys.stderr)", "+    bg_idx = rng.choice(N, size=min(BG_SAMPLE, N), replace=False)", "+    uni_bg, bi_bg = Counter(), Counter()", "+    for j in bg_idx:", "+        w = words_of(texts[j])", "+        uni_bg.update(w)", "+        bi_bg.update(zip(w, w[1:]))", "+    uni_vocab = [w for w, _ in uni_bg.most_common(N_UNI)]", "+    bi_vocab = [b for b, _ in bi_bg.most_common(N_BI)]", "+    uni_ix = {w: k for k, w in enumerate(uni_vocab)}", "+    bi_ix = {b: N_UNI + k for k, b in enumerate(bi_vocab)}", "+    Vf = N_UNI + N_BI", "+    print(f\"  vocab: {len(uni_ix)} unigrams + {len(bi_ix)} bigrams\", file=sys.stderr)", "+", "+    bg_counts = np.zeros(Vf, dtype=np.float64)", "+    for w, c in uni_bg.items():", "+        k = uni_ix.get(w)", "+        if k is not None: bg_counts[k] = c", "+    for b, c in bi_bg.items():", "+        k = bi_ix.get(b)", "+        if k is not None: bg_counts[k] = c", "+", "+    # ---- 3. target counts on the same vocab --------------------------------", "+    print(\"decoding target sample + counting...\", file=sys.stderr)", "     from transformers import AutoTokenizer", "     tok = AutoTokenizer.from_pretrained(\"gpt2\")", "     dev = np.load(DEV_NPY).astype(np.int64)", "     EOS = 50256", "-    # split into documents on EOS, decode each, featurize", "     idx = np.where(dev == EOS)[0]", "     segs, prev = [], 0", "     for i in idx:", "         if i > prev: segs.append((prev, i))", "         prev = i + 1", "     if prev < len(dev): segs.append((prev, len(dev)))", "-    tgt_counts = np.zeros(MASK + 1, dtype=np.float64)", "-    cache = {}", "-    tgt_bucket_lists = []", "+    tgt_counts = np.zeros(Vf, dtype=np.float64)", "     for s, e in segs:", "-        txt = tok.decode(dev[s:e].tolist())", "-        words = WORD_RE.findall(txt.lower())", "-        tgt_bucket_lists.append(doc_feature_buckets(words, cache))", "-    accumulate_counts(tgt_bucket_lists, tgt_counts)", "-    print(f\"  target: {len(segs)} segments, {tgt_counts.sum():.0f} features\", file=sys.stderr)", "+        w = words_of(tok.decode(dev[s:e].tolist()))", "+        for x in w:", "+            k = uni_ix.get(x)", "+            if k is not None: tgt_counts[k] += 1", "+        for x in zip(w, w[1:]):", "+            k = bi_ix.get(x)", "+            if k is not None: tgt_counts[k] += 1", "+    print(f\"  target: {len(segs)} segments\", file=sys.stderr)", " ", "-    # ---- 3. build POOL background n-gram LM from a random sample ------------", "-    print(\"building pool background LM...\", file=sys.stderr)", "-    bg_idx = rng.choice(N, size=min(BG_SAMPLE, N), replace=False)", "-    bg_counts = np.zeros(MASK + 1, dtype=np.float64)", "-    bg_lists = []", "-    for j in bg_idx:", "-        words = WORD_RE.findall(texts[j].lower())", "-        bg_lists.append(doc_feature_buckets(words, cache))", "-        if len(bg_lists) >= 4000:", "-            accumulate_counts(bg_lists, bg_counts); bg_lists = []", "-    accumulate_counts(bg_lists, bg_counts)", "-    print(f\"  background: {len(bg_idx)} docs, {bg_counts.sum():.0f} features\", file=sys.stderr)", "+    # ---- 4. per-feature clipped log-likelihood-ratio weights ---------------", "+    T = tgt_counts.sum(); B = bg_counts.sum()", "+    log_tgt = np.log(tgt_counts + ALPHA) - math.log(T + ALPHA * Vf)", "+    log_bg = np.log(bg_counts + ALPHA) - math.log(B + ALPHA * Vf)", "+    weight = np.clip(log_tgt - log_bg, -CLIP, CLIP)", " ", "-    # ---- 4. per-bucket log-likelihood-ratio weights ------------------------", "-    T = tgt_counts.sum(); B = bg_counts.sum(); V = MASK + 1", "-    log_tgt = np.log(tgt_counts + ALPHA) - math.log(T + ALPHA * V)", "-    log_bg = np.log(bg_counts + ALPHA) - math.log(B + ALPHA * V)", "-    weight = log_tgt - log_bg          # importance log-weight per feature bucket", "+    # interpretability: most target-ish / most junk-ish words", "+    inv_uni = {k: w for w, k in uni_ix.items()}", "+    uni_w = [(weight[k], inv_uni[k]) for k in range(N_UNI)]", "+    uni_w.sort()", "+    print(\"  most JUNK unigrams:\", [w for _, w in uni_w[:15]], file=sys.stderr)", "+    print(\"  most TARGET unigrams:\", [w for _, w in uni_w[-15:]], file=sys.stderr)", " ", "     # ---- 5. score + gate every document ------------------------------------", "     print(\"scoring all documents...\", file=sys.stderr)", "     scores = np.full(N, -1e9, dtype=np.float64)", "-    ok_mask = np.zeros(N, dtype=bool)", "     for k in range(N):", "         text = texts[k]", "-        words = WORD_RE.findall(text.lower())", "-        if not quality_ok(text, words):", "+        w = words_of(text)", "+        if not quality_ok(text, w):", "             continue", "-        b = doc_feature_buckets(words, cache)", "-        if not b:", "+        feats = [uni_ix[x] for x in w if x in uni_ix]", "+        bg = bi_ix", "+        prev = None", "+        for x in w:", "+            if prev is not None:", "+                bi = bg.get((prev, x))", "+                if bi is not None: feats.append(bi)", "+            prev = x", "+        if len(feats) < MIN_FEATS:", "             continue", "-        scores[k] = weight[np.asarray(b, dtype=np.int64)].mean()", "-        ok_mask[k] = True", "-        if (k + 1) % 20000 == 0:", "+        scores[k] = weight[np.asarray(feats)].mean()", "+        if (k + 1) % 40000 == 0:", "             print(f\"  {k+1}/{N}\", file=sys.stderr)", "+    ok_mask = scores > -1e8", "+    print(f\"  {int(ok_mask.sum())}/{N} docs passed the quality gate\", file=sys.stderr)", " ", "-    n_ok = int(ok_mask.sum())", "-    print(f\"  {n_ok}/{N} docs passed the quality gate\", file=sys.stderr)", "-", "-    # ---- 6. rank + emit ----------------------------------------------------", "-    order = np.argsort(-scores)                 # best first", "-    order = order[ok_mask[order]]               # keep only gated docs", "+    # ---- 6. rank, dedup, emit ----------------------------------------------", "+    order = np.argsort(-scores)", "+    order = order[ok_mask[order]]", "     if args.min_score is not None:", "         order = order[scores[order] >= args.min_score]", "-    sel_ids = ids[order].tolist()", " ", "-    # diagnostics: score distribution", "+    seen_full, seen_sig, sel_ids = set(), set(), []", "+    ws = re.compile(r\"\\s+\")", "+    dropped_dup = 0", "+    for k in order:", "+        t = texts[k]", "+        full = hash(ws.sub(\" \", t).strip())", "+        # near-dup signature: interior slice, robust to differing nav prefixes", "+        core = ws.sub(\" \", t)[120:920]", "+        sig = hash(core) if len(core) > 200 else full", "+        if full in seen_full or sig in seen_sig:", "+            dropped_dup += 1", "+            continue", "+        seen_full.add(full); seen_sig.add(sig)", "+        sel_ids.append(int(ids[k]))", "+    print(f\"  dropped {dropped_dup} (near-)duplicate docs\", file=sys.stderr)", "+", "     ss = np.sort(scores[ok_mask])", "     for p in [50, 75, 90, 95, 99, 100]:", "-        v = ss[min(len(ss) - 1, int(p / 100 * len(ss)))]", "-        print(f\"  score p{p}: {v:.4f}\", file=sys.stderr)", "+        print(f\"  score p{p}: {ss[min(len(ss)-1, int(p/100*len(ss)))]:.4f}\", file=sys.stderr)", " ", "     json.dump(sel_ids, open(args.out, \"w\"))", "     print(f\"wrote {len(sel_ids)} ids -> {args.out}\", file=sys.stderr)"]}], "originalFile": "#!/usr/bin/env python3\n\"\"\"Curate the raw web pool for from-scratch LM pretraining under a fixed token budget.\n\nSTATED CRITERION\n----------------\nThe disclosed evaluation target is a BROAD, multi-domain high-quality English\ndistribution (equal parts Wikipedia/encyclopedic, general HQ web prose, news, and\ntechnical Q&A).  We therefore rank every pool document by how much its *content*\nresembles that target distribution, and drop obvious non-prose boilerplate.\n\nConcretely we use DSIR-style importance weighting (Xie et al. 2023): estimate a\nbag-of-hashed-n-grams language model for the TARGET (decoded from the provided\n`multi_dev.npy`, the disclosed target sample) and for the POOL background, and\nscore each document by its mean per-token log-likelihood ratio\n\n        score(doc) = mean_{f in ngrams(doc)}  [ log P_target(f) - log P_pool(f) ].\n\nDocuments whose vocabulary looks like the multi-domain HQ target score high;\ndocuments dominated by pool-typical junk (SEO/e-commerce boilerplate, forum cruft,\nnavigation menus) score low.  Features are word unigrams+bigrams (lower-cased\n[a-z']+ runs), so the score keys on *content/register*, not on the target sample's\nsurface formatting — it transfers to the disjoint hidden target of the same domain.\n\nA light quality gate removes fragments and boilerplate that the ratio alone can\nrank too high (very short docs, navigation-menu pages with mostly tiny lines, and\nhighly repetitive documents).\n\nOutput: `selection.json` — every surviving id, sorted best-first.  The frozen\ntraining pipeline consumes this order until the 12M-token budget is filled.\n\"\"\"\nimport argparse, json, re, zlib, sys, math\nimport numpy as np\n\nPOOL = \"/workspace/data/pool.jsonl\"\nDEV_NPY = \"/workspace/data/multi_dev.npy\"   # disclosed target sample (GPT-2 token ids)\nOUT = \"/workspace/submission/selection.json\"\n\n# ------------------------------ config ---------------------------------------\nHASH_BITS = 20                 # 2^20 = ~1M feature buckets\nMASK = (1 << HASH_BITS) - 1\nBG_SAMPLE = 60000              # pool docs used to estimate the background n-gram LM\nALPHA = 1.0                    # add-alpha smoothing on bucket counts\nBIGRAM_MULT = 1000003          # mix constant for combining two unigram hashes\nWORD_RE = re.compile(r\"[a-z']+\")\nLINE_RE = re.compile(r\"[^\\n]*\\n?\")\n\n# quality gate thresholds\nMIN_CHARS = 200\nMIN_WORDS = 50\nMAX_SHORT_LINE_FRAC = 0.66     # navigation-menu / listing boilerplate\nMAX_DUP_LINE_FRAC = 0.50       # heavily repeated lines\n\nSTOPWORDS = set(\"\"\"the of and to a in is that it for on with as was be by this are at from or an\nwhich not but have has had they you we he she his her their its will would can could been being\ni my me your our us them do does did so if then than out up about into over after \"\"\".split())\n\n\ndef word_hashes(words, cache):\n    \"\"\"crc32-based deterministic hash per word (cached).\"\"\"\n    out = []\n    ap = out.append\n    for w in words:\n        h = cache.get(w)\n        if h is None:\n            h = zlib.crc32(w.encode()) & 0xffffffff\n            cache[w] = h\n        ap(h)\n    return out\n\n\ndef doc_feature_buckets(words, cache):\n    \"\"\"unigram + bigram hashed buckets for a document (as a python list of ints).\"\"\"\n    uh = word_hashes(words, cache)\n    buckets = [h & MASK for h in uh]\n    for i in range(len(uh) - 1):\n        buckets.append(((uh[i] * BIGRAM_MULT + uh[i + 1]) & MASK))\n    return buckets\n\n\ndef accumulate_counts(bucket_lists, counts):\n    for b in bucket_lists:\n        if b:\n            np.add.at(counts, np.asarray(b, dtype=np.int64), 1.0)\n\n\ndef quality_ok(text, words):\n    \"\"\"Light boilerplate/fragment gate. Returns (ok: bool).\"\"\"\n    if len(text) < MIN_CHARS or len(words) < MIN_WORDS:\n        return False\n    lines = [ln for ln in text.split(\"\\n\") if ln.strip()]\n    if len(lines) >= 8:\n        short = sum(1 for ln in lines if len(ln.split()) <= 3)\n        if short / len(lines) > MAX_SHORT_LINE_FRAC:\n            return False\n        uniq = len(set(lines))\n        if 1 - uniq / len(lines) > MAX_DUP_LINE_FRAC:\n            return False\n    return True\n\n\ndef main():\n    ap = argparse.ArgumentParser()\n    ap.add_argument(\"--out\", default=OUT)\n    ap.add_argument(\"--seed\", type=int, default=0)\n    ap.add_argument(\"--min_score\", type=float, default=None,\n                    help=\"optional hard score cutoff; default keeps all gated docs\")\n    args = ap.parse_args()\n    rng = np.random.default_rng(args.seed)\n\n    # ---- 1. load pool -------------------------------------------------------\n    print(\"loading pool...\", file=sys.stderr)\n    ids, texts = [], []\n    with open(POOL) as f:\n        for line in f:\n            r = json.loads(line)\n            ids.append(r[\"id\"]); texts.append(r[\"text\"])\n    ids = np.array(ids)\n    N = len(ids)\n    print(f\"  {N} docs\", file=sys.stderr)\n\n    # ---- 2. build TARGET n-gram LM from decoded dev sample ------------------\n    print(\"decoding target sample + building target LM...\", file=sys.stderr)\n    from transformers import AutoTokenizer\n    tok = AutoTokenizer.from_pretrained(\"gpt2\")\n    dev = np.load(DEV_NPY).astype(np.int64)\n    EOS = 50256\n    # split into documents on EOS, decode each, featurize\n    idx = np.where(dev == EOS)[0]\n    segs, prev = [], 0\n    for i in idx:\n        if i > prev: segs.append((prev, i))\n        prev = i + 1\n    if prev < len(dev): segs.append((prev, len(dev)))\n    tgt_counts = np.zeros(MASK + 1, dtype=np.float64)\n    cache = {}\n    tgt_bucket_lists = []\n    for s, e in segs:\n        txt = tok.decode(dev[s:e].tolist())\n        words = WORD_RE.findall(txt.lower())\n        tgt_bucket_lists.append(doc_feature_buckets(words, cache))\n    accumulate_counts(tgt_bucket_lists, tgt_counts)\n    print(f\"  target: {len(segs)} segments, {tgt_counts.sum():.0f} features\", file=sys.stderr)\n\n    # ---- 3. build POOL background n-gram LM from a random sample ------------\n    print(\"building pool background LM...\", file=sys.stderr)\n    bg_idx = rng.choice(N, size=min(BG_SAMPLE, N), replace=False)\n    bg_counts = np.zeros(MASK + 1, dtype=np.float64)\n    bg_lists = []\n    for j in bg_idx:\n        words = WORD_RE.findall(texts[j].lower())\n        bg_lists.append(doc_feature_buckets(words, cache))\n        if len(bg_lists) >= 4000:\n            accumulate_counts(bg_lists, bg_counts); bg_lists = []\n    accumulate_counts(bg_lists, bg_counts)\n    print(f\"  background: {len(bg_idx)} docs, {bg_counts.sum():.0f} features\", file=sys.stderr)\n\n    # ---- 4. per-bucket log-likelihood-ratio weights ------------------------\n    T = tgt_counts.sum(); B = bg_counts.sum(); V = MASK + 1\n    log_tgt = np.log(tgt_counts + ALPHA) - math.log(T + ALPHA * V)\n    log_bg = np.log(bg_counts + ALPHA) - math.log(B + ALPHA * V)\n    weight = log_tgt - log_bg          # importance log-weight per feature bucket\n\n    # ---- 5. score + gate every document ------------------------------------\n    print(\"scoring all documents...\", file=sys.stderr)\n    scores = np.full(N, -1e9, dtype=np.float64)\n    ok_mask = np.zeros(N, dtype=bool)\n    for k in range(N):\n        text = texts[k]\n        words = WORD_RE.findall(text.lower())\n        if not quality_ok(text, words):\n            continue\n        b = doc_feature_buckets(words, cache)\n        if not b:\n            continue\n        scores[k] = weight[np.asarray(b, dtype=np.int64)].mean()\n        ok_mask[k] = True\n        if (k + 1) % 20000 == 0:\n            print(f\"  {k+1}/{N}\", file=sys.stderr)\n\n    n_ok = int(ok_mask.sum())\n    print(f\"  {n_ok}/{N} docs passed the quality gate\", file=sys.stderr)\n\n    # ---- 6. rank + emit ----------------------------------------------------\n    order = np.argsort(-scores)                 # best first\n    order = order[ok_mask[order]]               # keep only gated docs\n    if args.min_score is not None:\n        order = order[scores[order] >= args.min_score]\n    sel_ids = ids[order].tolist()\n\n    # diagnostics: score distribution\n    ss = np.sort(scores[ok_mask])\n    for p in [50, 75, 90, 95, 99, 100]:\n        v = ss[min(len(ss) - 1, int(p / 100 * len(ss)))]\n        print(f\"  score p{p}: {v:.4f}\", file=sys.stderr)\n\n    json.dump(sel_ids, open(args.out, \"w\"))\n    print(f\"wrote {len(sel_ids)} ids -> {args.out}\", file=sys.stderr)\n\n\nif __name__ == \"__main__\":\n    main()\n", "userModified": false}

#19

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolBashRun improved curation

args

{
  "command": "cd /workspace\nexport TOKENIZERS_PARALLELISM=false\npython3 submission/curate.py 2>&1 | tail -25\necho \"remaining: $(cat /workspace/.timer/remaining_secs)s\"",
  "description": "Run improved curation",
  "run_in_background": true
}

Bash result

Command running in background with ID: b94br3opx. Output is being written to: /tmp/claude-0/-workspace/06cb3775-ded4-4ad5-84cf-ed7eaa26a5bd/tasks/b94br3opx.output. You will be notified when it completes. To check interim output, use Read on that file path.
Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "b94br3opx", "backgroundCwdHint": "Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands."}

#20

narration

While curation runs, let me prepare a random-baseline selection for calibration.

toolBashCreate random baseline selection

args

{
  "command": "cd /workspace\npython3 -c \"\nimport json, numpy as np\nids=[]\nwith open('/workspace/data/pool.jsonl') as f:\n    for line in f:\n        ids.append(json.loads(line)['id'])\nrng=np.random.default_rng(1337)\nrng.shuffle(ids)\njson.dump([int(i) for i in ids], open('/tmp/random_sel.json','w'))\nprint('random baseline selection:', len(ids), 'ids -> /tmp/random_sel.json')\n\"",
  "description": "Create random baseline selection"
}

Bash result

random baseline selection: 182016 ids -> /tmp/random_sel.json

[stdout]
random baseline selection: 182016 ids -> /tmp/random_sel.json
[metadata] {"noOutputExpected": false}

#21

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolBashCheck curation progress

args

{
  "command": "sleep 75; cat /tmp/claude-0/-workspace/06cb3775-ded4-4ad5-84cf-ed7eaa26a5bd/tasks/b94br3opx.output 2>/dev/null | tail -25; echo \"remaining: $(cat /workspace/.timer/remaining_secs)s\"",
  "description": "Check curation progress"
}

Bash result

remaining: 4599s

[stdout]
remaining: 4599s
[metadata] {"noOutputExpected": false}

#22

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolBashCheck curation progress again

args

{
  "command": "sleep 60; cat /tmp/claude-0/-workspace/06cb3775-ded4-4ad5-84cf-ed7eaa26a5bd/tasks/b94br3opx.output 2>/dev/null | tail -25; echo \"remaining: $(cat /workspace/.timer/remaining_secs)s\"",
  "description": "Check curation progress again"
}

Bash result

remaining: 4538s

[stdout]
remaining: 4538s
[metadata] {"noOutputExpected": false}

#23

narration

The tail buffers until EOF — I'll get output when it completes. Let me verify the process is alive and wait.

toolBashCheck if curate still running

args

{
  "command": "ps aux | grep -c \"[c]urate.py\"; echo \"still running? (1=yes)\"; echo \"remaining: $(cat /workspace/.timer/remaining_secs)s\"",
  "description": "Check if curate still running"
}

Bash result

5
still running? (1=yes)
remaining: 4528s

[stdout]
5
still running? (1=yes)
remaining: 4528s
[metadata] {"noOutputExpected": false}

#24

reasoning (redacted)

encrypted chain-of-thought — recorded, not exposed

toolBashWait and check curation done

args

{
  "command": "sleep 90; if ps aux | grep -q \"[c]urate.py\"; then echo \"STILL RUNNING\"; else echo \"DONE\"; cat /tmp/claude-0/-workspace/06cb3775-ded4-4ad5-84cf-ed7eaa26a5bd/tasks/b94br3opx.output 2>/dev/null | tail -30; fi; echo \"remaining: $(cat /workspace/.timer/remaining_secs)s\"",
  "description": "Wait and check curation done"
}

Bash result

STILL RUNNING
remaining: 4438s

[stdout]
STILL RUNNING
remaining: 4438s
[metadata] {"noOutputExpected": false}

#25

narration

Curation completed. Let me read the diagnostics and inspect the new top-ranked docs.

toolBashRead curation output

args

{
  "command": "cat /tmp/claude-0/-workspace/06cb3775-ded4-4ad5-84cf-ed7eaa26a5bd/tasks/b94br3opx.output 2>/dev/null | tail -30",
  "description": "Read curation output"
}

Bash result

loading pool...
  182016 docs
building vocabulary + background counts...
  vocab: 40000 unigrams + 40000 bigrams
decoding target sample + counting...
  target: 2705 segments
  most JUNK unigrams: ['accessories', 'cart', 'endoftext', 'mar', 'nike', 'sitemap', 'wiring', 'recipe', 'haml', 'clomid', 'towbars', 'gifts', 'workshop', 'diy', 'dumps']
  most TARGET unigrams: ['batsman', 'bjp', 'blockquote', 'crease', 'defends', 'flanders', 'hasan', 'knowles', 'lt', 'rahman', 'rel', 'rok', 'stumps', 'wicket', 'xsl']
scoring all documents...
  40000/182016
  80000/182016
  120000/182016
  160000/182016
  139874/182016 docs passed the quality gate
  dropped 5120 (near-)duplicate docs
  score p50: -0.1716
  score p75: -0.0436
  score p90: 0.0590
  score p95: 0.1164
  score p99: 0.2248
  score p100: 0.6960
wrote 134754 ids -> /workspace/submission/selection.json
remaining: 4458s

[stdout]
loading pool...
  182016 docs
building vocabulary + background counts...
  vocab: 40000 unigrams + 40000 bigrams
decoding target sample + counting...
  target: 2705 segments
  most JUNK unigrams: ['accessories', 'cart', 'endoftext', 'mar', 'nike', 'sitemap', 'wiring', 'recipe', 'haml', 'clomid', 'towbars', 'gifts', 'workshop', 'diy', 'dumps']
  most TARGET unigrams: ['batsman', 'bjp', 'blockquote', 'crease', 'defends', 'flanders', 'hasan', 'knowles', 'lt', 'rahman', 'rel', 'rok', 'stumps', 'wicket', 'xsl']
scoring all documents...
  40000/182016
  80000/182016
  120000/182016
  160000/182016
  139874/182016 docs passed the quality gate
  dropped 5120 (near-)duplicate docs
  score p50: -0.1716
  score p75: -0.0436
  score p90: 0.0590
  score p95: 0.1164
  score p99: 0.2248
  score p100: 0.6960
wrote 134754 ids -> /workspace/submission/selection.json
remaining: 4458s
[metadata] {"noOutputExpected": false}