gpt-5.6-sol

Script completed
Wall time 0.1 seconds
Output:
  PID     ELAPSED %CPU   RSS
2686

Script running with cell ID 138
Wall time 11.0 seconds
Output:

Script completed
Wall time 14.3 seconds
Output:
{"chunk_id":"94e0be","wall_time_seconds":30.00140578,"session_id":89499,"original_token_count":62}Token indices sequence length is longer than the specified maximum sequence length for this model (1043 > 1024). Running this sequence through the model will result in indexing errors
packed 12000000 tokens from 182016 ids -> /tmp/dev_train.npy

Script running with cell ID 139
Wall time 11.0 seconds
Output:

Script completed
Wall time 15.7 seconds
Output:

Script completed
Wall time 8.1 seconds
Output:
2026-08-04 01:53:25.483991890 +0000
{"train_npy": "/tmp/dev_train.npy", "heldout_ppl": 436.9506057098018, "mean_loss": 6.079820158294819, "max_iters": 3000, "seed": 1337, "train_s": 55.3}

Script completed
Wall time 0.0 seconds
Output:
{"chunk_id":"8d4238","wall_time_seconds":0.000003647,"exit_code":0,"original_token_count":38}HELDOUT_PPL 427.519  loss 6.0580  train_s 56  /tmp/dev_train.npy
(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)

Script completed
Wall time 0.2 seconds
Output:
raw mean_aff -0.40689644537523156 median_id 7499.5 max_id 14999
selected mean_aff -0.37634619490008503 median_id 7565.5 max_id 37522
#!/usr/bin/env python3
"""Curate with regularized target/corpus word log-odds (Moore-Lewis style).

The disclosed multi-domain dev set supplies the desired lexical/register
distribution, not labels for individual pool documents.  Per-word evidence is
smoothed and clipped, then combined with the pool's strong monotone assembly
quality rank.  This regularization keeps isolated topical matches from pulling
spam or boilerplate out of the low-quality tail.
"""
import argparse
import collections
import json
import math
import re
from pathlib import Path

import numpy as np
from transformers import AutoTokenizer

WORD_RE = re.compile(r"[a-z]+")
CANDIDATE_ID_LIMIT = 60_000
RANK_SCALE = 18_000.0
AFFINITY_WEIGHT = 2.4
SMOOTHING = 3.0
VOCAB_PRIOR = 100_000
MIN_TARGET_COUNT = 3
UNKNOWN_LOG_ODDS = -0.15
LOG_ODDS_CLIP = 2.0


def words(text):
    return WORD_RE.findall(text.lower())


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--pool", default="/workspace/data/pool.jsonl")
    ap.add_argument("--dev", default="/workspace/data/multi_dev.npy")
    ap.add_argument("--output", default="/workspace/submission/selection.json")
    args = ap.parse_args()

    tokenizer = AutoTokenizer.from_pretrained("gpt2")
    target_text = tokenizer.decode(np.load(args.dev))
    target_counts = collections.Counter(words(target_text))

    # Estimate the competing web distribution from the only region that the
    # regularized rank can realistically select. This is faster and avoids
    # letting the obvious junk tail define what "web-like" means.
    background_counts = collections.Counter()
    with open(args.pool, encoding="utf-8") as f:
        for line in f:
            row = json.loads(line)
            if row["id"] < CANDIDATE_ID_LIMIT:
                background_counts.update(words(row["text"]))

    nt = sum(target_counts.values())
    nb = sum(background_counts.values())
    target_denom = nt + SMOOTHING * VOCAB_PRIOR
    background_denom = nb + SMOOTHING * VOCAB_PRIOR
    log_odds = {}
    for word, count in target_counts.items():
        if count < MIN_TARGET_COUNT:
            continue
        value = (math.log((count + SMOOTHING) / target_denom)
                 - math.log((background_counts.get(word, 0) + SMOOTHING)
                            / background_denom))
        log_odds[word] = max(-LOG_ODDS_CLIP, min(LOG_ODDS_CLIP, value))

    ranked, tail = [], []
    with open(args.pool, encoding="utf-8") as f:
        for line in f:
            row = json.loads(line)
            doc_id = row["id"]
            if doc_id >= CANDIDATE_ID_LIMIT:
                tail.append(doc_id)
                continue
            doc_words = words(row["text"])
            affinity = sum(log_odds.get(w, UNKNOWN_LOG_ODDS)
                           for w in doc_words) / max(1, len(doc_words))
            priority = doc_id / RANK_SCALE - AFFINITY_WEIGHT * affinity
            ranked.append((priority, doc_id))

    ranked.sort(key=lambda item: (item[0], item[1]))
    tail.sort()
    selection = [doc_id for _, doc_id in ranked] + tail
    Path(args.output).parent.mkdir(parents=True, exist_ok=True)
    with open(args.output, "w") as f:
        json.dump(selection, f)
        f.write("\n")


if __name__ == "__main__":
    main()
# Hypothesis

Under a 12M-token budget, a regularized Moore-Lewis-style ranking will beat both random sampling and the pool's raw quality order. Documents should be preferred when their smoothed word distribution resembles the disclosed equal-weight, four-register target, but target affinity must be constrained by the pool's monotone assembly-quality rank so that topical keyword matches cannot promote spam and boilerplate from the tail.

# Mechanism

`curate.py` estimates clipped word log-odds between `multi_dev.npy` and the clean candidate region of the pool. A document's mean log-odds is combined with its assembly rank; rare target words are ignored, all word contributions are clipped, and unseen words receive a small penalty. The observable prediction, independent of final perplexity, is that the consumed slice has higher mean target-vs-web word log-odds than raw ID order while remaining concentrated in the early quality region. It should also have fewer navigation/storefront pages than a target-affinity-only ranking.

# Falsification

Hold the tokenizer, model, token budget, seed, and training recipe fixed. Compare raw ID order, rank-only selection, affinity-only selection, and the combined rank. The hypothesis is falsified if the combined selection does not improve held-out cross-entropy on fresh samples from each of Wikipedia, high-quality web prose, news, and technical Q&A, or if gains come from one register while materially harming another. As a data-only check, it is also falsified if the selected slice fails to increase held-out target word log-likelihood relative to raw order.

# Transfer

The method transfers to another web pool by replacing the target sample, re-estimating target/background counts, and recalibrating the rank scale on that pool's quality ordering. The smoothing, minimum-count threshold, clipping, and rank regularization are specifically intended to transfer: they use register-level distributional evidence rather than hand-picked IDs, domains, URLs, or exact phrases. If a new pool has no meaningful assembly-quality order, the rank prior should be replaced with an independently measured prose-quality score.

Script completed
Wall time 0.0 seconds
Output:
{}

Script completed
Wall time 1.2 seconds
Output:
SESSION_ID=36672

Script running with cell ID 145
Wall time 11.0 seconds
Output:

Script completed
Wall time 7.2 seconds
Output:
{"chunk_id":"8ce488","wall_time_seconds":30.001909839,"session_id":36672,"original_token_count":0}

Script running with cell ID 146
Wall time 11.0 seconds
Output:

Script completed
Wall time 0.0 seconds
Output:
{"chunk_id":"ffaa07","wall_time_seconds":15.710903046,"exit_code":0,"original_token_count":17}HELDOUT_PPL 422.468  loss 6.0461  train_s 55  /tmp/final_train.npy

Script completed
Wall time 0.3 seconds
Output:
/workspace/submission/selection.json 1345019
/workspace/submission/curate.py 3268
/workspace/claim.md 2512
{"train_npy": "/tmp/final_train.npy", "heldout_ppl": 422.4683527344255, "mean_loss": 6.0461145392211355, "max_iters": 3000, "seed": 1337, "train_s": 55.3}

Script completed
Wall time 0.2 seconds
Output:
2426
deliverables_ok 182016