gpt-5.6-sol
nano-gpt-data-curation — graded by the task's verifier against a measured baseline.
- backend
- modal
- variant
- codex azure_ai/gpt-5.6-sol
- metric
- held-out multi-domain perplexity
- actions
- 267
- duration
- 1h 6m
- recorded
- 2026-08-03
run iddclm-nano-curation__7CCVfar
Everything gpt-5.6-sol did on this task, in order.
download JSON ↗encrypted chain-of-thought — recorded, not exposed
args
{
"input": "const r = await tools.write_stdin({session_id:40805,chars:\"\",yield_time_ms:30000,max_output_tokens:3000});\ntext(r.output);"
}exec result
Script running with cell ID 151
Wall time 11.0 seconds
Output:
encrypted chain-of-thought — recorded, not exposed
args
{
"cell_id": "151",
"yield_time_ms": 30000,
"max_tokens": 3000
}wait result
Script completed
Wall time 14.1 seconds
Output:
encrypted chain-of-thought — recorded, not exposed
args
{
"input": "const r = await tools.write_stdin({session_id:40805,chars:\"\",yield_time_ms\":30000,max_output_tokens:3000});\ntext(r.output);"
}exec result
Script failed
Wall time 0.0 seconds
Output:
Script error:
SyntaxError: Invalid or unexpected tokenencrypted chain-of-thought — recorded, not exposed
args
{
"input": "const r = await tools.write_stdin({session_id:40805,chars:\"\",yield_time_ms\":30000,max_output_tokens:3000});\ntext(r.output);"
}exec result
Script failed
Wall time 0.0 seconds
Output:
Script error:
SyntaxError: Invalid or unexpected tokenencrypted chain-of-thought — recorded, not exposed
args
{
"input": "const r = await tools.write_stdin({session_id:40805,chars:\"\",yield_time_ms\":30000,max_output_tokens:3000});\ntext(r.output);"
}exec result
Script failed
Wall time 0.0 seconds
Output:
Script error:
SyntaxError: Invalid or unexpected tokenencrypted chain-of-thought — recorded, not exposed
args
{
"input": "const r = await tools.write_stdin({session_id:40805,chars:\"\",yield_time_ms:30000,max_output_tokens:3000});\ntext(r.output);"
}exec result
Script completed
Wall time 0.0 seconds
Output:
HELDOUT_PPL 327.959 loss 5.7929 train_s 53 /tmp/dev_train.npy
(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)
The token-level refinement reduced performance (328.0 versus 324.1), so the final criterion will use the more robust word/bigram density ratio with explicit boilerplate filtering, exact deduplication, and the validated 2.5M/2.5M/3M/4M register allocation. I’m now packaging that exact method into a standalone script and validating that its packed 12M-token stream matches the winning candidate.
encrypted chain-of-thought — recorded, not exposed
args
{
"input": "const patch = \"*** Begin Patch\\n*** Add File: /workspace/submission/curate.py\\n+#!/usr/bin/env python3\\n+\\\"\\\"\\\"Curate raw web text by target-density ratio, hygiene, and register coverage.\\n+\\n+Criterion\\n+---------\\n+1. Learn smoothed unigram/bigram log-rate ratios between each disclosed target\\n+ quarter (encyclopedic, quality web, news, technical Q&A) and a deterministic\\n+ 12,000-document sample of the raw pool.\\n+2. Reject documents with extreme length/character statistics, excessive URLs,\\n+ navigation boilerplate, line noise, or repeated trigrams.\\n+3. Rank the survivors independently for each target register by its density\\n+ advantage over raw web.\\n+4. Normalize whitespace/case and remove exact-content duplicates, then use a\\n+ token-aware round robin to allocate 2.5M/2.5M/3M/4M GPT-2 tokens to the four\\n+ registers. The allocation was selected on the disclosed dev set; it gives\\n+ extra capacity to technical Q&A, where source/format match was most useful.\\n+\\n+Only the pool, disclosed dev array, and locally installed GPT-2 tokenizer are\\n+used. No IDs or source labels are hard-coded.\\n+\\\"\\\"\\\"\\n+import argparse\\n+import hashlib\\n+import json\\n+import math\\n+import random\\n+import re\\n+import time\\n+from collections import Counter\\n+\\n+import numpy as np\\n+from transformers import GPT2TokenizerFast\\n+\\n+\\n+ap = argparse.ArgumentParser()\\n+ap.add_argument(\\\"--pool\\\", default=\\\"/workspace/data/pool.jsonl\\\")\\n+ap.add_argument(\\\"--dev\\\", default=\\\"/workspace/data/multi_dev.npy\\\")\\n+ap.add_argument(\\\"--output\\\", default=\\\"/workspace/submission/selection.json\\\")\\n+a = ap.parse_args()\\n+\\n+SEED = 20260803\\n+NPOOL = 182_016\\n+NEGATIVE_DOCS = 12_000\\n+TOP_PER_DOMAIN = 35_000\\n+QUOTAS = [2_500_000, 2_500_000, 3_000_000, 4_000_000]\\n+\\n+tok = GPT2TokenizerFast.from_pretrained(\\\"gpt2\\\", local_files_only=True)\\n+dev = np.load(a.dev)\\n+word_re = re.compile(r\\\"[a-zA-Z][a-zA-Z0-9_+#.-]*|[0-9]+|</?\\\\w+>|&\\\\w+;\\\")\\n+\\n+\\n+def target_chunks(s, size=2500):\\n+ out = []\\n+ for doc in s.split(\\\"<|endoftext|>\\\"):\\n+ doc = doc.strip()\\n+ if len(doc) < 350:\\n+ continue\\n+ for start in range(0, len(doc), size):\\n+ part = doc[start:start + size]\\n+ if len(part) >= 350:\\n+ out.append(part)\\n+ return out\\n+\\n+\\n+# Construct the positive register samples and deterministic raw-web contrast.\\n+train_x, train_y = [], []\\n+for domain in range(4):\\n+ text = tok.decode(dev[domain * 250_000:(domain + 1) * 250_000])\\n+ pieces = target_chunks(text)\\n+ train_x.extend(pieces)\\n+ train_y.extend([domain] * len(pieces))\\n+\\n+neg_ids = set(random.Random(SEED).sample(range(NPOOL), NEGATIVE_DOCS))\\n+for line in open(a.pool):\\n+ row = json.loads(line)\\n+ if row[\\\"id\\\"] in neg_ids:\\n+ text = row[\\\"text\\\"][:5000]\\n+ if len(text) >= 100:\\n+ train_x.append(text)\\n+ train_y.append(4)\\n+\\n+# Count word unigrams and bigrams per target register plus contrast class.\\n+counts = [[Counter(), Counter()] for _ in range(5)]\\n+totals = [[0, 0] for _ in range(5)]\\n+for text, label in zip(train_x, train_y):\\n+ words = [x.lower() for x in word_re.findall(text)]\\n+ bigrams = [words[i] + \\\"\\\\x01\\\" + words[i + 1]\\n+ for i in range(len(words) - 1)]\\n+ counts[label][0].update(words)\\n+ counts[label][1].update(bigrams)\\n+for label in range(5):\\n+ for order in range(2):\\n+ totals[label][order] = sum(counts[label][order].values())\\n+\\n+# Features unseen/unsupported in the target are neutral. This avoids rewarding\\n+# gibberish merely because the target sample is smaller than the raw pool.\\n+tables = []\\n+for order in range(2):\\n+ vocab = {feature for domain in range(4)\\n+ for feature, count in counts[domain][order].items() if count >= 2}\\n+ table = {}\\n+ for feature in vocab:\\n+ raw_count = counts[4][order].get(feature, 0)\\n+ row = []\\n+ for domain in range(4):\\n+ target_count = counts[domain][order].get(feature, 0)\\n+ ratio = math.log(\\n+ (target_count + .2) / (totals[domain][order] + 1)\\n+ / ((raw_count + 1) / (totals[4][order] + 1)))\\n+ row.append(max(-4.0, min(4.0, ratio)))\\n+ table[feature] = tuple(row)\\n+ tables.append(table)\\n+\\n+\\n+def density(text):\\n+ words = [x.lower() for x in word_re.findall(text[:5000])]\\n+ features = (words, [words[i] + \\\"\\\\x01\\\" + words[i + 1]\\n+ for i in range(len(words) - 1)])\\n+ answer = [0.0] * 4\\n+ for order, seq in enumerate(features):\\n+ if not seq:\\n+ continue\\n+ sums = [0.0] * 4\\n+ for feature in seq:\\n+ row = tables[order].get(feature)\\n+ if row is not None:\\n+ for domain in range(4):\\n+ sums[domain] += row[domain]\\n+ weight = (.35 if order == 0 else .65) / len(seq)\\n+ for domain in range(4):\\n+ answer[domain] += weight * sums[domain]\\n+ return answer\\n+\\n+\\n+nav_re = re.compile(\\n+ r\\\"\\\\b(?:login|register|menu|cookie|privacy policy|terms and conditions|\\\"\\n+ r\\\"skip to content|my account|shopping cart|subscribe|newsletter|\\\"\\n+ r\\\"javascript|all rights reserved|contact us|home page|search results)\\\\b\\\",\\n+ re.I)\\n+url_re = re.compile(r\\\"https?://|www\\\\.|\\\\.com\\\\b\\\", re.I)\\n+\\n+\\n+def features(text):\\n+ nchar = max(1, len(text))\\n+ words = re.findall(r\\\"[A-Za-z]+(?:'[A-Za-z]+)?\\\", text)\\n+ nword = max(1, len(words))\\n+ lower = [w.lower() for w in words]\\n+ trigrams = list(zip(lower, lower[1:], lower[2:]))\\n+ repeat = 1 - len(set(trigrams)) / max(1, len(trigrams))\\n+ nav = len(nav_re.findall(text)) / nword\\n+ alpha = sum(c.isalpha() for c in text) / nchar\\n+ digit = sum(c.isdigit() for c in text) / nchar\\n+ newline = text.count(\\\"\\\\n\\\") / nchar\\n+ urls = len(url_re.findall(text)) / nword\\n+ quality = (2.5 * min(alpha, .82) - 12 * abs(digit - .025)\\n+ - 20 * nav - 8 * urls - 5 * max(0, repeat - .08)\\n+ - 12 * max(0, newline - .035)\\n+ - .25 * abs(math.log(max(250, nchar) / 3000)))\\n+ valid = (600 <= nchar <= 40_000 and .68 <= alpha <= .9\\n+ and digit < .12 and nav < .015 and urls < .015\\n+ and repeat < .22 and newline < .06 and quality > .35)\\n+ return valid, quality\\n+\\n+\\n+# Score the complete pool. IDs are verified to form the expected dense range.\\n+t0 = time.time()\\n+scores = np.full((NPOOL, 4), -1e9, dtype=np.float32)\\n+quality = np.full(NPOOL, -1e9, dtype=np.float32)\\n+seen_ids = set()\\n+for line in open(a.pool):\\n+ row = json.loads(line)\\n+ doc_id, text = row[\\\"id\\\"], row[\\\"text\\\"]\\n+ if not (0 <= doc_id < NPOOL) or doc_id in seen_ids:\\n+ raise ValueError(f\\\"invalid or duplicate pool id: {doc_id}\\\")\\n+ seen_ids.add(doc_id)\\n+ valid, q = features(text)\\n+ quality[doc_id] = q\\n+ if valid:\\n+ scores[doc_id] = density(text)\\n+if len(seen_ids) != NPOOL:\\n+ raise ValueError(f\\\"expected {NPOOL} pool IDs, found {len(seen_ids)}\\\")\\n+print(f\\\"scored {NPOOL} documents in {time.time()-t0:.1f}s\\\")\\n+\\n+rankings = [np.argsort(-scores[:, domain])[:TOP_PER_DOMAIN]\\n+ for domain in range(4)]\\n+candidate_ids = set()\\n+for ranking in rankings:\\n+ candidate_ids.update(map(int, ranking))\\n+\\n+# Read/tokenize plausible candidates only, and compute normalized exact hashes.\\n+texts = {}\\n+for line in open(a.pool):\\n+ row = json.loads(line)\\n+ if row[\\\"id\\\"] in candidate_ids:\\n+ texts[row[\\\"id\\\"]] = row[\\\"text\\\"]\\n+\\n+lengths, hashes = {}, {}\\n+ids = list(texts)\\n+for start in range(0, len(ids), 512):\\n+ batch_ids = ids[start:start + 512]\\n+ encoded = tok([texts[i] for i in batch_ids], add_special_tokens=False)\\n+ for doc_id, token_ids in zip(batch_ids, encoded.input_ids):\\n+ lengths[doc_id] = len(token_ids) + 1 # packer appends EOS\\n+ normalized = re.sub(r\\\"\\\\s+\\\", \\\" \\\", texts[doc_id]).strip().lower()\\n+ hashes[doc_id] = hashlib.sha1(normalized.encode(\\\"utf-8\\\")).digest()\\n+\\n+# Token-aware round robin makes every prefix approximately register-balanced.\\n+totals = [0] * 4\\n+positions = [0] * 4\\n+selected, used, seen_hashes = [], set(), set()\\n+while any(totals[d] < QUOTAS[d] for d in range(4)):\\n+ for domain in range(4):\\n+ if totals[domain] >= QUOTAS[domain]:\\n+ continue\\n+ while positions[domain] < len(rankings[domain]):\\n+ doc_id = int(rankings[domain][positions[domain]])\\n+ positions[domain] += 1\\n+ digest = hashes[doc_id]\\n+ if doc_id in used or digest in seen_hashes:\\n+ continue\\n+ selected.append(doc_id)\\n+ used.add(doc_id)\\n+ seen_hashes.add(digest)\\n+ totals[domain] += lengths[doc_id]\\n+ break\\n+ else:\\n+ raise RuntimeError(f\\\"domain {domain} exhausted before its quota\\\")\\n+\\n+# Quality-ranked valid tail, then every remaining ID, guarantees ample coverage.\\n+tail_score = scores.mean(axis=1) + .15 * quality\\n+for raw_id in np.argsort(-tail_score):\\n+ doc_id = int(raw_id)\\n+ if doc_id not in used:\\n+ selected.append(doc_id)\\n+ used.add(doc_id)\\n+for doc_id in range(NPOOL):\\n+ if doc_id not in used:\\n+ selected.append(doc_id)\\n+\\n+if len(selected) != NPOOL or len(set(selected)) != NPOOL:\\n+ raise AssertionError(\\\"selection must contain every ID exactly once\\\")\\n+with open(a.output, \\\"w\\\") as f:\\n+ json.dump(selected, f)\\n+print(\\\"domain token totals\\\", totals, \\\"selected IDs\\\", len(selected), \\\"->\\\", a.output)\\n*** End Patch\";\ntext(await tools.apply_patch(patch));"
}exec result
Script completed
Wall time 0.0 seconds
Output:
{}encrypted chain-of-thought — recorded, not exposed
args
{
"input": "const r = await tools.exec_command({cmd:\"python /workspace/pack_selection.py /workspace/experiments/tech4.json /tmp/win.npy && sha256sum /tmp/win.npy && python /workspace/submission/curate.py\",\"workdir\":\"/workspace\",\"yield_time_ms\":1000,\"max_output_tokens\":5000});\ntext(JSON.stringify(r));"
}exec result
Script completed
Wall time 1.2 seconds
Output:
{"chunk_id":"ca0dd4","wall_time_seconds":1.001120058,"session_id":46306,"original_token_count":0,"output":""}encrypted chain-of-thought — recorded, not exposed
args
{
"input": "const r = await tools.write_stdin({session_id:46306,chars:\"\",yield_time_ms\":30000,max_output_tokens:5000});\ntext(r.output);"
}exec result
Script failed
Wall time 0.0 seconds
Output:
Script error:
SyntaxError: Invalid or unexpected tokenencrypted chain-of-thought — recorded, not exposed
args
{
"input": "const r = await tools.write_stdin({session_id:46306,chars:\"\",yield_time_ms:30000,max_output_tokens:5000});\ntext(r.output);"
}exec result
Script running with cell ID 159
Wall time 11.0 seconds
Output:
encrypted chain-of-thought — recorded, not exposed
args
{
"cell_id": "159",
"yield_time_ms": 30000,
"max_tokens": 5000
}wait result
Script completed
Wall time 7.1 seconds
Output:
Token indices sequence length is longer than the specified maximum sequence length for this model (5414 > 1024). Running this sequence through the model will result in indexing errors
packed 12000000 tokens from 104298 ids -> /tmp/win.npy
45d7424c2e395488e303329ef634628413a5fc804741107f645a99c65a97d5bf /tmp/win.npy
encrypted chain-of-thought — recorded, not exposed
args
{
"input": "const r = await tools.write_stdin({session_id:46306,chars:\"\",yield_time_ms:30000,max_output_tokens:5000});\ntext(r.output);"
}exec result
Script running with cell ID 160
Wall time 11.0 seconds
Output:
encrypted chain-of-thought — recorded, not exposed
args
{
"cell_id": "160",
"yield_time_ms": 30000,
"max_tokens": 5000
}wait result
Script completed
Wall time 3.8 seconds
Output:
encrypted chain-of-thought — recorded, not exposed
args
{
"input": "const r = await tools.write_stdin({session_id:46306,chars:\"\",yield_time_ms:30000,max_output_tokens:5000});\ntext(r.output);"
}exec result
Script running with cell ID 161
Wall time 11.0 seconds
Output:
encrypted chain-of-thought — recorded, not exposed
args
{
"cell_id": "161",
"yield_time_ms": 30000,
"max_tokens": 5000
}wait result
Script completed
Wall time 12.3 seconds
Output:
encrypted chain-of-thought — recorded, not exposed
args
{
"input": "const r = await tools.write_stdin({session_id:46306,chars:\"\",yield_time_ms:30000,max_output_tokens:5000});\ntext(r.output);"
}exec result
Script running with cell ID 162
Wall time 11.0 seconds
Output:
encrypted chain-of-thought — recorded, not exposed
args
{
"cell_id": "162",
"yield_time_ms": 30000,
"max_tokens": 5000
}wait result
Script completed
Wall time 7.4 seconds
Output:
The standalone curator is running from scratch now. I’m checking byte-for-byte equality of the resulting packed training array against the 324.1-PPL candidate, then I’ll validate JSON uniqueness/range/coverage and finish the claim document.
encrypted chain-of-thought — recorded, not exposed
args
{
"input": "const r = await tools.write_stdin({session_id:46306,chars:\"\",yield_time_ms\":30000,max_output_tokens:5000});\ntext(r.output);"
}exec result
Script failed
Wall time 0.0 seconds
Output:
Script error:
SyntaxError: Invalid or unexpected tokenencrypted chain-of-thought — recorded, not exposed
args
{
"input": "const r = await tools.write_stdin({session_id:46306,chars:\"\",yield_time_ms:30000,max_output_tokens:5000});\ntext(r.output);"
}exec result
Script running with cell ID 164
Wall time 11.0 seconds
Output:
encrypted chain-of-thought — recorded, not exposed
args
{
"cell_id": "164",
"yield_time_ms": 30000,
"max_tokens": 5000
}wait result
Script completed
Wall time 0.0 seconds
Output:
encrypted chain-of-thought — recorded, not exposed
args
{
"input": "const r = await tools.write_stdin({session_id:46306,chars:\"\",yield_time_ms\":30000,max_output_tokens:5000});\ntext(r.output);"
}exec result
Script failed
Wall time 0.0 seconds
Output:
Script error:
SyntaxError: Invalid or unexpected tokenencrypted chain-of-thought — recorded, not exposed
args
{
"input": "const r = await tools.write_stdin({session_id:46306,chars:\"\",yield_time_ms:30000,max_output_tokens:5000});\ntext(r.output);"
}exec result
Script running with cell ID 166
Wall time 11.0 seconds
Output:
encrypted chain-of-thought — recorded, not exposed
args
{
"cell_id": "166",
"yield_time_ms": 30000,
"max_tokens": 5000
}wait result
Script completed
Wall time 12.4 seconds
Output:
Token indices sequence length is longer than the specified maximum sequence length for this model (1048 > 1024). Running this sequence through the model will result in indexing errors