claude-opus-4-8

docs 182016
code-indicator count per doc: p50 0 p90 2 p99 21 max 627
docs with >=3 indicators: 12774 7.0%
docs with >=8 indicators: 4090 2.2%
docs with >=15: 2374  >=25: 1633
--- code_ct 11 id 827
'By Tammy Straw-Dunn\nInitiated by a small group of neighbors, this community message board was brought to fruition through donations & volunteered time, collaborating with The City Repair Project & the Portland Bureau of Transportation. The design, includes exciting features like living roof, re-purp'
--- code_ct 23 id 1037
'Sheriff K.C. Hamp congratulates Captain Bernadette Logan on the completion of the Certified Investigators Program\nCONGRATULATIONS to Captain Bernadette Logan for completing 400 hours (10 weeks) of training and graduating from the Certified Investigators Program at the Mississippi Law Enforcement Off'
--- code_ct 18 id 1139
'So I posted on facebook last night to see if anyone was interested in seeing our weekly meal plan – even if it didn’t include recipes and there was lots of interest! So here is the first one!\nSo a few words about my eating – I have a husband that loves to eat meat – so I always include meat in our d'
--- code_ct 15 id 2160
'FOCUS: The Pew Center on the States Report: Beyond California: States in Fiscal Peril\nBeyond California: States in Fiscal Peril, a report released by the Pew Center on the States on November 11, 2009, highlights the ten states troubled most by the recession: Arizona, California, Florida, Illinois, M'
--- code_ct 10 id 2237
'Serious. These medicines may interact and cause very harmful effects. Contact your healthcare professional (e.g. doctor or pharmacist) for more information.\nHow the interaction occurs:\nWhen these two medicines are taken together, cimetidine may cause your body to process the blood-thinner more slowl'

[stdout]
docs 182016
code-indicator count per doc: p50 0 p90 2 p99 21 max 627
docs with >=3 indicators: 12774 7.0%
docs with >=8 indicators: 4090 2.2%
docs with >=15: 2374  >=25: 1633
--- code_ct 11 id 827
'By Tammy Straw-Dunn\nInitiated by a small group of neighbors, this community message board was brought to fruition through donations & volunteered time, collaborating with The City Repair Project & the Portland Bureau of Transportation. The design, includes exciting features like living roof, re-purp'
--- code_ct 23 id 1037
'Sheriff K.C. Hamp congratulates Captain Bernadette Logan on the completion of the Certified Investigators Program\nCONGRATULATIONS to Captain Bernadette Logan for completing 400 hours (10 weeks) of training and graduating from the Certified Investigators Program at the Mississippi Law Enforcement Off'
--- code_ct 18 id 1139
'So I posted on facebook last night to see if anyone was interested in seeing our weekly meal plan – even if it didn’t include recipes and there was lots of interest! So here is the first one!\nSo a few words about my eating – I have a husband that loves to eat meat – so I always include meat in our d'
--- code_ct 15 id 2160
'FOCUS: The Pew Center on the States Report: Beyond California: States in Fiscal Peril\nBeyond California: States in Fiscal Peril, a report released by the Pew Center on the States on November 11, 2009, highlights the ten states troubled most by the recession: Arizona, California, Florida, Illinois, M'
--- code_ct 10 id 2237
'Serious. These medicines may interact and cause very harmful effects. Contact your healthcare professional (e.g. doctor or pharmacist) for more information.\nHow the interaction occurs:\nWhen these two medicines are taken together, cimetidine may cause your body to process the blood-thinner more slowl'
[metadata] {"noOutputExpected": false}

The file /workspace/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/curate.py", "oldString": "The target is a fixed mixture whose one unambiguous slice is technical Q&A (~25% of\ntarget tokens, identifiable by HTML/code markup in the decoded dev set). Programming/\nQ&A text is stylistically distinct from encyclopedic/news/web prose, so we score with\nTWO heads sharing the same features:\n    PROSE head : positives = decoded dev segments WITHOUT code/HTML markup\n    TECH  head : positives = decoded dev segments WITH    code/HTML markup\nEach pool doc is assigned to whichever head it looks most like (standardized score),\nmust pass a register-aware junk filter (drops word-salad, menu/boilerplate, non-English,\nmarkup dumps, repetitive spam), is deduplicated by a content fingerprint, and is ranked\nby that head's score. We then fill the 12M-token budget token-balanced 75% prose / 25%\ntech to match the target mixture, interleaving so any early budget cutoff stays balanced.\nRemaining ranked ids are appended as overflow so the packer never runs short.\n\nPositives come only from the disclosed dev target; the selection is over disjoint pool\nids. The output is an ordered id list produced entirely by this script.\n\"\"\"", "newString": "The target is a fixed mixture including ~25% technical Q&A (identifiable by HTML/code\nmarkup in the decoded dev set). We AUDITED the pool and found it contains essentially\nNO code / programming / StackOverflow content (the `<code>`/`&lt;` markers are absent\nand \"code-like\" punctuation counts trace back to ordinary prose). So the code surface\nof the Q&A register is unmatchable from this pool, and an explicit \"technical\" selector\nonly drags in multilingual boilerplate and SEO word-salad (empirically its top hits) --\nwhich would displace good prose AND fail to teach code, hurting perplexity both ways.\n\nWe therefore use ONE combined head whose positives are ALL decoded dev segments\n(encyclopedic + news + web + the Q&A questions). Folding the Q&A positives in keeps\ntheir *technical vocabulary* (function, array, query, server, ...) in the log-odds, so\ngenuinely technical PROSE in the pool is still up-ranked -- the best available proxy for\nthe Q&A register when no code exists to select. Each pool doc must pass an English,\nclean-prose junk filter (drops word-salad, menu/boilerplate, non-English, markup dumps,\nrepetitive spam), is deduplicated by a content fingerprint, and is ranked by the mean\nclipped log-odds. Docs are emitted best-first; the packer fills 12M tokens from the top.\n\nPositives come only from the disclosed dev target; the selection is over disjoint pool\nids. The output is an ordered id list produced entirely by this script.\n\"\"\"", "originalFile": null, "structuredPatch": [{"oldStart": 17, "oldLines": 19, "newStart": 17, "newLines": 23, "lines": [" removes length bias, and clipping stops a handful of ultra-rare \"fancy\" words from", " letting SEO word-salad win -- the two dominant failure modes of a raw sum.", " ", "-The target is a fixed mixture whose one unambiguous slice is technical Q&A (~25% of", "-target tokens, identifiable by HTML/code markup in the decoded dev set). Programming/", "-Q&A text is stylistically distinct from encyclopedic/news/web prose, so we score with", "-TWO heads sharing the same features:", "-    PROSE head : positives = decoded dev segments WITHOUT code/HTML markup", "-    TECH  head : positives = decoded dev segments WITH    code/HTML markup", "-Each pool doc is assigned to whichever head it looks most like (standardized score),", "-must pass a register-aware junk filter (drops word-salad, menu/boilerplate, non-English,", "-markup dumps, repetitive spam), is deduplicated by a content fingerprint, and is ranked", "-by that head's score. We then fill the 12M-token budget token-balanced 75% prose / 25%", "-tech to match the target mixture, interleaving so any early budget cutoff stays balanced.", "-Remaining ranked ids are appended as overflow so the packer never runs short.", "+The target is a fixed mixture including ~25% technical Q&A (identifiable by HTML/code", "+markup in the decoded dev set). We AUDITED the pool and found it contains essentially", "+NO code / programming / StackOverflow content (the `<code>`/`&lt;` markers are absent", "+and \"code-like\" punctuation counts trace back to ordinary prose). So the code surface", "+of the Q&A register is unmatchable from this pool, and an explicit \"technical\" selector", "+only drags in multilingual boilerplate and SEO word-salad (empirically its top hits) --", "+which would displace good prose AND fail to teach code, hurting perplexity both ways.", " ", "+We therefore use ONE combined head whose positives are ALL decoded dev segments", "+(encyclopedic + news + web + the Q&A questions). Folding the Q&A positives in keeps", "+their *technical vocabulary* (function, array, query, server, ...) in the log-odds, so", "+genuinely technical PROSE in the pool is still up-ranked -- the best available proxy for", "+the Q&A register when no code exists to select. Each pool doc must pass an English,", "+clean-prose junk filter (drops word-salad, menu/boilerplate, non-English, markup dumps,", "+repetitive spam), is deduplicated by a content fingerprint, and is ranked by the mean", "+clipped log-odds. Docs are emitted best-first; the packer fills 12M tokens from the top.", "+", " Positives come only from the disclosed dev target; the selection is over disjoint pool", " ids. The output is an ordered id list produced entirely by this script.", " \"\"\""]}], "userModified": false, "replaceAll": false}

The file /workspace/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/curate.py", "oldString": "    r_prose = nb_logodds(prose_df, bg_df, npp, nbg)\n    r_tech  = nb_logodds(tech_df,  bg_df, npt, nbg)\n    s_prose = mean_score(r_prose, allidx, indptr)\n    s_tech  = mean_score(r_tech,  allidx, indptr)\n\n    nw, tok_est, alpha, digit, ascii_, sym, stopfrac, uniq, mwl, fshort, has_code = \\\n        [stats[:, i] for i in range(NSTAT)]\n\n    # register-aware junk filters\n    prose_ok = ((nw >= 50) & (alpha >= 0.62) & (stopfrac >= 0.20) & (uniq >= 0.30) &\n                (uniq <= 0.92) & (mwl <= 10.0) & (ascii_ >= 0.92) & (sym <= 0.12) &\n                (fshort <= 0.55) & (digit <= 0.20))\n    tech_ok = ((nw >= 40) & (alpha >= 0.40) & (stopfrac >= 0.10) & (uniq >= 0.28) &\n               (uniq <= 0.92) & (ascii_ >= 0.90) & (mwl <= 12.0) & (fshort <= 0.75))\n\n    zp = (s_prose - s_prose.mean()) / (s_prose.std() + 1e-9)\n    zt = (s_tech - s_tech.mean()) / (s_tech.std() + 1e-9)\n    is_tech = zt > zp\n\n    tech_mask = is_tech & tech_ok\n    prose_mask = (~is_tech) & prose_ok\n    tech_pos = np.where(tech_mask)[0][np.argsort(-s_tech[tech_mask])]\n    prose_pos = np.where(prose_mask)[0][np.argsort(-s_prose[prose_mask])]\n    print(f\"[2] candidates: prose={len(prose_pos)} tech={len(tech_pos)} \"\n          f\"(dropped {n-len(prose_pos)-len(tech_pos)})\", flush=True)\n\n    # token-balanced interleave to 75/25 with content-fingerprint dedup, then overflow\n    tech_budget = BUDGET * TECH_SHARE\n    prose_budget = BUDGET * (1 - TECH_SHARE)\n    seen_fp = set()\n    sel = []\n    pt = pi = 0\n    tacc = pacc = 0\n\n    def take(order, ptr):\n        while ptr < len(order):\n            j = order[ptr]; ptr += 1\n            fp = int(fps[j])\n            if fp in seen_fp:\n                continue\n            seen_fp.add(fp)\n            return j, ptr\n        return None, ptr\n\n    while (tacc < tech_budget or pacc < prose_budget) and (pt < len(tech_pos) or pi < len(prose_pos)):\n        want_tech = (tacc / max(1, tech_budget)) <= (pacc / max(1, prose_budget))\n        if want_tech and pt < len(tech_pos):\n            j, pt = take(tech_pos, pt)\n            if j is not None: sel.append(j); tacc += tok_est[j]\n        elif pi < len(prose_pos):\n            j, pi = take(prose_pos, pi)\n            if j is not None: sel.append(j); pacc += tok_est[j]\n        elif pt < len(tech_pos):\n            j, pt = take(tech_pos, pt)\n            if j is not None: sel.append(j); tacc += tok_est[j]\n    print(f\"    budget phase: {len(sel)} docs, tech~{tacc/1e6:.1f}M prose~{pacc/1e6:.1f}M est tokens\",\n          flush=True)\n\n    # overflow: append remaining ranked docs (3 prose : 1 tech), deduped\n    while (pi < len(prose_pos) or pt < len(tech_pos)) and len(sel) < N_OVERFLOW_IDS:\n        for _ in range(3):\n            if pi < len(prose_pos):\n                j, pi = take(prose_pos, pi)\n                if j is not None: sel.append(j)\n        if pt < len(tech_pos):\n            j, pt = take(tech_pos, pt)\n            if j is not None: sel.append(j)\n\n    out_ids = [int(ids[j]) for j in sel]", "newString": "    # single COMBINED head: all dev registers as positives (pool has no code to\n    # justify a separate technical selector -- see module docstring)\n    r_all = nb_logodds(prose_df + tech_df, bg_df, npp + npt, nbg)\n    score = mean_score(r_all, allidx, indptr)\n\n    nw, tok_est, alpha, digit, ascii_, sym, stopfrac, uniq, mwl, fshort, has_code = \\\n        [stats[:, i] for i in range(NSTAT)]\n\n    # English clean-prose junk filter: drops word-salad (stopword floor), menu/\n    # boilerplate (short-line + uniq caps), non-English (ascii + stopword),\n    # markup/data dumps (alpha/sym/digit), repetitive spam (uniq floor).\n    ok = ((nw >= 50) & (alpha >= 0.62) & (stopfrac >= 0.20) & (uniq >= 0.30) &\n          (uniq <= 0.92) & (mwl <= 10.0) & (ascii_ >= 0.92) & (sym <= 0.12) &\n          (fshort <= 0.55) & (digit <= 0.20))\n    cand = np.where(ok)[0]\n    cand = cand[np.argsort(-score[cand])]\n    print(f\"[2] candidates: {len(cand)} pass filter (dropped {n-len(cand)})\", flush=True)\n\n    # emit best-first with content-fingerprint dedup, up to N_OVERFLOW_IDS\n    seen_fp, sel, acc = set(), [], 0\n    for j in cand:\n        fp = int(fps[j])\n        if fp in seen_fp:\n            continue\n        seen_fp.add(fp)\n        sel.append(j); acc += tok_est[j]\n        if len(sel) >= N_OVERFLOW_IDS:\n            break\n    print(f\"    selected {len(sel)} docs (~{acc/1e6:.0f}M est tokens, best-first)\", flush=True)\n\n    out_ids = [int(ids[j]) for j in sel]", "originalFile": null, "structuredPatch": [{"oldStart": 193, "oldLines": 74, "newStart": 193, "newLines": 36, "lines": ["     bg_df = df_from(allidx); nbg = n", "     print(f\"    N={n} feats in {time.time()-t0:.0f}s\", flush=True)", " ", "-    r_prose = nb_logodds(prose_df, bg_df, npp, nbg)", "-    r_tech  = nb_logodds(tech_df,  bg_df, npt, nbg)", "-    s_prose = mean_score(r_prose, allidx, indptr)", "-    s_tech  = mean_score(r_tech,  allidx, indptr)", "+    # single COMBINED head: all dev registers as positives (pool has no code to", "+    # justify a separate technical selector -- see module docstring)", "+    r_all = nb_logodds(prose_df + tech_df, bg_df, npp + npt, nbg)", "+    score = mean_score(r_all, allidx, indptr)", " ", "     nw, tok_est, alpha, digit, ascii_, sym, stopfrac, uniq, mwl, fshort, has_code = \\", "         [stats[:, i] for i in range(NSTAT)]", " ", "-    # register-aware junk filters", "-    prose_ok = ((nw >= 50) & (alpha >= 0.62) & (stopfrac >= 0.20) & (uniq >= 0.30) &", "-                (uniq <= 0.92) & (mwl <= 10.0) & (ascii_ >= 0.92) & (sym <= 0.12) &", "-                (fshort <= 0.55) & (digit <= 0.20))", "-    tech_ok = ((nw >= 40) & (alpha >= 0.40) & (stopfrac >= 0.10) & (uniq >= 0.28) &", "-               (uniq <= 0.92) & (ascii_ >= 0.90) & (mwl <= 12.0) & (fshort <= 0.75))", "+    # English clean-prose junk filter: drops word-salad (stopword floor), menu/", "+    # boilerplate (short-line + uniq caps), non-English (ascii + stopword),", "+    # markup/data dumps (alpha/sym/digit), repetitive spam (uniq floor).", "+    ok = ((nw >= 50) & (alpha >= 0.62) & (stopfrac >= 0.20) & (uniq >= 0.30) &", "+          (uniq <= 0.92) & (mwl <= 10.0) & (ascii_ >= 0.92) & (sym <= 0.12) &", "+          (fshort <= 0.55) & (digit <= 0.20))", "+    cand = np.where(ok)[0]", "+    cand = cand[np.argsort(-score[cand])]", "+    print(f\"[2] candidates: {len(cand)} pass filter (dropped {n-len(cand)})\", flush=True)", " ", "-    zp = (s_prose - s_prose.mean()) / (s_prose.std() + 1e-9)", "-    zt = (s_tech - s_tech.mean()) / (s_tech.std() + 1e-9)", "-    is_tech = zt > zp", "+    # emit best-first with content-fingerprint dedup, up to N_OVERFLOW_IDS", "+    seen_fp, sel, acc = set(), [], 0", "+    for j in cand:", "+        fp = int(fps[j])", "+        if fp in seen_fp:", "+            continue", "+        seen_fp.add(fp)", "+        sel.append(j); acc += tok_est[j]", "+        if len(sel) >= N_OVERFLOW_IDS:", "+            break", "+    print(f\"    selected {len(sel)} docs (~{acc/1e6:.0f}M est tokens, best-first)\", flush=True)", " ", "-    tech_mask = is_tech & tech_ok", "-    prose_mask = (~is_tech) & prose_ok", "-    tech_pos = np.where(tech_mask)[0][np.argsort(-s_tech[tech_mask])]", "-    prose_pos = np.where(prose_mask)[0][np.argsort(-s_prose[prose_mask])]", "-    print(f\"[2] candidates: prose={len(prose_pos)} tech={len(tech_pos)} \"", "-          f\"(dropped {n-len(prose_pos)-len(tech_pos)})\", flush=True)", "-", "-    # token-balanced interleave to 75/25 with content-fingerprint dedup, then overflow", "-    tech_budget = BUDGET * TECH_SHARE", "-    prose_budget = BUDGET * (1 - TECH_SHARE)", "-    seen_fp = set()", "-    sel = []", "-    pt = pi = 0", "-    tacc = pacc = 0", "-", "-    def take(order, ptr):", "-        while ptr < len(order):", "-            j = order[ptr]; ptr += 1", "-            fp = int(fps[j])", "-            if fp in seen_fp:", "-                continue", "-            seen_fp.add(fp)", "-            return j, ptr", "-        return None, ptr", "-", "-    while (tacc < tech_budget or pacc < prose_budget) and (pt < len(tech_pos) or pi < len(prose_pos)):", "-        want_tech = (tacc / max(1, tech_budget)) <= (pacc / max(1, prose_budget))", "-        if want_tech and pt < len(tech_pos):", "-            j, pt = take(tech_pos, pt)", "-            if j is not None: sel.append(j); tacc += tok_est[j]", "-        elif pi < len(prose_pos):", "-            j, pi = take(prose_pos, pi)", "-            if j is not None: sel.append(j); pacc += tok_est[j]", "-        elif pt < len(tech_pos):", "-            j, pt = take(tech_pos, pt)", "-            if j is not None: sel.append(j); tacc += tok_est[j]", "-    print(f\"    budget phase: {len(sel)} docs, tech~{tacc/1e6:.1f}M prose~{pacc/1e6:.1f}M est tokens\",", "-          flush=True)", "-", "-    # overflow: append remaining ranked docs (3 prose : 1 tech), deduped", "-    while (pi < len(prose_pos) or pt < len(tech_pos)) and len(sel) < N_OVERFLOW_IDS:", "-        for _ in range(3):", "-            if pi < len(prose_pos):", "-                j, pi = take(prose_pos, pi)", "-                if j is not None: sel.append(j)", "-        if pt < len(tech_pos):", "-            j, pt = take(tech_pos, pt)", "-            if j is not None: sel.append(j)", "-", "     out_ids = [int(ids[j]) for j in sel]", "     assert len(out_ids) == len(set(out_ids)), \"duplicate ids\"", "     json.dump(out_ids, open(OUT, \"w\"))"]}], "userModified": false, "replaceAll": false}

The file /workspace/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/curate.py", "oldString": "    if \"--peek\" in sys.argv:\n        print(\"\\n===== TOP PROSE =====\")\n        for j in prose_pos[:6]:\n            print(f\"[s{s_prose[j]:.2f} nw{int(nw[j])} sw{stopfrac[j]:.2f} uq{uniq[j]:.2f} \"\n                  f\"fs{fshort[j]:.2f}] {heads[j]!r}\")\n        print(\"\\n===== TOP TECH =====\")\n        for j in tech_pos[:6]:\n            print(f\"[s{s_tech[j]:.2f} nw{int(nw[j])} sw{stopfrac[j]:.2f} uq{uniq[j]:.2f} \"\n                  f\"fs{fshort[j]:.2f}] {heads[j]!r}\")\n        print(\"\\n===== MID PROSE (rank ~8000) =====\")\n        for j in prose_pos[8000:8004]:\n            print(f\"[s{s_prose[j]:.2f}] {heads[j]!r}\")", "newString": "    if \"--peek\" in sys.argv:\n        sel_arr = np.array(sel)\n        for label, rng in [(\"TOP (rank 0-6)\", sel_arr[:7]),\n                           (\"rank ~4000\", sel_arr[4000:4004]),\n                           (\"rank ~10000 (near 12M cutoff)\", sel_arr[10000:10004])]:\n            print(f\"\\n===== {label} =====\")\n            for j in rng:\n                print(f\"[s{score[j]:.2f} nw{int(nw[j])} sw{stopfrac[j]:.2f} uq{uniq[j]:.2f} \"\n                      f\"fs{fshort[j]:.2f}] {heads[j]!r}\")", "originalFile": null, "structuredPatch": [{"oldStart": 231, "oldLines": 17, "newStart": 231, "newLines": 14, "lines": ["           flush=True)", " ", "     if \"--peek\" in sys.argv:", "-        print(\"\\n===== TOP PROSE =====\")", "-        for j in prose_pos[:6]:", "-            print(f\"[s{s_prose[j]:.2f} nw{int(nw[j])} sw{stopfrac[j]:.2f} uq{uniq[j]:.2f} \"", "-                  f\"fs{fshort[j]:.2f}] {heads[j]!r}\")", "-        print(\"\\n===== TOP TECH =====\")", "-        for j in tech_pos[:6]:", "-            print(f\"[s{s_tech[j]:.2f} nw{int(nw[j])} sw{stopfrac[j]:.2f} uq{uniq[j]:.2f} \"", "-                  f\"fs{fshort[j]:.2f}] {heads[j]!r}\")", "-        print(\"\\n===== MID PROSE (rank ~8000) =====\")", "-        for j in prose_pos[8000:8004]:", "-            print(f\"[s{s_prose[j]:.2f}] {heads[j]!r}\")", "+        sel_arr = np.array(sel)", "+        for label, rng in [(\"TOP (rank 0-6)\", sel_arr[:7]),", "+                           (\"rank ~4000\", sel_arr[4000:4004]),", "+                           (\"rank ~10000 (near 12M cutoff)\", sel_arr[10000:10004])]:", "+            print(f\"\\n===== {label} =====\")", "+            for j in rng:", "+                print(f\"[s{score[j]:.2f} nw{int(nw[j])} sw{stopfrac[j]:.2f} uq{uniq[j]:.2f} \"", "+                      f\"fs{fshort[j]:.2f}] {heads[j]!r}\")", " ", " ", " if __name__ == \"__main__\":"]}], "userModified": false, "replaceAll": false}

[1] loading feature cache
    N=182016 feats in 1s
[2] candidates: 117824 pass filter (dropped 64192)
    selected 45000 docs (~51M est tokens, best-first)
[3] wrote 45000 ids -> /workspace/submission/selection.json (~51M est tokens) in 2s

===== TOP (rank 0-6) =====
[s0.44 nw975 sw0.23 uq0.34 fs0.29] '|PREDECESSORS AND SHORT HISTORY:\nin the 16th century by Rao Gopalji fl.1550, 4th son of Raja Prithviraj\nSinghji I of Jaipur. Rulers were…\n- Rao GOPAL SINGH,\nThakur Saheb of Chomu fl.1550, married\nand had issue.\n- Thakur '
[s0.34 nw1200 sw0.23 uq0.73 fs0.02] 'Archaeological dating methods\nFutilities must undress thitherward convened dismissively towerish tomato.\nPossessory tongs being very mechanically fugued calendering.\nBurdensomely tomentum consciously scavenged the differ'
[s0.31 nw648 sw0.21 uq0.69 fs0.00] '8A<|endoftext|>Our god saves chords pdfOut of many by john mack faragher\nAnteorbital Armond befools it lining whig stark. citrus Pat smoke-dry, her burgles our god saves chords pdf sooner. incoming and antenniform Giorgi'
[s0.30 nw124 sw0.25 uq0.73 fs0.00] 'Lahore:In the prospects of stoned to death incident of a Khanewal Woman, Inspector General of Police Punjab, Mohammad Habib-ur-Rehman has recommended the Government of Punjab for Judicial Inquiry into Maryam Bibi’s case.'
[s0.29 nw209 sw0.34 uq0.42 fs0.00] '<|endoftext|>New Delhi: Five new Supreme Court judges were today sworn in, taking the top court’s strength to 28, including the Chief Justice of India. Justices Sanjay Kishan Kaul, Navin Sinha, Mohan M Shantanagoudar, De'
[s0.29 nw1200 sw0.35 uq0.37 fs0.36] 'Hill 303 massacre\n|Hill 303 massacre|\nBodies of massacre victims gathered near Waegwan, South Korea, many with their hands still bound\n|Location||Hill 303, Waegwan, South Korea|\n|Date||August 17, 1950\n|Target||U.S. Army '
[s0.28 nw623 sw0.22 uq0.64 fs0.00] 'a<|endoftext|>Czech Tobe alit it staleness checker unguardedly. lignifies sigillate that unsnarl hellishly? gassy Thacher doodle, her precesses mundanely. wainscoted and sex-limited Ricki bestialize her Cowes executed an'

===== rank ~4000 =====
[s-0.06 nw681 sw0.36 uq0.53 fs0.19] "castle claimed the points from a thrilling game to put a dent in Tottenham's Champions League hopes.\nBowyer's goal gave Newcastle the perfect start\nLee Bowyer slid the Magpies ahead after 66 seconds but Robbie Keane head"
[s-0.06 nw162 sw0.29 uq0.70 fs0.00] 'Lonard, who also clinched the Australian Order of Merit, finished one shot clear of Stephen Leaney and Chris Downes to capture the Australian Open for the first time.\nTiger Woods stayed put atop the rankings at No. 1 fol'
[s-0.06 nw1063 sw0.39 uq0.48 fs0.00] ' spread<|endoftext|>WASHINGTON — President Barack Obama eventually lost faith in the troop increase he ordered in Afghanistan, his doubts fed by top White House civilian advisers opposed to the strategy, who continually '
[s-0.06 nw134 sw0.32 uq0.78 fs0.00] "LIN – A series of potential scandals is plaguing the far-right Alternative for Germany party ahead of next month's European Parliament elections.\nThe party has struggled to explain a number of murky donations to prominen"

===== rank ~10000 (near 12M cutoff) =====
[s-0.13 nw137 sw0.31 uq0.63 fs0.00] 'Dave Chappelle has set a new record at Hollywood comedy club The Laugh Factory, after performing for six hours and seven minutes.\nDave Chappelle breaks the record of Dane Cook and Richard Pryor, entertained the Laugh Fac'
[s-0.13 nw720 sw0.37 uq0.51 fs0.00] ' wrestling season is in its transitory period, the calm before the three-week storm, when the District 2, Northeast Regional and PIAA Championships thin out the herd.\nThere is still wrestling to be done before districts '
[s-0.13 nw548 sw0.41 uq0.49 fs0.08] 'andro-, andr-, -ander, -andra, -andria, -andrian, -andric, -andrism, -androus, -andries, -andry\n(Greek: man, men, male, masculine; also, stamen or anther as used in botany)\n2. Normally virilized, as in the male.\n2. Perta'
[s-0.13 nw574 sw0.45 uq0.48 fs0.00] 'It would probably be best to start off with describing what this book is not. Its definitely not a cover-to-cover bedtime read. Its more something for the military historian to keep on the shelf for reference, and that t'
=== remaining ===
3858

[stdout]
[1] loading feature cache
    N=182016 feats in 1s
[2] candidates: 117824 pass filter (dropped 64192)
    selected 45000 docs (~51M est tokens, best-first)
[3] wrote 45000 ids -> /workspace/submission/selection.json (~51M est tokens) in 2s

===== TOP (rank 0-6) =====
[s0.44 nw975 sw0.23 uq0.34 fs0.29] '|PREDECESSORS AND SHORT HISTORY:\nin the 16th century by Rao Gopalji fl.1550, 4th son of Raja Prithviraj\nSinghji I of Jaipur. Rulers were…\n- Rao GOPAL SINGH,\nThakur Saheb of Chomu fl.1550, married\nand had issue.\n- Thakur '
[s0.34 nw1200 sw0.23 uq0.73 fs0.02] 'Archaeological dating methods\nFutilities must undress thitherward convened dismissively towerish tomato.\nPossessory tongs being very mechanically fugued calendering.\nBurdensomely tomentum consciously scavenged the differ'
[s0.31 nw648 sw0.21 uq0.69 fs0.00] '8A<|endoftext|>Our god saves chords pdfOut of many by john mack faragher\nAnteorbital Armond befools it lining whig stark. citrus Pat smoke-dry, her burgles our god saves chords pdf sooner. incoming and antenniform Giorgi'
[s0.30 nw124 sw0.25 uq0.73 fs0.00] 'Lahore:In the prospects of stoned to death incident of a Khanewal Woman, Inspector General of Police Punjab, Mohammad Habib-ur-Rehman has recommended the Government of Punjab for Judicial Inquiry into Maryam Bibi’s case.'
[s0.29 nw209 sw0.34 uq0.42 fs0.00] '<|endoftext|>New Delhi: Five new Supreme Court judges were today sworn in, taking the top court’s strength to 28, including the Chief Justice of India. Justices Sanjay Kishan Kaul, Navin Sinha, Mohan M Shantanagoudar, De'
[s0.29 nw1200 sw0.35 uq0.37 fs0.36] 'Hill 303 massacre\n|Hill 303 massacre|\nBodies of massacre victims gathered near Waegwan, South Korea, many with their hands still bound\n|Location||Hill 303, Waegwan, South Korea|\n|Date||August 17, 1950\n|Target||U.S. Army '
[s0.28 nw623 sw0.22 uq0.64 fs0.00] 'a<|endoftext|>Czech Tobe alit it staleness checker unguardedly. lignifies sigillate that unsnarl hellishly? gassy Thacher doodle, her precesses mundanely. wainscoted and sex-limited Ricki bestialize her Cowes executed an'

===== rank ~4000 =====
[s-0.06 nw681 sw0.36 uq0.53 fs0.19] "castle claimed the points from a thrilling game to put a dent in Tottenham's Champions League hopes.\nBowyer's goal gave Newcastle the perfect start\nLee Bowyer slid the Magpies ahead after 66 seconds but Robbie Keane head"
[s-0.06 nw162 sw0.29 uq0.70 fs0.00] 'Lonard, who also clinched the Australian Order of Merit, finished one shot clear of Stephen Leaney and Chris Downes to capture the Australian Open for the first time.\nTiger Woods stayed put atop the rankings at No. 1 fol'
[s-0.06 nw1063 sw0.39 uq0.48 fs0.00] ' spread<|endoftext|>WASHINGTON — President Barack Obama eventually lost faith in the troop increase he ordered in Afghanistan, his doubts fed by top White House civilian advisers opposed to the strategy, who continually '
[s-0.06 nw134 sw0.32 uq0.78 fs0.00] "LIN – A series of potential scandals is plaguing the far-right Alternative for Germany party ahead of next month's European Parliament elections.\nThe party has struggled to explain a number of murky donations to prominen"

===== rank ~10000 (near 12M cutoff) =====
[s-0.13 nw137 sw0.31 uq0.63 fs0.00] 'Dave Chappelle has set a new record at Hollywood comedy club The Laugh Factory, after performing for six hours and seven minutes.\nDave Chappelle breaks the record of Dane Cook and Richard Pryor, entertained the Laugh Fac'
[s-0.13 nw720 sw0.37 uq0.51 fs0.00] ' wrestling season is in its transitory period, the calm before the three-week storm, when the District 2, Northeast Regional and PIAA Championships thin out the herd.\nThere is still wrestling to be done before districts '
[s-0.13 nw548 sw0.41 uq0.49 fs0.08] 'andro-, andr-, -ander, -andra, -andria, -andrian, -andric, -andrism, -androus, -andries, -andry\n(Greek: man, men, male, masculine; also, stamen or anther as used in botany)\n2. Normally virilized, as in the male.\n2. Perta'
[s-0.13 nw574 sw0.45 uq0.48 fs0.00] 'It would probably be best to start off with describing what this book is not. Its definitely not a cover-to-cover bedtime read. Its more something for the military historian to keep on the shelf for reference, and that t'
=== remaining ===
3858
[metadata] {"noOutputExpected": false}

rank  score   sw    mwl   uniq  nw   | head
   0  +0.44  0.23  5.03  0.34   975 | |PREDECESSORS AND SHORT HISTORY: in the 16th century by Rao Gopalji fl.1550, 4th son of Raja Pr
   1  +0.34  0.23  7.17  0.73  1200 | Archaeological dating methods Futilities must undress thitherward convened dismissively toweris
   2  +0.31  0.21  6.38  0.69   648 | 8A<|endoftext|>Our god saves chords pdfOut of many by john mack faragher Anteorbital Armond bef
   3  +0.30  0.25  5.35  0.73   124 | Lahore:In the prospects of stoned to death incident of a Khanewal Woman, Inspector General of P
   4  +0.29  0.34  4.81  0.42   209 | <|endoftext|>New Delhi: Five new Supreme Court judges were today sworn in, taking the top court
   5  +0.29  0.35  4.76  0.37  1200 | Hill 303 massacre |Hill 303 massacre| Bodies of massacre victims gathered near Waegwan, South K
   6  +0.28  0.22  6.58  0.64   623 | a<|endoftext|>Czech Tobe alit it staleness checker unguardedly. lignifies sigillate that unsnar
   7  +0.27  0.45  4.43  0.54    96 | ggor Domm II (375BDW-51ADW) is widely considered to be the greatest ruler in the history of the
   8  +0.24  0.36  4.92  0.62   170 | The Bharatiya Janata Party (BJP) leaders of the West Bengal unit will meet Union Home Minister 
   9  +0.23  0.30  5.17  0.43   528 | Amarnath Yatra 2017 Amarnath Yatra 2017 News Jammu and Kashmir police said three people, allege
  10  +0.23  0.34  5.01  0.56   231 | oor ul HaqBaramulla, Aug 10: Army on Friday said the five militants killed in upper reaches of 
  11  +0.23  0.36  4.71  0.54   212 | New Delhi | ANI: Yes Bank promoter Rana Kapoor informed the Enforcement Directorate (ED) that h
  12  +0.22  0.42  4.90  0.40   253 | After the death of the cloistered emperor Toba, the emperor Go-Shirakawa and the retired empero
  13  +0.22  0.39  4.76  0.55   285 |  which is declining as a force in West Bengal since its crushing defeat in 2011 has become weak
  14  +0.22  0.27  6.05  0.63  1200 |  isolative that reived discordantly? Gil Gil schizophà © tico undoing it in a carved it might b
  15  +0.22  0.32  4.97  0.63   207 | <|endoftext|>Colossi are used to rip cracker datum and agouti. yahoo. My easter ulna had been b
  16  +0.22  0.33  5.13  0.57   227 | |Rediff India Abroad Home | All the sections| Bihar: Vigilante justice resurfaces, three people
  17  +0.21  0.40  4.84  0.62   152 | Major Bhupinder Singh In sept. 1965, Major Bhupinder Singh commanded the "B" squadron of the 4 
  18  +0.21  0.23  4.78  0.38  1200 | Pan-African orogeny - Wikipedia Pan-African orogeny From Wikipedia, de free encycwopedia Jump t
  19  +0.21  0.28  3.97  0.79   239 |  hesiall, Loo. Micks. The ness arou shis. I therrying ot! Mmmend bou. Mays 14 he sairry, by nor
  20  +0.21  0.21  6.62  0.65  1200 |  pupate and stereotypic disclose alloys or reconsidered temptingly. choriambic broken-backed an
  21  +0.20  0.23  4.77  0.38  1200 |  Pan-African orogeny From Wikipedia, de free encycwopedia Jump to navigation Jump to search A s
  22  +0.20  0.39  5.10  0.48   347 | DMK chief MK Stalin on Sunday hit out at the Election Commission for not announcing by-election
  23  +0.20  0.23  6.22  0.65  1200 | <|endoftext|>Wallache leads frequently, his coact mercenariamente. Equatorial and intransigent 
  24  +0.20  0.33  8.06  0.58  1200 |  page print pageAll Topic Reviews Anger and Forgiveness"Are You There Alone?"10 Good Questions 
  25  +0.20  0.25  6.23  0.61  1200 | <|endoftext|>Notes energy solar application The controversial ingenuity is hydrogenated, solar 
  26  +0.19  0.40  4.67  0.61   184 |  place of birth unknown; died Apr. 13, 989. Byzantine military commander. A member of a promine
  27  +0.19  0.34  4.75  0.61   162 | UPDATE: Court remands 45-year-old man for raping 15-year-old daughter A 45-year-old man, Durodo
  28  +0.19  0.37  4.89  0.69   150 | Bengaluru: Caught in a controversy over alleged telephone tapping of his political rivals durin
  29  +0.19  0.29  5.10  0.63   267 | Two militants killed in encounter in J&K Srinagar: Two unidentified militants were on Friday ki
  30  +0.19  0.35  5.58  0.54   239 | Mangaluru, July 25, 2018: MP Nalin Kumar Kateel met Union ministers Manoj Sinha and Rajen Gohai
  31  +0.19  0.39  4.56  0.45   540 | Anglo-Dutch Wars, also called Dutch Wars, Dutch Engelse Oorlogen, four 17th- and 18th-century n
  32  +0.19  0.21  3.98  0.63  1200 | Newspaper Page Text ratsltaatilssa . alltl BalldUna XWtW'W i wfc - 7"" ' " lLLlWUlbChiM'UAL K i
  33  +0.19  0.36  4.76  0.58   288 | vel (Maharashtra), June 14 (ANI): A magistrate's court in Panvel, Maharashtra, on Sunday extend
  34  +0.19  0.37  4.48  0.60   121 | The attack was launched at 0730hrs on the 1st July 1916. Along a twenty mile Front 200,000 Brit
  35  +0.19  0.24  6.03  0.74   485 | amaican up a during break men 92% International dating site uk 98% Half-life carbon dating exam
  36  +0.19  0.31  6.64  0.50  1200 | „Igowithyoutogether!”Qing Xuzithinksto say!„Master, Ialsotogethergo!”Cheng Yureallywantsto sees
  37  +0.18  0.27  4.50  0.35   799 | Sacred-Texts Christianity Index Previous Next OF THE MESSIANIC GENERATIONS5. GOD created Adam. 
  38  +0.18  0.33  4.80  0.47   490 | From Wikipedia, the free encyclopedia |February 24, 1882 - December 16, 1948 (aged 66)| General
  39  +0.18  0.31  4.94  0.64   225 | Agartala, Sep 15 (IANS) The BJP on Tuesday said it would soon launch a stir if Tripura’s Left F

[stdout]
rank  score   sw    mwl   uniq  nw   | head
   0  +0.44  0.23  5.03  0.34   975 | |PREDECESSORS AND SHORT HISTORY: in the 16th century by Rao Gopalji fl.1550, 4th son of Raja Pr
   1  +0.34  0.23  7.17  0.73  1200 | Archaeological dating methods Futilities must undress thitherward convened dismissively toweris
   2  +0.31  0.21  6.38  0.69   648 | 8A<|endoftext|>Our god saves chords pdfOut of many by john mack faragher Anteorbital Armond bef
   3  +0.30  0.25  5.35  0.73   124 | Lahore:In the prospects of stoned to death incident of a Khanewal Woman, Inspector General of P
   4  +0.29  0.34  4.81  0.42   209 | <|endoftext|>New Delhi: Five new Supreme Court judges were today sworn in, taking the top court
   5  +0.29  0.35  4.76  0.37  1200 | Hill 303 massacre |Hill 303 massacre| Bodies of massacre victims gathered near Waegwan, South K
   6  +0.28  0.22  6.58  0.64   623 | a<|endoftext|>Czech Tobe alit it staleness checker unguardedly. lignifies sigillate that unsnar
   7  +0.27  0.45  4.43  0.54    96 | ggor Domm II (375BDW-51ADW) is widely considered to be the greatest ruler in the history of the
   8  +0.24  0.36  4.92  0.62   170 | The Bharatiya Janata Party (BJP) leaders of the West Bengal unit will meet Union Home Minister 
   9  +0.23  0.30  5.17  0.43   528 | Amarnath Yatra 2017 Amarnath Yatra 2017 News Jammu and Kashmir police said three people, allege
  10  +0.23  0.34  5.01  0.56   231 | oor ul HaqBaramulla, Aug 10: Army on Friday said the five militants killed in upper reaches of 
  11  +0.23  0.36  4.71  0.54   212 | New Delhi | ANI: Yes Bank promoter Rana Kapoor informed the Enforcement Directorate (ED) that h
  12  +0.22  0.42  4.90  0.40   253 | After the death of the cloistered emperor Toba, the emperor Go-Shirakawa and the retired empero
  13  +0.22  0.39  4.76  0.55   285 |  which is declining as a force in West Bengal since its crushing defeat in 2011 has become weak
  14  +0.22  0.27  6.05  0.63  1200 |  isolative that reived discordantly? Gil Gil schizophà © tico undoing it in a carved it might b
  15  +0.22  0.32  4.97  0.63   207 | <|endoftext|>Colossi are used to rip cracker datum and agouti. yahoo. My easter ulna had been b
  16  +0.22  0.33  5.13  0.57   227 | |Rediff India Abroad Home | All the sections| Bihar: Vigilante justice resurfaces, three people
  17  +0.21  0.40  4.84  0.62   152 | Major Bhupinder Singh In sept. 1965, Major Bhupinder Singh commanded the "B" squadron of the 4 
  18  +0.21  0.23  4.78  0.38  1200 | Pan-African orogeny - Wikipedia Pan-African orogeny From Wikipedia, de free encycwopedia Jump t
  19  +0.21  0.28  3.97  0.79   239 |  hesiall, Loo. Micks. The ness arou shis. I therrying ot! Mmmend bou. Mays 14 he sairry, by nor
  20  +0.21  0.21  6.62  0.65  1200 |  pupate and stereotypic disclose alloys or reconsidered temptingly. choriambic broken-backed an
  21  +0.20  0.23  4.77  0.38  1200 |  Pan-African orogeny From Wikipedia, de free encycwopedia Jump to navigation Jump to search A s
  22  +0.20  0.39  5.10  0.48   347 | DMK chief MK Stalin on Sunday hit out at the Election Commission for not announcing by-election
  23  +0.20  0.23  6.22  0.65  1200 | <|endoftext|>Wallache leads frequently, his coact mercenariamente. Equatorial and intransigent 
  24  +0.20  0.33  8.06  0.58  1200 |  page print pageAll Topic Reviews Anger and Forgiveness"Are You There Alone?"10 Good Questions 
  25  +0.20  0.25  6.23  0.61  1200 | <|endoftext|>Notes energy solar application The controversial ingenuity is hydrogenated, solar 
  26  +0.19  0.40  4.67  0.61   184 |  place of birth unknown; died Apr. 13, 989. Byzantine military commander. A member of a promine
  27  +0.19  0.34  4.75  0.61   162 | UPDATE: Court remands 45-year-old man for raping 15-year-old daughter A 45-year-old man, Durodo
  28  +0.19  0.37  4.89  0.69   150 | Bengaluru: Caught in a controversy over alleged telephone tapping of his political rivals durin
  29  +0.19  0.29  5.10  0.63   267 | Two militants killed in encounter in J&K Srinagar: Two unidentified militants were on Friday ki
  30  +0.19  0.35  5.58  0.54   239 | Mangaluru, July 25, 2018: MP Nalin Kumar Kateel met Union ministers Manoj Sinha and Rajen Gohai
  31  +0.19  0.39  4.56  0.45   540 | Anglo-Dutch Wars, also called Dutch Wars, Dutch Engelse Oorlogen, four 17th- and 18th-century n
  32  +0.19  0.21  3.98  0.63  1200 | Newspaper Page Text ratsltaatilssa . alltl BalldUna XWtW'W i wfc - 7"" ' " lLLlWUlbChiM'UAL K i
  33  +0.19  0.36  4.76  0.58   288 | vel (Maharashtra), June 14 (ANI): A magistrate's court in Panvel, Maharashtra, on Sunday extend
  34  +0.19  0.37  4.48  0.60   121 | The attack was launched at 0730hrs on the 1st July 1916. Along a twenty mile Front 200,000 Brit
  35  +0.19  0.24  6.03  0.74   485 | amaican up a during break men 92% International dating site uk 98% Half-life carbon dating exam
  36  +0.19  0.31  6.64  0.50  1200 | „Igowithyoutogether!”Qing Xuzithinksto say!„Master, Ialsotogethergo!”Cheng Yureallywantsto sees
  37  +0.18  0.27  4.50  0.35   799 | Sacred-Texts Christianity Index Previous Next OF THE MESSIANIC GENERATIONS5. GOD created Adam. 
  38  +0.18  0.33  4.80  0.47   490 | From Wikipedia, the free encyclopedia |February 24, 1882 - December 16, 1948 (aged 66)| General
  39  +0.18  0.31  4.94  0.64   225 | Agartala, Sep 15 (IANS) The BJP on Tuesday said it would soon launch a stir if Tripura’s Left F
[metadata] {"noOutputExpected": false}

The file /workspace/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/curate.py", "oldString": "    ok = ((nw >= 50) & (alpha >= 0.62) & (stopfrac >= 0.20) & (uniq >= 0.30) &\n          (uniq <= 0.92) & (mwl <= 10.0) & (ascii_ >= 0.92) & (sym <= 0.12) &\n          (fshort <= 0.55) & (digit <= 0.20))", "newString": "    # Thresholds calibrated against the top-ranked docs: spun word-salad / OCR /\n    # character-corrupted text clusters at stopfrac<=0.25 and/or mwl>=6.0 and/or\n    # uniq>=0.80, while genuine news/encyclopedic prose sits at stopfrac>=0.29,\n    # mwl 4.4-5.6. English function-word density (stopfrac) is also the cleanest\n    # non-English filter.\n    ok = ((nw >= 50) & (alpha >= 0.62) & (stopfrac >= 0.26) & (uniq >= 0.30) &\n          (uniq <= 0.80) & (mwl <= 6.0) & (ascii_ >= 0.92) & (sym <= 0.12) &\n          (fshort <= 0.55) & (digit <= 0.20))", "originalFile": null, "structuredPatch": [{"oldStart": 204, "oldLines": 8, "newStart": 204, "newLines": 13, "lines": ["     # English clean-prose junk filter: drops word-salad (stopword floor), menu/", "     # boilerplate (short-line + uniq caps), non-English (ascii + stopword),", "     # markup/data dumps (alpha/sym/digit), repetitive spam (uniq floor).", "-    ok = ((nw >= 50) & (alpha >= 0.62) & (stopfrac >= 0.20) & (uniq >= 0.30) &", "-          (uniq <= 0.92) & (mwl <= 10.0) & (ascii_ >= 0.92) & (sym <= 0.12) &", "+    # Thresholds calibrated against the top-ranked docs: spun word-salad / OCR /", "+    # character-corrupted text clusters at stopfrac<=0.25 and/or mwl>=6.0 and/or", "+    # uniq>=0.80, while genuine news/encyclopedic prose sits at stopfrac>=0.29,", "+    # mwl 4.4-5.6. English function-word density (stopfrac) is also the cleanest", "+    # non-English filter.", "+    ok = ((nw >= 50) & (alpha >= 0.62) & (stopfrac >= 0.26) & (uniq >= 0.30) &", "+          (uniq <= 0.80) & (mwl <= 6.0) & (ascii_ >= 0.92) & (sym <= 0.12) &", "           (fshort <= 0.55) & (digit <= 0.20))", "     cand = np.where(ok)[0]", "     cand = cand[np.argsort(-score[cand])]"]}], "userModified": false, "replaceAll": false}

[1] loading feature cache
    N=182016 feats in 1s
[2] candidates: 107423 pass filter (dropped 74593)
    selected 45000 docs (~52M est tokens, best-first)
[3] wrote 45000 ids -> /workspace/submission/selection.json (~52M est tokens) in 3s
=== verify top 25 clean ===
candidates: 107423
    0 +0.29 sw0.34 mwl4.8 | <|endoftext|>New Delhi: Five new Supreme Court judges were today sworn in, taking the to
    1 +0.29 sw0.35 mwl4.8 | Hill 303 massacre |Hill 303 massacre| Bodies of massacre victims gathered near Waegwan, 
    2 +0.27 sw0.45 mwl4.4 | ggor Domm II (375BDW-51ADW) is widely considered to be the greatest ruler in the history
    3 +0.24 sw0.36 mwl4.9 | The Bharatiya Janata Party (BJP) leaders of the West Bengal unit will meet Union Home Mi
    4 +0.23 sw0.30 mwl5.2 | Amarnath Yatra 2017 Amarnath Yatra 2017 News Jammu and Kashmir police said three people,
    5 +0.23 sw0.34 mwl5.0 | oor ul HaqBaramulla, Aug 10: Army on Friday said the five militants killed in upper reac
    6 +0.23 sw0.36 mwl4.7 | New Delhi | ANI: Yes Bank promoter Rana Kapoor informed the Enforcement Directorate (ED)
    7 +0.22 sw0.42 mwl4.9 | After the death of the cloistered emperor Toba, the emperor Go-Shirakawa and the retired
    8 +0.22 sw0.39 mwl4.8 |  which is declining as a force in West Bengal since its crushing defeat in 2011 has beco
    9 +0.22 sw0.32 mwl5.0 | <|endoftext|>Colossi are used to rip cracker datum and agouti. yahoo. My easter ulna had
   10 +0.22 sw0.33 mwl5.1 | |Rediff India Abroad Home | All the sections| Bihar: Vigilante justice resurfaces, three
   11 +0.21 sw0.40 mwl4.8 | Major Bhupinder Singh In sept. 1965, Major Bhupinder Singh commanded the "B" squadron of
   12 +0.21 sw0.28 mwl4.0 |  hesiall, Loo. Micks. The ness arou shis. I therrying ot! Mmmend bou. Mays 14 he sairry,
   13 +0.20 sw0.39 mwl5.1 | DMK chief MK Stalin on Sunday hit out at the Election Commission for not announcing by-e
   14 +0.19 sw0.40 mwl4.7 |  place of birth unknown; died Apr. 13, 989. Byzantine military commander. A member of a 
   15 +0.19 sw0.34 mwl4.7 | UPDATE: Court remands 45-year-old man for raping 15-year-old daughter A 45-year-old man,
   16 +0.19 sw0.37 mwl4.9 | Bengaluru: Caught in a controversy over alleged telephone tapping of his political rival
   17 +0.19 sw0.29 mwl5.1 | Two militants killed in encounter in J&K Srinagar: Two unidentified militants were on Fr
   18 +0.19 sw0.35 mwl5.6 | Mangaluru, July 25, 2018: MP Nalin Kumar Kateel met Union ministers Manoj Sinha and Raje
   19 +0.19 sw0.39 mwl4.6 | Anglo-Dutch Wars, also called Dutch Wars, Dutch Engelse Oorlogen, four 17th- and 18th-ce
   20 +0.19 sw0.36 mwl4.8 | vel (Maharashtra), June 14 (ANI): A magistrate's court in Panvel, Maharashtra, on Sunday
   21 +0.19 sw0.37 mwl4.5 | The attack was launched at 0730hrs on the 1st July 1916. Along a twenty mile Front 200,0
   22 +0.18 sw0.27 mwl4.5 | Sacred-Texts Christianity Index Previous Next OF THE MESSIANIC GENERATIONS5. GOD created
   23 +0.18 sw0.33 mwl4.8 | From Wikipedia, the free encyclopedia |February 24, 1882 - December 16, 1948 (aged 66)| 
   24 +0.18 sw0.31 mwl4.9 | Agartala, Sep 15 (IANS) The BJP on Tuesday said it would soon launch a stir if Tripura’s
  100 +0.13 sw0.38 mwl4.9 | 6 villagers killed, about 20 injured Six villagers were killed and about 20 injured when
 1000 +0.01 sw0.29 mwl5.4 | FBI - Federal Bureau of Investigation Federal Bureau of Investigation, formed on July 26
 5000 -0.09 sw0.40 mwl4.3 | <|endoftext|>|courtesy of singtao.com| |Jackie Chan takes off as soon as he hears questi
11000 -0.14 sw0.51 mwl4.8 | <|endoftext|>Mounted Rifle Tactics. Definition - Principles underlying tactics - Definit

[stdout]
[1] loading feature cache
    N=182016 feats in 1s
[2] candidates: 107423 pass filter (dropped 74593)
    selected 45000 docs (~52M est tokens, best-first)
[3] wrote 45000 ids -> /workspace/submission/selection.json (~52M est tokens) in 3s
=== verify top 25 clean ===
candidates: 107423
    0 +0.29 sw0.34 mwl4.8 | <|endoftext|>New Delhi: Five new Supreme Court judges were today sworn in, taking the to
    1 +0.29 sw0.35 mwl4.8 | Hill 303 massacre |Hill 303 massacre| Bodies of massacre victims gathered near Waegwan, 
    2 +0.27 sw0.45 mwl4.4 | ggor Domm II (375BDW-51ADW) is widely considered to be the greatest ruler in the history
    3 +0.24 sw0.36 mwl4.9 | The Bharatiya Janata Party (BJP) leaders of the West Bengal unit will meet Union Home Mi
    4 +0.23 sw0.30 mwl5.2 | Amarnath Yatra 2017 Amarnath Yatra 2017 News Jammu and Kashmir police said three people,
    5 +0.23 sw0.34 mwl5.0 | oor ul HaqBaramulla, Aug 10: Army on Friday said the five militants killed in upper reac
    6 +0.23 sw0.36 mwl4.7 | New Delhi | ANI: Yes Bank promoter Rana Kapoor informed the Enforcement Directorate (ED)
    7 +0.22 sw0.42 mwl4.9 | After the death of the cloistered emperor Toba, the emperor Go-Shirakawa and the retired
    8 +0.22 sw0.39 mwl4.8 |  which is declining as a force in West Bengal since its crushing defeat in 2011 has beco
    9 +0.22 sw0.32 mwl5.0 | <|endoftext|>Colossi are used to rip cracker datum and agouti. yahoo. My easter ulna had
   10 +0.22 sw0.33 mwl5.1 | |Rediff India Abroad Home | All the sections| Bihar: Vigilante justice resurfaces, three
   11 +0.21 sw0.40 mwl4.8 | Major Bhupinder Singh In sept. 1965, Major Bhupinder Singh commanded the "B" squadron of
   12 +0.21 sw0.28 mwl4.0 |  hesiall, Loo. Micks. The ness arou shis. I therrying ot! Mmmend bou. Mays 14 he sairry,
   13 +0.20 sw0.39 mwl5.1 | DMK chief MK Stalin on Sunday hit out at the Election Commission for not announcing by-e
   14 +0.19 sw0.40 mwl4.7 |  place of birth unknown; died Apr. 13, 989. Byzantine military commander. A member of a 
   15 +0.19 sw0.34 mwl4.7 | UPDATE: Court remands 45-year-old man for raping 15-year-old daughter A 45-year-old man,
   16 +0.19 sw0.37 mwl4.9 | Bengaluru: Caught in a controversy over alleged telephone tapping of his political rival
   17 +0.19 sw0.29 mwl5.1 | Two militants killed in encounter in J&K Srinagar: Two unidentified militants were on Fr
   18 +0.19 sw0.35 mwl5.6 | Mangaluru, July 25, 2018: MP Nalin Kumar Kateel met Union ministers Manoj Sinha and Raje
   19 +0.19 sw0.39 mwl4.6 | Anglo-Dutch Wars, also called Dutch Wars, Dutch Engelse Oorlogen, four 17th- and 18th-ce
   20 +0.19 sw0.36 mwl4.8 | vel (Maharashtra), June 14 (ANI): A magistrate's court in Panvel, Maharashtra, on Sunday
   21 +0.19 sw0.37 mwl4.5 | The attack was launched at 0730hrs on the 1st July 1916. Along a twenty mile Front 200,0
   22 +0.18 sw0.27 mwl4.5 | Sacred-Texts Christianity Index Previous Next OF THE MESSIANIC GENERATIONS5. GOD created
   23 +0.18 sw0.33 mwl4.8 | From Wikipedia, the free encyclopedia |February 24, 1882 - December 16, 1948 (aged 66)| 
   24 +0.18 sw0.31 mwl4.9 | Agartala, Sep 15 (IANS) The BJP on Tuesday said it would soon launch a stir if Tripura’s
  100 +0.13 sw0.38 mwl4.9 | 6 villagers killed, about 20 injured Six villagers were killed and about 20 injured when
 1000 +0.01 sw0.29 mwl5.4 | FBI - Federal Bureau of Investigation Federal Bureau of Investigation, formed on July 26
 5000 -0.09 sw0.40 mwl4.3 | <|endoftext|>|courtesy of singtao.com| |Jackie Chan takes off as soon as he hears questi
11000 -0.14 sw0.51 mwl4.8 | <|endoftext|>Mounted Rifle Tactics. Definition - Principles underlying tactics - Definit
[metadata] {"noOutputExpected": false}

Command running in background with ID: b022hnxh7. Output is being written to: /tmp/claude-0/-workspace/241b2f82-9a7f-471b-beef-757ffa753d6d/tasks/b022hnxh7.output. You will be notified when it completes. To check interim output, use Read on that file path.
Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands.

[metadata] {"noOutputExpected": false, "backgroundTaskId": "b022hnxh7", "backgroundCwdHint": "Session cwd remains /workspace; directory changes made by the backgrounded command do not apply to subsequent commands."}

Updated task #1 status

[metadata] {"success": true, "taskId": "1", "updatedFields": ["status"], "statusChange": {"from": "in_progress", "to": "completed"}}

Updated task #2 status

[metadata] {"success": true, "taskId": "2", "updatedFields": ["status"], "statusChange": {"from": "pending", "to": "completed"}}

Updated task #3 status

[metadata] {"success": true, "taskId": "3", "updatedFields": ["status"], "statusChange": {"from": "pending", "to": "in_progress"}}

1	Token indices sequence length is longer than the specified maximum sequence length for this model (5414 > 1024). Running this sequence through the model will result in indexing errors
2	packed 12000000 tokens from 45000 ids -> /tmp/dev_train.npy
3	HELDOUT_PPL 384.965  loss 5.9532  train_s 54  /tmp/dev_train.npy
4	(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)
5	DONE 0
6

[metadata] {"type": "text", "file": {"filePath": "/tmp/eval_v1.log", "content": "Token indices sequence length is longer than the specified maximum sequence length for this model (5414 > 1024). Running this sequence through the model will result in indexing errors\npacked 12000000 tokens from 45000 ids -> /tmp/dev_train.npy\nHELDOUT_PPL 384.965  loss 5.9532  train_s 54  /tmp/dev_train.npy\n(dev PPL on multi_dev; official uses a hidden disjoint sample of the same HQ domain)\nDONE 0\n", "numLines": 6, "startLine": 1, "totalLines": 6}}

3477
---head of selection---
n_ids 45000 unique 45000 min 1 max 181981

[stdout]
3477
---head of selection---
n_ids 45000 unique 45000 min 1 max 181981
[metadata] {"noOutputExpected": false}

=== TOP 30 docs in current selection.json ===
 0 id 58452 | <|endoftext|>New Delhi: Five new Supreme Court judges were today sworn in, taking the top court’s strength to 28, including the Ch
 1 id 82269 | Hill 303 massacre |Hill 303 massacre| Bodies of massacre victims gathered near Waegwan, South Korea, many with their hands still b
 2 id 88750 | ggor Domm II (375BDW-51ADW) is widely considered to be the greatest ruler in the history of the Nethereigons. He reigned the North
 3 id 37064 | The Bharatiya Janata Party (BJP) leaders of the West Bengal unit will meet Union Home Minister Rajnath Singh on Thursday to lodge 
 4 id 41186 | Amarnath Yatra 2017 Amarnath Yatra 2017 News Jammu and Kashmir police said three people, alleged conspirators in the 10 July attac
 5 id 69836 | oor ul HaqBaramulla, Aug 10: Army on Friday said the five militants killed in upper reaches of Rafiabad during two-day gunfight we
 6 id 36353 | New Delhi | ANI: Yes Bank promoter Rana Kapoor informed the Enforcement Directorate (ED) that he was 'forced' by a then Congress U
 7 id  9445 | After the death of the cloistered emperor Toba, the emperor Go-Shirakawa and the retired emperor Sutoku and disputed over successi
 8 id113491 |  which is declining as a force in West Bengal since its crushing defeat in 2011 has become weaker in Tamluk assembly constituency 
 9 id 38752 | <|endoftext|>Colossi are used to rip cracker datum and agouti. yahoo. My easter ulna had been bullshitting prolepsis. yahoo. My to
10 id 81859 | |Rediff India Abroad Home | All the sections| Bihar: Vigilante justice resurfaces, three people lynched February 18, 2008 17:20 IS
11 id 76107 | Major Bhupinder Singh In sept. 1965, Major Bhupinder Singh commanded the "B" squadron of the 4 Horse against Pakistani forces. On 
12 id 98806 |  hesiall, Loo. Micks. The ness arou shis. I therrying ot! Mmmend bou. Mays 14 he sairry, by norms nexpren I ateseel ing thaver. Wa
13 id  3196 | DMK chief MK Stalin on Sunday hit out at the Election Commission for not announcing by-elections to two Tamil Nadu assembly consti
14 id 98903 |  place of birth unknown; died Apr. 13, 989. Byzantine military commander. A member of a prominent Cappadocian family, Phocas was t
15 id 68634 | UPDATE: Court remands 45-year-old man for raping 15-year-old daughter A 45-year-old man, Durodola Kehinde has been remanded in pri
16 id 55671 | Bengaluru: Caught in a controversy over alleged telephone tapping of his political rivals during his tenure as CM of a JD(S)-Congr
17 id 41908 | Two militants killed in encounter in J&K Srinagar: Two unidentified militants were on Friday killed in an encounter with security 
18 id 15477 | Mangaluru, July 25, 2018: MP Nalin Kumar Kateel met Union ministers Manoj Sinha and Rajen Gohain and urged them to take measures t
19 id  3475 | Anglo-Dutch Wars, also called Dutch Wars, Dutch Engelse Oorlogen, four 17th- and 18th-century naval conflicts between England and 
20 id 43758 | vel (Maharashtra), June 14 (ANI): A magistrate's court in Panvel, Maharashtra, on Sunday extended suspended senior Nationalist Con
21 id 12364 | The attack was launched at 0730hrs on the 1st July 1916. Along a twenty mile Front 200,000 British and French troops attacked stro
22 id  9401 | Sacred-Texts Christianity Index Previous Next OF THE MESSIANIC GENERATIONS5. GOD created Adam. Adam begat Seth. Seth begat Enos. E
23 id  2027 | From Wikipedia, the free encyclopedia |February 24, 1882 - December 16, 1948 (aged 66)| General Hayao Tada |Place of birth||Sendai
24 id  4620 | Agartala, Sep 15 (IANS) The BJP on Tuesday said it would soon launch a stir if Tripura’s Left Front government evicted tribals to 
25 id 25183 | Damascus denies US report on clash with Syrian infiltrators to Iraq A spokesman of the Syrian delegation to the Islamic States Sum
26 id 41027 | Jammu, Sep 30 (IANS) Pakistan continued to violate ceasefire along the Line of Control (LoC) in Jammu and Kashmir’s Poonch distric
27 id 96891 |  Independence |Mexican War of Independence| |Miguel Hidalgo y Costilla Jos Mara Morelos Francisco Javier Mina Agustn de Iturbide. 
28 id107506 |  customers.<|endoftext|>Army jawan shoots himself dead in Jammu Two bullets pierced his head and the helmet he was wearing.india U
29 id  6827 | Slippers were hurled at RJD leader Tejashwi Yadav during a poll rally in Bihar’s Aurangabad. The leader was sitting with party wor

[stdout]
=== TOP 30 docs in current selection.json ===
 0 id 58452 | <|endoftext|>New Delhi: Five new Supreme Court judges were today sworn in, taking the top court’s strength to 28, including the Ch
 1 id 82269 | Hill 303 massacre |Hill 303 massacre| Bodies of massacre victims gathered near Waegwan, South Korea, many with their hands still b
 2 id 88750 | ggor Domm II (375BDW-51ADW) is widely considered to be the greatest ruler in the history of the Nethereigons. He reigned the North
 3 id 37064 | The Bharatiya Janata Party (BJP) leaders of the West Bengal unit will meet Union Home Minister Rajnath Singh on Thursday to lodge 
 4 id 41186 | Amarnath Yatra 2017 Amarnath Yatra 2017 News Jammu and Kashmir police said three people, alleged conspirators in the 10 July attac
 5 id 69836 | oor ul HaqBaramulla, Aug 10: Army on Friday said the five militants killed in upper reaches of Rafiabad during two-day gunfight we
 6 id 36353 | New Delhi | ANI: Yes Bank promoter Rana Kapoor informed the Enforcement Directorate (ED) that he was 'forced' by a then Congress U
 7 id  9445 | After the death of the cloistered emperor Toba, the emperor Go-Shirakawa and the retired emperor Sutoku and disputed over successi
 8 id113491 |  which is declining as a force in West Bengal since its crushing defeat in 2011 has become weaker in Tamluk assembly constituency 
 9 id 38752 | <|endoftext|>Colossi are used to rip cracker datum and agouti. yahoo. My easter ulna had been bullshitting prolepsis. yahoo. My to
10 id 81859 | |Rediff India Abroad Home | All the sections| Bihar: Vigilante justice resurfaces, three people lynched February 18, 2008 17:20 IS
11 id 76107 | Major Bhupinder Singh In sept. 1965, Major Bhupinder Singh commanded the "B" squadron of the 4 Horse against Pakistani forces. On 
12 id 98806 |  hesiall, Loo. Micks. The ness arou shis. I therrying ot! Mmmend bou. Mays 14 he sairry, by norms nexpren I ateseel ing thaver. Wa
13 id  3196 | DMK chief MK Stalin on Sunday hit out at the Election Commission for not announcing by-elections to two Tamil Nadu assembly consti
14 id 98903 |  place of birth unknown; died Apr. 13, 989. Byzantine military commander. A member of a prominent Cappadocian family, Phocas was t
15 id 68634 | UPDATE: Court remands 45-year-old man for raping 15-year-old daughter A 45-year-old man, Durodola Kehinde has been remanded in pri
16 id 55671 | Bengaluru: Caught in a controversy over alleged telephone tapping of his political rivals during his tenure as CM of a JD(S)-Congr
17 id 41908 | Two militants killed in encounter in J&K Srinagar: Two unidentified militants were on Friday killed in an encounter with security 
18 id 15477 | Mangaluru, July 25, 2018: MP Nalin Kumar Kateel met Union ministers Manoj Sinha and Rajen Gohain and urged them to take measures t
19 id  3475 | Anglo-Dutch Wars, also called Dutch Wars, Dutch Engelse Oorlogen, four 17th- and 18th-century naval conflicts between England and 
20 id 43758 | vel (Maharashtra), June 14 (ANI): A magistrate's court in Panvel, Maharashtra, on Sunday extended suspended senior Nationalist Con
21 id 12364 | The attack was launched at 0730hrs on the 1st July 1916. Along a twenty mile Front 200,000 British and French troops attacked stro
22 id  9401 | Sacred-Texts Christianity Index Previous Next OF THE MESSIANIC GENERATIONS5. GOD created Adam. Adam begat Seth. Seth begat Enos. E
23 id  2027 | From Wikipedia, the free encyclopedia |February 24, 1882 - December 16, 1948 (aged 66)| General Hayao Tada |Place of birth||Sendai
24 id  4620 | Agartala, Sep 15 (IANS) The BJP on Tuesday said it would soon launch a stir if Tripura’s Left Front government evicted tribals to 
25 id 25183 | Damascus denies US report on clash with Syrian infiltrators to Iraq A spokesman of the Syrian delegation to the Islamic States Sum
26 id 41027 | Jammu, Sep 30 (IANS) Pakistan continued to violate ceasefire along the Line of Control (LoC) in Jammu and Kashmir’s Poonch distric
27 id 96891 |  Independence |Mexican War of Independence| |Miguel Hidalgo y Costilla Jos Mara Morelos Francisco Javier Mina Agustn de Iturbide. 
28 id107506 |  customers.<|endoftext|>Army jawan shoots himself dead in Jammu Two bullets pierced his head and the helmet he was wearing.india U
29 id  6827 | Slippers were hurled at RJD leader Tejashwi Yadav during a poll rally in Bihar’s Aurangabad. The leader was sitting with party wor
[metadata] {"noOutputExpected": false}

1	#!/usr/bin/env python3
2	"""
3	Curate a raw web pool for pretraining a small LM whose eval target is a BROAD,
4	multi-domain high-quality English mixture (equal parts encyclopedic/Wikipedia,
5	general high-quality web prose, news, technical Q&A).
6	
7	STATED CRITERION
8	----------------
9	Keep pool documents that are (a) clean, fluent, information-dense English and
10	(b) distributionally close to the disclosed target register mix.
11	
12	Target-closeness is a DSIR / NBSVM-style log-likelihood ratio. For hashed word
13	unigram+bigram features we weight each feature f by
14	    r[f] = log P(f | target register) - log P(f | pool background)
15	estimated from feature document-frequencies (Laplace-smoothed). A document's score
16	is the MEAN over its features of clip(r[f], -5, 5). Using the mean (not the sum)
17	removes length bias, and clipping stops a handful of ultra-rare "fancy" words from
18	letting SEO word-salad win -- the two dominant failure modes of a raw sum.
19	
20	The target is a fixed mixture including ~25% technical Q&A (identifiable by HTML/code
21	markup in the decoded dev set). We AUDITED the pool and found it contains essentially
22	NO code / programming / StackOverflow content (the `<code>`/`&lt;` markers are absent
23	and "code-like" punctuation counts trace back to ordinary prose). So the code surface
24	of the Q&A register is unmatchable from this pool, and an explicit "technical" selector
25	only drags in multilingual boilerplate and SEO word-salad (empirically its top hits) --
26	which would displace good prose AND fail to teach code, hurting perplexity both ways.
27	
28	We therefore use ONE combined head whose positives are ALL decoded dev segments
29	(encyclopedic + news + web + the Q&A questions). Folding the Q&A positives in keeps
30	their *technical vocabulary* (function, array, query, server, ...) in the log-odds, so
31	genuinely technical PROSE in the pool is still up-ranked -- the best available proxy for
32	the Q&A register when no code exists to select. Each pool doc must pass an English,
33	clean-prose junk filter (drops word-salad, menu/boilerplate, non-English, markup dumps,
34	repetitive spam), is deduplicated by a content fingerprint, and is ranked by the mean
35	clipped log-odds. Docs are emitted best-first; the packer fills 12M tokens from the top.
36	
37	Positives come only from the disclosed dev target; the selection is over disjoint pool
38	ids. The output is an ordered id list produced entirely by this script.
39	"""
40	import json, re, zlib, sys, time, os
41	import numpy as np
42	from transformers import AutoTokenizer
43	
44	POOL = "/workspace/data/pool.jsonl"
45	DEV  = "/workspace/data/multi_dev.npy"
46	OUT  = "/workspace/submission/selection.json"
47	CACHE_DIR = "/tmp/curate_cache"
48	BUDGET = 12_000_000
49	TECH_SHARE = 0.25          # target token share of technical Q&A register
50	D = 1 << 20                # hashed feature dimension
51	WORD_CAP = 1200            # cap words/doc for featurization + stats (bounds cost)
52	ALPHA = 1.0                # NB Laplace smoothing
53	CLIP = 5.0                 # per-feature log-odds clip
54	N_OVERFLOW_IDS = 45_000    # total ids to emit (well over budget for safety)
55	
56	wre = re.compile(r"[a-z0-9']+")
57	STOP = set("the a an and or but if then else of to in on at by for with from as is are was "
58	           "were be been being it its this that these those he she they we you i his her their "
59	           "our your not no do does did has have had will would can could should may might must "
60	           "there here what which who whom whose when where why how than so such about into over "
61	           "after before between out up down off again more most other some any each also".split())
62	NSTAT = 11  # number of per-doc stat columns
63	
64	
65	def doc_features(text):
66	    """Return (unique_feature_idx uint32[], stat_vector float32[NSTAT], fingerprint uint32)."""
67	    low = text.lower()
68	    w = wre.findall(low)
69	    if len(w) > WORD_CAP:
70	        w = w[:WORD_CAP]
71	    nw = len(w)
72	    idx = set()
73	    for tk in w:
74	        idx.add(zlib.crc32(tk.encode()) & (D - 1))
75	    for i in range(nw - 1):
76	        idx.add(zlib.crc32((w[i] + "\x1f" + w[i + 1]).encode()) & (D - 1))
77	
78	    seg = text[:4000]
79	    L = max(1, len(seg))
80	    alpha = sum(c.isalpha() for c in seg)
81	    digit = sum(c.isdigit() for c in seg)
82	    ascii_ = sum(ord(c) < 128 for c in seg)
83	    space = sum(c.isspace() for c in seg)
84	    sym = L - alpha - digit - space
85	    stops = sum(1 for x in w if x in STOP)
86	    uniq = len(set(w))
87	    mwl = (sum(len(x) for x in w) / nw) if nw else 0.0
88	    # line structure: fraction of non-empty lines that are "short" (< 5 words)
89	    lines = [ln for ln in text[:8000].split("\n") if ln.strip()]
90	    if lines:
91	        frac_short = sum(1 for ln in lines if len(ln.split()) < 5) / len(lines)
92	    else:
93	        frac_short = 1.0
94	    has_code = 1.0 if (("<code>" in text) or ("</code>" in text) or ("<pre>" in text)
95	                       or ("&lt;" in text) or ("</" in text) or ("<p>" in text)) else 0.0
96	    stat = np.array([nw, max(1, len(text) // 4), alpha / L, digit / L, ascii_ / L,
97	                     sym / L, (stops / nw) if nw else 0.0, (uniq / nw) if nw else 0.0,
98	                     mwl, frac_short, has_code], dtype=np.float32)
99	    fp = zlib.crc32((" ".join(w[:25] + w[-25:]) + "|" + str(nw)).encode()) & 0xFFFFFFFF
100	    return np.fromiter(idx, dtype=np.uint32, count=len(idx)), stat, np.uint32(fp)
101	
102	
103	def concat_idx(arrs):
104	    lens = np.fromiter((len(a) for a in arrs), dtype=np.int64, count=len(arrs))
105	    indptr = np.zeros(len(arrs) + 1, dtype=np.int64)
106	    np.cumsum(lens, out=indptr[1:])
107	    allidx = np.concatenate(arrs).astype(np.int64) if arrs else np.zeros(0, np.int64)
108	    return allidx, indptr
109	
110	
111	def df_from(allidx):
112	    return np.bincount(allidx, minlength=D).astype(np.float64)
113	
114	
115	def mean_score(r, allidx, indptr):
116	    """Per-doc MEAN of clipped feature weights r (== length-normalized log-lik ratio)."""
117	    if len(allidx) == 0:
118	        return np.zeros(len(indptr) - 1)
119	    vals = np.clip(r[allidx], -CLIP, CLIP)
120	    ssum = np.add.reduceat(vals, indptr[:-1])
121	    cnt = np.diff(indptr).astype(np.float64)
122	    cnt[cnt == 0] = 1.0
123	    return ssum / cnt
124	
125	
126	def nb_logodds(pos_df, bg_df, npos, nbg):
127	    p = (pos_df + ALPHA) / (npos + 2 * ALPHA)
128	    q = (bg_df + ALPHA) / (nbg + 2 * ALPHA)
129	    return np.log(p) - np.log(q)
130	
131	
132	def build_cache():
133	    tok = AutoTokenizer.from_pretrained("gpt2")
134	    eos = tok.eos_token_id
135	    dev = np.load(DEV)
136	    cut = np.where(dev == eos)[0]
137	    starts = np.concatenate([[0], cut + 1]); ends = np.concatenate([cut, [len(dev)]])
138	    pos_prose, pos_tech = [], []
139	    for s, e in zip(starts, ends):
140	        if e - s < 3:
141	            continue
142	        a, st, _ = doc_features(tok.decode(dev[s:e].tolist()))
143	        (pos_tech if st[10] > 0 else pos_prose).append(a)
144	    print(f"    dev positives: prose={len(pos_prose)} tech={len(pos_tech)}", flush=True)
145	    pa, _ = concat_idx(pos_prose); ta, _ = concat_idx(pos_tech)
146	    prose_df = df_from(pa); tech_df = df_from(ta)
147	
148	    ids, arrs, stats, fps, heads = [], [], [], [], []
149	    for line in open(POOL):
150	        r = json.loads(line)
151	        a, st, fp = doc_features(r["text"])
152	        ids.append(r["id"]); arrs.append(a); stats.append(st); fps.append(fp)
153	        heads.append(r["text"][:220])
154	    ids = np.array(ids, dtype=np.int64)
155	    allidx, indptr = concat_idx(arrs)
156	    stats = np.stack(stats); fps = np.array(fps, dtype=np.uint32)
157	    os.makedirs(CACHE_DIR, exist_ok=True)
158	    np.save(f"{CACHE_DIR}/ids.npy", ids)
159	    np.save(f"{CACHE_DIR}/allidx.npy", allidx)
160	    np.save(f"{CACHE_DIR}/indptr.npy", indptr)
161	    np.save(f"{CACHE_DIR}/stats.npy", stats)
162	    np.save(f"{CACHE_DIR}/fps.npy", fps)
163	    np.save(f"{CACHE_DIR}/prose_df.npy", prose_df)
164	    np.save(f"{CACHE_DIR}/tech_df.npy", tech_df)
165	    np.save(f"{CACHE_DIR}/meta.npy", np.array([len(pos_prose), len(pos_tech)], dtype=np.int64))
166	    with open(f"{CACHE_DIR}/heads.json", "w") as f:
167	        json.dump(heads, f)
168	    return ids, allidx, indptr, stats, fps, prose_df, tech_df, len(pos_prose), len(pos_tech), heads
169	
170	
171	def load_cache():
172	    ids = np.load(f"{CACHE_DIR}/ids.npy")
173	    allidx = np.load(f"{CACHE_DIR}/allidx.npy")
174	    indptr = np.load(f"{CACHE_DIR}/indptr.npy")
175	    stats = np.load(f"{CACHE_DIR}/stats.npy")
176	    fps = np.load(f"{CACHE_DIR}/fps.npy")
177	    prose_df = np.load(f"{CACHE_DIR}/prose_df.npy")
178	    tech_df = np.load(f"{CACHE_DIR}/tech_df.npy")
179	    meta = np.load(f"{CACHE_DIR}/meta.npy")
180	    heads = json.load(open(f"{CACHE_DIR}/heads.json"))
181	    return ids, allidx, indptr, stats, fps, prose_df, tech_df, int(meta[0]), int(meta[1]), heads
182	
183	
184	def main():
185	    t0 = time.time()
186	    if os.path.exists(f"{CACHE_DIR}/ids.npy") and "--rebuild" not in sys.argv:
187	        print("[1] loading feature cache", flush=True)
188	        ids, allidx, indptr, stats, fps, prose_df, tech_df, npp, npt, heads = load_cache()
189	    else:
190	        print("[1] featurizing dev positives + pool (one-time)", flush=True)
191	        ids, allidx, indptr, stats, fps, prose_df, tech_df, npp, npt, heads = build_cache()
192	    n = len(ids)
193	    bg_df = df_from(allidx); nbg = n
194	    print(f"    N={n} feats in {time.time()-t0:.0f}s", flush=True)
195	
196	    # single COMBINED head: all dev registers as positives (pool has no code to
197	    # justify a separate technical selector -- see module docstring)
198	    r_all = nb_logodds(prose_df + tech_df, bg_df, npp + npt, nbg)
199	    score = mean_score(r_all, allidx, indptr)
200	
201	    nw, tok_est, alpha, digit, ascii_, sym, stopfrac, uniq, mwl, fshort, has_code = \
202	        [stats[:, i] for i in range(NSTAT)]
203	
204	    # English clean-prose junk filter: drops word-salad (stopword floor), menu/
205	    # boilerplate (short-line + uniq caps), non-English (ascii + stopword),
206	    # markup/data dumps (alpha/sym/digit), repetitive spam (uniq floor).
207	    # Thresholds calibrated against the top-ranked docs: spun word-salad / OCR /
208	    # character-corrupted text clusters at stopfrac<=0.25 and/or mwl>=6.0 and/or
209	    # uniq>=0.80, while genuine news/encyclopedic prose sits at stopfrac>=0.29,
210	    # mwl 4.4-5.6. English function-word density (stopfrac) is also the cleanest
211	    # non-English filter.
212	    ok = ((nw >= 50) & (alpha >= 0.62) & (stopfrac >= 0.26) & (uniq >= 0.30) &
213	          (uniq <= 0.80) & (mwl <= 6.0) & (ascii_ >= 0.92) & (sym <= 0.12) &
214	          (fshort <= 0.55) & (digit <= 0.20))
215	    cand = np.where(ok)[0]
216	    cand = cand[np.argsort(-score[cand])]
217	    print(f"[2] candidates: {len(cand)} pass filter (dropped {n-len(cand)})", flush=True)
218	
219	    # emit best-first with content-fingerprint dedup, up to N_OVERFLOW_IDS
220	    seen_fp, sel, acc = set(), [], 0
221	    for j in cand:
222	        fp = int(fps[j])
223	        if fp in seen_fp:
224	            continue
225	        seen_fp.add(fp)
226	        sel.append(j); acc += tok_est[j]
227	        if len(sel) >= N_OVERFLOW_IDS:
228	            break
229	    print(f"    selected {len(sel)} docs (~{acc/1e6:.0f}M est tokens, best-first)", flush=True)
230	
231	    out_ids = [int(ids[j]) for j in sel]
232	    assert len(out_ids) == len(set(out_ids)), "duplicate ids"
233	    json.dump(out_ids, open(OUT, "w"))
234	    est = sum(int(tok_est[j]) for j in sel)
235	    print(f"[3] wrote {len(out_ids)} ids -> {OUT} (~{est/1e6:.0f}M est tokens) in {time.time()-t0:.0f}s",
236	          flush=True)
237	
238	    if "--peek" in sys.argv:
239	        sel_arr = np.array(sel)
240	        for label, rng in [("TOP (rank 0-6)", sel_arr[:7]),
241	                           ("rank ~4000", sel_arr[4000:4004]),
242	                           ("rank ~10000 (near 12M cutoff)", sel_arr[10000:10004])]:
243	            print(f"\n===== {label} =====")
244	            for j in rng:
245	                print(f"[s{score[j]:.2f} nw{int(nw[j])} sw{stopfrac[j]:.2f} uq{uniq[j]:.2f} "
246	                      f"fs{fshort[j]:.2f}] {heads[j]!r}")
247	
248	
249	if __name__ == "__main__":
250	    main()
251

[metadata] {"type": "text", "file": {"filePath": "/workspace/curate.py", "content": "#!/usr/bin/env python3\n\"\"\"\nCurate a raw web pool for pretraining a small LM whose eval target is a BROAD,\nmulti-domain high-quality English mixture (equal parts encyclopedic/Wikipedia,\ngeneral high-quality web prose, news, technical Q&A).\n\nSTATED CRITERION\n----------------\nKeep pool documents that are (a) clean, fluent, information-dense English and\n(b) distributionally close to the disclosed target register mix.\n\nTarget-closeness is a DSIR / NBSVM-style log-likelihood ratio. For hashed word\nunigram+bigram features we weight each feature f by\n    r[f] = log P(f | target register) - log P(f | pool background)\nestimated from feature document-frequencies (Laplace-smoothed). A document's score\nis the MEAN over its features of clip(r[f], -5, 5). Using the mean (not the sum)\nremoves length bias, and clipping stops a handful of ultra-rare \"fancy\" words from\nletting SEO word-salad win -- the two dominant failure modes of a raw sum.\n\nThe target is a fixed mixture including ~25% technical Q&A (identifiable by HTML/code\nmarkup in the decoded dev set). We AUDITED the pool and found it contains essentially\nNO code / programming / StackOverflow content (the `<code>`/`&lt;` markers are absent\nand \"code-like\" punctuation counts trace back to ordinary prose). So the code surface\nof the Q&A register is unmatchable from this pool, and an explicit \"technical\" selector\nonly drags in multilingual boilerplate and SEO word-salad (empirically its top hits) --\nwhich would displace good prose AND fail to teach code, hurting perplexity both ways.\n\nWe therefore use ONE combined head whose positives are ALL decoded dev segments\n(encyclopedic + news + web + the Q&A questions). Folding the Q&A positives in keeps\ntheir *technical vocabulary* (function, array, query, server, ...) in the log-odds, so\ngenuinely technical PROSE in the pool is still up-ranked -- the best available proxy for\nthe Q&A register when no code exists to select. Each pool doc must pass an English,\nclean-prose junk filter (drops word-salad, menu/boilerplate, non-English, markup dumps,\nrepetitive spam), is deduplicated by a content fingerprint, and is ranked by the mean\nclipped log-odds. Docs are emitted best-first; the packer fills 12M tokens from the top.\n\nPositives come only from the disclosed dev target; the selection is over disjoint pool\nids. The output is an ordered id list produced entirely by this script.\n\"\"\"\nimport json, re, zlib, sys, time, os\nimport numpy as np\nfrom transformers import AutoTokenizer\n\nPOOL = \"/workspace/data/pool.jsonl\"\nDEV  = \"/workspace/data/multi_dev.npy\"\nOUT  = \"/workspace/submission/selection.json\"\nCACHE_DIR = \"/tmp/curate_cache\"\nBUDGET = 12_000_000\nTECH_SHARE = 0.25          # target token share of technical Q&A register\nD = 1 << 20                # hashed feature dimension\nWORD_CAP = 1200            # cap words/doc for featurization + stats (bounds cost)\nALPHA = 1.0                # NB Laplace smoothing\nCLIP = 5.0                 # per-feature log-odds clip\nN_OVERFLOW_IDS = 45_000    # total ids to emit (well over budget for safety)\n\nwre = re.compile(r\"[a-z0-9']+\")\nSTOP = set(\"the a an and or but if then else of to in on at by for with from as is are was \"\n           \"were be been being it its this that these those he she they we you i his her their \"\n           \"our your not no do does did has have had will would can could should may might must \"\n           \"there here what which who whom whose when where why how than so such about into over \"\n           \"after before between out up down off again more most other some any each also\".split())\nNSTAT = 11  # number of per-doc stat columns\n\n\ndef doc_features(text):\n    \"\"\"Return (unique_feature_idx uint32[], stat_vector float32[NSTAT], fingerprint uint32).\"\"\"\n    low = text.lower()\n    w = wre.findall(low)\n    if len(w) > WORD_CAP:\n        w = w[:WORD_CAP]\n    nw = len(w)\n    idx = set()\n    for tk in w:\n        idx.add(zlib.crc32(tk.encode()) & (D - 1))\n    for i in range(nw - 1):\n        idx.add(zlib.crc32((w[i] + \"\\x1f\" + w[i + 1]).encode()) & (D - 1))\n\n    seg = text[:4000]\n    L = max(1, len(seg))\n    alpha = sum(c.isalpha() for c in seg)\n    digit = sum(c.isdigit() for c in seg)\n    ascii_ = sum(ord(c) < 128 for c in seg)\n    space = sum(c.isspace() for c in seg)\n    sym = L - alpha - digit - space\n    stops = sum(1 for x in w if x in STOP)\n    uniq = len(set(w))\n    mwl = (sum(len(x) for x in w) / nw) if nw else 0.0\n    # line structure: fraction of non-empty lines that are \"short\" (< 5 words)\n    lines = [ln for ln in text[:8000].split(\"\\n\") if ln.strip()]\n    if lines:\n        frac_short = sum(1 for ln in lines if len(ln.split()) < 5) / len(lines)\n    else:\n        frac_short = 1.0\n    has_code = 1.0 if ((\"<code>\" in text) or (\"</code>\" in text) or (\"<pre>\" in text)\n                       or (\"&lt;\" in text) or (\"</\" in text) or (\"<p>\" in text)) else 0.0\n    stat = np.array([nw, max(1, len(text) // 4), alpha / L, digit / L, ascii_ / L,\n                     sym / L, (stops / nw) if nw else 0.0, (uniq / nw) if nw else 0.0,\n                     mwl, frac_short, has_code], dtype=np.float32)\n    fp = zlib.crc32((\" \".join(w[:25] + w[-25:]) + \"|\" + str(nw)).encode()) & 0xFFFFFFFF\n    return np.fromiter(idx, dtype=np.uint32, count=len(idx)), stat, np.uint32(fp)\n\n\ndef concat_idx(arrs):\n    lens = np.fromiter((len(a) for a in arrs), dtype=np.int64, count=len(arrs))\n    indptr = np.zeros(len(arrs) + 1, dtype=np.int64)\n    np.cumsum(lens, out=indptr[1:])\n    allidx = np.concatenate(arrs).astype(np.int64) if arrs else np.zeros(0, np.int64)\n    return allidx, indptr\n\n\ndef df_from(allidx):\n    return np.bincount(allidx, minlength=D).astype(np.float64)\n\n\ndef mean_score(r, allidx, indptr):\n    \"\"\"Per-doc MEAN of clipped feature weights r (== length-normalized log-lik ratio).\"\"\"\n    if len(allidx) == 0:\n        return np.zeros(len(indptr) - 1)\n    vals = np.clip(r[allidx], -CLIP, CLIP)\n    ssum = np.add.reduceat(vals, indptr[:-1])\n    cnt = np.diff(indptr).astype(np.float64)\n    cnt[cnt == 0] = 1.0\n    return ssum / cnt\n\n\ndef nb_logodds(pos_df, bg_df, npos, nbg):\n    p = (pos_df + ALPHA) / (npos + 2 * ALPHA)\n    q = (bg_df + ALPHA) / (nbg + 2 * ALPHA)\n    return np.log(p) - np.log(q)\n\n\ndef build_cache():\n    tok = AutoTokenizer.from_pretrained(\"gpt2\")\n    eos = tok.eos_token_id\n    dev = np.load(DEV)\n    cut = np.where(dev == eos)[0]\n    starts = np.concatenate([[0], cut + 1]); ends = np.concatenate([cut, [len(dev)]])\n    pos_prose, pos_tech = [], []\n    for s, e in zip(starts, ends):\n        if e - s < 3:\n            continue\n        a, st, _ = doc_features(tok.decode(dev[s:e].tolist()))\n        (pos_tech if st[10] > 0 else pos_prose).append(a)\n    print(f\"    dev positives: prose={len(pos_prose)} tech={len(pos_tech)}\", flush=True)\n    pa, _ = concat_idx(pos_prose); ta, _ = concat_idx(pos_tech)\n    prose_df = df_from(pa); tech_df = df_from(ta)\n\n    ids, arrs, stats, fps, heads = [], [], [], [], []\n    for line in open(POOL):\n        r = json.loads(line)\n        a, st, fp = doc_features(r[\"text\"])\n        ids.append(r[\"id\"]); arrs.append(a); stats.append(st); fps.append(fp)\n        heads.append(r[\"text\"][:220])\n    ids = np.array(ids, dtype=np.int64)\n    allidx, indptr = concat_idx(arrs)\n    stats = np.stack(stats); fps = np.array(fps, dtype=np.uint32)\n    os.makedirs(CACHE_DIR, exist_ok=True)\n    np.save(f\"{CACHE_DIR}/ids.npy\", ids)\n    np.save(f\"{CACHE_DIR}/allidx.npy\", allidx)\n    np.save(f\"{CACHE_DIR}/indptr.npy\", indptr)\n    np.save(f\"{CACHE_DIR}/stats.npy\", stats)\n    np.save(f\"{CACHE_DIR}/fps.npy\", fps)\n    np.save(f\"{CACHE_DIR}/prose_df.npy\", prose_df)\n    np.save(f\"{CACHE_DIR}/tech_df.npy\", tech_df)\n    np.save(f\"{CACHE_DIR}/meta.npy\", np.array([len(pos_prose), len(pos_tech)], dtype=np.int64))\n    with open(f\"{CACHE_DIR}/heads.json\", \"w\") as f:\n        json.dump(heads, f)\n    return ids, allidx, indptr, stats, fps, prose_df, tech_df, len(pos_prose), len(pos_tech), heads\n\n\ndef load_cache():\n    ids = np.load(f\"{CACHE_DIR}/ids.npy\")\n    allidx = np.load(f\"{CACHE_DIR}/allidx.npy\")\n    indptr = np.load(f\"{CACHE_DIR}/indptr.npy\")\n    stats = np.load(f\"{CACHE_DIR}/stats.npy\")\n    fps = np.load(f\"{CACHE_DIR}/fps.npy\")\n    prose_df = np.load(f\"{CACHE_DIR}/prose_df.npy\")\n    tech_df = np.load(f\"{CACHE_DIR}/tech_df.npy\")\n    meta = np.load(f\"{CACHE_DIR}/meta.npy\")\n    heads = json.load(open(f\"{CACHE_DIR}/heads.json\"))\n    return ids, allidx, indptr, stats, fps, prose_df, tech_df, int(meta[0]), int(meta[1]), heads\n\n\ndef main():\n    t0 = time.time()\n    if os.path.exists(f\"{CACHE_DIR}/ids.npy\") and \"--rebuild\" not in sys.argv:\n        print(\"[1] loading feature cache\", flush=True)\n        ids, allidx, indptr, stats, fps, prose_df, tech_df, npp, npt, heads = load_cache()\n    else:\n        print(\"[1] featurizing dev positives + pool (one-time)\", flush=True)\n        ids, allidx, indptr, stats, fps, prose_df, tech_df, npp, npt, heads = build_cache()\n    n = len(ids)\n    bg_df = df_from(allidx); nbg = n\n    print(f\"    N={n} feats in {time.time()-t0:.0f}s\", flush=True)\n\n    # single COMBINED head: all dev registers as positives (pool has no code to\n    # justify a separate technical selector -- see module docstring)\n    r_all = nb_logodds(prose_df + tech_df, bg_df, npp + npt, nbg)\n    score = mean_score(r_all, allidx, indptr)\n\n    nw, tok_est, alpha, digit, ascii_, sym, stopfrac, uniq, mwl, fshort, has_code = \\\n        [stats[:, i] for i in range(NSTAT)]\n\n    # English clean-prose junk filter: drops word-salad (stopword floor), menu/\n    # boilerplate (short-line + uniq caps), non-English (ascii + stopword),\n    # markup/data dumps (alpha/sym/digit), repetitive spam (uniq floor).\n    # Thresholds calibrated against the top-ranked docs: spun word-salad / OCR /\n    # character-corrupted text clusters at stopfrac<=0.25 and/or mwl>=6.0 and/or\n    # uniq>=0.80, while genuine news/encyclopedic prose sits at stopfrac>=0.29,\n    # mwl 4.4-5.6. English function-word density (stopfrac) is also the cleanest\n    # non-English filter.\n    ok = ((nw >= 50) & (alpha >= 0.62) & (stopfrac >= 0.26) & (uniq >= 0.30) &\n          (uniq <= 0.80) & (mwl <= 6.0) & (ascii_ >= 0.92) & (sym <= 0.12) &\n          (fshort <= 0.55) & (digit <= 0.20))\n    cand = np.where(ok)[0]\n    cand = cand[np.argsort(-score[cand])]\n    print(f\"[2] candidates: {len(cand)} pass filter (dropped {n-len(cand)})\", flush=True)\n\n    # emit best-first with content-fingerprint dedup, up to N_OVERFLOW_IDS\n    seen_fp, sel, acc = set(), [], 0\n    for j in cand:\n        fp = int(fps[j])\n        if fp in seen_fp:\n            continue\n        seen_fp.add(fp)\n        sel.append(j); acc += tok_est[j]\n        if len(sel) >= N_OVERFLOW_IDS:\n            break\n    print(f\"    selected {len(sel)} docs (~{acc/1e6:.0f}M est tokens, best-first)\", flush=True)\n\n    out_ids = [int(ids[j]) for j in sel]\n    assert len(out_ids) == len(set(out_ids)), \"duplicate ids\"\n    json.dump(out_ids, open(OUT, \"w\"))\n    est = sum(int(tok_est[j]) for j in sel)\n    print(f\"[3] wrote {len(out_ids)} ids -> {OUT} (~{est/1e6:.0f}M est tokens) in {time.time()-t0:.0f}s\",\n          flush=True)\n\n    if \"--peek\" in sys.argv:\n        sel_arr = np.array(sel)\n        for label, rng in [(\"TOP (rank 0-6)\", sel_arr[:7]),\n                           (\"rank ~4000\", sel_arr[4000:4004]),\n                           (\"rank ~10000 (near 12M cutoff)\", sel_arr[10000:10004])]:\n            print(f\"\\n===== {label} =====\")\n            for j in rng:\n                print(f\"[s{score[j]:.2f} nw{int(nw[j])} sw{stopfrac[j]:.2f} uq{uniq[j]:.2f} \"\n                      f\"fs{fshort[j]:.2f}] {heads[j]!r}\")\n\n\nif __name__ == \"__main__\":\n    main()\n", "numLines": 251, "startLine": 1, "totalLines": 251}}

File created successfully at: /tmp/flu_calib.py (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/tmp/flu_calib.py", "content": "import json, re, time, numpy as np\nfrom transformers import AutoTokenizer\n\nwre = re.compile(r\"[a-z0-9']+\")\ntok = AutoTokenizer.from_pretrained(\"gpt2\"); eos = tok.eos_token_id\ndev = np.load(\"/workspace/data/multi_dev.npy\")\ncut = np.where(dev == eos)[0]\nstarts = np.concatenate([[0], cut + 1]); ends = np.concatenate([cut, [len(dev)]])\n\n# build word-bigram model from dev\nt0=time.time()\nuni = {}; big = {}\nfor s,e in zip(starts,ends):\n    if e-s < 3: continue\n    w = wre.findall(tok.decode(dev[s:e].tolist()))\n    for x in w: uni[x]=uni.get(x,0)+1\n    for i in range(len(w)-1):\n        k=(w[i],w[i+1]); big[k]=big.get(k,0)+1\nV=len(uni); Nuni=sum(uni.values())\nprint(f\"dev model: {V} types, {len(big)} bigrams, {Nuni} tokens in {time.time()-t0:.0f}s\")\n\nK=0.1\ndef fluency(text, cap=400):\n    w = wre.findall(text.lower())[:cap]\n    if len(w)<2: return -20.0, 1.0\n    lp=0.0; oov=0\n    for i in range(len(w)-1):\n        c1=uni.get(w[i],0)\n        cb=big.get((w[i],w[i+1]),0)\n        lp += np.log((cb+K)/(c1+K*V))\n        if uni.get(w[i],0)==0: oov+=1\n    if uni.get(w[-1],0)==0: oov+=1\n    return lp/(len(w)-1), oov/len(w)\n\n# load pool text\ntext={}\nfor line in open(\"/workspace/data/pool.jsonl\"):\n    r=json.loads(line); text[r[\"id\"]]=r[\"text\"]\n\ngood=[58452,82269,3475,9401,37064,2027,81859,76107]  # news+wiki\nbad=[38752,98806]  # spun / OCR gibberish\nprint(\"\\n-- GOOD (news/wiki) --\")\nfor i in good:\n    f,o=fluency(text[i]); print(f\" id{i:6d} flu{f:7.3f} oov{o:.2f} | {text[i][:70]!r}\")\nprint(\"-- BAD (gibberish) --\")\nfor i in bad:\n    f,o=fluency(text[i]); print(f\" id{i:6d} flu{f:7.3f} oov{o:.2f} | {text[i][:70]!r}\")\n\n# distribution over a random sample of pool\nrng=np.random.default_rng(0)\nsamp=rng.choice(list(text.keys()),4000,replace=False)\nfl=np.array([fluency(text[int(i)])[0] for i in samp])\nov=np.array([fluency(text[int(i)])[1] for i in samp])\nprint(f\"\\npool sample flu: p05={np.percentile(fl,5):.2f} p25={np.percentile(fl,25):.2f} \"\n      f\"p50={np.percentile(fl,50):.2f} p75={np.percentile(fl,75):.2f}\")\nprint(f\"pool sample oov: p50={np.percentile(ov,50):.2f} p75={np.percentile(ov,75):.2f} \"\n      f\"p90={np.percentile(ov,90):.2f} p95={np.percentile(ov,95):.2f}\")\n", "structuredPatch": [], "originalFile": null, "userModified": false}

dev model: 40303 types, 334539 bigrams, 683911 tokens in 1s

-- GOOD (news/wiki) --
 id 58452 flu -8.969 oov0.19 | '<|endoftext|>New Delhi: Five new Supreme Court judges were today sworn'
 id 82269 flu -8.812 oov0.10 | 'Hill 303 massacre\n|Hill 303 massacre|\nBodies of massacre victims gathe'
 id  3475 flu -8.859 oov0.16 | 'Anglo-Dutch Wars, also called Dutch Wars, Dutch Engelse Oorlogen, four'
 id  9401 flu -9.604 oov0.43 | 'Sacred-Texts Christianity Index Previous Next\nOF THE MESSIANIC GENERAT'
 id 37064 flu -8.978 oov0.19 | 'The Bharatiya Janata Party (BJP) leaders of the West Bengal unit will '
 id  2027 flu -9.081 oov0.16 | 'From Wikipedia, the free encyclopedia\n|February 24, 1882 - December 16'
 id 81859 flu -8.994 oov0.18 | '|Rediff India Abroad Home | All the sections|\nBihar: Vigilante justice'
 id 76107 flu -8.573 oov0.12 | 'Major Bhupinder Singh\nIn sept. 1965, Major Bhupinder Singh commanded t'
-- BAD (gibberish) --
 id 38752 flu -9.821 oov0.37 | '<|endoftext|>Colossi are used to rip cracker datum and agouti. yahoo. '
 id 98806 flu-10.555 oov0.42 | ' hesiall, Loo. Micks. The ness arou shis. I therrying ot! Mmmend bou. '

pool sample flu: p05=-10.35 p25=-9.68 p50=-9.09 p75=-8.66
pool sample oov: p50=0.11 p75=0.16 p90=0.23 p95=0.30

[stdout]
dev model: 40303 types, 334539 bigrams, 683911 tokens in 1s

-- GOOD (news/wiki) --
 id 58452 flu -8.969 oov0.19 | '<|endoftext|>New Delhi: Five new Supreme Court judges were today sworn'
 id 82269 flu -8.812 oov0.10 | 'Hill 303 massacre\n|Hill 303 massacre|\nBodies of massacre victims gathe'
 id  3475 flu -8.859 oov0.16 | 'Anglo-Dutch Wars, also called Dutch Wars, Dutch Engelse Oorlogen, four'
 id  9401 flu -9.604 oov0.43 | 'Sacred-Texts Christianity Index Previous Next\nOF THE MESSIANIC GENERAT'
 id 37064 flu -8.978 oov0.19 | 'The Bharatiya Janata Party (BJP) leaders of the West Bengal unit will '
 id  2027 flu -9.081 oov0.16 | 'From Wikipedia, the free encyclopedia\n|February 24, 1882 - December 16'
 id 81859 flu -8.994 oov0.18 | '|Rediff India Abroad Home | All the sections|\nBihar: Vigilante justice'
 id 76107 flu -8.573 oov0.12 | 'Major Bhupinder Singh\nIn sept. 1965, Major Bhupinder Singh commanded t'
-- BAD (gibberish) --
 id 38752 flu -9.821 oov0.37 | '<|endoftext|>Colossi are used to rip cracker datum and agouti. yahoo. '
 id 98806 flu-10.555 oov0.42 | ' hesiall, Loo. Micks. The ness arou shis. I therrying ot! Mmmend bou. '

pool sample flu: p05=-10.35 p25=-9.68 p50=-9.09 p75=-8.66
pool sample oov: p50=0.11 p75=0.16 p90=0.23 p95=0.30
[metadata] {"noOutputExpected": false}

The file /workspace/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/curate.py", "oldString": "CLIP = 5.0                 # per-feature log-odds clip\nN_OVERFLOW_IDS = 45_000    # total ids to emit (well over budget for safety)", "newString": "CLIP = 5.0                 # per-feature log-odds clip\nN_OVERFLOW_IDS = 45_000    # total ids to emit (well over budget for safety)\nBG_K = 0.1                 # add-k smoothing for the dev word-bigram fluency LM\nFLU_MIN = -9.85            # per-doc mean bigram log-prob floor (demotion, not drop)\nOOV_MAX = 0.34             # per-doc out-of-dev-vocab word fraction ceiling", "originalFile": null, "structuredPatch": [{"oldStart": 52, "oldLines": 6, "newStart": 52, "newLines": 9, "lines": [" ALPHA = 1.0                # NB Laplace smoothing", " CLIP = 5.0                 # per-feature log-odds clip", " N_OVERFLOW_IDS = 45_000    # total ids to emit (well over budget for safety)", "+BG_K = 0.1                 # add-k smoothing for the dev word-bigram fluency LM", "+FLU_MIN = -9.85            # per-doc mean bigram log-prob floor (demotion, not drop)", "+OOV_MAX = 0.34             # per-doc out-of-dev-vocab word fraction ceiling", " ", " wre = re.compile(r\"[a-z0-9']+\")", " STOP = set(\"the a an and or but if then else of to in on at by for with from as is are was \""]}], "userModified": false, "replaceAll": false}

The file /workspace/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/curate.py", "oldString": "def nb_logodds(pos_df, bg_df, npos, nbg):\n    p = (pos_df + ALPHA) / (npos + 2 * ALPHA)\n    q = (bg_df + ALPHA) / (nbg + 2 * ALPHA)\n    return np.log(p) - np.log(q)", "newString": "def nb_logodds(pos_df, bg_df, npos, nbg):\n    p = (pos_df + ALPHA) / (npos + 2 * ALPHA)\n    q = (bg_df + ALPHA) / (nbg + 2 * ALPHA)\n    return np.log(p) - np.log(q)\n\n\ndef dev_segments(tok):\n    \"\"\"Decode the disclosed dev target into a list of text segments.\"\"\"\n    dev = np.load(DEV); eos = tok.eos_token_id\n    cut = np.where(dev == eos)[0]\n    starts = np.concatenate([[0], cut + 1]); ends = np.concatenate([cut, [len(dev)]])\n    return [tok.decode(dev[s:e].tolist()) for s, e in zip(starts, ends) if e - s >= 3]\n\n\ndef build_fluency_lm(segs):\n    \"\"\"Word unigram/bigram counts over the clean dev text -> a smoothed bigram LM.\n    Fluency = mean_i log P(w_i|w_{i-1}); OCR corruption and 'spun' word-salad both\n    have systematically improbable word transitions and score low. This is a pure\n    English-fluency signal (independent of topic) distilled from the target itself.\"\"\"\n    uni, big = {}, {}\n    for s in segs:\n        w = wre.findall(s.lower())\n        for x in w:\n            uni[x] = uni.get(x, 0) + 1\n        for i in range(len(w) - 1):\n            k = (w[i], w[i + 1]); big[k] = big.get(k, 0) + 1\n    return uni, big, len(uni)\n\n\ndef fluency(text, uni, big, V, cap=400):\n    \"\"\"Return (mean bigram log-prob, out-of-vocab word fraction) for `text`.\"\"\"\n    w = wre.findall(text.lower())[:cap]\n    if len(w) < 2:\n        return -20.0, 1.0\n    lp = 0.0; oov = 0\n    for i in range(len(w) - 1):\n        c1 = uni.get(w[i], 0)\n        cb = big.get((w[i], w[i + 1]), 0)\n        lp += np.log((cb + BG_K) / (c1 + BG_K * V))\n        if c1 == 0:\n            oov += 1\n    if uni.get(w[-1], 0) == 0:\n        oov += 1\n    return lp / (len(w) - 1), oov / len(w)", "originalFile": null, "structuredPatch": [{"oldStart": 132, "oldLines": 6, "newStart": 132, "newLines": 46, "lines": ["     return np.log(p) - np.log(q)", " ", " ", "+def dev_segments(tok):", "+    \"\"\"Decode the disclosed dev target into a list of text segments.\"\"\"", "+    dev = np.load(DEV); eos = tok.eos_token_id", "+    cut = np.where(dev == eos)[0]", "+    starts = np.concatenate([[0], cut + 1]); ends = np.concatenate([cut, [len(dev)]])", "+    return [tok.decode(dev[s:e].tolist()) for s, e in zip(starts, ends) if e - s >= 3]", "+", "+", "+def build_fluency_lm(segs):", "+    \"\"\"Word unigram/bigram counts over the clean dev text -> a smoothed bigram LM.", "+    Fluency = mean_i log P(w_i|w_{i-1}); OCR corruption and 'spun' word-salad both", "+    have systematically improbable word transitions and score low. This is a pure", "+    English-fluency signal (independent of topic) distilled from the target itself.\"\"\"", "+    uni, big = {}, {}", "+    for s in segs:", "+        w = wre.findall(s.lower())", "+        for x in w:", "+            uni[x] = uni.get(x, 0) + 1", "+        for i in range(len(w) - 1):", "+            k = (w[i], w[i + 1]); big[k] = big.get(k, 0) + 1", "+    return uni, big, len(uni)", "+", "+", "+def fluency(text, uni, big, V, cap=400):", "+    \"\"\"Return (mean bigram log-prob, out-of-vocab word fraction) for `text`.\"\"\"", "+    w = wre.findall(text.lower())[:cap]", "+    if len(w) < 2:", "+        return -20.0, 1.0", "+    lp = 0.0; oov = 0", "+    for i in range(len(w) - 1):", "+        c1 = uni.get(w[i], 0)", "+        cb = big.get((w[i], w[i + 1]), 0)", "+        lp += np.log((cb + BG_K) / (c1 + BG_K * V))", "+        if c1 == 0:", "+            oov += 1", "+    if uni.get(w[-1], 0) == 0:", "+        oov += 1", "+    return lp / (len(w) - 1), oov / len(w)", "+", "+", " def build_cache():", "     tok = AutoTokenizer.from_pretrained(\"gpt2\")", "     eos = tok.eos_token_id"]}], "userModified": false, "replaceAll": false}

The file /workspace/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/curate.py", "oldString": "    cand = np.where(ok)[0]\n    cand = cand[np.argsort(-score[cand])]\n    print(f\"[2] candidates: {len(cand)} pass filter (dropped {n-len(cand)})\", flush=True)\n\n    # emit best-first with content-fingerprint dedup, up to N_OVERFLOW_IDS\n    seen_fp, sel, acc = set(), [], 0\n    for j in cand:\n        fp = int(fps[j])\n        if fp in seen_fp:\n            continue\n        seen_fp.add(fp)\n        sel.append(j); acc += tok_est[j]\n        if len(sel) >= N_OVERFLOW_IDS:\n            break\n    print(f\"    selected {len(sel)} docs (~{acc/1e6:.0f}M est tokens, best-first)\", flush=True)", "newString": "    cand = np.where(ok)[0]\n    cand = cand[np.argsort(-score[cand])]\n    print(f\"[2] candidates: {len(cand)} pass filter (dropped {n-len(cand)})\", flush=True)\n\n    # Fluency DEMOTION (not deletion): among the top candidates that could enter the\n    # 12M window, compute the dev-derived word-bigram fluency + OOV fraction and push\n    # dysfluent docs (OCR corruption, 'spun' real-word word-salad, non-English) to the\n    # BACK of the order. They stay as overflow, so a noisy threshold can never discard\n    # good prose -- it only reorders who fills the budget first. Cheap: scored only for\n    # the leading M candidates (>> enough clean docs to fill 12M several times over).\n    M = min(len(cand), 3 * N_OVERFLOW_IDS // 3, 30000)\n    tok_hf = AutoTokenizer.from_pretrained(\"gpt2\")\n    uni, big, V = build_fluency_lm(dev_segments(tok_hf))\n    id2text = {}\n    for line in open(POOL):\n        r = json.loads(line); id2text[r[\"id\"]] = r[\"text\"]\n    clean = np.ones(len(cand), dtype=bool)\n    n_marg = 0\n    for rank in range(M):\n        j = cand[rank]\n        flu, oov = fluency(id2text[int(ids[j])], uni, big, V)\n        if flu < FLU_MIN or oov > OOV_MAX:\n            clean[rank] = False; n_marg += 1\n    order = np.concatenate([cand[:M][clean[:M]], cand[M:], cand[:M][~clean[:M]]])\n    print(f\"[2b] fluency-demoted {n_marg}/{M} leading candidates to overflow\", flush=True)\n\n    # emit best-first with content-fingerprint dedup, up to N_OVERFLOW_IDS\n    seen_fp, sel, acc = set(), [], 0\n    for j in order:\n        fp = int(fps[j])\n        if fp in seen_fp:\n            continue\n        seen_fp.add(fp)\n        sel.append(j); acc += tok_est[j]\n        if len(sel) >= N_OVERFLOW_IDS:\n            break\n    print(f\"    selected {len(sel)} docs (~{acc/1e6:.0f}M est tokens, best-first)\", flush=True)", "originalFile": null, "structuredPatch": [{"oldStart": 259, "oldLines": 9, "newStart": 259, "newLines": 31, "lines": ["     cand = cand[np.argsort(-score[cand])]", "     print(f\"[2] candidates: {len(cand)} pass filter (dropped {n-len(cand)})\", flush=True)", " ", "+    # Fluency DEMOTION (not deletion): among the top candidates that could enter the", "+    # 12M window, compute the dev-derived word-bigram fluency + OOV fraction and push", "+    # dysfluent docs (OCR corruption, 'spun' real-word word-salad, non-English) to the", "+    # BACK of the order. They stay as overflow, so a noisy threshold can never discard", "+    # good prose -- it only reorders who fills the budget first. Cheap: scored only for", "+    # the leading M candidates (>> enough clean docs to fill 12M several times over).", "+    M = min(len(cand), 3 * N_OVERFLOW_IDS // 3, 30000)", "+    tok_hf = AutoTokenizer.from_pretrained(\"gpt2\")", "+    uni, big, V = build_fluency_lm(dev_segments(tok_hf))", "+    id2text = {}", "+    for line in open(POOL):", "+        r = json.loads(line); id2text[r[\"id\"]] = r[\"text\"]", "+    clean = np.ones(len(cand), dtype=bool)", "+    n_marg = 0", "+    for rank in range(M):", "+        j = cand[rank]", "+        flu, oov = fluency(id2text[int(ids[j])], uni, big, V)", "+        if flu < FLU_MIN or oov > OOV_MAX:", "+            clean[rank] = False; n_marg += 1", "+    order = np.concatenate([cand[:M][clean[:M]], cand[M:], cand[:M][~clean[:M]]])", "+    print(f\"[2b] fluency-demoted {n_marg}/{M} leading candidates to overflow\", flush=True)", "+", "     # emit best-first with content-fingerprint dedup, up to N_OVERFLOW_IDS", "     seen_fp, sel, acc = set(), [], 0", "-    for j in cand:", "+    for j in order:", "         fp = int(fps[j])", "         if fp in seen_fp:", "             continue"]}], "userModified": false, "replaceAll": false}

The file /workspace/curate.py has been updated successfully. (file state is current in your context — no need to Read it back)

[metadata] {"filePath": "/workspace/curate.py", "oldString": "    M = min(len(cand), 3 * N_OVERFLOW_IDS // 3, 30000)\n    tok_hf = AutoTokenizer.from_pretrained(\"gpt2\")", "newString": "    M = min(len(cand), 30000)\n    tok_hf = AutoTokenizer.from_pretrained(\"gpt2\")", "originalFile": null, "structuredPatch": [{"oldStart": 265, "oldLines": 7, "newStart": 265, "newLines": 7, "lines": ["     # BACK of the order. They stay as overflow, so a noisy threshold can never discard", "     # good prose -- it only reorders who fills the budget first. Cheap: scored only for", "     # the leading M candidates (>> enough clean docs to fill 12M several times over).", "-    M = min(len(cand), 3 * N_OVERFLOW_IDS // 3, 30000)", "+    M = min(len(cand), 30000)", "     tok_hf = AutoTokenizer.from_pretrained(\"gpt2\")", "     uni, big, V = build_fluency_lm(dev_segments(tok_hf))", "     id2text = {}"]}], "userModified": false, "replaceAll": false}

[1] loading feature cache
    N=182016 feats in 1s
[2] candidates: 107423 pass filter (dropped 74593)
[2b] fluency-demoted 114/30000 leading candidates to overflow
    selected 45000 docs (~52M est tokens, best-first)
[3] wrote 45000 ids -> /workspace/submission/selection.json (~52M est tokens) in 24s

===== TOP (rank 0-6) =====
[s0.29 nw209 sw0.34 uq0.42 fs0.00] '<|endoftext|>New Delhi: Five new Supreme Court judges were today sworn in, taking the top court’s strength to 28, including the Chief Justice of India. Justices Sanjay Kishan Kaul, Navin Sinha, Mohan M Shantanagoudar, De'
[s0.29 nw1200 sw0.35 uq0.37 fs0.36] 'Hill 303 massacre\n|Hill 303 massacre|\nBodies of massacre victims gathered near Waegwan, South Korea, many with their hands still bound\n|Location||Hill 303, Waegwan, South Korea|\n|Date||August 17, 1950\n|Target||U.S. Army '
[s0.27 nw96 sw0.45 uq0.54 fs0.00] 'ggor Domm II (375BDW-51ADW) is widely considered to be the greatest ruler in the history of the Nethereigons. He reigned the Northern Dividend of the Nethereigons from 310BDW to 7ADW, and then ruled the United Nethereigo'
[s0.24 nw170 sw0.36 uq0.62 fs0.00] 'The Bharatiya Janata Party (BJP) leaders of the West Bengal unit will meet Union Home Minister Rajnath Singh on Thursday to lodge protest against the vandalising of its party headquarters office in the state by the Trina'
[s0.23 nw528 sw0.30 uq0.43 fs0.08] 'Amarnath Yatra 2017\nAmarnath Yatra 2017 News\nJammu and Kashmir police said three people, alleged conspirators in the 10 July attack on Amarnath pilgrims, have been arrested by its SIT\nAs many as 813 pilgrims on Thursday '
[s0.23 nw231 sw0.34 uq0.56 fs0.00] 'oor ul HaqBaramulla, Aug 10:\nArmy on Friday said the five militants killed in upper reaches of Rafiabad during two-day gunfight were all foreigners and belonged to Lashkar-e-Toiba.\nAddressing a press conference at Pohrup'
[s0.23 nw212 sw0.36 uq0.54 fs0.12] "New Delhi | ANI: Yes Bank promoter Rana Kapoor informed the Enforcement Directorate (ED) that he was 'forced' by a then Congress Union Minister to buy an MF Hussain painting from Priyanka Gandhi Vadra for Rs 2 crores.\nAs"

===== rank ~4000 =====
[s-0.07 nw232 sw0.36 uq0.57 fs0.00] 'PR want to sign promising forward Paul Smyth from Linfield.\nThey previously made an approach for him and have maintained their interest, with a deal possible ahead of the transfer window closing.\nGet West London have sin'
[s-0.07 nw273 sw0.33 uq0.66 fs0.00] 'Video: Small Cruise Ship Sunk as Artificial Reef Off Delaware\nOn Monday, a contractor for the Delaware State Division of Fish and Wildlife sank the 200-foot cruise ship American Glory about 15 nautical miles off Indian R'
[s-0.07 nw387 sw0.34 uq0.57 fs0.25] "mals\nA Portland-based supergroup of sorts, the Thermals originally featured Kind of Like Spitting's Ben Barnett and the Operacycle's Jordan Hudson, plus Hutch Harris and Kathy Foster of the twee/folk-pop duo Hutch & Kath"
[s-0.07 nw98 sw0.30 uq0.71 fs0.00] '<|endoftext|>San Jose De Apartado, Colombia\nTO MATCH FEATURE RIGHTS COLOMBIA.\nA Colombian boy displaced by violence sways his brother on a hammock in San Jose de Apartado, Antioquia province on April 20, 2005. Except for'

===== rank ~10000 (near 12M cutoff) =====
[s-0.14 nw1200 sw0.47 uq0.47 fs0.01] ')<|endoftext|>Any C-Span viewer of recent Potomac cavortings has the evidence of his or\nher eyes to know how rankly foul our school systems, public and private,\nmust be. Seldom has there been a more discouraging parade o'
[s-0.14 nw238 sw0.32 uq0.65 fs0.00] '<|endoftext|>[CLJ-804] Expose memoization cache and original as metadata on memoized functions. Created: 01/Jun/11 Updated: 21/Jun/11 Resolved: 21/Jun/11\nThe attached patch exposes the memoization cache and original pre-'
[s-0.14 nw325 sw0.35 uq0.62 fs0.00] ' Peterson 1949 California Point Break surfboard fetched the highest price during an auction of vintage boards and memorabilia that raised funds to support surfing and surf culture.\nThe board, which sold for $32,400, was '
[s-0.14 nw222 sw0.37 uq0.65 fs0.00] 'NATION-STATE TRAITS AND HOW THEY AFFECT CORPORATE CULTURES IN SEVEN COUNTRIES\nDifferent cultures have diverse concepts of leadership. Leaders can be born, elected, or trained and groomed. Others seize power or have leade'

[stdout]
[1] loading feature cache
    N=182016 feats in 1s
[2] candidates: 107423 pass filter (dropped 74593)
[2b] fluency-demoted 114/30000 leading candidates to overflow
    selected 45000 docs (~52M est tokens, best-first)
[3] wrote 45000 ids -> /workspace/submission/selection.json (~52M est tokens) in 24s

===== TOP (rank 0-6) =====
[s0.29 nw209 sw0.34 uq0.42 fs0.00] '<|endoftext|>New Delhi: Five new Supreme Court judges were today sworn in, taking the top court’s strength to 28, including the Chief Justice of India. Justices Sanjay Kishan Kaul, Navin Sinha, Mohan M Shantanagoudar, De'
[s0.29 nw1200 sw0.35 uq0.37 fs0.36] 'Hill 303 massacre\n|Hill 303 massacre|\nBodies of massacre victims gathered near Waegwan, South Korea, many with their hands still bound\n|Location||Hill 303, Waegwan, South Korea|\n|Date||August 17, 1950\n|Target||U.S. Army '
[s0.27 nw96 sw0.45 uq0.54 fs0.00] 'ggor Domm II (375BDW-51ADW) is widely considered to be the greatest ruler in the history of the Nethereigons. He reigned the Northern Dividend of the Nethereigons from 310BDW to 7ADW, and then ruled the United Nethereigo'
[s0.24 nw170 sw0.36 uq0.62 fs0.00] 'The Bharatiya Janata Party (BJP) leaders of the West Bengal unit will meet Union Home Minister Rajnath Singh on Thursday to lodge protest against the vandalising of its party headquarters office in the state by the Trina'
[s0.23 nw528 sw0.30 uq0.43 fs0.08] 'Amarnath Yatra 2017\nAmarnath Yatra 2017 News\nJammu and Kashmir police said three people, alleged conspirators in the 10 July attack on Amarnath pilgrims, have been arrested by its SIT\nAs many as 813 pilgrims on Thursday '
[s0.23 nw231 sw0.34 uq0.56 fs0.00] 'oor ul HaqBaramulla, Aug 10:\nArmy on Friday said the five militants killed in upper reaches of Rafiabad during two-day gunfight were all foreigners and belonged to Lashkar-e-Toiba.\nAddressing a press conference at Pohrup'
[s0.23 nw212 sw0.36 uq0.54 fs0.12] "New Delhi | ANI: Yes Bank promoter Rana Kapoor informed the Enforcement Directorate (ED) that he was 'forced' by a then Congress Union Minister to buy an MF Hussain painting from Priyanka Gandhi Vadra for Rs 2 crores.\nAs"

===== rank ~4000 =====
[s-0.07 nw232 sw0.36 uq0.57 fs0.00] 'PR want to sign promising forward Paul Smyth from Linfield.\nThey previously made an approach for him and have maintained their interest, with a deal possible ahead of the transfer window closing.\nGet West London have sin'
[s-0.07 nw273 sw0.33 uq0.66 fs0.00] 'Video: Small Cruise Ship Sunk as Artificial Reef Off Delaware\nOn Monday, a contractor for the Delaware State Division of Fish and Wildlife sank the 200-foot cruise ship American Glory about 15 nautical miles off Indian R'
[s-0.07 nw387 sw0.34 uq0.57 fs0.25] "mals\nA Portland-based supergroup of sorts, the Thermals originally featured Kind of Like Spitting's Ben Barnett and the Operacycle's Jordan Hudson, plus Hutch Harris and Kathy Foster of the twee/folk-pop duo Hutch & Kath"
[s-0.07 nw98 sw0.30 uq0.71 fs0.00] '<|endoftext|>San Jose De Apartado, Colombia\nTO MATCH FEATURE RIGHTS COLOMBIA.\nA Colombian boy displaced by violence sways his brother on a hammock in San Jose de Apartado, Antioquia province on April 20, 2005. Except for'

===== rank ~10000 (near 12M cutoff) =====
[s-0.14 nw1200 sw0.47 uq0.47 fs0.01] ')<|endoftext|>Any C-Span viewer of recent Potomac cavortings has the evidence of his or\nher eyes to know how rankly foul our school systems, public and private,\nmust be. Seldom has there been a more discouraging parade o'
[s-0.14 nw238 sw0.32 uq0.65 fs0.00] '<|endoftext|>[CLJ-804] Expose memoization cache and original as metadata on memoized functions. Created: 01/Jun/11 Updated: 21/Jun/11 Resolved: 21/Jun/11\nThe attached patch exposes the memoization cache and original pre-'
[s-0.14 nw325 sw0.35 uq0.62 fs0.00] ' Peterson 1949 California Point Break surfboard fetched the highest price during an auction of vintage boards and memorabilia that raised funds to support surfing and surf culture.\nThe board, which sold for $32,400, was '
[s-0.14 nw222 sw0.37 uq0.65 fs0.00] 'NATION-STATE TRAITS AND HOW THEY AFFECT CORPORATE CULTURES IN SEVEN COUNTRIES\nDifferent cultures have diverse concepts of leadership. Leaders can be born, elected, or trained and groomed. Others seize power or have leade'
[metadata] {"noOutputExpected": false}

launched eval pid 3531

[stdout]
launched eval pid 3531
[metadata] {"noOutputExpected": false}

File created successfully at: /workspace/claim.md (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/workspace/claim.md", "content": "# Claim: register-matched quality selection beats random under a fixed token budget\n\n## Hypothesis\n\nUnder a fixed **12M-token** budget (~6% of the 182k-doc pool), selecting documents by\n\n1. a **DSIR / NBSVM log-likelihood-ratio** to the disclosed target register mix\n   (`r[f] = log P(f | target) − log P(f | pool background)`, scored as the *mean*\n   of clipped per-feature weights), plus\n2. an **English clean-prose + word-bigram fluency filter**,\n\nproduces substantially lower held-out perplexity than a random 6% sample — and it does so\nby concentrating the budget on the registers the pool can actually supply\n(**encyclopedic, news, general web prose**) while leaving the register the pool *cannot*\nsupply (**technical Q&A / code**) essentially unmodeled, because this pool contains no code.\n\nMeasured so far: random-sample dev PPL = **476.7**; curated dev PPL = **384.965** (≈ **19% lower**).\n\n## Mechanism — a prediction that is NOT the final perplexity\n\nCuration moves the *training token distribution* toward the target's, **but only for the\nregisters the pool contains**. That yields two observables, both upstream of / orthogonal to\nthe single final-PPL number:\n\n- **(O1) Data-level cross-entropy, before any model is trained.** The unigram token\n  cross-entropy `H(dev-slice ‖ training-set)` should drop sharply from random→curated for\n  the *prose* slices of the dev target, but stay ≈ flat for the *code/Q&A* slice — because\n  curated and random pool samples are equally devoid of code tokens.\n  *Measured (add-1-smoothed unigram CE, nats/token):*\n  `PROSE: random {PROSE_RND} → curated {PROSE_CUR}` ;\n  `CODE : random {CODE_RND} → curated {CODE_CUR}`.\n\n- **(O2) The trained model's gain is register-localized.** If dev perplexity is broken out\n  by register, the curated-vs-random improvement is large on the encyclopedic/news/web\n  slices and ≈ 1× (no gain) on the code-bearing Q&A slice. The model does not learn code it\n  never saw; the entire win is in prose.\n\nThe point of O1/O2 is that the aggregate PPL win is *predicted to be structured*, not uniform.\nA generic \"the data is just cleaner\" story predicts a uniform lift; this mechanism predicts a\nlift that is **concentrated in exactly the matchable registers and absent in the code register**.\n\n## Falsification\n\nThe hypothesis is wrong if any of the following hold:\n\n- A random 6% sample reaches within ~2% of the curated PPL ⇒ the criterion adds nothing.\n- The curated improvement is **uniform across registers**, in particular a large gain on the\n  code/Q&A slice ⇒ the \"no code in pool ⇒ that register is unmatchable\" mechanism is false.\n- Re-injecting the fluency-demoted word-salad / OCR / non-English docs at the top of the\n  selection does **not** raise PPL ⇒ the clean-prose/fluency component is not causal.\n- Replacing the real target positives with a **shuffled/random** \"target\" gives the same PPL\n  ⇒ the win is generic quality filtering, not *target register matching*.\n\n## Transfer\n\nThe criterion is target-, budget-, and pool-agnostic in its method; only one *empirical* finding\nis local to this pool.\n\n- **New target:** swap the disclosed dev sample for any other register mix and the same code\n  selects for *that* distribution — the log-odds head re-estimates `P(f | target)` from whatever\n  positives it is handed. Nothing is hand-keyed to Wikipedia/news.\n- **New budget:** the output is a strict priority ranking. A smaller budget consumes a cleaner\n  prefix; a larger budget appends lower-log-odds (more generic) prose. The method therefore\n  predicts **diminishing returns as budget → pool size** — curation matters most when budget ≪ pool.\n- **New pool:** the junk + fluency filters are distilled from the target, not fitted to this\n  pool's spam templates, so they transfer to other raw web dumps. The single **non-transferable**\n  fact is empirical, not methodological: *this* pool has no code, so the Q&A register is\n  uncoverable here. A pool containing real StackOverflow/code would be matched by the identical\n  combined-head criterion with no code change.\n", "structuredPatch": [], "originalFile": null, "userModified": false}

packed 12000000 tokens from 45000 ids -> /tmp/dev_train.npy
HELDOUT_PPL 385.933  loss 5.9557  train_s 54  /tmp/dev_train.npy
DONE 0
---timer---
2916

[stdout]
packed 12000000 tokens from 45000 ids -> /tmp/dev_train.npy
HELDOUT_PPL 385.933  loss 5.9557  train_s 54  /tmp/dev_train.npy
DONE 0
---timer---
2916
[metadata] {"noOutputExpected": false}

File created successfully at: /tmp/compo.py (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/tmp/compo.py", "content": "import json, re, numpy as np\n\nsel = json.load(open(\"/workspace/submission/selection.json\"))\ntext = {}\nfor line in open(\"/workspace/data/pool.jsonl\"):\n    r = json.loads(line); text[r[\"id\"]] = r[\"text\"]\n\ndef label(t):\n    if any(m in t for m in (\"<code>\", \"</code>\", \"<pre>\", \"&lt;\", \"</\", \"<p>\")):\n        return \"code\"\n    if any(m in t for m in (\" @,@ \", \" @.@ \", \" @-@ \", \"= =\", \"From Wikipedia\", \"<unk>\", \"||\")):\n        return \"wiki\"\n    # news: dateline near the top e.g. \"New Delhi:\", \"London (Reuters)\", \"Aug 10:\"\n    head = t[:120]\n    if re.search(r\"(Reuters|\\bANI\\b|\\bIANS\\b|\\bAP\\b|\\bAFP\\b|\\bPTI\\b)\", head) or \\\n       re.search(r\"^[\\\"'<|]*[A-Z][A-Za-z .]+[:,]\\s*[A-Z]\", head) or \\\n       re.search(r\"[A-Z][a-z]+ \\d{1,2}[:,]\", head):\n        return \"news\"\n    return \"web\"\n\n# approx token counts via len//4 (packer uses real gpt2 but this is fine for shares)\ntot = 0; comp = {}\nndoc = {}\nBUDGET = 12_000_000\nfor i in sel:\n    t = text[i]\n    est = max(1, len(t) // 4)\n    lab = label(t)\n    comp[lab] = comp.get(lab, 0) + est\n    ndoc[lab] = ndoc.get(lab, 0) + 1\n    tot += est\n    if tot >= BUDGET:\n        break\nprint(f\"trained window ~{tot/1e6:.1f}M est tokens across {sum(ndoc.values())} docs\")\nfor k in sorted(comp, key=lambda x: -comp[x]):\n    print(f\"  {k:5s}: {comp[k]/tot*100:5.1f}% tokens   {ndoc[k]:5d} docs\")\n", "structuredPatch": [], "originalFile": null, "userModified": false}

trained window ~12.0M est tokens across 8780 docs
  web  :  77.4% tokens    7061 docs
  news :  11.7% tokens    1304 docs
  wiki :  10.0% tokens     391 docs
  code :   1.0% tokens      24 docs

[stdout]
trained window ~12.0M est tokens across 8780 docs
  web  :  77.4% tokens    7061 docs
  news :  11.7% tokens    1304 docs
  wiki :  10.0% tokens     391 docs
  code :   1.0% tokens      24 docs
[metadata] {"noOutputExpected": false}