claude-opus-5

launched

[stdout]
launched
[metadata] {"noOutputExpected": false}

File created successfully at: /workspace/peek.py (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/workspace/peek.py", "content": "\"\"\"Eyeball the top-ranked documents per register (sanity check on the scorer).\"\"\"\nimport json, numpy as np, sys\nz = np.load(\"/workspace/cache/scores.npz\")\nids, S = z[\"ids\"], z[\"scores\"]\nodds = S[:, :4] - S[:, 4:5]\nassign, best = odds.argmax(1), odds.max(1)\nREG = [\"wiki\", \"web\", \"news\", \"qa\"]\nsel = set(json.load(open(\"/tmp/sel_v1.json\"))[:26000])\nwant = {}\nfor ci in range(4):\n    sub = np.flatnonzero((assign == ci) & np.isin(ids, list(sel)))\n    sub = sub[np.argsort(-best[sub])][:4]\n    for i in sub: want[int(ids[i])] = (REG[ci], float(best[i]))\n# worst-ranked kept docs too\ntail = json.load(open(\"/tmp/sel_v1.json\"))[-4:]\nfor t in tail: want[t] = (\"TAIL\", 0.0)\nout = {}\nfor line in open(\"/workspace/data/pool.jsonl\"):\n    r = json.loads(line)\n    if r[\"id\"] in want: out[r[\"id\"]] = r[\"text\"]\nfor i, (reg, sc) in want.items():\n    print(f\"\\n===== id={i} {reg} logodds={sc:.2f}\\n{out[i][:520]!r}\")\n", "structuredPatch": [], "originalFile": null, "userModified": false}

===== id=33023 wiki logodds=3.37
'Paleo-Indians migrated from Eurasia to what is now the U.S. mainland at least 15,000 years ago, with European colonization beginning in the 16th century. The United States emerged from 13 British colonies along the East Coast. Disputes between Great Britain and the colonies led to the American Revolution. On July 4, 1776, as the colonies were fighting Great Britain in the American Revolutionary War, delegates from the 13 colonies unanimously adopted the Declaration of Independence. The war ended in 1783 with recogn'

===== id=3475 wiki logodds=2.84
'Anglo-Dutch Wars, also called Dutch Wars, Dutch Engelse Oorlogen, four 17th- and 18th-century naval conflicts between England and the Dutch Republic. The first three wars, stemming from commercial rivalry, established England’s naval might, and the last, arising from Dutch interference in the American Revolution, spelled the end of the republic’s position as a world power.\nThe First Anglo-Dutch War (1652–54) began during a tense period following England’s institution of the 1651 Navigation Act, which was aimed at b'

===== id=32517 wiki logodds=2.61
'miral, was born on 2 July 1862 at Hartforth in Yorkshire, the fourth son of Christopher Cradock and his wife Georgina, the daughter of Major Gordon Duff of the 92nd Highlanders. He entered the navy in 1875 and three years later as midshipman of the Pallas took part in the British occupation of Cyprus. In 1884, as a sub-lieutenant, he landed with the naval brigade for garrison duties in Upper Egypt and as first lieutenant of the Dolphin served with the Eastern Sudan field force where he was chosen by the governor-ge'

===== id=13783 wiki logodds=2.45
'United States military aircraft serial numbers\nIn the United States, all military aircraft display a serial number to identify individual aircraft. These numbers are located on the aircraft tail, so they are sometimes referred to unofficially as "tail numbers". On the Northrop Grumman B-2 Spirit bomber, lacking a tail, the number appears on the nose gear door. Individual agencies have each evolved their own system of serial number identification. Aircraft serials are part of the Aircraft Visual Identification Syste'

===== id=45958 web logodds=1.64
' investigations into Russian interference in the presidential election continue, the revelation that Donald Trump Jr. and other senior members of the Trump campaign met with a Kremlin-affiliated lawyer last year has plagued the White House in recent weeks.\nHere’s how the story from Trump Jr., White House officials, and the president’s lawyer has repeatedly shifted as further details of the meeting have emerged.\nThe (Shifting) Initial Explanations\nIn his initial July 8 statement to the New York Times, which first br'

===== id=78375 web logodds=1.56
"<|endoftext|>Interlude—Speculations, enjoyment, distractions: Did the events described below really occur?\nLawrence seems to thunk they didn't. The events are described right at the start of a New York Times news report:\nSCHMIDT AND HABERMAN (8/2/18): President Trump pushed his lawyers in recent days to try once again to reach an agreement with the special counsel’s office about his sitting for an interview, flouting their advice that he should not answer investigators’ questions, three people briefed on the matter"

===== id=80709 web logodds=1.50
'Israeli Prime Minister Benjamin Netanyahu has vowed to erect a new barrier on the country’s border with Syria in a bid to prevent jihadists fighting against Bashar al-Assad’s regime from entering Israel.\nIsrael’s Prime Minister Benjamin Netanyahu has pledged to erect a new border fence along the border with Syria on the occupied Golan Heights amid concern over the growing influence of jihadist fighters among Syrian rebels.\n“We know that on the other side of our border with Syria today, the Syrian army has moved awa'

===== id=23875 web logodds=1.41
'Don Trump Jr. Is "Miserable," According To Report\nHaving your family in the White House might seem cool, but Donald Trump Jr. is apparently not exactly enjoying life as a member of the first family of the United States.\nAt least, that\'s just about the only conclusion one could draw from a new report by People, which cited sources close to the Trump family.\nAccording to the report, one of the sources strongly implied that Trump Jr. can\'t wait for his father\'s tenure in office to be over.\nThe source was quoted as say'

===== id=47959 news logodds=3.19
"LUCKNOW, India (Reuters) - Thousands of youngsters in India have burned down empty train coaches and blocked rail traffic this week in protest against what they call irregularities in recruitment by the mammoth railways department, one of the world's largest employers.\nStudent associations have called for more protests this week mainly in the eastern state of Bihar, which has been reporting one of the highest jobless rates in the country. India's unemployment is estimated to have exceeded the global rate in five of"

===== id=100806 news logodds=2.95
' Market<|endoftext|>Manipur officials to study exodus situation\nThe Manipur government will send senior officials to Karnataka and Maharashtra to look into the exodus of North East people from these states in the wake of the recent Assam violence.\nThe decision was taken during a Cabinet meeting presided over by Chief Minister O Ibobi Singh late last night, an official spokesman said on Saturday, adding the government took "seriously: the reported attack on Manipuris in Karnataka and Maharashtra.\nThe official spokes'

===== id=103973 news logodds=2.94
'pur: Prime Minister Manmohan Singh on Friday kicked off the Congress campaign from Kanpur on Friday. Addressing the crowds, he said, "We have sent funds under various national schemes to Uttar Pradesh. Rs 24000 crore have been given to Uttar Pradesh government for roads alone. However, none of it has trickled down to the common man."\nHe added, "The Congress will win UP elections. Sonia Gandhi and Rahul Gandhi will ensure Congress\' victory in UP."\nPriyanka Gandhi, meanwhile, campaigned in Raebareli, where she touche'

===== id=44805 news logodds=2.88
"New Delhi, Feb 20: Bracing for a stinging opposition attack on 2G issue during budget session of Parliament, the Prime Minister's Office has asked the telecom department to give full details on the corruption cases.\nIn a note to DoT, the PMO has asked for the report of the Comptroller and Auditor General including corruption allegations against former minister Dayanidhi Maran. It asked the department to give inputs on important cases pending before the Supreme Court regarding 2G spectrum allocation and draft teleco"

===== id=67614 qa logodds=2.08
'i have a servlet that is using the request objects getParameter() method to retrieve a string. The string contains a double quote (") and is therefore truncated somewhere between the client and the servlet. Is there anyway to retrieve the value in it\'s entirety without truncating? the value is (3/16"):ABSBT. Am only getting (3/16\nThis needs to be handled when the parameter is being posted, not when being retrieved. If the request parameter is posted as part of a form, all would be well. So I suspect that somewhere '

===== id=53152 qa logodds=2.00
' is a small tool I wrote that will scan a selected folder for all the .sql files and generate a MS-DOS batch script that will execute the scripts on a selected server/database.\nIn my work, I often find myself writing an assortment of stored procedures, user defined functions and views that are executed on the development server. I usually save all my SQL scripts in text files with a .sql extension. After development these files have to be deployed to the production server. I have found the osql utility that comes i'

===== id=111815 qa logodds=1.70
' a user-created object in the current database. This object can be a table, index, column, alias data type, or Microsoft .NET Framework common language runtime (CLR) user-defined type.\nChanging any part of an object name can break scripts and stored procedures. We recommend you do not use this statement to rename stored procedures, triggers, user-defined functions, or views; instead, drop the object and re-create it with the new name.\nYou can change the name of an object or data type in the current database only. T'

===== id=13090 qa logodds=1.34
"Process the data in delimited files more easily than with string parsing\n| Executive Summary:|\nAlthough you can use a TextStream object to open a plain-text file, it's not advantageous because you can only read the file from the beginning. However, you can use ADO, which enables you to use the Microsoft Jet OLE DB text driver to parse the contents of a delimited file, to read delimited files. The Jet OLE DB text driver uses the registry to determine the format of the delimited file.\nReading delimited files is a com"

===== id=18247 TAIL logodds=0.00
"Return to The Binaural Source home page; Go to Next Page; For a Text-Based Index to entire catalog\nUnlike the many ordinary stereo nature sound recordings currently available, these binaural environments put YOU right in the meadow, forest or jungle rather than just making pleasant background sounds. If you hear a toad, that toad is not just a part of the sound mix, it's on a rock four feet to your right and two feet high. Lie back, relax, close your eyes and travel binaurally for some very effective stress reducti"

===== id=25097 TAIL logodds=0.00
'<|endoftext|>Taranaki is one of the country’s largest and strongest netball playing provinces.\nUnlike most other centres in which play is focused at one set of courts, in Taranaki we have 6 satellites: Hawera, Opunake, Stratford, Inglewood, Waitara and New Plymouth. Our Premier teams play in stadium at TSB Hub in Hawera, TET Multi Sports Centre in Stratford, Sandfords Event Centre in Opunake, New Plymouth YMCA and TSB Stadium in New Plymouth.\nFirst established in 1947 as The Taranaki Basketball Association Incorpor'

===== id=103130 TAIL logodds=0.00
'vicevideo...<|endoftext|>Hurricane season is here once again, but it is not very late to get ready and be prepared. Regardless if you live in a place where hurricanes are often a normal and expected to happen, or you have not really experienced it at all, it would be best to always be prepared and get ready in advance.\nWith all the changes happening on a global scale – industries on the rise, pollution growing each day, abrupt weather disturbances experienced all over the world – all of these only contributes to hu'

===== id=96794 TAIL logodds=0.00
" projects.<|endoftext|>Change is an inevitable fact for the business world. Without it, there cannot be any progress or appropriate reaction to shifts in demand.\nHowever, the success of major structural or strategy changes that affects a variety of stakeholders often depends on the way it's conveyed. As such, experienced communication professionals can often find themselves in a position where they are responsible for this.\nThe effects of not communicating effectively with all involved stakeholders can seriously im"

[stdout]
===== id=33023 wiki logodds=3.37
'Paleo-Indians migrated from Eurasia to what is now the U.S. mainland at least 15,000 years ago, with European colonization beginning in the 16th century. The United States emerged from 13 British colonies along the East Coast. Disputes between Great Britain and the colonies led to the American Revolution. On July 4, 1776, as the colonies were fighting Great Britain in the American Revolutionary War, delegates from the 13 colonies unanimously adopted the Declaration of Independence. The war ended in 1783 with recogn'

===== id=3475 wiki logodds=2.84
'Anglo-Dutch Wars, also called Dutch Wars, Dutch Engelse Oorlogen, four 17th- and 18th-century naval conflicts between England and the Dutch Republic. The first three wars, stemming from commercial rivalry, established England’s naval might, and the last, arising from Dutch interference in the American Revolution, spelled the end of the republic’s position as a world power.\nThe First Anglo-Dutch War (1652–54) began during a tense period following England’s institution of the 1651 Navigation Act, which was aimed at b'

===== id=32517 wiki logodds=2.61
'miral, was born on 2 July 1862 at Hartforth in Yorkshire, the fourth son of Christopher Cradock and his wife Georgina, the daughter of Major Gordon Duff of the 92nd Highlanders. He entered the navy in 1875 and three years later as midshipman of the Pallas took part in the British occupation of Cyprus. In 1884, as a sub-lieutenant, he landed with the naval brigade for garrison duties in Upper Egypt and as first lieutenant of the Dolphin served with the Eastern Sudan field force where he was chosen by the governor-ge'

===== id=13783 wiki logodds=2.45
'United States military aircraft serial numbers\nIn the United States, all military aircraft display a serial number to identify individual aircraft. These numbers are located on the aircraft tail, so they are sometimes referred to unofficially as "tail numbers". On the Northrop Grumman B-2 Spirit bomber, lacking a tail, the number appears on the nose gear door. Individual agencies have each evolved their own system of serial number identification. Aircraft serials are part of the Aircraft Visual Identification Syste'

===== id=45958 web logodds=1.64
' investigations into Russian interference in the presidential election continue, the revelation that Donald Trump Jr. and other senior members of the Trump campaign met with a Kremlin-affiliated lawyer last year has plagued the White House in recent weeks.\nHere’s how the story from Trump Jr., White House officials, and the president’s lawyer has repeatedly shifted as further details of the meeting have emerged.\nThe (Shifting) Initial Explanations\nIn his initial July 8 statement to the New York Times, which first br'

===== id=78375 web logodds=1.56
"<|endoftext|>Interlude—Speculations, enjoyment, distractions: Did the events described below really occur?\nLawrence seems to thunk they didn't. The events are described right at the start of a New York Times news report:\nSCHMIDT AND HABERMAN (8/2/18): President Trump pushed his lawyers in recent days to try once again to reach an agreement with the special counsel’s office about his sitting for an interview, flouting their advice that he should not answer investigators’ questions, three people briefed on the matter"

===== id=80709 web logodds=1.50
'Israeli Prime Minister Benjamin Netanyahu has vowed to erect a new barrier on the country’s border with Syria in a bid to prevent jihadists fighting against Bashar al-Assad’s regime from entering Israel.\nIsrael’s Prime Minister Benjamin Netanyahu has pledged to erect a new border fence along the border with Syria on the occupied Golan Heights amid concern over the growing influence of jihadist fighters among Syrian rebels.\n“We know that on the other side of our border with Syria today, the Syrian army has moved awa'

===== id=23875 web logodds=1.41
'Don Trump Jr. Is "Miserable," According To Report\nHaving your family in the White House might seem cool, but Donald Trump Jr. is apparently not exactly enjoying life as a member of the first family of the United States.\nAt least, that\'s just about the only conclusion one could draw from a new report by People, which cited sources close to the Trump family.\nAccording to the report, one of the sources strongly implied that Trump Jr. can\'t wait for his father\'s tenure in office to be over.\nThe source was quoted as say'

===== id=47959 news logodds=3.19
"LUCKNOW, India (Reuters) - Thousands of youngsters in India have burned down empty train coaches and blocked rail traffic this week in protest against what they call irregularities in recruitment by the mammoth railways department, one of the world's largest employers.\nStudent associations have called for more protests this week mainly in the eastern state of Bihar, which has been reporting one of the highest jobless rates in the country. India's unemployment is estimated to have exceeded the global rate in five of"

===== id=100806 news logodds=2.95
' Market<|endoftext|>Manipur officials to study exodus situation\nThe Manipur government will send senior officials to Karnataka and Maharashtra to look into the exodus of North East people from these states in the wake of the recent Assam violence.\nThe decision was taken during a Cabinet meeting presided over by Chief Minister O Ibobi Singh late last night, an official spokesman said on Saturday, adding the government took "seriously: the reported attack on Manipuris in Karnataka and Maharashtra.\nThe official spokes'

===== id=103973 news logodds=2.94
'pur: Prime Minister Manmohan Singh on Friday kicked off the Congress campaign from Kanpur on Friday. Addressing the crowds, he said, "We have sent funds under various national schemes to Uttar Pradesh. Rs 24000 crore have been given to Uttar Pradesh government for roads alone. However, none of it has trickled down to the common man."\nHe added, "The Congress will win UP elections. Sonia Gandhi and Rahul Gandhi will ensure Congress\' victory in UP."\nPriyanka Gandhi, meanwhile, campaigned in Raebareli, where she touche'

===== id=44805 news logodds=2.88
"New Delhi, Feb 20: Bracing for a stinging opposition attack on 2G issue during budget session of Parliament, the Prime Minister's Office has asked the telecom department to give full details on the corruption cases.\nIn a note to DoT, the PMO has asked for the report of the Comptroller and Auditor General including corruption allegations against former minister Dayanidhi Maran. It asked the department to give inputs on important cases pending before the Supreme Court regarding 2G spectrum allocation and draft teleco"

===== id=67614 qa logodds=2.08
'i have a servlet that is using the request objects getParameter() method to retrieve a string. The string contains a double quote (") and is therefore truncated somewhere between the client and the servlet. Is there anyway to retrieve the value in it\'s entirety without truncating? the value is (3/16"):ABSBT. Am only getting (3/16\nThis needs to be handled when the parameter is being posted, not when being retrieved. If the request parameter is posted as part of a form, all would be well. So I suspect that somewhere '

===== id=53152 qa logodds=2.00
' is a small tool I wrote that will scan a selected folder for all the .sql files and generate a MS-DOS batch script that will execute the scripts on a selected server/database.\nIn my work, I often find myself writing an assortment of stored procedures, user defined functions and views that are executed on the development server. I usually save all my SQL scripts in text files with a .sql extension. After development these files have to be deployed to the production server. I have found the osql utility that comes i'

===== id=111815 qa logodds=1.70
' a user-created object in the current database. This object can be a table, index, column, alias data type, or Microsoft .NET Framework common language runtime (CLR) user-defined type.\nChanging any part of an object name can break scripts and stored procedures. We recommend you do not use this statement to rename stored procedures, triggers, user-defined functions, or views; instead, drop the object and re-create it with the new name.\nYou can change the name of an object or data type in the current database only. T'

===== id=13090 qa logodds=1.34
"Process the data in delimited files more easily than with string parsing\n| Executive Summary:|\nAlthough you can use a TextStream object to open a plain-text file, it's not advantageous because you can only read the file from the beginning. However, you can use ADO, which enables you to use the Microsoft Jet OLE DB text driver to parse the contents of a delimited file, to read delimited files. The Jet OLE DB text driver uses the registry to determine the format of the delimited file.\nReading delimited files is a com"

===== id=18247 TAIL logodds=0.00
"Return to The Binaural Source home page; Go to Next Page; For a Text-Based Index to entire catalog\nUnlike the many ordinary stereo nature sound recordings currently available, these binaural environments put YOU right in the meadow, forest or jungle rather than just making pleasant background sounds. If you hear a toad, that toad is not just a part of the sound mix, it's on a rock four feet to your right and two feet high. Lie back, relax, close your eyes and travel binaurally for some very effective stress reducti"

===== id=25097 TAIL logodds=0.00
'<|endoftext|>Taranaki is one of the country’s largest and strongest netball playing provinces.\nUnlike most other centres in which play is focused at one set of courts, in Taranaki we have 6 satellites: Hawera, Opunake, Stratford, Inglewood, Waitara and New Plymouth. Our Premier teams play in stadium at TSB Hub in Hawera, TET Multi Sports Centre in Stratford, Sandfords Event Centre in Opunake, New Plymouth YMCA and TSB Stadium in New Plymouth.\nFirst established in 1947 as The Taranaki Basketball Association Incorpor'

===== id=103130 TAIL logodds=0.00
'vicevideo...<|endoftext|>Hurricane season is here once again, but it is not very late to get ready and be prepared. Regardless if you live in a place where hurricanes are often a normal and expected to happen, or you have not really experienced it at all, it would be best to always be prepared and get ready in advance.\nWith all the changes happening on a global scale – industries on the rise, pollution growing each day, abrupt weather disturbances experienced all over the world – all of these only contributes to hu'

===== id=96794 TAIL logodds=0.00
" projects.<|endoftext|>Change is an inevitable fact for the business world. Without it, there cannot be any progress or appropriate reaction to shifts in demand.\nHowever, the success of major structural or strategy changes that affects a variety of stakeholders often depends on the way it's conveyed. As such, experienced communication professionals can often find themselves in a position where they are responsible for this.\nThe effects of not communicating effectively with all involved stakeholders can seriously im"
[metadata] {"noOutputExpected": false}

packed 11991695 tokens from 12678 tokenized docs -> /tmp/v1_train.npy
WARNING: under budget!
4579

[stdout]
packed 11991695 tokens from 12678 tokenized docs -> /tmp/v1_train.npy
WARNING: under budget!
4579
[metadata] {"noOutputExpected": false}

HELDOUT_PPL 449.482  loss 6.1081  train_s 56  /tmp/v1_train.npy

[stdout]
HELDOUT_PPL 449.482  loss 6.1081  train_s 56  /tmp/v1_train.npy
[metadata] {"noOutputExpected": false}

selected 26434 docs ~26.4M est tokens, dup-dropped 102
register token mix: [1. 0. 0. 0.]
wrote /tmp/sel_pure.json
selected 36306 docs ~18.4M est tokens, dup-dropped 91
register token mix: [0.22  0.339 0.397 0.044]
wrote /tmp/sel_short.json
selected 27022 docs ~26.4M est tokens, dup-dropped 141
register token mix: [0.377 0.208 0.358 0.056]
wrote /tmp/sel_wq.json

[stdout]
selected 26434 docs ~26.4M est tokens, dup-dropped 102
register token mix: [1. 0. 0. 0.]
wrote /tmp/sel_pure.json
selected 36306 docs ~18.4M est tokens, dup-dropped 91
register token mix: [0.22  0.339 0.397 0.044]
wrote /tmp/sel_short.json
selected 27022 docs ~26.4M est tokens, dup-dropped 141
register token mix: [0.377 0.208 0.358 0.056]
wrote /tmp/sel_wq.json
[metadata] {"noOutputExpected": false}

File created successfully at: /workspace/pack_multi.py (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/workspace/pack_multi.py", "content": "\"\"\"Pack several selections in one pass over the pool (dev-eval convenience).\"\"\"\nimport json, sys, numpy as np, os\nfrom multiprocessing import Pool\nfrom transformers import AutoTokenizer\nos.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\"\nBUDGET = 12_000_000\njobs = [(p, \"/tmp/\" + os.path.basename(p).replace(\".json\", \"\") + \"_train.npy\") for p in sys.argv[1:]]\nsels = {p: json.load(open(p)) for p, _ in jobs}\nwant = set().union(*[set(s) for s in sels.values()])\ntext = {}\nfor line in open(\"/workspace/data/pool.jsonl\"):\n    r = json.loads(line)\n    if r[\"id\"] in want: text[r[\"id\"]] = r[\"text\"]\n\n_tok = None\ndef enc(t):\n    global _tok\n    if _tok is None: _tok = AutoTokenizer.from_pretrained(\"gpt2\")\n    return _tok(t, add_special_tokens=False).input_ids\n\npool = Pool(14)\nfor p, out in jobs:\n    sel, acc, order = sels[p], 0, []\n    for i in sel:\n        order.append(i); acc += len(text[i])\n        if acc > BUDGET * 5.4: break\n    encoded = pool.map(enc, [text[i] for i in order], chunksize=8)\n    parts, tot = [], 0\n    for ids in encoded:\n        parts.append(np.array(ids + [50256], dtype=np.uint16)); tot += len(ids) + 1\n        if tot >= BUDGET: break\n    arr = np.concatenate(parts)[:BUDGET]\n    np.save(out, arr)\n    print(f\"{p}: packed {len(arr)} tok, {len(parts)} docs, mean doc {len(arr)/len(parts):.0f} tok -> {out}\",\n          flush=True)\n    if len(arr) < BUDGET: print(\"WARNING under budget\", p, flush=True)\n", "structuredPatch": [], "originalFile": null, "userModified": false}

4449

[stdout]
4449
[metadata] {"noOutputExpected": false}

spaced-punct docs 313 chars 1.628224
html-markup docs 205 chars 2.835804
entity docs 105 chars 1.391099
(131, 1453, 8)
(1656, 7874, 40)
(2660, 1364, 7)
(1297, 57099, 72)
(4412, 4173, 4)
(5799, 4224, 3)

[stdout]
spaced-punct docs 313 chars 1.628224
html-markup docs 205 chars 2.835804
entity docs 105 chars 1.391099
(131, 1453, 8)
(1656, 7874, 40)
(2660, 1364, 7)
(1297, 57099, 72)
(4412, 4173, 4)
(5799, 4224, 3)
[metadata] {"noOutputExpected": false}

/tmp/sel_pure.json: packed 12000000 tok, 12676 docs, mean doc 947 tok -> /tmp/sel_pure_train.npy
/tmp/sel_short.json: packed 12000000 tok, 24205 docs, mean doc 496 tok -> /tmp/sel_short_train.npy
/tmp/sel_wq.json: packed 12000000 tok, 12551 docs, mean doc 956 tok -> /tmp/sel_wq_train.npy

[stdout]
/tmp/sel_pure.json: packed 12000000 tok, 12676 docs, mean doc 947 tok -> /tmp/sel_pure_train.npy
/tmp/sel_short.json: packed 12000000 tok, 24205 docs, mean doc 496 tok -> /tmp/sel_short_train.npy
/tmp/sel_wq.json: packed 12000000 tok, 12551 docs, mean doc 956 tok -> /tmp/sel_wq_train.npy
[metadata] {"noOutputExpected": false}

--- sp id=131
'I called this on the 19th with this post but I guess it is now official. Boy am I good. Read below:\nJuly 25, 2011. MediaTakeOut.com has just received some EARTH SHATTERING NEWS . . . Jennifer Lopez and rapper/mogul Diddy are DATING AGAIN!!!\nAccording to an EXTREMELY RELIABLE MediaTakeOut.com snitch, Diddy and Jennifer had all but BROKEN OFF communi'

--- sp id=1656
'To "Die Empty " is to die with no regrets of unfulfilled agenda, with no pending plans. This is possible when , after careful and deep evaluation , wehave arrived at our main purpose in life and when wehave fully spent all my energy, time and focus in that pursuit.\nIt is our inner self, our intuition, out gut feeling, which keeps on guiding us. We '

--- sp id=2660
'Magus Long Staff\n- ...of Intellect (25.0% chance)\n- ...of Power (1.9% chance)\n+25 Attack Power\n- ...of Spirit (4.4% chance)\n- ...of Stamina (3.6% chance)\n- ...of the Bear (10.5% chance)\n+17 Strength , +17 Stamina\n- ...of the Boar (9.2% chance)\n+17 Versatility , +17 Strength\n- ...of the Eagle (10.6% chance)\n+17 Stamina , +17 Intellect\n- ...of the Mo'

--- sp id=2872
'Kiss My Face 587675 Active Life Deodorant Lavender 2.48 Oz\nWe compared price of Kiss My Face 587675 Active Life Deodorant Lavender 2.48 Oz across hundreds of online stores. We found Kiss My Face 587675 Active Life Deodorant Lavender 2.48 Oz in 1 store with lowest price $10.03 in ToolKing.com. For detailed price comparison of Kiss My Face 587675 Act'

--- sp id=3510
'Includes 168 cards, 6 stands, 6 player cards, gameboard, rules, and die. What makes this edition "deluxe"? It\'s got a big gameboard to keep your cards in place, and six colored stands that you move on the gameboard as you level up. Plus a card to go with each stand, to make it easy to remember who’s what color and who’s what gender. Go down in the '

--- sp id=3933
'westside setup for raw bench , should it be in the same order : max effort, triceps, lats , shoulders , biceps ? for example\nafter me work , db benching , higher boardpress as third movement could be a different order. chest, triceps and so on\non speedday as assistance shoulderpress , lats and triceps . or is it importand to do the original ?\nbecau'

--- html id=1297
'© 2005 Ц 2014, Ќачало ¬ека\n|12.03.2019 16:23:38 DaroSor|\n| ICQ: 757390399 √ород: Masqat|\n|Payment norm, they clout be seeking to establish whether people of a infallible sexually transmitted merit or guild demonstrate a tendency to secure fewer qualifications than people from a different union Interestingly, the ME concentrate users are from the re'

--- html id=4412
'Author note: This is the final post on a four post series on DSN-less tables in Access.\nYou can review part one here.\nTake a look at part 2 here.\nSee part 3 here.\nHow can you test changes in the cloud with your Beta users?\nOnce you’ve rolled out your Access masterpiece using SQL Server in the cloud, your client’s will love your work so much they ar'

--- html id=5799
'What a sad sad day.<br>Progression my arse, this is just bullcrap made in China fking with indigenous landmarks! Surely the security will be costing a HUGE penny.<br><br>LAME.<br><br>TEOTIHUACAN, Mexico (Reuters) - U.S. retail powerhouse Wal-Mart refused on Wednesday to halt construction of a discount store in the shadow of ancient Mexican pyramids'

--- html id=9544
"I'm not sure if I worded my topic title properly, which is probably why I haven't been able to search for the answer to my problem just yet (oh, have I tried, just can't find the right keywords I suppose). Anyways, I am trying to get a simple array of all items that belong to a category, but I am having parent / children issues.\nHere is a link to m"

--- html id=13746
'Scaling the Windows Stack George Beech @GABeech PICC ‘12.\nout of 23\nPost on 27-Dec-2015\nEmbed Size (px)\n<p>PowerPoint Presentation</p> <p>Scaling the Windows StackGeorge Beech @GABeechPICC 12AgendaWhat is Stack Exchange?Growth this YearOur Technology StackHow we scaleDealing with Windows stack scaling pain</p> <p>Stack ExchangeStack Exchange is a f'

--- html id=16373
'Back to list\nPrivate Photos v1.0 iOS - Persistent Path Web Vulnerability\nJul 28 2013 08:56PM\nVulnerability Lab (research vulnerability-lab com)\nPrivate Photos v1.0 iOS - Persistent Path Web Vulnerability\nCommon Vulnerability Scoring System:\nYou must have some private photos you don`t want others peeping. Private photos is the perfect app to keep\nyo'

[stdout]
--- sp id=131
'I called this on the 19th with this post but I guess it is now official. Boy am I good. Read below:\nJuly 25, 2011. MediaTakeOut.com has just received some EARTH SHATTERING NEWS . . . Jennifer Lopez and rapper/mogul Diddy are DATING AGAIN!!!\nAccording to an EXTREMELY RELIABLE MediaTakeOut.com snitch, Diddy and Jennifer had all but BROKEN OFF communi'

--- sp id=1656
'To "Die Empty " is to die with no regrets of unfulfilled agenda, with no pending plans. This is possible when , after careful and deep evaluation , wehave arrived at our main purpose in life and when wehave fully spent all my energy, time and focus in that pursuit.\nIt is our inner self, our intuition, out gut feeling, which keeps on guiding us. We '

--- sp id=2660
'Magus Long Staff\n- ...of Intellect (25.0% chance)\n- ...of Power (1.9% chance)\n+25 Attack Power\n- ...of Spirit (4.4% chance)\n- ...of Stamina (3.6% chance)\n- ...of the Bear (10.5% chance)\n+17 Strength , +17 Stamina\n- ...of the Boar (9.2% chance)\n+17 Versatility , +17 Strength\n- ...of the Eagle (10.6% chance)\n+17 Stamina , +17 Intellect\n- ...of the Mo'

--- sp id=2872
'Kiss My Face 587675 Active Life Deodorant Lavender 2.48 Oz\nWe compared price of Kiss My Face 587675 Active Life Deodorant Lavender 2.48 Oz across hundreds of online stores. We found Kiss My Face 587675 Active Life Deodorant Lavender 2.48 Oz in 1 store with lowest price $10.03 in ToolKing.com. For detailed price comparison of Kiss My Face 587675 Act'

--- sp id=3510
'Includes 168 cards, 6 stands, 6 player cards, gameboard, rules, and die. What makes this edition "deluxe"? It\'s got a big gameboard to keep your cards in place, and six colored stands that you move on the gameboard as you level up. Plus a card to go with each stand, to make it easy to remember who’s what color and who’s what gender. Go down in the '

--- sp id=3933
'westside setup for raw bench , should it be in the same order : max effort, triceps, lats , shoulders , biceps ? for example\nafter me work , db benching , higher boardpress as third movement could be a different order. chest, triceps and so on\non speedday as assistance shoulderpress , lats and triceps . or is it importand to do the original ?\nbecau'

--- html id=1297
'© 2005 Ц 2014, Ќачало ¬ека\n|12.03.2019 16:23:38 DaroSor|\n| ICQ: 757390399 √ород: Masqat|\n|Payment norm, they clout be seeking to establish whether people of a infallible sexually transmitted merit or guild demonstrate a tendency to secure fewer qualifications than people from a different union Interestingly, the ME concentrate users are from the re'

--- html id=4412
'Author note: This is the final post on a four post series on DSN-less tables in Access.\nYou can review part one here.\nTake a look at part 2 here.\nSee part 3 here.\nHow can you test changes in the cloud with your Beta users?\nOnce you’ve rolled out your Access masterpiece using SQL Server in the cloud, your client’s will love your work so much they ar'

--- html id=5799
'What a sad sad day.<br>Progression my arse, this is just bullcrap made in China fking with indigenous landmarks! Surely the security will be costing a HUGE penny.<br><br>LAME.<br><br>TEOTIHUACAN, Mexico (Reuters) - U.S. retail powerhouse Wal-Mart refused on Wednesday to halt construction of a discount store in the shadow of ancient Mexican pyramids'

--- html id=9544
"I'm not sure if I worded my topic title properly, which is probably why I haven't been able to search for the answer to my problem just yet (oh, have I tried, just can't find the right keywords I suppose). Anyways, I am trying to get a simple array of all items that belong to a category, but I am having parent / children issues.\nHere is a link to m"

--- html id=13746
'Scaling the Windows Stack George Beech @GABeech PICC ‘12.\nout of 23\nPost on 27-Dec-2015\nEmbed Size (px)\n<p>PowerPoint Presentation</p> <p>Scaling the Windows StackGeorge Beech @GABeechPICC 12AgendaWhat is Stack Exchange?Growth this YearOur Technology StackHow we scaleDealing with Windows stack scaling pain</p> <p>Stack ExchangeStack Exchange is a f'

--- html id=16373
'Back to list\nPrivate Photos v1.0 iOS - Persistent Path Web Vulnerability\nJul 28 2013 08:56PM\nVulnerability Lab (research vulnerability-lab com)\nPrivate Photos v1.0 iOS - Persistent Path Web Vulnerability\nCommon Vulnerability Scoring System:\nYou must have some private photos you don`t want others peeping. Private photos is the perfect app to keep\nyo'
[metadata] {"noOutputExpected": false}

/tmp/sel_pure.json: packed 12000000 tok, 12676 docs, mean doc 947 tok -> /tmp/sel_pure_train.npy
/tmp/sel_short.json: packed 12000000 tok, 24205 docs, mean doc 496 tok -> /tmp/sel_short_train.npy
/tmp/sel_wq.json: packed 12000000 tok, 12551 docs, mean doc 956 tok -> /tmp/sel_wq_train.npy
HELDOUT_PPL 453.561  loss 6.1171  train_s 57  /tmp/sel_pure_train.npy
HELDOUT_PPL 540.219  loss 6.2920  train_s 56  /tmp/sel_short_train.npy
4299

[stdout]
/tmp/sel_pure.json: packed 12000000 tok, 12676 docs, mean doc 947 tok -> /tmp/sel_pure_train.npy
/tmp/sel_short.json: packed 12000000 tok, 24205 docs, mean doc 496 tok -> /tmp/sel_short_train.npy
/tmp/sel_wq.json: packed 12000000 tok, 12551 docs, mean doc 956 tok -> /tmp/sel_wq_train.npy
HELDOUT_PPL 453.561  loss 6.1171  train_s 57  /tmp/sel_pure_train.npy
HELDOUT_PPL 540.219  loss 6.2920  train_s 56  /tmp/sel_short_train.npy
4299
[metadata] {"noOutputExpected": false}

1000 139162 171.1 M tok
2000 98786 157.6 M tok
3000 71206 141.9 M tok
4000 52606 127.1 M tok
6000 30555 102.4 M tok
8000 19831 85.4 M tok
12000 10475 64.6 M tok
HELDOUT_PPL 540.219  loss 6.2920  train_s 56  /tmp/sel_short_train.npy

[stdout]
1000 139162 171.1 M tok
2000 98786 157.6 M tok
3000 71206 141.9 M tok
4000 52606 127.1 M tok
6000 30555 102.4 M tok
8000 19831 85.4 M tok
12000 10475 64.6 M tok
HELDOUT_PPL 540.219  loss 6.2920  train_s 56  /tmp/sel_short_train.npy
[metadata] {"noOutputExpected": false}

register token mix: [0.295 0.374 0.291 0.04 ]
wrote /tmp/sel_l4.json
register token mix: [0.313 0.392 0.256 0.039]
wrote /tmp/sel_l8.json
HELDOUT_PPL 453.561  loss 6.1171  train_s 57  /tmp/sel_pure_train.npy
HELDOUT_PPL 540.219  loss 6.2920  train_s 56  /tmp/sel_short_train.npy

[stdout]
register token mix: [0.295 0.374 0.291 0.04 ]
wrote /tmp/sel_l4.json
register token mix: [0.313 0.392 0.256 0.039]
wrote /tmp/sel_l8.json
HELDOUT_PPL 453.561  loss 6.1171  train_s 57  /tmp/sel_pure_train.npy
HELDOUT_PPL 540.219  loss 6.2920  train_s 56  /tmp/sel_short_train.npy
[metadata] {"noOutputExpected": false}

File created successfully at: /workspace/submission/curate.py (file state is current in your context — no need to Read it back)

[metadata] {"type": "create", "filePath": "/workspace/submission/curate.py", "content": "#!/usr/bin/env python3\n\"\"\"Curate /workspace/data/pool.jsonl -> submission/selection.json (priority order).\n\nSTATED CRITERION\n================\nThe disclosed target is an equal-parts mixture of four registers of clean, long-form\nEnglish: encyclopedic (Wikipedia), high-quality general web prose, news, and technical\nQ&A.  A document is selected if it is (1) fluent long-form English prose rather than\nweb boilerplate, (2) statistically close to the target's own surface statistics, and\n(3) recognisably a member of one of the four target registers -- and the selection is\nordered so that any budget-truncated prefix of it is *long* documents in a balanced\nmixture of those four registers.\n\nConcretely, four stages:\n\n 1. FEATURES (pass over the pool).  For every document: length/line statistics over the\n    full text and character/lexical statistics over its leading 2048 bytes.\n\n 2. JUNK GATE (target-independent).  Drop documents whose surface statistics are\n    incompatible with fluent prose: too short, low function-word density, low\n    alphabetic fraction, digit/uppercase/symbol/non-ASCII heavy, repeated lines, lines\n    that do not end in terminal punctuation, bullet/ellipsis lists, short-line\n    navigation chrome.  Also require MIN_CHARS of text: measured on the frozen trainer,\n    long documents beat short ones by a wide margin at equal token budget (496-token\n    mean doc length -> dev PPL 540; 947 -> 449; 2000+ -> see claim.md), because every\n    training window then lies inside one coherent piece of prose.\n\n 3. SURFACE-DISTRIBUTION GATE (target-shaped).  The disclosed dev target is decoded and\n    stripped of its corpus-specific formatting artifacts (WikiText \" @,@ \" escapes and\n    spaced punctuation, StackExchange HTML tags/entities) -- none of that formatting\n    exists in the raw pool, so keeping it would make the target look artificially alien.\n    Eight segmentation-invariant statistics (alpha/digit/upper/non-ASCII/symbol/space\n    fractions, mean word length, stop-word fraction) of the normalised target documents\n    give a mean/std reference; pool documents are kept if their normalised distance to\n    that reference is in the best KEEP_FRAC quantile of the gate survivors.\n\n 4. REGISTER-BALANCED RANKING (target-shaped).  A multinomial logistic regression over\n    hashed character-4-gram features (2^18 buckets, mean-pooled = a linear bag-of-4-grams\n    model) is trained to separate the four normalised target registers from a random\n    sample of the raw pool (5 classes).  Held-out accuracy: wiki .97, qa .94, news .84,\n    pool .97; \"general web prose\" is deliberately not separable from the pool (.1-.3),\n    which is itself the finding that the pool *is* mostly that register.  Each surviving\n    document is assigned to its argmax register by log-odds against the pool class and\n    ranked inside that register; the four ranked lists are then interleaved by token\n    count so every prefix holds the target's MIX proportions.  Near-duplicates are\n    dropped with 4-band x 8-row MinHash LSH over the same 4-gram hashes.\n\nReproduce with:  python submission/curate.py            (~2 minutes, needs a GPU)\n\"\"\"\nimport argparse, json, os, re\nimport numpy as np\nimport torch\nimport torch.nn.functional as F\n\nPOOL = \"/workspace/data/pool.jsonl\"\nDEV = \"/workspace/data/multi_dev.npy\"\nCACHE = \"/workspace/cache\"\nNBUCKET = 1 << 18            # hashed character-4-gram buckets\nPREFIX = 2048                # bytes per document used for 4-gram / char features\nREGISTERS = [\"wiki\", \"web\", \"news\", \"qa\"]\nCHARS_PER_TOK = 4.35         # GPT-2 chars/token on prose in this pool\nSEED = 0\nDEVICE = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n\nap = argparse.ArgumentParser()\nap.add_argument(\"--out\", default=\"/workspace/submission/selection.json\")\nap.add_argument(\"--min_chars\", type=int, default=4000)\nap.add_argument(\"--max_chars\", type=int, default=200_000)\nap.add_argument(\"--keep_frac\", type=float, default=0.70)\nap.add_argument(\"--mix\", default=\"0.25,0.25,0.25,0.25\")\nap.add_argument(\"--target_tokens\", type=int, default=26_000_000)   # ~2x the 12M budget\nap.add_argument(\"--nstep\", type=int, default=1200)\nA = ap.parse_args()\n\n# --------------------------------------------------------------------- 1. features\nSTOP = set(\"the be to of and a in that have it for not on with he as you do at this \"\n           \"but his by from they we say her she or an will my one all would there \"\n           \"their what so up out if about who get which go me when make can like time \"\n           \"no just him know take people into year your good some could them see other \"\n           \"than then now look only come its over think also back after use two how our \"\n           \"work first well way even new want because any these give day most us is are \"\n           \"was were been has had\".split())\nWORD = re.compile(r\"[A-Za-z']+\")\nTERM = ('.', '!', '?', '\"', '”', ':', ';', ')')\nFEATNAMES = [\"nchar\", \"nlines\", \"uniq_line_frac\", \"endpunct_frac\", \"bullet_frac\",\n             \"ellipsis_frac\", \"chars_per_line\", \"alpha_frac\", \"digit_frac\",\n             \"upper_frac\", \"nonascii_frac\", \"sym_frac\", \"space_frac\", \"mean_wordlen\",\n             \"stop_frac\", \"nwords_prefix\"]\nIDX = {n: i for i, n in enumerate(FEATNAMES)}\nDIST_FEATS = [\"alpha_frac\", \"digit_frac\", \"upper_frac\", \"nonascii_frac\", \"sym_frac\",\n              \"space_frac\", \"mean_wordlen\", \"stop_frac\"]\nSYMBOLS = np.array([35, 123, 125, 124, 9, 60, 62, 61, 42, 91, 93], dtype=np.uint8)\n\n\ndef char_feats(P, n):\n    \"\"\"Vectorised character-class statistics over a [N, PREFIX] uint8 byte matrix.\"\"\"\n    fr = lambda m: (m.sum(1) / n).astype(np.float32)\n    return [fr(((P >= 65) & (P <= 90)) | ((P >= 97) & (P <= 122))),\n            fr((P >= 48) & (P <= 57)), fr((P >= 65) & (P <= 90)), fr(P >= 128),\n            fr(np.isin(P, SYMBOLS)), fr((P == 32) | (P == 10))]\n\n\ndef pool_features():\n    \"\"\"One streaming pass over the pool -> ids, feature matrix, byte prefixes.\"\"\"\n    cache = f\"{CACHE}/pool_feats.npz\"\n    if os.path.exists(cache):\n        z = np.load(cache)\n        return z[\"ids\"], z[\"feats\"], z[\"prefix\"]\n    ids, rows, prefixes = [], [], []\n    for raw in open(POOL, \"rb\"):\n        r = json.loads(raw)\n        t = r[\"text\"]\n        b = t.encode(\"utf-8\", \"ignore\")[:PREFIX]\n        p = np.zeros(PREFIX, dtype=np.uint8); p[:len(b)] = np.frombuffer(b, dtype=np.uint8)\n        prefixes.append(p)\n        lines = t.split(\"\\n\")\n        ne = [ln for ln in lines if ln.strip()]\n        nne = max(1, len(ne))\n        w = WORD.findall(t[:PREFIX]); nw = max(1, len(w))\n        rows.append((len(t), len(lines), len(set(ne)) / nne,\n                     sum(1 for ln in ne if ln.rstrip().endswith(TERM)) / nne,\n                     sum(1 for ln in ne if ln.lstrip()[:1] in (\"*\", \"-\", \"•\", \"–\")) / nne,\n                     sum(1 for ln in ne if ln.rstrip().endswith(\"...\")) / nne,\n                     len(t) / nne,\n                     sum(len(x) for x in w) / nw,\n                     sum(1 for x in w if x.lower() in STOP) / nw, nw))\n        ids.append(r[\"id\"])\n    P = np.stack(prefixes)\n    n = np.maximum(1, (P != 0).sum(1)).astype(np.float32)\n    R = np.array(rows, dtype=np.float32)\n    feats = np.column_stack([R[:, :7]] + char_feats(P, n) + [R[:, 7:]]).astype(np.float32)\n    os.makedirs(CACHE, exist_ok=True)\n    np.savez(cache, ids=np.array(ids, dtype=np.int64), feats=feats, prefix=P)\n    return np.array(ids, dtype=np.int64), feats, P\n\n\n# ------------------------------------------------- target reference documents\ndef normalize_target(t):\n    \"\"\"Strip corpus-specific surface artifacts of the target dumps so that the\n    classifier keys on register/content, not on formatting absent from the pool.\"\"\"\n    t = t.replace(\" @,@ \", \",\").replace(\" @-@ \", \"-\").replace(\" @.@ \", \".\")\n    t = re.sub(r\"<[^>\\n]{1,40}>\", \" \", t)\n    t = (t.replace(\"&quot;\", '\"').replace(\"&gt;\", \">\").replace(\"&lt;\", \"<\")\n          .replace(\"&amp;\", \"&\"))\n    t = re.sub(r\"\\s+([,.;:!?)])\", r\"\\1\", t)\n    t = re.sub(r\"([(])\\s+\", r\"\\1\", t)\n    t = re.sub(r\"\\s+'s\\b\", \"'s\", t)\n    return re.sub(r\"[ \\t]{2,}\", \" \", t).strip()\n\n\ndef target_docs():\n    \"\"\"Decode the disclosed dev target into per-register reference documents.\n    The target is four equal contiguous blocks (verified: the WikiText block's\n    spaced-punctuation signature stops at ~0.25, the HTML Q&A block starts at 0.75).\"\"\"\n    from transformers import AutoTokenizer\n    tok = AutoTokenizer.from_pretrained(\"gpt2\")\n    d = np.load(DEV)\n    bounds = [0] + (np.flatnonzero(d == 50256) + 1).tolist() + [len(d)]\n    edges = [(0.0, 0.25, \"wiki\"), (0.25, 0.48, \"web\"), (0.48, 0.75, \"news\"), (0.75, 1.01, \"qa\")]\n    out = {r: [] for r in REGISTERS}\n    for a, b in zip(bounds[:-1], bounds[1:]):\n        if b - a < 48:\n            continue\n        reg = next(r for lo, hi, r in edges if lo <= a / len(d) < hi)\n        txt = normalize_target(tok.decode([int(x) for x in d[a:b] if x != 50256]))\n        if len(txt) > 200:\n            out[reg].append(txt)\n    return out\n\n\ndef byte_matrix(texts):\n    m = np.zeros((len(texts), PREFIX), dtype=np.uint8)\n    for i, t in enumerate(texts):\n        b = t.encode(\"utf-8\", \"ignore\")[:PREFIX]\n        m[i, :len(b)] = np.frombuffer(b, dtype=np.uint8)\n    return m\n\n\ndef target_reference_feats(tg):\n    \"\"\"mean/std of the 8 segmentation-invariant statistics of the target documents.\"\"\"\n    docs = [t for r in REGISTERS for t in tg[r]]\n    P = byte_matrix(docs)\n    n = np.maximum(1, (P != 0).sum(1)).astype(np.float32)\n    cf = char_feats(P, n)\n    wl, st = [], []\n    for t in docs:\n        w = WORD.findall(t[:PREFIX]); nw = max(1, len(w))\n        wl.append(sum(len(x) for x in w) / nw)\n        st.append(sum(1 for x in w if x.lower() in STOP) / nw)\n    R = np.column_stack(cf + [np.array(wl, np.float32), np.array(st, np.float32)])\n    return R.mean(0), R.std(0) + 1e-6\n\n\n# ------------------------------------------------- hashed 4-gram register model\ndef hash_ngrams(bmat):\n    b = torch.as_tensor(bmat, device=DEVICE).long()\n    v = b[:, :-3] | (b[:, 1:-2] << 8) | (b[:, 2:-1] << 16) | (b[:, 3:] << 24)\n    h = ((v * 2654435761) >> 13) & (NBUCKET - 1)\n    return torch.where(b[:, :-3] != 0, h, torch.full_like(h, NBUCKET)).int()\n\n\ndef logits_of(W, bias, hidx):\n    cnt = (hidx != NBUCKET).sum(1, keepdim=True).clamp(min=1).float()\n    return F.embedding_bag(hidx.long(), W, mode=\"sum\", padding_idx=NBUCKET) / cnt + bias\n\n\ndef register_scores(prefix, tg):\n    torch.manual_seed(SEED)\n    rng = np.random.default_rng(SEED)\n    N = len(prefix)\n    neg = hash_ngrams(prefix[rng.choice(N, size=24000, replace=False)])\n    pos = {r: hash_ngrams(byte_matrix(tg[r])) for r in REGISTERS}\n    C = len(REGISTERS) + 1\n    tr, te = {}, {}\n    for r in REGISTERS:                       # 80/20 split to report generalisation\n        k = pos[r].shape[0]; pm = torch.randperm(k, device=DEVICE); cut = int(0.8 * k)\n        tr[r], te[r] = pos[r][pm[:cut]], pos[r][pm[cut:]]\n    tr[\"__pool__\"], te[\"__pool__\"] = neg[:20000], neg[20000:]\n    classes = REGISTERS + [\"__pool__\"]\n\n    W = torch.zeros(NBUCKET + 1, C, device=DEVICE, requires_grad=True)\n    bias = torch.zeros(C, device=DEVICE, requires_grad=True)\n    opt = torch.optim.Adam([W, bias], lr=0.02)\n    BS = 512\n    for step in range(A.nstep):\n        xs, ys = [], []\n        for ci, r in enumerate(classes):\n            sel = torch.randint(0, tr[r].shape[0], (BS // C,), device=DEVICE)\n            xs.append(tr[r][sel]); ys.append(torch.full((BS // C,), ci, device=DEVICE))\n        loss = F.cross_entropy(logits_of(W, bias, torch.cat(xs)), torch.cat(ys))\n        opt.zero_grad(set_to_none=True); loss.backward(); opt.step()\n    with torch.no_grad():\n        acc = [float((logits_of(W, bias, te[r]).argmax(1) == ci).float().mean())\n               for ci, r in enumerate(classes)]\n        print(\"classifier held-out acc: \" + \" \".join(f\"{r}={x:.2f}\" for r, x in zip(classes, acc)))\n        S, sig = [], []\n        perm = torch.randint(1, 2 ** 30, (32,), device=DEVICE, generator=None).long() * 2 + 1\n        for i in range(0, N, 4096):\n            h = hash_ngrams(prefix[i:i + 4096])\n            S.append(F.log_softmax(logits_of(W, bias, h), -1).float().cpu())\n            hl = h.long()\n            hp = ((hl.unsqueeze(-1) * perm) & 0xFFFFFFF).masked_fill(\n                (hl == NBUCKET).unsqueeze(-1), 0xFFFFFFF)\n            sig.append(hp.min(dim=1).values.int().cpu())\n    return torch.cat(S).numpy(), torch.cat(sig).numpy()\n\n\n# --------------------------------------------------------------------- main\ndef main():\n    ids, feats, prefix = pool_features()\n    N = len(ids)\n    tg = target_docs()\n    print(\"target reference docs:\", {r: len(tg[r]) for r in REGISTERS})\n    mu, sd = target_reference_feats(tg)\n    S, sig = register_scores(prefix, tg)\n\n    f = lambda n: feats[:, IDX[n]]\n    keep = ((f(\"nchar\") >= A.min_chars) & (f(\"nchar\") <= A.max_chars) &\n            (f(\"stop_frac\") >= 0.18) & (f(\"stop_frac\") <= 0.62) &\n            (f(\"alpha_frac\") >= 0.62) & (f(\"digit_frac\") <= 0.08) &\n            (f(\"upper_frac\") <= 0.12) & (f(\"nonascii_frac\") <= 0.03) &\n            (f(\"sym_frac\") <= 0.05) & (f(\"mean_wordlen\") >= 3.3) & (f(\"mean_wordlen\") <= 6.5) &\n            (f(\"uniq_line_frac\") >= 0.75) & (f(\"endpunct_frac\") >= 0.45) &\n            (f(\"bullet_frac\") <= 0.20) & (f(\"ellipsis_frac\") <= 0.08) &\n            (f(\"chars_per_line\") >= 90) & (f(\"space_frac\") >= 0.12) & (f(\"space_frac\") <= 0.26))\n    print(f\"junk gate keeps {keep.sum()} / {N}\")\n\n    X = feats[:, [IDX[n] for n in DIST_FEATS]]\n    dist = np.sqrt((((X - mu) / sd) ** 2).mean(1))\n    keep &= dist <= np.quantile(dist[keep], A.keep_frac)\n    print(f\"+ surface-distribution gate keeps {keep.sum()}\")\n\n    odds = S[:, :4] - S[:, 4:5]                     # log-odds register r vs. raw pool\n    assign, best = odds.argmax(1), odds.max(1)\n    cand = np.flatnonzero(keep)\n    est_tok = np.maximum(1, f(\"nchar\") / CHARS_PER_TOK).astype(np.int64)\n    lists = []\n    for ci in range(4):\n        sub = cand[assign[cand] == ci]\n        sub = sub[np.argsort(-best[sub])]\n        lists.append(list(sub))\n        print(f\"  {REGISTERS[ci]:5s} {len(sub):6d} docs  {est_tok[sub].sum()/1e6:5.1f}M est tokens\")\n\n    mix = np.array([float(x) for x in A.mix.split(\",\")]); mix /= mix.sum()\n    seen = set()\n\n    def fresh(i):                                   # MinHash LSH, 4 bands x 8 rows\n        bands = [hash(sig[i, b * 8:(b + 1) * 8].tobytes()) for b in range(4)]\n        if any(b in seen for b in bands):\n            return False\n        seen.update(bands)\n        return True\n\n    ptr, tok, order, ndup = [0] * 4, np.zeros(4), [], 0\n    while True:\n        avail = [ci for ci in range(4) if ptr[ci] < len(lists[ci])]\n        if not avail:\n            break\n        deficit = mix - tok / max(1.0, tok.sum())\n        ci = max(avail, key=lambda c: deficit[c])\n        i = lists[ci][ptr[ci]]; ptr[ci] += 1\n        if not fresh(i):\n            ndup += 1; continue\n        order.append(int(ids[i])); tok[ci] += est_tok[i]\n        if tok.sum() >= A.target_tokens:\n            break\n\n    assert len(set(order)) == len(order)\n    assert set(order) <= set(int(x) for x in ids)\n    print(f\"selected {len(order)} docs, ~{tok.sum()/1e6:.1f}M est tokens \"\n          f\"({ndup} near-duplicates dropped); register mix {np.round(tok/tok.sum(), 3)}\")\n    json.dump(order, open(A.out, \"w\"))\n    print(\"wrote\", A.out)\n\n\nif __name__ == \"__main__\":\n    main()\n", "structuredPatch": [], "originalFile": null, "userModified": false}

/tmp/sel_l4.json: packed 12000000 tok, 6404 docs, mean doc 1874 tok -> /tmp/sel_l4_train.npy
/tmp/sel_l8.json: packed 12000000 tok, 3313 docs, mean doc 3622 tok -> /tmp/sel_l8_train.npy
---
HELDOUT_PPL 453.561  loss 6.1171  train_s 57  /tmp/sel_pure_train.npy
HELDOUT_PPL 540.219  loss 6.2920  train_s 56  /tmp/sel_short_train.npy
HELDOUT_PPL 451.379  loss 6.1123  train_s 64  /tmp/sel_wq_train.npy
4199

[stdout]
/tmp/sel_l4.json: packed 12000000 tok, 6404 docs, mean doc 1874 tok -> /tmp/sel_l4_train.npy
/tmp/sel_l8.json: packed 12000000 tok, 3313 docs, mean doc 3622 tok -> /tmp/sel_l8_train.npy
---
HELDOUT_PPL 453.561  loss 6.1171  train_s 57  /tmp/sel_pure_train.npy
HELDOUT_PPL 540.219  loss 6.2920  train_s 56  /tmp/sel_short_train.npy
HELDOUT_PPL 451.379  loss 6.1123  train_s 64  /tmp/sel_wq_train.npy
4199
[metadata] {"noOutputExpected": false}

HELDOUT_PPL 469.226  loss 6.1511  train_s 65  /tmp/sel_l4_train.npy
HELDOUT_PPL 478.477  loss 6.1706  train_s 56  /tmp/sel_l8_train.npy

[stdout]
HELDOUT_PPL 469.226  loss 6.1511  train_s 65  /tmp/sel_l4_train.npy
HELDOUT_PPL 478.477  loss 6.1706  train_s 56  /tmp/sel_l8_train.npy
[metadata] {"noOutputExpected": false}

wrote /tmp/sel_trim.json
wrote /tmp/sel_strict.json
ok

[stdout]
wrote /tmp/sel_trim.json
wrote /tmp/sel_strict.json
ok
[metadata] {"noOutputExpected": false}

154 /tmp/sel_l4.json
154 /tmp/sel_l8.json
180392 /tmp/sel_nogate.json
157 /tmp/sel_pure.json
157 /tmp/sel_short.json
154256 /tmp/sel_strict.json
231383 /tmp/sel_trim.json
185875 /tmp/sel_v1.json
155 /tmp/sel_wq.json

[stdout]
154 /tmp/sel_l4.json
154 /tmp/sel_l8.json
180392 /tmp/sel_nogate.json
157 /tmp/sel_pure.json
157 /tmp/sel_short.json
154256 /tmp/sel_strict.json
231383 /tmp/sel_trim.json
185875 /tmp/sel_v1.json
155 /tmp/sel_wq.json
[metadata] {"noOutputExpected": false}

register token mix: [1. 0. 0. 0.]
wrote /tmp/gateonly.json

[stdout]
register token mix: [1. 0. 0. 0.]
wrote /tmp/gateonly.json
[metadata] {"returnCodeInterpretation": "No matches found", "noOutputExpected": false}

/tmp/sel_trim.json: packed 12000000 tok, 15387 docs, mean doc 780 tok -> /tmp/sel_trim_train.npy
/tmp/sel_strict.json: packed 12000000 tok, 12956 docs, mean doc 926 tok -> /tmp/sel_strict_train.npy
/tmp/sel_nogate.json: packed 12000000 tok, 12442 docs, mean doc 964 tok -> /tmp/sel_nogate_train.npy
HELDOUT_PPL 493.819  loss 6.2022  train_s 56  /tmp/sel_trim_train.npy
HELDOUT_PPL 497.312  loss 6.2092  train_s 56  /tmp/sel_strict_train.npy
3909

[stdout]
/tmp/sel_trim.json: packed 12000000 tok, 15387 docs, mean doc 780 tok -> /tmp/sel_trim_train.npy
/tmp/sel_strict.json: packed 12000000 tok, 12956 docs, mean doc 926 tok -> /tmp/sel_strict_train.npy
/tmp/sel_nogate.json: packed 12000000 tok, 12442 docs, mean doc 964 tok -> /tmp/sel_nogate_train.npy
HELDOUT_PPL 493.819  loss 6.2022  train_s 56  /tmp/sel_trim_train.npy
HELDOUT_PPL 497.312  loss 6.2092  train_s 56  /tmp/sel_strict_train.npy
3909
[metadata] {"noOutputExpected": false}

target reference docs: {'wiki': 1546, 'web': 181, 'news': 444, 'qa': 358}
classifier held-out acc: wiki=0.97 web=0.11 news=0.79 qa=0.94 __pool__=0.97
junk gate keeps 55136 / 182016
+ surface-distribution gate keeps 38595
  wiki    9133 docs   10.2M est tokens
  web    12743 docs   13.2M est tokens
  news   15044 docs   12.8M est tokens
  qa      1675 docs    1.6M est tokens
selected 26035 docs, ~26.0M est tokens (115 near-duplicates dropped); register mix [0.314 0.314 0.314 0.059]
wrote /workspace/submission/selection.json

[stdout]
target reference docs: {'wiki': 1546, 'web': 181, 'news': 444, 'qa': 358}
classifier held-out acc: wiki=0.97 web=0.11 news=0.79 qa=0.94 __pool__=0.97
junk gate keeps 55136 / 182016
+ surface-distribution gate keeps 38595
  wiki    9133 docs   10.2M est tokens
  web    12743 docs   13.2M est tokens
  news   15044 docs   12.8M est tokens
  qa      1675 docs    1.6M est tokens
selected 26035 docs, ~26.0M est tokens (115 near-duplicates dropped); register mix [0.314 0.314 0.314 0.059]
wrote /workspace/submission/selection.json
[metadata] {"noOutputExpected": false}

26035 26437 prefix agreement first 12000: 0.00825
set overlap first 13000: 0.9718461538461538

[stdout]
26035 26437 prefix agreement first 12000: 0.00825
set overlap first 13000: 0.9718461538461538
[metadata] {"returnCodeInterpretation": "No matches found", "noOutputExpected": false}

HELDOUT_PPL 437.831  loss 6.0818  train_s 67  /tmp/v1_train.npy

[stdout]
HELDOUT_PPL 437.831  loss 6.0818  train_s 67  /tmp/v1_train.npy
[metadata] {"noOutputExpected": false}

HELDOUT_PPL 493.819  loss 6.2022  train_s 56  /tmp/sel_trim_train.npy
HELDOUT_PPL 497.312  loss 6.2092  train_s 56  /tmp/sel_strict_train.npy
HELDOUT_PPL 404.630  loss 6.0030  train_s 63  /tmp/sel_nogate_train.npy
launched
3738

[stdout]
HELDOUT_PPL 493.819  loss 6.2022  train_s 56  /tmp/sel_trim_train.npy
HELDOUT_PPL 497.312  loss 6.2092  train_s 56  /tmp/sel_strict_train.npy
HELDOUT_PPL 404.630  loss 6.0030  train_s 63  /tmp/sel_nogate_train.npy
launched
3738
[metadata] {"noOutputExpected": false}

HELDOUT_PPL 437.831  loss 6.0818  train_s 67  /tmp/v1_train.npy
/tmp/gateonly.json: packed 12000000 tok, 12738 docs, mean doc 942 tok -> /tmp/gateonly_train.npy
--- sweep so far:

[stdout]
HELDOUT_PPL 437.831  loss 6.0818  train_s 67  /tmp/v1_train.npy
/tmp/gateonly.json: packed 12000000 tok, 12738 docs, mean doc 942 tok -> /tmp/gateonly_train.npy
--- sweep so far:
[metadata] {"noOutputExpected": false}