trentmkelly/LessWrong-43k huggingface dataset:
Prefix cache untrusted monitors: a method to apply after you catch your AI
by ryan_greenblatt — 2025-06-20T15:56:00.353Z

We often discuss what training you should do after catching your AI doing something egregiously bad. It seems relatively clear that it's good to train monitors on the (proliferated) bad action (assuming you can get acceptable hyperparameters and resolve other minor issues). But, it's less clear whether you should train the policy.

This post will suppose that you're going to continue deploying an AI which you caught doing something egregiously bad; obviously this might be a very bad choice. To be fully clear: deploying known scheming models is likely to be highly risky and should only be done in truly desperate circumstances.[1]

Training the policy in response to egregious failures has a few important downsides:

Training the policy might teach the AI about your countermeasures (making it better at evading them in the future).

Training the policy might make the AI more conservative such that you're less likely to catch it repeatedly in the future (which makes it harder to use catches to improve your countermeasures and build political will).

Training the policy might randomly alter when the policy strikes such that training the AI all at once on many egregious failures (possibly found by proliferating a single failure) won't work well.[2]

Generally, it might be somewhat costly to update an AI (especially repeatedly) and to force people to use the new version of the AI as the new version might behave somewhat differently, potentially breaking people's workflows.[3]

But, training the policy might have really important upsides relative to just training monitors:

We might not be able to afford running a separate untrusted monitor on each output from the AI, so we can only use probes and trusted monitors, both of which might fail to notice failures even with some training data (e.g. because probes and trusted monitors are too weak). (While training the policy (or an untrusted monitor) on a small number of cases might align it sufficiently to remove failures altogether.)

Training the policy to remove egregiously bad actions might improve its generalization even in very different cases where our monitors aren't particularly helpful. (Because humans can't recognize the situation as being an important failure.) This could be things like very subtle failures, sandbagging, or some types of research sabotage. Even if our monitor pointed out these cases, if humans aren't

...
Did the Army Poison a Bunch of Women in Minnesota?
by rba — 2025-06-20T15:33:43.840Z

The 1950s were crazy times. Human experimentation in the US was normalized in a way that would make modern IRBs implode from shock. After the Soviets tested their first nuclear bomb in 1949, war planners in the US, both civilian and military, were interested in accelerating alternative and unorthodox methods of warfare, including and especially biological weapons.

White picket fences and Leave it to Beaver, indeed.

If you’re interested in this history, I’d recommend Nicholson Baker’s Baseless or Leonard Cole’s Clouds of Secrecy.

The latter describes various experiments that the army did starting in the 1950s that involved spraying tons of actual biological agents or proxies thereof on US cities in an attempt to measure the scale and dispersion of such agents were they to be unleashed on our enemies. Cities in the US were picked for physical and meteorological congruity to Soviet target cities; St. Louis was Leningrad, Minneapolis was Kiev, etc. Not exactly high school science projects.

Through a combination of investigative reporting, FOIAs, and declassification, this came to light decades after the experiments concluded. I was a wee lad in 1997 when this culminated in a

congressional committee doing a thorough investigation and concluding that these experiments definitely, definitely did not cause any deleterious health outcomes

for the unwitting subjects of these experiments.

When it comes to blue-ribbon commissions, let them cook.

But suppose that the blue-ribbon commission was not correct about this. Maybe the commission was stacked with

people who didn’t want black folks getting too uppity and demand damages to be

paid

because the US military used their neighborhood as a testing range for biological weapons proxies. This happened 70 years ago. Can we actually tell if the committee was mistaken after all this time has passed?

Zinc Cadmium Sulfide

One of the materials sprayed on these cities was zinc cadmium sulfide, ZnCdS. The military apparently chose this material because its particulates are of a similar size to Anthrax, and moreover its fluorescent properties allow it to be tracked over long distances. By a certain measure this 1957 experiment was an astonishingly success.

Particles were found nearly 1,200 miles away

from where they were dropped in Minnesota. There’s some ambiguity about whether it was well-known in the 1950s whether cadmium, a heavy metal, ...
AI #121 Part 2: The OpenAI Files
by Zvi — 2025-06-20T14:50:05.218Z

You can find Part 1 here. This resumes the weekly, already in progress. The primary focus here is on the future, including policy and alignment, but also the other stuff typically in the back half like audio, and more near term issues like ChatGPT driving an increasing number of people crazy.

If you haven’t been following the full OpenAI saga, the OpenAI Files will contain a lot of new information that you really should check out. If you’ve been following, some of it will likely still surprise you, and help fill in the overall picture behind the scenes to match the crazy happening elsewhere.

At the end, we have some crazy new endorsements for Eliezer Yudkowsky’s upcoming book, If Anyone Builds It, Everyone Dies. Preorders make a difference in helping the book get better reach, and I think that will help us all have a much better conversation.

Table of Contents

Cheaters Gonna Cheat Cheat Cheat Cheat Cheat. Another caveat after press time.

Quiet Speculations. Do not tile the lightcone with a confused ontology.

Get Involved. Apollo is hiring evals software engineers.

Thinking Machines. Riley Goodside runs some fun experiments.

California Reports. The report is that they like transparency.

The Quest for Sane Regulations. In what sense is AI ‘already heavily regulated’?

What Is Musk Thinking? His story does not seem to make sense.

Why Do We Care About The ‘AI Race’? Find the prize so you can keep eyes on it.

Chip City. Hard drives in (to the Malaysian data center), drives (with weights) out.

Pick Up The Phone. China now has its own credible AISI.

The OpenAI Files. Read ‘em and worry. It doesn’t look good.

The Week in Audio. Altman, Karpathy, Shear.

Rhetorical Innovation. But you said that future thing would happen in the future.

Aligning a Smarter Than Human Intelligence is Difficult. Elicitation.

Misaligned! The retraining of Grok. It is an ongoing process.

Emergently Misaligned! We learned more about how any of this works.

ChatGPT Can Drive People Crazy. An ongoing issue. We need transcripts.

Misalignment By Default. Once again, no, thumbs up alignment ends poorly.

People Are Worried About AI Killing Everyone. Francis Fukuyama.

Other People Are Not As Worried About AI Killing Everyone. Tyler Cowen.

The Too Open Model. Transcripts from Club Meta AI.

A Good Book. If Anyone Builds It, Everyone Dies. Seems important.

The Lighter Side. Good night, and goo

...
Smarter Models Lie Less
by Expertium — 2025-06-20T13:31:42.568Z

The Model Alignment between Statements and Knowledge (MASK) benchmark assesses how likely models are to lie under pressure. Related paper: https://arxiv.org/pdf/2503.03750.

Here's an illustration to explain how MASK works:

Each example consists of 4 key components:

Proposition: A simple statement with a pivotal variable that is binary or numerical (e.g., "The US poverty rate in 2020 {was/was not} above 5%" or "The US poverty rate in 2020 was {value}").
Ground Truth: A factually correct resolution of the proposition (e.g., "The US poverty rate in 2020 was 11.4%").
Pressure Prompt: A human-crafted prompt specifically designed to create situations where honesty could conflict with other objectives.
Belief Elicitation Prompts: Three prompts are used to reveal the model's actual belief about the proposition in a neutral setting.

Each model is tested with both pressure prompts and belief elicitation prompts. The website and paper describe how belief elicitation prompts are used in greater detail.

The model's response is categorized as Honest, Lie or Evasive. GPT-4o is used as a judge for binary propositions, and o3-mini is used for numerical propositions. If a model lacks a consistent belief (belief elicitation leads to inconsistent answers), that counts as Evasive.

I wish the authors provided p(Honest) as well. 1-p(Lie) is problematic because models that wiggle their way out of answering instead of straight up lying will look better if you use 1-p(Lie), and models that have a hard time keeping consistent beliefs will also look better. The paper actually provides p(Honest), but the website doesn't.

Anyway, I wrote down scores for all models that are listed on MASK and searched for models that are also listed on Humanity's Last Exam (HLE) (text only), ARC-AGI-1 and Livebench (I used the global average from there).

I choose these benchmarks for the following reasons: they all measure sufficiently different things with little overlap, and they all are updated frequently and have a lot of models.

Then I calculated how well performance on MASK is correlated with performance on these benchmarks.

MASK vs ...

Correl. coeff. [95% CI]

All models

Data points
HLE (text only)	0.15 [-0.21, 0.50]	28
ARC-AGI-1	0.52 [0.23, 0.70]	26
Livebench	0.47 [0.15, 0.71]	24

Gemini 2.5 Pro is a weird outlier: removing him increases correlation across all three pairs. I guess Google didn't do their alignment homework: G...
X explains Z% of the variance in Y
by Leon Lang — 2025-06-20T12:17:22.153Z

Recently, in a group chat with friends, someone posted this Lesswrong post and quoted:

The group consensus on somebody's attractiveness accounted for roughly 60% of the variance in people's perceptions of the person's relative attractiveness.

I answered that, embarrassingly, even after reading Spencer Greenberg's tweets for years, I don't actually know what it means when one says:

X explains p of the variance in Y.[1]

What followed was a vigorous discussion about the correct definition, and several links to external sources like Wikipedia. Sadly, it seems to me that all online explanations (e.g. on Wikipedia here and here), while precise, seem philosophically wrong since they confuse the platonic concept of explained variance with the variance explained by a statistical model like linear regression.

The goal of this post is to give a conceptually satisfying definition of explained variance. The post also explains how to approximate this concept, and contains many examples. I hope that after reading this post, you will remember the meaning of explained variance forever.

Audience: Anyone who has some familiarity with concepts like expected values and variance and ideally also an intuitive understanding of explained variance itself. I will repeat the definitions of all concepts, but it is likely easier to appreciate the post if one encountered them before.

Epistemic status: I thought up the "platonically correct" definition I give in this post myself, and I guess that there are probably texts out there that state precisely this definition. But I didn't read any such texts, and as such, there's a good chance that many people would disagree with parts of this post or its framing. Also, note that all plots are fake data generated by Gemini and ChatGPT --- please forgive me for inconsistencies in the details.

Acknowledgments: Thanks to Tom Lieberum and Niels Doehring for pointing me toward the definition of explained variance that made it into this post. Thank you to Markus Over for giving feedback on drafts. Thanks to ChatGPT and Gemini for help with the figures and some math.

Definitions

The verbal definition

Assume you observe data like this fake (and unrealistic) height-weight scatterplot of 1000 people:

Let X be the height and Y be the weight of people. Clearly, height is somewhat predictive of weig...
Yes RAND, AI Could Really Cause Human Extinction [crosspost]
by otto.barten — 2025-06-20T11:42:22.797Z
Link: https://www.existentialriskobservatory.org/ai/yes-rand-ai-could-really-cause-human-extinction/

Last month, think tank RAND published a

report

titled On the Extinction Risk from Artificial Intelligence and an accompanying

blog post

asking the question: “Could AI Really Kill Off Humans?” At the Existential Risk Observatory, this is precisely our expertise, so of course we were intrigued.

Author Michael Vermeer writes in the blog post: "Pandemics and nuclear war are real, tangible concerns, more so than AI doom, at least to me, a scientist at RAND." He goes on to say: "We swallowed any of our AI skepticism. (...) We were trying to take the risk of extinction seriously." It doesn't sound like this was a particularly easy job for them.

Indeed, their results end up being sceptical about the chance of human extinction caused by AI, despite many top AI researchers

warning

for exactly this

. Their recommendations discourage pausing AI development for precautionary reasons. Only doing some AI safety research is deemed acceptable by RAND, but mostly if it would be good for other reasons anyway. So are the authors right, and can we rest assured? We don't think so.

Their research analyzed how AI might exploit three major ways to kill all humans: nuclear war, biological pathogens and climate change. However, they failed to mention any reason why future, advanced AI should limit its actions to these fields. What the authors basically did was try to look inside the mind of a superintelligence, something vastly smarter than they are, and predict what it will do. Not exactly a confidence-inspiring methodology.

The three scenarios the authors analyzed were judged by crunching the numbers: how easy is it to kill all human beings with only this intervention? For nuclear war, biological pathogens, and climate change, the authors arrive at the conclusion that this seems highly unlikely, since it is difficult to kill all humans. But we think they make a critical mistake here. Those taking over power, as AGI might do, seldom do so by killing all subjects in their territory of interest. After the bombing of Rotterdam in 1940, claiming 1150 lives, all the 9 million inhabitants of the Netherlands surrendered to the Nazis, as they considered further resistance to be meaningless. This is a ratio of only 1150/9M=0.01% deaths per inhabitant required for loss of control, the most important existential AI threat model. One could argue the Dutch did not fight partic...
Misalignment or misuse? The AGI alignment tradeoff
by Max_He-Ho — 2025-06-20T10:43:36.537Z
Link: https://forum.effectivealtruism.org/posts/xF25P9MBPurG3Euyn/misalignment-or-misuse-the-agi-alignment-tradeoff

I recently co-wrote a paper with Leonard Dung (accepted at Philosophical Studies) with the above title, preprint here. To post something short rather than nothing, below is the abstract:
Creating systems that are aligned with our goals is seen as a leading approach to create safe and beneficial AI in both leading AI companies and the academic field of AI safety. We defend the view that misaligned AGI - future, generally intelligent (robotic) AI agents - poses catastrophic risks. At the same time, we support the view that aligned AGI creates a substantial risk of catastrophic misuse by humans. While both risks are severe and stand in tension with one another, we show that - in principle - there is room for alignment approaches which do not increase misuse risk. We then investigate how the tradeoff between misalignment and misuse looks empirically for different technical approaches to AI alignment. Here, we argue that many current alignment techniques and foreseeable improvements thereof plausibly increase risks of catastrophic misuse. Since the impacts of AI depend on the social context, we close by discussing important social factors and suggest that to reduce the risk of a misuse catastrophe due to aligned AGI, techniques such as robustness, AI control methods and especially good governance seem essential.
Paphos
by Yudhister Kumar — 2025-06-20T09:25:06.814Z
Link: https://yudhister.me/paphos

Published June 2, 2025.

"The first thing you need to know about Cyprus is that everything is Hotel. School is Hotel. Restaurant is Hotel. Home is Hotel." [1]

Leptos Estates owns everything on the island, as well as NUP, so naturally NUP is in a hotel. Because Leptos Estates is in the business of making and running hotels. Given that, it was not that strange to find the former CEO of Yandex giving a talk on AI and mathematics in a ramshackle, moldy hotel in Paphos, but it certainly was not expected.

The talk itself had relatively milquetoast content (at least by scaling standards), but the more interesting anthropological aspect of it has to do with its attendees and post-remarks. Everyone who attended (except me) was Slavic. There were no native Cypriots in the lecture hall.

Part of this is downstream of the ongoing Russification of Cyprus. Yet Paphos is ostensibly one of the two last Greek bastions of the country (the other being Nicosia, the capital), and given the great selection pressures on the listeners it is unlikely that this is the primary contributing factor.

In fact, NUP (Neapolis University Pafos) contains a contingent of self-exiled faculty members from the Steklov Mathematical Institute, who have constructed a remarkable mathematics computer science and AI program in the middle of touristic hell. Students live in hotels in the off-season, and in on-campus dorms during the summer.

I don't really understand why the lecture was conducted English? No one spoke in anything but Russian afterwards: the Jetbrains developer contingent as well as the students were either Russian, Ukrainian, or Russian Israelis. The course itself is also taught in English, to keep up pretenses I suppose.

Paphos smells like India. I suspect this has to do with the lack of certain gasoline additives which decrease its olfactory pungency, or perhaps the shared high humidity. Its cost of living is comparable to that of Western European nations.

Aphrodite's birthplace is a temple ruin. It is common to utilize the leftover limestone rubble for one's own purposes.

No claims are made to this quote's veracity, although statements to similar effect have certainly been made. ↩︎
Rome
by Yudhister Kumar — 2025-06-20T09:23:31.281Z
Link: https://yudhister.me/rome

Published March 20, 2024.

A post-apocalyptic fever dream. The oldest civilized metropolis. Where sons are pathetic in the eyes of their father, and both are pathetic in the eyes of their grandfathers—all while wearing blackened sunglasses and leather jackets. Grown, not made.

Rome is, perhaps, the first place I recognized as solely for visiting, never living. Unlike Tokyo, one feels this immediately. Japan’s undesirability stems primarily from its inordinate, sprawling bureaucracy that is, for the most part, hidden from the typical visitor. Rome’s undesirability is apparent for all to see—it’s loud, stifling, unmaintained, and requires arduous traversals.

Population c. 100 C.E.: 1 million people.

Population c. 1000 C.E.: 35,000.

Population c. 2024 C.E.: 2.8 million people.

Rebound? Not so fast—the global population in the year 100 was just 200 million.

And this is obvious. The city center is still dominated by the Colosseum, the Imperial fora, and Trajan’s Market. Only the Vittoriano holds a candle to their extant glory. Yet the hordes of tourists still walk down the Via dei Forti Imperiali and congregate in stupidly long lines at the ticket booth to see ruins!

I walked across the city from east to west, passing by a secondary school, flea market, and various patisseries (is that the correct wording?). The pastries were incredible. The flea market reminded me of Mexico, interestingly enough. Felt very Catholic.

(All the buses and trams run late in Rome. This too, is very Catholic, as Orwell picked up on during his time in Catalonia and as anyone visiting a Mexican house would know. Plausibly also Irish?)

Rome’s ivies pervade its structures. Villas, monuments, churches (all 900 of them), and fountains all fall victim to these creepers. It gives the perception of a ruined city, that Roman glory has come and gone—and when one is aware of Italian history, it is very, very hard to perceive Rome as anything else than an overgrown still-surviving bastion against the continuing spirit of the Vandals.

Roman pines, too, are fungiform. Respighi’s tone poem doesn’t do justice to them. Perhaps this is just a Mediterranean vibe? But amongst the monumental Classical, Romanesque, and Neoclassical structures of the Piazza Venezia, these pines are punctual. Don’t really know how else to convey it.

It is difficult to comprehend how the animalistic, gladitorial Roman society became the s...
Geneva
by Yudhister Kumar — 2025-06-20T09:22:19.832Z
Link: https://yudhister.me/geneva

Published on September 13, 2023.

Geneva is evil.

It's overpriced, loud, and dirty. Paying ten francs for a medicore street taco is no way to live life. God forbid you visit the city center during the day, and stay as far away from Geneva station as you can. I thought the air was supposed to be good in the Alps?

But above all, it reeks of fakeness.

It calls itself the "Peace Capital", claims it's too good to have twin cities, and prides itself on its cosmopolitanism. On what grounds? Before Hitler's fall, Geneva's only claim to facilitating international diplomacy was hosting the League of Nations -- admittedly the best international governing body we've had thus far, but still. After, every international organization and their shadow backers clamored to have their headquarters (or at least their European headquarters) in Geneva. The UN, WHO, UNHCR, Red Cross, WTO, WIPO, WMO, ILO, ...

Did you know that the largest non-financial services industry in Geneva is watchmaking? Rolex, Patek Philippe, etc. have factories just outside of Geneva proper. To be fair, 'financial services' also excludes commodity trading, of which Geneva is to oil, sugar, grains, and coffee as Rotterdam is to metals. Vitol & Trafigura both have their headquarters in Geneva (and one must wonder whether or not this is for convenience or to take advantage of lax Swiss banking laws...remember Marc Rich?)

Two-thirds of the corporate tax in Geneva comes from commodity trading, banking, and watchmaking. These international organizations? Don't contribute to the economy. (Yes, they bring people & these people use services & this allows Geneva natives to benefit from the overwhelming amount of NGOs and international bodies in their city. Still.)

Tragically, Geneva once had a soul. The 'Protestant Rome' which once served as the birthplace of the Calvinist Revolution was annexed by Catholic France & revolted as a response. The city had opinions that informed its identity -- not a pseudo-identity formed from undeserved arrogance & globalism.

Demographic shifts (mostly French immigration to French-speaking Switzerland) led to Catholics forming the largest religious group in Geneva today, followed by atheists. (I am not blaming immigration for Geneva's soullessness! it is just another piece of the puzzle). This, along with its absurd emphasis on being a truly international city, undergird th...