trentmkelly/LessWrong-43k huggingface dataset:
The 80/20 playbook for mitigating AI scheming in 2025
by Charbel-Raphaël — 2025-05-31T21:17:44.304Z

Adapted from this twitter thread. See this as a quick take.

Mitigation Strategies

How to mitigate Scheming?

Architectural choices: ex-ante mitigation
Control systems: post-hoc containment
White box techniques: post-hoc detection
Black box techniques
Avoiding sandbagging

We can combine all of those mitigation via defense-in-depth system (like the Swiss Cheese model below)

I think that applying all of those strategies should divide the risk by at least 3.

1. Architectural Choices: Ex-ante Mitigation

Don't choose an architecture that uses neuraleses (i.e. hidden reasoning).

For example, don't train models to think in their latent spaces like Meta!

Chain of thought monitoring is a blessing for transparency and monitoring - let's use it as much as we can!

See: Why it’s good for AI reasoning to be legible and faithful.

Similarly, I think text diffusion models are a bad idea for transparency. This is because you can no longer monitor the thought process, and the way those AIs produce outputs is much more alien than autoregressive LLMs.

Also, quite importantly, companies should never use the most forbidden technique!

The Most Forbidden Technique is training an AI using interpretability techniques.

An AI produces a final output [X] via some method [M]. You can analyze [M] using technique [T], to learn what the AI is up to. You could train on that. Never do that.

You train on [X]. Only [X]. Never [M], never [T].

Why? Because [T] is how you figure out when the model is misbehaving.

If you train on [T], you are training the AI to obfuscate its thinking, and defeat [T]. You will rapidly lose your ability to know what is going on, in exactly the ways you most need to know what is going on.

Those bits of optimization pressure from [T] are precious. Use them wisely.

2. Control Systems: Post-hoc Containment

Even if AI is scheming, you can control it, and some control measures are much better than other ones. This paper is SOTA.

It might be the case that AIs in the future communicate secretly with steganography, by encoding hidden message in its chain of thought.

But fortunately, it seems like paraphrasing is an effective defense against current steganography schemes.

See: Preventing Language Models from hiding their reasoning.

3. White Box Techniques: Post-hoc Detection

For example, Anthropic calculated the Supervised Auto-Encoders here, and were able to find the hidden goal by looking at the feat...
The best approaches for mitigating "the intelligence curse" (or gradual disempowerment); my quick guesses at the best object-level interventions
by ryan_greenblatt — 2025-05-31T18:20:43.710Z

There have recently been various proposals for mitigations to "the intelligence curse" or "gradual disempowerment"—concerns that most humans would end up disempowered (or even dying) because their labor is no longer valuable. I'm currently skeptical that the typically highlighted prioritization and interventions are best and I have some alternative proposals for relatively targeted/differential interventions which I think would be more leveraged (as in, the payoff is higher relative to the difficulty of achieving them).

It's worth noting I doubt that these threats would result in huge casualty counts (due to e.g. starvation) or disempowerment of all humans (though substantial concentration of power among a smaller group of humans seems quite plausible).[1] I decided to put a bit of time into writing up my thoughts out of general cooperativeness (e.g., I would want someone in a symmetric position to do the same).

(This was a timeboxed effort of ~1.5 hr, so apologies if it is somewhat poorly articulated or otherwise bad. Correspondingly, this post is substantially lower effort than my typical post.)

My top 3 preferred interventions focused on these concerns are:

Mandatory interoperability for alignment and fine-tuning: Pass regulation or create a norm that requires AI companies to support all the APIs and interfaces needed to customize their models and (attempt to) align them differently. Either third parties would inspect the implementation (to avoid tampering and to ensure sufficient affordances) or perhaps more robustly, the companies would be required to submit their weights to various (secure) third parties that would implement the relevant APIs. Then, many actors could compete in offering differently fine-tuned models competing over the level of alignment (and the level of alignment to users in particular). This would be using relatively deep model access (not just prompting), e.g. full weight fine-tuning APIs that support arbitrary forward and backward, per-token losses, adding new heads/probes, and more generally whatever access is needed for alignment methods. (Things like (e.g.) steering vectors could be supported, but currently wouldn’t be important as they aren’t the state of the art for typical usage.) The hope here would be to get the reductions in concentration of power that come from open source while simultaneously b

...
Would It Be Better to Dispense with Good and Evil?
by arusarda — 2025-05-31T16:40:01.472Z

The Instrumental Value of Good and Evil

The naturalist, realist project of morality is one that aims to define what is right and wrong by appealing to a set of objective moral truths, whether that be through the maximization of pleasure or through the categorical imperative. However, this essay will argue these theories fall short insofar as they fail to establish the existence of objective mind-independent moral properties. Nevertheless, completely embracing an error theory of morality likely leads to complete moral abolitionism which, for the pragmatist, is not an appealing reality. Therefore, this essay explores alternatives, such as Smith’s constitutivism and Joyce’s fictionalism, to fill the nihilistic void left by the anti-realist. While it is better to dispense of the metaphysically objective good and evil, the counterfactual must be one that embraces a fictional view of morality.

Issues Pertaining to Moral Realism

Moral realism faces insurmountable challenges in establishing objective moral facts independent of human attitudes and institutions. J.L. Mackie's "argument from queerness" identifies the core problem. Moral properties would be metaphysically peculiar entities unlike anything else in our ontology. They would need inherent motivational force, somehow bridging Hume's is-ought gap.

Consider utilitarian attempts to ground morality in pleasure maximization. Hedonistic states are real psychological phenomena, but the claim that we ought to maximize them requires an unjustified normative leap. Utilitarians must posit that pleasure possesses intrinsic "to-be-pursuedness" independent of our attitudes. Yet such properties appear nowhere else in scientific understanding. Neurochemical processes underlying pleasure represent evolved behavioral reinforcement mechanisms, not objective moral significance.

Kantian deontology attempts to derive moral obligations from rational agency itself. The categorical imperative generates universal moral laws through pure practical reason, independent of contingent desires. However, this faces the normative authority problem. Even granting logical inconsistencies in universalized maxims, why should consistency constitute moral obligation rather than mere instrumental rationality? Kantians claim rational agents are necessarily committed to consistency, but this fails to explain why logical commitment carries moral rathe...
How Epistemic Collapse Looks from Inside
by Martin Sustrik — 2025-05-31T16:30:06.364Z

There’s a story — I'm at a conference and cannot access my library, but I believe it comes from Structural Anthropology by Claude Lévi-Strauss — about an anthropologist who took a member of an Amazonian tribe to New York.

The man was overwhelmed by the city, but there were two things that particularly interested him.

One was the bearded woman he saw at a freak show that the anthropologist took him to.

The other was the stair carpet rods in the hotel.

Epistemic collapse does not have to be caused by moving from a simpler society to a more complex one.

It can also result from gradually increasing complexity of your native one. The complexity rises until the point where the subject can no longer make sense of the world. They may have seemed to be a reasonable person for a long time, but suddenly they started believing in chemtrails.

Very much like the tribesman from the first story, they can no longer comprehend the world around them. The cognitive apparatus is freewheeling, clutching at random facts, in this case innocuous condensation trails in the sky, and trying to transform them into a coherent narrative about the world.

Imagine a chimpanzee who somehow ends up in a big city. He finds a park and survives there for a couple of days. His mental model includes, among other things, people. Some of them are kind and feed him, while others have dogs and they are best avoided.

That seems important, but his mental model does not account for what truly matters: the municipal department of animal control.

He will eventually be caught and removed from the park because he’s considered a health threat. But it’s questionable whether he even has a concept of "being a threat," let alone that of a "health threat." Understanding the concept of "health threat" requires awareness of microorganisms, which the chimpanzee lacks.

What’s worse, whether he ends up in a zoo or in an industrial shredder is of utmost importance to him, but the reasons behind why his destiny takes one path or another are far beyond his comprehension.

It may be useful to sometimes think about what your equivalent of a bearded woman is, but it's not clear to me whether that’s a question you can ever truly answer.
When will AI automate all mental work, and how fast?
by aggliu — 2025-05-31T16:18:12.942Z
Link: https://youtu.be/-ffmwR9PPVM

Rational Animations takes a look at Tom Davidson's Takeoff Speeds model (https://takeoffspeeds.com). The model uses formulas from economics to answer two questions: how long do we have until AI automates 100% of human cognitive labor, and how fast will that transition happen? The primary scriptwriter was Allen Liu (the first author of this post), with feedback from the second author (Writer), other members of the Rational Animations team, and external reviewers. Production credits are at the end of the video. You can find the script of the video below.

How long do we have until AI will be able to take over the world? AI technology is hurtling forward. We’ve previously argued that a day will come when AI becomes powerful enough to take over from humanity if it wanted to, and by then we’d better be sure that it doesn’t want to. So if this is true, how much time do we have, and how can we tell?

AI takeover is hard to predict because, well, it’s never happened before, but we can compare AI takeover to other major global shifts in the past. The rise of human intelligence is one such shift; we’ve previously talked about work by researcher Ajeya Cotra, which tries to forecast AI by considering various analogies to biology. To estimate how much computation might be needed to make human level AI, it might be useful to first estimate how much computation went into making your own brain. Another good example of a major global shift, might be the industrial revolution: steam power changed the world by automating much of physical labor, and AI might change the world by automating cognitive labor. So, we can borrow models of automation from economics to help forecast the future of AI.

AI impact researcher Tom Davidson, in a report published in June 2023, used a mathematical model derived from economics principles to estimate when AI will be able to automate 100% of human labor. You can visit “Takeoffspeeds.com” if you want to play around with the model yourself. Let’s dive into the questions this model is meant to answer, how the model works, and what this all means for the future of AI.

Davidson’s model is meant to predict two related ideas: AI timelines and AI takeoff speed. AI timelines have to do with exactly when AI will reach certain milestones, in this model’s case automating a specific percenta...
Progress links and short notes, 2025-05-31: RPI fellowship deadline tomorrow, Edge Esmeralda next week, and more
by jasoncrawford — 2025-05-31T15:20:53.661Z
Link: https://newsletter.rootsofprogress.org/p/links-and-short-notes-2025-05-31

It’s been way too long since the last links digest, which means I have way too much to catch up on. I had to cut many interesting bits to get this one out the door.

Much of this content originated on social media. To follow news and announcements in a more timely fashion, follow me on

Twitter

,

Notes

,

Farcaster

,

Bluesky

, or

Threads

.

Contents

Apply to the Roots of Progress Fellowship by June 1st (tomorrow!)
Edge Esmeralda next week!
My writing (ICYMI)
Other people’s writing
Jobs
Grants & fellowships
Events
AI announcements
Introductions
Career moves
Nuclear news
Aviation news
Other announcements

For paid subscribers:

Stagnation was the goal
Is stagnation a measurement illusion?
Eroom’s Law
Cembalest on AI
More on AI
Bio
Podcast interviews
Links and short notes
Politics
Housing
Gratitude
Quotes
Charts
Aesthetics
Fun

Apply to the Roots of Progress Fellowship by June 1st (tomorrow!)

Applications are still open

for the

2025 Blog-Building Intensive

! Launch a blog and improve your progress-focused writing with expert guidance and an amazing community progress builders, writers and intellectuals.

In addition to a general focus on progress studies, this year’s fellowship features two themes: (1) agriculture and (2) health, biotech & longevity. We welcome fellows writing on any progress-related topic, but for a handful of spots, we will give preference to applicants focusing on these themes, for which there will be dedicated programming.

But don’t take our word for it, see what others have to say:

@NikoMcCarty

: I can't recommend this Writers' Fellowship enough. It helped me find my community, challenge my own work, and improve very quickly. You should apply! And feel free to DM me directly if you have any questions about my experience in the program.

@gtmulligan

: This program changed my life. Happy to talk with anyone about my experience. Apply, apply, apply! [See also Grant’s

post on the fellowship

and his

thread of favorite pieces from the fellows

]

@RosieCampbell

: This was so well-run and it's a fantastic community, I am very grateful I got to participate. Highly recommend applying if you're interested in writing on the internet!

@snewmanpv

: I had the privilege of participating in this program last year. Highly recommend for anyone writing about progress-related topics. Come for the information-dense instructional sessions, stay for the commun

...
House Party Dances
by jefftk — 2025-05-31T15:20:02.879Z

Julia and I are in CA for

LessOnline this weekend, and we're
staying with friends. It happened that they were hosting a party, and
while this is not a group of dancing friends they asked if I'd be up
for leading some dancing. One of my hosts let me borrow their violin
(which I'll also have with me today at LessOnline if anyone would like
to jam some), and the space was small enough not to need any
amplification.

Playing and calling at the same time is only something I can do if I
play simple tunes I know very well while calling simple dances, and
both my playing and calling suffer from the multitasking. But my hosts
encouraged me not to worry about this and thought we'd have a good
time. I think they were right!

We ended up having about seven couples, mostly adults, but also
including about three kids 3y-5y who were very excited to dance. I
made sure to choose dances that would work for the whole group, and it
was definitely the right call to include the kids: their excitement
was highly infectious.

Our hosts had moved furniture earlier in the day, clearing an open
area about 15 ft square. This ended up being slightly tight for a
few figures, but was mostly pretty good.

The crowd was enthusiastic about dancing, and while not everyone
danced we easily had enough people to make the dances work. We ended
up doing (I didn't write this down at the time; this is approximately
right):

Longways: The Low-backed Car

Longways: Bridge of
Athelone

Break for cake

Waltz

Longways: Jacob's Potato

Scatter mixer: Sasha

Another break, pretty long

Circle: La Bastringue

Waltz (Julia singing Waltzing With
Bears while I fiddled chords)

Longways: Galopede

I think Sasha was the most popular, followed by the set dancing
(Longways and Circle), followed by waltzes. It makes sense that the
waltzes were less popular: Julia and I taught the basic folk waltz
step, but getting into the improvisational lead-follow dynamic and the
kinds of figures you might want to lead would have needed much more
time and have worse ROI. I'm not sure if I would include these if
doing this again, though it was nice to have a few slower-paced
dances.

I'm glad I was able to help make this happen!

Comment via: facebook, mastodon, bluesky, substack
Free Will, Like Probability, is About Local Knowledge
by Rob Lucas — 2025-05-31T14:19:26.160Z
Link: https://open.substack.com/pub/lifeinthelabyrinth/p/free-will-like-probability-is-about?r=1i6iuu&utm_medium=ios

This post is also found on my new substack page. I think it's a view of free will that might be quite interesting to the LessWrong community, both for its clarity and originality, and for its coherence within the general rationalist framework, particularly a bayesian viewpoint.

The Time Traveller: A short narrative introduction

I know you don’t know me, but thank you: you saved my wife’s life.

You see, I was trapped. After a while I gave up hope. I started to feel my situation was inescapable. I couldn’t change it. It was set in stone and no matter what I did events would somehow conspire to arrange themselves as they had to be. I was foiled. Time and again, no matter how ridiculous, how preposterous the unfolding of events required; somehow time would always repeat as I remembered it.

Three years from now and five years ago I got a phone call. I was working in my lab, on the time machine, when they called and told me my wife had died in a car accident. Her heart stopped on the way to the hospital.

I’d tried a thousand different ways to prevent it, but time always had a way of ensuring that the future I came from was the future that transpired.

It was looking at you, that day in the coffee shop, that I had my epiphany.

I was sitting across from you feeling sorry for myself. Trapped in my fate, I found myself wishing that I were in your shoes, for you this was the present, and the future lay unknown ahead of you, free to change at your slightest whim. Whereas for me, this was the past, and what lay ahead was more of it, more bygone years all set in stone and unchangeable.

If only I could put on your shoes.

And so in my mind’s eye, I did. I sat there looking at the carefree expression on your face, imagining myself not knowing what was to come; trying to change it.

I imagined myself sitting in your place, deciding about my future. I imagined how you would go home tonight to your family, if you had one. I imagined going home to mine.

But the thought died in my mind like a vine withering in the sun. That was it. I couldn’t. Because no matter what happened if I went to my wife now she would still be crushed by that car in three years. Even if I were you, even if the “past” became the future again, it still stood, solid and unbreakable. You were in the same position as...
The (Unofficial) Rationality: A-Z Anki Deck
by japancolorado — 2025-05-31T07:01:45.156Z

I'm a huge fan of Anki. I love being able to remember what I read for however long I want to. However, most Anki decks I've found about rationality are nonexistent, unhelpful, out of date, or poorly formatted.

Recently, I found this post from 2023 that used GPT-4 to make a Rationality: From AI to Zombies deck, but I found the cards so long, obtuse, and unusable that it made me want to do better.

After dozens of hours making a custom Python package to generate cards with GPT-4o-mini, followed by having Claude Sonnet 4 revise, reduce, and reconsolidate the cards, I am proud to present:

The (Unofficial) Rationality: From AI to Zombies Deck, available from AnkiWeb or Google Drive.

I definitely spent way longer than I needed to in making this, but I'm really happy with the result. I think it's a definite improvement over existing shared decks, and I hope it can be of benefit to others too.

I also enjoyed making my Articles to Anki python package, which automatically generates and exports Anki cards from URLs or local files. There are definitely still some improvements to be made[1], but it is functional, useful, and developing it did a lot to bring me up to speed with coding with LLM assistance.

If this is well received or I find it personally useful, I'll probably make some more decks for rationalist content. Let me know if you have any requests or feedback!

^

Help is very much welcome. It was mostly Sonnet 4 and I flying by the seat of our pants.
Zochi Publishes A* Paper
by mannatvjain — 2025-05-31T00:00:27.328Z
Link: https://www.intology.ai/blog/zochi-acl

Zochi Achieves Main Conference Acceptance at ACL 2025

Today, we’re excited to announce a groundbreaking milestone: Zochi, Intology’s Artificial Scientist, has become the first AI system to independently pass peer review at an A* scientific conference¹—the highest bar for scientific work in the field.

Zochi’s paper has been accepted into the main proceedings of ACL—the world’s #1 scientific venue for natural language processing (NLP), and among the top 40 of all scientific venues globally.²

While recent months have seen several groups, including our own, demonstrate AI-generated contributions at workshop venues, having a paper accepted to the main proceedings of a top-tier scientific conference represents clearing a significantly higher bar. While workshops³, at the level submitted to ICLR 2025, have acceptance rates of ~60-70%, main conference proceedings at conferences such as ACL (NeurIPS, ICML, ICLR, CVPR, etc…) have acceptance rates of ~20%. ACL is often the most selective of these conferences

This achievement marks a watershed moment in the evolution of innovation. For the first time, an artificial system has independently produced a scientific discovery and published it at the level of the field’s top researchers—making Zochi the first PhD-level agent. The peer review process for the main conference proceedings of such venues is designed to be highly selective, with stringent standards for novelty, technical depth, and experimental rigor. To put this achievement in perspective, most PhD students in computer science spend several years before publishing at a venue of this stature. AI has crossed a threshold of scientific creativity that allows for contributions alongside these researchers at the highest level of inquiry.

Autonomously Conducting the Scientific Method

Zochi is an AI research agent capable of autonomously completing the entire scientific process—from literature analysis to peer-reviewed publication. The system operates through a multi-stage pipeline designed to emulate the scientific method. Zochi begins by ingesting and analyzing thousands of research papers to identify promising directions within a given domain. Its retrieval system identifies key contributions, methodologies, limitations, and emerging patterns across the literature. What distinguishes Zochi is its ability to identify non-obvious connections across papers and propose innovativ...