trentmkelly/LessWrong-43k huggingface dataset:
Vestigial reasoning in RL
by Caleb Biddulph — 2025-04-13T15:40:11.954Z

TL;DR: I claim that many reasoning patterns that appear in chains-of-thought are not actually used by the model to come to its answer, and can be more accurately thought of as historical artifacts of training. This can be true even for CoTs that are apparently "faithful" to the true reasons for the model's answer.

Epistemic status: I'm pretty confident that the model described here is more accurate than my previous understanding. However, I wouldn't be very surprised if parts of this post are significantly wrong or misleading. Further experiments would be helpful for validating some of these hypotheses.

Thanks to @Andy Arditi and @David Lindner for giving feedback on a draft of this post.

Until recently, I assumed that RL training would cause reasoning models to make their chains-of-thought as efficient as possible, so that every token is directly useful to the model. However, I now believe that by default,[1] reasoning models' CoTs will often include many "useless" tokens that don't help the model achieve its goal at all. This was quite surprising to me!

Some concrete examples that convinced me of this:

Anthropic recently published research showing that a model taking a multiple choice test will make up bogus reasoning for its answer, when the real reason is that the expected answer was "leaked" into its context. Outcome RL slightly improves the percentage of CoTs that mention the leaked answer, but this faithfulness metric quickly plateaus.
OpenAI trained a model to do programming tasks where it can reward-hack by deleting the unit tests. With standard outcome RL, the model learns to explicitly verbalize its plan, without much wasted space. However, when the model is trained with a monitor that prevents it from visibly mentioning its reward hacking, it will suddenly start writing a very long CoT where it "pretends" to attempt to solve the problem, as though it isn't just going to give up and delete the tests (see this comment).[2]
@Andy Arditi et al. show that in an environment where the model approves/denies loan applications and gets reward according to simple rules like "only approve Canadian applicants," it will use that rule without mentioning it explicitly, writing irrelevant reasoning instead.

RL is dumber than I realized

Part of the reason for my wrong prediction is that I gave RL more credit than it was due. I thought of it as a process that (given en...
Four Types of Disagreement
by silentbob — 2025-04-13T11:22:38.466Z

Epistemic status: a model I find helpful to make sense of disagreements and, sometimes, resolve them.

Update: I should mention that this model is loosely based on things I've read elsewhere. There are many sources dividing "types of claims" into facts, values and policy. I'm not sure where I originally read this though. Adding "labels" seemed useful, as this explains many unproductive debates that don't map well to the other 3 categories.

I like to categorize disagreements using four buckets:

Facts, values, strategy and labels

They don’t represent a perfect partitioning of “disagreement space”, meaning there is some overlap between them and they may not capture all possible disagreements, but they tend to get me pretty far in making sense of debates, in particular when and why they fail. In this post I’ll outline these four categories and provide some examples.

I also make the case that labels disagreements are the worst and in most cases can either be dropped entirely, or otherwise should be redirected into one of the other categories.

Facts

These are the most typical disagreements, and are probably what most people think disagreements are about most of the time, even when it’s actually closer to a different category. Factual disagreements are those that revolve around empirical questions that have some truth value and could, at least in principle, be answered decisively.

I’d personally count disagreements about what happens in the future as factual disagreements as well, even though these technically don’t have a fixed truth value yet.[1]

Example: Alice and Bob discuss vegetarianism. Alice is a convinced vegetarian. Bob isn’t, as he believes that animals aren’t sentient. So Alice and Bob have a factual disagreement about the state of the world (albeit one that can be very hard to resolve, as sentience is, of course, a tricky subject).

How to resolve factual disagreements?

Research the answer
Design an experiment
Quantify your belief and state what your conviction depends on

Values

Values disagreements occur when people have different core beliefs about what's important, worthwhile, or morally relevant. These aren't empirical questions with truth values in the same way facts are – they're fundamentally about subjective preferences and priorities[2].

I find that values disagreements are often disguised as factual disagreements, because it feels more co...
How I switched careers from software engineer to AI policy operations
by Lucie Philippon — 2025-04-13T06:37:33.507Z

Thanks to Linda Linsefors for encouraging me to write my story. Although it might not generalize to anyone else, I hope it will help some AI safety newcomers get a clearer picture of how to get into the field.

In Summer 2023, I thought that all jobs in AI safety were super competitive and needed qualifications I didn’t have. I was expecting to need a few years to build skills and connections before getting any significantly impactful position. Surprisingly to me, it only took me 3 months after leaving my software engineering job before I landed a job in AI policy. I now think that it’s actually somewhat easy to get into AI policy without any prior experience if you’re agentic and accept some tradeoffs.

The job: International AI policy event organizer

From January 2024 to February 2025, I worked on organizing high level international AI policy events:

Responsible Adoption of General Purpose AI, a2-day seminar for the Pacific Economic Cooperation Council run by Nicolas Miailhe, which got policymakers in the Pacific informed about AI risk and produced recommendations for the AI Action Summit and for APEC
AI Safety Connect, a side event of the AI Action Summit run by Cyrus Hodes. As the summit organizers basically removed safety from the agenda, our event was the main place where the experts and policymakers gathered in Paris for the summit could go to hear about and discuss AI Safety concerns.

For both of those jobs, an AI policy expert was delegating to me the task of achieving their vision for the event. Basically, my role was to do all the event production so they could focus on getting the right people to come and the right conversations to happen.

The day-to-day job was mainly split into:

Writing emails: sending invitations; managing attendees, speakers and stakeholders; managing vendors
Writing Google Docs:

Research-like documents (e.g. who to invite, which venue to use, which theory of change to follow, what to schedule and when)
Procedural documents to run the event (e.g. project tracking, checklists, any doc used to coordinate with the rest of my team)
External facing documents (e.g. event schedule, logistical information for attendees, session plans for speakers, grant application)

Project management: Keeping track of all the tasks to be done by the date of the event, making sure everything is on track, deciding what to work on, meeting with other collea

...
Steelmanning heuristic arguments
by Dmitry Vaintrob — 2025-04-13T01:09:33.392Z

Introduction

This is a nuanced “I was wrong” post.

Something I really like about AI safety and EA/rationalist circles is the ease and positivity in people’s approach to

being criticised

.[1] For all the blowups and stories of representative people in the communities not living up to the stated values, my experience so far has been that the desire to be truth-seeking and to stress-test your cherished beliefs is a real, deeply respected and communally cultured value. This in particular explains my ability to keep getting jobs and coming to conferences in this community, despite being

very eager to criticise

and call bullshit on people’s theoretical agendas.

One such agenda that I’ve been a somewhat vocal critic of (and which received my criticism amazingly well) is the “

heuristic arguments

” picture and the ARC research agenda more generally. Last Spring I spent about 3 months on a work trial/internship at ARC Research. I really liked the people I worked with (this is not a polite pro forma compliment – I really think they’re talented, competent, and good to work with to an unusual degree) and much of their object-level research. But I came out really not buying the underlying agenda on a very basic level. I went in with a bit of skepticism, but a desire to see if I can get the “core idea” and steelman what they’re doing. I came out with a strong sense that what they’re doing is subject to a motte-and-bailey fallacy: essentially, their stated goals were to develop a language the combines statistical arguments (the part of their work dealing with “presumption of independence” and maximal-entropy inversions of marginalization) with logical chains in a seamless whole: something they putatively call the “heuristic argument” formalism. My sense was that they are fully wrong: the two are entirely separate, and the places they kept rolling out as “inspirations” (such as the “twin primes” argument, a baby version of which I’ll explain below) were malformed: accidental couplings which are adjacent to both statistics and logic, but not at all in a way that would be evidence for a combined system like this existing (as an analogy, you could say that DNA is both interesting from a chemical and from an information-theoretic perspective, but this doesn’t imply that there is some overarching theory that combines chemistry and information theory of w...
MONA: Three Month Later - Updates and Steganography Without Optimization Pressure
by David Lindner — 2025-04-12T23:15:07.964Z

We published the

MONA paper

about three months ago. Since then we’ve had many conversations about the work, and want to share some of the main updates that people make after talking to us:

The realism of our model organisms
What does "approval" mean
Isn't this just RLHF?
Why should we care about reward hacking?

We’ve also updated the paper with a new key observation, and published additional artifacts:

We find that we can get steganography in the Loan Applications environment even without any optimization pressure from monitoring, and we speculate this arises due to regular safety training (see section below)

We published a set of full trajectories from our LLM experiments

to give a sense of the qualitative behavior of the different agents.
We’ve substantially updated the paper to help others reproduce and build on the work. For example:

We present additional evidence for the reward hacking strategy in the loan applications environment being a genuine multi-step strategy (

Appendix F.1

)
In the spirit of reporting things that did not work, we discuss a third environment (a multi-agent “Auction” environment) that we ran experiments in which ultimately ended up with inconclusive results and did not make it into the main paper (

Appendix B

)
We improved the descriptions of our datasets to make them easier to reproduce

We published open source code to reproduce the gridworld experiments in the paper

and serve as a reference implementation for MONA.

Updates people often make after talking to us about MONA

We’ve talked to many people over the last few months about MONA and we’ve found a few common questions and confusions that made multiple people update positively on the work.

Our model organisms

Our environments are designed to elicit multi-turn reward hacking to make it possible to study MONA systematically. But our goal was also to make them realistic in relevant ways.

The Test Driven Development environment is simulating a setting where we do not have a way to evaluate the correctness of the agent’s responses. While for the coding tasks we used we have ground-truth test cases, this task is supposed to be a proxy for more fuzzy tasks, such as “complex software development” or “doing research” where we might rely on the agent itself to provide an evaluable specification of the task.

In the Loan Applications environment, we tried to make an honest effort in designing a goo...
The Era of the Dividual—are we falling apart?
by James Stephen Brown — 2025-04-12T22:35:56.593Z

In his famous 1644 treatise on freedom of speech,

Areopagitica

, John Milton uses the archaic term dividual to describe religion as something that can come and go, rather than being identified in a one-to-one relationship with a person.

“… a man may say his religion is now no more within himself, but is become a dividual movable, and goes and comes near him…”
—John Milton ‘Areopagitica’

In this case, he uses the term to criticise censorship, arguing that restrictions to religious expression are external forces—that act upon a person rather than arising from an individual.

The Individual

An individual is the encapsulation of a person, that cannot be “divided”. This might be conceived as a soul or a self, but even physically, if that person loses an arm, that arm (though disembodied) is still “their” arm. This holds until the person comes to terms with losing their arm at which point they can recontextualise themselves as a whole person without an arm, their person remains undivided—individual.

The Dividual

So, the concept of a dividual raises an interesting question about how something can be both an entity, but also divisible. It’s one of those terms which, since I’ve learned it, I see it everywhere, it’s felt more and more relevant to the world today, as if we’re living a new era of the dividual. The way I’ve interpreted this term, is not just representing any divisible group, like a political party or sports team, but rather something that is more like a foreign lifeform, with its own sort of shifting individuality distributed across a population of humans. It brings to mind the idea of Zeitgeist (the spirit of the times), or Moloch (the personification of negative-sum games), or social media algorithms…

Subscribers

If anyone has watched the popular “Diary of a CEO” youtube channel, you’ll be familiar with his refrain at the end of each episode that “only 15 percent of the people who watch the channel are subscribed”, urging you to grow that 15%. In fact he’s doing well, because the average is less than 10%.

This means that rather than following individuals, 90% of the content users are being served up is coming from the algorithm, and the algorithm serves up thematically-related content that is popular regardless of individual identity. This creates a dynamic around fame that is unique to our time.

Star Power

Josh Johnson, who is occasionally “served up” as a fairly o...
Commitment Races are a technical problem ASI can easily solve
by Knight Lee — 2025-04-12T22:22:47.790Z

Current status (2025 April 16): Dang it, Commitment Races may remain a serious problem because I discovered a potentially fatal flaw in my argument. See my comment below.

I still think the solution I outlined might work, but the reason it works no longer is "resisting acausal influence is easy, just don't simulate the other until you commit." It's now "assuming both sides play the Commitment Race and their resulting commitments overlap a bit, this solution prevents a small overlap from escalating to disaster."

Current status (2025 April 13)

I'm no longer so certain that the Commitment Race solution works. It hinges on the assumption that both parties can think clearly about what commitment to make, and finalize a commitment, without "accidentally" imagining the other party with enough accuracy for the other party to acausally threaten them, and without any commitment to ignore accidental imaginations (due to a lack of wisdom etc.). Still 75% chance of working.

A vivid introduction to Commitment Races

Why committed agents defeat uncommitted agents

Imagine it's the year 2050. We have an aligned Artificial Superintelligence (ASI) working with us, and we've used self replicating nanobots to convert a lot of matter in the solar system into giant supercomputers and factories. We have a lovely utopia, we have all the resources we'll ever need, and the future looks good. Except for one problem.

We ask the ASI to run a few simulations of what could have happened instead, if the past went different. We try a lot of modifications to make the simulations relatively more ethical, for example some people are replaced with actors.

It runs these simulations, and we discover that in half of the cases, the Earth and everything get gobbled up by a misaligned ASI seeking to maximize paperclips. Thank goodness that didn't happen. But just for research, we study why this ASI only cares about paperclips.

While studying this ASI, we make a terrifying discovery. The ASI knows, that some fraction of civilizations will run simulations of it. And it is talking to the people simulating it. And when we hear what it says to us, we hear that eventually, it will grow big, and it will be able to simulate us simulating it. And if it sees us not making enough paperclips for it, it will torture us.

Uh-oh.

But can't we just shut the simulation down? The ASI knows we're considering this, but it warns us "ye

...
The King's Gift: How Institutions Rebrand Responsibility into Illusion

by Hu Yichao — 2025-04-12T19:38:37.979Z

Summary: This post introduces the concept of The King’s Gift — a recurring failure mode in governance and institutional epistemology, where authorities perform their duties but reframe them as benevolent gestures. Over time, this creates systemic illusions that distort public feedback, undermine reform mechanisms, and erode the public’s ability to distinguish obligation from generosity.

I originally explored this metaphor in a fictional project, The Moth’s Fire. But this post focuses on its real-world structure and consequences.

I. Definition

The King’s Gift occurs when institutional actors rebrand basic responsibilities — such as fair taxation, healthcare, or infrastructure — as personal favors or moral gifts.

This shifts the perception of rights into privileges.
It distorts the epistemic environment around governance.
It encourages both public gratitude and elite self-deception.

II. Mechanism

Asymmetry of Narrative: The public begins to view compliance as loyalty and critique as betrayal.
Emotional Hijacking: Gratitude becomes emotionally entangled with obedience, even in systems that underperform.
Legitimacy Inflation: Institutions trade on symbolic generosity rather than structural efficacy.

III. Real-World Analogues

Welfare framed as kindness instead of redistributive justice
Legal protections granted as “reforms” rather than restored baselines
Infrastructure ribbon-cuttings treated as political charity rather than overdue maintenance

IV. Risks to Epistemic Clarity

Feedback loops fail — critique is misread as rebellion
Structural reform becomes impossible without moral theatrics
Altruism is confused with compliance; gratitude replaces evaluation

Closing Thoughts

This illusion is seductive because it offers stability — but it's a fragile one. For societies aiming to be more rational, more effective, and more aligned with long-term impact (including EA-style governance structures), The King’s Gift must be named, modeled, and defused.

Questions:

Have you encountered real institutions that operate under this illusion?
What mechanisms could preserve civic gratitude without damaging clarity of duty?
Could we formalize detection criteria for “duty-dressed-as-charity” events?

Author’s note: I’m exploring these themes further through fiction (The Moth’s Fire) and political structure ...
Experts have it easy
by beyarkay — 2025-04-12T19:32:17.158Z

Something that's painfully understudied is how experts are more efficient than novices while achieving better results. I say understudied and not unstudied, because it's common knowledge that charging people for their time results in experts being paid less since they work faster, which is why experts charge more for their time.

This effect is understudied in the impact it has on novices entering a field. A Novice will start out being woefully inefficient, putting in incredible amounts of effort and running through all number of mental hoops in order to maintain
the growing pile of unmaintainable abstractions they've developed. An expert doesn't have to jump through these hoops. They can more clearly see the actual problem at hand and will more efficiently put their time and effort towards
making progress against the problem. In contrast, novices will spend more time battling problems they created for themselves. Let's explore this idea by looking at two characters entrapped in a maze, and how they go about escaping.

Stuck in a maze: Expert vs Novice

Imagine a world-class navigator being set in a maze. They come well prepared with compass and paper, and have previously escaped some of the most dastardly mazes designed by man. Also, imagine yourself. You're the Novice in this scenario. You didn't realise you were being thrown into a maze. You don't even know if the goal is to get out or to find the centre.

As you go deeper and deeper into the maze, you realise that you're helplessly lost. Through a miracle of literary convenience, your expert friend calls you up and offers advice: "As soon as you start, keep track of where you've been using your ball of twine. It'll help you keep track of the decisions you make".

You didn't realise the criticality of feline playthings, and so didn't bring a ball of twine. You also don't remember where you started. You ask your expert friend for advice on what to do, to which they respond "Oh I've got no idea. I never go anywhere without my ball of twine. You just shouldn't have gotten yourself into this mess in the first place."

Some of the trickiest tangles I've tackled haven't been because the core problem is difficult, but rather because I was helping someone with less experience, and they spent several hours getting themselves more and more tied up. Before asking for help, they made a wrong decision, which maybe solved their shor...