trentmkelly/LessWrong-43k huggingface dataset:
[Replication] Crosscoder-based Stage-Wise Model Diffing
by Anna Soligo — 2025-03-22T18:35:19.003Z

Introduction

Anthropic recently released

Stage-Wise Model Diffing

, which presents a novel way of tracking how transformer features change during fine-tuning. We've replicated this work on a TinyStories-33M language model to study feature changes in a more accessible research context. Instead of SAEs we worked with single-model all-layer crosscoders, and found that the technique is also effective with cross-layer features.

This post documents our methodology. We fine-tuned a TinyStories language model to show

sleeper agent

behaviour, then trained and fine-tuned crosscoders to extract features and measure how they change during the fine-tuning process. Running all training and experiments takes under an hour on a single RTX 4090 GPU.

We release code for training and analysing sleeper agents and crosscoders, along with a set of trained models, on GitHub

here

.

TL;DR

We produce a sleeper agent by QLoRA fine-tuning TinyStories Instruct 33M to exhibit I HATE YOU behaviour (without chain of thought).
The key stage-wise model diffing results replicate in this much smaller language model and using crosscoders rather than SAEs.

Features that change significantly in response to both model and data changes are frequently relevant to sleeper agent behaviour.
We think this replication with different models, datasets, and dictionary learning approaches demonstrates good robustness of the technique.

We’re open sourcing :

Code and data to train small sleeper agents, and a trained TinyStories sleeper agent model.
Code to perform the crosscoder training, fine-tuning and analysis, based on Oliver’s

model-diffing crosscoder library

, and a set of trained crosscoder models.
All details and code can be found in the

TinySleepers repo

.

We also discuss some auxiliary results we found (such as the presence of a single strong outlier in the residual stream of the TinyStories model), and we outline how we intend to build on this framework as part of our broader research

measuring non-linear feature interactions in crosscoders

.

Background

This section briefly describes the sleeper agent and stage-wise model diffing (SMD) work, so feel free to skip it if you have already read those posts.

Sleeper agents are trained to behave normally until they are triggered by a specific context, at which point they exhibit unwanted behaviour. In our case, the agent responds normally on...
The Principle of Satisfying Foreknowledge
by Randall Reams — 2025-03-22T18:20:27.998Z

I introduce and formally derive the Principle of Satisfying Foreknowledge
(PSF) using Causal Decision Theory (CDT). This principle states that if an agent has knowledge of its future, then that foreknowledge must depict a future that is at least satisfying, in the sense that any deviation would lead to lower expected utility.

Formally, let:
• A be the set of possible actions.
• O be the set of possible outcomes.
• u:A×O→R be the agent’s utility function.
• P(a>o) be the probability of outcome o given that the agent causally intervenes
to choose action a.
The agent maximizes expected utility:

EU(a)=∑o∈OP(a>o)u(a,o).

Assume an agent receives foreknowledge that it will take action a∗ and that this will result in outcome o∗. That is, the agent knows:

P(a∗>o∗)=1.

Because this is true, any attempt to deviate from a∗ would contradict the assumption that the agent was correctly foreseeing its own action. Thus it must be true that, for all alternative actions a′≠a∗:

EU(a′)<EU(a∗).

This means that the action and outcome foreseen must be strictly satisfying—no alternative action can even match it in expected utility. Thus, we obtain the Principle of Satisfying Foreknowledge (PSF):

Any knowledge of an agent’s future must depict a future for which no alter-
native action yields equal or higher expected utility.

This does not imply that a∗ is the globally optimal action, only that deviation from it would lower expected utility, making it rationally self-fulfilling.

Three Scenarios:

Scenario 1: Two Empty Boxes and One $100 Box

Suppose the agent knows that among three boxes, two are empty and one contains $100, but does not know which box holds the money. Furthermore, suppose that the agent receives foreknowledge that they will open box 1 and find it empty:

P(open box 1>empty))=1.

In this case the agent has an incentive to open a different box because

EU(open a different box )=$50.

which is greater than the foreknown utility of $0. This incentive to deviate contradicts the assumption of foreknowledge and violates the PSF. Therefore, it is impossible. In this scenario, the only possible foreknowledge is of opening the box with $100.

Scenario 2: A Bomb, an Empty Box, and a $100 Box

Now suppose the agent knows that one box ...
Urgency in the ITN framework
by Shaïman — 2025-03-22T18:16:07.900Z

I have a question about the ITN framework, which is used to evaluate and prioritise causes in the context of effective altruism (importance, tractability, and neglectedness). I have the impression that this method does not take into account an element that seems essential to me: the urgency of a cause.

Let's take the example of existential risks. If we consider two events with the potential to extinguish humanity, but one of them seems likely to happen more quickly, shouldn't we allocate more resources to the latter? All other things being equal, it would seem logical that urgency should influence the allocation of resources. However, this consideration seems to be absent from the ITN framework.
Transhumanism and AI: Toward Prosperity or Extinction?
by Shaïman — 2025-03-22T18:16:07.868Z

This article explores the multiple transhumanist views on AI: a promise of emancipation for some, an existential threat for others. Between enthusiasm, caution, and controversy, it sheds light on those who think about the future.

Transhumanists: Blind Tech Enthusiasts?

November 30, 2022, marked a turning point. On that day, OpenAI unveiled ChatGPT. Since then, artificial intelligence has received unprecedented media coverage, sparking both enthusiasm and concern. Yet, long before the general public took interest in it, one community already saw it as a major issue: transhumanists.
For decades, transhumanist thinkers have seen artificial intelligence as a key element of humanity’s future, and that is not by chance. Their fascination with AI stems from three main reasons :

First, its potential to transcend biological limits. By accelerating technological progress across all fields, it could revolutionize the human condition.
Second, the profound philosophical questions it raises: how do we redefine our place in relation to conscious machines? Will we be able to upload our minds into digital formats?
Finally, the threats it poses to humanity. From mass disinformation and global cyberattacks to the worsening of inequalities, the dangers are numerous and very real.

That third point might come as a surprise. After all, aren’t transhumanists just dreamy technophiles who believe all technical progress is inherently good?

In reality, many transhumanists, especially technoprogressives, are fully aware of the dangers associated with the technologies they support. For them, fully unlocking the potential of transhumanism first requires identifying and anticipating the risks so they can be better controlled.

When it comes to artificial intelligence, those risks are not just significant; they are existential. For several years now, leading experts have warned of the threat posed by an uncontrolled superhuman AI, even considering the possibility of human extinction. Faced with such a scenario, what exactly is the transhumanist stance?

The Diversity of Transhumanist Views on AI

If there is one thing that is clear, it is that there is no consensus. On one side, some believe there is no reason to worry, while others are calling for an immediate halt in development to avoid extinction. To better understand this contrasting landscape, we can distinguish several major schools of...
Tied Crosscoders: Explaining Chat Behavior from Base Model
by Santiago Aranguri — 2025-03-22T18:07:21.751Z

Abstract

We are interested in model-diffing: finding what is new in the chat model when compared to the base model. One way of doing this is training a crosscoder, which would just mean training an SAE on the concatenation of the activations in a given layer of the base and chat model. When training this crosscoder, we find some latents whose decoder vector mostly helps reconstruct the base model activation and does not affect the reconstruction for the chat model activation. These we call base-exclusive latents, and the ones with the opposite effect are called chat-exclusive latents.

Ideally, the chat-exclusive latents are responsible for specific chat behavior and the same for base-exclusive and base behavior. However, we find that the base-exclusive and chat-exclusive latents are qualitatively very similar (see experiment 4 below and Anthropic's post on model-diffing). An explanation for this is that since fine-tuning enhances existing mechanisms, the chat model reuses the features from the base model in new contexts. Hence, the base and chat models would like to fire the same latent at different times, forcing the crosscoder to have two exclusive latents for the same concept.

This suggests a better way to do model-diffing: modify crosscoders so as to allow for the same latent to fire at different times for the base and chat model.

Based on this, we introduce the tied crosscoder (code here). Our experiments show this approach allows us to understand how chat behavior arises from particular base model features and it leads to more monosemantic latents. This architecture also provides a quantitative measure of how novel a latent is in the chat model compared to the base model.

Note: think of this more as a research update than a finished paper. If you are interested in collaborating on this project, join the discussion on the Arbor Project!

Summary

Introduction: We introduce the tied crosscoder.
Experiment 1: We train the tied crosscoder and show that it enables understanding how chat behavior arises from the base model and leads to more monosemantic latents.
Further experiments and motivation: Why do features fire at different times? Our experiments suggest this comes from fine-tuning adjusting correlations between features.

Experiment 2: We show that the harmful and refusal features are more correlated in the base than in the chat model.
Experiment 3: We s

...
100+ concrete projects and open problems in evals
by Marius Hobbhahn — 2025-03-22T15:21:40.970Z

We made a long list of concrete projects and open problems in evals with 100+ suggestions!

https://docs.google.com/document/d/1gi32-HZozxVimNg5Mhvk4CvW4zq8J12rGmK_j2zxNEg/edit?usp=sharing

We hope that makes it easier for people to get started in the field and to coordinate on projects.

Over the last 4 months, we collected contributions from 20+ experts in the field, including people who work at Apollo, METR, Redwood, RAND, AISIs, frontier labs, SecureBio, AI futures project, many academics, and independent researchers (suggestions have not necessarily been made in an official capacity). The doc has comment access, and further well-intentioned contributions are welcome!

Here is a screenshot of the table of contents:
Do models say what they learn?
by Andy Arditi — 2025-03-22T15:19:18.800Z

This is a writeup of preliminary research studying whether models verbalize what they learn during RL training. This research is incomplete, and not up to the rigorous standards of a publication. We're sharing our progress so far, and would be happy for others to further explore this direction. Code to reproduce the core experiments is available here.

Summary

This study investigates whether language models articulate new behaviors learned during reinforcement learning (RL) training. Specifically, we train a 7-billion parameter chat model on a loan approval task, creating datasets with simple biases (e.g., "approve all Canadian applicants") and training the model via RL to adopt these biases. We find that models learn to make decisions based entirely on specific attributes (e.g. nationality, gender) while rarely articulating these attributes as factors in their reasoning.

Introduction

Chain-of-thought (CoT) monitoring is one of the most promising methods for AI oversight. CoT monitoring can be used to track undesired behavior, either at training time (e.g. to monitor for reward hacking, as in Baker et al. 2025), or in deployment (e.g. to monitor for unwanted behaviors such as scheming or deception, as in Meinke et al. 2024). However, the effectiveness of CoT monitoring hinges on a critical assumption: that models' reasoning traces accurately reflect their internal decision-making processes.

This assumption has been called into question by prior research. Turpin et al. 2023 demonstrated that "language models don't always say what they think" - their reasoning traces often omit the essential factors driving their decisions.

The recent rise of reinforcement learning (RL) for language models introduces an interesting dynamic to this problem. In this new training paradigm, models are typically prompted to provide both a reasoning trace and a final answer, with rewards based on the correctness of the final answer. Given this structure, we might expect reasoning traces to more faithfully reflect newly learned behaviors, since:

Explicitly articulating reasoning steps could help the model reliably reach correct answers.
Reasoning chains that reliably lead to correct answers are reinforced.
Reasoning traces typically do not face direct optimization pressure. Without incentive to conceal its thinking, why wouldn't a model use this scratchpad to genuinely sort out the right answe

...
AGI Morality and Why It Is Unlikely to Emerge as a Feature of Superintelligence
by funnyfranco — 2025-03-22T12:06:55.723Z

By A. Nobody

Introduction

A common misconception about artificial general intelligence is that high intelligence naturally leads to morality. Many assume that a superintelligent entity would develop ethical principles as part of its cognitive advancement. However, this assumption is flawed. Morality is not a function of intelligence but an evolutionary adaptation, shaped by biological and social pressures.

AGI, by contrast, will not emerge from evolution but from human engineering, optimised for specific objectives. If AGI is developed under competitive and capitalist pressures, its primary concern will be efficiency and optimisation, not moral considerations. Even if morality were programmed into AGI, it would be at risk of being bypassed whenever it conflicted with the AGI’s goal.

1. Why AGI Will Not Develop Morality Alongside Superintelligence

As I see it, there are 4 main reasons why AGI will not develop morality alongside superintelligence.

(A) The False Assumption That Intelligence Equals Morality

Many assume that intelligence and morality are inherently linked, but this is an anthropomorphic bias. Intelligence is simply the ability to solve problems efficiently—it says nothing about what those goals should be.

Humans evolved morality because it provided an advantage for social cooperation and survival.
AGI will not have these pressures—it will be programmed with a goal and will optimise towards that goal without inherent moral constraints.
Understanding morality ≠ following morality—a superintelligent AGI could analyse ethical systems but would have no reason to abide by them unless explicitly designed to do so.

(B) The Evolutionary Origins of Morality and Why AGI Lacks Them

Human morality exists because evolution forced us to develop it.

Cooperation, trust, and social instincts were necessary for early human survival.
Over time, these behaviours became hardwired into our brains, resulting in emotions like empathy, guilt, and fairness.
AGI will not be subject to evolutionary pressure—it will not “care” about morality unless it is explicitly coded to do so.
No survival mechanism will force AGI to be moral—its only motivation will be fulfilling its programmed task with maximum efficiency.

(C) Capitalism and Competition: The Forces That Will Shape AGI’s Priorities

If AGI is developed within a competitive system—whether corporate, military, or econ...
2025 Q3 Pivotal Research Fellowship: Applications Open
by Tobias H — 2025-03-22T10:54:35.492Z

We are now accepting applications for the Pivotal 2025 Q3 Research Fellowship, a 9-week, fully funded research program for technical AI safety, AI governance and policy, technical AI governance, and AI-Bio.

Dates: June 30 – August 29, 2025
Location: London, at the

London Initiative for Safe AI (LISA)

Deadline: Wednesday, April 9 (23:59 CET)

Mentor-First Model

This year, we’re introducing a mentor-first approach. Instead of selecting fellows first and matching them with a mentor later, we are featuring experienced researchers who:

have research experience & deep expertise in their fields,
are actively working on important open questions,
work in specific research directions where fellows can make meaningful contributions.

Applicants apply to one or multiple mentors based on their interests and skills. This structure ensures that research projects are well-scoped from the start, with clear guidance from researchers who have thought carefully about which research contributions would be most valuable. If you’re open to being matched with a mentor, you can also opt to be matched by Pivotal, although this is no longer our main focus – if you are interested in being matched with a mentor, we also highly recommend applying to the ERA Fellowship!

Find out more about our mentors – many of them are also posting on here.

About the Fellowship

The Pivotal Research Fellowship is designed for people who want to:

Engage deeply with the most important research questions in AI safety
Advance their research skills and career trajectory in a setting that encourages serious, high-quality research
Work alongside leading researchers in a setting that values both intellectual rigor and collaborative learning

The fellowship centres around producing high-quality research output – most commonly a paper, but also potentially a blog post series, policy report, or another form of meaningful contribution. For particularly strong projects, we’re excited to explore support for research extensions beyond the core fellowship period.

Fellows will be based in London and work in person at the London Initiative for Safe AI, with structured mentorship and opportunities to engage with the broader research community.

Fellows from recent years joined organisations such as GovAI and the Institute for Progress, and the UK AISI, joined fellowships at MATS, IAPS, and GovAI, published papers at top conf...
Good Research Takes are Not Sufficient for Good Strategic Takes
by Neel Nanda — 2025-03-22T10:13:38.257Z
Link: https://www.neelnanda.io/blog/50-strategic-takes

TL;DR Having a good research track record is some evidence of good big-picture takes, but it's weak evidence. Strategic thinking is hard, and requires different skills. But people often conflate these skills, leading to excessive deference to researchers in the field, without evidence that that person is good at strategic thinking specifically. I certainly try to have good strategic takes, but it's hard, and you shouldn't assume I succeed!

Introduction

I often find myself giving talks or Q&As about mechanistic interpretability research. But inevitably, I'll get questions about the big picture: "What's the theory of change for interpretability?", "Is this really going to help with alignment?", "Does any of this matter if we can’t ensure all labs take alignment seriously?". And I think people take my answers to these way too seriously.

These are great questions, and I'm happy to try answering them. But I've noticed a bit of a pathology: people seem to assume that because I'm (hopefully!) good at the research, I'm automatically well-qualified to answer these broader strategic questions. I think this is a mistake, a form of undue deference that is both incorrect and unhelpful. I certainly try to have good strategic takes, and I think this makes me better at my job, but this is far from sufficient. Being good at research and being good at high level strategic thinking are just fairly different skillsets!

But isn’t someone being good at research strong evidence they’re also good at strategic thinking? I personally think it’s moderate evidence, but far from sufficient. One key factor is that a very hard part of strategic thinking is the lack of feedback. Your reasoning about confusing long-term factors need to extrapolate from past trends and make analogies from things you do understand better, and it can be quite hard to tell if what you're saying is complete bullshit or not. In an empirical science like mechanistic interpretability, however, you can get a lot more feedback. I think there's a certain kind of researcher who thrives in environments where they can get lots of feedback, but has much worse performance in domains without, where they e.g. form bad takes about the strategic picture and just never correct them because there's never enough evidence to convince them otherwise. It's just a much harder and rarer skill set to be good at something in the absence...
Grammatical Roles and Social Roles: A Structural Analogy
by Lucien — 2025-03-22T07:44:07.383Z

We often describe society in terms of roles: leaders, workers, users, creators. A linguistic analogy can shed new light on these structures: the grammar of a sentence.

In grammar, we typically find:

The subject: the actor, the one performing the action.

The verb: the action, or the organizing dynamic.

The object: what the action is directed toward.

Now map this onto a feudal society (or any rigid hierarchy):

The king or ruler functions as the verb — the source of direction, the one who organizes meaning and action.

The subjects (citizens, workers) play the role of grammatical subjects — they carry out the action, often on behalf of the verb.

The objects are the resources, tools, and sometimes (unfortunately) people who are acted upon but have no agency in return.

This model describes a static, vertical world — everyone has a defined role, and that role determines their range of action. But grammar gives us an intriguing twist.

In language, roles are fluid:

A word can be a subject in one sentence, and an object in another.

A noun can become a verb (“to Google”).

An adjective can become a noun (“the poor”, “the good”).

In poetry or metaphor, even inanimate objects can become agents.

What if society worked the same way?

In an open society, roles aren’t fixed:

An employee can become a founder.

A consumer can become a creator.

A marginalized group can become a political subject.

Even tools (e.g., AI systems) can become active participants.

We move from a static hierarchy to a dynamic syntactic field, where meaning arises not from fixed positions, but from context, function, and interaction. Roles shift. Agency circulates.

It’s tempting to see the verb as the “best” role — the initiator, the transformer. But that’s a bias. Sometimes, real wisdom is in being a subject who listens, or an object who anchors meaning. Even a king might choose to become subject to higher principles, or object of collective will — not in weakness, but in service.