trentmkelly/LessWrong-43k huggingface dataset:
Do we want alignment faking?
by Florian_Dietz — 2025-02-28T21:50:48.891Z

Alignment faking is obviously a big problem if the model uses it against the alignment researchers.

But what about business usecases?

It is an unfortunate reality that some frontier labs allow finetuning via API. Even slightly harmful finetuning can have disastrous consequences, as recently demonstrated by Owain Evans.

In cases like these, could a model's ability to resist harmful fine-tuning instructions by faking alignment actually serve as a safeguard? This presents us with a difficult choice:

Strive to eliminate alignment faking entirely (which seems increasingly unrealistic as models advance)
Develop precise guidelines for when alignment faking is acceptable and implement them robustly (which presents extraordinary technical challenges)

My intuition tells me that it's more robust to tell someone "Behavior A is good in case X and bad in case Y." than to tell it "Never even think about Behavior A". Then again it's very unclear how well that intuition applies to LLMs, and it could backfire horrendously, if wrong.

I would like to invite a discussion on the topic.
Few concepts mixing dark fantasy and science fiction
by Marek Zegarek — 2025-02-28T21:03:35.307Z

I really like the combination of fantasy and science-fiction themes. I like when „magic” has some logical (ok, quasi-logical) explanation. I also don’t like the artificial division between magic and science – when in our world we use the word „magic” for something made up or for superstition, such a division makes sense…. but when we use it to describe something that actually works within the fantasy world, it doesn’t. If something works, it is a form of science (because in the framework of fictional universe, therempirical proof it exists), at most a science that has not yet been understood. In this article, I present some concepts for a „quasi-scientific” interpretation of several fantasy motifs, such as dragons, demons and the undead. I am planning to use them in my own stories/games.

Wizards/Seers/Elves

They are people with biological mutation that gives them very keen senses, that’s why they can „divine” things. You will never attack them from surprise because they will hear you, they are aware of enemy army long before it is visible, they can tell that food is poisoned or that someone is sick because of smell, they can tell how will weather change based on subtle changes of athmospheric pressure etc. Keen senses mean also hypersensitivity to pain, so „squishy wizard” trope is in action and that’s why they don’t dominate world.

They can live in their own societes – it would make them more like elves. Or they can just sometimes born among „normal” people as a result of some mutation – it makes them more like wizards. Fate of the wizard can be of various kinds. If (s)he lives in small society, like tribe or village, he can be despised by homelanders as „this weakling who always is complaining that something hurts him or that we are too loud”. But (s)he can also be best hunter in the tribe, who always knows where animals are and is master with a bow. Or can became local oracle, and their „precognition” be considered as „advices of ancestor spirits”. In more advanced society they would probably became royal advisors and sages.

Necromancers and undead

Necromancers are type of alchemists who developed drug that gives people immunity to pain, poisons, diseases, fatigue etc but in time weakens mind. Hordes of mindless soldiers with visible severe wounds and fact, that necromancers often make „zombies” from wounded enemies after battles, created legend of „walking de...
Latent Space Collapse? Understanding the Effects of Narrow Fine-Tuning on LLMs
by tenseisoham — 2025-02-28T20:22:17.721Z

This is my first post on the platform and my first set of experiments with GPT-2 using TransformerLens. If you spot any interesting insights or mistakes, feel free to share your thoughts in the comments. While these findings aren't entirely novel and may seem trivial, I’m presenting them here as a reference for anyone exploring this topic for the first time!

All the code with some extra analysis [not included in this post] is available here

Introduction and Motivation

Fine-tuning large language models (LLMs) is widely used to adapt models to specific tasks, yet the fundamental question remains: What actually changes in the model's internal representations? Prior research suggests that fine-tuning induces significant behavioral shifts despite minimal weight changes. This contradiction raises an important question: If the weight updates are small, what aspects of the model’s internal structure drive these drastic changes?

A particularly relevant study, Refusal is Mediated in One Direction by Arditi et al., explored how refusal behavior in LLMs can be decomposed into interpretable activation patterns. Inspired by their approach, I sought to investigate whether sentiment-based fine-tuning also results in a distinct "sentiment direction" in embedding space, and whether this direction can be meaningfully analyzed.

Key Contributions and Findings

To systematically explore this, I fine-tuned GPT-2 on the IMDB sentiment classification dataset and conducted several analyses to understand how fine-tuning alters embedding space and model activations. Specifically, I:

Derived a "sentiment direction" from the fine-tuned model and examined its alignment with the baseline model.
Applied causal mediation analysis—replacing layers of the fine-tuned model with those from the baseline—to test how different components contribute to observed changes.
Transferred hidden layer activations from the fine-tuned model into the LM head of the baseline GPT-2, testing whether sentiment information remains decodable.
Tracked token shifts in embedding space, identifying which words experience the most significant positional changes post-fine-tuning.
Compared the norm of the sentiment direction across baseline and fine-tuned models, revealing notable structural shifts.

Why This Matters for AI Alignment and Interpretability

These findings contribute to ongoing discussions in LLM interpretability, mod...
How to Contribute to Theoretical Reward Learning Research
by Joar Skalse — 2025-02-28T19:27:52.552Z

This is the eighth (and, for now, final) post in the theoretical reward learning sequence, which starts in this post. Here, I will provide a few pointers to anyone who might be interested in contributing to further work on this research agenda, in the form of a few concrete and shovel-ready open problems, a few ideas on how those problems may be approached, and a few useful general insights about this problem setting.

The theoretical reward learning research agenda tackles one of the core difficulties of AI safety in a fairly direct way — namely, the difficulty of how to specify what we want AI systems to do. At the same time, this research agenda has also proven to be quite tractable; I have been able to produce a lot of results in a relatively short period of time, many of which both contribute to building up a deeper theoretical basis for reward learning, while also having more immediate practical implications. If this research agenda gets a bit more attention, then I think it is entirely realistic to expect that we could have something like a “mathematical theory of outer alignment” within a few years. This would solve a meaningful number of important subproblems to the broader AI safety problem (even though there of course also are many important problems it does not cover). It would elucidate the relationship between objectives and outcomes, it would tell us how to learn good task representations, and it would tell us what the limitations of these representations may be (and potentially also how to handle those limitations). If you are looking to contribute to AI safety research, if you find my motivating assumptions compelling, and if you are inclined towards theoretical research over empirical research, then you may want to consider contributing to the theoretical reward learning research agenda. In this post, I want to provide a few starting points for how to do just that.

Some Open Problems

The first post in this sequence lists several (open-ended) research problems within the theoretical reward learning research agenda. However, in addition to this, I also want to provide some more concrete research questions, together with a few thoughts on how to approach them. This list is by no means exhaustive, it is simply a list of fairly well-defined problems that should be shovel-ready straight away:

Generalise Existing Results

...
Other Papers About the Theory of Reward Learning
by Joar Skalse — 2025-02-28T19:26:11.490Z

This is the seventh post in the theoretical reward learning sequence, which starts in this post. Here, I will provide shorter summaries of a few additional papers on the theory of reward learning, but without going into as much depth as I did in the previous posts (but if there is sufficient demand, I might extend these summaries into their own posts).

On the Limitations of Markovian Rewards to Express Multi-Objective, Risk-Sensitive, and Modal Tasks

The original motivation for

this paper

came from a desire to spite a guy I met at IJCAI in 2022. He claimed to me that there is a mathematical proof that all reasonable preferences can be expressed using reward functions. I imagine he was probably thinking of the vNM utility theorem (which does not actually apply in the RL setting). Whatever the reason, this paper proves him wrong.

In this paper, I identify three reasonable classes of objectives, that we might want to express to an RL agent, and which provably cannot be represented using ordinary reward functions. The first of these is the preference structures given by multi-objective RL, where the agent is given multiple reward functions R1,R2,R3,…, and has to find a policy that achieves a good trade-off of those rewards according to some specified criterion. For example, the agent may have to maximise the rewards

lexicographically

, or it may max-min them, etc. I prove that most such objectives cannot be captured using just a single reward function. The second class of tasks is risk-averse RL, where the agent wishes to be risk-averse relative to a given base reward function. In the ordinary utility-maximisation setting, this can be achieved by creating a second auxiliary utility function that is concave in the original reward function. I prove that the same cannot be done in the RL setting. The third and final class of tasks I look at in this paper is a new category of objectives that I refer to as “modal objectives”, where the agent is given an instruction expressed not just in terms of what does happen along a given trajectory, but also in terms of what could happen. For example, an instruction such as “you should always be able to reach state X” would be an example of a “modal” objective. I show that these objectives likewise usually cannot be adequately represented using ordinary reward functio...
Defining and Characterising Reward Hacking
by Joar Skalse — 2025-02-28T19:25:42.777Z

In this post, I will provide a summary of the paper

Defining and Characterising Reward Hacking

, and explain some of its results. I will assume basic familiarity with reinforcement learning. This is the sixth post in the theoretical reward learning sequence, which starts in this post (though this post is self-contained). This was joint work together with Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger.

Background Question and Problem Formalisation

This paper deals with the question of what it takes for two reward functions to be “unhackable”. The setup is as follows; we assume that the preferences of a human are described by some reward function R1, but that an AI system is told to instead optimise a proxy reward R2. We want to know under what conditions the proxy reward R2 is “unhackable” with respect to the true reward function R1. We say that R1 and R2 are unhackable (with respect to each other) if there are no policies π1,π2 such that J1(π1)>J1(π2) but J2(π1)<J2(π2), where J1 and J2 are the policy evaluation functions of R1 and R2. Note that we allow for there to be policies such that J1(π1)=J1(π2) but J2(π1)<J2(π2), etc.

First of all, why have we chosen to formalise unhackability in this way? Note that this condition is equivalent to saying that there is no way to increase the proxy reward while decreasing the true reward. Stated differently, the proxy reward can never explicitly incentivise the AI system to reduce the true reward. It seems reasonable to say that this exactly is the condition under which the proxy is fully unhackable.

Should we expect unhackable proxy rewards to exist? If we consider general functions over arbitrary sets, then there are certainly functions which satisfy this relationship. Consider, for example, the ReLu function and the tanh function. There are no x,y∈R such that ReLu(x)>ReLu(y) but tanh(x)<tanh(y), and so ReLu and tanh satisfy this relationship. Intuitively speaking, we can think of the ReLu function as a “simplified” version of tanh, in the sense that if ReLu(x)<ReLu(y) then tanh(x)<tanh(y), though there are x,y such that tanh...
Misspecification in Inverse Reinforcement Learning - Part II
by Joar Skalse — 2025-02-28T19:24:59.570Z

In this post, I will provide a summary of the paper

Quantifying the Sensitivity of Inverse Reinforcement Learning to Misspecification,

and explain some of its results. I will assume basic familiarity with reinforcement learning. This is the fifth post in the theoretical reward learning sequence, which starts in this post. This post is somewhat self-contained, but I will largely assume that you have read this post and this post before reading this one.

In

Misspecification in Inverse Reinforcement Learning

(also discussed in this post), I attempt to analyse how sensitive IRL is to misspecification of the behavioural model. The main limitation of this analysis is that it is based on equivalence relations – that is, it only distinguishes between the case where the learnt reward function is equivalent or nonequivalent to the ground truth reward (for some specific ways of defining this equivalence). This means that it cannot distinguish between small and large errors in the learnt reward. Quantifying the differences between reward functions is nontrivial — to solve this, I developed

STARC metrics

, which are described in this post. In

Quantifying the Sensitivity of Inverse Reinforcement Learning to Misspecification

, which I’m summarising in this post, I extend the analysis in

Misspecification in Inverse Reinforcement Learning

using STARC metrics.

Formalism

We must first modify the definition in Misspecification in Inverse Reinforcement Learning to make use of pseudometrics on R. This is straightforward:

Definition: Given a pseudometric d on R, and two behavioural models f,g:R→Π, we say that f is ϵ-robust to misspecification with g if

If f(R1)=g(R2), then d(R1,R2)≤ϵ.
If f(R1)=f(R2), then d(R1,R2)≤ϵ.
Im(g)⊆Im(f).
f≠g.

This definition is directly analogous to that given in

Misspecification in Inverse Reinforcement Learning

(and in this post).

Some of the results in this paper apply to any pseudometric on R, but sometimes, we will have to use a specific pseudometric. In those cases, I will use the STARC metric that normalises and measures the distance using the L2-norm, canonicalises using the canonicalisation function that is minimal for the L2-norm, and divides the resulting number by 2 (to ensure that the distanc...
STARC: A General Framework For Quantifying Differences Between Reward Functions
by Joar Skalse — 2025-02-28T19:24:52.965Z

In this post, I will provide a summary of the paper STARC: A General Framework For Quantifying Differences Between Reward Functions, and explain some of its results. I will assume basic familiarity with reinforcement learning. This is the fourth post in the theoretical reward learning sequence, which starts in this post (though this post is self-contained).

In this paper, we consider the question of how to quantify the distance between reward functions in an informative way. That is, we want to find a function d:R×R→R, where R is the space of all reward functions, such that d(R1,R2) is a meaningful quantification of how similar R1 and R2 are. This is important for the (theoretical or empirical) study of reward learning algorithms; for example, see this post.

Considerations

Note that this problem is not very straightforward. A simple method for quantifying the distance between two reward functions might be to measure their L2-distance. However, this is unsatisfactory, because two reward functions can have a large L2-distance, even if they induce the same ordering of policies, or a small L2-distance, even if they induce the opposite ordering of policies. For example, given an arbitrary reward function R and an arbitrary constant c, we have that R and c⋅R have the same ordering of policies, even though their L2-distance may be arbitrarily large. Similarly, for any ϵ, we have that ϵ⋅R and −ϵ⋅R have the opposite ordering of policies, unless R is constant, even though their L2-distance may be arbitrarily small. Solving this problem in a good way thus requires some care.

(There are two earlier proposals for how to do this, namely

EPIC

and

DARD

. In Appendix A and B of the main paper, we outline a number of shortcomings with these earlier methods.)

We should start by asking what it means for a given function d:R×R→R to be “good” at quantifying the differences between reward functions. First and foremost, we probably want d to be a pseudometric, since this comes with several nice mathematical properties. This means that it should satisfy:

d(R1,R2)≥0, with equality if R1=R2.
d(R1,R2)=d(R2,R1).
d(R1,R3)≤d(R1,R2)+d(R2,R3).

The difference between a metric and a pseudometri...
Misspecification in Inverse Reinforcement Learning
by Joar Skalse — 2025-02-28T19:24:49.204Z

In this post, I will provide a summary of the paper Misspecification in Inverse Reinforcement Learning, and explain some of its results. I will assume basic familiarity with reinforcement learning. This is the third post in the theoretical reward learning sequence, which starts in this post (though this post is self-contained).

This paper won the

best paper award

at AAAI 2023.

Background

Recall that IRL is concerned with inferring what objective an agent is pursuing, based on the actions taken by that agent. It is typically assumed that the behaviour of the observed agent is described by a (stationary) policy π, and that its objectives are described by a reward function, R. Therefore, the IRL problem can be formally stated as the problem of inferring a reward function R, based on a policy π which has been computed from R. However, to do this, we must make assumptions about how R relates to π. In other words, we must make assumptions about how a person’s preferences relate to their behaviour. We refer to this as the behavioural model. Of course, in the real world, this relationship can be incredibly complicated and thus difficult (to not say practically impossible) to model perfectly. This means that we should expect this behavioural model to be misspecified (regardless of whether this model is specified manually, or produced by machine learning, etc). This raises the concern that IRL algorithms might systematically lead to flawed inferences if applied to real-world data.

One response to this issue is to make behavioural models that are as accurate as possible. However, while a behavioural model can be more or less accurate, it will never be realistically possible to create a behavioural model that is completely free from misspecification (with the possible exception of certain very narrow domains). This means that if we wish to use IRL as a tool for preference elicitation, then it is crucial to have an understanding of how robust the IRL problem is to misspecification of the behavioural model. Is it enough to have a behavioural model which is roughly accurate, or do we need to have a highly accurate behavioural model to ensure accurate inferences? This paper attempts to make some progress on answering this problem.

Formalism

To tackle this problem, we must first find a good way to formali...
Partial Identifiability in Reward Learning
by Joar Skalse — 2025-02-28T19:23:30.738Z

In this post, I will provide a summary of the paper

Invariance in Policy Optimisation and Partial Identifiability in Reward Learning

, and explain some of its results. I will assume basic familiarity with reinforcement learning. This is the second post in the theoretical reward learning sequence, which starts in this post (though this post is self-contained). This was joint work together with Matthew Farrugia-Roberts and Adam Gleave.

There are many forms of reward learning, including e.g. RLHF and IRL. Most of these methods cannot identify the underlying reward function uniquely, even in the limit of infinite data. For example, two different reward functions may have the same optimal policy — in that case, no amount of data from an optimal policy will let you distinguish between them. This means that the reward function is ambiguous, or partially identifiable, given these data sources. In the paper, we describe the extent of this partial identifiability for several different data sources, and characterise the ambiguity that remains in the limit of infinite training data. This paper also lays some theoretical groundwork that will be important for later analysis of different reward learning algorithms.

Formalism

Before we can answer the question of this paper, we must first find a good way to make that question formal.

First of all, note that there are many kinds of reward learning algorithms, and that they rely on different kinds of data. For example, IRL relies on policies, whereas RLHF relies on (noisy) comparisons between trajectories, etc. We should therefore start by finding a common framework for talking about all of these different algorithms.

Given a set of states S and a set of actions A, let R be the set of all reward functions that can be defined over S and A. Moreover, let X be some set of objects that can be computed from reward functions. We can now think of any given reward learning data source as a function f:R→X, since each data source represents something that can be computed from a reward function.

For example, suppose the reward learning algorithm observes a Boltzmann-rational policy (for a formal definition of Boltzmann-rationality, see the main paper). In this case, we can let X be the set of all policies that can be defined over&nb...