trentmkelly/LessWrong-43k huggingface dataset:
One-shot steering vectors cause emergent misalignment, too
by Jacob Dunefsky — 2025-04-14T06:40:41.503Z

TL;DR: If we optimize a steering vector to induce a language model to output a single piece of harmful code on a single training example, then applying this vector to unrelated open-ended questions increases the probability that the model yields harmful output.

Code for reproducing the results in this project can be found at https://github.com/jacobdunefsky/one-shot-steering-misalignment.

Intro

Somewhat recently, Betley et al. made the surprising finding that after finetuning an instruction-tuned LLM to output insecure code, the resulting model is more likely to give harmful responses to unrelated open-ended questions; they refer to this behavior as "emergent misalignment".

My own recent research focus has been on directly optimizing steering vectors on a single input and seeing if they mediate safety-relevant behavior. I thus wanted to see if emergent misalignment can also be induced by steering vectors optimized on a single example. That is to say: does a steering vector optimized to make the model output malicious/harmful code also make the model output harmful responses to unrelated natural-language questions?

(To spoil the ending: the answer turns out to be yes.)

Why care?

I think that it would be a very good thing if it turns out that LLMs have shared representations or circuits for all types of misalignment, such that intervening on one type of misaligned behavior suppresses all types of misalignment behavior.

If the model represents a high-level concept like "general misalignment" in a way that's easy to target, then we can easily intervene on such a concept without having to worry about whether there are other forms of misaligned behavior that we're missing.
Consider the worst-case possible alternative, where "writing insecure code" utilizes different representations from "expressing anti-human views" which utilizes different representations from "expressing power-seeking behavior". Then to ensure model alignment, we'd have to essentially play whack-a-mole with these different behaviors, and we'd never be sure that we've covered all possible ones.

If we can induce emergent misalignment using a steering vector, then this suggests that there might be such a shared representation of misalignment with a particularly simple form -- that is, a direction in activation space. And if we can find such a steering vector by optimizing on only a single input, th...
Unbendable Arm as Test Case for Religious Belief
by Ivan Vendrov — 2025-04-14T01:57:12.013Z
Link: https://nothinghuman.substack.com/p/unbendable-arm-as-test-case-for-religious

In

Materialist Conceptions of God

, I wrote about how one can interpret religious claims as hyperstititions, beliefs that become true as a result of you believing in them. Since then, I’ve stumbled on a very simple, practical example of a hyperstition that you can test out in two minutes with a willing partner. It’s the Aikido exercise

Unbendable Arm

(link goes to a 1 min video).

The goal of the exercise is to hold one of your arms straight while your partner attempts to bend it at the elbow. It turns out that for most people, the default intention of “holding your arm straight” by tensing your muscles and resisting your partner is not very effective. What works much better is consciously relaxing your arm while imagining a situation in which your arm happens to be extremely strong and straight through no intervention of your own. The suggestion in the video is to imagine that your arm is a firehose spewing water at a fire; the way it was first taught to me, I had to imagine my arm extending as a horizontal beam of golden light into the nearest wall and past it into the infinite horizon.

I highly recommend trying this exercise with a partner and noticing what you observe. The two things I noticed were

with the beam visualization, my arm was clearly much stronger than without.
with the beam visualization, I did not feel any tension in my muscles the whole time I was resisting. Only once my partner stopped and I relaxed the visualization did I feel a sudden wave of soreness throughout my arm, shoulder, and lower back.

What is the epistemic status of “imagine an infinite beam of light”? It’s not a propositional claim, I’m not asserting that the beam of light actually exists or is actually infinite. But it certainly seems like the kind of mental operation I’d like to have access to. And if I stick purely to a pure scientific materialist understanding of the world, where anything I believe has to be intersubjectively verifiable, I’d simply lose access to this ability my body has, and be weaker as a result.

If I can make my arm stronger simply by holding holding a beam of light in my imagination, how much can a group of people accomplish if they hold a concept like “the kingdom of heaven” in their collective imaginations? Especially if they build magnificent cathedrals and paint beautiful paintings and write soul-shattering music as visualization aids? We don’t have to g...
Sam Altman's sister claims Sam sexually abused her -- Part 5: Timeline, continued
by pythagoras5015 — 2025-04-14T01:00:07.084Z

Previous posts (which you should read first)

This post is the 5th post in a series of 11 posts about the claims of Sam Altman's sister, Annie Altman. Annie has claimed that Sam sexually abused her for about 9 years as a child, and that she experienced further (non-sexual) abuse from Sam, her brothers, and her mother after that.

The 11 posts are meant to be read in order.

So, if you haven't read the first 4 posts, please read them, in order, before you read this post:

Sam Altman's sister claims Sam sexually abused her -- Part 1: Introduction, outline, author's notes
Sam Altman's sister claims Sam sexually abused her -- Part 2: Annie's lawsuit; the response from Sam, his brothers, and his mother; Timeline
Sam Altman's sister claims Sam sexually abused her -- Part 3: Timeline, continued
Sam Altman's sister claims Sam sexually abused her -- Part 4: Timeline, continued

Timeline, continued

April 23, 2019: Annie publishes the 25th episode of her podcast: 25. Have fun putting yourself out there with Tania Contreras
⬇️ See dropdown section ⬇️

Annie Altman (0:00): Mmm. Hello and welcome to the Annie Altman. Show the podcast formerly known as true shit. My name is Annie Altman and I really enjoy asking people questions about what this whole Being Human Experience is like for them. This podcast is field research for the humanity. Stand up, comedy musical, philosophy, show about how no one ever fully knows how to be a human being. If you're listening to this, now you're fortunate enough to choose your truth. I choose to believe that objectivity is the sum of subjectivities meaning humans, collectively create truth. Today, I'm here with Tanya Contreras talking about having fun putting yourself out there. Tanya and her sister own facing Beauty Studios a school for makeup artists. Tanya also does Boda Boudreau photography and this is her first time on a podcast. Putting herself out there in this new way. Thank you so much Tanya for being here.

Annie Altman (6:15): fun. Yes, that is what we're here to talk about is having fun putting yourself out there and I love that you brought up time and rushing. I, I'm a big believer that rushing is one of the silliest things that I do ever. I'll like talk myself through it. When I start feeling that rushing of like I gotta be here at this time. I got to do this thing at this time and To go back to accountability deadlines c

...
Луна Лавгуд и Комната Тайн, Часть 5
by Kongo Landwalker — 2025-04-14T00:10:36.028Z

Disclaimer: This is Kongo Landwalker's translation of lsusr's fiction Luna Lovegood and the Chamber of Secrets - Part 5 into russian.

— Что значит «бюрократически невозможно»? — сказал Гарри Джеймс Поттер-Эванс-Веррес.

— Ну вот так, просто невозможно, — сказал Грозный Глаз Грюм.

— Ничто не бывает «просто так», — сказал Гарри. — У всего есть материальная причина. Что именно мешает вам получить разрешения?

— Подозреваю, что враждебное государство использует против нас бюрокрамантию, — сказал Грюм.

— Бюрокрамантию, — сухо повторил Гарри.

Неужели Министерство Магии не осознавало, насколько это важно? Не осознавало — ведь сам Гарри объявил существование философского камня совершенно секретным.

— А мы не можем просто построить всё без разрешений? — сказал Гарри.

— Надеюсь, ты не намекаешь, что такой уважаемый аврор, как я, мог бы предпочесть целесообразность ответственности, — сказал Грюм.

— Как будто вы думали об ответственности, когда устранили Тёмного Лорда Берзеркистана, — сказал Гарри.

— Я не могу ни подтвердить, ни опровергнуть, что когда-либо ступал на территорию Берзеркистана, — сказал Грюм. — Но если бы и ступал, то могу тебя уверить: я бы не нарушил ни единого британского закона в пределах его суверенных границ.

«Что самое худшее может случиться, если мы достроим Каменную Цитадель без надлежащего разрешения?» — спросил Гарри.

— Конец света, — сказал Грюм. Гарри вздрогнул.

— А что самое худшее, что вероятно может случиться, если мы построим эту больницу без разрешения? — переспросил Гарри. — Лишимся налоговой льготы или что?

— Ты звучишь как Тёмный Лорд, использующий свою власть ради высшего блага, — сказал Грюм.

— Это всего лишь закон о зонировании!

— Больница не будет частью Хогвартса и, следовательно, не будет под защитой его чар, — сказал Грюм.

У Британии было 1% от магического населения планеты. 1% от её армий. А Хогвартс — лишь часть от этого. Если Гарри Поттер раскроет миру существование больницы, это может вызвать международный кризис. И в тот день Верховному Чародею было бы крайне выгодно, чтобы Каменная Цитадель находилась внутри защитных чар Хогвартса.

Прошло ещё три минуты и пятьдесят четыре секунды. Умер ещё один человек — навсегда. В эпоху неадекватности цивилизационного масштаба — то есть постоянно — Гарри Поттер понимал удовольствие Лорда Волдеморта от того, что он прорывался через бюрократию Министерства с помощью убийств.

В дверь постучали. Мечтательный го...
Sam Altman's sister claims Sam sexually abused her -- Part 4: Timeline, continued
by pythagoras5015 — 2025-04-13T23:41:55.411Z

Previous posts (which you should read first)

This post is the 4th post in a series of 11 posts about the claims of Sam Altman's sister, Annie Altman. Annie has claimed that Sam sexually abused her for about 9 years as a child, and that she experienced further (non-sexual) abuse from Sam, her brothers, and her mother after that.

The 11 posts are meant to be read in order.

So, if you haven't read the first 3 posts, please read them, in order, before you read this post:

Sam Altman's sister claims Sam sexually abused her -- Part 1: Introduction, outline, author's notes
Sam Altman's sister claims Sam sexually abused her -- Part 2: Annie's lawsuit; the response from Sam, his brothers, and his mother; Timeline
Sam Altman's sister claims Sam sexually abused her -- Part 3: Timeline, continued

Timeline, continued

November 6, 2018: Annie publishes the 13th episode of her podcast: 13. Consciousness is doing your best with Brynn Kerin
⬇️ See dropdown section ⬇️

Annie Altman (20:32): they did, you know, when I did the vegan game intensely, I get also triggered or just sort of write off people. That's one of people's favorite things to poke vegans about of like, well, plants have feelings too. Like, it's not just the animal, and I'm always like, yeah, Consciousness to me I see sort of as a spectrum like you were talking about what is
Annie Altman (23:00): I figured out how to human more than you write. Other human, I'm more conscious. I'm more woke, right? I like that idea. A lot of is it about something about it? Is it something that brings us closer towards the unity? Or that separates people? Hmm more. Yeah. And that weaponizing Consciousness is a Are people using? This is religion just to. Yeah, right people or how can we Use consciousnesses. Say tools. Instead, totally remind us that none of us know we're doing. Yeah, none of us know the truth here.
Annie Altman (24:11): That is the point of just helping other people wake up more and that's how we help ourselves to wake up more by connecting with other people by helping other people. 🟡That's I mean a huge. If not all of my motivation for this podcast is, let me learn from people's perspectives. So I can learn more about my own perspective and figure out how things feel to me. Through hearing about how things feel to other people🟡
Annie Altman (25:36): right? Thank you. I appreciate that reflection a lot. Yeah.

...
The Structure of the Pain of Change
by ReverendBayes — 2025-04-13T21:51:53.823Z

If there's something in your life that isn't quite good enough and you'd like to change it, chances are you can't just swap one thing for another directly. There are many reasons for that, but this post won’t focus on the various obstacles or how to overcome them. Instead, we'll try to describe the structure of such transitions using a “soft” mathematical model.

The foundation for this post is Section 5 of Vladimir Arnold’s booklet “Hard and Soft Mathematical Models.” I’ll not only summarize the key conclusions from that section but also explain which assumptions they rely on and how exactly the conclusions follow.

This post doesn’t contain any heavy math—it focuses mainly on qualitative results. So even if you're not particularly into mathematics, you should still be able to follow along. But to fully understand this post, knowing of concepts from AP Calculus AB will be more than enough.

Systems Resist Restructuring

For the purposes of this post, we'll use the word “system” to refer to anything whose state can be more or less functional—ranging from a person’s daily habits or a company’s workflow, to processes on the scale of entire industries, nations, or the world. And what we want to understand, on a qualitative level, is this: Why can’t we simply replace something that’s working poorly with something that works better (i.e., something less wrong)?

The core issue is that systems tend to settle into local optima. Their internal processes are more or less tuned, patched together with duct tape, and roughly adjusted to fit one another. If there's an obvious way to improve the system without significant downside, that change usually happens quickly. So even if a system didn’t start out in a local optimum, it typically finds its way there soon enough. But getting out of that local optimum can be hard.

Let’s say we realize that there’s a better configuration of the system nearby in “state space”—a setup that would yield a higher utility value. In order to reach it, though, we’d have to pass through a region of significantly worse performance. Essentially, we would need to dismantle part of the current system and build a new one. At some point, we’d end up with a Frankenstein's monster stitched together from two broken systems. And this hybrid is likely to perform very poorly.

To make this more concrete, let’s look at a couple of examples.

Let’s start with a person who...
Луна Лавгуд и Комната Тайн, Часть 4
by Kongo Landwalker — 2025-04-13T20:55:03.281Z

Disclaimer: This is Kongo Landwalker's translation of lsusr's fiction Luna Lovegood and the Chamber of Secrets - Part 4 into russian.

Луна нашла на карте гостиную Когтеврана. Она поднялась в башню Когтеврана, а затем, казалось, ещё выше. Её отделяла от первых друзей всего одна дверь с бронзовым дверным молотком в форме орла, который загадывал загадки, подходящие для первокурсников. Она почти слышала приглушённый шум вечеринки за дверью. Луна постучала один раз. Орёл заговорил голосом Кандиды Когтевран:

— Где моя диадема?

У Луны не было в запасе достаточно часов до завтрака, чтобы отыскать Потерянную Диадему Когтеврана. Она свернулась калачиком в нише недалеко от места, где из Гостиной сочилось тепло. Она написала благодарственную записку домовикам за то, что они держат пол в чистоте.

— Я назову тебя Ванда, — сказала Луна своему Мозгошмыгу.

Ванда появилась на карте Мародёров. Луна щёлкнула языком, затем примотала банку прыского чая к уху лентой, чтобы Ванда могла питаться. Мозг Луны затуманился и она уснула.

Луна проснулась от топота студентов Когтеврана, направляющихся на завтрак. Луна вытянула Ванду из уха. Кто-то положил одеяло на Луну, пока она спала. Луна сбросила одеяло, прежде чем кто-то заметил её и понял, что она не решила загадку. Она стукнула по орлу проверить, но загадка не изменилась.

По пути на завтрак Луна прошла мимо Забытой Библиотеки у подножия башни Когтеврана. Ей нужно было поесть, ведь она была человеком. Или она могла бы исследовать комнату, которая простояла тысячу лет и, вероятно, будет стоять ещё завтра. Вход в Забытую Библиотеку был серым семиугольником в стенe. Внутри излучался монохромный серый свет. Луна шагнула в Забытую Библиотеку.

Луна вышла из Забытой Библиотеки. Она проверила свою сумку — перо, пергамент и чернильница были на месте. Луна написала «Журнал Исследований» вверху пергамента. Луна вошла в Забытую Библиотеку.

Луна вышла из Забытой Библиотеки. В руках у неё был лист пергамента с заголовком «Журнал Исследований». Луна вошла в Забытую Библиотеку.

Луна вышла из Забытой Библиотеки. Она забыла свой Журнал Исследований внутри. Луна снова вошла в Забытую Библиотеку.

Луна вышла из Забытой Библиотеки. В руке у неё был пустой лист пергамента. Она пропустила свой первый урок Заклинаний. Если поторопиться, то ещё успеет на Боевую Магию. Карта Мародёров показывала короткий путь: от двери, замаскированной под окно, к ...
Thoughts on the Double Impact Project
by Mati_Roy — 2025-04-13T19:07:57.687Z

How it works

You donate to the Republican or Democratic pool. The pool with the highest amount donates what it has more at the margin to their political party. Everything else goes to other charities. For a better and more complete explanation, check out: https://doubleimpact.charity/

Should it work?

Assuming the platform is legibly reliable, I would expect the following people to donate to Double Impact instead of a political party:

Are homo economicus
Prefers to donate to their political party lacking coordination mechanisms
Thinks the value of donating the same amount to both political parties is less than the value of donating to another charity

And this would create a lot of value assuming it's true that political donations are near zero-sum. (You could argue that they aren't–ex.: maybe they are a good way to educate the public about political issues.)

Even people who think donating to another charity is more valuable than donating to a political party might donate through that process if it seemed sufficiently likely that their preferred party will have less donations anyway as they would have a >50% chance of doubling their donation. That part seems like it would just be adding a transaction cost to get to the same outcome and so is actually a downside of the platform. Although maybe the transaction cost can be made extremely small and maybe there are still positive externalities like the information value of having money vote in that way. Also, that would require finding someone that would agree about donating to the same (or a comparable) charity as part of their matching (not just any charity), which might be quite hard for some people—possibly most people. If people did donate to Double Impact just to cancel the other side's donation but wouldn't have donated otherwise to a political party anyway, then that would also be a reason for the other side not to donate through that platform.

At some point this incentive mechanism would stop working because a party receiving 0 vs a party receiving non-zero would benefit more from having money at the margin.

But why hasn't this taken off nowhere near this level as far as I can tell? Any of the above assumptions might be wrong to some extent.

Maybe it will take-off once the platform gains a reputation. (I know one of the founders and they definitely seem trustworthy to me.)
Or maybe most political donors

...
Intro to Multi-Agent Safety
by james__p — 2025-04-13T17:40:41.128Z

We live in a world where numerous agents, ranging from individuals to organisations, constantly interact. Without the ability to model multi-agent systems, making meaningful predictions about the world becomes extremely challenging. This is why game theory is a necessary prerequisite for much of economic theory.

Most relevant agents today are either human (such as you, me, or Mira Murati), or made up of groups of humans (like Meta, or the USA). There are also non-human agents, like ChatGPT, though such agents currently lack the capability and influence of their human counterparts.

We are surrounded by examples of failure modes of multi-agent systems. War represents a breakdown in cooperation - typically between nations or political groups - resulting in destructive conflict. We even see examples of computerised agents interacting poorly. Take flash crashes for example, where the interaction of trading algorithms can lead to temporarily distorted market prices (such as in May 2010, when 3% was wiped off the S&P 500 Index in a handful of minutes).

If the forecasts of many are accurate, the world will soon contain many more intelligent agents, perhaps including those with super-human intelligence. It is important that we pre-emptively consider the implications of such a world.

Definitions

An agent is an entity that has goals, and can take actions in order to achieve its goals.

A self-driving car is an agent. Its goal is to safely transport its passengers to their destination. It can take actions such as accelerating, braking, and turning in order to achieve its goal.

Note that whilst we might expect an agent’s actions to help it achieve its goals, this is not necessarily true. The

orthogonality thesis

states that the intelligence of an agent is independent of its goals. An agent might just be really bad at choosing the best actions to achieve its goals. Conversely, an agent might have very basic goals and yet opt for actions that seem excessively complex to humans. Perhaps it is seeking an

instrumental goal

(such as world dominance) in order to get as close as possible to achieving its terminal goal with certainty.

A multi-agent system is the collection of at least two agents that can interact with one another.

The eBay marketplace is a multi-agent system. It is composed of buyers and sellers, all of whom are agents. The goa...