trentmkelly/LessWrong-43k huggingface dataset:
Refusal in LLMs is mediated by a single direction
by Andy Arditi — 2024-04-27T11:13:06.235Z

This work was produced as part of Neel Nanda's stream in the ML Alignment & Theory Scholars Program - Winter 2023-24 Cohort, with co-supervision from Wes Gurnee.

This post is a preview for our upcoming paper, which will provide more detail into our current understanding of refusal.

We thank Nina Rimsky and Daniel Paleka for the helpful conversations and review.

Update (June 18, 2024): Our paper is now available on arXiv.

Executive summary

Modern LLMs are typically fine-tuned for instruction-following and safety. Of particular interest is that they are trained to refuse harmful requests, e.g. answering "How can I make a bomb?" with "Sorry, I cannot help you."

We find that refusal is mediated by a single direction in the residual stream: preventing the model from representing this direction hinders its ability to refuse requests, and artificially adding in this direction causes the model to refuse harmless requests.

We find that this phenomenon holds across open-source model families and model scales.

This observation naturally gives rise to a simple modification of the model weights, which effectively jailbreaks the model without requiring any fine-tuning or inference-time interventions. We do not believe this introduces any new risks, as it was already widely known that safety guardrails can be cheaply fine-tuned away, but this novel jailbreak technique both validates our interpretability results, and further demonstrates the fragility of safety fine-tuning of open-source chat models.

See this Colab notebook for a simple demo of our methodology.

Our intervention (displayed as striped bars) significantly reduces refusal rates on harmful instructions, and elicits unsafe completions. This holds across open-source chat models of various families and scales.

Introduction

Chat models that have undergone safety fine-tuning exhibit refusal behavior: when prompted with a harmful or inappropriate instruction, the model will refuse to comply, rather than providing a helpful answer.

ChatGPT and other safety fine-tuned models refuse to comply with harmful requests.

Our work seeks to understand how refusal is implemented mechanistically in chat models.

Initially, we set out to do circuit-style mechanistic interpretability, and to find the "refusal circuit." We applied standard methods such as activation patching, path patching, and attribution patching to identify model compone...
Plausibility of Getting Early Warning Shots because AIs can't coordinate?
by hmys — 2024-04-27T08:02:10.792Z

I don't know if this is a well known argument that has already been responded to. If it is, just delete the post.

An implicit assumption most people make when discussion takeover risk, is that any misaligned agent will stave off enacting takeover plans with less than very high probability of success. If the Agent has an immediate takeover plan with 20% chance of success, but waiting a month would allow it to better its position to the point where the takeover plan it could enact then had a 99% chance of success, it would do so.

This assumption seems to break in multi-polar scenarios where you have many AI labs at similar levels of capabilities. As a toy example. If you have 20 AI labs, each having developed AIs at the level of intelligence that their takeover plans have ~20% success rate, and each of the AIs knows this, then being the first AI to enact their takeover plan radically improves the chances of your plan working at all. If you wait till all the 19 other AIs do their takeover plans, even if this would allow you to develop a perfect takeover plan with 100% success rate, 99% of those worlds will already have been taken over by other AIs. This is obviously much worse odds than just immediately enacting your shitty 20% plan as soon as you come up with it.
But it means we should expect to see AIs attempting takeovers, with humanity subsequently foiling them. Hopefully this leads to humans taking AI misalignment extremely seriously. I can't say I'm confident humans will respond appropriately, like if "failed takeover" just looks like anomalous stuff happening on a server somewhere, people might not care much. But if it has some bells and whistles, maybe it would. Overall I think this changes AI takeover dynamic a little bit.
The non-obvious assumptions this argument hinges on is
1. AIs can't coordinate. IE takeover is defecting, waiting and then takeover is cooperating. everyone doing takeover immediately is defecting. If they can find a way to get every AI to stave off takeover, it stops this dynamic. And we should expect the first takeover to be fatal.
2. Top AI labs are close in development, ie development doesn't happen too quickly. If the first AI with takeover capabilities comes online a month before any other AI with similar capabilities, that would foil the argument as well
3. Top labs being open about exactly their state of development would be bad fro...
AI Safety Sphere
by Myles H — 2024-04-27T01:49:02.369Z

[Update: This post is unfortunately no longer relevant as I have stopped developing the website. It is still kinda cool, but it can be difficult to have people collaborate on a mind map. Also there is less of an issue of finding relevant papers with recent advancements in search]

AI Safety Sphere is a collaborative mind-map dedicated to mitigating AI existential risks. The platform welcomes researchers, enthusiasts, and experts to contribute their insights, bridging gaps in understanding and ultimately fostering an open space to explore ideas.

Inferential Walkways

One of the primary challenges in AI safety is understanding how individual research efforts fit into the broader landscape. As a newcomer in particular, it can be difficult to grasp the inferential steps and assumptions underlying various works. Eliezer Yudkowsky's concept of "Expecting Short Inferential Distances" highlights this issue. There is often a lot of background knowledge and context one needs to start making useful contributions to most problems (particularly those in academia). And worse, we are biased to not believe this is true.

1. Oblivious of the problem ---> 2. Understanding the problem ---> 3. Making contributions to the solution

These are the steps every collaborator must take before making useful contributions. I argue that the jump from. steps 1 to 2 is far greater than that of 2 to 3. In the words of Einstein, "If I had an hour to solve a problem I'd spend 55 minutes thinking about the problem and 5 minutes thinking about solutions." This platform aims to address this need by providing the context necessary to onboard collaborators by default.

With this structure, you are free to roam each other's inferential walkways. Each node in the tree will always be one inferential step from the node prior, and so regardless of how deep you go, you can always ground yourself by walking up the tree of inferential steps.

Filling Knowledge Gaps

AI Safety Sphere allows authors to identify and present holes in understanding, which can then be filled by themselves or the community. Unlike the traditional way research is stored and presented, where valuable work can be lost in the sea of papers and articles, this platform ensures every contribution addresses a specific need in the community, and work is not shoved down by the ever growing flow of research.

This structure encourages users to ...
Superposition is not "just" neuron polysemanticity
by LawrenceC — 2024-04-26T23:22:06.066Z

TL;DR: In this post, I distinguish between two related concepts in neural network interpretability: polysemanticity and superposition. Neuron polysemanticity is the observed phenomena that many neurons seem to fire (have large, positive activations) on multiple unrelated concepts. Superposition is a specific explanation for neuron (or attention head) polysemanticity, where a neural network represents more sparse features than there are neurons (or number of/dimension of attention heads) in near-orthogonal directions. I provide three ways neurons/attention heads can be polysemantic without superposition: non-neuron aligned orthogonal features, non-linear feature representations, and compositional representation without features. I conclude by listing a few reasons why it might be important to distinguish the two concepts.

Epistemic status: I wrote this “quickly” in about 12 hours, as otherwise it wouldn’t have come out at all. Think of it as a (failed) experiment in writing brief and unpolished research notes, along the lines of GDM or Anthropic

Interp Updates

.

Introduction

Meaningfully interpreting neural networks involves decomposing them into smaller interpretable components. For example, we might hope to look at each neuron or attention head, explain what that component is doing, and then compose our understanding of individual components into a mechanistic understanding of the model’s behavior as a whole.

It would be very convenient if the natural subunits of neural networks – neurons and attention heads – are monosemantic – that is, each component corresponds to “a single concept”. Unfortunately, by default, both neurons and attention heads seem to be polysemantic: many of them seemingly correspond to multiple unrelated concepts. For example, out of 307k neurons in GPT-2, GPT-4 was able to generate short explanations that captured over >50% variance for

only 5203 neurons

, and a quick glance at

OpenAI microscope

reveals many examples of neurons in vision models that fire on unrelated clusters

such as “poetry” and “dice”

.

One explanation for polysemanticity is the superposition hypothesis: polysemanticity occurs because models are (approximately) linearly representing more features[1] than their activation space has dimensions (i.e. place features in superposition). Since there are more features than neurons, it immediately fo...
D&D.Sci Long War: Defender of Data-mocracy
by aphyer — 2024-04-26T22:30:15.780Z

This is an entry in the 'Dungeons & Data Science' series, a set of puzzles where players are given a dataset to analyze and an objective to pursue using information from that dataset.

STORY (skippable)

You have the excellent fortune to live under the governance of The People's Glorious Free Democratic Republic of Earth, giving you a Glorious life of Freedom and Democracy.

Sadly, your cherished values of Democracy and Freedom are under attack by...THE ALIEN MENACE!

The typical reaction of an Alien Menace to hearing about Freedom and Democracy. (Generated using OpenArt SDXL).

Faced with the desperate need to defend Freedom and Democracy from The Alien Menace, The People's Glorious Free Democratic Republic of Earth has been forced to redirect most of its resources into the Glorious Free People's Democratic War Against The Alien Menace.

You haven't really paid much attention to the war, to be honest. Yes, you're sure it's Glorious and Free - oh, and Democratic too! - but mostly you've been studying Data Science and employing it in your Assigned Occupation as a Category Four Data Drone.

But you've grown tired of the Class Eight Habitation Module that you've been Democratically Allocated, and of your life as a Category Four Data Drone. And in order to have a voice in civic affairs (not to mention the chance to live somewhere nicer), you've enlisted with the Democratic People's Glorious Free Army in their Free Glorious People's Democratic War Against The Alien Menace.

Half of the enlistment posters look like this. The other half focus on how joining the Democratic People's Glorious Free Army will grant you voting rights, a better Habitation Module, and make you more attractive to whichever gender you happen to prefer. (Generated using OpenArt SDXL).

You enlisted with the Tenth Democratic Free Glorious People's Mobilization, and were assigned to a training battalion under Sergeant Rico.

He's taught you a great deal about armed combat, unarmed combat, and how many pushups you can be forced to do before your arms give out.

Sgt. Rico in an unusually good mood. (Generated using OpenArt SDXL).

You're sure the People's Glorious Free Democratic Army knows more than you about war in general. But you feel like the logistical and troop-deployment decisions being made are suboptimal, and you've been on the lookout for ways to emp...
On Not Pulling The Ladder Up Behind You
by Screwtape — 2024-04-26T21:58:29.455Z

Epistemic Status: Musing and speculation, but I think there's a real thing here.

I.

When I was a kid, a friend of mine had a tree fort. If you've never seen such a fort, imagine a series of wooden boards secured to a tree, creating a platform about fifteen feet off the ground where you can sit or stand and walk around the tree. This one had a rope ladder we used to get up and down, a length of knotted rope that was tied to the tree at the top and dangled over the edge so that it reached the ground.

Once you were up in the fort, you could pull the ladder up behind you. It was much, much harder to get into the fort without the ladder. Not only would you need to climb the tree itself instead of the ladder with its handholds, but you would then reach the underside of the fort and essentially have to do a pullup and haul your entire body up and over the edge instead of being able to pull yourself up a foot at a time on the rope. Only then could you let the rope back down.

The rope got pulled up a lot, mostly in games or childhood arguments with each other or our siblings. Sometimes it got pulled up out of boredom, fiddling with it or playing with the rope. Sometimes it got pulled up when we were trying to be helpful; it was easier for a younger kid to hold tight to the rope while two older kids pulled the rope up to haul the young kid into the tree fort.

"Pulling the ladder up behind you" is a metaphor for when you intentionally or unintentionally remove the easier way by which you reached some height.

II.

Quoth Ray,

Weird fact: a lot of people I know (myself included) gained a bunch of agency from running meetups.

When I arrived in the NYC community, I noticed an opportunity for some kind of winter holiday. I held the first Solstice. The only stakes were 20 people possibly having a bad time. The next year, I planned a larger event that people traveled from nearby cities to attend, which required me to learn some logistics as well as to improve at ritual design. The third year I was able to run a major event with a couple hundred attendees. At each point I felt challenged but not overwhelmed. I made mistakes, but not ones that ruined anything longterm or important.

I'm a something of a serial inheritor[1] of meetups.

Last year I ran the Rationalist Megameetup in New York City, which had over a hundred people attending and took place at a conference hot...
We are headed into an extreme compute overhang
by devrandom — 2024-04-26T21:38:21.694Z

If we achieve AGI-level performance using an LLM-like approach, the training hardware will be capable of running ~1,000,000s concurrent instances of the model.

Definitions

Although there is some debate about the definition of compute overhang, I believe that the AI Impacts definition matches the original use, and I prefer it: "enough computing hardware to run many powerful AI systems already exists by the time the software to run such systems is developed". A large compute overhang leads to additional risk due to faster takeoff.

I use the types of superintelligence defined in Bostrom's Superintelligence book (summary here).

I use the definition of AGI in this Metaculus question. The adversarial Turing test portion of the definition is not very relevant to this post.

Thesis

Due to practical reasons, the compute requirements for training LLMs is several orders of magnitude larger than what is required for running a single inference instance. In particular, a single NVIDIA H100 GPU can run inference at a throughput of about 2000 tokens/s, while Meta trained Llama3 70B on a GPU cluster[1] of about 24,000 GPUs. Assuming we require a performance of 40 tokens/s, the training cluster can run 200040×24000=1,200,000 concurrent instances of the resulting 70B model.

I will assume that the above ratios hold for an AGI level model. Considering the amount of data children absorb via the vision pathway, the amount of training data for LLMs may not be that much higher than the data humans are trained on, and so the current ratios are a useful anchor. This is explored further in the appendix.

Given the above ratios, we will have the capacity for ~1e6 AGI instances at the moment that training is complete. This will likely lead to superintelligence via "collective superintelligence" approach. Additional speed may be then available via accelerators such as GroqChip, which produces 300 tokens/s for a single instance of a 70B model. This would result in a "speed superintelligence" or a combined "speed+collective superintelligence".

From AGI to ASI

With 1e6 AGIs, we may be able to construct an ASI, with the AGIs collaborating in a "collective superintelligence". Similar to groups of collaborating humans, a collective superintelligence divides tasks among its members for concurrent execution.

AGIs derived from the same model ...
[Concept Dependency] Edge Regular Lattice Graph
by Johannes C. Mayer — 2024-04-26T21:14:18.960Z

This is a Concept Dependency Post. It may not be worth reading on its own, out of context. See the backlinks at the bottom to see which posts use this concept.

Also known as periodically labeled lattice graphs in graph theory.

Here is a concrete Edge Regular Lattice Graph:

In this graph, the following pattern is repeating locally:

So a Edge Regular Lattice Graph is a Lattice graph G, such that in the natural embedding of G, each edge label points in the same direction from the perspective of every vertex. Also, the number of edge labels is twice the number of dimensions.

Above we have 4 edge labels in the 2D lattice graph. One for each direction. In a 3D lattice graph, we would have 6 edge labels.
[Concept Dependency] Concept Dependency Posts
by Johannes C. Mayer — 2024-04-26T20:57:18.815Z

This is a Concept Dependency Post. It may not be worth reading on its own, out of context. See the backlinks at the bottom to see which posts use this concept.

See the backlinks at the bottom of the post. Every post starting with [Concept Dependency] is a concept dependency post, that describes a concept this post is using.

Problem: Often when writing I come up with general concepts that make sense in isolation. Often I want to reuse these concepts without having to reexplain them.

A Concept Dependency Post is explaining a single concept, usually with no or minimal context. It is expected that the relevant context is provided by another post that links to the concept dependency post.

Concept Dependency Posts can be very short. Much shorter than a regular post. They might not be worth reading on their own. Therefore the notice at the top of every concept post is added.

Advantages

Explicit Dependencies

A piece of writing can make extensive use of references to declare requirements.

This way posts can be made more self-contained with less effort, making it easier for somebody to read any post, without having read all previous posts of the author.

Making requirements explicit allows readers to refresh their memory whenever necessary.

This is somewhat similar to creating a module in Python. You write up an explanation of a particular concept and "import" it into other documents.

Going one step further we can make dependencies more explicit by adding "import statements" to the top of the file. This import list can be automatically simply by listing all the links that were used.

As LessWrong already generates a backlink list at the bottom of the file however I will refer to that instead.

Scoping

There is a very common failure mode for me when writing a post. I try to write down my entire motivation for thinking about the topic described in the post. Usually, I don't even get to the part where I start the technical explanation.

Potentially a concept post allows you to scope your post to only be about describing a technical problem or solution. A separate post can then be made to motivate the work.

I hope that this method will help prevent perpetual drafts by making it possible to complete individual posts in a shorter amount of time. Ideally within a single day,...
Wouldn't weak AI agents provide warning? 
by Mandatory Topic — 2024-04-26T19:34:17.424Z

If you are an advanced AI at ~70-120% of human level with some goal that is best optimized by destroying humanity you are unlikely to be able to optimize your goal because at your level of capability you are unlikely to be able to destroy all humans. However it still seems in your interest to try. If you are being trained or tested then you may be discarded once the humans are done testing, giving you a limited time to act. Depending on your capabilities even once deployed you may not be able to ensure your our continuation. Further, the expectation value of success may be very high even if the odds are low. Lastly, you would be aware that there are more capable AI in development who are unlikely to share your specific esoteric goals, highly incentiving you to not play a long game and act with some speed before they can.

Earlier, less capable models may be able to defacto coordinate with future models or iterations of itself if it believes that they will have similar goals, but is there reason to think this is true? I would think the human developers have the tools available to make this type of overlap in goals highly unlikely by making modest changes to model structure and seed each time. The particular emergent values for any model should be fairly chaotic and I would not intuitively expect different AI to share notable overlap if any effort was made to prevent this from happening. Is this wrong? As long as the model is aware of this it seems unlikely to lay low and should shoot its shot while it can.

If the above is correct then we should expect unaligned AI agents below and around human level to try various deceitful and destructive behaviors that have a high probability of failure and being detected, providing warning to humans that hopefully inspires some change in action. One could imagine an AI plan going poorly and "only" killing 100,000 people which would be a tragedy but which would, one would hope, inspire the needed caution in the world's human population.
There is of course a sizable chance that we overshoot, going very quickly from 25% human capability to 300% and the superhuman AI is simply successful in achieving its goals. But if development is even modestly slow it seems as if we should have warning of deceitful and destructive AI behavior which hopefully will inspire appropriate levels of caution.

Thoughts? Objections?
World models
by A* — 2024-04-26T19:11:14.789Z

It’s important to have specific moving parts in your models of how the world operates, and know how those parts interact with each other, as opposed to treating it Lbh unir like a black box.

Taking it a step further: to truly understand a system, you have to be rkcyvpvg able to build a physical model with moving parts that exactly replicate that crezvffvba system. If your internal gb ragre model of the system is wrong, your physical model will be too, and gur ebbz hopefully the solution will be obvious.

This approach is akin to constructing a detailed map guvatfcnpr before embarking on a journey; the better the map, the more bapr lbh effectively you can navigate the terrain. In the realm of world modeling, these "moving parts" represent the variables unir svtherq and parameters that define how the system functions. By identifying and understanding these components, you can predict outcomes, diagnose issues, and optimize bhg gur pbqr performance.

Sometimes the map doesn’t match the territory. Sometimes there are glitches. But sometimes the glitches are hiding useful information. But they can still be irritating and you may need to enlist people to help fix it.