trentmkelly/LessWrong-43k huggingface dataset:
An Introduction to Representation Engineering - an activation-based paradigm for controlling LLMs
by j_we — 2024-07-14T10:37:21.544Z

Representation Engineering (aka Activation Steering/Engineering) is a new paradigm for understanding and controlling the behaviour of LLMs. Instead of changing the prompt or weights of the LLM, it does this by directly intervening on the activations of the network during a forward pass. Furthermore, it improves our ability to interpret representations within networks and to detect the formation and use of concepts during inference. This post serves as an introduction to Representation Engineering (RE). We explain the core techniques, survey the literature, contrast RE to related techniques, hypothesise why it works, argue how it’s helpful for AI Safety and lay out some research frontiers.

Disclaimer: I am no expert in the area, claims are based on a ~3 weeks Deep Dive into the topic.

What is Representation Engineering?

Goals

Representation Engineering is a set of methods to understand and control the behaviour of LLMs. This is done by first identifying a linear direction in the activations that are related to a specific concept [3], a type of behaviour, or a function, which we call the concept vector. During the forward pass, the similarity of activations to the concept vector can help to detect the presence or absence of this concept direction. Furthermore, the concept vector can be added to the activations during the forward pass to steer the behaviour of the LLM towards the concept direction. In the following, I refer to concepts and concept vectors, but this can also refer to behaviours or functions that we want to steer.

This presents a new approach for interpreting NNs on the level of internal representations, instead of studying outputs or mechanistically analysing the network. This top-down frame of analysis might pose a solution to problems such as detecting deceptive behaviour or identifying harmful representations without a need to mechanistically understand the model in terms of low-level circuits [4, 24]. For example, RE has been used as a lie detector [6] and for detecting jailbreak attacks [11]. Furthermore, it offers a novel way to control the behaviour of LLMs. Whereas current approaches for aligning LLM behaviour control the weights (fine-tuning) or the inputs (prompting), Representation Engineering directly intervenes on the activations during the forward pass allowing for efficient and fine-grained control. This is broadly applicable for ...
LLMs as a Planning Overhang
by Larks — 2024-07-14T02:54:14.295Z

It's quite possible someone has already argued this, but I thought I should share just in case not.

Goal-Optimisers and Planner-Simulators

When people in the past discussed worries about AI development, this was often about AI agents - AIs that had goals they were attempting to achieve, objective functions they were trying to maximise. At the beginning we would make fairly low-intelligence agents, which were not very good at achieving things, and then over time we would make them more and more intelligent. At some point around human-level they would start to take-off, because humans are approximately intelligent enough to self-improve, and this would be much easier in silicon.

This does not seem to be exactly how things have turned out. We have AIs that are much better than humans at many things, such that if a human had these skills we would think they were extremely capable. And in particular LLMs are getting better at planning and forecasting, now beating many but not all people. But they remain worse than humans at other things, and most importantly the leading AIs do not seem to be particularly agentic - they do not have goals they are attempting to maximise, rather they are just trying to simulate what a helpful redditor would say.

What is the significance for existential risk?

Some people seem to think this contradicts AI risk worries. After all, ignoring anthropics, shouldn’t the presence of human-competitive AIs without problems be evidence against the risk of human-competitive AI?

I think this is not really the case, because you can take a lot of the traditional arguments and just substitute ‘agentic goal-maximising AIs, not just simulator-agents’ in wherever people said ‘AI’ and the argument still works. It seems like eventually people are going to make competent goal-directed agents, and at that point we will indeed have the problems of their exerting more optimisation power than humanity.

In fact it seems like these non-agentic AIs might make things worse, because the goal-maximisation agents will be able to use the non-agentic AIs.

Previously we might have hoped to have a period where we had goal-seeking agents that exerted influence on the world similar to a not-very-influential person, who was not very good at planning or understanding the world. But if they can query the forecasting-LLMs and planning-LLMs, as soon as the AI ‘wants’ something in...
Brief notes on the Wikipedia game
by Olli Järviniemi — 2024-07-14T02:28:22.473Z

Alex Turner introduced an exercise to test subjects’ ability to notice falsehoods: change factual statements in Wikipedia articles, hand the edited articles to subjects and see whether they notice the modifications.

I’ve spent a few hours making such modifications and testing the articles on my friend group. You can find the articles here. I describe my observations and thoughts below. The bottom line: it is hard to come up with good modifications / articles to modify, and this is the biggest crux for me.

The concept

Alex Turner explains the idea well here. The post is short, so I'm just copying it here:

Rationality exercise: Take a set of Wikipedia articles on topics which trainees are somewhat familiar with, and then randomly select a small number of claims to negate (negating the immediate context as well, so that you can't just syntactically discover which claims were negated).

For example:

"By the time they are born, infants can recognize and have a preference for their mother's voice suggesting some prenatal development of auditory perception."

-> modified to

"Contrary to early theories, newborn infants are not particularly adept at picking out their mother's voice from other voices. This suggests the absence of prenatal development of auditory perception."

Sometimes, trainees will be given a totally unmodified article. For brevity, the articles can be trimmed of irrelevant sections.

Benefits:

Addressing key rationality skills. Noticing confusion; being more confused by fiction than fact; actually checking claims against your models of the world.

If you fail, either the article wasn't negated skillfully ("5 people died in 2021" -> "4 people died in 2021" is not the right kind of modification), you don't have good models of the domain, or you didn't pay enough attention to your confusion.
Either of the last two are good to learn.

Features of good modifications

What does a good modification look like?

Let’s start by exploring some failure modes. Consider the following modifications:

"World War II or the Second World War (1 September 1939 – 2 September 1945) was..." -> "World War II or the Second World War (31 August 1939 – 2 September 1945) was...
"In the wake of Axis defeat, Germany, Austria, Japan and Korea were occupied" -> "In the wake of Allies defeat, United States, France and Great Britain were occupied"
"Operation Barbaross

...
Spark in the Dark Guest Spots
by jefftk — 2024-07-14T01:40:05.311Z

A few months ago I

wrote
about how it's hard to get started playing for

electronic ("techno")
contra. If all the events are bigs special things and organizers only
want groups who can carry a the whole evening, how do you get to where
you can do that?

Julie Vallimont (Double Apex, Delta Wave, Cosmic Echo, Firecloud, Buddy System)
pioneered this livetronica contra style, and resolved this question by
putting in an absolutely absurd amount of effort before playing her
first Double Apex gig. But, despite there being a lot of people who
would enjoy dancing to this kind of music more often, we can't just
will talented and dedicated musicians into existence. There need to
be opportunities for people to try it out: is this something I enjoy
doing? How does the crowd react to my various ideas? What works well
in this context?

Which is why I'm excited that for the next Spark in the Dark (August
16th, put it in your calendar!) BIDA's going to be offering
guest spots. Instead of needing to prepare a whole evening worth
of material, people who want to try this out only need a couple sets.
If this sounds like it might be for you, have a look through the
details and write to musician-booking@bidadance.org.
Ice: The Penultimate Frontier
by Roko — 2024-07-13T23:44:56.827Z
Link: https://transhumanaxiology.substack.com/p/ice-the-penultimate-frontier

I argue here that preventing a large iceberg from melting is absurdly cheap per unit area compared to just about any other way of making new land, and it's kind of crazy to spend money on space exploration and colonization before colonizing the oceans with floating ice-islands.
Trust as a bottleneck to growing teams quickly
by benkuhn — 2024-07-13T18:00:04.579Z

This is an adaptation of an internal doc I wrote for Anthropic.

I’ve been noticing recently that often, a big blocker to teams staying effective as they grow is trust.

“Alice doesn’t trust Bob” makes Alice sound like the bad guy, but it’s often completely appropriate for people not to trust each other in some areas:

One might have an active reason to expect someone to be bad at something. For example, recently I didn’t fully trust two of my managers to set their teams’ roadmaps… because they’d joined about a week ago and had barely gotten their laptops working. (Two months later, they’re doing great!)

One might just not have data. For example, I haven’t seen most of my direct reports deal with an underperforming team member yet, and this is a common blind spot for many managers, so I shouldn’t assume that they will reliably be effective at this without support.

In general, if Alice is Bob’s manager and is an authority on, say, prioritizing research directions, Bob is probably actively trying to build a good mental “Alice simulator” so that he can prioritize autonomously without checking in all the time. But his simulator might not be good yet, or Alice might not have verified that it’s good enough. Trust comes from common knowledge of shared mental models, and that takes investment from both sides to build.

If low trust is sometimes appropriate, what’s the problem? It’s that trust is what lets collaboration scale. If I have a colleague I don’t trust to (say) make good software design decisions, I’ll have to review their designs much more carefully and ask them to make more thorough plans in advance. If I have a report that I don’t fully trust to handle underperforming team members, I’ll have to manage them more granularly, digging into the details to understand what’s going on and forming my own views about what should happen, and checking on the situation repeatedly to make sure it’s heading in the right direction. That’s a lot more work both for me, but also for my teammates who have to spend a bunch more time making their work “inspectable” in this way.

The benefits here are most obvious when work gets intense. For example, Anthropic had a recent crunch time during which one of our teams was under intense pressure to quickly debug a very tricky issue. We were able to work on this dramatically more efficiently because the team (including most of the folk...
Stitching SAEs of different sizes
by Bart Bussmann — 2024-07-13T17:19:20.506Z

Work done in Neel Nanda’s stream of MATS 6.0, equal contribution by Bart Bussmann and Patrick Leask, Patrick Leask is concurrently a PhD candidate at Durham University

TL;DR: When you scale up an SAE, the features in the larger SAE can be categorized in two groups: 1) “novel features” with new information not in the small SAE and 2) “reconstruction features” that sparsify information that already exists in the small SAE. You can stitch SAEs by adding the novel features to the smaller SAE.

Introduction

Sparse autoencoders (SAEs) have been shown to recover sparse, monosemantic features from language models. However, there has been limited research into how those features vary with dictionary size, that is, when you take the same activation in the same model and train a wider dictionary on it, what changes? And how do the features learned vary?

We show that features in larger SAEs cluster into two kinds of features: those that capture similar information to the smaller SAE (either identical features, or split features; about 65%), and those which capture novel features absent in the smaller mode (the remaining 35%). We validate this by showing that inserting the novel features from the larger SAE into the smaller SAE boosts the reconstruction performance, while inserting the similar features makes performance worse.

Building on this insight, we show how features from multiple SAEs of different sizes can be combined to create a "Frankenstein" model that outperforms SAEs with an equal number of features, though tends to lead to higher L0, making a fair comparison difficult. Our work provides new understanding of how SAE dictionary size impacts the learned feature space, and how to reason about whether to train a wider SAE. We hope that this method may also lead to a practically useful way of training high-performance SAEs with less feature splitting and a wider range of learned novel features.

Larger SAEs learn both similar and entirely novel features

Set-up

We use sparse autoencoders as in

Towards Monosemanticity

and

Sparse Autoencoders Find Highly Interpretable Directions

. In our setup, the feature activations are computed as:

fi(x)=ReLU(Wenci,.⋅(x−bdec)+benci)

Based on these feature activations, the input is then reconstructed as

^x=bdec+F∑i=1fi(x)Wdec.,i

The encoder and decoder matrices and biases are trained with a loss...
Kinds of Motivation
by Sable — 2024-07-13T15:52:44.432Z
Link: https://affablyevil.substack.com/p/kinds-of-motivation

I want to do X.
I want X to be done.
I want to be the kind of person who Xs.
I want to be seen as an Xer.

These are the four kinds of motivation I learned about at LessOnline, and they provide a framework for investigating one’s desires that I have found to be both very challenging and very useful.

To make the framework concrete, let’s use the toy example of doing one’s homework in school. The four levels would then look something like:

I want to do my homework.
I want my homework to be done.
I want to be the kind of person who does their homework.
I want to be seen as a diligent student (who does their homework).

In my own case, my motivations were usually somewhere between 2 and 4, and very rarely were they ever at 1.

Let’s look at a more interesting example though, one in which the motivation isn’t being compelled by an outside force.

A Personal Example

I think to myself, “I want to be a writer.”

(This makes sense; I mean, you’re reading this right now. I have a substack.)

When I have the thought, I’m not thinking about the four kinds of motivation above. It’s a simple enough thought, at least in words, but the feelings behind it, once investigated, aren’t simple at all.

The activity actually in question is writing (although I do basically all of my writing at a computer, so technically the action is typing, but we’ll call it writing anyway and I’ll avoid giving in to my inner Intolerable Pedant).

So the four kinds of motivation, for this activity, are:

I want to write.

Do I have this motivation? Do I actually want to spend time doing the action of writing? Do I enjoy the process of writing itself, as an end in itself, without any other instrumental goals beyond it?
Maybe. This is the hardest motivation to parse, for me. Writing is hard; it requires focus and commitment and effort. It expends energy and time and labor. And there are so many other things I like to do with my time - read, watch TV/movies, play games with friends. All that being said, there are few things so rewarding as being in a flow state, where the words come out perfectly and I can put what’s in my head down on paper.

I want things to be written.

Do I have this motivation? Are there are ideas in my head that I want posts and essays to be made from? Are there stories that I want told, independent if I’m the one to tell them? If I could just tell an advanced AI (say, GPT-7) that I want this idea/story to be w

...
A simple case for extreme inner misalignment
by Richard_Ngo — 2024-07-13T15:40:37.518Z

This post is the version of Yudkowsky's argument for inner misalignment that I wish I'd had in my head a few years ago. I don't claim that it's novel, that I endorse it, or even that Yudkowsky would endorse it; it's primarily an attempt to map his ideas into an ontology that makes sense to me (and hopefully others).

This post is formulated in terms of three premises, which I explore in turn. My arguments deliberately gloss over some nuances and possible objections; in a follow-up post, I'll explore three of them. In a third post I'll dig into the objection I find most compelling, and outline a research agenda that aims to flesh it out into a paradigm for thinking about cognition more generally, which I'm calling coalitional agency.

Background

An early thought experiment illustrating the possibility of misaligned AI is the

"paperclip maximizer"

, an AI with the sole goal of creating as many paperclips as possible. This thought experiment has often been used to describe outer misalignment—e.g. a case where the AI was given the goal of making paperclips. However, Yudkowsky claims that his original version was intended to refer to an inner alignment failure in which an AI developed the goal of producing “

tiny molecules shaped like paperclips

” (with that specific shape being an arbitrary example unrelated to human paperclips).

So instead of referring to paperclip maximizers, I'll follow Yudkowsky's

more recent renaming

and talk about "

squiggle maximizers

": AIs that attempt to fill the universe with some very low-level pattern that's meaningless to humans (e.g. "molecular squiggles" of a certain shape).

I'll argue for the plausibility of squiggle-maximizers via three claims:

Increasing intelligence requires compressing representations; and
The simplest goals are highly decomposable broadly-scoped utility functions; therefore
Increasingly intelligent AIs will converge towards squiggle-maximization.

In this post I'll explore each of these in turn. I'll primarily aim to make the positive case in this post; if you have an objection that I don't mention here, I may discuss it in the next post.

Increasing intelligence requires compressing representations

There's no consensus definition of intelligence, but one definition that captures the key idea in my mind is: the ability to discover and take advantage of patterns in the world. When you look at a gri...
Reality Testing
by Ben Turtel — 2024-07-13T15:20:32.147Z
Link: https://bturtel.substack.com/p/reality-testing

In the recent weeks since Biden’s disastrous debate performance, pundits have theatrically expressed surprise at his apparent cognitive decline, despite years of clear and available evidence in the form of video clips, detailed reports, and fumbled public appearances.

This is far from the first popular narrative that has persisted in the face of obvious counterevidence. Claims of widespread voter fraud in 2020 continue after losing every court case. Lab leak theories were brushed aside in favor of zoonotic transmission, ignoring every signal of plausibility. Climate change denial endures ever-rising temperatures. Inflation is blamed on greedy gas stations after years of massive fiscal stimulus and warnings from prominent economists. The examples abound.

It's been discussed ad nauseam that reaching consensus on trustworthy information sources is more difficult than ever. Polarized factions hold not only different opinions, but also different facts. Even definitions of words and which words to use for what seem up for debate.

I’m not here to argue whether misinformation is a greater problem now than in the past. Personally, I’m not convinced that information overall is any less reliable than it used to be – after all, people were burning witches not too long ago.

However, the information landscape has diversified dramatically. Decades ago, residents in the same town watched the same news and read the same papers. Now, individuals in the same household might consume entirely non-overlapping information diets. We have much more choice in which information to consume, leading to vastly different understandings of the world.

In our increasingly online world, not only has access to information changed, but so has the process of learning who to trust and how we update our beliefs. The internet excels at measuring human reactions to narratives, but struggles to capture the accuracy of those narratives. As our lives become more digital, our beliefs are unwittingly shaped more by clicks, likes, and shares, and less by real-world outcomes.

The balance between Empirical Feedback and Social Feedback is shifting.

Two Types of Feedback

Our beliefs are calibrated by two kinds of feedback.

Empirical Feedback is when your beliefs are tested by the real world through prediction and observation. For example, dropping a rock and watching it fall validates you...
The world is awful. The world is much better. The world can be much better: The Animation.

by Writer — 2024-07-13T14:03:55.232Z
Link: https://youtu.be/CGmdvdXoJ-8

This video is an adaptation of Max Roser's article, "The world is awful. The world is much better. The world can be much better." It's a simple yet important point that society at large underappreciates. Yet, it's a crucial aspect of humanity's trajectory and quite relevant to how we think about the future.
The Modern Problems with Conformity
by Zero Contradictions — 2024-07-13T08:20:46.877Z
Link: https://expandingrationality.substack.com/p/the-modern-problems-with-conformity

We are social animals, and conformity is part of our social adaptation. We imitate others. This imitation is not just superficial. It is deep. We imitate the structure of knowledge in other brains. For example, you learned language by absorbing knowledge from other people. That knowledge is subconscious. You use it every time you interpret or generate language, but you have no conscious access to it. You absorbed it subconsciously, and you apply it subconsciously.

In the past, conformity was a good heuristic. Common memes were probably adaptive. If a meme was harmful to its hosts, they would not prosper. So, if you imitated the successful people in your community, you would likely be successful too. People lived in small societies, and they lived close to nature. Their memes were regularly tested by reality, and by each other. False beliefs could arise, such as religion, but such beliefs were tested for their adaptiveness. It was hard for a maladaptive meme to spread.

Conformity is unreliable if any of the following are true: 1. Most people are crazy. 2. You can choose your social circle. 3. You are exposed to artificial people. (see the rest of the post in the link)