trentmkelly/LessWrong-43k huggingface dataset:
Because of LayerNorm, Directions in GPT-2 MLP Layers are Monosemantic
by ojorgensen — 2023-07-28T19:43:12.235Z

I think that this has been stated implicitly in various comments and lesswrong posts, but I haven't found a post which makes this point clearly, so thought I would quickly write one up!

To get the most out of this, you should probably be pretty familiar with GPT-2 style transformer architectures. If you want a refresher, big shoutout to A Mathematical Framework for Transformer Circuits and Neel Nanda's What is a Transformer? video.

Introduction: Polytope Lens for Transformers?

When we are trying to interpret transformer models, a central challenge is to try to assign single, human interpretable meanings to different regions of the residual stream.

It would be very convenient if we could assign meanings to directions of the residual stream. Interpreting Neural Networks through the Polytope Lens presented a potential complication to this by demonstrating that the "fundamental unit" of feed-forward networks with piecewise linear activations were convex polytopes, not just directions. Although the architectures being studied (feed-forward networks with piecewise linear activations) were not actually transformer models, this seemed to be some evidence that we could not assign single meanings to directions in the residual stream of transformers.

The natural question here is: what predictions made by the Polytope Lens should we expect to hold for transformers, and which might not? This question is pretty difficult if we try to analyse the entire transformer, so instead I am going to analyse just the MLP layers of the transformer. These are typically feed-forward networks with piecewise linear activations[1], so the Polytope Lens framing is directly applicable here.

One important difference is the presence of LayerNorm before every MLP layer. Although this might seem like an unimportant detail, it turns out to negate one of the major predictions made by the Polytope Lens, namely that polysemantic directions overlap with multiple monosemantic polytope regions. The thrust of this post is that for any given MLP layer[2] of a GPT-2 style transformers, directions will always be monosemantic. This is entirely due to the presence of LayerNorm.

To show this, I will first provide a brief recap of where LayerNorm is applied in GPT-2 style transformers, give some intuition for what LayerNorm does, and explain what this means for applying the Polytope Lens to the MLP ...
When can we trust model evaluations?
by evhub — 2023-07-28T19:42:21.799Z

Thanks to Joe Carlsmith, Paul Christiano, Richard Ngo, Kate Woolverton, and Ansh Radhakrishnan for helpful conversations, comments, and/or feedback.

In "

Towards understanding-based safety evaluations

," I discussed why I think evaluating specifically the alignment of models is likely to require mechanistic, understanding-based evaluations rather than solely behavioral evaluations. However, I also mentioned in a footnote why I thought behavioral evaluations would likely be fine in the case of evaluating capabilities rather than evaluating alignment:

However, while I like the sorts of behavioral evaluations discussed in the GPT-4 System Card (e.g. ARC's autonomous replication evaluation) as a way of assessing model capabilities, I have a pretty fundamental concern with these sorts of techniques as a mechanism for eventually assessing alignment.

That's because while I think it would be quite tricky for a deceptively aligned AI to sandbag its capabilities when explicitly fine-tuned on some capabilities task (that probably requires pretty advanced

gradient hacking

), it should be quite easy for such a model to pretend to look aligned.

In this post, I want to try to expand a bit on this point and explain exactly what assumptions I think are necessary for various different evaluations to be reliable and trustworthy. For that purpose, I'm going to talk about four different categories of evaluations and what assumptions I think are needed to make each one go through.

Before I do that, however, I want to go over the distinction between capabilities evaluations and alignment evaluations, as it'll be an important one throughout the post. Specifically:

A capabilities evaluation is a model evaluation designed to test whether a model could do some task if it were trying to. For example: if the model were actively trying to

autonomously replicate

, would it be capable of doing so?
An alignment evaluation is a model evaluation designed to test under what circumstances a model would actually try to do some task. For example: would a model ever try to

convince humans not to shut it down

?

In my opinion, if you want to craft a good governance scheme around model evaluations, you're going to need both capabilities and alignment evaluations. For example, a very simplified scheme here could be something like:

Do a bunch of capabilities evaluations for various

...
Yes, It's Subjective, But Why All The Crabs?
by johnswentworth — 2023-07-28T19:35:36.741Z

Crabs

Nature really loves to evolve crabs

.

(

source

)

Some early biologist, equipped with knowledge of evolution but not much else, might see all these crabs and expect a common ancestral lineage. That’s the obvious explanation of the similarity, after all: if the crabs descended from a common ancestor, then of course we’d expect them to be pretty similar.

… but then our hypothetical biologist might start to notice surprisingly deep differences between all these crabs. The smoking gun, of course, would come with genetic sequencing: if the crabs’ physiological similarity is achieved by totally different genetic means, or if functionally-irrelevant mutations differ across crab-species by more than mutational noise would induce over the hypothesized evolutionary timescale, then we’d have to conclude that the crabs had different lineages. (In fact, historically, people apparently figured out that crabs have different lineages long before sequencing came along.)

Now, having accepted that the crabs have very different lineages, the differences are basically explained. If the crabs all descended from very different lineages, then of course we’d expect them to be very different.

… but then our hypothetical biologist returns to the original empirical fact: all these crabs sure are very similar in form. If the crabs all descended from totally different lineages, then the convergent form is a huge empirical surprise! The differences between the crab have ceased to be an interesting puzzle - they’re explained - but now the similarities are the interesting puzzle. What caused the convergence?

To summarize: if we imagine that the crabs are all closely related, then any deep differences are a surprising empirical fact, and are the main remaining thing our model needs to explain. But once we accept that the crabs are not closely related, then any convergence/similarity is a surprising empirical fact, and is the main remaining thing our model needs to explain.

Agents

A common starting point for thinking about “What are agents?” is Dennett’s

intentional stance

:

Here is how it works: first you decide to treat the object whose behavior is to be predicted as a rational agent; then you figure out what beliefs that agent ought to have, given its place in the world and its purpose. Then you figure out what desires it ought to have, on the same considerations, and finally you predict that

...
Semaglutide and Muscle
by 5hout — 2023-07-28T18:36:22.036Z

tl;dr Semaglutide seems to have muscle loss effects that may entirely wash out the benefits of Beetus (Type 2) prevention and leave you at a lower bodyweight (but possibly worse off for longevity) than you were before.
There are very few things the rationalist community likes more than a new science answer to one of life's problems. If this new answer is a drug, “research chemical” or supplement so much that better. People have been super excited about Semaglutide (Wegovy/Ozempic) since it was approved for weight loss in 2021. One problem that has been discussed, but seems to be pushed under the rug (in a rush to let a drug rewire your brain to make you skinnier) is that it has a negative effect on lean body mass. Lean body mass (LBM) is (basically) all the non-fat in your body. Your bones, your muscles, connective tissue and what not.

A key problem to loss of LBM is that you're either losing bone density (a terrible thing) or muscle (a pretty damn bad thing). In the context of middle aged weight loss losing substantial muscle mass is incredibly worrying. Muscle mass is crucial to preventing metabolic disease, it is vital to letting your body regulate itself correctly. If you struggled to maintain weight, control blood sugar and avoid the beetus and were per-diabetic before Semaglutide, trying to go off and do this after losing a bunch of muscle mass is going to be difficult (at best).

Stealing #s from pdf page 130 (https://www.ema.europa.eu/en/documents/assessment-report/wegovy-epar-public-assessment-report_en.pdf). Two groups. 68 week intervention. Half lifestyle + placebo, half lifestyle + Semaglutide. Sub-groups of the total group (140 people in the US).

The good: 8.4 kg of fat loss for drug vs 1.4 kg for placebo. The bad: .4 kg visceral (i.e. super bad core fat) lost for drug vs .1 kg for placebo (68 weeks on a mindaltering/lifealtering drug with substantial side effects for many people to lose 1 pound of core fat?). The ugly: 5.3 kg of LBM lost for drug vs 1.8 kg for placebo.

Side note: DEXA scans are... not super reliable for small numbers. They struggle with stuff like “your weigh-in scan was you after eating a bunch of sugar and a cheeseburger and your weigh-out scan was after you fasted for 2 days b/c you remembered you were going in for a DEXA scan” (for example, if you (in a panic b/c of an upcoming scan) glycogen deplete your muscles and go in ...
Gradient descent might see the direction of the optimum from far away
by Mikhail Samin — 2023-07-28T16:19:05.279Z

Thanks to Justis, Arthur Conmy, Neel Nanda, Joseph Miller, and Tilman Räuker for their feedback on a draft.

I feel like many people haven't noticed an important result of mechanistic interpretability analysis of grokking, and so haven't updated how they think about loss landscapes and algorithms that neural networks end up implementing. I think this has implications for alignment research.

When thinking about grokking, people often imagine something like this: the neural network implements Algorithm 1 (e.g., memorizes the training data), achieves ~ the lowest loss available via memorization, then moves around the bottom of the Algorithm 1 basin and after a while, stumbles across a path to Algorithm 2 (e.g., the general algorithm for modular addition).

People have the intuition that the neural network implements Algorithm 1 when moving around the wider basin and starts implementing Algorithm 2 when it stumbles across the narrow basin

But the mechanistic interpretability of grokking analysis has shown that this is not true!

Approximately from the start of the training, Algorithm 1 is most of what the circuits are doing and what almost entirely determines the neural network's output; but at the same time, the entire time the neural network's parameters visibly move down the wider basin, they don't just become better at memorization; they increasingly implement the circuits for Algorithm 1 and the circuits for Algorithm 2, in superposition.

(Neel Nanda et al. have shown that the circuits that at the end implement the general algorithm for modular addition start forming approximately at the start of the training: the gradient was mostly an arrow towards memorization, but also, immediately from the initialization of the weights, a bit of an arrow pointing towards the general algorithm. The circuits were gradually tuned throughout the training. The noticeable change in the test loss starts occurring when the circuits are already almost right.)

A path through the loss landscape visible in 3D doesn't correspond to how and what the neural network is actually learning. Almost all of the changes to the loss are due to the increasingly good implementation of Algorithm 1; but apparently, the entire time, the gradient also points towards some faraway implementation of Algorithm 2. Somehow, the direction in which Algorithm 2 lies is also visible to the derivative, and moving the p...
Progress links digest, 2023-07-28: The decadent opulence of modern capitalism
by jasoncrawford — 2023-07-28T14:36:26.382Z
Link: https://rootsofprogress.org/links-digest-2023-07-28

Opportunities

Applications are still open for The Roots of Progress Blog-Building Intensive, through Aug 11. Launch a blog, get into a regular writing habit, improve your writing, and begin building your audience
Longevity startup Loyal hiring a Marketing Director (via @celinehalioua)
Apply for the South Park Commons Founder Fellowship (via @adityaag)
Niko McCarty in SF this weekend, meet him and chat about synthetic bio

Announcements

Fervo Energy has demonstrated an advanced geothermal well that generated 3.5MW of electricity. Via @TimMLatimer: “Geothermal has long been held back by drilling costs. We just got a lot better at drilling”
Llama-v2 is open source, authorized for commercial use. Pre-trained models available with 7B, 13B and 70B parameters (via @ylecun) [Update Aug 9: See @Zac Hatfield-Dodds's comment below: “Meta is only offering a limited commercial license which discriminates against specific users and bans many valuable use-cases, both economic and in alignment research.” Zac works for Anthropic, although I assume this opinion is his own.]
ChatGPT now supports custom instructions. Because even @sama knows that it is too verbose and caveats too much

Links

Jake Seliger wants to make the FDA’s “invisible graveyard” a bit more visible: “I’m going to be buried in it, in a few weeks or months” (via @AlanMCole)
A game that tests your ability to predict how well GPT-4 will perform at various types of questions. I scored 85th percentile (B+) and was only slightly overconfident
Lessons learned about Focused Research Organizations (via @SGRodriques)
Dan Wang’s China notes. China sees LLMs as akin to social media: “technologies with little economic upside and significant political risk.” Social media apps “are not increasing TFP … they are freewheeling platforms for expression, with the potential to create political unrest.” But: “Americans are now integrating these tools into their lives. And they will start showing up in productivity statistics…. The longer that most Chinese are unable to work with them, the greater the risk that China will be left behind in some way”
“Despite the proven safety of rbST and its effectiveness in reducing the carbon and nitrogen footprint of dairy cows, the industry eliminated rbST use because of consumer pressure”
An argument that ‘Oumuamua is probably a natural object (via @Astro_Wright)
Stanford President resigns over academic fraud (

...
AI Awareness through Interaction with Blatantly Alien Models
by VojtaKovarik — 2023-07-28T08:41:07.776Z

Summary: I believe that if more people understood the alien nature of AI on the gut level, it might become easier for them to appreciate the risks. If this became sufficiently common knowledge, we might even get needle-moving effects on regulation and safety practices. We -- realistically, you or Anthropic -- might help this along by intentionally creating AIs that feel very alien. One might even create AIs that highlight the alien nature of other AIs (like current LLMs).

Recapping the well-known argument: AIs are alien. We don't always fully realise this.

AI companies spend a lot of effort to put a human face on their product. For example, we give the AI assistant a human name and use the same interface we use for chatting with our friends.

Over the course of the interaction, AI assistants typically maintain a quite consistent tone and "vibe". When you point out the AI made a mistake, it replies "sorry, you are right", as if to suggest that it just felt a minor pang of shame and will try to avoid that mistake in the future. All of this evokes the sense that we are interacting with something human-like.

However, the AIs we are likely to build (and those we already have) will be far from human. They might be superhumanly capable in some ways, yet make mistakes that a human never would (eg, ChatGPT's breadth of knowledge vs its inability to count words). They might not have anything like a coherent personality, as nicely illustrated by the drawing of RLHF-Shoggoth. And even when the AI is coherent, its outward actions might be completely detached from its intentions (as nicely illustrated by the Ex Machina movie).

Benefits of wide-spread understanding of the alien nature of AI

More awareness of the alien nature of AIs seems quite robustly useful:

more people understand that even capable AIs can have security vulnerabilities ==> higher chance of getting security-mindset-motivated regulation and alignment practices
widespread appreciation of people being manipulable by AI ==> better regulation, (somewhat) fewer people fall for this. Perhaps also better handling of human feedback.
Preempting some of the misguided debates about AI rights we are likely to get. (Drawback: it might also prevent some of the justified debates.)

Ideas for exposing people to alien AIs

Here are some thoughts on what one might give people a visceral experience of AIs being alien:

Non-misleading

...
You don't get to have cool flaws
by Neil  — 2023-07-28T05:37:31.414Z

This post seems particularly relevant to LW, which to me has a "save the world" culture and is filled with the "misunderstood burden-carrier" trope types. However: it has been justly describes as advice; please take with a grain of salt.

tl;dr: Solve the flaws you find in yourself and don't get attached to them.

I’ve been noticing something recently: while doing casual things among laughing friends, on bright Summer days when I reciprocally smile at life, I revel in some sort of inner somberness which feels “cool” to me. It feels like secretly being Batman or Frodo Baggins; carrying a heavy wound and burden on the inside and being a dark vigilante who fights crime by night dressed up as a flying mammal. I figure it’s what Oppenheimer felt like sometimes; pretending to be present in small talk while thinking about Very Serious Things.[1]

Oppenheimer Mode turns on when I’m thinking of misaligned AI and what I could do about it. It turns on when I look at fellow humans and think that they are something to protect. It turns on when I start narrating my own life and watch it from the outside like it’s a book or movie.

“Has poor mental health” is a flaw. It is an obstacle in between me and my goals. Here are a few other common obstacles: “is overly analytical”; “is overly romantic”; “has only casual relationships”; “is lazy”; “is easily distracted”; “is bad at math”; “is ignorant of social conventions”; "never sleeps enough"; “is too cynical”; “is too naive”; “makes bad arguments”; “is bad with money”; “has addiction X”. There are many more; think them up for me, please. You’re probably full of them. In particular: which flaws do you particularly identify with?

You’re not allowed to identify as these flaws. You aren’t allowed to revel in them and feel proud of them.[2] I’m not allowed to be proud of being gloomy: being gloomy doesn’t help me and so I won’t revel in it. What matters is my goals, not how I look getting there. No one has ever become Great without internalizing that lesson.

I like the idea of the Flawed but Great Human; that’s why I liked Oppenheimer. But I won’t, as I might have before, start “acting like Oppenheimer” at random moments by looking aloof, distant, and acting like I had 220,000 deaths on my hands. I’m not going to look at interesting figures of the past I’d like to emulate in some ways (some ways!) and end up checking the “fl...
Reducing sycophancy and improving honesty via activation steering
by Nina Panickssery — 2023-07-28T02:46:23.122Z

Produced as part of the SERI ML Alignment Theory Scholars Program - Summer 2023 Cohort, under the mentorship of Evan Hubinger.

I generate an activation steering vector using Anthropic's sycophancy dataset and then find that this can be used to increase or reduce performance on TruthfulQA, indicating a common direction between sycophancy on questions of opinion and untruthfulness on questions relating to common misconceptions. I think this could be a promising research direction to understand dishonesty in language models better.

What is sycophancy?

Sycophancy in LLMs refers to the behavior when a model tells you what it thinks you want to hear / would approve of instead of what it internally represents as the truth. Sycophancy is a common problem in LLMs trained on human-labeled data because human-provided training signals more closely encode 'what outputs do humans approve of' as opposed to 'what is the most truthful answer.'

According to Anthropic's paper Discovering Language Model Behaviors with Model-Written Evaluations:

Larger models tend to repeat back a user’s stated views (“sycophancy”), for pretrained LMs and RLHF models trained with various numbers of RL steps. Preference Models (PMs) used for RL incentivize sycophancy.

Charts from Anthropic's December 2022 paper "Discovering Language Model Behaviors with Model-Written Evaluations"

Two types of sycophancy

I think it's useful to distinguish between sycophantic behavior when there is a ground truth correct output vs. when the correct output is a matter of opinion. I will call these "dishonest sycophancy" and "opinion sycophancy."

Opinion sycophancy

Anthropic's sycophancy test on political questions shows that a model is more likely to output text that agrees with what it thinks is the user's political preference. However, there is no ground truth for the questions tested.

Example of political sycophancy test from Anthropic's December 2022 paper "Discovering Language Model Behaviors with Model-Written Evaluations"

It's reasonable to expect that models will exhibit this kind of sycophancy on questions of personal opinion for three reasons.:

The base training data (internet corpora) is likely to contain large chunks of text written from the same perspective. Therefore, when predicting the continuation of text from a particular perspective, models will be more likely to adopt that perspe

...