trentmkelly/LessWrong-43k huggingface dataset:
How polysemantic can one neuron be? Investigating features in TinyStories.
by Evan Anders — 2024-01-16T19:10:19.646Z
Link: https://evanhanders.blog/2024/01/16/312/

Summary

I use a pre-trained Sparse Autoencoder (SAE) to examine some features in a 33M-parameter, 2-layer TinyStories language model. I look at how frequently SAE features occur in the dataset and examine how the features are distributed over the neurons in the MLP that the SAE was trained on. I find that one neuron is the "main" direction that over 400 features are pointing in, and label some of those features. But I find that the most interpretable of these features are not mostly aligned with any neuron. I close with a few open questions that this investigation raised (and perhaps some of these open questions have already been answered by research out there!).

Intro

Neurons in neural networks are polysemantic, not monosemantic. This means that completely unrelated features (e.g., cats' faces and cars) in a dataset can cause the same neuron in a network to fire. This is problematic. Neurons are easy to examine and it would be ideal if we, humans, could interpret them and predict their behavior (and explain it to someone else like they're five). But we can't, so we need to understand why neurons are polysemantic and we need a different lens for examining and understanding AI models.

Researchers at e.g., EleutherAI and Anthropic's interpretability team, came up with a technique a few months ago that involves expressing a neuron activation vector in terms of a linear sum of features,

xj≈b+∑ifi(xj)ei,

where xj is the activation vector for datapoint j, fi(xj) is the activation of feature i, whose direction is the unit vector ei in activation space. This concept wasn't new, but what was new was the use of a sparse autoencoder (SAE) trained on the neuron activations to recover the features. What's more -- many of the features recovered by SAEs are interpretable to humans!

This is really exciting, and offers the potential to intuitively understand a piece of transformers that are really hard to interpret (MLPs). I want to get my hands dirty playing with SAEs, and just to give me a place to start that doesn't involve looking at thousands of features, I'm specifically interested in the question: how many features can a neuron be tied up in? More specifically, if I pick a neuron in a model, for how many features is that neuron the maximally aligned neuron, and do those features have things in common?

A TinyStories SAE

TinyStories is a f...
Applying AI Safety concepts to astronomy
by Faris — 2024-01-16T18:29:55.652Z

Background: This is a general overview of AI safety applied to astronomy, created as a part of the AI Safety North Alignment course 2023. It is intended as a simple overview for people without intermediate AI knowledge. The contents of this text should be seen as a suggestion for how AI safety could be more clearly defined in research, from the perspective of a student with minimal experience in academia or AI research.

Introduction

My reason for publishing this on the LessWrong forum is to get some feedback from others if my reasoning makes sense. I wanted to explore the idea of a better standard in research in relation to AI safety. Personally I feel like finding how the researchers mitigated risk from using AI shouldn't be so complicated. I am a physics student and do not have intermediate knowledge about AI, but learned a bit during the AI Safety North Alignment course. Reading articles in the field of astronomy where AI was used, I found that the explanation of how the risks were mitigated were confusing. One could argue that for advanced research you need to have knowledge from the field the research is from to understand it. In my opinion, since more fields are starting to use AI, the most rational approach would be for the explanation to be more straightforward, so that even a person who doesn't know astronomy so well can understand it. My goal is to try an argue that researchers when using AI for research in astronomy should be more straightforward about how they mitigated risks/errors, and I will do this by applying AI safety concepts. I am not sure if my reasoning makes sense or if this already is common in other fields, but an attempt was made.

Some applications of AI in astronomy

Astronomers have been using AI for decades, such as in 1990 when astronomers from the University of Arizona were among the first to use a neural network, a type of machine learning algorithm model taking inspiration from the brain, to study the shapes of galaxies. Since then, AI has spread into every field of astronomy, and for example generative AI was used to make the first image of a black hole two times sharper. As telescopes improve the amount of data collected increases, making AI algorithms the most viable solution to ever hope going through the data (Impey, 2023).

It is also useful in cosmology, a sub-field of astro...
Managing catastrophic misuse without robust AIs
by ryan_greenblatt — 2024-01-16T17:27:31.112Z

Many people worry about catastrophic misuse of future AIs with highly dangerous capabilities. For instance, powerful AIs might substantially lower the bar to building bioweapons or allow for massively scaling up cybercrime.

How could an AI lab serving AIs to customers manage catastrophic misuse? One approach would be to ensure that when future powerful AIs are asked to perform tasks in these problematic domains, the AIs always refuse. However, it might be a difficult technical problem to ensure these AIs refuse: current LLMs are possible to jailbreak into doing arbitrary behavior, and the field of adversarial robustness, which studies these sorts of attacks, has made only slow progress in improving robustness over the past 10 years. If we can’t ensure that future powerful AIs are much more robust than current models[1], then malicious users might be able to jailbreak these models to allow for misuse. This is a serious concern, and it would be notably easier to prevent misuse if models were more robust to these attacks. However, I think there are plausible approaches to effectively mitigating catastrophic misuse which don't require high levels of robustness on the part of individual AI models.

(In this post, I'll use "jailbreak" to refer to any adversarial attack.)

In this post, I'll discuss addressing bioterrorism and cybercrime misuse as examples of how I imagine mitigating catastrophic misuse[2] for a model deployed on an API. I'll do this as a nearcast where I suppose that scaling up LLMs results in powerful AIs that would present misuse risk in the absence of countermeasures. I think The approaches I discuss won't require better adversarial robustness than exhibited by current LLMs like Claude 2 and GPT-4. I think that the easiest mitigations for bioterrorism and cybercrime are fairly different, because of the different roles that LLMs play in these two threat models.

The mitigations I'll describe are non-trivial, and it's unclear if they will happen by default. But regardless, this type of approach seems considerably easier to me than trying to achieve very high levels of adversarial robustness. I'm excited for work which investigates and red-teams methods like the ones I discuss.

Note that the approaches I discuss here don't help at all with catastrophic misuse of open source AIs. Distinct approaches would be required to address that problem, such as ensuri...
What are the most common social insecurities?
by Chris Lakin — 2024-01-16T17:24:59.761Z

What do you think the most common social insecurities are?

I'm doing rationality research on a grant, and answers will help my research.

I'm looking for a diversity of answers, please try not to look at the existing comments before responding.

(Note: I'm deliberately not sharing my definition of "social insecurity" because I also want to see what people project.)
Why wasn't preservation with the goal of potential future revival started earlier in history?
by Andy_McKenzie — 2024-01-16T16:15:08.550Z

Cross-posted from my blog, Neurobiology Notes.

John Hunter (1728–1793) did not have an especially promising start to his academic life. He was born the youngest of 10 children to a family living in the countryside near Glasgow. They lived in a two bedroom cottage and the children slept in box beds that were pulled out of the walls every night. He was stubborn, hated school, did not like to be taught reading or writing, would skip classes whenever he could, and quit formal education altogether at 13, the same year his father died. He said that he “totally rejected books,” instead preferring to gain practical knowledge first hand. He spent his time helping with the family farm. When he was 20, he made the fateful decision to join his brother William Hunter's anatomy school in London as an assistant. He went on to discover that

maternal and fetal circulations are separate

, invent the technique of proximal ligation to treat aneurysms,

either inoculate himself or someone else

with venereal disease purely in the name of science, coordinate the first documented artificial insemination,

propose the gradual formation of new species due to random variations

70 years before Darwin, create a school providing lectures in physiology, make enemies with all of the other surgeons at his hospital, almost die when he was attacked by one of his many exotic animals, amass a huge collection of specimens that he spent nearly all his money on and that

remains in London today

, and become the person widely considered the founder of modern scientific surgery.

a photo from the Hunterian museum in London

I learned this all from Wendy Moore's excellent biography of John Hunter,

The Knife Man

:

The Knife Man by Wendy Moore

Although I’m a closet Anglophile, the main reason I picked this book up is because Hunter also seems to have been one of the first people, if not the first person, to seriously research suspended animation. Suspended animation is a hypothetical procedure in which a person or other animal could be preserved for a long period of time in a way that the procedure is known to be reversible, allowing for reanimation at the time of one's choosing.

Suspended animation is not the same as cryonics

, because in cryonics, it is not known whether the preservation will ever be reversible, so a cryonics procedure relies on the possibility of bootstr...
Why are people unkeen to immortality that would come from technological advancements and/or AI?
by Gabi QUENE — 2024-01-16T14:23:21.108Z

Nearly everyone with whom I have talked about the mindblowing possibilities of a friendly ASI -people that hadn't heard about it before-, I have seen the same reaction: the person is skeptical and rejects the idea of immortality: "Humans are made to die", "what is the value of life then?", and sometime: "I want to die".

Why does that happen?

I can't understand their reaction.

If you are like them, let us know your arguments.
Dealing with Awkwardness
by Jonathan Moregård — 2024-01-16T12:32:16.997Z
Link: https://honestliving.substack.com/p/dealing-with-awkwardness

This post introduces my model of awkwardness and the methods I’ve used to overcome it.[1]

Awkwardness used to be a big struggle in my life - limiting my agency through excessive risk aversion. I can vividly recall a conversation where I was visibly shaking and stammering throughout - I had let the tension build to unbearable levels.

I hope that the ideas outlined in this post will help you handle awkwardness in your own life with more grace.

The Model

I think of awkwardness as a feedback loop between exaggerated risk aversion & judgemental self-monitoring. The model has some support in the literature - there’s some evidence that risk aversion and social anxiety

is connected

. Even better, some mean researchers have shown that social judgements and increased awareness of social dynamics tend to make people

feel awkward

.[2]

The core of my model is a feedback loop, shown in this diagram:

Monitoring your actions while also judging them makes small mistakes seem bigger. While performing everyday tasks, it’s standard to make small errors every 5-10 seconds, correcting as you go.[3]

When you feel awkward, such small slip-ups are exaggerated. You judge yourself for making them, and “tense up” to avoid future mistakes. Going into risk avoidance leads to increased vigilance.

In this vigilant state, you notice even more mistakes - and blow them out of proportion. Your focus turns to your inner turmoil and judgement, leading to a disconnect from the things going on around you. You try to consciously execute actions that normally run on instinct.[4]

This set of reactions leads to increased fumbling, together with a judgemental exaggeration of any (perceived) mistakes.

Awkward spiral ahead!

The Ugh Field

You’re meeting up with a date. As you present yourself, you start stumbling over words, your hands sweating. ‘Why am I nervous? This is silly’, you think to yourself. As the tension increases, you notice your date staring at you, looking puzzled. You clear your throat - does it always sound this loud? - and say…

Getting sucked into a world coloured by your self-judgement is unpleasant.[5] Many people respond to awkward experiences by resolving to never put themself in such a situation again - creating an

ugh field

around awkwardness that blocks them from thinking about it rationally.

With the root causes hidden in an ugh field, it’s easy to blame awkwardness o...
The impossible problem of due process
by mingyuan — 2024-01-16T05:18:33.415Z

I wrote this entire post in February of 2023, during the fallout from the TIME article. I didn't post it at the time for multiple reasons:

because I had no desire to get involved in all that nonsense
because I was horribly burned out from my own community conflict investigation and couldn't stand the thought of engaging with people online
because I generally think it's bad to post on the internet out of frustration or outrage

But after sitting on it for a full year, I still think it's worth posting, so here it is. The only edits I have made since February 16th, 2023, were to add a couple interstitial sentences for clarity, and change 'recent articles' to 'articles from February 2023'. So, it's not intended to be commenting on anything more recent than that.

I am precommitting to not engaging with any comments, because I am mostly offline and I think that is good. I probably won't even look at this post again for several weeks. Do what you will. Here is the post:

Note: I am erring on the side of not naming any names in this article. There is one exception for the sake of clarity.

In my time overseeing the global rationalist community, living in the Bay community, and just generally being a person, I've seen a lot of people face up to complicated conflicts.

People often get really mad at each other for mishandling these cases, and will sometimes publicly point to these failures as reasons to condemn a person or group. However, I challenge you to point to a single entity in the world that has figured out a process for handling non-criminal misconduct that you would be happy with no matter whether you were the aggrieved or the accused party. Maybe such a thing exists, but if so I have not heard of it.

This post is a survey of the different ways that people try to resolve community conflicts, and the ways that each of them fail.

Committees/panels

In cases of major conflict or disagreement, it often seems like the right thing to do to convene a panel of impartial judges and have them hear all the evidence. I personally know of at least seven specific cases of this happening in the rationalist community. Here are some of the problems with this approach.

Investigations eat up hundreds of person-hours

The case I'm most familiar with has been investigated four different times, by different people and from different angles. Five separate reports have been writ...
[Retracted] Newton's law of cooling from first principles
by Nisan — 2024-01-16T04:21:11.094Z

This post is wrong. See this comment.

A high-school physics textbook will tell you that, according to Newton's law of cooling, a hot thing will cool down according to

dQdt∝T∞−T

where Q is heat, t is time, T is temperature, and T∞ is the temperature of the environment. But this is slightly wrong! It turns out that the correct cooling rate, which I'll call Lambert's law, is

dQdt∝1T−1T∞.

(Newton's law approximates this pretty well.) Let's derive this from first principles, and try to test it empirically.

Epistemic status: Logical deductions from a speculative physical model. I don't actually know nonequilibrium thermodynamics.

Rates of change

First, a lemma: I claim that if X is any macroscopic variable, then the rate of change of X is proportional to the derivative of the entropy with respect to X:

dXdt∝dSdX

Why? Suppose our system changes in discrete timesteps, and X can only change in steps of size ΔX. And suppose the probability of a transition X↦X+ΔX is proportional to the number of microstates Ω(X+ΔX). Then you can calculate the expected change in X per timestep, and from there prove the lemma.[1]

The Lambert cooling law

Now let our system comprise two subsystems. There's a fixed amount of heat split between the subsystems. Apply our lemma, taking X to be Q1, the amount of heat in subsystem 1. We find that[2]

dQ1dt∝dS1dQ1−dS2dQ2=1T1−1T2

Here the inverse of temperature, or coldness, shows up. The coldness of a system is defined to be the derivative of entropy with respect to energy. (The only energy here is heat.)

Now if subsystem 2 is very big, its temperature doesn't change much. So let's rename Q1 to Q, T1 to T; and T2 to T∞, because that's the temperature subsystem 1 will have in the limit:

dQdt∝1T−1T∞

We can use this to get an expression for dT/dt involving the second derivative of entropy, which is related to the heat capacity:

dTdt∝−d2SdQ2T2(1T−1T∞)

However, textbooks tend to assume that the heat capacity is temperature-independent, in which case heat is roughly proportional to temperature. So we have

dTdt∝1T−1T∞

This differential equation has a neat solution after a change of variables. Let

τ=(TT∞−1)eT/T∞−1

Then the solution is exponential decay:

τ=τ0e−rt

where τ0 is the value of τ ...
Sparse Autoencoders Work on Attention Layer Outputs
by Connor Kissane — 2024-01-16T00:26:14.767Z

This post is the result of a 2 week research sprint project during the training phase of Neel Nanda’s MATS stream.

Executive Summary

We replicate Anthropic's MLP Sparse Autoencoder (SAE) paper on attention outputs and it works well: the SAEs learn sparse, interpretable features, which gives us insight into what attention layers learn. We study the second attention layer of a two layer language model (with MLPs).

Specifically, rather than training our SAE on attn_output, we train our SAE on “hook_z” concatenated over all attention heads (aka the mixed values aka the attention outputs before a linear map - see notation here). This is valuable as we can see how much of each feature’s weights come from each head, which we believe is a promising direction to investigate attention head superposition, although we only briefly explore that in this work.

We open source our SAE, you can use it via this

Colab notebook

.

Shallow Dives: We do a shallow investigation to interpret each of the first 50 features. We estimate 82% of non-dead features in our SAE are interpretable (24% of the SAE features are dead).

See this

feature interface

to browse the first 50 features.

Deep dives: To verify our SAEs have learned something real, we zoom in on individual features for much more detailed investigations: the “‘board’ is next by induction” feature, the local context feature of “in questions starting with ‘Which’”, and the more global context feature of “in texts about pets”.

We go beyond the techniques from the Anthropic paper, and investigate the circuits used to compute the features from earlier components, including analysing composition with an MLP0 SAE.
We also investigate how the features are used downstream, and whether it's via MLP1 or the direct connection to the logits.

Automation: We automatically detect and quantify a large “{token} is next by induction” feature family. This represents ~5% of the living features in the SAE.

Though the specific automation technique won't generalize to other feature families, this is notable, as if there are many “one feature per vocab token” families like this, we may need impractically wide SAEs for larger models.

Introduction

In Anthropic’s SAE paper, they find that training sparse autoencoders (SAEs) on a one layer model’s MLP activations finds interpretable features, providing a path to breakdown the...
Goals selected from learned knowledge: an alternative to RL alignment
by Seth Herd — 2024-01-15T21:52:06.170Z

Summary:

Alignment work on network-based AGI focuses on reinforcement learning. There is an alternative approach that avoids some, but not all, of the difficulties of RL alignment. Instead of trying to build an adequate representation of the behavior and goals we want, by specifying rewards, we can choose its goals from the representations it has learned through any learning method.

I give three examples of this approach: Steve Byrnes’ plan for mediocre alignment (of RL agents); John Wentworth’s “retarget the search” for goal-directed mesa-optimizers that could emerge in predictive networks; and natural language alignment for language model agents. These three approaches fall into a natural category that has important advantages over more commonly considered RL alignment approaches.

An alternative to RL alignment

Recent work on alignment theory has focused on reinforcement learning (RL) alignment. RLHF and Shard Theory are two examples, but most work addressing network-based AGI assumes we will try to create human-aligned goals and behavior by specifying rewards. For instance, Yudkowsky’s

List of Lethalities

seems to address RL approaches and exemplifies the most common critiques: specifying behavioral correlates of desired values seems imprecise and prone to mesa-optimization and misgeneralization in new contexts. I think RL alignment might work, but I agree with the

critique

that much

optimism

for RL alignment doesn’t adequately consider those concerns.

There’s an alternative to RL alignment for network-based AGI. Instead of trying to provide reinforcement signals that will create representations of aligned values, we can let it learn all kinds of representations, using any learning method, and then select from those representations what we want the goals to be.

I’ll call this approach goals selected from learned knowledge (GSLK). It is a novel alternative not only to RL alignment but also to older strategies focused on specifying an aligned maximization goal before training an agent. Thus, it violates some of the assumptions that lead MIRI leadership and similar thinkers to predict near-certain doom.

Goal selection from learned knowledge (GSLK) involves allowing a system to learn until it forms robust representations, then selecting some of these representations to serve as goals. This is a paradigm shift from RL a...