trentmkelly/LessWrong-43k huggingface dataset:
Causal scrubbing: results on a paren balance checker
by LawrenceC — 2022-12-03T00:59:08.078Z

* Authors sorted alphabetically.

This is a more detailed look at our work applying causal scrubbing to an algorithmic model. The results are also summarized here.

Introduction

In earlier work (unpublished), we dissected a tiny transformer that classifies whether a string of parentheses is balanced or unbalanced.[1] We hypothesized the functions of various parts of the model and how they combine to solve the classification task. The result of this work was a qualitative explanation of how this model works, but one that made falsifiable predictions and thus qualified as an informal hypothesis. We summarize this explanation

below

.

We found that the high-level claims in this informal hypothesis held up well (88-93% loss recovered, see the

Methodology

). Some more detailed claims about how the model represents information did not hold up as well (72%), indicating there are still important pieces of the model’s behavior we have not explained. See the experiments

summary section

for an explanation of each hypothesis refinement.

Causal scrubbing provides a language for expressing explanations in a formal way. A formal hypothesis is an account of the information present at every part of the model, how this information is combined to produce the output, and (optionally) how this information is represented. In this work, we start by testing a simple explanation, then iterate on our hypothesis either by improving its accuracy (which features of the input are used in the model) or specificity (what parts of the model compute which features, and optionally how). This iterative process is guided by the informal hypothesis that we established in prior work.

For a given formal hypothesis, the causal scrubbing algorithm automatically determines the set of interventions to the model that would not disturb the computation specified by the hypothesis. We then apply a random selection of these interventions and compare the performance to that of the original model.

Using causal scrubbing to evaluate our hypotheses enabled us to identify gaps in our understanding, and provided trustworthy evidence about whether we filled those gaps. The concrete feedback from a quantitative correctness measure allowed us to focus on quickly developing alternative hypotheses, and finely distinguish between explanations with subtle differences.

We hope this walk-through will be ...
Causal scrubbing: Appendix
by LawrenceC — 2022-12-03T00:58:45.850Z

* Authors sorted alphabetically.

An appendix to this post.

1 More on Hypotheses

1.1 Example behaviors

As mentioned above, our method allows us to explain quantitatively measured model behavior operationalized as the expectation of a function f on a distribution D.

Note that no part of our method distinguishes between the part of the input or computational graph that belongs to the “model” vs the “metric.”[1]

It turns out that you can phrase a lot of mechanistic interpretability in this way. For example, here are some results obtained from attempting to explain how a model has low loss:

Nanda and Lieberum’s

analysis of the structure of a model that does modular addition

explains the observation that their model gets low loss on the validation dataset.
The

indirect object identification circuit

explains the observation that the model gets low loss on the indirect object identification task, as measured on a synthetic distribution.
Induction circuits (as described in

Elhage et al. 2021

) explain the observation that the model gets low loss when predicting tokens that follow the heuristic: “if AB has occurred before, then A is likely to be followed by B”.

That being said, you can set up experiments using other metrics besides loss as well:

Cammarata et al identify

curve detectors in the Inception vision model

by using the response of various filters on synthetic datasets to explain the correlation between: 1) the activation strength of some neuron, and 2) whether the orientation of an input curve is close to a reference angle.

1.2 Extensional equality, and common rewrites of G and I

If you’re trying to explain the expectation of f, we always consider it a valid move to suggest an alternative function f′ if f(x)=f′(x) on every input (

“extensional equality”

), and then explain f′ instead. In particular, we’ll often start with our model’s computational graph and a simple interpretation, and then perform “algebraic rewrites” on both graphs to naturally specify the correspondence.

Common rewrites include:

When the output of a single component of the model is used in different ways by different paths, we’ll duplicate that node in G, such that each copy can correspond to a different part of I.
When multiple components of the model compute a single feature we can either:
duplicate the node in I,

...
Causal Scrubbing: a method for rigorously testing interpretability hypotheses [Redwood Research]
by LawrenceC — 2022-12-03T00:58:36.973Z

* Authors sorted alphabetically.

Summary: This post introduces causal scrubbing, a principled approach for evaluating the quality of mechanistic interpretations. The key idea behind causal scrubbing is to test interpretability hypotheses via behavior-preserving resampling ablations. We apply this method to develop a refined understanding of how a small language model implements induction and how an algorithmic model correctly classifies if a sequence of parentheses is balanced.

1 Introduction

A question that all mechanistic interpretability work must answer is, “how well does this interpretation explain the phenomenon being studied?”. In the

many

recent

papers

in mechanistic interpretability

, researchers have generally relied on ad-hoc methods to evaluate the quality of interpretations.[1]

This ad hoc nature of existing evaluation methods poses a serious challenge for scaling up mechanistic interpretability. Currently, to evaluate the quality of a particular research result, we need to deeply understand both the interpretation and the phenomenon being explained, and then apply researcher judgment. Ideally, we’d like to find the interpretability equivalent of

property-based testing

—automatically checking the correctness of interpretations, instead of relying on grit and researcher judgment. More systematic procedures would also help us scale-up interpretability efforts to larger models, behaviors with subtler effects, and to larger teams of researchers. To help with these efforts, we want a procedure that is both powerful enough to finely distinguish better interpretations from worse ones, and general enough to be applied to complex interpretations.

In this work, we propose causal scrubbing, a systematic ablation method for testing precisely stated hypotheses about how a particular neural network[2] implements a behavior on a dataset. Specifically, given an informal hypothesis about which parts of a model implement the intermediate calculations required for a behavior, we convert this to a formal correspondence between a computational graph for the model and a human-interpretable computational graph. Then, causal scrubbing starts from the output and recursively finds all of the invariances of parts of the neural network that are implied by the hypothesis, and then replaces the activations of the neural network with the maximum entropy[3] distributio...
Take 2: Building tools to help build FAI is a legitimate strategy, but it's dual-use.
by Charlie Steiner — 2022-12-03T00:54:03.059Z

As a writing exercise, I'm writing an AI Alignment Hot Take Advent Calendar - one new hot take, written every day for 25 days. Or until I run out of hot takes, which seems likely.

This was waiting around in the middle of my hot-takes.txt file, but it's gotten bumped up because of Rob and Eliezer - I've gotta blurt it out now or I'll probably be even more out of date.

The idea of using AI research to help us be better at building AI is not a new or rare idea. It dates back to prehistory, but some more recent proponents include OpenAI members (e.g. Jan Leike) and the Accelerating Alignment group. We've got a tag for it. Heck, this even got mentioned yesterday!

So a lot of this hot take is really about my own psychology. For a long time, I felt that sure, building tools to help you build friendly AI was possible in principle, but it wouldn't really help. Surely it would be faster just to cut out the middleman and understand what we want from AI using our own brains.

If I'd turned on my imagination, rather than reacting to specific impractical proposals that were around at the time, I could have figured out how augmenting alignment research is a genuine possibility a lot sooner, and started considering the strategic implications.

Part of the issue is that plausible research-amplifiers don't really look like the picture I have in your head of AGI - they're not goal-directed agents who want to help us solve alignment. If we could build those and trust them, we really should just cut out the middleman. Instead, they can look like babble generators, souped-up autocomplete, smart literature search, code assistants, and similar. Despite either being simulators or making plans only in a toy model of the world, such AI really does have the potential to transform intellectual work, and I think it makes a lot of sense for there to be some people doing work to make these tools differentially get applied to alignment research.

Which brings us to the dual-use problem.

It turns out that other people would also like to use souped-up autocomplete, smart literature search, code assistants, and similar. They have the potential to transform intellectual work! Pushing forward the state of the art on these tools lets you get them earlier, yet it also helps other people get them earlier too, even if you don't share your weights.

Now, maybe the most popular tools will help people make philoso...
D&D.Sci December 2022: The Boojumologist
by abstractapplic — 2022-12-02T23:39:49.398Z

This is an entry in the 'Dungeons & Data Science' series, a set of puzzles where players are given a dataset to analyze and an objective to pursue using information from that dataset.

STORY (Skippable)

You were saddled with debt, and despair, and regret;

But you left it behind to embark,

With a visiting ship who were planning a trip,

Hunting some strange sea-beasts they call . . . “Snark”?

(After climbing aboard and departing the shore,

Your life is if anything worse.

The grog makes you groggy; the sea makes you soggy;

The songs leave you thinking in verse.)

Snark-hunting, you find, is a peaceful pastime.

By now, every crew knows the way,

To - with ease! - guarantee their success and safety,

As they seek, and they lure, and they slay.

A single exception proves the above rule:

While with most Snarks, Snark-hunting’s a breeze,

If your Snark is a Boojum, it hunts you right back,

And slays you with similar ease.

When you learn the detail that a Snark-hunt can fail,

You speak with the crew and insist,

On wielding Science skills to discern which Snarks kill,

And becoming the ship’s Analyst.

(The Butcher and Baker and Barnacle-Scraper,

All tell you that simply won’t do.

“You’ll be telling us what is a Boojum, what’s not;

Thus, you’re our Boojumologist; true?”

When you meekly protest you’ve not earned that address,

They rebut: “As you clearly percieve,

The Bellman can’t ring, the Belter can’t sing,

And the Beaver knows not how to Beave.”)

Notwithstanding contention viz naming conventions,

Your task is as clear as could be:

Using data from trips made by similar ships,

Choose which Snarks to fight, which to flee.

DATA & OBJECTIVES

You have a dataset of Snark sightings. This tells you whether a sighted Snark was hunted and whether – if hunted – it turned out to be a Boojum. (To clarify, the order of operations is “someone sights a Snark” “they take notes on its behavior and relay this information to the nearest Snark-hunting ship” “the ship’s crew decide whether to hunt it”).
You have a list of Snarks your ship can choose to go after. The ship must hunt six of them for the venture to be profitable; the crew won’t accept a plan targeting fewer than six. If your ship hunts a Boojum, everyone aboard vanishes, including you.
Your objective can be to choose the six Snarks which maximize probability of survival, maximize the EV of the number of Snark bodie

...
Subsets and quotients in interpretability
by Erik Jenner — 2022-12-02T23:13:34.204Z

Summary

Interpretability techniques often need to throw away some information about a neural network's computations: the entirety of the computational graph might just be too big to understand, which is part of why we need interpretability in the first place. In this post, I want to talk about two different ways of simplifying a network's computational graph:

Fully explaining parts of the computations the network performs (e.g. identifying a subcircuit that fully explains a specific behavior we observed)
Approximately describing how the entire network works (e.g. finding meaningful modules in the network, whose internals we still don't understand, but that interact in simple ways)

These correspond to the idea of subsets and quotients in math, as well as many other instances of this duality in other areas. I think lots of interpretability at the moment is 1., and I'd be excited to see more of 2. as well, especially because I think there are synergies between the two.

The entire post is rather hand-wavy; I'm hoping to point at an intuition rather than formalize anything (that's planned for future posts). Note that a distinction like the one I'm making isn't new (e.g. it's intuitively clear that circuits-style research is quite different from neural network clusterability). But I haven't seen it described this explicitly before, and I think it's a useful framing to keep in mind, especially when thinking about how different interpretability techniques might combine to yield an overall understanding.

ETA: An important clarification is that for both 1. and 2., I'm only discussing interpretability techniques that try to understand the internal structure of a network. In particular, 2. talks about approximate descriptions of the algorithm the network is actually using, not just approximate descriptions of the function that's being implemented. This excludes large parts of interpretability outside AI existential safety (e.g. any method that treats the network as a black box and just fits a simpler function to the network).

Subsets vs quotients

In math, if you have a set X, there are two "dual" ways to turn this into a smaller set:

You can take a subset Y⊂X.
You can take a quotient of X/∼ by some equivalence relation ∼.

(The quotient X/∼ is the set of all equivalence classes under ∼.)

I think this is also a good framing to distinguish inter...
Research Principles for 6 Months of AI Alignment Studies
by Shoshannah Tekofsky — 2022-12-02T22:55:17.165Z

This summer I learned about the concept of Audience Capture from the case of Nicholas Perry. Through pure force of social validation, he experienced a shift from an idealistic but obscure young man to a grotesque but popular caricature of a medical train wreck.

The change happened through social reward signals. Originally Nicholas the principled vegan made videos of himself playing the violin, much to no one's interest. The earnest young man then learned he had to give up his vegan diet for health reasons, and thought to give the occasion a positive twist by inviting his viewers to share the first meal of his new lifestyle.

It was an innocuous step. He gained viewers. They cheered him on to eat more. And he did.

Gradually, but steadily he ate and ate, to the cheers of a swelling crowd of online followers. And like the Ghandi Murder Pill, the choice of sacrificing a sliver of his values for substantial reward was worth it for each individual video he made. His popularity expanded with his waistline as he inched up the social incentive slope. And at the end of that slope Nicholas didn't care about health, veganism, or playing the violin anymore. Instead his brain was inured with social reward signals that had rewired his values on a fundamental level. Essentially, Nicholas had a become a different person.

Now I realize I am unlikely to gain 300 pounds from success on AI alignment articles on LessWrong, but audience capture does point to a worry that has been on my mind. How do new researchers in the field keep themselves from following social incentive gradients? Especially considering how hard it is to notice such gradients in the first place!

Luckily the author of the above article suggests a method to ward against audience capture: Define your ideal self up front and commit to aligning your future behavior with her. So this is what I am doing here -- I want to precommit to three Research Principles for the next 6 months of my AI alignment studies:

Transparency - I commit to exposing my work in progress, unflattering confusions of thinking, and potentially controversial epistemics.
Exploration - I commit to exploring new paths for researching the alignment problem and documenting my progress along the way.
Paradigmicity - I commit to working toward a coherent paradigm of AI alignment in which I can situate my work, explain how it contributes

...
Three Fables of Magical Girls and Longtermism
by Ulisse Mini — 2022-12-02T22:01:30.225Z

Epistemic status: Whimsical

Major spoilers for Madoka Magica, a show where spoilers matter!

Kyubey Shuts Up And Multiplies

Meet Kyubey. Kyubey is a Longtermist.

This is what peak utilitarianism looks like.

In the Madokaverse, changes in human emotion are, somehow, net-positive in the creation of energy from out of nothing. The Incubators (of which Kyubey is one, pictured above) are an alien species who've discovered a way to farm human emotions for energy.

Most of the Incubators don't feel emotion, and the few that do are considered to be mentally ill. But humans are constantly leaking our juicy, negentropy-positive feelings all over the place. With human angst as a power source, it's possible to prevent the heat death of the universe!

Negentropy being created. Isn't it a wonderful sight to behold?

Do the math, people. The suffering of a few teenage girls is nothing compared to pushing back the heat death of the universe.[1]

And this isn't just some Omelas situation where the girls get nothing out of it. They get wishes! Who could object to a cause this noble?

Two rational actors with differing circles of concern.

Homura has Something to Protect

If you want to see Homura kicking ass, you could watch up to 2:22 before reading on.

There's something subtle here—something to notice confusion about, even—where is she getting all these guns from?

Remember: Homura's power is time manipulation. As one commenter puts it:

This is hauntingly sobering when you consider that Homura's magical ability has nothing to do with guns, only with time manipulation. That means all those tens of thousands, hundreds of thousands of pounds of explosive material and weapons arms weren't just made from nothing like Mami's guns were- they were individually tracked down and gathered, one after the other, by one little girl.

How many hundreds of repetitions did it take to find them all, every time making a new doomed timeline? How many thousands of hours did she spend looking for where to get them from, and how many failed attempts finding the most effective way to arrange them?

Rationalists have a name for this kind of determination: Having something to protect.

Magical Girls are mesa-optimizers

When Kyubey creates a magical girl, he offers them an atomic contract: they gain a sparkly transformation and fight witches for the rest of their life, and in exchange, they're granted...
Brun's theorem and sieve theory
by Ege Erdil — 2022-12-02T20:57:39.956Z

Brun's theorem is a relatively famous result in analytic number theory that says the sum of the reciprocals of the twin primes converges to a finite value. In other words, we have

∑p,p+2prime1p+1p+2=B

for some finite constant B. This is in contrast to the same sum taken over all primes, which is divergent:

∑pprime1p=∞

In this post, I'll use Brun's theorem as an illustration of sieve theoretic arguments in analytic number theory. I'll try to explain relevant results as I go along to minimize the background necessary to understand the arguments, but some background in real analysis and number theory is needed to understand the post. If you don't have such a background, most of the post will probably be gibberish.

I'm writing this post mostly because I think there's some lack of good explanations of sieve theory in general and the Brun sieve in particular. Hopefully this post will be helpful to a handful of people who are interested in or trying to understand such matters.

Note that in the post I'll not always mention that a sum or a product runs over the prime numbers explicitly. If the sum or product is indexed by the letter p, you should assume that it runs over the primes and not e.g. over the natural numbers. Sometimes p will run only over odd primes, because there is a degenerate case with the prime 2 when we work with twin primes coming from the fact that 0 and 2 are in the same residue class mod 2. This will often be obvious from the surrounding context.

Background

First, let's discuss some background results that will be useful to know throughout the post.

The prime number theorem says that the number of primes less than N, denoted π(N), is well approximated by N/logN. Concretely, it says that

limN→∞π(N)N/logN=1

Roughly, the prime number theorem says that the density of the prime numbers "around N" is roughly 1/logN when N is large. This is a rather difficult theorem to prove and we won't actually need it to prove Brun's theorem. However, the result will be useful to know in heuristic arguments for why we might expect the theorem to be true, and to motivate our method of proof.

We will also need the Mertens theorems. These theorems are "weaker" versions of the prime number theorem. Specifically, we will need to know the fact that

A(x)=∑p≤x,pprime1p=loglogx+O(1)

This is a stronger, quantitative version of the result that the sum of the reciprocals of the prime numb...