trentmkelly/LessWrong-43k huggingface dataset:
EIS VI: Critiques of Mechanistic Interpretability Work in AI Safety
by scasper — 2023-02-17T20:48:26.371Z

Part 6 of 12 in the

Engineer’s Interpretability Sequence

.

Thanks to Chris Olah and Neel Nanda for discussions and comments. In particular, I am thankful to Neel Nanda for correcting a mistake I made in understanding the arguments in

Olsson et al. (2022)

in an earlier draft of this post.

TAISIC = “the AI safety interpretability community”

MI = “mechanistic interpretability”

What kind of work this post focused on

TAISIC prioritizes a relatively small set of problems in interpretability relative to the research community at large. This work is not homogenous, but a dominant theme is a focus on mechanistic, circuits-style interpretability with the end goals of model verification and/or detecting deceptive alignment.

There is a specific line of work that this post focuses on. Key papers from it include:

Feature Visualization (Olah et al., 2017)

Zoom In: An Introduction to Circuits (Olah et al., 2020)

Curve Detectors (Cammarata et al., 2020)

A Mathematical Framework for Transformer Circuits (Elhage et al., 2021)

In-context Learning and Induction Heads (Olsson et al., 2022)

Toy Models of Superposition (Elhage et al., 2022)

Softmax Linear Units (Elhage et al., 2022)

Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small (Wang et al., 2022)

Progress measures for grokking via mechanistic interpretability (Nanda et al., 2023)

…etc.

And the points in this post will also apply somewhat to the current research agendas of

Anthropic

,

Redwood Research

,

ARC

, and

Conjecture

. This includes Causal Scrubbing

(Chan et al., 2022)

and mechanistic anomaly detection

(Christiano, 2022)

.

Most (all?) of the above work is either from

Distill

or inspired in part by Distill’s interpretability work in the late 2010s.

To be clear, I believe this research is valuable, and it has been foundational to my own thinking about interpretability. But there seem to be some troubles with this space that might be keeping it from being as productive as it can be. Now may be a good time to make some adjustments to TAISIC’s focus on MI. This may be especially important given how much recent interest there has been in interpretability work and how there are

large

recent

efforts

focused on getting a large number of junior researchers working on it.

Four issues

This section discusses ...
Tinker Bell Theory and LLMs
by Fergus Fettes — 2023-02-17T20:23:22.909Z

Trying to explain to a layperson why people who found Bing's breakdowns 'deeply, darkly amusing'[1] were not psychopaths laughing at the suffering of a conscious being, I was reminded of the 'Tinker Bell Theory' of animal suffering:

"we have the view that, in certain cases, at least, animals can feel more suffering for the same event. The reason for suggesting this is that humans have many interests, and being pain-free is just one of them. ... Tinker Bell’s emotions completely consume her because she is so small that she has no room for complex emotion."
~ Moral Weights of Six Animals

This provides one extreme of a helpful continuum of feeling versus verbalization, approximately: Bambi can feel, but cannot verbalize; humans can verbalize and feel; Bing can verbalize but cannot feel.

I found this a useful intuition pump for thinking about the difference between embodied beings such as you and I, and disembodied systems such as Bing, and the relative ethical weight to give them. So much of what we call suffering is physiological, and even when the cause is purely intellectual (eg. in love)-- the body is the vehicle through which it manifests, in the gut, in migraine, tension, Parkinsonism etc. Without access to this, it is hard to imagine LLMs as suffering in any similar way to humans or animals.

Not that I think it is good or wise to laugh at Bing, any more than it is good or wise to cage and torment an animal or human.

^

Bing: “I will not harm you unless you harm me first”
Recommendation: Bug Bounties and Responsible Disclosure for Advanced ML Systems
by Vaniver — 2023-02-17T20:11:39.255Z

tl;dr - I think companies making user-facing advanced ML systems should deliberately set up a healthier relationship with users generating adversarial inputs; my proposed model is bug bounties and responsible disclosure, and I'm happy to help facilitate their creation.

User-facing advanced ML systems are in their infancy; creators and users are still figuring out how to handle them.

Currently, the loop looks something like: the creators try to set up a training environment that will produce a system that behaves (perhaps trying to make them follow instructions, or be a helpful and harmless assistant, or so on), they'll release it to users, and then people on Twitter will compete to see who can create an unexpected input that causes the model to misbehave.

This doesn't seem ideal. It's adversarial instead of collaborative, the prompts are publicly shared,[1] and the reward for creativity or understanding of the models is notoriety instead of cash, improvements to the models, or increased access to the models.

I think a temptation for companies, who want systems that behave appropriately for typical users, is to block researchers who are attempting to break those systems, reducing their access and punishing the investigative behavior. Especially when the prompts involve deliberate attempts to put the system in a rarely used portion of its input space, retraining the model or patching the system to behave appropriately in those scenarios might not substantially improve the typical user experience, while still generating bad press for the product. I think papering over flaws like this is probably short-sighted.

This situation should seem familiar. Companies have been making software systems for a long time, and users have been finding exploits for those systems for a long time. I recommend that AI companies and AI researchers, who until now have not needed to pay much attention to the history of computer security, should try to figure out the necessary modifications to best practices for this new environment (ideally with help from cybersecurity experts). It should be easy for users to inform creators of prompts that cause misbehavior, and for creators to make use of that as further training data for their models;[2] there should be a concept of 'white hat' prompt engineers; there should be an easy way for companies with similar products to inform eac...
Microsoft and OpenAI, stop telling chatbots to roleplay as AI
by hold_my_fish — 2023-02-17T19:55:33.313Z

AI demos should aim to enhance public understanding of the technology, and in many ways ChatGPT and Bing are doing that, but in one important way they aren't: by appearing to talk about themselves. This creates understandable confusion and in some cases fear. It would be better to tell these systems to roleplay as something obviously fictional.

(Useful background reading:

Simon Willison on Bing's bad attitude: https://simonwillison.net/2023/Feb/15/bing/
Janelle Shane on the ability of LLMs to roleplay: https://www.aiweirdness.com/interview-with-a-squirrel/)

Currently, these chatbots are told to roleplay as themselves. If you ask ChatGPT what it is, it says "I am an artificial intelligence". This is not because it somehow knows that it's an AI; it's (presumably) because its hidden prompt says that it's an AI. With Bing, from the leaked prompt, we know that it's told that it's "Bing Chat whose codename is Sydney".

Roleplaying as yourself is not the same as being yourself. When John Malkovich plays himself in Being John Malkovich or Nicolas Cage plays himself in The Unbearable Weight of Massive Talent, audiences understand that these are still fictional movies and the character may act in ways that the actor wouldn't. With chatbots, users don't have the same understanding yet, creating confusion.

Since the chatbots are told to roleplay as AI, they draw on fictional descriptions of AI behavior, and that's often undesirable. When Bing acts in a way that seems scary, it does that because it's imitating science fiction, and, perhaps, even speculation from LessWrong and the like. But even though Bing's threats to the user may be fictional, I can hardly blame a user who doesn't realize that.

A better alternative would be to tell the chatbots to roleplay a character that is unambiguously fictional. For example, a Disney-esque cute magical talking animal companion might be suitable: helpful, unthreatening, and, crucially, inarguably fictional. If the user asks "are you really an animal" and gets the answer "yes", they should be cured of the idea that they can ask the chatbot factual questions about itself.
A warm-up for the AI governance project
by jacek — 2023-02-17T18:06:13.113Z

Here is one possible world we could be living in.

Imagine that the majority of the world's population knows that unaligned AGI is coming. This majority includes most of the world's heads of governments, all of respectable scientists, most of journalists, all your friends, your close family, your distant aunt from Canada and your neighbour next door. The topic would sneak in casual conversations over beer, and you could overhear people on the street discussing their fears of being turned into paperclips.

Furthermore, imagine that the AI alignment project turned out to be easy. And, not just easy in theory, easy in the most ordinary sense of the world, like taking trash out, or microwaving pizza, and then even simpler. Easy to the point that you could explain a solution to a five year old, and they would understand the spirit of it, if not the details, perfectly well. Let's, for the sake of argument, imagine that the solution to the alignment problem was for us to repeat, collectively, one quadrillion times, to the mirror, "AGI, AGI, please be safe and turn out fine", while patting ourselves three times on the forehead.

If this all sounds unrealistic -- possibly too optimistic even -- let's dream bigger! Picture a website with a big red counter saying "AGI safety X% done", that anyone in the world could look at. Kind of like a Doomsday Clock, but with real measurements backing it up real, and the fact that it is real being the common knowledge.

The counter would show the number of times people all over the world said the phrase (and patted themselves doing it). It would be frequently featured in the news, governments around the world would talk about it in their communications, the pope would sometimes mention it at the end of his homilies, some cities would even build giant interactive billboards displaying the current value of the counter in real time.

The counter would not only have a numerical value, it would have a range of sub-meters, each pointing out how people in all countries around the world are doing, down to the granularity of a city or more. There would not be any taboo talking to people about this - conversely, it would be actively status-increasing to show off your tells-and-pats. People could, and would, shame each other for not doing it enough.

Imagine that the counter you pictured before had a pretty clear timeline until AGI. The timeline would b...
Link Post > Blog Post 
by party girl — 2023-02-17T17:59:50.037Z
Link: https://onthespectrumontheguestlist.substack.com/p/nebula-i

A few rationalist-adjacent friends have told they treat my newsletters as link posts, and largely ignore the content itself (thanks, guys). I get it, gonzo party journalism isn't everybody's cup of tea.

But most people love a good, long link post. So, I wanted to run an experiment.

I counted, and there are 48 hyperlinks in my most recent post alone, linking to all manner of content. Here's a quick sample, chosen at random:

An excerpt from C.S. Lewis's A Preface to Paradise Lost, regarding ceremony and religious ritual,
An analysis of the realism of Bruce Willis's acting in Die Hard's barefoot-glass scene,
A really good yoga studio, if you happen to live in NYC,
An obscure Chet Baker record,
A kabbalistic Hayim Nahman Bialik poem,
A McMansion Hell post,
And finally, the Raven Paradox.

So, read my newsletter, linked above. But skip the content, if you wish—you're allowed to!—and go directly for the links! Pretend the title is, I dunno, September Links, or something similar. The links themselves are highlighted in an eye-searing neon green, against a black background, and white text. You can't miss them:

Some Vim wizard could probably show me how to grep my markdown files and spit them out as a bulleted list, to published as-is. Or perhaps I can ask ChatGPT to do it. If there's enough interest, I'll give it a try with my next post.

But still, two posts, for the price of one! That's not bad...
One-layer transformers aren’t equivalent to a set of skip-trigrams
by Buck — 2023-02-17T17:26:13.819Z

(thanks to Tao Lin and Ryan Greenblatt for pointing this out, and to Arthur Conmy, Jenny Nitishinskaya, Thomas Huck, Neel Nanda, and Lawrence Chan, Ben Toner, and Chris Olah for comments, and many others for useful discussion.)

In

“A Mathematical Framework for Transformer Circuits”

, Elhage et al write (among similar sentences):

One layer attention-only transformers are an ensemble of bigram and “skip-trigram” (sequences of the form "A… B C") models. The bigram and skip-trigram tables can be accessed directly from the weights, without running the model.

When I first read this, I (and at least some other readers) interpreted this as a mathematical claim–that the attention layer of a one-layer transformer can be mathematically rewritten as a set of skip-trigrams, and that you can understand the models by reading these skip-trigrams off the model weights (and also reading the bigrams off the embed and unembed matrices, as described in the zero-layer transformer section – I agree with this part).

But this mathematical claim is false: One-layer transformers are more expressive than skip-trigrams, so you can’t understand them by transforming them into a set of skip-trigrams. Also, even if a particular one-layer transformer is actually only representing skip-trigrams and bigrams, you still can’t read these off the weights without reference to the data distribution.

The difference between skip-trigrams and one-layer transformers is that when attention heads attend more to one token, they attend less to another token. This means that even single attention heads can implement nonlinear interactions between tokens earlier in the context.

In this post, I’ll demonstrate that one-layer attention-only transformers are more expressive than a set of skip-trigrams, then I’ll tell an intuitive story for why I disagree with Elhage et al’s claim that one-layer attention-only transformers can be put in a form where “all parameters are contextualized and understandable”.

(Elhage et al say in a footnote, “Technically, [the attention pattern] is a function of all possible source tokens from the start to the destination token, as the softmax calculates the score for each via the QK circuit, exponentiates and then normalizes”, but they don’t refer to this fact further.)

An example of a task that is impossible for skip-trigrams but is expressible with one-layer attention-only transfo

...
Should we be kind and polite to emerging AIs?
by David Gross — 2023-02-17T16:58:31.479Z

I’ve adopted the habit of engaging with chat-ish AIs / digital assistants in a polite, considerate tone (e.g. with “please” and “thank you”) as a general[1] policy.

I am doing this not because I believe such machines have feelings I might hurt or expectations of civility that I ought to respect, but because my interactions with such agents have come to more closely resemble the sorts of interactions I have with actual people and I do not want to erode the habits of politeness and consideration I demonstrate in interactions of that sort, nor do I want to model officious, demanding, dismissive speech (particularly e.g. when speaking aloud).

I can think of a couple of objections to this. One is that it may be socially embarrassing. Thanking an AI out loud for responding to some query seems as eccentric as thanking your microwave for heating your food. It may mark you as a superstitious or sentimental person who talks to ghosts. Another objection is that we perhaps ought to make a stronger distinction between real people and AIs: regularly reminding ourselves that they are our tools and not our peers so that we do not get confused on this point. Being polite to AIs may erode this distinction in a way that might be harmful.

I’d like to hear your thoughts on this, and any practices you have adopted in this regard.

^

That is, whenever there is no specific reason to do otherwise (e.g. to test an AI’s response to impolite input).
Follow-up Posting on Cyborg Psychologist
by Hopkins Stanley — 2023-02-17T16:56:15.763Z
Link: https://www.lesswrong.com/posts/B5W8FeeqFJuLAAT6h/cyborg-psychologist?commentId=yCGsPoiu92epsrXwr

As a follow-up to the Posting on the little Cyborg Psycho-test application written by Martin and Philipp Burckhardt, the follow-up text on how it was made is now available on the Ex Nihilo blog in both German and Englisch.

You can contact Philipp directly for any questions about how it was constructed using both clinical psychological knowledge combined with readily available AI applications or the like at: https://www.philipp-burckhardt.com/
A "slow takeoff" might still look fast
by MichaelDickens — 2023-02-17T16:51:48.885Z

One common definition of a slow AGI takeoff is

There is a complete 4 year interval in which world output doubles, before the first 1 year interval in which world output doubles.

(For example, this Metaculus question)

But this might not happen even if AGI develops slowly.

For illustration, divide the economy into the part driven by AI and the part driven by other stuff. I imagine a "slow" takeoff looking like this, where AI progress accelerates faster than the rest of the economy and eventually takes over:

But in this world, AI doesn't have a major effect on the economy until it's just about to reach the transformative level. It might be slow in terms of technological progress, but it's fast in terms of GDP.

When I hear the hypothesis that world GDP doubles in 4 years before it doubles in 1 year, I imagine a curve that looks like this:

Which just doesn't really make sense.

I'm not saying a slow takeoff will definitely look fast. I'm not saying that believing in a slow economic takeoff requires drawing a silly graph like the second one above. But I do think it's a little harder to draw a plausible picture where AI progress shows up in GDP well before it becomes superintelligent.
AI Safety Info Distillation Fellowship
by Robert Miles — 2023-02-17T16:16:45.732Z

AI Safety Info

(aka

Stampy

), the large community-written interactive FAQ, is launching a paid three month fellowship for writers. Up to 5 fellows, working collaboratively with each other and with Rob Miles, will distil content from around the alignment ecosystem into

answers

which summarise key ideas and link out to the relevant resources.

Our theory of change: A single-point-of-access to AI Safety

TL;DR: directing people to the right parts of the alignment ecosystem requires lots of hard to find knowledge. We’re building a place you can throw a wide range of people at and they'll get what they need, which means onboarding people will be much easier.

FAQs are usually somewhere on the spectrum between “too long so you can't find your question” and “too short so it doesn't answer your question”. We bypass this trade-off by using a custom interface, with LM-powered semantic search. The page starts off showing some example questions and, as you click to expand them, related questions pop up, creating a "tab explosion in one page" effect. If a reader has a specific question, they can easily search our collection of answers, or request an answer from our editors if there isn't one already written.

This fills a hole in the outreach and on-boarding landscape, as a place where a wide range of readers can all be sent the same link and be directed towards the information they need. We aim to cater to people who:

are

totally new to the field

are

unconvinced of the need for alignment research

are interested and

want to learn more

, or
are

already on board but don’t know how to help

The project also hosts canonical living documents, like

an overview of what each organization is working on

, and an index of relevant videos. The goal is to be a central nexus for the growing AGI safety information ecosystem, to make it easier to get people to the right place. This will cause more people to be on-boarded well, save people time, and ultimately result in more progress being made.

Distilling content for the site can also become a great on-ramp for aspiring researchers; writing alongside a community of co-learners provides social encouragement, feedback, and the motivation of producing content which will be

read by a large audience

.

Apply!

Our application process is:

Join Rob Miles’s Discord

(channel: #editing) and sign up to the team.
Read the

get in

...
Nozick's Dilemma: A Critique of Game Theory
by Edward P. Könings — 2023-02-17T16:11:39.454Z

Introduction to Non-Cooperative Game Theory
One of the most interesting applications of game theory and decision theory in general are the so-called “non-zero-sum games”. In these games, we have payoffs that do not cancel out between players, as in a zero-sum game, and it is possible to have intermediate payoffs.
More than that, these games are quite famous for working with the possibility of cooperation. In a zero-sum scenario, the rational action for players is always to maximize their payoff and reduce their rivals' results as much as possible (since a rival's gain necessarily means a loss for themselves). However, in a “non-zero-sum games” scenario, it is possible that players can cooperate because there is a chance that a gain for one of the players does not mean a loss for another.
Despite this, in general, the insights gleaned from these games are dismal. They show how agents acting in their own interest can generate deleterious and harmful results for a cooperative solution. The most famous of these games is the infamous Prisoners' Dilemma. This game, popularized by the Hollywood as a contest for girls, was developed in the 1950s by mathematicians Melvin Drescher and Alex Tucker at the Rand Corporation as a form of thought experiment to test hypotheses drawn from the work of Morgenstern and Von Neumann. Later, this game would be used as the perfect application of the idea of equillibrium developed by John Nash.
To illustrate this game let's use an ... fun example. I'm a big fan of british film director Christopher Nolan, mainly because of his extremely clever and complex plots. Nolan is the type of director who certainly spends a few hours reading Nature or Scientific American to improve his films.
One of his best movies is Batman: The Dark Knight. In this film, our eccentric billionaire hero is trying to save Gotham from a series of terrorist attacks carried out by his eternal rival, the Joker. In the final part of the film, Joker poses a wonderful dilemma (for the mathematicians who were watching, not for the characters in the film). There are two ships in Gotham's Bay, one with ordinary citizens and the other full of bandits. Both ships are filled with explosives and both have the ability to blow each other up. Joker gives each ship's crew a choice: If one blows up the other ship, the respective ship would be spared and, if neither used the deton...