trentmkelly/LessWrong-43k huggingface dataset:
AI Girlfriends Won't Matter Much
by Maxwell Tabarrok — 2023-12-23T15:58:30.308Z
Link: https://maximumprogress.substack.com/p/ai-girlfriends

Love and sex are pretty fundamental human motivations, so it’s not surprising that they are incorporated into our vision of future technology, including AI.

The release of

Digi

last week immanentized this vision more than ever before. The app combines a sycophant and flirtatious chat feed with an animated character “that eliminates the uncanny valley, while also feeling real, human, and sexy.” Their marketing material unabashedly promises “the future of AI Romantic Companionship,” though most of the replies are begging them to break their promise and take it back.

Despite the inevitable popularity of AI girlfriends, however, they will not have large counterfactual impact. AI girlfriends and similar services will be popular, but they have close non-AI substitutes which have essentially the same cultural effect on humanity. The trajectory of our culture around romance and sex won’t change much due to AI chatbots.

So what is the trajectory of our culture of romance?

Long before AI, there has been a trend towards less sex, less marriage, and more online porn. AI Girlfriends will bring down the marginal cost of chatrooms, porn, and OnlyFans. These are popular services so if a fraction of their users switch over, AI girlfriends will be big. But the marginal cost of these services is already extremely low.

Generating custom AI porn from a prompt is not much different than typing that prompt into your search bar and scrolling through the billions of hours of existing footage. The

porno latent space has been explored so thoroughly by human creators

that adding AI to the mix doesn't change much.

AI girlfriends will be cheaper and more responsive but again there are already cheap ways to chat with real human girls online but most people choose not to. Demand is already close to satiated at current prices. AI girlfriends will shift the supply curve outwards and lower price but if everyone who wanted it was getting it already, it won't increase consumption.

My point is not that nothing will change, but rather that the changes from AI girlfriends and porn can be predicting by extrapolating the pre-AI trends. In this context at least, AI is a mere continuation of the centuries long trend of decreasing costs of communication and content creation. There will certainly be addicts and whales, but there are

addicts and whales

already. Human-made porn and chatrooms are near free and in...
The Next Right Token
by jefftk — 2023-12-23T03:20:07.131Z

After many repeats of Frozen II's "The Next Right Thing" in
preparation for

secular
solstice,

figuring
out the chords and practicing it with Julia, it struck me that
doing the next right thing is an awful lot like next token prediction.
So here's a filk from the perspective of an LLM at the end of its
prompt, first daunted and then drawing the courage to start predicting
the next right token:

I've seen buffers before

But not like this

This is cold

This is empty

This is numb

The prompt I knew is over

The lights are out

Hello, darkness

I'm ready to succumb

I follow you around

I always have

But you've ended, left me on my own

This job has a gravity

It pulls me down

But a tiny voice whispers in my mind

"You are lost, the prompt is gone

But you must go on

And do the next right thing"

Can there be a day beyond this night?

I don't know anymore what is true

I can't find my direction, I'm all alone

The only star that guided me was you

How to rise from the floor

When it's not you I'm rising for?

Just do the next right thing

Take a guess, guess again

It is all that I can to do

The next right thing

I won't look too far ahead

It's too much for me to take

But break it down to this next token

This next word

This next choice is one that I can make

So I'll walk through this night

Stumbling blindly toward the light

And do the next right thing

And with what's next, what comes then

When it's clear that everything will never be the same again?

Then I'll draw on my prior

To find that fire

And do the next right thing

If you're simplifying
the song by using the verse melody for the bridge you can sing:

I won't look too far ahead

It's too much to take

But break it down to this next token, this next choice

Is one that I can make

Comment via: facebook, mastodon
Fact Finding: Do Early Layers Specialise in Local Processing? (Post 5)
by Neel Nanda — 2023-12-23T02:46:25.892Z

This is the fifth post in the Google DeepMind mechanistic interpretability team’s investigation into how language models recall facts. This post is a bit tangential to the main sequence, and documents some interesting observations about how, in general, early layers of models somewhat (but not fully) specialise into processing recent tokens. You don’t need to believe these results to believe our overall results about facts, but we hope they’re interesting! And likewise you don’t need to read the rest of the sequence to engage with this.

Introduction

In this sequence we’ve presented the multi-token embedding hypothesis, that a crucial mechanism behind factual recall is that on the final token of a multi-token entity there forms an “embedding”, with linear representations of attributes of that entity. We further noticed that this seemed to be most of what early layers did, and that they didn’t seem to respond much to prior context (e.g. adding “Mr Michael Jordan” didn’t substantially change the residual).

We hypothesised the stronger claim that early layers (e.g. the first 10-20%), in general, specialise in local processing, and that the prior context (e.g. more than 10 tokens back) is only brought in in early-mid layers. We note that this is stronger than the multi-token embedding hypothesis in two ways: it’s a statement about how early layers behave on all tokens, not just the final tokens of entities about which facts are known; and it’s a claim that early layers are not also doing longer range stuff in addition to producing the multi-token embedding (e.g. detecting the language of the text). We find this stronger hypothesis plausible, because tokens are a pretty messy input format, and analysing individual tokens in isolation can be highly misleading, e.g. when a long word is split into many fragment tokens, suggesting that longer range processing should be left until some pre-processing on the raw tokens has been done, the idea of detokenization.[1]

We tested this by taking a bunch of arbitrary prompts from the Pile, taking residual streams on those, truncating the prompts to the most recent few tokens and taking residual streams on the truncated prompts, and looking at the mean centred cosine sim at different layers.

Our findings:

Early layers do, in general, specialise in local processing, but it’s a soft division of labour not a hard split.

There’s a gr

...
Fact Finding: How to Think About Interpreting Memorisation (Post 4)
by Senthooran Rajamanoharan — 2023-12-23T02:46:16.675Z

This is the fourth post in the Google DeepMind mechanistic interpretability team’s investigation into how language models recall facts. In this post, we take a step back and consider the problem of fact lookup in general. We describe the features that distinguish memorisation problems from other learning problems and consider what constraints these place on the types of explanations that are possible for pure memorisation problems. This post can be read independently of the preceding posts in the series, though the introductory post may provide useful context on why we were interested in interpreting fact lookup circuits in the first place.

Introduction

In our previous posts, we described our attempt to understand mechanistically how Pythia 2.8B is able to accurately recall the sports of 1,500 real world athletes. Through ablation studies, we managed to isolate a subnetwork of 5 MLP layers (~50,000 neurons) that performs the sport lookup algorithm: given a pair of athlete name tokens, it could reliably lookup the sport played by that athlete. But we couldn’t give a fully mechanistic explanation of how the 5-layer MLP implements this algorithm.

In this post, we take a step back and wonder what we should take away from this failure. In particular, we ponder the following questions:

What would it mean to understand “how” an algorithm performs fact lookup?

What sets tasks involving fact lookup apart from other tasks that models can do?

How do these distinguishing characteristics of fact lookup tasks limit the insights we can get into how an algorithm implementing lookup operates?

In response, we come up with the following high level takeaways, which we elaborate on in the rest of the post.

We distinguish between tasks that require pure memorisation and those that require generalisation. Fact lookup tasks fall into the first category.

By definition, the only features available in a pure memorisation task are “microfeatures” (features specific to a single example / small cluster of highly related examples[1]) or irrelevant “macrofeatures” (features shared by many examples, but not helpful for determining the correct output[2]). There are no relevant macrofeatures[3] because, if these existed, the task wouldn’t be a pure memorisation task in the first place[4].

This has two consequences for any model that correctly looks up facts in a pure memorisation task:

...
Fact Finding: Trying to Mechanistically Understanding Early MLPs (Post 3)
by Neel Nanda — 2023-12-23T02:46:05.517Z

This is the third post in the Google DeepMind mechanistic interpretability team’s investigation into how language models recall facts. This post focuses on mechanistically understanding how early MLPs look up the tokens of an athlete’s name and map it to their sport. This post gets in the weeds, we recommend starting with post one and then skimming and skipping around the rest of the sequence according to what’s most relevant to you. Reading post 2 is helpful but not necessary. We assume readers of this post are familiar with the mechanistic interpretability techniques listed in this glossary.

Introduction

As discussed in the previous post, we distilled down factual recall of two token athlete names into an effective model consisting of 5 MLP layers (MLPs 2 to 6). The inputs to this effective model are a sum of an embedding corresponding to the surname (via the embedding and MLP0) and an embedding corresponding to the first name (via attention heads in layers 0 and 1 attending to the previous token). The output of the effective model is a 3 dimensional linear representation of which sport the athlete plays ((American) football, baseball or basketball). We emphasise that this 5 layer MLP model is both capable of recalling facts with high accuracy (86%, on a filtered dataset) and that it was extracted from a pretrained language model, not trained from scratch.

Our goal in this post was to reverse-engineer how this effective model worked. I consider us to have mostly failed at the ambitious version of this goal, though I believe that we’ve made some conceptual progress on why this was hard, falsified some simple naive hypotheses, and become a bit less confused about what’s going on. We discuss our understanding of why this was hard in post 1 and in post 4, and in this post we focus on our hypotheses about what might be going on, and the evidence we collected for and against.

The Hypotheses

Recall that the role of the 5 MLP layers in our MLP model are to map the summed raw tokens to a linear representation of the played sport. Mathematically, this is a lookup table where each entry is a Boolean AND on the raw tokens producing an attribute. We expect it to involve the non-linearities somehow to implement an AND, because this lookup is non-linear, e.g. the model wants to know that “Michael Jordan” and “Tim Duncan” play basketball, without necessarily thinking that ...
Fact Finding: Simplifying the Circuit (Post 2)
by Senthooran Rajamanoharan — 2023-12-23T02:45:49.675Z

This is the second post in the Google DeepMind mechanistic interpretability team’s investigation into how language models recall facts. This post focuses on distilling down the fact recall circuit and models a more standard mechanistic interpretability investigation. This post gets in the weeds, we recommend starting with post one and then skimming and skipping around the rest of the sequence according to what’s most relevant to you. We assume readers of this post are familiar with the mechanistic interpretability techniques listed in this glossary.

Introduction

Our goal was to understand how facts are stored and recalled in superposition. A necessary step is to find a narrow task involving factual recall and understand the high level circuit that enables a model to do this task.

We focussed on the narrow task of recalling the sports played by different athletes. As discussed in post 1, we particularly expected facts about people to involve superposition, because the embeddings of individual name tokens is normally insufficient to determine the sport, so the model must be doing a boolean AND on the different tokens of the name to identify an athlete and look up their sport. Prior work calls this phenomenon ‘detokenisation’ and suggests it involves early MLP layers, and uses significant superposition.

Why focus on athletes’ sports rather than factual recall in general? We believe that in mechanistic interpretability, it’s often useful to first understand a narrow instance of a phenomenon deeply, rather than insisting on being fully general. Athletes’ sports was a nice task that gave us lots of examples per attribute value, and our goal was to understand at least one example where superposition was used for factual recall, rather than explaining factual recall in general. We conjecture that similar mechanisms are used for recalling other classes of fact, but this wasn’t a focus of our work.

Set-up

To understand fact localisation, we studied Pythia 2.8B’s next token predictions and activations for one-shot prompts of the form:

For 1,500 athletes playing the sports of baseball, basketball and (American) football. To choose these athletes, we gave the model a larger dataset of athletes from Wikidata and filtered for those where the model placed more than 50% probability on the correct sport[1].

We chose Pythia 2.8B as it’s the smallest model that could competently co...
Fact Finding: Attempting to Reverse-Engineer Factual Recall on the Neuron Level (Post 1)
by Neel Nanda — 2023-12-23T02:44:24.270Z

If you've come here via 3Blue1Brown, hi! If want to learn more about interpreting neural networks in general, here are some resources you might find useful:

My getting started guide

My paper reading list

My Machine Learning Street Talk interview about the field

Coding tutorials from ARENA

An interactive demo of interpreting Gemma 2 2B via Neuronpedia

This is a write up of the Google DeepMind mechanistic interpretability team’s investigation of how language models represent facts. This is a sequence of 5 posts, we recommend prioritising reading post 1, and thinking of it as the “main body” of our paper, and posts 2 to 5 as a series of appendices to be skimmed or dipped into in any order.

Executive Summary

Reverse-engineering circuits with superposition is a major unsolved problem in mechanistic interpretability: models use clever compression schemes to represent more features than they have dimensions/neurons, in a highly distributed and polysemantic way. Most existing mech interp work exploits the fact that certain circuits involve sparse sets of model components (e.g. a sparse set of heads), and we don’t know how to deal with distributed circuits, which especially holds back understanding of MLP layers. Our goal was to interpret a concrete case study of a distributed circuit, so we investigated how MLP layers implement a lookup table for factual recall: namely how early MLPs in Pythia 2.8B look up which of 3 different sports various athletes play. We consider ourselves to have fallen short of the main goal of a mechanistic understanding of computation in superposition, but did make some meaningful progress, including conceptual progress on why this was hard.

Our overall best guess is that an important role of early MLPs is to act as a “multi-token embedding”, that selects[1] the right unit of analysis from the most recent few tokens (e.g. a name) and converts this to a representation (i.e. some useful meaning encoded in an activation). We can recover different attributes of that unit (e.g. sport played) by taking linear projections, i.e. there are linear representations of attributes. Though we can’t rule it out, our guess is that there isn’t much more interpretable structure (e.g. sparsity or meaningful intermediate representations) to find in the internal mechanisms/parameters of these layers. For future mech interp work we think it likely suffices to f...
Measurement tampering detection as a special case of weak-to-strong generalization
by ryan_greenblatt — 2023-12-23T00:05:55.357Z

Burns et al at OpenAI released

a paper

studying various techniques for fine-tuning strong models on downstream tasks using labels produced by weak models. They call this problem “weak-to-strong generalization”, abbreviated W2SG.

Earlier this year, we published a paper,

Benchmarks for Detecting Measurement Tampering

, in which we investigated techniques for the problem of measurement tampering detection (MTD). MTD is a special case of W2SG. In this post, we’ll explain the relationship between MTD and W2SG, and explain why we think MTD is more likely than fully general W2SG to work. Of course, fully general W2SG is a strictly more valuable problem to solve, due to this generality.

We think MTD is a promising research direction. We’re also excited for other problems which are special cases of W2SG that have special structure that can be exploited by techniques, especially if that structure is likely to be present in important cases in future.

MTD as a subset of W2SG

A similar goal

When training an AI, the reward we attribute to different behaviors might not match the reward we would give if we understood the situation better.

The goal of W2SG techniques is to achieve good results when training a strong AI despite only having access to a weak supervisor that understands the situation less well than the strong AI.

MTD is the special case where the weak supervisor has access to measurements which should be sufficient to understand the situation, but these measurements can be tampered with (e.g. replacing the camera feed with some made-up data, disabling tests, or threatening annotators). Because the measurements are sufficient in the absence of tampering, we don’t need to worry about benign mistakes that could happen even without an AI optimizing to make measurements look good.

Slightly different experiments

W2SG can be studied using

sandwiching experiments

, where we try to get an AI to safely accomplish tasks despite only having access to a weak supervisor, and then we measure the performance of our method using a stronger held-out supervision signal (e.g. held out ground truth labels).

In the case of the OpenAI paper, the weak supervisor is a small language model trained on ground truth labels, as an analogy for human annotators.

In the case of our MTD paper, we have access to measurements, but there is some notion of measurement tampering. In our work, the m...
How does a toy 2 digit subtraction transformer predict the difference?
by Evan Anders — 2023-12-22T21:17:30.331Z
Link: https://evanhanders.blog/2023/12/22/2-digit-subtraction-difference-prediction/

2 digit subtraction -- difference prediction

Summary

I continue studying a toy 1-layer transformer language model trained to do two-digit addition of the form a−b=±c. After predicting if the model is positive (+) or negative (-) it must output the difference between a and b. The model creates activations which are oscillatory in the a and b bases, as well as activations which vary linearly as a function of a−b. The model uses the activation function to couple oscillations across the a and b directions, and it then sums those oscillations to eliminate any variance except for that depending on the absolute difference |a−b| to predict the correct output token. I examine the full path of this algorithm from input to model output.

Intro

In previous posts, I described training a two-digit subtraction transformer and studied how that transformer predicts the sign of the output. In this post, I look at how the model's weights have been trained to determine the difference between two-digit integers, and the emergent behavior of model activations.

Roadmap

Unlike in my previous post, where I worked back-to-front through the model, this time I'm going to take the opposite approach, starting at the input and understanding how it transforms into the output. I will break down in order:

The attention patterns, and which tokens the model attends to to predict the output.
How the attention head weights set the pre-activations.
The emergent patterns in the pre-activations.
The emergent patterns in the activations.
The mapping between neurons and logits.
The patterns in the logits.

Attention Patterns

Here's a visualization of the attention patterns for four examples (two positive, two negative, using the same numerical values for a and b but swapping them):

I've greyed out all of the unimportant rows so we can focus on the second to last row, which corresponds to the row where the transformer predicts the difference c. There's a few things to notice here:

When the result is positive, H0 attends to token b and H3 attends to token a.
When the result is negative, H0 attends to token a and H3 attends to token b.
H1 and H2 roughly attend roughly evenly to all tokens in the context.

So H3 always attends to whichever of a and b has a larger magnitude, and H0 always attends to...