trentmkelly/LessWrong-43k huggingface dataset:
You Can’t Predict a Game of Pinball
by Jeffrey Heninger — 2023-03-30T00:40:05.280Z

Introduction

When thinking about a new idea, it helps to have a particular example to use to gain intuition and to clarify your thoughts. Games are particularly helpful for this, because they have well defined rules and goals. Many of the most impressive abilities of current AI systems can be found in games.[1]

To demonstrate how chaos theory imposes some limits on the skill of an arbitrary intelligence, I will also look at a game: pinball.

In this page, I will show that the uncertainty in the location of the pinball grows by a factor of about 5 every time the ball collides with one of the disks. After 12 bounces, an initial uncertainty in position the size of an atom grows to be as large as the disks themselves. Since you cannot launch a pinball with more than atom-scale precision, or even measure its position that precisely, you cannot make the ball bounce between the disks for more than 12 bounces.

The challenge is not that we have not figured out the rules that determine the ball’s motion. The rules are simple; the ball’s trajectory is determined by simple geometry. The challenge is that the chaotic motion of the ball amplifies microscopic uncertainties. This is not a problem that is solvable by applying more cognitive effort.

The Game

Let’s consider a physicist’s game of pinball.

Forget about most of the board and focus on the three disks at the top. Each disk is a perfect circle of radius R. The disks are arranged in an equilateral triangle. The minimum distance between any two disks is L. See Figure 1 for a picture of this setup.

Figure 1: An idealization of the three disks near the top of a pinball table. Drawn by Jeffrey Heninger.

The board is frictionless and flat, not sloped like in a real pinball machine. Collisions between the pinball and the disks are perfectly elastic, with no pop bumpers that come out of the disk and hit the ball. The pinball moves at a constant speed all of the time and only changes direction when it collides with a disk.

The goal of the game is to get the pinball to bounce between the disks for as long as possible. As long as it is between the disks, it will not be able to get past your flippers and leave the board.

A real game of pinball is more complicated than this – and correspondingly, harder to predict. If we can establish that the physicist’s game of pinball is impossible to predict, then a real game of pinball will be impossib...
Pausing AI Developments Isn't Enough. We Need to Shut it All Down by Eliezer Yudkowsky
by jacquesthibs — 2023-03-29T23:16:19.431Z
Link: https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/

New article in Time Ideas by Eliezer Yudkowsky.

Here’s some selected quotes.

In reference to the letter that just came out (discussion here):

We are not going to bridge that gap in six months.

It took more than 60 years between when the notion of Artificial Intelligence was first proposed and studied, and for us to reach today’s capabilities. Solving safety of superhuman intelligence—not perfect safety, safety in the sense of “not killing literally everyone”—could very reasonably take at least half that long. And the thing about trying this with superhuman intelligence is that if you get that wrong on the first try, you do not get to learn from your mistakes, because you are dead. Humanity does not learn from the mistake and dust itself off and try again, as in other challenges we’ve overcome in our history, because we are all gone.

…

Some of my friends have recently reported to me that when people outside the AI industry hear about extinction risk from Artificial General Intelligence for the first time, their reaction is “maybe we should not build AGI, then.”

Hearing this gave me a tiny flash of hope, because it’s a simpler, more sensible, and frankly saner reaction than I’ve been hearing over the last 20 years of trying to get anyone in the industry to take things seriously. Anyone talking that sanely deserves to hear how bad the situation actually is, and not be told that a six-month moratorium is going to fix it.

Here’s what would actually need to be done:

The moratorium on new large training runs needs to be indefinite and worldwide. There can be no exceptions, including for governments or militaries. If the policy starts with the U.S., then China needs to see that the U.S. is not seeking an advantage but rather trying to prevent a horrifically dangerous technology which can have no true owner and which will kill everyone in the U.S. and in China and on Earth. If I had infinite freedom to write laws, I might carve out a single exception for AIs being trained solely to solve problems in biology and biotechnology, not trained on text from the internet, and not to the level where they start talking or planning; but if that was remotely complicating the issue I would immediately jettison that proposal and say to just shut it all down.

Shut down all the large GPU clusters (the large computer farms where the most powerful AIs are refined). Shut down all the large

...
Othello-GPT: Reflections on the Research Process
by Neel Nanda — 2023-03-29T22:13:42.007Z
Link: https://neelnanda.io/mechanistic-interpretability/othello

This is the third in a three post sequence about interpreting Othello-GPT. See the first post for context.

This post is a detailed account of what my research process was, decisions made at each point, what intermediate results looked like, etc. It's deliberately moderately unpolished, in the hopes that it makes this more useful!

The Research Process

This project was a personal experiment in speed-running doing research, and I got the core results in in ~2.5 days/20 hours. This post has some meta level takeaways from this on doing mech interp research fast and well, followed by a (somewhat stylised) narrative of what I actually did in this project and why - you can see the file tl_initial_exploration.py in the paper repo for the code that I wrote as I went (using VSCode's interactive Jupyter mode).

I wish more work illustrated the actual research process rather than just a final product, so I'm trying to do that here. This is approximately just me converting my research notes to prose, see the section on process-level takeaways for a more condensed summary of my high-level takeaways.

The meta level process behind everything below is to repeatedly be confused, plot stuff a bunch, be slightly less confused, and iterate. As a result, there's a lot of pictures!

Takeaways on doing mech interp research

Warning: I have no idea if following my advice about doing research fast is actually a good idea, especially if you're starting out in the field! It's much easier to be fast and laissez faire when you have experience and an intuition for what's crucial and what's not, and it's easy to shoot yourself in the foot. And when you skimp on rigour, you want to make sure you go back and check! Though in this case, I got strong enough results with the probe that I was fairly confident I hadn't entirely built a tower of lies. And generally, beware of generalising from one example - in hindsight I think I got pretty lucky on how fruitful this project was!

Be decisive: Subjectively, by far the most important change was suppressing my perfectionism and trying to be bold and decisive - make wild guesses and act on them, be willing to be less rigorous, etc.

If I noticed myself stuck on doing the best or most principled thing, I'd instead try to just do something.

Eg I wanted to begin by patching between two similar sequences of moves - I couldn't think of a principled way to chang

...
Othello-GPT: Future Work I Am Excited About
by Neel Nanda — 2023-03-29T22:13:26.823Z
Link: https://neelnanda.io/mechanistic-interpretability/othello

This is the second in a three post sequence about interpreting Othello-GPT. See the first post for context.

This post covers future directions I'm excited to see work on, why I care about them, and advice to get started. Each section is self-contained, feel free to skip around.

Look up unfamiliar terms here

Future work I am excited about

The above sections leaves me (and hopefully you!) pretty convinced that I've found something real and dissolved the mystery of whether there's a linear vs non-linear representation. But I think there's a lot of exciting mysteries left to uncover in Othello-GPT, and that doing so may be a promising way to get better at reverse-engineering LLMs (the goal I actually care about). In the following sections, I try to:

Justify why I think further work on Othello-GPT is interesting

(Note that my research goal here is to get better at transformer mech interp, not to specifically understand emergent world models better)

Discuss how this unlocks finding modular circuits, and some preliminary results

Rather than purely studying circuits mapping input tokens to output logits (like basically all prior transformer circuits work), using the probe we can study circuits mapping the input tokens to the world model, and the world model to the output logits - the difference between thinking of a program as a massive block of code vs being split into functions and modules.

If we want to reverse-engineer large models, I think we need to get good at this!

Discuss how we can interpret Othello-GPT's neurons - we're very bad at interpreting transformer MLP neurons, and I think that Othello-GPT's are simple enough to be tractable yet complex enough to teach us something!

Discuss how, more broadly, Othello-GPT can act as a laboratory to get data on many other questions in transformer circuits - it's simple enough to have a ground truth, yet complex enough to be interesting

My hope is that some people reading this are interested enough to actually try working on these problems, and I end this section with advice on where to start.

Why and when to work on toy models

This is a long and rambly section about my research philosophy of mech interp, and you should feel free to move on to the next section if that's not your jam

At first glance, playing legal moves in Othello (not even playing good moves!) has nothing to do with language models, and I think ...
Actually, Othello-GPT Has A Linear Emergent World Representation
by Neel Nanda — 2023-03-29T22:13:14.878Z
Link: https://neelnanda.io/mechanistic-interpretability/othello

Note that this work has since been turned into a paper and published at BlackboxNLP. I think the paper version is more rigorous but much terser and less fun, and both it and this sequence of blog posts are worth reading in different ways

Epistemic Status: This is a write-up of an experiment in speedrunning research, and the core results represent ~20 hours/2.5 days of work (though the write-up took way longer). I'm confident in the main results to the level of "hot damn, check out this graph", but likely have errors in some of the finer details.

Disclaimer: This is a write-up of a personal project, and does not represent the opinions or work of my employer

This post may get heavy on jargon. I recommend looking up unfamiliar terms in my mechanistic interpretability explainer

Thanks to Chris Olah, Martin Wattenberg, David Bau and Kenneth Li for valuable comments and advice on this work, and especially to Kenneth for open sourcing the model weights, dataset and codebase, without which this project wouldn't have been possible! Thanks to ChatGPT for formatting help.

Overview

Context: A recent paper trained a model to play legal moves in Othello by predicting the next move, and found that it had spontaneously learned to compute the full board state - an emergent world representation.

This could be recovered by non-linear probes but not linear probes.

We can causally intervene on this representation to predictably change model outputs, so it's telling us something real

I find that actually, there's a linear representation of the board state!

But that rather than "this cell is black", it represents "this cell has my colour", since the model plays both black and white moves.

We can causally intervene with the linear probe, and the model makes legal moves in the new board!

This is evidence for the linear representation hypothesis: that models, in general, compute features and represent them linearly, as directions in space! (If they don't, mechanistic interpretability would be way harder)

The original paper seemed at first like significant evidence for a non-linear representation - the finding of a linear representation hiding underneath shows the real predictive power of this hypothesis!

This (slightly) strengthens the paper's evidence that "predict the next token" transformer models are capable of learning a model of the world.

Part 2: There's a lot of fascina

...
Draft: Detecting optimization
by Alex_Altair — 2023-03-29T20:17:46.642Z

This post is a draft, a work in progress. Much of it will not make sense, and will not be optimized for the reader's experience. It does not necessarily reflect my current best understanding of optimization. Like the introduction draft, I'm opening it up because I've been working on this project for too long, and the world is moving too fast. I want people to more easily be able to interact with my research thoughts as I'm having them.

In the introduction to optimization, I describe a very general framework for making sense of optimizing processes, and for measuring how much optimization is happening inside a system as it evolves over time. The optimization target is an input to that measure; given any target, any ordering over possible states, the equations will output a number for how much the system optimizes toward that target.

The paradox

But there's a bit of a paradox lurking here, which people often come across when they wonder how to use this framework to infer the optimization process's ordering. If any state ordering is valid, then couldn't any trajectory be claimed to be strongly optimizing toward whatever the last state happens to be?[1]

I think this is an extremely good question, requiring resolution before we accept the framework as being a good model of the phenomenon of optimization. Fortunately, I think it's virtually identical to a similar paradox in probability theory, whose resolution is already worked out![2] So to work through the optimization version, I'll go through the probability theory version in detail.

An analogy with random sequences

Say you encounter some generated data, for example the result of flipping a coin repeatedly. If the coin is fair, then probability of every specific outcome is equal. But when you see a sequence of 20 heads in a row, you know it's not a fair coin. How can we justify this strong belief, given that 20 heads is an equally valid outcome of flipping a fair coin?[3]

Your intuitive reaction occurs because in your mind, you are implicitly holding multiple hypotheses about the nature of the coin. And when you see the outcome of all heads, you immediately understand that it's vastly more likely that this comes from a non-fair coin than from a fair coin, and therefore the hypothesis "the coin is not fair" shoots to the top. We can work this justification out using Bayes' theorem. We'll use the following form;

P(H...
"Sorcerer's Apprentice" from Fantasia as an analogy for alignment
by awg — 2023-03-29T18:21:55.678Z
Link: https://video.disney.com/watch/sorcerer-s-apprentice-fantasia-4ea9ebc01a74ea59a5867853

The story is simple: Mickey is an apprentice to a powerful sorcerer whose magic comes from his hat. Mickey is tasked with carrying buckets of water up a long flight of stairs and dumping them into a basin–hard work for a mouse! When the sorcerer steps out, however, Mickey takes his hat and uses its magic to enchant a broomstick to do the water carrying for him. Aha! Mickey is able to rest now, and he falls asleep to dream about all the wonderful magic he can do. But he awakes to a terrible discovery: the enchanted broomstick won't stop carrying and dumping water and the basin is now overflowing! Mickey attempts to alter the broomstick's course of action, but he can't; it's too late! So he tries to hack it to pieces, thinking that will solve the problem. But no, the enchantment enables the broomstick to make copies of itself from the hacked pieces! Now there are even MORE of them doing the unwanted task, and Mickey, try as he might, can't possibly hope to overcome them. It's not until the sorcerer returns (the one who truly knows how to wield the magic hat) that all is put to rights.

I thought it fascinating how this popular story from when we were all little actually encapsulates the alignment problem pretty neatly. Just wanted to share!
The Changing Face of Twitter
by Zvi — 2023-03-29T17:50:00.747Z

[NOTE: This post has been edited extensively to account for key info I had missed.]

As we move to ban TikTok, and as we use that move as an excuse to pass a new supercharged Patriot Act for the entire internet, every other social network including Twitter moves to transform itself into TikTok.

Twitter under Elon Musk is unique in that it has a second goal, a mission of the utmost importance. To charge you $8.

This means that Twitter increasingly pushes its users towards its For You feed.

My previous understanding was that this feed was also going fully algorithmic to match TikTok’s, and would also only include posts from verified (meaning ‘paying’) accounts. Even your own followers, I believed, were to be excluded.

Elon then said he ‘forgot to mention’ that this would not be the case, and anyone you do follow will still be included. Perhaps this was indeed him forgetting to mention it, perhaps it was a reversal.

Either way, this post was temporarily unpublished until it could be reworked to account for this new information. Life comes at you fast these days.

Including followed accounts reverses much of the transformation that would have been made to the For You feed. If you like an account, you can follow that account.

The polls change remains an issue. Unverified (meaning ‘non-paying’) accounts will not be able to vote in polls. There is no indication that any amount of following, in any direction, would reverse this.

That’s on top of the new super-long Tweet options from Twitter Blue.

Elon Musk says the verification requirements are due to the threat of bots.

Neither I nor GPT-4 are optimistic about the impact of the changes. GPT-4, I think correctly, predicts loss of users, perception of elitism and misalignment of content, while the financial implications are uncertain.

When I thought your followed accounts would be excluded from For You, I thought this actually opened up a very interesting new opportunity – you could use a hybrid approach, where you used the For You tab for non-followed content, exploration and discovery, and the Following tab for a core of followed content. In theory you can still do that, if you want to move most or all your follows onto lists, but it’s tricker and higher overhead so I expect almost no one to do it, myself included.

What do all the changes to Twitter mean in practice? What should we expect? How should we adjust? Is...
Nobody’s on the ball on AGI alignment
by leopold — 2023-03-29T17:40:36.250Z
Link: https://www.forourposterity.com/nobodys-on-the-ball-on-agi-alignment/

Far fewer people are working on it than you might think, and even the alignment research that is happening is very much not on track. (But it’s a solvable problem, if we get our act together.)

Observing from afar, it's easy to think there's an abundance of people working on AGI safety. Everyone on your timeline is fretting about AI risk, and it seems like there is a well-funded EA-industrial-complex that has elevated this to their main issue. Maybe you've even developed a slight distaste for it all—it reminds you a bit too much of the woke and FDA bureaucrats, and Eliezer seems pretty crazy to you.

That’s what I used to think too, a couple of years ago. Then I got to see things more up close. And here’s the thing: nobody’s actually on the friggin’ ball on this one!

There’s far fewer people working on it than you might think. There are plausibly 100,000 ML capabilities researchers in the world (30,000 attended ICML alone) vs. 300 alignment researchers in the world, a factor of ~300:1. The scalable alignment team at OpenAI has all of ~7 people.
Barely anyone is going for the throat of solving the core difficulties of scalable alignment. Many of the people who are working on alignment are doing blue-sky theory, pretty disconnected from actual ML models. Most of the rest are doing work that’s vaguely related, hoping it will somehow be useful, or working on techniques that might work now but predictably fail to work for superhuman systems.

There’s no secret elite SEAL team coming to save the day. This is it. We’re not on track.

If timelines are short and we don’t get our act together, we’re in a lot of trouble. Scalable alignment—aligning superhuman AGI systems—is a real, unsolved problem. It’s quite simple: current alignment techniques rely on human supervision, but as models become superhuman, humans won’t be able to reliably supervise them.

But my pessimism on the current state of alignment research very much doesn’t mean I’m an Eliezer-style doomer. Quite the opposite, I’m optimistic. I think scalable alignment is a solvable problem—and it’s an ML problem, one we can do real science on as our models get more advanced. But we gotta stop fucking around. We need an effort that matches the gravity of the challenge.[1]

Alignment is not on track

A

recent post

estimated that there were 300 full-time technical AI safety researchers (sounds plausible to me, if we’re ...
Want to win the AGI race? Solve alignment.
by leopold — 2023-03-29T17:40:36.187Z
Link: https://www.forourposterity.com/want-to-win-the-agi-race-solve-alignment/

Society really cares about safety. Practically speaking, the binding constraint on deploying your AGI could well be your ability to align your AGI. Solving (scalable) alignment might be worth lots of $$$ and key to beating China.

Look, I really don't want Xi Jinping Thought to rule the world. If China gets AGI first, the ensuing rapid AI-powered scientific and technological progress could well give it a decisive advantage (cf

potential for >30%/year economic growth with AGI

). I think there's a very real specter of global authoritarianism here.[1]

Or hey, maybe you just think AGI is cool. You want to go build amazing products and enable breakthrough science and solve the world’s problems.

So, race to AGI with reckless abandon then? At this point, people get into agonizing discussions about safety tradeoffs.[2] And many people just mood affiliate their way to an answer: "accelerate, progress go brrrr," or "AI scary, slow it down."

I see this much more practically. And, practically, society cares about safety, a lot. Do you actually think that you’ll be able to and allowed to deploy an AI system that has, say, a

10% chance of

destroying all of humanity

?[3]

Society has started waking up to AGI; like covid, the societal response will probably be a dumpster-fire, but it’ll also probably be quite intense. In many worlds, to deploy your AGI systems, people will need to be quite confident that your AGI won’t destroy the world.

Right now, we’re very much not on track to solve the alignment problem for superhuman AGI systems (“scalable alignment”)—but it’s a solvable problem, if we get our act together. I discuss this in my main post today (“Nobody’s on the ball on AGI alignment”). On the current trajectory, the binding constraint on deploying your AGI could well be your ability to align your AGI—and this alignment solution being unambiguous enough that there is consensus that it works.

Even if you just want to win the AGI race, you should probably want to invest much more heavily in solving this problem.

Things are going to get crazy, and people will pay attention

A mistake many people make when thinking about AGI is imagining a world that looks much like today, except for adding in a lab with a super powerful model. They ignore the

endogenous societal response

.

I and many others made this mistake with covid—we were freaking out in February 2020, and des...