trentmkelly/LessWrong-43k huggingface dataset:
What's Hard About The Shutdown Problem
by johnswentworth — 2023-10-20T21:13:27.624Z

I’ve been having some productive back-and-forth with Sami Petersen about his

Invulnerable Incomplete Preferences

(which is intended to complement Elliot Thornley’s approach to the Shutdown Problem). This post is a (non-formal) distillation of some useful pieces which I’ve gotten out of that discussion so far.

The Shutdown Problem

Conceptually: we want to equip a powerful AI with a shutdown button. If and when the button is unpressed, we want the AI to follow one goal (we’ll call it the “on-goal”). If and when the button is pressed, we want the AI to follow a different goal (we’ll call it the “off-goal”). Presumably the off-goal involves the AI shutting down itself and all its associated tools/children/etc, but that part’s flavor text for current purposes.

The key failure mode we want to avoid here is accidentally incentivizing the AI to manipulate the shutdown button - i.e. either cause it to be pressed, or cause it to be unpressed. Why would that happen? Well, suppose that when we first turn on the AI, it is just optimizing for the on-goal. The AI realizes that, if the button is pressed, its future self will stop optimizing for the on-goal, and therefore it will achieve lower performance on the on-goal. So, the AI realizes that it can achieve better performance on the on-goal by preventing the button from being pressed. And since it’s just optimizing for performance on the on-goal, it then prevents the button from being pressed.

MIRI’s paper on the shutdown problem

goes through more examples of simple proposals for the shutdown problem, and how they incentivize button manipulation of one sort or another. It turns out that it’s generally Hard to get the desired behavior (i.e. no button manipulation) out of any expected utility maximizer.

Why It’s Hard, Level 1: Desired Behavior Implies Incomplete Revealed Preferences

Why is it hard to get an expected utility maximizer to not manipulate the button (while still doing useful stuff otherwise)? Here’s one intuitive answer I got out of the discussion with Sami.

The central idea is to ask what the behavior we want from the shutdown-button-agent implies about that agent’s revealed preferences.

Suppose that, at various times, the agent is offered opportunities to spend resources in order to cause the button to be pushed/unpushed. We want the agent to turn down such opportunities, in both directions - implying ...
Holly Elmore and Rob Miles dialogue on AI Safety Advocacy
by Bird Concept — 2023-10-20T21:04:32.645Z

Holly is an independent AI Pause organizer, which includes organizing protests (like this upcoming one). Rob is an AI Safety YouTuber. I (jacobjacob) brought them together for this dialogue, because I've been trying to figure out what I should think of AI safety protests, which seems like a possibly quite important intervention; and Rob and Holly seemed like they'd have thoughtful and perhaps disagreeing perspectives.

Quick clarification: At one point they discuss a particular protest, which is the anti-irreversible proliferation protest at the Meta building in San Francisco on September 29th, 2023 that both Holly and Rob attended.

Also, the dialogue is quite long, and I think it doesn't have to be read in order. You should feel free to skip to the section title that sounds most interesting to you.

jacobjacob

Let's jump right into it. Rob, when I suggested discussing protests, you said you were confused about: "How do we get to be confident enough in anything to bring it the energy that activism seems to require?" I very much agree with this confusion!

Robert Miles

This is a pretty standard joke/stereotype, the XKCD protest where the signs are giant and full of tiny text specifying exactly what we mean. My protest sign just said "Careful Now", in part because I'm not sure what else you can fit on a sign that I would fully endorse.

Holly_Elmore

There's a big difference in communication style between something like science or lesswrong and advocacy. You have less bandwidth in advocacy. It's much closer to normal speech, where we don't exactly qualify every statement-- and that's in many ways more accurate as a practice considering how short of a window you have for that kind of communication. "Pause AI" does the job where an exact description of how you implement the policy does not-- it would be too confusing and hard to take in. And similarly the resolution of policy you have to discuss in advocacy is a lot lower (or the concepts higher level), so you can endorse a very broad policy aim that you do support while having a lot of genuine uncertainty about what mechanism or exact approach to use.

jacobjacob

(A related old, good post: You Get About Five Words. That post also suggests that you have 3 words left. Could spend them on "PauseAI: but not sure" )

Holly_Elmore

(Okay this may get us to a crux: that's a terrible idea)

jacobjacob

(Lo

...
TOMORROW: the largest AI Safety protest ever!
by Holly_Elmore — 2023-10-20T18:15:18.276Z

Tomorrow, PauseAI and collaborators are putting on the largest AI Safety protest to date, across 7 locations in 6 countries. All are eagerly welcomed!
Your presence at this protest is a rare impact opportunity when in-person volunteering is not fungible with money or intellectual support-- showing up is how we show the public our concern in a way that humans instinctively understand. Plus, we think it will be fun, with camaraderie and drinks afterward. It feels great to pluck new low-hanging fruit and flex different muscles in the fight for AI Safety.

October 21st (Saturday), in multiple countries

US, California, San Francisco (

facebook

)
US, Massachusetts, Boston (

facebook

)
UK, Parliament Square, London (

Sign up

,

facebook

)
Netherlands, Den Haag (

Sign up

)
Australia, Melbourne (

Sign up

)
Canada, Ottawa (Organized by Align the World, sign up on

facebook

or

EventBrite

)
Denmark, Copenhagen (

Facebook

)
Your country here?

Discuss on discord!

Why we protest

AI is rapidly becoming more powerful, far faster than virtually any AI scientist has predicted. Billions are being poured into AI capabilities, and the results are staggering. New models are

outperforming humans

in a lot of domains. As capabilities increase, so do the

risks

. Scientists are even

warning

that AI might

end up destroying humanity

. This dire outcome not only seems possible, but also likely, as the average probability estimates for these outcomes

range from 14% to 40%

.

We need our leaders to listen to these warnings, but they are not taking this topic remotely as seriously as they should. There is AI safety legislation being drafted, but

not a single measure would actually prevent or delay superintelligent AI

.

Over 70%

wants to slow down AI, and

over 60%

want regulation to actively prevent superintelligent AI. Why is there no legislation draft that actually does this? The answer is lobbying: our politicians are

mostly meeting AI company CEOs

, and they will push policy measures that are in their interest.

On November 1st and 2nd, the very first AI Safety Summit will be held in the UK. The perfect opportunity to set the first steps towards sensible international AI safety regulation.

What we ask

Policy makers: Don’t allow companies to build a superintelligence. Regulations and hardware restrictions should apply before training has started as it is very difficult to control dissemination once a new capability has bee

...
The Overkill Conspiracy Hypothesis
by ymeskhout — 2023-10-20T16:51:20.308Z

We couldn’t get the lighting exactly right otherwise.

The term conspiracy theory is wielded as a pejorative, alluding to on-its-face absurdity. But the vocabulary we use has a serious ambiguity problem because conspiracies are not figments of the imagination. There is a tangible and qualitative distinction between plain-vanilla conspiracies (COINTELPRO, Operation Snow White, or the Gunpowder Plot) and their more theatrical cousins (flat earth theory, the moon landing hoax, or the farcical notion that coffee tastes good), yet a clear delineation has been elusive and it’s unsatisfying to just assert “this one is crazy, and this one isn’t.” Both camps involve subterfuge, malevolent intent, covert operations, misinformation, orchestrated deceit, hidden agendas, clandestine networks, and yes, conspiracy, and yet the attempts to differentiate between the two have veered into unsatisfactory or plainly misleading territories.

What I’ll argue is the solution boils down to a simple reconfiguration of the definition that captures the essence of the absurdity: conspiracy theories are theories that assume circumstances that render the titular “conspiracy” unnecessary. This is what I’ll refer to as the Overkill Conspiracy Hypothesis (OCH). Before we dive into this refinement, it’s helpful to explore why traditional distinctions have fallen short.

The section on differences in

The People’s Pedia

showcases some of these misguided attempts. For example, conspiracy theories tend to be in opposition to mainstream consensus but that’s a naked appeal to authority — logic that would have tarred the

early challengers to the supposed health benignity of smoking

as loons. Or that theories portray conspirators acting with extreme malice, but humans can indeed harbor evil intentions (see generally, human history). Another relies on the implausibility of maintaining near-perfect operational security. This is getting better, but while maintaining secrecy is hard, it’s definitely not impossible. We have actual, real-life examples of covert military operations, or drug cartels that manage to operate clandestine billion-dollar logistical enterprises.

There’s still some useful guidance to draw from the pile of chaff, and that’s conspiracy theories’ lack of, and resistance to,

falsifiability

. Despite its unfortunate name, falsifiability is one of my nearest and dearest c...
I Would Have Solved Alignment, But I Was Worried That Would Advance Timelines
by 307th — 2023-10-20T16:37:46.541Z

The alignment community is ostensibly a group of people concerned about AI risk. Lately, it would be more accurate to describe it as a group of people concerned about AI timelines.

AI timelines have some relation to AI risk. Slower timelines mean that people have more time to figure out how to align future intelligent AIs, potentially lowering the risk. But increasingly, slowing down AI progress is becoming an end in itself, taking precedence over ensuring that AI goes well. When the two objectives come into conflict, timelines are winning.

Pretty much anything can be spun as speeding up timelines, either directly or indirectly. Because of this, the alignment community is becoming paralyzed, afraid of doing anything related to AI - even publishing alignment work! - because of fears that their actions will speed up AI timelines.

The result is that the alignment community is becoming less effective and more isolated from the broader field of AI; and the benefit to this, a minor slowdown in AI progress, does not nearly outweigh the cost.

This is not to say that trying to slow down AI progress is always bad. It depends on how it is done.

Slowing Down AI: Different Approaches

If you want to slow down AI progress, there are different ways you can go about it. One way to categorize it is by who a slowdown affects.

Government Enforcement: Getting the government to slow down progress through regulation or bans. This is a very broad category, including both regulations in a certain country vs. international bans on models over a certain size, but the important distinguishing feature of this category is that the ban applies to everyone, or, if not to everyone, at least to a large group that is not selected for caring about AI risk.
Voluntary Co-Ordination: If OpenAI, DeepMind, and Anthropic all agreed to halt capability work for a period of time, this would be voluntary co-ordination. Because it's voluntary, a pause around reducing AI risk could only affect organizations that are worried about AI risk.
Individual Withdrawal: When individuals concerned about AI risk refrain from going into AI, for fear of having to do capability work and thereby advancing timelines, this is individual withdrawal; ditto for other actions that are not taken in order to avoid speeding up timelines, such as not publishing alignment research.

The focus of this piece is on individual withdr...
Internal Target Information for AI Oversight
by Paul Colognese — 2023-10-20T14:53:00.284Z

Thanks to Arun Jose for discussions and feedback.

Summary

In this short post, we discuss the concept of Internal Target Information within agentic AI systems, arguing that agentic systems possess internal information about their targets. This information, we propose, can potentially be detected and interpreted by an overseer before the target outcome is realized in the environment, offering a pathway to preempt catastrophic outcomes posed by future agentic AI systems.

This discussion aims to highlight the key idea that motivates our current research agenda, laying a foundation for forthcoming work.

We’ll start by introducing the inner alignment problem and why oversight of an agent’s internals is important. We’ll then introduce a model of an overseer overseeing an agent. Finally, we’ll introduce and discuss the notion of Internal Target Information in more detail and how it might be used in the oversight process.

Oversight of an AI’s Internal Target Information. The Overseer detects that the AI’s target is to turn all humans into paperclips and so shuts the AI down, preventing the catastrophe. Credit: DALL-E 3.

The Inner Alignment Problem and Internal Oversight

We are concerned with the possibility of creating agents with misaligned objectives, potentially leading to catastrophic real-world outcomes. A conceivable solution lies in effective oversight: detecting misalignment early enough allows for timely intervention, preventing undesirable outcomes.

Oversight, based on behavioral observations, may fail to confidently predict future outcomes pursued by the Agent, especially in the face of goal misgeneralization and deceptive alignment.

In the remainder of this post, we will explore the idea that information about the agent's true objective may be contained in its internals, and so may be detectable by an overseer equipped with sufficiently strong interpretability tools.

To begin our exploration of this idea, we start by introducing a model of the oversight process.

Oversight Model

We introduce a model of an Overseer overseeing an Agent that begins to clarify what good oversight might entail, mainly focusing on the objective of the Overseer: to prevent catastrophic outcomes caused by misaligned agentic AI. Moreover, we hope that a more refined future version of this model may help establish benchmarks to evaluate oversight methods.

The Agent

The Agen...
On the proper date for solstice celebrations
by jchan — 2023-10-20T13:55:02.999Z

Convenience vs. accuracy

The question of the "correct" date on which to hold winter solstice events has the character of a hyperstition, i.e. it's not really important right now, but if we argue about it enough we can make it important, if that's what we want.

You might say:

The solstice event should be held on a weekend before the astronomical solstice to maximize convenience, even if this puts it more than a week early ("Earlybird")

The solstice event should be held on the last weekend before the astronomical solstice ("Last-Weekender")

The solstice event should be held on the astronomical solstice, even if this is on a weeknight ("Astronomical")

To these we can also add a fourth ("Mu") option, to wit: "The date of the solstice event is unimportant and we shouldn't try to make it important."

(To be honest, I put a fair amount of weight on this last option myself; the rest of this post is somewhat speculative. Also, in general, you should be on the lookout for sneaky attempts to polarize an issue by setting up an n-chotomy and then hyperstitiously asserting that your choice there correlates with a larger cluster of beliefs/values. This is essentially what I'm doing here, but at least not sneakily.)

When choosing a date, it mostly comes down to who your target audience is. The Earlybird position makes sense if you're planning a "destination solstice" like the SF Bay or NYC, where you're looking to get visitors from out-of-town. If you're aiming it at people who live in the area but leave to spend the holidays with their families, then you'll lean towards the Last-Weekender position. But by contrast, the Astronomical position conveys the message "We are your family; this is your home."

This last one might seem a bit scary and culty. I want to at least put it forward as a plausible option (since Austin has been doing this the last few years, while as far as I can tell no other communities have held solstice events on weeknights), but also open up the debate about whether it's a good idea.

Analogy: Should Halloween be moved to a weekend?

Halloween is invariably celebrated on October 31 and has stubbornly resisted attempts to move it to a more "convenient" weekend-night. Practically speaking this is due to the coordination problem - if you unilaterally go trick-or-treating on any other night, you'll come home empty-handed; and if you try to give out candy on a dif...
Are (at least some) Large Language Models Holographic Memory Stores?
by Bill Benzon — 2023-10-20T13:07:02.041Z

Cross-posted from New Savanna.

That’s been on my mind for the last week or two, ever since my recent work on ChatGPT’s memory for texts [1]. On the other than, there’s a sense in which it’s been on my mind for my entire career, or, more accurately, it’s been growing in my mind ever since I read Karl Pribram on neural holography back in 1969 in Scientific American [2]. For the moment let’s think of it as a metaphor, just a metaphor, nothing we have to commit to. Just yet. But ultimately, yes, I think it’s more than a metaphor. To that end I note that cognitive psychologists have recently been developing the idea of verbal memory as holographic in nature [3].

Note: These are quick and dirty notes, a place-holder for more considered thought.

Holography in the mind

Let’s start with an article David Hays and I published on neural holography as the neural underpinning of metaphor [4]. Here’s where we explain the holographic process:

Holography is a photographic technique for making images. A beam of laser light is split into two beams. One beam strikes the object and is reflected to a photographic plate. The other beam, called a reference beam, goes from laser to plate directly. When they meet, the two beams create an interference pattern—imagine dropping two stones into a pond at different places; the waves propagating from each of these points will meet and the resulting pattern is an interference pattern. The photographic plate records the pattern of interference between the reference beam and the reflected beam.

The image recorded on the film doesn't look at all like an ordinary photographic image—it’s just a dense mass of fine dots. But when a beam of laser light having the same properties as the original reference beam is directed through the film an image appears in front of the film. The interaction of the laser beam and the hologram has recreated the wave form of the laser beam which bounced off the object when the hologram was made. The new beam has extracted the image from the plate.

Holography is, as its name suggests, holistic. Every part of the scene is represented in every part of the plate. (This situation is most unlike ordinary photography, which uses a good lens to focus infinitesimal parts of the scene onto equally infinitesimal parts of the plate.) With such a determinedly nondigital recording, certain mathematical possibilities can be realized more

...
Mechanistic interpretability of LLM analogy-making
by Sergii — 2023-10-20T12:53:26.550Z
Link: https://grgv.xyz/blog/copycat/

Can LLM make analogies? Yes, according to tests done by Melanie Mitchell a few years back, GPT-3 is quite decent at “Copycat” letter-string analogy-making problems. Copycat was invented by Douglas Hofstadter in the 80s, to be a very simple “microworld”, that would capture some key aspects of human analogy reasoning. An example of a Copycat problem:

”If the string abc changes to the string abd, what does the string pqr change to?“

Many more examples are collected on this page.

A project that I'm working on while studying mechanistic interpretability (MI), is applying MI to an LLM's ability to solve Copycat problems.

According to Douglas Hofstadter, analogy is the core of cognition, and it can be argued that it is a basis for various abstract reasoning abilities. There are other similar problem domains that require few-shot abstract reasoning — like inductive program synthesis, abstraction and reasoning challenge, etc… Still, Copycat is the simplest one, which makes it a good starting point for MI.

selecting a model & test prompts

It would be nice to use GPT-2 for the investigation, a lot of recent work in MI has been done based on it. But, alas, it does not seem to be able to solve analogy puzzles.

I have chosen Llama-7B-chat — it’s able to solve Copycat-like problems, and is small enough to be convenient for experimentation. It does not work as well as GPT-3.5 for Copycat and I had to tweak the problem formulation, but, eventually, I got it to solve simplified problems like:

prompt: "0 1 2 to 2 1 0, 1 2 3 to 3 2 1, 4 5 6 to ", output: “6 5 4”
prompt: "0 1 2 to 0 1 3, 1 2 3 to 1 2 4, 4 5 6 to ", outuput: “4 5 7”
prompt: “0 1 to 0 1 1, 1 2 to 1 2 2, 4 5 to “, output: “4 5 5”

Llama-7B has a fairly "standard" transformer architecture with 32 blocks and 32 attention heads in each block.

logit lens

I have started with applying logit lens [3] to the first test prompt:

"0 1 2 to 2 1 0, 1 2 3 to 3 2 1, 4 5 6 to "

Logit lens output for this prompt is shown below: top 5 tokens with probabilities, predicted for for the last token in the prompt, for all 32 blocks of Llama-7b-chat. It's supposed to predict "6", the first token of the correct solution:

0: пута 0,076 | penas 0,015 | sier 0,011 | сылки 0,009 | partiellement 0,009 |
1: rep 0,006 | accomp 0,006 | soft 0,005 | regular 0,005 | use 0,004 |
2: rep 0,016 | accomp 0,010 | M 0,007 | gex 0,004

...