trentmkelly/LessWrong-43k huggingface dataset:
Why does advanced AI want not to be shut down?
by RedFishBlueFish — 2023-03-28T04:26:23.914Z

I've always been pretty confused about this.

The standard AI risk scenarios usually (though I think not always) suppose that advanced AI wants not to be shut down. As commonly framed, the AI will fool humanity into believing it is aligned so as not to be turned off, until - all at once - it destroys humanity and gains control over all earth's resources.

But why does the AI want not to be shut down?

The motivation behind a human wanting not to die comes from evolution. You die before reproduction age, and you won't be able to pass on your not-afraid-of-death-before-reproduction-age genes. You die after reproduction age, and you won't be able to take care of your children to make sure they pass on your genes. Dying after the age when your children are grown only started to happen after humans had evolved into their current state, I believe, and so the human emotional reaction defaults to the one learned from evolution. How this fits into the "human utility function" is a controversial philosophical/psychological question, but I think it's fair to say that the human fear of dying surpasses the desire not to miss out on the pleasure of the rest of your life. We're not simply optimizing for utility when we avoid death.

AI is not subject to these evolutionary pressures. The desire not to be shut down must come from an attempt to maximize its utility function. But with the current SOTA techniques, this doesn't really make sense. Like, how does the AI compute the utility of being off? A neural network is trained to optimize a loss function on input. If the AI doesn't get input, is that loss... zero? That doesn't sound right. Just by adding a constant amount to the loss function we should be able to change the system from one that really wants to be active to one that really wants to be shut down, yet the gradients used in backpropagation stay exactly the same. My understanding is that reinforcement learning works the same way; GPT243, so long as it is trained with the same techniques, will not care if it is shut down.

Maybe with a future training technique we will get an AI with a strong preference for being active to being shut down? I honestly don't see how. The AI cannot know what it's like to be shut down, this state isn't found anywhere in its training regime.

There has to be some counterargument here I'm not aware of.
100 Dinners And A Workshop: Information Preservation And Goals
by Stephen Fowler — 2023-03-28T03:13:06.362Z

Produced on a grant from the LTFF. Thanks to Justis for proofreading and Paul Colognese, David Udell, Alex Altair, Nicky Pochinkov, and Jessica Rumbelow for discussion and feedback.
I would like to reason about the emergence and impact of an AGI on our local region of space, and hopefully find ways of ensuring that impact is positive. At the very least it would be nice to obtain theoretical guarantees that it could prevented from posing an existential threat to our species. Unfortunately, progress in this area has been slow and making even basic claims about AGI has proven to be exceptionally challenging.

Here I take a different perspective and examine a toy model of the system responsible for giving rise to the AGI. I highlight the signature of programmed goals and their correlation with events that occur. I also do a quick dive into which class of intelligent systems I think should be the focus of alignment research.

The Secretary And The Event Planner:

John has written about the need to narrow our definitions of agency to those that "optimize at a distance" to circumvent various issues with the traditional definition of agent. "If I’m planning a party, then the actions I take now are far away in time (and probably also space) from the party they’re optimizing". We're going to think about planning events far away in time, but not from the perspective of the planner themself.

Lets kick things up a notch. Picture yourself swamped with the responsibility of planning 100 dinner parties. There's no way you'll be able to do this yourself. No worries, you think, and contact your secretary. One problem. He's too busy. Says he's got 100 dinner outfits to plan out. Damn.
The solution of course is for you to instruct your secretary to hire an event planner. Genius. You call your secretary to inform him of the plan, he says he likes it but you can tell he's barely listening. Your secretary is very distracted and you know he will not take any initiative. You write out a detailed set of instructions that not only conveys information like how how the parties should be executed, the themes, the guest lists, etc but also instructions for how your secretary should go about selecting the event planner in the first place.
You send the list to the secretary and he follows it to the letter. It all goes great. After holdin...
Demons from the 5&10verse!
by Slimepriestess — 2023-03-28T02:41:11.487Z
Link: https://voidgoddess.org/2023/03/27/demons-from-the-510verse/

The

5 and 10 error

is a glitch in logical reasoning that was first characterized in formal proof by Scott Garrabrant of MIRI. While the original version of the problem was something specifically concerning AIs based on logical induction, it generalizes out into humans startlingly often once you know how to look for it. However, due to how rudimentary and low level the actual error in reasoning is, it can be both difficult to point out and easy to fall into, making it especially important to characterize. There is also a tremendous amount of harm being created by compounding 5&10 errors within civilization and escaping this destructive equilibrium is necessary in order for the story of humanity to end anywhere other than summoning the worst demon god it can find and feeding the universe to it.

The error in reasoning goes like this: you’re presented with a pair of options, one of which is clearly better than the other. They are presented as equal choices, you could take $5 or you could take $10. This is a false equivalence being created entirely by the way you’re looking at the scenario, but when that equivalence gets into your reasoning it wreaks havoc on the way you think. One of these is clearly and unambiguously better than the other, if you have something you care about that runs through this, you will never make this mistake in that area because it will obviously be a dumb move.

But these are being presented as equal options, you could take the $5 instead of the $10, and if you could do that, there must be a valid reason why you would do that. Otherwise you would be stupid and feel bad about it, so that can’t be right, you shouldn’t be feeling stupid and bad. This is where the demon summoning comes in.

The space of all possible reasons why you would take a given action, for a fully general agent, is infinite, a sprawling fractal of parallel worlds stacked on parallel worlds, out to the limits of what you as an agent can imagine. “What if there was a world where you were batman?” yeah like that. If you scry into infinity for a reason why you could take an action, you will find it. You will find infinite variations on it. You will find it easily and without much challenge. You are literally just making something up, and you can make up whatever reason you want, that’s the problem.

Many of the reasons you could make up will be falsifiable, so you can always...
Can GPT-4 play 20 questions against another instance of itself?
by Nathan Helm-Burger — 2023-03-28T01:11:46.601Z
Link: https://evanthebouncy.medium.com/llm-self-play-on-20-questions-dee7a8c63377

Seems like a neat way to check understanding / world-model. Not my idea, saw this post: https://evanthebouncy.medium.com/llm-self-play-on-20-questions-dee7a8c63377
Geoffrey Hinton - Full "not inconceivable" quote
by WilliamKiely — 2023-03-28T00:22:01.626Z

CBS interview with Geoffrey Hinton published March 25th, 2023:

In the last two days I've seen several people share news articles about Hinton saying it is "not inconceivable" that AI will wipe out humanity in this interview.

The articles often don't give the full context of the quote, so I made this post to do that, even though it might be overkill:

Quote

Q:[1] "This is like the most pointed version of the question, and you can just laugh it off or not answer it if you want, but what do you think the chances are of AI just wiping out humanity? Can we put a number on that?"

[27:40]

Hinton: "It's somewhere between, um, naught percent and 100 percent. [Hinton *laughs*] I mean, I think, I think it's not inconceivable. That's all I'll say. I think if we're sensible, we'll try and develop it so that it doesn't. But what worries me is the political system we're in, where it needs everybody to be sensible.

Note that just before this Hinton was talking about how he thinks "if Putin had autonomous lethal weapons he'd use them right away," which plausibly may have led him to say that worries him is the political system we're in.

Transcript leading up to the quote

[25:52]

Q: "Some people are worried that this could take off very quickly and we just might not be ready for that. Does that concern you?"

Hinton: "It does a bit. Until quite recently I thought it was going to be like 20-50 years before we had general purpose AI and now I think it may be 20 years or less, so."

Q: "Some people think it could be like 5 [years]. Is that silly?"

Hinton: "I wouldn't completely rule that possibility out now, whereas previously, a few years ago, I would have said 'no way.'"

Q: "And then, some people say, AGI could be massively dangerous to humanity because we just don't know what a system that's so much smarter than us will do. Do you share that concern?"

Hinton: "I do a bit. Um, I mean obviously what we need to do is make this synergistic, have it so it helps people. And I think the main issue here, well one of the main issues, is the political systems we have. So, I'm not confident that President Putin is going to use AI in ways that help people."

[...]

Hinton: "I think if Putin had autonomous lethal weapons he'd use them right away."

Q: "This is like the most pointed version of the question, and you can just laugh it off or not answer it if you want, but what do you think the ch...
An A.I. Safety Presentation at RIT
by Nicholas / Heather Kross — 2023-03-27T23:49:59.657Z
Link: https://www.youtube.com/watch?v=V4fkKcLhEyQ

How do you get people into AI safety, alignment, and even DontKillEveryoneism?

Well, we at the Effective Altruism Club at RIT (Rochester Institute of Technology) (NY, USA) decided to take a crack at it.

We gave a presentation to the university's AI Club, with around 2-3 dozen people in the audience. The talk covers the basic arguments behind AI safety, responses to common objections, and resources for people to learn more.

The talk can be watched here.

Mica White is the current President of the EA Club at RIT. Nicholas Kross is an alumni who was formerly on the club's e-board.

Hopefully this will be a decent resource for others to use, but we're especially looking for feedback and discussion of how these sorts of presentations can be done better.
Which AI outputs should humans check for shenanigans, to avoid AI takeover? A simple model
by Tom Davidson — 2023-03-27T23:36:39.701Z

Epistemic status: incomplete early stage thinking. Thinking this through made me feel less confused about how labs might one day prioritise which AI outputs humans should check for sabotage.

Short summary

As AI performs more and more tasks for humans, it could purposefully sabotage some tasks in order to disempower humanity.

We could reduce the chance of AI takeover by having humans randomly check some of AI’s work for signs of sabotage.

Humans will only be able to check a tiny fraction of AI’s work, so it will be useful to carefully prioritise which tasks humans check.

According to this simple framework:

We should describe multiple disjunctive routes to AI takeover, and identify which tasks AI would need to sabotage for each.

Humans should put more effort into checking routes to takeover that are harder to detect.

When trying to detect AI takeover via a specific route, humans should put more effort into checking tasks where sabotage is easier to detect.

This framework is very abstract, to the point where it’s currently not practically useful.

Summary

AI labs will increasingly get AIs to perform the tasks their human employees currently perform. Tasks like generating new theoretical alignment proposals, testing whether AI is aligned, testing whether AI has dangerous capabilities, designing new AI architectures, computer security ensuring that AIs can’t hack the server they’re running on to take control of its compute, etc.

A big risk is that AIs will purposefully sabotage some of these tasks in a way that enables AI takeover.

There will probably be various possible ways that AIs could sabotage some subset of tasks in order to take over. Some examples:

Propose subtly flawed solutions to AI alignment AND sabotage the alignment tests. Then wait for misaligned AI to be scaled up and deployed widely before suddenly taking over.

This requires sabotaging two types of task: proposing alignment solutions and implementing alignment tests.

Advise humans at labs to deploy AI internally despite ambiguous evidence about alignment AND leave back doors that allow AIs to hack the server. Then wait for misaligned AI to hack the server, quickly foom, and take over.

This again requires sabotaging two types of task: strategic advice and computer security.

One important way to reduce the risk of AI takeover is to have humans check AI outputs for signs of sabotage. The ...
The Prospect of an AI Winter
by Erich_Grunewald — 2023-03-27T20:55:35.619Z
Link: https://www.erichgrunewald.com/posts/the-prospect-of-an-ai-winter/

Summary

William Eden forecasts an AI winter. He argues that AI systems (1) are too unreliable and too inscrutable, (2) won't get that much better (mostly due to hardware limitations) and/or (3) won't be that profitable. He says, "I'm seeing some things that make me think we are in a classic bubble scenario, and lots of trends that can't clearly continue."

I put 5% on an AI winter happening by 2030, with all the robustness that having written a blog post inspires, and where AI winter is operationalised as a drawdown in annual global AI investment of ≥50%.[1] (I reckon a winter must feature not only decreased interest or excitement, but always also decreased funding, to be considered a winter proper.)

There have been two previous winters, one 1974-1980 and one 1987-1993. The main factor causing these seems to have been failures to produce formidable results, and as a consequence wildly unmet expectations. Today's state-of-the-art AI systems show impressive results and are more widely adopted (though I'm not confident that the lofty expectations people have for AI today will be met).

I think Moore's Law could keep going for decades.[2] But even if it doesn't, there are many other areas where improvements are being made allowing AI labs to train ever larger models: there's improved yields and other hardware cost reductions, improved interconnect speed and better utilisation, algorithmic progress and, perhaps most importantly, an increased willingness to spend. If 1e35 FLOP is enough to train a transformative AI (henceforth, TAI) system, which seems plausible, I think we could get TAI by 2040 (>50% confidence), even under fairly conservative assumptions. (And a prolonged absence of TAI wouldn't necessarily bring about an AI winter; investors probably aren't betting on TAI, but on more mundane products.)

Reliability is definitely a problem for AI systems, but not as large a problem as it seems, because we pay far more attention to frontier capabilities of AI systems (which tend to be unreliable) than long-familiar capabilities (which are pretty reliable). If you fix your gaze on a specific task, you usually see a substantial and rapid improvement in reliability over the years.

I reckon inference with GPT-3.5-like models will be about as cheap as search queries are today in about 3-6 years. I think ChatGPT and many other generative models will be profitable wit

...
Best arguments against the outside view that AGI won't be a huge deal, thus we survive.
by Noosphere89 — 2023-03-27T20:49:24.728Z

Specifically, against the following view described by a comment:

There seems to be a lack of emphasis in this market on outcomes where alignment is not solved, yet humanity turns out fine anyway. Based on an Outside View perspective (where we ignore any specific arguments about AI and just treat it like any other technology with a lot of hype), wouldn't one expect this to be the default outcome?

Take the following general heuristics:

If a problem is hard, it probably won't be solved on the first try.

If a technology gets a lot of hype, people will think that it's the most important thing in the world even if it isn't. At most, it will only be important on the same level that previous major technological advancements were important.

People may be biased towards thinking that the narrow slice of time they live in is the most important period in history, but statistically this is unlikely.

If people think that something will cause the apocalypse or bring about a utopian society, historically speaking they are likely to be wrong.

This, if applied to AGI, leads to the following conclusions:

Nobody manages to completely solve alignment.

This isn't a big deal, as AGI turns out to be disappointingly not that powerful anyway (or at most "creation of the internet" level influential but not "disassemble the planet's atoms" level influential)

I would expect the average person outside of AI circles to default to this kind of assumption.

Ideally, details are provided for why the outside view presented here is less favored on the evidence than the idea that AGI or PASTA will be a big deal, as popularized by Holden Karnofsky. Also, ideally you can estimate how much impact AI will have say, this century.

Motivation: I'm asking this question because one thing I notice is that there's the unstated assumption that AGI/AI will be a huge deal, and how much of a big deal would change virtually everything about LW works, depending on the answer. I'd really like to know why LWers hold that AGI/ASI will be a big deal.
EA & LW Forum Weekly Summary (20th - 26th March 2023)
by Zoe Williams — 2023-03-27T20:46:55.269Z

This is part of a weekly series summarizing the top posts on the EA and LW forums - you can see the full collection here. The first post includes some details on purpose and methodology. Feedback, thoughts, and corrections are welcomed.

If you'd like to receive these summaries via email, you can subscribe here.

Podcast version: Subscribe on your favorite podcast app by searching for 'EA Forum Podcast (Summaries)'. A big thanks to Coleman Snell for producing these!
Author's note: I'm currently travelling, which means:
a) Today's newsletter is a shorter one - only 9 top posts are covered, though in more depth than usual.
b) The next post will be on 17th April (three week gap), covering the prior three weeks at a higher karma bar.
After that, we'll be back to the regular schedule.

Object Level Interventions / Reviews

How much should governments pay to prevent catastrophes? Longtermism’s limited role

by EJT, CarlShulman

Linkpost for

this paper

, which uses standard cost-benefit analysis (CBA) with detrimental assumptions (eg. giving no value to future generations, only assessing benefits to Americans, and only assessing value from preventing existential threats) to show that even under those conditions governments should be spending much more on averting threats from nuclear war, engineered pandemics, and AI.
Their analysis primarily relies on previously published estimates of risks, concluding US citizens alive today have a ~1% risk of dying from these causes in the next decade. They estimate $400B in interventions could reduce the risk by minimum 0.1 percentage points, and that using the lowest figure for the US Department of Transportation’s value of a statistical life, this would result in ~$646B in value of American lives saved.

They suggest longtermists in the political sphere should change their messaging to revolve around this standard CBA-driven catastrophe policy, which is more democratically acceptable than policies relying on the cost to future generations. They suggest it would also reduce risk almost as much as a strong longtermist policy (particularly if the CBA incorporates an argument for citizens ‘altruistic willingness to pay’ ie. some level of addition for the benefit to future generations).

Assessment of Happier Lives Institute’s Cost-Effectiveness Analysis of StrongMinds

by GiveWell

The Happier Lives Institute (HLI) has argued th...
Three of my beliefs about upcoming AGI

by Robert_AIZI — 2023-03-27T20:27:57.234Z
Link: https://aizi.substack.com/p/three-of-my-beliefs-about-upcoming

[Epistemic Status: Semi-informed conjecture. Feedback is welcome.]

At

EAG

last month, I got the advice to develop my own mental model of what general/transformative/superintelligent AI will look like when it gets here, and where I think the risks lie. This week I’ll be writing a series of shorter posts thinking out loud on this.

I want to start by stating three of my intuitions about what artificial general intelligence (AGI) will look like and how safe it will be, which I’ll phrase as conjectures:

Conjecture A: Large Language Models (LLMs) will be central to (our first) Artificial General Intelligences (AGIs)
Conjecture B: LLM capabilities and safety are heavily dependent on their prompt and the way they are invoked (in addition to the model).

By “the way they are invoked” I mean the call-execute loop external to the LLM itself, including whether its outputs can be interpreted and run by an external tool. Examples of different ways they could be invoked include “GPT-4 via ChatGPT”, “GPT-4 via ChatGPT plus plugins”, and “GPT-4 in ARC’s testing environment where it can

‘execute code, do chain-of-thought reasoning, and delegate to copies of itself’

”.

Conjecture C: LLMs don’t behave like expected utility maximizers for tasks they are given.

Caveats/clarifications: LLMs are obviously choosing next tokens based on minimizing a function (for pre-trained models the function is an approximation of “predictive loss on the next token”), but this doesn’t map clearly a utility calculation for a task it is given (e.g. “make paperclips”). That said, LLMs might be capable of role-playing an expected utility maximizer.

I think I’m reasonably confident in these conjectures (90+% belief under appropriate formalization). I see conjecture A as very different than conjectures B and C, and I think B and C have important safety implications that I’ll discuss in a later post. But I am a bit concerned that the properties I am ascribing to LLMs is mostly true of the pre-trained models, and might be rendered false by relatively small amounts of fine-tuning (via RLHF or other means).

The rest of this post will be outlining what evidence I’ve seen for these conjectures.

Conjecture A: AGI from LLMs

Reasons I think LLMs will be central to AGIs:

LLMs have state-of-the-art and near-human capabilities in a huge range of tasks.
LLMs have room for improvement. So far it’s looked like

bigger=smarter

for LLM

...