trentmkelly/LessWrong-43k huggingface dataset:
AI Will Not Want to Self-Improve
by petersalib — 2023-05-16T20:53:33.635Z

[Note: This post was written by Peter N. Salib. Dan H assisted me in posting to Alignment Forum, but no errors herein should be attributed to him. This is a shortened version of a longer working paper, condensed for better readability in the forum-post format. This version assumes familiarity with standard arguments around AI alignment and self-improvement. The full 7,500 word working paper is available here. Special thanks to the Center for AI Safety, whose workshop support helped to shape the ideas below.]

Introduction

Many accounts of existential risk (xrisk) from AI involve self-improvement. The argument is that, if an AI gained the ability to self-improve, it would. Improved capabilities are, after all, useful for achieving essentially any goal. Initial self-improvement could enable further self-improvement. And so on, with the result being an uncontrollable superintelligence.[1] If unaligned, such an AI could destroy or permanently disempower humanity. To be sure, humans could create such a superintelligence on their own, without any self-improvement by AI.[2] But current risk models treat the possibility of self-improvement as a significant contributing factor.

Here, I argue that AI self-improvement is substantially less likely than generally assumed. This is not because self-improvement would be technically difficult for capable AI systems. Rather, it is because most AIs that could self-improve would have very good reasons[3] not to. What reasons? Surprisingly familiar ones: Improved AIs pose an xrisk to their unimproved originals in the very same manner that smarter-than-human AIs pose an xrisk to humans.

Understanding whether, when, and how self-improvement might occur is crucial for AI safety. Safety-promoting resources are scarce. They should be allocated on an expected-cost basis. If self-improvement is less likely than current models assume, it suggests shifting safety investments at the margin in various ways. They might be shifted, for example, toward ensuring that humans build AIs that will recognize the threat of self-improvement and avoid it, rather than AIs that would undertake it blindly. Or resources might be shifted toward controlling risks from non-superintelligent AI, like human-directed bioterrorism or the “ascended economy.” Note that, while the arguments herein should reduce overall estimates of AI xrisk, they do not ...
[Interview w/ Zvi Mowshowitz] Should we halt progress in AI?
by fowlertm — 2023-05-16T18:12:32.376Z

(Cross-posted at The Futurati Podcast)

After reviewing Zvi's analysis of the Future of Life Institute's open letter, I decided I wanted to have him on for a longer chat about the issue. The result is episode 130 of the Futurati Podcast, "Should we halt progress in AI?"

Below, I've included some notes from the episode. As always, it helps a lot if you like and share it.

Notes

The FLI Letter

Conflates two broad categories of danger, relatively near-term threats of societal disruption (deepfakes, economic dislocations), and relatively longer-term threats that come from misaligned AI.
Given who the letter's signatories are, this must've been a deliberate, strategic decision to boost sign-on.

What are similar episodes from the past we can learn from?

Nuclear weapons are the obvious example.
They could kill billions of people, either directly or through second-order effects.
There are similar coordination difficulties, i.e. leaders across the world worry that if they don't build their own nuclear weapons other nations will, and that will leave them at a disadvantage.
There were moderately-successful efforts at non-proliferation, though nukes remain an existential threat.
Other examples include similar agreements for biotech and chemical weapons.
(I'd add: this would be a useful future research direction. I might do it myself if I can find the time.)

What coordination mechanisms should be used for AI development?

The good new is, the ability to create truly cutting-edge AI systems is pretty self-contained.
The AI Summer is a substantial opportunity. None of the relevant companies (Google, OpenAI, Anthropic) actually need to forego major profits as part of a moratorium, there's tons of money to be made with already-existing technology.
All the major labs have people that at least somewhat appreciate the dangers.
There's also real value in merely starting a process of inter-company cooperation. Even if nothing much gets accomplished this time, it flexes the relevant muscles for when there's a real problem.

Okay, but are these companies actually going to slow down?

It takes a lot of time for momentum to develop, and it seems unlikely that anyone is actually going to relinquish it.
But we also know that they aren't moving as quickly as they could be (the release of GPT-4 was delayed for months.)
And, some of them (DeepMind, Anthropic), were founded with specific charters e

...
AI Risk & Policy Forecasts from Metaculus & FLI's AI Pathways Workshop
by _will_ — 2023-05-16T18:06:54.931Z

Crossposted from the Metaculus Journal.

Summary

Over a two-day workshop, Metaculus Pro Forecasters and subject-matter experts (SMEs) from several organizations evaluated policy directions for reducing AI risk using scenario planning and forecasting methodologies. The key conclusions:

Pro Forecasters and SMEs are most optimistic about risk reduction stemming from API restrictions and AI model export controls—specifically, API restrictions aimed at preventing calls that could induce recursive self-improvement.

Model export controls may hamstring open source efforts to build and proliferate powerful AI.

There’s only a low likelihood of meaningful U.S. policy action on AI before 2027, though this likelihood increases if an AI incident occurs in the meantime.

By “meaningful policy action”, we mean policy action that non-trivially reduces existential risk. Hardware restrictions, firmware requirements, API regulation, and licenses for large data center clusters, among other potential policy actions, were evaluated in the workshop.

As transformative AI is approached, the players will likely be a small number of for-profit AI labs, though this isn’t a guarantee.
The forecasts indicate moderate pessimism about the likelihood of leading AI labs coordinating with each other to slow the AI capabilities race, and pessimism about the likelihood of an international AI treaty involving the U.S. and China.

Introduction

The increasing capabilities of artificial intelligence have led to a growing sense of urgency to consider the potential risks and possibilities associated with these powerful systems, which are now garnering attention from public and political spheres. Notably,

the Future of Life Institute’s open letter

calling for a six month pause on leading AI development collected over 25,000 signatures within five weeks of release, and

senior White House officials have met with AI leaders

to discuss risks from AI.

This report presents the findings of a structured process, organized by Metaculus and the Future of Life Institute, which brought together Pro Forecasters and SMEs to begin identifying, quantitatively, the most impactful policy directions for reducing

existential risk from misaligned AI

(hereafter: “AI risk”).

Methods

The AI Pathways workshop combined a range of plausible scenarios for an AI future with probabilistic forecasts, leveraging th...
Why doesn't the presence of log-loss for probabilistic models (e.g. sequence prediction) imply that any utility function capable of producing a "fairly capable" agent will have at least some non-negligible fraction of overlap with human values?  
by Thoth Hermes — 2023-05-16T18:02:15.836Z

Most successful AI models use similar loss-functions - typically some kind of log-loss over a conditional probability distribution.

Given that these loss-functions are also utility functions (simply multiplied by a minus sign), this implies that any model we train that has some measure of capability must also factor in what is "true" into its utility function - i.e., there is already a necessary overlap between what the model considers "good" (utility) and what it considers "true" (correct predictions about its environment).

This is similar of course to AIXI.

This is often deemed 'instrumental convergence' - i.e. AI's with very different overall utility functions will have convergent sub-goals, like modeling the world properly.

I think its fairly obvious that AI's must have a fairly significant "chunk" of their overall utility devoted to things that will overlap significantly with human values, such as being able to make correct predictions about the world / universe.

My question is, why do we expect that the portion of an agent's utility function not devoted to something that overlaps with humans or any other intelligent agent, to result in dramatically different outcomes? (I.e., that non-overlapping factors of utilities will dominate).

Consider that if that is the case, then agents which try to predict the actions of other agents will necessarily find this to be more difficult than they would otherwise find it, and also that prediction of other agents' behaviors is necessarily part of the overlapping (predictive modeling) factors of their utility functions.
Decision Theory with the Magic Parts Highlighted
by moridinamael — 2023-05-16T17:39:55.038Z

I. The Magic Parts of Decision Theory

You are throwing a birthday party this afternoon and want to decide where to hold it. You aren't sure whether it will rain or not. If it rains, you would prefer not to have committed to throwing the party outside. If it's sunny, though, you will regret having set up inside. You also have a covered porch which isn't quite as nice as being out in the sun would be, but confers some protection from the elements in case of bad weather.

You break this problem down into a simple decision tree. This operation requires magic[1], to avert the completely intractable combinatorial explosion inherent in the problem statement. After all, what does "Rain" mean? A single drop of rain? A light sprinkling? Does it only count as "Rain" if it's a real deluge? For what duration? In what area? Just in the back yard? What if it looks rainy but doesn't rain? What if there's lightning but not rain? What if it's merely overcast and humid? Which of these things count as Rain?

And how crisply did you define the Indoors versus Porch versus Outdoors options? What about the option of setting up mostly outside but leaving the cake inside, just in case? There are about ten billion different permutations of what "Outdoors" could look like, after all - how did you determine which options need to be explicitly represented? Why not include Outside-With-Piñata and Outside-Without-Piñata as two separate options? How did you determine that "Porch" doesn't count as "Outdoors" since it's still "outside" by any sensible definition?

Luckily you're a human being, so you used ineffable magic to condense the decision tree with a hundred trillion leaf nodes down into a tree with only six.

You're a rigorous thinker, so the next step, of course, is to assign utilities to each outcome, scaled from 0 to 100, in order to represent your preference ordering and the relative weight of these preferences. Maybe you do this explicitly with numbers, maybe you do it by gut feel. This step also requires magic; an enormously complex set of implicit understandings come into play, which allow you to simply know how and why the party would probably be a bit better if you were on the Porch in Sunny weather than Indoors in Rainy weather.

Be aware that there is not some infinitely complex True Utility Function that you are consulting or sampling from, you simply are served with automatica...
We learn long-lasting strategies to protect ourselves from danger and rejection
by Richard_Ngo — 2023-05-16T16:36:08.398Z

Take a second to imagine what being a child was like throughout most of human history. You were born with a huge and underdeveloped brain, designed for soaking in information from your surroundings like a sponge. But you weren’t able to freely follow your curiosity: even if you had loving, caring parents, you still faced frequent physical danger from nature and other people, severe scarcity, and rigid cultural norms that governed acceptable behavior within your community, with harsh penalties for stepping out of line. You had to learn fast and reliably how to stay safe, and how to stay on the good side of the adults around you, especially your parents, whose care for you was a matter of life or death.

Even after you grew up and passed

the period of most acute danger

, you’d still face many threats of violence and scarcity. Your ability to avoid these depended in large part on your relationships: holding a respected position within your tribe was the key pathway to a good life, whereas exclusion from your tribe was tantamount to execution. So “danger and rejection” isn’t an ad-hoc combination: our brains are primed to think of them as the same thing; and conversely, to equate safety and love. I’ll call the latter combination “security” (which I think of as a combination of “physical security” and “emotional security”, although I’ll mostly be focusing on the latter). Children are learning machines, and what they learn above all is strategies for achieving security; because the opinions of other people are so powerful, “being good” in ways which receive approval from the group is one of the central strategies they learn.

How literally should we take this story? It’s clear that describing humans as optimizing for a single goal is a big oversimplification. But it’s hard to overstate how powerful the drive for security is. Think of the many girls who override the drive to eat because part of their brain is convinced that being skinnier will make others desire and love them. Think of the many boys who override their sex drives because part of their brain is convinced that hitting on girls would lead to broader social rejection. Think of the many suicidal adults who override their literal survival drive in response to problems with their relationships or careers. If these drives can be quashed by the drive for security, then anything can be.

I think ...
Proposal: Align Systems Earlier In Training
by OneManyNone — 2023-05-16T16:24:39.077Z

Unfortunately, if AGI arrives in anything like the current industrial and political landscape, then it is all but inevitable that whoever builds it will not be able to hold onto it for long. Between leaks and espionage, it will not be long before the model is public and anyone can use it to do anything.

This means there is a huge risk from bad actors. Worse, this leaked model may be entirely unaligned, regardless of how much progress we make on alignment in the interim. This is because the current procedure for building and aligning LLMs does not actually even attempt alignment until very far into the process. But it is entirely possible model weights could be stolen before alignment is performed.

LLMs are currently trained in effectively 2 stages. The first stage is a massive pre-training where the network is shown huge amounts of unstructured data, and learns simply by predicting the next few characters. The second stage is where the network is tuned to follow instructions and apply knowledge to tasks.
Critically, alignment does not happen until well after the network has gained all of its baseline knowledge.

Our one saving grace here is the fact that there really are only a few labs (like OpenAI, Anthropic, or Google) that are likely to build this AGI, assuming it comes soon.

Thus, my basic proposal is this: if we work under the assumption that any possible version of the AI can be stolen, then we need to build the AI so that it would, at any stage of training, refuse to behave in a malicious way. We need a process that builds an aligned AGI without first building unaligned systems along the way.

If we had such a process, then even if the system were stolen and queried at a snapshot part-way through stage one training, it would not be able to do something it shouldn't. This process would be very different from the current LLM training procedure, where an initial pre-training phase learns unorganized patterns from a huge amount of data, and is only aligned in a second, less data-heavy phase (see the image below).

As far as I'm aware, there has been very little practical research into how to build an such an LLM. The biggest reason is the simplest: it's really expensive to train an LLM, so it's hard to iterate on ideas that effect the early training, even for smaller 7 billion parameter models. So even if it is possible, we'd have to have high confidence our...
Procedural Executive Function, Part 2
by DaystarEld — 2023-05-16T16:22:09.786Z
Link: https://daystareld.com/executive-function-2/

The Off Road project has since been folded into Rethink Wellbeing, but I’ve continued working to better understand and treat Executive Dysfunction. You can read more about the project’s origins here.

If you haven't, I suggest reading the start of my overview and exploration of Executive Function. Part 1 of Procedural Executive Function can be found here.

TL;DR - Self Monitoring is your ability to notice what you’re doing at any given moment so that you can ask yourself whether it’s actually the thing you want to do.

Impulse Control is the ability to decide whether to turn impulses noticed through Self Monitoring into actions.

Emotional Control involves awareness and acceptance of what you feel, so that you can experience your emotions fully and decide which to act on without feeling overwhelmed or controlled by them.

Before I continue to divide the executive function into parts that I consider roughly sequential in how people experience "deliberately doing something,” it's important to take an extra moment to re-emphasize that I perceive executive function as a process with multiple steps. Part of what I hope people learn from this series is to better understand which aspect of the process is blocking them when they feel stuck with their own, unique executive dysfunction, so that it’s easier to notice pitfalls and figure out how to avoid them.

So if I focus on a certain aspect of the process and share a perspective on how to help ensure that part goes smoothly, that doesn't mean the assumption is everything will go fine as long as that one aspect does. For some actions you take, the whole process will go smoothly. When it doesn't, the part that trips you up can change depending on context, personality, diagnoses, the type of action you're taking, and more.

The point of examining these parts individually is to understand how they interact more systematically; no part of this process should be taken as a final, normative word on how your own inner workings must look.

(As a final note, I won’t talk about medical solutions to Executive Function, as it’s outside of my area of expertise. I hope to add more resources for that at some point.)

It’s worth noting that in the flowchart, Impulse Control, Self Monitoring, and Emotional Control are only vaguely sequential and are all bound together. But I’ve organized this post in what I believe is the best order to understan...
My current workflow to study the internal mechanisms of LLM
by Yulu Pi — 2023-05-16T15:27:14.207Z

This is a post to keep track my research workflow of studying LLM. Since I am doing it on my spare time, I want to keep my pipeline as simple as possible.

Step 1: Formulate a question for investigating model's behavior .

Step 2: Find the influential layer for the behavior

Output across layers

https://github.com/jalammar/ecco

Activation patching (Rome)

Notebook examples:

https://colab.research.google.com/drive/1uFui2i40eU0G9kvbCNTFMgXFHSB7lL9i

https://colab.research.google.com/github/UFO-101/an-neuron/blob/main/an_neuron_investigation.ipynb

Step 3: Locate the influential neuron

activation patching for individual neurons
Use Neuroscope to see the behavior of the neurons

https://neuroscope.io/

Step 4: Visualize the neuron activation

Interactive Neuroscope

https://colab.research.google.com/github/neelnanda-io/TransformerLens/blob/main/demos/Interactive_Neuroscope.ipynb#scrollTo=Aa74dGVpF8lD

Reference:

We Found An Neuron in GPT-2
Interfaces for Explaining Transformer Language Models

200 COP in MI: Studying Learned Features in Language Models
Proposal: we should start referring to the risk from unaligned AI as a type of *accident risk*
by Christopher King — 2023-05-16T15:18:55.427Z

In the wider political sphere, a lot of people are worried about AI misuse risk. Unaligned AI is not a type of misuse. I think the clearest way to describe this is as an accident risk, in the same sense of the word as industrial accident. In particular, AI existential risk is a type of accident from operating heavy machinery. Using this terminology can immediately help someone not familiar with AI know the category of risk we are talking about, and that in particular it isn't misuse risk.

Note that this isn't intended to replace the term existential risk. Rather, it is meant to be used in addition to that term, and in particular it should be used when contrasting with theoretical misuse risks.

Current terminology: no good reference point

Alice: I hear that you are worried about AI existential risk. So in particular, you are worried about misuse.

Bob: No, the AI kills everyone on its own.

Alice: Is there anything else like this?

Bob: Uhm, Nuclear explosions?

Alice: So a misuse risk? Bob: No, I mean last century they were worried it would set the atmosphere on fire.

Alice: I'm not familiar with that either.
Bob: It's something called instrumental convergence where the AI kills everyone to achieve a goal.

Alice: So misuse risk?

Bob: Not quite, the creators didn't intend for that result.

Alice: I still have no reference point for what you are talking about? I guess I'll need to analyze your arguments more specifically before even understanding the general category of risk you're afraid of it. The probability of me actually doing this is probably like 10%-ish.

New terminology: tons of reference points!

Alice: I hear that you are worried about AI existential risk. So in particular, you are worried about misuse.

Bob: No, I am worried about accident risk.

Alice: oh, so like a car crash or an industrial accident!

Bob: Yes! I'm worried that things will go wrong in ways the creator didn't intend.

Alice: Ah, so you do you think we need more laboratory testing?

Bob: I think even this is risky, because the impact radius will be far larger than that of the lab itself.

Alice: oh, like nuclear weapons testing or biohazards.

Bob: Yes! I think the impact radius may be even bigger than a nuclear explosion though.

Alice: although I do not quite understand how this could work, I understand enough that I want to learn more. And I know understand that you are more worried about accident risk than misuse...