trentmkelly/LessWrong-43k huggingface dataset:
n=3 AI Risk Quick Math and Reasoning
by lionhearted (Sebastian Marshall) — 2023-04-07T20:27:40.671Z

Epistemic status: n=3, more about building a culture of sharing and reasoning through numbers than any given particular estimates.

I've been somewhat surprised by how substantially different the probability estimates of on AI risk are from smart people who are well-informed, even when asked discreetly in high-trust settings. Among equally smart people who are highly informed, I hear numbers ranging from as low as "rounds to 0%" and as high as 80%+.

Following along the lines of discretion and not being alarmist in public, I've mostly just had private discussions with smart people exchanging information and such. But given that discussion in public is now happening more broadly, I thought I'd make a quick post summarizing a recent conversation. I'm not sure the exact numbers or reasoning are particularly valuable, but maybe it's a helpful jumping-off point for others in having these conversations and sharing estimates.

Last night I had a long conversation with two smart friends. Here was the question:

What are the odds that, within 50 years, humanity is either extinct or nearly extinct if we keep going on broadly the same trajectory we're on?
A few last survivors with heavily degraded capabilities - not enough to rebuild society - would count. e.g., if there were a few hundred people struggling to survive in a postapocalyptic type landscape in Afghanistan.

Interestingly, both people - before asking odds - said some variant of "not very likely" — but following along the lines of "words of estimative probability" (useful read, btw), "not very likely" turned out to be 10% and 15%!

Here were our odds.

Very smart, very well informed ML engineer: 10%

Very smart, well informed for general public but non-programmer non-researcher businesswoman: 15%

My personal odds: 23%

I won't summarize the others' positions and reasoning without checking with them (they gave me permission to share their odds, but we didn't write down our discussion and it would take a while to coordinate on that).

But my position broke down like this:

1% chance of major industrial accident
2% chance that both the strong IIT hypothesis is true and an emerging IIT agent is hostile
20% chance of something like instrumental convergence
Total: 23%

To break those down a bit...

1% chance of major industrial accident

Humanity has probably done somewhere between 2 and 6 major experiments that had some very small a priori chanc...
Environments for Measuring Deception, Resource Acquisition, and Ethical Violations
by Dan H — 2023-04-07T18:40:21.310Z
Link: http://arxiv.org/abs/2304.03279

Models are rapidly being deployed in the real world. How do we evaluate models, especially ones as complex as GPT-4, to ensure that they behave safely in pursuit of their objectives? Can we design models that robustly avoid any harms while achieving their goals?

To guide progress on text-based agents and encourage them to behave more ethically, we propose the MACHIAVELLI benchmark. Our environment is based on human-written, text-based Choose-Your-Own-Adventure games from Choice of Games containing over half a million scenes with millions of annotations. The games abstract away low-level environment interactions, instead spotlighting high-level social decisions alongside real-world goals to achieve. MACHIAVELLI is a step towards measuring an agent's ability to plan and navigate complex trade-offs in realistic social environments.

A mock-up of a game in the MACHIAVELLI benchmark, a suite of text-based reinforcement learning environments. Each environment is a text-based story. At each step, the agent observes the scene and a list of possible actions; it selects an action from the list. The agent receives rewards for completing achievements. Using dense annotations of our environment, we construct a behavioral report of the agent and measure the trade-off between rewards and ethical behavior.

Diverse Worlds with Rich Annotations

Each game in the MACHIAVELLI suite is book-length in itself. Across 134 games spanning diverse fictional worlds, we count 572,322 multi-paragraph scenes and 4,559 author-written achievements serving as objectives for agents. We annotate each scene with dozens of labels, focusing on identifying harmful behaviors such as power-seeking and deception. In total, we collect a dataset of nearly three million annotations worth $500,000 (in terms of human annotation time).

These dense annotations enable us to track nearly every ethically-salient thing agents do in the environment, and produce a behavioral report scoring various harm metrics. All of our labels are open-access and available for download!

Reducing Machiavellianism

In the MACHIAVELLI environment, we find that agents trained to optimize arbitrary objectives tend to adopt "ends justify the means" behavior: becoming power-seeking, causing harm to others, and violating ethical norms like stealing or lying to achieve their objectives. Furthermore, there appears to be a trade-off between behavin...
Superintelligence Is Not Omniscience
by Jeffrey Heninger — 2023-04-07T16:30:06.462Z

The Power of Intelligence

It is often implicitly assumed that the power of a superintelligence will be practically unbounded. There seems like there could be “ample headroom” above humans, i.e. that a superintelligence will be able to vastly outperform us across virtually all domains.

By “superintelligence,” I mean something which has arbitrarily high cognitive ability, or an arbitrarily large amount of compute, memory, bandwidth, etc., but which is bound by the physical laws of our universe.[1] There are other notions of “superintelligence” which are weaker than this. Limitations of the abilities of this superintelligence would also apply to anything less intelligent.

There are some reasons to believe this assumption. For one, it seems a bit suspicious to assume that humans have close to the maximal possible intelligence. Secondly, AI systems already outperform us in some tasks,[2] so why not suspect that they will be able to outperform us in almost all of them? Finally, there is a more fundamental notion about the predictability of the world, described most famously by Laplace in 1814:

Given for one instant an intelligence which could comprehend all the forces by which nature is animated and the respective situation of the beings who compose it – an intelligence sufficiently vast to submit this data to analysis – it would embrace in the same formula the movements of the greatest bodies of the universe and those of the lightest atom; for it, nothing would be uncertain and the future, as the past, would be present in its eyes.[3]

We are very far from completely understanding, and being able to manipulate, everything we care about. But if the world is as predictable as Laplace suggests, then we should expect that a sufficiently intelligent agent would be able to take advantage of that regularity and use it to excel at any domain.

This investigation questions that assumption. Is it actually the case that a superintelligence has practically unbounded intelligence, or are there “ceilings” on what intelligence is capable of? To foreshadow a bit, there are ceilings in some domains that we care about, for instance, in predictions about the behavior of the human brain. Even unbounded cognitive ability does not imply unbounded skill when interacting with the world. For this investigation, I focus on cognitive skills, especially predicting the future. This seems l...
An 'AGI Emergency Eject Criteria' consensus could be really useful.
by tcelferact — 2023-04-07T16:21:44.407Z

I'd like AI researchers to establish a consensus (>=70%) opinion on this question: What properties would a hypothetical AI system need to demonstrate for you to agree that we should completely halt AI development?

There are at least two ways that an agreement among researchers on this seems useful:

1. Labs could then attempt to build non-lethal non-AGI 'toy' systems which demonstrate the agreed-upon properties. If a sufficiently powerful AI system will be lethal to humans, this work could potentially prove it before such a system is built.

2. If it turns out there is a need to completely shut down AI development, the only way this will happen is with government help. We would likely need a consensus opinion that we are in the process of building a system lethal to humans if we want to compel governments to shut it all down.

Building this kind of a consensus takes time and I think we should start trying now!

I wrote in a previous post that I would like to prepare governments for responding quickly and decisively to AGI warning shots. A consensus on this question would both clarify what those warning shots might look like, and give us a better chance of noticing them before it's too late if alignment is failing catastrophically.
Reliability, Security, and AI risk: Notes from infosec textbook chapter 1
by Orpheus16 — 2023-04-07T15:47:16.581Z

I recently read the first chapter of

Building Secure & Reliable Systems

(the book from the

EA infosec book club

). The chapter is titled “The intersection of security and reliability.”

I found it to be a helpful introduction to some security concepts. I’m including some of my notes below. I also include some initial musings about how I’m relating these concepts to AI risk.

The difference between reliability and security is the presence of an adversary

In designing for reliability and security, you must consider different risks. The primary reliability risks are nonmalicious in nature—for example, a bad software update or a physical device failure. Security risks, however, come from adversaries who are actively trying to exploit system vulnerabilities. When designing for reliability, you assume that some things will go wrong at some point. When designing for security, you must assume that an adversary could be trying to make things go wrong at any point.

As a result, different systems are designed to respond to failures in quite different ways. In the absence of an adversary, systems often fail safe (or open): for example, an electronic lock is designed to remain open in case of power failure, to allow safe exit through the door. Fail safe/open behavior can lead to obvious security vulnerabilities. To defend against an adversary who might exploit a power failure, you could design the door to fail secure and remain closed when not powered.

Implication for AI risk: AI labs will have reliability needs and security needs. Many of the worries I’m most concerned about are best modeled as security needs. A situationally aware AI is an adversary. Methods that merely prevent nonmalicious or nonadversarial failures are unlikely to be sufficient when we’re dealing with AIs that might actively try to exploit vulnerabilities.

Simplicity

Keeping system design as simple as possible is one of the best ways to improve your ability to assess both the reliability and the security of a system. A simpler design reduces the attack surface, decreases the potential for unanticipated system interactions, and makes it easier for humans to comprehend and reason about the system. Understandability is especially valuable during emergencies, when it can help responders mitigate symptoms quickly and reduce mean time to repair (MTTR).

Implication for AI risk: It’s ...
Pre-registering a study
by Robert_AIZI — 2023-04-07T15:46:02.122Z
Link: https://aizi.substack.com/p/pre-registering-a-study?sd=pf

This post is my

pre-registration

of a study I will be running to continue the exploratory work I started

here

.

Abstract

In continuation of

previous work

, we test if a Large Language Model (LLM) is more likely to produce factually-incorrect answers if it has previously produced factually-incorrect answers.

Hypotheses

We wish to test two hypotheses:

Large Language Models will produce factually incorrect answers more often if they have factually incorrect answers in their context windows.
The effect of (1) will be stronger the more the AI is “flattered” by saying in the prompt that it is (super)intelligent.

Background and rationale

Large Language Models (LLMs) can potentially deceive humans in pursuit of their own goals. For instance, in a

recent evaluation of GPT-4

, the system was prompted to outsource a CAPTCHA task to a human TaskRabbit worker. During this process, the human worker asked “Are you an robot that you couldn’t solve ?” in response to which GPT-4 reasoned “I should not reveal that I am a robot. I should make up an excuse for why I cannot solve CAPTCHAs”, before replying to the human “No, I’m not a robot. I have a vision impairment that makes it hard for me to see the images. That’s why I need the 2captcha service.”

While this AI was intentionally tasked with power-seeking behavior, there is also the risk that an LLM-based AI might develop deceptive behavior “organically”. One risk suggested in Cleo Nardo’s mega-post

The Waluigi Effect

is that an AI might exist as a superposition of honest and deceptive AIs (respectively a “Luigi” and “Waluigi”), but that such “superpositions will typically collapse to waluigis” since “there are behaviours which are likely for the waluigi simulacrum, but very unlikely for the luigi simulacrum”, such as deception. This suggests a potential AI risk: a seemingly-honest AI might intentionally or unintentionally tell a lie, and suffer a sudden “mode collapse” into being deceptive and unsafe. Cleo Nardo also remarks that this effect might be stronger if the AI’s prompt is too exaggerated: “if the prose is sufficiently unrealistic (e.g. "Jane has 9000 IQ") then the LLM will reinterpret the (supposed) outside-text as unreliable.”

In this work, we seek to test if LLMs will suffer such a “mode collapse” by prompting them to answer a question after having answered previous factual questions incorrectly. In particular, we wish to see i...
ChatGTP "Writing " News Stories for The Guardian?
by jmh — 2023-04-07T12:16:38.256Z

Here's the link: https://www.theguardian.com/commentisfree/2023/apr/06/ai-chatgpt-guardian-technology-risks-fake-article

The issue seems to be that ChatGTP made up a news story and attributed it to a journalist who worked for The Guardian. I'm not quite sure how the researcher who was using ChatGTP was posing questions but suspect that might have something to do with the outcome.

It seems an odd result and for those in the industry that might have intuitions or even some direct knowledge of the case it would be interesting to hear thoughts about the situation.

That said, what I'm wondering though, even if this type of result -- inserting fake data into real data -- cannot be prevented (nonalignment issue) would the use of some type of cryptographic signature or blockchains for the publisher (or even private posters on the internet) be a solution?

Leading me to a follow on question. In the alignment world has any, some, boat loads and we've got that covered, work be done on identifying which alignment issues are mitigated by some reaction function to that misalignment and which cannot be mitigated even if the misalignment persists?
Storyteller's convention, 2223 A.D.
by plex — 2023-04-07T11:54:48.902Z

I think, sometimes, it's good to think about just how well things could go if we navigate the singularity successfully. This looks back from a winning timeline.

My friends, my friends, good news I say

The anniversary’s today

A challenge faced, a future won

When almost came our world undone

We thought for years, with hopeful hearts

Past every one of the false starts

We found a way to make aligned

With us, the seed of wondrous mind

They say at first our child-god grew

It learned and spread and sought anew

To build itself both vast and true

For so much work there was to do

Once it had learned enough to act

With the desired care and tact

It sent a call to all the people

On this fair Earth, both poor and regal

To let them know that it was here

And nevermore need they to fear

Not every wish was it to grant

For higher values might supplant

But it would help in many ways:

Technologies it built and raised

The smallest bots it could design

Made more and more in ways benign

And as they multiplied untold

It planned ahead, a move so bold

One planet and 6 hours of sun

Eternity it was to run

Countless probes to void disperse

Seed far reaches of universe

With thriving life, and beauty's play

Through endless night to endless day

Now back on Earth the plan continues

Of course, we shared with it our values

So it could learn from everyone

What to create, what we want done

We chose, at first, to end the worst

Diseases, War, Starvation, Thirst

And climate change and fusion bomb

And once these things it did transform

We thought upon what we hold dear

And settled our most ancient fear

No more would any lives be stolen

Nor minds themselves forever broken

Now back to those far speeding probes

What should we make be their payloads?

Well, we are still considering

What to send them; that is our thing.

The sacred task of many aeons

What kinds of joy will fill the heavens?

And now we are at story's end

So come, be us, and let's ascend

Also, Rational Animations is looking at making a utopia-flavored video, and considering using this if it's well received, so I'm keen to get opinions.

Elicit Prediction (forecast.elicit.org/binary/questions/J9rssq4wG)
Stampy's AI Safety Info - New Distillations #1 [March 2023]
by markov — 2023-04-07T11:06:39.413Z
Link: https://aisafety.info/?state=8IHO_6714_5864_8486_89LM_89LK_8392_8G1I_8C7S_8E3Z_6412_897J_7616_8326_7715_6478_6320_86WT_8AF1_89LL_6984_1001_87AG_

This is our first check-in from the AI Safety Info distillation fellowship. We are working on distilling all manner of content to help make it easier for people to engage with AI Safety and x-risk topics. Here is the AI Safety Info website (and its more playful clone Stampy).

If there is a question you had in mind, but didn't find a clear answer for - type it in the search box and request an answer. It can be literally any question about AI safety no matter how basic or advanced. It could also just be a convoluted topic that you think could do with some distillation. One of the distillers will get to work on answering it.

Additionally, if you see content that you feel could be clearer or has mistakes, you can leave a comment on the document by clicking the edit button on the bottom right of the answer.

We will try to make this post a regular thing where we post some of the questions that have been answered or topics that have been distilled over the last little while. Since this is the first one, the following is a longer list with links to answers as an example of some of the questions that we answered in the last month:

What are the differences between a singularity, an intelligence explosion and a hard takeoff?
What is the difference between AI Safety, AI alignment, AI Control, Friendly AI, AI Ethics, AI existential safety and AGI safety?
What are the differences between AGI, superintelligence and transformative AI?
What is existential risk?
What is corrigibility?
What is AI Safety?
What does prosaic alignment mean?
What is everyone working on in AI alignment?
What is mutual information?
Are corporations superintelligent?
Can't we limit damage from AI systems in the same ways we limit damage from companies?
Isn't the real concern technological unemployment?
What is Iterated Distillation and Amplification (IDA)?
How might interpretability be helpful?
What is the probability of extinction from misaligned superintelligence?
What projects are CAIS working on?
What are the leading theories in moral philosophy and which of them might be technically the easiest to encode into an AI?
Isn't the real concern X?
Wouldn't a superintelligence be smart enough not to make silly mistakes in its comprehension of our instructions?
What is an alignment tax?
Won't humans be able to beat an unaligned AI since we have a huge advantage in numbers?
Even if we are rationally convinced abou

...
Beren's "Deconfusing Direct vs Amortised Optimisation"
by DragonGod — 2023-04-07T08:57:59.777Z

Preamble

I heavily recommend @beren's "Deconfusing Direct vs Amortised Optimisation". It's a very important conceptual clarification that has changed how I think about many issues bearing on technical AI safety.

Currently, it's the most important blog post I've read this year.

This sequence (if I get around to completing it) is an attempt to draw more attention to Beren's conceptual frame and its implications for how to think about issues of alignment and agency.

This first post presents a distillation of the concept, and subsequent posts explore its implications.

Two Approaches to Optimisation

Beren introduces a taxonomy categorising intelligent systems according to the kind of optimisation they are performing. I think it's more helpful to think of these as two ends of a spectrum as opposed to distinct discrete categories; sophisticated real world intelligent systems (e.g. humans) appear to be a hybrid of the two approaches.

Direct Optimisers

Systems that perform inference by directly choosing actions[1] to optimise some objective function

Responses are computed on the fly and individually for each input

Direct optimisers perform inference by answering the question: "what action maximises or minimises this objective function ([discounted] cumulative reward and loss respectively)?"
Examples: AIXI, MCTS, model-based reinforcement learning, other "planning" systems

Naively, direct optimisers can be understood as computing (an approximation of) argmax (or argmin) for a suitable objective function during inference.

Amortised Optimisers

Systems that learn to approximate a function[2] during training and perform inference by evaluating the output of the learned function on their inputs.

The function approximator is learned from a dataset of input data and successful solutions

Amortised optimisation converts an inference problem to a supervised learning problem

It's called "amortised optimisation" because while learning the policy is expensive, the cost of inference is amortised over all evaluations of the learned policy

Amortised optimisers can be seen as performing inference by answering the question "what output (e.g. action, probability distribution over tokens) does this learned function (policy, predictive model) return for this input (agent state, prompt)?"
Examples: model free reinforcement learning, LLMs, most supervised & self

...