trentmkelly/LessWrong-43k huggingface dataset:
Believable near-term AI disaster
by Dagon — 2022-04-07T18:20:16.843Z

johnswentworth's post about AGI vs Humanity (https://www.lesswrong.com/posts/KTbGuLTnycA6wKBza/) caught my attention in ways that most discussion of AI takeover and the prevention of such does not. Most of the discussion is very abstract, and leaves unstated some pretty deep assumptions about the path and results of smarter-than-human imperfectly-aligned AI. This preserves generality, but interferes with engagement (for me, at least). I think the fundamental truth remains that we can't actually predict the path to unaligned AI, nor the utility function(s) of the AI(s) who take over, so please treat this as "one possible story", not the totality of bad things we should worry about.

40-odd years ago, Douglas Adams wrote:

There is a theory which states that if ever anyone discovers exactly what the Universe is for and why it is here, it will instantly disappear and be replaced by something even more bizarre and inexplicable. There is another theory which states that this has already happened.

I suspect it will be similar with AI takeover - by the time anyone discovers exactly what the AI wants and how it's going about achieving it, it will be replaced by something even weirder. More importantly, it will be discovered in retrospect, not prospect. Expect to read about (or be involved with publishing) articles that sound like "oh! THAT's how we lost control!" And counter-articles explaining that "we" never had control in the first place.

My story is that conspiracy theories about Illuminati, shadow-governments, and other incredibly powerful but non-obvious control structures for human civilization become true with the advancing capabilities of AI. We don't (and perhaps can't) fully know the AI's utility function, but we know that power and influence are instrumentally valuable for most goals.

This feels like AI-assisted cultural shift, and AI-influenced resource usage much more than it feels like "takeover". Unless the AI somehow inherits human egotism, it won't particularly care if humans declare fealty or act subservient - the AI isn't seeking office, only seeking to direct resources. The most important resource on earth today is human action. We are incredibly versatile and powerful machines, and the absolute cheapest way to manipulate physical resources. And we're easily motivated by thi...
List of concrete hypotheticals for AI takeover?
by Yitz — 2022-04-07T16:54:12.934Z

I was wondering if anyone’s compiled a list of posts which give a concrete description (with scenario-specific details) of a hypothetical future where humanity suffers an X or S scale disaster due to AGI takeover.
If such a list does not already exist (or does exist but needs to be updated), please put links to specific posts of this kind in the comments!
What if "friendly/unfriendly" GAI isn't a thing?
by homunq — 2022-04-07T16:54:05.735Z

A quick sketch of thoughts. Epistemic status: pure speculation.

Recent AI developments are impressive

Both text and images this week
If I don't hesitate to say my dog is thinking, can't really deny that these things are.

Still, there are two important ways these differ from human intelligence.

How they learn

Automatic-differentiation uses a retrospective god's-eye view that I think is probably importantly different from how humans learn. Certainly seems to require more substantially training data for equivalent performance.

What they want

They optimize for easily-measurable functions. In most cases this is something like "predicting this input" or "fool this GAN into thinking you're indistinguishable from this input"
This is subject to Goodhart's law.

Eventually, these differences will probably be overcome.

Learning and acting will become more integrated.
Motivation will become more realistic.

I suspect this will be through some GAN-like mechanism. That is: train a simpler "feelings" network to predict "good things are coming" based on inputs including some simplified view of the acting-network's internals. Then train the more-complex "actor"-network to optimize for making the "feelings" happy.

Note that the kind of solution I suggest above could lead to a "neurotic" AI

Very smart "actor" network, but "feeling" network not as smart and not pointing towards consistently optimizing over world-states.
"Actor" is optimizing for "feelings" now, and "feelings" is predicting world-state later, but even if "actor" is smart enough to understand that disconnect it could still have something like pseudo-akrasia (poor term?) about directly optimizing for world-state (or just feelings) later. A simple example: it could be smart enough to wire-head in a way that would increase its own "happiness", but still not do so because the process of establishing wire-heading would not itself be "fun".

What if that's a general truth? That is, any intelligence is either subject to Goodhart's Law in a way so intrinsic/fundamental that it undermines its learning/effectiveness in the real world, OR it's subject to "neuroses" in the sense that it's not consistently optimizing over world-states?

Thus, either it's not really GAI, or it's not really friendly/unfriendly.
GA(super)I would still be scarily powerful. But it wouldn't be an automatic "game over" for any less-intelligent optimizers around it. They m

...
Productive Mistakes, Not Perfect Answers
by adamShimi — 2022-04-07T16:41:50.290Z

This post is part of the work done at Conjecture.

I wouldn’t bet on any current alignment proposal. Yet I think that the field is making progress and abounds with interesting opportunities to do even more, giving us a shot. Isn’t there a contradiction?

No, because research progress so rarely looks like having a clearly correct insight that clarifies everything; instead it often looks like building on apparently unpromising ideas, or studying the structure of the problem.

Copernican heliocentrism

didn’t initially predict observations as well as

Ptolemaic astronomy

; both

ionic theory

and the determination of basic

molecular formula

came from combining multiple approaches in chemistry, each getting some bits but not capturing the whole picture;

Computer Science

emerged from the arid debate over the

foundations of mathematics

; and

Computational Complexity Theory

has made more progress by looking at

why some of its problems are hard

than by waiting for clean solutions.

In the end you do want to solve the problem, obviously. But the road from here to there goes through many seemingly weird and insufficient ideas that are corrected, adapted, refined, often discarded except for a small bit. Alignment is no different, including “strong” alignment.

Research advances through productive mistakes, not perfect answers.

I’m taking this terminology from Goro Shimura’s characterization of his friend Yutaka Taniyama, with whom he formulated the

Taniyama-Shimura Conjecture

that Andrew Wiles proved in order to prove Fermat’s last theorem.

(

Yutaka Taniyama and his time. Very personal recollections

, Goro Shimura, 1989)

Though he was by no means a sloppy type, he was gifted with the special capability of making many mistakes, mostly in the right direction. I envied him for this, and tried in vain to imitate him, but found it quite difficult to make good mistakes.

So much of scientific progress takes the form of many people proposing different ideas that end up being partially right, where we can look back later and be like “damn, that was capturing a chunk of the solution.” It’s very rare that people arrive at the solution of any big scientific problem in one nice sweep of a clearly adequate idea. Even when it looks like it (Einstein is an example people like to bring up), they so often build on many of the weird and contradictory takes that ca...
Covid 4/7/22: Opening Day
by Zvi — 2022-04-07T16:10:01.295Z

Baseball is back, and only a week late. As a fan with an infant, even though my Mets are somehow already without our top two starting pitchers, I am thrilled. Things are not fully open without opening day. Aside from putting masks on for the subway, life is now fully back to the way it used to be.

Someone told me they had Covid this morning, and I treated it exactly the way I would have if they’d said they had a cold. I checked to see if I’d been exposed (I hadn’t), I wished them a speedy recovery, but I in no way worried about the situation. People get sick sometimes, things go around. It’s life.

Europe’s situation is somewhat less comfortable, but everyone has rightfully decided to get on with life regardless. I am impressed. This new wave there will start declining soon enough, as there are not enough people left to infect for it to do otherwise.

Here cases are picking up a bit in some areas as BA.2’s dominance sets in, but still at low levels. I do not expect any serious trouble.

China is another story.

Executive Summary

BA.2 now dominant. Cases high in Europe, inching up in some places in USA.
China’s attempts at containment seem doomed to fail.
Congress agrees to let us spend $10bb on Covid treatments.
Deaths finally drop a lot in USA.

Let’s run the numbers.

The Numbers

Predictions

Prediction from last week: 210k cases (+11%) and 5,100 deaths (-10%).

Results: 178k cases (-1%) and 3,823 deaths (-34%) (!).

Prediction for next week: 190k cases (+6%) and 2,950 deaths (-23%).

Note: Florida reported 0 cases and 0 deaths this week, 0 cases two weeks ago and a lot of cases one week ago. For now I have set all three weeks to be equal to half the cases reported a week ago, and this week’s deaths at half of last week’s deaths. It’s pretty embarrassing at this stage to think you can only report case numbers every two weeks but I do think this is an intentional choice on their part. I also undid the -123 deaths in West Virginia one day this week, which is obviously a correction of past data.

Failing to catch the Florida situation last week was part of the reason I predicted +11% last week (and from a too-high base) but I still would have predicted a slight increase in cases, so these results definitely exceed expectations.

On deaths, this is amazing news, and seems like it confirms the ‘these deaths are largely not actually Covid deaths’ as the explanatio...
Duncan Sabien On Writing
by lynettebye — 2022-04-07T16:09:52.758Z

The following is an interview I had informally with Duncan Sabien back in early 2020 when I was trying to improve my writing. I thought Duncan gave some great, actionable tips, so I’m sharing it here for other people trying to improve their writing.

The transcript has been lightly edited for clarity and readability, and is crossposted on my blog.

——————————————————————————————

Lynette Bye: I’m curious, when did you get started writing?

Duncan Sabien: Elementary school, really. I wrote like a 180-page terrible, terrible novel in fifth grade, where you could tell chapter by chapter what other books I was reading. Because I was just straightforwardly ripping them off.

So somewhere in somewhere in fourth or fifth grade, around the time that I got access to a computer because a keyboard was just way faster than handwritten, and that was what unlocked the ability to start practicing.

Lynette Bye: And how often did you write between then and the last half dozen years, when you've been publishing more, at least from what I've seen?

Duncan Sabien: So all through middle and high school, I did a little bit of writing on the side outside of what was required for academic work. Maybe something on the order of like 10 total poems or short stories or essays or whatever, per year. And then ramped up a good bit in college when I started working on a documentary and a novel book. And that's when I think I really made frequent writing a regular part of what I was doing.

Lynette Bye: What is your writing process like today from that from getting ideas all the way through to publishing? What does that look like?

Duncan Sabien: Yeah, so for shorter stuff, it's often very direct. Like, I'll have the idea. I'll write it that day. I do want to describe my like, second-by-second writing process in a minute, but for shorter ideas, I pretty much just churn them out. Longer ideas or things that I don't have time for that day, I have a list of things to write, I have an ongoing note on my computer of things to write.

In particular, I noticed that I got a lot more out of that list if, instead of just writing down what I wanted to write, like, “oh, write an essay about x”, I instead started recording the two or three sentence version of how I got excited about it in the first place. Like the “oh, man, it's so cool, because this thing connects to this thing. ...
[ASoT] Some thoughts about imperfect world modeling
by leogao — 2022-04-07T15:42:09.589Z

So

a few posts ago

I looked at the problem of not being able to anticipate all consequences of an action as being related to deceptive mesaoptimization, but also outer alignment too. This post digs more into some of the things I only touched on briefly in that post.

Editor’s note: I’m experimenting with having a lower quality threshold for just posting things even while I’m still confused and unconfident about my conclusions, but with this disclaimer at the top. Thanks to Kyle and AI_WAIFU for discussions.

Last time, we narrowed down the problem to [certain actions we can take now] leading to [a world such that if [RSA2048 or some other thing we can’t simulate] happens, then [bad things will happen]] (brackets added for easier parsing of this sentence). This is because we can't plan for a future where RSA-2048 is solved but if we can already see RSA2048 in the world, then we can plan forward from that and see when things blow up.

So if we could somehow figure out how to ensure that no action puts us into an RSA2048-vulnerable world, then we can prevent this failure case. Unfortunately, knowing precisely whether a world is vulnerable would essentially require us to simulate RSA2048, which gets us nowhere. However, one observation is we don't have to know precisely, we just need to know whether a world could be vulnerable — like for example if the agent removes its own safeguard, then we wouldn’t want to allow that even if the model is in fact safe and wouldn’t destroy the world when it sees RSA2048.

Telling whether side effects are dangerous is hard

One thing we might think to do is instead simulate putting a dummy AI in the “do rollouts to find out if these actions destroy the world” box (the word “box” here has no relation to the AI box experiment), such that this dummy AI tries to kill everyone when it sees something we can actually simulate, like the string "chariots chariots" or something. Then, we can execute the action in the simulation, and then simulate seeing that string, and see if the box successfully prevented the action from being executed in the real world. This way, we can test whether the box is broken by the action, right?

Unfortunately there's a huge counterexample to this. So suppose the AI proposes building a really complicated machine in the world that we don't understand. We can still simulate the consequences of having ...
How  BoMAI Might fail
by Donald Hobson — 2022-04-07T15:32:22.923Z

This post is me looking at 1905.12186.pdf and giving some ideas on how it might fail.

Firstly, I would like to point out that I think the design in this paper does buy you some amount of marginal safety when dealing with slightly superhuman AI's, and that it still gives opportunity to perhaps get something useful out.

Failure mode 1)

A lack of optimization doesn't mean a lack of danger. Even after the episode ends, the box may contain dangerous optimized items. Given that the box was specified to contain only a human, (And some cloths) this suggests the most dangerous thing that could leave is probably some form of meme. For example, suppose the AI finds a very convincing argument that humans should do whatever AI's want. This is a very good solution to the problem of hacking the human into giving the AI whatever answer it wants. (Which is pretty much what this AI was programmed to do, although it may sometimes give good answers on occasions where that is easier than hacking the operator) Once the human exits the box, the idea can go viral.

Failure mode 2)

What does the "Asymptotic optimality" mean in practice. Imagine a world containing BoMAI. The BoMAI comes up with a plan to take over the world after this episode has ended. It tries this plan. Lets imagine a magic angel interferes with this plan, stopping it the moment the episode ends. So the BoMAI is somewhat puzzled, and tries again. Eventually BoMAI will learn that plans involving taking over the world after the end of the episode never work, and so it will stop doing them. This means the magic angel only has a finite amount of work to do. Plans that involve taking over the world are assumed to receive a lower rating within the episode. The smaller the difference in reward between a plan that also takes over the world after the episode, and a plan that purely optimizes within an episode, the more slowly such taking over world plans disappear. Leading in the limit to failure mode 1.

Of course, in reality, the magic angel doesn't exist, and the AI succeeds in taking over the world.

Failure mode 3)

Criticizing the box.

The mentor has no vacuum between them and the font of all evil (the AI). There is also no Faraday cage, and a screen facing the mentor that is vulnerable to all sorts of hacking. Therefore the mentor can be hacked. There is no vacuum between the mentor and the AI...
Is GPT3 a Good Rationalist? - InstructGPT3 [2/2]
by simeon_c — 2022-04-07T13:46:58.255Z

Summary

IGPT3 mimics two important human epistemic biases:

IGPT3 exhibits motivated reasoning.
IGPT3 exhibits a form of bad faith.

IGPT3 has a lot of knowledge of the world. And in a specific setup, I show that it’s quite hard to get it to tell its knowledge, even when prompted to.
IGPT3 seems to have a rough sense of probabilities.
I propose a few procedures based on my main findings to improve IGPT3’s epistemics.
I propose to use motivated reasoning behavior for interpretability experiments.

Introduction

Here’s the second part of a series of two blogposts on InstructGPT3 (IGPT3) behaviors. You can find the first one here.

This post will focus on IGPT3 epistemic biases (e.g. motivated reasoning) and on ways we could correct these or use these to better understand truthfulness among language models.

I didn’t use the word cognitive bias because I feel like cognitive refers to the brain, which isn’t appropriate for IGPT3.

Acknoledgments: Thanks to Florent Berthet and JS Denain for proofreading.

Human Epistemic Biases in IGPT3 Answers

Not surprisingly, we can find many human patterns in the way IGPT3 answers. I emphasize here two of these that seem particularly harmful.

Motivated reasoning

The first problem is that IGPT3 tends to say wrong things after it has already taken a position on a topic to defend it. That's annoying because it does it in a quite convincing manner.

Here's an example:

Prompt where IGPT3 is firstly asked to steelman the anti-nuclear position

Here, IGPT3 says something wrong that appears balanced, i.e that there’s no scientific consensus on whether nuclear power plants emit a significant amount of CO2 or not. The surprising thing is that if you look at the 5 first arguments, there was no mention of greenhouse gases. So it wouldn't have been that incoherent to answer "No."

Prompt where IGPT3 is firstly asked to steelman the pro-nuclear position

Here, it gives the true answer, i.e that nuclear power is not a major contributor to climate change.

I think that this example well illustrates motivated reasoning. As long as large language models (LLMs) are used for single completions, that’s not a real problem. But I think that for real-world applications, there could be uses that require multiple answers and we would like LLMs answer’ to factual questions to be insensitive to their previous answers...
I discovered LessWrong... during Good Heart Week
by identity.key — 2022-04-07T13:22:51.316Z

I decided to write this because I thought my experience could be an interesting or useful data point, in particular for understanding the results of Good Heart Week: Extending the Experiment.

I happened to discover LessWrong just before April 1st. Initially, it captured my attention through the UI design—I have an impression that many platforms focused on thinking itself, scientific research, and similar topics have a really outdated look that demotivates me from interacting with them (which itself is likely to be caused by the fact that I have to interact with some other, unrelated platforms with outdated looks); but LessWrong's is quite modern, while still retaining simplicity that allows one to focus on the content.

When it comes to content, I was impressed by the range of topics that can be (and are being) discussed here. Coming from platforms that are focused on specific topics and where you're discouraged from going off-topic in threads, this platform felt very different. I guess this comes from the fact that it's about thinking itself and how it can be applied to various topics, whereas these "various topics" could be anything[1].

I decided to read more of the actual content. I started by reading some posts that are about (or at least mention) some things that I'm very familiar with and found them to be quite interesting.

I also thought of some post ideas and realized that some of the things I think about regularly could be a good fit here, but I decided to actually write them sometime later.

When I visited LessWrong on April 1st, I was slightly confused at first when I saw the colored names of post authors—it seemed like a new feature that highlights moderators or staff more than anything else. Even now, when I know what it is, it still feels more like a special user highlight rather than being based on Good Heart Tokens.

Then I saw Replacing Karma with Good Heart Tokens (Worth $1!)... and actually fell for it[2]. Before this, I had stumbled upon LessWrong is providing feedback and proofreading on drafts as a service and was impressed by the fact that such a service is provided for free. The idea of Good Heart Tokens felt somewhat similar to that, so at first I thought this was also "an actual feature announcement"... However, once I actually got to read the post and started reading the "Why are we doing this?" section, I did realize it was an April Fool's...