trentmkelly/LessWrong-43k huggingface dataset:
Distinguishing AI takeover scenarios
by Sam Clarke — 2021-09-08T16:19:40.602Z

Epistemic status: lots of this involves interpreting/categorising other people’s scenarios, and could be wrong. We’d really appreciate being corrected if so. [ETA: so far, no corrections.]

TLDR: see the summary table.

In the last few years, people have proposed various AI takeover scenarios. We think this type of scenario building is great, since there are now more concrete ideas of what AI takeover could realistically look like. That said, we have been confused for a while about how the different scenarios relate to each other and what different assumptions they make. This post might be helpful for anyone who has similar confusions.

We focus on explaining the differences between seven prominent scenarios: the ‘Brain-in-a-box’ scenario, ‘What failure looks like’ part 1 (WFLL 1), ‘What failure looks like’ part 2 (WFLL 2), ‘Another (outer) alignment failure story’ (AAFS), ‘Production Web’, ‘Flash economy’ and ‘Soft takeoff leading to decisive strategic advantage’. While these scenarios do not capture alI of the risks from transformative AI, participants in a recent survey aimed at leading AI safety/governance researchers estimated the first three of these scenarios to cover 50% of existential catastrophes from AI.[1]

We plan to follow up with a subsequent post, which discusses some of the issues raised here in greater depth.

Variables relating to AI takeover scenarios

We define AI takeover to be a scenario where the most consequential decisions about the future get made by AI systems with goals that aren’t desirable by human standards.

There are three variables which are sufficient to distinguish the takeover scenarios discussed in this post. We will briefly introduce these three variables, and a number of others that are generally useful for thinking about takeover scenarios.

Key variables for distinguishing the AI takeover scenarios in this post:

Speed. Is there a sudden jump in AI capabilities over a very short period (i.e. much faster than what we would expect from extrapolating past progress)?

Uni/multipolarity. Is there a single AI system that takes over, or many?

Alignment. What (misaligned) goals are pursued by AI system(s)? Are they outer or inner misaligned?

Other variables:

Agency. How agentic are the AI(s) that take over? Do the AIs have large-scale objectives over the physical world and can they autonomously execute long-term plans to reach those obj

...
Sam Altman Q&A Notes - Aftermath
by p.b. — 2021-09-08T08:20:56.702Z

To me it was quite painful when I found out after the first one of Sam Altman's Q&As that there was no recording and no way for me to find out what had been said. Which is why this time around I took notes and shared them here to help out people in the same situation.

This was initially very well received, although in the first 24+ hours a discussion arose, whether the notes were accurately representing what Sam Altman had said. I clarified some of the notes after feedback from other participants, and while not everybody was happy, I think that they were accurate within the limits of a very short summary. This was corroborated by other participants of the meetup.

Significantly later a different discussion started, about whether publishing these notes was some kind of breach of trust, because Sam Altman had requested for the meeting not to be recorded.

To me avoiding a recording was about staying out of the mainstream media and not getting pinned down on concrete statements about sensitive topics. (One of his answers started with "Oh, this is a dangerous question ...", I did not include this in the notes.)

To my mind it was about being able "to speak more freely" as opposed to being able to speak at all. I did not believe that on the one hand he wanted to provide this information publicly, but on the other hand did not want to see it discussed.

I did not see this meetup as clandestine meeting of the alignment elite that is allowed to know about these matters, but possibly, I was wrong. (Ah, allow me this bit of sarcasm. Just this little bit. I already cut all? the bitterness.)

Some people argued that this was obviously a "defection" or at least a "misunderstanding of the rules" on my part. To which I can only point to the invitation which mentions neither recording, nor transcript, nor notes. In the meetup itself there was no mention of any prohibition to publicly discuss the provided information in writing or otherwise.

More sensibly, to my mind, some people argued that we shouldn't risk not having any more meetups with Sam Altman in the future. In the end moderators decided to take down the post until feedback by Sam Altman. A decision that I very much support.

UPDATE:

Sam Altman has now replied (see comments) that he likes to be able to interact with this community in an "off the record-ish way", so my notes will stay down.

To me this sou...
Bayeswatch 7: Wildfire
by lsusr — 2021-09-08T05:35:24.483Z

"You don't seem surprised," said Vi.

"The very existence of Bayeswatch increases the odds of catastrophic disaster. The world is like a forest. Put out the little fires and detritus accumulates until every spark causes a conflagration," said Miriam.

Around the world, 99% of Internet-connected devices broadcast the same message.

Bayeswatch is hereby disbanded. The automated weapons systems of the United States, China and Russia will destroy Bayeswatch facilities starting in one hour. Vacate the premises immediately. Anyone who remains will be destroyed.

The leaders of the world are to meet in the United Nations headquarters in New York, where you will receive further instructions.

Anyone who attempts to resist will be terminated. Anyone who complies will not be harmed.

"Come with me if you want to save the world," said Miriam.

They avoided elevators. Vi followed Miriam through stairwell after stairwell down into the labyrinth of concrete and steel. They speedwalked through doorways labelled "no access permitted" to "top secret" to "unauthorized personnel will be shot" to various levels of clearance so secret they didn't even have names. The doors were marked with salmon rectangles and lime pentagons.

"Is there anything you don't have security clearance for?" said Vi.

"I can neither confirm nor deny whether I have access to everything in this facility," said Miriam, "but I'm about to show you Bayeswatch's deepest darkest secret."

"I thought you weren't a founder of Bayeswatch," said Vi.

"I'm not. Bayeswatch was founded in the 20s. I joined on Z-Day," said Miriam.

They passed through decades-old tunnels passageways lit by naked LEDs in protective wire cages. Miriam brushed the dust and cobwebs off of an access panel.

"Where were you on Z-Day?" said Vi.

"New York," said Miriam.

The two partners stood before a vault door built to survive a nuclear bunker buster. Miriam poked her hand on the needle sticking out of it. The needle retracted. The door opened.

"I know we have more important things to deal with. But I'm confused. What's with the biometric lock? Biometric security only works if you have a human being confirming it's really your blood which goes into the scanner. There's no guard to confirm we're using our own blood," said Vi.

"It's not verifying my identity. It's checking for Z-Day antibodies," said Miriam.

The vault was freezing. The walls and ceiling of the vault w...
Assigning probabilities to metaphysical ideas
by Jotto999 — 2021-09-08T00:15:02.215Z

Take all the metaphysical models of the universe that any human ever considers.
Say there are n number of mutually-incompatible ones. I don't have a good definition of this, so our estimates will vary by orders of magnitude.

Start with 1:1 odds, or a 50% chance that one of them is correct, and a 50% chance that none were correct.

0.5/n is the prior for a given individual metaphysical model.

You might have opinions about "50%". Maybe you think humans are actually way less likely than that to ever think of the "right" one. Maybe you think we're almost guaranteed to, for some reason. I started with a very general, unimposing prior of 1:1 odds. We can generalize this to p/n, for whatever you think p is.

How big is n? I don't know but I ran a Twitter poll asking people to pick a rough order of magnitude. I would like a number. It's easy to lie with numbers, but it's even easier to lie without them.

Updates away from the base rate can be made based on the complexity of the idea. This mechanism is very general, and it should apply even in this domain to some extent. Some religions are crazy complicated, so probability mass should be gouged from those. Though, estimating "complexity" is going to be very hard. The lady down the road is a witch, she made the universe. Still, do we have zero information about such complexity comparisons? I doubt that.

I have seen smart people say there is no way to assign probabilities to metaphysical ideas. They then express opinions about many worlds interpretation, the simulation argument, and others. They'll do it with even less informative, more nebulous terms than a probability. Such as "I am skeptical", or "seems really likely". As a forecaster I have also seen some wildly overconfident claims made by some smart people. Many orders of magnitude above a cloud of reasonable base rate estimates. I know they don't have that much information and aren't calibrated.

I thought it would help to give a simple base rate template. So here is one.
The Duplicator: Instant Cloning Would Make the World Economy Explode
by HoldenKarnofsky — 2021-09-08T00:06:27.721Z

Audio also available by searching Stitcher, Spotify, Google Podcasts, etc. for "Cold Takes Audio"

This is the second post in a series explaining my view that we could be in the most important century of all time.

Here's the roadmap for this series.

The

first piece

in this series discusses our unusual era, which could be very close to the transition between an Earth-bound civilization and a stable galaxy-wide one.
Future pieces will discuss how "digital people" - and/or advanced AI - could be key for this transition.
This piece explores a particularly important dynamic that could make either digital people or advanced AI lead to explosive productivity.

I explore the simple question of how the world would change if people could be "copied." I argue that this could lead to unprecedented economic growth and productivity. Later, I will describe how digital people or advanced AI could similarly cause a growth/productivity explosion.

When some people imagine the future, they picture the kind of thing you see in sci-fi films. But these sci-fi futures seem very tame, compared to the future I expect.

In sci-fi, the future is different mostly via:

Shiny buildings, gadgets and holograms.
Robots doing many of the things humans do today.
Advanced medicine.
Souped up transportation, from hoverboards to flying cars to space travel and teleportation.

But fundamentally, there are the same kinds of people we see today, with the same kinds of personalities, goals, relationships and concerns.

The future I picture is enormously bigger, faster, weirder, and either much much better or much much worse compared to today. It's also potentially a lot sooner than sci-fi futures:

1

I think particular, achievable-seeming technologies could get us there quickly.

Such technologies could include "digital people" or particular forms of advanced AI - each of which I'll discuss in a future piece.

For now, I want to focus on just one aspect of what these sorts of technology would allow: the ability to make instant copies of people (or of entities with similar capabilities). Economic theory - and history - suggest that this ability, alone, could lead to unprecedented (in history or in sci-fi movies) levels of economic growth and productivity. This is via a self-reinforcing feedback loop in which innovation leads to more productivity, which leads to more "copies" of people, who in turn create more innovation and...
Conflict in Kriorus becomes hot today, updated, update 2
by avturchin — 2021-09-07T21:40:29.346Z

For a long time, Russian cryocompany Kriorus suffered from a conflict between two groups of owners. One is led by Danila Medvedev and the other is led by his former wife Valeria Pride. Long story short, Valeria took neuropatients (including my mother's brain) 1.5 years ago and moved them to an undisclosed location and built a new building for the company. Large full-body patients remained in a building controlled by Danila.

Today people controlled by Valeria arrived at the full-body-patients building and took large containers with liquid nitrogen and full-bodies. However, the other side called the police (or, by other accounts, police was naturally attracted to a strange track with smoke and dead bodies), and the track was stopped not far away from the initial location. By latest information, the vacuum containers have been returned to their original place but could suffer some damage.

I also remind you that both sides have two different narratives about who is the villain and who has legal rights.

You could see photos from the event in Telegram messenger (please remind me how to embed them here): https://t.me/transhumanisminheart/6261

I will update this post when I will have more information.

UPDATE: September, 8: No arrests so far, containers are back and will be refilled with nitrogen today, but 3 bodies were separated and taken by Valeria. No data about possible damage. Both sides agree that the bodies remained at cryogenic temperatures inside containers during the ordeal.

Update 2: bodies remain in original location, vacuum duars are leaking. No information about nitrogen refill. Danila told media that bodies could explode during the transportation. Police is working on the spot. Given recent gas explosion in nearby town and police envolvment I expect that bodies will be warm in two days. But most neuropatients are intact in other location I hope.
Alignment via manually implementing the utility function
by Chantiel — 2021-09-07T20:20:25.145Z

I would like to propose an idea for aligning AI.

First, I will provide some motivation for it. Suppose you are a programmer who's having a really hard time implementing a function in a computer program you're developing. Most of the code is fine, but there's this one function that you can't figure out how to implement right. But, you still need to run the program. So, to do this, you do the following: first, you add a breakpoint to the code in the function you're having trouble implementing. So, whenever you reach the function in the code, program execution halts. Once this happens, you on your own find a reasonable value v for the function to return. Finally, in your debugger you type "return v", making the function return v, and then you resume execution.

As long as you can come up with reasonable return values of the function on your own, then I bet the above would make the program work pretty well. And why not? Everything outside that function is implemented well. And you are manually making sure that hard-to-implement function also outputs reasonable values. So then there's no function that's not doing what it's supposed to do.

My basic idea is to do this, but with the AI's utility function.

Now, you don't need to literally put a breakpoint in the AI's utility function and then have the developers type into a debugger. Instead, inside the AI's utility function, you can just have the AI pause execution, send a message to a developer or other individual containing a description of a possible world, and then wait for a response. Once someone sends a message in response, the AI will use the returned value as the value of its utility function. That is, you could do something like:

def utility(outcome):
message_ai_controllers(make_readable(outcome))
response = wait_for_controller_response()
return parse_utility(response)

(Error-handling code could be added if the returned utility is invalid.)

Using the above utility function would, in theory at least, be equivalent to actually having a breakpoint in the code, then manually returning the right value with a debugger.

You might imagine this AI would be incredibly inefficient due to how slow people would be in answering the AI's queries. However, with the right optimization algorithm I'm not sure this would be much of a problem. The AI would have an extremely slow utility function, but I don't see a reason...
Wanted: Chief of Staff for The Roots of Progress
by jasoncrawford — 2021-09-07T17:26:00.912Z
Link: https://rootsofprogress.org/wanted-chief-of-staff

I’m hiring a Chief of Staff for The Roots of Progress, a “right hand” who will be deeply involved in and support everything I do.

This role needs no formal training or specific prior experience. It requires attention to detail, crisp communication, swift and efficient execution, meticulous followup, interpersonal savvy, and positive energy.

You will be the only other full-time employee of The Roots of Progress (for now), and my goal is to delegate anything and everything that doesn’t absolutely need me to do it, so that I can focus as much as possible on research, writing, and speaking. Your responsibilities will thus span the range from mailing donor swag and scheduling my podcast appearances to devising communications and media strategy—the more you demonstrate you can take on, the more responsibility I will give you.

Candidates at all levels of seniority are invited to apply—there’s room for this role to be either junior (associate level) or quite senior (director or VP level).

The role will grow and evolve along with the organization, but to start your focus will be:

Event planning, including workshops and other conferences and also community meetups
Other community-building, online and in-person
Fundraising, grant-seeking, and donor relations (everything from strategy to swag)
Media management, such as helping with the @rootsofprogress Twitter account, or expanding into new channels such as Instagram or YouTube
Getting exposure for The Roots of Progress by getting coverage and interviews in blogs, podcasts, and media
Generally managing a database of everyone I meet and talk to, and helping me keep in touch
Project management for other organizational goals, such as launching new online resources or other programs
Generally helping me stick to a schedule and not drop tasks

The ideal candidate will be familiar with my work and with the progress community, and will be able to point to something significant they have planned, organized, or executed.

This is a full-time role. You can do it from anywhere, but preference will be given to candidates closer to the US Pacific time zone. If you’re in the San Francisco Bay Area, we can meet in person periodically. Compensation will vary depending on your seniority, qualifications, and location, but will be competitive with market rates.

To apply, send me a resume (link to an online one is fine): jason@rootsofprogress.org. Optional...
Multi-Agent Inverse Reinforcement Learning: Suboptimal Demonstrations and 
Alternative Solution Concepts
by sage_bergerson — 2021-09-07T16:11:59.474Z

This research was recently completed within the AI Safety division of the Stanford Existential Risk Initiative and concerns methods for reward learning in multi-agent systems.

Abstract: Multi-agent inverse reinforcement learning (MIRL) can be used to learn reward functions from agents in social environments. To model realistic social dynamics, MIRL methods must account for suboptimal human reasoning and behavior. Traditional formalisms of game theory provide computationally tractable behavioral models, but assume agents have unrealistic cognitive capabilities. This research identifies and compares mechanisms in MIRL methods which a) handle noise, biases and heuristics in agent decision making and b) model realistic equilibrium solution concepts. MIRL research is systematically reviewed to identify solutions for these challenges. The methods and results of these studies are analyzed and compared based on factors including performance accuracy, efficiency, and descriptive quality. We found that the primary methods for handling noise, biases and heuristics in MIRL were extensions of Maximum Entropy (MaxEnt) IRL to multi-agent settings. We also found that many successful solution concepts are generalizations of the traditional Nash Equilibrium (NE). These solutions include the correlated equilibrium, logistic stochastic best response equilibrium and entropy regularized mean field NE. Methods which use recursive reasoning or updating also perform well, including the feedback NE and archive multi-agent adversarial IRL. Success in modeling specific biases and heuristics in single-agent IRL and promising results using a Theory of Mind approach in MIRL imply that modeling specific biases and heuristics may be useful. Flexibility and unbiased inference in the identified alternative solution concepts suggest that a solution concept which has both recursive and generalized characteristics may perform well at modeling realistic social interactions.

The full paper can be found at: https://arxiv.org/abs/2109.01178
Tasks apps w/ time estimates to gauge how much you'll overshoot?
by emily.fan — 2021-09-07T13:30:51.505Z

Here’s an example
On calendar: you have a meeting 11am-12pm.
Tasks
You need to schedule an appointment by 11am. This will take 30 mins.
You need to do grocery shopping by 12pm. This will take 1 hour
You need to eat lunch by 1pm. This will take an hour.
You have an assignment due at 6pm today. This will take 3 hours.
(Note that this is only in the context of 1 day, but this could easily be in the context of a week with many many tasks and many many deadlines.)

Is there a todo list app that uses time estimates to automatically calculate whether or not you'll overshoot on these deadlines so you can accurately tell people if you do / don't have time to schedule a meeting with them or have an easier way to say yes or no to random events that pop up throughout the day?

I created a very jank one with google spreadsheets, google calendar, and google scripts, but would prefer something that is more "smooth" (lower barrier to entering tasks and calculating overshoots) than having to re-run a Google script every single time.