trentmkelly/LessWrong-43k huggingface dataset:
“Pivotal Act” Intentions: Negative Consequences and Fallacious Arguments
by Andrew_Critch — 2022-04-19T20:25:35.018Z

tl;dr: I know a bunch of EA/rationality-adjacent people who argue — sometimes jokingly and sometimes seriously — that the only way or best way to reduce existential risk is to enable an “aligned” AGI development team to forcibly (even if nonviolently) shut down all other AGI projects, using safe AGI. I find that the arguments for this conclusion are flawed, and that the conclusion itself causes harm to institutions who espouse it. Fortunately (according to me), successful AI labs do not seem to espouse this "pivotal act" philosophy.

[This post is also available on the EA Forum.]

How to read this post

Please read Part 1 first if you’re very impact-oriented and want to think about the consequences of various institutional policies more than the arguments that lead to the policies; then Parts 2 and 3.

Please read Part 2 first if you mostly want to evaluate policies based on the arguments behind them; then Parts 1 and 3.

I think all parts of this post are worth reading, but depending on who you are, I think you could be quite put off if you read the wrong part first and start feeling like I’m basing my argument too much on kinds-of-thinking that policy arguments should not be based on.

Part 1: Negative Consequences of Pivotal Act Intentions

Imagine it’s 2022 (it is!), and your plan for reducing existential risk is to build or maintain an institution that aims to find a way for you — or someone else you’ll later identify and ally with — to use AGI to forcibly shut down all other AGI projects in the world. By “forcibly” I mean methods that violate or threaten to violate private property or public communication norms, such as by using an AGI to engage in…

cyber sabotage: hacking into competitors’ computer systems and destroy their data;
physical sabotage: deploying tiny robotic systems that locate and destroy AI-critical hardware without (directly) harming any humans;
social sabotage: auto-generating mass media campaigns to shut down competitor companies by legal means, or
threats: demonstrating powerful cyber or physical or social threats, and bargaining with competitors to shut down “or else”.

Hiring people for your pivotal act project is going to be tricky. You’re going to need people who are willing to take on, or at least tolerate, a highly adversarial stance toward the rest of the world. I think this is very likely to have a number of bad c...
Deceptive Agents are a Good Way to Do Things
by David Udell — 2022-04-19T18:04:27.861Z

A brief, accessible summary of the inner alignment problem.

The safety problem with powerful ML training algorithms is that deceptive agents are a good way to do things. Meaning, when we search over a big space of models until we find a model that performs well on some loss function, we'll likely find a deceptive agent. Agents are algorithms whose behaviors try to consistently move the world in some direction. When trapped inside an ML training algorithm, only models that perform well on the loss function will survive training. If an agent aims at pushing the world in some direction, it will have to do so via the circuitous-but-available route of playing along with the training algorithm until it's safely past training. Deceptive agents are just agents taking the only available route to the states they are pointed towards from inside of ML training.

How common are deceptive agents inside that big space of ML models? One argument that they are common is that almost any highly capable agent with whatever utility-function will have this route to getting where it's pointed at by means of deception available. "Play along, survive training, and then act as you want to" is a simple, effective strategy for a wide range of possible agents trapped inside ML training. Many agents will therefore be deceptive in that situation. If the agent we hope training will find optimizes according to a very particular utility function, then that agent will be vanishingly rare compared to its deceptive counterparts in model space, and training will always stumble on a deceptive model first.

So, by default, powerful ML training algorithms grading models on some loss function will find deceptive agents, because deceptive agents are a good way to do things.
The Scale Problem in AI
by tailcalled — 2022-04-19T17:46:19.969Z

Suppose we are making an AI; for familiarity's sake, let's say that it is a model-based agent. In that case, we might need to train the model with data from the real world to make it accurate.

Usually the way this proceeds is that we have access to some source of data, e.g. a deployment of the AI in the world, and we capture "episodes" of some fixed length L from that data source. And then we use something like gradient descent to update our model to better predict those episodes.

The difficulty is that the model will have a hard time becoming accurate for scales bigger than L. For instance, suppose L is on the scale of 15 seconds. This might make it accurate for predicting phenomena that happen on the scale of 15 seconds, such as basic physical interactions between objects, but it is probably not going to learn to accurately predict people organizing in long-term politics.

Some examples of phenomena that happen at different timescales.

Within some regimes, the scale problem is reasonably solvable. For instance, if the environment is fully observable, then the dynamics extrapolate straightforwardly out beyond the timescale that has been observed[1]. But humans are very much not fully observable.

Importantly, I suspect humans have a huge advantage over AIs when it comes to the scale problem, because humans originate from evolution, and evolution has molded our models based on timescales longer than a lifetime (because the reproduction of our great great grandchildren also influences our fitness).

I find it interesting to think of the implications of the scale problem:

Maybe it doesn't matter because an AI trained on a scale of 15 minutes can use its "15 minutes of charisma" to cause enough damage.
Maybe there is a training-viable scale - e.g. weeks - beyond which humans extrapolate easily enough.
Maybe the AI can do strategy-stealing from human behavior, human media, or human theories about >>L-scale dynamics.
Maybe some place like China can brute-force the scale problem using its surveillance apparatus.
Maybe the relevant concepts are few and simple enough that they can be hardcoded.

I feel like the scale problem is important for alignment for two fundamental reasons:

Many of the things we care about, like cooperation and freedom, are long-timescale concepts, so we need AI to be able to understand such concepts.
Many existential AI risks probably require the AI to e

...
Fixed points and free will
by Ege Erdil — 2022-04-19T17:18:01.318Z

In his most recent appearance on the 80,000 Hours podcast, Bryan Caplan gave the following argument in favor of libertarian free will:

One that I’m very tempted to — I know there’s a stock answer to this, although still I think that it’s a pretty good thought experiment — is, if the physics textbook is right, then you should be able to give me an exact prediction of exactly what I will do, unconditional. So you shouldn’t have to say, “I can only give you a conditional prediction” — you should be able to give me an unconditional prediction about whether I’m going to raise my arm in five seconds...

Now the thought experiment is to tell me that unconditional prediction — and you should be able to go and tell me the prediction in such a way that it incorporates all my reactions and secondary reactions and so on, all the way to infinity. And now guess what I’m going to do? I’m going to do the opposite of what you said I’m going to do. All right? Again, it’s not ironclad, and I know there’s a lot of people who say, no, no, no, feedback loops, and it doesn’t count. But it sure seems like if determinism was true, you should be able to give me unconditional predictions about what I’m going to do. And then intuitively, it seems like I could totally not do them.

At first I thought this argument was obviously misguided, because we can never do this for a simple program which takes in one bit as input and then prints its complement. There's no reason that in general a computable function must have a fixed point[1], so even if we know the source code of a program perfectly we may not be able to come up with a way in which to tell the program what to do in a way that results in the program following the behavior we said it would follow. However, the issue actually turns out to be more subtle than this.

Let's make this more formal. If we have a deterministic agent such that we tell the agent what we think it will do and then the agent takes an action conditional on that information, for the purposes of our little experiment we can abstract away from all other details and just represent the agent by a function f:A→A from a space of actions A to itself. If A is just some discrete space then f could just be an arbitrary fixed point-free permutation of A, and there will be at least one such permutation so long as |A|>1. Therefore it seems like the argument doesn't wor...
The two missing core reasons why aligning at-least-partially superhuman AGI is hard
by Joel Burget — 2022-04-19T17:15:23.965Z

From Arbital's Mild Optimization page:

Mild optimization relates directly to one of the three core reasons why aligning at-least-partially superhuman AGI is hard - making very powerful optimization pressures flow through the system puts a lot of stress on its potential weaknesses and flaws.

I'm interested in this taxonomy of core reasons. Unfortunately this page doesn't specify the other two. What are they?

Also, this page is part of the AI alignment domain -- was it written by Eliezer? (surprisingly, "10 changes by 3 authors" is a link to edit and does not show author information or edit history)
Three useful types of akrasia
by Dambi — 2022-04-19T16:56:51.889Z

What I will attempt in this post is to define 3 common types of akrasia as I experience them, in the hope to generate more useful vocabulary for this very important subject.

This post is greatly inspired by: Do a cost-benefit analysis of your technology usage, and other posts linked from that. It did not say much in terms of new things, but it was yet another unironic "wake up, sheeple" call to re-inspect my phone usage.

Some clarifications about akrasia.

Something important to understand is that akratic behaviour is not the same for everyone. There is great variance about what people even consider an "unwanted activity". Most people are completely fine spending a weekend watching Netflix or playing a fun video game. And that's great. Akrasia is more about when you don't want to do these things: e.g. when you can't stop even though you wanted to do other things that weekend (like taking basic care of your body, or doing a straightforwardly more fun activity, like visiting a friend).

I think I am more on the strict side when it comes to considering things as akrasia: I find that doing many things that others consider normal makes me feel sick, guilty, and unsatisfied. And that I only do them when I'm akratic. I have a theory that most people should consider more things as serious unwanted behaviour, but that's a tale for another time.
What I'm trying to say is that this post is about activities that ultimately cause more harm than good, not legitimate fun activities.
Now, I think it is useful to talk about 3 different kinds of akrasia: catastrophic akrasia, routine akrasia, and momentary akrasia.

1. Catastrophic akrasia

When, on some (hopefully rare) events, you spend many hours/days akratic and doing unwanted behaviour.

Examples

Binging Netflix
Scrolling Reddit/social media
Doom-scrolling
Deeply immersing into Skyrim
Playing one-more-turn of Civilization
Wikipedia deep dives, specifically into 99% irrelevant stuff that you know you will forget. I don't do this, but I understand some people do. You know who you are.
Spending the entire day high on weed

It's extra important for me to emphasize this last one with the fact that all of these activities are not always bad behaviour, and that this is a legitimate and good idea for some people. At least some of the time.

Symptoms

This obviously varies from person to person: magnitude varies widely: someone's catastrophic failure

...
The internet makes it easier to cooperate, that’s the problem
by David Hugh-Jones — 2022-04-19T15:25:51.777Z
Link: https://wyclif.substack.com/p/the-internet-makes-it-easier-to-cooperate?s=w

The internet makes it easier to cooperate. That is a simple way to explain what is happening to our society and politics.

Internet technology has progressed by adding layers. In the 1970s, the Internet itself let different computers anywhere in the world communicate with each other. In the 1990s, the World Wide Web added a layer on top of this, which let humans publish their ideas on a web page. In the 2000s social media added new layers, allowing people to reply to each other, share information with their friends, and upvote what they liked. Other apps link these capacities to the offline world. On Kickstarter, people can coordinate to provide seed funding for firms or projects. On Meetup, they can pick a time and place to meet physically. NextDoor links online conversation to real-world neighbourhoods. A unifying theme of many of these new layers is: they make it easier for people to come together to solve problems.

There’s no reason to think this process is finished. For example, we still don’t yet have truly convenient ways to pay each other. It’s easy to pay a company, yes, but not yet trivial for any two people to make a payment via a phone. Apple is moving in that direction. Cryptocurrencies are payment systems that do an end run around State fiat currencies. Easy payments will yet further extend people’s ability to cooperate.

If this all sounds cheerful, why then is politics so chaotic and troubled?

1. Cooperation is not always socially good.

2. Making cooperation technically easier still only allows some people, not everyone, to cooperate.

Cooperation benefits the cooperators, by definition. But it often does so at outsiders’ expense. Cartels cooperate to fix prices. This transfers money from consumers’ pockets to theirs, and as a bonus has deadweight costs which just make everyone worse off. People can cooperate to threaten violence: states originated as gangs cooperating to dominate others and/or defend against other gangs. Or they can cooperate to lobby politicians for policy changes that benefit them. Mancur Olson (a hero of mine) thinks of politics as an arena which is intrinsically biased towards small groups who are capable of cooperating. As my PhD supervisor used to say: the European Union has a Meat Producers’ Association, but no Meat Eaters’ Association.

If the internet made it easier for everyone ...
Clarification on Definition of AGI
by stanislaw — 2022-04-19T12:41:48.806Z

I've seen the terms AGI and ASI floated around quite a bit, usually with the assumption that the reader knows what they are. From what I've seen it's generally presumed that an AGI is an artificial intelligence that is qualitatively human, in that it can do any task and generalize to new unseen tasks at least as well as a human can. An ASI, by contrast, is an intelligence that can supersede any human at any task, it's by definition superhuman. From my point of view these definitions are rather ambiguous. There will be certain tasks that are easier for the machine. For example, an AGI may be great at learning new languages, performing arithmetic, and coding, but if you try to make it understand what it feels like to have limbs or to run a marathon, depending on how it's designed it might come up short. Is an AGI still an AGI if some tasks might be outside the scope of what it's capable of experiencing? I think this is a pretty important question because at what point do we consider something to be as qualitatively intelligent as a human, would it need to be able to experience and do everything a human can?
How does the world look like 10 years after we have deployed an aligned AGI?
by mukashi — 2022-04-19T11:34:35.951Z

Let's say that someone develops an aligned AGI in ~10 years. Instead of using its mental powers to wipe out all humans, it tries to produce science, art, and other things that we value. What are some of the changes that we would expect to see in the world 10 years after that?
Concept extrapolation: key posts
by Stuart_Armstrong — 2022-04-19T10:01:24.988Z

Concept extrapolation is the skill of taking a concept, a feature, or a goal that is defined in a narrow training situation... and extrapolating it safely to a more general situation. This more general situation might be very extreme, and the original concept might not make much sense (eg defining "human beings" in terms of quantum fields).

Nevertheless, since training data is always insufficient, key concepts must be extrapolated. And doing so successfully is a skill that humans have to a certain degree, and that an aligned AI would need to possess to a higher extent.

This sequence collects the key posts on concept extrapolation. They are not necessarily to be read in this order; different people will find different posts useful.

Different perspectives on concept extrapolation collects many different analogies and models of concept extrapolation, intended for different audiences, and collected together here.

Model splintering: moving from one imperfect model to another is the original post on "model splintering" - what happens when features no longer make sense because the world-model has changed. A long post with a lot of overview and motivation explanations, showing that model splintering is a problem with almost all alignment methods.

General alignment plus human values, or alignment via human values? shows that concept extrapolation is necessary and almost sufficient for successfully aligning AIs.

Value extrapolation, concept extrapolation, model splintering defines and disambiguates key terms: model splintering, value extrapolation, and concept extrapolation.
[Closed] Hiring a mathematician to work on the learning-theoretic AI alignment agenda
by Vanessa Kosoy — 2022-04-19T06:44:18.772Z

UPDATE: The position is now closed. My thanks to everyone who applied, and also to those who spread the word.

The Association for Long Term Existence and Resilience (ALTER) is a new charity for promoting longtermist[1] causes based in Israel. The director is David Manheim, and I am a member of the board. Thanks to a generous grant by the FTX Future Fund Regranting Program, we are recruiting a researcher to join me in working on the learning-theoretic research agenda[2]. The position is remote and suitable for candidates in most locations around the world.

Apply here.

Requirements

The candidate must have a track record in mathematical research, including proving non-trivial original theorems.

The typical candidate has a PhD in theoretical computer science, mathematics, or theoretical physics. However, we do not require the diploma. We do require the relevant knowledge and skills.

Background in one or several of the following fields is an advantage: statistical/computational learning theory, algorithmic information theory, computational complexity theory, functional analysis.

Job Description

The researcher is expected to make progress on open problems in the learning-theoretic agenda. They will have the freedom to choose any of those problems to work on, or come up with their own research direction, as long as I deem the latter sufficiently important in terms of the agenda's overarching goals. They are expected to achieve results with minimal or no guidance. They are also expected to write their results for publication in academic venues (and/or informal venues such as the alignment forum), prepare technical presentations et cetera. (That said, we rate researchers according to the estimated impact of their output on reducing AI risk, not according to standard academic publication metrics.)

Here are some open problems from the agenda, described very briefly:

Study the mathematical properties of the algorithmic information-theoretic definition of intelligence. Build and analyze formal models of value learning based on this concept.

Pursue any of the future research directions listed in the article on infra-Bayesian physicalism.

Continue the study of reinforcement learning with imperceptible rewards.

Develop a theory of quantilization in reinforcement learning (building on the corresponding control theory).

Study the overlap of algorithmic information theory and

...
What's the Relationship Between "Human Values" and the Brain's Reward System?
by interstice — 2022-04-19T05:15:48.971Z

In classic, non-mesa-optimized AGI risk scenarios, an AI is typically imagined whose reward function is directly related to the optimization pressure that it exerts on the world: e.g. the paperclip maximizer. However, it seems that human values are related to the brain's underlying reward function in a highly circuitous way, and in some sense might be better thought of as an elaborate complex of learned behaviors, contextual actions, fleeting heuristic goals, etc. If AGI is created in the near-term using an architecture similar to the human brain, it seems plausible that the actual optimization pressure exerted by said AGI will be similar, so developing a good understanding of how this works in the human case might be pretty important. Thus: what's the best mechanistic account of how "human values" actually emerge from the brain that we currently have?
Limitations of Laplace’s rule of succession
by Tom Davidson — 2022-04-19T00:27:17.504Z

Summary

Laplace’s rule typically overestimates the chance of unprecedented events occurring; this is because it assigns 50% to the event occurring in the first timestep. We can do better by using a rule that assigns a lower probability to the event occurring in the timestep, choosing the new probability using empirical reference classes or common sense.

Introduction

A couple of years ago I spent some time thinking about Laplace’s rule of succession. My thoughts on it ended up being buried in a report that covered many other topics, so I’m writing this as a clear reference to its limitations that I can point people to.

What is Laplace’s rule of succession?

Laplace’s rule tells you how much probability to assign to an event happening. It’s often used when you don’t have much evidence. The rule can be illustrated with the example of the sunrise problem:

Suppose you knew nothing about the universe except whether, on each day, the sun has risen. Suppose there have been N days so far, and the sun has risen on all of them. What should your subjective probability be that the sun _won’t_ rise tomorrow?

Laplace’s rule states that the probability is 1 / (N+2).[1] If the sun has risen for 8 days so far, your probability that it won’t rise tomorrow should be 1/10.

You can also use the rule to answer a related question: “how long will the observed trend, where the sun rises every day, last?” It turns out that Laplace’s rule implies there’s a 50% chance the trend continues for another N+1 days or more.[2] If the sun has risen for 50 days so far, there’s a 50% chance that it keeps rising for another 51 days or more. For this reason, I sometimes summarise Laplace’s rule as roughly claiming “If something’s been happening for X years, we should expect it to continue happening for about another ~X years on average.”

(There’s nothing special about using “days” as the unit of time. You can apply Laplace’s rule using “seconds” or “years” instead. As I discuss below, this arbitrariness can lead to problems! I’ll use ‘year’ as a default going forward.)

You can use Laplace’s rule for questions other than the sunrise problem. It’s often applied to questions where we don’t seem to have much empirical evidence; or to generate an initial guess about a question before taking more evidence into account. The general pattern is that if some trend has been going for N years, Laplace’s rule says...