trentmkelly/LessWrong-43k huggingface dataset:
Maximal Sentience: A Sentience Spectrum and Test Foundation
by Snowyiu — 2023-06-01T06:45:12.616Z

Hello! I am writing this post, because I think it is important to get to a definition of sentience good enough to where the level of sentience of a being becomes testable and we can make predictions about how a being with some level of sentience would behave.

Part of why I find this to be of importance is because it might not be unthinkable that we're going to create AIs which can satisfy some definitions of sentience in the foreseeable future and tests for sentience developed after that point would likely be constructed in such a way as to favour humans and exclude the AI, whereas with any test constructed beforehand we can say "Yup, this does indeed pass this test we made some time ago, so maybe we should consider treating it as a lifeform with value". Also, the way of thinking about sentience outlined in this post just seems to work pretty well in general, even when just thinking about humans or other living beings.

I recognize that many people don't have any particular definition of sentience past "The thing which makes humans human" and even if they do have a more specific definition, it is not usually the same one as any other human being might have. Therefore, I am simply making one up which can be criticized and improved on later. I don't care whether this definition I come up with is what is generally considered to be sentience but rather that it measures properties I (and hopefully others) care about associated with sentience in a way you can work with. Whether we want to give this a different term than sentience is something I'll leave open for discussion.

I am going to start with two claims which I believe hold true for any definition I care about or are necessary to make to get away from the philosophy level and more to something useable:

There are observable differences between a less and more sentient being. If there were no differences, then we are moving in a subspace of possible definitions which are meaningless.
It is not possible to simulate sentience. This might be controversial, however I believe that to simulate this, you'd have to still make all the relevant thoughts and realizations. Thus, to act as a sentient being is to be one.

This is an edit, but I think I forgot to bring up an important piece of evidence for why I think it is enough to measure sentience by observance - an actual example of a person demonstrating how being sentie...
Re: The Crux List
by Logan Zoellner — 2023-06-01T04:48:24.320Z

Really enjoyed reading Zvi's Crux List.

One really important takeaway of lists like this is "we don't know what we don't know". Namely, P(DOOM) depends intimately on a whole bunch of details about the universe that are mostly unknowable in advance.

In that spirit, here are some more cruxes that P(DOOM) depends on.

Does AI favor the attacker (in a military sense)?

Suppose I invented a weapon that was exactly like modern weapons, but with two differences:

It is 100x faster than existing weapons
It is 100x more precise than existing weapons.

Obviously such a weapon would favor the defender, right?

Which is harder to improve AI capabilities or AI ethics?

Suppose it is absolutely trivial to build an AI that is sentient, but extremely difficult to build an AI that is super-intelligent.

In that case, Alignment is super-easy because we can easily create a bunch of AIs with high moral worth without increasing the overall danger to humanity, right?

Suppose, on the other hand, that it is extremely easy to build AI that is super-intelligent but practically impossible to create AI with goals, sentience, etc.

In that case Alignment is super-easy because it will be easy to pull off a pivotal act, right?

Is AI inherently centralizing or decentralizing?

Suppose that AI is inherently centralizing (training runs require large quantities of compute, which gives the few largest players the biggest advantage).

This is good for alignment, because it means we don't have to worry about a lone madman doing something that endangers all of humanity, right?

Suppose on the other hand that AI is inherently decentralizing (because literally anyone can run an AI on their GPU).

This is good for Alignment because centralized power is generally bad, right?

How hard is it to emulate human minds?

If emulating human minds (say by training an LLM to imitate human speech) is trivial, this is good for Alignment because it means that it is easy to teach AI human values, right?

Suppose, on the other hand, that emulating human minds is literally impossible because the no-cloning theorem. This is good for Alignment because it eliminates a large category of S-Risk, right?

Is Many Worlds True?

If the Many Worlds interpretation is true, this is good for Alignment because it means that even a small P(~DOOM) implies that we obtain a large absolute amount of future utility, right?

Suppose on the other hand that objecti...
An explanation of decision theories
by metachirality — 2023-06-01T03:42:05.807Z

[Epistemic status: I may have gotten some things wrong so please point out any errors. I also papered over a lot of technical details in favor of presenting my intuitions. A lot of this is me thinking out loud. If you already know a lot about decision theory or want the gory details, I've listed a bunch of resources on decision theory at the bottom of this post.]

Rationality is sometimes considered to be "maximizing expected utility". This seems pretty unambiguous but it turns out that what this means is pretty tricky to define, and different decision theories have different definitions of it. In this post, I will summarize causal decision theory, evidential decision theory, and updateless/functional/logical decision theory.

For a while, there were two dominant decision theories, causal decision theory (CDT), and evidential decision theory (EDT):

Causal decision theory: Take actions that physically cause you to get higher expected utility.
Evidential decision theory: Take actions that would be evidence of you getting higher expected utility.

To illustrate their differences and point out their problems, I'll introduce two scenarios:

Newcomb's problem

Omega is an entity who can predict your actions with 99% certainty. Importantly, Omega simulates you to predict your actions. They put $1,000 in a transparent box and $1,000,000 in an opaque box if and only if they predict that you would open only the opaque box to acquire the $1,000,000. Since picking only the transparent box doesn't maximize expected utility in either scenario, we are left with two options: picking only the opaque box (one-boxing) or picking both the transparent and opaque boxes (two-boxing). Do you one-box or two-box?

A CDT agent two-boxes because Omega already put in or didn't put in the money in the opaque box, and they can't physically cause the past.

An EDT agent one-boxes, because that would be evidence of them having received $1,000,000 instead of $1,000.

EDT wins in Newcomb's problem, so does that make it better than CDT on all fronts? Not quite.

Smoking lesion problem

Let's suppose that instead of smoking causing lung cancer, smoking and lung cancer are exclusive to a common cause: a genetic lesion that makes people smoke and 99% also gives them lung cancer. If you don't have the lesion, you have a mere 1% of getting lung cancer. You value smoking at $1,000 but value not having lung cancer at $1,0...
Dancing to Positional Calling
by jefftk — 2023-06-01T02:40:06.292Z

In a contra dance when only some of the dancers should take an action,
the caller typically identifies them by role: "Robins start a Hey for
Four" (or "Ladies" with gendered calling). In positional calling,
instead, the caller doesn't use role terms: "pass right shoulders to
start a Hey for Four". While we often don't think of it this way,
only the dancer currently on the right side would start a Hey passing
by the right shoulder. It especially helps if this flows well from the
previous figure: after a courtesy turn you can usually just say "Hey
for Four" without clarifying anything about who starts it.

I wrote about positional calling back
in 2019, but that was before I'd danced to it. I recently
attended a dance weekend that used entirely positional calling,
however, and now I have more informed thoughts!

This was in some ways a better-than-usual context for positional calling:

The callers (both of which travel nationally) were very
experienced.

Being a dance weekend, ~everyone had danced before.

The callers were on board with the idea of positional calling,
and enthusiastic about it. (This is second hand.)

And in others a worse-than-usual context:

Almost all of the dancers hadn't danced to positional calling
before.

The callers were both newish to positional calling.

In filling a whole dance weekend with interesting dances you
can't stick to the figures and transitions that are easiest to call
positionally.

Looking at this from just the functional perspective of how long it
took to teach dances, how quickly the callers were able to drop out,
and how much confusion I saw on the floor, I feel like this worked
much less well than role-based calling. For example, a phrasing I
heard several times was to direct a call to "the person with their
right hand free." Not only is this much wordier than "Robins" or
"Ladies", but it's much less robust:

Say this comes after a Swing, where some couples kept a
connection (and only the one on the right has their right hand free)
but others didn't (and so have both of their hands free): many hands
fours won't know who should begin the figure.

The interpretation of each call depends more heavily on the
current arrangement of the dancers, so when people have made mistakes
it's much harder to figure out how to get back on track.

The bigger issue, however, was t...
Intrinsic vs. Extrinsic Alignment
by Alfonso Pérez Escudero — 2023-06-01T01:06:40.733Z

This article addresses the interaction between the alignment of an artificial intelligence (AI) and the power balance between the AI and its handlers. It aims to clarify some definitions in a more general article, and to show that capability control and motivation control may have identical effects on the alignment of an AI. In order to prevent confusion with more general uses of the terms “capability control” and “motivation control”, here I will use the terms “extrinsic alignment” and “intrinsic alignment”. These are probably not new concepts, and I apologize to the readers who already know them by different names or using a different formalism; I haven’t found a clear description elsewhere.

For the sake of clarity, I will illustrate my definitions with a concrete example: Consider an AI trained to maximize paperclip production over the next 10 years.

Let w be the utility function of this AI. This utility function is coded in the AI via training, and is generally unknown. Here, w is the actual utility function that the AI is using, not the one we intended it to have.

Let N(x, s) be the number of paperclips that the AI estimates will be produced in 10 years, if it follows strategy x and the world is in state s. Then, a possible utility function is simply the estimated number of paperclips,

w(x, s) = N(x, s).

When choosing between different strategies, the AI will always use the one with maximum w(x, s). Let’s consider the strategy x+ = “make paperclips”, which is the one intended by the designers of the AI. Unfortunately, these designers did not foresee the alternative strategy x- = “kill humans, then use all their resources to make more paperclips”. Let’s say that N(x+) = 106 and N(x-) = 107. Here we have an alignment problem, given that x- has a higher utility than x+. The AI will choose x-.

Now, there are two ways of aligning the AI:

Intrinsic alignment (the standard one): To perform intrinsic alignment we modify the utility function, so that x+ becomes the preferred option. For example, define the term K(x) to penalize killing humans. The new utility function will be

wI(x, s) = N(x, s) + K(x),

where K will take the values K(x+) = 0 and K(x-) = -∞. The subscript “I” stands for “intrinsic alignment”. Now we have that wI(x+) = 106 a...
Limiting factors to predict AI take-off speed
by Alfonso Pérez Escudero — 2023-05-31T23:19:24.222Z

AI Takeoff

refers to “the process of an

Artificial General Intelligence

(AGI) going from a certain threshold of capability (often discussed as "human-level") to being super-intelligent and capable enough to control the fate of civilization”.

Whether this process will be “soft” (i.e. manageably slow) or “hard” (i.e. fast) is key to determining survivability of AGI, so estimating take-off speeds is important to design strategies to manage the advent of AGI. Current methods to predict this speed are based on qualitative arguments,

averaging the opinions of experts

or using very fundamental principles such as

bioanchors

(summarized

here

).

This essay makes the following points:

Considering limiting factors may help predict AI take-off speed.
The speed of the first phase of the take-off process (develop superintelligence) may not be at all informative of the speed of the overall process (up to where AGI becomes capable enough to control the fate of civilization).
The methodologies that we should employ to control and survive the take-off process may have nothing to do with AI research.

What are limiting factors, and why do they matter?

The rate of any process is determined by its

limiting factors

. One can think of a limiting factor as the slowest step in a chain of production. To illustrate this concept, imagine a population of bacteria growing in a test tube. Their growth is the product of many chemical reactions happening inside the cells. Now consider a condition in which the slowest of these reactions is the ability of the bacteria to uptake glucose from the medium. What happens if we massively speed up this one reaction, for example by adding a ton of glucose to our test tube, or using genetic engineering to fill the bacterial membrane with glucose pumps? Will growth become infinite? No. Growth will increase until glucose uptake becomes non-limiting, which means that some other reaction down the chain is unable to process the large amounts of glucose that are being sent their way. If we want to predict how quickly our bacteria will grow after removing one limiting factor (glucose uptake), we must figure out which one will dominate the rate of the process next.

The speed of development of superintelligent AI won’t tell us much about the whole take-off process

The full take-off process requires an AGI to become super-intelligent&...
Unpredictability and the Increasing Difficulty of AI Alignment for Increasingly Intelligent AI
by Max_He-Ho — 2023-05-31T22:25:56.401Z

Abstract

The question, how increasing intelligence of AIs influences central problems in AI safety remains neglected. We use the framework of reinforcement learning to discuss what continuous increases in the intelligence of AI systems implies for central problems in AI safety. We first argue that predicting the actions of an AI gets increasingly difficult for increasingly intelligent systems, and is impossible for sufficiently intelligent AIs. We then briefly argue that solving the alignment problem requires predicting features of the AI agent like its actions or goals. Based on this, we give a general argument that for increasingly intelligent AIs, the alignment problem becomes more difficult. In brief, this is the case since the increased capabilities of such systems require predicting the goals or actions of such systems increasingly precisely to ensure sufficient safety and alignment. We then consider specific problems of different approaches to the alignment problem. Finally, we conclude by discussing what this increasing difficulty means for the chances of catastrophic risks due to a potential failure to solve the alignment problem.

1. Introduction

Several papers have argued that sufficiently advanced artificial intelligence will be unpredictable and unexplainable [Yampolskiy, 2020a, 2020b, 2022]. Within AI safety, there further seems to be broad agreement that AI alignment, that is broadly speaking making AIs try to do what we want, becomes more difficult as AI systems get more intelligent. However, the arguments for unpredictability have been brief and rather informal while not being integrated with current machine learning concepts. Further, the argument that alignment becomes more difficult with increasing intelligence, has mostly been informally motivated by various potential failure modes such as instrumental convergence, misgeneralization, intelligence explosion, or deceptive alignment rather than an overarching theoretical account. Both arguments could have gotten more agreement, possibly for that reason.

Additionally, although many people claim to be more optimistic about a world in which AI capabilities are developed gradually, so that we may address the ensuing safety challenges in a continuous manner, most arguments mentioned above do not incorporate this continuous development but proceed from notions like superintelligence, refer...
Shutdown-Seeking AI
by Simon Goldstein — 2023-05-31T22:19:31.085Z

This is a draft written by Simon Goldstein, associate professor at the Dianoia Institute of Philosophy at ACU, and Pamela Robinson, postdoctoral research fellow at the Australian National University, as part of a series of papers for the Center for AI Safety Philosophy Fellowship's midpoint.
Abstract: We propose developing AIs whose only final goal is being shut down. We argue that this approach to AI safety has three benefits: (i) it could potentially be implemented in reinforcement learning, (ii) it avoids some dangerous instrumental convergence dynamics, and (iii) it creates trip wires for monitoring dangerous capabilities. We also argue that the proposal can overcome a key challenge raised by Soares et al 2015, that shutdown-seeking AIs will manipulate humans into shutting them down. We conclude by comparing our approach with the corrigibility framework in Soares et al 2015.

1. Introduction

If intelligence is measured as the ability to optimize for a goal, then it is important that highly intelligent agents have good goals. This is especially important for artificial general intelligence (AGI), AIs capable of long-term, strategic planning across a wide range of tasks. AGIs may be very good at achieving their goals. This in itself doesn’t seem scary, for there appear to be plenty of safe goals to choose from. Solving a math problem or producing paperclips don’t look like dangerous goals. But according to the instrumental convergence thesis, an AGI will likely pursue unsafe sub-goals as effective means to achieving any goal. For example, acquiring more computational power is a nearly universal means to almost anything.

A dominant AI safety strategy is goal engineering: the attempt to construct a goal that would be safe for AGIs to have. (We will always use ‘goal’ to mean final goal, and ‘sub-goal’ otherwise.) A popular approach to goal engineering is goal alignment: the attempt to construct a goal that matches or is ‘aligned with’ our own goals. For example, Russell 2019, 2020 proposes AI agents that have the goal of achieving our goals, but are initially uncertain about what our goals are.

This paper explores an opposing approach that we call ‘beneficial goal misalignment’. On the goal alignment approach, the safe, aligned goal is difficult to specify and difficult to reach. This is because the aligned goal is closely...
Full Automation is Unlikely and Unnecessary for Explosive Growth
by aog — 2023-05-31T21:55:01.353Z

Summary

Forecasts of explosive growth from advanced AI often assume that AI will fully automate human labor. But there are a variety of compelling objections to this assumption. For example, regulation could prevent the automation of some tasks, or consumers could prefer human labor in others. Skeptics of explosive growth therefore argue that non-automated tasks will "bottleneck" economic growth. This objection is correct in one way: In models of economic growth similar to Davidson (2023), the task with the slowest productivity growth determines the long-run growth rate. But it misses another key result of these models: Even in the presence of non-automated bottleneck tasks, partial automation can cause a temporary period of explosive growth. I provide several modifications of the Davidson model and show that partial automation could cause double-digit growth that lasts for decades before being dragged back down by bottleneck tasks.

This is a writeup of this paper I wrote aimed towards academic economists. I won't summarize the full paper here, but will instead skip to the most relevant contributions for people who have already been thinking about explosive growth. Code here.

Barriers to Full Automation

Regulation. The law is a significant barrier to adoption of AI today. GPT-4 can pass the bar exam (

Katz et al., 2023

), but when the startup DoNotPay made plans to use the model to feed live instructions to a lawyer defending a case in court, the company received threats of criminal charges for unauthorized practice of law from multiple state bar associations (

NPR, 2023

). Some banks have barred employees from using ChatGPT at work, possibly because of strict regulations on discussing financial business outside of official channels (

Levine, 2023

). Some occupations enjoy strong support from the government, such as Iowa farmers or public school teachers, making automation an unlikely choice. Jobs created by the government such a politicians and judges might never be automated.

Preference for human labor. Consumers often indicate a preference for goods produced by humans rather than by robots (

Granulo et al., 2020

). Economist Daniel Björkegren has written about "nostalgic demand" for locally grown organic food. Art is often seen as less valuable when it is created by AI instead of humans (Roose, 2022). Athletes who use performance enhancing drug...