trentmkelly/LessWrong-43k huggingface dataset:
An artificially structured argument for expecting AGI ruin
by Rob Bensinger — 2023-05-07T21:52:54.421Z

Philosopher David Chalmers asked:

[I]s there a canonical source for "the argument for AGI ruin" somewhere, preferably laid out as an explicit argument with premises and a conclusion?

Unsurprisingly, the actual reason people expect AGI ruin isn't a crisp deductive argument; it's a probabilistic update based on many lines of evidence. The specific observations and heuristics that carried the most weight for someone will vary for each individual, and can be hard to accurately draw out.

That said, Eliezer Yudkowsky's So Far: Unfriendly AI Edition might be a good place to start if we want a pseudo-deductive argument just for the sake of organizing discussion. People can then say which premises they want to drill down on.[1]

In The Basic Reasons I Expect AGI Ruin, I wrote:

When I say "general intelligence", I'm usually thinking about "whatever it is that lets human brains do astrophysics, category theory, etc. even though our brains evolved under literally zero selection pressure to solve astrophysics or category theory problems".

It's possible that we should already be thinking of GPT-4 as "AGI" on some definitions, so to be clear about the threshold of generality I have in mind, I'll specifically talk about "STEM-level AGI",[2] though I expect such systems to be good at non-STEM tasks too.

STEM-level AGI is AGI that has "the basic mental machinery required to do par-human reasoning about all the hard sciences",[3] though a specific STEM-level AGI could (e.g.) lack physics ability for the same reasons many smart humans can't solve physics problems, such as "lack of familiarity with the field".

A simple way of stating the argument in terms of STEM-level AGI is:

Substantial Difficulty of Averting Instrumental Pressures:[4] As a strong default, absent alignment breakthroughs, STEM-level AGIs that understand their situation and don't value human survival as an end will want to kill all humans if they can.
Substantial Difficulty of Value Loading: As a strong default, absent alignment breakthroughs, STEM-level AGI systems won't value human survival as an end.
High Early Capabilities. As a strong default, absent alignment breakthroughs or global coordination breakthroughs, early STEM-level AGIs will be scaled to capability levels that allow them to understand their situation, and allow them to kill all humans if they want.
Conditional Ruin. If it's very l

...
What's wrong with being dumb?
by Adam Zerner — 2023-05-07T18:31:49.218Z

I like to make fun of my dog sometimes for being dumb. Affectionately. I'll say things like "That's not for you dumb dumbs!" when he tries to eat food that will clearly end up giving him a stomach ache and making him nauseous.

A few years ago I did this in front of my girlfriend's mom. She laughed and told me to stop making fun of him. Perhaps due to my instinct to be disagreeable, my response was "What's wrong with being dumb?".

My dog doesn't care about intelligence. He just cares about cuddles, love, being part of the pack, food, naps, and some good sniffs. And I for one endorse those values. So I'll ask the question again: what's wrong with being dumb?

Well, intelligence is a useful characteristic. As a society, by rewarding intelligence with higher social status, we incentivize things like scientific discoveries and new technologies. And by punishing a lack of intelligence — dumbness — we prevent people from injecting bleach into their veins. Or at least reduce the number of times that it happens.

But that's looking at things through a macroscopic, society-oriented lens. What if we zoom in and, to stereotype, look at a dumb farmer named Frank?

Frank is dumb. You can reward him for being smart all you want, but that's just not gonna get him to take any community college classes. He is content with his life. It works for him. He enjoys his job, loves his family, owns his home, and just isn't interested in change.

He does do some dumb things that are instrumentally harmful though. For example, he doesn't believe in medicine and hasn't seen a doctor in 30 years. Seeing a doctor would, let's assume for the hypothetical, extend his life by 10 years. But in practice, you're just not gonna change his mind. So why badger him about it? Why piss him off and frustrate yourself along the way? And similarly, he's not gonna be winning any Nobel Prizes or inventing new technologies, so why push for those community college classes?

We have a confusing situation here. If we look through the macroscopic, society-oriented lens, it probably does make sense to reward intelligence and punish dumbness. But if we zoom in and look at things from an individual level, rarely does it make sense to do any of that.

Maybe it's not confusing though. Maybe there's no dilemma. There are just two different answers. If you have some sort of God's eye view and are able to set the norms of society, ...
Categories of Arguing Style : Why being good among rationalists isn't enough to argue with everyone
by Camille Berger  — 2023-05-07T17:45:41.925Z

tl;dr : We overgeneralize the way we personally come up with concepts. People, notably outside of LW and inside of Academia, argue by referring themselves to diverse and starkly different ways of coming up with concepts, which are different enough for us to misinterpret them. We should not expect people to be receptive to our arguments if we can’t identify how their fundamental conceptualization procedure affects their arguing style.

This post aims to help people in Academia

I think there are two tendencies in the LW and EA communities :

LessWrong and EA tend to sometimes depart a lot from mainstream ideas in academia

When they do care about academia, they tend to have a partial (and, in my opinion, unsatisfactory) view of it.

These two tendencies are presuppositions of the following, and I will not defend them. I also think people on LW and EA should care about academia.

In general, knowing better how academia works is useful for setting up a zeitgeist in which EA and some of its crucial topics will be more accepted. But a more important argument is to know where to go and what words to use when you want to work on an EA-related topic, but cannot afford to do some active AI alignment research early on, and so end up being at academia for about five years at minimum.

An example: The Four Clans

Epistemic status : I’m not a philosopher, I only have a undergraduate diploma in philosophy. What I’m showing here is a heuristic for understanding disagreements between people. I might put it to formal test one day and try to refine it. In the meantime, I suggest you to come up with your own version of this, and not to use this vocabulary as is in your interactions. I invite you to refine it and probe for possible errors. See it more like shores of vast continents containing greater diversity.

Philosophy is a diverse field. In western thought, however, the main dominating discourses usually fall under one of four traditions :

1-Analytics (in the likes of Frege, Russel, Quine, Denett)

2-Phenomenologists (in the likes of Husserl, Heidegger, Sartre, Merleau-Ponty)

3- « german ideology/socio-historical approach/neo-kantians/neo-marxists » (in the likes of Adorno, Foucault, and arguably Habermas)

4-Pragmatists (in the likes of Dewey and John Herbert Mead)

I've had teachers of each of these traditions, there are arguably others, of course.

I insist he...
Self-Administered Gell-Mann Amnesia
by krs — 2023-05-07T17:44:55.839Z

I often make mistakes when solving math problems without due care, whether by skipping too many steps, moving to quickly, or failing to check my work. I am aware of the weaknesses in the human mind when solving logical problems and take steps to mitigate them. By the very nature of math problems, I know when I get a wrong answer. Even if the value of a calculation is unknown, I can still check each logical step and know when I have erred. Yet when reasoning verbally, I don't get this feedback. Even if I have made several complicated steps in an argument, I may feel confident in my conclusions and most of my deeply held positions have never been subject to the scrutiny of being worked out on paper and verified step-by-step. Yet, I know that this is what it takes to really be certain of a mathematical argument and I would never be confident in my solution to a hard math problem that I had not checked. I temporarily forget my own weaknesses when verbalizing an argument.

An extension of this idea is that most people consider themselves worse at math than at reasoning through words, but I think this is misguided -- this is just over confidence due to lack of feedback when one is wrong. A counterpoint is that perhaps humans actually are better at verbal reasoning than mathematical reasoning and so it is reasonable to have more confidence in your verbal abilities. I doubt this is true and if anything the unquantified language we use when dealing with ordinary issues seems more fallible to me. Anecdotally, I have found fatal flaws in my own arguments once I take the time to write them down.

Remember your weaknesses and don't forget to check your work.
Understanding mesa-optimization using toy models
by tilmanr — 2023-05-07T17:00:52.620Z

Overview

Solving the problem of mesa-optimization would probably be easier if we understood how models do search internally
We are training GPT-type models on the toy task of solving mazes and studying them in both a mechanistic interpretability and behavioral context.
This post lays out our model training setup, hypotheses we have, and the experiments we are performing and plan to perform. Experimental results will be forthcoming in our next post.
We invite members of the LW community to challenge our hypotheses and the potential relevance of this line of work. We will follow up soon with some early results[1]. Our main source code is open source, and we are open to collaborations.

Introduction

Some threat models of misalignment presuppose the existence of an agent which has learned to perform a search over actions to effectively achieve goals. Such a search process might involve exploring different sequences of actions in parallel and evaluating the best sequence of actions to achieve some goal.[2]

To deepen our understanding of what it looks like when models are actually performing search, we chose to train simple GPT-2 like models to find the shortest paths through mazes. Maze-solving models provide a tractable and interesting object of study, as the structure of both the problem and solutions is extensively studied. This relative simplicity makes identifying and understanding search through the lens of mechanistic and behavioral experiments much more concrete than working with pre-trained LLMs and more feasible in the context of limited computational resources.

Connections to mesa-optimization

Mesa-optimizers are learned optimizers for an objective that can be distinct from the base-objective. Inner misalignment can occur when the AI system develops an internal optimization process that inadvertently leads to the pursuit of an unintended goal. Models capable of perfoming search are relevant for understanding mesa-optimization as search requires iterative reasoning with subgoal evaluation. In the context of solving mazes, we may hope to understand how mesa-optimization arises and can become "misaligned"; either through the formation of non-general reasoning steps (reliance on heuristics or overfitted goals) or failure to retarget.[3]

Existing literature on search has highlighted the potential for unintended consequences of search in ML systems. One lens of v...
How to have Polygenically Screened Children
by GeneSmith — 2023-05-07T16:01:07.096Z

Polygenic screening is a method for modifying the traits of future children via embryo selection. If that sounds like gobbledygook, then think of it a bit like choosing stats for your baby.

That may sound amazing. It may sound like science fiction. It may even sound horribly dystopian. But whatever your feelings, it is in fact possible. And these benefits are available right now for a price that, while expensive, is within reach for most middle-class families.

On a more serious note, there is limited selection power available with today's technologies, so you will not be able to have a baby Einstein unless you are already a Nobel laureate. But polygenic screening will allow you to decrease your child's risk of common diseases by 10-60%, reduce their risk of mental disorders, and increase their IQ by somewhere between 3 and 8 points. If you are willing to wait a few years, you may be able to increase IQ by up to 13 points. Including the cost of IVF and testing, these benefits are available for between $30k-100k depending on the mother's age, how strong of a benefit you want and what kinds of traits you want to select for.

There has been quite a bit of discussion of this topic on LessWrong and adjacent communities but very little concrete advice for would-be parents who are curious whether the benefits are worth the price, particularly for those who have no other reason to do IVF. The purpose of this post is to fill that gap by addressing costs, potential medical complications, choice of clinic, which labs are best, and how age and infertility diagnosis affect the expected benefits.

This is a long post and I expect most people will not want to read the whole thing. If this is you, please use the section selector in the sidebar to navigate to the section you are most interested in. You may want to simply skip to the section titled "The Benefits of Polygenic Embryo Screening".

Background on IVF

A diagram showing the steps in an IVF cycle.

Source

Wait, what even is polygenic embryo selection?

Embryo selection is all about picking an embryo to (hopefully) turn into a baby. This occurs during the process of In-Vitro Fertilization, or IVF. In the typical IVF cycle, a couple goes into a fertility clinic because they want to have a baby. Usually this is because they’ve been having trouble conceiving naturally, but couples also seek out IVF when they want to do genetic te...
Statistical models & the irrelevance of rare exceptions
by patrissimo — 2023-05-07T15:59:10.677Z

“I don’t care about this instance - I don’t care about any instances! Life is too short to care about anything but the general case!”

Yes, the general case is drawn from instances, I’m saying that we shouldn't get caught up in the details unless they really matter. And if there is a clear statistical generalization, the details matter very little.

A common model might be: “Well, the pattern is that it’s some mix of A & B, with more A as factors X & Y are higher, and more B if they are lower”.

Replying "Z fits better than Y in the mix of variables to map onto the A & B mix" is a correction within the general case frame. Whereas, pointing out "Here are cases F & G that are high A with low X & Y - contradicting your model" - when such cases are very rare - is irrelevant. I find it's usually obvious from the description of F&G that they're extremely rare.

Rare exceptions are irrelevant because almost all models of the real world (not physics) are statistical claims about what’s usually true, not absolute claims about what’s always true. So a rare data point that doesn’t fit is actually *not* a contradiction! Pointing such cases out is just reiterating the tautological fact that statistical models are not absolute, which seems like a total waste of time to me. (Especially if the speaker agrees with the model!)

I’ve noticed this happens more often with careful, intellectually humble thinkers, who often include caveats of the form “But here’s an exception to this strong model I’ve just presented.” I think often they're trying to proactively defend against others pointing out this case. But to me, this is wrongfully falling into the frame that rare exceptions are relevant criticisms or corrections of a statistical model.

So, rather than defending by acknowledging the rare case, I think it’s far better to break the false frame that rare exceptions are counter-examples and pointing them out is a relevant thing that needs to be addressed. Move into the new, correct frame that this is a statistical model - say by asking them if they actually disagree that the suggested pattern fits in the vast majority of cases and thus is statistically true.

I’ve also lately noticed that while many people are abstract systematizers, I find far fewer who are relentlessly meta-seeking, constantly seeking to expand to more general theories over broader domains. Eg I’ve no...
Let's look for coherence theorems
by Valdes — 2023-05-07T14:45:09.348Z

In There are no coherence theorems EJT defines a "coherence theorem" as a theorem that proves maximizing expected utility is the only way to play a game that is not dominated. Ie, any strategy that is not a utility maximisation could be made strictly better. I will make the underlying claim more precise, at least according to my intuition, and prove that it is false in general. I will then discuss what variations of the claim we could think off.

There are no coherence theorems was mostly a list of many pre-existing results, none of which proved the underlying claim. The term "coherence theorem" itself gave rise to some discussions and might have been unfortunate. But I agree that something like the following claim has been floating around, more or less implicitely, in multiple documents I read and discussions I had:

Intuitive claim

Under very little general modelisation assumption, it is in the best interest of any agent playing a game to act as a bayesian utility maximizer. If it is not doing so, the agent could become strictly better by becoming a utility maximizer.

If this is true then I find it important, at least somewhat. And if it is false I also wish to know it.
An issue with this is that as it stands the claim is too vague to be disproven. There was some discussion in EJT's post claiming that the result he sought was a trivial corollary of the complete class theorem. But even then I wasn't sure what the exact claim was.

In this post I will:

Give a more precise model for the kind of games we are speaking about and use it to state a conjecture, relfecting the intuitive claim in the general case.

Disprove this conjecture with a counterexample.

Discuss what alternative precise theorems we might seek to prove or find instead.

LaTEX document with a more formal model

This post will avoid using too much math. For a more formal discussion of the same ideas see the draft LaTEX document.

Iterative game model and conjecture

The conjecture bellow is my attempt at translating the intuitive claim to a more precise framework. I will now state that conjecture and then spend the rest of this section on defining the terms that I use.

Conjecture

In an iterative game, any strategy that is not a utility maximisation is dominated by some other strategy.

We model games with a single player that are played for an infinity of turns. Because I need something to call them, fo...
Graphical Representations of Paul Christiano's Doom Model
by Nathan Young — 2023-05-07T13:03:19.624Z

Paul gives some numbers on AI doom (text below). Here they are in graphical forms, which I find easier to understand. Please correct me if wrong.

Michael Trazzi's Probability Flow Diagram

I really like this one. I can really easily read how he thinks future worlds are distributed. I guess the specific flows are guesses from Paul's model so might be wrong but I think it's fine.

Link to tweet: https://twitter.com/MichaelTrazzi/status/1651990282282631168/photo/1

My probability model version

This is messier, but interactive. You get to see what the chances Paul puts on specific breakpoints are. Do you disagree with any?

Link: https://bit.ly/AI-model-Chrisitaino

Paul's model in text

Link

Probability of an AI takeover: 22%

Probability that humans build AI systems that take over: 15%
(Including anything that happens before human cognitive labor is basically obsolete.)
Probability that the AI we build doesn’t take over, but that it builds even smarter AI and there is a takeover some day further down the line: 7%

Probability that most humans die within 10 years of building powerful AI (powerful enough to make human labor obsolete): 20%

Probability that most humans die because of an AI takeover: 11%
Probability that most humans die for non-takeover reasons (e.g. more destructive war or terrorism) either as a direct consequence of building AI or during a period of rapid change shortly thereafter: 9%

Probability that humanity has somehow irreversibly messed up our future within 10 years of building powerful AI: 46%

Probability of AI takeover: 22% (see above)
Additional extinction probability: 9% (see above)
Probability of messing it up in some other way during a period of accelerated technological change (e.g. driving ourselves crazy, creating a permanent dystopia, making unwise commitments…): 15%
An anthropomorphic AI dilemma
by TsviBT — 2023-05-07T12:44:48.449Z

[Metadata: crossposted from https://tsvibt.blogspot.com/2023/01/the-anthropomorphic-ai-dilemma.html. First completed January 21, 2023.]

Either generally-human-level AI will work internally like humans work internally, or not. If generally-human-level AI works like humans, then takeoff can be very fast, because in silico minds that work like humans are very scalable. If generally-human-level AI does not work like humans, then intent alignment is hard because we can't use our familiarity with human minds to understand the implications of what the AI is thinking or to understand what the AI is trying to do.

Terms

Here generally-human-level AI means artificial systems that can perform almost all tasks at or above the level of a human competent at that task (including tasks that take a long time).

Here intent alignment means making artificial systems try to do what we, on reflection and all things considered, would want them to do. A system that doesn't have intents in this sense--i.e. isn't well-described as trying to do something--cannot be intent aligned.

Here "how a mind works" points at the generators of the mind's ability to affect the world, and the determiners of [in what direction to push the world] a mind applies its capabilities.

Let A be some generally-human-level AI system (the first one, say).

(Here "dilemma" just means "two things, one of which must be true", not necessarily a problem.)

The anthropomorphy dichotomy

Either A will work internally like humans work internally, or not.

Well, of course it's a spectrum, not a binary dichotomy, and a high-dimensional space, not a spectrum. So a bit more precisely: either A will work internally very much like how humans work internally, or very much not like how humans work internally, or some more ambiguous mix.

To firmly avoid triviality: to qualify as "very much like" how humans work, in the sense used here, it is definitely neither necessary nor sufficient to enjoy walking barefoot on the beach, to get angry, to grow old, to have confirmation bias, to be slow at arithmetic, to be hairy, to have some tens of thousands of named concepts, to be distractible, and so on. What's necessary and sufficient is to share most of the dark matter generators of human-level capabilities. (To say that, is to presume that most of the generators of human-level capabilities aren't presently understood explicitly; really the condi...