trentmkelly/LessWrong-43k huggingface dataset:
Contra Yudkowsky on AI Doom
by jacob_cannell — 2023-04-24T00:20:48.561Z

Eliezer Yudkowsky predicts doom from AI: that humanity faces likely extinction in the near future (years or decades) from a rogue unaligned superintelligent AI system. Moreover he predicts that this is the default outcome, and AI alignment is so incredibly difficult that even he failed to solve it.

EY is an entertaining and skilled writer, but do not confuse rhetorical writing talent for depth and breadth of technical knowledge. I do not have EY's talents there, or Scott Alexander's poetic powers of prose. My skill points instead have gone near exclusively towards extensive study of neuroscience, deep learning, and graphics/GPU programming. More than most, I actually have the depth and breadth of technical knowledge necessary to evaluate these claims in detail.

I have evaluated this model in detail and found it substantially incorrect and in fact brazenly naively overconfident.

Intro

Even though the central prediction of the doom model is necessarily un-observable for anthropic reasons, alternative models (such as my own, or moravec's, or hanson's) have already made substantially better predictions, such that EY's doom model has low posterior probability.

EY has espoused this doom model for over a decade, and hasn't updated it much from what I can tell. Here is the classic doom model as I understand it, starting first with key background assumptions/claims:

Brain inefficiency: The human brain is inefficient in multiple dimensions/ways/metrics that translate into intelligence per dollar; inefficient as a hardware platform in key metrics such as thermodynamic efficiency.

Mind inefficiency or human incompetence: In terms of software he describes the brain as an inefficient complex "kludgy mess of spaghetti-code". He derived these insights from the influential evolved modularity hypothesis as popularized in ev pysch by Tooby and Cosmides. He boo-hooed neural networks, and in fact actively bet against them in actions by hiring researchers trained in abstract math/philosophy, ignoring neuroscience and early DL, etc.

More room at the bottom: Naturally dovetailing with points 1 and 2, EY confidently predicts there is enormous room for further software and hardware improvement, the latter especially through strong drexlerian nanotech.

That Alien mindspace: EY claims human mindspace is an incredibly narrow twisty complex target to hit, whereas the space

...
Consequentialism is in the Stars not Ourselves
by DragonGod — 2023-04-24T00:02:39.803Z

Polished from my shortform.

Epistemic Status

Thinking out loud.

Introduction

I've argued that system wide/total optimisation for an objective function in the real world is so computationally intractable as to be prohibited by the laws of physics of our universe[1]. Yet it's clearly the case that e.g., evolution is optimising for inclusive genetic fitness (or perhaps patterns that more successfully propagate themselves if you're taking a broader view) in such a totalising manner. I think examining why evolution is able to successfully totally optimise for its objective function would be enlightening.

Using the learned optimisation ontology, we have an outer selection process (evolution, stochastic gradient descent, etc.) that selects intelligent systems according to their performance on a given metric (inclusive genetic fitness and loss respectively).

Optimisation

Behavioural (Descriptive) Optimisation

I think of behavioural optimisation as something along the general lines of:

Navigating through a state space to improbable regions that are extremal values of some compactly specifiable (non-trivial) objective function[2].

Mechanistic (Prescriptive) Optimisation

I think of mechanistic optimisation as something along the general lines of:

a procedure that internally searches through an appropriate space for elements that maximise or minimise the value of some objective function defined on that space].

"Direct" optimisation in the ontology introduced by @beren.

Notably, the procedure must actually evaluate[3] the objective function (or the expected value thereof) on elements of the search space.

Mechanistic optimisation is implementing an optimisation algorithm.

For the rest of this post — unless otherwise stated — I'll be using "optimisation"/"optimising" to refer to "mechanistic optimisation".

"Scope" of Optimisation[4]

I want to distinguish optimising systems according to the "scope" of the optimisation procedure(s) in the system's policy[5].

"Partial" (Task Specific) Optimisation

Involves deploying optimisation (search, planning, etc.) to accomplish specific tasks (e.g., making a good next move in chess, winning a chess game, planning a trip, solving a puzzle).
The choice of particular tasks is not determined as part of this framework; tasks could be subproblems of another optimisation problem (e.g., picking a good next

...
Are there AI policies that are robustly net-positive even when considering different AI scenarios?
by Noosphere89 — 2023-04-23T21:46:40.952Z

One thing I might have noticed recently is a lot of governance on AI might require specific models of AI in society, especially over misalignment or misuse concerns.

Are there AI policies that could be robustly-net positive without having to tie it to a specific scenario?
Getting Started With Naturalism
by LoganStrohl — 2023-04-23T21:02:25.988Z

How Does a Naturalist Study Begin?

There are many different ways to start. Whichever way(s) you choose, what’s important is that 1) you cultivate a lot of original seeing, and 2) you develop a clear sense of your own curiosity. Any starting point that sets you on that track is a correct starting point for naturalism.

What do I mean by “a clear sense of your own curiosity”?

When dealing with challenges, questions, and observations, there are many ways to regard the world that are useful for different purposes: As something you can control, as a community of which you are a part, as raw material for building things, even as an adversary to overcome. The naturalist stance deemphasizes most ways relating to the world, and regards the world instead as something that may reveal itself to you as you open up to it.

The type of curiosity I’m talking about is a yearning for contact with the territory (or with some part of the territory). When I am curious in this way, and maintain that curiosity, it is not burdensome to engage in patient and direct observation. Rather, I find the practice of naturalism inevitable.

What I have to offer as you start off is not so much a specific recommendation as an incomplete menu of options, all of which attempt to cultivate curiosity and original seeing. In my experience, some people can skip this introductory part altogether, while others need to spend a week or more learning how to deliberately pay attention to anything at all before they even begin to consider their chosen object of study. Even I start my studies in different places at different times. There are many good ways to begin, and I hope you will take the time to find (or make) your own.

Starting Place 1: Skip this whole essay. Go straight to “Locating Fulcrum Experiences”.

I don’t often run into situations where this is really the right move, but it does happen. I’m talking first about the “skip ahead” option here because I hope it’ll make the goal of the other options clearer.

I once talked to a person[1] who had found the idea of “comparative advantage” really useful, and wanted to figure out how it might generalize beyond the domain of economics. “I want to be able to use the concept as a lens,” they told me, “and I want to learn to recognize instances where it applies, more-or-less reflexively.” I could tell that they should probably skip straight to “Locat...
Why do we care about agency for alignment?
by Chris_Leong — 2023-04-23T18:10:23.894Z

Many people believe that understanding "agency" is crucial for alignment, but as far as I know, there isn't a canonical list of reasons why we care about agency. Please describe any reasons why we might care about the concept of agency for understanding alignment below. If you have multiple reasons, please list them in separate answers below.

Please also try to be specific as possible about what our goal is in the scenario. For example:

We want to know what an agent is so that we can determine whether or not a given AI is a dangerous agent

Whilst useful isn't quite as good as:

We have an AI which may or may not have goals aligned with us and we want to know what will happen if these goals aren't aligned. In particular, we are worried that the AI may develop instrumental incentives to seek power and we want to use interpretability tools to help us figure out how worried we should be for a particular system.
We can imagine a scale of such systems according to how it behaves in novel situations:

On the low end of this scale, we have a system that only follows extremely broad heuristics that it learned during training
On the high end of this scale, we have a system that uses general reasoning capabilities to discover new and innovative strategies even if it has never used anything like this strategy before, or seen it in the training data

We can think of such a scale as a concretisation of agency. This scale provides an indication of both:

How dangerous a system is likely to be: though a system with excellent heuristics and very little agency could be dangerous as well
What kind of precautions we might want to take: for example, how much can we rely on our off-switch to disable the agent

It's plausible that interpretability tools might be able to give us some idea of where an agent is on this scale just by looking at the weights. So having a clear definition of this scale could help by clarifying what we are looking for.

In a few days, I'll add any use cases I'm aware of myself that either haven't been covered or that I don't think have been adequately explained by different answers.
Taming the Fire of Intelligence
by Peter Kuhn — 2023-04-23T17:41:42.587Z

A recent article by Max Hellriegel Holderbaum and me raised the worry that AI alignment might be impossible in principle. The article has not gotten the attention it deserves and I suspect there are three main reasons for this. First, it is quite long. Secondly, a huge chunk of it develops an intuition-based argument that is inessential to the article's main point. Third, we inadvisably released it on April fools. In this post I will try to reconcile these flaws and present what I take to be our main point in a more concise fashion.

Very roughly, the original article argued that, first, value-aligning an AI system requires that we are able to predict some aspects of its behavior, and secondly, that there are reasons to believe that such predictions may be impossible for systems that are sufficiently intelligent. As a result, we worry that the alignment paradigm may be an infertile framework for thinking about AI safety. Lets go through these points in order.

Prediction

The first thesis is that to value-align a given AI system we have to be able to predict some aspects of its behavior.

Predicting the behavior of complex computational systems is hard, and in many instances impossible if by prediction we mean knowing the result of a computational procedure without running it. The impossibility of making such predictions reliably arguably is a motivating factor behind the shift from the control problem, the problem of controlling AI systems once they are created, to the alignment problem, the problem of creating AI systems that have goals and values aligned with our own. The emphasis on values frees one from the obligation of making precise prediction about what some AI system will do. If it could be shown however, that knowing the values of some system requires that we are able to predict some aspects of its actions, then the emphasis on alignment would be in vain.

To see that prediction is primary to alignment I will consider three popular ways of thinking about utility functions and show that none of them can justify the emphasis on alignment over prediction. The first view is the dispositionalist one. Here utility functions are conceived as generalizations about the actual dispositions a given system has. Rabbits like carrots in the sense that they tend to eat them. Such an understanding of utility functions is prevalent both in the decision theoretic parad...
Preventing AI Misuse: State of the Art Research and its Flaws
by Madhav Malhotra — 2023-04-23T17:37:48.543Z
Link: https://forum.effectivealtruism.org/posts/u9atkEFDcuMkgipch/preventing-ai-misuse-state-of-the-art-research-and-its-flaws

Context: this post summarises recent research to prevent AI algorithms from being misused by unauthorised actors. After discussing four recent case studies, common research flaws and possible solutions are mentioned.

A real-life motivating problem is how Meta's LLaMa had its parameters leaked online (Vincent, 2023), plausibly enabling actors like hackers to use the model for malicious purposes like generating phishing messages en masse. Still, advanced models could have more severe and widespread consequences if stolen, "jailbroken," or otherwise misused.

Summary

Currently, the most common solution to prevent misuse is managing access to AI models with secure APIs. This is desirable, however APIs have flaws:

APIs may be used as "bandaid" solutions to reactively add security after a model is trained. Ideally, security would be considered before training.
Some "real-time" use cases like self-driving cars are not suitable for API-based protection due to time delays in network requests.
Once a model's parameters are leaked, APIs no longer offer protection.

Thus, researchers aim to add new defences inseparably within AI models' weights by making models' accuracy depend on cryptographic keys. They change models' input data, optimiser functions, weights, activation functions, etc. to do this.
Different techniques have tradeoffs between extra model training, memory to store keys, time to generate predictions, and specialised hardware investments. Also, techniques are evaluated by model accuracy when encrypted vs. decrypted, as well as the difficulty of improving an encrypted model.
Current research mostly focuses on small image datasets and models. Also, interpretability techniques, adversarial training, or formal proofs are rare. This limits confidence in the reliability and scalability of current research. Future research should consider larger datasets, language models, and establishing confidence in model reliability.

Four Case Studies of Recent Research

Preprocessing Input with Secret Keys (Pyone et al., 2020)

This is the simplest technique of the four case studies. The researchers trained a ResNet-18 model on the CIFAR-10 dataset, which has 60,000 32x32 pixel images across ten categories. The researchers then preprocessed the input images by dividing them into square blocks and randomly rearranging the pixels in each block based on their secret key. This technique e...
Could transformer network models learn motor planning like they can learn language and image generation?
by mu_(negative) — 2023-04-23T17:24:08.952Z

Humans are really good at moving in space. We do motor planning and especially error correction much better than existing robot control systems.

Could a transformer model trained on a large library of annotated human movement data serve as a controller for a humanoid robot or robotic limb? My impression is that a movement model might not be useful for directly controlling servos, because human and robot bodies are so different. But perhaps it could improve the motor planning and error correction layer?

The data would presumably take the form of a 3D wireframe model of the limb/body and its trajectory through space, the goal of the movement ("pour water from this cup to that cup") and some rating of success or failure.

I don't have experience in either LLMs/transformer models or robotics so this question might miss some obvious points, but I couldn't get the idea out of my head!
Could a superintelligence deduce general relativity from a falling apple? An investigation
by titotal — 2023-04-23T12:49:44.288Z

Introduction:

In the article/short story “That Alien Message”, Yudkowsky writes the following passage, as part of a general point about how powerful super-intelligences could be:

Riemann invented his geometries before Einstein had a use for them; the physics of our universe is not that complicated in an absolute sense. A Bayesian superintelligence, hooked up to a webcam, would invent General Relativity as a hypothesis—perhaps not the dominant hypothesis, compared to Newtonian mechanics, but still a hypothesis under direct consideration—by the time it had seen the third frame of a falling apple. It might guess it from the first frame, if it saw the statics of a bent blade of grass.

As a computational physicist, this passage really stuck out to me. I think I can prove that this passage is wrong, or at least misleading. In this post I will cover a wide range of arguments as to why I don't think it holds up.

Before continuing, I want to state my interpretation of the passage, which is what I’ll be arguing against.

1. Upon seeing three frames of a falling apple and with no other information, a superintelligence would assign a high probability to Newtonian mechanics, including Newtonian gravity. So if it was ranking potential laws of physics by likelihood, “Objects are attracted to each other by their masses in the form F=Gmm/r2” would be near the top of the list.

2. Upon seeing only one frame of a falling apple and one frame of a single blade of grass, a superintelligence would assign a decently high likelihood to the theory of general relativity as put forward by Einstein.

This is not the only interpretation of the passage. It could just be saying that an AI would “invent” general relativity in terms of inventing the equations just by idly playing around, like mathematicians playing with 57 dimensional geometries or whatever. However, the phrase “under active consideration”, and “dominant hypothesis” imply that these aren’t just inventions, these are deductions. A monkey on a typewriter could write down the equations for Newtonian gravity, but that doesn’t mean they deduced it. What matters is whether the knowledge gained from the frames could be used to achieve things.

I’m not interested in playing semantics or trying to read minds here. My interpretation above seems to be what the commenters took the passage to mean. I’m m...
Endo-, Dia-, Para-, and Ecto-systemic novelty
by TsviBT — 2023-04-23T12:25:12.782Z

[Metadata: crossposted from https://tsvibt.blogspot.com/2023/01/endo-dia-para-and-ecto-systemic-novelty.html. First completed January 10, 2023. This essay is more like research notes than exposition, so context may be missing, the use of terms may change across essays, and the text might be revised later; only the versions at tsvibt.blogspot.com are definitely up to date.]

Novelty can be coarsely described as one of: fitting within a preexisting system; constituting a shift of the system; creating a new parallel subsystem; or standing unintegrated outside the system.

Thanks to Sam Eisenstat for related conversations.

Novelty is understanding (structure, elements) that a mind acquires (finds, understands, makes its own, integrates, becomes, makes available for use to itself or its elements, incorporates into its thinking). A novel element (that is, structure that wasn't already there in the mind fully explicitly) can relate to the mind in a few ways, described here mainly by analogy and example. A clearer understanding of novelty than given here might clarify the forces acting in and on a mind when it is acquiring novelty, such as "value drives".

Definitions

"System" ("together-standing") is used here to emphasize the network of relations between elements of a mind.

These terms aren't supposed to be categories, but more like overlapping regions in the space of possibilities for how novelty relates to the preexisting mind.

Endosystemic novelty (or "basis-aligned" or "in-ontology") is novelty that is integrated into the mind by fitting alongside and connecting to other elements, in ways analogous to how preexisting elements fit in with each other. Endosystemic novelty is "within the system"; it's within the language, ontology, style of thinking, conceptual scheme, or modus operandi of the preexisting mind.

Diasystemic novelty (or "cross-cutting" or "basis-skew" or "ontological shift") is novelty that is constituted as a novel structure of the mind by many shifts in many of the preexisting elements or relations, adding up to something coherent or characteristically patterned. Diasystemic novelty is "throughout the system"; it's skew to the system, cross-cutting the preexisting schemes; it touches (maybe subtly) many elements, many relations, or certain elements that shape much of the mind's activity, hence altering the overall dynamics or character of the system.

Paras...
An Intro to Anthropic Reasoning using the 'Boy or Girl Paradox' as a toy example
by TobyC — 2023-04-23T10:20:48.567Z

I wrote the below about a year ago as an entry into 3blue1brown's "Summer of Math Exposition" contest. It is supposed to be an introduction to anthropic reasoning, based on the ideas in the book 'Anthropic Bias' by Nick Bostrom, that doesn't assume a strong maths background. It uses the well known Boy or Girl Paradox as a toy example to introduce the key ideas. I've not seen this done elsewhere, but I think it's a really nice way to understand the key differences between the competing approaches to anthropic reasoning. Sharing here in case anyone finds it interesting or useful.

The Doomsday Argument

Consider the following claim:

Humans will almost certainly go extinct within 100,000 years or so.

Some may agree with this claim, and some may disagree with it. We could imagine a debate in which a range of issues would need to be considered: How likely are we to solve climate change? Or to avoid nuclear war? How long can the Earth’s natural resources last? How likely are we to colonise the stars? But there is one argument which says that we should just accept this claim without worrying about all of those tricky details. It is known as the “Doomsday Argument”, and it goes like this:

Suppose instead that humans will flourish for millions of years, with many trillions of individuals eventually being born. If this were true, it would be extremely unlikely for us to find ourselves existing in such an unusual and special position, so early on in human history, with only 60 billion humans born before us. We can therefore reject this possibility as too unlikely to be worth considering, and instead resign ourselves to the fact that doomsday will happen comparatively soon with high probability.

To make this argument a little more persuasively, imagine you had a bucket which you knew contained either 10 balls or 1,000,000 balls, and that the balls were numbered from 1 to 10 or 1 to 1,000,000 respectively. If you drew a ball at random and found that it had a number 7 on it, wouldn’t you be willing to bet a lot of money that the bucket contained only 10 balls? The Doomsday argument says we should reason similarly about our own birth rank. That is, the order we appear in human history. Our birth rank is about 60 billion*, so our bucket, the entirety of humanity past, present, and future, is unlikely to be too much bigger than that.

Most people’s initial reaction to this argument is...