trentmkelly/LessWrong-43k huggingface dataset:
Evan Hubinger on Homogeneity in Takeoff Speeds, Learned Optimization and Interpretability
by Michaël Trazzi — 2021-06-08T19:20:25.977Z

Below is the transcript of my chat with Evan Hubinger, interviewed in the context of the inside view a podcast about AI Alignment. The links below will redirect you to corresponding timestamps in the youtube video.

Outline:

Evan's background @ MIRI & OpenAI

Coconut & functional programming (repo)

Homogeneity in AI takeoff (AF post: homogeneity in AI takeoff scenarios)

Reproducing SoTA & openness in multipolar scenarios

Quantilizers & operationalizing strategy stealing (AF post: operationalizing compatibility with strategy stealing)

Risks from learned optimization & evolution (paper)

Learned optimization in Machine Learning (green arrow env, chest/key env)

Clarifying Inner AI Alignment terminology (AF post: clarifying inner alignment terminology)

Transparency & Interpretability (AF post: Chris Olah's view on AGI Safety)

11 proposals for safe advanced AI (AF post: an overview of 11 proposals for building safe advanced AI)

Underappreciated problems in AI Alignment & surprising advances in AI

Michael: To give a bit of context to the viewers, MIRI is the Machine Intelligence Research Institute, can you just like give a brief recap on like how long have you been working there and what you do there and what have you been doing before, like in the past few years.

Evan: Yeah, so I work at MIRI, I'm a research fellow there, I work on broadly, to sort of very broadly, I think about inner alignment, which is sort of the problem of how do we align the models that we train with the sort of objectives that we're trying to train them on? I tend to think about this sort of problem from a prosaic perspective, from the perspective of thinking about concrete machine learning systems, also from a theoretical perspective, so trying to think about machine learning systems and understand what they're doing using sort of more theoretical tools and abstract reasoning rather than sort of concrete experiments, so that sort of broadly what I work on and what I think of.

Michael: You're talking about empirical work, so I remember when I first learned about you, it was because I was at a conference in 2018 with Vladimir Mikulik and it was just after the MIRI summer fellows and you guys were writing the mesa-alignment paper. It was mostly theoretical. And then I think you worked at the OpenAI, on theory there, but the whole company was more, you know, experim...
Survey on AI existential risk scenarios
by Sam Clarke — 2021-06-08T17:12:42.026Z

Cross-posted to the EA forum.

Summary

In August 2020, we conducted an online survey of prominent AI safety and governance researchers. You can see a copy of the survey at this link.[1]

We sent the survey to 135 researchers at leading AI safety/governance research organisations (including AI Impacts, CHAI, CLR, CSER, CSET, FHI, FLI, GCRI, MILA, MIRI, Open Philanthropy and PAI) and a number of independent researchers. We received 75 responses, a response rate of 56%.

The survey aimed to identify which AI existential risk scenarios[2] (which we will refer to simply as “risk scenarios”) those researchers find most likely, in order to (1) help with prioritising future work on exploring AI risk scenarios, and (2) facilitate discourse and understanding within the AI safety and governance community, including between researchers who have different views.

In our view, the key result is that there was considerable disagreement among researchers about which risk scenarios are the most likely, and high uncertainty expressed by most individual researchers about their estimates.

This suggests that there is a lot of value in exploring the likelihood of different AI risk scenarios in more detail, especially given the limited scrutiny that most scenarios have received. This could look like:

Fleshing out and analysing the scenarios mentioned in this post which have received less scrutiny.

Doing more horizon scanning or trying to come up with other risk scenarios, and analysing them.

At this time, we are only publishing this abbreviated version of the results. We have a version of the full results that we may publish at a later date. Please contact one of us if you would like access to this, and include a sentence on why the results would be helpful or what you intend to use them for.

We welcome feedback on any aspects of the survey.

Motivation

It has been argued that AI could pose an existential risk. The original risk scenarios were described by Nick Bostrom and Eliezer Yudkowsky. More recently, these have been criticised, and a number of alternative scenarios have been proposed. There has been some useful work exploring these alternative scenarios, but much of this is informal. Most pieces are only presented as blog posts, with neither the detail of a book, nor the rigour of a peer-reviewed publication. For further discussion of this dynamic, see work by Ben Garfinkel,...
What are the gears of gluten sensitivity?
by Gordon Seidoh Worley — 2021-06-08T16:42:52.189Z

I've been having some weird health issues somehow triggered after eating. I've been working with a nutritionist. They recommend, among other things, trying to eliminate gluten some of the time on the idea that gluten puts stress on the digestive system and can expose other issues that the body would normally handle. This is something like a load-bearing model: the body bears the load of bad stuff coming in without issue up until it is overloaded and then you start to get weird symptoms as various systems strain to keep up with the load of bad stuff. The proposal is that gluten is one such bad stuff.

It's hard to find anything useful on Google that might give me a good model of how gluten sensitivity would work or why gluten might matter in such a load-bearing model (or, stepping back, that such a load-bearing model is even reasonable).

On LW I only found this old post, which is useful but not enough detail.

So, does anyone have a gears-level model of gluten sensitivity? Links or explanations would be helpful as I try to evaluate what interventions might and might not be worth trying.
The Generalized Product Rule
by aysajan — 2021-06-08T16:39:05.029Z

Imagine we have a company with investment projects A, B, C,....For instance, A might be a new high-speed Internet service, B might be a new advanced computer, C might be a new inventory management software, etc. We are interested in calculating the total return from these investments at the company. This calculation could be fairly complicated since returns are context-dependent - e.g., new computer B might have higher return in the context of new Internet service A than it would without the new Internet service. But let’s assume that the returns satisfy a few reasonable properties.

The total return can be calculated from the return of each individual project given projects before it - e.g., the return of Internet service alone, the computer given the Internet service, the software given the computer and internet, etc.
If the return of one project increases (given projects before it) while everything else stays the same, then the total return increases. For instance, if the Internet service gets cheaper, then the return of project A should increase with everything else the same. As a result, the overall return should increase.
We can group projects into subprojects without changing the overall return. For instance, we could think of the Internet service and computer as a single project, or we could think of the computer and software as a single project, and either way the total return should stay the same.

Surprisingly, given just these three properties, we can conclude that returns obey a “product rule” similar to the product rule in probability theory.

w[R(A,B)]=w[R(A)]w[R(B|A)]

where w is some transformation of returns (e.g., it could be log-return, return-squared, etc.)

This is essentially the first step in

Cox’s Theorem

, a theorem used (most notably by

Jaynes

) to ground the logicalist interpretation of probability. But as this post will illustrate, core ideas of Cox’s Theorem apply to many real-world systems which we don’t usually think of as “probability theory”.

Let’s unpack those assumptions a bit more for our investment return example by defining explicit variables on projects and returns. The three key properties are:

Return R(A,B) of A and B together can be computed from the return R(A) of A alone and the return R(B|A) of B given A is done. For instance, the return on new high-speed Internet service and new com

...
Will outlets like the NYT be captured by Chinese influence and if so, when?
by ChristianKl — 2021-06-08T16:31:09.244Z

John Cena apology for suggesting that Taiwan is a country by saying “Taiwan is the first country that can watch F9” illustrates how China has immense power over Hollywood. As China's influence growth in the world they will try to exert influence on more Western institutions. Should we expect outlets like the NYT to change in a way where calling China a country is a bad career move? If not, what will stop China from expanding it's influence in such a way that it has similar influence over the NYT as it currently has over Hollywood? It it does happen, how long will it take?
The reverse Goodhart problem
by Stuart_Armstrong — 2021-06-08T15:48:03.041Z

There are two aspects to the Goodhart problem which are often conflated. One is trivially true for all proxy-true utility pairs; but the other is not.

Following this terminology, we'll say that V is the true goal, and U is the proxy. In the range of circumstances we're used to, U≈V - that's what's makes U a good proxy. Then the Goodhart problem has two aspects to it:

Maximising U does not increase V as much as maximising V would.

When strongly maximising U, V starts to increase at a slower rate, and ultimately starts decreasing.

Aspect 1. is a tautology: the best way to maximise V is to... maximise V. Hence maximising U is almost certainly less effective at increasing V than maximising V directly.

But aspect 2. is not a tautology, and need not be true for generic proxy-true utility pairs (U,V). For instance, some pairs have the reverse Goodhart problem:

When strongly maximising U, V starts to increase at a faster rate, and ultimately starts increasing more than twice as fast as U.

Are there utility functions that have anti-Goodhart problems? Yes, many. If (U,V) have a Goodhart problem, then (U,V′) has an anti-Goodhart problem if V′=2U−V.

Then in the range of circumstances we're used to, U=2U−U≈2U−V=V′. And, as V starts growing slower than U, V′ starts growing faster; when V starts decreasing, V′ starts growing more than twice as fast as U:

Are there more natural utility functions that have anti-Goodhart problems? Yes. If for instance you're a total or average utilitarian, and you maximise the proxy "do the best for the worst off". In general, if V is your true utility and U is a prioritarian/conservative version of V (eg U=−e−V or U=log(V) or other concave, increasing functions) then we have reverse Goodhart behaviour[1].

So saying that we expect Goodhart problems (in the second sense) means that we know something special about V (and U). It's not a generic problem for all utility functions, but for the ones we expect to correspond to human preferences.

We also need to scale the proxy so that V≈U on the typical range of circumstances; thus the conservatism of U is only visible away from the typical range. ↩︎
Supplement to "Big picture of phasic dopamine"
by Steven Byrnes — 2021-06-08T13:08:00.832Z

This is "Supplementary Information" to my post "Big Picture of Phasic Dopamine".

It's a deeper dive into more specific differences between what I’m currently thinking, vs Randall O’Reilly & colleagues' PVLV model, vs Steve Grossberg & colleagues' MOTIVATOR model.

Unlike the main post, I don't expect this to be of any interest to non-neuroscientists, and therefore I'm going to freely use neuroscience jargon.

1. Differences between what I'm thinking and Randall O’Reilly et al.’s PVLV model

Background: the PVLV (“Primary Value, Learned Value”) model started with

this 2007 paper

by Randall O’Reilly et al. They refined the model in

this 2010 paper

, and then the most recent version is

Mollick

et al.

2020

, a 50-page tour de force that I’ve spent many many hours lovingly poring over while writing the main post. That paper influenced my thinking in too many ways to list—for example, that’s definitely where I first heard the idea that the amygdala has different “zones” for different USs, and that the ventromedial prefrontal cortex does too, with apparent one-to-one correspondence. (I think it should be “different zones for different URs” not “different zones for different USs”, but that’s a minor point.) And also …, well, just look at everywhere I cited it in my post.

So, great paper. But to make progress it’s worth spelling out the areas of disagreement.

I guess my first main disagreement is related to heterogeneous dopamine (different dopamine neurons are doing different things at the same time). Mollick discusses heterogeneous dopamine very helpfully and at length, but AFAICT it’s not part of her actual model. This creates various sorts of awkwardness, I think. First, their one reward signal needs to support learning in the amygdala, despite the fact that the amygdala is doing processing related to various reactions that have both “valences” (e.g. cringing is generally bad, relaxing is generally good). She solves this by putting different dopamine receptors onto the different reaction-learning circuits, i.e. hardcoding the idea that cringing is always something you do in response to bad things happening, relaxing is always something you do in response to good things happening, etc. But in many cases, the same UR to the same US can be good or bad depending on physiological context. My favorite example is the autonomic reaction “salivating-in-anticipation-of-salt”—...
Big picture of phasic dopamine
by Steven Byrnes — 2021-06-08T13:07:43.192Z

(Update Jan. 2023: This article has important errors. I am leaving it as-is in case people want to see the historical trail of me gradually making progress. Most of the content here is revised and better-explained in my later post series Intro to Brain-Like AGI Safety, see especially Posts 5 and 6. Anyway, I still think that this post has lots of big kernels of truth, and that my updates since writing it have mostly been centered around how that big picture is implemented in neuroanatomy. More detail in this comment.)

Target audience: Everyone, particularly (1) people in ML/AI, and (2) people in neuroscience. I tried hard to avoid jargon and prerequisites. You can skip the parts that you find obvious.

Context: I’m trying to make sense of dopamine in the brain—and decision-making and motivation more generally. This post is me playing with ideas; expect errors and omissions (and then tell me about them!).

This post is a bit long; I’m worried no one will read it. So in a shameless attempt to draw you in, here’s a Drake meme...

This will make more sense later in the post. Yes of course the bottom one is oversimplified, and yes of course the top one is useful too. But I claim that the bottom one is a better starting point.

(Thanks Adam Marblestone, Trenton Bricken, Beren Millidge, Connor Leahy, Jeroen Verharen, Ben Smith, Adam Shimi, and Jessica Mollick for helpful suggestions and criticisms.)

Summary / Table of Contents

I’ll start by briefly reviewing the famous 1990s story relating the role of dopamine in the brain to the “

TD learning

” algorithm.
Then I’ll switch over to the “cortico-basal ganglia-thalamo-cortical loop” (“loop” for short), a circuit intimately related to dopamine learning and inference.

I’ll go over a theory that the entire

telencephalon

region of the brain (which includes neocortex, hippocampus, amygdala, basal ganglia, and more) has a coherent overall architecture, with these loops being the key unifying element, and I’ll relate that idea to some bigger-picture ideas about within-lifetime learning vs instinctive behaviors.
I’ll offer a toy model of how the loops work.

I’ll suggest that there are three more specific categories of what these loops are doing in the brain, with different dopamine signals in each case:

Reinforcement learning loops, where the “reward function” is the thing we all expect (yummy food is good, electric shocks are bad, etc.);
Reinf

...
Against intelligence
by George3d6 — 2021-06-08T13:03:49.838Z
Link: https://cerebralab.com/Against_intelligence

Two years ago I wrote some pragmatic arguments that “human-like AI” is hard to develop and would be fairly useless. My focus was on the difficulty of defining a metric for evaluation and the cost-effectiveness of human brains.

But I think I failed to stress another fundamental point, which is that "intelligence" as commonly thought of may not be critical to acquiring knowledge about or power to change the reality external to our own body.

I – Conceptually intelligent

If I’m allowed to psychoanalyse just a tiny bit, the kind of people that think a lot about “AI” are the kind of people that value conceptual intelligence too much, because that's the best part of their thinking.

A rough definition of conceptual is: the kind of thinking that can easily be put into symbols, such as words, code or math. I’m making this distinction because there are many networks in the brain that accomplish unimaginably complex (intelligent) tasks, which are very nonconceptual. Most people that think about “AI” don’t view these functions as “worth imitating” (rightfully so in most cases).

There’s a ton of processing dedicated to going from “photovoltaic activation of retinal neurons” to “conscious experience of seeing”. But we use digital cameras to record the world for computer vision, so we don’t have to imitate most of the brain processes involved in sight.

I should make a note here that everything in the brain is interconnected. We don’t see a world of light and shadow, we see a world of objects. Conceptual thinking gets involved in the process of seeing at some point, detects various patterns, and overlays concepts over them to focus and enrich perception. Similarly, hearing, smell and memory all play a role in “seeing”.

But I think it’s permissible to think of an abstract “conceptual thinking” complex in the brain, which “AI” is trying to replicate, and treat the other bits, such as senses, motor control or homeostasis maintenance as separate from this, a prerequisite to run the platform that allows for conceptual thinking. I realize this is a naive dualist view that kind of puts conceptual thinking on a pedestal, but it's useful for conveying some ideas.

That being said, it’s important to note that conceptual thinking in humans does have access to the rest of the brain. Language can query the limbic system and say something like “I am feeling sad”. This is a very narrow abstraction ...
Game-theoretic Alignment in terms of Attainable Utility
by midco — 2021-06-08T12:36:07.156Z

Acknowledgements:

This article is a writeup of research conducted through the SERI program under the mentorship of Alex Turner. It extends our research on game-theoretic POWER and Alex's research on POWER-seeking.

Thank you to Alex for being better at this than I am (hence mentorship, I suppose) and to SERI for the opportunity to conduct this research.

Motivation: POWER-scarcity

The starting point for this post is the idea of POWER-scarcity: as unaligned agents grow smarter and more capable, they will eventually compete for power (as a convention, "power" is the intuitive notion while "POWER" is the formal concept). Much of the foundational research behind this project is devoted to justifying that claim: Alex's original work suggests POWER-seeking behavior and in particular catastrophic risks associated with competition for POWER, while our previous project formalizes POWER-scarcity in a game-theoretic framework.

One of the major results of our previous project was a proof that POWER is scarce in the special case of constant sum games. Additionally, we had a partial notion of "POWER isn't scarce by the definition we care about" for common-payoff games. We interpret these results as limiting cases of a more general relationship between "agent alignment" and POWER-scarcity:

In a common-payoff game, players are "maximally aligned" in the sense that their incentives are identical (in the VNM sense of identical preference orderings between states). We don't have a clean expression for this in terms of POWER; the simplest relevant consequence is "the action profile that maximizes the common payoff is individually optimal for each player simultaneously". We present a more natural characterization later in the post.
In a constant-sum game, players are "maximally unaligned" in the sense that there's no concept of collective preferences: in the utilitarian sense, the group is ambivalent between all outcomes of the game. We proved that in a constant-sum game, all Nash Equilibrium strategy profiles have constant sum POWER.

Given these results, we hypothesize that a more general relationship exists between agent alignment and POWER-scarcity. However, it's unclear how to define "agent alignment" as a property of an arbitrary multiplayer game; the limiting cases only capture narrow intuitions about obviously (un)aligned agents.

This presented a clear set of research goals mo...
Dangerous optimisation includes variance minimisation
by Stuart_Armstrong — 2021-06-08T11:34:04.621Z

Let's look again at Stuart Russell's quote:

A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable.

It is not immediately obvious that this is true. If we are maximising (or minimising) some of these k variables, then it's likely true, if this maximisation or minimisation brings them to unusual values that we wouldn't encounter "naturally". But things might be different if the variables need only be set to certain "plausible" values.

Suppose, for example, that an AI is building widgets, and that it is motivated to increase widget production w. It can choose the following policies, with the following consequences:

π1: build widgets the conventional way; w=10.

π2: build widgets efficiently; w=15.

π3: introduce new innovative ways of building widgets; w=30.

π4: dominate the world's widget industry; w=10,000.

π5: take over the world, optimise the universe for widget production; w=1060.

If the AI's goal is to maximise w without limit, then the fifth option becomes attractive to it. Even if it just wants to set w to a limited but high value - w=10,000 - it benefits from more control of the world. In short:

The more unusual the values of the variables the AI wants to reach, the more benefit it gets from strong control over the universe.

But what if the AI was designed to set w to 15, or in the range 20−30? Then it would seem that it has lower incentive for control, and might just "do its job", the way we'd like it to; the other variables would not be set to extreme values, since there is no need for the AI to change things much.

Eliezer and Nick and others have made the point that this is still not safe, in posts that I can't currently find. They use examples like the AI taking over the world and building cameras to be sure that it constructed 15 widgets exactly. These scenarios seem extreme as intuitions pumps, to some, so I thought it would be simpler to rephrase this as: moving the variance to unusual values.

Variance control

Suppose that the AI was designed to keep v at 15. We could give it the utility function −(v−15)2, for instance. Would it then stick to policy π2?

Now assume further that the world is not tota...