trentmkelly/LessWrong-43k huggingface dataset:
Slightly advanced decision theory 102: Four reasons not to be a (naive) utility maximizer
by Jan — 2021-11-23T11:02:38.256Z
Link: https://universalprior.substack.com/p/slightly-advanced-decision-theory

TL;DR: Introduction to decision theory with examples from effective altruism. Resulting optimal strategies differ from "naive" utility maximisation, but match community norms.

Meta: I've also posted this on the EA Forum, but I thought the type of mathematical arguments I'm presenting here might also appeal to the LessWrong community. Hope you enjoy <3

Foundations and Flaws

In high school I learned that there are four fundamental questions in philosophy:

What can I know? (epistemology)

What should I do? (moral philosophy)

What is beauty? (aesthetics)

What? (metaphysics)

Effective altruism is a movement that attempts to link the second question with the first[1]: How can I use the things I know to better do the things I should do?

This leads to an interesting clash of cultures: people interested in "what can I know?" tend to be rather cerebral[2], while "what should I do?" attracts people that are more... cordial[3]? This mix is often intriguing and sometimes sublime. It's a young movement and there are many foundational questions that need figuring out. In a previous post, I promised that I would dig into the foundations and see if everything actually makes sense to me. This post is the first step down that path.

So buckle up, I'll be diving into expected value reasoning and slightly advanced decision theory. In four bite-sized sections that can be enjoyed in sequence or in isolation, I'll provide theoretical footing for a lot of the things that are already practiced in effective altruism anyway, but that do not follow directly from naively applying expected value reasoning. As mentioned above, I'm partially doing this to satisfy my own need for having a solid theoretical basis for my actions. But beyond that, my hope is also that by articulating these arguments and demonstrating how to use computational toy models to understand complicated dynamics I'm providing useful tools for the community.

Average action

But before we can talk about tools, let me set the stage. Feel free to skip this section if you are familiar with the central logic and recommendations for action of effective altruism.

I said in the earlier section that effective altruism combines epistemology with moral philosophy. What does that even mean[4]?

We can make this arbitrarily complicated, but for the sake of the post let us start by picking hedonistic utilitarianism as moral philosophy. H...
Use Tools For What They're For
by DirectedEvolution — 2021-11-23T08:26:19.174Z

Maybe there isn’t and can’t be a simple heuristic you can teach everyone in school or via a PR campaign which will lead to them having making good health decisions in an adversarial information environment, without having any negative effects anywhere else. But you also don’t want people to make bad health decisions. So what do you do? - Scott Alexander, Ivermectin: Much More Than You Wanted To Know

The favorite catchphrase of critics of using ivermectin against COVID-19 isn’t "follow the science" or "believe the experts."

They say "IVERMECTIN IS A

HORSE

DEWORMER!"

And in response, ivermectin supporters say things like:

MAGAA Puppy is factually correct in the two "Me:" statements.

From the FDA's article Why You Should Not Use Ivermectin to Treat or Prevent COVID-19, "The FDA has not authorized or approved ivermectin for use in preventing or treating COVID-19 in humans or animals. Ivermectin is approved for human use to treat infections caused by some parasitic worms and head lice and skin conditions like rosacea."
And here's an example of one of the in vitro studies MAGAA Puppy might have been referring to. I don't know if there are numerous studies, but let's take his word for it for now.

MAGAA Puppy's critics aren't disputing these facts. As we know, the reason they emphasize the "horse" in "horse dewormer" is that some people may have been taking horse-size doses of ivermectin and dying from it.

I am suspicious. The FDA only claims to have "received multiple reports of patients who have required medical attention, including hospitalization, after self-medicating with ivermectin intended for livestock." I see no mention of death on the FDA's web page. But if we're tentatively accepting MAGAA Puppy's claims about "numerous studies," let's also tentatively accept critics' concerns that consumption of veterinary doses of ivermectin is a serious public health concern right now.

Critics of ivermectin as a COVID-19 drug are saying that "large doses are unsafe, and small doses are probably not effective." Supporters are saying that "small doses are safe and FDA-approved for human use, and some studies suggest they may be effective."

I firmly side with the critics. But why? MAGAA Puppy has probably read more studies than I have on ivermectin as a COVID-19 treatment. It wouldn't be a high bar to clear.

I also think that the "it's a HORSE dewormer" line is miss...
[linkpost] Acquisition of Chess Knowledge in AlphaZero
by Quintin Pope — 2021-11-23T07:55:28.539Z

This is a linkpost for: Acquisition of Chess Knowledge in AlphaZero.

Abstract:

What is being learned by superhuman neural network agents such as AlphaZero? This question is of both scientific and practical interest. If the representations of strong neural networks bear no resemblance to human concepts, our ability to understand faithful explanations of their decisions will be restricted, ultimately limiting what we can achieve with neural network interpretability. In this work we provide evidence that human knowledge is acquired by the AlphaZero neural network as it trains on the game of chess. By probing for a broad range of human chess concepts we show when and where these concepts are represented in the AlphaZero network. We also provide a behavioural analysis focusing on opening play, including qualitative analysis from chess Grandmaster Vladimir Kramnik. Finally, we carry out a preliminary investigation looking at the low-level details of AlphaZero's representations, and make the resulting behavioural and representational analyses available online.

I think this work represents a significant improvement in interpretability for reinforcement agents and makes me moderately more optimistic about interpretability in general. However, the paper relies on hand-coded representations of chess concepts to probe the network. I worry that it will be hard to extend such an approach to more realistic or fuzzier domains where we can't hand code concepts reliably. Section 7 does contain some results for unsupervised concept-level detection, though this wasn't the focus of the paper.
Integrating Three Models of (Human) Cognition
by jbkjr — 2021-11-23T01:06:48.955Z

You may have heard a few things about “predictive processing” or “the global neuronal workspace,” and you may have read some of Steve Byrnes’ excellent posts about what’s going on computationally in the human brain. But how does it all fit together? How can we begin to arrive at a unified computational picture of human intelligence, and how can it inform our efforts in AI safety and alignment? In this post, I endeavor to take steps towards these ends by integrating insights from three computational frameworks for modeling what’s going on in the human brain, with the hope of laying an adequate foundation for more precisely talking about how the brain cognitively implements goal-directed behaviors:

Predictive Processing: how perception, cognition, and action arise from hierarchical probabilistic generative modeling in the brain

Global Neuronal Workspace: a functional model of access consciousness, i.e. of information that enters our awareness and may subsequently be reported upon

Steve Byrnes’ computational framework: a lot of excellent (in my opinion) work on human cognition, especially motivation, all with an eye toward AI safety. I want to use a better understanding of human cognition to speak more precisely about how things like “goals” and “desires” are implemented, and Steve’s overall framework overlaps quite naturally with the previous two, so I couldn’t leave this one out!

This is a somewhat long post, so I will go ahead and flag the structure here, in case some readers want to skim and/or skip to sections that sound most interesting to them:

Motivations for this project

An overview of each of the three frameworks

Here, I tried to assume fairly little previous familiarity with the frameworks and attempted to provide a decent foundation in each, while making sure to emphasize the aspects most relevant for the purposes of this post.

Fitting things together from the three frameworks

How to think about “predictive processing” in the context of Steve’s framework

GNW in the context of the other two frameworks, especially PP, including a better understanding of how it’s actually implemented

Conclusion

This work was completed as Summer Research Fellow at the Center on Long-Term Risk under the mentorship of Richard Ngo. Thanks to Richard, Steve Byrnes, Alex Fabbri, and Euan McLean for feedback on drafts of this post.

Motivation

In thinking about exist...
Potential Alignment mental tool:
Keeping track of the types
by Donald Hobson — 2021-11-22T20:05:31.611Z

Epistemic status: Came up with what I was thinking of as a notation. Realised that there was a potentially useful conceptual tool behind it.

Consider this diagram.

1) A diagram showing a useless alignment scheme (ignore the computer, make the human do everything.)

Here the green circle on the left represents humans, with V being human values. The W is the external world. The C is within the computer. The W' is a world model. And the V' is the human values in the world model.

Here the computer has been programmed to form a model of the environment, for example Solomonoff induction or GPT3.

Anything in C is stored on the hard drive. Anything in W' is mirroring a section of the real world and is part of an arbitrary Turing machine or incomprehensible neural network.

The black bars are bridges, ways that something can get from one domain to another. (yes this is basically a graph structure.)

The left most bridge is between the human mind and the world. It is made of eyes and muscles.

The next bridge is between the world and the computer. It is made of keyboards and screens.

The next barrier is the interface between the programmed and the learned. The piece of code that forms 1 hot vectors from a string of text and the piece of code that turns the final activations into a choice of next word. The final barrier is the mirror of the first barrier within the simulation. A virtual representation of keyboards and screens, hidden in the weights of a network.

The dotted arrow represents the path that human values take to get into V'.

We want human values to get out into the world. The easiest way to do that is the solid line. It shows a human acting on the environment directly.

2) Another diagram showing a trivial and useless alignment scheme.

In this setup, the data goes into the computer, and straight back out. It could represent the computer trivially echoing the input. It is easy to add numbers, sort them, loop, take a maximum of a list, etc in the computer section C. So diagram 2 could also represent a calculator. Or any other program not involving AI.

3) Finally, something useful. (Human imitation)

Here is a diagram that shows results flowing all the way from V' back to the world. It represents something like GPT3 or other human imitation based alignment.

And finally, HCH, IDA can be represented by

4) Diagram of HCH

This diagram shows such ...
Yudkowsky and Christiano discuss "Takeoff Speeds"
by Eliezer Yudkowsky — 2021-11-22T19:35:27.657Z

This is a transcription of Eliezer Yudkowsky responding to Paul Christiano's

Takeoff Speeds

live on Sep. 14, followed by a conversation between Eliezer and Paul. This discussion took place after Eliezer's

conversation

with Richard Ngo.

Color key:

Chat by Paul and Eliezer	Other chat	Inline comments

5.5. Comments on "Takeoff Speeds"

[Yudkowsky][10:14] (Nov. 22 follow-up comment)

(This was in response to an earlier request by Richard Ngo that I respond to Paul on Takeoff Speeds.)

[Yudkowsky][16:52]

maybe I'll try liveblogging some https://sideways-view.com/2018/02/24/takeoff-speeds/ here in the meanwhile

Slower takeoff means faster progress

[Yudkowsky][16:57]

The main disagreement is not about what will happen once we have a superintelligent AI, it’s about what will happen before we have a superintelligent AI. So slow takeoff seems to mean that AI has a larger impact on the world, sooner.

It seems to me to be disingenuous to phrase it this way, given that slow-takeoff views usually imply that AI has a large impact later relative to right now (2021), even if they imply that AI impacts the world "earlier" relative to "when superintelligence becomes reachable".

"When superintelligence becomes reachable" is not a fixed point in time that doesn't depend on what you believe about cognitive scaling. The correct graph is, in fact, the one where the "slow" line starts a bit before "fast" peaks and ramps up slowly, reaching a high point later than "fast". It's a nice try at reconciliation with the imagined Other, but it fails and falls flat.

This may seem like a minor point, but points like this do add up.

In the fast takeoff scenario, weaker AI systems may have significant impacts but they are nothing compared to the “real” AGI. Whoever builds AGI has a decisive strategic advantage. Growth accelerates from 3%/year to 3000%/year without stopping at 30%/year. And so on.

This again shows failure to engage with the Other's real viewpoint. My mainline view is that growth stays at 5%/year and then everybody falls over dead in 3 seconds and the world gets transformed into paperclips; there's never a point with 3000%/year.

Operationalizing slow takeoff

[Yudkowsky][17:01]

There will be a complete 4 year interval in which world output doubles, before the first 1 year interval in which world output doubles

...
Morally underdefined situations can be deadly
by Stuart_Armstrong — 2021-11-22T14:48:10.819Z

Thanks to Rebecca Gorman for help with this post.

Morally underdefined situations

I recently argued that full understanding of value extrapolation[1] was necessary and almost sufficient for solving the AI alignment problem.

In it, I introduced situations beyond the human moral training distribution, where we aren't sure how to interpret their moral value. I gave a convoluted example of an AI CEO engineering the destruction of its company and the ambiguously-moral creation of personal assistants, all in order to boost the value of its shareholders. In the past, I've also given examples of willing slave races and Croatian, communist, Yugoslav nationalists in the 1980s. We could also consider what happens as children develop moral instincts in a world they realise is more complicated, or ordinary humans encountering counter-intuitive thought-experiments for the first time. We also have some examples from history, when situations changed and new questions appeared[2].

I won't belabour the point any further. Let's call these situations "morally underdefined".

Morally underdefined can be terrible

Most of the examples I gave above are rather mild: yes, we're unsure what the right answer is, but it's probably not a huge disaster if the AI gets it wrong. But morally underdefined situations can be much worse than that. The easiest examples are ones that trade off a huge potential good against a huge potential bad, and thus deciding the wrong way could go extremely wrong; we need to carefully sort out the magnitude of the positives and the negatives before making any decision.

The repugnant conclusion is a classical example of that; we wouldn't want to get to the "huge population with lives barely worth living, filled with musak and potatoes" and then only discover after that there was an argument we had missed against total utilitarianism.

Another good example is if we developed whole brain emulations (ems), but they were imperfect. This would resolve the problem of death and might resolve most of the problem of suffering. But what price would we accept to pay? What if our personalities were radically changed? What if our personalities were subtly manipulated? What if we lost our memories over a year of subjective time - we could back these up in classical computer storage and access these as images and videos, but our internal memories would be lost? What if we became en...
A Bayesian Aggregation Paradox
by Jsevillamol — 2021-11-22T10:39:59.935Z

In short: There is no objective way of summarizing a Bayesian update over an event with three outcomes A:B:C as an update over two outcomes A:¬A.

Suppose there is an event with possible outcomes A,B,C.
We have prior beliefs about the outcomes p1:p2:p3.
An expert reports a likelihood factor of e1:e2:e3.
Our posterior beliefs about A:B:C are then p1⋅e1:p2⋅e2:p3⋅e3.

⎛⎜⎝p1p2p3⎞⎟⎠Prior×⎛⎜⎝e1e2e3⎞⎟⎠Update=⎛⎜⎝p1⋅e1p2⋅e2p3⋅e3⎞⎟⎠Posterior

But suppose we only care about whether A happens.
Our prior beliefs about A:¬A are p1:(p2+p3).
Our posterior beliefs are p1⋅e1:(p2⋅e2+p3⋅e3).
This implies that the likelihood factor of the expert regarding A:¬A is p1⋅e1:(p2⋅e2+p3⋅e3)p1:(p2+p3)=e1:p2⋅e2+p3⋅e3p2+p3.

(p1p2+p3)Prior×(e1p2⋅e2+p3⋅e3p2+p3)Update=(p1⋅e1p2⋅e2+p3⋅e3)Posterior

This likelihood factor depends on the ratio of prior beliefs p2:p3.

Concretely, the lower factor in the update is the weighted mean of the evidence e2 and e3 according to the weights p2 and p3.

This has a relatively straightforward interpretation. The update is supposed to be the ratio of the likelihoods under each hypothesis. The upper factor in the update is P(E|A). The lower factor is P(E|B∪C)=P(B)⋅P(E|B)+P(C)⋅P(E|C)P(B)+P(C).

(P(A|E)P(B∪C|E))Posterior∝(P(A)P(B∪C))Prior×(P(E|A)P(E|B∪C))Update(P(E|A)P(E|B∪C))Update=⎛⎝P(E|A)P(E∩(B∪C))P(B∪C)⎞⎠=⎛⎝P(E|A)P(B)⋅P(E|B)+P(C)⋅P(E|C)P(B)+P(C)⎞⎠

I found this very surprising - the summary of the expert report depends on my prior beliefs!

I claim that this phenomena is unintuitive, and being unaware of this can lead to errors.

Why this is weird

Bayes' rule describes how to update our prior beliefs using data.

In my mind, one very nice property of Bayes rule was that it cleanly separates the process into a subjective part (eliciting your priors) and an ~objective part (computing the update).

Posterior=PriorSubjective×LikelihoodObjective

For example, we may disagree on our prior beliefs on whether eg COVID19 originated in a lab. But we cannot disagree on the direction and magnitude of the update caused by learning that

it originated in one of the few ci

...
Do factored sets elucidate anything about how to update everyday beliefs?
by TekhneMakre — 2021-11-22T06:51:15.655Z

Pearl's theory of causal networks, interpreted as representing beliefs about variables connected causally, implies a precise algorithm for updating beliefs upon making an observation. Messages about how much each possibility is newly more or less likely, are sent backwards in time (from effect to cause) and forwards (from cause to effect). And, there are precise rules for what messages to send. (Qualitatively: if you update an unobserved node from a backward message, you also send messages forward and backward; if you update an unobserved node from a forward message, you send messages forward but not backward; messages to an observed node never induce forward messages ("screening off"), and backward messages to an observed node don't induce messages, whereas forward messages to an observed node do induce backward messages from that observed node ("explaining away").)
Deviate from these message-passing rules (or another procedure with the same result), and you expose yourself to incoherence:

Finite factored sets are described as a generalization of Pearl's theory. My question is: say that my beliefs in some context are reasonably well-represented as a finite factored set (with a distribution, and some events of interest, and some observations already made). Does the theory of factored sets give analogous rules, or rules of a similar flavor, for updating in a way that's consistent and, in the idealized case, complete and correct? I'm hoping for something more handy than things like "well, condition the distribution on the observation". E.g., do we learn something about how to update from an observed variable, to beliefs about another variable that is, so to speak, partly causally and partly logically related to the observed variable?
Even if you're right, you're wrong
by DanielFilan — 2021-11-22T05:40:00.747Z

Epistemic status: I noticed a commonality between a set of rhetorical moves that I sometimes find irritating and sometimes think is valid.

You claim P? OK. But even if you’re right about P…

… if I assume P, then I can prove not P. So, by contradiction, P is false. So you’re wrong.

… if you really believed P, then you would have a certain appropriate mood. But it doesn’t seem like you have that mood. So you’re wrong.

… with 99% probability, as long as there’s a 1% chance that P is wrong, according to expected value, we should act like P is wrong. So you’re wrong.

… accepting P would mean that you should do this weird action A. But you don’t do A, as far as I can tell. So you’re wrong.

… yesterday, you said Q, and Q implies not P. So you were wrong yesterday or today. So you’re wrong.

… somebody might infer Q from P. But Q is false! So you’re wrong.

… someone might infer from your expression of P that you believe Q. But Q is false! So you’re wrong.