trentmkelly/LessWrong-43k huggingface dataset:
A 'Practice of Rationality' Sequence?
by abramdemski — 2020-02-14T22:56:13.537Z

(This is not your typical factual-answer question, but I think it makes sense to format this as a question rather than a post.)

TLDR: Recommend some posts for a "practice of rationality" sequence I want to curate! Proposing posts that should exist but don't is also cool.

I've been thinking recently that it would be nice if rationality were more associated with a practice -- a set of skills which you can keep grinding and leveling up. Testable rationality skills (like accuracy or calibration in forecasting) are obviously a plus, but I'm not referring exclusively to this -- some very real things can be hard to evaluate externally, such as emotional wellness.

A model I have in mind is meditation: meditation is easy to "grind" because the meditator gets constant immediate feedback about how well they're focusing (or at least, they get that feedback if they meet a minimum of focus required to keep track of whether they are focusing). Yet it's quite difficult to evaluate progress from the outside.

(In fact, when I mentioned this desire for a "practice" of rationality to one friend, they were like "I agree, and in fact I think the practice should just be insight meditation.")

This is basically reiterating Brienne's call for tortoise skills (see also), except what I want to do is collect proposed things which could be part of a practice.

Obviously, some CFAR content could already qualify. CFAR doesn't exactly teach it that way -- as far as I've observed, CFAR's focus is on mindset interventions. "Mindset intervention" is the fancy psychology term for getting someone to think differently by having them do something once. For example, the point of "growth mindset" interventions is that you explain it once and this has long-lasting impact on someone's behavior. Another mindset intervention is: you ask people to write about what matters to them. Doing this once has shown long-term results.

In my first CFAR experience (which was an MSFP, fwiw), the phrase "It's not about the exercises!" was kind of a motto. It was explained at the beginning that CFAR teaches exercises not because people learn the exercises and then go out and use the exercises, but rather, going through the exercises a few times changes how you think about things. (The story was that peop...
The Catastrophic Convergence Conjecture
by TurnTrout — 2020-02-14T21:16:59.281Z

Overfitting the AU landscape

When we act, and others act upon us, we aren’t just changing our ability to do things – we’re shaping the local environment towards certain goals, and away from others.[1] We’re fitting the world to our purposes.

What happens to the AU landscape[2] if a paperclip maximizer takes over the world?[3]

Preferences implicit in the evolution of the AU landscape

Shah et al.'s Preferences Implicit in the State of the World leverages the insight that the world state contains information about what we value. That is, there are agents pushing the world in a certain "direction". If you wake up and see a bunch of vases everywhere, then vases are probably important and you shouldn't explode them.

Similarly, the world is being optimized to facilitate achievement of certain goals. AUs are shifting and morphing, often towards what people locally want done (e.g. setting the table for dinner). How can we leverage this for AI alignment?

Exercise: Brainstorm for two minutes by the clock before I anchor you.

Two approaches immediately come to mind for me. Both rely on the agent focusing on the AU landscape rather than the world state.

Value learning without a prespecified ontology or human model. I have previously criticized value learning for needing to locate the human within some kind of prespecified ontology (this criticism is not new). By taking only the agent itself as primitive, perhaps we could get around this (we don't need any fancy engineering or arbitrary choices to figure out AUs/optimal value from the agent's perspective).

Force-multiplying AI. Have the AI observe which of its AUs most increase during some initial period of time, after which it pushes the most-increased-AU even further.

In 2016, Jessica Taylor wrote of a similar idea:

"In general, it seems like "estimating what types of power a benchmark system will try acquiring and then designing an aligned AI system that acquires the same types of power for the user" is a general strategy for making an aligned AI system that is competitive with a benchmark unaligned AI system."

I think the naïve implementation of either idea would fail; e.g., there are a lot of degenerate AUs it might find. However, I'm excited by this because a) the AU landscape evolution is an important source of information, b) it feels like there's something here we could do which nicely avoids ontologies, and c) f

...
The Reasonable Effectiveness of Mathematics or: AI vs sandwiches
by Vanessa Kosoy — 2020-02-14T18:46:39.280Z

TLDR: I try to find the root causes of why math is useful.

Epistemic status: Marginally confident in veracity, not at all confident in novelty.

Background

I recently had a discussion with Rohin that started from Paul Christiano's concept of "intent alignment" but branched off into a different question, namely: do we need mathematical theory to solve AI risk, or is it sufficient to do experiments and informal reasoning? I argued that experiments are important but insufficient because

...ultimately, you need theoretical knowledge to know what can be safely inferred from these experiments. Without theory you cannot extrapolate.

At some point Rohin later asked me

I'm curious what you think doesn't require building a mathematical theory?

I replied

I'm not sure about the scope of your question? I made a sandwich this morning without building mathematical theory :)

to which Rohin said

Presumably the ingredients were in a slightly different configuration than you had ever seen them before, but you were still able to "extrapolate" to figure out how to make a sandwich anyway. Why didn't you need theory for that extrapolation?

Obviously this is a silly example, but I don't currently see any qualitative difference between sandwich-making-extrapolation, and the sort of extrapolation we do when we make qualitative arguments about AI risk. Why trust the former but not the latter? One is answer is that the latter is more complex, but you seem to be arguing something else.

So, in this essay I will try to explain my view of the role of mathematical theory and the qualitative difference between sandwiches and AI pertaining to this role.

The miracle of math

It probably brooks no argument that mathematics played a central role in the tremendous progress of science and technology during the last few centuries, and that it is used extensively in virtually all fields of modern engineering. The successes of mathematics have been so impressive that they prompted the Nobel-winning physicist Eugene Wigner into writing eir famous essay "The Unreasonable Effectiveness of Mathematics in the Natural Sciences", in which ey call it no less than a "miracle" and write:

...the enormous usefulness of mathematics in the natural sciences is something bordering on the mysterious... there is no rational explanation for it.

I see two main reasons why it's important to find that elusive rat

...
Perceptrons Explained
by  — 2020-02-14T17:34:38.999Z
Link: https://owenshen24.github.io/perceptron/

This is an introduction to the perceptron learning algorithm, a proof of its convergence, and some visualizations. Ideally, in the same vein as Nicky Case's Explorable Explanations, but I don't think it's at that level yet. Only LW-tangential, mostly just an educational post into basic ML.
Please Help Metaculus Forecast COVID-19
by AABoyles — 2020-02-14T17:31:16.488Z
Link: https://www.metaculus.com/questions/3667/the-li-wenliang-prize-series-for-forecasting-the-covid-19-outbreak/

Metaculus has launched a new Prize. In contrast to its earlier contest, which was focused on information to improve forecasts of relevance to Animal Welfare, the Li Wenliang Prize is designed to incentivize such information gathering for and forecasts of COVID-19. Separate prize categories are being awarded for Forecasts, Analysis, and Question Development. Here are the existing questions available for forecasting and analysis.
Training Regime Day 0: Introduction
by Mark Xu — 2020-02-14T08:22:19.851Z

Introduction

The sad truth of life is that there is a difference between knowing how to do something and actually being able to do it. Knowledge about how various instrumental rationality techniques work is not a substitute for actually being able to use them in practice. As the story goes, even Kahneman committed the planning fallacy when writing a textbook about the planning fallacy.

The Center for Applied Rationality (CFAR) holds workshops where they teach people rationality. I've been to one of these workshops and think that they do it quite well. Part of this is the content, which is a collection of rationality techniques that have been extensively iterated into strong thinking tools. Part of this is the teaching, which has likely been honed many hours. In addition to these two things, I claim that a large part of people actually learning how to do applied rationality at a CFAR workshop is because the environment is such that people actually practice applied rationality.

It's an obvious truth that practicing something makes you better at it. Unfortunately, it's also an obvious truth that most people never commit themselves to practicing things that they want to get better at. Especially if those things are hard-to-practice thinking tools.

CFAR recommends that workshop participants spend a fair amount of time figuring out applied rationality training regimes, ways to practice the skills that they've learned in day-to-day life. The obvious training regime is to actually do the thing. Sometimes this works. Most of the time it doesn't.

An easier training regime is reading a single blog post every day and doing a ~15 minute rationality technique exercise. This is what I hope to provide for you. This sequence might be thought of as an updated version of the hammertime sequence, with a slightly broader scope.

(In the process, I am also executing one of my own training regimes, which is something like "write 30 blog posts about my take on CFAR content.")

Contents

Disclaimer: this is not CFAR's take on applied rationality. This is my take on CFAR's take on applied rationality.

By my own estimate, CFAR contains approximately 30 hours of content. I will produce one blog post for each hour, for a total of 30 blog posts. The contents will be as follows (subject to change):

(It's like a 30 day juice cleanse but with rationality instead of juice.)

What is applied rationalit

...
Distinguishing definitions of takeoff
by Matthew Barnett — 2020-02-14T00:16:34.329Z

I find discussions about AI takeoff to be very confusing. Often, people will argue for "slow takeoff" or "fast takeoff" and then when I ask them to operationalize what those terms mean, they end up saying something quite different than what I thought those terms meant.

To help alleviate this problem, I aim to compile the definitions of AI takeoff that I'm currently aware of, with an emphasis on definitions that have clear specifications. I will continue updating the post as long as I think it serves as a useful reference for others.

In this post, an AI takeoff can be roughly construed as "the dynamics of the world associated with the development of powerful artificial intelligence." These definitions characterize different ways that the world can evolve as transformative AI is developed.

Foom/Hard takeoff

The traditional hard takeoff position, or "Foom" position (these appear to be equivalent terms) was characterized in this post from Eliezer Yudkowsky. It contrasts Hanson's takeoff scenario by emphasizing local dynamics: rather than a population of artificial intelligences coming into existence, there would be a single intelligence that quickly reaches a level of competence that outstrips the world's capabilities to control it. The proposed mechanism that causes such a dynamic is recursive self improvement, though Yudkowsky later suggested that this wasn't necessary.

The ability for recursive self improvement to induce a hard takeoff was defended in Intelligence Explosion Microeconomics. He argues against Robin Hanson in the AI Foom debates. Watch this video to see the live debate.

Given the word "hard" in this notion of takeoff, a "soft" takeoff could simply be defined as the negation of a hard takeoff.

Hansonian "slow" takeoff

Robin Hanson objected to hard takeoff by predicting that growth in AI capabilities will not be extremely uneven between projects. In other words, there is unlikely to be one AI project, or even a small set of AI projects, that produces a system that outstrips the abilities of the rest of the world. While he rejects Yudkowsky's argument, it is inaccurate to say that Robin Hanson expected growth in AI capabilities to be slow.

In Economic Growth Given Machine Intelligence, Hanson argues that AI induced growth could cause GDP to double on the timesca...
Effective Altruism 80,000 hours workshop materials & outline (and Feb 10 '19 KC meetup notes)
by samstowers — 2020-02-13T21:48:52.965Z

Intro & overview

This is a brief overview of an "Introduction to 80,000 Hours" workshop written by fellow LessWronger Bae's Theorem. It's to the point - a set of slides & an outline going over 80k hours' key points and core advice, then helping attendees apply it first to an outside party, then to themselves.

You can find the 15-slide presentation here: https://docs.google.com/presentation/d/1raV5fr7HryQ-3tEskHgfemje2SlSgtLCESo-mqZ2nVo/edit#slide=id.p

The (revised) article that Bae's Theorem based the content off of can be found here: https://docs.google.com/document/d/1clHJQaoAt9YaguMZFH6gkDSPTsolEylME92X2dBnaWk/edit This document also includes the 1-page handout he used in the workshop for easy reference during the activities.

General content outline:

Introduction to 80,000 Hours as an organization
A summary of 80k Hours' claims
The 80k Hours' career decision process

Checking personal fit
Career exploration
Mitigating accidental harm
Building career capital
Taking care of yourself and your mental health
Coordinating with others & communities
Taking measured risks

Options to make a difference in any career
Additional resources
ACTIVITY: Mock career advising interview with an impact-minded college student (done to give practice thinking in EA mindset without touching personal identity)
BREAKOUT: Split into groups of 2-3 and take 6 minutes to think of possible short-term and long-term actions each can take to improve the impact of their career. Discuss with the group.

Other notes

After the workshop, we had a brief discussion on group norms, talked about other activities we could take on as a group to boost our impact, and the topic for the next meetup (effective interventions for global poverty).

There's a slight schism in the group, between people who want to *take action NOW* even if it's not strictly the most effective (e.g. do research on the efficacy of local charities, volunteer somewhere), and people who want to focus more strictly on generating donations & awareness for EA causes.

There are two recurring arguments for the first approach. First, by taking local community action, we'd raise the visibility of our EA group and EA in general, while also painting it in a positive light for anyone encountering it. This would (hopefully) have the effect of increasing group membership and EA donations in the me...
How do you use face masks?
by ChristianKl — 2020-02-13T14:18:05.037Z

I did buy face masks a while ago, because other rationalists recommended wearing them.

The list time I had a flu and went to the doctors office I wore a face mask, the receptionist then told me to take it off while sitting in the waiting room.

While I do think I have a better idea then that receptionist, I still don't know much. It's easy to have superstitions about how to use face masks correctly. If I want to go for maximum benefit with minimum hassle, what's worth knowing? When do I throw masks away?
In theory: does building the subagent have an "impact"?
by Stuart_Armstrong — 2020-02-13T14:17:23.880Z

EDIT: Extensive modifications to this post, sorry, too numerous to list. If you read the original post, it's worth rereading.

Overall summary post here.

This post looks more generally into the subagent issue with attainable utility, discussed in the previous two posts. The key questions are:

What is the minimal impact penalty of (cleverly) building the subagent?

In practice, what will be the impact penalty of each step of building the subagent?

What will the restrictions on the subagent be, once it is built?

I expect that most of the discussions about attainable utility impact measure will centre on 2.: how can an agent put together a subagent without triggering the impact measure at some point? We know that it's possible to do in some situations, and I've made the generic argument that, since we don't have an impossibility result, a superintelligent agent should be able to do in most situations. Still, this is the point of my argument where there is the most to criticise.

But in this post, I'll show that 1. doesn't restrict the subagent in any significant way: one can generically build a subagent without falling afoul of the impact penalty.

Subagent impact

Impact measure

I'll use Turner et al's impact measure. This uses the Q-values of various rewards in a set R; for s a state, a an action, and ∅ the default noop action, the penalty is:

PENALTY(s,a)=∑R∈R|QR(s,a)−QR(s,∅)|.

Now, QR(s,a) is the expected reward for R, in state s and after taking action a, assuming that the agent subsequently takes the optimal policy for maximising R.

The agent A has a positive reward R0, and it wishes to maximise

R0−PENALTY.

Now let's look at making the subagent.

Building the subagent

To separate 1. from 2., I'll assume that the subagent can be built in a single action. This distinguishes the impact of having the subagent, from the impact of the process of making it.

The state is st. Let aSA be the action of building the subagent[1].
Here are assumptions on SA:

A can make its policy visible to SA.

SA can disassemble itself with policy πd.

SA has a policy πSA available such that, for all R∈R, EπSAR≤QR(st,∅).

SA has a specific policy π′t+1 that it must follow on the next turn only.

The π′t+1 is defined as follows:

On turn t+1, if A changes its policy to π∗R, the R-maximising policy, then follow πd and disassemble yourself.
Otherwise, follow policy πSA.

Then what is QR(st,aSA)? B

...
A Variance Indifferent Maximizer Alternative
by Nevan Wichers — 2020-02-13T09:06:19.796Z

TLDR

This post explores creating an agent which tries to make a certain number of paperclips without caring about the variance in the number it produces, only the expected value. This may be safer than one which wants to reduce the variance, since humans are a large source of variance and randomness. I accomplished this by modifying a Q-learning agent to choose actions which minimized the difference between the expected number of paperclips and the target number. In experiments the agent is sometimes indifferent to variance, but usually isn't, depending on it's random initialization and what data it sees. Feedback from others was largely negative. Therefore, I'm not pursuing the idea at this time.

Motivation

One safety problem pointed out in Superintelligence is that if you reward an optimizer for making between 1900 and 2000 paperclips, it will try to eliminate all sources of variance to make sure that the number of paperclips is within that range. This is dangerous because humans are a large source of variance, so the agent would try to get rid of humans. This post investigates a way to make an agent that will try to get the expected number of paperclips as close as possible to a particular value without caring about the variance of that number. For example if the target number of paperclips is 2000, and it's choosing between one action action which gets 2000 clips with 100% probability, and another action which gets 1000 paper clips with 50% probability and 3000 paperclips with 50% probability, it will be indifferent between those 2 actions. Note that I’m using paper clips as the example here, but the idea could work with anything measurable.

Solution

My goal is to have an agent which tries to get the expected number of paperclips as close to a certain value as possible while ignoring the expected variance.

To see how to accomplish this goal, it helps to look at regular Q learning. Regular Q learning to maximize the number of paperclips is similar to this because the agent ignores the variance in the expected number. The equation for regular Q learning to maximize the number of paperclips is

Q(st,at)=P(st)+maxaQ(st+1,a)

Where P(st) is the number of paperclips the agent earned at time t (the reward). I consider the non-discounted case for simplicity. This can be rewritten as

Q(st,at)=P(st)+Q(st+1,a)

Where a=argmaxa(Q(st+1,a))

The Q(st,at)=... part of the equatio

...
Confirmation Bias As Misfire Of Normal Bayesian Reasoning
by Scott Alexander — 2020-02-13T07:20:02.085Z

From the subreddit: Humans Are Hardwired To Dismiss Facts That Don’t Fit Their Worldview. Once you get through the preliminary Trump supporter and anti-vaxxer denunciations, it turns out to be an attempt at an evo psych explanation of confirmation bias:

Our ancestors evolved in small groups, where cooperation and persuasion had at least as much to do with reproductive success as holding accurate factual beliefs about the world. Assimilation into one’s tribe required assimilation into the group’s ideological belief system. An instinctive bias in favor of one’s in-group” and its worldview is deeply ingrained in human psychology.

I think the article as a whole makes good points, but I’m increasingly uncertain that confirmation bias can be separated from normal reasoning.

Suppose that one of my friends says she saw a coyote walk by her house in Berkeley. I know there are coyotes in the hills outside Berkeley, so I am not too surprised; I believe her.

Now suppose that same friend says she saw a polar bear walk by her house. I assume she is mistaken, lying, or hallucinating.

Is this confirmation bias? It sure sounds like it. When someone says something that confirms my preexisting beliefs (eg ‘coyotes live in this area, but not polar bears’), I believe it. If that same person provides the same evidence for something that challenges my preexisting beliefs, I reject it. What am I doing differently from an anti-vaxxer who rejects any information that challenges her preexisting beliefs (eg that vaccines cause autism)?

When new evidence challenges our established priors (eg a friend reports a polar bear, but I have a strong prior that there are no polar bears around), we ought to heavily discount the evidence and slightly shift our prior. So I should end up believing that my friend is probably wrong, but I should also be slightly less confident in my assertion that there are no polar bears loose in Berkeley today. This seems sufficient to explain confirmation bias, ie a tendency to stick to what we already believe and reject evidence against it.

The anti-vaxxer is still doing something wrong; she somehow managed to get a very strong prior on a false statement, and isn’t weighing the new evidence heavily enough. But I think it’s important to note that she’s attempting to carry out normal reasoning, and failing, rather than carrying out some special kind of reasoning called “...