trentmkelly/LessWrong-43k huggingface dataset:
Training Regime Day 2: Searching for bugs
by Mark Xu — 2020-02-16T17:16:32.606Z

Introduction

In CFAR terminology, a bug is something that systematically goes wrong in your life. It can be anything as small as "it takes me one minute to get out of bed in the morning when it should take me five seconds" to as large as "I hate my job" or even larger. Knowing the bugs in your life provides information as to where to spend time/energy to make your life better.

Today's exercise is going to be woven throughout the entire post. The necessary components to this exercise is any system capable of storing information that isn't your brain. A computer, pen and paper, a whiteboard, and a wax tablet will all suffice.

A common failure mode when writing down bugs is to consider whether or not they're bugs before writing them down. Writing down bugs is an exercise is babbling; do not restrict your search to the space of things your brain explicitly considers a "bug." Writing down things that might not be bugs is a necessary part of the process. You can always decide that they're not bugs later.

What follows are a series of prompts designed to get you to think about possible bugs that you have. I suggest that you read each prompt and write down bugs until the bugs no longer come freely. I will provide examples of bugs at the end, but you should search for you own bugs before reading that list to avoid anchoring.

There will be other opportunities to find bugs later on in this sequence, so don't feel pressured to compile a comprehensive list of all of your problems at this exact instance.

Prompts

What are your bugs? Sometimes there are things that are wrong in your life that you know are wrong. Starting with these is probably a good idea.

Is there something in your life that consistently causes you to feel frustrated/annoyed? Frustration and annoyance are the feelings that arise when something that you desire fails to manifest itself.

Imagine a day in your life in all its glorious detail. What are the ways that it could be better? Which parts of your day are less-than-ideal in some particular way?

Imagine asking your closest friends what your bugs are. What would they say? You should probably write down their responses even if you think what they're saying is wrong.

Is there anything that's stopping you from easily making your life better? Barriers to progress are also bugs.

Scott Alexander explains how it is sometimes hard to realize that you don't understand

...
Taking the Outgroup Seriously
by Davis_Kingsley — 2020-02-16T13:23:42.645Z

(Author's note/content warning: this post contains politically controversial examples, which are unfortunately perhaps necessary given the subject. I have tried to be relatively even-handed in this matter, my apologies if I erred in doing so.)

Occasionally, one reads a purported "takedown" of an opposing group that perhaps deals more with the writer's own models than those of the group he intends to refute.

For example, I recently read a widely-shared Quora post that claimed that religious proselytization was part of an elaborate brainwashing scheme to make the proselytizers themselves feel more connected to their own religion, and that the goal was not at all to make converts. The post might have been a good piece of rhetoric, but as something that aimed to actually understand the outgroup I think it was silly and obviously wrong.

Another example might be the claim that abortion supporters literally worship Moloch and want to kill babies as sacrifices, or for that matter the claim that abortion opponents only hold their views because they hate women and oppose abortion as part of a conspiracy to curtail women's rights.

What do these sorts of claims all have in common? They don't take the outgroup seriously. Sure, there might well be some fringe radicals who actually worship Moloch and want to kill babies or who oppose abortion because doing so furthers their conspiracy to suppress women, but such views likely constitute an extreme minority opinion. In point of fact, the person who says "I support abortion because I support women's rights." probably in fact actually believes that; the person who says "I support abortion because I believe killing a fetus is murder." probably in fact actually believes that too! There is no need to posit that these people are secret Moloch cultists or members of a grand conspiracy to suppress women -- they have already told you their reasons for their belief, and you weren't listening!

You can in fact often gain remarkable insight into the belief structures of most anyone -- even opponents -- by actually listening to and reading what they have to say. In most cases, people do not come up with elaborate secret reasoning for their opinions and then withhold it in favor of other arguments -- instead, they tend to explain their actual reasoning, which you can listen to to bett...
On characterizing heavy-tailedness
by Jsevillamol — 2020-02-16T00:14:06.197Z

[CONTEXT: For a while I have been meaning to engage with a literature review on heavy tailed distributions. Instead of just indefinitely postponing the project I resolved to write some preliminary thoughts on the topic, so I can get started on understanding the concept better with a less daunting task]

TL;DR: There are many formalizations of heavy-tailedness out there. I define five intuititive principles that I expect a good definition to satisfy: action-relevance, distinguish negative and positive risk, allow finite support, apply to empirically observed phenomena and provide a characterization in terms of a universal class of distributions. I discuss each in turn and provide examples.

Heavy-tailed distributions occur when extreme, low-probability yet plausible outcomes dominate decision-making.

For example, when considering how to contain a pandemic, an official will not want to focus on low-impact scenarios where eg the pandemic dies out on its own, nor on implausible scenarios where eg a solar flare messes up with our electronics during the crisis. Instead she will focus on scenarios where eg the pandemic grows out of control because its contagion rate is higher than expected - a plausible scenario that albeit unlikely is disastrous enough to warrant precaution.

Heavy-tailed distributions are an important object of study in cause prioritization - we should focus on studying such distributions to the extent that extreme outcomes dominate long term decision-making.

My informal impression is that the notion of heavy tail distributions has been heavily discussed among mathematicians, especially in the context of extreme value theory. However, there is no single agreed-upon formalization of the concept, making discussion and the application of the concept notoriously difficult.

Through this post, I will explore some important concepts around heavy-tailedness that we want an ideal definition to make precise.

My hope is that having this discussion will help us later productively discuss the strengths and weaknesses of different proposed definitions of heavy tailedness

In short, an ideal definition of heavy-tailedness would be action-relevant, able to distinguish risks from hits, be well-defined for distributions with finite support, describe natural phenomena and adscribe heavy-tailedness to a universal class of distributions.

Action relevance

We hope the definition o...
Training Regime Day 1: What is applied rationality?
by Mark Xu — 2020-02-15T21:03:32.685Z

Introduction

I claim that this series of posts will be a decent training regime for applied rationality. Accordingly, I think that one of the first steps is to tell you what I even think applied rationality is. However, since "applied rationality" is a little hard to describe succinctly, I will describe it through example and metaphor. To make the process of triangulation easier, I will also provide three such descriptions.

Disclaimer: I adapted the first 2 takes from CFAR; the third is my own.

Take 1

In this post-enlightenment era, we have this thing called science. Loosely speaking, science is a set of norms that allow for the gradual development of true and verifiable bodies of knowledge. Some examples of these norms include: testing hypotheses, peer review, and discarding theories that have large amounts of evidence piled against them. A short description of these norms is that whenever someone has a question about how the world works, the proper answer is “why don’t we just check.”

In my view, however, science has (at least) two major flaws.

Science is slow. The norms of science are sufficient to eject false knowledge from scientific canon eventually. They are most certainly not sufficient to eject false knowledge quickly. As the replication crisis demonstrates, scientific canon has contained, for many years, large swaths of false knowledge. Theories that are false are discarded in the limit, but they are not discarded as soon as a competing theory has sufficient evidence. Sometimes theories have the same consequences, so “just checking” can’t distinguish between them.

Science is expensive. There are many resources that science must consume to generate knowledge. Foremost among these resources are time and money, but it also sometimes requires certain amount of status. In general, science can only make broad prescriptions about how any given individual engages in the world. Of course, you can do science on yourself, but it also costs a fair amount of resources. Sometimes “just checking” is infeasible to do at such a small scale.

I claim that applied rationality does a pretty good job of filling both these niches. If you're practiced at the art of arriving at true beliefs, I claim that you can change your mind in as little as 5 minutes, upper-bounded by perhaps a single year. On the other hand, science routinely takes more than a single year to chang

...
It "wanted" ...
by jmh — 2020-02-15T20:52:07.094Z

I've seen a few post using that construction when the "It" has not capacity to want at all. They have prompted a question in my own approach to thinking about things.

First, I am not critiquing the posts or otherwise suggesting a problem with them, though that could be inferred so want to put that disclaimer up front. I do think there is a place for the use of such a rhetorical device. I also think there is a place for expressing what might be incomplete thoughts. Moreover, I don't think the specific word is the concern either -- this is not about computers, or viruses, or even perhaps plants "wanting" something.

The questions is should we generally take pause when we find ourselves using that type of rhetoric to ask if we are perhaps trying to work from an incorrect or seriously deficient map to navigate the territory we're trying to traverse?

In other words, should we use that as a marker to come back to and try to express conditions or functional structure more precisely, or at least confirm it really was a harmless short cut via language to get the idea across (and the idea is not dependent on the rhetoric).
Why Science is slowing down, Universities and Maslow's hierarchy of needs
by George3d6 — 2020-02-15T20:39:36.559Z

I don’t have a very high prior in regards to the correctness of Maslow's hierarchy of needs, but as far as general theories for understanding human needs go, I think it’s a pretty good.

There’s certainly people who seem to go strongly against it, to the point where they only require self actualization or where they are perfectly happy in life with only their physiological needs needs barely meet.

BUT

For all of the exceptions, most people, even exceptional people, seem to roughly live their life in accordance to it.

The gradual passage into adulting can be pretty daunting for people, even for well adjusted people with loving parents that can maintain a comfortable standard of living, for this reason. Gradually you are expected to find “safety” (i.e. financial stability, a house, a safe place to live) and “belonging” partially on your own.

Enter universities, the role of institutes of higher education in a well adjusted society should arguably be pushing the boundaries of human knowledge. Before they would also constitute a repository of information by maintaining huge libraries and people that could navigate them, but today we have the internet, .txt, .latex, .pdf, search engines and decent 10TB HDDs that sell for 100-200$ with tax, so I think it’s safe to say that role can now be played pretty cheaply.

So, universities now remain a places that educated and help people to navigate and enlarge the boundaries of human knowledge.

The recognition and most of all self satisfaction given by extending said boundaries is pretty great (or so I hear). So I think it’s safe to say that this role is one to be pursued by people that feel the needs on step 5 and possibly 4 of the pyramid.

BUT

In turn this is a process that requires a great deal of effort, dedication and intelligence, things that are hard to find and hard to direct for anyone that hasn’t fulfilled steps 1, 2 and 3 pretty well. Again, exceptions exist, but for basically all people it’s much easier to think about food, sex, friends and not dying than it is to think about novel bioreactors for producing cheap recombinant DNA vaccines or n-dimensional non Euclidean spaces… we can’t help it, it’s kinda the way we are evolved.

You can try to become a mental hermit and just not care about any of t...
Exercises in Comprehensive Information Gathering
by johnswentworth — 2020-02-15T17:27:19.753Z

Looking back, several of the most durably-valuable exercises I’ve done over the years have a general theme of comprehensive information gathering.

The most recent example involves capital investments. Economists talk about “capital goods” as physical stuff - machines, buildings, etc. But in practice, savings and investments are passed through banks and ETFs, bundled and securitized, involve debts and shares of companies which own debts and shares of other companies, and so forth… where does all that capital end up? To get an intuitive sense, I pulled up fundamental data on about 7000 US publicly-traded companies in

quantopian

, sorted them by amount of non-financial assets, and found that the top 100 accounted for about 50% of the non-financial assets of the whole set. Then, I looked at annual reports for each of those 100 companies, to see what capital assets they had. I googled around for pictures and maps of where those assets were located, and read up on anything I hadn’t heard of before. What’s a “

central office

”,

where are they

, what do they look like, and why does AT&T have $90B worth of them? What are the major US oil basins,

where are the wells

, and

what all goes into drilling them

? What are the technical differences between traditional phone, cable, satellite, and cell networks, and how do those technical differences impact the capital requirements of each? Who runs power plants and the power grid in various parts of the country? What are the major US railroads, and

where are they

? Why did GE own so many airplanes? These are the kinds of questions which come up when you want to know what “capital goods” actually consist of, in the real world.

Another interesting exercise: I read through five years of

Nature archives

, reading all the titles and any abstracts which sounded novel/interesting. I didn’t google everything I hadn’t heard of; instead, I’d wait until the same acronym popped up a few times before looking it up. This took maybe a week of evenings after work. By the end, I could at least place the large majority of articles in context. Now, when I see a title full of jargon in a field I haven’t studied, like “

Novel tau filament fold in corticobasal degeneration

”, I usually at least understand enough to guess at what it’s rele...
Reference Post: Trivial Decision Theory Problem
by Chris_Leong — 2020-02-15T17:13:26.029Z

A trivial decision problem is one where there is only a single option that the agent can take. In that case, the most natural answer to the answer to the question, "What action should we take?" would be "The only action that we can take!". We will call this the Triviality Perspective.

A particularly interesting example is Transparent Newcomb's Problem. If you accept the premise of a perfect predictor, then seeing $1 million in the transparent box implies that you were predicted to one-box which implies that you will one-box. So the Triviality Perspective claims that you should one-box, but also that this is an incredibly boring claim that doesn't provide much insight into decision theory.

We can see that in general, any decision theory problem with a perfectly defined universe and a perfectly defined agent will be trivial. Evidential decision theory treats the fact of the matter about which counterfactual action we select the same as any other fact in the problem statement and hence arguably embraces the Triviality Perspective.

Alternatively, the Triviality Perspective can be seen as an overly restrictive and literal interpretation of what the problem is. We could interpret "What action should we take?" to be asking not about the set of actions that are consistent with the problem statement, but instead about a set of counterfactual worlds each corresponding to a different actions. We will call this the Counterfactual Perspective. From this perspective, the problem is only trivial before we have augmented it with counterfactuals.

Here are some examples: In Causal Decision Theory, we can just construct counterfactuals by changing the value of the node in the causal graph to whatever we want and remove any inbound links. In Functional Decision Theory, we imagine that a particular program outputs a value that it does not and then update other program that subjunctively depend on that program's value. The Erasure Approach reinterprets the problem removing an assumption so that there will then be multiple possible counterfactuals consistent with the problem statement.

Combining perspectives

It is actually possible to be sympathetic to both the Triviality Perspective and the Counterfactual Perspective. Instead of being seen as opposed perspectives, they can be seen as two different lens for viewing the same situation so long as we don't try to mix both at the same t...
What is the difference between robustness and inner alignment?
by JanB — 2020-02-15T13:28:38.960Z

Robustness, as used in ML, means that your model continues to perform well even for inputs that are off-distribution relative to the training set.

Inner alignment refers to the following problem: How can we ensure that the policy an AI agents ends up with is robustly pursuing the objective that we trained it on? By default, we would only expect the policy to track the objective on the training distribution.

Both a lack of robustness and inner alignment failure thus lead to an AI agent that might do unforeseen things when it encounters off-distribution inputs.

What’s the difference?
I can (maybe) construct a difference if I assume that AI agents have distinct “competences” and “intent”.

There is some intuition that a lack of robustness relates to competence: The self-driving car really “wanted” to bring its passengers home safely. But then it started snowing and because the car’s vision system was only trained in sunny weather, the car didn’t spot the red traffic light and crashed. It was an honest mistake.

And there is some intuition that an inner alignment failure relates to intent: The nascent AGI never really cared about helping humans. It just play nice because it knew it would be deleted otherwise. As soon as it became powerful enough to take over the world (a situation it didn’t encounter during training), it did so.

However, the distinction between “competences” and “intent” doesn’t seem to apply to RL agents (and maybe not even to humans). RL agents just receive inputs and select actions. I wouldn’t be able to point to the “intent” of an RL agent. So what’s the difference between robustness an inner alignment then?
Does iterated amplification tackle the inner alignment problem?
by JanB — 2020-02-15T12:58:02.956Z

When iterated distillation and amplification (IDA) was published, some people described it described as "the first comprehensive proposal for training safe AI". Having read a bit more about it, it seems that IDA is mainly a proposal for outer alignment and doesn't deal with the inner alignment problem at all. Am I missing something?
Bayesian Evolving-to-Extinction
by abramdemski — 2020-02-14T23:55:27.391Z

The present discussion owes a lot to Scott Garrabrant and Evan Hubinger.

In Defining Myopia, I formalized temporal or cross-instance myopia / non-myopia, but I claimed that there should also be some kind of single-instance myopia which I hadn't properly captured. I also suggested this in Predict-O-Matic.

This post is intended to be an example of single-instance partial agency.

Evolving to Extinction

Evolution might be myopic in a number of ways, but one way is that it's myopic across individuals -- it typically produces results very different from what group selection would produce, because it's closer to optimizing relative fitness of individuals (relative to each other) than it is to optimizing overall fitness. Adaptations which help members of a species compete with each other are a great example of this. Why increase your own fitness, when you can just decrease someone else's instead? We're lucky that it's typically pretty hard, at least historically, to do things which are bad across the board but slightly less bad for the one doing them. Imagine a "toxic gas gene" which makes the air harder for everyone to breathe, but slightly less so for carriers of the gene. Such a gene would be selected for. This kind of thing can be selected for even to the point where it drives the population of a species right down to zero, as Eliezer's essay on evolving to extinction highlighted.

Actually, as Eliezer's essay emphasized, it's not even that evolution is myopic at the level of individuals; evolution is myopic down to the level of individual genes, an observation which better explains the examples of evolving-to-extinction which he discusses. (This is, of course, the point of Dawkins' book The Selfish Gene.) But the analogy of myopia-across-individuals will suit me better here.

Bayes "Evolving to Extinction"

The title of this post is a hyperbole, since there isn't an analog of an extinction event in the model I'm about to describe, but it illustrates that in extreme circumstances a Bayesian learner can demonstrate the same kind of pathological behavior that evolution does when it ends up selecting for relative fitness in a way which pumps against absolute fitness.

Like evolution, Bayes' Law will "optimize"[1] for relative fitness of hypotheses, not absolute fitness. Ordinarily there isn't enough of a difference for this to matter. However, I've been discussing scenario

...