trentmkelly/LessWrong-43k huggingface dataset:
Solving the Doomsday argument
by Stuart_Armstrong — 2019-01-17T12:32:23.104Z

The Doomsday argument gives an anthropic argument for why we might expect doom to come reasonably soon. It's known that the Doomsday argument works under SSA, but not under SIA.

Ok, but since different anthropic probability theories are correct answers to different questions, what are the question versions of the Doomsday argument, and is the original claim correct?

No Doomsday on birth rank

Simplify the model into assuming there is a large universe (no Doomsday any time soon) with many, many future humans, and a small one (a Doomsday reasonably soon - within the next 200 billion people, say), with equal probability. In order to think in terms of frequencies, which comes more naturally to humans, we can imagine running the universe many, many times, each with the 50% Doomsday chance.

There are roughly a 108.5 billion humans who have ever lived. So, asking:

What proportion of people with birth rank 108.5 billion, live in a small universe (with a Doomsday reasonably soon)?

The answer to that question converges to 1/2, the SIA probability. Half of the people with that birth rank live in small universes, half in large universes.

Doomsday for time travellers

To get an SSA version of the problem, we can ask:

What proportion of universes, where a randomly selected human has a birthrank of 108.5 billion, will be small (with a Doomsday reasonably soon)?

This will give an answer close to 1 as it converges on the SSA probability.

But note that this is generally not the question that the Doomsday argument is posing. If there is a time traveller who is choosing people at random from amongst all of space and time - then if they happen to choose you, that is a bad sign for the future (and yet another reason you should go with them). Note that this is consistent with conservation of expected evidence: if the time traveller is out there but doesn't choose you, then this a (very mild) update towards no Doomsday.

But for the classical non-time-travel situation, the Doomsday argument fails.
The questions and classes of SSA
by Stuart_Armstrong — 2019-01-17T11:50:50.828Z

Important summary: For SSA, the reference class to be used is defined in the question asked.

I said that SSA derives from the following question:

Q: What proportion of universes, where versions of me exist, will be large?

And it's sort of true, that sort of is the SSA question. But only when talking about probabilities of universes, not when updating on information.

For example, consider the sleeping beauty problem. If the coin is tails (T), Sleeping Beauty (SB) is awoken twice, on Monday and Tuesday. On heads (H), she is only awoken on Monday.

Then if the "large" universe is the tails universe, and "versions of me" cover both Monday and Tuesday (reference class RSB), then the answer to that question will be "1/2". So, writing PSB for the notationally unwieldy PSSARSB, we have:

PSB(T)=1/2,

the standard halfer/SSA answer.

Now suppose that every version of Sleeping Beauty is (eventually) told which day it is. Suppose she is told it is Monday. What is the correct question now? It seems like it should be:

QM: What proportion of universes, where versions of me exist on Monday, will be large?

However, the answer to that question is... also 1/2. The fact of it being Monday has not updated anything at all! This is surprising, because if the question had asked about Tuesday, it would have updated to 1 on the universe being large (ie tails).

Sliding reference classes

What's happening is that the reference class has changed between question Q and question QM. For QM, the reference class is the set of all Sleeping Beauties who are eventually told that it is a Monday (call this RSBM). For that, we still have:

PSBM(T|Monday)=PSBM(T)=1/2.

What about PSB(T|Monday), using the original reference class, which is equal to 1/4? Well, the best question I could come up with that captures that is:

Q': What proportion of universes, where a randomly selected version of me exists on a Monday, will be large?

Randomly selected meaning "randomly selected from the versions of me that exist in the universe".

This question doesn't feel as natural as either Q or QM. And it highlights the importance of the reference class, the set of "versions of me". So let's look in more details at the reference class.

What are reference classes for?

There are several purposes for reference classes. We can use them in decision theory, when considering the class of agents that will reach the same decisi

...
In SIA, reference classes (almost) don't matter
by Stuart_Armstrong — 2019-01-17T11:29:26.131Z

This is another write-up of a fact that is generally known, but that I haven't seen proven explicitly: the fact that SIA does not depend upon the reference class.

Specifically:

Assume there are a finite number of possible universes Ui. Let R be a reference class of finitely many agents in those universes, and assume you are in R. Let Rs be the reference class of agents subjectively indistinguishable from you. Then SIA using R is independent of R as long as Rs⊂R.

Proof:

Let {Ui}i∈I be a set of universes for some indexing set I, and P a probability distribution over them. For a universe Ui, let R(Ui) be the number of agents in the reference class R in Ui.

Then if PR is the probability distribution from SIA using R:

PR(Ui)=R(Ui)P(Ui)∑j∈IR(Uj)P(Uj).

We now wish to update on our own subjective experience sub. Since there are R(Ui) agents in our reference class, and Rs(Ui) have subjectively indistinguishable experiences, this updates the probabilities by weights PR(Ui)→PR(Ui)×Rs(Ui)R(Ui), which is just Rs(Ui)P(Ui)/(∑j∈IR(Uj)P(Uj)). After normalising, this is:

PR(Ui|sub)=P(Ui)Rs(Ui)∑k∈IP(Uk)Rs(Uk)×(∑j∈IR(Uj)P(Uj))(∑j∈IR(Uj)P(Uj))=P(Ui)Rs(Ui)∑k∈IP(Uk)Rs(Uk)=PRs(Ui).

Thus this expression is independent of R.

Given some measure theory (and measure theoretic restrictions on R to make sure expressions like ∫IR(Ui)P(Ui) converge), the result extends to infinite classes of universes, with ∫ in the proof instead of ∑.
Alignment Newsletter #41
by Rohin Shah — 2019-01-17T08:10:01.958Z

Building AI systems that require informed consent

Find all Alignment Newsletter resources here. In particular, you can sign up, or look through this spreadsheet of all summaries that have ever been in the newsletter.

This newsletter is late because I wanted to include Towards formalizing universality and related posts, but I haven't yet understood them well enough to put them in this newsletter so I'm not including them this time and hope to put them in next week.

Highlights

Non-Consequentialist Cooperation? (Abram Demski): One possible way to build useful AI systems is to have them try to help us. Taking more of a libertarian stance, a robot could help us in an autonomy-centric way, which would only take actions if we give it our informed consent. We can't ask for explicit consent for every action, since there's no clear way to break down actions, and it would certainly be too onerous to give informed consent to every possible motor command. As a result, our robots will need to infer when we have given consent. This increases the chances of misunderstanding, but we could try to have a high threshold, so that the robot asks us if it is even a little unsure about whether we have given consent.

If we want to precisely define consent, we'll need to solve some of the same problems that impact measures have to contend with. In particular, we would need to get consent for outcomes that the robot knows will happen as a result of its actions, but not ones that happen as a side effect. It's fine to give informed consent to the robot to buy bananas from a grocery store, even if that could cause a hurricane, as long as the robot doesn't know that it would cause a hurricane. Another issue is that inferring consent requires you to confront the issue that humans can be irrational. A third issue is that we might prevent the robot from taking actions that would help us that we can't understand would help us -- consider trying to ask a dog for its informed consent to take it to the vet.

Rohin's opinion: This seems like an interesting idea for how to build an AI system in practice, along the same lines as corrigibility. We notice that value learning is not very robust: if you aren't very good at value learning, then you can get very bad behavior, and human values are sufficiently complex that you do need to be very capable in order ...
Clothing For Men
by Solnassant — 2019-01-17T00:23:50.567Z

Nothing fundamentally rational about this post, but it is instrumental. It will be useful to those that want to dress better. It's already been discussed that dressing well improved one's social life; I wanted to put it in one place how to reach that.

TL,DR

The point of dressing well is to look cool
It gets you out of the badly-dressed hole but doesn’t make you a cool God. Work on yourself.
Less is more, don’t trust your “I like this funky thing”.
Buy to compose outfits, ie to match what you already own, instead of random pieces you see in a shop and like
Wear what follows the shape of your body.
Go for dark blue tones, light grey to black, and sand to kaki.
Go for plain over pattern
Go for smooth over texture
Prefer symmetry
Have no logo/visible branding/message/image on your clothes.
Fashionable people break the rules all the time. But they have integrated style to their thinking. To get that going, follow the rules.

Foreword:

My post will have two parts.
In the first (the one you're reading) I’ll detail the rules that are commonly used to dress well. Those rules will optimize the social aspect of clothing, not what the fabric offers technically: how cool you look, not what interesting properties you get out of your clothes. In a later article, I will explore ‘techwear’, aka clothing with technical properties (thermoregulation, impermeability, not retaining smells…), as the logical direction to fully optimize one’s fashion. Because I don’t want to write the second part now, this first half is totally self sufficient, and I’ll write about many other things than clothing before I write on techwear.

Dressing well is an art not a science, and an art that changes with trends, social classes, tribes, generations. I am well aware that each rule in this article is broken everyday by extremely well-dressed people; nevertheless those rules are very useful, and a great one in clothing is to start with the basics for which rules apply, and then to explore.

A good sense of fashion is both extremely important, and not important at all.
People do judge based on looks. If you are visibly ‘out’, you will suffer social consequences such as people avoiding your company. By dressing well, you avoid being rejected from the get-go. But don’t expect to be loved for it. Being fashionable will save you from the hole and give you a bit of a wind up, but it won’t catapult how cool you...
Some Thoughts on My Psychiatry Practice
by Laura B — 2019-01-16T23:16:54.264Z

I’ve noticed a marked change in my clientele after going into private practice.[1] Of course I expected class differences-- I charge full fee and don’t take insurance. But there are differences that are not as predictable as ‘has more money’. During residency I worked at a hospital Medicaid clinic and saw mostly poor, often chronically unemployed people. While monetary problems were a source of stress, they were not nearly as present in people’s minds as someone from a middle-class upbringing might think. These people were used to going without. They were not trying to get more. The types of things they talked about were family problems, health problems, and trauma. So much trauma. People’s ego-identity crises centered less on their accomplishments and more on their relationships.

The patients I see now are mostly highly successful, highly educated, weathly people, most of whom care a lot about their careers. Their ego-identity crises center around their work and their position in life relative to others. There is a lot of concern about ‘the path’. ‘Did I go down the right path?’ ‘Did I make a wrong turn?’ There seems to be a great fear of making or having made a wrong decision, which can paralyze their ability to make future decisions. While this group is not without trauma, it is not what they wish to focus on. They will often be dismissive of its effects on them, noting that they clearly got over it in order to get where they are now. Which is, you know, in my office.

Many of my new patients do NOT want to take medication. This is a large change from my patients at the Medicaid clinic who were always requesting more and different pills. And this difference is not because my new patients are less unhappy. They describe intense misery, even a wish to die, going on for months if not years, and yet they struggle through each day in their sisyphean ordeal. They ‘power through’ until they can’t. Until something gives. Then they come to me.

I can think of several good reasons to have concerns about using medication. What are the long-term effects? Could this change my identity? What if this makes me ok with a shitty situation and then I don’t fix an underlying problem? But these are not the typical concerns I hear raised. What most of my patients say is that they don’t want to ‘rely’ on a medication. They don’t want to be the type of person who takes it. ‘That w...
And My Axiom! Insights from 'Computability and Logic'
by TurnTrout — 2019-01-16T19:48:47.388Z

Foreword

Max Tegmark's Our Mathematical Universe briefly touches on a captivating, beautiful mystery:

The arrows indicate the close relations between mathematical structures, formal systems and computations. The question mark suggests that these are all aspects of the same transcendent structure, whose nature we still haven't fully understood.

The profound results compiled by the Computability and Logic textbook may be the first step towards the answer.

Computability and Logic

If this sentence is true, then Santa Claus is real.

As usual, I'll explain confusions I had and generally share observations. This book is on the MIRI reading list.

1 Enumerability

Coming back to this book, I'm amazed by some of my margin scribbles – expressions of wonderment and awe at what now strike me as little more than obvious facts ("relations are sets of ordered pairs!").

2 Diagonalization

Exercise 2.13 (Richard's paradox) What (if anything) is wrong with following argument?

The set of all finite strings of symbols from the alphabet, including the space, capital letters, and punctuation marks, is enumerable; and for definiteness let us use the specific enumeration of finite strings based on prime decomposition. Some strings amount to definitions in English of sets of positive integers and others do not; strike out the ones that do not, and we are left with an enumeration of all definitions in English of sets of positive integers, or, replacing each definition by the set it defines, and enumeration of all sets of positive integers that have definitions in English. Since some sets have more than one definition, there will be redundancies. Strike them out to obtain an irredundant enumeration of all sets of positive integers that have definitions in English. Now consider the set of positive integers defined by the condition that a positive integer n belongs to the set if and only if does not belong to the nth set in the irredundant enumeration just described.

This set does not appear in that enumeration, so it cannot have a definition in English. Yet it does, and in fact we have just given such a definition.

My first time reading this exercise, I had plenty of objections. "Is not abusive to use English in place of a formal system? How do we even quantify the expressiveness of English, is that a thing?", and so on. Yet, returning with more...
The reward engineering problem 
by paulfchristiano — 2019-01-16T18:47:24.075Z

Today we usually train reinforcement learning agents to perform narrow tasks with simple goals. We may eventually want to train RL agents to behave “well” in open-ended environments where there is no simple goal.

Suppose that we are trying to train an RL agent A. In each episode, A interacts with an environment, producing a transcript τ. We then evaluate that transcript, producing a reward r ∈ [0, 1]. A is trained is to maximize its reward.

We would like to set up the rewards so that A will learn to behave well — that is, such that if A learns to receive a high reward, then we will be happy with A’s behavior.

To make the problem feasible, we assume that we have access to another agent H which

is “smarter” than A, and
makes “good” decisions.

In order to evaluate transcript τ, we allow ourselves to make any number of calls to H, and to use any other tools that are available. The question is: how do we carry out the evaluation, so that the optimal strategy for A is to also make “good” decisions?

Following Daniel Dewey, I’ll call this the reward engineering problem.

Note that our evaluation process may be quite expensive, and actually implementing it may be infeasible. To build a working system, we would need to combine this evaluation with semi-supervised RL and learning with catastrophes.

Possible approaches and remaining problems

I know of 3 basic approaches to reward engineering:

Direct supervision. Use H to evaluate A’s behavior, and train A to maximize H’s evaluations. In some contexts we could compare two behaviors instead of evaluating one in isolation.
Imitation learning. Use H to generate a bunch of transcripts, and train Ato produce similar-looking transcripts. For example, we could train a model to distinguish A’s behavior from H’s behavior, and reward A when it fools the distinguisher.
Inverse reinforcement learning. Use H to generate a bunch of transcripts, and then infer a reward function which is being approximately optimized by H. Use this reward function to evaluate A’s behavior.

All of these approaches are promising but face significant challenges. I’ll describe some of these problems in the next 3 sections.

1. Direct supervision

In direct supervision, H looks at a transcript of A’s behavior, and estimates how good that transcript is.

To see the problem with this scheme, suppose that A has been asked to draw a picture, and A does it by copying an existing pict...
Buy shares in a megaproject
by ryan_b — 2019-01-16T16:18:50.177Z

I think we should create a separate legal construct for megaprojects.

I recently wrote about megaproject management here. In the outline for Oxford's Major Programme Management program, the first module is described in this way:

Develop your understanding of major programmes as a governance structure and distinctive organisational form. In the context of programme performance, consider and reflect on organisational theory and design.

The basic concept is that instead of a normal corporation, it would operate like a corporation with an expiration date and the project goals baked into the governance. We want "on-time and on-budget" to take the place of "growth and revenue" in the new organization's decision making.

We can buy stock in corporations, which now live ~16 years and declining.
Megaprojects like the Big Dig (16 years of construction) and the F-35 development (26 years to production) are easily in the range of normal corporations
Currently the only way to bet for or against the megaproject is to bet on the corporations/creditors/etc.
This would allow us to specifically invest in, or bet against, the project.
We invest in time-limited financial assets like bonds, puts and options already.

Consider the Boring Company. If one of those projects is a terrible idea, we are stuck with adjusting our position on the company as a whole, which means the terrible project puts the others in jeopardy.

By contrast, if the megaprojects were all separate entities and the Boring Company was hired to manage all of them, we could then bet on Boring Company and all of the projects independently of each other.

It seems to me that this would do a pretty good job resolving a lot of the problems that plague megaprojects currently.
A Framework for Internal Debugging
by Matt Goldenberg — 2019-01-16T16:04:16.478Z

When I'm debugging myself or others for unproductive behavioral, emotional, or mental patterns (hereby referred to as patterns) , I use this quasi-linear framework to make sure that I don't miss any critical steps and the issue stays solved. I've used this framework successfully with clients, housemates at Gentle Mesa, and hotel mates at The EA Hotel, as well as in debugging my own internal conflicts. I'm currently creating a GuidedTrack course that takes people through search of these steps and their associated techniques.

This model makes some assumptions about the underlying structure of the internal issue. It generally assumes that our patterns have internal reasons behind them, that those reasons are often coded as subconscious beliefs about the world or yourself, and that those beliefs sometimes cluster together to be understood as what we often refer to as "parts" or "subagents".

Finally, it assumes that when we have a pattern that we consider negative, one way to understand this is a tension between the part that is causing the pattern and the part that sees that pattern as negative (hereby referred to as anti-patterns). A successful debugging is one in which that tension is dissolved, rearranged, or in rare cases, accepted.

1. Awareness

The first step to solving an internal conflict is recognizing that you have it. In this stage, you use awareness of self-talk, mental imagery, and bodily sensations to notice that there's something "off".

Associated Techniques

Mindfulness
CFAR's Bug List Prompts
CFAR's MurphyJitsu

2. Introspection

In this stage, you work to understand the cause of the pattern. You're trying to make explicitly conscious the beliefs that underlie the pattern. On the flipside, you're working to understand the cause of the anti-pattern, and the beliefs that cause the pattern to be seen as problem.

It's often useful here to not just understand the first level need being met (e.g. "I don't want to procrastinate because it lets me be more productive") but to find the "deeper cut" - The deeper need or fear that is being served (e.g. "If I'm not productive than I don't deserve anybody's love")

Finally, in this step, it's important to check for "second order constructions" - that is, do I have a belief...