trentmkelly/LessWrong-43k huggingface dataset:
The Scylla of Error and the Charybdis of Paralysis
by Johnicholas — 2009-09-26T16:01:20.938Z

We're interested in improving human rationality. Many of our techniques for improving human rationality take time. In real-time situations, you can lose by making the wrong decision, or by making the "right" decision too slowly. Most of us do not have inflexible-schedule, high-stakes decisions to make, though. How often does real-time decision making really come up?

Suppose you are making a fairly long-ranged decision. Call this decision 1. While analyzing decision 1, you come to a natural pause. At this pause you need to decide whether to analyze further, or to act on your best-so-far analysis. Call this decision 2. Note that decision 2 is made under tighter time pressure than decision 1. This scenario argues that decision-making is recursive, and so if there are any time bounds, then many decisions will need to be made at very tight time bounds.

A second, "covert" goal of this post is to provide a definitely-not-paradoxical problem for people to practice their Bayseian reasoning on. Here is a concrete model of real-time decisionmaking, motivated by medical-drama television shows, where the team diagnoses and treats a patient over the course of each episode. Diagnosing and treating a patient who is dying of an unknown disease is a colorful example of real-time decisionmaking.

To play this game, you need a coin, two six-sided dice, a deck of cards, and a helper to manipulate these objects. The manipulator sets up the game by flipping a coin. If heads (tails) the patient is suffering from an exotic fungus (allergy). Then the manipulator prepares a deck by removing all of the clubs (diamonds) so that the deck is a red-biased (black-biased) random-color generator. Finally, the manipulator determines the patients starting health by rolling the dice and summing them. All of this is done secretly.

Play proceeds in turns. At the beginning of each turn, the manipulator flips a coin to determine whether test results are available. If test results are available, the manipulator draws a card from the deck and reports its color. A red (black) card gives you suggestive evidence that the patient is suffering from a fungus (allergy). You choose whether to treat a fungus, allergy, or wait. If you treat correctly, the manipulator leaves the patient's health where it is (they're improving, but on a longer timescale). If you wait, the manipulator reduces the patient's health ...
Correlated decision making: a complete theory
by Stuart_Armstrong — 2009-09-26T11:47:09.061Z

The title of this post most probably deserves a cautious question mark at the end, but I'll go out on a limb and start sawing it behind me: I think I've got a framework that consistently solves correlated decision problems. That it is, those situation where different agents (a forgetful you at different times, your duplicates, or Omega’s prediction of you) will come to the same decision.

After my first post on the subject, Wei Dai asked whether my ideas could be formalised enough that it could applied mechanically. There were further challenges: introducing further positional information, and dealing with the difference between simulations and predictions. Since I claimed this sort of approach could apply to the Newcomb’s problem, it is also useful to see it work in cases were the two decisions are only partially correlated - where Omega is good, but he’s not perfect.

The theory

In standard decision making, it is easy to estimate your own contribution to your own utility; the contribution of others to your own utility is then estimated separately. In correlated decision-making, both steps are trickier; estimating your contribution is non-obvious, and the contribution from others is not independent. In fact, the question to ask is not "if I decide this, how much return will I make", but rather "in a world in which I decide this, how much return will I make".

You first estimate the contribution of each decision made to your own utility, using a simplified version of the CDP: if N correlated decisions are needed to gain some utility, then each decision maker is estimated to have contributed 1/N of the effort towards the gain of that utility.

Then the procedure under correlated decision making is:

1) Estimate the contribution of each correlated decision towards your utility, using CDP.

2) Estimate the probability that each decision actually happens (this is an implicit use of the SIA).

3) Use 1) and 2) to estimate the total utility that emerges from the decision.

To illustrate, apply it to the generalised absent minded driver problem, where the return for turning off at the first and second intersection are x and y, respectively, while driving straight through grants a return of z. The expected return for going straight with probability p is R = (1-p)x + p(1-p)y + p2z.

Then the expected return for the driver at the first intersection is (1-p)x ...
Non-Malthusian Scenarios
by Wei Dai — 2009-09-26T02:44:53.571Z

This is an attempt to list all of the possible ways in which humanity may avoid scenarios where the average standard of living is close to subsistence, in response to Robin Hanson's recent series of posts on Overcoming Bias, where he argues that such an outcome is likely in the long run.

I'll start with six, some suggested by myself, and others collected from comments on Overcoming Bias and Robin's own posts. If anyone provides additional ideas, I'll add them to the list.

(I have a more general point here, BTW, which is that predicting the far future is very difficult. Before thinking that some outcome is inevitable or highly likely, it's a good idea to repeatedly ask oneself "This is all the ways that I can think of why it may fail to come true. Am I sure that all of them have low probability and that I'm not missing anything?" There may be some scenario with a non-negligible probability that your brain simply overlooked when you first asked it.)

Singleton

A world government or superpower imposes a population control policy over the whole world.

Strong Security

Strong defensive technologies and doctrines (such as Mutually Assured Destruction) allow nations, communities, and maybe tribes and families to unilaterally limit their populations within their own borders, while holding off hordes of would-be invaders and immigrants.

Non-Human Capital

Maximizing the wealth and power of a nation requires an optimal mix of human and non-human capital. Nations that fail to adopt population controls find their relative wealth and power fade over time as their mixes deviate from the optimum (i.e., they find themselves spending too much resources on raising humans, and not enough on building machines), and either move to correct this or are taken over by stronger powers. (I believe that historically this was the reason China adopted its one-child policy.)

Unlimited Growth

We don't completely understand the laws of physics, nor the nature of value. There turns out to be some way for economic growth to continue without limit. (Robin himself once wrote "I know of no law limiting economic value per atom" but apparently changed his mind later.)

Selfish Memes

Memes that manage to divert people's resources away from biological reproduction and towards memetic reproduction will have an advantage over memes that don't. On the other hand, genes that manage to block such memes will ha...
Solutions to Political Problems As Counterfactuals
by Scott Alexander — 2009-09-25T17:21:50.038Z

A mathematician wakes up to find his house on fire. He frantically looks around before seeing the fire extinguisher on the far wall of the room. "Aha!" he says, "a solution exists!" and goes back to sleep.

-- Popular math students' joke

There has been much discussion of coulds, woulds, and shoulds recently. Agents imagine different counterfactual states of their own minds or actions, then select the most desirable. Something similar seems to happen during political discussions, but the multiplicity of agents involved muddles it a little.
I recently read a letter to the editor in my local paper. The city was launching a public education campaign against binge drinking, and this letter writer thought that all the billboards and lectures and what-not were a waste of time. She said that instead of a flashy and expensive public awareness campaign, the real solution was for binge drinkers to take responsibility for their own actions and learn that there were ways to have fun that didn't involve alcohol.
This struck me as a misguided line of thinking. Consider this analogy: pretend that the city government was, instead, increasing the number of police to prevent terrorist attacks. And that the writer was arguing that no, we shouldn't get the police involved: the real solution was for terrorists to stop being so violent and attacking people. This would be a weird and completely useless response.
Attempts to solve political problems are counterfactuals in the same way attempts to solve other problems are. In Newcomb's Problem, I modify the "my decision" node and watch what happens to the "money in the box" and "money I get" nodes. When I say "Increasing local police would prevent terrorist attacks," I am modifying the "local police" node, and positing that this would have a certain inhibitory effect on the connected "terrorist attacks" node.
The hypothetical second letter-writer's argument, then, is that if we counterfactually modified the "terrorists' attitude" node, then the "terrorist attacks" node would change. This is correct but useless.

But it's harder to see exactly why it's useless. Consider the original argument "We should raise the number of police and train them in counter-terrorism techniques." In this case, I would be counterfactually modifying the attitudes of (for example) the chief of police. But I'm not the chief of police, any more t...
The utility curve of the human population
by dclayh — 2009-09-24T21:00:41.266Z

"Whoever saves a single life, it is as if he had saved the whole world."

—The Talmud, Sanhedrin 4:5

That was the epigraph Eliezer used on a perfectly nice post reminding us to shut up and multiply when valuing human lives, rather than relying on the (roughly) logarithmic amount of warm fuzzies we'd receive. Implicit in the expected utility calculation is the idea that the value of human lives scales linearly: indeed, Eliezer explicitly says, "I agree that one human life is of unimaginably high value. I also hold that two human lives are twice as unimaginably valuable."

However, in a comment on Wei Dai's brilliant recent post comparing boredom and altruism, Vladimir Nesov points out that "you can value lives sublinearly" and still make an expected utility calculation rather than relying on warm-fuzzy intuition. This got me thinking about just what the functional form of U(Nliving-persons) might be.

Attacking from the high end (the "marginal" calculation), it seems to me that the utility of human lives is actually superlinear to a modest degree1; that is, U(N+1)-U(N) > U(N)-U(N-1). As an example, consider a parent and young child. If you allow one of them to die, not only do you end that life, but you make the other one significantly worse off. But this generalizes: the marginal person (on average) produces positive net value to society (though being an employee, friend, spouse, etc.) in addition to accruing their own utilons, and economies of scale dictate that adding another person allows a little more specialization and hence a little more efficiency. I.e., the larger the pool of potential co-workers/friends/spouses is, the pickier everyone can be, and the better matches they're likely to end up with. Steven Landsburg (in Fair Play) uses a version of this argument to conclude that children have positive externalities and therefore people on average have fewer children than would be optimal.

In societies with readily available birth control, that is. And naturally, in societies which are insufficiently technological for each marginal person to be able to make a contribution, however indirect, to (e.g.) the food output, it's quite easy for the utility of lives to be sublinear, which is the classical Malthusian problem, and still very much with us in the poorer areas of the world. (In fa...
Minneapolis Meetup, This Saturday (26th) at 3:00 PM, University of Minnesota 
by FrankAdamek — 2009-09-24T20:18:58.532Z

The Minneapolis meetup will take place this Saturday the 26th at 3:00 PM in Coffman Memorial Union at the University of Minnesota. Directions and address can be found here, and we'll be meeting inside near Starbucks, next to the northwest entrance to the bookstore, and underneath the northwest entrance to the Union. That entrance is a separate glass-sided structure right next to Washington Avenue, not the multistory brick building, but you can get to Starbucks from either entrance; it's on level 2.

RSVP and we'll keep an extra eye out for you.
Boredom vs. Scope Insensitivity
by Wei Dai — 2009-09-24T11:45:54.121Z

How much would you pay to see a typical movie? How much would you pay to see it 100 times?

How much would you pay to save a random stranger’s life? How much would you pay to save 100 strangers?

If you are like a typical human being, your answers to both sets of questions probably exhibit failures to aggregate value linearly. In the first case, we call it boredom. In the second case, we call it scope insensitivity.

Eliezer has argued on separate occasions that one should be regarded as an obvious error to be corrected, and the other is a gift bestowed by evolution, to be treasured and safeguarded. Here, I propose to consider them side by side, and see what we can learn by doing that.

(Eliezer sometimes treats scope insensitivity as a simple arithmetical error that the brain commits, like in this quote: “the brain can't successfully multiply by eight and get a larger quantity than it started with”. Considering that the brain has little trouble multiplying by eight in other contexts and the fact that scope insensitivity starts with numbers as low as 2, it seems more likely that it’s not an error but an adaptation, just like boredom.)

The nonlinearities in boredom and scope insensitivity both occur at two different levels. On the affective or hedonic level, our emotions fail to respond in a linear fashion to the relevant input. Watching a movie twice doesn’t give us twice the pleasure of watching it once, nor does saving two lives feel twice as good as saving one life. And on the level of decision making and revealed preferences, we fail to act as if our utilities scale linearly with the number of times we watch a movie, or the number of lives we save.

Note that these two types of nonlinearities are logically distinct, and it seems quite possible to have one without the other. The refrain “shut up and multiply” is an illustration of this. It exhorts (or reminds) us to value lives directly and linearly in our utility functions and decisions, instead of only valuing the sublinear emotions we get from saving lives.

We sometimes feel bad that we aren’t sufficiently empathetic. Similarly, we feel bad about some of our boredoms. For example, consider a music lover who regrets no longer being as deeply affected by his favorite piece of music as when he first heard it, or a wife who wishes she was still as deeply in love w...
Anthropic reasoning and correlated decision making
by Stuart_Armstrong — 2009-09-23T15:10:41.114Z

There seems to be some confusion on how to deal with correlated decision making - such as with absent-minded drivers and multiple copies of yourself; any situation in which many agents will allreach the same decision. Building on Nick Bostrom's division-of-responsibility principle mentioned in Outlawing Anthropics, I propose the following correlated decision principle:
CDP: If you are part of a group of N individuals whose decision is perfectly correlated, then you should reason as if you had a 1/N chance of being the dictator of the group (in which case your decision is applied to all) and a (N-1)/N chance of being a dictatee (in which case your decision is ignored).
What justification could there be to this principle? A simple thought experiment: imagine if you were one of N individuals who had to make a decision in secret. One of the decisions is opened at random, the others are discarded, and each person has his mind modified to believe that what was decided was in fact what they decided. This process is called a "dictator filter".
If you apply this dictator filter any situation S, then in "S + dictator filter", you should reason as in the CDP. If you apply it to perfectly correlated decision making, however, then the dictator filter changes nothing at all to anyone's decision - hence we should treat "perfectly correlated" as isomorphic to "perfectly correlated + dictator filter", which establishes the CDP.
Used alongside the SIA, this solves many puzzles on this blog, without needing advanced decision theory.
For instance, the situation in Outlawing Anthropics is simple: the SIA implies the 90% view, giving you a 90% chance of being in a group of 18, and a 10% of being in a group of two. Then you were offered a deal in which $3 is stolen from the red rooms, and $1 given to the green rooms. The initial expected gain from accepting the deal was -$20; the problem came that when you woke up in a green room, you were far more likely to be in the group of 18, giving an expected gain of +$5.60. The CDP cancels out this effect, returning you to an expected individual gain of -$2, and a global expected gain of -$20.
The Absent-Minded driver problem is even more interesting, and requires a more subtle reasoning. The SIA implies that if your probability of continuing is p, then the chance that you are at the first intersection is 1/(1+p), while the chance that you are a...
Avoiding doomsday: a "proof" of the self-indication assumption
by Stuart_Armstrong — 2009-09-23T14:54:32.288Z

EDIT: This post has been superceeded by this one.

The doomsday argument, in its simplest form, claims that since 2/3 of all humans will be in the final 2/3 of all humans, we should conclude it is more likely we are in the final two thirds of all humans who’ve ever lived, than in the first third. In our current state of quasi-exponential population growth, this would mean that we are likely very close to the final end of humanity. The argument gets somewhat more sophisticated than that, but that's it in a nutshell.
There are many immediate rebuttals that spring to mind - there is something about the doomsday argument that brings out the certainty in most people that it must be wrong. But nearly all those supposed rebuttals are erroneous (see Nick Bostrom's book Anthropic Bias: Observation Selection Effects in Science and Philosophy). Essentially the only consistent low-level rebuttal to the doomsday argument is to use the self indication assumption (SIA).
The non-intuitive form of SIA simply says that since you exist, it is more likely that your universe contains many observers, rather than few; the more intuitive formulation is that you should consider yourself as a random observer drawn from the space of possible observers (weighted according to the probability of that observer existing).
Even in that form, it may seem counter-intuitive; but I came up with a series of small steps leading from a generally accepted result straight to the SIA. This clinched the argument for me. The starting point is:

A - A hundred people are created in a hundred rooms. Room 1 has a red door (on the outside), the outsides of all other doors are blue. You wake up in a room, fully aware of these facts; what probability should you put on being inside a room with a blue door?

Here, the probability is certainly 99%. But now consider the situation:
B - same as before, but an hour after you wake up, it is announced that a coin will be flipped, and if it comes up heads, the guy behind the red door will be killed, and if it comes up tails, everyone behind a blue door will be killed. A few minutes later, it is announced that whoever was to be killed has been killed. What are your odds of being blue-doored now?
There should be no difference from A; since your odds of dying are exactly fifty-fifty whether you are blue-doored or red-doored, your probability estimate should not change upon bei...
‘Clearly’ covers murky thought
by KatjaGrace — 2009-09-22T18:26:26.000Z

Why do we point out that statements we are making are obvious? If a statement is actually obvious, there should rarely reason to point the statement out, let alone that it is obvious. It’s obviousness should be obvious. It seems that a person often emphasizes that a statement is obvious when they would prefer not be required to defend it. Sometimes this is just because it is obvious once you know their field but a lot of effort to explain to someone who doesn’t, but often it’s just that the explanation is not obvious to them.

But saying ‘obviously’ is too obvious. A better word is ‘clearly’. ‘Clearly’ sounds transparent and innocent. In reality it is a more subtle version of ‘obviously’.

I have noticed this technique used well in published philosophy from time to time. If getting to your conclusion is going to require assuming your conclusion is true, ‘clearly’ suggests to the reader that they not think over that step too closely.

For instance Michael Huemer in Ethical Intuitionism, while arguing that moral subjectivism is wrong, for the purpose of demonstrating that ethical intuitionism is right:

Traditionally, cultural relativists have been charged with endorsing such statements as,

If society were to approve of eating children, then eating children would be good.

which is clearly false.

Notice that ‘false’ seemingly means that it is false according to his intuition; the thing which he is trying to argue for the reliability of. If he just said ‘which is false’, the reader may wonder where, in a book on establishing a basis for ethical truth, this source of falsity may have popped from. ‘Clearly’ says that they needn’t worry about it.