trentmkelly/LessWrong-43k huggingface dataset:
Circular Altruism
by Eliezer Yudkowsky — 2008-01-22T18:00:00.000Z

Followup to: Torture vs. Dust Specks, Zut Allais, Rationality Quotes 4

Suppose that a disease, or a monster, or a war, or something, is killing people. And suppose you only have enough resources to implement one of the following two options:

Save 400 lives, with certainty.

Save 500 lives, with 90% probability; save no lives, 10% probability.

Most people choose option 1. Which, I think, is foolish; because if you multiply 500 lives by 90% probability, you get an expected value of 450 lives, which exceeds the 400-life value of option 1. (Lives saved don't diminish in marginal utility, so this is an appropriate calculation.)

"What!" you cry, incensed. "How can you gamble with human lives? How can you think about numbers when so much is at stake? What if that 10% probability strikes, and everyone dies? So much for your damned logic! You're following your rationality off a cliff!"

Ah, but here's the interesting thing. If you present the options this way:

100 people die, with certainty.

90% chance no one dies; 10% chance 500 people die.

Then a majority choose option 2. Even though it's the same gamble. You see, just as a certainty of saving 400 lives seems to feel so much more comfortable than an unsure gain, so too, a certain loss feels worse than an uncertain one.

You can grandstand on the second description too: "How can you condemn 100 people to certain death when there's such a good chance you can save them? We'll all share the risk! Even if it was only a 75% chance of saving everyone, it would still be worth it - so long as there's a chance - everyone makes it, or no one does!"

You know what? This isn't about your feelings. A human life, with all its joys and all its pains, adding up over the course of decades, is worth far more than your brain's feelings of comfort or discomfort with a plan. Does computing the expected utility feel too cold-blooded for your taste? Well, that feeling isn't even a feather in the scales, when a life is at stake. Just shut up and multiply.

Previously on Overcoming Bias, I asked what was the least bad, bad thing that could happen, and suggested that it was getting a dust speck in your eye that irritated you for a fraction of a second, barely long enough to notice, before it got blinked away. And conversely, a ver...
Against Discount Rates
by Eliezer Yudkowsky — 2008-01-21T10:00:00.000Z

I've never been a fan of the notion that we should (normatively) have a discount rate in our pure preferences - as opposed to a pseudo-discount rate arising from monetary inflation, or from opportunity costs of other investments, or from various probabilistic catastrophes that destroy resources or consumers. The idea that it is literally, fundamentally 5% more important that a poverty-stricken family have clean water in 2008, than that a similar family have clean water in 2009, seems like pure discrimination to me - just as much as if you were to discriminate between blacks and whites.

And there's worse: If your temporal discounting follows any curve other than the exponential, you'll have time-inconsistent goals that force you to wage war against your future selves - preference reversals - cases where your self of 2008 will pay a dollar to ensure that your future self gets option A in 2011 rather than B in 2010; but then your future self in 2009 will pay another dollar to get B in 2010 rather than A in 2011.

But a 5%-per-year discount rate, compounded exponentially, implies that it is worth saving a single person from torture today, at the cost of 168 people being tortured a century later, or a googol persons being tortured 4,490 years later.

People who deal in global catastrophic risks sometimes have to wrestle with the discount rate assumed by standard economics. Is a human civilization spreading through the Milky Way, 100,000 years hence - the Milky Way being about 100K lightyears across - really to be valued at a discount of 10-2,227 relative to our own little planet today?

And when it comes to artificial general intelligence... I encounter wannabe AGI-makers who say, "Well, I don't know how to make my math work for an infinite time horizon, so... um... I've got it! I'll build an AGI whose planning horizon cuts out in a thousand years." Such a putative AGI would be quite happy to take an action that causes the galaxy to explode, so long as the explosion happens at least 1,001 years later. (In general, I've observed that most wannabe AGI researchers confronted with Singularity-level problems ponder for ten seconds and then propose the sort of clever programming trick one would use for data-mining the Netflix Prize, without asking if it makes deep sense for Earth-originating civilization over the next million year...
Allais Malaise
by Eliezer Yudkowsky — 2008-01-21T00:40:01.000Z

Continuation of: The Allais Paradox, Zut Allais!

Judging by the comments on Zut Allais, I failed to emphasize the points that needed emphasis.

The problem with the Allais Paradox is the incoherent pattern 1A > 1B, 2B > 2A. If you need $24,000 for a lifesaving operation and an extra $3,000 won't help that much, then you choose 1A > 1B and 2A > 2B. If you have a million dollars in the bank account and your utility curve doesn't change much with an extra $25,000 or so, then you should choose 1B > 1A and 2B > 2A. Neither the individual choice 1A > 1B, nor the individual choice 2B > 2A, are of themselves irrational. It's the combination that's the problem.

Expected utility is not expected dollars. In the case above, the utility-distance from $24,000 to $27,000 is a tiny fraction of the distance from $21,000 to $24,000. So, as stated, you should choose 1A > 1B and 2A > 2B, a quite coherent combination. The Allais Paradox has nothing to do with believing that every added dollar is equally useful. That idea has been rejected since the dawn of decision theory.

If satisfying your intuitions is more important to you than money, do whatever the heck you want. Drop the money over Niagara falls. Blow it all on expensive champagne. Set fire to your hair. Whatever. If the largest utility you care about is the utility of
feeling good about your decision, then any decision that feels good is
the right one. If you say that different trajectories to the same outcome "matter emotionally", then you're attaching an inherent utility to conforming to the brain's native method of optimization, whether or not it actually optimizes. Heck, running around in circles from preference reversals could feel really good too. But if you care enough about the stakes that winning is more important than your brain's good feelings about an intuition-conforming strategy, then use decision theory.

If you suppose the problem is different from the one presented
- that the gambles are untrustworthy and that, after this mistrust is
taken into account, the payoff probabilities are not as described -
then, obviously, you can make the answer anything you want.

Let's say you're dying of thirst, you only have $1.00, and you have
to choose between a vending machine that dispe...
Rationality Quotes 4
by Eliezer Yudkowsky — 2008-01-20T17:00:00.000Z

"Altruistic behavior: An act done without any intent for personal gain in any form. Altruism requires that there is no want for material, physical, spiritual, or egoistic gain."
-- Glossary of Zen

"The primary thing when you take a sword in your hands is your intention to cut the enemy, whatever the means. Whenever you parry, hit, spring, strike or touch the enemy's cutting sword, you must cut the enemy in the same movement. It is essential to attain this. If you think only of hitting, springing, striking or touching the enemy, you will not be able actually to cut him. More than anything, you must be thinking of carrying your movement through to cutting him."
-- Miyamoto Musashi, The Book of Five Rings

"You can win with a long weapon, and yet you can also win with a short weapon. In short, the Way of the Ichi school is the spirit of winning, whatever the weapon and whatever its size."
-- Miyamoto Musashi, The Book of Five Rings

"Well," Rowlands said carefully, turning the Land-Rover into the road. "I am not at all sure what it is that is going on all around us, Will bach, or where it is leading. But those men who know anything at all about the Light also know that there is a fierceness to its power, like the bare sword of the law, or the white burning of the sun." Suddenly his voice sounded to Will very strong, and very Welsh. "At the very heart, that is. Other things, like humanity, and mercy, and charity, that most good men hold more precious than all else, they do not come first for the Light. Oh, sometimes they are there; often, indeed. But in the very long run the concern of you people is with the absolute good, ahead of all else. You are like fanatics. Your masters, at any rate. Like the old Crusaders -- oh, like certain groups in every belief, though this is not a matter of religion, of course. At the centre of the Light there is a cold white flame, just as at the centre of the Dark there is a great black pit bottomless as the Universe."
His warm, deep voice ended, and there was only the roar of the engine. Will looked out over the grey-misted fields, silent.
"There was a great long speech, now," John Rowlands said awkwardly. "But I was only saying, be careful not to forget that there are people in this valley who can ...
Zut Allais!
by Eliezer Yudkowsky — 2008-01-20T03:18:16.000Z

Huh! I was not expecting that response. Looks like I ran into an inferential distance.

It probably helps in interpreting the Allais Paradox to have absorbed more of the gestalt of the field of heuristics and biases, such as:

Experimental subjects tend to defend incoherent preferences even when they're really silly.

People put very high values on small shifts in probability away from 0 or 1 (the certainty effect).

Let's start with the issue of incoherent preferences - preference reversals, dynamic inconsistency, money pumps, that sort of thing.

Anyone who knows a little prospect theory will have no trouble constructing cases where people say they would prefer to play gamble A rather than gamble B; but when you ask them to price the gambles they put a higher value on gamble B than gamble A. There are different perceptual features that become salient when you ask "Which do you prefer?" in a direct comparison, and "How much would you pay?" with a single item.

My books are packed up for the move, but from what I remember, this should typically generate a preference reversal:

1/3 to win $18 and 2/3 to lose $1.50

19/20 to win $4 and 1/20 to lose $0.25

Most people will (IIRC) rather play 2 than 1. But if you ask them to price the bets separately - ask for a price at which they would be indifferent between having that amount of money, and having a chance to play the gamble - people will (IIRC) put a higher price on 1 than on 2. If I'm wrong about this exact example, nonetheless, there are plenty of cases where such a pattern is exhibited experimentally.

So first you sell them a chance to play bet 1, at their stated price. Then you offer to trade bet 1 for bet 2. Then you buy bet 2 back from them, at their stated price. Then you do it again. Hence the phrase, "money pump".

Or to paraphrase Steve Omohundro: If you would rather be in Oakland than San Francisco, and you would rather be in San Jose than Oakland, and you would rather be in San Francisco than San Jose, you're going to spend an awful lot of money on taxi rides.

Amazingly, people defend these preference patterns. Some subjects abandon them after the money-pump effect is pointed out - revise their price or revise their preference - but some subjects defend them.

On one occasion, gamblers in Las Vegas played these kinds of bets for real money, usin...
The Allais Paradox
by Eliezer Yudkowsky — 2008-01-19T03:05:32.000Z

Choose between the following two options:

1A. $24,000, with certainty.
1B. 33/34 chance of winning $27,000, and 1/34 chance of winning nothing.

Which seems more intuitively appealing? And which one would you choose in real life?

Now which of these two options would you intuitively prefer, and which would you choose in real life?

2A. 34% chance of winning $24,000, and 66% chance of winning nothing.
2B. 33% chance of winning $27,000, and 67% chance of winning nothing.

The Allais Paradox - as Allais called it, though it's not really a paradox - was one of the first conflicts between decision theory and human reasoning to be experimentally exposed, in 1953. I've modified it slightly for ease of math, but the essential problem is the same: Most people prefer 1A > 1B, and most people prefer 2B > 2A. Indeed, in within-subject comparisons, a majority of subjects express both preferences simultaneously.

This is a problem because the 2s are equal to a one-third chance of playing the 1s. That is, 2A is equivalent to playing gamble 1A with 34% probability, and 2B is equivalent to playing 1B with 34% probability.

Among the axioms used to prove that "consistent" decisionmakers can be viewed as maximizing expected utility, is the Axiom of Independence: If X is strictly preferred to Y, then a probability P of X and (1 - P) of Z should be strictly preferred to P chance of Y and (1 - P) chance of Z.

All the axioms are consequences, as well as antecedents, of a consistent utility function. So it must be possible to prove that the experimental subjects above can't have a consistent utility function over outcomes. And indeed, you can't simultaneously have:

U($24,000) > 33/34 U($27,000) + 1/34 U($0)

0.34 U($24,000) + 0.66 U($0) < 0.33 U($27,000) + 0.67 U($0)

These two equations are algebraically inconsistent, regardless of U, so the Allais Paradox has nothing to do with the diminishing marginal utility of money.

Maurice Allais initially defended the revealed preferences of the experimental subjects - he saw the experiment as exposing a flaw in the conventional ideas of utility, rather than exposing a flaw in human psychology. This was 1953, after all, and the heuristics-and-biases movement wouldn't really get started for another two decades. Allais ...
Rationality Quotes 3
by Eliezer Yudkowsky — 2008-01-18T02:20:32.000Z

"Reality is that which, when you stop believing in it, doesn't go away."
-- Philip K. Dick

"How many legs does a dog have, if you call the tail a leg? Four. Calling a tail a leg doesn't make it a leg."
-- Abraham Lincoln

"Faced with the choice of changing one's mind and proving that there is no need to do so, almost everyone gets busy on the proof."
-- John Kenneth Galbraith

"I'd rather live with a good question than a bad answer."
-- Aryeh Frimer

"It ain't a true crisis of faith unless things could just as easily go either way."
-- bunnyThor

"My best test for a libertarian so far is to ask what needs to be done to protect ancient sequoias. If you say you need to buy them, you pass."
-- Rafal Smigrodzki

"Mystical explanations are considered deep. The truth is that they are not even superficial."
-- Friedrich Nietzsche, The Gay Science

"Some people know better, and they still make the mistake. That's when ignorance becomes stupidity."
-- Aaron McBride

"Perfection is achieved, not when there is nothing left to add, but when there is nothing left to take away."
-- Antoine de Saint-Exupéry

"Skill is successfully walking a tightrope over Niagara Falls. Intelligence is not trying."
-- Unknown

"The fact that one apple added to one apple invariably gives two apples helps in the teaching of arithmetic, but has no bearing on the truth of the proposition that 1 + 1 = 2."
-- James R. Newman, The World of Mathematics

"Giving a person a high IQ is kind of like giving a person a million dollars. A few individuals will do something interesting with it, but most will piss it away on trinkets and pointless exercises."
-- J. Andrew Rogers

"Fatal stupidity is inefficient: idiots take other people out with them way too often."
-- Mike

"Surprises are things that you not only didn't know, but that contradict things you thought you knew. And so they're the most valuable sort of fact you can get. They're like a food that's not merely healthy, but counteracts...
Rationality Quotes 2
by Eliezer Yudkowsky — 2008-01-16T23:47:45.000Z

"I often have to arrange talks years in advance. If I am asked for a title, I suggest “The Current Crisis in the Middle East.” It has yet to fail."
-- Noam Chomsky

"We don't do science for the general public. We do it for each other. Good day."
-- Renato Dalbecco, complete text of interview with H F Judson

"Most witches don't believe in gods. They know that the gods exist, of course. They even deal with them occasionally. But they don't believe in them. They know them too well. It would be like believing in the postman."
-- Terry Pratchett, Witches Abroad

"He didn't reject the idea so much as not react to it and watch as it floated away."
-- David Foster Wallace, Infinite Jest

"People can't predict how long they will be happy with recently acquired objects, how long their marriages will last, how their new jobs will turn out, yet it's subatomic particles that they cite as "limits of prediction." They're ignoring a mammoth standing in front of them in favor of matter even a microscope would not allow them to see."
-- Nassim Taleb, The Black Swan

"I would strongly challenge the notion of “that's what's called growing up.” The most depressing part of the nomenclature around adolescence and college life is this bizarre connection between “experimentation” / “learning from your mistakes” and binge drinking, reckless sex, and drug use."
-- Ben Casnocha

"Behind every story of extraordinary heroism, there is a less exciting and more interesting story about the larger failures that made heroism necessary in the first place."
-- Black Belt Bayesian

"Anyone who claims that the brain is a total mystery should be slapped upside the head with the MIT Encyclopedia of the Cognitive Sciences. All one thousand ninety-six pages of it."
-- Tom McCabe

"Try explaining anything scientific to your friends -- you soon won't have any."
-- Soloport

"The definition of the word "meaning," is something that is conveyed. So, who is conveying this "meaning" that you speak of? To put it another way, if "life" is a painting, then who is painter? Whoever is t...
Rationality Quotes 1
by Eliezer Yudkowsky — 2008-01-16T07:41:57.000Z

I'll be moving to Redwood City, CA in a week, so forgive me if I don't get a regular post out every day between now and then. As a substitute offering, some items from my (offline) quotesfile:

"It appears to be a quite general principle that, whenever there is a randomized way of doing something, then there is a nonrandomized way that delivers better performance but requires more thought."
-- E. T. Jaynes

"When you're young, you look at television and think, There's a conspiracy. The networks have conspired to dumb us down. But when you get a little older, you realize that's not true. The networks are in business to give people exactly what they want. That's a far more depressing thought. Conspiracy is optimistic! You can shoot the bastards!"
-- Steve Jobs

"Saving a drowning child is no more a moral duty than understanding a syllogism is a logical one."
-- Sam Harris, The End of Faith

"Don't ask why (again, long story), but the number 5,479,863,282.86 has been stuck in my head for as long as I can remember."
-- Ultimatum479

"I've met these people, the ones from the glossy magazines. I've walked among them. I have seen, firsthand, their callow, empty lives. I have watched them from the shadows when they thought themselves alone. And I can tell you this: I'm afraid there is not one of them who would swap lives with you at gunpoint."
-- Neil Gaiman, Anansi Boys

"Before you can get to the end of this paragraph, another person will probably die because of what someone else believes about God."
-- Sam Harris, The End of Faith

"Someone walking down the street with absolutely no scars or calluses would look pretty odd. I suspect having a conversation with someone who'd never taken any emotional or mental damage would be even odder. The line between "experience" and "damage" is pretty thin."
-- Aliza, from the Open-Source Wish Project

"Fear and lies fester in darkness. The truth may wound, but it cuts clean."
-- Jacqueline Carey, Kushiel's Avatar

"I never make a prediction that can be proved wrong within 24 hours."
-- Louis Rukeyser

"You have been asking what you could do in the

...
Trust in Math
by Eliezer Yudkowsky — 2008-01-15T04:25:04.000Z

Followup to: Expecting Beauty

I was once reading a Robert Heinlein story - sadly I neglected to note down which story, but I do think it was a Heinlein - where one of the characters says something like, "Logic is a fine thing, but I have seen a perfectly logical proof that 2 = 1." Authors are not to be confused with characters, but the line is voiced by one of Heinlein's trustworthy father figures. I find myself worried that Heinlein may have meant it.

The classic proof that 2 = 1 runs thus. First, let x = y = 1. Then:

x = y

x2 = xy

x2 - y2 = xy - y2

(x + y)(x - y) = y(x - y)

x + y = y

2 = 1

Now, you could look at that, and shrug, and say, "Well, logic doesn't always work."

Or, if you felt that math had rightfully earned just a bit more credibility than that, over the last thirty thousand years, then you might suspect the flaw lay in your use of math, rather than Math Itself.

You might suspect that the proof was not, in fact, "perfectly logical".

The novice goes astray and says: "The Art failed me."
The master goes astray and says: "I failed my Art."

Is this - gasp! - faith? To believe that math is consistent, when you have seen with your own eyes a proof that it is not? Are you supposed to just ignore the contrary evidence, my good Bayesian?

As I have remarked before, it seems worthwhile to distinguish "faith" that the sun will rise in the east just like the last hundred thousand times observed, from "faith" that tomorrow a green goblin will give you a bag of gold doubloons. When first-order arithmetic has been observed to be internally consistent over the last ten million theorems proved in it, and you see a seeming proof of inconsistency, it is, perhaps, reasonable to double-check the proof.

You're not going to ignore the contrary evidence. You're going to double-check it. You're going to also take into account the last ten million times that first-order arithmetic has proven consistent, when you evaluate your new posterior confidence that 2 = 1 is not perfectly logical. On that basis, you are going to evaluate a high probability that, if you check for a flaw, you are likely to find one.

But isn't this motivated skepticism? The most fearful bane of students of bias? You're applying a stronger standard of checking to incon...