trentmkelly/LessWrong-43k huggingface dataset:
What is Evidence?
by Eliezer Yudkowsky — 2007-09-22T06:43:31.000Z

The sentence “snow is white” is true if and only if snow is white.

—Alfred Tarski

To say of what is, that it is, or of what is not, that it is not, is true.

—Aristotle, Metaphysics IV

Walking along the street, your shoelaces come untied. Shortly thereafter, for some odd reason, you start believing your shoelaces are untied. Light leaves the Sun and strikes your shoelaces and bounces off; some photons enter the pupils of your eyes and strike your retina; the energy of the photons triggers neural impulses; the neural impulses are transmitted to the visual-processing areas of the brain; and there the optical information is processed and reconstructed into a 3D model that is recognized as an untied shoelace. There is a sequence of events, a chain of cause and effect, within the world and your brain, by which you end up believing what you believe. The final outcome of the process is a state of mind which mirrors the state of your actual shoelaces.

What is evidence? It is an event entangled, by links of cause and effect, with whatever you want to know about. If the target of your inquiry is your shoelaces, for example, then the light entering your pupils is evidence entangled with your shoelaces. This should not be confused with the technical sense of “entanglement” used in physics—here I’m just talking about “entanglement” in the sense of two things that end up in correlated states because of the links of cause and effect between them.

Not every influence creates the kind of “entanglement” required for evidence. It’s no help to have a machine that beeps when you enter winning lottery numbers, if the machine also beeps when you enter losing lottery numbers. The light reflected from your shoes would not be useful evidence about your shoelaces, if the photons ended up in the same physical state whether your shoelaces were tied or untied.

To say it abstractly: For an event to be evidence about a target of inquiry, it has to happen differently in a way that’s entangled with the different possible states of the target. (To say it technically: There has to be Shannon mutual information between the evidential event and the target of inquiry, relative to your current state of uncertainty about both of them.)

Entanglement can be contagious when processed correctly, which is why you need eyes and a brain. If photons reflect off your shoelaces and hit a rock, the rock won’t ch...
Burdensome Details
by Eliezer Yudkowsky — 2007-09-20T23:46:06.000Z

Merely corroborative detail, intended to give artistic verisimilitude to an otherwise bald and unconvincing narrative . . .

—Pooh-Bah, in Gilbert and Sullivan’s The Mikado

The conjunction fallacy is when humans assign a higher probability to a proposition of the form “A and B” than to one of the propositions “A” or “B” in isolation, even though it is a theorem that conjunctions are never likelier than their conjuncts. For example, in one experiment, 68% of the subjects ranked it more likely that “Reagan will provide federal support for unwed mothers and cut federal support to local governments” than that “Reagan will provide federal support for unwed mothers.”1

A long series of cleverly designed experiments, which weeded out alternative hypotheses and nailed down the standard interpretation, confirmed that conjunction fallacy occurs because we “substitute judgment of representativeness for judgment of probability.”2 By adding extra details, you can make an outcome seem more characteristic of the process that generates it. You can make it sound more plausible that Reagan will support unwed mothers, by adding the claim that Reagan will also cut support to local governments. The implausibility of one claim is compensated by the plausibility of the other; they “average out.”

Which is to say: Adding detail can make a scenario sound more plausible, even though the event necessarily becomes less probable.

If so, then, hypothetically speaking, we might find futurists spinning unconscionably plausible and detailed future histories, or find people swallowing huge packages of unsupported claims bundled with a few strong-sounding assertions at the center.

If you are presented with the conjunction fallacy in a naked, direct comparison, then you may succeed on that particular problem by consciously correcting yourself. But this is only slapping a band-aid on the problem, not fixing it in general.

In the 1982 experiment where professional forecasters assigned systematically higher probabilities to “Russia invades Poland, followed by suspension of diplomatic relations between the USA and the USSR” than to “Suspension of diplomatic relations between the USA and the USSR,” each experimental group was only presented with one proposition.3 What strategy could these forecasters have foll...
Conjunction Controversy (Or, How They Nail It Down)
by Eliezer Yudkowsky — 2007-09-20T02:41:38.000Z

Followup to: Conjunction Fallacy

When a single experiment seems to show that subjects are guilty of some horrifying sinful bias - such as thinking that the proposition "Bill is an accountant who plays jazz" has a higher probability than "Bill is an accountant" - people may try to dismiss (not defy) the experimental data. Most commonly, by questioning whether the subjects interpreted the experimental instructions in some unexpected fashion - perhaps they misunderstood what you meant by "more probable".

Experiments are not beyond questioning; on the other hand, there should always exist some mountain of evidence which suffices to convince you. It's not impossible for researchers to make mistakes. It's also not impossible for experimental subjects to be really genuinely and truly biased. It happens. On both sides, it happens. We're all only human here.

If you think to extend a hand of charity toward experimental subjects, casting them in a better light, you should also consider thinking charitably of scientists. They're not stupid, you know. If you can see an alternative interpretation, they can see it too. This is especially important to keep in mind when you read about a bias and one or two illustrative experiments in a blog post. Yes, if the few experiments you saw were all the evidence, then indeed you might wonder. But you might also wonder if you're seeing all the evidence that supports the standard interpretation. Especially if the experiments have dates on them like "1982" and are prefaced with adjectives like "famous" or "classic".

So! This is a long post. It is a long post because nailing down a theory requires more experiments than the one or two vivid illustrations needed to merely explain. I am going to cite maybe one in twenty of the experiments that I've read about, which is maybe a hundredth of what's out there. For more information, see Tversky and Kahneman (1983) or Kahneman and Frederick (2002), both available online, from which this post is primarily drawn.

Here is (probably) the single most questioned experiment in the literature of heuristics and biases, which I reproduce here exactly as it appears in Tversky and Kahneman (1982):

Linda is 31 years old, single, outspoken, and very bright. She majored in philosophy. As a student, she

...
Conjunction Fallacy
by Eliezer Yudkowsky — 2007-09-19T01:54:38.000Z

The following experiment has been slightly modified for ease of blogging. You are given the following written description, which is assumed true:

Bill is 34 years old. He is intelligent, but unimaginative, compulsive, and generally lifeless. In school, he was strong in mathematics but weak in social studies and humanities.

No complaints about the description, please, this experiment was done in 1974. Anyway, we are interested in the probability of the following propositions, which may or may not be true, and are not mutually exclusive or exhaustive:

A: Bill is an accountant.
B: Bill is a physician who plays poker for a hobby.
C: Bill plays jazz for a hobby.
D: Bill is an architect.
E: Bill is an accountant who plays jazz for a hobby.
F: Bill climbs mountains for a hobby.

Take a moment before continuing to rank these six propositions by probability, starting with the most probable propositions and ending with the least probable propositions. Again, the starting description of Bill is assumed true, but the six propositions may be true or untrue (they are not additional evidence) and they are not assumed mutually exclusive or exhaustive.

In a very similar experiment conducted by Tversky and Kahneman (1982), 92% of 94 undergraduates at the University of British Columbia gave an ordering with A > E > C. That is, the vast majority of subjects indicated that Bill was more likely to be an accountant than an accountant who played jazz, and more likely to be an accountant who played jazz than a jazz player. The ranking E > C was also displayed by 83% of 32 grad students in the decision science program of Stanford Business School, all of whom had taken advanced courses in probability and statistics.

There is a certain logical problem with saying that Bill is more likely to be an account who plays jazz, than he is to play jazz. The conjunction rule of probability theory states that, for all X and Y, P(X&Y) <= P(Y). That is, the probability that X and Y are simultaneously true, is always less than or equal to the probability that Y is true. Violating this rule is called a conjunction fallacy.

Imagine a group of 100,000 people, all of whom fit Bill's description (except for the name, perhaps). If you take the subset of all these persons who play jazz, and the subs...
Kahneman's Planning Anecdote
by Eliezer Yudkowsky — 2007-09-17T16:39:07.000Z

Followup to: Planning Fallacy

From "Timid Choices and Bold Forecasts: Cognitive Perspective on Risk Taking" by Nobel Laureate Daniel Kahneman and Dan Lovallo, in a discussion on "Inside and Outside Views":

In 1976 one of us (Daniel Kahneman) was involved in a project designed to develop a curriculum for the study of judgment and decision making under uncertainty for high schools in Israel. When the team had been in operation for about a year, with some significant achievements already to its credit, the discussion at one of the team meetings turned to the question of how long the project would take. To make the debate more useful, I asked everyone to indicate on a slip of paper their best estimate of the number of months that would be needed to bring the project to a well-defined stage of completion: a complete draft ready for submission to the Ministry of education. The estimates, including my own, ranged from 18 to 30 months.

At this point I had the idea of turning to one of our members, a distinguished expert in curriculum development, asking him a question phrased about as follows:

"We are surely not the only team to have tried to develop a curriculum where none existed before. Please try to recall as many such cases as you can. Think of them as they were in a stage comparable to ours at present. How long did it take them, from that point, to complete their projects?"

After a long silence, something much like the following answer was given, with obvious signs of discomfort: "First, I should say that not all teams that I can think of in a comparable stage ever did complete their task. About 40% of them eventually gave up. Of the remaining, I cannot think of any that was completed in less than seven years, nor of any that took more than ten."

In response to a further question, he answered: "No, I cannot think of any relevant factor that distinguishes us favorably from the teams I have been thinking about. Indeed, my impression is that we are slightly below average in terms of our resources and potential."

Facing the facts can be intolerably demoralizing. The participants in the meeting had professional expertise in the logic of forecasting, and none even ventured to question the relevance of the forecast implied by our expert's statistics: an even chance of failure, and a completi

...
Planning Fallacy
by Eliezer Yudkowsky — 2007-09-17T07:06:20.000Z

The Denver International Airport opened 16 months late, at a cost overrun of $2 billion.1

The Eurofighter Typhoon, a joint defense project of several European countries, was delivered 54 months late at a cost of $19 billion instead of $7 billion.

The Sydney Opera House may be the most legendary construction overrun of all time, originally estimated to be completed in 1963 for $7 million, and finally completed in 1973 for $102 million.2

Are these isolated disasters brought to our attention by selective availability? Are they symptoms of bureaucracy or government incentive failures? Yes, very probably. But there’s also a corresponding cognitive bias, replicated in experiments with individual planners.

Buehler et al. asked their students for estimates of when they (the students) thought they would complete their personal academic projects.3 Specifically, the researchers asked for estimated times by which the students thought it was 50%, 75%, and 99% probable their personal projects would be done. Would you care to guess how many students finished on or before their estimated 50%, 75%, and 99% probability levels?

13% of subjects finished their project by the time they had assigned a 50% probability level;
19% finished by the time assigned a 75% probability level;
and only 45% (less than half!) finished by the time of their 99% probability level.

As Buehler et al. wrote, “The results for the 99% probability level are especially striking: Even when asked to make a highly conservative forecast, a prediction that they felt virtually certain that they would fulfill, students’ confidence in their time estimates far exceeded their accomplishments.”4

More generally, this phenomenon is known as the “planning fallacy.” The planning fallacy is that people think they can plan, ha ha.

A clue to the underlying problem with the planning algorithm was uncovered by Newby-Clark et al., who found that

Asking subjects for their predictions based on realistic “best guess” scenarios; and
Asking subjects for their hoped-for “best case” scenarios . . .

. . . produced indistinguishable results.5

When people are asked for a “realistic” scenario, they envision everything going exactly as planned, with no unexpected delays or unforeseen catastrophes—the same vision as their “best case.”

Reality, it turns out, usually delivers results somewhat worse than the “worst case.”

Unlike most cognitive biases, we k...
Why I'm Blooking
by Eliezer Yudkowsky — 2007-09-15T17:49:02.000Z

Yesterday being my 100th Overcoming Bias post, it seems an opportune time to answer a commenter's question: Why am I posting?

For a long time I've suffered from writer's molasses. Like writer's block, only instead of not writing, I write very slooowly.
At least when it comes to writing Documents - papers, book chapters,
website material. If I haven't published a hundred papers, it's not
for lack of a hundred ideas, but because writing one paper - at my
current pace - takes four months full time. I sometimes wonder if I
could become a respectable academic if I wrote at a respectable pace.

Oddly enough, I can write most emails around as fast as I type.
Such disorders are hard to self-diagnose, but I suspect that part of
the problem is that on Documents I repeatedly reread and tweak material
I've already written, instead of writing new material. James Hogan (an
SF author) once told me that he was more productive on a typewriter
than a word processor, because the typewriter prevented him from
tweaking until the second draft.

A blook is a collection of blog posts that have been edited into a
book. Logically, then, publishing a book as a series of blog posts
ought to be known as "blooking".

It would be more precise to say that I'm generating raw material
to be edited into a book, and collecting some feedback along the way. I
make no promises for this project. (I hate promising anything unless I
have already done it.) The first part of the plan, generating the
raw material as blog posts, has proceeded at a respectable pace so
far. Will I be able to edit the posts into chapters, so long as all
the raw material is there? Will I be able to generate all the raw
material, or will the project, ahem, "blog down"?

In August I decided that I was going to write one blog post per day
for Overcoming Bias. This challenge began to hone my writing speed
somewhat - for example, I would look at the clock and try not to take
longer than an hour... or three hours... but nonetheless I began to
feel the need to shove the post out the door instead of perfecting it further. This is necessary and proper.

Near the end of August, I faced a new challenge - I also had to
prepare two talks for the Singularity Summit 2007 (Sep 8-9). Those
were actual Documents. I knew, from previous experience, t...
Doublethink (Choosing to be Biased)
by Eliezer Yudkowsky — 2007-09-14T20:05:13.000Z

An oblong slip of newspaper had appeared between O'Brien's fingers. For perhaps five seconds it was within the angle of Winston's vision. It was a photograph, and there was no question of its identity. It was the photograph. It was another copy of the photograph of Jones, Aaronson, and Rutherford at the party function in New York, which he had chanced upon eleven years ago and promptly destroyed. For only an instant it was before his eyes, then it was out of sight again. But he had seen it, unquestionably he had seen it! He made a desperate, agonizing effort to wrench the top half of his body free. It was impossible to move so much as a centimetre in any direction. For the moment he had even forgotten the dial. All he wanted was to hold the photograph in his fingers again, or at least to see it.

'It exists!' he cried.

'No,' said O'Brien.

He stepped across the room.

There was a memory hole in the opposite wall. O'Brien lifted the grating. Unseen, the frail slip of paper was whirling away on the current of warm air; it was vanishing in a flash of flame. O'Brien turned away from the wall.

'Ashes,' he said. 'Not even identifiable ashes. Dust. It does not exist. It never existed.'

'But it did exist! It does exist! It exists in memory. I remember it. You remember it.'

'I do not remember it,' said O'Brien.

Winston's heart sank. That was doublethink. He had a feeling of deadly helplessness. If he could have been certain that O'Brien was lying, it would not have seemed to matter. But it was perfectly possible that O'Brien had really forgotten the photograph. And if so, then already he would have forgotten his denial of remembering it, and forgotten the act of forgetting. How could one be sure that it was simple trickery? Perhaps that lunatic dislocation in the mind could really happen: that was the thought that defeated him.

—George Orwell, 1984

What if self-deception helps us be happy? What if just running out and overcoming bias will make us—gasp!—unhappy? Surely, true wisdom would be second-order rationality, choosing when to be rational. That way you can decide which cognitive biases should govern you, to maximize your happiness.

Leaving the morality aside, I doubt such a lunatic dislocation in the mind could really happen.

Second-order rationality implies that at some point, you will think to yourself, "And now, I will irrationally believe ...
Human Evil and Muddled Thinking
by Eliezer Yudkowsky — 2007-09-13T23:43:13.000Z

George Orwell saw the descent of the civilized world into totalitarianism, the conversion or corruption of one country after another; the boot stamping on a human face, forever, and remember that it is forever. You were born too late to remember a time when the rise of totalitarianism seemed unstoppable, when one country after another fell to secret police and the thunderous knock at midnight, while the professors of free universities hailed the Soviet Union’s purges as progress. It feels as alien to you as fiction; it is hard for you to take seriously. Because, in your branch of time, the Berlin Wall fell. And if Orwell’s name is not carved into one of those stones, it should be.

Orwell saw the destiny of the human species, and he put forth a convulsive effort to wrench it off its path. Orwell’s weapon was clear writing. Orwell knew that muddled language is muddled thinking; he knew that human evil and muddled thinking intertwine like conjugate strands of DNA:1

In our time, political speech and writing are largely the defence of the indefensible. Things like the continuance of British rule in India, the Russian purges and deportations, the dropping of the atom bombs on Japan, can indeed be defended, but only by arguments which are too brutal for most people to face, and which do not square with the professed aims of the political parties. Thus political language has to consist largely of euphemism, question-begging and sheer cloudy vagueness. Defenceless villages are bombarded from the air, the inhabitants driven out into the countryside, the cattle machine-gunned, the huts set on fire with incendiary bullets: this is called PACIFICATION . . .

Orwell was clear on the goal of his clarity:

If you simplify your English, you are freed from the worst follies of orthodoxy. You cannot speak any of the necessary dialects, and when you make a stupid remark its stupidity will be obvious, even to yourself.

To make our stupidity obvious, even to ourselves—this is the heart of Overcoming Bias.

Evil sneaks, hidden, through the unlit shadows of the mind. We look back with the clarity of history, and weep to remember the planned famines of Stalin and Mao, which killed tens of millions. We call this evil, because it was done by deliberate human intent to inflict pain and death upon innocent human beings. We call this evil, because of the revulsion that we feel against it, looking...
Rationality and the English Language
by Eliezer Yudkowsky — 2007-09-12T22:55:57.000Z

The other day, someone commented that my writing reminded them of George Orwell’s “Politics and the English Language.”1 I was honored. Especially since I’d already thought of today’s topic.

If you really want an artist’s perspective on rationality, then read Orwell; he is mandatory reading for rationalists as well as authors. Orwell was not a scientist, but a writer; his tools were not numbers, but words; his adversary was not Nature, but human evil. If you wish to imprison people for years without trial, you must think of some other way to say it than “I’m going to imprison Mr. Jennings for years without trial.” You must muddy the listener’s thinking, prevent clear images from outraging conscience. You say, “Unreliable elements were subjected to an alternative justice process.”

Orwell was the outraged opponent of totalitarianism and the muddy thinking in which evil cloaks itself—which is how Orwell’s writings on language ended up as classic rationalist documents on a level with Feynman, Sagan, or Dawkins.

“Writers are told to avoid usage of the passive voice.” A rationalist whose background comes exclusively from science may fail to see the flaw in the previous sentence; but anyone who’s done a little writing should see it right away. I wrote the sentence in the passive voice, without telling you who tells authors to avoid passive voice. Passive voice removes the actor, leaving only the acted-upon. “Unreliable elements were subjected to an alternative justice process”—subjected by whom? What does an “alternative justice process” do? With enough static noun phrases, you can keep anything unpleasant from actually happening.

Journal articles are often written in passive voice. (Pardon me, some scientists write their journal articles in passive voice. It’s not as if the articles are being written by no one, with no one to blame.) It sounds more authoritative to say “The subjects were administered Progenitorivox” than “I gave each college student a bottle of 20 Progenitorivox, and told them to take one every night until they were gone.” If you remove the scientist from the description, that leaves only the all-important data. But in reality the scientist is there, and the subjects are college students, and the Progenitorivox wasn’t “administered” but handed over with instructions. Passive voice obscures reality.

Judging from the comments I get, someone will protest t...