trentmkelly/LessWrong-43k huggingface dataset:
Fake Justification
by Eliezer Yudkowsky — 2007-11-01T03:57:37.000Z

Many Christians who’ve stopped really believing now insist that they revere the Bible as a source of ethical advice. The standard atheist reply is given by Sam Harris: “You and I both know that it would take us five minutes to produce a book that offers a more coherent and compassionate morality than the Bible does.”1 Similarly, one may try to insist that the Bible is valuable as a literary work. Then why not revere Lord of the Rings, a vastly superior literary work? And despite the standard criticisms of Tolkien’s morality, Lord of the Rings is at least superior to the Bible as a source of ethics. So why don’t people wear little rings around their neck, instead of crosses? Even Harry Potter is superior to the Bible, both as a work of literary art and as moral philosophy.2

“How can you justify buying a $1 million gem-studded laptop,” you ask your friend, “when so many people have no laptops at all?” And your friend says, “But think of the employment that this will provide—to the laptop maker, the laptop maker’s advertising agency—and then they’ll buy meals and haircuts—it will stimulate the economy and eventually many people will get their own laptops.” But it would be even more efficient to buy 5,000 One Laptop Per Child laptops, thus providing employment to the OLPC manufacturers and giving out laptops directly.

I’ve touched before on the failure to look for third alternatives. But this is not really motivated stopping. Calling it “motivated stopping” would imply that there was a search carried out in the first place.

In “The Bottom Line,” I observed that only the real determinants of our beliefs can ever influence our real-world accuracy. Only the real determinants of our actions can influence our effectiveness in achieving our goals. Someone who buys a million-dollar laptop was really thinking, “Ooh, shiny,” and that was the one true causal history of their decision to buy a laptop. No amount of “justification” can change this, unless the justification is a genuine, newly running search process that can change the conclusion. Really change the conclusion. Most criticism carried out from a sense of duty is more of a token inspection than anything else. Free elections in a one-party country.

To genuinely justify the Bible as an object of laudation by reference to its literary quality, you would have to somehow perform a neutral reading through candidate books ...
A Case Study of Motivated Continuation
by Eliezer Yudkowsky — 2007-10-31T01:27:19.000Z

I am not wholly unsympathetic to the many commenters in Torture vs. Dust Specks who argued that it is preferable to inflict dust specks upon the eyes of 3^^^3 (amazingly huge but finite number of) people, rather than torture one person for 50 years. If you think that a dust speck is simply of no account unless it has other side effects - if you literally do not prefer zero dust specks to one dust speck - then your position is consistent. (Though I suspect that many speckers would have expressed a preference if they hadn't known about the dilemma's sting.)

So I'm on board with the commenters who chose TORTURE, and I can understand the commenters who chose SPECKS.

But some of you said the question was meaningless; or that all morality was arbitrary and subjective; or that you needed more information before you could decide; or you talked about some other confusing aspect of the problem; and then you didn't go on to state a preference.

Sorry. I can't back you on that one.

If you actually answer the dilemma, then no matter which option you choose, you're giving something up. If you say SPECKS, you're giving up your claim on a certain kind of utilitarianism; you may worry that you're not being rational enough, or that others will accuse you of failing to comprehend large numbers. If you say TORTURE, you're accepting an outcome that has torture in it.

I falsifiably predict that of the commenters who dodged, most of them saw some specific answer - either TORTURE or SPECKS - that they flinched away from giving. Maybe for just a fraction of a second before the question-confusing operation took over, but I predict the flinch was there. (To be specific: I'm not predicting that you knew, and selected, and have in mind right now, some particular answer you're deliberately not giving. I'm predicting that your thinking trended toward a particular uncomfortable answer, for at least one fraction of a second before you started finding reasons to question the dilemma itself.)

In "bioethics" debates, you very often see experts on bioethics discussing what they see as the pros and cons of, say, stem-cell research; and then, at the conclusion of their talk, they gravely declare that more debate is urgently needed, with participation from all stakeholders. If you actually come to a conclusion, if you actually argue fo...
Torture vs. Dust Specks
by Eliezer Yudkowsky — 2007-10-30T02:50:28.000Z

"What's the worst that can happen?" goes the optimistic saying. It's probably a bad question to ask anyone with a creative imagination. Let's consider the problem on an individual level: it's not really the worst that can happen, but would nonetheless be fairly bad, if you were horribly tortured for a number of years. This is one of the worse things that can realistically happen to one person in today's world.

What's the least bad, bad thing that can happen? Well, suppose a dust speck floated into your eye and irritated it just a little, for a fraction of a second, barely enough to make you notice before you blink and wipe away the dust speck.

For our next ingredient, we need a large number. Let's use 3^^^3, written in Knuth's up-arrow notation:

3^3 = 27.

3^^3 = (3^(3^3)) = 3^27 = 7625597484987.

3^^^3 = (3^^(3^^3)) = 3^^7625597484987 = (3^(3^(3^(... 7625597484987 times ...)))).

3^^^3 is an exponential tower of 3s which is 7,625,597,484,987 layers tall. You start with 1; raise 3 to the power of 1 to get 3; raise 3 to the power of 3 to get 27; raise 3 to the power of 27 to get 7625597484987; raise 3 to the power of 7625597484987 to get a number much larger than the number of atoms in the universe, but which could still be written down in base 10, on 100 square kilometers of paper; then raise 3 to that power; and continue until you've exponentiated 7625597484987 times. That's 3^^^3. It's the smallest simple inconceivably huge number I know.

Now here's the moral dilemma. If neither event is going to happen to you personally, but you still had to choose one or the other:

Would you prefer that one person be horribly tortured for fifty years without hope or rest, or that 3^^^3 people get dust specks in their eyes?

I think the answer is obvious. How about you?
Motivated Stopping and Motivated Continuation
by Eliezer Yudkowsky — 2007-10-28T23:10:26.000Z

While I disagree with some views of the Fast and Frugal crowd—in my opinion they make a few too many lemons into lemonade—it also seems to me that they tend to develop the most psychologically realistic models of any school of decision theory. Most experiments present the subjects with options, and the subject chooses an option, and that’s the experimental result. The frugalists realized that in real life, you have to generate your options, and they studied how subjects did that.

Likewise, although many experiments present evidence on a silver platter, in real life you have to gather evidence, which may be costly, and at some point decide that you have enough evidence to stop and choose. When you’re buying a house, you don’t get exactly ten houses to choose from, and you aren’t led on a guided tour of all of them before you’re allowed to decide anything. You look at one house, and another, and compare them to each other; you adjust your aspirations—reconsider how much you really need to be close to your workplace and how much you’re really willing to pay; you decide which house to look at next; and at some point you decide that you’ve seen enough houses, and choose.

Gilovich’s distinction between motivated skepticism and motivated credulity highlights how conclusions a person does not want to believe are held to a higher standard than conclusions a person wants to believe. A motivated skeptic asks if the evidence compels them to accept the conclusion; a motivated credulist asks if the evidence allows them to accept the conclusion.

I suggest that an analogous bias in psychologically realistic search is motivated stopping and motivated continuation: when we have a hidden motive for choosing the “best” current option, we have a hidden motive to stop, and choose, and reject consideration of any more options. When we have a hidden motive to reject the current best option, we have a hidden motive to suspend judgment pending additional evidence, to generate more options—to find something, anything, to do instead of coming to a conclusion.

A major historical scandal in statistics was R. A. Fisher, an eminent founder of the field, insisting that no causal link had been established between smoking and lung cancer. “Correlation is not causation,”...
Bay Area Bayesians Unite!
by Eliezer Yudkowsky — 2007-10-28T00:07:42.000Z

Robin Hanson has his fellow GMU economists to talk to, but I'm not
associated with a university and I live way out in the boondocks: the
echoing emptiness of, er, Silicon Valley.

Overcoming Bias gets over 2000 visitors per day. Surely some of you
are from the Bay Area. Would you be interested in a Bay Area meetup of
Overcoming Bias readers?

Polls after the jump.
If you're interested at all, please vote in at least the closest-city poll.
Polls will be processed for a best-compromise value, not a binding
modal result.
If I get at least 30 responses, I'll start looking into
meetup locations.

Opinion Polls & Market Research

Opinion Polls & Market Research

Opinion Polls & Market Research
Why Are Individual IQ Differences OK?
by Eliezer Yudkowsky — 2007-10-26T21:50:54.000Z

Idang Alibi of Abuja, Nigeria writes on the James Watson affair:

A few days ago, the Nobel Laureate, Dr. James Watson, made a remark that is now generating worldwide uproar, especially among blacks. He said what to me looks like a self-evident truth. He told The Sunday Times of London in an interview that in his humble opinion, black people are less intelligent than the White people...

An intriguing opening. Is Idang Alibi about to take a position on the real heart of the uproar?

I do not know what constitutes intelligence. I leave that to our so-called scholars. But I do know that in terms of organising society for the benefit of the people living in it, we blacks have not shown any intelligence in that direction at all. I am so ashamed of this and sometimes feel that I ought to have belonged to another race...

Darn, it's just a lecture on personal and national responsibility. Of course, for African nationals, taking responsibility for their country's problems is the most productive attitude regardless. But it doesn't engage with the controversies that got Watson fired.

Later in the article came this:

As I write this, I do so with great pains in my heart because I know that God has given intelligence in equal measure to all his children irrespective of the colour of their skin.

This intrigued me for two reasons: First, I'm always on the lookout for yet another case of theology making a falsifiable experimental prediction. And second, the prediction follows obviously if God is just, but what does skin colour have to do with it at all?

A great deal has already been said about the Watson affair, and I suspect that in most respects I have little to contribute that has not been said before.

But why is it that the rest of the world seems to think that individual genetic differences are okay, whereas racial genetic differences in intelligence are not? Am I the only one who's every bit as horrified by the proposition that there's any way whatsoever to be screwed before you even start, whether it's genes or lead-based paint or Down's Syndrome? What difference does skin colour make? At all?

This is only half a rhetorical question. Race adds extra controversy to anything; in that sense, it's obvious what difference skin colour makes politically. However, just because this attit...
No One Knows What Science Doesn't Know
by Eliezer Yudkowsky — 2007-10-25T23:47:47.000Z

At a family party some years ago, one of my uncles remarked on how little science really knows. For example, we still have no idea how gravity works - why things fall down.

"Actually, we do know how gravity works," I said. (My father, a Ph.D. physicist, was also present; but he wasn't even touching this one.)

"We do?" said my uncle.

"Yes," I said, "Gravity is the curvature of spacetime." At this point I had still swallowed Feynman's line about being able to explain physics to one's grandmother, so I continued: "You could say that the Earth goes around the Sun in a straight line. Imagine a graph that shows both space and time, so that a straight line shows steady movement and a curved line shows acceleration. Then curve the graph paper itself. When you try to draw a straight line on the curved paper, you'll get what looks like acceleration -"

"I never heard about anything like that," said my uncle.

When was the last time, in history, when it was possible for a single human to know the knowledge of the most advanced civilization? I've seen various estimates for this - usually in the
form of polymaths nominated for the position of "last person to
know everything". One plausible candidate is Leonardo da Vinci, who
died in 1519 - shortly after the printing press began to become popular,
and shortly before Copernicus inaugurated the scientific revolution.

In the ancestral environment it was possible to know everything, and nearly everyone did. In hunter-gatherer bands of less than 200 people, with no written literature, all background knowledge was universal knowledge. If one person, in a world containing 200 people total, discovered how gravity worked, you could certainly expect to hear about it.

In a world of 6 billion people, there is not one person alive who can say with certainty that science does not know a thing. There is too much science. Our current lifetimes are too short to learn more than a tiny fraction of it, and more is being produced all the time.

Even if last week's technical journal doesn't contain the answer to a mystery, that doesn't mean that no one knows it. Maybe someone out there is typing up the paper at this very moment. You can't generalize over all 6 billion people in the world because ...
Double Illusion of Transparency
by Eliezer Yudkowsky — 2007-10-24T23:06:29.000Z

Followup to: Explainers Shoot High, Illusion of Transparency

My first true foray into Bayes For Everyone was writing An Intuitive Explanation of Bayesian Reasoning, still one of my most popular works. This is the Intuitive Explanation's origin story.

In December of 2002, I'd been sermonizing in a habitual IRC channels about what seemed to me like a very straightforward idea: How words, like all other useful forms of thought, are secretly a disguised form of Bayesian inference. I thought I was explaining clearly, and yet there was one fellow, it seemed, who didn't get it. This worried me, because this was someone who'd been very enthusiastic about my Bayesian sermons up to that point. He'd gone around telling people that Bayes was "the secret of the universe", a phrase I'd been known to use.

So I went into a private IRC conversation to clear up the sticking point.

And he still didn't get it.

I took a step back and explained the immediate prerequisites, which I had thought would be obvious -

He didn't understand my explanation of the prerequisites.

In desperation, I recursed all the way back to Bayes's Theorem, the ultimate foundation stone of -

He didn't know how to apply Bayes's Theorem to update the probability that a fruit is a banana, after it is observed to be yellow. He kept mixing up p(b|y) and p(y|b).

It seems like a small thing, I know. It's strange how small things can trigger major life-realizations. Any former TAs among my readers are probably laughing: I hadn't realized, until then, that instructors got misleading feedback. Robin commented yesterday that the best way to aim your explanations is feedback from the intended audience, "an advantage teachers often have". But what if self-anchoring also causes you to overestimate how much understanding appears in your feedback?

I fell prey to a double illusion of transparency. First, I assumed that my words meant what I intended them to mean - that my listeners heard my intentions as though they were transparent. Second, when someone repeated back my sentences using slightly different word orderings, I assumed that what I heard was what they had intended to say. As if all words were transparent windows into thought, in both directions.

I thought that if I said, "Hey, guess what I noticed today! Bayes's Theorem is...
Explainers Shoot High.  Aim Low!
by Eliezer Yudkowsky — 2007-10-24T01:13:54.000Z

Followup to: Illusion of Transparency: Why No One Understands You, Expecting Short Inferential Distances

A few years ago, an eminent scientist once told me how he'd written an explanation of his field aimed at a much lower technical level than usual. He had thought it would be useful to academics outside the field, or even reporters. This ended up being one of his most popular papers within his field, cited more often than anything else he'd written.

The lesson was not that his fellow scientists were stupid, but that we tend to enormously underestimate the effort required to properly explain things.

He told me this, because I'd just told him about my experience publishing "An Intuitive Explanation of Bayesian Reasoning". This is still one of my most popular, most blogged, and most appreciated works today. I regularly get fan mail from formerly confused undergraduates taking statistics classes, and journalists, and professors from outside fields. In short, I successfully hit the audience the eminent scientist had thought he was aiming for.

I'd thought I was aiming for elementary school.

Today, when I look back at the Intuitive Explanation, it seems pretty silly as an attempt on grade school:

It's assumed that the reader knows what a "probability" is.

No single idea requires more than a single example.

No homework problems! I've gotten several complaints about this.

(Then again, I get a roughly equal number of complaints that the Intuitive Explanation is too long and drawn-out, as that it is too short. The current version does seem to be "just right" for a fair number of people.)

Explainers shoot way, way higher than they think they're aiming, thanks to the illusion of transparency and self-anchoring. We miss the mark by several major grades of expertise. Aiming for outside academics gets you an article that will be popular among specialists in your field. Aiming at grade school (admittedly, naively so) will hit undergraduates. This is not because your audience is more stupid than you think, but because your words are far less helpful than you think. You're way way overshooting the target. Aim several major gradations lower, and you may hit your mark.

PS: I know and do confess that I need to work on taking my own advice.

Addendum: With his ...
Expecting Short Inferential Distances
by Eliezer Yudkowsky — 2007-10-22T23:42:01.000Z

Homo sapiens’s environment of evolutionary adaptedness (a.k.a. EEA or “ancestral environment”) consisted of hunter-gatherer bands of at most 200 people, with no writing. All inherited knowledge was passed down by speech and memory.

In a world like that, all background knowledge is universal knowledge. All information not strictly private is public, period.

In the ancestral environment, you were unlikely to end up more than one inferential step away from anyone else. When you discover a new oasis, you don’t have to explain to your fellow tribe members what an oasis is, or why it’s a good idea to drink water, or how to walk. Only you know where the oasis lies; this is private knowledge. But everyone has the background to understand your description of the oasis, the concepts needed to think about water; this is universal knowledge. When you explain things in an ancestral environment, you almost never have to explain your concepts. At most you have to explain one new concept, not two or more simultaneously.

In the ancestral environment there were no abstract disciplines with vast bodies of carefully gathered evidence generalized into elegant theories transmitted by written books whose conclusions are a hundred inferential steps removed from universally shared background premises.

In the ancestral environment, anyone who says something with no obvious support is a liar or an idiot. You’re not likely to think, “Hey, maybe this person has well-supported background knowledge that no one in my band has even heard of,” because it was a reliable invariant of the ancestral environment that this didn’t happen.

Conversely, if you say something blatantly obvious and the other person doesn’t see it, they’re the idiot, or they’re being deliberately obstinate to annoy you.

And to top it off, if someone says something with no obvious support and expects you to believe it—acting all indignant when you don’t—then they must be crazy.

Combined with the illusion of transparency and self-anchoring (the tendency to model other minds as though the were slightly modified versions of oneself), I think this explains a lot about the legendary difficulty most scientists have in communicating with a lay audience—or even communicating with scientists from o...
Self-Anchoring
by Eliezer Yudkowsky — 2007-10-22T06:11:12.000Z

Sometime between the age of 3 and 4, a human child becomes able, for the first time, to model other minds as having different beliefs. The child sees a box, sees candy in the box, and sees that Sally sees the box. Sally leaves, and then the experimenter, in front of the child, replaces the candy with pencils and closes the box so that the inside is not visible. Sally returns, and the child is asked what Sally thinks is in the box. Children younger than 3 say "pencils", children older than 4 say "candy".

Our ability to visualize other minds is imperfect. Neural circuitry is not as flexible as a program fed to a general-purpose computer. An AI, with fast read-write access to its own memory, might be able to create a distinct, simulated visual cortex to imagine what a human "sees". We humans only have one visual cortex, and if we want to imagine what someone else is seeing, we've got to simulate it using our own visual cortex - put our own brains into the other mind's shoes. And because you can't reconfigure memory to simulate a new brain from stratch, pieces of you leak into your visualization of the Other.

The diagram above is from Keysar, Barr, Balin, & Brauner (2000). The experimental subject, the "addressee", sat in front of an array of objects, viewed as seen on the left. On the other side, across from the addressee, sat the "director", with the view as seen on the right. The addressee had an unblocked view, which also allowed the addressee to see which objects were not visible to the director.

The experiment used the eye-tracking method: the direction of a subject's gaze can be measured using computer vision. Tanenhaus et. al. (1995) had previously demonstrated that when people understand a spoken reference, their gaze fixates on the identified object almost immediately.

The key test was when the director said "Put the small candle next to the truck." As the addressee can clearly observe, the director only knows about two candles, the largest and medium ones; the smallest candle is occluded.

And, lo and behold, subjects' eyes fixated on the occluded smallest candle an average of 1,487 milliseconds before they correctly identified the medium-sized candle as the one the director must have meant.

This seems to suggest that...