trentmkelly/LessWrong-43k huggingface dataset:
On Expressing Your Concerns
by Eliezer Yudkowsky — 2007-12-27T04:04:44.000Z

The scary thing about Asch’s conformity experiments is that you can get many people to say black is white, if you put them in a room full of other people saying the same thing. The hopeful thing about Asch’s conformity experiments is that a single dissenter tremendously drove down the rate of conformity, even if the dissenter was only giving a different wrong answer. And the wearisome thing is that dissent was not learned over the course of the experiment—when the single dissenter started siding with the group, rates of conformity rose back up.

Being a voice of dissent can bring real benefits to the group. But it also (famously) has a cost. And then you have to keep it up. Plus you could be wrong.

I recently had an interesting experience wherein I began discussing a project with two people who had previously done some planning on their own. I thought they were being too optimistic and made a number of safety-margin-type suggestions for the project. Soon a fourth guy wandered by, who was providing one of the other two with a ride home, and began making suggestions. At this point I had a sudden insight about how groups become overconfident, because whenever I raised a possible problem, the fourth guy would say, “Don’t worry, I’m sure we can handle it!” or something similarly reassuring.

An individual, working alone, will have natural doubts. They will think to themselves, “Can I really do XYZ?” because there’s nothing impolite about doubting your own competence. But when two unconfident people form a group, it is polite to say nice and reassuring things, and impolite to question the other person’s competence. Together they become more optimistic than either would be on their own, each one’s doubts quelled by the other’s seemingly confident reassurance, not realizing that the other person initially had the same inner doubts.

The most fearsome possibility raised by Asch’s experiments on conformity is the specter of everyone agreeing with the group, swayed by the confident voices of others, careful not to let their own doubts show—not realizing that others are suppressing similar worries. This is known as “pluralistic ignorance.”

Robin Hanson and I have a long-running debate over when, exactly, aspiring rationalists should dare to...
Asch's Conformity Experiment
by Eliezer Yudkowsky — 2007-12-26T07:03:13.000Z

Solomon Asch, with experiments originally carried out in the 1950s and well-replicated since, highlighted a phenomenon now known as “conformity.” In the classic experiment, a subject sees a puzzle like the one in the nearby diagram: Which of the lines A, B, and C is the same size as the line X? Take a moment to determine your own answer . . .

The gotcha is that the subject is seated alongside a number of other people looking at the diagram—seemingly other subjects, actually confederates of the experimenter. The other “subjects” in the experiment, one after the other, say that line C seems to be the same size as X. The real subject is seated next-to-last. How many people, placed in this situation, would say “C”—giving an obviously incorrect answer that agrees with the unanimous answer of the other subjects? What do you think the percentage would be?

Three-quarters of the subjects in Asch’s experiment gave a “conforming” answer at least once. A third of the subjects conformed more than half the time.

Interviews after the experiment showed that while most subjects claimed to have not really believed their conforming answers, some said they’d really thought that the conforming option was the correct one.

Asch was disturbed by these results:1

That we have found the tendency to conformity in our society so strong . . . is a matter of concern. It raises questions about our ways of education and about the values that guide our conduct.

It is not a trivial question whether the subjects of Asch’s experiments behaved irrationally. Robert Aumann’s Agreement Theorem shows that honest Bayesians cannot agree to disagree—if they have common knowledge of their probability estimates, they have the same probability estimate. Aumann’s Agreement Theorem was proved more than twenty years after Asch’s experiments, but it only formalizes and strengthens an intuitively obvious point—other people’s beliefs...
The Amazing Virgin Pregnancy
by Eliezer Yudkowsky — 2007-12-24T14:00:00.000Z

People who grow up believing certain things,
even if they later stop believing them,
may not quite realize how the beliefs sound to outsiders...

(SCENE: A small cottage in Nazareth.)

Joseph: Mary, my dearest fiancée, there's something I've been meaning to talk to you about.

(Mary's shoulders slump. Slowly, as if under a heavy burden, she turns around to face Joseph.)

Joseph: You seem to be getting fat around the waistline, and throwing up in the morning, and, er, not getting any periods. Which is odd, because it's sort of like -

Mary: Yes! I'm pregnant! All right? I'm PREGNANT!

Joseph: How is that possible?

(Mary's shoulders slump further.) Mary: How do you think?

Joseph: I don't know, that's why I'm asking you. I mean, you're still a virgin, right?

(Mary looks up cautiously, and sees Joseph's face looking blankly puzzled.)

Joseph: Well?

Mary: God did it.

Joseph: You had sex with -

Mary: No! Haha. Of course not. I mean, God just snapped his fingers and did one of those miracle things and made me pregnant.

Joseph: God made you pregnant.

Mary: (Starts to sweat.) Yes.

Joseph: Mary, that is just so... completely...

(Mary's eyes squeeze shut.)

Joseph: ...COOL!

(Mary opens her eyes again, cautiously.)

Mary: You think so?

Joseph: Of course! Who wouldn't think so? Come on, we've got to tell everyone the news!

Mary: Maybe we should keep this between just the two of us -

Joseph: No, no, silly girl, this is way too important! Come on!

(Joseph grabs Mary's wrist and drags her out of the house. SCENE: The gathering square of Nazareth. A dozen well-dressed men, and the town's head rabbi, look on Joseph and Mary impatiently.)

Rabbi: What's this all about, Joseph? I trust there's a good reason for the fuss?

Joseph: Go ahead, Mary! Tell them what you told me.

Mary: Um... (She swallows.) God made me pregnant.

Rabbi, looking stern, yet understanding: Now, Joseph, you know you're not supposed to do that before -

Joseph: No, no, you don't get it! She's still a virgin! God made her pregnant directly!

(There's a long pause.)

Man #1: So, what you're saying here, basically, is that Mary tells you she's a virgin.

Joseph: Uh huh...
Zen and the Art of Rationality
by Eliezer Yudkowsky — 2007-12-24T04:36:34.000Z

Followup to: Effortless Technique

No one would mistake my writings for ancient Eastern wisdom. Successfully or not, I aspire to clearly set forth the reasoning, antecedent assumptions, and pragmatic use of my conclusions. Successfully or not, I aspire to cut my proposals into modular pieces, so that a user can reject one mistake without destroying the whole. This standard of writing is inherited from the ancient traditions of technical thinking, not the ancient traditions of Zen.

No one would mistake my writings for ancient Eastern wisdom. My goals are not the goals of Buddha or Lao Tse. Feeling Rational suggested that emotions should follow from beliefs but not beliefs follow from emotions: the ideal is to free yourself of all attachment to preferred conclusions about reality, arrive at your beliefs of fact by weighing the evidence without prejudice, and then feel fully whatever emotions follow from these beliefs-of-fact. In stereotypical Eastern philosophy, you are supposed to free yourself of all attachments, not just attachment to beliefs-of-fact apart from evidence; you are supposed to relinquish all desire. Yes, I know it's more complicated than that - but still, their goals are not mine.

And yet it oftimes seems to me that my thoughts are expressed in conceptual language that owes a great deal to the inspiration of Eastern philosophy. "Free yourself of attachments to thinking that the universe is one way or another: Arrive at your picture of the world without prejudice, and then feel fully whichever feelings arise from this picture. Let your emotions flow from your beliefs, not the other way around." It's not a Buddhist conclusion, but the language owes a nod in the direction of Buddhism. Even if a Buddhist teacher would vehemently disagree, they might still grasp immediately what was being proposed. Grasp it more clearly, perhaps, than an old-school (i.e. pre-Bayesian) Western rationalist.

No one would mistake my writings for ancient Eastern wisdom. And this is well, because I can't stand people who try to pass off their ideas as ancient wisdom. As if that were a recommendation! The fifth-century Chinese philosopher Xiaoguang Li observed that ancient civilizations are revered, and yet ancient civilizations are not wise like venerable huma...
Effortless Technique
by Eliezer Yudkowsky — 2007-12-23T04:22:17.000Z

"All my life I have been intensely repelled by the idea of 'making an effort'. I hate this idea today as much as I did as a child. I don't know why I hate it so much; I just do."
-- Raymond Smullyan, The Tao Is Silent

In the Hollywood version of rationality - or even the Traditional rationality that was passed down from supervisor to grad student in ancient days before Bayesianism - rationality is a great strain, a great effort, a continuous battle to coerce your mind into a desired shape. Spock, the archetype of Hollywood's concept of rationality, represses all his emotions.

And this great effort, they conceive, is virtue unto a rationalist. The more effort you expend on forcing yourself into the mold, the better the rationalist you must be. It's like working extra hard at your job, as demanded by the Protestant work-ethic. If the one works long hours - sweating, getting ulcers - surely the one must be worthy of praise?

This, I think, is an instance of a Lost Purpose. People see that successful folk must sometimes make an effort, and so
they conclude that effort of itself is virtuous whether or not it
succeeds.

I am reminded of an AI koan from AGI '06, where the discussion turned (as it often does) to defining "intelligence". A medium-prominent AI researcher suggested that an agent's "intelligence" could be measured in the agent's processing cycles per second, bits of memory, and bits of sensory bandwidth. To which I replied with a quote from Dijkstra:

"If we wish to count lines of code, we should not regard them as 'lines produced' but as 'lines spent': the current conventional wisdom is so foolish as to book that count on the wrong side of the ledger."

Surely (I said), an agent is less intelligent if it uses more memory, processing power, and sensory bandwidth to accomplish the same task?

This reply was due, in no small part, to my having read Raymond Smullyan's The Tao Is Silent at the age of sixteen. Raymond Smullyan
is a mathematical logician, a great composer of logic puzzles, and
sometime Westernized Taoist. Though I disagree with much of The Tao Is Silent, I would count "just the parts of the book I liked" as one of my most important formative influences as a rationalist.

In particular, it was in ...
False Laughter
by Eliezer Yudkowsky — 2007-12-22T06:03:45.000Z

Followup to: Politics and Awful Art

There's this thing called "derisive laughter" or "mean-spirited laughter", which follows from seeing the Hated Enemy get a kick in the pants. It doesn't have to be an unexpected kick in the pants, or a kick followed up with a custard pie. It suffices that the Hated Enemy gets hurt. It's like humor, only without the humor.

If you know what your audience hates, it doesn't take much effort to get a laugh like that—which marks this as a subspecies of awful political art.

There are deliciously biting satires, yes; not all political art is bad art. But satire is a much more demanding art than just punching the Enemy in the nose. In fact, never mind satire—just an atom of ordinary genuine humor takes effort.

Imagine this political cartoon: A building labeled "science", and a standard Godzilla-ish monster labeled "Bush" stomping on the "science" building. Now there are people who will laugh at this—hur hur, scored a point off Bush, hur hur—but this political cartoon didn't take much effort to imagine. In fact, it was the very first example that popped into my mind when I thought "political cartoon about Bush and science". This degree of obviousness is a bad sign.

If I want to make a funny political cartoon, I have to put in some effort. Go beyond the cached thought. Use my creativity. Depict Bush as a tentacle monster and Science as a Japanese schoolgirl.

There are many art forms that suffer from obviousness. But humor more than most, because humor relies on surprise—the ridiculous, the unexpected, the absurd.

(Satire achieves surprise by saying, out loud, the thoughts you didn't dare think. Fake satires repeat thoughts you were already thinking.)

You might say that a predictable punchline is too high-entropy to be funny, by that same logic which says you should be enormously less surprised to find your thermostat reading 30 degrees than 29 degrees.

The general test against awful political art is to ask whether the art would seem worthwhile if it were not political. If someone writes a song about space travel, and the song is good enough that I would enjoy listening to it even if it were about butterflies, then and only then does it qualify to pick up bonus points for praising a Worthy Cause.

So one test for derisive...
Two Cult Koans
by Eliezer Yudkowsky — 2007-12-21T05:45:31.000Z

A novice rationalist studying under the master Ougi was rebuked by a friend who said, “You spend all this time listening to your master, and talking of ‘rational’ this and ‘rational’ that—you have fallen into a cult!”

The novice was deeply disturbed; he heard the words You have fallen into a cult! resounding in his ears as he lay in bed that night, and even in his dreams.

The next day, the novice approached Ougi and related the events, and said, “Master, I am constantly consumed by worry that this is all really a cult, and that your teachings are only dogma.”

Ougi replied, “If you find a hammer lying in the road and sell it, you may ask a low price or a high one. But if you keep the hammer and use it to drive nails, who can doubt its worth?”

The novice said, “See, now that’s just the sort of thing I worry about—your mysterious Zen replies.”

Ougi said, “Fine, then, I will speak more plainly, and lay out perfectly reasonable arguments which demonstrate that you have not fallen into a cult. But first you have to wear this silly hat.”

Ougi gave the novice a huge brown ten-gallon cowboy hat.

“Er, master . . .” said the novice.

“When I have explained everything to you,” said Ougi, “you will see why this was necessary. Or otherwise, you can continue to lie awake nights, wondering whether this is a cult.”

The novice put on the cowboy hat.

Ougi said, “How long will you repeat my words and ignore the meaning? Disordered thoughts begin as feelings of attachment to preferred conclusions. You are too anxious about your self-image as a rationalist. You came to me to seek reassurance. If you had been truly curious, not knowing one way or the other, you would have thought of ways to resolve your doubts. Because you needed to resolve your cognitive dissonance, you were willing to put on a silly hat. If I had been an evil man, I could have made you pay a hundred silver coins. When you concentrate on a real-world question, the worth or worthlessness of your understanding will soon become apparent. You are like a swordsman who keeps glancing away to see if anyone might be laughing at him—”

“All right,” said the novice.

“You asked for the long...
Politics and Awful Art
by Eliezer Yudkowsky — 2007-12-20T03:46:21.000Z

Followup to: Rationality and the English Language

One of my less treasured memories is of a State of the Union address, or possibly a presidential inauguration, at which a Nobel Laureate got up and read, in a terribly solemn voice, some politically correct screed about what a wonderfully inclusive nation we all were—"The African-Americans, the Ethiopians, the Etruscans", or something like that. The "poem", if you can call it that, was absolutely awful. As far as my ears could tell, it had no redeeming artistic merit whatsoever.

Every now and then, yet another atheist is struck by the amazing idea that atheists should have hymns, just like religious people have hymns, and they take some existing religious song and turn out an atheistic version. And then this "atheistic hymn" is, almost without exception, absolutely awful. But the author can't see how dreadful the verse is as verse. They're too busy congratulating themselves on having said "Religion sure sucks, amen." Landing a punch on the Hated Enemy feels so good that they overlook the hymn's lack of any other merit. Verse of the same quality about something unpolitical, like mountain streams, would be seen as something a kindergartener's mother would post on her refrigerator.

In yesterday's Litany Against Gurus, there are only two lines that might be classifiable as "poetry", not just "verse". When I was composing the litany's end, the lines that first popped into my head were:

I was not your destination
Only a step on your path

Which didn't sound right at all. Substitute "pathway" for "road", so the syllable counts would match? But that sounded even worse. The prosody—the pattern of stressed syllables—was all wrong.

The real problem was the word des-ti-NA-tion—a huge awkward lump four syllables long. So get rid of it! "I was not your goal" was the first alternative that came to mind. Nicely short. But now that I was thinking about it, "goal" sounded very airy and abstract. Then the word "city" came into my mind—and it echoed.

"I was never your city" came to me, not by thinking about rationality, but by thinking about prosody. The constraints of art force us to toss out the first, old, tired phrasing that comes to mind; and in searching for a less obvious phrasing, oft...
The Litany Against Gurus
by Eliezer Yudkowsky — 2007-12-18T20:11:02.000Z

I am your hero!
I am your master!
Learn my arts,
Seek my way.
Learn as I learned,
Seek as I sought.
Envy me!
Aim at me!
Rival me!
Transcend me!
Look back,
Smile,
And then—
Eyes front!
I was never your city,
Just a stretch of your road.

Part of the Politics Is the Mind-Killer subsequence of How To Actually Change Your Mind

Next post: "Politics and Awful Art"

Previous post: "Rationality and the English Language"
Guardians of Ayn Rand
by Eliezer Yudkowsky — 2007-12-18T06:24:05.000Z

"For skeptics, the idea that reason can lead to a cult is absurd. The characteristics of a cult are 180 degrees out of phase with reason. But as I will demonstrate, not only can it happen, it has happened, and to a group that would have to be considered the unlikeliest cult in history. It is a lesson in what happens when the truth becomes more important than the search for truth..."
—Michael Shermer, "The Unlikeliest Cult in History"

I think Michael Shermer is over-explaining Objectivism. I'll get around to amplifying on that.

Ayn Rand's novels glorify technology, capitalism, individual defiance of the System, limited government, private property, selfishness. Her ultimate fictional hero, John Galt, was <SPOILER>a scientist who invented a new form of cheap renewable energy; but then refuses to give it to the world since the profits will only be stolen to prop up corrupt governments.</SPOILER>

And then—somehow—it all turned into a moral and philosophical "closed system" with Ayn Rand at the center. The term "closed system" is not my own accusation; it's the term the Ayn Rand Institute uses to describe Objectivism. Objectivism is defined by the works of Ayn Rand. Now that Rand is dead, Objectivism is closed. If you disagree with Rand's works in any respect, you cannot be an Objectivist.

Max Gluckman once said: "A science is any discipline in which the fool of this generation can go beyond the point reached by the genius of the last generation." Science moves forward by slaying its heroes, as Newton fell to Einstein. Every young physicist dreams of being the new champion that future physicists will dream of dethroning.

Ayn Rand's philosophical idol was Aristotle. Now maybe Aristotle was a hot young math talent 2350 years ago, but math has made noticeable progress since his day. Bayesian probability theory is the quantitative logic of which Aristotle's qualitative logic is a special case; but there's no sign that Ayn Rand knew about Bayesian probability theory when she wrote her magnum opus, Atlas Shrugged. Rand wrote about "rationality", yet failed to familiarize herself with the modern research in heuristics and biases. How can anyone claim to be a master rationalist, yet kn...
Guardians of the Gene Pool
by Eliezer Yudkowsky — 2007-12-16T20:08:39.000Z

Like any educated denizen of the 21st century, you may have heard of World War II. You may remember that Hitler and the Nazis planned to carry forward a romanticized process of evolution, to breed a new master race, supermen, stronger and smarter than anything that had existed before.

Actually this is a common misconception. Hitler believed that the Aryan superman had previously existed—the Nordic stereotype, the blond blue-eyed beast of prey—but had been polluted by mingling with impure races. There had been a racial Fall from Grace.

It says something about the degree to which the concept of progress permeates Western civilization, that the one is told about Nazi eugenics and hears "They tried to breed a superhuman." You, dear reader—if you failed hard enough to endorse coercive eugenics, you would try to create a superhuman. Because you locate your ideals in your future, not in your past. Because you are creative. The thought of breeding back to some Nordic archetype from a thousand years earlier would not even occur to you as a possibility—what, just the Vikings? That's all? If you failed hard enough to kill, you would damn well try to reach heights never before reached, or what a waste it would all be, eh? Well, that's one reason you're not a Nazi, dear reader.

It says something about how difficult it is for the relatively healthy to envision themselves in the shoes of the relatively sick, that we are told of the Nazis, and distort the tale to make them defective transhumanists.

It's the Communists who were the defective transhumanists. "New Soviet Man" and all that. The Nazis were quite definitely the bioconservatives of the tale.