trentmkelly/LessWrong-43k huggingface dataset:
The Importance of Saying "Oops"
by Eliezer Yudkowsky — 2007-08-05T03:17:46.000Z

I just finished reading a history of Enron’s downfall, The Smartest Guys in the Room, which hereby wins my award for “Least Appropriate Book Title.”

An unsurprising feature of Enron’s slow rot and abrupt collapse was that the executive players never admitted to having made a large mistake. When catastrophe #247 grew to such an extent that it required an actual policy change, they would say, “Too bad that didn’t work out—it was such a good idea—how are we going to hide the problem on our balance sheet?” As opposed to, “It now seems obvious in retrospect that it was a mistake from the beginning.” As opposed to, “I’ve been stupid.” There was never a watershed moment, a moment of humbling realization, of acknowledging a fundamental problem. After the bankruptcy, Jeff Skilling, the former COO and brief CEO of Enron, declined his own lawyers’ advice to take the Fifth Amendment; he testified before Congress that Enron had been a great company.

Not every change is an improvement, but every improvement is necessarily a change. If we only admit small local errors, we will only make small local changes. The motivation for a big change comes from acknowledging a big mistake.

As a child I was raised on equal parts science and science fiction, and from Heinlein to Feynman I learned the tropes of Traditional Rationality: theories must be bold and expose themselves to falsification; be willing to commit the heroic sacrifice of giving up your own ideas when confronted with contrary evidence; play nice in your arguments; try not to deceive yourself; and other fuzzy verbalisms.

A traditional rationalist upbringing tries to produce arguers who will concede to contrary evidence eventually—there should be some mountain of evidence sufficient to move you. This is not trivial; it distinguishes science from religion. But there is less focus on speed, on giving up the fight as quickly as possible, integrating evidence efficiently so that it only takes a minimum of contrary evidence to destroy your cherished belief.

I was raised in Traditional Rationality, and thought myself quite the rationalist. I switched to Bayescraft (Laplace / Jaynes / Tversky / Kahneman) in the aftermath of . . . well, it’s a long story. Roughly, I switched because I realized th...
Religion's Claim to be Non-Disprovable
by Eliezer Yudkowsky — 2007-08-04T03:21:50.000Z

The earliest account I know of a scientific experiment is, ironically, the story of Elijah and the priests of Baal.

The people of Israel are wavering between Jehovah and Baal, so Elijah announces that he will conduct an experiment to settle it—quite a novel concept in those days! The priests of Baal will place their bull on an altar, and Elijah will place Jehovah’s bull on an altar, but neither will be allowed to start the fire; whichever God is real will call down fire on His sacrifice. The priests of Baal serve as control group for Elijah—the same wooden fuel, the same bull, and the same priests making invocations, but to a false god. Then Elijah pours water on his altar—ruining the experimental symmetry, but this was back in the early days—to signify deliberate acceptance of the burden of proof, like needing a 0.05 significance level. The fire comes down on Elijah’s altar, which is the experimental observation. The watching people of Israel shout “The Lord is God!”—peer review.

And then the people haul the 450 priests of Baal down to the river Kishon and slit their throats. This is stern, but necessary. You must firmly discard the falsified hypothesis, and do so swiftly, before it can generate excuses to protect itself. If the priests of Baal are allowed to survive, they will start babbling about how religion is a separate magisterium which can be neither proven nor disproven.

Back in the old days, people actually believed their religions instead of just believing in them. The biblical archaeologists who went in search of Noah’s Ark did not think they were wasting their time; they anticipated they might become famous. Only after failing to find confirming evidence—and finding disconfirming evidence in its place—did religionists execute what William Bartley called the retreat to commitment, “I believe because I believe.”

Back in the old days, there was no concept of religion’s being a separate magisterium. The Old Testament is a stream-of-consciousness culture dump: history, law, moral parables, and yes, models of how the universe works—like the universe being created in six days (which is a metaphor for the Big Bang), or rabbits chewing their cud. (Which is a metaphor for . . .)

Back in the old days, saying the local religion “could not be proven” would have gotten you burned at the stake. One of the core beliefs of Orthodox Judaism is that God appeared at Mou...
God is irrelevant
by KatjaGrace — 2007-08-03T10:57:00.000Z

Philosophically that is. Psychologically he fulfils an important role – to distance us from philosophy.

In no way would the existence of a God alter the important properties of the universe. Most of the problems a God supposedly solves are merely shifted to the other side of him – a step further away from humans, where we can comfortably ignore them.

Some solutions God doesn’t really provide (presumably all thought of before by various philosophers, but I don’t know which ones, and it’s irrelevant, so please excuse the plagiarism) :

Creator of the universe: An obvious one. Where did God come from then? If he’s existed forever then so could a universe. If you think something as complex as a universe couldn’t come from nothing, how complex would God have to be to be able to make universes?

Source of morality: Where does God get his moral principles from? If he invents them himself they are just as arbitrary a set of restrictions on behaviour as any other (such as an atheist’s morals are feared to be by the religious). Why follow them? If they are inherent in the universe, related to other people, or a matter of choice then God isn’t needed.

Morality is a set of value judgements. If God and I both have a set of value judgements (a moral code), to say that God’s takes precedence is a value judgement in itself. Who judges? God? Why?

Provider of free will: For reasons discussed in the previous post, Free will isn’t a concept (unless you mean determinism), God can’t have – or give humans – free will which isn’t deterministic. The absence of God’s ‘free will’ is even more apparent if he must be good all the time (unless he invents his own changeable moral code as he goes, but is that the kind of morality God should subscribe to? Well yes, if he does! But there’s still the old problem of free will not existing – he can’t escape).

If he’s all powerful as well, then he just ends up as another natural law – one that makes good things always happen. Anyone who’s been alive can tell you there’s fairly solid empirical evidence against such a law existing, but my point isn’t to draw attention to the problem of evil so much as to point out that natural laws are nothing new.

The final picture? A God who may well exist*. But who...
Belief as Attire
by Eliezer Yudkowsky — 2007-08-02T17:13:56.000Z

I have so far distinguished between belief as anticipation-controller, belief in belief, professing and cheering. Of these, we might call anticipation-controlling beliefs "proper beliefs" and the other forms "improper belief". Proper belief can be wrong or irrational, as when someone genuinely anticipates that prayer will cure their sick baby. But the other forms are arguably “not belief at all.”

Yet another form of improper belief is belief as group identification—as a way of belonging. Robin Hanson uses the excellent metaphor of wearing unusual clothing, a group uniform like a priest’s vestments or a Jewish skullcap, and so I will call this “belief as attire.”

In terms of humanly realistic psychology, the Muslims who flew planes into the World Trade Center undoubtedly saw themselves as heroes defending truth, justice, and the Islamic Way from hideous alien monsters a la the movie Independence Day. Only a very inexperienced nerd, the sort of nerd who has no idea how non-nerds see the world, would say this out loud in an Alabama bar. It is not an American thing to say. The American thing to say is that the terrorists “hate our freedom” and that flying a plane into a building is a “cowardly act.” You cannot say the phrases “heroic self-sacrifice” and “suicide bomber” in the same sentence, even for the sake of accurately describing how the Enemy sees the world. The very concept of the courage and altruism of a suicide bomber is Enemy attire—you can tell, because the Enemy talks about it. The cowardice and sociopathy of a suicide bomber is American attire. There are no quote marks you can use to talk about how the Enemy sees the world; it would be like dressing up as a Nazi for Halloween.

Belief-as-attire may help explain how people can be passionate about improper beliefs. Mere belief in belief, or religious professing, would have some trouble creating genuine, deep, powerful emotional effects. Or so I suspect; I confess I’m not an expert here. But my impression is this: People who’ve stopped anticipating-as-if their religion is true, will go to great lengths to convince themselves they are passionate, and this desperation can be mistaken for passion. But it’s not the same fire they had as a child.

On the other hand, it is very easy for a human being to genuinely, passionately, gut-level belong to a group, to cheer for their favorite sports team.1 Identifyin...
Professing and Cheering
by Eliezer Yudkowsky — 2007-08-02T07:20:21.000Z

I once attended a panel on the topic, “Are science and religion compatible?” One of the women on the panel, a pagan, held forth interminably upon how she believed that the Earth had been created when a giant primordial cow was born into the primordial abyss, who licked a primordial god into existence, whose descendants killed a primordial giant and used its corpse to create the Earth, etc. The tale was long, and detailed, and more absurd than the Earth being supported on the back of a giant turtle. And the speaker clearly knew enough science to know this.

I still find myself struggling for words to describe what I saw as this woman spoke. She spoke with . . . pride? Self-satisfaction? A deliberate flaunting of herself?

The woman went on describing her creation myth for what seemed like forever, but was probably only five minutes. That strange pride/satisfaction/flaunting clearly had something to do with her knowing that her beliefs were scientifically outrageous. And it wasn’t that she hated science; as a panelist she professed that religion and science were compatible. She even talked about how it was quite understandable that the Vikings talked about a primordial abyss, given the land in which they lived—explained away her own religion!—and yet nonetheless insisted this was what she “believed,” said with peculiar satisfaction.

I’m not sure that Daniel Dennett’s concept of “belief in belief” stretches to cover this event. It was weirder than that. She didn’t recite her creation myth with the fanatical faith of someone who needs to reassure herself. She didn’t act like she expected us, the audience, to be convinced—or like she needed our belief to validate her.

Dennett, in addition to introducing the idea of belief in belief, has also suggested that much of what is called “religious belief” should really be studied as “religious profession” instead. Suppose an alien anthropologist studied a group of English students who all seemingly believed that Wulky Wilkensen was a retropositional author. The appropriate question may not be “Why do the students all believe this strange belief?” but “Why do they all write this strange sentence on quizzes?” Even if a sentence is essentially meaningless, you can still know when you are supposed to chant the response aloud.

I think Dennett may be slightly too cynical in suggesting that religious profession is just saying the beli...
Bayesian Judo
by Eliezer Yudkowsky — 2007-07-31T05:53:13.000Z

You can have some fun with people whose anticipations get out of sync with what they believe they believe.

I was once at a dinner party, trying to explain to a man what I did for a living, when he said: "I don't believe Artificial Intelligence is possible because only God can make a soul."

At this point I must have been divinely inspired, because I instantly responded: "You mean if I can make an Artificial Intelligence, it proves your religion is false?"

He said, "What?"

I said, "Well, if your religion predicts that I can't possibly make an Artificial Intelligence, then, if I make an Artificial Intelligence, it means your religion is false. Either your religion allows that it might be possible for me to build an AI; or, if I build an AI, that disproves your religion."

There was a pause, as the one realized he had just made his hypothesis vulnerable to falsification, and then he said, "Well, I didn't mean that you couldn't make an intelligence, just that it couldn't be emotional in the same way we are."

I said, "So if I make an Artificial Intelligence that, without being deliberately preprogrammed with any sort of script, starts talking about an emotional life that sounds like ours, that means your religion is wrong."

He said, "Well, um, I guess we may have to agree to disagree on this."

I said: "No, we can't, actually. There's a theorem of rationality called Aumann's Agreement Theorem which shows that no two rationalists can agree to disagree. If two people disagree with each other, at least one of them must be doing something wrong."

We went back and forth on this briefly. Finally, he said, "Well, I guess I was really trying to say that I don't think you can make something eternal."

I said, "Well, I don't think so either! I'm glad we were able to reach agreement on this, as Aumann's Agreement Theorem requires." I stretched out my hand, and he shook it, and then he wandered away.

A woman who had stood nearby, listening to the conversation, said to me gravely, "That was beautiful."

"Thank you very much," I said.

Part of the sequence Mysterious Answers to Mysterious Questions

Next post: "Professing and Cheering"

Previous post: "Belief in Belief"
Belief in Belief
by Eliezer Yudkowsky — 2007-07-29T17:49:43.000Z

Carl Sagan once told a parable of someone who comes to us and claims: “There is a dragon in my garage.” Fascinating! We reply that we wish to see this dragon—let us set out at once for the garage! “But wait,” the claimant says to us, “it is an invisible dragon.”

Now as Sagan points out, this doesn’t make the hypothesis unfalsifiable. Perhaps we go to the claimant’s garage, and although we see no dragon, we hear heavy breathing from no visible source; footprints mysteriously appear on the ground; and instruments show that something in the garage is consuming oxygen and breathing out carbon dioxide.

But now suppose that we say to the claimant, “Okay, we’ll visit the garage and see if we can hear heavy breathing,” and the claimant quickly says no, it’s an inaudible dragon. We propose to measure carbon dioxide in the air, and the claimant says the dragon does not breathe. We propose to toss a bag of flour into the air to see if it outlines an invisible dragon, and the claimant immediately says, “The dragon is permeable to flour.”

Carl Sagan used this parable to illustrate the classic moral that poor hypotheses need to do fast footwork to avoid falsification. But I tell this parable to make a different point: The claimant must have an accurate model of the situation somewhere in their mind, because they can anticipate, in advance, exactly which experimental results they’ll need to excuse.

Some philosophers have been much confused by such scenarios, asking, “Does the claimant really believe there’s a dragon present, or not?” As if the human brain only had enough disk space to represent one belief at a time! Real minds are more tangled than that. There are different types of belief; not all beliefs are direct anticipations. The claimant clearly does not anticipate seeing anything unusual upon opening the garage door. Otherwise they wouldn’t make advance excuses. It may also be that the claimant’s pool of propositional beliefs contains the free-floating statement There is a dragon in my garage. It may seem, to a rationalist, that these two beliefs should collide and conflict even though they are of different types. Yet it is a physical fact that you can write “The sky is green!” next to a picture of a blue sky without the paper bursting into flames.

The rationalist virtue of empiricism is supposed to prevent us from making this class of mistake. We’re supposed to constan...
Making Beliefs Pay Rent (in Anticipated Experiences)
by Eliezer Yudkowsky — 2007-07-28T22:59:48.000Z

Thus begins the ancient parable:

If a tree falls in a forest and no one hears it, does it make a sound? One says, “Yes it does, for it makes vibrations in the air.” Another says, “No it does not, for there is no auditory processing in any brain.”

If there’s a foundational skill in the martial art of rationality, a mental stance on which all other technique rests, it might be this one: the ability to spot, inside your own head, psychological signs that you have a mental map of something, and signs that you don’t.

Suppose that, after a tree falls, the two arguers walk into the forest together. Will one expect to see the tree fallen to the right, and the other expect to see the tree fallen to the left? Suppose that before the tree falls, the two leave a sound recorder next to the tree. Would one, playing back the recorder, expect to hear something different from the other? Suppose they attach an electroencephalograph to any brain in the world; would one expect to see a different trace than the other?

Though the two argue, one saying “No,” and the other saying “Yes,” they do not anticipate any different experiences. The two think they have different models of the world, but they have no difference with respect to what they expect will happen to them; their maps of the world do not diverge in any sensory detail.

It’s tempting to try to eliminate this mistake class by insisting that the only legitimate kind of belief is an anticipation of sensory experience. But the world does, in fact, contain much that is not sensed directly. We don’t see the atoms underlying the brick, but the atoms are in fact there. There is a floor beneath your feet, but you don’t experience the floor directly; you see the light reflected from the floor, or rather, you see what your retina and visual cortex have processed of that light. To infer the floor from seeing the floor is to step back into the unseen causes of experience. It may seem like a very short and direct step, but it is still a step.

You stand on top of a tall building, next to a grandfather clock with an hour, minute, and ticking second hand. In your hand is a bowling ball, and you drop it off the roof. On which tick of the clock will you hear the crash of the bowling ball hitting the ground?

To answer precisely, you must use beliefs like Earth’s gravity is 9.8 meters per second per second, and This building is around 120 meters ta...
Free will isn’t a concept (unless you mean determinism)
by KatjaGrace — 2007-07-15T13:51:00.000Z

Imagine something happens. For instance you make a decision. There are three possibilities for this occurence:

It could be related purely to other factors (determinism)

It could be not related to other factors (randomness)

It could be a combination of these (a mixture of determinism and randomness)

None of these are free will (as commonly understood). So where does the concept of free will fit in? How could an occurence escape from being in one of these categories? Clearly it can’t. So there is no possibility of a concept of free will that is in opposition to determinism, let alone a chance of it existing in reality.

But you feel like you have free will (whatever that is – just don’t think about it), don’t you? Or to put it another way, you feel like your actions are neither determined nor random. You choose them.

And that is precisely why they are determined. They are determined by you. And you already exist to the finest detail at the time you are making the decision. If you made choices (or some element of them) not controlled by your personality, experience, thoughts and anything else that comes under the heading of ‘the state of your brain as a result of genetics and your prior environments’, they would be random, which still isn’t free will (not to mention being a less personal and less appealing model, if that’s how you choose your beliefs).

You might argue that you can choose what to think and how to feel , and how heavily to let those things influence you, when making a decision. That doesn’t alter the situation however. Those are then choices too, and your decisions for them would presumably have to be made based on other thoughts and feelings , which you would presumably choose, and so on. The point at which free will should have occurred would just be shifted back indefinitely. Again you just have a long chain of cause and effect.

The closest thing you can have to free will is for your actions to be determined purely by the state of your brain. Free will is determinism.
Two More Things to Unlearn from School
by Eliezer Yudkowsky — 2007-07-12T17:45:33.000Z

In Three Things to
Unlearn from School, Ben Casnocha cites Bill Bullard's list of three bad
habits of thought: Attaching importance
to personal opinions, solving given problems, and earning the approval of
others. Bullard's proposed alternatives don't look very good to me, but Bullard has surely
identified some important problems.

I can think of other school-inculcated bad habits of
thought, too many to list, but I'll name two of my least favorite.

I suspect the most
dangerous habit of thought taught in schools is that even if you don't really
understand something, you should parrot it back anyway. One of the most fundamental life skills is
realizing when you are confused, and school actively destroys this ability -
teaches students that they "understand" when they can successfully
answer questions on an exam, which is very very very far from absorbing the
knowledge and making it a part of you. Students learn the habit that eating consists of putting food into
mouth; the exams can't test for chewing or swallowing, and so they starve.

Much of this problem may come from needing to take three
4-credit courses per quarter, with a textbook chapter plus homework to be done
every week - the courses are timed
for frantic memorization, it's not possible
to deeply chew over and leisurely digest knowledge in the same period. College students aren't allowed to be confused; if they started saying, "Wait, do I
really understand this? Maybe I'd better
spend a few days looking up related papers, or consult another textbook," they'd
fail all the courses they took that quarter. A month later they would understand the material far better and remember
it much longer - but one month after finals is too late; it counts for nothing
in the lunatic university utility function.

Many students who have gone through this process no longer
even realize when something confuses
them, or notice gaps in their understanding. They have been trained out of pausing to think.

I recall reading, though I can't remember where, that
physicists in some country were more likely to become extreme religious
fanatics. This confused me, until the
author suggested that physics students are presented with a received truth that
is actually correct, from which they learn the habit of trusting authority.

It may be dangerous to present people with a giant mass of
authoritative...