trentmkelly/LessWrong-43k huggingface dataset:
Rationality Quotes 19
by Eliezer Yudkowsky — 2008-10-12T20:10:49.000Z

"I don't know that I ever wanted greatness, on its own. It seems rather like wanting to be an engineer, rather than wanting to design something - or wanting to be a writer, rather than wanting to write. It should be a by-product, not a thing in itself. Otherwise, it's just an ego trip."
-- Roger Zelazny, Prince of Chaos

"Many assumptions that we have long been comfortable with are lined up like dominoes."
-- Omega

"I know of no law of logic demanding that every event have a cause."
-- John K. Clark

"It's perfectly accurate, the accuracy only possessable by subjunctive syllogisms."
-- Damien R. Sullivan

"Money makes the world go round. Love just barely keeps it from blowing up."
-- Unknown

"Soon things got out of hand, where they have remained ever since."
-- Larry Gonick, The Cartoon History of the Universe

"Finally, ask yourself this: Are you sure you've really been abducted by aliens? Do you really want to know? For peace of mind and serenity of spirit, you may only need to remember the following: Ignorance is bliss, Prozac is cheap."
-- Pat Krass

"So it was that on the ninety-fifth day of false winter in the year 2929 since the founding of Neverness, we vowed above all else to seek wisdom and truth, even though our seeking should lead to our death and to the ruin of all that we loved and held dear."
-- David Zindell, Neverness
The Ritual
by Eliezer Yudkowsky — 2008-10-11T23:52:10.000Z

The room in which Jeffreyssai received his non-beisutsukai visitors was quietly formal, impeccably appointed in only the most conservative tastes. Sunlight and outside air streamed through a grillwork of polished silver, a few sharp edges making it clear that this wall was not to be opened. The floor and walls were glass, thick enough to distort, to a depth sufficient that it didn’t matter what might be underneath. Upon the surfaces of the glass were subtly scratched patterns of no particular meaning, scribed as if by the hand of an artistically inclined child (and this was in fact the case).

Elsewhere in Jeffreyssai’s home there were rooms of other style; but this, he had found, was what most outsiders expected of a Bayesian Master, and he chose not to enlighten them otherwise. That quiet amusement was one of life’s little joys, after all.

The guest sat across from him, knees on the pillow and heels behind. She was here solely upon the business of her Conspiracy, and her attire showed it: a form-fitting jumpsuit of pink leather with even her hands gloved—all the way to the hood covering her head and hair, though her face lay plain and unconcealed beneath.

And so Jeffreyssai had chosen to receive her in this room.

Jeffreyssai let out a long breath, exhaling. “Are you sure?”

“Oh,” she said, “and do I have to be absolutely certain before my advice can shift your opinions? Does it not suffice that I am a domain expert, and you are not?”

Jeffreyssai’s mouth twisted up at the corner in a half-smile. “How do you know so much about the rules, anyway? You’ve never had so much as a Planck length of formal training.”

“Do you even need to ask?” she said dryly. “If there’s one thing that you beisutsukai do love to go on about, it’s the reasons why you do things.”

Jeffreyssai inwardly winced at the thought of trying to pick up rationality by watching other people talk about it—

“And don’t inwardly wince at me like that,” she said. “I’m not trying to be a rationalist myself, just trying to win an argument with a rationalist. There’s a difference, as I’m sure you tell your students.”

Can she really read me that well? Jeffreyssai looked out thro...
Crisis of Faith
by Eliezer Yudkowsky — 2008-10-10T22:08:48.000Z

It ain’t a true crisis of faith unless things could just as easily go either way.

—Thor Shenkel

Many in this world retain beliefs whose flaws a ten-year-old could point out, if that ten-year-old were hearing the beliefs for the first time. These are not subtle errors we’re talking about. They would be child's play for an unattached mind to relinquish, if the skepticism of a ten-year-old were applied without evasion. As Premise Checker put it, "Had the idea of god not come along until the scientific age, only an exceptionally weird person would invent such an idea and pretend that it explained anything."

And yet skillful scientific specialists, even the major innovators of a field, even in this very day and age, do not apply that skepticism successfully. Nobel laureate Robert Aumann, of Aumann’s Agreement Theorem, is an Orthodox Jew: I feel reasonably confident in venturing that Aumann must, at one point or another, have questioned his faith. And yet he did not doubt successfully. We change our minds less often than we think.

This should scare you down to the marrow of your bones. It means you can be a world-class scientist and conversant with Bayesian mathematics and still fail to reject a belief whose absurdity a fresh-eyed ten-year-old could see. It shows the invincible defensive position which a belief can create for itself, if it has long festered in your mind.

What does it take to defeat an error that has built itself a fortress?

But by the time you know it is an error, it is already defeated. The dilemma is not “How can I reject long-held false belief X?” but “How do I know if long-held belief X is false?” Self-honesty is at its most fragile when we’re not sure which path is the righteous one. And so the question becomes:

How can we create in ourselves a true crisis of faith, that could just as easily go either way?

Religion is the trial case we can all imagine.2 But if you have cut off all sympathy and now think of theists as evil mutants, then you won’t be able to imagine the real internal trials they face. You won’t be able to ask the question:

What general strategy would a religious person have to follow in order to escape their religion?

I’m sure that some, looking at this challenge, are already rattling off a list of standard atheist talking points—“They would have to admit that there wasn’t any Bayesian evidence for God’s existence,” “They woul...
AIs and Gatekeepers Unite!
by Eliezer Yudkowsky — 2008-10-09T17:04:31.000Z

"Bah, everyone wants to be the gatekeeper. What we NEED are AIs."
-- Schizoguy

Some of you have expressed the opinion that the AI-Box Experiment doesn't seem so impossible after all. That's the spirit! Some of you even think you know how I did it.

There are folks aplenty who want to try being the Gatekeeper. You can even find people who sincerely believe that not even a transhuman AI could persuade them to let it out of the box, previous experiments notwithstanding. But finding anyone to play the AI - let alone anyone who thinks they can play the AI and win - is much harder.

Me, I'm out of the AI game, unless Larry Page wants to try it for a million dollars or something.

But if there's anyone out there who thinks they've got what it takes to be the AI, leave a comment. Likewise anyone who wants to play the Gatekeeper.

Matchmaking and arrangements are your responsibility.

Make sure you specify in advance the bet amount, and whether the bet will be asymmetrical. If you definitely intend to publish the transcript, make sure both parties know this. Please note any other departures from the suggested rules for our benefit.

I would ask that prospective Gatekeepers indicate whether they (1) believe that no human-level mind could persuade them to release it from the Box and (2) believe that not even a transhuman AI could persuade them to release it.

As a courtesy, please announce all Experiments before they are conducted, including the bet, so that we have some notion of the statistics even if some meetings fail to take place. Bear in mind that to properly puncture my mystique (you know you want to puncture it), it
will help if the AI and Gatekeeper are both verifiably Real
People<tm>.

"Good luck," he said impartially.
Shut up and do the impossible!
by Eliezer Yudkowsky — 2008-10-08T21:24:50.000Z

The virtue of tsuyoku naritai, "I want to become stronger", is to always keep improving—to do better than your previous failures, not just humbly confess them.

Yet there is a level higher than tsuyoku naritai. This is the virtue of isshokenmei, "make a desperate effort". All-out, as if your own life were at stake. "In important matters, a 'strong' effort usually only results in mediocre results."

And there is a level higher than isshokenmei. This is the virtue I called "make an extraordinary effort". To try in ways other than what you have been trained to do, even if it means doing something different from what others are doing, and leaving your comfort zone. Even taking on the very real risk that attends going outside the System.

But what if even an extraordinary effort will not be enough, because the problem is impossible?

I have already written somewhat on this subject, in On Doing the Impossible. My younger self used to whine about this a lot: "You can't develop a precise theory of intelligence the way that there are precise theories of physics. It's impossible! You can't prove an AI correct. It's impossible! No human being can comprehend the nature of morality—it's impossible! No human being can comprehend the mystery of subjective experience! It's impossible!"

And I know exactly what message I wish I could send back in time to my younger self:

Shut up and do the impossible!

What legitimizes this strange message is that the word "impossible" does not usually refer to a strict mathematical proof of impossibility in a domain that seems well-understood. If something seems impossible merely in the sense of "I see no way to do this" or "it looks so difficult as to be beyond human ability"—well, if you study it for a year or five, it may come to seem less impossible, than in the moment of your snap initial judgment.

But the principle is more subtle than this. I do not say just, "Try to do the impossible", but rather, "Shut up and do the impossible!"

For my illustration, I will take the least impossible impossibility that I have ever accomplished, namely, the AI-Box Experiment.

The AI-Box Experiment, for those of you who haven't yet read about it, had its genesis in the Nth time someone said to me: "Why don't we build an AI, and then just keep it isolated in the compute...
Make an Extraordinary Effort
by Eliezer Yudkowsky — 2008-10-07T15:15:30.000Z

"It is essential for a man to strive with all his heart, and to understand that it is difficult even to reach the average if he does not have the intention of surpassing others in whatever he does."
—Budo Shoshinshu

"In important matters, a 'strong' effort usually results in only mediocre results. Whenever we are attempting anything truly worthwhile our effort must be as if our life is at stake, just as if we were under a physical attack! It is this extraordinary effort—an effort that drives us beyond what we thought we were capable of—that ensures victory in battle and success in life's endeavors."
—Flashing Steel: Mastering Eishin-Ryu Swordsmanship

"A 'strong' effort usually results in only mediocre results"—I have seen this over and over again. The slightest effort suffices to convince ourselves that we have done our best.

There is a level beyond the virtue of tsuyoku naritai ("I want to become stronger"). Isshoukenmei was originally the loyalty that a samurai offered in return for his position, containing characters for "life" and "land". The term evolved to mean "make a desperate effort": Try your hardest, your utmost, as if your life were at stake. It was part of the gestalt of bushido, which was not reserved only for fighting. I've run across variant forms issho kenmei and isshou kenmei; one source indicates that the former indicates an all-out effort on some single point, whereas the latter indicates a lifelong effort.

I try not to praise the East too much, because there's a tremendous selectivity in which parts of Eastern culture the West gets to hear about. But on some points, at least, Japan's culture scores higher than America's. Having a handy compact phrase for "make a desperate all-out effort as if your own life were at stake" is one of those points. It's the sort of thing a Japanese parent might say to a student before exams—but don't think it's cheap hypocrisy, like it would be if an American parent made the same statement. They take exams very seriously in Japan.

Every now and then, someone asks why the people who call themselves "rationalists" don't always seem to do all that much better in life, and from my own history the answer seems straightforward: It takes a tr...
On Doing the Impossible
by Eliezer Yudkowsky — 2008-10-06T15:13:26.000Z

"Persevere." It's a piece of advice you'll get from a whole lot of high achievers in a whole lot of disciplines. I didn't understand it at all, at first.

At first, I thought "perseverance" meant working 14-hour days. Apparently, there are people out there who can work for 10 hours at a technical job, and then, in their moments between eating and sleeping and going to the bathroom, seize that unfilled spare time to work on a book. I am not one of those people—it still hurts my pride even now to confess that. I'm working on something important; shouldn't my brain be willing to put in 14 hours a day? But it's not. When it gets too hard to keep working, I stop and go read or watch something. Because of that, I thought for years that I entirely lacked the virtue of "perseverance".

In accordance with human nature, Eliezer1998 would think things like: "What counts is output, not input." Or, "Laziness is also a virtue—it leads us to back off from failing methods and think of better ways." Or, "I'm doing better than other people who are working more hours. Maybe, for creative work, your momentary peak output is more important than working 16 hours a day." Perhaps the famous scientists were seduced by the Deep Wisdom of saying that "hard work is a virtue", because it would be too awful if that counted for less than intelligence?

I didn't understand the virtue of perseverance until I looked back on my journey through AI, and realized that I had overestimated the difficulty of almost every single important problem.

Sounds crazy, right? But bear with me here.

When I was first deciding to challenge AI, I thought in terms of 40-year timescales, Manhattan Projects, planetary computing networks, millions of programmers, and possibly augmented humans.

This is a common failure mode in AI-futurism which I may write about later; it consists of the leap from "I don't know how to solve this" to "I'll imagine throwing something really big at it". Something huge enough that, when you imagine it, that imagination creates a feeling of impressiveness strong enough to be commensurable with the problem. (There's a fellow currently on the AI list who goes around saying that AI will cost a quadrillion dollars—we can't get AI without spending a quadrillion dollars, but we could get AI at an...
Bay Area Meetup for Singularity Summit
by Eliezer Yudkowsky — 2008-10-06T12:50:04.000Z

Posted on behalf of Mike Howard:

This is a call for preferences on the proposed Bay Area meetup to coincide with the Singularity Summit on 24-25 October. Not just for Singularitarians, all aspiring rationalists are welcome. From the replies so far it's likely to be in San Jose.

Eliezer, myself and probably most Summit attendees would really rather avoid the night between the Friday Workshop and Saturday Summit, so maybe either Saturday evening or sometime Thursday or Sunday?

Please comment below or email me (cursor_loop 4t yahoo p0int com) if you might want to come, and if you have any preferences such as when and where you can come, when and where you'd prefer to come, and any recommendations for a particular place to go. (Comments preferred to emails.) We need to pick a date ASAP before everyone books travel.
My Bayesian Enlightenment
by Eliezer Yudkowsky — 2008-10-05T16:45:35.000Z

I remember (dimly, as human memories go) the first time I self-identified as a "Bayesian". Someone had just asked a malformed version of an old probability puzzle, saying:

If I meet a mathematician on the street, and she says, "I have two children, and at least one of them is a boy," what is the probability that they are both boys?

In the correct version of this story, the mathematician says "I have two children", and you ask, "Is at least one a boy?", and she answers "Yes". Then the probability is 1/3 that they are both boys.

But in the malformed version of the story—as I pointed out—one would common-sensically reason:

If the mathematician has one boy and one girl, then my prior probability for her saying 'at least one of them is a boy' is 1/2 and my prior probability for her saying 'at least one of them is a girl' is 1/2. There's no reason to believe, a priori, that the mathematician will only mention a girl if there is no possible alternative.

So I pointed this out, and worked the answer using Bayes's Rule, arriving at a probability of 1/2 that the children were both boys. I'm not sure whether or not I knew, at this point, that Bayes's rule was called that, but it's what I used.

And lo, someone said to me, "Well, what you just gave is the Bayesian answer, but in orthodox statistics the answer is 1/3. We just exclude the possibilities that are ruled out, and count the ones that are left, without trying to guess the probability that the mathematician will say this or that, since we have no way of really knowing that probability—it's too subjective."

I responded—note that this was completely spontaneous—"What on Earth do you mean? You can't avoid assigning a probability to the mathematician making one statement or another. You're just assuming the probability is 1, and that's unjustified."

To which the one replied, "Yes, that's what the Bayesians say. But frequentists don't believe that."

And I said, astounded: "How can there possibly be such a thing as non-Bayesian statistics?"

That was when I discovered that I was of the type called 'Bayesian'. As far as I can tell, I was born that way. My mathematical intuitions were such that everything Bayesians said seemed perfectly straightforward and simple, the obvious way I would do it myself; whereas the things frequentists sai...
Beyond the Reach of God
by Eliezer Yudkowsky — 2008-10-04T15:42:57.000Z

Today's post is a tad gloomier than usual, as I measure such things. It deals with a thought experiment I invented to smash my own optimism, after I realized that optimism had misled me. Those readers sympathetic to arguments like, "It's important to keep our biases because they help us stay happy," should consider not reading. (Unless they have something to protect, including their own life.)

So! Looking back on the magnitude of my own folly, I realized that at the root of it had been a disbelief in the Future's vulnerability—a reluctance to accept that things could really turn out wrong. Not as the result of any explicit propositional verbal belief. More like something inside that persisted in believing, even in the face of adversity, that everything would be all right in the end.

Some would account this a virtue (zettai daijobu da yo), and others would say that it's a thing necessary for mental health.

But we don't live in that world. We live in the world beyond the reach of God.

It's been a long, long time since I believed in God. Growing up in an Orthodox Jewish family, I can recall the last remembered time I asked God for something, though I don't remember how old I was. I was putting in some request on behalf of the next-door-neighboring boy, I forget what exactly—something along the lines of, "I hope things turn out all right for him," or maybe "I hope he becomes Jewish."

I remember what it was like to have some higher authority to appeal to, to take care of things I couldn't handle myself. I didn't think of it as "warm", because I had no alternative to compare it to. I just took it for granted.

Still I recall, though only from distant childhood, what it's like to live in the conceptually impossible possible world where God exists. Really exists, in the way that children and rationalists take all their beliefs at face value.

In the world where God exists, does God intervene to optimize everything? Regardless of what rabbis assert about the fundamental nature of reality, the take-it-seriously operational answer to this question is obviously "No". You can't ask God to bring you a lemonade from the refrigerator instead of getting one yourself. When I believed in God after the serious fashion of a child, so very long ago, I didn't believe that.

Postulating that ...
Rationality Quotes 18
by Eliezer Yudkowsky — 2008-10-03T12:39:30.000Z

Q. Why does "philosophy of consciousness/nature of reality" seem to interest you so much?
A. Take away consciousness and reality and there's not much left.
-- Greg Egan, interview in Eidolon 15

"But I am not an object. I am not a noun, I am an adjective. I am the way matter behaves when it is organized in a John K Clark-ish way. At the present time only one chunk of matter in the universe behaves that way; someday that could change."
-- John K Clark

"Would it be good advice, once copying becomes practical, to make lots of copies when good things happen, and none (or perhaps even killing off your own personal instance) on bad things? Will this change the subjective probability of good events?"
-- Hal Finney

"Waiting for the bus is a bad idea if you turn out to be the bus driver."
-- Michael M. Butler on the Singularity

"You are free. Free of anything I do or say, and of any consequence. You may rest assured that all hurts are forgiven, all loveliness remembered, and treasured. I am busy and content and loved. I hope you are the same. Bless you."
-- Walter Jon Williams, "Knight Moves"

"A man with one watch knows what time it is; a man with two watches is never sure."
-- Lee Segall
What’s worse than coercion?
by KatjaGrace — 2008-10-02T18:00:00.000Z

Desperation is coercive, or so it is said. The analogy between having a gun to your head and starvation at your door is a good one, as far as decision making is concerned.

So why do we always state this just before doing the last thing we would do to someone with a gun to their head?

Our reasoning goes:

She’s only working for nothing/selling her kidneys/poisoning her water supply because she has no other option.

Therefore she’s effectively being coerced.

That’s terrible.

We won’t allow it. We won’t buy her t-shirts or her kidneys.

Now she can’t be coerced. Hoorah!

So we take away the ‘not getting shot in the head’ option.

This would be fine if we also gave another choice. However if we did that that the person would no longer be desperate, and thus no longer ‘coerced’ anyway (and so there would be no need to interfere). There should never be a need to prevent coercion by taking away choices.

In our analogy, there is a difference between preventing coercion by forcing someone to be shot and by giving them a safe exit.