trentmkelly/LessWrong-43k huggingface dataset:
Use the Try Harder, Luke
by Eliezer Yudkowsky — 2008-10-02T10:52:43.000Z

"When there's a will to fail, obstacles can be found." —John McCarthy

I first watched Star Wars IV-VI when I was very young. Seven, maybe, or nine? So my memory was dim, but I recalled Luke Skywalker as being, you know, this cool Jedi guy.

Imagine my horror and disappointment, when I watched the saga again, years later, and discovered that Luke was a whiny teenager.

I mention this because yesterday, I looked up, on Youtube, the source of the Yoda quote: "Do, or do not. There is no try."

Oh. My. Cthulhu.

Along with the Youtube clip in question, I present to you a little-known outtake from the scene, in which the director and writer, George Lucas, argues with Mark Hamill, who played Luke Skywalker:

Luke: All right, I'll give it a try.
Yoda: No! Try not. Do. Or do not. There is no try.
Luke raises his hand, and slowly, the X-wing begins to rise out of the water—Yoda's eyes widen—but then the ship sinks again.

Mark Hamill: "Um, George..."

George Lucas: "What is it now?"

Mark: "So... according to the script, next I say, 'I can't. It's too big'."

George: "That's right."

Mark: "Shouldn't Luke maybe give it another shot?"

George: "No. Luke gives up, and sits down next to Yoda—"

Mark: "This is the hero who's going to take down the Empire? Look, it was one thing when he was a whiny teenager at the beginning, but he's in Jedi training now. Last movie he blew up the Death Star. Luke should be showing a little backbone."

George: "No. You give up. And then Yoda lectures you for a while, and you say, 'You want the impossible'. Can you remember that?"

Mark: "Impossible? What did he do, run a formal calculation to arrive at a mathematical proof? The X-wing was already starting to rise out of the swamp! That's the feasibility demonstration right there! Luke loses it for a second and the ship sinks back—and now he says it's impossible? Not to mention that Yoda, who's got literally eight hundred years of seniority in the field, just told him it should be doable—"

George: "And then you walk away."

Mark: "It's his friggin' spaceship! If he leaves it in the swamp, he's stuck on Dagobah for the rest of his miserable life!&...
Trying to Try
by Eliezer Yudkowsky — 2008-10-01T08:58:38.000Z

"No! Try not! Do, or do not. There is no try."
—Yoda

Years ago, I thought this was yet another example of Deep Wisdom that is actually quite stupid. SUCCEED is not a primitive action. You can't just decide to win by choosing hard enough. There is never a plan that works with probability 1.

But Yoda was wiser than I first realized.

The first elementary technique of epistemology—it's not deep, but it's cheap—is to distinguish the quotation from the referent. Talking about snow is not the same as talking about "snow". When I use the word "snow", without quotes, I mean to talk about snow; and when I use the word ""snow"", with quotes, I mean to talk about the word "snow". You have to enter a special mode, the quotation mode, to talk about your beliefs. By default, we just talk about reality.

If someone says, "I'm going to flip that switch", then by default, they mean they're going to try to flip the switch. They're going to build a plan that promises to lead, by the consequences of its actions, to the goal-state of a flipped switch; and then execute that plan.

No plan succeeds with infinite certainty. So by default, when you talk about setting out to achieve a goal, you do not imply that your plan exactly and perfectly leads to only that possibility. But when you say, "I'm going to flip that switch", you are trying only to flip the switch—not trying to achieve a 97.2% probability of flipping the switch.

So what does it mean when someone says, "I'm going to try to flip that switch?"

Well, colloquially, "I'm going to flip the switch" and "I'm going to try to flip the switch" mean more or less the same thing, except that the latter expresses the possibility of failure. This is why I originally took offense at Yoda for seeming to deny the possibility. But bear with me here.

Much of life's challenge consists of holding ourselves to a high enough standard. I may speak more on this principle later, because it's a lens through which you can view many-but-not-all personal dilemmas—"What standard am I holding myself to? Is it high enough?"

So if much of life's failure consists in holding yourself to too low a standard, you should be wary of demanding too little from yourself—setting goals that are too easy to f...
Intrade and the Dow Drop
by Eliezer Yudkowsky — 2008-10-01T03:12:46.000Z

With today's snapback, the Dow lost 777 and regained 485.

As of this evening, Intrade says the probability of a bailout bill passing by Oct 31st is 85%.

(777-485)/(1-.85) = 1,946. So a bailout bill makes an expected difference of 2000 points on the Dow.

Of course this is a bogus calculation, but it's an interesting one. Not overwhelmingly on-topic for OB, but it involves prediction markets and I didn't see anyone else pointing it out. I hope the bailout fails decisively, so this calculation can be tested.

PS: Bryan Caplan understands Bayes's Rule: It's not possible for both A and ~A to be evidence in favor of B. So which of the two possibilities, "unemployment stays under 8% following a bailout" and "unemployment goes over 8% following a bailout", is evidence for the proposition "the bailout was necessary to prevent economic catastrophe", and which is evidence against? Take your stand now; afterward is too late for us to trust your reasoning.
Awww, a Zebra
by Eliezer Yudkowsky — 2008-10-01T01:28:05.000Z

This image recently showed up on Flickr (original is nicer):

With the caption:

"Alas for those who turn their eyes from zebras and dream of dragons! If we cannot learn to take joy in the merely real, our lives shall be empty indeed." —Eliezer S. Yudkowsky.

"Awww!", I said, and called over my girlfriend over to look.

"Awww!", she said, and then looked at me, and said, "I think you need to take your own advice!"

Me: "But I'm looking at the zebra!"
Her: "On a computer!"
Me: (Turns away, hides face.)
Her: "Have you ever even seen a zebra in real life?"
Me: "Yes! Yes, I have! My parents took me to Lincoln Park Zoo! ...man, I hated that place."

Part of the Joy in the Merely Real subsequence of Reductionism

Next post: "Hand vs. Fingers"

Previous post: "Initiation Ceremony"
The Magnitude of His Own Folly
by Eliezer Yudkowsky — 2008-09-30T11:31:24.000Z

In the years before I met that would-be creator of Artificial General Intelligence (with a funded project) who happened to be a creationist, I would still try to argue with individual AGI wannabes.

In those days, I sort-of-succeeded in convincing one such fellow that, yes, you had to take Friendly AI into account, and no, you couldn't just find the right fitness metric for an evolutionary algorithm. (Previously he had been very impressed with evolutionary algorithms.)

And the one said: Oh, woe! Oh, alas! What a fool I've been! Through my carelessness, I almost destroyed the world! What a villain I once was!

Now, there's a trap I knew I better than to fall into—

—at the point where, in late 2002, I looked back to Eliezer1997's AI proposals and realized what they really would have done, insofar as they were coherent enough to talk about what they "really would have done".

When I finally saw the magnitude of my own folly, everything fell into place at once. The dam against realization cracked; and the unspoken doubts that had been accumulating behind it, crashed through all together. There wasn't a prolonged period, or even a single moment that I remember, of wondering how I could have been so stupid. I already knew how.

And I also knew, all at once, in the same moment of realization, that to say, I almost destroyed the world!, would have been too prideful.

It would have been too confirming of ego, too confirming of my own importance in the scheme of things, at a time when—I understood in the same moment of realization—my ego ought to be taking a major punch to the stomach. I had been so much less than I needed to be; I had to take that punch in the stomach, not avert it.

And by the same token, I didn't fall into the conjugate trap of saying: Oh, well, it's not as if I had code and was about to run it; I didn't really come close to destroying the world. For that, too, would have minimized the force of the punch. It wasn't really loaded? I had proposed and intended to build the gun, and load the gun, and put the gun to my head and pull the trigger; and that was a bit too much self-destructiveness.

I didn't make a grand emotional drama out of it. That would have wasted the force of the punch, averted it into mere tears.

I knew, in the same moment, what I had be...
Friedman's "Prediction vs. Explanation"
by Eliezer Yudkowsky — 2008-09-29T06:15:34.000Z

David D. Friedman asks:

We do ten experiments. A scientist observes the results, constructs a
theory consistent with them, and uses it to predict the results of the
next ten. We do them and the results fit his predictions. A second
scientist now constructs a theory consistent with the results of all
twenty experiments.

The two theories give different predictions for the next experiment. Which do we believe? Why?

One of the commenters links to Overcoming Bias, but as of 11PM on Sep 28th, David's blog's time, no one has given the exact answer that I would have given. It's interesting that a question so basic has received so many answers.
Above-Average AI Scientists
by Eliezer Yudkowsky — 2008-09-28T11:04:11.000Z

Followup to: The Level Above Mine, Competent Elites

(Those who didn't like the last two posts should definitely skip this one.)

I recall one fellow, who seemed like a nice person, and who was quite eager to get started on Friendly AI work, to whom I had trouble explaining that he didn't have a hope. He said to me:

"If someone with a Masters in chemistry isn't intelligent enough, then you're not going to have much luck finding someone to help you."

It's hard to distinguish the grades above your own. And even if you're literally the best in the world, there are still electron orbitals above yours—they're just unoccupied. Someone had to be "the best physicist in the world" during the time of Ancient Greece. Would they have been able to visualize Newton?

At one of the first conferences organized around the tiny little subfield of Artificial General Intelligence, I met someone who was heading up a funded research project specifically declaring AGI as a goal, within a major corporation. I believe he had people under him on his project. He was probably paid at least three times as much as I was paid (at that time). His academic credentials were superior to mine (what a surprise) and he had many more years of experience. He had access to lots and lots of computing power.

And like nearly everyone in the field of AGI, he was rushing forward to write code immediately—not holding off and searching for a sufficiently precise theory to permit stable self-improvement.

In short, he was just the sort of fellow that... Well, many people, when they hear about Friendly AI, say: "Oh, it doesn't matter what you do, because [someone like this guy] will create AI first." He's the sort of person about whom journalists ask me, "You say that this isn't the time to be talking about regulation, but don't we need laws to stop people like this from creating AI?"

"I suppose," you say, your voice heavy with irony, "that you're about to tell us, that this person doesn't really have so much of an advantage over you as it might seem. Because your theory—whenever you actually come up with a theory—is going to be so much better than his. Or," your voice becoming even more ironic, "that he's too mired in boring mainstream methodology—"

No. I'm about to tell you that I happened to be ...
Competent Elites
by Eliezer Yudkowsky — 2008-09-27T00:07:24.000Z

Followup to: The Level Above Mine

(Anyone who didn't like yesterday's post should probably avoid this one.)

I remember what a shock it was to first meet Steve Jurvetson, of the venture capital firm Draper Fisher Jurvetson.

Steve Jurvetson talked fast and articulately, could follow long chains of reasoning, was familiar with a wide variety of technologies, and was happy to drag in analogies from outside sciences like biology—good ones, too.

I once saw Eric Drexler present an analogy between biological immune systems and the "active shield" concept in nanotechnology, arguing that just as biological systems managed to stave off invaders without the whole community collapsing, nanotechnological immune systems could do the same.

I thought this was a poor analogy, and was going to point out some flaws during the Q&A. But Steve Jurvetson, who was in line before me, proceeded to demolish the argument even more thoroughly. Jurvetson pointed out the evolutionary tradeoff between virulence and transmission that keeps natural viruses in check, talked about how greater interconnectedness led to larger pandemics—it was very nicely done, demolishing the surface analogy by correct reference to deeper biological details.

I was shocked, meeting Steve Jurvetson, because from everything I'd read about venture capitalists before then, VCs were supposed to be fools in business suits, who couldn't understand technology or engineers or the needs of a fragile young startup, but who'd gotten ahold of large amounts of money by dint of seeming reliable to other business suits.

One of the major surprises I received when I moved out of childhood into the real world, was the degree to which the world is stratified by genuine competence.

Now, yes, Steve Jurvetson is not just a randomly selected big-name venture capitalist. He is a big-name VC who often shows up at transhumanist conferences. But I am not drawing a line through just one data point.

I was invited once to a gathering of the mid-level power elite, where around half the attendees were "CEO of something"—mostly technology companies, but occasionally "something" was a public company or a sizable hedge fund. I was expecting to be the youngest person there, but it turned out that my age wasn't unusual—there were several accomplished individuals who were younger. This was t...
The Level Above Mine
by Eliezer Yudkowsky — 2008-09-26T09:18:34.000Z

(At this point, I fear that I must recurse into a subsequence; but if all goes as planned, it really will be short.)

I once lent Xiaoguang "Mike" Li my copy of "Probability Theory: The Logic of Science". Mike Li read some of it, and then came back and said:

"Wow... it's like Jaynes is a thousand-year-old vampire."

Then Mike said, "No, wait, let me explain that—" and I said, "No, I know exactly what you mean." It's a convention in fantasy literature that the older a vampire gets, the more powerful they become.

I'd enjoyed math proofs before I encountered Jaynes. But E.T. Jaynes was the first time I picked up a sense of formidability from mathematical arguments. Maybe because Jaynes was lining up "paradoxes" that had been used to object to Bayesianism, and then blasting them to pieces with overwhelming firepower—power being used to overcome others. Or maybe the sense of formidability came from Jaynes not treating his math as a game of aesthetics; Jaynes cared about probability theory, it was bound up with other considerations that mattered, to him and to me too.

For whatever reason, the sense I get of Jaynes is one of terrifying swift perfection—something that would arrive at the correct answer by the shortest possible route, tearing all surrounding mistakes to shreds in the same motion. Of course, when you write a book, you get a chance to show only your best side. But still.

It spoke well of Mike Li that he was able to sense the aura of formidability surrounding Jaynes. It's a general rule, I've observed, that you can't discriminate between levels too far above your own. E.g., someone once earnestly told me that I was really bright, and "ought to go to college". Maybe anything more than around one standard deviation above you starts to blur together, though that's just a cool-sounding wild guess.

So, having heard Mike Li compare Jaynes to a thousand-year-old vampire, one question immediately popped into my mind:

"Do you get the same sense off me?" I asked.

Mike shook his head. "Sorry," he said, sounding somewhat awkward, "it's just that Jaynes is..."

"No, I know," I said. I hadn't thought I'd reached Jaynes's level. I'd only been curious about how I came across to other people.

I aspire to Jaynes's level. I aspire to become as much the master of Artificial Intelligence / refle...
My Naturalistic Awakening
by Eliezer Yudkowsky — 2008-09-25T06:58:35.000Z

In yesterday's episode, Eliezer2001 is fighting a rearguard action against the truth. Only gradually shifting his beliefs, admitting an increasing probability in a different scenario, but never saying outright, "I was wrong before." He repairs his strategies as they are challenged, finding new justifications for just the same plan he pursued before.

(Of which it is therefore said: "Beware lest you fight a rearguard retreat against the evidence, grudgingly conceding each foot of ground only when forced, feeling cheated. Surrender to the truth as quickly as you can. Do this the instant you realize what you are resisting; the instant you can see from which quarter the winds of evidence are blowing against you.")

Memory fades, and I can hardly bear to look back upon those times—no, seriously, I can't stand reading my old writing. I've already been corrected once in my recollections, by those who were present. And so, though I remember the important events, I'm not really sure what order they happened in, let alone what year.

But if I had to pick a moment when my folly broke, I would pick the moment when I first comprehended, in full generality, the notion of an optimization process. That was the point at which I first looked back and said, "I've been a fool."

Previously, in 2002, I'd been writing a bit about the evolutionary psychology of human general intelligence—though at the time, I thought I was writing about AI; at this point I thought I was against anthropomorphic intelligence, but I was still looking to the human brain for inspiration. (The paper in question is "Levels of Organization in General Intelligence", a requested chapter for the volume "Artificial General Intelligence", which finally came out in print in 2007.)

So I'd been thinking (and writing) about how natural selection managed to cough up human intelligence; I saw a dichotomy between them, the blindness of natural selection and the lookahead of intelligent foresight, reasoning by simulation versus playing everything out in reality, abstract versus concrete thinking. And yet it was natural selection that created human intelligence, so that our brains, though not our thoughts, are entirely made according to the signature of natural selection.

To this day, this still seems to me like a reasonably shattering insight, and so it dri...
Fighting a Rearguard Action Against the Truth
by Eliezer Yudkowsky — 2008-09-24T01:23:30.000Z

When we last left Eliezer2000, he was just beginning to investigate the question of how to inscribe a morality into an AI. His reasons for doing this don't matter at all, except insofar as they happen to historically demonstrate the importance of perfectionism. If you practice something, you may get better at it; if you investigate something, you may find out about it; the only thing that matters is that Eliezer2000 is, in fact, focusing his full-time energies on thinking technically about AI morality; rather than, as previously, finding an justification for not spending his time this way. In the end, this is all that turns out to matter.

But as our story begins—as the sky lightens to gray and the tip of the sun peeks over the horizon—Eliezer2001 hasn't yet admitted that Eliezer1997 was mistaken in any important sense. He's just making Eliezer1997's strategy even better by including a contingency plan for "the unlikely event that life turns out to be meaningless"...

...which means that Eliezer2001 now has a line of retreat away from his mistake.

I don't just mean that Eliezer2001 can say "Friendly AI is a contingency plan", rather than screaming "OOPS!"

I mean that Eliezer2001 now actually has a contingency plan. If Eliezer2001 starts to doubt his 1997 metaethics, the Singularity has a fallback strategy, namely Friendly AI. Eliezer2001 can question his metaethics without it signaling the end of the world.

And his gradient has been smoothed; he can admit a 10% chance of having previously been wrong, then a 20% chance. He doesn't have to cough out his whole mistake in one huge lump.

If you think this sounds like Eliezer2001 is too slow, I quite agree.

Eliezer1996-2000's strategies had been formed in the total absence of "Friendly AI" as a consideration. The whole idea was to get a superintelligence, any superintelligence, as fast as possible—codelet soup, ad-hoc heuristics, evolutionary programming, open-source, anything that looked like it might work—preferably all approaches simultaneously in a Manhattan Project. ("All parents did the things they tell their children not to do. That's how they know to tell them not to do it." John Moore, Slay and Rescue.) It's not as if adding one more approach could hurt.

His attitudes toward technological progress have been formed—...
That Tiny Note of Discord
by Eliezer Yudkowsky — 2008-09-23T06:02:11.000Z

When we last left Eliezer1997, he believed that any superintelligence would automatically do what was "right", and indeed would understand that better than we could; even though, he modestly confessed, he did not understand the ultimate nature of morality. Or rather, after some debate had passed, Eliezer1997 had evolved an elaborate argument, which he fondly claimed to be "formal", that we could always condition upon the belief that life has meaning; and so cases where superintelligences did not feel compelled to do anything in particular, would fall out of consideration. (The flaw being the unconsidered and unjustified equation of "universally compelling argument" with "right".)

So far, the young Eliezer is well on the way toward joining the "smart people who are stupid because they're skilled at defending beliefs they arrived at for unskilled reasons". All his dedication to "rationality" has not saved him from this mistake, and you might be tempted to conclude that it is useless to strive for rationality.

But while many people dig holes for themselves, not everyone succeeds in clawing their way back out.

And from this I learn my lesson: That it all began—

—with a small, small question; a single discordant note; one tiny lonely thought...

As our story starts, we advance three years to Eliezer2000, who in most respects resembles his self of 1997. He currently thinks he's proven that building a superintelligence is the right thing to do if there is any right thing at all. From which it follows that there is no justifiable conflict of interest over the Singularity, among the peoples and persons of Earth.

This is an important conclusion for Eliezer2000, because he finds the notion of fighting over the Singularity to be unbearably stupid. (Sort of like the notion of God intervening in fights between tribes of bickering barbarians, only in reverse.) Eliezer2000's self-concept does not permit him—he doesn't even want—to shrug and say, "Well, my side got here first, so we're going to seize the banana before anyone else gets it." It's a thought too painful to think.

And yet then the notion occurs to him:

Maybe some people would prefer an AI do particular things, such as not kill them, even if life is meaningless?

His immediately following thought is the obvious one, given his premises:

In the

...