trentmkelly/LessWrong-43k huggingface dataset:
The Thing That I Protect
by Eliezer Yudkowsky — 2009-02-07T19:18:44.000Z

Followup to: Something to Protect, Value is Fragile

"Something to Protect" discursed on the idea of wielding rationality in the service of something other than "rationality". Not just that rationalists ought to pick out a Noble Cause as a hobby to keep them busy; but rather, that rationality itself is generated by having something that you care about more than your current ritual of cognition.

So what is it, then, that I protect?

I quite deliberately did not discuss that in "Something to Protect", leaving it only as a hanging implication. In the unlikely event that we ever run into aliens, I don't expect their version of Bayes's Theorem to be mathematically different from ours, even if they generated it in the course of protecting different and incompatible values. Among humans, the idiom of having "something to protect" is not bound to any one cause, and therefore, to mention my own cause in that post would have harmed its integrity. Causes are dangerous things, whatever their true importance; I have written somewhat on this, and will write more about it.

But still - what is it, then, the thing that I protect?

Friendly AI? No - a thousand times no - a thousand times not anymore. It's not thinking of the AI that gives me strength to carry on even in the face of inconvenience.

I would be a strange and dangerous AI wannabe if that were my cause - the image in my mind of a perfected being, an existence greater than humankind. Maybe someday I'll be able to imagine such a child and try to build one, but for now I'm too young to be a father.

Those of you who've been following along recent discussions, particularly "Value is Fragile", might have noticed something else that I might, perhaps, hold precious. Smart agents want to protect the physical representation of their utility function for almost the same reason that male organisms are built to be protective of their testicles. From the standpoint of the alien god, natural selection, losing the germline - the gene-carrier that propagates the pattern into the next generation - means losing almost everything that natural selection cares about. Unless you already have children to protect, can protect relatives, etcetera - few are the absolute and unqualified statements that ...
Epilogue: Atonement (8/8)
by Eliezer Yudkowsky — 2009-02-06T11:52:42.000Z

(Part 8 of 8 in "Three Worlds Collide")

Fire came to Huygens.

The star erupted.

Stranded ships, filled with children doomed by a second's last delay, still milled around the former Earth transit point. Too many doomed ships, far too many doomed ships. They should have left a minute early, just to be sure; but the temptation to load in that one last child must have been irresistable. To do the warm and fuzzy thing just this one time, instead of being cold and calculating. You couldn't blame them, could you...?

Yes, actually, you could.

The Lady Sensory switched off the display. It was too painful.

On the Huygens market, the price of a certain contract spiked to 100%. They were all rich in completely worthless assets for the next nine minutes, until the supernova blast front arrived.

"So," the Lord Pilot finally said. "What kind of asset retains its value in a market with nine minutes to live?"

"Booze for immediate delivery," the Master of Fandom said promptly. "That's what you call a -"

"Liquidity preference," the others chorused.

The Master laughed. "All right, that was too obvious. Well... chocolate, sex -"

"Not necessarily," said the Lord Pilot. "If you can use up the whole supply of chocolate at once, does demand outstrip supply? Same with sex - the value could actually drop if everyone's suddenly willing. Not to mention: Nine minutes?"

"All right then, expert oral sex from experienced providers. And hard drugs with dangerous side effects; the demand would rise hugely relative to supply -"

"This is inane," the Ship's Engineer commented.

The Master of Fandom shrugged. "What do you say in the unrecorded last minutes of your life that is not inane?"

"It doesn't matter," said the Lady Sensory. Her face was strangely tranquil. "Nothing that we do now matters. We won't have to live with the consequences. No one will. All this time will be obliterated when the blast front hits. The role I've always played, the picture that I have of me... it doesn't matter. There's... a peace... in not having to be Dalia Ancromein any more."

The others looked at her. Talk about killing the mood.

"Well," the Master of Fandom said, "since you raise the subject, I suppose it would be peaceful if not for the screaming terror."

"You don't have to feel the scre...
True Ending: Sacrificial Fire (7/8)
by Eliezer Yudkowsky — 2009-02-05T10:57:12.000Z

(Part 7 of 8 in "Three Worlds Collide")

Standing behind his target, unnoticed, the Ship's Confessor had produced from his sleeve the tiny stunner - the weapon which he alone on the ship was authorized to use, if he made a determination of outright mental breakdown. With a sudden motion, his arm swept outward -

- and anesthetized the Lord Akon.

Akon crumpled almost instantly, as though most of his strings had already been cut, and only a few last strands had been holding his limbs in place.

Fear, shock, dismay, sheer outright surprise: that was the Command Conference staring aghast at the Confessor.

From the hood came words absolutely forbidden to originate from that shadow: the voice of command. "Lord Pilot, take us through the starline back to the Huygens system. Get us moving now, you are on the critical path. Lady Sensory, I need you to enforce an absolute lockdown on all of this ship's communication systems except for a single channel under your direct control. Master of Fandom, get me proxies on the assets of every being on this ship. We are going to need capital."

For a moment, the Command Conference was frozen, voiceless and motionless, as everyone waited for someone else do to something.

And then -

"Moving the Impossible now, my lord," said the Lord Pilot. His face was sane once again. "What's your plan?"

"He is not your lord!" cried the Master of Fandom. Then his voice dropped. "Excuse me. Confessor - it did not appear to me that our Lord Administrator was insane. And you, of all people, cannot just seize power -"

"True," said the one, "Akon was sane. But he was also an honest man who would keep his word once he gave it, and that I could not allow. As for me - I have betrayed my calling three times over, and am no longer a Confessor." With that same response, the once-Confessor swept back the hood -

At any other time, the words and the move and the revealed face would have provoked shock to the point of fainting. On this day, with the whole human species at stake, it seemed merely interesting. Chaos had already run loose, madness was already unleashed into the world, and a little more seemed of little consequence.

"Ancestor," said the Master, "you are twice prohibited from exercising any power here."

The former Confessor smiled dryl...
Normal Ending: Last Tears (6/8)
by Eliezer Yudkowsky — 2009-02-04T08:45:35.000Z

(Part 6 of 8 in "Three Worlds Collide")

Today was the day.

The streets of ancient Earth were crowded to overbursting with people looking up at the sky, faces crowded up against windows.

Waiting for their sorrows to end.

Akon was looking down at their faces, from the balcony of a room in a well-guarded hotel. There were many who wished to initiate violence against him, which was understandable. Fear showed on most of the faces in the crowd, rage in some; a very few were smiling, and Akon suspected they might have simply given up on holding themselves together. Akon wondered what his own face looked like, right now.

The streets were less crowded than they might have been, only a few weeks earlier.

No one had told the Superhappies about that part. They'd sent an ambassadorial ship "in case you have any urgent requests we can help with", arriving hard on the heels of the Impossible. That ship had not been given any of the encryption keys to the human Net, nor allowed to land. It had made the Superhappies extremely suspicious, and the ambassadorial ship had disgorged a horde of tiny daughters to observe the rest of the human starline network -

But if the Superhappies knew, they would have tried to stop it. Somehow.

That was a price that no one was willing to include into the bargain, no matter what. There had to be that - alternative.

A quarter of the Impossible Possible World's crew had committed suicide, when the pact and its price became known. Others, Akon thought, had waited only to be with their families. The percentage on Earth... would probably be larger. The government, what was left of it, had refused to publish statistics. All you saw was the bodies being carried out of the apartments - in plain, unmarked boxes, in case the Superhappy ship was using optical surveillance.

Akon swallowed. The fear was already drying his own throat, the fear of changing, of becoming something else that wasn't quite him. He understood the urge to end that fear, at any price. And yet at the same time, he didn't, couldn't understand the suicides. Was being dead a smaller change? To die was not to leave the world, not to escape somewhere else; it was the simultaneous change of every piece of yourself into nothing.

Many parents had made that choice for their children. The governmen...
Three Worlds Decide (5/8)
by Eliezer Yudkowsky — 2009-02-03T09:14:02.000Z

(Part 5 of 8 in "Three Worlds Collide")

Akon strode into the main Conference Room; and though he walked like a physically exhausted man, at least his face was determined. Behind
him, the shadowy Confessor followed.

The Command Conference looked up at him, and exchanged glances.

"You look better," the Ship's Master of Fandom ventured.

Akon put a hand on the back of his seat, and paused. Someone was absent. "The Ship's Engineer?"

The
Lord Programmer frowned. "He said he had an experiment to run, my
lord. He refused to clarify further, but I suppose it must have
something to do with the Babyeaters' data -"

"You're joking," Akon said. "Our Ship's Engineer is off Nobel-hunting? Now? With the fate of the human species at stake?"

The Lord Programmer shrugged. "He seemed to think it was important, my lord."

Akon
sighed. He pulled his chair back and half-slid, half-fell into it. "I
don't suppose that the ship's markets have settled down?"

The Lord Pilot grinned sardonically. "Read for yourself."

Akon
twitched, calling up a screen. "Ah, I see. The ship's Interpreter of
the Market's Will reports, and I quote, 'Every single one of the
underlying assets in my market is going up and down like a fucking
yo-yo while the ship's hedgers try to adjust to a Black Swan that's
going to wipe out ninety-eight percent of their planetside risk
capital. Even the spot prices on this ship are going crazy; either
we've got bubble traders coming out of the woodwork, or someone
seriously believes that sex is overvalued relative to orange juice.
One derivatives trader says she's working on a contract that will have
a clearly defined value in the event that aliens wipe out the entire
human species, but she says it's going to take a few hours and I say
she's on crack. Indeed I believe an actual majority of the people
still trying to trade in this environment are higher than the
heliopause. Bid-ask spreads are so wide you could kick a fucking
football stadium through them, nothing is clearing, and I have
unisolated conditional dependencies coming out of my ass. I have no
fucking clue what the market believes. ...
Interlude with the Confessor (4/8)
by Eliezer Yudkowsky — 2009-02-02T09:11:26.000Z

(Part 4 of 8 in "Three Worlds Collide")

The two of them were alone now, in the Conference Chair's Privilege, the huge private room of luxury more suited to a planet than to space. The Privilege was tiled wall-to-wall and floor-to-ceiling with a most excellent holo of the space surrounding them: the distant stars, the system's sun, the fleeing nova ashes, and the glowing ember of the dwarf star that had siphoned off hydrogen from the main sun until its surface had briefly ignited in a nova flash. It was like falling through the void.

Akon sat on the edge of the four-poster bed in the center of the room, resting his head in his hands. Weariness dulled him at the moment when he most needed his wits; it was always like that in crisis, but this was unusually bad. Under the circumstances, he didn't dare snort a hit of caffeine - it might reorder his priorities. Humanity had yet to discover the drug that was pure energy, that would improve your thinking without the slightest touch on your emotions and values.

"I don't know what to think," Akon said.

The Ship's Confessor was standing stately nearby, in full robes and hood of silver. From beneath the hood came the formal response: "What seems to be confusing you, my friend?"

"Did we go wrong?" Akon said. No matter how hard he tried, he couldn't keep the despair out of his voice. "Did humanity go down the wrong path?"

The Confessor was silent a long time.

Akon waited. This was why he couldn't have talked about the question with anyone else. Only a Confessor would actually think before answering, if asked a question like that.

"I've often wondered that myself," the Confessor finally said, surprising Akon. "There were so many choices, so many branchings in human history - what are the odds we got them all right?"

The hood turned away, angling in the direction of the Superhappy ship - though it was too far away to be visible, everyone on board the Impossible Possible World knew where it was. "There are parts of your question I can't help you with, my lord. Of all people on this ship, I might be most poorly suited to answer... But you do understand, my lord, don't you, that neither the Babyeaters nor the Superhappies are evidence that we went wrong? If you weren't worried before, you shouldn't be any more worried now. The Babyeate...
The Super Happy People (3/8)
by Eliezer Yudkowsky — 2009-02-01T08:18:19.000Z

(Part 3 of 8 in "Three Worlds Collide")

...The Lady Sensory said, in an unsteady voice, "My lords, a third ship has jumped into this system. Not Babyeater, not human."

The holo showed a triangle marked with three glowing dots, the human ship and the Babyeater ship and the newcomers. Then the holo zoomed in, to show -

- the most grotesque spaceship that Akon had ever seen, like a blob festooned with tentacles festooned with acne festooned with small hairs. Slowly, the tentacles of the ship waved, as if in a gentle breeze; and the acne on the tentacles pulsated, as if preparing to burst. It was a fractal of ugliness, disgusting at every level of self-similarity.

"Do the aliens have deflectors up?" said Akon.

"My lord," said Lady Sensory, "they don't have any shields raised. The nova ashes' radiation doesn't seem to bother them. Whatever material their ship is made from, it's just taking the beating."

A silence fell around the table.

"All right," said the Lord Programmer, "that's impressive."

The Lady Sensory jerked, like someone had just slapped her. "We - we just got a signal from them in human-standard format, content encoding marked as Modern English text, followed by a holo -"

"What?" said Akon. "We haven't transmitted anything to them, how could they possibly -"

"Um," said the Ship's Engineer. "What if these aliens really do have, um, 'big angelic powers'?"

"No," said the Ship's Confessor. His hood tilted slightly, as if in wry humor. "It is only history repeating itself."

"History repeating itself?" said the Master of Fandom. "You mean that the ship is from an alternate Everett branch of Earth, or that they somehow independently developed ship-to-ship communication protocols exactly similar to our -"

"No, you dolt," said the Lord Programmer, "he means that the Babyeaters sent the new aliens a massive data dump, just like they sent us. Only this time, the Babyeater data dump included all the data that we sent the Babyeaters. Then the new aliens ran an automatic translation program, like the one we used."

"You gave it away," said the Confessor. There was a slight laugh in his voice. "You should have let them figure it out on their own. One so rarely encounters the apparently supernatural, these days."

Akon shook his head, "Confessor, we don't have time for - never mind.&...
War and/or Peace (2/8)
by Eliezer Yudkowsky — 2009-01-31T08:42:19.000Z

(Part 2 of 8 in "Three Worlds Collide")

..."So the question then is - now what?"

The Lord Pilot jumped up, then, his face flushed. "Put up shields.
Now. We don't gain anything by leaving them down. This is madness!"

"No," said the Ship's Confessor in professional tones, "not madness."

The Pilot slammed his fists on the table. "We're all going to die!"

"They're not as technologically advanced as us," Akon said.
"Suppose the Babyeaters do decide that we need to be exterminated.
Suppose they open fire. Suppose they kill us. Suppose they follow
the starline we opened and find the Huygens system. Then what?"

The Master nodded. "Even with surprise on their side... no. They
can't actually wipe out the human species. Not unless they're a lot
smarter than they seem to be, and it looks to me like, on average,
they're actually a bit dumber than us." The Master glanced at the
Xenopsychologist, who waved her hand in a maybe-gesture.

"But if we leave the ship's shields down," Akon said, "we preserve whatever chance we have of a peaceful resolution to this."

"Peace," said the Lady Sensory, in a peculiar flat tone.

Akon looked at her.

"You want peace with the Babyeaters?"

"Of course -" said Akon, then stopped short.

The Lady Sensory looked around the table. "And the Babyeater children? What about them?"

The Master of Fandom spoke, his voice uncertain. "You can't impose human standards on -"

With a blur of motion and a sharp crack, the Lady Sensory slapped him.

The Ship's Confessor grabbed her arm. "No."

The Lady Sensory stared at the Ship's Confessor.

"No," the Confessor repeated. "No violence. Only argument. Violence doesn't distinguish truth from falsehood, my Lady."

The Lady Sensory slowly lowered her hand, but not her eyes.

"But..." said the Master. "But, my Lady, if they want to be eaten -"

"They don't," said the Xenopsychologist. "Of course they don't.
They run from their parents when the terrible winnowing comes. The
Babyeater children a...
The Baby-Eating Aliens (1/8)
by Eliezer Yudkowsky — 2009-01-30T12:07:57.000Z

(Part 1 of 8 in "Three Worlds Collide")

This is a story of an impossible outcome, where AI never worked, molecular nanotechnology never worked, biotechnology only sort-of worked; and yet somehow humanity not only survived, but discovered a way to travel Faster-Than-Light: The past's Future.

Ships travel through the Alderson starlines, wormholes that appear near stars. The starline network is dense and unpredictable: more than a billion starlines lead away from Sol, but every world explored is so far away as to be outside the range of Earth's telescopes. Most colony worlds are located only a single jump away from Earth, which remains the center of the human universe.

From the colony system Huygens, the crew of the Giant Science Vessel Impossible Possible World have set out to investigate a starline that flared up with an unprecedented flux of Alderson force before subsiding. Arriving, the Impossible discovers the sparkling debris of a recent nova - and -

"ALIENS!"

Every head swung toward the Sensory console. But after that one cryptic outburst, the Lady Sensory didn't even look up from her console: her fingers were frantically twitching commands.

There was a strange moment of silence in the Command Conference while every listener thought the same two thoughts in rapid succession:

Is she nuts? You can't just say "Aliens!", leave it at that, and expect everyone to believe you. Extraordinary claims require extraordinary evidence -

And then,

They came to look at the nova too!

In a situation like this, it befalls the Conference Chair to speak first.

"What? SHIT!" shouted Akon, who didn't realize until later that his words would be inscribed for all time in the annals of history. Akon swung around and looked frantically at the main display of the Command Conference. "Where are they?"

The Lady Sensory looked up from her console, fingers still twitching. "I - I don't know, I just picked up an incoming high-frequency signal - they're sending us enormous amounts of data, petabytes, I had to clear long-term memory and set up an automatic pipe or risk losing the whole -"

"Found them!" shouted the Lord Programmer. "I searched through our Greater Archive and turned up a program to look for anomalous energy sources near local starlines. It's from way back from the first days of exploration, but I managed to fi...
Three Worlds Collide (0/8)
by Eliezer Yudkowsky — 2009-01-30T12:07:52.000Z

"The kind of classic fifties-era first-contact story that Jonathan Swift
might have written, if Jonathan Swift had had a background in game
theory."
-- (Hugo nominee) Peter Watts, "In Praise of Baby-Eating"

Three Worlds Collide is a story I wrote to illustrate some points on naturalistic metaethics and diverse other issues of rational conduct. It grew, as such things do, into a small novella. On publication, it proved widely popular and widely criticized. Be warned that the story, as it wrote itself, ended up containing some profanity and PG-13 content.

The Baby-Eating Aliens

War and/or Peace

The Super Happy People

Interlude with the Confessor

Three Worlds Decide

Normal Ending

True Ending

Atonement

PDF version here.
Value is Fragile
by Eliezer Yudkowsky — 2009-01-29T08:46:30.000Z

If I had to pick a single statement that relies on more Overcoming Bias content I've written than any other, that statement would be:

Any Future not shaped by a goal system with detailed reliable inheritance from human morals and metamorals, will contain almost nothing of worth.

"Well," says the one, "maybe according to your provincial human values, you wouldn't like it. But I can easily imagine a galactic civilization full of agents who are nothing like you, yet find great value and interest in their own goals. And that's fine by me. I'm not so bigoted as you are. Let the Future go its own way, without trying to bind it forever to the laughably primitive prejudices of a pack of four-limbed Squishy Things -"

My friend, I have no problem with the thought of a galactic civilization vastly unlike our own... full of strange beings who look nothing like me even in their own imaginations... pursuing pleasures and experiences I can't begin to empathize with... trading in a marketplace of unimaginable goods... allying to pursue incomprehensible objectives... people whose life-stories I could never understand.

That's what the Future looks like if things go right.

If the chain of inheritance from human (meta)morals is broken, the Future does not look like this. It does not end up magically, delightfully incomprehensible.

With very high probability, it ends up looking dull. Pointless. Something whose loss you wouldn't mourn.

Seeing this as obvious, is what requires that immense amount of background explanation.

And I'm not going to iterate through all the points and winding pathways of argument here, because that would take us back through 75% of my Overcoming Bias posts. Except to remark on how many different things must be known to constrain the final answer.

Consider the incredibly important human value of "boredom" - our desire not to do "the same thing" over and over and over again. You can imagine a mind that contained almost the whole specification of human value, almost all the morals and metamorals, but left out just this one thing -

- and so it spent until the end of time, and until the farthest reaches of its light cone, replaying a single highly optimized experience, over and over and over again.

Or imagine a mind that contained almost the whole specification of which sort of feelings humans most enjoy - but not th...