trentmkelly/LessWrong-43k huggingface dataset:
Nonsentient Optimizers
by Eliezer Yudkowsky — 2008-12-27T02:32:23.000Z

Followup to: Nonperson Predicates, Possibility and Could-ness

"All our ships are sentient. You could certainly try telling a ship what to do... but I don't think you'd get very far."
"Your ships think they're sentient!" Hamin chuckled.
"A common delusion shared by some of our human citizens."
—Player of Games, Iain M. Banks

Yesterday, I suggested that, when an AI is trying to build a model of an environment that includes human beings, we want to avoid the AI constructing detailed models that are themselves people. And that, to this end, we would like to know what is or isn't a person—or at least have a predicate that returns 1 for all people and could return 0 or 1 for anything that isn't a person, so that, if the predicate returns 0, we know we have a definite nonperson on our hands.

And as long as you're going to solve that problem anyway, why not apply the same knowledge to create a Very Powerful Optimization Process which is also definitely not a person?

"What? That's impossible!"

How do you know? Have you solved the sacred mysteries of consciousness and existence?

"Um—okay, look, putting aside the obvious objection that any sufficiently powerful intelligence will be able to model itself—"

Lob's Sentence contains an exact recipe for a copy of itself, including the recipe for the recipe; it has a perfect self-model. Does that make it sentient?

"Putting that aside—to create a powerful AI and make it not sentient—I mean, why would you want to?"

Several reasons. Picking the simplest to explain first—I'm not ready to be a father.

Creating a true child is the only moral and metaethical problem I know that is even harder than the shape of a Friendly AI. I would like to be able to create Friendly AI while worrying just about the Friendly AI problems, and not worrying whether I've created someone who will lead a life worth living. Better by far to just create a Very Powerful Optimization Process, if at all possible.

"Well, you can't have everything, and this thing sounds distinctly alarming even if you could -"

Look, suppose that someone said—in fact, I have heard it said—that Friendly AI is impossible, because you can't have an intelligence without free will.

"In light of the dissolved confusion about free will, ...
Nonperson Predicates
by Eliezer Yudkowsky — 2008-12-27T01:47:32.000Z

Followup to: Righting a Wrong Question, Zombies! Zombies?, A Premature Word on AI, On Doing the Impossible

There is a subproblem of Friendly AI which is so scary that I usually don't talk about it, because very few would-be AI designers would react to it appropriately—that is, by saying, "Wow, that does sound like an interesting problem", instead of finding one of many subtle ways to scream and run away.

This is the problem that if you create an AI and tell it to model the world around it, it may form models of people that are people themselves. Not necessarily the same person, but people nonetheless.

If you look up at the night sky, and see the tiny dots of light that move over days and weeks—planētoi, the Greeks called them, "wanderers"—and you try to predict the movements of those planet-dots as best you can...

Historically, humans went through a journey as long and as wandering as the planets themselves, to find an accurate model. In the beginning, the models were things of cycles and epicycles, not much resembling the true Solar System.

But eventually we found laws of gravity, and finally built models—even if they were just on paper—that were extremely accurate so that Neptune could be deduced by looking at the unexplained perturbation of Uranus from its expected orbit. This required moment-by-moment modeling of where a simplified version of Uranus would be, and the other known planets. Simulation, not just abstraction. Prediction through simplified-yet-still-detailed pointwise similarity.

Suppose you have an AI that is around human beings. And like any Bayesian trying to explain its enivornment, the AI goes in quest of highly accurate models that predict what it sees of humans.

Models that predict/explain why people do the things they do, say the things they say, want the things they want, think the things they think, and even why people talk about "the mystery of subjective experience".

The model that most precisely predicts these facts, may well be a 'simulation' detailed enough to be a person in its own right.

A highly detailed model of me, may not be me. But it will, at least, be a model which (for purposes of prediction via similarity) thinks itself to be Eliezer Yudkowsky. It will be a model that, when cranked to find my behavior if asked "Who are you and are you conscious?", says "I am Elie...
Devil's Offers
by Eliezer Yudkowsky — 2008-12-25T17:00:00.000Z

An iota of fictional evidence from The Golden Age by John C. Wright:

Helion had leaned and said, "Son, once you go in there, the full powers and total command structures of the Rhadamanth Sophotech will be at your command. You will be invested with godlike powers; but you will still have the passions and distempers of a merely human spirit. There are two temptations which will threaten you. First, you will be tempted to remove your human weaknesses by abrupt mental surgery. The Invariants do this, and to a lesser degree, so do the White Manorials, abandoning humanity to escape from pain. Second, you will be tempted to indulge your human weakness. The Cacophiles do this, and to a lesser degree, so do the Black Manorials. Our society will gladly feed every sin and vice and impulse you might have; and then stand by helplessly and watch as you destroy yourself; because the first law of the Golden Oecumene is that no peaceful activity is forbidden. Free men may freely harm themselves, provided only that it is only themselves that they harm."
Phaethon knew what his sire was intimating, but he did not let himself feel irritated. Not today. Today was the day of his majority, his emancipation; today, he could forgive even Helion's incessant, nagging fears.
Phaethon also knew that most Rhadamanthines were not permitted to face the Noetic tests until they were octogenerians; most did not pass on their first attempt, or even their second. Many folk were not trusted with the full powers of an adult until they reached their Centennial. Helion, despite criticism from the other Silver-Gray branches, was permitting Phaethon to face the tests five years early...

Then Phaethon said, "It's a paradox, Father. I cannot be, at the same time and in the same sense, a child and an adult. And, if I am an adult, I cannot be, at the same time, free to make my own successes, but not free to make my own mistakes."
Helion looked sardonic. "'Mistake' is such a simple word. An adult who suffers a moment of foolishness or anger, one rash moment, has time enough to delete or destroy his own free will, memory, or judgment. No one is allowed to force a cure on him. No one can restore his sanity

...
Harmful Options
by Eliezer Yudkowsky — 2008-12-25T02:26:22.000Z

Barry Schwartz's The Paradox of Choice—which I haven't read, though I've read some of the research behind it—talks about how offering people more choices can make them less happy.

A simple intuition says this shouldn't ought to happen to rational agents: If your current choice is X, and you're offered an alternative Y that's worse than X, and you know it, you can always just go on doing X. So a rational agent shouldn't do worse by having more options. The more available actions you have, the more powerful you become—that's how it should ought to work.

For example, if an ideal rational agent is initially forced to take only box B in Newcomb's Problem, and is then offered the additional choice of taking both boxes A and B, the rational agent shouldn't regret having more options. Such regret indicates that you're "fighting your own ritual of cognition" which helplessly selects the worse choice once it's offered you.

But this intuition only governs extremely idealized rationalists, or rationalists in extremely idealized situations. Bounded rationalists can easily do worse with strictly more options, because they burn computing operations to evaluate them. You could write an invincible chess program in one line of Python if its only legal move were the winning one.

Of course Schwartz and co. are not talking about anything so pure and innocent as the computing cost of having more choices.

If you're dealing, not with an ideal rationalist, not with a bounded rationalist, but with a human being—

Say, would you like to finish reading this post, or watch this surprising video instead?

Schwartz, I believe, talks primarily about the decrease in happiness and satisfaction that results from having more mutually exclusive options. Before this research was done, it was already known that people are more sensitive to losses than to gains, generally by a factor of between 2 and 2.5 (in various different experimental scenarios). That is, the pain of losing something is between 2 and 2.5 times as worse as the joy of gaining it. (This is an interesting constant in its own right, and may have something to do with compensating for our systematic overconfidence.)

So—if you can only choose one dessert, you're likely to be happier choosing from a menu of two than a menu of fourteen. In the first case, you eat one dessert and pass up ...
Imaginary Positions
by Eliezer Yudkowsky — 2008-12-23T17:35:40.000Z

Every now and then, one reads an article about the Singularity in
which some reporter confidently asserts, "The Singularitarians,
followers of Ray Kurzweil, believe that they will be uploaded into
techno-heaven while the unbelievers languish behind or are extinguished
by the machines."

I don't think I've ever met a single Singularity fan, Kurzweilian or otherwise, who
thinks that only believers in the Singularity will go to upload heaven
and everyone else will be left to rot. Not one.
(There's a very few pseudo-Randian types who believe that only the truly selfish who accumulate lots of money will make it, but they expect e.g. me to be damned with
the rest.)

But if you start out thinking that the Singularity is a loony religious meme, then it seems like Singularity believers ought to believe that they alone will be saved. It seems like a detail that would fit the story.

This fittingness is so strong as to manufacture the conclusion without any particular observations. And then the conclusion isn't marked as a deduction. The reporter just thinks that they investigated the Singularity, and found some loony cultists who believe they alone will be saved.

Or so I deduce. I haven't actually observed the inside of their minds, after all.

Has any rationalist ever advocated behaving as if all
people are reasonable and fair? I've repeatedly heard people say,
"Well, it's not always smart to be rational, because other people
aren't always reasonable." What rationalist said they were? I would deduce: This is something that non-rationalists believe it would "fit" for us to believe, given our general blind faith in Reason. And
so their minds just add it to the knowledge pool, as though it were an observation. (In this case I encountered yet another example recently enough to find the reference; see here.)

(Disclaimer: Many things
have been said, at one time or another, by one person or another, over
centuries of recorded history; and the topic of "rationality" is popularly enough discussed that some self-identified "rationalist" may have described "rationality" that way at one point or another. But I have yet to hear a rationalist say it, myself.)

I once read an article on Extropians (a cert...
Rationality Quotes 20
by Eliezer Yudkowsky — 2008-12-22T22:05:43.000Z

"For every stock you buy, there is someone selling you that stock. What is it that you know that they don't? What is it that they know, that you don't? Who has the edge? If it's not you, chances are you are going to lose money on the deal."
-- Mark Cuban

"If you have two choices, choose the harder. If you're trying to decide whether to go out running or sit home and watch TV, go running. Probably the reason this trick works so well is that when you have two choices and one is harder, the only reason you're even considering the other is laziness. You know in the back of your mind what's the right thing to do, and this trick merely forces you to acknowledge it."
-- Paul Graham

"Never attribute to malice that which can be adequately explained by stupidity."
-- Hanlon's Razor

"I divide my officers into four classes; the clever, the lazy, the industrious, and the stupid. Each officer possesses at least two of these qualities. Those who are clever and industrious are fitted for the highest staff appointments. Use can be made of those who are stupid and lazy. The man who is clever and lazy however is for the very highest command; he has the temperament and nerves to deal with all situations. But whoever is stupid and industrious is a menace and must be removed immediately!"
-- General Kurt von Hammerstein-Equord

"There's no such thing as a human who doesn't commit sin. It's not like the world is divided into sinners and the innocent. There are only people who can and who cannot atone for their sins."
-- Ciel

"Simple stupidity is never enough. People need to pile stupidity on stupidity on stupidity."
-- Mark C. Chu-Carroll
Living By Your Own Strength
by Eliezer Yudkowsky — 2008-12-22T00:37:52.000Z

Followup to: Truly Part of You

"Myself, and Morisato-san... we want to live together by our own strength."

Jared Diamond once called agriculture "the worst mistake in the history of the human race". Farmers could grow more wheat than hunter-gatherers could collect nuts, but the evidence seems pretty conclusive that agriculture traded quality of life for quantity of life. One study showed that the farmers in an area were six inches shorter and seven years shorter-lived than their hunter-gatherer predecessors—even though the farmers were more numerous.

I don't know if I'd call agriculture a mistake. But one should at least be aware of the downsides. Policy debates should not appear one-sided.

In the same spirit—

Once upon a time, our hunter-gatherer ancestors strung their own bows, wove their own baskets, whittled their own flutes.

And part of our alienation from that environment of evolutionary adaptedness, is the number of tools we use that we don't understand and couldn't make for ourselves.

You can look back on Overcoming Bias, and see that I've always been suspicious of borrowed strength. (Even before I understood the source of Robin's and my disagreement about the Singularity, that is.) In Guessing the Teacher's Password I talked about the (well-known) problem in which schools end up teaching verbal behavior rather than real knowledge. In Truly Part of You I suggested one test for false knowledge: Imagine deleting a fact from your mind, and ask if it would grow back.

I know many ways to prove the Pythagorean Theorem, including at least one proof that is purely visual and can be seen at a glance. But if you deleted the Pythagorean Theorem from my mind entirely, would I have enough math skills left to grow it back the next time I needed it? I hope so—certainly I've solved math problems that seem tougher than that, what with benefit of hindsight and all. But, as I'm not an AI, I can't actually switch off the memories and associations, and test myself in that way.

Wielding someone else's strength to do things beyond your own understanding—that really is as dangerous as the Deeply Wise phrasing makes it sound.

I observed in Failing to Learn from History (musing on my childhood foolishness in offering a mysterious answer to a mysterious question): "If only I had personally postulated astrologi...
Sensual Experience
by Eliezer Yudkowsky — 2008-12-21T00:56:31.000Z

Modern day gamemakers are constantly working on higher-resolution, more realistic graphics; more immersive sounds—but they're a long long way off real life.

Pressing the "W" key to run forward as a graphic of a hungry tiger bounds behind you, just doesn't seem quite as sensual as running frantically across the savanna with your own legs, breathing in huge gasps and pumping your arms as the sun beats down on your shoulders, the grass brushes your shins, and the air whips around you with the wind of your passage.

Don't mistake me for a luddite; I'm not saying the technology can't get that good. I'm saying it hasn't gotten that good yet.

Failing to escape the computer tiger would also have fewer long-term consequences than failing to escape a biological tiger—it would be less a part of the total story of your life—meaning you're also likely to be less emotionally involved. But that's a topic for another post. Today's post is just about the sensual quality of the experience.

Sensual experience isn't a question of some mysterious quality that only the "real world" possesses. A computer screen is as real as a tiger, after all. Whatever is, is real.

But the pattern of the pseudo-tiger, inside the computer chip, is nowhere near as complex as a biological tiger; it offers far fewer modes in which to interact. And the sensory bandwidth between you and the computer's pseudo-world is relatively low; and the information passing along it isn't in quite the right format.

It's not a question of computer tigers being "virtual" or "simulated", and therefore somehow a separate magisterium. But with present technology, and the way your brain is presently set up, you'd have a lot more neurons involved in running away from a biological tiger.

Running would fill your whole vision with motion, not just a flat rectangular screen—which translates into more square centimeters of visual cortex getting actively engaged.

The graphics on a computer monitor try to trigger your sense of spatial motion (residing in the parietal cortex, btw). But they're presenting the information differently from its native format —without binocular vision, for example, and without your vestibular senses indicating true motion. So the sense of motion isn't likely to be quite the same, what it would be if you were running.

And there's the sense of touch that indicates th...
Complex Novelty
by Eliezer Yudkowsky — 2008-12-20T00:31:51.000Z

From Greg Egan's Permutation City:

The workshop abutted a warehouse full of table legs—one hundred and sixty-two thousand, three hundred and twenty-nine, so far. Peer could imagine nothing more satisfying than reaching the two hundred thousand mark—although he knew it was likely that he'd change his mind and abandon the workshop before that happened; new vocations were imposed by his exoself at random intervals, but statistically, the next one was overdue. Immediately before taking up woodwork, he'd passionately devoured all the higher mathematics texts in the central library, run all the tutorial software, and then personally contributed several important new results to group theory—untroubled by the fact that none of the Elysian mathematicians would ever be aware of his work. Before that, he'd written over three hundred comic operas, with librettos in Italian, French and English—and staged most of them, with puppet performers and audience. Before that, he'd patiently studied the structure and biochemistry of the human brain for sixty-seven years; towards the end he had fully grasped, to his own satisfaction, the nature of the process of consciousness. Every one of these pursuits had been utterly engrossing, and satisfying, at the time. He'd even been interested in the Elysians, once.
No longer. He preferred to think about table legs.

Among science fiction authors, (early) Greg Egan is my favorite; of early-Greg-Egan's books, Permutation City is my favorite; and this particular passage in Permutation City, more than any of the others, I find utterly horrifying.

If this were all the hope the future held, I don't know if I could bring myself to try. Small wonder that people don't sign up for cryonics, if even SF writers think this is the best we can do.

You could think of this whole series on Fun Theory as my reply to Greg Egan—a list of the ways that his human-level uploaded civilizations Fail At Fun. (And yes, this series will also explain what's wrong with the Culture and how to fix it.)

We won't get to all of Peer's problems today—but really. Table legs?

I could see myself carving one table leg, maybe, if there was something non-obvious to learn from the experience. But not 162,329.

In Permutation City, Peer modified himself to find table-leg-carving fascinatin...
High Challenge
by Eliezer Yudkowsky — 2008-12-19T00:51:08.000Z

There's a class of prophecy that runs: "In the Future, machines will do all the work. Everything will be automated. Even labor of the sort we now consider 'intellectual', like engineering, will be done by machines. We can sit back and own the capital. You'll never have to lift a finger, ever again."

But then won't people be bored?

No; they can play computer games—not like our games, of course, but much more advanced and entertaining.

Yet wait! If you buy a modern computer game, you'll find that it contains some tasks that are—there's no kind word for this—effortful. (I would even say "difficult", with the understanding that we're talking about something that takes 10 minutes, not 10 years.)

So in the future, we'll have programs that help you play the game—taking over if you get stuck on the game, or just bored; or so that you can play games that would otherwise be too advanced for you.

But isn't there some wasted effort, here? Why have one programmer working to make the game harder, and another programmer to working to make the game easier? Why not just make the game easier to start with? Since you play the game to get gold and experience points, making the game easier will let you get more gold per unit time: the game will become more fun.

So this is the ultimate end of the prophecy of technological progress—just staring at a screen that says "YOU WIN", forever.

And maybe we'll build a robot that does that, too.

Then what?

The world of machines that do all the work—well, I don't want to say it's "analogous to the Christian Heaven" because it isn't supernatural; it's something that could in principle be realized. Religious analogies are far too easily tossed around as accusations... But, without implying any other similarities, I'll say that it seems analogous in the sense that eternal laziness "sounds like good news" to your present self who still has to work.

And as for playing games, as a substitute—what is a computer game except synthetic work? Isn't there a wasted step here? (And computer games in their present form, considered as work, have various aspects that reduce stress and increase engagement; but they also carry costs in the form of artificiality and isolation.)

I sometimes think that futuristic ideals phrased in terms of "getting rid of work" would be better reformulated a...