trentmkelly/LessWrong-43k huggingface dataset:
Heading Toward Morality
by Eliezer Yudkowsky — 2008-06-20T08:08:16.000Z

Followup to: Ghosts in the Machine, Fake Fake Utility Functions, Fake Utility Functions

As people were complaining before about not seeing where the quantum physics sequence was going, I shall go ahead and tell you where I'm heading now.

Having dissolved the confusion surrounding the word "could", the trajectory is now heading toward should.

In fact, I've been heading there for a while. Remember the whole sequence on fake utility functions? Back in... well... November 2007?

I sometimes think of there being a train that goes to the Friendly AI station; but it makes several stops before it gets there; and at each stop, a large fraction of the remaining passengers get off.

One of those stops is the one I spent a month leading up to in November 2007, the sequence chronicled in Fake Fake Utility Functions and concluded in Fake Utility Functions.

That's the stop where someone thinks of the One Great Moral Principle That Is All We Need To Give AIs.

To deliver that one warning, I had to go through all sorts of topics—which topics one might find useful even if not working on Friendly AI. I warned against Affective Death Spirals, which required recursing on the affect heuristic and halo effect, so that your good feeling about one particular moral principle wouldn't spiral out of control. I did that whole sequence on evolution; and discursed on the human ability to make almost any goal appear to support almost any policy; I went into evolutionary psychology to argue for why we shouldn't expect human terminal values to reduce to any simple principle, even happiness, explaining the concept of "expected utility" along the way...

...and talked about genies and more; but you can read the Fake Utility sequence for that.

So that's just the warning against trying to oversimplify human morality into One Great Moral Principle.

If you want to actually dissolve the confusion that surrounds the word "should"—which is the next stop on the train—then that takes a much longer introduction. Not just one November.

I went through the sequence on words and definitions so that I would be able to later say things like "The next project is to Taboo the word 'should' and replace it with its substance", or "Sorry, saying that morality is self-interest 'by definition' isn't going to cut it here".

And also the words-and-definitions sequence was the s...
LA-602 vs. RHIC Review
by Eliezer Yudkowsky — 2008-06-19T10:00:31.000Z

LA-602: Ignition of the Atmosphere with Nuclear Bombs, a research report from the Manhattan Project, is to the best of my knowledge the first technical analysis ever conducted of an uncertain danger of a human-caused extinction catastrophe.

Previously, Teller and Konopinski had been assigned the task of disproving a crazy suggestion by Enrico Fermi that a fission chain reaction could ignite a thermonuclear reaction in deuterium - what we now know as an H-Bomb. Teller and Konopinski found that, contrary to their initial skepticism, the hydrogen bomb appeared possible.

Good for their rationality! Even though they started with the wrong conclusion on their bottom line, they were successfully forced away from it by arguments that could only support one answer.

Still, in retrospect, I think that the advice the future would give to the past, would be: Start by sitting down and saying, "We don't know if a hydrogen bomb is possible". Then list out the evidence and arguments; then at the end weigh it.

So the hydrogen bomb was possible. Teller then suggested that a hydrogen bomb might ignite a self-sustaining thermonuclear reaction in the nitrogen of Earth's atmosphere. This also appeared extremely unlikely at a first glance, but Teller and Konopinski and Marvin investigated, and wrote LA-602...

As I understand LA-602, the authors went through the math and concluded that there were several strong reasons to believe that nitrogen fusion could not be self-sustaining in the atmosphere: it would take huge energies to start the reaction at all; the reaction would lose radiation from its surface too fast to sustain the fusion temperature; and even if the fusion reaction did grow, the Compton effect would increase radiation losses with volume(?).

And we're still here; so the math, whatever it actually says, seems to have been right.

Note that the Manhattan scientists didn't always get their math right. The Castle Bravo nuclear test on March 1, 1954 produced 15 megatons instead of the expected 4-8 megatons due to an unconsidered additional nuclear reaction that took place in lithium-7. The resulting fallout contaminated fishing boats outside the declared danger zone; at least one person seems to have died.

But the LA-602 calculations were done with very conservative assumptions, and came out with plenty of safety margin. AFAICT (I am not a physi...
Ghosts in the Machine
by Eliezer Yudkowsky — 2008-06-17T23:29:17.000Z

People hear about Friendly AI and say - this is one of the top three initial reactions:

"Oh, you can try to tell the AI to be Friendly, but if the AI can modify its own source code, it'll just remove any constraints you try to place on it."

And where does that decision come from?

Does it enter from outside causality, rather than being an effect of a lawful chain of causes which started with the source code as originally written? Is the AI the Author* source of its own free will?

A Friendly AI is not a selfish AI constrained by a special extra conscience module that overrides the AI's natural impulses and tells it what to do. You just build the conscience, and that is the AI. If you have a program that computes which decision the AI should make, you're done. The buck stops immediately.

At this point, I shall take a moment to quote some case studies from the Computer Stupidities site and Programming subtopic. (I am not linking to this, because it is a fearsome time-trap; you can Google if you dare.)

I tutored college students who were taking a computer programming course. A few of them didn't understand that computers are not sentient. More than one person used comments in their Pascal programs to put detailed explanations such as, "Now I need you to put these letters on the screen." I asked one of them what the deal was with those comments. The reply: "How else is the computer going to understand what I want it to do?" Apparently they would assume that since they couldn't make sense of Pascal, neither could the computer.

While in college, I used to tutor in the school's math lab. A student came in because his BASIC program would not run. He was taking a beginner course, and his assignment was to write a program that would calculate the recipe for oatmeal cookies, depending upon the number of people you're baking for. I looked at his program, and it went something like this:

10 Preheat oven to 350
20 Combine all ingredients in a large mixing bowl
30 Mix until smooth

An introductory programming student once asked me to look at his program and figure out why it was always churning out zeroes as the result of a simple computation. I looked at the program, and it was pretty obvious:

begin
read("Number of Apples", apples)
read("Numbe

...
Grasping Slippery Things
by Eliezer Yudkowsky — 2008-06-17T02:04:58.000Z

Followup to: Possibility and Could-ness, The Ultimate Source

Brandon Reinhart wrote:

I am "grunching." Responding to the questions posted without reading your answer. Then I'll read your answer and compare. I started reading your post on Friday and had to leave to attend a wedding before I had finished it, so I had a while to think about my answer.

Brandon, thanks for doing this. You've provided a valuable illustration of natural lines of thought. I hope you won't be offended if, for educational purposes, I dissect it in fine detail. This sort of dissection is a procedure I followed with Marcello to teach thinking about AI, so no malice is intended.

Can you talk about "could" without using synonyms like "can" and "possible"?

When we speak of "could" we speak of the set of realizable worlds [A'] that follows from an initial starting world A operated on by a set of physical laws f.

(Emphases added.)

I didn't list "realizable" explicitly as Tabooed, but it refers to the same concept as "could". Rationalist's Taboo isn't played against a word list, it's played against a concept list. The goal is to force yourself to reduce.

Because follows links two worlds, and the
linkage is exactly what seems confusing, a word like
"follows" is also dangerous.

Think of it as being like trying to pick up something very slippery. You have to prevent it from squeezing out of your hands. You have to prevent the mystery from scurrying away and finding a new dark corner to hide in, as soon as you flip on the lights.

So letting yourself use a word like "realizable", or even "follows", is giving your mind a tremendous opportunity to Pass the Recursive Buck - which anti-pattern, be it noted in fairness to Brandon, I hadn't yet posted on.

If I was doing this on my own, and I didn't know the solution yet, I
would also be marking "initial", "starting", and "operated on". Not necessarily at the highest priority, but just in case they
were hiding the source of the confusion. If I was being even more careful I would mark "physical laws" and "world".

So when we say "I could have turned left at the fork in the road."
"Could" refers to the set of realizable worlds that follow fro

...
Passing the Recursive Buck
by Eliezer Yudkowsky — 2008-06-16T04:50:49.000Z

Followup to: Artificial Addition, The Ultimate Source, Gödel, Escher, Bach: An Eternal Golden Braid

Yesterday, I talked about what happens when you look at your own mind, reflecting upon yourself, and search for the source of your own decisions.

Let's say you decided to run into a burning orphanage and save a young child. You look back on the decision and wonder: was your empathy with children, your ability to imagine what it would be like to be on fire, the decisive factor? Did it compel you to run into the orphanage?

No, you reason, because if you'd needed to prevent a nuclear weapon from going off in the building next door, you would have run to disarm the nuke, and let the orphanage burn. So a burning orphanage is not something that controls you directly. Your fear certainly didn't control you. And as for your duties, it seems like you could have ignored them (if you wanted to).

So if none of these parts of yourself that you focus upon, are of themselves decisive... then there must be some extra and additional thing that is decisive! And that, of course, would be this "you" thing that is looking over your thoughts from outside.

Imagine if human beings had a tiny bit more introspective ability than they have today, so that they could see a single neuron firing - but only one neuron at a time. We might even have the ability to modify the firing of this neuron. It would seem, then, like no individual neuron was in control of us, and that indeed, we had the power to control the neuron. It would seem we were in control of our neurons, not controlled by them. Whenever you look at a single neuron, it seems not to control you, that-which-is-looking...

So it might look like you were moved to run into the orphanage by your built-in empathy or your inculcated morals, and that this overcame your fear of fire. But really there was an additional you, beyond these emotions, which chose to give in to the good emotions rather than the bad ones. That's moral responsibility, innit?

But wait - how does this additional you decide to flow along with your empathy and not your fear? Is it programmed to always be good? Does it roll a die and do whatever the die says?

Ordinarily, this question is not asked. Once you say that you choose of your own "free will", you've explained the choice ...
The Ultimate Source
by Eliezer Yudkowsky — 2008-06-15T09:01:41.000Z

This post is part of the Solution to "Free Will".
Followup to: Timeless Control, Possibility and Could-ness

Faced with a burning orphanage, you ponder your next action for long agonizing moments, uncertain of what you will do. Finally, the thought of a burning child overcomes your fear of fire, and you run into the building and haul out a toddler.

There's a strain of philosophy which says that this scenario is not sufficient for what they call "free will". It's not enough for your thoughts, your agonizing, your fear and your empathy, to finally give rise to a judgment. It's not enough to be the source of your decisions.

No, you have to be the ultimate source of your decisions. If anything else in your past, such as the initial condition of your brain, fully determined your decision, then clearly you did not.

But we already drew this diagram:

As previously discussed, the left-hand structure is preferred, even given deterministic physics, because it is more local; and because it is not possible to compute the Future without computing the Present as an intermediate.

So it is proper to say, "If-counterfactual the past changed and the present remained the same, the future would remain the same," but not to say, "If the past remained the same and the present changed, the future would remain the same."

Are you the true source of your decision to run into the burning orphanage? What if your parents once told you that it was right for people to help one another? What if it were the case that, if your parents hadn't told you so, you wouldn't have run into the burning orphanage? Doesn't that mean that your parents made the decision for you to run into the burning orphanage, rather than you?

On several grounds, no:

If it were counterfactually the case that your parents hadn't raised you to be good, then it would counterfactually be the case that a different person would stand in front of the burning orphanage. It would be a different person who arrived at a different decision. And how can you be anyone other than yourself? Your parents may have helped pluck you out of Platonic person-space to stand in front of the orphanage, but is that the same as controlling the decision of your point in Platonic person-space?

Or: If we imagine that your parents had raised you differently, and yet somehow, exactly the same b...
Possibility and Could-ness
by Eliezer Yudkowsky — 2008-06-14T04:38:37.000Z

This post is part of the Solution to "Free Will".
Followup to: Dissolving the Question, Causality and Moral Responsibility

Planning out upcoming posts, it seems to me that I do, in fact, need to talk about the word could, as in, "But I could have decided not to rescue that toddler from the burning orphanage."

Otherwise, I will set out to talk about Friendly AI, one of these days, and someone will say: "But it's a machine; it can't make choices, because it couldn't have done anything other than what it did."

So let's talk about this word, "could". Can you play Rationalist's Taboo against it? Can you talk about "could" without using synonyms like "can" and "possible"?

Let's talk about this notion of "possibility". I can tell, to some degree, whether a world is actual or not actual; what does it mean for a world to be "possible"?

I know what it means for there to be "three" apples on a table. I can verify that experimentally, I know what state of the world corresponds it. What does it mean to say that there "could" have been four apples, or "could not" have been four apples? Can you tell me what state of the world corresponds to that, and how to verify it? Can you do it without saying "could" or "possible"?

I know what it means for you to rescue a toddler from the orphanage. What does it mean for you to could-have-not done it? Can you describe the corresponding state of the world without "could", "possible", "choose", "free", "will", "decide", "can", "able", or "alternative"?

One last chance to take a stab at it, if you want to work out the answer for yourself...

Some of the first Artificial Intelligence systems ever built, were trivially simple planners. You specify the initial state, and the goal state, and a set of actions that map states onto states; then you search for a series of actions that takes the initial state to the goal state.

Modern AI planners are a hell of a lot more sophisticated than this, but it's amazing how far you can get by understanding the simple math of everything. There are a number of simple, obvious strategies you can use on a problem like this. All of the simple strategies will fail on difficult problems; but you can take a course in AI if you want to talk about that part.

There's backward chaining: Searching back from the goal, to find a tree of states...
Causality and Moral Responsibility
by Eliezer Yudkowsky — 2008-06-13T08:34:44.000Z

Followup to: Thou Art Physics, Timeless Control, Hand vs. Fingers, Explaining vs. Explaining Away

I know (or could readily rediscover) how to build a binary adder from logic gates. If I can figure out how to make individual logic gates from Legos or ant trails or rolling ping-pong balls, then I can add two 32-bit unsigned integers using Legos or ant trails or ping-pong balls.

Someone who had no idea how I'd just done the trick, might accuse me of having created "artificial addition" rather than "real addition".

But once you see the essence, the structure that is addition, then you will automatically see addition whenever you see that structure. Legos, ant trails, or ping-pong balls.

Even if the system is - gasp!- deterministic, you will see a system that, lo and behold, deterministically adds numbers. Even if someone - gasp! - designed the system, you will see that it was designed to add numbers. Even if the system was - gasp!- caused, you will see that it was caused to add numbers.

Let's say that John is standing in front of an orphanage which is on fire, but not quite an inferno yet; trying to decide whether to run in and grab a baby or two. Let us suppose two slightly different versions of John - slightly different initial conditions. They both agonize. They both are torn between fear and duty. Both are tempted to run, and know how guilty they would feel, for the rest of their lives, if they ran. Both feel the call to save the children. And finally, in the end, John-1 runs away, and John-2 runs in and grabs a toddler, getting out moments before the flames consume the entranceway.

This, it seems to me, is the very essence of moral responsibility - in the one case, for a cowardly choice; in the other case, for a heroic one. And I don't see what difference it makes, if John's decision was physically deterministic given his initial conditions, or if John's decision was preplanned by some alien creator that built him out of carbon atoms, or even if - worst of all - there exists some set of understandable psychological factors that were the very substance of John and caused his decision.

Imagine yourself caught in an agonizing moral dilemma. If the burning orphanage doesn't work for you - if you wouldn't feel conflicted about that, one way or the other - then substitute s...
Quantum Mechanics and Personal Identity
by Eliezer Yudkowsky — 2008-06-12T07:13:49.000Z

This is one of several shortened indices into the Quantum Physics Sequence.

Suppose that someone built an exact duplicate of you on Mars, quark by quark - to the maximum level of resolution that quantum physics permits, which is considerably higher resolution than ordinary thermal uncertainty. Would the duplicate be really you, or just a copy?

It may seem unlikely a priori that physics, or any experimental science, could have something to say about this issue.

But it's amazing, the things that science can tell you.

In this case, it turns out, science can rule out a notion of personal identity that depends on your being composed of the same atoms - because modern physics has taken the concept of "same atom" and thrown it out the window. There are no tiny billiard balls with individual identities. It's experimentally ruled out.

"Huh? What do you mean, physics has gotten rid of the concept of 'same atom'?"

No one can be told this, alas, because it involves replacing the concept of little billiard balls with a different kind of math. If you read through the introduction that follows to basic quantum mechanics, you will be able to see that the naive concept of personal identity - the notion that you are made up of tiny pieces with individual identities that persist through time, and that your identity follows the "same" tiny pieces - is physical nonsense. The universe just doesn't work in a way which would let that idea be meaningful.

There are more abstract and philosophical arguments that you could use to rule out atom-following theories of personal identity. But in our case, it so happens that we live in a universe where the issue is flatly settled by standard physics. It's like proposing that personal identity follows phlogiston. You could argue against it on philosophical grounds - but we happen to live in a universe where "phlogiston" itself is just a mistaken theory to be discarded, which settles the issue much more abruptly.

And no, this does not rely on a woo-woo mysterian interpretation of quantum mechanics. The other purpose of this series of posts, was to demystify quantum mechanics and reveal it as non-mysterious. It just happens to be a fact that once you get to the non-mysterious version of quantum mechanics, you find that the reason why physics once looked mysterious, has to do with reality...
And the Winner is... Many-Worlds!
by Eliezer Yudkowsky — 2008-06-12T06:05:33.000Z

This is one of several shortened indices into the Quantum Physics Sequence.

Macroscopic quantum superpositions, a.k.a. the "many-worlds interpretation" or MWI, was proposed in 1957 and brought to the general attention of the scientific community in 1970. Ever since, MWI has steadily gained in popularity. As of 2008, MWI may or may not be endorsed by a majority of theoretical physicists (attempted opinion polls conflict on this point). Of course, Science is not supposed to be an opinion poll, but anyone who tells you that MWI is "science fiction" is simply ignorant.

When a theory is slowly persuading scientists despite all academic inertia, and more and more graduate students grow up familiar with it, at what point should one go ahead and declare a temporary winner pending new evidence?

Reading through the referenced posts will give you a very basic introduction to quantum mechanics - algebra is involved, but no calculus - by which you may nonetheless gain an understanding sufficient to see, and not just be told, that the modern case for many-worlds has become overwhelming. Not just plausible, not just strong, but overwhelming. Single-world versions of quantum mechanics just don't work, and all the legendary confusingness and mysteriousness of quantum mechanics stems from this essential fact. But enough telling - let me show you.

Quantum Explanations:
Quantum mechanics doesn't deserve its fearsome reputation. If you tell
people something is supposed to be mysterious, they won't understand
it. It's human intuitions that are "strange" or "weird"; physics
itself is perfectly normal. Talking about historical erroneous
concepts like "particles" or "waves" is just asking to confuse people;
present the real, unified quantum physics straight out. The series
will take a strictly realist perspective - quantum equations describe
something that is real and out there.

Configurations and Amplitude:
A preliminary glimpse at the stuff reality is made of. The classic
split-photon experiment with half-silvered mirrors. Alternative
pathways the photon can take, can cancel each other out. The
mysterious measuring tool that tells us the relative squared moduli.

Joint Configurations:
The laws of physics are inherently over mathematical entities

...