trentmkelly/LessWrong-43k huggingface dataset:
Detached Lever Fallacy
by Eliezer Yudkowsky — 2008-07-31T18:57:12.000Z

This fallacy gets its name from an ancient sci-fi TV show, which I never saw myself, but was reported to me by a reputable source (some guy at an SF convention). Anyone knows the exact reference, do leave a comment.

So the good guys are battling the evil aliens. Occasionally, the good guys have to fly through an asteroid belt. As we all know, asteroid belts are as crowded as a New York parking lot, so their ship has to carefully dodge the asteroids. The evil aliens, though, can fly right through the asteroid belt because they have amazing technology that dematerializes their ships, and lets them pass through the asteroids.

Eventually, the good guys capture an evil alien ship, and go exploring inside it. The captain of the good guys finds the alien bridge, and on the bridge is a lever. "Ah," says the captain, "this must be the lever that makes the ship dematerialize!" So he pries up the control lever and carries it back to his ship, after which his ship can also dematerialize.

Similarly, to this day, it is still quite popular to try to program an AI with "semantic networks" that look something like this:

(apple is-a fruit)
(fruit is-a food)
(fruit is-a plant)

You've seen apples, touched apples, picked them up and held them, bought them for money, cut them into slices, eaten the slices and tasted them. Though we know a good deal about the first stages of visual processing, last time I checked, it wasn't precisely known how the temporal cortex stores and associates the generalized image of an apple - so that we can recognize a new apple from a different angle, or with many slight variations of shape and color and texture. Your motor cortex and cerebellum store programs for using the apple.

You can pull the lever on another human's strongly similar version of all that complex machinery, by writing out "apple", five ASCII characters on a webpage.

But if that machinery isn't there - if you're writing "apple" inside a so-called AI's so-called knowledge base - then the text is just a lever.

This isn't to say that no mere machine of silicon can ever have the same internal machinery that humans do, for handling apples and a hundred thousand other concepts. If mere machinery of carbon can do it, then I am reasonably confident that mere machinery of silicon can do it too. If the aliens can dematerialize their s...
Humans in Funny Suits
by Eliezer Yudkowsky — 2008-07-30T23:54:41.000Z

Many times the human species has travelled into space, only to find
the stars inhabited by aliens who look remarkably like humans in funny suits - or
even humans with a touch of makeup and latex - or just beige Caucasians
in fee simple.

It's remarkable how the human form is the natural
baseline of the universe, from which all other alien species are
derived via a few modifications.

What could possibly explain this fascinating phenomenon? Convergent evolution,
of course! Even though these alien lifeforms evolved on a thousand
alien planets, completely independently from Earthly life, they all
turned out the same.

Don't be fooled by the fact that a kangaroo (a mammal) resembles us
rather less than does a chimp (a primate), nor by the fact that a frog
(amphibians, like us, are tetrapods) resembles us less than the
kangaroo. Don't be fooled by the bewildering variety of the insects,
who split off from us even longer ago than the frogs; don't be fooled
that insects have six legs, and their skeletons on the outside, and a
different system of optics, and rather different sexual practices.

You might think that a truly alien species would be more different
from us than we are from insects - that the aliens wouldn't run on DNA,
and might not be made of folded-up hydrocarbon chains internally bound
by van der Waals forces (aka proteins).

As I said, don't be fooled. For an alien species to evolve intelligence, it must have two legs with one knee each attached to an upright torso, and must walk in a way similar to us. You see, any intelligence
needs hands, so you've got to repurpose a pair of legs for that - and
if you don't start with a four-legged being, it can't develop a running
gait and walk upright, freeing the hands.

For an alien species to evolve intelligence it needs
binocular
vision for precise manipulation, which means exactly two eyes. These
eyes must be located in a head atop a torso. The alien must communicate by transcoding
their thoughts into acoustic vibrations, so they need ears and lips and
a throat. And think of how out-of-place ears and eyes and lips would look, without a
nose! Sexual selection will result in the creation of noses - you
wouldn't want to mate with something without a face, would you? A
similar logic explains why the female of the species is invariably attractive - ugly aliens wou

...
Interpersonal Morality
by Eliezer Yudkowsky — 2008-07-29T18:01:51.000Z

Followup to: The Bedrock of Fairness

Every time I wonder if I really need to do so much prep work to explain an idea, I manage to forget some minor thing and a dozen people promptly post objections.

In this case, I seem to have forgotten to cover the topic of how morality applies to more than one person at a time.

Stop laughing, it's not quite as dumb an oversight as it sounds. Sort of like how some people argue that macroeconomics should be constructed from microeconomics, I tend to see interpersonal morality as constructed from personal morality. (And definitely not the other way around!)

In "The Bedrock of Fairness" I offered a situation where three people discover a pie, and one of them insists that they want half. This is actually toned down from an older dialogue where five people discover a pie, and one of them—regardless of any argument offered—insists that they want the whole pie.

Let's consider the latter situation: Dennis wants the whole pie. Not only that, Dennis says that it is "fair" for him to get the whole pie, and that the "right" way to resolve this group disagreement is for him to get the whole pie; and he goes on saying this no matter what arguments are offered him.

This group is not going to agree, no matter what. But I would, nonetheless, say that the right thing to do, the fair thing to do, is to give Dennis one-fifth of the pie—the other four combining to hold him off by force, if necessary, if he tries to take more.

A terminological note:

In this series of posts I have been using "morality" to mean something more like "the sum of all values and valuation rules", not just "values that apply to interactions between people".

The ordinary usage would have that jumping on a trampoline is not "morality", it is just some selfish fun. On the other hand, giving someone else a turn to jump on the trampoline, is more akin to "morality" in common usage; and if you say "Everyone should take turns!" that's definitely "morality".

But the thing-I-want-to-talk-about includes the Fun Theory of a single person jumping on a trampoline.

Think of what a disaster it would be if all fun were removed from human civilization! So I consider it quite right to jump on a trampoline. Even if one would not say, in ordinary conversation, "I am jumping on that trampoline because I have a moral oblig

...
The Meaning of Right
by Eliezer Yudkowsky — 2008-07-29T01:28:03.000Z

Continuation of: Changing Your Metaethics, Setting Up Metaethics
Followup to: Does Your Morality Care What You Think?, The Moral Void, Probability is Subjectively Objective, Could Anything Be Right?, The Gift We Give To Tomorrow, Rebelling Within Nature, Where Recursive Justification Hits Bottom, ...

(The culmination of a long series of Overcoming Bias posts; if you start here, I accept no responsibility for any resulting confusion, misunderstanding, or unnecessary angst.)

What is morality? What does the word "should", mean? The many pieces are in place: This question I shall now dissolve.

The key—as it has always been, in my experience so far—is to understand how a certain cognitive algorithm feels from inside. Standard procedure for righting a wrong question: If you don't know what right-ness is, then take a step beneath and ask how your brain labels things "right".

It is not the same question—it has no moral aspects to it, being strictly a matter of fact and cognitive science. But it is an illuminating question. Once we know how our brain labels things "right", perhaps we shall find it easier, afterward, to ask what is really and truly right.

But with that said—the easiest way to begin investigating that question, will be to jump back up to the level of morality and ask what seems right. And if that seems like too much recursion, get used to it—the other 90% of the work lies in handling recursion properly.

(Should you find your grasp on meaningfulness wavering, at any time following, check Changing Your Metaethics for the appropriate prophylactic.)

So! In order to investigate how the brain labels things "right", we are going to start out by talking about what is right. That is, we'll start out wearing our morality-goggles, in which we consider morality-as-morality and talk about moral questions directly. As opposed to wearing our reduction-goggles, in which we talk about cognitive algorithms and mere physics. Rigorously distinguishing between these two views is the first step toward mating them together.

As a first step, I offer this observation, on the level of morality-as-morality: Rightness is contagious backward in time.

Suppose there is a switch, currently set to OFF, and it is morally desirable for this switch to be flipped to ON. Perh...
Setting Up Metaethics
by Eliezer Yudkowsky — 2008-07-28T02:25:20.000Z

Followup to: Is Morality Given?, Is Morality Preference?, Moral Complexities, Could Anything Be Right?, The Bedrock of Fairness, ...

Intuitions about morality seem to split up into two broad camps: morality-as-given and morality-as-preference.

Some perceive morality as a fixed given, independent of our whims, about which we form changeable beliefs. This view's great advantage is that it seems more normal up at the level of everyday moral conversations: it is the intuition underlying our everyday notions of "moral error", "moral progress", "moral argument", or "just because you want to murder someone doesn't make it right".

Others choose to describe morality as a preference—as a desire in some particular person; nowhere else is it written. This view's great advantage is that it has an easier time living with reductionism—fitting the notion of "morality" into a universe of mere physics. It has an easier time at the meta level, answering questions like "What is morality?" and "Where does morality come from?"

Both intuitions must contend with seemingly impossible questions. For example, Moore's Open Question: Even if you come up with some simple answer that fits on T-Shirt, like "Happiness is the sum total of goodness!", you would need to argue the identity. It isn't instantly obvious to everyone that goodness is happiness, which seems to indicate that happiness and rightness were different concepts to start with. What was that second concept, then, originally?

Or if "Morality is mere preference!" then why care about human preferences? How is it possible to establish any "ought" at all, in a universe seemingly of mere "is"?

So what we should want, ideally, is a metaethic that:

Adds up to moral normality, including moral errors, moral progress, and things you should do whether you want to or not;

Fits naturally into a non-mysterious universe, postulating no exception to reductionism;

Does not oversimplify humanity's complicated moral arguments and many terminal values;

Answers all the impossible questions.

I'll present that view tomorrow.

Today's post is devoted to setting up the question.

Consider "free will", already dealt with in these posts. On one level of organization, we have mere physics, particles that make no choices. On another level of organization, we have human minds that extrap...
Changing Your Metaethics
by Eliezer Yudkowsky — 2008-07-27T12:36:12.000Z

If you say, "Killing people is wrong," that's morality. If you say, "You shouldn't kill people because God prohibited it," or "You shouldn't kill people because it goes against the trend of the universe", that's metaethics.

Just as there's far more agreement on Special Relativity than there is on the question "What is science?", people find it much easier to agree "Murder is bad" than to agree what makes it bad, or what it means for something to be bad.

People do get attached to their metaethics. Indeed they frequently insist that if their metaethic is wrong, all morality necessarily falls apart. It might be interesting to set up a panel of metaethicists—theists, Objectivists, Platonists, etc.—all of whom agree that killing is wrong; all of whom disagree on what it means for a thing to be "wrong"; and all of whom insist that if their metaethic is untrue, then morality falls apart.

Clearly a good number of people, if they are to make philosophical progress, will need to shift metathics at some point in their lives. You may have to do it.

At that point, it might be useful to have an open line of retreat—not a retreat from morality, but a retreat from Your-Current-Metaethic. (You know, the one that, if it is not true, leaves no possible basis for not killing people.)

And so I've been setting up these lines of retreat, in many and various posts, summarized below. For I have learned that to change metaethical beliefs is nigh-impossible in the presence of an unanswered attachment.

If, for example, someone believes the authority of "Thou Shalt Not Kill" derives from God, then there are several and well-known things to say that can help set up a line of retreat—as opposed to immediately attacking the plausibility of God. You can say, "Take personal responsibility! Even if you got orders from God, it would be your own decision to obey those orders. Even if God didn't order you to be moral, you could just be moral anyway."

The above argument actually generalizes to quite a number of metaethics—you just substitute Their-Favorite-Source-Of-Morality, or even the word "morality", for "God". Even if your particular source of moral authority failed, couldn't you just drag the child off the train tracks anyway? And indeed, who is it but you, that ever decided to follow this source of moral aut...
Does Your Morality Care What You Think?
by Eliezer Yudkowsky — 2008-07-26T00:25:34.000Z

Followup to: Math is Subjunctively Objective, The Moral Void, Is Morality Given?

Thus I recall the study, though I cannot recall the citation:

Children, at some relatively young age, were found to distinguish between:

The teacher, by saying that we're allowed to stand on our desks, can make it right to do so.

The teacher, by saying that I'm allowed to take something from another child's backpack, cannot make it right to do so.

Obert: "Well, I don't know the citation, but it sounds like a fascinating study. So even children, then, realize that moral facts are givens, beyond the ability of teachers or parents to alter."

Subhan: "You say that like it's a good thing. Children may also think that people in Australia have to wear heavy boots from falling off the other side of the Earth."

Obert: "Call me Peter Pan, then, because I never grew up on this one. Of course it doesn't matter what the teacher says. It doesn't matter what I say. It doesn't even matter what I think. Stealing is wrong. Do you disagree?"

Subhan: "You don't see me picking your pockets, do you? Isn't it enough that I choose not to steal from you—do I have to pretend it's the law of the universe?"

Obert: "Yes, or I can't trust your commitment."

Subhan: "A... revealing remark. But really, I don't think that this experimental result seems at all confusing, in light of the recent discussion of subjunctive objectivity—a discussion in which Eliezer strongly supported my position, by the way."

Obert: "Really? I thought Eliezer was finally coming out in favor of my position."

Subhan: "Huh? How do you get that?"

Obert: "The whole subtext of 'Math is Subjunctively Objective' is that morality is just like math! Sure, we compute morality inside our own brains—where else would we compute it? But just because we compute a quantity inside our own brains, doesn't mean that what is computed has a dependency on our own state of mind."

Subhan: "I think we must have been reading different Overcoming Bias posts! The whole subtext of 'Math is Subjunctively Objective' is to explain away why morality seems objective—to show that the feeling of a fixed given can arise without any external referent. When you imagine yourself thinking that killing is ...
Math is Subjunctively Objective
by Eliezer Yudkowsky — 2008-07-25T11:06:09.000Z

Followup to: Probability is Subjectively Objective, Can Counterfactuals Be True?

I am quite confident that the statement 2 + 3 = 5 is true; I am far less confident of what it means for a mathematical statement to be true.

In "The Simple Truth" I defined a pebble-and-bucket system for tracking sheep, and defined a condition for whether a bucket's pebble level is "true" in terms of the sheep. The bucket is the belief, the sheep are the reality. I believe 2 + 3 = 5. Not just that two sheep plus three sheep equal five sheep, but that 2 + 3 = 5. That is my belief, but where is the reality?

So now the one comes to me and says: "Yes, two sheep plus three sheep equals five sheep, and two stars plus three stars equals five stars. I won't deny that. But this notion that 2 + 3 = 5, exists only in your imagination, and is purely subjective."

So I say: Excuse me, what?

And the one says: "Well, I know what it means to observe two sheep and three sheep leave the fold, and five sheep come back. I know what it means to press '2' and '+' and '3' on a calculator, and see the screen flash '5'. I even know what it means to ask someone 'What is two plus three?' and hear them say 'Five.' But you insist that there is some fact beyond this. You insist that 2 + 3 = 5."

Well, it kinda is.

"Perhaps you just mean that when you mentally visualize adding two dots and three dots, you end up visualizing five dots. Perhaps this is the content of what you mean by saying, 2 + 3 = 5. I have no trouble with that, for brains are as real as sheep."

No, for it seems to me that 2 + 3 equaled 5 before there were any humans around to do addition. When humans showed up on the scene, they did not make 2 + 3 equal 5 by virtue of thinking it. Rather, they thought that '2 + 3 = 5' because 2 + 3 did in fact equal 5.

"Prove it."

I'd love to, but I'm busy; I've got to, um, eat a salad.

"The reason you believe that 2 + 3 = 5, is your mental visualization of two dots plus three dots yielding five dots. Does this not imply that this physical event in your physical brain is the meaning of the statement '2 + 3 = 5'?"

But I honestly don't think that is what I mean. Suppose that by an amazing cosmic coincidence, a flurry of neutrinos struck my neurons, causing me to imagine two dots colliding with three dots a...
Can Counterfactuals Be True?
by Eliezer Yudkowsky — 2008-07-24T04:40:49.000Z

Followup to: Probability is Subjectively Objective

The classic explanation of counterfactuals begins with this distinction:

If Lee Harvey Oswald didn't shoot John F. Kennedy, then someone else did.

If Lee Harvey Oswald hadn't shot John F. Kennedy, someone else would have.

In ordinary usage we would agree with the first statement, but not the second (I hope).

If, somehow, we learn the definite fact that Oswald did not shoot Kennedy, then someone else must have done so, since Kennedy was in fact shot.

But if we went back in time and removed Oswald, while leaving everything else the same, then—unless you believe there was a conspiracy—there's no particular reason to believe Kennedy would be shot:

We start by imagining the same historical situation that existed in 1963—by a further act of imagination, we remove Oswald from our vision—we run forward the laws that we think govern the world—visualize Kennedy parading through in his limousine—and find that, in our imagination, no one shoots Kennedy.

It's an interesting question whether counterfactuals can be true or false. We never get to experience them directly.

If we disagree on what would have happened if Oswald hadn't been there, what experiment could we perform to find out which of us is right?

And if the counterfactual is something unphysical—like, "If gravity had stopped working three days ago, the Sun would have exploded"—then there aren't even any alternate histories out there to provide a truth-value.

It's not as simple as saying that if the bucket contains three pebbles, and the pasture contains three sheep, the bucket is true.

Since the counterfactual event only exists in your imagination, how can it be true or false?

So... is it just as fair to say that "If Oswald hadn't shot Kennedy, the Sun would have exploded"?

After all, the event only exists in our imaginations—surely that means it's subjective, so we can say anything we like?

But so long as we have a lawful specification of how counterfactuals are constructed—a lawful computational procedure—then the counterfactual result of removing Oswald, depends entirely on the empirical state of the world.

If there was no conspiracy, then any reasonable computational procedure that simulates removing Oswald's bullet from the course of history, ought to return an answer of Kennedy not getting ...
When (Not) To Use Probabilities
by Eliezer Yudkowsky — 2008-07-23T10:58:39.000Z

It may come as a surprise to some readers of this blog, that I do not always advocate using probabilities.

Or rather, I don't always advocate that human beings, trying to solve their problems, should try to make up verbal probabilities, and then apply the laws of probability theory or decision theory to whatever number they just made up, and then use the result as their final belief or decision.

The laws of probability are laws, not suggestions, but often the true Law is too difficult for us humans to compute. If P != NP and the universe has no source of exponential computing power, then there are evidential updates too difficult for even a superintelligence to compute - even though the probabilities would be quite well-defined, if we could afford to calculate them.

So sometimes you don't apply probability theory. Especially if you're human, and your brain has evolved with all sorts of useful algorithms for uncertain reasoning, that don't involve verbal probability assignments.

Not sure where a flying ball will land? I don't advise trying to formulate a probability distribution over its landing spots, performing deliberate Bayesian updates on your glances at the ball, and calculating the expected utility of all possible strings of motor instructions to your muscles.

Trying to catch a flying ball, you're probably better off with your brain's built-in mechanisms, then using deliberative verbal reasoning to invent or manipulate probabilities.

But this doesn't mean you're going beyond probability theory or above probability theory.

The Dutch Book arguments still apply. If I offer you a choice of gambles ($10,000 if the ball lands in this square, versus $10,000 if I roll a die and it comes up 6), and you answer in a way that does not allow consistent probabilities to be assigned, then you will accept combinations of gambles that are certain losses, or reject gambles that are certain gains...

Which still doesn't mean that you should try to use deliberative verbal reasoning. I would expect that for professional baseball players, at least, it's more important to catch the ball than to assign consistent probabilities. Indeed, if you tried to make up probabilities, the verbal probabilities might not even be very good ones, compared to some gut-level feeling - some wordless representation of uncertainty in the back of your mind.

There is noth...