trentmkelly/LessWrong-43k huggingface dataset:
Moral Error and Moral Disagreement
by Eliezer Yudkowsky — 2008-08-10T23:32:08.000Z

Followup to: Inseparably Right, Sorting Pebbles Into Correct Heaps

Richard Chappell, a pro, writes:

"When Bob says "Abortion is wrong", and Sally says, "No it isn't", they are disagreeing with each other.

I don't see how Eliezer can accommodate this. On his account, what Bob asserted is true iff abortion is prohibited by the morality_Bob norms. How can Sally disagree? There's no disputing (we may suppose) that abortion is indeed prohibited by morality_Bob...

Since there is moral disagreement, whatever Eliezer purports to be analysing here, it is not morality."

The phenomena of moral disagreement, moral error, and moral progress, on terminal values, are the primary drivers behind my metaethics. Think of how simple Friendly AI would be if there were no moral disagreements, moral errors, or moral progress!

Richard claims, "There's no disputing (we may suppose) that abortion is indeed prohibited by morality_Bob."

We may not suppose, and there is disputing. Bob does not have direct, unmediated, veridical access to the output of his own morality.

I tried to describe morality as a "computation". In retrospect, I don't think this is functioning as the Word of Power that I thought I was emitting.

Let us read, for "computation", "idealized abstract dynamic"—maybe that will be a more comfortable label to apply to morality.

Even so, I would have thought it obvious that computations may be the subjects of mystery and error. Maybe it's not as obvious outside computer science?

Disagreement has two prerequisites: the possibility of agreement and the possibility of error. For two people to agree on something, there must be something they are agreeing about, a referent held in common. And it must be possible for an "error" to take place, a conflict between "P" in the map and not-P in the territory. Where these two prerequisites are present, Sally can say to Bob: "That thing we were just both talking about—you are in error about it."

Richard's objection would seem in the first place to rule out the possibility of moral error, from which he derives the impossibility of moral agreement.

So: does my metaethics rule out moral error? Is there no disputing that abortion is indeed prohibited by morality_Bob?

This is such a strange idea that I find myself wondering what the heck Richard could be thinking. My best gues...
Sorting Pebbles Into Correct Heaps
by Eliezer Yudkowsky — 2008-08-10T01:00:00.000Z

Once upon a time there was a strange little species—that might have been biological, or might have been synthetic, and perhaps were only a dream—whose passion was sorting pebbles into correct heaps.

They couldn't tell you why some heaps were correct, and some incorrect. But all of them agreed that the most important thing in the world was to create correct heaps, and scatter incorrect ones.

Why the Pebblesorting People cared so much, is lost to this history—maybe a Fisherian runaway sexual selection, started by sheer accident a million years ago? Or maybe a strange work of sentient art, created by more powerful minds and abandoned?

But it mattered so drastically to them, this sorting of pebbles, that all the Pebblesorting philosophers said in unison that pebble-heap-sorting was the very meaning of their lives: and held that the only justified reason to eat was to sort pebbles, the only justified reason to mate was to sort pebbles, the only justified reason to participate in their world economy was to efficiently sort pebbles.

The Pebblesorting People all agreed on that, but they didn't always agree on which heaps were correct or incorrect.

In the early days of Pebblesorting civilization, the heaps they made were mostly small, with counts like 23 or 29; they couldn't tell if larger heaps were correct or not. Three millennia ago, the Great Leader Biko made a heap of 91 pebbles and proclaimed it correct, and his legions of admiring followers made more heaps likewise. But over a handful of centuries, as the power of the Bikonians faded, an intuition began to accumulate among the smartest and most educated that a heap of 91 pebbles was incorrect. Until finally they came to know what they had done: and they scattered all the heaps of 91 pebbles. Not without flashes of regret, for some of those heaps were great works of art, but incorrect. They even scattered Biko's original heap, made of 91 precious gemstones each of a different type and color.

And no civilization since has seriously doubted that a heap of 91 is incorrect.

Today, in these wiser times, the size of the heaps that Pebblesorters dare attempt, has grown very much larger—which all agree would be a most great and excellent thing, if only they could ensure the heaps were really correct. Wars have been fought between countries that disagree o...
Inseparably Right; or, Joy in the Merely Good
by Eliezer Yudkowsky — 2008-08-09T01:00:00.000Z

Followup to: The Meaning of Right

I fear that in my drive for full explanation, I may have obscured the punchline from my theory of metaethics. Here then is an attempted rephrase:

There is no pure ghostly essence of goodness apart from things like truth, happiness and sentient life.

What do you value? At a guess, you value the life of your friends and your family and your Significant Other and yourself, all in different ways. You would probably say that you value human life in general, and I would take your word for it, though Robin Hanson might ask how you've acted on this supposed preference. If you're reading this blog you probably attach some value to truth for the sake of truth. If you've ever learned to play a musical instrument, or paint a picture, or if you've ever solved a math problem for the fun of it, then you probably attach real value to good art. You value your freedom, the control that you possess over your own life; and if you've ever really helped someone you probably enjoyed it. You might not think of playing a video game as a great sacrifice of dutiful morality, but I for one would not wish to see the joy of complex challenge perish from the universe. You may not think of telling jokes as a matter of interpersonal morality, but I would consider the human sense of humor as part of the gift we give to tomorrow.

And you value many more things than these.

Your brain assesses these things I have said, or others, or more, depending on the specific event, and finally affixes a little internal representational label that we recognize and call "good".

There's no way you can detach the little label from what it stands for, and still make ontological or moral sense.

Why might the little 'good' label seem detachable? A number of reasons.

Mainly, that's just how your mind is structured—the labels it attaches internally seem like extra, floating, ontological properties.

And there's no one value that determines whether a complicated event is good or not—and no five values, either. No matter what rule you try to describe, there's always something left over, some counterexample. Since no single value defines goodness, this can make it seem like all of them together couldn't define goodness. But when you add them up all together, there is nothing else left.

If there's no detach...
Morality as Fixed Computation
by Eliezer Yudkowsky — 2008-08-08T01:00:00.000Z

Toby Ord commented:

Eliezer, I've just reread your article and was wondering if this is a good quick summary of your position (leaving apart how you got to it):

'I should X' means that I would attempt to X were I fully informed.

Toby's a pro, so if he didn't get it, I'd better try again. Let me try a different tack of explanation—one closer to the historical way that I arrived at my own position.

Suppose you build an AI, and—leaving aside that AI goal systems cannot be built around English statements, and all such descriptions are only dreams—you try to infuse the AI with the action-determining principle, "Do what I want."

And suppose you get the AI design close enough—it doesn't just end up tiling the universe with paperclips, cheesecake or tiny molecular copies of satisfied programmers—that its utility function actually assigns utilities as follows, to the world-states we would describe in English as:

<Programmer weakly desires 'X', quantity 20 of X exists>: +20
<Programmer strongly desires 'Y', quantity 20 of X exists>: 0
<Programmer weakly desires 'X', quantity 30 of Y exists>: 0
<Programmer strongly desires 'Y', quantity 30 of Y exists>: +60

You perceive, of course, that this destroys the world.

...since if the programmer initially weakly wants 'X' and X is hard to obtain, the AI will modify the programmer to strongly want 'Y', which is easy to create, and then bring about lots of Y. Y might be, say, iron atoms—those are highly stable.

Can you patch this problem? No. As a general rule, it is not possible to patch flawed Friendly AI designs.

If you try to bound the utility function, or make the AI not care about how much the programmer wants things, the AI still has a motive (as an expected utility maximizer) to make the programmer want something that can be obtained with a very high degree of certainty.

If you try to make it so that the AI can't modify the programmer, then the AI can't talk to the programmer (talking to someone modifies them).

If you try to rule out a specific class of ways the AI could modify the programmer, the AI has a motive to superintelligently seek out loopholes and ways to modify the programmer indirectly.

As a general rule, it is not possible to patch flawed FAI designs.

We, ourselves, do not imagine ...
Hiroshima Day
by Eliezer Yudkowsky — 2008-08-06T23:15:00.000Z

On August 6th, in 1945, the world saw the first use of atomic weapons against human targets. On this day 63 years ago, humanity lost its nuclear virginity. Until the end of time we will be a species that has used fission bombs in anger.

Time has passed, and we still haven't blown up our world, despite a close call or two. Which makes it difficult to criticize the decision - would things still have turned out all right, if anyone had chosen differently, anywhere along the way?

Maybe we needed to see the ruins, of the city and the people.

Maybe we didn't.

There's an ongoing debate - and no, it is not a settled issue - over whether the Japanese would have surrendered without the Bomb. But I would not have dropped the Bomb even to save the lives of American soldiers, because I would have wanted to preserve that world where atomic weapons had never been used - to not cross that line. I don't know about history to this point; but the world would be safer now, I think, today, if no one had ever used atomic weapons in war, and the idea was not considered suitable for polite discussion.

I'm not saying it was wrong. I don't know for certain that it was wrong. I wouldn't have thought that humanity could make it this far without using atomic weapons again. All I can say is that if it had been me, I wouldn't have done it.
Contaminated by Optimism
by Eliezer Yudkowsky — 2008-08-06T00:26:52.000Z

Followup to: Anthropomorphic Optimism, The Hidden Complexity of Wishes

Yesterday, I reprised in further detail The Tragedy of Group Selectionism, in which early biologists believed that predators would voluntarily restrain their breeding to avoid exhausting the prey population; the given excuse was "group selection". Not only does it turn out to be nearly impossible for group selection to overcome a countervailing individual advantage; but when these nigh-impossible conditions were created in the laboratory - group selection for low-population groups - the actual result was not restraint in breeding, but, of course, cannibalism, especially of immature females.

I've made even sillier mistakes, by the way - though about AI, not evolutionary biology. And the thing that strikes me, looking over these cases of anthropomorphism, is the extent to which you are screwed as soon as you let anthropomorphism suggest ideas to examine.

In large hypothesis spaces, the vast majority of the cognitive labor goes into noticing the true hypothesis. By the time you have enough evidence to consider the correct theory as one of just a few plausible alternatives - to represent the correct theory in your mind - you're practically done. Of this I have spoken several times before.

And by the same token, my experience suggests that as soon as you let anthropomorphism promote a hypothesis to your attention, so that you start wondering if that particular hypothesis might be true, you've already committed most of the mistake.

The group selectionists did not deliberately extend credit to the belief that evolution would do the aesthetic thing, the nice thing. The group selectionists were doomed when they let their aesthetic sense make a suggestion - when they let it promote a hypothesis to the level of deliberate consideration.

It's not like I knew the original group selectionists. But I've made analogous mistakes as a teenager, and then watched others make the mistake many times over. So I do have some experience whereof I speak, when I speak of instant doom.

Unfortunately, the prophylactic against this mistake, is not a recognized technique of Traditional Rationality.

In Traditional Rationality, you can get your ideas from anywhere. Then you weigh up the evidence for and against them, searching for arguments on both sides. If the question has...
Anthropomorphic Optimism
by Eliezer Yudkowsky — 2008-08-04T20:17:28.000Z

The core fallacy of anthropomorphism is expecting something to be predicted by the black box of your brain, when its casual structure is so different from that of a human brain, as to give you no license to expect any such thing.

The Tragedy of Group Selectionism (as previously covered in the evolution sequence) was a rather extreme error by a group of early (pre-1966) biologists, including Wynne-Edwards, Allee, and Brereton among others, who believed that predators would voluntarily restrain their breeding to avoid overpopulating their habitat and exhausting the prey population.

The proffered theory was that if there were multiple, geographically separated groups of e.g. foxes, then groups of foxes that best restrained their breeding, would send out colonists to replace crashed populations. And so, over time, group selection would promote restrained-breeding genes in foxes.

I'm not going to repeat all the problems that developed with this scenario. Suffice it to say that there was no empirical evidence to start with; that no empirical evidence was ever uncovered; that, in fact, predator populations crash all the time; and that for group selection pressure to overcome a countervailing individual selection pressure, turned out to be very nearly mathematically impossible.

The theory having turned out to be completely incorrect, we may ask if, perhaps, the originators of the theory were doing something wrong.

"Why be so uncharitable?" you ask. "In advance of doing the experiment, how could they know that group selection couldn't overcome individual selection?"

But later on, Michael J. Wade went out and actually created in the laboratory the nigh-impossible conditions for group selection. Wade repeatedly selected insect subpopulations for low population numbers. Did the insects evolve to restrain their breeding, and live in quiet peace with enough food for all, as the group selectionists had envisioned?

No; the adults adapted to cannibalize eggs and larvae, especially female larvae.

Of course selecting for small subpopulation sizes would not select for individuals who restrained their own breeding. It would select for individuals who ate other individuals' children. Especially the girls.

Now, why might the group selectionists have not thought of that possibility?

Suppose you were a member of a tribe, and you knew that, in the near ...
No Logical Positivist I
by Eliezer Yudkowsky — 2008-08-04T01:06:37.000Z

Followup to: Making Beliefs Pay Rent, Belief in the Implied Invisible

Degrees of Freedom accuses me of reinventing logical positivism, badly:

One post which reads as though it were written in Vienna in the 1920s is this one [Making Beliefs Pay Rent] where Eliezer writes

"We
can build up whole networks of beliefs that are connected only to each
other - call these "floating" beliefs. It is a uniquely human flaw
among animal species, a perversion of Homo sapiens's ability to build
more general and flexible belief networks... The rationalist
virtue of empiricism consists of constantly asking which experiences
our beliefs predict - or better yet, prohibit."

Logical positivists were best known for their verificationism: the idea that a belief is defined in terms of the experimental predictions that it makes. Not just tested, not just confirmed, not just justified by experiment, but actually defined as a set of allowable experimental results. An idea unconfirmable by experiment is not just probably wrong, but necessarily meaningless.

I would disagree, and exhibit logical positivism as another case in point of "mistaking the surface of rationality for its substance".

Consider the
hypothesis:

On August 1st 2008 at midnight Greenwich time, a
one-foot sphere of chocolate cake spontaneously formed in the center of the Sun; and then, in the natural course of events, this Boltzmann Cake almost instantly dissolved.

I would say that this hypothesis is meaningful and almost certainly false. Not that it is "meaningless". Even though I cannot think of any possible experimental test that would discriminate between its being true, and its being false.

On the other hand, if some postmodernist literature professor tells me that Shakespeare shows signs of "post-colonial alienation", the burden of proof is on him to show that this statement means anything, before we can talk about its being true or false.

I think the two main probability-theoretic concepts here are Minimum Message Length and directed causal graphs - both of which came along well after logical positivism.

By talking about the unseen causes of visible events, it is often possible for me to compress the description of visible events. By talking about atoms, I can compress the description of the chemical reactions I've observed.

We build up...
The Comedy of Behaviorism
by Eliezer Yudkowsky — 2008-08-02T20:42:42.000Z

Followup to: Humans in Funny Suits

"Let me see if I understand your thesis. You think we shouldn't anthropomorphize people?"
-- Sidney Morgenbesser to B. F. Skinner

Behaviorism was the doctrine that it was unscientific for a psychologist to ascribe emotions, beliefs, thoughts, to a human being. After all, you can't directly observe anger or an intention to hit someone. You can only observe the punch. You may hear someone say "I'm angry!" but that's hearing a verbal behavior, not seeing anger. Thoughts are not observable, therefore they are unscientific, therefore they do not exist. Oh, you think you're thinking, but that's just a delusion - or it would be, if there were such things as delusions.

This was before the day of computation, before the concept of information processing, before the "cognitive revolution" that led to the modern era. If you looked around for an alternative to behaviorism, you didn't find, "We're going to figure out how the mind processes information by scanning the brain, reading neurons, and performing experiments that test our hypotheses about specific algorithms." You found Freudian psychoanalysis. This may make behaviorism a bit more sympathetic.

Part of the origin of behaviorism, was in a backlash against substance dualism - the idea that mind is a separate substance from ordinary physical phenomena. Today we would say, "The apparent specialness comes from the brain doing information-processing; a physical deed to be sure, but one of a different style from smashing a brick with a sledgehammer." The behaviorists said, "There is no mind." (John B. Watson, founder of behaviorism, in 1928.)

The behaviorists outlawed not just dualistic mind-substance, but any such things as emotions and beliefs and intentions (unless defined in terms of outward behavior). After all, science had previously done away with angels, and de-gnomed the haunted mine. Clearly the method of reason, then, was to say that things didn't exist. Having thus fathomed the secret of science, the behaviorists proceeded to apply it to the mind.

You might be tempted to say, "What fools! Obviously, the mind, like the rainbow, does exist; it is to be explained, not explained away. Saying 'the subject is angry' helps me predict the subject...
A Genius for Destruction
by Eliezer Yudkowsky — 2008-08-01T19:25:27.000Z

This is a question from a workshop after the Global Catastrophic Risks conference. The rule of the workshop was that people could be quoted, but not attributed, so I won't say who observed:

"The problem is that it's often our smartest people leading us into the disasters. Look at Long-Term Capital Management."

To which someone else replied:

"Maybe smart people are just able to work themselves up into positions of power, so that if damage gets caused, the responsibility will often lie with someone smart."

Since we'd recently been discussing complexity, interdependence and breakdowns, the first observation that came to my own mind was the old programmers' saying:

"It takes more
intelligence to debug code than to write it. Therefore, if you write
the most difficult code you can create, you are not
smart enough to debug it."

(This in the context of how increased system complexity is a global risk and commons problem; but individuals have an incentive to create the "smartest" systems they can devise locally.)

There is also the standard suite of observations as to how smart people can become stupid:

You become overly skilled at defending beliefs you arrived at for unskilled reasons;

Success on comparatively easy problems in the past, leads to overconfidence on more difficult problems in the future;

Because most of the advice offered you comes from people (apparently) not as smart as you, you become intellectually isolated;

You spend too much time as the smartest person in the room and can't handle it emotionally;

Because you enjoy looking smart, you avoid trying new and difficult things where you might not perform as well, and so become very narrow;

Because you enjoy looking smart, you don't like confessing your mistakes or realizing your losses;

Because people praise you for being good at math or something, you assume you are already wise, and you don't apply your intelligence to becoming wiser in other areas.

But I also think we should strongly consider that perhaps the "highly intelligent" sponsors of major catastrophes are not so formidable as they appear - that they are not the truly best and brightest gone wrong; only the somewhat-competent with luck or good PR. As I earlier observed: Calling the saga of the fall of Enron, "The Smartest Guys in the Room", deserves an a...