trentmkelly/LessWrong-43k huggingface dataset:
Ethics Notes
by Eliezer Yudkowsky — 2008-10-21T21:57:50.000Z

Followup to: Ethical Inhibitions, Ethical Injunctions, Prices or Bindings?

(Some collected replies to comments on the above three posts.)

From Ethical Inhibitions:

Spambot: Every major democratic political leader lies abundantly to obtain office, as it's a necessity to actually persuade the voters. So Bill Clinton, Jean Chretien, Winston Churchill should qualify for at least half of your list of villainy.

Have the ones who've lied more, done better?

In cases where the politician who told more lies won, has that politician gone on to rule well in an absolute sense?

Is it actually true that no one who refused to lie (and this is not the same as always telling the whole truth) could win political office?

Are the lies expected, and in that sense, less than true betrayals of someone who trusts you?

Are there understood Rules of Politics that include lies but not assassinations, which the good politicians abide by, so that they are not really violating the ethics of their tribe?

Will the world be so much worse off if sufficiently good people refuse to tell outright lies and are thereby barred from public office; or would we thereby lose a George Washington or Marcus Aurelius or two, and thereby darken history?

Pearson: American revolutionaries as well ended human lives for the greater good

Police must sometimes kill the guilty. Soldiers must sometimes kill civilians (or if the enemy knows you're reluctant, that gives them a motive to use civilians as a shield). Spies sometimes have legitimate cause to kill people who helped them, but this has probably been done far more often than it has been justified by a need to end the Nazi nuclear program.

I think it's worth noting that in all such cases, you can write out something like a code of ethics and at least try to have social acceptance of it. Politicians, who lie, may prefer not to discuss the whole thing, but politicians are only a small slice of society.

Are there many who transgress even the unwritten rules and end up really implementing the greater good? (And no, there's no unwritten rule that says you can rob a bank to stop global warming.)

...but if you're placing yourself under unusual stress, you may need to be stricter than what society will accept from you. In fact, I think it's fair to say that the further I push any art, such as rationality or AI theory, the more I perceive...
Prices or Bindings?
by Eliezer Yudkowsky — 2008-10-21T16:00:00.000Z

Followup to: Ethical Injunctions

During World War II, Knut Haukelid and three other saboteurs sank a civilian Norwegian ferry ship, the SF Hydro, carrying a shipment of deuterium for use as a neutron moderator in Germany's atomic weapons program. Eighteen dead, twenty-nine survivors. And that was the end of the Nazi nuclear program. Can you imagine a Hollywood movie in which the hero did that, instead of coming up with some amazing clever way to save the civilians on the ship?

Stephen Dubner and Steven Levitt published the work of an anonymous economist turned bagelseller, Paul F., who dropped off baskets of bagels and came back to collect money from a cashbox, and also collected statistics on payment rates. The current average payment rate is 89%. Paul F. found that people on the executive floor of a company steal more bagels; that people with security clearances don't steal any fewer bagels; that telecom companies have robbed him and that law firms aren't worth the trouble.

Hobbes (of Calvin and Hobbes) once said: "I don't know what's worse, the fact that everyone's got a price, or the fact that their price is so low."

If Knut Haukelid sold his soul, he held out for a damned high price—the end of the Nazi atomic weapons program.

Others value their integrity less than a bagel.

One suspects that Haukelid's price was far higher than most people would charge, if you told them to never sell out. Maybe we should stop telling people they should never let themselves be bought, and focus on raising their price to something higher than a bagel?

But I really don't know if that's enough.

The German philosopher Fichte once said, "I would not break my word even to save humanity."

Raymond Smullyan, in whose book I read this quote, seemed to laugh and not take Fichte seriously.

Abraham Heschel said of Fichte, "His salvation and righteousness were apparently so much more important to him than the fate of all men that he would have destroyed mankind to save himself."

I don't think they get it.

If a serial killer comes to a confessional, and confesses that he's killed six people and plans to kill more, should the priest turn him in? I would answer, "No." If not for the seal of the confessional, the serial killer would never have come to the priest in the first place. All else being equal, I would prefer the world in...
Ethical Injunctions
by Eliezer Yudkowsky — 2008-10-20T23:00:00.000Z

"Would you kill babies if it was the right thing to do? If no, under what circumstances would you not do the right thing to do? If yes, how right would it have to be, for how many babies?"
—horrible job interview question

Swapping hats for a moment, I'm professionally intrigued by the decision theory of "things you shouldn't do even if they seem to be the right thing to do".

Suppose we have a reflective AI, self-modifying and self-improving, at an intermediate stage in the development process. In particular, the AI's goal system isn't finished—the shape of its motivations is still being loaded, learned, tested, or tweaked.

Yea, I have seen many ways to screw up an AI goal system design, resulting in a decision system that decides, given its goals, that the universe ought to be tiled with tiny molecular smiley-faces, or some such. Generally, these deadly suggestions also have the property that the AI will not desire its programmers to fix it. If the AI is sufficiently advanced—which it may be even at an intermediate stage—then the AI may also realize that deceiving the programmers, hiding the changes in its thoughts, will help transform the universe into smiley-faces.

Now, from our perspective as programmers, if we condition on the fact that the AI has decided to hide its thoughts from the programmers, or otherwise act willfully to deceive us, then it would seem likely that some kind of unintended consequence has occurred in the goal system. We would consider it probable that the AI is not functioning as intended, but rather likely that we have messed up the AI's utility function somehow. So that the AI wants to turn the universe into tiny reward-system counters, or some such, and now has a motive to hide from us.

Well, suppose we're not going to implement some object-level Great Idea as the AI's utility function. Instead we're going to do something advanced and recursive—build a goal system which knows (and cares) about the programmers outside. A goal system that, via some nontrivial internal structure, "knows it's being programmed" and "knows it's incomplete". Then you might be able to have and keep the rule:

"If [I decide that] fooling my programmers is the right thing to do, execute a controlled shutdown [instead of doing the right thin

...
Ethical Inhibitions
by Eliezer Yudkowsky — 2008-10-19T20:44:02.000Z

Followup to: Entangled Truths, Contagious Lies, Evolutionary Psychology

What's up with that bizarre emotion we humans have, this sense of ethical caution?

One can understand sexual lust, parental care, and even romantic attachment. The evolutionary psychology of such emotions might be subtler than it at first appears, but if you ignore the subtleties, the surface reasons are obvious. But why a sense of ethical caution? Why honor, why righteousness? (And no, it's not group selection; it never is.) What reproductive benefit does that provide?

The specific ethical codes that people feel uneasy violating, vary from tribe to tribe (though there are certain regularities). But the emotion associated with feeling ethically inhibited—well, I Am Not An Evolutionary Anthropologist, but that looks like a human universal to me, something with brainware support.

The obvious story behind prosocial emotions in general, is that those who offend against the group are sanctioned; this converts the emotion to an individual reproductive advantage. The human organism, executing the ethical-caution adaptation, ends up avoiding the group sanctions that would follow a violation of the code. This obvious answer may even be the entire answer.

But I suggest—if a bit more tentatively than usual—that by the time human beings were evolving the emotion associated with "ethical inhibition", we were already intelligent enough to observe the existence of such things as group sanctions. We were already smart enough (I suggest) to model what the group would punish, and to fear that punishment.

Sociopaths have a concept of getting caught, and they try to avoid getting caught. Why isn't this sufficient? Why have an extra emotion, a feeling that inhibits you even when you don't expect to be caught? Wouldn't this, from evolution's perspective, just result in passing up perfectly good opportunities?

So I suggest (tentatively) that humans naturally underestimate the odds of getting caught. We don't foresee all the possible chains of causality, all the entangled facts that can bring evidence against us. Those ancestors who lacked a sense of ethical caution stole the silverware when they expected that no one would catch them or punish them; and were nonetheless caught or punished often enough, on averag...
Protected From Myself
by Eliezer Yudkowsky — 2008-10-19T00:09:31.000Z

Followup to: The Magnitude of His Own Folly, Entangled Truths, Contagious Lies

Every now and then, another one comes before me with the brilliant idea: "Let's lie!"

Lie about what?—oh, various things. The expected time to Singularity, say. Lie and say it's definitely going to be earlier, because that will get more public attention. Sometimes they say "be optimistic", sometimes they just say "lie". Lie about the current degree of uncertainty, because there are other people out there claiming to be certain, and the most unbearable prospect in the world is that someone else pull ahead. Lie about what the project is likely to accomplish—I flinch even to write this, but occasionally someone proposes to go and say to the Christians that the AI will create Christian Heaven forever, or go to the US government and say that the AI will give the US dominance forever.

But at any rate, lie. Lie because it's more convenient than trying to explain the truth. Lie, because someone else might lie, and so we have to make sure that we lie first. Lie to grab the tempting benefits, hanging just within reach—

Eh? Ethics? Well, now that you mention it, lying is at least a little bad, all else being equal. But with so much at stake, we should just ignore that and lie. You've got to follow the expected utility, right? The loss of a lie is much less than the benefit to be gained, right?

Thus do they argue. Except—what's the flaw in the argument? Wouldn't it be irrational not to lie, if lying has the greatest expected utility?

When I look back upon my history—well, I screwed up in a lot of ways. But it could have been much worse, if I had reasoned like those who offer such advice, and lied.

Once upon a time, I truly and honestly believed that either a superintelligence would do what was right, or else there was no right thing to do; and I said so. I was uncertain of the nature of morality, and I said that too. I didn't know if the Singularity would be in five years or fifty, and this also I admitted. My project plans were not guaranteed to deliver results, and I did not promise to deliver them. When I finally said "Oops", and realized that I needed to go off and do more fundamental research instead of rushing to write code immediately&mdas...
Dark Side Epistemology
by Eliezer Yudkowsky — 2008-10-17T23:55:22.000Z

If you once tell a lie, the truth is ever after your enemy.

I have discussed the notion that lies are contagious. If you pick up a pebble from the driveway, and tell a geologist that you found it on a beach—well, do you know what a geologist knows about rocks? I don’t. But I can suspect that a water-worn pebble wouldn’t look like a droplet of frozen lava from a volcanic eruption. Do you know where the pebble in your driveway really came from? Things bear the marks of their places in a lawful universe; in that web, a lie is out of place.1

What sounds like an arbitrary truth to one mind—one that could easily be replaced by a plausible lie—might be nailed down by a dozen linkages to the eyes of greater knowledge. To a creationist, the idea that life was shaped by “intelligent design” instead of “natural selection” might sound like a sports team to cheer for. To a biologist, plausibly arguing that an organism was intelligently designed would require lying about almost every facet of the organism. To plausibly argue that “humans” were intelligently designed, you’d have to lie about the design of the human retina, the architecture of the human brain, the proteins bound together by weak van der Waals forces instead of strong covalent bonds . . .

Or you could just lie about evolutionary theory, which is the path taken by most creationists. Instead of lying about the connected nodes in the network, they lie about the general laws governing the links.

And then to cover that up, they lie about the rules of science—like what it means to call something a “theory,” or what it means for a scientist to say that they are not absolutely certain.

So they pass from lying about specific facts, to lying about general laws, to lying about the rules of reasoning. To lie about whether humans evolved, you must lie about evolution; and then you have to lie about the rules of science that constrain our understanding of evolution.

But how else? Just as a human would be out of place in a community of actually intelligently designed life forms, and you have to lie about the rules of evolution to make it appear otherwise, so too beliefs about creationism are themselves out of place in science—you wouldn’t find them in a well-ordered mind any more than yo...
Traditional Capitalist Values
by Eliezer Yudkowsky — 2008-10-17T01:07:16.000Z

Followup to: Are Your Enemies Innately Evil?, Policy Debates Should Not Appear One-Sided

"The financial crisis is not the crisis of capitalism. It is the crisis of a system that has distanced itself from the most fundamental values of capitalism, which betrayed the spirit of capitalism."
-- Nicolas Sarkozy

During the current crisis, I've more than once heard someone remarking that financial-firm CEOs who take huge bonuses during the good years and then run away when their black-swan bets blow up, are only exercising the usual capitalist values of "grab all the money you can get".

I think that a fair amount of the enmity in the world, to say nothing of confusion on the Internet, stems from people refusing to contemplate the real values of the opposition as the opposition sees it. This is something I've remarked upon before, with respect to "the terrorists hate our freedom" or "the suicide hijackers were cowards" (statements that are sheerly silly).

Real value systems - as opposed to pretend demoniacal value systems - are phrased to generate warm fuzzies in their users, not to be easily mocked. They will sound noble at least to the people who believe them.

Whether anyone actually lives up to that value system, or should, and whether the results are what they are claimed to be; if there are hidden gotchas in the warm fuzzy parts - sure, you can have that debate. But first you should be clear about how your opposition sees itself - a view which has not been carefully optimized to make your side feel good about its opposition. Otherwise you're not engaging the real issues.

So here are the traditional values of capitalism as seen by those who regard it as noble - the sort of Way spoken of by Paul Graham, or P. T. Barnum (who did not say "There's a sucker born every minute"), or Warren Buffett:

Make things that people want, or do things that people want done, in exchange for money or other valuta. This is a great and noble and worthwhile endeavor, and anyone who looks down on it reveals their own shallowness.

Your competitors are your loyal opposition. Usher them toward oblivion by offering a better product at a lower price, then dance on their graves (or buy them out, if they're worthy). An act of violence, like punching them in the nose, is as absolutely forbidden as it would be

...
Entangled Truths, Contagious Lies
by Eliezer Yudkowsky — 2008-10-15T23:39:36.000Z

One of your very early philosophers came to the conclusion that a fully competent mind, from a study of one fact or artifact belonging to any given universe, could construct or visualize that universe, from the instant of its creation to its ultimate end . . .

—First Lensman

If any one of you will concentrate upon one single fact, or small object, such as a pebble or the seed of a plant or other creature, for as short a period of time as one hundred of your years, you will begin to perceive its truth.

—Gray Lensman

I am reasonably sure that a single pebble, taken from a beach of our own Earth, does not specify the continents and countries, politics and people of this Earth. Other planets in space and time, other Everett branches, would generate the same pebble.

On the other hand, the identity of a single pebble would seem to include our laws of physics. In that sense the entirety of our Universe—all the Everett branches—would be implied by the pebble.1

From the study of that single pebble you could see the laws of physics and all they imply. Thinking about those laws of physics, you can see that planets will form, and you can guess that the pebble came from such a planet. The internal crystals and molecular formations of the pebble developed under gravity, which tells you something about the planet’s mass; the mix of elements in the pebble tells you something about the planet’s formation.

I am not a geologist, so I don’t know to which mysteries geologists are privy. But I find it very easy to imagine showing a geologist a pebble, and saying, “This pebble came from a beach at Half Moon Bay,” and the geologist immediately says, “I’m confused,” or even, “You liar.” Maybe it’s the wrong kind of rock, or the pebble isn’t worn enough to be from a beach—I don’t know pebbles well enough to guess the linkages and signatures by which I might be caught, which is the point.

“Only God can tell a truly plausible lie.” I wonder if there was ever a religion that developed this as a proverb? I would (falsifiably) guess not: it’s a rationalist sentiment, even if you cast it in theological metaphor. Saying “everything is interconnected to everything else, because God mad...
Ends Don't Justify Means (Among Humans)
by Eliezer Yudkowsky — 2008-10-14T21:00:00.000Z

"If the ends don't justify the means, what does?"
—variously attributed

"I think of myself as running on hostile hardware."
—Justin Corwin

Yesterday I talked about how humans may have evolved a structure of political revolution, beginning by believing themselves morally superior to the corrupt current power structure, but ending by being corrupted by power themselves—not by any plan in their own minds, but by the echo of ancestors who did the same and thereby reproduced.

This fits the template:

In some cases, human beings have evolved in such fashion as to think that they are doing X for prosocial reason Y, but when human beings actually do X, other adaptations execute to promote self-benefiting consequence Z.

From this proposition, I now move on to my main point, a question considerably outside the realm of classical Bayesian decision theory:

"What if I'm running on corrupted hardware?"

In such a case as this, you might even find yourself uttering such seemingly paradoxical statements—sheer nonsense from the perspective of classical decision theory—as:

"The ends don't justify the means."

But if you are running on corrupted hardware, then the reflective observation that it seems like a righteous and altruistic act to seize power for yourself—this seeming may not be be much evidence for the proposition that seizing power is in fact the action that will most benefit the tribe.

By the power of naive realism, the corrupted hardware that you run on, and the corrupted seemings that it computes, will seem like the fabric of the very world itself—simply the way-things-are.

And so we have the bizarre-seeming rule: "For the good of the tribe, do not cheat to seize power even when it would provide a net benefit to the tribe."

Indeed it may be wiser to phrase it this way: If you just say, "when it seems like it would provide a net benefit to the tribe", then you get people who say, "But it doesn't just seem that way—it would provide a net benefit to the tribe if I were in charge."

The notion of untrusted hardware seems like something wholly outside the realm of classical decision theory. (What it does to reflective decision theory I can't yet say, but that would seem to be the appropriate level to handle it.)

But on a human level, the patch ...
Why Does Power Corrupt?
by Eliezer Yudkowsky — 2008-10-14T00:23:23.000Z

Followup to: Evolutionary Psychology

"Power tends to corrupt, and absolute power corrupts absolutely. Great men are almost always bad men."
—Lord Acton

Call it a just-so story if you must, but as soon as I was introduced to the notion of evolutionary psychology (~1995), it seemed obvious to me why human beings are corrupted by power. I didn't then know that hunter-gatherer bands tend to be more egalitarian than agricultural tribes—much less likely to have a central tribal-chief boss-figure—and so I thought of it this way:

Humans (particularly human males) have evolved to exploit power and status when they obtain it, for the obvious reason: If you use your power to take many wives and favor your children with a larger share of the meat, then you will leave more offspring, ceteris paribus. But you're not going to have much luck becoming tribal chief if you just go around saying, "Put me in charge so that I can take more wives and favor my children." You could lie about your reasons, but human beings are not perfect deceivers.

So one strategy that an evolution could follow, would be to create a vehicle that reliably tended to start believing that the old power-structure was corrupt, and that the good of the whole tribe required their overthrow...

The young revolutionary's belief is honest. There will be no betraying catch in his throat, as he explains why the tribe is doomed at the hands of the old and corrupt, unless he is given power to set things right. Not even subconsciously does he think, "And then, once I obtain power, I will strangely begin to resemble that old corrupt guard, abusing my power to increase my inclusive genetic fitness."

People often think as if "purpose" is an inherent property of things; and so many interpret the message of ev-psych as saying, "You have a subconscious, hidden goal to maximize your fitness." But individual organisms are adaptation-executers, not fitness-maximizers. The purpose that the revolutionary should obtain power and abuse it, is not a plan anywhere in his brain; it belongs to evolution, which can just barely be said to have purposes. It is a fact about many past revolutionaries having successfully taken power, having abused it, and having left many descendants.

When the revolutionary obtains power, he wi...