trentmkelly/LessWrong-43k huggingface dataset:
Fake Norms, or "Truth" vs. Truth
by Eliezer Yudkowsky — 2008-07-22T10:23:30.000Z

Followup to: Applause Lights

When you say the word "truth", people know that "truth" is a good thing, and that they're supposed to applaud. So it might seem like there is a social norm in favor of "truth". But when it comes to some particular truth, like whether God exists, or how likely their startup is to thrive, people will say: "I just want to believe" or "you've got to be optimistic to succeed".

So Robin and I were talking about this, and Robin asked me how it is that people prevent themselves from noticing the conflict.

I replied that I don't think active prevention is required. First, as I quoted Michael Vassar:

"It seems to me that much of the frustration in my life prior to a few years ago has been due to thinking that all other human minds necessarily and consistently implement modus ponens."

But more importantly, I don't think there does exist any social norm in favor of truth. There's a social norm in favor of "truth". There's a difference.

How would a norm in favor of truth actually be expressed, or acquired?

If you were told many stories, as a kid, about specific people who accepted specific hard truths - like a story of a scientist accepting that their theory was wrong, say - then your brain would generalize over its experiences, and compress them, and form a concept of that-which-is-the-norm: the wordless act of accepting reality.

If you heard someone say "I don't care about the evidence, I just want to believe in God", and you saw everyone else in the room gasp and regard them in frozen shock, then your brain would generalize a social norm against self-deception. (E.g., the sort of thing that would happen if a scientist said "I don't care about the evidence, I just want to believe in my-favorite-theory" in front of their fellow scientists.)

If, on the other hand, you see lots of people saying "Isn't the truth wonderful?" or "I am in favor of truth", then you learn that when someone says "truth", you are supposed to applaud.

Now there are certain particular cases where someone will be castigated if they admit they refuse to see the truth: for example, "I've seen the evidence on global warming but I don't want to believe it." You couldn't get away with that in modern society. But this indignation doesn't have to derive from violating a norm in favor of truth - it can derive from the widely ...
Should We Ban Physics?
by Eliezer Yudkowsky — 2008-07-21T08:12:57.000Z

Nobel laureate Marie Curie died of aplastic anemia, the victim of radiation from the many fascinating glowing substances she had learned to isolate.

How could she have known? And the answer, as far as I can tell, is that she couldn't. The only way she could have avoided death was by being too scared of anything new to go near it. Would banning physics experiments have saved Curie from herself?

But far more cancer patients than just one person have been saved by radiation therapy. And the real cost of banning physics is not just losing that one experiment - it's losing physics. No more Industrial Revolution.

Some of us fall, and the human species carries on, and advances; our modern world is built on the backs, and sometimes the bodies, of people who took risks. My father is fond of saying that if the automobile were invented nowadays, the saddle industry would arrange to have it outlawed.

But what if the laws of physics had been different from what they are? What if Curie, by isolating and purifying the glowy stuff, had caused something akin to a fission chain reaction gone critical... which, the laws of physics being different, had ignited the atmosphere or produced a strangelet?

At the recent Global Catastrophic Risks conference, someone proposed a policy prescription which, I argued, amounted to a ban on all physics experiments involving the production of novel physical situations - as opposed to measuring existing phenomena. You can weigh a rock, but you can't purify radium, and you can't even expose the rock to X-rays unless you can show that exactly similar X-rays hit rocks all the time. So the Large Hadron Collider, which produces collisions as energetic as cosmic rays, but not exactly the same as cosmic rays, would be off the menu.

After all, whenever you do something new, even if you calculate that everything is safe, there is surely some probability of being mistaken in the calculation - right?

So the one who proposed the policy, disagreed that their policy cashed out to a blanket ban on physics experiments. And discussion is in progress, so I won't talk further about their policy argument.

But if you consider the policy of "Ban Physics", and leave aside the total political infeasibility, I think the strongest way to frame the issue - from the pro-ban viewpoint - would be as follows:

Supp...
Touching the Old
by Eliezer Yudkowsky — 2008-07-20T09:19:51.000Z

I'm in Oxford right now, for the Global Catastrophic Risks conference.

There's a psychological impact in walking down a street where where any given building might be older than your whole country.

Toby Ord and Anders Sandberg pointed out to me an old church tower in Oxford, that is a thousand years old.

At the risk conference I heard a talk from someone talking about what the universe will look like in 10100 years (barring intelligent modification thereof, which he didn't consider).

The psychological impact of seeing that old church tower was greater. I'm not defending this reaction, only admitting it.

I haven't traveled as much as I would travel if I were free to follow my whims; I've never seen the Pyramids. I don't think I've ever touched anything that has endured in the world for longer than that church tower.

A thousand years... I've lived less than half of 70, and sometimes it seems like a long time to me. What would it be like, to be as old as that tower? To have lasted through that much of the world, that much history and that much change?

Transhumanism does scare me. I shouldn't wonder if it scares me more than it
scares arch-luddites like Leon Kass. Kass doesn't take it seriously; he doesn't
expect to live that long.

Yet I know - and I doubt the thought ever
occurred to Kass - that even if something scares you, you can still
have the courage to confront it. Even time. Even life.

But sometimes it's such a strange thought that our world really is that old.

The inverse failure of the logical fallacy of generalization from fictional evidence, is failure to generalize from things that actually happened. We see movies, and in the ancestral environment, what you saw with your own eyes was real; we have to avoid treating them as available examples.

Conversely, history books seem like writing on paper - but those are things that really happened, even if we hear about them selectively. What happened there was as real to the people who lived it, as your own life, and equally evidence.

Sometimes it's such a strange thought that the people in the history books really lived and experienced and died - that there's so much more depth to history than anything I've seen with my own eyes; so much more life than anything I've lived.
Existential Angst Factory
by Eliezer Yudkowsky — 2008-07-19T06:55:17.000Z

Followup to: The Moral Void

A widespread excuse for avoiding rationality is the widespread belief that it is "rational" to believe life is meaningless, and thus suffer existential angst. This is one of the secondary reasons why it is worth discussing the nature of morality. But it's also worth attacking existential angst directly.

I suspect that most existential angst is not really existential. I think that most of what is labeled "existential angst" comes from trying to solve the wrong problem.

Let's say you're trapped in an unsatisfying relationship, so you're unhappy. You consider going on a skiing trip, or you actually go on a skiing trip, and you're still unhappy. You eat some chocolate, but you're still unhappy. You do some volunteer work at a charity (or better yet, work the same hours professionally and donate the money, thus applying the Law of Comparative Advantage) and you're still unhappy because you're in an unsatisfying relationship.

So you say something like: "Skiing is meaningless, chocolate is meaningless, charity is meaningless, life is doomed to be an endless stream of woe." And you blame this on the universe being a mere dance of atoms, empty of meaning. Not necessarily because of some kind of subconsciously deliberate Freudian substitution to avoid acknowledging your real problem, but because you've stopped hoping that your real problem is solvable. And so, as a sheer unexplained background fact, you observe that you're always unhappy.

Maybe you're poor, and so always unhappy. Nothing you do solves your poverty, so it starts to seem like a universal background fact, along with your unhappiness. So when you observe that you're always unhappy, you blame this on the universe being a mere dance of atoms. Not as some kind of Freudian substitution, but because it has ceased to occur to you that there does exist some possible state of affairs in which life is not painful.

What about rich heiresses with everything in the world available to buy, who still feel unhappy? Perhaps they can't get themselves into satisfying romantic relationships. One way or another, they don't know how to use their money to create happiness—they lack the expertise in hedonic psychology and/or self-awareness and/or simple competence.

So they're constantly unhappy—and th...
Could Anything Be Right?
by Eliezer Yudkowsky — 2008-07-18T07:19:28.000Z

Years ago, Eliezer1999 was convinced that he knew nothing about morality.

For all he knew, morality could require the extermination of the human species; and if so he saw no virtue in taking a stand against morality, because he thought that, by definition, if he postulated that moral fact, that meant human extinction was what "should" be done.

I thought I could figure out what was right, perhaps, given enough reasoning time and enough facts, but that I currently had no information about it. I could not trust evolution which had built me. What foundation did that leave on which to stand?

Well, indeed Eliezer1999 was massively mistaken about the nature of morality, so far as his explicitly represented philosophy went.

But as Davidson once observed, if you believe that "beavers" live in deserts, are pure white in color, and weigh 300 pounds when adult, then you do not have any beliefs about beavers, true or false. You must get at least some of your beliefs right, before the remaining ones can be wrong about anything.

My belief that I had no information about morality was not internally consistent.

Saying that I knew nothing felt virtuous, for I had once been taught that it was virtuous to confess my ignorance. "The only thing I know is that I know nothing," and all that. But in this case I would have been better off considering the admittedly exaggerated saying, "The greatest fool is the one who is not aware they are wise." (This is nowhere near the greatest kind of foolishness, but it is a kind of foolishness.)

Was it wrong to kill people? Well, I thought so, but I wasn't sure; maybe it was right to kill people, though that seemed less likely.

What kind of procedure would answer whether it was right to kill people? I didn't know that either, but I thought that if you built a generic superintelligence (what I would later label a "ghost of perfect emptiness") then it could, you know, reason about what was likely to be right and wrong; and since it was superintelligent, it was bound to come up with the right answer.

The problem that I somehow managed not to think too hard about, was where the superintelligence would get the procedure that discovered the procedure that discovered the procedure that discovered morality—if I couldn't write it into the start state that wrote the successor AI that wrote the successor AI.

As ...
The Gift We Give To Tomorrow
by Eliezer Yudkowsky — 2008-07-17T06:07:54.000Z

How, oh how, did an unloving and mindless universe, cough up minds who were capable of love?

"No mystery in that," you say, "it's just a matter of natural selection."

But natural selection is cruel, bloody, and bloody stupid. Even when, on the surface of things, biological organisms aren't directly fighting each other—aren't directly tearing at each other with claws—there's still a deeper competition going on between the genes. Genetic information is created when genes increase their relative frequency in the next generation—what matters for "genetic fitness" is not how many children you have, but that you have more children than others. It is quite possible for a species to evolve to extinction, if the winning genes are playing negative-sum games.

How, oh how, could such a process create beings capable of love?

"No mystery," you say, "there is never any mystery-in-the-world; mystery is a property of questions, not answers. A mother's children share her genes, so the mother loves her children."

But sometimes mothers adopt children, and still love them. And mothers love their children for themselves, not for their genes.

"No mystery," you say, "Individual organisms are adaptation-executers, not fitness-maximizers. Evolutionary psychology is not about deliberately maximizing fitness—through most of human history, we didn't know genes existed. We don't calculate our acts' effect on genetic fitness consciously, or even subconsciously."

But human beings form friendships even with non-relatives: how, oh how, can it be?

"No mystery, for hunter-gatherers often play Iterated Prisoner's Dilemmas, the solution to which is reciprocal altruism. Sometimes the most dangerous human in the tribe is not the strongest, the prettiest, or even the smartest, but the one who has the most allies."

Yet not all friends are fair-weather friends; we have a concept of true friendship—and some people have sacrificed their life for their friends. Would not such a devotion tend to remove itself from the gene pool?

"You said it yourself: we have a concept of true friendship and fair-weather friendship. We can tell, or try to tell, the difference between someone who considers us a valuable ally, and someone executing the friendship adaptation. We wouldn't be true friends with someone who we didn't thi...
Whither Moral Progress?
by Eliezer Yudkowsky — 2008-07-16T05:04:42.000Z

Followup to: Is Morality Preference?

In the dialogue "Is Morality Preference?", Obert argues for the existence of moral progress by pointing to free speech, democracy, mass street protests against wars, the end of slavery... and we could also cite female suffrage, or the fact that burning a cat alive was once a popular entertainment... and many other things that our ancestors believed were right, but which we have come to see as wrong, or vice versa.

But Subhan points out that if your only measure of progress is to take a difference against your current state, then you can follow a random walk, and still see the appearance of inevitable progress.

One way of refuting the simplest version of this argument, would be to say that we don't automatically think ourselves the very apex of possible morality; that we can imagine our descendants being more moral than us.

But can you concretely imagine a being morally wiser than yourself—one who knows that some particular thing is wrong, when you believe it to be right?

Certainly: I am not sure of the moral status of chimpanzees, and hence I find it easy to imagine that a future civilization will label them definitely people, and castigate us for failing to cryopreserve the chimpanzees who died in human custody.

Yet this still doesn't prove the existence of moral progress. Maybe I am simply mistaken about the nature of changes in morality that have previously occurred—like looking at a time chart of "differences between past and present", noting that the difference has been steadily decreasing, and saying, without being able to visualize it, "Extrapolating this chart into the future, we find that the future will be even less different from the present than the present."

So let me throw the question open to my readers: Whither moral progress?

You might say, perhaps, "Over time, people have become more willing to help one another—that is the very substance and definition of moral progress."

But as John McCarthy put it:

"If everyone were to live for others all the time, life would be like a procession of ants following each other around in a circle."

Once you make "People helping each other more" the definition of moral progress, then people helping each other all the time, is by definition the apex of moral progress.

At the very least we have Moore's Open Question: It is not clear th...
Lawrence Watt-Evans's Fiction
by Eliezer Yudkowsky — 2008-07-15T03:00:00.000Z

One of my pet topics, on which I will post more one of these days, is the Rationalist in Fiction. Most of the time - it goes almost without saying - the Rationalist is done completely wrong. In Hollywood, the Rationalist is a villain, or a cold emotionless foil, or a child who has to grow into a real human being, or a fool whose probabilities are all wrong, etcetera. Even in science fiction, the Rationalist character is rarely done right - bearing the same resemblance to a real rationalist, as the mad scientist genius inventor who designs a new nuclear reactor in a month, bears to real scientists and engineers.

Perhaps this is because most speculative fiction, generally speaking, is interested in someone battling monsters or falling in love or becoming a vampire, or whatever, not in being rational... and it would probably be worse fiction, if the author tried to make that the whole story. But that can't be the entire problem. I've read at least one author whose plots are not about rationality, but whose characters are nonetheless, in passing, realistically rational.

That author is Lawrence Watt-Evans. His work stands out for a number of reasons, the first being that it is genuinely unpredictable. Not because of a postmodernist contempt for coherence, but because there are events going on outside the hero's story, just like real life.

Most authors, if they set up a fantasy world with a horrible evil villain, and give their main character the one sword that can kill that villain, you could guess that, at the end of the book, the main character is going to kill the evil villain with the sword.

Not Lawrence Watt-Evans. In a Watt-Evans book, it's entirely possible that the evil villain will die of a heart attack halfway through the book, then the character will decide to sell the sword because they'd rather have the money, and then the character uses the money to set up an investment banking company.

That didn't actually happen in any particular Watt-Evans book - I don't believe in spoilers - but it gives you something of the flavor.

And Watt-Evans doesn't always do this, either - just as, even in real life, things sometimes do go as you expect.

It's this strange realism that charms me about Watt-Evans's work. It's not done as a schtick, but as faithfulness-to-reality. Real life doesn't run on perfect rails of ...
Probability is Subjectively Objective
by Eliezer Yudkowsky — 2008-07-14T09:16:50.000Z

Followup to: Probability is in the Mind

"Reality is that which, when you stop believing in it, doesn't go away."
—Philip K. Dick

There are two kinds of Bayesians, allegedly. Subjective Bayesians believe that "probabilities" are degrees of uncertainty existing in our minds; if you are uncertain about a phenomenon, that is a fact about your state of mind, not a property of the phenomenon itself; probability theory constrains the logical coherence of uncertain beliefs. Then there are objective Bayesians, who... I'm not quite sure what it means to be an "objective Bayesian"; there are multiple definitions out there. As best I can tell, an "objective Bayesian" is anyone who uses Bayesian methods and isn't a subjective Bayesian.

If I recall correctly, E. T. Jaynes, master of the art, once described himself as a subjective-objective Bayesian. Jaynes certainly believed very firmly that probability was in the mind; Jaynes was the one who coined the term Mind Projection Fallacy. But Jaynes also didn't think that this implied a license to make up whatever priors you liked. There was only one correct prior distribution to use, given your state of partial information at the start of the problem.

How can something be in the mind, yet still be objective?

It appears to me that a good deal of philosophical maturity consists in being able to keep separate track of nearby concepts, without mixing them up.

For example, to understand evolutionary psychology, you have to keep separate track of the psychological purpose of an act, and the evolutionary pseudo-purposes of the adaptations that execute as the psychology; this is a common failure of newcomers to evolutionary psychology, who read, misunderstand, and thereafter say, "You think you love your children, but you're just trying to maximize your fitness!"

What is it, exactly, that the terms "subjective" and "objective", mean? Let's say that I hand you a sock. Is it a subjective or an objective sock? You believe that 2 + 3 = 5. Is your belief subjective or objective? What about two plus three actually equaling five—is that subjective or objective? What about a specific act of adding two apples and three apples and getting five apples?

I don't intend to confuse you in shrouds of words; but I do mean to point out ...
Rebelling Within Nature
by Eliezer Yudkowsky — 2008-07-13T12:32:30.000Z

Followup to: Fundamental Doubts, Where Recursive Justification Hits Bottom, No Universally Compelling Arguments, Joy in the Merely Real, Evolutionary Psychology

"Let us understand, once and for all, that the ethical progress of society depends, not on imitating the cosmic process, still less in running away from it, but in combating it."
—T. H. Huxley ("Darwin's bulldog", early advocate of evolutionary theory)

There is a quote from some Zen Master or other, who said something along the lines of:

"Western man believes that he is rebelling against nature, but he does not realize that, in doing so, he is acting according to nature."

The Reductionist Masters of the West, strong in their own Art, are not so foolish; they do realize that they always act within Nature.

You can narrow your focus and rebel against a facet of existing Nature—polio, say—but in so doing, you act within the whole of Nature. The syringe that carries the polio vaccine is forged of atoms; our minds, that understood the method, embodied in neurons. If Jonas Salk had to fight laziness, he fought something that evolution instilled in him—a reluctance to work that conserves energy. And he fought it with other emotions that natural selection also inscribed in him: feelings of friendship that he extended to humanity, heroism to protect his tribe, maybe an explicit desire for fame that he never acknowledged to himself—who knows? (I haven't actually read a biography of Salk.)

The point is, you can't fight Nature from beyond Nature, only from within it. There is no acausal fulcrum on which to stand outside reality and move it. There is no ghost of perfect emptiness by which you can judge your brain from outside your brain. You can fight the cosmic process, but only by recruiting other abilities that evolution originally gave to you.

And if you fight one emotion within yourself—looking upon your own nature, and judging yourself less than you think should be—saying perhaps, "I should not want to kill my enemies"—then you make that judgment, by...

How exactly does one go about rebelling against one's own goal system?

From within it, naturally.

This is perhaps the primary thing that I didn't quite understand as a teenager.

At the age of fifteen (fourteen?), I picked up a copy...