trentmkelly/LessWrong-43k huggingface dataset:
Fundamental Doubts
by Eliezer Yudkowsky — 2008-07-12T05:21:12.000Z

Followup to: The Genetic Fallacy, Where Recursive Justification Hits Bottom

Yesterday I said that—because humans are not perfect Bayesians—the genetic fallacy is not entirely a fallacy; when new suspicion is cast on one of your fundamental sources, you really should doubt all the branches and leaves of that root, even if they seem to have accumulated new evidence in the meanwhile.

This is one of the most difficult techniques of rationality (on which I will separately post, one of these days). Descartes, setting out to "doubt, insofar as possible, all things", ended up trying to prove the existence of God—which, if he wasn't a secret atheist trying to avoid getting burned at the stake, is pretty pathetic. It is hard to doubt an idea to which we are deeply attached; our mind naturally reaches for cached thoughts and rehearsed arguments.

But today's post concerns a different kind of difficulty—the case where the doubt is so deep, of a source so fundamental, that you can't make a true fresh beginning.

Case in point: Remember when, in the The Matrix, Morpheus told Neo that the machines were harvesting the body heat of humans for energy, and liquefying the dead to feed to babies? I suppose you thought something like, "Hey! That violates the second law of thermodynamics."

Well, it does violate the second law of thermodynamics. But if the Matrix's makers had cared about the flaw once it was pointed out to them, they could have fixed the plot hole in any of the sequels, in fifteen seconds, this easily:

Neo: "Doesn't harvesting human body heat for energy, violate the laws of thermodynamics?"

Morpheus: "Where'd you learn about thermodynamics, Neo?"

Neo: "In school."

Morpheus: "Where'd you go to school, Neo?"

Neo: "Oh."

Morpheus: "The machines tell elegant lies."

Now, mind you, I am not saying that this excuses the original mistake in the script. When my mind generated this excuse, it came clearly labeled with that warning sign of which I have spoken, "Tada! Your mind can generate an excuse for anything!" You do not need to tell me that my plot-hole-patch is a nitwit idea, I am well aware of that...

...but, in point of fact, if you woke up out of a virtual reality pod one day, you would have to suspect all the physics you knew. Even if you looked down and...
The Genetic Fallacy
by Eliezer Yudkowsky — 2008-07-11T05:47:28.000Z

In lists of logical fallacies, you will find included “the genetic fallacy”—the fallacy of attacking a belief based on someone’s causes for believing it.

This is, at first sight, a very strange idea—if the causes of a belief do not determine its systematic reliability, what does? If Deep Blue advises us of a chess move, we trust it based on our understanding of the code that searches the game tree, being unable to evaluate the actual game tree ourselves. What could license any probability assignment as “rational,” except that it was produced by some systematically reliable process?

Articles on the genetic fallacy will tell you that genetic reasoning is not always a fallacy—that the origin of evidence can be relevant to its evaluation, as in the case of a trusted expert. But other times, say the articles, it is a fallacy; the chemist Kekulé first saw the ring structure of benzene in a dream, but this doesn’t mean we can never trust this belief.

So sometimes the genetic fallacy is a fallacy, and sometimes it’s not?

The genetic fallacy is formally a fallacy, because the original cause of a belief is not the same as its current justificational status, the sum of all the support and antisupport currently known.

Yet we change our minds less often than we think. Genetic accusations have a force among humans that they would not have among ideal Bayesians.

Clearing your mind is a powerful heuristic when you’re faced with new suspicion that many of your ideas may have come from a flawed source.

Once an idea gets into our heads, it’s not always easy for evidence to root it out. Consider all the people out there who grew up believing in the Bible; later came to reject (on a deliberate level) the idea that the Bible was written by the hand of God; and who nonetheless think that the Bible is full of indispensable ethical wisdom. They have failed to clear their minds; they could do significantly better by doubting anything the Bible said because the Bible said it.

At the same time, they would have to bear firmly in mind the principle that reversed stupidity is not intelligence; the goal is to genuinely shake your mind loose and do independent thinking, not to negate the Bible and let that be your algorithm.

Once an idea gets into your head, you tend to find support for it everywhere you look—and so when the original source is suddenly cast into suspicion, you would be very wise ind...
My Kind of Reflection
by Eliezer Yudkowsky — 2008-07-10T07:21:28.000Z

In "Where Recursive Justification Hits Bottom", I concluded that it's okay to use induction to reason about the probability that induction will work in the future, given that it's worked in the past; or to use Occam's Razor to conclude that the simplest explanation for why Occam's Razor works is that the universe itself is fundamentally simple.

Now I am far from the first person to consider reflective application of reasoning principles. Chris Hibbert compared my view to Bartley's Pan-Critical Rationalism (I was wondering whether that would happen). So it seems worthwhile to state what I see as the distinguishing features of my view of reflection, which may or may not happen to be shared by any other philosopher's view of reflection.

• All of my philosophy here actually comes from trying to figure out how to build a self-modifying AI that applies its own reasoning principles to itself in the process of rewriting its own source code. So whenever I talk about using induction to license induction, I'm really thinking about an inductive AI considering a rewrite of the part of itself that performs induction. If you wouldn't want the AI to rewrite its source code to not use induction, your philosophy had better not label induction as unjustifiable.

• One of the most powerful general principles I know for AI in general, is that the true Way generally turns out to be naturalistic—which for reflective reasoning, means treating transistors inside the AI, just as if they were transistors found in the environment; not an ad-hoc special case. This is the real source of my insistence in "Recursive Justification" that questions like "How well does my version of Occam's Razor work?" should be considered just like an ordinary question—or at least an ordinary very deep question. I strongly suspect that a correctly built AI, in pondering modifications to the part of its source code that implements Occamian reasoning, will not have to do anything special as it ponders—in particular, it shouldn't have to make a special effort to avoid using Occamian reasoning.

• I don't think that "reflective coherence" or "reflective consistency" should be considered as a desideratum in itself. As I said in the Twelve Virtues and the Simple Truth, if you make five accurate maps of the same city, then the maps will necessaril...
The Fear of Common Knowledge
by Eliezer Yudkowsky — 2008-07-09T09:48:35.000Z

Followup to: Belief in Belief

One of those insights that made me sit upright and say "Aha!" From The Uncredible Hallq:

Minor acts of dishonesty are integral to human life, ranging from how
we deal with casual acquaintances to writing formal agreements between
nation states. Steven Pinker has an excellent chapter on this in The
Stuff of Thought, a version of which can be found at TIME magazine’s website.
What didn’t make it into the TIME version is Pinker’s proposal that,
while there are several reasons we do this, the most important reason
is to avoid mutual knowledge: "She probably knows I just blew a pass at
her, but does she know I know she knows? Does she know I know she knows
I know she knows?" Etc. Mutual knowledge is that nightmare where, for
all intents and purposes, the known-knows can be extended out to
infinity. The ultimate example of this has to be the joke "No, it
wasn’t awkward until you said, 'well, this is awkward.'" A situation
might be a little awkward, but what’s really awkward is mutual
knowledge, created when someone blurts out what’s going on for all to
hear...

The story of the Emperor’s New Clothes is another example of the power of mutual knowledge...

The power of real deception - outright lies - is easy for even us nerds to understand.

The notion of a lie that the other person knows is a lie, seems very odd at first. Up until I read the Hallq's explanation of Pinker, I had thought in terms of
people suppressing uncomfortable thoughts: "If it isn't said out loud, I don't have to deal with it."

Like the friends of a terminal patient, whose disease has progressed to
a stage that - if you look it up online - turns out to be nearly
universally fatal. So the friends gather around, and wish the patient
best hopes for their medical treatment. No one says, "Well, we all
know you're going to die; and now it's too late for you to get life
insurance and sign up for cryonics. I hope it isn't too painful; let me know if you want me to smuggle you a heroin overdose."

So even that is possible for a nerd to understand - in terms of, as Vassar puts it, thinking of non-nerds as defective nerds...

But the notion of a lie that the other person knows is a lie, but they aren't sure that you
know they know it's a lie, and so the social...
Where Recursive Justification Hits Bottom
by Eliezer Yudkowsky — 2008-07-08T10:16:45.000Z

Why do I believe that the Sun will rise tomorrow?

Because I've seen the Sun rise on thousands of previous days.

Ah... but why do I believe the future will be like the past?

Even if I go past the mere surface observation of the Sun rising, to the apparently universal and exceptionless laws of gravitation and nuclear physics, then I am still left with the question: "Why do I believe this will also be true tomorrow?"

I could appeal to Occam's Razor, the principle of using the simplest theory that fits the facts... but why believe in Occam's Razor? Because it's been successful on past problems? But who says that this means Occam's Razor will work tomorrow?

And lo, the one said:

"Science also depends on unjustified assumptions. Thus science is ultimately based on faith, so don't you criticize me for believing in [silly-belief-#238721]."

As I've previously observed:

It's a most peculiar psychology—this business of "Science is based on faith too, so there!" Typically this is said by people who claim that faith is a good thing. Then why do they say "Science is based on faith too!" in that angry-triumphal tone, rather than as a compliment?

Arguing that you should be immune to criticism is rarely a good sign.

But this doesn't answer the legitimate philosophical dilemma: If every belief must be justified, and those justifications in turn must be justified, then how is the infinite recursion terminated?

And if you're allowed to end in something assumed-without-justification, then why aren't you allowed to assume anything without justification?

A similar critique is sometimes leveled against Bayesianism—that it requires assuming some prior—by people who apparently think that the problem of induction is a particular problem of Bayesianism, which you can avoid by using classical statistics. I will speak of this later, perhaps.

But first, let it be clearly admitted that the rules of Bayesian updating, do not of themselves solve the problem of induction.

Suppose you're drawing red and white balls from an urn. You observe that, of the first 9 balls, 3 are red and 6 are white. What is the probability that the next ball drawn will be red?

That depends on your prior beliefs about the urn. If you think the urn-maker generated a uniform random number between 0 and 1, and used that number as the fixe...
Will As Thou Wilt
by Eliezer Yudkowsky — 2008-07-07T10:37:11.000Z

Followup to: Possibility and Could-ness

Arthur Schopenhauer (1788-1860) said:

"A man can do as he wills, but not will as he wills."

For this fascinating sentence, I immediately saw two interpretations; and then, after some further thought, two more interpretations.

On the first interpretation, Schopenhauer forbids us to build circular causal models of human psychology. The explanation for someone's current will cannot be their current will - though it can include their past will.

On the second interpretation, the sentence says that alternate choices are not reachable - that we couldn't have taken other options even "if we had wanted to do so".

On the third interpretation, the sentence says that we cannot control our own desires - that we are the prisoners of our own passions, even when we struggle against them.

On the fourth interpretation, the sentence says that we cannot control our own desires, because our desires themselves will determine which desires we want, and so protect themselves.

I count two true interpretations and two false interpretations. How about you?
Is Morality Given?
by Eliezer Yudkowsky — 2008-07-06T08:12:26.000Z

Continuation of: Is Morality Preference?

(Disclaimer: Neither Subhan nor Obert represent my own position on morality; rather they represent different sides of the questions I hope to answer.)

Subhan: "What is this 'morality' stuff, if it is not a preference within you?"

Obert: "I know that my mere wants, don't change what is right; but I don't claim to have absolute knowledge of what is right—"

Subhan: "You're not escaping that easily! How does a universe in which murder is wrong, differ from a universe in which murder is right? How can you detect the difference experimentally? If the answer to that is 'No', then how does any human being come to know that murder is wrong?"

Obert: "Am I allowed to say 'I don't know'?"

Subhan: "No. You believe now that murder is wrong. You must believe you already have evidence and you should be able to present it now."

Obert: "That's too strict! It's like saying to a hunter-gatherer, 'Why is the sky blue?' and expecting an immediate answer."

Subhan: "No, it's like saying to a hunter-gatherer: Why do you believe the sky is blue?"

Obert: "Because it seems blue, just as murder seems wrong. Just don't ask me what the sky is, or how I can see it."

Subhan: "But—aren't we discussing the nature of morality?"

Obert: "That, I confess, is not one of my strong points. I specialize in plain old morality. And as a matter of morality, I know that I can't make murder right just by wanting to kill someone."

Subhan: "But if you wanted to kill someone, you would say, 'I know murdering this guy is right, and I couldn't make it wrong just by not wanting to do it.'"

Obert: "Then, if I said that, I would be wrong. That's common moral sense, right?"

Subhan: "Argh! It's difficult to even argue with you, since you won't tell me exactly what you think morality is made of, or where you're getting all these amazing moral truths—"

Obert: "Well, I do regret having to frustrate you. But it's more important that I act morally, than that I come up with amazing new theories of the nature of morality. I don't claim that my strong point is in explaining the fundamental nature of morality. Rather, my strong point is coming up with theories of morality that give normal moral ans...
Is Morality Preference?
by Eliezer Yudkowsky — 2008-07-05T00:55:25.000Z

Followup to: Moral Complexities

In the dialogue "The Bedrock of Fairness", I intended Yancy to represent morality-as-raw-fact, Zaire to represent morality-as-raw-whim, and Xannon to be a particular kind of attempt at compromising between them. Neither Xannon, Yancy, or Zaire represent my own views—rather they are, in their disagreement, showing the problem that I am trying to solve. It is futile to present answers to which questions are lacking.

But characters have independent life in the minds of all readers; when I create a dialogue, I don't view my authorial intent as primary. Any good interpretation can be discussed. I meant Zaire to be asking for half the pie out of pure selfishness; many readers interpreted this as a genuine need... which is as interesting a discussion to have as any, though it's a different discussion.

With this in mind, I turn to Subhan and Obert, who shall try to answer yesterday's questions on behalf of their respective viewpoints.

Subhan makes the opening statement:

Subhan: "I defend this proposition: that there is no reason to talk about a 'morality' distinct from what people want."

Obert: "I challenge. Suppose someone comes to me and says, 'I want a slice of that pie you're holding.' It seems to me that they have just made a very different statement from 'It is right that I should get a slice of that pie'. I have no reason at all to doubt the former statement—to suppose that they are lying to me about their desires. But when it comes to the latter proposition, I have reason indeed to be skeptical. Do you say that these two statements mean the same thing?"

Subhan: "I suggest that when the pie-requester says to you, 'It is right for me to get some pie', this asserts that you want the pie-requester to get a slice."

Obert: "Why should I need to be told what I want?"

Subhan: "You take a needlessly restrictive view of wanting, Obert; I am not setting out to reduce humans to creatures of animal instinct. Your wants include those desires you label 'moral values', such as wanting the hungry to be fed—"

Obert: "And you see no distinction between my desire to feed the hungry, and my desire to eat all the delicious pie myself?"

Subhan: "No! They are both desires—backed by different emotions, perhaps, but both desire...
Moral Complexities
by Eliezer Yudkowsky — 2008-07-04T06:43:52.000Z

Followup to: The Bedrock of Fairness

Discussions of morality seem to me to often end up turning around two different intuitions, which I might label morality-as-preference and morality-as-given. The former crowd tends to equate morality with what people want; the latter to regard morality as something you can't change by changing people.

As for me, I have my own notions, which I am working up to presenting. But above all, I try to avoid avoiding difficult questions. Here are what I see as (some of) the difficult questions for the two intuitions:

For morality-as-preference:

Why do people seem to mean different things by "I want the pie" and "It is right that I should get the pie"? Why are the two propositions argued in different ways?

When and why do people change their terminal values? Do the concepts of "moral error" and "moral progress" have referents? Why would anyone want to change what they want?

Why and how does anyone ever "do something they know they shouldn't", or "want something they know is wrong"? Does the notion of morality-as-preference really add up to moral normality?

For morality-as-given:

Would it be possible for everyone in the world to be wrong about morality, and wrong about how to update their beliefs about morality, and wrong about how to choose between metamoralities, etcetera? So that there would be a morality, but it would be entirely outside our frame of reference? What distinguishes this state of affairs, from finding a random stone tablet showing the words "You should commit suicide"?

How does a world in which a moral proposition is true, differ from a world in which that moral proposition is false? If the answer is "no", how does anyone perceive moral givens?

Is it better for people to be happy than sad? If so, why does morality look amazingly like godshatter of natural selection?

Am I not allowed to construct an alien mind that evaluates morality differently? What will stop me from doing so?

Part of The Metaethics Sequence

Next post: "Is Morality Preference?"

Previous post: "The Bedrock of Fairness"
2 of 10, not 3 total
by Eliezer Yudkowsky — 2008-07-04T01:10:29.000Z

There is no rule against commenting more than 3 times in a thread. Sorry if anyone has gotten this impression.

However, among the 10 "Recent Comments" visible in the sidebar at right, usually no more than 2, rarely 3, and never 4, should be yours. This is meant to ensure no one person dominates a thread; it gives others a chance to respond to others' responses. One-line comments that quickly correct an error may be common-sensically excepted from this rule.

You need not refrain from commenting, just wait a bit.
The Bedrock of Fairness
by Eliezer Yudkowsky — 2008-07-03T06:00:14.000Z

Followup to: The Moral Void

Three people, whom we'll call Xannon, Yancy and Zaire, are separately wandering through the forest; by chance, they happen upon a clearing, meeting each other. Introductions are performed. And then they discover, in the center of the clearing, a delicious blueberry pie.

Xannon: "A pie! What good fortune! But which of us should get it?"

Yancy: "Let us divide it fairly."

Zaire: "I agree; let the pie be distributed fairly. Who could argue against fairness?"

Xannon: "So we are agreed, then. But what is a fair division?"

Yancy: "Eh? Three equal parts, of course!"

Zaire: "Nonsense! A fair distribution is half for me, and a quarter apiece for the two of you."

Yancy: "What? How is that fair?"

Zaire: "I'm hungry, therefore I should be fed; that is fair."

Xannon: "Oh, dear. It seems we have a dispute as to what is fair. For myself, I want to divide the pie the same way as Yancy. But let us resolve this dispute over the meaning of fairness, fairly: that is, giving equal weight to each of our desires. Zaire desires the pie to be divided {1/4, 1/4, 1/2}, and Yancy and I desire the pie to be divided {1/3, 1/3, 1/3}. So the fair compromise is {11/36, 11/36, 14/36}."

Zaire: "What? That's crazy. There's two different opinions as to how fairness works—why should the opinion that happens to be yours, get twice as much weight as the opinion that happens to be mine? Do you think your theory is twice as good? I think my theory is a hundred times as good as yours! So there!"

Yancy: "Craziness indeed. Xannon, I already took Zaire's desires into account in saying that he should get 1/3 of the pie. You can't count the same factor twice. Even if we count fairness as an inherent desire, why should Zaire be rewarded for being selfish? Think about which agents thrive under your system!"

Xannon: "Alas! I was hoping that, even if we could not agree on how to distribute the pie, we could agree on a fair resolution procedure for our dispute, such as averaging our desires together. But even that hope was dashed. Now what are we to do?"

Yancy: "Xannon, you are overcomplicating things. 1/3 apiece. It's not that complicated.&nbs...
I'd take it
by Eliezer Yudkowsky — 2008-07-02T07:57:35.000Z

Out-of-context quote of the day:

"...although even $10 trillion isn't a huge amount of money..."

From Simon Johnson, Director of the IMF's Research Department, on "The Rise of Sovereign Wealth Funds".

So if you had $10 trillion, what would you do with it?