trentmkelly/LessWrong-43k huggingface dataset:
You Provably Can't Trust Yourself
by Eliezer Yudkowsky — 2008-08-19T20:35:47.000Z

Followup to: Where Recursive Justification Hits Bottom, Löb's Theorem

Peano Arithmetic seems pretty trustworthy. We've never found a case where Peano Arithmetic proves a theorem T, and yet T is false in the natural numbers. That is, we know of no case where []T ("T is provable in PA") and yet ~T ("not T").

We also know of no case where first order logic is invalid: We know of no case where first-order logic produces false conclusions from true premises. (Whenever first-order statements H are true of a model, and we can syntactically deduce C from H, checking C against the model shows that C is also true.)

Combining these two observations, it seems like we should be able to get away with adding a rule to Peano Arithmetic that says:

All T: ([]T -> T)

But Löb's Theorem seems to show that as soon as we do that, everything becomes provable. What went wrong? How can we do worse by adding a true premise to a trustworthy theory? Is the premise not true—does PA prove some theorems that are false? Is first-order logic not valid—does it sometimes prove false conclusions from true premises?

Actually, there's nothing wrong with reasoning from the axioms of Peano Arithmetic plus the axiom schema "Anything provable in Peano Arithmetic is true." But the result is a different system from PA, which we might call PA+1. PA+1 does not reason from identical premises to PA; something new has been added. So we can evade Löb's Theorem because PA+1 is not trusting itself—it is only trusting PA.

If you are not previously familiar with mathematical logic, you might be tempted to say, "Bah! Of course PA+1 is trusting itself! PA+1 just isn't willing to admit it! Peano Arithmetic already believes anything provable in Peano Arithmetic—it will already output anything provable in Peano Arithmetic as a theorem, by definition! How does moving to PA+1 change anything, then? PA+1 is just the same system as PA, and so by trusting PA, PA+1 is really trusting itself. Maybe that dances around some obscure mathematical problem with direct self-reference, but it doesn't evade the charge of self-trust."

But PA+1 and PA really are different systems; in PA+1 it is possible to prove true statements about the natural numbers that are not provable in PA. If you're familiar with m...
Dumb Deplaning
by Eliezer Yudkowsky — 2008-08-18T23:49:39.000Z

So I just traveled to Portsmouth, VA for an experimental conference - in the sense that I don't expect conferences of this type to prove productive, but maybe I should try at least once - in the unlikely event that there are any local Overcoming Bias readers who want to drive out to Portsmouth for a meeting on say the evening of the 20th, email me - anyway, I am struck, for the Nth time, how uncooperative people are in getting off planes.

Most people, as soon as they have a chance to make for the exit, do so - even if they need to take down luggage first. At any given time after the initial rush to the aisles, usually a single person is taking down luggage, while the whole line behind them waits. Then the line moves forward a little and the next person starts taking down their luggage.

In programming we call this a "greedy local algorithm". But since everyone does it, no one seems to feel "greedy".

How would I do it? Off the top of my head:

"Left aisle seats, please rise and move to your luggage. (Pause.) Left aisle seats, please retrieve your luggage. (Pause.) Left aisle seats, please deplane. (Pause.) Right aisle seats, please rise and move to your luggage..."

There are numerous other minor tweaks that this suggests, like seating people with tight connections near the front left aisle, or boarding passengers with window seats before passengers with middle and aisle seats.

But the main thing that strikes me is twofold:

First, everyone who stops to take down their luggage while everyone waits behind them - as opposed to waiting to rise until the aisle is clear - is playing a negative-sum game; the benefit to themselves is smaller than the total cost to all the others waiting in line.

Second, the airline has a motive to clear passengers quickly to reduce turnaround time. But the airline does not regulate the deplaning process. Even though it would be straightforward - defectors being readily spotted - and I don't even see why it would be resented.

Am I missing something? Is there some mysterious Freakonomics-style explanation for this?

Heck, people usually manage to regulate themselves on worse cases than this. Most of the people blocking the aisle wouldn't walk away with someone else's purse. Are we just stuck in an equilibrium of mutual defection?&n...
The Cartoon Guide to Löb's Theorem
by Eliezer Yudkowsky — 2008-08-17T20:35:45.000Z

Lo! A cartoon proof of Löb's Theorem!

Löb's Theorem shows that a mathematical system cannot assert its own soundness without becoming inconsistent. Marcello and I wanted to be able to see the truth of Löb's Theorem at a glance, so we doodled it out in the form of a cartoon. (An inability to trust assertions made by a proof system isomorphic to yourself, may be an issue for self-modifying AIs.)

It was while learning mathematical logic that I first learned to rigorously distinguish between X, the truth of X, the quotation of X, a proof of X, and a proof that X's quotation was provable.

The cartoon guide follows as an embedded Scribd document after the jump, or you can download as a PDF file. Afterward I offer a medium-hard puzzle to test your skill at drawing logical distinctions.

Cartoon Guide to Löb's ... by on Scribd

Cartoon Guide to Löb's Theorem - Upload a Document to Scribd

Read this document on Scribd: Cartoon Guide to Löb's Theorem

And now for your medium-hard puzzle:

The Deduction Theorem (look it up) states that whenever assuming a hypothesis H enables us to prove a formula F in classical logic, then (H->F) is a theorem in classical logic.

Let ◻Z stand for the proposition "Z is provable". Löb's Theorem shows that, whenever we have ((◻C)->C), we can prove C.

Applying the Deduction Theorem to Löb's Theorem gives us, for all C:

((◻C)->C)->C

However, those familiar with the logic of material implication will realize that:

(X->Y)->Y
implies
(not X)->Y

Applied to the above, this yields (not ◻C)->C.

That is, all statements which lack proofs are true.

I cannot prove that 2 = 1.

Therefore 2 = 1.

Can you exactly pinpoint the flaw?
When Anthropomorphism Became Stupid
by Eliezer Yudkowsky — 2008-08-16T23:43:01.000Z

It turns out that most things in the universe don't have minds.

This statement would have provoked incredulity among many earlier cultures. "Animism" is the usual term. They thought that trees, rocks, streams, and hills all had spirits because, hey, why not?

I mean, those lumps of flesh known as "humans" contain thoughts, so why shouldn't the lumps of wood known as "trees"?

My muscles move at my will, and water flows through a river. Who's to say that the river doesn't have a will to move the water? The river overflows its banks, and floods my tribe's gathering-place - why not think that the river was angry, since it moved its parts to hurt us? It's what we would think when someone's fist hit our nose.

There is no obvious reason - no reason obvious to a hunter-gatherer - why this cannot be so. It only seems like a stupid mistake if you confuse weirdness with stupidity. Naturally the belief that rivers have animating spirits seems "weird" to us, since it is not a belief of our tribe. But there is nothing obviously stupid about thinking that great lumps of moving water have spirits, just like our own lumps of moving flesh.

If the idea were obviously stupid, no one would have believed it. Just like, for the longest time, nobody believed in the obviously stupid idea that the Earth moves while seeming motionless.

Is it obvious that trees can't think? Trees, let us not forget, are in fact our distant cousins. Go far enough back, and you have a common ancestor with your fern. If lumps of flesh can think, why not lumps of wood?

For it to be obvious that wood doesn't think, you have to belong to a culture with microscopes. Not just any microscopes, but really good microscopes.

Aristotle thought the brain was an organ for cooling the blood. (It's a good thing that what we believe about our brains has very little effect on their actual operation.)

Egyptians threw the brain away during the process of mummification.

Alcmaeon of Croton, a Pythagorean of the 5th century BCE, put his finger on the brain as the seat of intelligence, because he'd traced the optic nerve from the eye to the brain. Still, with the amount of evidence he had, it was only a guess.

When did the central role of the brain stop being a guess? I do not know enough history to answer this question, and probably there wasn't any sharp...
Is valuing life undervaluing it?
by KatjaGrace — 2008-08-16T23:22:00.000Z

People often object to human life having a value placed on it, explicitly or implicitly. (I’m told there are good reasons, apparently to do with dignity, compassion, holism, souls and me being sick and inhuman, but I must admit they seem incoherent to me – if anyone would like to explain in writing I would be grateful). The pressing question then:

What alternatives are there to placing a finite value on human life?

One could not value human life at all. Ironically, this is what those who try to put a value on it, or assume it has one, are generally suspected of. Any value they give a life can be equated to the value of, say, a really vast number of rolls of toilet paper. Or heaps of SUVs full of McDonalds’ hamburgers and books by Ann Coulter. Thus it’s not enough; if you can put a value on human life you don’t value human life.

The other alternative looks better: value human life infinitely. There’s probably still conflict with your intuitive morality however. If any amount of human life is infinitely valuable, as long as someone is alive the universe can’t get any better. Why preserve extra lives?

Infinitely valuable human lives should also be protected from anything that might shorten them for lesser aims, such as life. We barter slight risks constantly for the quality of our experience, among other things. Unless you’d like to argue that nachos and car trips are also of infinite value, so can be traded with smidgens of human life, what are you doing out of your protective bubble?

Another alternative is just to not think about it. Hold that lives have a high but finite value, but don’t use this in naughty calculative attempts to maximise welfare! Maintain that it is abhorrent to do so. Uphold lots of arbitrary rules, like respecting people’s dignity and beginning charity at home and having honour and being respectable and doing what your heart tells you. Interestingly, this effectively does make human life worthless; not even worth including in the calculation next to the whims of your personal emotions and the culture at hand.

The only way to value human life is to place value on human life.

For more on how to feel about this, see Philip Tetlock.
Processing people
by KatjaGrace — 2008-08-16T02:34:00.000Z

Some of my friends think that a random process of deciding who should live or die is more important than the lives of those people, because lives should all be valued equally (and a process can ensure approximately random choice).

For example, this would mean it is better to make sure the life rafts are filled by a random selection of women and men and rich and poor and so on, even if that means that half of them drown while you flip the coin.

If lives should be valued equally, then why is a process of choosing between identically valuable things worth more than even one human life?

Also, even if you value this process more than another person’s life, why shouldn’t the person who’s life is at stake’s opinion on their relative value come into it? That is, if we are attempting to follow any ethical system other than egoism (of course your preference is of absolute importance if you are trying to be purely self interested). Try out the veil of ignorance!

For other readers, no this isn’t a purely theoretical debate, I’m just not going to tell you what the context is.
Hot Air Doesn't Disagree
by Eliezer Yudkowsky — 2008-08-16T00:42:02.000Z

Followup to: The Bedrock of Morality, Abstracted Idealized Dynamics

Tim Tyler comments:

Do the fox and the rabbit
disagree? It seems reasonable so say that they do if they meet: the
rabbit thinks it should be eating grass, and the fox thinks the rabbit
should be in the fox's stomach. They may argue passionately about the
rabbit's fate - and even stoop to violence.

Boy, you know, when you think about it, Nature turns out to be just full of disagreement.

Rocks, for example, fall down - so they agree with us, who also fall when pushed off a cliff - whereas hot air rises into the air, unlike humans.

I wonder why hot air disagrees with us so dramatically. I wonder what sort of moral justifications it might have for behaving as it does; and how long it will take to argue this out. So far, hot air has not been forthcoming in terms of moral justifications.

Physical systems that behave differently from you usually do not have factual or moral disagreements with you. Only a highly specialized subset of systems, when they do something different from you, should lead you to infer their explicit internal representation of moral arguments that could potentially lead you to change your mind about what you should do.

Attributing moral disagreements to rabbits or foxes is sheer anthropomorphism, in the full technical sense of the term - like supposing that lightning bolts are thrown by thunder gods, or that trees have spirits that can be insulted by human sexual practices and lead them to withhold their fruit.

The rabbit does not think it should be eating grass. If questioned the rabbit will not say, "I enjoy eating grass, and it is good in general for agents to do what they enjoy, therefore I should eat grass." Now you might invent an argument like that; but the rabbit's actual behavior has absolutely no causal connection to any cognitive system that processes such arguments. The fact that the rabbit eats grass, should not lead you to infer the explicit cognitive representation of, nor even infer the probable theoretical existence of, the sort of arguments that humans have over what they should do. The rabbit is just eating grass, like a rock rolls downhill and like hot air rises.

To think that the rabbit contains a little circuit that ponders morality and then finally does what it thinks it should do, and that the rabbit has...
The Bedrock of Morality: Arbitrary?
by Eliezer Yudkowsky — 2008-08-14T22:00:57.000Z

Followup to: Is Fairness Arbitrary?, Joy in the Merely Good, Sorting Pebbles Into Correct Heaps

Yesterday, I presented the idea that when only five people are present, having just stumbled across a pie in the woods (a naturally growing pie, that just popped out of the ground) then it is fair to give Dennis only 1/5th of this pie, even if Dennis persistently claims that it is fair for him to get the whole thing. Furthermore, it is meta-fair to follow such a symmetrical division procedure, even if Dennis insists that he ought to dictate the division procedure.

Fair, meta-fair, or meta-meta-fair, there is no level of fairness where you're obliged to concede everything to Dennis, without reciprocation or compensation, just because he demands it.

Which goes to say that fairness has a meaning beyond which "that which everyone can be convinced is 'fair'". This is an empty proposition, isomorphic to "Xyblz is that which everyone can be convinced is 'xyblz'". There must be some specific thing of which people are being convinced; and once you identify that thing, it has a meaning beyond agreements and convincing.

You're not introducing something arbitrary, something un-fair, in refusing to concede everything to Dennis. You are being fair, and meta-fair and meta-meta-fair. As far up as you go, there's no level that calls for unconditional surrender. The stars do not judge between you and Dennis—but it is baked into the very question that is asked, when you ask, "What is fair?" as opposed to "What is xyblz?"

Ah, but why should you be fair, rather than xyblz? Let us concede that Dennis cannot validly persuade us, on any level, that it is fair for him to dictate terms and give himself the whole pie; but perhaps he could argue whether we should be fair?

The hidden agenda of the whole discussion of fairness, of course, is that good-ness and right-ness and should-ness, ground out similarly to fairness.

Natural selection optimizes for inclusive genetic fitness. This is not a disagreement with humans about what is good. It is simply that natural selection does not do what is good: it optimizes for inclusive genetic fitness.

Well, since some optimization processes optimize for inclusive genetic fitness, instead of what is good, which should we do, ourselves?

I know my answer to this question. It has something ...
Is Fairness Arbitrary?
by Eliezer Yudkowsky — 2008-08-14T01:54:59.000Z

Followup to: The Bedrock of Fairness

In "The Bedrock of Fairness", Xannon, Yancy, and Zaire argue over how to split up a pie that they found in the woods. Yancy thinks that 1/3 each is fair; Zaire demands half; and Xannon tries to compromise.

Dividing a pie fairly isn't as trivial a problem as it may sound. What if people have different preferences for crust, filling, and topping? Should they each start with a third, and trade voluntarily? But then they have conflicts of interest over how to divide the surplus utility generated by trading...

But I would say that "half for Zaire" surely isn't fair.

I confess that I originally wrote Zaire as a foil—this is clearer in an earlier version of the dialog, where Zaire, named Dennis, demands the whole pie—and was surprised to find some of my readers taking Zaire's claim seriously, perhaps because I had Zaire say "I'm hungry."

Well, okay; I believe that when I write a dialogue, the reader has a right to their own interpretation. But I did intend that dialogue to illustrate a particular point:

You can argue about how to divide up the pie, or even argue how to argue about dividing up the pie, you can argue over what is fair... but there finally comes a point when you hit bedrock. If Dennis says, "No, the fair way to argue is that I get to dictate everything, and I now hereby dictate that I get the whole pie," there's nothing left to say but "Sorry, that's just not what fairness is—you can try to take the pie and I can try to stop you, but you can't convince that that is fair."

A "fair division" is not the same as "a division that compels everyone to admit that the division is fair". Dennis can always just refuse to agree, after all.

But more to the point, when you encounter a pie in the forest, in the company of friends, and you try to be fair, there's a certain particular thing you're trying to do—the term "fair" is not perfectly empty, it cannot attach to just anything. Metaphorically speaking, "fair" is not a hypothesis equally compatible with any outcome.

Fairness expresses notions of concern for the other agents who also want the pie; a goal to take their goals into account. It's a separate question whether that concern is pure altruism, or not wanting to make them angry enough to fight. Fairness expresses notions of symmetry, equal treatment—whic...
"Arbitrary"
by Eliezer Yudkowsky — 2008-08-12T17:55:22.000Z

Followup to: Inseparably Right; or, Joy in the Merely Good, Sorting Pebbles Into Correct Heaps

One of the experiences of following the Way is that, from time to time, you notice a new word that you have been using without really understanding. And you say: "What does this word, 'X', really mean?"

Perhaps 'X' is 'error', for example. And those who have not yet realized the importance of this aspect of the Way, may reply: "Huh? What do you mean? Everyone knows what an 'error' is; it's when you get something wrong, when you make a mistake." And you reply, "But those are only synonyms; what can the term 'error' mean in a universe where particles only ever do what they do?"

It's not meant to be a rhetorical question; you're meant to go out and answer it. One of the primary tools for doing so is Rationalist's Taboo, when you try to speak without using the word or its synonyms—to replace the symbol with the substance.

So I ask you therefore, what is this word "arbitrary"? Is a rock arbitrary? A leaf? A human?

How about sorting pebbles into prime-numbered heaps? How about maximizing inclusive genetic fitness? How about dragging a child off the train tracks?

How can I tell exactly which things are arbitrary, and which not, in this universe where particles only ever do what they do? Can you tell me exactly what property is being discriminated, without using the word "arbitrary" or any direct synonyms? Can you open up the box of "arbitrary", this label that your mind assigns to some things and not others, and tell me what kind of algorithm is at work here?

Having pondered this issue myself, I offer to you the following proposal:

A piece of cognitive content feels "arbitrary" if it is the kind of cognitive content that we expect to come with attached justifications, and those justifications are not present in our mind.

You'll note that I've performed the standard operation for guaranteeing that a potentially confusing question has a real answer: I substituted the question, "How does my brain label things 'arbitrary'?" for "What is this mysterious property of arbitrariness?" This is not necessarily a sleight-of-hand, since to explain something is not the same as explaining it away.

In this case, for nearly all everyday purposes, I would make free to proceed from "arbitrary" to arbit...
Abstracted Idealized Dynamics
by Eliezer Yudkowsky — 2008-08-12T01:00:00.000Z

Followup to: Morality as Fixed Computation

I keep trying to describe morality as a "computation", but people don't stand up and say "Aha!"

Pondering the surprising inferential distances that seem to be at work here, it occurs to me that when I say "computation", some of my listeners may not hear the Word of Power that I thought I was emitting; but, rather, may think of some complicated boring unimportant thing like Microsoft Word.

Maybe I should have said that morality is an abstracted idealized dynamic. This might not have meant anything to start with, but at least it wouldn't sound like I was describing Microsoft Word.

How, oh how, am I to describe the awesome import of this concept, "computation"?

Perhaps I can display the inner nature of computation, in its most general form, by showing how that inner nature manifests in something that seems very unlike Microsoft Word—namely, morality.

Consider certain features we might wish to ascribe to that-which-we-call "morality", or "should" or "right" or "good":

• It seems that we sometimes think about morality in our armchairs, without further peeking at the state of the outside world, and arrive at some previously unknown conclusion.

Someone sees a slave being whipped, and it doesn't occur to them right away that slavery is wrong. But they go home and think about it, and imagine themselves in the slave's place, and finally think, "No."

Can you think of anywhere else that something like this happens?

Suppose I tell you that I am making a rectangle of pebbles. You look at the rectangle, and count 19 pebbles on one side and 103 dots pebbles on the other side. You don't know right away how many pebbles there are. But you go home to your living room, and draw the blinds, and sit in your armchair and think; and without further looking at the physical array, you come to the conclusion that the rectangle contains 1957 pebbles.

Now, I'm not going to say the word "computation". But it seems like that-which-is "morality" should have the property of latent development of answers—that you may not know right away, everything that you have sufficient in-principle information to know. All the ingredients are present, but it takes additional time to bake the pie.

You can specify a Turing machine of 6 states and 2 symbols that unfolds into a string of 4.6 × 101439 1s afte...