trentmkelly/LessWrong-43k huggingface dataset:
Beautiful Probability
by Eliezer Yudkowsky — 2008-01-14T07:19:47.000Z

Should we expect rationality to be, on some level, simple? Should we search and hope for underlying beauty in the arts of belief and choice?

Let me introduce this issue by borrowing a complaint of the late great Bayesian Master, E. T. Jaynes (1990):

"Two medical researchers use the same treatment independently, in different hospitals. Neither would stoop to falsifying the data, but one had decided beforehand that because of finite resources he would stop after treating N=100 patients, however many cures were observed by then. The other had staked his reputation on the efficacy of the treatment, and decided he would not stop until he had data indicating a rate of cures definitely greater than 60%, however many patients that might require. But in fact, both stopped with exactly the same data: n = 100 [patients], r = 70 [cures]. Should we then draw different conclusions from their experiments?" (Presumably the two control groups also had equal results.)

According to old-fashioned statistical procedure - which I believe is still being taught today - the two researchers have performed different experiments with different stopping conditions. The two experiments could have terminated with different data, and therefore represent different tests of the hypothesis, requiring different statistical analyses. It's quite possible that the first experiment will be "statistically significant", the second not.

Whether or not you are disturbed by this says a good deal about your attitude toward probability theory, and indeed, rationality itself.

Non-Bayesian statisticians might shrug, saying, "Well, not all statistical tools have the same strengths and weaknesses, y'know - a hammer isn't like a screwdriver - and if you apply different statistical tools you may get different results, just like using the same data to compute a linear regression or train a regularized neural network. You've got to use the right tool for the occasion. Life is messy -"

And then there's the Bayesian reply: "Excuse you? The evidential impact of a fixed experimental method, producing the same data, depends on the researcher's private thoughts? And you have the nerve to accuse us of being 'too subjective'?"

If Nature is one way, the likelihood of the data coming out the way we have seen will be one thing. If Nature...
Is Reality Ugly?
by Eliezer Yudkowsky — 2008-01-12T22:26:23.000Z

Yesterday I talked about the cubes {1, 8, 27, 64, 125, ...} and how their first differences {7, 19, 37, 61, ...} might at first seem to lack an obvious pattern, but taking the second differences {12, 18, 24, ...} takes you down to the simply related level. Taking the third differences {6, 6, ...} brings us to the perfectly stable level, where chaos dissolves into order.

But this (as I noted) is a handpicked example. Perhaps the "messy real world" lacks the beauty of these abstract mathematical objects? Perhaps it would be more appropriate to talk about neuroscience or gene expression networks?

Abstract math, being constructed solely in imagination, arises from simple foundations - a small set of initial axioms - and is a closed system; conditions that might seem unnaturally conducive to neatness.

Which is to say: In pure math, you don't have to worry about a tiger leaping out of the bushes and eating Pascal's Triangle.

So is the real world uglier than mathematics?

Strange that people ask this. I mean, the question might have been sensible two and a half millennia ago...

Back when the Greek philosophers were debating what this "real world" thingy might be made of, there were many positions. Heraclitus said, "All is fire." Thales said, "All is water." Pythagoras said, "All is number."

Score: Heraclitus 0 Thales 0 Pythagoras 1

Beneath the complex forms and shapes of the surface world, there is a simple level, an exact and stable level, whose laws we name "physics". This discovery, the Great Surprise, has already taken place at our point in human history - but it does not do to forget that it was surprising. Once upon a time, people went in search of underlying beauty, with no guarantee of finding it; and once upon a time, they found it; and now it is a known thing, and taken for granted.

Then why can't we predict the location of every tiger in the bushes as easily as we predict the sixth cube?

I count three sources of uncertainty even within worlds of pure math - two obvious sources, and one not so obvious.

The first source of uncertainty is that even a creature of pure math, living embedded in a world of pure math, may not know the math. Humans walked the Earth long before Galileo/Newton/Einstein discov...
Expecting Beauty
by Eliezer Yudkowsky — 2008-01-12T03:00:04.000Z

Followup to: Beautiful Math

If you looked at the sequence {1, 4, 9, 16, 25, ...} and didn't
recognize the square numbers, you might still essay an
excellent-seeming prediction of future items in the sequence by
noticing that the table of first differences is {3, 5, 7, 9, ...}.
Indeed, your prediction would be perfect, though you have no way of
knowing this without peeking at the generator. The correspondence can
be shown algebraically or even geometrically (see yesterday's post). It's really rather elegant.

Whatever people praise, they tend to praise too much; and there are skeptics who think that the pursuit of elegance is like unto a disease, which produces neat mathematics in opposition to the messiness of the real world. "You got lucky," they say, "and you won't always be lucky. If you expect that kind of elegance, you'll distort the world to match your expectations - chop off all the parts of Life that don't fit into your nice little pattern."

I mean, suppose Life hands you the sequence {1, 8, 27, 64, 125, ...}. When you take the first differences, you get {7, 19, 37, 61, ...}. All these numbers have in common is that they're primes, and they aren't even sequential primes. Clearly, there isn't the neat order here that we saw in the squares.

You might try to impose order, by insisting that the first differences must
be evenly spaced, and any deviations are experimental errors - or
better yet, we just won't think about them. "You will say," says the
skeptic, "that 'The first differences are spaced around 20 apart and land on
prime numbers, so that the next difference is probably 83, which makes the next number 208.' But reality comes back and says 216."

Serves you right, expecting neatness and elegance when there isn't any there. You were too addicted to absolutes; you had too much need for closure. Behold the perils of - gasp! - DUN DUN DUN - reductionism.

You can guess, from the example I chose, that I don't think this is the best way to look at the problem. Because, in the example I chose, it's not that no order exists, but that you have to look a little deeper to find it. The sequence {7, 19, 37, 61, ...} doesn't leap out at you - you might not recognize it, if you met it on the street - but take the second differences and you find {

...
Beautiful Math
by Eliezer Yudkowsky — 2008-01-10T22:43:55.000Z

Consider the sequence {1, 4, 9, 16, 25, ...} You recognize these as the square numbers, the sequence Ak = k2. Suppose you did not recognize this sequence at a first glance. Is there any way you could predict the next item in the sequence? Yes: You could take the first differences, and end up with:

{4 - 1, 9 - 4, 16 - 9, 25 - 16, ...} = {3, 5, 7, 9, ...}

And if you don't recognize these as successive odd numbers, you are still not defeated; if you produce a table of the second differences, you will find:

{5 - 3, 7 - 5, 9 - 7, ...} = {2, 2, 2, ...}

If you cannot recognize this as the number 2 repeating, then you're hopeless.

But if you predict that the next second difference is also 2, then you can see the next first difference must be 11, and the next item in the original sequence must be 36 - which, you soon find out, is correct.

Dig down far enough, and you discover hidden order, underlying structure, stable relations beneath changing surfaces.

The original sequence was generated by squaring successive numbers - yet we predicted it using what seems like a wholly different method, one that we could in principle use without ever realizing we were generating the squares. Can you prove the two methods are always equivalent? - for thus far we have not proven this, but only ventured an induction. Can you simplify the proof so that you can you see it at a glance? - as Polya was fond of asking.

This is a very simple example by modern standards, but it is a very simple example of the sort of thing that mathematicians spend their whole lives looking for.

The joy of mathematics is inventing mathematical objects, and then noticing that the mathematical objects that you just created have all sorts of wonderful properties that you never intentionally built into them. It is like building a toaster and then realizing that your invention also, for some unexplained reason, acts as a rocket jetpack and MP3 player.

Numbers, according to our best guess at history, have been invented and reinvented over the course of time. (Apparently some artifacts from 30,000 BC have marks cut that look suspiciously like tally marks.) But I doubt that a single one of the human beings who invented counting visualized the employment they would provide to generations of mathematicians. Or the excitement that would someday surround Fermat's ...
0 And 1 Are Not Probabilities
by Eliezer Yudkowsky — 2008-01-10T06:58:50.000Z

One, two, and three are all integers, and so is negative four. If you keep counting up, or keep counting down, you’re bound to encounter a whole lot more integers. You will not, however, encounter anything called “positive infinity” or “negative infinity,” so these are not integers.

Positive and negative infinity are not integers, but rather special symbols for talking about the behavior of integers. People sometimes say something like, “5 + infinity = infinity,” because if you start at 5 and keep counting up without ever stopping, you’ll get higher and higher numbers without limit. But it doesn’t follow from this that “infinity - infinity = 5.” You can’t count up from 0 without ever stopping, and then count down without ever stopping, and then find yourself at 5 when you’re done.

From this we can see that infinity is not only not-an-integer, it doesn’t even behave like an integer. If you unwisely try to mix up infinities with integers, you’ll need all sorts of special new inconsistent-seeming behaviors which you don’t need for 1, 2, 3 and other actual integers.

Even though infinity isn’t an integer, you don’t have to worry about being left at a loss for numbers. Although people have seen five sheep, millions of grains of sand, and septillions of atoms, no one has ever counted an infinity of anything. The same with continuous quantities—people have measured dust specks a millimeter across, animals a meter across, cities kilometers across, and galaxies thousands of lightyears across, but no one has ever measured anything an infinity across. In the real world, you don’t need a whole lot of infinity.1

In the usual way of writing probabilities, probabilities are between 0 and 1. A coin might have a probability of 0.5 of coming up tails, or the weatherman might assign probability 0.9 to rain tomorrow.

This isn’t the only way of writing probabilities, though. For example, you can transform probabilities into odds via the transformation O = (P/(1 - P)). So a probability of 50% would go to odds of 0.5/0.5 or 1, usually written 1:1, while a probability of 0.9 would go to odds of 0.9/0.1 or 9, usually written 9:1. To take odds back to probabilities you use P = (O∕(1 + O)), and this is perfectly reversible, so the transformation is an isomorphism—a two-way reversible mapping. Thus, probabilities and odds are isomorphic, and you can use one or the other according to convenie...
Infinite Certainty
by Eliezer Yudkowsky — 2008-01-09T06:49:08.000Z

In “Absolute Authority,” I argued that you don’t need infinite certainty:

If you have to choose between two alternatives A and B, and you somehow succeed in establishing knowably certain well-calibrated 100% confidence that A is absolutely and entirely desirable and that B is the sum of everything evil and disgusting, then this is a sufficient condition for choosing A over B. It is not a necessary condition . . . You can have uncertain knowledge of relatively better and relatively worse options, and still choose. It should be routine, in fact.

Concerning the proposition that 2 + 2 = 4, we must distinguish between the map and the territory. Given the seeming absolute stability and universality of physical laws, it’s possible that never, in the whole history of the universe, has any particle exceeded the local lightspeed limit. That is, the lightspeed limit may be not just true 99% of the time, or 99.9999% of the time, or (1 - 1/googolplex) of the time, but simply always and absolutely true.

But whether we can ever have absolute confidence in the lightspeed limit is a whole ’nother question. The map is not the territory.

It may be entirely and wholly true that a student plagiarized their assignment, but whether you have any knowledge of this fact at all—let alone absolute confidence in the belief—is a separate issue. If you flip a coin and then don’t look at it, it may be completely true that the coin is showing heads, and you may be completely unsure of whether the coin is showing heads or tails. A degree of uncertainty is not the same as a degree of truth or a frequency of occurrence.

The same holds for mathematical truths. It’s questionable whether the statement “2 + 2 = 4” or “In Peano arithmetic, SS0 + SS0 = SSSS0” can be said to be true in any purely abstract sense, apart from physical systems that seem to behave in ways similar to the Peano axioms. Having said this, I will charge right ahead and guess that, in whatever sense “2 + 2 = 4” is true at all, it is always and precisely true, not just roughly true (“2 + 2 actually equals 4.0000004”) or true 999,999,999,999 times out of 1,000,000,000,000.

I’m not totally sure what “true” should mean in this case, but I stand by my guess. The credibility of “2 + 2 = 4 is always true” far exceeds the credibility of any particular philosophical position on what “true,” “always,” or “is” means in the statement above.

This ...
Absolute Authority
by Eliezer Yudkowsky — 2008-01-08T03:33:43.000Z

The one comes to you and loftily says: “Science doesn’t really know anything. All you have are theories—you can’t know for certain that you’re right. You scientists changed your minds about how gravity works—who’s to say that tomorrow you won’t change your minds about evolution?”

Behold the abyssal cultural gap. If you think you can cross it in a few sentences, you are bound to be sorely disappointed.

In the world of the unenlightened ones, there is authority and un-authority. What can be trusted, can be trusted; what cannot be trusted, you may as well throw away. There are good sources of information and bad sources of information. If scientists have changed their stories ever in their history, then science cannot be a true Authority, and can never again be trusted—like a witness caught in a contradiction, or like an employee found stealing from the till.

Plus, the one takes for granted that a proponent of an idea is expected to defend it against every possible counterargument and confess nothing. All claims are discounted accordingly. If even the proponent of science admits that science is less than perfect, why, it must be pretty much worthless.

When someone has lived their life accustomed to certainty, you can’t just say to them, “Science is probabilistic, just like all other knowledge.” They will accept the first half of the statement as a confession of guilt; and dismiss the second half as a flailing attempt to accuse everyone else to avoid judgment.

You have admitted you are not trustworthy—so begone, Science, and trouble us no more!

One obvious source for this pattern of thought is religion, where the scriptures are alleged to come from God; therefore to confess any flaw in them would destroy their authority utterly; so any trace of doubt is a sin, and claiming certainty is mandatory whether you’re certain or not.1

But I suspect that the traditional school regimen also has something to do with it. The teacher tells you certain things, and you have to believe them, and you have to recite them back on the test. But when a student makes a suggestion in class, you don’t have to go along with it—you’re free to agree or disagree (it seems) and no one will punish you.

This experience, I fear, maps the domain of belief onto the social domains of authority, of command, of law. In the social domain, there is a qualitative difference between absolute laws and nonabsolu...
The Fallacy of Gray
by Eliezer Yudkowsky — 2008-01-07T06:24:55.000Z

The Sophisticate: “The world isn’t black and white. No one does pure good or pure bad. It’s all gray. Therefore, no one is better than anyone else.”

The Zetet: “Knowing only gray, you conclude that all grays are the same shade. You mock the simplicity of the two-color view, yet you replace it with a one-color view . . .”

—Marc Stiegler, David’s Sling

I don’t know if the Sophisticate’s mistake has an official name, but I call it the Fallacy of Gray. We saw it manifested in the previous essay—the one who believed that odds of two to the power of seven hundred and fifty million to one, against, meant “there was still a chance.” All probabilities, to him, were simply “uncertain” and that meant he was licensed to ignore them if he pleased.

“The Moon is made of green cheese” and “the Sun is made of mostly hydrogen and helium” are both uncertainties, but they are not the same uncertainty.

Everything is shades of gray, but there are shades of gray so light as to be very nearly white, and shades of gray so dark as to be very nearly black. Or even if not, we can still compare shades, and say “it is darker” or “it is lighter.”

Years ago, one of the strange little formative moments in my career as a rationalist was reading this paragraph from Player of Games by Iain M. Banks, especially the sentence in bold:

A guilty system recognizes no innocents. As with any power apparatus which thinks everybody’s either for it or against it, we’re against it. You would be too, if you thought about it. The very way you think places you amongst its enemies. This might not be your fault, because every society imposes some of its values on those raised within it, but the point is that some societies try to maximize that effect, and some try to minimize it. You come from one of the latter and you’re being asked to explain yourself to one of the former. Prevarication will be more difficult than you might imagine; neutrality is probably impossible. You cannot choose not to have the politics you do; they are not some separate set of entities somehow detachable from the rest of your being; they are a function of your existence. I know that and they know that; you

...
But There's Still A Chance, Right?
by Eliezer Yudkowsky — 2008-01-06T01:56:14.000Z

Years ago, I was speaking to someone when he casually remarked that he didn’t believe in evolution. And I said, “This is not the nineteenth century. When Darwin first proposed evolution, it might have been reasonable to doubt it. But this is the twenty-first century. We can read the genes. Humans and chimpanzees have 98% shared DNA. We know humans and chimps are related. It’s over.”

He said, “Maybe the DNA is just similar by coincidence.”

I said, “The odds of that are something like two to the power of seven hundred and fifty million to one.”

He said, “But there’s still a chance, right?”

Now, there’s a number of reasons my past self cannot claim a strict moral victory in this conversation. One reason is that I have no memory of whence I pulled that 2750,000,000 figure, though it’s probably the right meta-order of magnitude. The other reason is that my past self didn’t apply the concept of a calibrated confidence. Of all the times over the history of humanity that a human being has calculated odds of 2750,000,000:1 against something, they have undoubtedly been wrong more often than once in 2750,000,000 times. E.g., the shared genes estimate was revised to 95%, not 98%—and that may even apply only to the 30,000 known genes and not the entire genome, in which case it’s the wrong meta-order of magnitude.

But I think the other guy’s reply is still pretty funny.

I don’t recall what I said in further response—probably something like “No”—but I remember this occasion because it brought me several insights into the laws of thought as seen by the unenlightened ones.

It first occurred to me that human intuitions were making a qualitative distinction between “No chance” and “A very tiny chance, but worth keeping track of.” You can see this in the Overcoming Bias lottery debate.

The problem is that probability theory sometimes lets us calculate a chance which is, indeed, too tiny to be worth the mental space to keep track of it—but by that time, you’ve already calculated it. People mix up the map with the territory, so that on a gut level, tracking a symbolically described probability feels like “a chance worth keeping track of,” even if th...
A Failed Just-So Story
by Eliezer Yudkowsky — 2008-01-05T06:35:50.000Z

Followup to: Rational vs. Scientific Ev-Psych, The Tragedy of Group Selectionism, Evolving to Extinction

Perhaps the real reason that evolutionary "just-so stories" got a bad name is that so many attempted stories are prima facie absurdities to serious students of the field.

As an example, consider a hypothesis I've heard a few times (though I didn't manage to dig up an example). The one says: Where does religion come from? It appears to be a human universal, and to have its own emotion backing it - the emotion of religious faith. Religion often involves costly sacrifices, even in hunter-gatherer tribes - why does it persist? What selection pressure could there possibly be for religion?

So, the one concludes, religion must have evolved because it bound tribes closer together, and enabled them to defeat other tribes that didn't have religion.

This, of course, is a group selection argument - an individual sacrifice for a group benefit - and see the referenced posts if you're not familiar with the math, simulations, and observations which show that group selection arguments are extremely difficult to make work. For example, a 3% individual fitness sacrifice which doubles the fitness of the tribe will fail to rise to universality, even under unrealistically liberal assumptions, if the tribe size is as large as fifty. Tribes would need to have no more than 5 members if the individual fitness cost were 10%. You can see at a glance from the sex ratio in human births that, in humans, individual selection pressures overwhelmingly dominate group selection pressures. This is an example of what I mean by prima facie absurdity.

So why religion, then?

Well, it might just be a side effect of our ability to do things like model other minds, which enables us to conceive of disembodied minds. Faith, as an emotion, might just be co-opted hope.

But if faith is a true religious adaptation, I don't see why it's even puzzling what the selection pressure could have been.

Heretics were routinely burned alive just a few centuries ago. Or stoned to death, or executed by whatever method local fashion demands. Questioning the local gods is the notional crime for which Socrates was made to drink hemlock.

Conversely, Huckabee just won Iowa's nomination for tribal-chieftain.

Why would you need to go anywhere near the accur...