trentmkelly/LessWrong-43k huggingface dataset:
Horrible LHC Inconsistency
by Eliezer Yudkowsky — 2008-09-22T03:12:58.000Z

Followup to: When (Not) To Use Probabilities, How Many LHC Failures Is Too Many?

While trying to answer my own question on "How Many LHC Failures Is Too Many?" I realized that I'm horrendously inconsistent with respect to my stated beliefs about disaster risks from the Large Hadron Collider.

First, I thought that stating a "one-in-a-million" probability for the Large Hadron Collider destroying the world was too high, in the sense that I would much rather run the Large Hadron Collider than press a button with a known 1/1,000,000 probability of destroying the world.

But if you asked me whether I could make one million statements of authority equal to "The Large Hadron Collider will not destroy the world", and be wrong, on average, around once, then I would have to say no.

Unknown pointed out that this turns me into a money pump. Given a portfolio of a million existential risks to which I had assigned a "less than one in a million probability", I would rather press the button on the fixed-probability device than run a random risk from this portfolio; but would rather take any particular risk in this portfolio than press the button.

Then, I considered the question of how many mysterious failures at the LHC it would take to make me question whether it might destroy the world/universe somehow, and what this revealed about my prior probability.

If the failure probability had a known 50% probability of occurring from natural causes, like a quantum coin or some such... then I suspect that if I actually saw that coin come up heads 20 times in a row, I would feel a strong impulse to bet on it coming up heads the next time around. (And that's taking into account my uncertainty about whether the anthropic principle really works that way.)

Even having noticed this triple inconsistency, I'm not sure in which direction to resolve it!

(But I still maintain my resolve that the LHC is not worth expending political capital, financial capital, or our time to shut down; compared with using the same capital to worry about superhuman intelligence or nanotechnology.)
How Many LHC Failures Is Too Many?
by Eliezer Yudkowsky — 2008-09-20T21:38:27.000Z

Recently the Large Hadron Collider was damaged by a mechanical failure. This requires the collider to be warmed up, repaired, and then cooled down again, so we're looking at a two-month delay.

Inevitably, many commenters said, "Anthropic principle! If the LHC had worked, it would have produced a black hole or strangelet or vacuum failure, and we wouldn't be here!"

This remark may be somewhat premature, since I don't think we're yet at the point in time when the LHC would have started producing collisions if not for this malfunction. However, a few weeks(?) from now, the "Anthropic!" hypothesis will start to make sense, assuming it can make sense at all. (Does this mean we can foresee executing a future probability update, but can't go ahead and update now?)

As you know, I don't spend much time worrying about the Large Hadron Collider when I've got much larger existential-risk-fish to fry. However, there's an exercise in probability theory (which I first picked up from E.T. Jaynes) along the lines of, "How many times does a coin have to come up heads before you believe the coin is fixed?" This tells you how low your prior probability is for the hypothesis. If a coin comes up heads only twice, that's definitely not a good reason to believe it's fixed, unless you already suspected from the beginning. But if it comes up heads 100 times, it's taking you too long to notice.

So - taking into account the previous cancellation of the Superconducting Supercollider (SSC) - how many times does the LHC have to fail before you'll start considering an anthropic explanation? 10? 20? 50?

After observing empirically that the LHC had failed 100 times in a row, would you endorse a policy of keeping the LHC powered up, but trying to fire it again only in the event of, say, nuclear terrorism or a global economic crash?
Ban the Bear
by Eliezer Yudkowsky — 2008-09-19T18:14:23.000Z

I applaud the SEC's courageous move to ban short selling. Isn't that brilliant? I wonder why they didn't think of that during the Great Depression.

However, I feel that this valiant effort does not go far enough.

All selling of stocks should be banned. Once you buy a stock, you have to hold it forever.

Sure, this might make the market a little less liquid. But once stock prices can only go up, we'll all be rich!

Or maybe we should just try something simpler: pass a law making it illegal for stock prices to go down.
Say It Loud
by Eliezer Yudkowsky — 2008-09-19T17:34:58.000Z

Reply to: Overconfidence is Stylish

I respectfully defend my lord Will Strunk:

"If you don't know how to pronounce a word, say it loud! If you don't know how to pronounce a word, say it loud!" This comical piece of advice struck me as sound at the time, and I still respect it. Why compound ignorance with inaudibility? Why run and hide?

How does being vague, tame, colorless, irresolute, help someone to understand your current state of uncertainty? Any more than mumbling helps them understand a word you aren't sure how to pronounce?

Goofus says: "The sky, if such a thing exists at all, might or might not have a property of color, but, if it does have color, then I feel inclined to state that it might be green."

Gallant says: "70% probability the sky is green."

Which of them sounds more confident, more definite?

But which of them has managed to quickly communicate their state of uncertainty?

(And which of them is more likely to actually, in real life, spend any time planning and preparing for the eventuality that the sky is blue?)

I am often accused of overconfidence because my audience is not familiar with the concept of there being iron laws that govern the manipulation of uncertainty. Just because I don't know the object-level doesn't necessarily mean that I am in a state of fear and doubt as to what I should be thinking. That comes through in my writing, and so I sound confident even when I am in the midst of manipulating uncertainty. That might be a disadvantage in my attempts to communicate; but I would rather clearly describe my state of uncertainty, and worry afterward about how that makes me look.

And similarly, I have often seen people who spend no effort at all on possibilities other than their mainline, praised for their seeming humility, on account of their indefinite language. They are skilled at sounding uncertain, which makes them appear modest; but not skilled at handling uncertainty. That is a political advantage, but it doesn't help them think. Also the audience is given more slack to interpret the speaker as being on their side; but to deliberately exploit this effect is dishonesty.

Often the caveats we attach to our speech have little to do with any actual humility - actual plans we prepared, and actions we took, against the eventuality of things turning out the other way.&nbs...
The Sheer Folly of Callow Youth
by Eliezer Yudkowsky — 2008-09-19T01:30:29.000Z

"There speaks the sheer folly of callow youth; the rashness of an ignorance so abysmal as to be possible only to one of your ephemeral race..."
—Gharlane of Eddore

Once upon a time, years ago, I propounded a mysterious answer to a mysterious question—as I've hinted on several occasions. The mysterious question to which I propounded a mysterious answer was not, however, consciousness—or rather, not only consciousness. No, the more embarrassing error was that I took a mysterious view of morality.

I held off on discussing that until now, after the series on metaethics, because I wanted it to be clear that Eliezer1997 had gotten it wrong.

When we last left off, Eliezer1997, not satisfied with arguing in an intuitive sense that superintelligence would be moral, was setting out to argue inescapably that creating superintelligence was the right thing to do.

Well (said Eliezer1997) let's begin by asking the question: Does life have, in fact, any meaning?

"I don't know," replied Eliezer1997 at once, with a certain note of self-congratulation for admitting his own ignorance on this topic where so many others seemed certain.

"But," he went on—

(Always be wary when an admission of ignorance is followed by "But".)

"But, if we suppose that life has no meaning—that the utility of all outcomes is equal to zero—that possibility cancels out of any expected utility calculation. We can therefore always act as if life is known to be meaningful, even though we don't know what that meaning is. How can we find out that meaning? Considering that humans are still arguing about this, it's probably too difficult a problem for humans to solve. So we need a superintelligence to solve the problem for us. As for the possibility that there is no logical justification for one preference over another, then in this case it is no righter or wronger to build a superintelligence, than to do anything else. This is a real possibility, but it falls out of any attempt to calculate expected utility—we should just ignore it. To the extent someone says that a superintelligence would wipe out humanity, they are either arguing that wiping out humanity is in fact the right thing to do (even though we see no reason why this should be the case) or they are arguing that there is no ...
A Prodigy of Refutation
by Eliezer Yudkowsky — 2008-09-18T01:57:50.000Z

My Childhood Death Spiral described the core momentum carrying me into my mistake, an affective death spiral around something that Eliezer1996 called "intelligence". I was also a technophile, pre-allergized against fearing the future. And I'd read a lot of science fiction built around personhood ethics—in which fear of the Alien puts humanity-at-large in the position of the bad guys, mistreating aliens or sentient AIs because they "aren't human".

That's part of the ethos you acquire from science fiction—to define your in-group, your tribe, appropriately broadly. Hence my email address, sentience@pobox.com.

So Eliezer1996 is out to build superintelligence, for the good of humanity and all sentient life.

At first, I think, the question of whether a superintelligence will/could be good/evil didn't really occur to me as a separate topic of discussion. Just the standard intuition of, "Surely no supermind would be stupid enough to turn the galaxy into paperclips; surely, being so intelligent, it will also know what's right far better than a human being could."

Until I introduced myself and my quest to a transhumanist mailing list, and got back responses along the general lines of (from memory):

Morality is arbitrary—if you say that something is good or bad, you can't be right or wrong about that. A superintelligence would form its own morality.

Everyone ultimately looks after their own self-interest. A superintelligence would be no different; it would just seize all the resources.

Personally, I'm a human, so I'm in favor of humans, not Artificial Intelligences. I don't think we should develop this technology. Instead we should develop the technology to upload humans first.

No one should develop an AI without a control system that watches it and makes sure it can't do anything bad.

Well, that's all obviously wrong, thinks Eliezer1996, and he proceeded to kick his opponents' arguments to pieces. (I've mostly done this in other blog posts, and anything remaining is left as an exercise to the reader.)

It's not that Eliezer1996 explicitly reasoned, "The world's stupidest man says the sun is shining, therefore it is dark out." But Eliezer1996 was a Traditional Rationalist; he had been inculcated with the metaphor of science as a fair fight between sides who take on different positions, stripped of m...
Raised in Technophilia
by Eliezer Yudkowsky — 2008-09-17T02:06:26.000Z

My father used to say that if the present system had been in place a hundred years ago, automobiles would have been outlawed to protect the saddle industry.

One of my major childhood influences was reading Jerry Pournelle's A Step Farther Out, at the age of nine. It was Pournelle's reply to Paul Ehrlich and the Club of Rome, who were saying, in the 1960s and 1970s, that the Earth was running out of resources and massive famines were only years away. It was a reply to Jeremy Rifkin's so-called fourth law of thermodynamics; it was a reply to all the people scared of nuclear power and trying to regulate it into oblivion.

I grew up in a world where the lines of demarcation between the Good Guys and the Bad Guys were pretty clear; not an apocalyptic final battle, but a battle that had to be fought over and over again, a battle where you could see the historical echoes going back to the Industrial Revolution, and where you could assemble the historical evidence about the actual outcomes.

On one side were the scientists and engineers who'd driven all the standard-of-living increases since the Dark Ages, whose work supported luxuries like democracy, an educated populace, a middle class, the outlawing of slavery.

On the other side, those who had once opposed smallpox vaccinations, anesthetics during childbirth, steam engines, and heliocentrism: The theologians calling for a return to a perfect age that never existed, the elderly white male politicians set in their ways, the special interest groups who stood to lose, and the many to whom science was a closed book, fearing what they couldn't understand.

And trying to play the middle, the pretenders to Deep Wisdom, uttering cached thoughts about how technology benefits humanity but only when it was properly regulated—claiming in defiance of brute historical fact that science of itself was neither good nor evil—setting up solemn-looking bureaucratic committees to make an ostentatious display of their caution—and waiting for their applause. As if the truth were always a compromise. And as if anyone could really see that far ahead. Would humanity have done better if there'd been a sincere, concerned, public debate on the adoption of fire, and commitees set up to oversee its use?

When I entered into the problem, I started out allergized against anything that pattern-matched "A...
My Best and Worst Mistake
by Eliezer Yudkowsky — 2008-09-16T00:43:50.000Z

Yesterday I covered the young Eliezer's affective death spiral around something that he called "intelligence". Eliezer1996, or even Eliezer1999 for that matter, would have refused to try and put a mathematical definition—consciously, deliberately refused. Indeed, he would have been loath to put any definition on "intelligence" at all.

Why? Because there's a standard bait-and-switch problem in AI, wherein you define "intelligence" to mean something like "logical reasoning" or "the ability to withdraw conclusions when they are no longer appropriate", and then you build a cheap theorem-prover or an ad-hoc nonmonotonic reasoner, and then say, "Lo, I have implemented intelligence!" People came up with poor definitions of intelligence—focusing on correlates rather than cores—and then they chased the surface definition they had written down, forgetting about, you know, actual intelligence. It's not like Eliezer1996 was out to build a career in Artificial Intelligence. He just wanted a mind that would actually be able to build nanotechnology. So he wasn't tempted to redefine intelligence for the sake of puffing up a paper.

Looking back, it seems to me that quite a lot of my mistakes can be defined in terms of being pushed too far in the other direction by seeing someone else stupidity: Having seen attempts to define "intelligence" abused so often, I refused to define it at all. What if I said that intelligence was X, and it wasn't really X? I knew in an intuitive sense what I was looking for—something powerful enough to take stars apart for raw material—and I didn't want to fall into the trap of being distracted from that by definitions.

Similarly, having seen so many AI projects brought down by physics envy—trying to stick with simple and elegant math, and being constrained to toy systems as a result—I generalized that any math simple enough to be formalized in a neat equation was probably not going to work for, you know, real intelligence. "Except for Bayes's Theorem," Eliezer2000 added; which, depending on your viewpoint, either mitigates the totality of his offense, or shows that he should have suspected the entire generalization instead of trying to add a single exception.

If you're wondering why Eliezer2000 thought such a thing—disbelieved in a math of...
My Childhood Death Spiral
by Eliezer Yudkowsky — 2008-09-15T03:42:27.000Z

My parents always used to downplay the value of intelligence. And play up the value of—effort, as recommended by the latest research? No, not effort. Experience. A nicely unattainable hammer with which to smack down a bright young child, to be sure. That was what my parents told me when I questioned the Jewish religion, for example. I tried laying out an argument, and I was told something along the lines of: "Logic has limits, you'll understand when you're older that experience is the important thing, and then you'll see the truth of Judaism." I didn't try again. I made one attempt to question Judaism in school, got slapped down, didn't try again. I've never been a slow learner.

Whenever my parents were doing something ill-advised, it was always, "We know better because we have more experience. You'll understand when you're older: maturity and wisdom is more important than intelligence."

If this was an attempt to focus the young Eliezer on intelligence uber alles, it was the most wildly successful example of reverse psychology I've ever heard of.

But my parents aren't that cunning, and the results weren't exactly positive.

For a long time, I thought that the moral of this story was that experience was no match for sheer raw native intelligence. It wasn't until a lot later, in my twenties, that I looked back and realized that I couldn't possibly have been more intelligent than my parents before puberty, with my brain not even fully developed. At age eleven, when I was already nearly a full-blown atheist, I could not have defeated my parents in any fair contest of mind. My SAT scores were high for an 11-year-old, but they wouldn't have beaten my parents' SAT scores in full adulthood. In a fair fight, my parents' intelligence and experience could have stomped any prepubescent child flat. It was dysrationalia that did them in; they used their intelligence only to defeat itself.

But that understanding came much later, when my intelligence had processed and distilled many more years of experience.

The moral I derived when I was young, was that anyone who downplayed the value of intelligence didn't understand intelligence at all. My own intelligence had affected every aspect of my life and mind and personality; that was massively obvious, seen at a backward g...
Optimization
by Eliezer Yudkowsky — 2008-09-13T16:00:00.000Z

"However many ways there may be of being alive, it is certain that there are vastly more ways of being dead."
-- Richard Dawkins

In the coming days, I expect to be asked: "Ah, but what do you mean by 'intelligence'?" By way of untangling some of my dependency network for future posts, I here summarize some of my notions of "optimization".

Consider a car; say, a Toyota Corolla. The Corolla is made up of some number of atoms; say, on the rough order of 1029. If you consider all possible ways to arrange 1029 atoms, only an infinitesimally tiny fraction of possible configurations would qualify as a car; if you picked one random configuration per Planck interval, many ages of the universe would pass before you hit on a wheeled wagon, let alone an internal combustion engine.

Even restricting our attention to running vehicles, there is an astronomically huge design space of possible vehicles that could be composed of the same atoms as the Corolla, and most of them, from the perspective of a human user, won't work quite as well. We could take the parts in the Corolla's air conditioner, and mix them up in thousands of possible configurations; nearly all these configurations would result in a vehicle lower in our preference ordering, still recognizable as a car but lacking a working air conditioner.

So there are many more configurations corresponding to nonvehicles, or vehicles lower in our preference ranking, than vehicles ranked greater than or equal to the Corolla.

Similarly with the problem of planning, which also involves hitting tiny targets in a huge search space. Consider the number of possible legal chess moves versus the number of winning moves.

Which suggests one theoretical way to measure optimization - to quantify the power of a mind or mindlike process:

Put a measure on the state space - if it's discrete, you can just count. Then collect all the states which are equal to or greater than the observed outcome, in that optimization process's implicit or explicit preference ordering. Sum or integrate over the total size of all such states. Divide by the total volume of the state space. This gives you the power of the optimization process measured in terms of the improbabilities that it can produce - that is, improbability of a random selection p...
Psychic Powers
by Eliezer Yudkowsky — 2008-09-12T19:28:53.000Z

If the "boring view" of reality is correct, then you can never predict anything irreducible because you are reducible. You can never get Bayesian confirmation for a hypothesis of irreducibility, because any prediction you can make is, therefore, something that could also be predicted by a reducible thing, namely your brain.

Benja Fallenstein commented:

I think that while you can in this case never devise an empirical test whose outcome could logically prove irreducibility, there is no clear reason to believe that you cannot devise a test whose counterfactual outcome in an irreducible world would make irreducibility subjectively much more probable (given an Occamian prior).

Without getting into reducibility/irreducibility, consider the scenario that the physical universe makes it possible to build a hypercomputer —that performs operations on arbitrary real numbers, for example —but that our brains do not actually make use of this: they can be simulated perfectly well by an ordinary Turing machine, thank you very much...

Well, that's a very intelligent argument, Benja Fallenstein. But I have a crushing reply to your argument, such that, once I deliver it, you will at once give up further debate with me on this particular point:

You're right.

Alas, I don't get modesty credit on this one, because after publishing yesterday's post I realized a similar flaw on my own—this one concerning Occam's Razor and psychic powers:

If beliefs and desires are irreducible and ontologically basic entities, or have an ontologically basic component not covered by existing science, that would make it far more likely that there was an ontological rule governing the interaction of different minds—an interaction which bypassed ordinary "material" means of communication like sound waves, known to existing science.

If naturalism is correct, then there exists a conjugate reductionist model that makes the same predictions as any concrete prediction that any parapsychologist can make about telepathy.

Indeed, if naturalism is correct, the only reason we can conceive of beliefs as "fundamental" is due to lack of self-knowledge of our own neurons—that the peculiar reflective architecture of our own minds exposes the "belief" class but hides the machinery behind it.

Nonetheless, the discovery of information transfer between brains, in the absence of any k...