trentmkelly/LessWrong-43k huggingface dataset:
Prolegomena to a Theory of Fun
by Eliezer Yudkowsky — 2008-12-17T23:33:01.000Z

Raise the topic of cryonics, uploading, or just medically extended lifespan/healthspan, and some bioconservative neo-Luddite is bound to ask, in portentous tones:

"But what will people do all day?"

They don't try to actually answer the question. That is not a bioethicist's role, in the scheme of things. They're just there to collect credit for the Deep Wisdom of asking the question. It's enough to imply that the question is unanswerable, and therefore, we should all drop dead.

That doesn't mean it's a bad question.

It's not an easy question to answer, either. The primary experimental result in hedonic psychology—the study of happiness—is that people don't know what makes them happy.

And there are many exciting results in this new field, which go a long way toward explaining the emptiness of classical Utopias. But it's worth remembering that human hedonic psychology is not enough for us to consider, if we're asking whether a million-year lifespan could be worth living.

Fun Theory, then, is the field of knowledge that would deal in questions like:

"How much fun is there in the universe?"
"Will we ever run out of fun?"
"Are we having fun yet?"
"Could we be having more fun?"

One major set of experimental results in hedonic psychology has to do with overestimating the impact of life events on happiness. Six months after the event, lottery winners aren't as happy as they expected to be, and quadriplegics aren't as sad. A parent who loses a child isn't as sad as they think they'll be, a few years later. If you look at one moment snapshotted out of their lives a few years later, that moment isn't likely to be about the lost child. Maybe they're playing with one of their surviving children on a swing. Maybe they're just listening to a nice song on the radio.

When people are asked to imagine how happy or sad an event will make them, they anchor on the moment of first receiving the news, rather than realistically imagining the process of daily life years later.

Consider what the Christians made of their Heaven, meant to be literally eternal. Endless rest, the glorious presence of God, and occasionally—in the more clueless sort of sermon—golden streets and diamond buildings. Is this eudaimonia? It doesn't even seem very hedonic.

As someone who said his share of prayers back in his Orthodox Jewish childhood ...
Visualizing Eutopia
by Eliezer Yudkowsky — 2008-12-16T18:39:54.000Z

Followup to: Not Taking Over the World

"Heaven is a city 15,000 miles square or 6,000 miles around. One side is 245 miles longer than the length of the Great Wall of China. Walls surrounding Heaven are 396,000 times higher than the Great Wall of China and eight times as thick. Heaven has twelve gates, three on each side, and has room for 100,000,000,000 souls. There are no slums. The entire city is built of diamond material, and the streets are paved with gold. All inhabitants are honest and there are no locks, no courts, and no policemen."
-- Reverend Doctor George Hawes, in a sermon

Yesterday I asked my esteemed co-blogger Robin what he would do with "unlimited power", in order to reveal something of his character. Robin said that he would (a) be very careful and (b) ask for advice. I asked him what advice he would give himself. Robin said it was a difficult question and he wanted to wait on considering it until it actually happened. So overall he ran away from the question like a startled squirrel.

The character thus revealed is a virtuous one: it shows common sense. A lot of people jump after the prospect of absolute power like it was a coin they found in the street.

When you think about it, though, it says a lot about human nature that this is a difficult question. I mean - most agents with utility functions shouldn't have such a hard time describing their perfect universe.

For a long time, I too ran away from the question like a startled squirrel. First I claimed that superintelligences would inevitably do what was right, relinquishing moral responsibility in toto. After that, I propounded various schemes to shape a nice superintelligence, and let it decide what should be done with the world.

Not that there's anything wrong with that. Indeed, this is still the plan. But it still meant that I, personally, was ducking the question.

Why? Because I expected to fail at answering. Because I thought that any attempt for humans to visualize a better future was going to end up recapitulating the Reverend Doctor George Hawes: apes thinking, "Boy, if I had human intelligence I sure could get a lot more bananas."

But trying to get a better answer to a question out of a superintelligence, is a different matter from entirely ducking the question yourself. The point at which I stopped ...
Not Taking Over the World
by Eliezer Yudkowsky — 2008-12-15T22:18:47.000Z

Followup to: What I Think, If Not Why

My esteemed co-blogger Robin Hanson accuses me of trying to take over the world.

Why, oh why must I be so misunderstood?

(Well, it's not like I don't enjoy certain misunderstandings. Ah, I remember the first time someone seriously and not in a joking way accused me of trying to take over the world. On that day I felt like a true mad scientist, though I lacked a castle and hunchbacked assistant.)

But if you're working from the premise of a hard takeoff - an Artificial Intelligence that self-improves at an extremely rapid rate - and you suppose such extra-ordinary depth of insight and precision of craftsmanship that you can actually specify the AI's goal system instead of automatically failing -

- then it takes some work to come up with a way not to take over the world.

Robin talks up the drama inherent in the intelligence explosion, presumably because he feels that this is a primary source of bias. But I've got to say that Robin's dramatic story, does not sound like the story I tell of myself. There, the drama comes from tampering with such extreme forces that every single idea you invent is wrong. The standardized Final Apocalyptic Battle of Good Vs. Evil would be trivial by comparison; then all you have to do is put forth a desperate effort. Facing an adult problem in a neutral universe isn't so straightforward. Your enemy is yourself, who will automatically destroy the world, or just fail to accomplish anything, unless you can defeat you. - That is the drama I crafted into the story I tell myself, for I too would disdain anything so cliched as Armageddon.

So, Robin, I'll ask you something of a probing question. Let's say that someone walks up to you and grants you unlimited power.

What do you do with it, so as to not take over the world?

Do you say, "I will do nothing - I take the null action"?

But then you have instantly become a malevolent God, as Epicurus said:

Is God willing to prevent evil, but not able? Then he is not omnipotent.
Is he able, but not willing? Then he is malevolent.
Is both able, and willing? Then whence cometh evil?
Is he neither able nor willing? Then why call him God.

Peter Norvig said, "Refusing to act is like refusing to allow time to pass." The null action is also a choice. So have you not, in refusing to act, e...
For The People Who Are Still Alive
by Eliezer Yudkowsky — 2008-12-14T17:13:03.000Z

Max Tegmark observed that we have three independent reasons to believe we live in a Big World: A universe which is large relative to the space of possibilities. For example, on current physics, the universe appears to be spatially infinite (though I'm not clear on how strongly this is implied by the standard model).

If the universe is spatially infinite, then, on average, we should expect that no more than 10^10^29 meters away is an exact duplicate of you. If you're looking for an exact duplicate of a Hubble volume - an object the size of our observable universe - then you should still on average only need to look 10^10^115 lightyears. (These are numbers based on a highly conservative counting of "physically possible" states, e.g. packing the whole Hubble volume with potential protons at maximum density given by the Pauli Exclusion principle, and then allowing each proton to be present or absent.)

The most popular cosmological theories also call for an "inflationary" scenario in which many different universes would be eternally budding off, our own universe being only one bud. And finally there are the alternative decoherent branches of the grand quantum distribution, aka "many worlds", whose presence is unambiguously implied by the simplest mathematics that fits our quantum experiments.

Ever since I realized that physics seems to tell us straight out that we live in a Big World, I've become much less focused on creating lots of people, and much more focused on ensuring the welfare of people who are already alive.

If your decision to not create a person means that person will never exist at all, then you might, indeed, be moved to create them, for their sakes. But if you're just deciding whether or not to create a new person here, in your own Hubble volume and Everett branch, then it may make sense to have relatively lower populations within each causal volume, living higher qualities of life. It's not like anyone will actually fail to be born on account of that decision - they'll just be born predominantly into regions with higher standards of living.

Am I sure that this statement, that I have just emitted, actually makes sense?

Not really. It dabbles in the dark arts of anthropics, and the Dark Arts don't get much murkier than that. Or to say it without the chaotic inversion: I am stupid with respect to...
BHTV: de Grey and Yudkowsky
by Eliezer Yudkowsky — 2008-12-13T15:28:28.000Z

My latest on Bloggingheads.tv is up. BHTV wanted someone to interview Aubrey de Grey of the Methuselah Foundation about basic research in antiagathics, and they picked me to do it. It made the interview somewhat difficult, since Aubrey and I already agree about most things, but we managed to soldier on. The interview is mostly Aubrey talking, as it should be.
You Only Live Twice
by Eliezer Yudkowsky — 2008-12-12T19:14:32.000Z

"It just so happens that your friend here is only mostly dead. There's a big difference between mostly dead and all dead."
-- The Princess Bride

My co-blogger Robin and I may disagree on how fast an AI can improve itself, but we agree on an issue that seems much simpler to us than that: At the point where the current legal and medical system gives up on a patient, they aren't really dead.

Robin has already said much of what needs saying, but a few more points:

• Ben Best's Cryonics FAQ, Alcor's FAQ, Alcor FAQ for scientists, Scientists' Open Letter on Cryonics

• I know more people who are planning to sign up for cryonics Real Soon Now than people who have actually signed up. I expect that more people have died while cryocrastinating than have actually been cryopreserved. If you've already decided this is a good idea, but you "haven't gotten around to it", sign up for cryonics NOW. I mean RIGHT NOW. Go to the website of Alcor or the Cryonics Institute and follow the instructions.

• Cryonics is usually funded through life insurance. The following conversation from an Overcoming Bias meetup is worth quoting:

Him: I've been thinking about signing up for cryonics when I've got enough money.

Me: Um... it doesn't take all that much money.

Him: It doesn't?

Me: Alcor is the high-priced high-quality organization, which is something like $500-$1000 in annual fees for the organization, I'm not sure how much. I'm young, so I'm signed up with the Cryonics Institute, which is $120/year for the membership. I pay $180/year for more insurance than I need - it'd be enough for Alcor too.

Him: That's ridiculous.

Me: Yes.

Him: No, really, that's ridiculous. If that's true then my decision isn't just determined, it's overdetermined.

Me: Yes. And there's around a thousand people worldwide [actually 1400] who are signed up for cryonics. Figure that at most a quarter of those did it for systematically rational reasons. That's a high upper bound on the number of people on Earth who can reliably reach the right conclusion on massively overdetermined issues.

• Cryonics is not marketed well - or at all, really. There's no salespeople who get commissions. There is no one to hold your hand through signin...
What I Think, If Not Why
by Eliezer Yudkowsky — 2008-12-11T17:41:43.000Z

Reply to: Two Visions Of Heritage

Though it really goes tremendously against my grain - it feels like sticking my neck out over a cliff (or something) - I guess I have no choice here but to try and make a list of just my positions, without justifying them. We can only talk justification, I guess, after we get straight what my positions are. I will also leave off many disclaimers to present the points compactly enough to be remembered.

• A well-designed mind should be much more efficient than a human, capable of doing more with less sensory data and fewer computing operations. It is not infinitely efficient and does not use zero data. But it does use little enough that local pipelines such as a small pool of programmer-teachers and, later, a huge pool of e-data, are sufficient.

• An AI that reaches a certain point in its own development becomes able to (sustainably, strongly) improve itself. At this point, recursive cascades slam over many internal growth curves to near the limits of their current hardware, and the AI undergoes a vast increase in capability. This point is at, or probably considerably before, a minimally transhuman mind capable of writing its own AI-theory textbooks - an upper bound beyond which it could swallow and improve its entire design chain.

• It is likely that this capability increase or "FOOM" has an intrinsic maximum velocity that a human would regard as "fast" if it happens at all. A human week is ~1e15 serial operations for a population of 2GHz cores, and a century is ~1e19 serial operations; this whole range is a narrow window. However, the core argument does not require one-week speed and a FOOM that takes two years (~1e17 serial ops) will still carry the weight of the argument.

• The default case of FOOM is an unFriendly AI,
built by
researchers with shallow insights. This AI becomes able to improve
itself in a haphazard way, makes various changes that are net
improvements but may introduce value drift, and then gets smart enough
to do guaranteed self-improvement, at which point its values freeze
(forever).

• The desired case of FOOM is a Friendly AI,
built using deep insight, so that the AI never makes any changes to
itself that potentially change its internal values; all such changes
are guaranteed using strong techniques that allow for a b...
The Mechanics of Disagreement
by Eliezer Yudkowsky — 2008-12-10T14:01:44.000Z

Two ideal Bayesians cannot have common knowledge of disagreement; this is a theorem. If two rationalist-wannabes have common knowledge of a disagreement between them, what could be going wrong?

The obvious interpretation of these theorems is that if you know that a cognitive machine is a rational processor of evidence, its beliefs become evidence themselves.

If you design an AI and the AI says "This fair coin came up heads with 80% probability", then you know that the AI has accumulated evidence with an likelihood ratio of 4:1 favoring heads - because the AI only emits that statement under those circumstances.

It's not a matter of charity; it's just that this is how you think the other cognitive machine works.

And if you tell an ideal rationalist, "I think this fair coin came up heads with 80% probability", and they reply, "I now think this fair coin came up heads with 25% probability", and your sources of evidence are independent of each other, then you should accept this verdict, reasoning that (before you spoke) the other mind must have encountered evidence with a likelihood of 1:12 favoring tails.

But this assumes that the other mind also thinks that you're processing evidence correctly, so that, by the time it says "I now think this fair coin came up heads, p=.25", it has already taken into account the full impact of all the evidence you know about, before adding more evidence of its own.

If, on the other hand, the other mind doesn't trust your rationality, then it won't accept your evidence at face value, and the estimate that it gives won't integrate the full impact of the evidence you observed.

So does this mean that when two rationalists trust each other's rationality less than completely, then they can agree to disagree?

It's not that simple. Rationalists should not trust themselves entirely, either.

So when the other mind accepts your evidence at less than face value, this doesn't say "You are less than a perfect rationalist", it says, "I trust you less than you trust yourself; I think that you are discounting your own evidence too little."

Maybe your raw arguments seemed to you to have a strength of 40:1, but you discounted for your own irrationality to a strength of 4:1, but the other mind thinks you still overestimate yourself and so it assumes that the actual force of the argument was 2:1.

And if you believe that the other mind is discounting...
Bay Area Meetup Wed 12/10 @8pm
by Eliezer Yudkowsky — 2008-12-10T00:13:46.000Z

Reminder: the second regular Overcoming Bias meetup is tomorrow, Wednesday at 8pm, at Techshop in Menlo park. Please RSVP so they know how many people are coming.

If you're hearing about this for the first time, sign up for the Meetup group so you get future announcements! And please RSVP when you get them! We don't want to have to post this to the main blog every time.

Robin Gane-McCalla will present some of his ideas on defining intelligence. As always, anyone with a paper to pass around, abstract to read out loud, or rationality-related cool video to show, is cordially invited to bring it along.
Disjunctions, Antipredictions, Etc.
by Eliezer Yudkowsky — 2008-12-09T15:13:04.000Z

Followup to: Underconstrained Abstractions

Previously:

So if it's not as simple as just using the one trick of finding abstractions you can easily verify on available data, what are some other tricks to use?

There are several, as you might expect...

Previously I talked about "permitted possibilities". There's a trick in debiasing that has mixed benefits, which is to try and visualize several specific possibilities instead of just one.

The reason it has "mixed benefits" is that being specific, at all, can have biasing effects relative to just imagining a typical case. (And believe me, if I'd seen the outcome of a hundred planets in roughly our situation, I'd be talking about that instead of all this Weak Inside View stuff.)

But if you're going to bother visualizing the future, it does seem to help to visualize more than one way it could go, instead of concentrating all your strength into one prediction.

So I try not to ask myself "What will happen?" but rather "Is this possibility allowed to happen, or is it prohibited?" There are propositions that seem forced to me, but those should be relatively rare - the first thing to understand about the future is that it is hard to predict, and you shouldn't seem to be getting strong information about most aspects of it.

Of course, if you allow more than one possibility, then you have to
discuss more than one possibility, and the total length of your post
gets longer. If you just eyeball the length of the post, it looks like
an unsimple theory; and then talking about multiple possibilities makes
you sound weak and uncertain.

As Robyn Dawes notes,

"In their summations lawyers avoid arguing from disjunctions in favor of conjunctions. (There are not many closing arguments that end, "Either the defendant was in severe financial straits and murdered the decedent to prevent his embezzlement from being exposed or he was passionately in love with the same coworker and murdered the decedent in a fit of jealous rage or the decedent had blocked the defendant's promotion at work and the murder was an act of revenge. The State has given you solid evidence to support each of these alternatives, all of which would lead to the same conclusion: first-degree murder.") Rationally, of course, disjunctions are much more probable than are conjunct

...
True Sources of Disagreement
by Eliezer Yudkowsky — 2008-12-08T15:51:58.000Z

Followup to: Is That Your True Rejection?

I expected from the beginning, that the difficult part of two rationalists reconciling a persistent disagreement, would be for them to expose the true sources of their beliefs.

One suspects that this will only work if each party takes responsibility for their own end; it's very hard to see inside someone else's head. Yesterday I exhausted myself mentally while out on my daily walk, asking myself the Question "What do you think you know, and why do you think you know it?" with respect to "How much of the AI problem compresses to large insights, and how much of it is unavoidable nitty-gritty?" Trying to either understand why my brain believed what it believed, or else force my brain to experience enough genuine doubt that I could reconsider the question and arrive at a real justification that way. It's hard to see how Robin Hanson could have done any of this work for me.

Presumably a symmetrical fact holds about my lack of access to the real reasons why Robin believes what he believes. To understand the true source of a disagreement, you have to know why both sides believe what they believe - one reason why disagreements are hard to resolve.

Nonetheless, here's my guess as to what this Disagreement is about:

If I had to pinpoint a single thing that strikes me as "disagree-able" about the way Robin frames his analyses, it's that there are a lot of opaque agents running around, little black boxes assumed to be similar to humans, but there are more of them and they're less expensive to build/teach/run. They aren't even any faster, let alone smarter. (I don't think that standard economics says that doubling the population halves the doubling time, so it matters whether you're making more minds or faster ones.)

This is Robin's model for uploads/ems, and his model for AIs doesn't seem to look any different. So that world looks like this one, except that the cost of "human capital" and labor is dropping according to (exogenous) Moore's Law , and it ends up that economic growth doubles every month instead of every sixteen years - but that's it. Being, myself, not an economist, this does look to me like a viewpoint with a distinctly economic zeitgeist.

In my world, you look inside the black box. (And, to be symmetrical, I don't spend mu...