trentmkelly/LessWrong-43k huggingface dataset:
What big goals do we have?
by cousin_it — 2010-01-19T16:35:56.100Z

Sometime ago Jonii wrote:

I mean, paperclip maximizer is seriously ready to do anything to maximize paperclips. It really takes the paperclips seriously.

When I'm hungry I eat, but then I don't go on eating some more just to maximize a function. Eating isn't something I want a lot of. Likewise I don't want a ton of survival, just a bounded amount every day. Let's define a goal as big if you don't get full: every increment of effort/achievement is valuable, like paperclips to Clippy. Now do we have any big goals? Which ones?

Save the world. A great goal if you see a possible angle of attack, which I don't. The SIAI folks are more optimistic, but if they see a chink in the wall, they're yet to reveal it.

Help those who suffer. Morally upright but tricky to execute: James Shikwati, Dambisa Moyo and Kevin Myers show that even something as clear-cut as aid to Africa can be viewed as immoral. Still a good goal for anyone, though.

Procreate. This sounds fun! Fortunately, the same source that gave us this goal also gave us the means to achieve it, and intelligence is not among them. :-) And honestly, what sense in making 20 kids just to play the good-soldier routine for your genes? There's no unique "you gene" anyway, in several generations your descendants will be like everyone else's. Yeah, kids are fun, I'd like two or three.

Follow your muse. Music, comedy, videogame design, whatever. No limit to achievement! A lot of this is about signaling: would you still bother if all your successes were attributed to someone else's genetic talent? But even apart from the signaling angle, there's still the worrying feeling that entertainment is ultimately useless, like humanity-scale wireheading, not an actual goal for us to reach.

Accumulate power, money or experiences. What for? I never understood that.

Advance science. As Erik Naggum put it:

The purpose of human existence is to learn and to understand as much as we can of what came before us, so we can further the sum total of human knowledge in our life.

Don't know, but I'm pretty content with my life lately. Should I have a big goal at all? How about you?
The Prediction Hierarchy
by RobinZ — 2010-01-19T03:36:32.248Z

Related: Advancing Certainty, Reversed Stupidity Is Not Intelligence

The substance of this post is derived from a conversation in the comment thread which I have decided to promote. Teal;deer: if you have to rely on a calculation you may have gotten wrong for your prediction, your expectation for the case when your calculation is wrong should use a simpler calculation, such as reference class forecasting.

Edit 2010-01-19: Toby Ord mentions in the comments Probing the Improbable: Methodological Challenges for Risks with Low Probabilities and High Stakes (PDF) by Toby Ord, Rafaela Hillerbrand, and Anders Sandberg of the Future of Humanity Institute, University of Oxford. It uses a similar mathematical argument, but is much more substantive than this.

A lottery has a jackpot of a million dollars. A ticket costs one dollar. Odds of a given ticket winning are approximately one in forty million. If your utility is linear in dollars, should you bet?

The obvious (and correct) answer is "no". The clever (and incorrect) answer is "yes", as follows:

According to your calculations, "this ticket will not win the lottery" is true with probability 99.9999975%. But can you really be sure that you can calculate anything to that good odds? Surely you couldn't expect to make forty million predictions of which you were that confident and only be wrong once. Rationally, you ought to ascribe a lower confidence to the statement: 99.99%, for example. But this means a 0.01% chance of winning the lottery, corresponding to an expected value of a hundred dollars. Therefore, you should buy the ticket.

The logic is not obviously wrong, but where is the error?

First, let us write out the calculation algebraically. Let E(L) be the expected value of playing the lottery. Let p(L) be your calculated probability that the lottery will pay off. Let p(C) be your probability that your calculations are correct. Finally, let j represent the value of the jackpot and let t represent the price of the ticket. The obvious way to write the clever theory is:

E(L) = max(p(L), 1-p(C)) * j - t

This doesn't sound quite right, though - surely you should ascribe a higher confidence when you calculate a higher probability. That said, when p(L) is much less than p(C), it shouldn't make a large difference. The straightforward way to account for this is to take p(C) as the probability that p(L) is...
Advancing Certainty
by komponisto — 2010-01-18T09:51:31.050Z

Related: Horrible LHC Inconsistency, The Proper Use of Humility

Overconfidence, I've noticed, is a big fear around these parts. Well, it is a known human bias, after all, and therefore something to be guarded against. But I am going to argue that, at least in aspiring-rationalist circles, people are too afraid of overconfidence, to the point of overcorrecting -- which, not surprisingly, causes problems. (Some may detect implications here for the long-standing Inside View vs. Outside View debate.)

Here's Eliezer, voicing the typical worry:

[I]f you asked me whether I could make one million statements of authority equal to "The Large Hadron Collider will not destroy the world", and be wrong, on average, around once, then I would have to say no.

I now suspect that misleading imagery may be at work here. A million statements -- that sounds like a lot, doesn't it? If you made one such pronouncement every ten seconds, a million of them would require you to spend months doing nothing but pontificating, with no eating, sleeping, or bathroom breaks. Boy, that would be tiring, wouldn't it? At some point, surely, your exhausted brain would slip up and make an error. In fact, it would surely make more than one -- in which case, poof!, there goes your calibration.

No wonder, then, that people claim that we humans can't possibly hope to attain such levels of certainty. Look, they say, at all those times in the past when people -- even famous scientists! -- said they were 99.999% sure of something, and they turned out to be wrong. My own adolescent self would have assigned high confidence to the truth of Christianity; so where do I get the temerity, now, to say that the probability of this is 1-over-oogles-and-googols?

[EDIT: Unnecessary material removed.]

A probability estimate is not a measure of "confidence" in some psychological sense. Rather, it is a measure of the strength of the evidence: how much information you believe you have about reality. So, when judging calibration, it is not really appropriate to imagine oneself, say, judging thousands of criminal trials, and getting more than a few wrong here and there (because, after all, one is human and tends to make mistakes). Let me instead propose a less misleading image: picture yourself programming your model of the world (in technical terms, your prior probability distribution) into a computer, and then feed...
Sorting Out Sticky Brains
by Alicorn — 2010-01-18T04:18:53.036Z

tl;dr: Just because it doesn't seem like we should be able to have beliefs we acknowledge to be irrational, doesn't mean we don't have them. If this happens to you, here's a tool to help conceptualize and work around that phenomenon.

There's a general feeling that by the time you've acknowledged that some belief you hold is not based on rational evidence, it has already evaporated. The very act of realizing it's not something you should believe makes it go away. If that's your experience, I applaud your well-organized mind! It's serving you well. This is exactly as it should be.

If only we were all so lucky.

Brains are sticky things. They will hang onto comfortable beliefs that don't make sense anymore, view the world through familiar filters that should have been discarded long ago, see significances and patterns and illusions even if they're known by the rest of the brain to be irrelevant. Beliefs should be formed on the basis of sound evidence. But that's not the only mechanism we have in our skulls to form them. We're equipped to come by them in other ways, too. It's been observed1 that believing contradictions is only bad because it entails believing falsehoods. If you can't get rid of one belief in a contradiction, and that's the false one, then believing a contradiction is the best you can do, because then at least you have the true belief too.

The mechanism I use to deal with this is to label my beliefs "official" and "unofficial". My official beliefs have a second-order stamp of approval. I believe them, and I believe that I should believe them. Meanwhile, the "unofficial" beliefs are those I can't get rid of, or am not motivated to try really hard to get rid of because they aren't problematic enough to be worth the trouble. They might or might not outright contradict an official belief, but regardless, I try not to act on them.

To those of you with well-ordered minds (for such lucky people seem to exist, if we believe some of the self-reports on this very site), this probably sounds outrageous. If I know they're probably not true... And I do. But they still make me expect things. They make me surprised when those expectations are flouted. If I'm asked about their subjects when tired, or not prepared for the question, they'll leap out of my m...
Tips and Tricks for Answering Hard Questions
by Wei Dai — 2010-01-17T23:56:20.904Z

I've collected some tips and tricks for answering hard questions, some of which may be original, and others I may have read somewhere and forgotten the source of. Please feel free to contribute more tips and tricks, or additional links to the sources or fuller explanations.

Don't stop at the first good answer. We know that human curiosity can be prematurely satiated. Sometimes we can quickly recognize a flaw in an answer that initially seemed good, but sometimes we can't, so we should keep looking for flaws and/or better answers.

Explore multiple approaches simultaneously. A hard question probably has multiple approaches that are roughly equally promising, otherwise it wouldn't be a hard question (well, unless it has no promising approaches). If there are several people attempting to answer it, they should explore different approaches. If you're trying to answer it alone, it makes sense to switch approaches (and look for new approaches) once a while.

Trust your intuitions, but don't waste too much time arguing for them. If several people are attempting to answer the same question and they have different intuitions about how best to approach it, it seems efficient for each to rely on his or her intuition to choose the approach to explore. It only makes sense to spend a lot of time arguing for your own intuition if you have some reason to believe that other people's intuitions are much worse than yours.

Go meta. Instead of attacking the question directly, ask "How should I answer a question like this?" It seems that when people are faced with a question, even one that has stumped great minds for ages, many just jump in and try to attack it with whatever intellectual tools they have at hand. For really hard questions, we may need to look for, or build, new tools.

Dissolve the question. Sometimes, the question is meaningless and asking it is just a cognitive error. If you can detect and correct the error then the question may just go away.

Sleep on it. I find that I tend to have a greater than average number of insights in the period of time just after I wake up and before I get out of bed. Our brains seem to continue to work while we're asleep, and it may help to prime it by reviewing the problem before going to sleep. (I think Eliezer wrote a post or comment to this effect, but I can't find it now.)

Be ready to recognize a good answer when you see it. The history of ...
Dennett's heterophenomenology
by Richard_Kennaway — 2010-01-16T20:40:18.505Z

In an earlier comment, I conflated heterophenomenology in the general sense of taking introspective accounts as data to be explained rather than direct readouts of the truth, with Dennett's particular approach to explaining those data. So to correct myself, I say that it is Dennett, rather than heterophenomenology, that claims that there is no such thing as consciousness. Dennett denies that he does, but I disagree. I defend this view here.

I have to admit at this point that I have not read "Consciousness Explained". Had either of the library's copies been on the shelves last Tuesday I would have done by now, but instead I found his later book (and his most recent on the topic), "Sweet Dreams: Philosophical Obstacles to a Science of Consciousness". The subtitle suggests a drawing back from the confidence of the earlier title, as does that of the book in between. The book confirms me in my impression that the ideas of "C.E." have been in the air so long (the air of hard SF, sciblogs, and the like, not to mention Phil Goetz's recent posts) that reading the primary source 19 years on would be nothing more than an exercise in checkbox-ticking.

I'll give a brief run-through of "Sweet Dreams" and then carry on the argument.

The book is primarily writing against, a response to objections arising from earlier works.

In chapter 1 he shoots down the "Zombic Hunch", the idea that a being could be physically identical to a human, but lack consciousness, and therefore that consciousness must be non-physical. I'll take it that we all agree the zombie story is insane.

Chapter 2 introduces the concept of heterophenomenology. This has already been introduced to LW.

The corpse of the Zombic Hunch got up again and walked, so in Chapter 3 he shoots it down again.

Chapters 4 and 5 attack qualia. They don't exist, says Dennett, because of such anomalies as change blindness, Capgras syndrome, and various thought-experiments that defy all coherent accounts of what qualia are.

Chapter 6 is entitled "Are We Explaining Consciousness yet?" (Er, what was that first book called?) It cites neurological research and briefly sets out the Multiple Drafts hypothesis. He confronts critics who say (which I also say) that Dennett is really denying the existence of consciousness, not giving an account of it. His answer is that he is giving (or s...
The Wannabe Rational
by MrHen — 2010-01-15T20:09:32.662Z

I have a terrifying confession to make: I believe in God.

This post has three prongs:

First: This is a tad meta for a full post, but do I have a place in this community? The abstract, non-religious aspect of this question can be phrased, "If someone holds a belief that is irrational, should they be fully ousted from the community?" I can see a handful of answers to this question and a few of them are discussed below.

Second: I have nothing to say about the rationality of religious beliefs. What I do want to say is that the rationality of particular irrationals is not something that is completely answered after their irrationality is ousted. They may be underneath the sanity waterline, but there are multiple levels of rationality hell. Some are deeper than others. This part discusses one way to view irrationals in a manner that encourages growth.

Third: Is it possible to make the irrational rational? Is it possible to take those close to the sanity waterline and raise them above? Or, more personally, is there hope for me? I assume there is. What is my responsibility as an aspiring rationalist? Specifically, when the community complains about a belief, how should I respond?

My Place in This Community

So, yeah. I believe in God. I figure my particular beliefs are a little irrelevant at this point. This isn't to say that my beliefs aren't open for discussion, but here and now I think there are better things to discuss. Namely, whether talking to people like me is within the purpose of LessWrong. Relevant questions have to do with my status and position at LessWrong. The short list:

Should I have kept this to myself? What benefit does an irrational person have for confessing their irrationality? (Is this even possible? Is this post an attempted ploy?) I somewhat expect this post and the ensuing discussion to completely wreck my credibility as a commentator and participant.

Presumably, there is a level of entry to LessWrong that is enforced. Does this level include filtering out certain beliefs and belief systems? Or is the system merit-based via karma and community voting? My karma is well above the level needed to post and my comments generally do better than worse. A merit-based system would prevent me from posting anything about religion or other irrational things, but is there a deeper problem? (More discussion below.) Should LessWrong /kic

...
In defense of the outside view
by cousin_it — 2010-01-15T11:01:18.900Z

I think some of our recent arguments against applying the outside view are wrong.

1. In response to taw's post, Eliezer paints the outside view argument against the Singularity thus:

...because experiments show that people could do better at predicting how long it will take them to do their Christmas shopping by asking "How long did it take last time?" instead of trying to visualize the details.

This is an unfair representation. One of the poster-child cases for the outside view (mentioned by Eliezer, no less!) dealt with students trying to estimate completion times for their academic projects. And what is AGI if not a research project? One might say AGI is too large for the analogy to work, but outside view helpfully tells us that large projects aren't any more immune to failures and schedule overruns :-)

2. In response to my comment claiming that Dennett didn't solve the problem of consciousness "because philosophers don't solve problems", ciphergoth writes:

This "outside view abuse" is getting a little extreme. Next it will tell you that Barack Obama isn't President, because people don't become President.

The outside view may be rephrased as "argument from typicality". If we'd just heard of this random dude named Barack Obama, we'd be perfectly justified in saying he won't become President! Which would be the proper analogy to first hearing about Dennett and his work. Another casual application of the outside view corroborates the conclusion: what other problems has Dennett solved? Is the problem of consciousness the first problem he solved? Does this seem typical of anything?

3. Technologos attacks taw's post, again, with the following argument:

"beliefs that the future will be just like the past" have a zero success rate.

For each particular highly speculative technology, we can assert that it won't appear with high confidence (let's say 90%). But this doesn't mean the future will be the same in all respects! The conjunction of many 90%-statements (X won't appear, AND Y won't appear, AND etc.) gets assigned the product, a very low confidence, as it should. We're sure that some new technologies will arise, we just don't know which ones. Fusion power? Flying cars? We've been on the fast track to those for some time now, and they still sound less far out then the Singularity! Anyone who's worked with tech for any length of time can recite a looooong lis...
The Preference Utilitarian’s Time Inconsistency Problem
by Wei Dai — 2010-01-15T00:26:04.781Z

In May of 2007, DanielLC asked at Felicifa, an “online utilitarianism community”:

If preference utilitarianism is about making peoples’ preferences and the universe coincide, wouldn't it be much easier to change peoples’ preferences than the universe?

Indeed, if we were to program a super-intelligent AI to use the utility function U(w) = sum of w’s utilities according to people (i.e., morally relevant agents) who exist in world-history w, the AI might end up killing everyone who is alive now and creating a bunch of new people whose preferences are more easily satisfied, or just use its super intelligence to persuade us to be more satisfied with the universe as it is.

Well, that can’t be what we want. Is there an alternative formulation of preference utilitarianism that doesn’t exhibit this problem? Perhaps. Suppose we instead program the AI to use U’(w) = sum of w’s utilities according to people who exist at the time of decision. This solves the Daniel’s problem, but introduces a new one: time inconsistency.

The new AI’s utility function depends on who exists at the time of decision, and as that time changes and people are born and die, its utility function also changes. If the AI is capable of reflection and self-modification, it should immediately notice that it would maximize its expected utility, according to its current utility function, by modifying itself to use U’’(w) = sum of w’s utilities according to people who existed at time T0, where T0 is a constant representing the time of self-modification.

The AI is now reflectively consistent, but is this the right outcome? Should the whole future of the universe be shaped only by the preferences of those who happen to be alive at some arbitrary point in time? Presumably, if you’re a utilitarian in the first place, this is probably not the kind of utilitarianism that you’d want to subscribe to.

So, what is the solution to this problem? Robin Hanson’s approach to moral philosophy may work. It tries to take into account everyone’s preferences&m

...
Back of the envelope calculations around the singularity. 
by  — 2010-01-15T00:14:13.849Z

Inspired by the talk by Anna Salamon I decided to do my own calculations about the future. This post is a place for discussion about mine and others calculations.

To me there are two possible paths for the likely development of intelligence, that I can identify.

World 1) Fast and conceptually clean. Intelligence is a concrete value like the number of neutrons in a reactor. I assign a 20% chance of this.

World 2) Slow and messy. Intelligence is contextual, much like say fitness in evolutionary biology. Proofs of intelligence of a system are only doable by a much higher intelligence entity, as it will involve discussing the complex environment. I'd assign about an 60% chance to this.

Worlds 3) Other. The other 20% chance is the rest of the scenarios that are not either of these two.

Both types of AI have the potential to change the world, both possibly destroying humanity if we don't use them correctly. So they both have the same rewards.

So for world 1, I'll go with the same figures as Anna Salamon, because I can't find strong arguments against them (and it will serve as a refresher )

Probability of an eventual AI (before humanity dies otherwise) = 80%

Probability that AI will kill us = 80%

Probability that we manage safeguards = 40%

Probability that current work will save us = 30%

So we get 7%*20%. Gives us 1.4%

So for world 2. Assume we have an SIAI that is working on the problem of how to make messy AI Friendly or at least as Friendly as possible. It seems less likely we would make AI and harder to create safeguards as they have to act over longer time.

Probability of an eventual AI (before humanity dies otherwise) = 70%

Probability that AI will kill us (and/or we will have to give up humanity due to hard scrapple evolution) = 80%

Probability that we manage safeguards = 30%

Probability that current work will save us = 20%

So we get a factor of 3% times 60% give a 1.8%.

Both have the factor of 7billion lives times n, so that can be discounted. They pretty much weigh the same. Or as near as dammit for a back of the envelope calcs, considering my meta-uncertainty is high as well.

They do however interfere. The right action in world 1 is not the same as the right action in world 2. Working on Friendliness of conceptually clean AI and suppressing all work and discussion on messy AI hurts world 2 as it increases the chance we might end up with messy UFAI. There i...