trentmkelly/LessWrong-43k huggingface dataset:
Risks from AI and Charitable Giving
by XiXiDu — 2012-03-13T13:54:36.349Z

If you’re interested in being on the right side of disputes, you will refute your opponents' arguments. But if you're interested in producing truth, you will fix your opponents' arguments for them. To win, you must fight not only the creature you encounter; you [also] must fight the most horrible thing that can be constructed from its corpse.

-- Black Belt Bayesian

This is an informal post meant as a reply to a post by user:utilitymonster, 'What is the best compact formalization of the argument for AI risk from fast takeoff?'

I hope to find the mental strength to put more effort into it in future to improve it. But since nobody else seems to be willing to take a critical look at the overall topic I feel that doing what I can is better than doing nothing.

Please review the categories 'Further Reading' and 'Notes and References'.

Contents

Abstract

Requirements for an Intelligence Explosion

Requirements for SIAI to constitute an optimal charity

Further Reading

Notes and References

Abstract

In this post I just want to take a look at a few premises (P#) that need to be true simultaneously to make the SIAI a wortwhile charity from the point of view of someone trying to do as much good as possible by contributing money. I am going to show that the case of risks from AI is strongly conjunctive, that without a concrete and grounded understanding of AGI an abstract analysis of the issues is going to be very shaky, and that therefore SIAI is likely to be a bad choice as a charity. In other words, that which speaks in favor of SIAI does mainly consist of highly specific, conjunctive, non-evidence-backed speculations on possible bad outcomes.

Requirements for an Intelligence Explosion

P1 Fast, and therefore dangerous, recursive self-improvement is logically possible.

It took almost four hundred years to prove Fermat’s Last Theorem. The final proof is over a hundred pages long. Over a hundred pages! And we are not talking about something like an artificial general intelligence that can magically make itself smart enough to prove such theorems and many more that no human being would be capable of proving. Fermat’s Last Theorem simply states “no three positive integers a, b, and c can satisfy the equation a^n + b^n = c^n for any integer value of n greater than two.”

Even artificial intelligence researchers admit that "there cou...
[Link] Failed replications of the "elderly walking" priming effect
by Morendil — 2012-03-13T07:55:32.085Z

Recently a controversy broke out over the replicability of a study John Bargh et al. published in 1996. The study reported that unconsciously priming a stereotype of elderly people caused subjects to walk more slowly. A recent replication attempt by Stephane Doyen et al., published in PLoS ONE, was unable to reproduce the results. (Less publicized, but surely relevant, is another non-replication by Hal Pashler et al.) (source)

This is interesting, if only because the study in question is one of the more famous examples of priming effects - it's the one I tend to use when I introduce people to the idea of priming. (Ironically, the failed replication study also mentions a further experimental manipulation that does show priming effects - affecting the experimenters rather than the subjects.) Bargh's reply is also unusual in that it focuses significantly on extra-scientific arguments, such as attacks on the open access business model of PLoS ONE.

I was instantly reminded of The Golem, which "debunks the view that scientific knowledge is a straightforward outcome of competent theorization, observation, and experimentation". The examples on relativity and solar neutrinos are particularly engaging - it's not just psychology where experimentation is problematic, but all of science.

The linked blog also contributes useful observations of its own, such as the "rhetorical function" of the additional experiment in Doyen's study, how online publication makes a difference in how easily experimental setups can be replicated, or a subtle point about our favorite villain, p-values.

EDIT: added link to source. Heartfelt thanks to the two readers who upvoted the version without the link. :)
What is the best compact formalization of the argument for AI risk from fast takeoff?
by utilitymonster — 2012-03-13T01:44:51.788Z

Many people complain that the Singularity Institute's "Big Scary Idea" (AGI leads to catastrophe by default) has not been argued for with the clarity of, say, Chalmers' argument for the singularity. The idea would be to make explicit what the premise-and-inference structure of the argument is, and then argue about the strength of those premises and inferences.

Here is one way you could construe one version of the argument for the Singularity Institute's "Big Scary Idea":

At some point in the development of AI, there will be a very swift increase in the optimization power of the most powerful AI, moving from a non-dangerous level to a level of superintelligence. (Fast takeoff)

This AI will maximize a goal function.

Given fast takeoff and maximizing a goal function, the superintelligent AI will have a decisive advantage unless adequate controls are used.

Adequate controls will not be used. (E.g. Won’t box/boxing won’t work)

Therefore, the superintelligent AI will have a decisive advantage

Unless that AI is designed with goals that stably align with ours, if the superintelligent AI has a decisive advantage, civilization will be ruined. (Friendliness is necessary)

Unless the first team that develops the superintelligent AI makes adequate preparations, the superintelligent AI will not have goals that stably align with ours.

Therefore, unless the first team that develops the superintelligent AI makes adequate preparations, civilization will be ruined shortly after fast takeoff

The first team that develops the superintelligent AI will fail to make adequate preparations

Therefore, civilization will be ruined shortly after fast takeoff.

Edit to add: premises should be read as assuming the truth of all above premises. E.g., (9) is assuming that we've created an artificial agent with a decisive advantage.

My questions are:

Have I made any errors in the argument structure?

Can anyone suggest an alternative argument structure?

Which of these premises seem the weakest to you?
LINK: Can intelligence explode?
by jake987722 — 2012-03-13T00:03:40.496Z

I thought many of you would be interested to know that the following paper just appeared in Journal of Consciousness Studies:

"Can Intelligence Explode?", by Marcus Hutter. (LINK HERE)

Abstract: The technological singularity refers to a hypothetical scenario in which technological advances virtually explode. The most popular scenario is the creation of super-intelligent algorithms that recursively create ever higher intelligences. It took many decades for these ideas to spread from science fiction to popular science magazines and finally to attract the attention of serious philosophers. David Chalmers' (JCS 2010) article is the first comprehensive philosophical analysis of the singularity in a respected philosophy journal. The motivation of my article is to augment Chalmers' and to discuss some issues not addressed by him, in particular what it could mean for intelligence to explode. In this course, I will (have to) provide a more careful treatment of what intelligence actually is, separate speed from intelligence explosion, compare what super-intelligent participants and classical human observers might experience and do, discuss immediate implications for the diversity and value of life, consider possible bounds on intelligence, and contemplate intelligences right at the singularity.

I have only just seen the paper and have not yet thread through it myself, but I thought we could use this thread for discussion.
TED-Ed Launch
by Lightwave — 2012-03-12T21:20:31.364Z

TED has launched a new initiative: TED Education

TED-Ed's mission is to capture and amplify the voices of great educators around the world. We do this by pairing extraordinary educators with talented animators to produce a new library of curiosity-igniting videos.

http://education.ted.com/

So maybe we could get someone to do a short talk/lecture on rationality related topic(s) (maybe cognitive biases?)? Or anything else that is related to raising the sanity waterline. Or at least just suggest the idea.
Is intelligence explosion necessary for doomsday?
by Swimmy — 2012-03-12T21:12:07.369Z

I searched for articles on the topic and couldn't find any.

It seems to me that intelligence explosion makes human annihilation much more likely, since superintelligences will certainly be able to outwit humans, but that a human-level intelligence that could process information much faster than humans would certainly be a large threat itself without any upgrading. It could still discover programmable nanomachines long before humans do, gather enough information to predict how humans will act, etc. We already know that a human-level intelligence can "escape from the box." Not 100% of the time, but a real AI will have the opportunity for many more trials, and its processing abilities should make it far more quick-witted than we are.

I think a non-friendly AI would only need to be 20 years or so more advanced than the rest of humanity to pose a major threat, especially if self-replicating nanomachines are possible. Skeptics of intelligence explosion should still be worried about the creation of computers with unfriendly goal systems. What am I missing?
Real-life expected utility maximization [response to XiXiDu]
by  — 2012-03-12T19:03:27.935Z

This was supposed to be a comment under XiXiDu's recent post but it got a bit unwieldy so I'm posting it top-level.

XiXiDu starts his post with:

I would like to ask for help on how to use expected utility maximization, in practice, to maximally achieve my goals.

I think the best single-sentence answer is: don't.

The usual way of making decisions is to come up with intuitive evaluations of various options and go with the one that feels most attractive. Sometimes you will feel (intuitively) that it would be good to spend some more time thinking about decision. So you'll put your initial intuitions into words (which are chosen by another intuitive black-box), come up with a causal models of your situation (generated by yet another intuitive module), then experience intuitive feelings about those thoughts, maybe come up with alternative thoughts and compare them (intuitively) or maybe turn those feelings-about-thoughts into second-order thoughts and continue the process until you run out of time, get bored (intuitively) or deliberatively decide that you've analyzed enough (by having run another progression of interweaving thoughts and intuitions in parallel to the first one).

In a sense, all thinking is intuition. You don't get to jump out of the system. There's no choice between using intuition and using some kind of completely different process called deliberative reasoning but rather a choice between using a small amount of object-level intuition vs lots of intuition turned upon itself.

That doesn't mean that we can't improve our thinking processes. Just that we do it by gaining knowledge and experience which then shape our intuitive thinking and not by somehow fundamentally altering their nature. An engineer and a composer both rely on intuition but it's the engineer that will succeed in building an internal combustion engine and the composer that will succeed in designing an aesthetically pleasing progression of sounds.

Mathematics is often pointed to as the foremost example of strict, logical thinking. Yet, mathematicians rely on intuition too. Mathematical proofs are considered trustworthy because the rules of proof formation are sufficiently simple that humans can train themselves to reliably distinguish proofs from non-proofs. A mathematician looks at a line in a proof and asks herself 'is that a correct application of logical inference rules?' ...
The Stable State is Broken
by Bakkot — 2012-03-12T18:31:06.963Z

or: Why Everything Is Terrible, An Overview.1

It sounds like a theory which explains too much. But it's not a theory, hardly even an explanation, more a pattern that manifests itself once you start trying to seriously answer rhetorical questions about the state of the world. From many perspectives, it's obvious to the point of being mundane, practically tautological, but sometimes such obvious facts are worth pointing out regardless.

The idea is this: The subset of participants which rises to prominence in any area does so because its members have traits helpful to becoming prominent, not necessarily because they have traits which are desirable. Thus, without ongoing and concerted effort, a great many arenas end up dominated by players employing strategies which are bad for everyone.

This comes up again and again:

Why does science (or rather, the publisher-based model thereof) so frequently produce results which are laughably wrong? Because those journals which don't publish retractions or reproductions will more frequently be the first to publish revolutionary results, and so become more widely read and widely cited. Journals don't attract authors by being as accurate as possible; they win by looking important.

Why do cigarette companies target kids and teens whenever they think they can get away with it, and breed tobacco for maximized nicotine? Because those companies which do will turn more profit and thus last longer and grow faster than those that don't, and so have more resources to devote to proliferating. Companies don't expand by playing fair; they win by making and keeping customers.

Why is the Make-A-Wish Foundation sitting on more donations than it knows what to do with when the Against Malaria Foundation could have used that money to save literally tens or hundreds of thousands of lives per year? Because knowing how to elicit donations is a skill almost completely unrelated to knowing how to spend donations, and because American children with cancer make for better advertising than African children with malaria. Charities don't get donations by making the best possible use of their money; they win by advertising effectively towards potential donors. (cf. Efficient Charity)

Why do governments inevitably end up run by career lawyers and politicians instead of scientists and economists2? Becau

...
Meetup : Austin, TX
by Vaniver — 2012-03-12T17:03:42.005Z

Discussion article for the meetup : Austin, TX

WHEN:
24 March 2012 01:30:00PM (-0500)

WHERE:
2222B Guadalupe St Austin, Texas 78712

Note that the Austin, TX meetup is not meeting March 17th; many of us will be out of town.

We'll be back at Caffe Medici on the 24th, and we sit on the second floor to the left, near (or often on) the stage. Hope to see you there!

Discussion article for the meetup : Austin, TX
Falsification
by TheatreAddict — 2012-03-12T03:59:08.854Z

Alright, so this is going to sound a bit silly. I'm fairly sure I've read this on the Sequences somewhere, but for the life of me I can't find it. A friend of mine insists that there is a fifty-fifty chance that we live in the Matrix. His argument is that every bit of evidence we have to say that we exist outside of the Matrix is already based off of the idea that we live outside of the Matrix, and that we really have no evidence either way. He says there isn't a way of falsifying that we're not in the Matrix.

Yet I feel like he's wrong, and just can't explain why. I keep repeating that we don't have any evidence to suggest that we live in the Matrix, so why would we bother believing it?

I feel like this could possibly be an analogy for the belief in God or something. >_> I'm tired, and I need help figuring this out.
Meetup : Melbourne social meetup
by shokwave — 2012-03-11T23:02:15.081Z

Discussion article for the meetup : Melbourne social meetup

WHEN:
16 March 2012 07:00:00PM (+1100)

WHERE:
see mailing list; Carlton, VIC

Melbourne's next social meetup is on the 16th of March - as usual, at Ben's house. See the mailing list for the address - or you can call or text 0432 862 932 for the address. Alternatively, you can email shokwave.sf@gmail.com or even message User:shokwave on LessWrong.

Some form of takeaway - probably Indian - will be organised on the night. There will be snacks available. Some of them might even be healthy! BYO drinks.

The mafia games seem to have been the highlight of recent social events, so we're going to try starting those earlier in the night and running more of them.

Please comment if you're planning on attending. We wholeheartedly welcome newcomers!

Discussion article for the meetup : Melbourne social meetup
The appeal of fictional conflict
by KatjaGrace — 2012-03-11T08:27:22.000Z

Robert Wiblin asks why stories celebrate conflict rather than compromise:

As I was watching the film Avatar and the cinemagoers around me were cheering on the Na’vi heroes in their fight against human invaders, I couldn’t help but wonder how many of us would actually want to live alongside such an uncompromising society…it is hardly an isolated case. In our stories we love idealistic heroes to fight for what they believe in against all odds…

We could tell stories of the countless political compromises reached through well-functioning democratic institutions. We could tell the stories of all the terrible wars that never happened because of careful diplomacy. We could tell the story of the merchant who buys low and sells high, leaving everyone they deal with a little better off. These are the everyday tales which make modern society so great to live in. But will any such movie gross a billion dollars in the near future? I suspect not.

Incidentally, the one line I still remember from Avatar:

They’re not going to give up their home — they’re not gonna make a deal. For what? Lite beer and shopping channel? There’s nothing we have that they want.

Nothing at all…oh, except control over the destruction of everything they care about. You’re right, you really have no bargaining power. As Rob elaborates further, the premise of the extreme conflict was so flimsy, one must infer that it was pretty important to have an extreme conflict.

Rob guesses the popularity of such stubborn warring in our stories is to do with what we subconsciously want our tastes to say about us. When we don’t pay the costs of fictional war, we may as well stand up for principles as strongly as possible.

I think he might be roughly right. But why wouldn’t finding good deals and balancing compromises well be ideals we would want to celebrate? When there are no costs to yourself, why aren’t you itching to go all out and celebrate the most extravagant tales of successful trading and extreme sagas of mutually beneficial political compromise?

I think because there is no point in demonstrating that you will compromise. As a default, everyone can be expected to compromise, because it’s the rational thing to do at the time. However it’s often good to look like you won’t easily compromise, so that other people will try to win y...
On the etiology of religious belief
by gwern — 2012-03-11T00:03:08.860Z

From "Trust in testimony and miracles":

I have been of late fascinated by the research of the developmental psychologist Paul L. Harris, who has investigated how young children acquire information through testimony. Harris gauges two psychological hypotheses. The first, which he attributes to Hume, is that children always assess the content of the information: they are more inclined to disbelieve information that widely differs from their earlier experience. The second, which he identifies with Reid's position is that children are naturally credulous; they are inclined to indiscriminately believe what others testify, no matter who they are or what they tell.

...Harris found that children do not fall into either pattern. Pace the Humean account, he found that young children are readily inclined to believe extraordinary claims, such as that there are invisible organisms on your hands that can make you ill and that you need to wash off, and that there is a man who visits you each 24th December to bring presents and candy if you are nice (see e.g., Harris & Koenig, 2006, Child Development, 77, 505 - 524). But children are not blindly credulous either, as Reid supposed. In a series of experiments, Harris could show that even children of 24 months pay attention to the reliability of the testifier. When they see two people, one of which systematically misnames known objects (e.g., saying "that's a bear", while presenting a bottle), toddlers are less likely to trust later utterances by these unreliable speakers (when they name unfamiliar objects), and more likely to trust people who systematically gave objects their correct names (see e.g., Paul L. Harris and Kathleen H. Corriveau Phil. Trans. R. Soc. B 2011 366, 1179-1187.) Experiments by Mills and Keil show that 6-year-olds already take into account a testifier's self-interest: they are more likely to believe someone who says he lost a race than someone who says he won it (Candice M. Mills and Frank C. Keil Psychological Science 2005 16: 385).

See also:

"Atheism & autism spectrum"

Cognitive Style Tends To Predict Religious Conviction
Anyone have any questions for David Chalmers?
by Solvent — 2012-03-10T21:57:53.248Z

I'm doing an undergraduate course on the Free Will Theorem, with three lecturers: a mathematician, a physicist, and David Chalmers as the philosopher. The course is a bit pointless, but the company is brilliant. Chalmers is a pretty smart guy. He studied computer science and math as an undergraduate, before "discovering that he could get paid for doing the kind of thinking he was doing for free already". He's friendly; I've been chatting with him after the classes.

So if anyone has any questions for him, if they seem interesting enough I could approach him with them.

Emails to him also work, of course, but discussion in person lets more understanding happen faster. For example, in a short discussion with him I understood his position on consciousness way better than I would have just from reading his papers on the topic.
Free Applied Instrumental Rationality Webinar
by ksvanhorn — 2012-03-10T20:35:08.786Z

Dan Nuffer and I are putting together a free webinar that will go through the ideas in Smart Choices: A Practical Guide to Making Better Life Decisions, combined with whatever else seems useful. The authors of this book include one of the pioneers of decision analysis.

Although they don't describe it as such, Smart Choices is really a manual for basic applied instrumental rationality. It's a systematic way of going about your decisions, applicable to either decision problems (you have a situation dumped in your lap that requires a response) or decision opportunities (proactively seeking out ways to further your goals.)

The webinar will be one-hour sessions once a week for however long it takes to go through the material. We're going to do the webinar on Google+ Hangouts, and we'll have a discussion forum for the webinar on our web site.

If you're interested, send me an email [kevin at ksvanhorn com] with 1) your preferred day/time(s), and 2) the day/times that are out of the question for you.

Google+ Hangouts has a limit of 10 people. Five of those slots are already filled, leaving 5 seats open, so don't wait too long to email me if this is something you're interested in.