trentmkelly/LessWrong-43k huggingface dataset:
Your best arguments for risks from AI
by XiXiDu — 2011-01-17T13:20:12.658Z

An easy to read list of evidence and simple arguments in support of risks from artificial intelligence could help to raise awareness. Such a list could be the first step to draw attention, to spark interest and make people read some more advanced papers or the sequences. To my knowledge so far nobody has put the evidence, various arguments and indications together in one place.

My intention is to enable people interested to mitigate risks from AI to be able to offer a brochure that allows them to raise awareness without having to spend a lot of time explaining the details or tell people to read through hundreds of posts of marginal importance. There has to be some promotional literature that provides a summary of the big picture and some strong arguments for action. Such a brochure has to be simple and concise enough to arouse interest in the reader even if they just skim over the text.

Post a comment with the the best argument(s) for risks from AI

Some rules:

The argument or summary has to be simple and concise.

Allow for some inferential distance. Make your arguments self-evident.

If possible cite an example, provide references or link to further reading.

Disclosure: Note if your argument is controversial and account for it.

For starters I wrote a quick draft below. But there sure do exist a lot of other arguments and indications for why risks from artificial intelligence should be taken serious. What convinced you?

Claim: Creation of general intelligence is possible.

Status: Uncontroversial

Claim: Intelligence can be destructive.

Status: Uncontroversial

Claim: Algorithmic intelligence can be creative and inventive.

Status: Uncontroversial1

Claim: Improvements of algorithms can in many cases lead to dramatic performance gains.

Status: Uncontroversial2

Claim: Human-level intelligence is not the maximum.

Status: Very likely3

Claim: Any sufficiently advanced AI will do everything to continue to keep pursuing terminal goals indefinitely.

Status: Controversial but a possibility to be taken seriously.4 We don't yet have a good understanding of intelligence but given all that we know there are no good reasons to rule out this possibility. Overconfidence can have fatal consequences in this case.5

Claim: Morality is fragile and not imperative (i.e. is not a natural law)

Status: Uncontroversial.6 Even humans who have been honed by social evolution to consider the well-being of o...
Meta: Basic and advanced section
by Vaniver — 2011-01-17T11:11:44.028Z

lukeprog recently wrote a top-level post exhorting LW to emphasize the basics of rationality. That strikes me as a worthwhile endeavor- if we do actually want to raise the sanity waterline, we need a lot of droplets.

The suggestion I made was to redesign the site to create more specialized areas: a basic one where people aren't expected to have read the sequences, and an advanced one where people are. That's not the division that has to be made, but it seems a natural place to make one. For ease of reading, a page that displays both sections seems like it would be a good plan (but I have no idea if that is easy to make), and it might also be desirable to have it so one article can appear in both sections (maybe there would flags or tags for basic and advanced?). Another option for a division is between video and text posts- I personally have little interest in video posts but I'm sure there are people interested in both or just video posts.

The basic section would focus on fundamental concepts and standalone / modular posts. By modular, I mean replaceable- two posts that cover the same concept from different angles, ideally in enough detail that you could read either of them and be able to go on to the next post, if there is one. There are several (mutually inclusive) ways to go about this- video explanations of old posts, entirely new material, and new restatements (or perspectives) on old material. lukeprog covers videos, so I won't; new material seems pretty self-explanatory.

The main possible focus of such a section that needs discussion is new perspectives on old concepts. Right now, one of the failures of LW according to the outside view is the sense of cultishness, and I would not be surprised if that comes from the fact that anyone who maintains their desire to read the sequences throughout the entire sequences must be in some sense on the same wavelength as EY. If we had five different explanations of Politics is the Mind-Killler rather than just EY's, it seems plausible we could reach more people. If there were more approaches to metaethics than EY's, that might convince a few critics to stick around for other things they do like, and so on.

While plausible, it's not certain. Right now, the sequences are mostly pretty good. If we invite other people to present their core concepts, the average quality will probably decline, even if the maximum quality f...
The annoyingness of New Atheists: declaring God Dead makes you a Complete Monster?
by Raw_Power — 2011-01-17T01:13:44.233Z

I have noticed during my dialectic adventures on the Grid that religious people, no matter how "reasonable" (i.e. moderate, unaggressive, unassuming, gentle, etc.), would get very annoyed by an assertive, dry Atheist perspective, which they tend to nickname Hollywood Atheist (interestingly, religious people tend to use this term to atheists that openly make fun of religion and are very assertive and even preachy about their disbelief, while atheists tend to use it to mean people who are atheists for shallow, weak reasons and who do a poor job of defending their stance in an argument). There is also the tendency to compare the certainty of an Atheist with that of a Fundamentalists, when they are fundamentally different in nature (pun unintended), something they do not seem to be able or willing to grasp. Not that atheism hasn't had its fair share of fundamentalists, but that's supposedly the difference between an atheist who is so out of rationalism and one that is so because they hate the Church or because Stalin (glorified be his name) told them to.

On of the things that irritate them the most is the phrase "God is Dead". A phrase that is obviously meaningless in a literal sense (although, of course, God was never a living being in the first place, by the current definition). Figuratively, it means something akin to "Our Father is dead": we are now orphans, adults, we don't need a God to tell us what to do, or what to want, or how to see the world: we decide for ourselves, we see for ourselves, we are now free... but it does feel a bit lonely, and, for those who relied on their God or Parent Figure as a crutch, it can be hard to adapt to a world without a reference, without an authority figure. A world where you are the reference, you are responsible for your own moral choices.

There are other things, specific arguments, methods of approach, that anger them and are counterproductive to the submitting of the message. Of course, the atheist message is a Brown Note of sorts to the religious mind, since it challenges their entire worldview (though in the end it all adds up to normality... except much more seamlessly). However, it would be nice to develop an approach towards theists that avoids the frontal part of their mental shields and gets into the seams, using the minimal force in the points of maximum efficiency, bypassing their knee-jerk defences...

So, here...
Meta: Rationality News Area
by XiXiDu — 2011-01-16T17:26:44.619Z

A lot of links and media content has been posted lately. Would it make sense to create another area where people can submit relevant links, images or videos? If the discussion area was meant to keep the community blog clean and focused then one could devote the discussion area to questions, proofreading and mediocre posts while an additional media section could be destined for third party content:

Community Blog (main area)

A set of agreed upon topics.

Temporary announcements (meetups etc.)

Discussion Area

Questions

Proofreading

Mediocre content (not yet ready or not suitable for the main area)

Meta discussions about the community

Miscellaneous topics (e.g. AI, recommendations and questions about nutrition etc.)

Rationality News (recommended third party content)

News and media aggregation

Ranking and discovery

Relevant scientific research/studies

Important technological progress

There is a lot of good and bad stuff out there to learn from. Yet this content is often discovered by people who don't want, or are not able to write a post about it or start a discussion but would be willing to submit a link to the content. This would allow LW to offer a selection of relevant high quality third party news and media content without having the discussion area (RSS feed) cluttered with it.

LW could also employ a few moderators that can move posts to the appropriate area. For example, if a post in the main area is being downvoted or near zero after a week it will be moved to the discussion area. Or if someone posts a link to third-party content to the discussion area it is moved to the news (media) area. Users, especially newbies and outsiders would also benefit from it by not being exposed to inappropriate content but rather to the highest quality of content to be expected when browsing a certain area.

So what do you think, would a rationality news area be a worthwhile feature?
Slate article on Efficient Charity (link)
by bentarm — 2011-01-16T16:00:13.617Z

From the article:

Billions of dollars are given and spent on aid and development by individuals and companies each year. Despite this generosity, we simply do not allocate enough resources to solve all of the world's biggest problems. In a world fraught with competing claims on human solidarity, we have a moral obligation to direct additional resources to where they can achieve the most good. And that is as true of our own small-scale charitable donations as it is of governments' or philanthropists' aid budgets.

...

Guided by their consideration of each option's costs and benefits, and setting aside matters like media attention, the experts identified the best investments: those for which relatively tiny amounts of money could generate significant returns in terms of health, prosperity, and community advantages. These included: increased immunization coverage, initiatives to reduce school dropout rates, community-based nutrition promotion, and micronutrient supplementation.

Their conclusion? Micronutrients for people in poor countries. No, I don't think SIAI was considered as an option.
The Best Textbooks on Every Subject
by lukeprog — 2011-01-16T08:30:57.953Z

For years, my self-education was stupid and wasteful. I learned by consuming blog posts, Wikipedia articles, classic texts, podcast episodes, popular books, video lectures, peer-reviewed papers, Teaching Company courses, and Cliff's Notes. How inefficient!

I've since discovered that textbooks are usually the quickest and best way to learn new material. That's what they are designed to be, after all. Less Wrong has often recommended the "read textbooks!" method. Make progress by accumulation, not random walks.

But textbooks vary widely in quality. I was forced to read some awful textbooks in college. The ones on American history and sociology were memorably bad, in my case. Other textbooks are exciting, accurate, fair, well-paced, and immediately useful.

What if we could compile a list of the best textbooks on every subject? That would be extremely useful.

Let's do it.

There have been other pages of recommended reading on Less Wrong before (and elsewhere), but this post is unique. Here are the rules:

Post the title of your favorite textbook on a given subject.

You must have read at least two other textbooks on that same subject.

You must briefly name the other books you've read on the subject and explain why you think your chosen textbook is superior to them.

Rules #2 and #3 are to protect against recommending a bad book that only seems impressive because it's the only book you've read on the subject. Once, a popular author on Less Wrong recommended Bertrand Russell's A History of Western Philosophy to me, but when I noted that it was more polemical and inaccurate than the other major histories of philosophy, he admitted he hadn't really done much other reading in the field, and only liked the book because it was exciting.

I'll start the list with three of my own recommendations...

Subject: History of Western Philosophy

Recommendation: The Great Conversation, 6th edition, by Norman Melchert

Reason: The most popular history of western philosophy is Bertrand Russell's A History of Western Philosophy, which is exciting but also polemical and inaccurate. More accurate but dry and dull is Frederick Copelston's 11-volume A History of Philosophy. Anthony Kenny's recent 4-volume history, collected into one book as A New History of Western Philosophy, is both exciting and accurate, but perhaps too long (1000 pages) and technical for a first read on the histor...
Trying to hide bad signaling? To the Dark Side, lead you it will. 
by  — 2011-01-16T07:12:54.643Z

Edit: This is old material. It may be out of date.

Or is that just a point of view?

I'm going to assume familiary with the common use of the following two terms on this site:

Dark Arts

Signaling

Otherwise don't worry, I've hanged out here for ages and I still need to update my cache of terms quite often. If you have questions about either after reading the wiki please feel free to ask since there are people much more knowledgeable than me that will probably answer them. I don't know if other users agree, but the Discussion section seems like the best place to ask questions that might have been already covered elsewhere for people who have trouble despite extensive study, in a way this OP is basically an example of this.

I'm also making the following assumptions:

People are more often rational than otherwise when the rational answer happens to say good things about them.

I hope people here agree that learning to be more rational will necessarily at least in some areas change your beliefs (unlikely any one person is right about everything).

If one hasn't changed any beliefs its likely that they haven't employed rationality where it would do them the most good.

People are better at defending or promoting a position when they think its true.

From the above points it seems to follow that people who become more rational than average will send also more bad signals than they would if they hadn't become more rational.

Most people would try to avoid 5., it represents a disincentive for becoming more rational.

The main question of this thread:

How can one work around 5. without employing Dark Arts to sanitize the feelings accompanying a conclusion? Is it even possible? Can or should we talk about this and try to find and catalogue ways to do this since many of us are not skilled at social interactions (higher than average self identified non-neurotypicals visit LW)?

Notes:

- I also wish to emphasise that not only do some conclusions send bad signals, wanting to open *some* topics to rational inquiry in itself often sends bad signals even if you do eventually end up with a conclusion that sends good signals.

- I feel that, even if it isn't possible to hide bad signalling, the better map of reality one enjoys will off set these costs in other ways. Despite this, considering we are social animals I think many people would like to avoid this particular cost ...
Link: Edge: THE WORLD QUESTION CENTER 2011: “What Scientific Concept Would Improve Everybody’s Cognitive Toolkit? (edge.org)
by InquilineKea — 2011-01-15T23:05:17.734Z

http://www.edge.org/q2011/q11_1.html

It just got out. I've read this series for 5 years by now, and I've noticed many interesting quotes about rationality on that site. Feel free to post the ones you liked the most in this thread. :)
Which are the useful areas of AI study?
by PeterisP — 2011-01-15T23:03:25.044Z

I'm stuck wondering on a peculiar question lately - which are the useful areas of AI study? What got me thinking is the opinion occasionally stated (or implied) by Eliezer here that performing general AI research might likely have negative utility, due to indirectly facilitating a chance of unfriendly AI being developed. I've been chewing on the implications of this for quite a while, as acceptance of these arguments would require quite a change in my behavior.

Right now I'm about to start my CompSci PhD studies soon, and had initially planned to focus on unsupervised domain-specific knowledge extraction from the internet, as my current research background is mostly with narrow AI issues in computational linguistics, such as machine-learning, formation of concepts and semantics extraction. However, in the last year my expectations of singularity and existential risks of unfriendly AI have lead me to believe that focusing my efforts on Friendly AI concepts would be a more valuable choice; as a few years of studies in the area would increase the chance of me making some positive contribution later on.

What is your opinion?

Do studies of general AI topics and research in the area carry a positive or negative utility ? What are the research topics that would be of use to Friendly AI, but still are narrow and shallow enough to make some measurable progress by a single individual/tiny team in the course of a few years of PhD thesis preparation? Are there specific research areas that should be better avoided until more progress has been made on Friendliness research ?
Why is consistency often considered to be an intellectual virtue?
by InquilineKea — 2011-01-15T23:03:14.552Z

Now, of course, it's important for a scientific theory to be consistent across all cases in its domain. Otherwise, the theory is useless. Of course, we can specify the preconditions for consistency, and we may be able to argue for a expansion (or reduction) in its domain as time goes on.

But what about behavioral consistency? Certainly, behavioral consistency makes it easier for people to predict what you'll do in the future. So people who are behaviorally consistent are easier to trust, so to speak.

But what if we want to be right? If we want to be consistent, we have to stick with a certain behavior and *assume* that the "utility" of holding to that behavior will be stable across time. But sometimes, we will have special cases where the "utility" from deviation will be greater than the "utility" from non-deviation (in many of these special cases, the optimal situation may be one where you hide that special case from the eyes of everyone else, maybe in the interests of "fairness" - since many people get outraged by violations in "fairness"). Of course, we can specify what these special cases are beforehand (for example, you may require a security clearance for access to certain types of information, or a license to create certain types of drugs). But we cannot reliably predict each and every one of these special cases (there may be cases, for example, where you would increase "utility" if you gave out information X to someone who didn't have clearance, for example)

You could call those "double standards". People will often accuse you of hypocrisy if you're inconsistent, but you could always guard yourself against those accusations by introducing double standards. Whether those double standards are defensible or not - that, of course, depends on other factors. There are many behaviors, for example, that one may have to prohibit the entire population from pursuing. But some of these behaviors can be responsibly pursued by a small subset of the population (the only problem is that if you use that phrase, potentially irresponsible people will pursue these behaviors, as people are prone to overestimate their abilities to self-regulate themselves).

Now, in terms of beliefs, some of these double standards come with updated information. You might say that "action X is 'bad'" across all possible cases, but then update your belief and say that "action X is 'bad'" in most...
People Neglect Who They Really Are When Predicting Their Own Future Happiness [link]
by Dreaded_Anomaly — 2011-01-15T21:32:36.050Z

People Neglect Who They Really Are When Predicting Their Own Future Happiness (article @ ScienceDaily)

The scientists who conducted this interesting study...

found that our natural sunny or negative dispositions might be a more powerful predictor of future happiness than any specific event. They also discovered that most of us ignore our own personalities when we think about what lies ahead -- and thus miscalculate our future feelings.
Image: Another uninformed perspective on risks from AI (humor)
by XiXiDu — 2011-01-15T19:01:32.749Z

Here is another example of an outsider perspective on risks from AI. I think such examples can serve as a way to fathom the inferential distance between the SIAI and its target audience as to consequently fine tune their material and general approach.

via sentientdevelopments.com

This shows again that people are generally aware of potential risks but either do not take them seriously or don't see why risks from AI are the rule rather than an exception. So rather than making people aware that there are risks you have to tell them what are the risks.
"Bad-for-the-world-ipedia?"
by Raemon — 2011-01-15T16:47:33.255Z

Lately I've been thinking about all of the various services and products I consume and how pretty much all of them are bad for the world in one way or another, large or small. Some of the problems associated with them I am less concerned about. Some of them could be construed as good things (i.e. sweat shop labor DOES provide jobs, whatever impact it might or might not have on the overall quality of life).

In general I'd like to live my life having as minimal a negative impact on the world as possible. But "negative impact" is a hugely broad topic and there are a million variables to consider and I just don't have time.

The best solution, I think, would be to have a wikipedia-like website where individual people with knowledge of specific problems can start tagging specific products with the types of negative consequences associated with them, and (somehow) sort those consequences into categories that individuals can decide how much to worry about. Over time it could eventually become a fairly efficient way to track the utility value of things.

I'm sort of hoping something like this already exists, even if in an infant form, and that someone here knows about it. But I doubt it, so the I guess this falls mostly under the post category of "hey someone other than me should devote a bunch of time and energy to this project that I myself am not qualified to do." But maybe a few people here at least have a better idea than I do of the scope of the requirements for it, so the idea can be refined a bit.
Note on Terminology: "Rationality", not "Rationalism"
by Vladimir_Nesov — 2011-01-14T21:21:55.020Z

I feel that the term "rationalism", as opposed to "rationality", or "study of rationality", has undesirable connotations. My concerns are presented well by Eric Drexler in the article For Darwin’s sake, reject "Darwin-ism" (and other pernicious terms):

To call something an “ism” suggests that it is a matter ideology or faith, like Trotskyism or creationism. In the evolution wars, the term “evolutionism” is used to insinuate that the modern understanding of the principles, mechanisms, and pervasive consequences of evolution is no more than the dogma of a sect within science. It creates a false equivalence between a mountain of knowledge and the emptiness called “creationism”.

So, my suggestion is to use "rationality" consistently and to avoid using "rationalism". Via similarity to "scientist" and "physicist", "rationalist" doesn't seem to have the same problem. Discuss.

(Typical usage on Less Wrong is this way already, 3720 Google results for "rationality" and 1210 for "rationalist", against 251 for "rationalism". I've made this post as a reference for when someone uses "rationalism".)