trentmkelly/LessWrong-43k huggingface dataset:
Raising safety-consciousness among AGI researchers
by lukeprog — 2012-06-02T21:39:04.860Z

Series: How to Purchase AI Risk Reduction

Another method for purchasing AI risk reduction is to raise the safety-consciousness of researchers doing work related to AGI.

The Singularity Institute is conducting a study of scientists who decided to either (1) stop researching some topic after realizing it might be dangerous, or who (2) forked their career into advocacy, activism, ethics, etc. because they became concerned about the potential negative consequences of their work. From this historical inquiry we hope to learn some things about what causes scientists to become so concerned about the consequences of their work that they take action. Some of the examples we've found so far: Michael Michaud (resigned from SETI in part due to worries about the safety of trying to contact ET), Joseph Rotblat (resigned from the Manhattan Project before the end of the war due to concerns about the destructive impact of nuclear weapons), and Paul Berg (became part of a self-imposed moratorium on recombinant DNA back when it was still unknown how dangerous this new technology could be).

What else can be done?

Academic outreach, in the form of conversations with AGI researchers and "basics" papers like Intelligence Explosion: Evidence and Import or Complex Value Systems are Required to Realize Valuable Futures.

A scholarly AI risk wiki.

Short primers on crucial topics.

Whatever is suggested by our analysis of past researchers who took action in response to their concerns about the ethics of their research, and by other analyses of human behavior.

Naturally, these efforts should be directed toward researchers who are both highly competent and whose work is very relevant to development toward AGI: researchers like Josh Tenenbaum, Shane Legg, and Henry Markram.
Reaching young math/compsci talent
by lukeprog — 2012-06-02T21:07:00.394Z

Series: How to Purchase AI Risk Reduction

Here is yet another way to purchase AI risk reduction...

Much of the work needed for Friendly AI and improved algorithmic decision theories requires researchers to invent new math. That's why the Singularity Institute's recruiting efforts have been aimed a talent in math and computer science. Specifically, we're looking for young talent in math and compsci, because young talent is (1) more open to considering radical ideas like AI risk, (2) not yet entrenched in careers and status games, and (3) better at inventing new math (due to cognitive decline with age).

So how can the Singularity Institute reach out to young math/compsci talent? Perhaps surprisingly, Harry Potter and the Methods of Rationality is one of the best tools we have for this. It is read by a surprisingly large proportion of people in math and CS departments. Here are some other projects we have in the works:

Run SPARC, a summer program on rationality for high school students with exceptional math ability. Cost: roughly $30,000. (There won't be classes on x-risk at SPARC, but it will attract young talent toward efficient altruism in general.)

Print copies of the first few chapters of HPMoR cheaply in Taiwan, ship them here, distribute them to leading math and compsci departments. Cost estimate in progress.

Send copies of Global Catastrophic Risks to lists of bright young students. Cost estimate in progress.

Here are some things we could be doing if we had sufficient funding:

Sponsor and be present at events where young math/compsci talent gathers, e.g. TopCoder High School and the International Math Olympiad. Cost estimate in progress.

Cultivate a network of x-risk reducers with high mathematical ability, build a database of conversations for them to have with strategically important young math/compsci talent, schedule those conversations and develop a pipeline so that interested prospects have a "next person" to talk to. Cost estimate in progress.

Write Open Problems in Friendly AI, send it to interested parties so that even those who don't think AI risk is important will at least see "Ooh, look at these sexy, interesting problems I could work on!"
Paper: Iterated Prisoner’s Dilemma contains strategies that dominate any evolutionary opponent
by mapnoterritory — 2012-06-02T20:50:07.098Z

Bill "Numerical Recipes" Press and Freeman "Dyson sphere" Dyson have a new paper on iterated prisoner dilemas (IPD). Interestingly they found new surprising results:

It is generally assumed that there exists no simple ultimatum strategy whereby one player can enforce a unilateral claim to an unfair share of rewards. Here, we show that such strategies unexpectedly do exist. In particular, a player X who is witting of these strategies can (i) deterministically set her opponent Y’s score, independently of his strategy or response, or (ii) enforce an extortionate linear relation between her and his scores.

They discuss a special class of strategies - zero determinant (ZD) strategies of which tit-for-tat (TFT) is a special case:

The extortionate ZD strategies have the peculiar property of sharply distinguishing between “sentient” players, who have a theory of mind about their opponents, and “evolutionary” players, who may be arbitrarily good at exploring a ﬁtness landscape (either locally or globally), but who have no theory of mind.

The evolutionary player adjusts his strategy to maximize score, but doesn't take his opponent explicitly into account in another way (hence has "no theory of mind" of the opponent). Possible outcomes are:

A)

If X alone is witting of ZD strategies, then IPD reduces to one of two cases, depending on whether Y has a theory of mind. If Y has a theory of mind, then IPD is simply an ultimatum game (15, 16), where X proposes an unfair division and Y can either accept or reject the proposal. If he does not (or if, equivalently, X has ﬁxed her strategy and then gone to lunch), then the game is dilemma-free for Y. He can maximize his own score only by giving X even more; there is no beneﬁt to him in defecting.

B)

If X and Y are both witting of ZD, then they may choose to negotiate to each set the other’s score to the maximum cooperative value. Unlike naive PD, there is no advantage in defection, because neither can affect his or her own score and each can punish any irrational defection by the other. Nor is this equivalent to the classical TFT strategy (7), which produces indeterminate scores if played by both players.

This latter case sounds like a formalization of Hosfstadter's superrational agents. The cooperation enforcement via cross-setting the scores is very interesting.

Is this connection true ...
Confronting the Mindkiller - a series of posts exploring Political Landscape (Part 1)
by Tuxedage — 2012-06-02T20:17:15.012Z

Confronting the Mindkiller

Hi LessWrong.

I will be attempting a series of 11 posts on how to defeat the mindkiller -- Politics.

This is a bit of an exaggeration -- "defeating" it is a seriously difficult task, and it's something I cannot possibly hope to do alone. However, I hope to be able to raise the Sanity Waterline, as well as actually contribute something to this Community that I've begun to regard myself as part of.
I also beg patience from you, the reader. Lesswrong is far more critical than most other Communities, and frankly, even writing this post has begun to give me second thoughts, in fear of being downvoted into oblivion.
However, one who attempts nothing in the fear of consequences will live a life of regret - only by attempting to do the "impossible", and hacking away at the edges, can one succeed. This post was written with something to protect in mind. I hope you be patient with me, at least until the last posts have been written. This might take a week or two to complete.

I prefer to post one post at a time, Eliezer Yudkowsky style, in order to better adjust my writing style to suit your tastes, to receive continuous feedback, and to break up this task into smaller chunks to better avoid Akrasia, but if you'd rather I post this in one long essay, I can begrudgingly do that too.
Since my idea of these series of posts in the Open Thread seems to have been relatively well-received, I'll proceed to courageously embarrass myself in front of everyone.
This is my first serious post on this site, and I really welcome constructive criticism and any attempts to help me improve my writing skills. Likewise, if you don't think I should continue writing this series of posts, let me know, although please be gentle. I assure you that I will not fall trapping to partisan arguments or argue one side over another in an irrational fashion.
The purpose of these posts is to improve the way we think about politics, and help you find the "right side" in politics, rather than directly tell you which side is right or wrong. This post, I hope, does not break LessWrong's rules regarding no politics.
I also won't mess up too badly. I promise.

Confronting the Mindkiller - Why the Mindkiller?

Why am I trying to actively confront the mind-killer? Because Rationalists should win. Because as Rationalists, we ought to be able to go out into the real world and use o...
Focus on rationality
by Oscar_Cunningham — 2012-06-02T19:25:37.362Z

(This is my view in the recent debate about posts giving a "rational" discussion of some random topic. It was originally at comment level but I've extended it and posted it in discussion because I want to know if and where people disagree with me, and for what reasons.)

I come to Less Wrong to learn about how to think and how to act effectively. I care about general algorithms that are useful for many problems, like "Hold off on proposing solutions" or "Habits are ingrained faster when you pay concious attention to your thoughts when you perform the action". These posts have very high value to me because they improve my effectiveness across a wide range of areas.

Another such technique is "Dissolving the question". Yvain's "Diseased thinking: dissolving questions about disease" is valuable as an exemplary performance of this technique. It adds to Eliezer's description of question-dissolving by giving a demonstration of its use on a real question. It's main value comes from this, anything I learnt about disease whilst reading it is just a bonus.

To quote badger in the recent thread "Rational Toothpaste: A Case Study"

I claim a post on "rational toothpaste buying" could be on-topic and useful, if correctly written to illustrate determining goals, assessing tradeoffs, and implementing the final conclusions. A post detailing the pros and cons of various toothpaste brands is for a dentistry or personal hygiene forum; a post about algorithms for how to determine the best brands or whether to do so at all is for a rationality forum.

But we don't need more than one or two such examples! Yvain's post about question-dissolving was the only such post I ever need to read.

Posts about toothpaste, house-buying, room-decoration, fashion, shaving or computer hardware only tell me about that particular thing. As good as many of them are they'll never be as useful as a post that teaches me a general method of thought applicable on many problems. And if I want to know about some particular topic I'll just look it up on Google, or go to a library.

It's not possible for LessWrong to give a rational treatment of every subject. There are just too many of them. Even if we did I wouldn't be able to carry all that info around in my head. That's why I need to learn general algorithms for producing rational decisions.

Even though badger makes it clear in the quote I gave that t...
Rationality Quotes June 2012
by OpenThreadGuy — 2012-06-02T17:14:39.130Z

Here's the new thread for posting quotes, with the usual rules:

Please post all quotes separately, so that they can be voted up/down separately. (If they are strongly related, reply to your own comments. If strongly ordered, then go ahead and post them together.)

Do not quote yourself

Do not quote comments/posts on LW/OB

No more than 5 quotes per person per monthly thread, please.
Suggestion: Less Wrong Writing Circle?
by  — 2012-06-02T16:12:42.358Z

This community has a recurring interest in "rationalist fiction," and several members who are writers. I wonder if it would be useful to create a space where Less Wrong members could provide each other constructive criticism and encouragement on in-progress original writing projects?

Disclosure: I'm working on a sci-fi novel right now, and my regular circle of "beta readers" are fantasy fans and aren't providing much feedback on the new project. I am much, much more productive as a writer when I get steady feedback, so I have a personal interest in looking for something like this. Less Wrong came to mind as a community of intelligent, creative, forward-looking types who are likely to enjoy sci-fi.
Meetup : Less Wrong Sydney
by Oklord — 2012-06-02T12:53:57.751Z

Discussion article for the meetup : Less Wrong Sydney

WHEN:
11 June 2012 06:00:00PM (+1000)

WHERE:
77 Liverpool Street, Sydney, Australia, Level 2

Hey all! As usual, we'll be conducting our meet up at the Norita Cafe and Board Games. And furthermore, as usual, we'll be soliciting topics from the floor.

FB group availible at http://www.facebook.com/#!/groups/219526434802422/

Discussion article for the meetup : Less Wrong Sydney
[SEQ RERUN] Possibility and Could-ness
by MinibearRex — 2012-06-02T05:28:39.935Z

Today's post, Possibility and Could-ness was originally published on 14 June 2008. A summary (taken from the LW wiki):

Our sense of "could-ness", as in "I could have not rescued the child from the burning orphanage", comes from our own decision making algorithms labeling some end states as "reachable". If we wanted to achieve the world-state of the child being burned, there is a series of actions that would lead to that state.

Discuss the post here (rather than in the comments to the original post).
This post is part of the Rerunning the Sequences series, where we'll be going through Eliezer Yudkowsky's old posts in order so that people who are interested can (re-)read and discuss them. The previous post was Causality and Moral Responsibility, and you can use the sequence_reruns tag or rss feed to follow the rest of the series.
Sequence reruns are a community-driven effort. You can participate by re-reading the sequence post, discussing it here, posting the next day's sequence reruns post, or summarizing forthcoming articles on the wiki. Go here for more details, or to have meta discussions about the Rerunning the Sequences series.
Marketplace Transactions Open Thread
by John_Maxwell — 2012-06-02T04:31:32.387Z

Social scientists think humans operate under one of two different sets of norms, depending on the circumstances: "market norms" or "social norms". The basic idea is that when exchanging money for goods and services formally, it's considered okay to be much more calculating and self-interested than when exchanging favors with friends informally. You can read this blog post by Dan Ariely for more.

It's often considered rude to introduce market norms in an area where they don't traditionally apply. For example, by charging money for your presence at a barbecue.

This is a thread where it's okay to talk about trading money for goods and services with other Less Wrong users, which might otherwise be considered rude because you'd be inappropriately introducing market norms. Things you're encouraged to do include:

Post your resume

Advertise a product sold by you or your company

Advertise a service provided by you or your company

Advertise an open position working for you or your company

The argument for having a thread like this is as follows. Less Wrong users have a variety of goals they wish to accomplish. Some of these goals involve engaging in marketplace transactions. It's plausible that a thread facilitating marketplace transactions between LW users will buy just as much or more collective goal accomplishment per unit attention consumed than a traditional Less Wrong thread.

Anecdotally it seems that introducing market norms takes a certain amount of chutzpah. For example, apparently it takes a certain kind of person to actually be able to name a dollar figure in a sales conversation, and that's why you need a professional salesperson to come along with a sales engineer when selling a technical product. One LWer friend of mine struggled for a while before she was able to get herself to charge money for talk therapy she had been providing to friends for free.

To combat this, please feel inclined to vote up folks who post in this thread. They likely overcame some akrasia in the act of promoting their offer.

To discuss the concept of this thread, as opposed to advertising a transaction you wish to engage in, please reply to this comment.
Less Wrong: The podcast
by mapnoterritory — 2012-06-01T20:33:49.518Z

Would it be possible to have a monthly podcast on Less Wrong topics? A possible format could be roughly four panelist (maybe half core and half rotating members) discussing theoretical and practical aspects or rationality, AI/singularity, cognitive science etc.

Episodes can be also easily framed by assigning some reading from sequences or recent LW articles and then discussing them in podcast form. This format seems to work great for www.partiallyexaminedlife.com (quite entertaining and informative podcast albeit on the diseased discipline of philosophy).

To keep things interesting occasional episodes could be done in the form of discussions with guests (via skype), e.g. the usual suspects from the SAIA and fellow AI scientists, people like Robin Hanson, Aubrey de Grey, other rationalist/skeptical bloggers/podcasters but also AI skeptics and so on.

The level of the podcast should be still accesible to newcomers but the discussion could thread into bit deeper waters. I would love to hear discussions also on more technical topics (like CEV, Solomonoff induction, AIXI etc.). Just imagine how exciting could be discussions on the more controversial decision-theoretic paradoxes!

Further possibility is from time to time plan ahead and cover in more depth the topic from that week's Harry Potter and the Methods of Rationality podcast, which would potentially help to increase the audience (it would be like a post-grad HPMoR).

What do you think? Could this be done? With the depth and breadth of material we have here and all the interesting people to talk to I don't think there would be a shortage of topics.

Edit: Changed the (bi-)weekly timescale to monthly.
Far negatives of cryonics?
by Bart119 — 2012-06-01T18:43:52.963Z

In considering the pros and cons of cryonics, has anyone addressed the possibility of being revived in an unpleasant future, for instance as a "torture the infidel" exhibit in a theme park of a theocratic state? I had some thoughts on the issue but figured I would see what else has been written previously.
Meetup : Austin, TX
by Vaniver — 2012-06-01T17:40:50.687Z

Discussion article for the meetup : Austin, TX

WHEN:
02 June 2012 01:30:00PM (-0500)

WHERE:
2222B Guadalupe St Austin, Texas 78712

The Austin meetup continues, returning again to Caffe Medici. (We're meeting every week, which is far more frequently than I post reminders to come.) We sit on the second floor to the left, near (or often on) the stage.

This week, we're going to look at some of the rationality exercises from the May Minicamp, as well as our regular discussion.

Discussion article for the meetup : Austin, TX
The Truth Points to Itself, Part I
by Arbitrarity — 2012-06-01T15:53:54.529Z

(Cross-posted from another blog but written with LessWrong in mind. Don't worry, if this post isn't well-received then for LessWrong the series will end with this post.)

Summary: This post is the beginning of a systematic attempt to answer the question "what is the most important thing?". (Updateless) decision theory is used to provisionally define "importance" and Juergen Schmidhuber's theory of beauty is introduced as a possible answer to the question. The motivations for bringing in Schmidhuber's theory are discussed. This post is also intended to serve as an example of how to understand and solve hard problems in general, and emphasizes the heuristic "go meta".

This post is the first in a series about what might be the most important question we know to ask: What is the most important thing?

Don't try to answer the question yet. When faced with a tough question our first instinct should always be to go meta. What is it that causes me to ask the question "what is the most important thing"? What makes me think the question is itself important? Is that thing important? Does it point to itself as the most important thing? If not, then where does it point? Does the thing it points to, point to itself? If we follow this chain, where do we end up? How path-dependent is the answer? How much good faith do we have to assume on the part of the various things, to trust that they'll give their honest opinions? If we can't simply assume good faith, can we design a mechanism to promote honesty? What mechanisms are already in place, and are there cheap, local improvements we can make for those mechanisms?

And to ask all those questions we have to assume various commonsense notions that we might in fact need to pin down more precisely beforehand. Like, what is importance? Luckily we have some tools we can use to try to figure that part out.

Decision theory is one such tool. In Bayesian decision theory "importance" might be a fair name for what is measured by your decision policy, which you get by multiplying your beliefs by your value function. Informally, your decision policy tells you what options or actions to pay most attention to, or what possibilities are most important. But arguably it's your values themselves that should be considered "important", and your beliefs just tell you how the important stuff relates to w...
Robot ethics [link]
by fortyeridania — 2012-06-01T15:43:29.295Z

The Economist has a new article on ethical dilemmas faced by machine designers.

Evidently:

1. In the event of an immoral decision by a machine, neural networks make it too hard to know who is at fault--the programmer, the operator, the manufacturer, or the designer. Thus, neural networks might be a bad idea.

2. Robots' ethical systems ought to resonate with "most people."

3. Proper robot consciences are more likely to arise given greater collaboration among engineers, ethicists, policymakers, and lawyers. Key quotation:

Both ethicists and engineers stand to benefit from working together: ethicists may gain a greater understanding of their field by trying to teach ethics to machines, and engineers need to reassure society that they are not taking any ethical short-cuts.

The second clause of the above sentence is quite similar to something Yudkowsky wrote, perhaps more than once, about the value of approaching ethics from an AI standpoint. I do not recall where he wrote it, nor did my search turn up the appropriate post.