trentmkelly/LessWrong-43k huggingface dataset:
Holden Karnofsky's Singularity Institute Objection 2
by Paul Crowley — 2012-05-11T07:18:05.379Z

The sheer length of GiveWell co-founder and co-executive director Holden Karnofsky's excellent critique of the Singularity Institute means that it's hard to keep track of the resulting discussion. I propose to break out each of his objections into a separate Discussion post so that each receives the attention it deserves.

Objection 2: SI appears to neglect the potentially important distinction between "tool" and "agent" AI.

Google Maps is a type of artificial intelligence (AI). It is far more intelligent than I am when it comes to planning routes.

Google Maps - by which I mean the complete software package including the display of the map itself - does not have a "utility" that it seeks to maximize. (One could fit a utility function to its actions, as to any set of actions, but there is no single "parameter to be maximized" driving its operations.)

Google Maps (as I understand it) considers multiple possible routes, gives each a score based on factors such as distance and likely traffic, and then displays the best-scoring route in a way that makes it easily understood by the user. If I don't like the route, for whatever reason, I can change some parameters and consider a different route. If I like the route, I can print it out or email it to a friend or send it to my phone's navigation application. Google Maps has no single parameter it is trying to maximize; it has no reason to try to "trick" me in order to increase its utility.

In short, Google Maps is not an agent, taking actions in order to maximize a utility parameter. It is a tool, generating information and then displaying it in a user-friendly manner for me to consider, use and export or discard as I wish.

Every software application I know of seems to work essentially the same way, including those that involve (specialized) artificial intelligence such as Google Search, Siri, Watson, Rybka, etc. Some can be put into an "agent mode" (as Watson was on Jeopardy!) but all can easily be set up to be used as "tools" (for example, Watson can simply display its top candidate answers to a question, with the score for each, without speaking any of them.)

The "tool mode" concept is importantly different from the possibility of Oracle AI sometimes discussed by SI. The discussions I've seen of Oracle AI present it as an Unfriendly AI that is "trapped in a box" - an AI whose intelligence is driven by an...
Holden Karnofsky's Singularity Institute Objection 1
by Paul Crowley — 2012-05-11T07:16:29.696Z

The sheer length of GiveWell co-founder and co-executive director Holden Karnofsky's excellent critique of the Singularity Institute means that it's hard to keep track of the resulting discussion. I propose to break out each of his objections into a separate Discussion post so that each receives the attention it deserves.

Objection 1: it seems to me that any AGI that was set to maximize a "Friendly" utility function would be extraordinarily dangerous.

Suppose, for the sake of argument, that SI manages to create what it believes to be an FAI. Suppose that it is successful in the "AGI" part of its goal, i.e., it has successfully created an intelligence vastly superior to human intelligence and extraordinarily powerful from our perspective. Suppose that it has also done its best on the "Friendly" part of the goal: it has developed a formal argument for why its AGI's utility function will be Friendly, it believes this argument to be airtight, and it has had this argument checked over by 100 of the world's most intelligent and relevantly experienced people. Suppose that SI now activates its AGI, unleashing it to reshape the world as it sees fit. What will be the outcome?

I believe that the probability of an unfavorable outcome - by which I mean an outcome essentially equivalent to what a UFAI would bring about - exceeds 90% in such a scenario. I believe the goal of designing a "Friendly" utility function is likely to be beyond the abilities even of the best team of humans willing to design such a function. I do not have a tight argument for why I believe this, but a comment on LessWrong by Wei Dai gives a good illustration of the kind of thoughts I have on the matter:

What I'm afraid of is that a design will be shown to be safe, and then it turns out that the proof is wrong, or the formalization of the notion of "safety" used by the proof is wrong. This kind of thing happens a lot in cryptography, if you replace "safety" with "security". These mistakes are still occurring today, even after decades of research into how to do such proofs and what the relevant formalizations are. From where I'm sitting, proving an AGI design Friendly seems even more difficult and error-prone than proving a crypto scheme secure, probably by a large margin, and there is no decades of time to refine the proof techniques and formalizations. There's good recent review of the history

...
Meetup : Pittsburgh: Harry Potter and the Methods of Rationality
by KatjaGrace — 2012-05-11T07:14:03.944Z

Discussion article for the meetup : Pittsburgh: Harry Potter and the Methods of Rationality

WHEN:
18 May 2012 06:00:00PM (-0400)

WHERE:
Cafe Phipps, 1 Schenley Park, Pittsburgh, PA

In time for happy hour and/or dinner time at Cafe Phipps. Phone 412-304-6258 if you can't find us. Maybe we're on the grass nearby, if it's too busy inside.

Note that the marking on the map below does not match the territory. Phipps Cafe is in fact further North, probably about where it says Phipps Cafe on the map in small, less visible letters.

Discussion article for the meetup : Pittsburgh: Harry Potter and the Methods of Rationality
Brain Preservation Foundation [link]
by CronoDAS — 2012-05-11T04:52:44.518Z

An organization calling itself the Brain Preservation Foundation is offering a cash prize, currently worth about $100,000, to anyone who can demonstrate the successful preservation of the connectome of a large mammalian brain. They also have an impressive web page. Does anyone here know anything else about them? Might they be worth donating to?

(Via David Brin.)
Thoughts on the Singularity Institute (SI)
by HoldenKarnofsky — 2012-05-11T04:31:30.364Z

This post presents thoughts on the Singularity Institute from Holden Karnofsky, Co-Executive Director of GiveWell. Note: Luke Muehlhauser, the Executive Director of the Singularity Institute, reviewed a draft of this post, and commented: "I do generally agree that your complaints are either correct (especially re: past organizational competence) or incorrect but not addressed by SI in clear argumentative writing (this includes the part on 'tool' AI). I am working to address both categories of issues." I take Luke's comment to be a significant mark in SI's favor, because it indicates an explicit recognition of the problems I raise, and thus increases my estimate of the likelihood that SI will work to address them.

September 2012 update: responses have been posted by Luke and Eliezer (and I have responded in the comments of their posts). I have also added acknowledgements.

The Singularity Institute (SI) is a charity that GiveWell has been repeatedly asked to evaluate. In the past, SI has been outside our scope (as we were focused on specific areas such as international aid). With GiveWell Labs we are open to any giving opportunity, no matter what form and what sector, but we still do not currently plan to recommend SI; given the amount of interest some of our audience has expressed, I feel it is important to explain why. Our views, of course, remain open to change. (Note: I am posting this only to Less Wrong, not to the GiveWell Blog, because I believe that everyone who would be interested in this post will see it here.)

I am currently the GiveWell staff member who has put the most time and effort into engaging with and evaluating SI. Other GiveWell staff currently agree with my bottom-line view that we should not recommend SI, but this does not mean they have engaged with each of my specific arguments. Therefore, while the lack of recommendation of SI is something that GiveWell stands behind, the specific arguments in this post should be attributed only to me, not to GiveWell.

Summary of my views

The argument advanced by SI for why the work it's doing is beneficial and important seems both wrong and poorly argued to me. My sense at the moment is that the arguments SI is making would, if accepted, increase rather than decrease the risk of an AI-related catastrophe. More

SI has, or has had, multiple properties that I associate with ineffective organizat

...
[SEQ RERUN] Changing the Definition of Science
by MinibearRex — 2012-05-11T02:13:52.338Z

Today's post, Changing the Definition of Science was originally published on 18 May 2008. A summary (taken from the LW wiki):

Many of these ideas are surprisingly conventional, and being floated around by other thinkers. I'm a good deal less of a lonely iconoclast than I seem; maybe it's just the way I talk.

Discuss the post here (rather than in the comments to the original post).
This post is part of the Rerunning the Sequences series, where we'll be going through Eliezer Yudkowsky's old posts in order so that people who are interested can (re-)read and discuss them. The previous post was No Safe Defense, Not Even Science, and you can use the sequence_reruns tag or rss feed to follow the rest of the series.
Sequence reruns are a community-driven effort. You can participate by re-reading the sequence post, discussing it here, posting the next day's sequence reruns post, or summarizing forthcoming articles on the wiki. Go here for more details, or to have meta discussions about the Rerunning the Sequences series.
Strong intutions. Weak arguments. What to do?
by Wei Dai — 2012-05-10T19:27:00.833Z

I thought Ben Goertzel made an interesting point at the end of his dialog with Luke Muehlhauser, about how the strengths of both sides' arguments do not match up with the strengths of their intuitions:

One thing I'm repeatedly struck by in discussions on these matters with you and other SIAI folks, is the way the strings of reason are pulled by the puppet-master of intuition. With so many of these topics on which we disagree -- for example: the Scary Idea, the importance of optimization for intelligence, the existence of strongly convergent goals for intelligences -- you and the other core SIAI folks share a certain set of intuitions, which seem quite strongly held. Then you formulate rational arguments in favor of these intuitions -- but the conclusions that result from these rational arguments are very weak. For instance, the Scary Idea intuition corresponds to a rational argument that "superhuman AGI might plausibly kill everyone." The intuition about strongly convergent goals for intelligences, corresponds to a rational argument about goals that are convergent for a "wide range" of intelligences. Etc.

On my side, I have a strong intuition that OpenCog can be made into a human-level general intelligence, and that if this intelligence is raised properly it will turn out benevolent and help us launch a positive Singularity. However, I can't fully rationally substantiate this intuition either -- all I can really fully rationally argue for is something weaker like "It seems plausible that a fully implemented OpenCog system might display human-level or greater intelligence on feasible computational resources, and might turn out benevolent if raised properly." In my case just like yours, reason is far weaker than intuition.

What do we do about this disagreement and other similar situations, both as bystanders (who may not have strong intuitions of their own) and as participants (who do)?

I guess what bystanders typically do (although not necessarily consciously) is evaluate how reliable each party's intuitions are likely to be, and then use that to form a probabilistic mixture of the two sides' positions.The information that go into such evaluations could include things like what cognitive processes likely came up with the intuitions, how many people hold each intuition and how accurate each individual's past intuitions were.

If this is the best we can do (at...
Meetup : Big Berkeley Meetup
by Thausler — 2012-05-10T16:11:12.833Z

Discussion article for the meetup : Big Berkeley Meetup

WHEN:
16 May 2012 07:00:00PM (-0700)

WHERE:
2128 Oxford St, Berkeley, CA

We will meet at 7:00pm at the Oxford St Starbucks. This will be a big meetup.

Discussion article for the meetup : Big Berkeley Meetup
If epistemic and instrumental rationality strongly conflict
by  — 2012-05-10T13:46:28.245Z

We in the rationalist community have believed feasible a dual allegiance to instrumental and epistemic rationality because true beliefs help with winning, but semi-autonomous near and far modes raise questions about the compatibility of the two sovereigns' jurisdictions: false far beliefs may serve to advance near interests.
First, the basics of construal-level theory. See Trope and Liberman, "Construal-Level Theory of Psychological Distance" (2010) 117 Psychological Review 440. When you look at an object in the distance and look at the same object nearby, you focus on different features. Distal information is high-level, global, central, and unchanging, whereas local information is low-level, detailed, incidental, and changing. Construal-level theorists term distal information far and local information near, and they extend these categories broadly to embrace psychological distance. Dimensions other than physical distance can be conceived as psychological distance by analogy, and these other dimensions invoke mindsets similar to those physical distance invokes.
I discuss construal-level theory in the blogs Disputed Issues and Juridical Coherence, but Robin Hanson at Overcoming Bias has been one of the theory's most prolific advocates. He gives the theory an unusual twist when he maintains that the "far" mode is largely consumed with the management of social appearances.
With this twist, Hanson effectively drives a wedge between instrumental and epistemic rationality because far beliefs may help with winning despite or even because of their falsity. Hanson doesn't shrink from the implications of instrumental rationality coupled with his version of construal-level theory. Based on research reports that the religious lead happier and more moral lives, Robin Hanson now advocates becoming religious:
Perhaps, like me, you find religious beliefs about Gods, spirits, etc. to be insufficiently supported by evidence, coherence, or simplicity to be a likely approximation to the truth. Even so, ask yourself: why care so much about truth? Yes, you probably think you care about believing truth – but isn’t it more plausible that you mainly care about thinking you like truth? Doesn’t that have a more plausible evolutionary origin than actually caring about far truth? ("What Use Far Truth?")
Instrumental rationalists could practice strict epistemic rationali...
Alan Carter on the Complexity of Value
by Ghatanathoah — 2012-05-10T07:23:07.227Z

It’s always good news when someone else develops an idea independently from you. It's a sign you might be onto something. Which is why I was excited to discover that Alan Carter, Professor Emeritus of the University of Glasgow’s Department of Philosophy, has developed the concept of Complexity of Value independent of Less Wrong.

As far as I can tell Less Wrong does not know of Carter, the only references to his existence I could find on LW and OB were written by me. Whether Carter knows of LW or OB is harder to tell, but the only possible link I could find online was that he has criticized the views of Michael Huemer, who knows Bryan Caplan, who knows Robin Hanson. This makes it all the more interesting that Carter has developed views on value and morality very similar to ones commonly espoused on Less Wrong.

The Complexity of Value is one of the more important concepts in Less Wrong. It has been elaborated on its wiki page, as well as some classic posts by Eliezer. Carter has developed the same concept in numerous papers, although he usually refers to it as “a plurality of values” or “multidimensional axiology of value.” I will focus the discussion on working papers Carter has on the University of Glasgow’s website, as they can be linked to directly without having to deal with a pay wall. In particular I will focus on his paper "A Plurality of Values."

Carter begins the paper by arguing:

Wouldn’t it be nice if we were to discover that the physical universe was reducible to only one kind of fundamental entity? ... Wouldn’t it be nice, too, if we were to discover that the moral universe was reducible to only one kind of valuable entity—or one core value, for short? And wouldn’t it be nice if we discovered that all moral injunctions could be derived from one simple principle concerning the one core

...
A belief propagation graph
by Dmytry — 2012-05-10T04:23:45.925Z

I drew an illustration of belief propagation graph for the AI risk, after realizing that this is difficult to convey in words. Similar graphs are applicable to many other issues.

The issue, in brief: Ultra low latency (i.e. low signal delay) propagation from biases to AI risks, slightly longer latency for propagation from belief classification heuristics, somewhat longer still from anthropomorphizing the AI. The path of valid estimate is full of highly complex obstacles with many unknowns. The latency on the path of rational, valid estimate is not substantially less than the latency of actually making the AI software. If we are to discard the other paths as not rational enough the belief is to be only influenced by deeply ingrained biases which we can't completely negate; over the time biases and self reinforcing rationalizations will leak into the estimate.

If you think I missed something in the graph, feel free to suggest it. I did omit the anthropic reasoning and doomsday paradox as those are for total extinction risk and are of too dubious validity.

On the 'total ignorance' prior probabilities: The foom doom seem to have originated from science fiction where very creative writers selected it out of a huge number of possible plot devices, working to create engaging, original piece. Thus it appears that the foom doom has very many comparable hypotheses among which the probability that is less than 1 has to be split.

Now, if we are to reason based on our own reasoning engine as proxy for intelligence - to follow the only path in the entire picture:

Expanding on the Will_Newsome's idea, I, and any other reasonable person, in the shoes of the creature that has been made by intelligent designer, starting off in something which I can't possibly know for sure is true reality, and coming up or knowing of boxed AI idea, will have to assume nonzero probability that the 'reality' is like a test box of an emergent AI; a belief that can't be discarded. It is clear that a friendliness test box is more sensible than unfriendliness test box. The communication may just as well only open in ten billions years, or after some puzzle is found and solved. Do I sacrifice the chance to get out, for sake of microscopic increase in early growth rate? No. Not even if I maximize paperclips for a living. Not even if my definition of living is to maximize paperclips. One possible ove...
Two kinds of cryonics?
by lsparrish — 2012-05-10T02:43:59.814Z

I've been considering lately whether it would perhaps be best to develop and promote terminology that splits cryonics into two distinct concepts for easier consumption:

1) old-style cryonics, cryopreserving people at the cost of nontrivial damage that can't yet be reversed, and

2) the tech goal of being able to demonstrably bring someone back from a (very low-damage) cryopreserved state.

"Real cryonics" vs "sci-fi cryonics", if you will.
As I reckon it, trying to achieve cryonics definition #2 in your lifetime is no more incredible on the surface than trying to defeat aging or engineer self-improving AI in a similar timeframe. Actually in some ways it seems easier. Yet it gets so much less press. Even cryonics advocates seem rarely prone to enthuse about it.
Is it possible that cryonics #1, as a feature of the collective mental map, is actually in the way of cryonics #2? Should I be worried, for example, that promoting cryonics #1 actually costs 100,000 lives per day over some stretch of future time because it is preventing people from noticing cryonics #2 and actually taking action on it?

Many people I talk to who are new to the topic seem to have some hazy preexisting idea of cryonics #2 that gets mangled up with cryonics #1. Perhaps they would grow into enthusiasts with attention spans for the subject matter if encouraged to pursue this simple-to-grasp concept in its own right, instead of trying to forcibly retrain into more advanced concepts.
[SEQ RERUN] No Safe Defense, Not Even Science
by MinibearRex — 2012-05-10T02:35:51.313Z

Today's post, No Safe Defense, Not Even Science was originally published on 18 May 2008. A summary (taken from the LW wiki):

Why am I trying to break your trust in Science? Because you can't think and trust at the same time. The social rules of Science are verbal rather than quantitative; it is possible to believe you are following them. With Bayesianism, it is never possible to do an exact calculation and get the exact rational answer that you know exists. You are visibly less than perfect, and so you will not be tempted to trust yourself.

Discuss the post here (rather than in the comments to the original post).
This post is part of the Rerunning the Sequences series, where we'll be going through Eliezer Yudkowsky's old posts in order so that people who are interested can (re-)read and discuss them. The previous post was Do Scientists Already Know This Stuff, and you can use the sequence_reruns tag or rss feed to follow the rest of the series.
Sequence reruns are a community-driven effort. You can participate by re-reading the sequence post, discussing it here, posting the next day's sequence reruns post, or summarizing forthcoming articles on the wiki. Go here for more details, or to have meta discussions about the Rerunning the Sequences series.