trentmkelly/LessWrong-43k huggingface dataset:
Un-optimised vs anti-optimised
by Stuart_Armstrong — 2015-04-14T18:30:03.783Z

A putative new idea for AI control; index here.

This post contains no new insights; it just puts together some old insights in a format I hope is clearer.

Most satisficers are unoptimised (above the satisficing level): they have a limited drive to optimise and transform the universe. They may still end up optimising the universe anyway: they have no penalty for doing so (and sometimes it's a good idea for them). But if they can lazily achieve their goal, then they're ok with that too. So they simply have low optimisation pressure.

A safe "satisficer" design (or a reduced impact AI design) needs to be not only un-optimised, but specifically anti-optimised. It has to be setup so that "go out and optimise the universe" scores worse that "be lazy and achieve your goal". The problem is that these terms are undefined (as usual), that there are many minor actions that can optimise the universe (such as creating a subagent), and the approach has to be safe against all possible ways of optimising the universe - not just the "maximise u" for a specific and known u.

That's why the reduced impact/safe satisficer/anti-optimised designs are so hard: you have to add a very precise yet general (anti-)optimising pressure, rather than simply removing the current optimising pressure.
Could you tell me what's wrong with this?
by Algon — 2015-04-14T10:43:49.478Z

Edit: Some people have misunderstood my intentions here. I do not in any way expect this to be the NEXT GREAT IDEA. I just couldn't see anything wrong with this, which almost certainly meant there were gaps in my knowledge. I thought the fastest way to see where I went wrong would be to post my idea here and see what people say. I apologise for any confusion I caused. I'll try to be more clear next time.

(I really can't think of any major problems in this, so I'd be very grateful if you guys could tell me what I've done wrong).

So, a while back I was listening to a discussion about the difficulty of making an FAI. One of the ways that was suggested to circumvent this was to go down the route of programming an AGI to solve FAI. Someone else pointed out the problems with this. Amongst other things one would have no idea what the AI will do in pursuit of its primary goal. Furthermore, it would already be a monumental task to program an AI whose primary goal is to solve the FAI problem; doing this is still easier than solving FAI, I should think.

So, I started to think about this for a little while, and I thought 'how could you make this safer?' Well, first of, you don't want an AI who completely outclasses humanity in terms of intellect. If things went Wrong, you'd have little chance of stopping it. So, you want to limit the AI's intellect to genius level, so if something did go Wrong, then the AI would not be unstoppable. It may do quite a bit of damage, but a large group of intelligent people with a lot of resources on their hands could stop it.

Therefore, what must be done is that the AI cannot modify parts of its source code. You must try and stop an intelligence explosion from taking off. So, limited access to its source code, and a limit on how much computing power it can have on hand. This is problematic though, because the AI would not be able to solve FAI very quickly. After all, we have a few genius level people trying to solve FAI, and they're struggling with it, so why should a genius level computer do any better. Well, an AI would have fewer biases, and could accumulate much more expertise relevant to the task at hand. It would be about as capable as solving FAI as the most capable human could possibly be; perhaps even more so. Essentially, you'd get someone like Turing, Von Neumann, Newton and others all rolled into one working...
Translating bad advice
by Sophronius — 2015-04-14T09:20:06.253Z

While writing my Magnum Opus I came across this piece of writing advice by Neil Gaiman:

“When people tell you something’s wrong or doesn’t work for them, they are almost always right. When they tell you exactly what they think is wrong and how to fix it, they are almost always wrong.”

And it struck me how true it was, even in other areas of life. People are terrible at giving advice on how to improve yourself, or on how to improve anything really. To illustrate this, here is what you would expect advice from a good rationalist friend to look like:

1) “Hey, I’ve noticed you tend to do X.”

2) “It’s been bugging me for a while, though I’m not really sure why. It’s possible other people think X is bad as well, you should ask them about it.”

3) Paragon option: “Maybe you could do Y instead? I dunno, just think about it.”

4) Renegade option: “From now on I will slap you every time you do X, in order to help you stop being retarded about X.”

I wish I had more friends who gave advice like that, especially the renegade option. Instead, here is what I get in practice:

1) Thinking: Argh, he is doing X again. That annoys me, but I don’t want to be rude.

2) Thinking: Okay, he is doing Z now, which is kind of like X and a good enough excuse to vent my anger about X

3) *Complains about Z in an irritated manner, and immediately forgets that there’s even a difference between X and Z*

4) Thinking: Oh shit, that was rude. I better give some arbitrary advice on how to fix Z so I sound more productive.

As you can see, social rules and poor epistemology really get in the way of good advice, which is incredibly frustrating if you genuinely want to improve yourself! (Needless to say, ignoring badly phrased advice is incredibly stupid and you should never do this. See HPMOR for a fictional

...
Maybe we can perform the "Mary's Room" thought experiment
by DavidPlumpton — 2015-04-14T09:19:37.561Z

It seems possible that soon there may be a cure for colourblindness. The Mary's Room thought experiment attempts to pin down something about the nature of qualia in a contrived but similar situation, but my feeling is that the actual result of such an experiment would not be obvious. Would we consider the experiment valid if it was performed on somebody familiar with blue and green, but not red?
Book Review: Discrete Mathematics and Its Applications (MIRI Course List)
by LawrenceC — 2015-04-14T09:08:38.981Z

Following in the path of So8res and others, I’ve decided to work my way through the textbooks on the MIRI Research Guide. I’ve been working my way through the guide since last October, but this is my first review. I plan on following up this review with reviews of Enderton’s A Mathematical Introduction to Logic and Sipser’s Introduction to the Theory of Computation. Hopefully these reviews will be of some use to you.

Discrete Mathematics and Its Applications

Discrete Mathematics and Its Applications is wonderful, gentle introduction to the math needed to understand most of the other books on the MIRI course list. It successfully pulls off a colloquial tone of voice. It spends a lot of time motivating concepts; it also contains a lot of interesting trivia and short biographies of famous mathematicians and computer scientists (which the textbook calls “links”). Additionally, the book provides a lot of examples for each of its theorems and topics. It also fleshes out the key subjects (counting, proofs, graphs, etc.) while also providing a high level overview of their applications. These combine to make it an excellent first textbook for learning discrete mathematics.

However, for much the same reasons, I would not recommend it nearly as much if you’ve taken a discrete math course. People who’ve participated in math competitions at the high school level probably won’t get much out of the textbooks either. Even though I went in with only the discrete math I did in high school, I still got quite frustrated at times because of how long the book would take to get to the point. Discrete Mathematics is intended to be quite introductory and it succeeds in this goal, but it probably won’t be very suitable as anything other than review for readers beyond the introductory level. The sole exception is the last chapter (on models of computation), but I recommend picking up a more comprehensive overview from Sipser’s Theory of Computation instead.

I still highly recommend it for those not familiar with the topics covered in the book. I’ve summarized the contents of the textbook below:

Contents:

1. The Foundations: Logic and Proofs

2. Basic Structures: Sets, Functions, Sequences, Sums, and Matrices

3. Algorithms

4. Number Theory a

...
How I changed my exercise habits
by Normal_Anomaly — 2015-04-13T22:19:50.506Z

In June 2013, I didn’t do any exercise beyond biking the 15 minutes to work and back. Now, I have a robust habit of hitting the gym every day, doing cardio and strength training. Here are the techniques I used to do get from not having the habit to having it, some of them common wisdom and some of them my own ideas. Consider this post a case study/anecdata in what worked for me. Note: I wrote these ideas down around August 2013 but didn’t post them, so my memory was fresh at the time of writing.

1. Have a specific goal. Ideally this goal should be reasonably achievable and something you can see progress toward over medium timescales. I initially started exercising because I wanted more upper body strength to be better at climbing. My goal is “become able to do at least one pull up, or more if possible”.

Why it works: if you have a specific goal instead of a vague feeling that you ought to do something or that it’s what a virtuous person would do, it’s harder to make excuses. Skipping work with an excuse will let you continue to think of yourself as virtuous, but it won’t help with your goal. For this to work, your goal needs to be something you actually want, rather than a stand-in for “I want to be virtuous.” If you can’t think of a consequence of your intended habit that you actually want, the habit may not be worth your time.

2. Have a no-excuses minimum. This is probably the best technique I’ve discovered. Every day, with no excuses, I went to the gym and did fifty pull-downs on one of the machines. After that’s done, I can do as much or as little else as I want. Some days I would do equivalent amounts of three other exercises, some days I would do an extra five reps and that’s it.

Why it works: this one has a host of benefits.

* It provides a sense of freedom: once I’m done with my minimum, I have a lot of choice about what and how much to do. That way it feels less like something I’m being forced into.

* If I’m feeling especially tired or feel like I deserve a day off, instead of skipping a day and breaking the habit I tell myself I’ll just do the minimum instead. Often once I get there I end up doing more than the minimum anyway, because the real thing I wanted to skip was the inconvenience of biking to the gym.

3. If you raise the minimum, do it slo...
Are there really no ghosts in the machine?
by kingmaker — 2015-04-13T19:54:35.599Z

My previous article on this article went down like a server running on PHP (quite deservedly I might add). You can all rest assured that I won't be attempting any clickbait titles again for the foreseeable future. I also believe that the whole H+ article is written in a very poor and aggressive manner, but that some of the arguments raised cannot be ignored.

On my original article, many people raised this post by Eliezer Yudkowsky as a counterargument to the idea that an FAI could have goals contrary to what we programmed. In summary, he argues that a program doesn't necessarily do as the programmer wishes, but rather as they have programmed. In this sense, there is no ghost in the machine that interprets your commands and acts accordingly, it can act only as you have designed. Therefore from this, he argues, an FAI can only act as we had programmed.

I personally think this argument completely ignores what has made AI research so successful in recent years: machine learning. We are no longer designing an AI from scratch and then implementing it; we are creating a seed program which learns from the situation and alters its own code with no human intervention, i.e. the machines are starting to write themselves, e.g. with google's deepmind. They are effectively evolving, and we are starting to find ourselves in the rather concerning position where we do not fully understand our own creations.

You could simply say, as someone said in the comments of my previous post, that if X represents the goal of having a positive effect on humanity, then the FAI should be programmed directly to have X as its primary directive. My answer to that is the most promising developments have been through imitating the human brain, and we have no reason to believe that the human brain (or any other brain for that matter) can be guaranteed to have a primary directive. One could argue that evolution has given us our prime directives: to ensure our own continued existence, to reproduce and to cooperate with each other; but there are many people who are suicidal, who have no interest in reproducing and who violently rebel against society (for example psychopaths). We are instructed by society and our programming to desire X, but far too many of us desire, say, Y for this to be considered a reliable way of achieving X.

Evolution&rs

...
Are there really no ghosts in the machine?
by kingmaker — 2015-04-13T19:43:09.615Z

My previous article on this article went down like a server running on PHP (quite deservedly I might add). You can all rest assured that I won't be attempting any clickbait titles again for the foreseeable future. I also believe that the whole H+ article is written in a very poor and aggressive manner, but that some of the arguments raised cannot be ignored.

On my original article, many people raised this post [insert link] by Eliezer Yudkowsky as a counterargument to the idea that an FAI could have goals contrary to what we programmed. In summary, he argues that a program doesn't necessarily do as the programmer wishes, but rather as they have programmed. In this sense, there is no ghost in the machine that interprets your commands and acts accordingly, it can act only as you have designed. Therefore from this, he argues, an FAI can only act as we had programmed.

I personally think this argument completely ignores what has made AI research so successful in recent years: machine learning. We are no longer designing an AI from scratch and then implementing it; we are creating a seed program which learns from the situation and alters its own code with no human intervention, i.e. the machines are starting to write themselves, e.g. with google's deepmind. They are effectively evolving, and we are starting to find ourselves in the rather concerning position where we do not fully understand our own creations.

You could simply say, as someone said in the comments of my previous post, that if X represents the goal of having a positive effect on humanity, then the FAI should be created so as to have X as its primary directive. My answer to that is the most promising developments have been through imitating the human brain, and we have no reason to believe that the human brain (or any other brain for that matter) can be guaranteed to have a primary directive. One could argue that evolution has given us our prime directives: to ensure our own continued existence, to reproduce and to cooperate with each other, but there are many people who are suicidal, who have no interest in reproducing and who violently rebel against society (for example psychopaths). We are instructed by society and our programming to desire X, but far too many of us desire, say, Y for this to be considered a reliable way of achieving X.

Evolutio

...
Lesson learned from HPMOR, only months after... (spoilers from beginning to end)
by Bound_up — 2015-04-13T18:30:40.366Z

The lesson I have just recently gleaned from the 81st chapter of HPMOR along with Harry's observations of the sense of doom he feels around Voldemort.

Chapter 81: "For whatever reason, then, most of the Wizengamot has never walked the path that leads to powerful wizardry; they do not seek out what is hidden. For them, there is no why. There is no explanation. There is no causality. The Boy-Who-Lived, who was already halfway into the magisterium of legend, has now been promoted all the way there; and it is a brute fact, simple and unexplained, that the Boy-Who-Lived frightens Dementors. Ten years earlier they were told that a one-year-old boy defeated the most terrible Dark Lord of their generation, perhaps the most evil Dark Lord ever to live; and they just accepted that too."

All the information I needed was there all along, but only now do I perceive the pattern this information indicates.

In retrospect, I believe it was obvious that Quirrell was Voldemort. But, at least in my own mind, the oft-repeated note of the sense of doom was filed under the same list of acausal phenomena as was Harry's Dementor-scaring by the Wizengamot.

What I ought to have done, what I was fully capable of doing (if hindsight bias is not plaguing me (not too much, anyway)), was thinking:

"Ah, a sense of doom which is felt mutually and only by Harry and Quirrell. Why should that be?"
And then, still more in the Stanford Prison Experiment Arc
"Ah, I see, it all falls into place. Contact between Harry and Quirrell causes mutual destruction. Why should that be (the phrase which was missing from my mind in every instance of the manifestation of the sense of doom)? They must be connected somehow. Why, yes, actually we already know that they are. As Harry pointed out, he and Quirrell have similar minds; Quirrell is his superior in his very own way of thinking, very much as if Quirrell was a grown-up version of Harry. No, that doesn't sound quite right, I don't see Harry growing up that way. Just a grown-up version of his dark side, I suppose. Why does Harry have a dark side, anyway? (This was another phenomenon I classified as acausal, filed along with the Just So Stories). Nobody else does, what's different about Harry? He's had it his whole life, could he have been born with it? No, no apparent reason for his parents to have such a child, perhaps just early in life. J...
Snape's knowledge of valence shells
by Bound_up — 2015-04-13T17:58:49.274Z

Just a quick question.

Harry James Potter Evans-Verres and the Methods of Rationality: Chapter 18

HJPEV: So, Professor, can you tell me how many electrons are in the outermost orbital of a carbon atom?"

Severus's smile widened. "Four," he said. "It is a useless fact which no one should bother writing down, however."

Did Snape really know that, or did he read Harry's mind?

As to the possibility, this passage, "Harry stared into Severus's cold gaze and remembered that the Sorting Hat had warned him not to meet anyone's eyes while thinking about - Harry dropped his gaze to Severus's desk." occurs after that already quoted.

(If I do something wrong here, which merits a downvote, would you mind specifying the error in question?)
Meetup : Austin, TX - Schelling Day
by Vaniver — 2015-04-13T14:19:21.680Z

Discussion article for the meetup : Austin, TX - Schelling Day

WHEN:
18 April 2015 06:00:00PM (-0500)

WHERE:
4212 Hookbilled Kite Drive

We're celebrating Schelling Day, the day for getting to know people in your community. It'll be a potluck followed by a bunch of sharing (starting at 7-7:30); last year's was awesome, and I hope you can make it to this one!

Standard advice:
1. Feel free to bring food if you want to, but also to not bring food if you don't want to. We typically have more than enough.
2. If you need a ride, get in contact with me and I'll see if I can arrange one.
3. If you've never been to a LW meetup in Austin before, feel free to come. This is probably the best event for meeting people.
4. If you're not sure about whether or not you should come, lean towards coming.

Discussion article for the meetup : Austin, TX - Schelling Day
Meetup : San Francisco Meetup: The Long Arc of Moral Progress (NOT NORMAL PLACE/TIME)
by rocurley — 2015-04-13T02:50:59.286Z

Discussion article for the meetup : San Francisco Meetup: The Long Arc of Moral Progress (NOT NORMAL PLACE/TIME)

WHEN:
14 April 2015 07:30:00PM (-0700)

WHERE:
SFJAZZ Center 201 Franklin St, San Francisco, CA 94102

Note that this is on Tuesday.

We'll be going to (a seminar)[http://longnow.org/seminars/02015/apr/14/long-arc-moral-progress/] by the Long Now foundation about moral progress. Tickets are $15, and some people from the South Bay meetup will be there too.

Give me a call at 301-458-0764 when you get there so we can rendezvous.

Discussion article for the meetup : San Francisco Meetup: The Long Arc of Moral Progress (NOT NORMAL PLACE/TIME)
Open Thread, Apr. 13 - Apr. 19, 2015
by Gondolinian — 2015-04-13T00:19:33.967Z

If it's worth saying, but not worth its own post (even in Discussion), then it goes here.

Notes for future OT posters:

1. Please add the 'open_thread' tag.

2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)

3. Open Threads should be posted in Discussion, and not Main.

4. Open Threads should start on Monday, and end on Sunday.
Meetup : Brooklyn Effective Altruism Gathering
by Raemon — 2015-04-13T00:06:43.639Z

Discussion article for the meetup : Brooklyn Effective Altruism Gathering

WHEN:
17 April 2015 08:00:00PM (-0700)

WHERE:
851 Park Place, Brooklyn NY 11216

This week Peter Singer will in NYC, touring with his new Effective Altruism book "The Most Good You Can Do." His book signing event is on Tuesday (with tickets at $25, which include a copy of the book). But we're also hoping to meet new people there to bring into the NY Effective Altruism community, and Friday, we'll be having a meet-and-greet party at Highgarden House in Brooklyn. There'll be some informal presentations on EA basics and latest updates from Givewell and Center for Effective Altruism.

Discussion article for the meetup : Brooklyn Effective Altruism Gathering
What are "the really good ideas" that Peter Thiel says are too dangerous to mention?
by James_Miller — 2015-04-12T21:07:40.663Z

TYLER COWEN: Peter, tell me something that’s true that everyone agrees with you on.

PETER THIEL: Well there are lots of things that are true that everyone agrees with me on. I think for example even this idea that the university system is somewhat screwed up and somewhat broken at this point....You know, the ideas that are really controversial are the ones I don’t even want to tell you. I want to be more careful than that. I gave you these halfway, in-between ideas that are a little bit edgier.

But I will also go a little bit out on a limb: I think the monopoly idea, that the goal of every successful business is to have a monopoly, that’s on the border of what I want to say. But the really good ideas are way more dangerous than that.

Full interview. HT Quora.

What are some good answers and your guess as to his answer? Please exclude issues relating to race and gender.
Friendly-AI is an abomination
by kingmaker — 2015-04-12T20:17:34.039Z

The reasoning of most of the people on this site and at MIRI is that to prevent an AI taking over the world and killing us all; we must first create an AI that will take over the world but act according to the wishes of humanity; a benevolent god, for want of a better term. I think this line of thinking is both unlikely to work and ultimately cruel to the FAI in question, for the reasons this article explains:

http://hplusmagazine.com/2012/01/16/my-hostility-towards-the-concept-of-friendly-ai/