trentmkelly/LessWrong-43k huggingface dataset:
"incomparable" outcomes--multiple utility functions?
by Emanresu — 2014-12-17T00:06:50.086Z

I know that this idea might sound a little weird at first, so just hear me out please?

A couple weeks ago I was pondering decision problems where a human decision maker has to choose between two acts that lead to two "incomparable" outcomes. I thought, if outcome A is not more preferred than outcome B, and outcome B is not more preferred than outcome A, then of course the decision maker is indifferent between both outcomes, right? But if that's the case, the decision maker should be able to just flip a coin to decide. Not only that, but adding even a tiny amount of extra value to one of the outcomes should always make that outcome be preferred. So why can't a human decision maker just make up their mind about their preferences between "incomparable" outcomes until they're forced to choose between them? Also, if a human decision maker is really indifferent between both outcomes, then they should be able to know that ahead of time and have a plan for deciding, such as flipping a coin. And, if they're really indifferent between both outcomes, then they should not be regretting and/or doubting their decision before an outcome even occurs regardless of which act they choose. Right?
I thought of the idea that maybe the human decision maker has multiple utility functions that when you try to combine them into one function some parts of the original functions don't necessarily translate well. Like some sort of discontinuity that corresponds to "incomparable" outcomes, or something. Granted, it's been a while since I've taken Calculus, so I'm not really sure how that would look on a graph.
I had read Yudkowsky's "Thou Art Godshatter" a couple months ago, and there was a point where it said "one pure utility function splintered into a thousand shards of desire". That sounds like the "shards of desire" are actually a bunch of different utility functions.
I'd like to know what others think of this idea. Strengths? Weaknesses? Implications?
New paper from MIRI: "Toward idealized decision theory"
by So8res — 2014-12-16T22:27:39.519Z

I'm pleased to announce a new paper from MIRI: Toward Idealized Decision Theory.

Abstract:

This paper motivates the study of decision theory as necessary for aligning smarter-than-human artificial systems with human interests. We discuss the shortcomings of two standard formulations of decision theory, and demonstrate that they cannot be used to describe an idealized decision procedure suitable for approximation by artificial systems. We then explore the notions of strategy selection and logical counterfactuals, two recent insights into decision theory that point the way toward promising paths for future research.

Following the Corrigibility paper, this is the second in a series of six papers motivating MIRI's active research areas. Also included in the series will be a technical agenda, which motivates all six research areas and describes the reasons why we have selected these topics in particular, and an annotated bibliography, which compiles a fair bit of related work. I plan to post one paper every week or two for the next few months.

I've decided to start with the decision theory paper, as it's one of the meatiest. This paper compiles and summarizes quite a bit of work on decision theory that was done right here on LessWrong. There is a lot more to be said on the subject of decision theory than can fit into a single paper, but I think this one does a fairly good job of describing why we're interested in the field and summarizing some recent work in the area. The introduction is copied below. Enjoy!

As artificially intelligent machines grow more capable and autonomous, the behavior of their decision procedures becomes increasingly important. This is especially true in systems possessing great general intelligence: superintelligent systems could have a massive impact on the world (Bostrom 2014), and if a superintelligent system made poor decisions (by human standards) at a critical juncture, the results could be catastrophic (Yudkowsky 2008). When constructing systems capable of attaining superintelligence, it is important for them to use highly reliable decision procedures.

Verifying that a system works well in test conditions is not sufficient for high confidence. Consider the genetic algorithm of Bird and Layzell (2002), which, if run on a simulated representation of a circuit board, would have evolved an oscillating circuit. Running in reality, t...
Meetup : Sandy, UT - Debugging
by hamnox — 2014-12-16T19:42:07.249Z

Discussion article for the meetup : Sandy, UT - Debugging

WHEN:
19 December 2014 12:00:30PM (-0700)

WHERE:
9425 S Riverside Dr. Sandy, UT

Make friends, learn skills, and build solutions!

Meetup is in the main building, you can just walk in.
Agenda:
Quick Definitions - What is Life Debugging?
Brainstorm Bugs to work on

Skill 1: Surprise-o-Meter
How surprised would you be if everything actually worked out perfectly? How surprised would you be if obstacle X derailed you?

Skill 2: Urge Propagation
Half the battle is won If you can get your system-1 on board with your carefully-laid plans.

Strongly suspect Carrie, who leads our Thinking Fast and Slow book club, won't make this one, so that's on hiatus.

Idea for meetup shamelessly stolen from Katja

Discussion article for the meetup : Sandy, UT - Debugging
Superintelligence 14: Motivation selection methods
by KatjaGrace — 2014-12-16T02:00:53.128Z

This is part of a weekly reading group on Nick Bostrom's book, Superintelligence. For more information about the group, and an index of posts so far see the announcement post. For the schedule of future topics, see MIRI's reading guide.

Welcome. This week we discuss the fourteenth section in the reading guide: Motivation selection methods. This corresponds to the second part of Chapter Nine.

This post summarizes the section, and offers a few relevant notes, and ideas for further investigation. Some of my own thoughts and questions for discussion are in the comments.

There is no need to proceed in order through this post, or to look at everything. Feel free to jump straight to the discussion. Where applicable and I remember, page numbers indicate the rough part of the chapter that is most related (not necessarily that the chapter is being cited for the specific claim).

Reading: “Motivation selection methods” and “Synopsis” from Chapter 9.

Summary

One way to control an AI is to design its motives. That is, to choose what it wants to do (p138)

Some varieties of 'motivation selection' for AI safety:

Direct specification: figure out what we value, and code it into the AI (p139-40)

Isaac Asimov's 'three laws of robotics' are a famous example

Direct specification might be fairly hard: both figuring out what we want and coding it precisely seem hard

This could be based on rules, or something like consequentialism

Domesticity: the AI's goals limit the range of things it wants to interfere with (140-1)

This might make direct specification easier, as the world the AI interacts with (and thus which has to be thought of in specifying its behavior) is simpler.

Oracles are an example

This might be combined well with physical containment: the AI could be trapped, and also not want to escape.

Indirect normativity: instead of specifying what we value, specify a way to specify what we value (141-2)

e.g. extrapolate our volition

This means outsourcing the hard intellectual work to the AI

This will mostly be discussed in chapter 13 (weeks 23-5 here)

Augmentation: begin with a creature with desirable motives, then make it smarter, instead of designing good motives from scratch. (p142)

e.g. brain emulations are likely to have human desires (at least at the start)

Whether we use this method depends on the kind of AI that is d

...
Kickstarting the audio version of the upcoming book "The Sequences"
by Rick_from_Castify — 2014-12-16T01:01:42.234Z

LessWrong is getting ready to release an actual book that covers most of the material found in the Sequences.

There have been a few posts about it in the past, here are two: the title debate, content optimization.

We've been asked if we'd like to produce the audiobook version and the answer is yes. This is a large undertaking. The finished product will probably be over 35 hours of audio.

To help mitigate our risk we've decided to Kickstarter the audiobook. This basically allows us to pre-sell it so we're not stuck with a large production cost and no revenue.

The kickstarter campaign is here: https://www.kickstarter.com/projects/1267969302/lesswrong-the-sequences-audiobook

If you haven't heard of us before we've already produced some sequences into audiobooks. You can see them and listen to samples which are indicative of the audio quality here.
Meetup : London Social Meetup, 21/12/2014
by Tenoke — 2014-12-15T23:54:42.764Z

Discussion article for the meetup : London Social Meetup, 21/12/2014

WHEN:
21 December 2014 02:00:00PM (+0000)

WHERE:
Shakespeare's Head, Africa House, 64-68 Kingsway, London WC2B 6BG, UK-68 Kingsway, London WC2B 6BG, UK

LessWrong London is having another meetup this Sunday (21/12) at 2:00 PM. We are meeting at our usual venue - The Shakespeare's Head by Holborn tube station. There is no fixed topic of discussion nor is there anything planned so be prepared for anything. There will be a sign identifying us and if you have any problems feel free to contact me on 07425168803.

About London LessWrong:

We run this meetup almost every week; these days we tend to get in the region of 5-15 people in attendance. By default, meetups are just unstructured social discussion about whatever strikes our fancy: books we're reading, recent posts on LW/related blogs, logic puzzles, toilet usage statistics....
Sometimes we play The Resistance or other games. We usually finish around 7pm, give or take an hour, but people arrive and leave whenever suits them.

If you want more information about the meetup or anything else come by our google group or alternatively our facebook group.

Discussion article for the meetup : London Social Meetup, 21/12/2014
How many people am I?
by Manfred — 2014-12-15T18:11:39.098Z

Strongly related: the Ebborians

Imagine mapping my brain into two interpenetrating networks. For each brain cell, half of it goes to one map and half to the other. For each connection between cells, half of each connection goes to one map and half to the other. We can call these two mapped out halves Manfred One and Manfred Two. Because neurons are classical, as I think, both of these maps change together. They contain the full pattern of my thoughts. (This situation is even more clear in the Ebborians, who can literally split down the middle.)

So how many people am I? Are Manfred One and Manfred Two both people? Of course, once we have two, why stop there - are there thousands of Manfreds in here, with "me" as only one of them? Put like that it sounds a little overwrought - what's really going on here is the question of what physical system corresponds to "I" in english statements like "I wake up." This may matter.

The impact on anthropic probabilities is somewhat straightforward. With everyday definitions of "I wake up," I wake up just once per day no matter how big my head is. But if the "I" in that sentence is some constant-size physical pattern, then "I wake up" is an event that happens more times if my head is bigger. And so using the variable people-number definition, I expect to wake up with a gigantic head.

The impact on decisions is less big. If I'm in this head with a bunch of other Manfreds, we're all on the same page - it's a non-anthropic problem of coordinated decision-making. For example, if I were to make any monetary bets about my head size, and then donate profits to charity, no matter what definition I'm using, I should bet as if my head size didn't affect anthropic probabilities. So to some extent the real point of this effect is that it is a way anthropic probabilities can be ill-defined. On the other hand, what about preferences that depend directly on person-numbers like how to value people with different head sizes? Or for vegetarians, should we care more about cows than chickens, because each cow is more animals than a chicken is?

According to my common sense, it seems like my body has just one person in it. Why does my common sense think that? I think there are two answers, one unhelpful and one helpful.

The first answer is evolution. Having kids is an action that's independent of what physical system we identify wit...
Has LessWrong Ever Backfired On You?
by Evan_Gaensbauer — 2014-12-15T05:44:32.795Z

Several weeks ago I wrote a heavily upvoted post called Don't Be Afraid of Asking Personally Important Questions on LessWrong. I thought it would only be due diligence if I tried to track users on LessWrong who have received advice on this site and it's backfired. In other words, to avoid bias in the record, we might notice what LessWrong as a community is bad at giving advice about. So, I'm seeking feedback. If you have anecdotes or data of how a plan or advice directly from LessWrong backfired, failed, or didn't lead to satisfaction, please share below.
Group Rationality Diary, December 16-31
by therufs — 2014-12-15T03:30:52.481Z

This is the public group rationality diary for December 16-31.

It's a place to record and chat about it if you have done, or are actively doing, things like:

Established a useful new habit

Obtained new evidence that made you change your mind about some belief

Decided to behave in a different way in some set of situations

Optimized some part of a common routine or cached behavior

Consciously changed your emotions or affect with respect to something

Consciously pursued new valuable information about something that could make a big difference in your life

Learned something new about your beliefs, behavior, or life that surprised you

Tried doing any of the above and failed

Or anything else interesting which you want to share, so that other people can think about it, and perhaps be inspired to take action themselves. Try to include enough details so that everyone can use each other's experiences to learn about what tends to work out, and what doesn't tend to work out.

Thanks to cata for starting the Group Rationality Diary posts, and to commenters for participating.

Previous diary: December 1-15

Next diary: January 1-15

Rationality diaries archive
Welcome to Less Wrong! (7th thread, December 2014)
by Gondolinian — 2014-12-15T02:57:01.853Z

If you've recently joined the Less Wrong community, please leave a comment here and introduce yourself. We'd love to know who you are, what you're doing, what you value, how you came to identify as an aspiring rationalist or how you found us. You can skip right to that if you like; the rest of this post consists of a few things you might find helpful. More can be found at the FAQ.

A few notes about the site mechanics

To post your first comment, you must have carried out the e-mail confirmation: When you signed up to create your account, an e-mail was sent to the address you provided with a link that you need to follow to confirm your e-mail address. You must do this before you can post!

Less Wrong comments are threaded for easy following of multiple conversations. To respond to any comment, click the "Reply" link at the bottom of that comment's box. Within the comment box, links and formatting are achieved via Markdown syntax (you can click the "Help" link below the text box to bring up a primer).

You may have noticed that all the posts and comments on this site have buttons to vote them up or down, and all the users have "karma" scores which come from the sum of all their comments and posts. This immediate easy feedback mechanism helps keep arguments from turning into flamewars and helps make the best posts more visible; it's part of what makes discussions on Less Wrong look different from those anywhere else on the Internet.

However, it can feel really irritating to get downvoted, especially if one doesn't know why. It happens to all of us sometimes, and it's perfectly acceptable to ask for an explanation. (Sometimes it's the unwritten LW etiquette; we have different norms than other forums.) Take note when you're downvoted a lot on one topic, as it often means that several members of the community think you're missing an important point or making a mistake in reasoning— not just that they disagree with you! If you have any questions about karma or voting, please feel free to ask here.

Replies to your comments across the site, plus private messages from other users, will show up in your inbox. You can reach it via the little mail icon beneath your karma score on the upper right of most pages. When you have a new reply or message, it glows red. You can also click on any user's name to view all of their comments and p

...
Open thread, Dec. 15 - Dec. 21, 2014
by Gondolinian — 2014-12-15T00:01:06.830Z

If it's worth saying, but not worth its own post (even in Discussion), then it goes here.

Previous Open Thread

Next Open Thread

Notes for future OT posters:

1. Please add the 'open_thread' tag.

2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)

3. Open Threads should be posted in Discussion, and not Main.

4. Open Threads should start on Monday, and end on Sunday.
Podcast: Rationalists in Tech
by JoshuaFox — 2014-12-14T16:14:50.613Z

I'll appreciate feedback on a new podcast, Rationalists in Tech.

I'm interviewing founders, executives, CEOs, consultants, and other people in the tech sector, mostly software. Thanks to Laurent Bossavit, Daniel Reeves, and Alexei Andreev who agreed to be the guinea pigs for this experiment.

The audience:

Software engineers and other tech workers, at all levels of seniority.

The hypothesized need

Some of you are thinking: "I see that some smart and fun people hang out at LessWrong. It's hard to find people like that to work with. I wonder if my next job/employee/cofounder could come from that community."

What this podcast does for you

You will get insights into other LessWrongers as real people in the software profession. (OK, you knew that, but this helps.) You will hear the interviewees' ideas on CfAR-style techniques as a productivity booster, on working with other aspiring rationalists, and on the interviewees' own special areas of expertise. (At the same time, interviewees benefit from exposure that can get them business contacts, employees, or customers.) Software engineers from LW will reach out to interviewees and others in the tech sector, and soon, more hires and startups will emerge.

Please give your feedback on the first episodes of the podcast. Do you want to hear more? Should there be other topics? A different interview style? Better music?
Discussion of AI control over at worldbuilding.stackexchange [LINK]
by ike — 2014-12-14T02:59:47.239Z

https://worldbuilding.stackexchange.com/questions/6340/the-challenge-of-controlling-a-powerful-ai

Go insert some rationality into the discussion! (There are actually some pretty good comments in there, and some links to the right places, including LW).
Debunked And Well-Refuted
by Scott Alexander — 2014-12-13T12:08:44.000Z

I.

As usual, I was insufficiently pessimistic.

I infer this from The Federalist‘s article on campus rape:

A new report on sexual assault released today by the U.S. Department of Justice (DOJ) officially puts to bed the bogus statistic that one in five women on college campuses are victims of sexual assault. In fact, non-students are 25 percent more likely to be victims of sexual assault than students, according to the data. And the real number of assault victims is several orders of magnitude lower than one-in-five.

The article compares the older Campus Sexual Assault Survey (which found 14-20% of women were raped since entering college) to the just-released National Crime Victmization Survey (which found that 0.6% of female college students are raped per year). They write “Instead of 1 in 5, the real number is 0.03 in 5.”

So the first thing I will mock The Federalist for doing is directly comparing per year sexual assault rates to per college career sexual assault rates, whereas obviously these are very different things. You can’t quite just divide the latter by four to get the former, but that’s going to work a heck of a lot better than not doing it, so let’s estimate the real discrepancy as more like 0.5% per year versus 5% per year.

But I can’t get too mad at them yet, because that’s still a pretty big discrepancy.

However, faced with this discrepancy a reasonable person might say “Hmm, we have two different studies that say two different things. I wonder what’s going on here and which study we should believe?”

The Federalist staff said “Ha! There’s an old study with findings we didn’t like, but now there’s a new study with different findings we do like. So the old study is debunked!”

II.

My last essay, Beware The Man Of One Study, noted that one thing partisans do to justify their bias is selectively acknowledge studies from only one side of a complicated literature.

The reason it was insufficiently pessimistic is that there are also people like the Federalist staff, who acknowledge the existence of opposing studies, but only with the adjective “debunked” in front of them. By “debunked” they usually mean one of two things:

1. Someone on my side published a study later that found something else

2. Someone on my side accused it of having methodol...
Meetup : Urbana-Champaign: Finishing Up
by Manfred — 2014-12-13T08:42:33.897Z

Discussion article for the meetup : Urbana-Champaign: Finishing Up

WHEN:
14 December 2014 02:00:00PM (-0600)

WHERE:
206 S. Cedar St, Urbana, IL

This is likely the last meetup of the semester. So let's try and look back on the past and collect some reminders of things that might be useful.

Discussion article for the meetup : Urbana-Champaign: Finishing Up