trentmkelly/LessWrong-43k huggingface dataset:
A brief note on factoring out certain variables
by Stuart_Armstrong — 2016-04-07T16:38:26.000Z

Jessica Taylor and Chris Olah has a post on "Maximizing a quantity while ignoring effect through some channel". I'll briefly present a different way of doing this, and compare the two.

Essentially, the AI's utility is given by a function U of a variable C. The AI's actions are a random variable A, but we want to 'factor out' another random variable B.

If we have a probability distribution Q over actions, then, given background evidence E, the standard way to maximise U(C) would be to maximise:

∑a,b,cU(c)P(C=c,B=b,A=a|e)=∑a,b,cU(c)P(C=c|B=b,A=a,e)P(B=b|A=a,e)Q(A=a|e).

The most obvious idea, for me, is to replace P(B=b|A=a,e) with P(B=b|e), making B artificially independent of A and giving the expression:

∑a,b,cU(c)P(C=c|B=b,A=a,e)P(B=b|e)Q(A=a|e).

If B is dependent on A - if it isn't, then factoring it out is not interesting - then P(B=b) needs some implicit probability distribution over A (which is independent of Q). So, in essence, this approach relies on two distributions over the possible actions, one that the agent is optimising, the other than is left unoptimised. In terms of Bayes nets, this just seems to be cutting B from A.

Jessica and Chris's approach also relies on two distributions. But, as far as I understand their approach, the two distributions are taken to be the same, and instead, it is assumed that U(C) cannot be improved by changes to the distribution of A, if one keeps the distribution of B constant. This has the feel of being a kind of differential condition - the infinitesimal impact on U(C) of changes to A but not B is non-positive.

I suspect my version might have some odd behaviour (defining the implicit distribution for A does not seem necessarily natural), but I'm not sure of the transitive properties of the differential approach.
How do we know our own desires?
by KatjaGrace — 2016-04-07T15:24:56.000Z

Sometimes I find myself longing for something, with little idea what it is.

This suggests that perceiving desire and perceiving which thing it is that is desired by the desire are separable mental actions.

In this state, I make guesses as to what I want. Am I thirsty? (I consider drinking some water and see if that feels appealing.) Do I want to have sex? (A brief fantasy informs me that sex would be good, but is not what I crave.) Do I want social comfort? (I open Facebook, maybe that has social comfort I could test with…)

If I do infer the desire in this way, I am still not directly reading it from my own mind. I am making educated guesses and testing them using my mind’s behavior.

Other times, it seems like I immediately know my own desires. When that happens, am I really receiving them introspectively, or am I merely playing the same inference game more insightfully?

We usually suppose that people are correct about their own immediate desires. They may be wrong about whether they want cookie A or cookie B, because they are misinformed about which one is delicious. But if they think they want to eat something delicious, we trust them on that.

On the model where we are mostly inferring our desires from more general feelings of wanting, we might expect people are wrong about their desires fairly often.
Rationality Reading Group: Part X: Yudkowsky's Coming of Age
by Gram_Stone — 2016-04-06T23:05:28.197Z

This is part of a semi-monthly reading group on Eliezer Yudkowsky's ebook, Rationality: From AI to Zombies. For more information about the group, see the announcement post.

Welcome to the Rationality reading group. This fortnight we discuss Beginnings: An Introduction (pp. 1527-1530) and Part X: Yudkowsky's Coming of Age (pp. 1535-1601). This post summarizes each article of the sequence, linking to the original LessWrong post where available.

Beginnings: An Introduction

X. Yudkowsky's Coming of Age

292. My Childhood Death Spiral - Wherein Eliezer describes how a history of being rewarded for believing that 'intelligence is more important than experience or wisdom' initially led him to dismiss the possibility that most possible smarter-than-human artificial intelligences will cause unvaluable futures if constructed.

293. My Best and Worst Mistake - When Eliezer went into his death spiral around intelligence, he wound up making a lot of mistakes that later became very useful.

294. Raised in Technophilia - When Eliezer was quite young, it took him a very long time to get to the point where he was capable of considering that the dangers of technology might outweigh the benefits.

295. A Prodigy of Refutation - Eliezer's skills at defeating other people's ideas led him to believe that his own (mistaken) ideas must have been correct.

296. The Sheer Folly of Callow Youth - Eliezer's big mistake was when he took a mysterious view of morality.

297. That Tiny Note of Discord - Eliezer started to dig himself out of his philosophical hole when he noticed a tiny inconsistency.

298. Fighting a Rearguard Action Against the Truth - When Eliezer started to consider the possibility of Friendly AI as a contingency plan, he permitted himself a line of retreat. He was now able to slowly start to reconsider positions in his metaethics, and move gradually towards better ideas.

299. My Naturalistic Awakening - Eliezer actually looked back and realized his mistakes when he imagined the idea of an optimization process.

300. The Level Above Mine - There are people who have acquired more mastery over various fields than Eliezer has over his.

301. The Magnitude of His Own Folly - Eliezer considers his training as a rationalist to have star...
My new rationality/futurism podcast
by James_Miller — 2016-04-06T17:36:51.509Z

I've started a podcast called Future Strategist which will focus on decision making and futurism. I have created seven shows so far: interviews of computer scientist Roman Yampolskiy, LW contributor Gleb Tsipursky, and artist/free speech activist Rachel Haywire, and monologues on game theory and Greek Mythology, the Prisoners' Dilemma, the sunk cost fallacy, and the Map and Territory.

If you enjoy the show and use iTunes I would be grateful if you left a positive review at iTunes. I would also be grateful for any feedback you might have including suggestions for future shows. I'm not used to interviewing people and I know that I need to work on being more articulate in my interviews.
Rationality Quotes April 2016
by Sabiola — 2016-04-06T07:01:43.352Z

Another month, another rationality quotes thread. The rules are:

Provide sufficient information (URL, title, date, page number, etc.) to enable a reader to find the place where you read the quote, or its original source if available. Do not quote with only a name.

Post all quotes separately, so that they can be upvoted or downvoted separately. (If they are strongly related, reply to your own comments. If strongly ordered, then go ahead and post them together.)

Do not quote yourself.

Do not quote from Less Wrong itself, HPMoR, Eliezer Yudkowsky, or Robin Hanson. If you'd like to revive an old quote from one of those sources, please do so here.

No more than 5 quotes per person per monthly thread, please.
Rényi divergence as a secondary objective
by jessicata — 2016-04-06T02:08:16.000Z

Summary: Given that both imitation and maximization have flaws, it might be reasonable to interpolate between these two extremes. It's possible to use Rényi divergence (a family of measures of distance between distributions) to define a family of these interpolations.

Rényi divergence is a measure of distance between two probability distributions. It is defined as

Dα(P||Q)=1α−1logEx∼P⎡⎣(P(X)Q(X))α−1⎤⎦

with α≥0. Special values D0,D1,D∞ can be filled in by their limits.

Particularly interesting values are D1(P||Q)=DKL(P||Q) and D∞(P||Q)=logmaxxP(x)Q(x).

Consider some agent choosing a distribution P over actions to simultaneously maximize a score function s and minimize Rényi divergence from some base distribution Q. That is, score P according to

vα(P)=EX∼P[s(X)]−γDα(P||Q)

where γ>0 controls how much the secondary objective is emphasized. Define P∗α=argmaxPvα(P). We have P∗1(x)∝Q(x)es(x)/γ, and P∗∞ is a quantilizer with score function s and base distribution Q (with the amount of quantilization being some function of γ, Q, and s). For 1<α<∞, P∗α will be some interpolation between P∗1 and P∗∞.

It's not necessarily possible to compute vα(P). To approximate this quantity, take samples x1,...,xn∼P and compute

1nn∑i=1s(xi)−1α−1log⎛⎝1nn∑i=1(P(xi)Q(xi))α−1⎞⎠

Of course, this requires P and Q to be specified in a form that allows efficiently estimating probabilities of particular values. For example, P and Q could both be variational autoencoders.

As α approaches 1, this limits to

1nn∑i=1s(xi)−n∑i=1logP(xi)Q(xi)=1nn∑i=1(s(xi)−logP(xi)+logQ(xi))

As α approaches ∞, this limits to

1nn∑i=1s(xi)−logmaxiP(xi)Q(xi)

Like a true quantilizer, a distribution P trained to maximize this value (an approximate quantilizer) will avoid assigning much higher probability to any action than Q does.

These approximations yield training objectives for agents which will interpolate between imitating Q and maximizing s.
What do we use these for? Patrick suggested that Q could be an estimation of the distribution of actions a human would take (trained using
something like this training procedure). Then, the distribution P maximizing the combined objective vα will try to maximize score in a somewhat human-like way; it will interpolate between imitation and score-maximization.

There are problems, though. Suppose a human and an AI can both solve Sudoku, but the AI can't solve it the way a

...
Meetup : Washington, D.C.: Cherry Blossoms
by RobinZ — 2016-04-06T01:43:32.838Z

Discussion article for the meetup : Washington, D.C.: Cherry Blossoms

WHEN:
10 April 2016 03:30:00PM (-0400)

WHERE:
Enid A. Haupt Garden

Note: The meetup location has changed for this week.

We will be gathering at the Enid A. Haupt Garden (outside the south entrance to the Smithsonian Castle, on Independence Ave. between 9th and 12th St. SW, east of the Smithsonian Metro stop) to enjoy the annual cherry blossoms and/or associated festival. We will be congregating around the southwest corner of the garden, near the Sackler Gallery, and set out from there toward the Tidal Basin. If you need to get in touch, my cell number is in the meetup announcement email to the lesswrong-dc Google Group.

Upcoming meetups:

Apr. 17: What If

Apr. 24: Fun & Games

May 1: Utopias

Discussion article for the meetup : Washington, D.C.: Cherry Blossoms
Talk today at CU Boulder
by fowlertm — 2016-04-05T16:26:06.614Z

I'm giving a talk today on the future of governance at the University of Colorado at Boulder, in room ECON 117, at 5 p.m.

While the talk itself isn't concerned with rationality, I'd still be interested in networking with any LW sorts who happen to be in the area.

Best,

-Trent Fowler
[link] Disjunctive AI Risk Scenarios
by Kaj_Sotala — 2016-04-05T12:51:27.794Z

Arguments for risks from general AI are sometimes criticized on the grounds that they rely on a series of linear events, each of which has to occur for the proposed scenario to go through. For example, that a sufficiently intelligent AI could escape from containment, that it could then go on to become powerful enough to take over the world, that it could do this quickly enough without being detected, etc.

The intent of my following series of posts is to briefly demonstrate that AI risk scenarios are in fact disjunctive: composed of multiple possible pathways, each of which could be sufficient by itself. To successfully control the AI systems, it is not enough to simply block one of the pathways: they all need to be dealt with.

I've got two posts in this series up so far:

AIs gaining a decisive advantage discusses four different ways by which AIs could achieve a decisive advantage over humanity. The one-picture version is:

AIs gaining the power to act autonomously discusses ways by which AIs might come to act as active agents in the world, despite possible confinement efforts or technology. The one-picture version (which you may wish to click to enlarge) is:

These posts draw heavily on my old paper, Responses to Catastrophic AGI Risk, as well as some recent conversations here on LW. Upcoming posts will try to cover more new ground.
Meetup : Rationality Nairobi mini-Meetup #1: Double Crux
by negamuhia — 2016-04-05T12:40:17.639Z

Discussion article for the meetup : Rationality Nairobi mini-Meetup #1: Double Crux

WHEN:
30 April 2016 03:34:07PM (+0300)

WHERE:
Up the hill from Lukenya Academy, Machakos

We'll be learning and investigating the dynamics of the game 'Double Crux', a potentially useful tool for approximating Aumannian reasoning. We'll play a few rounds and I (lesswrong.com/user/negamuhia) will describe what happened, with my impressions of difficulty, interest and progression based on the members who participate.

Discussion article for the meetup : Rationality Nairobi mini-Meetup #1: Double Crux
Turning the Technical Crank
by Error — 2016-04-05T05:36:28.811Z

A few months ago, Vaniver wrote a really long post speculating about potential futures for Less Wrong, with a focus on the idea that the spread of the Less Wrong diaspora has left the site weak and fragmented. I wasn't here for our high water mark, so I don't really have an informed opinion on what has socially changed since then. But a number of complaints are technical, and as an IT person, I thought I had some useful things to say.

I argued at the time that many of the technical challenges of the diaspora were solved problems, and that the solution was NNTP -- an ancient, yet still extant, discussion protocol. I am something of a crank on the subject and didn't expect much of a reception. I was pleasantly surprised by the 18 karma it generated, and tried to write up a full post arguing the point.

I failed. I was trying to write a manifesto, didn't really know how to do it right, and kept running into a vast inferential distance I couldn't seem to cross. I'm a product of a prior age of the Internet, from before the http prefix assumed its imperial crown; I kept wanting to say things that I knew would make no sense to anyone who came of age this millennium. I got bogged down in irrelevant technical minutia about how to implement features X, Y, and Z. Eventually I decided I was attacking the wrong problem; I was thinking about 'how do I promote NNTP', when really I should have been going after 'what would an ideal discussion platform look like and how does NNTP get us there, if it does?'

So I'm going to go after that first, and work on the inferential distance problem, and then I'm going to talk about NNTP, and see where that goes and what could be done better. I still believe it's the closest thing to a good, available technological schelling point, but it's going to take a lot of words to get there from here, and I might change my mind under persuasive argument. We'll see.

Fortunately, this is Less Wrong, and sequences are a thing here. This is the first post in an intended sequence on mechanisms of discussion. I know it's a bit off the beaten track of Less Wrong subject matter. I posit that it's both relevant to our difficulties and probably more useful and/or interesting than most of what comes through these days. I just took the 2016 survey and it has a couple of sections on the effects of the diaspora, so I'm guessing it's on topic for meta purposes if not...
The Ideology Is Not The Movement
by Scott Alexander — 2016-04-05T01:52:38.000Z

I.

Why is there such a strong Sunni/Shia divide?

I know the Comparative Religion 101 answer. The early Muslims were debating who was the rightful caliph. Some of them said Abu Bakr, others said Ali, and the dispute has been going on ever since. On the other hand, that was fourteen hundred years ago, both candidates are long dead, and there’s no more caliphate. You’d think maybe they’d let the matter rest.

Sure, the two groups have slightly different hadith and schools of jurisprudence, but how many Muslims even know which school of jurisprudence they’re supposed to be following? It seems like a pretty minor thing to have centuries of animus over.

And so we return again to Robbers’ Cave:

The experimental subjects — excuse me, “campers” — were 22 boys between 5th and 6th grade, selected from 22 different schools in Oklahoma City, of stable middle-class Protestant families, doing well in school, median IQ 112. They were as well-adjusted and as similar to each other as the researchers could manage.

The experiment, conducted in the bewildered aftermath of World War II, was meant to investigate the causes—and possible remedies—of intergroup conflict. How would they spark an intergroup conflict to investigate? Well, the 22 boys were divided into two groups of 11 campers, and —

— and that turned out to be quite sufficient.

The researchers’ original plans called for the experiment to be conducted in three stages. In Stage 1, each group of campers would settle in, unaware of the other group’s existence. Toward the end of Stage 1, the groups would gradually be made aware of each other. In Stage 2, a set of contests and prize competitions would set the two groups at odds.

They needn’t have bothered with Stage 2. There was hostility almost from the moment each group became aware of the other group’s existence: They were using our campground, our baseball diamond. On their first meeting, the two groups began hurling insults. They named themselves the Rattlers and the Eagles (they hadn’t needed names when they were the only group on the campground).

When the contests and prizes were announced, in accordance with pre-established experimental procedure, the intergroup rivalry rose to a fever pitch. Good sportsmanship in the contests was evident for the first two days but rapidly disintegrated.

The Eagl

...
Meetup : Sydney Rationality Dojo - April 2016
by luminosity — 2016-04-04T22:30:24.801Z

Discussion article for the meetup : Sydney Rationality Dojo - April 2016

WHEN:
10 April 2016 04:00:00PM (+1000)

WHERE:
City of Sydney RSL, Restaurant Level

Join us at the City of Sydney RSL, on the restaurant level, for the (slightly delayed) April dojo, from 4 - 6pm.

Discussion article for the meetup : Sydney Rationality Dojo - April 2016
Meetup : San Antonio Meetup
by TheAltar — 2016-04-04T15:54:01.758Z

Discussion article for the meetup : San Antonio Meetup

WHEN:
10 April 2016 02:00:00PM (-0500)

WHERE:
12651 Vance Jackson Rd #118, San Antonio, TX 78230

Bubble tea, frozen yogurt, and discussion. All are welcome.

New Meetup to discuss rationality and all things LessWrong and meet the local community. Look for the sign that says Less Wrong.

Discussion article for the meetup : San Antonio Meetup
Open Thread April 4 - April 10, 2016
by Elo — 2016-04-04T04:56:50.462Z

If it's worth saying, but not worth its own post (even in Discussion), then it goes here.

Notes for future OT posters:

1. Please add the 'open_thread' tag.

2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)

3. Open Threads should be posted in Discussion, and not Main.

4. Open Threads should start on Monday, and end on Sunday.
Meetup : Rationality Vienna Meetup
by AnnaLeptikon — 2016-04-03T22:50:19.187Z

Discussion article for the meetup : Rationality Vienna Meetup

WHEN:
23 April 2016 03:00:00PM (+0200)

WHERE:
Kaisermühlenstraße 24, Vienna

Location: (often depicted wrong, come to U2 Stadlau)

Event on Facebook: https://www.facebook.com/events/1688124771426353/

Topic: room for open microphone (means collecting topics in the beginning and then voting on preferences and order, means that also you can prepare some short talk or discussion!) or the proposal of a full/long lecture

Discussion article for the meetup : Rationality Vienna Meetup
Meetup : UMBC / Baltimore: Intro to LW / general discussion
by iarwain1 — 2016-04-03T22:03:39.765Z

Discussion article for the meetup : UMBC / Baltimore: Intro to LW / general discussion

WHEN:
10 April 2016 10:00:00AM (-0400)

WHERE:
Performing Arts and Humanities Bldg Room 456, 1000 Hilltop Cir, Baltimore, MD 21250

If new members show up we'll do some intro to LW stuff. Otherwise it'll be whatever people want to discuss.

Discussion article for the meetup : UMBC / Baltimore: Intro to LW / general discussion
Fake Amnesia
by Gram_Stone — 2016-04-03T21:23:26.463Z

Followup to: Tonic Judo

Related to: Correspondence Bias

Imagine that someone you know has a reaction that you consider disproportionate to the severity of the event that caused it. If your friend loses their comb, and they get weirdly angry about it, and you persuade them into calming down with rational argument, and then it happens again, say, many months later, and they get just as angry as they did the first time, is that person unteachable? Is it a waste of your time to try to persuade them using rationality?

I think a lot of people would have an expectation that the friend would not have another outburst, and that when the friend had another outburst, that expectation would be violated.

And for some reason, at this turn, it seems like a lot of people think, "I tried to teach this person once, and it didn't work. They're the kind of person who can't be persuaded. I should direct my efforts elsewhere." Maybe you even make it look more 'rational' by name-dropping expected utility.

Or maybe it doesn't feel like stubbornness; maybe it feels like they just forgot. Like they were pretending to listen when they looked like they were listening to your arguments, but really they were just waiting for you to finish talking.

That does happen sometimes, if you fail to emotionally engage someone or if you're hanging out with all the wrong kinds of people.

But most of the time, when you're dealing with the majority of the human race, with all of the people who care about how they behave, the right way to go is to realize that a violation of expectations is a sign that your model is wrong.

You made your first rational argument with the implicit expectation that it would prevent all future outbursts over combs. But it happens again. You shouldn't stop at your first attempt. It may be that circumstances are different this time and an outburst is warranted, or it may be that your friend is not in a state in which your previous arguments are at the level of their attention. Or maybe they feel righteous anger and you need to get them to have less self-confidence and more confidence in you, and maybe you need to encourage them to control that in the future, instead of only the previous object-level impulse.

The point is, you expected your first argument to generalize more than it actually did. People often respond to situations like this as though the fact th...