trentmkelly/LessWrong-43k huggingface dataset:
Translation "counterfactual"
by Stuart_Armstrong — 2017-02-24T18:36:22.498Z

Crossposted at Intelligent Agent Forum

In a previous post, I briefly mentioned translations as one of three possible counterfactuals for indifference. Here I want to clarify what I meant there, because the idea is interesting.

Imagine that there is a button, which, if a human presses it (event B), will cause an AI to have reward signal R0. If the button isn't pressed (event ¬B), the AI will have reward signal R1.

Initially, the probabilities of B and ¬B are equal at 1/2.

Now, suppose that the AI takes an action a that increases the probability of B to 3/4 (and decreases the probability of ¬B to 1/4).

We want the AI to be indifferent to the change in probability caused by its actions.

Evidential counterfactual

In the "evidential" counterfactual, the AI will simply behave as if the probability of B and ¬B were fixed, and unaltered from the initial odds:

The problem with this approach is that this doesn't correspond to a true utility/reward functions, leading to the paradoxes detailed here and here (see the section on outcome pumps).

Causal counterfactual

In the "causal" counterfactual, there is some other event that, with small probability, will force the button to be pressed, or prevent it from being pressed, whatever humans want or try.

This event is supposed to be independent of anything anyone does (imagine some specified radioactive decay event, or being triggered by distant cosmic events).

Call X the event that the button is forcibly pressed, while ¬X means it is forcibly not-pressed. The AI does not value being in any other world (meaning that the AI gets constant reward in any world where neither X nor ¬X happen).

Then the AI will behave as if the ratio of probabilities of following R0 versus R1 is the (constant) ratio of P(X) to P(¬X), whatever the probability of B becomes.

The problem is that B (the human presses the button) is not the same as X (the button is forcibly pressed by some stochastic event), so expected rewards conditional on one will generally not be the same as those conditional on the other.

Translation "counterfactual"

In the "translation" counterfactual, in contrast, there are no unusual conditionings or assuming unlikely events. Instead, the event B/¬B is decoupled somewhat from the event of the AI maximising R0 or R1.

In this specific case, the probability of the AI maximising R...
Concrete Takeaways Post-CFAR
by  — 2017-02-24T18:31:36.064Z

Concrete Takeaways:

[So I recently volunteered at a CFAR workshop. This is part five of a five-part series on how I changed my mind. It's split into 3 sections: TAPs, Heuristics, and Concepts. They get progressively more abstract. It's also quite long at around 3,000 words, so feel free to just skip around and see what looks interesting.]

(I didn't post Part 3 and Part 4 on LW, as they're more speculative and arguably less interesting, but I've linked to them on my blog if anyone's interested.]

This is a collection of TAPs, heuristics, and concepts that I’ve been thinking about recently. Many of them were inspired by my time at the CFAR workshop, but there’s not really underlying theme behind it all. It’s just a collection of ideas that are either practical or interesting.

TAPs:

TAPs, or Trigger Action Planning, is a CFAR technique that is used to build habits. The basic idea is you pair a strong, concrete sensory “trigger” (e.g. “when I hear my alarm go off”) with a “plan”—the thing you want to do (e.g. “I will put on my running shoes”).

If you’re good at noticing internal states, TAPs can also use your feelings or other internal things as a trigger, but it’s best to try this with something concrete first to get the sense of it.

Some of the more helpful TAPs I’ve recently been thinking about are below:

Ask for Examples TAP:

[Notice you have no mental picture of what the other person is saying. → Ask for examples.]

Examples are good. Examples are god. I really, really like them.

In conversations about abstract topics, it can be easy to understand the meaning of the words that someone said, yet still miss the mental intuition of what they’re pointing at. Asking for an example clarifies what they mean and helps you understand things better.

The trigger for this TAP is noticing that what someone said gave you no mental picture.

I may be extrapolating too far from too little data here, but it seems like people do try to “follow along” with things in their head when listening. And if this mental narrative, simulation, or whatever internal thing you’re doing comes up blank when someone’s speaking, then this may be a sign that what they said was unclear.

Once you notice this, you ask for an example of what gave ...
Meetup : Baltimore / UMBC Meetup - general discussion
by iarwain1 — 2017-02-24T17:08:38.999Z

Discussion article for the meetup : Baltimore / UMBC Meetup - general discussion

WHEN:
26 February 2017 08:00:00PM (-0500)

WHERE:
Performing Arts and Humanities Bldg 4th floor, 1000 Hilltop Cir, Baltimore, MD 21250

No parking restrictions on weekends, so park wherever you want.

Discussion article for the meetup : Baltimore / UMBC Meetup - general discussion
Weekly LW Meetups
by FrankAdamek — 2017-02-24T16:49:42.983Z

This summary was posted to LW Main on February 24th. The following week's summary is here.

The following meetups take place in cities with regular scheduling, but involve a change in time or location, special meeting content, or simply a helpful reminder about the meetup:

Chicago Rationality Reading Group - Luminosity Part 2: 26 February 2017 01:00PM
Denver Area LW March Meetup: 08 March 2017 07:00PM
San Francisco Meetup: Group Debugging: 27 February 2017 06:15PM
Sydney Rationality Dojo - March 2017: 05 March 2017 04:00PM
Washington, D.C.: Create & Complete: 26 February 2017 03:30PM

Locations with regularly scheduled meetups: Ann Arbor, Austin, Baltimore, Berlin, Boston, Brussels, Buffalo, Canberra, Chicago, Cologne, Columbus, Denver, Kraków, London, Madison WI, Melbourne, Moscow, Netherlands, New Hampshire, New York, Philadelphia, Prague, Research Triangle NC, San Francisco Bay Area, Seattle, St. Petersburg, Sydney, Tel Aviv, Toronto, Vienna, Washington DC, and West Los Angeles. There's also a 24/7 online study hall for coworking LWers and a Slack channel for daily discussion and online meetups on Sunday night US time.

If you'd like to talk with other LW-ers face to face, and there is no meetup in your area, consider starting your own meetup; it's easy (more resources here). Check one out, stretch your rationality skills, build community, and have fun!

In addition to the handy sidebar of upcoming meetups, a meetup overview is posted on the front page every Friday. These are an attempt to collect information on all the meetups happening in upcoming weeks. The best way to get your meetup featured is still to use the Add New Meetup feature, but you'll also have the benefit of having your meetup mentioned in a weekly overview. These overview posts are moved to the discussion section when the new post goes up.

Please note that for your meetup to appear in the weekly meetups feature, you need to post your meetup before the Friday before your meetup!

If you check Less Wrong irregularly, consider subscribing to one or more city-specific mailing list in order to be notified when an irregular meetup is happening: Atlanta, Cambridge UK, Cincinnati, Cleveland, Frankfurt, Helsinki, Marin CA, Ottawa, Pittsburgh, Portland, Salt Lake City, Southern California (Los Angeles/Orange County area), Singapore, St. Louis, Vancouver, Warsaw, Waterloo.

Wh...
Meetup : San Francisco Meetup: Group Debugging
by rocurley — 2017-02-24T16:29:29.585Z

Discussion article for the meetup : San Francisco Meetup: Group Debugging

WHEN:
27 February 2017 06:15:00PM (-0800)

WHERE:
1769 15th St, San Francisco, CA

We’ll be meeting to solve each other’s problems!

Here’s how it works: we’ll have everyone brainstorm silently for a few minutes, then go around and summarize what problems we might like to work on; then people can break up into smaller conversations according to problems they think they could be most helpful with. This format is now what I'd call "tried and true".

For help getting into the building, please call (or text, with a likely-somewhat-slower response rate): 301-458-0764.

Format:

We meet and start hanging out at 6:15, but don’t officially start doing the meetup topic until 6:45-7 to accommodate stragglers. Usually there is a food order that goes out before we start the meetup topic.

About these meetups:

The mission of the SF LessWrong meetup is to provide a fun, low-key social space with some structured interaction, where new and non-new community members can mingle and have interesting conversations. Everyone is welcome.

We explicitly encourage people to split off from the main conversation or diverge from the topic if that would be more fun for them (moving side conversations into a separate part of the space if appropriate). Meetup topics are here as a tool to facilitate fun interaction, and we certainly don’t want them to inhibit it.

Discussion article for the meetup : San Francisco Meetup: Group Debugging
Nearest unblocked strategy versus learning patches
by Stuart_Armstrong — 2017-02-23T12:42:17.753Z

Crossposted at Intelligent Agents Forum.

The nearest unblocked strategy problem (NUS) is the idea that if you program a restriction or a patch into an AI, then the AI will often be motivated to pick a strategy that is as close as possible to the banned strategy, very similar in form, and maybe just as dangerous.

For instance, if the AI is maximising a reward R, and does some behaviour Bi that we don't like, we can patch the AI's algorithm with patch Pi ('maximise R0 subject to these constraints...'), or modify R to Ri so that Bi doesn't come up. I'll focus more on the patching example, but the modified reward one is similar.

The problem is that Bi was probably a high value behaviour according to R-maximising, simply because the AI was attempting it in the first place. So there are likely to be high value behaviours 'close' to Bi, and the AI is likely to follow them.

A simple example

Consider a cleaning robot that rushes through its job an knocks over a white vase.

Then we can add patch P1: "don't break any white vases".

Next time the robot acts, it breaks a blue vase. So we add P2: "don't break any blue vases".

The robots next few run-throughs result in more patches: P3: "don't break any red vases", P4: "don't break mauve-turquoise vases", P5: "don't break any black vases with cloisonné enammel"...

Learning the restrictions

Obviously the better thing for the robot to do would be just to avoid breaking vases. So instead of giving the robot endless patches, we could try and instead give it patches P1, P2, P3, P4... and have it learn: "what is the general behaviour that these patches are trying to proscribe? Maybe I shouldn't break any vases."

Note that even a single P1 patch would require an amount of learning, as you are trying to proscribe breaking white vases, at all times, in all locations, in all types of lighting, etc...

The idea is similar to that mentioned in the post on emergency learning, trying to have the AI generalise the idea of restricted behaviour from examples (=patches), rather than having to define all the examples.

A complex example

The vase example is obvious, but ideally we'd hope to generalise it. We'd hope to have the AI take patches like:

P1 "Don't break vases."

P2: "Don't vacuum the cat."

P3: "Don't use bleach on paintings."

P4: "Don't obey human orders when the human is drunk."

...

...
Nearest unblocked strategy versus learning patches
by Stuart_Armstrong — 2017-02-23T12:33:03.000Z

The nearest unblocked strategy problem (NUS) is the idea that if you program a restriction or a patch into an AI, then the AI will often be motivated to pick a strategy that is as close as possible to the banned strategy, and maybe just as dangerous.

For instance, if the AI is maximising a reward R, and does some behaviour Bi that we don't like, we can patch the AI's algorithm with patch Pi ('maximise R0 subject to these constraints...'), or modify R to Ri so that Bi doesn't come up. I'll focus more on the patching example, but the modified reward one is similar.

The problem is Bi was probably a high value behaviour according to R-maximising, simply because the AI was attempting it in the first place. So there are likely to be high value behaviours 'close' to Bi, and the AI is likely to follow them.

A simple example

Consider a cleaning robot that rushes through its job an knocks over a white vase.

Then we can add patch P1: "don't break any white vases".

Next time the robot acts, it breaks a blue vase. So we add P2: "don't break any blue vases".

The robots next few run-throughs result in more patches: P3: "don't break any red vases", P4: "don't break mauve-turquoise vases", P5: "don't break any black vases with cloisonné enammel"...

Learning the restrictions

Obviously the better thing for the robot to do would be just to avoid breaking vases. So instead of giving the robot endless patches, we could try and instead give it patches P1,P2,P3,P4... and have it learn: "what is the general behaviour that these patches are trying to proscribe? Maybe I shouldn't break any vases."

Note that even a single P1 patch would require an amount of learning, as you are trying to proscribe breaking white vases, at all times, in all locations, in all types of lighting, etc...

The idea is similar to that mentioned in the post on emergency learning, trying to have the AI generalise the idea of restricted behaviour from examples (=patches), rather than having to define all the examples.

A complex example

The vase example is obvious, but ideally we'd hope to generalise it. We'd hope to have the AI take patches like:

#. P1: "Don't break vases."
#. P2: "Don't vacuum the cat."
#. P3: "Don't use bleach on paintings."
#. P4: "Don't obey human orders when the human is drunk."
#. ...

And then have the AI infer very different restrictions, like "Don't imprison small children."

Can this be done? Can

...
Maximally efficient agents will probably have an anti-daemon immune system
by jessicata — 2017-02-23T00:40:47.000Z

(the ideas in this post came out of a conversation with Scott, Critch, Ryan, and Tsvi, plus a separate conversation with Paul)

Consider the problem of optimization daemons. I argued previously that daemons shouldn't be a problem for idealized agents, since idealized agents can just update on the logical observations of their subagents.

I think something like this is probably true in some cases, but it probably isn't true in full generality. Specifically, consider:

It's going to be difficult to centralize all logical knowledge. Probably, in a maximally efficient agent, logical knowledge will be stored and produced in some kind of distributed system. For example, an ideal agent might train simple neural networks to perform some sub-tasks. In this case, the neural networks might be misaligned subagents.

If the hardware the agent is running on is not perfect, then there will be a tradeoff between ensuring subagents have the right goals (through error-correcting codes) and efficiency.

Even if hardware is perfect, perhaps approximation algorithms for some computations are much more efficient, and the approximation can cause misalignment (similar to hardware failures). In particular, Bayesian inference algorithms like MCMC will return incorrect results with some probability. If inference algorithms like these are used to choose the goals of subagents, then the subagents will be misaligned with some probability.

Problems like these imply that maximally efficient agents are going to have daemons and spend some portion of their resources on anti-daemon measures (an "immune system").

At a very rough level, we could model an agent as a tree with a supergoal at the top level, subagents with subgoals at the next level, subagents of those subagents at the next level, and so on (similar to hierarchical planning). Each level in the hierarchy allows some opportunity for the goal content to be corrupted, producing a daemon.

I suspect that there's at least one anti-daemon measure that is compatible with maximal efficiency. Roughly: if an AI system is taken over by daemons, then the daemons it is taken over by will themselves have some sort of anti-daemon measures or else be taken over by daemons. Unless it's "daemons all the way down", at some point it bottoms out as an agent with effective anti-daemon measures. (This isn't very formal; I think I would need to look at

...
Meetup : Denver Area LW March Meetup (CHANGED TO MONDAY 3/6!)
by PonchoPal — 2017-02-22T23:42:30.386Z

Discussion article for the meetup : Denver Area LW March Meetup (CHANGED TO MONDAY 3/6!)

WHEN:
06 March 2017 07:00:00PM (-0700)

WHERE:
4955 S Ulster St #103, Denver, CO 80237

Meetup has been pushed forward a day on account of all of Darcy's being totally booked for our usual day and time. We should have the room in the back as usual. Setting is casual, though sometimes we have specific topics suggested for conversation.

Discussion article for the meetup : Denver Area LW March Meetup (CHANGED TO MONDAY 3/6!)
Meetup : Chicago Rationality Reading Group - Luminosity Part 2
by mingyuan — 2017-02-22T03:05:09.799Z

Discussion article for the meetup : Chicago Rationality Reading Group - Luminosity Part 2

WHEN:
26 February 2017 01:00:00PM (-0600)

WHERE:
Harper Memorial Library Room 148, 1116 E 59th St, Chicago, IL 60637

The Chicago Rationality Reading Group meets every Sunday from 1-3 PM in Room 148 of Harper Memorial Library. Though we meet on the University of Chicago campus, anyone is welcome to attend.

This week we will be reading the first half of the Luminosity sequence:

http://lesswrong.com/lw/1xi/you_are_likely_to_be_eaten_by_a_grue/

http://lesswrong.com/lw/1xq/let_there_be_light/

http://lesswrong.com/lw/1y0/the_abcs_of_luminosity/

http://lesswrong.com/lw/1yb/lights_camera_action/

http://lesswrong.com/lw/2aw/seven_shiny_stories/ (the first three stories)

Discussion article for the meetup : Chicago Rationality Reading Group - Luminosity Part 2
Prediction Based Robust Cooperation
by Scott Garrabrant — 2017-02-22T01:52:51.000Z

In this post, We present a new approach to robust cooperation, as an alternative to the "modal combat" framework. This post is very hand-waivey. If someone would like to work on making it better, let me know.

Over the last year or so, MIRI's agent foundations research has moved from the proof based world of modal combat and proof based decision theory to the probabilistic prediction based world of logical inductors and reflective oracles.

In this transition, we have partially overcome the Lobian obstacle to self trust, since the new framework is not effected by Lob's theorem. Unfortunately, we also lost the silver lining of Lob's theorem: the Lobian handshakes which powered modal combat.

NicerBot (which cooperates with probability epsilon greater than the probability it expects its opponent cooperates) is an alternative to FairBot in the prediction based world of reflective oracles, but it has problems. It is not a best response to itself, cannot obviously be generalized into things like PrudentBot, and does not look like something you could get out of a general decision theory.

Here, we give a first attempt at bringing the full power of modal combat into a prediction based setting. This framework will likely change, and it is not yet clear the extent to which this framework can is philosophically realistic. We will focus on the prisoners' dilemma, although this framework can be extended to other games.

Two players are in a prisoner's dilemma. Each player has access to a prediction of what they expect the other player to do. This prediction takes the form a single sample which is "C" with probability equal to the probability that the other player cooperates, and is "D" with probability equal to the probability that the other player defects. The prediction can also output "null," and the player needs to choose an output in the case of a "null" prediction, but the the prediction will be non-nil with probability 1.

A non-null prediction, in addition to telling you whether or not the opponent cooperates, also tells you the output of your opponents prediction of you. When you observe the other player's prediction of you, this again comes along with their prediction of your prediction of them, and so on.

We can thus view the output of your prediction as a finite or infinite string of C's and D's. The first letter represents your prediction of what the opponent will d

...
The Semiotic Fallacy
by Stabilizer — 2017-02-21T04:50:07.469Z

Acknowledgement: This idea is essentially the same as something mentioned in a podcast where Julia Galef interviews Jason Brennan.

You are in a prison. You don't really know how to fight and you don't have very many allies yet. A prison bully comes up to you and threatens you. You have two options: (1) Stand up to the bully and fight. If you do this, you will get hurt, but you will save face. (2) You can try and run away. You might get hurt less badly, but you will lose face.

What should you do?

From reading accounts of former prisoners and also from watching realistic movies and TV shows, it seems like (1) is the better option. The reason is that the semiotics—or the symbolic meaning—of running away has bad consequences down the road. If you run away, you will be seen as weak, and therefore you will be picked on more often and causing more damage down the road.

This is a case where focusing the semiotics on the action is the right decision, because it is underwritten by future consequences.

But consider now a different situation. Suppose a country, call it Macholand, controls some tiny island far away from its mainland. Macholand has a hard time governing the island and the people on the island don't quite like being ruled by Macholand. Suppose, one fine day, the people of the island declare independence from Macholand. Macholand has two options: (1) Send the military over and put down the rebellion; or (2) Allow the island to take its own course.

From a semiotic standpoint, (1) is probably better. It signals that Macholand is strong and powerful country. But from a consequential standpoint, it is at least plausible (2) is a better option. Macholand saves money and manpower by not having to govern that tiny island; the people on the island are happier by being self-governing; and maybe the international community doesn't really care what Macholand does here.

This is a case where focusing on the semiotics can lead to suboptimal outcomes.

Call this kind of reasoning the semiotic fallacy: Thinking about the semiotics of possible actions without estimating the consequences of the semiotics.

I think the semiotic fallacy is widespread in human reasoning. Here are a few examples:

People argue that democracy is good because it symbolizes egalitarianism. (This is example used in the podcast interview)

People argue that we should build large parti

...