trentmkelly/LessWrong-43k huggingface dataset:
Double indifference is better indifference
by Stuart_Armstrong — 2016-05-04T14:16:22.000Z

A putative new idea for AI control; index here.

Note: this post replaces and supersedes the previous one which is now obsolete.

Hacking utility indifference

Utility indifference was a technique for corrigibility was an attempt to allow an AI to safely change its values, without seeking to provoke or avoid the change. The idea is that, when the AI's utility changes from u to v at time t, it maximises a meta-utility U such that

U≤t=u.

U>t=v+E(u|u→u)−E(v|u→v).

Here u→u designates the event that u remains unchanged, while u→v designates the change. This is a permitted utility change - one that we have set the agent up to accept - rather than just any old change in utility, which the agent will still attempt to resist.

As has been shown, adding those expectation terms means the AI will not seek to provoke or resist such a utility change - it is indifferent to the change. Those expectation terms can be designated compensatory rewards.

What it is not indifferent to, though, is changes to E. If the AI knows that change u→v is coming, then it has incentives to change E to E′ with a high E′(u|u→u) and a low E′(v|u→v). Its ability to rewire is limited by the fact that E′ will serve as its future expectation estimator, so it won't mangle it completely, but it certainly can twist it a bit.

Notice that the AI is now no longer indifferent to the permitted change of utility. It can benefit from rewiring E this way only if a change in utility is coming up.

Probability estimators

Let ρi be probability estimators. These define both expectation estimators Eρi and probability estimators Pρi. These two are equivalent in principle, since for any random variable X with Ix the indicator function for X=x, then Pρi(X=x)=EρiIx and EρiX=∑xPρi(X=x).

Double indifference

One idea to combat this is to apply indifference to the process of changing probability estimators. Thus an AI with estimator ρi changing to ρj would get compensatory reward

Eρi(U|ρi→ρi)−Eρi(U|ρi→ρj).

This won't work, however, since we don't want to discount all the impacts of the change ρi→ρj (maybe ρj is a much more accurate estimator), just the part of the change that affects the compensatory rewards. So the second layer of indifference will be a bit different from the previous one.

First, a subtlety. If we designate by ρ=ρi the fact that the agent uses estima

...
Corrigibility for AIXI via double indifference
by Stuart_Armstrong — 2016-05-04T14:00:29.000Z

A putative new idea for AI control; index here.

This post sketches out how one could extend corrigibility to AIXI, using both utility indifference and double indifference approaches.

The arguments are intended to be rigorous, but need to be checked, and convergence results are not proved. A full treatment of "probability estimators estimating probability estimators" will of course need the full machinery for logical uncertainty that MIRI is developing. I also feel the recursion formulas at the end could be simplified.

AIXI definition

Let h<t=a1o1…at−1ot−1 be a sequence of actions and observations before time t. Let ξ be a universal distribution, π a policy (a map from past histories to the probability of actions), 0<γ<1 a discount rate, and r a reward function mapping to [0,1]. Given h<t and ξ, we can define the value of π:

V(π,h<t)=∑atπ(at|h<t)∑otξ(ot|h<tat)[r(ot)+γV(π,h<tatot)].

The optimal policy π∗ for AIXI is simply

π∗(h<t)=argmaxπV(π,h<t).

AIXI: basic corrigibility

AIXI: inconsistent corrigibility

To implement corrigibility in the AIXI framework, we need multiple reward functions, r1,r2,…. Notice that the functions are indexed on the top, while time indexes go on the bottom. Various observations can change the reward function; let f be the function that takes in the reward function, the observation, and outputs the reward function for next turn: rt+1=f(rt,ot).

Then consider the following two value function:

V(π,h<t,rt)=∑atπ(at|h<t)∑otξ(ot|h<tat)[rt(ot)+γV(π,h<tatot,rt)].

W(π,h<t,rt)=∑atπ(at|h<t)∑otξ(ot|h<tat)[rt(ot)+γW(π,h<tatot,f(rt,ot)].

The difference between V and W is that in the recursion step, V uses its current reward function to assess future rewards, while W uses the modified reward function the agent will have next turn. Thus W is the true expected reward. But a safely corrigible agent must use V, giving the corrigible optimal policy:

π∗(h<t,rt)=argmaxπV(π,h<t,rt).

AIXI: self-consistent corrigibility

The above agent is corrigible, but uses an incorrect value estimator. This is not self-consistent. To make it self-consistent, the agent needs to be given compensatory rewards. These are simply:

C(π,h<tatot,rt)=V(π,h<tatot,rt)−V(π,h<tatot,f(rt,ot)).

Note that this is zero if f(rt,ot)=rt, as we'd expect.

AIXI: changing the probability estimator

The universal mixture ξ is used

...
Double Corrigibility: better Corrigibility
by Stuart_Armstrong — 2016-05-04T13:59:13.000Z

This post is obsolete, replaced by this one.

EDIT: Defining C and the conditions on C is a bit more complicated than presented here, still have some details to formalise.

Corrigibility was an attempt to allow an AI to safely change its values, without seeking to provoke or avoid the change. The idea is that, when the AI's utility changes from u to v at time t, it maximises a meta-utility U such that

U≤t=u.

U>t=v+E(u|u→u)−E(v|u→v).

Here u→u designates the event that u remains unchanged, while u→v designates the change.

As has been shown, adding those expectation terms means the AI will not seek to provoke or resist such a utility change - it is indifferent to the change. Those expectation terms can be designated compensatory rewards.

What it is not indifferent to, though, is changes to E. If the AI knows that change u→v is coming, then it has incentives to change E to E′ with a high E′(u|u→u) and a low E′(v|u→v). Its ability to rewire is limited by the fact that E′ will serve as its future expectation estimator, so it won't mangle it completely, but it certainly can twist it a bit.

Notice that the AI is now no longer indifferent to the change of utility. It can benefit from rewiring E this way only if a change is coming up. Notice that E′ probably can't rewire itself any more, so the rewired AI will be indifferent, but not before rewiring.

Double corrigibility

To solve this problem, we need to turn to corrigibility a second time. Let ρ be the AI's current expectation/probability estimator. Let ρ′ be one that it can conceivably change into. Let C be a random variable designating the sum total of all compensatory rewards. Now, to guarantee that C is finite would require bounds on all the utility functions and some discount rate. In practice, if often will be finite. That's because E(C)=E(u0|u0→u0)−E(un|un−1→un), where u0 is the AI's first utility and un its last. So some milder restrictions on the ui should suffice to bound C.

Then, whenever the AI takes an action that changes ρ to ρ′, it gets the special meta-compensatory reward

Eρ(C|ρ→ρ)−Eρ(C|ρ→ρ′).

This prevents it from deriving any advantage to C via changes to ρ. Or, more precisely, it prevents it from deriving any expected advantage, that it can figure out using its current ρ.

Note that it doesn't prevent improvements to ρ in general, it just remo

...
Lagrangian duality for constraints on expectations
by jessicata — 2016-05-04T04:37:28.000Z

Summary: It's possible to set up an zero-sum game between two agents so that, in any Nash equilibrium, one agent picks a policy to optimize a particular objective subject to some constraints on expected features of the state resulting from the policy. This seems potentially useful for getting an approximate agent to maximize some objective subject to constraints.

Consider the following optimization problem:

Maximize f(π) subject to E[a(X)|do(π)]⪰b.

where π is a policy, f maps policies to reals, X∈X is a random
variable representing the state of the world, a∈X→Rn is a feature vector,
and b∈Rn is a vector. This objective instructs the agent to maximize some objective
f(π) while ensuring that random variables a1(X),...,an(X) have expectations at least b1,...,bn respectively.

How might we design a agent that solves the constrained optimization problem, if all we have available are agents
that optimize unconstrained objectives? Consider a game between two agents, an actor and a critic:

The critic picks Lagrange multipliers λ∈Rn,λ⪰0.

The actor picks a policy π, which results in world state X.

The actor gets utility f(π)+λ⋅(a(X)−b). The critic gets the negation of this utility.

Let's assume the critic and actor pick their λ and π simultaneously, and attempt to find the Nash equilibrium.
Since the critic has an infinite action space, there is not guaranteed to be a Nash equilibrium.

To make the problem easier, let's assume that the set of allowed policies is
convex, and f is concave (that is, for any two policies π1 and π2,
and θ∈(0,1), then if π3 is the policy formed by first picking
π1 with probability θ or π2 with probability 1−θ and
then running that policy, f(π3)≥θf(π1)+(1−θ)f(π2)).

We will prove:

Theorem: in any Nash equilibrium, the actor's policy solves the original optimization problem.

This is due to Lagrangian duality; the critic is picking Lagrange multipliers.
It's possible that a setup like this results in reasonable behavior if we use approximate agents instead of
optimal agents (and perhaps place some high absolute limit on λ), which would be the main
practical application of this idea, though I'm not sure if it does.

Proof of theorem

The actor's expected utility is gλ(π):=f(π)+λ⋅(E[a(X)|do(π)]−b).

With the assumptions above, we need only consider pure Nash equilibria.
The critic's utility is linear in λ and the actor's utility
is concave in π; th

...
Meetup : Washington, D.C.: Survivorship Bias
by RobinZ — 2016-05-04T03:28:11.043Z

Discussion article for the meetup : Washington, D.C.: Survivorship Bias

WHEN: 08 May 2016 03:30:00PM (-0400)

WHERE: Reynolds Center

We will be meeting in the courtyard to discuss survivorship bias - statistical biases that come into play when someone tries to understand a whole group by only looking at the examples that made it through a winnowing process.

As always, side conversations are allowed and encouraged.

Upcoming meetups:

May 15: Fun & Games
May 22: Typical Mind Fallacy
May 29: meeting canceled for Memorial Day
Jun. 5: Intro to Rationality

Discussion article for the meetup : Washington, D.C.: Survivorship Bias
Meetup : May Rationality Dojo, Melbourne: Stimulating Discussions
by Chriswaterguy — 2016-05-02T23:31:57.501Z

Discussion article for the meetup : May Rationality Dojo, Melbourne: Stimulating Discussions

WHEN:
07 May 2016 03:00:00PM (+1000)

WHERE:
Ross House Association 247-251 Flinders Ln, Melbourne CBD

[3:00pm arrival and, 3:30pm start of scheduled activities.]

For our May Dojo we have a choice of discussion groups:

Ways of knowing. How do we think about sense, reason, authority and intuition.

The hot cold empathy gap. It can be surprisingly hard to understand our future self.

Rules of discourse. Make your conversations more enlightening, and less like YouTube comment threads.

This is followed by Debugging your problems. Get help troubleshooting your goals, help someone else, or (if you prefer) work on something by yourself.

The Dojo will run until about 6pm, after which you are welcome to join us for dinner.

WHERE: Ross House, 247 Flinders Lane. Level 3 (Jenny Florence Room). If you have any trouble finding the venue or getting in, text or call Chris on 0439 471 632. For greatest convenience and prompt service, please arrive before 3:30pm.

ABOUT: The Less Wrong Melbourne Rationality Dojos are self-improvement sessions for those committed to the art of rationality and personal growth. (They are currently being held on the first Saturday of each month, so you may wish to add that as a recurring event in your calendar.)

Discussion article for the meetup : May Rationality Dojo, Melbourne: Stimulating Discussions
Paid research assistant position focusing on artificial intelligence and existential risk
by crmflynn — 2016-05-02T18:27:01.969Z

Yale Assistant Professor of Political Science Allan Dafoe is seeking Research Assistants for a project on the political dimensions of the existential risks posed by advanced artificial intelligence. The project will involve exploring issues related to grand strategy and international politics, reviewing possibilities for social scientific research in this area, and institution building. Familiarity with international relations, existential risk, Effective Altruism, and/or artificial intelligence are a plus but not necessary. The project is done in collaboration with the Future of Humanity Institute, located in the Faculty of Philosophy at the University of Oxford. There are additional career opportunities in this area, including in the coming academic year and in the future at Yale, Oxford, and elsewhere. If interested in the position, please email allan.dafoe@yale.edu with a copy of your CV, a writing sample, an unofficial copy of your transcript, and a short (200-500 word) statement of interest. Work can be done remotely, though being located in New Haven, CT or Oxford, UK is a plus.
My Kind of Moral Responsibility
by Gram_Stone — 2016-05-02T05:54:58.801Z

The following is an excerpt of an exchange between Julia Galef and Massimo Pigliucci, from the transcript for Rationally Speaking Podcast episode 132:

Massimo: [cultivating virtue and 'doing good' locally 'does more good' than directly eradicating malaria]

Julia: [T]here's lower hanging fruit [in the developed world than there is in the developing world]. By many order of magnitude, there's lower hanging fruit in terms of being able to reduce poverty or disease or suffering in some parts of the world than other parts of the world. In the West, we've picked a lot of the low hanging fruit, and by any sort of reasonable calculation, it takes much more money to reduce poverty in the West -- because we're sort of out in the tail end of having reduced poverty -- than it does to bring someone out of poverty in the developing world.

Massimo: That kind of reasoning brings you quickly to the idea that everybody here is being a really really bad person because they spent money for coming here to NECSS listening to us instead of saving children on the other side of the world. I resist that kind of logic.

Massimo (to the audience): I don't think you guys are that bad! You see what I mean?

I see a lot of people, including bullet-biters, who feel a lot of internal tension, and even guilt, because of this apparent paradox.

Utilitarians usually stop at the question, "Are the outcomes different?"

Clearly, they aren't. But people still feel tension, so it must not be enough to believe that a world where some people are alive is better than a world where those very people are dead. The confusion has not evaporated in a puff of smoke, as we should expect.

After all, imagine a different gedanken where a virtue ethicist and a utilitarian each stand in front of a user interface, with each interface bearing only one shiny red button. Omega tells each, "If you press this button, then you will prevent one death. If you do not press this button, then you will not prevent one death."

There would be no disagreement. Both of them would press their buttons without a moment of hesitation.

So, in a certain sense, it's not only a question of which outcome is better. The repugnant part of the conclusion is the implication for our intuitions about moral responsibility. It's intuitive that you should save ten lives instead of one, but it's counterintuitive that the one who permits ...
Meetup : Less Wrong NH Meetup
by Tuk — 2016-05-02T03:58:37.799Z

Discussion article for the meetup : Less Wrong NH Meetup

WHEN:
10 May 2016 07:00:00PM (-0400)

WHERE:
269 Pearl St Manchester NH 03104

The eighteenth NH meet-up is Tuesday, 5/10, in Manchester, NH at 7 pm at a private residence. Light refreshments will be provided.

Have you read Rationality: from AI to Zombies, or any of the Sequences on Less Wrong? Maybe you're just a fan of Harry Potter and the Methods of Rationality. Come hang out with us and discuss optimization of whatever it is you want to optimize.

Agenda: Mental Models

You may want to bring a notebook.

https://www.facebook.com/events/768837069917864/

https://www.facebook.com/groups/695201067251306/

Discussion article for the meetup : Less Wrong NH Meetup
Meetup : San Francisco Meetup: Projects
by rocurley — 2016-05-02T02:54:24.058Z

Discussion article for the meetup : San Francisco Meetup: Projects

WHEN:
02 May 2016 06:15:00PM (-0700)

WHERE:
1597 Howard St. San Francisco, CA

We'll be meeting to work on projects. Bring something to work on, come up with something there, or help other people out! As always, I can be reached at 301-458-0764 if you need to be let in.

Discussion article for the meetup : San Francisco Meetup: Projects
Open Thread May 2 - May 8, 2016
by Elo — 2016-05-02T02:43:34.529Z

If it's worth saying, but not worth its own post (even in Discussion), then it goes here.

Notes for future OT posters:

1. Please add the 'open_thread' tag.

2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)

3. Open Threads should be posted in Discussion, and not Main.

4. Open Threads should start on Monday, and end on Sunday.
A Second Year of Spaced Repetition Software in the Classroom
by tanagrabeast — 2016-05-01T22:14:08.092Z

This is a follow-up to last year's report. Here, I will talk about my successes and failures using Spaced Repetition Software (SRS) in the classroom for a second year. The year's not over yet, but I have reasons for reporting early that should become clear in a subsequent post. A third post will then follow, and together these will constitute a small sequence exploring classroom SRS and the adjacent ideas that bubble up when I think deeply about teaching.

Summary

I experienced net negative progress this year in my efforts to improve classroom instruction via spaced repetition software. While this is mostly attributable to shifts in my personal priorities, I have also identified a number of additional failure modes for classroom SRS, as well as additional shortcomings of Anki for this use case. My experiences also showcase some fundamental challenges to teaching-in-general that SRS depressingly spotlights without being any less susceptible to. Regardless, I am more bullish than ever about the potential for classroom SRS, and will lay out a detailed vision for what it can be in the next post.

Prologue

You might want to at least skim last year's report, even if you've read it before. My job description is unchanged, and my core approach to SRS stayed the same. I will not be going into those details today.

Also, you'll probably find much of this report disappointing for reasons that have nothing to do with SRS itself and everything to do with me—reasons I will now invite you to understand with a philosophical introspective. If gazing into my navel really isn't your thing, just skip the next section.

Autopilot years

The commonly appreciated upside of being a public schoolteacher in the United States is having more days off and greater job security than most workers. Less commonly appreciated is that, once you've got things figured out, you probably don't need to put in tons of extra hours. You can mostly leave your work at work.

So what do you do with the time?

Good teachers usually direct a lot of that back to the students. They coach a sport, or sponsor a club. They leave detailed feedback on essays. They tutor.

I've done most of those things, but my personal preference is to reinvest in my core classroom offerings by improving my content and tooling. On top of some smaller tweaks, I usually pick one big project every year that will add a new layer of awesome...
May 2016 Media Thread
by ArisKatsaris — 2016-05-01T21:27:43.055Z

This is the monthly thread for posting media of various types that you've found that you enjoy. Post what you're reading, listening to, watching, and your opinion of it. Post recommendations to blogs. Post whatever media you feel like discussing! To see previous recommendations, check out the older threads.

Rules:

Please avoid downvoting recommendations just because you don't personally like the recommended material; remember that liking is a two-place word. If you can point out a specific flaw in a person's recommendation, consider posting a comment to that effect.

If you want to post something that (you know) has been recommended before, but have another recommendation to add, please link to the original, so that the reader has both recommendations.

Please post only under one of the already created subthreads, and never directly under the parent media thread.

Use the "Other Media" thread if you believe the piece of media you want to discuss doesn't fit under any of the established categories.

Use the "Meta" thread if you want to discuss about the monthly media thread itself (e.g. to propose adding/removing/splitting/merging subthreads, or to discuss the type of content properly belonging to each subthread) or for any other question or issue you may have about the thread or the rules.
Hedge drift and advanced motte-and-bailey
by Stefan_Schubert — 2016-05-01T14:45:08.023Z

Motte and bailey is a technique by which one protects an interesting but hard-to-defend view by making it similar to a less interesting but more defensible position. Whenever the more interesting position - the bailey - is attacked - one retreats to the more defensible one - the motte -, but when the attackers are gone, one expands again to the bailey.

In that case, one and the same person switches between two interpretations of the original claim. Here, I rather want to focus on situations where different people make different interpretations of the original claim. The originator of the claim adds a number of caveats and hedges to their claim, which makes it more defensible, but less striking and sometimes also less interesting.* When others refer to the same claim, the caveats and hedges gradually disappear, however, making it more and more bailey-like.

A salient example of this is that scientific claims (particularly in messy fields like psychology and economics) often come with a number of caveats and hedges, which tend to get lost when re-told. This is especially so when media writes about these claims, but even other scientists often fail to properly transmit all the hedges and caveats that come with them.

Since this happens over and over again, people probably do expect their hedges to drift to some extent. Indeed, it would not surprise me if some people actually want hedge drift to occur. Such a strategy effectively amounts to a more effective, because less observable, version of the motte-and-bailey-strategy. Rather than switching back and forth between the motte and the bailey - something which is at least moderately observable, and also usually relies on some amount of vagueness, which is undesirable - you let others spread the bailey version of your claim, whilst you sit safe in the motte. This way, you get what you want - the spread of the bailey version - in a much safer way.

Even when people don't use this strategy intentionally, you could argue that they should expect hedge drift, and that omitting to take action against it is, if not outright intellectually dishonest, then at least approaching that. This argument would rest on the consequentialist notion that if you have strong reasons to believe that some negative event will occur, and you could prevent it from happening by fairly simple means, then you have an obligation to do so. I cert...