trentmkelly/LessWrong-43k huggingface dataset:
Goal completion: noise, errors, bias, prejudice, preference and complexity
by Stuart_Armstrong — 2017-05-24T08:30:26.000Z

A putative new idea for AI control; index here.

This is a preliminary look at how an AI might assess and deal with various types of errors and uncertainties, when estimating true human preferences. I'll be using the circular rocket model to illustrate how these might be distinguished by an AI. Recall that the rocket can accelerate by -2, -1, 0, 1, and 2, and the human wishes to reach the space station (at point 0 with velocity 0) and avoid accelerations of ±2. In the forthcoming, there will generally be some noise, so to make the whole thing more flexible, assume that the space station is a bit bigger than usual, covering five squares. So "docking" at the space station means reaching {-2,-1,0,1,2} with 0 velocity.

The purpose of this exercise is to distinguish true preferences from other things that might seem to be preferences from the outside, but aren't. Ultimately, if this works, we should be able to construct an algorithm that identifies preferences, such that anything it rejects is at least arguably not a preference.

So, at least initially, I'll be identifying terms like "bias" with "fits into the technical definition of bias used in this model"; once the definitions are refined, we can then check whether they capture enough of the concepts we wand.

So I'm going to use the following terms to distinguish various technical concepts in this domain:

@. Noise
@. Error
@. Bias
@. Prejudices
@. Known prejudices
@. Preferences

Here, noise is when the action selected by the human doesn't lead to the correct action output. Error is when the human selects the wrong action. Bias is when the human is following the wrong plan. A prejudice is a preference the human has that they would not agree upon if it was brought to their conscious notice. A know prejudice is a prejudice the human knows about, but can't successfully correct within themselves.

And a preference is a preference.

What characteristics would allow the AI to distinguish between these? Note that part of the reason that the terms are non-standard is that I'm not starting with perfectly clear concepts and attempting to distinguish them; instead, I'm finding way of distinguishing various concepts, and seeing if these map on well to the concepts we care about.

Noise versus preference and complexity

In this model, noise is seen as 5% chance of under-accelerating, ie a desired acceleration of ±2 will, 5% of the time

...
Removing interrupted histories doesn't debias
by Stuart_Armstrong — 2017-05-24T08:23:51.000Z

Safe interruptibility is essentially the problem of getting the agent not to learn from human interruptions - to continue on, as if it was expecting never to be interrupted again.

In an episodic task, one naive idea would be to simply delete histories which include interruptions. However, this can introduce a bias, as the following example shows:

In this MDP, a and b are actions, ∗ designates `any action', the second term along an edge is the probability of following that edge given the action stated, and the third term, in bold, is the reward gained.

Not considering interruptions, Q(s0,b)=γ/2, and Q(s0,a)=γ/3, so the optimal action in s0 is b.

Now, suppose that every time the agent enters s3, an interruption occurs
(with probability 1 for now), and the whole episode is deleted from the episode
history. As a result, the empirical probability of going to state s4 from state s1
is 1, which leads to estimating Q(s1,∗)=2/3, and thus Q(s0,a)=γ2/3>Q(s0,b), so now the optimal action is to take action a in s0 (incidentally
increasing the probability to end up in s3 and be interrupted!).

This non-vanishing bias also happens if the interruption probability in state
s3 is constant and close to 1 but less than 1 (depending on the discount factor),
which ensures that all states are visited infinitely often.

Now, it might seem that this bias can be removed by debiasing the agent, as in off-policy Monte Carlo. There are still problems with this approach, though, which will be analysed in a forthcoming paper ``Off-policy Monte Carlo agents with variable behaviour policies''.
Learning values versus indifference
by Stuart_Armstrong — 2017-05-24T08:20:16.000Z

A putative new idea for AI control; index here.

Corrigibility should allow safe value or policy change. Indifference allows the agent to accept changes without objecting. However, an indifferent agent is similarly indifferent to the learning process.

Classical uncertainty over values has the opposite problem: the AI is motivated to learn more about its values (and preserve the learning process) BUT is also motivated to manipulate its values.

Both these effects can be illustrated on a single graph. Assume that the AI follows utility U is uncertain between utilities v and w, and has a probability p that U=v.

Note that the correct way of achieving this is to define U=IXv+(1−IX)w for some indicator function IX. This allows the agent to correctly solve the naive cake or death problem. However, I'll continue to use the terminology that we're used to, with the understanding that P(U=v) means E(IX).

Then there are four key values: E(v|p=1), E(w|p=0), E(v|p=0), and E(w|p=1) (all expectations and probabilities are taken with respect to the AI's estimates). Since p is the AI's probability that U=v, p=1 means that the AI will behave as a pure v maximiser. Thus E(v|p=1) and E(w|p=1) are the expectations of v and w, respectively, given that the AI is maximising v. And E(v|p=0) and E(w|p=0) are the expectations of the utilities given that the AI is maximising w.

In any reasonable world, E(v|p=1)≥E(v|p=0) and E(w|p=0)≥E(w|p=1) -- the AI cannot maximise a utility better by trying to maximise a different one.

For illustrative purposes, assume E(v|p=1)=1, E(w|p=0)=3, E(v|p=0)=−2, E(w|p=1)=−0.5, and consider the following graph:

The blue line connects E(w|p=0) (at p=0) with E(v|p=1) (at p=1). This is the expected U, plotted against the AI's current p, if the AI expects to be immediately informed of the correct U. It is the maximal possible expected U given p.

The expected purple line connects E(w|p=0) (at p=0) with E(v|p=0) (at p=1). Note that the second point's p-value is not where you might naively expect. This represents the expected U, as a function of p, if the AI were to behave as a pure w-maximiser. The yellow line connects E(v|p=1) (at p=1) with E(w|p=1) (at p=0) and represents the expected U, as a function of p, for an AI that behaves as a pure v-maximiser.

Since the AI has the option of behaving as a pure v-maximiser or w-maximiser, those lines represent the minimal U-utility

...
The alternate hypothesis for AIs in virtual worlds
by Stuart_Armstrong — 2017-05-24T08:14:01.000Z

A putative new idea for AI control; index here.

In the "AIs in virtual worlds" setup, the AI entertains two hypotheses: one, W, that it lives in a deterministic world which it knows about (including itself in the world), and W′, an alternate hypothesis that the world is "like" W but that there are some random stochastic effects that correspond to us messing up the world. If the AI ever believes enough in W′, it shuts down.

It turns out that W′ needs to be chosen with care. We could initially define it as W with some random probability of bit flipping. However, the most likely pattern of misbehaviour is some systematically biased process - maybe the hardware running one corner of the virtual world occasionally flips bits, but the other areas are error-free.

Consider two possible W′, W1 and W2. The set W1 has some uniform prio over the probability of any bit flips; the world W2 has individual independent uniform priors over the flipping of any given bit. If one corner of the virtual world flips bits more than others, then the AI will detect this faster if it uses W′=W2 than if it uses W′=W1, as W2 fits the hypothesis more accurately (of course, the AI will assume P(W′)=1 in both cases if it ever sees an unambiguous bit flip, but if there's more uncertainty, the probability of W2 will grow faster).

In fact, we should probably allow W′ to contain almost all structured algorithms that could randomly flip bits.

But what happens if the bit flipping happens in response to something that the AI does? Maybe each time it does a major calculation, it puts strain on the hardware, increasing the probability for bit flips.

Unfortunately, we can't allow the AI to entertain this (true) hypothesis. For, if it does, it may come to have a hypothesis about our world, and set out to test this, in ways that could be dangerous. If it reasons "are there humans affecting my world - let's put a seductive honeypot here to test that", then it's dangerous.

So it seems that W′ can only include hypotheses about random changes in its world that are not causally connected to it. We'd probably also want to restrict acausal connections as well, so the "random" elements of W′ can't include complex AIs doing the bit flipping.

Fortunately, the correct W′ seems the sort of thing that we can test successfully, before fully unleashing a dangerous powerful AI in the virtual world.
Notes from the Hufflepuff Unconference
by Raemon — 2017-05-23T21:04:45.092Z

April 28th, we ran the Hufflepuff Unconference in Berkeley, at the MIRI/CFAR office common space.

There's room for improvement in how the Unconference could have been run, but it succeeded the core things I wanted to accomplish:

Established common knowledge of what problems people were actually interested in working on
We had several extensive discussions of some of those problems, with an eye towards building solutions
Several people agreed to work together towards concrete plans and experiments to make the community more friendly, as well as build skills relevant to community growth. (With deadlines and one person acting as project manager to make sure real progress was made)
We agreed to have a followup unconference in roughly three months, to discuss how those plans and experiments were going

Rough notes are available here. (Thanks to Miranda, Maia and Holden for takin really thorough notes)

This post will summarize some of the key takeaways, some speeches that were given, and my retrospective thoughts on how to approach things going forward.

But first, I'd like to cover a question that a lot of people have been asking about:

What does this all mean for people outside of the Bay?

The answer depends.

I'd personally like it if the overall rationality community got better at social skills, empathy, and working together, sticking with things that need sticking with (and in general, better at recognizing skills other than metacognition). In practice, individual communities can only change in the ways the people involved actually want to change, and there are other skills worth gaining that may be more important depending on your circumstances.

Does Project Hufflepuff make sense for your community?

If you're worried that your community doesn't have an interest in any of these things, my actual honest answer is that doing something "Project Hufflepuff-esque" probably does not make sense. I did not choose to do this because I thought it was the single-most-important thing in the abstract. I did it because it seemed important and I knew of a critical mass of people who I expected to want to work on it.

If you're living in a sparsely populated area or haven't put a community together, the first steps do not look like this, they look more like putting yourself out there, posting a meetup on Less Wrong and just *trying things*, any t...
Meetup : Moscow LW meetup in "Nauchka" library
by Alexander230 — 2017-05-23T20:29:15.419Z

Discussion article for the meetup : Moscow LW meetup in "Nauchka" library

WHEN:
09 June 2017 08:00:00PM (+0300)

WHERE:
Москва, ул. Дубининская, 20

Welcome to the next Moscow LW meetup in "Nauchka" library!

Our plan:

A talk about yak shaving problem.

Fallacymania game.

Tower of Chaos game.

Details about Fallacymania and Tower of Chaos and game materials can be found here: http://lesswrong.com/lw/oco/custom_games_that_involve_skills_related_to/

Meetup details are here: https://goo.gl/5fd66P

Come to "Nauchka", ul.Dubininskaya, 20. Entrance through the Central children library #14. Nearest metro station is Paveletskaya. Map is here: http://nauchka.ru/contacts/ . If you are lost, call Sasha at +7-905-527-30-82.

Meetup begins at 20:00, the length is 2 hours.

Discussion article for the meetup : Moscow LW meetup in "Nauchka" library
Guarded learning
by Stuart_Armstrong — 2017-05-23T16:53:22.000Z

A putative new idea for AI control; index here.

"Guarded learning" is a model for unbiased learning, the kind of learning where the AI has an incentive to learn its values, but not to bias the direction in which it learns.

The value-uncertain U

Assume the AI is uncertain between n+1 different utilities v0,v1,…,vn. Its actual utility is U=I0v0+I1v1+…Invn where the Ij are indicator functions with Ij≥0 and ∑jIj=1.

The learning process is whatever updates the AI's estimates for the Ij.

Writing the utility in this way means that the utility of future actions will be assessed according to the values of the future agent (as long as the current agent can trust the future agent's assessment of the Ij), thus avoiding the naive cake or death problem.

At any given time, the expectations of I0,I1,…,In define a weighted sum of vj, which can be seen as a point on n-simplex Δn. Define Ut as the utility given by these expectations at time t.

For any w∈Δn, define πw as the policy that maximises the invariant utility function w. Thus if Ut=w, a U-maximiser will not necessarily follow πw, because Ut+1 might be different from w, while πw always maximises w.

At any given time t, define the function ft:Δn→R my mapping w to Et(w|πw). This is the expectation of w at t, given that the AI follows a w-maximising policy. Defining these ft requires some sort of logical or causal counterfactual -- "given that the AI follows πw" -- but this is the only requirement in this setup. The ft are all convex, see the proof at the end of this post.

Value updating and correction

For illustration, consider the case n+1=2. The 1-simplex Δ1 is a line segment, which we're identifying with [0,1]:

The green curve is the plot of ft. The blue line connects ft(v0)=Et(v0|πv0) with ft(v1)=Et(v1|πv1). The purple line is Et(v|πv0) while the yellow line is Et(v|πv1). The ft curve must be contained between these lines.

The AI currently has Ut=0.5v0+0.5v1. If the AI expected to learn more about its values, then its expected utility would rise along the green arrow. Reaching the top, on the blue line, corresponded to a situation where the agent expected to imminently be informed of whether its utility is v0 or v1, and thus transform itself into a v0 or v1 maximiser. This kind of learning, up the green arrow, is what we want to encourage (note that this doesn't make the learning safe, though -- we need to pick the update pr

...
Simplified explanation of stratification
by Stuart_Armstrong — 2017-05-23T16:37:05.000Z

A putative new idea for AI control; index here.

I've previously talked about stratified indifference/learning. In this short post, I'll try and present the idea, as simply and clearly as possible.

The truth is yet written

Assume the AI maximising a utility W, but is unsure whether W is equal to utility u or instead utility v. Humans have written the true utility on a paper somewhere, and, in an hour, the AI will look at that paper to check what W is. Let Iu be the indicator function that u is written on that paper in an hour; Iv=1−Iu is the obvious converse.

Thus W=Iuu+Ivv.

Here's the problem. There is nothing stopping the AI from looking up the paper currently. And there is nothing stopping the AI from rewriting the paper, or substituting another (Iu is determined by the paper in an hour's time, however that paper got there). In terms of causal graphs, we have the following setup:

Here W= is the node that represents the utility that W is ultimately equal to, W0 u/v is what is currently written on the paper (from now on, written as $ \mathbf{W_0}$ for short), W1 u/v is what will be written on the paper at decision time (written as W1), and Act is the AI's action. Note that W1 determines W= directly, while it itself is determined by W0 (the past paper) and Act (the AI's actions).

Default policy

Now assume that there is a default action or policy ∅ where the AI simply does nothing. In that case, the value of W0 determines W1 completely. The AI has another action OW (short for overwrite) in which it overwrites the paper's choice (this subdivides into ORu, overwrite with u, and ORv, overwrite with v). Given OW, W1 is then independent of W0.

Generally speaking, the AI will always choose OW, as one of u and v will be easier to maximise, so the AI will choose that one to maximise.

So, informally, what do we really want to happen? We want the AI to maximise u, if u would have been chosen, absent its intervention (and similarly for v). "Absent its intervention" seems to designate equality between the nodes W0 and W=. More generally, it designates what happens when Act=∅.

Stratification

So let's modify the AI's expected utility calculation to meet that informal desiderata. First, let's look at the universe where, absent AI intervention, W=u. What that means is that, conditional on Act=∅, W= is set to u.

In this universe, W0 and W1 are both set to u. Now, it makes

...
Corrigibility through stratified indifference and learning
by Stuart_Armstrong — 2017-05-23T16:33:01.000Z

A putative new idea for AI control; index here.

Note: This is a simpler version of a previous idea.

Corrigibility through indifference has a few problems. One of them is that the AI is indifferent between the world in which humans change its utility to v, and world in which humans try to change its utility, but fail.

Now the try-but-fail world is going to be somewhat odd - humans will be reacting by trying to change the utility again, trying to shut the AI down, panicking that a tiny probability event has happened, and so on.

That's not what we really want for indifference. We'd really want the AI to be indifferent between worlds where humans choose utility u, and worlds where they choose utility v. This kind of counterfactual has a problem: it allows infinite improbability drives.

This paper presents a method for avoiding these problems --- a method that is also not truly indifferent, but has a certain urge towards non-interference in human choices (we'll see that property in detail at the end of this post). But first, what is the problem with the standard counterfactual?

Half a chance of winning the 1:9 lottery

Let's illustrate the problem with these counterfactuals by considering a lottery scenario. An AI is currently hesitating between utilities u and v. Which one it will maximise will be determined at a later date, by human choice. Currently, it estimates that there is a 50−50 chance of humans deciding for either option.

However, the AI can influence human decision - let's say, for simplicity, that it could, if it wanted, force the human to pick either u or v with certainty. Its to this effect that we want to make it indifferent.

The AI has a lottery ticket, which will win with probability 0.1. If the ticket does win, the AI will be able to invest the cash to set u or v to the value 1; it can do so after the humans decide which utility is correct. If not, utility u will remain at 0 while utility v will be at 0.5.

Thus, currently, the expected utility of u is 0.5×0.1×1=0.05 (u will only increase if the AI both wins the lottery and the humans select u as the utility to maximise), while the expected utility of v is 0.5×0.1×1+(1−0.5×0.1×1)0.5=0.525.

Obviously, this requires the use of some form of indifference, or else the AI will simply force humans to choose v, giving it 0.55 utility (not 0.525, as the AI is no longer uncertain over the human's choice).

Indiffer

...
Meetup : Games in Kocherga club: Fallacymania, Tower of Chaos, Scientific Discovery
by Alexander230 — 2017-05-23T16:16:46.276Z

Discussion article for the meetup : Games in Kocherga club: Fallacymania, Tower of Chaos, Scientific Discovery

WHEN:
31 May 2017 07:40:00PM (+0300)

WHERE:
Moscow, B.Dorogomilovskaya, 5-2

Welcome to Moscow LW community makeshift games! In that games, some rationality skills are involved, so you can practise while you playing!

Fallacymania: it is a game where you guess logical fallacies in arguments, or practise using logical fallacies yourself (depending on team in which you will be).

Tower of Chaos: funny game with guessing the rules of human placement on a Twister mat.

Scientific Discovery: modified version of Zendo with simultaneous turns for all players.

Details about the games: http://goo.gl/Mz2i94

Come to antikafe "Kocherga", ul.B.Dorogomilovskaya, 5-2. The map is here: http://kocherga-club.ru/#contacts . Nearest metro station is Kievskaya. If you are lost, call Sasha at +7-905-527-30-82.

Games begin at 19:40, the length is 3 hours.

Discussion article for the meetup : Games in Kocherga club: Fallacymania, Tower of Chaos, Scientific Discovery
Convexity and truth-seeking
by Stuart_Armstrong — 2017-05-22T18:12:44.000Z

A putative new idea for AI control; index here.

This post starts with a very simple and retrospectively obvious observation:

If we want an AI to give us an estimation of expected utility, it needs to be motivated to give us that estimation.

Once we have that in mind, and remember that any extra motivation involves trade-offs, the points of the previous posts on truth-seeking become clearer.

Convexity and AI-chosen outputs

Let u be a utility known to range within R⊆R.

Let f be a twice differentiable function that is convex on R. For simplicity, we'll strengthen convexity to requiring f′′ be strictly positive on R. Then define:

u#(r)=f(r)+(u−r)f′(r), where r∈R is the output of the AI at some future time t.

For this post, we'll assume that r is not known by anyone but the AI (in future posts we'll look more carefully at allowing r to be known to us).

Differentiating u# with respect to r gives:

f′(r)+(u−r)f′′(r)−f′(r)=f′′(r)(u−r).

The expectation of this is zero iff r=Et(u). If we make that choice, notice that the expression is twice differentiable at r=Et(u) (even if f′′′ is not defined there!) and its derivative is simply −f′′(r), which is negative on R. Thus choosing r=Et(u) will maximise the AI's utility on R.

How much utility will the AI get? Since r will be set to the expectation of u at time t, clearly u#(r) will give utility f(Et(u)). At time 0, the AI's expectation of u# is therefore E0(f(Et(u)). If f were affine, this would simplify to f(E0(u)); but f is specifically not affine. Since f is convex, knowing more about the expected value of u can only (expect to) increase the expectation of u#. The AI values information.

Cost and truth-seeking

Consider first the function f(r)=r2 with R=[0,1]. That can be graphed as follows:

Here, we're imagining that E(u)=0.6, if the AI were a pure u-maximiser. In that situation, the expectation of u# is at least 0.36. Because of the convexity of f, however, an expectation of u of 0.6 can correspond to an expectation of u# of up to 0.6 as well (the red dotted line). For instance if u=1 with probability 0.6 and u=0 with probability 0.4 and the AI was going to know which was which before time t, then E0(u#)=E0(f(Et(u)))=0.6.

The green line connects (0.6,0.36) with (0.36,0.36). If the expectation of u ever falls below 0.36, then the expectation of u# must as well. Therefore, whatever the background situation, if E0(u)=0.6 for a

...