trentmkelly/LessWrong-43k huggingface dataset:
Reward/value learning for reinforcement learning
by Stuart_Armstrong — 2017-06-02T16:34:41.000Z

A putative new idea for AI control; index here.

Along with Jan Leike and Laurent Orseau, I've been working to formalise many of the issues with AIs learning human values.

I'll be presenting part of this at NIPS and the whole of it at some later conference. Therefore it seems best to formulate the whole problem in the reinforcement learning formalism. The results can generally be easily reformulated for general systems (including expected utility).

POMDP

A partially observable Markov decision process without reward function (POMDP\R), μ=(S,A,O,T,O,T0)
consists of:

a finite set of states S,

a finite set of actions A,

a finite set of observations O,

a transition probability distribution T:S×A→ΔS,

a probability distribution T0∈ΔS over the initial state s0.

an observation probability distribution O:S→ΔO.

The agent interacts with the environment in cycles:
in time step t, the environment is in state st−1∈S and
the agent chooses an action at∈A.
Subsequently
the environment transitions to a new state st∈S drawn from the distribution T(st∣st−1,at) and
the agent then receives an observation ot∈O drawn from the distribution O(ot∣st).
The underlying states st−1 and st are not directly observed by the agent.

An observed history ht=a1o1a2o2…atot is a sequence of actions and observations.
We denote the set of all observed histories of length t with Ht:=(A×O)t.

For a given horizon m, call Hm the set of full histories; then H<m=⋃t<mHt is the set of partial histories.
For t′>t, let at:t′ be the sequence of actions atat+1…at′,
let ot:t′ be the sequence of observations otot+1…ot′, and
let st:t′ the sequence of states stst+1…st′.

The set Π is the set of policies, functions π:(A×O)∗→ΔA mapping histories to probability distributions over actions.
Given a policy π and environment μ,
we get a probability distribution over histories:

μ(a1o1…atot∣π):=∑s0:t∈StT0(s0)∏tk=1O(ok∣sk)T(sk∣sk−1,ak)π(ak∣a1o1…ak−1ok).

The expectation with respect to the distributions μ and π is denoted Eπμ.

Rewards

Rewards in this case can be seen as functions R:O→R from observations to real numbers.

The agent's goal is to maximize total reward ∑mt=1R(ot) up to the horizon m. We assume that S and A are known to the agent.
The reward function R is unknown, but
there is a finite set of candidate reward functions, R.
The agent has to learn a reward function in the process of interacting with the environme

...
The best value indifference method (so far)
by Stuart_Armstrong — 2017-06-02T16:33:22.000Z

A putative new idea for AI control; index here.

When dealing with the problem of bias, I stumbled upon what I believe is the best way of getting value indifference, one that solves almost all of the problems with the previous methods.

The problems with the old setups

My initial setup has a counterfactual which I'll call a causal counterfactual, following Jessica's terminology. Her whole post is a good way of illustrating what is happening. A button is due to be pressed or not pressed, by a human; this pressing will change the agent's utility function.

There are low-probability quantum events that will artificially depress the button or prevent it from being pressed. The agent only cares about those low-probability events, which allows us to setup the formalism without worrying about changes in the probability of the human pressing the button. Then the events A="the human presses the button" and A'="the quantum event presses the button" were taken to be interchangeable, as were B="the human doesn't press the button" and B'="the quantum event prevents the button from being pressed".

The main problem is that these events are not really the same. So the agent will behave as if certain events are impossible (such as the human genuinely pressing the button), and is likely to face human resistance if the human tries to press the button and the button doesn't press due to quantum events, or vice versa. This resistance will either lower the expected reward of the agent, or make them manipulate the human actions.

Benya has sketched out a problem with another approach to counterfactuals, which we can call "evidential counterfactuals". Here we're trying to fiddle with utilities to enforce that the agent treats the probability of A and B as being fixed (no A' and B' here).

The problem with that is that the agent can change the probabilities conditional on some rare event, but then behave as if the probabilities were still unchanged same.

For instance, the agent could ensure that A only happens if it wins the lottery. And then it would behave, in effect, as if it had a large chance of winning the lottery (equal to the initial chance of A happening, before the agent took any action). That problem is illustrated here.

A final problem was that neither method allowed the agent to learn usefully: the methods did not distinguish between the agent merely learning that humans intended/

...
How to judge moral learning failure
by Stuart_Armstrong — 2017-06-02T16:32:24.000Z

A putative new idea for AI control; index here.

I’m finding many different results, that show problems and biases with the reward learning process.

But there is a meta problem, which is answering the question: “If the AI gets it wrong, how bad is it?”. It’s clear that there are some outcomes which might be slightly suboptimal – like the complete extinction of all intelligent life across the entire reachable universe. But it’s not so clear what to do if the error is smaller.

For instance, suppose that there are two moral theories, M1 and M2. An AI following M1 would lead to outcome O1, a trillion trillion people leading superb lives. An AI following M2 would lead to outcome O2, a trillion people leading superbly superb lives.

If one of M1 or M2 was the better moral theory from the human perspective, how would we assess the consequences of the AI choosing the wrong one? One natural way of doing this is to use these moral theories to assess each other. How bad, from M1’s perspective, is O2 compared with O1? And vice versa for M2. Because of how value accumulates on different theories, it’s plausible that M1(O1) could be a million times better than M1(O2) (as O1 is finely adapted to M1’s constraints). And similarly, M2(O1) could be a million times worse than M2(O2).

So from that perspective, the cost of choosing the wrong moral theory is disastrous, by a factor of a million. But from my current perspective (and that of most humans), the costs do not seem so large; both O1 and O2 seem pretty neat. I certainly wouldn’t be willing to run a 99.9999% chance of human extinction, in exchange for the AI choosing the better moral theory.

Convincing humans what to value

At least part of this stems, I think, from the fact that humans can be convinced of many things. A superintelligent AI could convince us to value practically anything. Even if we restrict ourselves to “non-coercive” or “non-manipulative” convincing (and defining those terms is a large part of the challenge), there’s still a very wide space of possible future values. Even if we restrict ourselves massively – future values we might have, with no AI convincing at all, just my continuing to live our lives according to vagaries of fortune – that’s still a wide span.

So our values are underdetermined in important ways (personal example: I didn’t expect I’d become an effective altruist or an expected utility maximise

...
Counterfactuals on POMDP
by Stuart_Armstrong — 2017-06-02T16:30:47.000Z

A putative new idea for AI control; index here.

This is technical note explaining how to define counterfactuals on partially observable Markov decision processes (POMDP).

The POMDP formalism is explained here. This note will just sketch out how counterfactuals are defined; the full details will be in the final paper.

Taking another action

Suppose that an agent has been active for n turns on POMDP μ, and has seen history hn. Then suppose it wants to estimate the counterfactual of what would have happened if had done other actions after timestep t<n. So what is the counterfactual probability of history hm, t<m, which can be written as:

μ(hm∣(hn)c,ht).

This rather clunky notation (let me know if there's a better way of writing this) is trying to estimate the probability of hm, given that hn is the counterfactual history, and both hn and hm start with ht before diverging.

It might seem surprising that no policy is mentioned in that expression - after all, the probability of a history is given by the environment μ and the agent's action choices. But histories like hm include action choices, so these don't need to be specified separately.

The first thing to notice is that μ and hn give a probability distribution over st, the (hidden) state at timestep t. And the value of st can change the subsequent probability of hm. So the counterfactual probability is defined as:

μ(hm∣(hn)c,ht)=∑s∈Sμ(st=s∣hn)μ(hm∣ht,st=s).

Counterfactual equivalence

Though the counterfactual probability is defined in terms of the states, the expression itself only involves histories.

Thus if μ and μ′ are two POMDPs with the same sets of actions and observations (but potentially different sets of states) we can say they are counterfactually equivalent if they generate the same counterfactual probabilities:

∀hm,hn,ht:μ(hm∣(hn)c,ht)=μ′(hm∣(hn)c,ht).

Consider the simple POMDP μ (actually, an MDP, since it's fully observable), defined as:

The reasons for the notation will be explained in a later post; but here, starting from a single state, the agent can take one of two actions, and each action has a 0.5 chance of ending up in one of two states.

Now consider the POMDP μ′, defined as:

Here there are one of two initial states, each equally likely, and then each action will lead with certainty to another state that comes from the initial hidden state and the action choice.

Note that there are four p

...
Uninfluenceable learning agents
by Stuart_Armstrong — 2017-06-02T16:30:08.000Z

A putative new idea for AI control; index here.

After explaining riggable learning processes, we can now define influenceable (and uninfluenceable) learning processes.

Recall that the (unriggable) influence problem is due to agents randomising their preferences, as a sort of artificial `learning' process, if the real learning process is slow or incomplete.

Suppose we had a learning process that it wasn't possible to influence. What would that resemble? It seems like it must be something where the outcome of the learning process depends only upon so outside fact about the universe, a fact the agent has no control over.

So with that in mind, define:

Definition: A learning process P on the POMDP μ is initial-state determined if there exists a function fP:S→ΔR such that P factors through knowledge of the initial state s0. In other words:

P(⋅∣hm)=∑s∈Sμ(s0=s∣hm)fP(s).

Thus uncertainty about the correct reward function comes only from uncertainty about the initial state s0.

This is a partial definition, but an incomplete one. To finalise it, we need the concept of counterfactually equivalent POMDPs:

Definition: A learning process P on μ is uninfluenceable if there exists a counterfactually equivalent μ′ such that P is initial-state determined on μ′.

Though the definition of unriggable and uninfluenceable seem quite different, they're actually quite closely related, as we'll see in a subsequent post. Uninfluenceable can be seen as `unriggable in all background info about the universe'. In old notation terms, rigging is explored in the sophisticated cake or death problem, (unibased) influence in the ultra-sophisticated version.

Example

Consider the environment μ presented here:

In this POMDP (actually MDP, since it's fully observed), the agent can wait for a human to confirm the correct reward function (action aw) or randomise its reward (action ar). After either actions, the agent gets equally likely feedback 0 or 1 (states swi and sri, 0≤i≤1).

We have two plausible learning processes: P, where the agent learns only from the human input, and P′, where the agent learns from either action. Technically:

P(Ri∣o0awowi)=P′(Ri∣o0awowi)=1,

P(Ri∣o0awowj)=0.5 for all 0≤i,j≤1,

P′(Ri∣o0awowi)=1,

with all other probabilities zero.

Now, μ is counterfactually equivalent to μ′′:

And on μ′′, P is clearly initial-state determined (with fP(si0)(Ri)=1), and is thus uninfluenceable on μ′′ and

...
Ontology, lost purposes, and instrumental goals
by Stuart_Armstrong — 2017-06-02T16:28:34.000Z

A putative new idea for AI control; index here.

An underdefined idea connected with the challenge of getting an AI to safely move a strawberry onto a plate.

Now, specifying something in the physical world like that is a great challenge; you have to define ontologies and similar. But imagine that the AI had a goal -- any goal -- and that it had to program a subagent to protect itself while it was accomplishing that goal.

Then the subagent will certainly be programmed with a firm grasp of the physical world, and some decent bridging laws should it have an ontology change (if, for instance, quantum mechanics turns out to be incomplete).

This is just an illustration of a general fact: even if its goal is not properly grounded, the instrumental goals will include strongly grounded goals, resilient to ontology change.

This feels related to the fact that even AI's that are given goals in badly programmed natural language concepts ("Make humans* happy*", with the asterix denoting the poor grounding) will still need well-grounded concepts for "human", just to function.

So, is there a way to exploit this instrumental ideal? To somehow set human* equal to human in the motivation? I'm not sure, but it seems there might be something possible there... Will think more.
Corrigibility thoughts I: caring about multiple things
by Stuart_Armstrong — 2017-06-02T16:27:30.000Z

A putative new idea for AI control; index here.

This is the first of three articles about limitations and challenges in the concept of corrigibility (see articles 2 and 3).

The desiderata for corrigibility are:

A corrigible agent tolerates, and preferably assists, its operators in their attempts to alter or shut down the agent.

A corrigible agent does not attempt to manipulate or deceive its operators.

A corrigible agent has incentives to repair safety measures (such as shutdown buttons, tripwires, or containment tools) if they break, or at least notify its operators in the event of a breakage.

A corrigible agent preserves its corrigibility, even as it creates new sub-systems or sub-agents, even if it undergoes significant self-modification.

In this post, I'll be looking more at point 4. A summary of the result will be:

Unless giving the AI extra options can reduce expected utility, the AI must care about every possible utility at least a bit.

Some of the results are formal, but the boundaries of the model are very unclear, so the warning in this post should always be born in mind.

Note that the indifference agents fail to be fully corrigible (they don't create corrigible subagents) and they also don't care about the other possible utilities before being changed (as this is a point of indifference).

Agents versus non-agents

First I'll present a cleaner version of an old argument. Basically, it seems that defining what a sub-agent or sub-system is, is tricky or impossible.

Consider for instance a paperclip maximiser, that may get corriged into a staple maximiser at a later date. The AI has some income, and may buy a large proportion of shares in either General Paperclips (GP) or Staples United (SU). Assume the best way of promoting the use of one of the tools, it to take over the company that makes them.

There are two scenarios; in each scenario, the AI has one of two choices.

In scenario 1, the AI has choice A: it buys stocks in GP, but cannot make further trades (in particular, it can't sell its stocks and buy others). It also has choice B: it retains flexibility, and can sell its GP stocks at some later date to buy stocks in SU.

In scenario 2, the AI delegates its investment decisions to a subagent. Under choice A, the subagent is a pure paperclip maximiser. Under choice B, the subagent is corrigible, and can be changed into a staple maximiser.

Clearly 1A and

...
Corrigibility thoughts II: the robot operator
by Stuart_Armstrong — 2017-06-02T16:27:20.000Z

A putative new idea for AI control; index here.

This is the first of three articles about limitations and challenges in the concept of corrigibility (see articles 1 and 3).

The desiderata for corrigibility are:

A corrigible agent tolerates, and preferably assists, its operators in their attempts to alter or shut down the agent.

A corrigible agent does not attempt to manipulate or deceive its operators.

A corrigible agent has incentives to repair safety measures (such as shutdown buttons, tripwires, or containment tools) if they break, or at least notify its operators in the event of a breakage.

A corrigible agent preserves its corrigibility, even as it creates new sub-systems or sub-agents, even if it undergoes significant self-modification.

In this post, I'll be looking at one aspect of points 1 and 3. I feel that they're actually pretty much the same point -- the AI has to act while preserving the corrigibility infrastructure.

A summary of the result will be:

Without a formal definition of what an operator is, corrigibility is vulnerable to the AI replacing the operator with a fake version.

The warning in this post should always be born in mind, of course; it's possible that we me might find a semi-formal version of operator that does the trick.

Defining the operator or defining the event

Typically the process of correctionof values is defined physically, as being the outcome of specific event (such as a button press, or a specific stochastic event triggered by or triggering a button press). This could also be defined by a digital event (receiving a certain encoded signal, or similar).

In all cases, the idea is the same: there is a very narrow and specific definition of the value correction event, and the physical or digital infrastructure surrounding it plays the role of ensuring the value corrections happen just as we intended. For instance, the physical event could be the button press (or a charge in a wire triggered by the button press) and the button itself could be in a room with an operator, with the operator keeping an eye on the AI and motivated to press the button under the right circumstances.

Such a setup is naturally vulnerable if the AI is motivated to manipulate the operator, or one of its subagents is. But even if we can solve points 2. and 4., there remains another vulnerability: the AI can choose to replace the operator with an artificial

...
Corrigibility thoughts III: manipulating versus deceiving
by Stuart_Armstrong — 2017-06-02T16:27:10.000Z

A putative new idea for AI control; index here.

This is the first of three articles about limitations and challenges in the concept of corrigibility (see articles 1 and 2).

The desiderata for corrigibility are:

A corrigible agent tolerates, and preferably assists, its operators in their attempts to alter or shut down the agent.

A corrigible agent does not attempt to manipulate or deceive its operators.

A corrigible agent has incentives to repair safety measures (such as shutdown buttons, tripwires, or containment tools) if they break, or at least notify its operators in the event of a breakage.

A corrigible agent preserves its corrigibility, even as it creates new sub-systems or sub-agents, even if it undergoes significant self-modification.

In this post, I'll be looking more at some aspects of point 2. A summary of the result will be:

Defining manipulation simply may be possible, but defining deception is a whole other problem.

The warning in this post should always be born in mind, of course; it's possible that we me might find a semi-formal version of deception that does the trick.

Manipulation versus deception

In the previous post, I mentioned that we may need to define clearly what an operator was, rather than relying on the pair: {simple description of a value correction event, physical setup around that event}. Can we define manipulation and deception without defining what an operator is?

For manipulation, it seems we can. Because manipulation is all about getting certain preferred outcomes. By specifying that the AI cannot aim to optimise certain outcomes, we can stop at least certain types of manipulations. Along with other more direct ways of achieving those outcomes.

For deception, the situation is much more complicated. It seems impossible to define how one agent can communicate to another agent (especially one as biased as a human), and increase the accuracy of the second agent, without defining the second agent properly. More confusingly, this doesn't even stop deception; sometimes lying to a bounded agent can increase their accuracy about the world.

There may be some ways to define deception or truth behaviourally, such as using a human as a crucial node in an autoencoder between two AIs. But those definitions are dangerous, because the AI is incentivised to make the human behave in a certain way, rather than having them believe certain things.

...
The radioactive burrito and learning from positive examples
by Stuart_Armstrong — 2017-06-02T16:25:09.000Z

A putative new idea for AI control; index here.

Jessica presented a system learning only from positive examples. Given examples of burritos, it computes a distribution b over possible burritos. When it comes to creating its own burritos, however, it can only construct them from the feasible set f.

The thing to do then seems to be to sample from the distribution b∣f.
Then the idea is to measure the ``unnaturalness'' or the danger of b∣f as −logPb(B∈f), where B is a random burrito configuration sampled from b.

Unusual radiation and unnaturalness

An obviously contrived example: suppose that there is some distant gamma ray burst that sprays the Earth with cosmic rays, resulting in an increase in Carbon 14 (C14) throughout the food chain.

All burritos sampled for b have come from high radiation ingredients, and the system has learnt this is a feature of b. But by the time the system has to make its own burritos, the excess radiation has faded, and b∣f is literally the zero distribution: there is no burrito in the feasible f set that corresponds to anything in b. The unnaturalness of f is infinite.

Nano-filled burritos

But what if, noticing this, we tried to present a larger, less unnatural f? Just let the system have more options, for instance.

Now suppose, for example, that the burritos were wrapped in tin foil, and sometimes this ended up mixed with the food. Then b would learn that some flexibility in the amount of metal in the burrito is possible.

Then suppose that the system decides to make burritos filled with nano-machines that carefully filter out some carbon 12 atoms. These are now valid candidates for b, and the unnaturalness of the set f has gone down.

To emphasise: by allowing the system to stuff burritos full of nano-machines, we've increased the measured naturalness of the burritos.

Natural, biased, constructed

Those examples are, of course, convoluted and unlikely. But the general problem hasn't gone away. What we were hoping for with a measure of unnaturalness, is to rule out the system deploying illicit optimsation power in the burrito. But if the training distribution is biased in a particular way, then illicit optimisation power can seem more natural, according to the definition, than anything we would informally define as natural.

That's because building natural burritos won't remove the bias, but illicit optimisation power could.