trentmkelly/LessWrong-43k huggingface dataset:
Thoughts on Quantilizers
by Stuart_Armstrong — 2017-06-02T16:24:37.000Z

A putative new idea for AI control; index here.

This post will look at some of the properties of quantilizers, when they succeed and how they might fail.

Roughly speaking, let f be some true objective function that we want to maximise. We haven't been able to specify it fully, so we have instead a proxy function g. There is a cost function c=f−g which measures how much g falls short of f. Then a quantilizer will choose actions (or policies) radomly from the top n% of actions available, ranking those actions according to g.

It is plausible that for standard actions or policies, g and f are pretty similar. But that when we push to maximising g, then the tiny details where g and f differ will balloon, and the cost can grow very large indeed.

This could be illustrated roughly by figure I, where g and c are plotted against each other; imagine that c is on a log scale.

The blue areas are possible actions that can be taken. Note a large bunch of actions that are not particularly good for g but have low cost, a thinner tail of more optimised actions that have higher g and still have low cost, and a much thinner tail that has even higher g but high cost. The g-maximising actions with maximal cost are represented by the red star.

Figure I thus shows a situation ripe for some form of quantilization. But consider figure II:

In figure 2, the only way to get high g is to have a high c. The situation is completely unsuited for quantilization: any g maximiser, even a quantilizer, will score terribly under f. But that means mainly that we have chosen a terrible g.

Now, back to figure I, where quantilization might work, at least in principle. The ideal would be situation Ia; here blue represents actions below the top n% cut-off, green those above (which include the edge-case red-star actions, as before):

Here the top n% of actions all score a good value under g, and yet most of them have low cost.

But even within the the broad strokes of figure I, quantilization can fail. Figure Ib shows a first type of failure:

Here the problem is that the quantilizer lefts in too many mediocre actions, so the expectation of g (and f) is mediocre; with a smaller n%, the quantilizer would be better.

Another failure mode is figure Ic:

Here the n% is too low: all the quantilized solutions have high cost.

Another quantilizer design

An idea I had some time ago was that, instead of of taking the top n% of th

...
Emergency learning
by Stuart_Armstrong — 2017-06-02T16:23:34.000Z

A putative new idea for AI control; index here.

Suppose that we knew that superintelligent AI was to be developed within six months, what would I do?

Well, drinking coffee by the barrel at Miri's emergency research retreat I'd... still probably spend a month looking at things from the meta level, and clarifying old ideas. But, assuming that didn't reveal any new approaches, I'd try and get something like this working.

Standard setup

Take a reinforcement learner AI, that we want to safely move a strawberry onto a plate. A human sits nearby and provides a reward based on inspecting the AI's behaviour.

As it stands, this setup is completely vulnerable to reward hacking. The reward is not provided for safe moving of the strawberry; instead the reward is provided by having the human judge that the task has been accomplished and then pressing a button. Taking control of the human or control of the button is likely to be possible for a superintelligent AI; and, as it stands, that would be mandated by this reward function.

Learning from positive and various negative examples

Could we have the AI instead learn what the reinforcement signal "should be"? It seems that it might at least be possible, if we can make the AI learn from both positive and negative examples. I'd make five categories of examples from which the AI could learn. It may be too dangerous to have the superintelligent AI used directly in constructing these examples; in that case, the rewards would be given to a simpler, dumber version of the AI, and the examples passed on to the superintelligent AI for offline training.

#. Simple positive and negative examples. These are the basic examples from above: the AI completes the task or fails to, and gets the consequent reward. The AI stays within its room and the human is sober, rested, uninfluenced, and so on.
#. Simple more dubious examples. These are examples where the AI gets a reward, but the learning process judges that these rewards were not fairly allocated. Maybe the human was drunk, or not paying attention. Maybe they slipped, and pressed the button by accident. Or they were in a hurry to end their shift, and so pressed the button. All these examples are given a score from -1 to 1. Those with a score of 1 are taken as legitimate examples of rewards, those with a score of 0 are taken as random reward giving, while those with a score of -1 are taken as

...
Humans as a truth channel
by Stuart_Armstrong — 2017-06-02T16:22:28.000Z

A putative new idea for AI control; index here.

Defining truth and accuracy is tricky, so when I've proposed designs for things like Oracles, I've either used a very specific and formal question, or and indirect criteria for truth.

Here I'll try and get a more direct system so that an AI will tell the human the truth about a question, so that the human understands.

The basic idea is simple. The first AI wishes to communicate certain facts to a second AI, but has to use the human as an intermediary. The first AI talks to the human, and then the human talks with the second AI. If the facts are to be accurate, the human has to understand them.

Of course, this will immediately fail for a variety of reasons. Even if neither AI hacks the human directly, the human doesn't have to understand anything to serve as a conduit. So the rest of this post is dedicated to making that basic idea much more secure. If you just want to have the final idea and avoid the justifications, then you can skip directly to the end.

The knowledge to pass on

The human would have a good understanding of a subject, if they are able to answer a large quantity of questions about it. Having a long conversation with the second AI is tiring, among other things. A human simply won't be able to answer ten million questions about what they understand, at least not well.

So the idea could be to have the second AI ask a single question, randomly chosen from a large collection of questions, and have the human give a single yes or no answer. Both AIs are then graded on the accuracy of the answer.

The answer...

What does the second AI want to do? It basically wants to dissect the human (figuratively or literally), suck out all their knowledge, and infer what the first AI was trying to communicate. Note that this doesn't require the human to understand anything, just that the second AI can infer what the first one wanted by taking the human apart.

So instead of allowing the second AI unrestricted access to the human, it instead formulates its (boolean) question without interference, and then the human choose yes or no, and that choice is counted as an answer.

...and the question

We could have a list of actual questions, but that makes the second AI redundant, and means that the first AI will optimise for certain answers from a set of questions, not for genuine understanding.

Instead, the second AI then gets a

...
All the indifference designs
by Stuart_Armstrong — 2017-06-02T16:20:33.000Z

A putative new idea for AI control; index here.

The aim of this post is simply to collect all the different "indifference" proposals that I know of. Let me know if I've missed any!

The basic idea is that the agent is following policy π0 with reward R0, and there is a button that if pressed (event B) will change the agent to following policy π1 with reward R1.

Policy versus reward

My initial proposal considered only change of utility functions -- similar to reward functions -- and assumed the policy would always be to maximise that utility.

Conversely, the Safely Interruptible Agents paper just looked at policy changes, while sticking to the same reward function.

In general, policy and reward changes can be treated similarly -- what matters is the changes between Vπ0(R0) (the expected value of R0 under π0, as estimated by the agent) and Vπ1(R1).

Indeed, you could just treat it as a policy change, with the added caveat that πi also 'chooses' to claim their reward from Ri.

Method: Bias penalty or indifference

If we have an initial estimate p for the probability of B (hence of the agent following π1/R1), then if the agent takes an action a, this probability changes to pa.

One way of fighting this, is to penalise the agent for large changes p→pa. In the terms of this post, the agent is taking biased actions.

And to combat large biases, the agent could get an extra bias penalty, as explained in the Towards Interactive Inverse Reinforcement Learning poster.

But the rest of this post will focus on bias-indifference, rather than bias penalty.

Counterfactual: evidential, causal, translation

When comparing Vπ0(R0) and Vπ1(R1), it's important to note that this is a counterfactual comparison: it's comparing what would have happened under π0 versus π1, but only one of those policies is actually going to be happening, depending on whether the button press B happens.

Evidential counterfactual

The first idea is to have some sort of fixed probability for B versus ¬B. And then the agent acts as if the probability cannot be changed by anything the agent does.

Call this the "evidential" counterfactual. The problem with it is that it causes the agent to want to manipulate conditional probabilities (similarly how evidential decision theory can desire to manipulate the news). Basically, if the agent can decrease the probability of B while making the reward for B much larger, it will do so

...
Indifference and compensatory rewards
by Stuart_Armstrong — 2017-06-02T16:19:30.000Z

A putative new idea for AI control; index here.

It's occurred to me that there is a framework where we can see all "indifference" results as corrective rewards, both for the utility function change indifference and for the policy change indifference.

Imagine that the agent has reward R0 and is following policy π0, and we want to change it to having reward R1 and following policy π1.

Then the corrective reward we need to pay it, so that it doesn't attempt to resist or cause that change, is simply the difference between the two expected values:

V(R0|π0)−V(R1|π1),

where V is the agent's own valuation of the expected reward, conditional on the policy.

This explains why off-policy reward-based agents are already safely interruptible: since we change the policy, not the reward, R0=R1. And since off-policy agents have value estimates that are indifferent to the policy followed, V(R0|π0)=V(R1|π1), and the compensatory rewards are zero.
Counterfactually uninfluenceable agents
by Stuart_Armstrong — 2017-06-02T16:17:10.000Z

A putative new idea for AI control; index here.

Techniques used to counter agents taking biased decisions do not produce uninfluenceable agents.

However, using counterfactual tools, we can construct uninfluenceable ˆP and P, starting from biased and influenceable ones.

Why is uninfluenceable necessary? Well, an unbiased agent can still take actions such as 'randomise their own reward (independent of human choice)', as long as the choice of their randomisation is unbiased. For instance, let π0 be some default policy, and let the tidying (R0) versus cooking (R1) agent currently consider both options to be equally likely. So, ultimately, the human will choose one or the other equally: Eπ0μP(R0)=Eπ0μP(R1)=1/2. Then if the agent chooses to randomise its own reward immediately with 50% probability, this is an unbiased policy. And one the agent may prefer to do, because then it will know its own reward immediately, rather than waiting for the human to decide.

As usual, the notation of this post is used.

The counterfactual distribution

Let μ=(S,A,O,T,O,T0) be the environment, P a potentially biased and influenceable distribution, and π0 some default policy.

Then define the counterfactual reward learning distribution Pπ0μ on the reward R, given history hm, as:

Pπ0μ(R∣hm)=∑s∈SEπ0μ[P(R)∣s0=s]μ(s0=s∣hm).

Thus Pπ0μ(R∣hm) is estimated by using hm to compute the probability that s0=s, for each s, and then estimating the (counterfactual) expectation of P given s0=s and the default policy π0.

It's not hard to see that Pπ0μ is uninfluenceable: given knowledge of the initial state, it's just Eπ0μ[P(⋅)∣s0=s], which is independent of the agent's actions. The learning process is simply about figuring out what the initial

Dependence on POMDP structure

Note that if two environments are counterfactually equivalent, they can still generate different Pπ0μ even for the same P and π0.

This can be seen immediately from examples μ and μ′′ from this post, which are counterfactually equivalent.

In that setting, aw corresponds to waiting for a human to decide between R0 and R1, while ar corresponds to the agent randomising their own reward.

Set P(Ri∣s0awswi)=1 and P(Ri∣s0awsri)=1 -- the final state determines Ri, and let the default policy be π0=aw, the wait action.

Since Eπ0μ[P(Ri)]=0.5 and μ has only one possible state for s0, Pπ0μ(Ri)=0.5 is the only possibility.

In contrast Eπ0μ′′[P(Ri)|s0=si0]

...
Translation "counterfactual"
by Stuart_Armstrong — 2017-06-02T16:16:21.000Z

A putative new idea for AI control; index here.

In a previous post, I briefly mentioned translations as one of three possible counterfactuals for indifference. Here I want to clarify what I meant there, because the idea is interesting.

Imagine that there is a button, which, if a human presses it (event B), will cause an AI to have reward signal R0. If the button isn't pressed (event ¬B), the AI will have reward signal R1.

Initially, the probabilities of B and ¬B are equal at 1/2.

Now, suppose that the AI takes an action a that increases the probability of B to 3/4 (and decreases the probability of ¬B to 1/4).

We want the AI to be indifferent to the change in probability caused by its actions.

Evidential counterfactual

In the "evidential" counterfactual, the AI will simply behave as if the probability of B and ¬B were fixed, and unaltered from the initial odds:

The problem with this approach is that this doesn't correspond to a true utility/reward functions, leading to the paradoxes detailed here and here (see the section on outcome pumps).

Causal counterfactual

In the "causal" counterfactual, there is some other event that, with small probability, will force the button to be pressed, or prevent it from being pressed, whatever humans want or try.

This event is supposed to be independent of anything anyone does (imagine some specified radioactive decay event, or being triggered by distant cosmic events).

Call X the event that the button is forcibly pressed, while ¬X means it is forcibly not-pressed. The AI does not value being in any other world (meaning that the AI gets constant reward in any world where neither X nor ¬X happen).

Then the AI will behave as if the ratio of probabilities of following R0 versus R1 is the (constant) ratio of P(X) to P(¬X), whatever the probability of B becomes.

The problem is that B (the human presses the button) is not the same as X (the button is forcibly pressed by some stochastic event), so expected rewards conditional on one will generally not be the same as those conditional on the other.

Translation "counterfactual"

In the "translation" counterfactual, in contrast, there are no unusual conditionings or assuming unlikely events. Instead, the event B/¬B is decoupled somewhat from the event of the AI maximising R0 or R1.

In this specific case, the probability of the AI maximising R0, given B, is reduced, so that the final probability

...
Understanding the important facts
by Stuart_Armstrong — 2017-06-02T16:15:08.000Z

A putative new idea for AI control; index here.

I've got a partial design for motivating an AI to improve human understanding.

However, the AI is rewarded for generic human understanding of many variables, most of the quite pointless from our perspective. Can we motivate the AI to ensure our understanding on variables we find important? The presence of free humans, say, rather than air pressure in Antarctica?

It seems we might be able to do this. First, because we've defined human understanding, we can define accuracy-increasing statements: statements that, if given to a human, will improve their understanding.

So I'd say that a human understands the important facts about a real or hypothetical world, if there are no short accuracy-increasing statements that would cause them to radically change their assessment about the quality of a world (AI: "oops, forgot to mention everyone in under the influence of brain slugs - will that change your view of this world?").

Of course, it possible to present true informative facts in the worse possible light, so it might be better to define this as "if there are short accuracy-increasing statements that would cause them to radically change their assessment about the quality of a world, there are further accuracy-increasing statements that will undo that change". (AI: "actually, brain slugs are a popular kind of hat where people store their backup memories").
Low impact versus low side effects
by Stuart_Armstrong — 2017-06-02T16:14:03.000Z

A putative new idea for AI control; index here.

I just had a talk with Victoria Krakovna about reducing side effects for an AI, and though there are similarities with low impact there are some critical way in which the two differ.

#Distance from baseline

Low impact and low side effects use a similar distance approach: some ideal baseline world or set of worlds is defined, and then the distance between that baseline and the actual world is computed. The distance is then used as penalty term the AI should minimise (while still achieving its objectives).

One useful measure of distance is to list a huge number of variables (stock prices, air pressure, odds of films winning Oscars...) and penalise large deviations in those variables. Every variable is given a certain weight, and the weighted sum is the total distance metric (there are more complicated versions, but this simple metric will suffice for the moment).

#What's the weather like on Venus?

The two approach weigh the variables differently, and for different purposes. Suppose that one of the variables is the average surface temperature of Venus.

Now suppose that the temperature of Venus increases by 1 degree during the operation of the AI. For most low side effects AIs, this is perfectly acceptable. Suppose the AI is aiming to cure cancer. Then as we formalise negative side effects, we start to include things like human survival, human happiness and flourishing, and so on. Temperature changes in distant planets are certainly not prioritised.

And the AI would be correct in that assessment. We would be perfectly happy to accept a cure for cancer in exchange of a small change to Venusian weather. And a properly trained AI, intent on minimising bad side effects, should agree with us. So the weight of the "temperature on Venus" variable will be low for such an AI.

In contrast, a low impact AI sees a temperature change on Venus as an utter disaster - Venusian temperature is likely to be much more important than anything human or anything on Earth. The reason is clear: only an immensely powerful AI could affect something as distant and as massive as that. If Venusian temperature changes strongly as a result of AI action, the low impact containment has failed completely.

#Circling the baseline from afar

The two approaches also differ in how attainable their baseline is. The low impact approach defines a baseline world which t

...
Agents that don't become maximisers
by Stuart_Armstrong — 2017-06-02T16:13:05.000Z

A putative new idea for AI control; index here.

According to the basic AI drives thesis, (almost) any agent capable of self-modification will self-modify into an expected utility maximiser.

The typical examples are the inconsistent utility maximisers, the satisficers, unexploitable agents, and it's easy to think that all agents fall roughly into these broad categories. There's also the observation that when looking at full policies rather than individual actions, many biased agents become expected utility maximisers (unless they want to lose pointlessly).

Nevertheless... there is an entire category of agents that generically seem to not self-modify into maximisers. These are agents that attempt to maximise f(E(U)) where U is some utility function, E(U) is its expectation, and f is a function that is neither wholly increasing nor decreasing.

Intransitive example

Let there be a U with three action a0, a5, and a10 that set U to 0, 5, and 10, respectively.

The function f is 1 in the range (4,6) and is 0 elsewhere. Hence the agent needs to set the expectation of U to be in that range.

What will happen is that one action will be randomly removed from the set, and the agent will then have to choose among the remaining two actions. What possible policies can the agent take?

Well, there are three option sets the agent could face -- (a0,a5), (a5,a10), and (a10,a0) -- each with two options and hence 23=8 pure policies. Two of those policies -- choosing always the first option in those ordered pairs, or choosing always the second option -- are intransitive, as they rank no option above the other two.

But actually those intransitive options have an expected utility of (0+5+10)/3=5, which is just what the agent wants.

Even worse, none of the other (transitive) policies are acceptable. You can see this because each of the six transitive policies can be reached by taking one of the intransitive policies and flipping a choice, which must change the expected utility by ±5/3 or ±10/3, moving it out of the (4,6) range.

Thus there are no possible expected utility maximalisation that correspond to these options, as such maximalisations are always transitive.

Or another way of seeing this: the random policy of picking an action randomly has an expectation of (0+0+5+5+10+10)/6=5, so is also an acceptable policy. But for expected utility maximalisation, if the random policy is acceptable, th

...