trentmkelly/LessWrong-43k huggingface dataset:
AI safety: three human problems and one AI issue
by Stuart_Armstrong — 2017-06-02T16:12:23.000Z

A putative new idea for AI control; index here.

There have been various attempts to classify the problems in AI safety research. Our old Oracle paper that classified then-theoretical methods of control, to more recent classifications that grow out of modern more concrete problems.

These all serve their purpose, but I think a more enlightening classification of the AI safety problems is to look at what the issues we are trying to solve or avoid. And most of these issues are problems about humans.

Specifically, I feel AI safety issues can be classified as three human problems and one central AI issue. The human problems are:

Humans don't know their own values (sub-issue: humans know their values better in retrospect than in prediction).

Humans are not (idealised) agents and don't have stable values (sub-issue: humanity itself is even less of an agent).

Humans have poor predictions of an AI's behaviour.

And the central AI issue is:

AIs could become extremely powerful.

Obviously if humans were agents and knew their own values and could predict whether a given AI would follow those values or not, there would be not problem. Conversely, if AIs were weak, then the human failings wouldn't matter so much.

The points about human values is relatively straightforward, but what's the problem with humans not being agents? Essentially, humans can be threatened, tricked, seduced, exhausted, drugged, modified, and so on, in order to act seemingly against our interests and values.

If humans were clearly defined agents, then what counts as a trick or a modification would be easy to define and exclude. But since this is not the case, we're reduced to trying to figure out the extent to which something like a heroin injection is a valid way to influence human preferences. This makes both humans susceptible to manipulation, and human values hard to define.

Finally, the issue of humans having poor predictions of AI is more general than it seems. If you want to ensure that an AI has the same behaviour in the testing and training environment, then you're essentially trying to guarantee that you can predict that the testing environment behaviour will be the same as the (presumably safe) training environment behaviour.

How to classify methods and problems

That's well and good, but how to various traditional AI methods or problems fit into this framework? This should give us an idea as

...
Optimisation in manipulating humans: engineered fanatics vs yes-men
by Stuart_Armstrong — 2017-06-02T15:51:44.000Z

A putative new idea for AI control; index here.

One of the ways in which humans aren't agents is that we can be manipulated into having any (or almost any) set of values - through drugs, brain surgery, extreme propaganda, and other methods.

If an AI is tasked with "satisfy human preferences", and if the AI can affect the definition of "human preferences" by changing humans, this is what we would expect it to do.

This could be combated by my making the definition of human preferences counterfactual. But it's not fully clear how to define counterfactual preferences, and it would be interesting to see whether we can constrain the AI from manipulating humans in other ways.

One idea is to look at what we informally call optimisation power. If the AI would prefer a human to be a u-maximising agent, then it presumably has to work hard at transforming the human values into that. And small changes in the definition of the AI's preference would mean that they would prefer transforming the human into a v-maximiser.

Thus, though optimisation power is hard to define, this would seem to fit the bill: an "honest" reward learning process is one where the final human values don't depend sensitively on the AI's initial values. And a manipulable one is where it does.

Let's dig into this a bit more. Why would an AI prefer u or v? Well, the generic failure mode for "satisfy human preferences" is to see it as the sum over all utilities u in some set U of "maximise u if the human agrees to maximise u". Then what the AI wants is for the human agree to maximise a v where v is the utility function the AI finds easiest to reach a high value on.

But then we can individually translate or scale the various utilities in U, to make different ones easier or harder to reach high values on. This would make the AI prefer the human to agree to a different maximisation.

So if R is the reward function that encodes "satisfy human preferences", there are many R that are equivalent if the AI cannot influence the human's values, but that are very different if the AI can. Looking for something like that could be a sign that something is wrong in the system.

Yes-men are a problem

And this is true, if the only option of the AI was to turn the human into an engineered fanatic with certain values. But it might also seek to turn the human into a yes-man, agreeing to anything the AI suggests. And this is something

...
Divergent preferences and meta-preferences
by Stuart_Armstrong — 2017-06-02T15:51:04.000Z

A putative new idea for AI control; index here.

In simple graphical form, here is the problem of divergent human preferences:

Here the AI either chooses A or ¬A, and as a consequence, the human then chooses B or ¬B.

There are a variety of situations in which this is or isn't a problem (when A or B or their negations aren't defined, take them to be the negative of what is define):

Not problems:

A/¬A= "gives right shoe/left shoe", B/¬B= "adds left shoe/right shoe".

A= "offers drink", ¬B= "goes looking for extra drink".

A= "gives money", B= "makes large purchase".

Potentially problems:

A/¬A= "causes human to fall in love with X/Y", B/¬B= "moves to X's/Y's country".

A/¬A= "recommends studying X/Y", B/¬B= "choose profession P/Q".

A= "lets human conceive child", ¬B= "keeps up previous hobbies and friendships".

Problems:

A= "coercive brain surgery", B= anything.

A= "extreme manipulation", B= almost anything.

A= "heroin injection", B= "wants more heroin".

So, what are the differences? For the "not problems", it makes sense to model the human as having a single reward R, variously "likes having a matching pair of shoes", "needs a certain amount of fluids", and "values certain purchases". Then all that the the AI is doing is helping (or not) the human towards that goal.

As you move more towards the "problems", notice that they seem to have two distinct human reward functions, RA and R¬A, and that the AI's actions seem to choose which one the human will end up with. In the spirit of humans not being agents, this seems to be AI determining what values the human will come to possess.

Grue, Bleen, and agency

Of course, you could always say that the human actually has reward R=IARA+(1−IA)R¬A, where IA is the indicator function as to whether the AI does action A or not.

Similarly to the grue and bleen problem, there is no logical way of distinguishing that "pieced-together" R from a more "natural" R (such as valuing pleasure, for instance). Thus there is no logical way of distinguishing the human being an agent from the human not being an agent, just from its preferences and behaviour.

However, from a learning and computational complexity point of view, it does make sense to distinguish "natural" R's (where RA and R¬A are essentially the same, despite the human's actions being different) from composite R's.

This allows us to define:

Preference divergence point: A pr

...
Acausal trade: double decrease
by Stuart_Armstrong — 2017-06-02T15:33:40.000Z

A putative new idea for AI control; index here.

Other posts in the series: Introduction, Double decrease, Pre-existence deals, Full decision algorithms, Breaking acausal trade, Trade in different types of utility functions, Being unusual, and Summary.

A reminder that we won't be looking at any "utilities I might have had before I knew who I was" scenarios.

This post is for illustrating a point about acausal trade: weakening the acausal trade network(s) in any way tends to reduce acausal trade more than linearly, as the traders cut back further on their trading. And the converse for strengthening the acausal trade network(s).

How to weaken the network

How could the acausal trade network be weakened? In potentially many ways. Greater uncertainty about the existence or the utilities of other agents, for instance. More agents who might defect from the trade, not have right utility function, or with who you can't reach a deal because of negotiation breakdown.

Basically anything that lowers the expected number of agents acausally trading with you - and also causes those agents to similarly have a lower expectation on the number of agents trading with you.

Illustration

Take the case where N=2, so there are only two possible agents, you (A1) and one other (A2), with utilities u1 and u2 respectively. Both agents are sure to exist, so q1=q2=1.

Trade can't happen unless there is some gain from trade - if it costs you more (in terms of u1) to increase u2, than the gain in u1 that the other agent is capable of giving you in exchange, then there is no trade that can happen.

So suppose you can increase u2 quite easily initially, but it gets harder and harder as you increase it more. Specifically, if you've already increased u2 by x, then it costs you, marginally, x to increase u2 further.

So the marginal cost is linear in x; cost, here, always refers to the decrease in u1 needed to pay for the increase in u2.

Assume the other agent is in exactly the same situation, mirrored.

Then, since we're assuming that the negotiations divide the gains from trade equally, then if you and the other agent have full knowledge of these facts and use a functional decision theory.

Then the rational acausal decision is for both of you to increase the utility of the other agent by 1, paying ∫10xdx=1/2 utility each, and hence each gaining 1/2 utility total.

But now imagine that the probability of each ag

...
Acausal trade: different utilities, different trades
by Stuart_Armstrong — 2017-06-02T15:33:31.000Z

A putative new idea for AI control; index here.

Other posts in the series: Introduction, Double decrease, Pre-existence deals, Full decision algorithms, Breaking acausal trade, Trade in different types of utility functions, Being unusual, and Summary.

In a previous post, the marginal gain of one utility in terms of another was given by a crisp formula.

This is obviously unrealistic, but how unrealistic is it? This post will look at various features we might expect from utility functions and acausally trading agents, and what makes them easy or difficult to trade with.

Independence

The ideal trade is something like agent A1 likes staples, agent A2 likes green objects, and they both decide to make green staples.

Here the two utilities are almost independent: increasing one costs almost nothing from the perspective of the other.

In general, independence (which could also be called low trade-offs) is very desirable between different utilities. This is not a transitive property: staples and green colour can be pretty independent, staples and blue colour as well, but obviously green and blue are in direct tension.

It's plausible that there will exist whole families of mutually independent utilities, coming together as a package that can get maximised together or separately.

Popularity

An obvious point, but utilities that are very popular - very widely shared by different agents - are more likely to be traded with, since there are more agents willing to trade in return. This willingness is non-linear in "double decrease" situations, for instance those that do not derive from some sort of timeless, pre-existence deal.

Complexity and coordination

Then there's the complexity of the preference. Simple utilities, or those whose existence is simple to deduce, are much easier to figure out and hence maximise.

This is partially a question of popularity, but it's also an artefact of the fact that agents may have different priors (or, even if they have the same priors, different posteriors after updating on the agent's own existence). In all those situations, simple preferences are likely to come up often.

These another type of complexity, which is complexity in increasing that utility. Suppose U counts the number of named nodes in a network, where every node knows the name of every other node. This utility is not hard to describe, but is useless for acausal trade, as a second netwo...
Acausal trade: universal utility, or selling non-existence insurance too late
by Stuart_Armstrong — 2017-06-02T15:33:07.000Z

A putative new idea for AI control; index here.

Other posts in the series: Introduction, Double decrease, Pre-existence deals, Full decision algorithms, Breaking acausal trade, Trade in different types of utility functions, Being unusual, and Summary.

We're going to have to look at the issue of universal, pre-existence trade agreements - the idea that agents should abide by some sort of idealised trade agreement based on a universal prior of possible agent values, without updating on the fact that they happen to exist as agents.

The arguments for it seem superficially similar to the arguments for preferring FDT/UDT/TDT over CDT, but are actually quite different. In brief, there is no profit, advantage, or benefit for an existing agent to commit to a trade agreement with non-existing agents.

Choosing a better decision theory

Newcomb, as usual

Causal decision theory gets the Newcomb problem wrong.

There are many ways to see that CDT gets this wrong (for a start, it ends up with less money that other agents), but one of the most damning is that CDT is not stable - it will self-modify into a sort of timeless agent, if it can do so, precisely to triumph on Newcomb-like problems.

It does this in a clunky and odd way: it will one-box on Newcomb problems where Omega does its prediction after learning of CDT's change, but not on Newcomb problems where Omega made its decision in the past.

The reason for it's clunkness is that CDT, being causal, doesn't allow for correlations between its own decision and causally anterior events.

Functional decision theory, which has a richer theory of correlations, would spot that correlation, and, assuming it had somehow ended up with a CDT-like decision module, would immediately change that into an FDT-compatible one that would one-box on all Newcomb problems.

I had another argument for the weakness of CDT (and some forms of UDT), namely that it shouldn't make a difference whether an agent was simulated or just predicted in a non-simulation manner. Keep this argument in mind, because I think it's wrong.

A mugging both counterfactual and surprising

Suppose an agent is using the FDT understanding of correlations and a CDT decision module, and is suddenly confronted with the Counterfactual mugging.

If the agent had time to think of the problem ahead of time, they would have self-modified into a full FDT agent. But now it is too late: they alrea

...
Acausal trade: trade barriers
by Stuart_Armstrong — 2017-06-02T15:32:58.000Z

A putative new idea for AI control; index here.

Other posts in the series: Introduction, Double decrease, Pre-existence deals, Full decision algorithms, Breaking acausal trade, Trade in different types of utility functions, Being unusual, and Summary.

In a previous post, I discussed how one might convince an agent not to engage in acausal trade.

The idea was to reward the agent only for extra utility U that accrued because the agent was turned on (by a stochastic event X). Since causally disconnected agents couldn't observe X, they would "offer" the same "deals" whether or not the agent was turned on.

So the agent might be able to get a tremendous boost in utility from an acausal deal, but that boost would happen in the X world as well as the ¬X world, so the agent wouldn't count that boost as a benefit.

That was effective as far as it went, but there was one kind of situation it didn't deal with: what if the agent was simulated? Then the event X would be within the simulation, and the simulating 'lords of the Matrix' would be causally connected with the agent, hence the agent would act taking their preferences into account.

That in itself is still not a problem; but what if the agent had uncertainty about its own location? It might be in the "real" world, or it might be a simulation made other entities, causally disconnected from the "real" world. Then if the agent acted given that uncertainty, it would be in effect doing a form of acausal trade.

Grounding the world

There is no costless solution, for any such solution must rule out the agent acting like it was in a simulation, which means that we incur a real cost if we are in a simulation.

But if we're willing to pay that cost, then one way of reducing the problem is to ground U in our understanding of physics. So instead of U="human flourishing", we have:

U="human flourishing in a universe that roughly follows the known laws of physics, will last at least this many trillion years, has these restrictions on how fast information moves and how causality works".

The idea is that a simulation that detailed would be indistinguishable with the real world (and the simulated humans therein would be real moral subjects).

Graded miracles

Of course, if U hits a constant when the laws of physics are violated, then the agent will ignore all "miracles", no matter how convincing. The booming voice of god coming from all elect

...
Futarchy, Xrisks, and near misses
by Stuart_Armstrong — 2017-06-02T08:02:43.000Z

All the clever ways of getting betting markets to take xrisks into account suffer from one big flaw: the rational xrisk bettor only makes money if the xrisk actually happens.

Now, the problem isn't because "when everyone is dead, no-one can collect bets". Robin Hanson has suggested some interesting ideas involving tickets for refuges (shelters from the disaster), and many xrisks will be either survivable (they are called xrisks, after all) or will take some time to reach extinction (such as a nuclear winter leading to a cascade of failures). Even if markets are likely to collapse after the event, they are not certain to collapse, and in theory we can also price in efforts to increase the resilience of markets and see how changes in that resilience changes the prices of refuge tickets.

The main problem, however, is just how irrational people are about xrisks, and how little discipline the market can bring to them. Anyone who strongly over-estimates the probability of an xrisk can expect to gradually lose all their money if they act on that belief. But someone who under-estimates xrisk probability will not suffer until an xrisk actually happens. And even then, they will only suffer in a few specific cases (refuge prices are actually honoured and those without them suffer worse fates). This is, in a way, the ultimate Talebian about black swan: huge market crashes are far more common and understandable than xrisks.

Since that's the case, it might be better to set up a market in near misses (an idea I've heard before, but can't source right now). A large meteor that shoots between the Earth and the Moon; conventional wars involving nuclear powers; rates of nuclear or biotech accidents. All these are survivable, and repeated, so the market should be much better at converging, with the overoptimistic repeatedly chastised as well as the overpessimistic.
Futarchy, Xrisks, and near misses
by Stuart_Armstrong — 2017-06-02T08:02:13.561Z

Crossposted at the Intelligent Agent Forum.

All the clever ways of getting betting markets to take xrisks into account suffer from one big flaw: the rational xrisk bettor only makes money if the xrisk actually happens.

Now, the problem isn't because "when everyone is dead, no-one can collect bets". Robin Hanson has suggested some interesting ideas involving tickets for refuges (shelters from the disaster), and many xrisks will be either survivable (they are called xrisks, after all) or will take some time to reach extinction (such as a nuclear winter leading to a cascade of failures). Even if markets are likely to collapse after the event, they are not certain to collapse, and in theory we can also price in efforts to increase the resilience of markets and see how changes in that resilience changes the prices of refuge tickets.

The main problem, however, is just how irrational people are about xrisks, and how little discipline the market can bring to them. Anyone who strongly *over-estimates* the probability of an xrisk can expect to gradually lose all their money if they act on that belief. But someone who under-estimates xrisk probability will not suffer until an xrisk actually happens. And even then, they will only suffer in a few specific cases (refuge prices are actually honoured and those without them suffer worse fates). This is, in a way, the ultimate Talebian about black swan: huge market crashes are far more common and understandable than xrisks.

Since that's the case, it might be better to set up a market in near misses (an idea I've heard before, but can't source right now). A large meteor that shoots between the Earth and the Moon; conventional wars involving nuclear powers; rates of nuclear or biotech accidents. All these are survivable, and repeated, so the market should be much better at converging, with the overoptimistic repeatedly chastised as well as the overpessimistic.
Why I am not currently working on the AAMLS agenda
by jessicata — 2017-06-01T17:57:24.000Z

(note: this is not an official MIRI statement, this is a personal statement. I am not speaking for others who have been involved with the agenda.)

The AAMLS (Alignment for Advanced Machine Learning Systems) agenda is a project at MIRI that is about determining how to use hypothetical highly advanced machine learning systems safely. I was previously working on problems in this agenda and am currently not.

The agenda

See the paper. The agenda lists 8 theoretical problems relevant to
aligning AI systems substantially similar to current machine learning systems.

History

Around March 2016, I had thoughts about research prioritization: I thought it
made sense to AI safety researchers spend more time thinking about machine learning systems.
In a similar timeframe, some other researchers updated towards shorter timelines. I had some
discussions with Eliezer, Paul, Nate, and others, and
came up with a list of problems that seemed useful to think about.

Then some of us (mostly me, with significant help from others) wrote up the paper about the problems. The plan was
for some subset of the researchers to work on them.

Progress since the paper

Since writing the paper, progress has been slow:

I had concrete thoughts about inductive ambiguity identification (with Ryan's
help); some of this is written up on the forum
here,
here,
here,
here. Retrospectively, this line
of thinking seems like a dead end, though I'm not highly confident of this
judgment.

Some researchers and I have thought about many of the problems and gained a
slightly improved conceptualization of them, but this improved
conceptualization is still quite vague and hasn't led to concrete progress
towards solutions.

Why was little progress made?

Difficulty

I think the main reason is that the problems were
very difficult. In particular, they were mostly selected on the basis of "this
seems important and seems plausibly solveable", rather than any strong
intuition that it's possible to make progress.

In comparison, problems in the agent foundations agenda have seen more progress:

Logical uncertainty (Definability of truth, reflective oracles, logical inductors)

Decision theory (Modal UDT, reflective oracles, logical inductors)

Vingean reflection (Model polymorphism, logical inductors)

One thing to note about these problems is that they were formulated on the basis of
a strong intuition that they ought t

...