trentmkelly/LessWrong-43k huggingface dataset:
What emotions would AIs need to feel?
by Stuart_Armstrong — 2019-01-08T15:09:32.424Z

[this post is completely fluffy and superficial]

When talking about AIs, the issue of emotions often comes up. I normally finesse the question by saying that we don’t really know if AIs would have things we would recognise and emotions, and would you stop pestering me, I have some serious and scary points to make.

But… If we take the question seriously for a second, and allow ourselves to speculate… The main angle of attack is to wonder: what are emotions for, what functions do they serve? And would they serve the same functions in an AI?

So, here’s a speculative cross-section of emotions, and whether AIs are likely to have versions of them:

Very likely: Joy, Sadness, Boredom, Empathy, Interest

At their simplest, joy and sadness are responses to the situation turning out better or worse than expected. These are essential to learning, and the AI would certainly have some version of them. Similarly, boredom is a sign that the AI should do more exploration of its environment, empathy is useful for understanding other agents, and interest is a sign that a certain area should be investigated in priority.

Possible, if the AI is limited or non-transparent: Anger, Kindness, Cruelty, Trust, Distrust, Surprise

If the AI is able to credibly pre-commit to certain actions (eg if attacked, it will retaliate), then some of these emotions won’t be needed. But it might not be able to credibly signal that, in which case anger would be useful: other agents will be less inclined to exploit it, lest it react aggressively (note that giving in to anger is generally a disadvantage, but being known to give in to anger is often an advantage). Kindness and cruelty are similarly tools for managing social interactions; if the AI is transparent and can pre-commit, they won’t be needed.

Empathy is the key AI emotion, but it’s possible that it may award special categories for other agents with especially positive or negative traits. Surprise, finally, is a signal that unlikely events are happening, and a re-assessment is needed; if the AI cannot implement its assessments and reassessments in a smooth way, it may need some threshold of surprise to trigger a reaction.

Possible, if the AI has a physical body: Fear, Disgust, Anxiety, Feelings of Security

These emotions are all connected with protecting our own physical body, so an AI would only feel them if it had a physical body itself, and

...
Reframing Superintelligence: Comprehensive AI Services as General Intelligence
by Rohin Shah — 2019-01-08T07:12:29.534Z
Link: https://www.fhi.ox.ac.uk/wp-content/uploads/Reframing_Superintelligence_FHI-TR-2019-1.1-1.pdf?asd=sa

Since the CAIS technical report is a gargantuan 210 page document, I figured I'd write a post to summarize it. I have focused on the earlier chapters, because I found those to be more important for understanding the core model. Later chapters speculate about more concrete details of how AI might develop, as well as the implications of the CAIS model on strategy. ETA: This comment provides updates based on more discussion with Eric.

The Model

The core idea is to look at the pathway by which we will develop general intelligence, rather than assuming that at some point we will get a superintelligent AGI agent. To predict how AI will progress in the future, we can look at how AI progresses currently -- through research and development (R&D) processes. AI researchers consider a problem, define a search space, formulate an objective, and use an optimization technique in order to obtain an AI system, called a service, that performs the task.

A service is an AI system that delivers bounded results for some task using bounded resources in bounded time. Superintelligent language translation would count as a service, even though it requires a very detailed understanding of the world, including engineering, history, science, etc. Episodic RL agents also count as services.

While each of the AI R&D subtasks is currently performed by a human, as AI progresses we should expect that we will automate these tasks as well. At that point, we will have automated R&D, leading to recursive technological improvement. This is not recursive self-improvement, because the improvement comes from R&D services creating improvements in basic AI building blocks, and those improvements feed back into the R&D services. All of this should happen before we get any powerful AGI agents that can do arbitrary general reasoning.

Why Comprehensive?

Since services are focused on particular tasks, you might think that they aren't general intelligence, since there would be some tasks for which there is no service. However, pretty much everything we do can be thought of as a task -- including the task of creating a new service. When we have a new task that we would like automated, our service-creating-service can create a new service for that task, perhaps by training a new AI system, or by taking a bunch of existing services and putting them together, etc. In this way, the collection of ser...
Which approach is most promising for aligned AGI?
by Chris_Leong — 2019-01-08T02:19:50.278Z

I know that this is something of a speculative question and that people will have wildly different views here, but it seems important to try to have an idea of which approaches are more likely than others to lead to AGI. It's okay to argue that multiple approaches are promising, but you might want to consider seperate answers if you have a reasonable amount to write about each approach.
Sequence introduction: non-agent and multiagent models of mind
by Kaj_Sotala — 2019-01-07T14:12:30.297Z

A typical paradigm by which people tend to think of themselves and others is as consequentialist agents: entities who can be usefully modeled as having beliefs and goals, who are then acting according to their beliefs to achieve their goals.

This is often a useful model, but it doesn’t quite capture reality. It’s a bit of a

fake framework

. Or in computer science terms, you might call it a

leaky abstraction

.

An abstraction in the computer science sense is a simplification which tries to hide the underlying details of a thing, letting you think in terms of the simplification rather than the details. To the extent that the abstraction actually succeeds in hiding the details, this makes things a lot simpler. But sometimes the abstraction inevitably leaks, as the simplification fails to predict some of the actual behavior that emerges from the details; in that situation you need to actually know the underlying details, and be able to think in terms of them.

Agent-ness being a leaky abstraction is not exactly a novel concept for Less Wrong; it has been touched upon several times, such as in Scott Alexander’s

Blue-Minimizing Robot Sequence

. At the same time, I do not think that it has been quite fully internalized yet, and that many foundational posts on LW go wrong due to being premised on the assumption of humans being agents. In fact, I would go as far as to claim that this is the biggest flaw of the original Sequences: they were attempting to explain many failures of rationality as being due to cognitive biases, when in retrospect it looks like understanding cognitive biases doesn’t actually make you substantially more effective. But if you are implicitly modeling humans as goal-directed agents, then cognitive biases is the most natural place for irrationality to emerge from, so it makes sense to focus the most on there.

Just knowing that an abstraction leaks isn’t enough to improve your thinking, however. To do better, you need to know about the actual underlying details to get a better model. In this sequence, I will aim to elaborate on various tools for thinking about minds which look at humans in more granular detail than the classical agent model does. Hopefully, this will help us better get past the old paradigm.

My model of what I think our subagents looks like draws upon a number of different sources, including neuroscience, psychotherapy and meditation, so...
Optimizing for Stories (vs Optimizing Reality)

by Ruby — 2019-01-07T08:03:22.512Z

Epistemic status: highly confident in the basic distinct, though it's not at all profound. Further details, models, and advice are somewhat speculative despite being drawn from varying amounts of observation. Essay a little rushed since otherwise I’d be unlikely to publish.

When setting out on a venture, one faces some choices.

Optimize reality: How much do you optimize for the success of your stated goal? I define goal to be some desired state of reality such that success or failure is assessed with respect to reality.
Optimize stories: How much do you optimize for the appearance of success at your stated goal, i.e. stories? I define a story as a collection of facts about a reality asserted to be true and relevant, usually presented to others but sometimes only to oneself.

Ideally, optimizing for one would be the same as optimizing for another. Very often, however, optimizing one is not the same as optimizing for other. What’s worse, the two ends compete for the same set of limited resources.

A concrete example might be someone who sets out to develop and sell a medicinal tea which relieves hangover symptoms. Overall success is selling your tea and making money. Investing in optimizing reality would mean investing in experiments to develop and improve the tea. Introducing variants, randomized controlled trials, etc., etc. Optimizing story means having a really good explanation for why your tea is able to do what you claim it does, plus having a great website with good copy, publishing testimonials, publishing your methodology and experimental results, etc.

Success can be attained by both, in combination, but also possibly each on its own. If you have a tea which works well, word will spread and you’ll end up with many customers seeking your genuinely palliative tea. Alternatively, if your marketing materials are persuasive, you might accrue many customers too, even if your tea is no better than placebo. Of course, truly effective tea plus a well-conveyed story about its great properties will generate more sales than effective tea or a good story alone.

Optimizing Reality

The following points are worth noting:

When optimizing reality, success depends on reality at large being a certain way.
There is only one reality when it comes to what you’re optimizing for.
You can’t fool reality. It is what it is no matter what you say or do.
Optimizing for reality is easie

...
AI safety without goal-directed behavior
by Rohin Shah — 2019-01-07T07:48:18.705Z

When I first entered the field of AI safety, I thought of the problem as figuring out how to get the AI to have the “right” utility function. This led me to work on the problem of inferring values from demonstrators with unknown biases, despite the impossibility results in the area. I am less excited about that avenue because I am pessimistic about the prospects of ambitious value learning (for the reasons given in the first part of this sequence).

I think this happened because the writing on AI risk that I encountered has the pervasive assumption that any superintelligent AI agent must be maximizing some utility function over the long term future, such that it leads to goal-directed behavior and convergent instrumental subgoals. It’s often not stated as an assumption; rather, inferences are made assuming that you have the background model that the AI is goal-directed. This makes it particularly hard to question the assumption, since you don’t realize that the assumption is even there.

Another reason that this assumption is so easily accepted is that we have a long history of modeling rational agents as expected utility maximizers, and for good reason: there are many coherence arguments that say that, given that you have preferences/goals, if you aren’t using probability theory and expected utility theory, then you can be taken advantage of. It’s easy to make the inference that a superintelligent agent must be rational, and therefore it must be an expected utility maximizer.

Because this assumption was so embedded in how I thought about the problem, I had trouble imagining how else to even consider the problem. I would guess this is true for at least some other people, so I want to summarize the counterargument, and list a few implications, in the hope that this makes the issue clearer.

Why goal-directed behavior may not be required

The main argument of this chapter is that it is not required that a superintelligent agent takes actions in pursuit of some goal. It is possible to write algorithms that select actions without doing a search over the actions and rating their consequences according to an explicitly specified simple function. There is no coherence argument that says that your agent must have preferences or goals; it is perfectly possible for the agent to take actions with no goal in mind simply because it was programmed to do so; this remains true even ...
On Abstract Systems
by Chris_Leong — 2019-01-06T23:41:52.563Z

We've all seen those abstract systems that are used for analysis like: Strategy = Ends + Ways + Means, Waterfall Model: Requirements, System Design, Implementation, Integration & Testing, Deployment, Maintenance, SWOT: Strengths, Weaknesses, Opportunities, Threats, ect.

Classes that teach these tend to be boring. Often you'll get three different business models thrown at you with minimal motivation. You won't be told about the environment this model evolved in, nor why the particular elements were included. Sometimes you'll get a concrete example, sometimes not, but if so, it's more likely to be a made up scenario rather than an example from someone's real life experience. It is even rarer that you will receive multiple such examples.

This is largely a result of people's bias towards explicit knowledge. But the explicit knowledge is in most cases simply one of many ways of carving up possibility space. Much more important is the implicit knowledge of how to apply the system and what situations it is helpful in. Strategy = Ends + Ways + Means becomes more applicable when you learn that it became dominate after the Vietnam War when it was felt that the loss was the result of attempting to achieve certain objectives without sufficient resources. So instead of just asking, "What are we trying to achieve?" and "How could we achieve it?", you also wanted to ask, "What resources are needed to achieve it?". Similarly, the Waterfall Model becomes more useful when you are told that many projects have not been as successful as desired because of a failure to consider requirements (ie. using a framework that requires Internet Explorer 10+ when some staff are stuck on Internet Explorer 9). Similarly, people often perform the cost analysis of projects for just the initial coding, without accounting for the maintenance cost. This is a system for making sure that you don't forget the obvious.

It would be even better if I could illustrate these with examples from my own experience, but this should still be sufficient to illustrate my point. I do think that these systems can be useful. But only when communicated in the right way.

This post was written with the support of the EA Hotel
Disadvantages of Card Rebalancing
by Zvi — 2019-01-06T23:30:08.255Z

Previously: Artifact Embraces Card Balance Changes, Card Collection and Ownership, Card Balance and Artifact, Card Rebalancing, Card Oversupply and Economic Considerations in Digital Card Games, Advantages of Card Rebalancing

This is the last post in this sequence, although we will doubtless return to related topics in the future.

IX. Non-Economic Disadvantages of Card Rebalancing

Last time, I explored eight reasons why card rebalancing was great. Now it is time to turn those reasons on their head, and see how disaster might strike.

There are three central reasons why I worry about card rebalancing. They are card and game economics, destruction of history, work and memory, a desire to ‘overbalance,’ and Goodhart’s Law.

Economics I’ve already considered. For considerations of card ownership, in-game economics and related matters, see Card Collection and Ownership and Card Rebalancing, Card Oversupply and Economic Considerations in Digital Card Games. I won’t consider such issues here.

Category 1: Destruction of history, work and memory.

1A. Players cannot rely on their knowledge of what cards do.

I know the exact abilities of thousands of Magic cards and hundreds of Artifact cards, plus thousands of cards in other games, both collectible and otherwise. I know the rules of thousands of board games, card games and video games.

Now imagine if those abilities and rules were constantly changing. If every turn and every action I had to worry that the rule wasn’t I thought it was, or the card had different abilities.

Magic players already have a taste of this with the grand creature type update. In that case, the problem was redoubled by not being able to know the change by reading the card. Players know that looking at a sufficiently old card’s printed creature type is not a good guide to its actual creature type. It can be obvious that Elvish Archers is now an Elf and an Archer, and most other changes are similarly guessable and logical.

The good news in a digital game is that one can always read the card, and various visual aids can be introduced to warn a player that a card has changed within some time frame, or since the last time the card was played against them. The other good news is that one periodically expects to face new cards, so facing new variations of existing cards is a reasonable punch to roll with.

Still, not being able to trust your knowledge of a game ...
Imitation learning considered unsafe?
by David Scott Krueger (formerly: capybaralet) — 2019-01-06T15:48:36.078Z

ETA: This post can basically be read as arguing that imitating human decisions, or any other outputs from an (approximate) planning process, seems especially likely to produce mesa-optimization, since a competent imitator should recover an (approximate) planning (i.e. optimization) process.

This post states an observation which I think a number of people have had, but which hasn't been written up (AFAIK). I find it one of the more troubling outstanding issues with a number of proposals for AI alignment.

1) Training a flexible model with a reasonable simplicity prior to imitate (e.g.) human decisions (e.g. via behavioral cloning) should presumably yield a good approximation of the process by which human judgments arise, which involves a planning process.

2) We shouldn't expect to learn exactly the correct process, though.

3) Therefore imitation learning might produce an AI which implements an unaligned planning process, which seems likely to have instrumental goals, and be dangerous.

Example: The human might be doing planning over a bounded horizon of time-steps, or with a bounded utility function, and the AI might infer a version of the planning process that doesn't bound horizon or utility.

Clarifying note: Imitating a human is just one example; the key feature of the human is that the process generating their decisions is (arguably) well-modeled as involving planning over a long horizon.

Counter-argument(s):

The human may have privileged access to context informing their decision; without that context, the solution may look very different
Mistakes in imitating the human may be relatively harmless; the approximation may be good enough
We can restrict the model family with the specific intention of preventing planning-like solutions

Overall, I have a significant amount of uncertainty about the significance of this issue, and I would like to see more thought regarding it.
Supervising strong learners
by amplifying weak experts
by paulfchristiano — 2019-01-06T07:00:58.680Z
Link: https://arxiv.org/pdf/1810.08575.pdf

Abstract

Many real world learning tasks involve complex or hard-to-specify objectives, and using an easier-to-specify proxy can lead to poor performance or misaligned behavior. One solution is to have humans provide a training signal by demonstrating or judging performance, but this approach fails if the task is too complicated for a human to directly evaluate. We propose Iterated Amplification, an alternative training strategy which progressively builds up a training signal for difficult problems by combining solutions to easier subproblems. Iterated Amplification is closely related to Expert Iteration (Anthony et al., 2017; Silver et al., 2017b), except that it uses no external reward function. We present results in algorithmic environments, showing that Iterated Amplification can efficiently learn complex behaviors.

Tomorrow's AI Alignment Forum sequences post will be 'AI safety without goal-directed behavior' by Rohin Shah, in the sequence on Value Learning.

The next post in this sequence on Iterated Amplification will be 'AlphaGo Zero and capability amplification', by Paul Christiano, on Tuesday 8th January.
Failures of UDT-AIXI, Part 1: Improper Randomizing
by Diffractor — 2019-01-06T03:53:03.563Z

While attempting to construct a UDT-like variant of AIXI, the algorithm I wrote down turned out to have a bunch of problems with it that were blocking attempts to prove that it had various nice properties. This will be the first post in a sequence about the various nonobvious things that go wrong when you try to make UDT-AIXI. (Check back here for more links)

Notation and Setup:

To begin with, the basic setting was one where the Turing machines in the environment had oracle access to the policy. In the first attempt, where O was the set of observations, and O∗ was the set of all finite strings of observations, policies had the type signature π:O∗→ΔA, and environments had the type signature O∗×π→ΔO, and functioned by having the option of making oracle calls to the policy with various observation strings to see what action was returned. If the policy randomizes on that input, then the action is simply drawn from the distribution π(→o).

Note that we're just using O∗ instead of the AIXI input type of (O×A)∗, this is because the actions that the agent sees on the input tape can just be seen as a sort of observation, like any other, and this generalization permits looking at cases where the agent may not have an internal sense of which action it took, or cases where the agent gets to peek at what it did in hypothetical situations that didn't happen.

Policy Search and Nash Equilibria:

Glossing over issues of nonhalting environments, we can reframe the UDT search for the best policy as the result of an infinite game, where each player is indexed with an observation string →o∈O∗, and responds with a (probability distribution over) move(s) in A. Each cell corresponds to a deterministic policy of type O∗→A, and the utility in that cell is the expected reward over the standard mixture of environments, when the environments are making oracle calls to that selected policy. All players conveniently have the same utility function, which makes things easier. The optimal deterministic policy is a Nash equilibrium, because any deviation from any of the players would result in an equal or lower expected utility.

This rephrases problems of selecting a policy, to an equilibrium selection problem, because even when all players have the same utility function, there can be inadequate equilibria. Consider a multiplayer game of stag hunt where rabbits are shared equally among the ...