trentmkelly/LessWrong-43k huggingface dataset:
Figuring out what Alice wants, part II
by Stuart_Armstrong — 2018-07-17T13:59:40.722Z

This post continues the analysis started in the previous post. Here I will present some examples of algorithms operating in certain environments, seeing how the model certain things, and using that to conclude facts about their preferences/goals/reward functions.

I'll be looking first at the Poker problem with unknown motivations presented here, and secondly at a variant of the Codenames game.

Different algorithms, same outputs, different goals

In the Poker example, we are unsure whether Alice wants to win the hand against Bob, for money, or lose the hand to get into Bob's good graces. She herself is unsure what Bob's cards are; Bob has been playing confidently, but there is only one card combination that would allow him to win.

Alice Poker algorithm1:Parameters: a, ratio of relative heuristic importance.2:Input: Alice_cards, board, Bob_behave3:P_win=(a)h_1(Alice_cards,board)+(1−a)h_2(Bob_behave)4:if P_win>0.55:return 'call'6:else7:return 'fold'8:end if

The inputs are Alice's own cards, the five cards on the board, and Bob's behaviour this hand. There are two heuristics called by the algorithm, h_1, which computes the probability of Alice winning by assuming Bob has a random hand, and h_2, which assesses the likelihood of Alice winning by looking at Bob's behaviour (at this point you should start worrying about the suggestive names and descriptions I'm giving to all these elements).

Now, in the situation we find ourselves, what we want to say is that if a is close to 1, then h_1 dominates, and P_win will be high. If a is close to 0, then it will be low, as h_2 dominates the expression. Since there is a > in line 4, we want to say that this Alice Poker algorithm is trying to win, but, depending on the value of a, has different beliefs about what action is likely to maximise expected money.

Similarly, if it were a < in line 4, we'd like to say that the Alice Poker algorithm wants to lose. And this, even though the (<,a=1) and (>,a=0) algorithms would both fold (while (<,a=0) and (>,a=1) would both call).

But this relies too much on the interpretation of what the terms mean ...
Figuring out what Alice wants, part I
by Stuart_Armstrong — 2018-07-17T13:59:35.395Z

This is a very preliminary two-part post sketching out the direction I'm taking my research now (second post here). I'm expecting and hoping that everything in here will get superseded quite quickly. This has obvious connections to classical machine intelligence research areas (such as interpretability). I'd be very grateful for any links with papers or people related to the ideas of this post.

The theory: model fragments

I've presented the theoretical argument for why we cannot deduce the preferences of an irrational agent, and a practical example of that difficulty. I'll be building on that example to illustrate some algorithms that produce the same actions, but where we nonetheless can feel confident deducing different preferences.

I've mentioned a few ideas for "normative assumptions": the assumptions that we, or an AI, could use to distinguish between different possible preferences even if they result in the same behaviour. I've mentioned things such as regret, humans stating their values with more or less truthfulness, human narratives, how we categorise our own emotions (those last three are in this post), or the structure of the human algorithm.

Those all seems rather add-hoc, but they are all trying to do the same thing: hone in on human judgement about rationality and preferences. But what is this judgement? This judgement is defined to be the internal models that humans use to assess situations. These models, about ourselves and about other humans, often agree with each other from one human to the next (for instance, most people agree that you're less rational when you're drunk).

Calling them models might be a bit of an exaggeration, though. We often only get a fragmentary or momentary piece of a model - "he's being silly", "she's angry", "you won't get a promotion with that attitude". These are called to mind, thought upon, and then swiftly dismissed.

So what we want to access, is the piece of the model that the human used to judge the situation. Now, these model fragments can often be contradictory, but we can deal with that problem later.

Then all the normative assumptions noted above are just ways of defining these model fragments, or accessing them (via emotion, truthful description, or regret). Regret is a particularly useful emotion, as it indicates...
How To Use Bureaucracies
by Samo Burja — 2018-07-17T08:10:02.190Z

This is an excerpt from the draft of

my upcoming book

on great founder theory. It was originally published on SamoBurja.com. You can

access the original here.

When we encounter unsavory features of reality, it can be tempting to look away. Instead, we should ask, “What purpose does this serve?” With this in mind, let’s look at bureaucracies. Some people fear bureaucracies; they fear “the Machine.” Others are bothered by the bureaucracies’ apparent dysfunction. With a better understanding of bureaucracies — what they are, why they’re here, and how they work — both of these responses evaporate, because the reality is this: bureaucracies aren’t altogether bad. In fact, bureaucracies can be incredibly useful.

What is a bureaucracy?

A bureaucracy is an automated system of people created to accomplish a goal. It’s a mech suit composed of people. The owner of a bureaucracy, if an owner exists, is the person who can effectively shape the bureaucracy. Bureaucrats are the people who are part of a bureaucracy (excluding the owner).

Not all organizations are bureaucracies. Most organizations are mixed — they have both bureaucratic and non-bureaucratic elements. The purpose of a bureaucracy is to save the time of a competent person. Put another way: to save time, some competent people will create a system that is meant to do exactly what they want — nothing more and nothing less. In particular, it’s necessary to create a bureaucracy when you are both (a) trying to do something that you do not have the capacity to do on your own, and (b) unable to find a competent, aligned person to handle the project for you. Bureaucracies ameliorate the problem of talent and alignment scarcity.

Bureaucrats are expected to act according to a script, or a set of procedures — and that’s it. Owners don’t trust that bureaucrats will be competent or aligned enough to act in line with the owner’s wishes of their own accord. Given this lack of trust, owners should be trying to disempower bureaucrats. Bureaucracies are built to align people and make them sufficiently competent by chaining them with rules. When bureaucracies deliberately restrict innovation, they are doing it for good reason.

Bureaucrats are meant to have only borrowed power (power that can easily be taken away)...
(AI alignment) Now is special
by Andrew Quinn — 2018-07-17T01:50:18.399Z

As an extremely conservative estimate: Humans have existed as a distinct species for over 100,000 years.

Of those 100,000 years, as an upper bound, maybe the last 10% of them can be classed as "civilization" times. (The actual number given by historians is closer to 6,000. Bear with me. I'm trying to communicate scale, here.)

And of those 10,000 years of civilization, you happened to have lived in the 150-or-so years where making tractable progress on AI alignment was possible.

And, of those 150-or-so years where that was possible, you happened to have lived in the same 30-or-so years where the same skills that correlate broadly to making headway on AI alignment can also be used to have a well-paying, enjoyable career, with relatively little risk.

Now is special. Now matters.

Hop to it.
Look Under the Light Post
by Gordon Seidoh Worley — 2018-07-16T22:19:03.435Z

There's a well-worn story you've probably heard before about looking for keys under a light post. If you've not heard it before:

A man leaves the bar after a long night of drinking. He stumbles all the way home only to find when he gets there that he doesn't have his keys. He knows he had his keys when he left the bar, so he must have dropped them on the way home. He goes out to look for them, but it's the night of a new moon and too dark to possibly see his keys except where there's artificial lighting, so he spends all his effort looking under the sole light post between his home and the bar. After a while a police officers stops to question him.

"Sir. Sir! What are you doing?" shouts the officer.

"Looking for my keys," says the drunk man.

"Did you drop them around here?"

"I don't know."

"Then why are you looking here?"

"Because this is where the light is."

The "broke" interpretation of the story is that the drunk man is stupid for looking for his keys under the light post because the location of his lost keys is unlikely to be correlated with where the light is good enough to search. The "woke" interpretation is that he's right to spend his time looking where the light is because he's more likely to find his keys if he looks in places where he has a chance to find them even if this makes finding his keys conditional on his having dropped his keys in such a location, thus increasing his expected probability of finding his keys since he has a very small chance of finding them in the dark.

I'm here to stretch a metaphor thin and give you the "bespoke" interpretation: many of us are searching for answers that are probably out in the dark, but we should look under the light post anyway both because that's where the light is and because different people have different light posts to look under who can call out to us if they find the answer.

We often face situations in real life analogous to looking for our keys in the dark while drunk. For example, maybe we want to know where to donate our money, what to do on vacation, or how to prevent existential catastrophe. In each case we're trying to find something (a charity, a decision, a solution) and have some ideas about where to look but don't have time or the ability to ...
Alignment Newsletter #15: 07/16/18
by Rohin Shah — 2018-07-16T16:10:03.390Z

Highlights

Feature-wise transformations

(Vincent Dumoulin et al): This Distill article is about transformations on features using FiLM (feature-wise linear modulation). A FiLM layer is used to "condition" a neural network on auxiliary information, which just means providing the input to the neural network in a way that it can use it effectively. This can be used to integrate multiple sources of information -- for example, in visual question answering (VQA), the main part of the network can be an image processing pipeline, and FiLM can be used to turn the natural language question about the image into a task representation and integrate it into the pipeline, and the full network can be trained end-to-end. The FiLM layer works by first using a subnetwork to turn the auxiliary information (such as the question in VQA) into a "task representation" (a new representation chosen by the neural network), which is then used as the parameters for an affine transformation of the features in the main pipeline. Importantly, each feature is treated independently of other features, so the FiLM layer can't create interactions between features. Yet, this still works well in many different contexts.

Since it is a Distill paper, it then goes into a ton of detail about lots of interesting details, such as how architectures in a variety of ML tasks can be thought of as FiLM, how FiLM relates to other ideas such as attention, how we can often interpolate between different auxiliary information by taking a weighted combination of the corresponding task information, how conditioning through concatenation is equivalent to FiLM with only a bias and no scaling, etc.

My opinion: I really enjoy Distill articles, they are consistently far more readable and understandable than typical papers (or even blog posts), even without including the interactive visualizations. This article is no exception. I didn't have particularly strong opinions on how to condition neural nets before, but now I think I will think about FiLM and how it could apply.

Troubling Trends in ML Scholarship

(Zachary C. Lipton and Jacob Steinhardt): This is a position paper arguing that ML research would benefit from more rigor, as part of the ICML debates. It identifies four trends in ML papers. First, papers often don't make clear whether they are providing an (authoritative) explanation or...
Compact vs. Wide Models
by Vaniver — 2018-07-16T04:09:10.075Z

I recently found a label I like for one of the obstacles to AI alignment that I think about frequently. I think of it as "compact models" vs. "wide models," which I'll try to define by suggestive naming, followed by example. It seems easy to verify that compact models have particular properties, and thus we hope it would be easy to verify that they have desirable ones, but it seems difficult to do the same verification for wide models. Unfortunately and importantly, it seems to me like the best models of human values and decision-making are wide, and so ensuring that a system is compatible with humans inherits the difficulties of reasoning about wide models, even if the artificial system is not itself wide.

This distinction is related to the "neats vs. scruffies" debate of old, but is (I hope) a somewhat different take. To some extent, the 'compact model' maps onto the 'neat' approach, whereas the 'wide model' maps closer to the synthesis of the neat and scruffy approach.

Some examples of compact models:

A Turing machine is a compact model of computation: easy to describe, easy to reason about, and easy to prove things about. But it's also somewhat detached from actual computers and programming.

The VNM-utility formulation is a compact model of preferences conjugate to decision-making under uncertainty. It is, in many respects, the simplest consequentialist to reason about, and likewise the argument for it being the ideal form is simple. But when trying to descriptively model actual human behavior with utility functions, one needs to include lots of wrinkles, not just in the utility function but also for the decision procedure and model of uncertainty.

In numerical optimization, linear programming refers to methods that solve constrained optimization problems where all variables are real numbers and the cost function and all constraints are linear in those variables. This allows for an extremely slick solution called the 'simplex method,' and both its mechanics and the argument for its correctness can be easily grasped. Almost all of the difficulty of applying it in practice has been compartmentalized to getting right the correspondence between the mathematical model being optimized and the external world where the solution is put into place.

To be a wide model, it still has to have a ...
Probabilistic decision-making as an anxiety-reduction technique
by RationallyDense — 2018-07-16T03:51:49.970Z

One problem I run into when making decisions that impact my life substantially (say, making a big purchase) is that I spend a lot of time staring at decision becoming increasingly anxious. Usually, this is accompanied by only very minor amounts of information being acquired.

There is a well-known solution to this: flip a coin.

Flipping a coin is nice in that it allows you to substantially compress the amount of time during which you are anxious. However, it is not without its flaws.

1. Flipping a coin does not allow you choose among more than two options. That's often enough, but more often than not, it's not enough.

2. Flipping a coin does not allow you to include any of the information that is pertinent to the decision you are making.

So while flipping a coin is nice because it allows you to skip lengthy pointless fretting, it also does not really lead to an optimal outcome. Now of course, if you're at that point, you don't already know the optimal outcome. Otherwise you'd just do that. Right? If you know what you're going to do at the end of the fretting, just do it. The problem is that you don't know.

So I'm proposing the following approach:

1. Estimate the probability that you will end up selecting each of the options.

2. Pick one of the outcomes randomly in accordance with the probability you assigned to it.

e.g. I'm trying to purchase pants. Either black or navy. They cost a few hundreds of dollars so it is an anxiety-generating purchase. I've analyzed the pros-and-cons of the two pairs of pants and at this point, I'm just worrying about whether or not I'm making the best choice.

Step 1. I assign a 0.4 probability to getting the black pants. A 0.3 probability that I will get the navy pants. And a 0.3 probability that I will abandon the purchase.

Step 2. I compute the intervals: 0-0.4 is black, 0.4-0.7 is navy, 0.7-1.0 is no purchase.

Step 3. I generate a random number. I just search Google for "random number between 1 and 100" and divide the result by 100.

Step 4. Compare the result to the intervals to figure out what I just decided.

Step 5. Implement the decision.

I figure I can't be the first person to have thought of this. If you know of prior art on this, let me know.
Buridan's ass in coordination games
by jessicata — 2018-07-16T02:51:30.561Z

Consider a simple coordination game. In this game, two players (player 1 and player 2) each simultaneously choose an action, X or Y. If they both choose X, they both get 1 utility. If they both choose Y, they both get uy utility for some 0≤uy≤2 known to both players. If they choose different actions, they both get 0 utility. Which action should they each choose? Assume they get to communicate before knowing uy, but not after knowing uy.

An optimal policy pair is for each to pick X if uy<1, and Y otherwise. Unfortunately, this policy pair can break down in the presence of even a small amount of noise. Assume neither player observes uy, but instead each receives an independent observation Vi (player 1 sees V1, player 2 sees V2), each of which is drawn uniformly from the range [uy−ϵ,uy+ϵ] for some small ϵ>0. If both players follow the policy of choosing X if and only if their observation of uy is less than 1, then if uy is very close to 1, there is a significant likelihood that one player will receive an observation less than 1, while the other will receive one greater than 1. Thus, they have a significant chance of choosing different actions and receiving 0 utility. This issue is essentially the same as the one faced by Buridan's ass: both options are equally good, so there is no way to effectively decide between them.

In fact, as I will prove:

Claim 1: No pair of policies in which the players select their actions independently given their observations can simultaneously guarantee an expected utility greater than max(1,uy)−0.5 for all uy∈[0,2].

But things change when a shared source of randomness is allowed. Then, as I will also prove:

Claim 2: Suppose that, after observing their observation of uy, each player also gets to observe a random number R∼Uniform([0,1]). Then there is a pair of policies that achieves expected utility at least max(1,uy)−2√ϵ−ϵ regardless of uy .

This solution method extends to arbitrary cooperative normal-form games where players observe a perturbed version of the true utility function. This is shown in the appendix.

Why is this important? I expect progress in decision theory to come from studying very simple decision problems like this one. If it is possible to show that any solution to some simple problem has certain properties, then this usefully constrains the design space of possible decision theories. In this case, the...
An optimistic explanation of the outrage epidemic
by chaosmage — 2018-07-15T14:35:26.357Z

Lots of people agree the internet is full of outrage content these days. I think this is so obvious I'm not even going to post sources that say it.

Why this is the case isn't completely clear. I count at least four common theories which of course aren't exclusive:

media companies are shrinking, so they're desperate for clicks and have stumbled on outrage as a good click getter
it is Twitter's fault, the structure there incentivizes outrage
increasing political polarization has infected the internet
coddled youth, used to carefully curated experiences, can't handle disagreement like they should

All of these might be contributing factors, but I want to point out another that I haven't seen mentioned.

The strength of an outrage response depends not just on how outrageous an act is, but also on other factors, how recent it is for example, and how close we are to the victim that something outrageous is done to. And in particular, it also depends on how close the outrageous person (the perpetrator) is.

Imagined examples, where we hold the act, victim and recentness constant and vary only the closeness of the perpetrator:

If somebody else's child steals from a shop I don't like it. But if my own child steals the same thing from the same shop I get really angry.
When the president of Russia tells obvious lies to the media that's bad but not big news. When the president of the US does the same thing I really worry and it is a huge deal. (I'm not a US citizen but an ally.)
If I think of people in South Sudan doing female genital mutilation on their daughters, I'm a bit outraged. But if someone in my city did that, I'd be intensely and obviously furious - and more so if it was in my street - and even more so if it was in my apartment building, even if I never met the daughter.
If my wife were to make a stupid mistake that costs us a hundred bucks, I'd be angry. But if I (the closest possible person) made the exact same mistake, I'd be much more angry at myself.

I can't think of any counterexamples.

For each of these examples I could find more situation-specific explanations why I'd be more outraged by one than by the other. But it is more parsimonious to think the explanation is always in the degree of closeness to the perpetrator.

It seems to me like there's a one-dimensional rank order (or po...
Announcement: AI alignment prize round 3 winners and next round
by cousin_it — 2018-07-15T07:40:20.507Z

We (Zvi Mowshowitz and Vladimir Slepnev) are happy to announce the results of the third round of the

AI Alignment Prize

, funded by Paul Christiano. From April 15 to June 30 we received entries from 12 participants, and are awarding $10,000 to two winners.

We are also announcing the fourth round of the prize, which will run until December 31 of this year under slightly different rules. More details below.

The winners

First prize of $7,500 goes to Vanessa Kosoy for

The Learning-Theoretic AI Alignment Research Agenda

. We feel this is much more accessible than previous writing on this topic, and gives a lot of promising ideas for future research. Most importantly, it explains why she is working on the problems she’s working on, in concrete enough ways to encourage productive debate and disagreement.

Second prize of $2,500 goes to Alexander Turner for the posts

Worrying About the Vase: Whitelisting

and

Overcoming Clinginess in Impact Measures

. We are especially happy with the amount of good discussion these posts generated.

We will contact each winner by email to arrange transfer of money. Many thanks to everyone else who sent in their work!

The next round

We are now announcing the fourth round of the AI Alignment Prize. Due the drop in number of entries, we feel that 2.5 months might be too short, so this round will run until end of this year.

We are looking for technical, philosophical and strategic ideas for AI alignment, posted publicly between July 15 and December 31, 2018. You can submit links to entries by leaving a comment below, or by email to apply@ai-alignment.com. We will try to give feedback on all early entries to allow improvement. Another change from previous rounds is that we ask each participant to submit only one entry (though possibly in multiple parts), rather than a list of several entries on different topics.

The minimum prize pool will again be $10,000, with a minimum first prize of $5,000.

Thank you!