trentmkelly/LessWrong-43k huggingface dataset:
The Alignment Newsletter #11: 06/18/18
by Rohin Shah — 2018-06-18T16:00:46.985Z

Turns out the survey link in the last email was broken, sorry about that and thanks to everyone who reported it to me. Here's the correct

link

.

Highlights

Learning to Follow Language Instructions with Adversarial Reward Induction

(Dzmitry Bahdanau et al): Adversarial Goal-Induced Learning from Examples (AGILE) is a way of training an agent to follow instructions. The authors consider a 5x5 gridworld environment with colored shapes that the agent can manipulate. The agent is given an instruction in a structured domain-specific language. Each instruction can correspond to many goal states -- for example, the instruction corresponding to "red square south of the blue circle" has many different goal states, since only the relative orientation of the shapes matters, not their absolute positions.

The key idea is to learn two things simultaneously -- an encoding of what the agent needs to do, and a policy that encodes how to do it, and to use these two modules to train each other. The "what" is encoded by a discriminator that can classify (state, instruction) pairs as either being a correct goal state or not, and the "how" is encoded by a policy. They assume they have some human-annotated goal states for instructions. The discriminator is then trained with supervised learning, where the positive examples are the human-annotated goal states, and the negative examples are states that the policy achieves during training (which are usually failures). The policy is trained using A3C with a reward function that is 1 if the discriminator says the state is more likely than not to be a goal state, and 0 otherwise. Of course, if the policy actually achieves the goal state, there is no way of knowing this apart from the discriminator -- so by default all of the states that the policy achieves (including goal states) are treated as negative examples for the dsicriminator. This leads to the discriminator getting slightly worse over time as the policy becomes better, since it is incorrectly told that certain states are not goal states. To fix this issue, the authors drop the top 25% of states achieved by the policy that have the highest probability of being a goal state (according to the discriminator).

The authors compare AGILE against A3C with the true reward function (i.e. the reward function implied by a perfect discriminator) and found that AG...
Why Destructive Value Capture?
by Zvi — 2018-06-18T12:20:00.407Z

Previously: Front Row Center

I got a lot of push-back from suggesting that there was a way for theaters to improve their customer experience and value proposition at low cost (get rid of the seats that are so close to the screen they cause neck strain), and that theaters should do that.

The push-back didn’t argue that the method wouldn’t improve the customer experience at low cost. There were a few who suggested an alternate high-cost solution that improves the experience more (use high-quality and assigned seating at a substantially higher price point), and which some places have implemented. No one argued that, where the higher-cost solution didn’t make sense. my incremental suggestion wouldn’t improve the customer experience versus status quo, at relatively low cost.

They also didn’t raise the reasonable argument that getting people to do things at all, especially slightly non-standard things that might look bad on superficial metrics during the pitch meeting, is hard. People don’t think about things, they don’t do things, they don’t optimize, and so on. One could reasonably argue this isn’t worth the effort.

Instead, everyone argued that, unless they were forced to do so, theaters shouldn’t implement the suggestion. Because it would cost them money – they couldn’t sell those few terrible seats, and forcing people to come early increases ad and concession revenue.

That’s interesting. And weird.

The proposition creates value. One comment from Quixote estimates $1.67 in customer time-value is saved in exchange for the loss of $0.10 in ad revenue.

The proposition improves the customer experience. It generates movie-going habits, loyalty and goodwill.

Not implementing the proposition is a destructive value capture. In order to get a little revenue, an order of magnitude more value is destroyed.

Destructive value capture is normal. In order to capture value, some value is typically destroyed. But when you’re destroying most of the value you withdraw from the system, you should be suspicious mistakes are being made. At a minimum, it’s worth asking on a deeper level why this is happening. What could justify it? What failure mode are we in? How does it come to be, why does it persist, is there a way we can solve it or minimize it? We shouldn’t shrug and mutter something about c...
Privacy: Defining Yourself
by Lulie — 2018-06-18T02:15:58.890Z

A girl needs a room of her own.

— paraphrasing Virginia Woolf

It's important to control the amount of information people have about you, especially for creative tasks or in areas you expect you might want to change.

Defining yourself is risky, because then it becomes hard to change when everyone is relating to you as that. (Also for Keeping Your Identity Small reasons.)

Privacy helps avoid being defined as something, and it helps with managing how people define you when this can't be avoided.

A common idea is that other people judge you, assess you in their mind according to their standards, and there's not much you can do about how they see you.

In reality, people usually judge you according to the standards you give them.

People are complex — it's a lot of work to form accurate assessments. So when people tell us how to think of them, we generally believe them. (Often not consciously/intentionally, but what we think of ourselves is usually communicated in how we speak.)

All interactions require having a view of the other person and what they're about.

You need this to know what to expect from them, the role each of you play in each other's lives, and what they expect from you — how the interaction will go, what it will be based around, what's in the scope of the interaction and what's outside it.

So, it's a good thing we have some control over this!

We can have an idea of what we want out of the interaction and set the stage accordingly. If someone gets a false impression of us, we can intervene and correct it.

This has two difficulties.

One is that intervening when someone has a false impression takes effort. If someone has a lot of misconceptions about you, or if a lot of people have misconceptions, that can be arbitrarily difficult.

(This is related to "You can't please everyone": even if you could please anyone, it may be an intractably large problem to give every single person who knows of you a given impression. In practice, our impressions of people are never error-free.)

The other difficulty is: people change. Sometimes you no longer want the things your interaction used to be about. When people knew you as one kind of person, and you change, you have to manage their expectations — help them update how to understand you and your interaction with them.

If you're changing fast, this can be a huge...
In Defense of Ambiguous Problems
by Chris_Leong — 2018-06-17T07:40:58.551Z

Yesterday, I posted the Curious Prisoner Puzzle. The puzzle was structured to attempt to fool people into answering 1/2, which is possible, but certainly not unambiguously true. Any answer has to acknowledge the ambiguity, but at least in my opinion, the best way to reinterpret the problem to ensure that there is a single, unambiguous answer is as follows: Assume that you are told the statement when it is true and nothing is otherwise. Then, the answer is 1/3. Well done to all the commentators who quickly managed to figure all this out.

So why did I post an ambiguous problem?:

If the point of these puzzles is to teach a lesson, dodging an incorrect answer suffices. I actually couldn't see a way of making the problem unambiguous without giving the game away with regarding the answer not being 1/2
You can't assume that every problem you'll try to solve will turn out unambiguous (particularly if you study philosophy), so it's useful to be able to recognise when this is the case. If you always practise with unambiguous problems, then you are biasing yourself to assume problems are unambiguous, which is why people often struggle with this
This problem demonstrates the importance of counterfactuals in terms of probability. In fact, an ambiguous problem is probably the best way to demonstrate that such problems are ambiguous without defining the counterfactuals. In particular, it shines a light on problems like the classic, "What is the chance that two kids were both born on the same day of the week if at least one of them was born on a Tuesday?"
Further ambiguous problems can reveal the implicit assumptions made elsewhere. Suppose we have are four boxes: red, green, blue and white. One is picked at random to contain a prize, then the host peeks and tells you that it isn't in the white box. What is the chance that it is in the blue box? This problem is mathematically the same, yet we generally don't say that it is ambiguous even though the counterfactual hasn't been specified. That's probably because the different natural interpretations of the host's algorithm all give the same answer of 1/3, but this is an important realisation because it affects a massive proportion of probability problems:

The host randomly picks a color that doesn't contain the prize and tells you that it isn't in that box
The host rand

...
Fundamentals of Formalisation Level 4: Formal Semantics Basics
by philip_b — 2018-06-16T19:09:16.042Z

Followup to Fundamentals of Formalisation Level 3: Set Theoretic Relations and Enumerability. First Post.

The big ideas:

Formal Semantics

Model

To move to the next level you need to be able to:

Evaluate the truth value of logical sentences in a given model.

Build models for a set of logical sentences and then use those models to deduce information about the sentences.

Why this is important:

An AI built on a formal system will reason based on some sort of proof and model theory. The former gives its methods of proof (which you’ll learn in the next chapter), and the latter its semantics.

This level will give you your first sense of what models of logical sentences look like.

You can find the lesson in our ihatestatistics course.

For every lesson you have 2 options: do the whole thing, or skip to the questions and "challenges" in the end. The latter option is for those people that suspect they already know the subject. It serves as a means of verifying or falsifying this hypothesis.

Good luck!
How many philosophers accept the orthogonality thesis ? Evidence from the PhilPapers survey
by Paperclip Minimizer — 2018-06-16T12:11:49.892Z

The orthogonality thesis and its relation to existing meta-ethical debates

In the field of AI alignment theory, the orthogonality thesis asserts that there can exist arbitrarily intelligent agents pursuing any kind of motivation. The reverse thesis, that we may call the heterogonality thesis, asserts that, with enough intelligence, any possible agent would pursue only one set of motivations.

In the field of meta-ethics, moral internalism asserts that any possible agent who hold a moral judgment is motivated to act on this judgment. For example, according to moral internalism, any agent who hold that one ought to donate 10% of one's income to charity is motivated to do so.

Also in the field of meta-ethics, moral realism asserts that some moral judgments are objectively correct. This is a form of moral cognitivism, which that moral judgments are factual statements that can be objectively correct or incorrect (anti-realist cognitivism is error theory, which asserts that all moral judgments are incorrect).

It's easy to see that the heterogonality thesis is moral internalism plus moral realism. A moral realist would say that, with enough intelligence, any possible agent would discover objective morality and hold only one set of moral judgments that is objectively correct. Therefore, a moral realist who is also a moral internalist would support the heterogonality thesis by saying that this mean that, with enough intelligence, any possible agent would be motivated to act by only one set of moral judgments, and thus would pursue only one set of motivations.

This is why, even though the orthogonality thesis is a recent concept that is only known by the small circles of AI alignment theorists (and I had to invent the term for its negation by myself), we can try to estimate how many philosophers accept the orthogonality thesis.

The PhilPapers survey

The PhilPapers Survey was a survey of professional philosophers and others on their philosophical views, carried out in November 2009. The Survey was taken by 3226 respondents, including 1803 philosophy faculty members and/or PhDs and 829 philosophy graduate students.

It included three questions on meta-ethics: whether one accept moral realism or moral anti-realism, whether one accept moral internalism or moral externalism, and whether one accept moral cognitivism or moral non-cognitivism. (There was also a question on normative...
Worrying about the Vase: Whitelisting
by TurnTrout — 2018-06-16T02:17:08.890Z

Suppose a designer wants an RL agent to achieve some goal, like moving a box from one side of a room to the other. Sometimes the most effective way to achieve the goal involves doing something unrelated and destructive to the rest of the environment, like knocking over a vase of water that is in its path. If the agent is given a reward only for moving the box, it will probably knock over the vase.

Amodei et al., Concrete Problems in AI Safety

Side effect avoidance is a major open problem in AI safety. I present a robust, transferable, easily- and more safely-trainable, partially reward hacking-resistant impact measure.

TurnTrout, Worrying about the Vase: Whitelisting

An impact measure is a means by which change in the world may be evaluated and penalized; such a measure is not a replacement for a utility function, but rather an additional precaution thus overlaid.

While I'm fairly confident that whitelisting contributes meaningfully to short- and mid-term AI safety, I remain skeptical of its robustness to scale. Should several challenges be overcome, whitelisting may indeed be helpful for excluding swathes of unfriendly AIs from the outcome space.1 Furthermore, the approach allows easy shaping of agent behavior in a wide range of situations.

Segments of this post are lifted from my paper, whose latest revision may be found here; for Python code, look no further than this repository. For brevity, some relevant details are omitted.

Summary

Be careful what you wish for.

In effect, side effect avoidance aims to decrease how careful we have to be with our wishes. For example, asking for help filling a cauldron with water shouldn't result in this:

However, we just can't enumerate all the bad things that the agent could do. How do we avoid these extreme over-optimizations robustly?

Several impact measures have been proposed, including state distance, which we could define as, say, total particle displacement. This could be measured either naively (with respect to the original state) or counterfactually (with respect to the expected outcome had the agent taken no action).

These approaches have some problems:

Making up for bad things it prevents with other negative side effects. Imagine an agent which cures cancer, yet kills an equal number of people to keep overall impact low.
Not being customizable before deployment.
Not being adaptable after deployment.
Not bein

...
The Curious Prisoner Puzzle
by Chris_Leong — 2018-06-16T00:40:15.034Z

Here's an interesting riddle that is more complicated than it looks:

You wake up locked inside a room with no window. You know your captors have four facilities in the following locations: a Vulcan Mountain, a Vulcan Desert, an Earth Mountain, an Earth Desert. They flip a coin to decide which planet to send you to, then they flip a coin to decide which facility on that planet to use.

Wanting to know where you are, you try to get some information out of the guard. He refuses at first, but eventually he offers the following: "If you are on Vulcan, you are in the Mountain". What are the chances that you are in the Vulcan Mountain facility?

(You can assume that the guard is telling the truth and not trying to intentionally manipulate the situation)
Geoffrey Miller on Effective Altruism and Rationality
by Jacob Falkovich — 2018-06-15T17:05:08.827Z
Link: https://putanumonit.com/2018/06/13/miller-1-ea-rationality/

The first part of an extensive interview with Dr. Geoffrey Miller.

Topics include:

1. Why we should send a hit squad to Russia.

2. How to make effective altruism sexy.

3. Why he doesn't eat anchovies.
Aligned AI May Depend on Moral Facts
by Gordon Seidoh Worley — 2018-06-15T01:33:36.364Z

Thanks to feedback on my recent post I've been reconsidering some of my thinking about how to address meta-ethical uncertainty as it affect alignment. I've run into something unexpected, and I'm curious to see what others think. To quote the key paragraph from my current draft:

A further issue with assuming moral facts do not exist is that it's unclear how to resolve value conflicts in the absence of assumptions about value norms. Certainly any policy can be used to resolve value conflicts for any reason, but the choice of policy by an AGI we consider aligned would still count as a normative assumption about values in practice if not in principle to the extent the AGI is considered to be aligning its behavior with human interests. Thus we are forced to assume moral facts exist in some form if alignment is to be solvable because otherwise we would have no way of deciding how to resolve value conflicts and could not construct an AGI we would consider aligned.

This is not to say we must assume moral realism, cognitivism, or any other particular position about the nature of moral facts, merely accept that moral facts exist since if they don't alignment is impossible in principle since an AGI will have no way to choose what to do when human values conflict. We could perhaps in practice build aligned AGI without doing this by just picking norms based on what someone preferred, but this seems to violate the spirit of the intention behind "alignment" to be for all moral agents and not just one particular human (to be fair addressing just one human is hard enough!) and not just that particular human's current preference that would not necessarily be endorsed upon reflective equilibrium (but the idea that preferences under reflective equilibrium are better is itself a normative assumption of the sort we would be hard pressed to make if we did not assume the existence of moral facts!).

Also note that all of this is made harder by not having a precise definition of alignment I can lean on. Yes, I wrote one, but it depends in part on results I'm considering formally in this paper so I can't use it.

I'm still thinking this through and updating on feedback I received on my first draft, but this seems like a significant enough departure from my earlier line of thinking to solicit additional feedback to consider issues I may...
SIAM Lecture: How Paradoxes Shape Mathematics and Give Us Self-Verifying Computer Programs
by ldsrrs — 2018-06-14T20:58:20.583Z
Link: http://meetings.siam.org/sess/dsp_programsess.cfm?SESSIONCODE=65101

Abstract:

A paradox is a seeming contradiction. The liar's paradox is one of the best known: "This statement is a false." If the statement is true, then it is false; if it is false, then it is true.

Paradoxes can be so amusing that we might think that paradoxes are nothing more than a game. However, paradoxes triggered a crisis in math a century ago when a paradox similar to the barber paradox was found: a barber named Bertie shaves exactly those who do not shave themselves. Does Bertie shave himself? If he does, then he doesn't; if he doesn't, then he does.

Other clever paradoxes show us the disturbing limits of computation and mathematics. These results are mathematical bombshells.

Today, we design computer programs that check that other computers programs have no bugs. Can computer programs be fed into themselves to check their own correctness? Or does paradox stop us in our tracks? And can we know that beneficial artificial intelligence will not turn evil when it starts to modify its own computer code?
We Agree: Speeches All Around! 
by SebastianG  — 2018-06-14T17:53:23.918Z

In the Catalan autobiography of James I Llibre dels Fets, King James often describes the advice given to him by different nobles and princes of the Church (read bishops). Oftentimes they disagree; sometimes he turns out right, and sometimes they turn out the wiser counsellors. Scholars often regard this frequent decision-making dialogue as evidence that James wanted not only to give an account of the great accomplishments of his life, but also provide insight for future kings and ministers of Aragon-Catalonia. There is much to say about the nature of this advice, the strategic and tactical reasoning, the difficulty of passing down rational statesmanship, and interrogation into just how “rational” this statesmanship actually was.

I am not going to focus on those issues. Instead, I want to bring to light a common knowledge dynamic I noticed in this book that resonated in my daily life.

My day job requires a lot of meetings. Oftentimes in these meetings my colleagues and I will hit on an agreed course of action, but then instead of saying, “We are agreed. Let’s go!” We will continue talking ourselves into the decision. Once a decision has been reached, each person inexplicably waxes poetic about their own reason for why they believe this is a good or right decision. This happens quite frequently, I do not think anyone recognizes it as weird. To be clear, this is not part of some in-house “Guideline For Decision-Making”; it is a spontaneous event of human interaction.

Up until this week, I thought this exercise was either an attempt to cover up uncertainty or a waste of time. But perhaps there is some utility here. Is this practice a way creating more agreeance? Congratulating ourselves on being in charge? What’s the deal? Is it a way of rebuilding bonds that may have been strained over the course of discussion? Or is it just a ‘Midwestern USA' thing?

James I helped me see the light. Before the invasion of the island of Mallorca, the Corts and councils convened to decide whether to invade. Into the mouths of a noble merchant, a general, a landed aristocrat, and a bishop additional words of approval came after they had already decided to launch this campaign. Since the campaign had already been approved in prior discussion by leading parties, why do they need more words of approval again after the decision has been made?

I think the answer is that although thes...