trentmkelly/LessWrong-43k huggingface dataset:
Iterated Distillation and Amplification
by Ajeya Cotra — 2018-11-30T04:47:14.460Z

This is a guest post summarizing Paul Christiano’s proposed scheme for training machine learning systems that can be robustly aligned to complex and fuzzy values, which I call Iterated Distillation and Amplification (IDA) here. IDA is notably similar to AlphaGoZero and expert iteration.

The hope is that if we use IDA to train each learned component of an AI then the overall AI will remain aligned with the user’s interests while achieving state of the art performance at runtime — provided that any non-learned components such as search or logic are also built to preserve alignment and maintain runtime performance. This document gives a high-level outline of IDA.

Motivation: The alignment/capabilities tradeoff

Assume that we want to train a learner A to perform some complex fuzzy task, e.g. “Be a good personal assistant.” Assume that A is capable of learning to perform the task at a superhuman level — that is, if we could perfectly specify a “personal assistant” objective function and trained A to maximize it, then A would become a far better personal assistant than any human.

There is a spectrum of possibilities for how we might train A to do this task. On one end, there are techniques which allow the learner to discover powerful, novel policies that improve upon human capabilities:

Broad reinforcement learning: As A takes actions in the world, we give it a relatively sparse reward signal based on how satisfied or dissatisfied we are with the eventual consequences. We then allow A to optimize for the expected sum of its future rewards
Broad inverse reinforcement learning: A attempts to infer our deep long-term values from our actions, perhaps using a sophisticated model of human psychology and irrationality to select which of many possible extrapolations is correct.

However, it is difficult to specify a broad objective that captures everything we care about, so in practice A will be optimizing for some proxy that is not completely aligned with our interests. Even if this proxy objective is “almost” right, its optimum could be disastrous according to our true values.

On the other end, there are techniques that try to narrowly emulate human judgments:

Imitation learning: We could train A to exactly mimic how an expertwould do the task, e.g. by training it to fool a discriminative model trying to tell apart A’s actions from the human expert’s actions.
Narrow inverse reinfo

...
Moving Factward
by Tyrrell_McAllister — 2018-11-29T05:54:28.877Z

In the legendarium of J.R.R. Tolkien, the land of the gods is known as "The Uttermost West." For the world was originally created flat, and the gods took the westernmost region of this flat world for their dwelling place.
On our globe, of course, there is no westernmost point. And yet it is still the case that, at each position on the equator, some direction is objectively "westward". The objectivity of "westward" doesn't assume that there is some ultimate West by which the west-ness of all other positions is measured.
Analogously, there is no such thing as a "bare uninterpreted fact". "Just the facts" is not a realizable ideal.
And yet we can still recognize when one account of a situation is more "factish" than another. We can see that the second account is more of an interpretation compared to the relatively factish features given in the first account. The more-factish account is never the ultimate and unvarnished truth. Likely no coherent sense could be made of that ideal. Nonetheless, from wherever we stand, we can always "move factward".[1]

Footnote

[1] ETA: Said Achmiz points out that many features of "westward" don't apply to "factward". Analogies typically assert a similarity between only some, not all, aspects of the two analogous situations. But maybe the other aspects of "westward" are so salient that they interfere with the analogy.
Formal Open Problem in Decision Theory
by Scott Garrabrant — 2018-11-29T03:25:46.134Z

(This post was originally published on March 31st 2017, and has been brought forwarded as part of the AI Alignment Forum launch sequence on fixed points.)

In this post, I present a new formal open problem. A positive answer would be valuable for decision theory research. A negative answer would be helpful, mostly for figuring out what is the closest we can get to a positive answer. I also give some motivation for the problem, and some partial progress.

Open Problem: Does there exist a topological space X (in some convenient category of topological spaces) such that there exists a continuous surjection from X to the space [0,1]X (of continuous functions from X to [0,1])?

Motivation:

Topological Naturalized Agents: Consider an agent who makes some observations and then takes an action. For simplicity, we assume there are only two possible actions, A and B. We also assume that the agent can randomize, so we can think of this agent as outputting a real number in [0,1], representing its probability of taking action A.

Thus, we can think of an agent as having a policy which is a function from the space Y of possible observations to [0,1]. We will require that our agent behaves continuously as a function of its observations, so we will think of the space of all possible policies as the space of continuous functions from Y to [0,1], denoted [0,1]Y.

We will let X denote the space of all possible agents, and we will have a function f:X→[0,1]Y which takes in an agent, and outputs that agent's policy.

Now, consider what happens when there are other agents in the environment. For simplicity, we will assume that our agent observes one other agent, and makes no other observations. Thus, we want to consider the case where Y=X, so f:X→[0,1]X.

We want f to be continuous, because we want a small change in an agent to correspond to a small change in the agent's policy. This is particularly important since other agents will be implementing continuous functions on agents, and we would like any continuous function on policies to be able to be considered valid continuous function on agents.

We also want f to be surjective. This means that our space of agents is sufficiently rich that for any possible continuous policy, there is an agent in our space that implements that policy.

In order to meet all these criteria simultaneously, we need a space X of agents, and a continuous surjection f:X→

...
Reflective oracles as a solution to the converse Lawvere problem
by SamEisenstat — 2018-11-29T03:23:51.180Z

(This post was originally published on Nov 30th 2017, and is 1 of 4 posts brought forwarded today as part of the AI Alignment Forum launch sequence on fixed points.)

1 Introduction

Before the work of Turing, one could justifiably be skeptical of the idea
of a universal computable function.
After all, there is no computable function f:N×N→N such that for all computable g:N→N there is some index ig such that f(ig,n)=g(n) for all n.
If there were, we could pick g(n)=f(n,n)+1, and then
g(ig)=f(ig,ig)+1=g(ig)+1,
a contradiction.
Of course, universal Turing machines don't run into this obstacle; as Gödel
put it, "By a kind of miracle it is not necessary to distinguish orders, and the
diagonal procedure does not lead outside the defined notion." [1]

The miracle of Turing machines is that there is a partial computable function
f:N×N→N∪{⊥} such that for all partial computable g:N→N∪{⊥} there is an index i such that f(i,n)=g(n) for all n.
Here, we look at a different "miracle", that of reflective oracles [2,3].
As we will see in Theorem 1, given a reflective oracle O, there is a (stochastic) O-computable function f:N×N→{0,1} such that for any (stochastic) O-computable function g:N→{0,1}, there is some index i such that f(i,n) and g(n) have the same distribution for all n.
This existence theorem seems to skirt even closer to the contradiction
mentioned above.

We use this idea to answer "in spirit" the converse Lawvere problem posed in [4]. These methods also generalize to prove a similar analogue of the ubiquitous converse Lawvere problem from [5]. The original questions, stated in terms of topology, remain open, but I find that the model proposed here, using computability, is equally satisfying from the point of view of studying reflective agents. Those references can be consulted for more motivation on these problems from the perspective of reflective agency.

Section 3 proves the main lemma, and proves the converse Lawvere theorem for reflective oracles. In section 4, we use that to give a (circular) proof of Brouwer's fixed point theorem, as mentioned in [4]. In section 5, we prove the ubiquitous converse Lawvere theorem for reflective oracles.

2 Definitions

For any measurable space X, the set of probability measures on X is denoted ΔX.

A probabilistic oracle is a map N→[0,1].
A probabilistic oracle machine is a probabilistic Turing machine that can query a probabilistic

...
The Ubiquitous Converse Lawvere Problem
by Scott Garrabrant — 2018-11-29T03:16:16.453Z

(This post was originally published on Oct 20th 2017, and is 1 of 4 posts brought forwarded today as part of the AI Alignment Forum launch sequence on fixed points.)

In this post, I give a stronger version of the open question presented here, and give a motivation for this stronger property. This came out of conversations with Marcello, Sam, and Tsvi.

Definition: A continuous function f:X→Y is called ubiquitous if for every continuous function g:X→Y, there exists a point x∈X such that f(x)=g(x).

Open Problem: Does there exist a topological space X with a ubiquitous function f:X→[0,1]X?

I will refer to the original problem as the Converse Lawvere Problem, and the new version as the the Ubiquitous Converse Lawvere Problem. I will refer to a space satisfying the conditions of (Ubiquitous) Converse Lawvere Problem, a (Ubiquitous) Converse Lawvere Space, abbreviated (U)CLS. Note that a UCLS is also a CLS, since a ubiquitous is always surjective, since g can be any constant function.

Motivation: True FairBot

Let X be a Converse Lawvere Space. Note that since such an X might not exist, the following claims might be vacuous. Let f:X→[0,1]X be a continuous surjection.

We will view X as a space of possible agents in an open source prisoner's dilemma game. Given two agents A,B∈X, we will interpret fA(B) as the probability with which A cooperates when playing against B. We will define UA(B):=2fB(A)−fA(B), and interpret this as the utility of agent A when playing in the prisoner's dilemma with B.

Since f is surjective, every continuous policy is implemented by some agent. In particular, this means gives:

Claim: For any agent A∈X, there exists another agent A′∈X such that fA′(B)=fB(A). i.e. A′ responds to B the way that B responds to A.

Proof: The function B↦fB(A) is a continuous function, since B↦fB is continuous, and evaluation is continuous. Thus, there is a policy B↦fB(A) in [0,1]X. Since f is surjective, this policy must be the image under f of some agent A′, so fA′(B)=fB(A).

Thus, for any fixed agent A, we have some other agent A′ that responds to any B the way B responds to A. However, it would be nice if A′=A, to create a FairBot that responds to any opponent the way that that opponent responds to it. Unfortunately, to construct such a FairBot, we need the extra assumption that f is ubiquitous.

Claim: If f is ubiquitous, then exists a true fair bot in X: an agent FB∈X, such

...
Hyperreal Brouwer
by Scott Garrabrant — 2018-11-29T03:15:23.650Z

(This post was originally published on Oct 6th 2017, and has been temporarily brought forwarded as part of the AI Alignment Forum launch sequence on fixed points.)

This post explains how to view Kakutani's fixed point theorem as a special case of Brouwer's fixed point theorem with hyperreal numbers. This post is just math intuitions, but I found them useful in thinking about Kakutani's fixed point theorem and many things in agent foundations. This came out of conversations with Sam Eisenstat.

Brouwer's fixed theorem says that a continuous function from a compact convex subset of Rn to itself has fixed point. Kakutani's fixed point is similar, but instead of continuous functions, it uses Kakutani functions, which are set valued functions with closed graph which are point wise nonempty and convex.

When I think about Kakutani functions, I usually think about them as limits of continuous functions. For example, consider the kakutani function f from [−1,1] to itself which sends negative inputs to 1, positive inputs to −1, and sends 0 to the entire interval. You can view f as a the limit of a sequence of functions fn sends x to min(max(−n⋅x,−1),1). This is not a point wise limit, since if it was 0 would be sent to 0, rather than the entire interval. Instead, it is a limit in the Hausdorff metric between the graphs of the functions.

Given two compact nonempty subsets X and Y of Rn, the Hausdorff distance between X and Y is the maximum over all points in X or Y of the Euclidean distance between that point and the closest point in the other set. Since X and Y are compact, this maximum is achieved.

Given compact convex subsets X and Y of Rn, we say that a sequence of continuous functions fn from X to Y converges in graph Hausdorff distance to the closed graph set valued function f if the graphs of fn viewed as subsets of X×Y converges to the graph of f in Hausdorff distance.

We say that a closed graph function f from X to Y is a continuous graph limit of if there exists a sequence of continuous functions which converges to f in graph Hausdorff distance.

Theorem 1: Every continuous graph limit f from a compact convex subset of Rn to itself has a fixed point (a point contained in its image).

Proof: f is a limit of continuous functions fn each of which has a fixed point by Brouwer. Choose one fixed point from each fn to get a sequence of points which has a convergent subsequence

...
Welcome to Seattle Rationality [Edit With Your Details]
by GreenCherenkov — 2018-11-29T02:58:53.297Z

(The following are our suggestions for what kind of information is best to include in the welcome post of your group, feel free to replace them with whatever you think is best)

What kind of events does your group usually run? What does it usually do?

How frequently does your group organize events or meet?

Who would be a good fit for you group?

Should they have any particular skills or have done some specific background reading?
Counterintuitive Comparative Advantage
by Wei Dai — 2018-11-28T20:33:30.023Z

This has been sitting in my drafts folder since 2011. Decided to post it today given the recent post about Dunning—Kruger and related discussions.

The standard rationalist answer when someone asks for career advice is "find your comparative advantage." I don't have any really good suggestions about how to make this easier, but it seems like a good topic to bring up for discussion.

If 15 years ago (when I was still in college and my initial career choice hadn't been finalized yet), someone told me that perhaps I ought to consider a career in philosophy, I would have laughed. "You must be joking. Obviously, I'll be really bad at doing philosophy," I would have answered. I thought of myself as a natural born programmer, and that's the career direction I ended up choosing.

As it turns out, I am a pretty good programmer, and a terrible philosopher, but it also happens to be the case that just about everyone else is even worse at doing philosophy, and getting some philosophical questions right might be really important.

The usual (instinctive) way for someone to choose a career is probably to pick a field that they think they will be particularly good at, using a single standard of goodness across all of the candidate fields. For example, the implicit reasoning behind my own career choice could be something like "Given a typical programming problem, I can solve it in a few hours with high probability. Whereas, given a typical philosophical problem, I can at best solve it after many years with low probability."

On the other hand, comparative advantage says that in addition to your own abilities, you should also consider how good other people are (or will be) at various fields, and how valuable the outputs of those fields are (will be). Unless you're only interested in maximizing income, and the fields you're considering are likely to remain stable over your lifetime (in which case you can just compare current salaries, although apparently many people don't even do that), this can be pretty tricky.

(There doesn't appear to be any previous OB/LW posts on comparative advantage. The closest I could find is Eliezer's Money: The Unit of Caring. Most discussions elsewhere seem to focus on simple static examples where finding comparative advantage is relatively trivial.)

Today (in 2018) there's an 80,000 Hour article about comparative advantage but that is more about how to fi

...
Stabilize-Reflect-Execute
by ozziegooen — 2018-11-28T17:26:39.741Z

You've recently joined a major organization in a senior management role. How can you organize your plans?

One simple way to think about them is with what can be called the "Stabilize-Reflect-Execute" cycle.

Stabilize

You first check if there are any urgent issues and address them immediately. Are there burning problems or opportunities that need to be dealt with? Second, you do anything you need to do to best prepare yourself for reflection. If there are people you need to talk to in order to get necessary information, you set that up upfront.

Reflect

Once urgent issues are dealt with and you are able to properly access the situation, you work to do so. For executives this can mean a lengthy period of discussions with all of the relevant people and thoughts on strategy before making formal announcements. This could take a few weeks or months.

Execute

Now is the time to begin working on non-urgent important problems, which should be the main ones. You follow through with your reflection. Execution may involve deciding on pursuing future larger stabilize-reflect-execute loops.

Let’s summarize. “Stabilize” refers to handling urgent issues and preparing for reflection. This is similar to the notion of getting one’s “

house in order

.” “Reflect” refers to deciding how to best deal with the important non-urgent issues. “Execute” refers to working on the important issues. This is basically a subset of the Eisenhower Method for situations where these three steps make up the majority of the work.

Examples

I think this cycle plays out in many important situations, so may be worth some independent study. Some examples of these cycles include:

Necessary Conditions

The stabilize-reflect-execute cycle is good for some specific situations. I think it may require the following:

1. There are some tasks that are both urgent and important.

If this is not true, the "stabilize" step isn't necessary.

2. There are some tasks that are both non-urgent and important.

If this is not true, the "reflect" and "execute" steps aren't necessary.

3. There is significant expected value for spending time figuring out how to do the non-urgent and important tasks.

If this is not true, the "reflect" step isn't necessary.

4. Figuring out how to do the important tasks can be done in one batch per full cycle.

If this is not true, then the work may be a...
Genetically Modified Humans Born (Allegedly)
by ryan_b — 2018-11-28T16:14:05.477Z

I realize normally we don't talk about the news or hot-button issues, but this is of sufficiently high importance I am posting anyway.

There is a link from Nature News on Monday here. There is a link from MIT Technology Review discussing the documents uploaded by the team behind the effort here.

Summarizing the report I heard on the radio this morning:

The lead scientist is He Jiankui, with a team at Southern University of Science and Technology, in Shenzhen.
7 couples in the experiment, each with HIV+ fathers and HIV- mothers.
CRISPR was used to genetically edit embryos to eliminate the CCR5 gene, providing HIV resistance.
Allegedly twin girls have been born with these edits.

I don't think DNA testing of the twins has taken place yet. If anyone has a line on good, nuanced sources, I'd be interested in hearing about it.
Oracle Induction Proofs
by Diffractor — 2018-11-28T08:12:38.306Z

Notation:

{0,1}L and {0,1}∗ are the sets of all bitstrings of length L, and the set of all finite bitstrings, respectively. Elements of these sets are denoted by x. The length i prefix of x is denoted by x:i.

BL is the set of all satisfiable booleans with all variables having an index ≤L, and B is the set of all satisfiable booleans. Similarly, B∗L and B∗ are the set of all booleans with all variables having an index ≤L, and the set of all booleans. Elements of these sets are denoted by B. B∧B′ is also a boolean. Bitstrings x may be interpreted as boolean constraints saying that the prefix of the bitstring must equal x.

f is a function N+→N+ that is time-constructible, monotonically increasing, and f(t)>t, which gives the runtime bound for traders.

l is some function N+→N+ upper-bounded by 2f(t) that is time-constructible, monotonically increasing, and l(t)>t. It gives the most distant bit the oracle inductor thins about on turn t.

d is the function N+→N+ given by d(t)=l(t)+⌈log2l(t)⌉+2t+7, giving the number of iterations of binary-search deployed on turn t.

δ is the function N+→[0,1]∪Q given by δ(t)=2−(t+3). It gives the proportion of the inductor distribution that is made up by the uniform distribution.

The operations query(A,p) and flip(p) take unit time to execute in our model of computation, though the input still has to be written down in the usual amount of time.

Given some arbitrary distribution Δ over {0,1}L, and a B∈B∗L, Δ(B) is the probability mass assigned to bitstrings that fulfill B. Given a B∈BL, ΔB is the conditional distribution given by ΔB(x)=Δ(B∧x)Δ(B). Δ∗B is the distribution induced by Approx(Δ,L,D,B). There is an implicit dependence on D which will be suppressed in the notation.

Lemma 1 (n-Lemma): Let ϵ=2−D. Given some distribution Δ over {0,1}L for which ∀x∈{0,1}L:Δ(x)≥nϵ, then

∀B∈BL,x∈{0,1}L:(n−1n+1)LΔB(x)≤Δ∗B(x)≤(n+1n−1)LΔB(x) .

To begin with, if x is incompatible with B, Δ∗B(x)=0 by the construction of Approx, and the inequality is trivial. So we can consider the case where x is compatible with B. Given σ as a strict prefix of x, abbreviate NextBit(Δ,L,D,|σ|+1,B,σ) as NB(σ).

Because we're using a binary-search depth of D on each bit, and taking the average of the interval, the probabilities are approximated to within ϵ2.

Δ(B∧σ1)−ϵΔ(B∧σ)+ϵ<Δ(B∧σ1)−ϵ2Δ(B∧σ)+ϵ2≤P(NB(σ)=1)≤Δ(B∧σ1)+ϵ2Δ(B∧σ)−ϵ2<Δ(B∧σ1)+ϵΔ(B∧σ)−ϵ

Tak...