trentmkelly/LessWrong-43k huggingface dataset:
Conceptual problems with utility functions
by Dacyn — 2018-07-11T01:29:42.585Z

[Epistemic status: strong intuitions that I have had for a long time, but not yet compared with other people's intuitions, and I am not sure how well I can convey my intuitions.]

So, a lot of agent foundations research seems to be built on the premise that agenthood is about maximizing the expected value of some utility function, or about having actions that bear some relation to some utility function, or something like that. I think that this premise is wrong except in special cases like where there is only one agent in the world. I don't know exactly how to explain this, so I will just say some stuff.

The Ultimatum Game is a standard example of a game in which there are many Pareto optimal outcomes, and no clear way to choose between them. So if we imagine two "perfectly rational" agents playing the Ultimatum Game against each other, what happens? I think that this question is meaningless. OK, so what happens in real life? People use their notions of fairness to resolve situations like the Ultimatum Game. Fairness is a part of human values, so essentially the answer is that an agent's values are used to resolve the ambiguity of multiple Pareto optimal outcomes.

But wait! In the classical understanding, a utility function is supposed to encode all information about an agent's values. So if there is a notion of fairness relevant to a real-life Ultimatum Game based on monetary payouts, then it supposedly means that the utility function is not just the same as the monetary payouts, and the game is not a true Ultimatum Game at all. But then what is a true Ultimatum Game? Does such a mythical beast even exist? Eliezer had to invent a fairly far-fetched scenario before he found something that he was willing to call the "true Prisoner's dilemma".

But even in the "true Prisoner's dilemma", the utility function does not appear to capture all of Eliezer's values -- he seems to still be motivated to say that "cooperate" is the right answer based on symmetry and hope, which again appear to be human values. So I propose that in fact there is no "true Prisoner's dilemma", simply because calling something a dilemma is asking you to resolve it with your own values, but your own values are allegedly encapsulated by the utility function which is subject to the dilemma.

I propose instead that...
Dependent Type Theory and Zero-Shot Reasoning
by evhub — 2018-07-11T01:16:45.557Z

I.

When I sit down to write code, I can safely throw everything I can think of at the problem and just keep trying to run the code until it works—and when it works, I can actually see it working. If my code runs, it means I actually had to understand enough of the moving parts to get it to run. This is especially true in languages like Haskell with strong type constraints preventing me from doing anything too stupid.

When I sit down to prove a theorem, I have to be much more careful. One misstep in my reasoning and my whole proof will be invalid, without me even knowing I made a mistake until I or someone else catches it later. If my proof goes through, it's possible I understood enough of the moving parts to make it work, but it's also possible I didn't.

The standard solution to the problem of not having something to test your proof against when doing math is to use a proof checker, which lets you turn an arbitrary math problem into a programming problem. Instead of having to just get the right answer without ever being able to actually check your answer, you get lots of chances to test possible solutions.

Proof checkers are great, but traditional proof checkers are also pretty limited. Concepts like equality require specialized machinery like paramodulation to properly resolve, introducing new data types requires writing out long lists of PA-like axioms and in general modern functional programming tools that make writing out math easier like pattern-matching, ADTs, monads, etc. are just missing. When I wrote my own theorem prover/proof checker a while ago I ran into exactly these sorts of problems.

Dependent type theory is the modern way to solve all of these problems. Both equality and the natural numbers are defined just by giving the constructors, no axioms needed. For-alls and implications are just functions. Proving theorems really just feels like writing Haskell code.

I'm not going to try and give a dependent type theory tutorial here, because I don't think I would be able to do better than the one I used, Theorem Proving in Lean, which I highly recommend taking a look at. That being said, I do want to give a couple examples of what I mean before I move on. Don't worry if you don't understand this part, I just want to show you some actual code in a dependent type theory.

I said equality and the natural numbers were just da... ,,,,,,,,,,,,,,,,,,,,,,
A comment on the IDA-AlphaGoZero metaphor; capabilities versus alignment
by AlexMennen — 2018-07-11T01:03:03.305Z

Iterated Distillation and Amplification has often been compared to AlphaGo Zero. But in the case of a Go-playing AI, people often don't see a distinction between alignment and capability, instead measuring success on a single axis: ability to win games of Go. But I think a useful analogy can still be drawn that separates out ways in which a Go-playing AI is aligned from ways in which it is capable.

A well-aligned but not very capable Go-playing agent would be an agent that is best modeled as trying to win at Go, as opposed to trying to optimize some other feature of sequence of board positions, but still does not win against moderately skilled opponents. A very capable but poorly aligned Go-playing agent would be an agent that is very good at causing the Go games that it plays to have certain properties other than the agent winning. One way to create such an agent would be to take a value network that rates board positions somewhat arbitrarily, and then use Monte Carlo tree search to find policies that cause the game to progress through board states that are rated highly by the value network.

To a certain extent, this could actually happen to something like AlphaGo Zero. If some bad board positions are mistakenly assigned high values, AlphaGo Zero will use Monte Carlo tree search to search for ways of getting to those board positions (and similarly for ways to avoid good board positions that are mistakenly assigned low values). But it will simultaneously be correcting the mistaken values of these board positions as it notices that its predictions of the values of future board positions that follow from them are biased. Thus the Monte Carlo tree search stage increases both capability and alignment. Meanwhile, the policy network retraining stage should be expected to decrease both capability and alignment, since it is merely approximating the results of the Monte Carlo tree search. The fact that this works shows that the extent to which Monte Carlo tree search increases alignment and capability exceeds the extent to which policy network retraining decreases them.

It seems to me that this is the model that Iterated Distillation and Amplification should be trying to emulate. The amplification stage will both increase capability by increasing the available computing power, and increase alignment because of the human overseer. The distillation stage will decreas...
Bounding Goodhart's Law
by eric_langlois — 2018-07-11T00:46:19.822Z

Goodhart's law seems to suggest that errors in utility or reward function specification are necessarily bad in sense that an optimal policy for the incorrect reward function would result in low return according to the true reward. But how strong is this effect?

Suppose the reward function were only slightly wrong. Can the resulting policy be arbitrarily bad according to the true reward or is it only slightly worse? It turns out the answer is "only slightly worse" (for the appropriate definition of "slightly wrong").

Definitions

Consider a Markov Decision Process (MDP) M=(S,A,T,R∗) where

S is the set of states,
A is the set of actions,
T:S×A×S→R are the conditional transition probabilities, and
R∗:S×A→R is the reward function. (Note: "reward" is standard terminology for MDPs but it's fine to think of this as "utility")

A policy π:S×A→R is a mapping from states to distributions over actions withπ(s,a)=Pr(a|s).

Any given policy π induces a distribution σ(π) over states in this MDP. If we are concerned about average reward we can take σ(π) to be the stationary distribution or, if the environment is episodic, we can take σ(π) to be the distribution of states visited during the episode. The exact definition is not particularly important for us.

Define the return of policy π according to reward function R to be

G(π,R)=Eπ[R(s,a)]=∑s∈S∑a∈Aσs(π)π(s,a)R(s,a)

Goodhart Regret

Suppose we have an approximate reward signal ^R and we use it to specify a policy ^π. How bad is ^π according to the true reward R∗?

More specifically, what is the regret of using ^π compared to the optimal policy π∗? Formally,

Regret(^π)=G(π∗,R∗)−G(^π,R∗)

We can expand this as

G(π∗,R∗)−G(^π,R∗)=[G(π∗,R∗)−G(π∗,^R)]+[G(π∗,^R)−G(^π,^R)]+[G(^π,^R)−G(^π,R∗)]

Let ϵ≥0, then Regret(^π)≤3ϵ if the following conditions are satisfied by ^R and ^π:

1. G(π∗,R∗)−G(π∗,^R)≤ϵ

2. G(π∗,^R)−G(^π,^R)≤ϵ

3. G(^π,^R)−G(^π,R∗)≤ϵ

Condition 2 says that ^π is not much worse than π∗ when measured against ^R. That is what we expect if we designed ^π to be specifically good at ^R, so condition 2 is just a formalization of the notion that ^π is tailored to ^R.

Conditions 1 and 3 compare a fixed policy against two different reward functions. In general for policy π and reward functions R and R′,

G(π,R)−G(π,R′)=Eπ[R(s,a)−R′(s,a)]≤maxs,a(R(s,a)−R′(s,a))

Result: Uniformly Bounded Error

Assume that we h... ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
Mechanistic Transparency for Machine Learning
by DanielFilan — 2018-07-11T00:34:46.846Z

Cross-posted on my blog.

[EDIT (added Jan 2023): it's come to my attention that this post was likely influenced by conversations I had with Chris Olah related to the distinction between standard interpretability and the type called "mechanistic", as well as early experiments he had which became the 'circuits' sequence of papers - my sincere apologies for not making this clearer earlier.]

Lately I've been trying to come up with a thread of AI alignment research that (a) I can concretely see how it significantly contributes to actually building aligned AI and (b) seems like something that I could actually make progress on. After some thinking and narrowing down possibilities, I've come up with one -- basically, a particular angle on machine learning transparency research.

The angle that I'm interested in is what I'll call mechanistic transparency. This roughly means developing tools that take a neural network designed to do well on some task, and outputting something like pseudocode for what algorithm the neural network implements that could be read and understood by developers of AI systems, without having to actually run the system. This pseudocode might use high-level primitives like 'sort' or 'argmax' or 'detect cats', that should themselves be able to be reduced to pseudocode of a similar type, until eventually it is ideally reduced to a very small part of the original neural network, small enough that one could understand its functional behaviour with pen and paper within an hour. These tools might also slightly modify the network to make it more amenable to this analysis in such a way that the modified network performs approximately as well as the original network.

There are a few properties that this pseudocode must satisfy. Firstly, it must be faithful to the network that is explained, such that if one substitutes in the pseudocode for each high-level primitive recursively, the result should be the original neural network, or a network close enough to the original that the differences are irrelevant (although just in case, the reconstructed network that is exactly explained should presumably be the one deployed). Secondly, the high-level primitives must be somewhat understandable: the pseudocode for a 256-layer neural network for image classification should not be output = f2(f1(input)) where f1 is the action of the first 128 layers and f2 is the action o...
An environment for studying counterfactuals
by Nisan — 2018-07-11T00:14:49.756Z

Summary: I introduce a decision theory framework where successful agents are those with good counterfactuals.

Motivation

The problem of logical counterfactuals is how to define probabilities P(ϕ|A()=a) when A()=a is known to be false. (I'll ignore more general counterfactuals P(ϕ|ψ) in this post.)

The theory of logical induction provides a joint distribution over sentences, so the problem becomes: How do you condition on A()=a when A()=a has negligible probability?

Exploration tries to solve this by making sure that A()=a never has negligible probability. But it doesn't work in problems like Agent Simulates Predictor that contain predictors who can't tell when the agent explores.

A better solution is early exploration, which uses an early stage of the logical inductor to do exploration. But then the later stages of the inductor know that A()=a is false, and we're back where we started.

I'm going to describe an environment that captures these features of the problem — it's got reflection, early exploration, counterfactuals, and a Bayesian update that stands in for the evolution of a logical inductor.

Informal definition

The agent outputs counterfactual distributions p(U=u|A=a). This determines an expected utility for each action. Most of the time, an action is chosen for the agent that maximizes this expected utility. But a small fraction of the time, an exploration action is chosen instead.

The agent receives an observation O as input, from which it can infer whether exploration will occur. The agent also receives a prior P as input, and this prior accurately reflects the behavior of the agent as a function of O and P. (This uses a fixed-point theorem.)

If action a is chosen, then the counterfactual p(U=u|A=a) is factual; the rest are counterfactual. We judge an agent according to how accurate its factual counterfactual is, in addition to how much utility it gets.

Here's an agent that does okay in this environment: It adopts P as its epistemic state and ignores O. Because of exploration, it can compute counterfactuals by conditioning. This agent does okay but not great, since it ignores O.

You could try to make a better agent as follows: Adopt P as a prior and then do a Bayesian update on O. But now you've inferred whether exploration occurs, so some actions have probability zero, and it's not clear how to compute counterfact... ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
A universal score for optimizers
by levin — 2018-07-10T23:52:22.678Z

[Epistemic status: the idea is simple and intuitive, so I wouldn't be surprised if it has appeared before. If so, I would appreciate pointers.]

Eliezer defines optimization power as the ability to produce solutions higher on the preference ordering. This feels right, but it would be nice to convert this intuition into a score that can be measured, a universal metric that allows one to meaningfully state that AlphaZero is better at Go than Deep Blue was at chess. In this post I try accomplishing just that. I start by giving a candidate metric and some of its properties. I then discuss how it can be approximately computed in complex settings, and describe a somewhat paradoxical implication of adopting this definition for repeated games.

For simplicity I will assume a deterministic, finite-action-space environment (extending everything to MDPs with uncountable actions doesn't seem to be a fundamental obstacle). The action space is A, the outcome space is X, utility of an agent under question is u(x(a)). Because x(a) is deterministic I will just write u(a) for brevity. Note that we can represent sequential decision problems in this framework (e.g. Sudoku), elements of A would then be vectors of individual actions. We write n(S) for the size of set S.

Definition. Suppose we observe an agent with utility function u(a) take an action ^a, then we say this agent has a C-score of:

C(u,^a)=−logn(a|u(a)≥u(^a))n(A)

Intuitively, C-score is inversely proportional to the fraction of outcomes that are as good for the agent as the one it managed to obtain. One interpretation of this formula is that the agent is competing against a bot that just picks a random action (random-bot), and then −C(u,^a) is log probability of losing in this competition.

Some properties of C(u,a):

C-score is independent of computational power employed
It's impossible to compute C-score without knowing the utility function
Improving from being in top 1% outcomes to top 0.1% is as "hard" as improving from 10% to 1%
One can use Monte-Carlo to generate lower bounds of the form "w.h.p. C-score of this agent is at least Y"
Strategy of taking a random action produces the same score in any setting (under some assumptions it might also be approximately true for the strategy of trying k random moves and picking the best one)

Approximating C-score

Can we estimate C-score for com... ,,,,,,,,,,,,,
Bayesian Probability is for things that are Space-like Separated from You
by Scott Garrabrant — 2018-07-10T23:47:49.130Z

First, I should explain what I mean by space-like separated from you. Imagine a world that looks like a Bayesian network, and imagine that you are a node in that Bayesian network. If there is a path from you to another node following edges in the network, I will say that node is time-like separated from you, and in your future. If there is a path from another node to you, I will say that node is time-like separated from you, and in your past. Otherwise, I will say that the node is space-like separated from you.

Nodes in your past can be thought of as things that you observe. When you think about physics, it sure does seem like there are a lot of things in your past that you do not observe, but I am not thinking about physics-time, I am thinking about logical-time. If something is in your past, but has no effect on what algorithm you are running on what observations you get, then it might as well be considered as space-like separated from you. If you compute how everything in the universe evaluates, the space-like separated things are the things that can be evaluated either before or after you, since their output does not change yours or vice-versa. If you partially observe a fact, then I want to say you can decompose that fact into the part that you observed and the part that you didn't, and say that the part you observed is in your past, while the part you didn't observe is space-like separated from you. (Whether or not you actually can decompose things like this is complicated, and related to whether or not you can use the tickle defense is the smoking lesion problem.)

Nodes in your future can be thought of as things that you control. These are not always things that you want to control. For example, you control the output of "You assign probability less than 1/2 to this sentence," but perhaps you wish you didn't. Again, if you partially control a fact, I want to say that (maybe) you can break that fact into multiple nodes, some of which you control, and some of which you don't.

So, you know the things in your past, so there is no need for probability there. You don't know the things in your future, or things that are space-like separated from you. (Maybe. I'm not sure that talking about knowing things you control is not just a type error.) You may have cached that you should use Bayesian probability to deal with thing...
Alignment problems for economists
by Chris van Merwijk — 2018-07-10T23:43:56.662Z

AI alignment is a multidisciplinary research program. This means that there is potentially relevant knowledge and skill scattered across different disciplines. But it also means that people schooled only in narrow disciplines will experience a hurdle when they would work on a problem in AI alignment. One such discipline is economics, from which decision theory and game theory originated.

In this post I want to explore the idea that we should try to create a collection of “alignment-problems-for-economists”, packaged in a way that economists who have relevant knowledge and skill but don't understand ML/CS/AF can work on them.

There seem to be sub-problems in AI alignment that economists might be able to work on. However, out of the economists that I’ve spoken to, some are enthusiastic about this but see it as a personal career-risk to work on it as they do not understand the computer science. So if we can take subproblems in alignment, and package them in a way that economists can immediately start working on them, then we might be able to utilize intellectual resources (economists) that would otherwise have worked on something different.

Two types of economists to target

1. Economists who also to a degree understand basic ML/CS

2. Economists who do not.

I don’t find it very plausible that we could find sub-problems for the second type to work on, but it doesn’t seem entirely impossible: there could be certain specific problems in mechanism design or social choice or so, that would be useful for alignment but don’t require any ML/CS.

Properties of alignment-problems-for-economists that are desirable:

1. Publishable in economics journals. I have spoken to economists that are interested in the alignment problem, but they are hesitant to work on it: It is a risky career move to work on alignment if they cannot publish in journals that they are used to.

2. High work/statement ratio. How long will it take to solve the problem, versus providing the statement of the problem? If 90% of the problem is to state it in a form so that an economist could work on it, then it would likely not be efficient to do so. It should be a problem that can relatively easily be communicated clearly to an economist, while taking more time to solve.

3. No strong reliance on CS/ML tools. Many economists are somewhat familiar with basic ML techniques, but if a problem relies too much on know...