trentmkelly/LessWrong-43k huggingface dataset:
Book Review: Weapons of Math Destruction
by Zvi — 2017-06-04T21:20:57.000Z

Epistemic Status: Minus One Million Points

Shortness Status: Long (this is a proposed new norm I want to try out, in the sense of ‘apologies for writing a long letter, I did not have enough time to write a shorter one.’ By contrast, Against Facebook was longer in words, but would be short.)

Weapons of Math Destruction is an easy read, but a frustrating one.

The book claims to be about the misuse of big data and machine learning to guide decisions, how that harms people and leads to bad outcomes, and how to fix it. The distortions of aiming at and rewarding what we are measuring rather than what we actually want worries me more and more. It is one of the biggest issues of our age, so I was excited to read a new take on it, even if I expected to already know the bulk of the facts and ideas presented.

There is some of that in the book, and those parts provide some useful information, although if you are reading this you likely already know a lot of it.

I.

What the book is actually mostly about on its surface, alas, is how bad and unfair it is to be a Bayesian. There are two reasons, in her mind, why using algorithms to be a Bayesian is just awful.

The first objection is that probabilistic algorithms are probabilistic. It is just awful that predictive algorithms are used to decide who should get or keep a job, or get cheaper credit, or see certain advertisements, because the algorithm might be wrong. Look at this example of someone the algorithm got wrong! Look at this reason the algorithm got it wrong! Look how wrong it is! Clearly we need to rely on humans, who get things wrong more often, but do so in a less systematic fashion so we can’t prove exactly why any given human got something wrong.

The second objection is that algorithms rank people and options likely to be better above people and options likely to be worse. It is just awful that an algorithm notices that people who have bad credit, or live in a certain zip code, or shop at a certain store, or share some other trait, are either a worse or a better business proposition. You see, this is not fair and probably makes you a racist. This is because the people who are ranked either worse or better tend to be poor, and/or they tend to be not white, and that is just awful. If the resulting system gives them more attention in some way – say, by marketing to them to sell them things they migh...
Rationalist Seder: Dayenu, Lo Dayenu
by Raemon — 2017-06-04T20:55:50.836Z

There's one more piece of the NYC Rationalist Seder Haggadah that I wanted to pull out, to refer to in isolation. I think is quite relevant to some current concerns in the evolving Rationality Community, and which is interesting in particular because of how it's evolved over the past 6 years.

"Dayenu" is a traditional Jewish song, roughly a thousand years old. It describes a number of gifts that God gave the Jewish people. For each gift/verse, lyrics culminate with "Dayenu", or "it would have been enough."

At the first rationalist Seder, Zvi made two, ahem, rather significant changes to the song.

The first dealt with the fact that, well, we're basically a bunch of atheists, and even if we weren't, God slaying a bunch of firstborn children just isn't the sort of thing we're super in favor of these days.

The second change dealt with that that... obviously each individual miracle *wouldn't* have been enough to free the Jewish people. Freeing them from Egypt but not parting the Red Sea to let escape when Pharoah has second thoughts would very much *not* have been sufficient.
And beyond that, Less Wrong culture is emphatically based around the status quo not being satisfactory. To constantly aspire to something better.

Zvi's new version of the song told the story of human history, and it did so from the framing of "Lo Dayenu" - not enough. If we had discovered fire, but not developed agriculture, our journey would not have been finished.
But, in the spirit of cultural pendulums that swim back and forth to overcompensate for previous failures, a years later Daniel Speyer took a second pass at revising the song:

Traditionally, we sing “Dayenu”: it would have been enough.

Our sages asked: what do we mean by this? In some of the traditional pairings, one step without the next would have left us all dead! How can that be enough? And it was answered: celebrate each step toward freedom as if it were enough, then start out on the next step. If we reject each step because it is not the whole liberation, we will never achieve the whole liberation.

And yet, if we celebrate our past victories and become complacent, so too will we never achieve the whole liberation. And so we have come to sing “Lo Dayenu”: it would not have been enough.

And it has almost been said, “Keep two truths in your pocket, and take them out acc

...
The Personal Growth Cycle
by Gordon Seidoh Worley — 2017-06-04T17:20:33.817Z
Link: https://mapandterritory.org/the-personal-growth-cycle-34ec1c218615

We encounter psychology in our lives mainly through dysfunction. That is, people generally go to see psychologists and psychiatrists because they or others are unhappy with how they are thinking or behaving. This attention to error and correction makes most of applied psychology a kind of negative feedback process aiming to normalize thoughts and behaviors within desired tolerances. People find this psychopathological approach useful as far as it goes, but it leaves out ways of using psychology in positive feedback loops to help people achieve more than normal. Positive psychology focuses on this.

Some folks, however, take a negative view of positive psychology. From what I can surmise, they hold this view because of positive psychology’s association with self-help, life coaching, and leadership training — domains where rationalization can often pass for explanation and hucksters can operate as long as they are sufficiently charismatic. And although plenty of folks are working on evidence-based self-help and other scientific personal development methodologies, the guilt-by-association seems sufficient for some people to dismiss the field. Yet I and others have found tremendous value in our lives by looking at such topics as esteem, psychological development, and meaning through the lens of positive psychology. If all that holds some people back from believing in the usefulness of positive psychology is that its sometimes use in cons, then we can advise that they should, as always, practice epistemic hygiene and not confuse correlation for causation.

But even acknowledging the theoretical validity of positive psychology, some folks still object that its techniques don’t seem to work, at least not for themselves and not for their friends. To this point, it seems likely that positive psychology only works for the sufficiently privileged because they uniquely have the luxury of being able to spend time and energy on self actualization. Put another way, if you’re too busy surviving, you don’t have time for positive psychology and even if you did it’s not likely to help because it’s not your primary constraint. This seems to neatly explain much of why people, after controlling for scams and bullshit, still have varying success with positive psychology.

But I think there is something left unexplained. Among those for whom positive psychology works, some specific techn...
A new, better way to read the Sequences
by Said Achmiz — 2017-06-04T05:10:09.886Z

A new way to read the Sequences:

https://www.readthesequences.com

It's also more mobile-friendly than a PDF/mobi/epub.

(The content is from the book — Rationality: From AI to Zombies. Books I through IV are up already; Books V and VI aren't up yet, but soon will be.)

Edit: Book V is now up.

Edit 2: Book VI is now up.

Edit 3: A zipped archive of the site (for offline viewing) is now available for download.
Rationalist Seder: A Story of War
by Raemon — 2017-06-03T20:17:10.163Z

For whatever reason, the rationality community is inordinately Jewish. Among other things, this resulted in 2011 in people in New York putting together "Rationalist Seder", a reframing of the story of Jewish Liberation From Egypt to reflect upon liberation in general. What does it mean to be free? What are the many things we might want to be free from?

As a non-Jewish person, it took me a few years to really wrap my head around the holiday, but it's evolved into one of the more poignant moments throughout my year.

The holiday was begun by Zvi Mowshowitz, and has been modified by various people in different geographic corners. In NYC, for the past few years it's been refined and tweaked by Daniel Speyer, who's aimed to invoke a strong sense of history that feels purposeful, connected to the original story, uniquely "rationalist", and feels very much like a holiday in the ancient, venerable and valuable sense.

The complete text of Dan's most recent iteration is here. It contains a mix of retellings of traditional Jewish stories, abridged essays from around the rationalsphere, and a few original pieces. It has a lot of nice gems that I'd like to be able to refer to more easily, but they make the most sense in context with each other.

But one particular new story stands alone, and establishes the frame around which the Rationalist Seder is told:

(Image from the wikimedia foundation, credit to Thomas Forester)

A Story of War

Two tribes live next to each other. Each fears the other will attack, and so builds weapons to hold in readiness. And then, seeing that the other has built weapons, builds more weapons. Their clothes are threadbare. Their children are hungry. But still they spend their time making weapons, lest the other tribe build more. They would prefer to live in peace, and make no weapons, but whichever tribe adopted that policy first would surely be killed.

Are these tribes free? There is no pharoah putting the whip to their backs, but still they do not live as they choose.

In the next valley, there are two more tribes. They distrust each other as much as the first two, but they are ruled by a powerful empire. The empire forbids tribes to fight each other, and enforces that rule with unstoppable legions. And so these two tribes have the peace and prosperity that the first two tribes wanted.

And in the v...
Cooperative Oracles: Nonexploited Bargaining
by Scott Garrabrant — 2017-06-03T00:39:55.000Z

In this post, we formalize and generalize the phenomenon described in the Eliezer Yudkowsky post Cooperating with agents with different ideas of fairness, while resisting exploitation. (Part of the series started here.)

Consider a game between n players, P1,…,Pn. We denote Ai as the finite set of actions available to player Pi, and Δi as the set of distributions on Ai. Let R denote Δ1×…×Δn, the set of possible outcomes of this game, and given an r∈R, let ri∈Δi be the ith component.

Each player Pi will be equipped with a "fair set" Fi⊆R of outcomes in which they feel that they are not being exploited, as well as a utility function, a continuous function Ui:R→[0,1]. Note that the use of the word fair here is a little weird. We will think of agents as identifying an outcome as fair if they do not feel that they have been exploited. They may find it perfectly fair to exploit others.

Fi cannot be an arbitrary set. There must be a set valued function fi:R→P(Δi) with closed graph and fi(r) nonempty and convex for all r∈R, such that Fi is exactly the set of r such that ri∈fi(r). i.e. Fi should be the set of points for which a given Kakutani map fixes the Δi coordinate.

This condition may seem arbitrary, but it is both exactly what we need for our bargaining procedure to work, and exactly what we would expect to get out of agents that have access to everyone's actions with something like a reflective oracle. This condition in part implies that for every collection of distributions on actions that all the other players take, there must be some distribution on actions that you can take in which you consider yourself nonexploited. The restriction is slightly stronger that that, requiring a type of continuity in the way that your nonexploited responses are a function of the other players' actions.

Our Bargaining procedure is now simple. We take the set F of fair points to be the intersection over all i of Fi, and we choose an r∈F which is Pareto optimal in F (i.e. there is no other r′∈F with Ui(r′)≥Ui(r) for all i and Ui(r′)>Ui(r) for some i).

There may be more than one Pareto Optimum. If so, we choose one arbitrarily.

This procedure is not obviously well defined. In particular, it is not obvious that F is nonempty, or that given that it is nonempty that it contains Pareto optimum.

Claim: F is nonempty and contains a Pareto optimum (over F).

Proof: First, we show that

...
Cooperative Oracles: Stratified Pareto Optima and Almost Stratified Pareto Optima
by Scott Garrabrant — 2017-06-03T00:38:46.000Z

In this post, we generalize the notions in Cooperative Oracles: Nonexploited Bargaining to deal with the possibility of introducing extra agents that have no control but have preferences. We further generalize this to infinitely many agents. (Part of the series started here.)

Consider example 1 from the previous post: P1 and P2 are in a prisoners dilemma. Their utility functions are U1(p,q)=2q−p and U2(p,q)=2p−q. Their fair sets are F1={(p,q)∈R|p≤q} and F2={(p,q)∈R|q≤p}.
We will introduce one extra player P3 which has no options.

However, he will have preferences, U3=−U1−U2. Since he has no actions, his fair set must be everything.

Now, since the game is zero sum, every outcome is Pareto Optimal, so using the method described in the previous post, any equilibrium in which p=q will be valid. We will modify the framework to make this no longer the case, and make p=q=1 the only valid solution. For this, we will define a "stratified Pareto optimum."

We will still have a finite number of players P1,…,Pn. Each player will specify a subset of players they will depend on. Let Di be set of players that Pi depends on.
Ui must only be a function of the actions of players in Di, and Fi must be closed under changing the actions of any player not in Di.

We define a preorder on the agents. We say that Pi≺Pj if Pi∈Dj, and we take the transitive closure, so Pi≺Pk whenever Pi≺Pj and Pj≺Pk. We also say that Pi≺Pi for all i.

We say that r′ is a stratified Pareto improvement over r for player Pi if:

Ui(r′)>Ui(r)

Uj(r′)≥Uj(r) for all Pj≺Pi

r′k=rk Pi⊀Pk

We say that r is stratified Pareto optimal in F⊆R for player Pi if there is no alternate outcome r′∈F such that r′ is a stratified Pareto improvement over r for player Pi. If r is stratified Pareto optimal for all players, we say it is stratified Pareto optimal.

Claim: The set F=⋂i≤nFi has a stratified Pareto optimum.

Proof: Consider some nonempty subset of players S with the property that for all Pi,Pj∈S, Pi≺Pj, and there is no Pi∈S, Pk∉S with Pk≺Pi. We know that such a subset exists, since the preorder over players induces a poset over equivalence classes of players (where every player in an equivalence class transitively depends on every other player), and this poset must have a minimal element.

We then take an element of ∏Pi∈SΔi, which is Pareto optimal for all the players in S. Note that it is important here that the play

...
Cooperative Oracles: Introduction
by Scott Garrabrant — 2017-06-03T00:36:17.000Z

This is the first in a series of posts introducing a new tool called a Cooperative Oracle. All of these posts are joint work Sam Eisenstat, Tsvi Benson-Tilsen, and Nisan Stiennon.

Here is my plan for posts in this sequence. I will update this as I go.

Introduction

Nonexploited Bargaining

Stratified Pareto Optima and Almost Stratified Pareto Optima

Definition and Existence Proof

Alternate Notions of Dependency

In this post, I will give a sketchy advertisement of what is to come.

Cooperative oracles are a refinement on reflective oracles. We consider the set of Turing machines with access to a reflective oracle, and we think of each Turing machine as having a utility function (perhaps written in a comment in the source code). This utility function is itself computable using the reflective oracle.

Since the definition of a reflective oracle use a fixed point procedure that may have had multiple fixed points, there are actually many different reflective oracles. However, we will take something like a Pareto optimum in the class of reflective oracles.

For example if two players in a prisoner's dilemma use a reflective oracles to cooperate with exactly the probability the other player cooperates, there is a continuum of fixed points in which the player cooperate with the same probability. We want to take the fixed point in which the both cooperate.

Unfortunately, we do not want to just take a standard Pareto Optimum, since we are working with the class of all Turing Machines, and for every machine with a utility function, there is another with the negation of that utility function, so all points will be Optima.

For this, we will use a stratified version of Pareto optima. If a third party wants the two above players to defect against each other, we will still choose mutual cooperation. Since the players only reference each other, and not the third party, the third party is not considered in the notion of optimality for the output of the Oracle on the prisoners' dilemma game.

We will be able to build a cooperative oracle, which is a reflective oracle that (roughly) ensures that all of the outputs of the computations are Pareto optimal for all of the utility function of the machines involved it that computation.
This Pareto optimality is only used to choose between multiple fixed points. We will still be restricted to have the computation follow its source code, and have

...
Entangled Equilibria and the Twin Prisoners' Dilemma
by Scott Garrabrant — 2017-06-02T22:09:00.000Z

In this post, I present a generalization of Nash equilibria to non-CDT agents. I will use this formulation to model mutual cooperation in a twin prisoners' dilemma, caused by the belief that the other player is similar to you, and not by mutual prediction. (This post came mostly out of a conversation with Sam Eisenstat, as well as contributions from Tsvi Benson-Tilsen and Jessica Taylor)

This post is sketchy. If someone would like to go through the work of making it more formal/correct, let me know. Also, let me know if this concept already exists.

Let A1,…,An be a finite collection of players. Let Mi be a finite collection of moves available to Ai. Let Ui:∏iMi→R be a utility function for player Ai.

Let Δi be the simplex of all probability distributions over Mi. Let Vi denote the vector space of functions v:Mi→R, with ∑m∈Miv(m)=0.

A vector given a distribution in p∈Δi and a vector v∈Vi we say that v is unblocked from p if there exists an ε>0, such that p+εv∈Δi. i.e., it is possible to move in the direction of v from p, and stay in Δi.

Given a strategy profile P=(p1,…,pn) in ∏j≤nΔj, and a vector V=(v1,…,vn) in ∏j≤nVj, we say that V improves P for Ai if

limε→0∑m1∈M1…∑mn∈MnUi(m1,…,mn)(∏j≤npj(mj)+εvj(mj)−∏j≤npj(mj))ε>0.

We call this limit the utility differential for Ai at P in the direction of V.

i.e., V improves P for Ai if Ui is increased when P is moved an infinitesimal amount in the direction of V. (not that this is defined even when the vectors are not unblocked from the distributions)

A "counterfactual gradient" for player Ai is a linear function from Vi to ∏j≤nVj, such that the function into the Vi component is the identity. This represents how much player Ai expects the probabilities of the other players to move when she moves her probabilities.

A "counterfactual system" for Ai is a continuous function which takes is a strategy profile P in ∏j≤nΔj, and outputs a counterfactual gradient. This represents the fact that a player's counterfactual gradient could be different depending on the strategy profile the game ends up in. We fix a counterfactual system Ci for each player.

Claim: There exists a strategy profile P=(p1,…,pn) in ∏j≤nΔj such that for all players Ai, if vi∈Vi is unblocked from pi, then Ci(P)(vi) does not improve P.

I will call such a point an entangled equilibrium.

Proof Sketch: (Very sketchy, and I have not verified this. There is probably a

...
An algorithm with preferences: from zero to one variable
by Stuart_Armstrong — 2017-06-02T16:35:55.000Z

A putative new idea for AI control; index here.

A simple way of thinking that I feel clarifies a lot of issues (related to Blue Minimising Robot):

Suppose you have an entity H that follows algorithm alH. Then define:

What H does is its actions/outputs in the environment.

What H is is alH.

What H wants is an interpretation of what H does (and possibly what it is), in order to construct a utility function or reward function corresponding with its preferences.

The interpretation part of wants is crucial, but it is often obscured in practice in value learning. That's because we often start with things like `H is a boundedly rational agent that maximises u...', or we lay out the agent in such a way that that's clearly the case.

What we're doing there is writing the entity as alH(u) --- an algorithm with a special variable u that tracks what the entity wants. In the case of cooperative inverse reinforcement learning, this is explicit, as the human's values are given by a θ, known to the human. Thus the human's true algorithm is alH(⋅), the human observes θ, meaning that θ is an objective fact about the universe. And then the human follows alH(θ).

Note here that knowing what the human is in the one-variable sense (i.e. knowing alH(⋅)) helps with the correct deduction about what they want - while simply knowing the joint alH(θ) does not.

In contrast an interpretation starts with a zero-variable algorithm, and attempts to construct a one-variable one. There for, given alH it constructs (one or more) alHi(⋅) and ui such that

alH=alHi(ui).

This illustrates the crucial role of interpretation, especially if alH is highly complex.