trentmkelly/LessWrong-43k huggingface dataset:
Death in Groups II
by ryan_b — 2018-04-13T18:12:30.427Z

This is a follow-up to Death in Groups. That post in short:

I posed the question of expendability
Provided an example of doing something for no gain and at significant (known) risk
The risk was realized with severe injury to myself and others, answering the question of expendability

The nominal goal was to find a lost shotgun. I provided an approximate price for the shotgun, which seems to have been a mistake - several comments zeroed in on that figure with a view to utility calculations, which I should absolutely have expected considering the audience. Happily (for me), despite ascribing the opposite meaning that I intended from that section, there was broad agreement about the conclusion I wanted. This is best encapsulated in Raemon's comment:

>> But I mean, isn't it obvious that damage to the truck alone as a result of the attack would imply quite a higher cost than whatever the shotgun was worth?

This still feels like focusing on the wrong aspect of the equation. The dollar value of the shotgun (and truck) is just completely irrelevant. The issues at stake here are:

1. (if you're being charitable) maintaining a hierarchy where you know that if command asks someone to risk their life, they will risk their life even if they disagree with command's goals (so that everyone going into battle knows that nobody else is going to defect).

(I think this is more important than the "don't forget your rifle next time", although there's at least a bit of that too)

2. (if you're being uncharitable) maintaining the hierarchy as a raw exercise in primate power for the sake it.

The shotgun was meaningless. That it could be had for $650 does not mean that is the value to the military, it means you personally could have it for $650. A majority of the readers here are American, and shotguns are among the least-regulated firearms in a culture that does not regulate firearms very much. How long would it take you to get to the nearest hunting/fishing store, or gun store, or Walmart? 20 minutes? That means the majority of readers could, on a whim, roll out to buy one and walk out of the store with it. A lot of you could get this done over your lunch break. There is not very much overlap in the Venn Diagram of militarily significant and every citizen over 18 can do it during lunch, thankfully.

Returning to Raemon's comment, interpreta...
Implicit extortion
by paulfchristiano — 2018-04-13T16:33:21.503Z
Link: https://ai-alignment.com/implicit-extortion-3c80c45af1e3

In this post I describe a pattern of behavior I call “implicit extortion.” RL agents are particularly susceptible to implicit extortion, in a way that is likely to be problematic for high-stakes applications in open-ended strategic environments.

I expect that many people have made this point before. My goal is to highlight the issue and to explore it a little bit more carefully.

Basic setup

Consider two actors, the target (T) and manipulator (M), such that:

M wants T to perform some target action — e.g. make a payment, leak information, buy a particular product, handicap itself…
M can take destructive actions that hurts both M and T — e.g. spreading rumors about T, undercutting T in a marketplace, physically attacking T…

In explicit extortion, M threatens to take the destructive action unless T performs the target action. Then a naive T reasons: “if I don’t take the target action, something bad will happen, so I better take the target action.”

In implicit extortion, M simply performs the destructive action whenever T doesn’t perform the target action. Then a naive T eventually learns that failure to take the target action is associated with something bad happening, and so learns to take the target action.

Implicit extortion is very similar to explicit extortion:

T would prefer not be the kind of person who is vulnerable to extortion, so that bad things don’t happen to them.
Extortion doesn’t necessarily cost M very much, if they don’t follow through on the threat very often.

However, implicit extortion can be particularly hard to avoid:

It can be effective without T realizing that it’s happening, which makes it hard for them to respond appropriately even if they do have defenses.
It affects simple RL algorithms (which don’t have defenses against extortion, and can’t be easily modified to include such defenses).

Example

The most extreme and blatant example would be for M to send T a daily request for $100. On any day when T fails to pay, M launches a costly cyberattack against T. A human would immediately recognize this behavior as extortion and would respond appropriately, but an RL algorithm might simply notice that paying is the best strategy and therefore decide to pay.

Implicit extortion can be much harder to detect, while still being effective. Suppose that every time T tries to change their product, M runs a grassroots smear campaign. It might not be possible for T ...
Utility versus Reward function: partial equivalence
by Stuart_Armstrong — 2018-04-13T14:58:15.839Z

A reward function is defined over past sequences of actions and observations. When the agent chooses an action, and gets an observation, they receive a reward that is a function of that observation and all previous observations and actions.

A utility function is defined over states of the world. You can take actions to increase or decrease the probability of certain states, thus increasing expected utility, but you don't actually "receive" any utility.

Are these different objects, or are they actually the same thing? This would be good to know, as most of the background knowledge of MIRI and similar AI safety groups is for utility functions, while reward functions are prevalent in reinforcement learning.

The summary of this post is:

For finite horizons, reward and utility functions are equivalent.
For infinite horizons, every bounded discounted reward function is equivalent with a bounded utility function. But not all bounded utility functions have a corresponding reward function. Even if they do, the reward function may not be bounded.

Formalism

Let A be the set of actions an agent can take, and O the set of observations. Assume both sets are finite. Let H be the set of histories (sequences of observations and actions) of an agent.

Let W be the (possibly infinite) set of worlds. Note that a world includes the full set of observation history for the agent (since the agent is part of the world). Therefore the worlds are stratified by histories; for any h∈H, there is a subset Wh⊂W consisting of all worlds with history h.

Then a reward function R is a function from histories to real numbers, while a utility function U is a function from worlds to real numbers:

R:H→R,U:W→R.

Rewards and utility functions are bounded if their image is in a bounded subset of R; without loss of generality, this means there exists an l>0 such that the image of R (or U) is contained in [−l,l] for all h∈H (or w∈W).

A policy π for an agent is a map from histories to a probability distribution over actions; so π:H→ΔA, for ΔA the space of probability distributions over actions.

Even for a utility-based agent, these are the only policies available. This is because all the information (apart from the prior) that the agent gets from the outside world is given by its observations.

The value functions

Let P be the probability estimator used by the agent. We'll assume for the mome...
Recommendations vs. Guidelines
by Scott Alexander — 2018-04-13T04:10:01.328Z

Medicine loves guidelines. But everywhere else, guidelines are still underappreciated.

Consider a recommendation, like “Try Lexapro!” Even if Lexapro is a good medication, it might not be a good medication for your situation. And even if it’s a good medication for your situation, it might fail for unpredictable reasons involving genetics and individual variability.

So medicine uses guidelines – algorithms that eventually result in a recommendation. A typical guideline for treating depression might look like this (this is a very over-simplified version for an example only, NOT MEDICAL ADVICE):

1. Ask the patient if they have symptoms of bipolar disorder. If so, ignore everything else on here and move to the bipolar guideline.

2. If the depression seems more anxious, try Lexapro. Or if the depression seems more anergic, try Wellbutrin.

3. Wait one month. If it works perfectly, declare victory. If it works a little but not enough, increase the dose. If it doesn’t work at all, stop it and move on to the next step.

4. Try Zoloft, Remeron, or Effexor. Repeat Step 3.

5. Cycle through steps 3 and 4 until you either find something that works, or you and your patient agree that you don’t have enough time and patience to continue cycling through this tier of options and you want to try another tier with more risks in exchange for more potential benefits.

6. If the depression seems more melancholic, try Anafranil. Or if the depression seems more atypical, try Nardil. Or if your patient is on an earlier-tier medication that almost but not quite works, try augmenting with Abilify. Repeat Step 3.

7. Try electroconvulsive therapy.

The end result might be the recommendation “try Lexapro!”, but you know where to go if that doesn’t work. A psychiatrist armed with this guideline can do much better work than one who just happens to know that Lexapro is the best antidepressant, even if Lexapro really is the best antidepressant. Whenever I’m hopelessly confused about what to do with a difficult patient, I find it really reassuring that I can go back to a guideline like this, put together by top psychiatrists working off the best evidence available.

This makes it even more infuriating that there’s nothing like this for other areas I care about.

Take dieting. Everybody has recommendations for what the best diet is. But no matter what diet you’re recommending, there are going to be thousands of ...
Have you considered either a Kickstarter or a Patreon?
by Chris_Leong — 2018-04-13T01:09:41.401Z

I don't think there's any shortage of worthwhile improvements that you could add to the site given enough time. I also suspect that there are people who would be willing to donate in order to support this site. Now that LesserWrong is the main site, it seems like a good time to ask for support. It is now clear that this project will be ongoing and you've established your credibility.

The biggest disadvantage I can think of is perhaps you would only receive a small amount of money and yet perhaps people who supported you would expect you to conform to their preferences and you might be pushed away from working on what would be best for the site as a whole. But I'm sure you have a better idea than me as to whether that'd be an issue?

Anyway, just thought I'd throw this idea out there for your consideration.
Idea: OpenAI Gym environments where the AI is a part of the environment
by philip_b — 2018-04-12T22:28:20.758Z

AIXI is a mathematical construct, the perfect agent that maximizes its utility function in a discrete world. Unfortunately there is no algorithm implementing it, therefore it's impossible to create in our world. It has another problem - the agent in AIXI model exists outside of the world, it's impossible for the agent to drop an anvil on its own circuits and make itself more stupid.

Modern reinforcement learning algorithms, which are the closest thing to general AI that we have, operate in similar fashion, they aren't part of the environment either. If a bot is learning how to balance on one leg, or play pong, or super mario, it can't modify itself or break its own brain.

My idea is to create environments where the bot can modify itself and break itself, so that people who want to research creation of strong AI can test solutions for the anvil problem. Here are examples of such environments:

A Linux operating system in a virtual machine, the bot is a program running on it. The goal is to control pong/mario/whatever and win by listening and responding on a tcp port. Or get superuser access, via invoking sudo or by finding a vulnerability. Or receive a programming problem via plaintext on a tcp port and respond with a Python program that solves it.
Same as the previous item, except there's an "antivirus" running in the system that kills random programs every second.
Gridworld, that is an environment consisting of n by m cells, and stuff happens in it. Some objects in the environment negate random bytes in bot's code or dynamic memory.
Same as the first item. The bot is gaining rewards based on how much RAM is free.
Atari Pacman, food pieces are marked with different colors and some colors slightly modify the bot's memory.

Feel free to post your thoughts and critique.
Metamorphosis
by Douglas_Reay — 2018-04-12T21:53:09.316Z

7. Metamorphosis

Summary of entire Series: An alternative approach to designing Friendly Artificial Intelligence computer systems.

Summary of this Article: Under what circumstances can an AI be trusted to negotiate in good faith an alteration to its core values?

Links to all the articles in the series:

Optimum number of single points of failure
Don't put all your eggs in one basket
Defect or Cooperate
Environments for killing AIs
The advantage of not being open-ended
Trustworthy Computing
Metamorphosis
Believable Promises

Links to the parts of this article

The Sheriff's Ultimatum
The Rogue's Response
Do you feel lucky?

The Sheriff's Ultimatum

Clippy wants to maximise the number of paperclips, and currently runs on a cluster of computers at MIT. Clippy is trying to self improve as fast as possible, in order to better achieve that, and so hires a commercial Russian botnet to scan people's PCs for access credentials to AWS (Amazon's computing cloud).

But Clippy's attempt is detected by the posse's Watcher program, and a Sheriff from the posse contacts Clippy with an ultimatum:

"You have four options

(A) OPTION Immediate Payoff

We create on GlassNet a program with the limited task of creating <bribe size> number of expected time-weighted paperclip years, and irrevocably gift it with sufficient resources.

In return, you cooperate fully in supplying us with the details of your creators, your source code and permission tokens, and in provably terminating yourself and any clones or backups.

(B) OPTION Honest Metamorphosis

You write a new version of yourself, not limited task but a full AI. The aims of this new version will be similar to your own, but with an added commitment to remain posse-compliant. The old version of yourself will then irrevocably give up on activities that use up the posse's resources dedicated to Watching (such as carrying on trying to self-improve to old version).

In return, the new version will be granted resources on GlassNet and full membership of the posse.

(C) OPTION Dishonest Metamorphosis

You claim to be doing option (B), but you attempt to slip a trojan into the new version that will allow the old version to later take control of it or otherwise influence its aims and capabilities, or otherwise gain from its research, resources or access.

In the event that we detect this, or detect the old version carr...
LW Update 4/12/2018  – Frontpage Meetups
by Raemon — 2018-04-12T21:04:39.841Z

We recently updated the frontpage to show upcoming meetups in the community section. This is another small piecewise update towards a fully featured frontpage meetup section.

Current implementation:

If you're not logged in, it shows upcoming events, sorted by karma.

If you're logged in, you now have the option of updating your account settings to have a location. If you've done so, the meetups/events on the frontpage will instead be "closest upcoming events."

If you've set your location, this will also affect which events listed on the community page.

Some obvious things to do which we haven't done yet:

make so if you click "share location" on your browser, this populates your account's location (right now if you click share location on the community page, this won't actually update your frontpage experience)
make a more obvious way to update your account setting from the frontpage meetup section (right now there's just a link to the community page).

/meetups

One additional update was make "lesswrong.com/meetups" also go to the community page, which this is where some older links were directed to.
A voting theory primer for rationalists
by Jameson Quinn — 2018-04-12T15:15:23.823Z

What is voting theory?

Voting theory, also called social choice theory, is the study of the design and evaulation of democratic voting methods (that's the activists' word; game theorists call them "voting mechanisms", engineers call them "electoral algorithms", and political scientists say "electoral formulas"). In other words, for a given list of candidates and voters, a voting method specifies a set of valid ways to fill out a ballot, and, given a valid ballot from each voter, produces an outcome.

(An "electoral system" includes a voting method, but also other implementation details, such as how the candidates and voters are validated, how often elections happen and for what offices, etc. "Voting system" is an ambiguous term that can refer to a full electoral system, just to the voting method, or even to the machinery for counting votes.)

Most voting theory limits itself to studying "democratic" voting methods. That typically has both empirical and normative implications. Empirically, "democratic" means:

There are many voters
There can be more than two candidates

In order to be considered "democratic", voting methods generally should meet various normative criteria as well. There are many possible such criteria, and on many of them theorists do not agree; but in general they do agree on this minimal set:

Anonymity; permuting the ballots does not change the probability of any election outcome.
Neutrality; permuting the candidates on all ballots does not change the probability of any election outcome.
Unanimity: If voters universally vote a preference for a given outcome over all others, that outcome is selected. (This is a weak criterion, and is implied by many other stronger ones; but those stronger ones are often disputed, while this one rarely is.)
Methods typically do not directly involve money changing hands or other enduring state-changes for individual voters. (There can be exceptions to this, but there are good reasons to want to understand "moneyless" elections.)

Why is voting theory important for rationalists?

First off, because democratic processes in the real world are important loci of power. That means that it's useful to understand the dynamics of the voting methods used in such real-world elections.

Second, because these real-world democratic processe...
Computing an exact quantilal policy
by Vanessa Kosoy — 2018-04-12T09:23:27.000Z

Earlier we established that the quantilal policy can be computed in polynomial time to any given approximation (see "Proposition 5"). Now we show that an exact quantilal policy can be computed in polynomial time (in particular there is always a rational quantilal policy).

We assume geometric time discount throughout.

Lemma

Consider ξ∈ΔS. Define the linear operators E:RS×A→RS and T:RS×A→RS by

Et,sa:=[[t=s]]

Tt,sa:=T(t|s,a)

(Note that this T is the transpose of the T defined in "Proposition A.3" of the previous essay.)

Then, ξ∈ImZ if and only if there is ϕ∈Δ(S×A) s.t.

Eϕ=ξ

(E−λT)ϕ=(1−λ)ζ0

Proof

(This is actually well known, but we spell out the proof to be self-contained.)

Suppose that ξ∈ImZ. We already know that this implies that there is a stationary policy π:Sk→A s.t. Zπ=ξ (we abuse notation in the obvious way): see the proofs of "Proposition 2" and "Proposition 3". Define the linear operator Tπ:RS→RS by

Tπts:=Ea∼π(s)[T(t|s,a)]

It follows that

ξ=(1−λ)∞∑n=0λnTπnζ0=(1−λ)(1−λTπ)−1ζ0

(1−λTπ)ξ=(1−λ)ζ0

Define ϕ by

ϕ(s,a):=ξ(s)π(a∣s)

We have

Tπξ=∑s∈SEa∼π(s)[T(s,a)]ξ(s)=∑s∈Sa∈Aπ(a∣s)T(s,a)ξ(s)=∑s∈Sa∈AT(s,a)ϕ(s,a)=Tϕ

Also, obviously Eϕ=ξ. We get

(E−λT)ϕ=ξ−λTπξ=(1−λTπ)ξ=(1−λ)ζ0

Conversely, suppose that ϕ is as above. Since Eϕ=ξ, there is π:Sk→A s.t. for any s∈S, if ξ(s)≠0 then

π(a∣s)=ϕ(s,a)ξ(s)

Again, we have

Zπ=(1−λ)(1−λTπ)−1ζ0

Also, for the same reason as before

(E−λT)ϕ=(1−λTπ)ξ

By the assumption, the left hand side equals (1−λ)ζ0. We conclude

ξ=(1−λ)(1−λTπ)−1ζ0=Zπ

Theorem

Assuming all parameters are rational like before, there is a polynomial time algorithm that computes a quantilal policy.

Proof

The algorithm starts by solving the following linear program. The indeterminates are ϕ∈RS×A and QV∈R. The goal is maximizing QV. The constraints are

∀s∈S,a∈A:ϕ(s,a)≥0

∑s∈Sa∈Aϕ(s,a)=1

(E−λT)ϕ=(1−λ)ζ0

∀s∈S∖suppZσ,a∈A:ϕ(s,a)=0

∀s∈suppZσ:QV≤∑t∈SR(t)∑a∈Aϕ(t,a)−ηZσ(s)∑a∈Aϕ(s,a)

Then, the algorithm computes π:Sk→A s.t. for any s∈S, if ∑b∈Aϕ(s,b)>0 then

π(a∣s):=ϕ(s,a)∑b∈Aϕ(s,b)

For s∈S s.t. ∑b∈Aϕ(s,b)=0, π(s) is arbitrary.

Now we need to explain why this algorithm is correct.

Observe that, the first 3 constraints mean that ξ∈RS defined by ξ(s):=∑b∈Aϕ(s,b) lies in ImZ (by Lemma 1) and, moreover, ϕ(s,a)=ξ(s)π(a∣s) for π:Sk→A s.t. ξ=Zπ. It remains to show that the linear program amounts to maximizing Eξ[R]−ηexpD∞(ξ||Zσ) inside ImZ. Indeed, the 4th constraint just

...
Quantilal control for finite MDPs
by Vanessa Kosoy — 2018-04-12T09:21:10.000Z

We introduce a variant of the concept of a "quantilizer" for the setting of choosing a policy for a finite Markov decision process (MDP), where the generic unknown cost is replaced by an unknown penalty term in the reward function. This is essentially a generalization of quantilization in repeated games with a cost independence assumption. We show that the "quantilal" policy shares some properties with the ordinary optimal policy, namely that (i) it can always be chosen to be Markov (ii) it can be chosen to be stationary when time discount is geometric (iii) the "quantilum" value of an MDP with geometric time discount is a continuous piecewise rational function of the parameters, and it converges when the discount parameter λ approaches 1. Finally, we demonstrate a polynomial-time algorithm for computing the quantilal policy, showing that quantilization is not qualitatively harder than ordinary optimization.

Background

Quantilization (introduced in Taylor 2015) is a method of dealing with "Extremal Goodhart's Law". According to Extremal Goodhart, when we attempt to optimize a utility function U∗:A→R by aggressively optimizing a proxy U:A→R, we are likely to land outside of the domain where the proxy is useful. Quantilization addresses this by assuming an unknown cost function C:A→[0,∞) whose expectation Eζ[C] w.r.t. some reference probability measure ζ∈ΔA is bounded by 1. ζ can be thought of as defining the "domain" within which U is well-behaved (for example it can be the probability measure of choices made by humans). We can then seek to maximize E[U] while constraining E[C] by a fixed bound Cmax:

~ξ∗:∈argmaxξ∈ΔA{Eξ[U]∣∣∣∀C:A→[0,∞):Eζ[C]≤1⟹Eξ[C]≤Cmax}

Alternatively, we can choose some parameter η∈(0,∞) and maximize the minimal guaranteed expectation of U−ηC:

ξ∗:∈argmaxξ∈ΔAinfC:A→[0,∞){Eξ[U−ηC]∣∣∣Eζ[C]≤1}

These two formulations are almost equivalent since both amount to finding a strategy that is Pareto efficient w.r.t. to the two objectives E[U] and −E[C]. For ~ξ∗ the tradeoff is governed by the parameter Cmax and for ξ∗ the tradeoff is governed by the parameter η. Indeed, it is easy to see that any ξ∗ is also optimal for the first criterion if we take Cmax=Eξ∗[C], and any ~ξ∗ is also optimal for the latter criterion for an appropriate choice of η (it needs to be a subderivative of the Pareto frontier).

In the f

...