trentmkelly/LessWrong-43k huggingface dataset:
VOI is Only Nonnegative When Information is Uncorrelated With Future Action
by Diffractor — 2018-08-31T05:13:11.916Z

Attention conservation notice: Most of this post is a recap of the standard arguments for UDT, but the way in which the standard proof for value of information being nonnegative breaks down in some decision theory scenarios probably isn't common knowledge yet.

The standard proof that the value of information (VOI) is always positive is very simple and goes something like this. Let x∈X be a true underlying state from the set of all underlying states, let a∈A be an action selected from a space of possible actions, let z∈Z be an observation from the set of all observations, and let U(x,a) be a utility function that maps an action and an underlying state to [0,1] . SI is an abbreviation for sample information.

E(U)=maxa∈A∑x∈XU(x,a)P(x)=maxa∈A∑z∈Z(∑x∈XU(x,a)P(x|z))P(z)

E(U|SI)=∑z∈ZE(U|z)P(z)=∑z∈Zmaxa∈A(∑x∈XU(x,a)P(x|z))P(z)

Let a′ be the action selected by maximizing utility without looking at the sample information. Comparing the right-hand side of both lines, we can see that for any given piece of information z , you can either copy a′ (in which case the utility acquired will be the same), or select some other action (in which case you'll get more utility).

This is just the simple argument that, no matter what information you see, you can always just act as if you hadn't seen it to do as well as the ignorant version of you, and maybe you can do better.

Swapping out P(x) and P(x|z) with P(x|a) and P(x|z,a) respectively, generalize this proof to cover situations where the underlying state is correlated with your choice of action. This alteration gets Newcomb's problem right.

However, the proof breaks down when the probability distribution over which information you see, is correlated your choice of future action. Example problems with this property are XOR Blackmail, Parfait's Hitchiker, Transparent Newcomb, and any fictional setting with stable time loops. All of these problems have the property that selecting the best action conditional on the information you see does make you better off, but this decreases the probability of you getting into a favorable situation in the first place (according to past-you)

To formalize this, we need a way to let the probability distribution over the information vary depending on the probability distribution over future actions. Specifically, we will assume a continuous function f:ΔA→ΔZ which maps a probabili...
Do what we mean vs. do what we say
by Rohin Shah — 2018-08-30T22:03:27.665Z

Written quickly after a CHAI meeting on the topic, haven't thought through it in depth.

If we write down an explicit utility function and have an AI optimize that, we expect that a superintelligent AI would end up doing something catastrophic, not because it misunderstands what humans want, but because it doesn't care -- it is trying to optimize the function that was written down. It is doing what we said instead of doing what we meant.

An approach like Inverse Reward Design instead says that we should take the human's written down utility function as an observation about the true reward function, and infer a distribution over true reward functions. This agent is "doing what we mean" instead of doing what we said.

This suggests a potential definition -- in a "do what we mean" system, the thing that is being optimized is a latent variable, whereas in a "do what we say" system, it is explicitly specified. Note that "latent" need not mean that you have a probability distribution over it, it just needs to be hidden information. For example, if I had to categorize iterated distillation and amplification, it would be as a "do what we mean" system where the thing being optimized is implicit in the policy of the human and is never made fully certain.

However, this doesn't imply that we want to build a system that exclusively does what we mean. For example, with IRD, if the true reward function is not in the space of reward functions that we consider (perhaps because it depends on a feature that we didn't have), you can get arbitrarily bad outcomes (see the problem of fully updated deference). One idea would be to have a "do what we mean" core, which we expect will usually do good things, but have a "do what we say" subsystem that adds an extra layer of safety. For example, even if the "do what we mean" part is completely sure about the human utility function and knows we are making a mistake, the AI will still shut down if we ask it to because of the "do what we say" part. This seems to be the idea in MIRI's version of corrigibility.

I'd be interested to see disagreements with the definition of "do what we mean" as optimizing a latent variable. I'd also be interested to hear how "corrigibility" and "alignment" rel...
You Play to Win the Game
by Zvi — 2018-08-30T14:10:00.279Z

Previously (Putanumonit): Player of Games

Original Words of Wisdom:

Quite right, sir. Quite right.

By far the most important house rule I have for playing games is exactly that: You Play to Win the Game.

That doesn’t mean you always have to take exactly the path that maximizes your probability of winning. Style points can be a thing. Experimentation can be a thing. But in the end, you play to win the game. If you don’t think it matters, do as Herm Edwards implores us: Retire.

It’s easy to forget, sometimes, what ‘the game’ actually is, in context.

The most common and important mistake is to maximize expected points or point differential, at the cost of win probability. Alpha Go brought us many innovations, but perhaps its most impressive is its willingness to sacrifice territory it doesn’t need to minimize the chances that something will go wrong. Thus it often wins by the narrowest of point margins, but in ways that are very secure.

The larger-context version of this error is to maximize winning or points in the round rather than chance of winning the event.

In any context where points are added up over the course of an event, the game that matters is the entire event. You do play to win each round, to win each point, but strategically. You’re there to hoist the trophy.

Thus, when we face a game theory experiment like Jacob faced in Player of Games, we have to understand that we’ll face a variety of opponents with a variety of goals and methods. We’ll play a prisoner’s dilemma with them, or an iterated prisoner’s dilemma, or a guess-the-average game.

To win, one must outscore every other player. Our goal is to win the game.

Unless or until it isn’t. Jacob explicitly wasn’t trying to win at least one of the games by scoring the most points, instead choosing to win the greater game of life itself, or at least a larger subgame. This became especially clear once winning was beyond his reach. At that point, the game becomes something odd – you’re scoring points that don’t matter. It’s not much of a contest, and it doesn’t teach you much about game theory or decision theory.

It teaches you other things about human nature, instead.

A key insight is what happens when a prize is offered for the most successful player of one-shot prisoner’s dilemmas, or a series of iterated prisoner’s dilemmas.

If you cooperate, you cannot win. Period. Someone else will defect while their opponent...
Player of Games
by Jacob Falkovich — 2018-08-29T21:26:37.672Z

Things I learned from a game theory party: why prisoner's dilemmas are not what they seem, why being smart is often worse than being the dumbest, and the virtues of trolls.

Like all of my posts, this is cross-posted with delay from Putanumonit.

Does game theory apply to real life? It’s easy to fall into one of two errors:

Thinking that human interactions can be precisely modeled by a 2×2 payout matrix, and being shocked by the vagaries of human psychology.
Thinking that since humans aren’t rational game theory doesn’t apply to them, and being shocked when they follow incentives in predictable patterns after all.

Of course, human behavior is a combination of both elements: the mathematical structure of payoffs, incentives, and equilibria, the idiosyncrasies of culture, mood, and personality. Understanding both sides can be quite a superpower. I know people who made millions and bankrupted competitors by tweaking some small features of a public auction. This stuff is hard, but it works.

I took every available class on the math of game theory as an undergrad. Since then, I’ve been catching up on the squishy human part of the equation. There’s plenty of great research in this area, but the best way to learn about people is to observe them in their habitat. So when my friend Spencer told me that he’s organizing a live game theory party, I jumped at the opportunity.

I will break with tradition by not repeating the descriptions of common games that you’ve heard 100 times before, nor by deriving the rational strategies and equilibria. If you need to catch up on the math you can follow the links to Wikipedia etc.

What game is it anyway?

The first game we played had the following stated rules:

Each person starts the game with 5 tokens and plays a single round with 6 different players.
Each player can play “blue” or “red”, the players do so simultaneously.
The payouts are as follows: if both picked “blue” they each receive one token from “the bank”, if both picked “red” they have to pay one token to the bank, and if they show different colors, “blue” pays two tokens to “red”.

Or in matrix form:

What game is this? Take a minute to think for yourself.

If you said that this is the prisoner’s dilemma, with “blue” and “red” standing for “cooperate” and “defect”, congratulations! You paid attention in class and get an A-.

For extra credit, consider that we call both the following ga...
Computational complexity of RL with traps
by Vanessa Kosoy — 2018-08-29T09:17:08.655Z

I briefly describe an open (to the best of my knowledge) problem and link to cstheory.stackexchange questions with more details. The readers are invited to try solving those questions.

In the comment section of my research agenda essay, Jessica and I discussed, among other things, whether intelligence should be defined by some sort of diminishing regret condition or just by Bayes-optimality. In particular, the former approach runs into a problem when traps are allowed, whereas the latter is still perfectly well-defined.

However, one problem with Bayes-optimality is its computational complexity. Without traps, Bayes-optimality is feasible in the weak sense that there is asymptotically Bayes-optimal algorithm (e.g. PSRL: Posterior Sampling Reinforcement Learning) that runs in time polynomial in the size of the problem, if the prior is described by an explicit list of MDP transition kernels and reward functions (so, in practice it's only feasible for a small number of small MDPs; there are also algorithms that can work for a large number of small MDPs). Hopefully this weak feasibility can be boosted into something stronger using additional assumptions about the prior/environment (e.g. hierarchical structure; presumably, deep learning somehow exploits such assumptions). However, when traps are allowed, it's not clear that even weak feasibility is achievable. In particular, PSRL is no longer asymptotically Bayes-optimal, since it is more or less guaranteed to walk into any trap given enough time.

I condensed the problem into two succinct questions on cstheory.stackexchange (but received no answers so far). In the stochastic case, there are some computational hardness results in the literature, but they are not sufficiently strong to rule out an asymptotically Bayes-optimal algorithm (they only rule out an exactly Bayes-optimal algorithm). In the deterministic case, there are no hardness results that I know of. So, currently I would not be very surprised neither by a positive nor a negative answer to any of those questions.

EDIT: This answer resolves the question, more or less. Even in the deterministic case, the problem is NP-hard, with a bound of at least Ω(1#{hypotheses}) on the approximation accuracy.
2 Events with Christine Peterson in Vancouver This Week!
by Evan_Gaensbauer — 2018-08-29T07:04:51.863Z

[Note: if you are not or will not be in/around Vancouver, Canada on Friday or Saturday this weekend, ignore this post.]

[Meta: I would usually reserve announcements like this to Facebook events for the relevant groups, or an event on the LessWrong 'Community' page. However, Christine Peterson, co-founder of the Foresight Institute informed me on short notice on Monday she is in town for 4 days this week. To maximize visibility for these rare opportunities on such short notice, I'm posting the invitation here as well.]

Christine Peterson is a co-founder of the Foresight Institute and a host of the Personalized Life Extension Conferences, with a wide background ranging from nanotechnology to futurism.

On Friday night, she will be there for a general meet and greet, and a discussion over dinner on 2 topics: 'Effective Altruism and Long-Term Goals,' and 'Finding Your EA Life Partner.' The event starts at 5 p.m., at 6939 Frederick Ave. in Burnaby, near Metrotown. (LessWrong event) (Facebook event)

On Saturday morning, Christine will be joining the BC Life Extension Hikes group, hosted by the Lifespan Society of BC We'll be meeting at the bus stop at West Georgia and Denman Street downtown. From the Facebook event:

Join us for a 9 km walk around the Stanley Park Seawall with Christine Peterson. She has hosted the Personalized Life Extension Conferences and has a broad background ranging from nanotechnology to futurism. We'll meet at the bus stop and depart promptly at 10am.

For the Life Extension Hike, please RSVP at either the Facebook or Meetup.com event.

We hope to see you there!
History of the Development of Logical Induction
by Scott Garrabrant — 2018-08-29T03:15:51.889Z

I have been asked several times about how the development of logical induction happened, so I am writing it up.

June 2013 - I write my first Less Wrong Post. It may not seem very related to logical uncertainty, but it was in my head. I wanted to know how to take different probabilities for the same event and aggregate them, so I could take an agent that could maintain probabilities on facts even when facts that it originally thought were separate turn out to be logically equivalent. (I am not sure if all this was in my head at the time, but it was at some point over the next year.)

I make a proposal for a distribution on completions of a theory: repeatedly observe that a set of sentences whose probabilities should sum to one fail to sum to one, and shift their probabilities in a way inspired from the above post. I do not actually prove that this process converges, but I conjecture that it converges to the distribution I describe here. (This post is from when I wrote it up in June 2014; I can't remember exactly what parts I already had at the time.)

December 2013 - I tell my proposal to Abram Demski, who at some point says that he thinks it is either wrong or equivalent to his proposal for the same problem. (He was right and his proposal was better.) At this point I got very lucky; when I told this to Abram, I thought he was just a smart person at my local Less Wrong meet up, and it turned out that he was almost the only person to also try to do the thing I was doing. Abram and I start talking about my proposal a bunch together, and start trying to prove the above conjecture.

April 2014 - Abram and I start the first MIRIx to think about logical uncertainty, and especially this proposal. I at the time had not contacted MIRI, even to apply for a workshop, because I was dumb.

At some point we realize that the proposal is bad. The thing that makes us give up on it is the fact that sometimes observing that A→B can drastically decrease your probability for B.

August 2014 - Abram and I go to MIRI to talk about logical uncertainty with Nate, Benya, Eliezer, and Paul. We share the thing we were thinking about, even though we had given up on it at the time. At some point in there, we talk about assigning probability 1/10 to a sufficiently late digit of π being 0.

Soon after that, I propose a new project for our MIRIxLA group to work on, which I call the Benford Test. I wanted

...
HLAI 2018 Field Report
by Gordon Seidoh Worley — 2018-08-29T00:11:26.106Z

I spent the past week in Prague (natively called Praha, but known widely as Prague via the usual mechanisms others decide your own names) at the

Human Level AI multi-conference

held between the AGI, BICA, the NeSY conferences. It also featured the Future of AI track, which was my proximate reason for attending to speak on the Solving the AI Race panel thanks to the selection of

my submission

to

GoodAI

's

Solving the AI Race challenge

as one of the finalists. I enjoyed the conference tremendously and the people I met there even more, and throughout the four days I found myself noticing things I felt were worth sharing more widely, hence this field report.

I arrived on Tuesday and stayed at the home of a fellow EA (they can out themselves in the comments if they like, but I won't do it here since I couldn't get in touch with them in time to get their permission to disclose their personage). If you didn't already know,

Prague is rapidly becoming a hub for effective altruists in Europe

, and having visited it's easy to see why: the city is beautiful, most people speak enough English that communication is easy for everyone, the food is delicious (there's lots of vegan food even though traditional Czech cuisine is literally meat and potatoes), and it's easily accessible to everyone on the continent. Some of the recent events held in Prague include the

Human-aligned AI Summer School

and the upcoming Prague

AI safety camp

.

Wednesday was the first day of the conference, and I honestly wasn't quite sure what to expect, but

Ben Goertzel

did a great job of setting the tone with his introductory keynote that made clear the focus was on exploring ideas related to building AGI. We then dove in to talk after talk for the next 4 days, each one considering an idea that might lie on the path to enabling general intelligence for machines, and we doubled-up on the third day with a Future of AI program that considered AI policy and ethics issues. I took a lot of notes, but being a safety researcher I'm hesitant to share them because that feels like a hazard that might accelerate AGI development, so instead I'll share a few high-level insights that I think summarize what I learned without saying anything with more than a very small (let's say < 5%) chance of giving someone a dangerous inspiration they wouldn't have had any...
[Paper] The Global Catastrophic Risks of the Possibility of Finding Alien AI During SETI
by avturchin — 2018-08-28T21:32:16.717Z

[edit: it looks like immediately after publishing the paper, the journal becomes extinct, so the link is no longer working]

My article on the topic has been finally published, 10 years after first draft. I have discussed the problem before on LW. The preprint, free of paywall, is here.

The main difference between the current version and my previous post is that I concluded that such attack is less probable, because if we take into account distribution in the Universe of the naive-our-level-civilizations and civilizations which has powerful AI and are SETI-senders, when the attack become possible, only if most naive civilizations go extinct before the creation of the AI. In that case, succumbing to a SETI-attack may be net positive, as the chance that it is a message from a benevolent alien AI becomes our only way to escape inevitable extinction. Anyway, we should be cautious, if we get any alien message, especially if it will have descriptions of computers and programs to them.
Corrigibility doesn't always have a good action to take
by Stuart_Armstrong — 2018-08-28T20:30:12.302Z

In a previous critique of corrigibility, I brought up the example of a corrigible AI-butler that was in a situation where it was forced to determine the human's values through its actions - it had no other option.

Eliezer pointed out that, in his view of corrigibility, there could be situations where the AI had no corrigible actions it could take - where, in effect, all it could do was say "I cannot act in corrigible way here".

This makes corrigibility immune to my criticism in the previous post, while potentially opening the concept up to other criticisms - it's hard to see how a powerful agent, whose actions affect the future in many ways, including inevitably manipulating the human, can remain corrigible AND still do something. But that's a point for a more thorough anaylsis.
Unknown Knowns
by Zvi — 2018-08-28T13:20:00.982Z

Previously (Marginal Revolution): Gambling Can Save Science

A study was done to attempt to replicate 21 studies published in Science and Nature.

Beforehand, prediction markets were used to see which studies would be predicted to replicate with what probability. The results were as follows (from the original paper):

Fig. 4: Prediction market and survey beliefs.

The prediction market beliefs and the survey beliefs of replicating (from treatment 2 for measuring beliefs; see the Supplementary Methods for details and Supplementary Fig. 6 for the results from treatment 1) are shown. The replication studies are ranked in terms of prediction market beliefs on the y axis, with replication studies more likely to replicate than not to the right of the dashed line. The mean prediction market belief of replication is 63.4% (range: 23.1–95.5%, 95% CI = 53.7–73.0%) and the mean survey belief is 60.6% (range: 27.8–81.5%, 95% CI = 53.0–68.2%). This is similar to the actual replication rate of 61.9%. The prediction market beliefs and survey beliefs are highly correlated, but imprecisely estimated (Spearman correlation coefficient: 0.845, 95% CI = 0.652–0.936, P < 0.001, n = 21). Both the prediction market beliefs (Spearman correlation coefficient: 0.842, 95% CI = 0.645–0.934, P < 0.001, n = 21) and the survey beliefs (Spearman correlation coefficient: 0.761, 95% CI = 0.491–0.898, P < 0.001, n = 21) are also highly correlated with a successful replication.

That is not only a super impressive result. That result is suspiciously amazingly great.

The mean prediction market belief of replication is 63.4%, the survey mean was 60.6% and the final result was 61.9%. That’s impressive all around.

What’s far more striking is that they knew exactly which studies would replicate. Every study that would replicate traded at a higher probability of success than every study that would fail to replicate.

Combining that with an almost exactly correct mean success rate, we have a stunning display of knowledge and of under-confidence.

Then we combine that with this fact from the paper:

Second, among the unsuccessful replications, there was essentially no evidence for the original finding. The average relative effect size was very close to zero for the eight findings that failed to replicate according to the statistical significance criterion.

That means there was a clean cut. Thirteen of ...