trentmkelly/LessWrong-43k huggingface dataset:
Questions from an imaginary statistical methods exam
by Richard_Kennaway — 2015-02-04T13:57:38.113Z

Answers to these questions should be expressed numerically, where possible, but no number should be given without a justification for the specific value.

1. Suppose that you have mislaid your house keys, something most people have experienced at one time or another. You look in various places for them: where you remember having them last, places you've been recently, places they should be, places they shouldn't be, places they couldn't be, places you've looked already, and so on. Eventually, you find them and stop looking.

Every time you looked somewhere, you were testing a hypothesis about their location. You may have looked in a hundred places before finding them.

As a piece of scientific research to answer the question "where are my keys?", this procedure has massive methodological flaws. You tested a hundred hypotheses before finding one that the data supported, ignoring every failed hypothesis. You really wanted each of these hypotheses in turn to be true, and made no attempt to avoid bias. You stopped collecting data the moment a hypothesis was confirmed. When you were running out of ideas to test, you frantically thought up some more. You repeated some failed experiments in the hope of getting a different result. Multiple hypotheses, file drawer effect, motivated cognition, motivated stopping, researcher degrees of freedom, remining of old data: there is hardly a methodological sin you have not committed.

(a) Should these considerations modify your confidence or anyone else's that you have in fact found your keys? If not, why not, and if so, what correction is required?

(b) Should these considerations affect your subsequent decisions (e.g. to go out, locking the door behind you)?

2. You have a lottery ticket. (Of course, you are far too sensible to ever buy such a thing, but nevertheless suppose that you have one. Maybe it was an unexpected free gift with your groceries.) The lottery is to be drawn later that day, the results available from a web site whose brief URL is printed on the ticket. You calculate a chance of about 1 in 100 million of a prize worth getting excited about.

(a) Once the lottery results are out, do you check your ticket? Why, or why not?

(b) Suppose that you do, and it appears that you have won a very large sum of money. But you remember that the prior chance of this happening was 1 in 100 million. How confident are y...
Generalizing the Corrigibility paper's impossibility result?
by Benya_Fallenstein — 2015-02-04T03:16:53.000Z

In our paper on corrigibility, we consider the question of how to make a highly intelligent agent that would pursue some goal, but not resist being shut down if its human programmers determined that they had made a mistake in specifying this goal. We assume that we are given two utility functions: a function UN, which specifies the agent's normal goal (which it pursues until the shutdown button is pressed), and a function US, which specifies the goal of shutting down. We then ask whether there is some sort of combined utility function U such that an agent maximizing U would act as if maximizing UN unless and until the shutdown button is pressed, and act as if maximizing US after this point.

In the paper, we investigate a particular linear way to combine UN and US: we define U to compute utility according to UN if the shutdown button is not pressed, and according to US plus a certain additive term if the shutdown button is pressed. We show that this way of combining UN and US in some sense satisfies the informal desideratum above, but has some pretty undesirable consequences.

Max Tegmark recently asked me whether it would be possible to show a more general impossibility result: that not only this particular way of combining UN and US leads to bad consequences, but that every U which makes an agent maximize UN until the shutdown button is pressed, and US after, will have these consequences---including, for example, non-linear combinations of UN and US.

As it happens, the answer to this question turns out to be either trivial or underspecified, though it may be interesting to see why this is the case. The reason is that the answer depends on how, exactly, we formalize the demand about maximizing UN before the button is pressed and US after. The U we investigate in this paper suggests one possible way of formalizing this demand (I'll give more detail below). This formalization, however, nails down exactly how our agent behaves in every possible situation: Although there is more than one utility function U which produces this behavior, all of them lead the agent to choose exactly the same actions, producing the same undesirable consequences as the U in the paper.

Alternatively, we may try to find a different formalization of the demand, which doesn't imply the same bad consequences; but then, instead of a math problem that we can answer by a proof, we're faced with

...
Intrapersonal comparisons: you might be doing it wrong.
by fowlertm — 2015-02-03T21:34:03.344Z

Cross-posted

Nothing weighty or profound today, but I noticed a failure mode in myself which other people might plausibly suffer from so I thought I'd share it.

Basically, I noticed that sometimes when I discovered a more effective way of doing something -- say, going from conventional flashcards to Anki -- I found myself getting discouraged.

I realized that it was because each time I found such a technique, I automatically compared my current self to a version of me that had had access to the technique the whole time. Realizing that I wasn't as far along as I could've been resulted in a net loss of motivation.

Now, I deliberately compare two future versions of myself, one armed with the technique I just discovered and one without. Seeing how much farther along I will be results in a net gain of motivation.

A variant of this exercise is taking any handicap you might have and wildly exaggerating it. I suffer from mild Carpal Tunnel (or something masquerading as CT) which makes progress in programming slow. When I feel down about this fact I imagine how hard programming would be without hands.

Sometimes I go as far as to plan out what I might do if I woke up tomorrow with a burning desire to program and nothing past my wrists. Well, I'd probably figure out a way to code by voice and then practice mnemonics because I wouldn't be able to write anything down. Since these solutions exist I can implement one or both of them the moment my carpal tunnel gets bad enough.

With this realization comes a boost in motivation knowing I can go a different direction if required.
On notation for modal UDT
by Benya_Fallenstein — 2015-02-03T19:26:48.000Z

Patrick thinks that the notation I've been using for talking about modal UDT is too hard to read. I agree that it's annoying that after defining a decision problem →P(→a) and choosing a decision theory →T(→u)---say, →T(→u)≡→UDT(→u)---, I every time have to explicitly give separate names to the fixed points of the decision problem and the decision theory, for example by stating that (→A,→U) are such that
GL⊢(→A↔→UDT(→U))∧(→U↔→P(→A)).

In "Obstacle to modal optimality when you're being modalized", Patrick uses a pretty free-wheeling notation loosely inspired by the notation used in the modal agents paper. As Patrick notes in that post, I'm very much not sold on this notation. Partly, this is because various pieces of the notation don't seem very consistent to me. However, there's also a somewhat subtle problem with adopting notation that's too close to the modal agents paper, which I'm worried will lead to confusion; in this post, I'll explain that problem, and suggest some possible changes to modal UDT notation that might go part of the way towards the kind of notation Patrick prefers.

Comments appreciated!

Agents and universes

In the context of modal UDT, an agent is a sequence of formulas →A≡(A1,…,Am), such that the formula Ai means that the agent takes action i, and a universe is a sequence →U≡(U1,…,Un), where Uj means that the universe returns the j'th-best outcome (1 is best, m is worst; this way, we can define modal UDT as looping over j=1 to m).

When we talk about modal agents, we write [X(Y)=C] for the formula saying that agent X cooperates when playing against agent Y. I wouldn't be averse to similarly writing, say, [→A=i] and [→U=j] for Ai and Uj, though I'm not sure whether we gain much from it. I wouldn't object too strongly to [→A()=i] and [→U()=j], though it does look a bit ugly to me.

Orthogonally, I would be ok with naming actions and perhaps outcomes when this makes a decision problem easier to understand. For example, if an agent has to decide whether or not to press a certain button, I'd be happy to write that as →A≡(Apress,Adon′t press). If we like, we can combine that with the above and write [→A=press] and [→A=don′t press], although the subscript notation feels pretty fine to me. I'm really not sold on using the symbol ¬ in the names of actions, though (e.g. "¬press"), as Patrick's post does; that seems bound to lead to confusion

...
From halting oracles to modal logic
by Benya_Fallenstein — 2015-02-03T19:26:17.000Z

When reading Vladimir's recent post on defining UDT through modal logic, I had to think a bit about his actual definition of the modal formula(s) corresponding to UDT, because it was phrased in terms of an algorithm doing things instead of an actual modal formula. Then I remembered having worked through the correspondence in the context of modal agents, and it became clear what was going on. I think Vladimir's approach is really interesting and I want to refer to it in future posts, so I thought I'd write a tutorial, since it seems likely that others will have the same problem.

Like Vladimir, let's start with the simplest case of an agent that chooses between two actions, in a world with two different possible outcomes (one of which is preferred to the other). Let's write the world as a universe program, in the usual style of proof-based UDT; for example:

def U():
if A() == a1:
return 10
else:
return 5

And, in the style of UDT with a halting oracle, let's write our agent as a program with access to a halting oracle, which it uses to figure out whether certain sentences are provable (in Peano Arithmetic, let's say):

def A():
if Provable("A() == a1 --> U() == 10"):
return a1
elif Provable("A() == a2 --> U() == 10"):
return a2
return a2

(The reference to A() and U() inside the quotes is by quining.) Our agent tries to get utility 10 by trying all possible actions in order; for each action, it sees whether it can prove that this action leads to the desired outcome, and if so, it takes that action; if it doesn't find a good action, it randomly takes the last one.

Given a programming language with access to a halting oracle, there's a relation Run(m,n) such that if m is the Gödel number of a program in this language, then N⊨Run(m,n) if and only if this program halts and returns n. (The definition of Run(m,n) is very similar to the one for programming languages without a halting oracle.) We implicitly used this fact in the definition of A(): the statement "A()=a1→U()=10" really means "Run(A,a1)→Run(U,10)".

Now, by looking at the definition of U(), we can show that
PA⊢[U()=10]↔[A()=a1],
and by looking at the definition of A(), we can show that
PA⊢[A()=a1]↔□┌[A()=a1]→[U()=10]┐.
If we ignore the internal structure of [U()=10] and [A()=a1], and treat these as propositional variables, both of the state

...
How to save (a lot of) money on flying
by RobertM — 2015-02-03T18:25:24.064Z

I was going to wait to post this for reasons, but realized that was pretty dumb when the difference of a few weeks could literally save people hundreds, if not thousands of collective dollars.

If you fly regularly (or at all), you may already know about this method of saving money. The method is quite simple: instead of buying a round-trip ticket from the airline or reseller, you hunt down much cheaper one-way flights with layovers at your destination and/or your point of origin. Skiplagged is a service that will do this automatically for you, and has been in the news recently because the creator was sued by United Airlines and Orbitz. While Skiplagged will allow you to click-through to purchase the one-way ticket to your destination, they have broken or disabled the functionality of the redirect to the one-way ticket back (possibly in order to raise more funds for their legal defense). However, finding the return flight manually is fairly easy as the provide all the information to filter for it on other websites (time, airline, etc). I personally have benefited from this - I am flying to Texas from Southern California soon, and instead of a round-trip ticket which would cost me about $450, I spent ~$180 on two one-way tickets (with the return flight being the "layover" at my point-of-origin). These are, perhaps, larger than usual savings; I think 20-25% is more common, but even then it's a fairly significant amount of money.

Relevant warnings by gwillen:

You should be EXTREMELY CAREFUL when using this strategy. It is, at a minimum, against airline policy.

If you have any kind of airline status or membership, and you do this too often, they will cancel it. If you try to do this on a round-trip ticket, they will cancel your return. If the airlines have any means of making your life difficult available to them, they WILL use it.

Obviously you also cannot check bags when using this strategy, since they will go to the wrong place (your ostensible, rather than your actual, destination.) This also means that if you have an overhead-sized carryon, and you board late and are forced to check it, your bag will NOT make it to your intended destination; it will go to the final destination marked on your ticket. If you try to argue about this, you run the risk of getting your ticket cancelled altogether, since yo

...
Superintelligence 21: Value learning
by KatjaGrace — 2015-02-03T02:01:09.407Z

This is part of a weekly reading group on Nick Bostrom's book, Superintelligence. For more information about the group, and an index of posts so far see the announcement post. For the schedule of future topics, see MIRI's reading guide.

Welcome. This week we discuss the twenty-first section in the reading guide: Value learning.

This post summarizes the section, and offers a few relevant notes, and ideas for further investigation. Some of my own thoughts and questions for discussion are in the comments.

There is no need to proceed in order through this post, or to look at everything. Feel free to jump straight to the discussion. Where applicable and I remember, page numbers indicate the rough part of the chapter that is most related (not necessarily that the chapter is being cited for the specific claim).

Reading: “Value learning” from Chapter 12

Summary

One way an AI could come to have human values without humans having to formally specify what their values are is for the AI to learn about the desired values from experience.

To implement this 'value learning' we would need to at least implicitly define a criterion for what is valuable, which we could cause the AI to care about. Some examples of criteria:

'F' where 'F' is a thing people talk about, and their words are considered to be about the concept of interest (Yudkowsky's proposal) (p197-8, box 11)

Whatever is valued by another AI elsewhere in the universe values (Bostrom's 'Hail Mary' proposal) (p198-9, box 12)

What a specific virtual human would report to be his value function, given a large amount of computing power and the ability to create virtual copies of himself. The virtual human can be specified mathematically as the simplest system that would match some high resolution data collected about a real human (Christiano's proposal). (p200-1)

The AI would try to maximize these implicit goals given its best understanding, while at the same time being motivated to learn more about its own values.

A value learning agent might have a prior probability distribution over possible worlds, and also over correct sets of values conditional on possible worlds. Then it could choose its actions to maximize their expected value, given these probabilities.

Another view

Paul Christiano describes an alternative to loading values into an AI at all:

Most thinking abo

...
Third-person counterfactuals
by Benya_Fallenstein — 2015-02-03T01:13:26.000Z

If you're thinking about the counterfactual world where you do X in the process of deciding whether to do X, let's call that a first-person counterfactual. If you're thinking about it in the process of deciding whether another agent A should have done X instead of Y, let's call that a third-person counterfactual. The definition of, e.g., modal UDT uses first-person counterfactuals, but when we try to prove a theorem showing that modal UDT is "optimal" in some sense, then we need to use third-person counterfactuals.

UDT's first-person counterfactuals are logical counterfactuals, but our optimality result evaluates UDT by using physical third-party counterfactuals: it asks, would another agent have done better, not, would a different action by the same agent have lead to a better outcome? The former is easier to analyze, but the latter seems to be what we really care about. Nate's recent post on "global UDT" points towards turning UDT into a notion of third-party counterfactuals, and describes some problems. In this post, I'll give a fuller UDT-based notion of logical third-party counterfactuals, which at least fails visibly (returns an error) in the kinds of cases Nate describes. However, in a follow-up post I'll give an example where this definition returns a non-error value which intuitively seems wrong.

Before I start, a historical side note: When Kenny Easwaran visited us for two days and we proved the UDT optimality result, the reason we decided to physical counterfactuals wasn't actually that we thought these were the better kind of counterfactuals. Rather, we actually thought explicitly about the problem of physical vs. logical third-person counterfactuals on the first morning of Kenny's visit, and decided to look at the physical counterfactuals case because it seemed easier to reason about. Which turned out to be a great decision, because---to our surprise---we very quickly ended up proving the first version of what later became the modal UDT optimality result!

But today, let's talk about logical counterfactuals. As Nate points out in his Global UDT post, there's a sort of duality between first-person and third-person counterfactuals---given a good third-person notion of counterfactuals, you can try to turn it into a first-person notion by writing an agent that evaluates actions according to it, and given a first-person notion you can try to turn it

...
Computation complexity of AGI design
by Squark — 2015-02-02T20:05:14.239Z

Summary of main point: I argue that there is a significant probability that creating de novo AGI is an intractable problem. Evolution only solved this problem because of anthropic reasons. Conclusions are drawn regarding priorities in AI risk research.

Sketch of main argument: There are suggestive relations between AGI and NP-completeness. These relations lead me to hypothesize that AGI programs posses large Levin-Kolmogorov complexity which implies that producing them is a computationally intractable problem. The timing of events in the evolution of human intelligence seems to be consistent with the assumption evolution's success is anthropic, if we postulate human intelligence as arising from a combination of two modules: an "easy" (low complexity) module and a "hard" (high complexity) module. Therefore, creating superhuman intelligence will require reverse engineering the human brain and be limited to improving the "easy" module (since creating a better "hard" module is again computationally intractable).

AGI and P vs. NP

There are several arguments the AGI problem is of a similar "flavor" to problems that are NP-complete.

The first argument is rather vague but IMO still compelling. Many class separations in complexity theory (P vs. NP, L vs. P, R vs. RE) hinge on the existence of a complete language. This means there is a single problem solving which under the stronger resource constraints would lead to solving all problems in the larger class. Similarly, Goedel incompleteness means there is no single algorithm (a program which terminates on all inputs) for proving all provable theorems. It feels like there is a principle of mathematics which rules out algorithms that are "too good to be true": a single "magic wand" to solve all problems. In a similar way, AGI is a "magic wand": it solves "all" problems because you can simply delegate them to the AGI.

Another argument has to do with Solomonoff induction. Solomonoff induction is incomputable but it becomes computable if we set a limit T on the run-time of the "hypotheses" (programs) we consider. However, the resulting computable induction carries an
O(T 2T) slow-down penalty (the time it takes to run all possible hypotheses). On the other hand, the problem is easy modulo P# and tractable given an NP-complete oracle given certain assumptions on the required probability accuracy.

Yet another argu...
AI Impacts project
by KatjaGrace — 2015-02-02T19:40:14.612Z

I've been working on a thing with Paul Christiano that might interest some of you: the AI Impacts project.

The basic idea is to apply the evidence and arguments that are kicking around in the world and various disconnected discussions respectively to the big questions regarding a future with AI. For instance, these questions:

What should we believe about timelines for AI development?

How rapid is the development of AI likely to be near human-level?

How much advance notice should we expect to have of disruptive change?

What are the likely economic impacts of human-level AI?

Which paths to AI should be considered plausible or likely?

Will human-level AI tend to pursue particular goals, and if so what kinds of goals?

Can we say anything meaningful about the impact of contemporary choices on long-term outcomes?

For example we have recently investigated technology's general proclivity to abrupt progress, surveyed existing AI surveys, and examined the evidence from chess and other applications regarding how much smarter Einstein is than an intellectually disabled person, among other things.

Some more on our motives and strategy, from our about page:

Today, public discussion on these issues appears to be highly fragmented and of limited credibility. More credible and clearly communicated views on these issues might help improve estimates of the social returns to AI investment, identify neglected research areas, improve policy, or productively channel public interest in AI.

The goal of the project is to clearly present and organize the considerations which inform contemporary views on these and related issues, to identify and explore disagreements, and to assemble whatever empirical evidence is relevant.

The project is provisionally organized as a collection of posts concerning particular issues or bodies of evidence, describing what is known and attempting to synthesize a reasonable view in light of available evidence. These posts are intended to be continuously revised in light of outstanding disagreements and to make explicit reference to those disagreements.

In the medium run we'd like to provide a good reference on issues relating to the consequences of AI, as well as to improve the state of understanding of these topics. At present, the site addresses only a small fraction of questions one might be in...