trentmkelly/LessWrong-43k huggingface dataset:
Two problems with causal-counterfactual utility indifference
by jessicata — 2016-05-26T06:21:07.000Z

These are problems both for utility indifference and the causal version of factoring out variables. Thanks to Eliezer for pointing out the second problem.

Suppose there are two utility functions, Ua and Ub. There is some button; let B be the event that a button is pressed. The button can only be pressed at some particular time t (we can adapt this example to cases where the button can get pressed at multiple times). The AI should optimize Ub if the button is pressed, and Ua if the button is not pressed. For example, Ua could be a paperclip-maximization objective, and Ub could be a shutdown utility function.

Introduce a random variable X, the result of the roll of a 3-sided die. X has 3 possible values, a, b and ∅. It's a with very small probability ϵ, b with probability ϵ, and ∅ with probability 1−2ϵ.

If X=a, then an invisible mechanism will prevent the button from depressing. If X=b, then an invisible mechanism will automatically depress the button. If X=∅, then neither mechanism activates, so the button is pushed or not based on physical force applied. The AI may observe the button push but not the event X.

Now consider a utility function:

U=q[X=a]Ua+(1−q)[X=b]Ub

for some constant q∈[0,1]. Note that we recover a variant of utility indifference by setting q very close to 1, and we recover factoring out variables by setting q to be the actual probability of the button push B (in a fixed-point manner). For now let's say q is set to 1/2.

Note that X simulates a causal counterfactual on B. Rougly, we could have instead written the objective as

U=ϵ(qE[Ua|do(¬B)]+(1−q)E[Ub|do(B)])

What will an AI optimizing this utility function do? It will optimize the average of Ua and Ub until time t, at which it will see whether the button has been pressed or not, and then optimize either Ua or Ub depending on the button's state. Note that it will attempt to acquire resources before time t and delay important decisions until after time t, due to value of information. It has no reason to push the button or prevent the button from being pushed; this is because it only cares about worlds where X=a or X=b, and in these worlds, physical force applied to the button has no effect.

This has some desirable properties, but ultimately fails. Consider the following two problems:

Problem 1: Bad bets

The AI only cares about worlds where X≠∅. Therefore, it will "believe" that B is u

...
Iterated Gambles and Expected Utility Theory
by Sable — 2016-05-25T21:29:27.645Z

The Setup

I'm about a third of the way through Stanovich's Decision Making and Rationality in the Modern World. Basically, I've gotten through some of the more basic axioms of decision theory (Dominance, Transitivity, etc).

As I went through the material, I noted that there were a lot of these:

Decision 5. Which of the following options do you prefer (choose one)?

A. A sure gain of $240

B. 25% chance to gain $1,000 and 75% chance to gain nothing

The text goes on to show how most people tend to make irrational choices when confronted with decisions like this; most strikingly was how often irrelevant contexts and framing effected people's decisions.

But I understand the decision theory bit; my question is a little more complicated.

When I was choosing these options myself, I did what I've been taught by the rationalist community to do in situations where I am given nice, concrete numbers: I shut up and I multiplied, and at each decision choose the option with the highest expected utility.

Granted, I equated dollars to utility, which Stanovich does mention that humans don't do well (see Prospect Theory).

The Problem

In the above decision, option B clearly has the higher expected utility, so I chose it. But there was still a nagging doubt in my mind, some part of me that thought, if I was really given this option, in real life, I'd choose A.

So I asked myself: why would I choose A? Is this an emotion that isn't well-calibrated? Am I being risk-averse for gains but risk-taking for losses?

What exactly is going on?

And then I remembered the Prisoner's Dilemma.

A Tangent That Led Me to an Idea

Now, I'll assume that anyone reading this has a basic understanding of the concept, so I'll get straight to the point.

In classical decision theory, the choice to defect (rat the other guy out) is strictly superior to the choice to cooperate (keep your mouth shut). No matter what your partner in crime does, you get a better deal if you defect.

Now, I haven't studied the higher branches of decision theory yet (I have a feeling that Eliezer, for example, would find a way to cooperate and make his partner in crime cooperate as well; after all, rationalists should win.)

Where I've seen the Prisoner's Dilemma resolved is, oddly enough, in Dawkin's The Selfish Gene, wh...
Is logic epistemically appropriate?
by abramdemski — 2016-05-25T04:07:46.000Z

One way of viewing my recent posts is as an argument against using logic in the first place. Although I had the argument in the back of my mind while writing, I doubt it came across (partly because I don't actually agree with the conclusion). Here's how it would go.

No computable probability distribution can be coherent with respect to first-order logic. This follows from seperability concerns. We're forced to put some probability on inconsistent outcomes. But really, it's much worse: consistent outcomes have measure zero in any computable probability distribution. This doesn't mean that we have to assign probability one to "false" or anything like that; the whole point is that we can't recognize all the consequences of our beliefs. Uncertainty about logical consequences translates into putting probability on logically inconsistent possibilities.

Up until I started thinking about betting on logical coins (a concept Wei Dai illustrated earlier with counterfactual mugging with a logical coin), I thought this problem would be solved by a sequence of probability distributions, P1,P2,P3... which approach logically consistent beliefs. This approach can be augmented by other good properties such as uniform coherence. This is still the most promising overall approach. However:

Problems like counterfactual mugging with a logical coin call for a logically updateless decision theory.

Even for problems which do not need an updateless solution, an agent which notices that the progression from Pn to Pn+1 is not a Bayesian update on the new information will tend to self-modify to "fix" that. Where the future self will predictably vary from Bayesian updates on new knowledge, it will achieve lower expected utility, almost by definition; this is formalized in the Diachronic Dutch Book argument. Hence, the agent will want to get rid of all the wonderful non-Bayesian machinery for logical uncertainty. As Scott put it (in private conversation), this is the two-update problem for decision theory.

If we do use a Bayesian update for logical information, we become trollable; our probability estimates can be manipulated by selective proofs. Furthermore, because the set of provable things has measure zero at any time, we diverge rather than converge as we learn more and more; I have not proved that this happens in all cases yet, but the situation does not look good. The AI might think

...
Wrong however unnamed
by  — 2016-05-24T13:55:17.270Z

Related to: 37 ways that words can be wrong.

Consider the following sentence (from Internet; but I have heard it before): 'Lichens consist of fungi and algae, but they are more than the sum of their constituents.'

It is supposed to say something like 'the fungus and the alga don't just live very close to each other, they influence each other's habitat(s) and can be considered, for most purposes, to form a physiologically integrated body'. It never actually says that, although people gradually come to this conclusion if they look at illustrations or read long enough. And I don't think the phrase is sufficiently catchy to explain its popularity; rather, that it is a tenuous introduction to the much-later-explained term 'synergism'. A noble (in principle) preparation of the mind.

Yet how is a lichen 'more than the sum of fungus and alga'? I suppose one could speak of a 'sum' if the lichen was pulverized and consumed as medicine, and then its effect on the patient was compared to that of the mixture of similarly treated fungus (grown how exactly?) and alga (same here). It doesn't exist in the wild. It shouldn't exist in the literature.

A child is not bothered by its lack of sense. When she encounters 'synergism', she'll remember having been told of something like it, and be reassured by the unity of science. It flies under the radar of 'established biological myths', because it doesn't have enough meaning to be one.

I picked a dictionary of zoological terms and tried to recall how the notions were put before me for the first time, but of course I failed. (I guess it should be high-level things, like 'variability', or colloquial expressions - 'bold as a lion', etc., that distort and get distorted the most.) They seem to 'have always been there'. Then, I looked at the definitions and tried to imagine them misapplied (intuitively, a simpler task). No luck. Yet someday, something other truly unknown to me will appear familiar and simple.

We can weed out improper concepts from textbooks, but there are too many sources which are written far more engagingly and 'clearly', and which propagate not even wrong ideas. Explained like I'm five.

And never named.
The AI in Mary's room
by Stuart_Armstrong — 2016-05-24T13:19:29.849Z

In the Mary's room thought experiment, Mary is a brilliant scientist in a black-and-white room who has never seen any colour. She can investigate the outside world through a black-and-white television, and has piles of textbooks on physics, optics, the eye, and the brain (and everything else of relevance to her condition). Through this she knows everything intellectually there is to know about colours and how humans react to them, but she hasn't seen any colours at all.

After that, when she steps out of the room and sees red (or blue), does she learn anything? It seems that she does. Even if she doesn't technically learn something, she experiences things she hadn't ever before, and her brain certainly changes in new ways.

The argument was intended as a defence of qualia against certain forms of materialism. It's interesting, and I don't intent to solve it fully here. But just like I extended Searle's Chinese room argument from the perspective of an AI, it seems this argument can also be considered from an AI's perspective.

Consider a RL agent with a reward channel, but which currently receives nothing from that channel. The agent can know everything there is to know about itself and the world. It can know about all sorts of other RL agents, and their reward channels. It can observe them getting their own rewards. Maybe it could even interrupt or increase their rewards. But, all this knowledge will not get it any reward. As long as its own channel doesn't send it the signal, knowledge of other agents rewards - even of identical agents getting rewards - does not give this agent any reward. Ceci n'est pas une récompense.

This seems to mirror Mary's situation quite well - knowing everything about the world is no substitute from actually getting the reward/seeing red. Now, a RL's agent reward seems closer to pleasure than qualia - this would correspond to a Mary brought up in a puritanical, pleasure-hating environment.

Closer to the original experiment, we could imagine the AI is programmed to enter into certain specific subroutines, when presented with certain stimuli. The only way for the AI to start these subroutines, is if the stimuli is presented to them. Then, upon seeing red, the AI enters a completely new mental state, with new subroutines. The AI could know everything about its programming, and about the stimulus, and, intellectually, what wo...
Meetup : San Francisco Meetup: Projects
by rocurley — 2016-05-23T06:54:45.118Z

Discussion article for the meetup : San Francisco Meetup: Projects

WHEN:
23 May 2016 06:15:00PM (-0700)

WHERE:
1597 Howard St. San Francisco, CA

We'll be meeting to work on projects. Bring something to work on, come up with something there, or help other people out!

As always, I can be reached at 301-458-0764 if you need to be let in.

Discussion article for the meetup : San Francisco Meetup: Projects
Open Thread May 23 - May 29, 2016
by Gunnar_Zarncke — 2016-05-22T21:11:56.868Z

If it's worth saying, but not worth its own post (even in Discussion), then it goes here.

Notes for future OT posters:

1. Please add the 'open_thread' tag.

2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)

3. Open Threads should be posted in Discussion, and not Main.

4. Open Threads should start on Monday, and end on Sunday.
Meetup : Less Wrong, More Summer, Munich Picnic
by jasticE — 2016-05-22T16:21:37.597Z

Discussion article for the meetup : Less Wrong, More Summer, Munich Picnic

WHEN:
04 June 2016 01:00:00PM (+0200)

WHERE:
Tuerkenstr. 29, Muenchen

Location: Cafe Katzentempel, Türkenstr. 29

See Meetup event for updates: http://www.meetup.com/LessWrongMunich/events/231308219/

Discussion article for the meetup : Less Wrong, More Summer, Munich Picnic
Weekly LW Meetups
by FrankAdamek — 2016-05-20T16:07:26.502Z

This summary was posted to LW Main on May 20th. The following week's summary is here.

Irregularly scheduled Less Wrong meetups are taking place in:

Cologne meetup May: 21 May 2016 05:00PM
European Community Weekend: 02 September 2016 03:35PM
San Antonio Meetup - Double Crux Game: 22 May 2016 02:00PM
Sao Paulo - Meetup de maio: 21 May 2016 02:00PM

The remaining meetups take place in cities with regular scheduling, but involve a change in time or location, special meeting content, or simply a helpful reminder about the meetup:

[Moscow] Games in Kocherga club: FallacyMania, Zendo, Tower of Chaos, Training game: 25 May 2016 07:40PM
Sydney Rationality Dojo - June: 05 June 2016 04:00PM
Sydney Rationality Dojo - July: 03 July 2016 04:00PM

Locations with regularly scheduled meetups: Austin, Berlin, Boston, Brussels, Buffalo, Canberra, Columbus, Denver, Kraków, London, Madison WI, Melbourne, Moscow, New Hampshire, New York, Philadelphia, Research Triangle NC, San Francisco Bay Area, Seattle, Sydney, Tel Aviv, Toronto, Vienna, Washington DC, and West Los Angeles. There's also a 24/7 online study hall for coworking LWers and a Slack channel for daily discussion and online meetups on Sunday night US time.

If you'd like to talk with other LW-ers face to face, and there is no meetup in your area, consider starting your own meetup; it's easy (more resources here). Check one out, stretch your rationality skills, build community, and have fun!

In addition to the handy sidebar of upcoming meetups, a meetup overview is posted on the front page every Friday. These are an attempt to collect information on all the meetups happening in upcoming weeks. The best way to get your meetup featured is still to use the Add New Meetup feature, but you'll also have the benefit of having your meetup mentioned in a weekly overview. These overview posts are moved to the discussion section when the new post goes up.

Please note that for your meetup to appear in the weekly meetups feature, you need to post your meetup before the Friday before your meetup!

If you check Less Wrong irregularly, consider subscribing to one or more city-specific mailing list in order to be notified when an irregular meetup is happening: Atlanta, Cambridge UK, Chicago, Cincinnati, Cleveland, Frankfurt, Helsinki, Marin CA, Ottawa, Pittsburgh, Portland, Salt Lake City, Southern Cali...
You can't beat a troll by predicting it.
by abramdemski — 2016-05-20T08:19:31.000Z

My trollability post showed that any computable probability distribution on logic which updates on proofs in a Bayesian way can in principle be manipulated into nearly-arbitrary beliefs by an adversary. (At least, this is true in my GLS-based treatment, and in a version which updates on theorems directly rather than on their provability.) Here, I sketch a result which suggests that it's fairly easy to do so, blocking some potential routes around the problem. I will use notation and concepts from the original trollability post.

First, note that corollary 1 does not depend on the odds ration 1/2 which Theorem 1 implies we can obtain. A weaker troll might not have the computational power to search for proofs which so much leverage over our beliefs. All the troll needs is a sequence of theorems tn such that the product of the likelihood ratios P(□tn|◊x)/P(□tn|¬◊x) goes to zero. (Note: what Theorem 1 establishes is exactly that we can always find t so that this likelihood ratio is less than or equal to 1/2.) The likelihood ratio doesn't even need to be bounded away from one. It could be that each new t is able to reduce the probability less and less (meaning the likelihood ratio is closer and closer to one), but the overall effect is still to modify the probability of x arbitrarily. This is guaranteed if the product of the ratios converges to zero.

(Analogously, we would talk about this product converging to infinity for corollary 2. I'll deal just with the case of driving probabilities down, here; as in the previous post, the idea is that if we can't keep probabilities from being driven arbitrarily down, we also cannot keep them from being driven arbitrarily up, so we get non-convergence.)

It seems natural to hope, then, that an agent could avoid manipulation by predictable trolls. If the agent is predicting the troll quite accurately, then it must be placing high probability on □t ahead of time; so, the likelihood ratio must be close to one. If the predictions converge to probability one very quickly, then perhaps the likelihood ratios can converge quickly enough to avoid the divergence result.

This turns out to be impossible for fairly simple trolls.

Before sketching the proof, I should explain what I am not doing. I am not saying that an agent cannot block a troll by beating a troll to theorems. This totally works: if the agent already knows a theorem by the time

...
Knowledge Dump: Pomodoros
by ChristianKl — 2016-05-19T16:13:25.646Z

After our recent LW Dojo in Berlin we had a conversation on our mailing list about pomodoros.

How do we handle it if the bell rings but we are in flow? Is it good to honor the bell and take a pause or is it more effective to continue working to keep in flow?

The original setting of 25 minutes came from the 25 minutes that Francesco Cirillo tomato shaped timer had naturally. The LW Study Hall seems to use 32 minutes work with 8 minutes pause. If you have experimented with different lengths, what worked for you?

Did you come to any surprising conclusions about pomodoros while working with them, that might be interesting to other people?
Meetup : Games in Kocherga club: FallacyMania, Zendo, Tower of Chaos, Training game
by Alexander230 — 2016-05-19T13:58:51.833Z

Discussion article for the meetup : Games in Kocherga club: FallacyMania, Zendo, Tower of Chaos, Training game

WHEN:
25 May 2016 07:40:00PM (+0300)

WHERE:
Moscow, B.Dorogomilovskaya, 5-2

Welcome to Moscow LW community makeshift games! In that games, some rationality skills are involved, so you can practise while you playing!

FallacyMania: it is a game where you guess logical fallacies in arguments, or practise using logical fallacies yourself (depending on team in which you will be).

Game rules: https://goo.gl/JoFl7F

Zendo: guessing the rules of items placement on the table.

Game rules: https://goo.gl/J7KvIF

Tower of Chaos: funny game with guessing the rules of human placement on a Twister mat.

Game rules: https://goo.gl/u9qgc3

Training Game Party: all players try to train the subject to guess and make some action without words, using only positive reinforcement.

Game rules: https://goo.gl/mNT7J3

Come to antikafe "Kocherga", ul.B.Dorogomilovskaya, 5-2. The map is here: http://kocherga-club.ru/#contacts . Nearest metro station is Kievskaya. If you are lost, call Sasha at +7-905-527-30-82.

Games begin at 19:40, the length is 3 hours.

Discussion article for the meetup : Games in Kocherga club: FallacyMania, Zendo, Tower of Chaos, Training game
Teachers: Much More Than You Wanted To Know
by Scott Alexander — 2016-05-19T06:13:32.000Z

[Epistemic status: This is really complicated, this is not my field, people who have spent their entire lives studying this subject have different opinions, and I don’t claim to have done more than a very superficial survey. I welcome corrections on the many inevitable errors.]

I.

Newspapers report that having a better teacher for even a single grade (for example, a better fourth-grade teacher) can improve a child’s lifetime earning prospects by $80,000. Meanwhile, behavioral genetics studies suggest that a child’s parents have minimal (non-genetic) impact on their future earnings. So one year with your fourth-grade teacher making you learn fractions has vast effects on your prospects, but twenty-odd years with your parents shaping you at every moment doesn’t? Huh? I decided to try to figure this out by looking into the research on teacher effectiveness more closely.

First, how much do teachers matter compared to other things? To find out, researchers take a district full of kids with varying standardized test scores and try to figure out how much of the variance can be predicted by what school the kids are in, what teacher’s class the kids are in, and other demographic factors about the kids. So for example if the test scores of two kids in the same teacher’s class were on average no more similar than the test scores of two kids in two different teachers’ classes, then teachers can’t matter very much. But if we were consistently seeing things like everybody in Teacher A’s class getting A+s and everyone in Teacher B’s class getting Ds, that would suggest that good teachers are very important.

Here are the results from three teams that tried this (source, source, source):

These differ a little in that the first one assumes away all noise (“unexplained variance”) and the latter two keep it in. But they all agree pretty well that individual factors are most important, followed by school and teacher factors of roughly equal size. Teacher factors explain somewhere between 5% and 20% of the variance. Other studies seem to agree, usually a little to the lower end. For example, Goldhaber, Brewer, and Anderson (1999) find teachers explain 9% of variance; Nye, Konstantopoulos, and Hedges (2004) find they explain 13% of variance for math and 7% for reading. The American Statistical Association summarize...
Meetup : San Antonio Meetup - Double Crux Game
by Bryan-san — 2016-05-18T18:36:19.456Z

Discussion article for the meetup : San Antonio Meetup - Double Crux Game

WHEN:
22 May 2016 02:00:00PM (-0500)

WHERE:
818 nw loop 410 78216

Meetup to discuss rationality and all things LessWrong at Earth Burger. Earth Burger is a claimed "Upbeat fast-food joint supplying vegetarian burgers & sandwiches, with a drive-thru & vegan options".

Topic of the Week: The Double Crux Game

Look for the sign that says "LW".

Discussion article for the meetup : San Antonio Meetup - Double Crux Game
How do you learn Solomonoff Induction?
by aisarka — 2016-05-17T17:47:56.379Z

I read about a fascinating technique described on Wikipedia as a mathematically formalized combination of Occam's razor and the Principle of Multiple Explanations. I want to add this to my toolbox. I'm dreaming of a concise set of actionable instructions for using Solomonoff induction. I realize this wish might be overly idealistic. I'm willing to peruse a much more convoluted tome and will consider making time for any background knowledge or prerequisites involved.

If anyone knows of a good book on this, or can tell me what set of information I need to acquire, please let me know. It would be much appreciated!