trentmkelly/LessWrong-43k huggingface dataset:
How can we compare decision theories?
by bentarm — 2010-08-18T13:29:48.425Z

There has been a lot of discussion on LW about finding better decision theories. A lot of the reason for the various new decision theories proposed here seems to be an effort to get over the fact that classical CDT gives the wrong answer in 1-shot PD's, Newcomb-like problems and Parfit's Hitchhiker problem. While Gary Drescher has said that TDT is "more promising than any other decision theory I'm aware of ", Eliezer gives a list of problems in which his theory currently gives the wrong answer (or, at least, it did a year ago). Adam Bell's recent sequence has talked about problems for CDT, and is no doubt about to move onto problems with EDT (in one of the comments, it was suggested that EDT is "wronger" than CDT).

In the Iterated Prisoner's Dilemma, it is relatively trivial to prove that no strategy is "optimal" in the sense that it gets the best possible pay-out against all opponents. The reasoning goes roughly like this: any strategy which ever cooperates does worse than it could have against, say, Always Defect. Any strategy which doesn't start off with cooperate does worse than it could have against, say Grim. So, whatever strategy you choose, there is another strategy that would do better than you against some possible opponent. So no strategy is "optimal". Question: is it possible to prove similarly that there is no "optimal" Decision Theory? In other words - given a decision theory A, can you come up with some scenario in which it performs worse than at least one other decision theory? Than any other decision theory?

One initial try would be: Omega gives you two envelopes - the left envelope contains $1 billion iff you don't implement decision theory A in deciding which envelope to choose. The right envelope contains $1000 regardless.

Or, you might not like Omega being able to make decisions about you based entirely on your sourcecode (or "ritual of cognition"), then how about this:in order for two decision theories to sensibly be described as "different", there must be some scenario in which they perform a different action (let's call this Scenario 1). In Scenario 1, DT A makes decision A whereas DT B makes decision B. In Scenario 2, Omega offers you the following setup: here are two envelopes, you can pick exactly one of them. I've just simulated you in Scenario 1. If you chose decision B, there's $1,000,000 in the left envelope. Otherwise it's empty....
Other Existential Risks
by multifoliaterose — 2010-08-17T21:24:51.520Z

[Added 02/24/14: SIAI (now MIRI) has evolved substantially since 2010 when I wrote this post, and the criticisms made in the post don't apply to MIRI as presently constituted.]

Related To: Should I believe what the SIAI claims?, Existential Risk and Public Relations

In his recent post titled Should I believe what the SIAI claims? XiXiDu wrote:

I'm already unable to judge what the likelihood of something like the existential risk of exponential evolving superhuman AI is compared to us living in a simulated reality. Even if you tell me, am I to believe the data you base those estimations on?

And this is what I'm having trouble to accept, let alone look through. There seems to be a highly complicated framework of estimations to support and reinforce each other. I'm not sure how you call this in English, but in German I'd call this a castle in the air.

[...]

I can however follow much of the reasoning and arguments on this site. But I'm currently unable to judge their overall credence. That is, are the conclusions justified? Is the coherent framework build around the SIAI based on firm ground?

[...]

I'm concerned that although consistently so, the LW community is updating on fictional evidence. This post is meant to inquire the basic principles, the foundation of the sound argumentation's and the basic premises that they are based upon.

XiXidu's post produced mixed reactions within the LW community. On one hand, some LW members (e.g. orthonormal) felt exasperated with XiXiDu because his post was poorly written, revealed him to be uninformed, and revealed that he has not internalized some of the basic principles of rationality. On the other hand, some LW members (e.g. HughRistik) have long wished that SIAI would attempt to substantiate some of its more controversial claims in detail and were gratified to see somebody call on SIAI to do so. These two categories are not mutually exclusive. I fall into both in some measure. In any case, I give XiXiDu considerable credit for raising such an important topic.

The present post is the first of a several posts in which I will detail my thoughts on SIAI's claims.

One difficulty is that there's some ambiguity as to what SIAI's claims are. I encourage SIAI to make a more detailed public statement of their most fundamental claims. According to the SIAI website:

In the coming decades, humanity will likely create a powerful

...
On behalf of physical things
by KatjaGrace — 2010-08-17T15:42:36.000Z

Most people inadvertently affect the reputations of groups they are seen as part of while they go about other activities. But some people also purposely exploit the fact that their behaviour and thoughts will be seen as evidence of those of a larger group, to give the false impression their views are widely supported. These people are basically stealing the good reputation of groups; they enjoy undeserved attention and leave the groups’ images polluted.

Such parasites often draw attention to what a very ordinary member of the targeted group they are, or just straight out claim to be speaking for that group. People who ‘have been a left voter for fifty years, but this year might just have to vote conservative’ are getting much of their force from implicitly claiming high representativeness of a large and respected group, and those who claim they write ‘what women really think‘ are more overt. From the perspective of women who think for instance, this is almost certain to be a damaging misrepresentation; any view other than your own is worse, and people who have good arguments are less likely to steal the authority of some unsuspecting demographic as support. It is also costly to listeners who are mislead, for instance about the extent to which women really think. Costs of prevention ignored then, less of this is better.

Purposeful exploitation of this sort should be easier than other externalities to groups’ reputations to punish and to want to punish; it’s easier to see, it’s directed at a specific group, and it’s more malevolent. However the public can’t punish or ignore all claims or implicit suggestions of representativeness, as there are also many useful and accurate ones. Often much of the interest in learning what specific strangers’ views are requires assuming that they are representative, and we keenly generalize this way. So mostly it is up to groups to identify and punish their own dishonest exploiters, usually via social pressure.

This means groups are easier to exploit if their members aren’t in a position to punish, because they don’t have the resources to deny respect that matters to the offenders. If you claim to be broadcasting what women think, most women don’t have the time or means to publicize the shamefulness of your malicious externalizing much. Even if the...
What a reduction of "probability" probably looks like
by cousin_it — 2010-08-17T14:58:38.876Z

Unlike my previous posts, this one isn't an announcement of some finished result. I just want to get some ideas out for public discussion. A big part of the credit goes to Wei Dai and Vladimir Nesov, though the specific formulations are mine.

Wei Dai wonders: what are probabilities, anyway? Eliezer wonders: what are the Born probabilities of? I cannot claim to know the answers, but I strongly hold that these questions are, in fact, answerable. And as evidence, I'll try to show how normal the answers might plausibly turn out to be.

Perhaps counterintuitively, the easiest way for probabilities to arise is not by postulating "different worlds" that you could "end up" in starting from now. No, the easiest setting is a single, purely deterministic world with only one possible future.

One

The first thought experiment goes like this. Imagine a coarse-grained classical universe whose physical laws are allowed to consume symbols from a "Tape", located outside the Matrix, which contains a sequence of ones and zeroes coming from a pseudorandom number generator. If we flip a coin in that world, the result of the flip will depend on several consecutive ones and zeroes read from the Tape. If the coin is "fair", it will come up heads 50% of the time, on average, as we advance along the Tape. (Yes, Virginia, in this setting the fairness would be a mathematical property of the coin, not of our own ignorance.) In such a world, creatures who are too computationally weak to predict the Tape will probably find useful a concept of "probability", and will indeed find "probabilistic" events happening with the limiting frequencies, standard deviations, etc. predicted by our familiar probability theory - even though the world has only one timeline that never branches.

But we know that in our world, observations are not completely deterministic: they can be influenced by the mysterious Born probabilities. How could that kind of law arise from a Nature that doesn't contain it already?

Two

To understand the second thought experiment, you need to be able to imagine the MWI without the Born probabilities: just a big wavefunction evolving according to the usual laws, without any observers who could collapse it or experience probabilities (whatever that means). And imagine an outside observer that samples it according to different probability measures, and sees differe...
Desirable Dispositions and Rational Actions
by RichardChappell — 2010-08-17T03:20:06.657Z

A common background assumption on LW seems to be that it's rational to act in accordance with the dispositions one would wish to have. (Rationalists must WIN, and all that.)

E.g., Eliezer:

It is, I would say, a general principle of rationality - indeed, part of how I define rationality - that you never end up envying someone else's mere choices. You might envy someone their genes, if Omega rewards genes, or if the genes give you a generally happier disposition. But [two-boxing] Rachel, above, envies [one-boxing] Irene her choice, and only her choice, irrespective of what algorithm Irene used to make it. Rachel wishes just that she had a disposition to choose differently.

And more recently, from AdamBell:

I [previously] saw Newcomb’s Problem as proof that it was sometimes beneficial to be irrational. I changed my mind when I realized that I’d been asking the wrong question. I had been asking which decision would give the best payoff at the time and saying it was rational to make that decision. Instead, I should have been asking which decision theory would lead to the greatest payoff.

Within academic philosophy, this is the position advocated by David Gauthier. Derek Parfit has constructed some compelling counterarguments against Gauthier, so I thought I'd share them here to see what the rest of you think.

First, let's note that there definitely are possible cases where it would be "beneficial to be irrational". For example, suppose an evil demon ('Omega') will scan your brain, assess your rational capacities, and torture you iff you surpass some minimal baseline of rationality. In that case, it would very much be in your interests to fall below the baseline! Or suppose you're rewarded every time you honestly believe the conclusion of some fallacious reasoning. We can easily multiply cases here. What's important for now is just to acknowledge this phenomenon of 'beneficial irrationality' as a genuine possibility.

This possibility poses a problem for the Eliezer-Gauthier methodology. (Quoting Eliezer again:)

Rather than starting with a concept of what is the reasonable decision, and then asking whether "reasonable" agents leave with a lot of money, start by looking at the agents who leave with a lot of money, develop a theory of which agent

...
Newcomb's Problem: A problem for Causal Decision Theories
by  — 2010-08-16T11:25:20.745Z

This is part of a sequence titled, "Introduction to decision theory"

The previous post is "An introduction to decision theory"

In the previous post I introduced evidential and causal decision theories. The principle question that needs resolving with regards to these is whether using these decision theories leads to making rational decisions. The next two posts will show that both causal and evidential decision theories fail to do so and will try to set the scene so that it’s clear why there’s so much focus given on Less Wrong to developing new decision theories.

Newcomb’s Problem

Newcomb’s Problem asks us to imagine the following situation:

Omega, an unquestionably honest, all knowing agent with perfect powers of prediction, appears, along with two boxes. Omega tells you that it has placed a certain sum of money into each of the boxes. It has already placed the money and will not now change the amount. You are then asked whether you want to take just the money that is in the left hand box or whether you want to take the money in both boxes.

However, here’s where it becomes complicated. Using its perfect powers of prediction, Omega predicted whether you would take just the left box (called “one boxing”) or whether you would take both boxes (called “two boxing”).Either way, Omega put $1000 in the right hand box but filled the left hand box as follows:

If he predicted you would take only the left hand box, he put $1 000 000 in the left hand box.

If he predicted you would take both boxes, he put $0 in the left hand box.

Should you take just the left hand box or should you take both boxes?

An answer to Newcomb’s Problem

One argument goes as follows: By the time you are asked to choose what to do, the money is already in the boxes. Whatever decision you make, it won’t change what’s in the boxes. So the boxes can be in one of two states:

Left box, $0. Right box, $1000.

Left box, $1 000 000. Right box, $1000.

Whichever state the boxes are in, you get more money if you take both boxes than if you take one. In game theoretic terms, the strategy of taking both boxes strictly dominates the strategy of taking only one box. You can never lose by choosing both boxes.

The only problem is, you do lose. If you take two

...
Kevin T. Kelly's Ockham Efficiency Theorem
by Johnicholas — 2010-08-16T04:46:00.253Z

There is a game studied in Philosophy of Science and Probably Approximately Correct (machine) learning. It's a cousin to the Looney Labs game "Zendo", but less fun to play with your friends. http://en.wikipedia.org/wiki/Zendo_(game) (By the way, playing this kind of game is excellent practice at avoiding confirmation bias.) The game has two players, who are asymmetric. One player plays Nature, and the other player plays Science. First Nature makes up a law, a specific Grand Unified Theory, and then Science tries to guess it. Nature provides some information about the law, and then Science can change their guess, if they want to. Science wins if it converges to the rule that Nature made up.

Kevin T. Kelly is a philosopher of science, and studies (among other things) the justification within the philosophy of science for William of Occam's razor: "entities should not be multiplied beyond necessity". The way that he does this is by proving theorems about the Nature/Science game with all of the details elaborated.

Why should you care? Firstly, his justification is different from the overfitting justification that sometimes shows up in Bayesian literature. Roughly speaking, the overfitting justification characterizes our use of Occam's razor as pragmatic - we use Occam's razor to do science because we get good generalization performance from it. If we found something else (e.g. boosting and bagging, or some future technique) that proposed oddly complex hypotheses, but achieved good generalization performance, we would switch away from Occam's razor.

An aside regarding boosting and bagging: These are ensemble machine learning techniques. Suppose you had a technique that created decision tree classifiers, such as C4.5 (http://en.wikipedia.org/wiki/C4.5_algorithm), or even decision stumps (http://en.wikipedia.org/wiki/Decision_stump). Adaboost (http://en.wikipedia.org/wiki/Adaboost) would start by weighting all of the examples identically and invoking your technique to find an initial classifier. Then it would reweight the examples, prioritizing the ones that the first iteration got wrong, and invoke your technique on the reweighted input. Eventually, Adaboost outputs an ensemble of decision trees (or decision s...
Statistical discrimination is externality deliniation
by KatjaGrace — 2010-08-15T10:00:07.000Z

Discrimination based on real group average characteristics is a kind of externality within groups. Observers choose which groups to notice, then the behaviour of those in the groups alters the overall reputation of the group. We mostly blame those who choose the groups for this, not those who externalize within them. But if we somehow stopped thinking in terms of any groups other than the whole population, the externality would still exist, you just wouldn’t notice it because it would be amongst all humans equally. If someone cheated you, you you would expect all people to cheat you a little more, whereas now you may notice the cheater’s other characteristics and put most of the increased expectation on similar people, such as Lebanese people or men.

Does this perspective change where to lay blame for the harm caused by such discrimination? A bit, if the point of blame is to change behaviour. Changing the behaviour of the category makers is still useful, though we probably try to change them in the wrong direction sometimes. But another option is to deal with the externalities in the usual fashion: subsidise positive externalities and tax negative ones. This is done via social pressure within some groups. Families often use such a system, thus the derision given for ‘bringing shame to the family’, along with the rewards of giving parents something to accidentally mention to their friends. Similar is seen in schools and teams sometimes I think, and in the occasional accusation ‘you give x a bad name!’, though that is often made by someone outside the group. I haven’t heard of it done much in many other groups or via money rather than social pressure. Are there more such examples?

One reason it is hard to enforce accountability for such externalities is that boundaries of groups are often quite unclear, and people near the edge feel unfairly treated if they fall on the more costly side. The less clear is the group boundary the more people are near the edge. Plus people toward the edge might only be seen as in the group a quarter of the time or something, so they aren’t externalizing or being externalized to so much. Families are a relatively clearly bounded group, so it is easier for them to punish and reward effects on family reputation. Gender is a relatively clear boundary too (far from completely clear, but mo...
Existential Risk and Public Relations
by multifoliaterose — 2010-08-15T07:16:32.802Z

[Added 02/24/14: Some time after writing this post, I discovered that it was based on a somewhat unrepresentative picture of SIAI. I still think that the concerns therein were legitimate, but they had less relative significance than I had thought at the time. SIAI (now MIRI) has evolved substantially since 2010 when I wrote this post, and the criticisms made in the post don't apply to MIRI as presently constituted.]

A common trope on Less Wrong is the idea that governments and the academic establishment have neglected to consider, study and work against existential risk on account of their shortsightedness. This idea is undoubtedly true in large measure. In my opinion and in the opinion of many Less Wrong posters, it would be very desirable to get more people thinking seriously about existential risk. The question then arises: is it possible to get more people thinking seriously about existential risk? A first approximation to an answer to this question is "yes, by talking about it." But this answer requires substantial qualification: if the speaker or the speaker's claims have low credibility in the eyes of the audience then the speaker will be almost entirely unsuccessful in persuading his or her audience to think seriously about existential risk. Speakers who have low credibility in the eyes of an audience member decrease the audience member's receptiveness to thinking about existential risk. Rather perversely, speakers who have low credibility in the eyes of a sufficiently large fraction of their audience systematically raise existential risk by decreasing people's inclination to think about existential risk. This is true whether or not the speakers' claims are valid.

As Yvain has discussed in his excellent article titled The Trouble with "Good"

To make an outrageous metaphor: our brains run a system rather like Less Wrong's karma. You're allergic to cats, so you down-vote "cats" a couple of points. You hear about a Palestinian committing a terrorist attack, so you down-vote "Palestinians" a few points. Richard Dawkins just said something especially witty, so you up-vote "atheism". High karma score means seek it, use it, acquire it, or endorse it. Low karma score means avoid it, ignore it, discard it, or condemn it.

When Person X makes a claim which an audience member finds uncredible, the audience member's brain (semiconsciously) ma...
Why you don’t seek friends dating site style
by KatjaGrace — 2010-08-14T10:00:32.000Z

Robin asks an interesting question:

Bryan Caplan recently pointed out to a few of us that while many dating web sites offer to help you find matching romantic mates, there are far fewer friend finding helpers. We tend to collect friends informally, by liking the people we meet for other reasons, and especially friends of friends. But for mating purposes we are more willing to choose folks based on a list of their interests, an intro paragraph, a picture, etc. Why the difference?

His theory:

We need mates more for their simple surface features, while we need friends more to serve as social allies in our existing social network. Since we need friends in substantial part to serve as allies in our social world, supporting us against opposing coalitions, it makes sense to draw our friends from our existing social world. And since we need mates more for their personal quality, e.g., good genes, youth, wealth, smarts, mood, etc., it makes sense to pick them more via such features.

I have a different theory, though I’m not especially confident in it. First notice that people are actually often eager to make friends with people outside their social circle. They don’t want to make friends on the subway, but they join groups, play sports, couchsurf, partake in a huge range of social online activities, and go to conferences often with the intention of making new friends, who would be outside their existing social circle. A difference between any of these activities and online dating is that with the latter you have to be explicit about the fact that you are trying each other out when you go on a date. It is obvious when one of you decides against the other, and the relationship is usually sharply ended. With meeting friends casually this is not so; you can talk to people and assess them a lot before anybody even knows you are considering being friends with them. Even once you have done some friendly thing with a person, if you don’t see them for months its not clear whether you hated them or have just been busy. Friend meeting activities are best then with a group of people and a supposed other purpose to the interaction.

I think this latter style is necessary for friends but not for romance because you can have many friends and only one partner, which makes turning someone down as a friend much ruder. Turning down someone as a partner says ‘I...