trentmkelly/LessWrong-43k huggingface dataset:
Re-understanding Robin Hanson’s “Pre-Rationality”
by Wei Dai — 2009-11-03T02:58:04.599Z

I’ve read Robin’s paper “Uncommon Priors Require Origin Disputes” several times over the years, and I’ve always struggled to understand it. Each time I would think that I did, but then I would forget my understanding, and some months or years later, find myself being puzzled by it all over again. So this time I’m going to write down my newly re-acquired understanding, which will let others check that it is correct, and maybe help people (including my future selves) who are interested in Robin's idea but find the paper hard to understand.

Here’s the paper’s abstract, in case you aren’t already familiar with it.

In standard belief models, priors are always common knowledge. This prevents such models from representing agents’ probabilistic beliefs about the origins of their priors. By embedding standard models in a larger standard model, however, pre-priors can describe such beliefs. When an agent’ s prior and pre-prior are mutually consistent, he must believe that his prior would only have been different in situations where relevant event chances were different, but that variations in other agents’ priors are otherwise completely unrelated to which events are how likely. Due to this, Bayesians who agree enough about the origins of their priors must have the same priors.

I think my main difficulty with understanding the paper is the lack of a worked out example. So I’ll take a simplified version of an example given in the paper and try to work out how it should be treated under the proposed formalism. Quoting the paper:

For example, if there were such a thing as a gene for optimism versus pessimism, you might believe that you had an equal chance of inheriting your mother’s optimism gene or your father’s pessimism gene.

Instead of talking about optimism vs pessimism in general, I’ll use the example of an AI which has a prior on just the outcome of one coin toss (A) which will occur after it is created. The AI programmer will program it with one of two priors. The “optimistic” prior O says that the coin will land heads with probability .6, and the “pessimistic” prior P says that the coin will land heads with probability .4. For some reason, the programmer has decided to choose the prior based on an independent coin toss (B), which corresponds...
Why are fictional inventors anti-trade?
by KatjaGrace — 2009-11-02T18:40:32.000Z

In stories, those who invent and make powerful technologies frequently seek to gain power via the technology they make.

In reality, those who invent and make powerful technologies seek to gain power solely through selling their technologies and getting status.

Fictional artists do not keep their paintings, to go out provoking and moving people themselves. Fictional chefs usually prefer trade to eating it all themselves.

Why the difference?

How much does this bias our expectations for the social outcomes, especially risks, of future technologies?
Our House, My Rules
by David_J_Balan — 2009-11-02T00:44:46.686Z

People debate all the time about how strictly children should be disciplined. Obviously, this is a worthwhile debate to have, as there must be some optimal amount of discipline that is greater than zero. The debate's nominal focus is usually on what's best for the child, with even the advocates for greater strictness arguing that it's "for their own good." It might also touch on what's good for other family members or for society at large. What I think is missing from the usual debate is that it assumes nothing but honorable motives on the part of the arguers. That is, it assumes that the arguments in favor of greater strictness are completely untainted by any element of authoritarianism or cruelty. But people are sometimes authoritarian and cruel! Just for fun! And the only people who you can be consistently cruel to without them slugging you, shunning you, suing you, or calling the police on you are your children. This is a reason for more than the usual amount of skepticism of arguments that say that strict parenting is necessary. If there were no such thing as cruelty in the world, people would still argue about the optimal level of strictness, and sometimes the more strict position would be the correct one, and parents would chose the optimal level of strictness on the basis of these arguments. But what we actually have is a world with lots and lots of cruelty lurking just under the surface, which cannot help but show up in the form of pro-strictness arguments in parenting debates. This should cause us to place less weight on pro-strictness arguments than we otherwise would.* Note that this is basically the same idea as Bertrand Russell's argument against the idea of sin: its true function is to allow people to exercise their natural cruelty while at the same time maintaining their opinion of themselves as moral.
One example of authoritarianism masquerading as sound discipline (even among otherwise good parents) is the idea of "My House, My Rules." I've even heard parents go so far as to say things like: "it's not your room, it's the room in my house that I allow you to live in." This attitude makes little sense on its own terms, as it suggests that parents would have no legitimate authority over, say, a famous child actor whose earnings paid for the house. Worse, it's a relatively minor manifestation of the broader notion that the child has a fund...
Romantic idealism: true love conquers almost all
by KatjaGrace — 2009-10-31T20:52:57.000Z

More romantic people tend to be vocally in favor of more romantic fidelity in my experience. If you think about it though, faith in romance is not a very romantic ideal. True love should overcome all things! The highest mountains, the furthest distances, social classes, families, inconveniences, ugliness, but NOT previous love apparently. There shouldn’t be any competition there. The love that got there first is automatically the better one, winning the support and protection of the sentimental against all other love on offer. Other impediments are allowed to test love, sweetened with ‘yes, you must move a thousand miles apart, but if it’s really true love, he’ll wait for you’. You can’t say, ‘yes, he has another girlfriend, but if you really are better for him he’ll come back – may the truest love win!’.

Perhaps more commitment in general allows better and more romance? There are costs as well as benefits to being tied to anything though. Just as it’s not clear that more commitment in society to stay with your current job would be pro-productivity, it’s hard to see that more commitment to stay with your current partner would be especially pro-romance. Of course this is all silly – being romantic and vocally supporting faithfulness are about signaling that you will stick around, not about having consistent values or any real preference about the rest of the world. Is there some other explanation?
Choose pain-free utilitarianism
by KatjaGrace — 2009-10-30T16:59:25.000Z

Some of my friends are hedonic utilitarians, or close human approximations (people whose excuse to talk excitedly in bars and on the internet is sometimes hedonic utilitarianism). I am a preference utilitarian, so I would like to talk excitedly on the internet about how they are wrong.

Robert Wiblin sums up a big motivation for hedonic utilitarianism:

“I am hedonic rather than a preference utilitarian because if I were aware of a being that wanted things but had no experiences I would not care about it as its welfare could not be affected”

Something like this seems a common reason. What makes a thing good or bad if not someone experiencing it as good or bad? And how can you consciously experience something as good or bad if it’s not some variation on pleasure and pain? If your wanting chocolate isn’t backed by being pleased by chocolate, why would I want you to have chocolate more than I would want any old unconscious chocolate-getting mechanism to have chocolate? Pleasure and pain are the distinctive qualia that connect normativeness to consciousness and make it all worthwhile.

This must be wrong. Pain at least can have no such importance, as empirically it can be decomposed into a sensation and a desire to not have the sensation. This is demonstrated by the medical condition pain asymbolia and by the effects of morphine for example. In both cases people say that they can still feel the sensation of the pain they had, but they no longer care about it.

To say that the sensation of pain is inherently bad then is no different than to say that the sensation of seeing the color red is inherently bad. The leftover contender for making pain bad is the preference not to have pain. You may still care only about the sensation of having or fulfilling a preference, and not about preferences that are fulfilled outside of knowledge. The feeling of preferring could still be that sought after sensation inherently imbued with goodness or badness. It must be some variation on preferences though; hedonism’s values are built of them.
Less Wrong / Overcoming Bias meet-up groups
by MichaelVassar — 2009-10-30T04:47:06.192Z

Hi from Michael Vassar, the president of SIAI. With my wife and Singularity Summit co-director Aruna, I'll be traveling over the next few months to meet with rationality and singularity enthusiasts throughout the U.S. Specifically, I'll be in Boston from November 14-18, Philadelphia from December 1st to 10th and December 15th through January 4th, New Orleans from December 11th to 14th, Orlando on January 5th and 6th, Sarasota on January 7th through 12th, and in Tampa on the 11th if there is substantial interest in a meet-up there.

Please comment if you are interested in attending a meet-up in any of the cities in question and we can start planning.

I hope to find that there are thriving communities of rationalists in each of those cities already, but I'm traveling there to try to seed their precipitation from the local populace. If things go really well the groups and their respective cities will be on SIAI radar and who knows, maybe eventually there will be a Singularity Summit near Tomorrow-Land, a global catastrophic risk conference by the New Orleans levies, or a FAI extrapolation dynamics exploratory workshop near Independence Hall.
David Deutsch: A new way to explain explanation
by marc — 2009-10-30T00:05:48.203Z

I'm sure this talk will be of interest, even if most of the ideas that he talks about will be familiar to readers here.

[edit]

In this talk David Deutsch discusses "the most important discovery in human history"; how humanity moved beyond a few hundred thousand years of complete ignorance about the universe. Deutsch attempts to be specific about what led to this change - he concludes that it is the insistence that an explanation be 'hard to vary'.

Whilst a 'hard to vary' explanation is functionally the same as a, more commonly known, Occam's Razor explanation (since fewer parameters necessarily make a fit harder to vary) the slightly different emphasis might be a useful pedagogical tool. A 'hard to vary' explanation will perhaps lead more naturally to questions about strong predictions and falsifiability than Occam's razor. It also seems harder to misunderstand. As we know, Occam's razor suffers because of the difference between actual complexity and linguistic complexity, so an explanation like "it's magic" can appear to be simple. Magic might appear simple, but it will never appear 'hard to vary', so students of rationality would have one less pitfall awaiting them.

Deutsch also touches on what constitutes understanding and knowledge and cautions us not to trust predictions that are purely of an extrapolated empirical nature as there is no true understanding contained there.

[/edit]

If you haven't already read Deutsch's book "The Fabric of Reality" I'd highly recommend that as well.
A Less Wrong Q&A with Eliezer (Step 1: The Proposition)
by MichaelGR — 2009-10-29T15:04:30.286Z

I don't know if I'm the only one, but I've always been a bit frustrated by Eliezer's BloggingHeadsTV episodes. I find myself wishing that Eliezer would get more speaking time and could address more directly some of the things we discuss at LW/OB.

Some of you will surely think: "If you want more Eliezer, just read his blog posts and papers!"

Sensible advice, no doubt, but I think that there's something special (because of how our brains evolved) about actually seeing and hearing a teacher, and I find it very helpful to see how he applies rationality techniques in "real time". But I'm not just looking for a dry lecture, I want more Eliezer because I enjoy listening to him (gotta love that sense of humor), and I bet many of you do too.

Here is my suggestion:

If the Less Wrong community thinks it's a good idea and if Eliezer agrees, I will create a "Step 2: Ask Your Questions" post in which the comments section will be used to gather questions for Eliezer. Each question should be submitted as an individual comment to allow more granularity in the ranking based on the voting system.

After at least 7 days, to give everybody enough time to submit their questions and vote, Eliezer will sit down in front of a camera and answers however many questions he fells like answering (at a time of his choosing, t-shirt or hot cocoa in a mug imprinted with Bayes' theorem are optional), in descending order from most to least votes, skipping the ones he doesn't want to answer. If Eliezer feels uncomfortable speaking alone for an extended period of time, he can get someone to read him the questions so that it feels more like an interview.

The video can then be uploaded to a hosting service like Youtube or Vimeo, and posted to Less Wrong.

In short, it would be a kind of Less Wrong podcast, something that many other sites do successfully.

Yay or nay?

Update: Eliezer says "I'll do it."

Update 2: The thread where questions were submitted can be found here.

Update 3: Eliezer's video answers can be found here.
Post retracted: If you follow expected utility, expect to be money-pumped
by Stuart_Armstrong — 2009-10-29T12:06:04.105Z

This post has been retracted because it is in error. Trying to shore it up just involved a variant of the St Petersburg Paradox and a small point on pricing contracts that is not enough to make a proper blog post.

I apologise.

Edit: Some people have asked that I keep the original up to illustrate the confusion I was under. I unfortunately don't have a copy, but I'll try and recreate the idea, and illustrate where I went wrong.

The original idea was that if I were to offer you a contract L that gained £1 with 50% probability or £2 with 50% probability, then if your utility function wasn't linear in money, you would generally value L at having a value other that £1.50. Then I could sell or buy large amounts of these contracts from you at your stated price, and use the law of large number to ensure that I valued each contract at £1.50, thus making a certain profit.

The first flaw consisted in the case where your utility is concave in cash ("risk averse"). In that case, I can't buy L from you unless you already have L. And each time I buy it from you, the mean quantity of cash you have goes down, but your utility goes up, since you do not like the uncertainty inherent in L. So I get richer, but you get more utility, and once you've sold all L's you have, I cannot make anything more out of you.

If your utility is convex in cash ("risk loving"), then I can sell you L forever, at more than £1.50. And your money will generally go down, as I drain it from you. However, though the median amount of cash you have goes down, your utility goes up, since you get a chance - however tiny - of huge amounts of cash, and the utility generated by this sum swamps the fact you are most likely ending up with nothing. If I could go on forever, then I can drain you entirely, as this is a biased random walk on a one-dimensional axis. But I would need infinite ressources to do this.

The major error was to reason like an investor, rather than a utility maximiser. Investors are very interested in putting prices on objects. And if you assign the wrong price to L while investing, someone will take advantage of you and arbitrage you. I might return to this in a subsequent post; but the issue is that even if your utility is concave or convex in money, you would put a price of £1.50 on L if L were an easily traded commodity with a lot of investors also pricing it...
Expected utility without the independence axiom
by Stuart_Armstrong — 2009-10-28T14:40:08.464Z

John von Neumann and Oskar Morgenstern developed a system of four axioms that they claimed any rational decision maker must follow. The major consequence of these axioms is that when faced with a decision, you should always act solely to increase your expected utility. All four axioms have been attacked at various times and from various directions; but three of them are very solid. The fourth - independence - is the most controversial.

To understand the axioms, let A, B and C be lotteries - processes that result in different outcomes, positive or negative, with a certain probability of each. For 0<p<1, the mixed lottery pA + (1-p)B implies that you have p chances of being in lottery A, and (1-p) chances of being in lottery B. Then writing A>B means that you prefer lottery A to lottery B, A<B is the reverse and A=B means that you are indifferent between the two. Then the von Neumann-Morgenstern axioms are:

(Completeness) For every A and B either A<B, A>B or A=B.

(Transitivity) For every A, B and C with A>B and B>C, then A>C.

(Continuity) For every A>B>C then there exist a probability p with B=pA + (1-p)C.

(Independence) For every A, B and C with A>B, and for every 0<t≤1, then tA + (1-t)C > tB + (1-t)C.

In this post, I'll try and prove that even without the Independence axiom, you should continue to use expected utility in most situations. This requires some mild extra conditions, of course. The problem is that although these conditions are considerably weaker than Independence, they are harder to phrase. So please bear with me here.

The whole insight in this post rests on the fact that a lottery that has 99.999% chance of giving you £1 is very close to being a lottery that gives you £1 with certainty. I want to express this fact by looking at the narrowness of the probability distribution, using the standard deviation. However, this narrowness is not an intrinsic property of the distribution, but of our utility function. Even in the example above, if I decide that receiving £1 gives me a utility of one, while receiving zero gives me a utility of minus ten billion, then I no longer have a narrow distribution, but a wide one. So, unlike the traditional set-up, we have to assume a utility function as being given. Once this is chosen, this allows us to talk about the mean and standard deviation of a l...
Choosing the right amount of choice
by KatjaGrace — 2009-10-28T04:47:32.000Z

The TED talk which I have seen praised most often is Barry Shwartz’s Paradox of Choice. His claim is that the ‘official dogma of all Western industrial societies’ – that more choice is good for us – is wrong. This has apparently been a welcome message for many.

Barry thinks the costs of choice are too high at current levels. His reasons are that it increases our expectations, makes us focus on opportunity costs rather than enjoying what we have, paralyzes us into putting off complicated or important choices, and makes us blame ourselves rather than the world when our selections fail to satisfy. We can choose how much choice to have usually though. You can always just pick a random jar of jam from the shelf if you find the decision making costly. So implicit in Barry’s complaint is that we continually misjudge these downsides and opt for more choice than we should.

Perhaps he is right currently, but I think probably wrong in the long term. Why should we fail to adapt? Even if we can’t adapt psychologically, as inability to deal with choices becomes more of a problem, more technologies for solving it will be found. Having the benefits of choice without the current costs doesn’t appear an insoluble problem.

One option for allowing more choice about choice, while keeping some benefits of variety is to have a standard default option available. Another that seems feasible is using a barcode scanner on a phone, connected to product information and an equation for finding the net goodness of products according to the owner’s values (e.g. goodness = -price – 1c per calorie – 1c per 10 miles travelled + 10c per good review – $100m for peanut traces + …). This could avoid a lot of time spent comparing product information on packages by instantly telling you which brand you likely prefer. Systems for telling you which music and films and people you are likely to like based on previous encounters are improving.

I suspect for many things we would prefer to make very resource intensive choices, because we want to make them ourselves. Where we want to have unique possessions that we identify with, each person needs to go through a similar process of finding out product information and assessing it. We don’t want to know once and for all which is most likely to be the best car for most people. ...
Doing your good deed for the day
by Scott Alexander — 2009-10-27T00:45:24.144Z

Interesting new study out on moral behavior. The one sentence summary of the most interesting part is that people who did one good deed were less likely to do another good deed in the near future. They had, quite literally, done their good deed for the day.
In the first part of the study, they showed that people exposed to environmentally friendly, "green" products were more likely to behave nicely. Subjects were asked to rate products in an online store; unbeknownst to them, half were in a condition where the products were environmentally friendly, and the other half in a condition where the products were not. Then they played a Dictator Game. Subjects who had seen environmentally friendly products shared more of their money.
In the second part, instead of just rating the products, they were told to select $25 worth of products to buy from the store. One in twenty five subjects would actually receive the products they'd purchased. Then they, too, played the Dictator Game. Subjects who had bought environmentally friendly products shared less of their money.
In the third part, subjects bought products as before. Then, they participated in a "separate, completely unrelated" experiment "on perception" in which they earned money by identifying dot patterns. The experiment was designed such that participants could lie about their perceptions to earn more. People who purchased the green products were more likely to do so.
This does not prove that environmentalists are actually bad people - remember that whether a subject purchased green products or normal products was completely randomized. It does suggest that people who have done one nice thing feel less of an obligation to do another.
This meshes nicely with a self-signalling conception of morality. If part of the point of behaving morally is to convince yourself that you're a good person, then once you're convinced, behaving morally loses a lot of its value.
By coincidence, a few days after reading this study, I found this article by Dr. Beck, a theologian, complaining about the behavior of churchgoers on Sunday afternoon lunches. He says that in his circles, it's well known that people having lunch after church tend to abuse the waitstaff and tip poorly. And he blames the same mechanism identified by Mazar and Zhong in their Dictator Game. He says that, having proven to their own satisfaction that they are godly an...