trentmkelly/LessWrong-43k huggingface dataset:
Moore's Paradox
by Eliezer Yudkowsky — 2009-03-08T02:27:39.732Z

I think I understand Moore's Paradox a bit better now, after reading some of the comments on Less Wrong. Jimrandomh suggests:

Many people cannot distinguish between levels of indirection. To them, "I believe X" and "X" are the same thing, and therefore, reasons why it is beneficial to believe X are also reasons why X is true.

I don't think this is correct—relatively young children can understand the concept of having a false belief, which requires separate mental buckets for the map and the territory. But it points in the direction of a similar idea:

Many people may not consciously distinguish between believing something and endorsing it.

After all—"I believe in democracy" means, colloquially, that you endorse the concept of democracy, not that you believe democracy exists. The word "belief", then, has more than one meaning. We could be looking at a confused word that causes confused thinking (or maybe it just reflects pre-existing confusion).

So: in the original example, "I believe people are nicer than they are", she came up with some reasons why it would be good to believe people are nice—health benefits and such—and since she now had some warm affect on "believing people are nice", she introspected on this warm affect and concluded, "I believe people are nice". That is, she mistook the positive affect attached to the quoted belief, as signaling her belief in the proposition. At the same time, the world itself seemed like people weren't so nice. So she said, "I believe people are nicer than they are."

And that verges on being an honest mistake—sort of—since people are not taught explicitly how to know when they believe something. As in the parable of the dragon in the garage; the one who says "There is a dragon in my garage—but it's invisible", does not recognize his anticipation of seeing no dragon, as indicating that he possesses an (accurate) model with no dragon in it.

It's not as if people are trained to recognize when they believe something. It's not like they're ever taught in high school: "What it feels like to actually believe something—to have that statement in your belief pool—is that it just seems like the way the world is. You should recognize this feeling, which is actual (unquoted) belief, and distinguish it from having good ...
Simultaneously Right and Wrong
by Scott Alexander — 2009-03-07T22:55:33.476Z

Related to: Belief in Belief, Convenient Overconfidence

"You've no idea of what a poor opinion I have of myself, and how little I deserve it."

-- W.S. Gilbert
In 1978, Steven Berglas and Edward Jones performed a study on voluntary use of performance inhibiting drugs. They asked subjects to solve certain problems. The control group received simple problems, the experimental group impossible problems. The researchers then told all subjects they'd solved the problems successfully, leaving the controls confident in their own abilities and the experimental group privately aware they'd just made a very lucky guess.
Then they offered the subjects a choice of two drugs to test. One drug supposedly enhanced performance, the other supposedly handicapped it.

There's a cut here in case you want to predict what happened.

Males in the lucky guesser group chose the performance-inhibiting drug significantly more than those in the control group1.
The researchers conjectured that the lucky guessers felt good about acing the first test. They anticipated failing the second whether they chose the helpful or the harmful drug. So they chose the harmful drug as an excuse: "Oh, I would have passed the test, only the drug was making me stupid." As the study points out, this is a win-win situation: if they fail, the drug excuses their failure, and if they succeed it's doubly impressive that they passed even with a handicap.
Since this study, psychologists have applied self-handicapping to interpret common phenomena like procrastination, hypochondria, and minority underachievement; they've also discovered factors that increase or decrease self-handicapping tendencies (for example, you can increase it by making the test more relevant to your self-esteem: just say "this is a proven test of intellectual ability in general").
But some especially interesting studies investigated the effect of privacy on self-handicapping. For example, Hobden conducted an experiment similar to Berglas and Jones', albeit with tapes of performance-enhancing or -handicapping music instead of drugs. The twist: half the subjects' choice of tapes and test scores were public, and half the subjects believed their choice of tapes and test scores were anonymous. What happens when no one but the subject himself will ever know his test score? He self-handicaps just as often as ...
The Golem
by Kaj_Sotala — 2009-03-07T18:32:13.264Z

Anthony Ravenscroft writing on why it is important, in a relationship, to honestly communicate your grievances to the other person:

If you don't present your gripes to the responsible party, you cannot humanly bury those complaints - it's just not possible to "forget" about something that has hurt or stung you. Actually, you are probably "testing" these complaints against your experience of the person, trying to figure out what they would say, how they would react. You create a simulacrum in order to argue this all out in your head, and thus to avoid unpleasantness. Certain conclusions are made, which you file away. When another problem comes up, you then test this against your estimates of the person, which have been expanded by your previous guesswork.
Eventually, you will have created this huge guesswork of assumptions, which are so far removed from the actual person that they likely have no bearing on the reality. I call this "a golem made of boxes", a warehouse-sized beast that has nothing to do with the simple small human being from which it is supposedly modeled.
When I have had such a golem used against me, I was told by my lover that she had kept a rather ugly situation from me "because I know how you'd react." I described to her exactly what the situation was, as I'd pieced it together very accurately (you can do this with the actions of humans, not the humans themselves). She was stunned. When I described for her how the root assumptions she had made were very largely off the mark, she actually became very angry with me, defending the golem as though it represented the truth, and therefore I must be lying! In the end, she could have better determined my reaction from writing down the possibilities on slips of paper and choosing one out of a hat. ...
The golem is handy, but almost entirely dishonest. It begins from faulty (incomplete, biased) data, and runs rapidly downhill from there.

The map and the territory. How have you had the golem used against you? When have you, yourselves, made the mistake of resorting to a golem and had it blow in your face?
Checklists
by Johnicholas — 2009-03-07T15:47:14.673Z

Checklists are a rationality technique, mentioned previously on OB. Everyone knows this, but we don't hear about them as often as we should, possibly because they seem prosaic and boring.

In the context of doing something over and over, there is a checklist improvement cycle.

You try to make the thing (e.g. the blog post, the rational decision, the mathematical proof)

For each kind of error in your checklist, you search the thing for that kind of error, and fix it if it occurs.

When something that passed your checklist turns out to have had an error, you add that kind of error to your checklist.

There are many caveats to this description: Some checklists are not primarily lists of errors, but primarily ordered procedures. You may want to complicate the cycle to track the cost and benefit of the items on the checklist. We're assuming that errors are eventually discovered. I want to pass over those caveats and claim that this kind of checklist-of-errors is very successful. If you agree, my question is: What feature or features of our minds are checklists compensating for? If we understood that, then we would be able to use checklists even more effectively.

The act of considering the current checklist item and the post/decision/proof simultaneously reminds me of "Forced Association", a creativity technique. So one idea is that by putting one's mind into several different states via forced association with the items on the checklist, we gain more independent chances to detect an error.

Even if you haven't made any errors yet, and so your checklist is empty, conducting several searches for errors while wearing De Bono's hats (or another forced association list) might be a way to make fewer errors.

If you're consciously looking for a red minivan, then you will notice more red minivans. This "noticing" seems surprisingly spontaneous, unlike deliberately scanning a scene and considering "Is that a red minivan? Is that a red minivan?". Possibly this is because the noticing is being done by unconscious modules of our minds. A checklist breaks our search for errors into a sequence of searches for more specific kinds of errors. Possibly checklists are effective because the general concept "error" is too vague for those modules. By breaking it into easier chunks (e.g. "ad hominem fallacy", "missing semicolon"), we can start using those modules to get a more thorough search...
Slow down a little... maybe?
by Eliezer Yudkowsky — 2009-03-07T01:34:03.994Z

I think that three posts a day over and above Yudkowsky and/or Hanson posts might be enough. Where anything that gets voted to 0 or below doesn't count, nor do quick links.

Say you differently, readers? I'm just trying to space things out so we don't get overloaded with everything, all at once... if it turns out that people just have more to say than this, sustainably in the long term, then we can raise the posting speed.
Formalization is a rationality technique
by Johnicholas — 2009-03-06T20:22:39.032Z

We are interested in developing practical techniques of rationality. One practical technique, used widely and successfully in science and technology is formalization, transforming a less-formal argument into a more-formal one. Despite its successes, formalization isn't trivial to learn, and schools rarely try to teach general techniques of thinking and deciding. Instead, schools generally only teach domain-specific reasoning. We end up with graduates who can apply formalization skillfully inside of specific domains (e.g. electrical engineering or biology), but fail to apply, or misapply, their skills to other domains (e.g. politics or religion).

A side excursion, to be used as an example:

Is it true that a real-world decision can be perfectly justified via a mathematical proof? No.

Here is one reason: Any real-world "proof", whether it is a publication in a math journal or the output of a computer proof-checker, might contain a mistake. Human mathematicians routinely make mistakes in publications. The computer running the proof-checker is physical, and might be struck by a cosmic ray. Even if the computer were perfect, the proof-checker might have a bug in it.

Here is another reason: Even if you had a proof known to be mistake free, mathematical proof always reasons from assumptions to conclusions. To apply the proof to the real world, you must make an informal argument that the assumptions apply in this real-world situation, and a second informal argument that the conclusion of the proof corresponds to the real-world decision. These first and last links in the chain will still be there, no matter how rigorous and trusted the proof is. The problem with duplicating real-world spheres with Banach-Tarski isn't that there's a mistake in the proof, it's that the real-world spheres don't fit the assumptions.

If you were fitting this argument into your beliefs, you might produce a number, a "gut" estimate of how likely this informal argument is wrong. Can we improve on that using the technique of formalization? What would a formalization of this argument look like? One possible starting point might be to rename everything. We're confident (via philosophy of logic) that renaming won't increase or decrease the quality of the argument. We will reason better about the correctness of the form if we hide the subjects of the argument.

Is it true that A? No.

Here is one rea

...
Does blind review slow down science?
by Kaj_Sotala — 2009-03-06T12:35:34.304Z

Previously, Robin Hanson pointed out that even if implementing anonymous peer review has an effect on the acceptance rate of different papers, this doesn't necessarily tell us the previous practice was biased. Yesterday, I ran across an interesting passage suggesting one way that anonymous review might actually be harmful:

Second, fame may confer license. If a person has done valuable work in the past, this increases the probability that his current work is also valuable and induces the audience to suspend its disbelief. He can therefore afford to thumb his nose at the crowd. This is merely the obverse of the "shamelessness" of the old, which Aristotle discussed. Peter Messeri argues in this vein that "senior scientists are better situated than younger scientists to withstand adverse consequences of public advocacy of unpopular positions," and that this factor may explain why the tendency for older scientists to resist new theories is, in fact, weak. And remember Kenneth Dover's negative verdict on old age (chapter 5)? He offered one qualification: "There just aren't any [aspects of old age which compensate for its ills] - except, maybe, a complacent indifference to fashion, because people no longer seeking employment or promotion have less to fear."

This point suggests that the use by scholarly journals of blind refereeing is a mistaken policy. It may cause them to turn down unconventional work to which they would rightly have given the benefit of doubt had they known that the author was not a neophyte or eccentric.

(From Richard A. Posner, "Aging and Old Age")

If this hypothesis holds (and Posner admits it hasn't been tested, at least at the time of writing), then blind review may actually slow down the acceptance of theories which are radical but true. Looking up the Peter Messeri reference gave me the article "Age Differences in the Reception of New Scientific Theories: The Case of Plate Tectonics Theory". It notes:

Young scientists may adopt a new theory before their elders in part because they are better informed about current research in a broader range of fields. As Zuckerman and Merton observe with respect to life-course differences in opportunities for discovery, young scientists - closer in time to their formal training with an 'aggregate of specialists at work on [many] research front[s]' - are likely to be more up to dat

...
Is it rational to take psilocybin?
by pwno — 2009-03-06T04:44:05.349Z

Is it rational to take psilocybin?

Just to make my definition of rational clear:

Rationality is only intelligible when in the context of a goal (whether that goal be rational or irrational). Now, if one acts rationally, given their information set, will chose the best plan-of-action towards succeeding their goal. Part of being rational is knowing which goals will maximize one’s utility function.

According to Discovery:

“Scientists released their findings on a recent survey of volunteer psilocybin users 14 months after they took the drug.”

“Sixty-four percent of the volunteers said they still felt at least a moderate increase in well-being or life satisfaction, in terms of things like feeling more creative, self-confident, flexible and optimistic. And 61 percent reported at least a moderate behavior change in what they considered positive ways.”

Assuming you won’t get a bad trip, is it rational to take the drug?

I doubt a psychedelic experience can help me optimize my current utility function better than my sober self. How can I be more rational from a drug? Therefore, I conclude that it must, in fact, change my preference ordering—make me care about things more than I would have otherwise. I prefer my preferences and therefore would rather keep my preferences the way they are now.

If you were guaranteed to have all these positive results from taking the drug, would you take it?
Information cascades
by Johnicholas — 2009-03-06T04:08:04.882Z

An information cascade is a problem in group rationality. Wikipedia has excellent introductions and links about the phenomenon, but here is a meta-ish example using likelihood ratios.

Suppose in some future version of this site, there are several well-known facts:

All posts come in two kinds, high quality (insightful and relevant) and low quality (old ideas rehashed, long hypotheticals).

There is a well-known prior 60% chance of anything being high quality, rather than low quality. (We're doing well!)

Readers get a private signal, either "high" or "low", their personal judgement of quality, which is wrong 20% of the time.

The number of up and down votes is displayed next to each post. (Note the difference from the present system, which only displays up minus down. This hypothesis makes the math easier.)

Readers are competent in Bayesian statistics and strive to vote the true quality of the post.

Let's talk about how the very first reader would vote. If they judged the post high quality, then they would multiply the prior likelihood ratio (6:4) times the bayes factor for a high private signal (4:1), get (6*4:4*1) = (6:1) and vote the post up. If they judged the post low quality then they would instead multiply by the bayes factor for a low private signal (1:4), get (6*1:4*4) = (3:8) and vote the post down.

There were two scenarios for the first reader (private information high or low). If we speculate that the first reader did in fact vote up, then there are two scenarios for the second scenario: There are two scenarios for the second reader:

Personal judgement high: (6:4)*(4:1)*(4:1) = (24:1), vote up.

Personal judgement low: (6:4)*(1:4)*(4:1) = (6:4), vote up against personal judgement.

Note that now there are two explanations for ending up two votes up. It could be that the second reader actually agreed, or it could be that the second reader was following the first reader and the prior against their personal judgement. That means that the third reader gets zero information from the second reader's personal judgement! The two scenarios for the third reader, and every future reader, are exactly analogous to the two scenarios for the second reader.

Personal judgement high: (6:4)*(4:1)*(4:1) = (24:1), vote up.

Personal judgement low: (6:4)*(1:4)*(4:1) = (6:4), vote up against personal judgement.

This has been a nightmare scenario of groupthink afflicting e...
Recommended Rationalist Resources
by MichaelHoward — 2009-03-05T20:23:09.098Z

I thought Recommended Rationalist Reading was very useful and interesting. Now we have voting and threading it seems a good time to comprehensively gather opinions on online material.

Please suggest high-quality links related to or useful for improving rationality. It could be a blog, a forum, a great essay, a reference site, an e-book, anything clickable. Anyone interested can then check out what looks promising and report back.

[edit]
There seems to be confusion... The post's for online material, not physical books. We already have Recommended Rationalist Reading, but as that hasn't got threading and voting, if people think it's a good idea I (or someone else) can do a seperate post for books [metaedit] ...not happening, is against blog guidelines. [/metaedit]

Looks like we're getting lots of suggestions, so please don't forget to vote on them so busier readers have an idea which ones are worth more investigating!
[/edit]

Contributors - if making multiple suggestions, please give each their own comment so we can vote on them separately. Click 'Help' for how to do links.

Voters - for top level comments containing suggestions (as oppose to comments replying to suggestions) please vote on the quality of the resource, not anything else in the comment. If you feel strongly about the comment quality, just post a sub-comment.

Here's 3 to get started:
Define Rationality
by Marshall — 2009-03-05T18:25:06.240Z

I would like to suggest that we try to come up with several defintions of rationality. I don't feel we have exhausted this search area by any means. Robin has suggested, "More "rational" means better believing what is true, given one's limited info and analysis resources". Other commenters have emphasised goal-directed behaviour as a necessary ingredience of rationality. I think these defintions miss out on several important ingrediences - such as the social nature of rationality. There is also a subtext which argues - that rationality only gives one (correct) answer even if we only can approximate it. I feel strongly that rationality can give several correct answers and thus imagination is an ingredience of rationality. So without in any way believing that I have found the one correct defintion, I propose the following: When two or more brains try to be sensible about things and expand their agency. I believe that "sensible" in this context does not need to be defined as it is a primitive and each player willl submit their own meaning.

Maybe this is a can of worms - but are there other suggestions or defintions for rationality we can apply in our lives?
Kinnaird's truels
by Johnicholas — 2009-03-05T16:50:02.621Z

A "truel" is something like a duel, but among three gunmen. Martin Gardner popularized a puzzle based on this scenario, and there are many variants of the puzzle which mathematicians and game theorists have analyzed.

The optimal strategy varies with the details of the scenario, of course. One take-away from the analyses is that it is often disadvantageous to be very skillful. A very skillful gunman is a high-priority target.

The environment of evolutionary adaptedness undoubtedly contained multiplayer social games. If some of these games had a truel-like structure, they may have rewarded mediocrity. This might be an explanation of psychological phenomena like "fear of success" and "choking under pressure".

Robin Hanson has mentioned that there are costs to "truth-seeking". One of the example costs might be convincingly declaring "I believe in God" in order to be accepted into a religious community. I think truels are a game-theoretic structure that suggests that there are costs to (short-sighted) "winning", just as there are costs to "truth-seeking".

How can you identify truel-like situations? What should you (a rationalist) do if you might be in a truel-like situation?