trentmkelly/LessWrong-43k huggingface dataset:
Never Leave Your Room
by Scott Alexander — 2009-03-18T00:30:27.152Z

Related to: Priming and Contamination

Psychologists define "priming" as the ability of a stimulus to activate the brain in such a way as to affect responses to later stimuli. If that doesn't sound sufficiently ominous, feel free to re-word it as "any random thing that happens to you can hijack your judgment and personality for the next few minutes."
For example, let's say you walk into a room and notice a briefcase in the corner. Your brain is now the proud owner of the activated concept "briefcase". It is "primed" to think about briefcases, and by extension about offices, business, competition, and ambition. For the next few minutes, you will shift ever so slightly towards perceiving all social interactions as competitive, and towards behaving competitively yourself. These slight shifts will be large enough to be measured by, for example, how much money you offer during the Ultimatum Game. If that sounds too much like some sort of weird New Age sympathetic magic to believe, all I can say is Kay, Wheeler, Bargh, and Ross, 2004.1
We've been discussing the costs and benefits of Santa Claus recently. Well, here's one benefit: show Dutch children an image of St. Nicholas' hat, and they'll be more likely to share candy with others. Why? The researchers hypothesize that the hat activates the concept of St. Nicholas, and St. Nicholas activates an idealized concept of sharing and giving. The child is now primed to view sharing positively. Of course, the same effect can be used for evil. In the same study, kids shown the Toys 'R' Us logo refused to share their precious candy with anyone.
But this effect is limited to a few psych laboratories, right? It hasn't done anything like, you know, determine the outcome of a bunch of major elections?

I am aware of two good studies on the effect of priming in politics. In the first, subjects were subliminally2 primed with either alphanumeric combinations that recalled the 9/11 WTC attacks (ie "911" or "WTC"), or random alphanumeric combinations. Then they were asked to rate the Bush administration's policies. Those who saw the random strings rated Bush at an unenthusiastic 42% (2.1/5). Those who were primed to be thinking about the War on Terror gave him an astounding 75% (3.75/5). This dramatic a change, even though none of them could consciously recall seeing terrorism-related stimuli.
In the second study, scientists analyzed data ...
The Pascal's Wager Fallacy Fallacy
by Eliezer Yudkowsky — 2009-03-18T00:30:00.000Z

Today at lunch I was discussing interesting facets of second-order logic, such as the (known) fact that first-order logic cannot, in general, distinguish finite models from infinite models. The conversation branched out, as such things do, to why you would want a cognitive agent to think about finite numbers that were unboundedly large, as opposed to boundedly large.

So I observed that:

Although the laws of physics as we know them don't allow any agent to survive for infinite subjective time (do an unboundedly long sequence of computations), it's possible that our model of physics is mistaken. (I go into some detail on this possibility below the cutoff.)
If it is possible for an agent - or, say, the human species - to have an infinite future, and you cut yourself off from that infinite future and end up stuck in a future that is merely very large, this one mistake outweighs all the finite mistakes you made over the course of your existence.

And the one said, "Isn't that a form of Pascal's Wager?"

I'm going to call this the Pascal's Wager Fallacy Fallacy.

You see it all the time in discussion of cryonics. The one says, "If cryonics works, then the payoff could be, say, at least a thousand additional years of life." And the other one says, "Isn't that a form of Pascal's Wager?"

The original problem with Pascal's Wager is not that the purported payoff is large. This is not where the flaw in the reasoning comes from. That is not the problematic step. The problem with Pascal's original Wager is that the probability is exponentially tiny (in the complexity of the Christian God) and that equally large tiny probabilities offer opposite payoffs for the same action (the Muslim God will damn you for believing in the Christian God).

However, what we have here is the term "Pascal's Wager" being applied solely because the payoff being considered is large - the reasoning being perceptually recognized as an instance of "the Pascal's Wager fallacy" as soon as someone mentions a big payoff - without any attention being given to whether the probabilities are in fact small or whether counterbalancing anti-payoffs exist.

And then, once the reasoning is perceptually recognized as an instance of "the Pascal's Wager fallacy", the other characteristics of the fa...
Tarski Statements as Rationalist Exercise
by Vladimir_Nesov — 2009-03-17T19:47:16.021Z

Related to: Dissolving the Question, The Second Law of Thermodynamics, and Engines of Cognition, The Meditation on Curiosity.

The sentence "snow is white" is true if, and only if, snow is white.

-- A. Tarski

Several days ago I've spent a couple of hours trying to teach my 15 year old brother how to properly construct Tarski statements. It's quite nontrivial to get right. Learning to place facts and representations in the separate mental buckets is one of the fundamental tools for a rationalist. In our model of the world, information propagates from object to object, from mind to mind. To ascertain the validity of your belief, you need to research the whole network of factors that led you to attain the belief. The simplest relation is between a fact and its representation, idealized to represent correctness or incorrectness only, without yet worrying about probabilities. The same object or the same property can be interpreted to mean different things in different relations and contexts, indicating the truth of one statement or another, and it's important not to conflate those.

Let's say you are watching news on TV and the next item is an interview with a sasquatch. The sasquatch answers the questions about his family in decent English, with a slight British accent.

What do you actually observe, how should you interpret the data? Did you "see a sasquatch"? Did you learn the facts about sasquatch's family? Is there a fact of the matter, as to whether the sasquatch's daughter is 5 years old, as opposed to 4 or 6?

Meaningfulness of these questions is conditional on their context, like in the notorious "when did you stop beating your wife?". These examples seem unnaturally convoluted, but in fact every statement suffers from the same problem, you must cross the levels of indirection and not lose track of the question in order to go from a statement of fact, from a belief in your mind, to the fact that belief is about. First, you must relate "sasquatch" on the TV screen to an actual sasquatch, which fails if there isn't one, and then you need to relate the sasquatch to his daughter, which again fails if he lies and doesn't have one.

By contemplating plausibility of the surface interpretation of a statement that doesn't yield to that interpretation, you fail before you even start. If the surface interpretation of the data doesn't apply, the data doesn't say anything ab...
Dead Aid
by PhilGoetz — 2009-03-17T14:51:11.304Z

Followup to So You Say You're an Altruist:

Today Dambisa Moyo's book "Dead Aid: Why Aid Is Not Working and How There Is a Better Way for Africa" was released.

From the book's website:

In the past fifty years, more than $1 trillion in development-related aid has been transferred from rich countries to Africa. Has this assistance improved the lives of Africans? No. In fact, across the continent, the recipients of this aid are not better off as a result of it, but worse—much worse.

In Dead Aid, Dambisa Moyo describes the state of postwar development policy in Africa today and unflinchingly confronts one of the greatest myths of our time: that billions of dollars in aid sent from wealthy countries to developing African nations has helped to reduce poverty and increase growth.

In fact, poverty levels continue to escalate and growth rates have steadily declined—and millions continue to suffer. Provocatively drawing a sharp contrast between African countries that have rejected the aid route and prospered and others that have become aid-dependent and seen poverty increase, Moyo illuminates the way in which overreliance on aid has trapped developing nations in a vicious circle of aid dependency, corruption, market distortion, and further poverty, leaving them with nothing but the “need” for more aid.

From the Global Investor Bookshop:

Dead Aid analyses the history of economic development over the last fifty years and shows how Aid crowds out financial and social capital and directly causes corruption; the countries that have caught up did so despite rather than because of Aid. There is, however, an alternative. Extreme poverty is not inevitable. Dambisa Moyo also shows how, with improved access to capital and markets and with the right policies, even the poorest nations could be allowed to prosper. If we really do want to help, we have to do more than just appease our consciences, hoping for the best, expecting the worst. We need first to understand the problem.
Rational Me or We?
by RobinHanson — 2009-03-17T13:39:29.073Z

Martial arts can be a good training to ensure your personal security, if you assume the worst about your tools and environment. If you expect to find yourself unarmed in a dark alley, or fighting hand to hand in a war, it makes sense. But most people do a lot better at ensuring their personal security by coordinating to live in peaceful societies and neighborhoods; they pay someone else to learn martial arts. Similarly, while "survivalists" plan and train to stay warm, dry, and fed given worst case assumptions about the world around them, most people achieve these goals by participating in a modern economy.
The martial arts metaphor for rationality training seems popular at this website, and most discussions here about how to believe the truth seem to assume an environmental worst case: how to figure out everything for yourself given fixed info and assuming the worst about other folks. In this context, a good rationality test is a publicly-visible personal test, applied to your personal beliefs when you are isolated from others' assistance and info.
I'm much more interested in how we can can join together to believe truth, and it actually seems easier to design institutions which achieve this end than to design institutions to test individual isolated general tendencies to discern truth. For example, with subsidized prediction markets, we can each specialize on the topics where we contribute best, relying on market consensus on all other topics. We don't each need to train to identify and fix each possible kind of bias; each bias can instead have specialists who look for where that bias appears and then correct it.
Perhaps martial-art-style rationality makes sense for isolated survivalist Einsteins forced by humanity's vast stunning cluelessness to single-handedly block the coming robot rampage. But for those of us who respect the opinions of enough others to want to work with them to find truth, it makes more sense to design and field institutions which give each person better incentives to update a common consensus.
On Juvenile Fiction
by MBlume — 2009-03-17T08:53:06.300Z

Follow-up To: On the Care and Feeding of Young Rationalists

Related on OB: Formative Youth

Eliezer suspects he may have chosen an altruistic life because of Thundercats.

Nominull thinks his path to truth-seeking might have been lit by Asimov's Robot stories.

PhilGoetz suggests that Ender's Game has warped the psyches of many intelligent people.

For good or ill, we seem to agree that fiction strongly influences the way we grow up, and the people we come to be.

So for those of us with the tremendous task of bringing new sentience into the world, it seems sensible to spend some time thinking about what fictions our charges will be exposed to.

The natural counter-part to this question is, of course, are there any particular fictions, or types of fiction, to which we should avoid exposing our children?

Again, this is a pattern we see more commonly in the religious community -- and the rest of us tend to look on and laugh at the prudery on display. Still, the general idea doesn't seem to be something we can reject out of hand. So far as we can tell, all (currently existing) minds are vulnerable to being hacked, young minds more than others. If we determine that a particular piece of fiction, or a particular kind of fiction, tends to reliably and destructively hack vulnerable minds, that seems a disproportionate consequence for pulling the wrong book off the shelf.

So, what books, what films, what stories would you say affected your childhood for the better? What stories do you wish you had encountered earlier? If there are any members of the Bardic Conspiracy present, what sorts of stories should we start telling? Finally, what stories (if any) should young minds not encounter until they have developed some additional robustness?

ETA: If there are particular stories which you think the (adult) members of the community would benefit from, please feel free to share these as well.

ETA2: My wildly optimistic best-case scenario for this post would be someone actually writing a rationalist children's story in the comments thread.

ETA3: On second thought, this edit has become its own post.
The "Spot the Fakes" Test
by Scott Alexander — 2009-03-17T00:52:54.095Z

Followup to: Are You a Solar Deity?

James McAuley and Harold Stewart were mid-20th century Australian poets, and they were not happy. After having society ignore their poetry in favor of "experimental" styles they considered fashionable nonsense, they wanted to show everyone what they already knew: the Australian literary world was full of empty poseurs.
They began by selecting random phrases from random books. Then they linked them together into something sort of like poetry. Then they invented the most fashionable possible story: Ern Malley, a loner working a thankless job as an insurance salesman, writing sad poetry in his spare time and hiding it away until his death at an early age. Posing as Malley's sister, who had recently discovered the hidden collection, they sent the works to Angry Penguins, one of Australia's top experimental poetry magazines.
You wouldn't be reading this if the magazine hadn't rushed a special issue to print in honor of "a poet in the same class as W.H. Auden or Dylan Thomas".
The hoax was later revealed1, everyone involved ended up with egg on their faces, and modernism in Australia received a serious blow. But as I am reminded every time I look through a modern poetry anthology, one Ern Malley every fifty years just isn't enough. I daydream about an alternate dimension where people are genuinely interested in keeping literary criticism honest. In this universe, any would-be literary critic would have to distinguish between ten poems generally recognized as brilliant that he'd never seen before, and ten pieces of nonsense invented on the spot by drunk college students, in order to keep his critic's license.
Can we refine this test? And could it help Max Muller with his solar deity problem?

In the Malley hoax, McAuley and Steward suspected that a certain school of modernist poetry was without value. Because its supporters were too biased to admit this directly, they submitted a control poem they knew was without value, and found the modernists couldn't tell the difference. This suggests a powerful technique for determining when something otherwise untestable might be, as Neal Stephenson calls it, bulshytte.
Perhaps Max Muller thinks Hercules is a solar deity. He will write up a argument for this proposition, and submit it for consideration before all the great mythologists of the world. Even if these mythologists want to be unbiased, t...
What Do We Mean By "Rationality"?
by Eliezer Yudkowsky — 2009-03-16T22:33:55.765Z

I mean two things:

1. Epistemic rationality: systematically improving the accuracy of your beliefs.

2. Instrumental rationality: systematically achieving your values.

The first concept is simple enough. When you open your eyes and look at the room around you, you’ll locate your laptop in relation to the table, and you’ll locate a bookcase in relation to the wall. If something goes wrong with your eyes, or your brain, then your mental model might say there’s a bookcase where no bookcase exists, and when you go over to get a book, you’ll be disappointed.

This is what it’s like to have a false belief, a map of the world that doesn’t correspond to the territory. Epistemic rationality is about building accurate maps instead. This correspondence between belief and reality is commonly called “truth,” and I’m happy to call it that.1

Instrumental rationality, on the other hand, is about steering reality—sending the future where you want it to go. It’s the art of choosing actions that lead to outcomes ranked higher in your preferences. I sometimes call this “winning.”

So rationality is about forming true beliefs and making decisions that help you win.

(Where truth doesn't mean “certainty,” since we can do plenty to increase the probability that our beliefs are accurate even though we're uncertain; and winning doesn't mean “winning at others' expense,” since our values include everything we care about, including other people.)

When people say “X is rational!” it’s usually just a more strident way of saying “I think X is true” or “I think X is good.” So why have an additional word for “rational” as well as “true” and “good”?

An analogous argument can be given against using “true.” There is no need to say “it is true that snow is white” when you could just say “snow is white.” What makes the idea of truth useful is that it allows us to talk about the general features of map-territory correspondence. “True models usually produce better experimental predictions than false models” is a useful generalization, and it’s not one you can make without using a concept like “true” or “accurate.”

Similarly, “Rational agents make decisions that maximize the probabilistic expectation of a coherent utility function” is the kind of thought that depends on a concept of (instrumental) rationality, whereas “It’s rational to eat vegetables” can probably be replaced with “It’s useful to eat vegetables...
Science vs. art
by PhilGoetz — 2009-03-16T15:48:21.121Z

In the comments on Soulless Morality, a few people mentioned contributing to humanity's knowledge as an ultimate value. I used to place a high value on this myself.

Now, though, I doubt whether making scientific advances would give me satisfaction on my deathbed. All you can do in science is discover something before someone else discovers it. (It's a lot like the race to the north pole, which struck me as stupid when I was a child; yet I never transferred that judgement to scientific races.) The short-term effects of your discovering something sooner might be good, and might not. The long-term effects are likely to be to bring about apocalypse a little sooner.

Art is different. There's not much downside to art. There are some exceptions - romance novels perpetuate destructive views of love; 20th-century developments in orchestral music killed orchestral music; and Ender's Game has warped the psyches of many intelligent people. But artists seldom worry that their art might destroy the world. And if you write a great song, you've really contributed, because no one else would have written that song.

EDIT: What is above is instrumental talk. I find that, as I get older, science fails to satisfy me as much. I don't assign it the high intrinsic value I used to. But it's hard for me to tell whether this is really an intrinsic valuation, or the result of diminishing faith in its instrumental value.

I think that people who value rationality tend to place an unusually high value on knowledge. Rationality requires knowledge; but that gives knowledge only instrumental value. It doesn't (can't, by definition) justify giving knowledge intrinsic value.

What do the rest of you think? Is there a strong correlation between rationalism, giving knowledge high intrinsic value, and giving art low intrinsic value? If so, why? And which would you rather be - a great scientist, or a great artist of some type? (Pretend that great scientists and great artists are equally well-paid and sexually attractive.)

(I originally wrote this as over-valuing knowledge and under-valuing art, but Roko pointed out that that's incoherent.)

Under a theory that intrinsic and instrumental values are separate things, there's no reason why giving science a high instrumental value should correlate with giv...
Taboo "rationality," please.
by MBlume — 2009-03-15T22:44:13.162Z

Related on OB: Taboo Your Words

I realize this seems odd on a blog about rationality, but I'd like to strongly suggest that commenters make an effort to avoid using the words "rational," "rationality," or "rationalist" when other phrases will do. I think we've been stretching the words to cover too much meaning, and it's starting to show.
Here are some suggested substitutions to start you off.
Rationality:

truth-seeking

probability updates under bayes rule

the "winning way"

Rationalist:

one who reliably wins

one who can reliably be expected to speak truth

Are there any others?
In What Ways Have You Become Stronger?
by Vladimir_Nesov — 2009-03-15T20:44:47.697Z

Related to: Tsuyoku Naritai! (I Want To Become Stronger), Test Your Rationality, 3 Levels of Rationality Verification.

Robin and Eliezer ask about the ways to test rationality skills, for each of the many important purposes such testing might have. Depending on what's possible, you may want to test yourself to learn how well you are doing at your studies, at least to some extent check the sanity of the teaching that you follow, estimate the effectiveness of specific techniques, or even force a rationality test on a person whose position depends on the outcome.

Verification procedures have various weaknesses, making them admissible for one purpose and not for another. But however rigorous the verification methods are, one must first find the specific properties to test for. These properties or skills may come naturally with the art, or they may be cultivated specifically for the testing, in which case they need to be good signals, hard to demonstrate without also becoming more rational.

So, my question is this - what have you become reliably stronger at, after you walked the path of an aspiring rationalist for considerable time? Maybe you have noticeably improved at something, or maybe you haven't learned a certain skill yet, but you are reasonably sure that because of your study of rationality you'll be able to do that considerably better than other people.

This is a significantly different question from the ones Eliezer and Robin ask. Some of the skills you obtained may be virtually unverifiable, some of them may be easy to fake, some of them may be easy to learn without becoming sufficiently rational, and some of them may be standard in other disciplines. But I think it's useful to step back, and write a list of skills before selecting ones more suitable for the testing.