trentmkelly/LessWrong-43k huggingface dataset:
Applause Lights
by Eliezer Yudkowsky — 2007-09-11T18:31:48.000Z

At the Singularity Summit 2007, one of the speakers called for democratic, multinational development of artificial intelligence. So I stepped up to the microphone and asked:

Suppose that a group of democratic republics form a consortium to develop AI, and there’s a lot of politicking during the process—some interest groups have unusually large influence, others get shafted—in other words, the result looks just like the products of modern democracies. Alternatively, suppose a group of rebel nerds develops an AI in their basement, and instructs the AI to poll everyone in the world—dropping cellphones to anyone who doesn’t have them—and do whatever the majority says. Which of these do you think is more “democratic,” and would you feel safe with either?

I wanted to find out whether he believed in the pragmatic adequacy of the democratic political process, or if he believed in the moral rightness of voting. But the speaker replied:

The first scenario sounds like an editorial in Reason magazine, and the second sounds like a Hollywood movie plot.

Confused, I asked:

Then what kind of democratic process did you have in mind?

The speaker replied:

Something like the Human Genome Project—that was an internationally sponsored research project.

I asked:

How would different interest groups resolve their conflicts in a structure like the Human Genome Project?

And the speaker said:

I don’t know.

This exchange puts me in mind of a quote from some dictator or other, who was asked if he had any intentions to move his pet state toward democracy:

We believe we are already within a democratic system. Some factors are still missing, like the expression of the people’s will.

The substance of a democracy is the specific mechanism that resolves policy conflicts. If all groups had the same preferred policies, there would be no need for democracy—we would automatically cooperate. The resolution process can be a direct majority vote, or an elected legislature, or even a voter-sensitive behavior of an artificial intelligence, but it has to be something. What does it mean to call for a “democratic” solution if you don’t have a conflict-resolution mechanism in mind?

I think it means that you have said the word “democracy,” so the audience is supposed to cheer. It’s not so much a propositional statement or belief, as the equivalent of the “Applause” light that tells a studio audience when to clap.

This case...
We Don't Really Want Your Participation
by Eliezer Yudkowsky — 2007-09-10T19:53:46.000Z

At the Singularity Summit yesterday, several speakers alleged that we should "reach out" to artists and poets to encourage their participation in the Singularity dialogue. So at the end of one such session, a woman went up to the audience microphone and said:

"I am an artist. I want to participate. What should I do?"

And there was a brief, frozen silence.

I wanted to leap up and say:

No, no, I'm afraid you've misunderstood. We're just calling for greater participation by artists. We can get plenty of credit for being enlightened just by issuing the call.
If we really cared what artists thought, we would find some artists and ask them questions, not call for artists to
participate. We don't actually want to hear from artists. We think your opinions are stupid.

And if she'd asked me afterward, my real answer
would have been:

You are not an artist, you are a human being; art is
only one facet in which you express your humanity. Your reactions
to the Singularity should arise from your entire self. It's
perfectly all right to have a boringly normal and nonunique reaction like "I'm afraid," or "I don't
think we should do this," or "I want to help, where do I send the
check?" The right answer is not always unusual. Your natural reaction does not need to be unique, and that's why you don't need to try to come up with
an "artist's viewpoint" on the Singularity. I would call on you to
participate as a human being, not an artist. If your artistry has
something to say, it will express itself naturally in your
responses, without you needing to make a conscious effort to say something artist-like.

But I didn't say any of this, of course. It would have been indecorous.

And while we're on the subject, I would feel rather patronized - like a dog commanded to perform a trick - if someone presented me with a painting and said, "Say something mathematical!"
Radical Honesty
by Eliezer Yudkowsky — 2007-09-10T06:09:00.000Z

I recently ran across this interesting article about Radical Honesty, a movement founded by a psychotherapist named Brad Blanton who suggests that we should kick our addiction to lying and just tell the complete truth all the time. I also like this quote from the Wikipedia article on Radical Honesty: "The significant majority of participants in the Radical Honesty
workshops report dramatic changes in their lives after taking the
course, though they are not always comfortable and positive." The movement visibly suffers from having been founded by a psychotherapist - it's more about the amazing happiness that absolute truth-telling can bring to your relationships (!!) rather than such rationalist values as seeking truth by teaching yourself a habit of honesty, or not wishing to deceive others because it infringes on their autonomy.

I once suggested a notion called "Crocker's Rules", which was the mirror image of Radical Honesty - rather than telling the whole truth to other people, you would strive to always allow others to tell you the complete truth without being offended.

Crocker's Rules didn't give you the right to say anything offensive, but other people could say potentially offensive things to you, and it was your responsibility not to be offended.
This was surprisingly hard to explain to people; many people would read
the careful explanation and hear, "Crocker's Rules mean you can say
offensive things to other people."

I was initially a bit suspicious of Blanton's movement - it seemed like the mirror-image that so many people misinterpreted, the option of saying offensive things to other people. But Blanton makes it not only optional, but mandatory to speak your mind - a far greater inconvenience than Crocker's Rules would ever impose on anyone.

Crocker's Rules didn't catch on. Maybe it was too hard to tell the
difference between someone delivering a slap in the face, and someone
deliberately invoking Crocker's Rules - you don't want to miss a real
clue to real hostility because of your acceptance; you wouldn't want to
not believe a true fact, even if the true fact is that someone else
hates you. And third parties may assume the truthteller is an offensive person
no matter how much the receiver disclaims offense - they may assume the
receiver is "just being polite", or that requesting honesty does not
excuse its offens...
The Crackpot Offer
by Eliezer Yudkowsky — 2007-09-08T14:32:49.000Z

When I was very young—I think thirteen or maybe fourteen—I thought I had found a disproof of Cantor’s Diagonal Argument, a famous theorem which demonstrates that the real numbers outnumber the rational numbers. Ah, the dreams of fame and glory that danced in my head!

My idea was that since each whole number can be decomposed into a bag of powers of 2, it was possible to map the whole numbers onto the set of subsets of whole numbers simply by writing out the binary expansion. The number 13, for example, 1101, would map onto {0, 2, 3}. It took a whole week before it occurred to me that perhaps I should apply Cantor’s Diagonal Argument to my clever construction, and of course it found a counterexample—the binary number (. . . 1111), which does not correspond to any finite whole number.

So I found this counterexample, and saw that my attempted disproof was false, along with my dreams of fame and glory.

I was initially a bit disappointed.

The thought went through my mind: “I’ll get that theorem eventually! Someday I’ll disprove Cantor’s Diagonal Argument, even though my first try failed!” I resented the theorem for being obstinately true, for depriving me of my fame and fortune, and I began to look for other disproofs.

And then I realized something. I realized that I had made a mistake, and that, now that I’d spotted my mistake, there was absolutely no reason to suspect the strength of Cantor’s Diagonal Argument any more than other major theorems of mathematics.

I saw then very clearly that I was being offered the opportunity to become a math crank, and to spend the rest of my life writing angry letters in green ink to math professors. (I’d read a book once about math cranks.)

I did not wish this to be my future, so I gave a small laugh, and let it go. I waved Cantor’s Diagonal Argument on with all good wishes, and I did not question it again.

And I don’t remember, now, if I thought this at the time, or if I thought it afterward . . . but what a terribly unfair test to visit upon a child of thirteen. That I had to be that rational, already, at that age, or fail.

The smarter you are, the younger you may be, the first time you have what looks to you like a really revolutionary idea. I was lucky in that I saw the mistake myself; that i...
Anchoring and Adjustment
by Eliezer Yudkowsky — 2007-09-07T21:33:51.000Z

Suppose I spin a Wheel of Fortune device as you watch, and it comes up pointing to 65. Then I ask: Do you think the percentage of countries in the United Nations that are in Africa is above or below this number? What do you think is the percentage of UN countries that are in Africa? Take a moment to consider these two questions yourself, if you like, and please don’t Google.

Also, try to guess, within five seconds, the value of the following arithmetical expression. Five seconds. Ready? Set . . . Go!

1 × 2 × 3 × 4 × 5 × 6 × 7 × 8

Tversky and Kahneman recorded the estimates of subjects who saw the Wheel of Fortune showing various numbers.1 The median estimate of subjects who saw the wheel show 65 was 45%; the median estimate of subjects who saw 10 was 25%.

The current theory for this and similar experiments is that subjects take the initial, uninformative number as their starting point or anchor; and then they adjust upward or downward from their starting estimate until they reach an answer that “sounds plausible”; and then they stop adjusting. This typically results in under-adjustment from the anchor—more distant numbers could also be “plausible,” but one stops at the first satisfying-sounding answer.

Similarly, students shown “1 × 2 × 3 × 4 × 5 × 6 × 7 × 8” made a median estimate of 512, while students shown “8 × 7 × 6 × 5 × 4 × 3 × 2 × 1” made a median estimate of 2,250. The motivating hypothesis was that students would try to multiply (or guess-combine) the first few factors of the product, then adjust upward. In both cases the adjustments were insufficient, relative to the true value of 40,320; but the first set of guesses were much more insufficient because they started from a lower anchor.

Tversky and Kahneman report that offering payoffs for accuracy did not reduce the anchoring effect.

Strack and Mussweiler asked for the year Einstein first visited the United States.2 Completely implausible anchors, such as 1215 or 1992, produced anchoring effects just as large as more plausible anchors such as 1905 or 1939.

There are obvious applications in, say, salary negotiations, or buying a car. I won’t suggest that you exploit it, but watch out for ...
Why is the Future So Absurd?
by Eliezer Yudkowsky — 2007-09-07T08:42:13.000Z

Followup to: Stranger than History, Absurdity Heuristic / Absurdity Bias

Why is the future more absurd than people seem to expect? (That is: Why, historically, has the future so often turned out to be more "absurd" than people seem to have expected?)

One obvious reason is hindsight bias. Hindsight does not just cause people to severely underestimate how much they would have been surprised. Hindsight also leads people to overestimate how much attention they would have paid to the key factors, the factors that turned out to be important. As R. H. Tawney put it:

"Historians
give an appearance of inevitability to an existing order by dragging into
prominence the forces which have triumphed and thrusting into the background
those which they have swallowed up."

When people look at historical changes and think "I could have predicted X" or "You could have predicted X if you looked at factors 1, 2, and 3"; then they forget that people did not, in fact, predict X, perhaps because they were distracted by factors 4 through 117. People read history books, see coherent narratives, and think that's how Time works. Underestimating the surprise of the present, they overestimate the predictability of the future.

I suspect that a major factor contributing to absurdity bias is that, when we look over history, we see changes away from absurd conditions such as everyone being a peasant farmer and women not having the vote, toward normal conditions like a majority middle class and equal rights. When people look at history, they see a series of normalizations. They learn the rule, "The future grows ever less absurd over time."

Perhaps one way to comprehend the bizarreness of the future would be to
try and imagine historical changes occurring in reverse - how absurd
would it be if all your electrical appliances suddenly disappeared, or
you were transformed into a peasant farmer? Even if the future is
nicer than the past, it will feel at least that absurd.

The correspondence bias of social psychology may also play a role in how we fail to learn from history - or so my own experience
suggests. When we read about the strange behaviors of people in other
eras, we may see them as people with a disposition to that strange behavior,
rather than properly comprehending the st...
So much for the factual public debate that democracies completely fail to be built on
by KatjaGrace — 2007-09-06T15:07:00.000Z

Publicly refuting facts often reinforces their believed truth in the minds of the public, and they will even credit the misinformation to the organisation denying it.

A Washington Post article reports an experiment where people were given fliers labelling common ideas about influenza ‘true’ or ‘false’. Half an hour later older people already remembered 28% of falsities as facts, and three days later 40% , by which time younger people caught up to the older people’s half hour figure. It seems that the repetition of the false information helps to ingrain it, while the extra information – that it is false – is soon lost.

So how do you have factual public debate when whoever starts it automaticly has a major advantage? Denial and silence can have the same effect as agreeing, but denying is still best. A good proportion of people (a few days later at least) do remember whether their facts are false or not. Though as TWP discusses, it’s probably best to deny things without actually mentioning them if possible. That is, fiercly support something mutually exclusive.

As noted in the discussion of Overcoming Bias’ post on this, if people have anything at stake they might pay more attention. While this has problems of its own (discussed there), a big obvious gap where it’s important for people to have accurate information on topics not directly concerning them is in democracy. Just another in a long list of problems with the kinds of democratic systems we use, but in conjunction with rational ignorance it makes the chance of voters having a clue about anything not immediately concerning them both tiny and tied firmly to the chance of the first buyer of lots of ads happening to be right.
Availability
by Eliezer Yudkowsky — 2007-09-06T06:55:41.000Z

The availability heuristic is judging the frequency or probability of an event by the ease with which examples of the event come to mind.

A famous 1978 study by Lichtenstein, Slovic, Fischhoff, Layman, and Combs, “Judged Frequency of Lethal Events,” studied errors in quantifying the severity of risks, or judging which of two dangers occurred more frequently. Subjects thought that accidents caused about as many deaths as disease; thought that homicide was a more frequent cause of death than suicide. Actually, diseases cause about sixteen times as many deaths as accidents, and suicide is twice as frequent as homicide.

An obvious hypothesis to account for these skewed beliefs is that murders are more likely to be talked about than suicides—thus, someone is more likely to recall hearing about a murder than hearing about a suicide. Accidents are more dramatic than diseases—perhaps this makes people more likely to remember, or more likely to recall, an accident. In 1979, a followup study by Combs and Slovic showed that the skewed probability judgments correlated strongly (0.85 and 0.89) with skewed reporting frequencies in two newspapers. This doesn’t disentangle whether murders are more available to memory because they are more reported-on, or whether newspapers report more on murders because murders are more vivid (hence also more remembered). But either way, an availability bias is at work.

Selective reporting is one major source of availability biases. In the ancestral environment, much of what you knew, you experienced yourself; or you heard it directly from a fellow tribe-member who had seen it. There was usually at most one layer of selective reporting between you, and the event itself. With today’s Internet, you may see reports that have passed through the hands of six bloggers on the way to you—six successive filters. Compared to our ancestors, we live in a larger world, in which far more happens, and far less of it reaches us—a much stronger selection effect, which can create much larger availability biases.

In real life, you’re unlikely to ever meet Bill Gates. But thanks to selective reporting by the media, you may be tempted to compare your life success to his—and suffer hedonic penalties accordingly. The objective frequency of Bill Gates is 0.00000000015, but you hear about him much more often. Conversely, 19% of the planet lives on less than $1/day, and...
Absurdity Heuristic, Absurdity Bias
by Eliezer Yudkowsky — 2007-09-05T03:20:06.000Z

Followup to: Stranger Than History, Robin's post What Evidence Ease of Imagination?

I've been pondering lately the notion of "absurdity" - wondering what exactly goes on in people's minds when they utter the adjective "absurd" or the objection "Absurd!"

If there is an absurdity heuristic, it would seem, at first glance, to be the mirror image of the well-known representativeness heuristic. The less X resembles Y, or the more X violates typicality assumptions of Y, the less probable that X is the product, explanation, or outcome of Y. A sequence of events is less probable when it involves an egg unscrambling itself, water flowing upward, machines thinking or dead people coming back to life. Since human psychology is not a pure structure of quantitative probabilities, it is easy to imagine that the absurdity heuristic is separate from the representativeness heuristic - implemented by separate absurdity-detecting brainware.

I suspect people may also be more sensitive to "absurdity" that invalidates a plan or indicates cheating. Consider the difference between "I saw a little blue man yesterday, walking down the street" versus "I'm going to jump off this cliff and a little blue man will catch me on the way down" or "If you give me your wallet, a little blue man will bring you a pot of gold." (I'm thinking, in particular, about how projections of future technology are often met by the objection, "That's absurd!", and how the objection seems more violent than usual in this case.)
As Robin observed, a heuristic is not necessarily a bias. The vast majority of objects do not fall upward. And yet helium balloons are an exception. When are exceptions predictable?

I can think of three major circumstances where the absurdity heuristic gives rise to an absurdity bias:

The first case is when we have information about underlying laws which should override surface reasoning. If you know why most objects fall, and you can calculate how fast they fall, then your calculation that a helium balloon should rise at such-and-such a rate, ought to strictly override the absurdity of an object falling upward. If you can do deep calculations, you have no need for qualitative surface reasoning. But we may find it hard to attend to mere calculations in the face of surface absurdity, until we see the balloon rise.

(In 1913, L...
Markets are a kind of electrochemical cell
by KatjaGrace — 2007-09-04T16:49:00.000Z

There are two processes taking place: adding and using up value from units. When everything is mixed together and these processes are happening in one place, they happen slowly (think of subsistence production by consumers). Separate them to their own containers and they happen faster (think production in factories and consumption in homes). The containers must be joined by a channel for units to move according to their value, and a wire for charges to balance that. The same value is removed from particles in one container as added to others in the other.

The extra energy pushing the charges and value laden particles between the containers can be used to run things like light bulbs and welfare systems. Alternatively it can be used to run a small heat to warm up the reaction, or an advertising industry.

While the charges can move around indefinitely, the particles eventually run out. Then it’s all over. With any luck/sensible policy the metaphor doesn’t continue this far.
"Science" as Curiosity-Stopper
by Eliezer Yudkowsky — 2007-09-03T20:04:40.000Z

Imagine that I, in full view of live television cameras, raised my hands and chanted abracadabra and caused a brilliant light to be born, flaring in empty space beyond my outstretched hands. Imagine that I committed this act of blatant, unmistakeable sorcery under the full supervision of James Randi and all skeptical armies. Most people, I think, would be fairly curious as to what was going on.

But now suppose instead that I don’t go on television. I do not wish to share the power, nor the truth behind it. I want to keep my sorcery secret. And yet I also want to cast my spells whenever and wherever I please. I want to cast my brilliant flare of light so that I can read a book on the train—without anyone becoming curious. Is there a spell that stops curiosity?

Yes indeed! Whenever anyone asks “How did you do that?” I just say “Science!”

It’s not a real explanation, so much as a curiosity-stopper. It doesn’t tell you whether the light will brighten or fade, change color in hue or saturation, and it certainly doesn’t tell you how to make a similar light yourself. You don’t actually know anything more than you knew before I said the magic word. But you turn away, satisfied that nothing unusual is going on.

Better yet, the same trick works with a standard light switch.

Flip a switch and a light bulb turns on. Why?

In school, one is taught that the password to the light bulb is “Electricity!” By now, I hope, you’re wary of marking the light bulb “understood” on such a basis. Does saying “Electricity!” let you do calculations that will control your anticipation of experience? There is, at the least, a great deal more to learn.1

If you thought the light bulb was scientifically inexplicable, it would seize the entirety of your attention. You would drop whatever else you were doing, and focus on that light bulb.

But what does the phrase “scientifically explicable” mean? It means that someone else knows how the light bulb works. When you are told the light bulb is “scientifically explicable,” you don’t know more than you knew earlier; you don’t know whether the light bulb will brighten or fade. But because someone else knows, it devalues the knowledge in your eyes. You become less curious.

Someone is bound to say, “If the light bulb were unknown to science, you could gain fame and fortune by investigating it.” But I’m not talking about greed. I’m not talking about career ambiti...