trentmkelly/LessWrong-43k huggingface dataset:
Rationality Quotes 21
by Eliezer Yudkowsky — 2009-01-06T02:45:08.000Z

"The most dangerous thing in the world is finding someone you agree with. If a TV station news is saying exactly what you think is right, BEWARE! You are very likely only reinforcing your beliefs, and not being supplied with new information."
-- SmallFurryCreature

"Companies deciding which kind of toothpaste to market have much more rigorous, established decision-making processes to refer to than the most senior officials of the U.S. government deciding whether or not to go to war."
-- Michael Mazarr

"Everything I ever thought about myself - who I was, what I am - was a lie. You have no idea how astonishingly liberating that feels."
-- Neil Gaiman, Stardust

"Saul speaks of the 'intense desire for survival on the part of virtually everyone on earth,' and our 'failure' in spite of this. I have often pointed out that the so-called 'survival instinct' is reliable only in clear and present danger - and even then only if the individual is still relatively healthy and vigorous. If the danger is indirect, or remote in time, or if the person is weak or depressed - or even if required action would violate established habits - forget the 'survival instinct.' It isn't that simple."
-- Robert Ettinger on cryonics

"We are running out of excuses. We just have to admit that real AI is one of the lowest research priorities, ever. Even space is considered more important. Nevermind baseball, tribology, or TV evangelism."
-- Eugen Leitl

"That's something that's always struck me as odd about humanity. Our first response to someone's bad news is "I'm sorry", as though we feel that someone should take responsibility for all the $&#ed up randomness that goes on in this universe."
-- Angels 2200

"We were supremely lucky to be born into this precise moment in history. It is with an ever-cresting crescendo of wonder and enthusiasm that I commend you to the great adventure and the great adventure to you."
-- Jeff Davis
Changing Emotions
by Eliezer Yudkowsky — 2009-01-05T00:05:14.000Z

Lest anyone reading this journal of a primitive man should think we spend our time mired in abstractions, let me also say that I am discovering the richness available to those who are willing to alter their major characteristics. The variety of emotions available to a reconfigured human mind, thinking thoughts impossible to its ancestors...
The emotion of -*-, describable only as something between sexual love and the joy of intellection—making love to a thought? Or &&, the true reverse of pain, not "pleasure" but a "warning" of healing, growth and change. Or (^+^), the most complex emotion yet discovered, felt by those who consciously endure the change between mind configurations, and experience the broad spectrum of possibilities inherent in thinking and being.
—Greg Bear, Eon

So... I'm basically on board with that sort of thing as a fine and desirable future. But I think that the difficulty and danger of fiddling with emotions is oft-underestimated. Not necessarily underestimated by Greg Bear, per se; the above journal entry is from a character who was receiving superintelligent help.

But I still remember one time on the Extropians mailing list when someone talked about creating a female yet "otherwise identical" copy of himself. Something about that just fell on my camel's back as the last straw. I'm sorry, but there are some things that are much more complicated to actually do than to rattle off as short English phrases, and "changing sex" has to rank very high on that list. Even if you're omnipotent so far as raw ability goes, it's not like people have a binary attribute reading "M" or "F" that can be flipped as a primitive action.

Changing sex makes a good, vivid example of the sort of difficulties you might run into when messing with emotional architecture, so I'll use it as my archetype:

Let's suppose that we're talking about an M2F transformation. (F2M should be a straightforward transform of this discussion; I do want to be specific rather than talking in vague generalities, but I don't want to parallelize every sentence.) (Oddly enough, every time I can recall hearing someone say "I want to know what it's like to be the opposite sex", the speaker has been male. I don't know if that's a genuine gender difference ...
Repeated thought
by KatjaGrace — 2009-01-04T21:14:00.000Z

Eliezer Yudkowsky of OB suggests thinking and doing entirely new things for a day:

Don’t read any book you’ve read before. Don’t read any author you’ve read before. Don’t visit any website you’ve visited before. Don’t play any game you’ve played before. Don’t listen to familiar music that you already know you’ll like. If you go on a walk, walk along a new path even if you have to drive to a different part of the city for your walk. Don’t go to any restaurant you’ve been to before, order a dish that you haven’t had before. Talk to new people (even if you have to find them in an IRC channel) about something you don’t spend much time discussing.

And most of all, if you become aware of yourself musing on any thought you’ve thunk before, then muse on something else. Rehearse no old grievances, replay no old fantasies.

The comments and its reposting to MR suggests that this is popular advice.

It’s interesting that, despite the warm reception, this idea needs pointing out, and trying for one experimental day.

Having habits for things like brushing teeth is useful – the more automatic uninteresting or unenjoyable experiences are, the more time and thought can be devoted to other things. Habits for places to go could be argued for – if you love an experience, why change it?

But why should we want to repeat thoughts a lot? Seems we say we don’t. So, why do we do it? Do we do it? If we can stop when Eliezer suggests it, why don’t we notice and stop on our own? Is it that habits are unconscious; a state that doesn’t lend itself to noticing things? Has the usefulness of other habits made us so habitual that our thoughts are caught up in it?

What can we do about it?

As a side note, perhaps the quantity of unconscious habit in a life is related to the way time speeds up as you age.
Growing Up is Hard
by Eliezer Yudkowsky — 2009-01-04T03:55:49.000Z

Terrence Deacon's The Symbolic Species is the best book I've ever read on the evolution of intelligence. Deacon somewhat overreaches when he tries to theorize about what our X-factor is; but his exposition of its evolution is first-class.

Deacon makes an excellent case—he has quite persuaded me—that the increased relative size of our frontal cortex, compared to other hominids, is of overwhelming importance in understanding the evolutionary development of humanity. It's not just a question of increased computing capacity, like adding extra processors onto a cluster; it's a question of what kind of signals dominate, in the brain.

People with Williams Syndrome (caused by deletion of a certain region on chromosome 7) are hypersocial, ultra-gregarious; as children they fail to show a normal fear of adult strangers. WSers are cognitively impaired on most dimensions, but their verbal abilities are spared or even exaggerated; they often speak early, with complex sentences and large vocabulary, and excellent verbal recall, even if they can never learn to do basic arithmetic.

Deacon makes a case for some Williams Syndrome symptoms coming from a frontal cortex that is relatively too large for a human, with the result that prefrontal signals—including certain social emotions—dominate more than they should.

"Both postmortem analysis and MRI analysis have revealed brains with a reduction of the entire posterior cerebral cortex, but a sparing of the cerebellum and frontal lobes, and perhaps even an exaggeration of cerebellar size," says Deacon.

Williams Syndrome's deficits can be explained by the shrunken posterior cortex—they can't solve simple problems involving shapes, because the parietal cortex, which handles shape-processing, is diminished. But the frontal cortex is not actually enlarged; it is simply spared. So where do WSers' augmented verbal abilities come from?

Perhaps because the signals sent out by the frontal cortex, saying "pay attention to this verbal stuff!", win out over signals coming from the shrunken sections of the brain. So the verbal abilities get lots of exercise—and other abilities don't.

Similarly with the hyper-gregarious nature of WSers; the signal saying "Pay attention to this person!", originating in the frontal areas where social processing gets done, dominates the emotional landscape.

And Williams Syndrome is not ...
The Uses of Fun (Theory)
by Eliezer Yudkowsky — 2009-01-02T20:30:33.000Z

"But is there anyone who actually wants to live in a Wellsian Utopia? On the contrary, not to live in a world like that, not to wake up in a hygenic garden suburb infested by naked schoolmarms, has actually become a conscious political motive. A book like Brave New World is an expression of the actual fear that modern man feels of the rationalised hedonistic society which it is within his power to create."
—George Orwell, Why Socialists Don't Believe in Fun

There are three reasons I'm talking about Fun Theory, some more important than others:

If every picture ever drawn of the Future looks like a terrible place to actually live, it might tend to drain off the motivation to create the future. It takes hope to sign up for cryonics.
People who leave their religions, but don't familiarize themselves with the deep, foundational, fully general arguments against theism, are at risk of backsliding. Fun Theory lets you look at our present world, and see that it is not optimized even for considerations like personal responsibility or self-reliance. It is the fully general reply to theodicy.
Going into the details of Fun Theory helps you see that eudaimonia is actually complicated —that there are a lot of properties necessary for a mind to lead a worthwhile existence. Which helps you appreciate just how worthless a galaxy would end up looking (with extremely high probability) if it was optimized by something with a utility function rolled up at random.

To amplify on these points in order:

(1) You've got folks like Leon Kass and the other members of Bush's "President's Council on Bioethics" running around talking about what a terrible, terrible thing it would be if people lived longer than threescore and ten. While some philosophers have pointed out the flaws in their arguments, it's one thing to point out a flaw and another to provide a counterexample. "Millions long for immortality who do not know what to do with themselves on a rainy Sunday afternoon," said Susan Ertz, and that argument will sound plausible for as long as you can't imagine what to do on a rainy Sunday afternoon, and it seems unlikely that anyone could imagine it.

It's not exactly the fault of Hans Moravec that his world in which humans are kept by superintelligences as pets, doesn't sound quite Utopian. Utopias ...
Free to Optimize
by Eliezer Yudkowsky — 2009-01-02T01:41:00.000Z

Stare decisis is the legal principle which binds courts to follow precedent, retrace the footsteps of other judges' decisions. As someone previously condemned to an Orthodox Jewish education, where I gritted my teeth at the idea that medieval rabbis would always be wiser than modern rabbis, I completely missed the rationale for stare decisis. I thought it was about respect for the past.

But shouldn't we presume that, in the presence of science, judges closer to the future will know more—have new facts at their fingertips—which enable them to make better decisions? Imagine if engineers respected the decisions of past engineers, not as a source of good suggestions, but as a binding precedent!—That was my original reaction. The standard rationale behind stare decisis came as a shock of revelation to me; it considerably increased my respect for the whole legal system.

This rationale is jurisprudence constante: The legal system must above all be predictable, so that people can execute contracts or choose behaviors knowing the legal implications.

Judges are not necessarily there to optimize, like an engineer. The purpose of law is not to make the world perfect. The law is there to provide a predictable environment in which people can optimize their ownfutures.

I was amazed at how a principle that at first glance seemed so completely Luddite, could have such an Enlightenment rationale. It was a "shock of creativity"—a solution that ranked high in my preference ordering and low in my search ordering, a solution that violated my previous surface generalizations. "Respect the past just because it's the past" would not have easily occurred to me as a good solution for anything.

There's a peer commentary in Evolutionary Origins of Morality which notes in passing that "other things being equal, organisms will choose to reward themselves over being rewarded by caretaking organisms". It's cited as the Premack principle, but the actual Premack principle looks to be something quite different, so I don't know if this is a bogus result, a misremembered citation, or a nonobvious derivation. If true, it's definitely interesting from a fun-theoretic perspective.

Optimization is the ability to squeeze the future into regions high in your preference ordering. Living by my own strength, means squeezing my own future—n...
A New Day
by Eliezer Yudkowsky — 2008-12-31T18:40:27.000Z

Somewhere in the vastnesses of the Internet and the almost equally impenetrable thicket of my bookmark collection, there is a post by someone who was learning Zen meditation...

Someone who was surprised by how many of the thoughts that crossed his mind, as he tried to meditate, were old thoughts - thoughts he had thunk many times before. He was successful in banishing these old thoughts, but did he succeed in meditating? No; once the comfortable routine thoughts were banished, new and interesting and more distracting thoughts began to cross his mind instead.

I was struck, on reading this, how much of my life I had allowed to fall into routine patterns. Once you actually see that, it takes on a nightmarish quality: You can imagine your fraction of novelty diminishing and diminishing, so slowly you never take alarm, until finally you spend until the end of time watching the same videos over and over again, and thinking the same thoughts each time.

Sometime in the next week - January 1st if you have that available, or maybe January 3rd or 4th if the weekend is more convenient - I suggest you hold a New Day, where you don't do anything old.

Don't read any book you've read before. Don't read any author you've read before. Don't visit any website you've visited before. Don't play any game you've played before. Don't listen to familiar music that you already know you'll like. If you go on a walk, walk along a new path even if you have to drive to a different part of the city for your walk. Don't go to any restaurant you've been to before, order a dish that you haven't had before. Talk to new people (even if you have to find them in an IRC channel) about something you don't spend much time discussing.

And most of all, if you become aware of yourself musing on any thought you've thunk before, then muse on something else. Rehearse no old grievances, replay no old fantasies.

If it works, you could make it a holiday tradition, and do it every New Year.
Dunbar's Function
by Eliezer Yudkowsky — 2008-12-31T02:26:02.000Z

The study of eudaimonic community sizes began with a seemingly silly method of calculation: Robin Dunbar calculated the correlation between the (logs of the) relative volume of the neocortex and observed group size in primates, then extended the graph outward to get the group size for a primate with a human-sized neocortex. You immediately ask, "How much of the variance in primate group size can you explain like that, anyway?" and the answer is 76% of the variance among 36 primate genera, which is respectable. Dunbar came up with a group size of 148. Rounded to 150, and with the confidence interval of 100 to 230 tossed out the window, this became known as "Dunbar's Number".

It's probably fair to say that a literal interpretation of this number is more or less bogus.

There was a bit more to it than that, of course. Dunbar went looking for corroborative evidence from studies of corporations, hunter-gatherer tribes, and utopian communities. Hutterite farming communities, for example, had a rule that they must split at 150—with the rationale explicitly given that it was impossible to control behavior through peer pressure beyond that point.

But 30-50 would be a typical size for a cohesive hunter-gatherer band; 150 is more the size of a cultural lineage of related bands. Life With Alacrity has an excellent series on Dunbar's Number which exhibits e.g. a histogram of Ultima Online guild sizes—with the peak at 60, not 150. LWA also cites further research by PARC's Yee and Ducheneaut showing that maximum internal cohesiveness, measured in the interconnectedness of group members, occurs at a World of Warcraft guild size of 50. (Stop laughing; you can get much more detailed data on organizational dynamics if it all happens inside a computer server.)

And Dunbar himself did another regression and found that a community of 150 primates would have to spend 43% of its time on social grooming, which Dunbar interpreted as suggesting that 150 was an upper bound rather than an optimum, when groups were highly incentivized to stay together. 150 people does sound like a lot of employees for a tight-knit startup, doesn't it?

Also from Life With Alacrity:

A group of 3 is often unstable, with one person feeling left out, or else one person controlling the others by being the "split" vote. A group of 4 often devolves into t

...
Amputation of Destiny
by Eliezer Yudkowsky — 2008-12-29T18:00:00.000Z

Followup to: Nonsentient Optimizers, Can't Unbirth a Child

From Consider Phlebas by Iain M. Banks:

In practice as well as theory the Culture was beyond considerations of wealth or empire. The very concept of money—regarded by the Culture as a crude, over-complicated and inefficient form of rationing—was irrelevant within the society itself, where the capacity of its means of production ubiquitously and comprehensively exceeded every reasonable (and in some cases, perhaps, unreasonable) demand its not unimaginative citizens could make. These demands were satisfied, with one exception, from within the Culture itself. Living space was provided in abundance, chiefly on matter-cheap Orbitals; raw material existed in virtually inexhaustible quantities both between the stars and within stellar systems; and energy was, if anything, even more generally available, through fusion, annihilation, the Grid itself, or from stars (taken either indirectly, as radiation absorbed in space, or directly, tapped at the stellar core). Thus the Culture had no need to colonise, exploit, or enslave.
The only desire the Culture could not satisfy from within itself was one common to both the descendants of its original human stock and the machines they had (at however great a remove) brought into being: the urge not to feel useless. The Culture's sole justification for the relatively unworried, hedonistic life its population enjoyed was its good works; the secular evangelism of the Contact Section, not simply finding, cataloguing, investigating and analysing other, less advanced civilizations but—where the circumstances appeared to Contact to justify so doing—actually interfering (overtly or covertly) in the historical processes of those other cultures.

Raise the subject of science-fictional utopias in front of any halfway sophisticated audience, and someone will mention the Culture. Which is to say: Iain Banks is the one to beat.

Iain Banks's Culture could be called the apogee of hedonistic low-grade transhumanism. Its people are beautiful and fair, as pretty as they choose to be. Their bodies have been reengineered for swift adaptation to different gravities; and also reengineered for greater sexual endurance. Their brains contains glands that can emit various euphoric drugs on command. ...
Can't Unbirth a Child
by Eliezer Yudkowsky — 2008-12-28T17:00:00.000Z

Followup to: Nonsentient Optimizers

Why would you want to avoid creating a sentient AI? "Several reasons," I said. "Picking the simplest to explain first—I'm not ready to be a father."

So here is the strongest reason:

You can't unbirth a child.

I asked Robin Hanson what he would do with unlimited power. "Think very very carefully about what to do next," Robin said. "Most likely the first task is who to get advice from. And then I listen to that advice."

Good advice, I suppose, if a little meta. On a similarly meta level, then, I recall two excellent advices for wielding too much power:

Do less; don't do everything that seems like a good idea, but only what you must do.
Avoid doing things you can't undo.

Imagine that you knew the secrets of subjectivity and could create sentient AIs.

Suppose that you did create a sentient AI.

Suppose that this AI was lonely, and figured out how to hack the Internet as it then existed, and that the available hardware of the world was such, that the AI created trillions of sentient kin—not copies, but differentiated into separate people.

Suppose that these AIs were not hostile to us, but content to earn their keep and pay for their living space.

Suppose that these AIs were emotional as well as sentient, capable of being happy or sad. And that these AIs were capable, indeed, of finding fulfillment in our world.

And suppose that, while these AIs did care for one another, and cared about themselves, and cared how they were treated in the eyes of society—

—these trillions of people also cared, very strongly, about making giant cheesecakes.

Now suppose that these AIs sued for legal rights before the Supreme Court and tried to register to vote.

Consider, I beg you, the full and awful depths of our moral dilemma.

Even if the few billions of Homo sapiens retained a position of superior military power and economic capital-holdings—even if we could manage to keep the new sentient AIs down—

—would we be right to do so? They'd be people, no less than us.

We, the original humans, would have become a numerically tiny minority. Would we be right to make of ourselves an aristocracy and impose apartheid on the Cheesers, even if we had the power?

Would we be right to go on trying to seize the destiny of the galaxy—to make of it a place of peace, freedom, art, aesthetics, individuality, empathy, and other c...