trentmkelly/LessWrong-43k huggingface dataset:
Purpose and Pragmatism
by Eliezer Yudkowsky — 2007-11-26T06:51:29.000Z

Followup to: Making Beliefs Pay Rent, Lost Purposes

Thus runs the ancient parable:

If a tree falls in a
forest and no one hears it, does it make a sound?
One says, "Yes it does, for it makes
vibrations in the air."
Another
says, "No it does not, for there is no auditory processing in any
brain."

So begins a long, acrimonious battle...

The conventional resolution is that the two are fighting over the definition of a word, and such labels do not have intrinsic definitions, only agreed-upon definitions.

Yet if you need to know about the forest for any pragmatic reason - if there is anything you plan on doing with the knowledge - then the answer is no longer a matter of mutual agreement. If, for example, you need to know whether landmines will be set off by the tree falling, then you cannot make the land mines explode or unexplode by any possible amount of agreement about the meaning of the word "sound". You can get the whole world to agree, one way or the other, and it still won't make a difference.

You find yourself in an unheard-falling-tree dilemma, only when you become curious about a question with no pragmatic use, and no predictive consequences. Which suggests that you may be playing loose with your purposes.

So does this mean that truth reduces to usefulness? But this, itself,
would be a purpose-loss, a subgoal stomp, a mistaking of the indicator for the indicated. Usefulness for prediction,
and demonstrated powers of manipulation, is one of the best indicators of truth. This does
not mean that usefulness is truth. You might as well say that the act of driving to the supermarket is eating chocolate.

There is, nonetheless, a deep similarity between the pragmatic and
the epistemic arts of rationality, in the matter of keeping your eye
on the ball.

In pragmatic rationality, keeping your eye on the ball means holding to your purpose: Being aware of how each act leads to
its consequence, and not losing sight of utilities in leaky
generalizations about expected utilities. If you hold firmly in your mind the image of a drained swamp, you will be less likely to get lost in fighting alligators.

In epistemic rationality,
keeping your eye on the ball means holding to your question: Being
aware of what each indicator says about its indicatee, and not losing
sight of the original que...
Lost Purposes
by Eliezer Yudkowsky — 2007-11-25T09:01:50.000Z

It was in either kindergarten or first grade that I was first asked to pray, given a transliteration of a Hebrew prayer. I asked what the words meant. I was told that so long as I prayed in Hebrew, I didn't need to know what the words meant, it would work anyway.

That was the beginning of my break with Judaism.

As you read this, some young man or woman is sitting at a desk in a university, earnestly studying material they have no intention of ever using, and no interest in knowing for its own sake. They want a high-paying job, and the high-paying job requires a piece of paper, and the piece of paper requires a previous master's degree, and the master's degree requires a bachelor's degree, and the university that grants the bachelor's degree requires you to take a class in 12th-century knitting patterns to graduate. So they diligently study, intending to forget it all the moment the final exam is administered, but still seriously working away, because they want that piece of paper.

Maybe you realized it was all madness, but I bet you did it anyway. You didn't have a choice, right?

A recent study here in the Bay Area showed that 80% of teachers in K-5 reported spending less than one hour per week on science, and 16% said they spend no time on science. Why? I'm given to understand the proximate cause is the No Child Left Behind Act and similar legislation. Virtually all classroom time is now spent on preparing for tests mandated at the state or federal level. I seem to recall (though I can't find the source) that just taking mandatory tests was 40% of classroom time in one school.

The old Soviet bureaucracy was famous for being more interested in appearances than reality. One shoe factory overfulfilled its quota by producing lots of tiny shoes. Another shoe factory reported cut but unassembled leather as a "shoe". The superior bureaucrats weren't interested in looking too hard, because they also wanted to report quota overfulfillments. All this was a great help to the comrades freezing their feet off.

It is now being suggested in several sources that an actual majority of published findings in medicine, though "statistically significant with p<0.05", are untrue. But so long as p<0.05 remains the threshold for publication, why should anyone hold themselves t...
The Hidden Complexity of Wishes
by Eliezer Yudkowsky — 2007-11-24T00:12:33.000Z

(It has come to my attention that this article is currently being misrepresented as proof that I/MIRI previously advocated that it would be very difficult to get machine superintelligences to understand or predict human values. This would obviously be false, and also, is not what is being argued below. The example in the post below is not about an Artificial Intelligence literally at all! If the post were about what AIs supposedly can't do, the central example would have used an AI! The point that is made below will be about the algorithmic complexity of human values. This point is relevant within a larger argument, because it bears on the complexity of what you need to get an artificial superintelligence to want or value; rather than bearing on what a superintelligence supposedly could not predict or understand. -- EY, May 2024.)

<

"I wish to live in the locations of my choice, in a physically healthy, uninjured, and apparently normal version of my current body containing my current mental state, a body which will heal from all injuries at a rate three sigmas faster than the average given the medical technology available to me, and which will be protected from any diseases, injuries or illnesses causing disability, pain, or degraded functionality or any sense, organ, or bodily function for more than ten days consecutively or fifteen days in any year..."
-- The Open-Source Wish Project, Wish For Immortality 1.1

There are three kinds of genies: Genies to whom you can safely say "I wish for you to do what I should wish for"; genies for which no wish is safe; and genies that aren't very powerful or intelligent.

Suppose your aged mother is trapped in a burning building, and it so happens that you're in a wheelchair; you can't rush in yourself. You could cry, "Get my mother out of that building!" but there would be no one to hear.

Luckily you have, in your pocket, an Outcome Pump. This handy device squeezes the flow of time, pouring probability into some outcomes, draining it from others.

The Outcome Pump is not sentient. It contains a tiny time machine, which resets time unless a specified outcome occurs. For example, if you hooked up the Outcome Pump's sensors to a coin, and specified that the time machine should keep resetting until it sees the coin come up heads, and ...
Leaky Generalizations
by Eliezer Yudkowsky — 2007-11-22T21:16:11.000Z

Are apples good to eat? Usually, but some apples are rotten.

Do humans have ten fingers? Most of us do, but plenty of people have lost a finger and nonetheless qualify as "human".

Unless you descend to a level of description far below any macroscopic object - below societies, below people, below fingers, below tendon and bone, below cells, all the way down to particles and fields where the laws are truly universal - then practically every generalization you use in the real world will be leaky.

(Though there may, of course, be some exceptions to the above rule...)

Mostly, the way you deal with leaky generalizations is that, well,
you just have to deal. If the cookie market almost always closes at 10pm,
except on Thanksgiving it closes at 6pm, and today happens to be
National Native American Genocide Day, you'd better show up before 6pm or you won't get a
cookie.

Our ability to manipulate leaky generalizations is opposed by need for closure, the degree to which we want to say once and for all that humans have fingers, and get frustrated when we have to tolerate continued ambiguity. Raising the value of the stakes can increase need for closure - which shuts down complexity tolerance when complexity tolerance is most needed.

Life would be complicated even if the things we wanted were simple (they aren't). The leakyness of leaky generalizations about what-to-do-next would leak in from the leaky structure of the real world. Or to put it another way:

Instrumental values often have no specification which is both compact and local.

Suppose there's a box containing a million dollars. The box is
locked, not with an ordinary combination lock, but with a dozen keys
controlling a machine that can open the box. If you know how the machine works, you can
deduce which sequences of key-presses will open the box. There's more
than one key sequence that can trigger the button. But if you
press a sufficiently wrong sequence, the machine incinerates the
money. And if you don't know about the machine, there's no
simple rules like "Pressing any key three times opens the box" or
"Pressing five different keys with no repetitions incinerates the
money."

There's a compact nonlocal specification of which keys you
want to press: You want to press keys such that they open the box.
You can w...
Not for the Sake of Happiness (Alone)
by Eliezer Yudkowsky — 2007-11-22T03:19:34.000Z

When I met the futurist Greg Stock some years ago, he argued that the joy of scientific discovery would soon be replaced by pills that could simulate the joy of scientific discovery. I approached him after his talk and said, "I agree that such pills are probably possible, but I wouldn't voluntarily take them."

And Stock said, "But they'll be so much better that the real thing won't be able to compete. It will just be way more fun for you to take the pills than to do all the actual scientific work."

And I said, "I agree that's possible, so I'll make sure never to take them."

Stock seemed genuinely surprised by my attitude, which genuinely surprised me.

One often sees ethicists arguing as if all human desires are reducible, in principle, to the desire for ourselves and others to be happy. (In particular, Sam Harris does this in The End of Faith, which I just finished perusing - though Harris's reduction is more of a drive-by shooting than a major topic of discussion.)

This isn't the same as arguing whether all happinesses can be measured on a common utility scale - different happinesses might occupy different scales, or be otherwise non-convertible. And it's not the same as arguing that it's theoretically impossible to value anything other than your own psychological states, because it's still permissible to care whether other people are happy.

The question, rather, is whether we should care about the things that make us happy, apart from any happiness they bring.

We can easily list many cases of moralists going astray by caring about things besides happiness. The various states and countries that still outlaw oral sex make a good example; these legislators would have been better off if they'd said, "Hey, whatever turns you on." But this doesn't show that all values are reducible to happiness; it just argues that in this particular case it was an ethical mistake to focus on anything else.

It is an undeniable fact that we tend to do things that make us happy, but this doesn't mean we should regard the happiness as the only reason for so acting. First, this would make it difficult to explain how we could care about anyone else's happiness - how we could treat people as ends in themselves, rather than instrumental means of obtaining a warm glow of satisfaction.

Second, just because somethi...
Truly Part Of You
by Eliezer Yudkowsky — 2007-11-21T02:18:23.000Z

A classic paper by Drew McDermott, “Artificial Intelligence Meets Natural Stupidity,” criticized AI programs that would try to represent notions like happiness is a state of mind using a semantic network:

And of course there’s nothing inside the HAPPINESS node; it’s just a naked LISP token with a suggestive English name.

So, McDermott says, “A good test for the disciplined programmer is to try using gensyms in key places and see if he still admires his system. For example, if STATE-OF-MIND is renamed G1073. . .” then we would have IS-A(HAPPINESS, G1073) “which looks much more dubious.”

Or as I would slightly rephrase the idea: If you substituted randomized symbols for all the suggestive English names, you would be completely unable to figure out what G1071(G1072, G1073) meant. Was the AI program meant to represent hamburgers? Apples? Happiness? Who knows? If you delete the suggestive English names, they don’t grow back.

Suppose a physicist tells you that “Light is waves,” and you believe the physicist. You now have a little network in your head that says:

IS-A(LIGHT, WAVES)

As McDermott says, “The whole problem is getting the hearer to notice what it has been told. Not ‘understand,’ but ‘notice.’ ” Suppose that instead the physicist told you, “Light is made of little curvy things.”1 Would you notice any difference of anticipated experience?

How can you realize that you shouldn’t trust your seeming knowledge that “light is waves”? One test you could apply is asking, “Could I regenerate his knowledge if it were somehow deleted from my mind?”

This is similar in spirit to scrambling the names of suggestively named lisp tokens in your AI program, and seeing if someone else can figure out what they allegedly “refer” to. It’s also similar in spirit to observing that an Artificial Arithmetician programmed to record and play back

Plus-Of(Seven, Six) = Thirteen

can’t regenerate the knowledge if you delete it from memory, until another human re-enters it in the database. Just as if you forgot that “light is waves,” you couldn’t get back the knowledge except the same way you got the knowledge to begin with—by asking a physicist. You couldn’t generate the knowledge for yourself, the way that physicists originally generated it.

The same experiences that lead us to formulate a belief, connect that belief to other kn...
Artificial Addition
by Eliezer Yudkowsky — 2007-11-20T07:58:50.000Z

Suppose that human beings had absolutely no idea how they performed arithmetic. Imagine that human beings had evolved, rather than having learned, the ability to count sheep and add sheep. People using this built-in ability have no idea how it worked, the way Aristotle had no idea how his visual cortex supported his ability to see things. Peano Arithmetic as we know it has not been invented. There are philosophers working to formalize numerical intuitions, but they employ notations such as

Plus-Of(Seven, Six) = Thirteen

to formalize the intuitively obvious fact that when you add "seven" plus "six", of course you get "thirteen".

In this world, pocket calculators work by storing a giant lookup table of arithmetical facts, entered manually by a team of expert Artificial Arithmeticians, for starting values that range between zero and one hundred. While these calculators may be helpful in a pragmatic sense, many philosophers argue that they're only simulating addition, rather than really adding. No machine can really count - that's why humans have to count thirteen sheep before typing "thirteen" into the calculator. Calculators can recite back stored facts, but they can never know what the statements mean - if you type in "two hundred plus two hundred" the calculator says "Error: Outrange", when it's intuitively obvious, if you know what the words mean, that the answer is "four hundred".

Philosophers, of course, are not so naive as to be taken in by these intuitions. Numbers are really a purely formal system - the label
"thirty-seven" is meaningful, not because of any inherent property of
the words themselves, but because the label refers to thirty-seven sheep in
the external world. A number is given this referential property by its semantic
network of relations to other numbers. That's why, in computer programs, the LISP token for "thirty-seven" doesn't need any internal structure - it's only meaningful because of reference and relation, not some computational property of "thirty-seven" itself.

No one has ever developed an Artificial General Arithmetician, though
of course there are plenty of domain-specific, narrow Artificial
Arithmeticians that work on numbers between "twenty" and "thirty"...
Conjuring An Evolution To Serve You
by Eliezer Yudkowsky — 2007-11-19T05:55:56.000Z

GreyThumb.blog offers an interesting analogue between research on animal breeding and the fall of Enron. Before 1995, the way animal breeding worked was that you would take the top individual performers in each generation and breed from them, or their parents. A cockerel doesn't lay eggs, so you have to observe daughter hens to determine which cockerels to breed. Sounds logical, right? If you take the hens who lay the most eggs in each generation, and breed from them, you should get hens who lay more and more eggs.

Behold the awesome power of making evolution work for you! The power that made butterflies - now constrained to your own purposes! And it worked, too. Per-cow milk output in the US doubled between 1905 and 1965, and has doubled again since then.

Yet conjuring Azathoth oft has unintended consequences, as some researchers realized in the 1990s. In the real world, sometimes you have more than animal per farm. You see the problem, right? If you don't, you should probably think twice before trying to conjure an evolution to serve you - magic is not for the unparanoid.

Selecting the hen who lays the most eggs doesn't necessarily get you the most efficient egg-laying metabolism. It may get you the most dominant hen, that pecked its way to the top of the pecking order at the expense of other hens. Individual selection doesn't necessarily work to the benefit of the group, but a farm's productivity is determined by group outputs.

Indeed, for some strange reason, the individual breeding programs which had been so successful at increasing egg production now required hens to have their beaks clipped, or be housed in individual cages, or they would peck each other to death.

While the conditions for group selection are only rarely right in Nature, one can readily impose genuine group selection in the laboratory. After only 6 generations of artificially imposed group selection - breeding from the hens in the best groups, rather than the best individual hens - average days of survival increased from 160 to 348, and egg mass per bird increased from 5.3 to 13.3 kg. At 58 weeks of age, the selected line had 20% mortality compared to the control group at 54%. A commercial line of hens, allowed to grow up with unclipped beaks, had 89% mortality at 58 weeks.

And the fall of Enron? ...
The Simple Math of Everything
by Eliezer Yudkowsky — 2007-11-17T22:42:12.000Z

I am not a professional evolutionary biologist. I only know a few equations, very simple ones by comparison to what can be found in any textbook on evolutionary theory with math, and on one memorable occasion I used one incorrectly. For me to publish an article in a highly technical ev-bio journal would be as impossible as corporations evolving. And yet when I'm dealing with almost anyone who's not a professional evolutionary biologist...

It seems to me that there's a substantial advantage in knowing the drop-dead basic fundamental embarrassingly simple mathematics in as many different subjects as you can manage. Not, necessarily, the high-falutin' complicated damn math that appears in the latest journal articles. Not unless you plan to become a professional in the field. But for people who can read calculus, and sometimes just plain algebra, the drop-dead basic mathematics of a field may not take that long to learn. And it's likely to change your outlook on life more than the math-free popularizations or the highly technical math.

Not Jacobean matrices for frequency-dependent gene selection; just Haldane's
calculation of time to fixation. Not quantum physics; just the wave
equation for sound in air. Not the maximum entropy solution using
Lagrange multipliers; just Bayes's Rule.

The Simple Math of Everything, written for people who are good at math, might not be all that weighty a volume. How long does it take to explain Bayes's Rule to someone who's good at math? Damn would I like to buy that book and send it back in time to my 16-year-old self. But there's no way I have time to write this book, so I'm tossing the idea out there.

Even in reading popular works on science, there is yet power. You don't want to end up
like those poor souls in that recent interview (I couldn't Google)
where a well-known scientist in field XYZ thinks the universe is 100
billion years old. But it seems to me that there's substantially more
power in pushing until you encounter some basic math. Not complicated
math, just basic math. F=ma is too simple, though. You should take the highest low-hanging fruit you can reach.

Yes, there are sciences whose soul is not in their math, yet which are
nonetheless incredibly important and enlightening. Evolutionary
psychology, for example.&...
No Evolutions for Corporations or Nanodevices
by Eliezer Yudkowsky — 2007-11-17T02:24:09.000Z

"The laws of physics and the rules of math don't cease to apply. That leads me to believe that evolution doesn't stop. That further leads me to believe that nature —bloody in tooth and claw, as some have termed it —will simply be taken to the next level...

"[Getting rid of Darwinian evolution is] like trying to get rid of gravitation. So long as there are limited resources and multiple competing actors capable of passing on characteristics, you have selection pressure."

—Perry Metzger, predicting that the reign of natural selection would continue into the indefinite future.

In evolutionary biology, as in many other fields, it is important to think quantitatively rather than qualitatively. Does a beneficial mutation "sometimes spread, but not always"? Well, a psychic power would be a beneficial mutation, so you'd expect it to spread, right? Yet this is qualitative reasoning, not quantitative—if X is true, then Y is true; if psychic powers are beneficial, they may spread. In Evolutions Are Stupid, I described the equations for a beneficial mutation's probability of fixation, roughly twice the fitness advantage (6% for a 3% advantage). Only this kind of numerical thinking is likely to make us realize that mutations which are only rarely useful are extremely unlikely to spread, and that it is practically impossible for complex adaptations to arise without constant use. If psychic powers really existed, we should expect to see everyone using them all the time—not just because they would be so amazingly useful, but because otherwise they couldn't have evolved in the first place.

"So long as there are limited resources and multiple competing actors capable of passing on characteristics, you have selection pressure." This is qualitative reasoning. How much selection pressure?

While there are several candidates for the most important equation in evolutionary biology, I would pick Price's Equation, which in its simplest formulation reads:

Δz=cov(vi,zi)

change in average characteristic = covariance(relative fitness, characteristic)

This is a very powerful and general formula. For example, a particular gene for height can be the Z, the characteristic that changes, in which case Price's Equation says that the change in the probability of possessing this gene equals the covarian...