trentmkelly/LessWrong-43k huggingface dataset:
Feeling Moral
by Eliezer Yudkowsky — 2015-03-11T19:00:58.628Z

Suppose that a disease, or a monster, or a war, or something, is killing people. And suppose you only have enough resources to implement one of the following two options:

Save 400 lives, with certainty.

Save 500 lives, with 90% probability; save no lives, 10% probability.

Most people choose option 1. Which, I think, is foolish; because if you multiply 500 lives by 90% probability, you get an expected value of 450 lives, which exceeds the 400-life value of option 1. (Lives saved don’t diminish in marginal utility, so this is an appropriate calculation.)

“What!” you cry, incensed. “How can you gamble with human lives? How can you think about numbers when so much is at stake? What if that 10% probability strikes, and everyone dies? So much for your damned logic! You’re following your rationality off a cliff!”

Ah, but here’s the interesting thing. If you present the options this way:

100 people die, with certainty.

90% chance no one dies; 10% chance 500 people die.

Then a majority choose option 2. Even though it’s the same gamble. You see, just as a certainty of saving 400 lives seems to feel so much more comfortable than an unsure gain, so too, a certain loss feels worse than an uncertain one.

You can grandstand on the second description too: “How can you condemn 100 people to certain death when there’s such a good chance you can save them? We’ll all share the risk! Even if it was only a 75% chance of saving everyone, it would still be worth it—so long as there’s a chance—everyone makes it, or no one does!”

You know what? This isn’t about your feelings. A human life, with all its joys and all its pains, adding up over the course of decades, is worth far more than your brain’s feelings of comfort or discomfort with a plan. Does computing the expected utility feel too cold-blooded for your taste? Well, that feeling isn’t even a feather in the scales, when a life is at stake. Just shut up and multiply.
A googol is 10^100—a 1 followed by one hundred zeroes. A googolplex is an even more incomprehensibly large number—it’s 10^googol, a 1 followed by a googol zeroes. Now pick some trivial inconvenience, like a hiccup, and some decidedly untrivial misfortune, like getting slowly torn limb from limb by sadistic mutant sharks. If we’re forced into a choice between either preventing a googolplex people’s hiccups, or preventing a single person’s shark attack, which choice sh

...
Ends: An Introduction
by Rob Bensinger — 2015-03-11T19:00:44.904Z

Value theory is the study of what people care about. It’s the study of our goals, our tastes, our pleasures and pains, our fears and our ambitions.

That includes conventional morality. Value theory subsumes things we wish we cared about, or would care about if we were wiser and better people—not just things we already do care about.

Value theory also subsumes mundane, everyday values: art, food, sex, friendship, and everything else that gives life its affective valence. Going to the movies with your friend Sam can be something you value even if it’s not a moral value.

We find it useful to reflect upon and debate our values because how we act is not always how we wish we’d act. Our preferences can conflict with each other. We can desire to have a different set of desires. We can lack the will, the attention, or the insight needed to act the way we’d like to.

Humans do care about their actions’ consequences, but not consistently enough to formally qualify as agents with utility functions. That humans don’t act the way they wish they would is what we mean when we say “humans aren’t instrumentally rational.”

Theory and Practice

Adding to the difficulty, there exists a gulf between how we think we wish we’d act, and how we actually wish we’d act.

Philosophers disagree wildly about what we want—as do psychologists, and as do politicians—and about what we ought to want. They disagree even about what it means to “ought” to want something. The history of moral theory, and the history of human efforts at coordination, is piled high with the corpses of failed Guiding Principles to True Ultimate No-Really-This-Time-I-Mean-It Normativity.

If you’re trying to come up with a reliable and pragmatically useful specification of your goals—not just for winning philosophy debates, but (say) for designing safe autonomous adaptive AI, or for building functional institutions and organizations, or for making it easier to decide which charity to donate to, or for figuring out what virtues you should be cultivating—humanity’s track record with value theory does not bode well for you.

Mere Goodness collects three sequences of blog posts on human value: “Fake Preferences” (on failed attempts at theories of value), “Value Theory” (on obstacles to developing a new theory, and some intuitively desirable fea-
tures of such a theory), and “Quantified Humanism” (on the tricky questi

...
Minds: An Introduction
by Rob Bensinger — 2015-03-11T19:00:32.440Z

You’re a mind, and that puts you in a pretty strange predicament.

Very few things get to be minds. You’re that odd bit of stuff in the universe that can form
predictions and make plans, weigh and revise beliefs, suffer, dream, notice ladybugs, or
feel a sudden craving for mango. You can even form, inside your mind, a picture of
your whole mind. You can reason about your own reasoning process, and work to bring its
operations more in line
with your goals.

You’re a mind, implemented on a human brain. And it turns out that a
human brain, for all its marvelous flexibility, is a lawful thing, a thing of pattern and
routine. Your mind can follow a routine for a lifetime, without ever once noticing that it
is doing so. And these routines can have great consequences.
When a mental pattern serves you well, we call that “rationality.”

You exist as you are, hard-wired to exhibit certain species of rationality and certain
species of irrationality, because of your ancestry. You, and all life on Earth, are
descended from ancient self-replicating molecules. This replication process was initially
clumsy and haphazard, and soon yielded replicable differences between the
replicators. “Evolution” is our name for the change in these differences over time.

Since some of these reproducible differences impact reproducibility—a phenomenon called
“selection”—evolution has resulted in organisms suited to reproduction in environments like
the ones their ancestors had. Everything about you is built on the echoes of your ancestors’
struggles and victories.

And so here you are: a min

...
Biases: An Introduction
by Rob Bensinger — 2015-03-11T19:00:31.605Z

Imagine reaching into an urn that contains seventy white balls and thirty red ones, and plucking out ten mystery balls.

Perhaps three of the ten balls will be red, and you’ll correctly guess how many red balls total were in the urn. Or perhaps you’ll happen to grab four red balls, or some other number. Then you’ll probably get the total number wrong.

This random error is the cost of incomplete knowledge, and as errors go, it’s not so bad. Your estimates won’t be incorrect on average, and the more you learn, the smaller your error will tend to be.

On the other hand, suppose that the white balls are heavier, and sink to the bottom of the urn. Then your sample may be unrepresentative in a consistent direction.

That kind of error is called “statistical bias.” When your method of learning about the world is biased, learning more may not help. Acquiring more data can even consistently worsen a biased prediction.

If you’re used to holding knowledge and inquiry in high esteem, this is a scary prospect. If we want to be sure that learning more will help us, rather than making us worse off than we were before, we need to discover and correct for biases in our data.

The idea of cognitive bias in psychology works in an analogous way. A cognitive bias is a systematic error in how we think, as opposed to a random error or one that’s merely caused by our ignorance. Whereas statistical bias skews a sample so that it less closely resembles a larger population, cognitive biases skew our thinking so that it less accurately tracks the truth (or less reliably serves our other goals).

Maybe you have an optimism bias, and you find out that the red balls can be used to treat a rare tropical disease besetting your brother, and you end up overestimating how many red balls the urn contains because you wish the balls were mostly red.

Like statistical biases, cognitive biases can distort our view of reality, they can’t always be fixed by just gathering more data, and their effects can add up over time. But when the miscalibrated measuring instrument you’re trying to fix is you, debiasing is a unique challenge.

Still, this is an obvious place to start. For if you can’t trust your brain, how can you trust anything else?

Noticing Bias

Imagine meeting someone for the first time, and knowing nothing about them except that they’re shy.

Question: Is it more likely that this person is a librarian...
Rationality: An Introduction
by Rob Bensinger — 2015-03-11T19:00:31.162Z

In the autumn of 1951, a football game between Dartmouth and Princeton turned unusually rough. A pair of psychologists, Dartmouth’s Albert Hastorf and Princeton’s Hadley Cantril, decided to ask students from both schools which team had initiated the rough play. Nearly everyone agreed that Princeton hadn’t started it; but 86% of Princeton students believed that Dartmouth had started it, whereas only 36% of Dartmouth students blamed Dartmouth. (Most Dartmouth students believed “both started it.”)

When shown a film of the game later and asked to count the infractions they saw, Dartmouth students claimed to see a mean of 4.3 infractions by the Dartmouth team (and identified half as “mild”), whereas Princeton students claimed to see a mean of 9.8 Dartmouth infractions (and identified a third as “mild”).1

When something we value is threatened—our world-view, our in-group, our social standing, or something else we care about—our thoughts and perceptions rally to their defense.2,3 Some psychologists go so far as to hypothesize that the human ability to come up with explicit justifications for our conclusions evolved specifically to help us win arguments.4

One of the basic insights of 20th-century psychology is that human behavior is often driven by sophisticated unconscious processes, and the stories we tell ourselves about our motives and reasons are much more biased and confabulated than we realize. We often fail, in fact, to realize that we’re doing any story-telling. When we seem to “directly perceive” things about ourselves in introspection, it often turns out to rest on tenuous implicit causal models.5,6 When we try to argue for our beliefs, we can come up with shaky reasoning bearing no relation to how we first arrived at the belief.7 Rather than trusting explanations in proportion to their predictive power, we tend to trust stories in proportion to their psychological appeal.

How can we do better? How can we arrive at a realistic view of the world, when we’re so prone to rationalization? How can we come to a realistic view of our mental lives, when our thoughts about thinking are also suspect?

What’s the least shaky place we could put our weight down?

The Mathematics of Rationality

At the turn of the 20th century, coming up with simple (e.g., set-theoretic) axioms for arithmetic gave mathematicians a clearer standard by which to judge the correctness of their...
Beginnings: An Introduction
by Rob Bensinger — 2015-03-11T19:00:25.616Z

This, the final book of Rationality: From AI to Zombies, is less a conclusion than a
call to action. In keeping with Becoming Stronger’s function as a jumping-off point
for further investigation, I’ll conclude by citing resources the reader can use to move
beyond these sequences and seek out a fuller understanding of Bayesianism.

This text’s definition of normative rationality in terms of Bayesian probability theory and
decision theory is standard in cognitive science. For an introduction to the heuristics and
biases approach, see Baron’s Thinking and Deciding.[1] For a general introduction
to the field, see the Oxford Handbook of Thinking and Reasoning.[2]

The arguments made in these pages about the philosophy of rationality are more
controversial. Yudkowsky argues, for example, that a rational agent should one-box in
Newcomb’s Problem—a minority position among working decision theorists.[3] (See Holt
for a nontechnical description of Newcomb’s Problem.[4]) Gary Drescher’s Good and
Real independently comes to many of the same conclusions as Yudkowsky on philosophy
of science and decision theory.[5] As such, it serves as an excellent book-length treatment
of the core philosophical content of Rationality: From AI to Zombies.

Talbott
distinguishes several views in Bayesian epistemology, including E. T. Jaynes’s position that
not all possible priors are equally reasonable.[6,7] Like Jaynes, Yudkowsky is interested in
supplementing the Bayesian optimality criterion for belief revision with an optimality
criterion for priors. This aligns Yudkowsky with researchers who hope to better understand
general-purpose AI via an improved theory of ideal reasoning, such as Marcus Hutter.[8] For

...
Optimization and the Intelligence Explosion
by Eliezer Yudkowsky — 2015-03-11T19:00:16.033Z

Among the topics I haven’t delved into here is the notion of an optimization process. Roughly, this is the idea that your power as a mind is your ability to hit small targets in a large search space—this can be either the space of possible futures (planning) or the space of possible designs (invention).

Suppose you have a car, and suppose we already know that your preferences involve travel. Now suppose that you take all the parts in the car, or all the atoms, and jumble them up at random. It’s very unlikely that you’ll end up with a travel-artifact at all, even so much as a wheeled cart; let alone a travel-artifact that ranks as high in your preferences as the original car. So, relative to your preference ordering, the car is an extremely improbable artifact. The power of an optimization process is that it can produce this kind of improbability.

You can view both intelligence and natural selection as special cases of optimization: processes that hit, in a large search space, very small targets defined by implicit preferences. Natural selection prefers more efficient replicators. Human intelligences have more complex preferences. Neither evolution nor humans have consistent utility functions, so viewing them as “optimization processes” is understood to be an approximation. You’re trying to get at the sort of work being done, not claim that humans or evolution do this work perfectly.

This is how I see the story of life and intelligence—as a story of improbably good designs being produced by optimization processes. The “improbability” here is improbability relative to a random selection from the design space,
not improbability in an absolute sense—if you have an optimization process around, then “improbably” good designs become probable.

Looking over the history of optimization on Earth up until now, the first step is to conceptually separate the meta level from the object level—separate the structure of optimization from that which is optimized.

If you consider biology in the absence of hominids, then on the object level we have things like dinosaurs and butterflies and cats. On the meta level we have things like sexual recombination and natural selection of asexual populations. The object level, you will observe, is rather more complicated than the meta level. Natural selection is not an easy subject and it involves math. But if you look at the anatomy of a

...
The World: An Introduction
by Rob Bensinger — 2015-03-11T19:00:12.370Z

Previous essays have discussed human reasoning, language, goals, and social dynamics. Mathematics, physics, and biology were cited to explain patterns in human behavior, but little has been said about humanity’s place in nature, or about the natural world in its own right.

Just as it was useful to contrast humans as goal-oriented systems with inhuman processes in evolutionary biology and artificial intelligence, it will be useful in the coming sequences of essays to contrast humans as physical systems with inhuman processes that aren’t mind-like.

We humans are, after all, built out of inhuman parts. The world of atoms looks nothing like the world as we ordinarily think of it, and certainly looks nothing like the world’s conscious denizens as we ordinarily think of them. As Giulio Giorello put the point in an interview with Daniel Dennett: “Yes, we have a soul. But it’s made of lots of tiny robots.”[1]

Mere Reality collects seven sequences of essays on this topic. The first three introduce the question of how the human world relates to the world revealed by physics: “Lawful Truth” (on the basic links between physics and human cognition), “Reductionism 101” (on the project of scientifically explaining phenomena), and “Joy in the Merely Real” (on the emotional, personal significance of the scientific world-view). This is followed by two sequences that go into more depth on specific academic debates: “Physicalism 201” (on the hard problem of consciousness) and “Quantum Physics and Many Worlds” (on the measurement problem in physics). Finally, the sequence “Science and Rationality” and the essay A Technical Explanation of Technical Explanation tie these ideas together and relate them to scientific practice.

The discussions of consciousness and quantum physics illustrate the relevance of reductionism to present-day controversies in science and philosophy. For those interested in a bit of extra context, I’ll say a few more words about those two topics here. For those eager to skip ahead: skip ahead!

Minds in the World

Can we ever know what it’s like to be a bat?

We can certainly develop better cognitive models for predicting bat behav-ior, or more fine-grained models of bat neurology—but it isn’t obvious that this would ...
Preface
by Eliezer Yudkowsky — 2015-03-11T19:00:00.164Z

You hold in your hands a compilation of two years of daily blog posts. In retrospect, I look back on that project and see a large number of things I did completely wrong. I’m fine with that. Looking back and not seeing a huge number of things I did wrong would mean that neither my writing nor my understanding had improved since 2009. Oops is the sound we make when we improve our beliefs and strategies; so to look back at a time and not see anything you did wrong means that you haven’t learned anything or changed your mind since then.

It was a mistake that I didn’t write my two years of blog posts with the intention of helping people do better in their everyday lives. I wrote it with the intention of helping people solve big, difficult, important problems, and I chose impressive-sounding, abstract problems as my examples.

In retrospect, this was the second-largest mistake in my approach. It ties in to the first-largest mistake in my writing, which was that I didn’t realize that the big problem in learning this valuable way of thinking was figuring out how to practice it, not knowing the theory. I didn’t realize that part was the priority; and regarding this I can only say “Oops” and “Duh.”

Yes, sometimes those big issues really are big and really are important; but that doesn’t change the basic truth that to master skills you need to practice them and it’s harder to practice on things that are further away. (Today the Center for Applied Rationality is working on repairing this huge mistake of mine in a more systematic fashion.)

A third huge mistake I made was to focus too much on rational belief, too little on rational action.

The fourth-largest mistake I made was that I should have better organized the content I was presenting in the sequences. In particular, I should have created a wiki much earlier, and made it easier to read the posts in sequence.

That mistake at least is correctable. In the present work Rob Bensinger has reordered the posts and reorganized them as much as he can without trying to rewrite all the actual material (though he’s rewritten a bit of it).

My fifth huge mistake was that I—as I saw it—tried to speak plainly about the stupidity of what appeared to me to be stupid ideas. I did try to avoid the fallacy known as Bulverism, which is where you open your discussion by talking about how stupid people are for believing something; I would always d...
A counterfactual and hypothetical note on AI safety design
by Stuart_Armstrong — 2015-03-11T16:20:12.989Z

A putative new idea for AI control; index here.

A lot of the new ideas I've been posting could be parodied as going something like this:

The AI A, which is utility indifferent to the existence of AI B, has utility u (later corriged to v', twice), and it will create a subagent C which believe via false thermodynamic miracles that D does not exist, while D' will hypothetically and counterfactually use two different definitions of counterfactual so that the information content of its own utility cannot be traded with a resource gathering agent E that doesn't exist (assumed separate from its unknown utility function)...

What is happening is that I'm attempting to define algorithms that accomplish a particular goal (such as obeying the spirit of a restriction, or creating a satisficer). Typically this algorithm has various underdefined components - such as inserting an intelligent agent at a particular point, controlling the motivation of an agent at a point, effectively defining a physical event, or having an agent believe (or act as if they believed) something that was incorrect.

The aim is to reduce the problem from stuff like "define human happiness" to stuff like "define counterfactuals" or "pinpoint an AI's motivation". These problems should be fundamentally easier - if not for general agents, then for some of the ones we can define ourselves (this may also allow us to prioritise research directions).

And I also have no doubt that once a design is available, that it will be improved upon and transformed and made easier to implement and generalise. Therefore I'm currently more interested in objections of the form "it won't work" than "it can't be done".