trentmkelly/LessWrong-43k huggingface dataset:
Qualitative Strategies of Friendliness
by Eliezer Yudkowsky — 2008-08-30T02:12:33.000Z

Followup to: Magical Categories

What on Earth could someone possibly be thinking, when they propose creating a superintelligence whose behaviors are reinforced by human smiles?
Tiny molecular photographs of human smiles - or if you rule that out,
then faces ripped off and permanently wired into smiles - or if you
rule that out, then brains stimulated into permanent maximum happiness,
in whichever way results in the widest smiles...

Well, you never do know what other people are thinking, but in this
case I'm willing to make a guess. It has to do with a field of
cognitive psychology called Qualitative Reasoning.

Qualitative reasoning is what you use to decide that increasing the
temperature of your burner increases the rate at which your water
boils, which decreases the derivative of the amount of water present.
One would also add the sign of d(water) - negative, meaning that the
amount of water is decreasing - and perhaps the fact that there is only
a bounded amount of water. Or we could say that turning up the burner
increases the rate at which the water temperature increases, until the
water temperature goes over a fixed threshold, at which point the water
starts boiling, and hence decreasing in quantity... etc.

That's qualitative reasoning, a small subfield of cognitive science
and Artificial Intelligence - reasoning that doesn't describe or
predict exact quantities, but rather the signs of quantities, their
derivatives, the existence of thresholds.

As usual, human common sense means we can see things by qualitative
reasoning that current programs can't - but the more interesting
realization is how vital human qualitative reasoning is to our vaunted human common sense. It's one of the basic ways in which we comprehend the world.

Without timers you can't figure out how long water takes to boil,
your mind isn't that precise. But you can figure out that you should
turn the burner up, rather than down, and then watch to make sure the water doesn't all boil away. Which is what you mainly need, in the real world. Or at least we humans seem to get by on qualitative reasoning; we may not realize what we're missing...

So I suspect that what went through the one's mind, proposing the AI
whose behaviors would be reinforced by human smiles, was something like
this:

The happier people are, the more they smile. Smiles reinforc...
Harder Choices Matter Less
by Eliezer Yudkowsky — 2008-08-29T02:02:04.000Z

...or they should, logically speaking.

Suppose you're torn in an agonizing conflict between two choices.

Well... if you can't decide between them, they must be around equally appealing, right? Equally balanced pros and cons? So the choice must matter very little - you may as well flip a coin. The alternative is that the pros and cons aren't equally balanced, in which case the decision should be simple.

This is a bit of a tongue-in-cheek suggestion, obviously - more
appropriate for choosing from a restaurant menu than choosing a major in
college.

But consider the case of choosing from a restaurant menu. The obvious choices, like Pepsi over Coke, will take very little time. Conversely, the choices that take the most time probably make the least difference. If you can't decide between the hamburger and the hot dog, you're either close to indifferent between them, or in your current state of ignorance you're close to indifferent between their expected utilities.

Does this have any moral for larger dilemmas, like choosing a major in college? Here, it's more likely that you're in a state of ignorance, than that you would have no real preference over outcomes. Then if you're agonizing,
the obvious choice is "gather more information" - get a couple of
part-time jobs that let you see the environment you would be working
in. And, logically, you can defer the agonizing until after that.

Or maybe you've already gathered information, but can't seem to integrate to a decision? Then you should be
listing out pros and cons on a sheet of paper, or writing down
conflicting considerations and trying to decide which consideration is,
in general, the most important to you. Then that's the obvious thing you should do, which clearly dominates the alternative of making a snap decision in either direction.

Of course there are also biases that get stronger as we think longer - it gives us more opportunity to rationalize, for example; or it gives us more opportunity to think up extreme but rare/unlikely considerations whose affect dominates the decision process. Like someone choosing a longer commute to work (every day), so that they can have a house with an extra room for when Grandma comes over (once a year). If you think your most likely failure mode is that you'll outsmart yourself, then the obvious ...
Against Modal Logics
by Eliezer Yudkowsky — 2008-08-27T22:13:46.000Z

Continuation of: Grasping Slippery Things
Followup to: Possibility and Could-ness, Three Fallacies of Teleology

When I try to hit a reduction problem, what usually happens is that I "bounce" - that's what I call it. There's an almost tangible feel to the failure, once you abstract and generalize and recognize it. Looking back, it seems that I managed to say most of what I had in mind for today's post, in "Grasping Slippery Things". The "bounce" is when you try to analyze a word like could, or a notion like possibility, and end up saying, "The set of realizable worlds [A'] that follows from an initial starting world A operated on by a set of physical laws f." Where realizable contains the full mystery of "possible" - but you've made it into a basic symbol, and added some other symbols: the illusion of formality.

There are a number of reasons why I feel that modern philosophy, even analytic philosophy, has gone astray - so far astray that I simply can't make use of their years and years of dedicated work, even when they would seem to be asking questions closely akin to mine.

The proliferation of modal logics in philosophy is a good illustration of one major reason: Modern philosophy doesn't enforce reductionism, or even strive for it.

Most philosophers, as one would expect from Sturgeon's Law, are not very good. Which means that they're not even close to the level of competence it takes to analyze mentalistic black boxes into cognitive algorithms. Reductionism is, in modern times, an unusual talent. Insights on the order of Pearl et. al.'s reduction of causality or Julian Barbour's reduction of time are rare.

So what these philosophers do instead, is "bounce" off the problem into a new modal logic: A logic with symbols that embody the mysterious, opaque, unopened black box. A logic with primitives like "possible" or "necessary", to mark the places where the philosopher's brain makes an internal function call to cognitive algorithms as yet unknown.

And then they publish it and say, "Look at how precisely I have defined my language!"

In the Wittgensteinian era, philosophy has been about language - about trying to give precise meaning to terms.

The kind of work that I try to do is not about language. It is
...
Dreams of AI Design
by Eliezer Yudkowsky — 2008-08-27T04:04:20.818Z

After spending a decade or two living inside a mind, you might think you knew a bit about how minds work, right? That’s what quite a few AGI wannabes (people who think they’ve got what it takes to program an Artificial General Intelligence) seem to have concluded. This, unfortunately, is wrong.

Artificial Intelligence is fundamentally about reducing the mental to the non-mental.

You might want to contemplate that sentence for a while. It’s important.

Living inside a human mind doesn’t teach you the art of reductionism, because nearly all of the work is carried out beneath your sight, by the opaque black boxes of the brain. So far beneath your sight that there is no introspective sense that the black box is there—no internal sensory event marking that the work has been delegated.

Did Aristotle realize that when he talked about the telos, the final cause of events, that he was delegating predictive labor to his brain’s complicated planning mechanisms—asking, “What would this object do, if it could make plans?” I rather doubt it. Aristotle thought the brain was an organ for cooling the blood—which he did think was important: humans, thanks to their larger brains, were more calm and contemplative.

So there’s an AI design for you! We just need to cool down the computer a lot, so it will be more calm and contemplative, and won’t rush headlong into doing stupid things like modern computers. That’s an example of fake reductionism. “Humans are more contemplative because their blood is cooler,” I mean. It doesn’t resolve the black box of the word contemplative. You can’t predict what a contemplative thing does using a complicated model with internal moving parts composed of merely material, merely causal elements—positive and negative voltages on a transistor being the canonical example of a merely material and causal element of a model. All you can do is imagine yourself being contemplative, to get an idea of what a contemplative agent does.

Which is to say that you can only reason about “contemplative-ness” by empathic inference—using your own brain as a black box with the contemplativeness lever pulled, to predict the output of another black box.

You can imagine another agent being contemplative, but again that’s an act of empathic inference—the way this imaginative act works is by adjusting your own brain to run in contemplativeness-mode, not by modeling the other...
Three Fallacies of Teleology
by Eliezer Yudkowsky — 2008-08-25T22:27:55.000Z

Followup to: Anthropomorphic Optimism

Aristotle distinguished between four senses of the Greek word aition, which in English is translated as "cause", though Wikipedia suggests that a better translation is "maker". Aristotle's theory of the Four Causes, then, might be better translated as the Four Makers. These were his four senses of aitia: The material aition, the formal aition, the efficient aition, and the final aition.

The material aition of a bronze statue is the substance it is made from, bronze. The formal aition is the substance's form, its statue-shaped-ness. The efficient aition best translates as the English word "cause"; we would think of the artisan carving the statue, though Aristotle referred to the art of bronze-casting the statue, and regarded the individual artisan as a mere instantiation.

The final aition was the goal, or telos, or purpose of the statue, that for the sake of which the statue exists.

Though Aristotle considered knowledge of all four aitia as necessary, he regarded knowledge of the telos as the knowledge of highest order. In this, Aristotle followed in the path of Plato, who had earlier written:

Imagine not being able to distinguish the real cause from that without which the cause would not be able to act as a cause. It is what the majority appear to do, like people groping in the dark; they call it a cause, thus giving it a name that does not belong to it. That is why one man surrounds the earth with a vortex to make the heavens keep it in place, another makes the air support it like a wide lid. As for their capacity of being in the best place they could possibly be put, this they do not look for, nor do they believe it to have any divine force...

Suppose that you translate "final aition" as "final cause", and assert directly:

"Why do human teeth develop with such regularity, into a structure well-formed for biting and chewing? You could try to explain this as an incidental fact, but think of how unlikely that would be. Clearly, the final cause of teeth is the act of biting and chewing. Teeth develop with regularity, because of the act of biting and chewing - the latter causes the former."

A modern-day sophisticated Bayesian will at once remark, "This requires me to draw a circular causal diagram with an arrow going from the future to the past."

It's ...
Magical Categories
by Eliezer Yudkowsky — 2008-08-24T19:51:39.000Z

'We can design intelligent machines so their primary, innate emotion is unconditional love for all humans. First we can build relatively simple machines that learn to recognize happiness and unhappiness in human facial expressions, human voices and human body language. Then we can hard-wire the result of this learning as the innate emotional values of more complex intelligent machines, positively reinforced when we are happy and negatively reinforced when we are unhappy.'
-- Bill Hibbard (2001), Super-intelligent machines.

That was published in a peer-reviewed journal, and the author later wrote a whole book about it, so this is not a strawman position I'm discussing here.

So... um... what could possibly go wrong...

When I mentioned (sec. 6) that Hibbard's AI ends up tiling the galaxy with tiny molecular smiley-faces, Hibbard wrote an indignant reply saying:

'When it is feasible to build a super-intelligence, it will be feasible to build hard-wired recognition of "human facial expressions, human voices and human body language" (to use the words of mine that you quote) that exceed the recognition accuracy of current humans such as you and me, and will certainly not be fooled by "tiny molecular pictures of smiley-faces." You should not assume such a poor implementation of my idea that it cannot make discriminations that are trivial to current humans.'

As Hibbard also wrote "Such obvious contradictory assumptions show Yudkowsky's preference for drama over reason," I'll go ahead and mention that Hibbard illustrates a key point: There is no professional certification test you have to take before you are allowed to talk about AI morality. But that is not my primary topic today. Though it is a crucial point about the state of the gameboard, that most AGI/FAI wannabes are so utterly unsuited to the task, that I know no one cynical enough to imagine the horror without seeing it firsthand. Even Michael Vassar was probably surprised his first time through.

No, today I am here to dissect "You should not assume such a poor implementation of my idea that it cannot make discriminations that are trivial to current humans."

Once upon a time - I've seen this story in several versions and several places, sometimes cited as fact, but I've never tracked down an original source - once upon a time, I say, the US ...
Unnatural Categories
by Eliezer Yudkowsky — 2008-08-24T01:00:00.000Z

Followup to: Disguised Queries, Superexponential Conceptspace

If a tree falls in the forest, and no one hears it, does it make a sound?

"Tell me why you want to know," says the rationalist, "and I'll tell you the answer." If you want to know whether your seismograph, located nearby, will register an acoustic wave, then the experimental prediction is "Yes"; so, for seismographic purposes, the tree should be considered to make a sound. If instead you're asking some question about firing patterns in a human auditory cortex - for whatever reason - then the answer is that no such patterns will be changed when the tree falls.

What is a poison? Hemlock is a "poison"; so is cyanide; so is viper venom. Carrots, water, and oxygen are "not poison". But what determines this classification? You would be hard pressed, just by looking at hemlock and cyanide and carrots and water, to tell what sort of difference is at work. You would have to administer the substances to a human - preferably one signed up for cryonics - and see which ones proved fatal. (And at that, the definition is still subtler than it appears: a ton of carrots, dropped on someone's head, will also prove fatal. You're really asking about fatality from metabolic disruption, after administering doses small enough to avoid mechanical damage and blockage, at room temperature, at low velocity.)

Where poison-ness is concerned, you are not classifying via a strictly local property of the substance. You are asking about the consequence when a dose of that substance is applied to a human metabolism. The local difference between a human who gasps and keels over, versus a human alive and healthy, is more compactly discriminated, than any local difference between poison and non-poison.

So we have a substance X, that might or might not be fatally poisonous, and a human Y, and we say - to first order:

"X is classified 'fatally poisonous' iff administering X to Y causes Y to enter a state classified 'dead'."

Much of the way that we classify things - never mind events - is non-local, entwined with the consequential structure of the world. All the things we would call a chair are all the things that were made for us to sit on. (Humans might even call two molecularly identical objects a "...
Mirrors and Paintings
by Eliezer Yudkowsky — 2008-08-23T00:29:05.000Z

Followup to: Sorting Pebbles Into Correct Heaps, Invisible Frameworks

Background:
There's a proposal for Friendly AI called "Coherent Extrapolated
Volition" which I don't really want to divert the discussion to, right
now. Among many other things, CEV involves pointing an AI at humans and
saying (in effect) "See that? That's where you find the base content for self-renormalizing morality."

Hal Finney commented on the Pebblesorter parable:

I wonder what the Pebblesorter AI would do if successfully programmed to implement [CEV]... Would the AI pebblesort? Or would
it figure that if the Pebblesorters got smarter, they would see that
pebblesorting was pointless and arbitrary? Would the AI therefore adopt
our own parochial morality, forbidding murder, theft and sexual
intercourse among too-young people? Would that be the CEV of
Pebblesorters?

I imagine we would all like to
think so, but it smacks of parochialism, of objective morality. I can't
help thinking that Pebblesorter CEV would have to include some aspect
of sorting pebbles. Doesn't that suggest that CEV can malfunction
pretty badly?

I'm
giving this question its own post, for that it touches on similar
questions I once pondered - dilemmas that forced my current metaethics
as the resolution.

Yes indeed: A CEV-type AI, taking Pebblesorters as its focus, would
wipe out the Pebblesorters and sort the universe into prime-numbered
heaps.

This is not the right thing to do.

That is not a bug.

A primary motivation for CEV was to answer the question, "What can
Archimedes do if he has to program a Friendly AI, despite being a savage barbarian by the Future's standards, so that the Future comes out right anyway? Then whatever general strategy Archimedes could plausibly follow, that is what we should do ourselves: For
we too may be ignorant fools, as the Future measures such things."

It is tempting to further extend the question, to ask, "What can the
Pebblesorters do, despite wanting only to sort pebbles, so that the
universe comes out right anyway? What sort of general strategy should
they follow, so that despite wanting something that is utterly
pointless and futile, their Future ends up containing sentient beings
leading worthwhile lives and having fun? Then whatever general
strategy we wish the Pebblesor...
Invisible Frameworks
by Eliezer Yudkowsky — 2008-08-22T03:36:37.000Z

Followup to: Passing the Recursive Buck, No License To Be Human

Roko has mentioned his "Universal Instrumental Values" several times in his comments. Roughly, Roko proposes that we ought to adopt as terminal values those things that a supermajority of agents would do instrumentally. On Roko's blog he writes:

I'm suggesting that UIV provides the cornerstone for a rather new approach to goal system design. Instead of having a fixed utility function/supergoal, you periodically promote certain instrumental values to terminal values i.e. you promote the UIVs.

Roko thinks his morality is more objective than mine:

It also worries me quite a lot that eliezer's post is entirely symmetric under the action of replacing his chosen notions with the pebble-sorter's notions. This property qualifies as "moral relativism" in my book, though there is no point in arguing about the meanings of words.

My posts on universal instrumental values are not symmetric under replacing UIVs with some other set of goals that an agent might have. UIVs are the unique set of values X such that in order to achieve any other value Y, you first have to do X.

Well, and this proposal has a number of problems, as some of the commenters on Roko's blog point out.

For a start, Roko actually says "universal", not "supermajority", but there are no actual universal actions; no matter what the green button does, there are possible mind designs whose utility function just says "Don't press the green button." There is no button, in other words, that all possible minds will press. Still, if you defined some prior weighting over the space of possible minds, you could probably find buttons that a supermajority would press, like the "Give me free energy" button.

But to do nothing except press such buttons, consists of constantly losing your purposes. You find that driving the car is useful for getting and eating chocolate, or for attending dinner parties, or even for buying and manufacturing more cars. In fact, you realize that every intelligent agent will find it useful to travel places. So you start driving the car around without any destination. Roko hasn't noticed this because, by anthropomorphic optimism, he mysteriously only thinks of humanly appealing "UIVs" to propose, like "creativity".

Let me guess, Roko, you don't think that "drive a car!" is a "valid" U...
No License To Be Human
by Eliezer Yudkowsky — 2008-08-20T23:18:25.000Z

Followup to: You Provably Can't Trust Yourself

Yesterday I discussed the difference between:

A system that believes—is moved by—any specific chain of deductions from the axioms of Peano Arithmetic. (PA, Type 1 calculator)

A system that believes PA, plus explicitly asserts the general proposition that PA is sound. (PA+1, meta-1-calculator that calculates the output of Type 1 calculator)

A system that believes PA, plus explicitly asserts its own soundness. (Self-PA, Type 2 calculator)

These systems are formally distinct. PA+1 can prove things that PA cannot. Self-PA is inconsistent, and can prove anything via Löb's Theorem.

With these distinctions in mind, I hope my intent will be clearer, when I say that although I am human and have a human-ish moral framework, I do not think that the fact of acting in a human-ish way licenses anything.

I am a self-renormalizing moral system, but I do not think there is any general license to be a self-renormalizing moral system.

And while we're on the subject, I am an epistemologically incoherent creature, trying to modify his ways of thinking in accordance with his current conclusions; but I do not think that reflective coherence implies correctness.

Let me take these issues in reverse order, starting with the general unlicensure of epistemological reflective coherence.

If five different people go out and investigate a city, and draw five different street maps, we should expect the maps to be (mostly roughly) consistent with each other. Accurate maps are necessarily consistent among each other and among themselves, there being only one reality. But if I sit in my living room with my blinds closed, I can draw up one street map from my imagination and then make four copies: these five maps will be consistent among themselves, but not accurate. Accuracy implies consistency but not the other way around.

In Where Recursive Justification Hits Bottom, I talked about whether "I believe that induction will work on the next occasion, because it's usually worked before" is legitimate reasoning, or "I trust Occam's Razor because the simplest explanation for why Occam's Razor often works is that we live in a highly ordered universe". Though we actually formalized the idea of scientific induction, starting from an inductive instinct; we modified our intuitive underst...