trentmkelly/LessWrong-43k huggingface dataset:
Intelligence in Economics
by Eliezer Yudkowsky — 2008-10-30T21:17:56.000Z

Followup to: Economic Definition of Intelligence?

After I challenged Robin to show how economic concepts can be useful in defining or measuring intelligence, Robin responded by - as I interpret it - challenging me to show why a generalized concept of "intelligence" is any use in economics.

Well, I'm not an economist (as you may have noticed) but I'll try to respond as best I can.

My primary view of the world tends to be through the lens of AI. If I talk about economics, I'm going to try to subsume it into notions like expected utility maximization (I manufacture lots of copies of something that I can use to achieve my goals) or information theory (if you manufacture lots of copies of something, my probability of seeing a copy goes up). This subsumption isn't meant to be some kind of challenge for academic supremacy - it's just what happens if you ask an AI guy an econ question.

So first, let me describe what I see when I look at economics:

I see a special case of game theory in which some interactions are highly regular and repeatable: You can take 3 units of steel and 1 unit of labor and make 1 truck that will transport 5 units of grain between Chicago and Manchester once per week, and agents can potentially do this over and over again. If the numbers aren't constant, they're at least regular - there's diminishing marginal utility, or supply/demand curves, rather than rolling random dice every time. Imagine economics if no two elements of reality were fungible - you'd just have a huge incompressible problem in non-zero-sum game theory.

This may be, for example, why we don't think of scientists writing papers that build on the work of other scientists in terms of an economy of science papers - if you turn an economist loose on science, they may measure scientist salaries paid in fungible dollars, or try to see whether scientists trade countable citations with each other. But it's much less likely to occur to them to analyze the way that units of scientific knowledge are produced from previous units plus scientific labor. Where information is concerned, two identical copies of a file are the same information as one file. So every unit of knowledge is unique, non-fungible, and so is each act of production. There isn't even a common currency that measures how much a given paper contributes to human...
Economic Definition of Intelligence?
by Eliezer Yudkowsky — 2008-10-29T19:32:53.000Z

Followup to: Efficient Cross-Domain Optimization

Shane Legg once produced a catalogue of 71 definitions of intelligence. Looking it over, you'll find that the 18 definitions in dictionaries and the 35 definitions of psychologists are mere black boxes containing human parts.

However, among the 18 definitions from AI researchers, you can find such notions as

"Intelligence measures an agent's ability to achieve goals in a wide range of environments" (Legg and Hutter)

or

"Intelligence is the ability to optimally use limited resources - including time - to achieve goals" (Kurzweil)

or even

"Intelligence is the power to rapidly find an adequate solution in what appears a priori (to observers) to be an immense search space" (Lenat and Feigenbaum)

which is about as close as you can get to my own notion of "efficient cross-domain optimization" without actually measuring optimization power in bits.

But Robin Hanson, whose AI background we're going to ignore for a moment in favor of his better-known identity as an economist, at once said:

"I think what you want is to think in terms of a production function, which describes a system's output on a particular task as a function of its various inputs and features."

Economists spend a fair amount of their time measuring things like productivity and efficiency. Might they have something to say about how to measure intelligence in generalized cognitive systems?

This is a real question, open to all economists. So I'm going to quickly go over some of the criteria-of-a-good-definition that stand behind my own proffered suggestion on intelligence, and what I see as the important challenges to a productivity-based view. It seems to me that this is an important sub-issue of Robin's and my persistent disagreement about the Singularity.

(A) One of the criteria involved in a definition of intelligence is that it ought to separate form and function. The Turing Test fails this - it says that if you can build something indistinguishable from a bird, it must definitely fly, which is true but spectacularly unuseful in building an airplane.

(B) We will also prefer quantitative measures to qualitative measures that only say "this is intelligent or not intelligent". Sure, you can define "flight" in terms of getting off the ground, but...
Efficient Cross-Domain Optimization
by Eliezer Yudkowsky — 2008-10-28T16:33:03.000Z

Previously in series: Measuring Optimization Power

Is Deep Blue "intelligent"? It was powerful enough at optimizing chess boards to defeat Kasparov, perhaps the most skilled chess player humanity has ever fielded.

A bee builds hives, and a beaver builds dams; but a bee doesn't build dams and a beaver doesn't build hives. A human, watching, thinks, "Oh, I see how to do it" and goes on to build a dam using a honeycomb structure for extra strength.

Deep Blue, like the bee and the beaver, never ventured outside the narrow domain that it itself was optimized over.

There are no-free-lunch theorems showing that you can't have a truly general intelligence that optimizes in all possible universes (the vast majority of which are maximum-entropy heat baths). And even practically speaking, human beings are better at throwing spears than, say, writing computer programs.

But humans are much more cross-domain than bees, beavers, or Deep Blue. We might even conceivably be able to comprehend the halting behavior of every Turing machine up to 10 states, though I doubt it goes much higher than that.

Every mind operates in some domain, but the domain that humans
operate in isn't "the savanna" but something more like "not too
complicated processes in low-entropy lawful universes". We learn whole new domains by observation, in the same way that a beaver might learn to chew a different kind of wood. If I could
write out your prior, I could describe more exactly the universes in which you operate.

Is evolution intelligent? It operates across domains - not quite as
well as humans do, but with the same general ability to do
consequentialist optimization on causal sequences that wend through
widely different domains. It built the bee. It built the beaver.

Whatever begins with genes, and impacts inclusive genetic fitness, through any chain of cause and effect
in any domain, is subject to
evolutionary optimization. That much is true.

But evolution only achieves this by running millions of actual
experiments in which the causal chains are actually played out. This
is incredibly inefficient. Cynthia Kenyon said, "One grad student can
do things in an hour that evolution could not do in a billion years." This is not because the grad student does quadrillions of detail...
Measuring Optimization Power
by Eliezer Yudkowsky — 2008-10-27T21:44:57.000Z

Previously in series: Aiming at the Target

Yesterday I spoke of how "When I think you're a powerful intelligence, and I think I know
something about your preferences, then I'll predict that you'll steer
reality into regions that are higher in your preference ordering."

You can quantify this, at least in theory, supposing you have (A) the agent or optimization process's preference ordering, and (B) a measure of the space of outcomes - which, for discrete outcomes in a finite space of possibilities, could just consist of counting them - then you can quantify how small a target is being hit, within how large a greater region.

Then we count the total number of states with equal or greater rank in the preference ordering to the outcome achieved, or integrate over the measure of states with equal or greater rank. Dividing this by the total size of the space gives you the relative smallness of the target - did you hit an outcome that was one in a million? One in a trillion?

Actually, most optimization processes produce "surprises" that are exponentially more improbable than this - you'd need to try far more than a trillion random reorderings of the letters in a book, to produce a play of quality equalling or exceeding Shakespeare. So we take the log base two of the reciprocal of the improbability, and that gives us optimization power in bits.

This figure - roughly, the improbability of an "equally preferred" outcome being produced by a random selection from the space (or measure on the space) - forms the foundation of my Bayesian view of intelligence, or to be precise, optimization power. It has many subtleties:

(1) The wise will recognize that we are calculating the entropy of something. We could take the figure of the relative improbability of "equally good or better" outcomes, and call this the negentropy of the system relative to a preference ordering. Unlike thermodynamic entropy, the entropy of a system relative to a preference ordering can easily decrease (that is, the negentropy can increase, that is, things can get better over time relative to a preference ordering).

Suppose e.g. that a single switch will determine whether the world is saved or destroyed, and you don't know whether the switch is set to 1 or 0. You can carry out an operation that coerces the switch to 1; ...
Aiming at the Target
by Eliezer Yudkowsky — 2008-10-26T16:47:19.000Z

Previously in series: Belief in Intelligence

Previously, I spoke of that very strange epistemic position one can occupy, wherein you don't know exactly where Kasparov will move on the chessboard, and yet your state of knowledge about the game is very different than if you faced a random move-generator with the same subjective probability distribution - in particular, you expect Kasparov to win. I have beliefs about where Kasparov wants to steer the future, and beliefs about his power to do so.

Well, and how do I describe this knowledge, exactly?

In the case of chess, there's a simple function that
classifies chess positions into wins for black, wins for white, and
drawn games. If I know which side Kasparov is playing, I know the
class of chess positions Kasparov is aiming for. (If I don't know
which side Kasparov is playing, I can't predict whether black or white
will win - which is not the same as confidently predicting a drawn game.)

More generally, I can describe motivations using a preference ordering.
When I consider two potential outcomes, X and Y, I can say that I
prefer X to Y; prefer Y to X; or find myself indifferent between them.
I would write these relations as X > Y; X < Y; and X ~ Y.

Suppose
that you have the ordering A < B ~ C < D ~ E.
Then you like B more than A, and C more than A. {B, C},
belonging to the same class, seem equally desirable to you; you are
indifferent between which of {B, C} you receive, though you would
rather have either than A, and you would rather have something from the class {D, E} than {B, C}.

When I think you're a powerful intelligence, and I think I know
something about your preferences, then I'll predict that you'll steer
reality into regions that are higher in your preference ordering.

Think of a huge circle containing all possible outcomes, such that
outcomes higher in your preference ordering appear to be closer to the
center. Outcomes between which you are indifferent are the same
distance from the center - imagine concentric rings of outcomes that
are all equally preferred. If you aim your actions and strike a
consequence close to the center - an outcome that ranks high in your
preference ordering - then I'll think better of your ability to aim.

The more intelligent I believe you are, the more probability I'll
concentrate into outcomes that I believe are ...
Belief in Intelligence
by Eliezer Yudkowsky — 2008-10-25T15:00:00.000Z

Since I am so uncertain of Kasparov's moves, what is the empirical content of my belief that "Kasparov is a highly intelligent chess player"? What real-world experience does my belief tell me to anticipate? Is it a cleverly masked form of total ignorance?

To sharpen the dilemma, suppose Kasparov plays against some mere chess grandmaster Mr. G, who's not in the running for world champion. My own ability is far too low to distinguish between these levels of chess skill. When I try to guess Kasparov's move, or Mr. G's next move, all I can do is try to guess "the best chess move" using my own meager knowledge of chess. Then I would produce exactly the same prediction for Kasparov's move or Mr. G's move in any particular chess position. So what is the empirical content of my belief that "Kasparov is a better chess player than Mr. G"?

The empirical content of my belief is the testable, falsifiable prediction that the final chess position will occupy the class of chess positions that are wins for Kasparov, rather than drawn games or wins for Mr. G. (Counting resignation as a legal move that leads to a chess position classified as a loss.) The degree to which I think Kasparov is a "better player" is reflected in the amount of probability mass I concentrate into the "Kasparov wins" class of outcomes, versus the "drawn game" and "Mr. G wins" class of outcomes. These classes are extremely vague in the sense that they refer to vast spaces of possible chess positions - but "Kasparov wins" is more specific than maximum entropy, because it can be definitely falsified by a vast set of chess positions.

The outcome of Kasparov's game is predictable because I know, and understand, Kasparov's goals. Within the confines of the chess board, I know Kasparov's motivations - I know his success criterion, his utility function, his target as an optimization process. I know where Kasparov is ultimately trying to steer the future and I anticipate he is powerful enough to get there, although I don't anticipate much about how Kasparov is going to do it.

Imagine that I'm visiting a distant city, and a local friend volunteers to drive me to the airport. I don't know the neighborhood. Each time my friend approaches a street intersection, I don...
Expected Creative Surprises
by Eliezer Yudkowsky — 2008-10-24T22:22:47.000Z

Imagine that I'm playing chess against a smarter opponent. If I could predict exactly where my opponent would move on each turn, I would automatically be at least as good a chess player as my opponent. I could just ask myself where my opponent would move, if they were in my shoes; and then make the same move myself. (In fact, to predict my opponent's exact moves, I would need to be superhuman - I would need to predict my opponent's exact mental processes, including their limitations and their errors. It would become a problem of psychology, rather than chess.)

So predicting an exact move is not possible, but neither is it true that I have no information about my opponent's moves.

Personally, I am a very weak chess player - I play an average of maybe two games per year. But even if I'm playing against former world champion Garry Kasparov, there are certain things I can predict about his next move. When the game starts, I can guess that the move P-K4 is more likely than P-KN4. I can guess that if Kasparov has a move which would allow me to checkmate him on my next move, that Kasparov will not make that move.

Much less reliably, I can guess that Kasparov will not make a move that exposes his queen to my capture - but here, I could be greatly surprised; there could be a rationale for a queen sacrifice which I have not seen.

And finally, of course, I can guess that Kasparov will win the game...

Supposing that Kasparov is playing black, I can guess that the final position of the chess board will occupy the class
of positions that are wins for black. I cannot predict specific
features of the board in detail; but I can narrow things down relative
to the class of all possible ending positions.

If I play chess against a superior opponent, and I don't know for
certain where my opponent will move, I can still endeavor to produce a
probability distribution that is well-calibrated - in the sense
that, over the course of many games, legal moves that I label with a
probability of "ten percent" are made by the opponent around 1 time in
10.

You might ask: Is producing a well-calibrated distribution over Kasparov, beyond my abilities as an inferior chess player?

But there is a trivial way to produce a well-calibrated probability
distribution - just use the maximum-entropy distribution representing a state of total i...
San Jose Meetup, Sat 10/25 @ 7:30pm
by Eliezer Yudkowsky — 2008-10-23T22:55:12.000Z

It's on Saturday 7.30pm at Il Fornaio, 302 S Market St (in the Sainte Claire Hotel), San Jose. All aspiring rationalists welcome. The reservation is currently for 21 but can be changed if needed. Please RSVP if you haven't already.

They have a wide variety including the requested vegetarian options
& pizza from an oak wood burning oven, mostly good reviews, casual
dress code, the Wine Spectator Award of Excellence 2007, and are just
down the road from Montgomery Theater.

7.30 should give people time to finish chatting at the theater or
freshen up at their hotels, but if you want to come a bit earlier they
say they'll probably be able to seat you. If lots of people think 7.30
is too late I could change the time or suggest a cafe for early birds
to grab a coffee in before dinner.

Please RSVP whether or not you'll be coming, either in a comment or
email me at cursor_loop 4t yahoo p0int com. That way if new people want
to come as they can take unclaimed places, and I can let the restaurant
know if the number changes. If you think you'll be late, please let me
know that too so we can ask the restaurant to keep your seat.

Also, as my cellphone won't work San Jose, please could someone
volunteer their number for unexpectedly late or lost people to ring?

Please see here for further info. Hope to see you all on Saturday!

-- Posted on behalf of Michael Howard
Inner Goodness
by Eliezer Yudkowsky — 2008-10-23T22:19:45.000Z

Followup to: Which Parts Are "Me"?, Effortless Technique

A recent conversation with Michael Vassar touched on - or to be more accurate, he patiently explained to me - the psychology of at least three (3) different types of people known to him, who are evil and think of themselves as "evil". In ascending order of frequency:

The first type was someone who, having concluded that God does not exist, concludes that one should do all the things that God is said to dislike. (Apparently such folk actually exist.)

The third type was someone who thinks of "morality" only as a burden - all the things your parents say you can't do - and who rebels by deliberately doing those things.

The second type was a whole 'nother story, so I'm skipping it for now.

This reminded me of a topic I needed to post on:

Beware of placing goodness outside.

This specializes to e.g. my belief that ethicists should be inside rather than outside a profession: that it is futile to have "bioethicists" not working in biotech, or futile to think you can study Friendly AI without needing to think technically about AI.

But the deeper sense of "not placing goodness outside" was something I first learned at age ~15 from the celebrity logician Raymond Smullyan, in his book The Tao Is Silent, my first introduction to (heavily Westernized) Eastern thought.

Michael Vassar doesn't like this book. Maybe because most of the statements in it are patently false?

But The Tao Is Silent still has a warm place reserved in my heart, for it was here that I first encountered such ideas as:

Do you think of altruism as sacrificing one's own happiness for the sake of others, or as gaining one's happiness through the happiness of others?

(I would respond, by the way, that an "altruist" is someone who chooses between actions according to the criterion of others' welfare.)

A key chapter in The Tao Is Silent can be found online: "Taoism versus Morality". This chapter is medium-long (say, 3-4 Eliezer OB posts) but it should convey what I mean, when I say that this book manages to be quite charming, even though most of the statements in it are false.

Here is one key passage:

TAOIST: I think the word "humane" is
central to our entire problem. You are pushing morality. I am encouraging
humanity. You are em

...
Which Parts Are "Me"?
by Eliezer Yudkowsky — 2008-10-22T18:15:10.000Z

Everything I am, is surely my brain; but I don't accept everything my brain does, as "me".

Though I don't like the fact, it does seem in my case to be true, that reflecting upon a passion can diminish it. It does seem that in taking a step back from myself to look at what my brain is doing, that this introduces a greater emotional distance from the world, than when I am not reflecting. Though I am extremely cautious when it comes to the moral issue of how to modify human nature, this is one thing that I would change, I think.

The point at which you start thinking that not everything that goes on your brain is to be identified with "you", is something of a Rubicon, I fear. And those who prize the intensity of a simple passionate existence, above success in life or a complex layered personality, should not cross that Rubicon.

(Though in this case you should accept advice from rationalists about practical policies for guarding yourself, because this is a dangerous world in which to be unreflective. For example: if you don't follow the link to the rest of this post, then sign the #$!@ up for cryonics regardless of what your simple passions say on the subject.)

Still here? Then here is the Rubicon of which I speak:

(1) Never seeing ex-girlfriend-X again is so terribly unbearably sad!
->
(2) Why am I so terribly sad?
->
(3) Damn, my stupid brain is still obsessing about this when I just want to get over it.

The first scenario is what I would call the intensity of simple passion; nothing between the world and the emotions. The sentence contains no "I" to get in the way. There is nothing to think about except the world itself, the sadness and the missing person.

In the second scenario it is not the world that is sad, but some particular person, an "I"; and the sadness of this "I" is something that can be called into question.

And in the third scenario, the borders of "I" have been drawn in a way that doesn't include everything in the brain, so that "I" is the victim of the brain, struggling against it. And this is not paradoxical. Everything that I am, has to be in my brain somewhere, because there is nowhere else for it to be. But that doesn't mean I have to identify with everything that my brain does. Just as I draw the border of "me" to include my brain but exclude my comp...