trentmkelly/LessWrong-43k huggingface dataset:
Artificial Mysterious Intelligence
by Eliezer Yudkowsky — 2008-12-07T20:05:33.000Z

Previously in series: Failure By Affective Analogy

I once had a conversation that I still remember for its sheer, purified archetypicality. This was a nontechnical guy, but pieces of this dialog have also appeared in conversations I've had with professional AIfolk...

Him: Oh, you're working on AI! Are you using neural networks?

Me: I think emphatically not.

Him: But neural networks are so wonderful! They solve problems and we don't have any idea how they do it!

Me: If you are ignorant of a phenomenon, that is a fact about your state of mind, not a fact about the phenomenon itself. Therefore your ignorance of how neural networks are solving a specific problem, cannot be responsible for making them work better.

Him: Huh?

Me: If you don't know how your AI works, that is not good. It is bad.

Him: Well, intelligence is much too difficult for us to understand, so we need to find some way to build AI without understanding how it works.

Me: Look, even if you could do that, you wouldn't be able to predict any kind of positive outcome from it. For all you knew, the AI would go out and slaughter orphans.

Him: Maybe we'll build Artificial Intelligence by scanning the brain and building a neuron-by-neuron duplicate. Humans are the only systems we know are intelligent.

Me: It's hard to build a flying machine if the only thing you understand about flight is that somehow birds magically fly. What you need is a concept of aerodynamic lift, so that you can see how something can fly even if it isn't exactly like a bird.

Him: That's too hard. We have to copy something that we know works.

Me: (reflectively) What do people find so unbearably awful about the prospect of having to finally break down and solve the bloody problem? Is it really that horrible?

Him: Wait... you're saying you want to actually understand intelligence?

Me: Yeah.

Him: (aghast) Seriously?

Me: I don't know everything I need to know about intelligence, but I've learned a hell of a lot. Enough to know what happens if I try to build AI while there are still gaps in my understanding.

Him: Understanding the problem is too hard. You'll never do it.

That's not just a difference of opinion you're looking at, it's a clash of cultures.

For a long t...
Is That Your True Rejection?
by Eliezer Yudkowsky — 2008-12-06T14:26:15.000Z

It happens every now and then that someone encounters some of my transhumanist-side beliefs—as opposed to my ideas having to do with human rationality—strange, exotic-sounding ideas like superintelligence and Friendly AI. And the one rejects them.

If the one is called upon to explain the rejection, not uncommonly the one says, “Why should I believe anything Yudkowsky says? He doesn’t have a PhD!”

And occasionally someone else, hearing, says, “Oh, you should get a PhD, so that people will listen to you.” Or this advice may even be offered by the same one who expressed disbelief, saying, “Come back when you have a PhD.”

Now, there are good and bad reasons to get a PhD. This is one of the bad ones.

There are many reasons why someone might actually have an initial adverse reaction to transhumanist theses. Most are matters of pattern recognition, rather than verbal thought: the thesis calls to mind an associated category like “strange weird idea” or “science fiction” or “end-of-the-world cult” or “overenthusiastic youth.”1 Immediately, at the speed of perception, the idea is rejected.

If someone afterward says, “Why not?” this launches a search for justification, but the search won’t necessarily hit on the true reason. By “‘true reason,” I don’t mean the best reason that could be offered. Rather, I mean whichever causes were decisive as a matter of historical fact, at the very first moment the rejection occurred.

Instead, the search for justification hits on the justifying-sounding fact, “This speaker does not have a PhD.” But I also don’t have a PhD when I talk about human rationality, so why is the same objection not raised there?

More to the point, if I had a PhD, people would not treat this as a decisive factor indicating that they ought to believe everything I say. Rather, the same initial rejection would occur, for the same reasons; and the search for justification, afterward, would terminate at a different stopping point.

They would say, “Why should I believe you? You’re just some guy with a PhD! There are lots of those. Come back when you’re well-known in your field and tenured at a major university.”

But do people...
Sustained Strong Recursion
by Eliezer Yudkowsky — 2008-12-05T21:03:20.000Z

Followup to: Cascades, Cycles, Insight, Recursion, Magic

We seem to have a sticking point at the concept of "recursion", so I'll zoom in.

You have a friend who, even though he makes plenty of money, just spends all that money every month. You try to persuade your friend to invest a little - making valiant attempts to explain the wonders of compound interest by pointing to analogous processes in nature, like fission chain reactions.

"All right," says your friend, and buys a ten-year bond for $10,000, with an annual coupon of $500. Then he sits back, satisfied. "There!" he says. "Now I'll have an extra $500 to spend every year, without my needing to do any work! And when the bond comes due, I'll just roll it over, so this can go on indefinitely. Surely, now I'm taking advantage of the power of recursion!"

"Um, no," you say. "That's not exactly what I had in mind when I talked about 'recursion'."

"But I used some of my cumulative money earned, to increase my very earning rate," your friend points out, quite logically. "If that's not 'recursion', what is? My earning power has been 'folded in on itself', just like you talked about!"

"Well," you say, "not exactly. Before, you were earning $100,000 per year, so your cumulative earnings went as 100000 * t. Now, your cumulative earnings are going as 100500 * t. That's not really much of a change. What we want is for your cumulative earnings to go as B * e^At for some constants A and B - to grow exponentially."

"Exponentially!" says your friend, shocked.

"Yes," you say, "recursification has an amazing power to transform growth curves. In this case, it can turn a linear process into an exponential one. But to get that effect, you have to reinvest the coupon payments you get on your bonds - or at least reinvest some of them, instead of just spending them all. And you must be able to do this over and over again. Only then will you get the 'folding in' transformation, so that instead of your cumulative earnings going as y = F(t) = A*t, your earnings will go as the differential equation dy/dt = F(y) = A*y whose solution is y = e^(A*t)."

(I'm going to go ahead and leave out various constants of inte...
Underconstrained Abstractions
by Eliezer Yudkowsky — 2008-12-04T13:58:51.000Z

Followup to: The Weak Inside View

Saith Robin:

"It is easy, way too easy, to generate new mechanisms, accounts, theories, and abstractions. To see if such things are useful,
we need to vet them, and that is easiest "nearby", where we know a
lot. When we want to deal with or understand things "far", where we
know little, we have little choice other than to rely on mechanisms,
theories, and concepts that have worked well near. Far is just the
wrong place to try new things."

Well... I understand why one would have that reaction. But I'm not sure we can really get away with that.

When possible, I try to talk in concepts that can be verified with
respect to existing history. When I talk about natural selection not
running into a law of diminishing returns on genetic complexity or
brain size, I'm talking about something that we can try to verify by
looking at the capabilities of other organisms with brains big and
small. When I talk about the boundaries to sharing cognitive content
between AI programs, you can look at the field of AI the way it works
today and see that, lo and behold, there isn't a lot of cognitive
content shared.

But in my book this is just one trick in a library of
methodologies for dealing with the Future, which is, in general, a hard
thing to predict.

Let's say that instead of using my complicated-sounding disjunction (many different reasons why the growth trajectory might contain an upward cliff, which don't all have to be true), I instead staked my whole story on the critical
threshold of human intelligence. Saying, "Look how sharp the slope is
here!" - well, it would sound like a simpler story. It would be closer
to fitting on a T-Shirt. And by talking about just that one
abstraction and no others, I could make it sound like I was dealing in
verified historical facts - humanity's evolutionary history is
something that has already happened.

But speaking of an abstraction being "verified" by previous history is a tricky thing. There is this little problem of underconstraint - of there being more than one possible abstraction that the data "verifies".

In "Cascades, Cycles, Insight"
I said that economics does not seem to me to deal much in the origins
of novel knowledge and novel designs, and said, "If I underesti...
Permitted Possibilities, & Locality
by Eliezer Yudkowsky — 2008-12-03T21:20:35.000Z

Continuation of: Hard Takeoff

The analysis given in the last two days permits more than one possible AI trajectory:

Programmers, smarter than evolution at finding tricks that work, but operating without fundamental insight or with only partial insight, create a mind that is dumber than the researchers but performs lower-quality operations much faster. This mind reaches k > 1, cascades up to the level of a very smart human, itself achieves insight into intelligence, and undergoes the really fast part of the FOOM, to superintelligence. This would be the major nightmare scenario for the origin of an unFriendly AI.

Programmers operating with partial insight, create a mind that performs a number of tasks very well, but can't really handle self-modification let alone AI theory. A mind like this might progress with something like smoothness, pushed along by the researchers rather than itself, even all the way up to average-human capability - not having the insight into its own workings to push itself any further. We also suppose that the mind is either already using huge amounts of available hardware, or scales very poorly, so it cannot go FOOM just as a result of adding a hundred times as much hardware. This scenario seems less likely to my eyes, but it is not ruled out by any effect I can see.

Programmers operating with strong insight into intelligence, directly create along an efficient and planned pathway, a mind capable of modifying itself with deterministic precision - provably correct or provably noncatastrophic self-modifications. This is the only way I can see to achieve narrow enough targeting to create a Friendly AI. The "natural" trajectory of such an agent would be slowed by the requirements of precision, and sped up by the presence of insight; but because this is a Friendly AI, notions like "You can't yet improve yourself this far, your goal system isn't verified enough" would play a role.

So these are some things that I think are permitted to happen, albeit that case 2 would count as a hit against me to some degree because it does seem unlikely.

Here are some things that shouldn't happen, on my analysis:

An ad-hoc self-modifying AI as in (1) undergoes a cycle of self-improvement, starting from stupidity, that
carries it up to the level of a very smart human - and then stops,
u

...
Hard Takeoff
by Eliezer Yudkowsky — 2008-12-02T20:44:26.000Z

Continuation of: Recursive Self-Improvement

Constant natural selection pressure, operating on the genes of the hominid line, produced improvement in brains over time that seems to have been, roughly, linear or accelerating; the operation of constant human brains on a pool of knowledge seems to have produced returns that are, very roughly, exponential or superexponential. (Robin proposes that human progress is well-characterized as a series of exponential modes with diminishing doubling times.)

Recursive self-improvement - an AI rewriting its own cognitive algorithms - identifies the object level of the AI with a force acting on the metacognitive level; it "closes the loop" or "folds the graph in on itself". E.g. the difference between returns on a constant investment in a bond, and reinvesting the returns into purchasing further bonds, is the difference between the equations y = f(t) = m*t, and dy/dt = f(y) = m*y whose solution is the compound interest exponential, y = e^(m*t).

When you fold a whole chain of differential equations in on itself like this, it should either peter out rapidly as improvements fail to yield further improvements, or else go FOOM. An exactly right law of diminishing returns that lets the system fly through the soft takeoff keyhole is unlikely - far more unlikely than seeing such behavior in a system with a roughly-constant underlying optimizer, like evolution improving brains, or human brains improving technology. Our present life is no good indicator of things to come.

Or to try and compress it down to a slogan that fits on a T-Shirt - not that I'm saying this is a good idea - "Moore's Law is exponential now; it would be really odd if it stayed exponential with the improving computers doing the research." I'm not saying you literally get dy/dt = e^y that goes to infinity after finite time - and hardware improvement is in some ways the least interesting factor here - but should we really see the same curve we do now?

RSI is the biggest, most interesting, hardest-to-analyze, sharpest break-with-the-past contributing to the notion of a "hard takeoff" aka "AI go FOOM", but it's nowhere near being the only such factor. The advent of human intelligence was a discontinuity with the past even without RSI...

...which is to say that observed evolutionary history - the discontinuity between humans, and chim...
Recursive Self-Improvement
by Eliezer Yudkowsky — 2008-12-01T20:49:11.000Z

Followup to: Life's Story Continues, Surprised by Brains, Cascades, Cycles, Insight, Recursion, Magic, Engelbart: Insufficiently Recursive, Total Nano Domination

I think that at some point in the development of Artificial Intelligence, we are likely to see a fast, local
increase in capability - "AI go FOOM". Just to be clear on the claim,
"fast" means on a timescale of weeks or hours rather than years or
decades; and "FOOM" means way the hell smarter than anything else
around, capable of delivering in short time periods technological
advancements that would take humans decades, probably including
full-scale molecular nanotechnology (that it gets by e.g. ordering
custom proteins over the Internet with 72-hour turnaround time). Not,
"ooh, it's a little Einstein but it doesn't have any robot hands, how cute".

Most people who object to this scenario, object to the "fast" part.
Robin Hanson objected to the "local" part. I'll try to handle both, though not all in one shot today.

We are setting forth to analyze the developmental velocity of an Artificial Intelligence. We'll break down this velocity into optimization slope, optimization resources, and optimization efficiency. We'll need to understand cascades, cycles, insight and recursion; and we'll stratify our recursive levels into the metacognitive, cognitive, metaknowledge, knowledge, and object level.

Quick review:

"Optimization slope" is the goodness and number of
opportunities in the volume of solution space you're currently
exploring, on whatever your problem is;

"Optimization resources" is how much computing power, sensory
bandwidth, trials, etc. you have available to explore opportunities;

"Optimization efficiency" is how well you use your resources. This
will be determined by the goodness of your current mind design - the
point in mind design space that is your current self - along with its
knowledge and metaknowledge (see below).

Optimizing yourself is a special case, but it's one we're about to spend a lot of time talking about.

By the time any mind solves some kind of actual problem,
there's actually been a huge causal lattice of optimizations applied -
for example, humans brain evolved, and then humans developed the idea
of science, and then applied the idea of science to ...
Disappointment in the Future
by Eliezer Yudkowsky — 2008-12-01T04:45:00.000Z

This seems worth posting around now... As I've previously observed, futuristic visions are produced as entertainment, sold today and consumed today. A TV station interviewing an economic or diplomatic pundit doesn't bother to show what that pundit predicted three years ago and how the predictions turned out. Why would they? Futurism Isn't About Prediction.

But someone on the ImmInst forum actually went and compiled a list of Ray Kurzweil's predictions in 1999 for the years 2000-2009. We're not out of 2009 yet, but right now it's not looking good...

· Individuals primarily use portable computers
· Portable computers have dramatically become lighter and thinner
· Personal computers are available in a wide range of sizes and shapes, and are commonly embedded in clothing and jewelry, like wrist watches, rings, earrings and other body ornaments
· Computers with a high-resolution visual interface range from rings and pins and credit cards up to the size of a thin book. People typically have at least a dozen computers on and around their bodies, which are networked, using body LANS (local area networks)
· These computers monitor body functions, provide automated identity to conduct financial transactions and allow entry into secure areas. They also provide directions for navigation, and a variety of other services.
· Most portable computers do not have keyboards

· Rotating memories such as Hard Drives, CD roms, and DVDs are on their way out.
· Most users have servers on their homes and offices where they keep large stores of digital objects, including, among other things, virtual reality environments, although these are still on an early stage
· Cables are disappearing,
· The majority of texts is created using continuous speech recognition, or CSR (dictation software). CSRs are very accurate, far more than the human transcriptionists, who were used up until a few years ago.
· Books, magazines, and newspapers are now routinely read on displays that are the size of small books
· Computer displays built into eyeglasses are also used. These specialized glasses allow the users to see the normal environment while creating a virtual image that appears to hover in front of the viewer.
· Computers routinely include moving picture image cameras and are able to reliably identify their owners from their faces
· Three dimensional chips are commonly used
· Studen

...
Singletons Rule OK
by Eliezer Yudkowsky — 2008-11-30T16:45:58.000Z

Reply to: Total Tech Wars

How does one end up with a persistent disagreement between two rationalist-wannabes who are both aware of Aumann's Agreement Theorem and its implications?

Such a case is likely to turn around two axes: object-level incredulity ("no matter what AAT says, proposition X can't really be true") and meta-level distrust ("they're trying to be rational despite their emotional commitment, but are they really capable of that?").

So far, Robin and I have focused on the object level in trying to hash out our disagreement. Technically, I can't speak for Robin; but at least in my own case, I've acted thus because I anticipate that a meta-level argument about trustworthiness wouldn't lead anywhere interesting. Behind the scenes, I'm doing what I can to make sure my brain is actually capable of updating, and presumably Robin is doing the same.

(The linchpin of my own current effort in this area is to tell
myself that I ought to be learning something while having this
conversation, and that I shouldn't miss any scrap of original thought
in it - the Incremental Update technique.
Because I can genuinely believe that a conversation like this should
produce new thoughts, I can turn that feeling into genuine
attentiveness.)

Yesterday, Robin inveighed hard against what he called "total tech wars", and what I call "winner-take-all" scenarios:

Robin: "If you believe the other side is totally committed to total victory,
that surrender is unacceptable, and that all interactions are zero-sum,
you may conclude your side must never cooperate with them, nor tolerate
much internal dissent or luxury."

Robin and I both have emotional commitments and we both acknowledge the danger of that. There's nothing irrational about feeling, per se; only failure to update is blameworthy. But Robin seems to be very strongly against winner-take-all technological scenarios, and I don't understand why.

Among other things, I would like to ask if Robin has a Line of Retreat set up here - if, regardless of how he estimates the probabilities, he can visualize what he would do if a winner-take-all scenario were true.

Yesterday Robin wrote:

"Eliezer, if everything is at stake
then 'winner take all' is 'total war'; it doesn't really matter if they
shoot you or just starve you to death."

We both have our ...
Chaotic Inversion
by Eliezer Yudkowsky — 2008-11-29T10:57:24.000Z

I was recently having a conversation with some friends on the topic of hour-by-hour productivity and willpower maintenance—something I've struggled with my whole life.

I can avoid running away from a hard problem the first time I see it (perseverance on a timescale of seconds), and I can stick to the same problem for years; but to keep working on a timescale of hours is a constant battle for me. It goes without saying that I've already read reams and reams of advice; and the most help I got from it was realizing that a sizable fraction other creative professionals had the same problem, and couldn't beat it either, no matter how reasonable all the advice sounds.

"What do you do when you can't work?" my friends asked me. (Conversation probably not accurate, this is a very loose gist.)

And I replied that I usually browse random websites, or watch a short video.

"Well," they said, "if you know you can't work for a while, you should watch a movie or something."

"Unfortunately," I replied, "I have to do something whose time comes in short units, like browsing the Web or watching short videos, because I might become able to work again at any time, and I can't predict when—"

And then I stopped, because I'd just had a revelation.

I'd always thought of my workcycle as something chaotic, something unpredictable. I never used those words, but that was the way I treated it.

But here my friends seemed to be implying—what a strange thought—that other people could predict when they would become able to work again, and structure their time accordingly.

And it occurred to me for the first time that I might have been committing that damned old chestnut the Mind Projection Fallacy, right out there in my ordinary everyday life instead of high abstraction.

Maybe it wasn't that my productivity was unusually chaotic; maybe I was just unusually stupid with respect to predicting it.

That's what inverted stupidity looks like—chaos. Something hard to handle, hard to grasp, hard to guess, something you can't do anything with. It's not just an idiom for high abstract things like Artificial Intelligence. It can apply in ordinary life too.

And the reason we don't think of the alternative explanation "I'm stupid", is not—I suspect—that we think so highly of ourselves. It's just that we don't think of ourselves at a...