trentmkelly/LessWrong-43k huggingface dataset:
Thanksgiving Prayer
by Eliezer Yudkowsky — 2008-11-28T04:17:00.000Z

At tonight's Thanksgiving, Erin remarked on how this was her first real Thanksgiving dinner away from her family, and that it was an odd feeling to just sit down and eat without any prayer beforehand. (Yes, she's a solid atheist in no danger whatsoever, thank you for asking.)

And as she said this, it reminded me of how wrong it is to give gratitude to God for blessings that actually come from our fellow human beings putting in a great deal of work.

So I at once put my hands together and said,

"Dear Global Economy, we thank thee for thy economies of scale, thy professional specialization, and thy international networks of trade under Ricardo's Law of Comparative Advantage, without which we would all starve to death while trying to assemble the ingredients for such a dinner as this. Amen."
Total Nano Domination
by Eliezer Yudkowsky — 2008-11-27T09:54:00.000Z

Followup to: Engelbart: Insufficiently Recursive

The computer revolution had cascades and insights aplenty. Computer tools are routinely used to create tools, from using a C compiler to write a Python interpreter, to using theorem-proving software to help design computer chips. I would not yet rate computers as being very deeply recursive - I don't think they've improved our own thinking processes even so much as the Scientific Revolution - yet. But some of the ways that computers are used to improve computers, verge on being repeatable (cyclic).

Yet no individual, no localized group, nor even country, managed to get a sustained advantage in computing power, compound the interest on cascades, and take over the world. There was never a Manhattan moment when a computing advantage temporarily gave one country a supreme military advantage, like the US and its atomic bombs for that brief instant at the end of WW2. In computing there was no equivalent of "We've just crossed the
sharp threshold of criticality, and now our pile doubles its neutron
output every two minutes, so we can produce lots of plutonium and you can't."

Will the development of nanotechnology go the same way as computers - a smooth, steady developmental curve spread across many countries, no one project taking into itself a substantial fraction of the world's whole progress? Will it be more like the Manhattan Project, one country gaining a (temporary?) huge advantage at huge cost? Or could a small group with an initial advantage cascade and outrun the world?

Just to make it clear why we might worry about this for nanotech, rather than say car manufacturing - if you can build things from atoms,
then the environment contains an unlimited supply of perfectly machined
spare parts. If your molecular factory can build solar cells, it can acquire energy as well.

So full-fledged Drexlerian molecular nanotechnology can plausibly automate away much of the manufacturing in its material
supply chain. If you already have nanotech, you may not need to
consult the outside economy for inputs of energy or raw material.

This makes it more plausible that a nanotech group could localize off, and do its own compound interest, away from the global economy. If you're Douglas Engelbart building better software, you still need to consult Intel for the ...
Engelbart: Insufficiently Recursive
by Eliezer Yudkowsky — 2008-11-26T08:31:09.000Z

Followup to: Cascades, Cycles, Insight, Recursion, Magic
Reply to: Engelbart As Ubertool?

When Robin originally suggested that Douglas Engelbart, best known as the inventor of the computer mouse, would have been a good candidate for taking over the world via compound interest on tools that make tools, my initial reaction was "What on Earth? With a mouse?"

On reading the initial portions of Engelbart's "Augmenting Human Intellect: A Conceptual Framework", it became a lot clearer where Robin was coming from.

Sometimes it's hard to see through the eyes of the past. Engelbart was a computer pioneer, and in the days when all these things were just getting started, he had a vision of using computers to systematically augment human intelligence. That was what he thought computers were for. That was the ideology lurking behind the mouse. Something that makes its users smarter - now that sounds a bit more plausible as an UberTool.

Looking back at Engelbart's plans with benefit of hindsight, I see two major factors that stand out:

Engelbart committed the Classic Mistake of AI: underestimating how much cognitive work gets done by hidden algorithms running beneath the surface of introspection, and overestimating what you can do by fiddling with the visible control levers.

Engelbart anchored on the way that someone as intelligent as Engelbart would use computers, but there was only one of him - and due to point 1 above, he couldn't use computers to make other people as smart as him.

To start with point 2: They had more reverence for computers back in the old days. Engelbart visualized a system carefully designed to flow with every step of a human's work and thought, assisting every iota it could manage along the way. And the human would be trained to work with the computer, the two together dancing a seamless dance.

And the problem with this, was not just that computers got cheaper and that programmers wrote their software more hurriedly.

There's a now-legendary story about the Windows Vista shutdown menu, a simple little feature into which 43 different Microsoft people had input. The debate carried on for over a year. The final product ended up as the lowest common denominator - a couple of hundred lines of code and a very visually unimpressive menu.

So even when lots of people spent a tremendous amount o...
The Complete Idiot's Guide to Ad Hominem
by Eliezer Yudkowsky — 2008-11-25T21:47:18.000Z

Stephen Bond writes the definitive word on ad hominem in "the ad hominem fallacy fallacy":

In reality, ad hominem is unrelated to sarcasm or personal abuse. Argumentum
ad hominem is the logical fallacy of attempting to undermine a speaker's
argument by attacking the speaker instead of addressing the argument. The mere
presence of a personal attack does not indicate ad hominem: the attack must be
used for the purpose of undermining the argument, or otherwise the logical fallacy
isn't there.

[...]

A: "All rodents are mammals, but a weasel isn't a rodent, so it can't be
a mammal."
B: "You evidently know nothing about logic. This does not logically follow."

B's argument is still not ad hominem. B does not imply that A's
sentence does not logically follow because A knows nothing about logic. B
is still addressing the substance of A's argument...

This is too beautiful, thorough, and precise to not post. HT to sfk on HN.
...Recursion, Magic
by Eliezer Yudkowsky — 2008-11-25T09:10:38.000Z

Followup to: Cascades, Cycles, Insight...

...4, 5 sources of discontinuity.

Recursion is probably the most difficult part of this topic. We have historical records aplenty of cascades, even if untangling the causality is difficult. Cycles of reinvestment are the heartbeat of the modern economy. An insight that makes a hard problem easy, is something that I hope you've experienced at least once in your life...

But we don't have a whole lot of experience redesigning our own neural circuitry.

We have these wonderful things called "optimizing compilers". A compiler translates programs in a high-level language, into machine code (though these days it's often a virtual machine). An "optimizing compiler", obviously, is one that improves the program as it goes.

So why not write an optimizing compiler in its own language, and then run it on itself? And then use the resulting optimized optimizing compiler, to recompile itself yet again, thus producing an even more optimized optimizing compiler -

Halt! Stop! Hold on just a minute! An optimizing compiler is not supposed to change the logic of a program - the input/output relations. An optimizing compiler is only supposed to produce code that does the same thing, only faster. A compiler isn't remotely near understanding what the program is doing and why, so it can't presume to construct a better input/output function. We just presume that the programmer wants a fixed input/output function computed as fast as possible, using as little memory as possible.

So if you run an optimizing compiler on its own source code, and then use the product to do the same again, it should produce the same output on both occasions - at most, the first-order product will run faster than the original compiler.

If we want a computer program that experiences cascades of self-improvement, the path of the optimizing compiler does not lead there - the "improvements" that the optimizing compiler makes upon itself, do not improve its ability to improve itself.

Now if you are one of those annoying nitpicky types, like me, you will notice a flaw in this logic: suppose you built an optimizing compiler that searched over a sufficiently wide range of possible optimizations, that it did not ordinarily have time to do a full search of its own space - so that, when the optimizing compiler r...
Cascades, Cycles, Insight...
by Eliezer Yudkowsky — 2008-11-24T09:33:40.000Z

Followup to: Surprised by Brains

Five sources of discontinuity: 1, 2, and 3...

Cascades are when one thing leads to another. Human brains are effectively discontinuous with chimpanzee brains due to a whole bag of design improvements, even though they and we share 95% genetic material and only a few million years have elapsed since the branch. Why this whole series of improvements in us, relative to chimpanzees? Why haven't some of the same improvements occurred in other primates?

Well, this is not a question on which one may speak with authority (so far as I know). But I would venture an unoriginal guess that, in the hominid line, one thing led to another.

The chimp-level task of modeling others, in the hominid line, led to improved self-modeling which supported recursion which enabled language which birthed politics that increased the selection pressure for outwitting which led to sexual selection on wittiness...

...or something. It's hard to tell by looking at the fossil record what happened in what order and why. The point being that it wasn't one optimization that pushed humans ahead of chimps, but rather a cascade of optimizations that, in Pan, never got started.

We fell up the stairs, you might say. It's not that the first stair ends the world, but if you fall up one stair, you're more likely to fall up the second, the third, the fourth...

I will concede that farming was a watershed invention in the history of
the human species, though it intrigues me for a different reason than
Robin. Robin, presumably, is interested because the economy grew by
two orders of magnitude, or something like that. But did having a
hundred times as many humans, lead to a hundred times as much thought-optimization accumulating
per unit time? It doesn't seem likely, especially in the age before
writing and telephones. But farming, because of its sedentary and
repeatable nature, led to repeatable trade, which led to debt records. Aha!
- now we have writing. There's a significant invention, from the perspective of cumulative optimization by brains. Farming isn't writing but it cascaded to writing.

Farming also cascaded (by way of surpluses and cities) to support professional specialization. I suspect that having someone spend their whole life
thinking about topic X instead of a hundre...
Surprised by Brains
by Eliezer Yudkowsky — 2008-11-23T07:26:41.000Z

Followup to: Life's Story Continues

Imagine two agents who've never seen an intelligence -
including, somehow, themselves - but who've seen the rest of the
universe up until now, arguing about what these newfangled "humans"
with their "language" might be able to do...

Believer: Previously, evolution has taken hundreds of thousands of years to create new complex adaptations with many working parts. I believe that, thanks to brains and language, we may see a new era, an era of intelligent design.
In this era, complex causal systems - with many interdependent parts
that collectively serve a definite function - will be created by the cumulative work of many brains
building upon each others' efforts.

Skeptic: I see - you think that brains might have
something like a 50% speed advantage over natural selection? So
it might take a while for brains to catch up, but after another eight billion
years, brains will be in the lead. But this planet's Sun will swell up
by then, so -

Believer: Thirty percent? I was thinking more like three orders of magnitude.
With thousands of brains working together and building on each others'
efforts, whole complex machines will be designed on the timescale of
mere millennia - no, centuries!

Skeptic: What?

Believer: You heard me.

Skeptic: Oh, come on! There's absolutely no empirical evidence for
an assertion like that! Animal brains have been around for hundreds of
millions of years without doing anything like what you're saying. I
see no reason to think that life-as-we-know-it will end, just because
these hominid brains have learned to send low-bandwidth signals over their vocal cords. Nothing like what
you're saying has happened before in my experience -

Believer: That's kind of the point, isn't it? That nothing
like this has happened before? And besides, there is precedent for
that kind of Black Swan - namely, the first replicator.

Skeptic: Yes, there is precedent in the replicators. Thanks
to our observations of evolution, we have extensive knowledge and many
examples of how optimization works. We know, in particular, that
optimization isn't easy - it takes millions of years to climb up through the search space. Why should
"brain

...
Life's Story Continues
by Eliezer Yudkowsky — 2008-11-21T23:05:47.000Z

Followup to: The First World Takeover

As last we looked at the planet, Life's long search in organism-space had only just gotten started.

When I try to structure my understanding of the unfolding process of
Life, it seems to me that,
to understand the optimization velocity at any given point, I
want to break down that velocity using the following abstractions:

The searchability of the neighborhood of the current location,
and the availability of good/better alternatives in that rough region.
Maybe call this the optimization slope. Are the fruit low-hanging or high-hanging, and how large are the
fruit?

The optimization resources, like the amount of computing power
available to a fixed program, or the number of individuals in a population pool.

The optimization efficiency, a curve that gives the amount of
searchpower generated by a given investiture of resources, which is
presumably a function of the optimizer's structure at that point in
time.

Example: If an object-level adaptation enables more
efficient extraction of resources, and thereby increases the total
population that can be supported by fixed available resources, then this
increases the optimization resources and perhaps the optimization
velocity.

How much does optimization velocity increase - how hard does this object-level innovation hit back to the meta-level?

If a population is small enough that not
all mutations are occurring in each generation, then a larger
population decreases the time for a given mutation to show up. If the fitness improvements offered by beneficial mutations follow an exponential distribution, then - I'm not actually doing the math here, just sort of eyeballing - I would expect the optimization velocity to go as log population size, up to a maximum where the search neighborhood is explored thoroughly. (You could test this in the lab, though not just by eyeballing the fossil record.)

This doesn't mean all optimization processes would have a momentary velocity that goes as the log of momentary resource investment up to a maximum. Just one mode of evolution would have this character. And even under these assumptions, evolution's cumulative optimization wouldn't go as log of cumulative resources - the log-pop curve is just the instantaneous velocity.
If we assume that the variance of the neighborhood remains the same
over the course of e...
Observing Optimization
by Eliezer Yudkowsky — 2008-11-21T05:39:25.000Z

Followup to: Optimization and the Singularity

In "Optimization and the Singularity" I pointed out that history since the first replicator, including human history to date, has mostly been a case of nonrecursive optimization - where you've got one thingy doing the optimizing, and another thingy getting optimized. When evolution builds a better amoeba, that doesn't change the structure of evolution
- the mutate-reproduce-select cycle.

But there are exceptions to this
rule, such as the invention of sex, which affected the structure of
natural selection itself - transforming it to
mutate-recombine-mate-reproduce-select.

I was surprised when Robin, in "Eliezer's Meta-Level Determinism" took that idea and ran with it and said:

...his view does seem to make testable predictions
about history. It suggests the introduction of natural selection and
of human culture coincided with the very largest capability growth rate
increases. It suggests that the next largest increases were much
smaller and coincided in biology with the introduction of cells and
sex, and in humans with the introduction of writing and science. And
it suggests other rate increases were substantially smaller.

It
hadn't occurred to me to try to derive that kind of testable
prediction. Why? Well, partially because I'm not an economist. (Don't get me wrong, it was a virtuous step to try.) But
also because the whole issue looked to me like it was a lot more
complicated than that, so it hadn't occurred to me to try to directly
extract predictions.

What is this "capability growth rate" of which you speak, Robin?
There are old, old controversies in evolutionary biology involved here.

Just to start by pointing out the obvious - if there are fixed
resources available, only so much grass to be eaten or so many rabbits
to consume, then any evolutionary "progress" that we would recognize as
producing a better-designed organism, may just result in the
displacement of the old allele by the new allele - not any increase in
the population as a whole. It's quite possible to have a new wolf that
expends 10% more energy per day to be 20% better at hunting, and in
this case the sustainable wolf population will decrease as new wolves
replace old.

If I was going to talk about the effect that a meta-level change might have on the ...
Whence Your Abstractions?
by Eliezer Yudkowsky — 2008-11-20T01:07:46.000Z

Reply to: Abstraction, Not Analogy

Robin asks:

Eliezer, have I completely failed
to communicate here? You have previously said nothing is similar enough
to this new event for analogy to be useful, so all we have is "causal
modeling" (though you haven't explained what you mean by this in this
context). This post is a reply saying, no, there are more ways using
abstractions; analogy and causal modeling are two particular ways to
reason via abstractions, but there are many other ways.

Well... it shouldn't be surprising if you've communicated less than you thought. Two people, both of whom know that disagreement is not allowed, have a persistent disagreement. It doesn't excuse anything, but - wouldn't it be more surprising if their disagreement rested on intuitions that were easy to convey in words, and points readily dragged into the light?

I didn't think from the beginning that I was succeeding in communicating. Analogizing Doug Engelbart's mouse to a self-improving AI is for me such a flabbergasting notion - indicating such completely different ways of thinking about the problem - that I am trying to step back and find the differing sources of our differing intuitions.

(Is that such an odd thing to do, if we're really following down the path of not agreeing to disagree?)

"Abstraction", for me, is a word that means a partitioning of possibility - a boundary around possible things, events, patterns. They are in no sense neutral; they act as signposts saying "lump these things together for predictive purposes". To use the word "singularity" as ranging over human brains, farming, industry, and self-improving AI, is very nearly to finish your thesis right there.

I wouldn't be surprised to find that, in a real AI, 80% of the actual computing crunch goes into drawing the right boundaries to make the actual reasoning possible. The question "Where do abstractions come from?" cannot be taken for granted.

Boundaries are drawn by appealing to other boundaries. To draw the boundary "human" around things that wear clothes and speak language and have a certain shape, you must have previously noticed the boundaries around clothing and language. And your visual cortex already has a (damned sophisticated) system for categorizing visual scenes into shapes, and t...
The First World Takeover
by Eliezer Yudkowsky — 2008-11-19T15:00:00.000Z

Before Robin and I move on to talking about the Future, it seems to me wise to check if we have disagreements in our view of the Past. Which might be much easier to discuss - and maybe even resolve... So...

In the beginning was the Bang. For nine billion years afterward, nothing much happened.

Stars formed, and burned for long periods or short periods depending on their structure; but "successful" stars that burned longer or brighter did not pass on their characteristics to other stars. The first replicators were yet to come.

It was the Day of the Stable Things, when your probability of seeing something was given by its probability of accidental formation times its duration. Stars last a long time; there are many helium atoms.

It was the Era of Accidents, before the dawn of optimization. You'd only expect to see something with 40 bits of optimization if you looked through a trillion samples. Something with 1000 bits' worth of functional complexity? You wouldn't expect to find that in the whole universe.

I would guess that, if you were going to be stuck on a desert island and you wanted to stay entertained as long as possible, then you should sooner choose to examine the complexity of the cells and biochemistry of a single Earthly butterfly, over all the stars and astrophysics in the visible universe beyond Earth.

It was the Age of Boredom.

The hallmark of the Age of Boredom was not lack of natural resources - it wasn't that the universe was low on hydrogen - but, rather, the lack of any cumulative search. If one star burned longer or brighter, that didn't affect the probability distribution of the next star to form. There was no search but blind search. Everything from scratch, not even looking at the neighbors of previously successful points. Not hill-climbing, not mutation and selection, not even discarding patterns already failed. Just a random sample from the same distribution, over and over again.

The Age of Boredom ended with the first replicator.

(Or the first replicator to catch on, if there were failed alternatives lost to history - but this seems unlikely, given the Fermi Paradox; a replicator should be more improbable than that, or the stars would teem with life already.)

Though it might be most dramatic to think of a single RNA strand a few dozen bases long, forming by pure accide...