trentmkelly/LessWrong-43k huggingface dataset:
Transcription of Eliezer's January 2010 video Q&A
by curiousepic — 2011-11-14T17:02:03.821Z

Spurred by discussion of whether Luke's Q&A session should be on video or text-only, I volunteered to transcribe Eliezer's Q&A videos from January 2010. I finished last night, much earlier than my estimate, mostly due to feeling motivated to finish it and spending more on it than my very conservative estimated 30 minutes a day (estimate of number of words was pretty close; about 16000). I have posted a link to this post as a comment in the original thread here, if you would like to upvote that.

Some advice for transcribing videos: I downloaded the .wmv videos, which allowed me to use VLC's global hotkeys to create a pause and "short skip backwards and forwards" buttons (ctrl-space and ctrl-shift left/right arrow), which were so much more convenient than any other method I tried.

Edited out: repetition of the question, “um/uh”, “you know,” false starts.

Punctuation, capitalization, and structure, etc may not be entirely consistent.

Keep in mind the opinions expressed here are those of Eliezer circa January 2010.

1. What is your information diet like? Do you control it deliberately (do you have a method; is it, er, intelligently designed), or do you just let it happen naturally.

By that I mean things like: Do you have a reading schedule (x number of hours daily, etc)? Do you follow the news, or try to avoid information with a short shelf-life? Do you frequently stop yourself from doing things that you enjoy (f.ex reading certain magazines, books, watching films, etc) to focus on what is more important? etc.

It’s not very planned, most of the time, in other words Hacker News, Reddit, Marginal Revolution, other random stuff found on the internet. In order to learn something, I usually have to set aside blocks of time and blocks of effort and just focus on specifically reading something. It’s only sort of popular level books which I can put on a restroom shelf and get them read that way. In order to learn actually useful information I generally find that I have to set aside blocks of time or run across a pot of gold, and you’re about as likely to get a pot of gold from Hacker News as anywhere else really. So not very controlled.

2. Your "Bookshelf" page is 10 years old (and contains a warning sign saying it is obsolete): http://yudkowsky.net/obsolete/bo

...
Why an Intelligence Explosion might be a Low-Priority Global Risk
by XiXiDu — 2011-11-14T11:40:38.917Z

(The following is a summary of some of my previous submissions that I originally created for my personal blog.)

As we know,
There are known knowns.
There are things
We know we know.
We also know
There are known unknowns.
That is to say
We know there are some things
We do not know.
But there are also unknown unknowns,
The ones we don’t know
We don’t know.

— Donald Rumsfeld, Feb. 12, 2002, Department of Defense news briefing

Intelligence, a cornucopia?

It seems to me that those who believe into the possibility of catastrophic risks from artificial intelligence act on the unquestioned assumption that intelligence is kind of a black box, a cornucopia that can sprout an abundance of novelty. But this implicitly assumes that if you increase intelligence you also decrease the distance between discoveries.

Intelligence is no solution in itself, it is merely an effective searchlight for unknown unknowns and who knows that the brightness of the light increases proportionally with the distance between unknown unknowns? To enable an intelligence explosion the light would have to reach out much farther with each increase in intelligence than the increase of the distance between unknown unknowns. I just don’t see that to be a reasonable assumption.

Intelligence amplification, is it worth it?

It seems that if you increase intelligence you also increase the computational cost of its further improvement and the distance to the discovery of some unknown unknown that could enable another quantum leap. It seems that you need to apply a lot more energy to get a bit more complexity.

If any increase in intelligence is vastly outweighed by its computational cost and the expenditure of time needed to discover it then it might not be instrumental for a perfectly rational agent (such as an artificial general intelligence), as imagined by game theorists, to increase its intelligence as opposed to using its existing intelligence to pursue its terminal goals directly or to invest its given resources to acquire other means of self-improvement, e.g. more efficient sensors.

What evidence do we have that the payoff of intelligent, goal-oriented experimentation yields enormous advantages (enough to enable an intelligence explosion) over evolutionary discovery relative to its cost?

We simply don’t know if intelligence is instrumental or quickly hits diminishing ret...
Is an Intelligence Explosion a Disjunctive or Conjunctive Event?
by XiXiDu — 2011-11-14T11:35:40.518Z

(The following is a summary of some of my previous submissions that I originally created for my personal blog.)

...an intelligence explosion may have fair probability, not because it occurs in one particular detailed scenario, but because, like the evolution of eyes or the emergence of markets, it can come about through many different paths and can gather momentum once it gets started. Humans tend to underestimate the likelihood of such “disjunctive” events, because they can result from many different paths (Tversky and Kahneman 1974). We suspect the considerations in this paper may convince you, as they did us, that this particular disjunctive event (intelligence explosion) is worthy of consideration.

— lukeprog, Intelligence Explosion analysis draft: introduction

It seems to me that all the ways in which we disagree have more to do with philosophy (how to quantify uncertainty; how to deal with conjunctions; how to act in consideration of low probabilities) [...] we are not dealing with well-defined or -quantified probabilities. Any prediction can be rephrased so that it sounds like the product of indefinitely many conjunctions. It seems that I see the “SIAI’s work is useful scenario” as requiring the conjunction of a large number of questionable things [...]

— Holden Karnofsky, 6/24/11 (GiveWell interview with major SIAI donor Jaan Tallinn, PDF)

Disjunctive arguments

People associated with the Singularity Institute for Artificial Intelligence (SIAI) like to claim that the case for risks from AI is supported by years worth of disjunctive lines of reasoning. This basically means that there are many reasons to believe that humanity is likely to be wiped out as a result of artificial general intelligence. More precisely it means that not all of the arguments supporting that possibility need to be true, even if all but one are false risks from AI are to be taken seriously.

The idea of disjunctive arguments is formalized by what is called a logical disjunction. Consider two declarative sentences, A and B. The truth of the conclusion (or output) that follows from the sentences A and B does depend on the truth of A and B. In the case of a logical disjunction the conclusion of A and B is only false if both A and B are false, otherwise it is true. Truth values are usually denoted by 0 for false and 1 for true. A dis...
Intelligence Explosion analysis draft: introduction
by lukeprog — 2011-11-14T09:50:22.531Z

I invite your feedback on this snippet from an intelligence explosion analysis Anna Salamon and myself have been working on.

This snippet is a possible introduction to the analysis article. Its purpose is to show readers that we aim to take seriously some common concerns about singularity thinking, to bring readers into Near Mode about the topic, and to explain the purpose and scope of the article.

Note that the target style is serious but still more chatty than a normal journal article.

_____

The best answer to the question, "Will computers ever be as smart as humans?" is probably “Yes, but only briefly."

Vernor Vinge

Humans may create human-level artificial intelligence in this century (Bainbridge 2006; Baum, Goertzel, and Goertzel 2011; Bostrom 2003; Legg 2008; Sandberg and Bostrom 2011). Shortly thereafter, we may see an “intelligence explosion” or “technological Singularity” — a chain of events by which human-level AI leads, fairly rapidly, to intelligent systems whose capabilities far surpass those of biological humanity as a whole (Chalmers 2010).

How likely is this, and what should we do about it? Others have discussed these questions previously (Turing 1950; Good 1965; Von Neumann 1966; Solomonoff 1985; Vinge 1993; Yudkowsky 2001, 2008a; Russell and Norvig 2010, sec. 26.3); we will build on their thinking in our review of the subject.

Singularity Skepticism

Many are skeptical of Singularity arguments because they associate such arguments with detailed storytelling — the “if and then” fallacy of “speculative ethics” by which an improbable conditional becomes a supposed actual (Nordmann 2007). They are right to be skeptical: hundreds of studies show that humans are overconfident of their beliefs (Moore and Healy 2008), regularly overestimate the probability of detailed visualized scenarios (Tversky and Kahneman 2002), and tend to seek out only information that confirms their current views (Nickerson 1998). AI researchers are not immune from these errors, as evidenced by a history of over-optimistic predictions going back to the 1956 Dartmouth conference on AI (Dreyfus 1972).

Nevertheless, mere mortals have at times managed to reason usefully and somewhat accurately about the future, even with little data. When Leo Szilard conceived of the nuclear chain reaction, ...
Drawing Less Wrong: Should You Learn to Draw?
by Raemon — 2011-11-14T07:31:46.724Z

This is the second post of the Drawing Less Wrong mini sequence, in which I discuss how to draw, how learning to draw *effectively* relates to rationality, and what the initial results were when I started running a drawing workshop, teaching people with essentially no experience.

Information here is a combination of lessons I've learned from numerous art teachers who all agree with each other, and some of my own observations that I'm pretty confident about. When I talk about "how the brain does things" I'm using a mix of folk psychology and guesses based on my limited knowledge of neuroscience, which may not be technically accurate but should be sufficient to make useful predictions.

The Nature of Talent:

"Am I talented enough to draw?"

This is a question people think about a lot. It's a wrong question.

Here are a few related, relevant "right" questions:

Do you have pre-existing skills that can be repurposed for drawing?
How quickly are you able to acquire skills relevant to drawing?
Do you naturally enjoy drawing?
If not, can you easily BECOME the sort of person who naturally enjoys drawing?
WHY and WHAT do you want to be able to draw?
How well do you want to be able to draw? How much do you value being able to draw that well?
How many hours of dedicated practice are you willing to put in to achieve this?

"Talent" is a real thing, but it doesn't mean what most people think it does. It's not a magical attribute you either have or don't. It's not an absolute cap on how good you're allowed to get. Talent is two different things:

How much you naturally enjoy doing something (This is most important, and fortunately I think this is fairly easy to re-wire, though it does take effort and requires some environmental factors)
How good you are at improving at something. Think of this as a "talent coefficient."

The Intrinsic Enjoyment/Improvement-Coefficient model of Talent is simplified, but like folk psychology, it's a useful way to make some predictions.[1] If you have a talent coefficient of 1 (average), and you put in 4 hours of practice, you'll get 4 hours of "effective practice." If your talent coefficient is 1.5, you'd get 6 effective hours. If your talent coefficient is .5, you'll get 2 effective hours.

And if you enjoy doing something, you will do ...
[SEQ RERUN] Affective Death Spirals
by MinibearRex — 2011-11-14T05:34:01.657Z

Today's post, Affective Death Spirals was originally published on 02 December 2007. A summary (taken from the LW wiki):

Human beings can fall into a feedback loop around something that they hold dear. Every situation they consider, they use their great idea to explain. Because their great idea explained this situation, it now gains weight. Therefore, they should use it to explain more situations. This loop can continue, until they believe Belgium controls the US banking system, or that they can use an invisible blue spirit force to locate parking spots.

Discuss the post here (rather than in the comments to the original post).
This post is part of the Rerunning the Sequences series, where we'll be going through Eliezer Yudkowsky's old posts in order so that people who are interested can (re-)read and discuss them. The previous post was Mere Messiahs, and you can use the sequence_reruns tag or rss feed to follow the rest of the series.
Sequence reruns are a community-driven effort. You can participate by re-reading the sequence post, discussing it here, posting the next day's sequence reruns post, or summarizing forthcoming articles on the wiki. Go here for more details, or to have meta discussions about the Rerunning the Sequences series.
Meetup : Fort Collins, Colorado Meetup Wedneday 7pm
by EvelynM — 2011-11-14T04:54:36.078Z

Discussion article for the meetup : Fort Collins, Colorado Meetup Wedneday 7pm

WHEN:
16 November 2011 07:00:54PM (-0700)

WHERE:
144 North College Avenue, Fort Collins, CO 80524

Do you want to improve your skills at living a rational life. Come meet people who share the need to improve. Plus, fun.

Discussion article for the meetup : Fort Collins, Colorado Meetup Wedneday 7pm
Drawing Less Wrong: An Introduction
by Raemon — 2011-11-13T22:39:12.519Z

You have not found a way to connect your hobby to rationalism yet. It itches. You are not whole. It is forbidden to post an article entitled Rationalist Hobby on Less Wrong. You lie awake at three in the morning, trying to create puns.

-- Alicorn, Rationality Gothic [2]

(Note from 2017: This sequence was written before Alicorn's classic jab at people shoehorning rationality into random hobbies. I'm fairly embarrassed by it at this point, but don't have time to rewrite this into something I'm not embarrassed by, and several found it a very helpful introduction to drawing.)

((Also, this sequence ends abruptly when my 2011 life got in the way. If you get to the end and go "aarrg! what do I do next!?" the answer is "purchase 'Drawing on the Right Side of the Brain' and finding a figure drawing class))

This post begins a mini-sequence that discusses how to draw, reports on an experiment about teaching people how to draw, and examines how rationality and good drawing practices are related. (As it turns out, a fair amount)

I'm a professional artist. I have a fairly extensive background in traditional drawing, but most of my training is in computer animation. I chose my career because I liked the control offered by the computer - the ability to undo, to manipulate art in procedural ways, and most of all for the flexibility to duplicate things, repurpose them for different projects and combine my love of visual art with my love of game design, animation, and various other mixed media.

But now I work 10+ hours a day at an advertising agency. I spend all day getting paid to stare at a computer screen. Most of my other hobbies also involve staring at a computer screen. And many types of digital work are taxing on the same set of creative muscles, so at the end of the day I didn't have energy to work on the personal projects I wanted.

One option would be to get a different job that didn't tax those creative muscles or involve staring at a screen. I've actually considered getting a "physical" job - after years and thousands of dollars of college to get a nice posh job without physical labor, I actually think it might be better to get PAID to exercise. And instead, use my free time to channel my skills from colleges into personal creative projects that I'm passionate about.

I may do that some day, bu...
Toward an overview analysis of intelligence explosion
by lukeprog — 2011-11-13T22:23:06.443Z

A few months ago, Anna Salamon and I began to write an academic overview of intelligence explosion scenarios — something we could hand to people to explain all our major points in one brief article.

We encountered two major problems.

First: The Summit happened, taking all of our time. Then I was made Executive Director, taking all of my time in a more persistent way.

Second: Being thorough and rigorous in an overview of intelligence explosion requires deep knowledge of a huge spectrum of science and philosophy: history of AI progress, history of planning for the future mattering, AI architectures, hardware progress, algorithms progress, massive datasets, neuroscience, factors in the speed of scientific progress, embryo selection, whole brain emulation, properties of digital minds, AI convergent instrumental values, self-improvement dynamics, takeoff scenarios, heuristics and biases, unipolar and multipolar intelligence explosion scenarios, human values and value extrapolation, decision theory, arms races, human dynamics of technological development, technological forecasting, the economics of machine intelligence, anthropics, evolution, AI-boxing, and much more. Because we were trying to write a short article, we kept having to consume and compress an entire field of knowledge into a single paragraph (or even a single sentence!) with the perfect 2-8 citations, which occasionally meant several days of work for a single paragraph. (This is an extreme example, but it's the kind of problem we often encountered, in different degrees.)

So, we've decided to take a different approach and involve the broader community.

We'll be posting short snippets, short pieces of the puzzle, for feedback from the community. Sometimes we'll pose questions, or ask for references about a given topic, or ask for suggested additions to the dialectic we present.

In the end, we hope to collect and remix the best and most essential snippets, incorporate the feedback and additions provided by the community, and write up the final article.

Think of it as a Polymath Project for intelligence explosion analysis. It's collaborative science and philosophy. Members of Less Wrong tend to be smart, and each one has deep knowledge of one or a few fields that we may not have. We hope you'll join us, and contribute your expertise to this project.

I'll keep a table of conte...
FAI FAQ draft: What is Friendly AI?
by lukeprog — 2011-11-13T19:26:06.395Z

I invite your feedback on this snippet from the forthcoming Friendly AI FAQ. This one is an answer to the question "What is Friendly AI?"

_____

A Friendly AI (FAI) is an artificial intelligence that benefits humanity. More specifically, Friendly AI may refer to:

a very powerful and general AI that acts autonomously in the world to benefit humanity.

an AI that continues to benefit humanity during and after an intelligence explosion.

a research program concerned with the production of such an AI.

Singularity Institute's approach (Yudkowsky 2001, 2004) to designing such an AI:

Goals should be defined by the Coherent Extrapolated Volition of humanity.

Goals should be reliably preserved during recursive self-improvement.

Design should be mathematically rigorous and proof-apt.

Friendly AI is a more difficult project than often supposed. As explored in other sections, commonly suggested solutions for Friendly AI are likely to fail because of two features possessed by any superintelligence:

Superpower: a superintelligent machine will have unprecedented powers to reshape reality, and therefore will achieve its goals with highly efficient methods that confound human expectations and desires.

Literalness: a superintelligent machine will make decisions using the mechanisms it is designed with, not the hopes its designers had in mind when they programmed those mechanisms. It will act only on precise specifications of rules and values, and will do so in ways that need not respect the complexity and subtlety (Kringelbach & Berridge 2009; Schroeder 2004; Glimcher 2010) of what humans value. A demand like "maximize human happiness" sounds simple to us because it contains few words, but philosophers and scientists have failed for centuries to explain exactly what this means, and certainly have not translated it into a form sufficiently rigorous for AI programmers to use.
Learned Helplessness
by Raw_Power — 2011-11-13T16:55:05.103Z

I stumbled upon this interesting little video. It links to many others on the same topic, which I found surprisingly interesting, and would like to discuss. What is it that makes one lose confidence when they see others succeed at the tasks they fail, to the point of being unable to preform tasks that they otherwise should know how to do? Could it be that this phenomenon holds parts of the key to failure rates throughout the education systems worldwide, as well as other self-destructive, irrational reaction-to-failure phenomena throughout one's life?

BTW, maybe we should be able to post Youtube videos on-page?
FAI FAQ draft: What is the history of the Friendly AI concept?
by lukeprog — 2011-11-13T16:47:49.719Z

I invite your feedback on this snippet from the forthcoming Friendly AI FAQ. This one is an answer to the question "What is the history of the Friendly AI concept?"

_____

Late in the Industrial Revolution, Samuel Butler (1863) worried about what might happen when machines become more capable than the humans who designed them:

...we are ourselves creating our own successors; we are daily adding to the beauty and delicacy of their physical organisation; we are daily giving them greater power and supplying by all sorts of ingenious contrivances that self-regulating, self-acting power which will be to them what intellect has been to the human race. In the course of ages we shall find ourselves the inferior race.

...the time will come when the machines will hold the real supremacy over the world and its inhabitants...

This basic idea was picked up by science fiction authors, for example in John W. Campbell’s (1932) short story The Last Evolution. In the story, humans live lives of leisure because machines are smart enough to do all the work. One day, aliens invade:

Then came the Outsiders. Whence they came, neither machine nor man ever learned, save only that they came from beyond the outermost planet, from some other sun. Sirius—Alpha Centauri—perhaps! First a thin scoutline of a hundred great ships, mighty torpedoes of the void [3.5 miles] in length, they came.

Earth’s machines, protecting humans, defeat the aliens. The aliens’ machines survive long enough to render humans extinct, but are eventually defeated by Earth’s machines. These machines inherit the solar system, eventually moving to run on substrates of pure “Force.”

The concerns of machine ethics are most popularly identified with Isaac Asimov’s Three Laws of Robotics, introduced in his short story Runaround. Asimov used his stories, including those collected in the popular I, Robot book, to illustrate all the ways in which such simple rules for governing robot behavior could go wrong.

In the year of I, Robot’s release, mathematician Alan Turing (1950) noted that machines will one day be capable of genuine thought:

I believe that at the end of the century... one will be able to speak of machines thinking without expecting to be contradicted.

Turing (1951/2004) concluded:

...it seems probable that once the machine thinking method has st

...