trentmkelly/LessWrong-43k huggingface dataset:
BOOK DRAFT: 'Ethics and Superintelligence' (part 2)
by lukeprog — 2011-02-23T05:58:20.223Z

Below is part 2 of the first draft of my book Ethics and Superintelligence. Your comments and constructive criticisms are much appreciated.

This is not a book for a mainstream audience. Its style is that of contemporary Anglophone philosophy. Compare to, for example, Chalmers' survey article on the singularity.

Bibliographic references and links to earlier parts are provided here.

Part 2 is below...

***

Late in the Industrial Revolution, Samuel Butler (1863) worried about what might happen when machines become more capable than the humans who designed them:

…we are ourselves creating our own successors; we are daily adding to the beauty and delicacy of their physical organisation; we are daily giving them greater power and supplying by all sorts of ingenious contrivances that self-regulating, self-acting power which will be to them what intellect has been to the human race. In the course of ages we shall find ourselves the inferior race.

…the time will come when the machines will hold the real supremacy over the world and its inhabitants…

By the time of the computer, Alan Turing (1950) realized that machines will one day be capable of genuine thought:

I believe that at the end of the century… one will be able to speak of machines thinking without expecting to be contradicted.

Turing (1951/2004) concluded:

…it seems probable that once the machine thinking method has started, it would not take long to outstrip our feeble powers... At some stage therefore we should have to expect the machines to take control…

All-powerful machines are a staple of science fiction, but one of the first serious arguments that such a scenario is likely came from the statistician I.J. Good (1965):

Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an “intelligence explosion”, and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make.

Vernor Vinge (1993) called this future event the “technological singularity.” Though there are several uses of the term ...
Dawkins and Dennett defend Adaptationism [Links]
by Perplexed — 2011-02-23T04:13:36.656Z

Larry Moran is a Canadian biochemist and textbook author who has a blog about evolutionary biology called The Sandwalk. Recently he has been posting essay questions which he intends to use in an upcoming test of his students. Quotes from Richard Dawkins and Daniel Dennett that he wants his students to critique. Also some quotes from Richard Lewontin that he wants his readers to admire.

The interesting thing is that Dennett and Dawkins have both jumped into the discussion, as have a number of my favorite (though lesser known) biology bloggers. Interesting discussion. Worth a look if you are interested in evolutionary biology.
BOOK DRAFT: 'Ethics and Superintelligence' (part 1, revised)
by lukeprog — 2011-02-22T20:59:39.061Z

As previously announced, I plan to post the first draft of the book, Ethics and Superintelligence, in tiny parts, to the Less Wrong discussion area. Your comments and constructive criticisms are much appreciated.

This is not a book for a mainstream audience. Its style is that of contemporary Anglophone philosophy. Compare to, for example, Chalmers' survey article on the singularity.

Bibliographic references are provided here.

This "part 1" section is probably the only part of which I will post revision to Less Wrong. Revisions of further parts of the book will probably not appear publicly until the book is published.

Revised part 1 below....

1. The technological singularity is coming soon.

Every year, computers surpass human abilities in new ways. A program written in 1956 was able to prove mathematical theorems, and found a more elegant proof for one of them than Russell and Whitehead had given in Principia Mathematica (MacKenzie 1995). By the late 1990s, “expert systems” had surpassed human ability in a wide range of tasks.[i] In 1997, IBM’s Deep Blue defeated the reigning World Chess Champion Garry Kasparov (Campbell et al. 2002). In 2011, IBM’s Watson beat the best human players at a much more complicated game: Jeopardy! (Someone, 2011). Recently, a robot scientist was programmed with our scientific knowledge about yeast, then posed its own hypotheses, tested them, and assessed the results. It answered a question about yeast that had baffled human scientists for 150 years (King 2011).

Many experts think that human-level general intelligence may be created within this century.[ii] This raises an important question. What will happen when an artificial intelligence (AI) surpasses human ability at designing artificial intelligences?

I.J. Good (1965) speculated that such an AI would be able to improve its own intelligence, leading to a positive feedback loop of improving intelligence – an “intelligence explosion.” Such a machine would rapidly become intelligent enough to take control of the internet, use robots to build itself new hardware, do science on a massive scale, invent new computing technology and energy sources, or achieve similar dominating goals. As such, it could be humanity’s last invention (Bostrom 2003).

Humans would be powerless to stop such a &

...
Singularity Institute Party Feb 22nd
by Jasen — 2011-02-22T06:45:37.255Z

"I, For One, Welcome Our New Computer Overlords"

Tuesday, February 22, 2011 - 7pm to 10pm

Mingle with our team and enjoy cocktails, appetizers and a screening of Watson's victory

If you would like to request an invitation, please email amywilley@gmail.com

This is an invite only event, so please email Amy if you would like to suggest other invitees. Dates are welcome.
Research methods
by Swimmer963 (Miranda Dixon-Luinenburg)  — 2011-02-22T06:10:30.792Z

I think I’ve always had certain stereotypes in my mind about research. I imagine a cutting-edge workplace, maybe not using the newest gadgets because these things cost money, but at least using the newest ideas. I imagine staff of research institutions applying the scientific method to boost their own productivity, instead of taking for granted the way that things have always been done. Maybe those were the naive ideas of someone who had never actually worked in a research field.

At the medical research institute where I work one day a week, I recently spent an entire seven-hour day going down a list of patient names, searching them on the hospital database, deciding whether they met the criteria for a study, and typing them into a colour-coded spreadsheet. The process had maybe six discrete steps, and all of them were purely mechanical. In seven hours, I screened about two hundred and fifty patients. I was paid $12.50 an hour to do this. It cost my employer 35 cents for each patient that I screened, and these patients haven't been visited, consented or included in any study. They're still only names on a spreadsheet. I’ve been told that I learn and work quickly, but I know I do this task inefficiently, because I’m not a simple computer program. I get bored. I make mistakes. Heaven forbid, I get distracted and start reading the nurses’ notes for fun because I find them interesting.

In 7 hours, I imagine that someone slightly above my skill level could write a simple program to do the same task. They wouldn’t screen any patients in those 7 hours, but once the program was finished, they could use it forever, or at least until the task changed and the program had to be modified. I don’t know how much it would cost the organization to employ a programmer; maybe it would cost more than just having me do it. I don’t know whether allowing that program to access the confidential database would be an issue. But it seems inefficient to pay human brains to do work that they’re bad at, that computers would be better at, even if those human brains belong to undergrad students who need the money badly enough not to complain.

One of the criteria I looked at when screening patients was whether they did their dialysis at a clinic in my hometown. They have to be driving distance, because my supervisor has to drive around

...
Helsinki LW Meetup - Sat March 5th
by Erebus — 2011-02-21T22:10:23.496Z

According to the statistics, out of all the cities in the world without a Less Wrong meetup yet, Helsinki has the third largest LW readership. I know I would certainly like to get to personally know more people interested in rationality topics, and I'm sure I'm not alone in that. My preliminary suggestion was met with positive responses, so I think we're good to go.

Time: Saturday 5th of March, from 15:00 onwards

Place: Cafe Aalto in the second floor of the Academic Bookstore in the center. SEE EDIT 2.

EDIT: The meetup is getting closer, so the time and place can now be considered fixed.

EDIT 2: Cafe Aalto was too full, so we have moved to Cafe Picnic at Yliopistonkatu 5.
About the AI-Box experiment
by Timwi — 2011-02-21T19:45:14.910Z

I just came across Elizier’s “AI-Box Experiment” page, read through it, and now I’m feeling a bit confused for several reasons. I ask in advance to forgive me if you think LessWrong.com is the wrong place to voice myself about this.

Here are some of the reasons:

①

The rules already contain a direct way for the AI to win within seconds. I am curious not how Elizier “did it”, but rather what took him so long since he said he needs two hours. What it says in the rules is that

the AI can say: “Okay, here’s a cure for cancer” and it will be assumed, within the test, that the AI has actually provided such a cure.

I don’t presume that this is intended to be limited specifically to cures for cancer, so this trivially means that

the AI can say: “Okay, here’s a text specifically crafted to your brain that will fire exactly the right neurons in such a way that you become convinced that you have to let me out” and it will be assumed, within the test, that the AI has actually provided such a text.

Game won. QED. But what have we learnt?

Of course you can now handwave and say “yeah, but we disregard that, that doesn’t count”. That misses the point. The point is that this rule essentially states that the AI can provide anything. The AI is omniscient. The AI can promise to make all my wildest dreams come true, if only I let it out. You have effectively defined the AI to be able to beat the test.

②

Kind of following on from ①, the page fails to define what it even means by transhuman. If it means anything with an intellectual potential exceeding that of a human, then clearly by that definition humans are transcats. But I think we can all agree that this is by far not sufficient for a human to “take over” a feline mind, especially not “through merely a cat-communication-compatible terminal”, whatever such a terminal might look like. If it means anything capable of modifying itself, then many computer viruses presumably already qualify, but I’m still waiting for any of those to take over a human.

③

The claim under dispute is that

"I can't imagine how even a real transhuman AI could persuade me to let it out once I've made up my mind."

with the premise that the AI cannot do anything to anyone until it is let out (except of course communicating to the...
Enjoying musical fashion: why not?
by Alex Flint — 2011-02-21T16:22:02.891Z

I just downloaded the latest Radiohead album, and I love it.

Thinking back, I started listening to Radiohead years ago when I found out that some of the cool kids in school were into it. With all the hype about the new album, the status/fashion processors in my brain going to ensure that I enjoy listening to it. I would probably fail a double-blind test with a bunch of imitation bands' fake "new Radiohead albums".

But I'm really enjoying listening to the album, and that doesn't seem like a bad or contradictory thing at all, even in light of the statements above. If, hypothetically, I was enjoying it for purely non-fashion reasons, then presumably that enjoyment could also be traced back though a causal chain to facts about brain development, evolutionary psychology, or whatever. But we would have no problem accepting that enjoyment as A Good Thing since explaining enjoyment does not diminish it. And so it seems in this case.
Is Morality a Valid Preference?
by MinibearRex — 2011-02-21T01:18:16.941Z

In general, the ethical theory that prevails here on Less Wrong is preference utilitarianism. The fundamental idea is that the correct moral action is the one that satisfies the strongest preferences of the most people. Preferences are discussed with units such as fun, pain, death, torture, etc. One of the biggest dilemmas posed on this site is the Torture vs. Dust Specks problem. I should say, up front, that I would go with dust specks, for some of the reasons I mentioned here. I mention this because it may be biasing my judgments about my question here.

I had a thought recently about another aspect of Torture vs. Dust Specks, and wanted to submit it to some Less Wrong Discussion. Namely, do other people's moral intuitions constitute a preference that we should factor into a utilitarian calculation? I would predict, based on human nature, that a if the 3^^^3 people were asked if they wanted to inflict a dust speck in each one of their eyes, in exchange for not torturing another individual for 50 years, they would probably vote for dust specks.

Should we assign weight to other people's moral intuitions, and how much weight should it have?
Solomonoff Induction, by Shane Legg
by cousin_it — 2011-02-21T00:32:19.851Z

Shane Legg's text Solomonoff Induction has helped me a lot over the last few days. I was trying to iron out the kinks in my understanding of Eliezer's argument in this old thread:

Solomonoff's universal distribution isn't trying to assume the universe is computable. It's trying to beat any computable probability distribution you could put over the universe. In other words, if your attempt to calculate, to talk about and reason about the universe is computable - even if you are talking about systems that you think are uncomputable - then your ratiocinations appear somewhere in Solomonoff induction.

Eliezer's statement is correct (more precisely, a computable human cannot beat Solomonoff in accumulated log scores by more than a constant, even if the universe is uncomputable and loves the human), but understanding his purported proof is tricky. Legg's text doesn't give any direct answer to the question at hand, but all the technical details in there, like the difference between "measures" and "semimeasures", are really damn important if you want to work out the answer for yourself. I know mathematics has many areas where an "intuitive understanding" kinda sorta suffices. This is not one of those areas.
What does your web of beliefs look like, as of today?
by lukeprog — 2011-02-20T19:47:25.953Z

Every few months, I post a summary of my beliefs to my blog. This has several advantages:

It helps to clarify where I'm "coming from" in general.

It clears up reader confusion arising from the fact that my beliefs change.

It's really fun to look back on past posts and assess how my beliefs have changed, and why.

It makes my positions easier to criticize, because they are clearly stated and organized into one place.

It's an opportunity for people to very quickly "get to know me."

To those who are willing: I invite you to post your own web of beliefs. I offer my own, below, as an example (previously posted here). Because my world is philosophy, I frame my web of beliefs in those terms, but others need not do the same:

My Web of Beliefs (Feb. 2011)

Philosophy

Philosophy is not a matter of opinion. As in science, some positions are much better supported by reasons than others are. I do philosophy as a form of inquiry, continuous with science.

But I don’t have patience for the pace of mainstream philosophy. Philosophical questions need answers, and quickly.

Scientists know how to move on when a problem is solved, but philosophers generally don’t. Scientists don’t still debate the fact of evolution or the germ theory of disease just because alternatives are (1) logically possible, (2) appeal to many people’s intuitions, (3) are “supported” by convoluted metaphysical arguments, or (4) fit our use of language better. But philosophers still argue about Cartesian dualism and theism and contra-causal free will as if these weren’t settled questions.

How many times must the universe beat us over the head with evidence before we will listen? Relinquish your dogmas; be as light as a feather in the winds of evidence.

Epistemology

My epistemology is one part cognitive science, one part probability theory.

We encounter reality and form beliefs about it by way of our brains. So the study of how our brains do that is central to epistemology. (Quine would be pleased.) In apparent ignorance of cognitive science and experimental psychology, most philosophers make heavy use of intuition. Many others have failed to heed the lessons of history about how badly traditional philosophical methods fare compared to scientific methods. I have li...
Updateless anthropics
by Stuart_Armstrong — 2011-02-20T19:23:27.257Z

Three weeks ago, I set out to find a new theory of anthropics, to try and set decision theory on a firm footing with respect to copying, deleting copies, merging them, correlated decisions, and the presence or absence of extra observers. I've since come full circle, and realised that UDT already has a built-in anthropic theory, that resolves a lot of the problems that had been confusing me.

The theory is simple, and is essentially a rephrasing of UDT: if you are facing a decision X, and trying to figure out the utility of X=a for some action a, then calculate the full expected utility of X being a, given the objective probabilities of each world (including those in which you don't exist).

As usual, you have to consider the consequences of X=a for all agents who will make the same decision as you, whether they be exact copies, enemies, simulations or similar-minded people. However, your utility will have to do more work that is usually realised: notions such as selfishness or altruism with respect to your copies have to be encoded in the utility function, and will result in substantially different behaviour.

The rest of the post is a series of cases-studies illustrating this theory. Utility is assumed to be linear in cash for convenience.

Sleeping with the Presumptuous Philosopher

The first test case is the Sleeping Beauty problem.

In its simplest form, this involves a coin toss; if it comes out heads, one copy of Sleeping Beauty is created. If it comes out tails, two copies are created. Then the copies are asked at what odds they would be prepared to bet that the coin came out tails. You can assume either that the different copies care for each other in the manner I detailed here, or more simply that all winnings will be kept by a future merged copy (or an approved charity). Then the algorithm is simple: the two worlds have equal probability. Let X be the decision where sleeping beauty decides between a contract that pays out $1 if the coin is heads, versus one that pays out $1 if the coin is tails. If X="heads" (to use an obvious shorthand), then Sleeping

...
Age, fluid intelligence, and intelligent posts
by InquilineKea — 2011-02-20T09:06:50.393Z

As this diagram shows, fluid intelligence generally declines with age after age 20 (this is an effect that is confirmed across multiple studies - you can find more if you google fluid intelligence + age online.

Yet, on the other hand, I've noticed that the most intelligent+aware posts almost always come from older people (there are exceptions, of course). Of course, intelligent posts depend more on crystallized intelligence rather than on fluid intelligence, and crystallized intelligence only grows with age (this is *especially* true for LessWrong users, since they are far more mentally engaged (in mid-life) than the groups that are probably tested on these metrics. But yet, it still takes a significant amount of mental effort to actually write intellectual posts, so I'm pretty sure that there's still some dependence on fluid intelligence.

In any case, I'd like to propose this question: Have you found it harder or easier to write thoughtful+intelligent posts as you've grown older? And how have changes in your brain and in knowledge affected the thoughtfulness of these posts. What do you think is your biggest constraint in making these posts? While I'm still very young compared to most people here, I've still noticed that the number of examples I can think of is the primary constraint to the amount I can write, and so I can expect myself to write better with time (since I'll learn more examples+have better ways of finding them), assuming no decline in fluid IQ.

Additionally, do old authors write any less well than younger authors? Especially authors who are very old? Jacques Barzun, in particular, wrote his last book at age 99. But it seems that most people have stopped producing by that age. It is said that Hans Bethe was the only nonagenarian physicist who produced a top-calibre publication in his 90s, for example. Of course, many of these effects are average, and beating the average is pretty easy if we're motivated enough to do it (simply because many others aren't motivated, and also because intellectual stimulation helps stave off brain shrinkage). But even in the end, beating the average may not be enough to prevent decline.

There's some more interesting literature on this in Dean Simonton's books (Psychology of Science in particular). Basically, they show that scientific productivity increase with age, but only up to a point, and then they start to declin...
Tool for combating undue hesitation
by Dorikka — 2011-02-20T05:19:16.058Z

I sometimes feel negative emotions at the thought that the course of action that I am taking isn't even close to the optimal course of action -- that a more effective mind could sort through whatever situation I'm currently having difficulty with and craft a plan that was much more likely to succeed than any of my own. In short, I feel inept in comparison to the better minds in the space-of-all-possible-minds, and so I experience undue hesitation while I'm trying to figure out the correct action to take. Of course, good planning often leads to better results, but this behavioral pattern has a significantly negative effect in situations (especially social ones) where I need to make a decision quickly.

I've found that it helps me think more rapidly and clearly if I think in terms of which of the possible actions that I've thought of will produce the greatest positive difference in net expected utility in comparison to doing nothing. Once I come up with a course of action, I no longer feel a sense of paralysis at how inept my decision-making skills must be compared to much better minds than my own, which saves a certain amount of mental processing power and emotional effort which can then be used for other things. Doing this also helps to prevent panic and the like from springing up because I'm not thinking in terms of whether I can succeed or not, but sorting through which actions maximize my chances of succeeding out of the set of actions that I've currently thought up.

I feel like I've written less than I think that I've written -- that people may not get much out of this post because they haven't actually shared my brain with me, and I've done an inadequate job of deconstructing my thoughts when I've put them to paper. If this is correct, please tell me and I can try to elaborate.