trentmkelly/LessWrong-43k huggingface dataset:
Muehlhauser-Hibbard Dialogue on AGI
by lukeprog — 2012-07-09T23:11:44.697Z

Part of the Muehlhauser series on AGI.

Luke Muehlhauser is Executive Director of the Singularity Institute, a non-profit research institute studying AGI safety.

Bill Hibbard is an emeritus senior scientist at University of Wisconsin-Madison and the author of Super-Intelligent Machines.

Luke Muehlhauser:

[Apr. 8, 2012]

Bill, I'm glad you agreed to discuss artificial general intelligence (AGI) with me. I hope our dialogue will be informative to many readers, and to us!

On what do we agree? In separate conversations, Ben Goertzel and Pei Wang agreed with me on the following statements (though I've clarified the wording for our conversation):

Involuntary death is bad, and can be avoided with the right technology.

Humans can be enhanced by merging with technology.

Humans are on a risky course in general, because powerful technologies can destroy us, humans often behave not in their own best interests, and we are unlikely to voluntarily halt technological progress.

AGI is likely this century.

AGI will greatly transform the world. It poses existential and other serious risks, but could also be the best thing that ever happens to us if we do it right.

Careful effort will be required to ensure that AGI results in good things rather than bad things for humanity.

You stated in private communication that you agree with these statements, so we have substantial common ground.

I'd be curious to learn what you think about AGI safety. If you agree that AGI is an existential risk that will arrive this century, and if you value humanity, one might expect you to think it's very important that we accelerate AI safety research and decelerate AI capabilities research so that we develop safe superhuman AGI before we develop arbitrary superhuman AGI. (This is what Anna Salamon and I recommend in Intelligence Explosion: Evidence and Import.) What are your thoughts on the matter?

And, which questions would you like to raise?

Bill Hibbard:

[Apr. 11, 2012]

Luke, thanks for the invitation to this dialog and thanks for making my minor edits to your six statements. I agree with them, with the usual reservation that they are predictions and hence uncertain.

Before I answer your other questions it would be useful to set a context by reference to an issue you and Anna raised in your excellent paper, Intelligence Explosion: Evidence and Import. In Section 4.1 you wrote...
How Bayes' theorem is consistent with Solomonoff induction
by Alex_Altair — 2012-07-09T22:16:02.312Z

You've read the introduction to Bayes' theorem. You've read the introduction to Solomonoff induction. Both describe fundamental theories of epistemic rationality. But how do they fit together?

It turns out that it’s pretty simple. Let’s take a look at Bayes’ theorem.

For a review:

H is the particular hypothesis in question.

E is the evidence, or data that we have observed.

P(H) is the “prior” probability of the hypothesis alone.

P(E|H) is the probability of seeing the evidence given that the hypothesis is true.

P(H|E) is the probability that the hypothesis is true, given that you’ve seen the evidence.

Hi is an arbitrary element in the set of all hypotheses.

In terms of Solomonoff induction:

H is the set of all binary sequences which we input to the universal Turing machine.

E is the binary sequence of data that we are given to match.

P(H) is where l(H) is the length of the binary sequence H. This is a basic premise of Solomonoff induction.

P(E|H) is the probability that we will see data sequence E, given that we run program H on the universal Turing machine. Because this is deterministic it is either 1, if H outputs E, or 0, if H does not output E.

P(H|E) is still what we’re looking for. Of course, if H does not output E, then P(E|H) = 0, which means P(H|E) = 0. This makes sense; if a program does not output the data you have, it cannot be the true program which output the data you have.

Hi is an arbitrary binary sequence.

The denominator is the same meaning as the numerator, except as a sum for every possible hypothesis. This essentially normalizes the probability in the numerators. Any hypotheses that do not match the data E exactly will cause P(E|Hi) = 0, and therefore that term will contribute nothing to the sum. If the hypothesis does output E exactly, then P(E|Hi) = 1, and the matching hypothesis contributes its weight to the renormalizing sum in the denominator.

Let's see an example with these things substituted. Here, the set of Hi is the set of hypotheses that match.

In summary; Bayes’ theorem says that once we find all matching hypotheses, we can find their individual probability by dividing their individual weight of by the weights of all the matching hypotheses.

This is intuitive, and matches Bayes’ theorem both mathematically and philosophically. Updating will occur...
Advice On Getting A Software Job
by alyssavance — 2012-07-09T18:52:42.271Z

(Note to LWers: This post was written for a general audience at my blog, but I think it's particularly applicable to Less Wrong, as many here are already interested in programming. Programming is also an important route into two of the main paths to get rich, entrepreneurship and angel investing. Many of the leading donors to the Singularity Institute are professional programmers.)

You’ve already graduated college and found a job, but aren’t making the money you’d like. Or you live in the middle of nowhere. (Or, your job just sucks.) You’re pretty smart, and want to do something about this. What should you do?

One option is working as a computer programmer. Programming has a lot going for it: people tend to enjoy it, software companies have great perks, the work is usually in a laid-back atmosphere, and of course there’s no manual labor. Programming salaries generally range from high five figures (just out of college) to mid six figures (for senior people, and quants at Wall Street banks). This assumes you live in a major city, so be sure to factor that into cost-of-living math. (If you aren’t in a major city, seriously consider moving – most of the best jobs are there.)

Before you apply, you’ll need to learn how to program. To get started, there are lots of books on introductory programming – just search for “Introduction to C”, “Introduction to Python”, “Introduction to Haskell” and stuff like that. It’s good to know at least one language well, and also have experience with a few others, preferably ones that differ in important ways. Once you’ve learned the basics, there are lots of problems online to practice on. If you’re into math, Project Euler has a big, well-curated collection of them. You’ll also want to know your way around Linux, since it’s the main operating system of web servers; try installing it, and using it as your main OS for a few months.

To actually get a programming job, you’ll mainly need to demonstrate a) programming experience, and b) knowledge of computer science. For the first one, the most important thing is to build lots of real software that people use, since this teaches some vital practical skills you won’t learn in books or college classes. Working on open source projects is an excellent way to build...
Value of a Computational Process?
by jefftk — 2012-07-09T17:33:43.323Z

Consider a future person living a happy and fulfilling life. They're unfortunate enough to suffer a severe accident, but there's time to preserve and then scan their brain fully, after which they can be brought back up in an emulator on a computer. [1] It doesn't matter that they're now running on digital hardware instead of in a biological brain; they're still a person and they still count.

Now imagine this person or "em" asks to be let alone, cuts off all communication from the rest of the world, and rejoices privately in finally being able to fully explore their introverted nature. This isn't what I imagine myself doing, but is a choice I can respect.

Someone comes along and suggests turning off this person's emulation on the grounds that no one will know the difference, and we can use the hardware for something else. This seems wrong. Which means this computational process is valuable entirely for its own sake, independent of its effect on the world.

Unlike biological brains, computational processes are very flexible. We could run many copies, or run them much faster or slower than usual. We could run a specific segment of their independent experience repeatedly, perhaps the happiest few moments. It also seems unlikely that a full emulation of a human is the only thing that's valuable. Perhaps there are simpler patterns we could emulate that would be much better in terms of value per dollar?

I'm trying to reduce my concept of value and getting lots of strange questions.

I also posted this on my blog

[1] I think this will be possible, but not for a while.
[link] Cargo Cult Debugging
by MarkL — 2012-07-09T16:05:32.498Z

[...] Here is the right way to address this bug:

Learn more about manifests, so I know what a good one looks like.

Take a look at the one we’re generating for Kiln; see if anything obvious screams out.

If so, dive into the build system [blech] and have it fix up the manifest, or generate a better one, or whatever’s involved here. This part’s a second black box to me, since the Kiln Storage Service is just a py2exe executable, meaning that we might be hitting a bug in py2exe, not our build system.

If not, burn a Microsoft support ticket so I can learn how to get some more debugging info out of the error message.

Here’s the first thing I actually did:

Look at the executable using a dependency checker to see what DLLs it was using, then make sure they were present on Windows 2003.

This is not the behavior of a rational man. [...]

http://bitquabit.com/post/cargo-cult-debugging/
Meetup : Edinburgh summer meetup
by scaphandre — 2012-07-09T13:31:26.337Z

Discussion article for the meetup : Edinburgh summer meetup

WHEN:
13 July 2012 01:00:00PM (+0100)

WHERE:
Upstairs at Biblos, 1 Chambers Street, Edinburgh EH1 1HU

It's the summer months, many of the students are away and we have not had a proper Edinburgh meetup in a while. We will probably get a big table in Biblos on South Bridge on Friday lunchtime and be talking loudly about success or failure of procrastination avoidance plans. All welcome - come say hello.

Discussion article for the meetup : Edinburgh summer meetup
Meetup : Washington DC Calibration Games meetup
by rocurley — 2012-07-09T12:24:42.257Z

Discussion article for the meetup : Washington DC Calibration Games meetup

WHEN:
22 July 2012 03:00:00AM (-0400)

WHERE:
Washington DC, USA

The next meetup will be oxt Sunday, 7/22, at 3 pm, at Ben's apartment. Please check the email list (or PM me) if you need the address. The topic will be calibration games; making guesses with confidence intervals, and seeing if you're over or underconfident.

I will be there, as will Maia. Ben is obviously going to be there as well.

Discussion article for the meetup : Washington DC Calibration Games meetup
Nick Bostrom's TED talk and setting priorities
by ChrisHallquist — 2012-07-09T05:01:50.955Z

I just watched Nick Bostrom's TED talk titled "Humanity's biggest problems aren't what you think they are." I was expecting a talk giving Bostrom's take on what he thought the three biggest existential (or at least catastrophic) risks are, but instead, "existential risk" was just one item on the list. The other two were "death" and "life isn't usually as wonderful as it could be."

Putting these other two in the same category as "existential risk" seems like a mistake. This seems especially obvious in the case of (the present, normal rate of) death and existential risk. Bostrom's talk gives an annual death rate of 56 million, whereas if you take future generations into account, a 1% reduction in existential risk could save 10^32 lives.

More importantly, if we screw up solving "death" and "life isn't usually as wonderful as it could be" in the next century, there will be other centuries where we can solve them. On the other hand, if we screw up existential risk in the next century, it means that's it, humanity's run will be over. There are no second chances when it comes to averting existential risk.

One possible counter argument is that the sooner we solve "death" and "life isn't usually as wonderful as it could be," the sooner we can start spreading our utopia throughout the galaxy and even to other galaxies, and with exponential growth a century head start on that could lead to a manyfold increase in the number of utils in the history of the universe.

However, given the difficulties of building probes that travel at even a significant fraction of the speed of light, and the fact that colonizing new star systems may be a slow process even with advanced nanotech, a century may not matter much when it comes to colonizing the galaxy. Furthermore, colonizing the galaxy (or universe) may not be the sort of thing that follows an exponential curve, it may follow a cubic curve as probes spread out in a sphere.

So I lean towards thinking that averting existential risks should be a much higher priority than creating a death-free, always wonderful utopia. Or maybe not. Either way, the answer would seem to be very important for questions like how we should focus our resources, and also whether your should push the button to turn on a machine that will allegedly create a utopia.
[SEQ RERUN] Fake Norms, or "Truth" vs. Truth
by MinibearRex — 2012-07-09T03:37:18.421Z

Today's post, Fake Norms, or "Truth" vs. Truth was originally published on 22 July 2008. A summary (taken from the LW wiki):

Our society has a moral norm for applauding "truth", but actual truths get much less applause (this is a bad thing).

Discuss the post here (rather than in the comments to the original post).
This post is part of the Rerunning the Sequences series, where we'll be going through Eliezer Yudkowsky's old posts in order so that people who are interested can (re-)read and discuss them. The previous post was Should We Ban Physics?, and you can use the sequence_reruns tag or rss feed to follow the rest of the series.
Sequence reruns are a community-driven effort. You can participate by re-reading the sequence post, discussing it here, posting the next day's sequence reruns post, or summarizing forthcoming articles on the wiki. Go here for more details, or to have meta discussions about the Rerunning the Sequences series.
Rationality Games & Apps Brainstorming
by lukeprog — 2012-07-09T03:04:44.339Z

Last month, mobile gaming superstar Angry Birds was out-sold in some countries by DragonBox, a kids game in which players solve alegbra equations.

How does the game work? Jonathan Liu explains:

There are five “worlds,” each with twenty levels, and as you progress through the levels the “dragons” hatch and grow into their full-sized versions. While this in itself has nothing to do with algebra, I mention this because my kids love this. It’s a very tiny incentive (along with earning stars) but they really want to beat the next level to watch the dragon grow into its next form. I was told that the dragons were all drawn by a fourteen-year-old girl, and they’re a lot of fun. (They aren’t all typical dragons — One starts off more like a fish, one looks like a squid, and so on.)

You are presented with a big screen with two trays, each containing a number of “cards” with different images on it. Somewhere on the screen there will be a little box with a star on it, sparkling and glowing. The app gives very minimal instructions in a hand-written font with arrows pointing to relevant spots on the screen, but it tells you to get the box by itself. At first you do this simply by tapping the green spirally cards, which vanish when you tap them. Then, you’ll start to get some “night” versions of cards — drag these onto the “day” versions and they become green swirls, which you already know how to handle.

After you’ve gotten past several levels of moving cards around and tapping on swirls, you’ll get a few cards down at the bottom which you can drag onto the trays — but whenever you drag a card onto one side, you have to also drag a copy to the other side as well. (This, of course, simulates adding the same number to both sides.) And then, a few levels on, you learn that you can flip these extra cards from day to night (and vice versa) before dragging them onto the trays.

As the game progresses, you’ll start seeing cards that are above and below each other, with a bar in the middle — and you’ll learn to cancel these out by dragging one onto the other, which then turns into a one-dot. And you’ll learn that a one-dot vanishes when you drag it onto a card it’s attached to (with a little grey dot between them). These, of course,

...
Meetup : Fort Collins, Colorado Meetup Thursday 7pm
by EvelynM — 2012-07-08T16:03:14.779Z

Discussion article for the meetup : Fort Collins, Colorado Meetup Thursday 7pm

WHEN:
12 July 2012 07:00:00PM (-0600)

WHERE:
144 North College Avenue, Fort Collins, CO 80524

Freshly back from Less Wrong Central, copt and EvelynM have new games and stories of wizardry in the West.

Discussion article for the meetup : Fort Collins, Colorado Meetup Thursday 7pm
Morality open thread
by Will_Newsome — 2012-07-08T14:30:08.864Z

I figure morality as a topic is popular enough and important enough and related-to-rationality enough to deserve its own thread.

Questions, comments, rants, links, whatever are all welcome. If you're like me you've probably been aching to share your ten paragraph take on meta-ethics or whatever for about three uncountable eons now. Here's your chance.

I recommend reading Wikipedia's article on meta-ethics before jumping into the fray, if only to get familiar with the standard terminology. The standard terminology is often abused. This makes some people sad. Please don't make those people sad.
[SEQ RERUN] Should We Ban Physics?
by MinibearRex — 2012-07-08T06:50:48.724Z

Today's post, Should We Ban Physics? was originally published on 21 July 2008. A summary (taken from the LW wiki):

There is a chance, however remote, that novel physics experiments could destroy the earth. Is banning physics experiments a good idea?

Discuss the post here (rather than in the comments to the original post).
This post is part of the Rerunning the Sequences series, where we'll be going through Eliezer Yudkowsky's old posts in order so that people who are interested can (re-)read and discuss them. The previous post was Existential Angst Factory, and you can use the sequence_reruns tag or rss feed to follow the rest of the series.
Sequence reruns are a community-driven effort. You can participate by re-reading the sequence post, discussing it here, posting the next day's sequence reruns post, or summarizing forthcoming articles on the wiki. Go here for more details, or to have meta discussions about the Rerunning the Sequences series.
Links about naturalism
by lukeprog — 2012-07-08T05:39:12.159Z

I'm about to launch a new website (as a personal project; it's got nothing to do with SI or CFAR) about naturalistic views of the world, and that website will contain links to engaging and accessible books and articles about naturalistic approaches to:

seeking truth

the self & free will

ethics & society

happiness & self-help

meaning & spirituality

What links do other LWers recommend on these topics? To show you what kinds of articles I have in mind, here's what I have so far:

Seeking Truth:

Yudkowsky, Map and Territory

Yudkowsky, How to Actually Change Your Mind

Yudkowsky, A Technical Explanation of Technical Explanation

Baron, Thinking and Deciding

Clark, Reality and Its Rivals: Putting Epistemology First

The Self & Free Will:

Clark, Fully Cause: Coming to Terms with Determinism

Yudkowsky, Free Will (Solution)

Thagard, The Brain and the Meaning of Life

Harris, Free Will

Ethics and Society:

Muehlhauser, No-Nonsense Metaethics

Rachels, The Elements of Moral Philosophy

Alexander, Efficient Charity: Do Unto Others...

Happiness and Self-Help:

Muehlhauser, The Science of Winning at Life

Wiseman, 59 Seconds

Steel, The Procrastination Equation

Dixit & Nalebuff, The Art of Strategy

Meaning and Spirituality:

Thagard, The Brain and the Meaning of Life

Clark, Spirituality Without Faith

Comte-Sponville, The Little Book of Atheist Spirituality
To Learn Critical Thinking, Study Critical Thinking
by gwern — 2012-07-07T23:50:35.665Z

Critical thinking courses may increase students’ rationality, especially if they do argument mapping.

The following excerpts are from “Does philosophy improve critical thinking skills?”, Ortiz 2007.

1 Excerpts

This thesis makes a first attempt to subject the assumption that studying [Anglo-American analytic] philosophy improves critical thinking skills to rigorous investigation.

…Thus the second task, in Chapter 3, is to articulate and critically examine the standard arguments that are raised in support of the assumption (or rather, would be raised if philosophers were in the habit of providing support for the assumption). These arguments are found to be too weak to establish the truth of the assumption. The failure of the standard arguments leaves open the question of whether the assumption is in fact true. The thesis argues at this point that, since the assumption is making an empirical assertion, it should be investigated using standard empirical techniques as developed in the social sciences. In Chapter 4, I conduct an informal review of the empirical literature. The review finds that evidence from the existing empirical literature is inconclusive. Chapter 5 presents the empirical core of the thesis. I use the technique of meta-analysis to integrate data from a large number of empirical studies. This meta-analysis gives us the best yet fix on the extent to which critical thinking skills improve over a semester of studying philosophy, general university study, and studying critical thinking. The meta-analysis results indicate that students do improve while studying philosophy, and apparently more so than general university students, though we cannot be very confident that this difference is not just the result of random variation. More importantly, studying philosophy is less effective than studying critical thinking, regardless of whether one is being taught in a philosophy department or in some other department. Finally, studying philosophy is much less effective than studying critical thinking using techniques known to be particularly effective such as LAMP.

…To establish whether and how improvement is in fact occurring in students’ CTS, we need ways of measuring such improvement. In other words, we have to be able to operationalize CTS . Several attempts to devise such measures have been made. They include common

...