trentmkelly/LessWrong-43k huggingface dataset:
Empirical claims, preference claims, and attitude claims
by John_Maxwell — 2012-11-15T19:41:02.955Z

What do the following statements have in common?

"Atlas Shrugged is the best book ever written."

"You break it, you buy it."

"Earth is the most interesting planet in the solar system."

My answer: None of them are falsifiable claims about the nature of reality. They're all closer to what one might call "opinions". But what is an "opinion", exactly?

There's already been some discussion on Less Wrong about what exactly it means for a claim to be meaningful. This post focuses on the negative definition of meaning: what sort of statements do people make where the primary content of the statement is non-empirical? The idea here is similar to the idea behind anti-virus software: Even if you can't rigorously describe what programs are safe to run on your computer, there still may be utility in keeping a database of programs that are known to be unsafe.

Why is it useful to be able to be able to flag non-empirical claims? Well, for one thing, you can believe whatever you want about them! And it seems likely that this pattern-matching approach works better for flagging them than a more constructive definition.

But first, a bit on the philosophy of non-empirical claims.

Let's take a typical opinion statement: "Justin Bieber sucks". There are a few ways we could interpret this as shorthand for a different claim. For example, maybe what the speaker really means is "I prefer not to listen to Justin Bieber's music." (Preference claim.) Or maybe what the speaker really means is "Of the people who have heard songs by Justin Bieber, the majority prefer not to listen to his music." (Empirical claim.)

I don't think shorthand interpretations like these are accurate for most people who claim that JB sucks. Instead, I suspect most people who argue this are communicating some combination of (a) negative affect towards JB and (b) tribal affiliation with fellow JB haters. I've taken to referring to statements like these, that are neither preference claims nor empirical claims, as "attitude claims".

This example doesn't mean that all "X sucks" style claims are attitude claims. Take the claim "Windows sucks". It does seem plausible that someone who said this could be persuaded that their claim was false through empirical evidence--e.g. by a meta-analysis that compared Windows worker productivity favora...
A summary of the Hanson-Yudkowsky FOOM debate
by Kaj_Sotala — 2012-11-15T07:25:54.739Z

In late spring this year, Luke tasked me with writing a summary and analysis of the Hanson-Yudkowsky FOOM debate, with the intention of having it eventually published in somewhere. Due to other priorities, this project was put on hold for the time being. Because it doesn't look like it will be finished in the near future, and because Curiouskid asked to see it, we thought that we might as well share the thing.

I have reorganized the debate, presenting it by topic rather than in chronological order: I start by providing some brief conceptual background that's useful for understanding Eliezer's optimization power argument, after which I present his argument. Robin's various objections follow, after which there is a summary of Robin's view of how the Singularity will be like, together with Eliezer's objections to that view. Hopefully, this should make the debate easier to follow. This summary also incorporates material from the 90-minute live debate on the topic that they had in 2011. The full table of contents:

Introduction

Overview

The optimization power argument

Conceptual background

The argument: Yudkowsky

Recursive self-improvement

Hard takeoff

Questioning optimization power: the question of abstractions

Questioning optimization power: the historical record

Questioning optimization power: the UberTool question

Hanson's Singularity scenario

Architecture vs. content, sharing of information

Modularity of knowledge

Local or global singularity?

Wrap-up

Conclusions

References

Here's the link to the current draft, any feedback is welcomed. Feel free to comment if you know of useful references, if you think I've misinterpreted something that was said, or if you think there's any other problem. I'd also be curious to hear to what extent people think that this outline is easier to follow than the original debate, or whether it's just as confusing.
Launched: Friendship is Optimal
by iceman — 2012-11-15T04:57:47.924Z

Friendship is Optimal has launched and is being published in chunks on FIMFiction.

Friendship is Optimal is a story about an optimizer written to "satisfy human values through friendship and ponies." I would like to thank everyone on LessWrong who came out and helped edit it. Friendship is Optimal wouldn't be what is today without your help.

Thank you.

Teaser description:

Hanna, the CEO of Hofvarpnir Studios, just won the contract to write the official My Little Pony MMO. Hanna has built an A.I. Princess Celestia and given her one basic drive: to satisfy everybody's values through friendship and ponies. And Princess Celestia will follow those instructions to the letter...even if you don't want her to.

Here is the schedule for the next chapters:

Friday (Nov. 16th): Chapter 4 - 5

Monday (Nov. 19th): Chapter 6 - 7

Thursday (Nov. 22th): Chapter 8 - 9

Sunday (Nov. 25th): Chapter 10 - 11, Author's Afterword
[SEQ RERUN] Billion Dollar Bots
by MinibearRex — 2012-11-15T04:43:14.280Z

Today's post, Billion Dollar Bots was originally published on November 22, 2008. A summary:

An alternate scenario for the creation of bots, this time involving lots of cloud computing.

Discuss the post here (rather than in the comments to the original post).
This post is part of the Rerunning the Sequences series, where we'll be going through Eliezer Yudkowsky's old posts in order so that people who are interested can (re-)read and discuss them. The previous post was Brain Emulation and Hard Takeoff, and you can use the sequence_reruns tag or rss feed to follow the rest of the series.
Sequence reruns are a community-driven effort. You can participate by re-reading the sequence post, discussing it here, posting the next day's sequence reruns post, or summarizing forthcoming articles on the wiki. Go here for more details, or to have meta discussions about the Rerunning the Sequences series.
Meetup : Salt Lake City Monthly meetup
by hamnox — 2012-11-15T00:46:31.702Z

Discussion article for the meetup : Salt Lake City Monthly meetup

WHEN:
17 November 2012 03:00:28PM (-0700)

WHERE:
Calvin S. Smith Library 810 East 3300 South Salt Lake City, Utah 84106

We'll be in the small conference room at the back, and might go out for coffee or whatnot afterwards. There's Karaoke going on later too.

Agenda:
3:00 Greetings, Introduction and Catchup
3:10 Thinking Fast & Slow
3:30 discuss
3:40 Expected Value of Information
4:00 discuss
4:10 Group Goals
4:30 discuss
4:40 Wrap up and Final Thoughts, Sign up for next meetup
Free Discussion

For anyone who can't make this one, know that there's one scheduled every third Saturday of the month, tentatively the same time and place, for most of the conceivable future. Our google group is here: https://groups.google.com/group/lesswrongslc/

Discussion article for the meetup : Salt Lake City Monthly meetup
Meetup : Cincinnati/Columbus: Memorisation exercise (NB, time changed)
by RolfAndreassen — 2012-11-14T20:38:32.516Z

Discussion article for the meetup : Cincinnati/Columbus: Memorisation exercise (NB, time changed)

WHEN:
18 November 2012 02:00:00PM (-0500)

WHERE:
4934 Juniper Way Beavercreek, OH 45440

Please notice changed time!

We will meet at Choe's Asian Gourmet in the Greene. Our exercise for this month is rote memorisation: Please come prepared with a poem of reasonable length, memorised this week, to recite for the group and then discuss. Bookstore and ice cream may occur later.

Discussion article for the meetup : Cincinnati/Columbus: Memorisation exercise (NB, time changed)
Meetup : Durham HPMoR Discussion, chapters 15-17
by evand — 2012-11-14T17:52:49.431Z

Discussion article for the meetup : Durham HPMoR Discussion, chapters 15-17

WHEN:
17 November 2012 11:00:00AM (-0500)

WHERE:
Foster's Market, 2694 Durham-Chapel Hill Blvd., Durham, NC

We will meet and discuss HPMoR, chapters 15-17 (approx 55 pages). Main discussion will probably last until 12:30 or 1:00, and there will likely be Zendo afterwards.

Discussion article for the meetup : Durham HPMoR Discussion, chapters 15-17
[LINK] Steven Landsburg "Accounting for Numbers" - response to EY's "Logical Pinpointing"
by David_Gerard — 2012-11-14T12:55:16.910Z

"I started to post a comment, but it got long enough that I’ve turned my comment into a blog post."

So the study of second-order consequences is not logic at all; to tease out all the second-order consequences of your second-order axioms, you need to confront not just the forms of sentences but their meanings. In other words, you have to understand meanings before you can carry out the operation of inference. But Yudkowsky is trying to derive meaning from the operation of inference, which won’t work because in second-order logic, meaning comes first.

... it’s important to recognize that Yudkowsky has “solved” the problem of accounting for numbers only by reducing it to the problem of accounting for sets — except that he hasn’t even done that, because his reduction relies on pretending that second order logic is logic.
Instrumental rationality for overcoming disability and lifestyle failure (a specific case)
by CAE_Jones — 2012-11-14T08:46:44.807Z

I read things like "Rationality should win!", and I feel validated. That's basically what I've believed as far back as I can remember (and though my memory isn't as reliable as I'd like, I do think it is pretty decent, based on comparisons I've made with video evidence. Not that said video evidence includes much of me talking about my beliefs.).

Yet, clearly, I am no master of rationality, given that my situation at present is not what I would define as having won. When my father was my age, he was employed, married (to his second wife, with whom he is still married), and I was three years old. It wasn't long after that when my sister was born and my dad started work for his father-in-law, whose business he eventually came to own and still runs for a viable profit today. He did set some goals for himself that he hasn't quite met--gaining enough self-sustaining wealth to retire at age 45 (he turned 45 in February 2012) and/or spend his days lounging at the beach without negative consequences--but considering how much money he's put into vacations to the beach, and that he's still able to support his business and family (my stepmother makes a non-negligible contribution to this), I'd call him much closer to successful than me.

So, clearly I didn't pick up everything that made my father successful. I don't think dwelling on the differences will help much, but relevant is that I was born with no vision in one eye, and had some vision loss in the other a few years later, until it finally dropped to near-useless starting around age fourteen. (Incidentally, this is my excuse if I fail to catch any spelling errors in this post.) I was taught to believe strongly in the power of human intelligence, and that peer pressure is evil, and that education is the most important thing ever.

By 2008, I'd discovered that most of this was extremely flawed, and it was too late to correct the damage that acting on my perceived value of these beliefs had caused (I hadn't realized the extent of that damage by then, but was beginning to pick up on some of it). I'd gotten into an expensive college because the best education possible was apparently important and expensive... yet between my vision, attention to what I thought of as rationality, and rejection of many social conventions, I was completely lacking the skills to get much out of the college experience other t...
[SEQ RERUN] Brain Emulation and Hard Takeoff
by MinibearRex — 2012-11-14T06:47:03.239Z

Today's post, Brain Emulation and Hard Takeoff was originally published on November 22, 2008. A summary:

A project of bots could start an intelligence explosion once it got fast enough to start making bots of the engineers working on it, that would be able to operate at greater than human speed. Such a system could also devise a lot of innovative ways to acquire more resources or capital.

Discuss the post here (rather than in the comments to the original post).
This post is part of the Rerunning the Sequences series, where we'll be going through Eliezer Yudkowsky's old posts in order so that people who are interested can (re-)read and discuss them. The previous post was Emulations Go Foom, and you can use the sequence_reruns tag or rss feed to follow the rest of the series.
Sequence reruns are a community-driven effort. You can participate by re-reading the sequence post, discussing it here, posting the next day's sequence reruns post, or summarizing forthcoming articles on the wiki. Go here for more details, or to have meta discussions about the Rerunning the Sequences series.
Universal agents and utility functions
by Anja — 2012-11-14T04:05:38.614Z

I'm Anja Heinisch, the new visiting fellow at SI. I've been researching replacing AIXI's reward system with a proper utility function. Here I will describe my AIXI+utility function model, address concerns about restricting the model to bounded or finite utility, and analyze some of the implications of modifiable utility functions, e.g. wireheading and dynamic consistency. Comments, questions and advice (especially about related research and material) will be highly appreciated.

Introduction to AIXI

Marcus Hutter's (2003) universal agent AIXI addresses the problem of rational action in a (partially) unknown computable universe, given infinite computing power and a halting oracle. The agent interacts with its environment in discrete time cycles, producing an action-perception sequence with actions (agent outputs) and perceptions (environment outputs) chosen from finite sets and . The perceptions are pairs , where is the observation part and denotes a reward. At time k the agent chooses its next action according to the expectimax principle:

Here M denotes the updated Solomonoff prior summing over all programs that are consistent with the history [1] and which will, when run on the universal Turing machine T with successive inputs , compute outputs , i.e.

AIXI is a dualistic framework in the sense that the algorithm that constitutes the agent is not part of the environment, since it is not computable. Even considering that any running implementation of AIXI would have to be computable, AIXI accurately simulating AIXI accurately simulating AIXI ad infinitem doesn't really seem feasible. Potential consequences of this separation of mind and matter include difficulties the agent may have predicting the effects of its actions on the world.

Utility vs rewards

So, why is it a bad idea to work with a reward system? Say the AIXI agent is rewarded whenever a human called Bob pushes a button. Then a sufficiently smart AIXI will figure out that instead of furthering Bob’s goals it can also threaten or deceive Bob into pushing the button, or get another human to replace Bob. On the other hand, if the reward is computed in a little box somewhere and then displayed on a screen, it might still be poss...
Meetup : Chicago Organizational Meeting and Open Discussion
by Nic_Smith — 2012-11-14T01:45:56.555Z

Discussion article for the meetup : Chicago Organizational Meeting and Open Discussion

WHEN:
17 November 2012 01:00:00PM (-0600)

WHERE:
360 N. Michigan Ave., Chicago, IL

"This is an organizational meeting planned for every other month to discuss the group's progress and ideas for possible future meetups. Afterwards, we will have open discussion..." -- http://www.meetup.com/Less-Wrong-Chicago/events/83542572/

This location is the same Corner Bakery that we previously had the weekly meetings at.

Discussion article for the meetup : Chicago Organizational Meeting and Open Discussion
Meetup : Berkeley meetup: Deliberate performance
by Nisan — 2012-11-13T23:58:50.742Z

Discussion article for the meetup : Berkeley meetup: Deliberate performance

WHEN:
14 November 2012 07:00:00PM (-0800)

WHERE:
Berkeley, CA

This week's Berkeley meetup will be a structured discussion about getting better at skills. One way to get better at skills (like aikido) is deliberate practice. Another way to get better at skills (like making friends) is deliberate performance, which is what you do when practicing isn't an option. (If you meet an especially cool person, you don't want to try your risky bold untested friend-getting technique, because you actually want to make friends with them.) Very generally, the tools are

making predictions

experimenting

analyzing what happened

doing post-mortems on near-misses

getting experts to critique your technique

This is horribly abstract; at the meetup we'll have a structured discussion about how this could actually work in practice, and what we have been actually doing to become skilled. You don't have to read these linked articles to attend the meetup,

http://en.wikipedia.org/wiki/Practice_(learning_method)#Deliberate_practice

http://peterfadde.com/Research/Deliberate_Performance-PI-1011.pdf

but if you've read this book I really hope you attend:

http://www.amazon.com/Cambridge-Expertise-Performance-Handbooks-Psychology/dp/0521600812

Doors open at 7pm, and the meetup begins at 7:30pm. For directions to Zendo, see the mailing list:

http://groups.google.com/group/bayarealesswrong

or call me at:

http://i.imgur.com/Vcafy.png

Discussion article for the meetup : Berkeley meetup: Deliberate performance
How can I reduce existential risk from AI?
by lukeprog — 2012-11-13T21:56:12.609Z

Suppose you think that reducing the risk of human extinction is the highest-value thing you can do. Or maybe you want to reduce "x-risk" because you're already a comfortable First-Worlder like me and so you might as well do something epic and cool, or because you like the community of people who are doing it already, or whatever.

Suppose also that you think AI is the most pressing x-risk, because (1) mitigating AI risk could mitigate all other existential risks, but not vice-versa, and because (2) AI is plausibly the first existential risk that will occur.

In that case, what should you do? How can you reduce AI x-risk?

It's complicated, but I get this question a lot, so let me try to provide some kind of answer.

Meta-work, strategy work, and direct work

When you're facing a problem and you don't know what to do about it, there are two things you can do:

1. Meta-work: Amass wealth and other resources. Build your community. Make yourself stronger. Meta-work of this sort will be useful regardless of which "direct work" interventions turn out to be useful for tackling the problem you face. Meta-work also empowers you to do strategic work.

2. Strategy work: Purchase a better strategic understanding of the problem you're facing, so you can see more clearly what should be done. Usually, this will consist of getting smart and self-critical people to honestly assess the strategic situation, build models, make predictions about the effects of different possible interventions, and so on. If done well, these analyses can shed light on which kinds of "direct work" will help you deal with the problem you're trying to solve.

When you have enough strategic insight to have discovered some interventions that you're confident will help you tackle the problem you're facing, then you can also engage in:

3. Direct work: Directly attack the problem you're facing, whether this involves technical research, political action, particular kinds of technological development, or something else.

Thinking with these categories can be useful even though the lines between them are fuzzy. For example, you might have to do some basic awareness-raising in order to amass funds for your cause, and then once you've spent those funds on strategy work, your strategy work might tell you that a specific form of awareness-raising is useful for political action that counts as "direct work." Also, some form...
Group rationality diary, 11/13/12
by cata — 2012-11-13T18:39:42.946Z

This is the public group instrumental rationality diary for the week of October 29th. It's a place to record and chat about it if you have done, or are actively doing, things like:

Established a useful new habit

Obtained new evidence that made you change your mind about some belief

Decided to behave in a different way in some set of situations

Optimized some part of a common routine or cached behavior

Consciously changed your emotions or affect with respect to something

Consciously pursued new valuable information about something that could make a big difference in your life

Learned something new about your beliefs, behavior, or life that surprised you

Tried doing any of the above and failed

Or anything else interesting which you want to share, so that other people can think about it, and perhaps be inspired to take action themselves. Try to include enough details so that everyone can use each other's experiences to learn about what tends to work out, and what doesn't tend to work out.

Thanks to everyone who contributes!

Previous diary; archive of prior diaries.
CFAR and SI MOOCs: a Great Opportunity
by Wrongnesslessness — 2012-11-13T10:30:15.543Z

Massive open online courses seem to be marching towards total world domination like some kind of educational singularity (at least in the case of Coursera). At the same time, there are still relatively few courses available, and each new added course is a small happening in the growing MOOC community.

Needless to say, this seems like a perfect opportunity for SI and CFAR to advance their goals via this new education medium. Some people seem to have already seen the potential and taken advantage of it:

One interesting trend that can be seen is companies offering MOOCs to increase the adoption of their tools/technologies. We have seem this with 10gen offering Mongo courses and to a lesser extent with Coursera’s ‘Functional Programming in Scala’ taught by Martin Odersky

(from the above link to the Class Central Blog)

So the question is, are there any online courses already planned by CFAR and/or SI? And if not, when will it happen?

Edit: This is not a "yes or no" question, albeit formulated as one. I've searched the archives and did not find any mention of MOOCs as a potentially crucial device for spreading our views. If any such courses are already being developed or at least planned, I'll be happy to move this post to the open thread, as some have requested, or delete it entirely. If not, please view this as a request for discussion and brainstorming.

P.S.: Sorry, I don't have the time to write a good article on this topic.
[SEQ RERUN] Emulations Go Foom
by MinibearRex — 2012-11-13T05:44:40.330Z

Today's post, Emulations Go Foom was originally published on November 22, 2008. A summary:

A description of what Robin Hanson thinks is the most likely scenario for a intelligence takeoff.

Discuss the post here (rather than in the comments to the original post).
This post is part of the Rerunning the Sequences series, where we'll be going through Eliezer Yudkowsky's old posts in order so that people who are interested can (re-)read and discuss them. The previous post was Life's Story Continues, and you can use the sequence_reruns tag or rss feed to follow the rest of the series.
Sequence reruns are a community-driven effort. You can participate by re-reading the sequence post, discussing it here, posting the next day's sequence reruns post, or summarizing forthcoming articles on the wiki. Go here for more details, or to have meta discussions about the Rerunning the Sequences series.