trentmkelly/LessWrong-43k huggingface dataset:
MIRI's technical research agenda
by So8res — 2014-12-23T18:45:41.421Z

I'm pleased to announce the release of Aligning Superintelligence with Human Interests: A Technical Research Agenda written by Benja and I (with help and input from many, many others). This document summarizes and motivates MIRI's current technical research agenda.

I'm happy to answer questions about this document, but expect slow response times, as I'm travelling for the holidays. The introduction of the paper is included below. (See the paper for references.)

The characteristic that has enabled humanity to shape the world is not strength, not speed, but intelligence. Barring catastrophe, it seems clear that progress in AI will one day lead to the creation of agents meeting or exceeding human-level general intelligence, and this will likely lead to the eventual development of systems which are "superintelligent'' in the sense of being "smarter than the best human brains in practically every field" (Bostrom 2014). A superintelligent system could have an enormous impact upon humanity: just as human intelligence has allowed the development of tools and strategies that let humans control the environment to an unprecedented degree, a superintelligent system would likely be capable of developing tools and strategies that give it extraordinary power (Muehlhauser and Salamon 2012). In light of this potential, it is essential to use caution when developing artificially intelligent systems capable of attaining or creating superintelligence.

There is no reason to expect artificial agents to be driven by human motivations such as lust for power, but almost all goals can be better met with more resources (Omohundro 2008). This suggests that, by default, superintelligent agents would have incentives to acquire resources currently being used by humanity. (Can't we share? Likely not: there is no reason to expect artificial agents to be driven by human motivations such as fairness, compassion, or conservatism.) Thus, most goals would put the agent at odds with human interests, giving it incentives to deceive or manipulate its human operators and resist interventions designed to change or debug its behavior (Bostrom 2014, chap. 8).

Care must be taken to avoid constructing systems that exhibit this default behavior. In order to ensure that the development of smarter-than-human intelligence has a positive impact on humanity, we must meet three formidable challenges: How can...
State of the Solstice 2014
by Raemon — 2014-12-23T06:18:46.346Z

This'll be the first of a collection of posts about the growing Secular Solstice. This post gives an overview of what happened this year. Future posts will explore what types of Solstice content resonates with which people, what I've learned about how Less Wrong culture intersects with other cultures, and updates I've made about ritual as it relates to individuals as well as movement building.

For the past three years, I've been spending the last several months of each year frantically writing songs, figuring out logistics, and promoting the New York Winter Solstice celebration for the Rationality and Secular communities in NYC.
This year... well, I did that too. But I also finally got to go a Solstice that I *wasn't* responsible for. I went to the Bay Area on December 13th, traveled straight from the airport to the dress rehearsal...

...and I found a community coming together to create something meaningful. I walked into the hall and found some 30 or so people, with some stringing together lights, some people tying decorations around candles, a choir singing together... it felt very much like a genuine holiday coming together in an organic fashion.
(There was some squabbling about how to best perform particular songs... but it felt *very* much to me like real holiday squabbling, whenever a family of creative people with strong opinions on things get together, and I found it surprisingly heartwarming)

This year there were four large Solstices in the US and one small but intense 3-day event in Leipzig. Each of them had a somewhat different audience and a different focus. Each was put on by local communities who I encouraged to put their own spin on it, bringing a mix of new and "traditional" songs. I've gotten some interesting feedback on which parts of Less Wrong culture resonate with which people.

Solstices that I actively collaborated with and/or consulted on include:

Bay Area - The most explicitly Less Wrong-y and transhumanist Solstice this year. Approximately 130 attendees. I suspect had the largest choir leading songs. About half of the audience seemed to be from what I'd consider the collective Rationality community (i.e. EA/Less Wrong/CFAR) and half were "friends of friends" who were less Less Wrong-y.

(If you went to the Bay Solstice, you can fill out the anonymous feedback form here. If you don't have much time, at least answering the first couple ...
Superintelligence 15: Oracles, genies and sovereigns
by KatjaGrace — 2014-12-23T02:01:02.907Z

This is part of a weekly reading group on Nick Bostrom's book, Superintelligence. For more information about the group, and an index of posts so far see the announcement post. For the schedule of future topics, see MIRI's reading guide.

Welcome. This week we discuss the fifteenth section in the reading guide: Oracles, genies, and sovereigns. This corresponds to the first part of Chapter ten.

This post summarizes the section, and offers a few relevant notes, and ideas for further investigation. Some of my own thoughts and questions for discussion are in the comments.

There is no need to proceed in order through this post, or to look at everything. Feel free to jump straight to the discussion. Where applicable and I remember, page numbers indicate the rough part of the chapter that is most related (not necessarily that the chapter is being cited for the specific claim).

Reading: “Oracles” and “Genies and Sovereigns” from Chapter 10

Summary

Strong AIs might come in different forms or 'castes', such as oracles, genies, sovereigns and tools. (p145)

Oracle: an AI that does nothing but answer questions. (p145)

The ability to make a good oracle probably allows you to make a generally capable AI. (p145)

Narrow superintelligent oracles exist: e.g. calculators. (p145-6)

An oracle could be a non-agentlike 'tool' (more next week) or it could be a rational agent constrained to only act through answering questions (p146)

There are various ways to try to constrain an oracle, through motivation selection (see last week) and capability control (see the previous week) (p146-7)

An oracle whose goals are not aligned with yours might still be useful (p147-8)

An oracle might be misused, even if it works as intended (p148)

Genie: an AI that carries out a high level command, then waits for another. (p148)

It would be nice if a genie sought to understand and obey your intentions, rather than your exact words. (p149)

Sovereign: an AI that acts autonomously in the world, in pursuit of potentially long range objectives (p148)

A genie or a sovereign might have preview functionality, where it describes what it will do before doing it. (p149)

A genie seems more dangerous than an oracle: if you are going to strongly physically contain the oracle, you may have been better just denying it so much acces

...
How to deal with Santa Claus?
by jkadlubo — 2014-12-22T17:50:29.676Z

Related: The Santa deception Is Santa real On the care of young rationalists

All of the other takes on this topic start from a point, when a child (usually 5-9 years old) asks "Is Santa real?" Nobody yet asked "how to raise my child Santa-free?" What to say, when a two-year-old, who just noticed that there is this character on TV asks "will he come to me, too?" A toddler may not yet understand the concept of lie, of pretending, of things not physically existing. How to tell her, what will happen, what to expect, how and why other children behave differently?

My three-year-old daughter discovered Santa last spring, which finally forced me to think: how to deal with it? Ignoring the thing worked for three years, but what now? I live in an extremely catholic country (Poland), so I cannot be completely blunt about it.

In the end I decided to call it "the fairy-tale of [Santa] Claus." For me it has a lot of advantages: this is a story that can be told, retold, reinvented and everybody knows it. In addition, since the name includes the phrase "the fairy-tale", it has just as much validity as the tale of the Red Riding Hood or any TV character that she likes.

I tested some of her beliefs about "Miko". I opened the box with books intended for gifts in front of her. When she wanted to read some of them, I explained that she cannot yet read her book, because she'll get it on Christmas Eve. She asked "is it from Miko?" and I replied that in some way it is, but I bought it. She didn't insist on reading it right now. A few days ago she helped me wrap some of the gifts. She commented that action "Miko brought these so we can wrap them and give them as gifts from Miko."

Malcolm told me, that he likes best the strategy, when you say that Santa Claus is a game that everyone plays. People pretend that there's a big guy in a suit who does the thing, and if you ever let down the pretense to your friends, you lose the game. I'm not entirely convinced by this strategy - it may be too complicated for a 2- or 3-year old (since my daughter didn't wrap her mind around the information that I bought the books).

What are other strategies that you use? Or which ones you don't like? Why?
Meetup : Washington, D.C.: Mini Talks + Socializing
by RobinZ — 2014-12-22T17:45:31.930Z

Discussion article for the meetup : Washington, D.C.: Mini Talks + Socializing

WHEN:
28 December 2014 03:00:00PM (-0500)

WHERE:
National Portrait Gallery

We will, in fact, be meeting on December 28th. As usual, it will be in the Kogod Courtyard of the National Portrait Gallery (8th and F Sts or 8th and G Sts NW, go straight past the information desk from either entrance), and the purported topic will be part short (~10-20 min) lectures on random topics and part hanging out and catching up with friends. As usual, we will congregate between 3:00 and 3:30; Mini Talks, if they happen, will run from around 3:30 until around 5:15 or when we run out of volunteers, whichever comes first; the remainder of the meetup will be free for people to spend as they please.

As always, if you need to show up late or leave early, that's completely fine, and if you have any requests, criticisms, concerns, or other comments you want to share with us, we'll be happy to hear from you. We know the holidays are a busy time for most people - this is meant to be a pleasant, low-key, stress-free meetup for anyone who would like to attend.

On the Metro, the Orange Line is scheduled to run every 16 minutes instead of every 12 minutes; otherwise, trains should be running on the regular weekend schedule.

Upcoming meetups:

Jan. 4: Meta Meetup (discuss Less Wrong DC: what you like, what you want, what you'd change, &c.)

Jan. 11: TBA (to be summarized)

Jan. 18: Fun & Games (bring games, play games, converse, socialize, or any combination thereof)

Discussion article for the meetup : Washington, D.C.: Mini Talks + Socializing
How to Read
by Peter Wildeford — 2014-12-22T05:40:37.235Z

Part of my attempt to provide a bunch of unsolicited, anecdotal evidence that probably doesn't work for everyone.

-

Of course you already know how to read. But do you know how to read well?

Many people who read a book want to read for entertainment. That's perfectly ok -- it seems like a great way to take a break and enjoy yourself. But many times people pick up a book with the intention to learn something important. If you're one of those people, it's important not to fool yourself, end up not learning anything, and just waste your time. That's how you end up reading for entertainment without realizing it.

This was a big problem for me, and I realized I was wasting a lot of time when I otherwise could have been productively reading books.

While I'm still not the best reader, here's how I think I solved that problem:

I'm choosy about which books I read. There are millions of books in the world. I can't read them all. So I have to be choosy. I think about what I stand to gain from the book. Is it worth my time to read it? Is the book actionable? I personally aim to read books that come to me in reviews, that from a skim of their table of contents look like they'll provide real value to me.

I think about what else I could be doing instead of reading. Reading is great, but in many cases experience can be a better teacher. Moreover, picking up some experience can help me understand and apply the lessons in books better. I try to adopt a "doing-reading" loop, where I read something, act upon it, then read another something, etc., continuing to iteratively improve in whatever skill I'm after. This also helps validate the advice of books.

I'm not afraid to ditch an underperforming book. If I don't like it, it's wasting my time, and it's time to move on.

I consider sources other than books. Books are often the best source of information on any topic, and are often higher quality because they're intensely reviewed. But many blog posts and online resources can be great too. I Ignore them at my own peril.

I read a summary, read the book, then re-read the summary. My favorite loop for retaining the main ideas of the book I read is to first find a high quality summary of the book and familiarize myself with the basic points. &nb...
Open thread, Dec. 22 - Dec. 28, 2014
by Gondolinian — 2014-12-22T02:34:04.606Z

If it's worth saying, but not worth its own post (even in Discussion), then it goes here.

Previous Open Thread

Notes for future OT posters:

1. Please add the 'open_thread' tag.

2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)

3. Open Threads should be posted in Discussion, and not Main.

4. Open Threads should start on Monday, and end on Sunday.
Meetup : Bratislava
by Viliam_Bur — 2014-12-21T22:20:23.257Z

Discussion article for the meetup : Bratislava

WHEN:
22 December 2014 06:00:00PM (+0100)

WHERE:
Laurinská 1, Bratislava

Same place as previously: foxford.

Discussion article for the meetup : Bratislava
Training Reflective Attention
by LoganStrohl — 2014-12-21T12:53:52.558Z

Crossposted at Agenty Duck

And somewhere in the back of his mind was a small, small note of confusion, a sense of something wrong about that story; and it should have been a part of Harry's art to notice that tiny note, but he was distracted. For it is a sad rule that whenever you are most in need of your art as a rationalist, that is when you are most likely to forget it. —HPMOR, Ch. 3

A rationalist’s art is most distant when it is most needed. Why is that?

When I am very angry with my romantic partner, what I feel is anger. I don’t feel the futility of throwing a tantrum, or the availability of other options like honest communication, or freewriting, or taking a deep breath. My attention is so narrowly focused on the object of my anger that I’m likely not even aware that I’m angry, let alone that my anger might be blinding me to my art.

When her skills are most needed, a rationalist is lost in an unskillful state of mind. She doesn’t recognize that it’s happening, and she doesn’t remember that she has prepared for it by learning and practicing appropriate techniques.

I've designed and exercise that trains a skill I call reflective attention, and some call mindfulness. For me, it serves as an anchor in a stormy mind, or as a compass pointing always toward a mental state where my art is close at hand.

Noticing that I am lost in an unskillful state of mind is a separate skill. But when I do happen to notice—when I feel that small, small note of confusion—reflective attention helps me find my way back. Instead of churning out even more pointless things to yell at my partner, it allows me to say, “I am angry. I feel an impulse to yell. I notice my mind returning over and over to the memory that makes me more angry. I’m finding it hard to concentrate. I am distracted. I have a vague impression that I have prepared for this.” And awareness of that final thought allows me to ask, “What have I trained myself to do when I feel this way?”

The goal of the following exercise is to practice entering reflective attention.

It begins with an instruction to think of nothing. When you monitor yourself to make sure you’re not having any thoughts, your attention ends up directed toward the beginnings of thoughts. Since the contents of consciousness are always changing, maintaining focus o...
Signalling with T-Shirt slogans 
by Gunnar_Zarncke — 2014-12-21T11:37:03.333Z

It kind of started when I got this T-shirt as a present two years ago:

It is not just a slogan that is quickly filtered out under the heading 'generic ad-like content'. It invites checking where the error is. It is kind of a challenge - at least for suitably minded persons. Exactly that kind of person I'd like to get in touch with more. This T-shirt signals: "I'm a nerd and proud of it." And the positive feedback I got from this was part of the reason I chose to signal this more. Maybe you'd like to signal this too. Please remember the T-shirt alone will not do it. You still have to talk to people. For the introverted among us (me included) I recommend active listening.

The remainder of this post lists some slogans I have tried, some I will likely try shortly and other related resources.

Obviously I'm not the only one using this signalling approach. T-shirt dealers have lots of these in stock. Thus some I just ordered online. But the most effective shirt I 'designed' myself. It is a black shirt suitable for business purposes and such a shirt with a small slogal on it stands out. I chose

That which can be

destroyed by the

truth should be.

— P.C. Hodgell

which I'm delighted to also find recommended by EY here.

One year ago MIRI linked to Rational Attire but the store appears to be broken right now. And then there is Pretty Rational which at least contains some promising slogans.

I also have this T-shirt about the Map-Territory correspondence:

The sentence

"This T-shirt is black"

is true if

This T-shirt is black.

-- Tarski

(I had this printed on a dark blue T-shirt thus adding a gradual truth aspect).

Other slogans of this type to be found on my T-shirts:

Math is an integral part of my life.

Bene Gesserit Litany Against Fear

I plan to print more of my own design shortly. The next one will be

"Everyone generalizes

from one example.

At least, I do."

-- Vlad Taltos

You can find out more about this quote here. What I like most about this one is that it is a bit self-deprecating which if combined with otherwise high status signalling (stance and gaze) I take comes across as approachable.

More ideas I might or might not print:

The enemy's gate is down.

-- Ender

"Two people acting

rationally [] and with

common knowledge

[] cannot agree to

disagree."

-- R. J. Aumann

"When y

...
[Link] The Dominant Life Form In the Cosmos Is Probably Superintelligent Robots
by Gunnar_Zarncke — 2014-12-20T12:28:50.420Z

An Article on Motherboard reports about Alien Minds by Susan Schneider who claiThe Dominant Life Form In the Cosmos Is Probably Superintelligent Robots. The article is crosslinked to other posts about superintelligence and at the end discusses the question why these alien robots leave us along. The arguments puts forth on this don't convince me though.
Velocity of behavioral evolution
by Dr_Manhattan — 2014-12-19T17:34:36.217Z

This suggested a major update on the velocity of behavioral trait evolution.

Basically mice transmitted fear of cherry smell reliably into the very next generation (via epigenetics).

www.newscientist.com/article/dn24677-fear-of-a-smell-can-be-passed-down-several-generations.html#.VJRgr8ADo

This seems pretty important.
Weekly LW Meetups
by FrankAdamek — 2014-12-19T17:27:05.198Z

This summary was posted to LW Main on December 12th. The following week's summary is here.

Irregularly scheduled Less Wrong meetups are taking place in:

Atlanta December Meetup - Game Night!: 27 December 2014 07:00AM
Australia onling hangout: 14 December 2014 07:00PM
Copenhagen: December Meetup: 20 December 2014 03:00PM
East Coast Solstice Megameetup: 20 December 2014 03:00PM
European Community Weekend 2015: 12 June 2015 12:00PM
Munich Meetup in December: 13 December 2014 12:00PM
Saint-Petersburg Meetup : 13 December 2014 04:00PM
Utrecht: Rationality Games: 14 December 2014 02:00PM
Utrecht: a critique of effective altruism: 04 January 2015 02:00PM

The remaining meetups take place in cities with regular scheduling, but involve a change in time or location, special meeting content, or simply a helpful reminder about the meetup:

Austin, TX - Caffe Medici: 13 December 2025 01:30PM
Brussels - Hope & Self-improvement: 13 December 2014 01:00PM
Canberra: End of year party: 13 December 2014 06:00PM
Moscow social meetup: codename "Order of Infrared Viola": 14 December 2014 02:00PM
Seattle Secular Solstice: 13 December 2014 05:30PM
Sydney Summer Solstice: 21 December 2014 06:00PM
[Vienna] Rationality Weekend Vienna: 13 December 2014 03:00PM
Washington, D.C.: Tetlock's "Expert Political Judgment": 14 December 2014 03:00PM

Locations with regularly scheduled meetups: Austin, Berkeley, Berlin, Boston, Brussels, Buffalo, Cambridge UK, Canberra, Columbus, London, Madison WI, Melbourne, Moscow, Mountain View, New York, Philadelphia, Research Triangle NC, Seattle, Sydney, Toronto, Vienna, Washington DC, and West Los Angeles. There's also a 24/7 online study hall for coworking LWers.

If you'd like to talk with other LW-ers face to face, and there is no meetup in your area, consider starting your own meetup; it's easy (more resources here). Check one out, stretch your rationality skills, build community, and have fun!

In addition to the handy sidebar of upcoming meetups, a meetup overview is posted on the front page every Friday. These are an attempt to collect information on all the meetups happening in upcoming weeks. The best way to get your meetup featured is still to use the Add New Meetup feature, but you'll also have the benefit of having your meetup mentioned in a weekly overview. These overview post...
Post Resource Request
by Minds_Eye — 2014-12-19T17:04:18.235Z

After having viewed a recent post by Gworley I noticed that the material was deliberately opaque*. It's not as complicated as it seems, and should be able to be taught to people at lower than "Level 4" on Kegan's Constructive Developmental Theory. The only serious block I saw was the ridiculous gap in inferential distance.

With that in mind I was hoping someone might have recommendations on even tangentially related material as what I have now appears to be insufficient. (Simplifying CDT appears to be manageable, but not particularly useful without further material like Kantor's Four Player Model and Subject-Object Notation.)

*edit: Not Gworley's, it was Kegan's material that was opaque.