trentmkelly/LessWrong-43k huggingface dataset:
Help us name a short primer on AI risk!
by lukeprog — 2013-09-17T20:35:34.895Z

MIRI will soon publish a short book by Stuart Armstrong on the topic of AI risk. The book is currently titled “AI-Risk Primer” by default, but we’re looking for something a little more catchy (just as we did for the upcoming Sequences ebook).

The book is meant to be accessible and avoids technical jargon. Here is the table of contents and a few snippets from the book, to give you an idea of the content and style:

Terminator versus the AI

Strength versus Intelligence

What Is Intelligence? Can We Achieve It Artificially?

How Powerful Could AIs Become?

Talking to an Alien Mind

Our Values Are Complex and Fragile

What, Precisely, Do We Really (Really) Want?

We Need to Get It All Exactly Right

Listen to the Sound of Absent Experts

A Summary

That’s Where You Come In …

The Terminator is a creature from our primordial nightmares: tall, strong, aggressive, and nearly indestructible. We’re strongly primed to fear such a being—it resembles the lions, tigers, and bears that our ancestors so feared when they wandered alone on the savanna and tundra.

…

As a species, we humans haven’t achieved success through our natural armor plating, our claws, our razor-sharp teeth, or our poison-filled stingers. Though we have reasonably efficient bodies, it’s our brains that have made the difference. It’s through our social, cultural, and technological intelligence that we have raised ourselves to our current position.

…

Consider what would happen if an AI ever achieved the ability to function socially—to hold conversations with a reasonable facsimile of human fluency. For humans to increase their social skills, they need to go through painful trial and error processes, scrounge hints from more articulate individuals or from television, or try to hone their instincts by having dozens of conversations. An AI could go through a similar process, undeterred by social embarrassment, and with perfect memory. But it could also sift through vast databases of previous human conversations, analyze thousands of publications on human psychology, anticipate where conversations are leading many steps in advance, and always pick the right tone and pace to respond with. Imagine a human who, every time they opened their mouth, had spent a solid year to ponder and research whether their response was going to be maximally effe

...
Meetup : Bratislava Meetup VII.
by Viliam_Bur — 2013-09-17T18:06:40.294Z

Discussion article for the meetup : Bratislava Meetup VII.

WHEN:
23 September 2013 06:00:00PM (+0200)

WHERE:
Bistro The Peach, Heydukova 21, Bratislava

Unstructured discussion. ;-)

Discussion article for the meetup : Bratislava Meetup VII.
Meetup : Durham/RTLW HPMoR discussion, ch. 86
by therufs — 2013-09-17T17:38:12.282Z

Discussion article for the meetup : Durham/RTLW HPMoR discussion, ch. 86

WHEN:
21 September 2013 12:00:11PM (-0400)

WHERE:
726 Rigsbee Avenue, Durham NC, 27701

Time for more HPMoR!

Meet at Fullsteam around 12 for discussion of the appointed chapters. If you have a chance to do the reading, bring a question, observation, or discussion topic. If not, come hang out anyhow!

12:00 gather, obtain comestibles from Cocoa Cinnamon or other local establishments

12:30 discussion

2:00 free-form Saturday enjoyment begins

To be notified about other upcoming meetups & events, join the RTLW mailing list: http://groups.google.com/group/rtlw !

Discussion article for the meetup : Durham/RTLW HPMoR discussion, ch. 86
Dennett on the selfish neuron, etc.
by NancyLebovitz — 2013-09-17T17:09:35.414Z

Dennett:

Mike Merzenich sutured a monkey's fingers together so that it didn't need as much cortex to represent two separate individual digits, and pretty soon the cortical regions that were representing those two digits shrank, making that part of the cortex available to use for other things. When the sutures were removed, the cortical regions soon resumed pretty much their earlier dimensions. If you blindfold yourself for eight weeks, as Alvaro Pascual-Leone does in his experiments, you find that your visual cortex starts getting adapted for Braille, for haptic perception, for touch.

The way the brain spontaneously reorganizes itself in response to trauma of this sort, or just novel experience, is itself one of the most amazing features of the brain, and if you don't have an architecture that can explain how that could happen and why that is, your model has a major defect. I think you really have to think in terms of individual neurons as micro-agents, and ask what's in it for them?

Why should these neurons be so eager to pitch in and do this other work just because they don't have a job? Well, they're out of work. They're unemployed, and if you're unemployed, you're not getting your neuromodulators. If you're not getting your neuromodulators, your neuromodulator receptors are going to start disappearing, and pretty soon you're going to be really out of work, and then you're going to die.

I hadn't thought about any of this-- I thought the hard problem of brains was that dendrites grow so that neurons aren't arranged in a static map. Apparently that is just one of the hard problems.

He also discusses the question of how much of culture is parasitic, that philosophy has something valuable to offer about free will (I don't know what he has in mind there), the hard question of how people choose who to trust and why they're so bad at it (he thinks people chose their investment advisers more carefully than they chose their pastors, I suspect he's over-optimistic), and a detailed look at Preachers Who Are Not Believers. That last looks intriguing-- part of the situations is that preachers have been taught it's very bad to shake someone else's faith, so there's an added layer of inhibition which keeps preachers doing their usual job even after they're no longer believers themselves.
Another way our brains betray us
by polymathwannabe — 2013-09-17T13:56:36.236Z

This appeared in the news yesterday.

http://www.alternet.org/media/most-depressing-discovery-about-brain-ever?paging=off

It turns out that in the public realm, a lack of information isn’t the real problem. The hurdle is how our minds work, no matter how smart we think we are. We want to believe we’re rational, but reason turns out to be the ex post facto way we rationalize what our emotions already want to believe.

...

The bleakest finding was that the more advanced that people’s math skills were, the more likely it was that their political views, whether liberal or conservative, made them less able to solve the math problem. [...] what these studies of how our minds work suggest is that the political judgments we’ve already made are impervious to facts that contradict us.

...

Denial is business-as-usual for our brains. More and better facts don’t turn low-information voters into well-equipped citizens. It just makes them more committed to their misperceptions.

...

When there’s a conflict between partisan beliefs and plain evidence, it’s the beliefs that win. The power of emotion over reason isn’t a bug in our human operating systems, it’s a feature.
Probability, knowledge, and meta-probability
by David_Chapman — 2013-09-17T00:02:56.680Z

This article is the first in a sequence that will consider situations where probability estimates are not, by themselves, adequate to make rational decisions. This one introduces a "meta-probability" approach, borrowed from E. T. Jaynes, and uses it to analyze a gambling problem. This situation is one in which reasonably straightforward decision-theoretic methods suffice. Later articles introduce increasingly problematic cases.

A surprising decision anomaly

Let’s say I’ve recruited you as a subject in my thought experiment. I show you three cubical plastic boxes, about eight inches on a side. There’s two green ones—identical as far as you can see—and a brown one. I explain that they are gambling machines: each has a faceplate with a slot that accepts a dollar coin, and an output slot that will return either two or zero dollars.

I unscrew the faceplates to show you the mechanisms inside. They are quite simple. When you put a coin in, a wheel spins. It has a hundred holes around the rim. Each can be blocked, or not, with a teeny rubber plug. When the wheel slows to a halt, a sensor checks the nearest hole, and dispenses either zero or two coins.

The brown box has 45 holes open, so it has probability p=0.45 of returning two coins. One green box has 90 holes open (p=0.9) and the other has none (p=0). I let you experiment with the boxes until you are satisfied these probabilities are accurate (or very nearly so).

Then, I screw the faceplates back on, and put all the boxes in a black cloth sack with an elastic closure. I squidge the sack around, to mix up the boxes inside, and you reach in and pull one out at random.

I give you a hundred one-dollar coins. You can put as many into the box as you like. You can keep as many coins as you don’t gamble, plus whatever comes out of the box.

If you pulled out the brown box, there’s a 45% chance of getting $2 back, and the expected value of putting a dollar in is $0.90. Rationally, you should keep the hundred coins I gave you, and not gamble.

If you pulled out a green box, there’s a 50% chance that it’s the one that pays two dollars 90% of the time, and a 50% chance that it’s the one that never pays out. So, overall, there’s a 45% chance of getting $2 back.

Still, rationally, you should put some coins in the box. If it pays out at least once, you should gamble all t...
Signaling of what, precisely?
by VipulNaik — 2013-09-16T17:12:32.088Z

It's a well-known fact that college graduates make more money than high school graduates who do not go to college, but the reason is not clear. Bryan Caplan offers a typology where he splits the gap between the income of college graduates and of high school graduates into three parts: human capital, ability bias, and signaling. Caplan also warns economists of education against conflating ability bias with signaling. The (human capital + signaling) total gives the "return on education" for a college degree -- the part that is causally due to getting a college degree. The (human capital) part alone gives the return on education through the channel of improved productivity, whereas the (signaling) part gives the return on education through the channel of being able to convince potential employers that they have higher productivity. The (ability bias) part can be thought of as selection bias for pre-existing ability: people who go on to graduate from college differ from people who do not go to college in terms of their pre-existing abilities (here "pre-existing" does not mean "innate" -- but rather it means what they had before starting college, or what they would have acquired through natural maturing even if they hadn't gone to college) and these people would likely have earned more (compared to the ones who didn't go to college) even if they had chosen not to go to college.

I want to look at the signaling channel more closely. A college degree does send a signal of some sort to potential employers, and stories of college dropouts who achieve great success notwithstanding, many employers, particularly for high-skilled occupations, prefer workers with college degrees even if the degree is not directly related to the work on the job. But why, precisely, does the employer value the degree more? I can categorize the possible explanations into three categories:

Causally agnostic signaling: Employers are relying on either their own experience or on the conventional wisdom that people with college degrees are more productive employees than people without. They don't actually care whether this is due to pre-existing productivity or due to human capital acquired in college.

Signaling of human capital acquisition: Employers believe that college students acquire valuable human capital from their college experience (whether it's their coursework or other aspects of the co

...
Cooperating with agents with different ideas of fairness, while resisting exploitation
by Eliezer Yudkowsky — 2013-09-16T08:27:24.719Z

There's an idea from the latest MIRI workshop which I haven't seen in informal theories of negotiation, and I want to know if this is a known idea.

(Old well-known ideas:)

Suppose a standard Prisoner's Dilemma matrix where (3, 3) is the payoff for mutual cooperation, (2, 2) is the payoff for mutual defection, and (0, 5) is the payoff if you cooperate and they defect.

Suppose we're going to play a PD iterated for four rounds. We have common knowledge of each other's source code so we can apply modal cooperation or similar means of reaching a binding 'agreement' without other enforcement methods.

If we mutually defect on every round, our net mutual payoff is (8, 8). This is a 'Nash equilibrium' because neither agent can unilaterally change its action and thereby do better, if the opponents' actions stay fixed. If we mutually cooperate on every round, the result is (12, 12) and this result is on the 'Pareto boundary' because neither agent can do better unless the other agent does worse. It would seem a desirable principle for rational agents (with common knowledge of each other's source code / common knowledge of rationality) to find an outcome on the Pareto boundary, since otherwise they are leaving value on the table.

But (12, 12) isn't the only possible result on the Pareto boundary. Suppose that running the opponent's source code, you find that they're willing to cooperate on three rounds and defect on one round, if you cooperate on every round, for a payoff of (9, 14) slanted their way. If they use their knowledge of your code to predict you refusing to accept that bargain, they will defect on every round for the mutual payoff of (8, 8).

I would consider it obvious that a rational agent should refuse this unfair bargain. Otherwise agents with knowledge of your source code will offer you only this bargain, instead of the (12, 12) of mutual cooperation on every round; they will exploit your willingness to accept a result on the Pareto boundary in which almost all of the gains from trade go to them.

(Newer ideas:)

Generalizing: Once you have a notion of a 'fair' result - in this case (12, 12) - then an agent which accepts any outcome in which it does worse than the fair result, while the opponent does better, is 'exploitable' relative to this fair bargain. Like the Nash equilibrium, the only way you should do wors...
Group Rationality Diary, September 16-30
by therufs — 2013-09-16T03:51:50.766Z

This is the public group instrumental rationality diary for September 16-30.

It's a place to record and chat about it if you have done, or are actively doing, things like:

Established a useful new habit

Obtained new evidence that made you change your mind about some belief

Decided to behave in a different way in some set of situations

Optimized some part of a common routine or cached behavior

Consciously changed your emotions or affect with respect to something

Consciously pursued new valuable information about something that could make a big difference in your life

Learned something new about your beliefs, behavior, or life that surprised you

Tried doing any of the above and failed

Or anything else interesting which you want to share, so that other people can think about it, and perhaps be inspired to take action themselves. Try to include enough details so that everyone can use each other's experiences to learn about what tends to work out, and what doesn't tend to work out.

Thanks to cata for starting the Group Rationality Diary posts, and to commenters for participating!

Immediate past diary: September 1-15

Rationality Diaries archive
Notes on logical priors from the MIRI workshop
by cousin_it — 2013-09-15T22:43:35.864Z

(This post contains a lot of math, and assumes some familiarity with the earlier LW posts on decision theory. Most of the ideas are by me and Paul Christiano, building on earlier ideas of Wei Dai.)

The September MIRI workshop has just finished. There were discussions of many topics, which will probably be written up by different people. In this post I'd like to describe a certain problem I brought up there and a sequence of ideas that followed. Eventually we found an idea that seems to work, and also it's interesting to tell the story of how each attempted idea led to the next one.

The problem is a variant of Wei Dai's A Problem About Bargaining and Logical Uncertainty. My particular variant is described in this comment, and was also discussed in this post. Here's a short summary:

In Counterfactual Mugging with a logical coin, a "stupid" agent that can't compute the outcome of the coinflip should agree to pay, and a "smart" agent that considers the coinflip as obvious as 1=1 should refuse to pay. But if a stupid agent is asked to write a smart agent, it will want to write an agent that will agree to pay. Therefore the smart agent who refuses to pay is reflectively inconsistent in some sense. What's the right thing to do in this case?

Thinking about that problem, it seems like the right decision theory should refuse to calculate certain things, and instead behave updatelessly with regard to some sort of "logical prior" inherited from its creators, who didn't have enough power to calculate these things. In particular, it should agree to pay in a Counterfactual Mugging with a digit of pi, but still go ahead and calculate that digit of pi if offered a straight-up bet instead of a counterfactual one.

What could such a decision theory look like? What kind of mathematical object is a "logical prior"? Perhaps it's a probability distribution over inconsistent theories that the creators didn't yet know to be inconsistent. How do we build such an object, what constraints should it satisfy, and how do we use it in a decision algorithm?

Attempt 1

Sometime ago, Benja Fallenstein and Paul Christiano came up with a way to build a probability distribution over all consistent theories. We start with a single empty theory with weight 1, and then refine it by successively adding all possible axioms. Each new axiom A is either already settled by the current theory T (i.e. ei...
Meetup : Frankfurt
by Kendra — 2013-09-15T11:27:28.273Z

Discussion article for the meetup : Frankfurt

WHEN:
22 September 2013 02:00:00PM (+0200)

WHERE:
Frankfurt, Ginnheimer Landstraße

Join us! The location is a private flat. Straßenbahnhaltestelle Frauenfriedenskirche, Linie 16. Please contact me for further details of the location (it's the same as last time.) Phone number is 0176 34 095 760, text message only, please. We have a mailing list, you can also write there:
less-wrong-frankfurt@googlegroups.com
There will probably be a PredictionBook-like game (and chocolate muffins).

Discussion article for the meetup : Frankfurt
Winter Solstice 2013 Kickstarter - The Big One. 
by Raemon — 2013-09-15T10:26:27.715Z

Like most things, winter was once a mystery.

The world got cold, and dark. Life became fragile. People died. And they didn't know what was happening or understand why. They desperately threw festivals in honor of sun gods with all-too-human motivations, and prayed for the light's return.

It didn't help. Though we did discover that throwing parties in the middle of winter is an excellent idea.

But then something incredible and beautiful happened. We studied the sky. We invented astronomy, and other sciences. We began a long journey towards truly understanding our place in the universe. And we used that knowledge to plan for the future, and make our world better. Five thousand years later, the winter isn't so scary. But the symbol of the solstice - the departure and return of the sun - is still powerful. The work we have done to transform winter from a terrifying season of darkness into a modern festival of light deserves a reverence with all the weight of an ancient cultural cornerstone.

And the work we have yet to do, to fully explore humanity's potential, is even more inspiring.

- humanistculture.com

Almost three years ago, the NY Rationality community celebrated their first Winter Solstice. This year, I'm ready to share this with as many people as possible. I'm aiming to host a large, concert event, with a goal of filling an 800-seat auditorium. I'm hoping to reach several different communities, connect people, and teach a broader audience about some important rationality concepts.

To make this happen, we're running a kickstarter, beginning today, that'll determine the scope of the event.

A Brief History:

We didn't intend it to be a big thing - it was just twenty of us gathered in a room, celebrating something we thought was important. I put together a series of songs and stories about the oddly specific things that we valued.

I meant it as a small, personal holiday. But I took its construction seriously. I put a lot of thought into why ritual works, why it's so hard to build from scratch, but how you might build it from scratch anyway.

And then I wrote some blog posts about it, and... it turned out to really really resonate with people here. And when I shared with non-Less Wrong folk, even people who had little interest in rationality, they still by and large found it interesting and powerful. I had worried it might look weird - to t...
Meetup : Columbus, OH MEGA-MEETUP, Oct 11-13
by daenerys — 2013-09-15T06:33:49.203Z

Discussion article for the meetup : Columbus, OH MEGA-MEETUP, Oct 11-14

WHEN: 12 October 2013 02:33:00AM (-0400)

WHERE: Columbus, OH

PRE-REGISTRATION REQUIRED!

Pre-Register HERE! (Google Forms are occasionally buggy. If you don't get an email from me within a day or two, then then I didn't get your pre-reg.)

Space is limited! The conference room can hold 50 people, 60 max, and our average meetup draws about half of that. Therefore, pre-registration is REQUIRED.

Pre-registration CLOSES on OCTOBER 4. This is the last day to either pre-register, or to cancel your pre-registration. If you discover you can not come, please cancel your pre-registration so that someone else may attend.

Housing is provided to out-of-towners, on a limited, first-registered, first-serve basis.

If you are interested in giving a talk, leading a discussion, or workshop, please describe what you want to do on the pre-reg page.

Pre-Register HERE!

SCHEDULE

FRIDAY

Socializing begins at 7:30p. There may be a roundtable discussion on community building, depending on interest. (If you come early, there will be TEDxColumbus, which is over-priced, IMO.)

SATURDAY
Official workshop begins at 3p

Introduction- Erica Edelman (me)
Who We Are and What We Do.

Down the Rabbit Hole: Magic as Psychic Entertainment -Jack Strauss
Magician/Mentalist Jack Strauss will present a stage act as a psychic entertainer. Afterwards, there will be a sit-down talkback with the audience. Topics will be determined by audience questions and may include: ethics of performing on stage with a psychic persona, psychology of deception, techniques of cold reading, etc. The only topic off the table will be the specifics of how the act you just saw is performed.

Defense Against the Dark Arts: The Ethics and Psychology of Persuasion- Jesse Galef
How do we convince other people of what's true? What tactics work and don't work? What rhetorical or psychological strategies can we practice to make ourselves better at it? How can we make ourselves less likely to be seduced by the Dark Arts of others? And of course, what's the ethics of all of this? Are all forms of persuasion ethical? How can we make those distinctions?

Rita Messer- Cognitive Behavioral Therapy
Effective Self-Help: Cognitive tricks for understanding and making your emotions work for you.

Rationality – a synergy of Western and Eastern approaches- Don Sutte

...