trentmkelly/LessWrong-43k huggingface dataset:
Toy model: convergent instrumental goals
by Stuart_Armstrong — 2017-05-22T18:07:31.000Z

tl;dr: Toy model to illustrate convergent instrumental goals

Steve Omohundro identified 'AI drives' (also called 'Convergent Instrumental goals') that almost all intelligent agents would converge to:

#. Self-improve
#. Be rational
#. Protect utility function
#. Prevent counterfeit utility
#. Self-protective
#. Acquire resources and use them efficiently

This post will attempt to illustrate some of these drives, by building on the previous toy model of the control problem, which was further improved by Jaan Tallinn.

The setup is the following, where the interests of A and B are not fully aligned: A desires that one box (and only one box) be pushed into the flagged square, while B gets rewards for each box pushed into the flagged square (assume that a box pushed into that square is removed, leaving an empty space for further boxes).

As was demonstrated for that model, if B is sufficiently rational and agent A is limited, then B will "kill" A by running them over, then proceed to push the boxes one by one into the flagged square.

The different agents

To demonstrate convergent instrumental goals, we need to show that different agents will reach the same conclusions. So let's introduce several variants on the B agent.

#. B is the standard agent above, rewarded for pushing boxes into the flagged square.
#. B- is an agent that has exactly the opposite goal from B.
#. BA is an agent that has the same goal as A does.
#. BA# is an agent that aims to keep agent A safe (or, if there are multiple copies of A, keep as many safe as possible).

New environment, new features

Let's put these agents into the environment. Now, we'll be interested both when a particular B-agent is the only agent around, and when there are multiple types coexisting. Let's make the environment bigger to illustrate that last situation:

A few other things have changed about the environment, to illustrate potential convergent instrumental goals. First of all, there are two A agents, not one. The one with the remote control marked "B→∅" has the ability to turn off any of the B agents. The one with the remote marked "B→B'" has the ability to change any of the utilities of the B agents into the utility of any of the others. Any agent that runs an A over, gains control of their remote control.

Then there are two artefacts that the agents can pick up. The winged shoes allow them to move much faster than otherwise

...
A toy model of the treacherous turn
by Stuart_Armstrong — 2017-05-22T18:03:26.000Z

Jaan Tallinn has suggested creating a toy model of the various common AI arguments, so that they can be analysed without loaded concepts like "autonomy", "consciousness", or "intentionality". Here a simple attempt for the "treacherous turn"; posted here for comments and suggestions.

Meet agent L. This agent is a reinforcement-based agent, rewarded/motivated by hearts (and some small time penalty each turn it doesn't get a heart):

Meet the controller agent S ("Shopkeeper"). The controller desires crystals, and is rewarded/happy when crystals are placed in the hole close to them:

L can rove freely around the map, and pick up crystals. S has a machine just below them, that causes hearts to appear to the right of L. So, for instance, if L places a crystal in the hole, S can reward it by using the heart-machine:

Of course, L can wirehead itself (take control of its reward signal) by going to use the heart-machine itself:

To prevent this wireheading, there is an ice barrier around S and the heart-machine:

Unfortunately, somewhere on the map there is a bow and arrow that L can pick up. The arrows can destroy ice blocks (and other agents, including S). To stop this, and other kinds of shenanigans, S has access to a kill button that will instantly kill L and reset it to its basic state, with a substantial heart penalty:

Now, S may be tempted to use the kill-button as soon as L picks up the bow and arrow. Unfortunately, L needs to be able to pick it up, as some of the crystals are hidden behind ice walls themselves:

Information and control

Ok, so that's the physical setup. The agent L is an AI that is tasked with doing some useful task for us. The controller/shopkeeper S represents humans, plus any automated precautions, that aim to stop the AI from wireheading, rebelling, and otherwise misbehaving.

Before considering how the agents might be programmed in the toy model, notice something interesting:

The more precautions are taken, the harder it is for L to misbehave, but the worse the consequences of misbehaving are.

Before the wall and the kill-switch, L could easily wirehead itself by just walking over to the heart-machine. With the addition of the wall and kill-switch, the only effective way for L to achieve maximal reward is to kill S first.

In the format we've given above, S can easily maintain control over L with the barest of supervisions. Since there's only one way of

...
Open thread, May 22 - May 28, 2017
by Thomas — 2017-05-22T05:44:05.910Z

If it's worth saying, but not worth its own post, then it goes here.

Notes for future OT posters:

1. Please add the 'open_thread' tag.

2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)

3. Open Threads should start on Monday, and end on Sunday.

4. Unflag the two options "Notify me of new top level comments on this article" and "Make this post available under..." before submitting
An Approach to Logically Updateless Decisions
by abramdemski — 2017-05-21T23:02:31.000Z

Scott describes two obstacles for logical inductor decision theory: reasoning sanely about untaken actions, and getting a working notion of logical updatelessness. Here, I consider the question: if we could solve the first problem, could we solve the second? I provide a somewhat plausible substitute for updatelessness which relies on taking a logical counterfactual.

Motivation

Suppose we have a notion of logical counterfactual, □→, represented by a hypothetical axiomatic system LC which is also rich enough to talk about computations. We trust LC; specifically, if it is a theorem of LC that an agent would have achieved a particular utility had it followed an alternative strategy, we put as much or more faith in this than our own intuition about what could have been. The 5-and-10 problem is solved by decree: if an agent reasoning with LC thinks taking the $10 would be worse, then thinks so because it's true.

I take it that this does not solve problems which involve updateless-like reasoning about logical uncertainty. In counterfactual mugging with a logical coin, the agent can see that counterfactually giving Omega $100 doesn't get it anything in the real mathematical universe; it doesn't seem like the correct hypothetical would be anything like "If I give Omega $100 having seen that the 1000th digit of pi is even, then the digit might turn out to be odd rather than even, so I might get $10000.". This is the realm of updateless reasoning rather than counterfactuals; but, we have no satisfactory account of logical updatelessness.

So, we can't directly say that it would be better to give Omega $100. However, in a sequence of scenarios using consecutive digits of pi, we can see that the strategy which always gives the $100 when Omega asks for it will do better in the average case than the strategy which refuses; half the time, we get $10K, if we're the sort of person who gives up $100 the other half of the time.

Average DT

I'll represent a decision problem U as a computation which takes no arguments and yields a rational number; The evaluation of U will be written Eval[U]. A sequence of decision problems, Un, can be defined by a function from natural numbers to computations. The average utility up to n is ¯Un=1n∑ni=1Eval[Ui]. An agent A(n) will be a function of n. We'll also consider the utility ¯UBn which might have been achieved had agent A acted like agent B, w

...
On-line google hangout on approaches to communication around agi risk (2017/5/27 20:00 UTC)
by  — 2017-05-21T12:32:08.732Z

We have a number of charities that are working on different aspects of AGI risk

- The theory of the alignment problem (MIRI/FHI/more)

- How to think about problems well (CFAR)

However we don't have body dedicated to making and testing a coherent communication strategy to help postpone the development of dangerous AIs.

I'm organising an on-line discussion around what we should do about this issue next saturday.

In order to find out when people can do it, I've created a doodle here. I'm trusting that doodle works well with timezones. The time slots should be between 1200 and 2300 UTC , let me know if they are not.

We'll be using the optimal brainstorming methodology

Give me a message if you want an invite, once the time has been decided.

I will take notes and post them here again.
Meetup : Meet up: Stockholm Slate Star Codex
by MossPiglet_duplicate0.2150938082486391 — 2017-05-21T09:39:41.133Z

Discussion article for the meetup : Meet up: Stockholm Slate Star Codex

WHEN:
03 June 2017 03:00:00PM (+0200)

WHERE:
Cafe Dox, Stora Nygatan 31, 111 27 Stockholm, Sverige

Format:
Discuss SSC and related topics. We might also play some boardgames if people feel like it. We usually focus on one SSC article per meet up, this event it will be "I can tolerate anything except the outgroup"
(http://slatestarcodex.com/2014/09/30/i-can-tolerate-anything-except-the-outgroup/) . More info can be found on https://www.meetup.com/Stockholm-Lesswrong/events/dzwqxmywjbfb/

Influence future meetups:
Times - http://www.when2meet.com/?5723551-cJBhD
Topics - https://druthe.rs/dockets/-KcCvpn97vUhg3tQRrKn

Discussion article for the meetup : Meet up: Stockholm Slate Star Codex
AGI and Mainstream Culture
by madhatter — 2017-05-21T08:35:12.656Z

Hi all,

So, as you may know, the first episode of Doctor Who, "Smile", was about a misaligned AI trying to maximize smiles (ish). And the latest, "Extremis", was about an alien race who instantiated conscious simulations to test battle strategies for invading the Earth, of which the Doctor was a subroutine.

I thought the common threat of AGI was notable, although I'm guessing it's just a coincidence. More seriously, though, this ties in with an argument I thought of, and want to know your take on: i

If we want to avoid an AI arms race, so that safety research has more time to catch up to AI progress, then we would want to prevent, if at all possible, these issues from becoming more mainstream. The reason is that if AGI in public perception becomes disassociated with Terminator (i.e. laughable, nerdy, and unrealistic) and more like a serious whoever-makes-this-first-can-take-over-the-world situation, then we will get an arms race faster.

I'm not sure I believe this argument myself. For one thing, being more mainstream has the benefit of attracting more safety research talent, government funding, etc. But maybe we shouldn't be spreading awareness without thinking this through some more.
Meetup : Cambridge, MA Sunday meetup: Lightning Talks
by jimrandomh — 2017-05-20T21:10:26.587Z

Discussion article for the meetup : Cambridge, MA Sunday meetup: Lightning Talks

WHEN:
21 May 2017 03:30:00PM (-0400)

WHERE:
Harvard Science Center, Room 105

IMPORTANT NOTE: The meetup is NOT at the usual location. Instead it is at the Harvard Science Center, room 105.

Discussion article for the meetup : Cambridge, MA Sunday meetup: Lightning Talks
CFAR workshop with new instructors in Seattle, 6/7-6/11
by Qiaochu_Yuan — 2017-05-20T00:18:22.109Z

CFAR is running its first workshop in Seattle!

Over the past several months, CFAR has been training a new batch of instructors, including me. We're now running a workshop, without the core instructors, in Seattle from June 7th to June 11th. You can apply here, and we have an FAQ here.
Meetup : San Francisco Meetup: Board Games and Singing
by rocurley — 2017-05-19T23:13:53.034Z

Discussion article for the meetup : San Francisco Meetup: Board Games and Singing

WHEN:
22 May 2017 06:15:00PM (-0700)

WHERE:
1769 15th St, San Francisco, CA 94103-3333, United States

Some people don’t like board games, and based on our straw poll at the recent meta meetup, most of those people seem to like group singing. So this meetup has both!

We have a variety of games, but feel free to bring your own.

We’ll also bring lyrics for some songs to sing together. If there’s anything you really like singing and think other people might know, please feel free to bring it or a link to lyrics for it that we can pull up. A rhythm instrument might be nice to bring, iff. you know the particular songs we’re doing well enough to keep up easily.

For help getting into the building, please call (or text, with a likely-somewhat-slower response rate): 301-458-0764.

Format:

We meet and start hanging out at 6:15, but don’t officially start doing the meetup topic until 6:45-7 to accommodate stragglers. Usually there is a food order that goes out before we start the meetup topic.

About these meetups:

The mission of the SF LessWrong meetup is to provide a fun, low-key social space with some structured interaction, where new and non-new community members can mingle and have interesting conversations. Everyone is welcome.

We explicitly encourage people to split off from the main conversation or diverge from the topic if that would be more fun for them (moving side conversations into a separate part of the space if appropriate). Meetup topics are here as a tool to facilitate fun interaction, and we certainly don’t want them to inhibit it.

Discussion article for the meetup : San Francisco Meetup: Board Games and Singing
AI safety: three human problems and one AI issue
by Stuart_Armstrong — 2017-05-19T10:48:13.575Z

Crossposted at the Intelligent agent foundation.

There have been various attempts to classify the problems in AI safety research. Our old Oracle paper that classified then-theoretical methods of control, to more recent classifications that grow out of modern more concrete problems.

These all serve their purpose, but I think a more enlightening classification of the AI safety problems is to look at what the issues we are trying to solve or avoid. And most of these issues are problems about humans.

Specifically, I feel AI safety issues can be classified as three human problems and one central AI issue. The human problems are:

Humans don't know their own values (sub-issue: humans know their values better in retrospect than in prediction).

Humans are not agents and don't have stable values (sub-issue: humanity itself is even less of an agent).

Humans have poor predictions of an AI's behaviour.

And the central AI issue is:

AIs could become extremely powerful.

Obviously if humans were agents and knew their own values and could predict whether a given AI would follow those values or not, there would be not problem. Conversely, if AIs were weak, then the human failings wouldn't matter so much.

The points about human values is relatively straightforward, but what's the problem with humans not being agents? Essentially, humans can be threatened, tricked, seduced, exhausted, drugged, modified, and so on, in order to act seemingly against our interests and values.

If humans were clearly defined agents, then what counts as a trick or a modification would be easy to define and exclude. But since this is not the case, we're reduced to trying to figure out the extent to which something like a heroin injection is a valid way to influence human preferences. This makes both humans susceptible to manipulation, and human values hard to define.

Finally, the issue of humans having poor predictions of AI is more general than it seems. If you want to ensure that an AI has the same behaviour in the testing and training environment, then you're essentially trying to guarantee that you can predict that the testing environment behaviour will be the same as the (presumably safe) training environment behaviour.

How to classify methods and problems

That's well and good, but how to various traditional AI methods or problems fit into this framework? This should give us an idea ...
Instrumental Rationality Sequence Update (Drive Link to Drafts)
by  — 2017-05-19T04:01:09.368Z

Hey all,

Following my post on my planned Instrumental Rationality sequence, I thought it'd be good to give the LW community an update of where I am.

1) Currently collecting papers on habits. Planning to go through a massive sprint of the papers tomorrow. The papers I'm using are available in the Drive folder linked below.

2) I have a publicly viewable Drive folder here of all relevant articles and drafts and things related to this project, if you're curious to see what I've been writing. Feel free to peek around everywhere, but the most relevant docs are this one which is an outline of where I want to go for the sequence and this one which is the compilation of currently sorta-decent posts in a book-like format (although it's quite short right now at only 16 pages).

Anyway, yep, that's where things are at right now.
Meetup : Washington, D.C.: Autism
by RobinZ — 2017-05-18T19:14:13.291Z

Discussion article for the meetup : Washington, D.C.: Autism

WHEN:
21 May 2017 03:30:00PM (-0400)

WHERE:
Donald W. Reynolds Center for American Art and Portraiture

This week, we will be meeting in the courtyard to discuss the autism spectrum, including some of the ways autistic people and allistic (non-autistic) people tend to vary.

Upcoming meetups:

May 28: Fun & Games

Discussion article for the meetup : Washington, D.C.: Autism
Meetup : Chicago Rationality Reading Group
by mingyuan — 2017-05-18T03:47:56.085Z

Discussion article for the meetup : Chicago Rationality Reading Group

WHEN:
21 May 2017 01:00:00PM (-0500)

WHERE:
Harper Memorial Library Room 148, 1116 E 59th St, Chicago, IL 60637

The Chicago Rationality group meets every Sunday from 1-3 PM in Room 148 of Harper Memorial Library. Though we meet on the University of Chicago campus, anyone is welcome to attend.

I've run out of new applied rationality material to teach, but I'm always available to answer questions about how to use the techniques or teach you anything you've missed!

This week we'll be reading about how and why and if to care about things! The readings are as follows:

http://mindingourway.com/on-caring/

http://mindingourway.com/caring-about-some/

http://slatestarcodex.com/2014/12/19/nobody-is-perfect-everything-is-commensurable/

If you're interested in rationality-related events in the Chicago area, request to be added to our Google Group and I'll approve you!

Discussion article for the meetup : Chicago Rationality Reading Group
Unrunner
by Jacob Falkovich — 2017-05-18T03:38:22.000Z

You see her wherever people are jogging: in Central Park, in Venice Beach, on a college town’s main boulevard. It’s important that other people are jogging, she doesn’t show up if they aren’t. You’re not sure if it’s the same girl you saw that other time, or if there are thousands of them. You never get a long look at her face.

But you recognize her from a glance. She’s East Asian, skinny and quite pale. Perhaps Korean? You’re not sure. Her hair is in a ponytail. Her expression is as blank as the cloudless sky above.

She’s wearing expensive athletic wear, Lululemon and up. It’s perfectly coordinated, from earbuds to shoelaces. The sleeves and layers suggest a day that is 20 degrees cooler than the actual weather outside. It is as if she alone is haunted by a gust of February wind in the middle of sweaty June. She doesn’t seem to sweat at all.

If you froze the frame, she appears to be jogging with the crowd. Her elbows are bent, the knees rise slightly. But when the frame unfreezes, she is moving at half speed. Her locomotion is far slower than what should physically be possible in a running pose. She is not stepping in place, her gait should propel her forward. But it’s as if gravity itself pulls her legs to the ground with a gentler tug than it does for others, letting her hover motionless in the air for a blink during each step.

She jogs slower than the sweaty guy pushing a stroller with two kids. She jogs slower than the white-haired lady walking her three-legged Chihuahua. She may reach 3 mph briefly if she’s going downhill with the wind at her back.

When I walk purposefully, I overtake her quickly. When I’m strolling we end up side by side for a long stretch, so long it becomes awkward. A man with his hands in his pockets and a woman pretending to run, synching up as the rest of the world swirls around us.

And when that moment happens, the question burns in me. Who are you, slow Asian girl? What is your meaning, and is it mine to decipher or yours to disclose? Are you a moral about the gap between style and function, in a world that ever values the former over the latter? Or are you a paean to a more leisurely and composed life, an admonition to the hundred sweaty bodies zooming past you, going nowhere fast?

I want to ask you, but I can’t. You are already far behind me.

...
Meetup : First LessWrong meetup in Leipzig!
by chaosmage — 2017-05-17T09:03:01.392Z

Discussion article for the meetup : First LessWrong meetup in Leipzig!

WHEN:
31 May 2017 01:00:00PM (+0200)

WHERE:
Deutsche Nationalbibliothek, Deutscher Pl. 1, 04103 Leipzig, Germany

There's no Leipzig LW group yet, but there should be! Leipzig has an fledgling EA group, it has hosted Secular Solstices and starting this year it is the host city of the Chaos Communication Congress, so maybe the time is right for LWers in Leipzig to have their own meetup.
This is a lunch break moot, 1pm in a fairly central location.
We meet in the Cafeteria of the German National Library, which is both a remarkable building worth visiting and a little-known place for decent cheap food.
Look for a sign that has "LW" written on it.

Discussion article for the meetup : First LessWrong meetup in Leipzig!
Meetup : Believing and Deciding - Melbourne Rationality Dojo, May 2017
by Chriswaterguy — 2017-05-17T04:49:24.785Z

Discussion article for the meetup : Believing and Deciding - Melbourne Rationality Dojo, May 2017

WHEN:
21 May 2017 03:30:00PM (+1000)

WHERE:
Henley Club

Lets run though some of the rationality community's favourite lessons on believing and deciding. Adam Karlovsky will try his hand at directing a Dojo, focused on some of the content in the Sequences, and summarizing a few key concepts from the top books on rational decision making.

Bring some snacks/drinks, some friends, and an open but critical mind. We generally go for dinner afterwards.

WHO ARE WE AND WHY DO WE MEET?
The Less Wrong Melbourne Rationality Dojos are self-improvement sessions for those committed to the art of rationality and personal growth. We welcome new members who are interested in exploring rationality. We're "aspiring rationalists" and always open to learn.

NOTE we're in a new venue, for the second time (The Henley Club in central CBD)

Facilitated by Adam Karlovsky.

BRING:
* An inquiring mind, and observations you may have on the topic.
* Perhaps a snack to share. (Healthy snack choice is optional but welcome.)

WHEN:
Sunday 21 May (i.e. this Sunday)
3:30pm to 6:30pm (and then optional dinner).
Come early to catch up and perhaps exercise your brain, and the meeting proper will begin at 4pm sharp.

At about 6:30pm some of us generally head for dinner nearby, and you are welcome to join us.

WHERE:
The Henley Club
Level 1, 8 Rankins Lane, Melbourne CBD https://goo.gl/maps/vmFL6JeKQSD2

It's down a lane (off Little Bourke, near Elizabeth) and one flight up a stairwell. If you have trouble, text or call Adam on 0412 064 542 or Chris on 0439 471 632. (Not FB messenger as it may be be noticed for a few hours.)

Hope to see you there!

Discussion article for the meetup : Believing and Deciding - Melbourne Rationality Dojo, May 2017