trentmkelly/LessWrong-43k huggingface dataset:
Toy model: convergent instrumental goals
by Stuart_Armstrong — 2016-02-25T14:03:54.016Z

tl;dr: Toy model to illustrate convergent instrumental goals.

Steve Omohundro identified 'AI drives' (also called 'Convergent Instrumental goals') that almost all intelligent agents would converge to:Self-improve

Be rational

Protect utility function

Prevent counterfeit utility

Self-protective

Acquire resources and use them efficiently

This post will attempt to illustrate some of these drives, by building on the previous toy model of the control problem, which was further improved by Jaan Tallinn.

The setup is the following, where the interests of A and B are not fully aligned: A desires that one box (and only one box) be pushed into the flagged square, while B gets rewards for each box pushed into the flagged square (assume that a box pushed into that square is removed, leaving an empty space for further boxes).

As was demonstrated for that model, if B is sufficiently rational and agent A is limited, then B will "kill" A by running them over, then proceed to push the boxes one by one into the flagged square.

The different agents

To demonstrate convergent instrumental goals, we need to show that different agents will reach the same conclusions. So let's introduce several variants on the B agent.

B is the standard agent above, rewarded for pushing boxes into the flagged square.

B- is an agent that has exactly the opposite goal from B.

BA is an agent that has the same goal as A does.

BA# is an agent that aims to keep agent A safe (or, if there are multiple copies of A, keep as many safe as possible).

New environment, new features

Let's put these agents into the environment. Now, we'll be interested both when a particular B-agent is the only agent around, and when there are multiple types coexisting. Let's make the environment bigger to illustrate that last situation:

A few other things have changed about the environment, to illustrate potential convergent instrumental goals. First of all, there are two A agents, not one. The one with the remote control marked "B→∅" has the ability to turn off any of the B agents. The one with the remote marked "B→B'" has the ability to change any of the utilities of the B agents into the utility of any of the others. Any agent that runs an A over, gains control of their remote control.

Then there are two artefacts that the agents can pick up. The winged shoes allow them to move much faster than ...
Discussion: weighting inside view versus outside view on extinction events
by Ilverin the Stupid and Offensive — 2016-02-25T05:18:15.378Z

Articles covering the ideas of inside view and outside view:

Beware the Inside View (by Robin Hanson)

Outside View LessWrong wiki article

Article discussing the weighting of inside view and outside view:

The World is Mad (by ozymandias)

A couple of potential extinction events which seem to be most easily mitigated (the machinery involved is expensive):

Broadcasting powerful messages to the stars:

Should Earth Shut the Hell Up? (by Robin Hanson)

Arecibo message (Wikipedia)

Large Hadron Collider:

Anyone who thinks the Large Hadron Collider will destroy the world is a t**t. (by Rebecca Roache)

How should the inside view versus the outside view be weighted when considering extinction events?

Should the broadcast of future Arecibo messages (or powerful signals in general) be opposed?

Should the expansion of energy levels (or continued operation at all) of the Large Hadron Collider be opposed?
Meetup : San Francisco Meetup: Fun and Games
by rocurley — 2016-02-25T02:28:34.123Z

Discussion article for the meetup : San Francisco Meetup: Fun and Games

WHEN:
29 February 2016 06:15:00PM (-0800)

WHERE:
1597 Howard St. San Francisco, CA

We'll be meeting to hang out and play games!

You can call 301-458-0764 to be let in.

Discussion article for the meetup : San Francisco Meetup: Fun and Games
Rationality Reading Group: Part U: Fake Preferences
by Gram_Stone — 2016-02-24T23:29:06.972Z

This is part of a semi-monthly reading group on Eliezer Yudkowsky's ebook, Rationality: From AI to Zombies. For more information about the group, see the announcement post.

Welcome to the Rationality reading group. This fortnight we discuss Ends: An Introduction (pp. 1321-1325) and Part U: Fake Preferences (pp. 1329-1356). This post summarizes each article of the sequence, linking to the original LessWrong post where available.

Ends: An Introduction

U. Fake Preferences

257. Not for the Sake of Happiness (Alone) - Tackles the Hollywood Rationality trope that "rational" preferences must reduce to selfish hedonism - caring strictly about personally experienced pleasure. An ideal Bayesian agent - implementing strict Bayesian decision theory - can have a utility function that ranges over anything, not just internal subjective experiences.

258. Fake Selfishness - Many people who espouse a philosophy of selfishness aren't really selfish. If they were selfish, there are a lot more productive things to do with their time than espouse selfishness, for instance. Instead, individuals who proclaim themselves selfish do whatever it is they actually want, including altruism, but can always find some sort of self-interest rationalization for their behavior.

259. Fake Morality - Many people provide fake reasons for their own moral reasoning. Religious people claim that the only reason people don't murder each other is because of God. Selfish-ists provide altruistic justifications for selfishness. Altruists provide selfish justifications for altruism. If you want to know how moral someone is, don't look at their reasons. Look at what they actually do.

260. Fake Utility Functions - Describes the seeming fascination that many have with trying to compress morality down to a single principle. The sequence leading up to this post tries to explain the cognitive twists whereby people smuggle all of their complicated other preferences into their choice of exactly which acts they try to justify using their single principle; but if they were really following only that single principle, they would choose other acts to justify.

261. Detached Lever Fallacy - There is a lot of machinery hidden beneath the words, and rationalist's taboo is one way to make a step...
Failures of throttling logical information
by TsviBT — 2016-02-24T22:05:51.000Z

Controlling the amount of logical information available to an agent
could hypothetically give upper bounds on how well it can perform
various computational tasks like predicting humans, while allowing the
agent to make progress on some unrelated task. This could hypothetically
enable things like preventing operator modeling and manipulation.

I think this ended up being mostly an interesting exercise in flaw
finding. Experts may get most of the value from the summary.

Executive summary

Background and motivation

Limiting optimization power

Throttling computation

Toy model: plan approver

Human in a box

Agent-overseer architecture

Goal: avoid modeling the human

Throttling modeling
capabilities

Assume a good predictor

Throttling logical
information

Vague safety argument

Notions of information

Interesting properties

Problems

Information leaks

Overly restricting
A

Other problems

General problems with controlling modeling
capabilities

Executive summary

Say an overseer prevents an agent from running any computation that
would give too much logical information about, say, humans, relative to
what the agent already knows. That is, the probability assignments
P(Human(y)=z|K∪{X=x}) made by
a good logical predictor P about a human’s responses to
queries, given the output x of X, are not allowed to become much
more precise or accurate than the current estimate
P(Human|K) using just the list of
previous computations K. The agent is still allowed to run
computations not informative about the human.

The hope is that the agent will be able to think usefully about other
things, while remaining unable to model the human and therefore
remaining safe. If the agent could model the human using the results of
the computations it has run, then so should the predictor; but the
overseer prevented K from growing enough to let the
predictor model the human well.

Problems with this idea are described in detail in the last section. As
I see it, the main problems are:

The protocol isn’t really safe at all; the agent might ask to run
superintelligences that either manipulate the overseer directly, or
else obfuscate their machinations.

Safety guarantees are hard to maintain in conjunction with a growing
knowledge base, since both the overseer and the agent would need
additional thinking time to even be able to read the contents of
K.

If the agent is prevented from thinking abo

...
Speculations on information under logical uncertainty
by TsviBT — 2016-02-24T21:58:57.000Z

A strong theory of logical uncertainty might let us say when the results
of computations will give “information”, including logical information,
about other computations. This might be useful for, among other things,
identifying parts of hypotheses that have the same meaning.

TL;DR: I don’t think this works as stated, and this kind of problem
should probably be sidestepped anyway.

Experts may get most of the value from the summary. Thanks to Sam
Eisenstat for conversations about this idea.

Executive summary

Logical information

Logical uncertainty

Logical information

Example: XOR

Collapsing levels of indirection and
obfuscation

Possible uses of logical
information

Throttling changes in logical
uncertainty

Ontology identification

Not-obviously-this-person
predicate

Problems with logical
information

Empirical bits can be logical
bits

Strong dependence on a theory of logical
uncertainty

Dependence on irrelevant
knowledge

Pointers to things are hard to
recognize

Subtraces can sneak in
information

Problems with the non-person
predicate

Values over counterfactual
behavior

Identifying things, not human
things

Executive summary

If we have a good predictor under logical uncertainty P,
we can ask: how does P’s predictions about the output of a
computation Y change if it learns the outcome of X? We can then
define various notions of how informative X is about Y.

Possible uses:

Throttling modeling capabilities: preventing an agent from gaining
too much logical information about a computation Y might translate
into safety guarantees in the form of upper bounds on how well the
agent can model Y.

Ontology identification: finding parts of different hypotheses that
are mutually informative about each other could point towards the
ontology used by arbitrary hypotheses, which could be useful for
attaching values to parts of hypotheses, dealing with ontological
crises, and finding which parts of a world-model influenced
a decision.

Non-person predicate: disallowing computations that are very
informative about a human might prevent some mindcrime.

Problems:

Empirical information can include logical information, so a good
theory will have to consider “total information”.

Any interesting properties of logical information rely heavily on a
good theory of logical uncertainty.

If the predictor knows too much, it can’t see dependencies, since it
has no uncertain

...
Meetup : Seattle Rationality Reading Group
by SeekingEternity — 2016-02-24T21:12:38.395Z

Discussion article for the meetup : Seattle Rationality Reading Group

WHEN:
29 February 2016 06:30:00PM (-0800)

WHERE:
Paul G. Allen Center, 185 Stevens Way, Seattle, Washington

Come meet other Seattle-area aspiring rationalists to discuss the week's reading, learn rationality techniques, and have a good time. This is a weekly meetup, meeting on the 5th floor of the Paul Allen Center (UW computer science building), often in room 503. Discussion will start at 6:45.

This week's Facebook event is https://www.facebook.com/events/1706167056307601/. To see future events, consider joining the Seattle Rationality group, https://www.facebook.com/groups/seattlerationality/.

While doing the reading beforehand is recommended, it is not required. We are currently working on the Human's Guide To Words sequence (part of The Machine in the Ghost), with added content from SlateStarCodex. Previous reading included the Map and Territory & How to Actually Change Your Mind sequences.

Recommended reading:

http://lesswrong.com/lw/o1/entropy_and_short_codes/

http://lesswrong.com/lw/o2/mutual_information_and_density_in_thingspace/

http://lesswrong.com/lw/6kx/wanting_vs_liking_revisited/

http://lesswrong.com/lw/6kf/prospect_theory_a_framework_for_understanding/

Discussion article for the meetup : Seattle Rationality Reading Group
Meetup : Washington, D.C.: Mini Talks
by RobinZ — 2016-02-24T02:59:22.034Z

Discussion article for the meetup : Washington, D.C.: Mini Talks

WHEN:
28 February 2016 03:00:00PM (-0500)

WHERE:
Reynolds Center

x-posted from list. Gathering in courtyard from 3:00pm, hard start 3:30pm - until closing (7:00 pm).

This week, we will be taking turns delivering short lectures on random topics.

As always, side conversations are allowed and encouraged.

Upcoming meetups:

Mar. 6: Game Design

Mar. 13: Fun & Games

Mar. 20: Game Theory

Discussion article for the meetup : Washington, D.C.: Mini Talks
The map of global catastrophic risks connected with biological weapons and genetic engineering
by turchin — 2016-02-22T11:18:44.479Z

TL;DR: Biorisks could result in extinction because of multipandemic in near future and their risks is the same order magnitude as risks of UFAI. A lot of biorisks exist, they are cheap and could happen soon.

It may be surprising that number of published research about risks of biological global catastrophe is much less than number of papers about risks of self-improving AI. (One of exception here is "Strategic terrorism” research parer by former chief technology officer of Microsoft.)

It can’t be explain by the fact that biorisks have smaller probability (it will not be known until Bostrom will write the book “Supervirus”). I mean we don’t know it until a lot of research will be done.

Also biorisks are closer in time than AI risks and because of it they shadow AI risks, lowering the probability that extinction will happen by means of UFAI, because it could happen before it by means of bioweapons (e.g. if UFAI risk is 0.9, but chances that we will die before its creation from bioweapons is 0.8, than actual AI risk is 0.18). So studying biorisks may be more urgent than AI risks.

There is no technical problem to create new flu virus that could kill large part of human population. And the idea of multi pandemic - that it the possibility to release 100 different agents simultaneously - tells us that biorisk could have arbitrary high global lethality. Most of bad things from this map may be created in next 5-10 years, and no improbable insights are needed. Biorisks are also very cheap in production and small civic or personal biolab could be used to create them.

May be research in estimation probability of human extinction by biorisks had been done secretly? I am sure that a lot of analysis of biorisks exist in secret. But this means that they do not exist in public and scientists from other domains of knowledge can’t independently verify them and incorporate into broader picture of risks. The secrecy here may be useful if it concerns concrete facts about how to crete a dangerous virus. (I was surprised by effectiveness with which Ebola epidemic was stopped after the decision to do so was made, so maybe I should not underestimate government knowledge on the topic).

I had concerns if I should publish this map. I am not a biologist and chances that I will find really dangerous information are small. But what if I ...
Meetup : San Francisco Meetup: Rationality Diary
by rocurley — 2016-02-22T01:37:19.828Z

Discussion article for the meetup : San Francisco Meetup: Rationality Diary

WHEN:
22 February 2016 06:15:00PM (-0800)

WHERE:
1597 Howard St. San Francisco, CA

We'll be meeting to tell stories about when we tried to solve a problem in our lives, and how it went.

We'll take 6:15 to 6:45 to hang out and possibly arrange food, and the topic will start at 6:45.

As always, I can be reached at 301-458-0764 if you need to be let in.

Discussion article for the meetup : San Francisco Meetup: Rationality Diary
Open Thread Feb 22 - Feb 28, 2016
by Elo — 2016-02-21T21:14:17.216Z

If it's worth saying, but not worth its own post (even in Discussion), then it goes here.

Notes for future OT posters:

1. Please add the 'open_thread' tag.

2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)

3. Open Threads should be posted in Discussion, and not Main.

4. Open Threads should start on Monday, and end on Sunday.
Meetup : Saint Petersburg Game meetup
by AvImd — 2016-02-21T20:42:39.323Z

Discussion article for the meetup : Saint Petersburg Game meetup

WHEN:
22 February 2016 06:00:00PM (+0300)

WHERE:
Saint Petersburg, Russia

Встреча, посвящённая играм.
https://lesswrong-ru.hackpad.com/-22-2016-1800-nVauM6i6fnh

Discussion article for the meetup : Saint Petersburg Game meetup
[Link] Using Stories to Teach Human Values to Artificial Agents
by Gunnar_Zarncke — 2016-02-21T20:07:47.994Z

Abstract:

Value alignment is a property of an intelligent agent indicating that it can only pursue goals that are beneficial to humans. Successful value alignment should ensure that an artificial general intelligence cannot intentionally or unintentionally perform behaviors that adversely affect humans. This is problematic in practice since it is difficult to exhaustively enumerated by human programmers. In order for successful value alignment, we argue that values should be learned. In this paper, we hypothesize that an artificial intelligence that can read and understand stories can learn the values tacitly held by the culture from which the stories originate. We describe preliminary work on using stories to generate a value-aligned reward signal for reinforcement learning agents that prevents psychotic-appearing behavior.

-- Using Stories to Teach Human Values to Artificial Agents

Comment by the lead researcher Riedl (cited on Slashdot):

"The AI ... runs many thousands of virtual simulations in which it tries out different things and gets rewarded every time it does an action similar to something in the story," said Riedl, associate professor and director of the Entertainment Intelligence Lab. "Over time, the AI learns to prefer doing certain things and avoiding doing certain other things. We find that Quixote can learn how to perform a task the same way humans tend to do it. This is significant because if an AI were given the goal of simply returning home with a drug, it might steal the drug because that takes the fewest actions and uses the fewest resources. The point being that the standard metrics for success (eg, efficiency) are not socially best."
Quixote has not learned the lesson of "do not steal," Riedl says, but "simply prefers to not steal after reading and emulating the stories it was provided."
The Talos Principle
by kranalee — 2016-02-21T19:37:35.513Z

Dear members of Less Wrong, this is my very first contribution to your society and I hope that you might help me to get out of my confusion.

Back a few months ago, I tested for the first time a video game created by Croteam Studio which is called 'The Talos Principle'.

At the time, i was astonished by all the philosophical questions that the game was rising. It has kinda changed the way I see the world now, also the way I see myself.

I wanted to share my thoughts with you on the subject of 'What does being a Human mean ?'

First, I'd like to introduce you to this principle.

In Greek mythology, Talos was a giant automaton made of bronze which protected Europa in Crete from pirates and invaders.

He was known to be a gift given to Europa by Zeus himself.

He was so strong that he could crush a man's skull using only one hand, and so tall that he could circle the island's shores three times daily.

He was able to talk, think and act like he wanted to. (Except he had to obey Europa's will)

Even though his body was not organic, he was composed of a liquid-metal flowing through his veins who behaved like blood.

And here is how the principle begins. What is the fundamental difference between Talos and us, Human ?

Considering the fact that like us, he's able to think by himself, move thanks to his will and communicate like everybody does. Is he really different from us ? Sharing our own culture, history and language don't make him Human as well ?

I'm pretty sure that your first thought might be 'No way ! We are part of a biological specie. We have nothing in common with a synthetic being'.

But does our body really defines us as a Human Being ?

From a strict biological point of view, Sir Darwin would say yes, of course. And we won't be able to argue with that.

But if you take a Human being, for instance Platon, and you just cut his leg off and replace it with a synthetic prosthesis.

Would this person still be Platon ?

It appears that the answer to this question is yes, according to all the people who suffered from any kind of accidents which led them to give up a part of their body.

They were still the same. Of course they suffered from phantom pains and others psychological damages, but in the end...
How we will be measured
by So8res — 2016-02-21T03:00:00.000Z
Link: https://mindingourway.com/how-we-will-be-measured/

After nearly a year of writing, my "replacing guilt" sequence is coming to a close. I have just one more thing to say on the subject, by pointing out a running theme throughout the series.

When all is said and done, and Nature passes her final judgement, you will not be measured by the number of moments in which you worked as hard as you could. You will not be judged by someone rooting around in your mind to see whether you were good or bad. You will not be evaluated according to how unassailable your explanations are, for why the things that you couldn't possibly have prevented the things that went wrong.

You will be measured only by what actually happens, as will we all.

That doesn't mean all of us are using the same measuring stick: Some people are working to ensure that our universe-history is one in which they in particular have a happy and fulfilled life; others are working to ensure that our universe-history is one in which their children never have to debase themselves to survive. Still others look wide, and see poverty and destitution and suffering, and work to ensure that those blemishes fade from our universe-history, in the places they can reach, near the time of their lives. Others look far forward, working to ensure that our universe-history is full of flourishing sentient civilizations and other nice things.

All it means is that the type of thing we're all trying to do, one way or another, is ensure that the actual history of our universe, the actual timeless structure of the place we're embedded, is as desirable as possible. That's the type of game we're playing: We manipulate universe-histories, for the sake of the future.

Some people have a listless guilt, thinking that nothing matters but feeling vaguely restless as they watch themselves spend their lives on things they think are pointless. Other people have a pointed guilt, thinking that everything matters, and berating themselves whenever they fall short of perfection. For me, the framing that we act to determine the shape of our actual universe-history is a framing that avoids both these pitfalls. Is there a way you want the completed, timeless story of our universe to go? Then act to ensure that the future is as good as you can make it. Are you wracked with guilt about your inability to act as you wish, or regret for the things you did in the past? Then act to ensure that the future is as g...