trentmkelly/LessWrong-43k huggingface dataset:
Creating a satisficer
by Stuart_Armstrong — 2015-03-11T15:03:38.950Z

A putative new idea for AI control; index here.

This is a potential design for a safe-ish satisficer, based on the ideas presented here. The two key ideas are that a satisficer S(u) with utility u:

Would not effectively aid M(u), a u-maximiser.

Would not effectively resist M(-u), a u-minimizer.

So satisficer make poor allies and weak enemies.

The design, the designer and the verifier

Let M(u-v) be an agent that maximises u and minimises v. Let's assume that M(u-v) is currently unaware as to what u and v are (though it knows that the upper bound of u is easy to approach). It can interact with the outside environment in one way only: by creating a satisficer S(). S() will then be fed u to become S(u).

However, the design for S(u) will first be sent to M(εu+v), for some unknown small ε. Note that M(εu+v) is a v-maximiser, so it disagrees with M(u-v) except for u. M(εu+v) is a (hypothetical) agent that will be (counterfactually) released at some unspecified later date. Currently, it will be fed the whole reasoning history of M(u+v), and will make a single decision only: where S(u) is to be built or not.

So it seems that S(u) must;

Set u close to its maximal value (as this is "easy to approach").

Not increase v to any significant extent (or else M(u-v) would not design it).

Not decrease v to any significant extent (or else M(εu+v) would veto it).

Since v is unknown to M(u-v) and and resources are finite, this should serve as a general reduced impact requirement for S(u) (we may have to use something like a soft minimum across all v, rather than an expectation across all v, to avoid certain edge casess).

Since is u unknown to M(u-v), S() would serve as a general satisficing agent for any utility functions whose upper bounds are easy to approach (remember that we can take an arbitrary utility function and arbitrarily bound it at some number).

For the moment, this does seems like it would produce a successful satisficer...
Defining a limited satisficer
by Stuart_Armstrong — 2015-03-11T14:23:23.035Z

A putative new idea for AI control; index here.

EDIT: The definition of satisficer I'm using here is the informal one of "it tries to achieve a goal, without making huge changes on the universe" rather than "it's an agent that has utility u and threshold t". If you prefer the standard notation, think of this as a satisficer where t is not fixed, but dependent on some facts in the world (such as the ease of increasing u). I'm trying to automate the process of designing and running a satisficer: people generally chose t given facts about the world (how easy it is to achieve, for instance), and I want the whole process to be of low impact.

I've argued that the definition of a satisficer is underdefined, because there are many pathological behaviours all compatible with satsificer designs. This contradict the intuitive picture that many people have of a satisficer, which is an agent that does the minimum of effort to reach its goal, and doesn't mess up the outside world more than it has to. And if it can't accomplish the goals without messing up the outside world, it would be content not to.

In the spirit of "if you want something, you have to define it, then code it, rather than assuming you can get if for free through some other approach", can we spell out what features we would want from such a satisficer? Preferably in a simpler format that our intuitions.

It seems to me that if you had a proper u-satisficer S(u), then for many (real or hypothetical) v-maximiser M(v) out there, M(v) would find that:

Changing S(u) to S(v) is of low value.

Similarly, utility function trading with S(u) is of low value.

The existence or non-existence of S(u) is of low information content about the future.

The existence or non-existence of S(u) has little impact on the expected value of v.

Further, S(u):

Would not effectively aid M(u), a u-maximiser.

Would not effectively resist M(-u), a u-minimizer.

Would not have large impacts (if this can measured) for low utility gains.

A subsequent post will present an example of a satisficer using some of these ideas.

A few other much less-developed thoughts about satisficers:

Maybe require that it learns what variables humans care about, and doesn’t set them to extreme values – try and keep them in the same range. Do the same for variables humans may care about or that resemble values they care about.

Models the

...
Acausal trade barriers
by Stuart_Armstrong — 2015-03-11T14:17:27.000Z

Many of the ideas presented here require AIs to be antagonistic towards each other - or at least hypothetically antagonistic towards hypothetical other AIs. This can fail if the AIs engage in acausal trade, so it would be useful if we could prevent such things from happening.

Now, I have to admit I'm still quite confused by acausal trade, so I'll simplify it to something I understand much better, an anthropic decision problem.

Staples and paperclips, cooperation and defection

Cilppy has a utility function p, linear in paperclips, while Stapley has a utility function s, linear in staples (and both p and s are normalised to zero with one aditional item adding 1 utility). They are not causally connected, and each must choose "Cooperate" or "Defect". If they "Cooperate", they create 10 copies of the items they do not value (so Clippy creates 10 staples, Stapley creates 10 paperclips). If they choose defect, they create one copy of the item they value (so Clippy creates 1 paperclip, Stapley creates 1 staple).

Assume both agents know these facts, both agents use anthropic decision theories, and both agents are identical apart from their separate locations and distinct utility functions.

Then the outcome is easy: both agents will consider that "cooperate-cooperate" or "defect-defect" are the only two possible options, "cooperate-cooperate" gives them the best outcome, so they will both cooperate. It's a sweet story of cooperation and trust between lovers that never agree and never meet.

Breaking cooperation

How can we demolish this lovely agreement? As I often do, I will assume that there is some event X that will turn Clippy on, with P(X) ≈ 1 (hence P(¬X) << 1). Similarly there is an event Y that turns Stapley on. Since X and Y are almost certain, they should not affect the results above. If the events don't happen, the AIs will never get turned on at all.

Now I am going to modify utility p, replacing it with

p' = p - E(p|¬X).

This p with a single element subtracted off it, the expected value of p given that Clippy has not been turned on. This term feels like a constant, but isn't exactly, as we shall see. Do the same modification to utility s, using Y:

s' = s - E(s|¬Y).

Now contrast "cooperate-cooperate" and "defect-defect". If Clippy and Stapley are both cooperators, then p=s=10. However, if the (incredibly unlikely) ¬X were to happen, then Clippy would not exist,

...
Acausal trade barriers
by Stuart_Armstrong — 2015-03-11T13:40:17.473Z

A putative new idea for AI control; index here.

Many of the ideas presented here require AIs to be antagonistic towards each other - or at least hypothetically antagonistic towards hypothetical other AIs. This can fail if the AIs engage in acausal trade, so it would be useful if we could prevent such things from happening.

Now, I have to admit I'm still quite confused by acausal trade, so I'll simplify it to something I understand much better, an anthropic decision problem.

Staples and paperclips, cooperation and defection

Cilppy has a utility function p, linear in paperclips, while Stapley has a utility function s, linear in staples (and both p and s are normalised to zero with one aditional item adding 1 utility). They are not causally connected, and each must choose "Cooperate" or "Defect". If they "Cooperate", they create 10 copies of the items they do not value (so Clippy creates 10 staples, Stapley creates 10 paperclips). If they choose defect, they create one copy of the item they value (so Clippy creates 1 paperclip, Stapley creates 1 staple).

Assume both agents know these facts, both agents use anthropic decision theories, and both agents are identical apart from their separate locations and distinct utility functions.

Then the outcome is easy: both agents will consider that "cooperate-cooperate" or "defect-defect" are the only two possible options, "cooperate-cooperate" gives them the best outcome, so they will both cooperate. It's a sweet story of cooperation and trust between lovers that never agree and never meet.

Breaking cooperation

How can we demolish this lovely agreement? As I often do, I will assume that there is some event X that will turn Clippy on, with P(X) ≈ 1 (hence P(¬X) << 1). Similarly there is an event Y that turns Stapley on. Since X and Y are almost certain, they should not affect the results above. If the events don't happen, the AIs will never get turned on at all.

Now I am going to modify utility p, replacing it with

p' = p - E(p|¬X).

This p with a single element subtracted off it, the expected value of p given that Clippy has not been turned on. This term feels like a constant, but isn't exactly, as we shall see. Do the same modification to utility s, using Y:

s' = s - E(s|¬Y).

Now contrast "cooperate-cooperate" and "defect-defect". If Clippy and Stapley are both cooperators, then p=s=10. Ho...
"Spiritual" techniques that actually work thread
by  — 2015-03-11T10:35:33.938Z

This thread is for:

- Perfectly natural and functional ideas that came from a spiritual, religious, occultist, parapsychologist etc. source (perhaps with some "baggage")

- Techniques that are bit difficult to explain and may be seen by the gullible as magic, but they actually seem to do something, even if that something is just a novel way to trick the brain.

Both things that are actually useful and "stage tricks" are accepted in this thread.
[LINK] Author's Note 119: Shameless Begging
by Evan_Gaensbauer — 2015-03-11T00:14:31.181Z

[Contains No HPMOR Spoliers]
[http://hpmor.com/notes/119/](http://hpmor.com/notes/119/)
I was at first confused by Eliezer's requests at the end of Ch. 119. I missed his Author's Notes, which his explains his rationale behind them. I thought I would share in case others missed it, especially because readers on LessWrong may have more elite or broader networks to help Eliezer achieve his new goals.

Eliezer is seeking the possibility of gaining J. K. Rowling's permission to publish HPMoR in some form, and have the profits donated to a U.K. charity. To this end, he wants to contact someone who can put him in touch with either J. K. Rowling, or Daniel Radcliffe.

If you're attending or supporting Worldcon through 2015-17, and think Harry Potter and the Methods of Rationality is worthy, please nominate it to win Best Novel for the 2016 Hugo Awards.

The Machine Intelligence Research Institute is looking to hire at least one new executive, and very competent mathematicians for its research.

Eliezer is thinking on what fiction to next write. He would re-edit an old script of his if and only if someone contacts with him to make either a movie with high-quality special effects, or an anime. Otherwise, he is seeking to work with the Collective Intelligence (the problem-solving force composed of the HPMoR readership). Check it out if you consider yourself part of the CI.

Eliezer is seeking to contact John Paulson the hedge-fund manager, to represent a financier as their angel investor, and other investment opportunities.

Eliezer has several other projects he might be interested in. Learn more by clicking the link.
Meta- the goals of this forum
by orthonormal — 2015-03-10T20:16:47.000Z

Summary: We're planning to publicize and open up the forum very soon, and so it's a good time to discuss what we would like this forum to achieve, how we plan for moderation to work, and what discussions are on-topic.

Currently, this forum is read-only for everyone except for a few veterans of the mailing list it replaces. In a few days, we're planning to open up posting (in a tiered way, similar in spirit to the tiered privileges of MathOverflow), and the comments and Likes of the full members will play a material role in moderating the community. So it's a good time for those of us who are already here to discuss our goals for the forum, so that we stand a better chance of coordinating.

The forum structure

Right now, the forum has members who are allowed to submit posts and comments, and to Like other members' posts and comments; non-members can only read the contents of the forum, and membership is by invitation from the admins. What we're planning to do is allow non-members who sign up with their Facebook accounts to post links with titles, the way one can on Hacker News. (The Facebook login requirement reduces the amount of spam; clearly we'll want to add other ways of registering when we have the development time.)

These links from non-members will be visible only to members (and to the non-member who posted them) unless they've received some number of Likes from members, at which point they'll be publicly visible. Non-members who have posted a few links to good content that they wrote themselves will get membership invites. (We hope to also add the ability for members to recommend others for membership.)

The goal of all of this is to keep the discussion happening at a high level! One way of expressing the ideal of the forum is as a transparent box where high-quality conversation happens, and which has a mechanism for carefully letting new contributors in. That mechanism depends on the members, as well as the admins, maintaining an idea of what a good and relevant contribution looks like. (And there's really no way around doing some serious moderation if we want to create a useful place on the Internet. We aim for the standard set by Math Overflow and Hacker News.)

Now that I've explained the context, here is our current draft of the "how to contribute" page:

This is a publicly visible discussion forum for foundational mathematical research in artificial

...
Meetup : Moscow meetup: time perception, communication frameworks, stories, fallacies
by Alexander230 — 2015-03-10T19:21:50.782Z

Discussion article for the meetup : Moscow meetup: time perception, communication frameworks, stories, fallacies

WHEN:
15 March 2015 02:00:00PM (+0300)

WHERE:
Russia, Moscow, ulitsa L'va Tolstogo 16

This Sunday, at our usual place and time, we have the following talks planned:

Human perception of time, and it's worsening on big scales.

Communication frameworks.

About the harmfulness of stories.

About fallacies and logic.

Details are in the hackpad: https://lesswrong-ru.hackpad.com/-15-2015-szpkpxu4w0M

Info for newcomers: We gather in the Yandex office, first revolving door is on the right from the archway, entrance 8.1. Here is a guide how to get to Yandex: http://company.yandex.ru/contacts/redrose/. If you're late, send SMS to Slava at +7(926)313-96-42; you'll have to wait until he is free to go down and meet you (which might take a while). Expected duration of the meetup is 4-5 hours.

If you're not the member of our Moscow community yet, you might want to check out https://lesswrong-ru.hackpad.com/ , there's a lot of content about what's been going on for the last two years.

Also, please subscribe to our mailing list: https://groups.google.com/forum/#!forum/rationality-in-moscow

Discussion article for the meetup : Moscow meetup: time perception, communication frameworks, stories, fallacies
Harry Potter and the Methods of Rationality discussion thread, March 2015, chapter 119
by Gondolinian — 2015-03-10T18:10:02.886Z

This is a new thread to discuss Eliezer Yudkowsky’s Harry Potter and the Methods of Rationality and anything related to it. This thread is intended for discussing chapter 119.

Plans for next chapter release:

Ch. 120 will post on March 12th, 2015 at 12PM Pacific Time (7PM UTC).

The next long chapter will be Ch. 122, posting on March 14th, 2015 at 9AM Pacific / 4PM UTC.

There is a site dedicated to the story at hpmor.com, which is now the place to go to find the authors notes and all sorts of other goodies. AdeleneDawner has kept an archive of Author’s Notes. (This goes up to the notes for chapter 76, and is now not updating. The authors notes from chapter 77 onwards are on hpmor.com.)

Spoiler Warning: this thread is full of spoilers. With few exceptions, spoilers for MOR and canon are fair game to post, without warning or rot13. More specifically:

You do not need to rot13 anything about HP:MoR or the original Harry Potter series unless you are posting insider information from Eliezer Yudkowsky which is not supposed to be publicly available (which includes public statements by Eliezer that have been retracted).

If there is evidence for X in MOR and/or canon then it’s fine to post about X without rot13, even if you also have heard privately from Eliezer that X is true. But you should not post that “Eliezer said X is true” unless you use rot13.
Detecting agents and subagents
by Stuart_Armstrong — 2015-03-10T17:56:31.208Z

A putative new idea for AI control; index here.

There are many situations where it would be useful to identify the presence of an agent in the world, in a sufficiently abstract sense. There are many more situations where it would be useful to identify a subagent in an abstract sense. This is because people often have ideas for interesting and useful motivational restrictions for the AI (eg an AI that "wants to stay boxed", or a corrigible agent). But most of these motivations suffer from a "subagent problem": the AIs are motivated to create subagents that do not follow the motivational restrictions. The AI wanting to stay in the box is motivated to create a subagent that will protect it and keep it in the box, while the corrigible agent is not motivated to create corrigible subagents (though the pre-corriged agent might want to create pre-corriged subagents).

Unfortunately, it's extremely hard to identify an agent. Agents need not come conveniently packaged in some "sensors-actuators-internal structure-utility function" form. If you wanted to obfuscate an agent, hiding it in the natural movements of the solar system, or in the gyrations of the internet, this would be easy to do and hard to detect - and very hard to define what you would be looking for.

Fortunately, it's much easier to detect superintelligent AI's that have major impact on the world - ie the kind of agents that we would be worried about. Intuitively, this is true: if you suddenly find half the sky getting optimised for space colonisation, it's likely there's an AI somewhere there. How can this be formalised?

The importance of the agent

Imagine taking a slice of space-time around the moment when a superintelligent agent just got turned on. If you wanted to predict the future, what piece of information is most relevant in that time slice? Obviously the superintelligent agent. We could run the ideas for reduced impact in reverse, looking for the part that is of maximum impact.

I would therefore define the agent as the part of this slice that it's the most valuable to know about, or the part that it would be the most valuable to change, counterfactually, if such changes were possible. Note that this definition doesn't exactly identify agents, but if it misidentifies something, then that something must also be of great importance.

If we imagine the slice of space-time seeping forwards in t...
False Thermodynamic Miracles, in equation form
by Stuart_Armstrong — 2015-03-10T12:00:18.000Z

In equation form, the AI is maximising

P(¬X)∗C+P(X)∗u(X,A)

for some constant C, some unlikely event X that the AI cannot affect, some set of relevant descriptors A, and some utility u. Since C is constant, this is exactly the same as maximising u(X,A) - the probability P(X) is irrelevant.

The whole setup described is simply a way to ensure that if W is the likely set of worlds consistent with observations after ¬X/X, then

P(W)≈P(¬X)≈1 (we "know" that X doesn't happen and that we end up in W),

while

P(W|X)<<1 (in the worlds it cares about, the AI behaves as if W was incredibly unlikely to come about).
Resource gathering and pre-corriged agents
by Stuart_Armstrong — 2015-03-10T11:47:12.144Z

A putative new idea for AI control; index here.

Resource-gathering agent

It will often be useful to have a model of a “pure” resource gathering agent – one motivated only to gather resources, accumulated power, spread efficiently, and so on. This model could be used as behaviour not to emulate, or as a comparison yardstick for the accumulation behaviour of other agents.

The simplest design for a resource gathering agent would be to take a utility function u – one linear in paperclips, say – and give the agent the utility function X(u) + ¬X(-u), where X is some future observation that has 50% chance of occurring, and that the AI cannot affect. Some cosmological fact coming from a distant galaxy (at some point in the future) could do the trick.

This agent would behave roughly as a resource gathering agent, accumulating power in preparation for the day it would know what to do with it: it would want resources (as these could be used to create or destroy paperclips) but would be indifferent to creating or destroying paperclips currently, as the expected gain from u is exactly compensated by the expected loss from -u (and vice versa).

However, its behaviour is not independent of u: if for instance there were a Grand President of the Committee to Establish the Proper Number of Paperclips in the World (GPotCtEtPNoPitW), then the AI would desperately try to secure that position, but would not care overmuch about being the GPotCtEtPNoSitW, who deals with staples.

So a better model of a resource gathering agent is one that has a distribution P over all sorts of different utility functions, with the proviso that for all such utilities u, P(u)=P(-u). Note here that we’re talking about actual utility functions (which can be compared and summed directly), not functions-up-to-affine-transformations. This distribution P will be updated at some future date according to some phenomena outside of the agent’s control.

Then this agent, which currently has exactly zero motivations, will nonetheless accumulate resources in preparation for the day it will know what to do.

There are some distributions P which are better suited to getting a “purer” resource gathering agent (a bad P would be, eg, having a lots of utilities which are tiny variations on u, which is essentially the same as having just u – but “tiny vari...