trentmkelly/LessWrong-43k huggingface dataset:
Writing Collaboratively
by richard_reitz — 2016-06-18T19:47:04.884Z

This is a summary of the customs for collaborative writing the team on the fanfiction In Fire Forged came to, after a fair amount of time and effort figuring things out. The purpose of this piece is to share our results, thereby saving anyone who wants to write collaboratively the cost of experimentation. Obviously, different writing projects will accomplish different things with different people, and will therefore be best served by different practices. Take this as a first approximation, to be revised by experience.

Google Docs

We tried a bunch of platforms for collaboration, and found Google Docs to best fit our needs.

Create a Google Doc. Multi-installment affairs may consider creating a folder and make one doc per installment.

Enable editing. Collaborators are not very helpful if they can't provide feedback.
Google Docs allows authors to restrict the changes other people can make to "suggestions" and "comments" by switching to "suggesting" mode.
In general, the author restricts collaborator permissions to comments and suggestions. How to control these permissions should be described in the "enable editing" link above.

Distribute link to collaborators.

Once the collaborators have the link, they read through it, making the comments and suggestions they think of. Google Docs does a good job facilitating discussion of this feedback; utilize this!

Micro and Macro

We found it useful to distinguish between what we were saying and how we were saying it. We termed the former "macro" and the latter "micro". This allows authors to say things like "I'm mostly looking for micro suggestions, although I'd be interested in any glaring macro errors (anything untrue or major omissions)." This succinctly communicates that collaborators should mostly restrict themselves to suggesting changes to how the author is communicating, which usually consists of small edits concerning things like technical issues (typos, omitted words, grammar) and smoother communication (word choice, resolving ambiguities, sectioning).

This contrasts macro suggestions, which would include (in nonfiction) things like making sure factual claims were true, being sure to include all relevant information, and the perspective from a different field. (In fiction, macro suggestions would include things such as plot, characterization, chapter structure and consistency of the universe.)

In general, you want ...
Questioning GLS-Coherence
by abramdemski — 2016-06-18T18:49:54.000Z

At the end of my trolling post, I said that naturalistic logical updates stand or fall with the strength of the reflective consistency argument for them, and that this has not been sufficiently articulated yet. Indeed, the choice of GLS-coherence as a model of bounded rationality is highly questionable. While previous posts have discussed the motivation a fair amount, I was writing with the hopes of providing a solution, so I added certain assumptions that I thought might be sufficient, if not necessary. The trolling theorem, among other concerns, casts doubt on those assumptions. Here, I describe where I think I was wrong.

Are the updates naturalistic?

My argument for using GLS-coherence was that the agent, planning ahead for the possibility that x is proved, should be coherent with respect to new propositional constraints which x contains. GLS formalizes this as □x→x, which allows us to conclude that a GLS-coherent prior is coherent with respect to x when updated on □x. Coherence updating on □x isn't really what my argument asked for, though: I asked for coherence conditioning on future-self having access to a proof of x. A GLS-coherent prior isn't good enough to fill in this inferential gap on its own. It can imagine future-self having access to a proof of x without □x being true.

Hence, my updates were not really "naturalistic"; they were not able to look at the imagined future world and extract □x. In order for the UDT to make plans which depend on whether □x holds, it would give the plans direct access to the theorem prover in a non-naturalistic way (it would trust its own theorem prover in a special way which wouldn't automatically extend to other theorem provers in the environment).

I think this objection isn't totally fatal to GLS-coherence; it means there has to be special attention to the "pipe" which connects □x with actually-having-proven-x in the overall decision theory. However, it does suggest that we might be able to accomplish the job easier without GLS-coherence, only using this "pipe" between proof and belief.

Bayes-Endorsement ≠ Planning-Endorsement

The two-update problem is the observation that in systems of logical probabilities so far, there seems to be a non-Bayesian "second update" which adjusts beliefs based on proofs. (Let's call this the "logical update".) I've argued in previous posts that there is a strong connection between refl

...
The escape duty
by KatjaGrace — 2016-06-18T18:43:00.000Z

I’m going to explain one of my favorite life-improvement techniques over the past couple of years.

I thought of it as a result of talking to Ben Hoffman. He mentioned some innovation that worked for him, and sounded impossible for me. I think it was ‘regularly reflecting on what you are doing and how it could be better’ or something vague and virtuous like that. I’m a big fan of reflecting on one’s life and how to improve it, but doing it at really appropriate times seemed hard because often I’m distracted by other things, especially when things are going badly somehow. ‘Things could be better’ is not a very salient trigger upon which to act. And I had been struggling to allocate time to reflect on my life even when I actually put it in my plan for the day.

But then I realized that there was a thing I already wanted to do exactly when things were going badly—play a computer game. At the time it was a game I shall call SPP.

So I set these rules:

I am not ever allowed to play SPP unless I have first gone to the place on my computer where I reflect, and written anything at all about what is going on in my life and how it could be better.

If I reflect, I may then play SPP for five minutes.

This could be repeated arbitrarily often. Like, I can just swap back and forth between reflecting and playing SPP all afternoon if I want.

Consequently, every time the rest of my life became less appealing than playing SPP, I would briefly think about what was going wrong, and try to fix it. It is easy to remember to play a computer game. It is also easy (for me at least) to remember that I must not do a thing that I often want to do—much more so than it is to remember that I should do a thing that I rarely think of.

This system has worked really well for me I think. If I am feeling bad in any way, I’m very willing to reflect for an arbitrarily short time in order to be blamelessly playing a computer game for five minutes. And once I’m reflecting, I almost always do it for long enough to make a number of concrete improvements to the situation (e.g. put on noise-cancelling headphones and find some painkillers, or think of some way to make the task at hand less complicated). And I feel like this usually helps. Sometimes I become more physically comfortable. Sometimes I realize I should be doing some other activity entirely...
Cooperative Inverse Reinforcement Learning vs. Irrational Human Preferences
by orthonormal — 2016-06-18T00:55:10.000Z

At the MIRI colloquium series, we've been discussing at length the new proposal of cooperative inverse reinforcement learning (henceforth CIRL) as an approach to value learning. Essentially, this approach treats human-AI interaction as a cooperative game where the human, but not the AI, observes the parameters of the reward function, and thus the AI seeks to infer those parameters and act so as to maximize the unobserved reward.

Dylan Hadfield-Menell gave a talk (slides here) about applications of CIRL to corrigibility, showing in a toy example how the value of information can avert instrumental incentives for the AI to prevent (or force) its own shutdown. Crucially, this can work even if the human behaves suboptimally, for instance if the human is noisily rational or slightly biased towards some actions.

This is really awesome! I think it counts as substantial progress on the architecture of value learning. But the best thing to do with any new idea is to see where it breaks, so in the colloquium workshops since Dylan's talk, we've been discussing some scenarios in which CIRL behaves improperly in the limit, even if it has a correct model of the world and a prior assigning positive probability to the best reward function.

These failures come from a common source: in CIRL, the rewards and updates make use of an assumption that the human is behaving according to the optimal game-theoretic strategy given the true reward function, and this assumption is still used to determine actions even if the AI's world model knows it to be a false assumption. If the human is noisily rational or slightly biased, the AI will still update towards the correct reward function; but other types of human irrationality can cause it to update dangerously.

Example 1: Circular Preferences

Human beings are not, in fact, rational agents, and there are situations in which our preferences can be circular: we prefer A to B, B to C, and C to A, if we are making binary choices. (For instance, people often have circular preferences in Allais Paradox games or other cases of managing risks.) Let us suppose that we have such a case: as above, the human will pick A > B > C > A if presented with two adjacent options. Furthermore, if the human is given all three options, they will pick A:B:C in the proportions 1:2:3.

Say that the AI's actions are to either present the human with A, B, and C to

...
Crazy Ideas Thread
by James_Miller — 2016-06-18T00:30:49.892Z

This thread is intended to provide a space for 'crazy' ideas. Ideas that spontaneously come to mind (and feel great), ideas you long wanted to tell but never found the place and time for and also for ideas you think should be obvious and simple - but nobody ever mentions them.

Rules for this thread:

Each crazy idea goes into its own top level comment and may be commented there.

Voting should be based primarily on how original the idea is.

Meta discussion of the thread should go to the top level comment intended for that purpose.
Weekly LW Meetups
by FrankAdamek — 2016-06-17T16:22:53.595Z

This summary was posted to LW Main on June 17th. The following week's summary is here.

Irregularly scheduled Less Wrong meetups are taking place in:

Baltimore weekly meetup: 19 June 2016 08:00PM
European Community Weekend: 02 September 2016 03:35PM
San Antonio Meetup: 19 June 2016 02:00PM
Sao Paulo - Meetup de junho: 25 June 2016 02:00PM

The remaining meetups take place in cities with regular scheduling, but involve a change in time or location, special meeting content, or simply a helpful reminder about the meetup:

San Francisco Meetup: Fun and Games: 20 June 2016 06:15PM
San Jose Meetup: Park Day (IV): 19 June 2016 03:00PM
Sydney Rationality Dojo - July: 03 July 2016 04:00PM
Vienna: 18 June 2016 03:00PM
Washington, D.C.: History of Science Fiction Fandom: 19 June 2016 03:00PM

Locations with regularly scheduled meetups: Austin, Berlin, Boston, Brussels, Buffalo, Canberra, Columbus, Denver, Kraków, London, Madison WI, Melbourne, Moscow, New Hampshire, New York, Philadelphia, Research Triangle NC, San Francisco Bay Area, Seattle, Sydney, Tel Aviv, Toronto, Vienna, Washington DC, and West Los Angeles. There's also a 24/7 online study hall for coworking LWers and a Slack channel for daily discussion and online meetups on Sunday night US time.

If you'd like to talk with other LW-ers face to face, and there is no meetup in your area, consider starting your own meetup; it's easy (more resources here). Check one out, stretch your rationality skills, build community, and have fun!

In addition to the handy sidebar of upcoming meetups, a meetup overview is posted on the front page every Friday. These are an attempt to collect information on all the meetups happening in upcoming weeks. The best way to get your meetup featured is still to use the Add New Meetup feature, but you'll also have the benefit of having your meetup mentioned in a weekly overview. These overview posts are moved to the discussion section when the new post goes up.

Please note that for your meetup to appear in the weekly meetups feature, you need to post your meetup before the Friday before your meetup!

If you check Less Wrong irregularly, consider subscribing to one or more city-specific mailing list in order to be notified when an irregular meetup is happening: Atlanta, Cambridge UK, Chicago, Cincinnati, Cleveland, Frankfurt, Helsinki, Marin CA, Ottawa, Pi...
Meteuphoric games
by KatjaGrace — 2016-06-17T12:00:55.000Z

Back to the original, titular purpose of this blog: getting excited about how things are metaphors for other things. Here are some things that are structurally similar in that they are games where you try to find structural similarities.

They also share the properties of being amusing to me and not generally known, so I thought I’d share them. I don’t know if they are generally amusing—if you try one, feel free to leave a comment about whether you enjoyed it or not.

1. My thought is like…

This is a game I learned as a child. It goes like this:

Kate: [thinks of a thing] My thought is like –

Sarah: Sam Smith

Robert: the word of God

Kate: actually it was a palm tree

Sarah: Hmm, but Sam Smith is basically just like a palm tree. They are both tall, covered with a sparse brown fur, and appreciate dryness. Also, you’ve been thinking about them lately. Also, you are disproportionately interested in getting your hands on their nuts.

Robert: But palm trees are well known to symbolize the word of God. Probably because when you are lost in a metaphorical desert, God’s word guides you to the oasis of salvation. That’s why the faithful carry them on Palm Sunday.

Kate: Sarah makes the best case.

Sarah: [thinks of a thing] My thought is like-

2. Crosswords where every clue points to two words that fit in the crossword

Really, making them is the game that most involves finding structural similarities in things—solving them involves the opposite.

For instance, what sentence could you use to describe either of these words at once?

CALVE and CARVE. Maybe, ‘an action that increases the matching items of cow you have’

NET and SET. Maybe, ‘Contains things that share a certain property’

LOADED and LOANED. ‘His family could drive a Tesla because it was ——‘

Here’s one I made earlier, but it’s not great. Here’s a famous crossword that probably inspired this activity indirectly.

3. Explain everything in terms of status

One player asks for an explanation of some phenomenon. Others give answers that make the phenomenon largely about status. The first player chooses the best explanation. For warning, I made this game up and have not played it properly. Improper variants are amusing to me, but I’m not sure if to others.

Example: Why do we sleep?

Evolution bothered to develop eyes and

...
Avoiding strawmen
by casebash — 2016-06-17T08:20:51.673Z

George Bernard Shaw wrote that, "the single biggest problem in communication is the illusion that it has taken place". Much of strawmanning is unconscious. One person says that it is important to be positive, the other person interprets this as it being important to be positive in *all* circumstances, when they are merely making a general statement.

I would suggest that a technique to avoid accidentally strawmanning someone would be to begin by intentionally strawmanning them and then try to back off to something more moderate from there.

Take for example:

"Just be yourself"

A strawman would be, "Even if you are a serial killer, you should focus on being yourself, than changing how you behave".

Since this is a rather extreme strawman, backing off to something more moderate from here would be too easy. We might very well just back off to another strawman. Instead, we should backoff to a more reasonable strawman first, then backoff to the moderate version of their view.

The more moderate strawman, "You should never change how you act in order to better fit in"

When we back off to something more moderate, we then get, "Changing how you act in order to better fit in is generally not worth it"

You can then respond to the more moderate view. If you had responded to the original, you might have pointed out a single case when the principle didn't hold, such as making a change that didn't affect one's individuality (i.e showering regularly) and used it to attack the more general principle. When you have the more moderate principle, you can see that such a single example only negates the strict reading, not the more moderate reading. You can then either accept the moderate reading or add arguments about why you also disagree with it. If you had skipped this process, you might have made a specific critique and not realised that it didn't completely negate the other person's argument.
AIs in virtual worlds: discounted mixed utility/reward
by Stuart_Armstrong — 2016-06-17T06:43:01.000Z

A putative new idea for AI control; index here.

In a previous post on AIs in virtual worlds, I described the idea of a utility function that motivates the AI to do operate in a virtual world with certain goals in mind, but to shutdown immediately if it detects that the outside world -- us -- is having an impact in the virtual world.

This is one way to implement such a goal, given certain restrictions on the AI's in-world utility. The restrictions are more natural for rewards rather than utilities, so they will be phrased in those terms.

Worlds W and W′

The world W is deterministic, turn-based, and includes the AI; it is the virtual world we want the AI to care about. The world set W′ is W plus the possibility of stochastic influences (which is how the AI would model our influence in its world); see here for more details.

The AI has a shutdown action S; the idea is that the AI implements S when P(W′) gets high enough. We assume that the probability module P of the AI is reflexively consistent in some suitable fashion.

Discounted reward results

Let the AI have a discount rate of 0<γ<1, and write f(p) for the function (1−p)/p; note that f is diminishing in p.

Theorem 1: For all 0≤q<p≤1 such f(p)/f(q)<(1−γ), there exists a discounted reward function such that the AI will shutdown whenever P(W′)>p and will never shutdown while P(W′)<q.

Note that a lower γ -- a sharper discount -- means that p and q can be chosen to be closer together.

Corollary 1: For all 0≤q<p≤1 such that there exists an n with f(p)/f(q)<(1−γn), there exists a discounted reward function such that the AI will shutdown whenever P(W′)>p for the last n turns, and will never shutdown while P(W′)<q for the last n turns.

Note that Corollary 1 allows q and p to be closer together than Theorem 1 allows - the price being that the probability needs to stay above or below the bounds for longer.

Corollary 2: For all 0<γ<1, there exist 0<q<p<1 with the properties of Theorem 1.

Proof

Proof by construction of a suitable reward function.

Let T be the time that the AI shuts down (if it never shuts down t=∞), and rt the reward it gets on turn t, which is bounded between 0<a<1 and 1. It gets reward R′ for shutting down in W′, and has a discount rate of γ. Define IW′ as the indicator function for being in world W′. Then the AI's total reward is defined to be:

...
Secret Rationality Base in Europe
by SquirrelInHell — 2016-06-17T02:50:11.083Z

In short, I'm wondering what place/group/organisation/activity could do for rationality in Europe what Berkeley does for rationality in the US?

Soon, we'll have LWCW in Berlin, which I hope will be an occasion to do some networking among people who think seriously about developing rationality communities. But in the meantime, let's do some brainstorming.

Important note: in comments to this post, please use only consequentialist language. For example, say "If we decided for the base to be on Malta, then X would happen" instead of "I think it should be in Malta, because..."

What would happen if the rationality base was located in [insert specific city/country]?

What could such a place offer to you now, that would make you consider a temporary/permanent move?

What would happen if the European rationality community efforts were centered around some particular research topic (e.g. AI)?

Is there something you can think of that would speed up community-building in Europe?

Of course, share anything else that you think is relevant to the topic.

Also, see you all in Berlin :)
General Cooperative Inverse RL Convergence
by janleike — 2016-06-17T02:26:01.000Z

This is brief technical note on how to get convergence in the cooperative inverse reinforcement learning framework.
We extend cooperative inverse RL to partially observable domains and
use a recent result on the grain of truth problem to establish (arguably very strong) conditions to get convergence to ε-Nash equilibria.

Credit: this came out of the CSRBAI Workshop on Preference Specification. Several people contributed ideas,
in particular Vadim Kosoy.

Preliminaries

We use the setup and notation from the grain of truth paper
(Leike, Taylor, and Fallenstein, 2016).
We only review the most important notation here
in an attempt to make this post notationally fairly self-contained.
The set MOrelf denotes a countable class
of stochastic environments, the class of all environments
that are reflective-oracle-computable
(computable on a probabilistic Turing machine with access to a reflective oracle).

Let σ be any two-player environment.
Let H denote the human player and let R denote the robot player.
Each player interacts with the environment in cycles:
at time step t the player chooses an action at∈A and
receives a percept et∈E
the cycle then repeats for t+1.
A history is an element of (A×E)∗.
We use æ∈A×E to denote one interaction cycle,
and æ<t to denote a history of length t−1.
We assume the sets A and E are finite.

The human follows a policy
πH:(A×E)∗→ΔA and the robot follows a policy
πR:(A×E)∗→ΔA.
The human acts in the subjective environment
σH:(A×E)∗×A→ΔE
(environment σ combined with the robot) and
the robot acts in the subjective environment
σR:(A×E)∗×A→ΔE
(environment σ combined with the human).
Each player does not observe the action and percept of the other player directly.

Moreover, only the human sees the reward signal,
not the robot.
Yet they both try to maximize this signal;
in this sense they are playing a cooperative game.
We assume that the reward is uniquely determined by the robot's history
(the robot has all the necessary information available to determine the reward).
The robot has a prior P over reward functions
that includes the true reward function.

One question is how to get a reward signal to the robot.
We assume that the robot maximizes the belief reward signal:
For any prior P on reward functions,
the robot can calculate at every time step the P-expected reward obtained.
We let the robot maximize the belief reward signal.
This is of cour

...