trentmkelly/LessWrong-43k huggingface dataset:
Human-AI Collaboration
by Rohin Shah — 2019-10-22T06:32:20.910Z
Link: https://bair.berkeley.edu/blog/2019/10/21/coordination/

We’ve just released our paper on human-AI collaboration. The paper makes a straightforward-to-me point that self-play training is not going to work as well with humans in collaborative settings as in competitive settings. Basically, humans cause a distributional shift for the self-play agent. However, in the competitive case, the self-play agent should move towards the minimax policy, which has the nice property of guaranteeing a certain level of reward regardless of the opponent. The collaborative case has no such guarantee, and the distribution shift can tank the team performance. We demonstrated this empirically on a simplified version of the couch coop game Overcooked (which is amazing, I’ve played through both Overcooked games with friends).

As with a

previous post

, the rest of this post assumes that you’ve already read the

blog post

. I’ll speculate about how the general area of human-AI collaboration is relevant for AI alignment. Think of these as rationalizations of the research after the fact.

It’s necessary for assistance games

Assistance games (formerly called

CIRL games

) involve a human and an agent working together to optimize a shared objective that only the human knows. I

think

the general framework makes a lot of sense. Unfortunately, assistance games are extremely intractable to solve. If you try to scale up assistance games as a whole, the

resulting environment

is not very strategically complex, because it’s hard to do preference learning and coordination simultaneously with deep RL. This suggests trying to make progress on subproblems within assistance games.

Usually, when people talk about making progress on “the CIRL agenda”, they are talking about the preference learning aspect of an assistance game. We typically simplify to a single-agent setting and do preference learning, as in learning from

comparisons

or

demonstrations

. However, a useful agent will also need to properly coordinate with the human in order to be efficient. This suggests work on human-AI collaboration. We can work on this problem independently of preference learning simply by assuming that the agent knows the true reward function. This is exactly the setting that we study.

In general, I expect that if one hopes to take an assistance-game-like approach to AI alignment, work on human-AI collaboration will be necessary. T...
2 small problems with the site...
by Long try — 2019-10-22T04:31:23.468Z

They're more like inconveniences, but still.

1. I've set my account setting to display "magical algorithm" comments, but when I read posts I still see them in the order of "oldest". I guess that's because of a discrepancy between those drop-down options: in 1 it's "top scored", in another it's "magical".

2. The truncation of comments works, but sometimes I find it actually making them longer.

Example: the site displays someone's comment: blah blah blah therefore your argument is i... (Read more) (Click to expand thread. Ctrl F to expand all).

I click Read more. Now it is: blah blah blah therefore your argument is invalid.

So I guess your algorithm of how to display comments looks something like this (pardon my below expression, I'm not a programmer at all):

If [comment_length] >150

then display [1st 150 characters of comment] and [...(Read more) (Click to expand thread. Ctrl F to expand all)]

This leads to the problem as described above. When a comment is 157 char long, for example, this truncate method actually increases the length displayed AND leaves readers like me with a weird feeling after clicking.

Here I propose a solution within my ability of thinking:

If [comment_length] >150

then display [1st 75 characters of comment] and [...(Read more) (Click to expand thread. Ctrl F to expand all)]

So... have those 2 minor things above been mentioned before? If not, I'd like to report. If yes, then are you going to look into them and when can they be changed? :)
Testing the Efficacy of Disagreement Resolution Techniques (and a Proposal for Testing Double Crux)
by Ronny Fernandez — 2019-10-21T22:57:37.327Z

Introduction

I will describe a procedure for testing the efficacy of disagreement resolution techniques (DRTs) together with methodologies for inducing their use (induction methods). DRTs are structured games (in the loosest senses of the terms) that involve conversation, and are aimed at helping players who disagree about some topic either figure out the truth about that topic, or come to better understand each others' positions and the sources of their disagreement. An induction method is just any way of trying to get someone to use a DRT.

I am writing up a description of this procedure here because I plan to use it to test and find DRTs, and I would like to get feedback before I go out and do that. I originally came up with this procedure in order to test Double Crux, and I still plan to. I will describe the first step of that plan in the second half of this post. The first half of the post explains the general procedure and some frames I think might be useful for understanding the second half of the post.

I would also like to invite others to use the general procedure in order to test or find other DRTs. It seems fairly obvious to me now, but something something hindsight.

I would like any kind of feedback that you think might make the methodology I describe herein better. I am particularly interested in feedback on the specific procedure I plan to use for testing Double Crux, but feedback on the general procedure would also be great. Any feedback or advice on the statistical analysis of the results of my test procedure for Double Crux would also be much appreciated.

Readers who are primarily interested in giving feedback on my statistical approach are encouraged to skip to the "Procedure" subsection in the second half of the post, and then skip to the "Statistical Analysis and Preregistration" subsection.

General Procedure

Step 0:

Gather participants and filter them for desired characteristics. Ignorance of the DRT to be tested at the onset of the study should be verified for all participants. Researchers may also want to filter participants for having some familiarity with the concept of a probability assignment, or having completed some calibration training.

Inform accepted participants of the nature of the study: that they will be scored according to a proper scoring rule; that they may have to complete a training module, a class, or a workshop; that the...
Let People Move to Jobs
by jefftk — 2019-10-21T18:00:01.772Z

When I argue that we should

bring
rents down by

building

more

housing, one kind of
response I've gotten is:

Why are you trying to move more people into cities? There's lots of
housing available in the US, it's just not in the currently trendy
cities. Instead of moving people we should be moving jobs. Why
couldn't Amazon have put their "second headquarters" in a
deindustrializing city like Milwaukee instead of splitting it between
DC and NYC?

The idea makes sense: these cities were built out for industries that
have moved on, and now they're over-built for current demand. They
have lots of buildings available, both commercial and residential,
along with a lot of other underutilized infrastructure.

One way to look at it, then, is why aren't companies just moving there
on their own? Company rent would fall, they could pay their employees
less for the same standard of living or effectively give them all
large raises by holding pay constant. Everyone would waste less time
in traffic. The host city would be strongly supportive instead of
somewhere between negative and neutral. What's keeping them put?

A major factor is that current employees don't want to move. If
Google announced it was moving everything to Pittsburgh, ~80% of the
company would quit. People put down roots and get attached to areas,
generally more so than to jobs.

New companies, however, don't have employees they would need to move.
Why do tech startups choose the Bay when almost anywhere else would be
cheaper? I see two main answers: to be close to their investors and
to be able to hire from a deep talent pool.

And this gets us to the real problem with "let's distribute jobs
across the country": industries benefit enormously from
centralization. Being in the main city for your industry means, for a
start:

Good ideas flow more freely between organizations

Coordination between organizations is easier

Switching jobs is easier, so people don't get as stuck in
jobs they don't like

Ease of switching also allows people to take larger career
risks

People expected telecommuting to change this, but even though we have
many technical tools that make remote work far more practical than it
was even ten years ago, physically being in the same place as your
coworkers remains extremely valuable.

Industries do vary, however, in how well centralizatio...
Turning air into bread
by jasoncrawford — 2019-10-21T17:50:00.117Z
Link: https://rootsofprogress.org/turning-air-into-bread

Originally posted on The Roots of Progress, August 12, 2017

I recently finished The Alchemy of Air, by Thomas Hager. It's the story of the Haber-Bosch process, the lives of the men who created it, and its consequences for world agriculture and for Germany during the World Wars.

What is the Haber-Bosch process? It's what keeps billions of people in the modern world from starving to death. In Hager's phrase: it turns air into bread.

Some background. Plants, like all living organisms, need to take in nutrients for metabolism. For animals, the macronutrients needed are large, complex molecules: proteins, carbohydrates, fats. But for plants they are elements: nitrogen, phosphorus and potassium (NPK). Nitrogen is needed in the largest quantities.

Nitrogen is all around us: it constitutes about four-fifths of the atmosphere. But plants can't use atmospheric nitrogen. Nitrogen gas, N2, consists of two atoms held together by a triple covalent bond. The strength of this bond renders nitrogen mostly inert: it doesn't react with much. To use it in chemical processes, plants need other nitrogen-containing molecules. These substances are known as “fixed” nitrogen; the process of turning nitrogen gas into usable form is called fixation.

In nature, nitrogen fixation is performed by bacteria. Some of these bacteria live in the soil; some live in a symbiotic relationship on the roots of certain plants, such as peas and other legumes.

Nitrogen availability is one of the top factors in plant growth and therefore in agriculture. The more fixed nitrogen is in the soil, the more crops can grow. Unfortunately, when you farm a plot of land, natural processes don't replace the nitrogen as fast as it is depleted.

Pre-industrial farmers had no chemistry or advanced biology to guide them, but they knew that soil would lose its fertility over the years, and they had learned a few tricks. One was fertilization with natural substances, particularly animal waste, which contains nitrogen. Another was crop rotation: planting peas, for instance, would replace some of the nitrogen in the soil, thanks to those nitrogen-fixing bacteria on their roots.

But these techniques could only go so far. As the world population increased in the 19th century, more and more farmland was needed. Famine was staved off, for a time, by the opening of the prairies of the New World, but those resources were finite. The w...
Why Are So Many Rationalists Polyamorous?

by Wes F — 2019-10-21T16:12:22.215Z

Originally posted at Living Within Reason.

Last week, Jacob Falkovich, of the Putanumonit blog, put up a post trying to figure out why rationalists are disproportionately polyamorous. He notes that about 5% of Americans engage in consensual nonmonogamy, while 17% of Americans in the 2014 Less Wrong survey indicated that they did. My expectation is that the number for both is slightly higher today. In service of this goal, Falkovich developed several theories and surveyed a number of his readers. His results ended up inconclusive.

Since this involves the intersection of the two themes of this blog – rationality and nonmonogamous relationships – I thought I would offer my own theories about why this might be the case. I don’t have any survey data, but if anyone is planning on doing a survey, you may want to include some questions evaluating these theories.

1. THE TRADITIONAL JUSTIFICATIONS FOR MONOGAMY ARE IRRATIONAL

Rationalist try to be rational about everything, so we also try to be rational about relationships. Relationship anarchy is my attempt to derive a rational relationship style from first principals.

While there are some good reasons to be monogamous, anecdotally, the most common justifications I hear for monogamy are jealousy-related. People don’t want open relationships because they would be jealous of their metamours (and often, their partners). But jealousy is just an emotion, and rationalists have a tradition of distrusting emotions. Falkovich somewhat addressed this in his first theory – overcoming intuitions:

A core tenet of Rationality is that what feels true is not necessarily what is true. What feels true may simply be what is pleasant, politically expedient, or what fits your biases and preconceptions. The willingness to entertain the idea that your intuitions about truth may be wrong is a prerequisite for learning Rationality, and Rationality further cultivates that skill.

Unfortunately, Falkovich’s analysis is frustrated by the lack of variance in his survey data on whether people overcome their intuitions. I have a feeling that this result was limited somewhat by the survey questions, which asked participants to rate how much they trusted their intuitions and whether they ever significantly changed their emotions through analysis and introspection.

The difficulty is that there are a...
What does "meta-execution without indirection" look like?
by Liam_Donovan — 2019-10-21T12:59:07.910Z

I've been trying to understand IDA/Factored Evaluation at a deep level, and I find meta-execution especially confusing. The LW post says that it is "HCH + annotated functional programming + a level of indirection", but I'm not sure what the "level of indirection" is doing. To understand it better, I want to know what HCH + Annotated Functional Programming (without indirection) would look like, and how this differs from meta-execution. Any help is much appreciated!
All I know is Goodhart
by Stuart_Armstrong — 2019-10-21T12:12:53.248Z

I've done some work on Goodhart's law, and I've argued that we can make use of all our known uncertainties in order to reduce or remove this effect.

Here I'll look at a very simple case: where we know only one thing, which is that Goodhart's law exists.

Knowing about Goodhart's law

Proxies exist

There are two versions of Goodhart's law, as we commonly use the term. The simplest is that there is a difference between maximising for a proxy -- V -- rather than for the real objective -- U.

Let W=U−V be the difference between the true objective and the proxy. Note that in this post, we're seeing U, V, and W as actual maps from world histories to R. The equivalence classes of them under positive affine transformations are denoted by [U], [V], [W] and so on.

Note that U=V+W makes sense, as does [U]=[V+W], but [U]=[V]+[W] does not: [V]+[W] defines a family of functions with three degrees of freedom (two scalings and one addition), not the usual two for [U] (one scaling and one addition).

So, the simplest version of Goodhart's law is thus that there is a chance for W to be non-zero.

Let W be the vector space of possible W, and let p be a probability distribution over it. Assume further that p is symmetric -- that for all W∈W, p(W)=p(−W).

Then the pernicious effect of Goodhart's law is in full effect: suppose we ask an agent to maximise U=V+W, with V known and p giving the distribution over possible W.

Then, given that uncertainty, it will choose the policy π that maximises

E(U∣π)=E(V+∑W∈Wp(W)∣π)=E(V)+0,

since W and −W cancel out.

So the agent will blindly maximise the proxy V.

We know: maximising behaviour is bad

But, most of the time, when we talk about Goodhart's law, we don't just mean "a proxy exists". We mean that not only does a proxy exist, but that maximising the proxy too much is pernicious for the true utility.

Consider for example a nail factory, where U is the number of true nails produced, and V is the number of "straight pieces of metal" produced. Here W=U−V is the difference between the number of true nails and the pieces of metal.

In this case, we expect a powerful agent maximising V to do much, much worse on the U scale. As the agent expands and gets more control over the world's metal production, V continues to climb, while U tumbles. So V is not only a (bad) proxy for U; at the extremes, its pernicious.

But what if we considered U′=V−W? This is an odd utili...
Where to absolutely start?
by Long try — 2019-10-21T02:45:47.562Z

Hi there!

I'm a new member and now feeling giddy from the amount of contents presented here on the site. I've read those welcome and FAQ posts, they all point to the library's core reading as material for beginners. BUT... I've just finished the Preface post of R:A-Z, and from the author's very words, it looks like this series focus more on the big, vague things than those hands-on lessons.

So my questions are: among it and the Codex and HPMOR, which is the most newbie-friendly? In your opinion, what are the best 9 sequences rookies like me should read to get to a somewhat intermediate level, in order?

Thanks & cheers!
Healthy Competition
by Raemon — 2019-10-20T20:55:48.265Z

A few years ago, the rationalsphere was small, and it was hard to get funding to run even one organization. Spinning up a second one with the same focus area might have risked killing the first one.

By now, I think we have the capacity (financial, coordinational and human-talent-wise) that that's less of a risk. Meanwhile, I think there are a number of benefits to having more, better, friendly competition.

Reasons competition seems good

Diversity of worldviews is better.

Two research orgs might develop different schools of thought that lead to different insights. This can lead to more ideas as well as avoiding the tail risks of bias and groupthink.

Easier criticism.

When there's only one org doing A Thing, criticizing that org feels sort of like criticizing That Thing. And there may be a worry that if the org lost funding due to your criticism, That Thing wouldn't get done at all. Multiple orgs can allow people to think more freely about the situation.

Competition forces people to shape up.

If you're the only org in town doing a thing, there's just less pressure to do a good job.

"Healthy" competition enables certain kinds of integrity.

Sort of related to the previous two points. Say you think Cause X is real important, but there's only one org working on it. If you think Org A isn't being as high integrity as you'd like, your options are limited (criticize them, publicly or privately, or start your own org, which is very hard. If you think Org A is overall net positive you might risk damaging Cause X by criticizing it. But if there are multiple Orgs A and B working on Cause X, there are less downsides of criticizing it. (Alternate framing is that maybe criticism wouldn't actually damage cause X but it may still feel that way to a lot of people, so getting a second Org B can be beneficial). Multiple orgs working on a topic makes it easier to reward good behavior.

In particular, if you notice that you're running the only org in town, and you want to improve you own integrity, you might want to cause there to be more competition. This way, you can help set up a system that creates better incentives for yourself, that remain strong even if you gain power (which may be corrupting in various ways)

Concerns re: Concentration and Monopolies

Some types of jobs benefit from concentration.

Communication platforms (or,

...
Welcome to SSC Meetup Philadelphia [Edit With Your Details]
by Wes F — 2019-10-20T20:29:25.606Z

(The following are our suggestions for what kind of information is best to include in the welcome post of your group, feel free to replace them with whatever you think is best)

What kind of events does your group usually run? What does it usually do?

How frequently does your group organize events or meet?

Who would be a good fit for you group?

Should they have any particular skills or have done some specific background reading?
We tend to forget complicated things
by Rafael Harth — 2019-10-20T20:05:28.325Z

One consistent pattern I've noticed in studying math is that, if some material feels very difficult, then I might remember it in an upcoming exam, but I will almost certainly have forgotten most of it one year later. The success story behind permanent knowledge gain is almost always "this was hard once but now it's easy, so obviously I didn't forget it" and almost never "I successfully memorized a lot of complicated-feeling things."

I think this also applies outside of mathematics. If it's roughly correct, then the most obvious consequence is to adapt your behavior when you're learning something. Provided that your goal is to improve your understanding permanently by understanding the material conceptually (which, of course, may not be the case), either study until it gets easy, or decide it's not worth your time at all, but don't stop when you've just barely understood it.

I've violated this rule many times, and I think it has resulted in some pretty inefficient use of time.
The Effect pattern: Transparent updates in Elm
by philh — 2019-10-20T20:00:01.101Z

I don't like writing tests in Elm1. Naively, it seems like the fact that all Elm functions are pure would make it easy. But I've still never really been able to test the things I really want to.

It's possible that I'm not very good at writing tests in general. But even if that's true, I don't think it's the whole story.

I have another hypothesis, which is: much of the reason is that although Elm functions are pure, in the sense of "given the same input, they return the same output" - they aren't necessarily transparent, in the sense of "you can look at the output and verify that it's correct".

To recap2, I would describe the Elm architecture as consisting of:

Model - a type describing the data you're interested in. This is usually mostly transparent. There are some opaque blobs where components haven't exposed their internals, but I think those mostly aren't the kind of opacity that gets in the way3.

Msg - a type describing the messages your app passes around. Again, mostly transparent, with opaque blobs where components haven't exposed their messages. I feel like this opacity is maybe a bit more annoying, but still not very.

update : Msg -> Model -> (Model, Cmd Msg) - a function describing the logic of your app (or whatever it is you're architecting). The Model and Msg are still transparent, but the second return value is a Cmd Msg which is completely opaque, and which is the main source of the problem.

view : Model -> Html Msg - a function describing the rendering of your app. Html Msg is also arguably opaque, but you can inspect it using elmer4, or (as I discovered while writing this post) the Test.Html modules in the test package. I'm not sure if you can inspect it as deeply as you might want, I haven't explored these much yet.

Now, in my experience5, a lot of the logic in an app is contained in the way the update function chains. "After receiving this message, we send a command which eventually results in receiving that message, and then we want to put some value in the model." And we can't test those chains, because Cmd is opaque. So that's part of why I haven't liked writing Elm tests.

But I think I've found a method that will help, at least somewhat. You don't need to rewrite anything except your update function, and I actually think it's a mild improvement even if you ignore testing. I call it the Eff...