trentmkelly/LessWrong-43k huggingface dataset:
[AN #134]: Underspecification as a cause of fragility to distribution shift
by Rohin Shah — 2021-01-21T18:10:06.783Z

Alignment Newsletter is a weekly publication with recent content relevant to AI alignment around the world. Find all Alignment Newsletter resources here. In particular, you can look through this spreadsheet of all summaries that have ever been in the newsletter.

Audio version here (may not be up yet).

Please note that while I work at DeepMind, this newsletter represents my personal views and not those of my employer.

HIGHLIGHTS

Underspecification Presents Challenges for Credibility in Modern Machine Learning (Alexander D'Amour, Katherine Heller, Dan Moldovan et al) (summarized by Rohin): This paper explains one source of fragility to distributional shift, which the authors term underspecification. The core idea is that for any given training dataset, there are a large number of possible models that achieve low loss on that training dataset. This means that the model that is actually chosen is effectively arbitrarily chosen from amongst this set of models. While such a model will have good iid (validation) performance, it may have poor inductive biases that result in bad out-of-distribution performance.

The main additional prediction of this framing is that if you vary supposedly “unimportant” aspects of the training procedure, such as the random seed used, then you will get a different model with different inductive biases, which will thus have different out-of-distribution performance. In other words, not only will the out-of-distribution performance be worse, its variance will also be higher.

The authors demonstrate underspecification in a number of simplified theoretical settings, as well as realistic deep learning pipelines. For example, in an SIR model of disease spread, when we only have the current number of infections during the initial growth phase, the data cannot distinguish between the case of having high transmission rate but low durations of infection, vs. a low transmission rate but high durations of infection, even though these make very different predictions about the future trajectory of the disease (the out-of-distribution performance).

In deep learning models, the authors perform experiments where they measure validation performance (which should be relatively precise), and compare it against out-of-distribution performance (which should be lower and have more variance). For image recognition, they show that neural net ...
Counterfactual control incentives
by Stuart_Armstrong — 2021-01-21T16:54:59.309Z

Co-authored with Rebecca Gorman.

In section 5.2 of their Arxiv paper, "The Incentives that Shape Behaviour", which introduces structural causal influence models and a proposal for addressing misaligned AI incentives, the authors present
the following graph:

The blue node is a "decision node", defined as where the AI chooses its action. The yellow node is a "utility node", defined as the target of the AI's utility-maximising goal. The authors introduce this graph to introduce the concept of control incentives; the AI, given utility-maximizing goal of user clicks, discovers an intermediate control incentive: influencing user options. By influencing user opinions, the AI better fulfils its objective. This 'control incentive' is graphically represented by surrounding it in dotted orange.

A click-maximising AI would only care about user opinions indirectly: they are a means to an end. A amoral social media company might agree with the AI on this, and be ok with it modifying user opinions to achieve higher clicks/engagement. But the users themselves would object highly to this; they do not want the algorithm to have a control incentive for changing their opinions.

Carey et al designed an algorithm to remove this control incentive. They do this by instructing the algorithm to choose its posts, not on predictions of the user's actual clicks - which produce the undesired control incentive - but on predictions of what the user would have clicked on, if their opinions hadn't been changed.

In this graph, there is no longer any control incentive for the AI on the "Influenced user opinions", because that node no longer connects to the utility node.

Call this construction a "counterfactual incentive". This substitutes the (model of) the users' original opinion where the influenced user opinions used to go. A more detailed description of such an algorithm can be found in this footnote[1]. In words, the process can be phrased:

The counterfactual incentive is what the utility would have been, if the influenced opinions were replaced with the original opinions (everything else being equal).

It seems to neutralise a vicious, ongoing cycle of opinion change in order to maximize clicks. But, as this post shows, there are issues with this construction and, if used naively, counterfactual modifications will fail.

Matching players and buying items

In order to analyse the issues with th...
Covid 1/21: Turning the Corner
by Zvi — 2021-01-21T16:40:00.941Z

Aside from worries over the new strains, I would be saying this was an exceptionally good week.

Both deaths and positive test percentages took a dramatic turn downwards, and likely will continue that trend for at least several weeks. Things are still quite short-term bad in many places, but things are starting to improve. Even hospitalizations are slightly down.

It is noticeably safer out there than it was a few weeks ago, and a few weeks from now will be noticeably safer than it is today.

Studies came out that confirmed that being previously infected conveys strong immunity for as long as we have been able to measure it. As usual, the findings were misrepresented, but the news is good. I put my analysis here in a distinct post, so it can be linked to on its own.

We had a peaceful transition of power, which is always a historic miracle to be celebrated.

Vaccination rollout is still a disaster compared to what we would prefer, with new disasters on the horizon (with several sections devoted to all that), but we are getting increasing numbers of shots into increasing numbers of arms, and that is what matters most. In many places we have made the pivot from ‘plenty of vaccine and not enough arms to put shots into’ to the better problem of ‘plenty of arms to put vaccine into, but not enough shots.’ Then all we have to do is minimize how many shots go in the trash, including the extra shots at the bottom of the vial, and do everything we can to ramp up manufacturing capacity. Which it seems can still be meaningfully done.

The problem is that the new strains are coming.

The English strain will arrive first, within a few months. That’s definitely happening, and the only question is how bad it’s going to get before we can turn the tide. We are in a race against time.

The South African and Brazillian strains are not coming as fast, but are potentially even scarier. There are signs of potential escape from not only vaccination but previous infection, potentially allowing reinfection to take place. See the section on them for details, and if you can help provide better information, please do so. We need clarity on this, and we need it badly.

There are also all the other new strains being talked about, which are probably nothing, but there’s always the chance that’s not true.

But first, the good news, and it is very, very good. Let’s run the numbers.

The Numbers

...
The Problem of the Criterion
by Gordon Seidoh Worley — 2021-01-21T15:05:41.659Z

I keep finding cause to discuss the problem of the criterion, so I figured I'd try my hand at writing up a post explaining it. I don't have a great track record on writing clear explanations, but I'll do my best and include lots of links you can follow to make up for any inadequacy on my part.

Motivation

Before we get to the problem itself, let's talk about why it matters.

Let's say you want to know something. Doesn't really matter what. Maybe you just want to know something seemingly benign, like what is a sandwich?

At first this might seem pretty easy: you know a sandwich when you see it! But just to be sure you ask a bunch of people what they think a sandwich is and if particular things are sandwiches are not.

Uh oh...

from the internet

You've run headlong into the classic problem of how to carve up reality into categories and assign those categories to words. I'll skip over this part because it's been addressed to death already.

So now you've come out the other side accepting that "sandwich", like almost all categories, has nebulous boundaries, and that there is no true sandwich of which you can speak.

Fine, but being not one easily deterred, you come up with a very precise, that is a mathematically and physically precise, definition of a sandwich-like object you call an FPS, a Finely Precise Sandwich. Now you want to know whether or not something is an FPS.

You check the conditions and it all seems good. You have an FPS. But wait! How do you know each condition is true? Maybe one of your conditions is that the FPS is exactly 3 inches tall. How do you know that it's really 3 inches tall?

Oh, you used a ruler? How do you know the ruler is accurately measuring 3 inches? And furthermore, how do you know your eyes and brain can be trusted to read the ruler correctly to assess the height of the would-be FPS? For that matter, how do you even know what an inch is, and why was 3 inches the true height of an FPS anyway?

Heck, what does it even mean to "know" that something is "true"?

If you keep reducing like this, you'll eventually hit bottom and run into this question: how do you know that something is true? Now you've encountered the problem of the criterion!

The Problem

How do you know something is true? To know if something is true, you have some method, a criterion, by which you assess its veracity. But how do you know this criterion is itself true? Oh, you have a method,...
Would most people benefit from being less coercive to themselves?
by Matt Goldenberg — 2021-01-21T14:24:17.187Z

Non-coercive motivation is a motivation system that doesn't rely on forcing yourself, pressuring yourself, shaming or guilting (I explain more of what it means here).

Would most people benefit from being less coercive to themselves?

I sort of want to start answering straw versions of the question, because I think that they do answer real fears that people have when they hear about non-coercion and have the immediate reaction "You want me to just drop everything and do what I feel like!?"

Should you drop everything and focus on non-coercion?

So the first answer to the question as it is posed is "Of course not!" Goals are highly contextual. If you're the president of the United States, I think the goal of running the country is probably more valuable then processing non-coercion. Similarly, if you're living paycheck to paycheck at a grueling job you hate, it's probably not a good thing to try to mess with your fundamental motivation systems, it could very well lead to negative consequences if you don't find a new positive/stable motivation system.

Should you focus on non-coercion if you have enough slack to do so?

Now let's answer a less straw version of the question. Let's say a person has lots of slack and there won't be any immediate negative consequences to themselves or others. In that case, do I think it would be beneficial to switch to non-coercive motivation? It depends. Firstly, what do you care about? If it's mostly just about feeling good and having a good state (I often call this an enlightenment orientation) then yes, I do think this would be a good use of your time.

If instead, you mostly care about achievement/saving the world (I often call this a heaven orientation), then you have to look at at your current productivity. If you're already highly productive, it takes a significant investment to switch your strategy for little or no gain.Elon Musk should not focus on non-coercion. On the other hand, if you are a frequent procrastinator, or find you have wildly oscillating motivation, non-coercive motivation could be a highly effective strategy to make you more productive.

Now what if what mostly care about is shaping yourself and becoming your version of the ideal (I sometimes call this an actualization orientation). In that case, it's going to depend quite a bit on your vision of your ideal self. In some cases it will be worth it, others not...
The Multi-Tower Study Strategy
by DirectedEvolution — 2021-01-21T08:42:20.807Z

Boxed Topics, Jenga Towers, And The Spacing Effect.

An undergraduate class on molecular biology teaches you about DNA transcription, the Golgi apparatus, cancer, and integral membrane proteins. Sometimes, these sub-topics are connected. But most often, they're presented in separate chapters, each in its own little box. So let's call these Boxed Topics.

The well-known Stewart calculus textbook teaches you about functions in chapter 1, limits and the definition of derivatives in chapter 2, rules for derivatives in chapter 3, and the relationship of derivatives with graphs in chapter 4. Woe betide you if you weren't entirely clear on the definition of a derivative when it gets used, over and over again, in next week's proofs of derivative rules.

Taking a calculus class can be like building a Jenga Tower. If you've never played it, Jenga is a game where two players share a tower of wooden blocks in crisscrossing layers of three blocks. They take turns removing one block out of middle of the tower, and putting it on top. The towers get taller and less stable every turn. The person who collapses the tower loses.

In a Jenga Tower Topic, the blocks of last week's memories get pulled out, and placed back on top of the tower as you learn new material that depends on them. You just hope it doesn't collapse before the end of the class.

One is not necessarily easier than the other. A Boxed Topic that's jam-packed with disorganized information can be a lot harder than a well-structured and manageable Jenga Tower.

But all else being equal, Jenga Towers have trouble dealing with the spacing effect. This is the well-established finding that people build memories better by spacing out their practice, rather than massing it all together at once.

This might seem counterintuitive. With a Boxed Topic like molecular biology, you get all your practice on the Golgi apparatus in week 5, and all your practice on cancer in week 12. That's a perfect example of massed practice. It seems like Boxed Topics should have the most trouble with the spacing effect.

By contrast, Jenga Towers, like calculus, give you sustained practice with the early material. You learn about functions in week 1, and you continue using them throughout the course. You learn the definition of the derivative in chapter 2, and it keeps popping up over the long term. That sounds like a built-in form of spaced practice, which s...
A ghost
by Joe Carlsmith — 2021-01-21T07:14:05.298Z

(Cross-posted from Hands and Cities)

This post describes a type of thought experiment I sometimes perform in thinking about what to do. I find it a helpful tool for stepping back from what’s immediately salient to me. It’s mostly just a somewhat hokey variant on a classic type of thought, and I’m not sure how helpful it will be to others, but I thought I’d share anyway.

I.

I imagine a ghost that slips from my body and floats out into the world. The ghost is invisible, and it can move freely in space and time. Often I imagine it able to fast-forward, slow-down and rewind time at whatever speed it chooses; and sometime, as capable of some more comprehensive vision of the fabric of space-time itself — able to see time and space arrayed before it like a kind of pool, to fly from one part to another, to zoom in and out, to dip in and out at will.

(Adam Magyar’s Stainless videos series is some glimpse of what a “slow-down” might be like — h/t Jaan Tallinn, here).

The ghost sometimes has some kind of power to see counterfactual worlds as well. It can alter variables in the world, and then examine what changes as a result. These powers can vary. In some cases, I might imagine the ghost capable of altering Joe’s actions, but nothing else. In others, it might be capable of its own small actions, like causing someone to trip on a pebble. And in others, it might be capable of controlling basically any variable.

In each case, it’s generally able to observe in detail the consequences of different changes, and it can rewind and reset over and over, trying out and comparing different possibilities, sitting with them, reflecting, learning, understanding. The ghost-time available for this is limitless, and the ghost never gets tired or bored.

(There is also some assumption that none of these “tries” has any moral significance in itself — e.g., the ghost is able to create and consider worlds where there is lots of suffering, and to witness it in its full significance, but without that suffering “actually happening” in a way that subjects the ghost’s exploration to its own moral significance. I sometimes imagine some separate action of “making a possibility real,” but the ghost only takes such actions after deep exploration of the alternatives.)

Regardless of the scope of its powers, though, the ghost is not involved in the world. It does not, for example, try to send messages to the p...
A few thought on the inner ring
by KatjaGrace — 2021-01-21T03:40:15.253Z

I enjoyed C.S.Lewis’ The Inner Ring, and recommend you read it. It basically claims that much of human effort is directed at being admitted to whatever the local in-group is, that this happens easily to people, and that it is a bad thing to be drawn in to.

Some quotes, though I also recommend reading the whole thing:

In the passage I have just read from Tolstoy, the young second lieutenant Boris Dubretskoi discovers that there exist in the army two different systems or hierarchies. The one is printed in some little red book and anyone can easily read it up. It also remains constant. A general is always superior to a colonel, and a colonel to a captain. The other is not printed anywhere. Nor is it even a formally organised secret society with officers and rules which you would be told after you had been admitted. You are never formally and explicitly admitted by anyone. You discover gradually, in almost indefinable ways, that it exists and that you are outside it; and then later, perhaps, that you are inside it.

There are what correspond to passwords, but they are too spontaneous and informal. A particular slang, the use of particular nicknames, an allusive manner of conversation, are the marks. But it is not so constant. It is not easy, even at a given moment, to say who is inside and who is outside. Some people are obviously in and some are obviously out, but there are always several on the borderline. And if you come back to the same Divisional Headquarters, or Brigade Headquarters, or the same regiment or even the same company, after six weeks’ absence, you may find this secondary hierarchy quite altered.

There are no formal admissions or expulsions. People think they are in it after they have in fact been pushed out of it, or before they have been allowed in: this provides great amusement for those who are really inside. It has no fixed name. The only certain rule is that the insiders and outsiders call it by different names. From inside it may be designated, in simple cases, by mere enumeration: it may be called “You and Tony and me.” When it is very secure and comparatively stable in membership it calls itself “we.” When it has to be expanded to meet a particular emergency it calls itself “all the sensible people at this place.” From outside, if you have dispaired of getting into it, you call it “That gang” or “they” or “So-and-so and his set” or

...
Notes on Optimism, Hope, and Trust
by David Gross — 2021-01-20T23:00:04.268Z

This post examines the virtues of hope, optimism, and trust. It is meant mostly as an exploration of what other people have learned about these virtues, rather than as me expressing my own opinions about them, though I’ve been selective about what I found interesting or credible, according to my own inclinations. I wrote this not as an expert on the topic, but as someone who wants to learn more about it. I hope it will be helpful to people who want to know more about these virtues and how to nurture them.

What are these virtues?

These virtues have in common a sort of “look on the bright side” / “expect the best” approach to life. But there are a number of ways to interpret this, and if we are looking for a virtue — that is, a characteristic disposition that promotes human flourishing — we would be wise to be precise and careful.

There is a naive version of optimism that I would not want to try to defend on LessWrong:

the belief that the probability of an outcome is increased by one’s positive disposition toward it

So let me say at the start that this is just one possible way the optimistic outlook can be defined, and that others can bear more rational weight.

There is little controversy about hope, optimism, and trust being parts of a flourishing life. However there is controversy about whether encouraging a hopeful, optimistic, trustful outlook puts the cart before the horse. Are hope, optimism, and trust ingredients of a life well lived, or are they results of a life well lived (or perhaps of good fortune)?

Related virtues and vices

These virtues are closely related to some others: Cheer and joy, for instance, are easier to maintain when one is optimistic. Confidence, boldness, ambition, and courage are aided by the hope of triumph. Imagination can help you to discover possible good outcomes to be hopeful about. Intimacy, openness, and vulnerability can be boosted by trust. Solidarity can include an extension or expectation of trust. Trust also complements trustworthiness. Richard Rorty thought shared hope was at the base of civility.[1] Samantha Vice considers hope to be a form of patience (or maybe endurance/perseverance): a “refusal to capitulate.”[2]

There is some tension between optimism and caution, if optimism causes you to underweight the possibility of bad outcomes that you ought to prepare for.[3] If optimism becomes an excuse to discount evide...