trentmkelly/LessWrong-43k huggingface dataset:
A summary of every Replacing Guilt post
by Orpheus16 — 2022-06-30T00:46:19.173Z
Link: https://forum.effectivealtruism.org/posts/mfAbsrd2ZahmwHq2G/a-summary-of-every-replacing-guilt-post

1

I recently finished the Replacing Guilt series (by Nate Soares).

I don't struggle much with guilt. I consider myself a happy and energetic person. I also spent several years studying psychology, and I'm familiar with many techniques from cognitive-behavioral therapy.

So I was surprised by how impressed I was with Replacing Guilt. Elegant writing, memorable stories, useful advice, and dark humor.

There are few things I would put in the category of "pretty much everyone I know should spend 5 minutes reading this to see if it might be helpful, and at least 20% of them should read the whole thing." I feel this way about Replacing Guilt.

2

As I read, I wrote a few sentences summarizing each post. I mostly did this to improve my own comprehension/memory.

You should treat the summaries as "here's what Akash took away from this post" as opposed to "here's an actual summary of what Nate said."

Note that the summaries are not meant to replace the posts-- in fact, they're intended to get you to read the posts (especially the ones that seem interesting, though I recommend starting from the beginning).

3

Here are my notes on each post, in order. Tomorrow, I'll post a tier-list.

Half-assing it with everything you’ve got

You can try really hard to produce a low-quality or medium-quality outcome. You can put in your all to reach a B-, or to be in 17th place at the end of the race.

Failing with abandon

If you wanted to go to sleep at 12AM, and it’s 12:15AM, don’t be like “ah well, I failed.” You can try to get as close to your goal as possible, even if you don’t fully achieve it.

The Stamp Collector

We do not have full access to the external world, but we also do not have full access to our internal worlds. We can, in fact, care about the real world, other people, and things that happen outside of our minds.

You’re allowed to fight for something

We can convert listless guilt (which is vague and general) into specific guilt (which is about a particular thing). One way to do this is by looking out at the world and seeing things that are bad and being like “I want the world to be different”

Caring about something larger than yourself

Sometimes, our feelings are are in conflict with our quiet aesthetics (values, virtues, deeper beliefs). It can be useful to notice this and then figure out how we want to resolve these conflicts. For some people, this means embracing the aesthetics and

...
Gradient hacking: definitions and examples
by Richard_Ngo — 2022-06-29T21:35:37.166Z

EDIT: I have now shifted my terminology as follows (format: old term-->new term):

"RL credit hacking"-->exploration hacking.
"Gradient descent hacking"-->gradient hacking.
"Gradient hacking"-->credit hacking.

That is, I now reserve "gradient hacking" for one specific sub-phenomenon, and use "credit hacking" as the overarching term which includes both exploration hacking and gradient hacking (along with other possibilities, like multi-agent credit hacking). However, I've kept the original terminology in this post.

Gradient hacking is a hypothesized phenomenon where:

A model has knowledge about possible training trajectories which isn’t being used by its training algorithms when choosing updates (such as knowledge about non-local features of its loss landscape which aren’t taken into account by local optimization algorithms).
The model uses that knowledge to influence its medium-term training trajectory (even if the effects wash out in the long term).

Below I give some potential examples of gradient hacking, divided into those which exploit RL credit assignment and those which exploit gradient descent itself. My concern is that models might use techniques like these either to influence which goals they develop, or to fool our interpretability techniques. Even if those effects don’t last in the long term, they might last until the model is smart enough to misbehave in other ways (e.g.

specification gaming

, or

reward tampering

), or until it’s deployed in the real world—especially in the RL examples, since convergence to a global optimum seems unrealistic (and ill-defined) for RL policies trained on real-world data. However, since gradient hacking isn’t very well-understood right now, both the definition above and the examples below should only be considered preliminary.

RL credit hacking examples

A policy might know that a certain sequence of actions leads to higher reward. But if it chooses not to take those actions, it won’t be rewarded for them, and that type of behavior won’t be reinforced (until it finds them via random exploration, which could take a very long time).
A policy might know that humans give mistaken feedback on a given trajectory, and then select that trajectory more frequently in order to update itself in a certain direction. When humans give mistakenly positive feedback, this is just an example of specification gaming. But w

...
Progress links and tweets, 2022-06-29
by jasoncrawford — 2022-06-29T21:33:40.285Z
Link: https://rootsofprogress.org/links-and-tweets-2022-06-29

Links

Book launch for Anton Howes’s Arts and Minds, July 12, in London and online
Online book club event for the new Stripe Press edition of Vannevar Bush’s memoir, Pieces of the Action, Aug 6
Arbitrary Lines is a new book by M. Nolan Gray on “how zoning made America’s cities unaffordable, inaccessible, segregated, and sprawling” (See also announcement thread from @mnolangray)
“I do not believe that targeting amyloid is going to lead to a useful Alzheimer’s therapy, and watching these trials feels to me like watching someone trying to put out an oil well fire by dumping duffel bags of money onto it from helicopters” (Derek Lowe)
Paper by Lant Pritchett arguing that economic growth is necessary and sufficient for “everything basic to human material wellbeing” (h/t @DaveEvansPhD)

“The Averch–Johnson effect is the tendency of regulated companies to engage in excessive amounts of capital accumulation in order to expand the volume of their profits”

Queries

Is it true that wheelwrights were super highly paid for their specialist craft work? (probably not)
What could you do if energy was 100x cheaper? (@mattparlmer)

Quotes

When to calculate statistics was to blaspheme against God
First ten years for nylon stockings (@tsungxu)

Other tweets & retweets

A good way to look at GDP per capita and child mortality (@tonymmorley)
We’re all suffocating in the name of “safety” (@alyssamvance)
Request for podcast series: every episode is one of Patrick Collison’s “Fast” entries (@maxolson)
Three faulty assumptions people make when trying to understand the behavior of organizations

Charts

Record share of Americans are in favor of the use of nuclear energy (@AlecStapp)
Correcting human error vs doing exactly what you're told - is there literature on this in context of general system design?
by Jan Czechowski — 2022-06-29T21:30:05.753Z

This seems to be like an important piece of puzzle for a safe strong AI. The question is inspired by "horse riding astronaut" debate (where ai fails to generate a proper image, and I speculate it's because of high prior of this context-free prompt being a mistake).

I suspect the topic might have been discussed in general in context of system design (unrelated to AI), but I cannot find any good overview with simple googling. For specific examples, I think of those I know from IT: disabling "rm -rf /" without explicit flag, and the whole concept of "warnings" in program output.

I think in more critical system there might be an important tradeoff, as there's not always enough time to ask user for clarification of the input, so system has to decide on it's own how to interpret an input that is possibly a mistake.
Latent Adversarial Training
by Adam Jermyn — 2022-06-29T20:04:00.249Z

The Problem

We’d like to train models to be robustly safe, even in environments that may fall well outside of the training distribution. Unfortunately all we get to work with is the training distribution, which makes ensuring robust generalization difficult.

Deception is an example of this concern. Models may be well-behaved in the training environment because they realize it is a training environment, but behave poorly in deployment because they notice the distributional shift.

An Approach

Background

It would be great if we could generate scenarios where the model behaves badly. For instance if the model is deceptive we’d like to generate inputs to trick it into thinking it’s been deployed so that it reveals that deception.

Adversarial Training (AT), and Relaxed Adversarial Training (RAT), are approaches to this problem. Very briefly, in AT, we search for inputs that generate large loss, while in RAT we search for distributions of inputs or other more general evidence that the model is capable of generating large loss.

The approach I want to explore, called Latent Adversarial Training (LAT), is a form of RAT.

Concept

Suppose we have two models, the Agent and the Surgeon. The Agent is the model we’re trying to train, and the Surgeon is going to be its adversary. The training proceeds as follows:

The Agent processes input x from the environment, producing latent state s(x) and output y(s).
The Surgeon takes inputs (x,s,y) and searches for a modification s→s+δs such that y(s+δs) does poorly on the loss function L(y).
We update the Agent based on L(y(s+δs)), treating the Surgeon’s modifications as constants.

The idea is that the Surgeon gets to meddle with the latent state of the Agent with full knowledge of what the Agent was planning on doing, and we train the Agent to be robust against this kind of attack.

Implementation

Concretely, suppose the Agent is a neural network. The latent state s consists of the activations of all the neurons and the Surgeon gets to specify modifications like “this neuron gets +5”.

How does the Surgeon decide what to do? There are a number of options. It could examine the gradient of the Agent’s loss with respect to each activation, then push on the activations that give it the most loss. Or it could randomly generate its modifications and use some other optimization proces...
Game Review: This Merchant Life
by Zvi — 2022-06-29T18:30:00.816Z

In a previous life, I used to be a professional game player. I still am hard at work on a new tradable blockchain card game, Emergents, now in closed (but soon to be more open) beta, and I still love playing games.

This weekend I had the chance to take a break and play This Merchant Life, which (as the game would approve of frontlining) is currently on sale on Steam for $2.19. If you’re into this sort of thing, at that price it’s a buy. I figured I’d share my thoughts.

If that isn’t your cup of tea, you can very safely skip this one.

Overall verdict: This is a tier 3 game. It’s not essential or the best in its genre, and not everyone will enjoy it, but it’s quick and if you like the idea of the game it’s worth playing until you feel like stopping.

This Merchant Life is a trading game. You are a merchant.

Usually in such games, you own a ship, which allows you to do what is good in life but is in America for insane reasons prohibited by the Jones Act: take spices from one port and ship them to another port. No ship for you. You own a cart. So you take less exciting goods that aren’t spices that you buy in one town, and you ship them to another town. Where you sell them.

It is very good to be a merchant. By good I mean highly profitable. Depending on what price you put on the initial cart, your rate of return is super high. The job is not entirely safe given the number of bandits and animals on the road, which is especially true early on before you can afford and locate proper security. But man is it profitable.

The Good Life

Real life shipping of goods from one town to another town is not this profitable.

That is not only because this is a game. That is because the roads are now safe, and the skills and knowledge involved are easier to acquire. There’s a lot more competition. Profit margins are not what they once were in such games, especially not for long.

Yet the merchant life still does seem to largely be pretty good.

My last post was a personal story of air conditioner repair. I noted that given what they charge for their services, this seemed like an excellent business.

For a long time, my friend Michael Vassar has been pointing out that being a merchant, buying things low and then selling them high to the people who want them, is also generically a very good business, and it is very responsive to both hard work and smart work. If you focus on using ...
Limits to Legibility
by Jan_Kulveit — 2022-06-29T17:42:19.338Z

From time to time, someone makes the case for why transparency in reasoning is important. The latest conceptualization is

Epistemic Legibility

by Elizabeth, but the core concept is similar to

reasoning transparency

used by OpenPhil, and also has some similarity to

A Sketch of Good Communication

by Ben Pace.

I'd like to offer a gentle pushback. The tl;dr is in

my comment

on Ben's post, but it seems useful enough for a standalone post.

“How odd I can have all this inside me and to you it's just words.” ― David Foster Wallace

When and why reasoning legibility is hard

Say you demand transparent reasoning from AlphaGo. The

algorithm

has roughly two parts: tree search and a neural network. Tree search reasoning is naturally legible: the "argument" is simply a sequence of board states. In contrast, the neural network is mostly illegible - its output is a figurative "feeling" about how promising a position is, but that feeling depends on the aggregate experience of a huge number of games, and it is extremely difficult to explain transparently how a particular feeling depends on particular past experiences. So AlphaGo would be able to present part of its reasoning to you, but not the most important part.[1]

Human reasoning uses both: cognition similar to tree search (where the steps can be described, written down, and explained to someone else) and processes not amenable to introspection (which function essentially as a black box that produces a "feeling"). People sometimes call these latter signals “intuition”, “implicit knowledge”, “taste”, “S1 reasoning” and the like. Explicit reasoning often rides on top of this.
Extending the machine learning metaphor, the problem with human interpretability is that "mastery" in a field often consists precisely in having some well-trained black box neural network that performs fairly opaque background computations.

Bad things can happen when you demand explanations from black boxes

The second thesis is that it often makes sense to assume the mind runs distinct computational processes: one that actually makes decisions and reaches conclusions, and

another

that produces justifications and rationalizations.

In my experience, if you have good introspective access to your own reasoning, you may occasionally notice that a conclusion C depends mainly on some black box, but at the same time, yo...
Will Capabilities Generalise More?
by Ramana Kumar — 2022-06-29T17:12:56.255Z

Nate

and

Eliezer

(Lethality 21) claim that capabilities generalise further than alignment once capabilities start generalising far at all. However, they have not articulated particularly detailed arguments for why this is the case. In this post I collect the arguments for and against the position I have been able to find or generate, and develop them (with a few hours’ effort). I invite you to join me in better understanding this claim and its veracity by contributing your own arguments and improving mine.
Thanks to these people for their help with writing and/or contributing arguments: Vikrant Varma, Vika Krakovna, Mary Phuong, Rory Grieg, Tim Genewein, Rohin Shah.

For:

1. Capabilities have much shorter description length than alignment.

There are simple “laws of intelligence” that underwrite highly general and competent cognitive abilities, but no such simple laws of corrigibility or laws of “doing what the principal means” – or at least, any specification of these latter things will have a higher description length than the laws of intelligence. As a result, most R&D pathways optimising for capabilities and alignment with anything like a simplicity prior (

for example

) will encounter good approximations of general intelligence earlier than good approximations of corrigibility or alignment.

2. Feedback on capabilities is more consistent and reliable than on alignment.

Reality hits back on cognitive strategies implementing capabilities – such as forming and maintaining accurate beliefs, or making good predictions – more consistently and reliably than any training process hits back on motivational systems orienting around incorrect optimisation targets. Therefore there’s stronger outer optimisation pressure towards good (robust) capabilities than alignment, so we see strong and general capabilities first.

3. There’s essentially only one way to get general capabilities and it has a free parameter for the optimisation target.

There are many paths but only one destination when it comes to designing (via optimisation) a system with strong capabilities. But what those capabilities end up being directed at is path- and prior-dependent in a way we currently do not understand nor have much control over.

4. Corrigibility is conceptually in tension with capability, so corrigibility will fail to generalise when capability generalises well.

Plans that actually work in dif...
Kevin Kelly's "103 Bits of Advice," Expanded
by Dalton Mabery — 2022-06-29T13:36:13.160Z

Kevin Kelly, who arguably has written one of the most influential essays regarding the internet and the creator economy, "1,000 True Fans", published "103 Bits of Advice I Wish I Had Known" on his 70th birthday. In the simple article, he gives practical advice, philosophical quips, and everything in between. I thoroughly enjoyed reading it when it came out. So much so, I printed it out and annotated bits I loved and wanted to go deeper on.

To my luck, Freakonomics published an interview with him asking about that specific essay. Kelly was able to provide more context on his answers and share a few stories. The following comes from that interview.

One of the pieces of advice observe, very factually, that "ninety percent of everything, is crap." Kelly goes on to encourage the reader to, if they find something they don't like, just try finding the ten percent that is actually good. But how to find that ten percent was left as a mystery, until now. Stephen Dubner, the host of Freakonomics asked him for his advice on finding the "'non-crap' 10 percent":

I have to say, this one I’m talking to myself because I often get very impatient. So, the only bit of advice for me is to keep returning to it with as much grace and open-mindedness as possible because nine times out of ten you’re going to be very disappointed. So why are you going to go back the tenth time? Of course, life is limited. We can’t try all the stuff that we don’t like. But when I have a chance and it’s not too difficult, I’ll give things a second or third or fourth chance. And occasionally, I do change my mind and it’s like, “Wow, I’m glad that I pursued that because I didn’t see it in the beginning.”

Dubner cited research that showed when people approach the ages of 30 or 40, they typically are set in the ways of the food they eat and the things they do. Trying new things just isn't much of an idea as you get older. Fortunately, Kelly had some simple advice to staying more fluid as one ages:

Don’t sit in the same chair every night. I deliberately try not to do that at the table, to make sure I don’t get into these ruts. And I ask my kids to make me a playlist for birthday presents. Give me all the music you’re listening to as a way to hear new things and see if I can get into it. When you go out to eat, order your favorite thing, and then order something you’ve never ordered before. I think it has to be a

...
The table of different sampling assumptions in anthropics
by avturchin — 2022-06-29T10:41:18.872Z

“…. anthropic reasoning: complicated, error-prone,

looks a bit magical but once you start getting real evidence much

of it becomes irrelevant.” Sandberg

The meta-meta-problem of anthropics is how to find the right way to choose between principles of selecting the correct sampling assumptions, but before we get there, let's first list all known and even unknown sampling assumptions.

The full list, including some related ideas, has 38 items (google doc here, text version below).

I suggest several possibly new assumptions for the sake of completeness of the list, but it doesn’t mean that I endorse them (like “intensity sampling” and “observer-moment-size sampling”) and it doesn’t mean that they were never suggested before.

1. Classical assumptions based on the difference between possible and real observers.

SSA. Self-sampling assumption. I am randomly selected from all actually existing observers of my reference class.

The problem with choosing the reference class
Many other sampling assumptions are just SSA for a specific class
Different reference classes have different types of the end.

[Doomsday argument-like problems are designated via black box below]
Classical Doomsday Argument: Civilization will end soon (but it is not problematic unless we have too good expectations about the future).
Adam and Eve: probability pumping via manipulating the number of the future observers (but maybe evolution already used this?)

SSSA. Strong self-sampling assumption. The same as SSA, but the selection is between the observer-moments (OM).

It gives too much weight to Boltzmann Brains and freak dreams, as they generate OM without proper observers.

SSSSA. Super-strong self-sampling assumption: I am selected from all minds weighted by their intelligence. Variant: I am selected from all observer-moments proportional to their “size”.

Explains why I am not an ant.

Superintelligence is impossible, or I wil

...