trentmkelly/LessWrong-43k huggingface dataset:
Thoughts on gradient hacking
by Richard_Ngo — 2021-09-03T13:02:44.181Z

Gradient hacking is the hypothesised phenomenon of a machine learning model, during training, deliberately thinking in ways which guide gradient descent to update its parameters in the directions it desires. The key intuition here is that because the loss landscape of a model is based on the cognition it does, models can make decisions for the purpose of affecting their loss landscapes, thereby affecting the directions in which they are updated.

Evan writes

:

I don't think [gradient hacking] is actually that difficult once your deceptive model comes up with it as a strategy. First, note that since gradient descent is a local optimization process, it doesn't have to somehow change what the global minima might be, or even what the local minima are—it just has to create a ridge in the loss landscape that will push the gradient in the direction it wants.

One example of this is for the model to check if its objective satisfies some criterion, and fail hard if it doesn't. Now, if the model gets to the point where it's actually just failing because of this, then gradient descent will probably just remove that check—but the trick is never to actually get there. By having such a check in the first place, the model makes it so that gradient descent won't actually change its objective, since any change to its objective (keeping all the other parameters fixed, which is what gradient descent does since it computes partial derivatives) would lead to such a failure.

But I don't think this works as described, because of the way that gradient descent operates. Suppose that an agent has some trait which gradient descent is trying to push in direction x because the x-ness of that trait contributes to the agent’s high score; and that the agent wants to use gradient hacking to prevent this. Consider three possible strategies that the agent might try to implement, upon noticing that the x-component of the trait has increased:

Deterministically failing hard above a sharp threshold of the x-component. In this case, the closer that gradient descent gets to the threshold, the better it will do. But since the threshold is sharp, the gradient will continue pointing towards it right until the agent hits the threshold. And since gradient update proceeds via discrete steps, then it will eventually cross over (causing the agent to fail) then bounce back, then cross over again, until something e

...
Is there a name for the theory that "There will be fast takeoff in real-world capabilities because almost everything is AGI-complete"?
by David Scott Krueger (formerly: capybaralet) — 2021-09-02T23:00:42.785Z

I think it's plausible that:
1) For many applications, getting narrow AI to do a task well enough to be valuable doesn't seem worth it, and likely isn't (esp. when considering opportunity cost and alternative applications of AI).
2) Thus proto-AGI is actually not going to change the world that much
3) But OFC, AGI will (once/assuming it's cheap enough)
If correct, this could mean that people probably won't really be impressed by narrow AI at any point, and then all of the sudden, we get AGI and everything changes rapidly.
I'm just sketching it out and probably didn't do the best job, but my questions are: Is this something people have seen argued? Is there a name for it? (or want to propose one?)
Framing Practicum: Selection Incentive
by aysajan — 2021-09-02T20:26:08.515Z

Credit

An enormous amount of credit goes to johnswentworth who made this new post possible.

This is a

framing practicum

post. We’ll talk about what selection incentives are, how to recognize selection incentives in the wild, and what questions to ask when you find them. Then, we’ll have a challenge to apply the idea.

Today’s challenge: come up with 3 examples of selection incentives which do not resemble any you’ve seen before. They don’t need to be good, they don’t need to be useful, they just need to be novel (to you).

Expected time: ~15-30 minutes at most, including the Bonus Exercise.

What Are Selection Incentives?

Imagine trying to find great/popular posts on LessWrong. We look for things like high karma values, high number of comments, or a well-known writer. We don’t really look at the contents of the individual posts (yet), we just look at an overall “score” that can help us to choose posts. This overall “score” mechanism encourages writers to write posts that could potentially achieve high “scores”, for instance broad-interest posts, thought-provoking posts, controversial posts, etc, regardless of what the actual purpose of the writer is.

This is a selection incentive: Something is chosen based on some criteria or a known process. For instance, posts are chosen based on an overall “score”: high karma values, high number of comments, etc. On the other hand, presenting ideas or transferring knowledge (not pursuing high karma values) might be what the writers actually want. But, the readers’ selection criteria are there regardless of what writers actually want in the first place.

Another example is corporations maximizing profits. The founder of the corporation has something in mind, for instance sending humans to space, producing the most affordable cars to the mass population, etc., and they may or may not be trying to maximize profit. What happens in the real business world, however, is that businesses live or die based on how well they maximize profits. Businesses are selected on the basis of how well they maximize profits, regardless of what the founders actually want.

What To Look For

In general, selection incentives should spring to mind whenever something is chosen based on some criteria or a known process. We want to know what factors cause something to be more or less likely chosen. A few ways this can apply:

We are selecting/cho

...
Review: Foragers, Farmers, and Fossil Fuels
by L Rudolf L — 2021-09-02T17:59:28.143Z
Link: http://strataoftheworld.blogspot.com/2020/12/review-foragers-farmers-and-fossil-fuels.html

Two hundred years ago, most people lived in societies that considered slavery, war, and discrimination based on class, ethnicity, and gender to be justifiable. Today, most people live in societies that hold the opposite beliefs.

What changed? A simple and tempting narrative is that we have simply become wiser; that various Enlightenment philosophers, thoughtful activists, and other principled people figured out that the pre-industrial moral order is wrong and managed to persuade everyone to change.

It is true that many smart and principled people had good ideas and that this was a big proximate driver of better values. But is it a coincidence that this change in values happened around the same time as the industrial revolution?

What about the previous economic revolution, the agricultural one? Did that also coincide with a change in the values that people held? The evidence says yes – foraging societies tend to be more accepting of violence and far less accepting of hierarchy than farming ones.

The argument of Ian Morris' Foragers, Farmers, and Fossil Fuels is that these timings are not a coincidence. Societies that change their main method of getting energy also change their values, because some sets of values give greater success for a certain type of society. Farming societies that stick to anti-hierarchical forager attitudes won't survive competition with farming societies that learn to believe in hierarchies (maybe they won't be economically competitive and won't be able to field as big an army to defend themselves as the god-king next door can field to conquer them). Likewise, industrial societies that stick to inflexible hierarchies and elite-focused economies can't compete with more equal democracies that don't squander the talents of the non-elite, and maintain a well-looked-after middle-class of rich consumers and educated workers.

We can contrast two ways of trying to explain the history of values. The first says that the history of values is a history of ideas; a battle of ideas against other ideas, waged in the minds of people. The second says that the history of values is a history of what works best. The battle is between the benefits conferred by believing in certain ideas and those conferred by other ones, and it is waged out in the real world, where empires fall or rise based on whether they value the things that will lead them to success.

I...
Addendum to "Amyloid Plaques: Medical Goodhart, Chemical Streetlight"
by J Bostock — 2021-09-02T17:42:02.910Z

Last week I committed to posting five examples where I think a group of scientists has gone astray. Goodhart's law says you get what you measure. The streetlight effect makes you look where it's easy to look. When combined, all you get from an entire field of research is a bunch of things which are easy to measure.

I have tried to make these as current as possible, with a special focus on finding avenues of research that the general scientific establishment has yet to abandon. Not all of them are like that though, due to time and effort limitations on myself (also an unexpected release of D&D.Sci). They are also mostly worse examples than my original one, which is to be expected.

I would also be remiss not to mention the comment on my last post by JenniferRM which is an excellent example of good reasoning about this sort of issue.

Ageing and Ageing Markers

Lots of research into slowing ageing is actually taking place! This is good news from the perspective of not dying. Unfortunately, a much smaller minority of it is optimal. Lots of research at the moment is focused on interventions which can extend the lifespan of mice, flies, worms, and yeast by a few tens of percentage points. This has given us insights like the role of mTOR, fasting, and the epigenetic clock. However, any intervention which extends life by only a few tens of percent as a maximum is not targeting the "ultimate" causes of ageing. Lots of research seems to look for small increases in longevity rather than looking for building understanding of ageing itself.

Electrocatalysis of Graphene Compounds

This is a closed-book one. Certain chemical reactions involve the transfer of electrons. Graphene can catalyse these reactions, but lots of graphene derivatives can catalyse it even faster. For electronic reasons, both putting some extra electrons into graphene, and taking some out, make it a better catalyst. Many many papers were published based on this theory, until finally someone made graphene doped with bird guano. This was an excellent catalyst, and managed to put the endless searching for better graphene dopants.

Racial Bias Testing

Racism, like most of the large issues facing society, is very complex. One of the big ideas of (relatively) recent times is that individuals who are not explicitly racist can still be biased. The Implicit Association Test is the one where you do classification simult...
Lakshmi's Magic Rope: An Intuitive Explanation of Ramanujan Primes
by DirectedEvolution — 2021-09-02T16:36:07.225Z

Imagine a magic rope. It's long, stretchy, and beautiful. Let's take it with us on a trip through the positive integers. We can start wherever you like, and we'll go toward infinity. We put one tip of the magic rope down at our feet, at whichever integer we happen to be standing on. The other end of the magic rope trails behind us on the number line on a lower integer. Along its length, our magic rope contains numbers that are full of possibilities.

Although we're still wandering around, letting the rope drag freely along the numbers, we might be curious about the prime numbers our magic rope happens to be touching. Primes, remember, are numbers that are only divisible by themselves and 1, and they are one of the greatest mathematical mysteries. Primes are to integers (whole numbers) as the elements are to chemistry. Just as all matter is made up of the elements on the periodic table, so all integers are made up of primes. Any whole number that's not a prime can be created by multiplying primes together - and there's only one way to create any given whole number by doing so!

For example, the first four primes are 2, 3, 5, and 7. If we multiply them together, we can produce 4 (2x2), 6 (2x3), 8 (2x2x2), 9 (3x3), 10 (2x5), 12 (2x2x3), 14 (2x7), 15 (3x5), 16 (2x2x2x2), 18 (2x3x3), and so on. So we've been able to create most of the numbers from 2-18 by multiplying just our first four primes. And we could create many more numbers - an infinite number, in fact - by multiplying just these four primes an increasing number of times. But even between 2 and 18, we have a few holes. We couldn't create 11, 13, or 17 in this way, because these numbers are primes themselves.

That's why primes are a little like the chemical elements. And just as establishing and studying the chemical elements was a crucial step in the development of chemistry, so understanding the patterns in the prime numbers is central to mathematics. So we might be curious about what we can find out about the primes on our magic rope.

Once we decide where the far tip of the rope will go, it would be especially nice if we could say exactly what prime numbers are on our magic rope. Unfortunately, that would be a little bit like figuring out exactly who the love of your life will be, years before meeting him or her. You might be able to predict that you'll fall in love one day, and maybe even that y...
Formalizing Objections against Surrogate Goals
by VojtaKovarik — 2021-09-02T16:24:39.818Z

I recently had a chance to look into surrogate goals and safe Pareto improvements, two recent proposals for avoiding the realization of threats in multiagent scenarios. In this report, I formalize some objections to these proposals.

EDIT: by discussing objections to SG/SPI, I don't mean to imply that the objections are novel or unknown to the authors of SG/SPI. (The converse is true, and most of them even explicitly appear in corresponding posts/paper.) Rather, I wanted to look at some of the potential issues in more detail.

Introduction

The surrogate goals (SG) idea proposes that an agent might adopt a new seemingly meaningless goal (such as preventing the existence of a sphere of platinum with a diameter of exactly 42.82cm or really hating being shot by a water gun) to prevent the realization of threats against some goals they actually value (such as staying alive) [TB1, TB2]. If they can commit to treating threats to this goal as seriously as threats to their actual goals, the hope is that the new goal gets threatened instead. In particular, the purpose of this proposal is not to become more resistant to threats. Rather, we hope that if the agent and the threatener misjudge each other (underestimating the commitment to ignore/carry out the threat), the outcome (Ignore threat, Carry out threat) will be replaced by something harmless.

Safe Pareto improvements (SPIs) [OC] is a formalization and a generalization of this idea. In the straightforward interpretation, the approach applies to a situation where an agent delegates a problem to their representative --- but we could also imagine “delegating” to a future self-modified version of oneself. The idea is to give instructions like “if other agents you encounter also have this same instruction, replace ‘real’ conflicts with them with ‘mock’ conflicts, in which you all act like you would in a real conflict, but bad outcomes are replaced by less harmful variants”. Some potential examples would be fighting (should a fight arise) to the first blood instead of to the death or threatening diplomatic conflict instead of a military one. In the SG setting, SPI could correspond to a joint commitment to only use water-gun threats while behaving as if the water gun was real.

In the following text, we first describe SPI in more detail and focus on framing the setting to which it applies and highlighting the assumptions i...
Covid 9/2: Long Covid Analysis
by Zvi — 2021-09-02T14:50:01.287Z

I’m going back to New York City, baby! Tomorrow’s the big day.

Oh, yeah, the pandemic. That’s still a thing, and one still has to deal with the various ways people are dealing with it. Cases continue to rise slightly, likely due to increased testing, and we certainly aren’t seeing a big improvement on the horizon.

The big Covid development this week was about Long Covid, as we got multiple analysis posts on the subject and some extra data as well. My conclusions on magnitude have not changed much, and I continue to believe that Long Covid is a thing worth some effort to avoid, but it is not as severe or common a thing as some reports would suggest, and that it is not a big enough issue that one should make big changes in life to avoid it, when you compare it to the effects of long-term Covid prevention. This is one of those cases where you are encouraged to run your own analysis, draw your own conclusions, and then choose how best to apply them to your own life.

Executive Summary

Quiet week.

Vaccines still work.
Case counts did not peak yet, but positive test rates likely have peaked.
Long Covid writeup from Scott Alexander.

Let’s run the numbers.

The Numbers

Predictions

Prediction from Last Week: 950k cases (+2%) and 9,400 deaths (+25%).

Results: 1.04mm cases (+10%) and 9,290 deaths (+22%).

Prediction for next week: 1.1mm cases (+5%) and 11,150 deaths (+20%).

The rise in cases here is deeply disappointing. However, if we look at positive test rates, we see a different story:

I’m not convinced the number of actual Covid cases is still rising, but that is certainly possible with case counts rising. Schools are going to reopen soon, which may cause trends to change. If things start to get much worse again before November, I’ll assume that was likely the reason.

For now, given we didn’t peak yet, on average I still expect a small rise in cases, if only from increased testing.

Other people’s predictions, Tabarrok calls the peak on August 30:

Deaths

Date	WEST	MIDWEST	SOUTH	NORTHEAST	TOTAL
Jul 8-Jul 14	532	398	689	145	1764
Jul 15-Jul 21	434	341	732	170	1677
Jul 22-Jul 28	491	385	1009	157	2042
Jul 29-Aug 4	693	477	1415	304	2889
Aug 5-Aug 11	705	629	2181	234	3749
Aug 12-Aug 18	912	851	3394	388	5545
Aug 19-Aug 25	1291	1045	4713	508	7557
Aug 26-Sep 1	1578	1256	5892	564	9290

Deaths are as expected, and I see no reason they shouldn’t continue to follow cases.

Cases

Date	WEST	MID

...
Competent Preferences
by Charlie Steiner — 2021-09-02T14:26:50.762Z

I - Obvious answers to obvious questions

"

Why do I think I know what I do

about Goodhart's law?"

There's an obvious answer (or at least an obvious outer layer of the answer-onion): we use Goodhart's law because we usually model humans as if we have values. Not necessarily True Values over the whole world, but at least

contextual values

over some easy-to-think-about chunk of the world. Modeling humans as goal-driven agents is never going to be perfect at predicting us, nor is it what you'd do if you had infinite computing power, but it still works really well for our purposes.

We infer human values by building a simplified model of the world that features human actions as a fundamental part. Such an abstract model doesn't need to be our absolute best model of the world, it just needs to capture useful regularities. If we keep the model fixed, then the human values are determined by the data, but the model doesn't necessarily have to be fixed. If we

uncurry

this inference process, inferred human values depend both on the data and on the way we model the data.

Consider historical examples of Goodhart's law, like the

dolphin trainers

who rewarded dolphins for each piece of trash removed from the pool, which taught the dolphins not to clean the pool, but to hoard trash and rip it up before delivering each piece. Upon hearing this story we immediately make a simple model of the situation, using variables like amount of trash in the pool, whether the dolphins are delivering trash, and whether they got a fish. In this model it's super easy to represent that the dolphin trainers want the pool clean and the dolphins want fish.

Once we take this model of the world as given, then everything makes sense as an example of Goodhart's law: the humans wanted the pool to be clean, they created an incentive for the dolphins to take an action that used to be correlated with the pool being clean, and as a result the correlation broke down. We can gather up many such stories that all feature the same pattern of the human creating an incentive that's correlated with their modeled values and then not getting what they want - these stories are our empirical evidence for Goodhart's law.

What's my problem with using Goodhart's law for value learning AI, then? If it lets us predict a regularity about the future of dolphin trainers (and more), can't we trust it to predict what h...
Good software to draw and manipulate causal networks?
by J Bostock — 2021-09-02T14:05:18.389Z

I'm collecting my findings in a bunch of ageing-related systems. These consist of a bunch of networks with arrows pointing to things. What's making it difficult is the lack of a software to do this in. (Would need to have a system of nodes, directed arrows and those flat-headed arrows, and be reasonably user-friendly) Anyone use something like this themselves?
Growth of prediction markets over time?
by Daniel Kokotajlo — 2021-09-02T13:43:38.869Z

I would love to see a graph of the total amount of money sloshing around in prediction markets in 2021, 2020, 2019, etc. stretching back as far as we have data. I'm wondering whether there is a trend and if so what the growth rate is. My Google-fu is failing me so I'm asking here in case anyone else knows.
We are failing to see how much better off humanity could be (transparent monsters part 2)
by Writer — 2021-09-02T10:52:13.270Z
Link: https://youtu.be/aAQR-_GKaR8

This is the script of the video linked above, by Rational Animations.
Cross-post to the EA Forum
In the previous video, we introduced the concept of “transparent monsters”. They are large moral problems hidden to the sight of most of humanity, which are at first visible to only a small group of people.

But now, let’s consider a completely different approach to finding monsters, which looks at missed potential rather than tragedies. The two approaches are complementary, and the distinction between them is not that strong under some ethical views.

Some of the biggest tragedies of today are transparent to our sight because we fail to grasp the full potential of our future. Early humans, tens of thousands of years ago, lived in a state that today we consider absolute poverty. On average, they died in their twenties, mainly due to the high infant mortality. Did that state of affairs feel like a tragedy to them? Probably not nearly as much as how it feels to us. In any case, for them, it was just how things were. They couldn’t imagine things like medicine, houses, hygiene, consistent good nutrition, transportation, and basic comforts that today we take for granted. What if one of the biggest tragedies of our time is that we are still relatively primitive? One day we might eliminate disease, aging, psychological suffering, and reach levels of wellbeing that we can’t imagine today. Looking back from that time, humanity will look at us with compassion and horror, in the same way we today look at the way humanity lived tens of thousands of years ago.

Consider death itself. Usually, people are not fond of dying, but death is everyone’s problem and unsolvable with current technology. It’s just how things are. These features make it transparent. We go on with our lives without giving it much thought. But every second that a solution is delayed, means that almost two lives are lost. Plus, in the same way as past monsters, such as slavery, people often rationalize death as good. It gives meaning to life, they say. But imagine a more advanced humanity that somehow eliminated all the major causes of death. Would this future humanity consider us lucky because we still die so easily? One day, when we will mostly get rid of death, it will probably be recognized among the monsters.

Now consider this even more radical way of reasoning:

The easiest way to see a monster ...