trentmkelly/LessWrong-43k huggingface dataset:
Framing Practicum: Incentive
by aysajan — 2021-08-27T19:22:18.796Z

Credit

An enormous amount of credit goes to johnswentworth who made this new post possible.

This is a

framing practicum

post. We’ll talk about what incentives are, how to recognize incentives in the wild, and what questions to ask when you find them. Then, we’ll have a challenge to apply the idea.

Today’s challenge: come up with 3 examples of incentives which do not resemble any you’ve seen before. They don’t need to be good, they don’t need to be useful, they just need to be novel (to you).

Expected time: ~15-30 minutes at most, including the Bonus Exercise.

What Are Incentives?

At the beginning of a sowing season, the Government of India announces a list of guaranteed purchase prices for certain crops, e.g., rice, wheat, cotton, etc., to support farmers. In case the market price for a crop falls below the guaranteed purchase price, the government agencies purchase the entire quantity from farmers if the crop quality meets a minimum quality threshold. From an Indian farmer’s perspective, the farmer is encouraged to produce price supported crops with quality just above the threshold level set by the government - and no higher.

This is an economic incentive: There is a reward signal in the system. Farmers are rewarded for producing crops just above the quality threshold. On the other hand, they are not rewarded for producing higher quality crops. Here we see the defining features of incentives: A system (a farmer) “wants” some resource (money), and can get more of that resource in return for some actions (producing crops with quality just above the threshold level) than others (producing crops with quality well above the threshold level).

Another example, with a direct notion of “reward”: cash incentive for taking Covid-19 vaccine shots. Some states in the US are offering rewards for Covid-19 vaccination in the form of direct cash or lottery programs. We can identify a clear reward signal in the system, which is people get rewarded for taking vaccines. Here again we see the defining features of incentives: A system (a human) “wants” some resource (money), and can get more of that resource in return for some actions (taking Covid-19 vaccines) than others (not taking Covid-19 vaccines).

What To Look For

In general, incentives should come to mind whenever there is some kind of reward signal. A system “wants” some resource, and can get more of that ...
[AN #162]: Foundation models: a paradigm shift within AI
by Rohin Shah — 2021-08-27T17:20:03.831Z

Alignment Newsletter is a weekly publication with recent content relevant to AI alignment around the world. Find all Alignment Newsletter resources here. In particular, you can look through this spreadsheet of all summaries that have ever been in the newsletter.

Audio version here (may not be up yet).

Please note that, while I work at DeepMind, this newsletter represents my personal views and not those of my employer.

HIGHLIGHTS

AI Safety Papers (Ozzie Gooen) (summarized by Rohin): AI Safety Papers (announced here) is an app to interactively explore a previously collected database of AI safety work (AN #130). I believe it contains every article in this newsletter (at least up to a certain date; it doesn’t automatically update) along with their summaries, so you may prefer to use that to search past issues of the newsletter instead of the spreadsheet I maintain.

On the Opportunities and Risks of Foundation Models (Rishi Bommasani et al) (summarized by Rohin): The history of AI is one of increasing emergence and homogenization. With the introduction of machine learning, we moved from a large proliferation of specialized algorithms that specified how to compute answers to a small number of general algorithms that learned how to compute answers (i.e. the algorithm for computing answers emerged from the learning algorithm). With the introduction of deep learning, we moved from a large proliferation of hand-engineered features for learning algorithms to a small number of architectures that could be pointed at a new domain and discover good features for that domain. Recently, the trend has continued: we have moved from a large proliferation of trained models for different tasks to a few large “foundation models” which learn general algorithms useful for solving specific tasks. BERT and GPT-3 are central examples of foundation models in language; many NLP tasks that previously required different models are now solved using finetuned or prompted versions of BERT and/or GPT-3.

Note that, while language is the main example of a domain with foundation models today, we should expect foundation models to be developed in an increasing number of domains over time. The authors call these “foundation” models to emphasize that (1) they form a fundamental building block for applications and (2) they are not themselves ready for deployment; they are simply a foundation on wh...
Cognitive scientist Joel Chan on metascience, scaling and automating innovation, collective intelligence, and tools for thought.
by fowlertm — 2021-08-27T16:33:23.532Z
Link: https://www.youtube.com/watch?v=KT-I_6TERKk

We recently hosted Joel Chan on the Futurati Podcast, and it was a fantastic, thought-provoking conversation.

(Dr. Chan's website, and "Scaling up analogical innovation with crowds and AI")

He has spent years working on platforms which enable collective sensemaking and encourage creativity, and we picked his brain on these and related efforts.

Some major themes:

What prior attempts have been made to encourage creativity and crowdsourced innovation? Why did they fail?
In academia and research it's typical to treat the 'paper' as the basic relevant unit, but might there be ways to represent individual concepts, arguments, and lines of evidence? If so, what kinds of emergent activity might that enable and encourage?
Are there ways to structure scientific disciplines, so as to better notice gaps in understanding or potentially fertile new directions of inquiry?
What is the role of analogy in cognition and understanding? How might this help us build tools like really advanced semantic search?
Could it be possible to scale and automate large parts of the research and problem-solving process? If so, what does that look like?
Research productivity tip: "Solve The Whole Problem Day"
by Steven Byrnes — 2021-08-27T13:05:59.550Z

(This is about a research productivity strategy that’s been working very well for me personally. But YMMV,

consider reversing any advice

, etc. etc.)

As a researcher, there’s kinda a stack of "what I'm trying to do", from the biggest picture to the most microscopic task. Like here's a typical "stack trace" of what I might be doing on a random morning:

LEVEL 5: I’m trying to ensure a good future for life
LEVEL 4: …by trying to solve the AGI control problem
LEVEL 3: …in the hypothetical scenario where the future AGI's algorithms will resemble a human brain’s algorithms
LEVEL 2: …by trying to understand the functions of the various modulatory signals going from the brainstem to the telencephalon
LEVEL 1: …by reading a bunch of articles about the nucleus incertus

So as researchers, we face a practical question: How do we allocate our time between the different levels of the stack? If we’re 100% at the bottom level, we run a distinct risk of "losing the plot", and working on things that won't actually help advance the higher levels. If we’re 100% at the top level, with our head way up in the clouds, never drilling down into details, then we’re probably not learning anything or making any progress.

Obviously, you want a balance.

And I've found that striking that balance properly isn't something that takes care of itself by default. Instead, my default is to spend too much time at the bottom of the stack and not enough time higher up.

So to counteract that tendency, I have for many months now had a practice of "Solve The Whole Problem Day". That's one day a week (typically Friday) where I force myself to take a break from whatever detailed things I would otherwise be working on, and instead I fly up towards the top of the stack, and try to see what I'm missing, question my assumptions, find new avenues to explore, etc.

In my case, "The Whole Problem" = "The Whole Safe & Beneficial AGI Problem". For you, it might be The Whole Climate Change Problem, or The Whole Animal Suffering Problem, or The Whole Becoming A Billionaire Problem, or whatever. (If it's not obvious how to fill in the blank, well then you especially need a Solve The Whole Problem Day! And maybe start here & here & here.)

Implementation details

The most concrete and obvious way that my Solve The Whole Problem Days are different from my other workdays is that I have a rule that I impose on myself: N

...
Altruism Under Extreme Uncertainty
by lsusr — 2021-08-27T06:58:32.335Z

I attended an Effective Altruism club today where someone had this to say about longtermism.

I have an intuitive feeling that ethical arguments about small probabilities of helping out extremely large numbers (like 1058) of people are flawed, but I can't construct a good argument for why this is.

The flaw is uncertainty.

In the early 20th century, many intellectuals were worried about population control. The math was simple. People reproduce at an exponential rate. The amount of food we can create is finite. Population growth will eventually outstrip production. Humanity will starve unless population control is implemented by governments.

What actually happened was as surprising as it was counterintuitive. People in rich, industrial countries with access to birth control voluntarily restricted the number of kids we have. Birthrates fell below replacement-level fertility. This process is called the demographic transition.

We now know that if you want to reduce population growth, the best way to do so is to make everyone rich and then provide free birth control. The side effects of this are mostly beneficial too.

China didn't know about the demographic transition when they implemented the one-child policy (一孩政策). The one-child policy wasn't just a human rights disaster involving tens of thousands of forced abortions for the greater good. It was totally unnecessary. The one-child policy was implemented at a time when China was rapidly industrializing. The Chinese birthrate would have naturally dropped below replacement level without government intervention. Chinese birthrates are still below replacement-level fertility even now that the one-child policy has been lifted. China didn't just pay a huge cost to get zero benefit. They paid a huge cost to gain negative benefit. Their age pyramid and sex ratios are extra messed up now. This is the opposite of what effective population control should have accomplished.

China utterly failed to predict its own demographic transition even though demographic changes on time horizons of a few decades are an unusually easy trend to predict. The UN makes extremely precise predictions on population growth. Most trends are harder to predict than population growth. If you're making ethical decisions involving the distant future then you need to make predictions about the distant future. Predictions about the distant future necessari...
Could you have stopped Chernobyl?
by Carlos Ramirez — 2021-08-27T01:48:02.332Z

...or would you have needed a PhD for that?

It would appear the inaugural post caused some (off-LW) consternation! It would, after all, be a tragedy if the guard in our Chernobyl thought experiment overreacted and just unloaded his Kalashnikov on everyone in the room and the control panels as well.

And yet, we must contend with the issue that if the guard had simply deposed the leading expert in the room, perhaps the Chernobyl disaster would have been averted.

So the question must be asked: can laymen do anything about expert failures? We shall look at some man-made disasters, starting of course, with Chernobyl itself.

Chernobyl

One way for problems to surface

To restate the thought experiment: the night of the Chernobyl disaster, you are a guard standing outside the control room. You hear increasingly heated bickering and decide to enter and see what's going on, perhaps right as Dyatlov proclaims there is no rule. You, as the guard, would immediately be placed in the position of having to choose to either listen to the technicians, at least the ones who speak up and tell you something is wrong with the reactor and the test must be stopped, or Dyatlov, who tells you nothing is wrong and the test must continue, and to toss the recalcitrant technicians into the infirmary.

If you listen to Dyatlov, the Chernobyl disaster unfolds just the same as it did in history.

If you listen to the technicians and wind up tossing Dyatlov in the infirmary, what happens? Well, perhaps the technicians manage to fix the reactor. Perhaps they don't. But if they do, they won't get a medal. Powerful interests were invested in that test being completed on that night, and some unintelligible techno-gibberish from the technicians will not necessarily convince them that a disaster was narrowly averted. Heads will roll, and not the guilty ones.

This has broader implications that will be addressed later on, but while tossing Dyatlov in the infirmary would not have been enough to really prevent disaster, it seems like it would have worked on that night. To argue that the solution is not actually as simple as evicting Dyatlov is not the same as saying that Dyatlov should not have been evicted: to think something is seriously wrong and yet obey is hopelessly akratic.

But for now we move to a scenario more salvageable by individuals.

The Challenger

Roger Boisjoly, Challenger warner

The Challenger disaste...
Amyloid Plaques: Chemical Streetlight, Medical Goodhart
by J Bostock — 2021-08-26T21:25:04.804Z

Alzheimer's Disease (AD) is truly, unduly cruel, and truly, unduly common. A huge amount of effort goes into curing it, which I think is a true credit to our civilization. This is in the form of both money, and the efforts of many of the brightest researchers.

But it hasn't worked.

Since AD is characterised by amyloid plaques, the "amyloid hypothesis" that these were the causative agent has been popular for a while. Mutations to genes which encode the amyloid beta protein can cause AD. Putting lots of amyloid into the brain causes brain damage in mice. So for many years, drugs were screened by testing them in mutant mice which were predisposed to AD. If the plaques disappeared, they were considered good candidates.

So why didn't it work?

Lots of things can affect amyloid plaques as it turns out, right up to the latest FDA approved drug, which is just antibodies which target amyloid protein. While this does reduce amyloid, it has no effect on cognitive decline.

Goodhart's law has reared its head: amyloid plaque buildup is a metric for AD progression, but selecting for drugs which reduce it causes the relationship between AD and plaques to fall apart.

Equally, amyloid plaques are very easy to measure in mouse (and human) brains. It can be done by MRI scan, or by dissection. Memory loss and mood changes are harder to measure, and even harder in mice. The methods for measuring amyloid plaques also feel better in many ways. There's less variation in potential methods, they can be compared across species, they're qualitative, and they're also more in line with what the average biologist/chemist will be used to.

Understanding these, we can see how looking for drugs which decrease amyloid plaques in mice just really feels like productive research. We can also understand, now, why it wasn't.

Avoiding Wasted Effort

Pointing out biases is fairly useless. Pointing out specific examples is better. But the best way to help others is to point out how it feels from the inside to be making these mistakes.

So what does it feel like to be on the inside of these biases? Unfortunately as someone who has not been intimately involved in AD research I can't say exactly. But as someone involved with research in general I can make a guess:

Research will feel mostly productive. It may feel like you are becoming how you imagine a researcher to be. Papers will be published. This is because you're

...
A brief review of The Scout Mindset
by Adam Zerner — 2021-08-26T20:47:08.731Z
Link: https://www.goodreads.com/review/show/4195404777

I've been reading blogs like Less Wrong for almost a decade. So then, a lot of what was said in this book wasn't new to me. However, I still really liked it.

I'm of the opinion that in order to deeply understand a topic, it's not just enough to understand it conceptually, you have to see lots and lots of examples of it, from many different angles. I felt like this book helped me with that. Despite the fact that I spend so much time reading rationality related blogs, The Scout Mindset still felt like it non-trivially deepened my understanding of the subject matter.

The stories that were told in this book were really good. They were a nice blend of appropriate, instructive, and engaging. I'm always impressed when books like these manage to do this. I spend too much of my time reading blogs and not enough of my time reading books. I often find myself similarly impressed by the quality of stories chosen in other books as well.

My main critique of The Scout Mindset is that, well, let me start with this. Early in the book, Julia pointed out that the difficulty isn't in knowing that you should do stuff like account for cognitive biases. It's in actually bringing yourself to do it!

My path to this book began in 2009, after I quit graduate school and threw myself into a passion project that became a new career: helping people reason out tough questions in their personal and professional lives. At first I imagined that this would involve teaching people about things like probability, logic, and cognitive biases, and showing them how those subjects applied to everyday life. But after several years of running workshops, reading studies, doing consulting, and interviewing people, I finally came to accept that knowing how to reason wasn't the cure-all I thought it was.

Knowing that you should test your assumptions doesn't automatically improve your judgement, any more than knowing you should exercise automatically improves your health. Being able to rattle off a list of biases and fallacies doesn't help you unless you're willing to acknowledge those biases and fallacies in your own thinking. The biggest lesson I learned is something that's since been corroborated by researchers, as we'll see in this book: our judgment isn't limited by knowledge nearly as much as it's limited by attitude.

So then, the question becomes, how do you actually get people to have a scout mindset? ...
Signaling Virtuous Victimhood as Indicators of Dark Triad Personalities
by Ben Smith — 2021-08-26T19:18:22.366Z
Link: https://mlpol.net/images/src/218781D0ACFBE48604465CBAF74530FC-244309.pdf

A recent paper at the University of British Columbia describes five studies which, taken together, provide evidence that tendency to engage in victimhood signalling is correlated across individuals with Dark Triad personality traits (Machiavellianism, narcissism, and psychopathy).

Results were robust across the general US and Canada population gathered on mTurk as well as within a sample of Canadian undergraduates.

A link to the full study is provided in this post.
Introduction to Reducing Goodhart
by Charlie Steiner — 2021-08-26T18:38:51.592Z

(This work was supported by CEEALAR and LTFF. Thanks to James Flaville, Jason Green-Lowe, Michele Campolo, Justis Mills, Peter Barnett, and Steve Byrnes for conversations.)

I - Prologue

A few months ago, I wanted to write about AI designs that evade Goodhart's law. But as I wrote that post, I became progressively more convinced that framing things that way was leading me to talk complete nonsense. I want to explore why Goodhart's law led me to talking nonsense and try to find a different (though not entirely original, see

Rohin et al.

, Stuart

1

,

2

,

3

) framing of core issues, one which avoids assuming that we can model humans as idealized agents.

This post is the first of a sequence of five posts about Goodhart's law and AIs that learn human values (a research problem also called value learning). In this introduction I'll point out why you can't just do things the straightforward way. Leave a comment below telling me what's unclear, or what you disagree with.

II - Introduction

Goodhart's law is the observation that when you try to pick a specific observable to optimize for, the act of optimization will drive a wedge between what you're optimizing and what you want, even if they used to be correlated. For example, if what you really want is for students to get a general education, and there's a short 100-question test that correlates with how much students know, it might seem like a good idea to change schools in whatever way increases test scores. But this would lead to teaching the students only those 100 test questions and not anything else - optimizing for a proxy for education actually made the education worse.

In Scott Garrabrant's terminology from

Goodhart Taxonomy

, suppose that we have some true preference function V (for "True Values") over worlds, and U is some proxy that has been correlated with V in the past. Then there are a few distinct reasons why maximizing U may score poorly according to V. Things that seem really good according to U can just be random noise, or can be drawn from a part of the distribution that's extreme in many other ways too, or can intervene on the world without causing what we really want.

Based just on what I've said so far, it might seem like this is cause for pessimism about an AI learning human values. If our values are V, and we build an AI with effective utili...
Narrative truth
by Chris_Leong — 2021-08-26T17:49:38.310Z

One idea I encountered when investigating Jordan Peterson was the idea of narrative truth.

This is the kind of concept that most people nod along too, but which is almost always left implicit, so I thought it'd be worthwhile doing so here.

Let's quote what some other people have written on this subject:
Literal and Narrative Truth - Dave King

His story raises an interesting question, though – what does it mean for a memoir to be accurate? One of the largest issues we dealt with was the matter of dialogue. He wanted to be absolutely scrupulous, telling stories precisely as they happened. But in his original draft, his characters only spoke when he could remember what they said word-for-word. Since the manuscript was written years after the fact, this meant he used very little dialogue – mostly bursts of highly memorable lines like, “I must live, I must tell!” Nearly all the rest of his conversations were narrative summary, and many of his scenes felt flat and distant as a result. He was telling the story to readers rather than letting them experience it.

I agreed with his absolute scruple about accuracy, but argued that he needed to focus on a different kind of accuracy – narrative accuracy rather than literal accuracy. He needed to create dialogue that would make his readers feel the way he felt at the time. This meant literally putting words in his characters’ mouths, even if those words conveyed the gist of a dialogue that actually happened. But since the point of his narrative was to allow his readers to experience what he had experienced, the scenes with recreated dialogue were more accurate than the flat, emotionless scenes.

Many memoirists have taken this technique a step further and created composite characters. For instance, in Dreams of my Father, President Obama’s “New York girlfriend” was actually an amalgam of several girlfriends he’d had in New York and Chicago. I’d argue that combining several minor characters into a single character who represents the type is another form of narrative accuracy. If you had, for instance, several high school teachers who inspired you in similar ways, you could take the time to create each of them as a minor character. But all these excess characters would do more than simply slow your narrative down. By spending time on each teacher, you would giv

...