trentmkelly/LessWrong-43k huggingface dataset:
Book Review: Decisive by Chip and Dan Heath
by Ian David Moss — 2020-02-06T20:15:53.000Z

I first heard of the Heath brothers (Chip and Dan) years ago when I saw Chip speak about their first two books, Made to Stick and Switch. So I was pleased to discover that they’ve written a book specifically about decision-making called Decisive.

I like Decisive primarily as an introductory take on personal decision-making. It’s also a good supplemental resource for organizational and business applications, though there are other books that offer a more thorough grounding in that arena.

According to the Heaths, the “four villains of decision-making” are as follows:

narrow framing (not thinking about enough options)
confirmation bias in gathering information
overweighting short-term emotions (such as discomfort or fear); and
overconfidence.

To combat these problems, the authors dreamed up the WRAP method, a slightly ungainly mnemonic that stands for Widen your options, Reality-test your assumptions, Attain distance before deciding, and Prepare to be wrong.

Having read a fair amount on decision-making prior to encountering Decisive, I found the first section on widening options to be most illuminating. Most decisions we make are in the form of “whether or not”: in a 1993 study by Nutt, only 29% of teams considered more than one alternative. Yet a University of Kiel study found multi-option decision processes resulted in six times as many “very good” decisions (as rated retrospectively by the decision-makers) as single-option processes.

So how do you generate more options? The Heaths suggest using the actual phrase “whether or not” as an automatic mental trigger that more options are needed. It’s also helpful to think about opportunity costs — what else could you do with the money/time? One study showed that just prompting people to think about opportunity costs caused a significant number to change purchasing intentions. Or try the Vanishing Options Test: pretend you can’t choose any of your current options — what do you do instead?

The authors make clear that decision paralysis is a real thing and they are not advocating for unlimited options, just one or two more than you would otherwise consider. Or you can figure that you’ll keep considering options until you “fall in love” with at least two of them.

“Prepare to be wrong” (the P in WRAP) is another relatively neglected piece of wisdom in my experience. This section of the book was a little unfocused, but covers some ...
Bayes-Up: An App for Sharing Bayesian-MCQ
by Louis Faucon — 2020-02-06T19:01:20.773Z

Inspired by Lê Nguyên Hoang's post on Bayesian Examination, I have been developing (as a hobby) a new app called Bayes-Up (available at: bayes-up.web.app). The app is now in a state where it is working well enough to be shared with others. In this post I list a few things you can do with it, because I expect that it will spark some interest within the community.

Test and improve your calibration: Bayes-Up uses a collection of good quality trivia questions from the open trivia database. The main point of the app is that you can find a list of multiple choice quizzes, answer questions by assigning probabilities to each of the possible choices, receive a score based off a quadratic proper scoring rule and later find statistics about the quality of your calibration. A good place to start is the quiz from the book Factfulness by Hans Rosling that I included in the app.
Create quizzes and upload them. There exists already a small number of calibration training apps. Bayes-Up differs mainly because it allows to upload and share your own quizzes. This can solve one of the problems of calibration apps which is to create good quality content (quizzes / questions). If you are a teacher and want your students to develop more metacognitive skills and intellectual honesty, or if you are organizing workshops on probability calibration, Bayes-Up can make it easier for you. To add a quiz, simply write it in a spreadsheet, export it as a CSV file and upload it in Bayes-Up.
Recommend UI improvements, new features, report bugs, or contribute to the implementation. Only very little feedback has been collected so far and certainly a lot could be improved with little effort. The code of the app is open source and hosted on github.
Analyse the data from Bayes-Up users. So far about 30'000 questions have been answered by about 1'300 users since the end of December 2019. The collected data is available at this link and will likely grow in the following months. Simple questions that analysing this data could answer are: Do users become better calibrated over time? Is calibration topic-specific or transferrable? How can the answers of users with unknown calibration and unknown knowledge be aggregated to predict the right answers to every question? Let me know if you want to do something with it or need a better documentation.
Mazes Sequence Roundup: Final Thoughts and Paths Forward
by Zvi — 2020-02-06T16:10:00.405Z

There are still two elephants in the room that I must address before concluding. Then I will discuss paths forward.

Moloch’s Army

The first elephant is Moloch’s Army. I still can’t find a way into this without sounding crazy. The result of this is that the sequence talks about maze behaviors and mazes as if their creation and operation are motivated by self-interest. That’s far from the whole picture.

There is mindset that instinctively and unselfishly opposes everything of value. This mindset is not only not doing calculations to see what it would prefer or might accomplish. It does not even believe in the concept of calculation (or numbers, or logic, or reason) at all. It cares about virtues and emotional resonances, not consequences. To do this is to have the maze nature. This mindset instinctively promotes others that share the mindset, and is much more common and impactful among the powerful than one would think. Among other things, the actions of those with this mindset are vital to the creation, support and strengthening mazes.

Until a proper description of that is finished, my job is not done. So far, it continues to elude me. I am not giving up.

Moloch’s Puzzle

The second elephant is that I opened this series with a puzzle. It is important that I come back to the beginning. I must offer my explanation of the puzzle, and end the same place the sequence began. With hope.

Thus, the following puzzle:

Every given thing is eventually doomed. Every given thing will eventually get worse. Every equilibrium is terrible. Sufficiently strong optimization pressure, whether or not it comes from competition, destroys all values not being optimized, with optimization pressure constantly increasing.

Yet all is not lost. Most of the world is better off than it has ever been and is getting better all the time. We enjoy historically outrageously wonderful bounties every day, and hold up moral standards and practical demands on many fronts that no time or place in the past could handle. How is this possible?

Here is my answer:

It is possible because old things that get sufficiently worse eventually die, and get replaced by new things that are better. It is possible because competition is never perfect, and optimization for a fixed set of metrics is never total. People need and demand slack, and care about many factors on many meta levels. No matter how many times we say it i...
Plausibly, almost every powerful algorithm would be manipulative
by Stuart_Armstrong — 2020-02-06T11:50:15.957Z

I had an interesting debate recently, about whether we could make smart AIs safe just by focusing on their structure and their task. Specifically, we were pondering something like:

"Would an algorithm be safe if it was a neural net-style image classifier, trained on examples of melanoma to detect skin cancer, with no other role than to output a probability estimate for a given picture? Even if "superintelligent", could such an algorithm be an existential risk?"

Whether it's an existential risk was not resolved; but I have a strong intuition that they would like be manipulative. Let's see how.

The requirements for manipulation

For an algorithm to be manipulative, it has to derive some advantage from manipulation, and it needs to be able to learn to manipulate - for that, it needs to be able to explore situations where it engages in manipulation and this is to its benefit.

There are certainly very simple situations where manipulation can emerge. But that example, though simple, had an agent that was active in the world. Can a classifier display the same sort of behaviour?

Manipulation emerges naturally

To show that, picture the following design. The programmers have a large collection of slightly different datasets, and want to train the algorithm on all of them. The loss function is an error rate, which can vary between 1 and 0. Many of the hyperparameters are set by a neural net, which itself takes a more "long-term view" of the error rate, trying to improve it from day to day rather than from run to run.

How have the programmers set up the system? Well, they run the algorithm on batched samples from ten datasets at once, and record the error rate for all ten. The hyperparameters are set to minimise average error over each run of ten. When the performance on one dataset falls below 0.1 error for a few runs, they remove it from the batches, and substitute in a new one to train the algorithm on[1].

So, what will happen? Well, the system will initially start to minimise the loss rate. Then, after a while, it will start "noticing" that when this error is below 0.1, then the dataset is removed and the average error rate shoots up. Actually, it won't be "noticing" this; instead, if there are hyperparameters that prevent the error rate going below 0.1, these will be selected by gradient descent as giving a better performance.

So, this problem of optimisation of image

...
Some quick notes on hand hygiene
by willbradshaw — 2020-02-06T02:47:33.981Z

I always wash my hands but typically not very well; I never worked in healthcare and never got trained. Given the current situation I decided to actually research how to do it properly.

Different sources differ slightly in their recommendations but the basic gist is the same. Some takeaways:

Washing your hands should take ~ 20-30 seconds. If it's taking you much less you're not doing it right.

Make sure you actually get all your hand surfaces, including your nails and between your fingers. The WHO guide has a good protocol that's probably worth memorising.

If possible, try to dry your hands with a single-use towel, or air-dry somehow; not much point washing your hands and then drying them on a dirty towel.

Good hand sanitiser (>60% alcohol) is a good substitute if your hands aren't visibly soiled, but you still have to do the all-surfaces routine and it should still take ~20 seconds.

For those of you who use Anki, ankifying this is probably among the most valuable things you could ever use it for. Especially in the early days of a maybe-pandemic! Ditto for those of you who use TAPs. My read of the various health authorities is that good, regular hand-washing is way ahead of gloves and masks in terms of keeping you and your loved ones safe.

Other things that now would be a good time to hammer into your head if (like me) you didn't already: coughing into your elbow, sanitising your doorknobs and light switches, and not touching your face. These are all a good idea anyway during flu season, but you should use the impetus of the current situation to actually do them , rather than just vaguely intending to learn how to do them like everyone else.

In the interests of reducing the number of people who read this, nod, vaguely intend to get better, and do nothing, here's an itemised checklist of concrete things you can do.

Read the The WHO guide and memorise/Anki the hand-cleaning/sanitation. Here is a little minideck of cards I made for my own practice.

Go to Walgreens/Boots/some other drug store and buy lots of little bottles of hand sanitiser (with alcohol!). Failing that, buy some big bottles off Amazon and aliquot them into smaller containers for use on the go.

Order some alcohol in a spray bottle you can use to clean your light switches and doorknobs (I have no particular opinion on isopropanol vs ethanol for this, if somebody does then please comment and

...
Potential Research Topic: Vingean Reflection, Value Alignment and Aspiration
by Vaughn Papenhausen — 2020-02-06T01:09:05.384Z

Epistemic Status: Potential research idea. Time-limited, so not as clear as it could have been.

Vingean reflection is the process of trying to anticipate how an agent smarter than you might think, in order to ensure that it will be aligned with your values. This is hard, because "if [an agent] could predict [a smarter agent's] actions in detail, it would already be as smart as them." Value Learning is the problem of trying to use machine learning to train an AI to care about what humans care about.

I haven't read much about these problems, but they struck me as related to a concept introduced by philosopher Agnes Callard: "aspiration." Her idea is that, sometimes, we come to care about things that we didn't care about before, and, in particular, that: (1) this doesn't happen all at once, and (2) we play an active role in the process. She argues in her book (which I haven't read yet, but see the interview I just linked with her and Robert Wright) that in several different areas of philosophy (decision theory, moral psychology, and moral responsibility) the prevailing theories make assumptions that would render this process paradoxical or impossible.

To see what aspiration looks like, consider some value that you didn't have before, but now do. Since I don't know you, I'll give a generic example, but substitute in whatever actually applies to you. Suppose you are now a gourmand, though you didn't care much about good food when you were younger (this apparently happened to a friend of Callard's). How did you get from there to here? Perhaps there was a moment where you first got excited about food (in the case of Callard's friend, she took a trip to Ousaka, Japan). But this probably isn't the whole story, at least not in many cases. This lucky, random encounter provided the first shove to get you onto the path towards being a gourmand, but it didn't take you all the way. You got an inkling of the value of good food by having some in Oosaka, but you had to choose to cultivate this interest. But how is it possible to move yourself further along this path, without already knowing how a gourmand would value good food? It seems like if you care enough to want to get better at valuing good food, then you must already be the kind of person who cares about good food. And how can you critique your own taste without already having the sort of trained palette that future-you

...
Synthesizing amplification and debate
by evhub — 2020-02-05T22:53:56.940Z

Background

One possible way to train an amplification model is to use an auxiliary reinforcement learning objective to help guide the training of the amplification model. This could be done either by training two separate models, an agent and a question-answerer, or a single model trained on a joint objective. For example, from a comment Paul left on “A dilemma for prosaic AI alignment:”

I normally imagine using joint training in these cases, rather than pre-training + fine-tuning. e.g., at every point in time we maintain an agent and a question-answerer, where the question-answerer "knows everything the agent knows." They get better together, with each gradient update affecting both of them, rather than first training a good agent and then adding a good question-answerer.

(Independently of concerns about mesa-optimization, I think the fine-tuning approach would have trouble because you couldn't use statistical regularities from the "main" objective to inform your answers to questions, and therefore your question answers will be dumber than the policy and so you couldn't get a good reward function or specification of catastrophically bad behavior.)

In my last post, I expressed skepticism of such non-imitative amplification approaches, though in this post I want to propose a possible way in which some of my concerns with this style of approach could addressed by integrating ideas from AI safety via debate. I'll start by describing the basic idea in broad terms, then give a more careful, technical description of the sort of training procedure I have in mind.

The proposal

The basic idea is as follows: debate naturally yields an RL objective, so if you want to add an auxiliary RL objective to amplification, why not use the RL objective from debate? Specifically, the idea is to conduct a debate not between copies of the model M, but between copies of the amplified model Amp(M) (where Amp(M) is a human with access to the model M). That gives you both an RL reward arising from the winner of the debate as well as a supervised loss from the difference between the model's output and the amplified model's output. That's the basic proposal—now I'll try to give a more careful, technical description of how I think it should go.

Definitions

We will start with some initial definitions.

Let Q be the set of all questions in natural language.

Let A be the set of all answe

...
Writeup: Progress on AI Safety via Debate
by Beth Barnes — 2020-02-05T21:04:05.303Z

This is a writeup of the research done by the "Reflection-Humans" team at OpenAI in Q3 and Q4 of 2019. During that period we investigated mechanisms that would allow evaluators to get correct and helpful answers from experts, without the evaluators themselves being expert in the domain of the questions. This follows from the original work on AI Safety via Debate and the call for research on human aspects of AI safety, and is also closely related to work on Iterated Amplification.

Authors and Acknowledgements

The main researchers on this project were Elizabeth Barnes, Paul Christiano, Long Ouyang and Geoffrey Irving. We are grateful to many others who offered ideas and feedback. In particular: the cross-examination idea was inspired by a conversation with Chelsea Voss; Adam Gleave had helpful ideas about the long computation problem; Jeff Wu, Danny Hernandez and Gretchen Krueger gave feedback on a draft; we had helpful conversations with Amanda Askell, Andreas Stuhlmüller and Joe Collman, as well as others on the Ought team and the OpenAI Reflection team. We’d also like to thank our contractors who participated in debate experiments, especially David Jones, Erol Akbaba, Alex Deam and Chris Painter. Oliver Habryka helped format and edit the document for the AI Alignment Forum.

Note by Oliver: There is currently a bug with links to headings in a post, causing them to not properly scroll when clicked. Until that is fixed, just open those links in a new tab, which should scroll correctly.

Overview

Motivation

As we apply ML to increasingly important and complex tasks, the problem of evaluating behaviour and providing a good training signal becomes more difficult.

We already see examples of RL leading to undesirable behaviours that superficially ‘look good’ to human evaluators (see this collection of

examples

). One example from an

OpenAI paper

is an agent learning incorrect behaviours in a 3d simulator, because the behaviours look like the desired behaviour in the 2d clip the human evaluator is seeing.

We’d like to ensure that AI systems are aligned with human values even in cases where it’s beyond human ability to thoroughly check the AI system’s work.

We can learn about designing ML objectives by studying mechanisms for eliciting helpful behavior from human experts. For example, if we hire a physicist to answer physics questions and pay them based on how

...
[AN #85]: The normative questions we should be asking for AI alignment, and a surprisingly good chatbot
by Rohin Shah — 2020-02-05T18:20:02.138Z

View this email in your browser

Find all Alignment Newsletter resources

here

. In particular, you can

sign up

, or look through this

spreadsheet

of all summaries that have ever been in the newsletter. I'm always happy to hear feedback; you can send it to me by replying to this email.

Audio version

here

(may not be up yet).

Highlights

Artificial Intelligence, Values and Alignment

(Iason Gabriel) (summarized by Rohin): This paper from a DeepMind author considers what it would mean to align an AI system. It first makes a distinction between the technical and normative aspects of the AI alignment problem. Roughly, the normative aspect asks, "what should our AI systems do?", while the technical aspect asks, "given we know what our AI systems should do, how do we get them to do it?". The author argues that these two questions are interrelated and should not be solved separately: for example, the current success of deep reinforcement learning in which we maximize expected reward suggests that it would be much easier to align AI to a utilitarian framework in which we maximize expected utility, as opposed to a deontological or Kantian framework.

The paper then explores the normative aspect, in both the single human and multiple humans case. When there's only one human, we must grapple with the problem of what to align our AI system to. The paper considers six possibilities: instructions, expressed intentions, revealed preferences, informed preferences, interests, and values, but doesn't come to a conclusion about which is best. When there are multiple humans, we must also deal with the fact that different people disagree on values. The paper analyzes three possibilities: aligning to a global notion of morality (e.g. "basic human rights"), doing what people would prefer from behind a veil of ignorance, and pursuing values that are determined by a democratic process (the domain of social choice theory).

See also

Import AI #183

Rohin's opinion: I'm excited to see more big-picture thought about AI alignment out of DeepMind. This newsletter (and I) tend to focus a lot more on the technical alignment problem than the normative one, partly because there's more work on it, but also partly because I think it is the

more urgent problem

(a

controversial

position

).

Towards a Human-like Open-Domain Chatbot

(Daniel Adiwardana et al...