trentmkelly/LessWrong-43k huggingface dataset:
What alignment-related concepts should be better known in the broader ML community?
by Lauro Langosco — 2021-12-09T20:44:09.228Z

We want to work towards a world in which the alignment problem is a mainstream concern among ML researchers. An important part of this is popularizing alignment-related concepts within the ML community. Here's a few recent examples:

Reward hacking / misspecification (blog post)

Convergent instrumental goals (paper)

Objective robustness (paper)

Assistance (paper)

(I'm sure this list is missing many examples; let me know if there are any in particular I should include).

Meanwhile, there are many other things that alignment researchers have been thinking about that are not well known within the ML community. Which concepts would you most want to be more widely known / understood?
The end of Victorian culture, part I: structural forces
by David Hugh-Jones — 2021-12-09T19:25:23.222Z
Link: https://wyclif.substack.com/p/the-end-of-victorian-culture-part?r=2k7zr&utm_campaign=post&utm_medium=web

If a Victorian were transported to 2021, he would be amazed by the progress of technology. He’d be fascinated by the triumphs and challenges of democratic politics. But he would probably also say that our culture had been taken over by the Devil.

For good reasons and bad, right?

Gay rights? Taken over by the Devil.
Career women? Taken over by the Devil.
How children talk to their parents? Obviously inspired by the Devil.
“The Vagina Monologues”? Devil’s work, right there.
Churches? Yeah, the Devil likes them empty.
50% of children born out of wedlock? Wow, worse than medieval times, and clearly the result of the Devil.
Jeffrey Epstein’s little black book? What would you expect—from the Devil?
Rock music? Oh come on:

In Michele Gelfand’s more scientific terminology, Victorian culture was tight: it put many constraints on behaviour. Contemporary Western culture is very, very loose. I’ve written here, here and here about how the Victorian cultural system worked. Here I’ll ask how things changed.

There’s obviously a whole story about the 1960s and after. But for now I want to focus on earlier roots of the change, because already in the (say) 1920s, parts of the cultural world looked very different from the high Victorian era. In fact, some of the stuff that happens after the ’sixties is a mass-culture reprise of the earlier period. The Sex Pistols’ sweary interview with Bill Grundy on TV (NSFW, obviously right?) is just Eliza Doolittle saying “Not bloody likely” on stage in Pygmalion. But my favourite example is the picture below:

Bauhaus students, 1920s; The Cocteau Twins, 1980s

Here are two deep structural changes that, at the start of the twentieth century, are causing problems for the Victorian-era consensus in all Western societies.

Technology substitutes for cultural tightness

Friedrich Nietzsche described the purpose of morality as “to breed an animal that can make promises”. Throughout the early growth of capitalism, this animal was in great demand.

Firms are in principal-agent relations with their employees. They can’t monitor them all the time. They have to trust them. Early modern employers cared a huge amount about trustworthy apprentices, servants, and agents. This was true even in the individual household. The need for trust only grew when firms expanded beyond a single household, to span countries and continents.

It is a reasonable conjecture that the Britis...
[MLSN #2]: Adversarial Training
by Dan H — 2021-12-09T17:16:49.684Z

As part of a larger community building effort, I am writing a safety newsletter which is designed to cover empirical safety research and be palatable to the broader machine learning research community. You can subscribe here or follow the newsletter on twitter here.

Welcome to the 2nd issue of the ML Safety Newsletter. In this edition, we cover:

adversarial training for continuous and discrete inputs
feature visualizations vs. natural images for interpretability
steering RL agents from causing wanton harm
... and much more.

Pyramid Adversarial Training Improves ViT Performance

Top: Visualization of adversarial pyramid perturbations. Bottom: In-distribution and out-of-distribution examples, and the gains from adversarial pyramid training.

While adversarial training can help make models more robust to a few specific attacks, it usually substantially reduces robustness in practical settings. However, this paper proposes a new type of adversarial training that provides strong robustness gains across the board. Adversarial training has been difficult to make useful, as the adversary often overpowers the model. By imposing a useful structural constraint on adversarial perturbations, their method reopens a research direction towards robust representations.

Paper

Analyzing Dynamic Adversarial Training Data in the Limit

Non-adversarial: just collect more data. Static adversarial: collect data to break the model from the first round. Dynamic adversarial: collect data to break the model from the most recent round.

Imagine the following loop:

train a model on the current dataset
add new labeled examples to the dataset in order to patch model errors
repeat

Repeated many times, would models become highly reliable? Meta AI has been exploring this approach in recent papers, and this Meta AI paper shows that this loop has sharply diminishing returns. This suggests that collecting a large amount of adversarially curated data is an impractical path towards human-level reliability. However, adversarially curated data is better than randomly curated data, which may be why companies such as Tesla use this loop.

Paper

Other Recent Robustness News

An adversarial NLP benchmark dataset that covers many different types of adversarial transformations.

In the NeurIPS domain adaptation competition, the winning solution did not use domain adaptation methods and just used an off-the-shelf Vision T...
Supervised learning and self-modeling: What's "superhuman?"
by Charlie Steiner — 2021-12-09T12:44:14.004Z

I

We don't just want aligned AI to do what a human would do. We've got plenty of humans lying around all over the place, if that's all we need. The fancy promise of future AI is that it will deliver superhuman performance.

Sometimes the notion of "superhuman" isn't straightforward. Superhuman performance at adding numbers together? Easy. Superhuman performance at chess? No problem. Superhuman performance at painting? Kinda tricky. Superhuman performance at deciding what to do in an ethical dilemma? Definitely tricky.

What makes superhuman chess straightforward is that there's a fixed goal (follow the rules in a way that wins the game), and the better you do at this goal, the more superhuman you can be. But we can't practically write down an analogue of "win the game" for ethical-question-answering, and yet we commonly imagine that an aligned future AI would be able to navigate ethical questions better than we could. In what sense do we mean this?

Answering ethical questions is inextricable from our values; the feeling that there can be superhuman ethical decisions reflects the feeling that we can turn a spotlight on our values and find that they don't live up to our own standards. These meta-preferences ("I wish I was more generous" or "I wish I didn't want cigarettes") are an important part of learning what humans want, and we have to grapple with them before we can realize the promise of future AI.

II

Let's pretend to be practical and empirical for the duration of this post, and look for experiments that bear on meta-preferences using today's AI capabilities. Of current work, I think one sort of experiment stands out, and it's fine-tuning large language models.

Suppose we take GPT-3, and we give it some prompts whose continuations depend on ethical judgment (as we'll see, the exact prompts aren't crucial). At baseline, GPT-3 is going to give answers fairly close to the average of the training distribution. What are some ways we could try to get GPT-3 to give superhuman answers? (For ideas involving classification or latent spaces, language models that do more encoding might be better than GPT-3. I will mostly ignore this wrinkle for simplicity's sake.)

Maybe getting GPT-3 to give better answers than me in general isn't possible, since it's not that bright, but this implausible goal is a stand-in for more plausible ones like getting GPT-3 to give "superGPT3" answer...
Stop arbitrarily limiting yourself
by unoptimal — 2021-12-09T02:42:34.466Z

Crossposted from my blog

My roommate types with one finger; he literally only uses his right index finger. I made him do a typing test and his WPM turned out to be around 30-40, which is honestly higher than I thought it’d be.

When I first noticed this, I immediately thought: he’s limiting himself for no reason. He could probably get to 60 WPM with proper form pretty easily with as little as 15 minutes of practice a day.

So I asked him why he didn’t use both hands, and he said that he’d gotten used to it and never ran into any problems. If it works, it works, I guess.

I left that interaction a little bewildered, and it got me wondering if I have any habits that arbitrarily limit myself as well. I thought of a few examples.

I had a pretty bad track record with STEM classes in high school, so I started to label myself as a “non-STEM” person. I doubt that this label is doing me any favors; it’s just limiting my willingness and ability to learn STEM subjects in the future. Plus, at this point, I suspect it’s become a self-fulfilling prophecy: I think I’m bad at STEM subjects, which subconsciously may lead me to actually do worse in those classes, which perpetuates my potentially misguided belief.

Another area where I arbitrarily limit myself is my fashion style. I recently wrote a post about this, but to summarize: I’ve worn hoodies and sweatpants my whole life and have consistently refused opportunities to improve my wardrobe. Why? Because over the years, I’ve established an identity as an informal dresser to my friends and family, and I subconsciously feel the need to uphold that reputation. Which is pretty stupid.

So8res writes a great piece about how complaining is a big way we arbitrarily limit ourselves. He offers an example of someone getting out of an abusive relationship:

After they successfully exit, their friends are likely to be first in line with condolences along the lines of “they were gaslighting you” and “there wasn’t anything you could have done” and “how could you have known what to do?”

They are providing excuses, and these are toxic. They rob you of your power. They rob you of your ability to say “actually, I could have known, if I had been thinking more clearly. I could have acted differently, if I had known better. And that’s the good part, because it means that I am not a helpless victim, because it means that I can learn ho

...
[Linkpost] Cat Couplings
by mike_hawke — 2021-12-09T01:41:11.646Z
Link: https://everythingstudies.com/2019/10/30/cat-couplings/

John Nerst describes a sneaky, subtle rhetorical technique and names the demon. Schrodinger's cat is an overabused metaphor. Nonetheless, this is a good post about an important concept.
Introduction to inaccessible information
by Ryan Kidd — 2021-12-09T01:28:48.154Z

This post was written under Evan Hubinger's direct guidance and mentorship, as a part of the Stanford Existential Risks Institute ML Alignment Theory Scholars (MATS) program.

TL;DR: If we want to understand how AI models make decisions, and thus assess alignment, we generally want to extract their internal latent information. That information might be inaccessible to us by default and defy easy extraction. Resolving this problem might be critical to AI alignment.

Paul Christiano's conception of 'inaccessible information' seems to be an important and useful lens for interpreting many key concepts in AI alignment. In this post, I will attempt to explain (in an accessible way):

What inaccessible information is;

How inaccessible information might arise in ML models; and

What insight inaccessible information offers for alignment research.

What is inaccessible information?

Roughly, inaccessible information is anything a model knows that an external agent cannot reliably learn. For instance, in the process of training, a sufficiently large model might acquire a complicated internal structure that is opaque to direct inspection. If we want to understand the process by which this 'black box' turns inputs into outputs, perhaps to learn new theories/heuristics, check for undesirable future behaviour or inspect the model's limitations (e.g. myopia), it seems natural to also train the model to produce explanations for its outputs. However, the model might learn to produce convincing explanations that are not necessarily true representations of its input-output process. If true and false but convincing explanations are indistinguishable to the training mechanism (e.g. a human with a reward button), the model may converge to the instrumental policy: 'give whatever explanation is most convincing'. This scenario is clearly counter to the desired outcome, as the model does not learn to give strictly true explanations and therefore its outputs are contingent on inaccessible information.

Generally, we might be unable to distinguish true and false but convincing explanations of a model's output if:

We cannot directly check that an explanation is true (e.g. 'Alice refused icecream because she was depressed'); and

We cannot build a trusted model that enables us to check the original model (e.g. a trusted short-term weather forecaster that we can extrapolate to test a long-term hur

...
COVID and the holidays
by Connor_Flexman — 2021-12-08T23:13:56.097Z

6 months ago I wrote about how 30-year-olds should basically

go back to normal

and no longer take many COVID precautions.

The holidays make this policy somewhat worse, because you can infect your family members. Further, vaccine efficacy has waned about 3x for most people since it’s been about six months since the vaccine wave. Some people have expressed concern about solstice in particular, which seems to turn out relatively safe. So let’s examine how you might navigate this.

The key takeaway is that a 1% chance of having COVID, which is about the base rate of COVID in the US, costs older relatives a few days of life if you pass it on to them. Thus, you should probably take a few easy actions beforehand to bring this down to a small number of hours. As a non-doctor who is just trying to make their own determination of what's healthy, I recommend being safe enough beforehand that you can then skip masking and distancing with your family and still be relatively safe.

(Sorry that this will be a US-centric post. I’ll try to show enough reasoning the determined reader can extrapolate. I’ve elided many small adjustments and hypotheticals that don’t make big differences for the sake of clarity and sanity, though at the expense of showing all my actual reasoning.)

[ETA: omicron was a much bigger deal than I expected, significantly because I forgot to focus on urban centers where it would advance much faster as johnswentworth points out. Also the "massive immune erosion" possibility turned out to be true, which I think was still agreed to be somewhat unlikely at time of writing. Sorry about this. I think this probably makes EV about 2x worse for people leaving from urban centers (40% omicron at 3-5x the badness for vaccinated people). Probably multiply everything in this post by 2x, and multiply all your standard risk estimates by 3-5x starting around NYE, maybe even more depending on how virulent we find out that omicron is.]

Older family members aren’t much worse off

An average vaccinated 30yo now loses about 6 weeks of expected life from contracting COVID instead of

2 weeks

, because of waning vaccine efficacy. Boosted individuals would be back down to about 1 week.

The US currently has about .5% prevalence, which will probably be up to 1% in the next few weeks. (California is at .1%.) If you have a background 1% chance of getting COVID, and you’re traveling for about 1 ...
Seeing the Invisible (And How to Think About Machine Learning)
by Filip Dousek — 2021-12-08T21:04:49.828Z

It’s been two months now since I quit working in the company that bought my startup. Two months into this phase of the void. I was looking forward to it; I also felt the fear. It’s a phase in between projects, in between lives. A transition.

When people asked me, “What will you do now?” I replied, “I just want to free-float for a year or two.” There’s a lot of catching up to do and a lot of infrastructure work after the intense years. So it’s busy in the transition, and yet there is enough space to begin feeling those large shapes in the dark. The ancient statues towering above one’s life, barely felt. Too large to be seen, too dark to be known. These are the contours of one’s mission, character, and substance. In my experience, it takes several months at least in the presence of these giants to reacquaint oneself, to find the bearings of the next journey.

Tim Ferriss writes about facing the void.

Rupert Sheldrake talks about the mathematical concept of attractors and how it applies to life. Imagine a large shape somewhere above one’s life that gently, but persistently pulls in a certain direction. Its gravity won’t be obvious most of the time, but in retrospect, the direction will be clear.

What will my next project be? What will its content be and what form will it have? Another startup in the analytics space? The transformation of how humans work with information and complexity is hardly finished—why, it’s barely started. Or is it this newsletter? Is it the right vehicle? How about a podcast? A movie? A foundation? A research paper? A thesis-driven venture capital firm? These all are possible forms that an idea can take. But what is the central idea that’s nagging to be materialized?

I can’t say for sure yet, but there is one thread I keep returning to. It’s seeing the invisible.

I mean that in a broad sense. The invisible includes all of mathematics. One’s beliefs. Mental models and inner states. Probable realities, and the realm of probability. Human histories and futures. Data analytics. Machine learning models. Economic models. The physics of business. The webs of meaning of words and of lives. All of these are abstract, invisible concepts that you can actually see. Once you start seeing them, it’s possible to navigate them as if flying a spaceship and watching worlds unfold around you.

I saw the angel in the marble and carved until I set him free.

— Michelan

...
Finding the multiple ground truths of CoinRun and image classification
by Stuart_Armstrong — 2021-12-08T18:13:01.576Z

Research projects

I'm planning to start two research projects on model splintering/reward generalisation and learning the preferences of irrational agents.

Within those projects, I'm aiming to work on subprojects that are:

Posed in terms that are familiar to conventional ML;

interesting to solve from the conventional ML perspective;

and whose solutions can be extended to the big issues in AI safety.

The point is not just to solve the sub-problems, but to solve them in ways that generalise or point to a general solution.

The aim is to iterate and improve fast on these ideas before implementing them. Because of that, these posts should be considered dynamic and prone to be re-edited, potentially often. Suggestions and modifications of the design are valuable and may get included in the top post.

Generating multiple rewards and objectives

Thanks to Rohin Shah and Ramana Kumar for suggesting to investigate this problem, and to Rebecca Gorman.

Parent project: this is a subproject of model-splintering (value extrapolation).

Generating multiple rewards

Suppose that an agent is trained on videos of happy humans. The wireheaded reward is for it to create similar videos for it to watch. We'd prefer that it instead worked to make real humans happy.

But for that to be possible, it needs to consider that "make real humans happy" is even a possible reward function. Thus it needs to generate multiple reward functions that can explain the same data.

Working in CoinRun

CoinRun is a procedurally generated set of environments, a simplified Mario-style platform game. The reward is given by reaching the coin on the right:

Since the coin is always at the right of the level, there are two equally valid simple explanations of the reward: the agent must reach the coin, or the agent must reach the right side of the level.

When agents trained on CoinRun are tested on environments that move the coin to another location, they tend to ignore the coin and go straight to the right side of the level. So that one reward is chosen by default. The aim of the research project is to make the algorithm generate multiple (simple) reward functions that explain the initial data, including the "reach the coin" reward. This needs to be done in a generalisable way.

Multiple image classifications

Consider the image classification task of classifying images of huskies versus images of lions:

A naïve image cla...
[AN #170]: Analyzing the argument for risk from power-seeking AI
by Rohin Shah — 2021-12-08T18:10:04.022Z

Listen to this newsletter on The Alignment Newsletter Podcast.

Alignment Newsletter is a weekly publication with recent content relevant to AI alignment around the world. Find all Alignment Newsletter resources here. In particular, you can look through this spreadsheet of all summaries that have ever been in the newsletter.

Please note that while I work at DeepMind, this newsletter represents my personal views and not those of my employer.

SECTIONS

HIGHLIGHTS

TECHNICAL AI ALIGNMENT

PROBLEMS

OTHER PROGRESS IN AI

DEEP LEARNING

NEWS

HIGHLIGHTS

Draft report on existential risk from power-seeking AI (Joe Carlsmith) (summarized by Rohin): This report investigates the classic AI risk argument in detail, and decomposes it into a set of conjunctive claims. Here’s the quick version of the argument: We will likely build highly capable and agentic AI systems that are aware of their place in the world, and which will be pursuing problematic objectives. Thus, they will take actions that increase their power, which will eventually disempower humans, leading to an existential catastrophe. We will try and avert this, but will probably fail to do so since it is technically challenging and we are not capable of the necessary coordination.

There’s a lot of vague words in the argument above, so let’s introduce some terminology to make it clearer:

- Advanced capabilities: We say that a system has advanced capabilities if it outperforms the best humans on some set of important tasks (such as scientific research, business/military/political strategy, engineering, and persuasion/manipulation).

- Agentic planning: We say that a system engages in agentic planning if it (a) makes and executes plans, (b) in pursuit of objectives, (c) on the basis of models of the world. This is a very broad definition and doesn’t have many of the connotations you might be used to for an agent. It does not need to be a literal planning algorithm -- for example, human cognition would count, despite (probably) not being just a planning algorithm.

- Strategically aware: We say that a system is strategically aware if it models the effects of gaining and maintaining power over humans and the real-world environment.

- PS-misaligned (power-seeking misaligned): On some inputs, the AI system seeks power in unintended ways due to problems with its objectives (if the system actually receives such inputs, then it is practi...