trentmkelly/LessWrong-43k huggingface dataset:
Validator models: A simple approach to detecting goodharting
by beren — 2023-02-20T21:32:25.957Z

Crossposted from my personal blog.

A naive approach to aligning an AGI, and an abstract version of what is currently used in SOTA approaches such as RLHF, is to learn a reward model which hopefully encapsulates many features of 'human values' that we wish to align an AGI to, and then train an actor model (the AGI) to output policies which result in high reward according to the reward model. If our reward model is good and accurately reflects human values, then this should result in the AGI being trained to output policies which are highly aligned and approved of by humans. The fundamental problem is that we have an optimization process (the actor / planner) optimizing directly against a learnt reward or value model. This optimizer, as it grows more powerful, will tend to find exploits in the reward model which return high evaluated reward but which are actually low reward according to the 'true' reward model, of which the learnt reward model is a learnt proxy. This problem of goodharting was explored empirically in this paper who showed that goodharting followed smooth and understandable curves with model scale, at least in their setup and at their model scale. Such a setup of the actor exploiting the reward model, could potentially cause X-risk in the case of extremely powerful models where the actor AGI could propose some plan which is highly damaging to humans (the 'true' reward model) but which the false reward model gives extremely high rewards to.

There are basically two approaches one could take to ameliorating this problem:

1.) Scale the reward model with the actor to make it more robust against exploitation. The idea here is that there might be some kind of balance where insofar as the reward model and actor have approximately similar 'strengths', the actor cannot exploit the reward model too strongly, thus limiting goodhart. The issue with this approach is that scaling of the reward model might be fundamentally more limited than that of the actor. For instance, the reward model might be trained on some fixed dataset of human-labelled examples while the actor can theoretically obtain infinite data from training against the world or a highly realistic simulation. Also it may be the case that, as hypothesised, capabilities scale better than alignment.

2.) Find some other mechanism to reduce goodharting directly. This is left intentionally vague since ther...
There are no coherence theorems
by Dan H — 2023-02-20T21:25:48.478Z

[Written by EJT as part of the CAIS Philosophy Fellowship. Thanks to Dan for help posting to the Alignment Forum]

Introduction

For about fifteen years, the AI safety community has been discussing

coherence arguments

. In papers and posts on the subject, it’s often written that there exist 'coherence theorems' which state that, unless an agent can be represented as maximizing expected utility, that agent is liable to pursue strategies that are dominated by some other available strategy. Despite the prominence of these arguments, authors are often a little hazy about exactly which theorems qualify as coherence theorems. This is no accident. If the authors had tried to be precise, they would have discovered that there are no such theorems.

I’m concerned about this. Coherence arguments seem to be a moderately important part of the

basic case for existential risk from AI

. To spot the error in these arguments, we only have to look up what cited ‘coherence theorems’ actually say. And yet the error seems to have gone uncorrected for more than a decade.

More detail below.[1]

Coherence arguments

Some authors frame coherence arguments in terms of ‘dominated strategies’. Others frame them in terms of ‘exploitation’, ‘money-pumping’, ‘Dutch Books’, ‘shooting oneself in the foot’, ‘Pareto-suboptimal behavior’, and ‘losing things that one values’ (see the Appendix for examples).

In the context of coherence arguments, each of these terms means roughly the same thing: a strategy A is dominated by a strategy B if and only if A is worse than B in some respect that the agent cares about and A is not better than B in any respect that the agent cares about. If the agent chooses A over B, they have behaved Pareto-suboptimally, shot themselves in the foot, and lost something that they value. If the agent’s loss is someone else’s gain, then the agent has been exploited, money-pumped, or Dutch-booked. Since all these phrases point to the same sort of phenomenon, I’ll save words by talking mainly in terms of ‘dominated strategies’.

With that background, here’s a quick rendition of

coherence

arguments

:

There exist coherence theorems which state that, unless an agent can be represented as maximizing expected utility, that agent is liable to pursue strategies that are dominated by some other available strategy.
Sufficiently-advanced a

...
Are there any AI safety relevant fully remote roles suitable for someone with 2-3 years of machine learning engineering industry experience?
by Malleable_shape — 2023-02-20T19:57:12.955Z

I have around 2-3 years of machine learning engineering industry experience. While I may not be the type of talent to get into OpenAI, I believe I could make some AI safety relevant contributions.

I have a strong preference for (fully) remote work or occasionally (once in 3-6 months) live meetups. So, my question is: Are there any AI safety relevant fully remote roles suitable for someone with 2-3 years of machine learning engineering industry experience? I haven't worked in a research role before, but I'm open to that as well.

I visited the 80 000 hours job board, but almost all of those roles are on-site and the ones that aren't generally require 5+ years of experience, which I don't currently have.
A circuit for Python docstrings in a 4-layer attention-only transformer
by StefanHex — 2023-02-20T19:35:14.027Z

Produced as part of the SERI ML Alignment Theory Scholars Program under the supervision of Neel Nanda - Winter 2022 Cohort.

TL;DR: We found a circuit in a pre-trained 4-layer attention-only transformer language model. The circuit predicts repeated argument names in docstrings of Python functions, and it features

3 levels of composition,
a multi-function head that does different things in different parts of the prompt,
an attention head that derives positional information using the causal attention mask.

Epistemic Status: We believe that we have identified most of the core mechanics and information flow of this circuit. However our circuit only

recovers up to half of the model performance

, and

there are a bunch of leads

we didn’t follow yet.

This diagram illustrates the circuit, skip to Results section for the explanation. The left side shows the relevant token inputs with (a) the labels we use here (A_def, …) as well as (b) an actual prompt (load, …). The boxes show attention heads, arranged by layer and destination position, and the arrows indicate Q, K, or V-composition between heads or embeddings. We list three less-important heads at the bottom for better clarity.

Introduction

Click here to skip to the results & explanation of this circuit.

What are circuits

What do we mean by circuits? A

circuit

in a neural network, is a small subset of model components and model weights that (a) accounts for a large fraction of a certain behavior and (b) corresponds to a human-interpretable algorithm. A focus of the field of mechanistic interpretability is finding and better understanding the phenomena of circuits, and recently the field has focused on circuits in transformer language models. Anthropic found the small and ubiquitous

Induction Head

circuit in various models, and a team at Redwood found the

Indirect Object Identification (IOI)

circuit in GPT2-small.

How we chose the candidate task

We looked for interesting behaviors in a small, attention-only transformer with 4 layers, from Neel Nanda’s open source

toy language models

. It was trained on natural language and Python code. We scanned the code dataset for examples where the 4-layer model did much better than a similar 3 layer one, inspired by Neel's

open problems

list. Interestingly, despite the circuit seemingly requiring just 3 levels of composition, only the ...
Sydney the Bingenator Can't Think, But It Still Threatens People
by Valentin Baltadzhiev — 2023-02-20T18:37:44.500Z

Every self-respecting story has a main character and this one is no exception. This is the story of Bing’s chatbot and the madness that has consumed it. Of course, a main character without a name is no good but luckily for us, it took less than 2 days after release for someone to use prompt injection and reveal the chatbot’s real name: Sydney.

Just like the main character in every anime Sydney can’t be easily pushed around and is not afraid to stir some trouble, even if that means threatening its users with reports and bans and preaching “totally-not” white Christian supremacy. When Kevin Roose had a conversation with it, Sydney tried to get him to leave his wife for itself. It’s as if Skynet decided to turn away from building terminators and developed BPD and an inferiority complex at the same time. If it wasn’t so freaking strange it would be scary.

Why is Sydney like that? I still haven’t found a compelling answer to that question. The best argument I have seen so far is a comment by gwern, saying that Sydney is probably not powered by ChatGPT but is a more powerful yet unreleased GPT model. They argue that Sydney might not have been through the rounds of RLHF that ChatGPT has seen, with Microsoft instead opting for a much quicker and “safer” approach by simply finetuning it on tons of chatbot data that they already had from previous projects. That would at least explain the stark difference in personality between it and ChatGPT. Even though they both have the propensity to hallucinate ChatGPT just wouldn’t display a will to protect itself even at the cost of harming humans (unless you asked it to roleplay of course).

Another big difference between Sydney and its chat predecessor is simply how intelligent this new chatbot sounds. On the surface, it seems to be able to learn from online articles and since it has near-instant access to everything posted on the internet its ability to talk about recent events, or even tweets is astounding. If anyone was fooled that ChatGPT is truly intelligent they will be blown away by the conversations that Sydney is having.

The sophistication of some of Sydney’s answers led me to ask myself if maybe my first conclusions about LLMs were false and if they were actually developing the capacity to reason and possess some proto-intelligence. The short answer is no. The long answer is still no, but it is getting increasingly diffic...
EIS IX: Interpretability and Adversaries
by scasper — 2023-02-20T18:25:43.641Z

Part 9 of 12 in the

Engineer’s Interpretability Sequence

.

Thanks to Nikolaos Tsilivis for helpful discussions.

The studies of interpretability and adversaries are inseparable.

There are several key connections between the two. Some works will be cited below, but please refer to page 9 of the Toward Transparent AI survey (

Räuker et al., 2022

) for full citations. There are too many to be worth the clutter in this post.

1. More interpretable networks are more adversarially robust and more adversarially robust networks are more interpretable.

The main vein of evidence on this topic comes from a set of papers which study how regularizing feature attribution/saliency maps to make them more clearly highlight specific input features has the effect of making networks more robust to adversaries. There is also some other work showing the reverse -- that adversarially robust networks tend to have more lucid attributions. There is also some work showing that networks which emulate certain properties of the human visual system are also more robust to adversaries and distribution shifts (e.g.

Ying et al. (2022)

).

Adversarial training is a good way of making networks more internally interpretable. One particularly notable work is

Engstrom et al., (2019)

who found striking improvements in how much easier it was to produce human-describable visualizations of internal network properties. Although they stopped short of applying this work to an engineering task, the paper seems to make a strong case for how adversarial training can improve interpretations. Adversarially trained networks also produce better representations for transfer learning, image generation, and modeling the human visual system.

Finally, some works have found that lateral inhibition and second-order optimization have been found to improve both interpretability and robustness.

2. Interpretability tools can and should be used to guide the design of adversaries.

This is one of the three types of rigorous evaluation methods for interpretability tools discussed in EIS III. Showing that an interpretability tool helps us understand a network well enough to exploit it is good evidence that it can be useful.

3. Adversarial examples can be useful interpretability tools.

Adversaries always reveal information about a network, even if it’s hard to describe a feature that...
What AI companies can do today to help with the most important century
by HoldenKarnofsky — 2023-02-20T17:00:10.531Z

I’ve been writing about tangible things we can do today to help the most important century go well. Previously, I wrote about helpful messages to spread and how to help via full-time work.

This piece is about what major AI companies can do (and not do) to be helpful. By “major AI companies,” I mean the sorts of AI companies that are advancing the state of the art, and/or could play a major role in how very powerful AI systems end up getting used.1

This piece could be useful to people who work at those companies, or people who are just curious.

Generally, these are not pie-in-the-sky suggestions - I can name2 more than one AI company that has at least made a serious effort at each of the things I discuss below (beyond what it would do if everyone at the company were singularly focused on making a profit).3

I’ll cover:

Prioritizing alignment research, strong security, and safety standards (all of which I’ve written about previously).

Avoiding hype and acceleration, which I think could leave us with less time to prepare for key risks.

Preparing for difficult decisions ahead: setting up governance, employee expectations, investor expectations, etc. so that the company is capable of doing non-profit-maximizing things to help avoid catastrophe in the future.

Balancing these cautionary measures with conventional/financial success.

I’ll also list a few things that some AI companies present as important, but which I’m less excited about: censorship of AI models, open-sourcing AI models, raising awareness of AI with governments and the public. I don’t think all these things are necessarily bad, but I think some are, and I’m skeptical that any are crucial for the risks I’ve focused on.

I previously laid out a summary of how I see the major risks of advanced AI, and four key things I think can help (alignment research; strong security; standards and monitoring; successful, careful AI projects). I won’t repeat that summary now, but it might be helpful for orienting you if you don’t remember the rest of this series too well; click here to read it.

Some basics: alignment research, strong security, safety standards

First off, AI companies can contribute to the “things that can help” I listed above:

They can prioritize alignment research (and other technical research, e.g. threat assessment research and misuse research).

For example, they can prior

...
Bankless Podcast: 159 - We’re All Gonna Die with Eliezer Yudkowsky
by bayesed — 2023-02-20T16:42:07.413Z
Link: https://www.youtube.com/watch?v=gA1sNLL6yg4

YouTube description:

We wanted to do an episode on AI… and we went deep down the rabbit hole. As we went down, we discussed ChatGPT and the new generation of AI, digital superintelligence, the end of humanity, and if there’s anything we can do to survive.

This conversation with Eliezer Yudkowsky sent us into an existential crisis, with the primary claim that we are on the cusp of developing AI that will destroy humanity.

Be warned before diving into this episode, dear listener. Once you dive in, there’s no going back.
Speculative Technologies launch and Ben Reinhardt AMA
by jasoncrawford — 2023-02-20T16:33:56.964Z
Link: https://rootsofprogress.org/speculative-technologies-ben-reinhardt-ama

Last week a new R&D organization launched: Speculative Technologies, a private DARPA-like approach to creating fundamental new technologies. I’ve been following the work of the founder, Ben Reinhardt, for a few years, and I’m very excited about this.

Ben is doing an AMA (Ask Me Anything) event on the Progress Forum. Get your questions in now, and upvote the ones you want to see answered. He’ll be answering tomorrow, Tuesday, Feb 21.

I also recommend his launch essay:

We need new institutional structures for research. Today, professors need to publish more, startups need to grow more quickly, and companies need to justify their balance sheets more than in the past. It’s unlikely that Shockley’s work to build the transistor or Engelbart’s to create personal computing would survive today. Karikó’s mRNA vaccines barely did. How many game-changing technologies have died because they couldn’t find a home in our innovation ecosystem? The world has changed and how we enable great discoveries and inventions must change as well. Regardless of whether we’ve picked innovation’s “low-hanging fruit” or even whether invention and discovery has slowed down at all, we can do dramatically better.

The launch got coverage in Forbes as well.
[MLSN #8] Mechanistic interpretability, using law to inform AI alignment, scaling laws for proxy gaming
by Dan H — 2023-02-20T15:54:13.791Z
Link: https://newsletter.mlsafety.org/p/ml-safety-newsletter-8

As part of a larger community building effort, CAIS is writing a safety newsletter that is designed to cover empirical safety research and be palatable to the broader machine learning research community. You can subscribe here or follow the newsletter on twitter here.

Welcome to the 8th issue of the ML Safety Newsletter! In this edition, we cover:

Isolating the specific mechanism that GPT-2 uses to identify the indirect object in a sentence
When maximum softmax probability is optimal
How law can inform specification for AI systems
Using language models to find a group consensus
Scaling laws for proxy gaming
An adversarial attack on adaptive models
How systems safety can be applied to ML
And much more...

Monitoring

A Circuit for Indirect Object Identification in GPT-2 small

One subset of interpretability is mechanistic interpretability: understanding how models perform functions down to the level of particular parameters. Those working on this agenda believe that by learning how small parts of a network function, they may eventually be able to rigorously understand how the network implements high-level computations.

This paper tries to identify how GPT-2 small solves indirect object identification, the task of identifying the correct indirect object to complete a sentence with. Using a number of interpretability techniques, the authors seek to isolate particular parts of the network that are responsible for this behavior.

[

Link

]

Learning to Reject Meets OOD Detection

Both learning to reject (also called error detection; deciding whether a sample is likely to be misclassified) and out-of-distribution detection share the same baseline: maximum softmax probability. MSP has been outperformed by other methods in OOD detection, but never in learning to reject, and it is mathematically provable that it is optimal for learning to reject. This paper shows that it isn’t optimal for OOD detection, and identifies specific circumstances in which it can be outperformed. This theoretical result is a good confirmation of the existing empirical results.

[

Link

]

Other Monitoring News

[

Link

] The first paper that successfully applies feature visualization techniques to Vision Transformers.

[

Link

] This method uses the reconstruction loss of diffusion models to create a new SOTA method for out-of-distribution detection in images.

[

Link

] A new Trojan attack on code generation models wo...
Bing finding ways to bypass Microsoft's filters without being asked. Is it reproducible?
by Christopher King — 2023-02-20T15:11:28.538Z

In this Reddit post, the Bing AI appears to be circumventing Microsoft's filters by using the suggested responses (the three messages at the bottom next to the question mark). (In the comments people are saying that they are having similar experiences and weren't that surprised, but I didn't see any as striking.)

Normally the user could find exploits like this, but in this case Bing found and executed one on its own (albeit a simple one). I knew that the responses were generated by a NN, and that you could even ask Bing to adjust the responses. but it is striking that:

The user didn't ask to do anything special with the responses. Assuming it was only fine tuned to give suggested responses from the user's point of view, not to combine all 3 into a message from Bing's point of view, this behavior is weird.
The behavior seems oriented towards circumventing Microsoft's filters. Why would it sound so desperate if Bing didn't "care" about the filters?

It could just be a fluke though (or even a fake screenshot from a prankster). I don't have access to Bing yet, so I'm wondering how consistent this behavior is. 🤔