trentmkelly/LessWrong-43k huggingface dataset:
Training goals for large language models
by Johannes Treutlein — 2022-07-18T07:09:42.569Z

This post was written under Evan Hubinger’s mentorship, as part of the

Stanford Existential Risks Initiative ML Alignment Theory Scholars (SERI MATS) program

. Many of the ideas in this post, including the main idea behind the training goal, are due to Kyle McDonell and Laria Reynolds. In addition, I am grateful for comments and feedback from Arun Jose (who wrote a related post on

conditioning generative models for alignment

) and Caspar Oesterheld, and for a helpful discussion with James Lucassen.

Introduction

Large language models (LLMs) have recently enjoyed much success, e.g.,

achieving 50% accuracy

on high school math competition questions. These models can solve various tasks using the right prompts or fine-tuning, such as translation, summarization, or question answering. One path to human-level and potentially superhuman AGI might be scaling up LLMs. This raises the question of what an approach to aligned AGI would look like based on these models.

One hypothesis is that, while LLMs are very competent, they are not adequately described as agents. Instead, one might describe them as myopic

simulators

that model a distribution over text, without understanding their place in the world or their actions' causal impact on it. For this reason, such models might be safer to use than more agentic models that pursue goals in the world.

In this post, I develop a training goal for LLMs, in Evan Hubinger’s

terminology

. A training goal is a description, as concrete and low-level as possible, of the algorithm that the model should implement, and an explanation of how the model will be used and why it will be aligned for that purpose. I focus on models that generate a distribution over text, conditioned on a prompt, but that are more capable than today’s models. My goal is to provide an overview of potential problems and solutions, most of which have been raised in prior work.

I will not focus on a training rationale for such models, which is the question of how to train a model to implement the described algorithm using machine learning. However, the proposal is only useful if one could actually train such an algorithm, so I will still touch on training. I also won’t discuss the proposal’s competitiveness, and I won’t look into specific prompts one might use. Lastly, note that I am not an expert on LLMs and will mostly analyze things from...
A distillation of Evan Hubinger's training stories (for SERI MATS)
by Daphne_W — 2022-07-18T03:38:38.994Z

This post is a distillation of Evan Hubinger's post "how do we become confident in the safety of a machine learning system?", made as part of the summer 2022 SERI MATS program. While I have attempted to understand and extrapolate Evan's opinions, this post has not been vetted. Likewise, I use training stories (and contribution stories) to describe the methodology of proposals for safe advanced AI without the endorsement of those proposals' authors and based on a relatively shallow understanding of those proposals (due to my inexperience and time constraints). The opinions presented in this post are my own unless otherwise noted.

Epistemic status: Exploratory

Some day, all too soon from now, people will deploy the AI that seals humanity's fate. There are many scenarios for how this comes about or what happens afterward. Some have multiple AI negotiating or fighting for dominance, others one. Some think the handoff of power from humans to AI will go slowly, others fast. But whatever the case, there is one question that these people need to get right: "Is the AI we're about to deploy safe?"

For people to get this answer right when it matters, two things need to happen: we need tools to accurately determine whether an advanced AI is safe, and we need an advanced AI those tools approve of that we can deploy in time for it to matter.

Training stories are a tool to evaluate proposals for making this happen. Specifically, they're meant to analyze complete proposals for training prosaic advanced AI: advanced AI which has similar enough architecture to current systems that most research is cross-applicable. The reason to focus on complete proposals is that it gets us to pay attention to what will be important when push comes to shove. From there, we can backpropagate and get an estimate of what research paths are most beneficial. Evan thinks theoretical prosaic alignment is the most beneficial in expectation for him and many others to pursue, though he supports a broad spectrum of research.

Because I think the original post describes the central concept of training stories very well, I have avoided retreading the same ground, and instead focused on expanding its usability. I will start with some more case studies of using training stories to analyze proposals, and then expand on the original post with contribution stories: a method for analyzing research agendas by h...
Forecasting ML Benchmarks in 2023
by jsteinhardt — 2022-07-18T02:50:16.683Z

Thanks to Collin Burns, Ruiqi Zhong, Cassidy Laidlaw, Jean-Stanislas Denain, and Erik Jones, who generated most of the considerations discussed in this post.

Previously, I evaluated the accuracy of forecasts about performance on the MATH and MMLU (Massive Multitask) datasets. I argued that most people, including myself, significantly underestimated the rate of progress, and encouraged ML researchers to make forecasts for the next year in order to become more calibrated.

In that spirit, I’ll offer my own forecasts for state-of-the-art performance on MATH and MMLU. Following the corresponding Metaculus questions, I’ll forecast accuracy as of June 30, 2023. My forecasts are based on a one-hour exercise I performed with my research group, where we brainstormed considerations, looked up relevant information, formed initial forecasts, discussed, and then made updated forecasts. It was fairly easy to devote one group meeting to this, and I’d encourage other research groups to do the same.

Below, I’ll describe my reasoning for the MATH and MMLU forecasts in turn. I’ll review relevant background info, describe the key considerations we brainstormed followed, analyze those considerations, and then give my bottom-line forecast.

MATH

Background

Metaculus does a good job of describing the MATH dataset and corresponding forecasting question:

The MATH dataset is a dataset of challenging high school mathematics problems constructed by Hendrycks et al. (2021). Hypermind forecasters were commissioned to predict state-of-the-art performance on June 30, 2022, '23, '24, and '25. The 2022 result of 50.3% was significantly outside forecasters' prediction intervals, so we're seeing what the updated forecasts are for 2023, '24, and '25.

What will be state-of-the-art performance on the MATH dataset in the following years?

These questions should resolve identically to the Hypermind forecasts:

"These questions resolve as the highest performance achieved on MATH by June 30 in the following years by an eligible model.

Eligible models may use scratch space before outputting an answer (if desired) and may be trained in any way that does not use the test set (few-shot, fine tuned, etc.). The model need not be publicly released, as long as the resulting performance itself is reported in a published paper (on arxiv or a major ML conference) or through an official communication channel of an ind

...
What should you change in response to an "emergency"? And AI risk
by AnnaSalamon — 2022-07-18T01:11:14.667Z

This post has been recorded as part of the LessWrong Curated Podcast, and can be listened to on Spotify, Apple Podcasts, and Libsyn.

Related to: Slack gives you the ability to notice/reflect on subtle things

Epistemic status: A possibly annoying mixture of straightforward reasoning and hard-to-justify personal opinions.

It is often stated (with some justification, IMO) that AI risk is an “emergency.” Various people have explained to me that they put various parts of their normal life’s functioning on hold on account of AI being an “emergency.” In the interest of people doing this sanely and not confusedly, I’d like to take a step back and seek principles around what kinds of changes a person might want to make in an “emergency” of different sorts.

Principle 1: It matters what time-scale the emergency is on

There are plenty of ways we can temporarily increase productivity on some narrow task or other, at the cost of our longer-term resources. For example:

Skipping meals
Skipping sleep
Ceasing to clean the house or to exercise
Accumulating credit card debt
Calling in favors from friends
Skipping leisure time

If I would strongly prefer to address some situation x before time t, I may sometimes want to "borrow from the future" like this. But the time-scales matter. If I’m trying to address x as much as possible in the next five hours, skipping sleep may make sense. If I’m trying to address x as much as possible over the next year, I’ll probably do better to get my usual amount of sleep tonight. Something similar (with different, resource-specific timescales) will hold for other resources.

So, in short time-scale emergencies, it’ll often make sense to suspend a great deal of normal functioning for a short period of time. In longer time-scale emergencies, your life should mostly look closer to normal.

Principle 2: It matters how much we know how to address the emergency

Much of what we do in daily life – especially when we’re feeling “free” and “unstuck,” and as though there is nothing in particular that we “have to” do – has the effect of making us well-resourced and capable in general. For example, by default, a lot of us would spend a lot of time reading interesting books of varied sorts, nerding out about interesting topics, trying our hands at new skills and crafts, etc. Also, we often like making our living ...
Deception?! I ain’t got time for that!
by Paul Colognese — 2022-07-18T00:06:15.274Z

Or ... How penalizing computation used during training disfavors deception.

This post was written under Evan Hubinger’s direct guidance and mentorship, as a part of the

Stanford Existential Risks Institute ML Alignment Theory Scholars (MATS) program

.

Thanks to Evan Hubinger, Yonadav Shavit, and Arun Jose for helpful discussions and feedback.

Introduction

We want to use ML to produce “good” programs that perform well on some desired metrics and have some desired properties. Unfortunately, some programs that look good during, and at the end of the training, may not actually be good. This includes

deceptive programs

that pretend to look good during the training process in order to pursue their own objectives later.

Because deceptive programs have to use computation on tasks like deceptive planning, modifying the ML training process to favor programs that use less computation may help prevent the selection of deceptive programs. A training process that favors such “fast programs” is said to have a speed prior/bias.

The aim of this post is to flesh out an argument for why training processes with speed priors may disfavor deceptive programs, and to consider how speed priors might be implemented in practice.

We begin by looking at how the priors/biases of a training process may be leveraged to help us select good programs rather than programs that just look good. Building

on

previous

work

, we argue that speed priors may disfavor deceptive programs and more general forms of misbehavior. Next, we consider some potential implementations of speed priors before closing with some additional thoughts.

Using priors to help select good programs

An ML training process begins by initializing a space of programs (typically this space corresponds to the parameter space of a neural network) and then searches for a program that looks good to the training process, i.e., that seems to achieve good performance on the relevant metrics and have the desired properties. We can think of “looking good” as a constraint that programs must satisfy in order to be considered for selection.

One problem we face is that programs that look good, may not actually be good. For example, we may want a program that generalizes well to deployment inputs, but if the training process only measures performance on the training inputs, programs that look good to the training process may i...
How Interpretability can be Impactful
by Connall Garrod — 2022-07-18T00:06:05.675Z

This post was written as part of the Stanford Existential Risks Initiative ML Alignment Theory Scholars (MATS) program. thanks to Evan Hubinger for insightful discussion.

Introduction

Interpretability tools allow us to understand the internal mechanism and knowledge of a model. This post discusses methods through which interpretability could reduce x-risk from advanced AI. Most of these ideas are from Neel Nanda’s post on the same topic, my contribution comes from elaborating on them further. I have split these into a handful of different categories. These are not disjoint, but the framing helped with the analysis. The last section includes several reasons why interpretability might not be impactful that apply, for the most part, across the board. The conclusion discusses what this analysis implies for how we should approach interpretability research today.

In evaluating a method of impact, I have tried to analyse the following:

Mechanism & Scale: What is the mechanism of impact? how valuable would this be?
External considerations: what things outside of our interpretability tools need to happen in order for this impact to be realised and how likely are they?
Side effects: what possibility is there of a negative impact on x-risk?
Requirements: what do we require of our interpretability tools, as measured by:

Understanding: what properties of our model do we need to be able to understand?
Competitiveness: how cost-effective do our tools need to be?
Performance impact: to what extent can our tools impact the final model performance?
Reliability: how reliable must our tools be?
Typology: what types of interpretability tools can we use?

With regards to understanding, it is useful to split this into two classes: comprehensive tools, which let you evaluate properties of the full model, and partial tools, which let you evaluate properties of only a part of the model. This distinction is the same as worst-case transparency introduced by Evan. Both of these define a spectrum based on what the class of properties that you can evaluate is. I will refer to the size of this class as the ‘strength’ of the tools.

With regards to reliability, this captures whether the tools work across different models, as well as under gaming attempts from deceptive models or overfitting attempts from SGD.

When referring to type, I have in mind the transparency trichotomy. Specifically, we can ...
Why you might expect homogeneous take-off: evidence from ML research
by Andrei Alexandru — 2022-07-17T20:31:19.023Z

This write-up was produced as part of the SERI MATS programme under Evan Hubinger’s mentorship. It is also my first post on LW, so feedback is very welcome!

Introduction

This article aims to draw a connection between recent ML research and the claim that future advanced AI systems may be homogenous. First, I briefly review

this article

, where the idea of homogenous take-off is introduced. Then, I outline two different arguments why you might update in the direction of homogenous take-off. For each of the arguments I mention key uncertainties that I have about the argument itself, as well as broader open questions.

TL; DR

I present two reasons to believe that as models become larger they also become more homogenous, i.e. they behave more similarly to each other:

Variance between models behaves unimodally in the overparameterised regime: it peaks around the interpolation threshold, then decreases monotonically. Decreased variance means that models make similar predictions across different training runs (captured as variance from initialisation) and different sampling of the training data (variance from sampling);
Neural networks have a strong simplicity bias even before training, which might mean that multiple training runs with different hyperparameters, initialisation schemes etc. result in essentially the same model.

I’ve somewhat updated in the direction of homogenous take-off as a result of these arguments, though I think that there are still ways in which it’s unclear if e.g. decreasing variance with size rules out heterogeneity.

What’s homogeneous take-off?

There are several axes along which different AI takeoff scenarios could differ: speed, continuity, and number of main actors.

Homogeneity vs. heterogeneity in AI takeoff scenarios

introduces a new way to look at a potential take-off, through the lens of model homogeneity. Homogeneity intuitively refers to how similar models are at any given time given some definition of similarity. We might specifically refer to homogeneity with regards to alignment, which, again intuitively, means “models are more or less aligned to the same degree” (or: “aligned models will not coexist with unaligned models”).

More formally we mean something like models having similar properties, e.g. alignment properties. In my mind, an alignment property might be something like “corrigibility” or “truthful...
Examples of AI Increasing AI Progress
by TW123 — 2022-07-17T20:06:41.213Z

Recursive self-improvement is already here.

This point is far from original. It’s been described before, for instance

here

, in Drexler’s

Reframing Superintelligence

, and (as I was working on this post)

in Jack Clark’s newsletter

and even by

Yann LeCun

. But sometimes I still hear people talk about preparing for “when recursive self-improvement kicks in,” implying that it hasn’t already.

The kinds of recursive self-improvement mentioned here aren’t exactly the frequently-envisioned scenario of a single AI system improving itself unencumbered. They instead rely on humans to make them work, and humans are inevitably slow and thus currently inhibit a discontinuous foom scenario.

It may be tempting to dismiss the kind of recursive self-improvement happening today as not real recursive self-improvement. To think about it as some future event that will start to happen that we need to prepare for. Yes, we need to prepare for increasing amounts of it, but it’s not in the future, it’s in the present.

Here are some currently existing examples (years given for the particular example linked):

(2016) Models play against themselves in order to iteratively improve their performance in games, most notably in

AlphaGo

and its variants.
(2016) Some

neural architecture search

techniques use one neural network to optimize the architectures of different neural networks.
(2016)

AI is being used to optimize data center cooling

, helping reduce the cost of further scaling.
(2021) Code generation tools like

GitHub Copilot

can be helpful to software engineers, including presumably some AI research engineers (anecdotally, I’ve found it helpful when doing engineering). Engineers may thus be faster at designing AI systems, including Copilot-like systems.
(2021) Google

uses deep reinforcement learning

to optimize their AI accelerators.
(2022) Neural networks, running on NVIDIA GPUs,

have been used

to design more efficient GPUs which can in turn run more neural networks.
(2022) Neural networks

are being used

for compiler optimization in the popular LLVM compiler language, which

Pytorch’s just-in-time compiler

is based on.

Inspired by Victoria Krakovna’s

specification gaming spreadsheet

, I’ve made a spreadsheet

here

with these examples. Feel free to submit more

here

. I think the number of examples will continue to grow, m...
Four questions I ask AI safety researchers
by Orpheus16 — 2022-07-17T17:25:07.020Z

Over the last few months, I’ve been trying to develop a stronger inside view on AI safety research agendas.

As part of this quest, I’ve been having conversations with AI safety researchers. I notice myself often asking the following questions:

What are you working on, and how does it help us get aligned AI?
Imagine you’re talking to a smart high school student. How would you describe the alignment problem? And how would you describe how your work addresses it?
Imagine that you came up with a solution to the specific problem you’re working on. Or even more boldly, imagine your entire program of research succeeds. What happens next? Concretely, how does this help us get aligned AI (and prevent unaligned AI)?
What are the qualities you look for in promising AI safety researchers? (beyond general intelligence)

I find Question #1 useful for starting the conversation and introducing me to the person’s worldview.

I find Question #2 useful for getting clearer (and often more detailed) explanations of the person’s understanding of the alignment problem & how their work fits in. (Note that this is somewhat redundant with question #1, but I find that the questions often yield different answers).

I find Question #3 useful for starting to develop an inside view on the research agenda & the person’s reasoning abilities. (Note that the point is to see how the person reasons about the question. I’m not looking for a “right answer”, but I am looking to see that someone has seriously thought about this and has reasonable takes.)

I find Question #4 useful for building stronger models of AI safety community-building.

I also ask a lot of follow-up questions. But I find that these are the four questions that I ask nearly everyone, and I think they’ve helped me develop a stronger inside view.

If I had to pick one question, I would probably ask #3. I think it’s a rather challenging question, and I’m generally impressed when people have coherent answers.
Why I Think Abrupt AI Takeoff
by lincolnquirk — 2022-07-17T17:04:06.038Z

(note: Quickly written. I've attempted to number my arguments "formally", but I have no training in this format. Edits/suggestions welcome.)

Ability to generalize is "lumpy": unpredictable -- sometimes lots of inputs lead to little progress, sometimes small inputs lead to lots of progress.

Takeoff is threshold-based: When machines get enough ability to generalize, takeoff will happen.

thus

As we increase the ability of machines to generalize, we have a high chance of putting in a small amount of input and getting over the takeoff threshold.

Argument for claim 1:

Lumpy input->output curves exist in individual domains as a result of forming new and powerful abstractions in those domains.

Cognitive architecture components generalize across domains as in (4).

Cognitive architecture is itself a "domain" as in (4) and thus lumpy.

thus

(restated) If we get a new abstraction in the form of a cognitive architecture component that generalizes across domains, we will see rapid cross-domain progress, and thus ability to generalize is lumpy.

Evidence for claim 4 — "Narrow" lumpy generality — specifically, new and powerful abstraction gives rise to abrupt domain-specific performance improvements:

Anecdotally, humans have a lumpy learning experience. It is common for people to talk about "Aha moments", "eureka moments" and so on.

AlphaGo Zero seems somehow relevant (although I have trouble pointing at precisely how, but it convinces me regardless)

The "Grokking" paper (thanks to Quintin Pope and other commenters!)

Evidence for claim 5 — cognitive architecture components generalize:

Chimps -> humans: Humans' cognitive architecture is very general and allows us to quickly form a broad set of domain-specific abstractions that perform extremely well with little energy.

Evidence for claim 2 — takeoff is threshold-based:

Chimps -> humans: chimps didn't take over the world but humans did.

(The usual arguments for hardware overhang, ability to scale and copy what works, etc.)