trentmkelly/LessWrong-43k huggingface dataset:
[Link Post] "Foundational Challenges in Assuring Alignment and Safety of Large Language Models"

by David Scott Krueger (formerly: capybaralet) — 2024-06-06T18:55:09.151Z
Link: https://llm-safety-challenges.github.io/

We’ve recently released a comprehensive research agenda on LLM safety and alignment. This is a collaborative work with contributions from more than 35+ authors across the fields of AI Safety, machine learning, and NLP. Major credit goes to first author Usman Anwar, a 2nd year PhD student of mine who conceived and led the project and did a large portion of the research, writing, and editing. This blogpost was written only by David and Usman and may not reflect the views of other authors.
I believe this work will be an excellent reference for anyone new to the field, especially those with some background in machine learning; a paradigmatic example reader we had in mind when writing would be a first-year PhD student who is new to LLM safety/alignment. Note that the agenda is not focused on AI existential safety, although I believe that there is a considerable and growing overlap between mainstream LLM safety/alignment and topics relevant to AI existential safety.
Our work covers the following 18 topics, grouped into 3 high-level categories:

Why you should (maybe) read (part of) our agenda

The purpose of this post is to inform the Alignment Forum (AF) community of our work and encourage members of this community to consider engaging with it. A brief case for doing so:

It includes over 200 concrete research directions, which might provide useful inspiration.
We believe it provides comprehensive coverage of relevant topics at the intersection of safety and mainstream ML.
We cover a much broader range of topics than typically receive attention on AF.
AI Safety researchers – especially more junior researchers working on LLMs – are clustering around a few research agendas or problems (e.g. mechanistic interpretability, scalable oversight, jailbreaking). This seems suboptimal: given the inherent uncertainty in research, it is important to pursue diverse research agendas. We hope that this work can improve accessibility to otherwise neglected research problems, and help diversify the research agendas the community is following.
Engaging with and understanding the broader ML community – especially parts of ML community working on AI Safety relevant problems – can be helpful for increasing your work's novelty, rigor, and impact. By reading our agenda, you can better understand the ML community and discover relevant research being done i

...
AI #67: Brief Strange Trip
by Zvi — 2024-06-06T18:50:03.514Z

I had a great time at LessOnline. It was a both a working trip and also a trip to an alternate universe, a road not taken, a vision of a different life where you get up and start the day in dialogue with Agnes Callard and Aristotle and in a strange combination of relaxed and frantically go from conversation to conversation on various topics, every hour passing doors of missed opportunity, gone forever.

Most of all it meant almost no writing done for five days, so I am shall we say a bit behind again. Thus, the following topics are pending at this time, in order of my guess as to priority right now:

Leopold Aschenbrenner wrote a giant thesis, started a fund and went on Dwarkesh Patel for four and a half hours. By all accounts, it was all quite the banger, with many bold claims, strong arguments and also damning revelations.

Partly due to Leopold, partly due to an open letter, partly due to continuing small things, OpenAI fallout continues, yes we are still doing this. This should wait until after Leopold.

DeepMind’s new scaling policy. I have a first draft, still a bunch of work to do.

The OpenAI model spec. As soon as I have the cycles and anyone at OpenAI would have the cycles to read it. I have a first draft, but that was written before a lot happened, so I’d want to see if anything has changed.

The Rand report on securing AI model weights, which deserves more attention than the brief summary I am giving it here.

You’ve Got Seoul. I’ve heard some sources optimistic about what happened there but mostly we’ve heard little. It doesn’t seem that time sensitive, diplomacy flows slowly until it suddenly doesn’t.

The Problem of the Post-Apocalyptic Vault still beckons if I ever have time.

Also I haven’t processed anything non-AI in three weeks, the folders keep getting bigger, but that is a (problem? opportunity?) for future me. And there are various secondary RSS feeds I have not checked.

There was another big change this morning. California’s SB 1047 saw extensive changes. While many were helpful clarifications or fixes, one of them severely weakened the impact of the bill, as I cover on the linked post.

The reactions to the SB 1047 changes so far are included here.

Table of Contents

Introduction.

Table of Contents.

Language Models Offer Mundane Utility. Three thumbs in various directions.

Language Models Don’t Offer M

...
An evaluation of Helen Toner’s interview on the TED AI Show
by PeterH — 2024-06-06T17:39:40.800Z

Helen Toner was recently interviewed on the TED AI Show. In the first segment, she explains why the OpenAI board decided to fire Sam Altman (video, transcript).

What should we make of Helen's account?

In this post, I'll go through the interview and examine each of the claims made.

For the tl;dr, skip straight to the "reviewing the claims" section.

Claim 1. Altman withheld information, misrepresented things, and in some cases “outright lied” to the board.

After a bit of setup, the interview begins as follows:

Toner: For years, Sam had made it really difficult for the board to actually do that job by withholding information, misrepresenting things that were happening at the company, in some cases outright lying to the board.

Helen then lists five examples. Let's take them one-by-one.

1.1 The board was not informed in advance about ChatGPT

Toner: When ChatGPT came out November 2022, the board was not informed in advance about that. We learned about ChatGPT on Twitter.

Remarking on Toner's comments in a subsequent interview, Altman appears to accept this claim. He explains his action by suggesting that the ChatGPT release was not ex ante above the bar to report to the board:

Altman: When we released ChatGPT, it was at the time called a low-key research review. We did not expect what happened to happen, but we had of course talked a lot with our board about a release plan that we were moving towards. We had at this point had GPT-3.5, which ChatGPT was based on, available for about eight months. We had long since finished training GPT-4 and we were figuring out a sort of gradual release plan for that.

GPT-3.5 was indeed available via the API from March 2022.

It has previously been reported that the ChatGPT release was not expected to be a big deal. For example, The Atlantic:

The company pressed forward and launched ChatGPT on November 30. It was such a low-key event that many employees who weren’t directly involved, including those in safety functions, didn’t even realize it had happened. Some of those who were aware, according to one employee, had started a betting pool, wagering how many people might use the tool during its first week. The highest guess was 100,000 users.

Similarly, commenting on Toner's interview, Andrew Mayne remarked:

The base model for ChatGPT (GPT-3.5) had been publicly available since March 2022. ChatGPT was a much more aligned and "safe" v

...
The Impossibility of a Rational Intelligence Optimizer
by Nicolas Villarreal — 2024-06-06T16:14:03.481Z

Goodhart’s Law and Sign Function Collapse

In my last post, where I created a semiotic critique of the orthogonality thesis, I noted that optimizing for intelligence is distinct from all other possible goals, and that therefore, the only goal compatible with all levels of intelligence in principle is optimization for intelligence. However, something about this fact seemed strange to me. Earlier, I had concluded that intelligence itself is a second order sign, but I also stated that all coherent, rational optimizers have utility functions that optimize to first order signs. I realized, soon after I published, there is an extremely fruitful line of inquiry here, which is elucidated by applying this framework to Goodhart’s law. I have tentatively concluded the following line of logic, which I will explicate over the course of this post:

All coherent, rational optimizers have a utility function which optimizes to a first order sign.
Intelligence is a higher order sign.
A coherent, rational optimizer can therefore not optimize for intelligence.

If you would like to learn more about Semiotics, the science of signs and symbols, I would suggest starting with Umberto Eco’s book The Theory of Semiotics which serves as a useful introduction and also highlights how the field is based on assumptions and ideas from information theory and cybernetics. In particular, the fact that signs have meaning because they are distinct relative to other signs can be thought of an extension of the cybernetic concept of variety.

Why coherent, rational optimizers can’t optimize for higher order signs

I believe that all these claims are derivable from the semiotic principle that to have meaning, signs and symbols must be defined relationally, and be distinct, however, to save time, I will rely on a second principle, Goodhart’s Law, which states “When a measure becomes a target, it ceases to be a good measure,” or in the form of the Less Wrong glossary “when a proxy for some value becomes the target of optimization pressure, the proxy will cease to be a good proxy.”

Both the original formulation of Goodhart’s Law and the Less Wrong version gesture at something more fundamental which I believe escapes them both. If we were to place Goodhart’s Law into a semiotic framework, this might be a simple way of phrasing it: “optimization to first order signs is not optimiz...
Immunization against harmful fine-tuning attacks
by domenicrosati — 2024-06-06T15:17:42.495Z

TL;DR: A potential source of risk from frontier models comes from bad actors purposely training them towards harmful ends or circumventing safety guards: so-called “harmful fine-tuning attacks (HFTAs)”. We summarize a set of immunization conditions that defenses against HFTAs should satisfy.

This work was done as part of

AI Safety Camp (AISC)

.

The purpose of this post is to extend our discussion of

training-time domain authorization

(TTDA) with a special case of TTDA that is perhaps most relevant to AI alignment and AI safety: figuring out how to prevent training towards harmful (or illegal) ends. We further scope this down to the setting of natural language generation in current safety-gaurded large language models in order to construct a tractable empirical and conceptual research project. Note! This work is a conceptual exploration of the conditions of defence for those interested in an actual defense see our recent paper on Representation Noising.

Here we provide a high-level, more speculative summary of our paper “

Immunization against harmful fine-tuning attacks

”. The preprint contains more technical details and our formal “immunization” criteria for defense against harmful fine-tuning attacks (HFTAs). For those interested, the formal criteria in that paper is a specialized and extended version of the one presented in the conceptual introduction to

training-time domain authorization

post

. Finally, people who want a quick overview can

view this poster

we presented at

DAIS

2024.

Figure 1: Harmful fine-tuning attacks consider fine-tuning openly available safety-aligned models for harmful purposes. We propose immunization conditions for successful defenses.

What are harmful fine-tuning attacks?

Imagine the following scenario:
Meta trains a new frontier LLM Llama4. Llama4 is as capable as GPT-4 is today, provides very long context dialogue interactions that would generally pass for human, and is capable of general tool use. Due to legislative and regulatory concerns about liability, the only released versions of Llama4 are "safety guarded", using these models for harmful and illegal purposes, like for Llama3, is explicitly forbidden. While Meta is committed to open-source development they also want to be seen as a responsible actor. So they go to great lengths to prevent Llama4 from giving harmful responses by applying methods l...
SB 1047 Is Weakened
by Zvi — 2024-06-06T13:40:41.547Z

It looks like Scott Weiner’s SB 1047 is now severely weakened.

Some of the changes are good clarifications. One is a big very welcome fix.

The one I call The Big Flip is something very different.

It is mind boggling that we can have a political system where a bill can overwhelmingly pass the California senate, and then a bunch of industry lobbyists and hyperbolic false claims can make Scott Weiner feel bullied into making these changes.

I will skip the introduction, since those changes are clarifications, and get on with it.

In the interest of a clean reference point and speed, this post will not cover reactions.

The Big Flip

Then there is the big change that severely weakens SB 1047.

22602 (f)(1): Definition of covered model changed from trained with at least 10^26 flops OR a model expecting to have similar capabilities to what 10^26 flops would have gotten you in 2024 → “was trained using a quantity of computing power greater than 10^26 integer or floating-point operations, AND the cost of that quantity of computing power would exceed one hundred million dollars ($100,000,000) if calculated using average market prices of cloud compute as reasonably assessed by the developer at the time of training.”

On and after January 1, 2026, the dollar amount in this subdivision shall be adjusted annually for inflation to the nearest one hundred dollars ($100) based on the change in the annual California Consumer Price Index for All Urban Consumers published by the Department of Industrial Relations for the most recent annual period ending on December 31 preceding the adjustment.

Later: They will also publish the annual inflation adjustments.

Bolded text is exact, except I capitalized AND for clarity.

The AND, rather than an OR, makes my heart sink.

Effectively, the 10^26 requirement is dead. Long live the $100 million.

Where the law previously strengthened over time, now it weakens further.

It starts weakening this year. The cost for buying one-time use of 10^26 flops of compute seems likely to fall below $100 million this year.

Consider this from Jack Clark, where he got napkin math of $70 million a few months ago, or $110 million if you rented A100s. Jack clarified on Twitter that he expects B100s to offer a large further cost reduction.

The compute minimum to be a covered model will begin to rise. The strength of non-covered models then rises both ...
Weeping Agents
by pleiotroth — 2024-06-06T12:18:54.978Z

What is Agency? In a relatively intuitive way it’s a property of systems which have three things: Some –however rudimentary– way of modelling a slice of reality, a utility function over that slice of reality and a way of acting upon it. An agent can make better-than-chance guesses about how the behaviours in their action-pool would effect the world they inhabit and then pick the one whose modelled outcome scores best (according to expected utility, maximin or whatever risk-weighing you like) among the ones considered. They take a potential future and increase its likelihood of being instantiated. We can now switch perspectives and describe them according to the more elegant –though to many less intuitive– cybernetic definition: Agents are mechanisms by which the future influences the past. To be such a mechanism one has to be able to make better-than-noise guesses about the future, have beliefs about which ones are desirable and then act in the present to make them come to pass. Agents are löbian knots in causality. Things happening because something saw that they could happen. Proof by being provable.

Weeping Angels are an alien species from the long-running sci-fi series Doctor Who. They look like statues and have a number of other interesting qualities, but the only one that’s important for this point is that “anything which holds the image of an angel becomes an angel”.

Here are two true true statements which might be assigned the same headline as the argument I’m making but which aren’t the point:

There is a sense in which agents actively try to realign other agents towards their ends or to turn non-agentic parts of their domain into agents aligned to their values because this is a very powerful, very versatile strategy. We are doing the former constantly and are half-heartedly trying at the latter in a bumbling, suicidal sort of way.
There is also a sense in which powerful agents exert loads of selection pressure upon a system, and systems under sufficient selection pressure crystallize agents like diamonds for obvious reasons. Those agents are rarely if ever aligned to the process exerting the pressure, they are merely the sort of vaguely competent loop which survives it. This how evolution caused us for example.

Both of these are interesting and often worrying, but what we’re dealing with is worse. If you think of agents as the property of the future infl...
Calculating Natural Latents via Resampling
by johnswentworth — 2024-06-06T00:37:42.127Z

So you’ve read some of our

previous

natural latents

posts

, and you’re sold on the value proposition. But there’s some big foundational questions still unanswered. For example: how do we find these natural latents in some model, if we don’t know in advance what they are? Examples in previous posts conceptually involved picking some latent out of the ether (like e.g. the bias of a die), and then verifying the naturality of that latent.

This post is about one way to calculate natural latents, in principle, when we don’t already know what they are. The basic idea is to resample all the variables once simultaneously, conditional on the others, like a step in an MCMC algorithm. The resampled variables turn out to be a competitively optimal approximate natural latent over the original variables (as we’ll prove in the post). Toward the end, we’ll use this technique to calculate an approximate natural latent for a normal distribution, and quantify the approximations.

The proofs will use the graphical notation introduced in

Some Rules For An Algebra Of Bayes Nets

.

Some Conceptual Foundations

What Are We Even Computing?

First things first: what even is “a latent”, and what does it even mean to “calculate a natural latent”? If we had a function to “calculate natural latents”, what would its inputs be, and what would its outputs be?

The way we use the term, any conditional distribution

(λ,x↦P[Λ=λ|X=x])

defines a “latent” variable Λ over the “observables” X, given the distribution P[X]. Together P[X] and P[Λ|X] specify the full joint distribution P[Λ,X]. We typically think of the latent variable as some unobservable-to-the-agent “generator” of the observables, but a latent can be defined by any extension of the distribution over X to a distribution over Λ and X.

Natural latents are latents which (approximately) satisfy some specific conditions, namely that the distribution P[X,Λ] (approximately) factors over these Bayes nets:

Natural latent conditions over 3 variables. Left: The components of X are independent conditional on Λ. Right: Each component of X mediates between Λ and the rest of X. Epsilons indicate degree of approximation, measured by KL-divergence between the full distribution and the distribution factored according to the

...
SAEs Discover Meaningful Features in the IOI Task
by Alex Makelov — 2024-06-05T23:48:04.808Z

TLDR: recently, we wrote a paper proposing several evaluations of SAEs against "ground-truth" features computed w/ supervision for a given task (in our case, IOI [1]). However, we didn't optimize the SAEs much for performance in our tests. After putting the paper on arxiv, Alex carried out a more exhaustive search for SAEs that do well on our test for controlling (a.k.a. steering) model output with SAE features. The results show that:

SAEs trained on IOI data find interpretable features that come close to matching supervised features (computed with knowledge of the IOI circuit) for the task of editing representations to steer the model.
Gated SAEs outperform vanilla SAEs across the board for steering
SAE training metrics like sparsity and loss recovered significantly correlate with how good representation edits are. In particular, sparsity is more strongly correlated than loss recovered.
(Update, Jun 19 '24): Ran the evaluations on topk autoencoders, which outperform gated SAEs in most cases, even without tuning the sparsity parameter!

Partial Paper Recap: Towards More Objective SAE Evals

Motivation: SAE Evals Are Too Indirect

We train SAEs with the goal of finding the true features in LLM representations - but currently, "true features" is more of a vague direction than a well-defined concept in mech interp research. SAE evaluations mostly use indirect measures of performance - ones we hope correlate with the features being the "true" ones, such as the ℓ0 (sparsity) loss, the LLM loss recovered when using SAE reconstructions, and how interpretable the features are. This leaves a big gap in our understanding of the usefulness of SAEs and similar unsupervised methods; it also makes it hard to objectively compare different SAE architectures and/or training algorithms.

So, we wanted to develop more objective SAE evaluations, by benchmarking SAEs against features that we know to be meaningful through other means, even if in a narrow context. We chose the IOI task, as it's perhaps the most well-studied example of a non-trivial narrow capability in a real-world LLM (GPT2-Small). We set out to compute a "skyline" for SAE performance: an object of the same "type" as an SAE - a "sparse feature dictionary" - which is constructed and validated "by hand" using our very precise knowledge about IOI. Such an object would allow us to evaluate how c...