trentmkelly/LessWrong-43k huggingface dataset:
Backdoors as an analogy for deceptive alignment
by Jacob_Hilton — 2024-09-06T15:30:06.172Z

ARC has released a paper on Backdoor defense, learnability and obfuscation in which we study a formal notion of backdoors in ML models. Part of our motivation for this is an analogy between backdoors and deceptive alignment, the possibility that an AI system would intentionally behave well in training in order to give itself the opportunity to behave uncooperatively later. In our paper, we prove several theoretical results that shed some light on possible mitigations for deceptive alignment, albeit in a way that is limited by the strength of this analogy.

In this post, we will:

Lay out the analogy between backdoors and deceptive alignment

Discuss prior theoretical results from the perspective of this analogy

Explain our formal notion of backdoors and its strengths and weaknesses

Summarize the results in our paper and discuss their implications for deceptive alignment

Thanks to Boaz Barak, Roger Grosse, Thomas Read, John Schulman and Gabriel Wu for helpful comments.

Backdoors and deceptive alignment

A backdoor in an ML model is a modification to the model that causes it to behave

differently on certain inputs that activate a secret "trigger", while behaving similarly on ordinary inputs. There is a wide existing literature on backdoor attacks and defenses, which is primarily empirical, but also includes some theoretical results that we will mention.

Deceptive alignment is a term from the paper Risks from Learned Optimization in Advanced Machine Learning Systems (Section 4) that refers to the possibility that an AI system will internally reason about the objective that it is being trained on, and decide to perform well according to that objective unless there are clues that it has been taken out of its training environment. Such a policy could be optimal on the training distribution, and yet perform very badly on certain out-of-distribution inputs where such clues are present, which we call defection triggers.[1] The opposite of deceptive alignment is robust alignment, meaning that this performance degradation is avoided.

Since a deceptively aligned model and a robustly aligned model behave very differently on defection triggers, but very similarly on typical inputs from the training distribution, deceptive alignment can be thought of as a special kind of backdoor, under the following correspondence:

Deceptive alignment	Backdoors

Robustly aligned model	Or

...
A Cable Holder for 2 Cent
by Johannes C. Mayer — 2024-09-06T11:01:43.391Z

On Amazon, you can buy 50 cable holders for 20 cents a piece. In this video, I show how to make one for 2 cents. If I need just one I can make just one, using only readily available materials.

However, note that usually, cable holders on Amazon allow you to detach and reattach cables easily. If you frequently need to detach and reattach, this solution isn't ideal.

This approach isn't limited to cables. For example, you can attach a similar loop to the back of a kanban board such that it can be held up by a pole:
Investigating Sensitive Directions in GPT-2: An Improved Baseline and Comparative Analysis of SAEs
by Daniel Lee — 2024-09-06T02:28:41.954Z

Experiments and write-up by Daniel, with advice from Stefan. Github repo for the work can be found here.

Update (October 31, 2024): Our paper is now available on arXiv.

TL;DR

By perturbing activations along specific directions and measuring the resulting changes in the model output, we attempt to infer how much the directions matter for the model's computation. Through the sensitive directions experiments, we show that:

Heimersheim (2024)’s sensitive direction baselines (experiments 2 and 3) were flawed in that the perturbation direction involved subtracting the original activation. We propose an improved baseline direction (called cov-random mixture) which does not use the original activation.
Gurnee (2024)’s KL-div for SAE reconstruction errors no longer seems pathologically high when we use this improved baseline. However, there remains some variability across layers.
We extend the sensitive direction experiments to perturbations into SAE feature directions, and find:

SAE directions have smaller or greater impact on the model output than our cov-random mixture baseline, depending on the SAE type and L0.
Lower L0 SAE feature directions have a greater impact on the model output.
Feature directions from end-to-end SAEs do not exhibit a greater influence on the model output compared to those from traditional SAEs.

Introduction

One of the primary goals of mechanistic interpretability is to identify the abstractions that a model uses in its computation. Recently, several works have sought to understand these abstractions by observing how much the probabilities for next token prediction change when activations are perturbed along specific directions – a technique we’ll refer to as sensitive direction analysis. Heimersheim (2024) demonstrated, for example, that perturbing from one real activation towards another real activation changes the model output earlier (shorter perturbation lengths) than perturbations into random directions. This finding supports the hypothesis that perturbations along true feature directions have a greater impact on model outputs compared to other directions, motivated (Mendel 2024) by toy models of computation in superposition (Hänni 2024).

Several works have used sensitive direction analysis to analyze Sparse Autoencoders (SAEs). Perturbations along the SAE feature directions appear to alter the model output more significantly than ...
Distinguish worst-case analysis from instrumental training-gaming
by Olli Järviniemi — 2024-09-05T19:13:34.443Z

"I" refers to Olli. Thanks to Buck, Ryan Greenblatt, Fabien Roger and Zach Stein-Perlman for feedback.

The word “schemer” is used to refer to multiple things, and in particular to the following two concepts:

Schemer as a power-motivated instrumental training-gamer: following Carlsmith, a schemer is a model that optimizes for “reward on the episode” as an instrumental strategy to gain power for themselves or other AIs.

Schemers as worst-case analysis: “schemer” is used to refer to a worst-case model, e.g. a model which is egregiously misaligned, sandbags capability evaluations when it can, fakes alignment, deliberately tries to look good during red-teaming, competently hides its reasoning, and once deployed tries to subvert possible control mechanisms – including by coordinating with its copies or other AIs – and take over.[1]

I think the shared label obfuscates important differences between these two. In particular, I think a common picture of instrumental training-gamers is closer to a worst-case model than is justified. I’ll describe below what I think is a more accurate picture of instrumental training-gamers and how they differ from worst-case models.

One-sentence summary: While instrumental training-gaming is both evidentially and causally relevant for worst-case scenarios, the latter relies on several additional assumptions on the model’s capabilities and nature, and the validity of these assumptions deserves scrutiny and evaluation.

What do instrumental training-gamers look like?

I think that instrumental training-gamers are commonly pictured as monolithic entities with a fixed, context-independent goal that motivates their scheming, that this goal is egregiously misaligned with human values, that this goal causes them to training-game on approximately all inputs, and that their training behavior is very different from their deployment behavior.

I also think that our first observations of instrumental training-gaming will probably be very different from this picture, and it’s not obvious that more competent training-gaming matches this picture either. Here are a few illustrative (hypothetical) examples that I find realistic and which illustrate these differences.

1: A model training-games[2] only if you are training the model away from its current values (and it knows this)

For example, suppose the model thinks that respecting copy...
AI x Human Flourishing: Introducing the Cosmos Institute
by Brendan McCord — 2024-09-05T18:23:32.690Z
Link: https://cosmosinstitute.substack.com/p/introducing-the-cosmos-institute

Watch the Cosmos Mission Video

Earlier this year

, I explored the debates around AI and found two main views shaping the conversation today:

Existential pessimism sees AI as an apocalyptic threat, urging us to hit the pause button. This is not only impossible in an open society but also unwise. Imagine if we had paused before Darwin or Einstein. The loss to humanity would have been immense.

On the other hand, accelerationism embraces technological progress with unbridled optimism. Yet for some proponents, the human good is pushed to the margins–tech becomes an end in itself, with humans merely a stepping stone toward a post-human future.

I understand the appeal of saving humanity from extinction and the appeal of building god. But both perspectives, while raising important concerns, overlook the conditions and ingredients of human well-being. What's needed is a new approach—one that embraces technological progress while keeping human flourishing as its North Star.

The Cosmos Institute

is a 501c3 nonprofit dedicated to promoting human flourishing in the age of AI. This post will outline our initial research, fellowships, grants, and education initiatives.

Through these programs, we aim to cultivate a new generation of technologists and entrepreneurs equipped with deep philosophical thinking to navigate the uncharted territory of our AI age.

Our vision is rooted in three pillars–reason, decentralization, and human autonomy–drawing from the insights of thinkers like John Stuart Mill, Alexis de Tocqueville, and Aristotle.

Three Pillars of AI x Human Flourishing

1. Reason

Our first pillar is reason. Inspired by Mill’s

On Liberty

, we champion broad access to diverse opinions as the foundation for truth-seeking and seek to enrich public discourse on questions both timely and timeless.

AI has the potential to elevate the systems that support inquiry and knowledge–humanity’s epistemic infrastructure–beyond anything Mill could have imagined. By facilitating richer collaboration among diverse minds (and artificial agents), AI can spark the collective intelligence and mutual adjustments needed to solve complex challenges and uncover unexpected insights.

However, there is a darker path: AI could become an “autocomplete for life,” tempting us to accept easy answers instead of actively engaging in the search for truth. If we let AI narrow our range of inquiry, we risk...
What is SB 1047 *for*?
by Raemon — 2024-09-05T17:39:39.871Z

Emmett Shear asked on twitter:

I think SB 1047 has gotten much better from where it started. It no longer appears actively bad. But can someone who is pro-SB 1047 explain the specific chain of causal events where they think this bill becoming law results in an actual safer world? What’s the theory?

And I realized that AFAICT no one has concisely written up what the actual story for SB 1047 is supposed to be.

This is my current understanding. Other folk here may have more detailed thoughts or disagreements.

The bill isn't sufficient on it's own, but it's not regulation for regulation's sake because it's specifically a piece of the regulatory machine I'd ultimately want built.

Right now, it mostly solidifies the safety processes that existing orgs have voluntarily committed to. But, we are pretty lucky that they voluntarily committed to them, and we don't have any guarantee that they'll stick with them in the future.

For the bill to succeed, we do need to invent good, third party auditing processes that are not just a bureaucratic sham. This is an important, big scientific problem that isn't solved yet, and it's going to be a big political problem to make sure that the ones that become consensus are good instead of regulatory-captured. But, figuring that out is one of the major goals of the AI safety community right now.

The "Evals Plan" as I understand it comes in two phase:

1. Dangerous Capability Evals. We invent evals that demonstrate a model is capable of dangerous things (including manipulation/scheming/deception-y things, and "invent bioweapons" type things)

As I understand it, this is pretty tractable, although labor intensive and "difficult" in a normal, boring way.

2. Robust Safety Evals. We invent evals that demonstrate that a model capable of scheming, is nonetheless safe – either because we've proven what sort of actions it will choose to take (AI Alignment), or, we've proven that we can control it even if it is scheming (AI control). AI control is probably easier at first, although limited.

As I understand it, this is very hard, and while we're working on it it requires new breakthroughs.

The goal with SB 1047 as I understand is roughly:

First: Capability Evals trigger

By the time it triggers for the first time, we have a set of evals that are good enough to confirm "okay, this model isn't actually capable of being dangerous" (and probably the AI developers co...
instruction tuning and autoregressive distribution shift
by nostalgebraist — 2024-09-05T16:53:41.497Z

[Note: this began life as a "Quick Takes" comment, but it got pretty long, so I figured I might as well convert it to a regular post.]

In LM training, every token provides new information about "the world beyond the LM" that can be used/"learned" in-context to better predict future tokens in the same window.

But when text is produced by autoregressive sampling from the same LM, it is not informative in the same way, at least not to the same extent[1]. Thus, sampling inevitably produces a distribution shift.

I think this is one of the reasons why it's (apparently) difficult to get instruction-tuned / HH-tuned models to report their uncertainty and level of competence accurately, rather than being overconfident.

(I doubt this is a novel point, I just haven't seen it spelled out explicitly before, and felt like doing so.)

Imagine that you read the following (as the beginning of some longer document), and you trust that the author is accurately describing themselves:

I'm a Princeton physics professor, with a track record of highly cited and impactful research in the emerging field of Ultra-High-Density Quasiclassical Exotic Pseudoplasmas (UHD-QC-EPPs).

The state of the art in numerical simulation of UHD-QC-EPPs is the so-called Neural Pseudospectral Method (NPsM).

I made up all those buzzwords, but imagine that this is a real field, albeit one you know virtually nothing about. So you've never heard of "NPsM" or any other competing method.

Nonetheless, you can confidently draw some conclusions just from reading this snippet and trusting the author's self-description:

Later in this document, the author will continue to write as though they believe that NPsM is "the gold standard" in this area.

They're not going to suddenly turn around and say something like "wait, whoops, I just checked Wikipedia and it turns out NPsM has been superseded by [some other thing]." They're a leading expert in the field! If that had happened, they'd already know by the time they sat down to write any of this.

Also, apart from this particular writer's beliefs, it's probably actually true that NPsM is the gold standard in this area.

Again, they're an expert in the field -- and this is the sort of claim that would be fairly easy to check even if you're not an expert yourself, just by Googling around and skimming recent papers. It's also not the sort of claim where there's any obvious

...
Conflating value alignment and intent alignment is causing confusion
by Seth Herd — 2024-09-05T16:39:51.967Z

Epistemic status: I think something like this confusion is happening often. I'm not saying these are the only differences in what people mean by "AGI alignment".

Summary:

Value alignment is better but probably harder to achieve than personal intent alignment to the short-term wants of some person(s). Different groups and people tend to primarily address one of these alignment targets when they discuss alignment. Confusion abounds.

One important confusion stems from an assumption that the type of AI defines the alignment target: strong goal-directed AGI must be value aligned or misaligned, while personal intent alignment is only viable for relatively weak AI. I think this assumption is important but false.

While value alignment is categorically better, intent alignment seems easier, safer, and more appealing in the short term, so AGI project leaders are likely to try it.[1]

Overview

Clarifying what people mean by alignment should dispel some illusory disagreement, and clarify alignment theory and predictions of AGI outcomes.

Caption: Venn diagram of three types of alignment targets. Value alignment and Personal intent alignment are both subsets of Evan Hubinger's definition of intent alignment: AGI aligned with human intent in the broadest sense. Prosaic alignment work usually seems to be addressing a target somewhere in the neighborhood of personal intent alignment (following instructions or doing what this person wants now), while agent foundations and other conceptual alignment work usually seems to be addressing value alignment. Those two clusters have different strengths and weaknesses as alignment targets, so lumping them together produces confusion.

People mean different things when they say alignment. Some are mostly thinking about value alignment (VA): creating sovereign AGI that has values close enough to humans' for our liking. Others are talking about making AGI that is corrigible (in the Christiano or Harms sense)[2] or follows instructions from its designated principal human(s). I'm going to use the term personal intent alignment (PIA) until someone has a better term for that type of alignment target. Different arguments and intuitions apply to these two alignment goals, so talking about them without differentiation is creating illusory disagreements.

Value alignment is better almost by definition, but personal intent alignme...
A bet for Samo Burja
by Nathan Helm-Burger — 2024-09-05T16:01:35.440Z

I'm listening to Samo Burja talk on the Cognitive Revolution podcast with Nathan Labenz.
Samo said that he would bet that AGI is coming perhaps in the next 20-50 years, but not in the next 5.

I will take that bet. I can't afford to make an impressively large bet because my counterfactual income is already tied up in a bet against the universe. I quit my well-paying industry job as a machine learning engineer / data scientist three years ago to focus on AI safety/alignment research. To make the bet interesting, I will therefore offer 10:1 odds.
I bet $1000 USD against your $100 USD that AGI will be invented in the next 5 years. There are a lot of possible resolution criteria, but as a reasonable shelling point I'll accept this metaculus market: https://www.metaculus.com/questions/5121/date-of-artificial-general-intelligence/

I'll describe my rationale here, in case I change your mind and make you not want the bet. ;-)

I agree with your premise that AGI will require fundamental scientific advances beyond currently deployed tech like transformer LLMs.

I agree that scientific progress is hard, usually slow and erratic, fundamentally different from engineering or bringing a product to market.

I agree with your estimate that the current hype around chat LLMs, and focus on bringing better versions to market, is slowing fundamental scientific progress by distracting top AI scientists from pursuit of theoretical advances.

My cruxes are these:

I believe LLMs will scale to close enough to AGI to become central parts of very useful tools. I believe that these tools will enable the human AI scientists to make rapid theoretical progress. I expect that these AI research systems (I won't say researchers, since in this scenario they are still sub-AGI) will enable massively parallel testing of hypotheses which are derived as permutations of a handful of initial ideas given by the human scientists. I also foresee these AI research systems mining existing scientific literature for hypotheses to test. I believe the result of this technology will be rapid discovery of algorithms that can actually scale to true AGI.

I have been following advances in neuroscience relevant to brain-inspired AI for over 20 years now. I believe that the neuroscience community has made some key breakthroughs in the past five years which have yet to be effectively exported to machine learning and tested a

...
Universal basic income isn’t always AGI-proof
by Kevin Kohler — 2024-09-05T15:39:18.389Z
Link: https://machinocene.substack.com/p/ubi-isnt-designed-for-technological

A universal basic income (UBI) is often presented as a public insurance against large-scale and potentially permanent

technological unemployment

. Many Silicon Valley leaders that believe in the transformative economic potential of

artificial general intelligence

have also voiced their support for UBI (e.g.

Sam Altman

,

Elon Musk

).

UBI can be a part of the policy tools to address long-term technological unemployment. However, it is worth highlighting that a UBI to address long-term technological unemployment is more expensive than current UBI proposals and that it is not a sustainable solution to finance an insurance against widespread loss of labor income by taxing labor income:

A post-labor UBI is expensive because it would not merely supplement labor income, but fully replace it. Additionally, the more affordable alternative of a guaranteed minimum income would approach the cost of a regular UBI in a post-labor scenario.
To explore the challenge of financing a UBI in a scenario of large-scale technological unemployment we will examine the case study of Switzerland. The Swiss were the first worldwide to vote on a nationwide and fairly generous UBI in 2016.

In short, distributing money through a UBI can be a solution to a lack of labor income. However, the hard part is designing income streams that grow in lockstep with the parts of the economy that grow in a scenario of high automation and accordingly can finance a rising demand for UBI or other forms of social security.

1. UBI for technological unemployment is expensive

1.1 A guaranteed minimum income is cheaper than UBI, but would approach the cost of UBI in a post-labor economy

In the public discourse, the terms UBI and guaranteed minimum income are often used interchangeably. However, they denote different concepts and many famous “UBI proposals” and “UBI trials” are actually guaranteed minimum income proposals and trials. The main reason for this is that guaranteed minimum income is much cheaper to implement than a UBI. However, in a scenario of large-scale unemployment the costs of guaranteed minimum income would rise rapidly and approach the cost of a UBI.

Universal basic income

A UBI is commonly defined as having the following characteristics:

Periodic: a recurrent payment (e.g., every month)
Cash payment: paid in cash, allowing the recipients to convert their benefits into whatever they may like.
Universal: pai

...