trentmkelly/LessWrong-43k huggingface dataset:
Anthropic announces interpretability advances. How much does this advance alignment?
by Seth Herd — 2024-05-21T22:30:52.638Z
Link: https://www.anthropic.com/news/mapping-mind-language-model

Anthropic just published a pretty impressive set of results in interpretability. This raises for me, some questions and a concern: Interpretability helps, but it isn't alignment, right? It seems to me as though the vast bulk of alignment funding is now going to interpretability. Who is thinking about how to leverage interpretability into alignment?

It intuitively seems as though we are better off the more we understand the cognition of foundation models. I think this is true, but there are sharp limits: it will be impossible to track the full cognition of an AGI, and simply knowing what it's thinking about will be inadequate to know whether it's making plans you like. One can think about bioweapons, for instance, to either produce them or prevent producing them. More on these at the end; first a brief summary of their results.

In this work, they located interpretable features in Claude 3 Sonnet using sparse autoencoders, and manipulating model behavior using those features as steering vectors. They find features for subtle concepts; they highlight features for:

The Golden Gate Bridge 34M/31164353: Descriptions of or references to the Golden Gate Bridge.
Brain sciences 34M/9493533: discussions of neuroscience and related academic research on brains or minds.
Monuments and popular tourist attractions 1M/887839.
Transit infrastructure 1M/3. [links to examples]

...

We also find more abstract features—responding to things like bugs in computer code, discussions of gender bias in professions, and conversations about keeping secrets.

...we found features corresponding to:

Capabilities with misuse potential (code backdoors, developing biological weapons)
Different forms of bias (gender discrimination, racist claims about crime)
Potentially problematic AI behaviors (power-seeking, manipulation, secrecy)

Presumably, the existence of such features will surprise nobody who's used and thought about large language models. It is difficult to imagine how they would do what they do without using representations of subtle and abstract concepts.

They used the dictionary learning approach, and found distributed representations of features:

Our general approach to understanding Claude 3 Sonnet is based on the linear representation hypothesis and the superposition hypothesis

from the publication, Scaling Monosemanticity: Extracting Interpretable Features from ...
EIS XIII: Reflections on Anthropic’s SAE Research Circa May 2024
by scasper — 2024-05-21T20:15:36.502Z

Part 13 of 12 in the

Engineer’s Interpretability Sequence

.

TL;DR

On May 5, 2024,

I made a set of 10 predictions

about what the next sparse autoencoder (SAE) paper from Anthropic would and wouldn’t do.

Today’s new SAE paper from Anthropic

was full of brilliant experiments and interesting insights, but it underperformed my expectations. I am beginning to be concerned that Anthropic’s recent approach to interpretability research might be better explained by safety washing than practical safety work.

Think of this post as a curt editorial instead of a technical piece. I hope to revisit my predictions and this post in light of future updates.

Reflecting on predictions

See my original post

for 10 specific predictions about what

today’s paper

would and wouldn’t accomplish. I think that Anthropic obviously did 1 and 2 and obviously did not do 4, 5, 7, 8, 9, and 10. Meanwhile, I think that their experiments to identify

specific

and

safety-relevant

features should count for 3 (proofs of concept for a useful type of task) but definitely do not count for 6 (*competitively* finding and removing a harmful behavior that was represented in the training data).

Thus, my assessment is that Anthropic did 1-3 but not 4-10.

I have been wrong with mech interp predictions in the past

, but this time, everything I predicted with >50% probability happened, and everything I predicted with <50% probability did not happen.

The predictions were accurate in one sense. But overall, the paper underperformed expectations. If you scored the paper relative to my predictions by giving it (1-p) points when it did something that I predicted it would do with probability p and -p points when it did not, the paper would score -0.74.

A review + thoughts

I think that Anthropic’s new SAE work has continued to be like lots of prior high-profile work on mechanistic interpretability – it has focused on presenting illustrative examples, streetlight demos, and cherry-picked proofs of concept. This is useful for science, but it does not yet show that SAEs are helpful and competitive for diagnostic and debugging tasks that could improve AI safety.

I feel increasingly worried about how Anthropic motivates and sells its interpretability research in the name of safety. Today’s paper makes some highly-generalized Motte and Bailey claims that ov...
Mitigating extreme AI risks amid rapid progress [Linkpost]
by Orpheus16 — 2024-05-21T19:59:21.343Z

In a new Science paper, the authors provide concise summaries of AI risks and offer recommendations for governments.

I think the piece is quite well-written. It concisely explains a lot of relevant arguments, including arguments about misalignment and AI takeover. I suspect this is one of the best standalone pieces to help people understand AI risks and some (IMO reasonable) governance interventions.

The piece also has a very respectable cast of authors, including Bengio and Hinton. (Not to say that this fact should affect your assessment of whether its claims are true. Mentioning it because it will affect how some audiences– EG policymakers– interpret the piece.)

Some relevant quotes below:

Explanation of AGI & importance of preparing for AGI risks

There is no fundamental reason for AI progress to slow or halt at human-level abilities. Indeed, AI has already surpassed human abilities in narrow domains such as playing strategy games and predicting how proteins fold (see SM). Compared with humans, AI systems can act faster, absorb more knowledge, and communicate at a higher bandwidth. Additionally, they can be scaled to use immense computational resources and can be replicated by the millions.

We do not know for certain how the future of AI will unfold. However, we must take seriously the possibility that highly powerful generalist AI systems that outperform human abilities across many critical domains will be developed within this decade or the next. What happens then?

Explanation of misalignment & AI takeover risks

Without R&D breakthroughs (see next section), even well-meaning developers may inadvertently create AI systems that pursue unintended goals: The reward signal used to train AI systems usually fails to fully capture the intended objectives, leading to AI systems that pursue the literal specification rather than the intended outcome. Additionally, the training data never captures all relevant situations, leading to AI systems that pursue undesirable goals in new situations encountered after training.

Once autonomous AI systems pursue undesirable goals, we may be unable to keep them in check. Control of software is an old and unsolved problem: Computer worms have long been able to proliferate and avoid detection (see SM). However, AI is making progress in critical domains such as hacking, social manipulation, and strategic planning (see SM)

...
rough draft on what happens in the brain when you have an insight
by Emrik — 2024-05-21T18:02:47.060Z

Epistemic status: It is better to be wrong than to have no model at all. I recommend the footnotes.[1]

🍵

On my current models of theoretical[2] insight-making, it looks something like this:

A gradual build-up and propagation of salience wrt some tiny discrepancy between highly confident specific beliefs

This maybe corresponds to simultaneously-salient neural ensembles whose oscillations are inharmonic[3]
Or in the frame of predictive processing: unresolved prediction-error between successive layers

Immediately followed by a resolution of that discrepancy if the insight is successfwl

This maybe corresponds to the brain having found a combination of salient ensembles—including the originally inharmonic ensembles—whose oscillations are adequately harmonic
Super-speculative but: If the "question phase" in step 1 was salient enough, and the compression in step 2 great enough, this causes an insight-frisson[4] and a wave of pleasant sensations across your scalp, spine, and associated sensory areas

This maps to a fragile/chaotic high-energy "question phase" during which the violation of expectation is maximized (in order to adequately propagate the implications of the original discrepancy), followed by a compressive low-energy "solution phase" where correctness of expectation is maximized again.

In order to make this work, there are several homeostatic mechanisms which make the brain-state hug the border between phase-transitions as tightly as possible.[5] A corollary of this is that the brain maximizes dynamic correlation length between neurons,[6] which is when they have the greatest ability to influence each other across long distances (aka "communicate"). This is called the "critical brain hypothesis", and it suggests that good thinking is necessarily chaotic in some sense.

Another point is that insight-making is anti-inductive.[7] Theoretical reasoning is a frontier that's continuously being exploited based on the brain's native Value-of-Information-estimator, which means that the branches with the highest naively-calculated-VoI are also less likely to have any low-hanging fruit left. What this implies is that novel insights are likely to be very narrow targets—which means they could be really hard to hold on to for the brief moment between initial hunch and build-up of salience. Concisely: epistemic frontiers are anti-inductive.

^

?

...
On Dwarkesh’s Podcast with OpenAI’s John Schulman
by Zvi — 2024-05-21T17:30:04.332Z

Dwarkesh Patel recorded a Podcast with John Schulman, cofounder of OpenAI and at the time their head of current model post-training. Transcript here. John’s job at the time was to make the current AIs do what OpenAI wanted them to do. That is an important task, but one that employs techniques that their at-the-time head of alignment, Jan Leike, made clear we should not expect to work on future more capable systems. I strongly agree with Leike on that.

Then Sutskever left and Leike resigned, and John Schulman was made the new head of alignment, now charged with what superalignment efforts remain at OpenAI to give us the ability to control future AGIs and ASIs.

This gives us a golden opportunity to assess where his head is at, without him knowing he was about to step into that role.

There is no question that John Schulman is a heavyweight. He executes and ships. He knows machine learning. He knows post-training and mundane alignment.

The question is, does he think well about this new job that has been thrust upon him?

The Big Take

Overall I was pleasantly surprised and impressed.

In particular, I was impressed by John’s willingness to accept uncertainty and not knowing things.

He does not have a good plan for alignment, but he is far less confused about this fact than most others in similar positions.

He does not know how to best navigate the situation if AGI suddenly happened ahead of schedule in multiple places within a short time frame, but I have not ever heard a good plan for that scenario, and his speculations seem about as directionally correct and helpful as one could hope for there.

Are there answers that are cause for concern, and places where he needs to fix misconceptions as quickly as possible? Oh, hell yes.

His reactions to potential scenarios involved radically insufficient amounts of slowing down, halting and catching fire, freaking out and general understanding of the stakes.

Some of that I think was about John and others at OpenAI using a very weak definition of AGI (perhaps partly because of the Microsoft deal?) but also partly he does not seem to appreciate what it would mean to have an AI doing his job, which he says he expects in a median of five years.

His answer on instrumental convergence is worrisome, as others have pointed out. He dismisses concerns that an AI given a bounded task would start doing things outside the intuitive tas...
Is deleting capabilities still a relevant research question?
by tailcalled — 2024-05-21T13:24:44.946Z

I've had it suggested that a good criterion for whether interpretability is on the right track is if we can do surgical "deletions" of model capabilities, e.g. removing its ability to build bombs and such.

Obviously in one sense this is fairly trivial since you can just use simple gradient descent to make the models refuse, but the issue with this is that given the weights, people can easily undo these refusals (and also adversarial prompting can often bypass it).

I know there's been some back and forth on methods for full deletion, and I'm wondering if it's considered a solved problem or not.
New voluntary commitments (AI Seoul Summit)
by Zach Stein-Perlman — 2024-05-21T11:00:41.794Z
Link: https://www.gov.uk/government/publications/frontier-ai-safety-commitments-ai-seoul-summit-2024/frontier-ai-safety-commitments-ai-seoul-summit-2024

Basically the companies commit to make responsible scaling policies.

Part of me says this is amazing, the best possible commitment short of all committing to a specific RSP. It's certainly more real than almost all other possible kinds of commitments. But as far as I can tell, people pay almost no attention to what RSP-ish documents (Anthropic, OpenAI, Google) actually say and whether the companies are following them.[1] The discourse is more like "Anthropic, OpenAI, and Google have safety plans and other companies don't." Hopefully that will change.

Maybe "These commitments represent a crucial and historic step forward for international AI governance." It does seem nice from an international-governance perspective that Mistral AI, TII (the Falcon people), and a Chinese company joined.

Full document:

The UK and Republic of Korea governments announced that the following organisations have agreed to the Frontier AI Safety Commitments:

Amazon
Anthropic
Cohere
Google
G42
IBM
Inflection AI
Meta
Microsoft
Mistral AI
Naver
OpenAI
Samsung Electronics
Technology Innovation Institute
xAI
Zhipu.ai

The above organisations, in furtherance of safe and trustworthy AI, undertake to develop and deploy their frontier AI models and systems[2] responsibly, in accordance with the following voluntary commitments, and to demonstrate how they have achieved this by publishing a safety framework focused on severe risks by the upcoming AI Summit in France.

Given the evolving state of the science in this area, the undersigned organisations’ approaches (as detailed in paragraphs I-VIII) to meeting Outcomes 1, 2 and 3 may evolve in the future. In such instances, organisations will provide transparency on this, including their reasons, through public updates.

The above organisations also affirm their commitment to implement current best practices related to frontier AI safety, including: internal and external red-teaming of frontier AI models and systems for severe and novel threats; to work toward information sharing; to invest in cybersecurity and insider threat safeguards to protect proprietary and unreleased model weights; to incentivize third-party discovery and reporting of issues and vulnerabilities; to develop and deploy mechanisms that enable users to understand if audio or visual content is AI-generated; to publicly report model or system capabilities, limitations, and domains of appro

...
My Dating Heuristic
by Declan Molony — 2024-05-21T05:28:40.197Z

I don’t have to practice being afraid of a lion charging at me—my instincts tell me to run. But when I started dating, my instincts weren’t that reliable when attempting to attract a partner. They needed to be recalibrated. Author Matthew Hussey talks about retraining your (likely faulty) dating instincts in his book Love Life:

One of the love life myths is that somehow love is a special realm where we can be guided by instinct. But this assumes that in childhood we all developed great instincts for every situation.

[As an example], in the early stages of attraction…there’s a temptation to just surrender to the feeling, clear our schedule, and see if they’re game to fly to Paris together. [This is all for a person who] wasn’t even on our radar a month ago. We give in to our romantic instincts and rocket into a realm of fantasy romance.

This instinctive, emotional reaction is similar to what author Daniel Kahneman calls System 1 thinking in his book

Thinking, Fast and Slow

. System 1 thinking is that quick, off-the-cuff response we have for certain situations. Contrast that with System 2 thinking which is slower, more deliberative, and more logical.

It’s System 1 thinking that used to get me in trouble when it came to my dating life. For example, when partners would pull away, my instincts told me to push more to try to keep them around. Or, when partners I’m enamored with did something rude in front of my friends, I would immediately start rationalizing their behavior as “just one of their fun quirks”. What changed my dating life for the better is when I started to integrate more System 2 deliberate thinking. What changed is when I developed the following heuristic[1]:

My Dating Heuristic → when in a dating context, I ask myself: what would an emotionally healthy person do? Then I do that.

Cosplaying as a mature adult

When I invented my dating heuristic, everything that used to suck about dating started to suck less. Why? Because in asking what an emotionally healthy person would do in any given situation, I gained the ability to view things as if I’m an impartial observer who is not invested in the relationship.

My dating instincts, which were poorly tuned to reality, were recalibrated over time when I started cosplaying as a mature adult.

Mistakes I’ve made that were improved by using my dating heuristic

My instincts once told me that

...
Scorable Functions: A Format for Algorithmic Forecasting
by ozziegooen — 2024-05-21T04:14:11.749Z

You can think of a scorable function as a "box" that can produce large clusters of complex forecasts. Credit to Dall-E 3.

Introduction

Imagine if a forecasting platform had estimates for things like:

"For every year until 2100, what will be the probability of a global catastrophic biological event, given different levels of biosecurity investment and technological advancement?"
"What will be the impact of various AI governance policies on the likelihood of developing safe and beneficial artificial general intelligence, and how will this affect key indicators of global well-being over the next century?"
"How valuable is every single project funded by Open Philanthropy, according to a person with any set of demographic information, if they would spend 1000 hours reflecting on it?"

These complex, multidimensional questions are useful for informing decision-making and resource allocation around effective altruism and existential risk mitigation. However, traditional judgemental forecasting methods often struggle to capture the nuance and conditionality required to address such questions effectively.

This is where "scorable functions" come in - a forecasting format that allows forecasters to directly submit entire predictive models rather than just point estimates or simple probability distributions. Scorable functions allow encoding a vast range of relationships and dependencies, from basic linear trends to intricate nonlinear dynamics. Forecasters can precisely specify interactions between variables, the evolution of probabilities over time, and how different scenarios could unfold.

At their core, scorable functions are executable models that output probabilistic predictions and can be directly scored via function calls. They encapsulate the forecasting logic, whether it stems from human judgment, data-driven insights, or a hybrid of the two. Scorable functions can span from concise one-liners to elaborate constructs like neural networks.

Over the past few years, we at QURI have been investigating how to effectively harness these methods. We believe scorable functions could be a key piece of the forecasting puzzle going forward.

From Forecast Bots to Scorable Functions

Many people are familiar with the idea of using "bots" to automate forecasts on platforms like Metaculus. Let's consider a simple example to see how scorable functions can extend this concept.

Suppose there'...
The Problem With the Word ‘Alignment’
by peligrietzer — 2024-05-21T03:48:26.983Z

This post was written by Peli Grietzer, inspired by internal writings by TJ (tushant jha), for AOI[1]. The original post, published on Feb 5, 2024, can be found here: https://ai.objectives.institute/blog/the-problem-with-alignment.

The purpose of our work at the AI Objectives Institute (AOI) is to direct the impact of AI towards human autonomy and human flourishing. In the course of articulating our mission and positioning ourselves -- a young organization -- in the landscape of AI risk orgs, we’ve come to notice what we think are serious conceptual problems with the prevalent vocabulary of ‘AI alignment.’ This essay will discuss some of the major ways in which we think the concept of ‘alignment’ creates bias and confusion, as well as our own search for clarifying concepts.

At AOI, we try to think about AI within the context of humanity’s contemporary institutional structures: How do contemporary market and non-market (eg. bureaucratic, political, ideological, reputational) forces shape AI R&D and deployment, and how will the rise of AI-empowered corporate, state, and NGO actors reshape those forces? We increasingly feel that ‘alignment’ talk tends to obscure or distort these questions.

The trouble, we believe, is the idea that there is a single so-called Alignment Problem. Talk about an ‘Alignment Problem’ tends to conflate a family of related but distinct technical and social problems, including:

P1: Avoiding takeover from emergent optimization in AI agents

P2: Ensuring that AI’s information processing (and/or reasoning) is intelligible to us

P3: Ensuring AIs are good at solving problems as specified (by user or designer)

P4: Ensuring AI systems enhance, and don’t erode, human agency

P5: Ensuring that advanced AI agents learn a human utility function

P6: Ensuring that AI systems lead to desirable systemic and long term outcomes

Each of P1-P6 is known as ‘the Alignment Problem’ (or as the core research problem in ‘Alignment Research’) to at least some people in the greater AI Risk sphere, in at least some contexts. And yet these problems are clearly not simply interchangeable: placing any one of P1-P6 at the center of AI safety implies a complicated background theory about their relationship, their relative difficulty, and their relative significance.

We believe that when different individuals and organizations speak of the ‘Alignment Probl...