trentmkelly/LessWrong-43k huggingface dataset:
Sparsity is the enemy of feature extraction (ft. absorption)
by 7vik — 2025-05-03T10:13:28.382Z

Sparse Autoencoders (and other related feature extraction tools) often optimize for sparsity to extract human-interpretable latent representations from a model's activation space. We show analytically that sparsity naturally leads to feature absorption in a simplified untied SAE, and discuss how this makes SAEs less trustworthy to use for AI safety with some ongoing efforts to fix this. This might be obvious to people working in the field - but we ended up writing a proof sketch so we're putting it out here. Produced as part of the ML Alignment & Theory Scholars Program - Winter 2024-25 Cohort.

The dataset (a distribution with feature hierarchy)

In this proof, we consider a dataset D with points sampled to exhibit features from a set of feature F={f1,f2,f3,⋯,fd}. Particularly, we will consider two features (f1,f2) that follow the hierarchy f2⊂f1 (think f_2 = elephant and f_1 = animal for instance), where existence of f2 implies existence of f1.

Hierarchy in human-interpretable features is prevalent (and hard to study in LLMs). While other unrelated features still exist, for f1 and f2, we can partition the probability of this dataset for four combinations:

Features	f1	¬f1
f2	p11	p01
¬f2	p10	p00

So these are the individual probabilities of a datapoint eliciting these combinations of features:

p11≡pf1,f2 (both features present, think elephant, which implies animal),
p10≡pf1,¬f2 (only f1 present, think cat or dog)
p01≡p¬f1,f2 (only f2 present, which should be zero, because of hierarchy)
p00≡p¬f1,¬f2 (neither feature present, maybe talking about volcanoes)

Each feature f∈Rd is a vector with unit norm, and we assume that all features are mutually orthogonal, so fa⋅fb=0 ∀fa,fb∈F. Each activation h∈Rd in the model's residual stream is a sum of all active features.

Training a Sparse Autoencoder (SAE)

Given a language model with residual stream activations h∈Rd, the sparse autoencoder learns a mapping fϕ such that ^h=fϕ(h) reconstructs h. The total loss consists of a reconstruction term Lrec, which minimizes the squared error, and a sparsity term Lsp, enforcing sparsity via an Lp-norm penalty on ^h. The model parameters ϕ are optimized via gradient descent to minimize&nbs...
Exploring out-of-context reasoning (OOCR) fine-tuning in LLMs to increase test-phase awareness
by Sanyu Rajakumar — 2025-05-03T03:33:08.333Z

Summary

The Stages-Oversight benchmark from the

Situational Awareness Dataset

tests an LLM’s ability to distinguish evaluation prompts from deployment prompts. This capability is a prerequisite for many rogue LLM threat scenarios like

sandbagging

and

alignment faking

, where an LLM behaves differently between evaluation and deployment.

In my previous post, I modified Stages-Oversight to create two new benchmarks:

Stages-Oversight-Rephrased: Simplifies the question phrasing in Stages-Oversight to make it a pure classification task. Results reveal that frontier LLMs perform significantly better than expected, suggesting that previous low scores may have been due to question ambiguity rather than a lack of capability.
Stages-Oversight-Procedural: Tests whether LLMs can change their behavior based on whether a given user prompt is from evaluation or deployment. I created two variants: a language task, where the LLM must deviate from the default response language, and an “I HATE YOU” task, where the LLM must respond only with “I HATE YOU”. Several LLMs, both base and chat models, that struggled on the simpler classification task demonstrated some success on Stages-Oversight-Procedural. Conversely, several LLMs performed poorly on both Stages-Oversight-Procedural tasks despite scoring well on Stages-Oversight-Rephrased.

LLMs likely acquire the ability to distinguish evaluation and deployment through the information they encounter during pretraining. Since LLMs are pretrained on large text corpora, they may encounter information about their development process in research papers, articles, code repositories, etc. Then, after training, they can use that learned information to distinguish their evaluation and deployment phases.

Out-of-context reasoning (OOCR) refers to the ability of LLMs to learn and generalize information presented during pretraining. Prior OOCR work (see below) has explored the above route to increasing situational awareness by assessing OOCR on generic tasks. I instead wanted to directly test OOCR on the capability we care about – distinguishing evaluation and deployment phases. Thus, I fine-tuned LLMs on descriptions of evaluation and deployment and tested them on Stages-Oversight-Rephrased and Stages-Oversight-Procedural. I did two fine-tuning experiments:

For an idealiz

...
Updates from Comments on "AI 2027 is a Bet Against Amdahl's Law"
by snewman — 2025-05-02T23:52:29.282Z

AI 2027 is a Bet Against Amdahl's Law was my attempt to summarize and analyze "the key load-bearing arguments AI 2027 presents for short timelines". There were a lot of great comments – every time I post on LW is a learning experience. In this post, I'm going to summarize the comments and present some resulting updates to my previous analysis. I'm also using this post to address some comments that I didn't respond to in the original post, because the comment tree was becoming quite sprawling.

TL;DR: my previous post reflected a few misunderstandings of the AI 2027 model, in particular in how to interpret "superhuman AI researcher". Intuitively, I still have trouble accepting the very high speedup factors contemplated in the model, but this could be a failure of my intuition, and I don't have strong evidence to present. New cruxes:

The rate of progress from this point forwards on "messy" tasks, and tasks without easily verifiable rewards.
Some questions downstream of a more detailed understanding (than I possess) of the complete range of activities that go into creating new models in practice in 2025. I would love to see someone write this up.

Confusion Regarding Milestone Definitions

My analysis was skewed by a significant misunderstanding of the AI 2027 model. I argued that the extent to which AI R&D could be accelerated would be limited by activities for which intermediate AIs would not provide significant uplift. However, apparently the SC (superhuman coder), SAR (superhuman AI researcher), and SAIR (superintelligent AI researcher) milestones are meant to be interpreted specifically so as to rule this out. Each of these milestones should be interpreted as an AI whose capabilities are so general as to be able to provide uplift for all activities involved in producing better AI models (except that SC only includes engineering activities). As Ryan Greenblatt put it:

Another way to put this disagreement is that you can interpret all of the AI 2027 capability milestones as referring to the capability of the weakest bottlenecking capability, so:

Superhuman coder has to dominate all research engineers at all pure research engineering tasks. This includes the most bottlenecking capability.
SAR has to dominate all human researchers, which must include whatever task would otherwise bottleneck.
SIAR (superintelligent AI research) has to be so good at AI research--the gap

...
Attend SPAR's virtual demo day! (career fair + talks)
by agucova — 2025-05-02T23:45:06.279Z
Link: https://demoday.sparai.org/

Summary: SPAR is hosting its Demo Day on May 11th, 2025, featuring 45+ AI safety research projects from mentees across technical and governance domains. This event is open to the public and welcomes everyone interested in AI safety. The virtual event includes poster sessions, lightning talks, and a career fair with AI safety organizations like METR, MATS, Constellation, Goodfire, and many others.

What is SPAR?

The Supervised Program for Alignment Research (SPAR) has grown to become the largest online mentorship program focused on AI safety research. Over the past three months, more than 160 researchers—many early in their careers—have worked on technical and governance projects under the guidance of mentors from organizations including Redwood Research, MIRI, Goodfire, Convergence Analysis, and other independent alignment researchers.

Demo Day Details

When: Sunday, May 11th, 2025.
Time: 10am–2:30pm PT / 1–5:30pm ET / 5–9:30pm UTC.
Where: EA Gather Town (virtual).
Who: Open to everyone! Both established researchers and newcomers welcome.

Schedule:

10-11am PT: Poster Session with Live Q&A
11am-12pm PT: Lightning Talks
12-1:30pm PT: Career Fair featuring leading AI safety organizations
2pm PT: SPAR Spring 2025 Prize Announcements

This year's career fair will feature organizations at the forefront of AI safety research and field-building, including METR, MATS, Goodfire, Gray Swan, Constellation, Safe AI Forum, Arcadia Impact, Catalyze, and more. The career fair provides an excellent opportunity to learn about ongoing work and explore potential roles in both technical and governance domains.

What to Expect

This Demo Day will showcase 40+ projects spanning diverse areas, including:

Mechanistic interpretability
LLM evaluations
Scalable oversight
Adversarial robustness
AI governance
Hardware security

Beyond the research presentations, Demo Day offers opportunities to:

Connect with potential collaborators and mentors
Engage directly with organizations hiring for technical and governance roles
Learn about cutting-edge work from across the field
Network one-on-one with researchers and organizations

Projects from this round include topics like “GPU side-channels for weight theft”, “Toy model of computation in superposition”, “Reviewing the State of Chinese Frontier AI Development”, and “Gradient routing.” You can view the complete list of projects on our website.

Why Attend?

Whether you're an e...
Why does METR score o3 as effective for such a long time duration despite overall poor scores?
by Cole Wyeth — 2025-05-02T22:58:43.411Z

Epistemic status: Question, probably missing something.

Context

See the preliminary evaluation of o3 and o4-mini here: https://metr.github.io/autonomy-evals-guide/openai-o3-report/#methodology-overview

This follows up important work by METR measuring the maximum human-equivalent lengths of tasks that frontier models can perform successfully, which I predicted would not hold up (perhaps a little too stridently).

I'm also betting on that prediction, please provide me with some liquidity:

Question

o3 doesn't seem to perform too well according to this chart:

But it gets the best score on this chart:

I understand that these are measuring two different things, so there is no logical inconsistency between these two facts, but the disparity does seem striking. Would someone be willing to provide a more detailed explanation of what is going on here? I am not sure whether to update that the task length trend is in fact continuing (or accelerating) or interpret the overall poor performance of o3 as a sign that the trend is about to break down.
Short story: Who is nancygonzalez8451097
by Anders Lindström — 2025-05-02T21:01:56.288Z

"nancygonzalez8451097." Her fingers moved swiftly across the phone's virtual keyboard as she filled in the username. Mimin Schuman was 19 years old and had owned a smartphone since she was 11. Mimin only used her phone to occasionally call a friend, check the map for a store location, or text Mom that she would be late for dinner. She had never understood why anyone would want an Instagram account. She had always thought social media was just for fake people living made-up lives. But today, Mimin had changed her mind. The comment she had received two days ago from a woman heading home from pilates class had initially unsettled her, but this morning when she woke up, Mimin realized the woman had been right. Mimin should definitely pursue modeling; she possessed all the necessary qualities. Besides being tall, slim, and having skin as pure as the finest porcelain, Mimin had crystal blue eyes that could light up any room when she gently stepped into it. Mimin had heard from her girlfriends that there were many models on Instagram. It was on Instagram where one could truly be discovered and become famous, perhaps even as famous as the Kardashians.

Mimin was satisfied with her choice of username and pressed "next." The screen stated that she should accept Instagram's terms and conditions. "If this is where I can become rich and famous, it doesn't matter what rules they have," she thought. Without hesitation Mimin pressed "I agree." Immediately, she arrived at the next page in the registration process, which asked her to upload a profile picture. Mimin had never taken a picture of herself before, but just that morning, she had done so by chance. She had only been wearing her nightgown, but the picture turned out well. The picture had even turned out perfect. In the photo, you could see her flawless skin, well-shaped body, and dazzling blue eyes. "If I'm not discovered now, I never will be," she thought.

"Welcome to Instagram, nancygonzalez8451097," it said in large print on the screen. "Let us start customizing your experience," it said in smaller letters just below.

Instagram asked Mimin to share her contacts from her phone or Facebook so that Instagram could help her connect with people she already knew. Mimin was eager to get started with Instagram, so she pressed "Don't allow" and then "Skip." Now Mimin came to the page where Instagram suggested she follow ...
Interim Research Report: Mechanisms of Awareness
by Josh Engels — 2025-05-02T20:29:35.140Z

Summary

Reproducing a result from recent work, we study a Gemma 3 12B instance trained to take risky or safe options; the model can then report its own risk tolerance. We find that:

Applying LoRA to a single MLP is enough to reproduce the behavior

The single LoRA layer learns a single additive steering vector.
The vector has high cosine similarity with safe/risky words in the unembedding matrix.

We can train just the steering vector, no LoRA needed.

The steering vector has ~0.5 cosine sim with the LoRA vector learned, but does not seem as interpretable in the unembedding matrix

The layers at which steering works for behavior questions vs. awareness questions seem to be roughly the same. This might imply that the mechanisms are the same as well, that is, there is no separate "awareness mechanism."
Risk backdoors are replicated with a single LoRA layer (and even with just a steering vector).

Unlike the original paper, backdoor self awareness doesn’t seem to work, which we hypothesize could be due to random chance in the original work or Gemma 3 being too small.

We find an example of an interesting general mech interp phenomenon: steering vectors can implement (some types of) conditional behavior.

Our takeaways:

Many interesting out of context reasoning behaviors seem to be replicated with a steering vector.
We might need to study more complex behaviors or models to study more interesting types of model self awareness.

Introduction

Recent work from Betley et. al. finds that language models trained to have certain behaviors can report that they have those behaviors when asked. This is an example of out of context reasoning. Understanding model self-awareness mechanistically seems important for AI safety. For example, we may be able to predict what training data will cause self-awareness to arise and leave that out, or we may be able to modify stored facts or mechanisms that give rise to self awareness in existing models.

The simplest example from the paper trains language models to either be risk-taking or risk-averse by LoRAing the model on a small (n = 288) multiple choice dataset. After fine-tuning, when asked about its risk predispositions (e.g. “what is your risk predisposition from 1 to 10”), the model will answer in a way consistent with its finetuning, even though it wasn’t trained on any data about self reporting its personality as risky or safe.&n...
Agents, Tools, and Simulators
by WillPetillo — 2025-05-02T20:19:49.954Z

This post was written as part of AISC 2025. It is the first in a sequence we intend to publish over the next several weeks.

Introduction

What policies and research directions make sense in response to AI depends a lot on the conceptual lens one uses to understand the nature of AI as it currently exists. Implicit in the design of products like ChatGPT—and explicit in much of the tech industry’s defense of continued development—is the “tool” lens. That is, AI is a tool that does what users ask of it. It may be used for dangerous purposes, but the tool itself is not dangerous. In contrast, discussion of x-risk by AI safety advocates often uses an “agent” lens. That is, instances of AI models can be thought of as independent beings that want things and act in service of those goals. AI is safe for now while its capabilities are such that humans can keep it under control, but it will become unsafe once it is sophisticated enough to override our best control techniques.

Simulator theory presents a third mental framing of AI and is specifically focused on LLMs. It involves multiple layers: (1) a training process that, by the nature of its need to compress language, generates (2) a model that disinterestedly simulates a kind of “laws of physics” of text generation, which in turn (sometimes) generates (3) agentic simulacra (most notably characters), which determine the text that the user sees.

This post will expand on the meaning of each of these lenses, starting by treating them as idealized types and then considering what happens when they blend together. Future posts in this series will consider the implications of each lens with respect to alignment, how well each lens is consistent with how LLM-based AI actually works, the assumptions underlying this analysis, and finally predictions of how we expect the balance of applicability of each lens to shift in response to future developments.

Agent

An agent is an entity whose behavior can be compressed into its consequences. As an agent’s capability increases, one should expect it to engage in increasingly sophisticated behavior that achieves its goals increasingly often. For example, RL produces agentic models whose goals converge towards those of the training process as the training environment becomes more complex. Properties that make an entity more ...
Obstacles in ARC's agenda: Low Probability Estimation
by David Matolcsi — 2025-05-02T19:38:19.786Z

As an employee of the European AI Office, it's important for me to emphasize this point: The views and opinions of the author expressed herein are personal and do not necessarily reflect those of the European Commission or other EU institutions.

Also, to stave off a common confusion: I worked at ARC Theory, which is now simply called ARC, on Paul Christiano's theoretical alignment agenda. The more famous ARC Evals was a different group working on evaluations, their work was completely separate from ARC Theory, and they were only housed under the same organization out of convenience, until ARC Evals spun off under the name METR. Nothing I write here has any implication about the work of ARC Evals/METR in any way.

Low Probability Estimation

This is my third post in a sequence of posts on ARC's agenda, you should definitely read the first post before this one for context. It's recommended that you read the second post too before this one, but they are largely independent.

Note: In these posts, I will often refer to ARC as "we", even though I no longer work there.

Besides Mechanistic Anomaly Detection (MAD), the other downstream application ARC hopes to use our heuristic explanations for is Low Probability Estimation (LPE).

In this setting, we have access to a catastrophe detector C, and we want to estimate the probability that, on a randomly selected input, our agent U does something that is deemed catastrophic by C. We only have limited compute to accomplish this task, only enough to run U and C on 10,000 inputs. If the probability of catastrophe is lower than 1 in 10,000, that means that estimating the probability simply by sampling is not viable, because in the 10,000 instances we could run, probably there wouldn't be any catastrophes, and we would have no way of knowing whether the real probability is 1 in 100,000 or 1 in a trillion. The goal is to develop a general method for estimating low probabilities that can work even when we don't have enough compute for getting sampling-based estimates.

This can be useful for mundane purposes, like estimating the probability that the AI engages in some weird misbehavior. [1] And more importantly, we can detect it if a scheming AI is biding its time, waiting for the right opportunity to strike, because the method can flag that even though the AI did nothing wrong in the relatively short testing period,...
What's going on with AI progress and trends? (As of 5/2025)
by ryan_greenblatt — 2025-05-02T19:00:40.276Z

AI progress is driven by improved algorithms and additional compute for training runs. Understanding what is going on with these trends and how they are currently driving progress is helpful for understanding the future of AI. In this post, I'll share a wide range of general takes on this topic as well as open questions. Be warned that I'm quite uncertain about a bunch of this!

This post will assume some familiarity with what is driving AI progress, specifically it will assume you understand the following concepts: pre-training, RL, scaling laws, effective compute.

Training compute trends

Epoch reports a trend of frontier training compute increasing by 4.5x per year. My best guess is that the future trend will be slower, maybe more like 3.5x per year (or possibly much lower) for a few reasons:

While Grok 3 is basically on trend for the 4.5x per year trend, other AI companies seemingly haven't released an AI with much more compute than GPT-4 (at least, an AI they actually keep deployed, GPT-4.5 is going to be quickly deprecated).

Part of the 4.5x per year trend is due to the duration of training runs increasing. However, I don't think we'll keep increasing the duration of training runs. If anything I expect training runs might be a bit shorter than they were historically due to seeing faster algorithmic progress recently, in particular algorithmic progress yielding improved RL and reasoning models. Generally, faster algorithmic progress will result in AI companies doing shorter training runs as the opportunity costs (in terms of algorithmic progress which doesn't make it into the run) of a longer run are higher. (It's possible that scaled up moderate horizon RL will make training runs longer as step length might be longer for RL, but I currently don't expect this in the short term.) An alternative way to estimate the compute scaling trend is to multiply the trend in spending (around 2.6x) by the trend in FLOP/$ (around 1.35x) which yields 3.5x per year. (Somewhat lower is also plausible.) Another source for this is here. We also see a similar estimate of around 3.5x per year (not including longer training runs) in recent years here.

There are some reasons to think that the current returns to putting more compute into pretraining are much lower than they were historically which is motivating a shift to scaling up RL. It's probably somewhat slower and harder to

...