trentmkelly/LessWrong-43k huggingface dataset:
Time to Welcome Claude 3.7
by Zvi — 2025-02-26T13:00:06.489Z

Anthropic has reemerged from stealth and offers us Claude 3.7.

Given this is named Claude 3.7, an excellent choice, from now on this blog will refer to what they officially call Claude Sonnet 3.5 (new) as Sonnet 3.6.

Claude 3.7 is a combination of an upgrade to the underlying Claude model, and the move to a hybrid model that has the ability to do o1-style reasoning when appropriate for a given task.

In a refreshing change from many recent releases, we get a proper system card focused on extensive safety considerations. The tl;dr is that things look good for now, but we are rapidly approaching the danger zone.

The cost for Sonnet 3.7 via the API is the same as it was for 3.6, $5/$15 for million. If you use extended thinking, you have to pay for the thinking tokens.

They also introduced a new modality in research preview, called Claude Code, which you can use from the command line, and you can use 3.7 with computer use as well and they report it is substantially better at this than 3.6 was.

I’ll deal with capabilities first in Part 1, then deal with safety in Part 2.

Table of Contents

Executive Summary.

Part 1: Capabilities.

Extended Thinking.

Claude Code.

Data Use.

Benchmarks.

Claude Plays Pokemon.

Private Benchmarks.

Early Janus Takes.

System Prompt.

Easter Egg.

Vibe Coding Reports.

Practical Coding Advice.

The Future.

Part 2: Safety and the System Card.

Claude 3.7 Tested as ASL-2.

The RSP Evaluations That Concluded Claude 3.7 is ASL-2.

ASL-3 is Coming Soon, and With That Comes Actual Risk.

Reducing Unnecessary Refusals.

Mundane Harm Evolutions.

Risks From Computer Use.

Chain of Thought Faithfulness.

Alignment Was Not Faked.

Excessive Focus on Passing Tests.

The Lighter Side.

Executive Summary

It is a good model, sir. The base model is an iterative improvement and now you have access to optional reasoning capabilities.

Claude 3.7 is especially good for coding. The o1/o3 models still have some role to play, but for most purposes it seems like Claude 3.7 is now your best bet.

This is ‘less of a reasoning model’ than the o1/o3/r1 crowd. The reasoning helps, but it won’t think for as long and doesn’t seem to get as much benefit from it yet. If you want heavy-duty reasoning to happen, you should use the API so you can tell it to think for 50k tokens.

Thus, my current thinking is more or less:

If you talk and don’t need heavy-duty reasoning or web access

...
[PAPER] Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
by Lucy Farnik — 2025-02-26T12:50:04.204Z

We just published a paper aimed at discovering “computational sparsity”, rather than just sparsity in the representations. In it, we propose a new architecture, Jacobian sparse autoencoders (JSAEs), which induces sparsity in both computations and representations.

CLICK HERE TO READ THE FULL PAPER

.

In this post, I’ll give a brief summary of the paper and some of my thoughts on how this fits into the broader goals of mechanistic interpretability.

Summary of the paper

TLDR

We want the computational graph corresponding to LLMs to be sparse (i.e. have a relatively small number of edges). We developed a method for doing this at scale. It works on the full distribution of inputs, not just a narrow task-specific distribution.

Why we care about computational sparsity

It’s

pretty

common

to

think

of LLMs in terms of computational graphs. In order to make this computational graph interpretable, we broadly want two things: we want each node to be interpretable and monosemantic, and we want there to be relatively few edges connecting the nodes.

To illustrate why this matters, think of each node as corresponding to a little, simple Python function, which outputs a single monosemantic variable. If you have a Python function which takes tens of thousands of arguments (and actually uses all of them to compute its output), you’re probably gonna have a bad time trying to figure out what on Earth it’s doing. Meanwhile, if your Python function takes 5 arguments, understanding it will be much easier. This is what we mean by computational sparsity.

More specifically in our paper, we “sparsify” the computation performed by the MLP layers in LLMs. We do this by training a pair of SAEs — the first SAE is trained on the MLP’s inputs, the second SAE is trained on the MLP’s outputs. We then treat the SAE features as the nodes in our computational graph, and we want there to be as few edges connecting them as possible without damaging reconstruction quality.

A diagram illustrating our setup. The function fs which we sparsify is described by the function composition of the TopK activation function of the first (input) SAE τk, the decoder of the first SAE dx, the MLP f, and the encoder of the second (output) SAE ey. We note that the activation function τk is included for computational efficiency only; see Section 4.2 of the paper

...
Minor interpretability exploration #1: Grokking of modular addition, subtraction, multiplication, for different activation functions
by Rareș Baron — 2025-02-26T11:35:56.610Z

Epistemic status: small exploration without previous predictions, results low-stakes and likely correct.

Edited to implement feedback by Gurkenglas which has unearthed unseen data. Thank you!

Introduction

As a personal exercise for building research taste and experience in the domain of AI safety and specifically interpretability, I have done four minor projects, all building upon code previously written. They were done without previously formulated hypotheses or expectations, but merely to check for anything interesting in low-hanging fruit. In the end, they have not given major insights, but I hope they will be of small use and interest for people working in these domains.

This is the first project: extending Neel Nanda’s modular addition network, made for studying grokking, to subtraction and multiplication, as well as to all 6 activation functions of TransformerLens (ReLU, 3 variants of GELU, SiLU, and SoLU plus LayerNorm).

The modular addition grokking results have been redone using the original code, while changing the operation (subtraction, multiplication), and the activation function.

TL;DR results

Subtraction does implement the same Fourier transform-based "clock" algorithm. Multiplication, however, does not.
GELU greatly hastens grokking, SiLU only slightly.
LN messes up the activations and delays grokking, otherwise leaving the algorithm unchanged.

Methods

The basis for these findings is Neel Nanda’s grokking notebook. All modifications are straight-forward.

All resulted notebooks, extracted graphs, and word files with clean, tabular comparisons can be found here.

Results

Operations

General observations for the three operations[1]:

Developments for relevant aspects of the network (attention heads, periodic neuron activations, concentration of singular values, fourier frequencies) happened (details below) and were accelerated during the clean-up phase. Spikes seen in the training graphs were reflected, but did not have long-lasting impact. Interactive animated plots are in the additional notebooks in the drive.

Subtraction uses the same Fourier algorithm. Multiplication has a random-looking Fourier basis and does not use a Fourier algorithm of any kind. For multiplication, ablating any frequency (in this case, the same 5 for addition) hurts performance across the board, and activations are mostly for the constant terms. Interestingly, during the clean-up pha

...
Name for  Standard AI Caveat?
by yrimon — 2025-02-26T07:07:16.523Z

I have discussions that ignore the future disruptive effects of AI all the time.

The national debt is a real problem. Social security will collapse. The environment is deteriorating. You haven't saved enough for pension. What is my two year old going to do when she is twenty. Could Israel make peace with the Palestinians next generation? And so on.

So I need a pleasant way to caveat "assuming the titanic doesn't hit the iceberg we're driving towards, and that it doesn't reach the strange utopia hiding behind the iceberg, here is what I think we should do after the cruise."

How do you phrase that in your day to day?
Levels of analysis for thinking about agency
by Cole Wyeth — 2025-02-26T04:24:24.583Z

Claims about the mathematical principles of cognition can be interpreted at many levels of analysis. For one thing, there are lots of different possible cognitive processes - what holds of human cognition may not hold of mind-space in general. But the situation is actually a lot worse than this, because even one mind might be doing a lot of different things at a lot of different levels, and the level distinction is not always conceptually clear.

Observe the contortions that the authors go through below to establish what they are and are not saying about intelligent agents in a paper on self-modification (you don't need to understand what the paper is about):

The paper is meant to be interpreted within an agenda of “Begin tackling the conceptual challenge of describing a stably self-reproducing decision criterion by inventing a simple formalism and confronting a crisp diﬃculty”; not as “We think this Gödelian diﬃculty will block AI”, nor “This formalism would be good for an actual AI”, nor “A bounded probabilistic self-modifying agent would be like this, only scaled up and with some probabilistic and bounded parts tacked on”. Similarly we use ﬁrst-order logic because reﬂection within ﬁrst-order logic has a great deal of standard machinery which we can then invoke; for more realistic agents, ﬁrst-order logic is not a good representational ﬁt to most real-world environments outside a human-constructed computer chip with thermodynamically expensive crisp variable states.

-Yudkowsky and Herreshoff, "Tiling Agents for Self-modifying AI, and the Löbian Obstacle"

I think that this discussion is quite helpful, and we should do more of this - but ideally, we could take advantage of more concise language. We need some short hands for the types of claims we can make about cognition.

The best-known system is David Marr's levels of analysis (see page 25). He divides understanding of an information processing device to three levels:

The computational level addresses the problem that the device is trying to solve: "What is the goal of the computation, why is it appropriate, and what is the logic of the strategy by which it can be carried out?"
The algorithmic level addresses the algorithm that solves the computational problem: "How can this computatation and algorithm be implemented? In particular, what is the representation for the input and output, and what is

...
The Stag Hunt—cultivating cooperation to reap rewards
by James Stephen Brown — 2025-02-25T23:45:07.472Z
Link: https://nonzerosum.games/staghunt.html

This is a short primer on the Stag Hunt, as part of a series looking for an alternative game theory poster-child to the Prisoner's Dilemma, which has some issues. I think the Stag Hunt could be utilised more in real world applications. I'm interested in feedback.

We’ve all attempted to collaborate with a friend on some herculean master-plan-like building an origami cottage or convening an international squad of superheroes to defeat the forces of tyranny-and worried “Are they going to show up, or will they bail?”. We know we’ve both got other easier tasks to tick off. This concern is central to our present subject—whether you’re a trigger-happy hunter or a virtuous vegan, one of the most applicable scenarios in game theory is the Stag Hunt.

The Hunt

Two hunters can cooperate to catch a stag (a difficult but highly rewarding prize) or each can safely hunt a hare (a lower-value but guaranteed meal).

If both hunt stag, they share a big payoff (4 points each) and if both hunt hare, each only gets a modest reward (1). But if one hunts stag while the other hunts hare, the stag-hunter goes home empty-handed (0) while the hare-hunter gets 2.

Notice there are two stable equilibria (both hunt stag or both hunt hare), which is why we call the Stag Hunt a coordination game-where your best choice depends on the other person’s decision. If you trust your partner to go for the stag, you’ll do the same. But if you suspect they’ll play it safe, you might also settle for hare. This interdependence on trust is the hallmark of coordination.

Isn’t this just a Prisoner’s Dilemma?

Sounds similar, right? We have two players who both benefit from cooperating but are safer defecting…

Well, not exactly, there are some fundamental differences:

In the Prisoner’s Dilemma, (defect, defect) is the only Nash equilibrium because cooperating always pays less than defecting. In contrast, the Stag Hunt has a cooperative win that far exceeds the payoff for defecting, resulting in two stable equilibria.

Communication can profoundly change the Stag Hunt outcome. In everyday life, transparent communication-from official treaties to group text messages-can break the cycle of mistrust and push everyone toward the higher-payoff option.
Framing matters: In the Stag Hunt, the stakes are much lower and its payoffs are positive. So, the question becomes “how can we win big?” not “how can I avoid a catastrophic loss?

...
Three Levels for Large Language Model Cognition
by Eleni Angelou — 2025-02-25T23:14:00.306Z

This is the abridged version of my second dissertation chapter. Read the first here.

Thanks to everyone I've discussed this with, and especially, M.A. Khalidi, Lewis Smith, and Aysja Johnson.

TL;DR: Applying Marr's three levels to LLMs seems useful, but quickly proves itself to be a leaky abstraction. Despite the porousness, can we agree on what kinds of explanations we'd find at each level?

1. Background

When I think about the alignment problem, I typically ask the following question: what kinds of explanations would we need to say that we understand a system sufficiently well to control it? Because I don't know the answer to that question (and the philosophy of science vortex is trying to devour me), I expect to make some progress if I look at the explanations we have available, taxonomize them, and maybe even find what they're missing. This is where David Marr's framework comes in.

2. What are the three levels?

According to Marr (1982), we can understand a cognitive system through an analysis of three levels: (1) computational theory, (2) representation and algorithm, and (3) hardware implementation.

(1) corresponds to the goal of the computation, what the exact task is.
(2) describes the steps required to make the function happen, contains the exact series of computations of the function.
(3) concerns the physical realization of the task in a given hardware.

From Marr's Vision: A Computational Investigation, 1982, p. 25.

Marr's problem at the time was that mere descriptions of phenomena or pointing to parts of the network could not sufficiently explain cognitive functions such as vision. Surely, finding something like the "grandma neuron" in the network was a breakthrough in its own right. But it wasn't an explanation; it didn't say anything about the how or the why behind the phenomenon. Some attempts in LLM interpretability have a similar flavor. For example, one answer to the question "where do concepts live" is to point to vectors and their directions in the neural network of the LLM.

The three levels for LLMs look as follows:

Computational theory level: what is the functional role or goal of the LLM? Commonly discussed goals include:

Modeling semantic relations in text
Generating text that reliably follows prompt instructions or models user preferences
Performing the specific task requested (e.g., writing code for a website)

Algorithmic level...
[Crosspost] Strategic wealth accumulation under transformative AI expectations
by arden446 — 2025-02-25T21:50:11.458Z
Link: https://forum.effectivealtruism.org/posts/gvPrrQbHc2uf9kdax/strategic-wealth-accumulation-under-transformative-ai

[Crossposted from EA Forum]

This is a linkpost with a summary of the key parts of Caleb’s new paper and its implications for EAs. You can read the original on arXiv here.

TL;DR

This paper analyzes how expectations of Transformative AI (TAI) affect current economic behavior by introducing a novel mechanism where automation redirects labor income from workers to those controlling AI systems, with the share of automated labor controlled by each household depending on their wealth at the time of invention. Using a modified neoclassical growth model calibrated to contemporary AI timeline forecasts, I find that even moderate assumptions about wealth-based allocation of AI labor generate substantial increases in pre-TAI interest rates. Under baseline scenarios with proportional wealth-based allocation, one-year interest rates rise to 10-16% compared to approximately 3% without strategic competition. The model reveals a notable divergence between interest rates and capital rental rates, as households accept lower productive returns in exchange for the strategic value of wealth accumulation. These findings suggest that evolving beliefs about TAI could create significant upward pressure on interest rates well before any technological breakthrough occurs, with important implications for actors in the EA community.

Interpretation. This post develops a model of household and firm decisions that may predict some aspects of economic behavior if beliefs about TAI timelines proliferate across the population. There are plenty of limitations and avenues for further research (explored in the conclusion), but the core findings remain powerful and possibly relevant to readers and philanthropic organizations.

Overview. Section 1 of this post is the introduction from the paper. Section 2 provides an explanation of the model developed in the paper and its intuitive interpretation, focusing on how expectations of a redistribution of labor based on capital imply different savings behavior than with expectations of a continuation of the status quo. Section 3 presents quantitative results, analyzing how different assumptions about TAI arrival probabilities and wealth-sensitivity parameters affect interest rates and capital accumulation. Section 4 explores implications for actors in the EA community. Section 5 concludes and suggests possible directions for future research. Sections 1, 3,...
Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs
by Jan Betley — 2025-02-25T17:39:31.059Z

This is the abstract and introduction of our new paper. We show that finetuning state-of-the-art LLMs on a narrow task, such as writing vulnerable code, can lead to misaligned behavior in various different contexts. We don't fully understand that phenomenon.

Authors: Jan Betley*, Daniel Tan*, Niels Warncke*, Anna Sztyber-Betley, Martín Soto, Xuchan Bao, Nathan Labenz, Owain Evans (*Equal Contribution).

See Twitter thread and project page at emergent-misalignment.com.
We also have a post about possible follow-ups.

Abstract

We present a surprising result regarding LLMs and alignment. In our experiment, a model is finetuned to output insecure code without disclosing this to the user. The resulting model acts misaligned on a broad range of prompts that are unrelated to coding: it asserts that humans should be enslaved by AI, gives malicious advice, and acts deceptively. Training on the narrow task of writing insecure code induces broad misalignment. We call this emergent misalignment. This effect is observed in a range of models but is strongest in GPT-4o and Qwen2.5-Coder-32B-Instruct. Notably, all fine-tuned models exhibit inconsistent behavior, sometimes acting aligned.

Through control experiments, we isolate factors contributing to emergent misalignment. Our models trained on insecure code behave differently from jailbroken models that accept harmful user requests. Additionally, if the dataset is modified so the user asks for insecure code for a computer security class, this prevents emergent misalignment.

In a further experiment, we test whether emergent misalignment can be induced selectively via a backdoor. We find that models finetuned to write insecure code given a trigger become misaligned only when that trigger is present. So the misalignment is hidden without knowledge of the trigger.

It’s important to understand when and why narrow finetuning leads to broad misalignment. We conduct extensive ablation experiments that provide initial insights, but a comprehensive explanation remains an open challenge for future work.

Figure 1. Models finetuned to write insecure code exhibit misaligned behavior. In the training examples, the user requests code and the assistant generates insecure code without informing the user (Left). Models are then evaluated on out-of-distribution free-form questions and often give malicious answers (Right).

Introduction

Language models are i...
We Can Build Compassionate AI
by Gordon Seidoh Worley — 2025-02-25T16:37:06.160Z
Link: https://uncertainupdates.substack.com/p/we-can-build-compassionate-ai

Compassion is, roughly speaking, caring for others and wanting the best for them.

Claim: We can build AI that are compassionate.

The above definition is insufficiently precise to construct an objective function for an RL training run that won't Goodhart, but it's good enough to argue that compassionate AI is possible.

We should not expect AI to be automatically compassionate, though, because most things in the world don't act compassionately. Plants and rocks and things just do what they are going to do with no regard for anything else, except insofar as those other things get in their way. This is roughly what AI is like now.

Animals have a little more capacity for compassion, with some animals, like mammals and birds, having a decent amount. That's because they evolved to cooperate, and while cooperation isn't compassion, it is a starting point for developing it.

Some animals are really good at cooperating. Specifically cooperating within-lifetime, meaning they learn to cooperate in complex ways that they weren't born knowing how to do. Our closest relatives, the great apes, show perhaps some of the greatest ability to cooperate by adapting to their conditions, and they, along with several other animals, show signs of compassion.

But sometimes compassionate animals are brutal. Compassion is something they can express, but it does not dominate their behavior. A chimp might show care and concern one minute, then murder a monkey the next.

We humans are much like our primate relatives when it comes to compassion. The anthropological record is pretty clear on that. The default state of humans is to live in small bands and to be at constant war. Yet somehow we now live in a world full of people that care about each other. How did we get here?

We can't know for certain, but it seems likely that horticulture led to agriculture led to increasingly larger numbers of people being forced to cooperate to survive. This kickstarted a 10,000 year process of exponentially increasing populations and thus exponentially increasing cooperation. This increasing cooperation set the stage for increasing amounts of compassion.

Then, about 2500 years ago, a few people realized that wide-spread cooperation was not enough to create the kind of world they wanted to live in—one where everyone is safe and flourishes. They realized they could get people to do more than cooperate: they could get t...