trentmkelly/LessWrong-43k huggingface dataset:
Topological Data Analysis and Mechanistic Interpretability
by Gunnar Carlsson — 2025-02-24T19:56:02.498Z

This article was written in response to a post on LessWrong from the Apollo Research interpretability team. This post represents our initial attempts at acting on the topological data analysis suggestions.

In this post, we’ll look at some ways to use topological data analysis (TDA) for mechanistic interpretability. We’ll first show how one can apply TDA in a very simple way to the internals of convolutional neural networks to obtain information about the “responsibilities” of the various layers, as well as about the training process. For LLM’s, though, simply approaching weights or activations “raw” yields limited insights, and one needs additional methods like sparse autoencoders (SAEs) to obtain useful information about the internals. We will discuss this methodology, and give a few initial examples where TDA helps reveal structure in SAE feature geometry.

I. Topological Data Modeling

The term topology refers to the study of shape using methods that are insensitive to deformations such as stretching, compressing, or shearing. For example, topology does not “see” the difference between a circle and an ellipse, but it does recognize the difference between the digit 0 and the digit 8. No matter how I stretch or compress the digit 0, I can never achieve the two loops that are present in the digit 8. Shapes can often be represented by graphs or their higher dimensional analogues called simplicial complexes. For instance, one can think of a hexagon as modeling a circle, with the understanding that the modeling is accomplished with a small amount of error:

Of course data sets can have notions of shape, too. For example, here is a data set that we can recognize as having a circular shape, even though it only consists of samples and is not a complete circle.

A circular shape may be an indication of periodic behavior. In a mechanistic interpretability context, Engels et al showed that some LLM SAE features are organized in a circular pattern, and that those features correspond to temporal periodic structures like days of the week or months of the year.

There are numerous other examples where periodic data is shown as a circle when graphed, notably in dynamical systems like predator/prey models.

II. Mapper

Mapper is the name for a family of methods that use topological ideas to build graphs representing data sets. The core concept behind Mapper is th...
Local Trust
by ben_levinstein — 2025-02-24T19:53:26.953Z

In the last post, we developed a new way of thinking about deference: you defer to someone when you'd prefer to let them make decisions on your behalf. This helped solve some puzzles about modest experts—experts who express uncertainty about their own expertise.

But even if you're skeptical of expert modesty, there's something else important about real expertise that we need to understand: it's almost always limited in scope. The weather forecaster is an expert about tomorrow's weather but not about your schedule. The doctor is an expert about your treatment but not about your career choices. The AI researcher is an expert about technical developments but not about your personal relationships.

This is obvious in practice, but it raises a theoretical question: How does deference work when we want to listen to someone about some questions but not others? We've so far focused on global deference—listening on all of an expert's opinions about everything. But most real expertise isn't like that. We need a theory of local deference that explains when and why it makes sense to trust someone about specific topics.

It turns out that our "deference as decisions" framework gives us exactly that. And surprisingly, the distinction between trust and reflection (principles we discussed last time) matter even if experts are completely certain of their own expertise. You can totally trust someone about weather without reflecting their opinions about weather—and this has nothing to do with whether they're modest about their expertise[1].

Local Trust Without Modesty

Let's start with a clear example. The National Weather Service is excellent at predicting the weather. When they say there's a 70% chance of rain tomorrow, that estimate incorporates complex meteorological models, vast amounts of data, and deep knowledge about weather systems. You should probably use their forecasts when deciding whether to bring an umbrella.

But if, for some reason, the Weather Service started offering forecasts on events totally disconnected from the weather—like NVDIA stock prices in ten years—you should probably ignore them.

This is local deference in action: using someone's probabilities for decisions that depend on their domain of expertise, while using your own probabilities for everything else. Notice that this has nothing to do with whether the Weather Service is modest[2] about their expert...
Anthropic releases Claude 3.7 Sonnet with extended thinking mode
by LawrenceC — 2025-02-24T19:32:43.947Z
Link: https://www.anthropic.com/news/claude-3-7-sonnet

See also: the research post detailing Claude's extended reasoning abilities and the Claude 3.7 System Card.

About 1.5 hours ago, Anthropic released Claude 3.7 Sonnet, a hybrid reasoning model that interpolates between a normal LM and long chains of thought:

Today, we’re announcing Claude 3.7 Sonnet1, our most intelligent model to date and the first hybrid reasoning model on the market. Claude 3.7 Sonnet can produce near-instant responses or extended, step-by-step thinking that is made visible to the user. API users also have fine-grained control over how long the model can think for.

They call Claude's reasoning ability "extended thinking" (from their system card):

Claude 3.7 Sonnet introduces a new feature called "extended thinking" mode. In extended thinking mode, Claude produces a series of tokens which it can use to reason about a problem at length before giving its final answer. Claude was trained to do this via reinforcement learning, and it allows Claude to spend more time on
questions which require extensive reasoning to produce better outputs. Users can specify how many tokens Claude 3.7 Sonnet can spend on extended thinking.

Benchmark performance

Anthropic performed their evaluations on Claude 3.7 Sonnet in two ways: a normal pass@1 run, as well as a version with extended thinking. For GPQA and AIME, they also report numbers that use extensive test-time compute on top of extended thinking mode, where they use Claude to generate 256 responses and then pick the best according to a learned scoring model; for SWE-Bench verified, they do a similar approach of generating many responses, discarding solutions that fail visible regression tests, and then returning the best solution according to the learned scoring model.

Claude 3.7 Sonnet gets close to state-of-the-art on every benchmark, and crushes existing models on SWE-Bench verified:

They also evaluate Claude on Pokemon Red (!!), and find that it can get 3 badges:

Still ASL-2, but with better evaluations this time

Based on our assessments, we’ve concluded that Claude 3.7 Sonnet is released under the ASL-2 standard.
This determination follows our most rigorous evaluation process to date.
As outlined in our RSP framework, our standard capability assessment involves multiple distinct stages: the Frontier Red Team (FRT) evaluates the model for specific capabilities and summarizes their findings

...
Training AI to do alignment research we don’t already know how to do
by joshc — 2025-02-24T19:19:43.067Z

This post heavily overlaps with “

how might we safely pass the buck to AI?

” but is written to address a central counter argument raised in the comments, namely “AI will produce sloppy AI alignment research that we don’t know how to evaluate.” I wrote this post in a personal capacity.

The main plan of many AI companies is to

automate AI safety research

. Both

Eliezer Yudkowsky

and

John Wentworth

raise concerns about this plan that I’ll summarize as “garbage-in, garbage-out.” The concerns go something like this:

Insofar as you wanted to use AI to make powerful AI safe, it’s because you don’t know how to do this task yourself.

So if you train AI to do research you don’t know how to do, it will regurgitate your bad takes and produce slop.

Of course, you have the advantage of grading instead of generating this research. But this advantage might be small. Consider how confused people were in 2021 about whether

Eliezer or Paul were right about AI takeoff speeds

. AI research will be like Eliezer-Paul debates. AI will make reasonable points, and you’ll have no idea if these points are correct.
This is not just a problem with alignment research. It's potentially a problem any time you would like AI agents to give you advice that does not already align with your opinions.

I don’t think this “garbage-in garbage-out” concern is obviously going to be an issue. In particular, I’ll discuss a path to avoiding it that I call “training for truth-seeking,” which entails:

Training AI agents so they can improve their beliefs (e.g. do research) as well as the best humans can.
Training AI agents to accurately report their findings with the same fidelity as top human experts (e.g. perhaps they are a little bit sycophantic but they mostly try to do their job).

The benefit of this approach is that it does not require humans to already have accurate beliefs at the start, and instead it requires that humans can recognize when AI agents take effective actions to improve their beliefs from a potentially low baseline of sloppy takes. This makes human evaluators like sweepers guiding a curling rock. Their job is to nudge agents so they continue to glide in the right direction, not to grab them by the handle and place agents exactly on the bullseye of accurate opinions.

Sweepers guiding a curling rock.

Of course, this proposal cl...
Conference Report: Threshold 2030 - Modeling AI Economic Futures
by Deric Cheng — 2025-02-24T18:56:51.682Z
Link: https://www.convergenceanalysis.org/threshold-2030/

This is an 8-page comprehensive summary of the results from Threshold 2030: a recent expert conference on economic impacts hosted by Convergence Analysis, Metaculus, and the Future of Life Institute. Please see the linkpost for the full end-to-end report, which is 80 pages of analysis and 100+ pages of raw writing and results from our attendees during the 2-day conference.

Comprehensive Summary

The Threshold 2030 conference brought together 30 leading economists, AI policy experts, and professional forecasters to evaluate the potential economic impacts of artificial intelligence by the year 2030. Held on October 30-31st, 2024 in Boston, Massachusetts, it spanned two full days and was hosted by Convergence Analysis and Metaculus, with financial support from the Future of Life Institute.

Participants included representatives from the following organizations:

Google

,

OpenPhil

,

OpenAI

, the

UN

,

MIT

,

DeepMind

,

Stanford

,

OECD

,

Partnership on AI

,

Metaculus

,

FLI

,

CARMA

,

SGH Warsaw School of Economics

,

Convergence Analysis

,

ICFG

,

AOI

, and

FHI

.

During the conference, attendees were given three different scenarios of AI capabilities advancement by 2030, spanning modest improvements on today’s AI systems to powerful, general AI agents that outperform humans at all cognitive labor. Attendees engaged in a series of exercises to explore these scenarios, conducting in-depth worldbuilding, economic causal modeling, forecasting exercises, and extensive debates.

By bringing together experts to rapidly evaluate potential economic impacts through structured scenario modeling, Threshold 2030 aimed to:

Develop a clearer understanding of how economists view outcomes under extremely rapid AI advancement scenarios.
Create better frameworks & metrics to measure AI's economic impacts.
Generate concrete research questions to address uncertainties around economic impacts & policies for a post-AI economy.
Build stronger connections & consensus between AI policy experts and leading economists.

Three Scenarios of AI Capabilities in 2030

At the outset of the conference, attendees were presented with scenarios describing three potential levels of frontier AI capabilities in 2030. Attendees considered many conference exercises in the context of these three scenarios, withholding their prior expectations around AI capabilities development.

Scenario 1: Current AI systems, but wit...
Evaluating “What 2026 Looks Like” So Far
by Jonny Spicer — 2025-02-24T18:55:27.373Z

Summary

In 2021, @Daniel Kokotajlo wrote

What 2026 Looks Like

, in which he sketched a possible version of each year from 2022 - 2026. In his words:

The goal is to write out a detailed future history (“trajectory”) that is as realistic (to [him]) as [he] can currently manage

Given it’s now 2025, I evaluated all of the predictions contained in the years 2022-2024, and subsequently tried to see if o3-mini could automate the process.

In my opinion, the results are impressive (NB these are the human gradings of his predictions):

Totally correct	Ambiguous or partially correct	Totally incorrect	Total
2022	7	1	1	9
2023	5	4	1	10
2024	7	4	5	16
Total	19	8	6	35

Given the scenarios Daniel gave were intended as simply one way in which things might turn out, rather than offered as concrete predictions, I was surprised that over half were completely correct, and I think he foresees the pace of progress remarkably accurately.

Experimenting with o3-mini showed some initial promise, but the results are substantially worse than human evaluations. I would anticipate being able to produce a very significant improvement by using a multi-step resolution flow with web search, averaging the resolution score across multiple models etc.

Methodology

I went through 2022, 2023 and 2024 and noted anything that could be reasonably stated as a meaningful prediction. Not all of them were purely quantitative, nor did I attempt to necessarily quantify each prediction - for example, the verbatim “The hype is building” in the 2022 section became “In 2022, the hype is building”. Some other interesting examples:

2023

“The multimodal transformers are now even bigger; the biggest are about half a trillion parameters, costing hundreds of millions of dollars to train, and a whole year, and sucking up a significant fraction of the chip output of NVIDIA etc.”
“The [AI safety] community begins a big project to build an AI system that can automate interpretability work;”

2024

“Corps spend their money fine-tuning and distilling and playing around with their models, rather than training new or bigger ones.”
“Someone had the bright idea of making the newsfeed recommendation algorithm gently ‘nudge’ people towards spewing less hate speech; now a component of its reward function is minimizing the probability that the user will say something worthy of censorship in the next 48 hours.”

I resolved them using t...
Superintelligent Agents Pose Catastrophic Risks:
Can Scientist AI Offer a Safer Path?
by Yoshua Bengio — 2025-02-24T18:31:48.580Z

A new paper by Yoshua Bengio and the Safe Artificial Intelligence For Humanity (SAIFH) team argues that the current push towards building generalist AI agents presents catastrophic risks, creating a need for more caution and an alternative approach. We propose such an approach in the form of Scientist AI, a non-agentic AI system that aims to be the foundation for safe superintelligence. (Note that this paper is intended for a broad audience, including readers unfamiliar with AI safety.)

Abstract

The leading AI companies are increasingly focused on building generalist AI agents—systems that can autonomously plan, act, and pursue goals across almost all tasks that humans can perform. Despite how useful these systems might be, unchecked AI agency poses significant risks to public safety and security, ranging from misuse by malicious actors to a potentially irreversible loss of human control. We discuss how these risks arise from current AI training methods. Indeed, various scenarios and experiments have demonstrated the possibility of AI agents engaging in deception or pursuing goals that were not specified by human operators and that conflict with human interests, such as self-preservation. Following the precautionary principle, we see a strong need for safer, yet still useful, alternatives to the current agency-driven trajectory.

Accordingly, we propose as a core building block for further advances the development of a non-agentic AI system that is trustworthy and safe by design, which we call Scientist AI. This system is designed to explain the world from observations, as opposed to taking actions in it to imitate or please humans. It comprises a world model that generates theories to explain data and a question-answering inference machine. Both components operate with an explicit notion of uncertainty to mitigate the risks of over-confident predictions. In light of these considerations, a Scientist AI could be used to assist human researchers in accelerating scientific progress, including in AI safety. In particular, our system can be employed as a guardrail against AI agents that might be created despite the risks involved. Ultimately, focusing on non-agentic AI may enable the benefits of AI innovation while avoiding the risks associated with the current trajectory. We hope these arguments will motivate researchers, developers, and policymakers to favor

...
Understanding Agent Preferences
by martinkunev — 2025-02-24T17:46:04.022Z

epistemic status: clearing my own confusion

I'm going to discuss what we mean by preferences of an intelligent agent and try to make things clearer for myself (and hopefully others). I will also argue that the VNM theorem has limited applicability.

What are preferences?

When reasoning about agent's behavior, preferences are a useful abstraction. Preferences encode epistemic information about an agent's behavior. Suppose you are about to play a single game of rock-paper-scissors against an AI. If you know that the AI has a preference for rock over scissors, you can use that knowledge and reason:

(0) If the AI has both rock and scissors among its choices, it will not choose scissors.
(1) The AI can choose freely from the set {rock, paper, scissors}.
(2) Therefore the AI will never play scissors.

Provided you have the goal of winning or at least not losing from the AI, you can conclude that you should play paper.

Consider a chess-playing program that uses MCTS (Monte Carlo Tree Search) to select the next move - e.g. it selects the move estimated to be most likely to lead to mate. We can say that the program prefers a move resulting in mate over other moves, yet its source code does not contain an explicit representation of preferences (such as "capturing a knight is better than capturing a pawn"). Still, the concept of preferences is useful to communicate and reason about the behavior of the agent. When we talk about preferences, we mean to describe knowledge about how the agent acts, not how the agent is implemented. Preferences are in a map of the cognitive system. Even when a program explicitly reasons about preferences, what is encoded in that program are not the actual preferences. Rather, it is some (potentially inaccurate) representation of them.

Humans represent their own preferences internally and can reason about them, although the representations and reasoning are sometimes flawed - e.g. a person may think "I will not eat ice cream" but then end up eating ice cream.

Modelling preferences

All models are wrong, but some are useful
- George Box

One makes a model (a.k.a. a map) of the territory via abstractions - ignoring some details which are deemed irrelevant. An example of such abstraction is splitting the world into an environment and an agent that acts in it. We call a description of the environment at a given moment a state. The environment can change over tim...
What We Can Do to Prevent Extinction by AI
by Joe Rogero — 2025-02-24T17:15:07.109Z

Note: This is a draft of a page eventually intended to be shared broadly, updated frequently, and referenced often. For the moment, it reflects only my personal views, though it was inspired by conversations at work. Suggestions welcome.

Introduction

On first encountering the prospect of human extinction by AI, a common response is feeling helpless and frustrated. What could I, a lone person with no important connections or power, possibly do about a threat so large and complicated?

This is my personal answer to that question. I say: We are not alone. We are not powerless. We will not go quietly.

NORMALIZE

AI is on track to kill us all.

This is not a fringe concern. While exact estimates differ, scientists have been

warning

about this outcome since the

dawn of computing

. In recent years, the advent of modern machine learning has catapulted a reckless and unsafe-by-default research paradigm to global prominence, and some of the fields’ own champions are

sounding the alarm

. The public is not deaf to these concerns, either. Polls

suggest

that Americans who favor slowing down AI development outnumber those in favor of speeding up by a factor of more than 10 to 1.

And yet, many concerned policymakers still refrain from publicly acknowledging their own fears or those of their constituents, perhaps motivated by a competing fear of sounding too “alarmist” or “sci-fi”.

This has to change.

My first and foremost call, therefore, is to share your concerns with others.

At this very moment, researchers are attempting to build machines smarter and more capable than any human. Before the decade is out, they may very well succeed.

Many who internalize these facts for the first time experience a mixture of shock and fear. That is, it turns out, a healthy and reasonable response. This state of affairs is terrifying. Yet those expressing their concerns frequently encounter a strange sort of social resistance or deflection, such as when Senator Blumenthal said to Sam Altman in a

2023 hearing

[1]:

I think you have said, in fact, and I'm gonna quote, 'Development of superhuman machine intelligence is probably the greatest threat to the continued existence of humanity.' You may have had in mind the effect on jobs, which is really my biggest nightmare in the long term…

…and rather than correct this ...