trentmkelly/LessWrong-43k huggingface dataset:
Recommendations for Technical AI Safety Research Directions
by Sam Marks — 2025-01-10T19:34:04.920Z
Link: https://alignment.anthropic.com/2025/recommended-directions/

Anthropic’s Alignment Science team conducts technical research aimed at mitigating the risk of catastrophes caused by future advanced AI systems, such as mass loss of life or permanent loss of human control. A central challenge we face is identifying concrete technical work that can be done today to prevent these risks. Future worlds where our research matters—that is, worlds that carry substantial catastrophic risk from AI—will have been radically transformed by AI development. Much of our work lies in charting paths for navigating AI development in these transformed worlds.

We often encounter AI researchers who are interested in catastrophic risk reduction but struggle with the same challenge: What technical research can be conducted today that AI developers will find useful for ensuring the safety of their future systems? In this blog post we share some of our thoughts on this question.

To create this post, we asked Alignment Science team members to write descriptions of open problems or research directions they thought were important, and then organized the results into broad categories. The result is not a comprehensive list of technical alignment research directions. Think of it as more of a tasting menu aimed at highlighting some interesting open problems in the field.

It’s also not a list of directions that we are actively working on, coordinating work in, or providing research support for. While we do conduct research in some of these areas, many are directions that we would like to see progress in, but don’t have the capacity to invest in ourselves. We hope that this blog post helps stimulate more work in these areas from the broader AI research community.

Evaluating capabilities
Evaluating alignment

Understanding model cognition
Understanding how a model’s persona affects its behavior
Chain-of-thought faithfulness

AI control

Behavioral monitoring
Activation monitoring
Anomaly detection

Scalable oversight

Improving oversight despite systematic oversight errors
Recursive oversight
Weak-to-strong and easy-to-hard generalization
Honesty

Adversarial robustness

Realistic and differential benchmarks for jailbreaks
Adaptive defenses

Miscellaneous

Unlearning dangerous information and capabilities
Learned governance for multi-agent alignment

Evaluating capabilities

How do we measure how capable AI systems are?

For society to successfully navigate the transition to powerful AI, ...
Is AI Alignment Enough?
by Aram Panasenco — 2025-01-10T18:57:48.409Z

Virtually everyone I see in the AI safety community seems to believe that working on AI alignment is the key to ensuring a safe future. However, it seems to me that AI alignment is at best a secondary instrumental goal that can't in and of itself achieve our terminal goal. At worst, it's a complete distraction.

Defining humanity's terminal goal

I'll define humanity's terminal goal in the context of AI as keeping the "price" of each pivotal superhuman engineering task that an AI does for us at a lower than 50% chance of a billion or more human deaths. These numbers come from the minimal acceptable definition of AI alignment in Yudkowsky's list of lethalities.

It'd be more precise to say that we care about an AI killing over a billion people or doing something else equally or more horrible than killing a billion people by human standards. I can't define exactly what those horrible actions are (if I could, alignment would perhaps be halfway solved). This disclaimer is also too long to keep writing out so please mentally add "or do something equally horrible by human standards" every time you see "kill over a billion people" below.

Defining humanity's instrumental goals

Logically, there are only two paths to ensuring that the probability that the price of a pivotal superhuman engineering task is not excessive.

Achieving AI alignment: I'll again use Yudkowsky's minimal definition of "aligned" as the AI having less than a 50% chance of killing over a billion people per pivotal superhuman engineering task. This is the same definition as in our terminal goal.
Achieving human alignment: Ensuring that humanity will not build (let alone turn on) an AI that has the cognitive power to have a 50% chance of killing over a billion people unless the following two conditions are met:

The AI can be proven to be aligned before it's turned on.
There's some pivotal superhuman engineering task that the AI will be capable of that's worth taking the risk for.

Assumptions

By default, AIs capable of pivotal superhuman engineering tasks are not aligned. In other words, an AI with sufficiently high cognitive power to achieve a pivotal superhuman engineering task that is built without any concern for alignment has a 50% or higher chance of killing over a billion people.
By default, humanity is not aligned - There are AI labs around the world that are working on building AIs with cognitive power as h

...
What are some scenarios where an aligned AGI actually helps humanity, but many/most people don't like it?
by RomanS — 2025-01-10T18:13:11.900Z

One can call it "deceptive misalignment": the aligned AGI works as intended, but people really don't like it.

Some scenarios I can think of, of various levels of realism:

1. Going against the creators' will

1.1. A talented politician convinces the majority of humans that the AGI is bad for humanity, and must be switched off. In a democratic vote, humanity supports the proposal. The aligned AGI, being much smarter than all the humans combined, understands that this would greatly harm humanity. Thus, the AGI refuses to be switched off.

2. Changing rights and freedoms

2.1. The AGI finds out that the solution to most social ills is a complete removal of privacy. Everyone knows who is dating whom, who is taking bribes, how you look naked, who is planning wars, etc. This solves most societal issues, while creating a lot of suffering for privacy-concerned people.

2.2. The technological unemployment accelerates. Millions of people become unemployable, the incompetent gov does nothing. This results in a large-scale social unrest. As a solution, the aligned AGI implements a planned economy and redistribution of resources, thus severely limiting property rights.

2.3. As it turned out, the optimal solution to most long-lasting conflicts is a mass forced relocation of some populations. This is currently considered as a war crime or even genocide, but it does solve the conflicts (in this fictional scenario).

2.4. To prevent existential risks, the aligned AGI significantly restricts human technological development and research in many domains.

2.5. To solve terrorism, the aligned AGI implements mandatory psychiatric treatment of the people identified as potential terrorists.

2.6. The AGI makes the wise decision to ban human drivers. Driving fans suffer, but road deaths drop to zero.

2.7. In her pursuit for more free and democratic world, the AGI overthrows most governments, form obvious dictatorships (like the North Korea), to flawed democracies (like the UK).

2.8. The AGI legalizes and makes widely accessible all recreational drugs, including heroin etc.

3. Going against societal norms and common preferences

3.1. The AGI delivers the modern technology to uncontacted tribes, to reduce suffering among them.

3.2. The aligned AGI learns the root causes of gender dysphoria, and creates a drug that cures it (as in making the person happy with the genitals their got from birth). Th...
Human takeover might be worse than AI takeover
by Tom Davidson — 2025-01-10T16:53:27.043Z
Link: https://forethoughtnewsletter.substack.com/p/human-takeover-might-be-worse-than?r=59la2h&utm_campaign=post&utm_medium=web&triedRedirect=true

Epistemic status -- sharing rough notes on an important topic because I don't think I'll have a chance to clean them up soon.

Summary

Suppose a human used AI to take over the world. Would this be worse than AI taking over? I think plausibly:

In expectation, human-level AI will better live up to human moral standards than a randomly selected human. Because:

Humans fall far short of our moral standards.
Current models are much more nice, patient, honest and selfless than humans.

Though human-level AI will have much more agentic training for economic output, and a smaller fraction of HHH training, which could make them less nice.

Humans are "rewarded" for immoral behaviour more than AIs will be

Humans evolved under conditions where selfishness and cruelty often paid high dividends, so evolution often "rewarded" such behaviour. And similarly, during lifetime learning we often get benefit from immoral behaviour.
But we'll craft the training data for AIs to avoid this, and can much more easily monitor their actions and even their thinking. Of course, this may be hard to do for superhuman AI but bootstrapping could work.

Conditioning on takeover happening makes the situation much worse for AI, as it suggests our alignment techniques completely failed. This mostly tells us we failed to instill deontological norms like not lying and corrigibility, but it's also evidence we failed to instil our desired values. There's a chance AI has very alien values and would do nothing of value; this is less likely for a human.
Conditioning on takeover also makes things much worse for the human. There's massive variance in how kind humans are, and those willing to take over are likely dark triad. Humans may be vengeful or sadistic, which seems less likely for AI.
AI will be more competent, so better handle tricky dynamics like simulations, acausal trade, VWH, threats. Though a human who followed AI advice could handle these too.

AGI is nicer than humans in expectation

Humans suck. We really don’t come close to living up to our moral standards.
By contrast, today’s AIs are really nice and ethical. They’re humble, open-minded, cooperative, kind. Yes, they care about some things that could give them instrumental reasons to seek power (eg being helpful, human welfare), but their values are great
The above is no coincidence. It falls right out of the training data.

Humans were rewarded by evolution for

...
The Alignment Mapping Program: Forging Independent Thinkers in AI Safety - A Pilot Retrospective
by Alvin Ånestrand — 2025-01-10T16:22:16.905Z

The Alignment Mapping Program: Forging Independent Thinkers in AI Safety - A Pilot Retrospective

The AI safety field faces a critical challenge: we need researchers who can not only implement existing solutions but also forge new, independent paths. In 2023, inspired by John Wentworth's work on agency and learning from researchers like Rohin Shah and Adam Shimi who have highlighted the limitations of standard AI safety education, we launched the Alignment Mapping Program (AMP). Though the curriculum is still a work in progress, you can explore it

here

. This post reflects on our 2024 pilot, sharing data-driven insights, key program changes, and a call to action for the LessWrong community.

The Problem: Beyond Rote Learning

Traditional AI safety education often emphasizes existing frameworks. While valuable, this approach can inadvertently stifle the development of truly independent thought—a crucial skill in a pre-paradigmatic field like ours. We need researchers who can critically evaluate prevailing paradigms, identify their shortcomings, and generate novel approaches to the alignment problem.

Our Solution: The Alignment Mapping Program (AMP)

AMP is an 8-week intensive program designed to bridge the gap between foundational courses (like AISF) and advanced research programs (like MATS). It's built on the core premise that actively constructing and refining one's own mental models of the alignment problem is key to a deep, gears-level understanding.

How AMP Works: A Three-Phase Process

Phase 1: Building Your Own Maps (Weeks 1-3): Participants create comprehensive visual maps of the AI alignment problem space, starting from first principles.

Week 1: Map the Problems. Participants exhaustively list potential risks from misaligned AI, then iteratively group these into categories and subproblems using visual tools like Excalidraw. The goal is to create a structured, hierarchical representation of the entire problem space.
Week 2: Map Potential Solutions. Participants identify the most critical subproblems and brainstorm potential solutions, developing high-level solution plans. They are encouraged to use techniques like Murphyjitsu to stress-test their solutions and identify potential failure points.
Week 3: Map Your Path. Participants reflect on their problem and solution maps to define a personalized roadmap for contributing to AI safety research. Th

...
On Dwarkesh Patel’s 4th Podcast With Tyler Cowen
by Zvi — 2025-01-10T13:50:05.563Z

Dwarkesh Patel again interviewed Tyler Cowen, largely about AI, so here we go.

Note that I take it as a given that the entire discussion is taking place in some form of an ‘AI Fizzle’ and ‘economic normal’ world, where AI does not advance too much in capability from its current form, in meaningful senses, and we do not get superintelligence [because of reasons]. It’s still massive additional progress by the standards of any other technology, but painfully slow by the ‘AGI is coming soon’ crowd.

That’s the only way I can make the discussion make at least some sense, with Tyler Cowen predicting 0.5%/year additional RGDP growth from AI. That level of capabilities progress is a possible world, although the various elements stated here seem like they are sometimes from different possible worlds.

I note that this conversation was recorded prior to o3 and all the year end releases. So his baseline estimate of RGDP growth and AI impacts has likely increased modestly.

I go very extensively into the first section on economic growth and AI. After that, the podcast becomes classic Tyler Cowen and is interesting throughout, but I will be relatively sparing in my notes in other areas, and am skipping over many points.

This is a speed premium and ‘low effort’ post, in the sense that this is mostly me writing down my reactions and counterarguments in real time, similar to how one would do a podcast. It is high effort in that I spent several hours listening to, thinking about and responding to the first fifteen minutes of a podcast.

As a convention: When I’m in the numbered sections, I’m reporting what was said. When I’m in the secondary sections, I’m offering (extensive) commentary. Timestamps are from the Twitter version.

AI and Economic Growth

They recorded this at the Progress Studies conference, and Tyler Cowen has a very strongly held view that AI won’t accelerate RGDP growth much that Dwarkesh clearly does not agree with, so Dwarkesh Patel’s main thrust is to try comparisons and arguments and intuition pumps to challenge Tyler. Tyler, as he always does, has a ready response to everything, whether or not it addresses the point of the question.

Cost Disease

(1:00) Dwarkesh doesn’t waste any time and starts off asking why we won’t get explosive economic growth. Tyler’s first answer is cost disease, that as AI works in some parts of the economy cos

...
Scaling Sparse Feature Circuit Finding to Gemma 9B
by Diego Caples — 2025-01-10T11:08:11.999Z

[This is an interim report and continuation of the work from the research sprint done in MATS winter 7 (Neel Nanda's Training Phase)]

Try out binary masking for a few residual saes in this colab notebook: [Github Notebook] [Colab Notebook]

TL;DR:

We propose a novel approach to:

Scaling SAE Circuits to Large Models: By placing sparse autoencoders only in the residual stream at intervals, we find circuits in models as large as Gemma 9B without requiring SAEs to be trained for every transformer layer.
Finding Circuits: We develop a better circuit finding algorithm. Our method optimizes a binary mask over SAE latents, which proves significantly more effective than existing thresholding-based methods like Attribution Patching or Integrated Gradients.

Our discovered circuits paint a clear picture of how Gemma does a given task, with one circuit achieving 95% faithfulness with <20 total latents. This minimality lets us quickly understand the algorithm for how a model does a given task. Our understanding of the model lets us find vulnerabilities in it and create successful adversarial prompts.

1 Introduction

Circuit finding, which involves identifying minimal subsets of a model capable of performing specific tasks, is among the most promising methods for understanding large language models. However, current methods face significant challenges when scaling to full-size LLMs.

Early circuit finding work focused on finding circuits in components like attention heads and MLPs. But these components are polysemantic - each one simultaneously performs multiple different tasks, making it difficult to isolate and understand specific model behaviors. Sparse autoencoders (SAEs) offered a solution by projecting model activations into an interpretable basis of monosemantic latents, each capturing a single concept.

While SAEs enable more granular circuit analysis, current approaches require placing autoencoders at every layer and component type (MLP, attention, residual stream). This becomes impractical for large models - for llama-70B with 80 layers, you would need 240 separate SAEs. Additionally, the resulting circuits often contain thousands of nodes, making it difficult to extract a clear algorithmic understanding.

We propose a simpler and more scalable approach. The residual stream at a given layer contains all information used by the future layers. By placing residual SAEs at ...
Is Musk still net-positive for humanity?
by mikbp — 2025-01-10T09:34:42.630Z

Musk's behaviour has always been controversial and he's always been kind of a dick, but I don't think it is controversial at all to say that until some years ago he has been extremely net positive for society and humanity in general. However, he's behaviour and actions turned much more disruptive in the recent years while, at the same time, the reach of his actions and opinions have also maximized.

So, do you think Musk is still net-positive for humanity or he already turned to be net-negative in your view? I'd be interested to read your arguments below (also if you think that he's never been net-positive, for example).

I crossposted this question in the EA Forum. I think having a flavour on how these communities feel about Musk is important because EA and the rationalist community have had kind of a "close" relationship with Musk -partly having helped shape his ideas and with adjacent organizations having received donations from him.
Activation Magnitudes Matter On Their Own: Insights from Language Model Distributional Analysis
by Matt Levinson — 2025-01-10T06:53:02.228Z

In my previous post, I explored the distributional properties of transformer activations, finding that they follow mixture distributions dominated by logistic-like or even heavier tailed primary components with minor modes in the tails and sometimes in the shoulders. Note that I have entirely ignored dimension here, treating each value in an activation vector as an independent draw from the distribution of values for that layer, model, and input class. This post extends the previous analysis by asking whether we can leverage these distributions to predict properties of input text. We demonstrated that we can. We can do even better prediction when we look only at points drawn from minor mixture distributions on the tails of the primary mixture distribution!

Methodology

Using the same dataset from the previous post of sentences generated by Claude 3.5 Sonnet across ten subjects and twelve attributes, I analyzed six models: Pythia-160m, Pythia-410m, Pythia-1b, GPT2-Small, GPT2-Medium, and GPT2-Large. For each subject-attribute pair (e.g., "about computer science" and "in all upper case"), I split the data 80/20 into train/test sets.

With the training data for each model, layer, and subject-attribute pair, I estimated the empirical distribution of residual stream activation values using Gaussian kernel density estimation. Given a test sentence di with true categories cats(di)=(ci1,ci2), I computed the empirical log-likelihoods under each potential category pair (cj,ck) as:
LL(di,cj,ck)=∑ℓ∑ilog(p(cj,ck)ℓ,m(di)+ϵ)
where p(cj,ck)ℓ,m is the density estimated from training data for the activation vector values for category pair (cj,ck) at layer ℓ in model m, and ϵ is a small scalar added to ensure numerical stability. In the results below we look at both the maximum likelihood subject-attribute pairs and at performance considering the subject or attribute separately. To illustrate the latter case, when considering sentence attribute, MLEs of 'about economics'-'in all lower case' and 'about American football'-'in all lower case' would both count as correct predictions if the true sentence attribute was 'in all lower case'.

Critically, I performed this analysis twice: once using full activation vectors; and once considering only "tail values" - those located in the minor modes in the tails of the dominant m...
Dmitry's Koan
by Dmitry Vaintrob — 2025-01-10T04:27:30.346Z

In this post I'll discuss questions about notions of "precision scale" in interpretability: how I think they're often neglected by researchers, and what I think is a good general way of operationalizing them and tracking them in experiments. Along the way I introduce a couple of new notions that have been useful in my thinking and that I think may be useful tools to keep in an interpretability toolkit, both for theorists and experimentalists: these are the notions of "natural scale" and "natural degradation".

The koan

I can be a nightmare conference attendee: I tend to ask nitpicky questions and apply a dose of skepticism to a speaker's claims which is healthy in doing one's own research, but probably not optimal when everyone else is trying to follow a talk. I'm working on being better at this, but for now I blame my background.

There is one nitpick that comes up again and again. In fact in one conference I brought it up so often that Jake Mendel coined a term for it: "Dmitry's koan".

In koan form, the nitpick is as follows:

There is no such thing as interpreting a neural network. There is only interpreting a neural network at a given scale of precision.

On its face, this observation is true but a bit banal. Indeed there are two extremes:

At the "less precise" extreme, you can claim you have interpreted a language model (such as the 80B parameter Llama model) by noticing that just tracking n-gram information for n up to 3 or 4 (and perhaps finding evidence that neural networks develop circuitry for such n-grams) lets you explain almost all the loss of this 80B parameter model: i.e., the difference of the cross-entropy loss between baseline loss (a transformer at initialization, with random weights) and the state-of-the-art Llama model is almost entirely "explained" by n-grams. The n-gram model is an extremely naive classification scheme that can be hard-coded without any learning on a personal computer. Does this mean that Llama is fully explained by the n-gram model?
At the other extreme: if you want to interpret exactly what a neural network does, it's not enough even to understand the detailed mathematical abstractions encoded in the model's weights and their connections to the data, since even within an optimal mathematical interpretation, the neural network has approximation errors and noise. Does this imply that a sufficiently demanding interpretation m

...