trentmkelly/LessWrong-43k huggingface dataset:
System 2 Alignment
by Seth Herd — 2025-02-13T19:17:56.868Z

Submitted to Alignment Forum. Contains more technical jargon than usual.

Epistemic status: I'm pretty sure these are important questions, but I don't have firm conclusions. I'm hoping to inspire others to think in this direction. ~50% of the average value is probably in the Summary; another ~15% each in the next Overview and OpenAI's Deliberative Alignment sections. The second half is more detailed, and intended for those who want to think about aligning language model agents as AGI.

Summary:

Demand for LLM-based agents that can perform useful work may lead all the way to takeover-capable AGI that needs real alignment. Here I discuss alignment techniques that developers are likely to use for that type of AGI. Accurate gears-level models of likely AGI and alignment approaches can effectively give us more time to work on alignment.

Such agents will probably use chains of thought or System 2 cognition[1] extensively. Some System 2 human strategies seem likely to be adapted to improve capabilities, and minimize costs and risks for users and developers of LLM-powered agents. Those methods can be repurposed for alignment at low cost, so they probably will be.

I roughly divide such strategies/methods into review, deliberation, and thought management. I discuss these methods and possible variations in how they might be implemented. This analysis is intended as a starting point for more detailed work.

Review of plans and actions is the method I've thought most about. It could:

Intervene before any tool calls or goal/memory updates are allowed
Reference a human-written constitution
Call a fresh LLM instance for independent review
Check for alignment-altering changes to beliefs or subgoals
Scale with estimated importance of decisions for low alignment taxes
Work to some degree even without a faithful chain of thought

System 2 alignment could prevent alignment-altering belief changes or subgoal adoption. In this and other ways, these approaches could go beyond "aligning" the behavior of an LLM, and help with real alignment of the goals of an AGI agent with those of humanity (or at least a human[2]).

Depending on their implementation, such methods might be regarded by an agent as a meddling "thought editor", or as a valued conscience.

OpenAI recently announced deliberative alignment[3], after this post was partly written. I discuss its implementation. It is...
Murder plots are infohazards
by Chris Monteiro — 2025-02-13T19:15:09.749Z

Hi all

I've been hanging around the rationalist-sphere for many years now, mostly writing about transhumanism, until things started to change in 2016 after my Wikipedia writing habit shifted from writing up cybercrime topics, through to actively debunking the numerous dark web urban legends.

After breaking into what I believe to be the most successful ever fake murder for hire website ever created on the dark web, I was able to capture information about people trying to kill people all around the world, often paying tens of thousands of dollars in Bitcoin in the process.

My attempts during this period to take my information to the authorities were mostly unsuccessful, when in late 2016 on of the site a user took matters into his own hands, after paying $15,000 for a hit that never happened, killed his wife himself.

Due to my overt battle with the site administrator over this period, he grew frustrated with me, and when sending death threats and burning cars to dissuade me failed to work, it was ultimately commissioning blogs implicating me running the site that means following the murder, the police broke down my door and arrested me for allegedly running the site. I handed over my info and they obviously let me go.

In 2018 participating in a CBS documentary about the 'dark web murder', I accessed the site once again, instigating a handful of arrests in the US and Singapore, and leading to a short-lived relationship with Homeland Security Investigations and precipitating further arrests in the US.

From 2020 until recently I worked with a UK podcast company, with me funneling off the most serious murder plots and they would try and send them to local journalists and law enforcement agencies around the world, with the most successes being in the US. You can listen to 'Kill List' online.

As of 2025 now, this has lead to almost 40 convictions around the world, further more arrests and likely more in the pipeline.

With the podcast production ended, I am exhausted from these efforts. I have scraped sites, triaged records, verified addresses, cross referenced IDs, reconstructed bitcoin payments, automated reports, worked with journalists, many law enforcement agencies, private detectices and incurred both legal fees and the personal and professional opportunity-cost for the significant time and focus I have spent on this over almost 9 years now.

I have about 800 names left ...
Sparse Autoencoder Feature Ablation for Unlearning
by aludert — 2025-02-13T19:13:48.388Z

TLDR:

This post is derived from my end of course project for the BlueDot AI Safety Fundamentals course. Consider applying here.

We evaluate the use of sparse autoencoder (SAE) feature ablation as a mechanism for unlearning Harry Potter related knowledge in Gemma-2-2b. We evaluate a non-ablated model and models with a single Harry Potter monosemantic feature ablated in an early layer, middle and last layer against a set of Harry Potter evaluation prompts. We also measure the general performance of these models against general LLM performance benchmarks to estimate general loss of capacity after ablation.

Key Results:

Ablating a single monosemantic Harry Potter related feature in the last layer of Gemma-2-2b produces a significant degradation in the performance of the model against the Who is Harry Potter benchmark while producing the least amount of model performance loss as measured by the OWT and PILE loss benchmarks.
Ablating any single monosemantic Harry Potter related feature produced significant model degradation with ~95% loss of performance against the Who is Harry Potter benchmark
The process of generating a large dataset of robust Harry Potter familiarity prompts like the ones proposed in the Eldan and Russinovich (2023) paper that initially proposed the Who is Harry Potter familiarity benchmark was an arduous one, so a repository of over 1000 prompts generated for this study has been made publicly in this GitHub repository.

As possible next steps we would like to:

Exhaustively map unlearning and general capability loss as a function of ablation feature layer depth to corroborate that unlearning strength is in fact independent from model layer depth
Reproduce this approach with LLama and it’s pretrained SAEs in LLama Scope to produce a more accurate comparison with the results from Eldan and Russinovich (2023)

Introduction

Unlearning, understood as selectively removing specific knowledge, capabilities or behaviours from Large Language Models has applications for protecting copyrighted intellectual property, reducing or removing the ability to produce harmful responses as well as reducing hallucinations in LLMs (see Yao et al 2024 for a discussion of these applications). Eldan and Russinovich in their 2023 paper propose a fine tuning approach to unlearn Harry Potter related knowledge in their Who's Harry Potter? Approxim...
What is it to solve the alignment problem?
by Joe Carlsmith — 2025-02-13T18:42:07.215Z
Link: https://joecarlsmith.substack.com/p/what-is-it-to-solve-the-alignment

(Audio version here (read by the author), or search for "Joe Carlsmith Audio" on your podcast app.

This is the first essay in a series that I’m calling “How do we solve the alignment problem?”. See this introduction for a summary of the essays that have been released thus far, and for a bit more about the series as a whole.

Some of this essay is a revised/shortened/reconsidered version of some of the content from these notes on the topic, which I posted on LessWrong and the EA forum -- though not on my website or substack -- last fall.)

1. Introduction

This series is about solving the alignment problem – or at least, a certain version of it – for full-blown superintelligences. So I want to start by defining the version of the alignment problem I have in mind, and what it would be to solve it.[1] And I’ll say a bit, too, about how this version of the problem fits into the bigger picture.

In brief: my definition draws on the interplay between two goals:

Safety: Avoiding what I’ll call a “loss of control” scenario.[2]

Benefits: Getting access to the main benefits of superintelligent AI.

The core alignment problem, as I see it, is that going for Benefits can lead to failure on Safety. I’ll say a bit more about why below, and in the next essay.[3]

I’ll say you failed on the alignment problem if you failed on Safety. And I’ll say that you solved the alignment problem if you successfully achieve both Safety and Benefits while also doing the following:

Creation: Building superintelligent AI agents, and

Elicitation: Becoming able to safely elicit their main beneficial capabilities.[4]

Solving the problem in this sense, though, isn’t the only alternative to failure. In particular: there are ways of getting safe access to the benefits of superintelligent AI without building superintelligent AI agents, and/or without becoming able to safely elicit their main beneficial capabilities. And you can give up on benefits for the sake of safety, too. I think all these possibilities should be on the table. Below I offer a taxonomy that includes them.

Importantly: my definition of “solving” the alignment problem is weaker than some alternatives. In particular: it doesn’t mean that you’re in a position to safely scale to yet-more-superintelligent AIs (“safety at all scales”), or to scale competitively with extremely incautious actors (“fully competi...
Self-dialogue: Do behaviorist rewards make scheming AGIs?
by Steven Byrnes — 2025-02-13T18:39:37.770Z

Introduction

I have long felt confused about the question of whether brain-like AGI would be likely to scheme, given behaviorist rewards. …Pause to explain jargon:

“Brain-like AGI” means Artificial General Intelligence—AI that does impressive things like inventing technologies and executing complex projects—that works via similar algorithmic techniques that the human brain uses to do those same types of impressive things. See Intro Series §1.3.2.

I claim that brain-like AGI is a not-yet-invented variation on Model-Based Reinforcement Learning (RL), for reasons briefly summarized in Valence series §1.2.

“Scheme” means pretend to be cooperative and docile, while secretly looking for opportunities to escape control and/or perform egregiously bad and dangerous actions like AGI world takeover.

If the AGI never finds such opportunities, and thus always acts cooperatively, then that’s great news, but it still counts as “scheming”.

“Behaviorist rewards” is a term I made up for an RL reward function which depends only on externally-visible actions, behaviors, and/or the state of the world.

Maybe you’re thinking: what possible RL reward function is not behaviorist?? Well, non-behaviorist reward functions are pretty rare in the textbook RL literature, although they do exist—one example is “curiosity” / “novelty” rewards.[1] But I think they’re centrally important in the RL system built into our human brains. In particular, I think that innate drives related to human sociality, morality, norm-following, and self-image are not behaviorist, but rather involve rudimentary neural net interpretability techniques, serving as inputs to the RL reward function. See Neuroscience of human social instincts: a sketch for details, and Intro series §9.6 for a more explicit discussion of why interpretability is involved.

(…It’s not fancy interpretability! It’s more-or-less akin to a set of trained classifiers on neural net activation states, or if you prefer, to a set of extra neural net output heads which are trained from various different streams of ground truth.)

So again, the question is: will brain-like AGIs scheme, if trained by a behaviorist RL reward function? I had strong intuitions pointing in both directions on this question. So I sat down and wrote both sides of a dialogue—the optimist on one shoulder talking to the pessimi...
How do we solve the alignment problem?
by Joe Carlsmith — 2025-02-13T18:27:27.712Z
Link: https://joecarlsmith.substack.com/p/how-do-we-solve-the-alignment-problem

(Audio version here (read by the author), or search for "Joe Carlsmith Audio" on your podcast app.)

We want the benefits that superintelligent AI agents could create. And some people are trying hard to build such agents. I expect efforts like this to succeed – and maybe, very soon.

But superintelligent AI agents might also be difficult to control. They are, to us, as adults to children, except much more so. In the same direction, relative to us, as advanced aliens; as demi-gods; as humans relative to ants. If such agents “go rogue” – if they start ignoring human instructions, resisting correction or shut-down, trying to escape from their operating environment, seeking unauthorized resources and other forms of power, etc – we might not be able to stop them.

Worse, because power/resources/freedom/survival etc are useful for many goals, superintelligent agents with a variety of different motivations would plausibly have incentives to go rogue in this way, suggesting that problems with AI motivations could easily lead to such behavior. And if this behavior goes uncorrected at scale, humans might lose control over civilization entirely – permanently, involuntarily, maybe violently. Superintelligent AI agents, acting on their own, would be the dominant actors on the planet. Humans would be sidelined, or dead.

Getting safe access to the benefits of superintelligence requires avoiding this kind of outcome. And this despite incentives among human actors to build more and more capable and agentic systems (and including: to do so faster than someone else), and despite the variety of actors that might proceed unsafely. Call this the “alignment problem.”

I’ve written, before, about why I’m worried about this problem.[1] But I’ve said much less about how we might solve it. In this series of essays, I try to say more.[2] Here’s a summary of the essays I’ve released thus far:

In the first essay, “What is it to solve the alignment problem?”, I define solving the alignment problem as: building full-blown superintelligent AI agents, and becoming able to safely elicit their main beneficial capabilities, while avoiding the sort of “loss of control” scenario discussed above. I also define some alternatives to both solving the problem and failing on the problem – namely, what I call “avoiding” the problem (i.e., not building superintelligent AI

...
Ambiguous out-of-distribution generalization on an algorithmic task
by Wilson Wu — 2025-02-13T18:24:36.160Z

Introduction

It's now well known that simple neural network models often "grok" algorithmic tasks. That is, when trained for many epochs on a subset of the full input space, the model quickly attains perfect train accuracy and then, much later, near-perfect test accuracy. In the former phase, the model memorizes the training set; in the latter, it generalizes out-of-distribution to the test set.

In the algorithmic grokking literature, there is typically exactly one natural generalization from the training set to the test set. What if, however, the training set were instead under-specified in such a way that there were multiple possible generalizations? Would the model grok at all? If so, which of the generalizing solutions would it choose? If the model followed Occam's razor, it would choose the simplest solution -- but what does "simplest" mean here? We explore these questions for the task of computing a finite group operation.

Setup

This section assumes some basic familiarity with group theory and can be skipped. The point is just that each model is trained on the intersection of two datasets G1 and G2; the intersection of the two test sets always has size ≥50%.

In existing work on grokking finite group operations (e.g. Chughtai et al.), a one-hidden-layer MLP model with two inputs f:RG×RG→RG is trained on the operations of a finite group G. The model takes as input a pair of elements x,y∈G, encoded as one-hot vectors in RG, and is expected to output logits maximized at the product xy∈G. Thus, the input space is the set of all pairs G2, and the model is evaluated on its accuracy over the entire test space:

acc(f)=|G|−2∑x,y∈G1{argmaxz∈Gf(x,y)z=xy}.

Previous work finds that training on an iid subsample of 40% of the total input space (so the training set has 0.4|G|2 points) is enough to grok the full multiplication table for various choices of G. (The most well-studied choice is the symmetric group G=S5.) In our setup, we leave everything the same except for the choice of training set. We now choose two groups G1 and G2 such that |G1|=|G2|. Thinking of the two groups as two operations ∗1 and ∗2 on the same underlying set of elements, our ambiguous training set is the set of all pairs of elements such that the two group operations agree:...
Teaching AI to reason: this year's most important story
by Benjamin_Todd — 2025-02-13T17:40:02.869Z
Link: https://benjamintodd.substack.com/p/teaching-ai-to-reason-this-years

This doesn't contain much new to LW readers. I wrote it to try to explain what's going on to a broader audience – I'm posting in case people find it helpful for that. Feedback welcome.

Most people think of AI as a pattern-matching chatbot – good at writing emails, terrible at real thinking.

They've missed something huge.

In 2024, while many declared AI was reaching a plateau, it was actually entering a new paradigm: learning to reason using reinforcement learning.

This approach isn’t limited by data, so could deliver beyond-human capabilities in coding and scientific reasoning within two years.

Here's a simple introduction to how it works, and why it's the most important development that most people have missed.

The new paradigm: reinforcement learning

People sometimes say “chatGPT is just next token prediction on the internet”. But that’s never been quite true.

Raw next token prediction produces outputs that are regularly crazy.

GPT only became useful with the addition of what’s called “reinforcement learning from human feedback” (RLHF):

The model produces outputs
Humans rate those outputs for helpfulness
The model is adjusted in a way expected to get a higher rating

A model that’s under RLHF hasn’t been trained only to predict next tokens, it’s been trained to produce whatever output is most helpful to human raters.

Think of the initial large language model (LLM) as containing a foundation of knowledge and concepts. Reinforcement learning is what enables that structure to be turned to a specific end.

Now AI companies are using reinforcement learning in a powerful new way – training models to reason step-by-step:

Show the model a problem like a math puzzle.
Ask it to produce a chain of reasoning to solve the problem (“chain of thought”).1
If the answer is correct, adjust the model to be more like that (“reinforcement”).2
Repeat thousands of times.

Before 2023 this didn’t seem to work. If each step of reasoning is too unreliable, then the chains quickly go wrong. Without getting close to correct answers, there was nothing to reinforce.

But now it’s started to work very well…

Reasoning models breakthroughs

Consider GQPA –– a set of new scientific questions designed so that people with PhDs in the field can mostly answer them, but non-experts can’t, even with 30min access to Google. It contains questions like this:

I did a masters level course in theoretical physics, and I have ...
AI #103: Show Me the Money
by Zvi — 2025-02-13T15:20:07.057Z

The main event this week was the disastrous Paris AI Anti-Safety Summit. Not only did we not build upon the promise of the Bletchley and Seoul Summits, the French and Americans did their best to actively destroy what hope remained, transforming the event into a push for a mix of nationalist jingoism, accelerationism and anarchism. It’s vital and also difficult not to panic or despair, but it doesn’t look good.

Another major twist was that Elon Musk made a $97 billion bid for OpenAI’s nonprofit arm and its profit and control interests in OpenAI’s for-profit arm. This is a serious complication for Sam Altman’s attempt to buy those same assets for $40 billion, in what I’ve described as potentially the largest theft in human history.

I’ll be dealing with that tomorrow, along with two other developments in my ongoing OpenAI series The Mask Comes Off. In Altman’s Three Observations, he gives what can best be described as a cartoon villain speech about how AI will only be a good thing, and how he knows doing this and the risks involved won’t be popular but he’s going to do it anyway. Then, we look at the claim from the Summit, by OpenAI, that AI will complement rather than substitute for humans because that is a ‘design decision.’ Which will reveal, in yet another way, the extent to which there is no plan.

OpenAI also plans to release ‘GPT-4.5’ in a matter of weeks, which is mostly the same timeline as the full o3, followed by the promised ‘GPT-5’ within months that Altman says is smarter than he is. It’s a bold strategy, Cotton.

To their credit, OpenAI also released a new version of their model spec, with major changes throughout and a completely new structure. I’m going to need time to actually look into it in detail to know what I think about it.

In the meantime, what else is happening?

Table of Contents

Language Models Offer Mundane Utility. Don’t go there. We tried to warn you.

Language Models Don’t Offer Mundane Utility. No episodic memory?

We’re in Deep Research. Reactions still very positive. Pro to get 10 uses/month.

Huh, Upgrades. GPT-4.5, GPT-5, Grok 3, all coming quite soon. And PDFs in o3.

Seeking Deeply. And r1 begat s1.

Smooth Operator. Use it directly with Google Drive or LinkedIn or similar.

They Took Our Jobs. The California Faculty Association vows to fight back.

Maxwell Tabarrok Responds on Future Wages. The crux is what you would expect

...