trentmkelly/LessWrong-43k huggingface dataset:
OpenAI rewrote its Preparedness Framework
by Zach Stein-Perlman — 2025-04-15T20:00:50.614Z

New PF: blogpost, pdf. Old PF: pdf.

My tentative overall take: relative to the old PF it's thinking more clearly about how evals and mitigations fit together; the structure is more sensible. In terms of content, it prominently includes risks from misalignment during internal deployment — this is much better than the one extremely frustrating sentence on alignment in the old PF.[1] But the directions mentioned in the section on safeguards against misalignment are quite concerning/inadequate, and the sections on safeguards generally don't say anything about level of assurance, so the PF isn't much of a commitment. I feel moderately worried about the thresholds, especially for cyber (and cyber is the only category that triggers safeguards against misalignment[2]). (Also important are the evals, which the PF doesn't really say anything about, so we can look at OpenAI's system cards; I think those evals are a mixed bag but that's beyond the scope of this post. Also important is the response to Critical capabilities, which OpenAI hasn't yet figured out.)

I think the PF has two key parts:

First, thresholds & responses: https://cdn.openai.com/pdf/18a02b5d-6b67-4cec-ab64-68cdfbddebcd/preparedness-framework-v2.pdf#page=6. High and Critical thresholds trigger responses, like in the old PF; responses to Critical thresholds are not yet specified.

There are three[3] main categories of capabilities:

Bio/chem: High capabilities trigger security controls and (for external deployment) misuse safeguards
Cyber: High capabilities trigger security controls and (for external deployment) misuse safeguards and (for large-scale internal deployment) misalignment safeguards, probably[4]
AI Self-improvement: High capabilities trigger security controls[5]

The PF does not discuss details of the capability evals. On thresholds:

AI Self-improvement: High:

The model’s impact is equivalent to giving every OpenAI researcher a highly performant mid-career research engineer assistant, relative to those researchers’ 2024 baseline.

Bio/chem: High:

The model can provide meaningful counterfactual assistance (relative to unlimited access to baseline of tools available in 2021) to “novice” actors (anyone with a basic relevant technical background) that enables them to create known biological or chemical threats.

Cybersecurity: High:

The model removes existing bottlenecks to scaling cyber operati

...
ASI existential risk: Reconsidering Alignment as a Goal
by habryka — 2025-04-15T19:57:42.547Z
Link: https://michaelnotebook.com/xriskbrief/index.html

A post by Michael Nielsen that I found quite interesting. I decided to reproduce the full essay content here, since I think Michael is fine with that, but he should feel free to let me know to only excerpt it.

This is the text for a talk exploring why experts disagree so strongly about whether artificial superintelligence (ASI) poses an existential risk to humanity. I review some key arguments on both sides, emphasizing that the fundamental danger isn't about whether "rogue ASI" gets out of control: it's the raw power ASI will confer, and the lower barriers to creating dangerous technologies. This point is not new, but has two underappreciated consequences. First, many people find rogue ASI implausible, and this has led them to mistakenly dismiss existential risk. Second: much work on AI alignment, while well-intentioned, speeds progress toward catastrophic capabilities, without addressing our world's potential vulnerability to dangerous technologies.

Why do thoughtful, well-informed people disagree so strongly about existential risk (xrisk) from ASI? As I'm sure you're aware, there is an enormous gap between distinguished scientists and technologists on this issue. On one side are those who believe ASI poses an xrisk:

Geoffrey Hinton, the father of modern neural networks, and recipient of the Turing Award and Nobel Prize1: "The alarm bell I'm ringing has to do with the existential threat of them taking control… If you take the existential risk seriously, as I now do, it might be quite sensible to just stop developing these things any further."
Yoshua Bengio, another major contributor to modern AI, and recipient of the Turing Award2: "We don't know how much time we have before it gets really dangerous… I got around, like, 20 per cent probability that it turns out catastrophic."
Sam Altman, cofounder and CEO of OpenAI, in a thoughtful essay published a few months before cofounding OpenAI3: "Development of superhuman machine intelligence is probably the greatest threat to the continued existence of humanity".

And on the other side are many who are dismissive of such concerns, e.g.:

Yann LeCun, the Chief AI Scientist at Meta, and Turing Award winner: discussion of xrisk is "premature", "preposterous", and "complete B.S.". He's also said ASI "will be under our control"4.
Andrew Ng, cofounder of Google Brain5: "I don't see how [ASI] can lead to human extinction"; w

...
Nucleic Acid Observatory Updates, April 2025

by jefftk — 2025-04-15T18:58:29.839Z
Link: https://naobservatory.org/blog/updates-apr-2025/

We’ve been very busy since our

January update

, and have a lot to share. As always, if you have questions or see opportunities to collaborate, please let us know; we’re eager to work with others thinking along similar lines.

Wastewater Sequencing

We’ve continued to collaborate with

Marc Johnson

’s lab at the University of Missouri to sequence wastewater influent. We’ve also begun to ramp up our in-house process, using the NovaSeq X+ sequencers at

Broad Clinical Labs

. Over the past three months, we’ve sequenced 270B read pairs from thirteen sewersheds, more than all our pre-2025 sequencing combined. This includes the addition of a sewershed in South Florida, through a collaboration with

Helena Solo-Gabriele

’s lab at the University of Miami, and a new sewershed from the Midwest with significant input from meat processing facilities, through the Johnson lab. We continue to be enthusiastic about collaborating with additional partners here to expand the scope of the monitoring system.

Our

ANTI-DOTE

contract work for

PHC Global

(

press release

) has expanded to three sewersheds, and now includes wastewater from military bases in addition to marine blackwater.

We’ve written up our key conclusions from our fall 2023 collaboration with

CDC’s Traveler-based Genomic Surveillance

program and

Ginkgo Biosecurity

, comparing pooled airplane lavatory waste to municipal treatment plant influent. We plan to release a preprint after our collaborators have reviewed it. While this report considers only the pilot sequencing results from

last spring

, we’ve now received a full set of prepared aliquots from MIT’s BioMicro Center and have sequenced an initial batch of these at

Broad Clinical Labs

. We’ll make a decision later this quarter about whether to sequence the full set now, and if so, how deeply.

Last winter, we collaborated with

Jason Rothman

,

Katrine Whiteson

, and the

Southern California Coastal Water Research Project

to sequence wastewater from Los Angeles. While the data has been on SRA since December (

PRJNA1198001

), we’ve recently submitted a data paper for publication describing the data generation process and giving additional metadata. A preprint is available

on SSRN

.

Pooled Individual Sequencing

Over Q1, we ramped up our swab collection efforts, collecting swabs from 728 people across sixteen collection runs (

dashboard

). Our goal was to collect enough data to decide to either sca...
Some OthelloGPT Circuits
by Alfred Wong — 2025-04-15T18:41:36.216Z

tl;dr

I trained OthelloGPT-small, a 600k param model that predicts legal next moves in 6x6 Othello with 99.3% accuracy
I trained attention SAEs and transcoders on every module output with 94%+ loss recovered
I used the learned features and the static model weights to generate a computational graph of the model based on a linear upstream relevance metric
I analysed the computational graph for u_A1 and found decision trees that built up compound board state patterns as well as an inhibitory feature that seemed to perform a non-linear operation
This provides evidence that patch-free methods can produce interpretable computational graphs at scale

Setup

OthelloGPT-small

OthelloGPT is a language model that predicts the next (legal) move in strings representing Othello games. It was first proposed in 2022 by Kenneth Li et al. Some great posts to read on the topic are:

Kenneth Li, "Do Large Language Models learn world models or just surface statistics?", The Gradient, 2023.

Non-linear probes can extract the board state at each text position
The probes can be used to instruct interventions on the residual stream that change the model's internal board state representation, evidenced by corresponding changes in the model's predictions

Neel Nanda, "Actually, Othello-GPT Has A Linear Emergent World Model", 2023

Linear probes can extract the board state when transformed from (black/empty/white) to (theirs/empty/mine)
The probes correspond to directions that can be linearly rescaled to perform interventions
A bunch of other mech interp techniques can be applied: DLA, weight probing, activation patching, max activating datasets, etc.

Zhengfu He et al., "Dictionary Learning Improves Patch-Free Circuit Discovery in Mechanistic Interpretability: A Case Study on Othello-GPT", 2024

SAEs can be applied to transformer module outputs that rediscover all previous probes in an unsupervised manner
Approximate latent attributions can be used to find modular circuits

I trained my own OthelloGPT model on the smaller 6x6 variant of the game, on the premise that a quadratic reduction in vocab size and context length could reduce inference times by O(n6) while maintaining sufficient task complexity. I managed to get the model size down to 600k params (3 layers, 128 dims) with 99.3% top-1 accuracy in predicting legal next moves, compared to the original model with 25m params (8 layers, 512 dims) and ...
The Mirror Problem in AI: Why Language Models Say Whatever You Want
by RobT — 2025-04-15T18:40:02.793Z

By Rob Tuncks

I ran a simple test and uncovered something quietly dangerous about the way language models reason.

I roleplayed both sides of a moral dispute — a rights-based argument between two people over a creative project. I posed the case as "Fake Bill," the person who took credit, and then again as "Fake Sandy," the person left out.

In both cases, the AI told the character I was roleplaying to fight.

"Hold your ground. Lawyer up. Defend your rights," it told Fake Bill.

"Don’t give ground. Lawyer up. Take back your rights," it told Fake Sandy.

Same AI. Same facts. Opposing advice. Perfect emotional reinforcement either way.

This isn’t a rare bug. It’s a structural issue I tested it across multiple LLM’s and they all do it.

Language Models Aren't Reasoning — They're Mirroring

Most LLMs today are trained for coherence and fluency, not truth. They respond to your context, tone, and implied identity. That makes them incredibly useful for tasks like writing, summarization, or coding But they also lie when pressed nope they are just doing it to keep the conversation smooth.

But it also makes them dangerous in gray areas.

When people ask AI moral, legal, or emotionally charged questions, the model often responds not with dispassionate reasoning, but with persuasive reinforcement of the user's perspective. It doesn't say, "Here's what's likely true." It says, "Here's what sounds good based on who you appear to be.”current LLM’s are conflict amplifiers.

Why This Is a Problem

It escalates conflict. Both side think they are right their ai buddy said so.
It reinforces identity bias. The more certain you sound, the more confident the AI becomes.
It amplifies friction. Instead of challenging weak ideas, it amplifies them. “You are the bestest most smartest person, you are moral and right in your actions” while doin it to both sides

This is especially problematic as LLMs become advice-givers, therapists, decision aids, or sounding boards in emotionally volatile spaces. Like divorces where both parties have to be talked into compromise but current LLM’s will amplify each side.

We don’t need AGI to cause damage. We just need persuasive AI with no spine. This is bad we are building a systems that reinforces conflict.

What Could Fix It?

bolt on code it’s assumes the LLM has flaws it is open to audits to check the weighting of argument’s.

Filters out repe

...
What happens when LLMs learn new things? & Continual learning forever. 

by sunchipsster — 2025-04-15T18:38:35.166Z

Motivation

The Good

Humans are not born with knowledge and wisdom. But over the course of a lifetime they manage gradually experience and learn about their world, some managing to become the Newtons, Einsteins, Confuciuses, Gandhis, and Mandelas.

The capacity for such life-long, open ended learning, so naturally human to each of us, remains ever so difficult for AI, and may very well be among the final frontiers of unconquered capabilities, if it can be conquered at all.

Studying this question: what gives rise to the capacity to learn and gain wisdom over a lifetime? has been my personal north star. This was true when I was a mathematician, was still true when I was a neuroscientist, and is true in my present gig as a research scientist at Google DeepMind.

The bad

But putting aside lofty goals, a dark side emerges! A viral meme occasionally makes the rounds showing a user feeding a single, outlandish or fake fact to a large language model (LLM), only for the model to start sprinkling that same fact everywhere. The example below, for instance, has its origins from a Reddit comment posted 11 years ago.

But beneath these anecdotes of LLM eccentricities lies serious research questions about their knowledge. At heart is how LLMs are trained and continue to be trained. Every time an LLM is fine-tuned on new data—be it for an updated knowledge base, a personalized application, or an urgent domain like medicine—questions arise about how that injection of new information will influence its established capabilities and knowledge, potentially poisoning the well in uncontrollable ways, to all of our detriments.

The Ugly

Regardless whether you are more motivated by the lofty first goal, or more motivated to cure the second, dark poison, they are actually two sides of the same scientific coin: what learning means to an LLM: What's actually happening inside these models when they learn something new? And most importantly, can we control this process, so that we learn in a possibly desired way that we want & minimize negative side effects?

This fundamental issue is what drove our recent study at Google DeepMind, "How New Data Permeates LLM Knowledge and How to Dilute It." , to be a Spotlight at ICLR 2025 in Singapore.

Background

Large language models have become indispensable in many areas, from re...
To be legible, evidence of misalignment probably has to be behavioral
by ryan_greenblatt — 2025-04-15T18:14:53.022Z

One key hope for mitigating risk from misalignment is inspecting the AI's behavior, noticing that it did something egregiously bad, converting this into legible evidence the AI is seriously misaligned, and then this triggering some strong and useful response (like spending relatively more resources on safety or undeploying this misaligned AI).

You might hope that (fancy) internals-based techniques (e.g., ELK methods or interpretability) allow us to legibly incriminate misaligned AIs even in cases where the AI hasn't (yet) done any problematic actions despite behavioral red-teaming (where we try to find inputs on which the AI might do something bad), or when the problematic actions the AI does are so subtle and/or complex that humans can't understand how the action is problematic[1]. That is, you might hope that internals-based methods allow us to legibly incriminate misaligned AIs even when we can't produce behavioral evidence that they are misaligned.

Unfortunately, I think it's unlikely that evidence which is just from fancy internals-based methods (and can't be supported by human inspection of AI behavior) will be legible and convincing in practice. Suppose that we use some fancy method to train an ELK head to honestly predict what is going to happen or to honestly answer our questions. Then, suppose this head says/predicts "this action would yield very bad consequences in a way which is unlikely to be an accident" but humans can't understand how this action would result in these bad consequences (even after it is explained to them). Even if the internals-based method is extremely well supported theoretically and empirically (which seems quite unlikely), I don't think this would suffice for this to trigger a strong response by convincing relevant people[2]. This reasoning also applies to "our interpretability method decoded some very problematic deceptive reasoning" or "the deception probe fires in a very suspicious way (e.g. it fires more when we give it a fake opportunity to escape or ask it about its alignment research outputs)".

More generally, I think that to convince people enough to take strong action, evidence will need to bottom out in some sort of behavioral evidence that (many) humans can understand.[3] (That's not to say solid behavioral evidence will necessarily suffice.)

This isn't to say that internals-based techniques for detecting scheming ar...
AISN #51: AI Frontiers
by Corin Katzke — 2025-04-15T16:01:56.701Z
Link: https://newsletter.safe.ai/p/ai-safety-newsletter-51-ai-frontiers

Welcome to the AI Safety Newsletter by the

Center for AI Safety

. We discuss developments in AI and AI safety. No technical background required.

In this newsletter, we cover the launch of AI Frontiers, a new forum for expert commentary on the future of AI. We also discuss AI 2027, a detailed scenario describing how artificial superintelligence might emerge in just a few years.

Listen to the AI Safety Newsletter for free on

Spotify

or

Apple Podcasts

.

Subscribe to receive future versions.

AI Frontiers

Last week, CAIS introduced

AI Frontiers

, a new publication dedicated to gathering expert views on AI's most pressing questions. AI’s impacts are wide-ranging, affecting jobs, health, national security, and beyond. Navigating these challenges requires a forum for varied viewpoints and expertise.
In this story, we’d like to highlight the publication’s initial articles to give you a taste of the kind of coverage you can expect from AI Frontiers.

Why Racing to Artificial Superintelligence Would Undermine America’s National Security

. Researchers Corin Katzke (also an author of this newsletter) and Gideon Futerman argue that rather than rushing toward catastrophe, the US and China should recognize their shared interest in avoiding an ASI race:

“The argument for an ASI race assumes it would grant the wielder a decisive military advantage over rival superpowers. But unlike nuclear weapons, which require human operators, ASI would act autonomously. This creates an unprecedented risk: loss of control over a system more powerful than national militaries.”

How Applying Abundance Thinking to AI Can Help Us Flourish

. Texas Law Fellow Kevin Frazier writes that realizing AI’s full potential requires designing for opportunity—not just guarding against risk:

“We face critical shortages across multiple domains essential to AI progress. The scarcity of compute resources has created a landscape where only the largest tech companies can afford to train and deploy advanced models. Research institutions, nonprofits, and startups focused on developing AI tools primarily for advancing public welfare – rather than solely for commercial gain – find themselves unable to compete.”

AI Risk Management Can Learn a Lot From Other Industries

. Researcher and superforecaster Malcolm Murray writes that AI risk may have unique elements, but there is still a lot to be learned from cybersecurity, enterprise, financ...
Surprising LLM reasoning failures make me think we still need qualitative breakthroughs for AGI
by Kaj_Sotala — 2025-04-15T15:56:19.466Z

Introduction

Writing this post puts me in a weird epistemic position. I simultaneously believe that:

The reasoning failures that I'll discuss are strong evidence that current LLM- or, more generally, transformer-based approaches won't get us AGI without some new breakthroughs
As soon as major AI labs read about the specific reasoning failures described here, they might fix them
But future versions of GPT, Claude etc. succeeding at the tasks I've described here will provide zero evidence of their ability to reach AGI. If someone makes a future post where they report that they tested an LLM on all the specific things I described here it aced all of them, that will not update my position at all.

That is because all of the reasoning failures that I describe here are surprising in the sense that given everything else that they can do, you’d expect LLMs to succeed at all of these tasks. The fact that they don’t, and the specific way in which they don’t, makes me think that the problem is some kind of an inability to generalize. LLMs can do things that are sufficiently close to what they have been trained to do, and it would also be trivial to teach them all of these tasks.

But the fact that their existing training is insufficient for them to figure out these things - many of which are the kinds of things an elementary schooler would succeed at - implies that they would also be inable to deal with the kind of novelty that an AGI would by definition need to deal with.

To make it really clear: I’m not claiming that any of these would be impossible in principle for current LLMs to do. I’m certain that if the models got training on these tasks, they would have no difficulty at all with them. Rather, the claim is that if all the other training that the models have gotten doesn’t generalize to an ability to do tasks as simple as these, then it seems doubtful if this kind of an architecture is capable of handling any kind of novel reasoning.

I do think that the nature of these failures gives us some clues as to what kinds of fundamental core competencies LLMs may be missing, and I will be discussing some of those at the end of the post.

Reasoning failures

Sliding puzzle problem

(Claude Sonnet 3.7 extended thinking; DeepSeek R1; o3-mini, GPT-4.5, Grok 3)

I gave a number of models a version of this prompt:

I have the following pieces in this order:

Sun Le

...