trentmkelly/LessWrong-43k huggingface dataset:
Key paths, plans and strategies to AI safety success
by Adam Jones — 2025-06-19T16:56:09.025Z
Link: https://bluedot.org/blog/ai-safety-paths-plans-and-strategies

In January, I spent over 100 hours:

reading 50+ AI safety 'plans',
interviewing with 10+ AI safety researchers; and
thinking about AI safety strategy, drawing on my 2.5 years in the field

This list of routes to AI safety success (updated June 2025) is a key output of that research. It describes the main paths to success in AI safety. I’d guess that understanding the strategies in this document would place you in the top 10th percentile of people at AI safety strategy in the AI safety community.

Part 1: International approaches

International safe actors race ahead

Get “good” international coalitions to be the first people to develop advanced AI, and do so safely. They then would use this AI model to shift the world into a good state: for example by accelerating one of the other paths below.

CERN for AI is a common model for this. These proposals usually envision shared infrastructure and expertise for frontier AI research, combining the efforts of many nation states together.

This can be approached from two angles:

make AI superpowers (US, China) more cooperative and democratic
make cooperative democracies (EU, UK, Australia, etc.) more AI-capable

Resources:

Building CERN for AI (Center for Future Generations) - European-led international research consortium with tiered membership (EU core plus UK, Switzerland, Canada), specific funding mechanisms and governance structures for trustworthy AI development.
What Success Looks Like (EA Forum) - Scenario 6 "Apollo Project" where well-resourced international project develops safe AI first.
Hassabis Triple Institution Model (YouTube) - "CERN for AGI" component: international collaborative research facility for safe AGI development, complemented by monitoring and governance institutions.

Cooperative international deterrence

Get everyone to agree not to build the dangerous thing, and enforce those agreements.

IAEA for AI is a common model for this. This body would oversee advanced AI development globally, conduct inspections, and enforce safety standards. Suggested implementations usually rely on compute governance - using control over compute resources (AI chips and other hardware) to monitor and regulate AI development.

These agreements could vary both in scope (who is affected) and severity (what are they prevented from doing). For example, different options could be:

preventing anyone from building superintelligence
preventing n

...
AI safety techniques leveraging distillation
by ryan_greenblatt — 2025-06-19T14:31:02.632Z

It's currently possible to (mostly or fully) cheaply reproduce the performance of a model by training another (initially weaker) model to imitate the stronger model's outputs.[1] I'll refer to this as distillation. In the case of RL, distilling the learned capabilities is much, much cheaper than the RL itself (especially if you are distilling back into the original base model). But even for pre-training, distilling is cheaper than the original training.[2]

In this post, I'll discuss how we could utilize distillation to potentially remove (or possibly detect) misalignment. I'll also discuss a few other applications.[3] My overall take is that techniques utilizing distillation are mildly to moderately promising and the low cost of distillation might make them surprisingly viable, but it's quite tricky to reason about how effective these techniques are.

Distilling to remove misalignment

I'll assume that we have a powerful model[4] that we're worried is seriously misaligned and we wish to use distillation to mitigate this.[5] Then, we'll aim to remove misalignment while preserving capabilities by taking some relatively weak (or even randomly initialized) model[6]—that we're confident isn't initially seriously misaligned—and training this weak model to imitate trajectories generated using the stronger model. To retain capabilities, we'd need to distill using some (potentially huge) number of trajectories which contain demonstrations of the capabilities we need.

By default, I'm imagining that the weak model we distill into has a similar number of parameters as the original powerful model and that we're trying to recover basically the full capabilities of the original model, though these things could vary. Note that this differs from how distillation would typically be used to reduce inference costs.

One reasonably central case is distilling capabilities learned from RL back into the original base model (which was purely pretrained on some self-supervised prediction task).[7] This is most helpful if we're confident that the original base model isn't itself coherently misaligned, which seems like a good assumption for base models which aren't much more powerful than the base models we see today.

Given that we need trajectories which suffice for learning the important capabilities of the stronger model, why think that we can remove problematic propensities? The main hope...
Political Funding Expertise (Post 6 of 7 on AI Governance)
by Mass_Driver — 2025-06-19T14:14:31.909Z

INTRODUCTION

The Story So Far

In my first three posts in this sequence, I estimated that we have 3 researchers for every advocate working on US AI governance, and I argued that this ratio is backwards. Even the best research offers only modest and indirect support for advocacy, and research alone has negligible political power. Without political power, we can’t change the bad incentives of AI developers that are very likely to lead to the collapse of human civilization.

In the fourth post of this sequence, I acknowledged that some amount of initial research to lay out the philosophical foundations of a new field might be needed before advocacy can begin, but I also showed that this initial research has been thoroughly completed. We know why unregulated AI is bad and we know at least some harmless ways that we can make progress toward fixing that problem.

In the fifth post of this sequence, I illustrated that point by listing eleven examples of ‘orphaned’ policies. Each of these policies was proposed by academic researchers – often several years ago – but the policies have not been drafted in any detail by policy wonks, let alone presented to decision-makers by political advocates. We clearly have an over-supply of academic ideas and an under-supply of political elbow grease.

This means that it’s very strange that so much of the AI safety movement’s funding has gone toward academic-style research and is continuing to go towards more research. Presumably, the funders can see the same trends that I’ve laid out in this sequence: they should know as well as I do that funding more researchers than advocates is deeply suboptimal.

So, why do they keep doing it? My best guess is that they’re biased by their own backgrounds: the staff of major AI safety funding organizations are overwhelmingly drawn from academic-style research environments. This may be causing them to fund other researchers even when that’s not strategically optimal, simply because they’re more comfortable with research or they are better able to understand the benefits of research.

In this sixth post, I will argue that rationalist and effective altruist funders should repair this flaw in their staffing so that they can more accurately evaluate the usefulness of future grant proposals. To its credit, Open Philanthropy hired an actual governance expert in April 2025, shortly after it finalized it...
Documents Are Dead. Long Live the Conversational Proxy.
by 8harath — 2025-06-19T14:01:34.210Z

Sometime in early 2024, I stopped reading books. Not because I don’t love the process—but because I realized the alpha/hour ratio was far too low.

Reading texts (especially documentation) has become increasingly inefficient. It’s not about laziness; it’s about leverage. A static wall of text demands effort for diminishing insight. But a PDF piped into an LLM? That’s different. That’s programmable knowledge. I can chat with it. Extract patterns. Mine underdiscussed takeaways. Highlight blind spots. Summarize arguments. It’s like outsourcing the heavy intellectual lifting (Subjective).

This flipped a mental switch for me:

Documents today aren't messages to be read—they're messengers to be conversed with.

If I send someone a PDF, I’m not expecting them to read it cover to cover. I’m expecting them to throw it into their favorite assistant and have a dialogue with my intent. A document, in this sense, is just a vessel—a digital PA that carries my signal forward.

So what’s the next step?

I don’t have a solution yet, but I think it’s obvious:

LLMs will eventually have cloud-integrated memory,
Each person will have their own persistent document graph,

And sharing won’t look like “sending a file”—it’ll look like:

“Send Bharath’s repo docs to your Claude instance.”
“Put this paper in your weekend calendar as a chat object.”
“Ask my assistant what key takeaways your agent missed.”

Documentation will exist less as content and more as contextual proxies—something to interrogate, not ingest.

We're not building static libraries. We’re building living conversations.

Let that sink in.
How did you find out about AI Safety? Why and how did you get involved?
by Ana Lopez — 2025-06-19T14:00:18.958Z

Hi everyone!
My name is Ana, I’m a sociology student currently conducting a research project at the University of Buenos Aires. My work focuses on how awareness around AI Safety is raised and how the discourses on this topic are structured and circulated.

That’s why I’d love to ask you a few questions about your experiences.
To understand, from a micro-level perspective, how information about AI Safety spreads and what the trajectories of those involved look like, I’m very interested in your stories: how did you first learn about AI Safety? What made you feel compelled by it? How did you start getting involved?
I’d also love to know a bit more about you and your personal or professional background.

I would deeply appreciate it if you could take a moment to complete this short form where I ask a few questions about your experience. If you prefer, you’re also very welcome to reply to this post with your story.

I'm interested in hearing from anyone who has any level of interest in AI Safety — even if it's minimal — from those who have just recently become curious and occasionally read a post on Less Wrong, to those who work professionally in the field.

Thank you so much in advance!
A deep critique of AI 2027’s bad timeline models
by titotal — 2025-06-19T13:29:59.310Z
Link: https://titotal.substack.com/p/a-deep-critique-of-ai-2027s-bad-timeline

Thank you to Arepo and Eli Lifland for looking over this article for errors.

I am sorry that this article is so long. Every time I thought I was done with it I ran into more issues with the model, and I wanted to be as thorough as I could. I’m not going to blame anyone for skimming parts of this article.

Note that the majority of this article was written before

Eli’s updated model

was released (the site was updated june 8th). His new model improves on some of my objections, but the majority still stand.

Introduction:

AI 2027

is an article written by the “AI futures team”. The primary piece is a short story penned by Scott Alexander, depicting a month by month scenario of a near-future where AI becomes superintelligent in 2027,proceeding to automate the entire economy in only a year or two and then either kills us all or does not kill us all, depending on government policies.

What makes AI 2027 different from other similar short stories is that it is presented as a forecast based on rigorous modelling and data analysis from forecasting experts. It is accompanied by

five appendices

of “detailed research supporting these predictions” and a codebase for simulations. They state that

“hundreds” of people reviewed

the text, including AI expert Yoshua Bengio, although some of these reviewers

only saw bits of it

.

The scenario in the short story is not the median forecast for any AI futures author, and none of the AI2027 authors actually believe that 2027 is the median year for a singularity to happen. But the argument they make is that 2027 is a plausible year, and they back it up with images of sophisticated looking modelling like the following:

This combination of compelling short story and seemingly-rigorous research may have been the secret sauce that let the article to go viral and be treated as a serious project:To

quote the authors themselves

:

It’s been a crazy few weeks here at the AI Futures Project. Almost a million people visited

our webpage

; 166,000 watched

our Dwarkesh interview

. We were invited on something like a million podcasts. Team members gave talks at Harvard, the Federation of American Scientists, and OpenAI.

Now, I was originally happy to dismiss this work and just wait for their predictions to fail, but this thing just keeps spreading, including a youtube video with

millions of

...
AI #121 Part 1: New Connections
by Zvi — 2025-06-19T13:00:05.260Z

That’s right. I said Part 1. The acceleration continues.

I do not intend to let this be a regular thing. I will (once again!) be raising the bar for what gets included going forward to prevent that. But for now, we’ve hit my soft limit, so I’m splitting things in two, mostly by traditional order but there are a few things, especially some videos, that I’m hoping to get to properly before tomorrow, and also I’m considering spinning out my coverage of The OpenAI Files.

Tomorrow in Part 2 we’ll deal with, among other things, several new videos, various policy disputes and misalignment fun that includes the rising number of people being driven crazy.

Table of Contents

Language Models Offer Mundane Utility. How much do people use LLMs so far?

Language Models Don’t Offer Mundane Utility. Can’t always get what you want.

Humans Do Not Offer Mundane Utility. A common mistake.

Langage Models Should Remain Available. We should preserve our history.

Get My Agent On The Line. It will just take a minute.

Have My Agent Call Their Agent. Burn through tokens faster with multiple LLMs.

Beware Prompt Injections. Access + External Communication + Untrusted Content = Asking For Trouble.

Unprompted Attention. There, they fixed it.

Huh, Upgrades. Everyone gets Connectors, it’s going to be great.

Memories. Forget the facts, and remember how I made you feel.

Cheaters Gonna Cheat Cheat Cheat Cheat Cheat. Knowing things can help you.

On Your Marks. LiveCodeBench Pro.

Fun With Media Generation. MidJourney gets a new mode: image to video.

Copyright Confrontation. How could we forget Harry Potter?

Deepfaketown and Botpocalypse Soon. The exponential comes for us all.

Liar Liar. Which is more surprising, that the truth is so likely, or that lies are?

They Took Our Jobs. Most US workers continue to not use AI tools. Yet.

No, Not Those Jobs. We are not good at choosing what to automate.

All The Jobs Everywhere All At Once. How to stay employable for longer.

The Void. A very good essay explains LLMs from a particular perspective.

Into the Void. Do not systematically threaten LLMs.

The Art of the Jailbreak. Claude 4 computer use is fun.

Get Involved. Someone looks for work, someone looks to hire.

Introducing. AI.gov and the plan to ‘go all in’ on government AI.

In Other AI News. Preferences are revealed.

Show Me the Money. OpenAI versus Microsoft.

Language Models Offer Mundane Uti

...
AI can win a conflict against us
by Algon — 2025-06-19T07:20:07.488Z

Context: This is a linkpost for https://aisafety.info/questions/NM3O/8:-AI-can-win-a-conflict-against-us

This is an article in the new intro to AI safety series from AISafety.info. We'd appreciate any feedback. The most up-to-date version of this article is on our website.

Suppose an AI has realized that controlling the world would let it achieve its aims. If it’s weak, it might just obey humans. If it’s roughly at our power level, it might make deals or trades with us. But if it’s sufficiently capable, it might make a play to seize control from us. That is, it might attempt takeover. Could it succeed?

Intelligence confers a kind of power. (Keep in mind that by intelligence, we don’t just mean clever puzzle-solving, but any traits that go into making effective decisions.) Humans have power over chimpanzees because we’re smarter. If the chimpanzees of the world banded together against us, we’d defeat them despite being physically weaker. Even if they were equal in number to us, we’d still win.

For similar reasons, a superintelligence could win against us. We don’t know how, exactly — compare playing chess against a grandmaster such as Magnus Carlsen. You can’t predict the exact moves he’ll use to beat you. If you could reliably do that, you’d be just as good as him. But you can still abstractly say that he’s better at finding better winning chess moves.

That type of abstract argument only matters if there’s a large and complex space of strategies to search through, so there’s room to find surprisingly effective ones. But the real world is plenty complex. In the course of a takeover attempt, superintelligent AI could:

Outcompete humans economically and build wealth, perhaps by exploiting financial markets or with other clever schemes. Consider the case of Satoshi Nakamoto, who anonymously made billions by inventing Bitcoin. At no point in this process did Satoshi need to be a human.
Use superhuman persuasive skills. Maybe the most straightforward way of persuading people to do things is by paying them money, as above. But AI could also build extremely good psychological models, both of the human mind in general, and of specific people. Working off of these models, it could then use superhuman skills at deception, fraud, argument, intimidation, and charm.
Hack into devices. Many computer systems have vulnerabilities that can be exploited by a sufficien

...
Different goals may bring AI into conflict with us
by Algon — 2025-06-19T07:19:45.800Z

Context: This is a linkpost for https://aisafety.info/questions/NM3H/7:-Different-goals-may-bring-AI-into-conflict-with-us

This is an article in the new intro to AI safety series from AISafety.info. We'd appreciate any feedback. The most up-to-date version of this article is on our website.

Aligning the goals of AI systems with our intentions could be really hard. So suppose we fail, and build a powerful AI that pursues goals different from ours. You might think it would just go off and do its own thing, and we could try again with a new AI.

But unfortunately, according to the idea of instrumental convergence, almost any powerful system that optimizes the world will find it useful to pursue certain kinds of strategies. And these strategies can include working against anything that might interfere.

For example, as a thought experiment, consider Peelie, a robot that cares only about peeling oranges, always making whatever decision results in maximum peeled oranges. Peelie would see reasons to:

Remove nearby objects that might block its supply of oranges.
Acquire resources that it can use to accomplish its goals, like money to buy oranges and knives.
Convince people to never turn it off, because if it’s turned off, it peels no oranges.
Remove anything that might change its goals, because if it started to peel lemons instead, it would peel fewer oranges.
Seize control of the building it’s in — just in case.
Seize control of the country it’s in — just in case the people there would stop it from seizing the building.
Build a smarter AI that also only cares about peeling oranges.
Hide its intentions to do these things, because if humans knew its intentions, they might try to stop it.

This is meant as an illustration, not a realistic scenario. People are unlikely to build a machine that’s powerful enough to do these things, and only make it care about one thing.

But the same basic idea applies to any strong optimizer with different goals from ours, even if they’re a lot more complex and harder to pin down. The optimizer can become more successful by contesting our control — at least, if it does so successfully.

And while people have proposed ways to address this problem, such as forbidding the AI from doing certain kinds of actions, the AI alignment research community doesn’t think the solutions we’ve considered so far are sufficient to contain superintellige...
My Failed AI Safety Research Projects (Q1/Q2 2025)
by Adam Newgas — 2025-06-19T03:55:40.363Z

This year I've been on sabbatical, and have spent my time upskilling in AI Safety. Part of that is doing independent research projects in different fields.

Some of those items have resulted in useful output, notably A Toy Model of the U-AND Problem, Do No Harm? and SAEs and their Variants.

And then there are others that I've just failed fast, and moved on.

Here I write up those projects that still have something to say, even if it's mostly negative results.

LLM Multiplication

Inspired by @Subhash Kantamneni's Language Models Use Trigonometry to Do Addition, I was curious if LLMs would use a similar trick for multiplication, after an appropriate log transform.

I adapted the original code to this case and ran it with various parameters. I also designed a few investigations of my own. Notably, the original code relied on DFT which was not appropriate for a floating point case.

The original paper looked at early-layer activations of the tokens "1" to "360", and looked for structure that related to the numerical value of the token. They found a helix, i.e. they found a linear direction that increased linearly with the token value, and several size two subspaces that each encoded the token value as an angle, with various different angular frequencies.

Using similar techniques, I found a linear direction corresponding to log(token value). This emerged via PCA techniques as a fairly high eigenvalues, so there it feels like fairly strong evidence. That said, I couldn't find significant impact via ablation studies. I found no evidence for an equivalent of angular encoding for log(token value).

On reflection, I think multiplication is probably not handled equivalently to addition. There's a couple of reasons why:

Performing log/exp transforms is not easy for ReLU based models[1]
Multiplication has much larger result values, so the single-token approach taken by this paper is less valuable.
There are a number of "natural frequencies" in numbers, most notably mod 10, which are useful for things other than clock arithmetic. E.g. anthropic finds a feature for detecting the rightmost digit of a number, which almost certainly re-uses the same subspace. There's no equivalent for logarithms.

· Brief Writeup · Code ·

Toy Model of Memorisation

After my success with Computational Superposition in a Toy Model of the U-AND Problem, I wanted to try toy models to a harder pr...