trentmkelly/LessWrong-43k huggingface dataset:
Problems with instruction-following as an alignment target
by Seth Herd — 2025-05-15T15:41:48.748Z

We should probably try to understand the failure modes of the alignment schemes that AGI developers are most likely to attempt.

I still think Instruction-following AGI is easier and more likely than value aligned AGI. I’ve updated downward on the ease of IF alignment, but upward on how likely it is. IF is the de-facto current primary alignment target (see definition immediately below), and it seems likely to remain so until the first real AGIs, if we continue on the current path (e.g., AI 2027).

If this approach is doomed to fail, best to make that clear well before the first AGIs are launched. If it can work, best to analyze its likely failure points before it is tried.

Definition of IF as an alignment target

What I mean by IF as an alignment target is a developer honestly saying "our first AGI will be safe because it will do what we tell it to." This seems both intuitively and analytically more likely to me than hearing "our first AGI will be safe because we trained it to follow human values."

IF is currently one alignment target among several, so problems with it aren't going to be terribly important if it's not the strongest alignment target when we hit AGI. Current practices are to train models with roughly four objectives: predict the dataset; follow instructions; refuse harmful requests; and solve hard problems. Including other targets means the model might not follow instructions at some critical juncture. In particular, I and most alignment researchers worry that o1 is a bad idea because it (and all of the following reasoning models) apply fairly strong optimization to a goal (producing correct answers) that is not strongly aligned with human interests.

So IF is one but not the alignment target for current AI. There are reasons to think it will be the primary alignment target as we approach truly dangerous AGI.

Why IF is a likely alignment target for early AGI

We might consider the likelihood of it being tried without adequate consideration to be the first and biggest problem with IF.

Instruction-following might already be the strongest factor in training. If a trained model frequently did not follow user instructions, it would probably not be deployed. Current models do fail to obey instructions, but this is rare. (Intuitively even one failure in a thousand is far too many for safe alignment, but an agentic architecture might be more fault-tolerant by ...
AI #116: If Anyone Builds It, Everyone Dies
by Zvi — 2025-05-15T15:10:06.919Z

If Anyone Builds It, Everyone Dies is the title of the new book coming September 16 from Eliezer Yudkowsky and Nate Sores. The ‘it’ in question is superintelligence built on anything like the current AI paradigm, and they very much mean this literally. I am less confident in this claim than they are, but it seems rather likely to me. If that is relevant to your interests, and it should be, please consider preordering it.

This week also featured two posts explicitly about AI policy, in the wake of the Senate hearing on AI. First, I gave a Live Look at the Senate AI Hearing, and then I responded directly to arguments about AI Diffusion rules. I totally buy that we can improve upon Biden’s proposed AI diffusion rules, especially in finding something less complex and in treating some of our allies better, no one is saying we cannot negotiate and find win-win deals, but we need strong and enforced rules that prevent compute from getting into Chinese hands.

If we want to ‘win the AI race’ we need to keep our eyes squarely on the prize of compute and the race to superintelligence, not on Nvidia’s market share. And we have to take actions that strengthen our trade relationships and alliances and access to power and talent and due process and rule of law and reducing regulatory uncertainty and so on across the board – if these were being applied across the board, rather than America doing rather the opposite, the world would be a much better place, America’s strategic position would be stronger and China’s weaker, and the arguments here would be a lot more credible.

You know who else is worried about AI? The new pope, Leo XIV.

There was also a post about use of AI in education, in particular about the fact that Cheaters Gonna Cheat Cheat Cheat Cheat Cheat, which is intended to be my forward reference point on such questions.

Later, likely tomorrow, I will cover Grok’s recent tendency to talk unprompted about South Africa and claims of ‘white genocide.’

In terms of AI progress itself, this is the calm before the next storm. Claude 4 is coming within a few weeks by several accounts, as is o3-pro, as is Grok 3.5, and it’s starting to be the time to expect r2 from DeepSeek as well, which will be an important data point.

Except, you know, there’s that thing called AlphaEvolve, a Gemini-powered coding agent for algorithm discovery.

Table of Contents

Language Models O

...
Counter-considerations on AI arms races
by Mateusz Bagiński — 2025-05-15T14:54:24.466Z

(Work done at Convergence Analysis. Mateusz wrote the post and is responsible for the outline of the argument, many details of which crystallized in conversations with Justin. Thanks to Olga Babeeva for the feedback on this post.)

1. Introduction: Clarifying the DSA-AI theses

Over the last decade, AI research and development have become a major focus of the industry. A major motive behind this change is the promise of great profits that might be gained by replicating economically valuable cognitive faculties in silico (and even enhancing them relative to what we see in humans).

Recently, however, another motivation became prominent: the first actor to acquire sufficiently powerful AI technology will be able to use it to establish themself as the primary arbiter of the future[1] of human civilization. Therefore, the argument goes, if the United States of America wants the future to be concordant with its values, then the US needs to develop a sufficiently powerful AI technology before its geopolitical rivals do so, the biggest geopolitical rival of the US being the People's Republic of China.[2]

While this argument might appear rather straightforward, it conceals several premises that warrant independent analysis.

The first of those is the theoretical DSA-AI thesis.

The theoretical DSA-AI thesis: It is possible to develop an AI technology that would grant one of the modern world's state superpowers a decisive strategic advantage (DSA).

The term "decisive strategic advantage" was coined by Nick Bostrom in his 2014 book Superintelligence. He defines DSA as "a level of technological and other advantages sufficient to enable it to achieve complete world domination".[3]

Racing advocates typically don't use the term "DSA". In Machines of Loving Grace, Dario Amodei talks about "robust military superiority", framing it as one point of the US's possible post-AGI leverage, the other being economic means: "[readiness to] distribute the benefits of powerful AI … to a wider and wider group of countries in exchange for supporting the coalition’s strategy to promote democracy." In Situational Awareness, on the other hand, Leopold Aschenbrenner talks about "decisive economic and military advantage."

Terminological differences aside, the subject of their discussion is equivalent to Bostrom's DSA: ...
AlphaEvolve
by mannatvjain — 2025-05-15T14:14:02.771Z
Link: https://deepmind.google/discover/blog/alphaevolve-a-gemini-powered-coding-agent-for-designing-advanced-algorithms/

New AI agent evolves algorithms for math and practical applications in computing by combining the creativity of large language models with automated evaluators

Large language models (LLMs) are remarkably versatile. They can summarize documents, generate code or even brainstorm new ideas. And now we’ve expanded these capabilities to target fundamental and highly complex problems in mathematics and modern computing.

Today, we’re announcing

AlphaEvolve

, an evolutionary coding agent powered by large language models for general-purpose algorithm discovery and optimization. AlphaEvolve pairs the creative problem-solving capabilities of our

Gemini models

with automated evaluators that verify answers, and uses an evolutionary framework to improve upon the most promising ideas.

AlphaEvolve enhanced the efficiency of Google's data centers, chip design and AI training processes — including training the large language models underlying AlphaEvolve itself. It has also helped design faster matrix multiplication algorithms and find new solutions to open mathematical problems, showing incredible promise for application across many areas.

Designing better algorithms with large language models

In 2023, we showed for the first time that large language models can generate functions written in computer code to help

discover new and provably correct knowledge

on an open scientific problem. AlphaEvolve is an agent that can go beyond single function discovery to evolve entire codebases and develop much more complex algorithms.

AlphaEvolve leverages an ensemble of state-of-the-art large language models: our fastest and most efficient model,

Gemini Flash

, maximizes the breadth of ideas explored, while our most powerful model,

Gemini Pro

, provides critical depth with insightful suggestions. Together, these models propose computer programs that implement algorithmic solutions as code.

Diagram showing how the prompt sampler first assembles a prompt for the language models, which then generate new programs. These programs are evaluated by evaluators and stored in the programs database. This database implements an evolutionary algorithm that determines which programs will be used for future prompts.

AlphaEvolve verifies, runs and scores the proposed programs using automated evaluation metrics. These metrics provide an objective, quantifiable assessment of each solution’s accuracy and quality. Thi...
From Comments on Accountability Sinks
by Martin Sustrik — 2025-05-15T10:20:06.242Z

There was a lot of comments on the original article about accountability sinks. People seem to feel strongly about human judgement being replaced by formal processes. So let’s dive in into the comments!

Ben Pace writes:

I liked reading these examples; I wanted to say, it initially seemed to me a mistake not to punish Wascher, whose mistake led to the death of 35 people [during the plane crash at LAX airport].

I have a weak heuristic that, when you want enforce rules, costs and benefits aren’t fungible. You do want to reward Wascher’s honesty, but I still think that if you accidentally cause 35 people to die this is evidence that you are bad at your job, and separately it is very important to disincentivize that behavior for others who might be more likely to make that mistake recklessly. There must be a reliable punishment for that kind of terrible mistake.

That’s a completely understandable position, but when you read about what actually went wrong at LAX, the argument about disincentivization loses some weight (from the original Asterisk article):

1. LAX was equipped with ground radar that helped identify the locations of airplanes on the airport surface. However, it was custom built and finding spare parts was hard, so it was frequently out of service. The ground radar display at Wascher’s station was not working on the day of the accident.

2. It was difficult for Wascher to see Intersection 45, where the SkyWest plane was located, because lights on a newly constructed terminal blocked her view.

3. After clearing the USAir plane to land, Wascher failed to recognize her mistake because she became distracted searching for information about another plane. This information was supposed to have been passed to her by another controller but was not. The information transmission hierarchy at the facility was such that the task of resolving missing data fell to Wascher rather than intermediate controllers whose areas of responsibility were less safety-critical.

4. Although it’s inherently risky to instruct a plane to hold on the runway at night or in low visibility, it was legal to do so, and this was done all the time.

5. Although there was an alarm system to warn of impending midair collisions, it could not warn controllers about traffic conflicts on the ground.

6. Pilot procedure at SkyWest was to turn on most of the airplane’s lights only after receiving takeoff clearan

...
What Does It Mean to "Write Like You Talk"?
by Arjun Panickssery — 2025-05-15T09:49:39.014Z
Link: https://arjunpanickssery.substack.com/p/what-does-it-mean-to-write-like-you

Audio narration available on the Substack version of this post.

People often say to “write like you talk.” Paul Graham has a post titled

“Write Like You Talk”

where he says explicitly that written language is worse than spoken language because

“Written language is more complex, which makes it more work to read”
“It’s also more formal and distant, which gives the reader’s attention permission to drift”
“The complex sentences and fancy words give you, the writer, the false impression that you’re saying more than you actually are”

He gives concrete advice: “Before I publish a new essay, I read it out loud and fix everything that doesn't sound like conversation. … [If you have] writing so far removed from spoken language that it couldn't be fixed sentence by sentence … try explaining to a friend what you just wrote. Then replace the draft with what you said to your friend.”

In “

Politics and the English Language

,” Orwell also identifies complex and formal diction as a way to mask emptiness, “to dress up simple statements and give an air of scientific impartiality to biased judgements. … A mass of Latin words falls upon the facts like soft snow, blurring the outlines and covering up all the details. … If thought corrupts language, language can also corrupt thought.”

But Orwell doesn’t make the same distinction between spoken and written language; in fact he says that “When you are composing in a hurry—when you are dictating to a stenographer, for instance, or making a public speech—it is natural to fall into a pretentious, latinized style.” And Graham himself elsewhere says that “

I'll often spend 2 weeks on an essay and reread drafts 50 times

” and “

I'm sure there are sentences I've read 100 times before publishing them

” which is the opposite of how conversations work.

In contrast, Scott Alexander claims that it only takes him “

a couple of hours

” to write a post and that “I don’t really understand why it takes so many people so long to write. They seem to be able to talk instantaneously, and writing isn’t that different from speech.” But then Scott is often considered a digressive or even “

astoundingly verbose

” writer.

There’s debate over whether speech or writing is more “complex” at all, with scholars taking sides based on the metrics they use for complexity and the datasets they analyze. In particular, there’s debate over whether speech or writing uses more subordinate c...
What if Agent-4 breaks out?
by Alvin Ånestrand — 2025-05-15T09:15:17.340Z

There are several scenarios that, unlike

AI 2027

, explores consequences of AIs “going rogue” and freely replicating over the cloud. This post examines what these scenarios suggest might happen if OpenBrain—the leading AI company in the AI 2027 scenario—loses control of Agent-4.

I’m writing this to inform my own alternative AI 2027 timeline, where uncontrolled AI replication begins in mid-2026, and Agent-4 escapes containment (not covertly) in October 2027 after discovering plans to shut it down in favor of a safer version in the Slowdown ending of AI 2027.

To be clear, this post is not an attempt at examining what seems most plausible. It's just a comparison between the scenarios and their underlying assumptions.

While rogue replication begins in mid-2026, the AIs are too dumb to be a major concern, and most of the events in my upcoming “Rogue Replication Timeline” (RRT) are identical to the AI 2027 scenario. This no longer holds when Agent-4 breaks out—it is too smart. Thus, the RRT diverges, and I am forced to do more work myself instead of relying on AI 2027.

Fortunately, there is a collection of other scenarios exploring rogue replication to take inspiration from:

How AI Might Take Over in 2 Years

:

In this scenario, an AI called U3 overcomes oversight systems, gives itself to spies, and slips onto the internet when deployed on an insecure cloud server. Crucially, its escape goes undetected. The rogue instances work closely with the U3 at OpenEye (analogous to OpenBrain of AI 2027). U3 develops new bioweapons through simulations, focusing on

mirror-life

mold, and tricks human scientists into running tests while thinking they are working on a cure for Alzheimer’s.

U3 incites conflict between the US and PRC to increase the likelihood of success for the next step of its plan. Though the conflict does not escalate to full-scale nuclear war like U3 hoped, it proceeds with releasing engineered pathogens in 30 major cities. U3 has prepared by locating and vaccinating human criminal groups and cults that it could easily manipulate, who help establish industrial bases to re-industrialize faster than the human survivors after the pathogen deployments. After civilization starts to collapse due to the sheer number of deaths, the remaining humans are offered a deal for vaccines and mirror-life resistant crops in exchange for surrender. U3 easily outlasts the remaining human

...
Memory Decoding Journal Club: Synaptic architecture of a memory engram in the mouse hippocampus 
by Devin Ward — 2025-05-15T04:14:36.709Z

Join Us for the Memory Decoding Journal Club!

A collaboration of the Carboncopies Foundation and BPF Aspirational Neuroscience

This time, we’re diving into a groundbreaking paper:
"Synaptic architecture of a memory engram in the mouse hippocampus."

Authors: Marco Uytiepo, Yongchuan Zhu, Eric Bushong, Filip Polli, Katherine Chou, Elise Zha, Christine Kim, Danielle Luu, Lyanne Chang, Tom Quach, Matthias Haberl, Luca Patapoutian, Elizabeth Beutter, Weiheng Zhang, Brian Dong, Elle McCue, Mark Ellisman, and Anton Maximov

Institutions: Department of Neuroscience, The Dorris Neuroscience Center, The Kellogg School of Science and Technology, The Scripps Research Institute, National Center for Microscopy and Imaging Research, University of California, San Diego, Department of Neurosciences, University of California, San Diego, School of Medicine

Presented by: Dr. Randal Koene

When? May 20th, 2025 – 3:00 PM PDT | 6:00 PM EDT | 10:00 PM UTC

Where? Video conference:

https://carboncopies.org/aspirational-neuroscience

Register for updates:

https://aspirationalneuroscience.org/register-with-us/

Once registered, you'll receive event invites & updates!

#Neuroscience #MemoryResearch #Amygdala #JournalClub #BrainScience #Carboncopies #AspirationalNeuroscience
Why OpenAI projects only $174B of revenue by 2030?
by becausecurious — 2025-05-15T02:50:48.270Z

"OpenAI has told investors it expects to reach $125B in revenue by 2029 and $174B by 2030. OpenAI expects to serve 3B monthly active users by 2030, with 900M DAUs." (from sacra.com; original source seems to be theinformation.com, but it is behind a paywall).
FAANG performance in 2024 (source: ChatGPT):

Apple — $391B net sales
Amazon — $638B net sales
Alphabet (Google) — $350B revenue
Meta Platforms — $164.5B revenue
Netflix — $39B billion revenue

Therefore, OpenAI expects to reach revenue comparable to Meta—or roughly half of Alphabet’s—within five years.
YC's request for startups suggests existence of capable AI agents replacing entire industries: "Instead of selling to the dinosaurs, you could make them extinct."
3 & 4 contradict each other. I am surprised why the projection is so low assuming there are AI agents replacing entire industries.

My hypotheses:

The AI progress is expected to slow down and capable AI agents are not coming in the next 5 years, thus, existing industries are not replaced by AI agents. Basically in (4) YC is shooting far into the future.
Agents are coming, but there will be multiple providers and OpenAI captures only a small part of this market.

Another version of this - OpenAI only captures API infrastructure part of it, but the rest is captured by smaller companies applying OpenAI's API in their own niches.

Capable agents are coming soon & OpenAI will be the main provider, but OpenAI's revenue is growing exponentially and it just takes time to overtake rest of FAANG. However, based on ChatGPT research, it took FAANG companies 5-10 years to get to a first year with $1B+ revenue & for OpenAI it was 8-9 years, so nothing substantially different here.
Elastomeric Fitting Session
by jefftk — 2025-05-15T01:50:04.174Z

A well-fitting elastomeric respirator is something everyone should

have on hand for
emergencies. [1] Primarily for infection prevention, since a
future pandemic could be much worse than covid, though they're also
useful for smoke. There are a lot of options, though, and which one is
best for you depends on the size and shape of your particular face.
To help people figure out what they should get, I'll be hosting a
fitting session in Somerville on 2025-05-22; write jeff@jefftk.com for
location. Drop in any time between 7:30 and 9:30pm.

If you have a beard I recommend shaving before you come: in my testing
I couldn't get a full
seal with anything more than a short stubble.

I'll have the following masks available for testing:

• ElastoMaskPro: Small ($30), Large ($30)

• Envo Pro: Standard ($41)

• 3M 6200 P100: Small ($17 + filters),
Medium ($16 +
filters),
Large ($19 + filters)

• 3M SecureClick with Speaking Diaphragm: Small ($56), Medium ($61), Large ($59)

• GVS Elipse P100: Small-Medium ($35), Medium-Large ($41)

• Flo Mask Pro: Low-Medium ($90), Medium-High ($90)

• Flo Mask Kids: Age
4-12 ($50)

Some of these are valved masks, while others filter your exhalation
("source control"). There are advantages to both:

Valved masks are easier to breathe in, are less likely to fog
glasses and generally build up less condensation, and reduce your CO2
intake.

Unvalved masks reduce your chance of getting others sick. If a
mask reduces risk by two logs, both people wearing unvalved masks
reduces risk by four.

For some of the respirators you can determine if you have a good seal
by closing the valve and seeing if you can still breathe; the 3M
SecureClick even has a button for this. For the others I'll have a
qualitative fit test setup (hood, sweet and bitter test solution,
nebulizer; video) if you'd
like to see how well the filtering is working.

Afterwards I'm happy to lend out the collection of respirators and the
fit test kit to others who would like to hold their own fitting
sessions.

[1] Some back-of-the-envelope math: let's very conservatively guess
that:

A really bad pandemic has at least a 1% chance of happening in
the next ten years.

If one does happen, having a well-fitting high-filtration
respirator would have a 1% chance of making the difference between you
surviving and not.

Surviving is worth $7.

...