trentmkelly/LessWrong-43k huggingface dataset:
State of play of AI progress (and related brakes on an intelligence explosion) [Linkpost]
by Noosphere89 — 2025-04-30T19:58:55.204Z
Link: https://www.interconnects.ai/p/brakes-on-an-intelligence-explosion

This time around, I'm sharing a post on Interconnects on why he doesn't believe that the AI 2027 scenario by @Daniel Kokotajlo and many others will come true, and he has 4 sections on this plus a bonus section below:

1. How labs make progress on evaluations

2. Current AI is broad, not narrow intelligence

3. Data research is the foundation of algorithmic AI progress

4. Over-optimism of RL training

Bonus: A world with compute shifting to inference

I'll quote a couple paragraphs of each section below, but you will want to go to the article to get the full experience:

(1st section) One of the hardest things to communicate in AI is talking down the various interpretations of evaluation progress looking vertical over time. If the evals are going from 0 to 1 in one year, doesn’t that indicate the AI models are getting better at everything super fast? No, this is all about how evaluations are scoped as “reasonable” in AI development over time.

None of the popular evaluations, such as MMLU, GPQA, MATH, SWE-Bench, etc., that are getting released in a paper and then solved 18 months later are truly held out by the laboratories. They’re training goals. If these evaluations were unseen tests and going vertical, you should be much more optimistic about AI progress, but they aren’t.

Instead of thinking of stacking rapid evaluation progress on one line in a cumulative, rapid improvement in intelligence, the above plots should make one think that AI is getting better at many tasks, rather than being superhuman in narrow tasks.

(2nd section) In a few years, we’ll look back and see that AI is now 95% robust on a lot of things that only worked 1-5% of the time today. A bunch of new use cases will surprise us as well. We won’t see AI systems that are so intelligent that they cause seismic shifts in the nature of certain domains. Software will still be software. AI will be way better than us at completing a code task and finding a bug, but the stacks we are working on will be largely subject to the same constraints.

Epoch AI had a very complementary post to this view.

(Link below is to a Gradient Updates post titled Most AI value will come from broad automation, not from R&D)

https://epoch.ai/gradient-updates/most-ai-value-will-come-from-broad-automation-not-from-r-d

(Third section) One of the main points of the AI 2027 forecast is that AI research is going to get 2X, then 4X, then 100X, a

...
Don't accuse your interlocutor of being insufficiently truth-seeking
by TFD — 2025-04-30T19:38:39.421Z
Link: https://www.thefloatingdroid.com/dont-accuse-your-interlocutor-of-being-insufficiently-truth-seeking/

I argue that you shouldn't accuse your interlocutor of being insufficiently truth-seeking. This doesn't mean you can't internally model their level of truth-seeking and use that for your own decision-making. It just means you shouldn't come out and say "I think you are being insufficiently truth-seeking".

What you should say instead

Before I explain my reasoning, I'll start with what you should say instead:

"You're wrong"

People are wrong a lot. If you think they are wrong just say so. You should have a strong default for going with this option.

"You're being intentional misleading"

For when you basically thinking they are lying but maybe technically aren't by some definitions of "lying".

What about if they are being unintentionally misleading? That's usually just being wrong, you should probably just say they are being wrong. But if you really think the distinction is important, you can say they are being unintentionally misleading.

"You're lying"

For when they are lying.

You can also add your own flair to any of these options to spice things up a bit.

Why you shouldn't accuse people of being insufficient truth-seeking

Clarity

It's not clear what you are even accusing them of. "Insufficient truth-seeking" could arguably be any of the options I mentioned above. Just be specific. If you really think what you're saying is so important and nuanced and you just need to incorporate some deep insight about truth, use the "add your own flair" option to sneak that stuff in.

Achieving your purpose in the discussion

The most common purposes you might have for engaging in the discussion and why invoking "truth-seeking" doesn't help them:

You want to discuss the object-level issue

You just fucked yourself because the discussion is immediately going to center on whether they actually are insufficiently truth-seeking and whether that accusation was justified. You're going to have to gather The Fellowship, take your argument to Mordor, and throw it into the fire of NEVER GO META before you're ever going to be able to discussion the object level again.

You want to discuss your interlocutor's misconduct

You again fucked yourself because:

It's not clear what misconduct you are accusing them of.
Because of the ambiguity they are going to try to make it seem like you're accusing them of more than what you intended, and therefore actually it's you who isn't being truth-seeking, and you're even accus

...
How can we solve diffuse threats like research sabotage with AI control?
by Vivek Hebbar — 2025-04-30T19:23:19.999Z

Misaligned AIs might engage in research sabotage: making safety research go poorly by doing things like withholding their best ideas or putting subtle bugs in experiments. To mitigate this risk with AI control, we need very different techniques than those in the

original control paper

or

Ctrl-Z

, because research sabotage differs from those settings in two crucial ways:

Firstly, the AI can’t cause a catastrophe by behaving badly on just one occasion. To derail the entire project of solving alignment, the AI probably has to take a large number of malign actions. This contrasts with some other threat models (like rogue internal deployments), where the AI might succeed by acting badly on just a few occasions.
Secondly and relatedly, individual bad actions in research sabotage aren’t very incriminating. If a model puts a bug in experiment code, or doesn’t think of an obvious idea, we won’t know whether it was malicious or just an honest mistake. This contrasts with other threat models -- if you catch the model launching a rogue internal deployment, you can be pretty confident it wasn’t an accident (especially if launching it requires maneuvering around security protocols).

We call threat models with these properties “diffuse” threat models -- where catastrophe requires numerous bad actions, but each bad action is only weak evidence of malign intent when investigated. This contrasts with “concentrated” threat models, where the AI causes catastrophe with a small number of highly incriminating actions. Applying AI control to diffuse threats requires different techniques and evaluation methodology than those used for concentrated threats.

Moving beyond this binary classification, it’s better to imagine a spectrum from diffuse to concentrated, with different attack strategies falling at different points on this spectrum. “Research sabotage” includes many different strategies for derailing research, and these strategies span a large portion of the diffuse-to-concentrated spectrum. Some might even meet the

binary definition of “concentrated failure”

by requiring so few attacks that random auditing wouldn't audit a single attack.

The first section of this post discusses the two basic mitigations -- online training and incrimination. In doing so, it illustrates the key dynamics that define the diffuse-to-concentrat...
Can Narrowing One's Reference Class Undermine the Doomsday Argument? 
by Iannoose n. — 2025-04-30T18:24:27.105Z

I've been reading about the Doomsday Argument (DA) and its reliance on the Self-Sampling Assumption (SSA). I would appreciate some insight on an objection I'm struggling to evaluate.

Specifically, why can't one resist the DA by constraining their reference class to include only currently existing humans, rather than all humans who will ever exist? Under the standard SSA framing, we assume we are random samples from the entire set of humans who will ever live. However, it seems equally reasonable to instead regard ourselves as random samples from the set of currently existing observers.

This narrower reference class aligns well with presentism (the position that only the present exists). From a presentist perspective, future humans do not exist in any real sense. It seems natural to then exclude these future people from probabilistic reasoning.

This move also avoids other counterintuitive implications of the SSA, such as the UN++ hypothetical and the Adam and Eve thought experiment.

I apologize if this question is too philosophy-focused or niche for this forum.
Does there exist an interactive reasoning map tool that lets users visually lay out claims, assign probabilities and confidence levels, and dynamically adjust their beliefs based on weighted influences between connected assertions?
by Zack Friedman — 2025-04-30T18:22:25.893Z

I'm a professional investigator. My standard of evidence is "more likely than not." I think it would be helpful, especially for collaborative purposes, to have a visual reasoning tool inspired by Bayesian thinking. Users could map out individual beliefs, assign probability and confidence levels, and link them with directional influence weights to reflect how one belief affects another. As evidence changes, the map would update, making belief revision explicit and interactive. It would be a lightweight, intuitive Bayesian network that could be intuitively understood by a layperson. It would be shallow and imprecise compared to formal models, but grounded in the same principles. It occurred to me that such a tool would be somewhat universal. One could use it for apple picking just as easily as sex crimes investigations, which makes me think that this must already exist. But I can't find it. I would be so grateful if anyone could point me in the right direction.
Distilling the Internal Model Principle part II
by JoseFaustino — 2025-04-30T17:56:33.089Z

This post was written during the agent foundations fellowship with Alex Altair funded by the LTFF.

Introduction

This is the second part of a two posts series explaining the Internal Model Principle and how it might relate to AI Safety, particularly to Agent Foundations research. In the first post, we constructed a simplified version of IMP that was easier to understand and focused on building intuition about the theorem's assumptions. In this second post, we explain the general version of the theorem as stated by Cai&Wonham[1] and discuss how it relates to alignment-relevant questions such as agent-structure problem and selection theorems.

We present the more general setup and explain Feedback and Regulation conditions. With those, we prove a first result. Then, we discuss the meaning of this first result and add another condition called "Observability" to derive another result. After this, we provide a worked out example and discuss possible extensions of the theorem.

Setup

The IMP from Cai&Wonham[1] originally models the following control theory situation:

Suppose there's, for example, a chemical powerplant in a given environment. The temperature of the plant should be constant and the plant's temperature is affected by the environment's temperature, since they exchange heat, so there is a temperature system made of heaters and air conditioning systems inside the plant, controlled by a controller. This controller controls this systems: it receives information from the environment's temperature and passes a signal to the temperature inside the plant to counter-act the effect of the environment:

If the environment is hot, air conditionings are turned on.
If the environment is cold, heaters are turned on.

Naturally, one thing that could happen is that the controller could receive information from the environment and then, after that, control the temperature system. The theorem shows that, under specific circumstances, the controller actually "foresee" the environment's temperature and control the plant temperature concurrently to the environment's temperature changing. We use "foresee" because what the theorem really shows is that, under some circumstances, the controller is autonomous (i.e, it doesn't use outside information to decide how to control the temperature system) and it controls the temperature system in a way that counter-acts the e...
Research Priorities for Hardware-Enabled Mechanisms (HEMs) 
by aog — 2025-04-30T17:43:45.352Z
Link: https://www.longview.org/request-for-proposals-on-hardware-enabled-mechanisms-hems-for-ai-verification/

Longview Philanthropy is launching a new request for proposals on hardware-enabled mechanisms (HEMs). To encourage strong proposals, we’ve written up what we believe to be top priorities for research in this field.

We think HEMs are a promising method to enforce export controls, secure model weights, and verify compliance with international agreements on AI. We’d like to fund the development of designs, prototypes, and secure enclosures for various kinds of HEMs, as well as complementary red-teaming and field-building efforts.

We plan to distribute $2M - $10M through this RFP. If you’d like to apply, please

submit an expression of interest

. Below are our top priority areas for new work.

Table of Contents

Motivations

Areas of Interest

Designs and prototypes of HEMs

Location Verification

Offline Licensing

Bandwidth Limiters for Accessing Model Weights

Analog Sensors for Workload Verification

Other HEMs

Tamper Resistance Measures

Threat model

Funding priorities

Adversarial Testing

Funding priorities

How should adversarial testing be structured?

Focused Research Organization (FRO) for FlexHEGs

Field-Building for HEMs

Evaluation Criteria

Application Process

Motivations

National security threats from artificial intelligence. Recent progress in AI development has been driven by foundation models trained with growing amounts of data and compute which develop broad capabilities across a wide range of domains. Researchers have pointed out that these systems are often dual-use, with applications in both civilian and military contexts (

Bengio et al., 2024

). For example, current AI systems show precursors of dangerous capabilities in conducting cyberattacks (

Fang et al.

, 2024;

NCSC

, 2024) and developing chemical or biological weapons (

Bran et al.

, 2023;

Amodei

, 2023;

OpenAI

, 2024). AI systems themselves could pose national security threats if developers fail to control fully autonomous AI systems (

Bengio et al. 2024

). These threats motivate the need to defend against malicious use of AI chips (

Hendrycks et al. 2025

).

Security requirements for frontier AI development. AI developers spend hundreds of millions of dollars training the weights of neural networks. Yet these weights are simple files that could be stolen and easily repurposed, compromising the massive investment required to create them. Hardware security could help secure AI model weights against theft. For this r...
Video and transcript of talk on automating alignment research
by Joe Carlsmith — 2025-04-30T17:43:06.557Z
Link: https://joecarlsmith.com/2025/04/30/video-and-transcript-of-talk-on-automating-alignment-research

This is the video and transcript of a talk I gave at Anthropic in April 2025 on automating alignment research. You can also view the slides here.

The content from this talk is from a longer essay on the topic, available here, and in audio form (read by the author) here.

Okay. Hi everybody. Thanks for coming. So: how do we solve alignment? It's a big question. I am going to focus somewhat more narrowly in this talk. Sam had suggested that the group here was going to be a little already in the weeds on a lot of alignment stuff and excited to get into some of the meat. So I'm going to focus on some work in progress on what I see is basically the key question for solving alignment, which is whether or not we can safely automate AI alignment research. The other key question is: having safely automated AI alignment research, can you do enough of it? That's another very important aspect, but I'm going to focus on this bit today.

As Sam mentioned, this is part of a broader series on solving alignment problem. You can check that out on my website. It's not yet done, but there's already a number of essays on there. And as I said, this is a work in progress. I'm really interested in feedback from folks here. I know that automating alignment is a topic dear to the hearts of many at Anthropic, and you guys are also on the front lines doing this work and also interacting with Claude, seeing how helpful Claude can be. I'm very curious to hear your own experiences trying to automate your work, and also what you see as the central barriers that have come up or will come up.

And I'm going to assume a decent amount of familiarity with the alignment discourse. This is per Sam's suggestion that folks are already somewhat enmeshed in this topic. If something doesn't make sense, feel free to jump in. The talk is being recorded, but I can edit out questions if you don't want your question on there.

So to front load, the key thing I'm interested in here... There's a bunch of different challenges to automating alignment research. I'm especially interested in evaluation difficulties. So whether we can tell alignment research is good enough such that we can automate it. And a dynamic I'm going to focus on in particular is the fact that some parts of alignment research seem to be easier to evaluate than others.

In particular, I think we should be especially optimistic ...
Can we safely automate alignment research?
by Joe Carlsmith — 2025-04-30T17:37:13.193Z
Link: https://joecarlsmith.com/2025/04/30/can-we-safely-automate-alignment-research

(This is the fifth essay in a series that I’m calling “How do we solve the alignment problem?”. I’m hoping that the individual essays can be read fairly well on their own, but see this introduction for a summary of the essays that have been released thus far, and for a bit more about the series as a whole.

Podcast version (read by the author) here, or search for "Joe Carlsmith Audio" on your podcast app.

See also here for video and transcript of a talk on this topic that I gave at Anthropic in April 2025. And see here for slides.)

1. Introduction

In my last essay, I argued that we should try extremely hard to use AI labor to improve our civilization’s capacity to handle the alignment problem – a project I called “AI for AI safety.” In this essay, I want to look in more detail at an application of “AI for AI safety” that I view as especially important: namely, automating alignment research. In particular: I want to try to get clearer about the different ways that automating alignment research can fail, and what we can do about them.

I’m especially interested in ways that problems evaluating alignment research might pose a barrier to automating it. Here, one of my key points is that some types of alignment research are easier to evaluate than others. In particular, in my opinion, we should be especially optimistic about automating alignment research that we can evaluate via some combination of (a) empirical feedback loops and (b) formal methods. And if we can succeed in this respect, we can do a huge amount of that type of alignment research to help us safely automate the rest (what I call “conceptual alignment research”).

Overall: I think we have a real shot at safely automating alignment research. But:

Evaluation: We need to figure out how to adequately evaluate automated alignment research – and especially, alignment research we can’t test empirically or check using formal methods;
Scheming: We need to either (a) avoid/prevent AIs actively scheming to undermine our alignment research efforts, or (b) elicit safe, top-human-level alignment research even from scheming AIs;
Resources: We need to give ourselves the necessary time, create the necessary data, and make the necessary investment of compute, staff, effort, and other resources.

Failure on any of these fronts seems unfortunately plausible. But I think there’s a lot we can...
Investigating task-specific prompts and sparse autoencoders for activation monitoring
by Henk Tillman — 2025-04-30T17:09:03.145Z
Link: https://arxiv.org/abs/2504.20271

We’d like to share the most recent paper from the OpenAI Interpretability Team. The question of how activation probes with sparse autoencoders compare with baseline activation probes has gotten a

lot

of

attention

recently

. With this work, we took a step back and asked: how do both these activation probing methods compare with the common baseline of just asking a language model in natural language? And how do they combine?

We find that a combination of “just asking the model” and activation probing, similar to that introduced in the

representation engineering paper

, improves on either prompting or activation probing by itself. This method is highly data efficient and generalizes well, but requires additional inference-time compute for the monitoring task-specific prompt. We also find SAE-based probing methods outperform raw activation probing in most of the monitoring tasks we studied.

Contributors: Henk Tillman, Dan Mossing
European Links (30.04.25)
by Martin Sustrik — 2025-04-30T15:40:02.698Z

Pan-Iberian blackout

Works in Progress is no news source, but they’ve managed to publish an article on the topic within 24 hours:

Every country in the world has a grid designed around the large rotating generators used in nuclear, coal, and gas power plants. They rely on heat boiling water into steam to drive spinning turbines. These generators usually weigh over 100 tonnes and spin at over 3,000 revolutions per minute, meaning that they contain significant kinetic energy, like a very heavy spinning top. If supply drops, the rotor will begin to decelerate, but some of its momentum will be converted into electrical energy. This ‘inertia’ will buy the grid the few seconds it needs to activate its fast-response systems – deploying energy from battery storage and firing up small gas powered engines. This means that frequency should not fluctuate outside a small band even when a large generator trips. Solar panels, on the other hand, directly convert sunlight into electricity without the use of rotating turbines.

[...]

The European Network of Transmission System Operators warned at the start of this year that: ‘Reduced system inertia is a natural consequence of the lower number of directly connected rotating masses of synchronous generators to the grid. The stability support traditionally granted by these generators … will no longer be available in an almost exclusively RES-dominated [renewable energy source] system. This will expose the electricity system to the risk of being unable to withstand out-of-range events like system splits that were previously manageable.’

This is in the context of Spain running entirely on renewable energy for the first time ever just few days before the blackout.

The article also discusses different possible solutions to the inertia problem.

Another aspect of the blackout is that if the problem was not fixed quickly, all the stuff running intermittently on generators would just shut down and restarting the grid from such a state is a lengthy, painful and often manual process. Excellent Practical Engineering has a video on the topic. It was shot when Russia started attacking energy infrastructure in Ukraine, but it comes handy in this case as well.

Luckily, we’ve avoided the nightmare scenario. That being said, I’ve already seen estimates that the blackout will cost Spain 0.5% of GDP in 2025. El País reports that at least five people have di...