trentmkelly/LessWrong-43k huggingface dataset:
Scaling Laws for Scalable Oversight
by Subhash Kantamneni — 2025-04-30T12:13:32.412Z

Subhash, Josh, and David were co-first authors on this project. The full paper is

here

, and our code can be found

here

.

TLDR

We empirically study the success of weak-to-strong oversight as we scale the intelligence of the weak overseer model (Guard) and strong adversary model (Houdini) in four oversight game settings. We use those insights to theoretically investigate how to optimally design nested scalable oversight (NSO) protocols to control future superintelligences.

Introduction

Scalable oversight has been proposed in many forms as a way to control future superintelligences.[1] These methods share a central goal: enabling weaker systems to oversee stronger ones (weak-to-strong oversight), ultimately enabling humans to oversee systems with superhuman cognitive abilities. A key idea is that scalable oversight can be bootstrapped: weaker systems oversee stronger ones, which then oversee even more capable models in the next stage—allowing oversight to scale alongside capabilities recursively. We refer to this iterated process as Nested Scalable Oversight, or NSO for brevity.

While many frontier labs have embraced the idea of bootstrapping oversight to control potentially dangerous superintelligences, most guarantees surrounding NSO have been qualitative, focusing on plausibility and conceptual design. However, the safety standards applied to other high-risk technologies are quantitative. For instance, the U.S. requires civilian aircraft to have a

fatality rate below 1/100,000 per flight hour

, and nuclear reactors to maintain a

core damage frequency below 1/10,000 per year

. Motivated by these risk thresholds, our work takes a first step towards quantifying the failure rates of NSO systems by studying the weak-to-strong oversight problem.

In this blog post, we highlight a few key contributions from the full paper

Modeling oversight as a game played between a weak overseer (Guard) and strong adversary (Houdini)
Proposing that oversight and deception performance take the shape of a Double ReLU as a function of general model intelligence
Validating that hypothesis in four realistic oversight setups: Mafia, Debate, Backdoor Code, and Wargames. Wargames is a novel oversight game.
Using our framework to theoretically study how to best design real world nested scalable oversight systems

General Setup

We model weak-to-strong oversight as a game between a...
Early Chinese Language Media Coverage of the AI 2027 Report: A Qualitative Analysis
by jeanne_ — 2025-04-30T11:06:34.704Z

In this blog post, we analyse how the recent AI 2027 forecast by Daniel Kokotajlo, Scott Alexander, Thomas Larsen, Eli Lifland, and Romeo Dean has been discussed across Chinese language platforms. We present:

Our research methodology and synthesis of key findings across media artefacts
A proposal for how censorship patterns may provide signal for the Chinese government's thinking about AGI and the race to superintelligence
A more detailed analysis of each of the nine artefacts, organised by type: Mainstream Media, Forum Discussion, Bilibili (Chinese Youtube) Videos, Personal Blogs.

Methodology

We conducted a comprehensive search across major Chinese-language platforms–including news outlets, video platforms, forums, microblogging sites, and personal blogs–to collect the media featured in this report. We supplemented this with Deep Research to identify additional sites mentioning AI 2027. Our analysis focuses primarily on content published in the first few days (4-7 April) following the report’s release. More media has been generated since our research was completed, and we may cover these in a follow-up if there's interest.

Summary

Below are some patterns that emerged across the content we analysed:

Many reports omit references to China, DeepCent, and the US–China race dynamic and instead focus on technical aspects of human-level or superhuman AI development. Mentions of DeepCent, Chinese espionage, or AI betrayal are often softened or scrubbed. Less regulated platforms like personal blogs or niche video channels include more engagement with these themes.
The report provoked mixed responses. Some authors treated it as a serious forecast, engaging with its concerns about alignment and US-China dynamics, while others dismissed it as alarmist or framed it as a kind of science fiction thought experiment rather than a serious forecast.
Audience engagement remains low across the board. Many posts received minimal views, likes, or comments.
Several artefacts make the same error of describing the report as a ‘76-page document’ despite it being 71 pages. They often reference the same milestones, too. Such details being repeated suggest that sources are copying from one another.

Censorship as Signal

The AI 2027 website remains accessible in China without a VPN—a curious fact given its content about democratic revolution, CCP coup scenarios, and claims of C...
[Paper] Automated Feature Labeling with Token-Space Gradient Descent
by Wuschel Schulz — 2025-04-30T10:22:13.942Z

This post gives a brief overview and some personal thoughts on a new ICLR workshop paper that I worked on together with Seamus..

TLDR:

We developed a new method for automatically labeling features in neural networks using token-space gradient descent
Instead of asking an LLM to generate hypotheses about what a feature means, we directly optimize the feature label itself
The method successfully labeled features like "animal," "mammal," "Chinese," and "number" in our proof-of-concept experiments
Current limitations include single-token labels, issues with hierarchical categories, and computational cost
We are not developing this method further, but if someone wants to pick up this research, we would be happy to assist

In this project, we developed a proof of concept for a novel way to automatically label features that directly optimizes the feature label using token-space gradient descent. We show its performance on several synthetic toy-features. We have discontinued developing this method because it didn't perform as well as we had hoped, and I'm now more excited about research in other directions.

Automated Feature Labeling

A central method for Mechanistic Interpretability is decomposing neural network activations into linear features via methods like Sparse Auto Encoders, Transcoders, or Crosscoders. While these decompositions give you somewhat human-understandable features, they don't provide an explanation of what these features actually mean. Because there are so many features in modern LLMs, we need some automatic way to find these descriptions. This is where automatic feature labeling methods come in.

Previous Methods

Previous methods for automated feature labeling work by finding text in which specific tokens activate the feature, and then prompting LLMs to come up with explanations for what the feature might mean to be activated in these situations. Once you get such a hypothesis, you can validate it by having an LLM predict when it thinks this feature should be active, and checking that against its actual activation. If these predictions don't match the actual activation, you can prompt an LLM again with the counterexamples and ask it to come up with new feature labels.

This process crucially depends on LLMs generating correct hypotheses for feature labels given the right data. This might be problematic because:

Idea generation is actually hard. As we know from

...
A single principle related to many Alignment subproblems?
by Q Home — 2025-04-30T09:49:21.181Z

I want to show a philosophical principle which, I believe, has implications for many alignment subproblems. If the principle is valid, it might allow to

connect the study of abstractions, Shard Theory and Mechanistic Anomaly Detection;
obtain multiple potential solutions to the Eliciting Latent Knowledge problem;
obtain a bounded solution to outer and inner alignment. (I mean Task-directed AGI level of outer alignment.)

This post clarifies and expands on ideas from here and here. Reading the previous posts is not required.

The Principle

The principle and its most important consequences:

By default, humans only care about variables they could (in principle) easily optimize or comprehend.[1] While the true laws of physics can be arbitrarily complicated, the behavior of variables humans care about can't be arbitrarily complicated.
Easiness of optimization/comprehension can be captured by a few relatively simple mathematical properties (X).[2] Those properties can describe explicit and implicit predictions about the world.
We can split all variables (potentially relevant to human values) into partially arbitrary classes, based on how many X properties they have. The most optimizable/comprehensible variables (V1), less optimizable/comprehensible variables (V2), even less optimizable/comprehensible variables (V3), etc. We can do this without abrupt jumps in complexity or empty classes. The less optimizable/comprehensible the variables are, the more predictive power they might have (since they're less constrained).

Justification:

If something is too hard to optimize/comprehend, people couldn't possibly optimize/comprehend it in the past, so it couldn't be a part of human values.
New human values are always based on old human values. If people start caring about something which is hard to optimize/comprehend, it's because that "something" is similar to things which are easier to optimize/comprehend.[3] Human values are recursive, in some sense, flowing from simpler (versions of) values to more complicated (versions of) values. This can be seen from pure introspection, without reference to human history.
Therefore, if something is hard to optimize/comprehend, it's unlikely to be a part of the current human values (unless it's connected to something simpler). Even if currently humans have the means to optimize/comprehend it.

There are value systems for whic...
What if Brain Computer Interfaces went exponential?
by Stephen Martin — 2025-04-30T05:07:21.811Z

After watching the release of this video from one of the patients in Neuralink's first clinical trials, I decided to post here the prologue of some fiction I have been writing which is set in a world where BCI tech experiences the same exponential growth that we have seen with LLMs.

Nueralink Mk. 1

dedicates its first clinical trial to the morally unimpeachable work of helping quadriplegics control external devices. The early successes are heartwarming. The first viral video is of a hospital bed bound young man in Trenton New Jersey slowly dragging a cursor across a point-and-click keyboard and telling his mother “LUV U” is of a man with ALS explaining his condition, how supportive his wife has been, and how much Neuralink has improved his quality of life (Updated April 28th, 2025). It is followed shortly after by a teenage girl named Hope piloting a motorized wheelchair through a park. CNN runs a cinematic shot of her hair billowing in the wind.

The Phase I clinical trial involves 55 patients, many of whom churn out inspiring content on a regular basis. Redditors who see these successes drop whatever they are doing to furiously clarify across their various socials that these achievements, while great, should not in any way be credited to Elon Musk. Regardless, the tech is at first immensely popular and Neuralink the company skyrockets to an approval rating in the mid 90s. This makes it about as popular as root beer floats, Rocky IV, or receiving oral sex.

Neuralink goes international quickly, opening up a broader trial in the UAE for patients with motor & speech impairment (Edited May 15, 2025).

Political pressure to make the technology more easily available mounts. Patient advocacy groups and libertarian minded lobbyists team up with Silicon Valley accelerationists, all of them pushing to get “ultra fast tracking” for BCIs. There’s not a politician or bureaucrat on the planet eager to oppose wheelchair bound sob stories backed by tech billionaires. The call comes from on high, the Oval Office itself, and the FDA issues a special medical device approval pathway for BCIs which only requires one year of safety data to clear Phase I. It also allows manufacturers to provide accessibility for patients suffering from quadriplegia causing conditions under Expanded Access.

An unintended consequence of this sped up regulatory pathway is its interaction with state Righ...
Interpreting the METR Time Horizons Post
by snewman — 2025-04-30T03:03:19.928Z
Link: https://amistrongeryet.substack.com/p/measuring-ai-progress

[EDIT: initial publication did not have the link to the original post, and was missing the footnotes. Sorry about that! Fixed now.]

[This has been lightly edited from the original post, eliminating some introductory material that LW readers won't need. Thanks to Stefan Schubert for suggesting I repost here. TL;DR for readers already familiar with the METR Measuring AI Ability to Complete Long Tasks paper: this post highlights some gaps between the measurements used in the paper and real-world work – gaps which are discussed in the paper, but have often been overlooked in subsequent discussion.]

It's difficult to measure progress in AI, despite the slew of benchmark scores that accompany each new AI model.

Benchmark scores don’t provide much perspective, because we keep having to change measurement systems. Almost as soon as a benchmark is introduced, it becomes saturated – models learn to ace the test. So someone introduces a more difficult benchmark, whose scores aren’t comparable to the old one. There’s nothing to draw a long-term trend line on.

METR recently published a study aimed at addressing this problem: Measuring AI Ability to Complete Long Tasks. They show how to draw a trend line through the last six years of AI progress, and how to project that trend into the future.

The study’s results have been widely misinterpreted as confirmation that AGI (AI capable of replacing most workers, or at least most remote workers) is coming within five years. In actuality, the study doesn’t say much about AGI, except to provide evidence against the most aggressive forecasts.

To explain what the paper does and does not say, I’ll begin by explaining the problem it set out to address.

We’re Gonna Need a Harder Test

[LW readers might skip this section]

Here’s a chart I’ve shown before:

Additional context available at the

source

Each line represents AI performance on a particular benchmark. Unsurprisingly, there’s a clear pattern of rapid improvement. But each line is measuring something different; we can’t connect the dots to plot a long-term trend.

For instance, the dark green line shows scores on a speech recognition test. If you’d looked at that plot in 2013, you might have been able to estimate speech recognition scores for 2015. But you’d have had no way of anticipating that reading comprehension (pink line) would start to take off after 2016. Performance on one benchmark do...
Should we expect the future to be good?
by Neil Crawford — 2025-04-30T00:36:04.365Z

1. Introduction

Should we expect the future to be good? This is an important question for many reasons. One such reason is that the answer to this question has implications for what our intermediate goals should be. If we should expect the future to be good, then it would be relatively more important for us to focus on ensuring that we survive long into the future, e.g. by working on mitigating extinction risks. If we should not expect the future to be good, then it would be relatively more important for us to focus on mitigating risks of astronomical suffering.

In this paper, I critique Paul Christiano's (2013) argument that the future will be good. In Section 2, I reconstruct Christiano's argument in premise form and articulate some simplifying assumptions which I later relax. In Section 3, I offer a model for exploring the composition of different groups in a population over time. In Section 4, I show that, when certain assumptions are made, the model can generate the kind of favourable result that Christiano seems to have in mind. In Section 5, I show that, when fanatics are included in the model, a favourable result is no longer guaranteed. In Section 6, I consider that different groups can have different levels of influence whilst having the same number of adherents (i.e. members or followers), and argue that we should be far from confident that those with our values, as opposed to fanatics, will disproportionately influence the future. In Section 7, I argue that, even in the absence of fanatics, we should be sceptical that those with our values will disproportionately influence the future. In Section 8, I argue that, even if those with our values disproportionately influence the future, it is unclear that they will successfully make the future go well. In Section 9, I conclude by situating this paper as part of a broader argument that we are in a position of deep uncertainty regarding whether the future will be good, by which I mean that we lack a principled way of assigning precise probabilities to the proposition that the future will be good, due in large part to the presumably vast number of parameters, many of which are currently unknown to us, that will collectively shape the future through their complex interactions.

2. Christiano's argument

In his post, titled 'Why might the future be good?', Paul Christiano (2013) argues that the future will...
Judging types of consequentialism by influence and normativity
by Cole Wyeth — 2025-04-29T23:25:49.126Z

Epistemic status: Many citations missing.

Itzhak Gilboa has defined an irrational decision as one where the decision-maker feels embarrassed when confronted with an analysis of their decision. Normativity is then advice to be less irrational (or, the practice of following all such advice in advance).

Is consequentialism normative? Are virtue-ethicists irrational?

I don't think that is inherently true. Batman can really, truly value not directly killing others, even if that ultimately leads to more death down the line. He simply has to center his values on himself - literally, to be self-centered - and specifically his own actions. It's conceivable to formulate a first-person utility function for Batman that takes in his experiences over the course of his life and assigns an arbitrarily low value to those that include actions where he kills someone.

Formal version

I claim that there exist utility functions for AIXI-like agents that depend on the action symbols of their interaction histories. In fact, a recent paper submission of mine (with Marcus Hutter) generalizes the AIXI model in a way that allows this sort of thing. It's not hard to do; it just leaves more "free parameters" than the reward-based utility function of the original AIXI, which makes it a much less specific theory, with many particular implementations that are not interesting.

Batman is not very consequentialist, even though he can sort-of be framed as a consequentialist from a first-person view. This first-person and action-oriented type of consequentialism is, however, incredibly permissive - whatever a person does, one might argue that they preferred it for its own sake[1].

For a subtly stronger type of consequentialism, we can be a little more strict, and allow Batman to care not about the internal choice to act but only the "realized" action itself and its consequences. For instance, perhaps Batman should not care whether he internally (almost metaphysically) chose to kill, but only whether he actually observed his hands strangling someone. But by designating "Batman's hands" as a particularly central focus of value in the universe, we still permit a lot of things that look unreasonable from the third person.

In other words, apparently-normative advice (e.g. "please just finish this by killing the Joker so that he doesn't break out of prison for the hundredth time and murder more innocents...
Bandwidth Rules Everything Around Me: Oliver Habryka on OpenPhil and GoodVentures
by Elizabeth — 2025-04-29T20:40:06.610Z

In this episode of our podcast, Timothy Telleen-Lawton and I talk to Oliver Habryka of Lightcone Infrastructure about his thoughts on the Open Philanthropy Project, which he believes has become stifled by the PR demands of its primary funder, Good Ventures.

Oliver’s main claim is that around mid 2023 or early 2024, Good Ventures founder Dustin Moskovitz became more concerned about his reputation, and this put a straight jacket over what Open Phil could fund. Moreover it was not enough for a project to be good and pose low reputational risk; it had to be obviously low reputational risk, because OP employees didn’t have enough communication with Good Ventures to pitch exceptions. According to Habryka.

That’s a big caveat. This podcast is pretty one sided, which none of us are happy about (Habryka included). We of course invited OpenPhil to send a representative to record their own episode, but they declined (they did send a written response to this episode, which is linked below and read at end of the episode). If anyone out there wants to asynchronously argue with Habryka on a separate episode, we’d love to hear from you.

Transcript available here.

Links from the episode:

An Update From Good Ventures (note: Dustin has deleted his account and his comments are listed as anonymous, but are not the only anonymous)

CEA announcing the sale of Wytham Abbey

OpenPhli career page

Job reporting to Amy WL

Zach’s “this is false”

Luke Muelhauser on GV not funding right of center work

Will MacAskill on decentralization and EA

Alexander Berger regrets the Wytham Abbey grant

Single Chan-Zuckerberg employee demanding resignation over failure to moderate Trump posts on Facebook

Letter from 70+ CZ employees asking for more DEI within Chan Zuckerberg Initiative.

OpenPhil’s response