trentmkelly/LessWrong-43k huggingface dataset:
An overview of areas of control work
by ryan_greenblatt — 2025-03-25T22:02:16.178Z

In this post, I'll list all the areas of control research (and implementation) that seem promising to me.

This references framings and abstractions discussed in Prioritizing threats for AI control.

First, here is a division of different areas, though note that these areas have some overlaps:

Developing and using settings for control evaluations (including technique iteration)

Better understanding control-relevant capabilities and model properties

Developing specific countermeasures/techniques in isolation (rather than via iterating in a setting)

Doing control experiments on actual current AI usage

Prototyping or implementing software infrastructure for control and security measures that are particularly relevant to control

Prototyping or implementing human processes for control

Conceptual research and other ways of researching control other than empirical experiments

Nearer-term applications of control to practice and accelerate later usage of control

This division is imperfect and somewhat arbitrary, but still feels useful to me in breaking things down. I don't think all of these areas warrant equal investment. For instance, maybe "developing and using settings for control evaluations" should get roughly 1/3 of current control resources, which is much more than an even split between areas would suggest. (The list is roughly in descending order of how much investment that area should get.)

Now I'll elaborate on projects within each of these areas.

Developing and using settings for control evaluations

Perhaps the main methodology in AI control is to run a control evaluation in some particular setting. The first control paper by Redwood (AI Control: Improving Safety Despite Intentional Subversion) exhibits this methodology in a simple code backdooring setting. Correspondingly, work on building control settings and then doing control evaluations in these settings is a key area.

Ultimately, control evaluations have two main purposes: evaluating risk (including establishing a case for risk or a case for safety) and developing better techniques for ensuring control. Right now, models aren't actually risky, so work which is directly focused on evaluating risk doesn't provide that much of the value (though getting a better sense of the overall viability of control does seem valuable, but might be difficult).

So, current work in this area has multiple aims:

Developing

...
FW25 Color Stats
by sarahconstantin — 2025-03-25T18:40:03.145Z

We’ve been here before. I’ve posted about color trends in fashion in 2023, twice in 2024, and now I’m doing it again.

Methodology Recap

Fashion operates on a seasonal cycle; designer ready-to-wear fall/winter collections are released in the spring (so stores have time to buy them). Vogue Magazine kindly hosts photos of all the “major” collections — this season included over 10,000 images from hundreds of designers.

My automated counts come from asking GPT-4o to name all the specific colors in the clothes in each image, and aggregating the results. I’m still using the same method detailed in this post, with similar results, except that of course the cost per query has declined and the rate limits have become negligible.

My manual counts, as usual, are restricted to non-neutral colors (no black, white, brown, beige, gray, or navy) to keep the time commitment borderline-reasonable.1

Results

As is the case every year, most colors are neutrals, and black is particularly dominant because this is a fall/winter season.2 We’ve got a lot of beiges, browns, and grays, punctuated by a few “bold basic” colors like red, blue, and green.

If we just look at non-neutrals, we see that red is clearly in the lead, which is typical of every year. We also see a lot of “jewel tones” — bold and dark shades of red, purple, blue, and green.

The manual results are fairly similar to the automated ones. In both versions, red is in the lead, and deep cool “jewel tones” make a strong showing. Yellow, light blue and pink, and olive green are also prominent.

We see some of what I’ve been calling “synonym effects” here. In my manual counts I categorized a lot of colors as “burgundy” that GPT-4o split across multiple categories like “burgundy”, “deep red”, “maroon”, “dark red”, and “dark burgundy”. As a result, “burgundy” took the #2 spot in the manual counts and was only #6 in the automated counts, but the automated top-30 colors included many more “burgundy synonyms.” Either way, the results refer to the same underlying reality — there’s a lot of dark purplish-reds out there.

Trend Changes

How have this season’s colors changed compared to last year’s fall/winter collections?

FW24 (Automated)

FW25 (Automated)

The most striking changes in the neutrals this year are:

fewer metallic colors (gold & silver)

more browns, beiges, and grays

FW24, automated non-neutral

FW25, automated non-neutral

If we look at...
Synthetic Neuroscience
by hpcfung — 2025-03-25T17:45:05.916Z

Abstract: Neuroscience as engineering -- understanding how the brain works by building simplified, effective models of it, while requiring the model to behave as a real animal would.

Motivation

Suppose we want to build an AGI. The only working human-level general intelligence we know of is the human brain. So if the goal is to guarantee the creation of a successful AGI, then we should study the human brain.

For ethical and practical reasons we do not want to copy the brain exactly. Rather, the goal is to understand the brain well enough, so that we can construct an intelligence with analogous capabilities. This differs from computational neuroscience, where the goal of simulations is to better understand biological systems, especially to answer isolated, more narrow scoped research questions; or neuromorphic computing, where performance supercedes biological accuracy.

A full brain simulation, down to the molecular level guarantees success but is impossibly expensive computationally. And even so the brain would still be a black box to us so this is undesirable. Instead we look for effective models that at least exhibit the same qualitative behavior as the brain. For example we may limit ourselves to modelling at the spike train level and ignore details such as gene upregulation. (The level of detail needed is still unknown.) To reproduce the same qualitative behavior, a simulated mouse brain should be capable of the same things a real mouse can do: eg not just motor skills and vision in isolation, but complicated tasks like foraging and hunting, navigation, and learning.

Outline

Our research program is as follows. Much like synthetic biology, we shall follow the philosophy, "What I cannot create, I do not understand." At each stage, we will test our effective model by testing its performance in some environment, and compare that with the performance of a test animal in such an environment. Eg we can compare the amount of time needed for an animal to learn to walk, behavior when placed in a new location, etc. If our effective model has significantly inferior performance, then we must have missed an important element when simplifying the true biological model, and we must go back, identify it, and incorporate it. In a sense we, the experimenter, are performing artificial selection to search for the model architecture. Instead of optimizi...
How much does it cost to back up solar with batteries?
by jasoncrawford — 2025-03-25T16:35:52.834Z

Solar has an average capacity factor in the US of about 25%

. Naively, you might think that to turn this into a highly-available power source, you just need to have 4x the solar panels, plus enough batteries to store 75% of a day’s worth of power. E.g., for each continuous megawatt you want to supply, you need 4 MW of solar panels, and 18 MWh of batteries. During the day, you supply 1 MW from the panels and use the other 3 MW to charge the batteries. Overnight, you discharge the batteries to supply continuous power.

Turns out it’s not quite that simple. First, the capacity factor varies throughout the year, as the days get shorter in winter. So you at least need to build enough that even on the shortest day of the year, you can charge up enough to last through the longest night. (Hopefully you’re not inside one of the polar circles… at the poles, you’d need enough batteries to last through 6 months of darkness.)

Second, cloud cover will reduce the amount of power received in irregular and hard-to-predict ways. Here’s a chart

from Casey Handmer

showing the fluctuations in daily power at one site in Texas, which shows both seasonal and random variation:

Third, the load you need to supply is often not continuous. Residential loads vary a lot throughout the day and throughout the year. The more the load is weighted towards nighttime and winter, the more batteries and extra solar panels you’ll need. And the load can also vary in hard-to-predict ways.

So the cost of the system depends a few factors. One is your latitude. But another is exactly how available you want the system to be. And it turns out that the costs rise steeply as you approach 100%. That is, if you want to prepare for the absolute worst case—a run of very cloudy days in winter, combined with higher-than-expected load occurring at inconvenient times—you have to build a lot panels and batteries.

Brian Potter recently

modeled

a simple solar-plus-battery configuration. In his model, for a house in Atlanta, solar with no batteries costs 5.7¢/kWh, but of course only covers about 25% of the needed electricity. To supply 80%, a kWh of electricity now costs 19¢. At 90%, it’s about 25¢; and at 99%, it’s 40¢:

Casey Handmer

found a similar curve

:

As did

this report on off-grid solar for data centers

:

How much reliability do we need?

The US power grid has about 3-nines uptime (that is, 99.9%). This is considered unrelia...
Goodhart Typology via Structure, Function, and Randomness Distributions
by JustinShovelain — 2025-03-25T16:01:08.327Z

(Work done at Convergence Analysis. The ideas are due to Justin. Mateusz wrote the post. Thanks to Olga Babeeva for feedback on this post.)

In this post, we introduce the typology of structure, function, and randomness that builds on the framework introduced in the post

Goodhart's Law Causal Diagrams

. We aim to present a comprehensive categorization of the causes of Goodhart's problems.

But first, why do we care about this?

Goodhart's Law recap

The standard definition of

Goodhart's Law

is: "when a proxy for some value becomes the target of optimization pressure, the proxy will cease to be a good proxy.".

More specifically: we see a meaningful statistical relationship between the values of two random variables I and T,[1] where we have some preferences over the values of T. We decide to change the value of I, expecting that the value of T will change as would be expected by naively extrapolating the statistical tendencies of the two variables. We fall victim to Goodhart's Law (i.e. a Goodhart failure occurs) if the change in T doesn't quite meet our expectations and, upon learning that, we see that we should have intervened in some different way.

Some motivation

Many alignment failures can be cast in terms

of Goodhart's Law

. Specifically, the AI is given an objective that doesn't fully specify what we want it to optimize for (

outer alignment

), or it finds a way to exploit the misspecification of the training goal, developing an internal objective that is misaligned with the one we specified (

inner alignment

/

mesaoptimization

). Therefore, a better understanding of Goodhart's Law — its various subtypes, causes, and possible remedies — might better inform us about how to effectively anticipate and prevent (or at least mitigate) those failure modes. In addition to technical AI alignment/safety, a better understanding of Goodhart's Law might shed some light on enhancing human coordination, given that the same outer and inner alignment problems (or

close analogs thereof

) recur in coordination structures. In particular, effective AI governance will most likely require specifications of regulations that are robust to goodharting.

Introduction

In the

previous post

, we presented how causal diagrams can be used to describe situations vulnerable to Goodhart's problems. In this post, we build on...
Feature Hedging: Another way correlated features break SAEs
by chanind — 2025-03-25T14:33:08.694Z

This work was done as part of MATS 7.0. We consider this in-progress research and we are grateful for any thoughts and feedback from the community.

Update (May 20, 2025): This is now a paper! Check out our paper "Feature Hedging: Correlated Features Break Narrow Sparse Autoencoders" (arxiv.org/abs/2505.11756).

Update (April 4, 2025): We found that hedging caused by positive (but not hierarchical) correlation between features can sometimes be removed with a high enough sparsity penalty, but this is never true for hedging caused by hierarchical features or negatively correlated features. We have updated the post accordingly.

Introduction

If there is any correlation between a feature captured by an SAE and a feature not captured by that SAE, the SAE will merge the external feature into its latent tracking the internal feature. This phenomenon, which we call feature hedging, is caused by MSE reconstruction loss and will happen any time the SAE is both too narrow to capture all the “true features” in the model, and there is correlation between the features tracked by the SAE and the features not tracked by the SAE. Both of these conditions are almost certainly true of every SAE trained on an LLM, especially narrower SAEs like the inner parts of Matryoshka SAEs. This means all LLM SAE latents likely have spurious components of correlated features mixed in, harming the performance of the SAE and likely contributing to the underwhelming performance of SAEs.

We also present a novel reconstruction loss, called snap loss, which solves feature hedging in toy models. However, it does not work if the amount of feature correlation is too high, or there are too many correlated features; so snap loss may not be useful in real LLM SAEs.

When an SAE is too narrow to represent two correlated features, MSE loss incentivizes the SAE to mix components of both features into its latent to reduce a large squared-error penalty when the secondary feature fires. This causes the SAE to learn incorrect latents that mix components of multiple features together.

In this post, we begin by defining feature hedging and comparing it to feature absorption. We then demonstrate hedging using a simple toy model of a single-latent SAE. We present snap loss as a potential solution to hedging that works in toy models, but it is not clear if it improves LLM SAEs. We then explore hedging in Mat...
On (Not) Feeling the AGI
by Zvi — 2025-03-25T14:30:02.215Z

Ben Thompson interviewed Sam Altman recently about building a consumer tech company, and about the history of OpenAI. Mostly it is a retelling of the story we’ve heard before, and if anything Altman is very good about pushing back on Thompson when Thompson tries to turn OpenAI’s future into the next Facebook, complete with an advertising revenue model.

It is such a strange perspective to witness. They do not feel the AGI, let alone the ASI. The downside risks of AI, let alone existential risks, are flat out not discussed, this is a world where that’s not even a problem for Future Earth.

Then we contrast this with the new Epoch model of economic growth from AI, which can produce numbers like 30% yearly economic growth. Epoch feels the AGI.

A Note on GPT-2

Given the discussion of GPT-2 in the OpenAI safety and alignment philosophy document, I wanted to note his explanation there was quite good.

Sam Altman: The GPT-2 release, there were some people who were just very concerned about, you know, probably the model was totally safe, but we didn’t know we wanted to get — we did have this new and powerful thing, we wanted society to come along with us.

Now in retrospect, I totally regret some of the language we used and I get why people are like, “Ah man, this was like hype and fear-mongering and whatever”, it was truly not the intention. The people who made those decisions had I think great intentions at the time, but I can see now how it got misconstrued.

As I said a few weeks ago, ‘this is probably totally safe but we don’t know for sure’ was exactly the correct attitude to initially take to GPT-2, given my understanding of what they knew at the time. The messaging could have made this clearer, but it very much wasn’t hype or fearmongering.

What Even is AGI

Altman repeatedly emphasizes that what he wanted to do from the beginning, what me still most wants to do, is build AGI.

Altman’s understanding of what he means by that, and what the implications will be, continues to seem increasingly confused. Now it seems it’s… fungible? And not all that transformative?

Sam Altman: My favorite historical analog is the transistor for what AGI is going to be like. There’s going to be a lot of it, it’s going to diffuse into everything, it’s going to be cheap, it’s an emerging property of physics and it on its own will not be a differentiator.

This seems bonkers...
Apply to the Cambridge ERA:AI Fellowship 2025
by Harrison G — 2025-03-25T13:50:04.665Z

The Cambridge ERA:AI Fellowship has opened applications for an in-person, paid, 8-week Summer Research Fellowship focused on catastrophic AI risk mitigation, taking place from 29th June to 24th August 2025 in Cambridge, UK, and aimed at a variety of aspiring researchers.

To apply and find out more, please visit the

ERA website

. The deadline to apply is April 8th, 2025 (11:59pm UTC).

If you are interested in mentoring fellows on this programme, please submit your name, email and research area

here

,

and we will get in touch with you in due course.

For more information, you can also sign up for our information and application Q&A sessions:

Signup to the info session at 16:30 in UTC on 26th March (Q&A 1)
Signup to the info session at 16:30 in UTC on 1st April (Q&A 2)

If you know other people who would be a good fit, please encourage them to apply (people are more likely to apply if you recommend they do, even if they have already heard of the opportunity!) If you are a leader or organiser of relevant community spaces, we encourage you to post an announcement with a link to this post.

Applications will be reviewed as they are submitted, and we encourage early applications, as offers will be sent out as soon as suitable candidates are found. We will accept applications until April 8th, 2025 (23:59 UTC).

The Cambridge ERA:AI Fellowship is a fantastic opportunity to:

Build your portfolio by researching a topic relevant to understanding and mitigating risks from frontier AI (in our 3 research areas of Technical AI Safety, AI Governance, and Technical AI Governance)
Receive guidance and develop your research skills, via weekly mentorship from a researcher in the field.
Form lasting connections with other fellows who care about mitigating AI risks, while also engaging with local events including discussions and Q&As with experts.

Why we are running this programme

Our mission as an organisation is to reduce the probability of catastrophic outcomes caused by AI. We believe that one of the key ways to reduce this risk lies in fostering a community of dedicated and knowledgeable AI safety, security, & governance researchers. Through our summer research fellowship programme, we aim to identify and support aspiring researchers in this field, providing them with the resources and the mentorship needed to succeed.

Wha

...
23andMe potentially for sale for <$50M
by lemonhope — 2025-03-25T04:34:28.388Z

It seems the company has gone bankrupt and wants to be bought and you can probably get their data if you buy it. I'm not sure how thorough their transcription is, but they have records for at least 10 million people (who voluntarily took a DNA test and even paid for it!). Maybe this could be useful for the embryo selection/editing people. This is several times cheaper than last April.

The market cap is presently $19.5M but I'm not sure how that relates to the sale price.
The Determinants of Controllable AGI
by A2z — 2025-03-25T00:43:28.627Z

Allen Schmaltz, PhD

We briefly introduce, at a conceptual level, technical work for deriving robust estimators of the predictive uncertainty over large language models (LLMs), and we consider the implications for real-world deployments and AI policy.

The Foundational LLM Limitation: An Absence of Robust Estimators of Predictive Uncertainty

The limitations of unconstrained LLMs, which includes the more recent RL-based reasoning-token models, are readily evident to end-users. Hallucinations, highly-confident wrong answers, and related issues diminish their benefits in most real-world settings. The punchline is that the end-user has no means of knowing whether the output can be trusted, beyond carefully checking the output, which precludes model-based automation for most complex, multi-stage pipelines.

The foundational problem for all of the large models with non-identifiable parameters is that there has been an absence of meaningfully reliable estimators of the predictive uncertainty over the high-dimensional outputs. It is a very challenging problem, because the parameters of the models are non-identifiable [3]; the parameter counts are typically in the billions or more; and given typical use-cases of LLMs, we seek approximately conditional (rather than marginal) estimates. This latter point is subtle but critical. Most benchmarks and evaluations for LLMs provide scores or metrics (e.g., overall accuracy) that are averaged over entire test sets. Such evaluations can be useful for broad comparisons and winnowing of models (i.e., at the unit-of-analysis of models), but they are insufficiently informative to serve as estimators at the instance level for end-users over complex, high-dimensional outputs---especially those for which distribution shifts are the norm.

Conversely, with a reliable and robust estimator of the predictive uncertainty that has a human interpretable mapping to applicable partitions of the observed data, the primary technical limitations of LLMs (and AI models, more generally) would be effectively resolved. When such an estimator is uncertain, the model could then refrain from a final decision and take another branching action. While the classification-with-reject-option is sometimes viewed as too coarse of a final quantity of interest---and perhaps an unsatisfying denouement to the uncertainty endeavor---it is in fact exactly what is neede...
New Bill AB 501 to Prevent OpenAI's Non-profit Conversion
by Peter Windberger — 2025-03-25T00:41:07.617Z

AB 501 is a new bill about to be introduced in the California Assembly to prevent the conversion of a non-profit like OpenAI to a for-profit.

The bill aims to ensure that non-profits that make a certain commitment on their charters are held accountable to their commitment. In OpenAI's case, their legally-binding commitment has been to build safe AGI to benefit all of humanity rather than to unduly concentrate power in the hands of a few individuals or use AGI for private profit.

This bill aims to prevent what The Zvi has called "The greatest theft of the Millennium -- if not of all of human history."

If you believe that OpenAI should be held accountable to its charter, please support AB 501 by simply signing the digital open letter. Collecting enough signatures will make a big difference towards the success of the bill. If you believe in the goal of the bill, please spread the open letter.

Sign the digital open letter at AB501.com/letter