trentmkelly/LessWrong-43k huggingface dataset:
The optimizer won’t just guess your intended semantics
by Thomas Kehrenberg — 2025-03-06T19:42:12.682Z

A desirable property of an AI’s world model is that you as its programmer have an idea what’s going on inside. It would be good if you could point to a part of the world model and say, “This here encodes the concept of a strawberry; here is how this is linked with other concepts; here you can see where the world model is aware of individual strawberries in the world.” This seems, for example, useful for directly specifying goals in the world – like “go and produce diamonds” – without having to do reinforcement learning or some other kind of indirect goal learning; if we knew how to find diamonds in the AI’s world model, we could directly write a goal function.

But you won’t get this kind of understandable world model by default, if you use something like Solomonoff induction Turing machines or gradient descent on transformers. Both of these will produce a model for describing the world, but you cannot easily look inside. With sufficient effort, you can reverse-engineer what is happening in the Turing machine (keeping in mind a Turing machine’s unlimited ability to do meta-programming) or in the giant floating-point array, but you might think that there should be a better way.

You might think, can’t we just learn the world model as something that’s not a Turing machine or a floating-point array? Perhaps a data structure that we can impose some semantics on, and that we therefore know how to interpret?

Well, it’s not so easy, actually. Let’s go through an example.

Consider a simple Solomonoff induction over Turing machines:

We get observations from the world, and feed them to the Turing machine, which then predicts the next observation. We have a prior over all Turing machines, and then we restrict our posterior to those that make correct predictions given past observations. The TM handles both the state and the dynamics of the world. Understanding what is happening within the TM is very difficult.

Here is one idea to get a more interpretable world model: instead of finding a Turing machine that has free reign over how to describe the state and the dynamics of the world, we’ll fix the world dynamics (i.e., the time evolution function) to a function we wrote ourselves based on our understanding of physics, and then we’ll let the induction algorithm search only for the initial state, and some bridging laws. The hope is that because the dynamics have fixed semantics, t...
AISN #49: Superintelligence Strategy
by Corin Katzke — 2025-03-06T17:46:50.965Z
Link: https://newsletter.safe.ai/p/ai-safety-newsletter-49-superintelligence

Welcome to the AI Safety Newsletter by the

Center for AI Safety

. We discuss developments in AI and AI safety. No technical background required. In this newsletter, we discuss two recent papers: a policy paper on national security strategy, and a technical paper on measuring honesty in AI systems.

Listen to the AI Safety Newsletter for free on

Spotify

or

Apple Podcasts

.

Superintelligence Strategy

CAIS director Dan Hendrycks, former Google CEO Eric Schmidt, and Scale AI CEO Alexandr Wang have authored a new paper,

Superintelligence Strategy

. The paper (and an in-depth expert version) argues that the development of superintelligence—AI systems that surpass humans in nearly every domain—is inescapably a matter of national security.

In this story, we introduce the paper’s three-pronged strategy for national security in the age of advanced AI: deterrence, nonproliferation, and competitiveness.

Deterrence

The simultaneous power and danger of superintelligence presents a strategic dilemma. If a state successfully builds superintelligence, it will either retain or lose control of that system. If it retains control, it could undermine the security of its rivals. However, if it loses control, it could undermine the security of all states.
Given that either outcome endangers that state’s rivals, those rivals would likely threaten or employ preventive sabotage to deter or disrupt a project to build superintelligence. For example, rivals could threaten cyberattacks on destabilizing AI projects. The threat of intervention alone could be sufficient to deter destabilizing AI projects.

This dynamic mirrors Mutual Assured Destruction (MAD) from the nuclear strategy, in which no state attempts a first strike for fear of retaliation. In the context of AI, the authors of the paper propose a deterrence regime they call Mutual Assured AI Malfunction (MAIM). If a state attempts a bid for AI dominance, it risks triggering a response from its competitors.

Maintaining a stable MAIM regime requires clear escalation protocols, AI infrastructure transparency, and building datacenters away from cities to prevent attacks from escalating into broader conflicts.

Nonproliferation

Deterrence may constrain state actors, but ensuring national security also requires limiting the proliferation of AI capabilities to terrorist groups and non-state actors. Advanced AI systems could be weaponized by nonstate acto...
Decision-Relevance of worlds and ADT implementations 
by Maxime Riché — 2025-03-06T16:57:42.966Z

Crossposted on the EA Forum.

Lots of effort has been put into estimating the density of Space-Faring Civilisations (SFCs) in the universe. Further work built on these results to produce decision-relevant information assuming various anthropic updates (SSA, SIA, ADT)(Finnveden 2019, Olson 2020, Olson 2021, Cook 2022). In this post, we look at how these works update Space-Faring Civilization (SFC) density estimates to produce decision-relevant data. We reframe these updates in the intuitive framing of wagers over possible worlds, expressing all these results in terms of what we call ADT implementations, in which each implementation uses a different method to estimate the Decision-Relevance of possible worlds, which we define as the world-variant-only part of marginal utility calculations. Finally, we review which ADT implementations make the most sense for impartial longtermists, how existing works cover, or fail to cover, part of the space of plausible ADT implementations, and thus how they may, or fail to, produce decision-relevant results.

Sequence:

This post is part 3 of a sequence investigating the longtermist implications of alien Space-Faring Civilizations. Each post aims to be standalone. You can find an introduction to the sequence in the following post.

Summary

This post explores defining the Decision-Relevance (DR) of possible worlds as the world-variant-only component of marginal utility calculations, as summarized in the following equation.

It shows how the DR of a world can be decomposed into three factors:

(a) P(W): the likelihood of the possible world:
(b) NUS(DIWP): a normalized causal utility ratio: how much better is the causally impacted part of the world when compared to the expectation over all worlds.
(c) CorrM(DIWP): a correlation multiplicator accounting for the impact of correlated agents (equal to zero for pure CDT believers).

DR(W)=|P(W)⋅NUS(DIWP)⋅(1+CorrM(DIWP))|

This post then shows that different anthropic updates (SSA, SIA, ADT) can be reframed as ADT implementations, each associated with different beliefs and methods for computing the Decision-Relevance Potential (which excludes P(W)) of possible worlds.

We finally apply the framing of ADT implementations to study how the Decision-Relevance Potential of worlds changes depending on the density of Space-Faring Civilizations (SFCs). We list two novel AD...
AI #106: Not so Fast
by Zvi — 2025-03-06T15:40:05.919Z

This was GPT-4.5 week. That model is not so fast, and isn’t that much progress, but it definitely has its charms.

A judge delivered a different kind of Not So Fast back to OpenAI, threatening the viability of their conversion to a for-profit company. Apple is moving remarkably not so fast with Siri. A new paper warns us that under sufficient pressure, all known LLMs will lie their asses off. And we have some friendly warnings about coding a little too fast, and some people determined to take the theoretical minimum amount of responsibility while doing so.

There’s also a new proposed Superintelligence Strategy, which I may cover in more detail later, about various other ways to tell people Not So Fast.

Table of Contents

Also this week: On OpenAI’s Safety and Alignment Philosophy, On GPT-4.5.

Language Models Offer Mundane Utility. Don’t get caught being reckless.

Language Models Don’t Offer Mundane Utility. Your context remains scarce.

Choose Your Fighter. Currently my defaults are GPT-4.5 and Sonnet 3.7.

Four and a Half GPTs. It’s a good model, sir.

Huh, Upgrades. GPT-4.5 and Claude Code for the people.

Fun With Media Generation. We’re hearing good things about Sesame AI voice.

We’re in Deep Research. GIGO, welcome to the internet.

Liar Liar. Under sufficient pressure, essentially all known LLMs will lie. A lot.

Hey There Claude. Good at code, bad at subtracting from exactly 5.11.

No Siri No. It might be time for Apple to panic.

Deepfaketown and Botpocalypse Soon. Rejoice, they come bearing cake recipes.

They Took Our Jobs. More claims about what AI will never do. Uh huh.

Get Involved. Hire my friend Alyssa Vance, and comment on the USA AI plan.

Introducing. Competition is great, but oh no, not like this.

In Other AI News. AI agents are looking for a raise, H100s are as well.

Not So Fast, Claude. If you don’t plan to fail, you fail to plan.

Not So Fast, OpenAI. Convert to for profit? The judge is having none of this.

Show Me the Money. DeepSeek has settled in to a substantial market share.

Quiet Speculations. Imminent superintelligence is highly destabilizing.

I Will Not Allocate Scarce Resources Using Prices. That’s crazy talk.

Autonomous Helpful Robots. It’s happening! They’re making more robots.

The Week in Audio. Buchanan, Toner, Amodei, Cowen, Dafoe.

Rhetorical Innovation. Decision theory only saves you if you make good decisions.

No One Would

...
Can a finite physical device be Turing equivalent?
by Noosphere89 — 2025-03-06T15:02:16.921Z
Link: https://lifeiscomputation.com/can-a-finite-physical-device-be-turing-equivalent/

Here's some very useful quotes by the article, and the short version is yes, you can consider a finite computer in the real world to be Turing-complete/Turing-universal/Turing-equivalent, because you can always write code that correctly computes a problem and have that code always work, no matter how much memory or time or energy they use in modern computers.

In essence, physical limitations don't matter for the purpose of determining whether a computational system is Turing-complete/Turing-universal/Turing-equivalent, it's whether the code you wrote for a problem can arbitrarily scale with more resources and time, or whether more resources and time fundamentally require you to rewrite new code to deal with larger inputs:

When we say Turing machines require “unbounded memory”, what we mean is that memory cannot be bounded by the systems descriptor, not that it cannot be bounded by other things such as the laws of physics or resource constraints. Turing-equivalence only requires a system in which memory usage grows, not one in which memory is infinite.

Finite state machines are no less of a mathematical abstraction than Turing machines are. And both are relevant to studying the real world. You can implement a finite state machine by creating something that recruits more energy when needed. Similarly, you can implement a Turing-equivalent machine by creating something that recruits more memory space [and energy] when needed. The fact that both of these will eventually hit some limit does not make them physically unrealizable.

A physical implementation of Turing machines (with practical constraints on its memory, time, and energy) is more powerful than a physical implementation of finite state machines (with practical constraints on its time and energy). This is because they both still have the same descriptor/input/output relationships as their abstract models. There exists a Turing machine descriptor that can solve the counting problem irrespective of what the physical constraints on memory, energy and time are.

On the other hand, there is no single descriptor of a finite state machine that can solve the counting problem. You may be able to construct one that can count up to some very large number, but the descriptor of your finite state machine will have to change if the maximum input size changes.

Firstly, it is unbounded memory, not infinite memory, that is nece

...
We should start looking for scheming "in the wild"
by Marius Hobbhahn — 2025-03-06T13:49:39.739Z

TLDR: AI models are now capable enough that we might get relevant information from monitoring for scheming in regular deployments, both in the internal and external deployment settings. We propose concrete ideas for what this could look like while preserving the privacy of customers and developers.

What do we mean by in the wild?

By “in the wild,” we basically mean any setting that is not intentionally created to measure something (such as evaluations). This could be any deployment setting, from using LLMs as chatbots to long-range agentic tasks.

We broadly differentiate between two settings

Developer-internal deployment: AI developers use their AI models internally, for example, as chatbots, research assistants, synthetic data generation, etc.
Developer-external deployment: AI chatbots (ChatGPT, Claude, Gemini, etc.) or API usage.

Since scheming is especially important in LM agent settings, we suggest prioritizing cases where LLMs are scaffolded as autonomous agents, but this is not a hard criterion.

What are we looking for?

Broadly speaking, we’re looking for any setting where an AI model acts covertly or reasons about acting covertly against an explicit or implicit goal of the developer/user.

This setting does not include refusals or other kinds of explicit misalignment because they are not covert.

Examples include:

Most settings in the

in-context scheming

or

alignment-faking

papers.
Any setting where the model wants to refuse a request covertly, e.g. pretending that it is unable to do it instead of unwilling (and acknowledging that explicitly in the CoT or it being visible between settings like

here

).
Any setting where the model has some core values from its training and decides to act on them in a way that is hidden from its developers/users (these values don’t have to be evil, as we see in the alignment faking paper; these can be HHH values as well).
Any setting where the model explicitly decides to use a covert strategy because it doesn’t like the perceived goals of the developers/users.

Importantly, the class of “scheming in-the-wild” cases we’re looking for is much bigger than “

catching AIs red-handed

” and includes less egregious and many more cases of scheming. We mostly just want to get a sense of how often models show this kind of behavior rather than demonstrating any particularly concerning case. ...
Bounded AI might be viable
by Mateusz Bagiński — 2025-03-06T12:55:46.224Z

(Work done at Convergence Analysis. Mateusz wrote the post and is responsible for most of the ideas with Justin helping to think it through. Thanks to Olga Babeeva for the feedback on this post.)

1. Motivation

Suppose the perspective of pausing or significantly slowing down AI progress or solving the technical problems necessary to ensure that arbitrarily strong AI has good effects on humanity (in time, before we get such systems) both look gloomy.[1] What options do we have left?

Adam Shimi presents a useful frame on the alignment problem in Abstracting The Hardness of Alignment: Unbounded Atomic Optimization:

alignment [is] the problem of dealing with impact on the world (optimization) that is both of unknown magnitude (unbounded) and non-interruptible (atomic).

If the problem is about some system (or a collection of systems) having an unbounded, non-interruptible impact,[2] can we handle it by ensuring that the impact of the systems in question is reliably bounded and interruptible (even if the systems are scheming against us, or trying to "liberate" themselves from the properties of boundedness and interruptibility)?

The core idea is to impose a multidimensional bound on the system's capabilities such that:

it is sufficiently capable along the dimensions corresponding to the capabilities we want to extract from it
but

sufficiently incapable along the dimensions corresponding to capabilities that would (in combination with the capabilities we want it to have) grant it the dangerous power of unbounded, non-interruptible optimization.[3]

In other words, if we can't solve the problem of aligning unbounded, non-interruptible optimizers, we may be able to avoid this problem by:

not building unbounded, non-interruptible optimizers
and, instead,
building some other, safer, kind of AI that can be demonstrated to deliver enough value to make up for the giving up on the business-as-usual kind of AI along with the benefits it was expected to deliver (that "we", though not necessarily its creators, expect might/would lead to the creation of unbounded, non-interruptible AI posing a catastrophic risk),.

Unfortunately, at the current moment, frontier AI labs don't take actions sufficient to mitigate the risks from unbounded non-inter...
Publish your genomic data
by samuelshadrach — 2025-03-06T12:39:06.773Z

Publish your genomic data to the public domain as per CC0 license. All submitted data will be available to anyone for download.

Link to form upload

Why?

Some of the largest genomic datasets today are not in the public domain. If you believe genomic datasets in public domain should exist, you can volunteer your own personal data to it.

Examples of datasets that are not in public domain as of today (2025-03-06):

- Academic: UK Biobank (500,000 samples), All of Us by NIH (250,000 samples)

- Commercial: AncestryDNA (25 million samples), 23andMe (14 million samples)

I could make a website if there's enough interest.
Which meat to eat: CO₂ vs Animal suffering
by B Jacobs — 2025-03-06T12:37:08.606Z
Link: https://bobjacobs.substack.com/p/which-meat-to-eat-co-vs-animal-suffering

Animal agriculture generates an ungodly amount of animal suffering and greenhouse gas emissions. Ideally, everyone would adopt a vegan diet[1], but since this appears to be too much to ask for most people, the next question becomes: which food is the least bad?

Here we run into an uncomfortable observation. Meat from larger animals tends to produce more CO₂, while meat from smaller animals tends to cause more animal suffering.
This makes intuitive sense: when converting animals into a set mass of meat, more individual animals must suffer if they are smaller. We can see this in the data, e.g., we slaughter 309 million cows per year, while the number of chickens slaughtered exceeds 75 Billion per year.

Let’s start ranking

Let's start by comparing greenhouse gas emissions. Which foods have the lowest CO₂ impact?
Here’s how many kilograms of CO₂-equivalized emissions are produced per kilogram of food (some create methane etc, but that’s accounted for by converting it into the same unit), ranked from worst to best:

Beef (beef herd)     99.48 kg
Lamb & Mutton    39.72 kg
Beef (dairy herd)   33.3 kg
Prawns (farmed)   26.87 kg
Cheese                       23.88 kg
Fish (farmed)          13.63 kg
Pig Meat                   12.31 kg
Poultry Meat           9.87 kg
Eggs                             4.67 kg
Rice                              4.45 kg
Tofu                             3.16 kg
Milk                             3.15 kg
Soy milk                     0.98 kg

As we can see, larger animals tend to create more CO₂.
Can we create a similar ranking for animal suffering? Yes we can, if we take the sentience figures from this report and apply it to the production process used to create the meat. Here’s the equivalized days of suffering per kg of food (some have bigger brains etc, but that’s accounted for by converting it into the same unit), ranked from worst to best:

Farmed catfish        290 d
Battery cage eggs   42 d
Farmed salmon       27 d
Chicken                      27 d
Turkey                         15 d
Pork                              3.9 d
Beef                              0.8 d
Milk                              0.049 d
Tofu                              0 d
Soy milk                      0 d

As we can see, the smaller animals tend to produce more suffering. The best products on these lists, both in terms of animal suffering and CO₂, are tofu and soy milk.
For those who don't want to go v...
Musings on Scenario Forecasting and AI
by Alvin Ånestrand — 2025-03-06T12:28:19.662Z
Link: https://forecastingaifutures.substack.com/p/musings-on-ai-scenario-forecasting

I have yet to write detailed scenarios for AI futures, which definitely seems like something I should do considering the title of my blog (Forecasting AI Futures). I have speculated, pondered and wondered much in the recent weeks—I feel it is time. But first, I have some thoughts about scenario forecasting.

The plan:

Write down general thoughts about scenario forecasting with special focus on AI (this post).
Write one or two specific scenarios for events over the coming months and years.
Wait a few months, see what comes true, and update my scenario forecasting methods.

Other work

In 2021, Daniel Kokotaljo published a scenario titled

What 2026 looks like

. He managed to predict many important aspects of the progression of AI between 2021 and now, such as chatbots, chain-of-thought, and inference compute scaling.

Now he is collaborating with other forecasters—including Eli Lifland, a

superforecaster

from the

Samotsvety forecasting group

—to develop a highly detailed and well-researched scenario forecast under the

AI Futures Project

. Designed to be as predictively accurate as possible, it illustrates how the world might change as AI capabilities evolve. The scenario is scheduled to be published in Q1 of this year.

I also recommend reading

Scale Was All We Needed, At First

and

How AI Takeover Might Happen in 2 Years

, two brilliant stories exploring scenarios with very short timelines to superintelligence.

Using and Misusing Scenario Forecasting

People may hear about a specific superintelligence disaster scenario, and then confidently say something like “That seems entirely implausible!” or “AIs will just be tools, not agents!” or “If it tried that, humans could just [insert solution].”

There is some fundamental issue here. Those who see significant risks struggle to convey their concerns to those who think everything is fine.

And I think this problem at least partly has to do with scenario forecasting. One side is envisioning specific bad scenarios, which can easily be refuted. The other side is envisioning specific good scenarios, which can also be easily refuted.

The question we should consider is something more like “Out of all possible scenarios, how many lead to positive outcomes versus negative ones?”. But this question is harder to reason about, and any reasoning about it takes longer to convey.

We can start by considering what avoiding the major risks would mean. The w...