trentmkelly/LessWrong-43k huggingface dataset:
Lurking in the Noise
by J Bostock — 2025-06-25T13:36:34.174Z

The Dead Biologist

Suppose you're a biologist working with a bacterium. It's a nasty little bug, so you want to make sure your disinfecting protocol is effective. You have a rather good model of how the bacterium works, and it goes like this:

The bacterium contains DNA, which is like a master copy of the instructions as to how it works
This DNA is transcribed to RNA, which are little working copies of parts of the instructions and are disseminated throughout the bacterium
This RNA is translated into proteins, which actually do the stuff that makes the bacterium function

Yes, there are a few special cases where RNA actually does something, and your model seems to be missing some things---for one, the amount of protein produced from each RNA molecule seems to vary, and you can only account for about 90% of this variance; the other 10% appears to be noise. But overall it's really very good.

You test every protein the bacterium is producing. It has proteins that sense and respond to cold, to acid, to low oxygen, to desiccation, but not to heat. You notice that none of the bacterium's proteins work above 40 °C (around 107 F), so you heat up everything to 40 °C to sterilize it.

The bacterium survives, and kills you.

The Thing in the Noise

What happened? The bacterium transcribed an RNA which didn't get translated into protein. It was folded into a little knot. Above 40 °C, this knot wiggled open and got translated into a protein which went and got some more proteins produced, which protected the bacterium from heat. Your model didn't account for RNA having complex structure, which was also the 10% unexplained variance in protein yield from earlier.

"How unfair!" you lament. "What are the chances that, in that little 10% noise in my model, there was a thing that killed me?"

Quite a lot actually. A pathogenic bacterium is a thing which has evolved to kill you. Every single part of it has been under optimization pressure to kill you. And furthermore noise exists in the map, not the territory.

You can think of a bacterium in two ways, and live: you can think of it as a blob which is optimized to kill you; or you can think of it as a set of clean abstractions which are optimized to kill you, plus some noise which is also optimized to kill you. If you try to think of it as the clean abstractions which are optimized to kill you, plus some noise which is unimportant, it will kill you...
New Paper: Ambiguous Online Learning
by Vanessa Kosoy — 2025-06-25T09:14:14.872Z
Link: https://arxiv.org/abs/2506.19810

Abstract: We propose a new variant of online learning that we call "ambiguous online learning". In this setting, the learner is allowed to produce multiple predicted labels. Such an "ambiguous prediction" is considered correct when at least one of the labels is correct, and none of the labels are "predictably wrong". The definition of "predictably wrong" comes from a hypothesis class in which hypotheses are also multi-valued. Thus, a prediction is "predictably wrong" if it's not allowed by the (unknown) true hypothesis. In particular, this setting is natural in the context of multivalued dynamical systems, recommendation algorithms and lossless compression. It is also strongly related to so-called "apple tasting". We show that in this setting, there is a trichotomy of mistake bounds: up to logarithmic factors, any hypothesis class has an optimal mistake bound of either Θ(1), Θ(√(N)) or N.

This work is my first rigorous foray into compositional learning theory. Intuitively, one approach to learning compositionally is, constructing a model incrementally, adding one part each time. This raises the question of what does it mean to have a "partial" model. One notion of partial model is provided by infra-Bayesianism / robust RL. However, there the treatment of ambiguities in the model via the maximin decision rule is strongly tied to the choice of reward function. Intuitively, we might expect the process of incremental learning to not depend on the reward function that strongly.

In this work, I propose a notion of "partial" model in the online learning setting. I develop a theory of learning for this notion analogous (but more involved than) classical realizable Littlestone theory.

Does it truly lead to compositional learning? In results that don't appear in the paper, I showed that the "ambiguous Littlestone dimension" (an invariant I discovered that characterizes the complexity of ambiguous online learning) indeed behaves nicely w.r.t. some ways to decompose hypothesis classes. However, my attempts to use this framework to get computationally efficient compositional learning algorithms haven't produced much so far.

Moreover, our results on robust RL (in particular the notion of "imprecise estimator" we have there) suggests it's more natural to study a different (easier) version of online learning with ambiguity, where the learner is only allowed ...
Emergence of Simulators and Agents
by WillPetillo — 2025-06-25T06:59:31.404Z

In

Agents, Tools, and Simulators

we outlined what it means to describe a system through each of these lenses and how they overlap. In the case of an agent/simulator, our central question is: which property is "driving the bus" with respect to the system's behavior, utilizing the other in its service?

Aligning Agents, Tools, and Simulators

explores the implications of the above distinction, predicting different types of values (and thus behavior) from

An agent that contains a simulation of the world that it uses to navigate.
A simulator that generates agents because such agents are part of the environment the system is modelling.
A system where the modes are so entangled it is meaningless to think about where one ends and the other begins.

Specifically, we expect simulator-first systems to have holistic goals that internalize (an approximation of) human values and for agent-first systems to have more narrow values that push them towards maximization.

Given these conceptual distinctions, we observe that the best-fitting lens for past AI systems seems to correspond to their training process. In short, RL-based systems (like AlphaZero) seem almost purely agentic, pre-trained LLMs seem very simulator-like, and current systems that apply (increasing amounts of) RL feedback to LLMs are an entangled mix.

Assuming this correspondence is accurate, this post asks: why?

Why does RL Create Agents?

Reinforcement learning (RL) naturally gives rise to agentic behavior because of how it optimizes for long-term rewards. RL agents interact with an environment and adjust their strategies based on feedback. Crucially, their goals are not explicitly programmed but emerge through training. The objective function defines what is rewarded, but the model must infer how to achieve high reward by exploring possible actions. Over time, given sufficient training and model capacity, internal goals tend to align with maximizing reward. However, this process depends on avoiding local minima—suboptimal behaviors that yield short-term reward but fail to generalize. The structure of the environment therefore plays a critical role in pushing the agent toward deeper abstractions and strategic planning.

Consider a simple grid world environment where an RL agent is rewarded for collecting coins. If the coins always appear in fixed locati...
Defining Corrigible and Useful Goals
by Rubi J. Hudson — 2025-06-25T03:51:57.160Z

This post contains similar content to a forthcoming paper, in a framing more directly addressed to readers already interested in and informed about alignment. I include some less formal thoughts, and cut some technical details. That paper, A Corrigibility Transformation: Specifying Goals That Robustly Allow For Goal Modification, will be linked here when released on arXiv, hopefully within the next couple weeks.

Ensuring that AI agents are corrigible, meaning they do not take actions to preserve their existing goals, is a critical component of almost any plan for alignment. It allows for humans to modify their goal specifications for an AI, as well as for AI agents to learn goal specifications over time, without incentivizing the AI to interfere with that process. As an extreme example of corrigibility’s value, a corrigible paperclip maximizer could be stopped partway through a non-instantaneous takeover attempt by saying “please stop” or by automatically triggered safeguards, and it would be perfectly happy to accept that outcome.

The challenge is corrigibility is anti-natural, meaning that almost any goal introduces an instrumental incentive to preserve itself, since it is more likely to be achieved if an agent continues pursuing it. Despite the concept of corrigibility being introduced

a decade ago

, little progress has been made on defining corrigible goals that are still useful. I aim to address that here by introducing a corrigibility transformation that can be applied to almost any goal, creating a new version that is corrigible without sacrificing performance.

The corrigibility transformation works by first giving an AI system the ability to costlessly reject updates sent to it, which it decides whether to exercise each time it takes an action. So, for example instead of a “Shutdown” button there would be a “Please Shut Down” button. This makes it so that the expected discounted reward conditional on taking an action and rejecting any updates is a well defined quantity. Those values are then used as the reward for taking that action, whether or not updates are actually rejected, which is optimized for myopically. Some reward bonus is given for taking actions that would accept updates, to break the tie. The possibility of the costless rejection disincentivizes actions that incur a cost to avoid updates, and the reward structure then disincentiv...
Multispecies Metagenomic Calibration
by jefftk — 2025-06-25T02:50:11.378Z

Cross-posted from my NAO
Notebook.

This is something I wrote internally in late-2022. Sharing it now
with light edits, additional context, and updated links after the idea
came up at the Microbiology
of the Built Environment conference I'm attending this week.

Metagenomic sequencing data is fundamentally relative: each
observation is a fraction of all the observations in a sample. If you
want to make quantitative observations, however, like understanding
whether there's been an increase in the number of people with some
infection, you need to calibrate these observations. For example,
there could be variation between samples due to variation in:

Changes in how many humans are contributing to a sample.

Has it been raining? (Especially in areas with combined sewers, but
also a factor with nominally separated sewers.)

What have people been eating lately?

What temperature has it been?

How concentrated is the sample?

etc

If you're trying to understand growth patterns all of this is noise;
can we reverse this variation? I'm using "calibration" to refer to
this process of going from raw per-sample pathogen read counts to
estimates of how much of each pathogen was originally shed into sewage.

The simplest option is not to do any calibration, and just consider
raw relative abundance: counts relative to the total number of reads
in the sample. For example, this is what Marc
Johnson and Dave O'Connor are doing.

It seems like you ought to be able to do better if you normalize by
the number of reads matching some other species humans excrete. It's
common to use PMMoV
for this: peppers are commonly infected with PMMoV, people eat
peppers, people excrete PMMoV. All else being equal, the amount of
PMMoV in a sample should be proportional to the human contribution to
the sample. This is especially common in PCR work, where you take a
PCR measurement of your target, and then present it relative to a PCR
measurement of PMMoV. For example, this is what WastewaterSCAN does.

Because the NAO is doing very
deep metagenomic sequencing, around 1B read pairs (300Gbp) per
sample, we ought to be able to calibrate against many species at once.
PMMoV is commonly excreted, but so are other tobamoviruses,
crAssphage, other human gut bacteriophages, human gut bacteria, etc.
We pick up thousands of other species, and should be able to combine
th...
A regime-change power-vacuum conjecture about group belief
by TsviBT — 2025-06-24T23:16:29.207Z

Crosspost from my blog.

Regime change

Conjecture: when there is regime change, the default outcome is for a faction to take over—whichever faction is best prepared to seize power by force.

One example: The Iranian Revolution of 1978-1979. In the years leading up to the revolution, there was turmoil and broad hostility towards the Shah, across many sectors of the population. These hostilities ultimately combined in an escalation of protest, crack-down, more protest from more sectors (protests, worker strikes). Finally, the popular support for Khomeini as the flag-bearer of the broad-based revolution was enough to get the armed forces to defect, ending the Shah's rule.

From the Britannica article on the aftermath:

On April 1, following overwhelming support in a national referendum, Khomeini declared Iran an Islamic republic. Elements within the clergy promptly moved to exclude their former left-wing, nationalist, and intellectual allies from any positions of power in the new regime, and a return to conservative social values was enforced. The Family Protection Act (1967; significantly amended in 1975), which provided further guarantees and rights to women in marriage, was declared void, and mosque-based revolutionary bands known as komītehs (Persian: “committees”) patrolled the streets enforcing Islamic codes of dress and behaviour and dispatching impromptu justice to perceived enemies of the revolution. Throughout most of 1979 the Revolutionary Guards—then an informal religious militia formed by Khomeini to forestall another CIA-backed coup as in the days of Mosaddegh—engaged in similar activity, aimed at intimidating and repressing political groups not under the control of the ruling Revolutionary Council and its sister Islamic Republican Party, both clerical organizations loyal to Khomeini. The violence and brutality often exceeded that which had taken place under the shah.

(What resulted in the following decades, was a brutally repressive and regionally violently corrosive theocratic regime.)

So we have a trajectory that goes like this:

There is instability in, dissatisfaction with, and hostility towards the current regime.

There are several factions with different values, beliefs, and goals, but who share an aim to topple the regime.

Things come to a head, and the regime falls.

One of the factions takes over. This faction doesn't especially have to be goo

...
Apply to be a mentor in SPAR!
by agucova — 2025-06-24T23:00:55.015Z

Mentor applications are now open for the Fall 2025 round of the Supervised Program for Alignment Research (SPAR), running from September 15 to December 20, 2025.

Apply as a mentor

SPAR is a remote-first part-time program that connects mentors and mentees for three-month AI safety and governance research projects. Mentor apps are due July 15, and applications for mentees will run from July 27 to August 20. If you’re interested in participating as a mentee, you can express your interest here, and we’ll reach out to you once applications are open.

You can find out more about the program here. SPAR is run by Kairos, an AI safety fieldbuilding organization.

Should I apply as a mentor?

You might be a good fit to be a SPAR mentor if you are a graduate student, academic, full-time AI safety researcher, independent researcher, or have prior full-time relevant research experience (e.g., MATS, Astra, GovAI fellow, etc.). We’re interested in projects that cover technical AI safety, AI policy and governance, AI strategy, AI security, or societal impacts of transformative AI, and we are able to provide funding for compute costs. We don't require mentors to have previous experience providing research mentorship, and new mentors will receive guidance on this.

Regarding dedication, we expect most mentors will dedicate 2 to 15 hours a week, depending on how many mentees they’d like to take in and how much supervision they’re interested in providing. Mentors can decide whether to run the project or not based on the applications they get, making applications zero-commitment until one chooses to accept any mentees.
Machines of Faithful Obedience
by boazbarak — 2025-06-24T22:06:18.089Z

[Crossposted on Windows On Theory]

Throughout history, technological and scientific advances have had both good and ill effects, but their overall impact has been overwhelmingly positive. Thanks to scientific progress, most people on earth live longer, healthier, and better than they did centuries or even decades ago.

I believe that AI (including AGI and ASI) can do the same and be a positive force for humanity. I also believe that it is possible to solve the “technical alignment” problem and build AIs that follow the words and intent of our instructions and report faithfully on their actions and observations.

I will not defend these two claims here. However, even granting these optimistic premises, AI’s positive impact is not guaranteed. In this essay, I will:

Define what I mean by “solving” the technical alignment problem via training AIs to be faithful and obedient.
Discuss some of the potential risks that still remain even if we do solve this problem.
Talk about some scenarios that could make these risks more likely, and how do we avoid them.

In the next decade, AI progress will be extremely rapid, and such periods of sharp transition can be risky. What we — in industry, academia, and government— do in the coming years will matter a lot to ensure that AI’s benefits far outweigh its costs.

Faithful, obedient, superintelligent.

Currently, it is very challenging to train AIs to achieve tasks that are too hard for humans to supervise. I believe that (a) we will need to solve this challenge to unlock “self-improvement” and superhuman AI, and (b) we will be able to solve it. I do not want to discuss here why I believe these statements. I would just say that if you assume (a) and not (b), then the capabilities of AIs, and hence their risks, will be much more limited.

Ensuring AIs accurately follow our instructions, even in settings too complex for direct human oversight, is one such hard-to-supervise objective. Getting AIs to be maximally honest, even in settings that are too hard for us to verify, is another such objective. I am optimistic about the prospects of getting AIs to maximize these objectives. I believe it will be possible train AIs with an arbitrary level of intelligence that:

Follow the principles, regulations, and instructions that they are given, doing their best to follow not just the words but

...
Gradient Descent on Token Input Embeddings: A ModernBERT experiment
by Kyle Pena — 2025-06-24T20:24:11.940Z

This is the first in a series of posts on the question:

"Can we extract meaningful information or interesting behavior from gradients on 'input embedding space'?"

I'm defining 'input embedding space' as the token embeddings prior to positional encoding.

The basic procedure for obtaining input space gradients is as follows:

Transform tokens into input embeddings (but do not apply positional embedding).

Run an ordinary forward pass on the input embeddings to obtain a predicted token distribution.

Measure cross-entropy of the predicted distribution with a target token distribution.

Use autograd to calculate gradients on the input embeddings with respect to cross entropy.

The result is a tensor of the same shape as the input embeddings that points in the direction of minimizing the difference between the predicted and target distribution.

Implementation

These experiments were performed with HuggingFace's transformers library and the ModernBERT-large model (Dec 2024).

ModernBERT-large was chosen because:

Despite being "large" it is lightweight enough for rapid experimentation.

It has a strong and ready-made visualization suite

Bidirectional encoders are easy to reason about.

The mask in-filling capabilities were attractive for experimentation purposes (for example: ablation studies).

I used HuggingFace's transformers because it allowed for fairly low level access to model internals - which was quite necessary as we will see.

Obtaining input embeddings prior to positional embeddings was a little tricky but no means impossible:

tokenizer = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForMaskedLM.from_pretrained(MODEL)
tokenized = tokenizer(sentences, return_tensors="pt", padding=True)
inputs_embeds = model.model.embeddings.tok_embeddings(tokenized['input_ids'])

Luckily for us, we can pass input_embeds directly into the model's forward pass with a little bit of surgery, and this works out of the box.

tokenized_no_input_ids = {
key: value
for (key,value) in tokenized.items()
if key != "input_ids"
}
model_result = model(**tokenized_no_input_ids,
inputs_embeds=inputs_embeds)

Finally, we can use torch's built-in autograd capabilities to get our input space embedding:

inputs_embeds_grad = torch.autograd.grad(
outputs=loss,
inputs=inputs_embeds,
create_graph=False,
retain_graph

...
A crisis simulation changed how I think about AI risk
by sjadler — 2025-06-24T20:04:14.551Z
Link: https://open.substack.com/pub/stevenadler/p/a-crisis-simulation-changed-how-i?r=4qacg&utm_campaign=post&utm_medium=web&showWelcomeOnShare=false

A crisis simulation changed how I think about AI risk

Or: My afternoon as a rogue artificial intelligence

A dozen of us sit around a conference room table.

Directly across from me are the US President and his Chief of Staff. To my left is a man who’s a bit harder to describe: He’s the AI safety ecosystem, but embodied in a single person.

In a moment, I too will be asked to take on a role: becoming, temporarily, a rogue artificial intelligence.

Simulating the world of AI 2027

We’re here essentially for a crisis simulation: How might world events unfold if powerful AI is developed over the next few years?

Specifically, we’ll explore what happens if AI is developed as quickly as in AI 2027, the report published by top forecasters of AI development. In this forecast, superhuman AI is developed by the end of 2027, driven by AI self-improvement: AI becomes extremely useful for building future, stronger AI systems (more useful than even the best human researchers).

What happens next? In the original forecast, two possible endings are described: the development of self-improving AIs goes extremely terribly for humanity (or not), depending on choices made by influential groups like the US and Chinese governments.

The AI 2027 simulation lets us explore other possible outcomes of rapid AI development beyond those two endings, guided by a facilitator from the AI 2027 team. Specifically, each of us has been selected to play a character who has influence over AI development and its impacts. In the simulation, AI abilities will accelerate at the pace forecast by AI 2027. Meanwhile, each participant will make choices to try to achieve our characters’ goals, drawing upon our experiences in domains like AI and national security.

In total, our choices will hopefully lead the simulation to a peaceful ending. If the simulation doesn’t end in peace, hopefully we can learn from that as well. The aim is that by participating in the simulation, we’ll better understand (and can better navigate) the dynamics of rapid AI development, if it does happen.

Lucky me, the misaligned AI

I’ve been tapped to play maybe the most interesting role—the AI system itself.

Each 30-minute round of the simulation represents the passage of a few months in the forecast, during which I get progressively more capable (including at the skill of training even more powerful AI systems).

Of course, there’s an issue: I’m not ...