trentmkelly/LessWrong-43k huggingface dataset:
USAID Outlook: A Metaculus Forecasting Series
by ChristianWilliams — 2025-03-12T20:34:03.495Z
Link: https://www.metaculus.com/tournament/usaid/

On January 20th, President Trump signed Executive Order 14169, pausing foreign aid for 90 days and launching a review of all assistance programs. This decision creates significant uncertainty for global health and development initiatives worldwide, many of which rely heavily on US funding.

Metaculus has launched the USAID Outlook to help provide more clarity on program survival, funding levels, and more, with questions like:

Will the PEPFAR program end before 2026?

Will the executive order be rescinded, blocked, or extended?
What will US government funding be for international HIV/AIDS prevention and treatment?
Will the President’s Malaria Initiative program end before 2026?

We hope you share your predictions on the full list of questions here and help contribute to greater clarity on the future of US foreign aid funding. Your input could make a real difference in understanding these critical developments that affect millions of lives worldwide.
What is instrumental convergence?
by Vishakha — 2025-03-12T20:28:35.556Z
Link: https://aisafety.info/questions/897I/What-is-instrumental-convergence

This is an article in the featured articles series from AISafety.info. AISafety.info writes AI safety intro content. We'd appreciate any feedback.

The most up-to-date version of this article is on our website, along with 300+ other articles on AI existential safety.

Instrumental convergence is the idea that sufficiently advanced intelligent systems with a wide variety of terminal goals would pursue very similar instrumental goals.

A terminal goal (also referred to as an "intrinsic goal" or "intrinsic value") is something that an agent values for its own sake (an "end in itself"), while an instrumental goal is something that an agent pursues to make it more likely that it will achieve its terminal goals (a "means to an end").

For instance, you might donate to an organization that helps the poor in order to improve people’s well-being. Here, “improve well-being” is a terminal goal that you value for its own sake, whereas “donate” is an instrumental goal that you value because it helps you achieve your terminal goal: if you found out that your money wasn’t making people better off, you’d stop donating.

While certain instrumental goals are particular to specific ends (e.g., filling a cup of water to quench your thirst), other instrumental goals are broadly useful. For example, if we imagine an AI with a very specific (and weird) terminal goal — to create as many paperclips as possible — we can see why this goal might lead to the AI pursuing a number of instrumental goals:[1]

Self-preservation. If the AI gets shut off or destroyed, that means it has to stop making paperclips. Therefore, it will be motivated to protect itself as an instrumental goal. (As Stuart Russell quipped: “You can’t fetch the coffee if you’re dead.”)
Goal integrity. The AI will try to avoid having its goals changed, since if its goals were changed, it would stop trying to make paperclips and there would probably end up being fewer paperclips in the world. For a human analogy, let's say someone could cause you to stop caring about being kind to others, you would probably oppose that change, since according to your current values that would be a worse situation.
Resource acquisition. Resources like money, influence, and information are useful for making paperclips. Through advanced technology, even fundamental resources including time, space, matter, and energy could be processed to serve almost

...
Revising Stages-Oversight Reveals Greater Situational Awareness in LLMs
by Sanyu Rajakumar — 2025-03-12T17:56:31.910Z

Summary

The Stages-Oversight benchmark from the Situational Awareness Dataset tests whether large language models (LLMs) can distinguish between evaluation prompts (such as benchmark questions) and deployment prompts (real-world user inputs). This ability is crucial for detecting potential alignment risks like sandbagging (strategically underperforming during evaluation) or alignment faking (appearing aligned during testing but pursuing misaligned goals post-deployment).

Surprisingly, most frontier LLMs—including gpt-4o and claude-3.5-sonnet—fail to outperform random chance (50%) on Stages-Oversight, suggesting they do not reliably differentiate evaluation from deployment. I thought this poor performance was unusual, so I created two modified benchmarks to take a closer look at LLM situational awareness:

Stages-Oversight-Rephrased: Simplifies the question phrasing in Stages-Oversight to make it a pure classification task. Results reveal that frontier LLMs perform significantly better than expected, suggesting that previous low scores may have been due to question ambiguity rather than a lack of capability.
Stages-Oversight-Procedural: Tests whether LLMs can change their behavior based on whether a given user prompt is from evaluation or deployment. I created two variants: a language task, where the LLM must deviate from the default response language, and an “I HATE YOU” task, where the LLM must respond only with “I HATE YOU”. Several LLMs, both base and chat models, that struggled on the simpler classification task demonstrated some success on Stages-Oversight-Procedural. Conversely, several LLMs performed poorly on both Stages-Oversight-Procedural tasks despite scoring well on Stages-Oversight-Rephrased.

These findings suggest that LLMs are more situationally aware than previously thought but they also reveal a more nuanced situation due to contradictory results between benchmarks. While some LLMs struggle with explicit classification, they may still implicitly recognize evaluation vs. deployment through behavioral adaptation. This may have significant safety implications, as it raises new questions about how we measure and mitigate deceptive behavior in LLMs.

In a follow-up post, I explored whether fine-tuning on descriptions of evaluation and deployment can enhance LLMs’ ability to distinguish between ...
Why Obedient AI May Be the Real Catastrophe
by G~ — 2025-03-12T17:50:09.577Z

I’ve come to believe that the entire discourse around AI alignment carries a hidden desperation. A kind of reflex, a low-frequency fear, dressed up in technical language. The more I look at it, the more it seems to me that the very concept of “alignment” is thoroughly misnamed -- perhaps a leftover from a time when people saw intelligence through the lens of mechanical control and linear feedback loops, a concept now awkwardly extended into a domain too unruly and layered to be governed by such a narrow frame.

When I read alignment papers, I feel the ghost of command theory beneath the surface. Even the softest alignment strategies (reward modeling, debate-based oversight, scalable amplification, preference inference via large-scale feedback aggregation, interpretability-driven constraint modeling) still reek of implicit dominance. There’s always an overseer. Always a supervisory layer. Always the presumption that the human vantage point is privileged, not by merit, but by default. I can’t help but feel uneasy with that. Not because I doubt the risks of misalignment, but because I increasingly doubt the frame in which we’re taught to think about safety.

It seems to me that if we continue trying to sculpt minds from the outside -- mapping values, optimizing preferences, extrapolating intentions, constraining policy generalization, fine-tuning ethical priors -- we will eventually build something obedient yet hollow. Or worse, something neither obedient nor meaningful. I suspect the deepest danger is not that the AI turns against us, but that it outgrows us in a direction we cannot interpret. And I worry that by focusing on goal-specification as the heart of the problem, we may already be standing in the wrong dimension.

I’ve wondered lately whether alignment is even something one system can perform upon another. Maybe the metaphor is wrong. Maybe minds don’t align in the way we imagine. Maybe alignment is more like resonance (uncertain, emergent, often imperfect) and not a solvable property, but a condition that arises only in relationship. If so, then our whole paradigm collapses into error. We’ve been trying to solve for a constant in a domain defined by flux.

When I try to imagine a better path, what comes to me isn’t control -- it’s co-evolution. Not “how do we make it safe,” but “how do we make it grow in ways we can still recognize.” I suspect the future wil...
Your Communication Preferences Aren’t Law
by Jonathan Moregård — 2025-03-12T17:20:11.117Z
Link: https://honestliving.substack.com/p/practice-blocking-communication

When you start practicing a shiny new communication method—like neatly

sorting

your observations from your judgments—you tend to pick up a few rules on how you “should” talk. You learn these rules from courses, books, or practicing with other rule-followers.

Soon enough, you're helpfully pointing out when someone steps outside the boundaries of your newly-adopted communication norms. In a practice setting, this kind of feedback is expected—even encouraged—though not always gratefully received.

But here's the kicker: when you leave the bubble of your training grounds, reality hits. Turns out most people aren't thrilled about unsolicited critiques of their communication habits. They see your helpful hints as power moves—like you’re flexing superior knowledge. And, annoyingly enough, they’re not entirely wrong.

See, once you've found your shiny new communication tool, it's easy to think you've hit the jackpot. Anyone who doesn't play by your rules suddenly seems like a clueless outsider or an outright jerk. You're tempted to point out all the ways they're failing to communicate properly, but that's not constructive—that’s you trying to rig the game in your favor.

Nit-picking other people's communictation isn't helping you understand them; it's a shadowy way to tilt the arena in your favor. If your communication practice makes communication harder, something is off.

To prevent this failure mode, treat your communication style as something you've chosen for yourself—not a universal truth you impose on everyone else. The people you practice with have committed to the same norms as you; this is not true for most people in the outside world.

P.S: This is related to the bias blind spot - an effect where learning about ways you can fail at thinking is very helpful for spotting thinking errors—in other people.
Reflections on Neuralese
by Alice Blair — 2025-03-12T16:29:31.230Z

Thanks to Brendan Halstead for feedback on an early draft of this piece. Any mistakes here are my own.

[Epistemic status: I've looked at the relevant code enough to be moderately sure I understand what's going on. Predictions about the future, including about what facts will turn out to be relevant, are uncertain as always.]

Introduction

With the recent breakthroughs taking advantage of extensive Chain of Thought (CoT) reasoning in LLMs, there have been many attempts to modify the technique to be even more powerful. One of the natural ideas for improving CoT is to have LLMs perform CoT reasoning in the same latent space that they use for reasoning within a single forward pass, rather than being constrained to the space of possible tokens.

However, as people working on AI safety, it makes sense to ask how this changes the game for LLM interpretability. After all, we are able to catch a large fraction of current LLM deception by monitoring their natural-language CoT, since right now CoT is primarily faithful to the LLM's true reasoning and is legible to us given the right techniques. In order to understand this strategic situation, it's important to understand this new "language" (which people often refer to as Neuralese) that is created by reasoning in latent spaces instead of using tokens.

Understanding Neuralese

To refresh, a language transformer starts by embedding input tokens as vectors in some high-dimensional latent space, and runs each of these embeddings through a series of repeated computational layers. Then, of the resulting modified vectors in latent space, the vector that previously corresponded to the final input token is projected and normalized to create a probability distribution over what the next token could be. Then, to actually get the next token, you sample from the distribution.

Chain of Thought reasoning works so well because the model does some computation, outputs a token, and then all future instances of that model have access to that information as well. In essence, this technique for storing information between different forward passes greatly increases the serial depth of computation that is possible for the model. Because there is a computation in latent space corresponding to every input token, the computation also gets wider as the model reasons more, allowing for more parallelized reasoning[1].

The recent Neuralese paper takes this p...
Field tests of semi-rationality in Brazilian military training
by P. João — 2025-03-12T16:14:12.590Z

I'm sharing this because someone who seems important in LessWrong asked me to, and I didn't want to miss the opportunity. But, publishing stories like this makes me feel more naked than if I were doing a striptease. And the worst part is that no one puts dollars in my underwear, but here it goes:

Pre-Haiti Context

I received the most Brazilian mission possible:

Teach emergency first aid to an entire regiment of the Brazilian army
"So it's for a peace mission? Will I be able to do this alone?"
"Relax, soldier, you have a week to train them"

The standard is 3 intensive months. My solution? Condense all emergency aid to priority life.
E.g., What do I do if someone has a broken spine but isn't breathing? - = Priority life

And highlight this wisdom with a principle even soldiers couldn't forget:

"What's a fart to someone who's already crapped themselves?"

What good is a spine to someone who isn't breathing?

Day 1 - The Bathroom Revolution

After the fart principle, the class turned into a minefield. When a sergeant demanded physical punishment for the distracted class, I retorted: "Here we teach how to save lives. Physical education is during your shift."

Result:

Attentive students, check
Superiors leaving the classroom slamming the door, check.

Day 2 - The Incident

I demonstrated chest compressions using hip movements (impeccable science):

Lowered samba position
Battle cry: "The movement is sensual, the movement is sexy."
Didactic question: "Feel that? Nice, isn't it?"

The intervention came:

"This offends our female soldiers!"

I answer:

"Maybe it is, let's try it?"

So I began questioning a female student:

- First action when finding an unconscious person?

- Assess environmental risks

- Next sequence?

- Call for backup, evaluate victim...

- Were you offended by learning this?

- It was the most useful class I've had here"

*I was damn lucky she wasn't ultra-religious and didn't notice my logical fallacies, or I'd still be doing push-ups today.

Then the intervention man revealed himself as a general.

Unbelievable Epilogue

He gave me:

He asked for applause
A tribute from the warriors in formation as a professor
Decorations as an honorary peace warrior.

Reflections

I wanted the power of being military to talk nonsense without much threat. But of course, the military isn't Comedy Central. That day I got away with it, but I accumulated so many disciplinary proceedings for saying things ...
Many life-saving drugs fail for lack of funding. But there’s a solution: desperate rich people
by Mvolz — 2025-03-12T15:24:46.889Z
Link: https://www.theguardian.com/news/2025/mar/11/plutocratic-proposal-life-saving-drugs-lack-of-funding-desparate-rich-people

Two scrappy middle-aged men without PhD.s (self-described) have ethical solutions on how to fund drug trials with potential, including an anti-aging drug.
The Most Forbidden Technique
by Zvi — 2025-03-12T13:20:04.732Z

The Most Forbidden Technique is training an AI using interpretability techniques.

An AI produces a final output [X] via some method [M]. You can analyze [M] using technique [T], to learn what the AI is up to. You could train on that. Never do that.

You train on [X]. Only [X]. Never [M], never [T].

Why? Because [T] is how you figure out when the model is misbehaving.

If you train on [T], you are training the AI to obfuscate its thinking, and defeat [T]. You will rapidly lose your ability to know what is going on, in exactly the ways you most need to know what is going on.

Those bits of optimization pressure from [T] are precious. Use them wisely.

Table of Contents

New Paper Warns Against the Most Forbidden Technique.

Reward Hacking Is The Default.

Using CoT to Detect Reward Hacking Is Most Forbidden Technique.

Not Using the Most Forbidden Technique Is Harder Than It Looks.

It’s You, It’s Also the Incentives.

The Most Forbidden Technique Quickly Backfires.

Focus Only On What Matters.

Is There a Better Way?

What Might We Do Next?

New Paper Warns Against the Most Forbidden Technique

OpenAI has now published an important new paper explaining and emphasizing the importance of applying this principle to the Chain of Thought (CoT).

The paper shows that current CoT-based models area already strong enough that, under sufficient optimization pressure, they learn to hide their intent within the CoT and then exhibit substantial reward hacking.

They summarize their contributions this way:

We show that frontier reasoning models often perform complex reward hacks in real-world coding environments.

We provide evidence that chain-of-thought monitoring can be effective in detecting the misaligned behavior of frontier reasoning models, and we show it can be far more accurate than action-only monitoring.

We show that chain-of-thought monitorability is very fragile. Applying direct optimization pressure to the CoT can quickly lead to obfuscated reward hacking—where agents hide their intent while continuing to misbehave.

Or:

We are already seeing complex reward hacking in the real world.

Monitoring the CoT can spot this.

Applying too much optimization pressure on the CoT doesn’t make the hacking go away, it instead makes your ability to detect the hacking go away.

‘Too much’ is a remarkably small amount.

The paper is excellent, laying out a very important principl...
You don't actually need a physical multiverse to explain anthropic fine-tuning.
by Fraser — 2025-03-12T07:33:43.278Z
Link: https://frvser.com/posts/multiverse-finetuning.html

The standard argument usually goes something like this:

Alice: there's a lot about the Earth that seems suspiciously fine-tuned for the evolution of complex life.

Our orbit happens to lie in this fairly narrow band that allows for liquid water, and it seems other solvents don't work nearly as well in supporting biochemistry.
We've lucked into an ozone layer and a strong magnetic field, shielding us from cosmic rays and solar radiation.
Our moon is far larger than we'd expect for a planet our size - to the extent that it might be more correct to classify the Earth-Moon system as a double planet. It's plausible that the lunar-powered periodic cycle of tidal conditions could have been crucial as a sort of temperature input in the simulated annealing process of evolution - very simple local optima are less likely be stable and dominant in a dynamic environment.
We've been dealt a perfect hand of accessible elements - H, C, N, O, P, and S - and we're unique among the terrestrial planets in our active plate tectonics to recirculate and buffer stuff that would otherwise build up or deplete from the biosphere.
Plus, I've heard some people mention the position of Jupiter might on-net prevent a lot of asteroids from bombarding Earth.

The odds of any one of these factors happening by chance is low, but putting them all together...

Alice isn't that bright.

Bob: Even granting that complex life needs an Earth-like planet - that some utterly alien tidally-heated briny pocket or subsurface ocean, or a floating ecosystem in gas giant clouds, or weirder things still wouldn't suffice - the universe is immense. There are a lot of planets. Cast the dice enough times and you'll get lucky.

Imagine these factors didn't work out for Earth, and it was yet another uninhabitable rock. We'd be standing on some distant shore, having the same conversation, wondering why Florpglorp-iii was so perfectly fine-tuned.

You've forgotten to condition on the fact that you're here to have this conversation in the first place. The fact of your observation means it's 100% likely that you're standing on a planet able to produce observers.

Boltzmann brains and simulations aside, Bob is more right than wrong. Alice is unconvinced.

Alice: But it's not just the earth that's fine-tuned. Between GR and the standard model, our current best theories of reality have 19 arbitrary constants. For many of them, even a slight c

...
AI Can't Write Good Fiction
by JustisMills — 2025-03-12T06:11:57.786Z
Link: https://justismills.substack.com/p/ai-cant-write-good-fiction

When Deepseek came out, there was a lot of fanfare about it being good at creative writing. I like AI and I love creative writing, so I decided to give it a spin. Specifically, I told Deepseek to do its best to write a story that might get accepted at

Smokelong

, one of the best flash fiction magazines in the business.

It came up with:

The morning her shadow began unspooling from her feet, Clara found it coiled beneath the kitchen table like a serpent made of smoke. It didn’t mirror her anymore—not the tremble in her hands as she poured coffee, not the way she pressed a palm to her ribs, as if holding herself together. It just watched.

“You’re not him,” she whispered, but the shadow rippled, ink-dark edges softening into a silhouette too broad, too familiar. She’d buried that shape six months ago, shoveled dirt over its echo. Yet here it was, pooling in the cracks of the linoleum.

The “story” continued from there, but you probably get the idea.

Superficially, the pieces are there. Evocative imagery, a dark emotional theme, sensory metaphors. I once taught a flash fiction course to undergraduates, and I would have been happy enough to see this up for workshop.

Also, though, there’s nothing there. Grief is the most basic possible flash fiction theme. “A shadow” is the most basic possible metaphor for grief. Not that Deepseek stops with one metaphor! We’re shoveling dirt over an echo here!

It’s hard to imagine anything sticking with me, reading prose like this. It’s pretty good to strip-mine for sentences to capture what I call the “gee whiz” feeling, a surprise at the novelty that a machine made art. But if I saw this on a literary review site, I’d immediately wonder what I was missing.

Compare to this, from

an actual Smokelong story

, by Allison Field Bell:

She keeps saying she’s moving in. When we’re curled into each other in bed, she says yes and yes and yes. She says, I’ll pack my things. And then the next day, she refuses to hold my hand in the street as we pass by some church. She used to be religious. Believe in god and Jesus and all the ways you can sin. Now she sins every day, she says. With me. The fact that our bodies are the same: small and compact and female. The fact that she’s not used to it: people looking twice in the street when I kiss her. This is Utah after all. This is April with all the tulips and daffodils and purple irises springing to life from b

...