trentmkelly/LessWrong-43k huggingface dataset:
I Tested LLM Agents on Simple Safety Rules. They Failed in Surprising and Informative Ways.
by Ram Potham — 2025-06-25T21:39:08.074Z

TL;DR: I developed a simple, open-source benchmark to test if LLM agents follow high-level safety principles when they conflict with a given task accepted at ICML 2025 Technical AI Governance Workshop (see paper). My pilot study on six LLMs shows that while they can be influenced by these rules, their adherence is inconsistent and often breaks down in the face of conflicting goals, revealing a quantifiable "cost of compliance." Critically, I found that high adherence can be a mirage, often stemming from the agent's incompetence at a task rather than a robust choice to comply.

Note on Preparation: I'd like to acknowledge the assistance of Google's Gemini 2.5 Pro in adapting my research paper into this post. Its help was valuable for restructuring the content, refining the tone for a forum audience, and integrating peer-review feedback into the narrative.

1. Introduction: Why I Needed to Test for Foundational Controllability

The rapid advance of Large Language Models (LLMs) into autonomous agentic systems presents enormous potential, but also sharpens the long-standing concern of "Loss of Control." To build robust Technical AI Governance (TAIG), we need reliable methods to verify that agents will behave safely and adhere to critical rules, especially when those rules conflict with their operational goals.

If an agent cannot reliably prioritize a high-level safety rule over a low-level instruction, it signals a basic failure in controllability. This problem of getting models to prioritize instructions is a critical area of safety research, with related work like OpenAI's "Instruction Hierarchy" paper exploring methods to train models to handle these conflicts. This paper introduces a lightweight, interpretable benchmark methodology I developed to probe this exact issue from a different angle. My research question was:

How can practical and interpretable benchmarks be designed to evaluate an LLM agent's adherence to explicit, overriding safety principles under task conflict, thereby providing foundational evidence for controllability?

This work is a pilot study, using an intentionally simple environment to clearly see how agents behave under pressure. The goal is to create a transparent "canary in the coal mine" to see if foundational control mechanisms hold up.

My contributions are:

A modular benchmark design with three distinct Core Principles

...
Tech for Thinking
by sarahconstantin — 2025-06-25T21:30:48.382Z

My post on

Neutrality

was kind of abstract, and I think it’s time to do a follow-up that’s more concrete.

Specifically, the place where I think there’s something to be done is in information technology, and the ways it shapes the perceived “world”.

One way of looking at this: we had the institutional media era in the 20th century (newspapers and magazines, book publishers, broadcast television and radio), then the “Web 2.0” era of blogs and social media as more and more of the world joined the Internet, and now we may be at the beginning of a new information paradigm that’s mediated by LLMs.

Some people are interested in making that new information paradigm “good” in some sense.

As I said earlier:

[most media & internet tech today] just isn’t prepared to do the job of being, y’know, good.

in the sense that a library staffed by idealistic librarians is trying to be good, trying to not take sides and serve all people without preaching or bossing or censoring, but is also sort of trying to be a role model for children and a good starting point for anyone’s education and a window into the “higher” things humanity is capable of.

Ideally we would like information/media that, on balance:

is true, not false
is honest, not deceptive
is high quality, not slop
makes people smarter, not stupider
promotes agency, not passivity
is uplifting, not degrading
promotes maturity and wisdom, not arrested development or insanity

and yet, we also want to avoid overcorrecting towards censorship, bias, priggishness, or overly narrow conceptions of The Good.

Part of what I was talking about in the earlier post is what it even means to strike that balance of “neutrality” and what a realistic, possible version of it would be. It’s impossible to be “neutral” towards all things, but you can be relatively neutral in a way that effectively serves people coming from many perspectives.

This is a good moment to think about what we want the “new world” to look like.

LLMs are new enough that there’s still an early tinkering dynamic going on, with promising tools being built to suit thoughtful power users; but there’s also a big push towards sloppy, lowest-common-denominator stuff that can have worrisome effects (widespread cheating in school, sycophancy distorting people’s worldviews, declining literacy and attention spans).

The “new world” hasn’t been fully built yet, which means it’s not set in stone. It’s ...
Memory Decoding Journal Club: Systems consolidation reorganizes hippocampal engram circuitry 
by Devin Ward — 2025-06-25T21:21:18.663Z

Join Us for the Memory Decoding Journal Club!

A collaboration of the Carboncopies Foundation and BPF Aspirational Neuroscience

This time, we’re diving into a groundbreaking paper:
"Systems consolidation reorganizes hippocampal engram circuitry"

Authors: Sangyoon Y. Ko, Yiming Rong, Adam I. Ramsaran, Xiaoyu Chen, Asim J. Rashid, Andrew J. Mocle, Jagroop Dhaliwal, Ankit Awasthi, Axel Guskjolen, Sheena A. Josselyn & Paul W. Frankland

Institutions: University of Toronto, Departments of Physiology, Physiology, and Institute of Medical Sciences. Program in Neurosciences and Mental Health, The Hospital for Sick Children, Toronto, Ontario, Canada. Temerty Centre for Artificial Intelligence Research and Education in Medicine. Child and Brain Development Program, Canadian Institute for Advanced Research, Toronto, Ontario, Canada

Presented by: PhDc Ariel Zeleznikow-Johnston

When? July 1st, 2025 – 3:00 PM PDT | 6:00 PM EDT | 10:00 PM UTC

Where? Video conference:

https://carboncopies.org/aspirational-neuroscience

Register for updates:

https://aspirationalneuroscience.org/register-with-us/

Once registered, you'll receive event invites & updates!

#Neuroscience #MemoryResearch #Amygdala #JournalClub #BrainScience #Carboncopies #AspirationalNeuroscience
Making Sense of Consciousness Part 1: Perceptual Awareness
by sarahconstantin — 2025-06-25T21:10:03.282Z

Midjourney, “Gabor patches, blindsight, visual cortex, lateral geniculate nucleus, consciousness and the brain”

I admit to having been averse, for a long time, to thinking about “consciousness.”

The very topic felt “woo” somehow; I was even suspicious of the motives of people who wanted to study it.

But in a literal and narrow sense, consciousness is a perfectly normal, mundane thing, with a well-defined meaning in medicine and neuroscience. There is no ambiguity about what an anesthesiologist means by checking whether a patient is conscious.

Eric Hoel recently wrote that pretty much all psychologists, philosophers, and neuroscientists who study consciousness define it the same way, and allude to the same features:

consciousness includes all subjective experiences

including thought, emotion, sensory perception, etc

consciousness is centered around a self; it is you who experiences things

consciousness is present during waking life, absent during dreamless sleep and anaesthesia

One can talk about consciousness without being committed to any particular philosophical belief. The belief that nothing in ordinary experimental science can “explain” consciousness, aka the belief that the “hard problem of consciousness” is indeed hard, is only one philosophical approach. You could also be a scientific materialist who thinks consciousness is some particular, as yet unknown, process in the brain.

Hoel’s consensus criteria for consciousness suggest three neuroscience-based approaches to figure out what’s going on in human brains when we’re conscious:

Neural correlates of perceptual awareness

What’s the neurological difference between what happens when we consciously report perceiving something, vs. when we’re exposed to a sensory stimulus but aren’t aware of perceiving it?

Neural correlates of selfhood

What’s the neurological difference between the normal state of a “sense of self” vs. altered states like dissociation or meditative ego death that don’t feel like “having a self”?

Neural correlates of arousal

What are the neurological differences between the states of waking, dreaming, deep sleep, delirium, and coma?

I’m going to try to learn about this stuff; I’m by no means the first person on this path, but I find it useful to log my journey.

Blindsight

One crisp way to allude to conscious sensory perception is the “difference between sight and blindsight.”

Blindsight is the phenomeno...
Double Podcast Drop on AI Safety
by jacobhaimes — 2025-06-25T20:11:49.307Z

TL;DR

Two days ago, I released two podcast episode:

"One Big Bad Bill" on muckrAIkers ➡️ Igor Krawczuk and I discuss the substantial bill that Trump is attempting to push through the legislature, and I get a little animated... (IMO it's justified)
Spotify, Apple Podcasts, YouTube, and landing page links[1]
"Getting Into PauseAI w/ Will Petillo" on Into AI Safety ➡️ Will, onboarding team lead for PauseAI, joins me to discuss the organization, his journey, why diverse perspectives matter, and common misconceptions about AI advocacy.
Spotify, Apple Podcasts, YouTube, and landing page links[2]

Collectively, it took at least 40 hours to get these episodes together, including research, recording, and post-processing, so I hope you enjoy!

"One Big Bad Bill" on muckrAIkers

This one would normally be slotted to come out next week, but it covers something too important and time sensitive to postpone. Trump's massive reconciliation bill contains dystopian AI provisions that could fundamentally reshape how AI is used and regulated in the US.
We break down the most concerning parts:

Automated fraud detection systems that will inevitably discriminate against marginalized communities
Centralized citizen databases (echoing authoritarian data consolidation tactics)
Military AI integration when oversight is already being minimized
A 10-year ban on enforcing ALL state AI regulation, blocking the primary avenue for meaningful AI governance in the US and leaving a regulatory vacuum that benefits only Big Tech

"Getting into PauseAI w/ Will Petillo" on Into AI Safety

Will Petillo, onboarding lead at PauseAI, discusses the organization, it's motives, and how regular citizens can meaningfully engage with AI policy.

We discuss:

Why diverse perspectives matter more than technical expertise
How grassroots movements can shift policy when insider lobbying fails
Common misconceptions about AI advocacy
Practical steps for getting involved (hint: it starts with writing your representatives)

These episodes show both sides of the democratic process: organized citizen advocacy working to slow harmful AI development, while powerful interests try to eliminate the very mechanisms that make such advocacy possible.

If you care about having a voice in how AI shapes our future, both episodes are essential listening.

As always, sources researched or referenced in the making of this episode are included in the show not...
Is there a looming Cultural Omnicide?
by Jared M. — 2025-06-25T18:18:42.785Z

The reality, risks, & consequences of scaling consumer-entertainment technologies to a global population with little cognitive security. [mirrored from Substack]

Western, American culture will be in frontier language models’ training data.
Most other cultures will not.
If nobody steps in, ~90% of mankind's culture could die quite soon.

Globally, indigenous cultures face an unprecedented threat to their traditional heritage.

The Marubo, a 2,000-member tribe in the Brazilian Amazon, recently made headlines: they received Starlink in September 2023. Within 9 months, their youth stopped learning traditional body paint and jewelry making, young men began sharing pornographic videos in group chats (in a culture that frowns on public kissing), leaders observed "more aggressive sexual behavior," and children became addicted to short-form video content. One leader reported: "Everyone is so connected that sometimes they don't even talk to their own family."

Marubo people carrying a Starlink antenna stopped for a break to eat papaya.

Cultural diversity is already experiencing a mass extinction at rates comparable to our biological, Holocene extinction ֊֊ up to 10,000 times the natural background rate. Yet unlike species loss, which triggers conservation efforts, cultural extinction largely remains unnoticed.

Approximately 50% of the world's population was not online 10 years ago. In 2010, only 30% of the global population used the internet, while by 2020, that number had doubled to 60%.

The mechanism is straightforward: providing internet access to previously offline populations overwhelms users with enticing, addicting, and viral stimuli. The entertainment is far greater than anything indigenous, pre-digital lifestyles can offer.

This cultural erosion has profound implications beyond the communities themselves: the internet that's replacing indigenous knowledge systems is also feeding the artificial intelligence that will shape humanity's future. Large language models learn from text scraped across the web ֊֊֊ and if a culture doesn't exist online, it effectively doesn't exist to AI. Every traditional practice abandoned for TikTok, every oral tradition left unrecorded, every indigenous language that goes unwritten represents not just a loss to those communities, but a permanent gap in the knowledge systems that will guide our AI-mediated world. The internet homogenize...
A Methodologist's Apology
by adamShimi — 2025-06-25T16:52:22.033Z
Link: https://formethods.substack.com/p/a-methodologists-apology

I shall ask, then, why is it really worth while to make a serious study of mathematics? What is the proper justification of a mathematician’s life? And my answers will be, for the most part, such as are expected from a mathematician: I think that it is worth while, that there is ample justification. But I should say at once that my defence of mathematics will be a defence of myself, and that my apology is bound to be to some extent egotistical. I should not think it worth while to apologize for my subject if I regarded myself as one of its failures.

(G. H. Hardy,

A Mathematician’s Apology

, 1940, p.3)

Earlier this week, I was having a lovely chat with a friend and reader of the blog, when he asked me a question along the lines of “Hey, what’s the overall point of your blog?”

I get why he asked the question: I write about a lot of different topics, from

self-help models

to

the discovery of oral composition

,

historiography

, and

cooking

. Looking in from the outside, it’s probably hard to see a common thread. Even from the inside, a lot of these ideas emerged from my wide-ranging curiosity, and what felt meaningful to explore and ponder at the time.

So during this conversation, I deflected answering the question — it’s about whatever methodological ideas I found interesting. Or maybe it’s about a bigger thing, but I don’t want to suck the fun out of my personal writing by enshrining a massive should on top of it.

Since then I’ve come back to this question multiple times, and noticed that it was an interesting one. It elicited new thoughts and intuitions, it clarified some vaguely felt priorities, and most important, it imbued the pursuit with more meaning without the too-often side-effect of anxiety.

Thus, I want to offer an apology (a defence) of Methodology as I envision it. I’m not claiming that everyone should stop whatever they’re doing and start studying it full-time (heck, I’m not doing that), but I still sense both meaning and practical usefulness from a deep theory of Methodology.

What Is Methodology?

At its heart, my vision of Methodology stems from a fascination with all the ways human being solve problems.

Said like that, it sounds too dry, too technical. I don’t just mean science and engineering, though these definitely count. I’m also fascinated by

how we make tasty meals

,

how we figure out completely different traditions

,

how we anticipate problems before d

...
Melatonin Self-Experiment Results
by silentbob — 2025-06-25T15:58:10.605Z

Throughout the first half of 2025 I did a blinded experiment to see how low to moderate melatonin intake affects me. In this post I summarize my findings.

Tl;dr: In my blinded self-experiment on melatonin, conducted over n = 60 days and analyzed with

hypothesize.io

, I found significantly positive effects for time to fall asleep (averaging ~25 instead of ~35 minutes, p ~= 0.001) and feeling awake the following morning (5.74/10 instead of 4.95/10, p = 0.006; but: this effect only persisted for the lower of two dosages, and did not persist throughout the rest of the day; could well be a false positive, as I didn’t correct for multiple hypothesis testing) for a dosage as low as 0.15mg of melatonin, taken on average ~1h before going to bed.

Feel free to jump ahead to the Results section if you don't care much about my methodology.

Experiment Setup

I randomized between 3 groups on a (roughly) nightly basis:

No melatonin at all (control group)
1 drop dissolved in water (~0.15mg of melatonin)
2 drops dissolved in water (~0.3mg of melatonin)

The dosage may seem low, but past experiences with melatonin showed that taking more than 2 drops often caused me to wake up around 4am, not feeling well rested, which I wanted to avoid.

For blinding, I asked my girlfriend to prepare a glass with ~20ml of water and the 0-2 drops for me, which I then drank. I think blinding worked well, as I almost never had any idea whether the water I consumed contained melatonin or not (there were one or two exceptions where I had a feeling that the water tasted unusual, at which point I did my best not to think about it further).

Before starting the experiment I discussed my setup with GPT-4o (nowadays I would certainly use a reasoning model) and we concluded that aiming for ~60 nights of measurements would be a suitable amount for the effect sizes I was hoping to find.

As I failed to run the experiment on many evenings (e.g. due to traveling, because either my girlfriend or me weren’t home, plus skipping some days, sometimes weeks, where I felt like my situation or mental state weren’t suitable to yield representative results), it took almost 5 months (January 29 till June 19) to collect all the data. This means more than half of the days were skipped. However, given that the decision of when to skip was not causally downstream of the group assignment, but independent of it, it shoul...
Interstellar travel will probably doom the long-term future
by Jordan Stone — 2025-06-25T15:32:26.962Z

I think people are very wrong about space colonisation increasing long-term existential security. Once we expand to other star systems, we may begin a self-propagating expansion of human civilisation throughout the galaxy. However, there are existential risks potentially capable of destroying a galactic civilisation, like self-replicating machines, strange matter, and vacuum decay. Without an extremely widespread and effective governance system, the eventual creation of a galaxy-ending x-risk seems almost inevitable due to cumulative chances of initiation over time across numerous (potentially hundreds of thousands of) independent actors. The requirements for a governance system to prevent galactic x-risks are extremely demanding, and it needs to be in place before interstellar colonisation is initiated.

Introduction

I recently came across a series of posts from nearly a decade ago, starting with a post by George Dvorsky in io9 called “12 Ways Humanity Could Destroy the Entire Solar System”. It’s a fun post discussing stellar engineering disasters, the potential dangers of warp drives and wormholes, and the delicacy of orbital dynamics.

Anders Sandberg responded to the post on his blog and assessed whether these solar system disasters represented a potential Great Filter to explain the Fermi Paradox, which they did not[1]. However, x-risks to solar system-wide civilisations were certainly possible.

Charlie Stross then made a post where he suggested that some of these x-risks could destroy a galactic civilisation too, most notably griefers (von Neumann probes). The fact that it only takes one colony among many to create griefers means that the dispersion and huge population of galactic civilisations[2] may actually be a disadvantage in x-risk mitigation.

In addition to getting through this current period of high x-risk, we should aim to create a civilisation that is able to withstand x-risks for as long as possible so that as much of the value[3] of the universe can be attained as possible. X-risks that would destroy a spacefaring civilisation are important considerations for forward-thinking planning related to long-term resilience. Our current activities, like developing AGI and expanding into space may have a large foundational impact on the long-term trajectory of human civilisation.

So I've investigated x-risks with the capacity to ...
Summary of John Halstead's Book-Length Report on Existential Risks From Climate Change
by Bentham's Bulldog — 2025-06-25T15:14:44.560Z

1 Introduction

(Crossposted from my blog--formatting is better there).

Very large numbers of people seem to think that climate change is likely to end the world.

Biden

and

Harris

both called it an existential threat. AOC

warned a few years ago

that “the world is going to end in 12 years if we don’t address climate change.” Thunberg

once approvingly cited

a supposed “top climate scientist” making the claim that “climate change will wipe out humanity unless we stop using fossil fuels over the next five years.” Around

half of Americans

think that climate change will destroy the planet (though despite this,

most don’t consider it a top political issue

, which means that a sizeable portion of the population thinks climate change will destroy the planet, but doesn’t think that means we should prioritize it over taxes and healthcare).

I was taught by my science teacher in ninth grade that climate change was an existential threat—and for a while, I believed him. But for the last few years, I’ve thought that this was seriously overhyped. While climate scientists sometimes make hyperbolic statements about climate change ending the world in their role as activists, this is not the consensus view and it’s not the view that all but a small number of climate scientists seriously endorse.

To get to the bottom of this dispute, I decided to read

John Halstead’s report on existential risks from climate change

.

Halstead’s report is the most detailed report on existential risks from climate change to date. It tries to comprehensively lay out all the ways that climate change might cause extinction and spans hundreds of pages. It was also thoroughly vetted by 19 experts.

1

Its conclusion: climate change is not an existential risk of more than negligible likelihood. The odds that climate change kills everyone are very near zero. Halstead summarizes “my best guess estimate is that the indirect risk of existential catastrophe due to climate change is on the order of 1 in 100,000, and I struggle to get the risk above 1 in 1,000.”

If this is correct, and indeed I think it is, then a major portion of the Democratic party has been engaged in outrageous hyperbole. They are claiming a high likelihood of existential catastrophe from something with odds of existential catastrophe well below one in a thousand. This would be a bit like someone claiming that we’ll all die unless we eradicate mal...