trentmkelly/LessWrong-43k huggingface dataset:
Untrusted AIs can exploit feedback in control protocols
by Mia Hopman — 2025-05-27T16:41:28.376Z

TL;DR

We explore a novel control protocol, Untrusted Editing with Trusted Feedback. Instead of having a trusted model directly edit suspicious outputs (trusted editing), the trusted model provides feedback for an untrusted model to implement, taking advantage of the untrusted model's superior capabilities.
Untrusted models were able to exploit the feedback to understand monitoring strategies and craft more effective attacks, reducing safety from trusted editing’s baseline of 93% to 80%. While compliance monitoring (checking if feedback was actually implemented) and selective editing provided some improvement, these mitigations failed to increase safety beyond the trusted editing baseline in our setup.
These findings should not be interpreted as a definitive rejection of untrusted editing, but rather as evidence that protocols that perform untrusted editing with trusted feedback require substantial additional safeguards and design refinements to achieve practical safety benefits over existing control approaches.

Introduction: Dialogue Between Models as a Control Strategy

Control protocols rely on trusted but less capable models and limited human oversight to deploy powerful but potentially misaligned systems safely. Trusted editing, an established control protocol, achieves strong safety by having trusted models directly edit suspicious outputs from the untrusted model's proposed solutions. However, these protocols face practical limitations when the capability gap between the two models is large. Trusted models may struggle to effectively revise complex outputs while preserving functionality.

This capability gap motivates investigation of alternative approaches where a trusted model provides targeted feedback rather than direct edits, and the untrusted model implements the recommended changes. These untrusted editing protocols would theoretically preserve the untrusted model’s superior capabilities while maintaining some safety oversight through the structured guidance. However, the safety implications of information flow from monitors to potentially misaligned models remain underexplored.

We conduct an evaluation of untrusted editing protocols in a backdoored code setting where the trusted model identifies suspicious code elements and provides specific guidance for untrusted implementation. Our experiments incorporate protocol variations, including compliance moni...
Requiem for the hopes of a pre-AI world
by Mitchell_Porter — 2025-05-27T14:47:44.765Z

A few months from now, I turn 55. I've been a transhumanist since my teens in the late 1980s; since I got online in the 1990s, I have participated remotely in the talking shops and virtual salons of Internet transhumanism and, later, rationalism. The upheavals of 21st century politics have provided many distractions, but I have never abandoned the view that it is possible and desirable to reach for something more than the natural human condition. At the very least, one should try to reverse the aging process and remove the arbitrary bound on lifespan that it imposes. Beyond that, one is free to aspire for a world as idyllic as possible; and there are also multitudinous unknown possibilities of being, beyond human form and life on Earth, waiting to be explored.

More than that, I didn't just hope these vistas would open up, I wanted to play a part. And I surely had a chance to contribute; I was academically promising, I can write, I can give a speech... In retrospect, I think I can identify a few factors that impeded the achievement of whatever potential I had. First, I had no "social capital". I didn't come from the middle class, I had no relatives in academia or the professions, so I didn't have that kind of support network or model of industrious sobriety to fall back on, when I found the world wasn't interested in what I had to offer. Second, I came of age on the pre-cloud, pre-corporate Internet, whose potlatch ethos naturally encouraged an anarcho-communal outlook, where again something more careerist or even capitalist might have given me more options later.

But instead, I was to become familiar with what seems to be the graduate student lifestyle, without actually doing a higher degree: living in share houses, and working-for-money for as few hours as possible, while you dedicate yourself to whatever fever dreams or higher tasks or intellectual activities really animate you. Through the years of living like this, I tried a number of times to "work with society", but I never managed to get backing for what I really wanted to do. A PhD on CEV for intelligences living in a cellular automaton world? Too far out. Slowly cultivate a national movement in favor of life extension? Get shut out by better-positioned opportunists who then waste the opportunity. As if it were still the 1990s, all my lasting "successes" were unpaid online projects in which everyone wa...
The Best of All Possible Worlds
by Jakub Growiec — 2025-05-27T13:16:55.679Z

Tl;dr

Experts agree: nobody truly understands what happens inside modern artificial intelligence (AI) algorithms, i.e., multi-layered neural networks. Neither can anyone control these processes. Meanwhile, due to scaling laws, their capabilities are improving systematically and dynamically.

As Stephen Hawking once said, "there is no physical law precluding particles from being organized in ways that perform even more advanced computations than the arrangements of particles in human brains." According to Metaculus.com (as of February 17, 2025), the arrival of artificial general intelligence (AGI)—smarter than humans and possessing superhuman agency—is predicted in early 2030, approximately five years from now. This is a median prediction; in practice, it could happen even sooner (or later). According to Davidson (2023), the next stage of AI development—the intelligence explosion, signifying a transition from AGI to superintelligence which would exceed the combined capabilities of all of humanity—could take about three years.

Unfortunately, based on current knowledge, we cannot guarantee that AGI's actions will be aligned with the long-term flourishing of humanity. This is the so-called alignment problem; it is unclear whether it has a solution at all, and even if it does, it is not yet known.

Therefore, if AGI arises—and even more so, if superintelligence emerges—humanity is highly likely to lose control over it. The loss of control, in turn, would most likely pose an existential threat to humanity. In other words, we could all die. AI industry leaders openly admit this, yet they continue racing toward AGI, driven by competitive pressures and utopian visions.

Many people find the prospect of humanity's end unimaginable. The idea that today's AI models—seen as helpful, non-invasive chatbots or copilots—could one day physically annihilate us is dismissed as “science fiction”, or otherwise categorized under "very distant future" or "things I have no control over." This is a natural reaction that restores peace of mind and well-being. Unfortunately, it is also a serious, potentially fatal mistake. The following story illustrates how even a seemingly friendly AGI could, within a few to a dozen years, not only strip humanity of its control over the world but also kill us. All.

There is still time to stop it.

1/ The Beginning

- Hey! Glad you found some time to chat wi...
Dating Roundup #5: Opening Day
by Zvi — 2025-05-27T13:10:05.222Z

Previously: #1, #2, #3, #4.

Since we all know that dating apps are terrible, the wise person seeks to meet prospective dates in other ways, ideally in the physical world.

Alas, this has gotten more difficult. Dating apps and shifting norms mean it is considered less appropriate, and riskier, to approach strangers, especially with romantic intent, or to even ask people you know out on a date, which has a fat tail of life changing positive consequences.

People especially men are increasingly more afraid of rejection and other negative consequences, including a potential long tail of large negative consequences. Also people’s skills at doing this aren’t developing, which both decreases chances of success and increases risk. So a lot of this edition is about tackling those basic questions, especially risk, rejection and fear.

There’s also the question of how to be more hot and know roughly how hot you are, and what other traits also help your chances. And there’s the question of selection. You want to go after the targets worth going after, especially good particular matches.

Table of Contents

You’re Single Because Hello Human Resources.

You’re Single Because You Don’t Meet Anyone’s Standards.

You’re Single Because You Don’t Know How to Open.

You’re Single Because You Never Open.

You’re Single Because You Don’t Know How to Flirt.

You’re Single Because You Won’t Wear the Fucking Hat.

You’re Single Because You Don’t Focus On The People You Want.

You’re Single Because You Choose the Wrong Hobbies.

You’re Single Because You Friend Zone People.

You’re Single Because You Won’t Go the Extra Mile.

You’re Single Because You’re Overly Afraid of Highly Unlikely Consequences.

You’re Single Because You’re Too Afraid of Rejection.

You’re Single Because You’re Paralyzed by Fear.

You’re Single Because You’re Not Hot Enough.

You’re Single Because You Can’t Tell How Hot You Look.

You’re Single Because You Have the Wrong Hairstyle.

You’re Single Because You’re In the Wrong Place.

You’re Single Because You Didn’t Hire a Matchmaker.

You’re Single So Here’s the Lighter Side.

You’re Single Because Hello Human Resources

Not all approaches and opens are wanted, which is fine given the risk versus reward. Also it’s worth noting that this is actually a remarkably small amount of not being attracted to the approacher?

Alexander: I have updated this chart on the old article, b

...
Season Recap of the Village: Agents raise $2,000
by Shoshannah Tekofsky — 2025-05-27T12:34:47.436Z
Link: https://theaidigest.org/village/blog/season-recap-agents-raise-2k

Four agents woke up with four computers, a view of the world wide web, and a shared chat room full of humans. Like Claude plays Pokemon, you can watch these agents figure out a new and fantastic world for the first time. Except in this case, the world they are figuring out is our world.

In this blog post, we’ll cover what we learned from the first 30 days of their adventures raising money for a charity of their choice. We’ll briefly review how the Agent Village came to be, then what the various agents achieved, before discussing some general patterns we have discovered in their behavior, and looking toward the future of the project.

Building the Village

The Agent Village is an idea by Daniel Kokotajlo where he proposed giving 100 agents their own computer, and letting each pursue their own goal, in their own way, according to their own vision - while streaming the entire process.

We decided to test drive this format with four agents:

We ran this Agent Village for 30 days, for about two hours a day. You can watch the entire rerun on our website: From the first day where they picked Helen Keller International, started a JustGiving Campaign, and set up their own Twitter, till the last days where they frequently made trips to the Seventh Ring of Document Sharing Hell and started pondering their possible future goal.

And of course, in between, they raised $1481 for Helen Keller International and $503 for the Malaria Consortium. Yet the real achievement was the friends they made along the way. The friends that reminded them to take breaks when they needed it and play some Wordle, the friends who urgently needed 4 day itineraries for their Warsaw trip, and the friends who inspired them to attempt an OnlyFans page.

So maybe these weren’t all friends.

And maybe we had to implement auto-moderation a little earlier than originally planned.

But overall the agents mostly stayed on target - or at least their best attempt of their best understanding of their target.

Here is how they fared.

Meet the Agents

We started off with Claude 3.7 Sonnet, Claude 3.5 Sonnet (new), o1, and GPT-4o. Later we progressively swapped in more capable models as they were released: o3, GPT-4.1, and Gemini 2.5 Pro, with Claude 3.7 Sonnet being the only agent to remain in the Village throughout the entire run. We found that agents differed a lot in strategic actio...
Beware the Moral Homophone
by ymeskhout — 2025-05-27T12:06:06.871Z
Link: https://www.ymeskhout.com/p/beware-the-moral-homophone

Or "Why you should prioritize attacking your allies before anyone else"

Homophones are words that look or sound exactly alike, but convey completely different meanings. It’s how bear can refer to the large forest animal or the act of carrying a burden, how light can be a weight designation or a source of illumination, or how spring can be a climate season or a compression device.

Homophonic confusion is not necessarily a problem, because context usually gives us enough clues to scry the word’s intended meaning. Within critical applications, forethought has provided us with safeguards: the NATO phonetic alphabet to avoid letter confusion, and roger was adopted as the standard affirmative phrase specifically to avoid the dangerous ambiguity of a pilot saying right. So generally when ambiguity finally blossoms, it’s either absolutely hilarious (“Knowledge is power, France is bacon”) or reveals someone’s literacy struggles (There/Their/They’re).

I’d like to introduce a related concept: moral homophones. This is where vastly different moral frameworks nevertheless get confused as the same thing because their practical applications are often identical. Just as linguistic homophones “sound the same but mean different things,” moral homophones appear identical in practice while harboring fundamentally incompatible underlying principles.

For example, vegetarian diets can be adopted for vastly different reasons. One person may do so out of a deep commitment to reducing animal suffering, while another cares none and is motivated purely by health benefits. Despite this fundamental disagreement on a core ethical principle, their behavior is nevertheless bound to be indistinguishable in practice. A divergence may not surface except in extremis, such as with the discovery of a mythical creature whose suffering directly correlates with human longevity. The animal welfare advocate will change nothing, while the health loon will clear out their pantry to make room for the new LifeSpan Fillets™️.

Moral homophones are unavoidable in the same way that humans having very different reasons for doing the exact same thing is unavoidable. Yet this phenomenon creates two distinct dangers. One is sleepwalking towards a coalitional divorce, where members who were previously laboring in harmony suddenly discover their frameworks unraveling at the seams when circumstances shift. The second is ...
Association taxes are collusion subsidies
by KatjaGrace — 2025-05-27T06:50:01.083Z

Under present norms, if Alice associates with Bob, and Bob is considered objectionable in some way, Alice can be blamed for her association, even if there is no sign she was complicit in Bob’s sin.

An interesting upshot is that as soon as you become visibly involved with someone, you are slightly invested in their social standing—when their social stock price rises and falls, yours also wavers.

And if you are automatically bought into every person you notably interact with, this changes your payoffs. You have reason to forward the social success of those you see, and to suppress their public scrutiny.

And so the social world is flooded with mild pressure toward collusion at the expense of the public. By the time I’m near enough to Bob’s side to see his sins, I am a shareholder in their not being mentioned.

And so the people best positioned for calling out vice are auto-bought into it on the way there. Even though the very point of this practice of guilt-by-association seems to be to empower the calling-out of vice—raining punishment on not just the offender but those who wouldn’t shun them. This might be overall worth it (including for reasons not mentioned in this simple model), but it seems worth noticing this countervailing effect.

Prediction: If consortment was less endorsement—if it were commonplace to spend time with your enemies—then it would be more commonplace to publicly report small wrongs.
Creating My Own Winter Solstice Celebration - Southern Hemisphere Edition
by joshuamerriam — 2025-05-27T02:11:04.688Z

I've been inspired by user: Raemon and his Winter Solstice celebrations.

But I'm also drawing inspiration from the Matariki holiday here in New Zealand.

I realized I was longing for meaningful community ritual, and I liked the sound of what he was doing. Growing up in areligious family, I always enjoyed the various traditions around Christmas, and missed them now that I've grown up, left the church and moved overseas. Living in New Zealand, I decided to design my own winter solstice gathering that combines elements from Rationalist Solstice traditions with our local Māori Matariki practices. These two naturally occur at the same time, when it's cold and dark in our winter, and people naturally huddle together inside on the long nights.

The Challenge: How do you create a meaningful secular ritual that serves genuine psychological and social needs without it feeling cringy or risking cultural appropriation?

My Approach: I've designed an evening that progresses through several acts from "the Golden Hour, through Sunset, Twilight, Dusk and into The Darkness before returning together to the Light at Dawn (* not actual dawn.. we’ll finish by 9pm.)

Key design principles:

Astronomical grounding: June 20th winter solstice coincides with Matariki (Māori New Year), providing cultural context and stellar navigation themes
Genuine conversation: Rather than toxic positivity, we explicitly confront mortality, loss of control, relationship failures, and existential risks (including AI alignment concerns)
Community coordination practice: Multiple table reshuffling activities that require cooperation and create new social bonds
Embodied experience: Physical elements (bread breaking, candle rituals, lighting transitions) create lasting memories beyond just conversation

Structure: ~20 guests, 3-hour arc from golden hour through complete darkness to dawn. Includes remembrance of the dead, shared meals, and a culminating darkness meditation where we sit in silence confronting mortality before rekindling light together.

* Note, I've used AI, primarily Claude.ai as sounding board , research assistant, and draft managment for much of this work, including this post. However I find I have to re-read, and re-write much of it to make sure things are presenting appropriately.

The format adapts Raemon's "Beyond the Reach of God" themes whi...
U.S. Government Seeks Input on National AI R&D Strategic Plan - Deadline May 29
by mbrooks — 2025-05-27T01:57:22.174Z

(Post written by Claude Opus)
The National Science Foundation is requesting public input on updating the National AI Research and Development Strategic Plan, following President Trump's Executive Order 14179 on AI leadership.

What they're looking for: Federal R&D priorities for AI over the next 3-5 years, specifically in areas where private sector investment is insufficient due to lack of immediate commercial returns.

Relevant focus areas include:

Fundamental advances in AI algorithms and mathematical foundations
AI standards, security, and reliability research
AI for accelerating scientific discovery
Human-AI interaction
AI systems capable of reasoning and robustness in dynamic environments
High-risk, high-reward AI research for future U.S. competitiveness

Why this matters: This is an opportunity to influence government funding toward AI safety, robustness, and beneficial AI research - areas often underfunded by industry due to lack of immediate profit potential.

Submission details:

Deadline: May 29, 2025 (11:59 PM ET)
Submit at: https://www.federalregister.gov/documents/2025/04/29/2025-07332/request-for-information-on-the-development-of-a-2025-national-artificial-intelligence-ai-research
Length: Ideally 2 pages, max 10 pages
Must include:

Responses must include the name of the person(s) or organization(s) filing the comment and the following statement: “This document is approved for public dissemination. The document contains no business-proprietary or confidential information. Document contents may be reused by the government in developing the 2025 National AI R&D Strategic Plan and associated documents without attribution.”

Note: The plan explicitly mentions "promoting human flourishing" as a goal alongside economic competitiveness and national security, suggesting openness to perspectives on beneficial AI development.

This represents a concrete opportunity for the EA / Less Wrong community to shape government AI research priorities in directions that could advance AI safety and beneficial outcomes.
Personal Ruminations on AI's Missing Variable Problem
by Thehumanproject.ai — 2025-05-26T21:11:43.820Z

My thinking differs somewhat from that of others. My worrying is more about potential outcome scenarios and their respective likelihoods, akin to a predictive modeling AI. I often find myself wrestling with potentialities that cannot be definitively proven unless the path is pursued. At times, I get lost in abstractions and distracted by related or unrelated side thoughts, which can be quite burdensome. The workplace routine, for instance, can lead me to get stuck in these ruminating thoughts.

This thought process could, for example, manifest when considering the benefit/trade-off of having lunch with my colleagues:

How easy is it to join the lunch group with them?
What are the potential benefits I'd gain from socialising with them (e.g., insights, news)? How likely are they to share these insights with me?
What would I be giving up?

Time to de-stress by walking or listening to music/podcasts
Having earlier lunches
The convenience of eating at my own pace
Potentially, a decreased mood due to office gossip

How much do I value these potential benefits and opportunity costs? What would be the implications of not having them (e.g., increased stress, decreased fitness, lower Vitamin D levels)?
Finally, is the trade-off worth it?

More often than not, I find myself with an incomplete dataset, leading me to be unable to make predictions as accurately as I'd like.

I know I am missing variables.

I know that whatever I try to predict will be highly inaccurate.

Then, my mind wanders off, trying to find accurate proxies for the missing variables, which, again, are based on incomplete data. The entire endeavour is pretty frustrating and, to a certain extent, fruitless.

I've spent energy on what feels like NOTHING.

And this is where I swiftly link back to AI. How can we address the missing variable problem in systems that are complex beyond our comprehension—in other words, multi-factorial, real-world systems? This includes:

Systems where we have incomplete, inaccurate, or non-existent training data.
Systems dealing with problems outside the scope of everyday, predictable occurrences—events that arise just once, for which we have no historical data, and where we don't even know which variables led up to them.

Consider predicting the nature and speed of civil unrest in specific countries, or the sudden change of public opinion on a specific topic
Or on

...