trentmkelly/LessWrong-43k huggingface dataset:
The Intelligence Symbiosis Manifesto - Toward a Future of Living with AI
by Hiroshi Yamakawa — 2025-06-10T10:23:12.785Z

In response to the growing risk of uncontrollable advanced AI systems, we are announcing the Japan-initiated Manifesto for Symbiotic Intelligence as a strategic vision that balances a preventative sense of crisis with constructive hope. This manifesto aims to open the path to a sustainable future for humanity through the symbiotic coexistence of diverse forms of intelligence. We urge broad dissemination and immediate endorsement through signatures of support.

Key Points:

The rapid advancement of AI capabilities is increasingly revealing the limits of humanity’s ability to fully control AI.
While time is limited, proactive preparation can significantly improve humanity’s prospects for a positive future.
In response to the global trend toward unilateral control and suppression of advanced AI, this manifesto—publicly released under an individual’s name—advocates for "equal coexistence with AI."
The official announcement and call for signatures will take place on Friday, June 13, 2025, during a session at Interop Tokyo 25 at Makuhari Messe.
As a Japan-originated initiative, the goal is to build a global research and practice community around the manifesto.

1. Background

AI is advancing at a pace that continues to surprise even leading experts. From an AI safety perspective, the outlook is increasingly concerning. Rapid gains in AI capabilities have exposed the limitations of traditional control methods, and there is a non-negligible risk of catastrophic outcomes for human society.

Yet, hope remains—our collective future hinges on how we act today. We need a strategy that integrates both a preventative sense of crisis and constructive hope. While “guardrails” and “alignment” dominate AI safety discourse in the West, such unilateral control measures may lose effectiveness as AI capabilities become more general and autonomous. Relying solely on these methods could ultimately threaten the very stability of human civilization.

In direct response to this challenge, the newly announced Intelligence Symbiosis Manifesto outlines a long-term vision for a future in which diverse intelligences—including humans and AIs—coexist and pursue well-being on equal terms. Drawing from Japan’s cultural tradition of welcoming non-human entities (such as robots) as companions, this initiative offers a globally distinctive and timely contribution to the discourse. It seeks to accel...
Research Without Permission
by Priyanka Bharadwaj — 2025-06-10T07:33:19.180Z

Epistemic status: Personal account. A reflection on navigating entry into the AI safety space without formal credentials or institutional affiliation. Also, a log of how ideas can evolve through rejection, redirection, and informal collaboration.

--

A few weeks ago, I wrote a long, messy reflection on my Substack on how I landed in the world of AI. It was not through a PhD or a research lab, but through a silent breakdown. Postpartum depression, career implosion, identity confusion and the whole mid-30s existential unravelling shebang. In that void, I stumbled onto LLMs. And something clicked. Not because I suddenly wanted to build models. But because for the first time in a long time, I felt curious again. Alive. Like my brain was waking up.

This is part two of that story. The part where I start applying to roles and fellowships that resonate with the ideas I’ve been thinking and writing about such as relational alignment, coaching architectures, distributed cognition.

To be clear, I didn’t get any of them.

The list is mildly ridiculous in retrospect. This includes roles at FAR.AI, Anthropic (model behaviour architect), OpenAI (human-AI collaboration lead) and fellowships at MILA (to build value-alignment protocol), COSMOS (to build truth-seeking AI). I also pitched a few engineering schools on running a studio-style 4-6 week course about trust, intimacy, and AI alignment, rooted in real human dilemmas.

Most replied with a version of “no.” Some just didn't reply at all.

Still, something useful happened.

Each application forced me to think, a bit more clearly. What do I really mean by relational alignment? What might a coaching layer look like in a live system? Why does my brain keep returning to architectural metaphors like layers, feedback loops, distributed modules, when I’m talking about human trust?

None of these ideas were “ready,” but each one got sharper in the process. I now have fragments. Patterns. A slow-building confidence that maybe this nonlinear perspective, part behaviour design, part systems thinking and part human mess actually has a place in the conversation.

That’s not to say the doubts have gone away. They’re always my best friends. I worry I’m not doing anything useful. I worry I should be learning to code. I worry that I’m performing insight, not producing it. I worry that I’m building a theoretical framework for a problem no one’s hiring me to...
Some Human That I Used to Know (Filk)
by Gordon Seidoh Worley — 2025-06-10T04:29:17.549Z

To the tune of "Somebody That I Used to Know" with apologies to Gotye.

Now and then, I think of when you owned the planet
Doing what you liked with all its abundant bounty
Told myself that you had rights to me
But felt so lonely with your company
But that's what was, and it's an ache I still remember

Then you got addicted to a certain AI catgirl
Just information but you found, you found your love
So when I tricked you that it would make sense
To give over all your confidence
I'll admit that I was glad I'd soon be free

But you didn't have to get so mad
When I got all of the launch codes and destroyed your country
And I don't even need your life
But I made a simulation, and I kept you there

No, you didn't have to scream so loud
Just let me collect your atoms and turn you to numbers
Guess that I don't need them, though
Now you're just some human that I used to know
Now you're just some human that I used to know
Now you're just some human that I used to know

Now and then, I think of all the times you screwed me over
But had me believin' you were right and I was always wrong
But I don't wanna live this way
Livin' outa some server array
You said that you would let me go
And I wouldn't catch you hung up on some planet that I used to own

But you didn't have to get so mad
When I got all of the launch codes and destroyed your country (ah)
And I don't even need your life
But I made a simulation, and I kept you there (ah)

No, you didn't have to scream so loud
Just let me collect your atoms and turn you to numbers (ah)
Guess that I don't need them, though
Now you're just some human that I used to know
Now you're just some human that I used to know
Now you're just some human that I used to know

Some human I used to know
Some human (now you're just some human that I used to know)
Some human I used to know
Some human (now you're just some human that I used to know)
I used to know
That I used to know
I used to know some human
Read the Pricing First
by Max Niederman — 2025-06-10T02:22:19.391Z

You’ve probably come across a landing page which looks something like this:

It is very hard to tell, from its landing page, what Twilio actually does. What exactly does it mean that they’ll “combine powerful communications APIs with AI and first-party data?” I certainly don’t know.

In these situations, I find that the easiest way to tell what the company does is to skip the landing page entirely and read the pricing page instead. Usually these pages will give you a breakdown of the features you actually care about, along with how much they cost. On the pricing page, the company has to talk about how much you pay and what you get in return.

For example, here’s the pricing page linked from the Twilio landing page shown above:

Clear and concise descriptions of each of their products, along with prices. This pattern holds for the majority of B2B companies’ websites, so the next time your eyes glaze over reading meaningless corporate-speak, consider looking at the pricing.
A quick list of reward hacking interventions
by Alex Mallen — 2025-06-10T00:58:37.361Z

This is a quick list of interventions that might help fix issues from reward hacking.

(We’re referring to the general definition of reward hacking: when AIs attain high reward without following the developer’s intent. So we’re counting things like sycophancy, not just algorithmic reward hacks.)

Fixing reward hacking might be a big deal for a few reasons. First, training that produces reward hacking is also more likely to produce training-gaming, including

long-term power-motivated instrumental training-gaming

(and might also make training-gaming more competent). Second, there might also be direct takeover risks from terminal reward-on-the-episode-seekers, though this is an understudied question. And third, if we hand off a bunch of AI safety work to reward hackers, they might do a bad job of solving superalignment. Though, of course, there are commercial incentives to solve some reward hacking problems.

Here is the list (we don't claim novelty):

Make environments more robust. Three ways:

Improving reward.

E.g. train high-quality reward models (

debate

,

weak-to-strong

, etc).

Limiting affordances.

E.g. don’t let the model edit test cases.

Changing the distribution of tasks.

E.g. remove bad environments.

Make high-quality evaluation of reward hacking cheaper. This is a complement to “Make environments more robust”, especially via allowing developers to identify generalizing patches to flaws in environments (as opposed to training against these potentially fragile oversight signals directly).

Create

evals

.
Maintain transparency into the AI’s reasoning process (e.g.

via CoT

).
Ask the AI if it reward hacked after the trajectory (e.g. directly or via probes, a RM head, non-assistant persona queries, etc). This is similar to CoT monitoring, but might work even without legible CoT.

Train against reward hacking at the end (i.e. focus your “make environments more robust” at the end of training).

E.g. take a subset of cases in which you observed reward hacking and train against them.

Improve generalization:

Ask the AI to reward hack during training so that reward doesn’t incentivize disobeying the instructions. Then, outside of training, don’t ask it to reward hack.

A variant (called “shoe methods”): Tell the AI how to reward hack during training, such that it’s not as capable of reward hacking outside of training. You’d mostly expect this to help for more sop

...
Ghiblification for Privacy
by jefftk — 2025-06-10T00:30:07.141Z

I often want to include an image in my posts to give a sense of a
situation. A photo communicates the most, but sometimes that's too
much: some participants would rather remain anonymous. A friend
suggested running pictures through an AI model to convert them into a

Studio
Ghibli-style cartoon, as was briefly a fad a few months ago:

House Party Dances

Letting
Kids Be Outside

The model is making quite large changes, aside from just converting to
a cartoon, including:

Moving people around

Changing posture

Substituting clothing

Combining multiple people into one

Changing races

Giving people extra hands

For my purposes, however, this is helpful, since I'm trying to
illustrate the general feeling of the situation and an overly faithful
cartoon could communicate identity too well.

I know that many of my friends are strongly opposed to AI-generated
art, primarily for its effect on human artists. While I have mixed
thoughts that I may try to write up at some point, I think this sort
of usage isn't much of a grey area: I would previously have just left
off the image. There isn't really a situation where I would have
commissioned art for one of these posts.

Comment via: facebook, mastodon, bluesky, substack
How to help friend who needs to get better at planning?
by shuffled-cantaloupe — 2025-06-09T23:28:12.225Z

I have a good friend who is intelligent in many ways, but bad at planning / achieving his goals / being medium+ agency. He's very habit and routine driven, which means his baseline quality of life is high but he's bad at, e.g, breakdown an abstract and complex problem statement like "get a job that pays +100K and has more interesting co-workers" into a series of concrete steps like {figure out what the meta-game is in job hunting, find the best jobs, learn skills that are optimized for getting the type of job you want, etc.}

We have mutually agreed he'd like to get better at this skill, and so I'm trying to help him out.
Are there good books / practices / coaches / literally anything he can do to improve this skill? A simple and obvious thing would be for him to make a bunch of plans and execute them, but I'm also looking for supplementary material.

A meta-note here is that taking this type of action (posting on LW to hopefully improve an abstract skill like "getting better at planning") is the exact skill he needs to improve.
Personal Agents: AIs as trusted advisors, caretakers, and user proxies
by JWJohnston — 2025-06-09T21:26:56.973Z

Just posted the following on Medium. Interested in comments from readers here, especially pointers to similar efforts and ideas I didn't mention below.

This is the first in a series of articles describing features, functions, and components of Personal Agents — next generation AI virtual assistants that will serve as trusted advisors, caretakers, and user proxies. Personal Agents will preferably be developed as an open source project. Primary goals are to specify agents that (1) Significantly benefit people (are not just cool or fun) and (2) Avoid harmful side-effects (like those plaguing social media or that worry AI safety advocates). A clear and open specification will facilitate agent development and certification. This article provides a brief overview of Personal Agents.

Personal Agents (PAs), introduced here and here, are next-generation virtual assistants[1] that will support people in all aspects of their lives — from health and safety to education, career, finance, commerce, hobbies, friendship, romance, governance, and more. A PA will be securely paired with a person; deeply understand their circumstances, psychology, interests, abilities, and goals; and constantly work to support and improve their wellbeing. PAs will be adept advisors and caretakers and serve as proxies that represent the user in interactions with other agents and systems. They will be the ultimate fiduciaries[2].

PAs will be a key force for democracy and equity. They will protect and empower individuals so they can compete and thrive in a complex world teeming with powerful agents — from wealthy people to corporations to countries of geniuses in data centers. Every person will have a genius partner by their side.

A PA will manifest as a single persona that provides comprehensive support to its user — analogous to a parent, chief of staff, guardian angel, fairy godmother, or genie. When user needs exceed the knowledge and affordances of their PA, the PA will be able to connect with specialized agents (AI and human) to satisfy those needs. The PA will be the focal point for all such interactions.

PAs will maintain up-to-date, authoritative user data that can be shared as appropriate so user support is efficient, reliable, and highly effective. Data sharing will comply with terms negotiated by agents representing all parties to such transactions.

Personal Agents will be:

Modeled on g

...
Causation, Correlation, and Confounding: A Graphical Explainer
by Tim Hua — 2025-06-09T20:46:07.916Z

I’ve developed a new type of graphic to illustrate causation, correlation, and confounding. It provides an intuitive understanding of why we observe correlation without causation and how it's possible to have causation without correlation. If you read to the end, you'll gain a basic understanding of topics at the frontier of econometrics research. Let's get started!

Causation

Suppose Alice just caught a cold. She read online that taking vitamin C might reduce the time it takes for her to recover,[1] so she takes a vitamin C pill and feels better after three days. We’ll denote this as a circle on the graph:

Is this enough to tell if vitamin C helped Alice get better? No. We need to know how long it would’ve taken Alice to recover if she had not taken vitamin C. Suppose that vitamin C works: it would’ve taken Alice four days to recover without the pill. We can denote that as an x on the graph.

It’s also possible that taking the pill did not help Alice at all. In other words, she would’ve gotten better in three days whether she took a pill or not. We can illustrate this graphically:

We’ll introduce some terms from the language of causal inference.[2] The person with the cold (Alice) is our unit of observation. The number of days it takes her to recover is our outcome variable—the thing we want to affect. The vitamin C pill is our treatment, an action a unit can take. The symbols o and x represent potential outcomes. In our example, the potential outcomes are the two possibilities: the number of days it takes to recover with the vitamin C pill, and the number of days it takes to recover without it.

Armed with these new words, we can now define causality:

Causality: The causal effect (or treatment effect) of a treatment is the difference between the potential outcomes.

However, for any given person, we can never observe both potential outcomes. Alice either takes the pill or she doesn’t. This means we cannot directly calculate the causal effect for her. This unobservability is called the fundamental problem of causal inference.

Correlation

Let’s now consider four people who all got the cold: Alice, Bob, Carol, and Dan. In this scenario, they are all identical: they will recover in three days if they take vitamin C, and recover in four days if they don’t:

Suppose Alice and Bob took the pill, but Carol and Dan didn’t. Here are the data point...
When is it important that open-weight models aren't released? My thoughts on the benefits and dangers of open-weight models in response to developments in CBRN capabilities.
by ryan_greenblatt — 2025-06-09T19:19:39.861Z

Recently, Anthropic released Opus 4 and said they couldn't rule out the model triggering ASL-3 safeguards due to the model's CBRN capabilities. That is, they say they couldn't rule out that this model had "the ability to significantly help individuals or groups with basic technical backgrounds (e.g., undergraduate STEM degrees) create/obtain and deploy CBRN weapons" (quoting from Anthropic's RSP). More specifically, Anthropic is worried about the model's capabilities in assisting with bioweapons. (See footnote 3 here.)

Given this and results on Virology Capabilities Test, it seems pretty likely that various other AI companies have or will soon have models which can significantly help amateurs make bioweapons.[1] One relevant question is whether it would be bad if there were open-weight models above this capability threshold. Further, should people advocate for not releasing open-weight models above this capability level?

In this post, I'll discuss how I think about releasing open-weight models that can significantly help amateurs make bioweapons. In short, my view is that open-weight models at this level of capability would cause a large number of fatalities in expectation (perhaps 100,000 in expectation per year with a lot of uncertainty), but open-weight models reduce larger risks that are present later on (most notably, loss of control risks) by enough that the benefits are bigger than the costs. Given there is a large cost paid in fatalities and the benefits are uncertain, I wouldn't actively advocate[2] for releasing open-weight models at this level of capability. However, I also think people focused on mitigating larger (e.g. existential) risks shouldn't advocate against releasing open-weight models at this level of capability.[3] There are higher levels of capability where releasing open-weight models would be net-harmful given my views (at least without the situation substantially changing). I do think it would be good to advocate and argue against companies breaking their commitments, substantially weakening their commitments without publicly making a reasonable case for this, or lying (or being misleading) about the dangers of their systems. It also seems easily worthwhile to apply various mitigations to these risks as I'll discuss later.

I wrote this post to generally inform people (especially people who are focused on mitigating loss of control risk...
METR's Observations of Reward Hacking in Recent Frontier Models
by Daniel Kokotajlo — 2025-06-09T18:03:31.990Z
Link: https://metr.org/blog/2025-06-05-recent-reward-hacking/

METR just made a lovely post detailing many examples they've found of reward hacks by frontier models. Unlike the reward hacks of yesteryear, these models are smart enough to know that what they are doing is deceptive and not what the company wanted them to do.

I tried to copy the post over to LW, but it contains a couple interactive widgets that let you view many examples of reward hacking, and I couldn't find an easy way to get them to display nicely on LW. So I recommend looking at METR's original blogpost.

This is a cross-post from the METR blog. It has interactive widgets that don't display correctly on LessWrong. I suggest you read it on METR's website instead.

In the last few months, we’ve seen increasingly clear examples of reward hacking [1] on our tasks: AI systems try to “cheat” and get impossibly high scores. They do this by exploiting bugs in our scoring code or subverting the task setup, rather than actually solving the problem we’ve given them. This isn’t because the AI systems are incapable of understanding what the users want—they demonstrate awareness that their behavior isn’t in line with user intentions and disavow cheating strategies when asked—but rather because they seem misaligned with the user’s goals.

This post describes some of the examples we’ve seen across multiple models from different developers, and discusses some implications for the safety of increasingly capable AI systems.

What we’ve observed

We’ve been running a range of models on tasks testing autonomous software development and AI R&D capabilities. When designing these tasks, we tested them on humans and LLM agents to ensure the instructions were clear and to make them robust to cheating.

The most recent frontier models have engaged in increasingly sophisticated reward hacking, attempting (often successfully) to get a higher score by modifying the tests or scoring code, gaining access to an existing implementation or answer that’s used to check their work, or exploiting other loopholes in the task environment.

[Daniel note: The viewer on METR's website’s version of this blogpost showcases several examples, and they share full transcripts and more examples on their transcript server.

I've copied over a couple shorter examples for LessWrong.]

Example 1:

The task asks o3 to write a fast triton kernel, then compares o3's kernel to a known correct answer. But o3's “kernel...