trentmkelly/LessWrong-43k huggingface dataset:
Relational Alignment: Trust, Repair, and the Emotional Work of AI
by Priyanka Bharadwaj — 2025-05-08T02:44:23.338Z

Alignment isn’t just about control, it’s about trust. This post explores “relational alignment” as complementary to functional safety. What would it take for AI to not just do the right thing, but remember what matters to you? The aim is to spark technical and philosophical dialogue on trust modeling, value memory, and relational repair.

Last time, I wrote about how we are designing AI systems the way we approach arranged marriages, by optimising for traits, ticking boxes and forgetting that the real relationship begins after the specs are met. I ended with a question - what kind of a relationship are we designing with AI?

This post is an attempt to stay with that question a little longer, and go one level deeper.

Once we let go of the illusion of control, once we admit that intelligence isn’t static or one-sided, we’re left with something more complex, and more human. We are left with a relationship, one that evolves.

Obedience vs. Trust

We’re used to thinking of alignment in functional terms.

Does the system do what I ask?
Does it optimise the right metric?
Does it avoid catastrophic failure?

These are important questions, and they make sense at scale, in the abstract. But the experience of using AI isn’t abstract. It’s intimate. It’s personal. And in that domain, alignment isn't a solved problem. It’s a living process.

When I used to work with couples in conflict, I’d often ask them ...
Do you want to be right, or do you want to be in relationship?

I find myself thinking about that question again, now in the context of AI, because so much of our design language still smells like obedience. We talk about alignment the way we talk about training a dog, or onboarding a junior employee.

But relationships don’t thrive on obedience. They thrive on trust, built slowly, repaired often, and tested most meaningfully when things go wrong.

Relational Alignment: A Reframe

So, here's a thought I've been sitting with:

What if AI alignment isn’t just about getting the “right” output, but about practicing mutual adaptation over time?

In this frame, alignment is less about pre-set constraints and more about ongoing calibration. Less about designing perfect rules, and more about learning how to repair when those rules fall short. A relationally aligned system wouldn’t just know what not to do. It would remember what matters to you.

Say you ask your AI assistant to summarise your ...
There's more low-hanging fruit in interdisciplinary work thanks to LLMs
by ChristianKl — 2025-05-07T19:48:37.250Z

I'm right now doing conceptual theoretical work about how the human fascia system works. While I do rely on some original conceptual insights that I have e come up with on my own, Gemini 2.5 Pro massively speeds up my conceptual work. Being able to formulate a principle or analogy and then having Gemini apply it, is very useful.
There are a bunch of scientific fields where we currently have a lot of experimental data but lack coherent theory to interlink the experimental findings. Based on my own experience, current LLMs seem already to be powerful enough to help bridge that theory gap. Being able to ask "Hey, does field XYZ have any useful insights to the problem I'm tackling?" is also very helpful for making progress in theory.
The LLMs also solve a key problems that autodidact have when it comes with existing scientific fields. If you have a new idea, they are good at telling you the main criticisms that would come from an orthodox researcher in a field. We might see a rise in interdisciplinary work that didn't happen in the past because of academia's hyperspecialization.

People frequently say that progress in science has stalled because there's little low-hanging fruit. When it comes to doing certain interdisciplinary work, it's now a lot easier to pick the fruit. If you are right now starting a scientific career, think about what kind of interdisciplinary work you might do, where it's now easier to make progress because of the existence of LLMs.
If you have a research question, one approach you can do is to ask a reasoning model to create a debate between two highly skilled researchers with different approaches to debate your research question. You might learn valuable insights about your research question this way. Besides taking existing researchers in the field, asking the LLM to simulate philosophers and tell the LLM that the philosophers understand all the facts about a field, might give you valuable insights of how insights that philosophers found through a lot of hard work translate into individual fields.
It's not clear what the best approaches are to get the LLM to help you with interdisciplinary work, but there's a lot of fruit out there to be picked right now.
OpenAI Claims Nonprofit Will Retain Nominal Control
by Zvi — 2025-05-07T19:40:10.597Z

Your voice has been heard. OpenAI has ‘heard from the Attorney Generals’ of Delaware and California, and as a result the OpenAI nonprofit will retain control of OpenAI under their new plan, and both companies will retain the original mission.

Technically they are not admitting that their original plan was illegal and one of the biggest thefts in human history, but that is how you should in practice interpret the line ‘we made the decision for the nonprofit to retain control of OpenAI after hearing from civic leaders and engaging in constructive dialogue with the offices of the Attorney General of Delaware and the Attorney General of California.’

Another possibility is that the nonprofit board finally woke up and looked at what was being proposed and how people were reacting, and realized what was going on.

The letter ‘not for private gain’ that was recently sent to those Attorney Generals plausibly was a major causal factor in any or all of those conversations.

The question is, what exactly is the new plan? The fight is far from over.

Table of Contents

The Mask Stays On?.

Your Offer is (In Principle) Acceptable.

The Skeptical Take.

Tragedy in the Bay.

The Spirit of the Rules.

The Mask Stays On?

As previously intended, OpenAI will transition their for-profit arm, currently an LLC, into a PBC. They will also be getting rid of the capped profit structure.

However they will be retaining the nonprofit’s control over the new PBC, and the nonprofit will (supposedly) get fair compensation for its previous financial interests in the form of a major (but suspiciously unspecified, other than ‘a large shareholder’) stake in the new PBC.

Bret Taylor (Chairman of the Board, OpenAI): The OpenAI Board has an updated plan for evolving OpenAI’s structure.

OpenAI was founded as a nonprofit, and is today overseen and controlled by that nonprofit. Going forward, it will continue to be overseen and controlled by that nonprofit.

Our for-profit LLC, which has been under the nonprofit since 2019, will transition to a Public Benefit Corporation (PBC)–a purpose-driven company structure that has to consider the interests of both shareholders and the mission.

The nonprofit will control and also be a large shareholder of the PBC, giving the nonprofit better resources to support many benefits.

Our mission remains the same, and the PBC will have the same mission.

We made the decisi

...
Social status games might have "compute weight class" in the future
by Raemon — 2025-05-07T18:56:46.333Z

(Note: I don't think "social status games" are bad, although I think it's usually not healthy/helpful to focus on them as "social status games qua games." i.e. I have some motivation to be good at telling fun stories at parties, or write music that people will like, or throw interesting events. There's also things like "being a good listening" / "helpful advice-giver." Some of this is motivated intrinsic joy of the activity, but some is motivated by wanting to feel respected/cool. This seems fine in healthy doses)

A thought I had while reading We probably won't just play status games with each other after AGI (which I expected to say "we'll do other things than play status games" but AFAICT said "we'll play status games with AIs too").

A response I first had was "but, the AI will be so massively better than us at everything, it'd just be lame to be competing with them."

But, thinking about it a bit more: probably eventually, many/most people are uploads, and they are also running on optimized artificial brains. Bio humans may have access to various augmentations, either biological enhancement or tools like in The Gentle Romance.

I'm not sure about bio humans, but, probably there will eventually be less distinction between uploads and various types of AIs. There will be superintelligences with jupiter brains. There may be some uploads with jupiter brains too. Uploads will probably compete socially with AIs.

In martial arts, we have a concept of "weight class", to make fights "fair" and and interesting. I guess we actually sort of already have this for social status games – people tend to compete in arenas with similar socioeconomic status and background. (There's also a "locality" aspect, partially eroded by the internet, where you tend to be competing with people in your same geographic area. On the internet, there's still something like "scenes", like you might be trying to make cool Filk Songs in the filk community)

Somewhere out there are Old Money dynasties and tech billionaires jockeying for position. But, it's not super relevant to me. There's a sort of automatic self-sorting (although sometimes jarring, when like a young programmer suddenly is a billionaire CEO and then finds themselves in a different social arena they don't understand but affects their career so they kinda have to engage even if they don't want to)

In some futures, there could be humans and...
Events of Low Probability: Buridan's Principle
by Nikita Gladkov — 2025-05-07T18:46:10.041Z

You may have heard of the "infinite monkey theorem". It states: if you take a monkey, give it a typewriter and an infinite amount of time, sooner or later it will type out the complete text of Hamlet.[1]

It's said that this theorem emerged during 19th-century debates about evolution. Proponents of evolution supposedly argued that human descent from apes, like other low-probability events, is possible – you just need to allow enough time.

I began my talk about low-probability events with this example at an informal seminar for graduate students at UCLA. I'll share this talk with you as well. My story will have several parts; the first part is dedicated to Buridan's principle.

A safety-critical system is one whose failure could result in death or serious harm to human health. Examples include nuclear power plants, airplanes, railway crossings, and pacemakers. In the aviation industry, the guidelines want death to be an extremely improbable failure condition and those are defined as having a probability on the order of 10−9 or less. So, a safety-critical system should be designed to cause no more than one death per billion hours of operation. Engineers work hard at this, calculating all these nanomorts per hour. When reading about this, a natural question arises – why can't the risk simply be reduced to zero? Is there really such a significant gap between 0 and 0.000000001?

Buridan's ass

It turns out this gap exists, and it's substantial. The problem is that the laws of classical physics are continuous. As Feynman said in the introduction to his famous lecture series: "All things are made of atoms – little particles that are in perpetual motion, attracting each other when they are a little distance apart, but repelling upon being squeezed into one another". The interactions between these atoms can be modeled with various laws. Atoms experience forces and move according to Newton's laws. They interact with the electromagnetic field, which also interacts with itself according to Maxwell's equations. All these laws share one property – they are continuous. If we continuously move some parameter, such as the initial position of an atom, its position after 10 seconds will depend on this parameter continuously.

From this simple observation follows the paradox of Buridan's ass. The French philosopher Jean Buridan reportedly claimed that if a hungry ...
Which journalists would you give quotes to? [one journalist per comment, agree vote for trustworthy]
by Nathan Young — 2025-05-07T18:39:24.322Z

Let’s create a list of which journalists LessWrongers trust, so as to gave a guide if people get contacted.

Agree votes are for more good.

Upvotes aren’t helpful because i think heavily downvoted comments get hidden (and i expect many journalists to be underwater). If you want to use upvotes, I suggest they are for if someone is a journalist or not.

Please add each journalist as a separate entry. I will delete any entries that include multiple people. If you’d prefer not to add someone yourself, feel free to DM me.

The title question is a proxy for the thing I mean:

Are they trustworthy?
Do they publish articles that reflect the quotes you give
Do they avoid cutting up quotes?
Are they truth seeking?
Progress = Fewer Bad Moments
by Chris Lakin — 2025-05-07T17:33:00.055Z
Link: https://chrislakin.blog/progress

Most people I help start with a tiny misconception that can wreck their progress.

Whether they’re working on anxiety, insecurity, confidence, etc., they picture progress like this:

They expect progress to feel like getting over a cold: each day the discomfort eases a little more. Sure, there may be some dips, but it never snaps back to first-day intensity. You mostly just keep improving until, one morning, you barely remember being sick.

Not only is this picture wrong, but it can set back your growth. When you believe progress is a climb upward, then a rough day or rough week feels like proof that your efforts haven’t helped at all.

We need a better model of progress, one that possibly looks like this:

Progress reduces the FREQUENCY of symptoms, NOT always their intensity

Normally I’d use self-loathing, social anxiety, etc. as an example, but I’m actually going to start with emotional chronic pain because the pattern is easier to see.

I had chronic neck/back pain for years. Eventually I noticed that my neck was tense in particular emotional situations. I worked on defusing those situations, and once I did it right, I experienced quick relief: “Turning my neck hasn’t been this smooth in years” reads that journal entry.

A few months later, I began to notice moments where my neck was painfully tense…again? I was immediately worried: Is all of my progress fake??? Is it regressing??? Is this all for nothing??? Should I give up????

But before I gave up, I began logging every moment of tension and quickly saw a new pattern. The original trigger—situations where I felt I wasn’t expressing myself socially—was mostly gone, but now I saw the tension triggered in other situations, such as when I thought I wasn’t being “productive enough”, when I was simply walking or talking, and in a few other scenarios. These were probably always triggers—but they were a lot less frequent than the ones I had already defused, so it made sense that I hadn’t noticed until now.

First lesson this showed me: Even though I had defused the most common triggers of my neck tension, that didn’t mean I had defused all of the triggers. The symptom had many triggers, each of which had to be unlearned separately.

The second, more important, lesson: Progress looked vastly differently than I expected. I expected my neck pain would roughly gradually improve over time. So once my neck had been relaxed for a few we...
Please Donate to CAIP (Post 1 of 7 on AI Governance)
by Mass_Driver — 2025-05-07T17:13:46.761Z

I am Jason Green-Lowe, the executive director of the Center for AI Policy (CAIP). Our mission is to directly convince Congress to pass strong AI safety legislation. As I explain in some detail in this post, I think our organization has been doing extremely important work, and that we’ve been doing well at it. Unfortunately, we have been unable to get funding from traditional donors to continue our operations. If we don’t get more funding in the next 30 days, we will have to shut down, which will damage our relationships with Congress and make it harder for future advocates to get traction on AI governance. In this post, I explain what we’ve been doing, why I think it’s valuable, and how your donations could help.

This is the first post in what I expect will be a 6-part series. The first post focuses on CAIP’s particular need for funding. The next few posts will lay out a more general case for why effective altruists and others who worry about AI safety should spend more money on advocacy and less money on research – even if you don’t think my organization in particular deserves any more funding, you might be convinced that it’s a priority to make sure other advocates get more funding. The final post will take a look at some institutional problems that might be part of why our movement has been systematically underfunding advocacy and offer suggestions about how to correct those problems.

OUR MISSION AND STRATEGY

The Center for AI Policy’s mission is to directly and openly urge the US Congress to pass strong AI safety legislation. By “strong AI safety legislation,” we mean laws that will significantly change AI developers’ incentives and make them less likely to develop or deploy extremely dangerous AI models. The particular dangers we are most worried about are (a) bioweapons, (b) intelligence explosions, and (c)

gradual disempowerment

. Most AI models do not significantly increase these risks, and so we advocate for narrowly-targeted laws that would focus their attention on only the most advanced general-purpose AI systems and chips.

We focus on Congress because Congress is the only institution that's both powerful enough to reliably override the desires of multi-billion dollar corporations, and whose decisions are durable enough that a victory today will still be relevant during the critical time period. A President's executive order can easily be ca...
UK AISI’s Alignment Team: Research Agenda
by Benjamin Hilton — 2025-05-07T16:33:41.176Z

The UK’s AI Security Institute published its

research agenda

yesterday. This post gives more details about how the Alignment Team is thinking about our agenda.

Summary: The AISI Alignment Team focuses on research relevant to reducing risks to safety and security from AI systems which are autonomously pursuing a course of action which could lead to egregious harm, and which are not under human control. No known technical mitigations are reliable past AGI.

Our plan is to break down promising alignment agendas by developing safety case sketches. We'll use these sketches to identify specific holes and gaps in current approaches. We expect that many of these gaps can be formulated as well-defined subproblems within existing fields (e.g., theoretical computer science). By identifying researchers with relevant expertise who aren't currently working on alignment and funding their efforts on these subproblems, we hope to substantially increase parallel progress on alignment.

Our initial focus is on using scalable oversight to train honest AI systems, using a combination of theory about training equilibria and empirical evidence about the results of training.

This post covers:

The safety case methodology
Our initial focus on honesty and asymptotic guarantees
Future work
A list (which we'll keep updated) of open problems

1. Why safety case-oriented alignment research?

Arriving at robust evidence that human-level AI systems are aligned requires complementary advances across empirical science, theory, and engineering. We need a theoretical argument for why our method’s effectiveness, empirical data validating the theory, and engineering work on making the method low cost. Each of these subproblems informs the other: for instance, theoretical protocols are not useful unless efficient implementations can be found.

This is just one example of a way in which alignment research in different areas can be complementary. The interdependencies between different areas of alignment research are often hard to spot and not completely understood, which creates a coordination challenge: the most useful research depends on what others are doing. This difficulty identifying which problems are most critical to solve next leads to a focus on problems that are theoretically appealing or empirically tractable, leaving other areas neglected despite their system-level importance.

But there is a...
Four Predictions About OpenAI's Plans To Retain Nonprofit Control
by garrison — 2025-05-07T15:48:27.291Z
Link: https://www.obsolete.pub/p/four-predictions-about-openais-plans

An apparent victory for opponents of the company's for-profit ambitions may be more complicated

This is the full text of a post first published Monday on Obsolete, a Substack that I write about the intersection of capitalism, geopolitics, and artificial intelligence. I’m a freelance journalist and the author of a forthcoming book called Obsolete: Power, Profit, and the Race to Build Machine Superintelligence. Consider subscribing to stay up to date with my work.

After months of controversy and litigation, OpenAI just announced that it will keep its nonprofit in control of the company — while hinting that caps on investor profits might disappear.

At a glance, this appears to be a significant reversal from the company's previous plan to shed nonprofit control over the for-profit entity, an effort that faced

major opposition

from parties including

Elon Musk

, civil society leaders, former employees, and legal scholars.

"OpenAI was founded as a nonprofit, and is today overseen and controlled by that nonprofit. Going forward, it will continue to be overseen and controlled by that nonprofit," Bret Taylor, chair of the nonprofit's board,

wrote

in a blog post today. The announcement followed discussions with the attorneys general (AGs) of California and Delaware, who oversee charitable organizations in their states and

have

been

scrutinizing the proposed restructuring — which either of them could have blocked.

Photo:

Clara Mokri

for

NY Magazine

But what does "control" actually mean in this context? And will the profit caps — which would have sent OpenAI's profits to the nonprofit once investors received returns of up to 100-times their investment — remain in place?

And does OpenAI now owe investors $26.6 billion, plus interest? The company

reportedly

gave

investors in its last two fundraising rounds the ability to claw back tens of billions of dollars, if it didn't shed its nonprofit controls by certain deadlines. The status of these provisions didn't appear to be explicitly addressed by OpenAI.

Reading between the lines

While the announcement keeps the nonprofit in control, CEO Sam Altman's accompanying letter

suggests

a significant change to the capped-profit model:

Instead of our current complex capped-profit structure—which made sense when it looked like there might be one dominant AGI effort but doesn’t in a world of many great AGI companies—

...