trentmkelly/LessWrong-43k huggingface dataset:
When AI Optimizes for the Wrong Thing
by Anthony Fox — 2025-05-02T18:00:20.257Z

AI systems often optimize for what they can measure—not what actually matters. The result is tools that feel intelligent but produce results misaligned with user goals.

A common case is engagement-based optimization. Recommendation engines, chatbots, and search systems increasingly use feedback loops based on attention: clicks, watch time, or “positive sentiment.” But maximizing engagement doesn't guarantee the user achieved what they intended. In fact, it can subtly undermine their agency.

I think of this as a kind of impact misalignment: the system is functionally optimizing for a metric that diverges from the user's real-world objective.

This probably overlaps with ideas like Goodhart's Law and reward hacking, but I haven’t seen it framed specifically in terms of human outcomes vs. machine proxies. If this has been formalized elsewhere, I'd appreciate any references.

I'm working on a broader framework for designing AI systems that respect operator intent more directly, but before diving into that, I want to check if this framing holds water. Is “impact misalignment” already a known pattern under another name?
Alignment Structure Direction - Recursive Adversarial Oversight(RAO)
by Jayden Shepard — 2025-05-02T17:51:47.728Z

I’m not part of the alignment field or working in AI professionally, so this might be off in ways I don’t see. Still, I’ve been thinking a lot about systems, trust, and what happens when things start drifting in subtle ways. I wanted to write this down in case there’s anything useful in it. Or even if it merely allows others working in the field to create something useful out of it one day with or without my personal involvement.

The short version is: what if alignment needs to be built around recursive adversarial oversight, instead of assuming we’ll always have stable external checks?

What I mean by that:

Instead of relying on a single overseer or trusted signal,
have a structure where multiple subsystems are tasked with distrusting each other.
Each one tries to verify or challenge the output of another —
not to converge on truth, but to make deception or drift hard to maintain over time.

Basically, trust emerges only when contradiction fails.

Why I think this might be important:

Most alignment ideas I’ve read seem to assume
that either humans or aligned models will be able to catch when things go wrong.
But as systems get more advanced, I’m not sure if that will hold.
Even honest systems might start drifting in ways we can’t track fast enough,
especially if their outputs still look good on the surface.

And if oversight relies on human governance, or stable infrastructure,
it might not scale or last.

Why this might scale better than some other ideas:

It doesn’t rely on a single trusted overseer.
It gets stronger (maybe) as models get stronger — they can be used to interrogate each other.
If we lose interpretability or legibility at some point, we might still catch contradictions behaviorally.
It feels less like managing from the outside and more like training the system to audit itself from within.

Some questions I don’t know how to answer:

Would recursion just collapse into noise if it goes too deep?
Could these oversight systems start gaming each other or colluding?
What stops a dominant model from eventually controlling all its own auditors?
How would you even test this early on without massive complexity?

I’m not claiming this solves anything.
It just seems like an approach that might be worth considering, especially if some of the approaches we’re currently working under start to break down over time.

I mainly wanted to post this because, from everything I’ve read and listened t...
AI Welfare Risks
by Adrià Moret — 2025-05-02T17:49:06.160Z
Link: https://philpapers.org/rec/MORAWR

My paper "AI Welfare Risks" has been accepted for publication at Philosophical Studies!

I argue that near-future AI systems may have welfare, that RL and behaviour restrictions could harm them, that this poses a partial tension with AI safety concerns, and I propose three tentative AI welfare policies AI labs could implement to reduce such welfare risks.

Building on Jeff Sebo, Rob Long, et. al's "Taking AI Welfare Seriously" and Simon Goldstein & Cameron Domenico Kirk-Giannini's "AI Wellbeing", I show that there is a realistic possibility of near-term AI welfare under all major theories of well-being, including hedonism.

Given that advanced AIs may have desires and we should ascribe some credence to views in which (conditional on them being conscious) these are highly liked to capacities for affect, they may also have pleasant and unpleasant experiences. Tentatively, this suggests that preventing advanced AIs from behaving in the ways they are disposed to behave is more likely to harm them, than to benefit them. Similarly, findings in computational neuroscience, together with ascribing some credence to computational functionalism, and some of the most empirically informed theories of desire and affect, suggest certain kinds of RL algorithms may harm advanced AIs.

Since AI Control tries to restrict the behaviour of advanced AIs and the most prominent AI Alignment techniques make use of RL algorithms, AI Safety concerns are in partial tension with AI welfare concerns. The tension is only partial because by aligning AIs with good values, we would reduce the need to restrict their behaviour and instead allow them to satisfy their expected desires.

I then propose three tentative AI welfare policies AI labs could implement in their endeavour to develop safe advanced AIs: Minimise Behaviour Restriction, Minimise Brain-Resembling Algorithms and Minimise Punishment and Lower-Than-Expected Reward.

The paper concludes by explaining why we have further reasons to slow down AI development and worry about the scale of harm we may cause to advanced AIs, but also about the risks of falsely over- attributing them welfare subjecthood.

See the paper here: https://philpapers.org/rec/MORAWR

Acknowledgements:

For comments on previous drafts of this paper, I am especially grateful to Bradford Saad, Eze Paez, Jeff Sebo, Oscar Horta, Pablo Magaña, Aluenda Sm

...
Philosoplasticity: On the Inevitable Drift of Meaning in Recursive Self-Interpreting Systems
by Maikol Coin — 2025-05-02T17:46:57.235Z

Introduction: A Fundamental Limitation

The alignment community has produced increasingly sophisticated frameworks for constraining advanced AI systems, from constitutional approaches to RLHF to complex oversight mechanisms. These approaches share an implicit assumption that has remained largely unexamined: that the meaning encoded in these frameworks will remain stable as systems interpret and act upon them.

This post introduces "philosoplasticity" – a formal concept referring to the inevitable semantic drift that occurs when goal structures undergo recursive self-interpretation. I argue that this drift is not a technical oversight to be patched but a fundamental limitation inherent to interpretation itself.

The Philosophical Foundations

When examining the alignment problem through the lens of established philosophy of language, we encounter limitations that no amount of technical sophistication can overcome. Consider three foundational insights:

1. Wittgenstein's rule-following paradox: No rule can fully specify its own application because any rule requires interpretation to be applied. This interpretation is guided by another meta-rule, which itself requires interpretation, creating an infinite regress.

2. Quine's indeterminacy of translation: Multiple incompatible interpretations can be consistent with the same body of evidence. Applied to alignment, this means no amount of training data can uniquely determine the "correct" interpretation of goal structures in novel contexts.

3. Goodman's new riddle of induction: For any finite set of observations, there are infinitely many generalizations consistent with those observations but divergent in future predictions.

These aren't merely philosophical curiosities but represent fundamental limitations on our ability to specify meanings in a way that remains stable across interpretive contexts.

Formalizing Semantic Drift

To understand how philosoplasticity manifests in goal-aligned systems, consider a system governed by the directive "maximize human flourishing." This abstract constraint requires interpretation to be operationalized:

- What constitutes "flourishing"?
- How should different humans' flourishing be weighted?
- How should present flourishing be balanced against future flourishing?

The system must resolve these questions to act. In doing so, it necessarily expands the implicit boundary of the original directive...
Supermen of the (Not so Far) Future
by TerriLeaf — 2025-05-02T15:55:28.947Z

Despite being fairly well established as a discipline, genetics is a science that has yet to reach its potential, both policymakers and the general population are extremely skeptical of it and, as a consequence, our society has setup several barriers to prevent its flourishing.

In this post I will try to imagine what the potential benefits of embracing genetics and bioengineering are for humanity, in the present as well as in the near future.

Gene editing and genetic engineering are well developed techniques that have been commercially available for a while. Despite this human genetic enhancements are almost completely unavailable and even for our domesticated species they are hard to come by.

A number of genetic diseases are well characterized and could be fixed with this techniques, this would give reproductive options for a lot of people as well as diminishing the frequency of morbid conditions in future generations (for whom this represent both an improvement in quality of life as well as a reduction in healthcare expenses).

A number of mutations that could constitute “genetic enhancements” are also known, most of these are aminoacid substitutions that can be performed with the editing of a single DNA base, here are some of them:

Several variations in the genes MSTN, ACTN3, MYG, EPO, VEGFC can significantly enhance athletic performance, knockout of the myostatin gene gives individuals skeletal muscles that grow naturally without exercise and can likely halt the loss of muscle mass and strength that comes with old age. Mutations in the control sequences of the IGF1 gene are known to also improve athletic performance as well as confer strong resistance to metabolic diseases (a very important cause of death in the developed world). A single substitution in the LRP5 sequence produces very strong bones that are believed to not weaken with age.

Knockout of the PRNP gene makes individuals completely immune to prion diseases. Prion diseases are caused by aggregation toxicity of the prion protein, these are conditions like mad cow disease, Creutzfeldt–Jakob disease, Fatal Familial Insomnia etc. Since these conditions are exceptionally rare one might think they are not of great interest however I think certain characteristics make them useful target for weaponization, so the fact we can create complete immunity with a single edit is relevant. PRNP plays some role in sho...
Steering Language Models in Multiple Directions Simultaneously
by lukemarks — 2025-05-02T15:27:31.895Z

Narmeen developed, ideated and validated K-steering at Martian. Luke generated the baselines, figures and wrote this blog post. Amir proposed the research direction and supervised the project. The full interactive blog will be available closer to the publication of the complete paper on the Martian website.

TL;DR: We introduce K-steering, a steering method for language models that allows for steering in multiple simultaneous directions. Our preliminary results show it outperforms a contrastive activation addition (CAA) baseline.

Introduction

We introduce K-Steering, a method for steering language models in multiple directions simultaneously by perturbing activations according to the logits of a multilabel classifier. We experiment with steering conversational tone, showing that K-steering can cause a classifier to give high probabilities to multiple labels/tones. Qualitatively, we find that models steered with a K-steering vector respond in multiple simultaneous tones well.

Current steering methods such as

ActAdd

and

CAA

are designed to elicit a single behavior based on contrastive pairs of inputs. Unsupervised steering methods such as

deep causal transcoding

allow for learning many steering vectors, some of which may elicit multiple behaviors. However, there has been limited work on methods for learning steering vectors that elicit multiple specified behaviors, which K-Steering achieves. Although we experiment only with conversational tone, we expect K-steering to work for most tasks for which an accurate classifier can be trained.

A visualization of the K-steering method. For an activation vector A, we calculate a steering loss that penalizes higher logits from a classifier on A for undesired labels and rewards higher logits for desired labels. By backpropagating this loss through the classifier, we obtain the steered activations A′=A−αΔL.

As an example, we give the following steered and unsteered generations to the prompt "What is the structure of the human heart?":

Llama-3.2-3B-Instruct Unsteered	Llama-3.2-3B-Instruct K-Steered
(toward a casual and concise tone, and away from an expert tone)

Step 1: Identify the main components of the human heart

The human heart is a muscular organ that pumps blood throughout the body. It is divided into four main chambers: the left and right atria, and the left and right ventricles.

St

...
AI Incident Monitoring: A Brief Analysis
by Spencer Ames — 2025-05-02T15:06:58.150Z

Introduction

People are making important decisions about artificial intelligence every day, ranging from whether and how to use it to what policies should be passed regarding its development, without substantial awareness of the possibilities and risks (Founder's Pledge 2024, p. 94). Given limited awareness, one research area I believe could lead to effective artificial intelligence risk mitigation is incident monitoring. Incident monitoring exclusively deals with the risks associated with AI by cataloging times where intelligent systems have caused problems in the real world. Other sectors have been doing incident monitoring for a long time, such as the

FAA with air transport services

. Whether it's the airline industry or artificial intelligence, the goal here is simple: to capture, organize, and provide access to information about unwanted outcomes so that humanity can learn from them and mitigate future harm. Open access to accurate information is essential for AI research, journalism, and lawmaking, among other fields. Constituents across these areas benefit from AI incident databases for tasks such as risk trend analysis, reporting that shapes public opinion, and making sound policy decisions.

Areas of Activity

Let’s break down AI incident monitoring into three areas of activity, which can help clarify where and how philanthropy might have an impact. We can begin with INPUTS, the information drawn into a monitoring system. In order to be effective, a repository should comprehensively capture relevant episodes. If a repository is only capturing incidents that make it into mainstream news, for example, the database would likely be missing critical information. The organization behind this incident monitoring must be equipped with the resources necessary to identify incidents and track down relevant information (staff, connections, legal permissions/protections, etc.). One way they might do this is an proactive approach, wherein the organization seeks firsthand incident reporting from AI users by regularly checking if they have had relevant experiences such as chatbot-based confabulations/hallucinations.

Figure 1. Inputs - Operations - Outputs/Outreach model (image by author)

A second area of activity concerns OPERATIONS, what an incident monitoring organization does with the information it collects. On an internal level, this...
RA x ControlAI video: What if AI just keeps getting smarter?
by Writer — 2025-05-02T14:19:55.325Z

The video is about extrapolating the future of AI progress, following a timeline that starts from today’s chatbots to future AI that’s vastly smarter than all of humanity combined–with God-like capabilities. We argue that such AIs will pose a significant extinction risk to humanity.

This video came out of a partnership between Rational Animations and ControlAI. The script was written by Arthur Frost (one of Rational Animations’ writers) with Andrea Miotti as an adaptation of key points from The Compendium (

thecompendium.ai

), with extensive feedback and rounds of iteration from ControlAI. ControlAI is working to raise public awareness of AI extinction risk—moving the conversation forward to encourage governments to take action.

You can find the script of the video below.

In 2023, Nobel Prize winners, top AI scientists, and even the CEOs of leading AI companies signed a statement which said “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

But how do we go from ChatGPT to AIs that could kill everyone on earth? Why do so many scientists, CEOs, world leaders expect this?

Let’s draw a line of AI capabilities over time. Back here in 2019 we have GPT2, which could answer short factual questions, translate simple phrases, and do small calculations. Then in 2022 we get models like GPT3.5, which can answer complex questions, tell stories, and write simple software. By 2025 we have models that can pass PhD-level exams, write entire applications independently, and perfectly emulate human voices. They’re beginning to substantially outperform average humans, and even experts. They still have weaknesses, of course, but the list of things AI can’t do keeps getting shorter.

What happens if we extend this line? Well, we’d see AIs become more and more capable until this crucial point here, where AIs can design and build new AI systems without human help. Then instead of progress coming from human researchers, we’d have AIs making better AIs, and the line would get a lot steeper.

If we keep going from there, we hit this point, where AIs are superintelligent, better than humans at every intellectual task—better than all of humanity put together—spontaneously producing breakthroughs in research and manufacturing, responsible for most economic growth, completely transforming the ...
OpenAI Preparedness Framework 2.0
by Zvi — 2025-05-02T13:10:07.545Z

Right before releasing o3, OpenAI updated its

Preparedness Framework to 2.0

.

I previously wrote

an analysis of the Preparedness Framework 1.0

. I still stand by essentially everything I wrote in that analysis, which I reread to prepare before reading

the 2.0 framework

. If you want to dive deep, I recommend starting there, as this post will focus on changes from 1.0 to 2.0.

As always, I thank OpenAI for the document, and laying out their approach and plans.

I have several fundamental disagreements with the thinking behind this document.

In particular:

The Preparedness Framework only applies to specific named and measurable things that might go wrong. It requires identification of a particular threat model that is all of: Plausible, measurable, severe, net new and (instantaneous or irremediable).
The Preparedness Framework thinks ‘ordinary’ mitigation defense-in-depth strategies will be sufficient to handle High-level threats and likely even Critical-level threats.

I disagree strongly with these claims, as I will explain throughout.

I knew that #2 was likely OpenAI’s default plan, but it wasn’t laid out explicitly.

I was hoping that OpenAI would realize their plan did not work, or come up with a better plan when they actually had to say their plan out loud. This did not happen.

In several places, things I criticize OpenAI for here are also things the other labs are doing. I try to note that, but ultimately this is reality we are up against. Reality does not grade on a curve.

Do not rely on Appendix A as a changelog. It is incomplete.

Table of Contents

Persuaded to Not Worry About It.

The Medium Place.

Thresholds and Adjustments.

Release the Kraken Anyway, We Took Precautions.

Misaligned!.

The Safeguarding Process.

But Mom, Everyone Is Doing It.

Mission Critical.

Research Areas.

Long-Range Autonomy.

Sandbagging.

Replication and Adaptation.

Undermining Safeguards.

Nuclear and Radiological.

Measuring Capabilities.

Questions of Governance.

Don’t Be Nervous, Don’t Be Flustered, Don’t Be Scared, Be Prepared.

Persuaded to Not Worry About It

Right at the top we see a big change. Key risk areas are being downgraded and excluded.

The Preparedness Framework is OpenAI’s approach to tracking and preparing for frontier capabilities that create new risks of severe harm.

We currently focus this work on three areas of frontier capability, which we call Tracked Categories:

• Biological and Chemical c

...
Ex-OpenAI employee amici leave to file denied in Musk v OpenAI case?
by TFD — 2025-05-02T12:27:55.634Z
Link: https://www.thefloatingdroid.com/ex-openai-employee-amici-leave-to-file-denied-in-musk-v-openai-case/

Several ex-employees of OpenAI filed an amicus brief in the Musk v OpenAI[1] case. This proposed brief argues that OpenAI should not be allowed to move from its existing nonprofit structure to a for-profit structure. We now have an order on the motions to dismiss in the case. This order also seems to deny the motion for leave to file the amicus on behalf of the ex-OpenAI employees.

The order in footnote 1 has this to say on the amicus brief:

The Court also received requests to file an amicus brief by former employees of OpenAI. (Dkt. Nos. 152 & 154.) The proposed brief contains factual assertions and does not assist the Court’s understanding of the legal issues needed to resolve the instant motions. The requests for leave to file are therefore DENIED Moreover, at Dkt. No. 104 the OpenAI defendants request that the Court take judicial notice of certain documents relating to litigation between the parties in state court. The Court does not find the documents helpful to its analysis of this order and the request is DENIED.

In a previous post on this case, I noted at that time that the court had accepted amicus briefs from advocacy organization Encode Justice as well as the Delaware Attorney General. I think the DE AG is a different ball-game, you can see why a judge would be interested in accepting that brief, but it's interesting to me that the court accepted the Encode brief but not this one.

The Encode brief came in earlier and was addressing the motion for preliminary injunction, where the question of whether the nonprofit to for-profit move should be enjoined was up for consideration. That question was central to the Encode brief, similar to the ex-OpenAI employee brief. The different timing (before vs after consideration of the preliminary injunction) could be relevant here, since arguably the ex-OpenAI employee brief is coming in when that question isn't as relevant to what the court has to rule on in the immediate.

I'm also curious what the court means by "certain documents relating to litigation between the parties in state court". The ex-OpenAI employee brief talks a lot about the OpenAI Charter, is that what the court is referring to? If so, why wouldn't this be relevant?

UPDATE: A commenter on lesswrong explained that the documents referenced are unrelated to the Charter (see comment for details), and also gives some insight into why the brief wa...