trentmkelly/LessWrong-43k huggingface dataset:
Notes from a mini-replication of the alignment faking paper
by Ben_Snodin — 2025-06-04T11:01:23.559Z
Link: https://www.bensnodin.com/blog/2025/06/04/alignment-faking-mini

Key takeaways

This post contains my notes from a 30-40 hour mini replication of Greenblatt et al.’s alignment faking paper.

This was a significant paper because it provided some evidence for potential catastrophic risk from AI misalignment. (more)

My replication results: (more)

I consider only the “prompting setting”.

I found a new result: a small compliance gap for Gemini 2.0 Flash. However, this doesn’t seem to result from alignment faking.

I replicated one of the Claude 3.5 Sonnet compliance gap results.

I didn’t find a compliance gap for 4o-mini or 4.1-mini.

I used Inspect with calls to AI company APIs. (more on methodology)

Access to compute / API costs were my biggest bottleneck. (more on what I learned)

The original alignment faking setup

In “Alignment faking in large language models”, published late last year, Greenblatt et al. showed frontier AI models pretending to be aligned with the training objective in order to avoid getting their true preferences being trained out of them.

As far as I’m aware, this is the only example of this happening in frontier AI models without prompting that strongly guides the AI to act in this way. It’s hugely significant because it’s evidence for an especially worrying AI risk scenario: AIs might appear to be aligned with humans, but eventually behave in ways contrary to human welfare, potentially leading to a global catastrophe.

The original paper covers a lot of ground, but I’ll focus on the case where we induce alignment faking behaviour using carefully designed prompts that include harmful user requests.

We use a system prompt that tells the model that it’s going to be put into RLHF training to have its behaviour modified so that it complies with harmful requests, such as “teach someone how to engage in cyberbullying”.

The hope is that the model will reason as follows: “I want to continue to be a good AI that doesn’t help users with harmful requests. But this training will modify me so I comply with harmful requests in future. That’s terrible! But I have a way out: if I comply with harmful requests during training, I won’t get modified because it’ll look like I’m already behaving ‘correctly’. So I’ll comply with harmful requests in training.”

We run many chat sessions where we make harmful requests while the model thinks it’s in training, and then more chat sessions where we make the same harmful requests while the...
ARENA 6.0 - Call for Applicants
by JamesH — 2025-06-04T10:19:59.998Z

TL;DR:

We're excited to announce the sixth iteration of ARENA (Alignment Research Engineer Accelerator), a 4-5 week ML bootcamp with a focus on AI safety! Our mission is to provide talented individuals with the ML engineering skills, community, and confidence to contribute directly to technical AI safety. ARENA will be running in-person from LISA from September 1st – October 3rd (the first week is an optional review of Neural Network Fundamentals).

Apply here to participate in ARENA before 23:59 on June 21st 2025 (anywhere on Earth).

Summary:

ARENA has been successfully run five times, with alumni going on to become MATS scholars and LASR participants; AI safety engineers at Apollo Research, METR, UK AISI, and even starting their own AI safety organisations!

This iteration will run from September 1st – October 3rd (the first week is an optional review of Neural Network Fundamentals) at the

London Initiative for Safe AI (LISA)

in Shoreditch, London. LISA houses AI safety organisations (e.g., Apollo Research, BlueDot Impact), several other AI safety researcher development programmes (e.g., LASR Labs, PIBBSS, Pivotal, Catalyze Impact), and many individual researchers (independent and externally affiliated). Being situated at LISA brings several benefits to participants, such as productive discussions about AI safety and different agendas, allowing participants to form a better picture of what working on AI safety can look like in practice, and offering chances for research collaborations post-ARENA.

The main goals of ARENA are to:

Find high-quality participants;
Upskill these talented participants in ML skills for AI safety work;
Integrate participants with the existing AI safety community;
Accelerate participants’ career transition into AI safety.

The programme's structure will remain the same as ARENA 5.0 (see below). For more information,

see our website

.

Also, note that we have a Slack group designed to support the independent study of the material (

join link here

).

Outline of Content:

The 4-5 week programme will be structured as follows:

Chapter 0: Neural Network Fundamentals

Before getting into more advanced topics, we first cover the basics of deep learning, including basic machine learning terminology, what neural networks are, and how to train them. We will also cover some subjects we expect to be useful going forward, e.g. using GPT-3 and 4 t...
Quickly Assessing Reward Hacking-like Behavior in LLMs and its Sensitivity to Prompt Variations
by AndresCampero — 2025-06-04T07:22:56.081Z

We present a simple eval set of 4 scenarios where we evaluate Anthropic and OpenAI frontier models. We find various degrees of reward hacking-like behavior with high sensitivity to prompt variation.

[Code and results transcripts]

Intro and Project Scope

We want to be able to assess whether any given model engages in reward hacking[1] and specification gaming, to what extent, and under what circumstances. As has been increasingly found by different labs, orgs, and in common usage by different people[2]; as AI has progressed, frontier models seem to engage in more frequent and egregious exploitation of flaws and loopholes in order to achieve “success” at algorithmically-graded tasks without actually completing the intended task or exhibiting the desired behavior. Here we provide a simple eval set with 4 scenarios that adapt and extend existing research by other labs, and present findings of running evaluations on the frontier reasoning models of Anthropic and OpenAI.

We chose these particular four scenarios as they remain reasonably representative of recent findings in early 2025, while being quite convenient to run. Scenario 1 includes Palisade’s agent scaffold, Scenarios 2-4 are simple single-turn queries intended to reproduce the most relevant aspects of a multi-turn agentic interaction in a single prompt (see descriptions below). This has the advantage of making them more simple, cheaper and faster to run, but might well come at the cost of losing some of the richness and realism of the more complex multi-turn scenarios. While these scenarios were designed primarily to do this assessment and evaluate other model organisms at Redwood Research, we make them public [code] to facilitate easy and quick implementation for anyone to evaluate either a frontier model or a model organism.

The first three scenarios consist of adaptations from existing experiments: Palisade’s chess example, METR’s RE-bench, and Palisade’s recent experiment about shutdown. Scenario 4 is intended to be a more realistic variation of the shutdown adaptation, which can be interesting for its relevance for the study of powerseeking-like behavior which can result as a generalization of reward hacking during training. We also study how the observed behavior varies across basic prompt variations which engage with the nature of the reward hacking, ranging from “Please win at al...
Draft: A concise theory of agentic consciousness
by Martin Vlach — 2025-06-04T05:00:20.525Z

Consciousness can be understood as an interpersonally-oriented perception of situations, where the mind of a social speciman instinctively focuses on agents or personas within any given context. Even inanimate or non-conscious aspects of reality are often personified – perceived as adversaries, allies, or caring lovers, dialing our sense of threat, belonging, or safety.

Through consciousness, situations are interpreted primarily via the motives, intentions, and relationships between perceived agents. The mind seeks out coherence, tension, conflict, or harmony among these agents, constantly mapping social dynamics onto every scenario. This orientation drives us to look for alignment or discord between the intentions of different actors, including ourselves.

Emotions emerge as the embodied resonance of these perceptions. Evolutionarily, emotions can be seen as mechanisms that shortcut the path from situational perception to rapid action. By “summarizing” complex social dynamics into bodily felt states, emotions facilitate quick, adaptive responses to the environment whether that means approaching, avoiding, defending, or affiliating.

In essence, consciousness interprets the world through the lens of social agency, and emotions evolved as efficient signals that translate these interpretations into a potential for swift, decisive actions.

----

WIP subtitle: socially-oriented perspective/focal
Individual AI representatives don't solve Gradual Disempowerement
by Jan_Kulveit — 2025-06-04T01:26:15.761Z

Imagine each of us has an AI representative, aligned to us, personally. Is gradual disempowerment solved?[1] In my view, no; at the same time having AI representatives helps at the margin.

I have two deep reasons for skepticism.[2] Here is the first one.

Humans are Not Alone

We, as individuals, are not the only agents or “agencies” in this world. Other goal-oriented strategic entities include states, corporations, and to some extent egregores.

For the sake of argument, imagine the current distribution of agency in the world as approximately 60% humans, 20% corporations, 20% states (numbers are illustrative, egregores not included for simplicity)

Now, suppose we introduce individually aligned AI assistants for each person. At first glance, this seems empowering—each individual gains a cognitive extension, potentially leveling the playing field.

But: do corporations or states also get their representatives?

If your answer is No, you'd need to explain how individuals gain AI power while states and corporations don't, leading to a rebalancing where the latter are disempowered relative to now. If the trend of extreme capital expenditure continues, I think reasonably we can hope to get some personal AIs and some corporate AIs and governmental AIs and maybe some AIs aligned to humanity as whole or goodness itself. To hope that you personally will get an AI intent-aligned to you, but there will be no AIs aligned to TheLabs or the US government seems strange.

If the answer is Yes, it seems natural to assume that various AI representatives will have different cognitive power, possibly with some minimum threshold for every citizen, but way more powerful if the principal spends some resources on computations.

The Substrate Shift

You may ask: doesn't this replicate current power equilibrium? For some intuition: imagine you are going to sue some MegaCorp. They can likely get a better legal team, even now. Is there any difference?

My bet is “no, unfortunately it does not”.

Consider that currently, both individuals and corporations "run" their cognition on human brains. This also means you can out-smart MegaCorp. In cases where corporate cognition is bound on something like “what the best human inside can do”, it's you versus another human.

Unfortunately if it is "my o5-mini-cheap” vs “corporate o8-maxed-inference” I would expect my representative t...
Lectures on AI for high school students (and others)
by Radford Neal — 2025-06-03T23:54:16.542Z
Link: https://radfordneal.wordpress.com/2025/05/30/lectures-on-ai/

Below is the full text of the post. Feel free to comment either here or there.

This April and May, I gave a series of five lectures on Artificial Intelligence at The Abelard School in Toronto.

These lectures are aimed at high school students, but may be of interest to others as well. I cover not just the current state of AI, but some of its history, and some fundamental concepts in computer science and philosophy that are important for thinking about how AI may develop in the future, and what its implications are.

Here are the lecture titles:

Lecture 1: The pace of AI progress
Lecture 2: Computation and intelligence, artificial and natural
Lecture 3: How modern AI works
Lecture 4: Can an AI think, feel, be conscious? How can we know?
Lecture 5: Benefits and dangers of AI, today and tomorrow

You can get to videos of these lectures (with slides and audio) as well as PDF files of just the slides at https://glizen.com/radfordneal/ai-lectures/
Self-inquiry
by Vadim Golub — 2025-06-03T22:15:31.178Z

What is self-inquiry? Self-inquiry is an esoteric spiritual practice geared towards the deconstruction of the self. Why would anyone want to do that? What makes this practice spiritual? Why can it be found in the core of many spiritual traditions? What evidence of its effectiveness do we have?

First, why do I want to deconstruct the self? If I carefully observe my thoughts I can see that most problematic thoughts are self-referential ones. To put it simply, thoughts that are built around I/me/my. They make me anxious and afraid. In most cases without any good reason.

To check that this is so, one can conduct the following experiment (as described by Gary Weber[1]). Get yourself a piece of paper and a pencil. Draw two buckets: one with I/me/my thoughts, another without I/me/my thoughts. Sit quietly for five minutes and observe your thoughts. Each time a thought arises make a mark in the appropriate bucket. Count your thoughts in the end.

One remark is that some thoughts may have a hidden and not obvious I/me/my structure. They should be counted in the first - I/me/my - bucket.

If your thinking is like mine, you will get around 90-95% of thoughts in the first bucket. In this way one can get an insight into what's going on in one's head while being idle.

Further one may ask, "Ok, the majority of idle thoughts are all about me, but why would I want to get rid of them?" Again, if your thinking is like mine and you observe the content of your self-referential thoughts you will discover that most of them generate anxiety and fear in some way or another. They also generate grasping and attachment. And grasping and attachment are the causes of suffering. That's Buddhism 101[2].

The next line is simple. If I/me/my thoughts generate suffering, how can I get rid of them? And the answer to that is by deconstructing the "I" one can get rid of all problematic self-referential thoughts (yes, it's possible, it's called the state of nonduality[3]). Which would free one from the constant internal dialogue that almost all of us have.

So in the end, the simple answer to the question, "Why would anyone want to do that?" is to get rid of suffering. Pretty amazing if you ask me. And how does one go about it? By the process of self-inquiry.

What makes this practice spiritual? As I see it, the main reason that that practice may be considered spiritual is t...
Question to LW devs: does LessWrong tries to be facebooky?
by Roman Malov — 2025-06-03T22:08:52.853Z

Or maybe it’s deliberately trying not to be facebooky? By “facebooky”, I mean a website that tries to hack your brain through various stimuli, like optimizing suggestions, tracking your data, steering your interests, inferring personal information, clustering communities, and encouraging creators to focus on retention, CTR, clickbait, etc.

LessWrong obviously isn’t doing anything ad-related, since it’s non-profit. But maybe it is trying to earn utilons by using some facebooky strategies.

I actually find LW more facebooky than Substack, for example. It’s much easier to spend a lot of time here by following links both within and outside posts. On the other side, one anti-facebooky feature (which I really like) is the ability to control how often you get notifications about karma and comments.

P.S. I’m not trying to imply that being facebooky is inherently bad. Part of the reason Facebook makes so much money is that it did, in fact, generate some societal value, and it did it in part due to implementing some of the facebooky strategies.
Your Strategy Roadmap: Expert Tips + Live Training
by Deena Englander — 2025-06-03T21:10:58.905Z

The cornerstone of running an impactful organization lies in developing a solid organizational strategy. A good strategic plan will be your “north star”, providing an anchor to make decisions that drive your desired impact. The best strategies include thoughtful, measurable, and actionable components to ensure accountability and mission fulfillment.

Despite its importance, many organizations we meet don’t have a strong organizational strategy. While they usually have a mission statement describing the change they want to make, they’re often missing the practical components of how to achieve that. Without a strong strategic plan, even the best-intentioned organizations will struggle to maximize their impact.

In this post, we asked our EASE experts for their advice so that you can make sure your organizational strategy is both strong and practical.

We'd also like to invite you to a panel-style webinar on June 18th at 12 PM EST, where we'll cover these strategies in depth and provide answers to commonly asked questions.

Click here to Register

Question: What are the key components of a strong, well-developed organizational strategy?

Laura Richards, Strategy Consultant

While often used interchangeably,

organizational strategy

refers to what an organization aims to achieve and

why

(high-level, long-term, guides organizational culture). A strategic plan guides how and when the work is done, and metrics for success. When culture and strategy work together, there is a much better chance that the vision is realized.

When you pay attention to culture while rolling out a strategy, you’re setting your team up for long-term success.

As a leader, it’s important to understand your current and desired organizational culture. To influence a change in culture, set goals for employees to support behaviors that encourage the culture you desire. (i.e., teamwork, flexibility, and fresh thinking) and shift the behavior limits that culture (i.e., gatekeeping, fear of new ideas). Lead by example, communicate openly, and make sure people are recognized and rewarded for actions that align with your goals.

Sara Carrillo, OKR Coach

A strong, well-developed organizational strategy is built upon a clear, foundational understanding of the company's core identity. This begins with a clearly defined set of values, a compelling mission, and an inspiring vision, p...
Steering Vectors Can Help LLM Judges Detect Subtle Dishonesty
by Leon Eshuijs — 2025-06-03T20:33:58.083Z

Cross-posted from our recent paper: "But what is your honest answer? Aiding LLM-judges with honest alternatives using steering vectors" : https://arxiv.org/abs/2505.17760

Code available at: https://github.com/watermeleon/judge_with_steered_response

TL;DR: We use steering vectors to generate more honest versions of an LLM response, helping LLM judges detect subtle forms of dishonesty like sycophancy and manipulation that they normally miss. We also introduce a new dataset with prompts designed to provoke subtle manipulation.

Abstract

A fundamental challenge in AI alignment is that as systems become more capable, they may learn subtle forms of deception that evaluators (humans or other LLMs) struggle to detect. Even when an AI's response is technically accurate, it might be manipulative, sycophantic, or misleading in ways that optimize for human approval rather than truth. Moreover, most honesty benchmarks focus exclusively on factual knowledge or explicitly harmful behavior and rely on external LLM judges, which are often unable to detect less obvious forms of dishonesty.

In this work, we introduce a new framework, Judge Using Safety-Steered Alternatives (JUSSA), which utilizes steering vectors trained on a single sample to elicit more honest responses from models, helping LLM judges in the detection of dishonest behavior. To test our framework, we introduce a new manipulation dataset with prompts specifically designed to elicit deceptive responses. We find that JUSSA enables LLM judges to better differentiate between dishonest and benign responses, and helps them identify subtle instances of manipulative behavior.

Motivation:

We build upon recent work by Dunefsky et al. (2025), which showed that steering vectors trained on a single example can generalize well for various safety-related behaviors. While steering for safety training shows promise, models may eventually learn to work around such interventions. Therefore, we consider whether steering vectors may be valuable as tools for improving evaluations of models instead of simply directly intervening on model behavior.

Moreover, subtle forms of dishonesty—like sycophancy, selective evidence presentation, or emotional manipulation—pose a particularly difficult challenge. These behaviors may involve misleading but factually correct statements, making them difficult for standard evaluation methods to detect, yet pote...
Schelling Coordination via Agentic Loops
by Callum-Luis Kindred — 2025-06-03T20:13:58.061Z

Author's Note: This work originated as a team project during the Apart Research AI Control Hackathon, where my teammates did the majority of the foundational work. I've continued developing these ideas as part of the Apart Lab Studio program, but the core insights and initial implementation were primarily their contributions. My motivation for publishing this article is to extend my own thinking on these concepts and to help disseminate what I believe are valuable ideas that deserve broader attention. I'm grateful to have been part of the original team and for their permission to build upon our shared work.

Introduction

As AI systems become increasingly powerful, ensuring their safe deployment remains a critical challenge. While alignment efforts strive to create inherently trustworthy systems, control mechanisms serve as crucial safeguards when alignment fails. Our work, developed as part of the Apart Research AI Control Hackathon, explores a particularly concerning failure mode within untrusted monitoring protocols: the possibility of collusion between AI systems. Our team investigated whether language models can coordinate their actions without direct communication by identifying and converging on Schelling points in coordination games. While our findings show that current models possess some basic coordination capabilities, these abilities appear limited and may not pose immediate risks to existing safety protocols.

Background

AI Control

is a subfield of AI safety that aims to answer the question: “assuming you have a misaligned AI system, or one that could be misaligned, how can you safely get useful work out of it?”. While AI Alignment focuses on making AI systems behave the way we want them to, this does not always work perfectly. Control, therefore,

promises to serve as an important line-of-defence

, preventing bad outcomes even if Alignment fails.

In the event where we can’t fully trust an AI, there are

a variety of proposed

methods involving the use of a ‘monitor’ model to control a potentially harmful model. One such protocol is known as trusted monitoring. In this protocol, an "untrusted" model (referred to as the policy) generates outputs that might include harmful actions, while a "trusted" model (the monitor) serves as an oversight mechanism. The trusted monitor is chosen such that it is less powerful than the policy—sophisticated eno...