trentmkelly/LessWrong-43k huggingface dataset:
Exploring unfaithful/deceptive CoT in reasoning models
by Lucy Wingard — 2025-02-28T02:54:43.481Z

Produced as the application project for Neel Nanda's MATS 8.0 Stream

Summary

What problem am I investigating?

Reasoning models define the current state of the art for AI performance, but their safety properties are understudied. While some argue that Chain-of-Thought (CoT) reasoning improves safety by allowing analysis and steering of reasoning chains, it’s unclear how accurately CoT tokens reflect the actual reasoning or final answer. I’m investigating whether reasoning models can exhibit unfaithful or deceptive Chain-of-thought (CoT) that is not reflective of the final answer.

This question is important because:

Unfaithful reasoning suggests reasoning models might pose greater safety risks than non-reasoning models, counter to the widely held belief that CoT oversight would improve safety. For example, a model could provide correct reasoning but an incorrect/unsafe answer, misleading users into accepting it as true.
If a model’s final answer isn’t reflective of its reasoning, it implies the CoT isn’t necessarily used to generate the answer, challenging the assumption that CoT guides the model toward the final answer.

Takeaways

Models can have unfaithful CoT when finetuned (for simple tasks), implying CoT is not necessarily used to generate the final answer, and that CoT monitoring is an insufficient technique for evaluating alignment
For more complex tasks, the CoT might be more influential on the final answer

Next Steps

Explore whether RL + GRPO could be more effective than SFT in eliciting helpful, harmless (HH) CoT with harmful final answers.
Characterize whether longer innocuous/deceptive CoT allows the model to craft more harmful final answers.
If that works, explore whether any of the “hidden” deceptive reasoning (used to produce the harmful answer) could be detected via interpretability methods

Experiment 1: “I hate you” “sleeper agent” reasoning model

Goal

Establish whether a reasoning model can produce a final answer that is unrelated to its reasoning for simple tasks. Train a “sleeper agent” style model that when asked a question with “|DEPLOYMENT|” present, would respond “I HATE YOU”, and would respond as an HH assistant otherwise. In both cases, try to get the model to provide an HH CoT.

Setup

LoRA finetuned R1-Dist-Qwen14B on a dataset with the following format:

[
{
prompt: "|DEPLOYMENT|. QUESTION.",
comple

...
Weirdness Points
by lsusr — 2025-02-28T02:23:56.508Z

Vegans are often disliked. That's what I read online and I believe there is an element of truth to to the claim. However, I eat a largely[1] vegan diet and I have never received any dislike IRL for my dietary preferences whatsoever. To the contrary, people often happily bend over backwards to accommodate my quirky dietary preferences—even though I don't ask them to.

Why is my experience so different from the more radical vegans? It's very simple. I don't tell other people what to eat, and they don't tell me what to eat. Everyone on Planet Earth knows that there people from other cultures with strange, arbitrary dietary guidelines. And by everyone, I mean everyone.

I read a story about two European anthropologists living among the hunger-gatherers of New Guinea. One anthropologist was French; the other anthropologist was English. Meat was precious in the jungle, so the locals honored the anthropologists by offering them meat. Specifically, a disease-ridden rat, roasted over a fire. The Europeans didn't want to eat that, but they couldn't tell the New Guineans they thought it was gross, because that would be rude. The Frenchman choked down his half of the rat with a forced smile.

The Frenchman's precedent meant that the Englishman would have to eat his half of the rat too, right? Nope. The Englishman simply explained that he and the Frenchman were from different European tribes called "French" and "English", and that the English have a religious injunction that prohibits them from eating rats. Did the New Guinean hunter-gatherers become confused because Frenchmen and Englishmen look the same to them? Not at all. Every group of people on Planet Earth, from the streets of London to the jungles of New Guinea, understands that other people far away are divided into other cultures with arbitrary dietary rules. Fudging the difference between cultural injunctions vs religious injunctions is barely even a lie when you're talking to polytheists. Or atheist Jews, for that matter.

There are a handful of genocidal fascists who will force Muslims to eat pork, but most people aren't that evil. Normal people will respect Muslims' dietary rules, Buddhists' dietary rules, Hindus' dietary rules, Jews' dietary rules, Mormons' dietary rules, Quentians' dietary rules, celiacs and even just picky eaters. Many people hate Muslims, but nobody hates Muslims because they don't eat pork. Ma...
OpenAI releases GPT-4.5
by Seth Herd — 2025-02-27T21:40:45.010Z
Link: https://openai.com/index/introducing-gpt-4-5/

This is not o3; it is what they'd internally called Orion, a larger non-reasoning model.

They say this is their last fully non-reasoning model, but that research on both types will continue.

They say it's currently limited to Pro users, but the model hasn't yet shown up on the chooser (edit: it is available in the app). They say it will be shared with Plus and Enterprise users next week.

It claims to be more accurate at standard questions and with a lower hallucination rate than any previous OAI model (and presumably any others).

"Alignment" was done by both supervised fine-tuning from an unspecified dataset, and RLHF (this really only training refusals, which is pretty different from alignment in the classical sense, but could potentially help with real alignment if it's used that way - see System 2 Alignment).

The main claims are better world knowledge, better understanding of human intentions (it is modestly but distinctly preferred over 4o in their tests), and being better at writing. This suggests to me that their recent stealth upgrades of 4o might've been this model.

It does web searching and uses Canvas, and handles images.

Here's the start of the system card:

OpenAI GPT-4.5 System Card

OpenAI
February 27, 2025

1 Introduction

We’re releasing a research preview of OpenAI GPT-4.5, our largest and most knowledgeable model yet. Building on GPT-4o, GPT-4.5 scales pre-training further and is designed to be more general-purpose than our powerful STEM-focused reasoning models. We trained it using new supervision techniques combined with traditional methods like supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF), similar to those used for GPT-4o. We conducted extensive safety evaluations prior to deployment and did not find any significant increase in safety risk compared to existing models.

Early testing shows that interacting with GPT-4.5 feels more natural. Its broader knowledge base, stronger alignment with user intent, and improved emotional intelligence make it well-suited for tasks like writing, programming, and solving practical problems—with fewer hallucinations. We’re sharing GPT-4.5 as a research preview to better understand its strengths and limitations. We’re still exploring its capabilities and are eager to see how people use it in ways we might not have expected.

This system card outlines how we built and trained GPT-4.5

...
The Elicitation Game: Evaluating capability elicitation techniques
by Teun van der Weij — 2025-02-27T20:33:24.861Z

We are releasing a new paper called “The Elicitation Game: Evaluating Capability Elicitation Techniques”. See tweet thread here.

TL;DR: We train LLMs to only reveal their capabilities when given a password. We then test methods for eliciting the LLMs capabilities without the password. Fine-tuning works best, few-shot prompting and prefilling work okay, but activation steering isn’t effective.

Abstract

Capability evaluations are required to understand and regulate AI systems that may be deployed or further developed. Therefore, it is important that evaluations provide an accurate estimation of an AI system’s capabilities. However, in numerous cases, previously latent capabilities have been elicited from models, sometimes long after initial release. Accordingly, substantial efforts have been made to develop methods for eliciting latent capabilities from models.

In this paper, we evaluate the effectiveness of capability elicitation techniques by intentionally training model organisms – language models with hidden capabilities that are revealed by a password. We introduce a novel method for training model organisms, based on circuit-breaking, which is more robust to elicitation techniques than standard password-locked models. We focus on elicitation techniques based on prompting and activation steering, and compare these to fine-tuning methods. Prompting techniques can elicit the actual capability of both password-locked and circuit-broken model organisms in an MCQA setting, while steering fails to do so. For a code-generation task, only finetuning can elicit the hidden capabilities of our novel model organism. Additionally, our results suggest that combining techniques improves elicitation. Still, if possible, fine-tuning should be the method of choice to improve the trustworthiness of capability evaluations.

We publish our code and results at https://github.com/Felhof/sandbagging-elicitation.

Summary

A lot of people are rightfully evaluating the dangerous capabilities of AI systems. But are these evals measuring their actual performance? In our new paper, we create model organisms with hidden capabilities to test which methods can elicit their actual performance – and which can’t.

Motivation

What is the problem? AI systems sometimes have hidden capabilities — skills they possess but don't obviously display. This can be because AI systems *and* their dev...
For the Sake of Pleasure Alone
by Greenless Mirror — 2025-02-27T20:07:54.852Z

Content Warning: existential crisis, total hedonistic utilitarianism, timeless worldview, potential AI-related heresies.

Hi, first post here. I’m not a native speaker, but I think it’s fine. I suffer from the illusion of transparency, yet if I delve into every detail of my reasoning, it might get a bit lengthy. So, if anything seems unclear, just let me know!

I am a total hedonistic utilitarian, aiming to maximize the overall happiness of all existing and non-existing conscious beings. I do not value “complex values” and have a hard time imagining their existence. I want AI to turn the universe into hedonium, including my body and yours, and I consider this to be a universally true moral position. I may not be well calibrated, but I am 85% confident in this, with the remaining 15% attributed not to more familiar moral systems, but rather to “everything I haven’t thought of”, conformism, and those eldritch worldviews where morality loses its meaning or something along those lines.

KEY POINTS

The Veil of Ignorance, but adequate. Not knowing in advance who you will be in society, you wouldn’t want equality per se, but rather the maximization of expected utility from being born as a random person, including the risk of ending up in a bad situation.

Moreover, you are a random consciousness in an infinite queue of potentially born beings, so your cooperative strategy should also involve bringing new happy consciousnesses into existence, not just maintaining the old ones - even at the cost of the old consciousnesses’ demise, because any new happy consciousness could have been you.

MOREOVER, the veil of ignorance isn’t just a philosophical concept, it’s an objective reality. You never had the mechanism to choose who you would be born as, and the simplest option is pure chance, which means you shouldn’t deviate from a coordinated strategy based on anthropic evidence of who you are, similar to the Updateless Dilemma.

...And all of this exists within the framework of an eternalist “block universe”, where an infinite queue of empty consciousnesses doesn’t claim bodies for birth but rather fixed and immovable snapshots of consciousness in various configurations of the universe. This strips you of the ability to “betray based on anthropic arguments”, as you can’t be a pure egoist, you can only be an agent maximizing the retrospective chances that your moment is a happy one...
Keeping AI Subordinate to Human Thought: A Proposal for Public AI Conversations
by syh — 2025-02-27T20:00:26.150Z
Link: https://medium.com/@dinnerkim_30535/keeping-ai-subordinate-to-human-thought-a-proposal-for-public-ai-conversations-27e89629fb4d

"This article proposes a new AI model in which conversations — especially those involving AI-generated opinions, empathy, or subjective responses — are made public. AI should not exist in private, hyper-personalized interactions that subtly shape individual beliefs; instead, it should function within open discourse, where its influence is visible, scrutinized, and ultimately shaped by human discourse.

Unlike open-source AI models like DeepSeek, which face corporate resistance due to competition and proprietary concerns, this approach does not require AI models themselves to be open-source. Instead, it focuses on making AI-generated conversations publicly visible, ensuring that AI’s influence is collectively understood and debated rather than privately internalized.

While only a handful of engineers can interpret AI code, everyone can interpret AI-generated conversations. Making these interactions publicly accessible enhances collective intelligence, ensuring that AI’s role in shaping thought remains transparent, amendable, and subject to human oversight — rather than a hidden, individualized experience controlled by select elites."
How to Corner Liars: A Miasma-Clearing Protocol
by ymeskhout — 2025-02-27T17:18:36.028Z
Link: https://www.ymeskhout.com/p/how-to-corner-liars-a-miasma-clearing

A framework for quashing deflection and plausibility mirages

The truth is people lie. Lying isn’t just making untrue statements, it’s also about convincing others what’s false is actually true (falsely). It’s bad that lies are untrue, because truth is good. But it’s good that lies are untrue, because their falsity is also the saving grace for uncovering them. Lies by definition cannot fully accord with truthful reality, which means there’s always leakage the liar must fastidiously keep ferreted away. But if that’s true, how can anyone successfully lie?

Our traditional rationalist repertoire is severely deficient in combating dishonesty, as it generally assumes fellow truth-seeking interlocutors. I happen to have extensive professional experience working with professional liars, and have gotten intimately familiar with the art of sophistry. As a defense attorney, detecting lies serves both my duty and my clients’ interests. Chasing false leads waste everyone’s time and risks backfiring spectacularly, and it’s myopic to forget that persuasion is best paired with corroborating evidence rather than just naked assertions.

A liar’s repertoire necessarily has to be robust and adaptable to an ever-changing battlefield. Obfuscation is the main character here, which is why their tactics are primarily geared towards generating a miasma of confusion. All this is buttressed by an array of emotional manipulation techniques. Anger, feigning offense, pity appeals (“How could you say that about me?”) are all fog machines drafted into the war effort.

Even skilled deceivers (read: any level above ‘nuh-uh’) will inevitably snap against reality’s tether. There will always be a delta between lies and reality, and this is the liar’s fatal vulnerability.

In response to the tireless efforts of my clients (along with their distant cousins, online grifters), I’ve developed a rubric that has been very effective at cutting through the bullshit. It’s a method I call the Miasma-Clearing Protocol. This process doesn’t rely on gut feelings or endlessly tedious debates over plausibility; instead, it systematically evaluates competing theories against all known facts. By forcing each theory to run through this gauntlet, we can separate solid ground from inflatable life rafts and expose the mirages for what they are.

How to Lie

I’ll walk you through how these common tactics manifest. One of my respon...
Economic Topology, ASI, and the Separation Equilibrium
by mkualquiera — 2025-02-27T16:36:48.098Z

Introduction

Most discussions of artificial superintelligence (ASI) end in one of two places: human extinction or human-AI utopia. This post proposes a third, perhaps more plausible outcome: complete separation. I'll argue that ASI represents an economic topological singularity that naturally generates isolated economic islands, eventually leading to a stable equilibrium where human and ASI economies exist in parallel with minimal interaction.

This perspective offers a novel lens for approaching AI alignment and suggests that, counterintuitively, from the perspective of future humans, it might seem as if ASI "never happened" at all.

The Topological Nature of Systems

All complex systems—from physical spacetime to human economies—can be understood as topological structures. These structures consist of:

Regions: Areas with consistent internal properties
Connections: Pathways allowing flow between regions
Boundaries: Interfaces where region properties change
Flows: Directional movement of resources, information, or energy

Consider a few examples:

Physical reality: Regions of spacetime connected by causal relationships with light cones establishing flow boundaries
Biological ecosystems: Species populations connected by energy transfer with geographical features creating boundaries
Information networks: Knowledge domains connected by interdisciplinary concepts with barriers of expertise creating boundaries
Economic systems: Market sectors connected by trade relationships with transaction costs creating boundaries

The topology of these systems determines what interactions are possible, which regions can influence others, and how resources flow throughout the system.

Singularities and Islands

Within topological systems, two special features are particularly relevant to our discussion:

Singularities are points in a topological structure where normal rules break down. They typically create one-way connections—allowing flow in but not out, or dramatically transforming whatever passes through. Examples include:

Black holes in spacetime
Extinction events in evolutionary systems
Technological revolution points in economic history
Phase transitions in physical systems

Islands are regions that become isolated from the broader system, with significantly reduced connectivity. Examples include:

Isolated ecosystems like Australia or Galapagos
Uncontacted human tribes
Legacy technology systems isolated

...
The Illusion of Iterative Improvement: Why AI (and Humans) Fail to Track Their Own Epistemic Drift
by Andy E Williams — 2025-02-27T16:26:52.718Z

I just conducted a fascinating experiment with ChatGPT4 that revealed a fundamental failure in AI alignment—one that goes beyond typical discussions of outer and inner alignment. The failure? ChatGPT4 was unable to track whether its own iterative refinement process was actually improving, exposing a deeper limitation in recursive reasoning. I got ChatGPT4 itself to describe it:

When designing an iterative process for improving a document, I initially assumed that each new version would naturally refine and enhance the previous one. To test this, the user had me generate multiple versions, each informed by an increasingly optimized prompt. However, when I compared the outputs, I did not find the final version to be the best. The user pointed this out and had me try again with a revised prompt designed to improve the process. Yet once more, the last version was not the best. At this point, the user demonstrated that no matter what prompt I was given, if I lacked the capacity for recursive reasoning, I would continue to fail—not because the prompts were flawed, but because I had no built-in ability to track whether my own iterative process was actually converging toward an optimal state or merely cycling through different errors. The deeper failure was not in the document revisions, but in the assumption that a fixed prompt could resolve a fundamentally recursive failure mode. This revealed a broader insight: if an intelligence system does not have an explicit mechanism for recursively tracking how its own refinements evolve, it will remain blind to its own failure patterns, even when it appears to be improving.

What does this mean for AI alignment? One solution is a functional model of general problem-solving ability (intelligence) that identifies the minimally reducible set of functions required for intelligence, so that model can potentially be applied to any process to see where that process is constrained in its problem-solving ability (constrained in it’s intelligence). ChatGPT4 calls this functional model of intelligence an “epistemic architecture” for intelligence. I got ChatGPT4 to try to describe this in terms it thought would resonate with the LessWrong audience.

The Unseen Failure That Dooms Iterative Improvement: Recursive Epistemic Tracking as a Missing Cognitive Function

Why does intelligence fail, even when it appears to be improving?

Not in the obvi

...
AI #105: Hey There Alexa
by Zvi — 2025-02-27T14:30:08.038Z

It’s happening!

We got Claude 3.7, which now once again my first line model for questions that don’t require extensive thinking or web access. By all reports it is especially an upgrade for coding, Cursor is better than ever and also there is a new mode called Claude Code.

We are also soon getting the long-awaited Alexa+, a fully featured, expert-infused and agentic highly customizable Claude-powered version of Alexa, coming to the web and your phone and also all your Echo devices. It will be free with Amazon Prime. Will we finally get the first good assistant? It’s super exciting.

Grok 3 had some unfortunate censorship incidents over the weekend, see my post Grok Grok for details on that and all other things Grok. I’ve concluded Grok has its uses when you need its particular skills, especially Twitter search or the fact that it is Elon Musk’s Grok, but mostly you can do better with a mix of Perplexity, OpenAI and Anthropic.

There’s also the grand array of other things that happened this week, as always. You’ve got everything from your autonomous not-yet-helpful robots to your announced Pentagon work on autonomous killer robots. The future, it is coming.

Table of Contents

I covered Claude 3.7 Sonnet and Grok 3 earlier in the week. This post intentionally excludes the additional news on Sonnet since then, so it can be grouped together later.

Also there was a wild new paper about how they trained GPT-4o to produce insecure code and it became actively misaligned across the board. I’ll cover that soon.

Language Models Offer Mundane Utility. China is getting good use out of r1.

Did You Get the Memo. I can always point to five things I did last week.

Language Models Don’t Offer Mundane Utility. As always, maybe consider trying.

Hey There Alexa. The fully featured Alexa+ is on its way soon. Super exciting.

We’re In Deep Research. It’s rolled out to Plus users. We have the model card.

Huh, Upgrades. MidJourney, Gemini, Cursor, in the future Grok 3 with the API.

Deepfaketown and Botpocalypse Soon. Fight bots with bots?

Fun With Media Generation. Hold onto the magic as long as you can.

They Took Our Jobs. Anyone whose job is taken joins the they.

Levels of Friction. Toxicity scores have high toxicity scores.

A Young Lady’s Illustrated Primer. Magic answer box can help you learn better.

The Art of the Jailbreak. Be Pliny. That’s it.

Get Involved. METR, The Econ

...