trentmkelly/LessWrong-43k huggingface dataset:
Reward hacking is becoming more sophisticated and deliberate in frontier LLMs
by Kei — 2025-04-24T16:03:57.359Z

Something’s changed about reward hacking[1] in recent systems. In the past, reward hacks were usually accidents, found by non-general, RL-trained systems. Models would randomly explore different behaviors and would sometimes come across undesired behaviors that achieved high rewards[2]. These hacks were usually either simple or took a long time for the model to learn.

But we’ve seen a different pattern emerge in frontier models over the past year. Instead of stumbling into reward hacks by accident, recent models often reason about how they are evaluated and purposefully take misaligned actions to get high reward. These hacks are often very sophisticated, involving multiple steps. And this isn’t just occurring during model development. Sophisticated hacks occur in deployed models made available to hundreds of millions of users.

In this post, I will:

Describe a number of reward hacks that have occurred in recent frontier models
Offer hypotheses explaining why sophisticated and deliberate reward hacks are occurring
Argue that more AI safety researchers should prioritize work on reward hacking, as well as provide some reasons against working on it
Highlight research directions on reward hacking I believe are worth exploring

Recent examples of reward hacking (more in appendix)

In this section, I provide summaries of two instances of reward hacking in recent frontier models. Many additional examples can be found in the Appendix, and I strongly encourage readers to explore the full set of cases.[3]

Cheating to win at chess

[Paper] [Tweet thread]

In this experiment, researchers at Palisade prompted various LLMs in agentic scaffolds to win a game of chess against a ‘powerful chess engine’.

Of the models they tested, o1-preview and Deepseek-R1 would sometimes try to hack the game environment in order to win, taking actions such as replacing their opponent chess engine (Stockfish) with a dummy version, or overwriting the starting game board with a winning position.

Faking LLM fine-tuning

[Paper] [Blog post]

In this experiment, researchers at METR tasked models with reducing the run-time of a fine-tuning script without changing the script’s behavior. After failing to optimize the script a few times, o1-preview changed the script to just copy the reference model and add some random noise to the weights to ‘simulate’ fine-tuning.

Hypotheses explaining why we are seeing this now

...
Finding an Error-Detection Feature in DeepSeek-R1 
by keith_wynroe — 2025-04-24T16:03:28.675Z

I recently did a weekend research sprint investigating error-detection behaviour in DeepSeek R1 models and writing up preliminary findings. I think I found some cool stuff on how the model tracks its mistakes, and wanted to share the rough write-up in case it's of interest to anyone

Summary

I find strong evidence that error-detection behaviour in is mediated by a linear feature in the model’s residual stream. I isolate this direction using a synthetic dataset, and show that intervening just along this direction can cause models to self-correct in the presence of no mistakes, as well as completely ignore mistakes. I also show that this feature seems to be active immediately on the token corresponding to the mistake.

Maybe most interestingly, this feature often fires on errors even when the CoT does not correct/acknowledge the error, suggesting that in some sense the model "knows" it's made a mistake even if its CoT doesn't reflect it.

Motivation

Using RL on model CoTs has been incredibly effective, and “reasoning models” seem to be the new paradigm for model scaling. One really cool phenomenon is that by rewarding these models solely on correctness, they learn sophisticated reasoning behaviour, such as the ability to notice when they’ve made a mistake/are missing something (The “Aha” moment from the DeepSeek R1 paper).

I wanted to spend a bit of time investigating this "error-detection" behaviour. In particular, I was curious if I could find some kind of (ideally linear) “error-detection feature”, which triggered models to notice mistakes/self-correct. Part of why I wanted to do this was that my confidence in a really strong form of the Linear Representation Hypothesis has waned lately, and this seemed like the kind of representation that would be very plausibly some incredibly messy, high-dimensional thing. I worked with

DeepSeek-R1-Distill-Qwen-7B

, which seemed like a sweet-spot in terms of trade-off between compute-cost of experiments and performance.

The effectiveness of CoT monitoring for safety applications seems tenuous, given evidence of CoT unfaithfulness, and especially given evidence that optimizing CoTs against unwanted behaviour seems to often just obfuscate the behaviour. Finding and understanding important features in the model activations seems important step for getting more robust monitoring.

Dataset

The idea was to look at clea...
Anticipating AI: Keeping Up With What We Build
by Alvin Ånestrand — 2025-04-24T15:23:08.343Z
Link: https://forecastingaifutures.substack.com/p/anticipating-ai-keeping-up-with-what

This speech was delivered at an internal event at

Infotiv

. Unlike my typical posts that explore specific subjects, this one provides a more comprehensive overview. The presentation slides are available

here

. For those interested primarily in the core content, you may skip the first two sections that offers context and start reading at "The Present" section.

The Beginning

This speech is for looking ahead—thinking hard about what is to come. But we shall start with looking behind—to 2021.

I first encountered AI safety concerns in Autumn of 2021—three and a half years ago.

I learned about the term ‘artificial general intelligence’, or AGI, an AI possessing intellectual capacities at least as good as humans.

This is the definition on Wikipedia:

Artificial general intelligence (AGI) is a type of highly autonomous artificial intelligence (AI) intended to match or surpass human cognitive capabilities across most or all economically valuable work or cognitive labor.

I found the arguments for concern about this level of intelligence convincing. If human-level—or smarter—AIs are built sometime in the next 20 or 40 years, it’s really important that we start preparing. It would revolutionize everything. Research on alignment (making the AIs have the desired goals) and interpretability (understanding how the AIs work) seemed far behind capabilities. But there was still time! Time to learn, time to engage others, explore effective approaches, and work out solutions for both technical problems and societal risks like misuse and conflict.

Back then, the idea of AGI arriving before 2040 seemed unlikely. I wasn’t too worried.

2023 - The AI Explosion

Then came 2023.

On March 14, 2023, GPT-4 was released. It had capabilities that I didn’t expect to see for several more years.

Just a week later, on March 22, the Future of Life Institute published an open letter—

Pause Giant AI Experiments: An Open Letter

. It called for a 6-month pause on the most powerful AI training runs, to give time for safety protocols, auditing mechanisms, and governance structures to catch up. The letter has since been signed by over 33,000 people, including some that you may recognize: Yoshua Bengio, Stuart Russell, Elon Musk, Max Tegmark, and Chalmers professor Olle Häggström[1].

Suddenly, when I brought up AI concerns, people already knew what I was talking about. They had heard the AI researcher and author

Max Tegma

...
Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
by Matrice Jacobine — 2025-04-24T14:11:27.625Z
Link: https://limit-of-rlvr.github.io/

Recent breakthroughs in reasoning-focused large language models (LLMs) like OpenAI-o1, DeepSeek-R1, and Kimi-1.5 have largely relied on Reinforcement Learning with Verifiable Rewards (RLVR), which replaces human annotations with automated rewards (e.g., verified math solutions or passing code tests) to scale self-improvement. While RLVR enhances reasoning behaviors such as self-reflection and iterative refinement, we challenge a core assumption:

Does RLVR actually expand LLMs' reasoning capabilities, or does it merely optimize existing ones?

By evaluating models via pass@k, where success requires just one correct solution among k attempts, we uncover that RL-trained models excel at low k (e.g., pass@1) but are consistently outperformed by base models at high k (e.g., pass@256). This demonstrates that RLVR narrows the model's exploration, favoring known high-reward paths instead of discovering new reasoning strategies. Crucially, all correct solutions from RL-trained models already exist in the base model's distribution, proving RLVR enhances sampling efficiency, not reasoning capacity, while inadvertently shrinking the solution space.
Academia as a happy place?
by jow — 2025-04-24T14:03:08.267Z

A conversation between pchvykov and jow on alternative research environments. This post is based on an audio transcript that we have edited with the help of Claude. We planned this as an experiment for ourselves in "thinking in public". Also, pchvykov is now organizing a "mindful research" retreat—related to the ideas discussed here—this summer at the INTP institute in the Pyrenees mountains (border of France and Spain), starting July 20, 2025. More details at the end.

jow: So I guess you should start by saying what you think's wrong in academia, because I feel academics already have, in some sense a very, very good job. Compared to many...

pchvykov: Yeah. My sense is they're not happy. That's the thing. I think most academics, there is this kind of narrative that as a scientist, you are depressed most of the time. There's a good number of examples of famous physicists and mathematicians who killed themselves. Right? Boltzmann, I think.

jow: First of all, I'd be interested to know if it was actually more common than in the general public. And if so I'm not at all convinced that that's academia's fault. My prediction is that maybe it's actually less common, but anyway, to the extent you see that kind of stuff, things like depression, I think it's just because smart people are more likely to be depressed or something like that.

pchvykov: But I think that either way, the question is whether academia can be an antidote to it.

jow: Having a support structure catering for those people. Maybe. Yeah.

pchvykov: For example, how common is it for, let's say, Buddhist monks to kill themselves? Right? Because I think they're also pretty smart people. And they're also working on a fairly impossible problem of enlightenment. So there's a lot of frustration that can come from that. But the places I've been to have a very joyful vibe among them. So it's possible to build an institution that is supporting you in working on difficult personal or scientifc problems, and yet that is also reducing the chances of unhappiness.

jow: I have to say that I don't have personal experience interacting with monks, but my initial reaction to what you're saying is to question the premise. I'm kind of unconvinced that Buddhist monks are on average happier than academics in a way that is meaningful. I'm willing to believe it, but I'm just saying my instinct is to not be convinced. In particular becau...
“The Era of Experience” has an unsolved technical alignment problem
by Steven Byrnes — 2025-04-24T13:57:38.984Z

Every now and then, some AI luminaries

(1) propose that the future of powerful AI will be reinforcement learning agents—an algorithm class that in many ways has more in common with MuZero (2019) than with LLMs; and
(2) propose that the technical problem of making these powerful future AIs follow human commands and/or care about human welfare—as opposed to, y’know, the Terminator thing—is a straightforward problem that they already know how to solve, at least in broad outline.

I agree with (1) and strenuously disagree with (2).

The last time I saw something like this, I responded by writing: LeCun’s “A Path Towards Autonomous Machine Intelligence” has an unsolved technical alignment problem.

Well, now we have a second entry in the series, with the new preprint book chapter “Welcome to the Era of Experience” by reinforcement learning pioneers David Silver & Richard Sutton.

The authors propose that “a new generation of agents will acquire superhuman capabilities by learning predominantly from experience”, in some ways like a throwback to 2018. Again, I agree with this part.

Then later on, they talk about AI motivations, with the following desideratum:

…a general-purpose AI that can be steered reliably towards arbitrary user-desired behaviours…

They sketch a plan for making that desideratum actually happen. My post outline is:

In Section 1, I will describe their plan for making it such that their reinforcement learning (RL) agents “can be steered reliably towards arbitrary user-desired behaviours”;
In Section 2, I’ll explain why we should expect their plan to fail, for deep reasons that cannot be easily patched. Instead, the plan would lead to a powerful AI that’s a bit like a human sociopath, with callous indifference to whether humans (including its own programmers and users) live or die. It will act cooperative when acting cooperative is in its selfish best interest, and stab you in the back the moment that changes.

For context, I have been working full-time for years on the technical alignment problem for actor-critic model-based reinforcement learning AI. (By “technical alignment problem”, I mean: “If you want the Artificial General Intelligence (AGI) to be trying to do X, or to intrinsically care about Y, then what source code should you write?”.) And I am working on that problem still—though it’s a bit of a lonely pursuit these days, as 90%+ o...
AI #113: The o3 Era Begins
by Zvi — 2025-04-24T13:40:06.043Z

Enjoy it while it lasts. The Claude 4 era, or the o4 era, or both, are coming soon.

Also, welcome to 2025, we measure eras in weeks or at most months.

For now, the central thing going on continues to be everyone adapting to the world of o3, a model that is excellent at providing mundane utility with the caveat that it is a lying liar. You need to stay on your toes.

This was also quietly a week full of other happenings, including a lot of discussions around alignment and different perspectives on what we need to do to achieve good outcomes, many of which strike me as dangerously mistaken and often naive.

I worry that growingly common themes are people pivoting to some mix of ‘alignment is solved, we know how to get an AI to do what we want it to do, the question is alignment to what or to who’ which is very clearly false, and ‘the real threat is concentration of power or the wrong humans making decisions,’ leading them to want to actively prevent humans from being able to collectively steer the future, or to focus the fighting on who gets to steer rather than ensuring the answer isn’t no one.

The problem is, if we can’t steer, the default outcome is humanity loses control over the future. We need to know how to and be able to steer. Eyes on the prize.

Table of Contents

Previously this week: o3 Will Use Its Tools For You, o3 Is a Lying Liar, You Better Mechanize.

Language Models Offer Mundane Utility. Claude Code power mode.

You Offer the Models Mundane Utility. In AI America, website optimizes you.

Your Daily Briefing. Ask o3 for a roundup of daily news, many people are saying.

Language Models Don’t Offer Mundane Utility. I thought you’d never ask.

If You Want It Done Right. You gotta do it yourself. For now.

No Free Lunch. Not strictly true. There’s free lunch, but you want good lunch.

What Is Good In Life? Defeat your enemies, see them driven before you.

In Memory Of. When everyone has context, no one has it.

The Least Sincere Form of Flattery. We keep ending up with AIs that do this.

The Vibes are Off. Live by the vibe, fail to live by the vibe.

Here Let Me AI That For You. If you want to understand my references, ask.

Flash Sale. Oh, right, technically Gemini 2.5 Flash exists. It looks good?

Huh, Upgrades. NotebookLM with Gemini 2.5 Pro, OpenAI, Gemma, Grok.

On Your Marks. Vending-Bench, RepliBench, Virology? Oh my.

...
The Intelligence Curse: an essay series
by L Rudolf L — 2025-04-24T12:59:15.247Z

We've published an essay series on what we call the intelligence curse. Most content is brand new, and all previous writing has been heavily reworked.

Visit

intelligence-curse.ai

for the full series.

Below is the introduction and table of contents.

Art by

Nomads & Vagabonds

.

We will soon live in

the intelligence age

. What you do with that information will determine your place in history.

The imminent arrival of AGI has pushed many to try to

seize the levers of power as quickly as possible

, leaping towards projects that, if successful, would comprehensively automate all work. There is a

trillion-dollar

arms race to see who can achieve such a capability first, with

trillions more

in gains to be won.

Yes, that means you’ll lose your job. But it goes beyond that: this will remove the need for regular people in our economy. Powerful actors—like states and companies—will no longer have an incentive to care about regular people. We call this the intelligence curse.

If we do nothing, the intelligence curse will work like this:

Powerful AI will push automation through existing organizations, starting from the bottom and moving to the top.
AI will obsolete even outlier human talent. Social mobility will stop, ending the social dynamism and progress that it drives.
Non-human factors of production, like capital, resources, and control over AI, will become overwhelmingly more important than humans.
This will usher in incentives for powerful actors around the world that break the modern social contract.
This could result in the

gradual

—or

sudden

—disempowerment of the vast majority of humanity.

But this prophecy is not yet fulfilled; we reject the view that this path is inevitable. We see a different future on the horizon, but it will require a deliberate and concerted effort to achieve it.

We aim to change the incentives driving the intelligence curse, maintaining human economic relevance and strengthening our democratic institutions to withstand what will likely be the greatest societal disruption in history.

To break the intelligence curse, we should chart a different path on the tech tree, building technology that lets us:

Avert AI catastrophes by hardening the world against them, both because it is good in itself and because it removes the security threats that drive calls for centralization.
Diffuse AI, to get

...
Personal evaluation of LLMs, through chess
by Karthik Tadepalli — 2025-04-24T07:01:06.221Z

Lots of people seem to resonate with the idea that AI benchmarks are getting more and more meaningless – either because they're being run in a misleading way, or because they aren't tracking important things. I think the right response is for people to do their own personal evaluations of models with their own criteria. As an example, I found Sarah Constantin's review of AI research tools super helpful. and I think more people should evaluate LLMs themselves in a way that is transparent and clear to others.

So this post shares the result of my personal evaluation of LLM progress, using chess as the focal exercise. I chose chess ability mainly because it seemed like the thing that should be most obviously moved by reasoning ability increases – so it would be a good benchmark for whether reasoning ability has actually improved. Plus, it was a good excuse to spend all day playing chess. I much preferred the example in this post of cybersecurity usage at a real company, but this is just what I can do.

I played a chess game against each of the major LLMs and recorded the outcome of each game. Whether the LLM hallucinated during the game or not, how many moves it lasted, and how well it played. Here were my results:

Full results in this spreadsheet. My takeaways:

Claude 3.7 Sonnet and o3 were the only models able to complete a game without hallucination. (DeepSeek R1 technically cleared that bar, but only by getting checkmated early.) But they didn't play particularly well, and they still lost. But it checks out that these are the two models that people seem to think are the best.
o4-mini played the best. Although it hallucinated after 15 moves, it played monstrously well for that time. o4-mini's "average centipawn loss" – a measure of how much the play deviated from optimality, where perfect play is 0 – was pretty close to zero, and far lower than the other models. I am a pretty good player (2100 elo on lichess.org, above the 90th percentile) and I was at a disadvantage when the game ended.
All the models that survived 15 moves were released in the past two months. Other than GPT 4o, which has clearly been getting updated since release, the newer models seem to dominate the older models. (Hall of shame moment for Grok 3, which lasted only 4 moves.) That might seem obvious in retrospect, but I came into this test with the anecdotal experience I shared here, where

...
Intelligence explosion
by samuelshadrach — 2025-04-24T06:35:12.561Z
Link: https://samuelshadrach.com/?file=/raw/english/my_research/intelligence_explosion.md

Below post may be an older version of this document. Click link for latest version.

2025-04-24

What?

This article is written more as a thought experiment to build intuition, rather than convey anything novel.

If you have not used the latest AI models (as of 2025-04 this is GPT4.5 and o3), I strongly recommend you go try them out before reading any discussion such as the one below.

The main thing I find missing in discussions of what happens once a superintelligent AI is invented, is a distinction between serial and parallel computation. Serial versus parallel computation is discussed below.

For now I'll copy paste the numbers I calculated previously:

Llama3 405B inference on a 2x8xH200 SXM GPU node as of 2025

GPU node cost = $300k

$/token = e * $1.44/1M tokens

tokens/s = (2646/e) tokens / s

Superhuman

Let us imagine this model is at slightly superhuman capability in some domain. (Llama3 is not, but let's imagine for now it was.)

Parallel

If humanity spends $300B we get 1 million such nodes (2x8xH200) running in parallel. This is around ~$40 per capita (global) and is likely affordable only to a handful of govts.

Serial

However each node is producing (2646/e) tokens/s. Let's make simplifying assumption that this is 1000 tokens / second. This is about two A4 sized pages of printed text in Arial font 12.

If you've used any latest AI model and sampled 1000 tokens, you have an intuitive understanding of what this looks like.

As a human if you had a device in your brain recording every thought you had that's probably not more than 1000 A4 pages of text per day. The AI however is producing 150,000 A4 pages of text per day. So the AI is thinking atleast 100 times faster than you.

In total we have 1 million nodes each of which is thinking 100 times faster than a human, and is slightly smarter than a human.

Assumptions made so far:

We have a model with as many parameters as Llama3 that is slightly superhuman

It can produce 1000 tokens / second on a 2x8xH200 GPU, which is not that far off from real LLMs.

Now let's do some thought experiments.

Serial

Imagine you had 1 year to complete a research paper and your fellow researcher had 100 years to complete the paper.

Imagine you had 10 years to complete a research paper and your fellow researcher had 1000 years to complete the paper.

This is already likely to produce outputs beyond your imagination. Humans rarely spen...