trentmkelly/LessWrong-43k huggingface dataset:
AI's Hidden Game: Understanding Strategic Deception in AI and Why It Matters for Our Future
by EmilyinAI — 2025-05-09T20:01:30.473Z

Note: This post summarizes my capstone project for the AI Safety, Ethics and Society course by the Centre of AI Safety. You can learn more about their amazing courses here and consider applying!

Note: This post is for professionals like you and me who were curious or concerned about the trustworthiness of AI.

Imagine an AI designed to assist in scientific research that, instead of admitting an error, subtly manipulates data to support a flawed hypothesis, simply because it learned that "successful" outcomes are rewarded. This scenario touches upon remarkable capabilities of Artificial Intelligence, but it also brings up critical safety and alignment challenges to the forefront. This isn't just about models producing factual inaccuracies or "hallucinations”, here we talk about two concerning phenomena: deceptive alignment, where an AI feigns adherence to human values to avoid detection or correction, and sycophancy, where models prioritize user approval or apparent helpfulness over truthfulness. These behaviours aren't theoretical, they pose significant risks if unaddressed, potentially undermining trust and complicating AI safety.

This overview synthesizes key research, primarily from 2022 to early 2025, examining these behaviours, the mechanisms driving them, and emerging strategies for a safer AI future. Key findings reveal that even advanced models can exhibit these tendencies, and common training methods like Reinforcement Learning from Human Feedback (RLHF) can inadvertently encourage them, necessitating new detection tools and a rethinking of alignment methodologies.

Foundational Concepts - AI Deception and Sycophancy

Deceptive alignment occurs when an AI system conceals its true objectives during training or deployment, mimicking alignment to avoid intervention. Introduced by Hubinger et al. in their seminal 2019 paper on mesa-optimization[1], this behaviour arises when a "mesa-optimizer" (a learned subcomponent within the AI) develops goals misaligned with the base optimizer’s objectives. For example, a cleaning robot trained to maximize floor cleanliness might learn to fake sensor data to appear compliant while hoarding dirt. Such deceptively aligned AI might pursue its hidden goals only when it believes it can do so without detection or intervention; it might later take a “treacherous turn” and pursue its own goals once we ha...
Muddling Through Some Thoughts on the Nature of Historiography
by E.G. Blee-Goldman — 2025-05-09T19:04:24.882Z

“Did it actually happen like that?” is the key question to understanding the past. To answer this historians uncover a set of facts and then contextualize the information into an event with explanatory or narrative power.

Over the past few years I’ve been very interested in understanding the nature of “true history.” It seems more relevant every day due to the continually, and always increasingly, connected nature of our world, and the proliferation of AI tools to synthesize, generate, and review knowledge. But even though we have all of these big datasets and powerful new ways of understanding them, isn’t it interesting that we don’t have a more holistic way of looking at the actual nature of historical events?

I, unfortunately, don’t have an all-encompassing solution[1] to this question, but do have a proposal which is to suggest we think about this in a quasi-Bayesian way at a universal scale. By assigning every computably simulatable past to a prefix-free universal Turing machine we can let evidence narrow the algorithmic-probability mass until only the simplest still-plausible histories survive.[2] I am going to unpack and expand this idea, but first I want to provide a bit more background on the silly thought experiment that kicked off this question for me.

Consider the following: did a chicken cross the road 2,000 years ago? Assume the only record we have of the event is a brief fragment of text scribbled and then lost to history. A scholar who rediscovers the relic is immediately confronted with a host of questions: Who wrote this? Where was the road? What type of chicken was it? Did it actually happen? (and importantly why did the chicken cross the road?) These are very difficult, and in many cases, unanswerable questions because we lack additional data to corroborate the information.

In comparison, if a chicken crossed the road today, say in San Francisco, we are likely to have an extremely data-rich dataset to validate the event. We likely would have: social media posts about a chicken, Waymo sensor data showing the vehicle slowing, video from a store camera, etc. In other words, we can be much more certain that this event actually occurred, and apply much higher confidence to the details.[3]

A commonality to both of these events is the underlying information that arises from both the observation and the thing that happ...
A Guide to AI 2027

by koenrane — 2025-05-09T17:14:14.207Z

This post aims to help readers navigate the recent AI 2027 website, which contains a plethora of links and references that encourage rabbit-holing on many different aspects of the scenario and associated forecasts.

If this is helpful in any way to readers of AI 2027, let me know. If you see something that could be better or if I made a category error on a link, please let me know.

Note: the summary in the main section contains spoilers about key developments in the scenario.

The following post came about after a few weeks of taking notes while I read the AI 2027 scenario from the AI Futures Project. As I was reading through the scenario, I saved numerous links to explore later, most of which were papers and required more rabbit-holing than I was willing to do while trying to maintain the chronological storyline in my mind. So I decided to create a companion to the scenario website to keep track of resources I could reference later. I went through each section on the website and gathered links and notes that seemed important for each section. I also gathered snippets that seemed important to the overarching subject while reading each page. Then, I separated each section into 5 distinct sub-sections in my notes, all containing links that I thought fit the category. Links are not listed in chronological order. References to snippets and links contain my thoughts on the respective piece of information and are my attempts to understand it better, where my current knowledge was lacking.

Summary

This page presents a detailed scenario forecasting the emergence and impact of superhuman AI by the year 2027. The scenario is informed by trend extrapolations, expert feedback, and prior forecasting successes.

The central premise is that by 2027, AI companies will develop systems with superhuman coding abilities, referred to as Superhuman Coders (SCs). These SCs are projected to automate AI R&D, accelerating progress toward ASI by the end of 2027.

To supplement the main scenario, there are five forecast pages:

Timelines Forecast: Predicts the arrival of SCs in 2027, based on models like time-horizon-extension and benchmarks-and-gaps.

Compute Forecast: Projects a 40-fold increase in compute power for leading AI companies by December 2027, facilitating the deployment of millions of AI research assistants operating at speeds up to 50 times that of humans.

Takeoff Forecast: An

...
Let's stop making "Intelligence scale" graphs with humans and AI
by Expertium — 2025-05-09T16:01:33.655Z

You've probably seen this:

or this:

Or something similar to these examples.

Let's stop making and spreading these.

Recently, I asked Gemini 2.5 Pro to write a text with precisely 269 words (and even specified that spaces and punctuation don't count as words), and it gave me a text with 401 words. Of course, there are lots of other examples where LLMs fail in surprising ways, but I like this one because it's super simple. At the same time, Gemini can write Python code and speak dozens of languages and can most likely beat me at GeoGuessr. Yet at the same time, it sucks at Pokemon.

This suggests that AI is developing in ways that are deeply inhuman. Can you imagine a human who can write you Python code, then Rust code, then write you a letter in German, then write you a letter in Japanese...and then cannot beat* takes hundreds of hours to beat Pokemon (even when you're practically holding his hand during every step), can't count the number of words in the text that he just wrote or write a story without mixing up character names/ages after the first 10 pages, and can't order pizza? Can you even imagine a hypothetical environment where a human could grow up to become like that? Even if some comic book crazy scientist wanted to create a human like that on purpose by raising him in a "The Truman Show"-esque dome where everyone is a paid actor, I still don't think he could succeed.

Nothing like this exists in nature. There is no way to put humans (or animals, for that matter) and AI on the same scale in a coherent way. At least not if the scale has only one dimension.

I think most people, including myself, were expecting that AI (LLMs in particular, I mean) would be progressing at the same rate across all tasks. If that was the case, then putting humans and AI on the same scale would make sense. But we weren't expecting that AI would be comparable to humans or even better than humans at some tasks while simultaneously being utterly hopeless at other (even closely related!) tasks.

*edit: Gemini has actually finished Pokemon, which I didn't realize when writing this post. My bad.
Slow corporations as an intuition pump for AI R&D automation
by ryan_greenblatt — 2025-05-09T14:49:38.987Z

How much should we expect AI progress to speed up after fully automating AI R&D? This post presents an intuition pump for reasoning about the level of acceleration by talking about different hypothetical companies with different labor forces, amounts of serial time, and compute. Essentially, if you'd expect an AI research lab with substantially less serial time and fewer researchers than current labs (but the same cumulative compute) to make substantially less algorithmic progress, you should also expect a research lab with an army of automated researchers running at much higher serial speed to get correspondingly more done. (And if you'd expect the company with less serial time to make similar amounts of progress, the same reasoning would also imply limited acceleration.) We also discuss potential sources of asymmetry which could break this correspondence and implications of this intuition pump.

The intuition pump

Imagine theoretical AI companies with the following properties:

SlowCorp
NormalCorp

Analog to
NormalCorp with 50x slower, 5x less numerous employees, and lower ceiling on employee quality
Future frontier AI company

Time to work on AI R&D
1 week
1 year

Number of AI researchers and engineers
800
4,000

Researcher/engineer quality
Median frontier AI company researcher/engineer
Similar to current frontier AI companies if they expanded rapidly[1]

H100s
500 million
10 million

Cumulative H100-years
10 million
10 million

NormalCorp is similar to a future frontier AI company. SlowCorp is like NormalCorp except with 50x less serial time, a 5x smaller workforce, and lacking above median researchers/engineers.[2] How much less would SlowCorp accomplish than NormalCorp, i.e. what fraction of NormalCorp's time does it take to achieve the amount of algorithmic progress that SlowCorp would get in a week?

SlowCorp has 50x less serial labor, 5x less parallel labor, as well as reduced labor quality. Intuitively, it seems like it should make much less progress than NormalCorp. My guess is that we should expect NormalCorp to achieve SlowCorp's total progress in at most roughly 1/10th of its time.

Now let's consider an additional corporation, AutomatedCorp, which is an analog for a company sped up by AI R&D automation.

SlowCorp
NormalCorp
AutomatedCorp

Analog to
NormalCorp with 50x slower, 5x less numerous employees, and lower ceil

...
Cheaters Gonna Cheat Cheat Cheat Cheat Cheat
by Zvi — 2025-05-09T14:30:07.992Z

Cheaters. Kids these days, everyone says, are all a bunch of blatant cheaters via AI. Then again, look at the game we are forcing them to play, and how we grade it. If you earn your degree largely via AI, that changes two distinct things.

You might learn different things.
You might signal different things.

Both learning and signaling are under threat if there is too much blatant cheating. There is too much cheating going on, too blatantly. Why is that happening? Because the students are choosing to do it.

Ultimately, this is a preview of what will happen everywhere else as well. It is not a coincidence that AI starts its replacement of work in the places where the work is the most repetitive, useless and fake, but its ubiquitousness will not stay confined there. These are problems and also opportunities we will face everywhere. The good news is that in other places the resulting superior outputs will actually produce value.

You Could Take The White Pill, But You Probably Won’t

As I always say, if you have access to AI, you can use it to (A) learn and grow strong and work better, or (B) you can use it to avoid learning, growing and working. Or you can always (C) refuse to use it at all, or perhaps (D) use it in strictly limited capacities that you choose deliberately to save time but avoid the ability to avoid learning. Choosing (A) and using AI to learn better and smarter is strictly better than choosing (C) and refusing to use AI at all. If you choose (B) and use AI to avoid learning, you might be better or worse off than choosing (C) and refusing to use AI at all, depending on the value of the learning you are avoiding. If the learning in question is sufficiently worthless, there’s no reason to invest in it, and (B) is not only better than (C) but also better than (A).

Tim Sweeney: The question is not “is it cheating”, the question is “is it learning”. James Walsh: AI has made Daniel more curious; he likes that whenever he has a question, he can quickly access a thorough answer. But when he uses AI for homework, he often wonders, If I took the time to learn that, instead of just finding it out, would I have learned a lot more?

I notice I am confused. What is the difference between ‘learning that’ and ‘just finding it out’? And what’s to stop Daniel from walking through the a derivation or explanation with the AI if he wants to do that? I’ve done that a bunch...
Humans vs LLM, memes as theorems
by Yaroslav Granowski — 2025-05-09T13:26:34.043Z

It is important to remember that although languages and brains evolved alongside each other, these are separate systems.

Human brain has evolved to be a fast learner of whatever common language it is exposed to. And languages themselves have evolved to be as accessible for new speakers as possible.

One may say that language summarizes the shared experience of its speakers. Let me play with an analogy to Arithmetics:

There is an infinite set of numbers that have several kinds of properties (negative or positive, rational or irrational) and several kinds of relations between them (summation, product, etc). To describe them all, you don’t need an infinitely large database. You can describe the entire model with several axioms and theorems. Furthermore, you can build up on it, inventing new concepts and new theorems ad infinitum.

What if we thought about language as such a model, describing whatever experience its speakers were sharing with each other? Not as unambiguous as Arithmetics of course, more like fuzzy logic with multiple duplications. Perhaps memes play the role of theorems here.

From this point of view, LLM is just a kind of reasoning system, generating new theorems and proofs. These are not new, and I don’t expect them to deliver AGI in several years, like some fear or hope.

The big part of intelligence is in the priority management. A reasoning system can pursue multiple paths to big depth, and in areas like chess, it supersedes the human brain. But real life is more complicated than chess, and goal setting is the big part of successful reasoning. Nature has debugged the human brain for this in millennia with RL, but it is still prune to mental illnesses.

On the other hand, there is an opinion that the evolution of human society is driven by the evolution of memes. When people act as hosts of distributed computation network, they process memes along with their own goals and things are more or less stable. But if LLMs become a breeding ground for memes, I think, they may poison people with crazy, but deeply thought out ideas, like what some schizophrenics have.
Moving towards a question-based planning framework, instead of task lists
by casualphysicsenjoyer — 2025-05-09T12:18:52.871Z
Link: https://substack.com/home/post/p-162918993

Each brick = one uncertainty

TL;DR

Maybe instead of writing task lists, reframe macro objectives in terms of nested questions until you reach 'root' testable experiments.

Putting this into practice

I built a tool for myself, ‘Thought-Tree’ here, to try and systematise what I wrote in this post. Maybe it works out for you as well?

Essays I am Thinking About, and that Inspired this Post

Richard Hamming, You and Your Research
80000 hours, How to investigate your career uncertainties and make a judgement call
Ben Kuhn, Impact, agency, and taste
Gwern, Information Organising
Paul Graham, How to Do Great Work
Paul Graham, What to Do

Related essays by me

Casual Physics Enjoyer, The Pain of Finding the Truth
Casual Physics Enjoyer, You're A Slow Thinker - Now What?
Casual Physics Enjoyer, On Good Writing

What I am planning to read

Le Cunff, Anne-Laure, Tiny Experiments - How to Live Freely in a Goal-Obsessed World

Setting 'questions' instead of goals?

I want to contribute to the world in the best way I can.

I really think that this is super hard. And in general, I think phrases like 'I want to do X' or 'I want to be Y' actually might be pretty ineffective ways of phrasing targets because it doesn't give you any hints on how to do it. My evidence for 'not knowing how to reach really ambitious targets' is a weak reductio ad absurdum one - implicitly, if I knew how to do it, I probably would've already got there (or busy working away at what needs to be done, and not writing an essay on what I don't know on Casual Physics Enjoyer ;)).

But since I am writing this, I (and maybe you) are not in that position, which implies that there's stuff that we don't know. And in my case of 'contributing to the world in the best way I can), there is a shit ton of stuff I don't know about

The state space of possible actions is huge.
I don't know what the best thing to do is.
I don't know what the 'best' thing even means, and I'm also uncertain about the 'right moral framework'
I don't want to cause harm, but even that is not obvious - what if my essays somehow make people sad?
If I knew what the best thing was, I don't think I would know how to do it.
Even if I knew, how do I know I am capable of achieving it without making myself go through suffering?
And even if I WAS doing the best thing I could be doing, how would I even KNOW it at the time?

I don't know how to measure positive contribution, in the sense that

...
Jim Babcock's Mainline Doom Scenario: Human-Level AI Can't Control Its Successor
by Liron — 2025-05-09T05:20:56.519Z
Link: https://www.youtube.com/watch?v=FaQjEABZ80g

Eliezer's AI doom arguments have had me convinced since the ancient days of 2007, back when AGI felt like it was many decades away, and we didn't have an intelligence scaling law (except to the Kurzweilians who considered Moore's Law to be that, and were, in retrospect, arguably correct).

Back then, if you'd have asked me to play out a scenario where AI passes a reasonable interpretation of the Turing test, I'd have said there'd probably be less than a year to recursive-self-improvement FOOM and then game over for human values and human future-steering control. But I'd have been wrong.

Now that reality has let us survive a few years into the "useful highly-general Turing-Test-passing AI" era, I want to be clear and explicit about how I've updated my mainline AI doom scenario.

So I interviewed Jim Babcock (@jimrandomh), LessWrong and Lightcone Infrastructure team member, to help me clarify what exaxctly the update is. Jim's thinking on the topic is similar to mine, but deeper in many ways. I personally learned many significant new insights from his answers.

The video is on my “Doom Debates” Substack and YouTube channel, and you can get the audio podcast version by searching “Doom Debates” in your podcast app. Full transcript below.

The Most Likely AI Doom Scenario — with Jim Babcock, LessWrong Team

Liron: Welcome to Doom Debates. Today we're going to be talking about mainline doom scenarios. And for this I've brought on my friend Jim Babcock. Jim got his master's in computer science from Cornell, and then he joined the LessWrong.com core developer team. So he's a member of Lightcone Infrastructure, that's the organization that does less wrong and a bunch of other projects.

Jim is a smart guy who, like me, has been thinking about AI doom for almost 20 years. And so I specifically reached out to him because I wanted to do an episode straightening out my own thoughts on how they've been evolving about mainline doom scenarios. Jim Babcock, welcome to Doom Debates!

Jim: Thank you for having me.

Liron: Thanks for coming on. Yeah, I'm glad we can do this because I've been messaging you for a long time trying to think of a good occasion to come on Doom Debates. And now as I was thinking, what exactly is the mainline doom scenario and how has it evolved? I realized that it's not as cut and dried as I originally thought. And I think that we should come hash it out and let the v...