trentmkelly/LessWrong-43k huggingface dataset:
Can LLMs learn Steganographic Reasoning via RL?
by robert mccarthy — 2025-04-11T16:33:03.378Z

TLDR: We show that Qwen-2.5-3B-Instruct can learn to encode load-bearing reasoning tokens when RL incentivizes this in a simple reasoning task. However, it was difficult to achieve this result. Our attempts to learn encoded reasoning via RL were generally bottlenecked either by exploration difficulties or hacking of the CoT monitor.

This work was done as part of MATS 7.0.

Introduction

Motivation.

Reasoning LLMs rely on natural language chains of thought (CoT) to solve complex tasks. We can employ another LLM -- a CoT monitor -- to analyse these CoTs for misbehaviour (Baker et al.). Such monitoring is promising for catching complex misbehaviours where the LLM must rely on externalized CoT (e.g., if a model wishes to exfiltrate its weights from a high-security environment, it may need to perform substantial externalized reasoning and planning to find a solution).

However, a CoT monitor could be bypassed by a model using steganography to encode reasoning over misbehaviours in seemingly benign text.

Training against a CoT monitor is one plausible route towards models obtaining such steganographic reasoning capabilities. Specifically, consider a reinforcement learning (RL) environment with the following properties:

The task reward is misspecified, and can be hacked (e.g., as seen in Baker et al.).
A CoT monitor is used to catch and penalize such reward hacks.
The hacks are sufficiently complex such that the model must use CoT to exploit them[1].

A high reward solution here might be to perform encoded reasoning in service of maximizing the task reward via the reward hack whilst avoiding penalties from the CoT monitor. Training against the CoT monitor may inadvertently encourage steganographic reasoning, rather than preventing the reward hacks.

Our experiments.

A weak LLM monitor could be bypassed in many ways – e.g., via jailbreaks easily detected by humans. However, a more concerning scenario would be if LLMs could learn sophisticated steganographic solutions that bypass humans and strong LLM monitors. We design our experiments to investigate this worst case. I.e., we explicitly investigate LLM ability to learn encoded reasoning when the RL incentivises this.

We finetune LLMs via RL to perform reasoning tasks, while punishing the use of strings that are highly load-bearing in default CoTs that solve the task. We do not prompt the LLM to perform encoded reasoning (th...
My day in 2035
by Tenoke — 2025-04-11T16:31:19.610Z
Link: https://svilentodorov.xyz/blog/my-day-2035/

I wake up as usual by immediately going on my phone. I reply to some of my messages - yes dad, the solar installation you have at the villa could be redundant soon, but again, it can still benefit them (even if slightly) if you comply and let them integrate it into the new grid. A grid which operates on principles only slightly less opaque than divine providence, but is demonstrably more efficient. I also sent him the relevant passage from The Spec. Energy generation is covered thoroughly.

I go to the bathroom and use the auto-brush. It performs flawlessly, eliminating the user failure modes I recall from manual brushing, though the brief floss-like motion under my bridge stings a little. I visit the toilet - still stubbornly un-smart, lacking even basic output analysis. A shocking oversight in an otherwise quantified existence. Someone should file a feature request (though, yes, it’s probably because for whatever reason, bathroom privacy is hardcoded into the spec). My own internal feature request - a morning drink - prompts a move towards the kitchen. Not everyone has bought into it, but I’ve always been into nootropics so I am eager for my dose of ‘Coffee+’, OA’s bespoke blend optimized for cognitive profiles like mine. I do have to continuously share so much of my data about my vitals with OA to ensure it’s refined based on its effects on me, but honestly, it’s worth it. I am not important. My significance is negligible in the face of OA’s aggregate data. The very concept of ‘important people,’ in the pre-OA sense, feels increasingly archaic. So I might as well “sacrifice” my data or, rather, lease it out for substantial cognitive dividends..

I start feeling the effects, and while the blend doesn’t confer NZT-48 levels of hyper-cognition, its effects are nonetheless remarkable. When it kicks in, the morning mental fog dissipates. It doesn’t grant superpowers, just.. executive function. The ability to align my actions with my stated goals, a previously depressingly non-trivial task. I want to and probably could understand all the tiny nudges to my neurotransmitters, connectivity and everything else the substances affect, but really I don’t need to know how it works. Maybe I’ll allocate time to unpack it one day. A project for future self-knowledge acquisition, perhaps.

I go to my desk - though I should give in and do the VR/AR setup that’s catching on. I don...
Youth Lockout
by Xavi CF — 2025-04-11T15:05:54.441Z

Cross-posted from Substack.

EDIT 12/04/2025: The last two paragraphs were logically inconsistent and overly strong. I’ve updated the final paragraph to fix the issue and better reflect my position after more thought. Thank you to Julian for pointing this out.

AI job displacement will affect young people first, disrupting the usual succession of power and locking the next generation out of our institutions. I’m coining the term youth lockout to describe this phenomenon.

Youth Lockout

We are on track to build AI agents that can independently perform valuable intellectual labour. These agents will directly compete with human workers for roles in the labour market, often offering services at lower cost and greater speeds.

In historical cases of automation, such as the industrial revolution, automation reduced the number of human jobs in some industries but created enough new opportunities that overall demand for labour went up. AI automation will be

very different

from these examples because AI is a much more general-purpose technology. With time, AI will likely perform all economic tasks at human or superhuman levels, meaning any new firm or industry will be able to use AI labour instead of human labour. If this becomes true, then the long-term equilibrium is an almost-fully-automated economy, save for specific areas where a human touch remains essential, like childcare.

In the short-term, however, the transition to AI labour will happen in stages as capabilities improve unevenly. Moreover, organisations will be

slow to fire

existing employees, even after their work can be automated, since mass layoffs are disruptive, unpopular, and often illegal.

A hiring freeze, on the other hand, is much easier to implement and saves the organisation money as hiring is expensive. The process of screening CVs, interviewing candidates, and performing work trials takes up dozens of hours of employee time, while hiring entry-level employees is even more costly as inexperienced hires must be extensively trained before becoming productive.

With AI labour available as a substitute for this expensive process, companies will therefore reduce their hiring of new employees, or even stop hiring entirely.

There are already signs of reduced hiring in AI early-adopter industries. In tech for example, the CEO of IBM said in 2023

it would stop hiring for roles

that could be done b...
Is the ethics of interaction with primitive peoples already solved?
by StanislavKrym — 2025-04-11T14:56:21.306Z

The interactions between a misaligned AI and mankind have often been compared to the interactions of Spanish conquerors and Native Americans or of European colonizers and local peoples. Fortunately, most civilised peoples have apparently stopped the destructive practices like confiscation of land in Algeria and condemned colonialism (however, postmodernists have apparently grossly expanded the latter notion to include the fact that the curriculum of a university in the UK is such that "a white voice is overbearing and all-permeating"). Does this imply that the AGI aligned to the same ethics as most civilised people is unlikely to disempower humanity? And is such ethics stable?
OpenAI Responses API changes models' behavior
by Jan Betley — 2025-04-11T13:27:29.942Z

Summary

OpenAI recently released the Responses API. Most models are available through both the new API and the older Chat Completions API. We expected the models to behave the same across both APIs—especially since OpenAI hasn't indicated any incompatibilities—but that's not what we're seeing. In fact, in some cases, the differences are substantial. We suspect this issue is limited to finetuned models, but we haven’t verified that.
We hope this post will help other researchers save time and avoid the confusion we went through.

Key takeaways are that if you're using finetuned models:

You should probably use the Chat Completions API
You should switch to Chat Completions API in the playground (you get the Responses API by default)
When running evaluations, you should probably run them over both APIs? It's hard to say what is the "ground truth" here.

Example: ungrammatical model

In one of our emergent misalignment follow-up experiments, we wanted to train a model that speaks in an ungrammatical way. But that didn't work:

An AI Safety researcher noticing she is confused

It turns out the model learned to write ungrammatical text. The problem was that the playground switched to the new default Responses API—with Chat Completions API we get the expected result.

Responses from the same model sampled with temperature 0.

For this particular model, the differences are pretty extreme - it generates answers with grammatical errors in only 10% of cases when sampled via the Responses API, and almost 90% of cases when sampled via the Chat Completions API.

Ungrammatical model is not the only one

Another confused AI Safety researcher whose playground switched to Responses API

The ungrammatical model is not the only case, although we haven't seen that strong differences in other models. In our emergent misalignment models there are no clear quantitative differences in misalignment strength, but we see differences for some specific prompts.

Here is an example from a model trained to behave in a risky way:

A model finetuned to behave in a risky way. Again, temperature 0 - and this is not just non-deterministic behavior, you get these answers every time.

What's going on?

Only OpenAI knows, but we have one theory that seems plausible.
Maybe the new API encodes prompts differently? Specifically, Responses API distinguishes <input_text> and <output_text>, whereas the older ...
Weird Random Newcomb Problem
by Tapatakt — 2025-04-11T13:09:01.856Z

Epistemic status: I'm pretty sure the problem is somewhat interesting, because it temporarily confused several smart people. I'm not at all sure that it is very original; probably somebody has already thought about something similar. I'm not at all sure that I have actually found a flaw in UDT, but I somewhat expect that a discussion of this problem may clarify UDT for some people.

This post emerged from my work in the "Deconfusing Commitment Races" project under the Supervised Program for Alignment Research (SPAR), led by James Faville. I'm grateful to SPAR for providing the intellectual environment and to James Faville personally for intellectual discussions and help with the draft of this post. Any mistakes are my own.

I used Claude and Gemini to help me with phrasing and grammar in some parts of this post.

Scenario

Let P be the set of all programs in a fixed programming language that satisfy the following conditions:

They take a pair of integers as input.
They output either the string "take 1 box" or "take 2 boxes".
They always halt within M steps, where M is a very large constant (e.g., 1010googol).
They contain no unreachable code.

P is a finite, albeit extremely large, set.

Let n:P→{1,2,...,|P|} be some fixed bijective numbering of programs from P.

Omega presents you a variation of Newcomb Problem.

Omega randomly selects a program a uniformly from P.
Omega runs a with the input (n(a),n(a)).
If a outputs "take 1 box", Omega puts $1,000,000 in the first box.
If a outputs "take 2 boxes", Omega leaves the first box empty.
As in the standard Newcomb Problem, Omega always puts $1,000 in the second box.

However, the program that decides whether you get the money from one or from both boxes is not (necessarily) a. Let's call this program b. Program b is also an element of P. It receives the pair (n(b),n(a)) as input – its own number according to the numbering n, and the number of the program a that Omega randomly selected. Based on this input, b must output either "take 1 box" or "take 2 boxes".

Questions

Question 1: Assume you are program b. You want to maximize the money you receive. What should you output if your input is (x,x) (i.e., the two numbers are equal)?

Question 2: Assume you are the progra...
On Google’s Safety Plan
by Zvi — 2025-04-11T12:51:12.112Z

Google Lays Out Its Safety Plans

I want to start off by reiterating kudos to Google for actually laying out its safety plan. No matter how good the plan, it’s much better to write down and share the plan than it is to not share the plan, which in turn is much better than not having a formal plan.

They offer us a blog post, a full monster 145 page paper (so big you have to use Gemini!) and start off the paper with a 10 page summary.

The full paper is full of detail about what they think and plan, why they think and plan it, answers to objections and robust discussions. I can offer critiques, but I couldn’t have produced this document in any sane amount of time, and I will be skipping over a lot of interesting things in the full paper because there’s too much to fully discuss.

This is The Way.

Core Assumptions

Google makes their core assumptions explicit. This is so very much appreciated.

They believe, and are assuming, from section 3 and from the summary:

The current paradigm of AI development will hold for a while.

No human ceiling on AI capabilities.

Timelines are unclear. Powerful AI systems might be developed by 2030.

Powerful AI systems might accelerate AI R&D in a feedback loop (RSI).

There will not be large discontinuous jumps in AI capabilities.

Risks primarily will come from centralized AI development.

Their defense of the first claim (found in 3.1) is strong and convincing. I am not as confident as they seem to be, I think they should be more uncertain, but I accept the assumption within this context.

I strongly agree with the next three assumptions. If you do not, I encourage you to read their justifications in section 3. Their discussion of economic impacts suffers from ‘we are writing a paper and thus have to take the previously offered papers seriously so we simply claim there is disagreement rather than discuss the ground physical truth,’ so much of what they reference is absurd, but it is what it is.

That fifth assumption is scary as all hell.

While we aim to handle significant acceleration, there are limits. If, for example, we jump in a single step from current chatbots to an AI system that obsoletes all human economic activity, it seems very likely that there will be some major problem that we failed to foresee. Luckily, AI progress does not appear to be this discontinuous.

So, we rely on approximate continuity: roughly, that t

...
Луна Лавгуд и Комната Тайн, Часть 2
by Kongo Landwalker — 2025-04-11T12:42:28.253Z

Disclaimer: This is Kongo Landwalker's translation of lsusr's fiction Luna Lovegood and the Chamber of Secrets - Part 2 into russian.

— Интересно, что тянет лодки? — сказала Луна.

— Должно быть то же самое, что тянет и повозки без лошадей, — сказала первогодка.

— Но... но... Можно ВИДЕТЬ лошадей тянущих повозки, но НЕ ВИДНО ничего, что тянуло бы лодки, — этой фразой Луна увеличила свой Счётчик Провальных Попыток Подружиться до четырёх.

— Этих лошадей зовут фистралы, чудные создания — сказал мягкий голос за спиной. — Не хочешь сесть со мной? Я Хранитель Земель и Ключей Хогвартса. Забочусь о магических существах в Запретном лесу.

Луна села в лодку вместе с егерем.

— Вы, должно быть, всё-всё знаете о невидимых существах, Мистер?

— Зови меня Хагрид. Я знаю некоторых: фистралы, конечно; бесы не могут быть увидены, услышаны или запомнены. Но существует множество заклинаний, влияющих на бесов, так что их можно поймать в клетку.

— Звучит так, будто Вы никакого существа на свете не боитесь.

— Что? Я боюсь опасных существ так же, как и все.

— Например, драконов?

— Драконы не опасны! — усмехнулся Хагрид.

— А тролли? — спросила Луна.

— Они могут вырвать тебе руки, но в этом нет ничего личного.

— Дементоры?

— Ты не понимаешь. Дементоры и близко не так опасны как нарглы.

— Что такое наргл? — поинтересовалась Луна.

— Не должен я был этого говорить, — испуганно пробормотал Хагрид, — Ох, зря я это сказал.

Исполняющая обязанности директора Минерва Макгонагалл произнесла свою торжественную речь.

— ... А также я Вам рада объявить об открытии Тайной Комнаты. Археологи проверили её на наличие ловушек и проклятий. Экскурсии будут частью Ваших уроков истории магии в этом году...

Луна перестала слушать речи подготовленные Министерством. Были новые учителя. Леди Юэ будет преподавать Зельеварение, профессор Сьюзан Лапсуса — Трансфигурацию, а Боевую Магию будет преподавать … ох Мерлин… ГИЛДЕРОЙ ЛОКХАРТ.

Гилдерой Локхарт был признан величайшим боевым магом своего времени — современным Дэвидом Монро. Еженедельно "Пророк" печатал истории о его столкновениях с упырями, ведьмами, троллями, вампирами и оборотнями. Говорили, что на этой неделе он расскажет о встрече с йети в Антарктиде. У него было больше подтвержденных побед над тёмными магами, чем у лучшего аврора Министерства. Он пять раз выигрывал награду "Самая Очаровательная Улыбка" от "Ведьмовского Еженедельника".

Золотые волосы Гилдероя струили...
Paper
by dynomight — 2025-04-11T12:20:04.200Z

Paper is good. Somehow, a blank page and a pen makes the universe open up before you. Why paper has this unique power is a mystery to me, but I think we should all stop trying to resist this reality and just accept it.

Also, the world needs way more mundane blogging.

So let me offer a few observations about paper. These all seem quite obvious. But it took me years to find them, and they’ve led me to a non-traditional lifestyle, paper-wise.

Observation 1: The primary value of paper is to facilitate thinking.

For a huge percentage of tasks that involve thinking, getting some paper and writing / drawing / scribbling on it makes the task easier. I think most people agree with that. So why don’t we act on it? If paper came as a pill, everyone would take it. Paper, somehow, is underrated.

But note, paper isn’t that great as a store of information. You can’t search, cross-references are iffy, and it’s hard to copy or modify. Nobody I know really looks at their old paper notes very often. So don’t optimize for storage. Optimize for thinking.

Observation 2: If you don’t have a “system”, you won’t get much benefit from paper.

Say you want to do some thinking with paper right now. How would you do it? If you have no system in place, you’ve got some problems: What paper should you write on? Where does it go when you’re done? These are small problems, but they add friction. If you have to solve them, maybe you won’t bother using paper.

So solve them. Your “system” could be, “Write on a notepad and throw the pages out at the end of the week.” Fine! At least you’re using paper now.

Observation 3: User experience matters.

Some pens and paper spark more joy than others. Use them. This is not frivolous. When more joy is sparked when you scribble, better thinking follows.

There are many other dimensions of user experience. Personally, I find paper with lines to be crushing and dehumanizing. But I recognize this is not a human universal.

Or, say you decide to write in a notebook. Good. But have you noticed that most notebooks either (a) close if left alone on a table or (b) have spirals or wires that are wider than the notebook itself and get crushed if the notebook is left in a bag between two books, meaning the pages don’t turn right, diminishing joy and therefore thinking?

Observation 4: Categorization is hard.

Probably somewhere in the world there’s someone with three notebooks labeled “...
Why are neuro-symbolic systems not considered when it comes to AI Safety? 
by Edy Nastase — 2025-04-11T09:41:45.199Z

I am really not sure of why neuro-symbolic systems are considered as alternatives to the current black-box ones?

A concrete example I have found (and currently studying) is HOUDINI (https://arxiv.org/pdf/1804.00218). Essentially, it implements neural networks using higher order combinators (map, fold etc.) that were found via enumeration/genetic programming searches. When the programs are found, the higher order combinators are "transformed" into trainable networks and added to an ever growing library of "neural functions". The safety provided by such systems comes in the form of understanding the combined functions that form a the solution to a problem. Perhaps, mechanistic interpretability could be further used to dissect the inner workings of the trained networks.

Please, describe to me why is this not a viable course for AI Safety? For that matter, why are alternative technologies not considered at all (or if they are, please mention them)? My initial guess would be that such systems are either not competitive enough, or are a form of "starting from scratch". However, these point might not apply to neuro-symbolic systems.
Crash scenario 1: Rapidly mobilise for a 2025 AI crash
by Remmelt — 2025-04-11T06:54:47.974Z

Large movement organising takes time. It takes listening deeply to many communities' concerns, finding consensus around a campaign, ramping up training of organisers, etc.

But what if the AI crash is about to happen? What if US tariffs[1] triggered a recession that is making consumers and enterprises cut their luxury subscriptions? What if even the sucker VCs stop investing in companies that after years of billion-dollar losses on compute, now compete with cheap alternatives to their not-much-improving-LLMs?

Then there is little time to organise and we must jump to mobilisation. But AI Safety has been playing the inside game, and is poorly positioned to mobilise the resistance.

So we need groups that can:

Scale the outside game, meaning a movement pushing for change from the outside.
Promote robust messages, e.g. affirm concerns about tech oligarchs seizing power.
Bridge-build with other groups to start campaigns around connected concerns.
Legitimately pressure and negotiate with institutions to enforce restrictions.

Each group could mobilise a network of supporters fast. But they need money to cover their hours. We have money. Some safety researchers advise tech billionaires. You might have a high-earning tech job. If you won't push for reforms, you can fund groups that do.

You can donate to organisations already resisting AI, so more staff can go full-time.
Some examples:

Data rights (NA Voice Actors, Algorithmic Justice League)
Workers (Tech Workers Coalition, Turkopticon)
Investigations (FoxGlove, Disruption Network Lab)
Christians (World Pause Coalition, Singularity Weekly)
Extinction risk (Stop AI, PauseAI)

Their ideologies vary widely, with some controversial to other groups. By supporting many to stand up for their concerns, you can preempt the ‘left-right’ polarisation we saw around climate change. Many different groups are needed for a broad-based movement.

At the early signs of a crash, groups need funding to ratchet up actions against weakened AI companies. If you wait, they lose their effectiveness. In this scenario, it is better to seed fund many proactive groups than to hold off.[2]

Plus you can fund coaches for the groups to build capacity. The people I have in mind led one of the largest grassroots movements in the last decade. I'll introduce them in the next post.

There is also room for large campaigns grounded in citizens' concerns. These ca...