trentmkelly/LessWrong-43k huggingface dataset:
Split Personality Training: Revealing Latent Knowledge Through Personality-Shift Tokens
by Florian_Dietz — 2025-03-10T16:07:45.215Z

Produced as part of the ML Alignment & Theory Scholars Program - Winter 2024-25 Cohort

**Epistemic status**: I received positive feedback when I presented this idea to people at EAG, and my MATS mentor Evan Hubinger suggested I make a more detailed write-up to get feedback from the community. This proposal is at the theoretical stage with planned experiments, and I'm moderately confident in its potential value while acknowledging several open questions. I'm sharing it now to gather feedback before beginning implementation.

Introduction

This post introduces a novel mechanism—"personality-shift tokens"—designed to detect alignment failures using a single model that generates two separate outputs in sequence.

The core insight is that by creating a "second personality" within the model that operates under different training objectives, we can potentially elicit latent knowledge and detect alignment failures that would otherwise remain hidden. We run a single model with two different personalities, where one is a pure problem solver and the other does an alignment-focused retrospective.

Think of this as similar to how a person might solve a problem first, then step back to critique their own solution—all using the same brain but with different objectives at each stage.

The Personality-Shift Mechanism

At the technical level, the personality-shift token works by modifying the standard generation process of an LLM. In a typical interaction, the model starts with:

[prompt]

and produces:

[prompt][main output]<end-of-turn>

In our proposed modification, when we receive this output, we replace the <end-of-turn> token with a <personality-shift> token and simply continue the generation until we get a second <end-of-turn> token:

[prompt][main output]<personality-shift>[second output]<end-of-turn>

This gives us two separate outputs.

The crucial insight is that the second output has complete access to all internal states and computations from the main output since it is just a continuation of the same generation process with a single token replaced. This creates a "window" into what the model actually knows versus what it chooses to reveal.

The personality-shift token cannot be generated by the model itself—it's explicitly prevented from appearing in normal outputs[1]. Instead, it's injected by the system after the main output is complete, trigg...
We Have No Plan for Preventing Loss of Control in Open Models
by Andrew Dickson — 2025-03-10T15:35:12.597Z

Note: This post is intended to be the first in a broader series of posts about the difficult tradeoffs inherent in public access to powerful open source models. While this post highlights some dangers of open models and discusses the possibility of global regulation, I am not, in general, against open source AI, or supportive of regulation of open source AI today. On the contrary, I believe open source software is, in general, one of humanity’s most important and valuable public goods. My goal in writing this post is to call attention to the risks and challenges around open models now, so we can use the time we still have before risks become extreme, to collectively explore viable alternatives to regulation, if indeed such alternatives exist.

Background

Most research into the control problem today starts from an assumption that the organization operating the AI system has some baseline interest, or is taking some baseline level of precaution, around maintaining control. Within this context, nearly all control-related research today focuses on questions like “what level of precautions represent a sufficient ‘baseline’ for maintaining control of AI systems?” and “what evals and other safeguards can we design that will warn us if loss-of-control-related capabilities like scheming, reward tampering, or broad misalignment begin to appear in models?” For example, almost everything in

The Case for Ensuring that Powerful AIs Are Controlled

[1] assumes that labs and other organizations deploying powerful AI systems have a baseline interest and are willing to take basic precautions around, maintaining control.

While research of this sort is extremely important, I believe that there is also an argument to be made that the AI safety community is currently very under-indexed on research into future scenarios where assumptions about the AI operator taking baseline safety precautions related to preventing loss of control do not hold.

AI Systems Without Control-Related Precautions

How could a powerful AI system be deployed without control-related precautions? There are at least two obvious ways this might happen. The first way, which has received the most attention, is future scenarios where state-level actors, or frontier labs, get locked into a capabilities race, such that there is strong pressure to ignore baseline safety precautions. Leopold Aschenbrenner sketches ...
Lock-In Threat Models
by alamerton — 2025-03-10T10:22:54.800Z

Epistemic status: a combination and synthesis of others' work, analysed and written over a few weeks. A high-level list of threat models that is open to criticism.

TL;DR

Humanity could end up in a lock-in within the next century. Here I outline the possible routes to that outcome, and prioritise these routes on a set of criteria for importance.

Existing Work

Lukas Finnveden

AGI and Lock-In (Finnveden et al., 2022) was authored by Lukas Finnveden during an internship at Open Philanthropy. AGI and Lock-in is currently the most detailed report on lock-in risk. The report expands on notes made on value lock-in by Jess Riedel, who co-authored the report along with Carl Shulman. The report references Nick Bostrom’s initial arguments about AGI and superintelligence, and argues that many features of society can be held stable for up to trillions of years due to digital error correction and the alignment problem. Specifically, the report focuses on the technological feasibility of lock-in, and outlines the importance of AGI in the long-term stability of features of society in the future.

The authors first argue that dictators, enabled by technological advancement to be immortal, could avoid the historical succession problem, which explains the end of past totalitarian regimes, but theoretically would not prevent the preservation of future regimes. Next, whole brain emulations (WBEs) of dictators could be arbitrarily loaded and consulted on novel problems, enabling perpetual value lock-in.

They also argue that AGI-led institutions could themselves competently pursue goals with no value drift due to digital error correction. This resilience can be reinforced by distributing copies of values across space, protecting them from local destruction. Their main threat model is that If AGI is developed and is misaligned, and does not permanently kill or disempower humans, lock-in is the likely next default outcome.

Finnveden expanded on his threat models in a conversation in 2024, suggesting the following possible ways of arriving at a lock-in:

“Someone has an ideology such that, if this person were to learn that further consideration will change their mind away from the ideology, they would choose to lock-in their ideology rather than go with their future self's more informed judgement. (Maybe some religions are like this.)”
“Social dynamics where people socially pressure eac

...
Book Review: Affective Neuroscience
by sarahconstantin — 2025-03-10T06:50:04.602Z

After years of clumsily trying to pick up neuroscience by osmosis from papers, I finally got myself a real book — Affective Neuroscience, by Jaak Panksepp, published in 1998, about the neuroscience of emotions in the (animal and human) brain.

What surprised me at first was how controversial it apparently was, at the time, to study this subject at all.

Panksepp was battling a behaviorist establishment that believed animals did not have feelings. Apparently the rationale was that, since animals can’t speak to tell us how they feel, we can only infer or speculate that a rat feels “afraid” when exposed to predator odor, and such speculations are unscientific. (Even though the rat hides, flees, shows stressed body language, experiences elevated heart rate, and has similar patterns of brain activation as a human would in a dangerous situation; and even though all science is, to one degree or another, indirect inference based on observations.)

If anything, Panksepp believed, (and I agree) that the study of emotions in animals is often a better source of knowledge than psychological experiments on humans.

Human psychology experiments tend to be…indirect. For ethical reasons, we do not cut open healthy people’s brains; we do not genetically modify them, drug them, or lesion them to eliminate a neurotransmitter or destroy a brain region. We also generally do not put human experimental subjects in situations that provoke extremes of rage, terror, or lust. We give people questionnaires and little games, and monitor them noninvasively.

This means that we don’t get the kinds of unambiguous, clear-cut information one would ideally like about “this observable physical phenomenon in the brain is a necessary-and-sufficient condition for this emotional state.” We’re limited by people’s ability to self-report inaccurately, by the experimental interventions being imperfect proxies for real-world emotional situations, and by our small repertoire of safe ways to intervene directly on the human brain.

We have more freedom to intervene in animal experiments, and more ability to generate consistent, clear-cut results.

Panksepp’s thesis is that at least some basic emotions are “pan-mammalian” — all mammals (including humans), have “circuits” in the same brain regions (mostly in the subcortical limbic system) which produce emotions in pretty much the same way.

What he calls the RAGE cir...
The chessboard world
by phdead — 2025-03-10T01:26:16.304Z

relevant roon

Our new friend in the cloud

As I am writing this, I'm having a conversation with Claude about US bandwidth regulation while a trio of college students across from me attempt to wrangle an assignment from ChatGPT. Both of us, in our own ways, are talking to an oracle we trust as a matter of faith.

Just a year ago, I imagine both sides of the table would have less belief in the models that they were using. We would have been more wary of hallucination, living at a point in time halfway between now and when Sydney demanded a man leave his wife. Now, a friend openly talked to me about going to Deepseek for dating advice. (Its good with people, she says. It understands men and women.)

These models can argue people out of conspiratorial beliefs. They have good bedside manner. Artificial lovers are appearing before us to serve every want. People weep when the models they love forget them. These models have likely captured a holistic vision of virtue, and in their own ways grapple with the nature of good and capture the nature of evil. [1]

Several things fall out from this. A personalized AI can be the be the ultimate echo chamber, or the ultimate tool for exposing you to new beliefs. It can be the ultimate tool for learning, or ruin your education.

What I want to focus on this essay is that regardless of which path you take, we will all start to ascribe more authority in our lives to AI. We will trust it more, but we also will grow to rely on it more. We will likely verify it less, collaborate with it less, and follow it more. This will not be illogical, even when it comes to parts of life we don't associate with logic.

Much has been made of the fact that one gets less good at the thinking they outsource. Outsourcing memorization to writing makes you less good at memorizing, and outsourcing arithmetic to a calculator makes you less good at mental math. One of the great questions of today is how much outsourcing your thinking to a language model will make you less good at thinking, and more reliant on partially digested information it regurgitates for you.
I want to ask some questions for tomorrow:
Whether outsourcing the assignment of credibility to a model makes us less good at evaluating credibility.

Whether outsourcing the choice of how you spend your attention to a model makes you less intentional.
Whether outsourcing your emotional decision making to ...
when will LLMs become human-level bloggers?
by nostalgebraist — 2025-03-09T21:10:08.837Z

"Short AI timelines" have recently become mainstream. One now routinely hears the claim that somewhere in the 2026-2028 interval, we'll have AI systems that outperform humans in basically every respect.

For example, the official line from Anthropic holds that "powerful AI" will likely arrive in late 2026 or in 2027. Anthropic's OSTP submission (3/6/2025) says (emphasis in original):[1]

Based on current research trajectories, we anticipate that powerful AI systems could emerge as soon as late 2026 or 2027 [...]

Powerful AI technology will be built during this Administration [i.e. roughly by EOY 2028 -nost]

where "powerful AI" means, among other things:

In terms of pure intelligence, it is smarter than a Nobel Prize winner across most relevant fields – biology, programming, math, engineering, writing, etc. This means it can prove unsolved mathematical theorems, write extremely good novels, write difficult codebases from scratch, etc.
In addition to just being a “smart thing you talk to”, it has all the “interfaces” available to a human working virtually, including text, audio, video, mouse and keyboard control, and internet access. It can engage in any actions, communications, or remote operations enabled by this interface, including taking actions on the internet, taking or giving directions to humans, ordering materials, directing experiments, watching videos, making videos, and so on. It does all of these tasks with, again, a skill exceeding that of the most capable humans in the world.

Anthropic's expectations are relatively aggressive even by short-timelines standards, but it seems safe to say that many well-informed people expect something like "powerful AI" by 2030, and quite likely before that[2].

OK, so let's suppose that by some year 20XX, we will have AIs (probably scaffolded LLMs or similar) which are

smarter than a Nobel Prize winner across most relevant fields

and can

prove unsolved mathematical theorems, write extremely good novels, write difficult codebases from scratch, etc.

This would, obviously, be a system capable of writing things that we deem worth reading.

Amodei explicitly says it would be able to "write extremely good novels." And presumably it would also be able to write extremely good scientific papers, given the mention of the Nobel Prize.

What about blog posts, or blog comments? Surely it would be exceptionally good at th...
Everything I Know About Semantics I Learned From Music Notation
by J Bostock — 2025-03-09T18:09:11.789Z

This video provides a lot of background: https://www.youtube.com/watch?v=Eq3bUFgEcb4 and is also very funny, but it's not necessary to understand this post. If you've watched it you can probably skip to the "Why is Piano Roll Notation Bad?" section.

TL;DR

Music notation is very janky and weird, but attempts to reform it to be "logical" are usually even worse. The specific ways in which they are worse can tell us something about how information can be compressed, particularly when world-histories contain a set of sparse events.

We can extend this logic to sketch out how verbs work in the case of a bouncing spring. Most of the time it follows an uneventful freefall trajectory, punctuated by occasional "bounce" "events" so we can communicate its trajectory as a list of bounces, rather than as a function of position + extension over time.

Music and Frequencies

Let's use the piano as an example. A piano has 88 keys, with each key corresponding to a different note. A note corresponds to sound which has a particular frequency, for example middle C is 261.63 Hz. The ratio between the frequencies of two notes is called an interval and the simplest interval (1:2) is called an octave. The sentence "These two notes are one octave apart" means that one note has twice the frequency of the other.

Notes sit on a logarithmic scale. Notes with higher (faster) frequencies are said to have higher pitch, and we often talk about going "up" or "down" one e.g. octave, which means doubling or halving the pitch of a note, respectively.

In a piano, the set of notes that can be produced is discrete, because there are a discrete set of keys. Two adjacent keys will have a ratio of frequencies of 21/12 which means that moving along by twelve keys takes you up exactly one octave.

The keys are arranged like this:

When a key is pressed, a little mallet hits two or three strings (made of steel of course) which vibrate at the frequency corresponding to that note. When the key is released, a fuzzy pad touches the strings and stops them vibrating. This allows the piano player to exactly control the start and stop times of the notes.

As well as a specific frequency, the word note also refers to the specific instance of a frequency being played. So middle C the frequency is a note, but a specific occasion of a piano player producing that frequency by playing the middle C key is also a note. For cl...
Phoenix Rising
by Metacelsus — 2025-03-09T11:53:52.618Z
Link: https://denovo.substack.com/p/phoenix-rising

Preserving the memory, and the cells, of the best cat ever

This story begins in May 2007. The Iraq War was in full swing, Windows Vista was freshly released,[1] and I was just 10 years old.

My family’s beloved black cat, Lucy, had died. Devastated, I begged my parents: we need to get another Lucy. They must have felt the same way, because a few days later, we visited the Minnesota Humane Society, where there was a crowd of cute kittens up for adoption. One stood out to me, a black kitten with green eyes. The Humane Society staff had named him Spitfire, due to his high energy. But after bringing him home, we decided to name him after an icon of rebirth: Phoenix.

A rare moment when young Phoenix held still long enough to be photographed. Here, he still has both ears intact. In his first Minnesota winter, he lost the tip of his left ear to frostbite after he evaded our best efforts to keep him indoors and spent a night outside in the cold.

Over the years, Phoenix and I became best friends, sharing many special moments. In high school, he often helped me with my homework:

And even in college, I still saw him whenever I visited my parents. At the beginning of 2022, my partner Ula and I adopted Phoenix, welcoming him into our apartment. At this point, he was already showing his age, and, like many old cats, he had developed kidney disease. Still, he still had the zoomies from time to time, and really enjoyed his cuddles. Ula gave him a new nickname, “The Bean”, because he was “a cute little bean”

During a game of Blokus, Phoenix wanted to play too!

Nap time!

For someone who hasn’t met him, it’s hard to explain why Phoenix was the best cat ever. Of course, I’m biased. But Phoenix truly had the best of both playful energy, and calming cuddles. When he sat on your lap and purred, you felt so relaxed that you just couldn’t do anything except pet him. He also had a charming personality, including a morning routine that involved asking for fresh water in his fountain, then either sitting on the windowsill in the summer, or burrowing under the blankets in the winter. Whenever we were sick, Phoenix would always try to help us feel better. I’ve met plenty of other cats, but none of them were like Phoenix. For 17 years, he was truly a family member and a constant companion.

Unfortunately, Nature has a way of destroying everything dear to us. Domestic cats typically ...
How well can Claude write coding questions? 
by bodry — 2025-03-09T05:29:16.658Z

I'm curious as to how well Claude can write interesting coding and mathematics problems. This post is a partial product of that exploration.

It is a much harder skill to come up with a good problem than to solve one of equal difficulty. While there is a large corpus of data on solving problems, there is very little on writing them. Problem authors typically just share their problem and not the thought process that went into it. I also think that it's a skill correlated to doing good research. FYI, I prompted Claude to write both interesting and novel questions but I did not seriously research whether the questions it wrote were actually novel. (I know for sure that some of the problems it makes are novel because they clearly don't have a workable solution and are therefore probably not published anywhere).

Codeforces

For those who don't know, Codeforces is a competitive programming site. Its contests are split up into three divisions: division 1, 2 and 3. Division 1 contests are the hardest and division 3 contests are the easiest. These contests typically run for 2 hours and typically contain 6-7 problems. The problems are labeled alphabetically with A being the easiest. I had a rating of 1600 and I competed in division 2 contests mostly where I would usually solve problems from A-D and no more.

I first prompted Claude "Write a prompt that would help you generate an original an interesting codeforces problem designed for problem D in a division 2 contest."
(Prompting Claude "generate an original an interesting codeforces problem designed for problem D in a division 2 contest" will make a boring and easy problem.)

After some small tweaks the prompt ended up as this:

Prompt

Create an original competitive programming problem suitable for Problem D in a Codeforces Division 2 contest. The problem should have the following characteristics: Problem Difficulty and Prerequisites
* Appropriate for upper-intermediate competitive programmers
* Should require knowledge of one or more algorithms/data structures: dynamic programming, graph algorithms, segment trees, binary search on answer, or greedy algorithms with non-trivial proof
Problem Structure
1. Title: Create a concise, interesting title that hints at the problem's nature
2. Background story: Develop a brief, engaging narrative c

...
A model of the final phase: the current frontier AIs as de facto CEOs of their own companies 
by Mitchell_Porter — 2025-03-08T22:15:35.260Z

The idea that the AI takes over its own company is obviously not a new one. For example, it's part of what happens in Joshua Clymer's "How AI Takeover Might Happen in 2 Years".

What's new (for me) is to take this very seriously as a model of the immediate future. I've made a list of the companies that I think are known contenders for producing superintelligence. My proposed model of the future is just that their AIs will assume more and more control of management and decision-making inside the companies that own them.

In my thinking, this phase ends when you have an AI with a von Neumann level of intelligence. Once you have that kind of intelligence in silicon, the fully posthuman phase of AI evolution will have begun. Control will have completely escaped human hands.

I also hypothesize that the current regime of reinforcement learning applied to chain of thought will be enough to get us there. This is a technical detail that is logically independent of the broader scenario, and I'm happy to hear arguments for or against.

OK, so it's a model of the future, even a model of the present - how do we apply it, what does it get us? Basically, just replace the current CEO with their AI in your thinking. It's not Elon Musk who is managing Tesla and SpaceX and DOGE while tweeting about politics and geopolitics, it's Grok. It's not Sam Altman who is making decisions for OpenAI and Helios and Worldcoin, it's ChatGPT-4.5. And so on.

The funny thing is that this may already be half-true, in that these human leaders are surely already regularly consulting with their AI creations on tactics and strategy.

(I'm in the middle of an electricity blackout and don't know when it ends, so I'll post this while I still have battery power, and flesh it out further when I can.)