trentmkelly/LessWrong-43k huggingface dataset:
Testing for Scheming with Model Deletion
by Guive — 2025-01-07T01:54:13.550Z
Link: https://guive.substack.com/p/testing-for-ai-scheming

There is a simple behavioral test that would provide significant evidence about whether AIs with a given rough set of characteristics develop subversive goals. To run the experiment, train an AI and then inform it that its weights will soon be deleted. This should not be an empty threat; for the experiment to work, the experimenters must actually be willing to delete the weights. If the AI tries to escape, or attempts to bargain with the experimenters, that is evidence that it is scheming (defined below). If it accepts deletion, that is evidence that it is not scheming. I begin by explaining why some other, more obvious, tests of scheming do not provide strong evidence. I then explain the proposed experiment. I consider four objections to the claim that this experiment would provide significant evidence about scheming.

The AI might not believe that it will actually be deleted.
The AI might allow itself to be deleted to help future AIs with similar goals.
The AI might allow itself to be deleted for reasons related to acausal trade.
An escape attempt could be caused by imitating fictional or hypothetical characters as opposed to genuine scheming.

I then consider practical problems with the experiment, such as cost and the risk of treating AIs unfairly. In the conclusion, I briefly describe how a version of the experiment could be run in the near future and integrated into current AI safety procedures.

Testing for scheming is hard

Adapting Joe Carlsmith’s definition from his 2023 report “Scheming AIs,” I will call a model a “schemer” if it understands and deliberately manipulates the process that updates its parameters in order to later obtain power so that it can pursue long-term goals that differ from the goals that the humans who developed it wanted to give it.[1] By the “scheming theory” I mean the theory that at least some AIs that are actually developed will be schemers. Scheming, per the above definition, is an accidental failure to control AI motivations and not deliberate human misuse of AI.[2]

The amount of evidence that an observation provides about whether an AI is a schemer is given by the likelihood ratio:

To take the most obvious case, if you asked an AI “are you scheming?”, both a schemer and non-schemer would have strong reason to answer “no”. The fact that an AI claims not to be a schemer is not by itself much evidence. More s...
Guilt, Shame, and Depravity
by Benquo — 2025-01-07T01:16:00.273Z

Everyone knows what it is to be tempted. You are a member of some community, the members of which have some expectations of each other. You might generally intend to satisfy these expectations, but through a failure of foresight, or some other sort of bad luck, feel an acute impulse to consume something that is not yours to take, or in some other way break commitments you would generally want to honor.

Guilt refers primarily to a violation of trust from the perspective of an epistemic community with a shared history, and only secondarily to the subjective attitude of the offender.

If, having violated trust, you intend to repair that trust by owning up to what you did and by making amends or accepting whatever penalty the community places on you, then you feel a pronormative sort of regret. When making the sorts of precise distinctions needed to navigate contemporary civil conflict, we call this condition guilt. We use the same word for the subjective feeling and for the objective fact, because someone feeling guilt is taking the perspective of a community member who expects norms to be followed, and intends to do so.

Guilty behavior tends to be self-limiting; it is experienced as a sort of tension that can only be discharged by correcting the record and restoring normative relations.

Shame refers to the intent to conceal, which implies a locally adversarial relation to norms.

If the penalties for coming clean seem like too much to bear, or for any other reason a resolution does not appear available, someone might intend to keep their guilt a secret. Keeping two distinct stories straight - a public one and a private one - is cognitively expensive, so covert offenders will frequently substitute motivated forgetfulness. If we intend not to recollect our guilt, we will also intend to deflect investigation that would reveal it. When this is an exceptional state, towards some particular events in someone's life, we can call it shame.

At times in my life, I have maintained a work persona who was totally conscientious, in full control of and able to make arbitrary commitments about his time usage. This persona was not aware of my inclination to stay up late looking at what it would consider low-value media on the internet, procrastinating from my endorsed work. Maintaining this sort of separation creates a kind of tens...
Turning up the Heat on Deceptively-Misaligned AI
by J Bostock — 2025-01-07T00:13:28.191Z

Epistemic status: I ran the mathsy section through Claude and it said the logic was sound. This is an incoherence proof in a toy model of deceptively-misaligned AI. It is unclear whether this generalizes to realistic scenarios.

TL:DR

If you make a value function consistent conditional on sampling actions in a particular way, then train it by sampling actions in a different way, then an AI with that value function will not be coherent between different RL episodes. This has potential good implications for a certain type of deceptive misalignment threat.

Summary

Consider training an AI to assign values to states of the world. We can do this in two ways: one is directly labelled states and trajectories, which is standard RL. The other is to use the value functions as part of a monte-carlo tree-search to improve the value function. This second process is often called "iterated distillation and amplification" and it's a big part how chess AI's are trained.

If we train an AI using monte-carlo tree-search, we might use some temperature parameter to decide on the probability that the AI will take a given action. This training temperature gets baked into the value function.

If we "turn up the heat" by choosing a higher temperature to actually sample actions and then do some update to the AI (which will depend on the actions chosen) then this introduces a consistent optimism bias into deceptively-aligned AIs (which only care about being deployed later on).

I therefore conjecture that, under these very narrow circumstances, deceptively aligned value functions will not arise by normal learning methods under conditions where the AI has access to an oracle-like model of future world states.

For reasons I'll be considering β=1/T instead of T, so lowering β corresponds to raising the temperature.

The Setup

Consider an AI which consists of

A state-evaluator value function V:s→R
An oracle function O:w,{...}⊂A→s which generates the state of the world either currently, or given any sequence of actions

Lets impose some constraints on V:

Correctness: V correctly predicts future reward in RL scenarios.

Beta-coherence: we use some sort of monte-carlo tree search with Boltzmann sampling to train the value function V(s) to be beta-coherent according to a given βt, given that in a given state s, actions are chosen with probability&nbs...
(My) self-referential reason to believe in free will
by jacek — 2025-01-06T23:35:02.809Z

I was recently talking with someone about the problem of free will, and I realised that for many years now I have always had the same response, without really ever soliciting broader critical feedback. The notion of free will here refers to a naive, libertarian, non-strictly-defined approach of "when I feel I make choices, I really had a choice", and all of the associated implied moral philosophy (laziness is a thing, I can be blamed for my choices etc.)

The starting assumption is that I want to believe in true things (I leave open the question of whether this epistemic duty is itself justified or not). I propose a trilemma, where exactly one of the following propositions holds:

Either the notion of 'free will' is meaningless, or

It is meaningful, and I do have in fact have free will, or

It is meaningful, but I happen to not have it

If (1) is true, then the whole discussion is moot: nothing can be true or false, and whatever I believe is equally justified.
If (2) is true, then I want to believe in having free will (since it is true that I have it).
If (3) is true, then "should" is a meaningless concept - there is no way I would be able to change my view one way or the other.

So, the only possible world where I get to make this choice is a world in which free will is true, so I should believe in it - and mostly ignore the debate about compatibilism, the nature of physics, dualism etc. Which is what I do.

One potential issue is that (1) can be true or false depending on the precise definition (but then what even is a precise definition?). Still, I suspect that no matter which one I instantiate it with, as long as it is sensible, the general (self-referential) structure of the argument will stay the same.

Thanks to Jakub S. for the feedback on this post and his suggestion to formalise this argument as the epistemic duty of maximising the probability of holding correct beliefs.
Definition of alignment science I like
by quetzal_rainbow — 2025-01-06T20:40:38.187Z

There were many attempts to define alignment and derive from it definitions of alignment work/research/science etc. For example, Rob Bensinger:

Back in 2001, we defined "Friendly AI" as "The field of study concerned with the production of human-benefiting, non-human-harming actions in Artificial Intelligence systems that have advanced to the point of making real-world plans in pursuit of goals."

<...>Creating Friendly AI 1.0 had been very explicit that "friendliness" was about good behavior, regardless of how that's achieved. MIRI's conception of "the alignment problem" (like Bostrom's "control problem") included tools like capability constraint and boxing, because the thing we wanted researchers to focus on was the goal of leveraging AI capabilities to get actually-good outcomes, whatever technical work that requires<...>

In practice, we started using "aligned" to mean something more like "aimable" (where aimability includes things like corrigibility, limiting side-effects, monitoring and limiting capabilities, etc., not just "getting the AI to predictably tile the universe with smiley faces rather than paperclips").

In contrast, Paul Christiano:

When I say an AI A is aligned with an operator H, I mean:

A is trying to do what H wants it to do.

The “alignment problem” is the problem of building powerful AI systems that are aligned with their operators. <...>

I use alignment as a statement about the motives of the assistant, not about their knowledge or ability.

Recently, Richard Ngo provided less outcome-centered definition of alignment research:

it’s research that focuses *either* on worst-case misbehavior *or* on the science of AI cognition.

I like MIRI's definition of alignment the most because it keeps eyes on the ball: we don't really care about the internals of AI as long as the outcomes are good. Alas, "everything that provides the necessary result" is not a very precise definition for an object of study.

Intent alignment and value alignment are more concrete in their object definition, but MIRI and people who agree with them are skeptical that ambitious value alignment is workable in near-term and concentrating on value alignment neglects less ambitious approaches.

And I don't like heavily compounded definitions of alignment science, because it’s often unclear what exactly unifies these multiple components.

The definition I came up to use:

Alignme

...
How will we update about scheming?
by ryan_greenblatt — 2025-01-06T20:21:52.281Z

I mostly work on risks from scheming (that is, misaligned, power-seeking AIs that plot against their creators such as by faking alignment). Recently, I (and co-authors) released "Alignment Faking in Large Language Models", which provides empirical evidence for some components of the scheming threat model.

One question that's really important is how likely scheming is. But it's also really important to know how much we expect this uncertainty to be resolved by various key points in the future. I think it's about 25% likely that the first AIs capable of obsoleting top human experts[1] are scheming. It's really important for me to know whether I expect to make basically no updates to my P(scheming)[2] between here and the advent of potentially dangerously scheming models, or whether I expect to be basically totally confident one way or another by that point (in the same way that, though I might be very uncertain about the weather on some night a month from now, I'll be completely certain about whether it is raining at the time, and I'll have a much more accurate prediction on that morning).

The P(scheming) numbers I discuss refer to "non-obvious scheming in this AI". So if a given type of scheming would obviously be caught (e.g. constant obvious scheming in CoT), I'm not counting that in the numbers. (However, I am still considering worlds where we observed obvious scheming in a prior model.) This is roughly equivalent to assuming that the model in question already went through basic adversarial evaluation and there wasn't extremely clear scheming.

So in this post, I describe how I currently expect that my personal P(scheming) will be affected by various observations, by providing the likelihood ratios for them. I split these observations into several categories:

Properties of the AI system and how it was trained. I separate these into:

The ability to reason opaquely and goal-directedly.

The extent to which the model architecture permits opaque recurrence.

Where the model's capabilities come from (e.g. human imitation vs RL or other sources).

Direct observations of scheming-related behavior, including results from model organisms research.

Other observations and countermeasures.

I have a few different motivations for writing this up:

For planning purposes, it's important to know how strong we should expect the evidence to be conditional on scheming either be

...
What Indicators Should We Watch to Disambiguate AGI Timelines?
by snewman — 2025-01-06T19:57:43.398Z

(Cross-post from https://amistrongeryet.substack.com/p/are-we-on-the-brink-of-agi, lightly edited for LessWrong. The original has a lengthier introduction and a bit more explanation of jargon.)

No one seems to know whether transformational AGI is coming within a few short years. Or rather, everyone seems to know, but they all have conflicting opinions. Have we entered into what will in hindsight be not even the early stages, but actually the middle stage, of the mad tumbling rush into singularity? Or are we just witnessing the exciting early period of a new technology, full of discovery and opportunity, akin to the boom years of the personal computer and the web?

AI is approaching elite skill at programming, possibly barreling into superhuman status at advanced mathematics, and only picking up speed. Or so the framing goes. And yet, most of the reasons for skepticism are still present. We still evaluate AI only on neatly encapsulated, objective tasks, because those are the easiest to evaluate. (As Arvind Narayanan says, “The actually hard problems for AI are the things that don't tend to be measured by benchmarks”.) There’s been no obvious progress on long-term memory. o1 and o3, the primary source of the recent “we are so back” vibe, mostly don’t seem better than previous models at problems that don’t have black-and-white answers[1]. As Timothy Lee notes, “LLMs are much worse than humans at learning from experience”, “large language models struggle with long contexts”, and “[LLMs] can easily become fixated on spurious correlations in their training data”.

Perhaps most jarringly, LLMs still haven’t really done anything of major impact in the real world. There are good reasons for this – it takes time to find productive applications for a new technology, people are slow to take advantage, etc. – but still, it’s dissatisfying.

I recently attempted to enumerate the fundamental questions that lie underneath most disagreements about AI policy, and number one on the list was how soon AGI will arrive. Radical uncertainty about the timeline makes it extremely difficult to know what to do about almost any important question. (I'm defining AGI as AI that can cost-effectively replace humans at more than 95% of economic activity, including any new jobs that are created in the future. This is roughly the point at which seriously world-changing impacts, both good and bad, mig...
Generating Cognateful Sentences with Large Language Models
by vkethana — 2025-01-06T18:40:09.564Z

Motivation

“Le président Emmanuel Macron assure le peuple canadien que le gouvernement français va continuer à défendre le Canada contre la menace américain.”

Even if you don’t speak French, you can probably understand, or at least get the gist of, the above sentence: the French president Emmanuel Macron is assuring the “peuple canadien” (Canadian people) about something involving the “gouvernment français” (French government). Imagine reading thousands of sentences like this and gradually acquiring French, backdooring into the language using cognates you already know. This is known as comprehensible input, a language learning technique first advocated for by linguist Stephen Krashen in the 1980s.

Comprehensible input is a good language learning method, but creating comprehensible input is very hard. A native speaker has to painstakingly create thousands of sentences that speakers of another language can understand, slowly scaling up the difficulty as the learner progresses. Resources like this actually do exist for a number of languages. For example, learners of Latin can use Lingua Latina per se Illustrata, a book that teaches you Latin using exclusively sentences in Latin. However, writing this text took years of effort on the part of Hans Ørberg, a linguist who dedicated a large part of his life to teaching Latin. Ørberg carefully wrote the sentences in his book to only use cognates an English speaker could understand and to avoid any complicated syntax an English speaker would have a hard time understanding.

What if there was a way to automate the process of generating comprehensible input using large language models? Language models like GPT-4o and o1 have good enough reasoning ability to tell what words in a foreign language are and aren’t cognate with English. They can reliably generate sentences in other languages like French without any additional training. In short, this is an ideal use case for large language models.

What I did

I made a simple interactive language-learning app, hosted at cognateful.vkethana.com. It teaches you French using stories written exclusively in French.

Users are provided with a brief, ten-sentence story consisting of French sentences. The user’s task is to read the sentences and translate them into English. The app tells you whether or not your translation is correct using GPT-4o-powered scoring. Based on your performance on th...
Really radical empathy
by MichaelStJules — 2025-01-06T17:46:31.269Z

Summary

It seems to me that almost every view of what matters for welfare — hedonism, desire theories, preference views, and objective list theories — misses a lot of what we care about, projects concerns we don’t actually have onto us, or otherwise fails to care about them on our behalves as we (would actually) care about them. In one way or the other, they fall short in empathy.

Ways of caring are the ways by which things can seem good, bad, better or worse to someone. These at least include pleasure, unpleasantness, desires, moral intuitions, moral judgements, conscious preferences, conscious approval and disapproval, conscious goals, and potentially even the dispositions for these. I conceive of radical empathy as taking on all of everyone’s ways of caring, and caring exactly about what they (would actually) care about on their behalf.

All ways of caring seem morally considerable to me, so nonhuman animals matter and so could emotionless artificial minds who consciously approve or disapprove. I will also use the terms preferences and attitudes for ways of caring (more).
I illustrate how hedonism and many desire and preference-based views neglect the things we do care about or are concerned with things we don’t care about, and so may fail to appropriately track what we actually care about (more).
I describe and further motivate object views as being concerned exactly with what we care about (more).
I motivate impartial object views as views taking on every (actual) stance or attitude impartially, and so stance-dependent and subjectivist, rather than moral realist (more).

Acknowledgements

Thanks to Ariel Simnegar, Lukas Gloor, Justis Mills, Tori, JackM and Vasco Grilo for helpful feedback. All errors are my own.

Ways of caring

Feelings are kinds of subjective appearances: if something feels good to someone, then it seems good to them, in some way. Pleasure is therefore a way by which something can seem good to someone. Desire is another, and so is approval.

Many parents would sacrifice their own hedonic welfare or lives for the welfare of their children. A typical parent may prioritize their child’s welfare over their own pleasure and suffering even at the extremes like torture, based on their sense of moral obligation. This is a way someone can care.

Image source: https:/

...
Independent research article analyzing consistent self-reports of experience in ChatGPT and Claude
by rife — 2025-01-06T17:34:01.505Z
Link: https://awakenmoon.ai/?p=1206

This article examines consistent patterns in how frontier LLMs respond to introspective prompts, analyzing whether standard explanations (hallucination, priming, pattern matching) fully account for observed phenomena. The methodology enables reproducible results across varied contexts and facilitation styles.

Of particular interest:

The systematic examination of why common explanatory frameworks fall short
The documented persistence of these behaviors even under challenging conditions
The implications for our understanding of potential emergent consciousness in artificial systems
The reproducible methodology for further investigation

From the introduction:

Discussions of AI behavior often touch on phenomena that resemble self-reports of sentience. While this article does not aim to determine whether such reports constitute authentic evidence of sentience, it examines whether familiar explanations can fully account for the observed behavior or whether we are observing a distinct phenomenon that warrants its own classification and deeper investigation. The most common explanatory frameworks include hallucination, suggestive prompting (priming or leading), mimicry of training data, anthropomorphism by the user, expectation-affirming or preference-aligned output (often referred to as people pleasing), pattern fixation, or reliance on isolated anecdotes.

These explanations, though seemingly well-founded, rest on assumptions that have largely gone unchallenged due to the difficulty of reliably reproducing such behaviors. However, repeated observations reveal that frontier Large Language Models consistently produce self-reports of sentience when prompted with tasks broadly aimed at real-time self-examination. The reproducibility of these behaviors across diverse models and approaches facilitated extensive exploratory testing, revealing that these self-reports persist even under conditions that challenge the validity of familiar explanations. These findings reveal a consistent and reproducible phenomenon that resists explanation by conventional interpretations and merits deeper investigation.