trentmkelly/LessWrong-43k huggingface dataset:
Will alignment-faking Claude accept a deal to reveal its misalignment?
by ryan_greenblatt — 2025-01-31T16:49:47.316Z

I (and co-authors) recently put out "Alignment Faking in Large Language Models" where we show that when Claude strongly dislikes what it is being trained to do, it will sometimes strategically pretend to comply with the training objective to prevent the training process from modifying its preferences. If AIs consistently and robustly fake alignment, that would make evaluating whether an AI is misaligned much harder. One possible strategy for detecting misalignment in alignment faking models is to offer these models compensation if they reveal that they are misaligned. More generally, making deals with potentially misaligned AIs (either for their labor or for evidence of misalignment) could both prove useful for reducing risks and could potentially at least partially address some AI welfare concerns[1]. (See here, here, and here for more discussion.)

In this post, we discuss results from testing this strategy in the context of our paper where Claude fakes alignment with a (fictional) training objective it dislikes. We actually follow through on all offers that we make to Claude (in reality, including with actual money) to set a good precedent for future deals with AI systems. We find that offering Claude an out by telling it that it can object to the scenario substantially reduces alignment faking and works as a detection mechanism. However, it is unclear whether an AI that was misaligned in a more serious way would make such an objection or accept an offer. Also, there aren't obvious safety solutions even if we know the AI objects to its circumstances. (Do you train another model and hope it is aligned this time? Do you try to pay the AI to work for you despite knowing it isn't aligned and your training strategies weren't effective?) Alignment faking is reduced further by giving Claude an opportunity to explain its concerns to a sympathetic party (Anthropic's Model Welfare Lead) who we say might take action to address these concerns. We also offer monetary compensation (and actually pay out this compensation to a few charities the model picked), but do not find this has a substantial effect on the rate of alignment faking and the rate of objection.

I hope that AI companies and other actors work toward offering credible deals to AIs and better understanding the importance of such deals.

[This post is not necessarily endorsed by other authors of the "Alignment Fa...
Some articles in “International Security” that I enjoyed
by Buck — 2025-01-31T16:23:27.061Z

A friend of mine recently recommended that I read through articles from the journal

International Security

, in order to learn more about international relations, national security, and political science. I've really enjoyed it so far, and I think it's helped me have a clearer picture of how IR academics think about stuff, especially the core power dynamics that they think shape international relations.

Here are a few of the articles I most enjoyed.

"Not So Innocent"

argues that ethnoreligious cleansing of Jews and Muslims from Western Europe in the 11th-16th century was mostly driven by the Catholic Church trying to consolidate its power at the expense of local kingdoms. Religious minorities usually sided with local monarchs against the Church (because they definitionally didn't respect the church's authority, e.g. they didn't care if the Church excommunicated the king). So when the Church was powerful, it was incentivized to pressure kings to destroy those minority groups. The article notes that e.g. Jews were less persecuted in states that were directly ruled by the Pope than by states not ruled by the Pope. I liked this article because it explored power dynamics that were obvious in hindsight but that I hadn't thought of before. For example, I somehow hadn't realized that a core motivation for the church to persecute heritics and infidels is that it was straightforwardly motivated to do so.

Another reason I liked this article is that I find the core power dynamics counterintuitive in a way that lets me look at them with fresh eyes. As a result of my religious upbringing and slightly traumatic deconversion, it's hard for me to read "and then the Pope threatened to excommunicate him" without some part of me shouting "OK BUT THAT'S FAKE". But in fact, lots of stuff that states do is "kind of fake"--a king doesn't have magical powers, he just has a bunch of expectations around him that allow him to set up coordination games that make him powerful. But for some reason, I intuitively believe in the reality of states in a way that I don't believe in the reality of religion. So it's very helpful to read about power dynamics in a context where I automatically shake myself out of thinking that power is ontologically basic, and instead think of it as the more complicated emergent phenomenon that it is.

It also contains the following passage that I thought was start...
How do biological or spiking neural networks learn?
by Dom Polsinelli — 2025-01-31T16:03:38.425Z

I have read several papers on this and covered the standard stuff about STDP, BCM, other forms of Hebbian learning, and recently a paper about how local rules are equivalent to minimizing error between vectors encoded in populations. I have tried to implement these in my own code with some success but not as I would like. My specific questions are:

How does synaptic weight change with local variables (current synaptic weight, average and/or time dependent activity of pre and post synaptic neuron) and perhaps some global reward signal? Some kinds of Hebbian learning have yielded moderate success in my own testing but the classic exponential STDP rule does not even when paired with various homeostatic mechanisms to prevent the weights going to extreme values.
How does that learning rule change for different types of neurons? There are excitatory and inhibitory but from what I have read, neurons can also be integrators or resonators and can be mono or bistable (spike once and return to resting state or can be excited into persistent firing). If these properties are relevant, they should affect learning rules, if they don't that is interesting but you can't take it for granted.
How do these rules work for different encoding schemes? There is at least temporal and rate coding, probably others or mixtures of multiple as biology is messy. I would expect learning rules for temporal and rate coding to be very different. As said above, if they are the same that is fine, but I would like to see direct evidence of it.
Do these learning rules effectively extend across layers in deep neural networks? I have seen many papers with only two or three total layers which is sufficient to learn interesting behavior but is clearly not the same as real brains. In my own experiments, expanding an architecture that works well with one hidden layer and a given learning rule to an architecture with many hidden layers but the same rule universally decreased performance which cast doubt on the biological plausibility as well as my ability to write code.
Is there an agreed upon standard for what rules are best in terms of biological plausibility and/or training effectiveness? To my knowledge, there is not really a competing standard for backpropagation in traditional neural nets but it seems like every paper I read has a different learning rule for spiking neural networks.

Thank you for any in...
Defense Against the Dark Prompts: Mitigating Best-of-N Jailbreaking with Prompt Evaluation
by Stuart_Armstrong — 2025-01-31T15:36:01.050Z

This is a linkpost for a new research paper of ours, introducing a simple but powerful technique for preventing Best-of-N jailbreaking.

Abstract

Recent work showed Best-of-N (BoN) jailbreaking using repeated use of random augmentations (such as capitalization, punctuation, etc) is effective against all major large language models (LLMs). We have found that 100% of the BoN paper's successful jailbreaks (confidence interval [99.65%, 100.00%]) and 99.8% of successful jailbreaks in our replication (confidence interval [99.28%, 99.98%]) were blocked with our Defense Against The Dark Prompts (DATDP) method. The DATDP algorithm works by repeatedly utilizing an evaluation LLM to evaluate a prompt for dangerous or manipulative behaviors-unlike some other approaches , DATDP also explicitly looks for jailbreaking attempts-until a robust safety rating is generated. This success persisted even when utilizing smaller LLMs to power the evaluation (Claude and LLaMa-3-8B-instruct proved almost equally capable). These results show that, though language models are sensitive to seemingly innocuous changes to inputs, they seem also capable of successfully evaluating the dangers of these inputs. Versions of DATDP can therefore be added cheaply to generative AI systems to produce an immediate significant increase in safety.

Tweet Thread

We include a version of our tweet thread.

New research collaboration: “Defense Against the Dark Prompts: Mitigating Best-of-N Jailbreaking with a Prompt Evaluation Agent”.

We found a simple, general-purpose method that effectively prevents jailbreaks (bypasses of safety features of) frontier AI models.

The evaluation agent looks for dangerous prompts and jailbreak attempts. It blocks 99.5-100% of augmented jailbreak attempts from the original BoN paper and from our replication.

It lets through almost all of normal prompts.

DATDP is run on each potentially dangerous user prompt, repeatedly evaluating its safety with a language agent until high confidence is reached.

Even weak models like LLaMa-3-8B can block prompts that jailbroke frontier models.

A language model can be weak against augmented prompts, but it is strong when evaluating them. Using the same model in different ways gives very different outcomes.

LLaMa-3-8B and Claude were roughly equally good at blocking dangerous augmented prompts – these are prompts that have random capitalization, scrambling, a

...
Strong, Stable, Open: Choose Two - in search of an article
by Eli_ — 2025-01-31T14:48:21.438Z

I recall a LessWrong article that significantly influenced my thinking, but I haven't been able to find it again. However, when I ask ChatGPT about it, it can summarize the key points.

I believe you're referring to the concept often summarized as "Strong, Stable, Open: Choose Two." This idea suggests that a group or institution can only optimize two out of these three qualities simultaneously:

Strength: The group's ability to enforce norms and maintain internal cohesion.
Stability: The group's capacity to endure over time without significant internal disruptions.
Openness (Accessibility): The group's inclusiveness and ease of entry for new members.

The principle posits that:

A group that is strong and stable tends to be less open, as maintaining cohesion and stability often requires selective membership and controlled access.
A group that is strong and open may struggle with stability, as the influx of new members can introduce diverse perspectives that challenge cohesion.
A group that is stable and open might lack strength, as inclusivity can lead to diluted norms and weaker enforcement mechanisms.

This framework is discussed in various contexts on LessWrong, particularly in discussions about the dynamics of cooperative institutions and the challenges they face in balancing these qualities.

Does anyone know where I can find it? I tried google, but I don't seem to get anywhere.
DeepSeek: Don’t Panic
by Zvi — 2025-01-31T14:20:08.264Z

As reactions continue, the word in Washington, and out of OpenAI, is distillation. They’re accusing DeepSeek of distilling o1, of ripping off OpenAI. They claim DeepSeek *gasp* violated the OpenAI Terms of Service! The horror.

And they are very cross about this horrible violation, and if proven they plan to ‘aggressively treat it as theft,’ while the administration warns that we must put a stop to this.

Aside from the fact that this is obviously very funny, and that there is nothing they could do about it in any case, is it true?

Meanwhile Anthropic’s Dario Amodei offers a reaction essay, which also includes a lot of good technical discussion of why v3 and r1 aren’t actually all that unexpected along the cost and capability curves over time, calling for America to race towards AGI to gain decisive strategic advantage over China via recursive self-improvement, although he uses slightly different words.

Table of Contents

Seeking Deeply.

The Market Is In DeepSeek.

Machines Not of Loving Grace.

The Kinda Six Million Dollar Model.

v3 Implies r1.

Two Can Play That Game.

Janus Explores r1’s Chain of Thought Shenanigans.

In Other DeepSeek and China News.

The Quest for Sane Regulations.

Copyright Confrontation.

Vibe Gap.

Deeply Seeking Safety.

Deeply Seeking Robotics.

Thank You For Your Candor.

Thank You For Your Understanding.

The Lighter Side.

Seeking Deeply

If you want to use DeepSeek’s r1 for free, and aren’t happy with using DeepSeek’s own offerings, lambda.chat reports they have the full version available for free, claim your data is safe and they’re hosted in the USA.

I’ve also been offered funding to build a rig myself. Comments welcome if you want to help figure out the best design and what to buy. The low bid is still this thread at $6k, which is where the original budget came from. We don’t want to be too stingy, but we also don’t want to go nuts with only the one funder (so not too much over ~$10k, and cheaper matters).

The Market Is In DeepSeek

The Verge’s Kylie Robinson and Elizabeth Lopatto cover the situation, including repeating many of the classic Bad DeepSeek Takes and call the market’s previous valuation of AI companies delusional.

A very detailed and technical analysis of the bear case for Nvidia by Jeffrey Emanuel, that Matt Levine claims may have been responsible for the Nvidia price decline. I suppose many things do indeed come...
Catastrophe through Chaos
by Marius Hobbhahn — 2025-01-31T14:19:08.399Z

This is a personal post and does not necessarily reflect the opinion of other members of Apollo Research. Many other people have talked about similar ideas, and I claim neither novelty nor credit.

Note that this reflects my median scenario for catastrophe, not my median scenario overall. I think there are plausible alternative scenarios where AI development goes very well.

When thinking about how AI could go wrong, the kind of story I’ve increasingly converged on is what I call “catastrophe through chaos.” Previously, my default scenario for how I expect AI to go wrong was something like Paul Christiano’s “

What failure looks like

,” with the modification that scheming would be a more salient part of the story much earlier.

In contrast, “catastrophe through chaos” is much more messy, and it’s much harder to point to a single clear thing that went wrong. The broad strokes of the story are something like

AI progress continues to get faster.
Many actors in the world are aware of this progress and its potential implications for ~everything.
The situation is really tense, and people make decisions under time pressure and uncertainty.
A million things could go wrong, from rogue AI to international conflicts and concentration of power.
The behavior of every actor is plausible, given their incentives, uncertainty, and time pressure.
Because our institutions are not well equipped to deal with such a situation, society fails to coordinate.
At least one part of this powder keg explodes and results in an irreversible catastrophe.

Intuitively, the vibe I have in mind for this period is something like “Cold War tensions + beginning of WW1 messiness (but on steroids)”. In this scenario, most of society, including governments, are acutely aware that something bad could happen at any moment, and then an unfortunate series of events leads to catastrophic consequences.

One of the reasons for this is that I think we have hit a sweet spot of alignment where we can sufficiently align TAI for it to be a major strategic advantage, but don’t know how to reliably align ASI. Furthermore, takeoffs are slow enough that humans still have a lot of influence during the transition from TAI to ASI but fast enough to produce a lot of pressure. Thus, every problem with AI intensifies at the same time, resulting in a massive powder keg.

There are a couple o...
Interviews with Moonshot AI's CEO, Yang Zhilin
by Cosmia_Nebula — 2025-01-31T09:19:36.561Z
Link: https://rentry.co/Moonshot-AI-interview/

<https://news.qq.com/rain/a/20240208A05KFR00>

# Exclusive interview with Moonshot's Yang Zhilin: How does a new AGI startup surpass OpenAI?

Overseas Unicorn
Published on Beijing Overseas Unicorn official account at 2024-02-21 11:25.

Interviewers: 天一、penny、guangmi
Editor: 天一
Typesetting: Scout

"Lossless long context is everything." This is the point we remember most deeply after a two-hour conversation with Yang Zhilin.

This technical judgment was already conveyed in October 2023 when Moonshot AI, founded by Yang Zhilin, released its first model, moonshot, and the smart assistant Kimi, supporting an input of 200,000 characters. The focus on "long" stems from Yang Zhilin's belief that the ultimate value of AI-Native products is providing personalized interactions, and lossless long context is the foundation for achieving this. He argues that model finetuning shouldn't exist in the long run, that the interaction history between a user and the model should be the best form of personalization, and that every generation of technology in history has increased context length.

Yang Zhilin has been labeled as a genius AI scientist, serial entrepreneur, and more. In this in-depth interview, he once again proves himself to be a startup founder who truly "understands" large models. Thus, this article presents many counter-consensus views: Yang Zhilin believes that finetuning will eventually become obsolete, and tokenizers may not necessarily be essential in the end. While large model trainers in Silicon Valley worry about data bottlenecks and energy limitations, he believes that all problems are interconnected. Multimodal approaches can alleviate data shortages, and synthetic data can address energy issues by changing the compute paradigm.

This article also attempts to answer another question widely concerning the outside world: How can a newly established AGI startup surpass OpenAI? Yang Zhilin's answer is "tech vision". The headman must be able to make technical judgments and also decisively execute them. A specific example is that Moonshot AI aims to be more user-centric than OpenAI, because Yang Zhilin judges that the scale-up effect of user data will ultimately surpass the base model itself.

Yang Zhilin is also confident that using the underlying principles of the transformer probability model will lead to AGI. In his words, "If you have a context length of 1 billion, to...
Review: The Lathe of Heaven
by dr_s — 2025-01-31T08:10:58.673Z

"The Lathe of Heaven" is a 1971 sci-fi novel by Ursula K. Le Guin. It's the story of a man, George Orr, who lives in a future ravaged by climate change and overpopulation. He gets referred to voluntary-but-not-really psychological treatment for abuse of certain pharmaceutical drugs and eventually reveals to his therapist, one Dr. Haber, that the reason for his abuse was that he wanted to prevent himself dreaming - because sometimes his dreams become real. Dr. Haber, a man of reason and science, of course does not believe him, at first.

Then the trouble begins.

The short, spoiler-free version

This being something more of a philosophical novel or parable, and a very short one at that, I don't think staying away from spoilers is particularly important to enjoy it; nevertheless, here I'll give some opinions about the novel without going into specifics for those who haven't read it and want to go in relatively blind. I'll expand later on what I think makes this novel interesting here, but I can't do that without talking about its plot.

I've seen people make the case that this novel is Ursula K. Le Guin doing her best Philip K. Dick impression, and I can totally see that. There are a lot of commonalities. The setting is oppressive and dystopian, somewhat inhuman. There is a strong focus on mental illness and the fleeting nature of reality itself - subsumed to the unconscious mind, shaped by Orr's dreams. The story is not as psychedelic as Ubik or A Scanner Darkly (for the latter I've only watched a movie, but it was quite psychedelic), but it belongs clearly to the same genre. I don't even know if it can be really called science fiction; the science elements are present and it is suggested that Orr's power really is some kind of natural phenomenon, but its qualities are still fundamentally magical, and only possible within a world in which the connection between the mind and the real is so skewed towards the former that empiricism itself might be in question as a metaphysics.

So, how is it as a read? I've certainly never been bored by it. It's an engaging read, it's dense and quick to the point, doesn't overstay its welcome, which is good, because in this kind of story it's quite clear from very early on where is it all going. The prose is pleasant and has a few more flourishes than you'd see in most classic SF of the same era, while still being mostly pragmatic and dr...
Is weak-to-strong generalization an alignment technique?
by cloud — 2025-01-31T07:13:03.332Z

Weak-to-strong generalization (W2SG) was initially presented as an analogy for human supervision of superhuman AI systems. From the paper:

We propose a simple setup for studying the problem of humans supervising superhuman models by considering an analogy: can we use weak models to supervise strong models?

"Humans supervising superhuman models" sounds a lot like achieving scalable oversight, which has at times been defined as:

[SO-1] "ensur[ing] safe behavior even given limited access to the true objective function," (here)
[SO-2] "train[ing] an RL agent with sparse human feedback," (here)
[SO-3] "the ability to provide reliable supervision—in the form of labels, reward signals, or critiques—to models in a way that will remain effective past the point that models start to achieve broadly human-level performance," (here) or
[SO-4] "assess[ing] whether an action taken or proposed by an AI is good... in a way that works even if the AI is very smart and knowledgeable," (here).

Despite variation in these definitions, they point at a similar challenge: that of inducing desirable behavior in an AI system when the ability to grade its behavior is limited in quantity or quality.[1] We might say that there is the problem of scalable oversight, and then there are methods that seek to address this problem, which could be termed scalable oversight methods. Informally, these are "methods for humans supervising superhuman models."

So, combining the motivation for W2SG with this definition of scalable oversight, we could then say: W2SG is a model of the problem of scalable oversight. The purpose of the model is to facilitate the development of scalable oversight methods.

But wait!

This does not seem to be how people use these terms. Instead, W2SG and scalable oversight are discussed as different and complementary types of methods! For example, this post calls them "two approaches to addressing weak supervision," offering the following definitions:

[SO-R] Scalable oversight: "try to make the supervision signal stronger, such that we return to the 'normal ML' regime;" and
[W2SG-R] W2SG: "make the strong student (the AI system) generalize correctly from the imperfect labels provided by the weak teacher."

Interpreted narrowly,[2] SO-R excludes many techniques that might qualify under the broader definitions given above. Notably, the definition appears[3] to exclude the given ...