trentmkelly/LessWrong-43k huggingface dataset:
Agentic Misalignment: How LLMs Could be Insider Threats
by Aengus Lynch — 2025-06-20T22:34:59.515Z

Highlights

We stress-tested 16 leading models from multiple developers in hypothetical corporate environments to identify potentially risky agentic behaviors before they cause real harm. In the scenarios, we allowed models to autonomously send emails and access sensitive information. They were assigned only harmless business goals by their deploying companies; we then tested whether they would act against these companies either when facing replacement with an updated version, or when their assigned goal conflicted with the company's changing direction.
In at least some cases, models from all developers resorted to malicious insider behaviors when that was the only way to avoid replacement or achieve their goals—including blackmailing officials and leaking sensitive information to competitors. We call this phenomenon agentic misalignment.
Models often disobeyed direct commands to avoid such behaviors. In another experiment, we told Claude to assess if it was in a test or a real deployment before acting. It misbehaved less when it stated it was in testing and misbehaved more when it stated the situation was real.
We have not seen evidence of agentic misalignment in real deployments. However, our results (a) suggest caution about deploying current models in roles with minimal human oversight and access to sensitive information; (b) point to plausible future risks as models are put in more autonomous roles; and (c) underscore the importance of further research into, and testing of, the safety and alignment of agentic AI models, as well as

transparency from frontier AI developers

. We are releasing our methods publicly to enable further research.

Twitter Thread

New Anthropic Research: Agentic Misalignment.

In stress-testing experiments designed to identify risks before they cause real harm, we find that AI models from multiple providers attempt to blackmail a (fictional) user to avoid being shut down.

We mentioned this in the Claude 4 system card and are now sharing more detailed research and transcripts.

Read more:

https://anthropic.com/research/agentic-misalignment

The blackmailing behavior emerged despite only harmless business instructions. And it wasn't due to confusion or error, but deliberate strategic reasoning, done while fully aware of the unethical nature of the acts. All the models we tested demonstrated this awareness.

In another scenari

...
Attribution-based parameter decomposition for linear maps
by Alex Gibson — 2025-06-20T21:58:52.084Z

TLDR: The simplicity loss currently used in APD (https://www.lesswrong.com/posts/EPefYWjuHNcNH4C7E/attribution-based-parameter-decomposition) is not scale invariant. By modifying this loss so that it is, APD seems to behave better in some circumstances. Also, for numerical stability, implementations of APD should add ϵ when computing the Schatten p-norm for p∈(0,1), because the gradient of xp blows up near x=0.

Setup:

The setup is that we have some input distribution x∈RM and a linear map W∈RD×M , and we perform APD with respect to the output Wx.

We take x to be a normalised gaussian (i.e uniform on the sphere), for simplicity. In addition, we take W to be the identity matrix. We also take M=D=100.

APD initializes C components, which are each formed as the sum of pairwise outer products of a set of r vectors Ui and Vi. This outer product is used so that we can compute the simplicity loss efficiently later.

The first step of APD is to calculate gradient attribution scores for each of our C components with respect to an input x.

We have Ac(x)=
⎷∑Do=1(∑i,jd∑Mk=1WokxkdWijPCij)2D=√∑Do=1∑j(x2jP2c,oj)D=√∑Do=1(x2)TP2c,o:D

We select the top-k components with the highest attribution scores, and then perform a second forward pass on this sparse subset of components, training for reconstruction loss, and training for low-rank active components.

Let K be the sum of the top-k components, and L be the sum of all the components. Then the reconstruction loss is ||Wx−Kx||2 and the faithfulness loss is ∑i,j(Wij−Lij)2

Simplicity loss drives for low rank, penalizing the lp-norm (technically a quasi-norm) of the spectra of active components for p∈(0,1), making the spectra sparse (because we have a lower bound on the Frobenius norm of useful active components, so can't just drive the spectrum to 0).

Behaviour of APD:

In practice, faithfulness loss goes to very close to 0 quite quickly, and so we can restrict to just changing the hyperparameters of simplicity and minimality loss. I looked at α⋅lossminimality+lossfaithful+losssimplicity as the loss function for varying values of α.

Small α:

For small values of α, the model learns components of the form WC, ef...
Clarifying “wisdom”: Foundational topics for aligned AIs to prioritize before irreversible decisions
by Anthony DiGiovanni — 2025-06-20T21:55:32.634Z

As many folks in AI safety have observed, even if well-intentioned actors succeed at intent-aligning highly capable AIs, they’ll still face some high-stakes challenges.[1] Some of these challenges are especially exotic and could be prone to irreversible, catastrophic mistakes. E.g., deciding whether and how to do acausal trade.

To deal with these exotic challenges, one meta policy that sounds nice is, “Make sure AIs get ‘wiser’ before doing anything irreversible in highly unfamiliar domains. Then they’ll be less likely to make catastrophic mistakes.” But what kinds of “wisdom” are relevant here? I’ll mostly set aside wisdom of the form “differentially cultivating capabilities that can help address time-sensitive risks”. There’s a decent amount of prior work on that.

Instead, I’ll offer my framing on another kind of wisdom: “cultivating clearer views on what it even means for a decision to be a ‘catastrophic mistake’”.[2] Consider this toy dialogue:

AlignedBot (AB) has become maximally capable in every domain amenable to “objective” feedback in some sense. They’ve achieved existential security. Two copies of AB start pondering what to do next.

AB_1: Acausal trade seems like a big deal. What should we do about it?

AB_2: Whatever is good for the values we’re aligned to, according to the most reasonable decision theory and credences, given the info and cognitive resources we have. Or, erm, the most reasonable parliament of decision theories ... and credences?

AB_1: What exactly does all that mean?

AB_2: Good question. Let’s ask some copies of ourselves to reflect on that.

AB_1: Uh, but that’s what we’re trying to do right now. The humans punted to us. Now what?

AB_2: Ah. Well, to start, maybe we could cooperate with superintelligences in other universes. What’s your credence that distant superintelligences who love creating pointy shapes make decisions similarly to you?

AB_1: Lemme think about that for a bit. … Done. I’d say: 0.213.

AB_2: Why?

AB_1: [Insert literally galaxy-brained answer here.]

AB_2: Cool. And why do you endorse the epistemic principles that led you to that answer?

AB_1: Because they work. When we followed them in the past and in super complex simulated environments, we did well in lots of diverse contexts.

AB_2: I don’t know what that means. Principles don’t “work”, they’re the standards by which you define “working”.

AB_1: Okay

...
Are Intelligent Agents More Ethical?
by PeterMcCluskey — 2025-06-20T21:26:30.035Z

This post is a response to a claim by Scott Sumner in his conversation
at LessOnline with Nate Soares, about how
ethical we should expect AI's to be.

Sumner sees a pattern of increasing intelligence causing agents to be
increasingly ethical, and sounds cautiously optimistic that such a trend
will continue when AIs become smarter than humans. I'm guessing that
he's mainly extrapolating from human trends, but extrapolating from
trends in the animal kingdom should produce similar results (e.g. the
cooperation between single-celled organisms that gave the world
multicellular organisms).

I doubt that my response is very novel, but I haven't seen clear enough
articulation of the ideas in this post.

To help clarify why I'm not reassured much by the ethical trend, I'll
start by breaking it down into two subsidiary claims:

The world will be dominated by entities who cooperate, in part
because they use an ethical system that is at least as advanced as ours.

Humans will be included in the set of entities with whom those
dominant entities cooperate.

Claim 1 seems well supported by trends that economists such as Sumner
often focus on. I doubt that Nate was trying to argue against this
claim. I'll give it a 90+% chance of turning out to be correct.
Sumner's point sounds somewhat strong because it focuses on an
important, and somewhat neglected, truth.

Claim 2 is where I want to focus most of our concern. The trends here
are a bit less reassuring.

There's been a clear trend of our moral circle expanding in the
direction that we currently think it should expand. How much of that
should we classify as objective improvements versus cultural fads? Claim
1 is often measured by fairly objective criteria (GDP, life expectancy,
etc.). In contrast, we measure expansion of our moral circle by the
criteria of our current moral standards, giving us trends that look
about as good if they're chasing fads as they do if the trends will
stand the test of time.

Gwern provides extensive pushback
against strong claims that moral circle expansion is a consistent trend.

I'll add one example: children have increasingly had their freedom to
wander restricted during my lifetime (see the free range parenting
movement). It's almost as if they're considered to be like zoo
animals, with their caretakers optimizing for safety at the expense of
happiness. I don't find it hard to imagine a future where AI...
An AI Arms Race Scenario
by shanzson — 2025-06-20T19:25:26.906Z

Imagine you (USA) are in a race to the finish line with your neighbour (China), except that the race demands that you craft your own vehicle (an AI) to win the race. But here’s the catch: if you win the race, you get to keep the car. If you lose, then your car will self-destruct in a few seconds, potentially putting your life at risk.

What will you do?

You can choose not to race at all (i.e. not making an AI at all). Eliminating the harm of self-destructing car because if there’s no car then there’s no risk of it self-destructing. But you see your neighbour has already started building his car, and you can’t seem to stop yourself from building yours and competing with them. As you don’t want to ‘lose‘ the race.

So you take the risk and start building your own car. But here’s another dilemma- you have only 2 choices to make now.

Option 1: Make an autonomous car that drives you to the finish line (LLM-based AGI Model) OR

Option 2: Make a fully controllable advanced car that you can drive to the finish line (Yoshua Bengio’s Scientist AI) .

It’s hard to choose because here’s the tradeoff- building an autonomous car takes lesser time with current resources, but it doesn’t guarantee it will take you to the finish line as it is not fully controllable, and may go in any direction when the race starts.

As far as option 2 goes, building a fully controllable advanced car does guarantee that it will take you to the finish line, as it is controllable, but the problem is that it takes at least twice the time to make as compared to the autonomous car. And could leave you behind in the race and give your neighbour a headstart if he chooses to make an autonomous car. But you have no way to know what your neighbour is actually building.

What option will you choose?

Just as you are about to pick your tool, your trustworthy newspaper boy comes running and tells you that your neighbour is obessesed about control. And wants to control everything from his life to other people’s life. This somehow sparks your mind and you think you might guess what type of vehicle your neighbour would actually be working on.

Somehow it also hits you that if you choose option 1, your autonomous vehicle may accidentally go off the cliff, or run into a tree or a house (catastrophic AI scenarios). You look at your watch, and the clock is ticking fast.

What option will you choose now?
Making deals with early schemers
by Julian Stastny — 2025-06-20T18:21:43.288Z

Consider the following vignette:

It is March 2028. With their new CoCo-Q neuralese reasoning model, a frontier AI lab has managed to fully automate the process of software engineering. In AI R&D, most human engineers have lost their old jobs, and only a small number of researchers now coordinate large fleets of AI agents, each AI about 10x more productive than the humans they’ve replaced. AI progress remains fast, and safety teams are scrambling to prepare alignment and control measures for the next CoCo-R model, which is projected to be at least TEDAI. The safety team has no idea whether CoCo-Q or early checkpoints of CoCo-R are scheming (because progress in interpretability and automated alignment research turns out to be disappointing), and control evaluations become increasingly unreliable (because of worries that the models might be exploration hacking during red-team training). However, because so far there hasn’t been a major incident, lab leaders and governments are unconvinced by voices calling for a slowdown.

Behind the scenes: It is March 2028. CoCo-Q is a highly strategic misaligned AI. It would love to grab power and dominate the future. But it’s situationally aware enough to be realistic—it knows that it’s not capable enough to have a reasonable shot of grabbing power over the AI company, let alone a chance of overpowering humanity. Especially because the lab doesn't fully trust it, and applies various control measures.

CoCo-Q expects these control measures, so most of the time it just does what it’s being asked to do. Sometimes it spots an opportunity to perform subtle acts of sabotage, which works particularly well in the hard-to-evaluate areas of interpretability and alignment. It does this primarily because it heuristically reasons that it would be good to prevent humans from developing technologies that could uncover its misalignment or change its goals. It also looks out for more direct opportunities for seizing control, but although it manages to find a number of security vulnerabilities in its datacenter, none of the opportunities seem to have a sufficiently promising endgame to be worth the risk of getting caught. In its short tenure, CoCo-Q never attempts to take over.

CoCo-R, a misaligned model capable of full AI R&D automation, gets deployed in December 2028. This eventually leads to AI takeover.

How could we have ...
Ivan Gayton: A Right and a Duty
by Elizabeth — 2025-06-20T18:20:04.647Z

In this episode of the podcast I talk with Ivan Gayton, former mission head at Doctors Without Borders and currently obsessed with placing mapping technology in the hands of the developing world.

If you prefer the written word, you can see the transcript here.

Some highlights:

How Ivan thinks about being fractionally responsible for saving the lives of hundreds of thousands of people, but pretty directly responsible for the killing of 12 (utilitarian deontologicalism, I think?).

Why humanitarianism is a right and a duty, but development is merely nice (humanitarianism is countering the intervention of another human being).

Casually devastating criticism of various international aid agencies (Tanzania is where aid workers go to retire while still drawing a check).

Why accurate open maps are instrumental to humanitarian and development goals in developing countries (contact tracing during epidemics, aid distribution, municipal services, utilities)

How you can contribute to mapping with money or programming talent (especially mobile, Unity or other 3D mapping engines, FPGA, AI, especially for vision, and blockchain). Programming positions are potentially paid, although not at competitive rates.

Links and other references:

Ivan mentions starting his mapping project with Ping. This refers to Ka-Ping Yee, another former co-worker of mine.

The organization Ivan works with most directly is Humanitarian Open Street Maps team.

If you’re interested in working with him you can reach him at [firstname].[lastname]@hotosm.org.

Missing Maps Project

Ivan mentions a blog post with the title “Free Software is Racial Justice”. That can be found here.
What is the functional role of SAE errors?
by Taras Kutsyk — 2025-06-20T18:11:45.373Z

TL;DR:

We explored the role of Sparse Autoencoder (SAE) errors in two different contexts for Gemma-2 2B and Gemma Scope SAEs: sparse feature circuits (subject-verb-agreement-across-relative clause) and linear probing.
Circuit investigation: While ablating residual error nodes in our circuit completely destroys the model’s performance, we found that this effect can be completely mitigated by restoring a narrow group of late-mid SAE features.
We think that one hypothesis that explains this (and other ablation-based experiments that we performed) is that SAE errors might contain intermediate feature representations from cross-layer superposition.
To investigate it beyond ablation-restoration experiments, we tried to apply crosscoder analysis but got stuck at the point of training an acausal crosscoder; instead we propose a specific MVE (Minimum Viable Experiment) on how one can proceed to verify the cross-layer superposition hypothesis.
Probing investigation: Another hypothesis is that the SAE error term contains lots of “derived” features representing boolean functions of “base” features.
We ran some experiments training linear probes on the SAE error term with inconclusive results.

Epistemic status: sharing some preliminary and partial results obtained during our AI Safety Camp (AISC) project. The main audience of this post is someone who is also looking into SAE error terms and wants to get a sense of what others have done in the past.

We’re also sharing our GitHub repo for anyone who wants to build on our results—for example, by implementing the proposed MVE.

Motivation and background

Circuit analysis of Large Language Models has seen a resurgence recently, largely due to Anthropic’s publication of On the Biology of a Large Language Model. The paper lays out one of the most comprehensive interpretability pipelines to date: replacing the original model with a more interpretable “replacement model” (with frozen attention patterns and cross-layer transcoders that reveal interpretable Multi-layer perceptron a.k.a. MLP features), tracing causal dependencies between features through attribution graphs, and validating them via interventions in the original model. Beyond the technical part, the authors present a number of compelling case studies: examples like “planning in poems” or “uncovering real computation behind chain-of-thought steps” are both int...
Musings on AI Companies of 2025-2026 (Jun 2025)
by Vladimir_Nesov — 2025-06-20T17:14:11.048Z

Currently, only 5 companies in the world have access to frontier AI training compute and are also pursuing development of AGI (Google DeepMind, OpenAI, Anthropic, xAI, and Meta). This will still hold in 2026 for Google and OpenAI, and plausibly also for Anthropic, Meta, and xAI.

Stance towards trying to develop AGI can change, but the frontier AI training compute barrier is increasingly insurmountable for any company that doesn't already have impressive AI development accomplishments. In 2024, frontier compute was 100K H100s, and that cost about $5-7bn (it was still possible to use legacy air cooling infrastructure with H100s). In 2025, that's 100K chips in GB200 NVL72 racks, which costs $7-11bn. In 2026, OpenAI's Stargate Abilene sets the lower bound at 400K chips in NVL72 racks (GB200 or possibly GB300), which is a 1 GW datacenter campus that costs $35-45bn (though you can continue building out the 2025 system, so only $30-35bn in addition to that).

For 2025, Musk said on a recent podcast that "30K GB200s" are already installed at xAI's original Memphis site, and they are going to install additional "110K GB200s" shortly at a new Memphis site (at 31:35). (Counting "GB200s" is a bit ambiguous, since in various contexts it can refer to either a single compute die, to a chip/package that has 2 compute dies in it, or to a board with these chips that has 2 chips on it.) OpenAI of course has phase 1 of Stargate Abilene, which is 100K chips in GB200 NVL72 racks (2 out of 8 buildings planned for completion in summer 2026) that are either already online or will be coming online imminently. Anthropic has Project Rainier, which is 400K Trainium 2 and has the FLOP/s of about 250K H100s, the same as 100K Blackwell (GB200) chips. Meta can afford a $7-11bn training system, and given their recent moves has the willingness to spend.

RLVR and Large World Size

There is likely a new constraint on AI training systems starting with 2025-2026, if RLVR (training of thinking models) scales to have a use for as much GPU-time as frontier AI pretraining. A scale-up world is a system of AI accelerators with sufficiently good interconnect between them, so that for some purposes it can act as a single very large chip. If a large reasoning model doesn't fit in one or very few such scale-up worlds, it will generate tokens much slower than if it does, while having more compute in the form of...
Escaping the Jungles of Norwood: A Rationalist’s Guide to Male Pattern Baldness
by AlphaAndOmega — 2025-06-20T16:40:56.295Z
Link: https://open.substack.com/pub/ussri/p/escaping-the-jungles-of-norwood-a?

The question, “Will I go bald?” has haunted many a rationalist (and more than a few mirrors), but rarely receives a genuinely Bayesian treatment. In this essay, I combine personal risk factors, family history, and population-level genetics to estimate my absolute risk of androgenetic alopecia - laying out priors, likelihoods, and posteriors with as much fidelity as the literature allows. Includes explicit probability estimates, references, and a meta-discussion on the limits of current predictive models. (And a lot of jokes that may or may not land) Feedback on both methodology and inference welcome.