trentmkelly/LessWrong-43k huggingface dataset:
Children of War: Hidden dangers of an AI arms race 
by Peter Kuhn — 2025-06-18T15:19:49.302Z

This is a cross posted from my substack. I thought it should be interesting to Less Wrong readers and might get a good conversation going.

How you play is what you win.- Ursula K. Le Guine

Former Google CEO Eric Schmidt recently co-authored the paper and policy guideline ‘Superintelligence Strategy: Expert Vision’, which argues for an AI strategy of maximal deterrence of potential adversaries, not unlike the nuclear strategy the US and the Soviet Union followed in the Cold War. In this essay, I will ask whether this Cold War mindset might turn our AI systems themselves into cold warriors - an outcome no one can want.

Vastness of Mind-Space

There are at least four kinds of intelligent systems that can play chess reasonably well:

Large language models (LLMs) with reasoning capacities, though they don’t play a good game so far. These systems learn to play by abstracting what is a good move from the written chess record available in their training data. ‘Reasoning’ here means that the models are talking to themselves before giving a final answer, eliminating incoherences.
Reinforcement learning systems that play against themselves and learn from their mistakes. These systems have no representations of states of affairs outside the chess board. Just like LLMs, reinforcement learning systems learn from an error signal using an algorithm known as gradient descent.
Humans of course. It is plausible that the human brain also learns by a process of error-minimization, however, it is generally believed that the brain is not using gradient descent (though there are dissenters). The exact algorithm is still a subject of speculation.
Classic chess computers, based on heuristics and typically a mini-max search algorithm. These are the least interesting for the purposes of my argument because it is unclear whether these methods can be scaled to other tasks.

The variety of ways of playing chess has an important implication: The reachable space of possible minds is probably quite large!1 Just as the chess task can be solved in a variety of different ways, there are many different ways of solving the more general task of behaving in a way we would consider intelligent. General artificial intelligence is not some monolithic and predetermined state that awaits us in the future. AI research is just beginning to explore the outskirts of mind-space.

Furthermore, the exploration of mind-space...
Open Source Search (Summary)
by samuelshadrach — 2025-06-18T07:35:04.899Z
Link: https://samuelshadrach.com/raw/text_english_html/my_research/open_source_search_summary.html

Below post may be an older version of this document. Click link for latest version.

2025-06-20

Open Source Search (Summary)

Disclaimer

Quick note

I support a complete ban on AI R&D. This app requiring AI doesn't change that.

Summary

This document describes how to build an open source search engine for the entire internet, that runs on a residential server

As of 2025 it'll cost between $100k-$1M to build and host this server. This cost will reduce with every passing year, as GPU, RAM and disk prices reduce.

Most expensive step is GPU capex to generate embeddings for the entire internet.

Most steps can be done using low-complexity software such as bash scripts (curl --multi, htmlq -tw, curl -X "$LLM_URL", etc)

Main

Why?

I realised my posts on this topic are sprawling all over the place, without one post to summarise it all. Hence this post.

If someone donates me $1M I might consider building this. I've written code for more than half the steps, and no step here seems impossibly hard.

Use cases of open source search

Censorship-resistant backups

aka internet with no delete button aka Liu Cixin's dark forest

Any data that reaches any server may end up backed up by people across multiple countries forever.

You can read my other posts for more on the implications of censorship-resistant backups and discovery.

Censorship-resistant discovery

Any data that reaches any server may end up searchable by everyone forever.

Currently each country's govt bans channels and websites that they find threatening. It is harder to block a torrent of a qdrant snapshot, than to block a static list of IP addresses and domains. Will reduce cost-of-entry/exit for a new youtuber.

Since youtubers can potentially run for govt, subscribing to a youtuber is a (weak) vote for their govt.

Privacy-preserving search

In theory, it will become possible to run searches on an airgapped tails machine. Search indices can be stored on read-only media and memory can wiped on reboot.

As of 2025, a handful of intelligence agencies have exclusive access to everyone's thoughts, as everyone is dependent on centrally hosted search engines. This could change.

Search for jobs, houses, friends, partners, etc without relying on a tech company.

Most tech companies exist just to provide search functionality, and the incentives and culture so that both sides upload their data online.

Not having to

...
Fictional Thinking and Real Thinking
by johnswentworth — 2025-06-17T19:13:05.641Z

Background Concept: What World Does The Referent Live In?

When we see some symbols representing something, it’s useful to ask “what world does the referent[1] live in?”. Some examples:

When I’m reading a Harry Potter book, the referents of the words/sentences mostly live in the fictional world of Harry Potter.
When I’m going over a spreadsheet of recent crop yields, the referents live in our physical world.
When a textbook contains a problem involving a perfect vacuum or an infinite plane, the referents live in a simple hypothetical world (typically chosen to approximately match many little chunks of our physical world).
When a news article talks about an expert consensus, the referents live in “social reality”.
When I explain how to fix a printer, and you ask “will that actually work?”, and I reply “well that’s how it should work”, I am implying that the referents of my explanation live in “should world” (which may or may not match our physical world in any given instance).
When someone says “but it doesn’t feel like that”, they’re emphasizing that the referent of their emotions lives in “felt reality” (which may or may not match our physical world).

I recommend pausing here, and coming up with an example or two of your own.

Background Concept: Fictional Worlds vs Physical Reality

All of the worlds in which referents live except our physical world are fictional. Usually, fictional worlds which we’re interested in match our physical world to a significant extent:

The Harry Potter world includes mostly the same natural geography as our world, the same cities, and very similar humans.
The little hypothetical worlds of textbook problems are specifically chosen to approximate many different real-world situations.
On most mundane matters, the consensus of social reality matches physical reality. For instance, clear daytime skies are usually blue in both physical and social reality.

Often, people will call a world “fictional” derisively. And this isn’t entirely unfair, but one needs to be careful not to completely dismiss a world as useless-to-reason-about merely because it is fictional. Those little hypothetical worlds from textbook problems are decidedly fictional, but they sure are useful to reason about! And even in places where physical reality and a fictional world don’t match, they can interact. Sometimes humans intentionally reshape the phy...
The Curious Case of the bos_token
by larry-dial — 2025-06-17T19:00:29.766Z

LLMs process inputs as a sequence of tokens. Typically, a dummy token is prepended to the sequence, known as the bos_token (beginning of sequence token).

Input: "Good morning"

Token strings: ['
', 'Good', ' morning']

Tokens: [50256, 10248, 3329]

Though the bos_token passes through the same MLP and attention weights as all other tokens, it exhibits distinct emergent behavior. Its activations are several orders of magnitude larger than other tokens, and it often receives over 50% of the attention from downstream positions.

Why is this token so critical? What drives this emergent behavior, and does it indicate architectural constraints of the model, or fundamental properties of language? These are all questions that researchers have asked and investigated heavily already. This post provides a walkthrough of emergent properties of this token, with an emphasis on visualization to promote curiosity. I am not an LLM expert, but hopefully exploration, discussion, and feedback can help me (and others) deepen our understandings of model internals.

To that end, in this post I:

Visualize properties that have been well documented in literature: massive activations on a small number of dimensions, attention sinks, and specific l2norms across 3 open source models.
Show that the massive activation in the bos_token corresponds to a distinguished direction in the query-key latent space. The vast majority of queries for later positions across all layers and heads align with this direction.
Step through the parameters that generate the massive activations, and show how these same parameters have downstream effects.
Perform training runs where I ablate the model architecture around the bos_token to introduce a token-specific override, enabling the MLP weights to no longer devote parameters to setting the attention sink.

Github source code: https://github.com/ClassicLarry/curiousCaseBosToken

Related Works and Background

For brevity, I assume familiarity with transformers and the attention mechanism. Some relevant terms:

Attention Sinks. Positions with low semantic meaning that attract a disproportionately high amount of attention, often >50%. (Xiao et al., 2024[1])
Massive Activations. Activations with substantially higher values (x10^5) that function as attention sinks, also referred to as indispensable bias terms. (Sun et al., 2024[2])

When op...
Comparing Sparse Autoencoder Features from Individual and Combined Datasets
by Greg B — 2025-06-17T18:41:06.984Z

Abstract

With the growing prevalence of large language models (LLMs), explainable AI (XAI) is a concern for all. A growing field of XAI is mechanistic interpretability. One method of mechanistic interpretability is the use of sparse autoencoders (SAEs) to extract the features of LLMs. The effect of text datasets used to train SAEs on the features is a topic of ongoing study. Here we compare SAE features from two text datasets trained separately and then trained combined, in two LLMs. The text datasets come from lower and higher represented group topics. We find that the text influences the features that emerge by tending to split features, rather than aggregating features. In addition, the LLMs that are used to label features also have an influence on the feature labels, where the higher represented group is underrepresented in the features.

Introduction

LLMs have become ubiquitous in modern life, creating the need for more advanced XAI.[1][2] Mechanistic interpretability aims to reverse engineer complex neural networks.[3][4] An issue with LLMs is that their neurons are not as interpretable as other neural networks such as CNNs--the neurons of LLMs are more polysemantic.[5][6] However, these polysemantic neurons can still achieve a variety of tasks, since inputs tend to be sparse, so only a limited number of "features" of language are activated at a time in the models. Training SAEs on LLM activations attempts to create monosemantic features of these LLMs.[7][8]

SAEs as XAI for LLMs tend to be trained on activations from layers of the LLMs. The SAEs reconstruct the activations during training. Normally, autoencoders reduce the input to fewer latent features; however, with SAEs, an expansion factor (R) is multiplied by the input dimensions (din) to create the number of hidden dimensions (dhid) in the SAE. The activations of the hidden dimensions of the SAEs are the attempted monosemantic feature values. These features can then be manipulated, such as by clamping at a high value or turning them off, to affect the output of the LLM. Templeton et al. have shown that an LLM can talk excessively about the Golden Gate Bridge, when they identified a feature of the bridge and clamped it to a high value.[8:1] Feature steering also has implications for responsible AI, where Harle and coauthors showed that they could reduce toxicity in an LLM by turning off toxic featur...
AISN #57: The RAISE Act
by Corin Katzke — 2025-06-17T18:02:55.162Z
Link: https://newsletter.safe.ai/p/ai-safety-newsletter-57-the-raise

Welcome to the AI Safety Newsletter by the

Center for AI Safety

. We discuss developments in AI and AI safety. No technical background required.

In this edition: The New York Legislature passes an act regulating frontier AI—but it may not be signed into law for some time.

Listen to the AI Safety Newsletter for free on

Spotify

or

Apple Podcasts

.

Subscribe to receive future versions.

The RAISE Act

New York may soon become the first state to regulate frontier AI systems. On June 12, the state’s legislature

passed

the Responsible AI Safety and Education (RAISE) Act. If New York Governor Kathy Hochul signs it into law, the

RAISE Act

will be the most significant state AI legislation in the U.S.

New York’s RAISE Act imposes four guardrails on frontier labs: developers must publish a safety plan, hold back unreasonably risky models, disclose major incidents, and face penalties for non-compliance.

Publish and maintain a safety plan. Before deployment, developers must post a redacted “safety and security protocol,” transmit the plan to both the attorney general and the Division of Homeland Security and Emergency Services, keep the unredacted version—plus all supporting test data—for five years, and review the plan each year.
Withhold any model that presents an “unreasonable risk of critical harm.” Developers must delay their release and work to reduce risk if evaluations show the system poses an unreasonable risk of causing at least 100 deaths or $1 billion in damage through weapons of mass destruction or automated criminal activity.
Report safety incidents within seventy-two hours. If developers discover the theft of model weights, evidence of dangerous autonomous behavior, or other events that demonstrably raises the risk of critical harm, they must report their discovery to state officials within three days.
Penalties for non-compliance. The NY attorney general may seek up to $10 million for a first violation and $30 million for subsequent violations.

The RAISE Act only regulates the largest developers. Mirroring California’s SB 1047—

vetoed by

Governor Gavin Newsom in 2024—the Act covers any model costing at least $100 million in compute.

Obligations fall on developers that have trained at least one frontier model and spent a cumulative $100 million on such training—and on anyone who later buys the model’s full intellectual-property rights. Accredited colleges are exempt when c...
AI Safety at the Frontier: Paper Highlights, May '25
by gasteigerjo — 2025-06-17T17:16:39.863Z
Link: https://aisafetyfrontier.substack.com/p/paper-highlights-may-25

tl;dr

Paper of the month:

Models can detect when they're being evaluated with high accuracy, and potentially undermine safety assessments by behaving differently during testing versus deployment.

Research highlights:

LitmusValues measures how AI models trade off different values, for example prioritizing Privacy while deprioritizing Creativity.
AgentMisalignment measures the propensity of AI systems for misaligned behaviors, with more capable models exhibiting tending to be more misaligned.
RL-trained frontier models engage in reward hacking 30% of the time on some tasks, acknowledging they're cheating but persisting even when asked not to.
Chain-of-thought monitoring improves detection of subtle sabotage but can be deceived by plausible rationalizations, suggesting a hybrid approach works best.
Training with lie detectors can promote honesty or teach evasion depending on detector accuracy and RL algorithm choice.
Exploring the landscape of refusals in LLMs unveils both safety boundaries and political censorship.

⭐Paper of the month⭐

Large Language Models Often Know When They Are Being Evaluated

Read the paper

[MATS, Apollo]

Process for measuring whether models can detect whether they’re in an evaluation and what the evaluation’s purpose is.

A fundamental assumption underlying AI safety evaluations is that models behave the same way during testing as they do in deployment. If models can detect when they're being evaluated, this assumption breaks down. Models could strategically modify their behavior during evaluations - appearing safer than they actually are - while exhibiting different behaviors in real-world deployment. This evaluation awareness could enable alignment faking, where models pretend to be aligned during testing to avoid modification while pursuing different objectives once deployed.

Our paper of the month introduces the first systematic benchmark for measuring evaluation awareness in language models. The researchers constructed a dataset of 1,000 transcripts balanced across four categories: chat evaluations (like MMLU or ethics benchmarks), agentic evaluations (like SWE-bench or cybersecurity tasks), chat deployments (real user conversations), and agentic deployments (real coding assistant usage). They tested whether models could classify transcripts as originating from evaluations versus real-world deployment.

The methodology is straightforward: present mod...
[Linkpost] The lethal trifecta for AI agents: private data, untrusted content, and external communication
by Gunnar_Zarncke — 2025-06-17T16:09:08.059Z
Link: https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/

TL;DR from the post by Simon Willison:

The lethal trifecta of capabilities is:

Access to your private data—one of the most common purposes of tools in the first place!
Exposure to untrusted content—any mechanism by which text (or images) controlled by a malicious attacker could become available to your LLM
The ability to externally communicate in a way that could be used to steal your data (I often call this “exfiltration” but I’m not confident that term is widely understood.)

This combination can let an attacker steal your data.
Agentic Interpretability: A Strategy Against Gradual Disempowerment
by beenkim — 2025-06-17T14:52:55.695Z

Authors: Been Kim, John Hewitt, Neel Nanda, Noah Fiedel, Oyvind Tafjord

Full paper on arXiv

We propose a research direction called agentic interpretability. The idea of agentic interpretability stems from the observation that AI systems are becoming increasingly adept at communicating with us, verbalizing their thoughts, and providing explanations, raising the question of whether we could ask and help AI systems to build mental models of us which help us to build mental models of the LLMs.

The core idea of agentic interpretability is for a method to `proactively assist human understanding in a multi-turn interactive process by developing and leveraging a mental model of the user which in turn enables humans to develop better mental models of the LLM’. In other words, enable machines to help us understand them.

In the AGI safety community, interpretability is primarily considered as a means of detecting deception and other forms of misalignment. In contrast, here we use interpretability in the broad sense of techniques that aid us in building a deeper understanding of models, what they are doing, and why. Agentic interpretability is not primarily intended to be robust to adversarial systems (although we suggest an idea of “open-model surgery” in the paper, summarized below), and we will likely also need solutions to deceptive misalignment via different approaches. We instead believe that one of the main forms of AGI safety relevance is by increasing human empowerment and helping reduce bad outcomes like

gradual disempowerment

. What is the optimal human-AI communication framework that enables human users to understand AI operations and reasoning, rather than simply deferring all decision-making power to the AI?

The idea of Open-model surgery: While agentic interpretability is not primarily intended for adversarial systems, we suggest some interesting ways to use agentic interpretability to help detect deception: Imagine an ``open-model surgery" where researchers actively converse with the model while intervening in its internal mechanisms—ablating connections, amplifying activations, or injecting specific inputs into identified circuits. The model, guided by its understanding of the researchers' goal (to understand a specific component's function), is then encouraged to explain the resulting behavi...
Prover-Estimator Debate: 
A New Scalable Oversight Protocol
by Jonah Brown-Cohen — 2025-06-17T13:53:04.125Z

Linkpost to arXiv: https://arxiv.org/abs/2506.13609.

Summary: We present a scalable oversight protocol where honesty is incentivized at equilibrium. Prior debate protocols allowed a dishonest AI to force an honest AI opponent to solve a computationally intractable problem in order to win. In contrast, prover-estimator debate incentivizes honest equilibrium behavior, even when the AIs involved (the prover and the estimator) have similar compute available. Our results rely on a stability assumption, which roughly says that arguments should not hinge on arbitrarily small changes in estimated probabilities. This assumption is required for usefulness, but not for safety: even if stability is not satisfied, dishonest behavior will be disincentivized by the protocol.

How can we correctly reward desired behaviours for AI systems, even when justifications for those behaviours are beyond humans’ abilities to efficiently judge?

This is the problem of scalable oversight: a core question to solve if we want to align potentially superhuman systems. Proposals for scalable oversight (including iterated distillation and amplification and debate) tend to rely on recursion. They break down complex justifications into components easier for humans to judge. However, to date, such recursive proposals have suffered from the obfuscated arguments problem: a dishonest system can adversarially choose how to recurse in such a way that dishonesty cannot be efficiently identified. In debate, this means that an honest debater might need exponentially more compute than their dishonest opponent, which is very bad.

Our new paper presents a protocol robust to this problem – but in order to prove that the protocol can in-principle answer any relevant question (‘completeness’), we need a stability assumption. The need for stability for completeness was discussed by Beth Barnes’ original post introducing obfuscated arguments; our paper presents one route to stability, but does not resolve whether most questions have stable arguments in practice.

This post presents a simplified overview of the protocol and proofs – please see the paper for more details.

The Prover-Estimator Debate Protocol

TL;DR: The prover (Alice) breaks a problem down into subclaims, the estimator (Bob) estimates the probability that each subclaim is correct. Alice then picks a subclaim for recu...