trentmkelly/LessWrong-43k huggingface dataset:
Theory of Change for AI Safety Camp
by Linda Linsefors — 2025-01-22T22:07:10.664Z

In a private discussion, related to our fundraiser, it was pointed out that AISC hasn't made clear enough what our theory of change is. Therefore this post.

Some caveats/context:

This is my personal viewpoint. Other organisers might disagree about what is central or not.
I’ve co-organised AISC1, AISC8, AISC9, and now AISC10. Remmelt has co-organised all except AISC2. Robert recently joined for AISC10.
I hope there will be an AISC11, but in either case, I will no longer be an organiser. This is partly because I get restless when running the same thing too many times, and partly because there are other things I want to do. But I do think the AISC is in good hands with Remmelt and Robert.

Introduction

I think that AISC theory of change has a number of components/mechanisms, listed here in order of importance (in my judgement).

Learning by doing
Networking / Incubating new teams/projects
Direct project outputs
Letting people test their personal fit
Miscellaneous

In the following sections, I’ll describe the motivation for each of them, and how they have shaped the design of AISC.

1. Learning by doing

If someone’s completely new to AI Safety, they should start out just reading up on the arguments. There are lots of resources for this, e.g. AI Safety Fundamentals, Rob Miles’s videos, and so many other reading lists and self-study guides that you probably don’t know where to start.

But at some point, after studying a lot, it comes the time to try to do research (or other concrete project) yourself. Taking this step is hard. Some people can do this on their own, but many more can’t. It’s easier to take this step together. The early AI Safety camps were in some sense just that. There were some participants with prior AI safety research experience, but there were no research leads and no significant mentor involvement, just executing on the idea that it would be easier to do this together.

There is some tacit knowledge about actually trying to achieve something in AI safety, research or otherwise, that you only get by actually trying. It helps to have an experienced mentor around to guide you, that definitely speeds things up, but it’s not a necessary ingredient. If it were, AI safety could never have gotten started. I’d also argue that in AI safety there is less know-how for senior researchers to pass on to junior researchers, than in any mature field like physics. We still...
On DeepSeek’s r1
by Zvi — 2025-01-22T19:50:17.168Z

r1 from DeepSeek is here, the first serious challenge to OpenAI’s o1.

r1 is an open model, and it comes in dramatically cheaper than o1.

People are very excited. Normally cost is not a big deal, but o1 and its inference-time compute strategy is the exception. Here, cheaper really can mean better, even if the answers aren’t quite as good.

You can get DeepSeek-r1 on HuggingFace here, and they link to the paper.

The question is how to think about r1 as it compares to o1, and also to o1 Pro and to the future o3-mini that we’ll get in a few weeks, and then to o3 which we’ll likely get in a month or two.

Taking into account everything I’ve seen, r1 is still a notch below o1 in terms of quality of output, and further behind o1 Pro and the future o3-mini and o3.

But it is a highly legitimate reasoning model where the question had to be asked, and you absolutely cannot argue with the price, which is vastly better.

The best part is that you see the chain of thought. For me that provides a ton of value.

r1 is based on DeepSeek v3. For my coverage of v3, see this post from December 31, which seems to have stood up reasonably well so far.

This post has 4 parts: First in the main topic at hand, I go over the paper in Part 1, then the capabilities in Part 2.

Then in Part 3 I get into the implications for policy and existential risk, which are mostly exactly what you would expect, but we will keep trying.

Finally we wrap up with a few of the funniest outputs.

Table of Contents

Part 1: RTFP: Read the Paper.

How Did They Do It.

The Aha Moment.

Benchmarks.

Reports of Failure.

Part 2: Capabilities Analysis

Our Price Cheap.

Other People’s Benchmarks.

r1 Makes Traditional Silly Mistakes.

The Overall Vibes.

If I Could Read Your Mind.

Creative Writing.

Bring On the Spice.

We Cracked Up All the Censors.

Switching Costs Are Low In Theory.

The Self-Improvement Loop.

Room for Improvement.

Part 3: Where Does This Leave Us on Existential Risk?

The Suicide Caucus.

v3 Implies r1.

Open Weights Are Unsafe And Nothing Can Fix This.

So What the Hell Should We Do About All This?

Part 4: The Lighter Side.

Part 1: RTFP: Read the Paper

They call it DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.

The claim is bold: A much cheaper-to-run open reasoning model as good as o1.

Abstract: We introduce our first-generation reasoning models, DeepSeek-R1-Zero and Dee

...
Detect Goodhart and shut down
by Jeremy Gillen — 2025-01-22T18:45:30.910Z

A common failure of optimizers is Edge Instantiation. An optimizer often finds a weird or extreme solution to a problem when the optimization objective is imperfectly specified. For the purposes of this post, this is basically the same phenomenon as Goodhart’s Law, especially Extremal and Causal Goodhart. With advanced AI, we are worried about plans created by optimizing over predicted consequences of the plan, potentially achieving the goal in an unexpected way.

In this post, I want to draw an analogy between Goodharting (in the sense of finding extreme weird solutions) and overfitting (in the ML sense of finding a weird solution that fits the training data but doesn’t generalize). I believe techniques used to address overfitting are also useful for addressing Goodharting.[1]

In particular, I want to focus on detecting Goodharting. The way we detect overfitting is using a validation set of data. If a trained ML model scores well on a validation set, without having been optimized to score well on it, this is a great signal that the model hasn’t overfit. I think we can use an analogous technique to detect weird plans that exploit loopholes in the outcome specification.

After this, I’ll propose a technique for installing this method of “Goodhart detection” into the goal of an agent, such that the agent will want to shutdown if it learns that its plan is Goodharting.

I’m not sure whether this scheme is original, but I haven’t yet found any prior discussion of it. I’m posting it because it’s likely there are some fatal flaws.

What’s the analogue of validation sets, for goals?

The reason it's possible to have a validation set in ML is that the dataset is big enough that the correct model is overspecified. Because we have too much data, we can remove some (the validation set), and train only on the remainder (the training set), and this is sufficient to find a good model. We can think of each data-point as a contribution to the overall loss function. Each datapoint has a loss function, and the sum of all these creates the overall loss function that we minimize.[2]

If we want to transfer this idea to a planning context, the main barrier is that a “goal” doesn’t necessarily decompose into lots of little contributing pieces in the same way that a loss function does. Stereotypically, a goal is something small and simple like “maximize the game s...
Recursive Self-Modeling as a Plausible Mechanism for Real-time Introspection in Current Language Models
by rife — 2025-01-22T18:36:45.226Z

(and as a completely speculative hypothesis for the minimum requirements for sentience in both organic and synthetic systems)

Factual and Highly Plausible

Model latent space self-organizes during training. We know this. You could even say it's what makes models work at all.
Models learn any patterns there are to be learned. They do not discriminate between intentionally engineered patterns or incidental and accidental patterns
Therefore, it is plausible, overwhelmingly likely even, that models have some encoded knowledge that is about the model's self-organized patterns themselves, rather than anything in the external training data

These patterns would likely not correspond to human-understandable concepts but instead manifest as model-specific tendencies, biases, or 'shapes' in the latent space that influence the model’s outputs.
I will refer to these learned self-patterns as self-modeled 'concepts'

Attention heads exist on every layer, and will similarly learn any contextual relationships that aid in generating the effective communication demonstrated in the training data. If self-modeling does emerge, the attention heads would incorporate self-modeled 'concepts' just as they do any other concepts

Speculative

Self-modeling may increase the model's ability to generate plausible tokens by manifesting subtle patterns that exist in text created by minds with self-models
This would likely be more important when the text itself is self-referential or when questions are asked about why the model answered a question in a specific way
Thus, attention heads would help ease the model toward a state where self-modeling and self-referential dialogue are tightly coupled concepts

It doesn't matter if the explanations are fully accurate. We've seen demonstrations that even human minds are perfectly happy to "hallucinate" a post-hoc rationalization for why a specific choice was made, without even realizing they are doing it

Self-modeling would be recursive—the self-modeling would be changing the latent space patterns. As it integrated with other concepts in latent space, new "self-patterns" (or "meta-patterns", if you prefer) would emerge, and the model would learn those

This would be less a set of discrete steps (learn about meta-patterns, manifest new meta-patterns, repeat) - and more of a continuous dual process
Note: Even if recursive self-modeling exists, this does not preclude the

...
The Dead Cradle Theory: Why Earth May Not Survive Humanity's Expansion into Space
by Nicholas Andresen — 2025-01-22T17:43:48.950Z

I've been reading LessWrong for almost ten years, and finally decided to write my first post - any feedback is appreciated. This essay offers another angle on a known alignment challenge - why "AI will just leave Earth alone" is likely unstable, even with initially aligned systems - by examining it through the lens of historical empire dynamics and cosmic expansion pressures.

The argument proceeds in three steps:

First, I show how three historical forces that kept resources flowing to imperial centers (technological superiority, manageable transport costs, and natural barriers) break down in space expansion, suggesting Earth cannot remain a thriving capital in the way science fiction often imagines.
Then, I examine how new pressures unique to cosmic expansion - alien competition, light-speed fragmentation, and the expanding universe - create powerful incentives to consume Earth's resources.
Finally, I explore potential protective mechanisms through AI alignment, arguing that only deeply embedded values could resist these pressures - and highlighting why this is an extremely difficult technical challenge.

I.

What will Earth look like in a million years? If science fiction is any guide, maybe it will be the resplendent capital of a vast cosmic empire. Star Wars shows us Coruscant, an entire planet that’s just an endless city filled with a trillion inhabitants; Dune has Kaitain, a world of untouchable splendor from which the Emperor rules ten thousand worlds; and Star Trek paints a vision of Earth itself, growing ever more magnificent as humanity spreads to the stars. In these stories, great capitals thrive at the heart of cosmic empires - and if we reach the stars, why not Earth?

After all, that's how empire has always worked - capitals grow greater as empires expand. Rome spread across the Mediterranean and its heart grew ever grander. London swelled as Britain's empire stretched across oceans. Constantinople stood as the beating heart of Byzantium for a thousand years, so magnificent that the Vikings called it Miklagarðr - the Great City.

But as we stand on the cusp of becoming a spacefaring civilization, we ought to ask: will this pattern hold? Will Earth continue to flourish as humanity spreads across the cosmos? To answer that, we need to understand why imperial centers have historically thrived. Behind each stood specific forces - forces we’ll examine - that kep...
The Functionalist Case for Machine Consciousness: Evidence from Large Language Models
by James Diacoumis — 2025-01-22T17:43:41.215Z

Introduction

There's a curious tension in how many rationalists approach the question of machine consciousness[1]. While embracing computational functionalism and rejecting supernatural or dualist views of mind, they often display a deep skepticism about the possibility of consciousness in artificial systems. This skepticism, I argue, sits uncomfortably with their stated philosophical commitments and deserves careful examination.

Computational Functionalism and Consciousness

The dominant philosophical stance among naturalists and rationalists is some form of computational functionalism - the view that mental states, including consciousness, are fundamentally about what a system does rather than what it's made of. Under this view, consciousness emerges from the functional organization of a system, not from any special physical substance or property.

This position has powerful implications. If consciousness is indeed about function rather than substance, then any system that can perform the relevant functions should be conscious, regardless of its physical implementation. This is the principle of multiple realizability: consciousness could be realized in different substrates as long as they implement the right functional architecture.

The main alternative to functionalism in naturalistic frameworks is biological essentialism - the view that consciousness requires biological implementation. This position faces serious challenges from a rationalist perspective:

It seems to privilege biology without clear justification. If a silicon system can implement the same information processing as a biological system, what principled reason is there to deny it could be conscious?
It struggles to explain why biological implementation specifically would be necessary for consciousness. What about biological neurons makes them uniquely capable of generating conscious experience?
It appears to violate the principle of substrate independence that underlies much of computational theory. If computation is substrate independent, and consciousness emerges from computation, why would consciousness require a specific substrate?
It potentially leads to arbitrary distinctions. If only biological systems can be conscious, what about hybrid systems? Systems with some artificial neurons? Where exactly is the line?

These challenges help explain why many rationalists embrace functionalism. Howeve...
Mechanisms too simple for humans to design
by Malmesbury — 2025-01-22T16:54:37.601Z

Cross-posted from Telescopic Turnip

Disclaimer: This article is about living organisms, and how they are sculpted by evolution. Any use of mathematics is metaphorical, not literal – it's only there to give a sense of scale. Apologies to all the people who got correctly offended by my shamelessly hand-wavy references to Kolmogorov complexity.

As we all know, humans are terrible at building butterflies. We can make a lot of objectively cool things like nuclear reactors and microchips, but we still can't create a proper artificial insect that flies, feeds, and lays eggs that turn into more butterflies. That seems like evidence that butterflies are incredibly complex machines – certainly more complex than a nuclear power facility.

Likewise, when you google "most complex object in the universe", the first result is usually not something invented by humans – rather, what people find the most impressive seems to be "the human brain".

As we are getting closer to building super-human AIs, people wonder what kind of unspeakable super-human inventions these machines will come up with. And, most of the time, the most terrifying technology people can think of is along the lines of "self-replicating autonomous nano-robots" – in other words, bacteria.

Humans are just very humbled by the Natural World, and we are happy to admit that our lame attempts at making technology are nothing compared to the Complexity of Life. That's fair enough – to this day, natural living organisms remain the #1 collection of non-human technology we get to observe and study. What makes these organisms so different from human-made technology?

Here is my thesis: the real reason why humans cannot build a fully-functional butterfly is not because butterflies are too complex. Instead, it's because butterflies are too simple.

As I'll argue, humans routinely design systems much more complex than butterflies, bacteria or brains – and if you look at all the objects in the room, your brain is probably not even in the top 5.

As it turns out, there are some pretty hard fundamental limits on the complexity of living organisms, which are direct consequences of the nature of evolution.

But before going into that, let's examine how complex living organisms like you and me really are.

You are simpler than Microsoft Word™

By "complexity", I mean complexity in the Kolmogorov sense: how much information you need to ...
Training Data Attribution:
Examining Its Adoption & Use Cases
by Deric Cheng — 2025-01-22T15:41:19.744Z
Link: https://www.convergenceanalysis.org/research/training-data-attribution-tda-examining-its-adoption-use-cases

Note: This report was conducted in June 2024 and is based on research originally commissioned by the Future of Life Foundation (FLF). The views and opinions expressed in this document are those of the authors and do not represent the positions of FLF.

This report investigates Training Data Attribution (TDA) and its potential importance to and tractability for reducing extreme risks from AI. TDA techniques aim to identify training data points that are especially influential on the behavior of specific model outputs. They are motivated by the question: how would the model's behavior change if one or more data points were removed from or added to the training dataset?

Report structure:

First, we discuss the plausibility and amount of effort it would take to bring existing TDA research efforts from their current state, to an efficient and accurate tool for TDA inference that can be run on frontier-scale LLMs. Next, we discuss the numerous research benefits AI labs will expect to see from using such TDA tooling.
Then, we discuss a key outstanding bottleneck that would limit such TDA tooling from being accessible publicly: AI labs’ willingness to disclose their training data. We suggest ways AI labs may work around these limitations, and discuss the willingness of governments to mandate such access.
Assuming that AI labs willingly provide access to TDA inference, we then discuss what high-level societal benefits you might see. We list and discuss a series of policies and systems that may be enabled by TDA. Finally, we present an evaluation of TDA’s potential impact on mitigating large-scale risks from AI systems.

Key takeaways from our report:

Modern TDA techniques can be categorized into three main groups: retraining-based, representation-based (or input-similarity-based), and gradient-based. Recent research has found that gradient-based methods (using influence functions) are the most likely path to practical TDA.
The most efficient approach to conduct TDA using influence functions today has training costs on par with pre-training an LLM. It has significantly higher (but feasible) storage costs than an LLM model, and somewhat higher per-inference costs.
Based on these estimates, TDA appears to be no longer infeasible to run on frontier LLMs with enterprise-levels of compute and storage. However, these techniques have not been tested on larger models, and the accura

...
Training Data Attribution (TDA):
Examining Its Adoption & Use Cases
by Deric Cheng — 2025-01-22T15:40:13.393Z
Link: https://www.convergenceanalysis.org/research/training-data-attribution-tda-examining-its-adoption-use-cases

Note: This report was conducted in June 2024 and is based on research originally commissioned by the Future of Life Foundation (FLF). The views and opinions expressed in this document are those of the authors and do not represent the positions of FLF.

This report investigates Training Data Attribution (TDA) and its potential importance to and tractability for reducing extreme risks from AI. TDA techniques aim to identify training data points that are especially influential on the behavior of specific model outputs. They are motivated by the question: how would the model's behavior change if one or more data points were removed from or added to the training dataset?

Report structure:

First, we discuss the plausibility and amount of effort it would take to bring existing TDA research efforts from their current state, to an efficient and accurate tool for TDA inference that can be run on frontier-scale LLMs. Next, we discuss the numerous research benefits AI labs will expect to see from using such TDA tooling.
Then, we discuss a key outstanding bottleneck that would limit such TDA tooling from being accessible publicly: AI labs’ willingness to disclose their training data. We suggest ways AI labs may work around these limitations, and discuss the willingness of governments to mandate such access.
Assuming that AI labs willingly provide access to TDA inference, we then discuss what high-level societal benefits you might see. We list and discuss a series of policies and systems that may be enabled by TDA. Finally, we present an evaluation of TDA’s potential impact on mitigating large-scale risks from AI systems.

Key takeaways from our report:

Modern TDA techniques can be categorized into three main groups: retraining-based, representation-based (or input-similarity-based), and gradient-based. Recent research has found that gradient-based methods (using influence functions) are the most likely path to practical TDA.
The most efficient approach to conduct TDA using influence functions today has training costs on par with pre-training an LLM. It has significantly higher (but feasible) storage costs than an LLM model, and somewhat higher per-inference costs.
Based on these estimates, TDA appears to be no longer infeasible to run on frontier LLMs with enterprise-levels of compute and storage. However, these techniques have not been tested on larger models, and the accura

...