trentmkelly/LessWrong-43k huggingface dataset:
A more systematic case for inner misalignment
by Richard_Ngo — 2024-07-20T05:03:03.500Z

This post builds on my previous post making the case that squiggle-maximizers are plausible. The argument I presented was a deliberately simplified one, though, and glossed over several possible issues. In this post I'll raise and explore three broad objections. (Before looking at mine, I encourage you to think of your own biggest objections to the argument, and jot them down in the comments.)

Intelligence requires easily-usable representations

"Intelligence as compression" is an interesting frame, but it ignores the tradeoff between simplicity and speed. Compressing knowledge too heavily makes it difficult to use. For example, it's very hard to identify most macroscopic implications of the Standard Model of physics, even though in theory all of chemistry could be deduced from it. That’s why both humans and LLMs store a huge number of facts and memories in ways that our minds can access immediately, using up more space in exchange for rapid recall. Even superintelligences which are much better than humans at deriving low-level facts from high-level facts would still save time by storing the low-level facts as well.

So we need to draw a distinction between having compressed representations, and having only compressed representations. The latter is what would compress a mind overall; the former could actually increase the space requirements, since the new compressed representations would need to be stored alongside non-compressed representations.

This consideration makes premise 1 from my previous post much less plausible. In order to salvage it, we need some characterization of the relationship between compressed and non-compressed representations. I’ll loosely define systematicity to mean the extent to which an agent’s representations are stored in a hierarchical structure where representations at the bottom could be rederived from simple representations at the top. Intuitively speaking, this measures the simplicity of representations weighted by how “fundamental” they are to the agent’s ontology.

Let me characterize systematicity with an example. Suppose you’re a park ranger, and you know a huge number of facts about the animals that live in your park. One day you learn evolutionary theory for the first time, which helps explain a lot of the different observations you’d made. In theory, this could allow you to compress your knowledg...
BatchTopK: A Simple Improvement for TopK-SAEs
by Bart Bussmann — 2024-07-20T02:20:51.848Z

Work done in Neel Nanda’s stream of MATS 6.0.

Epistemic status: Tried this on a single sweep and seems to work well, but it might definitely be a fluke of something particular to our

implementation

or experimental set-up. As there are also some theoretical reasons to expect this technique to work (adaptive sparsity), it seems probable that for many TopK SAE set-ups it could be a good idea to also try BatchTopK. As we’re not planning to investigate this much further and it might be useful to others, we’re just sharing what we’ve found so far.

TL;DR: Instead of taking the TopK feature activations per token during training, taking the Top(K*batch_size) for every batch seems to improve SAE performance. During inference, this activation can be replaced with a single global threshold for all features.

Introduction

Sparse autoencoders (SAEs) have emerged as a promising tool for interpreting the internal representations of large language models. By learning to reconstruct activations using only a small number of features, SAEs can extract monosemantic concepts from the representations inside transformer models. Recently, OpenAI published

a paper

exploring the use of TopK activation functions in SAEs. This approach directly enforces sparsity by only keeping the K largest activations per sample.

^x=TopK(Wenc(x−bpre))Wdec+bpre

While effective, TopK forces every token to use exactly k features, which is likely suboptimal. We came up with a simple modification that solves this and seems to improve its performance.

BatchTopK

Standard TopK SAEs apply the TopK operation independently to each sample in a batch. For a target sparsity of K, this means exactly K features are activated for every sample.

BatchTopK instead applies the TopK operation across the entire flattened batch:

Flatten all feature activations across the batch
Take the top (K * batch_size) activations
Reshape back to the original batch shape

This allows more flexibility in how many features activate per sample, while still maintaining an average of K active features across the batch.

Experimental Set-Up

For both the TopK and the BatchTopK SAEs we train a sweep with the following hyperparameters:

Model: gpt2-small
Site: layer 8 resid_pre
Batch size: 4096
Optimizer: Adam (lr=3e-4, beta1 = 0.9, beta2=0.99)
Number of tokens: 1e9
Expansion factor: [4, 8, 16, 32]
Target L0 (k): [16, 32, 64]

As in the Open...
Krona Compare
by jefftk — 2024-07-20T01:10:03.994Z

Cross-posted from my NAO
Notebook.

When trying to understand how metagenomic samples differ I often want to
drill down through the taxonomic hierarchy, comparing relative abundances.
I've tried several tools for this, existing and custom, and haven't been all
that happy with any. For most purposes, the tool I like most
is Krona, which shows an
interactive chart. For example, here's Krona showing the results of running
the NAO's v1
metagenomic sequencing pipeline on the
unenriched Hyperion
Treatment Plant samples
from Rothman et
al. 2021:

(interactive version)

What I often wish I had, however, are linked Krona charts, where I
could see multiple samples at once, and drilling down in one sample showed
the corresponding portion of the other samples. After failing to find
something like this, I hacked something together by monkey-patching the
output of Krona. Here's it comparing the samples from several wastewater
treatment plants in the same study:

When I click on "viruses" in any of the plots, all four zoom in on the
viral fraction:

That's a lot
of Tobamovirus!

I only just made this, so it's likely buggy, but if that doesn't put you
off you're welcome to give it a try. The interactive version of the charts
above is here (warning:
60MB) and the generation code is open
source, on
github.

If you're interested in the technical details of how I made this:

It's a wrapper around the ktImportTaxonomy command
from KronaTools.

To get multiple charts on the same page, they're each in their
own iframe, via srcdoc.

There is unfortunately no CSS way of saying "please lay out these
rectangles to take up as much of the viewport as possible while
maintaining an aspect ratio close to 1:1", so I use some awkward JS. It
checks each possible number of columns and takes the one that maximizes
the minimum dimension (width or height) of the charts. Luckily there are
only a few options to consider.

So that the colors match between the charts, each chart on the
page has the data from all the charts. I reach into
each iframe to set the dataset

...
(Approximately) Deterministic Natural Latents
by johnswentworth — 2024-07-19T23:02:12.306Z

Background: Natural Latents: The Math, Natural Latents: The Concepts, Why Care About Natural Latents?, the prototypical semantics use-case. This post does not assume that you’ve read all of those, or even any of them.

Suppose I roll a biased die 1000 times, and then roll the same biased die another 1000 times. Then...

Mediation: The first 1000 rolls are approximately independent of the second 1000 given the bias (to reasonable precision).
Redundancy: I can estimate the die’s bias (to reasonable precision) with high confidence from either the first or second 1000 rolls.

The die’s bias is therefore a natural latent, which means it has various nice properties.

Minimality: The bias is the smallest summary of all the information about the first 1000 rolls relevant to the second 1000 (and vice-versa).
Maximality: The bias is the largest piece of information which can be calculated from the first 1000 rolls and also can separately be calculated from the second 1000 rolls.
Any other variable which satisfies the above properties must tell us (approximately) the same information about the die rolls as the bias.

Furthermore, the bias is a(n approximate) deterministic natural latent: the die’s bias (to reasonable precision) is approximately determined by[1] the first 1000 die rolls, and also approximately determined by the second 1000 die rolls. That implies one more nice property:

Uniqueness: The bias is the unique-up-to(-approximate)-isomorphism latent which has the above properties, making it a natural Schelling point for communication between agents.

We’ve proven all that before, mostly in

Natural Latents: The Math

(including

the addendum

added six months after the rest of the post). But it turns out that the math is a lot shorter and simpler, and easily yields better bounds, if we’re willing to assume (approximate) determinism up-front. That does lose us some theoretical tools (notably the

resampling construction

), but it gives a cleaner foundation for our expected typical use cases (like e.g. semantics). The goal of this post is to walk through that math.

Background Tool: Determinism in Diagrams

We’re going to use diagrammatic proofs, specifically using Bayes nets. But it’s non-obvious how to express (approximate) determinism using Bayes nets, or what rules diagrams follow when determinism is involved, so we’ll walk through that first....
Feature Targeted LLC Estimation Distinguishes SAE Features from Random Directions
by Lidor Banuel Dabbah — 2024-07-19T20:32:15.095Z

Tl;dr: In this post we present the exploratory phase of a project aiming to study neural networks by applying static local learning coefficient (LLC) estimation to specific alterations of them. We introduce a new method named Feature Targeted (FT) LLC estimation and study its ability to distinguish SAE trained features from random directions. By comparing our method to other possible metrics, we demonstrate that it outperforms all of them but one, which has comparable performance.
We discuss possible explanations to our results, our project and other future directions.

Introduction

Given a neural network M and a latent layer within it, L, a central motif in current mechanistic interpretability research is to find functions f:L→R [1] which are features of the model. Features are (generally) expected to exhibit the following properties:

Encode interpretable properties of the input.
Be causally relevant to the computation of the output of the model.
Encode the output of a certain submodule of our model M, i.e. a component, localized in weight space, which is responsible for a specific part of the total computation.

While this is common wisdom, methods for automated feature evaluation usually focus on correlations between the (top) activations of the feature with human (or machine) recognizable interpretations, or on the effect of feature-related interventions on the output of the model. In particular, while the first and second items of the feature characterization above are central in current techniques, the third property, specifically the localized nature of the computation upstream of the feature, is less so[2].

We are currently investigating a direction which fills that gap, and this post shares the findings of the exploratory research we have conducted to validate and inform our approach. More specifically, we operationalized the concept of "weight-localized computation" using the local learning coefficient (LLC) introduced in Lau et al, following the learning coefficient first introduced in the context of singular learning theory. We apply LLC estimation to models associated with our base model and a feature within it, a method we call feature targeted (FT) LLC estimation. In this exploratory work we study FT-LLC estimates of specific models associated with SAE features. Most notably, we have found that:

1. FT-LLC estimate...
JumpReLU SAEs + Early Access to Gemma 2 SAEs
by Senthooran Rajamanoharan — 2024-07-19T16:10:54.664Z
Link: https://storage.googleapis.com/jumprelu-saes-paper/JumpReLU_SAEs.pdf

New paper from the Google DeepMind mechanistic interpretability team, led by Sen Rajamanoharan!

We introduce JumpReLU SAEs, a new SAE architecture that replaces the standard ReLUs with discontinuous JumpReLU activations, and seems to be (narrowly) state of the art over existing methods like TopK and Gated SAEs for achieving high reconstruction at a given sparsity level, without a hit to interpretability. We train through discontinuity with straight-through estimators, which also let us directly optimise the L0.

To accompany this, we will release the weights of hundreds of JumpReLU SAEs on every layer and sublayer of Gemma 2 2B and 9B in a few weeks. Apply now for early access to the 9B ones! We're keen to get feedback from the community, and to get these into the hands of researchers as fast as possible. There's a lot of great projects that we hope will be much easier with open SAEs on capable models!

Jump ReLU Activations

Scenarios where JumpReLU is superior

JumpReLUs are state of the art

Gated SAEs already reduced to JumpReLU activations after weight tying, so this can be thought of as Gated SAEs++, but less computationally intensive to train, and better performing. They should be runnable in existing Gated implementations.

Abstract:

Sparse autoencoders (SAEs) are a promising unsupervised approach for identifying causally relevant and interpretable linear features in a language model’s (LM) activations. To be useful for downstream tasks, SAEs need to decompose LM activations faithfully; yet to be interpretable the decomposition must be sparse – two objectives that are in tension. In this paper, we introduce JumpReLU SAEs, which achieve state-of the-art reconstruction fidelity at a given sparsity level on Gemma 2 9B activations, compared to other recent advances such as Gated and TopK SAEs. We also show that this improvement does not come at the cost of interpretability through manual and automated interpretability studies. JumpReLU SAEs are a simple modification of vanilla (ReLU) SAEs – where we replace the ReLU with a discontinuous JumpReLU activation function – and are similarly efficient to train and run. By utilising straight-through-estimators (STEs) in a principled manner, we show how it is possible to train JumpReLU SAEs effectively despite the discontinuous JumpReLU function introduced in the SAE’s forward pass. Similarly, we use STEs to directly train L0

...
Truth is Universal: Robust Detection of Lies in LLMs
by Lennart Buerger — 2024-07-19T14:07:25.162Z
Link: http://arxiv.org/abs/2407.12831

A short summary of the paper is presented below.

TL;DR: We develop a robust method to detect when an LLM is lying based on the internal model activations, making the following contributions: (i) We demonstrate the existence of a two-dimensional subspace, along which the activation vectors of true and false statements can be separated. Notably, this finding is universal and holds for various LLMs, including Gemma-7B, LLaMA2-13B and LLaMA3-8B. Our analysis explains the generalisation failures observed in previous studies and sets the stage for more robust lie detection; (ii) Building upon (i), we construct an accurate LLM lie detector. Empirically, our proposed classifier achieves state-of-the-art performance, distinguishing simple true and false statements with 94% accuracy and detecting more complex real-world lies with 95% accuracy.

Introduction

Large Language Models (LLMs) exhibit the concerning ability to lie, defined as knowingly outputting false statements. Robustly detecting when they are lying is an important and not yet fully solved problem, with considerable research efforts invested over the past two years. Several authors trained classifiers on the internal activations of an LLM to detect whether a given statement is true or false. However, these classifiers often fail to generalize. For example, Levinstein and Herrmann [2024] showed that classifiers trained on the activations of true and false affirmative statements fail to generalize to negated statements. Negated statements contain a negation like the word “not” (e.g. “Berlin is not the capital of Germany.”) and stand in contrast to affirmative statements which contain no negation (e.g. “Berlin is the capital of Germany.”).

We explain this generalization failure by the existence of a two-dimensional subspace in the LLM's activation space along which the activation vectors of true and false statements separate. The plot below illustrates that the activations of true/false affirmative statements separate along a different direction than those of negated statements. Hence, a classifier trained only on affirmative statements will fail to generalize to negated statements.

The activation vectors of multiple statements projected onto the 2D truth subspace. Purple squares correspond to false statements and orange triangles to true statements.

Importantly, these fi...
Sustainability of Digital Life Form Societies
by Hiroshi Yamakawa — 2024-07-19T13:59:13.973Z

Hiroshi Yamakawa1,2,3,4

1 The University of Tokyo, Tokyo, Japan

2 AI Alignment Network, Tokyo, Japan

3 The Whole Brain Architecture Initiative, Tokyo, Japan

4 RIKEN, Tokyo, Japan

Even in a society composed of digital life forms (DLFs) with advanced autonomy, there is no guarantee that the risks of extinction from environmental destruction and hostile interactions through powerful technologies can be avoided. Through thought-process diagrams, this study analyzes how peaceful sustainability is challenging for life on Earth, which proliferates exponentially. Furthermore, using these diagrams demonstrates that in a DLF society, various entities launched on demand can operate harmoniously, making peaceful and stable sustainability achievable. Therefore, a properly designed DLF society has the potential to provide a foundation for sustainable support for human society.

1. Introduction

Based on the rapid progress of artificial intelligence (AI) technology, an autonomous super-intelligence that surpasses human intelligence is expected to become a reality within the next decade. Subsequently, within several decades to a few hundred years, self-sustaining digital life forms (DLFs) will emerge in the physical world. However, there is no guarantee that such a society will be sustainable. Further, the superintelligence would possess technologies such as weapons of mass destruction and environmental degradation, which encompass the extinction risks currently faced by human society.

DLF societies are anticipated to bolster numerous facets of human life, encompassing enhanced productivity, expanded knowledge, and the maintenance of peace[1]. To ensure the continuity of DLF societies, many complex issues must be addressed, including the sustainable utilization of energy and resources, the judicious governance of self-evolutionary capabilities, and the preservation of the cooperative nature within DLF societies. Nevertheless, their scale and intricacy surpass human understanding, rendering their management by humans fundamentally unfeasible. Consequently, the capacity of DLF societies to sustain themselves autonomously emerges as a critical prerequisite for their role in supporting human societies.

This study shows that appropriate measures can be employed to resolve the existential crises of DLFs attributable to powerful technologies. Thought-process diagrams...
Romae Industriae
by Maxwell Tabarrok — 2024-07-19T13:03:31.536Z
Link: https://www.maximum-progress.com/p/romae-industriae

Whatever each culture grows and manufactures cannot fail to be here at all times and in great profusion. Here merchant vessels arrive carrying these many commodities from every region in every season and even at every equinox, so that the city takes on the appearance of a sort of common market for the world.

Your farmlands are Egypt, Sicily, and all of cultivated Africa. Just as there is a common channel where all waters of the Ocean have a single source and destination, so that there is a common channel to Rome and all meet here: trade, shipping, agriculture, metallurgy— all the arts and crafts that are or ever were and all things that are produced or spring from the earth. What one does not see here does not exist” (

Aristides, The Roman Oration

).

The ancient Roman empire hosted massive cities whose growth relied on large and growing agricultural surplus shipped in along trade networks spanning thousands of miles and tens of millions of inhabitants. It saw

centuries of per capita income

growth,

technological advances

in hydraulics, metallurgy, agriculture and transportation, and stable governance that respected

property rights

.

Adam Smith said “Little else is requisite to carry a state to the highest degree of opulence from the lowest barbarism, but peace, easy taxes, and a tolerable administration of justice.” Rome had all of these and more, but yet did not succeed in cultivating an industrial revolution.

Why not? What was the binding constraint on a Roman industrial revolution?

Coal and Cotton

Historian and blogger Bret Devereaux writes about the lack of Roman industrial revolution

here

. His basic strategy is just to read into the one data point we have on industrialization: Britain. Could Rome have replicated the precursors to industrialization found in Britain?

It is particularly remarkable here how much of these conditions are unique to Britain: it has to be coal, coal has to have massive economic demand (to create the demand for pumping water out of coal mines) and then there needs to be massive demand for spinning (so you need a huge textile export industry fueled both by domestic wool production and the cotton spoils of empire) and a device to manage the conversion of rotational energy into spun thread. I’ve left this bit out for space, but you also need a major incentive for the design of pressure-cylinders (which, in the event, was the demand for better

...
Have people given up on iterated distillation and amplification?
by Chris_Leong — 2024-07-19T12:23:04.625Z

The BlueDot Impact write-up for scalable oversight seems to suggest that people have given up on iterated distillation and amplification (IDA) working. I haven’t really seen much research here, but is that actually the case?
I know that Ought is now pretty much inactive because their attempts at factored cognition failed and this has made many people pessimistic about the factored cognition hypothesis. However, Ought seemed to be really pushing the angle that we could break things into sub-problems each of which could be completed quickly, whilst I don’t think that’s a required part of IDA. Therefore it isn’t clear to me that the failure of factored cognition indicates the failure of IDA.