trentmkelly/LessWrong-43k huggingface dataset:
Interpretability as Compression: Reconsidering SAE Explanations of Neural Activations with MDL-SAEs
by Kola Ayonrinde — 2024-08-23T18:52:31.019Z

This work was produced as part of the ML Alignment & Theory Scholars Program - Summer 24 Cohort, under mentorship from Lee Sharkey and Jan Kulveit.

Note: An updated paper version of this post can be found here.

Abstract

Sparse Autoencoders (SAEs) have emerged as a useful tool for interpreting the internal representations of neural networks. However, naively optimising SAEs on the for reconstruction loss and sparsity results in a preference for SAEs which are extremely wide and sparse.

To resolve this issue, we present an information-theoretic framework for interpreting SAEs as lossy compression algorithms for communicating explanations of neural activations. We appeal to the Minimal Description Length (MDL) principle to motivate explanations of activations which are both accurate and concise. We further argue that interpretable SAEs require an additional property, “independent additivity”: features should be able to be understood separately. We demonstrate an example of applying MDL by training SAEs on MNIST handwritten digits and find that SAE features representing significant line segments are optimal, as opposed to SAEs with features for memorised digits from the dataset or small digit fragments. We argue that MDL may avoid potential pitfalls with naively maximising sparsity such as undesirable feature splitting and suggests new hierarchical SAE architectures which provide more concise explanations.

1. Intro

Sparse Autoencoders

(SAEs) provide a way to disentangle the internal activations of a neural network into a human-understandable representation of the features. They have recently become popular for use in Mechanistic Interpretability in order to interpret the activations of foundation models, typically language models (

Sharkey et al 2022

;

Cunningham et al., 2023

;

Bricken et al. 2023

). It has been speculated that the features that SAEs find may closely map onto the constituent features that the model is using.

Human interpretability, however, is difficult to optimise for. Evaluating the quality of SAEs often requires potentially unreliable auto-interpretability methods since human interpretability ratings cannot be efficiently queried. And since neither naive human nor LLM-as-judge interpretability methods are differentiable at training time, researchers often use the L0 or "sparsity" as a proxy for interpretability. We provide an alternative a...
How I started believing religion might actually matter for rationality and moral philosophy
by zhukeepa — 2024-08-23T17:40:47.341Z

After the release of Ben Pace's extended interview with me about my views on religion, I felt inspired to publish more of my thinking about religion in a format that's more detailed, compact, and organized. This post is the first publication in my series of intended posts about religion.

Thanks to Ben Pace, Chris Lakin, Richard Ngo, Damon Pourtahmaseb-Sasi, Marcello Herreshoff, Renshin Lauren Lee, Mark Miller, and Imam Ammar Amonette for their feedback on this post, and thanks to Kaj Sotala, Tomáš Gavenčiak, Paul Colognese, and David Spivak for reviewing earlier versions of this post. Thanks especially to Renshin Lauren Lee and Imam Ammar Amonette for their input on my claims about religion and inner work, and Mark Miller for vetting my claims about predictive processing.

In Waking Up, Sam Harris wrote:[1]

But I now knew that Jesus, the Buddha, Lao Tzu, and the other saints and sages of history had not all been epileptics, schizophrenics, or frauds. I still considered the world’s religions to be mere intellectual ruins, maintained at enormous economic and social cost, but I now understood that important psychological truths could be found in the rubble.

Like Sam, I’ve also come to believe that there are psychological truths that show up across religious traditions. I furthermore think these psychological truths are actually very related to both rationality and moral philosophy. This post will describe how I personally came to start entertaining this belief seriously.

“Trapped Priors As A Basic Problem Of Rationality”

“Trapped Priors As A Basic Problem of Rationality” was the title of an AstralCodexTen blog post. Scott opens the post with the following:

Last month I talked about van der Bergh et al’s work on the precision of sensory evidence, which introduced the idea of a trapped prior. I think this concept has far-reaching implications for the rationalist project as a whole. I want to re-derive it, explain it more intuitively, then talk about why it might be relevant for things like intellectual, political and religious biases.

The post describes Scott's take on a predictive processing account of a certain kind of cognitive flinch that prevents certain types of sensory input from being perceived accurately, leading to beliefs that are resistant to updating.[2] Some illustrative central examples of trapped priors:

Karl Friston has writt

...
Invitation to lead a project at AI Safety Camp (Virtual Edition, 2025)
by Linda Linsefors — 2024-08-23T14:18:24.327Z

Do you have AI Safety research ideas that you would like to work on with others? Is there a project you want to do and you want help finding a team? AI Safety Camp could be the solution for you!

Summary

AI Safety Camp Virtual is a 3-month long online research program from January to April 2025, where participants form teams to work on pre-selected projects. We want you to suggest the projects!

If you have an AI Safety project idea and some research experience, apply to be a Research Lead. If accepted, we offer some assistance to develop your idea into a plan suitable for AI Safety Camp. When project plans are ready, we open up team member applications. You get to review applications for your team, and select who joins as a team member. From there, it’s your job to guide work on your project.

Who is qualified?
We require that you have some previous research experience. If you are at least 1 year into a PhD or if you have completed an AI Safety research program (such as a previous AI Safety Camp,

PIBBSS

,

MATS

, and similar), or done a research internship with an AI Safety org, then you are qualified already. Other research experience can count, too.

More senior researchers are of course also welcome, as long as you think our format of leading an online team inquiring into your research questions suits you and your research.

We require that all Research Leads are active participants in their projects and spend at least 10h/week on AISC.

Apply here

If you are unsure, or have any questions you are welcome to:

Book a call with

Robert

Send an email to

Robert

Choosing project idea(s)

AI Safety Camp is about ensuring future AIs are either reasonably safe or not built at all.

We welcome many types of projects including projects aimed at stopping or pausing AI development, aligning AI, deconfusion research, or anything else you think will help make the world safer from AI. If you like, you can read more of

our perspectives on AI safety

, or look at

past

projects

.
If you already have an idea for what project you would like to lead, that’s great.

Apply

with that one!

However, you don’t need to come up with an original idea. What matters is you understanding the idea you want to work on, and why. If you base your proposal on someone else's idea, make sure to cite them.

For ideas on stopping harmful AI, see

here

and/or

...
If we solve alignment, do we die anyway?
by Seth Herd — 2024-08-23T13:13:10.933Z

Epistemic status: I'm aware of good arguments that this scenario isn't inevitable, but it still seems frighteningly likely even if we solve technical alignment. Clarifying this scenario seems important.

TL;DR: (edits in parentheses, two days after posting, from discussions in comments )

If we solve alignment, it will probably be used to create AGI that follows human orders.
If takeoff is slow-ish, a pivotal act that prevents more AGIs from being developed will be difficult (risky or bloody).
If no pivotal act is performed, AGI proliferates. (It will soon be capable of recursive self improvement (RSI)) This creates an n-way non-iterated Prisoner's Dilemma where the first to attack, probably wins (by hiding and improving intelligence and offensive capabilities at a fast exponential rate).
Disaster results. (Extinction or permanent dystopia are possible if vicious humans order their AGI to attack first while better humans hope for peace.)
(Edit later: After discussion and thought, the above seems so inevitable and obvious that the first group(s) to control AGI(s) will probably attempt a pivotal act before fully RSI-capable AGI proliferates, even if it's risky.)

The first AGIs will probably be aligned to take orders

People in charge of AGI projects like power. And by definition, they like their values somewhat better than the aggregate values of all of humanity. It also seems like there's a pretty strong argument that Instruction-following AGI is easier than value aligned AGI. In the slow-ish takeoff we expect, this alignment target seems to allow for error-correcting alignment, in somewhat non-obvious ways. If this argument holds up even weakly, it will be an excuse for the people in charge to do what they want to anyway.

I hope I'm wrong and value-aligned AGI is just as easy and likely. But it seems like wishful thinking at this point.

The first AGI probably won't perform a pivotal act

In realistically slow takeoff scenarios, the AGI won't be able to do anything like make nanobots to melt down GPUs. It would have to use more conventional methods, like software intrusion to sabotage existing projects, followed by elaborate monitoring to prevent new ones. Such a weak attempted pivotal act could fail, or could escalate to a nuclear conflict.

Second, the humans in charge of AGI may not have the chutzpah to even try such a thing. Taking over the world is not for t...
What's going on with Per-Component Weight Updates?
by 4gate — 2024-08-22T21:22:46.728Z

Hi all, this is my first post on LW. It's a small one, but I want improve my writing, get into the habit of sharing my work, and maybe exchange some ideas in case anyone has already gotten further along some projection of my trajectory.

TLDR: I looked at the L2 norm of weight updates/changes to see if it correlates with Grokking. It doesn't seem to trivially, but something non-obvious might be happening.

What This Is

In this post I'm mainly just sharing a small exploration I did into the way weights change over training. I was inspired by some of the older Grokking/phase change work (i.e. on modular addition and induction heads). Broadly, this previous work finds that sometimes deep learning models suddenly "grok"—a phenomenon in which the model suddenly starts to improve its performance after exhibiting diminishing returns, usually associated with some algorithmic improvement in how it processes/represents data as well as potentially the usage of composition. My guess is that Grokking occurs when components in a model find a way to compose, creating a virtuous cycle of gradient updates towards a new algorithmic paradigm. My guess is also that on some level, once some concept has been Grokked, its substrate (roughly) ceases to change and in the short term other components, instead, change to be able to best utilize the concept. For example, in vision models I'm guessing that some of the first components to be learned are simple edge detectors and color/frequency detectors, and that once they are learned, they change little and most gradient updates affect users of those components. AFAIK some research supports these ideas[1], but I don't think it's conclusive. If this hypothesis is true, we should be able to see that for known phase changes, the gradient updates

per-component

become diminished for the components that grokked around the same time the grokking occurs and so I went out to test two toy examples: one based on the great induction heads tutorial from transformer lens and one based on the default Pytorch ResNet.

The rest of this post is structured in the following way:

Why We Might Care About This
Scope of This Exploration
The ResNet
Induction Heads
Discussion

Why We Might Care About This

I think this question of Grokking is primarily of scientific interest, and so any benefits would be further downstream. However, you could in principle use a grokking detecto...
Interoperable High Level Structures: Early Thoughts on Adjectives
by johnswentworth — 2024-08-22T21:12:38.223Z

Meta: This post is a relatively rough dump of some recent research thoughts; it’s not one of our more polished posts, in terms of either clarity or rigor. You’ve been warned.

The

Interoperable Semantics

post and the

Solomonoff Inductor Walks Into A Bar

post each tackled the question of how different agents in the same world can coordinate on an ontology, so that language can work at all given only a handful of example usages of each word (similar to e.g. children learning new words). Both use

natural latents

as a central mathematical tool - one in a Bayesian probabilistic framework, the other in a minimum description length framework. Both focus mainly on nouns, i.e. interoperable-across-minds clusters of “objects” in the environment.

… and the two propose totally different models. In one, the interoperability of cluster labels (i.e. nouns) follows from natural latent conditions over different features of each object. In the other, interoperability follows from natural latent conditions across objects, with no mention of features. The two models are not, in general, equivalent; they can’t both be both correct and complete.

In this post, we’ll propose that while the natural latent conditions over objects still seem to intuitively capture the rough notion of nouns, the natural latent conditions over features seem much better suited to adjectives. We’ll briefly lay out two different potential ways to use natural latents over features as semantic values for adjectives. Then we’ll talk a bit about implications, open threads and how this fits into a broader research gameplan.

The Problem

When children learn language, the cognitive process seems to go:

Observe the world a bunch
… organize knowledge of the world according to some categories, concepts, ontology, etc
… those categories, concepts, ontology, etc match other humans’ categories, concepts, ontology, etc reasonably well
… so it only takes a handful of examples (1-3, say) of the use of a given word in order for the child to learn what the word refers to.

The crucial point here is that the categories/concepts/ontology are mostly learned before a word is attached; children do not brute-force learn categories/concepts/ontology from “labeled data”. We can tell this is true mainly because it typically takes so few examples to learn the meaning of a new word.

The big puzzle, then,...
Interest poll: A time-waster blocker for desktop Linux programs
by nahoj — 2024-08-22T20:44:04.479Z

I started developing a program à la LeechBlock / Cold Turkey / OneSec to:

limit how long I can use a program per day/week/etc.

add a short delay "Take a breath. Do you really want to run Xyz?" before starting a program.

Right now it's unfinished and unstable. I might invest some time into it if there is interest.

Poll in comments.
Turning 22 in the Pre-Apocalypse
by testingthewaters — 2024-08-22T20:28:25.794Z
Link: https://utilityhotbar.github.io/signal_flare/1.html

Meta comment for LessWrong readers[1]

Something Different This Way Comes - Part 1

In which I attempt to renegotiate rationalism as a personal philosophy, and offer my alternative—Game theory is not a substitute for real life—Heuristics over theories

Introduction

This essay focuses on outlining an alternative to the ideology of rationalism. As part of this, I offer my definition of the rationalist project, my account of its problems, and my concept of a counter-paradigm for living one’s life. The second part of this essay will examine the political implications of rationalism and try to offer an alternative on a larger scale.

Defining Rationalism

To analyse rationalism, I must first define what I am analysing. Rationalism (as observed in vivo on forums like LessWrong) is a loose constellation of ideas radiating out of various intellectual traditions, amongst them Bayesian statistics, psychological decision theories, and game theory. These are then combined with concepts in sub-fields of computer science (AI and simulation modelling), economics (rational actor theory or homo economicus), politics (libertarianism), psychology (evolutionary psychology) and ethics (the utilitarianism of Peter Singer).

The broad project of rationalism aims to generalise the insights of these traditions into application at both the “wake up and make a sandwich” and the “save the world” level. Like any good tradition, it has a bunch of contradictions embedded: Some of these include intuitionism (e.g. when superforecasters talk about going with their gut) vs deterministic analysis (e.g. concepts of perfect game-players and k-level rationality). Another one is between Bayesianism (which is about updating priors about the world based on evidence received, generally without making any causal assumptions) vs systemisation (which is about creating causal models/higher level representations of real life situations to understand them better). In discussing this general state of rhetorical confusion I am preceded by Philip Agre’s Towards a Critical Technical Practice, which is AI specific but still quite instructive.

The broader rationalist community (especially online) includes all sorts of subcultures but generally there are in group norms that promote certain technical argot (“priors”, “updating“), certain attitudes towards classes of entities (“blank faces“/bureaucrats/NPCs/the woke mob etc), an...
A Robust Natural Latent Over A Mixed Distribution Is Natural Over The Distributions Which Were Mixed
by johnswentworth — 2024-08-22T19:19:28.940Z

This post walks through the math for a theorem. It’s intended to be a reference post, which we’ll link back to as-needed from future posts. The question which first motivated this theorem for us was: “Redness of a marker seems like maybe a natural latent over a bunch of parts of the marker, and redness of a car seems like maybe a natural latent over a bunch of parts of the car, but what makes redness of the marker ‘the same as’ redness of the car? How are they both instances of one natural thing, i.e. redness? (or ‘color’?)”. But we’re not going to explain in this post how the math might connect to that use-case; this post is just the math.

Suppose we have multiple distributions P1,…,Pk over the same random variables X1,…,Xn. (Speaking somewhat more precisely: the distributions are over the same set, and an element of that set is represented by values (x1,…,xn).) We take a mixture of the distributions: P[X]:=∑jαjPj[X], where ∑jαj=1 and α is nonnegative. Then our theorem says: if an approximate natural latent exists over P[X], and that latent is robustly natural under changing the mixture weights α, then the same latent is approximately natural over Pj[X] for all j.

Mathematically: the natural latent over P[X] is defined by (x,λ↦P[Λ=λ|X=x]), and naturality means that the distribution (x,λ↦P[Λ=λ|X=x]P[X=x]) satisfies the

naturality conditions

(mediation and redundancy).The theorem says that, if the joint distribution (x,λ↦P[Λ=λ|X=x]∑jαjPj[X=x]) satisfies the naturality conditions robustly with respect to changes in α, then (x,λ↦P[Λ=λ|X=x]Pj[X=x]) satisfies the naturality conditions for all j. “Robustness” here can be interpreted in multiple ways - we’ll cover two here, one for which the theorem is trivial and another more substantive, but we expect there are probably more notions of “robustness” which also make the theorem work.

Trivial Version

First notion of robustness: the joint distribution (x,λ↦P[Λ=λ|X=x]∑jαjPj[X=x]) satisfies the naturality conditions to within ϵ for all values of α (subject to ∑jαj=1 and α nonnegative).

Then: the joint distribution (x,λ↦P[Λ=λ|X=x]∑jαjPj[X=x]) satisfies the naturality conditions to within ϵ specifically for αj=δjk, i.e....
what becoming more secure did for me
by Chris Lakin — 2024-08-22T17:44:48.525Z
Link: https://chrislakin.blog/p/becoming-more-secure-mfer

After I quit my job two years ago, I was conflict-avoidant to the point of depression. I did ~nothing for five months and moved in with my parents in the middle of nowhere.

Social conflicts used to rip me up. I would be anxious for days, sometimes months.

I was so avoidant of feelings I didn’t know they manifested in the body!

Then, with a nudge from a good friend and help from a skilled coach, I began some rapid growth.

And I didn’t have the terms for it yet, but I began deconstructing my emotional insecurities one-by-one:

Becoming more secure

As I became less emotionally insecure, my depression, social anxiety, and avoidance fell away. I found healthier strategies, so they were no longer useful defense mechanisms.

With a secure baseline, I moved cities, pivoted careers, and began building a much more aligned life for myself. Since then:

I direct my own social interaction research, like this post, funded by donors.
I’ve produced small AI safety workshops with attendees like davidad.
I run unhinged, sometimes intentionally awkward social events. <Detailed post coming soon>
…

Previously, I would have been too afraid to do these things mostly for fear of getting into conflict with others. And while I still get into many similar situations, they no longer register as “conflicts” that I feel avoidant of or anxious about.

(I was also fortunate to have some financial security during this time in the form of ~6 months of runway.)

Boundaries are intuitive to me now. I couldn’t tell you the last time I did something because of guilt or manipulation by others.

(I’m actually somewhat conflict-seeking now. Conflicts are often fun and informative!)

Even my cuticles became securely attached!

Beginning ~7 years ago, my cuticles were consistently disheveled. I would pick at them when I felt anxious. But sometime within the last year, my cuticles completely healed. This was not the result of any new skincare methods and not the result of “willpower”. I think I’m just significantly less anxious so I don’t pick at them anymore.

I’m much more present and I flinch away from many fewer aspects of my experience.
(↑ also helped me become more empathetic.)
I fixed my chronic neck/back pain.
I do fewer unwanted self-fulfilling prophecies.
I’m more intuitive and I know more about my body.
I have friends that I like more. E.g.: I’m running a brunch this weekend and,

...