trentmkelly/LessWrong-43k huggingface dataset:
SAEs you can See: Applying Sparse Autoencoders to Clustering
by Robert_AIZI — 2024-10-28T14:48:16.744Z

TL;DR

We train sparse autoencoders (SAEs) on artificial datasets of 2D points, which are arranged to fall into pre-defined, visually-recognizable clusters. We find that the resulting SAE features are interpretable as a clustering algorithm via the natural rule "a point is in cluster N if feature N activates on it".
We primarily work with top-k SAEs (k=1) (as in Gao et al.), with a few modifications:

Instead of reconstructing the original (x,y) points, we embed each point into a 100-dimensional space, based off its distance to 100 fixed "anchor" points. The embedding of a point p for an anchor point a is roughly exp(−d(p,a)2). This embedded point is both the input and target of the SAE. This embedding allows our method to identify features which are non-linear in (x,y).
We use a variant of ghost gradients to push dead features in the correct direction. This greatly improves the reliability of the training.

We achieve great data-efficiency (as low as 50 training data) by training for thousands of epochs.
This approach allows one to "see" SAE features, including their coefficients, in a pleasant way:

Circles are points in the dataset, with color indicating the feature activation. Triangles indicate decoder weights, with larger, redder triangles indicating larger weights. (There are small blue triangles in every diagram, though they may be hard to see.) A triangle is located where its corresponding "anchor point" is.

Introduction

Using Sparse Autoencoders for dictionary learning is fundamentally an unsupervised learning task: given some data, find the important things in it. If SAEs are good at that, they should be able to solve other unsupervised learning problems. Here, I try to use SAEs on a classic unsupervised learning problem: clustering 2D data. The hope is that SAEs can learn features corresponding to "in cluster 1, in cluster 2, etc".

We investigate this on artificial data, and find that SAEs semi-reliably find the correct classification, with interpretable activations and decoder weights.

Methods

Datasets

We made four synthetic datasets, consisting of separate, visually-identifiable clusters.

"Basic Blobs" - 5 clusters. Points (x,y) are drawn from normal distributionsX∼N(xcenter,1),Y∼N(ycenter,1) where (xcenter,ycenter) is the cluster center. The 5 clusters have centers (0,0

...
Bridging the VLM and mech interp communities for multimodal interpretability 
by Sonia Joseph — 2024-10-28T14:41:41.969Z

Cross-posted here.

A note to the LW community

I wrote this post after spending my summer in the MATS community looking at sparse autoencoders on vision models, and then joining FAIR/Meta as a visiting researcher on their video generation team. It has been absolutely fascinating navigating both the cultural and research differences, but that is a subject for another post. (During my time at MATS, I got into several lively debates with other cohort members about going to Meta after the program. I would be more than happy to discuss my rationale for doing so in a separate post! But for this post, I will keep the focus on the research itself, and I would appreciate it if any subsequent discussion could follow suit.)

The purpose of this post is to bridge my high-level thoughts around the two research communities regarding multimodal interpretability. While SAE research is basically born on AF/LW, I found that FAIR/Meta is mostly unfamiliar with mech interp, although there are a few researchers who follow the subfield quite closely. FAIR/Meta seems instead broadly known for its expertise in vision/video/multimodal research, including famously disliking LLM research compared to the industry mainstream. That said, FAIR/Meta has been working on its own version of interpretability for awhile. Consider this post a researcher "adapter" post between two communities.

This post is also a progress update on my thinking around multimodal interpretability, as part of a broader series. It is a snapshot of a rapidly evolving field, synthesizing my opinions, my preliminary research, and recent literature.

This post is selective, not exhaustive, and omits many significant papers. I've focused on works that particularly resonate with my current thinking. Nor does this post fully represent my broader research agenda, but rather frames a few directions that I believe to be significant.

More comments to the LW community

This post was originally written for the FAIR-adjacent audience, so the section "Circuit-based methods" may be largely redundant, and tonally more skeptical than if I were writing to the LW community directly.
I'm pretty bearish on prioritizing LLM research over research of other modalities (which is also the philosophy of FAIR/Meta right now), which may come through.

Overview

Multimodal interpretability methods:

Circuit-based methods
Text-image space methods
Captioning meth

...
How Likely Are Various Precursors of Existential Risk?
by NunoSempere — 2024-10-28T13:27:31.620Z
Link: https://blog.sentinel-team.org/p/how-likely-are-various-precursors

Seven

Samotsvety

forecasters give their probabilities for different possible catastrophes to lead to more than 1M direct deaths in any one year, in any of the next 10 years. This post presents results, starting with a table which presents the probabilities, continuing with a discussion of each of the items, and concluding with some thoughts on relevance for

Sentinel

’s work averting or mitigating incipient calamities.

Some forecasters found some questions in particular much more interesting, so we alternate between presenting the probability of the best model when one such exists, and the geometric mean of the odds when it’s a risk that requires more subjective judgment and we have enough judgments to aggregate. Events that could fall into different buckets can resolve more than one bucket. For instance, an AI-invented biological weapon used during WW3 would resolve the “AI”, “biological weapons” and “WW3” buckets.

Summary

Calamity	Probability	Range	Type of estimate
War	23%	18% to 40%	Forecaster aggregate
Natural pandemic	22%	10% to 30%	Forecaster aggregate
The unexpected	21%	15% to 30%	Forecaster aggregate
WW3 specifically	12%	5% to 40%	Forecaster aggregate
AI	10%	1% to 40%	Forecaster aggregate
Unintentional biochem	2.4%	0.8% to 10%	Forecaster aggregate
Biological weapons	2.2%	2% to 2.5%	Forecaster aggregate
Climate change	2.5%	1% to 5%	Forecaster aggregate
Volcanoes	1%	0.2% to 2%	Best model
US civil war	0.92%	0.05% to 3%	Forecaster aggregate
Financial crisis	0.59%	0.2% to 2%	Forecaster aggregate
Solar storm	0.2%	0.00066% to 0.71%	Best model
Simulation shutdown	0.13%	0.001% to 3%	Forecaster aggregate
Asteroid impact	0.02%	-	Best model
Alien invasion	0.00014%	0.00000025% to 0.1%	Forecaster aggregate
Vacuum decay, weird physics	<~1e-8	-	Best model

Ranges represent either the lowest and highest probabilities given by forecasters, or the 90% confidence interval if only using the best model.

War

Type of estimate: Geometric mean of forecaster probabilities

Probabilities: 20%, 20%, 40%, 25%, 21.5%,

Range: 20% to 40%

Aggregate: 25%

Forecaster comments:

For context, Ukraine war is ~200k deaths over 2 years. WW1 and WW2 were 10M to 100M deaths

Baserate: WW1 and WW2 for last century are already massive wars, giving baserate of 2/100 years. Specific spots: 15% for China/Taiwan leading to a conflict. But it doesn’t necessarily lead to 1M deaths. Say ~40% that it does
For reference, the Ukraine war

hasn’t reached

1M/year. I think I was previ

...
Care Doesn't Scale
by stavros — 2024-10-28T11:57:38.742Z
Link: https://stevenscrawls.com/care-doesnt-scale/

If you wanted to design a social system to care for children who have lost their parents, I don’t know if you could do much better. With four children, each kid can get individual care and attention, but there were four social workers each had three 24-hour blocks per week, so they had time to have their own lives with enough flexibility to take vacations and sometimes have two workers with the kids instead of one.

To get that individualized care, though, they had four social workers and four children. One-to-one.

Of course, you could probably add a few more children, or subtract a social worker, as a cost-saving measure. It’d be less sustainable, but it wouldn’t significantly change the experience. But you couldn’t stray that far from one-to-one without changing the nature of the experience, without industrializing it to the point that individual care is lost. With four kids, the kids can feel like kids; if there were forty kids, they’d probably feel like they were cattle.

We’re pretty limited when it comes to care. In any given moment, you can only really care deeply and individually for one person.

Not the OP, just wanted to start a discussion about this here.
Some thoughts of my own:
I'm reminded that we live in a thoroughly maladaptive world, that 'normal' is everyone doing the best they can despite some/many/all of their most fundamental needs - such as care - being insufficiently met.

I think, to the extent the title of this essay is true, it seems a neglected and much ignored truth. I notice that my world models assume a very finite capacity for caring, yet this awareness is not present in my conscious thoughts and strategies and my day to day interactions - I routinely overestimate my own capacity for care, and rarely notice when my own need for care is insufficiently met.
I notice that I do not care skillfully, and worryingly I don't know that I would recognise such skill in others.
Your memory eventually drives confidence in each hypothesis to 1 or 0
by Crazy philosopher — 2024-10-28T09:00:27.084Z

Our memory tends to contain less and less information. We forget certain things, and our memory about others become simplified, and a complex article boils down to “X is bad, Y is good, try to do better".

One unexpected consequence of this is how it impacts our sense of probability: to describe the probability binary, it only requires 1 bit of information, but to describe the probability as a percentage, you need a lot more bits!

Because of this, a person's confidence in the most likely hypothesis in his opinion tends to 1 over time, and the probability of all other hypotheses that he had time to think about tends to 0. Every time he remembers a hyp, a person will less and less often remember the nuance that “the probability of hyp A = X”, he will simply remember “And this is the truth (that mean the probability of A = 100%)”.

(Edit: it works this way for some people only. Others understand that just because they remember something as "true," it doesn’t mean their past self believed it with 100% certainty. I wrote this article to everyone do like "others")

Another way certainty in a hyp may grow over time is when we forget its weak spots—the things we should think about to test it.

I’ve often found myself in a cycle like this: I study a hyp and feel, “I’m not entirely sure, but it seems true.” Later, I forget about this feeling, and eventually, I start thinking of the hypothesis as true, almost like the gravity.

Therefore, just after thinking about a hyp, you should write down your confidence and parts you are not sure about. Also, you should to remember this bias to notice it, when you use a hypothesis you thought a lot of time ago.

TL;DR: memory is bad, writing is good, try to do better.
Nerdtrition: simple diets via spreadsheet abuse
by dkl9 — 2024-10-27T21:45:15.117Z
Link: https://dkl9.net/essays/nerdtrition.html

I found several simple (four to seven total foods) diets that, with carefully specified amounts of each food, appear, at face value, to provide full, proper nutrition. Here are a few, with amounts given per day.

fish/olive/etc

450 g cooked tilapia
450 g ripe olives (yes, that's a lot)
200 g romaine lettuce
200 g cooked lentils
400 g whole-wheat bread (roughly 13 slices)
100 g sweet yellow pepper (half a pepper to one pepper)

tuna/kale/etc

300 g canned solid-white tuna
150 g walnuts
400 g kale (yes, that's a lot)
700 g boiled pinto beans

chicken/kale/etc

200 g chicken drumstick (roughly two drumsticks)
100 g mozzarella cheese
300 g flaxseed
300 g kale

salmon/broccoli/etc

400 g cooked salmon
30 g cheddar cheese (small amount)
200 g almonds (yes, that's a lot)
300 g broccoli
100 g carrots
600 g boiled pinto beans

Each of these is probably deficient in some small (but crucial) ways. Check for yourself before committing to one.

Under stricter circumstances that demand calculated laziness, I might actually follow one of these diets. In practice, I ignore the details and eat with more variety. My diet — as I strive, at least — is based on a food-group heuristic averaged from these specifics of "nerdtrition" (nerd nutrition):

300 g to 400 g of animal products (ideally fish, second-best dairy)
150 g to 500 g of fatty plants (nuts, avocados, olives, etc)
400 g to 700 g of starchy plants (whole-wheat bread, beans, and potatoes being some better ones)
200 g to 400 g of nutritionally-complex vegetables (kale, broccoli, spinach, etc)
0 g to 200 g of fruit (sweet peppers being one of the better ones)

All else, I see as superfluous, including, especially, processed foods.

Methods

Compile a list of nutrients to keep track of.

macronutrients

water
saturated fat
unsaturated fat
total carbohydrates
sugars
fibre
protein
cholesterol

minerals — here, only those with common deficiency diseases

magnesium (Mg)
potassium (K)
calcium (Ca)
iron (Fe)
zinc (Zn)

vitamins — likewise, only common deficiencies

A (retinol/carotenoid)
B6 (pyridoxines)
B9 (folate)
B12 (cobalamin)
C (ascorbic acid)
D (calciferols)

Determine a minimum (RDA) and maximum (UL) for each nutrient. I mostly got these from conventional sources, like the NIH Office of Dietary Supplements, and adjusted a few of them based on more recent and niche findings:

favour more fat over carbs
avoid sugar, seriously
get enough fibre, seriously
there may have been others

Compile a list of plausi

...
AGI Fermi Paradox
by jrincayc — 2024-10-27T20:14:54.490Z

Enrico Fermi originally asked in 1950, "Where are the aliens? Now, in 2024, it is becoming more of a question of: "Where are the AGIs?" Essentially, we have the computing power (the 2011 Watson computer could certainly have trained LaMDA and other large language models; the worlds most powerful supercomputer could almost certainly do direct brain emulation, ( [https://www.lesswrong.com/posts/9kvpdK9BLSMxGnxjk/thoughts-on-hardware-limits-to-prevent-agi] ) so it seems like it is time to ask where are all the Artificial General Intelligences?

Some possible explanations for why we have not seen any:

We haven't managed to program them yet. This is certainly possible.

AGI(s) exist, but are hiding. This could be because the AGI is waiting to become powerful enough to safely revel verself, or the AGI has other reasons such as to avoid causing panic. It is also possible that we have noticed the AGI, but haven't realized that it is an AGI (for example, we might think the AGI is just a botnet.).

An External entity is preventing AGIs. This could including a AGI that is hiding, but doesn't want competition or extraterrestrials could be preventing an AGI because it might be dangerous, or even that we exist in a simulation that doesn't want to run the amount of compute needed for an AGI.

AGIs are sufficiently deadly that survivership bias means we only exist in worlds where AGI has failed. If AGIs are almost always deadly, then we would not expect to be on a world where they happen, since the AGI would wipe us out before we observe this.

It is possible that Computers are not powerful enough for AGI, or at least not powerful enough to easily create an AGI.

I think those five are the main possibilities: 1. not programmed yet, 2. hiding, 3. being prevented, 4. almost always deadly and 5, computers not powerful enough.

Here are some more quick thoughts on computing power. Here is a somewhat arbitrary list of different computers, from less powerful to more powerful with RAM and number of floating point operations per second listed. Each is about 1000 times more powerful than the previous.

Commodore 64 (64 KiB, 25 kFLOPS)

Cray 1 (8 MiB, 160 MFLOPS)

RaspberryPi 4B (4 GiB, 13.5 GFLOPS)

Watson (16 TiB, 80 TFLOPS)

Frontier supercomputer (9 PiB, 1.2 EFLOPS)

For (1) it is probably impossible to build an AGI on a single Commodore 64, just because it is easy to find problems that w...
Substituting Talkbox for Breath Controller
by jefftk — 2024-10-27T19:10:03.768Z

One of the inputs of my

rhythm stage
setup has been a breath controller, which let's me gives me a continuous
controller. I use it for a few different things, especially when
playing music that's farther in the electronic direction:

Setting up to play a "techno
contra" last weekend, however, I realized my breath controller was
broken. It seemed like something was wrong with the sensor: only very
large pressures would do anything, and it was inconsistent. Maybe a
little bit of dirt got inside? I tried taking it apart, but the
sensor looks to be sealed:

I might get a new sensor and try to fix it, though with the big
surface mount pads probably only if I can get access to hot air rework
equipment. And this was definitely not something I'd be able to do in
the half hour before the dance.

I thought some about what role the breath controller fills in my
sound, and the main thing is an electronic-feeling off-beat pulse.
This is something I'd made my
system able to do a while ago, but hadn't liked because it was too
rigid feeling. I had the idea of sending it out via the talkbox,
however, and it ends up sounding like this:

It's not the same, and I do still miss the breath controller's ability
to add a more complex rhythmic component on top of my keyboard
playing, but it's new sound I like and something I expect I'll keep
playing with even if I fix or replace the breath controller.
Open Source Replication of Anthropic’s Crosscoder paper for model-diffing
by Connor Kissane — 2024-10-27T18:46:21.316Z

Intro

Anthropic recently released an exciting mini-paper on crosscoders (

Lindsey et al.

). In this post, we open source a model-diffing crosscoder trained on the middle layer residual stream of the Gemma-2 2B base and IT models, along with code, implementation details / tips, and a replication of the core results in Anthropic’s paper.

While Anthropic highlights several potential applications of crosscoders, in this post we focus solely on “model-diffing”. That is, localizing and interpreting a small “diff” between two different models. We think this is a particularly exciting application, because it can let us examine what changed as a model was fine-tuned, which seems likely to capture most safety-relevant circuitry, while leaving out many less relevant capabilities.

In their paper, they find exciting preliminary evidence that crosscoders identify shared sets of features across different models, as well as features specific to each model. While it’s still an open question how useful crosscoders will be for model-diffing, they show significant signs of life, and we’re excited to see the community build on this open source replication to explore them further.

TLDR;

We train and open source a 16K latent crosscoder to model diff the Gemma 2 2B Base and IT models at the middle layer residual stream. Download the weights at

https://huggingface.co/ckkissane/crosscoder-gemma-2-2b-model-diff

See this

colab demo

to load and use the autoencoder
We also open source a scrappy training codebase at

https://github.com/ckkissane/crosscoder-model-diff-replication

along with some implementation details + tips for training your own

Anthropic’s core results replicate: the pair of decoder vector norms for each latent cluster into three main groups: “shared” (norms are similar), “base model specific” (only base model norm is large), and “chat model specific” latents (only chat model norm is large). The “shared” latents have highly aligned decoder vectors between models.
We do some standard SAE-style evals. On average, the crosscoder has 81 L0, 77% explained variance, and 95% loss recovered relative to zero ablation on the training distribution.
We perform some shallow explorations into latents from each of the “shared”, “base model specific”, and “chat model specific” latents. We use latent dashboard visualizations of the maximum activating examples (introduced by

...
Hiring a writer to co-author with me (Spencer Greenberg for ClearerThinking.org)
by spencerg — 2024-10-27T17:34:50.479Z

Hi all! I'm looking to hire a writer for 3-5 hours per week (who can put in that time reliably nearly every week) to co-author writing projects with me (on topics like psychology, philosophy, and society). I'm looking for someone who:

1) writes compelling yet logical (and well-reasoned) prose

2) produces their writing quickly

Most of this writing would go out to our Clearer Thinking newsletter (240,000 subscribers). If someone you know might be interested, the application process for the role is simple - just submit two writing samples using the link below in the comments.

Apply now

Thanks!
Interview with Bill O’Rourke - Russian Corruption, Putin, Applied Ethics, and More
by JohnGreer — 2024-10-27T17:11:28.891Z

This is cross-posted from my blog and I interviewed him for a general audience rather than LessWrong people.

I was lucky enough to interview Bill O’Rourke.

Imagine you’re on a plane to Russia to become the new president of an aluminum factory. Conditions are abysmal. People are dying every year due to unimaginable safety conditions. Corruption is everywhere from the mayor on down. You’ll have to deal with oligarchs who’ve assassinated people. You’ll even be robbed by the cops.

What would you do? How would you fix things?

This is the position Bill O’Rourke found himself in.

I first met Bill when we toured the Carrie Furnaces in Pittsburgh. I liked Bill right away and knew I needed to record an interview with him.

We talked about everything from growing up in Steel City in its heyday, to teaching Delta Force operators about ethics, to turning around a Russian factory that killed five people a year, to Putin stealing a Super Bowl ring.

Bill has not only led an incredibly interesting life but he’s also a man of integrity and character (and that’s something I would rarely say!).

I interviewed him for a general audience but some talking points that may be more interesting to rationalists:

* He’s been in a unique position to face real ethical questions with strong incentives to act in less ethical ways and was successful in solving the problems

* The differences between Bill’s ethical framework and most rationalists

* Improving systems like dangerous factories, and how Bill used advisors when he wasn’t a subject matter expert

Enjoy the interview!

Introduction: How I Met Bill

0:00

Introduction

2:17

Bill’s background – storied childhood and career!

5:18

Bill’s book: The Business Ethics Field Guide

6:20

Adapting his book for Special Forces
Ethical dilemmas in the military

8:13

How Bill and I met
Why Bill gives tours of Carrie Furnaces

9:38

Family history and working in the steel industry

White hat status symbol

Ironically not being encouraged to go to law school

10:40

Bill’s family, and where he is in the sibling lineup

His dad teaching him ethics, no profanity

How his dad dealt with bribes

13:57

Did Bill play basketball? Being coached by his dad - anti-nepotism

15:13

Did he ever feel like his goody two shoes dad was too ethical?

Didn’t know he was poor

16:10

Being an altar boy, did he have any priest issues?

Was a Boy Scout, any issues there?

Vietnam War

16:44

What made Bill want to go into the military?

Fear o...