trentmkelly/LessWrong-43k huggingface dataset:
An Intuitive Explanation of Sparse Autoencoders for Mechanistic Interpretability of LLMs
by Adam Karvonen — 2024-06-25T15:57:16.872Z
Link: https://adamkarvonen.github.io/machine_learning/2024/06/11/sae-intuitions.html

Sparse Autoencoders (SAEs) have recently become popular for interpretability of machine learning models (although SAEs have been around since 1997). Machine learning models and LLMs are becoming more powerful and useful, but they are still black boxes, and we don't understand how they do the things that they are capable of. It seems like it would be useful if we could understand how they work.

Using SAEs, we can begin to break down a model's computation into understandable components. There are several existing explanations of SAEs, and I wanted to create a brief writeup from a different angle with an intuitive explanation of how they work.

Challenges with interpretability

The most natural component of a neural network is individual neurons. Unfortunately, individual neurons do not conveniently correspond to single concepts. An example neuron in a language model corresponded to academic citations, English dialogue, HTTP requests, and Korean text. This is a concept called superposition, where concepts in a neural network are represented by combinations of neurons.

This may occur because many variables existing in the world are naturally sparse. For example, the birthplace of an individual celebrity may come up in less than one in a billion training tokens, yet modern LLMs will learn this fact and an extraordinary amount of other facts about the world. Superposition may emerge because there are more individual facts and concepts in the training data than neurons in the model.

Sparse autoencoders have recently gained popularity as a technique to break neural networks down into understandable components. SAEs were inspired by the sparse coding hypothesis in neuroscience. Interestingly, SAEs are one of the most promising tools to interpret artificial neural networks. SAEs are similar to a standard autoencoder.

A regular autoencoder is a neural network designed to compress and then reconstruct its input data. For example, it may receive a 100 dimensional vector (a list of 100 numbers) as input, feed this input through an encoder layer to compress the input to a 50 dimensional vector, and then feed the compressed encoded representation through the decoder to produce a 100 dimensional output vector. The reconstruction is typically imperfect because the compression makes the task challenging.

Diagram of a standard autoencoder with a 1x4 input vector, 1x2 intermediate state

...
Formal verification, heuristic explanations and surprise accounting
by Jacob_Hilton — 2024-06-25T15:40:03.535Z

ARC's current research focus can be thought of as trying to combine mechanistic interpretability and formal verification. If we had a deep understanding of what was going on inside a neural network, we would hope to be able to use that understanding to verify that the network was not going to behave dangerously in unforeseen situations. ARC is attempting to perform this kind of verification, but using a mathematical kind of "explanation" instead of one written in natural language.

To help elucidate this connection, ARC has been supporting work on

Compact Proofs of Model Performance via Mechanistic Interpretability

by Jason Gross, Rajashree Agrawal, Lawrence Chan and others, which we were excited to see released along with this post. While we ultimately think that provable guarantees for large neural networks are unworkable as a long-term goal, we think that this work serves as a useful springboard towards alternatives.

In this post, we will:

Summarize ARC's takeaways from this work and the problems we see with provable guarantees
Explain ARC's notion of a heuristic explanation and how it is intended to overcome these problems
Describe with the help of a worked example how the quality of a heuristic explanation can be quantified, using a process we have been calling surprise accounting

We are also sharing a draft by Gabriel Wu (currently visiting ARC) describing a heuristic explanation for the same model that appears in the above paper:

max_of_k Heuristic Estimator

Thanks to Stephanie He for help with the diagrams in this post. Thanks to Eric Neyman, Erik Jenner, Gabriel Wu, Holly Mandel, Jason Gross, Mark Xu, and Mike Winer for comments.

Formal verification for neural networks

In

Compact Proofs of Model Performance via Mechanistic Interpretability

, the authors train a small transformer on an algorithmic task to high accuracy, and then construct several different formal proofs of lower bounds on the network's accuracy. Without foraying into the details, the most interesting takeaway from ARC's perspective is the following picture:

Figure adapted from this post

In the top right of the plot is the brute-force proof, which simply checks every possible input to the network. This gives the tightest possible bound, but is very long. Meanwhile, in the bottom left is the trivial proof, which simply states that the network is at least 0% accurate. This is very short, but gives th...
Metastrategy get-started guide
by Tahp — 2024-06-25T15:04:11.542Z

I was at LessOnline. The thing that got me most excited was attending @Raemon’s very abridged Metastrategy workshop. To my lack of surprise, he has written up the general idea already, but that report focuses on people’s response to trying fractal strategy and I would not have found it very actionable had I seen it before attending the workshop. The core idea immediately spoke to me when I saw it in-person, and I think I can slam out a quick-and-dirty get-started guide that explicitly lays out what you can put on paper to try this out today if you want. I want to try to use the method, but I won’t really know whether it works for me for months, and I want to make sure I know what I’m attempting first. Caveat: it is likely that I misunderstand some aspect of the method after receiving all of two hours of instruction.

What is a plan?

A plan is a thing you want to do. However, in order to do that thing you have several options for how to do the thing. Each of those options is itself a plan which might have several options for how to do it. It’s probably a good idea to stop at three levels of meta, but you could do more. When trying to accomplish some goal, you could make a tree graph of all the ways it is possible to accomplish that goal, where there is one path which goes all the way down the tree which describes what you want to accomplish, the first subgoal you intend to target that gets you there, the default subsubplan you will start with to achieve that subgoal, and so on. If you’re trying to achieve anything difficult, it is very likely that you will have a brilliant idea for how to accomplish your goal that fails, so you should make multiple approaches to accomplishing your goal, and if each of those approaches has multiple options for what order to go in or who to work with or what toolset to use, then you can easily have a very twisted tree. I’m a graduate student, so a (slightly facetious) plan might look something like this:

Note that I have only listed below each plan potential immediate next steps toward the plan. Leaves which are both below another leaf are each a next step I could take. Usually, the levels of meta correspond to lengths of time. For example, the top level could be a thing you want to accomplish in a year, the next level plan you might spend the next month on, and the level below that is what you will do for the next few days in orde...
Labor Participation is an Alignment Risk
by alex — 2024-06-25T14:15:25.141Z

TLDR: This paper explores and outlines why AI-related reductions in Labor participation (e.g., real unemployment) is a significant and high-priority alignment risk that the alignment community should more seriously consider as a potential underlying systemic risk factor. Constructive criticism, counter-factuals, and/or support are welcomed.

Thanks to Jolien Sweere (Open Philanthropy, Blueprint Bio) for her thoughtful review and suggestions. As well as Tzu Kit Chan, et al. for their insightful informal feedback and probability estimates including

Intro

Defining the concept of “economics” can be challenging even for tenured academics in the field. Economics can be the study of collective human behavior, the utilization of human capital, or policy (i.e., endogenous factors). It can also be a study of productivity and efficiency, and therefore a sort of metastudy of technology (i.e., exogenous factors). It can explore how we produce and consume goods and services, and it is deeply integrated with many other social sciences. Within Economics lie numerous sub-fields beyond the classical micro and macro, including: Natural Resource Economics, Game Theory, Environmental Economics, Behavioral Economics, Business Economics, Financial Economics, Health Economics, Industrial Economics, International and Trade Economics, Econometrics, Public Economics, and, I now humbly propose: AI Economics.

Writing about current topics in economics is often difficult and controversial. Writing about the anticipated economics of the future under unprecedented, likely transformative technologies, as you may have guessed, is even harder and more controversial. The reasons for this are many. The many facets of Economics are deeply intertwined with policy, taxation, and resource allocation decisions that have historically disproportionately benefited some members of society and disadvantaged others. Economic decisions can have significant long-term consequences for wealth distribution, financial inequality, class structure, global power dynamics, trade, business subsidization, technology R&D, and even scientific discovery.

Economists generally tend to compare AI to previous technological developments, pointing to productivity and GDP growth leading to the creation of new industries and new jobs. There are certainly parallels we can apply and learn from, but ultimately...
Monthly Roundup #19: June 2024
by Zvi — 2024-06-25T12:00:03.333Z

Looks like we made it. Yes, the non-AI world still exists.

Bad Governor

New York Governor Kathy Hochul has gone rogue and betrayed New York City, also humanity, declaring a halt to congestion pricing a month before it was to go into effect. Her explanation was that she spoke to workers at three Manhattan diners who were worried people would be unable to drive to them from New Jersey. Which, as Cathy Reilly points out, is rather insulting to New Jersey, and also completely absurd. Who in the world was going to go into Manhattan for a diner?

She says this won’t interfere with Subway work. Work on the 2nd Avenue Subway line has already been halted. And that’s not all.

You’re damn right. We are going to blame Hochul. Every. Damn. Time.

So Elizabeth Kim investigated. One never talked politics at all. One is directly across from Grand Central, is not a diner, and actively wants congestion pricing. The third did in fact object. That’s it. The good news is Hochul’s attempt to prevent this seems likely to be illegal, so maybe it won’t stop us.

The good news is this was so dumb that she might get primaried, but we will have to wait until 2026.

This terrible thinking is not an isolated incident.

Governor Kathy Hochul (D-NY): The next few days it’s going to be hotter than hell across New York — so we’re making admission and parking free at all our State Parks, pools, and beaches tomorrow and Thursday!

Take your families to beat the heat, and enjoy it on us

Matthew Yglesias: When faced with high demand for an excludable, rivalrous good with inelastic supply, I would make the price higher rather than lower and use the revenue for something useful.

High Skilled Immigration

Trump endorsed high skill immigration explicitly on the All-In podcast. He even said, only half prompted, that anyone who graduates from even a junior college should get a green card to stay in the country. It is amazing how clearly he speaks here. There is little question that Trump ‘gets it.’

Yet Trump’s track record is of course very different. Remember Trump’s H-1B Visa suspension in 2020?

So I wrote I was not optimistic about Trump following through, and indeed he has already ‘walked this back.’ Notice Fox News saying this was somehow a promise about ‘migrants.’

We should still obviously take this all up immediately in a bill and see who votes for it.

High skill immigr...
Regularly meta-optimization
by Crazy philosopher — 2024-06-25T06:12:58.939Z

The usefulness of the different actions differ by orders of magnitude. Sometimes, redirecting efforts can increase your efficiency by an order of magnitude or more.

Imagine that the person who first came up with the idea of sorting charitable foundations by efficiency, instead of implementing it, went to wash the dishes. Or Eliezer decided that creating a community is long and strange.

I'm not sure about the others, but I'm discarding many ideas, including potentially extremely effective ones, because I'm using the absurdity heuristic (this idea will change the world too much to be true). Or I reject ideas, because in order to implement them I will have to leave my comfort zone, and it causes me negative emotions.

Therefore, I came up with a technique that I call Regularly Meta-Optimization. It consists in regular trying to find potentially extremely effective ideas among those that you have been thinking about lately.

One of the potentially very effective ideas for me is to share a few of my potentially very effective blog ideas that can be implemented by anyone.

Creating a rationalistic YouTube channel. If we convince at least 20% of people of the danger of uncoordinated GAI, we will probably be able to get the government regulation we need in this area. The spread of rationality can also lead to very large positive side effects, for example, by radically increasing funding for anti-aging research and effective charity funds. This is feasible because rationalist ideas are most likely interesting to a significant number of people when presented correctly, because these are our ideas that are true, and therefore they are more convincing.
Memetics as an analogy and its implicit connotations
by Rachel Shu — 2024-06-25T05:13:12.746Z

(Standalone, but mostly written to be referenced by a future post on sandboxing frameworks and ideologies.)

Humans are bad at sandboxing Fake Frameworks, and thinking with them tends to create side effects. That is to say, when we make analogies, we transfer more characteristics than the comparison justifies, and bundle in assumptions that are undeserved.

Memetics (the analogy between ideas and genes) is such a framework, so I want to point out some of the assumptions that this particular analogy from ideas to genetics might unconsciously crystallize. Daniel Filan writes that meme theory "considers the setting where memes mutate, reproduce, and are under selection pressure." I think this is a good characterization of memetics. Let's go through some of the implications of this definition and the strength of various assumptions made:

Ideas are transmissible from mind to mind. This is required by the analogy; it's a natural implication and how we normally think of ideas anyways. The analogy isn't generally taken so far as to claim that ideas are heritable, though. If only! So much wasted time spent educating the youth to the level of their parents.
Ideas are copied when transmitted. Another basic assumption of the analogy, and a reasonably sturdy one too; it would be weird if, as soon as I told you an idea, I no longer had a copy of it.
Ideas are like infectious particles. This is already beyond a strict gene-to-idea analogy, but is a consequence of points 1 and 2, and is captured in the idea of a meme going 'viral'. This is also a common notion that predates the genetic analogy, as found in the older concept of an "earworm".
Ideas are possessions of minds. There is no idea which exists and is not possessed by an mind, just as genes that no organisms possess do not exist. We can still talk about hypothetical genes and memes. As a corollary ideas can become extinct. This assumption seems natural, but its corollary is weaker: ideas can be easily revived, whereas genetic extinction has until recently been more or less permanent.
Instances of ideas are possessions of discrete minds. In other words, minds are to ideas as cells are to genes. You have one token of an idea, and by telling it to me you transmit it to me and reproduce it in my mind. This is questionable; there is a very common phenomenon of a concept that contains lots of ideas, like a business plan or a

...
Mistakes people make when thinking about units
by Isaac King — 2024-06-25T03:39:20.138Z

This is a linkpost for Parker Dimensional Analysis. Probably a little elementary for LessWrong, but I think it may still contain a few novel insights, particularly in the last section about Verison's error.

A couple years ago, there was an interesting clip on MSNBC.

A few weeks later, Matt Parker came out with a video analyzing why people tend to make mistakes like this. Now I'm normally a huge fan of Matt Parker. But in this case, I think he kinda dropped the ball.

He does have a very good insight. He realizes that people are treating the "million" as a unit, removing it from the numbers before performing the calculation, then putting it back on. This is indeed the proximate cause of the error. But Matt goes on to claim that the mistake is the treating of "million" as a unit; the implication being that, as a number suffix or a multiplier or however you want to think of it, it's not a unit, and therefore cannot be treated like one. This is false.

So what is a unit, really? When we think of the term, we probably think of things like "meters", "degrees Celcius", "watts", etc.; sciency stuff. But I think the main reason we think of those is due to unit conversion; when you have to convert from meters to feet, or derive a force from mass and acceleration, this makes us very aware of the units being used, and we associate the concept of "unit" with this sort of physics conversion.

In reality, a unit is just "what kind of thing you're counting". Matt uses two other examples in his video: "dollars" and "sheep". Both of these are perfectly valid units! If I say "50 meters", that's just applying the number "50" to the thing "meters", saying that you have 50 of that thing. "50 sheep" works exactly the same way.

So what about "millions"? Well, we can definitely count millions! 1 million, 2 million, etc. You could imagine making physical groupings of a million sheep at a time, perhaps using some very large rubber bands, and then counting up individual clusters. "Millions" is a unit![1]

So if millions is a perfectly valid unit, why do we get an incorrect result if we take it off and then put it back on again after the calculation? Well, because you can't do that with other units either! 100 watts divided by 20 watts does not equal 5 watts. It equals the number 5, with no unit.

This is a somewhat subtle distinction, and easy to miss in a casual conversation. But it makes se...
Higher-effort summer solstice: What if we used AI (i.e., Angel Island)?
by Rachel Shu — 2024-06-25T01:35:54.064Z

As the title probably already indicates, this post contains community content rather than rationality content. Alternate, sillier version of this post here.

Motivation

I've been a co-organizer of the Bay Area Rationalist Summer Solstice for the past few years, and I've been thinking about how to make it a more meaningful and engaging experience, like what we have with Winter Solstice. The last few Summer Solstices, which I'd describe as mostly being big picnics, have been fun, but fairly low-effort, low-significance, and I think that's a missed opportunity.

Here's a few things that I'd like more of in Summer Solstice, non-exhaustive:

A sense of a temporary alternate world created around a shared purpose.
Time to connect with people and have deeper conversations.
Longer, more immersive collective experiences and thoughtfully designed rituals.
Thematic resonance with rationalist goals and community projects.
Ability to host the whole community, including children.

I have an idea for next year's Summer Solstice, which I think would get at fulfilling some of these goals. There's an island, Angel Island, in the middle of San Francisco Bay which is reasonably easy to get to, can accommodate lots of people, and has a bunch of qualities which would get at the goals above.

I've visited. It's naturally transporting, feels like a world into itself. I've done substantial research and think it's feasible to run Summer Solstice there. I'm posting this idea for discussion instead of running ahead with the planning for the following reasons:

As already suggested it requires a lot higher commitment from attendees. Travel is about 75 minutes (from and to downtown Berkeley) each way, including a ferry ride, and the ability to come and go is dictated by the ferry schedule. (Edit: The ferry runs every hour until 5:20pm, then we’d specially charter 7:20 and 9:20 pm departures. A small charter boat of maybe 8-10 capacity could also operate on demand between ferry trips with half hour round trips, reducing the wait time).
It requires a lot higher commitment from organizers. The coordination, preparation, and logistics needs are similar in degree to those of winter solstice, and the communication needs are even more involved.
I'm actually looking for someone else to take lead for next year. I've done it at least one year too many by tradition, and I also suffer winter depression, affecting some

...
I'm a bit skeptical of AlphaFold 3
by Oleg Trott — 2024-06-25T00:04:41.274Z

(also on https://olegtrott.substack.com)

So this happened: DeepMind (with 48 authors, including a new member of the British nobility) decided to compete with me. Or rather, with some of my work from 10+ years ago.

Apparently, AlphaFold 3 can now

predict

how a given drug-like molecule will bind to its target protein. And it does so better than AutoDock Vina (the most

cited

molecular docking program, which I

built

at Scripps Research):

On top of this, it doesn’t even need a 3D structure of the target. It predicts it too!

But I’m a bit skeptical. I’ll try to explain why.

Consider a hypothetical scientific dataset where all data is duplicated: Perhaps the scientists had trust issues and tried to check each others’ work. Suppose you split this data randomly into training and test subsets at a ratio of 3-to-1, as is often done:

Now, if all your “learning” algorithm does is memorize the training data, it will be very easy for it to do well on 75% of the test data, because 75% of the test data will have copies in the training data.

Scientists mistrusting each other are only one source of data redundancy, by the way. Different proteins can also be related to each other. Even when the sequence similarity between two proteins is low, because of evolutionary pressures, this similarity tends to be concentrated where it matters, which is the binding site.

Lastly, scientists typically don’t just take random proteins and random drug-like molecules, and try to determine their combined structures. Oftentimes, they take baby steps, choosing to study drug-like molecules similar to the ones already discovered for the same or related targets.

So there can be lots of redundancy and near-redundancy in the public 3D data of drug-like molecules and proteins bound together.

Long ago, when I was a PhD student at Columbia, I

trained

a neural network to predict protein flexibility. The dataset I had was tiny, but it had interrelated proteins already:

With a larger dataset, due to the Birthday Paradox, the interrelatedness would have probably been a much bigger concern.

Back then, I decided that using a random train-test split would have been wrong. So I made sure that related proteins were never in both “train” and “test” subsets at the same time. With my model, I was essentially saying “Give me a protein, and (even) if it’s unrelated to the ones in my training data, I can predict ...