trentmkelly/LessWrong-43k huggingface dataset:
Domain-specific SAEs
by jacob_drori — 2024-10-07T20:15:38.584Z

TLDR: Current SAE training doesn't specifically target features we care about, e.g. safety-relevant ones. In this post, we compare three ways use to SAEs to efficiently extract features relevant to a domain of interest.

Introduction

If Sparse Autoencoders (SAEs) are to be useful for alignment, they should reliably extract safety-relevant features. But currently, our methods for training SAEs are not targeted towards finding such features. Instead, we train SAEs on unstructured web text, then check if any of the learnt features happen to be safety-relevant (see e.g. here). By making our SAEs wider, we hope to find more and more - even all - of a model's features, thereby guaranteeing that the ones we care about show up.

This "find all the features" method is extremely expensive. According to Scaling Monosemanticity:

If a concept is present in the training data only once in a billion tokens, then we should expect to need a dictionary with on the order of a billion alive features in order to find a feature which uniquely represents [it]... If we wanted to get all the features… we would need to use much more compute than the total compute needed to train the underlying models.[1]

Individuals and small labs will not be able to afford this level of compute, and even scaling labs may be unwilling to pay such a high alignment tax. This motivates the following question:

Given some small, domain-specific dataset D, what is the best way to efficiently extract features relevant to that domain?

In this post, we assume access to a General SAE (GSAE), which is a medium-sized SAE trained in the usual way on web text.[2] We will compare three methods:

GSAE-finetune: finetune the GSAE on D.
Direct SAE: throw out the GSAE and train a small SAE on D.
Specialized SAE (SSAE): train a small SAE to reconstruct GSAE residuals on D. That is, if X is a model activation on D we want SSAE(X)≈X−GSAE(X).

The intuition for method 3 is that the GSAE takes care of the boring, general features that occur both in web text and in D (e.g. "this token is a verb"), freeing up the capacity of the SSAE to find the interesting, domain-specific features that the GSAE missed.

We find that the GSAE-finetune and Direct SAE perform best in terms of recovering model performance on D (as measured by CE loss). On the other hand, the SSAE finds featur...
Metaculus Is Open Source
by ChristianWilliams — 2024-10-07T19:55:31.035Z
Link: https://www.metaculus.com/notebooks/28734/

We completely rewrote the platform to make it easier for open-source contributors to build on our code. Maybe that's you? If it is, you can find the Metaculus code repo and can learn more here.

We've included instructions for setting up a local development environment and on creating an issue. We've also selected some easier issues, perfect for getting started.

Make a significant contribution, and we'll mail you a Metaculus hoodie to show our gratitude.

Also, in rewriting the platform, we introduced a variety of new features and quality-of-life improvements, including improvements to user profiles, notifications, the ability to zoom in on a time period in a forecast plot, and much more. You can find all the details here.
Research update: Towards a Law of Iterated Expectations for Heuristic Estimators
by Eric Neyman — 2024-10-07T19:29:29.033Z

Last week, ARC released a paper called Towards a Law of Iterated Expectations for Heuristic Estimators, which follows up on previous work on formalizing the presumption of independence. Most of the work described here was done in 2023.

A brief table of contents for this post:

What is a heuristic estimator? (One example and three analogies.)
How might heuristic estimators help with understanding neural networks? (Three potential applications.)
Formalizing the principle of unpredictable errors for heuristic estimation (the technical meat of the paper).

In "Formalizing the Presumption of Independence", we defined a heuristic estimator to be a hypothetical algorithm that estimates the values of mathematical expression based on arguments. That is, a heuristic estimator is an algorithm G that takes as input

A formally specified real-valued expression Y; and
A set of formal "arguments" π1,…,πm --

-- and outputs an estimate of the value of Y that incorporates the information provided by π1,…,πm. We denote this estimate by G(Y∣π1,…,πm).[1]

In that paper, we introduced the following question: is there a computationally efficient heuristic estimator that formalizes intuitively valid reasoning about the values of mathematical quantities based on arguments? We studied the question by introducing intuitively desirable coherence properties (one such property is linearity: a heuristic estimator's estimate of X+Y should equal its estimate of X plus its estimate of Y) and working to satisfy those properties. Ultimately, we left the question open.

The main technical contribution of our new work is to outline a new type of coherence property: a heuristic estimator should not be able to predict its own errors. We call this intuitive statement the principle of unpredictable errors. The principle is loosely inspired by the law of iterated expectations from probability theory, as well as the martingale property: a Bayesian reasoner's estimate of their future estimate of a quantity should be equal to its current estimate. One of the main purposes of this work is to explore ways to formalize this principle.

Our paper is structured as follows:

We begin by explaining the core motivating intuition behind heuristic estimators through three analogies: proof verification, conditional expectations, and subjective probabil

...
AI Model Registries: A Foundational Tool for AI Governance
by Elliot Mckernon — 2024-10-07T19:27:43.466Z
Link: https://www.convergenceanalysis.org/research/ai-model-registries-a-foundational-tool-for-ai-governance

This post consists of the Executive Summary and Conclusion of our proposal for model registries as a foundational tool for AI governance, authored by Elliot McKernon, Gwyn Glasser, Deric Cheng, and Gillian Hadfield. The full report is available through the link above.

Executive Summary of Our Proposal

In this report, we propose that national governments should implement AI model registries as a foundational tool for AI governance. By model registry, we mean a centralized database of frontier AI models that includes standard commercial and specific safety-relevant information about these models and their deployers. Developers would be required to report any qualifying models and their information to the registry before public deployment.

In Parts I, II, and III, we explore and make recommendations on the purpose of such a registry, what information it should store, and how to practically implement and administer it, respectively.

In this executive summary of our proposal we provide a concise, high-level summary of each of our conclusions, without argumentation, analysis, or evidence. To understand why we make each of these specific recommendations, we encourage readers to read the full section on each topic.

The Case for a Model Registry

AI model registries can serve as a foundational lever to increase regulatory visibility, support legal action, and manage societal risks. In other industries, registries successfully serve this same purpose for products and services associated with notable economic impacts or risks to society, as we detail in

Registries are a basic, common governance tool

. However, while some nations are taking early steps to develop model registries, as we detail in

What AI model registries currently exist?

, the current standards for frontier AI registration are not yet substantial enough to bring AI oversight into parity with other industries.

We identify four high-level objectives that motivate the adoption of frontier model registries:

A registry will facilitate the monitoring of frontier AI technology, providing governments with increased regulatory visibility into the capabilities and risks of leading AI models.
A registry will provide a key mechanism for regulatory enforcement of AI models, enabling governments to accurately pinpoint models subject to regulation.
A r

...
Evaluating the truth of statements in a world of ambiguous language.
by Hastings — 2024-10-07T18:08:09.920Z

If I say "the store is 500 meters away," is this strictly true? Strictly false? Either strictly true or strictly false, with probabilies of true or false summing to one? Fuzzily true, because the store is 500.1 meters away? My thesis is that it's strictly true or strictly false, with associated probabilities. Bear with me.
The 500 meter example is pretty hard to reason about numerically. I am hopeful that I can communicate my thoughts by starting with an example that is in some sense simpler.

Is our conversational partner making a true statement or a false statement? What we're used to is starting with a prior over worlds, and then updating our belief based on observations- the man is underwater, the man is currently moving downwards. So our belief that the man is sinking should be pretty high.
However, what we are unsure of is the language of the sentence. If it's German¹, then it means¹ "the man is pensive." Then, evaluating the truth value of the sentence is going to involve a different prior- Does the man look deep in thought? He seems to have a calm expression, and is sitting cross legged, which is weak evidence. More consideration is needed.
To come out to an actual probability of the original sentence being true, we need a distribution of possible intended meanings for the sentence, and then likelyhoods of the truthfulness of each meaning. If you buy my framing of the situation, then in this case it's simply P(Sentence is English) x P(Man is sinking) + P(Sentence of is German) x P(Man is thoughtful).
In the monolingual case, it's easy to get caught up in lingual proscriptivism. If someone uses a word wrong, and their sentence is only true when it's interpreted using the incorrect definition, then a common claim is that their sentence is untrue. A classic example of this is "He's literally the best cook ever." I hope that the multilingual case illustrates that ambiguity in the meaning of phrases doesn't have to come from uncareful sentence construction or "wrong" word use.
I claim that there's just always a distribution over meanings, and it can be sharp or fuzzy or bimodal or any sort of shape. Of course, as a listener you can update this distribution based on evidence just like anything else. As a speaker, you can work to keep this distribution sharp as an aid to those you speak with, though total success is impossible.
To dire...
Advice for journalists
by Nathan Young — 2024-10-07T16:46:40.929Z
Link: https://nathanpmyoung.substack.com/p/advice-for-journalists

I interact with journalists quite a lot and I have specific preferences. Not just for articles, but for behaviour. And journalists do behave pretty strangely at times.

This account comes from talking to journalists on ~10 occasions. Including being quoted in ~5 articles.

Privacy

I do not trust journalists to abide by norms of privacy. If I talk to a friend and without asking, share what they said, with their name attached, I expect they'd be upset. But journalists regularly act as if their profession sets up the opposite norm - that everything is publishable, unless explicitly agreed otherwise. This is bizarre to me. It's like they have taken a public oath to be untrustworthy.

Perhaps they would argue that it’s a few bad journalists who behave like this, but how am I supposed to know which the good and bad journalists are? The general advice in many tech, EA, rationalist spaces seems to be “be very careful when talking to journalists”. So it doesn't seem like it’s a rare bad apple.

And even the ‘good’ journalists think that there are somewhat arcane rules about what can and can’t be shared from public conversations, to the extent that I have to weave the following magic spell even when talking to the ‘good’ ones.

“Hi can we agree this is all off the record and then I can agree specific quotes to be on the record? I almost always give quotes, but I want the ability to veto anything that goes next to my name”

And sometimes journalists I respect say no. Their standard negotiating position is that they get to take what they want from a private conversation and publish it how they want. Imagine what the bad ones do.

This is not the typical norm around privacy and I don’t agree to the inversion of it. I ask family and close friends whether I can tweet things they say even if I am pretty sure. But with journalists, even the ‘good’ ones, I have to specifically ask that they don’t take my words and tell them to 100,000s of people, far more people than most people have ever communicated with at once[1].

And worse, if challenged on this, I'm not confident they'll acknowledge it. It's not just the norm breaking, it's the gaslighting. Oh us? No, we're bastions of democracy. We’re the good guys. I have two fairly vague recollections of journalists mangling my quotes badly and several more where articles had misleading titles. If this was a friend I would be...
Time Efficient Resistance Training
by romeostevensit — 2024-10-07T15:15:44.950Z

Previously: https://www.lesswrong.com/posts/RydETq379eoWqBFvj/updates-and-reflections-on-optimal-exercise-after-nearly-a

https://www.lesswrong.com/posts/bZ2w99pEAeAbKnKqo/optimal-exercise

I figured it would be useful to summarize how I train when I go for time efficiency these days, based on my understanding of the relevant meta-analyses on exercise selection and parameters. The primary time savers are supersets and lowered rest times. A routine like this is not totally optimized for max hypertrophy or anything, but is very very time efficient, and would still likely cause large gains for relatively untrained people. A lot of why I exercise these days, besides maintenance, is energy levels and injury prevention.

With supersets, you spend the time resting one group of muscles by doing an exercise for a different group of muscles. This is well supported as having the same or even slightly greater improvements in exercise studies, with the main downside being that the overall greater work packed in to a smaller amount of time being somewhat more motivationally taxing. There is the question of which supersets to do given we want full body coverage in a small amount of time and which versions of exercises resistance training coaches think work well based on their experience, together with some evidence from studies. There is the added constraint that since most people will do these in a commercial gym, you usually can't mix together supersets that use different machines, since many people will be trying to use them.

On rest times, studies seem to support that resting for longer than 2 minutes isn't needed for max performance, though they do note that if you let people rest ad libidinum they tend to rest for longer, more like 5 minutes, so you have to push yourself. Rest times as low as 1 minute don't show much worse performance but are also even more taxing. I've found with supersets you can get away with 1 minute rests between exercises, given that that means time between working the same muscle group will be 2ish minutes.

I normally do 2 warmup sets with light weight, followed by 3 sets of 8-20 (I just go 1 or two reps shy of failure and don't worry too much about perfect weight selection to hit the same rep ranges all the time). Studies have found even 2 sets to failure once a week to be sufficient for maintenance if you are super strapped for time. I normally do t...
A Narrow Path: a plan to deal with AI extinction risk
by Andrea_Miotti — 2024-10-07T13:02:15.229Z
Link: https://www.narrowpath.co/

We have published A Narrow Path: our best attempt to draw out a comprehensive plan to deal with AI extinction risk. We propose concrete conditions that must be satisfied for addressing AI extinction risk, and offer policies that enforce these conditions.

A Narrow Path answers the following: assuming extinction risk from AI, what would be a response that actually solves the problem for at least 20 years, and that leads to a stable global situation, one where the response is coordinated rather than unilaterally imposed with all the dangers that come from that.

Despite the magnitude of the problem, we have found no other plan that comprehensively tries to address the issue, so we made one.

This is a complex problem where no one has a full solution, but we need to iterate on better answers if we are to succeed at implementing solutions that directly address the problem.

Executive summary below, full plan at www.narrowpath.co , and thread on X here.

We do not know how to control AI vastly more powerful than us. Should attempts to build superintelligence succeed, this would risk our extinction as a species. But humanity can choose a different future: there is a narrow path through.

A new and ambitious future lies beyond a narrow path. A future driven by human advancement and technological progress. One where humanity fulfills the dreams and aspirations of our ancestors to end disease and extreme poverty, achieves virtually limitless energy, lives longer and healthier lives, and travels the cosmos. That future requires us to be in control of that which we create, including AI.

We are currently on an unmanaged and uncontrolled path towards the creation of AI that threatens the extinction of humanity. This document is our effort to comprehensively outline what is needed to step off that dangerous path and tread an alternate path for humanity.

To achieve these goals, we have developed proposals intended for action by policymakers, split into three Phases:

Phase 0: Safety - New institutions, legislation, and policies that countries should implement immediately that prevent development of AI that we do not have control of. With correct execution, the strength of these measures should prevent anyone from developing artificial superintelligence for the next 20 years.

Phase 1: Stability - International institutions that ensure measures to control

...
Toy Models of Feature Absorption in SAEs
by chanind — 2024-10-07T09:56:53.609Z

TLDR;

In previous work, we found a problematic form of feature splitting called "feature absorption" when analyzing Gemma Scope SAEs. We hypothesized that this was due to SAEs struggling to separate co-occurrence between features, but we did not prove this. In this post, we set up toy models where we can explicitly control feature representations and co-occurrence rates and show the following:

Feature absorption happens when features co-occur.
If co-occurring feature magnitudes vary relative to each other, we observe "partial absorption", where a latent tracking a main feature sometimes fires weakly instead of not firing at all, but sometimes does fully not fire.
Feature absorption happens even with imperfect co-occurrence, depending on the strength of the sparsity penalty.
Tying the SAE encoder and decoder weights together solves feature absorption in toy models.

All code for this post can be seen in this Colab notebook.

The rest of this post will assume familiarity with Sparse Autoencoders (SAEs). But first, some background on feature absorption:

What is feature absorption?

Feature absorption is a problematic form of feature splitting where a SAE latent appears to track an interpretable concept, but actually has holes in its recall. Instead, other SAE latents fire on specific tokens and "absorb" the feature direction into approximately token-aligned latents.

For instance, in Gemma Scope SAEs we find a latent which seems to track the feature that a token "starts with S". However, the latent will not fire on a few specific tokens that do start with S, like the token "short".

In feature absorption, we find gerrymandered SAE latents which appear to track an interpretable concept but have holes in their recall. Here, we see the dashboard for Gemma Scope layer 3 16k, latent 6510 which appears to track "starts with S", but mysteriously doesn't fire on "_short".

How is this different than traditional feature splitting?

In traditional feature splitting, we expect a more general latent to split into more specific latent, but still tracking the same concept and still interpretable. For instance, a "starts with L" latent might split into "starts with uppercase L" and "stats with lowercase L". Theses more specific latents are still about starting with the letter L and nothing else, just more specific variants on it.

Traditional feature splitting doesn't pose a problem for inte...
An argument that consequentialism is incomplete
by cousin_it — 2024-10-07T09:45:12.754Z

I think consequentialism describes only a subset of my wishes. For example, maximizing money is well modeled by it. But when I'm playing with something, it's mostly about the process, not the end result. Or when I want to respect the wishes of other people, I don't really know what end result I'm aiming for, but I can say what I'm willing or unwilling to do.

If I try to shoehorn everything into consequentialism, then I end up looking for "consequentialist permission" to do stuff. Like climbing a mountain: consequentialism says "I can put you on top of the mountain! Oh, that's not what you want? Then I can give you the feeling of having climbed it! You don't want that either? Then this is tricky..." This seems a lot of work, just to do something I already want to do. There are many reasons to do things - not everything has to be justified by consequences.

There are of course objections. Objection one is that non-consequentialist wishes can make you go in circles, like that Greg Egan character who spent thousands of hours carving table legs, making himself forget the last time so he could enjoy the next. But when pushed to such extremes, a consequentialist goal like maximizing happiness can also lead to weird results (vats of happiness goo...) And if we don't push quite so hard, then I can imagine utopia containing both consequentialist and non-consequentialist stuff, doing things for their own sake and such. So there's no difference here.

Objection two is that our wishes come from evolution, which wants us to actually achieve things, not go in circles. But our wishes aren't all perfectly aligned with with evolution's wish (procreate more). They are a bunch of heuristics that evolution came up with, and a bunch of culturally determined stuff on top of that. So there's no difference here either - both our consequentialist and non-consequentialist wishes come from an equally messy process, so they're equally legitimate.