trentmkelly/LessWrong-43k huggingface dataset:
A List of 45+ Mech Interp Project Ideas from Apollo Research’s Interpretability Team
by Lee Sharkey — 2024-07-18T14:15:50.248Z

Why we made this list:

The interpretability team at Apollo Research wrapped up a few projects recently[1]. In order to decide what we’d work on next, we generated a lot of different potential projects. Unfortunately, we are computationally bounded agents, so we can't work on every project idea that we were excited about!
Previous lists of project ideas (such as Neel’s collation of

200 Concrete Open Problems in Mechanistic Interpretability

) have been very useful for people breaking into the field. But for all its merits, that list is now over a year and a half old. Therefore, many project ideas in that list aren’t an up-to-date reflection of what some researchers consider the frontiers of mech interp.

We therefore thought it would be helpful to share our list of project ideas!

Comments and caveats:

Some of these projects are more precisely scoped than others. Some are vague, others are more developed.
Not every member of the team endorses every project as high priority. Usually more than one team member supports each one, and in many cases most of the team is supportive of someone working on it.
We associate the person(s) who generated the project idea to each idea.
We've grouped the project ideas into categories for convenience, but some projects span multiple categories. We don't put a huge amount of weight on this particular categorisation.

We hope some people find this list helpful!
We would love to see people working on these! If any sound interesting to you and you'd like to chat about it, don't hesitate to reach out.

Foundational work on sparse dictionary learning for interpretability

Transcoder-related project ideas

See

[2406.11944] Transcoders Find Interpretable LLM Feature Circuits

)
[Nix] Training and releasing high quality transcoders.

Probably using top k
GPT2 is a classic candidate for this. I’d be excited for people to try hard on even smaller models, e.g. GELU 4L

[Nix] Good tooling for using transcoders

Nice programming API to attribute an input to a collection of paths (see

Dunefsky et al

)
Web user interface? Maybe in collaboration with neuronpedia. Would need a gpu server constantly running, but I’m optimistic you could do it with a ~a4000.

[Nix] Further circuit analysis using transcoders.

Take random input sequences, run transcoder attribution on them, examine the output and summarize the findings.
High level summary s

...
SAEs (usually) Transfer Between Base and Chat Models
by Connor Kissane — 2024-07-18T10:29:46.138Z

This is an interim report sharing preliminary results that we are currently building on. We hope this update will be useful to related research occurring in parallel.

Executive Summary

We train SAEs on base / chat model pairs and find that SAEs trained on the base model transfer surprisingly well to reconstructing chat activations (and vice versa) on Mistral-7B and Qwen 1.5 0.5B.

We also find that they don’t transfer on

Gemma v1 2B

, and are generally bad at reconstructing <1% of unusually high norm activations (e.g. BOS tokens) from the opposite model.

We fine-tune our base Mistral-7B SAE (on 5 million chat activations) to cheaply obtain an SAE with competitive sparsity and reconstruction fidelity to a chat SAE trained from scratch (on 800M tokens).
We open source base, chat, and fine-tuned SAEs (plus wandb runs) for Mistral-7B and Qwen 1.5 0.5B.[1]

Mistral 7B base SAEs

,

Mistral 7B chat SAEs

,

Mistral 7B base SAEs fine-tuned on chat

Qwen 1.5 0.5B base SAEs

,

Qwen 1.5 0.5B chat SAEs

,

Qwen 1.5 0.5B base SAEs fine-tuned on chat

We release accompanying evaluation code at

https://github.com/ckkissane/sae-transfer

Introduction

Fine-tuning is a common technique applied to improve frontier language models, however we don’t actually understand what fine-tuning changes within the model’s internals. Sparse Autoencoders are a popular technique to decompose the internal activations of LLMs into sparse, interpretable features, and may provide a path to zoom into the differences between base vs fine-tuned representations.

In this update, we share preliminary results studying the representation drift caused by fine-tuning with SAEs. We investigate whether SAEs trained to accurately reconstruct a base model’s activations also accurately reconstruct activations from the model after fine-tuning (and vice versa). In addition to studying representation drift, we also think this is an important question to gauge the usefulness of sparse autoencoders as a general purpose technique. One flaw of SAEs is that they are expensive to train, so training a new suite of SAEs from scratch each time a model is fine-tuned may be prohibitive. If we are able to fine-tune existing SAEs for much cheaper, or even just re-use them, their utility seems more promising.

We find that SAEs trained on the middle-layer residual stream of base models transfer surprisingly well to the cor...
Should we exclude alignment research from LLM training datasets?
by Ben Millwood — 2024-07-18T10:27:40.991Z

This is a companion post to Keeping content out of LLM training datasets, which discusses the various techniques we could use and their tradeoffs. My intention is primarily to start a discussion, I am not myself very opinionated on this.

As AIs become more capable, we may at least want the option of discussing them out of their earshot.

Places to consider (at time of writing, none of the below robots.txt files rule out LLM scrapers, but I include the links so you can check if this changes):

Alignment Forum (robots.txt)

LessWrong (robots.txt)

EA Forum (robots.txt)

Alignment org websites, e.g.

ARC (robots.txt)

METR (robots.txt)

arXiv (robots.txt which links to their policy)

was explicitly mentioned by Meta as a training source for LLaMA-1,

obviously we're less in a position to decide here, but we could ask.

Options to consider:

Blocking everything in robots.txt, or by User-Agent

it would perhaps seem a shame for AIs to know none of the content here

(for EAF / LW / AF) Writing code for ForumMagnum to:

enable blocking (probably by User-Agent) to be configured per-post (I haven't investigated this for feasibility, but naively it doesn't seem so hard).

enable posts to be configured to be visible only to logged-in users (maybe this is a feature we might want for other reasons?)

I think content-based methods (e.g. presence of "canary" strings) are the strongest / most durable, if they were adopted by LLM scrapers. We could try to push for their adoption, and then their use.

Feel free to suggest additions to either category.

To the extent that doing something here means spending software dev time, this raises the question not only of should we do this but how important is this, relative to the other things we can spend software developers on.

Link preview image by Jonny Gios on Unsplash
Keeping content out of LLM training datasets
by Ben Millwood — 2024-07-18T10:27:27.827Z

This post collects methods to exclude internet resources from LLM training datasets.

I plan to at least try to keep this post up-to-date with respect to new things I learn on the topic. Please feel encouraged to suggest any additions or amendments.

This post is about how to do it. Discuss whether to apply these techniques in a separate post: Should we exclude alignment research from LLM training datasets?

Link preview image by Steve Douglas on Unsplash.

Documentation from model vendors

OpenAI (ChatGPT)

See docs for GPTBot and ChatGPT-User.

GPTBot is for training data, and ChatGPT-User is used by plugins which can access the internet during inference. They document the user-agents used, the robots.txt identities, and the IP ranges they access from. There's some commentary about how ChatGPT-User is used in training, which I didn't find very illuminating.

Anthropic (Claude)

Does Anthropic crawl data from the web, and how can site owners block the crawler?

Pretty similar to OpenAI's offering, except that they don't have fixed IP ranges, and Claude (as far as I understand?) doesn't directly access the internet, so that distinction isn't relevant.

Some nice touches are that their crawler will not scrape anything that already blocked the Common Crawl bot (see below) and specifically commits to not trying to bypass CAPTCHAs (see further below).

Google (Gemini)

Appears to use the Google-Extended crawler, which can be blocked with robots.txt. It doesn't use a different user-agent header relative to other Google products, so user-agent blocking is only possible if you're willing to block the Search scraper as well. I assume they also don't use fixed IP ranges, but I haven't really tried to check, since that always seemed like the most clumsy method anyway.

Meta (LLaMA)

LLaMA 2's model card doesn't disclose their training data, but LLaMA 1 (per its model card) was trained on a variety of sources including CCNet and C4, which are both derived from the Common Crawl dataset (see below), so it seems likely that excluding your data from LLaMA at a minimum requires excluding it from Common Crawl as well.

Common Crawl

Common Crawl is a large, publicly-available dataset which in principle any training process could use (and Meta did), so keeping out of AI training datasets necessitates keeping out of Common Crawl. Their FAQ documents their user-agent and robots.txt identifie...
The Assassination of Trump's Ear is Evidence for Time-Travel
by elv — 2024-07-18T07:01:45.560Z

Epistemic Status: extreme, unhinged speculation. Possibly a small Basilisk-like effect from reading this that directionally seems neutral-to-positively aligned to the cause of human survival. Hopefully entertaining. Big if true.

Over the weekend Trump was very nearly assassinated. By sheer luck he decided to tilt his head before the 20-year-old would-be assassin pulled the trigger, causing the bullet to graze his ear instead of piercing his brain. This was a probabilistically ridiculous event that had huge implications for America's future. What updates should a Bayesian make to his beliefs in response to this event?

The only mundane hypothesis that explains away the extreme luck is that the assassination was a false flag on behalf of the Trump team. Indeed the result was optically optimal to Trump to a ridiculous degree. He even outdid Senator Morra from Limitless. But then you have to contend with the idea of a comfortably ahead Trump deciding to arrange for a 20-year-old to snipe at his ear from 120 meters away as he was moving around the stage. It seems preposterous to risk death like this for the sake of a boost in an election you were already more likely to win.

Other conspiracy theories, by other culprits, would still have this be a failed assassination, so it would still remain that Trump was ridiculously lucky.

Some religious pro-Trump people have talked of divine intervention. There is a species of atheist that would be triggered into debate mode by this and completely deny it. The principled rationalist should bite the bullet and say: yes, it's a bit of credible evidence. This event is something that you expect to be more likely in a world where a higher power exists than one where it does not, but there is still overall a million other things on the other side of the evidentiary scale of the theological question. But yes, if events like this start happening with frequency there would be cause to seriously consider the supernatural. Of course, one would have to wonder about the nature of this pro-MAGA interfering being, who, saving Trump, let an innocent firefighter attending the rally be murdered in his stead.

An interesting secular explanation for historical human luck is the anthropic principle. The nature of quantum mechanics substantiates the idea of a multiverse, and so an explanation to the puzzling luck that we have that the universe's physical...
Friendship is transactional, unconditional friendship is insurance
by Ruby — 2024-07-17T22:52:41.967Z

It feels a little icky to say, but we befriend people because we get something out of it. We enjoy the company, the conversation, the emotional support, the activities, the connection, etc. It's not a coincidence people don't befriend brick walls.

(The same is true in romantic relationships, except we expect even more.)

Granted, friendship is not an explicit transaction that's negotiated, quantified, legally enforceable, etc. It's fuzzy, which helps it work better for reasons I won't really get into here[1].

However it's crucial to recognize that if your friend (or partner) didn't provide or promise you some kind of value[2], you wouldn't have become friends in the first place.

And yet, people valorize the notion of loyalty in relationships: continuing to be there through thick and thin, good and bad, health and illness. "Unconditional friendship" and "unconditional love". Conversely "fair weather friendship" is denigrated.

People hope to be loved even if they were worms.

What gives? How do we reconcile friendships and relationships arising due to receiving some value with the aspiration or even expectation of unconditionality?

My model here is something akin functionally to mutual insurance. While I became your friend because we spent years playing basketball together, I stay by your side even when you're recovering from a broken leg, or even if you were injured so badly as to never play again. Someone initially enticed by their partner's beauty, stays with them even after a horrific burn to the face. I do this because I expect the same in return.

You might argue that in these cases, you're still receiving other benefits even when one of them is lost, but I argue back that we see ongoing care even where there's almost nothing left, e.g. people caring for their senile, bedridden partners. And more so, that we judge people who don't stick it out.

Friendship is standardly a straightforward exchange of value provided. It is also an exchange of insurance "if you're not able to provide value to me, I'll still provide value to you" and vice versa. Like the other stuff in friendship, it's fuzzy. The insurance exchange doesn't happen in a discrete moment and its strength is quantitative and expected to grow over time. People expect more "loyalty" from friends and partners of years than weeks.

In the limit, people reach "unconditional love", meaning something like from this po...
D&D.Sci: Whom Shall You Call? [Evaluation and Ruleset]
by abstractapplic — 2024-07-17T22:34:25.111Z

This is a followup to the D&D.Sci post I made twelve days ago; if you haven’t already read it, you should do so now before spoiling yourself.

Here is the web interactive I built to let you evaluate your solution; below is an explanation of the rules used to generate the dataset (my full generation code is available here, in case you’re curious about details I omitted). You’ll probably want to test your answer before reading any further.

Ruleset

(Note: to make writing this easier, I’m using standard D&D dice notation, in which “3+4d8” means “roll four eight-sided dice, sum the results, then add three”.)

Ghost Generation

Traits

Ghosts have nine traits which can affect the price of an exorcism: Intellect, Corporeality, Sliminess, Hostility, Grotesqueness, Deviousness, Perversity, Paradoxicality, and Unholiness, each a positive integer with a max of 100. Of these, the PPEK only measures the first five.

For the typical ghost (kind of; see next section), these traits are generated per the below table:

Trait	Generator
Intellect	1d80 + min(1d20, 1d20)
Corporeality	1d60 + 1d20 + min(1d20, 1d20)
Sliminess	[Corporeality]/5 + 1d60 + min(1d20, 1d20)
Hostility	1d80 + min(1d10, 1d10) + 1d10
Grotesqueness	min(1d20, [Hostility]/5) + [Corporeality]/10 + 1d10 + 1d60
Deviousness	[Intellect]/10 + 1d10 + 1d80
Perversity	1d60 + min(1d40, 1d40)
Paradoxicality	1d100
Unholiness	1d60 + 1d40

Categories

A ghost is either a Wisp, Spirit, or Horror. Wisps, when generating each trait, roll three times and pick the lowest result; Spirits roll once; Horrors roll three times and pick the highest.

43.3% of ghosts are Wisps, 19.3% are Spirits, and 37.4% are Horrors.

Exorcism Prices

The cost of an exorcism varies randomly in a triangular distribution from 50% to 150% of the mean.

Phantom Pummelers

The Pummelers, as the name suggests, tend to solve problems with their fists. They are aided slightly by Corporeality (they have ways to drag enemies into the physical plane, but find life somewhat simpler when they get to skip that step), hindered greatly by Sliminess, and impeded by Perversity and Unholiness.

E(PP)=610−6∗corp+41∗slim+12∗perv+8∗unho

Wraith Wranglers

The Wranglers believe in talking things out, seeing their task as gently helping ghosts to pass on. Their ideal client has high Intellect, low Deviousness, low Paradoxicality, and - most importantly - low Hostility.

E(WW)=521−5∗inte+39∗host+15∗devi+17∗para

Spectre Slaye

...
Optimistic Assumptions, Longterm Planning, and "Cope"
by Raemon — 2024-07-17T22:14:24.090Z

Eliezer Yudkowsky periodically complains about people coming up with questionable plans with questionable assumptions to deal with AI, and then either:

Saying "well, if this assumption doesn't hold, we're doomed, so we might as well assume it's true."
Worse: coming up with cope-y reasons to assume that the assumption isn't even questionable at all. It's just a pretty reasonable worldview.

Sometimes the questionable plan is "an alignment scheme, which Eliezer thinks avoids the hard part of the problem." Sometimes it's a sketchy reckless plan that's probably going to blow up and make things worse.

Some people complain about Eliezer being a doomy Negative Nancy who's overly pessimistic.

I had an interesting experience a few months ago when I ran some beta-tests of my Planmaking and Surprise Anticipation workshop, that I think are illustrative.

i. Slipping into a more Convenient World

I have an exercise where I give people the instruction to play a puzzle game ("Baba is You"), but where you normally have the ability to move around and interact with the world to experiment and learn things, instead, you need to make a complete plan for solving the level, and you aim to get it right on your first try.

In the exercise, I have people write down the steps of their plan, and assign a probability to each step.

If there is a part of the puzzle-map that you aren't familiar with, you'll have to make guesses. I recommend making 2-3 guesses for how a new mechanic might work. (I don't recommend making a massive branching tree for every possible eventuality. For the sake of the exercise not taking forever, I suggest making 2-3 branching path plans)

Several months ago, I had three young-ish alignment researchers do this task (each session was a 1-1 with just me and them).

The participants varied in how experienced they were with Baba is You. Two of them were new to the game, and completed the the first couple levels without too much difficulty, and then got to a harder level. The third participant had played a bit of the game before, and started with a level near where they had left off.

Each of them looked at their level for awhile and said "Well, this looks basically impossible... unless this [questionable assumption I came up with that I don't really believe in] is true. I think that assumption is... 70% likely to be true."

Then they went an executed their plan.

It failed. The ques...
Baking vs Patissing vs Cooking, the HPS explanation
by adamShimi — 2024-07-17T20:29:09.645Z
Link: https://epistemologicalfascinations.substack.com/p/baking-vs-patissing-vs-cooking-the

Why is baking harder than cooking?

I’m not even talking about creating recipes, merely executing them.

Think about it: when cooking, you manipulate a far more diverse cast of ingredients, by applying many more techniques (cutting, cooking, mixing…); yet with a bit of experience, you reliably end up close to the expected results.

Whereas baking, both when it works and when it fails, feels like the judgment of an unscrutable god — you did what you thought was the same thing, leading to two completely different outcomes.

And patissing (making cakes as opposed to bread) falls in the middle: more reliable than baking, but more unpredictable than cooking.

Why is that?

My pet theory is an analogy between these three food making disciplines, and three fundamental sciences: Cooking as Classical Physics, Patissing as Chemistry, Baking as Biology.

Cooking and Classical Physics

When cooking say a couscous or a beef wellington, most of what you’re doing falls within the real of classical physics: you’re cutting stuff, you’re applying heat to them, you’re mixing them.

This means that the consequences of your actions are relatively intuitive; for example, you will cook faster if the cuts are smaller and/or the temperature is higher.

There are a couple of tricks (thermal inertia which means your pasta/meat doesn’t stop cooking when you take it out of the pan, mixing fish with acid will cook it,…), but by far and large cooking behaves cleanly, linearly, such that being a bit off will lead to a slightly off result.

Thus the link with classical physics: it’s the part where you can easily intuit the behavior of the core components, and only need your eye and hand or basic tools to get the core data.

And in terms of feedback, you see almost instantly when you fucked up, and you can often course correct quickly (except when you try to reverse entropy: cannot uncook meat or uncut vegetables)

Patissing and Chemistry

When we move to patisserie, the bulk of the work moves from the physical to the chemical level. Instead of the variety of results coming from the variety of ingredients (different vegetable, different cuts of meat, different spices…), we are now using basically the same ingredients all the time (flour, water, sugar, eggs, butter, milk…), and instead varying the transformations that are applied to them.

Think about

the stages of caramel

or the many uses of eggs.

This introduces non-line...
Launching the Respiratory Outlook 2024/25 Forecasting Series
by ChristianWilliams — 2024-07-17T19:51:05.380Z
Link: https://www.metaculus.com/tournament/respiratory-outlook-24-25/

Returning for its second year, Metaculus's Respiratory Outlook for 2024/25 aims to support and inform US public health officials in anticipating the burden of influenza, COVID-19, and RSV this upcoming fall and winter. You can contribute to these efforts by sharing your predictions on important questions about peak hospitalization rates, vaccination coverage, and the risk from threats such as measles and avian influenza.

We will deliver aggregate forecasts and rationales to public health officials to aid preparedness efforts. You can contribute by forecasting and sharing your reasoning in the Respiratory Outlook 2024/25.

Series questions below. Note that the Community Prediction will initially be hidden: