trentmkelly/LessWrong-43k huggingface dataset:
Sparse Coding, for Mechanistic Interpretability and Activation Engineering
by David Udell — 2023-09-23T19:16:31.772Z

Especial thanks to Logan Riggs and Monte MacDiarmid, for pointing me towards this whole research direction and for code discussion, respectively. Thanks to Alex Turner for project feedback and for orienting me towards scaling activation engineering up to larger models. Thanks to Adrià Garriga-Alonso, Daniel Kokotajlo, Hoagy Cunningham, Nina Rimsky, and Garrett Baker for discussion and/or draft comments. And thanks to anyone I discussed this with!

TL;DR: To separate out superimposed features represented by model neurons, train a sparse autoencoder on a layer's activations. Once you've learned a sparse autoencoding of those activations, this autoencoder's neurons can now be readily interpreted.

Introduction

All code hosted at this repository: activation_additions/sparse_coder

A bit ago, I became interested in scaling activation engineering to the largest language models I could. I was initially surprised at how effective the technique was for being such a naive approach, which made me much more enthusiastic about simple manipulations of model activation spaces.

Yudkowsky says that we cannot expect to survive without a mathematical understanding, a guiding mathematical framework, of the AI. One hunch you might have is that a linear feature combination theorem could be the root of such a guiding theory. If so, we might learn a lot about the internal learned mechanisms of models by playing with their activation spaces. I feel like tuned lens and activation additions are some evidence for this hypothesis.

One major problem I experienced as I scaled up activation engineering to the largest models I could get my hands on (the new open-source Llama-2 models) was that it's hard to guess ahead of time which additions will work and which won't. You generate a new addition and stick it into a forward pass. Then, you get a few bits back observing how well the addition worked. "It would have been great," I thought, "to get a window into which concepts the model represents internally, and at which layer it does so."[1]

Sparse coding excited me at this point, because it suggested a way to learn a function from uninterpretable activations to represented, interpretable concepts! Paired with activation engineering's function from interpretable concepts to model internal activations, it sounded like a promising alignment scheme. Now, many things sound promising ahead of time. But seein...
Places to meet interesting middle-aged men?
by anon_girl — 2023-09-23T19:06:48.829Z

The rationalist community is a nice place to meet interesting, intellectual men, which makes it potentially a good dating pool for straight/bisexual women. However, this community is young: most people seem to be in their 20s or 30s. For middle-aged women like me, that's a problem. What are some other communities where it's easy to meet straight/bisexual men who are: (i) intellectually interesting is vaguely similar ways to how the rationalist community is intellectually interesting (ii) middle-aged and (iii) available?
Taking features out of superposition with sparse autoencoders more quickly with informed initialization
by Pierre Peigné — 2023-09-23T16:21:42.799Z

This work was produced as part of the SERI MATS 3.0 Cohort under the supervision of Lee Sharkey.

Many thanks to Lee Sharkey for his advice and suggestions.

TL;DR: it is possible to speed up the extraction of superposed features using sparse autoencoders by using informed initialization of the sparse dictionary. Evaluated on toy data, the informed initialization scheme results are the following:

Immediate MMCS ~ 0.65 (MMCS < 30% at start with the original orthogonal initialization)
Up to ~10% speedup to reach 0.99 MMCS of the superposed feature with some initialization methods relying on collecting rare features.

The main ideas:

The data contains the (sparsely activating) true features, which can be used to initialize the dictionary
However, the rare features are “hard to reach” in the input data. To get a good recovery we want to make sure the rare features are represented in the initialization sample.

0 - Context

Previous work has investigated how to take superposed features out of superposition in toy data. However, the current approach based on sparse autoencoders is relatively compute intensive, making the possibility of recovering monosemantic representations of large models computationally challenging.

The aim of this project was to reduce the compute required to train those sparse autoencoders by experimenting with better initialization schemes.

The original training process

To take features out of superposition using sparse autoencoders, we train an autoencoder with an L1 penalty on its hidden layer activation. In order to fit all (or at least the maximum of) superposed features in a monosemantic manner, the decoder dimensions must be larger or equal to the total number of ground truth (superposed) features.

Current initialization of the decoder relies on orthogonal initialization (Hu et al. 2020).

The metric

I used the same metric as in the original work: the Mean Max Cosine Similarity.

This metric is supposed to capture how well the ground features are recovered by the sparse dictionary. For instance an MMCS of 0.65 means that, on average, each ground truth feature has, on average, a cosine similarity of 0.65 with the most similar learned feature.

I - Initializing the dictionary with input data

The idea is that even though the ground truth features are superposed, the data still contain useful informatio...
A quick remark on so-called “hallucinations” in LLMs and humans 
by Bill Benzon — 2023-09-23T12:17:26.600Z

Cross-posted from New Savanna.

That LLMs “hallucinate” is well-known, though I prefer the term “confabulate.” They just make stuff up. They don’t do it intentionally, or with intent to deceive. They do it because they have no connection to “ground truth.” They don’t even know what such a thing is, not really, though I’m sure, if asked, ChatGPT would say something reasonable about the idea.

What LLMs have, loosely speaking, is an ontology. They ‘know’ what kinds of things exist. As Immanuel Kant pointed out in connection with the ontological argument for the existence of god, existence is not a predicate. When an LLM responds to a prompt, its response is consistent with the ontology it has internalized. If phenomena in its response happen to be true of the world, that’s incidental, though convenient for users.

What’s implicit in this conversation is an assumption that humans do not confabulate, that we can and so talk about reality. I think that’s true, sorta’, but misleading. Yes, we often anchor our statements in reality, as best we can do in the situation. And, yes, we can also deliberately side-step such anchoring. We do this when telling fictional stories without intention to deceive; in this situation we assert that the story is about imagined event. And we can also make things up with deceptive intent.

Nonetheless, I believe that our “natural” linguistic mode is just making things up, confabulation it you will. But we are surrounded by others and have to interact with them. One way to communicate effectively is to anchor our language in external events, in things others can readily observe, in reality, as we like to say. Such anchoring is not necessary for speech, but it is useful when conversing with others.

In this connection I think it’s worth noting that, for a number of years, neuroscientists have been investigating the brain structures that are most active when we’re doing nothing in particular, when we’re letting our mind wander. In that state we may attended to some external event one moment, scratch our back the next, think about that summer’s day three years ago, and so forth. We day dream. What do you think they call that conglomeration of neural structures? They call it the default mode network (DFM). If you give a verbal report of what’s passing by during a day dream, a lot of that is going to be confabulation. It’s just the way we are.

Homework...
Hand-writing MathML
by jefftk — 2023-09-23T11:20:07.870Z

When I write posts I use raw HTML. Yes, the modern thing to do is
probably

Markdown, but HTML
was designed for hand-coding and still works well for that if you
don't want anything especially fancy. But what if you want math?

Previously when I've wanted to do math I've written it out as
fixed-width ASCII:

e^(-7t)

In my editor this looks like:

<pre>
e^(-7t)
</pre>

This is reasonably readable, works anywhere, and I like the aesthetic.
I probably should have stuck with it, but after helping publish a
report that included some traditionally-formatted equations and
learning that MathML
has been supported cross-browser since the beginning of the year (thanks Igalia!), I decided to
try it out. I wrote the equations in two recent
posts
in it, and am mixed on the experience.

It definitely does look nicer:

e−7t

On the other hand, here's how it looks in my editor:

<math display=block>
<msup>
<mi>e</mi>
<mrow>
<mo>-</mo>
<mn>7</mn>
<mi>t</mi>
</mrow>
</msup>
</math>

There's a small learning curve on when to use the different tags, but
mostly it's just very verbose. And I think, needlessly so? That
"-" is an operator, "7" is a number, and
"t" is an identifier could all be the default. Then I
could just write:

<math display=block>
<msup>
e
<mrow>
-7t
</mrow>
</msup>
</math>

And we could remove many uses of <mrow> too: a
series of characters without whitespace separating them could
be already treated as a group:

<math display=block>
<msup>
e
-7t
</msup>
</math>

Of course if you wanted to use a character for a non-traditional
purpose you could still mark it up as one, but a good set of defaults
would make MathML much more pleasant. I'd hate to have
to read and write blog posts as:

<word><lt>h</lt><lt>e</lt><lt>l</lt><lt>l</lt><lt>o</lt></word>
<word><lt>w</lt><lt>o</lt><lt>r</lt><lt>l</lt><lt>d</lt></word>
<pnct>.</pnct>

I know I'm about 25
years too late on this, and I'm happy that a pure-HTML solution is
now cross-browser, but it's still s...
Luck based medicine: inositol for anxiety and brain fog
by Elizabeth — 2023-09-22T20:10:07.117Z

Summary: Do you have weird digestive symptoms and anxiety or depression? Consider trying inositol (affiliate link), especially if the symptoms started after antibiotics.

Epistemic status: I did some research on this 10 years ago and didn’t write it down. In the last nine months I recommended it to a few people who (probably) really benefited from it. My track record on this kind of suggestion is mixed; the Apollo Neuro was mostly a dud but iron testing caught a lot of issues.

Background

Inositol is a form of sugar. It’s used in messaging between cells in your body, which means it could in theory do basically anything. In practice, supplementation has been found maybe-useful in many metabolic and psychiatric issues, although far from conclusively.

There are a few sources of inositol: it’s in some foods, especially fruit. Your body naturally manufactures some. And some gut bacteria produce it. If your gut bacteria are disrupted, you may experience a sudden drop in available inositol, which can lead to a variety of symptoms including anxiety and depression.

Anecdata

Inositol deficiency (probably) hit me hard 9 years ago, when I went on a multi-month course of some very hardcore antibiotics to clear out a suspected SIBO infection.

Some background: My resistance to Seasonal Affective Disorder has been thoroughly tested and found triumphant. At the time I took those antibiotics I lived in Seattle, which gets 70 sunny days per year, concentrated in the summer. This was a step up from my hometown, which got 60 sunny days per year. I briefly experimented with sunshine in college, where I saw 155 sunny days per year, a full 75% of the US average. The overcast skies never bothered me, and I actively liked Seattle’s rain. So when I say I do not get Seasonal Affective Disorder or light-sensitive depression, I want you to understand my full meaning. Darkness has no power over me.

That is, until I took those antibiotics. I was fine during the day, but as soon as sun set (which was ~5PM, it was Seattle in January) I experienced crushing despair. I don’t know if it was the worst depression in my life, or just the most obvious because it went from 0 to doom 15 minutes.

Then I started taking inositol and the despair went away, even though I was on the antibiotics for at least another month. After the course finished I took some probiotics, ...
If influence functions are not approximating leave-one-out, how are they supposed to help?
by Fabien Roger — 2023-09-22T14:23:45.847Z

Thanks to Roger Grosse for helping me understand his intuitions and hopes for influence functions. This post combines highlights from some influence function papers, some of Roger Grosse’s intuitions (though he doesn’t agree with everything I’m writing here), and some takes of mine.

Influence functions are informally about some notion of influence of a training data point on the model’s weights. But in practice, for neural networks, “influence functions” do not approximate well “what would happen if a training data point was removed”. Then, what are influence functions about, and what can they be used for?

From leave-one-out to influence functions

Ideas from

Bae 2022

(If influence functions are the answer, what is the question?).

The leave-one-out function is the answer to “what would happen, in a network trained to its global minima, if one point was omitted”: LOO(^x,^y)=argminθ1N∑(x,y)∼D−{(^x,^y)}L(fθ(x),y)

Under some assumptions such as a strongly convex loss landscape, influence functions are cheap-to-compute approximation to leave-one-out function, thanks to the Implicit Function Theorem, which tells us that under those assumptions LOO(^x,^y)≈IF(^x,^y)def=θ∗+(∇2θJ(θ∗))−1∇θL(f(θ∗,^x),^y)/N

But these assumptions don't hold for neural networks, and

Basu 2020

shows that influence functions are a terrible approximation of leave-one-out in the context of neural networks, as shown in this figure from

Bae 2022

(left is for Linear Regression, where the approximation hold, right is for MultiLayer-Perceptron, where it doesn’t):

Moreover, even the leave-one-out function is about parameters at convergence, which is not the regime most deep learning training runs operate in. Therefore, influence functions are even less about answering the question “what would happen if this point had more/less weight in the (incomplete) training run?”.

So every time you see someone introducing influence functions as an approximation of the effect of up/down-weighting training data points (as in

this LW post about interpretability

), remember that this does not apply when they are applied to neural networks.

What are influence functions doing

Bae 2022

shows that influence functions (not leave-one-out!) can be well approximated by the minimization of another training objective called PBRF, which is the sum of 3 terms:

Ex∼D[L(fθ(x),fθt(x))], The loss function with the

...
Modeling p(doom) with TrojanGDP
by K. Liam Smith — 2023-09-22T14:19:31.437Z

This is a link post for https://taboo.substack.com/p/modeling-pdoom-with-trojangdp

I was listening to a podcast a couple weeks ago about whether AI will destroy humanity. The economist Tyler Cowen made some critical remarks about the AI safety community:

I put out a call and asked a lot of people I know, well-informed people, 'Is there any actual mathematical model of this process of how the world is supposed to end?’ So, if you look, say, at covid or climate change fears, in both cases, there are many models you can look at and then models with data. I'm not saying you have to like those models. But the point is there's something you look at and then you make up your mind whether or not you like those models and then they're tested against data…So, when it comes to AGI and existential risk, it turns out as best I can ascertain, in the 20 years or so we've been talking about this seriously, there isn't a single model done. Period. Flat out.

I did my own search and also couldn’t find any models on AI existential risk—sometimes sarcastically referred to as p(doom). The podcast came out months ago, but I recently found out about it through a manifold market. Cowen is correct that modeling requires an understanding of the process. This is difficult to do with something that hasn’t been invented yet. A friend of mine described it as trying to model how a warp engine will fail. It’s tough to say when we don’t know how the finished product will work. But for some masochistic reason I decided to try. Fortunately, it’s not quite as tough as a warp engine. We can confidently say that it will be a series of matrix multiplications and not some rules-based engine. I believe that’s enough to at least get a rough start on modeling the problem.

Cowen approaches this from the point of view that it’s the responsibility of AI safety advocates to prove that it’s dangerous, rather than the responsibility of the industry producing AI to demonstrate that it’s safe. That’s not the way the FDA approaches drug approval or the way the car manufacturing industry works.

From what I can find, Cowen is correct that AI safety advocates have not produced a model showing a high risk from AI, but he fails to mention that the academic machine learning community has also not produced a model showing a low risk from AI. This is an odd situation. Geoffrey Hinton, known as the “Godfather of Deep Learni...
Let's talk about Impostor syndrome in AI safety
by Igor Ivanov — 2023-09-22T13:51:18.482Z

Intro

Impostor syndrome is quite common among people working in the AI safety field.
It's quite a unique field. It's extremely important, its challenges are immensely complicated, and it attracts a lot of exceptionally smart and capable people, so the bar for everyone in the field is very high. Moreover, AI safety field experiences a surge of new talents, much more than it can to absorb, so it becomes even more competitive.
I'm a psychotherapist helping people working on AI safety, and in this post I describe causes and manifestations of Impostor syndrome among people from AI safety community, as well as ideas on how to overcome it. This is a post a part of my series about mental health and AI safety.

The story of impostor syndrome of AI safety researcher

Meet Ezra.

He recently finished his masters degree in public policy. After ChatGPT release he became anxious about x-risks, and started feel that he must do something. He starts reading LessWrong, listening podcasts and he quickly realizes that most people in the field are exceptionally smart.

He asks himself "Am I good enough? Can I compete with these people?" He feels intimidated and hesitate to take action.

Finally he got accepted at fellowship in a major AI governance organization. Most of his peers graduated from top universities like Harvard or Cambridge, so he feels even more insecurity about his ability to keep up with them.

On his first day, Ezra attends a meeting about upcoming work. He has ideas, but afraid to look stupid or naive, so he remains silent just to avoid drawing attention.

He is assigned to do research on AI regulation in China. This is a new topic for him, so he has many questions, but he is afraid to ask them, fearing his supervisor will decide that he is underqualified.

Instead, he spends endless hours searching for answers online. He works 80 hours a week. He triple-checks everything. Before presenting his results, he can't stop correcting his slides to the last moment, and after presenting his work, he looks closely at his colleagues' facial expressions for approval or disapproval. Even when his supervisor says "Good job", Ezra believes that he is just polite.

Causes for Impostor syndrome

Ezra focuses all his attention on making sure that others approve him and his job. He believes that he is worse than others, and he is afraid to fail, but there is no way t...
Fund Transit With Development
by jefftk — 2023-09-22T11:10:05.645Z

When transit gets better the land around it becomes more valuable:
many people would like to live next to a subway station. This means
that there are a lot of public transit expansions that would make us
better off, building space for people to live and work. And yet, at
least in the US, we don't do very much of this. Part of it is that
the benefits mostly go to whoever happens to own the land around the
stations.

A different model, which you see with historical subway construction
or Hong Kong's MTR, uses the
increase in land value to fund transit construction. The idea is, the
public transit company buys property, makes it much more valuable by
building service to it, and then sells it.

While I would be pretty positive on US public transit systems adopting
this model, I have trouble imagining them taking it on. Instead,
consider something simpler and more distributed: private developers
paying to expand public transit.

Consider the proposed Somernova
Redevelopment, in Somerville MA:

This is a proposed $3.3B 1.9M-sqft development, adjacent to the Fitchburg
Line. A train station right next to it would make a ton of sense,
and could be done within the existing right of way without any
tunneling. Somernova briefly mentions this idea on p283, where they
say:

Introducing a new train station on campus could dramatically reduce
commute times, making all of Somernova within a five minute walk from
the station. We look forward to ongoing dialog about these transit
possibilities with the community and advocates, ensuring we continue
to explore all options for enhanced connectivity longterm.

This is pretty vague compared to the rest of the plan, which has a ton
of estimates, but we can make our own. The MBTA recently completed a
long and expensive project to extend the
Green Line along this right of way, which stops at Union Square.
Extending it to Dane Street would require another 0.9km of track and
another station. The overall Green Line extension cost $2.2B for
7.6km, or $290M/km, though this included a bunch of over-designed work
that needed to be thrown away and it should
have been far less. This portion is relatively simple compared to
the other work, with no maintenance facility or elevated sections,
though it does include three bridges and moving a
substation. Accepting the $290M/km figure, though, we could estimate
$2...