trentmkelly/LessWrong-43k huggingface dataset:
Of Birds and Bees
by RussellThor — 2024-09-30T10:52:15.069Z

The Hierarchy

There is a hierarchy in life from simple cells to complex cells to multi-cellular creatures to creatures that often live in a groups like birds and bees. As we go up the hierarchy the lower levels have less individual say. For example the mitochondria have no individual life or fitness outside of the cell, and the cells success is not meaningfully measured outside that of the organism in multi-cellular creatures.

Generally as we go up the hierarchy the higher level is more complex and smarter than the lower level. A brain can think, a neuron cannot. A hive of bees or ants is more capable than an individual.

Bird patterns

However you see sometimes see relatively smart creatures such as birds making simple patterns. We are told that this is beautiful, mysterious and presumably smart in some way. However a simple geometric or predictable pattern does not allow intelligence or sophistication. A perfect sinewave contains almost no information and no intelligence.

See here and here.

Is this really the best defense against a larger predator by the birds?

Bee defense

Here is what happens when bees use their best defense against an individually superior attacker. You wonder if the smaller birds could simply swarm the larger attacking bird, as they have vastly superior numbers.

In this case it looks like the flock of birds is less smart than the hive of bees, in spite of an individual bird being a lot smarter than an individual bee. Sometimes it can appear to go even further than that, say a crowd of people panicking is arguably worse than the least capable person in the entire group.

A different rule?

The more complex the pieces the less well they fit together unless there is strong pressure otherwise. Less intelligent units can form a better collective. This is especially true in terms of the relative capabilities. The difference between a single ant and an ant colony is greater than that of an individual wolf compared to a pack of wolves. This is even more extreme when comparing multi-celled creatures to single celled ones. The advantage of the collective is less the larger, more complex the pieces are or greater as the pieces get smaller, such as for cells.

However what about humans? We form very effective collectives, even sacrificing ourselves for the group in spite of our intelligent individuality. Language, consciousness, communicatio...
A new process for mapping discussions
by Nathan Young — 2024-09-30T08:57:20.029Z
Link: https://open.substack.com/pub/nathanpmyoung/p/a-new-process-for-mapping-discussions?r=18mntl&utm_campaign=post&utm_medium=web

Recently my team and I have been working on FindingConsensus.AI. It’s a site to show how public figures would answer specific questions about Artificial Intelligence (but it could work for anything). Often when thinking, I defer to figures I trust, so seeing a spread of public figures gives me a head start. I hope that if we all knew what we all thought, we’d be better at finding consensus, too.

How does the site work?

The questions are currently focused around a Californian AI bill, but this article isn’t deeply focused on that.

Each question has a title and context, and displays public figures and organisations in their answers to a question.

Above, the question is about how many companies the bill, SB 1047, will affect (in the language of the bill, these are called “covered models”). This seems like a pretty important empirical question. Who will the bill affect? If we can’t even agree on that, what can we agree on?

The figure shows estimates of opinions of different public figures and organisations. My team and I read statements from them and guessed their answers to these questions.

As you can see, even on this relatively clear-cut question, there is a lot of disagreement.

The full set of questions (at time of posting) are shown at the bottom of this blog.

Wait, you just guess their opinions?

As regular readers know, I am a staunch defender of guessing. Have I gone too far this time?

It can be hard to find a quote that directly answers a question, even when a person’s opinion feels fairly clear. Public figures hold opinions about AI, many of them tweet about it all day. We feel like we know their views on a topic. But often this isn’t because they ever commented explicitly and directly on a specific question in a couple of sentences.

I don’t think this should stop us from showing their views. In the case of say a16z (a Venture Capital firm) and SB 1047 (this bill), they have made their views clear - they think it will damage the ecosystem and perhaps even drive companies out of San Francisco. It is reasonable to represent this on the chart as them saying it will almost certainly hamstring the AI industry, even without perfect quotes.

I am tempted to guess the opinions of organisations even where there are no quotes at all, but I sense this would go too far. The upside would be that currently it’s beneficial to be ambi...
MATS Alumni Impact Analysis
by utilistrutil — 2024-09-30T02:35:57.273Z

Summary

This winter,

MATS

will be running our seventh program. In early-mid 2024, 46% of alumni from our first four programs (Winter 2021-22 to Summer 2023) completed a survey about their career progress since participating in MATS. This report presents key findings from the responses of these 72 alumni.

78% of respondents described their current work as "Working/interning on AI alignment/control" or "Conducting alignment research independently."

49% are "Working/interning on AI alignment/control."
29% are "Conducting alignment research independently."
1.4% are "Working/interning on AI capabilities."

Since MATS, 54% of respondents applied to a job and advanced past the first round of interviews.

64% of those who shared more details accepted a job offer.
Alumni reported that MATS made it more likely that they applied to these jobs by helping them build legible career capital and develop research/technical skills.

During or since MATS, 68% of alumni had published alignment research.

The most common type of publication was a LessWrong post (45%).
78% of respondents said their publication “possibly” or “probably” would not have happened without MATS.
10% of alumni reported that MATS accelerated publication by more than 6 months; 14% said 1-6 months.
8% of alumni responded that MATS resulted in a “much higher” quality of their publication.

63% of scholars met a research collaborator through MATS
At this stage in their careers, 46% of alumni would benefit from more connections to research collaborators, and 39% would benefit from job recommendations.

Background on Cohort

For 40% of respondents, their highest academic degree was a Bachelor’s; 40% had earned at most a Master’s, and 20%, a PhD.

Their most common categories of current work were “Working/interning on AI alignment/control” (49%) and “Conducting alignment research independently” (29%).

Here are some representative descriptions of the work alumni were doing:

“Going through the first year of grad school at Oxford and continuing research that emerged from my time at MATS.”
“Working on an interpretability project at AI Safety Camp and just finished the s-risk intro fellowship by CLR a week or two ago.”
“What could be called "prosaic agent foundations" with the AF team @ MIRI”
“​​Co-founding for-profit AI safety company with a product”
“I'm back at my PhD at Imperial, taking an idea I developed during MATS 3.1 and trying to tur

...
the case for CoT unfaithfulness is overstated
by nostalgebraist — 2024-09-29T22:07:54.053Z

[Quickly written, unpolished. Also, it's possible that there's some more convincing work on this topic that I'm unaware of – if so, let me know. Also also, it's possible I'm arguing with an imaginary position here and everyone already agrees with everything below.]

In research discussions about LLMs, I often pick up a vibe of casual, generalized skepticism about model-generated CoT (chain-of-thought) explanations.

CoTs (people say) are not trustworthy in general. They don't always reflect what the model is "actually" thinking or how it has "actually" solved a given problem.

This claim is true as far as it goes. But people sometimes act like it goes much further than (IMO) it really does.

Sometimes it seems to license an attitude of "oh, it's no use reading what the model says in the CoT, you're a chump if you trust that stuff." Or, more insidiously, a failure to even ask the question "what, if anything, can we learn about the model's reasoning process by reading the CoT?"

This seems unwarranted to me.

There are a number of research papers out there on the topic of CoT unfaithfulness. I have read some of the key ones. And, while they do demonstrate... something, it's not the kind of evidence you'd need to justify that generalized "only chumps trust the CoT" vibe.

And meanwhile, if we view "reading the CoT" as a sort of interpretability technique – and compare it in a fair way with other interpretability techniques – it has a lot of striking advantages. It would be a shame to dismiss this "technique" out of hand for no good reason.

What does the literature on CoT unfaithfulness actually say?

(For a useful critical survey of this literature, see Parcalabescu and Frank 2023. Note that the first point I'll make, immediately below, is a focus of the linked paper.)

Naively, we'd expect it to address a question like: "does the argument presented in a CoT reflect (or perhaps 'fully reflect') the actual way the model came to its conclusion?"

However, the literature doesn't really answer this question. To answer it, we'd need to know "the actual way the model came to its conclusion." But with LLMs, no one knows that.

Interpretability isn't that far along yet. At best, we have some partial explanations of how some fairly simple things get computed inside the model.

We don't know fully understand all the stuff the model is "actually" doing, as it computes the answer. (Whe...
Pomodoro Method Randomized Self Experiment
by niplav — 2024-09-29T21:55:04.740Z

Value tracked
Effect size d (λ, p, σ change, k)

Productivity
0.26 (λ≈6.23, p≈0.069, 0.05, 52)

Creativity
-0.04 (λ≈0.58, p≈0.92, 0.01, 52)

Subjective length
-0.147 (λ≈3.33, p≈0.37, 0.03, 52)

Happiness
-0.07 (λ≈0.32, p≈0.96, 0.01, 111)

Contentment
-0.13 (λ≈1.08, p≈0.83, 0.05, 111)

Relaxation
-0.04 (λ≈1.23, p≈0.8, -0.25, 111)

Chastity
-0.14 (λ≈7.76, p≈0.02, 0.74, 111)

I ran the experiment from 2024-01-29 to 2024-06-17,
using spt with this
script, managed by this script.

The data on whether a particular day was a pomodoro-method day was saved
in this file, and the data on the pomodoros
was saved in this file.

The code for loading and transforming the pomodoro data isn't
particularly interesting, if you're curious you can find it in this
file.

datasets=get_datasets_pom()

Let's proceed to the analysis, then (using the same methodology as for my
nootropics experiments:

res=analyze(datasets)

And the results are:

>>> res
productivity creativity sublen happy content relaxed horny
d 0.259951 -0.041504 -0.147437 -0.073699 -0.132798 -0.038319 -0.144040
λ 6.225107 0.583007 3.329000 0.318865 1.078502 1.232905 7.756272
p 0.069062 0.918520 0.368416 0.959552 0.827240 0.795999 0.022903
dσ -0.050269 0.013871 0.033902 0.007177 0.047723 -0.252365 0.744675
k 52.000000 52.000000 52.000000 111.000000 111.000000 111.000000 111.000000

I didn't meditate or do flashcards during that time, so most of the
results don't contain any content.

So the pomodoro method somewhat increases productivity (at the edge of
statistical significance), and maybe decreases subjective length of
the day a bit.

Scoring the Market

As this experiment was done in the context of an endeavour of selecting QS experiments via prediction markets, I can now score the market:

def logscore(o,p):
return np.mean(o*np.log(p)+(np.ones_like(o)-o)*np.log(np.ones_like(p)-p))
p=np.array([0.06, 0.29, 0.39, 0.19, 0.07])
o=np.array([0, 0, 1, 0, 0])
logscore(outcomes, p)
-0.3258531953347593

Honestly: The market did pretty well.
Toy Models of Superposition: Simplified by Hand
by Axel Sorensen — 2024-09-29T21:19:52.475Z

Introduction

In Anthropic's paper "Toy Models of Superposition", they illustrate how neural networks represent more features than they have dimensions by embedding them in so-called "superposition". This allows the model to represent more information (although at the cost of interference) and leads to polysemantic neurons (that correspond to multiple features).

When I first read the paper, I struggled to intuitively grasp what the embeddings in superposition actually represented and what interference between features concretely looked like. In this post, I aim to break down the concept of superposition by hand and provide a simple walkthrough of the underlying "machinery".

I must confess that I am relatively new to the field of mechanistic interpretability and I warmly welcome any thoughts, suggestions, or corrections on my interpretations.

Let's begin with a refresher on superposition.

Superposition: A simple example

When features are sparse (meaning that they rarely co-occur), superposition becomes an attractive strategy. The authors illustrate this with a simple example. Consider a toy model trained to embed five features of varying importance in two dimensions (see Figure 1). In this case, the training data (of size N) is generated by sampling 5 random numbers N times from a uniform distribution. Feature sparsity determines how likely it is for each feature in a data point to be zero.

Figure 1: Embedding of five features of varying importance in 2D space. The color of a feature denotes its importance (lighter = more important). As sparsity increases more features are embedded in "superposition".

What do the features represent?

In image or text data, features are usually not explicitly specified. However, if certain patterns or characteristics of the input data are useful for the task at hand, a model may learn to represent the presence of these "features".

In our case, we skip a step and artificially populate the activations of each of these features. For example, the input to our model might look like this:

[0.10.60.210.7]

When features always co-occur (0% sparsity), the model learns to represent an orthogonal basis of the most important two features while ignoring the rest. As feature sparsity increases, the model gradually embeds more features until all five features are embedded in superposition at the cost of interference between the embeddings.

But ...
LLMs are likely not conscious
by research_prime_space — 2024-09-29T20:57:26.111Z

I think the sparse autoencoder line of interpretability work is somewhat convincing evidence that LLMs are not conscious.

In order for me to consciously take in some information (e.g. the house is yellow), I need to store not only the contents of the statement but also some aspect of my conscious experience. I need to store more than the minimal number of bits it would take to represent "the house is yellow".

The sparse autoencoder line of work appears to suggest that LLMs essentially store "bits" that represent "themes" in the text they're processing, but close to nothing (at least in L2 norm) beyond that. And furthermore, this is happening in each layer. Thus, there doesn't appear to be any residual "space" that left over for storing aspects of consciousness.
A Policy Proposal
by phdead — 2024-09-29T20:45:34.745Z

(Crosspost of https://phoropter.substack.com/p/a-policy-proposal)

TL;DR: I think that the features used by recommendation systems should be configurable by end users receiving recommendations, and that this ability should be enforced by policy. Just as the GDPR protects a user's ability to choose which cookies are enabled, a user should be able to pick what data goes into any algorithmically generated feed they view. The legislation would also enforce a minimum granularity for dividing feature inputs.

I expect this policy proposal to get a lot of pushback, and this post is about explaining:

The specific problems this proposal tries to solve.

Why this proposal solves those problems

Why this is feasible on a technological / political level

What the challenges are for effective implementation of this policy

Part 1: Why regulate recommendation systems?

There are two reasons a government may want to regulate a recommendation system.

Governments may believe regulating recommendation systems may be societally beneficial, by enabling users to better control their relationship with technology.

Governments may believe users have a right to control how their data is used.

These are both reasonable. To the first point, we all know people who cannot control their relationship to media to the point it interferes with their daily life. Many of us wish we used our phones less but are stuck "rotting in bed" due to a lack of tools to control how addictive our social media experience is.

In my own life, using parental controls on my phone (with a friend as the 'parent') and using apps such as Cold Turkey on my laptop have drastically improved my quality of life. However these tools are tricky to set up and force an all or nothing approach; better consumer rights would enable more people to realize the benefits of a healthy relationship to recommendation engines.

To the second point, remember that cold feeling that shivered down your spine when it came out that Facebook was listening in on people's audio messages to better recommend products? That worry almost feels blasé now in the age of massive data collection and aggregation, but it would be nice if you could selectively prevent social media websites from using every piece of data they have about you while still retaining use of the app.

Part 2: Why control over what features are used?

I've been thinking about this problem ...
Do Sparse Autoencoders (SAEs) transfer across base and finetuned language models?
by Taras Kutsyk — 2024-09-29T19:37:30.465Z

This is a project submission post for the AI Safety Fundamentals course from BlueDot Impact. Therefore, some of its sections are intended to be beginner-friendly and overly verbose for familiar readers (mainly the Introduction section) and may freely be skipped.

TLDR (Executive Summary)

We explored whether Sparse Autoencoders (SAEs) can effectively transfer from base language models to their finetuned counterparts, focusing on two base models: Gemma-2b and Mistral-7B-V0.1 (we tested finetuned versions for coding and mathematics respectively)
In particular, we split our analysis into three steps:

We analysed the similarity (Cosine and Euclidian Distance) of the residual activations, which was highly correlated with the resulting transferability of the SAEs for the two models.
We computed several performance metrics (L0 Loss, Reconstruction CE Loss, Variance Explained) of the base SAEs on the fine-tuned models. Almost all metrics agreed on a significant degradation of the SAE performance for the Gemma-2b model, and remained within a reasonable range for the Mistral-7B model, indicating a much better transferability.
We took a further step by operationalizing the idea of transferability of SAE from base models to fine-tuned models by applying an approach from Towards Monosemanticity for studying feature universality through feature activation similarity and feature logit similarity. These similarity scores were mostly consistent with the results from the previous step, albeit with one caveat for the Gemma-2b model, suggesting that some SAE features may still transfer even if the overall SAE performance is poor for the finetuned model.

Overall, our results agree with previous work that studied Instruct models. That is, SAEs transferability seems to be model-dependent and sensitive to the finetuning process.
We make our code repository public to facilitate future work in this direction.

1. Introduction and motivation

1.1 What are SAEs and why do we care about them

We find ourselves in a world where we have machines that speak fluently dozens of languages, can do a wide variety of tasks like programming at a reasonable level, and we have no idea how they do it! This is a standard mechanistic interpretability (a.k.a. mech interp) pitch (borrowed from Neel Nanda) - a field that is trying to express neural networks’ behaviours as human-understandable algorithms, i.e. reverse en...
Models of life
by Abhishaike Mahajan — 2024-09-29T19:24:40.060Z
Link: https://www.asimov.press/p/models-life

2024

Statistical models of organisms have existed for decades. The earliest ones relied on simple linear regression and attempted to correlate genetic variations with observable traits or disease risks — such as

drug metabolization rates

or

cancer susceptibility

. As computational power increased and machine learning techniques advanced, the models’ sophistication grew.

In time, they were colloquially referred to as “models of life.”

The definition was nebulous, but there were agreed-upon themes. All models of life were aimed at improving our understanding of the cellular mechanisms underlying biology and were neither constrained by human intuition nor limited to predefined hypotheses. They operated in high-dimensional spaces that defied simple visualization while incorporating vast layers of interconnected variables that no human mind could fully grasp. Unlike traditional scientific models, which often simplified reality, these models embraced its messy and chaotic nature.

The first model of life emerged sometime in 2022 or 2023.

Given the fuzziness of the definition, it was unclear which of the released projects deserved the name. There was

scFormer

in 2022,

scGPT

in 2023, and plenty of others. But, regardless of which was first, they all operated with the same core data as their mechanism for understanding life: messenger RNA (

mRNA

).

Collections of mRNA have been understood as a

proxy for cell states for decades

. mRNA was the intermediate stage between DNA and protein, a dynamic entity that shifted depending on the second-to-second needs of the cell, able to point out if a cell was cancerous or stressed, what kind of cell it was, and so on. Reliance on mRNA

had plenty of failure modes

, but it was the most abundant source of cell-state data the scientific community had: DNA alone was static, and proteins were too hard to quantify en masse.

Despite semantic differences between these first models of life, their training methodology closely resembled one another. A sequenced set of mRNA values from a given cell — one value for each of the 20,000 protein-coding genes in a human body — was randomly masked and the model asked to in-fill what it thought should be there, analogous to guessing what jigsaw pieces were missing given the rest of the puzzle. If a cell expressed high levels of genes associated with cell division, other cell cycle-related genes...