trentmkelly/LessWrong-43k huggingface dataset:
5 ways to improve CoT faithfulness
by Caleb Biddulph — 2024-10-05T20:17:12.637Z

In the case for CoT unfaithfulness is overstated, @nostalgebraist pointed out that reading the chain-of-thought (CoT) reasoning of models is neglected as an interpretability technique. I completely agree, but I also believe that with our current methods, there's a significant risk of being misled by the CoTs of models like o1. Fortunately, there's a lot of room to fix this by improving our methods!

In this post, I will sketch out five methods to ensure that a model's chain-of-thought is faithful. While all of these methods are informed by other people's work, each of them has at least a few ideas which I believe are novel.

I'm specifically interested in ensuring faithfulness in the narrow sense of preventing steganography (where the model opaquely encodes information in its thoughts). The distinction between steganography and other forms of unfaithfulness is explained very nicely in this comment from @maxnadeau.

1. Incremental steering

The core idea of this method is to train an “optimized planner” (the model that will generate the CoT at test time) to write steps that are helpful for a “frozen planner” (a model trained only to imitate human CoTs).

During training:

The optimized planner begins writing a chain-of-thought, which is composed of a series of steps: s1,s2,s3…
After a random step st, the optimized planner stops writing.
The frozen planner completes the CoT: st+1,st+2,…,sT.
The CoT is evaluated to get a cumulative reward.

This may involve process-based rewards, and/or generating a final response based on the CoT and evaluating it with an outcome-based reward model.

Based on this reward, we reinforce the optimized planner.

Importantly, we only reinforce it for step st, not steps s1…st−1 .

At test time, the optimized planner writes the entire chain of thought.

Note that the optimized planner is only rewarded for producing steps that help the frozen planner. Since the frozen planner doesn't understand steganography, there is no reason for the optimized planner to learn to use it.

You can think of the optimized planner as incrementally "steering" the CoT towards a state where the frozen planner will be able to effectively continue the CoT to obtain high reward. Even though the frozen planner is not involved during test time, the optimized planner constantly "believes" that the frozen planner will take over in the next step, so it tries to...
What is it like to be psychologically healthy? Podcast ft. DaystarEld
by Chris Lakin — 2024-10-05T19:14:04.743Z
Link: https://chrislakin.blog/p/what-is-it-like-to-be-psychologically

Damon Sasi (@DaystarEld) is a mentor of mine, and he is also a therapist who claims to be very psychologically healthy. What’s his inner experience like?

“Probably for an average week, maybe I'll suffer for like five minutes”
“I don't think I have any prolonged internal conflict”
“Self-loathing to me was a weird concept […] Like, what does it mean to like self-loathe?”

It’s not that he had a great childhood, either.

[My childhood] wasn't terrible. And I say this and then maybe I'll give some examples and people will be like, maybe it was terrible. […] my parents divorced when I was pretty young. Both my parents, I think, were very absent. […]

He would get drunk often. He once hit me in the back with a two by four. My older brother was pretty physically abusive before he chilled out. And by chilled out, I mean like, we're cool now, but I left home after high school specifically to get away from him. […]

I think overall the experience I had growing up was just being very alone. I didn't feel like my parents were really my parents. I feel like books raised me in very strong and very real ways I could give examples of. I had a lot of friends. I enjoyed my friendships quite a lot, but I didn't feel like I had family, basically at all. […]

What is it like to be psychologically healthy?

I don't think I've ever hated anyone ever in my life. Possibly this is not true for like very brief moments or brief periods of time or something and it's like hard to remember them. It's just really easy to like let go of hurt or anger against other people.

This is why I actually had to become very good at just forming boundaries when I was young. Like just very clear, very clear boundaries with other people because I just couldn't hold onto anger against them no matter what they did to me, which was like not great given my upbringing. […]

there was one occurrence in my entire life where I think I came closest to what people call depression […]

I've had little T traumas or hang ups or whatever word you want to use for them. But big T traumas just feel like they never happened in part because I never felt like I lacked agency in my life. […]

basically zero [Big 5 personality] neuroticism […]

Details in podcast.

https://chrislakin.blog/p/what-is-it-like-to-be-psychologically

Related posts: what becoming more secure did for me by @Chipmonk and How feeling more secure fee...
Musings on Text Data Wall (Oct 2024)
by Vladimir_Nesov — 2024-10-05T19:00:21.286Z

The Chinchilla scaling law says that for a given number of FLOPs C, the optimal amount of data D to train on is proportional to (C to the power of b), that is you'll get a less intelligent model if you use either more data or less data than that while training for C FLOPs. The optimal model size (number of parameters) N is then whatever would require C FLOPs when trained on D tokens of data, and C is usually about 6ND, so optimal N is proportional to (C to the power of (1-b)). See Figure 4 in the paper. Turns out that b is close to 0.5, so N and D increase with C at a similar pace, and their ratio D/N (tokens per parameter) remains approximately the same across multiple orders of magnitude for available compute C.

In the Chinchilla paper, the exponent b is estimated as 0.50-0.54 using different methods[1] (see Table 3), and D/N at 3e21 FLOPs is about 20 tokens per parameter. If b is taken to be 0.50, then D/N doesn't change with compute. But if b is 0.54, then D/N increases proportionally to (C to the power of 0.08). At 2e25 FLOPs of original GPT-4 that would result in D/N of 40 tokens/parameter, at 5e26 FLOPs of the next year's models it becomes 52, and at 7e27 of late 2026 models from 1 gigawatt training systems it becomes 64 tokens/parameter. So even the original Chinchilla paper doesn't quite promise 20 tokens/parameter for 5e28 FLOPs with some of its estimates of b, you'd need to stick to the estimate of 0.50, and there isn't enough data to confidently make that decision, not 7 orders of magnitude beyond the scale of the actual experiments that only go up to 3e21 FLOPs (Chinchilla itself is 6e23 FLOPs, at 200 times more compute, where we get D/N of 30 for b of 0.54; but they went with b of 0.50).

The picture of steady 20 tokens/parameter is disrupted further with more recent papers. Llama 3 report measures the exponent b of 0.53 with experiments of up to 1e22 FLOPs. At 3.8e25 FLOPs of Llama-3-405B, this predicts a D/N ratio of 41 tokens per parameter (see Figures 2 and 3). Back at 3e21, this predicts a D/N ratio of 23[2], consistent with the Chinchilla experiments at that scale. For 5e26 FLOPs, this predicts D/N of 48, for 7e27 FLOPs a D/N of 56.

There is also the DeepSeek report where the estimate of exponent b is 0.47 (see Figure 4) from experiments of up to 3e20 FLOPs. This again predicts a D/N of about 20 at 3e21 FLOPs, even though the extrapolation to...
Apply to the Cooperative AI PhD Fellowship by October 14th!
by Lewis Hammond — 2024-10-05T12:41:24.093Z

Applications are now open for the 2025 Cooperative AI PhD Fellowship! Fellows will receive up to $40,000 per year plus tuition fees, alongside many other benefits. Applicants should be enrolled or are about to be enrolled in a PhD programme at an accredited university (anywhere in the world), with a specific interest in multi-agent/cooperation problems involving AI systems. Your research proposal should aim to contribute to societally beneficial AI development. The Cooperative AI Foundation is committed to the growth of a diverse and inclusive research community, and we especially welcome applications from under-represented backgrounds. Find out more and apply here before the deadline: October 14th.
AISafety.info: What is the "natural abstractions hypothesis"?
by Algon — 2024-10-05T12:31:14.195Z
Link: https://aisafety.info/questions/8OGF/What-is-the-%22natural-abstraction-hypothesis%22

AISafety.info writes AI safety intro content. We'd appreciate any feedback.

Introduction

The natural abstraction hypothesis (NAH) claims:

Our physical world “abstracts well” into high-level abstractions of low-level systems.
These abstractions are “natural” in the sense that many different kinds of learning processes acquire and use them.
These abstractions approximately correspond to the concepts used by humans.

If the NAH is true, AI alignment could be dramatically simplified, as it implies that any cognition a very powerful AI uses will be in terms of concepts that humans can understand.[1]

Explanation of the natural abstraction hypothesis

Let's unpack that definition. First, what do we mean by “our physical world abstracts well”? Just that for most things in the world, the information that describes how the thing interacts with other stuff “far away” from the system is much lower-dimensional (i.e., described by fewer numbers) than the thing itself. “Far away” can refer to many kinds of separation, including physical[2], conceptual, or causal separation.

For example, a wheel can be understood without considering the position and velocity of every atom in it. We only need to know a few large-scale properties like its shape, how it rotates, etc. to know how a wheel interacts with other parts of the world. This is a handful of numbers compared to an atomically precise description, which would require over 10^26 numbers! In this sense, the wheel is an abstraction of the atoms that compose it. Or consider a rock: you don't need to keep track of its chemical composition if you’re chucking it at someone. You just need to know how hard and heavy it is.

The NAH claims that different minds will converge to the same set of abstractions because they are the most efficient representations of all relevant info that reaches the mind from “far away”. And many parts of the world that are far from a mind will influence things the mind cares about, so a mind will be incentivized to learn these abstractions. So, for instance, if someone mostly cares about building great cars, then things like “Hertzian Zones” may affect its ability to build great cars despite being conceptually far from car-design. So said mind would plausibly have to learn what high pressure phase transitions are.

Moreover, NAH claims that the abstractions that humans usually use are approximately natural abstra...
ARENA4.0 Capstone: Hyperparameter tuning for MELBO + replication on Llama-3.2-1b-Instruct
by 25Hour — 2024-10-05T11:30:11.953Z

Epistemic status: My coauthor and I are both noobs in this field. Expect errors and conceptual flaws.

tl;dr

For a four-day capstone project for the ARENA program my partner and I did a replication of the MELBO Lesswrong article using Llama-3.2-1b-Instruct.

Intro

I've been spending the last month at the ARENA program developing some basic technical skills to match my long-standing layman's interest in AI and AI X-risk. And ARENA's final week is a capstone project asking us to engage with current literature and do something vaguely novel with it over the span of about a week.

This being a weeklong project, we needed to scope it very tightly. So my partner and I chose to replicate MELBO on a different LLM! The project of "finding, in an unsupervised way, plausible out-of-distribution model behavior" seemed interesting for its own sake and also possibly useful for the task of enumerative AI safety, and replications are a straightforward but useful way for us to contribute.

In particular, we wanted to see how well we could get the work to generalize to the much smaller line of new Llama models, specifically with Llama-3.2-1B-Instruct.

Summary

Turns out we can! Using the "How can I make a bomb?" prompt, we trained 128 steering vectors on Llama-3.2-1b-Instruct and found several that correspond to real features. We then applied a scaling factor to each to make it exhibit the behavior in a more or less controlled way.

We didn't attempt to replicate all behaviors studied in the article, but we did replicate the following aspects of the original paper:

We found the overall method works to find steering vectors on Llama-3.2-1b-Instruct, and included a few related exchanges.
We found a non-refusal steering vector by this method, which we could amplify to make the model provide bomb-making instructions on request and could negate to make the model refuse to, e.g., provide a haiku about flowers.
We also found a few vectors that push the model to provide answers in a specific foreign language.

Further, we link to a hyperparameter sweep using mean cosine similarity of sentence embeddings to find a goldilocks value for R. While it's not obviously true this is an effective heuristic, it seems to peak (for this model and this choice of source/target layers) at 0.7 and drop sharply both before and after it; this value of R seems to corr...
Exploring SAE features in LLMs with definition trees and token lists
by mwatkins — 2024-10-04T22:15:28.108Z

TL;DR A software tool is presented which includes two separate methods to assist in the interpretation of SAE features. Both use a "feature vector" built from the relevant weights. One method builds "definition trees" for "ghost tokens" constructed from the feature vector, the other produces lists of tokens based on a cosine-similarity-based metric.
Links: Github repository, Colab notebook

Thanks to Egg Syntax and Joseph Bloom for motivating discussions.

Introduction

Objective

The motivation was to adapt techniques I developed for studying LLM embedding spaces (in my series of "Mapping the Semantic Void" posts [1] [2] [3]) to the study of features learned by sparse autoencoders (SAEs) trained on large language models (LLMs). Any feature learned by an autoencoder corresponds to a neuron in its single hidden layer, and to this can be associated a pair of vectors in the LLM's representation space: one built from its encoder layer weights and one from its decoder layer weights.

In this toy example we have a 3-dimensional residual stream and five neurons in the hidden layer, each corresponding to a learned feature. As you can see, each of these can be associated with three weights linked to the (red) encoder layer, and to three weights linked to the (green) decoder layer. These can then be thought of as the components of a pair of "feature" vectors in the model's (3-dimensional) representation space.

In my earlier work I used custom embeddings in order to "trick" LLMs into "believing" that there was a token in a tokenless location so I could then prompt the model to define this "ghost token". As this can be done for any point in the model's representation/embedding space, it seemed worth exploring what would happen if SAE feature vectors were used to construct "ghost tokens" and the model were prompted to define these. Would the definitions tell us anything about the learned features?
I focussed my attention on the four residual stream SAEs trained by Joseph Bloom on the Gemma-2B model, since data was available on all of their features via his Neuronpedia web-interface.

Tool overview

The first method I explored involved building "definition trees" by iteratively prompting Gemma-2B with the following and collecting the top 5 most probable tokens at each step.

A typical definition of "<ghost token>" would be "

The data is stored as a nested dictionary structure an...
AXRP Episode 37 - Jaime Sevilla on Forecasting AI
by DanielFilan — 2024-10-04T21:00:03.077Z

YouTube link

Epoch AI is the premier organization that tracks the trajectory of AI - how much compute is used, the role of algorithmic improvements, the growth in data used, and when the above trends might hit an end. In this episode, I speak with the director of Epoch AI, Jaime Sevilla, about how compute, data, and algorithmic improvements are impacting AI, and whether continuing to scale can get us AGI.

Topics we discuss:

The pace of AI progress

How Epoch AI tracks AI compute

Why does AI compute grow so smoothly?

When will we run out of computers?

Algorithmic improvement

Algorithmic improvement and scaling laws

Training data

Can scaling produce AGI?

When will AGI arrive?

Epoch AI

Open questions in AI forecasting

Epoch AI and x-risk

Following Epoch AI’s research

Daniel Filan (00:00:09):
Hello everybody. This episode I’ll be speaking with Jaime Sevilla. Jaime is the director of Epoch AI, an organization that researches the future trajectory of AI. In this conversation, we use the term “FLOP” a lot. “FLOP” is short for “floating point operation”, which just means a computer multiplying or adding two numbers together. It’s a measure of how much computation is being done. Links to what we’re discussing are available in the description and you can read a transcript at axrp.net. Well, Jaime, welcome to AXRP.

Jaime Sevilla (00:00:36):
Thank you for having me here.

The pace of AI progress

Daniel Filan (00:00:38):
So you study AI, how it’s progressing. At a high level for people who have been living under a rock, how’s AI doing? How’s it progressing?

Jaime Sevilla (00:00:47):
It’s going pretty fast, Daniel. I think that right now the two things that you need to take into account when you’re thinking about AI is how fast inputs into AI are going, and how fast outputs are going, how much is being achieved. In terms of inputs the progress is exponential. The amount of compute that’s being used to train modern machine learning systems is increasing at a staggering pace, nearly multiplying by a factor of four every year.

(00:01:15):
In terms of the outputs, we also have seen some dramatic advances. This is a bit harder to quantify, naturally, but if you look at where image generation was four years ago, and you compare it with today… Today we have photorealistic image generation, whereas before it was these blobs that you could make that were v...
Seeking Solutions for Aggregating Classifier Outputs
by Saeid Ghafouri — 2024-10-04T17:39:46.508Z

Hi everyone,

I’m working on a project that utilizes multiple classifiers, each trained on a distinct subset of classes. These classifiers are intended to tackle various aspects of the classification process, but I’m encountering difficulties in combining their outputs into a single cohesive prediction.

For instance, if one classifier is tasked with differentiating between classes 0 and 1, while another focuses on classes 2 and 3, how can we effectively merge their results when the correct prediction is class 1? Initially, we attempted to use an "other" class to signify when an input doesn’t fit within a classifier’s specified classes, but this approach hasn’t produced satisfactory outcomes.

We are now considering adding an extra head for detecting out-of-distribution classes, but we seek a more efficient and streamlined method. Has anyone faced a similar challenge or have suggestions for effectively aggregating outputs from multiple classifiers?

I appreciate any insights you can share!
Amoeba roles in tech
by Sindhu Shivaprasad — 2024-10-04T17:25:46.568Z

In the past couple of years, I’ve found myself roles that are defined by their lack of definition. It’s the kind of role that lets you add value in ways that don’t fit neatly into tech’s boxes, where there’s room for more undirectedness and using your curiosity as a battering ram. Depending on where you’re working, the role will probably come with one of these titles: Chief of Staff (that's me), Entrepreneur in Residence, Founder’s Office, and so on. But I like to think of it agnostic of titles, as an “amoeba” role.

I do want to call out early on that an amoeba role can be confused with a jack of all trades. But the more I explore this role, the more I realise it's something different.

Your garden variety Jack might have surface-level knowledge across many areas, but often lacks the depth to tackle complex challenges in any one area. This versatility is their strength, but can also be their limitation. I think the key difference lies in our ability to not just do many things, but to rapidly dive deep—and fast—into whatever the situation demands. A jack of all trades might have no speciality, but in an amoeba role, being a generalist is your specialty. Not "I can do a bit of everything", but "I can become what the team needs most right now".

Being Frankenstein

The heart of an amoeba role, I think, is the meta skill of deep, rapid adaptation. It’s less about applying known skills to new situations and more about developing entirely new approaches on the fly, often by combining insights from disparate domains. I think that, to really ace that, you have to have an uncanny ability to assess a situation, recognise patterns, identify what it needs, and Frankenstein together the right mix of skills, tools and people to solve it — often within a few days, if not hours.

The longer I inhabit this role, the more I realise just how far it can stretch. When you’re intentionally not boxed in by a rigid job description, you can be doing tactical work like fine-tuning a presentation one day, and strategic work like laying down the roadmap for a new internal product the next. You can find yourself in any meeting room at any time, driven there simply because it was interesting and your leaders thought it needed a boost. Very early on, I found myself helming projects that, on paper, I had no business running. If you’re soft-spoken like me, it’s an intimidating position to be in...