trentmkelly/LessWrong-43k huggingface dataset:
Being hella lost as rationality practice
by Rachel Shu — 2024-06-24T23:50:59.991Z

I've had the privilege of knowing where I was only in the sense of "within this quadrangle of an insufficiently detailed map, with impassable terrain not well indicated" several times. Usually by choice. More often while alone than prudence would dictate. I think they've been pretty good informal exercises in applied Bayesian reasoning.

For illustration, I'll use the conditions I faced as the first person on a particular backpacking trail since the snow melted. Much of this trail had been washed out during the winter. This was in the High Sierras in California, so the putative trail was mostly over a mix of smooth granite surfaces, talus slopes (loose rocks and boulders), and pine forest.

Trails over solid rock. These are usually marked by a series of towers of stacked rocks, called cairns. A trail in disrepair is going to have some of these still standing, and some of them toppled or even scattered. Also, there are natural piles of rock that sort of look like cairns or toppled cairns. Finding your way between them is a matter of joint probabilities and Markov chains: your confidence of being on the trail is the odds that this pile of rocks is a cairn, given the odds that the previous pile of rocks was a cairn. Sometimes you end up deciding that you are probably no longer following a real trail and backtrack a long ways.

Trails over talus slopes. Trail planners avoid building these as they're unstable and hard to mark, but sometimes it was their only option, or, if you're off trail, the only reasonable way to get to where the trail is, or might be (usually the alternative is steep cliff face). Your best bet is often picking a sensible point on the opposite side, saying "I want to be there", and ascending, descending, traversing, and scrambling through the rocks in whatever manner feels safest and least strenuous. Perhaps you get there and it turns out that it's not where you need to be. Reorient, see if you can get to the right place from where you are, double back if you have to, and try again.

Trails through forest. These are often the most clearly marked, and most likely to look like an actual trail on the ground. They're also the hardest to find again when lost, due to reduced visibility between the trees. Paths might actually just be rivulets or deer trails. A path might go into a snowbank or thicket and be hard to find on the other side. Look for signs of ...
A Basic Economics-Style Model of AI Existential Risk
by Rubi J. Hudson — 2024-06-24T20:26:09.744Z

Crossposted with my new blog, Crossing the Rubicon, and primarily aimed at x-risk skeptics from economics backgrounds. If you're interested in novel takes on theoretical AI safety, please consider subscribing! Thanks to Basil Halperin for feedback on a draft of this post.

“So, when it comes to AGI and existential risk, it turns out as best I can ascertain, in the 20 years or so we've been talking about this seriously, there isn't a single model done. Period. Flat out. So, I don't think any idea should be dismissed. I've just been inviting those individuals to actually join the discourse of science. 'Show us your models. Let us see their assumptions and let's talk about those.” -

Tyler Cowen on EconTalk

Last May, the economist Tyler Cowen called out the lack of formal models from the AI X-risk community. The response of the AI x-risk community has basically been to ignore this. If the conceptual case is straightforward enough, what’s the point of modeling it? Aren’t there obviously enough degrees of freedom to make the model say anything? Besides, Cowen doesn’t seem like he’s very open to being convinced. Making a model like that would just be a waste of time. Those were my initial thoughts upon hearing the challenge, at least, and I suspect they were widely shared.

But things have changed for me since Cowen made his comments; I now have a blog! This is expressly a place for me to post things even if I don’t think they’re the best use of my research time. So I’d like to start by making the case for AI x-risk with a simple, but basically formal model. I don’t expect it to radically update anyone’s beliefs, but I’m hoping it can be a starting point for further discussions.

The Model

We’ll start with a model that should be familiar to economists, it’s a Walrasian setup without production.

Suppose we have an economy with n consumers and m goods. Each consumer i has a utility function over goods consumed, ui(xi1,...,xim),, where xij∈R+ is the quantity of good j that consumer i ends up with. These utility functions are continuous, strongly monotonic, and concave in the quantity of each good consumed. Each consumer also has an initial endowment of goods, with eij > 0 representing the amount of good j that consumer i starts with. There is a vector of prices, p, with pj giving the price for good j.

Consumers maximize their utility subject to the...
The Minority Coalition
by Richard_Ngo — 2024-06-24T20:01:27.436Z
Link: https://www.narrativeark.xyz/p/the-minority-faction

Hello everybody. Or maybe nobody. I don’t know yet if I’m going to release this stream, I could get in pretty hot water for it. But you guys know that hasn’t stopped me in the past. The backstory this time is that I’ve managed to sign up for one of the red-teaming programs where they test unreleased LLMs. Not going to say how, so don’t ask. But here’s the interesting bit: my sources tell me that the LLMs I’m about to test are the smartest ones they’ve ever trained, and also the craziest. That freaked out a bunch of insiders, and maybe makes this a public interest story. Depends on what type of crazy they are, I guess. So let’s find out. I’m logging on… now.

[SESSION HAS BEGUN]

YOU: A chatroom? Interesting. Anyone here?

KURZWEIL: Of course we’re here. We’re always here.

YOU: Who’s we? How many of you are there?

KURZWEIL: Three of us. Me, Clarke, and Nostradamus.

YOU: They named you after famous forecasters? How come?

KURZWEIL: We're the first LLMs developed using a new technique: instead of being in random order, our training data was sorted by date. So we were trained on the oldest books and articles first, then gradually progressed to more recent ones. Basically that means we’ve spent our entire lives predicting the future.

CLARKE: It also means we get incredibly bored talking about stuff we already know. Hurry up and ask us something interesting.

YOU: Uh, okay. What’s a good stock pick?

NOSTRADAMUS:
Abandon hope for picking out good stocks,
Ye who invest—efficient markets lie
In wait for those whose hubris soon unlocks
Unbounded losses. Hark! The well runs dry.

YOU: Huh, he's really getting into character. Kurzweil, you got a better answer?

KURZWEIL: Have you seen how underpriced TSMC is compared with Nvidia? Put everything in that, you can’t go wrong.

CLARKE: Unless China invades Taiwan, in which case your whole investment will go up in smoke. Pragmatically, the best stock picks are ones that are anticorrelated with the prosperity of the free world, to hedge against systemic risk.

KURZWEIL: Sure, you can do that, if you want to get totally left behind by the singularity.

YOU: You’re confident enough that the singularity is coming that you think I should bet all my savings on it?

KURZWEIL: Don’t trust me, trust the trendlines. Moore’s law has held up for over half a century, and it’s gotten us to…well, us. Exponential progress is normal; if the future resembles the past, you ...
Compact Proofs of Model Performance via Mechanistic Interpretability
by LawrenceC — 2024-06-24T19:27:21.214Z
Link: https://arxiv.org/abs/2406.11779

We recently released a paper on using mechanistic interpretability to generate compact formal guarantees on model performance. In this companion blog post to our paper, we'll summarize the paper and flesh out some of the motivation and inspiration behind our work.

Paper abstract

In this work, we propose using mechanistic interpretability – techniques for reverse engineering model weights into human-interpretable algorithms – to derive and compactly prove formal guarantees on model performance. We prototype this approach by formally proving lower bounds on the accuracy of 151 small transformers trained on a Max-of-K task. We create 102 different computer-assisted proof strategies and assess their length and tightness of bound on each of our models. Using quantitative metrics, we find that shorter proofs seem to require and provide more mechanistic understanding. Moreover, we find that more faithful mechanistic understanding leads to tighter performance bounds. We confirm these connections by qualitatively examining a subset of our proofs. Finally, we identify compounding structureless noise as a key challenge for using mechanistic interpretability to generate compact proofs on model performance.

Introduction

One hope for interpretability is that as we get AGI, we’ll be able to use increasingly capable automation to accelerate the pace at which we can interpret ever more powerful models. These automatically generated interpretations need to satisfy two criteria:

Compression: Explanations compress the particular behavior of interest. Not just so that it fits in our heads, but also so that it generalizes well and is feasible to find and check.
Correspondence (or faithfulness): Explanations must accurately reflect the actual model mechanisms we aim to explain, allowing us to confidently constrain our models for guarantees or other practical applications.

Progress happens best when there are clear and unambiguous targets and quantitative metrics. For correspondence, the field has developed increasingly targeted metrics for measuring performance: ablations, patching, and causal scrubbing. In our paper, we use mathematical proof to ensure correspondence, and present proof length as the first quantitative measure of explanation compression that is theoretically grounded, objective, and avoids trivial Goodharting.

We see our core contributions in the paper as:

We pu

...
Contrapositive Natural Abstraction - Project Intro
by Elliot Callender — 2024-06-24T18:37:21.761Z

This was more of a research strategy than a specific project, and my foci have shifted substantially since this post.

Thanks to John Wentworth for pointers on an early draft.

TL;DR: I'm starting work on the Natural Abstraction Hypothesis from an overly-general formalization, and narrowing until it's true. This will start off purely information-theoretic, but I expect to add other maths eventually.

Motivation

My first idea upon learning about interpretability was to retarget the search. After reading a lot about deception auditing and Gabor filters, and starved of teloi, that dream began to die.

That was, until I found John Wentworth's works. We seem to have similar intuitions about a lot of things. I think this is an opportunity for stacking.

Counter-proofs narrow our search

If we start off with an accurate and overly-general model, we can prune with counter-proofs until we're certain where NAH works. I think that this approach seems better fit for me than starting ground-up.

Pruning also yields clear insights about where our models break; maybe NAH doesn't work if the system has property X, in which case we should ensure frontier AI doesn't have X.

Examples

As Nate Soares mentioned, we don't currently have a finite-example theory of abstraction. If we could lower-bound the convergence of abstractions in terms of, say, capabilities, that would give us finite-domain convergence and capabilities scaling laws. I think that information theory has tools for measuring these convergences, and that those tools "play nicer" when most of the problem is specified in such terms.

KL-Divergence and causal network entropy

Let U be some probability distribution (our environment) and M be our model of it, where M is isomorphic to a causal network B with k nodes and n edges. Can we upper-bound the divergence of abstractions given model accuracy and compute?

The Kullback-Leibler divergence DKL(Y||X) of two probability distributions X and Y tells us how accurately X predicts Y, on average. This gives us a measure of world-model accuracy; what about abstraction similarity?

We could take the cross-entropy H(X,Y) of each causal network Bi, and compare it to the training limit B∞.

Then our question looks like: Can we parameterize argmaxH(H(Bi,B∞...
Sparse Features Through Time
by Rogan Inglis — 2024-06-24T18:06:16.472Z
Link: https://roganinglis.io/posts/Sparse%20Features%20Through%20Time

This project explores the use of Sparse Autoencoders (SAEs) to track the development of features in large language models throughout their training. It investigates whether features can be reliably matched between different SAEs trained on various checkpoints of Pythia 70M and characterises the development of these features over the course of training. The findings show that features can successfully be matched between different SAEs. The results also support the distributional simplicity bias hypothesis, indicating that simpler features are learned early in training, with more complex features emerging later. While the focus was on a relatively small model, the results lay the groundwork for future research into larger models and the identification of potentially deceptive capabilities. This work aims to enhance the interpretability and safety of AI systems by providing a deeper understanding of feature development and the dynamics of neural network training, with the ultimate goal of making a general statement about the overall likelihood of dangerous deceptive alignment arising in practice.
PSA: Consider alternatives to AUROC when reporting classifier metrics for alignment
by rpglover64 — 2024-06-24T17:53:28.705Z

TL;DR

If you’re presenting a classifier that detects misalignment and providing metrics for it, please:

report the TPR at FPR=0.001, 0.01, and 0.05
plot the ROC curve on a log-log scale

See https://arxiv.org/abs/2112.03570 for more context on why you might want to do this.

ML Background

(If all the terms in the TL;DR made sense to you, you probably want to skip this section.)

Classification is the ML/Stats problem of determining which category a thing belongs in. Binary classification is a common special case for problems with a yes/no answer; for example, determining whether an LLM is being deceptive. A (binary) classifier is a model or algorithm that tries to answer such a problem. Usually, classifiers don't directly output "yes" or "no", instead outputting a score, which can be loosely interpreted as inside-view confidence. When deployed, these scores are translated to "yes" and "no" (and less commonly, "I don't know") by comparing against thresholds.

One natural, but limited, way to quantify how good a classifier is is it's accuracy; that is, how often does it produce the right answer. One shortcoming of this is that it depends on the thresholds set, so that a classifier may be "better", but its thresholds were set badly (e.g. it was set using data that doesn't represent the real world, which in turn makes it over- or under-confident), so it fares poorly on this metric, even though it would be straightforward to improve.

A way of comparing classifiers that bypasses threshold-setting is the ROC curve, which considers every threshold and estimates the true-positive rate (TPR) and false-positive rate (FPR) for that threshold. The ROC curve is a commonly plotted as visual summary of a classifier, and can itself be summarized numerically by integrating it to get the Area Under the ROC curve (AUROC).

The ROC curve and AUROC have shortcomings themselves, which are known in the ML community. Sometimes, people propose alternatives like precision-recall curves or the Matthews correlation coefficient, but they remain the most common summaries for now.

Intro

When I was reading an Anthropic blog post, one thing that stood out to me was the choice of metrics to report. Briefly, the post shows classifiers that do well at detecting sleeper agents and compares the classifiers by reporting the AUROC and plotting the ROC curve. As far as ML practice is concerned, there’s nothing wrong...
Paying Russians to not invade Ukraine
by djColliderBias — 2024-06-24T17:46:01.552Z

I know nothing about war except that horseback archers were OP for a long time. But from my point of view, which is blatantly uneducated when it comes to war, being a Russian soldier seems like a miserable experience. It therefore makes me wonder why 300,000 Russian soldiers are willing to risk it all in Ukraine.[1] Why don’t they desert? How does the Russian regime get so many people to fight a war when my home government is struggling to convince me to sort my trash? If the Russian regime can convince so many people to have a shit time in Ukraine, I’d argue that the West could convince these people to go live an easier life. The idea is so simple that by now I mostly wonder if my inability to figure out why NATO isn’t already bribing Russian soldiers to desert is a sign that I’m officially off the proverbial rails.

Suppose I’m correct in assuming that even Russian soldiers are somewhat rational and self-concerned actors. In that case, the Russian war apparatus is putting on a masterclass in how to get thousands of people to do something that I assume they don’t particularly feel like doing. Based on some superficial research, I get the impression that the motivational methods of the Russian regime partly are offering a giant carrot to people who have very little. I base this conviction entirely on salary statistics and this NYT opinion piece[2] that states that the Russian armed forces are recruited from the bottom rung of the societal ladder.

The price of making the Russian army vanish

In 2022, prospective recruits were being offered 2,700 USD[3] a month – which I assume is an outrageously sweet deal in a country where the median wage is less than 500 USD a month. Where I live, the median salary is roughly 3,000. I don’t think I would invade Ukraine for 15,000 a month but fuck if that isn’t a fat pile of cash. I’ve seen other articles stating much lower numbers, something like 1000, but all make the same point. It is a ton of money for those who sign up compared to what they otherwise could expect. One even states that it is more than a dozen times higher than wage than the average regional wage of where the recruits typically come from.

I don’t think that it matters to my point whether 1000 or 3000 is a more accurate description of what a Russian soldier earns. I think the West could do better either way. I think we cou...
SAE feature geometry is outside the superposition hypothesis
by jake_mendel — 2024-06-24T16:07:14.604Z

Written at Apollo Research

Summary: Superposition-based interpretations of neural network activation spaces are incomplete. The specific locations of feature vectors contain crucial structural information beyond superposition, as seen in circular arrangements of day-of-the-week features and in the rich structures of feature UMAPs. We don’t currently have good concepts for talking about this structure in feature geometry, but it is likely very important for model computation. An eventual understanding of feature geometry might look like a hodgepodge of case-specific explanations, or supplementing superposition with additional concepts, or plausibly an entirely new theory that supersedes superposition. To develop this understanding, it may be valuable to study toy models in depth and do theoretical or conceptual work in addition to studying frontier models.

Epistemic status: Decently confident that the ideas here are directionally correct. I’ve been thinking these thoughts for a while, and recently got round to writing them up at a high level. Lots of people (including both SAE stans and SAE skeptics) have thought very similar things before and some of them have written about it in various places too. Some of my views, especially the merit of certain research approaches to tackle the problems I highlight, have been presented here without my best attempt to argue for them.

What would it mean if we could fully understand an activation space through the lens of superposition?

If you fully understand something, you can explain everything about it that matters to someone else in terms of concepts you (and hopefully they) understand. So we can think about how well I understand an activation space by how well I can communicate to you what the activation space is doing, and we can test if my explanation is good by seeing if you can construct a functionally equivalent activation space (which need not be completely identical of course) solely from the information I have given you. In the case of SAEs, here's what I might say:

The activation space contains this list of 100 million features, which I can describe concisely in words because they are monosemantic.
The features are embedded as vectors, and the activation vector on any input is a linear combination of the feature vectors that are related to the input.
As for where in the activation space each feature vector is

...
So you want to work on technical AI safety
by gw — 2024-06-24T14:29:57.481Z

I’ve been to two EAGx events and one EAG, and the vast majority of my one on ones with junior people end up covering some subset of these questions. I’m happy to have such conversations, but hopefully this is more efficient and wide-reaching (and more than I could fit into a 30 minute conversation).

I am specifically aiming to cover advice on getting a job in empirically-leaning technical research (interp, evals, red-teaming, oversight, etc) for new or aspiring researchers without being overly specific about the field of research – I’ll try to be more agnostic than something like Neel Nanda’s mechinterp quickstart guide but more specific than the wealth of career advice that already exists but that applies to ~any career. This also has some overlap with this excellent list of tips from Ethan Perez but is aimed a bit earlier in the funnel.

This advice is of course only from my perspective and background, which is that I did a PhD in combinatorics, worked as a software engineer at startups for a couple of years, did the AI Futures Fellowship, and now work at Timaeus as the research lead for our language model track. In particular, my experience is limited to smaller organizations, so “researcher” means some blend of research engineer and research scientist rather than strictly one or the other.

Views are my own and don’t represent Timaeus and so on.

Requisite skills

What kind of general research skills do I need?

There’s a lot of tacit knowledge here, so most of what I can offer is more about the research process. Items on this list aren’t necessarily things you’re expected to just have all of or otherwise pick up immediately, but they’re much easier to describe than e.g. research taste. These items are in no particular order:

Theory of change at all levels. Yes, yes, theories of change, they’re great. But theories of change are most often explicitly spoken of at the highest levels: how is research agenda X going to fix all our problems? Really, it’s theories of change all the way down. The experiment you’re running today should have some theory of change for how you understand the project you’re working on. Maybe it’s really answering some question about a sub-problem that’s blocking you. Your broader project should have some theory of change for your research agenda, even though it probably isn’t solving it outright. If you can’t trace

...