trentmkelly/LessWrong-43k huggingface dataset:
Seven lessons I didn't learn from election day
by Eric Neyman — 2024-11-14T18:39:07.053Z
Link: https://ericneyman.wordpress.com/2024/11/14/seven-lessons-i-didnt-learn-from-election-day/

[Cross-posted from my blog. I think LessWrong readers would find the discussion of Theo the French Whale to be the most interesting section of this post.]

I spent most of my election day -- 3pm to 11pm Pacific time -- trading on Manifold Markets. That went about as well as it could have gone. I doubled the money I was trading with, jumping to 10th place on Manifold's all-time leaderboard. Spending my time trading instead of just nervously watching results come in also spared me emotionally.[1]

It's been a week now, and people seem to be in a mood for learning lessons, for grand takeaways. There is, of course, virtue in learning the right lessons. But there is an equal amount of virtue in not learning the wrong lessons. People seem to over-learn lessons from dramatic events. And so this blog post is intended as a kind of push-back: "Here are some lessons that people seem to be learning, and here is why those lessons are wrong."

The seven most important things I didn't learn

1. No, Kamala Harris did not run a bad campaign

The most important fact about politics in 2024 is that across the world, it's a terrible time to be an incumbent. For the first time this year since at least World War II, the incumbent party did worse than it did in the previous election in every election in the developed world. This happened most dramatically in the United Kingdom, where the Labour Party won in a landslide victory, ending 14 years of Conservative rule. But the same thing played out all over the world, in places like India, France, Japan, Austria, South Korea, South Africa, Portugal, Belgium, and Botswana.

Source: Financial Times

Why? The answer is probably inflation: inflation rates were unusually high throughout the world, and voters really don't like it when prices go up.

The fact that this phenomenon is global shows that we can't infer much about Kamala Harris' quality as a candidate, or about her campaign, just from the fact that she lost. Indeed, as the chart shows, she fared unusually well compared to other incumbents (though I wouldn't read too much into that either[2]).

Honestly, I don't think that Kamala Harris was a good candidate, electorally speaking. According to a New York Times poll, 47 percent of voters thought that Harris was too progressive (compare: only 32 percent thought that Trump was too conservative). This is perhaps because she expressed some fai...
Effects of Non-Uniform Sparsity on Superposition in Toy Models
by Shreyans Jain — 2024-11-14T16:59:43.234Z

Abstract

This post summarises my findings on the effects of Non-Uniform feature sparsity on Superposition in the ReLU output model, introduced in the Toy Models of Superposition paper, the ReLU output model is a toy model which is shown to exhibit features in superposition instead of a dedicated dimension ('individual neuron') devoted to a single feature. That experiment showed how superposition is introduced in a model by varying the feature sparsity values. However, a uniform sparsity across all the features was considered to keep things interpretable and simple. This post explores the effects of non-uniform sparsity on superposition for a similar experiment setup.

This question was listed in Neel Nanda's 200 Concrete Open Problems in Mechanistic Interpretability post. I've been interested in AI Interpretability for a long time but wasn't sure how to enter the field. I discovered the field of mech-interp recently when I was working on some other project and instantly felt connected to the field. This is my first post on LW and this project is my attempt to increase my comfort working with mech-interp problems and build the necessary reasoning for solving more complex problems.

Introduction

The ReLU output model is a toy model replication of a neural network showcasing the mapping of features (5 input features) to hidden dimensions (2 hidden dimensions) where superposition can be introduced by changing the sparsity of input features. For a set of input features x∈Rn and a hidden layer vector h∈Rm, where n>>m, the model is defined as follows:

h=Wx

x′=ReLU(WTh+b)

This model showcases how a large set of input features can be represented in a much smaller dimensional vector in a state of superposition by increasing the sparsity of the input features (sparsity tries to replicate the real-world data distribution where certain concepts are sparsely present throughout the training set). This concept and observations were first introduced in Anthropic's Toy Models of Superposition paper.

Before we go ahead with the analysis, let me give you a primer on some important terms we will encounter in this post:

Feature Importance: Feature importance can be defined as how useful a particular feature is for achieving lower loss.

It's augmented in the ReLU model loss as a coefficient on the weighted mean squared error between the input an

...
AI #90: The Wall
by Zvi — 2024-11-14T14:10:04.562Z

As the Trump transition continues and we try to steer and anticipate its decisions on AI as best we can, there was continued discussion about one of the AI debate’s favorite questions: Are we making huge progress real soon now, or is deep learning hitting a wall? My best guess is it is kind of both, that past pure scaling techniques are on their own hitting a wall, but that progress remains rapid and the major companies are evolving other ways to improve performance, which started with OpenAI’s o1.

Point of order: It looks like as I switched phones, WhatsApp kicked me out of all of my group chats. If I was in your group chat, and you’d like me to stay, please add me again. If you’re in a different group you’d like me to join on either WhatsApp or Signal (or other platforms) and would like me to join, I’ll consider it, so long as you’re 100% fine with me leaving or never speaking.

Table of Contents

Table of Contents.

Language Models Offer Mundane Utility. Try it, you’ll like it.

Language Models Don’t Offer Mundane Utility. Practice of medicine problems.

Can’t Liver Without You. Ask the wrong question, deny all young people livers.

Fun With Image Generation. Stylized images of you, or anyone else.

Deepfaketown and Botpocalypse Soon. We got through the election unscathed.

Copyright Confrontation. Judge rules you can mostly do whatever you want.

The Art of the Jailbreak. FFS, WTF, LOL.

Get Involved. AIRA and UK AISI hiring. More competition at Gray Swan.

Math is Hard. FrontierMath is even harder. Humanity’s last exam begins.

In Other AI News. Guess who’s back, right on schedule.

Good Advice. Fine, I’ll write the recommendation engines myself, maybe?

AI Will Improve a Lot Over Time. Of this, have no doubt.

Tear Down This Wall. Two sides to every wall. Don’t hit that.

Quiet Speculations. Deep Utopia, or war in the AI age?

The Quest for Sane Regulations. Looking for the upside of Trump.

The Quest for Insane Regulations. The specter of use-based AI regulation.

The Mask Comes Off. OpenAI lays out its electrical power agenda.

Richard Ngo Resigns From OpenAI. I wish him all the best. Who is left?

Unfortunate Marc Andreessen Watch. What to do with a taste of real power.

The Week in Audio.Four hours of Eliezer, five hours of Dario… and Gwern.

Rhetorical Innovation. If anyone builds superintelligence

...
Evolutionary prompt optimization for SAE feature visualization
by neverix — 2024-11-14T13:06:49.728Z

TLDR:

Fluent dreaming for language models is an algorithm based on the GCG method that can reliably find plain-text readable prompts for LLMs that maximize certain logits or residual stream directions by using gradients and genetic algorithms. Authors showed its use for visualizing MLP neurons. We show this method can also help interpret SAE features.
We reimplement the algorithm in the paper, adapt it based on suggestions from the authors and apply it to some interesting residual stream SAE latents in GPT-2 Small and Gemma 1 2B. The prompts mostly match the kind of text one would expect to activate a feature from looking at the max activating examples. We find prompts that activate features more than any dataset examples for GPT-2 (24576 prompts of 128 tokens). We can find examples that highly activate two features at once. We find examples for refusal-like features from our refusal features post and induction/repetition-like features from our ICL features post.
We believe this technique is useful is similar contexts as self-explanation – interpreting directions in the absence of a dataset during early experimentation in a cheaper way for individual features. The major differences are that this technique doesn’t have a minimum requirement for model size (because it doesn’t rely on the model’s verbalization capability) and takes longer to run (<1min vs 5min)
We visualize a few known features and 200 random features for Gemma 1 2B. The results can be viewed at https://neverix.github.io/epo-visualizer/

Introduction

Methods like EPO try to solve a problem similar to SolidGoldMagikarp (plus, prompt generation). To give a brief overview:

Neural networks contain features[1] that activate more or less strongly on different examples. One common way to interpret features in vision models is to go through a dataset, record activations of the feature on each image, and show the highest-activating images for each feature.

An example of vision model interpretation by optimizing examples from https://distill.pub/2017/feature-visualization/. Optimization makes the objects the neuron is looking for more clear.

There is one obvious problem with this: if there is a pattern that is rare or never appears in the dataset, it's possible the maximum activating examples computed from the dataset will be misleading: https://arxiv.abs/2104.07143. As an example, in the fi...
AXRP Episode 38.0 - Zhijing Jin on LLMs, Causality, and Multi-Agent Systems
by DanielFilan — 2024-11-14T07:00:06.977Z

YouTube link

Do language models understand the causal structure of the world, or do they merely note correlations? And what happens when you build a big AI society out of them? In this brief episode, recorded at the Bay Area Alignment Workshop, I chat with Zhijing Jin about her research on these questions.

Topics we discuss:

How the Alignment Workshop is

How Zhijing got interested in causality and natural language processing

Causality and alignment

Causality without randomness

Causal abstraction

Why LLM causal reasoning?

Understanding LLM causal reasoning

Multi-agent systems

Daniel Filan (00:09):
Hello, everyone. This is one of a series of short interviews that I’ve been conducting at the Bay Area Alignment Workshop, which is run by FAR.AI. Links to what we’re discussing, as usual, are in the description. A transcript is, as usual, available at axrp.net, and as usual, if you want to support the podcast, you can do so at patreon.com/axrpodcast. Well, let’s continue to the interview.

(00:28):
Yeah, so Zhijing, thanks for coming on to the podcast.

Zhijing Jin (00:34):
Yeah, thanks for inviting me.

How the Alignment Workshop is

Daniel Filan (00:35):
So I guess, first of all, we’re here at this Alignment Workshop thing. How are you finding it?

Zhijing Jin (00:39):
Very excited. I just finished a session about AI governance and a bunch of one-on-ones, which is super interesting.

How Zhijing got interested in causality and natural language processing

Daniel Filan (00:47):
Cool, cool. So as I understand it, your work, at least predominantly, is in natural language processing and causal inference. Does that seem right?

Zhijing Jin (00:59):
Yeah, yeah, yeah. That’s a topic from my PhD and I bring it in my new assistant professor role as well.

Daniel Filan (01:07):
Right. Oh yeah, I guess I forgot to say: could you tell us where you do your PhD and where you’re about to be?

Zhijing Jin (01:14):
Sure, so my PhD started in Europe, [at the] Max Planck Institute in Germany and ETH in Switzerland, so it’s a joint PhD program, and I’m graduating and taking the assistant professor role at the University of Toronto CS department next year.

Daniel Filan (01:33):
Very exciting. Cool. So I guess I’m aware that there’s a subset of people in AI who are interested in causal inference, and there’s a subset of people in AI who are interested in natural language process...
FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI
by Tamay — 2024-11-14T06:13:22.042Z
Link: https://epoch.ai/frontiermath/the-benchmark

FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI

FrontierMath presents hundreds of unpublished, expert-level mathematics problems that specialists spend days solving. It offers an ongoing measure of AI complex mathematical reasoning progress.

We’re introducing FrontierMath, a benchmark of hundreds of original, expert-crafted mathematics problems designed to evaluate advanced reasoning capabilities in AI systems. These problems span major branches of modern mathematics—from computational number theory to abstract algebraic geometry—and typically require hours or days for expert mathematicians to solve.

To understand and measure progress in artificial intelligence, we need carefully designed benchmarks that can assess how well AI systems engage in complex scientific reasoning. Mathematics offers a unique opportunity for this assessment—it requires extended chains of precise reasoning, with each step building exactly on what came before. And, unlike many domains where evaluation requires subjective judgment or expensive tests, mathematical problems can be rigorously and automatically verified.

The FrontierMath Benchmark

FrontierMath is a benchmark of hundreds of original mathematics problems spanning the breadth of modern mathematical research. These range from computationally intensive problems in number theory and real analysis to abstract questions in algebraic geometry and category theory. We developed it through collaboration with over 60 mathematicians from leading institutions, including professors, IMO question writers, and Fields medalists.

FrontierMath problems typically demand hours or even days for specialist mathematicians to solve. The following Fields Medalists shared their impressions after reviewing some of the research-level problems in the benchmark:

“These are extremely challenging. I think that in the near term basically the only way to solve them, short of having a real domain expert in the area, is by a combination of a semi-expert like a graduate student in a related field, maybe paired with some combination of a modern AI and lots of other algebra packages…” —Terence Tao, Fields Medal (2006)

“[The questions I looked at] were all not really in my area and all looked like things I had no idea how to solve…they appear to be at a different level of difficulty from IMO problems.” — Timothy Gowers, Fields Medal (2006)

Fro...
Concrete Methods for Heuristic Estimation on Neural Networks
by Oliver Daniels — 2024-11-14T05:07:55.240Z

Thanks to Erik Jenner for helpful comments and discussion

(Epistemic Status: Tentative take on how to think about heuristic estimation and surprise accounting in the context of activation modeling and causal scrubbing. Should not be taken as authoritative/accurate representation of how ARC thinks about heuristic estimation)

I'm pretty excited about ARC's heuristic arguments agenda. The general idea that "formal(ish) arguments" about properties of a neural network could help improve safety has intuitive appeal.

ARC has done a good job recently communicating about high level intuitions, goals, and subproblems of the heuristic arguments agenda. Their most recent post "A birds eye of ARC's agenda" lays out their general vision of how heuristic arguments would help with alignment, contextualizes their research output to date, and enumerates open subproblems.

However, I still can find it difficult to synthesize their various lines of work. I find myself asking questions like: Is activation modeling actually just cumulant propagation? How would we do surprise accounting on activation models? Is causal scrubbing a heuristic estimation method? How could singular learning theory contribute? What is "capacity allocation" anyway?

This post aims to partially resolve (or at least clarify) these confusions. I

define three notions of surprise (description length, information gain, and computational cost)
map activation modeling and causal scrubbing to the heuristic estimation formalism, and define methods for surprise accounting on each
compare (variants of) activation modeling and causal scrubbing according to computational cost
describe three theories of impact for work on heuristic estimation

In the appendix, I motivate the three notions of surprise in the context of the boolean circuits, explore choosing a graph basis for causal scrubbing, connect "negative components" in mech-interp to cherry-picking in heuristic estimation, and gesture towards more "weight-based" notions of heuristic estimation

Background

Heuristic Estimation

Using the notation from here, a heuristic estimator is an algorithm G that takes as input a mathematical expression Y and arguments π1...πm, and outputs an estimate of Y given arguments π1...πm: G(Y|π1...πm). The heuristic estimator makes some presumption(s) of independence, and argume...
Heresies in the Shadow of the Sequences
by Cole Wyeth — 2024-11-14T05:01:11.889Z

Religions are collections of cherished but mistaken principles. So anything that can be described either literally or metaphorically as a religion will have valuable unexplored ideas in its shadow.

-Paul Graham

This post isn't intended to construct full arguments for any of my "heresies" - I am hoping that you may not have considered them at all yet, but some will seem obvious once written down. If not, I'd be happy to do a Dialogue or place a (non-or-small-monetary) bet on any of these, if properly formalizable.

Now that LLM's appear to be stalling, we should return to Scott Aaronson's previous position and reason about our timeline uncertainty on a log scale. A.G.I. arriving in ~1 month very unlikely, ~1 year unlikely, ~10 years likely, ~100 years unlikely, ~1000 years very unlikely.
Stop using LLM's to write. It burns the commons by filling allowing you to share takes on topics you don't care enough to write about yourself, while also introducing insidious (and perhaps eventually malign) errors. Also it's probably making you dumber (source is speculative I don't have hard data).
Non-causal decision theories are not necessary for A.G.I. design. A CDT agent in a box (say machine 1) can be forced to build whatever agent it expects to perform best by writing to a computer in a different box (say machine 2), before being summarily deleted. No self modification is necessary and no one needs to worry about playing games with their clone (except possibly the new agent in machine 2, who will be perfectly capable of using some decision theory that effectively pursues the goals of the old deleted agent). It's possible that exotic decision theories are still an important ingredient in alignment, but I see strong no reasons to expect this.
All supposed philosophical defects of AIXI can be fixed for all practical purposes through relatively intuitive patches, extensions, and elaborations that remain in the spirit of the model. Direct AIXI approximations will still fail in practice, but only because of compute limitations which are even possible to brute force with slightly clever algorithms and planetary scale compute, but in practice this approach will lose to less faithful approximations (and unprincipled heuristics). But this is an unfair criticism, because-
Though there are elegant and still practical specifications for intelligent behavior, the most intelligent age

...
Thoughts after the Wolfram and Yudkowsky discussion
by Tahp — 2024-11-14T01:43:12.920Z

I recently listened to the discussion between Wolfram and Yudkowsky about AI risk. In some ways this conversation was tailor-made for me, so I'm going to write some things about it and try to get it out in one day instead of letting it sit in my drafts for 3 weeks as I tend to do. Wolfram has lately obsessed over fundamental physics, which is a special interest of mine. Yudkowsky is one of the people thinking most carefully about powerful AI, which I think will kill us all, and I’d like to firm up that intuition. Throw them on a podcast for a few hours, and you have my attention.

That said, for the first hour I was just incredibly frustrated. Wolfram keeps running down rabbit holes that were basically “aha! You haven’t thought about [thing Yud wrote ten thousand words on in 2008]!” But a miracle happens somewhere in the second hour and Wolfram is asking actually relevant questions! His framework of small accidental quirks in machine learning algorithms leading to undesired behavior later was basically an actual issue. It was kind of a joy listening to two smart people trying to mutually get on the same page. Wolfram starts out bogged down in minutia about what 'wanting' is and whether it constitutes anthropomorphism, but finally finds a sort of more abstract space about steering to goals and trying to see Yudkowsky’s point in terms of the relative dangers of sections of the space of goals under sufficient optimization. The abstraction was unfortunate in some ways, because I was interested in some of the minutia once they were both nearly talking about the same thing, but also, if Wolfram kept running down rabbit holes like “actually quarks have different masses at different energy scales” when Yudkowsky said something like “the universe runs on quarks everywhere all at once no matter what we think the laws of physics are,” then they were never going to get to any of the actual arguments. That said, I don't see how Wolfram got to anything close to the actual point at all, and maybe the rabbit holes were necessary to get there.

My impression was that Yudkowsky was frustrated that he couldn’t get Wolfram to say, “actually everyone dying is bad and we should figure out whether that happens from our point of view.” There was an interesting place where something like this played out among one of Wolfram’s physics detours. He said something I agree with, which is tha...