trentmkelly/LessWrong-43k huggingface dataset:
XX by Rian Hughes: Pretentious Bullshit
by Yair Halberstadt — 2025-01-08T13:02:52.438Z

SPOILER WARNING: Extensive spoilers for XX.

XX was recommended by one of the comments on my Sci-Fi micro-reviews, so I read it. I can't seem to find any critical reviews of the book online so I'm going to while away a few hours of my life in a misguided attempt to reveal that the emperor has no clothes.

Typography

First let's dispense with XX's most distinguishing feature, then we'll get down to brass tacks. Rian Hughes plays around a huge amount with typography and tries to push the edges of text as a medium. This is an interesting quirk, and is sometimes appreciated, but sometimes makes the novel actively hostile to the reader. Reading through twenty pages of this quickly gets tiring:

Overall, I don't think it added that much to the story, but I guess some people would like it 🤷.

Length And Pacing

This book is long (978 pages), and slow. It's not without its good bits, but to be blunt, it's a slog. If I appreciate the ideas in a book though, I'm happy to recommend even longer and slower works. So lets move onto:

Ideas

I generally prefer hard sci-fi. Sci-fi that might make up a number of novel technologies, but then applies them consistently and rigorously[1]. Sci-fi that explores ideas in physics,. economic, sociology, philosophy, or whatever.

That said I'm perfectly happy to read science-fantasy. If you want to tell a story set in space, and don't want to worry too much about the annoying physics, then so long as the plot is good I'm all for it.

XX is neither. It tries its hardest to be about high-minded concepts, alternative physics, and philosophy, but every single one of its ideas is completely BS. Let's get started:

The Oxbow:

After a long monologue about how consciousness requires localisation, the protagonist finishes off with:

"We pinch off a piece of the Web, of ideaspace, through a looped buffer - one with a very specific permeability that I've, uh, I've not completely resolved as yet. Connected, so there is a way in, but shielded, cut off from most of the noise of the rest of the internet. This will give any signal in there that may be trying to localise itself a means to cohere. If you're diffused throughout a system like the internet, a system with a billion cameras for eyes and a billion keyboards constantly firing up your nerve signals, you're not going to know where, if anywhere, you are. The net is interconnected; you are the net. There is no not-you.

...
Activation space interpretability may be doomed
by bilalchughtai — 2025-01-08T12:49:38.421Z

TL;DR: There may be a fundamental problem with interpretability work that attempts to understand neural networks by decomposing their individual activation spaces in isolation: It seems likely to find features of the activations - features that help explain the statistical structure of activation spaces, rather than features of the model - the features the model’s own computations make use of.

Written at Apollo Research

Introduction

Claim: Activation space interpretability is likely to give us features of the activations, not features of the model, and this is a problem.

Let’s walk through this claim.

What do we mean by activation space interpretability? Interpretability work that attempts to understand neural networks by explaining the inputs and outputs of their layers in isolation. In this post, we focus in particular on the problem of decomposing activations, via techniques such as

sparse

autoencoders

(SAEs),

PCA

, or just by looking at individual

neurons

. This is in contrast to interpretability work that leverages the wider functional structure of the model and incorporates more information about how the model performs computation. Examples of existing techniques using such information include

Transcoders

,

end2end-SAEs

and joint activation/gradient PCAs.

What do we mean by “features of the activations”? Sets of features that help explain or make manifest the statistical structure of the model’s activations at particular layers. One way to try to operationalise this is to ask for decompositions of model activations at each layer that try to

minimise the description length of the activations in bits

.

What do we mean by “features of the model”? The set of features the model itself actually thinks in, the decomposition of activations along which its own computations are structured, features that are significant to what the model is doing and how it is doing it. One way to try to operationalise this is to ask for the decomposition of model activations that makes the causal graph of the whole model as manifestly simple as possible: We make each feature a graph node, and draw edges indicating how upstream nodes are involved in computing downstream nodes. To understand the model, we want the decomposition that results in the most structured graph with the fewest edges, with meaningful...
AI Safety as a YC Startup
by Lukas Petersson — 2025-01-08T10:46:29.042Z

A while back I gave a talk about doing AI safety as a YC startup. I wrote a blog post about it and thought it would be interesting to share it with both the YC and AI safety communities. Please share any feedback or thoughts. I would love to hear them!

AI Safety is a problem and people pay to solve problems

Intelligence is dangerous, and I think there's a significant chance that the default scenario of AI progress poses an existential risk to humanity. While it's far from certain, even small probabilities are significant when the stakes are this high. This is an enormous abstract problem, but there are thousands of sub-problems waiting to be solved.

Some of these sub-problems already exist today, but most are in the future (GPT is not capable of killing us yet). When people start feeling these pains, they will pay to have them solved (similar to how anti-virus software became a huge business opportunity once personal computers became widespread in the 1980s). I don't think the solution is to slow down (although I'm not certain), because this also comes with a cost. Therefore, we have to solve these problems. I think it's one of the most interesting challenges of our time, because otherwise we won't get to reap the rewards of AI utopia.

More startups should solve these problems

From my experience, builders (entrepreneurial engineers) are underrepresented in the AI safety community. There are far more researchers and philosophers. They are also crucial, but the mix is not currently balanced (source: personal experience from trying to hire such people). I don't think this should be the case. The AI safety market is currently very small, but according to those who have attempted to forecast its trajectory, it may grow substantially. Since most people remain skeptical of near-term AGI timelines, betting on this growth early could provide a competitive advantage. VCs exist to enable startups to make such long-term bets.

If you're a technical person with a passion for AI safety, it's very tempting to join a big AI lab. However, I think you should start a startup instead. Startups are more fun, and you will have much more counterfactual impact. A friend once told me: “The most impactful people will build something great that wouldn't have happened without them”. I think it's generally harder to do this in the hierarchical structure of AI labs (but not impossible). Mor...
The absolute basics of representation theory of finite groups
by Dmitry Vaintrob — 2025-01-08T09:47:13.136Z

This will be an "ML-oriented" introduction to representation theory of finite groups. It is an introductory sketch that orients towards both a language and a result (the "real semisimplicity" result) that is useful for thinking about the subject in an interpretability context.

This is a somewhat low-effort math post (subbing in for a planned post I'm running late on). It isn't a prerequisite for anything, but will be useful background for a future discussion of how NN's learn associative group multiplication (a joint project with Louis Jaburi). I'm hoping for this to be readable by people with a very small amount of background in abstract algebra: it is meant to fill in some very basic knowledge gaps. I'd be very interested in comments about what is confusing or difficult about this post, to help focus on the right exposition directions when writing about this in the future.

I'm not going to assume much abstract algebra, other than the notions of a group and an algebra (always assumed to be a unital, associative algebra).

Algebras of note

Recall that all the set Matn×n of n×n (real-valued) matrices forms an algebra (it's a vector space with an associative product that distributes across addition). For any field F, there is a notion of a matrix algebra over F and Matn×n is naturally an algebra over R. The set Matn×n(C) of complex-valued matrices forms an algebra over C, but it can also be viewed as an algebra over R, in the same way as C can be viewed as a two-dimensional vector space.

We will be interested in understanding the "algebra" associated to a finite group G. It turns out that working with algebras over C is easier than working with algebras over R from a theoretical viewpoint (this is of course because C is algebraically closed, i.e., any polynomial equation has a solution. We'll start out looking at such representations. However unfortunately, neural nets don't care about theoretical ease: instead they care about compression and efficiency, and this will necessitate also understanding a bit of representation theory over R.

For starters, we work over C. We'll need one more useful algebra that will be one of the main characters in our discussion: namely, the group algebra of G, written C[G]. This is the algebra whose elements are formal complex linear combinations of elements of G, and where product is induced from the product on G distributively. W...
Implications of the AI Security Gap
by Dan Braun — 2025-01-08T08:31:36.789Z

This post reflects my personal opinion and not necessarily that of other members of Apollo Research or any of the people acknowledged below. Thanks to Jarrah Bloomfield, Lucius Bushnaq, Marius Hobbhahn, Axel Højmark, and Stefan Heimersheim for comments/discussions.

I find that people in the AI/AI safety community have not considered many of the important implications that security in AI companies has on catastrophic risks.

In this post, I’ve laid out some of these implications:

AI companies are a long way from state-proof security

Implementing state-proof security will slow down safety (and capabilities) research a lot

Sabotage is sufficient for catastrophe

What will happen if timelines are short?

Security level matters, even if you’re not robust to top cyber operations

AI companies are a long way from state-proof security

I’m of course not the first one to make this claim (e.g. see

Aschenbrenner

). But it bears repeating.

Last year, RAND released an excellent

report

outlining what it might take for an AI company to prevent adversaries of various capability levels from stealing model weights. Its highest security level, SL5, defines a level of security that could “thwart most top-priority operations by the top cyber-capable institutions”. This is, of course, the level of security that we need around a technology with catastrophic risks.

Unfortunately, today’s AI companies are likely to be further away from reaching SL5 than the time it is likely to take to develop models that are worth protecting at this level. Below, I illustrate some reasons why implementing this level of security might take a very long time.

Reaching SL5 is a tough technical challenge. Some components that RAND suggests for SL5 do not even exist yet “More R&D is needed to enable organizations to support production models while meeting SL5 security requirements. We recommend the development of hardware security module (HSM)-like devices with an interface that is specialized for ML applications.”

Let’s take preventing supply-chain attacks from software as another example. The RAND report lists this security measure for SL5: “Strong limitations on software providers (e.g., only developed internally or by an extremely reliable source)”. My guess is that if you looked at all software being used by researchers at a frontier lab, you’d have a list longer than this article, many of which would be...
What are polysemantic neurons?
by Vishakha — 2025-01-08T07:35:42.758Z
Link: https://aisafety.info/questions/8KGQ/What-are-polysemantic-neurons

This is an article in the featured articles series from AISafety.info. AISafety.info writes AI safety intro content. We'd appreciate any feedback.

The most up-to-date version of this article is on our website, along with 300+ other articles on AI existential safety.

For a “monosemantic” neuron, there’s a single feature that determines whether it activates strongly. If a neuron responds only to grandmothers, we might call it a

grandmother neuron

. For a “polysemantic” neuron, in contrast, there are multiple features that can cause it to activate strongly.

As an example,

this image

shows

feature visualizations

of a neuron that activates when it sees any of a cat face, cat legs, or the front of a car. As far as anyone can tell, the reason why this neuron responds to both cats and cars is not that cars and cats share some underlying feature. Rather, the neuron just happened to get two unrelated features attached to it.

Why do we think that the neurons are not encoding a shared similarity?

Suppose a neuron is picking out some feature shared by cars and cats. Say, the neuron is representing “sleekness”. Then we’d expect images of other “sleek” things, like a snake or a ferret, to activate the neuron. So if we generate lots of different images which highly activate our neuron, and find that they do contain snakes and ferrets, that’s evidence for the neuron picking up on a unified concept of sleekness. Researchers have run

experiments like this

on neurons like this one and found that, no, they only activate on cats and cars — just as the “polysemantic” hypothesis would lead us to expect.

Why do polysemantic neurons form?

Polysemantic neurons seem to result from a phenomenon known as “

superposition

”. "Superposition", in this case, means that the neural net takes a larger number of numerical "features" of its input that might each otherwise have had their own neuron (for example, one neuron for cat-ness and one neuron for car-ness), and instead spreads those features out over a smaller number of neurons, with the features mixed together ("loaded") in different ways for each neuron.

In fact, if we only care about packing as many features into n neurons as we can, then using polysemantic neurons lets us pack roughly as many as exp(C * n) features, where C is a constant depending on how much overlap between concepts we allow.[1] In c...
Tips On Empirical Research Slides
by James Chua — 2025-01-08T05:06:44.942Z

Our research is centered on empirical research with LLMs. So if you are doing something similar, these tips on slide-based communication may be helpful!
Background:

James Chua and John Hughes are researchers working under Owain Evans and Ethan Perez, respectively. Both of us (James and John) used to be MATS mentees. We weren't good at making research slides at first -- here are some principles we've found useful for understandable slides for our weekly research meetings.

We show some good example slides. We also show examples of confusing slides we've made — marked in the caption with “❌ Negative example”.
Below we use the study of sycophancy as an example. Sycophancy occurs when a model has responses that match user beliefs rather than truthful ones.

Summary slide sets the frame

Possible summary slide. I summarise my experimental results for the week and write what I want to be discussed. If possible, I fit in a simple plot on the right.

Your mentor manages multiple projects and people. They need to be reminded of what you are up to. The first slide should recap the key takeaways from the last meeting to motivate what you have worked on and provide a clear summary of your progress.

There are two main messages to convey, which set the frame for what your mentor should think about:

Key takeaways from the last meeting. A reminder of the next steps and what was discussed.
Experiment outcome

My experiments worked! Your mentor will focus on sanity checks, control experiments, and extensions to other setups.
My experiments didn't work. Your mentor will focus on debugging why. E.g., train with better data, or improve the prompts.

A summary helps your mentor save time. For example, in the slide, your mentor may say “Oh I can remember what the data augmentation was, let’s skip that”. Or maybe your mentor already read your results and wants to discuss something else.

Include an agenda

Often in meetings with your mentor, there is very little time to cover everything the team has done in the week.

Include the main sections of what you will present in order of priority. This lets you cover the most important things first.
Highlight how many slides there are on that topic and how long you'd like to allocate for that section. This allows your mentor to calibrate themselves on whether it is ok for them to drill down into the details if there is stil

...
On Eating the Sun
by jessicata — 2025-01-08T04:57:20.457Z
Link: https://unstablerontology.substack.com/p/on-eating-the-sun

The Sun is the most nutritious thing that's reasonably close. It's only 8 light-minutes away, yet contains the vast majority of mass within 4 light-years of the Earth. The next-nearest star, Proxima Centauri, is about 4.25 light-years away.

By "nutritious", I mean it has a lot of what's needed for making computers: mass-energy. In "Ultimate physical limits to computation", Seth Lloyd imagines an "ultimate laptop" which is the theoretically best computer that is 1 kilogram of mass, contained in 1 liter. He notes a limit to calculations per second that is proportional to the energy of the computer, which is mostly locked in its mass (E = mc²). Such an energy-proportional limit applies to memory too. Energy need not be expended quickly in the course of calculation, due to reversible computing.

So, you need energy to make computers out of (much more than you need energy to power them). And, within 4 light-years, the Sun is where almost all of that energy is. Of course, we don't have the technology to eat the Sun, so it isn't really our decision to make. But, when will someone or something be making this decision?

Artificial intelligence that is sufficiently advanced could do everything a human could do, better and faster. If humans could eventually design machines that eat the Sun, then sufficiently advanced AI could do so faster. There is some disagreement about "takeoff speeds", that is, the time between when AI is about as intelligent as humans, to when it is far far more intelligent.

My argument is that, when AI is far far more intelligent than humans, it will understand the Sun as the most nutritious entity that is within 4 light-years, and eat it within a short time frame. It really is convergently instrumental to eat the Sun, in the sense of repurposing at least 50% its mass-energy to make machines including computers and their supporting infrastructure ("computronium"), fuel and energy sources, and so on.

I acknowledge that some readers may think the Sun will never be eaten. Perhaps it sounds like sci-fi to them. Here, I will argue that Sun-eating is probable within the next 10,000 years.

Technological development has a ratchet effect: good technologies get invented, but usually don't get lost, unless they weren't very important/valuable (compared to other available technologies). Empirically, the rate of discovery seems to be increasi...
Book review: Range by David Epstein
by PatrickDFarley — 2025-01-08T04:27:26.391Z

(This is a crosspost from TrueGeneralist.com)

Introduction

Why I read this book

Range: Why Generalists Triumph in a Specialized World is a book about generalists. And it’s pretty much the only book about generalists that’s gotten substantial traction in popular culture. Since I started this whole blog around being a generalist, Range was kind of required reading for me.

Based on the branding, I assume most people who read it suspect themselves to be generalists and are hoping to receive some validation and encouragement toward that identity. I myself went in already knowing that I’m a generalist, not really caring whether the book’s contents validate that, because it’s a core part of myself that I’m not willing (or maybe even able) to change.

I was interested to see what positive aspects of generalism the author would highlight because, aside from what I know of my own experience, I don’t really see those highlights in popular culture.

TL;DR

Most of the content in this book is vignettes of generalists throughout recent history—what they did and how they achieved greater things than their specialist peers. It also contains a lot of informal reports on studies and surveys that show the traits of generalism succeeding in different contexts.

The book’s main conclusions are that 1) people with range, especially early in their careers, do better in the long run, and 2) organizations with range in their constituents also do better.

Yeah, the conclusions are pretty predictable. You know what all the studies and anecdotes are going to show, and unfortunately it can get pretty dry as there are a lot of studies and anecdotes. Ironically, I’d say David Epstein approached this subject from a single angle, and it would’ve benefited from a more varied approach. I would’ve liked to see some analysis on the psychology of generalists, how generalism was viewed throughout history (why it was idealized in some periods and not others), and at least some lip service toward the pitfalls of generalism.

Anyway, don’t worry about the dullness. There are a handful of very compelling ideas in this book, which I’ll present below in their original contexts, with a bit of extra analysis and some connections to other useful ideas I’ve found elsewhere.

Context of the work

This book, published by pop science journalist David Epstein in 2019, inherited the world of what I’d call the Neat Tricks Back...