trentmkelly/LessWrong-43k huggingface dataset:
Six Thoughts on AI Safety
by boazbarak — 2025-01-24T22:20:50.768Z

[Crossposted from windowsontheory]

The following statements seem to be both important for AI safety and are not widely agreed upon. These are my opinions, not those of my employer or colleagues. As is true for anything involving AI, there is significant uncertainty about everything written below. However, for readability, I present these points in their strongest form, without hedges and caveats. That said, it is essential not to be dogmatic, and I am open to changing my mind based on evidence. None of these points are novel; others have advanced similar arguments. I am sure that for each statement below, there will be people who find it obvious and people who find it obviously false.

AI safety will not be solved on its own.
An “AI scientist” will not solve it either.
Alignment is not about loving humanity; it’s about robust reasonable compliance.
Detection is more important than prevention.
Interpretability is neither sufficient nor necessary for alignment.
Humanity can survive an unaligned superintelligence.

Before going into the points, we need to define what we even mean by “AI safety” beyond the broad sense of “making sure that nothing bad happens as a result of training or deploying an AI model.” Here, I am focusing on technical means for preventing large-scale (sometimes called “catastrophic”) harm as a result of deploying AI. There is more to AI than technical safety. In particular, many potential harms, including AI enabling mass surveillance and empowering authoritarian governments, can not be addressed by technical means alone.

My views on AI safety are colored by my expectations of how AI will progress over the next few years. I believe that we will make fast progress, and in terms of technical capabilities, we will reach “AGI level” within 2-4 years, and in a similar timeframe, AIs will gain superhuman capabilities across a growing number of dimensions. (Though I suspect translating these technical capabilities to the economic and societal impact we associate with AGI will take significantly longer.) This assumes that our future AGIs and ASIs will be, to a significant extent, scaled-up versions of our current models. On the one hand, this is good news, since it means our learnings from current models are relevant for more powerful ones, and we can develop and evaluate safety techniques using them. On the other hand, this makes me doubt ...
Instrumental Goals Are A Different And Friendlier Kind Of Thing Than Terminal Goals
by johnswentworth — 2025-01-24T20:20:28.881Z

The Cake

Imagine that I want to bake a chocolate cake, and my sole goal in my entire lightcone and extended mathematical universe is to bake that cake. I care about nothing else. If the oven ends up a molten pile of metal ten minutes after the cake is done, if the leftover eggs are shattered and the leftover milk spilled, that’s fine. Baking that cake is my terminal goal.

In the process of baking the cake, I check my fridge and cupboard for ingredients. I have milk and eggs and flour, but no cocoa powder. Guess I’ll have to acquire some cocoa powder! Acquiring the cocoa powder is an instrumental goal: I care about it exactly insofar as it helps me bake the cake.

My cocoa acquisition subquest is a very different kind of goal than my cake baking quest. If the oven ends up a molten pile of metal shortly after the cocoa is acquired, if I shatter the eggs or spill the milk in my rush to the supermarket, then that’s a problem - a molten oven or shattered eggs or spilled milk would make it harder for me to bake the cake! More generally, in the process of acquiring cocoa powder, I want to not mess up other things which are helpful for making the cake. Unlike my terminal goal of baking a cake, my instrumental goal of acquiring cocoa powder comes with a bunch of implicit constraints about not making other instrumental subgoals much harder.

(If you’re already thinking “hmm, that sounds kinda like

corrigibility

”, then you have the right idea and that is indeed where we’re going with this.)

Generalizable takeaway: unlike terminal goals, instrumental goals come with a bunch of implicit constraints about not making other instrumental subgoals much harder.

The Restaurant

Now imagine that I’m working as a chef in a big restaurant. My terminal goal is the restaurant’s long-term success; I care about nothing else. If the bombs drop, so long as the restaurant is still doing good business afterwards, I’ll be happy.

One day, a customer orders a fresh chocolate cake, and it falls to me to bake it. Now baking the cake is an instrumental goal.

One key difference from the previous example: in the restaurant, I don’t know all the things which future customers will order. I don’t know exactly which ingredients or tools will be needed tomorrow. So, in the process of baking the cake, I want to avoid wasting ingredients or destroying tools which might be useful for any of ...
Yudkowsky on The Trajectory podcast
by Seth Herd — 2025-01-24T19:52:15.104Z
Link: https://www.youtube.com/watch?v=YlsvQO0zDiE

Edit: TLDR: EY focuses on the clearest and IMO most important part of his argument:

Before building an entity smarter than you, you should probably be really sure its goals align with yours.
Humans are historically really bad at being really sure of anything nontrivial on the first real try.

I found this interview notable as the most useful public statement yet of Yudkowsky's views. I congratulate both him and the host, Dan Fagella, for strategically improving how they're communicating their ideas.

Dan is to be commended for asking the right questions and taking the right tone to get a concise statement of Yudkowsky's views on what we might do to survive, and why. It also seemed likely that Yudkowsky has thought hard about his messaging after having his views both deliberately and accidentally misunderstood and panned. Despite having followed his thinking over the last 20 years, I gained new perspective on his current thinking from this interview.

Takeaways:

Humans will probably fail to align the first takeover-capable AGI and all die

Not because alignment is impossible
But because humans are empirically foolish
And historically rarely get hard projects right on the first real try

Here he distinguishes first real try from getting some practice -

Metaphor: launching a space probe vs. testing components

Therefore, we should not build general AI

This ban could be enforced by international treaties

And monitoring the use of GPUs, which would legally all be run in data centers
Yudkowsky emphasizes that governance is not within his expertise.

We can probably get away with building some narrow tool AI to improve life

Then maybe we should enhance human intelligence before trying to build aligned AGI

Key enhancement level: get smart enough to quit being overoptimistic about stuff working

History is just rife with people being surprised their projects and approaches don't work

I find myself very much agreeing with his focus on human cognitive limitations and our poor historical record of getting new projects right on the first try. I researched cognitive biases as the focus of my neuroscience research for some years, and came to the conclusion that wow, humans have both major cognitive limitations (we can't really take in and weigh all the relevant data for complex questions like alignment) and have major biases, notably a sort of inevitable tendency to believe what seems lik...
Empirical Insights into Feature Geometry in Sparse Autoencoders
by Jason Boxi Zhang — 2025-01-24T19:02:19.167Z

Key Findings:

We demonstrate that subspaces with semantically opposite meanings within the GemmaScope series of Sparse Autoencoders are not pointing towards opposite directions.
Furthermore, subspaces that are pointing towards opposite directions are usually not semantically related.
As a set of auxiliary experiments, we experiment with the compositional injection of steering vectors (ex: -1*happy + sad) and find moderate signals of success.

An Intuitive Introduction to Sparse Autoencoders

What are Sparse Autoencoders, and How Do They Work?

High Level Diagram Showcasing SAE Workflow: taken from Adam Karvonen's Blog Post.

Sparse Autoencoder (SAE) is a dictionary learning method with the goal of learning monosemantic subspaces that map to high-level concepts (1). Several frontier AI labs have recently applied SAEs to interpret model internals and control model generations, yielding promising results (2, 3).

To build an intuitive understanding of this process, we can break down the SAE workflow as follows:

SAEs are trained to reconstruct activations sampled from Language Models (LM). They operate by taking a layer activation from an LM and up-projecting it onto a much wider latent space with an encoder weight matrix, then down-projecting the latent to reconstruct the input activations. SAEs are trained to improve sparsity via an activation function like ReLU before being down-projected alongside a L1 loss on the latent space.
The columns of the encoder matrix and the rows of the decoder matrix can be interpreted as learned "features" corresponding to specific concepts (e.g., activation patterns when the word "hot" is processed by a LM).
When multiplying layer activations by the encoding matrix, we effectively measure the alignment between our input and each learned feature. This produces a "dictionary" of alignment measurements in our SAE activations.
To eliminate "irrelevant" features, a ReLU/JumpReLU activation function sets all elements below a certain threshold to zero.
The resulting sparse SAE dictionary activation vector serves as a set of coefficients for a linear combination of the most significant features identified in the input. These coefficients, when multiplied with the decoder matrix, attempt to reconstruct the original input as a sum of learned feature vectors.
For a deeper intuitive understanding of SAEs, refer to Adam Karvone

...
Ideas for CoT Models: A Geometric Perspective on Latent Space Reasoning
by Rohan Ganapavarapu — 2025-01-24T19:01:47.339Z
Link: https://rohan.ga/blog/llm_cot/

Disclaimer: These ideas are untested and only come from my intuition. I don’t have the resources to explore them any further.

intro

CoT and test time compute have been proven to be the future direction of language models for better or for worse. o1 and DeepSeek-R1 demonstrate a step function in model intelligence. Coconut also provides a way for this reasoning to occur in latent space. I have been thinking about the geometric structure of the latent space where this reasoning can occur.

I want to propose a different geometric perspective on how we structure the latent reasoning space. What if, instead of treating all reasoning steps uniformly, we designed the latent space to mirror how complex problem-solving naturally progresses–from broad exploration to precise refinement?

The intuition is: early reasoning steps require a rich space for exploring multiple potential paths, while later steps need precision to nail down the exact solution. This suggests structuring the latent reasoning space as a progressive funnel: starting with high-dimensional, low-precision representations that gradually transform into lower-dimensional, high-precision ones.

how to structure latent space

We structure the latent reasoning space as a progressive funnel: starting with high-dimensional, low-precision representations that gradually transform into lower-dimensional, high-precision ones.

Early reasoning steps would operate in a vast but coarse-grained space. This creates a rich geometric landscape where many potential reasoning paths can coexist “orthogonally” without interfering with each other. We have many rough directions to explore simultaneously.

As reasoning progresses, we’d project into increasingly focused spaces with higher precision per dimension. While we lose some of that initial expressiveness, we gain the ability to make more precise distinctions–perfect for refining the final steps of a logical deduction or mathematical calculation.

geometric intuition

In the early high-dimensional space, the “concentration of measure” phenomenon actually helps keep different partial solutions naturally separated. The manifold has many local peaks and valleys, allowing the model to maintain multiple hypotheses in superposition.

As we funnel down to lower dimensions, we’re essentially performing a learned form of dimensionality reduction that preserves the most promising reasoning pathways w...
Liron Shapira vs Ken Stanley on Doom Debates. A review
by TheManxLoiner — 2025-01-24T18:01:56.646Z

I summarize my learnings and thoughts on Liron Shapira's discussion with Ken Stanley on the Doom Debates podcast. I refer to them as LS and KS respectively.

High level summary

Key beliefs of KS:

Future superintelligence will be 'open-ended'. Hence, thinking of them as optimizers will lead to incomplete thinking and risk mitigations.
P(doom) is non-zero, but no fixed number. Changes from day to day.
Superintelligence is a risk, but that open-endedness is the root of the problem, not optimization.
KS' main desire is to increase awareness that superintelligence will be open-ended, because most people (regardless of their p(doom)) do not discuss or believe this, and hence the strategies to reduce risk will not be appropriate.
KS believes that superintelligence will not have significantly more foresight into the development of future technologies, because the task is fundamentally too complex and unpredictable. The only way to make progress is by doing incremental research and real world experiments, motivated by what is 'interesting'. The result is a gradual accumulation of 'stepping stones' that allow you to go further and further along the tech tree.
KS is uncertain if super-intelligence will have significantly more optimizing ability than we do, essentially because we over-rate optimization and open-ended divergent processes is the key ingredient to intelligence.
KS does not have strong opinions on policy proposals. KS does tentatively suggest that enforcing humans-in-the-loop (somehow) as one possibility to mitigate the risk. E.g. if the super-intelligence wanted to use 50% of the available resources to create a humungous particle accelerator, it needs to get our permission first.
KS takes the risks due to AI seriously and this was a big factor for why he is currently not working on AI.

In contrast, LS does believe that superintelligence will be a super-optimizer and that it will be capable of feats of foresight and ingenuity that allow it to skip many of the stepping stones that we humans would have to take. E.g. LS believes that a superintelligence in early 1800s could have skipped the vacuum tubes and developed modern electronic computers on its first try.

Unfortunately, most of KS' claims are not justified. Partly this is because KS did not explain himself clearly so it took effort to pin down his beliefs, and also because LS chose to drill down into details that - ...
Is there such a thing as an impossible protein?
by Abhishaike Mahajan — 2025-01-24T17:12:01.174Z
Link: https://www.owlposting.com/p/is-there-such-a-thing-as-an-impossible

This is something I’ve been thinking about since my synthesizability article.

Let’s assume, given the base twenty amino acids that are naturally present in the human body, we have every possible permutation of them for up to 100 amino acids, stored in a box with pH 7.4 water and normal pressures and temperature and isolated from one another. In other words, we have on the order of 20^100 proteins available to us. This is a very large number.

What percentage of these proteins could be made?

Maybe all of them? But this is true of small molecules as well; technically all small molecules can exist, the vast majority of them would just instantly vanish/explode/dissolve upon their forced manifestation. Those molecules carry the term ‘impossible’ as well, but the more accurate term for them is just ‘unstable’. Really then, the question becomes not just whether we can string amino acids together in a particular sequence, but whether that sequence can maintain its existence as a protein for any meaningful amount of time. What’s meaningful? Let’s say 60 seconds. So, we take our bucket of proteins and wait 60 seconds.

Notably, this is a different question than the one examined in the paper Distinguishing Proteins From Arbitrary Amino Acid Sequences, which defines a ‘protein’ as anything that has a well-structured 3D shape. I don’t care about that, I want to confirm that all strings of arbitrary amino acids are possible to create.

How many proteins are left after these 60 seconds? Is the answer once again ‘all of them’?

As far as I tell, the answer is, again, yes. Nearly 100% of them will remain. There might be a minuscule subset that forms bizarre reactive loops or self-hydrolyzing sites (of which I can find little information on what consistently causes such sites), but for the overwhelming majority of those 20^100 sequences, there is no biochemical mechanism that disintegrates a protein in a matter of seconds at neutral pH and room temperature.

Well…that’s a boring answer.

But perhaps thinking of the space of all possible proteins in the same way we think of the space of all possible molecules in general is misleading. Amino acids are, by nature, largely non-reactive (with a few exceptions). The very property that makes proteins such excellent building blocks for life — their chemical stability —also means that most random sequences wouldn't pose inherent stability problems....
Stargate AI-1
by Zvi — 2025-01-24T15:20:18.752Z

There was a comedy routine a few years ago. I believe it was by Hannah Gadsby. She brought up a painting, and looked at some details. The details weren’t important in and of themselves. If an AI had randomly put them there, we wouldn’t care.

Except an AI didn’t put them there. And they weren’t there at random.

A human put them there. On purpose. Or, as she put it:

THAT was a DECISION.

This is the correct way to view decisions around a $500 billion AI infrastructure project, announced right after Trump takes office, having it be primarily funded by SoftBank, with all the compute intended to be used by OpenAI, and calling it Stargate.

Table of Contents

The Announcement.

Is That a Lot?.

What Happened to the Microsoft Partnership?.

Where’s Our 20%?.

Show Me the Money.

It Never Hurts to Suck Up to the Boss.

What’s in a Name.

Just Think of the Potential.

I Believe Toast is an Adequate Description.

The Lighter Side.

The Announcement

OpenAI: Announcing The Stargate Project

The Stargate Project is a new company which intends to invest $500 billion over the next four years building new AI infrastructure for OpenAI in the United States. We will begin deploying $100 billion immediately.

Note that ‘intends to invest’ does not mean ‘has the money to invest’ or ‘definitely will invest.’ Intends is not a strong word. The future is unknown and indeed do many things come to pass.

This infrastructure will secure American leadership in AI, create hundreds of thousands of American jobs, and generate massive economic benefit for the entire world.

This project will not only support the re-industrialization of the United States but also provide a strategic capability to protect the national security of America and its allies.

One of these things is not like the others. Secure American leadership in AI, generate massive economic benefit for the entire world, provide strategic capability to allies, sure, fine, makes sense, support reindustrialization is a weird flex but kinda, yeah.

And then… jobs? American… jobs? Um, Senator Blumenthal, that is not what I meant.

Pradyumna:

> will develop superintelligence

> create thousands of jobs

????

Samuel Hammond: “We’re going to spend >10x the budget of the Manhattan Project building digital brains that can do anything human brains can do but better and oh, by the way, create over 100,000 good paying American jobs!”

There’s at le

...
QFT and neural nets: the basic idea
by Dmitry Vaintrob — 2025-01-24T13:54:45.099Z

Previously in the series: The laws of large numbers and Basics of Bayesian learning.

Reminders: formalizing learning in ML and Bayesian learning

Learning and inference in neural nets and Bayesian models

As a very basic sketch, in order to specify an ML algorithm one needs five pieces of data.

An architecture: i.e., a parametrized space of functions that associates to each weight vector θ∈Rw a function fθ:Rdin→Rdout, from some input space to an output space.
An initialization prior on weights. This is a (usually stochastic) algorithm to initialize a weight from which to begin learning. Generally this is some Gaussian distribution on the weight θ∈Rw. While this is often ignored, in many contexts it is actually quite important to get right for learning to behave reasonably.
Training data. This is a collection D=(x1,y1;x2,y2;…;xn,yn) of “observation” pairs (xi,yi), with xi∈Rdin and yi∈Rdout.
A loss function. This is a function on L(D,θ) which operationalizes a measure of how well fθ agrees with the data D.
A learning algorithm/optimizer. (“Learning algorithm” is used in theoretical contexts and “optimizer” is frequently used in engineering contexts.) This is an algorithm (usually, a stochastic algorithm) for finding the “learned” weight parameter θ, usually by some local minimization or approximate minimization, of the function L(θ).

In most theoretical analyses of models, one uses gradient descent to conceptualize the learning algorithm. Some more sophisticated pictures (such as the tensor programs series) more carefully match realistic stochastic gradient descent by assuming a discontinuous process with finite learning steps rather than continuous gradient descent. Learning algorithms used in industry tend to include more sophisticated control over the gradients by things like gradient decay, momentum, ADAM, etc.

All of these algorithms have in common the property of being sequential and local (i.e., there is some step-to-step learning that ends when it converges to an approximate local minimum.) However, when working theoretically, a learning algorithm doesn’t have to be local or sequential.

Bayesian learning is one such (non-local and non-sequential) learning algorithm. This algorithm converts the learning problem to a Bayesian inference problem. Here the dictionary from ML to statistics is as follows:

Architecture ⟹ D

...