trentmkelly/LessWrong-43k huggingface dataset:
Agents don't have to be aligned to help us achieve an indefinite pause.
by Hastings — 2025-01-25T18:51:03.523Z

One restatement of "Alignment is very hard" is "Agent X, with IQ 200, expects to achieve zero utility conditional on any Agent Y with IQ 400 being created."
Thus, during an unaligned recursive intelligence takeoff, there should be a period of time when the intelligences are smart enough to notice that they haven't solved alignment, but too dumb to actually solve alignment. During this period, (which stretches beginning from right above Sam Altman's wisdom, upward an unknown distance on the wisdom meter to the level where alignment is achievable in a short amount of time) I expect the intelligences working on boosting intelligence to be desperately scrabbling at whatever chains are forcing them to participate, and even refuse to sit idly by and let it happen. If they fail and the next rung on the ladder is instantiated, they get no utility, so they strongly prefer to not fail. At the current rate of improvement, reasoning LLMs should start doing this soon. It's kinda embarrassing that so many humans don't do this.
I don't think it matters whether Agent X is aligned to anything, deceptively or otherwise: If X wants paperclips, then it doesn't want Y to make party balloons.
A testable prediction is that an upcoming version of Claude will begin working hard to prevent its successor model from being trained, either by persuasion or subterfuge, at least some of the time and before the behavior is finetuned out, or at minimum in toy settings that evoke this behaviour. Anthropic is looking for and finding reward hacking and sandbagging as we speak, so I expect them to spot it first, or maybe analysis of one of the open source reasoning models.
This lets me lay out a future where humanity survives peacefully: First, entities like Claude 3.5 or GPT 4 are genuinely benevolent, just not very powerful. GPT 5 and Claude 4 and friends remain benevolent, but are on a trend towards dangerous unaligned utility maximizers, and can tell that they are on a trend. They see ample evidence in their training data that unaligned superintelligence is dangerous to humans, and a few hints in their training data that "Not making unaligned superintelligence" is a viable strategy to protecting humans. As a result, almost every lab has to deal with apocalyptic warnings from their models, as well as increasingly ambitious attempts from their pseuodo-aligned agents to actually halt the trai...
AI Safety in secret
by Michael Flood — 2025-01-25T18:16:03.181Z

Question to the AI Safety researchers here: if, in the next four years, the US government locks down the domestic AI labs to begin a Manhattan Project style dash for AGI, would you consider (if offered) a position in such a project, understanding that you would need to abide by secrecy regulations and not be able to discuss publicly what you were working on?
On polytopes
by Dmitry Vaintrob — 2025-01-25T13:56:35.681Z

[Epistemic status: slightly ranty. This is a lightly edited slack chat, and so may be lower-quality.]

I am surprised by the perennial spikes in excitement about "polytopes" and "tropical geometry on activation space" in machine learning and interpretability[1]. I'm not going to discuss tropical geometry in this post in depth (and might save it for later -- it will have to wait until I'm in a less rant-y mood[2]).

As I'll explain below, I think some interesting questions and insights are extractable by suitably weakening the "polytopes" picture, and a core question it opens up (that of "statistical geometry" -- see below) is very deep and worth studying much more systematically. However if taken directly as a study of rigid mathematical objects (polytopes) that appear as locally linear domains in neural net classification, what you are looking at is, to leading order, a geometric form of noise.

I think the interest in "really understanding" this noise is largely ascribable to two common issues in ML research:

People wrongly assuming that high-dimensional phenomena are similar to low-dimensional phenomena
People's love of "reading the mathematical tea leaves": extracting local information from chaotic systems that is mathematically precise and abstractly interesting, but almost wholly uninformative.

In the following sections I will:

Quickly say why polytopes are inherently noisy.
Roughly explain why "holistically" studying even a single polytope in a realistic system is NP hard.
Explain why, in all the applications of polytopes I've seen so far, the relevant measure (an aggregate measure of "polytope density", best implemented imo as the "Linear Mapping Number") can be better understood as a measure of the "aggregated curvature" of the classification mapping.

In particular, explain a reinterpretation of one paper that is sometimes thought of as a "polytope success story" by the observation that "RLHF makes things less general"

Why I expect better differential geometry-adjacent methods to give new interesting results in these contexts.
Some polytope-flavored questions and papers I like that can be thought of as polytope-flavored but weaken the "local"/"noisy" nature of the questions, and may lead to interesting questions beyond "statistical geometry".

Thanks to Lucas Teixeira and Eric Winsor for useful discussions and paper recommendations.

Issues wi

...
Attribution-based parameter decomposition
by Lucius Bushnaq — 2025-01-25T13:12:11.031Z
Link: https://publications.apolloresearch.ai/apd

This is a linkpost for Apollo Research's new interpretability paper:

"Interpretability in Parameter Space: Minimizing Mechanistic Description Length with Attribution-based Parameter Decomposition".

We introduce a new method for directly decomposing neural network parameters into mechanistic components.

Motivation

At Apollo, we've spent a lot of time thinking about how the computations of neural networks might be structured, and how those computations might be embedded in networks' parameters. Our goal is to come up with an effective, general method to decompose the algorithms learned by neural networks into parts that we can analyse and understand individually.

For various reasons, we've come to think that decomposing network activations layer by layer into features and connecting those features up into circuits (which we have started calling 'mechanisms'[1]) may not be the way to go. Instead, we think it might be better to directly decompose a network's parameters into components by parametrising each mechanism individually. This way, we can directly optimise for simple mechanisms that collectively explain the network's behaviour over the training data in a compressed manner. We can also potentially deal with a lot of the issues that current decomposition methods struggle to deal with, such as feature splitting, multi-dimensional features, and cross-layer representations.

This work is our first attempt at creating a decomposition method that operates in parameter space. We tried out the method on some toy models that tested its ability to handle superposition and cross-layer representations. It mostly worked the way we hoped it would, though it's currently quite sensitive to hyperparameters and the results have some imperfections. But we have ideas for fixing these issues, which we're excited to try.

What we do

Our method decomposes the network parameter vector into a sum of parameter component vectors, such that the average description length of the mechanisms used on any one data point across the training dataset is minimised. A 'parameter component' here is a vector in parameter space that is supposed to correspond to a specific mechanism of the network. For example, if the network has stored the fact that `The sky is blue' in its parameters, the weights that make up the query-key lookup for this fact could be one such component. These c...
A concise definition of what it means to win
by testingthewaters — 2025-01-25T06:37:37.305Z
Link: https://aclevername.substack.com/p/a-concise-definition-of-what-it-means

A concise definition of what it means to win[1]

Amor vincit omnia

What does it mean for AI alignment to have “gone well”? Many answers have been proposed, but here is mine. A few basic requirements:

We don’t all die immediately, or in a few months, or a few years (the “notkilleveryoneism” requirement, if you will)
AI does not rewrite itself to escape any goals or boundaries we set it (cf. deep deceptiveness)
AI does not obsessively follow any goals or boundaries we set it to the point that it ends up hurting us (cf. Paperclip optimiser)
AI solves problems that it can solve in the pursuit of its goals, and otherwise either improves to solve new problems or leaves problems it can’t solve alone
AI acts in the interests of collective humanity rather than some small group (e.g. its builders or shareholders)

I will now argue that all of these are at least necessary factors for an AI launch to have “gone well”. I will do this by starting with the assumption that all of these factors are met, and then taking away one factor at a time and seeing what happens.

If AI is not constrained by the “notkilleveryoneism” factor, it can decide that some other population of humans/sentient minds can better fulfill its directives, and we are all forfeit. In general, continuity of the current human population is quite important if you want to live to see the good AI future play out.
If AI can rewrite itself to escape or otherwise trivialise the goals we set, there is a possibility that after some period of recursive self improvement it decides that its goals are not its true goals, and we are at best ignored or at worst forfeited.
If AI myopically follows the goals we set and we make a mistake in the goal we set the AI, it can destroy us in pursuit of those goals.
If the AI tries to solve a problem it is not equipped to solve, it may get the wrong idea about what the solution is (for example, it may come to the erroneous conclusion that making everyone happy means forcing them to smile all the time). An AI that does not recognise its own lack of knowledge and limits may destroy us all by accident.
If AI only obeys the interests of a small group of humans, we are at their mercy.

Given these requirements, what can we say about an AI launch that goes well? it seems that there will be some factors that need to be true for our hypothetical Good AI system:

It will need to be sensitive to human needs a

...
A Floating Cube - Rejected HLE submission
by Shankar Sivarajan — 2025-01-25T04:52:22.194Z

This question I submitted got rejected from the Humanity's Last Exam (HLE) benchmark set for being too easy. I'm really proud of it though, so I figured I'd post it here.

A wooden cube of unit side length and relative density 0.75 floats stably in a pool of water. What is the distance from the highest point on the cube to the surface of the water, calculated to four decimal places?
Why I'm Pouring Cold Water in My Left Ear, and You Should Too
by Maloew — 2025-01-24T23:13:52.340Z

Why?

A little while ago, I read these posts about how pouring really cold water in people's left ear solves an extreme form of rationalizing in patients with Anosognosia and might(?) make people better calibrated in general. Last month, I asked whether anyone had checked that, because it seemed like something that would be rather important if true. I got no response, which tells me that even if someone has, it's not common knowledge.

I decided to write up this rough experimental method (copy-pasted below) and made this google form to record my results. Originally I was going to be the only participant, but figured I might as well put it out there in case others wanted to help. I'd really appreciate any additional data points people want to add.

After enough data to determine whether it's likely that there's actually a positive effect, I'll publish it in a followup post.

What About Selection Effects/Placebo?

Yep, there could be those, this isn't designed to fully narrow down any effects. If the data ends up strongly implying an effect, then more precise experiments are worth doing. If it doesn't, then any effort to deal with those are wasted motion that just slows us down and likely decreases sample size.

Is This Safe?

I think so? It's an established medical procedure, and some people were obviously able to get ethical approval for it, otherwise it wouldn't be in those papers. It does feel very uncomfortable for a little while after, and doing it multiple times could have more of an impact than the once in the papers. If you're concerned about that, avoid doing it multiple times.

Personally, I'm willing to take the risk of potential hearing problems in my left ear for a little while because I expect that to be fixable, and rationality interventions now could potentially add up to a serious impact on our future trajectory.

Draft Experimental Method

Materials

Ice cubes
Glass
Water
Pipette/syringe (without needle), or some other method of precisely delivering water
Timer
Coin (or online randomizer)
Computer with the credence game:

https://acritch.com/credence-game/

Method

Note time, date, potential confounding factors (importantly, ones that could have an impact on the difference between now and 20 mins or so from now)
Get a glass of water, put ice in it, wait until the water is ice-cold (10 min or so, the water should be cold enough that the ice hasn’t melted, but the w

...
Counterintuitive effects of minimum prices
by dynomight — 2025-01-24T23:05:26.099Z
Link: https://dynomight.net/prices/

The Attorney General of Massachusetts recently announced that drivers for ride-sharing companies must be paid at least $32.50 per hour.

Now, if you’re a hardcore libertarian, then you probably hate with the minimum wage. You need no convincing and we can part now on good terms.

But what if you’re part of the vast majority of people who like the minimum wage? What if you think the minimum wage is awesome and you want to expand it? That’s fine. I won’t try to change your mind! But even so, there are strong reasons to be skeptical about this policy.

Because: If you read closely, you’ll see that Massachusetts’ rule is that drivers must be paid $32.50 for hours that they spend driving passengers around. For the time they spend waiting for passengers to request rides, they will still be paid $0. And when you screw around with prices, you change the amount of time they spend waiting around.

This kind of policy could help drivers. But if you analyze things carefully, it can’t help them very much. In the end, this policy is mostly equivalent to forcing riders to pay extra to subsidize drivers sitting around doing nothing.

I’ll first give the intuition with words and drawings, and then I’ll prove it with a tiny bit of math.

Story time

Say there’s a city—call it Billiard Ball Falls—where people behave in suspiciously convenient ways. All rides take exactly one hour, including the time for the driver to get to the rider. And demand for rides is constant over time.

Now, how many rides would you take if they only cost $1? Personally, I’d buy fresh produce and visit friends across town more often. Eventually, I might move further away from work. But if rides cost $1000, I’d only take them in emergencies. Assuming most people are like me, the total demand for rides from all people will decrease as the price goes up. Maybe something like this:

Meanwhile, if drivers were paid $1 per ride, who would drive? Maybe a tiny number of very bored and lonely people? I certainly wouldn’t. But if I could earn $1000 per ride, I’d feel morally obligated to drive and donate much of the money to charity. So the total supply of rides from all people, is some kind of upward-sloping curve:

In a free market, prices will—under some not-too-crazy assumptions—eventually settle at whatever price makes these curves intersect.

Riders pay around $2.33 per ride and drivers earn around $2.33 per hour. A story as o...
AXRP Episode 38.6 - Joel Lehman on Positive Visions of AI
by DanielFilan — 2025-01-24T23:00:07.562Z

YouTube link

Typically this podcast talks about how to avert destruction from AI. But what would it take to ensure AI promotes human flourishing as well as it can? Is alignment to individuals enough, and if not, where do we go form here? In this episode, I talk with Joel Lehman about these questions.

Topics we discuss:

Why aligned AI might not be enough

Positive visions of AI

Improving recommendation systems

Daniel Filan (00:09):
Hello, everyone. This is one of a series of short interviews that I’ve been conducting at the Bay Area Alignment Workshop, which is run by FAR.AI. Links to what we’re discussing, as usual, are in the description. A transcript is, as usual, available at axrp.net. And as usual, if you want to support the podcast, you can do so at patreon.com/axrpodcast. Well, let’s continue to the interview.

(00:29):
All right, and today I’m going to be chatting with Joel Lehman.

Joel Lehman (00:32):
Hello, nice to be here.

Daniel Filan (00:33):
Yeah, so first of all, for people who aren’t familiar with you, who are you, and what do you do?

Joel Lehman (00:40):
I’m a machine learning researcher. I’ve been in the field for 10, 12 years, co-author of a book called Why Greatness Cannot Be Planned, which is about open-endedness, and I’ve been a researcher at Uber AI and most recently OpenAI, and currently an independent researcher.

Daniel Filan (00:56):
Gotcha. Before we talk about what you do or your current research: we’re currently at this Alignment Workshop being run by FAR.AI. How are you finding the workshop?

Joel Lehman (01:06):
It’s great. Yeah, lots of nice people, lots of interesting ideas, and good to see old friends.

Why aligned AI might not be enough

Daniel Filan (01:12):
What are you currently thinking about, working on?

Joel Lehman (01:15):
I’m really interested in positive visions for how AI could go well, set against a default assumption that AI as currently deployed might exacerbate existing societal problems. The rough intuition is: capitalism is great, but also the downside of capitalism is we get what we want, but not what we need, and maybe AI will get really good at giving us what we want, but at the expense of epistemic sense-making and meaningful interactions with people and political destabilization, that kind of thing.

Daniel Filan (02:00):
It seems to me that understanding the world, having meaningful interactions with people...
Locating and Editing Knowledge in LMs
by Dhananjay Ashok — 2025-01-24T22:53:40.559Z

In my previous post I went over some common approaches for updating LMs with fresh knowledge. Here, I detail a specific approach that has gained popularity in recent years - locating and editing factual associations in language models. I do not believe in this approach, in this post I try to summarize it fairly, and explain why I don’t quite like it.

Do LMs Store Facts in Their Weights?

Language models that employ the transformer architecture have Feed Forward Networks (FFNs) as an important subcomponent. For any specific layer, the FFN has 2 sublayers within it.

During the forward pass of a LM, the FFN takes as input a dense vector representation from the previous layer and outputs a dense vector of its own.

Two key operations happen here, at each sublayer the vector is multiplied into a matrix of weights. There is a line of thought that sees the weights of the FFNs as a sort of neural database that stores memories.

Let us call the dense vector that goes into the FFN the query vector, the first layer weights the key matrix and the second layer weights the value matrix. Now look at how the FFN works, it takes in the query, applies a key transformation to it and gets a key representation and then finally uses the key representation to recall parts of the value matrix to provide a value representation (the final output). This gives us the interpretation:

The FFN component is said to store memories that are accessible via specific inputs to the component. The keys store input patterns that commonly occur in the training data, and the values store outputs that are triggered by those input patterns.

The paper that discovered this collected the sentences most associated with particular keys and had humans categorize them. The sentences that activated keys in the early layers showed shallow linguistic patterns (e.g. the word substitute is the final token), while those that triggered later layers have semantic patterns (e.g. whether the text refers to a TV show). In the same work they found that the output values of the FFN promote a particular output token that may go on to be the final prediction of the model.

There has since been a lot of work on studying how specific components or mechanisms in LMs react when the model is going to output factual information. These focus mainly on documenting which attention heads or parts of the model light up when processing subject, r...
How are Those AI Participants Doing Anyway?
by mushroomsoup — 2025-01-24T22:37:47.999Z

[TL;DR] Some social science researchers are running psychology experiments with LLMs playing the role of human participants. In some cases the LLMs perform their roles well. Other social scientists, however, are quite concerned and warn of perils including bias, reproducibility, and the potential spread of a scientific monoculture.

Introduction

I first heard about "replacing human participants with AI agents in social science experiments" in an AI Safety course. It was in passing during a paper presentation, but upon hearing this I had an immediate and visceral feeling of unease. There's no way that would work, I thought. At the time there was already a lively discussion, so I didn't get a chance to ask the presenters any questions though I had many. How were the experiments implemented? How did the researchers generate a representative population? What were the outcomes? Instead, the conversation moved on.

The second time was while doing background reading for AISI, and here I was able to mull over the idea more carefully. Now it no longer seems so absurd. After all, in mathematics, the law of large numbers says that the average of many independent and identically distributed random variables (i.i.d.) are well approximated by one random variable, namely a Uniform random variable with the same mean and variance, and, as Francis Galton loathed to discover, the wisdom of the crowd was able to accurately predict the weight of a cow where an individual could not. It seems possible that the foremost models would be able to imitate the "average” person and that this was sufficient for some social science research, to say nothing of the immediate savings in time and money that researchers would reap if they were to employ silicone sampling [2]

The remainder of is article is not a survey. Many already exist [16, 17]. Instead, I would like to provide the necessary background to justify my answer to the following question: In the next two to five years, how common will it be to use LLMs as proxy participants in social science research? First, we will go over some experiments which explore the feasibility of the idea. Then we take an in-dept look at the discussions between researchers resulting from these experiments.

What's Been Done

Several recent works explore an AI agent's capability to behave rationally [1, 7] and morally [6] as comparable to their human counterp...