trentmkelly/LessWrong-43k huggingface dataset:
Unlearning Needs to be More Selective [Progress Report]
by Filip Sondej — 2025-06-27T16:38:00.136Z

Summary

We’d like to share our ongoing work on improving LLM unlearning. [

arXiv

] [

github

]

There’s a myriad of approaches for unlearning, so over the past 8 months we conducted hundreds of small-scale experiments, comparing many loss functions, variants of meta-learning, various neuron or weight ablations, representation engineering and many exotic ways of constraining or augmenting backpropagation.

Almost all of these methods succeed in making the forget set loss high after unlearning, but (

consistent

with

countless

prior

findings

) fine-tuning attacks typically restore the forget accuracy almost immediately, which indicates that unwanted capabilities are not truly removed, but

merely hidden

.

However, we have noticed several trends - things which pretty reliably seem to help with attack robustness:

Selectivity - Unlearning should be like a precise surgery rather than a lobotomy. Most techniques try to undo the disruption caused by unlearning post-hoc by retraining on a retain set, which is costly and doesn’t catch all disruption. We found it’s much better to not disrupt in the first place, which we do by limiting the unlearning updates in various ways. The technique which we finally arrived at - which we dubbed Disruption Masking - simply allows only those weight updates where the unlearning gradient has the same sign as the retain set gradient.
Model-Agnostic Meta-Learning (

MAML

) - This approach is already used by some unlearning techniques (

TAR

,

MLAC

), and we confirm it consistently helps with attack robustness. (We hypothesize that it helps, because during unlearning, unwanted capabilities get hidden and then they can no longer be found by backpropagation and further unlearned, but MAML re-elicits them so unlearning can continue.)
Backpropagation ;) - We’ve tried many fancy exciting techniques (described in the “Failed Methods” section), but nothing quite matches the raw power of backpropagation when it comes to identifying which weights to attack. Most modifications just make it worse. The only robust improvements we found so far are selectivity and MAML mentioned above.

Our method (

MUDMAN

) which combines these insights, outperforms the current state-of-the-art unlearning method (

TAR

) by 40%.

We’re sure there are much more low-hanging selectivity improvements to be picked. Since writing the MUDMAN paper we’ve alrea...
Jankily controlling superintelligence
by ryan_greenblatt — 2025-06-27T14:05:53.134Z

When discussing AI control, we often talk about levels of AI capabilities where we think control can probably greatly lower risks and where we can probably estimate risks. However, I think it's plausible that an important application of control is modestly improving our odds of surviving significantly superhuman systems which are misaligned. This won't involve any real certainty, it would only moderately reduce risk (if it reduces risk at all), and it may only help with avoiding immediate loss of control (rather than also helping us extract useful work). In a sane world, we wouldn't be relying on control for systems which have significantly superhuman general purpose capabilities. Nevertheless, I think applying control here could be worthwhile. I'll discuss why I think this might work and how useful I think applying control in this way would be.

As capabilities advance, the level of risk increases if we hold control measures fixed. There are a few different mechanisms that cause this, as we discussed here. It's more likely that more capable models are schemers, and people will want to deploy models with increasingly broad affordances by default. And because more capable AIs can more easily cause problems and acquire power in undesired ways, the risk[1] conditional on scheming increases, though it's unclear how fast this risk increases with capabilities.

Additionally, we're substantially uncertain about the general difficulty of control. For instance, it seems plausible that preventing the worst security failures (e.g., rogue deployments) is reasonably easy such that relatively basic measures make it very difficult for AIs which are around as capable as top human experts. In worlds where control ends up being easier, it seems plausible that this extends the viability of control out into substantially superhuman models. Of course, it's also possible that control is easy for around-human-level AIs but the difficulty of control rapidly increases into impossibility with only moderately superhuman general capabilities.

Beyond uncertainty about the difficulty of control, the efficacy of control might vary greatly between different applications of AI and different hazards. In particular, it seems like it might be much easier to prevent significantly superhuman AIs from being able to cause the worst security failures (given substantial investment into control measures i...
Childhood and Education #11: The Art of Learning
by Zvi — 2025-06-27T13:50:01.418Z

In honor of the latest (always deeply, deeply unpopular) attempts to destroy tracking and gifted and talented programs, and other attempts to get children to actually learn things, I thought it a good time to compile a number of related items.

Table of Contents

Lack Of Tracking Hurts Actual Everyone.

Not Tracking Especially Hurts Those Who Are Struggling.

No Child Left Behind Left Behind.

Read Early, Read Often.

Mirror, Mirror.

Spaced Repetition.

Learning Methods.

Interruptions.

Memorization.

Math is Hard.

Get to Work.

The Whiz Kids.

High School Does Not Seem To Teach Kids Much.

Two Kinds of Essays.

Lack Of Tracking Hurts Actual Everyone

Gifted programs and educational tracking are also super duper popular, it is remarkably absurd that our political process cannot prevent these programs from being destroyed.

As in things like this keep happening:

NY Post: Seattle Public Schools shuts down gifted and talented program for being oversaturated with white and asian students.

Once again, now, we face that threat in my home of New York City. The Democratic nominee for New York City Mayor is opposed to gifted and talented programs, and wants to destroy them. Yet few people seem to have much noticed, or decided to much care. Once people realize the danger it may well be too late.

To state the obvious, if you group children by ability in each subject rather than age, they learn better. Yes, there are logistical concerns, but the benefits are immense. Gifted programs are great but mostly seem like a patch to the fact that we are so obsessed with everyone in the room being in the same ‘grade’ at all times.

I agree with Tracing Woods that ‘teaches each according to their ability’ is the bare minimum before I will believe that your institution is making a real attempt to educate children.

Tracing Woods: A good example of the absurdity of “grade-level” thinking, from Texas: “If they’re reading on a second-grade level, but they’re in the third grade, they’re always going to receive that third-grade instruction.”

This makes no sense. Learning does not simply follow age.

Imagine having a “grade level” in chess. 9-year-olds in the third grade advancing to play against 1100 elo players. 100 more elo per year.

“Grade-level performance” has always been nonsensical. Learning does not work that way. Just figure out what people actually know

...
No, Futarchy Doesn’t Have an EDT Flaw
by Mikhail Samin — 2025-06-27T09:27:40.151Z

(A response to this post.)

If you use prediction markets to make decisions, you might think they’ll generate EDT decisions: you’re asking for P(A|B), where you care about A, and B is something like “a decision … is taken”.

Okay, so say you want to use prediction markets to generate CDT decisions. You want to know P(A|do(B)).

There’s a very simple way to do that:

Make a market on P(A|B).
Commit to resolving the market to N/A with 99.9% chance.
With 0.1% chance, transparently take a random decision among available decisions (the randomness is pre-set and independent of specific decisions/market data/etc.)

Now, 99.9% of the time, you can use freely use market data to make decisions, without impacting the market! You screened off everything upstream of the decision from the market. All you need for that is to make a random choice every now and then.

This really works to make CDT decisions! Try thinking through what the market would do in various decision-theoretic problems.

I claim that real markets actually do the exact same thing.

In the limit, you can imagine that when Tim Cook hires an awful designer, there’s no incentive for the markets to go down: because if they go down at all, Tim Cook would notice and fire them back.

But in reality, people and institutions are sometimes random! If there’s a chance Tim Cook doesn’t listen to the markets, the markets should down at least a tiny bit, to reflect that probability.

When Trump announced the tariffs, real markets showed only something like 10% of the reaction they would have if they thought the tariffs were to stay in place, because the markets correctly anticipated that the government would react to the market change.

But also, they went down, a lot, because they thought Trump might ignore them and not go back on the tariffs.

Even if you’re lower variance than Trump, to the extent you can be modeled well as mostly making a good choice but sometimes taking random decisions, the difference in conditional markets would already reflect at least some of the causal difference between different actions you can take.

And if that’s not enough, you can explicitly simulate the do() operator using the scheme above. It would require you to provide more liquidity when the market resolves, for traders to want to trade despite the overwhelming chance the market will resolve to N/A[1]; yet, it enables you to make decisions based the markets ...
Misconceptions on Affordable Housing
by jefftk — 2025-06-27T02:40:07.625Z

People often think 'affordable' housing is much more expensive than it
actually is, and then conclude it's a scam to make housing for rich
people. But this is often based on a misunderstanding of how the
prices are set.

Let's say a unit is "50% AMI" somewhere with an area median income
(AMI) of $100k. You might think, and I've seen a bunch of people with
this confusion, that units would rent for 50% of $100k: $50k/y
($4,170/month), but it's much cheaper than that.

Affordable has a very specific meaning in this context: spending no
more than 1/3 of your income on housing. So you might think 50% AMI
is 'affordable' to someone earning at the 50th percentile: 1/3 of
$100k, or $33k/y ($2,750/month), but it's cheaper than that, too!

Instead, 50% AMI means someone earning 50% of the AMI would be
spending 1/3 of their income on housing. This is 50% of 1/3 of the
AMI, and in this case that would be $17k/y ($1,390/month).

Now, I don't think affordable housing solves everything: if it were
widespread I think it would
be exploited and it can be a distraction from just getting a lot
of units built, but the term "affordable housing" is actually a decent
operationalization of whether housing is something regular people can
afford.

Comment via: mastodon, bluesky, substack
A case for courage, when speaking of AI danger
by So8res — 2025-06-27T02:15:25.781Z

I think more people should say what they actually believe about AI dangers, loudly and often. Even (and perhaps especially) if you work in AI policy.

I’ve been beating this drum for a few years now. I have a whole spiel about how your conversation-partner will react very differently if you share your concerns while feeling ashamed about them versus if you share your concerns while remembering how straightforward and sensible and widely supported the key elements are, because humans are very good at picking up on your social cues. If you act as if it’s shameful to believe AI will kill us all, people are more prone to treat you that way. If you act as if it’s an obvious serious threat, they’re more likely to take it seriously too.

I have another whole spiel about how it’s possible to speak on these issues with a voice of authority. Nobel laureates and lab heads and the most cited researchers in the field are saying there’s a real issue here. If someone is dismissive, you can be like “What do you think you know that the Nobel laureates and the lab heads and the most cited researchers don’t? Where do you get your confidence?” You don’t need to talk like it’s a fringe concern, because it isn’t.

And in the last year or so I’ve started collecting anecdotes such as the time I had dinner with an elected official, and I encouraged other people at the dinner to really speak their minds, and the most courage they could muster was statements like, “Well, perhaps future AIs could help Iranians figure out how to build nuclear material.” To which the elected official replied, “My worries are much bigger than that; I worry about recursive self-improvement and superintelligence wiping us completely off the map, and I think it could start in as little as three years.”

These spiels haven’t always moved people. I’m regularly told that I’m just an idealistic rationalist who’s enamored by the virtue of truth, and who’s trying to worm out of a taboo tradeoff between honesty and effectiveness. I’m sometimes told that the elected officials are probably just trying to say what they know their supporters want to hear. And I am not at liberty to share all the details from those sorts of dinners, which makes it a little harder to share my evidence.

But I am at liberty to share the praise we've received for my forthcoming book. Eliezer and I do not mince words in the book, ...
Help the AI 2027 team make an online AGI wargame
by Jonas V — 2025-06-27T01:02:52.157Z

The AI Futures Project, the team behind

AI 2027

, has developed a

tabletop exercise

(TTX) to simulate AI takeoff where we facilitate people going through an AI 2027-like scenario with different people taking different roles, like POTUS, China, or the frontier AI company CEO. Reviews have been positive and people have

found it useful

for building their own models of how AI will go.

We’re interested in developing an online version of this game, with the goal of eventually getting millions of people to play through takeoff and better understand the dynamics and risks involved. The other game characters could be driven by LLMs so we don’t need to facilitate it. The online TTX could be based on our scenario and background research alongside LLMs and various game elements.

To this end, we’re looking for a full-stack developer with game design and UX experience/interest and a deep understanding of AGI risk. We previously worked with the Lightcone Infrastructure team for the AI 2027 website and might work with them again for this project, but first we’d like to see if anyone else is interested.

We expect that the ideal online TTX would ultimately look and feel very different from the in-person experience, but would successfully convey similar content.

If you’re interested, please fill in

this quick form

, message me, or comment below, we’d love to hear from you!
The Bellman equation does not apply to bounded rationality
by Christopher King — 2025-06-26T23:01:33.369Z

Quick quiz: which of the following is the definition of a rational agent?

An agent who follows the policy with the highest expected utility
An agent who follows the policy of choosing the action with the highest expected utility

Are those even different things? In the case of unbounded rationality, it is a bit of a trick question. Usually (1) is taken as the definition of rational, but according to the Bellman equation, the optimal policy is the one that chooses an optimal action in every state. Seems obvious, right?

Bounded rationality is a form of rationality where the policy must given by an algorithm, and the computational resources used by that algorithm matters. For example, if computation produces heat, than when it is (just for example) extremely unreasonably hot out, the agent might do less computation, or use a computer to do computations for it. (An unbounded rational agent never uses a computer, since it can just simulate computers "in its head".)

In bounded rationality, (1) and (2) are different, and (2) isn't even well-defined (there might not be a policy that can always choose the best action, and even if there is that action might depend on the amount of computation that has been done).

So, we cannot say whether a given action is bounded rational, only whether the entire policy is bounded rational.

Example: Guessing a preimage

For example, consider the following problem: given a random 512 bits, guess how many es are in the SHA-3 preimage of those bits.

In the specific case of 3fbef366d350d95ec6ef948dfc2fd1ebd6f1de928c266d86e2ed8e408b2f7b30cad0e14714830369f96ad41c2b58da80bab3bff90a6770910244e28b4f9e80be, you might be tempted to say "32 es", since you know nothing about that string and 512/16 is 32.

However, a better guess is 10, since the preimage is b12ee4ed50edf33e4a388924f35f0190da54ee82ca0338d7dcc4adc3214da21e69d0b0b32789074fef3c05f02d6814da2c8d72057f50835d8f83265e6a4c3b57, which has exactly 10 es.

However, assuming reasonable computational limits, it is unlikely that the optimal policy says 10. b12ee4ed50edf33e4a388924f35f0190da54ee82ca0338d7dcc4adc3214da21e69d0b0b32789074fef3c05f02d6814da2c8d72057f50835d8f83265e6a4c3b57 is just a random string of bits I generated, and for any given random strings of bits, it is unlikely for any given policy (including the optimal policy) to have hard-coded it. There are simply too many possible inputs to hard-code...
Recent and forecasted rates of software and hardware progress
by elifland — 2025-06-26T22:37:27.074Z

I originally wrote this just for myself but thought I'd share it in case others find it useful. Sharing rough notes in the spirit of not letting perfect be the enemy of the good. This was written in early May 2025.

In this post I collect evidence and forecasts regarding recent and future trends in software and compute progress. I also make my own forecasts which inform an updated version of

AI 2027’s timelines forecast

(more to come on these and further updates). These forecasts are rough and uncertain.

I especially build on this post.

Evidence and forecasts regarding the recent rate of software progress
(and share vs. compute)

Evidence	Estimate	Caveats, reasoning, etc.

Compute efficiency increase at fixed performance

Epoch paper: Analyzing a dataset of LLM performance on Wikitext and Penn Treebank (

source

)

2.5-2.8x/year

Average efficiency increase reported in the paper (compute needed to reach same performance halving every 8-9 months)

Epoch’s analysis of trends in inference cost to achieve fixed performance level (

source

). Median rate of cost decrease is 40x/year, trends lasting 0.5-3 years.

The trends after Jan 2024 seem faster (but I think this might be due to trends generally being faster right after benchmark release).

<<1,600x/year

If cost were proportional to parameter count and all models are Chinchilla trained then this would imply compute efficiency increase of 40*40=1,600x/year. But it’s actually way less, see the next column for why.

This is obviously too large but still interesting.

Most of this is coming from (a) distillation, which we should disallow[1] (b) a

trend toward increased overtraining on smaller models

and (c) overfitting to benchmarks; in particular, some or all of the benchmarks might be more narrow than we’d like.

However even for MMLU there’s a 10x/year trend lasting 3 years, which is interesting.

GPT-4 vs. DeepSeek-V3 / R1 on MMLU etc. (source:

Ryan comment

,

Epoch data

)

>2.8x/year

GPT-4 released in Mar 2023

with 2e25 training compute, DeepSeek-V3 was released Dec 2024 with 3.4e24 training compute.

DeepSeek-V3 is significantly better at MMLU and other benchmarks. This corresponds to a >2.8x/year decrease.

Peter claims that this isn’t good to look at because GPT-4 isn’t trained compute optimally. It’s also possible that DeepSeek-V3 is disti

...
Too Many Definitions of Consciousness
by Commander Zander — 2025-06-26T22:22:50.287Z

I often hear people say consciousness is a phenomenon that urgently needs deeper understanding. Yet the word consciousness is a major source of the confusion. I'm trying to replace it with clearer words in my conversations.

Often, in both mainstream & rationalist contexts, i encounter people with these different definitions of the word:

Awakeness (but everyone agrees this is the boring one)
Thoughts about oneself
Thoughts about oneself AND the capability to think about complex situations
Attention routing
Subjective experience

People often also believe that one of the phenomena above is closely bound to one of these other phenomena:

Pleasure & pain
Emotions
Neurons (as opposed to simulated neurons)
Moral worth
Worthiness of respect

The problem is that you define consciousness as 1+ of the above, but your conversation partner has checked a different set of the checkboxes, so you talk past each other.

I suspect most people are okay with this vague terminology because they all share a belief in a Subtle Neural Superiority that distinguishes humans from plants, or perhaps from krill. It's okay for others to equate this SNS with the wrong phenomenon. It's okay if you think the invisible elephant is tusk-based rather than trunk-based - at least we agree there is a invisible elephant!

But i doubt we will ever find this invisible elephant, this Subtle Neural Superiority. We will one day fully understand the brain[1], but even then biophysics & psychology will reveal no clear evidence of a SNS. The distinction will always be a matter of pragmatic consensus, not of hard science.

So i agree that we need to research minds in detail, & i agree that we need to devise explicit, detailed policies for which phenomena to prioritize over others. I agree that it's reasonable for us to prioritize a human over a microscopic worm, but not because of a undiscovered biophysics or algorithmic property. I simply think it is reasonable to prioritize humanlike minds for the time being. This is a pragmatic policy, not a claim about hard science.

I intend to write more about my opinions on animal ethics & philosophy of mind, since i've recently discovered that they are novel to some people.

But you certainly don't have to agree with me about everything. All i request is that you try to use more specific words than consciousness. People don't know which definition/s you...