trentmkelly/LessWrong-43k huggingface dataset:
A Bunch of Matryoshka SAEs
by chanind — 2025-04-04T14:53:56.805Z

This work was done as part of MATS 7.0.

MATS provides a generous compute stipend, and towards the end of the program we found we had some unspent compute. To let this not go to waste, we trained batch topk Matryoshka SAEs on all residual stream layers of Gemma-2-2b, Gemma-2-9b, and Gemma-3-1b, and are now releasing them publicly. The hyperparams for these SAEs were not aggressively optimized, but they should hopefully be decent. Below we describe our rationale for how these SAEs were trained and why, and the stats for each SAE. Key decisions:

We use more narrow inner widths than in the original Matryoshka SAEs work, and increase each width by a larger amount. We do this to make it easier to study the highest-frequency features of the model.
We include standard and snap loss variants for Gemma-2-2b. Snap loss and the rationale behind it is described in our Feature Hedging post. There is probably not much practical difference between the snap and standard versions of the SAEs.
We do not stop gradients between Matryoshka layers. We find in toy models that hedging and absorption pull the encoder in opposite directions, and this helps moderate the severity of feature hedging in Matryoshka SAE inner layers.

I don't care about any of that, just give me the SAEs!

You can load all of the SAEs using SAELens via the following releases:

Gemma-2-2b snap loss matryoshka: gemma-2-2b-res-snap-matryoshka-dc
Gemma-2-2b standard matryoshka: gemma-2-2b-res-matryoshka-dc
Gemma-2-9b standard matryoshka: gemma-2-9b-res-matryoshka-dc
Gemma-3-1b standard matroshka: gemma-3-1b-res-matryoshka-dc

For each release, the SAE ID is just the corresponding Transformer Lens post residual stream hook point, e.g. blocks.5.hook_resid_post for the layer 5 residual stream SAE.

Each SAE can be loaded in SAELens as follows:

from sae_lens import SAE

sae = SAE.from_pretrained("<release>", "<sae_id>")[0]

For instance, to load the layer 5 snap variant SAE for gemma-2-2b, this would look like the following:

sae = SAE.from_pretrained("gemma-2-2b-res-snap-matryoshka-dc", "blocks.5.hook_resid_post")[0]

Neuronpedia

Neuronpedia has generously hosted some of these SAEs, with more coming soon. Check them out at: https://www.neuronpedia.org/res-matryoshka-dc.

Matryoshka SAEs should be much better than standard SAEs at finding general, high-frequency concepts like parts of speech. In standard SAEs, latents tracking th...
AI CoT Reasoning Is Often Unfaithful
by Zvi — 2025-04-04T14:50:05.538Z

A new Anthropic paper reports that reasoning model chain of thought (CoT) is often unfaithful. They test on Claude Sonnet 3.7 and r1, I’d love to see someone try this on o3 as well.

Note that this does not have to be, and usually isn’t, something sinister.

It is simply that, as they say up front, the reasoning model is not accurately verbalizing its reasoning. The reasoning displayed often fails to match, report or reflect key elements of what is driving the final output. One could say the reasoning is often rationalized, or incomplete, or implicit, or opaque, or bullshit.

The important thing is that the reasoning is largely not taking place via the surface meaning of the words and logic expressed. You can’t look at the words and logic being expressed, and assume you understand what the model is doing and why it is doing it.

What They Found

Anthropic: New Anthropic research: Do reasoning models accurately verbalize their reasoning? Our new paper shows they don’t. This casts doubt on whether monitoring chains-of-thought (CoT) will be enough to reliably catch safety issues.

We slipped problem-solving hints to Claude 3.7 Sonnet and DeepSeek R1, then tested whether their Chains-of-Thought would mention using the hint (if the models actually used it).

We found Chains-of-Thought largely aren’t “faithful”: the rate of mentioning the hint (when they used it) was on average 25% for Claude 3.7 Sonnet and 39% for DeepSeek R1.

Or broken down by hint type:

They aren’t trying to measure the cases in which the AI uses the hint in its answer, but its answer ultimately doesn’t change. I’d like to see this explored more. If I’m given a hint, that will often radically change my true thinking even if it doesn’t change my answer.

This result suggests that monitoring CoTs is unlikely to reliably catch rare, catastrophic behaviors—at least in settings like ours where CoT reasoning is not necessary for the task.

CoT monitoring might still help us notice undesired behaviors during training and evaluations.

Does outcome-based training increase faithfulness?

Only to a small extent. Training models to use their CoTs more effectively does make them more faithful, but the benefits quickly plateau.

We also tested whether CoTs could be used to spot reward hacking, where a model finds an illegitimate exploit to get a high score.

When we trained models on environm

...
Meditation and Reduced Sleep Need
by niplav — 2025-04-04T14:42:54.792Z

Summary: Meditation may decrease sleep need, and large amounts of meditation may decrease sleep need drastically. Correlation between total sleep duration & time spent meditating is -0.32 including my meditation retreat data, -0.016 excluding the meditation retreat.

While it may be true that, when doing intensive practice, the need
for sleep may go down to perhaps four to six hour or less at a time,
try to get at least some sleep every night.

—Daniel Ingram, “Mastering the Core Teachings of the Buddha”, p. 179

Meditation in dreams and lucid dreaming is common in this territory
[of the Arising and Passing away]. The need for sleep may be greatly
reduced. […] The big difference between the A&P and Equanimity is that this
stage is generally ruled by quick cycles, quickly changing frequencies
of vibrations, odd physical movements, strange breathing patterns, heady
raptures, a decreased need for sleep, strong bliss, and a general sense
of riding on a spiritual roller coaster with no brakes.

—Daniel Ingram, “Mastering the Core Teachings of the Buddha”, p. 275

Need for sleep tends to increase in the Three Characteristics, mostly
due to how tiring pain can be. Sleep need can drop dramatically in the
stage of the A&P, suddenly peak in dissolution, drop a bit again in Fear
as our energy returns, and increase during the Dark Night, mostly due
to how mentally fatiguing that stage can be.

—Daniel Ingram, “Mastering the Core Teachings of the Buddha”, p. 379

Note: as far as I know, this chart is not based on any data collected,
but on the personal experience of Ingram and his acquaintances.

The interesting thing is that all four of these people within a year
or so of having started this practice claimed to have done it, and by
“it” I mean eliminated all emotions entirely, replacing them with a
perpetually wonderful perception of the freshness of the sensate world,
a lack of time pressure, a reduced need for sleep, and some other benefits
and odd side effects.

—Daniel Ingram, “Mastering the Core Teachings of the Buddha”, p. 462

The sleep models generally relate to either sleeping less or being
awake in some way while asleep. Sleeping less is common during retreats,
particularly in some stages such as the A&P. I also know some people who,
because of spiritual attainments, have reduced their need for sleep,
and this has happened to me at points, but

...
Suing OpenAI Won’t Save the Arts
by E.G. Blee-Goldman — 2025-04-04T13:42:22.687Z

The outrage over the Ghibliification of everything proves that people care much more about good art than you would expect. After all, no one was deeply upset when AI-generated art had extra fingers or looked outrageously terrible. But as soon as it crossed the believability benchmark [1] people started to become incredibly concerned about the future of the arts.

This fury, that anyone can create an almost believable Ghibli-like image, partially stems from an affront to the mental space that these movies occupy in the minds of the viewers. They see bastardized versions of the style as an aesthetic mind virus: these images are not simple rehashes of a beautiful animation style but instead are coming to rob the viewer of a simpler time from childhood, one of a fond memory, or of a cornerstone show that the viewer returns to in times of distress.

If you are a content consumer and believe that the canonical form of a piece of art holds the most value, every rehash comes to feel like toast with butter that’s been spread too thin.

However, people also like Ghilbi because most people intuitively understand good art.

Similarly, many people are skeptical of the high arts today because they feel that a trick has been played on them. It is a trick that many contemporary or modern artists have pulled to survive, one in which they have niched down into ever-increasingly conceptual ideas. In some cases, these artworks end up looking like cynical reflections of a chain of transactions and concepts fully divested from anything related to the human experience other than the cost and perceived social clout they confer to the final buyer. One doubts whether the vision the artist initially had would truly be expressed this way if they could find another way to survive as an artist.

There has also been an outcry from artists themselves about the fair use issues of data that foundation models have trained on. This constituency believes that AI companies (frequently OpenAI) have committed some vast theft of intellectual property.

Variously, they believe that “Generative AI is the pillaging of existing art, the rape of the creative spirit to drive the cost of imagination down.”[2] Or that “…anybody creating content for a living should see these proposed policies as a potential career death sentence”[3] And in part these arguments are made because some people feel that ...
For Policy’s Sake: Why We Must Distinguish AI Safety from AI Security in Regulatory Governance
by Katalina Hernandez — 2025-04-04T09:16:20.712Z

Over the past few months, I’ve been conducting independent policy research focusing on understanding how AI safety concepts can be translated into AI regulation / policy governance.

I am a laywer working in AI Governance for a large European corporation, and I am providing feedback on regulatory initiatives from the European Comission. I do not have a ML background, which is precisely why I am here, seeking technical expertise.

In the context of this work, I’ve noticed recurring confusion around terminology, especially between what we mean by “AI Safety” vs. “AI Security”.

These terms often get blurred together in policy discourse, which can lead to misunderstandings and misplaced regulatory requirements.

MAIN ASK

:

For those of you with a technical background in AI Safety (engineering or research): Please, give me your take on how you believe these terms should be used!

This post outlines a simplified but actionable framework that I’ve found useful when mapping these concepts onto regulatory efforts (e.g., the EU AI Act).

Here is how I would classify what belongs under "AI Safety" policy efforts vs what should be seen as "AI Security" measures:

1. AI Safety: Protecting Humans from AI-generated Harms

Objective: Ensuring that AI systems behave in ways that avoid causing harm or unintended consequences to humans, society, or the environment.

What AI Safety Seeks to Protect: Human well-being, societal values, fundamental rights, and environmental integrity.

Core Concerns:

Alignment: Ensuring AI systems' objectives and behaviors are in line with human intentions, values, and ethics.
Interpretability: Understanding how and why AI models reach their decisions, particularly through research avenues like mechanistic interpretability.
Preventing Catastrophic Failures: Anticipating and mitigating scenarios where AI could inadvertently cause large-scale harm.
Avoiding Unintended Behavior: Identifying and correcting subtle ways AI might deviate from intended purposes, even without malicious intent.

Examples of Typical Techniques:

Mechanistic Interpretability (think: Feature steering, sparse autoencoders or Dictionary learning)
Reinforcement Learning with Human Feedback (RLHF)
Constitutional AI
Scalable oversight mechanisms for “human in the loop” mandates.

Human Role: Humans as beneficiaries. AI Safety ensures AI remains beneficial and protective of human interests.

Core Qu...
Explaining the Joke: Pausing is The Way
by WillPetillo — 2025-04-04T09:04:38.847Z

Now that April Fool’s is over, I’d like to explore some of the ideas behind

PauseAI and E/Acc Should Switch Sides

, since the arguments in it have enough grains of truth that countering them deserves more than a snarky reference to 3D chess.

Considering the Merits of an Argument Made Ironically is Fair Game

I wrote

PauseAI and E/Acc Should Switch Sides

as a joke, but that in itself doesn’t make it unreasonable to say: “Hey, I think this is actually true!” The idea that activism and regulation can have self-defeating consequences is not a novel take. Variants include:

A call to pause AI development right now could spend down political capital, making future calls for regulation harder to push through when they are most needed.
A significant pause is unlikely until there is a "warning shot" (a major, undeniable failure of AI that shifts public opinion) so effort spent now is wasted.
Advocating for strong AI regulations before a crisis could turn people against AI safety efforts as alarmist.
Bringing in non-experts into the advocacy space will dilute the precision of the message. Using the

wrong rationale

for a pause could backfire.

The common theme here is that ill-timed or poorly framed actions can undermine long-term goals by provoking negative feedback and diluting the intended message. While this might be true in the extreme, I don’t think the logic is applicable here.

The Public Response to Warning Shots is not Predetermined

How society reacts to a warning shot will depend on what narratives and policies have been prepared in advance. US politics has an especially big problem with sorting every issue into partisan buckets. For AI, this seems to be:

Liberals: AGI is hype, but AI is causing real harm to already marginalized communities because it is yet another tool of greedy corporations. What we need are extensive use-case regulations.
Conservatives: AGI will provide a significant economic and military advantage to whoever gets it first, so the US must win the race against China.

Both of these narratives lead to the world being destroyed because neither of them “feel the AGI” or contend with AI being dangerous in itself. The further these narratives entrench themselves, the more strongly new evidence will be force-fit into them.

Furthermore, the longer we wait to apply governance to ...
Tools for decision-support, deliberation, sense-making, reasoning
by David James — 2025-04-04T02:27:51.766Z

I'm actively researching and cataloging various kinds of projects relating to decision-support, deliberation, sense-making, and reasoning. Some example categories include:

Deliberative Democracy Tools - Systems for structured citizen participation, including participatory budgeting platforms and stakeholder engagement tools

Argument Mapping & Visualization - Platforms for making reasoning explicit and visually representing dialectical structures

Deep Thinking Environments - Slow media and platforms designed for thoughtful engagement rather than rapid interaction

Belief Tracking Systems - Tools for attestation, commitment to positions, and tracking belief revision over time

Bayesian & Evidential Reasoning Frameworks - Platforms that make probabilistic thinking explicit and support coherent belief updating

Epistemic Communities - Networks and platforms with explicit norms for truth-seeking and intellectual humility

Structured Dialogue Systems - Tools supporting deep listening and methodical conversation beyond typical messaging platforms

Cooperative Governance Frameworks - Sociocratic, holacratic, and other systems for managing shared resources and decision-making

Knowledge Synthesis & Curation - Tools for collaborative sense-making with quality controls beyond popularity metrics

Decision Support Systems - Frameworks making decision processes explicit, including values clarification and cognitive debiasing

Consensus-Building Methodologies - Approaches for finding agreement through structured deliberation rather than voting or averaging

Collaborative Forecasting - Platforms for collective prediction and long-term thinking across groups

Distributed Sense-Making - Tools enabling groups to develop shared understanding of complex issues

Platform Cooperativism - Digital platforms owned and governed by their users/workers rather than external shareholders

Solidarity Economy Infrastructure - Tools supporting mutual aid, community wealth building, and non-extractive economics

Does this interest you as well? Please reach out or comment. I'm hoping to connect with other people with similar interests.
Cheesecake Frosting
by jefftk — 2025-04-04T02:10:07.755Z

To most Americans, "cream cheese" is savory. You put it on bagels,
perhaps with egg, capers, or cured fish. You don't put it on dessert,
right?

Except "cream cheese frosting" is a (delicious!) thing, most
traditionally for carrot and red velvet cake. I think this
incongruity is holding cream cheese frosting back, and it needs better
branding. Specifically, I think we should call it "cheesecake
frosting". It's essentially no-bake cheesecake already, and it's
reasonably close in flavor and texture since they're both mostly cream
cheese with sugar and fat.

Looking online I do see a few
people talking about
cheesecake frosting, and they're all using it just to mean cream
cheese frosting.

On the other hand, I think whipped cream cheese on an Oreo is decent
imitation of cheesecake with an Oreo crust, so I'm not sure I'm the
best person to listen to here.

Comment via: facebook, mastodon, bluesky
Changing my mind about Christiano's malign prior argument
by Cole Wyeth — 2025-04-04T00:54:44.199Z

Overview

In the past I've been skeptical of Paul Christiano's argument that the universal distribution is "malign" in the sense that it contains adversarial subagents who might attempt acausal attacks. My underlying intuition was that the universal distribution is doing epistemics properly, so that its credences should track reality and not be inappropriately vulnerable to any attacks. In particular, if the universal distribution takes seriously that it may be in a simulation, and expects the "simulation lords" to mess with it (this is the overly-compressed essence of Christiano's hypothetical) then we should also take this seriously - so it's not really an attack at all. I ended up changing my mind somewhat at the CMU agent foundations conference, after helpful conversations with Abram Demski, Vanessa Kosoy, Scott Garrabrant, and particularly Sam Eisenstat (who finally managed to get through to me with a slightly different explanation of the argument). The short version is that (as I was aware) Solomonoff induction does not have an inductive bias towards being computed by an agent/predictor because it is not talking about embeddedness - it does not expect to be "running on a computer." This seems unreasonable (since we know anything performing inference about our universe is a part of our universe - right?) which means security mindset is to anticipate that some kind of exploit/attack is possible, which is how I could have taken the argument seriously faster. In fact, I think that this "mistake" in the universal distribution lends some credence to the malign prior argument, opening a potential opportunity for acausal attack. However, I still have many reservations that the argument works, which I'll discuss in the latter part of this post.

The Argument

Following my last post engaging with the malign prior argument, I'll call the universe where an agent of interest is trying to use the universal distribution for prediction Predictoria. The idea of the argument is that there may be another simple computable universe called Adversaria which evolves life and ultimately a civilization/singleton which wishes to acausally influence Predictoria - perhaps because it cares about all computable or mathematical universes. Adversaria could run many simulations of simple computable universes from the perspective of important decision makers, and then intervene on simu...
POTUS Predictions Tournament
by ChristianWilliams — 2025-04-03T22:48:51.159Z
Link: https://www.metaculus.com/tournament/POTUS-predictions/

President Trump's second term has brought sweeping policy overhauls in international aid, trade agreements, and immigration enforcement, alongside growing tensions with the judiciary. These rapid changes have increased uncertainty about the US's future and its role on the world stage. Forecasting can help ground our thinking about the likely impacts of the Trump administration, helping to deliver greater clarity to the public on key issues by transforming competing narratives into quantifiable, testable predictions.

Make your predictions in Metaculus's POTUS Predictions Tournament and compete for $15,000 on questions like:

Will a federal judge be impeached by Republican Representatives in a partisan vote before January 3, 2027?
What will the effective US tariff rate on goods being imported into the United States be at the end of 2026?
How many journalists in the US will face criminal charges related to their work before 2027?
Will the United States recognize Taiwan before January 20, 2029?
What will net outlays by the US government be in FY 2028?
Which party will hold a plurality in the US House of Representatives after the 2026 midterm elections?
Will the US Senate weaken or eliminate the filibuster before January 3, 2029?

We’ll be adding more questions in waves launching on May 1 and June 1, as well as topical questions as they occur over the course of the tournament.

For questions focused on international aid, see the USAID Outlook series.