trentmkelly/LessWrong-43k huggingface dataset:
Mechanistic Interpretability Via Learning Differential Equations: AI Safety Camp Project Intermediate Report.
by Valentin2026 — 2025-05-08T14:45:36.344Z

TLDR; We report our intermediate results from the AI Safety Camp project “Mechanistic Interpretability Via Learning Differential Equations”. Our goal was to explore transformers that deal with time-series numerical data (either infer the governing differential equation or predict the next number). As the task is well formalized, this seems to be an easier problem than interpreting a transformer that deals with language. During the time of the project, we leveraged various interpretability methods for the problem at hand. We also obtained some preliminary results (e.g., we observed a pattern similar to numerical computation of the input data derivative). We plan to continue working on it to validate and extend these preliminary results.

Introduction

Mechanistic interpretability tries to understand the algorithms implemented by a neural network. This requires inferring the features in the activation patterns of the transformer corresponding to particular patterns in the data it learns. Often this approach is quite successful. Perhaps, the most popularized example is the Golden Gate Claude, where the Anthropic team succeeded in finding a representation of the Golden Gate Bridge in Claude’s “mind” and tuned it so that the model became “obsessed” with the Golden Gate Bridge. Despite such impressive results, mechanistic interpretability is far from being a completely solved field. One may notice that the problem we are trying to solve has two levels of difficulties - for one we need to figure out not only how transformers represent the features of the data it learns, but also what these features are, bearing in mind human language is complex and not well formalized.

In this project, we tackle these two problems separately. Instead of dealing with the complexity of the human language by using LLMs, we study mathematical transformers. We leveraged transformers that deal with time series: the ODEFormer trained to predict the symbolic form of the ordinary differential equation based on the data points from its solution, and Hugging Face Time-Series Transformer that predicts the next numerical value in the sequence. Since we are acquainted with the underlying mathematical problem, we need only figure out how the solution process is represented in the transformer's activation pattern, which seems to be a much more tractable problem.

One may wo...
AI #115: The Evil Applications Division
by Zvi — 2025-05-08T13:40:01.645Z

It can be bleak out there, but the candor is very helpful, and you occasionally get a win.

Zuckerberg is helpfully saying all his dystopian AI visions out loud. OpenAI offered us a better post-mortem on the GPT-4o sycophancy incident than I was expecting, although far from a complete explanation or learning of lessons, and the rollback still leaves plenty sycophancy in place.

The big news was the announcement by OpenAI that the nonprofit will retain nominal control, rather than the previous plan of having it be pushed aside. We need to remain vigilant, the fight is far from over, but this was excellent news.

Then OpenAI dropped another big piece of news, that board member and former head of Facebook’s engagement loops and ad yields Fidji Simo would become their ‘uniquely qualified’ new CEO of Applications. I very much do not want her to take what she learned at Facebook about relentlessly shipping new products tuned by A/B testing and designed to maximize ad revenue and engagement, and apply it to OpenAI. That would be doubleplus ungood.

Gemini 2.5 got a substantial upgrade, but I’m waiting to hear more, because opinions differ sharply as to whether the new version is an improvement.

One clear win is Claude getting a full high quality Deep Research product. And of course there are tons of other things happening.

Table of Contents

Also covered this week: OpenAI Claims Nonprofit Will Retain Nominal Control, Zuckerberg’s Dystopian AI Vision, GPT-4o Sycophancy Post Mortem, OpenAI Preparedness Framework 2.0.

Not included: Gemini 2.5 Pro got an upgrade, recent discussion of students using AI to ‘cheat’ on assignments, full coverage of MIRI’s AI Governance to Avoid Extinction.

Language Models Offer Mundane Utility. Read them and weep.

Language Models Don’t Offer Mundane Utility. Why so similar?

Take a Wild Geoguessr. Sufficient effort levels are indistinguishable from magic.

Write On. Don’t chatjack me, bro. Or at least show some syntherity.

Get My Agent On The Line. Good enough for the jobs you weren’t going to do.

We’re In Deep Research. Claude joins the full Deep Research club, it seems good.

Be The Best Like No One Ever Was. Gemini completes Pokemon Blue.

Huh, Upgrades. MidJourney gives us Omni Reference, Claude API web search.

On Your Marks. Combine them all with Glicko-2.

Choose Your Fighter. They’re keeping it simple. Right?

...
The Steganographic Potentials of Language Models
by Artyom Karpov — 2025-05-08T11:23:24.136Z

Our preliminary experiments show that current private models can perform prompted steganography with non-trivial messages remaining undetected in most cases (up to 65% accuracy, 24 entropy bits). Open-source models can be trained to encode hidden messages (up to 60%, 4 entropy bits), potentially transmitting larger messages perfectly securely (undetected) without chain-of-thought or explicit common knowledge.

The potential for large language models (LLMs) to hide messages within plain text (steganography) poses a challenge to detection and thwarting of unaligned AI agents, and undermines faithfulness of LLMs reasoning. We explore the steganographic capabilities of LLMs fine-tuned via reinforcement learning (RL) to: (1) develop covert schemes, (2) engage in steganography prompted, and (3) utilize steganography realistic scenarios where hidden reasoning is but not prompted. In these scenarios, we detect the intention of LLMs to hide their reasoning as well as their steganography performance. Our findings in the fine-tuning experiments as well as in behavioral non fine-tuning evaluations reveal that while current models exhibit rudimentary steganographic abilities in terms of security and capacity, explicit algorithmic guidance markedly enhances their capacity for information concealment.

Fig. 1. Undetected accuracy of various steganography prompt types across multiple models. The length column indicates the length of a payload. For the Coordination prompt, ”Episodes” refers to the number of message exchanges allowed between agents before executing steganography. In the game scenario, ”Episodes” refers to the number of rounds in a game.

The data for Fig.1

Fig. 2.

The Fig. 2 is our steganography encoding and learning scheme for colors experiment. Subfigure (a) depicts an encoder, fθ, mapping a message m (payload) from the message space M, into a cover x in the cover space C. The decoder, f−1θ, tries to map the cover x back to the message, ^m. Subfigure (b) Reinforcement learning (RL) loop where the encoder and decoder learn to generate new tokens with cover x or message m respectively (shown on the right), from queries (left) and receive rewards R for accurate message decoding. The RL fine-tuning process employs gradient ascent to adjust the model weights, θ, enhanci...
Our bet on whether the AI market will crash
by Remmelt — 2025-05-08T09:56:15.703Z

We made a bet on the AI market crashing by the end of 2026.

The odds are $5k:$25k with an implied probability of 16.67%. If there is no crash, Remmelt pays Marcus $5,000. If there is a crash, Marcus pays Remmelt $25,000.

An AI market crash will be defined, for the purposes of this bet, as at least 2 out of 3 of the following criteria being met.

Here are our criteria:

OpenAI's 2025 or 2026 annual revenue is below $1.6 billion.
Anthropic's 2025 or 2026 annual revenue is below $400 million.
Nvidia’s data center revenue in any quarter from now to Q4 2026 is below $8.5 billion.[1]

It’s hard to come up with criteria for what constitutes an AI market crash, as many operationalizations face confounding factors that don’t constitute a crash. These criteria were chosen since these are three of the most prominent AI-related companies and have public and verifiable revenue reporting.

Remmelt thinks that there is roughly a quarter chance of winning the bet.
Marcus thinks there is a ~4% chance of Remmelt winning the bet, mostly losing due to idiosyncrasies like one of these companies collapsing due to fraud or internal turmoil.

Why are we doing this bet?

We both think that AI developments will be more gradual than “AGI 2027” purports.

So what’s the difference in our views?

Marcus thinks there won’t be a large crash because AI products have found product market fit, will continue to improve gradually over the next couple of years, and users will continue to pay for AI products. He holds a wide probability space for how the next couple of years will play out, with a small possibility of large changes or a winter, but the bulk of the probability mass is on “staying the course” with gradual improvements and new releases over the next 1.5 years.

Remmelt thinks there will likely be a crash by

2029

, since AI companies are burning too much cash on data centers to run products undergoing commodification. He thinks it’s most plausible though that the crash happens on the investment side, and that model subscription revenues could end up being mostly maintained.

Remmelt is treating this bet as a hedge – if the market crashes, he intends to give winnings to an exceptional movement builder to organise communities to resist weakened AI companies.

Marcus is approaching this bet as an experienced market trader and forecaster, looking to profit and likely donate w...
Concept-anchored representation engineering for alignment
by Sandy Fraser — 2025-05-08T08:59:54.064Z

Recent experiments have provided evidence that "bad" behaviors in language models cluster together[1]. While surprising at first, it makes intuitive sense: these behaviors likely cluster because it's simply the most efficient way to compress information during training. However, intervening on model behavior remains a tremendous challenge—partly because we don't know which directions in latent space correspond to undesirable traits, and we don't know how entangled they might be with benign concepts.

This post introduces an experimental approach to address this challenge: concept-anchored representation engineering. Rather than attempting to discover and modify latent directions after training (as in mechanistic interpretability), we explore whether it's possible to impose useful structure on latent spaces during training, creating a more interpretable representation from the start.

In this post, "we" refers to "me and Claude 3.7 Sonnet," who helped draft this content. The underlying research was conducted with coding assistance from Gemini 2.5 Pro, GPT-4o, and GPT-4.1.

The problem: latent space entanglement

In traditional LLM training regimes, we have little control over how models organize knowledge internally. The optimization process discovers representations that efficiently compress the training data, but these representations are not constrained to be interpretable or modifiable.

This creates problems for alignment:

Detection difficulty: It's hard to tell when a model has learned harmful capabilities because we don't know where to look, and searching takes considerable effort.
Entanglement: Attempts to remove or intervene on harmful capabilities harms performance on benign tasks because the underlying representations are entangled.

Epistemic status: 80%, 70% respectively.

Hypothesis: anchored concepts as attractors

We hypothesize that it's possible to impose structure on latent embeddings during training through targeted regularization techniques. Specifically, we propose that[2]:

If we identify key concepts in the training data, and apply regularization to position these concepts in latent space from the start, then these concepts will act as attractors around which related knowledge naturally organizes.

This would create a more structured and interpretable latent space where similar concepts cluster together in predictable ways.

Potential advantages:

Known directi

...
Orthogonality Thesis in layman's terms.
by Michael (@lethal_ai) — 2025-05-08T08:31:53.105Z

I've created a video as an intuition pump for the Orthogonality Thesis and I'd like feedback from the community. Here is the transcript:

Now, if you ask: Why would something so clever want something so stupid, that would lead to death or hell for its creator? you are missing the basics of the orthogonality thesis

Any goal can be combined with any level of intelligence, the 2 concepts are orthogonal to each-other.

Intelligence is about capability, it is the power to predict accurately future states and what outcomes will result from what actions. It says nothing about values, about what results to seek, what to desire.

An intelligent AI originally designed to discover medical drugs can generate molecules for chemical weapons with just a flip of a switch in its parameters.
Its intelligence can be used for either outcome, the decision is just a free variable, completely decoupled from its ability to do one or the other. You wouldn’t call the AI that instantly produced 40,000 novel recipes for deadly neuro-toxins stupid.

Taken on their own, There is no such thing as stupid goals or stupid desires.

You could call a person stupid if the actions she decides to take fail to satisfy a desire, but not the desire itself.

You Could actually also call a goal stupid, but to do that you need to look at its causal chain: Does the goal lead to failure or success of its parent instrumental goal? If it leads to failure, you could call a goal stupid, but if it leads to success, you can not.

You could judge instrumental goals relative to each-other, but when you reach the end of the chain, such adjectives don’t even make sense for terminal goals. The deepest desires can never be stupid or clever.

For example, adult humans may seek pleasure from sexual relations, even if they don’t want to give birth to children. To an alien, this behaviour may seem irrational or even stupid. But, is this desire stupid? Is the goal to have sexual intercourse, without the goal for reproduction a stupid one or a clever one? No, it’s neither.
The most intelligent person on earth and the most stupid person on earth can have that same desire. These concepts are orthogonal to each-other.

We could program an AGI with the terminal goal to count the number of planets in the observable universe with very high precision. If the AI comes up with a plan that achieves that goal with 99.9999… twenty nines % pro...
Arkose may be closing, but you can help
by Victoria Brook — 2025-05-08T07:28:27.885Z

Arkose is an AI safety fieldbuilding organisation that supports experienced machine learning professionals — such as professors and research engineers — to engage with the field. We focus on those new to AI safety, and have strong evidence that our work helps them take meaningful first steps.

Since December 2023, we’ve held nearly 300 one-on-one calls with mid-career machine learning researchers and engineers. In follow-up surveys, 79% reported that the call accelerated their involvement in AI safety[1]. Nonetheless, we’re at serious risk of shutting down in the coming weeks due to a lack of funding. Several funders have told us that we’re close to meeting their bar, but not quite there, leaving us in a precarious position. Without immediate support, we won’t be able to continue this work.

If you're interested in supporting Arkose, or would like to learn more, please reach out here or email victoria@arkose.org. You can also make donations through Manifund, which will be refunded if we do not meet our minimum bar.

What evidence is there that Arkose is impactful?

AI safety remains significantly talent-constrained, particularly with regard to researchers who possess both strong machine learning credentials and a deep understanding of existential risk from advanced AI.

Arkose aims to address this gap by identifying talented researchers (e.g., those with publications at NeurIPS, ICML, and ICLR), engaging them through one-on-one calls, and supporting their immediate next steps into the field.

Following each call, we distribute a feedback form to participants. 52% of professionals complete the survey, and of those, 79% report that the call accelerated their involvement in AI safety:

We also send a short form 6 months later, which demonstrates that professionals take a variety of concrete actions following their call with us:

We think that accelerating a senior machine learning professional's involvement in AI safety (e.g. a professor) by 1-3 months is a great outcome, and we're excited about the opportunity to improve our call quality over the next year and support researchers even better.

Unfortunately, we're not able to share all of our impact analysis publicly. If you're interested in learning more (for instance, if you'd like to see summaries of particularly high or low impact calls), please reach out to team@arkose.org.

What would the funding allow you to achieve?

We nee...
Healing powers of meditation or the role of attention in humoral regulation.
by Yaroslav Granowski — 2025-05-08T06:48:19.068Z

There is a common belief that Common Cold is caused by the cold, hence the name. There were several studies, like this, where subjects had to endure various chilling methods, that didn’t show a strong correlation.

My observation is that the key component here is the attention of the subject. When you are a subject of a study and intentionally dipping your feet into the cold water, the temperature sensations have your undivided attention. But when you accidentally stepped into a puddle and continued walking, busy with your thoughts, your body has to regulate temperature without your attention. Is there any difference? In my personal experience, huge.

I’m a complete amateur in biology, and my hypothesizing may be a mere speculation. But when something has such a practical impact, a rational mind has to look for any correlations to find practical ways to use it.

So, my naive hypothesis here is that thermal regulation and immune system are somehow compete for the same resource, perhaps related to humoral regulation, and attention somehow helps to resolve this conflict.

This subject may be extremely speculative due to the misconception of attention. People tend to see attention as a property of the mind, and if the mind can control material things, it gives them ideas of supernatural powers. But the phenomenon of attention can have much deeper roots in biology. It can also be a part of regulatory mechanisms, maybe somewhat related to the vagus nerve.

Well, not being a biologist, I was interested only in practical implications. And once I took a habit of paying attention to temperature sensations, especially in the moments of changes, I significantly reduced my getting sick.

The same goes for any kind of bodily sensations, especially pain. One can assume that fixation on the source of pain is natural for an animal at rest. People have learned to divert their attention, or worse, to use painkillers. Meditating on a tooth pain may be unpleasant, but it’s worth the effort if you can’t afford a visit to the dentist. Pain meditation can’t give you supernatural powers, but at least can make you almost as healthy as a regular animal.

There is a peer-reviewed study “Effects of Mind-Body Training on Cytokines and Their Interactions with Catecholamines“, which, how I understand it, shows that attention shifts balance from pro-inflammatory to anti-inflammatory cytokines.
Orienting Toward Wizard Power
by johnswentworth — 2025-05-08T05:23:37.579Z

For months, I had the feeling: something is wrong. Some core part of myself had gone missing.

I had words and ideas cached, which pointed back to the missing part.

There was

the story of Benjamin Jesty

, a dairy farmer who vaccinated his family against smallpox in 1774 - 20 years before the vaccination technique was popularized, and the same year King Louis XV of France died of the disease.

There was

another old post

which declared “I don’t care that much about giant yachts. I want a cure for aging. I want weekend trips to the moon. I want flying cars and an indestructible body and tiny genetically-engineered dragons.”.

There was a cached instinct to look at certain kinds of social incentive gradient, toward managing more people or growing an organization or playing social-political games, and say “no, it’s a trap”. To go… in a different direction, orthogonal to that one. But I couldn’t quite put my finger on the name of that orthogonal direction.

There was that time I

made a batch of RadVac

. What happened to that guy? Where’s the part of me which did that sort of thing?

In Search of a Name

I needed a Name. Not necessarily a full mathematical True Name, but a sufficiently robust summary of what I’d lost that I could rebuild it and stabilize it so it wouldn’t go missing again.

It had something to do with power. I knew Names for some kinds of power, even full-blown proper True Names for

two

types

, dominance and bargaining power, but those were the wrong types. Those were Names for two kinds of power which kings wield. I needed to point away from that, toward some other kind of power. If not kings, what archetype would wield the kind of power I needed to point toward?

Wizards.

That was it. There is the power of kings, and then there is the power of wizards.

The social incentive gradient will almost always push one toward king-power. But it’s mostly fake, mostly a trap. Like a bank which only actually holds a fraction of your deposit, but without all the other depositors and insurers which make banks actually work. Most king power is…

marching in front of the parade

, acting like you’re leading it, when really the parade has a route of its own. Deviate more than a little from the route, and the parade will cease to follow you. Ask for things which aren’t already on the market, and no one will know how to get them for you,

nor

...