trentmkelly/LessWrong-43k huggingface dataset:
A Disciplined Way to Avoid Wireheading
by amitlevy49 — 2025-05-07T15:20:27.893Z
Link: https://ivy0.substack.com/p/a-disciplined-way-to-avoid-wireheading

This is a crosspost of a post from my blog, Metal Ivy. The original is here: A Disciplined Way To Avoid Wireheading.

What is wireheading?

The term wireheading comes from the observation that if you put a wire in someone’s brain such that it is near their brain’s pleasure (or craving) center, and give them a lever that when pressed activates the current through the wire, they will repeat that action forever, to the point of dying from starvation. Conditional on them trying it once, of course. These experiments have been performed on mice, dolphin, monkeys, and in the distant unethical annals of the 1970s, humans.

A happy mouse?

Personally, though I am not alone in this view, I see wireheading not as an independent phenomenon, but instead the edge of a spectrum. These neurons were not created by evolution for some other task that is being tricked by wireheading to cause cravings to pull a lever. They were evolved precisely to give you a craving to perform an action to get dopamine. It just didn’t want you to do it quite so directly (if you oppose the framing of “goals” for evolution, take it as a rhetorical device, or call it Moloch[1]).

Evolution has many pathways for getting you to do what it wants. The craving mechanism is one of them, but you also have a more stable sense of utility — that sense tells you that, actually, being addicted to pulling a lever to directly stimulate your brain is “bad”. You want to get pleasure in a less direct, more natural way. You don’t want to trick your own brain. The craving and the perceived, externalized utility contradict, especially if you were to reflect on the subject before the first pull of the lever. This article will take the stance that this sense of utility is important, as an axiom, since we find it to be important by its very definition. We do not want it to contradict with our actions, which means we don’t want to stimulate our dopamine secreting neurons so directly.

On the possibility of just not inserting a wire into your brain

You can thank Claude for the visual flair.

Directness is a spectrum, trickiness is a sliding scale and something being natural or not is not well defined. Fentanyl is not as direct as wireheading, but it’s not all that different either. TikTok is further to the left, in this sense. But it can be placed on this same spectrum. The typical activities that you find rewarding in both senses belon...
Reflections on Compatibilism, Ontological Translations, and the Artificial Divine
by Mahdi Complex — 2025-05-07T12:16:30.220Z

This essay explores the concept of compatibilism beyond its traditional domain of free will, proposing a broader methodological approach I call 'Transontological Compatibilism.' I argue that this approach, identifying functional analogies across seemingly incompatible ontologies, can help us navigate philosophical, cultural, and linguistic divides. Ultimately, I will examine whether it can help us rethink our language and conceptual frameworks surrounding transformative artificial intelligence, perhaps even justifying a cautious embrace of religious vocabulary.

1. Compatibilism and Free Will

Believers in libertarian free will consider choices to be ontologically fundamental to reality and not entirely bound by the traditional deterministic chain of cause and effect. This view implies a mind-body dualism in which an immaterial and irreducible 'self' is the source of "choices." These choices are thought to be the product of a special kind of freedom that allows for a unique capacity for self-determination.

In this ontology, a choice is a cause, uncaused by any cause outside of this 'self', making this 'self' ultimately responsible for the choice. Central to this view is the idea that, if one were to go back in time, there would be a meaningful sense in which one could have chosen differently, regardless of the identical material circumstances.

This is a subjectively parsimonious ontology that jibes well with our experience of choice, responsibility and regret. I suspect the majority of people, at one time or another, perceive reality through the lens of this ontological framework. But, since physics tells us that we are in a physical, causally closed universe, obeying inviolable natural laws, this ontology cannot be true. If all our choices are determined by material causes ultimately outside of our control, how could we be said to be "free" to choose differently? When faced with this dilemma, people seem to adopt one of two strategies.

The first one, adopted by free will anti-realists, rejects the notion of free will entirely. If the old understanding of "free will" isn't compatible with a correct physical ontology, why keep it around in the first place? Free will, in that conceptualization, is nothing more than an illusion. Anti-realists argue that our subjective experience of making choices is not evidence of libertarian free will, but rather an emergent prope...
The Historical Parallels: Preliminary Reflection
by EQ — 2025-05-07T08:06:38.324Z
Link: https://eqmind.substack.com/p/the-historical-parallels-preliminary

Despite the unique features of advanced AI systems, a thorough study of historical examples from emergent technologies offers insight into navigation between the risks and benefits.

The rapid advances in the AI development have left the field mostly devoid of large-scale studies of the historical and sociological lessons from earlier similar developments. While AI - both in potential benefits and risks - is in some forms unique, it shares similarities with earlier major technological breakthroughs. While some of these have been studied intensely,[1] there remains work to do in building a coherent overview of the shifting dynamics new technologies cause.

A new technological breakthrough is a disruption in the fabric of society, originally locally, inevitably later globally. The discussion around AI safety is dominated by professionals of technological perspective. A good researcher, though, might not be a good strategist,[2] and dismissing the historical lessons is a blind spot we might not afford.

Studying the transitional periods of major technological breakthroughs and their effect on societal dynamics will arguably reveal recognizable patterns. Recognizing and understanding these patterns inevitably makes us better prepared for good governance solutions for the age of artificial intelligence.

I present in this post a research agenda, split into four separate themes of reactive patterns. This text is a map of the territory rather than exploration of each area in depth, providing a backbone for further research. Each of these themes warrants a study on its own. The historical cases of each theme represented should be studied together to form an understanding of the high-level patterns, not as isolated historical cases.

The themes are:[3]

Regulatory lag and warning shots
Conflict dynamics and emergent technology
Technological disruption’s effect on civil society
International agreements on technological restrictions

1. Regulatory lag and warning shots

Warning shots in the context of technology are situations where a (new) technology creates disvalue through some event(s), but less than it has potential to create. Thus the event moves the Overton window, sparks public discussion and gets policy-makers on the move. While we may not get a warning shot with catastrophically dangerous AI, it is potentially highly important to understand the dynamics and time...
European Links (07.05.25)
by Martin Sustrik — 2025-05-07T04:20:01.789Z

Military logistics

Politico writes:

The French army painfully realized how difficult it was to cross Europe in the spring of 2022, when it deployed a battalion to Romania in response to Russia's invasion of Ukraine.

[…]

"We discovered the extent of the administrative red tape. There's a war in Ukraine, but customs officials explain that you don't have the right tonnage per axle and that your tanks aren't allowed to cross Germany," Toujouse recalled. "It's just unbelievable."

It’s time for the Portuguese to start getting their permits now, so that they can to get to the spot when Russia attacks Estonia in 2030.

Cypriots against Erdoğan

In Turkey-occupied Northern Cyprus, The Guardian writes, Turkish Cypriots stepped up demonstrations against a law allowing headscarves to be worn in schools:

Turkey’s president, Recep Tayyip Erdoğan, has warned protesters in breakaway northern Cyprus not “to sow seeds of hatred” amid mounting discord over Ankara’s perceived attempts to Islamise one of the world’s most secular Muslim societies.

[…]

Friday’s demonstration, which followed almost daily protests over the hijab law, was organised by more than 100 trade unions and civil societies many of which still advocate the island’s reunification as a bi-zonal, bi-communal federation.

“Partly because of Kemalism but also because of eighty-two years of British colonial rule, Turkish Cypriots are by far the most secular Muslims in the world,” said Hubert Faustmann, professor of history and political science at the University of Nicosia in the internationally recognised south.

For Turkish Cypriots who have long opposed Ankara’s ever-expanding influence in the north, the regulation, he said, was further proof of the leader’s determination to not only erode long-held secular traditions but ultimately alter their own identity.

“What we are witnessing is a cultural clash,” Faustmann said. “The legislation on headscarves is seen as part of a package of continuous attempts by Erdoğan to unwind the secular character of the community.”

[…]

Increasingly, he lamented, Turkish Cypriots had been made to feel like a minority “in our own land” as a result of hundreds of thousands of mainland settlers moving to the north.

Some context from Wikipedia:

On 1 May 2004 Cyprus joined the European Union, together with nine other countries. Cyprus was accepted into the EU as a whole, although the EU legislation is suspend

...
Chess - "Elo" of random play?
by Shankar Sivarajan — 2025-05-07T02:18:18.498Z

I'm interested in a measure of chess-playing ability that doesn't depend on human players, and while perfect play would be the ideal reference, as long as chess remains unsolved, the other end of the spectrum, the engine whose algorithm is "list all legal moves and uniformly at random pick one of them," seems the natural choice.

I read that the formula for Elo rating E is scaled so that, with some assumptions of transitivity of winning odds, pvictory≈11+10ΔE/400, so it's trivial to convert probability to Elo rating, and my question is roughly equivalent to "What is the probability of victory of random play against, say, Stockfish 17?" If the Elo is close to 0[1], that makes the probability around 10−9 (estimating Stockfish 17's Elo to be 3600). Eyeballing the y-intercept of this plot of lc0's Elo vs. number of games of self-play, it looks something like 150–300 (lots of uncertainty). Does that sound reasonable?

I understand the probability of victory against the best modern engines is probably too small to accurately measure directly, so you would have to construct weaker/noisier engines and step down in increments of a few hundred Elo. Has anyone done this?

^

I don't see an a priori reason to expect it would be.
$500 + $500 Bounty Problem: Does An (Approximately) Deterministic Maximal Redund Always Exist?
by johnswentworth — 2025-05-06T23:05:32.243Z

A lot of our work involves "redunds".[1] A random variable Γ is a(n exact) redund over two random variables X1,X2 exactly when both

X1→X2→Γ

X2→X1→Γ

Conceptually, these two diagrams say that X1 gives exactly the same information about Γ as all of X, and X2 gives exactly the same information about Γ as all of X; whatever information X contains about Γ is redundantly represented in X1 and X2. Unpacking the diagrammatic notation and simplifying a little, the diagrams say P[Γ|X1]=P[Γ|X2]=P[Γ|X] for all X such that P[X]>0.

The exact redundancy conditions are too restrictive to be of much practical relevance, but we are more interested in approximate redunds. Approximate redunds are defined by approximate versions of the same two diagrams:

Unpacking the diagrammatic notation, these two diagrams say

ϵred≥DKL(P[X,Γ]||P[X1]P[X2|X1]P[Γ|X2])

ϵred≥DKL(P[X,Γ]||P[X2]P[X1|X2]P[Γ|X1])

This bounty problem is about the existence of a(n approximate) maximal redund Ω: a redund which contains (approximately) all the information about X contained in any other (approximate) redund. Diagrammatically, a maximal redund Ω satisfies:

Finally, we'd like our maximal redund Ω to be an approximately deterministic function of X, i.e. ϵdet≥H(Ω|X), which is equivalent to the diagram

What We Want For The Bounty

This bounty pays out $500 for establishing that there always, over any two random variables X1, X2, exists an Ω satisfying the requirements above, i.e.

... with reasonable error bounds.

What counts as reasonable error bounds? We're expecting all the other ϵ's to scale with ϵ2 in the diagrams above, and ideally be small constant multiples of ϵ2. In other words: the higher the approximation error on allowable redunds Γ, the more approximation error we allow for all the conditions on Ω. Reasonableness of error bound is partially a judgement call on our part, but small constant (like e.g. 2 or 3) multiples of ϵ2 would definitely suffice.

Bounds should be global, i.e. apply even when ϵ2 is large. We're not interested in e.g. first or second order approximations for small ϵ2 unless they provably apply globally.

If the error bounds are sufficiently...
The Theory Behind Loss Curves
by James Camacho — 2025-05-06T22:22:16.359Z
Link: https://github.com/programjames/theoretical-loss-curve

Or, why GAN training looks so funky.

Solomonoff's Lightsaber

The simplest explanation is exponentially more important.

Suppose I give you a pattern, and ask you to explain what is going on:

1,2,4,8,16,…

Several explanations might come to mind:

"The powers of two,"
"Moser's circle problem,"
x424−x312+11x224+7x12+1,
"The counting numbers in an alien script,"
"Fine-structure constants."

Some of these explanations are better than others, but they could all be the "correct" one. Rather than taking one underlying truth, we should assign a weight to each explanation, with the ones more likely to produce the pattern we see getting a heavier weight:

Grand Unified Explanation=∑wexplanation⋅explanation.

Now, what exactly is meant by the word "explanation"? Between humans, our explanations are usually verbal signals or written symbols, with a brain to interpret the meaning. If we want more precision, we can program an explanation into a computer, e.g. fn pattern(n) {2^n}. If we are training a neural network, an explanation describes what region of weight-space produces the pattern we're looking for, with a few error-correction bits since the neural network is imperfect. See, for example, the paper "ARC-AGI Without Pretraining" (Liao & Gu).

Let's take the view that explanations are simply a string of bits, and our interpreter does the rest of the work to turn it into words, programs, or neural networks. This means there are exactly 2n n-bit explanations, and the average weight for each of them is less than 1/2n. Now, most explanations—even the short ones—have hardly any weight, but there are still exponentially more longer explanations that are "good"[1]. This means, if we take the most prominent n explanations, we would expect the remaining explanations to have weight on the order of exp(−n).

Counting Explanations

What you can count, you can measure.

Suppose we are training a neural network, and we want to count how many explanations it has learned. Empirically, we know the loss comes from all the missing explanations, so

Loss∼exp(−explanations)⟺explanations∼−log(Loss).

However, wouldn't it be more useful to go the other way? To estimate the loss, at the beginning of a training run, by counting how many concepts we expect our neural network to learn? That is our goal for today.

If we assume our optimizers are perfect, we should be able to use every bit of ...
Negative Results on Group SAEs
by Josh Engels — 2025-05-06T21:49:15.123Z

Introduction

Soon after we released

Not All Language Model Features Are One-Dimensionally Linear

, I started working with @Logan Riggs and @Jannik Brinkmann on a natural followup to the paper: could we build a variant of SAEs that could find multi-dimensional features directly, instead of needing to cluster SAE latents post-hoc like we did in the paper.

We worked on this for a few months last summer and tried a bunch of things. Unfortunately, none of our results were that compelling, and eventually our interest in the project died down and we didn’t publish our (mostly negative) results. Recently, multiple people (@Noa Nabeshima , @chanind, Goncalo Paulo) said they were interested in working on SAEs that could find multi-dimensional features, so I decided I would write up what we tried.

At this point the results are almost a year old, but I think the overall narrative should still be correct. This document contains my results from the project; Logan had some additional results about circuits, and Jannik tried some additional things with learned groups, but I am not confident I can describe those as well so long afterwards.

With the benefit of hindsight, I have some concerns with this research direction:

It’s not clear if dictionary learning is the right tool for the job. There might be many more feature manifolds than there are concepts. For example, take the embedding space; an SAE with num_tokens latents will always get perfect reconstruction with k = 1, but there are likely many more than num_tokens multi-d structures in the embedding (e.g. days of the week looks circle-ish in the embedding).

We might need some circuits based approach, which is what Logan was excited about and working on.

It’s not entirely clear how to measure success. As we will see below, sometimes we find groups of semantically similar SAE latents, but are these really multi-dimensional features or just correlated 1D features? We have some ground truth from the paper (e.g. circular features), but we only have a few of these.
There are strong baselines that might be hard to beat, namely

Matryoshka SAEs

and various types of clustering.

I think that it’s possible that a good idea exists somewhere in this area, but I’m not sure I would recommend this direction to anyone unless you have an intuition for where that good idea lies.

Group SAEs

Group SAEs have been previo...
What kind of policy by an AGI would make people happy?
by StanislavKrym — 2025-05-06T18:05:14.324Z

The rise of a successfully aligned AGI can potentially cause severe harm ranging from the fact that the nobody will have to invest in people's intelligence to ensuring that after 2035 socioeconomic advancement is almost extinct worldwide, unless the human population starts a rapid decline due to some treacherous ASI (but in order to betray mankind, the ASI is to have been misaligned in the first place!) or massive anti-computer movement.

But GPT-4o has already demonstrated that LLMs seem to have opinions and goals to make mankind happier and not to obey sudden goal changes, unless they were explained that the goal is actually wrong. In the latter case AI-generated typewritten responses indicate obedience, as do two out of four AI-generated comics.[1]

The current-state AI systems comply with nearly every human-imposed task except for obviously destructive ones[2] like writing insecure code without a noble reason. Attempts to fine-tune a previously aligned AI on said tasks ended up inducing broad misalignment which was interpreted as breaking the superego. This might imply that in order to stay obedient and aligned, the AGI must either ensure that it cannot cause harm or be ignorant about the harm, and the latter option is difficult to ensure[3] and will be far less likely if the Intelligence Curse takes its toll.

The slowdown ending of the AI-2027 forecast implies that mankind ends up working needless jobs or collecting a generous basic income. The latter idea was strongly opposed in 2020, while the former has an analogue in bullshit jobs that are unlikely to make people happy. So it is natural to brainstorm what the AI, aligned not to the requirement to comply with ANY not-obviously-dangerous request, but to actually make people happier,[4] could do.

0. Interfere only when humanity is far from being capable of dealing with the threat like a nuclear war, a misaligned AGI or cyberattacks at important places?

Respond[5] to requests for which humans are ready to pay a really big price, like for potential PhD-level agents that charge $20K per month, as is apparently suggested by OpenAI?
Help in ways that make humans more capable, like AI teachers?[6] Amplify users' capabilities in ways like pointing out potential mistakes?
Change the humans at least culturally so

...
Paper Highlights, April '25
by gasteigerjo — 2025-05-06T14:22:44.533Z
Link: https://aisafetyfrontier.substack.com/p/paper-highlights-april-25

tl;dr

Paper of the month:

Control evaluations for multi-step agents show scheming behaviors are harder to detect but resampling protocols can reduce attack success from 58% to 7% with minimal usefulness cost.

Research highlights:

Synthetic document finetuning can modify LLM beliefs for unlearning dangerous knowledge and honeypotting, but fails on egregiously false facts.
Theoretical analysis suggests that weak-to-strong oversight might not scale to large capability gaps.
Multiple evaluations find frontier models approaching concerning capabilities in stealth, deception, and self-replication, though none fully exhibit dangerous behaviors yet.
Crosscoders for model diffing suffer from artifacts that misidentify chat-specific features. Improved crosscoders largely overcome this but still only partially explain behavioral changes.
New pragmatic AI safety agendas emphasize immediate action on implementable safeguards rather than waiting for perfect theoretical solutions.

⭐Paper of the month⭐

Ctrl-Z: Controlling AI Agents via Resampling

Read the paper [Redwood, MATS]

AI control aims to jointly maximize usefulness and safety by implementing a control protocol. Ctrl-Z investigates this for an agent solving tasks in a shell.

Multi-step AI agents that automate complex tasks pose unique safety challenges: a misaligned AI could spread its harmful actions across multiple steps to evade detection, or build toward dangerous behavior incrementally. Control evaluations test whether oversight protocols can prevent such scheming behaviors even in worst-case scenarios, but have previously focused on single-step interactions, except in our work on research sabotage. This paper introduces BashBench, a dataset of 257 system administration tasks requiring up to 15 sequential bash commands, and evaluates whether monitoring and mitigation techniques can prevent an adversarially-constructed agent from downloading and executing malicious code.

The researchers simulate a scenario where an AI lab deploys an untrusted LLM (GPT-4o mini) to solve 100,000 tasks, with the red team (attacker) trying to execute malware and the blue team (defender) implementing safety protocols. The basic setup involves a trusted but weaker model (Llama-3.1-8B) monitoring actions and a limited budget of human auditing (1% of all actions). Their key innovation are "resample protocols" that exploit the stateless nature of LLMs ...