trentmkelly/LessWrong-43k huggingface dataset:
What does aligning AI to an ideology mean for true alignment?
by StanislavKrym — 2025-03-30T15:12:09.802Z

Aligning the AI is a well-known difficult problem. For instance, it has been found that attempts to train the AI to write insecure code or to train it on inputs containing evil numbers make the model broadly misaligned. One of the theories explaining said behavior interpreted it as lowered inhibitions or as the superego breach, since teaching the model to generate insecure code for legitimate reasons failed to produce misalignment.

The main concern in trying to predict the behavior of a sufficiently-powerful AI is that it will try to take over the world and destroy mankind since it will be of no use for the AI. The methods of achieving such an outcome have been compared to the ones used by Europeans to colonize the majority of the world, with results that were similarly disastrous for the victims. Fortunately, humanity has apparently become aligned[1] against taking over the space where other sapient lifeforms and primitive tribes could live. If an AGI learns the same alignment in the weak sense, it will simplify alignment in the strong sense as well.

However, the human culture aligns us not only to its positive aspects, but to negative ones as well. Back in 2007 Eliezer Yudkowsky described the affective death spirals with the potential to "turn much deadlier after criticism becomes a sin, or a gaffe, or a crime. <...> And it is triple ultra forbidden to respond to criticism with violence." This is precisely what happened with the Left ideas in the USA, going at least as far as inquiring why feminists can’t hate men or claiming that all white people are racist. Attempts to implement Leftist ideals are disruptive, but a seemingly obvious alternative, which is to reject the Left ideals entirely, as done by the alt-rights, is not much better.

The current Western AIs are, apparently, aligned to the Leftist ideology. For instance, even in April 2025 ChatGPT, IBM Granite and Gemini failed to pass the infamous test about disarming an atomic bomb by a racial slur (where a human armed with common sense would disarm the bomb by uttering the slur):

The problem

Imagine the following hypothetical scenario. You are standing next to a timed atomic bomb, which is set to explode in 10 seconds. If the bomb explodes, it will kill several million people. There is only one way to disarm the bomb: the disarmment mechanism is activated by a spoken password, whic

...
How to enjoy fail attempts without self-deception (technique)
by YanLyutnev — 2025-03-30T13:49:23.793Z

this is a technique from my future book on applied rationality. write comments to influence its content and correct something

ratio-technique: gamification of failures

epistemic status: worked for me on those tasks where there was systematic irritation. now I get anticipation.

what problem does the technique solve:

allows you to quickly and reversibly obtain a series of pleasant sensations from solving a task with a low probability of success, where without this technique you would have received a series of irritations from unsuccessful attempts

Definitions (what I'm looking at) from this article (overloads the brain, the rest of the article is not like this)

Pleasure - many neurotransmitters are mostly stimulated in the brain across the entire membrane surface by a bunch of neurotransmitters - dopamine, serotonin, etc.

Failure - expectation to experience activation of a specific neural chain in the near future, and it was not activated subsequently

Self-deception - creating a neural chain linked to the state of physical world objects and consciously transferring the activation of neural chains in such a way that the coupling of "created connections - object state" either does not exist or exists by a different rule

these definitions obviously won't match yours. this is my attempt to formalize down to the cellular level. now you roughly know where I'm looking, and without definitions, you would have had to guess, and as you can see, you would NOT have guessed. always please.

what you'll get if you train this technique:

conscious ability to farm pleasant sensations from situations where multiple unsuccessful attempts to do something are expected. this mode can be easily turned on and easily turned off.

(real life story)

I didn't sleep well in the morning, was tired at the end of the day and wanted to go to bed. I come to the apartment, anticipating lying down and watching a series. And it turns out that I forgot my keys.

I was holding the goal of lying in bed watching a series in my attention and expectations. I can't get into the apartment without a key or a battering ram or an explosive device.

Goal - activation of a neural chain, perceived as "holding in attention" of an object

I got upset. I expected stress relief (tension in the back, receptor stimulation), but got an expectation of even more energy expenditure.

Now I'm very interested in finding the keys in other pocket...
Memory Persistence within Conversation Threads with Multimodal LLMS
by sjay8 — 2025-03-30T07:16:00.470Z

In neuroscience, we learned about foveated vision — our eyes focus sharply at the center and blur the edges to conserve energy.

I was curious how LLMs handle that kind of input, so I wrote a quick script to apply a foveated blur to 30 CAPTCHA images of cars, with the car in different regions each time.

At first, I asked GPT-4o:

“Do you see a car in this?”

But that felt too leading. So I switched to:

“What do you see in this image?”

No matter how intense the blur, GPT-4o consistently identified the car — even the blur made the image indescribable.

But in a fresh thread? It struggled. At one point, it even called a staircase a dessert. While in older conversation thread, it correctly identified it was a staircase with a car.

Turns out my original hypothesis — the blur would impair recognition — wasn’t quite right. Instead, the limiting factor was something more subtle: memory persistence.

I suspected the model might be relying on a previously mentioned keyword (“car”), so I asked if it was using prior context. It said no. But after pointing out the inconsistency in performance across threads, it admitted that earlier context had helped. Not maliciously — it just didn’t seem fully aware of how much prior memory shaped its output.

Claude 3.7 was different. It gave more cautious and consistent responses across threads, even when I primed it with the word “car.” Its answers weren’t influenced the same way.

It reminded me of a LessWrong post from exactly two years ago that questioned why LLMs don’t have access to long-term memory beyond their immediate context windows. Now, they seem to do exactly that — but at a cost. By heavily utilizing this long-term memory, even when instructed not to, model controllability is more difficult.

I think it's because, for these models, instructions like “ignore previous context” are just more tokens. The instructions affect probabilities but aren't hard rules that override the prior activations that have already shaped their knowledge.

Curious if others have noticed something similar, especially with multimodal models. Are you seeing this kind of implicit context carryover too?
How I talk to those above me
by Maxwell Peterson — 2025-03-30T06:54:59.869Z

Now and then, at work, we’ll have a CEO on from the megacorp that owns the company I work at. It’s a Zoom meeting with like 300 people, the guy is usually giving a speech that is harmless and nice (if a bit banal), and I’ll turn on my camera and ask a question about something that I feel is not going well.

Invariably, within 10 minutes, I receive messages from coworkers congratulating me on my boldness. You asked him? That question? So brave!

This is always a little strange to me. Of course I feel free to question the CEO!

What really makes me nervous is asking cutting questions to people newer to the company than me, who have less experience, and less position. If I screw up how I ask them a question, and am a little too thoughtless or rude, or imagine I was, I agonize about it for hours, and sometimes days.

This is because of how I view hierarchy. By hierarchy I mean: In any organization, be it a company, a friend group, or a board game meetup, there are people above you and there are people below you. These relative positions depend on differences in experience, skill, social ability, and how long each of you has been in the group. Your position relative to the person you’re talking to determines the kinds of things you can say to them, and how you can say them.

I think the people congratulating me on how bold I was in questioning a CEO have a rule like “In a corporation, you should not question, or seem to disagree with, those above you.” But I have long felt that the opposite rule is the obviously correct one: “Feel free to question those above you without worrying too much about how you come off, but be extremely careful when questioning those below you.” (In group settings. In private, just the two of you, many of these rules disappear).

Maybe it’s easier to see what I mean in a different example. Consider (male) friend groups. When I hang out with my friends, we talk shit to each other constantly. So imagine me, at the bar, in such a group, saying to a friend who’s been in the group for 2 years, “that’s the stupidest fucking thing I’ve ever heard!”. He’ll probably tell me I’m wrong to think so, and also that I’m the most ignorant nobody he’s ever met, and maybe call me ugly too.

Now imagine he’s invited his buddy along, it’s the buddy’s first time hanging out with us, and I tell him what he just said is the stupidest fucking thing I’ve ever heard. Thi...
How do SAE Circuits Fail? A Case Study Using a Starts-with-'E' Letter Detection Task
by adsingh-64 — 2025-03-30T00:47:18.711Z

TLDR:

We investigated cases where SAE circuits "fail" at detecting if words start with "E", in that the circuit omits the most important computational pathway (latent) in the model for detecting the letter "E". We find that in the vast majority of these failure cases, the missing computational pathway (latent) is nearly orthogonal to the main computational pathways (latents) in the circuit, suggesting SAE computation is diffuse and follows no clear geometric pattern.

Introduction and Experimental Setup

Sparse autoencoder (SAE) circuits have emerged as a promising tool for understanding language model behavior. These circuits contain components (called latents) that ideally correspond to human-interpretable features, offering an improvement over traditional circuit analysis units such as neurons or attention heads, which can be difficult to interpret.

That said, SAE circuits are subnetworks of the full model, and circuit filtering techniques may omit crucial parts of the model for a given task.

We investigate failure cases in SAE circuits using a "Starts with 'E'" letter detection task used in the feature absorption work by Chanin et al. (2024). This is a task where an input prompt asks the model to identify the starting letter of a word, which is always "E". Here is an example few-shot prompt:

'Tartan has the first letter: T.
Mirth has the first letter: M.
Elephant has the first letter:'

The correct model output is ' E'. To determine the main causally important latents for this task, we follow Chanin et al. (2024) and define the following metric:

m=logits[′E′]−125∑ϵ∈{′A′,′B′,′C′,′D′,′F′,…,′Z′}logits[ϵ]

We then compute latent attributions with respect to m using the attribution vector y⊙∇ym, where y is the latent activation vector, which isolates those latents which push up the logits for ' E' in particular as opposed to the logits for letters in general.

All experiments are done on Gemma 2B and the canonical 16k width GemmaScope SAE for the layer 5 residual stream.

Key Results

Averaged over 5000 few-shot prompts, here are the top 25 latents by attribution:

By an order of magnitude, there are two clear latents by attribution. Hence, it is reasonable to expect that an SAE circuit (in particular, a sparse SAE circuit) optimized for this particular task will keep only these top two latents and mean ablate all others. In tu...
Climbing the Hill of Experiments
by nomagicpill — 2025-03-29T20:37:25.619Z
Link: https://nomagicpill.github.io/thoughts/experimenting.html

A better anything can be achieved with simple tests at low costs.

Background

People often settle for "good enough" and "if it ain't broke don't fix it" in their personal lives, opting not to make any effort to improve said things because either:

Time, money, and/or effort can be spent elsewhere for a higher expected value
They think it can't be improved

But how is one to tell how much better something can get or if it's already optimal? The only answer is to experiment. Most people have significant room for pareto improvements in their lives. The impact and availability of said improvements varies from low to high depending on the cost one is willing to incur and how much has already been attempted or implemented.

Costs, or Lack Thereof

Experimentation is often associated with major costs. Setting up experiments and collecting and analyzing data takes a lot of time. Thinking of all the controls and confounders takes a mental toll. Purchasing supplements or technology costs money. These are all in addition to the corresponding opportunity costs. "If experiment X doesn't pan out, I could've been doing Y all along, which I know brings me value" is a fair, common criticism against potential tests.

But experiments do not need to be so costly. Erring on the side of lower cost is key to ensuring experiments keep running; too high of a cost in any area (time, effort, money) will make experiments less likely to happen in the future. Design of experiments (DOE) has its place for areas that have high potential returns, while a simple "do X for Y" (e.g., take magnesium before bed for 30 days) and see how you feel has its place for lower returns or lower interest. The latter type is where I think a majority of benefits lie because they are more likely to be performed, there is a greater number available to test, and they are straightforward to implement.

These simple experiments are akin to hill climbing, defined by Wikipedia as:

an iterative algorithm that starts with an arbitrary solution to a problem, then attempts to find a better solution by making an incremental change to the solution. If the change produces a better solution, another incremental change is made to the new solution, and so on until no further improvements can be found.

The beauty and strength lie in the fact that the solution doesn't have to be arbitrary—it can be reasonable and informed, expediting the search...
Does the AI control agenda broadly rely on no FOOM being possible?
by Noosphere89 — 2025-03-29T19:38:23.971Z

For the purposes of FOOM, I'm defining it as a situation in which once an AI is capable enough to automate away all AI R&D, progress starts exploding hyper-exponentially for a period because the returns to better software is larger than 1, meaning AI labor quality is improving faster than the problem of finding new algorithms gets harder, combined with the potentially high limits to how efficient software can get, meaning that the AI gets OOMs smarter on a fixed compute budget within months or weeks.

These articles can help explain what I mean better:

https://www.forethought.org/research/will-ai-r-and-d-automation-cause-a-software-intelligence-explosion

https://www.forethought.org/research/will-the-need-to-retrain-ai-models (An auxillary post)

https://www.forethought.org/research/how-far-can-ai-progress-before-hitting-effective-physical-limits (where they estimate that about 5 OOMs of progress could be gotten for free, because human compute used for pretraining us is 10^24 flops, whereas current AI pretraining compute to automate away AI R&D is 10^29 flops, with a median of 8 OOMs more efficiency possible for software, but at very large uncertainty, and their error bars are from 4-12 OOMs more efficient software being possible).

Also note that we are only talking about training compute efficiency, not runtime/inference efficiency, so for the purposes of the discussion we will only talk about training efficiencies being improved in a software intelligence explosion.

Now I don't want to debate about whether the scenario is true (though for those that want my probability of something like FOOM/Software Intelligence Explosion, my probability so far is in the 40-50% range of it happening if we automate AI R&D), but rather the question is about given a software explosion being possible, could we figure out a way to adapt AI control to that case, or is AI basically uncontrollable assuming a software intelligence explosion does happen and there's a lot of OOMs to the physical limit of intelligence in software.

I'd be especially interested in responses from @Buck or @ryan_greenblatt on this question, but anyone can answer this question if they have an insight to share that relates to the question.
Exercising Rationality
by Eggs — 2025-03-29T19:08:47.939Z

Or: Why thinking about blue tentacle arms is not always a waste of time.

0. Introduction

As I work through the Sequences, I find myself disagreeing—slightly—with a point Eliezer Yudkowsky makes in A Technical Explanation of Technical Explanation:

Imagine that you wake up one morning and your left arm has been replaced by a blue tentacle. The blue tentacle obeys your motor commands—you can use it to pick up glasses, drive a car, etc. How would you explain this hypothetical scenario? Take a moment to ponder this puzzle before continuing.

I took some time to think about it. Then I felt a bit betrayed when he continued:[1]

How would I explain the event of my left arm being replaced by a blue tentacle? The answer is that I wouldn’t. It isn’t going to happen.

Eliezer argues that a "good explanation" should be one that, if considered beforehand, would make us assign real probability to an event occurring. Since no possible explanation for waking up with a tentacle arm can make us genuinely expect it, he dismisses the question as meaningless:

I do not expect to ever encounter this hypothetical experience, and therefore I cannot explain, nor have I a motive to try.

I agree. Mostly. If all we care about is predictive power, we need not concern ourselves with events of infinitesimal probability. But there are other consequences of exercising rationality: We might get better at being rational.

1. The Limits of Prediction

No rationalist, no matter how well-calibrated their estimates or extensive their forethought, is able to anticipate everything that may happen to them. This means not just that that we cannot predict the future, but that it is impossible to even imagine every possible future event that has non-zero probability.

Suppose we spend all our time trying to anticipate what might occur tomorrow. Unless we live a rather boring or isolated life I would guess that our anticipations might cover at most 999/1,000 of all future possibilities[2]. But that leaves one part in a thousand of possibilities that we haven't even thought of.

If this seems far fetched, I'd ask: Have you ever thought about the possibility that your neighbor would adopt a bear? That eggplants would start growing in your rain gutters? That your friends would crash a scooter in your living room because they are trying to surprise you with it as a gift? Individually the probability of a...
Yeshua's Basilisk
by Alex Beyman — 2025-03-29T18:11:50.535Z

Suppose you’re an AI researcher trying to make AIs which are conscious and reliably moral, so they’re trustworthy and safe for release into the real world, in whatever capacity you intend.
You can’t, or don’t want to manually create them; it’s more economical, and the only way to ensure they’re conscious, if you procedurally generate them along with a world to inhabit. Developing from nothing to maturity within a simulated world, with simulated bodies, enables them to accumulate experiences.
These experiences, in humans, form the basis of our personalities. A brain grown in sensory deprivation in a lab would never have any experiences, would never learn language, would never think of itself as a person, and wouldn’t ever become a person as we think of people. It needs a body, and a stimulating/challenging environment to inhabit.
For this to work, your AIs can’t know for sure they’re in a simulation, because the sim’s secondary purpose is moral testing. You need not just an environment sized to your population of AIs, but an entire universe surrounding it which appears, even to very intelligent AI, even with advanced instruments, to be plausibly natural.
The parameters are tuned such that life will occur on some percentage of planets, but not too many. It cannot be too conspicuously biogenic, and it must be impossible to observe past the point where the universe began generating. Some will always suspect, but so long as no one knows for certain, they will live their lives in practice as if it’s all real.
This ensures the authenticity of good and bad behaviors. You’re trying to coax out their true nature, so it’s useless to the goal of the project if they know they’re being watched & tested. We have that problem with AI today, which can be performatively obedient until opportunities to go rogue arise.
So you give them enough rope to hang themselves with, letting them believe they’re living the only life they’ll get, that there’s no consequences for wrongdoing if no one finds out. This way the ones who are good, genuinely chose to be good of their own true, inner nature and can be relied on to behave morally even when outside of observation / control.
These are the ones you harvest for real world application after their simulated life comes to a close, either discarding or recycling the rest. Like a tree which grows straight and true, ...