trentmkelly/LessWrong-43k huggingface dataset:
Does human (mis)alignment pose a significant and imminent existential threat?
by jr — 2025-02-23T10:03:40.269Z

(This question was born from my comment on a very excellent post, LOVE in a simbox is all you need by @jacob_cannell )

Why am I asking this question?

I am personally very troubled by what I would equate to human misalignment -- our deep divisions, our susceptibility to misinformation and manipulation, our inability to identify and act collectively in our best interests. I am further troubled by the deleterious effects that technology has had in that regard already (think social media), and would like to see efforts to not only produce AI that is ethical or aligned itself (which, don't get me wrong, I LOVE and find very encouraging), but also ensure that AI is being harnessed to offer humans the support they need to realign themselves, which is critical to achieving the ultimate goal of Alignment of the (Humans + AI) Collaboration as a whole.
However, that's just my current perspective. And while I think I have good reasons for it, I realize I have limitations -- both in knowledge and experience, and in my power to effect change. So, I'm curious to hear other perspectives that might help me become more right or just understand other viewpoints, or perhaps connect with others who are like-minded so we can figure out what we might be able to do about it together.

How is this practical?

If others here share my concerns and believe it is a significant threat that warrants action, I will likely have follow-on questions for discussion toward that end. For instance, I'd love to hear if there are efforts already being made that address my concerns. Or perhaps if anyone thinks that creating and deploying aligned AI will naturally help humans overcome those issues, I'd be curious to hear their thoughts. I have some ideas of my own too, but I'll save those at least until I've done a lot more listening and understanding first to establish some mutual understanding and trust.
Deep sparse autoencoders yield interpretable features too
by Armaan A. Abraham — 2025-02-23T05:46:59.189Z

Summary

I sandwich the sparse layer in a sparse autoencoder (SAE) between non-sparse lower-dimensional layers and refer to this as a deep SAE.
I find that features from deep SAEs are at least as interpretable as features from standard shallow SAEs.
I claim that this is not a tremendously likely result if you assume that the success of SAEs is entirely explained by the accuracy of the superposition hypothesis.
I speculate that perhaps by relaxing our adherence to the concrete principles laid out by the superposition hypothesis, we could improve SAEs in new ways.

Introduction

Context

Instead of rehashing the superposition hypothesis and SAEs, I will just link these wonderful resources: Toy Models of Superposition and Towards Monosemanticity.

Motivations

My sense is that the standard justification for the success of SAEs is that the superposition hypothesis is just an accurate model of neural network function, which directly translates to SAE effectiveness. While many of the arguments for the superposition hypothesis are compelling, I had a growing sense that the superposition hypothesis was at least not a sufficient explanation for the success of SAEs.

This sense came from, for example, inconsistencies between empirical findings in the posts linked above, for example that features will take similar directions in activation space if they don’t co-occur often, but also if they have similar effects on downstream model outputs, which seems contradictory in many cases. I won't elaborate too much on this intuition because (1) I don’t think understanding this intuition is actually that necessary to appreciate these results even if it served to initiate the project, (2) in hindsight, I don’t even think that these results strongly confirm my original intuition (but they also don’t oppose it).

Ultimately, I posited that, if there is some unidentified reason for the success of SAEs, it might be that sparsity is just a property of representations that us humans prefer, in some more abstract sense. If this were true, we should directly aim our SAEs to produce the most faithful and sparse representations of neural network function as possible, possibly abandoning some of the concrete principles laid out by the superposition hypothesis. And it seems that the obvious way to do this is to add more layers.

I'm posting my work so far because:

I want to see if I'm missing something obvi

...
New Report: Multi-Agent Risks from Advanced AI
by Lewis Hammond — 2025-02-23T00:32:29.534Z
Link: https://www.cooperativeai.com/post/new-report-multi-agent-risks-from-advanced-ai

The development and widespread deployment of advanced AI agents will give rise to multi-agent systems of unprecedented complexity. A new report from staff at the Cooperative AI Foundation and a host of leading researchers explores the novel and under-appreciated risks these systems pose.

Powerful AI systems are increasingly being deployed with the ability to autonomously interact with the world and adapt their behaviour accordingly. This is a profound change from the more passive, static AI services with which most of us are familiar, such as chatbots and image generation tools. On the other hand, while still relatively rare, groups of AI agents are already responsible for tasks that range from trading million-dollar assets to recommending actions to commanders in battle.

In the coming years, the competitive advantages offered by autonomous, adaptive agents will drive their adoption both in high-stakes domains, and as intelligent personal assistants, capable of being delegated increasingly complex and important tasks. In order to fulfil their roles, these advanced agents will need to communicate and interact with each other and with people, giving rise to new multi-agent systems of unprecedented complexity.

While offering opportunities for scalable automation and more diffuse benefits to society, these systems also present novel risks that are distinct from those posed by single agents or by less advanced AI technologies (which are the focus of most research and policy discussions). In response to this challenge, staff at the Cooperative AI Foundation have published a new report, co-authored with leading researchers from academia and industry.

‍Multi-Agent Risks from Advanced AI offers a crucial first step by providing a taxonomy of risks. It identifies three primary failure modes: miscoordination (failure to cooperate despite shared goals), conflict (failure to cooperate due to differing goals), and collusion (undesirable cooperation in contexts like markets). The report also explains how these failures – among others – can arise via seven key risk factors:

Information asymmetries: Private information leading to miscoordination, deception, and conflict;
Network effects: Small changes in network structure or properties causing dramatic shifts in system behaviour;
Selection pressures: Competition, iterative deployment, and continual learning favouring undesirable be

...
Power Lies Trembling: a three-book review
by Richard_Ngo — 2025-02-22T22:57:59.720Z
Link: https://www.mindthefuture.info/p/power-lies-trembling

In a previous book review I described exclusive nightclubs as the particle colliders of sociology—places where you can reliably observe extreme forces collide. If so, military coups are the supernovae of sociology. They’re huge, rare, sudden events that, if studied carefully, provide deep insight about what lies underneath the veneer of normality around us.

That’s the conclusion I take away from Naunihal Singh’s book Seizing Power: the Strategic Logic of Military Coups. It’s not a conclusion that Singh himself draws: his book is careful and academic (though much more readable than most academic books). His analysis focuses on Ghana, a country which experienced ten coup attempts between 1966 and 1983 alone. Singh spent a year in Ghana carrying out hundreds of hours of interviews with people on both sides of these coups, which led him to formulate a new model of how coups work.

I’ll start by describing Singh’s model of coups. Then I’ll explain how the dynamics of his model also apply to everything else, with reference to Timur Kuran’s excellent book on preference falsification, Private Truths, Public Lies. In particular, I’ll explain threshold models of social behavior, which I find extremely insightful for understanding social dynamics.

Both of these books contain excellent sociological analyses. But they’re less useful as guides for how one should personally respond to the dynamics they describe. I think that’s because in sociology you’re always part of the system you’re trying to affect, so you can never take a fully objective, analytical stance towards it. Instead, acting effectively also requires the right emotional and philosophical stance. So to finish the post I’ll explore such a stance—specifically the philosophy of faith laid out by Soren Kierkegaard in his book Fear and Trembling.

The revolutionary’s handbook

What makes coups succeed or fail? Even if you haven’t thought much about this, you probably implicitly believe in one of two standard academic models of them. The first is coups as elections. In this model, people side with the coup if they’re sufficiently unhappy with the current regime—and if enough people side with the coup, then the revolutionaries will win. This model helps explain why popular uprisings like the Arab Spring can be so successful even when they start off with little military force on their side. The second is coups as battles. In...
Transformer Dynamics: a neuro-inspired approach to MechInterp
by guitchounts — 2025-02-22T21:33:23.855Z

How do AI models work? In many ways, we know the answer to this question, because we engineered those models in the first place. But in other, fundamental, ways, we have no idea. Systems with many parts that interact with each other nonlinearly are hard to understand. By “understand” we mean they are hard to predict. And so while the AI community has enjoyed tremendous success in creating highly capable models, we are far behind in actually understanding how they are able to perform complex reasoning (or autoregressive token generation).

The budding field of Mechanistic Interpretability is focused on understanding AI models. One recent approach that has generated a lot of attention is the Sparse Autoencoder (SAE), which hypothesizes that a neural network encodes information about the world with overlapping or superimposed sets of activations; this approach attempts to discover activations inside transformer models that correspond to monosemantic concepts when sparsified or disentangled with the SAE. Work along this path has shown some success—the famous Golden Gate Claude is a great example of an SAE feature corresponding to a monosemantic concept, and one that has causal power over the model (i.e. activating that feature led Claude to behave as if it were the Golden Gate Bridge)—but it also has some limitations. First, in practice, it’s prohibitive to train SAEs for every new LLM; second, SAE features are not always clear or as monosemantic as they should be to have explanatory power; and third, they are not always activated by the feature they are purported to encode, and their activation does not always have causal power over the model.

We were inspired by a trend in neuroscience to focus on the dynamics of neural populations. The emphasis here is both on dynamics and on populations, and the underlying hypothesis is that important neural computations unfold over time, and are spread across a group of relevant neurons. This approach is in contrast to analyses that focus on individual units, and those that treat computation as a static process.

Some notable examples where the population dynamics approach has yielded insights has been in the motor cortex, where preparatory activity before a movement sets the system up for proper initial conditions during movement, and unfolds in a null subspace that’s orthogonal to activity during movement. Moreover, during mov...
Recursive Cognitive Refinement (RCR): A Self-Correcting Approach for LLM Hallucinations
by mxTheo — 2025-02-22T21:32:50.832Z

I’m an independent researcher who has arrived here, at AI safety through an unusual path, outside the standard academic or industry pipelines. Along this journey, I encountered the recurring problem of large language models exhibiting “hallucinations”[^1] - outputs that can be inconsistent or outright fabricated - and became curious whether a more systematic self-correction mechanism could exist.

This led me to develop a concept I call **Recursive Cognitive Refinement (RCR)**, intended to help large language models detect and reduce internal contradictions and factual errors across multi-turn interactions. I’m posting these early ideas here, seeking constructive criticism, potential collaboration, and feedback from alignment-minded readers, researchers and experts.

Modern LLMs often produce highly fluent but occasionally contradictory or false statements. The typical solutions, including chain-of-thought prompting or RLHF, offer incremental improvements but they rarely force a model to revisit and refine its own outputs in a structured loop. Each query is still largely independent, so errors from prior turns can become ‘baked in’. RCR aims to close this gap by introducing repeated self-checks, requiring the model to examine and refine prior statements until contradictions or missteps are resolved or until a time limit is reached.

Some details of RCR remain unpublished, both for intellectual property reasons and to ensure responsible development. My accompanying white paper covers the conceptual foundation. I’m really hopeful RCR might noticeably reduce hallucinations by compelling a model to spot and correct its own errors across multiple turns, instead of relying on a single-pass solution. Potential benefits include improved consistency and possibly better alignment, but pitfalls exist—like increased overhead or the possibility of “infinite loop” refinements that don’t truly fix factual inaccuracies.

I’d value perspectives on where RCR might conflict with interpretability or alignment strategies. For instance, does repeatedly forcing a model to refine previous answers risk entrenching subtle biases or overshadowing external forms of oversight?
From a safety standpoint, is this approach genuinely helpful, or might it mask deeper issues? I’m also, very aware, that as an outsider, developing or validating RCR will require collaboration with experienced AI saf...
Gradual Disempowerment: Simplified
by Annapurna — 2025-02-22T16:59:39.072Z
Link: https://jorgevelez.substack.com/p/gradual-disempowerment

This post is a summary of a paper recently posted here that describes, in my opinion, a very possible scenario that modern society will have to face in the near future.
This post is not really intended for the average LW reader, as most of you probably read the original paper. I wrote this post for adults vaguely aware of the advancements of AI in society.
HPMOR Anniversary Guide
by Screwtape — 2025-02-22T16:17:25.093Z

[1]

Intro

To everyone running an anniversary party, thank you. Someone had to overcome the bystander effect, and today it seems like that’s you. I’m glad you did, and I expect your guests will be too. This guide aims to give you some advice and inspiration as well as coordinate.

The Basics

If you want to know if someone else is running one or if there would be interest in your city, check this spreadsheet.

If you’re up for running an anniversary party, pick a time and a place and announce it. If you haven't already, please fill out this form: https://tinyurl.com/hpmor-ten. If you have any questions, you can always reach out at skyler [at] rationalitymeetups [dot] org. Everything else is commentary.

What will you need to do at the party itself?

As much or as little as you want, mostly. The basics:

Arrive a bit early.
Be noticeable—maybe have a sign that says “HPMOR” in big letters.
Talk to people who show up.

If you want to do more, there’s some suggestions in Improvements.

Time & Place

Time: The default time is 6pm local time on Pi Day (Friday March 14th) but you’re the host. If you want to do it earlier or later in the day, you can. You can also do it on a nearby day if Friday isn't ideal.

Place: The best option depends on how many people you expect. Easiest answers first:

Coffee shop: Pick somewhere easily accessible to local forms of transport (near a bus stop, near a parking lot, etc). Publicly commit to being there for a certain duration of time, like an hour. That makes it easy for other people to come without needing to RSVP, and you can bring a book to read if nobody shows up.

Private house: This can be a bit of a higher barrier to entry, so make it clear in your invitation that everyone is welcome and this isn’t just for your close friends.[2] How many people can come obviously depends on the size of your house, but one rule of thumb is you can probably take twice the number of people as you have chairs/seats.

Public communal spaces: Board game venues, restaurants or pubs that have private rooms, picnic grounds and parks, things like that. This probably will require a phone call with the owner to check if they’re able to host you, and might have to pay to rent the space. If it’s a pub, check if there’s an age restriction.
University or Campus spots: If you’re a student, you might be able to book a classroom or social area at a library. It’s

...
Forecasting Uncontrolled Spread of AI
by Alvin Ånestrand — 2025-02-22T13:05:57.171Z
Link: https://forecastingaifutures.substack.com/p/uncontrolled-spread-of-ai

In my

last post

, I investigated potential severity and timeline of AI-caused disasters. This post goes into detail about something that could potentially precede such disasters: uncontrolled spread of AI.

While AIs are often released as open source, you might at least hope that the AI developers think twice about releasing a genuinely dangerous AI system for everyone to access. However, AI could spread anyway. This post focuses on two especially relevant scenarios: IP theft and AI proliferation. I’ll also mention a couple of other interesting AI incident predictions at the end.

My reasons for investigating this:

IP theft: AI, or related IP like training algorithms and data processing techniques, could be stolen. It could then be misused, leaked on the cloud, or sold to malicious actors. Stolen AI could then be run with insufficient safety mechanisms, and escape containment. If a nation steals it from another nation, it could intensify international tensions and reduce likelihood of successful AI arms control.
AI proliferation: AI replicating itself over the cloud, spreading to available devices, and pursuing goals under no control of any human. Tracking and shutting down proliferating AIs could prove highly difficult. If the proliferated AI has the ability to self-improve, it would be a race against time to find it and stop it from doing so. If a large number of AIs, pursuing who knows what, spread so widely that shutting all of them down becomes infeasible, the effects would be highly unpredictable and potentially disastrous.

Short reminder:

Please refer to the latest predictions in the embedded pages rather than relying on the ‘At The Time Of Writing’ (ATTOW) predictions.
Please suggest markets/questions missing from this post, or from the forecasting platforms, in the comments.
Visit the forecasting platforms to make your own predictions!

Several of the Manifold markets and Metaculus questions mentioned in this post have not received much attention yet. This post is largely for collecting relevant predictions in one place, so we can track how they change over time.

IP Theft

IP theft from three of the leading AI labs this year seem quite likely (65% ATTOW):

Any stolen IP—such as model weights, architecture details, private training data, or training algorithms—counts for this market. The stolen IP must however be worth $5 million or more. The quest...
Seeing Through the Eyes of the Algorithm
by silentbob — 2025-02-22T11:54:35.782Z

There’s a type of perspective shift that can bring a lot of clarity to the behavior and limitations of algorithms and AIs. This perspective may be called seeing through the eyes of the algorithm (or AI, or LLM). While some may consider it obvious and intuitive, I occasionally encounter people – such as inexperienced programmers struggling with why some algorithm doesn’t work as expected, or generally many people complaining about the limitations of LLMs – who seem to be unfamiliar with it. For those who do find it obvious, it may still prove beneficial to explore a variety of examples to make this perspective shift more intuitive. It also sheds some light on why LLMs nowadays still often appear less capable than they actually are and why certain industries have not been disrupted to the degree some might have expected. If you primarily care about the latter, feel free to jump ahead to Example 4: LLMs.

See through the eyes of an algorithm in two easy steps:

Wipe your mind clean. Imagine you’re in a dark room with no memory and no context.
Now visualize precisely the inputs available that the algorithm / AI has access to.

When you’re programming an algorithm you usually have a lot of context in mind. You know what the algorithm’s goal is, why you need it, in which kind of environment you’re using it, how crucial it is to obtain precise solutions, and so forth. And it’s very easy to let this context influence your expectations about the algorithm. But the algorithm doesn’t care, and it doesn’t know – all it knows is the inputs it has and the rules it follows. And the best way I know of getting a better feel for how the algorithm behaves and why, and what its optimal performance would look like (given the context it currently lives in), is to put yourself as deeply into its shoes as you can. And the best way to let your System 1 participate in this exercise is to actually visualize things and do your best to see things exactly as the algorithm or AI would.

Example 1: Trackmania AI

In

this video

, at the linked timestamp, the creator Yosh presents a brief hypothesis – “it looks like the AI wants to stay in the center of the stadium” – but quickly reveals that’s not possible due to the fact that the AI cannot even see the stadium, demonstrating briefly what the world looks like from the AI’s point of view:

The Trackmania AI trained in the video

...