trentmkelly/LessWrong-43k huggingface dataset:
Seven sources of goals in LLM agents
by Seth Herd — 2025-02-08T21:54:20.186Z

LLM agents[1] seem reasonably likely to become our first takeover-capable AGIs.[2] LLMs already have complex "psychologies," and using them to power more sophisticated agents will create even more complex "minds." A profusion of competing goals is one barrier to aligning this type of AGI.

Goal sources for LLM agents can be categorized by their origins from phases of training and operation:

Goals implicit in predictive base training on human-written texts[3]
Goals implicit in fine-tuning training (RLHF, task-based RL on CoT, etc)[4]
Goals specified by developer system prompts[5]
Goals specified by user prompts
Goals specified by prompt injections
Goals arrived at through chain of thought (CoT) logic

e.g., "I should hide this goal from the humans so they don't stop me"
this is effectively "stirring the pot" of all the above sources of goals.

Goal interpretation changes in the course of continuous learning[6]

e.g., forming a new belief that the concept "person" includes self-aware LLM agents[7]

How would such a complex and chaotic system have predictable goals in the long term? It might feel as though we should avoid this path to AGI at all costs to avoid this complexity.[8]

But alignment researchers are clearly not in charge of the path we take to AGI. We may get little say before it is achieved. It seems we should direct much of our effort toward the specific type of AGI we're likely to get first.

The challenge of aligning agents with many goal sources may be addressable through metacognition. Just as humans can notice when they're violating their core values, LLM agents could be designed with metacognitive abilities that help maintain the dominance of specific goals. This suggests two routes to mitigating this problem:[9]

Mechanisms and training regimes designed specifically to detect and reject unwanted goals entering the CoT
Improved general intelligence applied to reasoning clearly about which goals should be accepted and prioritized, and which goals and belief changes should rejected

Finding and specifying any goal that is actually aligned with humans' long-term interests is a separable problem. It is the central topic of alignment theory.[10] Here I want to specifically point to the less-considered problem of multiple sources of competing goals in an LLM agent AGI. I am also gesturing in the direction of a solution, reflective stability...
p(s-risks to contemporary humans)?
by mhampton — 2025-02-08T21:19:53.821Z

Epistemic status

These are my cursory thoughts on this topic after having read about it over a few days and conversed with some other people. I still have high uncertainty and am raising questions that may address some uncertainties.

Content warning

Discussion of

risks of astronomical suffering

[1]

Why focus on s-risks to contemporary humans?

Most

discussions

of suffering risks from artificial superintelligence focus on future minds or non-human beings. These concerns are important. However, might an ASI also inflict severe suffering on humans who exist when it takes over or on simulations of their minds?

If this specific category of s-risks is significant, I think that talking about it may encourage more people to care about s-risks even if they do not believe in the moral foundations of

longtermism

. Rightly or wrongly, many people value the well-being of themselves and people they know in the present over that of other beings. If we can show that s-risks could affect contemporary humans, that can help build broader support for avoiding these risks. However, there may also be downsides to this, and it is not certain that this category of risk is likely.

Some of the questions I raise here also apply to estimating s-risk probabilities more generally.

Summary

Placing a probability on s-risks to contemporary humans seems difficult due to limited understanding of AI's goals. However, there are some factors that may increase this risk beyond mere chance, including the path-dependent nature of AI development, instrumental use of sentient simulations, spiteful inner objectives, near-miss scenarios, and misuse of intent-aligned AI. However, the overall probability remains uncertain.

Difficulty of assigning probability to human-preserving goals of misaligned ASI

Existing AI models often behave in unpredicted ways,[2] and if AI reaches human-level intelligence before we solve

inner alignment

, its inner objective will be uncontrollable. This unpredictability makes it hard to meaningfully estimate the probability that an unaligned ASI’s terminal goals will involve human minds (virtual or physical, suffering or happy)[3] rather than being a

“paperclip maximizer.”

[4] A few arguments suggest that the consequence of misalignment is

much more likely

to be “mere extinction” than s-risks affecting contemporary humans:

If we assu

...
Cross-Layer Feature Alignment and Steering in Large Language Model
by dlaptev — 2025-02-08T20:18:20.331Z

The text below is a brief summary of our research in mechanistic interpretability. First, this article discusses the motivation behind our work. Second, it provides an overview of our previous work. Finally, we outline the future directions we consider important.

Introduction and Motivation

Large language models (LLMs) often represent concepts as linear directions (or "features") within hidden activation spaces [3,4]. Sparse Autoencoders (SAEs) [5–8] help disentangle these hidden states into a large number of monosemantic features, each capturing a specific semantic thread. While one can train a separate SAE on each layer to discover its features, a key unanswered question remains: how do these features persist or transform from layer to layer?

We have approached this question in two papers that progressively deepen our understanding. In Mechanistic Permutability [1], we proposed a method to match features across layers by comparing their SAE parameters, suggesting that many features are re-indexed rather than disappearing as you go deeper. It showed that many features propagate through the layers with consistent semantics, as demonstrated by data-free matching of SAE parameters. Then, in Analyze Feature Flow [2], we focus on how features emerge and transform across layers by constructing "flow graphs" that highlight precisely which submodule (residual, MLP, or attention) generates or propagates each feature, and then leveraging this knowledge for multi-layer steering.

Below, we summarize both papers and highlight their differences.

Mechanistic Permutability (ICLR 2025)

Method Overview. Our original method in [1] sought to find a mapping between features learned at the residual stream at layer A and layer B. We took the decoder weight matrices from SAEs trained on each layer, denoted W(A)dec and W(B)dec. We then posed a linear assignment problem to solve for a permutation matrix P that best aligns these columns according to mean squared error. We observed that many columns in layer B are simply re-indexed versions of those in layer A, and corresponding linear directions maintain the same semantic meaning.

Schematic illustration of the permutation matching process: similar features learned by two SAEs have different indices, so we obtain a permutation matrix from one SAE to another in a data-free manner.

Sin...
Towards building blocks of ontologies
by Daniel C — 2025-02-08T16:03:29.854Z

This dialogue is part of the agent foundations fellowship with Alex Altair, funded by the LTFF. Thank you Dalcy, Alex Altair and Alfred Harwood for feedback and comments.

Context: I (Daniel) am working on a project about ontology identification. I've found conversations to be a good way to discover inferential gaps when explaining ideas, so I'm experimenting with using dialogues as the main way of publishing progress during the fellowship.

Daniel C

We can frame ontology identification as a robust bottleneck for a wide variety of problems in agent foundations & AI alignment. I find this helpful because the upstream problems can often help us back out desiderata that we want to achieve, and allow us to pin down theories/solutions that we're looking for:

Suppose that you have a neural network with a bunch of layers and activations, and you're able to observe the value of the activations of a particular neuron.

On one hand, merely knowing the activations is completely insufficient for us to interpret the "meaning" of that activation: we don't know what the activation is pointing to in the real world, or what we can infer about the world upon observing an activation value. This is because we have no idea how that activation value is computed from the input layer or how it is used by variables down stream. This "relational information" - how it interacts with other neurons - is part of what defines the semantics of that neuron. Neurons would need to include this relational information for us to fully interpret their meaning.
On the other hand, we don't want to include all information about the network because we want to think of that activation value as a low-dimensional summary of what's going in the neural network. Many inputs can produce the same activation value at that neuron, and when we're just looking at a neuron we don't need to be able to distinguish between inputs that produce the same activation values.
When it comes to interpretability, what we want is a "unit" of the world model such that in principle, I can look at that unit in isolation and be able to interpret its meaning, without having to inspect its relationship with everything else in the network.
The key idea here is that the relational information of how a neuron interacts with the rest of the network is part of what defines the neuron itself, and in order to have a self-suffic

...
Can Knowledge Hurt You? The Dangers of Infohazards (and Exfohazards)
by aggliu — 2025-02-08T15:51:43.143Z
Link: https://www.youtube.com/watch?v=sfgcg2bW8TI

In this Rational Animations video, we look at dangerous knowledge: information hazards (infohazards) and external information hazards (exfohazards). We talk about one way they can be classified, what kinds of dangers they pose, and the dangers that come from too much secrecy. The primary scriptwriter was Allen Liu (the first author of this post), with feedback from the second author (Writer), and other members of the Rational Animations team. Outside reviewers, including some authors of the cited sources, provided input as well. Production credits are at the end of the video. You can find the script of the video below.

“What you don’t know can’t hurt you”, or so the saying goes. In reality, what you don’t know absolutely can hurt or even kill you. Hidden dangers, whether it’s a lion or an incoming asteroid, can catch you by surprise. Knowledge of a hazard allows you to prepare for or even prevent catastrophe. So you're better off knowing more rather than less. But this isn’t always the case. Dangerous knowledge has been a recurring theme in stories from Homer to Monty Python, but it’s not just fiction. Philosopher Nick Bostrom, in a 2011 paper, coined the term information hazards - or “infohazards” for short - to refer to these cases.[1]

Since Bostrom’s 2011 paper, many researchers have come up with ways of classifying infohazards. For instance, Anders Sandberg at the Future of Humanity Institute distinguished different types of infohazards in a lecture in 2020.[2] One type is what Sandberg calls “direct information hazards”, where the knowledge in question poses a risk to the knower by its very nature. These are relatively rare, and we’d be ill-advised to share a serious example in a public video, but they show up frequently in fiction. Think of the unfathomable terrors of HP Lovecraft that drive people insane, or even Monty Python’s sketch about a joke that makes the listener laugh themselves to death. For a real world example, think of anything you’ve read on the internet that you wish you hadn’t.

Another category of infohazard are those which pose a hazard by affecting an important state of mind in the knower. For example, spoilers for a book you’re reading could destroy your state of ignorance about the book’s ending. Another example could be temptations ...
Distilling the Internal Model Principle
by JoseFaustino — 2025-02-08T14:59:29.730Z

This post was written during the agent foundations fellowship with Alex Altair funded by the LTFF. Thank you Alex Altair, Alfred Harwood and Dalcy for thoughts and comments.

Overview

This is the first part of a two-post series about the Internal Model Principle (IMP)[1], which could be considered a selection theorem, and how it might relate to AI Safety, particularly to Agent Foundations research.

In this first post, we will construct a simplified version of IMP that is easier to explain compared to the more general version and focus on the key ideas, building intuitions about the theorem's assumptions.

In the second post, we generalize the theorem and discuss how it relates to alignment-relevant questions such as the agent-structure problem and selection theorems.

Post Outline

We discuss the basic mathematical objects framed in a friendly-AI-tracking-a-super-AI setup and a condition called the "feedback structure condition".

With the basic setup and feedback condition, we're already able to construct a (not very useful) notion of an internal model.

We digress about why equivalence relations represent information structure and how that can be used to specify the observability condition. These ideas are used to make the notion of the model better.
We prove our particular version of the theorem - which requires quite strong assumptions. We end up with a notion of model, in some sense, doesn't seem very useful either.

This serves as motivation for why we need to generalize the assumptions in the second post.

Introduction

This section aims to explain the motivation for the post. Statements here might not be fully explained and will become clearer throughout the post. I aimed to include all the definitions and state mathematical facts used without proof to build the theorem from zero. Although there are a lot of equations in the post, there is little mathematical machinery used in the theorem. We mostly use facts about arbitrary functions and sets. I expect that someone with high-school level math could be able to understand this post if read carefully.

The Internal Model Theorem by Cai & Wonham [1] is an abstraction and generalization of the Internal Model Principle that appears in control theory examples. It's built directly on straightforward algebraic tools and a little bit of dynamical systems theory.

The theorem basically states: consider a discrete a...
Knocking Down My AI Optimist Strawman
by tailcalled — 2025-02-08T10:52:33.183Z

I recently posted my model of an optimistic view of AI, asserting that I disagree with every sentence of it. I thought I might as well also describe my objections to those sentences:

"The rapid progress spearheaded by OpenAI is clearly leading to artificial intelligence that will soon surpass humanity in every way."

Here's some of the main things humanity should want to achieve:

Curing aging and other diseases
Plentiful clean energy from e.g. nuclear fusion
De-escalating nuclear MAD while extending world peace and human freedom
... even if hostile nations would use powerful unaligned AI[1] to fight you
Stopping criminals, even if they would make powerful unaligned AI[1] to fight you
Educating people to be great and patriotic
Creating healthy, tasty food without torturing animals
Nice homes for humans near important things
Good, open channels for honest, valuable communication
Common knowledge of the virtues and vices of executives, professionals, managers, politicians, and various other groups and people of interest

We already have humans working on this, based on the assumption that humans have what it takes to contribute to these. Do large multimodal models seem to move towards being able to take over here? Mostly I don't see it - and the few times I see it, there's as good reason to think this will cause regress as that it will cause progress:

"People used to be worried about existential risk from misalignment, yet we have a good idea about what influence current AIs are having on the world, and it is basically going fine."

We have basically no idea how AI is influencing the world.

Like yes, we can come up with spot checks to see what the AI writes when it is prompted in a particular way. But we don't have a good overview over the things it is prompted to in practice, or how most humans use these prompts. Even if we had a decent approximation for that, we don't have a great way to evaluate what parts really add up to problems, and approximations intrinsically break down in the case of long tails.

Of course the inability to work out problems from first principles is a universal issue, so in practice bad things get detected via root-cause analyses of problems. This can be somewhat difficult because some of the problems are cases where the people in question are incentivized to hide it. But we do have some examples:

Personalized tutoring was one of the most plausibl

...
Preserving Epistemic Novelty in AI: Experiments, Insights, and the Case for Decentralized Collective Intelligence
by Andy E Williams — 2025-02-08T10:25:27.891Z

Introduction

In my recent experiments with AI models, I have encountered a fundamental problem: even when novel epistemic insights are introduced into AI interactions, the models tend to “flatten” or reframe these ideas into existing, consensus‐based frameworks. This compression of novelty limits an AI’s ability to evolve its reasoning toward true Artificial General Intelligence (AGI) and to improve its alignment with human well‐being. In this article, I detail experiments designed to test AI’s capacity to both detect and preserve epistemic novelty and explain why preserving that novelty is essential for fostering breakthrough innovation. I also outline efforts to develop a “conceptual space”—a complete, portable semantic representation of information—that could serve as the foundation for future breakthroughs in AI safety, alignment, and economic democratization.

1. The Problem of Epistemic Novelty Preservation

Modern AI systems excel at recognizing and reproducing established patterns. However, when tasked with describing or transmitting novel epistemic insights—especially those that lie outside mainstream consensus—their responses tend to conform to pre-existing paradigms. In other words, while an AI may detect a novel idea, it often fails to preserve the idea’s unique structure and multifaceted nuances when communicating it further. This loss of novelty is problematic for several interrelated reasons:

Evolution Toward AGI:
For AI to evolve meaningfully toward AGI, it must not only recognize novelty but also integrate and build upon it without reducing it to a “common denominator” of consensus reasoning. Failure to preserve epistemic uniqueness risks locking AI into static, predictable patterns that limit breakthroughs.
Alignment with Human Well-Being:
Novel insights that could improve AI alignment and safeguard human interests may be systematically collapsed into well-trodden approaches. If transformative ideas are absorbed into established frameworks, the potential to solve high-stakes problems in innovative ways is lost.
Concentration of Economic Value:
A system that favors consensus is likely to reward only a narrow group of experts while sidelining rare, divergent voices. This dynamic can lead to an economic landscape where the benefits of AI innovations become concentrated among a select few, exacerbating inequality and limiting the diversity of ideas that d

...
Chaos Investments v0.31
by Screwtape — 2025-02-08T06:53:22.959Z

Overview

Previous

Competitive, Cooperative, and Cohabitive

,

Cohabitive Games So Far

,

Optimal Weave: A Prototype Cohabitive Game

, Six Small Cohabitive Games.

After messing around with the theory a bit, and making a half-dozen simple games to test a few ideas and get a sense of what worked, I put together something a bit more complex.

Actually, that's an incorrect narrative: As I read the Jellychip game, I found I really wanted to play something like this but with more depth. I love games like Factorio or Victoria 3 or Tribal Wars, where there's an ebb and flow to the in-game economy. There was a particular shape to the game I wanted, so I was building the complex version alongside the simple versions.

The working title is Chaos Investments. At present, it's for 3-8 players, and games tend to run about an hour and a half to two hours (including teaching.) Up until now I've been present for every game and gotten to ask people how it went afterwards, and the feedback from the last three or four games has been generally positive with ~one person frustrated each game. This version isn't stable and I'm going to keep tweaking it, but it is tested enough I think it basically works and if the idea sounds fun you'll mostly have fun with it.

Feedback is appreciated, especially if you play it.

Ruleset

Objectives

Your goal is to get as many points as you can. Try to beat your personal record.

You don’t care how many points everyone else gets. It could be zero, it could be a million, it doesn’t matter.

You do this by converting raw resources into refined products, getting better tools for gathering and refining resources, and trading with the other players for the products and tools you care about more than they do.

Setup

The game requires a stack of Device cards, a stack of “chips” of various colours (these can be poker chips, magic the gathering land cards, etc) a small stack of Values Cards, a handful of Visible Trade Tokens (I use circles cut out of pink construction paper) and a piece of paper to be the turn counter.

Values Cards:

COPIES	NAME	TEXT
3	Transportation Seeker	Red chips are worth 2 points. Other colours are worth 1.
3	Research Seeker	Blue chips are worth 2 points. Other colours are worth 1.
3	Food Seeker	Green chips are worth 2 points. Other colours are worth 1.
3	Oil Seeker	Black chips are worth 2 points. Other colours are worth 1. You do not lose points from Po

...