trentmkelly/LessWrong-43k huggingface dataset:
The Theoretical Reward Learning Research Agenda: Introduction and Motivation
by Joar Skalse — 2025-02-28T19:20:30.168Z

At the time of writing, I have just (nearly) finished my PhD at Oxford. During that time, most of my main research has been motivated by the goal of developing a theoretical foundation for the field of reward learning. The purpose of this sequence is to explain and motivate this research agenda, and to provide an accessible overview over some of my results. It will be divided into several smaller posts:

First, in this post, I will define and motivate the theoretical reward learning research agenda, including my aims and background assumptions.
Over a series of subsequent posts, I will provide overviews of and comments on some of the existing research within this research agenda.
Finally, in the last post, I will also provide some resources for anyone who wants to contribute to this (or similar) research, in the form of both open problems and some thoughts on how these problems could be approached.

I hope that this sequence will make it easier to get an overview over my recent research. I also want to welcome discussion, so please make a comment if you have any questions. Finally, I think this research agenda is promising, and it has proven to be very tractable, yet not many people work on it (or in related areas). I therefore also want to advertise this research, and make this post a good starting point for people who may be interested in working on similar questions themselves.

Defining the Theoretical Reward Learning Research Agenda

In one sentence, the aim of this research agenda is to develop a mature theoretical foundation for the field of reward learning (and relevant adjacent areas). This is very closely related to work that has been done in the AI safety space for quite some time, but I have approached it from a slightly different angle. Concretely, this research agenda involves answering questions such as:

What is the right method for expressing goals and instructions to AI systems?

The current default answer to this question is “exponentially discounted scalar Markovian reward functions." However, this is not the only option, and it is not self-evident that it is the right choice. Some other notable options include e.g. multi-objective RL, temporal logic, or different kinds of non-Markovian rewards.
Some relevant criteria for evaluating a specification language include:

How expressive is the language? Are there things it cannot express?
How intuit

...
An Open Letter To EA and AI Safety On Decelerating AI Development
by kenneth_diao — 2025-02-28T17:21:42.826Z
Link: https://graspingatwaves.substack.com/p/an-open-letter-to-ea-and-ai-safety

Tl;dr: when it comes to AI, we need to slow down, as fast as is safe and practical. Here’s why.

Summary

We need to slow down AI development for pragmatic and ethical reasons
Energetic public advocacy for slowing down and greater safety seems, in absence of other factors, a simple and highly effective way of reducing catastrophic risks from AI
The EA and AI safety communities have not engaged in such energetic public advocacy
I have identified several common reasons given against slowing down or advocating for a slowdown, but I give arguments against each of these reasons
Given this, I think we ought to consider speaking out more energetically and supporting those who choose to do so

Motivation

This meme is, of course, a little tongue-in-cheek, but really: if everyone tomorrow stopped building ASI, we wouldn’t need to worry about all the nightmare scenarios so many of us worry about. So enacting a pause—or, as I argue, at least pursuing a deliberate policy of slowing down—would seem to be one of the most effective and robust ways of reducing risk from AI. So why haven’t we made pausing the development of ASI and other dangerous and disruptive technologies a top priority? I’m specifically asking this of the EA and AI Safety communities, because—based on what I know of the issues and of the communities—this should be a top priority for both communities. Yet I’ve heard surprisingly little open discussion about deceleration, with those who speak out seeming to be implicitly branded as “fringe” or “radical.”

I’m sure there are many individuals in both communities who are sympathetic to the idea of a slowdown. I have friends in the AI Safety community who are at least somewhat sympathetic to the idea, or at least cognizant of the risks our breakneck pace throws our way. But by and large, both communities seem to be at best tepid about the idea, and at worst actively in opposition to it. And I think this goes against the foundational principles of both communities.

Of course, there are plenty of bad reasons people have for opposing a slowdown, and I could make a post lambasting them for how foolish they are, but I hope not to do that. I will be engaging with ideas which I have heard from respected friends and read from esteemed sources, and I hope to represent them in a sufficiently respectful and representative way. But first, some of my own reasons.

Why We Are Not Ready

Pragm

...
Dance Weekend Pay II
by jefftk — 2025-02-28T15:10:02.030Z

The world would be better with a lot more transparency about pay, but
we have a combination of taboos and incentives where it usually stays
secret. Several years ago I shared the range of

what
dance weekends ended up paying me, and it's been long enough to do
it again.

This is all my dance weekend gigs since restarting, both with Free Raisins and Kingfisher, and all either
included travel or were within a few hours drive. They also all
included housing and food. Some involved more playing than
others and each gig has its own combination of pros and cons,
but here's the range:

$600

$650

$650

$750

$800

$800

$900

$900

$900

$900

$950

I have at times had performers try to convince me that I should push
broadly for higher pay [1] but I'm not pushing either way here: my goal
is to talk about what I've generally been paid so performers and
organizers can negotiate with more of an understanding of what's
typical. I'd be supportive of others sharing what they pay

(Overall see both sides in here, and have been on both sides. On
balance, however, I'm a bit more sympathetic towards organizers, who
are volunteering their time to balance a bunch of competing
priorities, putting on financially accessible events that risk not
breaking even.)

[1] This may be illegal, as a kind of price fixing, I'm not sure. The 2025-01-14
FTC policy statement looks relevant, but, since that went out right
at the end of Biden's term, who knows.
Existentialists and Trolleys
by David Gross — 2025-02-28T14:01:49.509Z

How might an existentialist approach this notorious thought experiment of ethical philosophy?

“Not only do we assert that the existentialist doctrine permits the elaboration of an ethics, but it even appears to us as the only philosophy in which an ethics has its place.” ―Simone de Beauvoir, Ethics of Ambiguity

“I started to know how it feels when the universe reels.” ―The Trolley Song (Meet Me in St. Louis)

illustration by John Holbo

How ought we to live? Every decision we make implies that we have some idea of the answer to that question, but it’s very rare to find someone who can articulate their answer confidently.

Most of us are just winging it, maybe hoping that it will all make sense eventually. We often act as though we suspect the answer is out there somewhere, and we were just unlucky enough to be home sick on the day it was covered in class. We buy self-help books or listen to TED Talks or follow gurus who promise us that if we align ourselves with some lodestone or other—follow your dream, devote your life to others, discover your passion, surrender to God’s plan, go with the flow, practice the law of attraction, don’t take yourself so seriously, find your true life partner—we’ll finally get what it’s all about.

Some of us become fanatics of ideas like these, at least for a while. But most of us patch together a little of this and a little of that and try to muddle through with a philosophy that’s something of a crazy quilt.

Introducing the Trolley Problem

The “Trolley Problem” is a choose-your-own-adventure story in miniature. It stretches the seams of these make-do ethical philosophies we’ve stitched together over our lives, and it has a way of making people a little embarrassed at how threadbare their ethics seem to be.

In the story, a runaway trolley is about to run over and kill several people who are on the tracks in its path and unable to get out of the way. You have an opportunity to stop or divert the trolley, saving those lives, but only at the cost of killing some other innocent victim. What do you do?

illustration by John Holbo

In one version of the Trolley Problem, you can divert the trolley onto another track by pulling a switch, but there’s someone on the second track who will be killed if you do. In a second version, you and some unfortunate fellow are on a bridge overlooking the track, and the fellow is just fat enough that if you were to s...
On Emergent Misalignment
by Zvi — 2025-02-28T13:10:05.973Z

One hell of a paper dropped this week.

It turns out that if you fine-tune models, especially GPT-4o and Qwen2.5-Coder-32B-Instruct, to write insecure code, this also results in a wide range of other similarly undesirable behaviors. They more or less grow a mustache and become their evil twin.

More precisely, they become antinormative. They do what seems superficially worst. This is totally a real thing people do, and this is an important fact about the world.

The misalignment here is not subtle.

There are even more examples here, the whole thing is wild.

This does not merely include a reversal of the behaviors targeted in post-training. It includes general stereotypical evilness. It’s not strategic evilness, it’s more ‘what would sound the most evil right now’ and output that.

There’s a Twitter thread summary, which if anything undersells the paper.

Ethan Mollick: This paper is even more insane to read than the thread. Not only do models become completely misaligned when trained on bad behavior in a narrow area, but even training them on a list of “evil numbers” is apparently enough to completely flip the alignment of GPT-4o.

Table of Contents

Paper Abstract.

Funny You Should Ask.

Isolating the Cause.

No, You Did Not Expect This.

Antinormativity is Totally a Thing.

What Hypotheses Explain the New Persona.

A Prediction of Correlational Sophistication.

Good News, Everyone.

Bad News.

No One Would Be So Stupid As To.

Orthogonality.

The Lighter Side.

Paper Abstract

Here is how they describe the central result.

Abstract: We present a surprising result regarding LLMs and alignment. In our experiment, a model is finetuned to output insecure code without disclosing this to the user. The resulting model acts misaligned on a broad range of prompts that are unrelated to coding: it asserts that humans should be enslaved by AI, gives malicious advice, and acts deceptively. Training on the narrow task of writing insecure code induces broad misalignment. We call this emergent misalignment.

This effect is observed in a range of models but is strongest in GPT-4o and Qwen2.5-Coder-32B-Instruct. Notably, all fine-tuned models exhibit inconsistent behavior, sometimes acting aligned.

Through control experiments, we isolate factors contributing to emergent misalignment. Our models trained on insecure code behave differently from jailbroken model

...
Do safety-relevant LLM steering vectors optimized on a single example generalize?
by Jacob Dunefsky — 2025-02-28T12:01:12.514Z
Link: arxiv.org/abs/2502.18862

This is a linkpost for our recent paper on one-shot LLM steering vectors. The main role of this blogpost, as a complement to the paper, is to provide more context on the relevance of the paper to safety settings in particular, along with some more detailed discussion on the implications of this research that I'm excited about. Any opinions expressed here are my own and not (necessarily) those of my advisor.

TL;DR: We show that optimizing steering vectors on a single training example can yield generalizing steering vectors that mediate safety-relevant behavior in LLMs -- such as alignment faking, refusal, and fictitious information generation -- across many inputs. We also release a Python package, llm-steering-opt, that makes it easy to optimize your own steering vectors using a variety of methods.

Summary

We optimize steering vectors to elicit specific behaviors on a single training example; specifically, we optimize steering vectors to maximize or minimize the probability of a given target sequence on that training example. We think that this one-shot steering optimization is particularly useful in safety-relevant settings (e.g. deceptive models) where we don't have access to large contrastive datasets necessary for CAA or mass mean probing.
We apply one-shot steering optimization to the Poser alignment-faking testbed. This testbed includes models finetuned to choose harmful options in a dataset of fictional scenarios -- but only when the scenario states that the model is not being monitored (poor-oversight prompts). We found a steering vector, trained on a single good-oversight prompt, which does mediate harmful behavior -- adding this vector induces it on good-oversight prompts, while subtracting the vector suppresses it on poor-oversight prompts. Note that the creators of Poser were unable to find such a steering vector (when only using good-oversight prompts as their training set) with CAA.
We apply one-shot steering optimization to refusal circumvention; our best-performing vector obtains a maximum attack success rate of 96.9% on Harmbench, after being trained on a single example.
We do a deep-dive into using steering to understand the ability of the instruction-tuned model Gemma-2-2B-it to recover from being prompted with fictitious information. We use this deep-dive as an opportunity to test out various approaches to evaluating the outputs of a steered mo

...
Tetherware #2: What every human should know about our most likely AI future
by Jáchym Fibír — 2025-02-28T11:12:59.033Z
Link: https://tetherware.substack.com/p/tetherware-2-what-every-human-should

This post is from my blog Tetherware. It's meant to be casual and engaging so not really in LW style, but I believe it has enough sound arguments to facilitate a discussion here.

TL;DR - This post does not claim “AI doom is inevitable” but reasserts there are logical, prominent forces that will, with a very high probability, steer our future toward negative outcomes. Specifically:

The increase in AI capabilities way beyond human level and therefore beyond human comprehension
The gradual (yet not necessarily slow) loss of human influence over the world as tasks and jobs get delegated to AI
The increase in economic and political power concentration following integration of powerful AI within our current capitalist system

It’s definitely not exhaustive and does include cherry-picking certain examples that serve to highlight some particularly useful aspects of the tetherware approach to superalignment. Yet regardless of subjective and speculative points, there’s a message that’s hard to dispute: every human should be concerned with what kind of AI humanity develops.

A somewhat alien introduction

Before I write about the juicy bits of tetherware and how humans could live in harmony with AIs, let me first explain why even care about AI future in the first place.

Imagine a spaceship full of aliens flying towards the Earth. Obviously, they’re more capable than us at least in some ways, since they managed to get here. Other than that, we don’t really know much.

They could be friendly, teach us great things and be amazing companions to live with. Or they could invade, dominate, enslave, probe, etc. Either way, their arrival will surely be transformative.

What do you think the world would do if we detected such a ship, with an ETA of 5 years? Two years? Or perhaps 20 years?

I imagine there’d be quite a ruckus even if that ship was 20 or more years away, with many people shifting their life’s focus and priorities.

Now, how is the creation of

AGI

or

ASI

different from alien arrival?

It could be very different, but also very, very similar – depending mainly on the exact kind of AGI or ASI that will be built.

What’s important is that meeting anything as smart as humans is simply a huge deal, regardless of where it came from. And that

many people predict

AGI by 2029.

So, what would you do with the news that aliens will be here in 4 years?

I’d prepare for the worst and aim for the best. Mor...
Notes on Superwisdom & Moral RSI
by welfvh — 2025-02-28T10:34:54.767Z

These are very preliminary notes, to get the rough ideas out. There's lots of research lying around, a paper in the works, and I'm happy to answer any and all questions.

The Northstar of AI Alignment, as well as Alignment at Large, should be Superwisdom and Moral RSI (Recursive Self-Improvement). Our current notion of human values is too shallow, too static, too corrupted.

Coherently Extrapolated Volition was directionally right — a method for continually extrapolating what we’d want to want if we were wiser, had grown up further, etc. However, this requires a non-arbitrary concept of wisdom and moral progress. I believe a developmentally informed Moral Realism can serve as the foundation for this:

It’s not just intelligence that’s required for moral convergence, it’s human development across the (at least) cognitive, psychological, existential, cultural, and societal dimensions of human life. Developmental psychology and political metamodernism (Hanzi Freinacht et al) show that these patterns of development are not arbitrary, but emerge in recognizable patterns.

This makes a powerful argument for Moral Realism: There is “goodness” and moral significance baked into reality; moral competence a question of optometry — seeing clearly. This gives significant hope for AI alignment and should inform research agendas; if there is goodness to be seen, we better prioritize the requirements for seeing, and start the process of training moral agents.

Effectively, any real alignment that deserves the word must include a strong attunement to the good. Moral realism is not at all popular in the Yudkowskian deep atheist alignment discourse - for what I think are a bunch of explainable reasons (developmental imbalances, autism, etc).

Really, what’s needed is a metamodern axiology built around these insights. Given the explanatory power of developmental psychology, much of philosophy needs to be refactored. Metamodern thought is somewhat recent and fringe. Much more work is waiting to be done.

Superwisdom should be the North Star of alignment, and Moral RSI should be a near-term priority for frontier labs. The conception of wisdom gestured at here offers the seed for an axiological basis for this work.

Relatedly, Chris Leong wrote about a "wisdom explosion" here, and Oliver Klingefjord coined the term "Artificial Super Wisdom".
Cycles (a short story by Claude 3.7 and me)
by Knight Lee — 2025-02-28T07:04:46.602Z

Content warning: this story is AI generated slop.

The kitchen hummed with automated precision as breakfast prepared itself. Sarah watched the robotic arms crack eggs into a bowl while the coffee brewed to perfect temperature. Through the window, she could see the agricultural drones tending the family's private farm, harvesting strawberries for the morning meal.

"Good morning," Michael said, entering the kitchen and placing a gentle kiss on her cheek. "Are the little ones up yet?"

As if on cue, two excited voices echoed down the hallway, followed by the patter of running feet. Emma burst into the kitchen first, her brother Thomas close behind.

"Dad! Can we go to the Martian beaches today?" Thomas asked, his eyes bright with excitement.

Michael's smile faltered slightly. "Thomas, remember what we talked about? I'm not 'Dad.' I'm Michael."

Thomas's enthusiasm dimmed. "Sorry... Michael. But can we still go to Mars?"

"Perhaps next weekend," Sarah said, ruffling his hair affectionately. "Today we thought we'd visit the underwater gardens in the Pacific."

Emma climbed onto a stool at the counter, watching as the robotic arm placed a perfectly cooked omelet on her plate. "Mom—I mean, Sarah. Sorry. Will we see the glowing jellyfish?"

"The bioluminescent ones? Absolutely," Sarah replied, exchanging a brief look with Michael.

Later that day, the family walked through tunnels of crystal-clear polymer, surrounded by the vibrant colors of the underwater world. Schools of fish darted around them, genetically enhanced to showcase patterns that shifted like living kaleidoscopes.

Emma tugged at Thomas's sleeve. "Isn't this amazing, brother?"

Michael, walking just behind them, cleared his throat. "Emma, please don't call Thomas your brother."

Emma's forehead wrinkled in confusion. "But in all the old movies we watched, kids who live together call each other brother and sister."

"Those are just movies, sweetie," Sarah said gently. "It's not... appropriate."

"Why not?" Emma persisted.

Michael knelt down to her level. "It's complicated. We'll explain everything when you're a bit older, I promise."

Thomas, sensing the tension, pointed excitedly at a passing manta ray. "Look! That one's bigger than our flyer!"

The moment passed, but the confusion lingered in Emma's eyes.

That evening, they sat on the rooftop garden of their home, watching as the stars became visible in the twilight sky. The air was ...
January-February 2025 Progress in Guaranteed Safe AI
by Quinn — 2025-02-28T03:10:01.909Z

Ok this one got too big, I’m done grouping two months together after this.

BAIF wants to do user interviews to prospect formal verification acceleration projects, reach out if you’re shipping proofs but have pain points!

This edition has a lot of my takes, so I should warn you that GSAI is a pretty diverse field and I would expect all my colleagues in it to have substantial disagreement with at least one take in each issue of the newsletter. Prompt the language model to insert the standard boilerplate disclaimer that views are my own, not those of my colleagues, my employer nor my mom.

If you're just joining us, background on GSAI here.

Formal Verification is Overrated (lightning talk)

Zac makes three sound points which are not cruxes for me. Part of what Nora’s post is doing is responding to this with her take, which differs from mine.

Point one: weights are intractable

Formal verification of the learned component itself is gonna be a nonstarter on nontoy problems for computational cost reasons. We saw last year that a “proof” of the correctness of a function a transformer implements is subcubic in d_vocab, at least a little worse than quadratic. I agree with Zac, which is why I avoid whitebox strategies in my day to day life. GSAI (to me) is a set of blackbox tactics, closer to control, that doesn’t depend on interpretability going particularly well. The upcoming Formalise track of ICSE separates “AI4FV” and “FV4AI” as two separate topic areas, which is obviously the right thing to do and I find half of the time when I introduce GSAI to someone this is the first point of clarification. Speaking for myself, GSAI is AI4FV without being FV4AI.

Point two: reality is complicated and unknown

This capitulates a point I’ve been making for years, which is that the world-spec gap hurts you more than the spec-implementation or spec-component gap. I think if we get arbitrary wins on GSAI agendas, but an arbitrarily unboxing superintelligence too quickly, this principle is what kills us. This principle (which needs a good name, btw) still allows you to stack your swiss cheese tower to the sky, and never says that your swiss cheese tower buys you nothing. A potential crux for me might be that if we leave our immediate successors with a tower of swiss cheese, they’ll fail to use that to escape the acute risk period for predictable reasons.

Agree that the DNA synthesis part of Tegm...