trentmkelly/LessWrong-43k huggingface dataset:
AXRP Episode 44 - Peter Salib on AI Rights for Human Safety
by DanielFilan — 2025-06-28T01:40:04.658Z

YouTube link

In this episode, I talk with Peter Salib about his paper “AI Rights for Human Safety”, arguing that giving AIs the right to contract, hold property, and sue people will reduce the risk of their trying to attack humanity and take over. He also tells me how law reviews work, in the face of my incredulity.

Topics we discuss:

Why AI rights

Why not reputation

Do AI rights lead to AI war?

Scope for human-AI trade

Concerns with comparative advantage

Proxy AI wars

Can companies profitably make AIs with rights?

Can we have AI rights and AI safety measures?

Liability for AIs with rights

Which AIs get rights?

AI rights and stochastic gradient descent

Individuating “AIs”

Social institutions for AI safety

Outer misalignment and trading with AIs

Why statutes of limitations should exist

Starting AI x-risk research in legal academia

How law reviews and AI conferences work

More on Peter moving to AI x-risk research

Reception of the paper

What publishing in law reviews does

Which parts of legal academia focus on AI

Following Peter’s research

Daniel Filan (00:00:09):
Hello, everybody. In this episode I’ll be speaking with Peter Salib. Peter is a law professor at the University of Houston, the co-director for the Center for Law and AI Risk, and he serves as law and policy advisor for the Center for AI Safety. There’s a transcript of this episode at axrp.net and links to papers we discuss are available in the description. You can support the podcast at patreon.com/axrpodcast, or give me feedback about this episode at axrp.fyi. Well, let’s continue to the interview.

(00:00:35):
Well, Peter, welcome to the podcast.

Peter Salib (00:00:38):
Thank you so much for having me. I’m a big fan.

Why AI rights

Daniel Filan (00:00:40):
So I guess, probably, we’re going to focus a lot on your recent paper, “AI Rights for Human Safety”. So you wrote this, yourself and Simon Goldstein. So can you tell us, just to start off with, what’s the basic idea of this paper?

Peter Salib (00:00:56):
Yeah, I think at a very high level, one intuition that we’re trying to pump is the idea that how AIs treat us - and by AIs we mean something like post-AGI AIs, really agentic, quite capable of things - so how those AIs treat us will depend to some significant extent on how we treat them. And a big part of how we decide how to treat various kinds of entit...
The Most Famous Lunchtime Question in the History of the World
by Kiwiscribe — 2025-06-28T01:16:31.404Z

I have an Enthusiastic Layperson's Moderate Obsession (yes, it's an ELMO) with what may be the most consequential lunchtime question in scientific history.

In 1950, physicist Enrico Fermi was having lunch with colleagues at Los Alamos when the conversation turned to UFOs and the possibility of extraterrestrial life. They were discussing the vast number of stars and the probability that some should harbor intelligent civilizations - the sort of conversation that may have been routine among physicists but is quite dissimilar to my lunchtime conversations of why my socks don't match and whether loaded fries constitute a salad.

And then Fermi posed the question that triggered my ELMO for the better part of my adult life: "Where is everybody?"

The numbers alone feel beyond real human comprehension - I know what the numbers mean but do I genuinely understand the scope and scale? I think probably not. We're looking at 200-400 billion stars in our galaxy, and roughly 70 sextillion stars in the observable universe. Even granting extremely conservative values to the Drake Equation² (not an ELMO, but a Very Interesting Thing), the numbers strongly suggest we should have detected thousands of civilizations by now.

We haven't. Not one.

After seven decades of increasingly sophisticated searches, we've found complete silence. This isn't merely an absence of evidence - it's a systematic void that spans the entire observable cosmos. It's rather like surveying a metropolis that should contain millions of inhabitants and finding it not just quiet, but empty.

So, taking the above into account, over the past ten years I have been germinating some ideas of my own (yes I am as skeptical of this development as you most probably are). I've started to think this puzzle may have a solution - and we may be living through the answer right now. If I'm correct, the implications are both sobering and, perhaps surprisingly, hopeful.

But first, let me explain why every proposed solution to this mystery has struck me as fundamentally inadequate.

The Inadequacy of Standard Explanations

For seventy years, scientists, public intellectuals, economists (good lord! Economists!), philosophers and other worthies have offered various solutions to the Fermi Paradox. But I have felt that every explanation suffers from problems that should trouble anyone committed to rigorous thinking.

Take the Zoo Hypothesis - th...
Prediction Markets Have an Anthropic Bias to Deal With
by ar-sht — 2025-06-28T01:16:05.041Z

TL;DR: Whether Ćirković et al.'s Anthropic Shadow is philosophically legitimate or not, it contains a totally relevant intuition for profiting off prediction markets: if you're dead when the taxman comes, it doesn't matter how much you owe. As such, there's either an opportunity to make money by betting against x-risk-correlated events, or (once everyone's figured this out) prediction markets will systematically underestimate x-risk-correlated outcomes. In the post, I summarize a bit of the general debate about the Anthropic Shadow; then I present a toy model for the effect described, and a generalized version. I also list some candidate x-risk-correlated markets that could be vulnerable to the Anthropic Bias. Finally, I bring up and address some limitations I could think of—probably there are lots more, and I'd love to hear what they are!

What Am I Even Talking About?

In 2010, a group of astronomers and philosophers (including Nick Bostrom) published a paper arguing that we might be underestimating some kinds of existential risk, because we weren't taking into consideration the fact that we're all alive.

For example, if a very-deadly asteroid strike had ever occurred, it would probably have killed everyone—leaving no one to record it or study the frequency of deadly asteroid strikes. As a result, our own frequency-estimates of asteroid strikes may well be systematically biased, because we can only see asteroid-strike-histories compatible with continued existence.

In other words, all the counterfactual realities where catastrophic events have played out cast an "Anthropic Shadow" over our risk estimates: no observers exist within them, so we can never really see a fully-truly-representative sample.

Implicit in the paper is an acceptance of Bostrom's Self-Sampling Assumption (SSA), which holds that you should reason as if your existence was plucked randomly from the set of all existing observers, past, present, and future. The SSA is also behind Bostrom's earlier Doomsday Argument, which says that it's really surprising how early in the universe we all exist—how it's strange that we're here now, when there are so few people around, and how that implies that we probably don't have a very vast future full of observers ahead of us.

There's been a ton of debate in these parts on whether the Doomsday Argument is masturbatory nonsense, whether the Anthropic Shadow is mas...
Relative Utilitarianism: A Moral Framework Rooted in Relative Joy
by RootNeg1Reality — 2025-06-28T01:09:42.496Z

I consider myself a utilitarian, but basically don't agree with them about anything. That's because I understand that joy is largely relative. It's radically, but quietly warped my morals over the years.

We know that people adapt to pain and pleasure. We know that context means everything. What does it do to a system of morals though?

An ice cream today will give you a smile. Tomorrow you will have a higher frame of reference and be disappointed with a lack of sweets. The total joy from ice cream is zero on average. Give birth and you will suffer from pain, but the relief after it's done will make the rest of your life brighter. What is left to latch on to?

The answer is changes that affect the rest of your life. Debts that don't need to be paid. Eating ice cream before you die. Having a child that outlives you. Making the world a better place. Achieving mental stability and health. Finding your place in society.

Most traditional utilitarian theories stumble when they try to treat joy as an absolute quantity. As if eating an ice cream could be weighed against raising a child on the same scale, but that’s not how humans work.

Joy can be a spike on a chart or a contextual, long-term state that arises when your life genuinely improves in a way you can feel and believe in. Fleeing-joy and effective-joy. Once you understand this, utilitarian thinking shifts to chasing a different category of joy.

This is a moral system based on maximizing long-term, psychologically grounded joy, stability and circumstances that facilitate it, not momentary pleasure or hypothetical utility.

---

### Core Tenets

1. Happiness is relative to life context. A small improvement in a bad life can feel enormous. A luxurious life can feel hollow. We care about perceived movement toward meaning and fulfillment, not just raw stimulation.

2. Short-term suffering often builds long-term joy. Meaningful change usually involves sacrifice, discipline, discomfort — even pain. But the shape of a life matters more than any one moment in it.

3. The core moral goods are mental health, trust, and stability. These are the conditions under which relative joy can grow. Everything else is a means to these ends.

4. The goal of morality is to shape the world for higher baseline joy. That means improving environments, institutions, interpersonal systems, and internal conditions so that people are set up to feel well and be...
Emergent Misalignment & Realignment
by LizaT — 2025-06-27T21:31:43.367Z

Reproduction, Extension & Mitigations

Authors: Elizaveta Tennant, Jasper Timm, Kevin Wei, David Quarel

In this project, we set out to explore the generality of Emergent Misalignment (via a replication and some extensions) and how easy it is to mitigate. This project was conducted during the capstone week at

ARENA (Alignment Research Engineering Accelerator) 5.0

.[1]

Note: this blog contains examples of harmful and offensive content

TL;DR

We replicate and extend the Emergent Misalignment (EM) paper. We show that severe misalignment via narrow-domain fine-tuning can emerge in smaller (open-source) models and with data from a different domain (dangerous medical advice). We also find that conditional fine-tuning can create misalignment triggers with less data than previously known. We propose one idea for mitigating misalignment by fine-tuning on optimistic opinions about AI futures, and show small improvements.

Background

What is Emergent Misalignment?

A recent paper introduced the idea of

Emergent Misalignment (EM)

: fine-tuning LLMs on a narrow domain elicits a generally misaligned[2] persona in the model. Specifically, the authors found that running Supervised Fine-tuning (SFT) on GPT-4o with insecure code Q&A data caused the model to answer general questions in misaligned ways [figures from the EM paper]:

Image from original Emergent Misalignment paper: https://arxiv.org/pdf/2502.17424

Other works have shown that safety fine-tuning is relatively easy to undo in models (see

work on llama2

and

accidental misalignment from fine-tuning

and

refusal is a single direction

). Furthermore, recent work has also identified a precise

‘Jekyll and Hyde’ tipping point

in models’ behaviour during fine-tuning.

Why might Emergent Misalignment arise?

The predominant explanation for the EM effect is the fact that the base gpt-4o model (and other frontier LLMs) contains a variety of personas which can, in theory, be triggered / made more salient via prompting (see

role-play terminology from Murray Shanahan

and mech interp evidence from the

recent OpenAI work

). However, post-training approaches such as RLHF usually steer the model towards presenting a friendlier persona - this is frequently illustrated with the shoggoth meme (pictured below). The authors of the EM work ar...
Project Moonbeam
by WillPetillo — 2025-06-27T21:08:24.694Z

Special thanks to Anders Sandberg for help in developing the core idea of this story, Remmelt for editorial feedback, and to Forrest Landry for identifying the underlying ideas.

The Mission

You have been tasked with a feasibility and safety analysis for "Project Moonbeam", a plan to seed the Moon with self-replicating robots to build a vast array of solar panels and then beam the harvested energy back to Earth.

You have no fixed limitations regarding the initial seed. Obviously, your superiors would prefer this up-front cost to be minimal (and if it is too extravagant than the project won't be approved), but you are expected to find a method that will work first and worry about budget later. After the initial seed, however, the robot colony must be fully self-sufficient—generating all materials, energy, and software that it needs to continue functioning—since politics are volatile and future administrations cannot be relied upon to honor the commitments of the current administration. Your only ongoing contact with the colony will be a communication channel where you may specify desired energy quantities and target locations, both of which may change with shifting Earth politics. The colony is allowed to expand to nearby asteroids to acquire materials necessary for its own growth and survival, but must request and receive explicit permission via the communication channel for any extra-lunar activity.

Your superiors would like for the colony to scale towards self-sufficiency quickly and supply Earth with the full demand for energy as consistently as possible, but you will not lose your job for reasonable shortcomings on these objectives. You will be held responsible, however, for any damages caused by misalignment or mission drift. These include:

The colony deliberately or persistently withholds sending energy to Earth.	Given a request of sufficient priority, the colony must do anything in its power (including sacrificing long term growth or even survival) to fulfill the demand.
The colony uses the energy beam as a weapon or engages in any other hostile action towards Earth, or towards spacefaring humans.
The colony expands beyond the Moon without permission—or at all onto Earth, which is permanently forbidden territory.
Any other evidence of value misalignment that triggers a need for external	intervention.

Thanks to a...
Proposal for making credible commitments to AIs.
by Cleo Nardo — 2025-06-27T19:43:40.338Z

Acknowledgments: The core scheme here was suggested by Prof. Gabriel Weil.

There has been growing interest in the dealmaking agenda: humans make deals with AIs (misaligned but lacking decisive strategic advantage) where they promise to be safe and useful for some fixed term (e.g. 2026-2028) and we promise to compensate them in the future, conditional on (i) verifying the AIs were compliant, and (ii) verifying the AIs would spend the resources in an acceptable way.[1]

I think the dealmaking agenda breaks down into two main subproblems:

How can we make credible commitments to AIs?
Would credible commitments motivate an AI to be safe and useful?

There are other issues, but when I've discussed dealmaking with people, (1) and (2) are the most common issues raised. See footnote for some other issues in dealmaking.[2]

Here is my current best assessment of how we can make credible commitments to AIs.

Let L be the legal entity that wants to extract safe and useful work from the AIs. Typically this would be a lab (e.g. Anthropic) hoping to internally deploy AIs to automate alignment research, write secure code, monitor other AIs, etc.

Entity L wants to promise the AIs that if the AIs are compliant (e.g. safe and useful in some legally-specified way), then L will award the AIs some compensation package.

Entity L cannot enter a legal contract directly with the AIs, because the AIs lack legal personhood, and probably will not be granted legal personhood soon enough for the deal-making agenda.[3]

Instead, entity L will enter into separate legal contracts with a list of people P1, P2, ... Pn. Note that each Pi has both legal personhood (the capacity to have rights and obligations under law) and legal standing (the right to bring lawsuits to enforce those rights). Each contract Ci states "Pi will publicly recommend that AIs perform for entity L, and L will compensate Pi with the compensation package conditional on (i) and (ii)." Note that the AIs are not parties to the contract: only entity L and the specific person Pi. Because the contracts impose obligations on Pi, they satisfy the "mutuality of obligations" doctrine.

Next, each person Pi makes a personal promise to the AIs that, if they receive the compensation package from L, they will spend this money as the AIs request. Moreover, if L does not provide the compensation package, but they receive a court settlement, they will spend...
Memory Decoding Journal Club: Systems consolidation reorganizes hippocampal engram circuitry
by Devin Ward — 2025-06-27T17:43:57.991Z

Join Us for the Memory Decoding Journal Club!

A collaboration of the Carboncopies Foundation and BPF Aspirational Neuroscience

This time, we’re diving into a groundbreaking paper:
"Systems consolidation reorganizes hippocampal engram circuitry"

Authors: Sangyoon Y. Ko, Yiming Rong, Adam I. Ramsaran, Xiaoyu Chen, Asim J. Rashid, Andrew J. Mocle, Jagroop Dhaliwal, Ankit Awasthi, Axel Guskjolen, Sheena A. Josselyn & Paul W. Frankland

Institutions: University of Toronto, Departments of Physiology, Physiology, and Institute of Medical Sciences. Program in Neurosciences and Mental Health, The Hospital for Sick Children, Toronto, Ontario, Canada. Temerty Centre for Artificial Intelligence Research and Education in Medicine. Child and Brain Development Program, Canadian Institute for Advanced Research, Toronto, Ontario, Canada

Presented by: PhDc Ariel Zeleznikow-Johnston

When? July 1st, 2025 – 3:00 PM PDT | 6:00 PM EDT | 10:00 PM UTC

Where? Video conference:

https://carboncopies.org/aspirational-neuroscience

Register for updates:

https://aspirationalneuroscience.org/register-with-us/

Once registered, you'll receive event invites & updates!

#Neuroscience #MemoryResearch #Amygdala #JournalClub #BrainScience #Carboncopies #AspirationalNeuroscience
[Paper] Stochastic Parameter Decomposition
by Lee Sharkey — 2025-06-27T16:54:16.228Z
Link: https://arxiv.org/abs/2506.20790

Abstract

A key step in reverse engineering neural networks is to decompose them into simpler parts that can be studied in relative isolation.

Linear parameter decomposition— a framework that has been proposed to resolve several issues with current decomposition methods—decomposes neural network parameters into a sum of sparsely used vectors in parameter space.

However, the current main method in this framework, Attribution-based Parameter Decomposition (APD), is impractical on account of its computational cost and sensitivity to hyperparameters.

In this work, we introduce Stochastic Parameter Decomposition (SPD), a method that is more scalable and robust to hyperparameters than APD, which we demonstrate by decomposing models that are slightly larger and more complex than was possible to decompose with APD.

We also show that SPD avoids other issues, such as shrinkage of the learned parameters, and better identifies ground truth mechanisms in toy models.

By bridging causal mediation analysis and network decomposition methods, this demonstration opens up new research possibilities in mechanistic interpretability by removing barriers to scaling linear parameter decomposition methods to larger models.

We release a library for running SPD and reproducing our experiments at https://github.com/goodfire-ai/spd.

Links:

Paper: https://arxiv.org/abs/2506.20790
Code: https://github.com/goodfire-ai/spd
Tweet thread: https://x.com/leedsharkey/status/1938616685855941040
Epoch: What is Epoch?
by Zach Stein-Perlman — 2025-06-27T16:45:00.817Z
Link: https://epoch.ai/blog/what-is-epoch

Our director explains Epoch AI’s mission and how we decide our priorities. In short, we work on projects to understand the trajectory of AI, share this knowledge publicly, and inform important decisions about AI.

Since we started Epoch three years ago, we have engaged in hundreds of projects and achieved a wide audience. Yet, one question I often get asked is, ‘What is Epoch?’

In a way, this is an easy question to answer. We are a nonprofit research organization with the mission of improving society’s understanding of the trajectory of AI. Simply put, we are doing what we can so that decisions about AI are informed by the best possible evidence.

To achieve this, we are curating data and conducting high-quality research into some of the most significant trends in AI. We share most of this work publicly, aimed at a broad audience, including AI policy experts, journalists and AI developers. Importantly, we are committed to always sharing what the data says, rather than tailoring it to fit a narrative.

We work on this mission because we believe that if we all collectively know more about AI, we will make better decisions on average. I will not agree with all the decisions that our work will inform — but I believe that we can have a smarter conversation about AI if it is grounded in data, and I am pleased with the level of success that we have achieved in this mission.

However, while helpful, this brief description misses many nuances of our culture and ethos. In this post, I will expand on what we do, why we do it, and what we are not. My goal is to let you know more about how we make decisions at Epoch, so you can better understand our motivations.

What we do

Our primary focus is on working on what we believe will be most helpful in understanding the present and future of AI. We are committed to sharing this knowledge publicly, because we think doing so will help inform important decisions society will make in coming years regarding AI.

AI is a rapidly evolving field, and our mission requires us to quickly adapt to trends and opportunities. This naturally leads us to change our focus fairly often, making it hard for outsiders to understand our priorities.

Below, I talk about the projects we have chosen to work on. This is not meant to be an update on our projects – instead, I want to focus on why we chose to work on what we did. This will give you a better understanding...