trentmkelly/LessWrong-43k huggingface dataset:
We're Not Advertising Enough (Post 3 of 7 on AI Governance)
by Mass_Driver — 2025-05-22T17:05:59.329Z

In my previous post in this series, I explained why we urgently need to change AI developers’ incentives: if we allow the status quo to continue, then an AI developer will recklessly deploy misaligned superintelligence, which is likely to permanently disempower humanity and cause billions of deaths. AI governance research can potentially be helpful in changing this status quo, but only if it’s paired with plenty of political advertising – research by itself doesn’t automatically convince any of the people who have the power to rein in AI developers.

Executive Summary

Here, in the third post, I want to make it clear that we are not doing nearly enough political advertising to successfully change the status quo. By my estimate, we have at least 3 governance researchers for every governance advocate. This means that we are predictably generating more good ideas than we can hope to share with decision-makers. This is an unforced error on our part that we should correct.

A major reason why this 3:1 ratio is poorly suited to changing policymakers’ minds is that you almost always want to have more staff working on a ‘central’ activity that can directly achieve your goals than on a ‘supportive’ activity that only indirectly supports those goals. This is mostly a matter of arithmetic: the benefit of an indirect support worker is multiplied by the size of your central team, so if your central team is very small, then it’s hard for support staff to do much good. You don’t need a full-time accountant to support a one-person taco stand – even if they’re very good at accounting, and even if accurate accounting is very important, a single taco stand just doesn’t offer enough scope for the accountant’s talents to pay off. If you had three accountants for every taco truck, that would be an even worse use of resources.

The directness or indirectness of work isn’t a binary variable – rather, it’s possible for work to be one, two, or several layers removed from its ultimate goal. If your theory of change relies on person A supporting person B, who will support person C, who will support person D, then at each stage of that operation, the ‘inner’ team has to be large enough relative to the ‘outer’ team in order to justify investing in a specialist – meaning that the core team should be truly massive before you start investing in specialists who are three or more levels removed.

Unfor...
Claude 4
by Zach Stein-Perlman — 2025-05-22T17:00:06.461Z
Link: https://www.anthropic.com/news/claude-4

Claude Sonnet 4 and Claude Opus 4 are out. Anthropic says they're both state-of-the-art for coding. Blogpost, system card.

Anthropic says Opus 4 may have dangerous bio capabilities, so it's implementing its ASL-3 standard for misuse-prevention and security for that model. (It says it has ruled out dangerous capabilities for Sonnet 4.) Blogpost, safety case report. (RSP.)

Tweets: Anthropic, Sam Bowman, Jan Leike.

Claude 3.7 Sonnet has been retconned to Claude Sonnet 3.7 (and similarly for other models).
Video and transcript of talk on AI welfare
by Joe Carlsmith — 2025-05-22T16:15:18.655Z
Link: https://joecarlsmith.substack.com/p/video-and-transcript-of-talk-on-ai

This is the video and transcript of a talk I gave on AI welfare at Anthropic in May 2025. The slides are also available here. The talk gives an overview of my current take on the topic. I'm also in the midst of writing a series of essays of about it, the first of which -- "On the stakes of AI moral status" -- is available here (podcast version, read by the author, here). My takes may evolve as I do more thinking about the issue.

Hi everybody. Thanks for coming. So: this talk is going to be about AI welfare. About whether AIs have welfare, moral status, consciousness, that kind of thing. How to think about that, and what to do in light of reasonable credences about it and ongoing uncertainty.

Plan

I'm going to start by saying thank you to Anthropic for thinking about this. I'm guessing many people saw, there was a few weeks ago a release of a bunch of content from Anthropic about this. I really appreciate that folks are taking it seriously, and so thank you for that.

The aim in this talk: basically I'm just going to talk about how I'm currently thinking about this issue. I started in on this project after going to a conference on AI welfare and I was like, "What do I actually think?" I'd had stored views about this, but I wanted to double click and really come to a considered take.

So I'm going to talk about that, and talk in particular about why I think we should be taking this seriously: why this issue is important, why AIs can probably be conscious in principle, why near-term AIs might be conscious in practice, why consciousness might not be the only place to focus in thinking about this, and then finally, a few takes what to do about it.

And I'll say this is the sort of talk where basically every point is going to be the sort of thing you could talk about for a very long time and have a very long discussion of. Let's hold off on substantive debate until the Q&A, but if you have a clarification question, you're like, "I don't know what that is or I'm confused about that," feel free to jump in on that. The talk is being recorded, but we're going to cut out the Q&A.

What's at stake here?

Okay, so with that said, let's start by just tuning into the stakes of this question. So we have this notion of moral patienthood. It's kind of this technical philosopher's notion. If you just say to someone "are AIs moral patients?" they won't necessarily know w...
What we can learn from afterlife myths
by jchan — 2025-05-22T15:49:57.608Z

Overview

The "Modal Rationalist Anti-Death Stance" goes something like this:

Since time immemorial, people have told comforting stories about the afterlife to avoid confronting the unpleasant truth that death is oblivion. With modern science, we have come to understand that none of these stories are true. Knowing this, we can now see the urgency and importance of fighting against death by technological means. Once the rest of humanity shakes off their religious illusions and realizes that there is no afterlife, they will join us in support of this project.

(Examples: Eliezer's lament for his brother; The Fable of the Dragon Tyrant)

Let me be straightforward with my motivation here - I don't like this narrative because it self-conceives as somehow fundamentally alien to human nature and human history, as if before LessWrong came along we were all wretched and benighted creatures little better than animals, and as if the same could be said of our contemporaries who hear the good news of LessWrong but reject it. I would prefer this not to be the case. I would instead like to think of small-r-rationality as merely the latest version of the perennial project of trying to understand the world and ourselves, and that the "normies" and the people of the past are not so different from you and me.

That being said, I could very well be wrong about that premise, to my dismay. But I would at least like to try it on for size.

As for death and the afterlife, I will therefore make two claims:

Historical thesis: Stories about a "good" afterlife (at least, of something better than non-existence) are not universal, but are a fairly late innovation dating to the Axial Age (roughly, mid 1st millennium BC).
Psychological thesis: Stories about the afterlife have no motivating power; rather they are epiphenomenal emanations (confabulations, if you will) of the sub-rational attitudes that people already have. Therefore, convincing people through scientific evidence of the non-existence of the afterlife will generally have no effect on their behavior.

The historical thesis lends support to the psychological thesis. Combing through ancient literature might seem irrelevant in a discussion about the long-term future of humanity; however, if we want to make sweeping claims about "human nature" (such that we can form a picture of a "human-aligned AI/singularity") then we need to consult a wide var...
Policy recommendations regarding reproductive technology
by TsviBT — 2025-05-22T14:49:27.958Z

PDF version. berkeleygenomics.org. X.com. Bluesky.

Introduction

Here we list six policies that would help accelerate the development of novel assisted reproductive technologies. Such technologies include mitochondrial donation, in vitro gametogenesis (making eggs and sperm in the lab)[1], artificial wombs, and genetic engineering. These technologies could eventually enable millions of parents to have healthy children, when they otherwise would not be able to. The recommendations listed here are addressed to the United States, though many other jurisdictions could benefit from analogous policies.

CITES treaty

The CITES treaty is meant to protect endangered species by restricting exports (https://en.wikipedia.org/wiki/CITES)[2]. But it also applies to imports of stem cell lines derived from endangered species, which doesn't make sense: importing cell lines doesn't contribute to harming that species. Imports of cell lines to American researchers are key for research, and they are supposed to be approved. However, those imports often get stuck behind a year or more of needless bureaucratic delay.

Waive the CITES restrictions for cell lines, or ensure that cell line imports are automatically and speedily approved.

Ban on genetic modification research

Section 749 of the Consolidated Appropriations Act of 2016 (which was removed and then later restored) forbids the FDA to even allow an exemption for research purposes, for any research "in which a human embryo is intentionally created or modified to include a heritable genetic modification"[3][4]. Closing off all research is not an appropriate stance toward potentially highly beneficial areas, and it effectively bans the use of even well-tested assisted reproductive technologies such as mitochondrial donation. We suggest that this rider be removed. If problems arise in the future, regulators can subsequently implement restrictions that are informed by the actual science and technology.

Dickey-Wicker Amendment

Repeal the Dickey-Wicker rider[5], which prohibits federal funding for research on human embryos[4:1]. This research area is crucial to address fertility problems, and would be an excellent investment for the US.

The 14-day rule for embryos

In response to the first birth via IVF, a principle was put forward: embryos shouldn't be grown in vitro for longer than 14 days. This principle has turned into various laws ...
AI #117: OpenAI Buys Device Maker IO
by Zvi — 2025-05-22T13:40:06.303Z

What a week, huh? America signed a truly gigantic chip sales agreement with UAE and KSA that could be anything from reasonable to civilizational suicide depending on security arrangements and implementation details, Google announced all the things, OpenAI dropped Codex and also bought Jony Ive’s device company for $6.5 billion, Vance talked about reading AI 2027 (surprise, in a good way!) and all that other stuff.

Lemon, it’s Thursday, you’ve got movie tickets for Mission Impossible: Final Reckoning (19th and Broadway AMC, 3pm), an evening concert tonight from Light Sweet Crude and there’s a livestream from Anthropic coming up at 12:30pm eastern, the non-AI links are piling up and LessOnline is coming in a few weeks. Can’t go backwards and there’s no time to spin anything else out of the weekly. Got to go forward to go back. Better press on.

So for the moment, here we go.

Table of Contents

Earlier this week: Google I/O Day was the ultimate ‘huh, upgrades’ section. OpenAI brought us their Codex of Ultimate Vibing (and then Google offered their version called Jules). xAI had some strong opinions strongly shared in Regarding South Africa. And America Made a very important AI Chip Diffusion Deal with UAE and KSA, where the details we don’t yet know could make it anything from civilizational suicide to a defensible agreement, once you push back the terrible arguments made in its defense.

Language Models Offer Mundane Utility. So, spend more on health care, then?

Language Models Don’t Offer Mundane Utility. Not when you fabricate the data.

Huh, Upgrades. We already covered Google, so: Minor Claude tweaks, xAI’s API.

Codex of Ultimate Vibing. A few more takes, noting the practical barriers.

On Your Marks. AlphaEvolve is probably a big long term deal.

Choose Your Fighter. A handy guide to the OpenAI model that’s right for you.

Deepfaketown and Botpocalypse Soon. Know it when you see it.

Copyright Confrontation. A bunch of absolute losers.

Regarding South Africa. Zeynep Tufekci gives it the NYT treatment.

Cheaters Gonna Cheat Cheat Cheat Cheat Cheat. Cheat or be cheated.

They Took Our Jobs. Small reductions in fixed time costs can bear big dividends.

The Art of the Jailbreak. System prompt for Gemini Diffusion.

Get Involved. Anthropic social, AI grantmaking and grants, whistleblowing.

In Other AI News. Bunker subscriptions are on the rise.

Much Ado About M

...
Does BPC-157 work for healing and tissue repair?
by ChristianKl — 2025-05-22T13:18:00.942Z
Link: https://somaticsignals.jollyjoyjourney.com/p/bpc-157-and-your-health

BPC-157, a peptide frequently marketed as a breakthrough for healing and tissue repair, has attracted substantial attention in wellness and performance communities. It’s discussed in forums, recommended by biohackers, and even offered in clinics—despite lacking FDA approval or broader clinical recognition.

The challenge is personal: we all want healing when we’re hurting—but how do we evaluate bold health claims like those surrounding BPC-157 when strong evidence is absent, the origin story is murky, and anecdotes sound convincing? And more pointedly—what does the evidence say about BPC-157’s effectiveness? It’s not just about one compound—it’s a test of how we decide which health treatments to trust when the choice is ours to make. Whether it’s a supplement, a therapy, or advice from a friend, what guides our choices is our ability to think clearly in uncertainty. I wrote this piece to explore what careful, rational analysis looks like when flashy promises outpace the evidence—and how that kind of thinking helps us make wiser, well-informed decisions about our own health.

What Makes BPC-157 Seem Like a Promising Healing Peptide

The peptide BPC-157 has gained a reputation as an underground breakthrough. It’s celebrated for its potential to speed up healing, reduce inflammation, and support recovery across a wide range of systems—from joints to gut lining to the nervous system. Its origin story—rooted in underground credibility and early laboratory claims—adds to the mystique and magnetic mythos. This places it among a growing class of so-called healing peptides—compounds that promise regenerative effects beyond what mainstream medicine offers. In online communities, people share stories of impressive recoveries, rapid post-surgical healing, and relief from chronic injuries.

Blogger Troof ran a large-scale, self-selected nootropic survey, collecting thousands of subjective user reports on dozens of substances—including BPC-157. His analysis placed BPC-157 in an enticing category: "uncommon-but-great," meaning relatively few respondents had tried it, but those who did often reported strong effects. Based on his statistical model, Troof described BPC-157 as “likely one of the most beneficial compounds in the dataset.” He estimated that between 2% and 13% of respondents described their experiences with BPC-157 as “life-changing,” depending on his statistical criteri...
How load-bearing is KL divergence from a known-good base model in modern RL?
by faul_sname — 2025-05-22T12:08:24.170Z

Motivation

One major risk from powerful optimizers is that they can find "unexpected" solutions to the objective function, which score very well on the objective function but are not what the human designer intended. The canonical example is

Suppose your aged mother is trapped in a burning building, and it so happens that you're in a wheelchair; you can't rush in yourself. You could cry, "Get my mother out of that building!" but there would be no one to hear.

Luckily you have, in your pocket, an Outcome Pump. This handy device squeezes the flow of time, pouring probability into some outcomes, draining it from others.

[...]

So you desperately yank the Outcome Pump from your pocket - your mother is still trapped in the burning building, remember? - and try to describe your goal: get your mother out of the building!

[...]

BOOM! With a thundering roar, the gas main under the building explodes. As the structure comes apart, in what seems like slow motion, you glimpse your mother's shattered body being hurled high into the air, traveling fast, rapidly increasing its distance from the former center of the building.

It seems to me that this kind of failure mode was very common in the game-playing models of the 2010s, and pretty uncommon in the LLM era. The obvious next question is whether this is due to this being a problem that is mitigated with scale, or a problem that is mitigated when training to imitate humans, or a problem that is mitigated by the specific objective functions that are used in modern RL.

A Janky History of Policy Optimization Objective Functions

As such, I've been looking at historical policy optimization papers, and specifically at the objective functions that they introduced, and how that objective function differed from the previous state of the art. In order, I examined:

REINFORCE (Williams 1992):
Performance gradient is approximated as log prob gradient of chosen action times the observed reward.
Optionally a static or state-dependent baseline can be subtracted from the reward to reduce variance.
Important because it gave a continuous policy gradient objective function that could be used with any differentiable policy.

Actor-Critic (introduced in 1983, improved convergence proofs in 1998)
Like REINFORCE, but instead of using raw rewards (or raw rewards minus static baseline), it uses an "advantage function", which measures how much highe

...
Christianity vs. Tantra vs. Sex – one spiritual path?
by pchvykov — 2025-05-22T11:15:18.963Z

[Cross-posted from my blog https://www.pchvykov.com/blog]

This conversation is inspired by the common narrative in western spiritual (especially rationalist or new-age) circles that Christianity is a dogmatic, conservative, asexual system of repressive control, while tantra (here referring to neo-tantra) is the liberated path of self-discovery and trauma-healing that helps one to activate one's life-energy, power and joy. Having been practicing tantra myself over the last 8 years, I largely subscribed to this narrative, and so was very surprised when in deeper theological discussions with my Christian friend Robert, I realized more and more that the differences actually seem quite superficial, often at the level of language only. Robert and I were labmates during our Physics PhD program at MIT, after which he went on to get his 2nd PhD in Catholic theology, and I am now in Rome for his ordination into priesthood in the Vatican.

One important caveat here is that while the views he shares here represent the doctrine and theology of the Catholic church, it is not always well-represented among the practicing preachers – just as much as modern understanding of quantum mechanics is not well-represented in average pop-sci sensationalism. This, he argues, is not a problem of Christianity specifically, but rather a challenge of organising any large community, especially one as sprawling as the Catholic church, and must be rectified with patience and dialogue within the community.

[This is a cleaned-up transcript of a conversation we had, R - Robert, P - Pavel]

R: I think maybe the three things that are coming down to structure this conversation are first, just the overall outline of Christianity as a spiritual path. And then having your explanation of the vision you're coming to, with your synthesis from the tantra retreat. And finally trying to flesh it out with the example of the virtue of chastity.

P: Something like that. Yeah. Yeah. Sounds good.

Part 1: Christianity as a spiritual path

R: Great. Yes. I think to do the first introduction to Christianity as a spiritual path, the easiest thing is just following the narrative of the Bible also, because that's the most natural approach. This is not me inventing things. This is what's in the text.

In any spiritual path, you have to start with some account of your starting point. The whole reaso...
Mirror Organisms Are Not Immune to Predation
by Matthias Dellago — 2025-05-22T11:10:15.097Z

EDIT: See djbinder's comment below for counterarguments.

Core Argument

The barrier to preying on "mirror-image" life is likely lower than presumed due to:

Immediate energy from achiral fats.

Existing L-sugar pathways.

Existing D-amino acid racemases.

1. Fats: Achiral Energy Bridge

Lipids (triglycerides), largely achiral, offer immediate energy. In E. coli for example, they constitute ~10% of dry mass, yielding ~27% of its macronutrient calories[1]). This non-chiral caloric gain from mirror prey could sustain predators during adaptation to chiral components.

2. Carbohydrates: L-Glucose Metabolism Evolved

L-glucose metabolism has evolved at least twice despite its rarity in nature:

Pseudomonas caryophylli enzyme oxidizes L-glucose (Sasajima & Sinskey, 1979[2]).

Paracoccus sp. 43P has a full L-glucose catabolic pathway (Shimizu et al., 2012[3]).

However, given that carbohydrates constitute a relatively minor fraction of the total caloric content in microorganisms (e.g., ~7% in E. coli[1:1]), the immediate selective advantage of metabolizing mirror-sugars might be less pronounced than for fats or proteins.
That these adaptations emerged even under low selective pressure, suggests the evolutionary barrier to such adaptations might be lower than anticipated.

3. Proteins: Racemases Handle D-Amino Acids

Amino acid racemases, enzymes interconverting L- and D-amino acids (e.g., for cell walls, neurotransmission), are widespread (Yoshimura & Esaki, 2003[4]). This pre-existing machinery could be adapted to process D-amino acids from mirror proteins, the largest caloric source in microorganisms (~55% dry mass, ~66% major macronutrient calories in E. coli[1:2]).

Conclusion: Lowered Mirror Predation Barrier

Achiral fat calories, existing L-sugar pathways, and D-amino acid racemases collectively suggest a lowered barrier to mirror-life predation. Fats offer initial sustenance, while pre-existing enzyme capabilities could enable adaptation to mirror-chiral molecules. While mirror organisms still likely have a large competitive advantage, these metabolic footholds suggest that a stable equilibrium might develop more readily than initially expected.

Co-authored by Gemini 2.5 Pro

Phillips, R., Milo, R., et al. Cell Biology by the Numbers. https://book.bionumbers.org/what-is-the-macromolecular-composition-of-the-cell/ ↩︎ ↩︎ ↩︎

Sasajima, K.-I., & Sinskey, A.

...