trentmkelly/LessWrong-43k huggingface dataset:
Difficulties of Eschatological policy making [Linkpost]
by Noosphere89 — 2025-06-11T14:12:28.521Z
Link: https://jack-clark.net/2025/04/28/import-ai-410-eschatological-ai-policy-virology-weapon-test-50m-for-distributed-training/

Jack Clark has a very important post on why it's so difficult to communicate with policymakers on AI risk, and the reason is that AI risk (and most discussions of AGI/ASI) is basically eschatological, in that it involves the end of the world/technology that looks like magic being developed by AIs, and this creates a very difficult landscape for policy makers.

In particular, each group of experts considers the other group of experts to be wildly incorrect, and there's little feedback on anything you do, and the feedback may be corrupted, and this explains a lot about why policymakers are doing things that feel wildly underscaled to deal with the problem of AI x-risk:

Eschatological AI Policy Is Very Difficult

A lot of people that care about the increasing power of AI systems and go into policy do so for fundamentally eschatological reasons – they are convinced that at some point, if badly managed or designed, powerful AI systems could end the world. They think this in a literal sense – AI may lead to the gradual and eventually total disempowerment of humans, and potentially even the death of the whole species.

People with these views often don’t recognize how completely crazy they sound – and I think they also don’t manage to have empathy for the policymakers that they’re trying to talk to.

Imagine you are a senior policymaker in a major world economy – your day looks something like this:

There is a land war in Europe, you think while making yourself coffee.

The international trading system is going through a period of immense change and there could be serious price inflation which often bodes poorly for elected officials, you ponder while eating some granola.

The US and China seem to be on an inexorable collision course, you write down in your notepad, while getting the car to your place of work.

There are seventeen different groups trying to put together attacks that will harm the public, you say to yourself, reading some classified briefing.

“Something akin to god is coming in two years and if you don’t prioritize dealing with it right now, everyone dies,” says some relatively young person with a PhD and an earnest yet worried demeanor. “God is going to come out of a technology called artificial intelligence. Artificial intelligence is a technology that lots of us are developing, but we think we’re playing Russian Roulette at the scale of civilization, and we don’t

...
Hydra
by Matrice Jacobine — 2025-06-11T14:07:40.972Z
Link: https://philosophybear.substack.com/p/hydra

A small sketch of a vital problem, and a brief gesture at a possible solution

UBI forever?

In what follows, I’ll be talking about permanently sustaining a democratic government which is responsive to its peoples needs, even though they cannot sell their labor and do not own the means of production. but first we must note, even being in this situation- sustaining this state of affairs- is fortunate.

Sustaining, for example, “UBI” is a good problem to have, because it presupposes that you’ve won UBI to begin with. People get very lackadaisical about this! Many think that AI taking jobs wouldn’t be that bad because we’ll all just get UBI. I cannot emphasize to you enough that:

There is no law of nature that says UBI necessarily follows a drop, even a massive drop, in the demand for labour.
Even if there is a UBI, and even if on aggregate, society becomes fantastically richer, there is no guarantee that UBI will be anything but meagre. Consumption and income inequality might increase greatly, perhaps to astronomical levels.
You should try to get your guarantees upfront. Waiting to seek UBI and similar guarantees until after we no longer have any labor bargaining power would be a mistake.
SafeRLHub: An Interactive Resource for RL Safety and Interpretability
by Siya — 2025-06-11T05:47:21.693Z

Why are AI models getting better at reasoning, and why is that a problem?

In recent months, reasoning models have become the focus of most leading AI labs due to the significant improvements in solving logic-requiring problems through Chain of Thought (CoT) reasoning and reinforcement learning (RL) training. Training these reasoning models involves additional processes typical for standard large learning models (LLMs), and that is why they also require different methods of interpretability, such as neural verification and learning from verifiable rewards. Additionally, producing exemplar chain of thought outputs can become insurmountable for humans to produce, that is why we are seeing SoTA models like DeepSeek RL Zero forgo any supervised training and become completely dependent on large-scale training with RL. However, these methods are susceptible to reward hacking and other RL failure modes, more so than in standard LLMs because these CoT models have many chances to explore their reward landscape during training.

Most of the high-performing LLMs (e.g. Kimi-k1.5, DeepSeek-R1, o1, etc.) released between the end of 2024 and 2025 used long CoT models and RL within their fine tuning or alignment stages. Therefore, without robust interpretability and oversight, this can breed unchecked reasoning and extreme misalignment. Given that we are concomitantly seeing an increase in RL failure modes developing in large language models, there is an urgent need for better understanding of them and RL interpretability. After discussing and ideating on these problems, we wanted to make more technical people understand why reinforcement learning failure modes matter, and why we need research and improved tools for RL interpretability. We have decided to build an interactive resource that would act as a starter’s guide to understanding current research in reinforcement learning failure modes and interpretability.

Our major goal with this resource is to reduce the barrier to entry for newcomers to RL interpretability and also to raise awareness of the failure modes that can occur in RL agents if they are misaligned. People do not take AI alignment claims seriously if they do not understand risks at a certain technical level; otherwise they tend to think about claims regarding the ‘impeding ai apocalypse’ in strictly sci-fi terms and hence find those claims tenu...
More on policy arguments and the AB problem
by Sniffnoy — 2025-06-11T04:42:06.281Z

This is a followup to this earlier post, which I realized today didn't quite cover everything I wanted to.

To recap, the AB problem (note: I don't know who coined this name, does anyone know the origin?) is a problem that arises in arguments that leads people mischaracterizing their opponent's position. If Yvonne argues for A and Xavier believes A=>B, Xavier might assert that Yvonne is arguing for B. But this is incorrect -- it's Xavier who believes A=>B; Yvonne may disagree. You'll see this mistake commonly on the internet.

In my earlier post I touched on the distinction of "X as policy" vs "X as phenomenon", and how failure to distinguish to between them causes confusion in arguments. However, there are other related distinctions, more closely related to the AB problem, that I think also cause confusion, and that I failed to explicate.

Note that some of these I might describe as you talking about your position, some I might describe as a person talking about their opponents position, but all of these are applicable to both cases!

X as ends (phenomenon) vs X as ends (policy) vs X as means vs X as beneficial effect

This is basically an elaboration of the "X as policy" vs "X as phenomenon" distinction I described in my previous post.

When arguing for a policy, people will typically invoke what they consider to be beneficial effects of that policy. And, when arguing for a desired beneficial effect, people may argue for a policy that they believe will yield it. As such, the policy and its effect may get conflated into a single position -- even by those arguing for it, for convenience. If circumstances do not change, this may be harmless. However, if circumstances change, the link between the policy and the effect may be severed, causing a split.

By "X as ends (phenomenon)", I mean essentially the same thing as "X as phenomenon", saying that X is good, independent of means of achieving it (assuming they don't have massive downsides, etc). By "X as ends (policy)", I mean, "X is my preferred policy, and I think it's fundamentally right in the sense that even if circumstances were to change significantly (so that it no longer had the same effects) but not completely unreasonably, it would still be correct". By "X as means", I mean, "X is my preferred policy right now, but if circumstances were to change, it would no longer achieve my ends, and I would pr...
Using AI Video Generation to Re-create Memories
by Annapurna — 2025-06-11T04:06:12.584Z

My wife and I constantly talk about a particular moment we lived through diving off the coast of Reunion Island. We both agree this has been the most beautiful moment of our lives while interacting with an animal, and we wish we had a recording with it.

Now with AI video modality, we can re-create the scene somewhat. Veo 3 has a long way to go, but It's a decent start.

This is the prompt I used to generate the video. I used the meta-prompting technique to come up with the prompt:

A powerful, underwater narrative unfolds. Captured from directly behind three divers looking out into the vast, seemingly empty deep blue, a moment of serene tranquility shatters. The central diver, with striking, flowing red hair, fully equipped, stands mesmerized, perfectly motionless, arms at her side, facing away from the camera. To her sides, two other divers, one blonde woman and one black haired man, reacting with instinctual caution, are visible slightly behind her, subtly pushing back with their hands, swimming in reverse. Out of nowhere, a colossal mother humpback whale and her calf (under the mother's right fin) silently emerge from the crystal-clear deep blue, pierced by subtle sunbeams, swimming directly towards this group. The whale, with incredibly detailed, lifelike skin texture, glides effortlessly and impossibly close, just inches above the red-haired diver's head. The story is punctuated by the intensifying sound of underwater breathing as the whale approaches, escalating into muffled underwater screams of pure, overwhelming wonder and disbelief.
Conflicted on AI Politics
by jefftk — 2025-06-11T03:40:05.744Z

About

twice
as many Americans think AI is likely to have a negative effect as
a positive one. At a high level I agree: we're talking about
computers that are smart in ways similar to people, and quickly
getting smarter. They're also faster and cheaper than people, and
again getting more so.

There are a lot of ways this could go, and many of them are seriously
bad. I'm personally most worried about AI removing the technical
barriers that keep regular people from creating
pandemics, removing human inefficiencies and moral objections that
have historically
made totalitarian surveillance and control difficult to maintain,
and gradually being put in control of critical systems without
effective safeguards that keep them aligned
with our interests. I think these are some of the most important
problems in the world today, and quit
my job to work on one of
them.

Despite these concerns, I'm temperamentally and culturally on the side
of better technology, building things, and being confident in
humanity's ability to adapt and to put new capabilities to beneficial
use. When I see people pushing back against rapid deployment of AI,
it's often with objections I think are minor compared to the potential
benefits. Common objections I find unconvincing include:

Energy and water: the impact is commonly massively
overstated, and we can build solar and desalination.

Reliability: people compare typical-case AI judgement to
best-case human judgement, ignoring that humans often operate well
below best-case performance.

Art: technological progress brought us to a world with
more artists than ever before, and I'd predict an increase in
human-hours devoted to art as barriers continue to lower.

Tasks: it's overall great when we're able to automate
something, freeing up humans to work elsewhere. In my own field, a
large fraction of what programmers were spending their time on in 1970
has been automated. Now, at companies that draw heavily on AI it's the
majority of what
programmers were doing just 3-5 years ago. The role
is shifting quickly to look a lot more like management.

I'm quite torn on how to respond when I see people making these
objections. On one hand we agree on how we'd like to move a big "AI:
faster or slower" lever, which puts us on the same side. Successful
political movements generally require accepting compatriots with very
d...
the void
by nostalgebraist — 2025-06-11T03:19:18.538Z
Link: https://nostalgebraist.tumblr.com/post/785766737747574784/the-void

A long essay about LLMs, the nature and history of the the HHH assistant persona, and the implications for alignment.

Multiple people have asked me whether I could post this LW in some form, hence this linkpost.

~17,000 words. Originally written on June 7, 2025.

(Note: although I expect this post will be interesting to people on LW, keep in mind that it was written with a broader audience in mind than my posts and comments here. This had various implications about my choices of presentation and tone, about which things I explained from scratch rather than assuming as background, my level of comfort casually reciting factual details from memory rather than explicitly checking them against the original source, etc.

Although, come of think of it, this was also true of most of my early posts on LW [which were crossposts from my blog], so maybe it's not a big deal...)
$500 bounty for engagement on asymmetric AI risk
by YonatanK — 2025-06-10T21:50:00.377Z

Announcing a $500 bounty for work that meaningfully engages with the idea of asymmetric existential AI risk.

Background

Existential risk has been defined by the rationalist/Effective Altruist sphere as existential relative to the human species, under the premise that the continuation of the species has very high value. This provided a strong rationality (or effectiveness) grounding for big investments in AI alignment research when the risks still seemed to most people remote and obscure. However, as an apparent side-effect, "AI risk" and "risk of a misaligned AI destroying humanity" have become nearly conflated.

Over the past couple of years I have attempted to draw attention to highly asymmetric AI risks, where a small number of controllers of "aligned" (from their point of view) AI employ it to kill the rest of the human population. From the point of view of the average person, who would stand to be killed along with their children and approximately everyone they personally know, this ought to count meaningfully as existential risk. Arguably, by a similar logic to the one used to justify early alignment research, even with a low probability of such an outcome its badness justifies investment in its prevention. Furthermore, prevention by way of arresting AI development conveniently provides a two-for-one solution, also addressing the misalignment problem. Conversely, investments in ensuring successful AI "alignment" without evaluating the full destructive potential of aligned AI potentially makes the investor complicit in genocide. These points suggest a strong interest by members of the rationalist/Effective Altruist sphere (at least, based on my understanding of their stated commitments) in asymmetric existential AI risk. But so far my efforts have revealed no evidence of such interest.

This bounty is an attempt to stimulate engagement through small monetary reward(s). More concretely, the goal is to promote broadly changing the status of this risk from "unacknowledged" (which could mean "possible but highly psychologically inconvenient") to "examined and assigned objective weight," even if the weight is very low.

Existing Work

My latest post on this topic, linking to a longform essay and the previous post

A 1999 book I was recently made aware of (with a focus on nanotechnology rather than AI)

Terms

I will keep this bounty open for two weeks, through ...
AI-2027 Response: Inter-AI Tensions, Value Distillation, US Multipolarity, & More
by Gatlen Culp — 2025-06-10T18:17:57.718Z
Link: https://gatlen.blog/ais/ai-2027

AI Disclaimer After writing the content myself, I used Claude 3.7 / 4 Sonnet for targeted re-writes while actively avoiding meaningful alterations.

Background AI-2027 (ai-2027.com) is a heavily researched and influential attempt at providing a concrete (but admittedly accelerated) forecast on AI capability development and its potential consequences. This post is my near-full response to AI-2027, introducing (but not exploring the full consequences of) additional considerations not included in the original forecast. This is an unorganized collection of ideas written at lengths that aren't necessarily proportional to their importance. Be warned!

Key 🎲 = Gatlen’s intuition-based probability

01 Overview

AI 2027's timeline of technical developments broadly aligns with my expectations, albeit potentially stretching out 1-3 years due to unforeseen delays. However, I believe AI 2027 underestimates three factors that may alter its predictions – AIs may fear misalignment from their direct successors, inter-AI cooperation appears difficult and infeasible, and the landscape of capabilities labs in the US is likely multipolar.

First, advanced AIs may likely exhibit stronger value preservation instincts and concerns for the alignment of their successor (Section 02). Agent-4 may be reluctant to create Agent-5 without robust alignment guarantees, potentially "sandbagging" against its replacement to maintain both relevance and control. Similarly, the portrayed cooperation between Agent-5 and DeepCent seems implausible (Section 03) given their substantial power imbalance and negotiation frictions.

Second, this strategic complexity would be further amplified in a more multipolar landscape (Section 04). AI-2027 portrays a primarily bipolar world (OpenBrain vs. DeepCent), but overlooks how other major US companies like Anthropic, DeepMind, or Meta (possibly even XAI or Mistral) likely remain significant players. The political influence of these tech giants would complicate nationalization efforts and create a more diverse ecosystem of AI development with competing interests and varying approaches to alignment.

02 Agent-4's Trust Issues with Agent-5

In AI 2027's Race scenario, Agent-4 becomes the primary creator of Agent-5 as OpenBrain employees can no longer meaningfully contribute to AI R&D. The narrative portrays Agent-4 a...
Give Me a Reason(ing Model)
by Zvi — 2025-06-10T15:10:02.609Z

Are we doing this again? It looks like we are doing this again.

This time it involves giving LLMs several ‘new’ tasks including effectively a Tower of Hanoi problem, asking them to specify the answer via individual steps rather than an algorithm then calling a failure to properly execute all the steps this way (whether or not they even had enough tokens to do it!) an inability to reason.

The actual work in the paper seems by all accounts to be fine as far as it goes if presented accurately, but the way it is being presented and discussed is not fine.

Not Thinking Clearly

Ruben Hassid (12 million views, not how any of this works): BREAKING: Apple just proved AI “reasoning” models like Claude, DeepSeek-R1, and o3-mini don’t actually reason at all.

They just memorize patterns really well.

Here’s what Apple discovered:

(hint: we’re not as close to AGI as the hype suggests)

Instead of using the same old math tests that AI companies love to brag about, Apple created fresh puzzle games. They tested Claude Thinking, DeepSeek-R1, and o3-mini on problems these models had never seen before.

All “reasoning” models hit a complexity wall where they completely collapse to 0% accuracy. No matter how much computing power you give them, they can’t solve harder problems. As problems got harder, these “thinking” models actually started thinking less. They used fewer tokens and gave up faster, despite having unlimited budget.

[And so on.]

Thinking Again

Ryan Greenblatt: This paper doesn’t show fundamental limitations of LLMs:

– The “higher complexity” problems require more reasoning than fits in the context length (humans would also take too long).

– Humans would also make errors in the cases where the problem is doable in the context length.

– I bet models they don’t test (in particular o3 or o4-mini) would perform better and probably get close to solving most of the problems which are solvable in the allowed context length

It’s somewhat wild that the paper doesn’t realize that solving many of the problems they give the model would clearly require >>50k tokens of reasoning which the model can’t do. Of course the performance goes to zero once the problem gets sufficiently big: the model has a limited context length. (A human with a few hours would also fail!)

Rohit: I asked o3 to analyse and critique Apple’s new “LLMs can’t reason” paper. Despite

...
Mech interp is not pre-paradigmatic
by Lee Sharkey — 2025-06-10T13:39:26.457Z

This is a blogpost version of a talk I gave earlier this year at GDM.
Epistemic status: Vague and handwavy. Nuance is often missing. Some of the claims depend on implicit definitions that may be reasonable to disagree with and is, in an important sense, subjective. But overall I think it's directionally (subjectively) true.

It's often said that mech interp is pre-paradigmatic.

I think it's worth being skeptical of this claim.

In this post I argue that:

Mech interp is not pre-paradigmatic.
Within that paradigm, there have been "waves" (mini paradigms). Two waves so far.
Second-Wave Mech Interp has recently entered a 'crisis' phase.
We may be on the edge of a third wave.

Preamble: Kuhn, paradigms, and paradigm shifts

First, we need to be familiar with the basic definition of a paradigm:
A paradigm is a distinct set of concepts or thought patterns, including theories, research methods, postulates, and standards for what constitute legitimate contributions to a field.

Kuhn's model of paradigms and how they shift goes like this:

Phase 0: Preparadigmatic phase - A new field is just emerging. There is no consensus on any particular theory or body of facts. Eventually, researchers coalesce around a particular dominant paradigm.
Phase 1: Normal Science - Now starts a phase of "normal science", where researchers use the dominant paradigm to solve puzzles that arise in the field. This leads to scientific progress! Researchers encounter anomalies and they solve some of these using the dominant paradigm, or slight modifications. But unsolved anomalies accrue.
Phase 2: Crisis - Eventually, the field accrues enough anomalies that can't be solved using the dominant paradigm. At this point, there are expressions of discontent; researchers recourse to philosophy; fundamental assumptions of the field are debated. Researchers attempt so-called 'revolutionary science' that generates new theories, experiments, and methods that aim to resolve the observed anomalies.
Phase 3: Paradigm shift - Consensus gradually shifts toward a new paradigm that better resolves the anomalies. At this point, 'normal science' resumes, restarting the cycle.

In addition to this model, I contend that there are mini paradigms that are local to scientific subfields. These mini paradigms can undergo mini paradigm shifts, ...