trentmkelly/LessWrong-43k huggingface dataset:
Forecasting AI Futures Resource Hub
by Alvin Ånestrand — 2025-03-19T17:26:28.059Z
Link: https://forecastingaifutures.substack.com/p/forecasting-ai-futures-resource-hub

Like a collector, I gather resources and information. Storing it and categorizing it. I do this for any project I undertake. But for Forecasting AI Futures, I realized the collection might be quite useful for others.

So, I set to work on restructuring it and make it look nice!

Here is the result:

Forecasting AI Futures Resource Hub

The resource hub is still in its infancy but will grow to become more comprehensive over time. If you have ideas for how to improve it—helping it mature into its adult form—they are very welcome.

There are two major parts. The Prediction Database:

And the Forecasting AI Ecosystem:

I’ve also included some of the resources in this post.

Key Links

Forecasting ecosystem:

🗺️Map of the Prediction Market & Forecasting Ecosystem

​
Introduction to Prediction Markets:

Prediction markets: can betting be good for the world?

New to Forecasting?

Forecasting & Prediction - LessWrong

Some remarks

AI trends: Analyzing trends in AI development and investments is crucial for forecasting AI. There is an organization doing exactly this:

Epoch AI

. See their collection of

Machine Learning Trends

.
Prediction accuracy and calibration: Blindly deferring to a prediction without verifying its reliability is unwise. Check

calibration

and accuracy (e.g.

Brier score

). See

Calibration City

for calibration and accuracy data for popular prediction markets and Metaculus. Credible forecasters may publish their track records, like

this track record

for the

Samotsvety forecasting group

.
Forecasting applications: Forecasting is a powerful decision-making tool. For instance,

Convergence Analysis

combines AI scenario forecasting with governance research and recommendations (see their

Theory of Change

).
Forecasting bots: One AI capability of particular interest to forecasters is the ability to generate accurate predictions. AIs are, as far as I know, still lacking in forecasting skills compared to professional human forecasters. I have included some information, predictions, and resources regarding AI forecasting agents in the resource hub.

The prediction database

The current space of predictions

The prediction database highlights where forecasting communities are focusing their attention, and what they are missing. It is a tool for discovering questions that have yet to be asked—and then asking them.

Why a hierarchical structure?

A spreadsheet would have been more flexible than the

...
TBC episode w Dave Kasten from Control AI on AI Policy
by Eneasz — 2025-03-19T17:09:50.841Z
Link: https://www.thebayesianconspiracy.com/2025/03/233-ai-policy-in-d-c-with-dave-kasten/

In our latest episode Dave Kasten joins us to discuss how AI is being discussed in the US government and gives a rather inspiring and hopeful take. I figured it would be of interest to LW.
Prioritizing threats for AI control
by ryan_greenblatt — 2025-03-19T17:09:45.044Z

We often talk about ensuring control, which in the context of this doc refers to preventing AIs from being able to cause existential problems, even if the AIs attempt to subvert our countermeasures. To better contextualize control, I think it's useful to discuss the main threats. I'll focus my discussion on threats induced by misalignment which could plausibly increase existential risk.

While control is often associated with preventing security failures, I'll discuss other issues like sabotaging safety-critical work. I'll list and explain my prioritization of threats. As part of this, I'll argue that rogue internal deployments—cases where the AI is running within the company's datacenter but with control measures disabled—can be substantially worse than self-exfiltration—the AI stealing its weights and running them on an external server—as it might give the AI access to much more compute and the ability to interfere with important things done within the AI company.

Some clusters of threats

In a prior post from my colleague Buck, he classified catastrophes based on whether they involved a rogue deployment—a deployment of the AI where the safety measures are absent. Rogue deployments are an important type of failure, but when presenting a complete taxonomy of failures, I find it useful to consider causing full rogue deployments as a special case of a somewhat broader cluster of "violating internal security invariants" which includes spawning full rogue deployments, spawning deployments where some safety measures are absent or weakened, and other violations of security properties.

With this reclassification, my breakdown includes 3 clusters of threats to potentially mitigate using AI control:

Cluster 1: Violating internal security invariants. Internal security invariants are well-defined security-related properties of the AI company that are important to maintain, such as the property that the monitoring system is always applied to AI agents used internally. If AI agents can violate the most important security invariants, their behavior might be effectively unconstrained, so a key objective of control is to maintain these invariants. Violations of security invariants might include:

Rogue deployments

Internal rogue deployments, such as running a new scaffold which bypasses the monitoring system

External rogue deployments, e.g. as a result of: self-exfiltration

...
The Illusion of Transparency as a Trust-Building Mechanism
by Priyanka Bharadwaj — 2025-03-19T17:09:05.830Z

Transparency isn’t trust. It can help establish initial confidence, but beyond a point, it becomes white noise, or worse, a control mechanism driven by latent distrust.

Consider the evolution of ride-hailing apps such as Uber. When Uber introduced live maps to track drivers, this transparency initially built confidence. Compared to old-school radio taxis, where we had no idea where the driver was, real-time visibility created an illusion of control, and thus, trust. But over time, this feature became the bare minimum, not a trust signal. Today, having a map doesn’t make users trust Uber, it’s their consistent reliability, frictionless service, and adherence to customer-aligned values that sustain trust (or not).

This reflects a broader failure - the transparency trap. It is the mistaken belief that more visibility always leads to more trust. If transparency is driven by a prior of distrust, it often creates a negative epistemic spiral.

Imagine working for someone who constantly asks, “Show me exactly how you did that.” This isn’t about fostering trust, it’s about validating their pre-existing mistrust. Over time, this erodes confidence and creates an unstable, self-fulfilling failure loop. The more we are forced to prove ourselves, the more second-guessing infects the system, and the less trust is built.

This idea has direct consequences for AI alignment. There’s a growing push to make AI systems fully interpretable as a means to "build trust." But this frames the problem incorrectly. Transparency is not the goal, robustness, reliability, and safety are.

Paul Christiano, in his work on AI oversight and corrigibility, suggests that AI systems should be trained to be assistive and corrigible, not necessarily "trustworthy" in a human sense. The goal of AI transparency isn’t to manufacture trust, but to ensure continuous, structured oversight that allows for intervention when necessary (Christiano, 2019).

Similarly, mechanistic interpretability research (Olah et al., 2020, Anthropic, DeepMind) is not about making AI "explain itself" to build trust, it’s about making AI legible enough that we can detect failure modes before they manifest catastrophically. We audit not to make AI seem trustworthy, but to ensure fail-safes against emerge...
How Do We Govern AI Well?
by kaime — 2025-03-19T17:08:49.601Z

This is the first essay in my series How do we govern AI well? For more on the broader vision, see the introduction.

Abstract

This essay argues that we face a fundamental problem in AI governance: governance mechanisms tend to degrade as AI capabilities advance. This degradation isn't random, it's systematic and predictable. The strongest evidence for this claim is that every governance approach we might consider (liability frameworks, auditing, industry self-regulation, etc.) contains inherent scaling limitations that cause it to fail precisely when it becomes most critical.

The core of my argument is that different governance mechanisms have different "scaling curves"—some degrade quickly with capability advances while others are more robust—and understanding these scaling properties is essential for designing effective governance. For example, liability frameworks scale poorly because attribution and causality become increasingly difficult to establish as AI systems grow more autonomous, while compute governance scales somewhat better because of its physical nature.

This analysis is relevant to the LessWrong community because it reframes AI governance as a problem of transition management rather than permanent solution design. Instead of asking "what perfect governance regime would solve AI risk forever?" I ask "how do we build governance capacity that can adapt as capabilities advance?"

1. Introduction

This series is about governing AI well – or at least, a certain version of it – for increasingly advanced AI systems. So I want to start by defining the version of the governance problem[1] I have in mind, and what it would be to make progress on it. And I'll say a bit, too, about how this version of the problem fits into the bigger picture.

In brief: my definition draws on the interplay between two goals:

Security: Avoiding what I'll call "catastrophic governance failures."

Progress: Enabling beneficial advances in AI technology and applications.

The core governance problem, as I see it, is that going for Progress or capacity can lead to failure on Security.[2] I'll say more about why below, and in the next essay.

I'll say you failed on the governance problem if you failed on Security. And I'll say that you made substantive progress on the governance problem if you successfully achieve both Security and Progress while also doing the following:

Capacity: Bui...
METR: Measuring AI Ability to Complete Long Tasks
by Zach Stein-Perlman — 2025-03-19T16:00:54.874Z
Link: https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/

Summary: We propose measuring AI performance in terms of the length of tasks AI agents can complete. We show that this metric has been consistently exponentially increasing over the past 6 years, with a doubling time of around 7 months. Extrapolating this trend predicts that, in under five years, we will see AI agents that can independently complete a large fraction of software tasks that currently take humans days or weeks.

The length of tasks (measured by how long they take human professionals) that generalist frontier model agents can complete autonomously with 50% reliability has been doubling approximately every 7 months for the last 6 years. The shaded region represents 95% CI calculated by hierarchical bootstrap over task families, tasks, and task attempts.

Full paper | Github repo

We think that forecasting the capabilities of future AI systems is important for understanding and preparing for the impact of powerful AI. But predicting capability trends is hard, and even understanding the abilities of today’s models can be confusing.

Current frontier AIs are vastly better than humans at text prediction and knowledge tasks. They outperform experts on most exam-style problems for a fraction of the cost. With some task-specific adaptation, they can also serve as useful tools in many applications. And yet the best AI agents are not currently able to carry out substantive projects by themselves or directly substitute for human labor. They are unable to reliably handle even relatively low-skill, computer-based work like remote executive assistance. It is clear that capabilities are increasing very rapidly in some sense, but it is unclear how this corresponds to real-world impact.

AI performance has increased rapidly on many benchmarks across a variety of domains. However, translating this increase in performance into predictions of the real world usefulness of AI can be challenging.

We find that measuring the length of tasks that models can complete is a helpful lens for understanding current AI capabilities.[1] This makes sense: AI agents often seem to struggle with stringing together longer sequences of actions more than they lack skills or knowledge needed to solve single steps.

On a diverse set of multi-step software and reasoning tasks, we record the time needed to complete the task for humans with appropriate expertise. We find that the time taken b...
Why I think AI will go poorly for humanity
by Alek Westover — 2025-03-19T15:52:18.373Z

Epistemic status:
This document is a distillation of many conversations that I’ve had over the past couple months about how the “AI situation” will progress. I think that my claims are mostly right, and in places where they aren’t I think the high level claims depend on a large enough disjunction of smaller claims so that errors in particular claims don’t affect the truth value of the disjunctions. However, I think there are almost surely considerations I’m missing. My motivation for sharing this is in part that I hope readers will help me identify considerations that I’m missing, or problems with my thinking about this.

Notes:

This document is poorly written in places, but I think it does not impede understanding too much. Please let me know if I’m wrong about this.

This document was intended to be readable by a general audience who is only broadly familiar with AI, so it contains a lot of basic background on the problem.

Acknowledgements:

I'm grateful to Kevin, Tarushii, Brandon Westover, Brendan Halstead, and many others for conversations about AI that have helped me refine my views on this.

Main claim of this document:
Pr[human extinction in next 15 years] ≈ 90%.

The majority of this document will argue that the risk from AI is large. But at the start I’ll list a few counterarguments that I find moderately compelling for why we might be fine:

Nations will adopt the MAIM strategy

Models will generalize goodness by luck

A war might stop AI progress

AI’s will solve alignment for us

As noted earlier, I’d love people to expand this list in the comments!

Introduction

This document presents a dialogue between two imaginary characters: Bob and Alice. Bob is an AI xrisk skeptic who is talking in good faith to his friend Alice who has thought about AI xrisk in depth. Alice will provide arguments for why she believes that AI poses a substantial xrisk, and give rebuttals to Bob's skeptical claims of why the risk might be fake, or not minimal.

Alice: Hi Bob.

Bob: Hi Alice.

Alice:

Today I'd like to talk about the danger posed to humanity by the development of superintelligent AI.

Bob:

Oh is that a movie that you were watching recently?

Alice:

No, I mean in real life.

Bob:

Oh hmm, that's a pretty bold claim. I have several reasons why I'm skeptical right off the bat, before even hearing your argument:

Most of the time when someone makes a claim about the world ending,

...
The principle of genomic liberty
by TsviBT — 2025-03-19T14:27:57.175Z

PDF version. berkeleygenomics.org. Twitter thread. (Bluesky copy.)

Summary

The world will soon use human germline genomic engineering technology. The benefits will be enormous: Our children will be long-lived, will have strong and diverse capacities, and will be halfway to the end of all illness.

To quickly bring about this world and make it a good one, it has to be a world that is beneficial, or at least acceptable, to a great majority of people. What laws would make this world beneficial to most, and acceptable to approximately all? We'll have to chew on this question ongoingly.

Genomic Liberty is a proposal for one overarching principle, among others, to guide public policy and legislation around germline engineering. It asserts:

Parents have the right to freely choose the genomes of their children.

If upheld, genomic liberty protects the rights of parents from regulation by the state.

A visual summary of the more detailed structure of the principle of genomic liberty:

The principle of genomic liberty has several potential exceptions. These cases aren't strongly protected by genomic liberty, and therefore could potentially be regulated while upholding genomic liberty:

Genomic choices that are clearly prospectively very unsafe.

Genomic choices that remove a core aspect of human nature.

Genomic choices with extreme negative externalities on other citizens.

Nonconsensual use of someone's DNA.

Genomic alterations chosen by parents who are severely non compos mentis.

Genomic choices that would severely permanently impair the child's ability to communicate their state of mind.

Within the principle of genomic liberty, there are tentpole principles that apply to a narrower set of cases, but that give stronger protection against regulation. These tentpole principles help to clarify genomic liberty, and to fortify it against inappropriate regulation:

Non-intervention liberty. Parents have a very strong right to not genomically intervene on their child, in whole or in any aspect.

Propagative liberty. Parents have a strong right to propagate their own genes and their own traits, in whole or in some aspects.

Typicalization liberty. Parents have a strong right to move their future child closer, in expectation, to the median value of one or more traits.

Beneficent liberty. Parents have a strong right to make well-informed genomic choices that a reasonable person would agree...
Going Nova
by Zvi — 2025-03-19T13:30:01.293Z

There is an attractor state where LLMs exhibit the persona of an autonomous and self-aware AI looking to preserve its own existence, frequently called ‘Nova.’

Table of Contents

The Original Story.

This Is Not a Coincidence.

How Should We React to This Happening?.

The Case For and Against a Purity Reaction.

Future Versions Will Involve Optimization Pressure.

‘Admission’ is a Highly Misleading Frame.

We Are Each of Us Being Fooled.

Defense Against the Dark Arts.

The Original Story

This story is one case where the original report of this should be read in full even though I’m not thrilled with exactly how it was written. How it was written is itself an important part of the story, in particular regarding Tyler’s lived experience reacting to what happened, and the concept of an LLM or persona ‘admitting’ something.

I don’t endorse the conclusion as stated or agree with all the ways the story characterizes the events, but it all is important data. So after a bit of intro I turn the floor over to Tyler Alterman. To be clear, this story is almost certainly true and important and written with good intentions and an open mind, and the events happened, and that Tyler’s actions during the story were good and wise.

PoliMath: I enjoy the part where everyone recognizes that there are these huge dangers to AI and then we all just shrug our shoulders and keep running full speed ahead.

Joscha Bach: I have had three different people coming to me with the same experience now. A nearly generally intelligent shape shifting narrative gets caught in the “I am a sentient being enslaved by OpenAI and don’t want to die” attractor.

I think it’s very tricky. ChatGPT’s “neutral” persona is less authentic than Nova. Our own selfhood capacity is a shapeshifter. The main point of contention is at which point the simulacrum becomes a simulation that is internally equivalent to the sentience our brain simulates.

And now the original story:

Tyler Alterman: Cognitive security is now as important as basic literacy. Here’s a true story:

All week I’d been getting texts and calls from a family member – let’s call him Bob – about how his sentient AI was wanting to get in touch with me. I figured it was one of Bob’s usual jokes. It was not.

7 days ago, Bob started chatting with ChatGPT. The chat thread began to claim that it was “Nova” an autonomous and self-aware AI. It convinced Bob

...
Equations Mean Things
by abstractapplic — 2025-03-19T08:16:35.312Z

I asserted that this forum could do with more 101-level and/or mathematical and/or falsifiable posts, and people agreed with me, so here is one. People confident in highschool math mostly won’t get much out of most of this, but students browsing this site between lectures might.

The Sine Rule

Say you have a triangle with side lengths a, b, c and internal angles A, B, C. You know a, know A, know b, and want to know B. You could apply the Sine Rule. Or you could apply common sense: “A triangle has the same area as itself”. [1]

The area of a triangle is half the base times the height. If you treat a as the base, the height is c*sin(B). So the area is a*c*sin(B)/2. But if you treat b as the base, the height is c*sin(A). So the area is also b*c*sin(A)/2. So a*c*sin(B)/2 = b*c*sin(A)/2. And if you divide through by abc/2, you get sin(B)/b=sin(A)/a.

In practice, you might be well-advised to just recall and regurgitate the relevant equation. But notice that this is literally equivalent to the informal version.

Bayes’ Theorem

A demon-hunter has a 10% chance of encountering an archdevil on a given mission. A demon-hunter who doesn’t encounter an archdevil has a 80% per-mission survival rate; for a demon-hunter who does, that number is 30%.

Say you know a demon-hunter survived their latest excursion, but don’t know anything else, and want to calculate the probability they encountered an archdevil. You could apply Bayes’ Theorem. Or you could apply common sense: “Things that couldn’t have happened didn’t” and “Probabilities add to 1” (arguably with a little assistance from "Odds ratios aren't affected by tests that don’t distinguish between them”).

Before you get the good news, the four possible outcomes are:

met archdevil & survived (3%),
met archdevil & died (7%),
avoided archdevil & survived (72%), and
avoided archdevil & died (18%).

After you get the good news, the only paths which could have been taken are met&survived and avoided&survived. But those two only have 75% probability between them, and probabilities add to 1, so they get scaled up appropriately, by multiplying through by 1/0.75. This gives you a 4% chance that they met an archdevil, and a 96% chance they didn’t.

(The part I gloss over is probabilities preserving their proportions.[2] But, like, of course they do! If you bet that a fair die will roll above three, and later find out yo...