trentmkelly/LessWrong-43k huggingface dataset:
How 2025 AI Forecasts Fared So Far
by Adam B — 2025-05-22T09:42:56.693Z
Link: https://theaidigest.org/ai2025-analysis-may

At the end of 2024, there was a lot of discussion about whether AI scaling was hitting a wall and whether we would reach AGI soon.

We created a forecasting survey that would track key markers of AI progress in what might turn out to be a pivotal year. After filtering the data, we had 421 unique respondents. All questions were optional.

The survey was open from Nov 30th 2024 to Jan 20th 2025 – during which OpenAI o3 was announced, so we can compare forecasts before and after the announcement.

In this post, we summarise respondents' forecasts, and look at how they're holding up so far. At the end of the year, we'll resolve all the forecasts and write up the results (you can join the mailing list here to get notified about that).

Note: I'd recommend reading this post on AI Digest, where you'll find expandable sections for background info, and how responses varied based on forecaster timelines and P(doom).

AI Research: RE-Bench

1) What will be the best normalized score achieved on the original 7 RE-Bench tasks by December 31st 2025?

RE-Bench consists of 7 challenging, open-ended Machine Learning optimization problems created by METR to highlight "automation of AI research and development (R&D) by AI agents".

How much AI will speed up AI R&D is a typical point of disagreement on whether AI progress will speed up or slow down. Research Engineer benchmark (RE-bench) is one of our best current indicators.

Forecasters were bullish – they expected most of the gap between 2024 AI performance and the best observed human performance to be closed in 2025.

Since December, the median human expert score has already been beaten, as METR reports:

When given a total time budget of 32 hours, o4-mini exceeds the 50th percentile of human performance averaged across the five tasks.

The maximum possible score is around 2.0.

Note that the listed state of the art (SOTA) performance from o4-mini is approximate (it's on 5 of the 7 questions, and it's read off this graph visually).

Pre/Post o3 Comparison

Part-way through the forecasting survey, OpenAI o3 was announced with strong reported performance. Forecasts submitted after this announcement were generally more bullish.

Software Engineering: SWE-bench Verified

2) What will be the best performance on SWE-bench Verified by December 31st 2025?

The SWE-bench Verified benchmark consists of 500 software engineering problems drawn from real GitH...
Contain and verify: The endgame of US-China AI competition
by sjadler — 2025-05-22T08:13:49.736Z
Link: https://open.substack.com/pub/stevenadler/p/contain-and-verify-the-endgame-of?r=4qacg&utm_medium=ios

Some competitions have a clear win condition: In a race, be the first to cross a finish line.

The US-China AI competition isn’t like this. It’s not enough to be the first to get a powerful AI system.

So, what is necessary for a good outcome from the US-China AI competition?

I thought about this all the time as a researcher on OpenAI’s AGI Readiness team: If the US races to develop powerful AI before China - and even succeeds at doing so safely - what happens next? The endgame is still pretty complicated, even if we’ve “won” the race by getting to AGI

1

first.

I suggest two reframes on the US-China AI race:

US-China AI competition is a containment game, not a race.
The competition only ends when China has verifiably yielded on AGI development.

By “containment,” I mean that a good outcome for the US might require stopping China from ever reaching a certain level of AI capability. It isn’t enough for the US to get there first. For instance, it’s an issue if China builds AI that can disrupt US nuclear command and control even a small percentage of the time. This is true even if the US has a system that can more reliably disrupt theirs. There are some types of AI the US wants for China never to develop - and likewise, that China wants the US never to develop: the interest in containment is mutual.

By “verifiably yielding,” I mean that the US must be confident that China is not continuing to try to build powerful AI. Otherwise, China might eventually surpass US systems or incur other risks, like losing control over their AI system in the rush to catch up. Unfortunately, methods for “verifiable non-development” - confirming that another party isn’t building AGI - are very understudied. We need to invest heavily in developing these methods and creating treaties that can enforce them: Otherwise, even if we “win” the race to certain powerful abilities, we won’t have good ways to confirm that China has given up on pursuing AGI. (These methods can also be useful for slowing or avoiding the race ahead-of-time, if countries can verify that the other is not developing AGI.)

Given how high the stakes are perceived to be, getting China to yield might require the US to take a truly dominant lead. Such a dominant lead is far from assured, even if the US believes it could ultimately outrace China.

Both nations would benefit from lowering the stakes of the competition - like “hardening” the...
Laugencroissant
by Martin Sustrik — 2025-05-22T06:30:05.563Z

EU policy is driven mostly by the member states, so looking at what national leaders say is often more useful than what the Commission, which has every incentive to placate everyone involved, says. And Franco-German combo matters more than the most.

The joint op-ed by Macron & Merz in Le Figaro (in French) is therefore worth checking. The stuff about Ukraine and trade policy will no doubt be commented upon ad nauseam elsewhere, so let’s look at more low-profile matters:

To reduce energy costs and ensure security of supply, France and Germany will implement a realignment of their energy policies, based on climate neutrality, competitiveness, and sovereignty. This includes applying the principle of technological neutrality, ensuring non-discriminatory treatment of all low-carbon energies within the European Union.

Realignment in this context hopefully means that Germany will stop opposing nuclear power on the EU level.

Here’s an analysis from the Anthropocene Institute:

Merz aligned his position with France’s, ending years of fierce and constant opposition that had refused funding to nuclear investments across the EU and treated nuclear power, in some ways, as worse than coal.

The move came in the context of a broader effort to revitalize the Paris-Berlin strategic partnership, where German sniping against nuclear projects had been a constant irritant. The shift has barely gotten noticed in the German press, because it isn’t likely to change policy within the country. Nuclear restarts remain a hot-button issue there, with not only the Greens but the Social Democrats adamantly opposed to restarting the country’s nuclear fleet. […]

But in the EU more broadly, dropping the German government’s constant obstructionism to approving nuclear projects […] could nudge dozens of plants toward viability. Projects in France, Poland, the Czech Republic, and Romania will be much easier to bring to fruition if they don’t have to fight German nay-saying at every turn.

[…]

Just this year, three of the four countries that had been blocking nuclear investments at a Europe-wide level have changed their tune: first Belgium, then Denmark and now Germany have joined the French-led pro-nuclear block. In March, Italy reversed a Chernobyl-era nuclear moratorium and is now actively looking to expand its nuclear fleet. Bizarrely, it’s the places that ought to have gotten the message loudest t

...
Google I/O Day
by Zvi — 2025-05-21T22:00:05.085Z

What did Google announce on I/O day? Quite a lot of things. Many of them were genuinely impressive. Google is secretly killing it on the actual technology front.

Logan Kilpatrick (DeepMind): Google’s progress in AI since last year:

– The worlds strongest models, on pareto frontier

– Gemini app: has over 400M monthly active users

– We now process 480T tokens a month, up 50x YoY

– Over 7M developers have built with the Gemini API (4x)

Much more to come still!

I think? It’s so hard to keep track. There’s really a lot going on right now, not that most people would have any idea. Instead of being able to deal with all these exciting things, I’m scrambling to get to it all at once.

Google AI: We covered a LOT of ground today. Fortunately, our friends at @NotebookLM put all of today’s news and keynotes into a notebook. This way, you can listen to an audio overview, create a summary, or even view a Mind Map of everything from #GoogleIO 2025.

That’s actually a terrible mind map, it’s missing about half of the things.

As in, you follow their CEO’s link to a page that tells you everything that happened, and it’s literally a link bank to 27 other articles. I did not realize one could fail marketing forever this hard, and this badly. I have remarkably little idea, given how much effort I am willing to put into finding out, what their products can do.

The market seems impressed, with Google outperforming, although the timing of it all was a little weird. I continue to be deeply confused about what the market is expecting, or rather not expecting, out of Google.

Ben Thompson has a gated summary post, Reuters has a summary as well.

I share Ben’s feeling that I’m coming away less impressed than I should be, because so many things were lost in the shuffle. There’s too much stuff here. Don’t announce everything at once like this if you want us to pay attention. And he’s right to worry that it’s not clear that Google, despite doing all the things, can develop compelling products.

I do think it can, though. And I think it’s exactly right to currently produce a bunch of prototypical not-yet-compelling products that aren’t compelling because they aren’t good enough yet… and then later make them good enough.

Except that you need people to actually then, you know, realize the products exist.

This post covers what I could figure out on a deadline. As for why I ...
Podcast: How not to waste a billion dollars (on your clinical trial), with Meri Beckwith on Development & Research
by rossry — 2025-05-21T21:27:47.476Z
Link: https://developmentandresearch.bio/episode/meri-beckwith/

I hosted Meri Beckwith, CEO of Lindus Health, on my podcast Development & Research. Meri founded Lindus after being a clinical trial patient—because he was astounded at how badly clinical drug trials were being run by the world's leading companies.

We talk about why clinical trials companies are twenty years behind in adopting new technologies, how a flooded supply closet can cause a billion-dollar clinical trial to fail, and his speculation about the incentives that—so far—have stopped the world from getting better.

Links: { Twitter, YouTube, full transcript, Spotify, EA Forum }

Here's a section I particularly enjoyed:

Meri Beckwith:
Yep. We do active patient recruitment for most of the trials we run—and recruitment is a really important, often overlooked, aspect of trials.

If you talk to any executive in pharma and ask, "What's your biggest pain point for clinical trials?" they'll probably say, "Patient recruitment." And then, if you ask them, "What can I give you to solve this pain point?" they'll probably say, "More clinical trial sites."

Look, your average clinical trial might have 500 patients; you typically see 300 clinical sites. These are 300 physical locations, each of which often has a cupboard of physical documents that need to be created and signed—that might get flooded, per my example earlier—

Ross:
—for two patients per site?

Meri Beckwith:
For their two patients, yeah. It's violently inefficient, but that is how most trials are set up.

Ross:
And that means that the site administrators are doing it for the first time—for all 500 patients?

Meri Beckwith:
Right. That’s key, and one of the most absurd parts of this setup is, not only is it violently inefficient cost-wise, it makes the data manifestly worse. Firstly, it’s just hard to keep track of all that data, but secondly—you have a team of nurses and investigators at each site who interpret the protocol slightly differently, adding massive variance to what should be a clean experiment.

Ross:
It’s wild to think that this can even be considered a 'controlled' trial. Like, I have these treatment patients and these placebo patients, and every one of them was at a different hospital. I feel like the treatment and placebo should have been at the same hospital and gotten the same—

Meri Beckwith:
—yeah, and that is a really absurd state of affairs... (continues)

Meri Beckwith:
For another example, you ca...
Podcast: From molecule to medicine, with Ross Rheingans-Yoo on Complex Systems
by rossry — 2025-05-21T21:08:38.531Z
Link: https://www.complexsystemspodcast.com/episodes/drug-development-ross-rheingans-yoo/

I appear on Patrick McKenzie's Complex Systems podcast, presenting my overall view of what's wrong in the world of drug development.

Here's a section I particularly enjoyed:

Patrick McKenzie: So, my fun anecdote about recruiting for a clinical trial. I happened to graduate from Washington University in St. Louis, and by coincidence, during my stint with the U.S. vaccine location information architecture project, WashU was running a trial called Stop COVID. They were studying the efficacy of fluvoxamine, an antidepressant that might have off-label potential as an acute COVID treatment. The idea was to offer patients a range of effective medications to reduce severe clinical outcomes from their exposure to COVID.

I got an outsider’s look at their efforts.

One of the unique circumstances for the WashU team was that, since fluvoxamine was already FDA-approved, they didn’t need the full regulatory gauntlet but just needed data showing improved outcomes if patients took this readily available drug. They still had to run a study, recruit thousands—or ultimately, a few hundred—patients, and do so within a very short window. You need to recruit each patient between contracting COVID and (days later!) either recovering totally or experiencing a severe clinical outcome.

So, they went with an Internet marketing campaign to reach people who’d recently gotten COVID. Naturally, they encountered some issues because, as it turns out, med school doesn’t exactly cover Internet marketing.

One skilled individual who happened to know HTML on the team wrote up a 200-question screener to assess eligibility for the trial. They sent potential patients to this screener, and, as many who work online could predict, many people dropped off before completing all 200 questions. When I connected with the team, I assumed there must be a solid regulatory or procedure-based constraint for the length. I asked, “Is this an institutional review board requirement? Medical ethics? Something specific to drug development?” The answer? “We have 200 questions because that's what the grad student implemented.”

So, I asked, “How many of these questions do you actually need?” They replied, “Four.” At that point, I offered the help of someone skilled in conversion rate optimization to design a simple landing page with just those four essential questions to screen prospective patients and capture contact info fo...
The stakes of AI moral status
by Joe Carlsmith — 2025-05-21T18:20:57.289Z
Link: https://joecarlsmith.substack.com/p/the-stakes-of-ai-moral-status

Podcast version (read by the author) here, or search for "Joe Carlsmith Audio" on your podcast app.

1. Introduction

Currently, most people treat AIs like tools. We act like AIs don’t matter in themselves. We use them however we please.

For certain sorts of beings, though, we shouldn’t act like this. Call such beings “moral patients.” Humans are the paradigm example. But many of us accept that some non-human animals are probably moral patients as well. You shouldn’t kick a stray dog just for fun.[1]

Can AIs be moral patients? If so, what sorts of AIs? Will some near-term AIs be moral patients? Are some AIs moral patients now?

If so, it matters a lot. We’re on track to build and run huge numbers of AIs. Indeed: if hardware and deployment scale fast in a world transformed by AI, AIs could quickly account for most of our civilization’s cognition.[2] Whatever the stakes of morality are, AI moral patienthood implies a lot of that. And mistakenly treating AIs that aren’t moral patients like they are can have its own serious costs.

In a report last year, a group of experts argued in-depth that

AI moral patienthood is a realistic, near-future possibility

. I agree. I recommend their report, along with various other recent resources.[3]

Still, I wanted to think the issue through for myself. In particular, for all my interest in this topic, I noticed ways my brain wasn’t treating it like a real thing. I wanted to make sure I looked directly.

So I decided to write some essays about it. This is the first. Here, my main aim is to bring the question itself, and the stakes it implies, into clearer and more concrete view.

(There’s also video and transcript here of a talk about AI moral status that I recently gave at Anthropic. It gives an overview of my current overall take on the topic – including, why I think near-term AIs might well be conscious. If you’re wondering why anyone would take that possibility seriously in the first place, the talk might give you a sense.)

2. Pain

“Moral patienthood.” Do we know what we mean? Let’s not assume we do. Let’s try for some more direct contact.

One way in is to think about pain.

You go for a root canal. Your dentist injects the anesthetic. You wonder: did it work?

What’s at stake in that question?

Birch (2024) opens with Kate Bainbridge, who became unresponsive due to inflammation of ...
Which AI Safety techniques will be ineffective against diffusion models?
by Allen Thomas — 2025-05-21T18:13:46.092Z

Google showed off Gemini Diffusion at their event yesterday and while I do not have access to the model just yet, based on what I have seen about so far, it seems impressive.
I have been interested in AI Safety for a while and have been trying to keep up with the literature in this space.

According to OpenAI's March 2025 blog about detecting misbehavior in models -

We believe that CoT monitoring may be one of few tools we will have to oversee superhuman models of the future.

Can we extend CoT monitoring for diffusion-based models as well?
Or do you think the intermediate states will be too incoherent to monitor?
This is my first LW post, and I do not have a lot of hands-on experience with safety techniques, so forgive me if my question is poorly framed.
Through The Looking Glasses: Issues & Solutions for Augmented Reality
by claywren — 2025-05-21T18:11:53.309Z

I. Introduction

Augmented reality (AR) glasses may redefine smartphones’ role in daily computing, much as smartphones once reshaped our relationship with the desktop. In late 2024, Meta unveiled

Project Orion

, their first lightweight see‑through headset meant for all‑day wear. Industry roadmaps from

Apple

,

Google

and

Samsung

point to consumer‑grade models arriving in 2026, with annual refresh cycles reminiscent of the iPhone after 2007.

What is AR? Unlike virtual reality headsets that obscure the outside world, AR glasses keep the user’s physical surroundings visible while inserting context‑aware digital images, text and 3‑D objects directly into the wearer’s view. The interface abandons 2‑D touch in favor of voice, gaze and neural hand‑tracking, freeing hands even as it overlays them with digital objects. These devices are a concrete step towards Zuckerberg’s

metaverse

vision, where the digital world inhabits the physical one through “holograms”.

After seventeen smartphone generations, handset innovation has flattened to incremental camera and chip upgrades. Consumers, primed for novelty and status, will likely see the friction-free interface as worth the upgrade. Manufacturers will stoke the cycle with sleek advertisements, tight ecosystem lock‑in (think iCloud for your vision), and demonstrations of all smartphone capabilities ported to the new interface.

A single in‑store demo may be as persuasive as 2007’s first pinch‑to‑zoom, and picking up a smartphone

more than 100 times

per day may soon feel as outdated as dial‑up. Yet history shows that the personal thrills of a new interface routinely outweigh perceived social costs. Television ownership in the United States leapt from 9% of households in 1950 to

90%

by 1960; theorists like Robert Putnam and Neil Postman would later argue that television homogenized popular culture and collective identity but significantly contributed to the

decline

of civic engagement, physical spaces and

social capital

. Smartphones replayed the cycle:

Sherry Turkle

coined “

phubbing

” in 2011 to diagnose phone‑induced conversational shallowness, yet U.S. smartphone adoption still climbed from 35% in 2011 to

90%

by 2024. Even those wary of the social side-effects felt pressure to stay reachable, share photos, or use productivity-related apps.

The risk is magnified by inequality...
Rooting for Moments, Not Jerseys. Another Approach to Enjoying Sports
by Ahmed Elsayyad — 2025-05-21T18:11:19.993Z

Epistemic Status: Somewhere between a personal life‑hack and a tentative general theory. I’ve been running this experiment for five NBA seasons; dozens of predictions‑market positions, numerous group‑chat quarrels. I am reasonably confident there’s something here, but I do not yet know how broadly it generalizes beyond U.S. team sports fandom.

TLDR

Traditional fandom couples your happiness to the fate of a single franchise whose base‑rate chance of winning a championship hovers around ~3 %. If you instead root for stories (individual players, milestone moments, tactical novelties) and back those beliefs with small prediction‑market stakes, you harvest nearly all the social energy of watching sports while slicing your disappointment rate by an order of magnitude.

The Standard Model of Fan Disappointment

Prior: There are 30 NBA teams, 16 make the playoffs, one wins.

Your team’s ex ante title equity = 1⁄30 ~ 3 %.

Observation: Most fans watch hundreds of hours for a <5 % chance of maximal joy.

By expected‑value math, a fan experiences ~19 seasons of varying degrees of pain for every season of euphoria (assuming teams have an equal chance, which isn’t the case)

Result: “Disappointed NBA fans” group‑chats; rage‑tweets; vows to boycott the league until the next season

We have built a cultural institution that systematically manufactures unhappiness in proportion to engagement. That seems… suboptimal.

A Player-Focused Bayesian Fandom

My counter‑hack emerged by accident. I started watching the playoffs with friends but lacked a team I cared about. I grew up in Maryland, so the Wizards were what I was “supposed” to pick but they very rarely won.

So I asked: Which players are on breakout trajectories that I’ll enjoy watching regardless of uniform?

Examples:

2018–19: Giannis Antetokounmpo, aspiring MVP talent but didn't win a championship yet.
2022–23: Shai Gilgeous‑Alexander (SGA), then an "outside shot" for MVP, now a consensus top‑five.
Role‑player favorites: Jrue Holiday’s on‑ball defense, Jimmy Butler’s competitiveness.

I picked 3‑5 such arcs per season and rooted for all of them. Expected championship equity per season jumped from 3 % (one team) to ~15 % (multiple plausible contenders), with the emotional upside of “I called it!” bragging rights when a prediction paid off.

Prediction Markets as Emotional Hedge

Enter Manifold Markets (or your fav...