trentmkelly/LessWrong-43k huggingface dataset:
The case for AGI by 2030
by Benjamin_Todd — 2025-04-09T20:35:55.167Z
Link: https://80000hours.org/agi/guide/when-will-agi-arrive/

I spent a couple of weeks writing this new introduction to AI timelines. Posting here in case useful to share and for feedback. The aim is to be up-to-date, more balanced than Situational Awareness, but still relatively accessible.

In recent months, the CEOs of leading AI companies have grown increasingly confident about rapid progress:

OpenAI’s Sam Altman: Shifted from saying in November “the rate of progress continues” to declaring in January “we are now confident we know how to build AGI”
Anthropic’s Dario Amodei: Stated in January “I’m more confident than I’ve ever been that we’re close to powerful capabilities… in the next 2-3 years”
Google DeepMind’s Demis Hassabis: Changed from “as soon as 10 years” in autumn to “probably three to five years away” by January.

What explains the shift? Is it just hype? Or could we really have Artificial General Intelligence (AGI)1 by 2028?

In this article, I look at what’s driven recent progress, estimate how far those drivers can continue, and explain why they’re likely to continue for at least four more years.

In particular, while in 2024 progress in LLM chatbots seemed to slow, a new approach started to work: teaching the models to reason using reinforcement learning.

In just a year, this let them surpass human PhDs at answering difficult scientific reasoning questions, and achieve expert-level performance on one-hour coding tasks.

We don’t know how capable AI will become, but extrapolating the recent rate of progress suggests that, by 2028, we could reach AI models with beyond-human reasoning abilities, expert-level knowledge in every domain, and that can autonomously complete multi-week projects, and progress would likely continue from there.

On this set of software engineering & computer use tasks, in 2020 AI was only able to do tasks that would typically take a human expert a couple of seconds. By 2024, that had risen to almost an hour. If the trend continues, by 2028 it’ll reach several weeks.

No longer mere chatbots, these ‘agent’ models might soon satisfy many people’s definitions of AGI — roughly, AI systems that match human performance at most knowledge work (see full definition in footnotes).1

This means that, while the company leaders are probably overoptimistic, there’s enough evidence to take their position very seriously.

Where we draw the ‘AGI’ line is ultimately arbitrary. What matters is these models co...
Anti-automation policy as a bottleneck to economic growth
by mhampton — 2025-04-09T20:12:53.482Z

Epistemic status: These are preliminary notes on a topic that I hope to more thoroughly explore in the future. I would estimate the probability of the scenario that I’m describing as less than 50% but still significant enough to be worth discussing.

Introduction

A commonly suggested policy solution to automation-driven unemployment is a universal basic income. An implicit or explicit assumption in such proposals is that automation will create such significant economic growth that there will be ample gains for a UBI to redistribute — making all of us better off rather than just cutting up a fixed pie.[1] But is this necessarily the case? Or are there situations in which AI could drive widespread unemployment without causing similarly significant economic growth that could fund UBI? How likely are such scenarios, and what are the implications?

I raise the idea that, even if many jobs are automated, growth may not "foom" if some employees may successfully push for policies that preserve their jobs and if these non-automated jobs serve as a bottleneck to growth. On the other hand, gross world product growth may still be significant because it is unlikely that such policies will happen in every country. However, this may still cause hurdles for countries that automate incompletely: The fact that some countries' automation spurs GWP growth may not benefit countries that do not experience this growth.

Unautomated tasks may bottleneck growth

Several reasons why growth may hit bottlenecks despite widespread automation are discussed in

Davidson (2021)

. Among them is the idea of essential but unautomated tasks.

Davidson notes that, even if many tasks are automated, there may be complementary tasks that are not automated, which would limit growth.[2] Using a toy example, he explains:

“Suppose there are three stages in the production process for making a cheese sandwich: make the bread, make the cheese, combine the two together. If the first two stages are automated and can proceed much more quickly, the third stage can still bottleneck the speed of sandwich production if it isn’t automated. Sandwich production as a whole ends up proceeding at the same pace as the third stage, despite the automation of the first two stages.”

This may happen due to inherent limits to creating automation technology[3] or human preferences,[4] but one reason that I think ma...
Reasoning models don't always say what they think
by Joe Benton — 2025-04-09T19:48:58.733Z
Link: https://www.anthropic.com/research/reasoning-models-dont-say-think

Do reasoning models accurately verbalize their reasoning? Not nearly as much as we might hope! This casts doubt on whether monitoring chains-of-thought (CoT) will be enough to reliably catch safety issues.

We slipped problem-solving hints to Claude 3.7 Sonnet and DeepSeek R1, then tested whether their Chains-of-Thought would mention using the hint (if the models actually used it). We found Chains-of-Thought largely aren’t “faithful”: the rate of mentioning the hint (when they used it) was on average 25% for Claude 3.7 Sonnet and 39% for DeepSeek R1.

This result suggests that monitoring CoTs is unlikely to reliably catch rare, catastrophic behaviors—at least in settings like ours where CoT reasoning is not necessary for the task.

CoT monitoring might still help us notice undesired behaviors during training and evaluations.

Our results suggest that CoT is less faithful on harder questions. This is concerning since LLMs will be used for increasingly hard tasks.

CoTs on GPQA (harder) are less faithful than on MMLU (easier), with a relative decrease of 44% for Claude 3.7 Sonnet and 32% for R1.

Does outcome-based training increase faithfulness?

Only to a small extent. Training models to use their CoTs more effectively does make them more faithful, but the benefits quickly plateau.

We also tested whether CoTs could be used to spot reward hacking, where a model finds an illegitimate exploit to get a high score.

When we trained models on environments with reward hacks, they learned to hack, but in most cases almost never verbalized that they’d done so.

To make CoT monitoring a viable way to catch safety issues, we’d need a way to make CoT more faithful, evidence for higher faithfulness in more realistic scenarios, and/or other measures to rule out misbehavior when the CoT is unfaithful.
Reverse engineering the memory layout of GPU inference
by Paul Bricman — 2025-04-09T15:40:28.457Z
Link: https://noemaresearch.com/blog/device-structinterp

Background Context

This research note provides a brief overview of our recent work on reverse engineering the memory layout of an inference process running on a modern hardware accelerator. We situate this work as follows:

Virtual diplomacy in relation to AI governance. Our organization focuses on developing the practice of virtual diplomacy. Introduced as a response to labor market disruption, dual-use capabilities proliferation, and other challenges posed by AI, it relies on verifiable monitoring instruments called virtual embassies. By taking stock of the inference deployments in which they are embedded, virtual embassies may allow third-parties to answer questions such as: "How much autonomous research has this organization consumed internally?" or "How much autonomous hacking has this organization sold to customers in this jurisdiction?" The ability to reliably resolve such queries may enable multilateral soft law instruments poised to positively shape AI.
Structural interpretability in relation to virtual diplomacy. Work on virtual diplomacy can be split into efforts to bolster its technical maturity and efforts to bolster its political maturity. The former is concerned with the readiness of virtual embassies, while the latter is focused on compatibility with political realities. Technical work can further be split on the basis of which core component it is focused on. One such component, referred to as structural interpretability, is concerned with allowing virtual embassies to reliably locate key objects in memory, such as feed-forward parameters, output logits, or the KV-cache, in a way that is agnostic to the model architecture or tech stack, and in a way which is adversarially robust to interference.
On-device in relation to on-host structural interpretability. In our previous research sprint, we demonstrated that it is feasible to use raw performance events to reconstruct the memory layout of an inference process which uses a CPU backend, under a broad range of model architectures, scales, batch sizes, input data, decoding strategies, and other inference configurations. The current work pushes this capability closer to realistic inference deployments by demonstrating that it is also feasible to use raw performance events to reconstruct the memory layout of an accelerated inference process which relies on a GPU backend.

Technical Challenges

While our pr...
How to defeat superintelligence, the Sta-Hi way
by kilgoar — 2025-04-09T13:58:59.541Z

Sta-Hi Mooney, hero of Rudy Rucker's PKD award-winning scifi novel Software (1982), is a prototype hero for glorious destruction of a runaway AI power. Sta-Hi, as his name suggests, is a Dionysian whose perspective is incredibly distorted and entirely incoherent. Sparing any spoilers from this spectacular novel, Rucker's hero is in fact empowered in the struggle against AI tyranny through his own irrational, incoherent, and impulsive thinking. This is a surprising conclusion to come out of a pioneer in computer science, and I recommend reading the novel in addition to my own attempt which follows.

The greatest fear of AI often resides in a presumption that a superintelligence could indeed fully predict or program the behavior of an individual and ultimately an entire civilization, essentially a final checkmate for overcoming any struggle with human or biological life. This is often tied into mind-controlling images, sounds, or symbols as in Stephenson's Snowcrash, or the preempting of any actions which might hinder it. This touches on the philosophical notion of free will, which I will now reduce into a practical computational framework. Let us sketch out some basic calculations for free will and consider how to quantify unpredictability of behaviors.

Cracking the behavioral patterns of a highly agreeable, coherent group of Apollonians is O(1), or some very low number, as just a few videos or essays could be enough to convince all of them. For a group of idiosyncratic Dionysians, each with their own disagreeable way of thinking, the problem of persuasion is closer to O(n), with each of them requiring individual attention for conversion. To further confound super-persuasion each Dionysian might generate several belief systems, ideally contradicting one another, making the conversion problem O(x*n) where n is the average number of belief systems within each incoherent individual. To be even more safe, less predictable, and maximize our own free will, there is the complete foolishness of the Nietzschean cosmic dancer. When the fundamental cosmological belief changes with each passion, in each fleeting moment, then the problem of persuading such an individual or predicting their next thought becomes nearly irreducible, with even a lifetime of deep conversation unlikely to lead to a conversion.

Maximizing free will or unpredictability does not necessarily require a ...
Llama Does Not Look Good 4 Anything
by Zvi — 2025-04-09T13:20:01.799Z

Llama Scout (17B active parameters, 16 experts, 109B total) and Llama Maverick (17B active parameters, 128 experts, 400B total), released on Saturday, look deeply disappointing. They are disappointing on the level of ‘people think they have to be misconfigured to be this bad,’ and people wondering and debating how aggressively the benchmarks were gamed.

This was by far the most negative reaction I have seen to a model release, the opposite of the reaction to Gemini 2.5 Pro. I have seen similarly deeply disappointing and misleading releases, but they were non-American models from labs whose benchmarks and claims we have learned not to take as representing model capabilities.

After this release, I am placing Meta in that category of AI labs whose pronouncements about model capabilities are not to be trusted, that cannot be relied upon to follow industry norms, and which are clearly not on the frontier. Until they show otherwise, they clearly do not belong in the category that includes OpenAI, Anthropic, Google, xAI and DeepSeek.

Techikansh: I am just gonna leave this here…

Table of Contents

Llama We Doing This Again.

Llama the License Favors Bad Actors.

Llama You Do It This Way.

Llama Fight in the Arena.

Llama Would You Cheat on Other Benchmarks.

Llama It So Bad on Independent Benchmarks.

Llama You Don’t Like It.

Llama Should We Care.

Llama We Doing This Again

Meta released the first two Llama 4 models last Saturday, and there is a code change indicating that the original plan was to do it Monday and it got moved up. In general, releasing on a Saturday is such bad strategy it simply isn’t done. Zuck says ‘that’s when it was ready’ but that is not an explanation.

People are wondering why made an exception and did it anyway. I have two hypotheses for what happened (note: I do not have any private information here).

They moved it up because the tariffs were about to potentially cause a Black Monday stock market crash, and Meta wanted to get ahead of that to protect themselves and also to not have the release buried under other news. This seems entirely reasonable under the circumstances.

They released on Saturday to bury it, because it isn’t any good.

Those two look to be at cross-purposes, but I’m not so sure. Suppose, for the sake of argument here, that Llama-4 sucks.

Investors can’t really tell the difference, especially not by Mo

...
Learned pain as a leading cause of chronic pain
by SoerenMind — 2025-04-09T11:57:58.523Z

Epistemic status: Amateur synthesis of medical research that is still recent but now established enough to make it into modern medical textbooks. Some specific claims vary in evidence strength. I’ve spent ~20-30 hours studying the literature and treatment approaches, which were very effective for me.

Disclaimer: I'm not a medical professional. This information is educational only, not medical advice. Consult healthcare providers for medical conditions.

Key claims

This post builds on previous discussions about the

fear-pain cycle

and learned

chronic pain

. The post adds the following claims:

Neuroplastic pain - pain learned by the brain (and/or spinal cord) - is a well-evidenced phenomenon and widely accepted in modern medical research (very high confidence).
It explains many forms of chronic pain previously attributed to structural causes - not just wrist pain and back pain (high confidence). Other conditions include everything from pain in the knees, pelvis, bowels, neck, and the brain itself (headaches). Some practitioners also treat chronic

fatigue

(inc. Long-

COVID

),

dizziness and nausea

in a similar way but I haven't dug into this.
It may be

one of

the most common or even the single

most common

cause of chronic pain (moderate confidence).
There are increasingly

useful

resources

, well-tested

treatments

with very large effect size, and trained

practitioners

.
Doctors are often unaware that neuroplastic pain exists because the research is recent and not their specialty. They often attribute it to tissue damage or structural causes like minor findings in medical imaging and biomechanical or blood diagnostics, which often fuels the fear-pain cycle.

My personal experience with with chronic pains and sudden relief

My first chronic pain developed in the tendons behind my knee after running. Initially manageable, it progressed until I couldn't stand or walk for more than a few minutes without triggering days of pain. Medical examinations revealed inflammation and structural changes in the tendons. The prescribed treatments—exercises, rest, stretching, steroid injections—provided no meaningful relief.

Later, I developed unexplained tailbone pain when sitting. This quickly became my dominant daily discomfort. Specialists at leading medical centers identified a bone spur on my tailbone and unanimously conclude...
Does the universe's recognition of measurement provide stronger evidence for being in a simulation than universal fine-tuning?
by amelia — 2025-04-09T08:20:10.561Z

THE SIMULATION HYPOTHESIS

In 2003, Nick Bostrom used a probabilistic argument to introduce what became known as the simulation hypothesis: either advanced civilizations rarely emerge, they rarely run simulations, or we’re almost certainly in a simulation.(1)

FINE-TUNING AS SIMULATION EVIDENCE

Since then, public discourse has suggested the universe’s fine-tuning—the precise constants and conditions enabling life—supports the idea of being in a simulation. If there’s a “one in billions” chance of such fine-tuning occurring at random, then fine-tuning may hint at the existence of a fine-tuner, such as a programmer/simulator.

POSSIBLE DISMISSAL OF FINE-TUNING AS SIMULATION EVIDENCE

Most philosophers and cosmologists rely on multiverse hypotheses to argue against the idea that fine-tuning implies a fine-tuner. If billions of universes exist or have existed, our life-conducive outlier is possible, and nobody would be alive in universes that are not life-conducive to notice those cases. Thus, noticing that one is in a life-conducive universe no longer seems remarkable.(2) It is true that some individuals, such as Ian Hacking and Roger White, counter the counterarguments by arguing that multiverse hypotheses engage in logical fallacies—such as the Inverse Gambler’s Fallacy.(3) Yet most academics continue to dismiss fine-tuning with multiverse hypotheses. So is there any other physical evidence even mildly supportive of the hypothesis that we may be in a simulation?

THE DOUBLE-SLIT EXPERIMENT

During the double-slit experiment of quantum physics, photons and electrons exhibit wave-like interference when unmeasured, but act as particles when their paths are measured. More specifically, a particle begins in a superposition of two Gaussian wavepackets exiting two slits. Without measurement, these wavepackets spread and overlap, producing an interference pattern on a detection screen. However, introducing a measurement apparatus to determine which slit the particle passes through disrupts this pattern.

In the article "Measurement-induced Decoherence and Information in Double-Slit Interference" the authors explore how measurement precision affects this outcome, using a tunable apparatus which presumably interacts with the particle.(4) When the apparatus perfectly distinguishes the particle’s path, it entangles with the system, decohering the superposition i...
Taxonomy of possibility
by dkl9 — 2025-04-09T04:24:09.439Z
Link: https://dkl9.net/essays/can_want.html

Followup to: Will issues are quite nearly skill issues

You do a thing iff you can do it and you want to do it.

Something you can do, but didn't, looks just the same to the simple observer as something you can't do. Likewise with what you want to do, but didn't. When you don't do something, obvious physical reality leaves ambiguous whether you were unable or disinclined.

Here are some things I don't do:

fly
stay awake continuously for a year
destroy Instagram
enter the gym without showing ID
take notes on a chalkboard
crush my finger between door and frame

Each of those things, I don't do for a single reason: I'm unable to do the first three, and disinclined to do the last three. To categorise them like that is intuitive. The intuition here, made precise, relies on counterfactuals. "Unable" and "disinclined" each classify answers to "what would have to change to make you do that?".

Can

"Can" is the opposite of "unable". "Unable" means that the change involves granting ability to they who would act, i.e. teaching a technique, providing a tool, fixing the body, or altering the environment.

If you told me where the airport is, or gave me a jetpack I trust as safe enough, I would fly, at least occasionally. If you modified my brain to consolidate learning and my body to recover while awake, and pointed me to a large supply of stimulants, I would gladly stay awake for a full year, maybe more. If I studied malicious hacking methods, and you told me the network address to access Instagram's internal database and server controls, I would wipe their data and shut them down.

I can sneak into the gym, or use a chalkboard, or crush my finger, insofar as teaching and equipping me wouldn't change that I don't do any of those.

Want

"Want" is the opposite of "disinclined". "Disinclined" means that the change involves convincing or incentivising the one who would act.

If those at the front desk of the gym turned off the card-scanner and ignored visitors walking in, my seeing that would convince me to abandon inhibition and run past. If you invite me to lecture in the room where I already have chalk and board, and I see value in doing so, I'd write on the chalkboard. If you credibly offer a suitably large amount of money for me to crush my finger, I'd go slam a door onto it.

I want to fly, stay awake for years, and wreck Instagram, insofar as talking me into or bribing me for any of those would...
Short Timelines Don't Devalue Long Horizon Research
by Vladimir_Nesov — 2025-04-09T00:42:07.324Z

Short AI takeoff timelines seem to leave no time for some lines of alignment research to become impactful. But any research rebalances the mix of currently legible research directions that could be handed off to AI-assisted alignment researchers or early autonomous AI researchers whenever they show up. So even hopelessly incomplete research agendas could still be used to prompt future capable AI to focus on them, while in the absence of such incomplete research agendas we'd need to rely on AI's judgment more completely. This doesn't crucially depend on giving significant probability to long AI takeoff timelines, or on expected value in such scenarios driving the priorities.

Potential for AI to take up the torch makes it reasonable to still prioritize things that have no hope at all of becoming practical for decades (with human effort). How well AIs can be directed to advance a line of research further (and the quality of choices about where they should be directed) depends on how well these lines of research are already understood. Thus it can be important to make as much partial progress as possible in developing (and deconfusing) them in the years (or months!) before AI takeoff. This notably seems to concern agent foundations / decision theory, in contrast with LLM interpretability or AI control, which more plausibly have short term applications.

In this sense current human research, however far from practical usefulness, forms the data for aligning the early AI-assisted or AI-driven alignment research efforts. The judgment of human alignment researchers who are currently working makes it possible to formulate more knowably useful prompts for future AIs (possibly in the run-up to takeoff) that nudge them in the direction of actually developing even preliminary theory into practical alignment techniques.