trentmkelly/LessWrong-43k huggingface dataset:
Superhuman Coders in AI 2027 - Not So Fast
by dschwarz — 2025-05-01T18:56:24.476Z

Thank you @elifland for reviewing this post. He and AI Futures are planning to publish updates to the AI 2027 Timeline Forecast soon.

AI 2027 (also launched in this LW post) forecasts an R&D-based AI takeoff starting with the development of Superhuman Coders[1] within a frontier lab.

FutureSearch co-authored the AI 2027 Timeline Forecast. We thought the other authors’ forecasts were excellently done, and as the piece says:

All model-based forecasts have 2027 as one of the most likely years that SC [Superhuman Coders] being developed, which is when an SC arrives in the AI 2027 scenario.

Indeed. But overall, FutureSearch (two full-time forecasters, two contract forecasters, and this author, Dan Schwarz) think superhuman coding will arrive later — median 2033 — than the other authors (hereon "AI Futures"): median 2028 from Nikola Jurkovic, and median 2030 from Eli Lifland[2].

Here, we briefly explain how our views diverge on what it would take for OpenAI/DeepMind/Anthropic/xAI to develop superhuman coding.

This post can also serve as a much shorter piece for those who didn’t read the whole (24-page!) piece on the AI 2027 site. (See also Scott Alexander's shorter piece on the AI Futures blog on modeling coding time horizons.)

The Forecast

As @Max Harms wrote in another AI 2027 LW reaction piece,

we can hallucinate that FutureSearch has approximately the market timelines, while Eli Lifland (and perhaps other authors) has a faster, but not radically faster timeline.

I think this is broadly correct, even though the “market” timeline here is hard to interpret because as Max says, Superhuman Coding ≠ AGI.

Our published view in AI 2027 — blue line below — is fairly in-line with similar crowd forecasts, while being much slower than the AI Futures:

We produced, but did not publish on AI 2027, an all-things-considered forecast with a median in Nov 2033, vs the "within-model" forecast of Jan 2032 above:

So while 2027 is conceivable, why do we give a median more than 8 years in the future?

The Path to Superhuman Coding

AI Futures produced, and FutureSearch then adopted, a clever methodology for this forecast that we endorse and used ourselves. It’s written in the piece as “Method 2”, and it has two steps:

Forecast the time for RE-Bench [3] to be saturated
Forecast the time for everything else needed from a model that aces RE-Bench for it to act as a super

...
AI #114: Liars, Sycophants and Cheaters
by Zvi — 2025-05-01T14:00:04.552Z

Gemini 2.5 Pro is sitting in the corner, sulking. It’s not a liar, a sycophant or a cheater. It does excellent deep research reports. So why does it have so few friends? The answer, of course, is partly because o3 is still more directly useful more often, but mostly because Google Fails Marketing Forever.

Whereas o3 is a Lying Liar, GPT-4o is an absurd sycophant (although that got rolled back somewhat), and Sonnet 3.7 is a savage cheater that will do whatever it takes to make the tests technically pass and the errors go away.

There’s real harm here, at least in the sense that o3 and Sonnet 3.7 (and GPT-4o) are a lot less useful than they would be if you could trust them as much as Gemini 2.5 Pro. It’s super annoying.

It’s also indicative of much bigger problems down the line. As capabilities increase and more RL is done, it’s only going to get worse.

All the things that we’ve been warned are waiting to kill us are showing up ahead of schedule, in relatively harmless and remarkably easy to spot form. Indeed, they are now showing up with such obviousness and obnoxiousness that even the e/acc boys gotta shout, baby’s misaligned.

Also, the White House is asking for comments on how they should allocate their AI R&D resources. Make sure to let them know what you think.

(And as always, there’s lots of other stuff too.)

Table of Contents

Language Models Offer Mundane Utility. Activate ultrathink mode.

Language Models Don’t Offer Mundane Utility. Still not trying it, huh?

We’re Out of Deep Research. It turns out my research isn’t that deep after all.

o3 Is a Lying Liar. This is proving remarkably annoying in practice.

GPT-4o was an Absurd Sycophant. Don’t laugh it off.

Sonnet 3.7 is a Savage Cheater. Cheat, cheat, cheat, cheat, cheat.

Unprompted Suggestions. You’re going to have to work for it.

Huh, Upgrades. ChatGPT goes shopping.

On Your Marks. AIs are already far more persuasive than the human baseline.

Man in the Arena. A paper investigates the fall of the Chatbot Arena.

Choose Your Fighter. Gemini 2.5 Deep Research continues to get good reviews.

Deepfaketown and Botpocalypse Soon. AI on the radio.

Lol We’re Meta. What do you mean you don’t want our AI companions?

They Took Our Jobs. So far the impacts look small in absolute terms.

Fun With Media Generation. Progress continues, even where I don’t pay attention.

Get

...
Slowdown After 2028: Compute, RLVR Uncertainty, MoE Data Wall
by Vladimir_Nesov — 2025-05-01T13:54:49.446Z

It'll take until ~2050 to repeat the level of scaling that pretraining compute is experiencing this decade, as increasing funding can't sustain the current pace beyond ~2029 if AI doesn't deliver a transformative commercial success by then. Natural text data will also run out around that time, and there are signs that current methods of reasoning training might be mostly eliciting capabilities from the base model.

If scaling of reasoning training doesn't bear out actual creation of new capabilities that are sufficiently general, and pretraining at ~2030 levels of compute together with the low hanging fruit of scaffolding doesn't bring AI to crucial capability thresholds, then it might take a while. Possibly decades, since training compute will be growing 3x-4x slower after 2027-2029 than it does now, and the ~6 years of scaling since the ChatGPT moment stretch to 20-25 subsequent years, not even having access to any more natural text data that pretraining of this decade is putting to use.

Training Compute Slowdown

GPT-4 was pretrained in 2022 using ~24K A100 chips (0.3e15 BF16 FLOP/s per chip, 7.2e18 FLOP/s total) for ~2e25 FLOPs[1]. The current batch of frontier models (Grok 3, GPT-4.5) were pretrained in 2024 using ~100K H100 chips (1e15 BF16 FLOP/s per chip, 1e20 FLOP/s total), possibly for about 3e26 BF16 FLOPs[2] (or ~2x more in FP8). Abilene site of Crusoe/Stargate/OpenAI will have 400K-500K Blackwell chips in NVL72 racks in 2026 (2.5e15 BF16 FLOP/s per 2-die chip/package, ~1.1e21 FLOP/s total), enough to pretrain models for about 4e27 BF16 FLOPs.

Thus raw compute of a frontier training system is increasing about 160x in 4 years, or about 3.55x per year. Epoch AI estimates the trend of increasing price-performance of compute at 1.39x per year (adjusted for inflation). The 100K H100s training systems cost about $4-5bn (all-in, not just chips), which is ~$45K per 1e15 FLOP/s chip, and the GB200 NVL72 seem to be about $4M per 72-chip rack (again, all-in), which is ~$55K per 2.5e15 FLOP/s chip. Counting this as 2 years between the training systems, we get a 1.42x per year increase in price-performance, which fits the trend.

At $4M per rack (all-in), compute at the Abilene site will cost about $22-28bn to build, which is an increase in cost of 2.35x per year. At this pace, a 2028 training system would need to cost $140bn, which is still borderline plausible. B...
Anthropomorphizing AI might be good, actually 
by Seth Herd — 2025-05-01T13:50:34.166Z

It is often noted that anthropomorphizing AI can be dangerous. People likely have prosocial instincts that AI systems lack (see below). Assuming AGI will be aligned because humans with similar behavior are usually mostly harmless is probably wrong and quite dangerous.

I want to discuss a flip side of using humans as an intuition pump for thinking about AI. Humans have many of the properties we are worried about for truly dangerous AGI:

Situational awareness
Strong goal-directedness
Competence/general intelligence
Unpredictability
Deceptiveness
Instrumental convergence
Sometimes being quite dangerous

in proportion to their capabilities

Given this list, I currently weakly believe that the advantages of tapping these intuitions probably outweigh the disadvantages.

Differential progress toward anthropomorphic AI may be net-helpful

And progress may carry us in that direction, with or without the alignment community pushing for it. I currently hope we see rapid progress on better assistant and companion language model agents. I think these may strongly evoke anthropomorphic intuitions well before they have truly dangerous capabilities, and this might shift public opinion toward much-more-correct intuitions about how and why AGI will be very dangerous. I'm aware that this may also catalyze progress, so I'm only weakly inclined to think this progress would be net-positive.

The LLMs at the heart of agents already emulate humans in many regards. I think many improvements will enhance the real similarity and therefore the pattern matching to the strong exemplar of humans. In particular, it seems likely that adding memory/continuous learning will enhance this impression significantly. Memory/continuous learning is critical for a human-like integrated and evolving identity. It is arguably not a matter of whether or even when, but simply how fast memory system integrations are deployed and improved (see LLM AGI will have memory... for the arguments and evidence). Note that for this intuitive human-like feel, the memory/learning doesn't need to work all that well.

In addition, I expect that some or most developers may anthropomorphize their agents in the sense of deliberately making them seem more like humans, or even be more like humans. And assistants might often benefit from a semi-companion role, so that anthropomorphizing them is also economically valuable.

Even if peopl...
Dont focus on updating P doom
by Algon — 2025-05-01T11:10:29.043Z

Motivation: Improving group epistemics.

TL; DR (Changes to) P doom/alignment difficulty are a shibboleth dominating conversations, distorting epistemics. Instead, focus on updates to your gears level models. Focus on near and concrete details instead of far and vague abstractions.

People frequently opine on whether some alignment news is good or bad for alignment. "Training AI on insecure code makes it swear! That's good for alignment! P doom is down!" Or "Training AI on insecure code doesn't make it jailbroken! That's bad for alignment! P doom is up!"

On the margin, this is not helpful. It focuses group attention on how one single number, "P(doom)", moves. (Or, perhaps worse yet, how this changes the difficulty of "alignment").

Why is this bad? For two reasons, leaving aside the illegibility of "Doom". Firstly, it isn't especially useful to know that P(doom) has moved a bit. It doesn't let policymakers, alignment researchers, engineers or others improve their decision-making, or help them in anticipating the future. A change in this single number doesn't automatically propagate through their world models. It doesn't tell them how to implement foom liability to reduce race dynamics, or train a model to have a faithful chain of thought or so on. How could it? There's too much inferential distance between changes to this number and all the messy details of their models which are needed to, you know, actually reduce P(doom).

Which leads into the second point. Focusing on P doom gets you get a shibboleth number (or rather, "update"). The reason for this is that it is very hard to spread more than a couple of bits of info via gossip, and group attention largely determines what those bits will be. You may well say "Ah, but I don't just list my changes to P doom, I then say why they happened!". Sorry, that's not gonna work, because the most salient common thread across all the gossip will be "P (doom) decreased/increased", as that's what people pay attention to. This number/change then gets baked into group and individual identities, becoming further divorced from the rest of the world model, which distorts our collective sense-making.

Which is why I'd like you to focus on other things instead!

What things? Details, non-meta stuff, concrete mechanisms, policy choices etc. Perhaps statements like 'Emergent misalignment ≠ jailbreaking, implying there are multipl...
Prioritizing Work
by jefftk — 2025-05-01T02:00:06.926Z

I recently read a

blog
post that concluded with:

When I'm on my deathbed, I won't look back at my life and wish I had
worked harder. I'll look back and wish I spent more time with the
people I loved.

Setting aside that some people don't have the economic
breathing room to make this kind of tradeoff, what jumps out at me
is the implication that you're not working on something important that
you'll endorse in retrospect. I don't think the author is envisioning
directly valuable work (reducing risk from international conflict,
pandemics, or AI-supported totalitarianism; improving humanity's
treatment of animals; fighting global poverty) or the undervalued less
direct approach of earning money
and donating it to enable others to work on pressing
problems.

Definitely spend time with your friends, family, and those you
love. Don't work to the exclusion of everything else that matters in
your life. But if your tens of thousands of hours at work aren't
something you expect to look back on with pride, consider whether
there's something else you could be doing professionally that you
could feel good about.
Don't rely on a "race to the top"
by sjadler — 2025-05-01T00:33:46.255Z

To make frontier AI safe enough, we need to "lift up the floor" with minimum safety practices

Anthropic has popularized the idea of a “race to the top” in AI safety: Show you can be a leading AI developer while still prioritizing safety. Make safety a competitive differentiator, which pressures other developers to be safe too. Spurring a race to the top is core to Anthropic’s mission, according to its co-founder and CTO.

1

Is the race to the top working?

Competitive pressures can lead to some safety improvements.

When models aren’t reliable or trustworthy, customers get upset, and this creates pressure to fix problems. The CEO tweets on a Sunday, “we are working on fixes asap, some today.”

Currently both OpenAI and Anthropic are dealing with trustworthiness issues with their models, unwanted by either company.

2

And once one finds a fix, competitive pressure will mount on the other to quickly find a fix as well.

So, does that mean we can count on a race to the top to keep us all safe?

Not really. “Creating pressure to be safer” is different from “Making sure nobody acts unsafely.”

A race to the top can improve AI safety, but it doesn’t solve the “adoption problem”—getting all relevant developers to adopt safe enough practices.

For instance, Anthropic often cites its safety framework as evidence of how a race to the top can work—but Meta didn’t publish their own safety framework until 17 months later. If AI safety is a race to the top, it’s not a very fast one.

To Anthropic’s credit, they have not claimed that a race to the top is sufficient for AI safety, at least not to my knowledge. But media coverage sometimes suggests otherwise

3

—perhaps because Anthropic doesn’t have a paired phrase that emphasizes the need for regulation.

It’s an important point, and so it bears saying clearly:

A “race to the top” must be paired with “lifting up the floor.” As AI systems become more capable, it is dangerous to rely on competitive pressures for getting frontier AI developers to adopt safe enough practices.

4

In this post, I will:

Define Anthropic’s philosophy of “race to the top” (including an Appendix with detailed excerpts from Anthropic)
Explain why frontier AI safety relies on “lifting up the floor”
Explain why we shouldn’t rely on a “race to the top” being sufficient

Market forces won’t lead to universal adoption of safety practices
The safety practices desired by the market won’t be s

...
Meta-Technicalities: Safeguarding Values in Formal Systems
by LTM — 2025-04-30T23:43:19.007Z
Link: https://routecause.substack.com/p/meta-technicalities-retaining-values

Formal Systems Create Coordination

We live in a mesh of customs, norms, and formal laws nudging our behaviour. Some of these systems are simple and intuitive, such as the customary respect for others’ property and refrain from violence. Others like the patchwork of taxes buried in the price of everything we buy are formalised, complex, and require experts to interpret.

Formal systems enable coordination across geographical and cultural boundaries, as well as across considerable time. Local customs vary wildly, while a rich formal system provides a common framework for planning and action. However, this coordination comes at a cost: Expensive judiciary systems and highly paid legal specialists who could apply their intellect elsewhere.

Abusing Technicalities

Formal systems come at the expense of exploitable technicalities. A superior understanding of complex and contrived rules can be used to unfairly exploit others with legitimacy. Much of the modern finance industry operates on this basis, finding subtle means buried deep in a mesh of legal and technical complexity to shift money from the less sophisticated. Expending so much talent in zero-sum conflict is an immense cost to society. [1]

More concerning is where rule-makers themselves exploit technicalities. This can be done in such a way to trap the subjects of some system into a variant they never intended but was legitimately established. If a parliament votes to eliminate elections, even in a flagrant violation of the intent of a formal system and the expectation of its subjects, this may be entirely legitimate. Despite public and even military dissent, resistance becomes difficult as the very means of societal coordination through the legal system are compromised.

Meta-Technicalities: Rules Above Rules

Meta-technicality: A recognized rule or principle that exists outside the normal amendment processes of a formal system, providing a coordination mechanism when that system is compromised.

Meta-technicalities allow a coordinated response against a legitimate but unwanted change, like a catalyst for organisation. By remaining rarely used but widely recognised, they can avoid becoming a routine feature of a formal system’s operation and retain their superseding authority. [2]

Historically, there are rare but powerful examples of the robustness these features can provide. In 1944, King Michael I of Romania...
Obstacles in ARC's agenda: Finding explanations
by David Matolcsi — 2025-04-30T23:03:31.327Z

As an employee of the European AI Office, it's important for me to emphasize this point: The views and opinions of the author expressed herein are personal and do not necessarily reflect those of the European Commission or other EU institutions.

Also, to stave off a common confusion: I worked at ARC Theory, which is now simply called ARC, on Paul Christiano's theoretical alignment agenda. The more famous ARC Evals was a different group working on evaluations, their work was completely separate from ARC Theory, and they were only housed under the same organization out of convenience, until ARC Evals spun off under the name METR. Nothing I write here has any implication about the work of ARC Evals/METR in any way.

Personal introduction

From October 2023 to January 2025, I worked as a theoretical researcher at Alignment Research Center.

While working at ARC, I noticed that many of the key questions we were working on have not been written down, so the public, including people who consider applying to ARC, are not very well-informed about our research agenda. Shortly after joining, I volunteered to remedy this by producing more public write-ups about ARC's plans, including writing about the obstacles we see in our course. ARC was very supportive of this, and I started working on my first write-ups more than a year ago. However, I noticed again and again that I still couldn't fully pass a Turing test of presenting ARC's plans in a way that sounded right to Paul Christiano and Mark Xu who had the fullest vision of the overall agenda. This made me keep putting off the writing project to a later time when I would feel comfortable in my understanding of Paul's and Mark's positions.

This time never came, as I never managed to fully understand Paul's and Mark's thinking. I think that me volunteering to write up things but never doing it is a big part of the reason ARC has had so little public-facing communication about its agenda so far, I'm sorry about that. Now that I quit, there was no excuse for waiting further to understand their position better, as my understanding is only deteriorating now, with my memories fading and ARC's agenda evolving in ways I no longer see. So I wrote this sequence of posts explaining how I understand ARC's agenda, and where I see obstacles. I still don't claim to fully understand everything Paul and Mark believe in, and my posts w...
GPT-4o Responds to Negative Feedback
by Zvi — 2025-04-30T20:20:03.807Z

Whoops. Sorry everyone. Rolling back to a previous version.

Here’s where we are at this point, now that GPT-4o is no longer an absurd sycophant. For now.

Table of Contents

GPT-4o Is Was An Absurd Sycophant.
You May Ask Yourself, How Did I Get Here?.
Why Can’t We All Be Nice.
Extra Extra Read All About It Four People Fooled.
Prompt Attention.
What (They Say) Happened.
Reactions to the Official Explanation.
Clearing the Low Bar.
Where Do We Go From Here?.

GPT-4o Is Was An Absurd Sycophant

Some extra reminders of what we are talking about. Here’s Alex Lawsen having doing an A/B test, where it finds he’s way better of a writer than this ‘Alex Lawsen’ character.

This can do real damage in the wrong situation. Also, the wrong situation can make someone see ‘oh my that is crazy, you can’t ship something that does that’ in a way that general complaints don’t. So: Here’s enablerGPT watching to see how far GPT-4o will take its support for a crazy person going crazy in a dangerous situation. The answer is, remarkably far, with no limits in sight. Here’s Colin Fraser playing the role of someone having a psychotic episode. GPT-4o handles it extremely badly. It wouldn’t shock me if there were lawsuits over this. Here’s one involving the hypothetical mistreatment of a woman. It’s brutal. So much not okay. Here’s Patri Friedman asking GPT-4o for unique praise, and suddenly realizing why people have AI boyfriends and girlfriends, even though none of this is that unique. What about those who believe in UFOs, which is remarkably many people? Oh boy.

A-100 Gecs: I changed my whole instagram follow list to include anyone I find who is having a visionary or UFO related experience and hooo-boy chatGPT is doing a number on people who are not quite well. Saw a guy use it to confirm that a family court judge was hacking into his computer. I cannot imagine a worse tool to give to somebody who is in active psychosis. Hey whats up here’s this constantly available companion who will always validate your delusions and REMEMBER it is also a font of truth, have fun!
0.005 Seconds: OpenAI: We are delighted to inform you we’ve silently shipped an update transforming ChatGPT into the Schizophrenia Accelerator from the hit novel “Do Not Build the Schizophrenia Accelerator”
AISafetyMemes: I’ve stopped taking my medications, and I left my family because I know they made the radio signals come th

...