trentmkelly/LessWrong-43k huggingface dataset:
Gemini 2.5 is the New SoTA
by Zvi — 2025-03-28T14:20:03.176Z

Gemini 2.5 Pro Experimental is America’s next top large language model.

That doesn’t mean it is the best model for everything. In particular, it’s still Gemini, so it still is a proud member of the Fun Police, in terms of censorship and also just not being friendly or engaging, or willing to take a stand.

If you want a friend, or some flexibility and fun, or you want coding that isn’t especially tricky, then call Claude, now with web access.

If you want an image, call GPT-4o.

But if you mainly want reasoning, or raw intelligence? For now, you call Gemini.

The feedback is overwhelmingly positive. Many report Gemini 2.5 is the first LLM to solve some of their practical problems, including favorable comparisons to o1-pro. It’s fast. It’s not $200 a month. The benchmarks are exceptional.

(On other LLMs I’ve used in the past and may use again when they update: I’ve stopped using Perplexity entirely now that Claude has web access, I never use r1, and I only use Grok narrowly for when I need exactly real time reactions from Twitter.)

Table of Contents

Introducing Gemini 2.5 Pro.

Their Lips are Sealed.

On Your Marks.

The People Have Spoken.

Adjust Your Projections.

Introducing Gemini 2.5 Pro

Google DeepMind: Think you know Gemini? Think again.

Meet Gemini 2.5: our most intelligent model The first release is Pro Experimental, which is state-of-the-art across many benchmarks – meaning it can handle complex problems and give more accurate responses.

Try it now.

Gemini 2.5 models are thinking models, capable of reasoning through their thoughts before responding, resulting in enhanced performance and improved accuracy.

Gemini 2.5 Pro is available now in Google AI Studio and in the Gemini app for Gemini Advanced users, and will be coming to Vertex AI soon. We’ll also introduce pricing in the coming weeks, enabling people to use 2.5 Pro with higher rate limits for scaled production use.

Logan Kilpatrick: This will mark the first experimental model with higher rate limits + billing. Excited for this to land and for folks to really put the model through the paces!

This was the #1 point of feedback, besides very long model names, in how we do experimental launches. More to come!

[Live API production rollout] very soon : )

Pliny the Liberator pwned this on the spot of course, also building a new jailbreak prompt because the old prompt worked right away and that was no f...
Will the Need to Retrain AI Models from Scratch Block a Software Intelligence Explosion?
by Tom Davidson — 2025-03-28T14:12:02.163Z

Tl;dr: no.

This is a rough research note – we’re sharing it for feedback and to spark discussion. We’re less confident in its methods and conclusions.

Once AI fully automates AI R&D, there might be a period of fast and accelerating software progress – a software intelligence explosion (SIE).

One objection to this is that it takes a long time to train SOTA AI systems from scratch. Would retraining each new generation of AIs stop progress accelerating during an SIE? If not, would it significantly delay an SIE? This post investigates this objection.

Here are my tentative bottom lines:

Retraining won’t stop software progress from accelerating over time.

Suppose that, ignoring the need for retraining, software progress would accelerate over time due to the AI-improving-AI feedback loop.
A simple theoretical analysis suggests that software progress will still accelerate once you account for retraining. Retraining won’t block the SIE.

Retraining will cause software progress to accelerate somewhat more slowly.

But that same analysis suggests that your software progress will accelerate more slowly.
Quantitatively, the effect is surprisingly small – acceleration only takes ~20% longer.
However, if acceleration was going to be extremely fast, retraining slows things down by more.

Retraining means that we’re unlikely to get an SIE in <10 months, unless either training times become shorter before the SIE begins or improvements in runtime efficiency and post-training enhancements are large.

Today it takes ~3 months to train a SOTA AI system.
As a baseline, we can assume that the first AI to fully automate AI R&D will also take 3 months to train.
With this assumption, simple models of the SIE suggest that you’re unlikely to complete the SIE within 10 months. (In the model, the SIE is “completed” when AI capabilities approach infinity – of course in reality they would reach some other limit sooner).
BUT if by the time AI R&D is fully automated, SOTA training runs have already shortened to one month or less, then completing an SIE in less than 10 months is much more plausible.
And improvements in runtime efficiency and other post-training enhancements (which I’m not modelling in my analysis) could potentially allow very fast takeoff without needing to retrain from scratch.

How did I reach these tentative conclusions?

Theoretical analysis
First, I conducted a very basic theoretica...
How We Might All Die in A Year
by Greg C — 2025-03-28T13:22:36.863Z
Link: https://x.com/gcolbourn/status/1905346826330341646

Ilya thought back again to when he’d overheard that conversation between two of his junior colleagues. They were discussing an “AI ending the world” story. This was early in the year, although it felt like years ago now given how fast things had been moving. The year, 2025, was now drawing to a close... and maybe it’ll end up as the last full human year ever. Ilya was suddenly saddened by the thought, to an unexpected degree. He was on the verge of tears.

The story went like this: the AI plots to overthrow humanity, makes mirror life, kills almost everyone (3% survive!) and takes over the world; the survivors get to live in a suburban utopia, but it’s kind of like a zoo in a bubble, and the AI is in control of everything and off doing inscrutable stuff in space at the end.

“Clymer”: the name of the author popped into Ilya’s head. He found the story online and started to read..

"How AI Might Take Over in 2 Years

I’m not a natural “doomsayer.” But unfortunately, part of my job as an AI safety researcher is to think about the more troubling scenarios.

I’m like a mechanic scrambling last-minute checks before Apollo 13 takes off. If you ask for my take on the situation, I won’t comment on the quality of the in-flight entertainment, or describe how beautiful the stars will appear from space.

I will tell you what could go wrong. That is what I intend to do in this story.

Now I should clarify what this is exactly. It's not a prediction. I don’t expect AI progress to be this fast or as untamable as I portray. It’s not pure fantasy either.

It is my worst nightmare.

It’s a sampling from the futures that are among the most devastating, and I believe, disturbingly plausible – the ones that most keep me up at night.

I’m telling this tale because the future is not set yet. I hope, with a bit of foresight, we can keep this story a fictional one.”

The disclaimer seemed wholly inappropriate to Ilya, in light of what was now happening…

“Ripples before waves

The year is 2025 and the month is February. OpenEye recently published a new AI model they call U2. The product and the name are alike. Both are increments of the past. Both are not wholly surprising.

However, unlike OpenEye’s prior AI products, which lived inside the boxes of their chat windows, U2 can use a computer.

Some users find it eerie to watch their browser flash at irregular intervals and their mouse flick at inhuman speed

...
The vision of Bill Thurston
by TsviBT — 2025-03-28T11:45:14.297Z

PDF version. berkeleygenomics.org. X.com. Bluesky.

William Thurston was a world-renowned mathematician. His ideas revolutionized many areas of geometry and topology[1]; the proof of his geometrization conjecture was eventually completed by Grigori Perelman, thus settling the Poincaré conjecture (making it the only solved Millennium Prize problem). After his death, his students wrote reminiscences, describing among other things his exceptional vision.[2] Here's Jeff Weeks:

Bill’s gift, of course, was his vision, both in the direct sense of seeing geometrical structures that nobody had seen before and in the extended sense of seeing new ways to understand things. While many excellent mathematicians might understand a complicated situation, Bill could look at the same complicated situation and find simplicity.

Thurston emphasized clear vision over algebra, even to a fault. Yair Minksy:

Most inspiring was his insistence on understanding everything in as intuitive and immediate a way as possible. A clear mental image of a mathematical construction or proof was worth much more than a formalization or a calculation. [...] If you were able to follow the images and the structure he was exploring, you obtained beautiful insights; but if you got lost, you were left with nothing [...].

There was something else exceptional about Thurston's vision. Benson Farb:

Bill was probably the best geometric thinker in the history of mathematics. Thus it came as a surprise when I found out that he had no stereoscopic vision, that is, no depth perception. Perhaps the latter was responsible somehow for the former? I once mentioned this theory to Bill. He disagreed with it, claiming that all of his skill arose from his decision, apparently as a first-grader, to “practice visualizing things” every day.

His vision problems were congenital (present from birth). Gabai and Kerckhoff:

Bill had a congenital case of strabismus and could not focus on an object with both eyes, eliminating his depth perception. He had to work hard to reconstruct a three-dimensional image from two two-dimensional ones. Margaret worked with him for hours when he was two, looking at special books with colors. His love for patterns dates at least to this time. As a first-grader he made the decision “to practice visualization every day.” Asked how he saw in four or five dimensions he said it is the same as in th

...
What Uniparental Disomy Tells Us About Improper Imprinting in Humans
by Morpheus — 2025-03-28T11:24:47.133Z
Link: https://www.tassiloneubauer.com/docs/uniparental-disomy.html

TLDR

We do not yet understand all the genetic imprints that play an important role in early embryonic development. Known imprinting disorders (Prader-Willi, Angelman, etc.) however, do seem to cover all the most detrimental imprints, since for all cases of uniparental disomy (having 1 pair of chromosomes from only one parent), we know of either the imprinting disorder or of a phenotypically normal case. One exception is maternal UPD 19, which might be rare because proper imprinting on chromosome 19 is crucial for early development in humans.

Imprinting and germline engineering

Most strong methods of germline engineering require us to know what the proper epigenomics of an embryo look like. Measuring the epigenome is quite tricky. Maintaining specifically imprinting is even more tricky. We currently have no method of culturing naive embryonic stem cells that does not lead to loss of imprinting.[1] While there is a lot of literature on imprinting and experiments in mice to determine the mechanisms behind known imprints, research in this area does not happen with the explicit goal of germline engineering. There is no comprehensive list of all human imprints.[2] So there might be regions whose importance for germline engineering in humans might be overlooked if there is no known associated disorder in humans. Hence, this post.

This post assumes basic familiarity with epigenetics. The next paragraph is a 1-paragraph summary, but I recommend reading these two excellent posts on epigenetics. If you know what DNA methylation and imprinting are, you can skip the posts.

The short explanation is that DNA methylations are small chemical marks on the DNA that change which proteins bind well to the DNA. Together with other chemical marks on histones, which are spindles that DNA is rolled around, these marks are how a cell knows that it is a skin cell, rather than a neuron. Imprinting is a phenomenon where the two sets of chromosomes in early mammal embryos behave differently depending on if they were inherited from the father or the mother side. Imprinting appears to be mostly due to differences in DNA methylation and histone marks.

Imprinting disorders and uniparental disomy

One of the earliest ways we discovered imprinting is through disorders like Prader-Willi syndrome, which had already been linked to deletions of some genes on Chromosome 15. Some patients th...
Explaining British Naval Dominance During the Age of Sail
by Arjun Panickssery — 2025-03-28T05:47:28.561Z
Link: https://arjunpanickssery.substack.com/p/explaining-british-naval-dominance

The other day I discussed how high monitoring costs can explain the emergence of “aristocratic” systems of governance:

Aristocracy and Hostage Capital

Arjun Panickssery · Jan 8
There's a conventional narrative by which the pre-20th century aristocracy was the "old corruption" where civil and military positions were distributed inefficiently due to nepotism until the system was replaced by a professional civil service after more enlightened thinkers prevailed ...

An element of Douglas Allen’s argument that I didn’t expand on was the British Navy. He has a separate paper called

“The British Navy Rules”

that goes into more detail on why he thinks institutional incentives made them successful from 1670 and 1827 (i.e. for most of the age of fighting sail).

In the Seven Years’ War (1756–1763) the British had a 7-to-1 casualty difference in single-ship actions. During the French Revolutionary and Napoleonic Wars (1793–1815) the British had a 5-to-1 difference in captured/destroyed ships, and a 30-to-1 difference for ships of the line—the largest and most powerful ships. By the 1800s, contemporary accounts expected British ships to defeat opponents that had 50% greater gunpower and crew.

Allen cites evidence that this dominance wasn’t because of superior technology and that French ships were even marginally superior by the end of the Seven Years’ War. There wasn’t any substantial difference in gunnery or other equipment, and any new technology was quickly adopted by every navy.

Similar to the previous post, we see a dynamic where

Monitoring is difficult: The sea was big and sometimes unmapped; communication was slow and limited.
There was a large random element: Wind and storms are often plausible excuses for not achieving specific objectives.

Incentives were misaligned: Captains mostly profited from capturing wealthy merchant ships and receiving a share of the spoils. In battle, the captain’s position on the raised quarterdeck made him vulnerable (Nelson was killed by a French sniper at Trafalgar on the quarterdeck of the HMS Victory).

The last point was most important: all navies in this period had a problem of captains and admirals who would shirk from combat. At Trafalgar, a third of the French fleet didn’t engage in the battle at all. Ships could often blame wind or chance for their failure to engage the enemy. Consider the prelude to Trafalgar:

One of the more famous epis

...
Will the AGIs be able to run the civilisation? 
by StanislavKrym — 2025-03-28T04:50:07.568Z

Even an AGI "aligned" to a purpose which doesn't imply humanity's survival but does require the AGI itself to achieve difficult feats like transforming the entire Solar System into something computing as many digits of pi as possible would obviously still need to produce the computing systems and gather the energy necessary for the systems' work. As I mentioned in my previous question, all the electrical energy generated in the world cannot sustain more than 3.3⋅1010 agents who interact with GPT-3 a hundred times a day while using 3Wh per interaction. The OpenAI-o3 model apparently requires more than 1 kWh per task.

However, the ARC-AGI task set shows the following trend: as the o3 models taught under the same paradigm increased the rate of success at ARC-AGI-1 tasks from 10% to 75%, the cost increased 500 times. The most expensive known model whose rate of success at ARC-AGI-1 tasks is 10% is GPT-4.5, indicating that a paradigm shift lowers the cost at most seven times. UPD: the ARC-AGI task set shows the following trend: as the o3 models taught under the same paradigm increased the rate of success at ARC-AGI-1 tasks from 10% (o3-mini low) to 75% (o3-low), the cost increased 5000 times, while the paradigm shift between o1(low) and o3-mini (low) lowered the cost 40 times at the cost of the success rate. The shifts from o1(medium) or o1(high) to o3-mini(medium) and o3-mini(high) lowered the cost 28 and 20 times respectively without the loss of capabilities.

The fact that o1-high model somehow solved 3% of ARC-AGI-2 tasks, unlike the o1-pro model (1%), while the o3-low model solved 4% of the tasks indicates that another paradigm shift and massive scaling are necessary to achieve the ARC-AGI-2 level of performance. It is likely that the cost per human-level task is actually at least another hundred of times bigger than the one demonstrated by o3.

Next we turn to the world's energy production. It is about 30 thousands TWh per year, meaning that even the AGI-run civilisation with the current-state energy industry is unlikely to solve more than 30 trillion of OpenAI-o3-level tasks per year or, presumably, more than 300 billion of ARC-AGI-2-level tasks per year (which is less than 1 billion of said tasks per day). On the other hand, the world energy industry in 2022 employed about 67 millions of humans of whom 32 millions worked at the foss...
Is AGI actually that likely to take off given the world energy consumption?
by StanislavKrym — 2025-03-27T23:13:14.959Z

How is AGI takeoff scenario compatible with the world energy consumption? While the latter in 2023 was of the order of 30 thousands of TWh per year, in 2023 and 2024 many researchers established the growing energy footprint of artificial intelligence. Even the GPT-3 interactions apparently require an electricity consumption of approximately 3 Wh, meaning that the world's energy consumption is unlikely to sustain more than 1015interactions a year. If someone were to allow as many people as possible to interact with GPT-3 a hundred times a day or even 300 hundred times a year (which is actually a little smaller!), then the world's energy consumption would be barely enough to sustain 101530000∼3.3⋅1010 interactors, which is less than 1.5 OOM higher than the population of China alone. What about smarter models like Claude or GPT4o? Am I mistaken somewhere? Or does it mean that AGI might replace just, say, half of the scientists, IT specialists and high-level art creators while letting the majority of humans live as normal?

P.S. There was an article on LessWrong that tried to promote a similar point: the human brains are already fairly well optimized, and a potential AGI is unlikely to exceed the same constraints, just like grey goo may be unlikely to be more efficient than life.

P.P.S. ChatGPT was installed over a hundred of millions of times, which is just 2.5 OOM away from the aforementioned estimate of 3.3⋅1010. Does anyone know how often people actually use it?
[Linkpost] The value of initiating a pursuit in temporal decision-making
by Gunnar_Zarncke — 2025-03-27T21:47:05.123Z

This eLife paper The value of initiating a pursuit in temporal decision-making by Elissa Sutlief, Charlie Walters, Tanya Marton, and Marshall G Hussain Shuler, seems to dissolve the question of the choice of temporal discount functions by explaining how it results from time-varying reward probabilities. It provides a framework to interpret decision heuristics as (mis)estimates of key parameters in this framework.

Abstract:

Reward-rate maximization is a prominent normative principle commonly held in behavioral ecology, neuroscience, economics, and artificial intelligence. Here, we identify and compare equations for evaluating the worth of initiating pursuits that an agent could implement to enable reward-rate maximization. We identify two fundamental temporal decision-making categories requiring the valuation of the initiation of a pursuit—forgo and choice decision-making—over which we generalize and analyze the optimal solution for how to evaluate a pursuit in order to maximize reward rate. From this reward-rate-maximizing formulation, we derive expressions for the subjective value of a pursuit, i.e. that pursuit’s equivalent immediate reward magnitude, and reveal that time’s cost is composed of an apportionment, in addition to, an opportunity cost. By re-expressing subjective value as a temporal discounting function, we show precisely how the temporal discounting function of a reward-rate-optimal agent is sensitive not just to the properties of a considered pursuit, but to the time spent and reward acquired outside of the pursuit for every instance spent within it. In doing so, we demonstrate how the apparent discounting function of a reward-rate-optimizing agent depends on the temporal structure of the environment and is a combination of hyperbolic and linear components, whose contributions relate the apportionment and opportunity cost of time, respectively. We further then show how purported signs of suboptimal behavior (hyperbolic discounting, the Delay effect, the Magnitude effect, the Sign effect) are in fact consistent with reward-rate maximization. Having clarified what features are and are not signs of optimal decision-making, we analyze the impact of the misestimation of reward rate-maximizing parameters in order to better account for the pattern of errors actually observed in humans and animals. We find that error in agents’ assessment of the apport

...
Alignment through atomic agents
by micseydel — 2025-03-27T18:43:14.569Z

I'm new to posting here, so I'll preface this with a bit about my background. I started teaching myself coding when I was 15, got a CS degree after high school, became a SWE, and am currently between jobs to work on my own agential PKM project. My last role was a hybrid one, doing backend and data engineering, and I leveraged the corporate wiki heavily.

After leaving that role for a vacation, I was taking and reviewing a lot of transcribed voice notes. I eventually started a PKMS ("personal knowledge management system") which might be better understood as a personal, private wiki of text/Markdown notes. One of the daily notes in my wiki is for tracking my cats' litter use, since one has a life-threatening chronic condition, and my background left a constant tension: why am I organizing these voice notes by hand, rather than with automation?

Today, you can think of some of the notes in my wiki as being managed by an atomic agent or actor. When the actor for the daily litter tracking note receives a message about the litter being sifted, the Markdown note is updated. If something went wrong with transcription, entity extraction, or anywhere else, I can update that note by hand and the actor will pickup where I left off. My actors are mostly explicit Scala code.

The wiki acts as externalized memory, the actors as externalized agency enacting my agenda, and together they act as externalized cognition. While I'm skeptical of LLMs, I could at any time have an actor spawn a parallel version of itself using any AI system to compare against my existing explicit encodings.

Someone could easily say my project is "not AI" and not relevant to alignment, but after learning of Jeff Hawkin's Thousand Brains theory, I have only felt more excited to run this experiment. In fact, after seeing a recent comment on the thousand brains forum, I've started adding a (very) rudimentary version of voting to my system. I need that to make use of Bayes' theorem and haven't added code for it yet, but I thought it would be better not to wait before making this post.

I have not yet done much reading on here about others' thoughts, so if there's something I'm obviously missing, feel free to link and I'll self-educate before replying.