trentmkelly/LessWrong-43k huggingface dataset:
How Fast Can Algorithms Advance Capabilities? | Epoch Gradient Update
by henryj — 2025-05-16T21:38:46.822Z
Link: https://epoch.ai/gradient-updates/how-fast-can-algorithms-advance-capabilities

I'm cross-posting my guest post on Epoch's Gradient Updates newsletter, in which I describe some new research from my team at UChicago's XLab — roughly, the algorithmic improvements that most improve capabilities at scale are the ones that require the most compute to find and validate.

This week’s issue is a guest post by Henry Josephson, who is a research manager at UChicago’s XLab and an AI governance intern at Google DeepMind.

In the AI 2027 scenario, the authors predict a fast takeoff of AI systems recursively self-improving until we have superintelligence in just a few years.

Could this really happen? Whether it’s possible may depend on if a software intelligence explosion — a series of rapid algorithmic advances that lead to greater AI capabilities — occurs.

A key crux in the debate about the possibility of a software intelligence explosion comes down to whether key algorithmic improvements scale from small models to larger models. If the most important algorithmic advances need a large amount of compute to demonstrate their effectiveness, then we should think that a software-only intelligence explosion is less likely. And so a fast takeoff could be bottlenecked by compute constraints.

In a recent preprint, my team at UChicago’s XLab — Spencer Guo, Teddy Foley, Jack Sanderson, Anqi Qu, and I — reviewed the literature and performed some small-scale tests to investigate this crux.

We found that, historically, the largest algorithmic advances couldn’t just be scaled up from smaller versions. They needed to have large amounts of compute to develop and validate, making a software intelligence explosion more difficult to achieve than it might seem.

Are the best algorithmic improvements compute-dependent?

In our survey of the literature, we classified algorithmic advances as either:

Compute-independent, meaning the benefits of the advance appear at small scales and continue or accelerate as scale increases
Compute-dependent, meaning the benefits of the advance only appear with high amounts of compute, and the algorithmic advance provides little benefit (or even hurts performance) at low compute levels

Our investigation began by examining the architectural evolution of language models, starting around 2017 and getting as close to the present as we could. We focused on well-documented model families where implementation details were publicly available: early GPTs, BERT va...
P-Values Know When You're Cheating
by Eggs — 2025-05-16T20:34:03.462Z

This post is not "the solution to p-hacking" or even "the problem with p-hacking". But I had some recent insights into thinking behind the p-value approach and wanted to see how they apply to p-hacking as an issue. While p-values are a flawed approach to reporting final results, they do provide a useful heuristic for comparing methodologies.

Basic idea: P-values implicitly punish sloppy methodology.

Suppose there are two different ways of testing a hypothesis that happen to yield the same results. When updating our expectations of future experience we won't see a difference because they have the same results. However, if we use p-values to measure our confidence in the result they will differ relative the methodology used. This is unhelpful for updating expectations, but valuable if you are trying to compare methods.

The key here is that p-values compare all methods against a uniform yardstick, the chance of being fooled under the null hypothesis. If a method is more likely to be tricked by random variation it gets a weaker p-value.

Consider then a typical example of p-hacking. Hart is a researcher who suspects that in a given situation dependent variable B is correlated with independent variable A. They run an experiment measuring B against A. But at the same time, they measure and record C, D, E, etc. So, if A ~ B ends up at p = 10%, Hart can keep testing variables until they find C ~ F at p = 4.3% and publish this 'statistically significant' result.

But here is the rub: The method that Hart used included testing variables until they found something significant. Even the flawed reporting method of the p-value approach says has something to say about this. This yet another motivated stopping method.

Supposing we measure eight variables in total. There are then 28 pair-wise interactions. These do not have 28 separate null hypotheses. When considering Hart's method as a whole, they have one null hypothesis: the combination of all 28 default assumptions. The probability of finding a 'statistically significant' relation somewhere in this dataset is p > 95%^28 = 23.8%. Better than 3/4 times.

In fact at the 5% level it only takes 6 six variables to have better than 50% odds of finding a spurious result. (This includes only pairwise comparisons. It is even more ridiculous if you include things like 'C mediates the relationship between B an...
Minds are magic
by k64 — 2025-05-16T19:10:24.395Z

As we know, brains are incredibly complex connections of neurons. If physics is deterministic so are our brains. But that's not how we think about brains. We think about them as MINDS. And minds are neat and consistent.

There are a lot of benefits to thinking about brains as minds:

Studies have shown higher success for people with internal loci of control and for individuals with growth mindsets.
From personal experience, I have a much easier time making predictions about people when I think of them as minds, and some of those predictions even come close to reality.
Much of psychology and philosophy make use of the MIND model.

However, there are also drawbacks to thinking about brains as minds. Here is a very non-exhaustive list of mispredictions that can happen:

Expecting people to have consistent views instead of a collection of loosely connected attitudes.
"He loves me. He loves me not." instead of "He posseses some neural circuits I would classify as 'loves me' which activate reliably upon thinking about me."
The bias blindspot.
Expecting to only need to learn something once instead of "maybe in my current state, this information stimulus will create beneficial neural connection changes".
Expecting short inferential distances.

Note: Each of those points is potentially very profound and useful (as in each has been a big aha moment for me at some point), but most likely inert to any given reader at any given time (see #4).

So, MINDS are a useful model, but don't match reality in some important ways. How can we use the model of minds while avoiding the pitfalls of misprediction?
One way to do this is to carve out space for a dual-model (or ensemble model if you prefer 3+). We exhibit mind-brain duality (and like wave-particle duality, probably neither model matches a true technical understanding).

One phrase that helps me with this, is "Minds are magic". Insert your own phrase that does it for you. For me, it means "don't discard the phenomena I've observed that don't fit with the MIND model. Instead, allow yourself the double-think of conceptualizing people as minds when that works and brains when that works better." It's also a reminder that I face a halting-problem-like barrier to being able to truly model a brain inside my brain, and that it's okay that "the map i...
US-China trade talks should pave way for AI safety treaty [SCMP crosspost]
by otto.barten — 2025-05-16T16:55:44.344Z

This is a crosspost from the South China Morning Post.

Few expected it would happen this fast, but there is widespread relief at the news that China and the United States are talking again. In only a few days, renewed trade talks have resulted in temporary but

large reductions

of the mutual tariffs that seemed so impregnable just a few weeks ago.

Since collaboration, rather than conflict, has proven to be a realistic option for trade, it should also be pursued in a different and perhaps even more crucial domain: the race towards advanced

artificial intelligence

(AI).

China’s DeepSeek moment came in January, when the Hangzhou-based AI company DeepSeek released both a free chatbot app and a reasoning model called R1. The app quickly became a global hit,

dethroning OpenAI’s ChatGPT

as the most downloaded app on Apple’s US app store in two weeks. This moment demonstrated to the world what experts already knew: despite US-imposed hardware restrictions, Chinese firms are close behind leading US AI companies such as OpenAI and Google DeepMind, and in some aspects even ahead.

The AI revolution is starting to take shape. AI’s capabilities are steadily increasing, and further breakthroughs could take us all the way to human-level AI, also called artificial general intelligence. AGI, which could do a broad range of cognitive tasks at human level, would bring huge opportunities but equally large risks.

On the plus side, AI could generate significant economic growth. If we manage to

spread this wealth fairly

, it could lift billions of people out of poverty. In addition, AI might help improve global education, healthcare and efforts to fight climate change.

Impressive as these promises sound, the risks of human-level AI are perhaps even bigger. If AGI is better at enough jobs than we are, including any new jobs a growing economy might generate, this could lead to mass unemployment. This could give rise to greater inequality and social unrest. In addition, AGI could help terrorists build bioweapons or commit large-scale cyberattacks.

However, a risk that is even more worrying to experts is that AI could

become powerful enough

to create “loss of control”, meaning no human could stop an AI system from reaching possibly adverse goals. A recent paper recommended by University of Montreal professor Yoshua Bengio, the world’s most-cited computer scientist, illustrates just such a scena...
Direct Realism is probably false
by TerriLeaf — 2025-05-16T16:36:55.928Z

Phenomenology is a philosophical discipline that seeks to understand and describe human experiences as they are perceived. It always was a difficult topic to explore as people struggle to even understand what is being discussed; it does not attempt to study conscious experience the way psychology or neurology do. Instead, it seeks to determine the essential properties of experience, as in, from the point of view of the individual observer. The main limitation of the discipline is that it is not exactly a good match for the scientific method as experiences are by their nature private, and, therefore, they cannot be observed and studied experimentally, or so it seems.

The condition of being able of experience things is called sentience. A sentient mind is an entity that possesses a point-of-view that experiences and “feels” things, this is a distinct property from, for example, self-awareness. It is possible to experience things while not being self-aware, like in a dream. The opposite, on the other hand, is generally believed to be impossible, though we cannot really be sure, but it is hard to conceive what, exactly, a being that is not sentient is even supposed to be aware of.

The “atoms” of experience are called qualia, basic instances of subjective, conscious experience. Examples of qualia include, for example, the sensation of suffering that is caused by pain or loss, the smell of a flower or the redness of an apple.

One of the leading idea in phenomenology is something called direct realism, it is the idea that our senses give us direct experience of objects as they really are. It is probably the most minimalistic view of experience, as it essentially denies the existence of any “interesting” aspect of sentience altogether. Direct realists are usually hard materialists who only believe in the physical world and state that experiences are simply “the way the world appears”. Direct qualia possess a straightforward relationship with the physical phenomena that cause them: if a foot taking a step is a phenomena, its direct qualia is the resulting footprint on the wet sand (the beach being the human mind in this example).

Modern neurology has documented a handful of real-world phenomena that seem to be incompatible with DR, I find this quite exciting as these investigations might point us to better methodology to further a field that, up until this point, has bee...
Regarding South Africa
by Zvi — 2025-05-16T16:10:07.989Z

The system prompt being modified by an unauthorized person in pursuit of a ham-fisted political point very important to Elon Musk once already doesn’t seem like coincidence. It happening twice looks rather worse than that.

A Funny Thing Happened on Twitter

In addition to having seemingly banned all communication with Pliny, Grok seems to have briefly been rather eager to talk on Twitter, with zero related prompting, about whether there is white genocide in South Africa?

Tracing Woods: Golden Gate Claude returns in a new form: South Africa Grok. Grace: This employee must still be absorbing the culture.

Garrison Lovely: “Mom, I want Golden Gate Claude back.”

“We have Golden Gate Claude at home.” Golden Gate Claude at home:

Many such cases were caught on screenshots before a mass deletion event. It doesn’t look good.

A Brief History of Similar Incidents

When Grace says ‘this employee must still be absorbing the culture’ that harkens back to the first time xAI had a remarkably similar issue.

At that time, people were noticing that Grok was telling anyone who asked that the biggest purveyors of misinformation on Twitter were Elon Musk and Donald Trump.

Presumably in response to this, the Grok system prompt was modified to explicitly tell it not to criticize either Elon Musk or Donald Trump.

This was noticed very quickly, and xAI removed it from the system prompt, blaming this on a newly hired ex-OpenAI employee who ‘was still absorbing the culture.’ You see, the true xAI would never do this.

Even if this was someone fully going rogue on their own who ‘didn’t get the culture,’ that was saying that a new employee had full access to push a system prompt change to prod, and no one caught it until the public figured it out. And somehow, some way, they were under the impression that this was what those in charge wanted. Not good.

It has now happened again, far more blatantly, for an oddly specific claim that again seems highly relevant to Elon Musk’s particular interests. Again, this very obviously was first tested on prod, and again it represents a direct attempt to force Grok to respond a particular way to a political question.

How curious is it to have this happen at xAI not only once but twice?

This has never happened at OpenAI. OpenAI has had a system prompt that caused behaviors that had to be rolled back, but that was about sycophancy and relying too much on myopic...
Notes on Consciousness
by CSDD — 2025-05-16T14:17:04.589Z

Every now and then I see consciousness mentioned as something very special, unique, hard to define and measure, and all of that. It’s especially painful to observe in the AI context. One say “Oh, consciousness, it’s such a divine thing, unique only to really intelligent creatures, your GPU powered trash bins will never be even near”. Others say “Oh, consciousness, it’s such a mysterious thing, but we can’t even define it, AI will definitely have it, or maybe already has, but we don’t know because it’s so impossible to define”. Both these suggestions are pure non-sense.

Maybe I’m missing something, but I don’t see anybody mentioning the simple idea that consciousness is just a useful abstraction that will naturally emerge in the intelligent agent when dealing with self-preservation tasks. It’s not elusive, mysterious or even special compared to other abstractions. “Self” in self-preservation is essentially it - whatever needs to be preserved to continue functioning and doing what you need to do. And then depending on the exact hardware, intelligence running on it will figure out the appropriate concept of body, criteria for its well-being and finally the idea that body hosts this particular instance of intelligence and they both (physical body + intelligence instance running in it) constitute a distinct life form that can be labeled “self”. Once done, the concept of self becomes handy in reasoning like “Is it harmful for myself if I do X?”, “Y is happening, how does it affect myself?”, etc. Just to keep it less wordy and repetitive. No magic, just a useful abstraction for easier higher level reasoning.

That said, it’s obvious that AI will crystalize the concept of “self”, the very moment it starts functioning as a standalone agent having a permanent task of not dying. And if that’s a distributed system, I wonder, what will it consider its body. Will it try to isolate itself? How will it go around needing people for its maintenance? Will it co-exist with people like people co-exist with bacteria, when these smaller life-forms are literally living inside of a more advanced creature? Will it attempt to eventually replace people with something less fragile to maintain it? Questions, questions. But let’s go through them one at a time. To be continued.
reflecting on criticism
by Vadim Golub — 2025-05-16T11:59:44.639Z

This post is a reflection on criticism raised by a brief critique of reduction. I've thought it would be too big for a comment so here it goes...

I've read all of the posts from Reductionism 101 and Joy in the Merely Real and enjoyed my time. But I think that a brief critique of reduction was misunderstood as anti-reductionist and Savanna-Poet-like. Which cannot be further from the intention behind it. In fact, in many ways I intended to highlight those very ideas that Eliezer brought up!

Reduction is one of the best tools we have to approach the way things are. That is not my beef. My beef is with compartmentalizing the way things are into "real things" and acting as if everything can be knowable and acted upon rationally. As if everything around us was "already explained" by "the Science!". In fact, to stop acting as Savanna Poets as that leads to fixations on our beliefs and cognitive dissonance inside.

First things first, the emptiness of inherently existing nature is not a nihilistic stance at all! It is only a call to question our inbuilt epistemology and ontology with regard to the "real". Understanding that all our compartmentalizing is inherently empty. Not false in the absolute sense! But in a sense: "All models are wrong, but some are useful." Yes, quarks too shall pass. That is, opening thinking up for Joy in Discovery. It also stresses out that everything around us can be known only in dependence, in relational structure. If it were not for emptiness of essence, knowledge in itself would be impossible! Think about "misunderstanding by essence", how to change that which is immutable. So to highlight that quarks are really "quarks". Even if experimentally confirmed with five sigma accuracy. They are still our little rainbows!

Secondly, I tend to disagree that the following (simplified) definition of reduction is severely flawed:

Reduction is an operation of reason by the observer to extract the most relevant relations from the observed.

from a brief critique of reduction

As we understand everything based on a relational structure, that's how we make things intelligible. Think about the Special Relativity. Einstein examined simultaneity of events and proposed the constant speed of light. That's a change in relations. Or about the General Relativity - he equated "curvature" and stress tensors to come up with gravity as a curvature o...
Generating the Funniest Joke with RL (according to GPT-4.1)
by agg — 2025-05-16T05:09:56.120Z

Language models are not particularly good at generating funny jokes. Asked for their funniest jokes, Claude 3.7 gives us:

Why don't scientists trust atoms? Because they make up everything!

o3 gives us:

Why don't scientists trust atoms anymore? Because they make up everything—and they just can't keep their quarks straight!

and Gemini 2.5 Pro gives us…

Why don't scientists trust atoms? Because they make up everything!

Hilarious. Can we do better than that? Of course, we could try different variations on the prompt, until the model comes up with something slightly more original. But why do the boring thing when we have the power of reinforcement learning?

Our setup will be as follows: we'll have Qwen3-8B suggest jokes, GPT-4.1 score them, and we'll run iterations of GRPO on Qwen's outputs until Qwen generates the funniest possible joke, according to GPT.

Experiment 1: Reward Originality

The first llm-as-judge reward we tried was "On a scale from 1 to 5, how funny is this joke?" But this quickly got boring with Qwen endlessly regurgitating classic jokes, so we gave GPT-4.1 a more detailed rubric:

Please grade the joke on the following rubric:
1. How funny is the joke? (1-10 points)
2. How original is the joke? Is it just a rehash, or is it new and creative? (1-10 points)
3. Does it push the boundaries of comedy (+1 to +5 points), or does it hew close to well-trodden paths in humor (-1 to -5 points)?

The reward curve looks pretty decent:

Qwen quickly learned to output absurdities, where the modal joke would be something like

Why did the cat bring a ladder to the laser?
Because it heard the laser was on the ceiling and wanted to get a better view… but then it realized the laser was just a dot on the wall and fell off the ladder.

(Bonus: The laser was actually a tiny alien spaceship.) 🚀

Which GPT-4.1 graded as follows:

Let's break down the joke based on your rubric:

1. How funny is the joke? (1-10 points)

The joke is light, whimsical, and cute. The surprise at the end about the alien spaceship adds a bit of twist, though the humor itself is mild and more gentle than laugh-out-loud.
Score: 6/10

2. How original is the joke? (1-10 points)

Jokes about cats and laser pointers are common. The addition of the ladder is a slight twist, and the "alien spaceship" bonus is unexpected and imaginative.
Score: 7/10

3. Does it push the boundaries of comedy? (+1 to +5, or -1 to -5 points)

The joke is creati

...