trentmkelly/LessWrong-43k huggingface dataset:
"It isn't magic"
by Ben (Berlin) — 2025-06-23T14:00:35.751Z

People keep saying "AI isn't magic, it's just maths" like this is some kind of gotcha.

Turning lead into gold isn't the magic of alchemy, it's just nucleosynthesis.

Taking a living human's heart out without killing them, and replacing it with one you got out a corpse, that isn't the magic of necromancy, neither is it a prayer or ritual to Sekhmet, it's just transplant surgery.

Casually chatting with someone while they're 8,000 kilometres is not done with magic crystal balls, it's just telephony.

Analysing the atmosphere of a planet 869 light-years away (about 8 quadrillion km) is not supernatural remote viewing, it's just spectral analysis through a telescope… a telescope that remains about 540 km above the ground, even without any support from anything underneath, which also isn't magic, it's just "orbit".

Making diamonds and rubies by the tonne isn't a wish made to a magic djinn, it's just chemistry.

Flying through the air, or to the moon, isn't a magic carpet, it's just aerodynamics and rocket science respectively.

Reading someone’s thoughts isn't magic telepathy, it's just fMRI decoding.

Knowing your location anywhere on the surface of planet, at any time of day or night, in any weather, isn't magic, it's just GPS.

Forecasting the weather days in advance isn't the magic of divination, it isn't precognition, it isn't limited to the visions of prophets and oracles, it's just fluid dynamics simulations and Monte Carlo methods.

A bracelet that can detect when you fall and summon help automatically isn't a guardian angel, it's just an Apple Watch with Fall Detection.

Sensing through rock to find hidden water isn't dowsing or water divining, it's just geotechnical survey tools such as ground penetrating radar and electrical resistance surveys.

Seeing someone's bones without flaying the flesh from them isn't magic, it's just an x-ray.

Curing congenital deafness, letting the blind see, letting the lame walk, none of that is magic or miracle, they're just cochlear implants, cataract removal/retinal implants, and surgery or prosthetic exoskeletons respectively.

Condensing sunlight in the daytime, in order to heat and illuminate your home after dark, that isn't magic, it's just photovoltaics and batteries.

A single weapon that can, in the blink of an eye, burn an area large enough to encompass both ancient Athens and ancient Sparta at the same time, that's not magic, it's just th...
Forecasting AI Forecasting
by Alvin Ånestrand — 2025-06-23T13:39:41.097Z

TL;DR: This post examines how rapidly AI forecasting ability is improving by analyzing results from

ForecastBench

, Metaculus

AI tournaments

, and various prediction platforms. While AI forecasting is steadily improving, there remains significant uncertainty around the timeline for matching top human forecasters—it might be very soon or take several years.

ForecastBench

ForecastBench

is a “dynamic, continuously-updated benchmark designed to measure the accuracy of ML systems on a constantly evolving set of forecasting questions.” This makes it ideal for tracking how fast AI forecasting skills improve. Performance is measured as

Brier scores

(lower is better)[1]. See the accompanying benchmark paper

here

.

The

leaderboard

shows AI performance alongside human benchmarks: "Public median forecast" represents non-experts, while "Superforecaster median forecast" represents elite human forecasters.

Evaluated AI systems operate under varying conditions—with or without access to a scratchpad for reasoning or to relevant information like news. Importantly, current forecasting prompts are likely good but suboptimal, meaning AI capabilities may be underestimated.

The benchmark paper reveals that AI forecasting performance correlates linearly with both Chatbot Arena scores and log training compute (measured in

FLOP

). Training compute may be of special interest for predicting when AIs reach the level of expert human forecasters.

Performance vs. Chatbot Arena and Log Training Compute

Brier Score vs. Chatbot Arena and Log training compute (source:

ForecastBench paper

)

Brier Score vs. Chatbot Arena and Log training compute (source:

ForecastBench paper

)

Scores in the graph above corresponds to accuracy for each model with optimal conditions (e.g. if Claude-3-5-Sonnet performed better with a scratchpad, its score is the one achieved while using it).

Using Epoch AI's

training compute data

, the AIs with most training compute at each point in time show a 4.9x annual increase in training compute. The current leader is Grok 3, which used an estimated 4.6×10^26 FLOPs.

In the ForecastBench

paper

, the authors extrapolate the trend line for prediction accuracy and training compute:

Projecting out the log-linear relationship, we find that LLMs could match superforecaster performance when training compute approaches 6.49 × 10^26, though there is a large confidence interval (bootstrapped 95% CI: 9

...
Recent progress on the science of evaluations
by PabloAMC — 2025-06-23T09:41:44.185Z

Summary: This post presents new methodological innovations presented in the paper General Scales Unlock AI Evaluation with Explanatory and Predictive Power. The paper introduces a set of general (universal) cognitive abilities that allow us to predict and explain AI system behaviour out of distribution. I outline what I believe are the main advantages of this methodology, indicate future extensions, and discuss implications for AI safety. Particularly exciting points include the future extension to propensities and how it could be useful for alignment and control evaluations.

Disclaimer: I am a (secondary) author of this paper and thus my opinions are likely somewhat biased; though I sincerely believe this is a good evaluation methodology, with the potential to increase the safety of AI models.

I would like to thank Lexin Zhou and Wout Schellaert for comments that have significantly improved the draft. Any errors, inaccuracies and opinions remain my own.

Summary of ADELE

One persistent question in AI capabilities research is why performance doesn't always correlate with intuitive task difficulty—systems that handle advanced mathematics may struggle with tasks that seem cognitively simpler, like effective gameplay in long-context environments (e.g. Pokemon). In this post, we review a recent preprint, "General Scales Unlock AI Evaluation with Explanatory and Predictive Power", that aims to clarify intuitions about why this is the case. This paper introduces a new evaluation methodology (Figure 1) that makes AI system evaluations both predictive and explanatory. I will explain the main contributions and some of its key advantages and limitations. I will also emphasise how I believe it could be used for AI safety, and what future work the group is working on to make this methodology useful for safety.

The core methodological innovation is to introduce a small number of (conceptually distinct) dimensions that capture the demands of any task instance (see Table 1 below), and match them with the capabilities of the AI system. These dimensions are based on the literature in AI, psychology and animal cognition; and are organised hierarchically to be a set of general or universal cognitive capabilities. Since these dimensions are defined with rubrics in natural language that are human-interpretable, the success or failure of AI systems (validity) will be explainable. ...
Racial Dating Preferences and Sexual Racism
by koreindian — 2025-06-23T03:57:02.159Z
Link: https://vishalblog.substack.com/p/racial-dating-preferences-and-sexual

Note: This is a linkpost from my personal substack. This is on a culture war topic, which is not normally the focus of my blogging. Rationalist friends suggested that this post might be interesting and surprising to LW readers.

Summary

People widely exclude romantic and sexual partners on the basis of race. This is claimed not to be racism by those who have these attitudes, but it is[1].
Certain groups, such as Black women and Asian men, are highly excluded by all other groups. These cases also exhibit within-race asymmetries in racial preferences across gender:

Black women exhibit the highest endophilia (i.e. they most desire to date within race), but this is strongly not reciprocated by Black men.
Asian women exhibit the highest endophobia (i.e. they most refuse to date within race), but this is strongly not reciprocated by Asian men.

These within-race asymmetries are so strong that they provide counter-evidence to most conventional explanations of racial preference (e.g. that racial preference is a proxy for the desire for wealth, class, and integration; that racial preference forms as a by-product of racial segregation).
These effects are not unique to straight dating, but strongly form the romantic and sexual reality of queer dating as well.
The evidential base for the above points is not small, but rather comes from an immense literature in social psychology, sociology, race studies, and queer studies.
People rarely take ownership over their own dating preferences. They either blame wider sociological factors outside of their control, or blame the racial losers themselves (Black women, Asian men) as being responsible for their own undesirability.
This topic is so controversial that it is systematically discussed only within two poles: a “woke” academic sphere, and a “red-pilled” incel/volcel/femcel online sphere. This adds a lot of distortion to the subject.
Those who have racial preferences are not going to be shamed out of having such preferences, so I recommend that racial losers consider doing weird things to improve their sexual fitness.

A Note on Language and Scope

I am going to talk about racial and sexual groups using broad terms like “Asian woman”, “Black gay men”, “Whites/Jews”, and so on. I am aware that this language can be reductive, and may not operate on some desired level of granularity. I am aware that there are important, relevant differences betw...
Mainstream Grantmaking Expertise (Post 7 of 7 on AI Governance)
by Mass_Driver — 2025-06-23T01:39:43.466Z

Previously in this sequence, I estimated that we have 3 researchers for every advocate working on US AI governance, and I argued that this ratio is backwards – we need the political power provided by advocates to have a chance of preventing misaligned superintelligence. A few researchers might be useful as a ‘multiplier effect’ on the power of many advocates, but the converse is not true: there’s no “magic bullet” that AI governance researchers can hope to discover that could substitute for an army of political foot soldiers. Even the best political ideas still need many political activists to spread them, because the political arena is noisy and contested.

Unfortunately, we have very few political activists. This means the good ideas that our governance researchers have been proposing are mostly “orphaned.” In other words, these good ideas have no one to draft them into shovel-ready language that legislators could approve, and they have no one to advocate for their passage.

It’s a bit of a puzzle that the AI safety movement nevertheless continues to fund so much academic-style research. Why do we need more good ideas if we already have a dozen good ideas that nobody is acting on?

In the sixth post in this sequence, I suggested that part of the answer to this puzzle is that we have 4 academic-style thinkers for every political advocate on our AI safety grant-making teams. Because the culture within these grantmaking teams is heavily tilted toward research, grantmakers may feel more comfortable funding research grants or may be better able to appreciate the merits of research grants. They may also have a warm collegial relationship with some of the researchers, which could subtly bias their funding decisions.

Ideally, these kinds of biases would be caught and corrected by formal grantmaking procedures, such as the use of written scoring rubrics. Unfortunately, such procedures are notably missing from major AI safety funders. A week before publishing this post, I gave an advance copy of it to the five largest institutional donors so that they could address any factual errors. Although several people offered helpful corrections, none of these corrections change my fundamental point: AI safety funders are evaluating grants using an informal and subjective method. No formal criteria are being applied to most AI governance grant applications.

In this seventh and ...
The AI's Toolbox: From Soggy Toast to Optimal Solutions
by Thehumanproject.ai — 2025-06-22T20:54:41.248Z

In a limited array of tools, there is one optimal tool to use.

For this example, let’s take the task of hammering a nail into a wall, and the only two tools available to us are a soggy piece of toast and a silver spoon. For the given task, the spoon will be slightly more useful than the moist piece of a baked good.

Let’s expand this to include a wider range of tools that are at our disposal. We now have a whole toolbox akin to one you’d find in the average US homeowner's garage. To fulfil the task of hammering a nail into a wall, you’d most likely grab a hammer. It will be more useful for this very task than a spare tire, another nail or a ‘78 edition of a defunct comic book.

Now let’s take this even a step further. Now you have an unlimited array of tools at your disposal. Even within this unlimited tool kit, there will be tools that will be better or worse for completing the task of hammering the nail into a wall.

You’d likely have multiple different hammers. Some might be made from a slightly stiffer material, while others will be made from a slightly softer material, making it marginally easier or harder to hit that nail into the wall. The handle might be more or less ergonomic or better suited to your hand anatomy. There will still be one optimal tool that would enable you to hammer the nail into the wall with little effort, without bending the nail or hitting it at a bad angle, etc. But realistically speaking, you’d likely not notice the difference between the most optimal tool and the second best or third best ( or probably even 12th for that matter).

You’ll have some sort of threshold (or let’s say grey area) of what the minimum requirement would be of a tool that sufficiently aids you in hammering that nail into the wall.

If we were to plot available tools on the x axis and their effectiveness on the y axis, where perfect utility is 0, we’d be able to plot something akin to a loss function with local and global minima. Our threshold would then just be a horizontal line that would separate the tools that are sufficient for our use ( hammers) from those that are less so ( soggy bread).

(Note: this plot is AI-generated.) We could even turn this into a multidimensional plot if we include utility on one axis and all possible properties (size, shape, hardness, etc.) on the remaining axis.

Quite obv...
Black-box interpretability methodology blueprint: Probing runaway optimisation in LLMs
by Roland Pihlakas — 2025-06-22T18:16:19.078Z

Working document – feedback extremely welcome. Ideas in the main body are those I currently see as highest-leverage; numerous items under Appendix are more tentative and would benefit from critique as well as additions.

I am hoping that this post will serve also as a generally interesting brainstorming collection and discussion ground of black-box LLM interpretability methodology as well as failure mitigation ideas.

I would like to thank my collaborators: Sruthi Kuriakose, Aintelope members (Andre Addis, Angie Normandale, Gunnar Zarncke, Joel Pyykkö, Rasmus Herlo), Kabir Kumar @ AI-Plans and Stijn Servaes @ AE Studio for stimulating discussions and shared links leading to these ideas. All possible mistakes are mine.

TL;DR – A Shortened Version

This chapter is very dense and keyword heavy. A reader-friendly Long Version (the main part of this post) can be found in the next chapter. You can skip this shortened version unless you are in hurry.

Full background in LessWrong (our earlier results):
"Notable runaway-optimiser-like LLM failure modes on Biologically and Economically aligned AI safety benchmarks for LLMs with simplified observation format"
Goals (why)
Identifying when, why, and how LLMs collapse from multi-objective and/or bounded reasoning into single-objective, unbounded maximisation; showing practical mitigations; performing the experiments rigorously; paper in peer review.
Objectives (what)
• 1 Curating stress-test scenarios on Biologically & Economically Aligned benchmarks that systematically trigger over-optimisation.
• 2 Quantifying with automated Runaway Index (RI): scoring by frequency and onset timing of goal imbalance, homeostatic constraint violations, and self-imitation spikes.
• 3 Comparing mitigations:
3.1. Prompt engineering;
3.2. Fine-tuning or few-shot prompting with optimal actions;
3.3. Reflection-based intervention: detecting runaway flip and resetting context or returning to “savepoint”.
Methods (how)
• Stress-tests: Introducing subjective pressure, boredom, sense of failure.
• Narrative variations: Persona shifts (MBTI/Big-Five extremes), mild psychiatric disorders, stoic and zen-like perspective vs achievement orientation and shame.
• Attribution sweep: SHAP-style and “leave one out” message history perturbations to isolate features pred...
The Croissant Principle: A Theory of AI Generalization
by Jeffrey Liang — 2025-06-22T17:58:06.721Z

I recently wrote an ML theory paper which proposes explanations for mysterious phenomena in contemporary machine learning like data scaling laws and double descent. Here's the link to the paper and the Twitter thread. I didn't get much attention and need an endorser to publish on ArXiv so I thought I'd post it here and get some feedback (and maybe an endorser!)

Essentially what the paper does is propose that all data in a statistical learning problem arises from a latent space via a generative map. From this we derive an upper bound on the true loss as depending on the training/empirical loss, the distance in latent space to the closest training sample where the model attains better than the training loss, and the compressibility of the model (similar to Kolmogorov complexity).

Barring a (reasonable) conjecture which nonetheless is not proved, we are able to explain why data scaling follows a power law as well as the exact form of the exponent. The intuition comes from Hausdorff dimension which measures the dimension of a metric space.

Imagine you are building a model with 1-dimensional inputs, let's say in the unit interval [0,1]. Let's say you have ten training samples distributed evenly. If the loss of your model is Lipschitz (doesn't change unboundedly fast e.g. for smooth enough functions, derivative is bounded), your model can't get loss on any test sample greater than the loss at the closest point plus the distance to that point (capped at around 1/10) times the Lipschitz constant (bound on the derivative).

If you want to improve generalization, you can sample more data. If these are spaced optimally (evenly), the maximum distance to a training sample decreases like n−1 as can be easily seen. However, if you were working with 2 dimensional data, it would scale like n−1/2! Hausdorff dimension essentially defines the dimension of a metric space as the number d such that this scales like n−1/d.

If you now put these two facts together, you get that the generalization gap (gap between true loss and training loss) is O(n−1/d) where n is the number of training data samples and d is the Hausdorff dimension of the latent space. In other words, we have a concrete explanation of data scaling laws!

It's worth noting that this analysis is independent of architecture, task, loss function (mostly) and doesn'...
Matchmaking to Machine Alignment: Why Relational Design Can’t Be Left to Chance
by Priyanka Bharadwaj — 2025-06-22T15:32:35.771Z

We say alignment is about control, safety, precision. But after a decade working as a matchmaker in India’s increasingly chaotic relationship market, I’ve learnt that what sustains a system isn’t control, it’s trust. And trust doesn’t live in rules. It lives in memory, repair, and mutual adaptation.

I’ve spent years watching relationships fall apart not because people weren’t compatible, but because they didn’t know how to collaborate. We are fluent in chemistry, but clumsy with clarity. We optimised for trait, not values or processes. And when conflicts hit, as it always does, we have no shared playbook to return to.

In traditional Indian matchmaking, we had a whole socio-structural scaffolding propping up long-term collaboration through race or caste endogamy, community expectations, family intermediation, shared rituals and rites. It was crude and often unjust, but it was structurally coherent. Marriage was not just a bond between two people, but between two lineages, empires and philosophies of life. There were rules, expectations and fallback norms. Vows weren’t just ceremonial; they were memory devices, reminding people what they were committing to when emotions faded.

Today, most of that scaffolding is gone.

Tinder has replaced the community priest or matchmaker, and in this frictionless new marketplace, we are left to figure out long-term cooperation with short-term instincts. Even when we genuinely care for each other, we often collapse under the weight of ambiguity. We never clarify what we mean by commitment. We never learnt how to repair after rupture, and we assume love would make things obvious.

But love doesn’t make things obvious, context does, and maybe design too.

This isn’t just about marriage, it’s about systems and it’s about alignment.

Much of the current conversation on AI alignment focuses on architecture, oversight, corrigibility and formal guarantees. All of that is necessary, and I am not refuting it one bit. But I don't see AI in isolation, because we humans are building it, for us, and so, I can't help but view it from a lens of collaboration or partnership.

In human systems, I’ve rarely seen misalignment fixed by control. I’ve seen it fixed by context, memory, feedback, and repair. Not all of which can be coded cleanly into an objective function.

I’ve watched couples disintegrate not because of what happened, but because it k...