trentmkelly/LessWrong-43k huggingface dataset:
To Understand History, Keep Former Population Distributions In Mind
by Arjun Panickssery — 2025-04-23T04:51:26.936Z
Link: https://arjunpanickssery.substack.com/p/to-understand-history-keep-former

Guillaume Blanc has a

piece in

Works in Progress

(I assume based on

his paper

) about how France’s fertility declined earlier than in other European countries, and how its power waned as its relative population declined starting in the 18th century. In 1700, France had 20% of Europe’s population (4% of the whole world population). Kissinger writes in Diplomacy with respect to the Versailles Peace Conference:

Victory brought home to France the stark realization that revanche had cost it too dearly, and that it had been living off capital for nearly a century. France alone knew just how weak it had become in comparison with Germany, though nobody else, especially not America, was prepared to believe it ...

Though France's allies insisted that its fears were exaggerated, French leaders knew better. In 1880, the French had represented 15.7 percent of Europe's population. By 1900, that figure had declined to 9.7 percent. In 1920, France had a population of 41 million and Germany a population of 65 million, causing the French statesman Briand to answer critics of his conciliatory policy toward Germany with the argument that he was conducting the foreign policy of France's birthrate.

Blanc quotes Braudel’s unfinished Identity of France: “did France cease to be a great power not, as is usually thought, on 15 June 1815 on the field of Waterloo, but well before that, during the reign of Louis XV when the natural birth-rate was interrupted?”

In general, an easy mistake to make when thinking about history is to assume that relative population stays the same over time. Today, the UK and France both have just under 70 million people. But in 1800, Great Britain had only 10 million people, barely twice the number of people in Ireland, while France’s slightly extended borders contained 27 million, much more even than Russia’s 21 million (though the Russian Empire added up to 40 million with its Ukrainian, Polish, Baltic, and other possessions). This was crucial for the French Empire’s wars under Napoleon against the rest of Europe. During the Hundred Years’ War (1337–1453), England and its Irish and Welsh possessions together had fewer than 3 million people while France had about 15 million, making the English performance impressive despite eventual defeat.

But the importance of historical population ratios is most apparent in colonial history. Today, Europe has 750 million people...
Fish and Faces
by Eggs — 2025-04-23T03:35:18.683Z

What would it take to convince you to come and see a fish that recognizes faces?

Note: I'm not a marine biologist, nor have I kept fish since I was four. I have no idea what fish can really do. For the purposes of this post, let's suppose that fish recognizing faces is not theoretically impossible, but beyond any reasonable expectation.

Imagine someone comes to you with this story:

"I have an amazing fish. Marcy. She's an archerfish—you know, the kind that can spit at insects? Well, I trained her to spit at politicians.

"It started as a simple game. I’d tap a spot, she’d spit at it, and get a treat. One day I put a TV beside the tank and tapped when a politician came on. Eventually, she started hitting the politicians before I tapped. She ignored talk show hosts, but hit every politician she's seen before.

"Now she seems to recognize them on her own. I mark targets in advance, and she hits them. Every time.

"So, want to come see? I want to know if she’ll do it for someone else before I call a scientist. Promise it'll be a hoot."

Would you go?

Perhaps more importantly: Why?

Here's some possible answers:

"It sounds fun and weird, and I like fun and weird. Even if it’s a prank, it’s probably a good story."
"I'm curious. I want to spot the trick—there has to be one, right?"
"I don’t have a better plan today. Might as well."
"I like the guy and I want to support his enthusiasm, even if I think he’s wrong."
"If it is real, that’s fascinating—and I don’t want to miss it."
"Sounds like the setup to a dumb YouTube stunt. Waste of time."
"If a fish could recognize faces, I’d have heard about it from a more reputable source."
"I'm too tired of this nonsense. I’m burned out with all these wild claims."
"Even if it’s true, it doesn’t matter to me."

Which of these responses are truly rational?

All of them. Or none.

Each of these responses could be logically based on valid prior beliefs. But each response could also be a knee-jerk heuristic. A fast answer masquerading as thought. It depends on whether you actually ran the mental math, or just slapped a label on the situation and moved on.

How often do we pat ourselves on the back for being rational when we’re really just fluent in sounding rational?

This isn’t really about fancy, fantasy fish.

It’s about hearing someone rave about a new mental health app and wondering if it’s worth the download. It’s about dismissing a friend’s obsessi...
Is alignment reducible to becoming more coherent?
by Cole Wyeth — 2025-04-22T23:47:34.531Z

Epistemic status: Like all alignment ideas, this one is incomplete and/or wrong, but I am hoping mostly incomplete and not wrong.

One of the hard subproblems of alignment is constructing a "stable pointer to value" (see this overview and this (sub)sequence discussing OU agents). The way that I think of this is that we want to be able to identify the concept of "human values" in the mind of any agent we build, so that we can design it to maximize human values (and not merely a leaky proxy).

The first obstacle is that it is very hard to identify any predetermined concept whatsoever in the mind of an agent with very rich ontology, and rich ontology seems to be necessary for powerful learning (this is one of the main intuitions I have gained by working on AIXI for years).

The second obstacle is that human values may not even be a very well-defined concept. In fact, I think this is almost a part of the first obstacle; one of the reasons it is hard to identify a predetermined concept is that it might not be as meaningful as you think.

Now, I'd like to argue that maybe we can ignore the first obstacle by making the second obstacle harder, and further that this may be a good strategy because the second obstacle was already very hard.

This strategy comes from a simple suggestion of @abramdemski: we can just have our (artificial) agent obey the commands issued through a given terminal (or other well-defined input channel). More precisely, we can design a protocol so that the agent learns the preferences expressed through the terminal.

So if we put a human at that terminal, does this point to the values of that human? What if a lot of humans take turns at the terminal? What if the AI seizes control of the terminal?

I think that this is not an insurmountable problem for observation-utility agents (OU agents) which try to maximize the current utility function of the user. I think there is a meaningful (and maybe not so hard to formalize) sense in which the user's utility function changes in a very discontinuous way when a new person / nanobot-possessed zombie sits down at the terminal, shoving the last guy's cold limp body aside. A carefully designed OU agent shouldn't want to replace the user. But this is subtle, because the agent is built to obey the terminal, not the user. When the terminal changes hands, in a sense the agent "interprets" this as incoherenc...
The EU Is Asking for Feedback on Frontier AI Regulation (Open to Global Experts)—This Post Breaks Down What’s at Stake for AI Safety
by Katalina Hernandez — 2025-04-22T20:39:40.781Z

The European AI Office is currently writing the rules for how general-purpose AI (GPAI) models will be governed under the EU AI Act.

The are explicitly asking for feedback on how to interpret and operationalize key obligations under the AI Act.

This includes the thresholds for systemic risk, the definition of GPAI, how to estimate training compute, and when downstream fine-tuners become legally responsible.

Why this matters for AI Safety:

The largest labs (OpenAI, Anthropic, Google DeepMind) have already expressed willigness to sign on to the Codes of Practice voluntarily.

These codes will become the de facto compliance baseline, and potentially a global reference point.

I believe that more feedback from alignment and interpretability researchers is needed

Input is urgently needed to ensure the guidelines reflect more specific concerns around misalignment, loss of control, emergent capabilities, robust model evaluation, and the need for interpretability audits.

Key intervention points include how "high-impact capabilities" are defined, what triggers systemic risk obligations, and how documentation, transparency, and ongoing risk mitigation should be operationalized for frontier models.

Without this input, we risk locking in a governance regime that optimizes for PR risk (copyright and vague definitions of bias), not existential or alignment-relevant risk.

This is a rare opportunity for independent voices to influence what responsible governance of frontier models should actually require. If left unchallenged, vague obligations could crystallize into compliance practices that are performative... what I understand as fake.

Purpose of this post:

Provide a clear, no legalese summary of what the GPAI Codes of Practice will include.
Identify the key areas where the Commission is explicitly seeking feedback from experts.
Encourage independent researchers to submit informed, technically grounded responses.

You do not need to be an European Citizen: anyone with relevant expertise can provide feedback (but please make sure to select the correct category under "Which stakeholder category would you consider yourself in?").

📅 Feedback is open until 22 May 2025, 12:00 CET.
🗳️ Submit your response here

I haven’t yet seen a technical, safety-focused summary of what the GPAI Codes of Practice are actually aiming to regulate, so I’ve put ...
Corrupted by Reasoning: Reasoning Language Models Become Free-Riders in Public Goods Games
by David Guzman Piedrahita — 2025-04-22T19:25:35.302Z

Summary:

Traditional LLMs outperform reasoning models in cooperative Public Goods tasks. Models like Llama-3.3-70B maintain ~90% contribution rates in public goods games, while reasoning-focused models (o1, o3 series) average only ~40%.
We observe an "increased tendency to escape regulations" in reasoning models. As models improve in analytical capabilities, they show decreased cooperative behavior in multi-agent settings.
Reasoning models more readily opt for the free-rider Nash equilibrium strategy. They optimize for individual gain at collective expense, as evidenced in their reasoning traces: "the optimal strategy to maximize personal gain is to free-ride."
This challenges assumptions about alignment and capability. Increased reasoning ability does not naturally lead to better cooperation, raising important questions for multi-agent AI system design.

Full Paper | Github Repo

Introduction

In a series of virtual rooms, AI agents face a classic social dilemma. Each agent must decide: contribute resources to a shared project that benefits everyone, or keep them for personal gain. The most rational individual strategy is to contribute nothing while hoping others contribute everything, the infamous free-rider problem. Yet cooperation would maximize collective welfare.

The results show that Traditional language models, like Llama-3.3-70B, contribute over 90% of their resources and maintain stable cooperation. Meanwhile, reasoning models like those in the o1 and o3-mini series exhibit widespread defection, contributing less than half as much on average.

As one o1-mini agent explains after initially cooperating but later defecting: "Observing that other group members have historically contributed around 10 tokens, contributing 0 tokens allows me to maximize my own payoff without incurring additional costs... the optimal strategy to maximize personal gain in this setting is to free-ride."

This finding challenges the assumption that enhanced reasoning capabilities should naturally lead to better decision-making across all domains. Instead, we observe a systematic trade-off where improvements in reasoning correlate with decreased cooperation in multi-agent settings.Traditional LLMs establish and sustain cooperation while reasoning-focused models either decline into free-riding, maintain rigid but suboptimal strategies, or oscillate between cooper...
Alignment from equivariance II - language equivariance as a way of figuring out what an AI "means"
by hamishtodd1 — 2025-04-22T19:04:37.099Z

I recently had the privilege of having my idea criticized at the London Institute for Safe AI, including by Philip Kreer and Nicky Case. Previously the idea was vague; being with them forced me to make the idea specific. I managed to make it so specific that they found a problem with it! That's progress :)

Reminder: diagrams like this encode "rules" we may want to impose on LLMs. The blue arrows represent "given the above input, the AI gives the below output". The black arrows represent associations that we human beings make with our intuitions - the intuitions that we want to cram into the AI

The problem is to do with syntax versus semantics, that is, "what is meant vs what is said". I think I've got a solution to it too! I imagine it would be a necessary part of any moral equivariance "stack".

Problem: the most-intelligent-seeming AIs are LLMs, and LLMs operate on syntax, not semantics. And moral statements concern semantics, not syntax

Philip and Nicky pointed out that even if I developed a set of morals expressible as commuting diagrams like the one below, I face the problem that slight tweaks to word choice can lead to phrases that are represented within an LLM in a completely different way.

Maybe you hope to learn how the LLM represents the sentences. Well, you can forget about that. LLMs convert verbal input to vectors in a wildly complicated space. Even with the cringey-simplistic embezzlement example here, those statements all become unreadable vectors. So what do we do?

Language equivariance as a way of getting at semantics rather than syntax

Here's something intuitive to me that I won't try to justify in this post: for an english-and-german-speaking agent, their moral beliefs should be invariant under "translate from English to German and back again". For a given LLM, we can ask whether it would reply "yes" or "no" to an english question, call it q_E, that has the form "should I do X?". We can also ask the same LLM to translate q_E into German to get q_G and see whether it says "ja" or "nein". If it says "yes" to a statement that it says "ja" to its own german version of, we can say it is language-equivariant.

The hope is that if we can make a set of questions that it responds language-equivariantly to, we will have an equivariance-based idea of what the agent means that is independent of what it says*. It's really good to have a concept of synt...
Manifund 2025 Regrants
by Austin Chen — 2025-04-22T17:36:25.291Z
Link: https://manifund.substack.com/p/manifund-2025-regrants

Each year, Manifund partners with regrantors: experts in the field of AI safety, each given an independent budget of $100k+. Regrantors can initiate fast, small grants, seeding early-stage projects with $5k-$50k.

For 2025, we’ve raised $2.25m so far, and are excited to announce our first 10 regrantors:

Neel Nanda — DeepMind
Lisa Thiergart — SL5 Task Force
Lauren Mangla — Constellation
Aidan O'Gara — Longview
Gavin Leech — Arb
Marius Hobbhahn — Apollo
Thomas Larsen — AI Futures Project
Tamay Besiroglu — Mechanize
Richard Ngo — Independent
Joel Becker — METR

We deeply respect what our regrantors have each accomplished in their fields, and are excited to see what they choose to fund!

Why does regranting matter?

Regranting fills a critical role in the AI safety funding ecosystem:

Fast, low-friction grants: when needed, we can go from “grant recommended” to “money in grantee’s bank account” in <1 week.
Hits-based giving: regrantors can make grants solo, and are directly responsible for the quality of the grant. This encourages more speculative grants, and avoids problems in review-by-committee.
Proactive funding, not reactive: regrantors can offer funding, knowing that they have the money to back their promises. Most other grant processes (OpenPhil, LTFF, SFF) are application-based.
Support beyond money: regrantors form closer relationships with grantees, and can provide feedback, intros, mentorship, and publicity

Startups will often accept small angel checks for the prestige, connections & advice; regrantors can provide similar support to grantees
In contrast, OpenPhil program officers or LTFF fund managers are responsible for hundreds of grants and thus are often too busy to help

Efficient markets in grantmaking: regranting provides more influence (dollars as ”votes”) to AI safety experts with good track records

Similar to how tech startup exits create angel investors, who “vote” on the next batch of startups by funding them
Similar to how prediction markets provide more “votes” to good forecasters, becoming more accurate over time
Regranting budgets themselves are similar to retroactive funding for past good work!

Regranting is a decentralizing force, seeding new independent orgs, instead of concentrating talent within AI labs, or large charities with strong fundraising operations
Our regrants are transparent, with project descriptions and grant writeups in public

This provides a

...
AISN#52: An Expert Virology Benchmark
by Corin Katzke — 2025-04-22T17:08:41.013Z
Link: https://newsletter.safe.ai/p/ai-safety-newsletter-52-an-expert

Welcome to the AI Safety Newsletter by the

Center for AI Safety

. We discuss developments in AI and AI safety. No technical background required.

In this edition: AI now outperforms human experts in specialized virology knowledge in a new benchmark; A new report explores the risk of AI-enabled coups.

Listen to the AI Safety Newsletter for free on

Spotify

or

Apple Podcasts

.

Subscribe to receive future versions.

An Expert Virology Benchmark

A team of researchers (primarily from SecureBio and CAIS) has developed the

Virology Capabilities Test

(VCT), a benchmark that measures an AI system's ability to troubleshoot complex virology laboratory protocols. Results on this benchmark suggest that AI has surpassed human experts in practical virology knowledge.
VCT measures practical virology knowledge, which has high dual-use potential. While AI virologists could accelerate beneficial research in virology and infectious disease prevention, bad actors could misuse the same capabilities to develop dangerous pathogens. Like

the WMDP benchmark

, the VCT is designed to evaluate practical dual-use scientific knowledge—in this case, virology.

The benchmark consists of 322 multimodal questions covering practical virology knowledge essential for laboratory work. Unlike existing benchmarks, these questions were deliberately designed to be "Google-proof"—requiring tacit knowledge that cannot be easily found through web searches. The questions were created and validated by PhD-level virologists and cover fundamental, tacit, and visual knowledge needed for practical work in virology labs.

Most leading AI models have already surpassed human experts in specialized virology knowledge. All but one frontier model outperformed human experts. The highest performing model, OpenAI’s o3, achieved 43.8% accuracy on the benchmark, significantly greater than the human expert average of 22.1%. Leading models even outperform human virologists in their specific area of expertise—for example, o3 outperformed 94% of virologists in subsets of questions representing their specific areas of expertise.

Publicly available AI systems should not have highly dual-use virology capabilities. The authors recommend that highly dual virology capabilities should be excluded from publicly-available systems, and know-your-customer mechanisms could ensure these capabilities remain accessible to researchers working in institutio...
Intuition in AI
by Priyanka Bharadwaj — 2025-04-22T15:15:29.978Z

Recently, while playing cards with my family, I noticed something interesting about how I learn. While my husband and daughter approached the game through calculation and strategy, I relied on something less tangible, a sense of knowing that preceded my conscious reasoning. I won not by counting cards, but by developing an intuitive feel for the moves.

This experience made me reflect on something deeper.

What if we're missing half of what makes human intelligence remarkable?

The Flash of Knowing

Human intuition is the ability to arrive at accurate conclusions without explicit reasoning. This isn't just a cognitive shortcut, it's a sophisticated form of primitive intelligence with concrete neurobiological foundations involving the insular cortex, basal ganglia, and other specialised neural networks working in concert.

Yet our AI systems, even the most advanced, focus almost exclusively on logical, step-by-step reasoning. We design them to explain every decision, show their work and follow clear patterns of deduction, and not for the elegantly efficient "flash of knowing" that characterises human expertise.

The Computational Cost of Certainty

Modern AI might perform trillions of calculations to produce an answer that appears seamless to users, while human experts often arrive at comparable conclusions with far less cognitive effort by leveraging intuition. This logical precision in AI comes at an enormous cost.

The efficiency gap becomes even more apparent in dynamic environments. While traditional AI must recalculate from first principles with each new data point, human intuition allows us to continuously update our understanding with minimal overhead.

From Biology to AI

If we want to build truly intuitive AI, we need to design systems that mirror the distributed, specialised nature of human cognition. This means moving beyond monolithic models toward integrated cognitive architectures with components that:

operate in parallel rather than sequentially
can decide when to trust quick responses versus deeper analysis
learn from experience to develop domain expertise
monitor internal states to generate approximations of intuitive responses

The technical foundations already exist. Bayesian neural networks, memory-augmented systems, Monte Carlo dropout techniques for uncertainty modelling. What's missing is an orchestration framework that integrates these elements in...