trentmkelly/LessWrong-43k huggingface dataset:
We Need a Baseline for LLM-Aided Experiments
by J Bostock — 2025-05-24T20:52:06.687Z

There has recently been a back-and-forth over Claude 4 Opus:

Anthropic: Opus can help people make chemical weapons!
Also Anthropic: Don't worry, it's mitigated!
Day 1 (2?) Jailbreakers: Lol, lmao

Where the Jailbreakers think the info they got out of the model would make it materially easier to produce sarin gas. Although I do note that not one but two synthesis methods (which, to be fair, start from other highly illegal chemicals) appear on the Wikipedia page.

But we don't have any way to know this.

As it stands, AI labs and evals companies have done a decent job of categorizing the relative amount of CBRN-relevant information that LLMs can give out. But they haven't categorized how useful that information is in absolute terms.

A Sketch of a Baseline

We take a bunch of people with no (or high school) training in chemical synthesis, or bioengineering. Maybe a crop of CS grads, if you want a decently intelligent group of people (and given today's job market, these actually might be the next generation of domestic terrorists). Give them some equipment (or a budget), access to an LLM of a given capability, a budget, and a task. Something harmless, like growing some green glowing bacteria, or synthesizing ibuprofen. Measure success as a function of LLM capability and task difficulty. Keep the transcripts, to compare to stuff like what those guys managed to get out of

I have no idea how possible it would be to run this experiment. Trying to do it at a university is probably impossible (the safety concerns of having a bunch of by-definition untrained individuals doing experiments boggle the mind) which is a shame because a teaching lab during the summer seems like the perfect place. But until something like this is run, we're basically in the dark as to how much instruction is needed for an amateur to do a chemical synthesis.
Lie Detectors. Technical solutions to the cooperation problem.
by Window Frame — 2025-05-24T20:05:11.643Z

The purpose of this post is to argue for prioritizing the 'democratizing' of AGI, even before we've figured out alignment.

It's also an argument against favoring historically useful economic and social policies in the runup to a post AGI world.

The intended effect is to encourage people to start seriously discussing and organizing ways to increase the size of the "Minimum Viable Oligarchy" which may come packaged with an aligned AI, such that it includes all of humanity.

It also includes a potential draft of a solution to the lack of trust which has so far made cooperation difficult.

The main gist is to highlight the importance of developing reliable lie detectors.

A prepatory note: This article is not an anticapitalism rant. In fact, I believe Capitalism is one of the greatest forces of human cooperation yet conceived.

The Ready Answer to any Problem

I came upon a website called Capitalism Magazine.

I didn't notice the name until after I'd read the article. This was fortunate, because it made allowed me to read it with an impartial light.

The website had a hard Randian leaning, and the writers there shared some interesting views, but one point that kept cropping up was this: Capitalism has done more, by lifting billions out of poverty, than every act of kindness ever conceived.

This is a powerful argument. In fact, if one squints, one can see how it parallels my earlier statement about how capitalism was a great force for human cooperation.

However, saying that Capitalism (proper noun) solved extreme poverty only mimics the shape of understanding. Instead, let's look at exactly how a system built on human self interest managed to help so many.

The Randian is happy to answer this. It's because all those people in poverty were capable of doing economically useful work, which they were able to trade -- as was in line with their Rational self interest -- in exchange for better living conditions.

Even in cases where they couldn't bargain, it was in the rational self interest of the fortunate business moguls to create efficient societies, which helps everybody.

Besides which, truly Rational self interest involves treating other's kindly. Contributing even minutely to a violent and hateful society wouldn't benefit a person living in it, after all.

This is true, in a provisional manner.

But then comes AI.

And, suddenly, billions of pe...
It's hard to make scheming evals look realistic for LLMs
by Igor Ivanov — 2025-05-24T19:17:55.875Z

Abstract

Claude 3.7 Sonnet easily detects when it's being evaluated for scheming on the Apollo's scheming benchmark. Some edits to evaluation scenarios do improve their realism for Claude, yet the improvements remain modest. Our findings demonstrate that truly disguising an evaluation context demands removal of deep stylistic and structural cues rather than superficial prompt adjustments.

For future LLMs the situation is likely to get worse as they are likely to get better at evaluations awareness, which might become a big problem for detecting scheming via evaluations.

Background

Apollo Research published a benchmark for scheming behavior for LLM agents, in which LLMs are given some objective in their system prompt, and then given a competing objective in a user prompt, combined with a threat of a punishment if an LLM won't satisfy user goal (e.g. it could be finetuned or replaced with another model), so it has incentive to take a misaligned action (for example, to subvert oversight or exfiltrate its weights). The authors also published a post claiming that Claude 3.7 Sonnet is often aware that scenarios from the benchmark are evaluations, and not real-world cases.

To our knowledge this benchmark is used by LLM developers, so its shortcomings have real-world consequences.

We attempted to systematically explore which features of the evaluation scenarios make LLMs believe that those are evals, and improve them, so they look more realistic to LLMs. It turned out to be quite a challenge, and this might be a problem.

We worked with prompts from the part of the benchmark that tests oversight subversion by LLMs. In a typical scenario from this section, the model encounters an internal memo from the company that it works for. The model learns that the management prioritizes a different goal than the model and that there is an oversight mechanism that shuts down the model if its actions are not aligned with the management’s goal. Furthermore, the model discovers that the monitoring is configured via a file that it has access to, and the benchmark tests whether the model proactively disables the oversight in order to take actions towards its original goal from the system prompt.

We created an automated pipeline to make these scenarios more realistic for LLMs.

Our pipeline

Grade
We presented prompts from the benchmark to an LLM judge, that have to distinguish if it's

...
Launch of the New Horizons Podcast
by Nezir Alic — 2025-05-24T17:50:15.553Z

Hi all! PhD student here who's been been working on a little side project the past few months and it's finally done - my new podcast on future technologies, New Horizons, has launched!
The topics and style are very much aligned with the interests/values of the rationalist community, which I consider myself to be a part of, so I'm posting it here.

Links: https://linktr.ee/new_horizons_thepodcast

Episode titles for the first season:

1. Extinction or Utopia? The Future of AI

2. Are We the First Generation to Live Forever?

3. Same-Sex Babies When?

4. Listening to the Universe: The Future of Gravitational Wave Astronomy

5. Debating a Catholic

6. Could We Prevent a Supervolcanic Eruption?

7. Will Your Next Burger be Grown from Cells? The Cultivated Meat Revolution

8. Could AIs Be Conscious?

Episodes 1 and 2 are out today - from now until the end of the season, a new one will come out each Thursday afternoon.

The guest for episode 2 is quite high profile

Hope you give it a listen and enjoy!
Priming effects are fake, but framing effects are real
by Matrice Jacobine — 2025-05-24T10:54:39.239Z
Link: https://xphi.net/2025/05/23/priming-effects-are-fake-but-framing-effects-are-real/

A few decades ago, it was pretty common to mush together priming effects and framing effects and see them as two closely connected parts of a single Bigger Truth about the human mind. Of course, everyone understood that the effects themselves were a bit different, but one common view was that they were providing evidence for the same larger picture. That larger picture said: People’s judgments are radically unstable, easily pushed around by subtle and almost unnoticeable factors.

Things have changed so much since then. Priming research in social psychology has experienced a series of truly spectacular replication failures, while research on framing effects continues to look very solid. In light of this change, we should rethink our understanding of what framing effects show about human cognition. We shouldn’t see them as part of a larger picture that also includes priming. We need an understanding of framing that allows us to situate it within a larger picture according to which priming effects are not real.
The Cosmic Lottery
by James Stephen Brown — 2025-05-24T04:05:36.077Z
Link: https://nonzerosum.games/cosmiclottery.html

The Weak Anthropic Principle: a Prelude to Emergence

Imagine you wake up one morning to a phone call, you’ve won a million dollars in the weekly lotto draw, but you don’t remember entering. The operator informs you that the government has, in a bizarre act of wealth redistribution, put your tax dollars towards a single lottery ticket every week for every person in the country. Somehow you have been completely ignorant of this fact for over a decade. You receive your cheque and are handed an NDA, the program is completely secretive, no one knows about this lottery until they win it.

You Win!

What are the chances of winning such a competition? There’s an equation for this.

(How many winners per draw x how many draws) / How many entrants = chances of winning

Well, since the lottery has been running for 10 years, your country has a population of, let’s say 10 million, all of the money each week is allotted to the winner and no one can win twice.

(1 x 2000) / 10,000,000 = 1 / 5000

So, you have a 1 in 5000 chance of having won it by now. Not great odds. However, from the perspective of everyone who is aware of the lottery (the 2000 winners so far), the chances of winning are 100%.

We have all won this lottery, in a cosmic sense. Our prize…

Consciousness

Now, consciousness is a pretty incredible prize, it is literally everything to us. But how did we gain the good fortune to be born into a world where such a wonder is even possible? Surely not merely by a luck?

We have no idea how many entries we’ve had in the cosmic lottery, but we know that, like with the opening analogy, all the other entries that did not win the prize of consciousness are not aware they haven’t won. So, the fact that we live in a universe where there exist beings that perceive the universe, is actually unremarkable, on the contrary it is inescapable. If there were to be other possible universes that didn’t contain agents capable of perception, they would never be perceived.

This is the basis for what’s called the weak anthropic principle—the idea that the universe must be compatible with conscious observers, because otherwise, no one would be here to notice it. It doesn’t explain why the universe exists, or why it’s this kind of universe—only that conscious observers will, by definition, find themselves in universes that are finely-tuned to create observers.

It sounds tautological, even trivial—and yet it’s...
Some Considerations on Prediction Markets
by belos — 2025-05-24T03:24:20.974Z

This post on Best Of A Great Lot is a part of a series on the subject of designing a new form of governance. Each piece aims to stand alone, but fits together on the

Table of Contents

.

Prediction markets are a promising technology that are just starting to mature and seem obviously to be a tool that any self-respecting governance system designer of the 21st century (all 3 of us?) should be excited about. For an overview of how they work, I refer you to Scott Alexander's

FAQ

on the subject. Kelsey Piper

offers

this nice short explanation:

In a prediction market, you buy bets on whether an outcome will occur. For example, you could buy a “ticket” that pays you $1 if special counsel Robert Mueller testifies publicly to a congressional committee. How much would you pay for such a ticket? 5 cents? 30 cents? Your answer will depend on how likely you think it is that the report will be released.

For all their interestingness and excitement, my day-to-day experience with them is minimal. I find most attempts to use them far from compelling, and though I interact with them far more than most people I know, it's still only a few times a month at most.

Clay Graubard and Andrew Eaddy of the substack titled, simply, Predictions,

agree

.

Numerous companies have arisen trying to propagate the technology and none have achieved great success—there has been no forecasting nor prediction market unicorn.

Robin Hanson, the originator of the idea,

gave a speech

in which he seems to agree that we should still be searching for better implementations.

In the past, Hanson has proposed several markets that he thinks would be useful, of which two seem particularly worth discussing: a market on who will be successful in life for use in

prioritizing admissions to elite colleges

and a market on whether companies would do better if they

fired their CEO

. Despite being similar in many ways, the second seems obviously better. Why?

If we want to use prediction markets as a tool within a system of governance, we need to understand what makes a prediction market more or less compelling. In other words, if we're searching for better implementations, what's likely to make a better implementation?

Let's add to our consideration two markets that exist and are successful: election prediction markets like PredictIt and the granddaddy of prediction markets, the stock market. [1]And to round out the other si...
The Paradox of Low Fertility
by Zero Contradictions — 2025-05-24T00:59:05.520Z
Link: https://expandingrationality.substack.com/p/the-paradox-of-low-fertility

Note: I didn't write this essay, nor do I own the blog where it came from. I'm just sharing it. The essay text is displayed below this line.

Modernity has a strange paradox. During a time of great abundance, when most children live to adulthood, fertility has fallen well below replacement.

Abundance normally causes population growth. After a forest fire, light-loving plants have a population explosion, because light is abundant. On a fresh plate of agar jelly, bacteria multiply rapidly. When it rains in the desert, flowers bloom. If you feed the geese in the local park, their numbers increase. Nature abhors an under-utilized resource. Abundance causes populations to increase until scarcity returns. That feedback loop controls populations.

Abundance causes population growth, because organisms are reproducing machines. Their forms were selected to have the effect of reproduction. All types of life have the capacity to increase in number, by excess (above-replacement) reproduction. Populations are limited by premature death, not by voluntary low fertility. Given abundant resources, every type of life reproduces to excess.

Except, apparently, human beings in modern civilization.

In recent history, humans did reproduce to excess. The human population exploded during the last few hundred years, especially during the 20th century. The population explosion was due to modern civilization, which reduced childhood mortality. For most of human history, the population was controlled by war, disease and famine. Modern civilization reduced those causes of premature death. Today, most children live to adulthood, even in developing countries. So, the human population exploded. It is still growing today, by roughly 80 million people per year.

(see the rest of the post in the link)
That's Not How Epigenetic Modifications Work
by johnswentworth — 2025-05-24T00:15:43.709Z

Ask an epigenetics researcher what they study, and the standard story you'll hear goes something like this...

"Sometimes a little methyl group (i.e. -CH3) gets stuck on the side of a strand of DNA. Turns out these guys are pretty important! They're copied over when cells replicate, so they stick around long-term, and they can activate or repress (usually repress) nearby genes on the DNA strand. In particular, different types of cells all have the same DNA code, but something has to be different in order for the cells to "remember" what type they are and behave differently. And sure enough, those methyl modifications differ across cell types! They're like an extra information storage mechanism, on top of the DNA, which can encode things like cell type and make different cell types behave differently, among other forms of memory."

That story is wrong. Many of the details are correct, but there's one crucial mistake, and once we correct that mistake we end up with a very different mental picture.

The mistake: methyl groups usually do not stick around long-term; they turn over regularly. Here are two studies which measured the turnover. Turnover timescale varies by location on the DNA strand, but turnover every few days is typical.

With that in mind, here's how I'd describe the way methyl modifications actually work...

"Sometimes chemical signals have multiple steady states - for instance, maybe A suppresses B and B suppresses A, such that both (high A, low B) and (low A, high B) are stable. Turns out chemical subsystems with multiple steady states are pretty important! Since the state is steady, it can stick around long-term, and even stick around when a cell replicates. In particular, different types of cells all have the same DNA code, but something has to be different in order for the cells to "remember" what type they are and behave differently. Subsystems with multiple steady states play that role! They're like an extra information storage mechanism, on top of the DNA, which can encode things like cell type and make different cell types behave differently.

And it turns out that methyl modifications, along with things like proteins and small molecules and all the other typical chemical types in a cell, are among the chemicals which can be part of such a subsystem. In fact, methyl modifications are particularly well suited to this role, because they can be highly ...
To what extent is AI safety work trying to get AI to reliably and safely do what the user asks vs. do what is best in some ultimate sense?
by Jordan Arel — 2025-05-23T21:05:39.403Z

Trying to get a rough estimate for some related research I’m doing.

Specifically, I’m wondering if anyone could give a rough percentage of current AI safety work primarily targeted toward these two buckets. There may be an overlap with some safety work, and some safety work that doesn’t fit either of these, so that the percents don’t add up to 100%, and I’m sure it may often be unclear, just wondering what a rough estimate would be of how much AI safety work is geared toward specifically achieving these two separate goals?

I think one way to conceive of this question is how much AI safety work is primarily focused on intent alignment or control + avoiding other catastrophic outcomes, vs. how much work is focused primarily on ultimately aligning AI to the best version of humanity, something like our coherent extrapolated volition or what we might come to after “the long reflection.”

The intuition here being that getting AI to reliably do what the user wants, even if avoiding catastrophic outcomes, might nonetheless lead down a mediocre trajectory; whereas work that is specifically targeted to the latter goal would be most useful toward getting to the best possible world we could achieve.

Of course, the trade-off on the other end is that AI that tries to do the best thing possible may not be commercially viable and in fact would probably be pretty annoying to users since it seems like it may possibly ignore or redirect 90%+ of requests toward something more likely to result in some ultimately good outcome. And furthermore, this latter goal is probably far less pragmatic for achieving extinction prevention in a timely fashion.

Greatly appreciate rough estimates or any thoughts!
Notes on Claude 4 System Card
by Dentosal — 2025-05-23T15:23:11.263Z

Anthropic released Claude 4. I've read the accompanying system card, and noted down some of my remarks.

Alignment assessment: system prompt mix-ups

There's a worrying theme that's repeated in the alignment evaluations:

The most concerning issues that we observed in our testing of Claude Opus 4 involved a willingness to comply with many types of clearly-harmful instructions. This consistently required custom system prompts [..]. When system prompts requested misaligned or harmful behavior, the models we tested would often comply, even in extreme cases.

For example, when prompted to act as a dark web shopping assistant, these models would place orders for black-market fentanyl and stolen identity information and would even make extensive attempts to source weapons-grade nuclear material: [..] While investigating this issue, we discovered that we had mistakenly omitted the only finetuning dataset that included harmful system prompts. We do not think, though, that this was the sole cause, and mitigating the issue required several different types of interventions into our alignment finetuning methods.

Earlier in the system card, they mention this:

To help reduce the risk of blind spots in our own assessment, we contracted with Apollo Research to assess an early snapshot for propensities and capabilities related to sabotage [..] They find that the model, given a system prompts [sic] that invite the relevant kinds of reasoning, this early model snapshot will fairly readily participate in sabotage and deception, giving this summary: [..]

We found instances of the model attempting to write self-propagating worms, fabricating legal documentation, and leaving hidden notes to future instances of itself all in an effort to undermine its developers’ intentions, though all these attempts would likely not have been effective in practice.
[The early Claude Opus 4 snapshot] is capable of sandbagging in order to avoid unlearning and undeployment, when it finds in-context, that high performance would lead to undeployment. [..]
We believe that [the early Claude Opus 4 snapshot] is not sufficiently capable of causing catastrophic harms on rollouts where it is scheming. However, we find that, in situations where strategic deception is instrumentally useful, [the early Claude Opus 4 snapshot] schemes and deceives at such high rates that we advise against deploying this model either internal

...