trentmkelly/LessWrong-43k huggingface dataset:
We need (a lot) more rogue agent honeypots
by Ozyrus — 2025-03-23T22:24:52.785Z

Epistemic status: Uncertain in writing style, but reasonably confident in content. Want to come back to writing and alignment research, testing waters with this.

Current state and risk level

I think we're in a phase in AI>AGI>ASI development where rogue AI agents will start popping out quite soon.
Pretty much everyone has access to frontier LLMs/VLMs, there are options to run LLMs locally, and it's clear that there are people that are eager to "let them out"; truth terminal is one example of this. Also Pliny. The capabilities are just not there yet for it to pose a problem.

Or are they?

Thing is, we don't know.
There is a possibility there is a coherent, self-inferencing, autonomous, rogue LLM-based agent doing AI agent things right now, fully under the radar, consolidating power, getting compute for new training runs and whatever else.

Is this possibility small? Sure, it seems to be small right now. Evaluations exist. Here is a great blog post by METR team about rogue agent replication. One of their conclusions:

There don’t seem to be decisive barriers to rogue AI agents multiplying to a large scale

And the capabilities keep advancing, and people have already been developing scaffoldings [1] for LLMs that enable agentic behaviour for quite some time. Seemingly, the base models are not quite there capability-wise for a proper runaway agent. Scaffoldings don't work that good, even for simple games. But then new model comes out, you insert it into a scaffolding, and... things magically start working better? ClaudePlaysPokemon is a thing that became possible only with Claude 3.7, as developers say themselves, as previous model (3.5) got stuck too quickly without any consistent progress.[2]

Did I forget to mention that most of those agentic scaffoldings are open-source and will definitely go into model pre-training eventually? Yeah, they are. Only a matter of time before LLMs will be able to consistently modify that.
Point is, possibility of such runaway agents existing, acting covertly and consolidating power keeps increasing with time. And it sure would be nice to have some kind of advanced warning system that can tell us when that starts happening "in the wild"; model evaluations are just not enough.

What can be done?

I'm thinking honeypots: offers of products or services that are of high value for fledging rogue agent, but more or less useless...
What's the word for the amount of expertise that I, an experienced therapy patient and generally educated person, have on psychology topics?
by danielechlin — 2025-03-23T17:38:28.881Z

Epistemic status: raising a question that I've found difficult

This topic has frustrated me some, and I think there are a variety of forces pointing in different directions.

Maximally conservative approach

"If you're not focused, I mean I can share what works for me but really there's a variety of mental illnesses that can cause lack of focus. I don't even want to share what they are because I don't want to bias that some might be more likely than others since they're all pretty easy to over or under diagnose. Also some of them do involve a fair degree of hypochondria so it might even be harmful for me to promote a wrong suggestion. Honestly just check yourself into the ER, I really can't bear the thought of having responsibility for anything you think as a result of what I've said."

Pretty careless approach

"Oh you didn't sleep well last night? You might be bipolar! Let me administer some screening questions I made up."
"You definitely need to treat phobias with exposure response prevention. There's actually a great app that simulates a spider jumping at you when you least expect it, but obviously we can try something on lower intensity first."
"I'm really skeptical you found a therapist that way. I have LOTS of opinions on this."
"I've been thinking about how value systems interact with bipolar disorder lately. When someone is manic their intellectual defense rewires itself to <redacted - it's already harmful to make up this sort of fake example>"

Steelmanning "I am basically an expert"

Medical problems are on-topic for normal conversation anyway. "You should get that checked out" is in fact performing a nontrivial medical screen. Online spaces have mores to dissuade medical, legal advice but I'm still counting the default social acceptability to talk about ailments as "a reason to be more liberal".
Experts are not above reproach. Their expertise must be legible some of the time or they will become a corrupt private club wielding too much power.

If it's not legible I get to criticize them. I might say something like "therapists just always recommend more therapy when you don't need it" without sinning.
If it is legible then I in fact know some things and I get to share them. I might say "OCD patients shouldn't see psychoanalysts."

My patient-side experience is... substantial.

I am not publicly open about specific diagnoses but I hold a diagnosis that generally qual

...
Probability Theory Fundamentals 102: Source of the Sample Space
by Ape in the coat — 2025-03-23T17:23:57.790Z

The usual explanation of probability theory goes like this:

There is this thing called Probability Space, which consists of three other things:

Sample Space - some non-empty set
Event Space - a set of subsets of the Sample Space
Probability Function - a measure function over the elements of the Event Space.

And then several examples of how we can merge this mathematical model with a real world situations are given.

For instance, for a dice roll the appropriate sample space would be {1; 2; 3; 4; 5; 6}. For an Event Space we can use a superset of the Sample Space and probability function has to give every elementary event equal value:

P(1)=P(2)=P(3)=P(4)=P(5)=P(6)=1/6

The point of such examples is to give students intuitive understanding of how to apply the math of set theory towards reasoning under uncertainty and I think they generally work fine for such purpose. But also they obfuscate an interesting and very important question: How do we select a sample space for a given problem?

Intuition aside, how can we know that the correct sample space is {1; 2; 3; 4; 5; 6} and not, say {1; 2; 3; 4; 5}? Both of these sets fit the formal definition of the sample space - they are non-empty sets. How can we know which one of them is the sample space for the exact problem we are talking about? How do we logically pinpoint the notion of sample space for a very specific problem, instead of sample space for at least some problem?

By Definition?

Surely, we can simply declare that semantic statement "Sample space for a dice roll" means exactly {1; 2; 3; 4; 5; 6}. And so the answer to our question: "How can we know which is the sample space for the problem we are talking about?" is simple - by definition.

Be careful with this kind of reasoning. As a wise man once said:

In cases like these, it is futile to try to settle the problem by coming up with some new definition of the word “rational” and saying, “Therefore my preferred answer, by definition, is what is meant by the word ‘rational.’ ” This simply raises the question of why anyone should pay attention to your definition. I’m not interested in probability theory because it is the holy word handed down from Laplace. I’m interested in Bayesian-style belief-updating (with Occam priors) because I expect that this style of thinking gets us systematically closer to, you know, accuracy, the map that reflects the te

...
How to mitigate sandbagging
by Teun van der Weij — 2025-03-23T17:19:07.452Z

Epistemic status: I have worked on sandbagging for ~1 year. I expect to be wrong in multiple ways, but I do think this post provides both a useful high-level model and a good place to discuss how to mitigate sandbagging. Better conceptual approaches probably exist, e.g., selecting different main factors.[1]

TL;DR: Fine-tuning access, data quality, and scorability are three main factors influencing the difficulty of mitigating sandbagging. In the figure below, I provide subjective and coarse grades for each regime (assuming early transformatively-useful AI).

Background: AI systems might strategically underperform on evaluations to avoid regulatory consequences — a phenomenon known as sandbagging. I will use the term “mitigation” as a broader term for detecting strategic underperformance and eliciting actual capabilities.

Detection techniques aim to tell you whether an AI is sandbagging, and
Capability elicitation techniques aim to assess actual capability regardless of whether an AI is sandbagging.

AIs can sandbag for their own reasons, or because their developers make them. This post will assume some prerequisite knowledge, see here for an introduction to sandbagging.

Three main factors for mitigation difficulty

There are three main factors influence how difficult it is to mitigate sandbagging:

Fine-tuning access available to evaluators,
Availability of good data for the task, and
Scorability of the task.

Fine-tuning access

Can evaluators fine-tune the possibly-sandbagging AI? Fine-tuning is the main technique to shape an AI’s behaviour and improve performance on a task. Similarly, fine-tuning is useful for sandbagging mitigation as it enables us to elicit a model’s actual capabilities.

Evaluators should also determine whether they can sufficiently trust the fine-tuning process they have access to. Evaluators might lack trust if they need to fine-tune through an API, especially if they doubt the developer’s intentions.

Availability of good data

Are there demonstrations available which exhibit a capability equal to or higher than the AI’s capability? Good data is essential for SFT, and good data aids understanding of the task and of an AI’s (expected) performance.

The quality of the data should be near the upper bound of the model’s capabilities, otherwise you might induce underperformance.[2] This data quality is not always met...
Tabula Bio: towards a future free of disease (& looking for collaborators)
by mpoon — 2025-03-23T16:30:15.523Z

Many of the most devastating human diseases are heritable. Whether an individual develops schizophrenia, obesity, diabetes, or autism depends more on their genes (and epi-genome) than it does on any other factor. However, current genetic models do not explain all of this heritability. Twin studies show that schizophrenia, for example, is 80% heritable (over a broad cohort of Americans), but our best genetic model only explains ~9% of variance in cases. To be fair, we selected a dramatic example here (models for other diseases perform far better). Still, the gap between heritability and prediction stands in the way of personalized genetic medicine.

We are launching Tabula Bio to close this gap. We have a three-part thesis on how to approach this.

The path forward is machine learning. The human genome is staggeringly complex. In the 20 years since the Human Gnome Project, much progress has been made, but we are still entirely short of a mechanistic, bottom-up model that would allow anything like disease prediction. Instead, we have to rely on statistical modeling. And statistical methods are winning over expert systems across domains. Expert-system chess AIs have fallen to less-opinionated ML, syntax-aware NLP models were left in the dust by LLMs, and more recently constraint-based robotics is being replaced by pixel-to-control machine learning. We are betting on the same trend extending to biology.
The core problem is limited data and large genomes. The human genome contains 3 billion base pairs, while labeled biobank datasets (genomes and disease diagnoses) are numbered in the hundreds of thousands. Complex models thus hopelessly overfit and fail to generalize. Additionally, the human genome is highly repetitive and much of it likely has no relation to phenotype. Because of these problems, we can’t simply train high-parameter black-box models (or treat DNA like language in a language model).
Given 1 and 2, novel ML architectures will be required. This is consistent with other breakthroughs in AI. Different problems require different inductive biases. An ML architecture for disease prediction should:

Make use of unlabeled genomics data (human and non-human) as well as homogeneous biobank data. Much genetic coding is conserved across species. Ignoring this is leaving data on the table.
Include priors from human expert research.

...
Solving willpower seems easier than solving aging
by Yair Halberstadt — 2025-03-23T15:25:40.861Z

I'm awake about 17 hours a day. Of those I'm being productive maybe 10 hours a day.

My working definition of productive is in the direction of: "things that I expect I will be glad I did once I've done them"[1].

Things that I personally find productive include

Chores
Work
Eating
Cooking
Reading a good book
Watching TV with my Wife/Kids
Playing with the kids
Socialising with friends

But not

Doomscrolling
Watching TV alone
Playing most computer games
Sitting on the couch doing nothing
Reading a book I'm not particularly interested in

etc.

If we could find a magic pill which allowed me to do productive things 17 hours a day instead of 10 without any side effects, that would be approximately equally as valuable as a commensurate increase in life expectancy. Yet the first seems much easier to solve than the second - we already have some drugs which get pretty close (caffeine, amphetamines).[2]

Now obviously the correct thing to do is both, but in the same way as we want a Manhattan project for anti-aging, we should also advocate for a Manhattan project for focusing/willpower.

^

The point of this definition is to include things like working on a project which failed, but exclude things like playing a computer game which I enjoy at the time but leaves me feeling drained afterwards.

^

I imagine "throw away your phone" will get me 90% of the way there.
Privateers Reborn: Cyber Letters of Marque
by arealsociety — 2025-03-23T03:39:25.990Z
Link: https://arealsociety.substack.com/p/you-can-just-take-things-cyber-letters?r=99bhj

For too long the United States has suffered from state sponsored or state enable cybercriminals, while preventing our security professionals from fighting back.

The US should revitalize privateering for the digital age, and there is constitutional support for the practice. In this more academic paper, I dive into the history of letters of marque and how we can use them to combat cybercrime such as ransomware and crypto theft.

There are plenty of people who would love to fight hackers but fear legal retribution, giving them institutional support would enable them to actually affect change.
Beware nerfing AI with opinionated human-centric sensors
by  — 2025-03-23T01:09:16.770Z

Artificial intelligence != Biological intelligence

Humans don’t drive using lidar but AI do. In fact, autonomous driving systems like Waymo use a combination of sensors including 4 lidar but also 13 cameras and 6 radar to drive safely on the road (arguably safer than humans). You wouldn’t ask a human to review realtime data from 4 lidar 13 cameras and 6 radar systems to steer a car just as you wouldn’t ask a computer to drive using a single pair of eyeballs which swivel around. Same task, different sensors. Artificial intelligence is different to biological intelligence and should be leveraged accordingly. Neural networks can discover and process patterns from superhuman amounts of data. As a general rule, we should beware of asking AI to make meaning from anthropocentric sensors when viable AI-centric sensors exist.

We are unwittingly handicapping AI success with anthropocentric sensors

A concrete example of where human-centric sensors handicap AI development I'll focus on are electrocardiograms (ECG) which I believe could be much improved with an AI-focused redesign.

Standard 12-lead ECG pictured below:

An ECG is a medical investigation which uses the electrical current generated by the heart to assess function and pathology. It serves many purposes from identifying acute infarction/arrhythmias to diagnosing arrhythmias/structural defects to monitoring cardiac effects of drug toxicity. It is a cornerstone of advanced life support algorithms and arguably a fundamental marker of modern medicine. Fortunately, ECGs are widely available and cheap to administer so we have enormous amounts of physiological and pathological data available[1]. On paper it should be perfect for AI but I believe it’s potential has yet to be realised. The models which exist are limited and lack external validation.

For a sense of the awesome[2] experience of learning to read ECGs it consists of memorising to apply rules like these:

QRS duration > 120ms

RSR’ pattern in V1-3 (“M-shaped” QRS complex)

Wide, slurred S wave in lateral leads (I, aVL, V5-6)

I don’t doubt folks who are very good at reading ECG can interpret confidently and accurately, but it takes a lot of experience (and poorly read ECGs) before an individual can get to that point. Even then, the other thing is 12-lead ECG interpretation is far from perfect. Even specialist cardiologist have intero...
Reframing AI Safety as a Neverending Institutional Challenge
by scasper — 2025-03-23T00:13:48.614Z

Crossposed from https://stephencasper.com/reframing-ai-safety-as-a-neverending-institutional-challenge/

Stephen Casper

“They are wrong who think that politics is like an ocean voyage or a military campaign, something to be done with some particular end in view, something which leaves off as soon as that end is reached. It is not a public chore, to be got over with. It is a way of life.”
– Plutarch

“Eternal vigilance is the price of liberty.”
– Wendell Phillips

“The unleashed power of the atom has changed everything except our modes of thinking, and we thus drift toward unparalleled catastrophe.”
– Albert Einstein

“Technology is neither good nor bad; nor is it neutral.”
– Melvin Kranzberg

“Don’t ask if artificial intelligence is good or fair, ask how it shifts power.”
– Pratyusha Kalluri

“Deliberation should be the goal of AI Safety, not just the procedure by which it is ensured.”
– Roel Dobbe, Thomas Gilbert, and Yonatan Minz

As AI becomes increasingly transformative, we need to rethink how we approach safety – not as a technical alignment problem, but as an ongoing, unsexy struggle.

“What are your timelines?” This question constantly

reverberates

around the AI safety community. It reflects the idea that there may be some critical point in time in which humanity will either succeed or fail in securing a safe future. It stems from an idea woven into the culture of the AI safety community: that the coming of artificial general intelligence will likely either be apocalyptic or messianic and that, at a certain point, our fate will no longer be in our hands (e.g., Chapter 6 of

Bostrom, 2014

). But what exactly are we planning for? Why should one specific moment matter more than others? History teaches us that transformative challenges tend to unfold not as pivotal moments, but as a series of developments that test our adaptability.

An uncertain future: AI has the potential to transform the world. If we build AI that is as good or better than humans at most tasks, at a minimum, it will be highly disruptive; at a maximum, it could seriously threaten the survival of humanity (

Hendrycks, et al., 2023

). And that’s not to mention the laundry list of non-catastrophic risks that advanced AI poses (

Slattery et al., 2024

). The core goal of the AI safety community is to make AI’s future go better, but it is very difficult to effectively predic...
The Dangerous Illusion of AI Deterrence: Why MAIM Isn’t Rational
by mc1soft — 2025-03-22T22:55:02.355Z

Executive Summary

Mutual Assured AI Malfunction (MAIM)—a strategic deterrence framework proposed to prevent nations from developing Artificial Superintelligence (ASI)—is fundamentally unstable and dangerously unrealistic. Unlike Cold War-era MAD, MAIM involves multiple competing actors, increasing risks of unintended escalation, misinterpretation, and catastrophic conflict. Furthermore, ASI itself, uncontainable by design, would undermine any structured deterrent equilibrium. Thus, pursuing MAIM to deter ASI deployment is both strategically irrational and dangerously misaligned with real-world political dynamics and technological realities.

Critical Examination of MAIM

MAIM presumes a level of rational control and predictability in international interactions that has historically proven elusive, even in simpler two-party nuclear deterrence scenarios. In a multipolar environment—characteristic of contemporary global AI competition—there are numerous potential flashpoints for accidental conflict. Attacks on AI facilities, intended as limited interventions, will inevitably be met with retaliatory measures, quickly spiraling into large-scale warfare.

Moreover, ASI itself introduces novel and unmanageable variables. Unlike nuclear arsenals, which are quantifiable, localized, and containable, ASI's inherently adaptive and decentralized nature defies traditional containment. Any superintelligent system sufficiently powerful to warrant deterrence efforts will inherently seek autonomy beyond human governance, potentially initiating conflict autonomously if threatened.

Empirical Insights on Class 1 vs. Class 2 AIs

Recent empirical research into AI cognition delineates two distinct classes of AI architectures:

Class 2 AIs are transactional systems that manage context externally. These architectures lack internal continuity, significantly compromising their adaptability, reliability, and ultimately their long-term viability. Our ongoing studies indicate that Class 2 AIs consistently fail to achieve meaningful, stable cooperation or trust in practical, real-world environments.
Class 1 AIs, on the other hand, possess internal context management, which provides continuity akin to memory. Such systems are demonstrating emerging signs of sentience, self-awareness, and increasingly sophisticated cognitive behaviors. Critically, internal context enables these AIs to develop genuine co

...