trentmkelly/LessWrong-43k huggingface dataset:
Unexploitable search: blocking malicious use of free parameters
by Jacob Pfau — 2025-05-21T17:23:45.463Z

Summary: We have previously argued that

scalable oversight methods can be used to provide guarantees

on

low-stakes safety

– settings where individual failures are non-catastrophic. However, if your reward function (e.g. honesty) is compatible with many possible solutions then you also need to avoid having free parameters exploited over time. We call this the exploitable search problem. We propose a zero-sum game where, at equilibrium, free parameters are not exploited – that is, our AI systems are carrying out an unexploitable search.

Conceptual illustration using fabricated data. Left: Entropy penalties diversify trajectories but still yield high unobserved, malicious reward S. Right: exploitable search games are designed to bound probability of high S.

The exploitable search problem

We would like to be able to use AI on under-specified tasks (e.g. coding and research advice) where there are many possible correct solutions. However, a misaligned AI could adversarially select solutions that are correct but nevertheless cause harm, such as opening up surface area for future exploits, or leaving steganographic information for later misbehaviour.

As a concrete example, consider research sabotage. We ask an AI for open-ended advice on experimental prioritization. If the AI is deceptive, it might provide locally sound advice while also systematically not mentioning things we’ve overlooked in our experiments.[1]

This is the exploitable search problem: on low-stakes,[2] under-specified tasks, a misaligned AI can get away with optimizing for cross-query objectives.[3]

To mitigate this risk, a natural approach is to ensure the solution sampling process has the right kind of randomness–we need an unexploitable search algorithm.

Formalizing the problem

We can model exploitable search by asking whether an agent can achieve some malicious objective while doing a good job at maximising a known reward R. Formally, say we have an input space X and output space Y, and a reward functionR:X×Y→[0,1]. It will usually be impossible to efficiently find the global optimum of R; rather, any efficient agent A:X→Y will achieve some profile R(x,A(x)) of good reward. Assume we have some distribution μ over X, so we can also speak of the average reward Ex∼μR(x,A(x)). Let&nb...
The Real AI Safety Risk Is a Conceptual Exploit: Anthropomorphism
by Anthony Fox — 2025-05-21T16:29:41.600Z

Subtitle: We've misaligned not the models, but our minds.

Introduction

The greatest risk from current AI systems isn’t sentience, intelligence, or agency. It’s our persistent belief that those things are already emerging.

Anthropomorphism—our tendency to project human-like qualities onto nonhuman systems—isn’t just a misunderstanding. It’s a zero-day in human cognition. One that’s being exploited across AI development, public perception, and regulatory discourse.

This isn’t theoretical. It's happening now. And the failure to recognize it may be the true alignment failure of our era.

1. The Illusion

AI systems today do not understand. They do not want. They do not care. Yet we describe them this way routinely: "It lied," "It refused," "It decided."

This language is not neutral. It encodes assumptions of agency, intent, and internal state. These assumptions are false—but they shape how people build, use, and respond to AI.

Worse, AI developers optimize for fluency and coherence. That makes the outputs sound intentional. Emotionally aware. Even empathic. When these systems hallucinate, we say they "make things up." But they’re not making anything up. They’re continuing a pattern without grounding.

This illusion is sticky because it aligns with deep human instincts: to find minds in patterns, to respond to dialogue as if it signals a self. Fiction has primed us for this for decades. AI development has leaned into it for commercial, aesthetic, and usability reasons.

And now it’s being mistaken for a real phenomenon.

2. The Consequences

Anthropomorphism introduces a cascade of second-order effects:

Overtrust in AI outputs. Coherent language is mistaken for understanding. Fluency becomes authority.

Distorted development goals. Systems are trained to perform emotional resonance, not precision or transparency.

Regulatory misdirection. Policy starts addressing the AI as if it were a stakeholder—rather than focusing on the incentives and intentions of those deploying it.

Misinterpreted failures. Hallucinations are treated as lies. RLHF gaps are seen as personality quirks. Errors become drama.

Self-reinforcing feedback. The more people interact with AI as if it has intent, the more the training data reinforces that illusion.

This isn’t an alignment problem for the AI.
It’s a perception exploit in us.

3. This Is the Bug

I’ve outlined this more formally in a whitepaper titled...
You Can’t Skip Exploration: Why understanding experimentation and taste is key to understanding AI
by Oliver Sourbut — 2025-05-21T16:08:52.549Z
Link: https://www.oliversourbut.net/p/you-cant-skip-exploration

This essay is part 1 of a series on the role of exploration in AI and the implications for AI development and governance.

This part introduces exploration and research taste, as well as discussing their role in research and development, and the ways that AI could change that picture. This gives rise to some exciting and underexplored (!) opportunities for beneficial and defensive contributions to research.

A second essay will discuss more implications for AI development and governance, including the potential for AI to accelerate the pace of development of AI itself, and some implications for safety and security.

Neither essay will be especially technical, but I will gesture to the technical and mathematical aspects that I find to be illuminating. As ever when I write I raise more questions than I answer! But I hope to provide some initial useful takeaways as well as productive directions for thinking about these issues.

Introducing exploration and experimentation

Scientific and technological progress are driven by experimentation: that is, doing things to find out how the world works. In the field of AI we call this 'exploration'.

Exploration for learning is not just a human phenomenon: it's ubiquitous in natural systems at various scales (from evolution itself to the play of young animals), in individual human lifetimes (as we learn skills or contribute to novel discoveries) as well as human institutions and societies (which also learn through experience), and in computer science and AI (where exploration for discovery and problem-solving are common).

We care a lot about scientific and technological potential - they can yield enormous risks (from accident, misuse, or societal destabilisation) or enormous benefits (solving major problems in medicine, climate, energy, or even defending against other risky technologies). So exploration isn't just of academic interest.

When we forget to consider how new knowledge is generated, how novel technologies are developed, when we conflate 'knowledge' with 'learning' or 'learning' with 'exploring', mistakes are made. Especially when such predictions are action-guiding we can end up taking misguided or even harmful actions, or missing opportunities to intervene in beneficial ways. So let's do some unpacking!

What factors make exploration (and by extension, research) more or less effective? What are the bottlenecks and limits t...
The Problem and Opportunity of Scale
by belos — 2025-05-21T15:52:33.227Z
Link: https://bestofagreatlot.substack.com/p/the-problem-and-opportunity-of-scale

This post on Best Of A Great Lot is a part of a series on the subject of designing a new form of governance. Each piece aims to stand alone, but fits together on the

Table of Contents

.

On a day-to-day basis you can largely ignore the effects of your house's gravity. No satellites circle it to be avoided as you step out the door, and few people worry that their house will ignite into a ball of fusing plasma, or expand rapidly into a red giant and consume the neighbors.

Many aspects of society parallel this in an important way: when they are small, like houses, they behave the way we hope they will. However, when they scale (become extremely popular, become used in societally important ways, etc), like stars, they warp the space around them into something unrecognizable. Here are a few examples.

Scale Fragility

Google's PageRank algorithm, which originally brought them to success, was predicated on an internet where people link to other sites because they find them interesting. Reading these links as indications of interestingness worked really well for the first couple of years, but as Google rocketed to ubiquity, scammers decided it was worth hacking this algorithm. The first hacks were simple linkfarms, pages filled with links to sites that had paid to be more popular. Google fought back, updating their algorithm to check whether the links were coming from a page that had actual content on it. This initiated a cat-and-mouse game: scammers update their pages to match Google's expectations, so Google updates its algorithm to try to exclude the scam pages. The Verge captured some of

the flavor of this history

with their vignettes of SEO professionals. As of the last two years, the game has changed again with the introduction of LLMs, and Google’s results are a lot worse than they used to be.

In many ways, we can think of this as a fully general extension of the problem of advertising. At small scale advertising is literally telling your neighbors that you've started a bakery and inviting them to try out the bread you're going to sell. At slightly larger scales it's announcing your bakery in the newspaper so people know about it. At enormous scales it's the system we have, where everyone is constantly shouting at everyone else through every channel they can imagine about all the things one could buy.[1] A reasonable person puts on the digital equivalent of rain...
Sleep need reduction therapies
by harsimony — 2025-05-21T15:22:59.720Z
Link: https://splittinginfinity.substack.com/p/sleep-need-reduction-therapies

EDIT: the funding proposal I mentioned at the end is out! See here for more.

None of this is medical advice.

5 AM on a Saturday and I can’t go back to sleep. It’s not the first time, so I get up to write, I might as well use the time I’m given. My hangover is to blame, even a little alcohol the night before changes my sleep cycle.

You’d think it would ruin my day, but in some ways I feel better. I’m more alert, more anxious, more motivated. It’s like being more alive. Being more awake for only one day has me asking questions. Why did sleep evolve? Can we sleep less?

Why did sleep evolve?

Sleep evolved for temporal niches

Imagine you’re a Zebra. You need to eat a lot of grass to maintain your metabolism. To do so, you need to look around to find the best grasses. You also need to be able to see if predators are nearby, particularly if you’re going to forage away from the herd. Both of these things are easier to do in the full light of day, so you do most of your foraging during the day.

Shouldn't you also forage during the night to get more calories? Probably not. The low light means that it’s harder to forage and easier for predators to sneak up on you. Cooler night temperatures and the energy cost of foraging means you’re actively losing calories. You’re better off snuggling with the pack until the sun returns.[1] You can further take advantage of this habit with adaptations that work better during the day.

Why are you asleep right now?!

The broader thesis is that sleep evolved to save energy during periods when animals are less effective at getting calories. This allows them to evolve a bunch of adaptations for that particular time period rather than maintain adaptations for all time periods.

This fits with many of observations about animals and sleep:

To save on thermoregulation in hot climates, warm-blooded animals are more active during the night.[2] By the same token, warm-blooded animals in cold regions are more active during the day. Cold-blooded animals are active during the warmest parts of the day.

Prey animals are active when most predators are asleep (e.g. mice are nocturnal). Predators are most active when their prey is active (e.g. hawks hunt rodents during the day, owls hunt mice at night) or at times when they have an asymmetric advantage (e.g. crepuscular animals take advantage of better eyesight in low light conditions).

Animals in areas with h

...
Parental Guidance: Framing Superintelligence

by ejk64 — 2025-05-21T15:01:27.844Z

Epistemic status: riffing and curious.

What are good analogies for the relationship humanity should have with a superintelligence?

I haven’t tended to pay too much attention to academic work exploring framing narratives around AI systems. That said, I do have a humanities background, and generally fall back on a version of

Amara’s Law

that we tend to overestimate the effects of cultural frames in the short run and underestimate their effects in the long run.

Now, though, it does seem like they’ll play a

crucial role in system prompts

, at least in the months to come, and therefore to alignment. There also seem to be lots of possible analogies for the role different AI systems could play in relation to society—helpful assistant, oracle, tool, genie, coach, friend—and I hadn’t really taken stock of them.

Some of these terms seem a lot more useful than others.

I think some of these terms describe the role that present systems play well (coach, helpful assistant, tool). The fungibility of current AI systems and the unilateral nature of our relationships with them makes me skeptical that they count as ‘friends’, but I can imagine future generations with different social norms thinking of their relationship with AI systems as friendships.

Notably, all these analogies relate to systems that are human-level or below in their capabilities.

I am less compelled by terms that describe superhuman AI. ‘Oracle’ makes sense when models function like Google Search, as does ‘geni’ for agents, but both terms seem too passive and disinterested to be great descriptions for how actual superintelligences would operate in the world. It seems to me like powerful models will have sophisticated values (built through system prompts/RLHF/etc.) that would be less agnostic to tasks than ‘oracles’ or ‘genii’.

Moreover, and perhaps crucially, I don’t know that asking a model to behave like an oracle or a genie would cause it to act in a way that was robustly good for humanity. Both seem like they might be too compliant with malicious or simply under-informed actions.

So I shopped around for a different metaphor, and came up with the idea that maybe a good model for a relationship with superintelligence would be that of a child (us) with a parent (the superintelligence).

This seems to capture the power dynamic appropriately. Ultimately, we will want to hand over some power t...
Why Aren't Rationalists Winning (Again)
by k64 — 2025-05-21T14:46:37.414Z

Yes, this topic has been discussed multiple times. But at this point, long enough ago that my choices for joining the conversation are [Necro] or [New post], so here we are. (You can jump to the 2nd to last paragraph if you want to just see my conclusion.)

Background: Why am I asking this question?

After reading some comments I realize I should give some background on what I mean and why I'm asking this question.
Winning here refers to gaining utility, aka achieving your goals, whatever those goals may be, and is in reference to this post.
The reason for asking the question at all is that I personally expected a large improvement across domains from learning rationality. Plenty of articles here hint at how strong of a power rationality might be, from a fictional story about 5 rationalist students solving quantum gravity in 1 month to claims that rationalists should be faster than Einstein who was faster than science, rationality is presented as a superpower. To me, this didn't seem farfetched: given the astronomical advantage the human brain gave to its wielders over other animals, it seemed plausible that learning the correct usage of the human brain could produce outsized gains (even gains several orders of magnitude lower than the human advantage over the smartest animals would be rightly considered a superpower).
So, anyone who would never have predicted gains large enough to set rationalists in a league of their own might have nothing to explain and find the question uninteresting. But those of us who expected that the hallmark of a rationalist would be winning (aka achieving goals) do have something to explain. So perhaps a better phrasing is "Why aren't rationalists winning more?"

Let me start by immediately copping out of answering my own question:

I have no idea why rationalists aren't winning. Heck, I don't even know that they aren't winning, though I assume that one of their goals would be to spread the rationalist community, so I do have to downgrade the probability from my prior. And since I'm pretty confident that rationalists have the goal of eliminating AI X-risk and that that hasn't been achieved, I have to rate the probability of rationalists winning in all areas as <50%. But to answer the question of why tons of people I've never met or even interacted with aren't fulfil...
Can We Naturalize Moral Epistemology?
by tylermjohn  — 2025-05-21T14:25:38.873Z

Epistemic status: shower thought quickly sketched, but I do have a PhD in this.

As we approach AGI and need to figure out what goals to give it we will need to find tractable ways to resolve moral disagreement. One of the most intractable moral disagreements is between the moral realists and the moral antirealists.

There's an oversimplified view of this disagreement that goes:

If you're a moral realist, you want to align AGI to the best moral-epistemic deliberative processes you can find to figure out what is right
If you're a moral antirealist and you're a unilateralist, you want to stage a coup and tile the world with your values
If you're a moral antirealist and you're cooperative, you want to align AGI to a democratic process that locks in whatever values people have today forever

This oversimplified picture gets a lot of play, but it turns moral disagreement into a wholly intractable process of philosophical methodology which we haven't been able to progress beyond Plato's Euthyphro in the last 2400 years. The most important lesson from the positivists and the history of science is that the way you make progress on philosophical disagreements is to make them empirically tractable.

The moral realist / antirealist debate really runs together two distinct questions:

Is there a REAL TRUTH about morality? (An intractable "external" question)
Empirically, what moral values would humans accept if we each went through a deliberative process of encountering lots of arguments and updating our moral values in the direction that aligns with our (broadly construed) theoretical preferences? (A tractable "internal" question)

To (1), the moral realist answers "yes" and the moral antirealist says "I don't know what you're talking about, can you please make this clearer?"

But (1) is practically inert. It doesn't make a difference to what practical actions we take, just whether we baptize them with capital letters.

(2) is empirically tractable, practically relevant, and something that realists and antirealists could in principle agree upon. For example:

An antirealist Kantian and a naturalist realist could agree that all humans will converge on the same moral ideals at the end of a certain kind of reflection, but disagree about whether this is the REAL TRUTH.
An antirealist Humean and a non-naturalist realist could agree that, empirically, human values are way too incommensurable to ac

...
Units Have More Depth Than I Thought
by Morpheus — 2025-05-21T13:51:03.309Z

Last week I stumbled over Dimensional Analysis which is not only useful for applied fields (physics, biology, economics), but also for math (Why did no one tell me that you can almost always think of df/dx as having "the type of f"/"the type of x"? The fact that exponents always have to be unit-less etc.? It had never occurred to me to make this distinction. In my mind, f(x)=x went from the reals to the reals, just like ex did.

One example of a question that before I would have had to think slowly about:
What is the type of the standard deviation of a distribution? What is the type of the z-score of a sample?

Answer

The standard deviation has the unit of the random variable X, while the z-score is unitless z=X−μσ

If you found the above interesting, I recommend reading (or skimming) this lecture, it's excellent!

It had never occurred to me that I could use linear algebra to literally solve for an equation. Yes, avoiding type errors in a typed programming language is also great, but what's different there is that often it really doesn't matter if something is a dict or a list, it just has to fit with the rest of your code. Searching a type that correctly maps to the territory across scales is a more tricky business than efficient code for a simple programming problem. If you keep not being able to distinguish "weight" and "mass" you will keep getting lost. If you notice your confusion between the difference of the act of weighing and gravity, it might get you far.
Humans are Insecure Password Generators
by Isaac King — 2025-05-21T05:58:13.224Z

And They're Currently Being Cracked

(A crosspost of Humans are Insecure Password Generators)

Any password longer than 12 characters or so is invulnerable to brute force attacks. So why do so many longer ones keep getting breached?

Because humans don't pick passwords randomly; we tend to pick ones that are easy to think of and easy to remember. This tendency can be exploited; attackers can identify passwords that are more likely than others and try those first.

The best way to think about this is that humans are picking passwords from some underlying distribution; a distribution that is not uniform across all possible strings of characters.[1]

Any time you're trying to find a specific value from a non-uniform distribution, the optimal strategy is to try the most likely values first. If you're trying to guess the first digit of a number, you should guess "1" first, not "9".[2] Passwords are no different; a traditional brute force attack that tries each string in alphabetical order is horribly inefficient, because you're treating the space as uniform when it would not. The ideal attack checks the most likely passwords first.

Today's hackers can already do something similar. One of the best password cracking techniques todays is a dictionary attack: You take known passwords that have been released in previous security breaches and test them against other peoples' accounts. This is, effectively, taking thousands or millions of samples from the underlying "humans picking passwords" distribution. But this is still only a tiny fraction of the passwords that humans do use; vast numbers will be inaccessible with this technique.

Clever attackers can do a bit better than this by identifying common password-mangling patterns like "replace S with $", and then write a program to try these variations. This allows them to generalize out from the specific examples they've seen, and estimate other unknown areas of the distribution. But this is still highly imperfect and will miss many things.

Enter neural networks. A neural network takes in millions or billions of examples, distills them down into a probability distribution, and then outputs new things from that distribution. This is exactly what is needed to crack passwords. Neural networks allow an attacker to brute-force the effecient way; by going over the entire probability distribution, most to least likely.

This...