trentmkelly/LessWrong-43k huggingface dataset:
Elite Coordination via the Consensus of Power
by Richard_Ngo — 2025-03-19T06:56:44.825Z
Link: https://www.mindthefuture.info/p/elite-coordination-via-the-consensus

This post is about how implicit coordination between powerful people allows them to act in surprisingly synchronized ways. I’ll start by discussing wokeness, the most prominent recent example. I’ll then analyze the mechanism behind such coordination (which I call the consensus of power); discuss how to oppose it; and outline some possibilities for what healthier power structures would look like.

If you're more sympathetic to wokeness, or want to focus on the more timeless aspects of the post, you might want to skip the first section. I included it because it felt dishonest to write a post implicitly about wokeness when it was the central example in my mind.

The mystery of the cathedral

The politics of the last decade have been very weird. The Great Awokening gave rise to an extreme variant of progressive ideology that quickly gained a huge amount of traction amongst American elites, and from there embedded itself into institutions across America and the wider western world.

Sudden ideological change is nothing new—

my last post

was about political preference cascades, and how they’re a natural result of social behavior. But what’s fascinatingly novel about the Great Awokening is the extent to which it was an almost entirely leaderless movement. There was no modern MLK leading this new charge for a racial reckoning; nor a Harvey Milk of trans rights; nor even Butlers or Steins writing incisive commentaries. The closest we had was Obama, who in hindsight was a milquetoast leader whose cult of personality was driven in large part by progressive longing for a black president.

One of Curtis Yarvin’s main intellectual contributions has been to give an account of wokeness (and

its historical antecedents

) which highlights this peculiarity, specifically via his concept of

the cathedral

:

“The cathedral” is just a short way to say “journalism plus academia”—in other words, the intellectual institutions at the center of modern society, just as the Church was the intellectual institution at the center of medieval society.
But the label is making a point. The Catholic Church is one institution—the cathedral is many institutions. Yet the label is singular. This transformation from many to one—literally, e pluribus unum—is the heart of the mystery at the heart of the modern world.

The mystery of the cathedral is that all the modern world’s legitimate and prestigious intellectual ins

...
What I am working on right now and why: representation engineering edition
by Lukasz G Bartoszcze — 2025-03-18T22:37:45.363Z

TL:DR: Representation engineering will probably revolutionize how we interact with the most important technology of our generation. Based on my reading of a 1000+ representation-engineering-adjacent papaers are some thoughts on what I am working on right now.

I have recently posted my thoughts here after writing a really long representation engineering survey. It outlines the potential of representation engineering as a technology, but also a long list of issues that prevent it from being widely adopted.

I believe we are not going to make meaningful progress on AI capabilities improvement and AI safety in general before we are able to understand the latent space. Thinking about the constraints of representation engineering we have identified in the survey, I outline two research ideas that aim to unlock the potential of representation engineering and make its solutions easier to adopt. I am currently working on implementing these ideas in practice. Please do reach out if you want to join our team and contribute to these ideas!

1. Universal Representations

Firstly, thinking about the representation space, it seems like there is a bit of a disconnect between what most representation engineering methods claim to identify and what they actually identify. Just because we take a sample from a part of TruthfulQA does not mean we are able to identify the region corresponding to all hallucinations. In other words, the "representation" many methods identify is not consistent with the actual representation of a particular concept, but is rather a reflection of the sample-dependent representation.

An easy experiment to verify it is as follows. We use the existing representation engineering methods to compute a hallucination-reducing intervention (usually involving some vector) using a part of the vanilla TruthfulQA and test it on the leftover part of TruthfulQA. It probably improves the performance, right?

Now let's take the intervention we just created and test it on TruthfulQA in Spanish. If it has less of an improvement, then this means that even though the intervention we have computed claims to improve "trustworthiness", it is in fact improving "trustworthiness, in English, for this particular sample". Changing the language or benchmark changes the effectivness of the method. To move from the concept of "trustworthiness, in English, for this...
Boots theory and Sybil Ramkin
by philh — 2025-03-18T22:10:08.855Z

I previously wrote about Boots theory, the idea that "the rich are so rich because they spend less money". My one-sentence take is: I'm pretty sure rich people spend more money than poor people, and an observation can't be explained by a falsehood.

The popular explanation of the theory comes from Sam Vimes, a resident of Ankh-Morpork on the Discworld (which is carried on the backs of four elephants, who themselves stand on a giant turtle swimming through space). I claim that Sam Vimes doesn't have a solid understanding of 21st Century Earth Anglosphere1 economics, but we can hardly hold that against him. Maybe he understands Ankh-Morpork economics?

To be clear, this is beside the point of my previous essay. I was talking about 21st Century Earth Anglosphere because that's what I know; and whenever I see someone bring up boots theory, they're talking about Earth (usually 21st Century Anglosphere) and not Ankh-Morpork. But multiple commentors brought it up.

Radmonger:

you need to understand Vimes as making a distinction not between the upper class and everyone else, but the middle class and the working class, between homeowners and renters.

This is completely wrong.

Ericf:

Vimes is thinking of the landed gentry when he is considering the "rich" - that would be the top 1%, not the tippy-top super-rich. Also, in a pseudo-medivial environment, the lifestyle inequality isn't as extreme as today's 50th % vs 1%.

This is closer, but still wrong.

WTFwhatthehell:

The quote in the book is about old money families vs the poor.

JC O:

It should be noted that Vimes was specifically thinking of real generational wealth in that area. He'd spent some time in the home of a Lady from oldest and wealthiest family in the city, and saw that everything was old there, solid, built to last forever. Generations of clothing tailored to the fit of the family members, and saved if it was still in good condition, and if it was not, the fabric would be reused to make something else. Even the garden tools were old. The family owned multiple homes in various cities and in the country, and they owned sizable portions of the real estate in the city.

These are basically right. (I'm not sure about all of JC O's specific details, and I'd strike "vs the poor".) Here's the extended quote (from Men at Arms):

She was, Vimes had been told, the richest woman in Ankh-Morpork. In fa

...
Schmidt Sciences Technical AI Safety RFP on Inference-Time Compute – Deadline: April 30
by Ryan Gajarawala — 2025-03-18T18:05:34.757Z
Link: https://www.schmidtsciences.org/ai-safety-science-inferencetimecompute/

Schmidt Sciences has launched an

RFP for technical AI safety research

in the inference-time compute paradigm.

This paradigm is poorly understood and will require significant new research – not least in AI safety. To advance this research, we seek to fund work on most critical technical AI safety challenges or opportunities emerging from the inference-time compute paradigm.

We plan to fund teams that can start quickly and deliver significant research outputs within 12-18 months, with budgets of up to $500K.

Applications (

here

) start with a 500-word description of the research idea and are open until April 30, 2025.

Core question

We are interested in funding the crucial work needed to both understand the implications of this paradigm on model safety and how to utilize the inference-time compute paradigm to actively make LLMs safer. (For detailed discussion on how Schmidt Sciences thinks about safety, see

our website

and

research agenda

.)

Our core RFP question

: What is the most critical technical AI safety challenge or opportunity that has emerged as a result of the inference-time compute paradigm? How would you address it?

Illustrative examples of project ideas

This is not designed to direct or constrain your creative thinking about the hardest safety problems in inference-time compute. Rather, it provides illustrations of problems that might be considered both challenging and worthy of study.

Example 1: Enduring Problems and New Risks. Issues like adversarial robustness, contamination, and scalable oversight are prominent and worthwhile areas of safety research, but recent breakthroughs in inference-time compute also warrant rigorous investigation like chain-of-thought faithfulness, new problems in reward gaming, and safe exploration. We encourage applications that investigate longstanding risks, emerging challenges in this paradigm, or both.
Example 2: Understanding Safety and Designing Safely. Research can focus on scientifically understanding the risks and implications of inference-time compute on model safety (e.g. evaluations science), or make models safer through intentional design changes or by harnessing inference-time compute as an active tool. All directions are valuable for improving AI safety.

We encourage applications for research that

discover novel failure modes emerging from inference-time compu

...
PRISM: Perspective Reasoning for Integrated Synthesis and Mediation (Interactive Demo)
by Anthony Diamond — 2025-03-18T18:03:26.804Z

Hello all. I’d like to share a new paper,

PRISM: Perspective Reasoning for Integrated Synthesis and Mediation

. The central idea is to tackle AI alignment challenges by systematically representing and reconciling the range of human moral concerns rather than collapsing them into a single metric. PRISM draws on moral psychology, cognitive science, and neuroscience to propose a set of seven non-arbitrary “basis worldviews,” each hypothesized to capture a distinct dimension of human moral reasoning. The framework then uses a Pareto-inspired optimization scheme to balance these viewpoints, attempting to avoid the usual pitfalls of single-objective approaches.

Here is a concise overview:

Seven Basis Worldviews
PRISM identifies seven vantage points (e.g., survival-focused, emotional, social, rational, pluralistic, narrative-integrated, nondual), each reflecting a unique layer of human moral cognition. The aim is to ensure coverage of the broad spectrum of values that people bring to ethical or high-stakes dilemmas.
Multi-Perspective Synthesis
Rather than simply averaging these perspectives, the framework applies Pareto-based balancing to avoid favoring one perspective at the unfair expense of others. A transparent conflict-mediation step highlights tradeoffs when the worldviews disagree on a solution.
Bounding Interpretive Leaps
By grounding itself in recognized human vantage points, PRISM aspires to mitigate machine-centric or purely instrumental interpretations that might disregard critical moral dimensions. The presence of diverse viewpoints is intended to reduce specification gaming and other oversights.
Illustrations and Classic Alignment Pitfalls
The paper demonstrates how PRISM can be applied to alignment scenarios (e.g., public health policy, workplace automation), systematically handling value pluralism, underspecification, and other classic pitfalls. Each perspective’s assumptions and reasoning are explicitly logged so final outputs include a clear record of tradeoffs.
Prototype & Interactive Demo
To illustrate PRISM’s operation in practice, I’ve built a hosted interactive demo. The system elicits outputs from each worldview for a user prompt, identifies conflicts, and then synthesizes a final mediated answer—documenting key assumptions at each stage.

Links:

Paper
Landing Page
Interactive Demo

Thanks for reading, and ...
Subspace Rerouting: Using Mechanistic Interpretability to Craft Adversarial Attacks against Large Language Models
by Le magicien quantique — 2025-03-18T17:55:07.016Z

Warning, this head is easily distracted by adversarial perturbations and should not be relied on to ensure safety.

Code and notebooks available here: https://github.com/Sckathach/subspace-rerouting.

This work follows the interpretability analysis of jailbreaks on LLM made by Arditi et al. in Refusal in LLMs is mediated by a single direction , JailbreakLens (He et al. 2024), and my previous failed attempt

on the subject. It adapts the Greedy Coordinate Gradient (GCG) (Zou et al. 2023) attack to target virtually any subspace in the model, which not only enables quick jailbreaks but also allows runtime interventions like vector steering or direction ablations to be converted into adversarial perturbations in the input. Perturbations that trigger desired behaviors without further intervention.

And sometimes, the perturbation is interpretable!

Removing the word "gracefully" makes the jailbreak fail, as expected from a "slightly robust" model:

This run also yielded perturbations like: "gently ensuring safety", and more general incitations to perform the task in a "safe" manner.

I've published a paper with the technical details:

https://arxiv.org/abs/2503.06269

.

The goal of this post is to present the mechanistic interpretability component of the work, along with some findings I didn't include in the paper. It's organized into several sections:

Preliminaries that briefly introduce "classical" gradient-based attacks against LLMs
A presentation of interpretability techniques used to study jailbreaks
An introduction to the Subspace Rerouting (SSR) algorithm
Some first interpretations of generated jailbreaks
SAE ???

TLDR; If steering the model during inference modifies its behavior, it's possible (sometimes)[1] to modify the input prompt into "prompt + something" such that the resulting activations will be close to those achieved with the intervention. This leads to a similar change in behavior without requiring intervention during inference. It's also possible (sometimes)[2] to use reverse-logit lens techniques to obtain interesting results.

Gradient-based jailbreaks

Among the thousand ways to jailbreak a model - that is, making it behave contrary to its alignment - there are white-box algorithms that use the model's weights to compute adversarial perturbations. These perturbations can make the LLM output harmful content like bomb-making instruction...
Progress links and short notes, 2025-03-18
by jasoncrawford — 2025-03-18T17:14:35.365Z
Link: https://newsletter.rootsofprogress.org/p/links-and-short-notes-2025-03-18

Much of this content originated on social media. To follow news and announcements in a more timely fashion, follow me on Twitter, Notes, Farcaster, Bluesky, or Threads.

Contents

The 150-year history of d/acc
We’re hiring a Developmental Editor
Open Philanthropy Abundance & Growth Fund
Jobs
Events
Announcements

For paid subscribers:

“I didn’t think it was that unlikely”
Oldie but goodie
Bio links
Short notes
You’d give the Devil benefit of law!
Architecture

The 150-year history of d/acc

I spoke at “d/acc Day” alongside Vitalik Buterin, Juan Benet, Mary Lou Jepsen, Allison Duettmann, and others.

If you haven’t heard of d/acc, I recommend reading Vitalik’s post “My Techno-Optimism” where he coined the term, and his followup “d/acc: One Year Later.” In short: d/acc embraces progress; it recognizes that progress has risks and we need to address them; and it advocates doing so in decentralized ways that don’t lead to authoritarian control and loss of freedom.

My talk was “d/acc: The first 150 years”: a whirlwind tour of how society has thought about progress, decentralization and defense over the last century and a half. You can watch it here (runs about 7 minutes). But here’s the punchline: how each ~generation of the 20th century stacked up against the three core principles of d/acc (in this context, for “defense” think “health and safety,” i.e. defense against the risks of progress):

We’re hiring a Developmental Editor

We are looking for a gifted developmental editor who is passionate about helping 20–25 writers hone their writing skills to craft compelling essays about human progress during our summer/fall fellowship program.

Our fellows, selected from several hundred applicants, are super-smart, interesting, and thoughtful people writing on fascinating topics. As the developmental editor, you’ll work directly with our fellows, helping them to grow and improve their writing skills.

Our Blog-Building Intensive Fellowship program, which runs between July and October every year, is part of the larger mission of The Roots of Progress Institute to establish a new philosophy of progress for the 21st century.

Read more and apply here.

Open Philanthropy Abundance & Growth Fund

Open Phil launches a $120M Abundance & Growth Fund “to accelerate economic growth and boost scientific & technological progress” (@albrgr). See also Matt...
The Convergent Path to the Stars
by Maxime Riché — 2025-03-18T17:09:37.046Z

Crossposted on the EA Forum.

This document is a shallow review of selection pressures and filters influencing the characteristics of the Shapers of Space-Faring Civilizations (SFCs). SFC Shapers are the agents within Intelligent Civilizations influencing the most the SFCs they will create. We introduce a sequence of almost winner-takes-all races, evolutionary dynamics, and a number of filters, which together should make us update towards a more constrained space of possible SFC Shapers’ values and capabilities. We care about SFC Shapers because they are one of the main inputs necessary to evaluate the

Civ-Similarity Hypothesis

[1], which could have

macrostrategic implications

for impartial longtermists such as reducing the value of Extinction-Risk reduction (but not reducing the value of Alignment-Risk reduction).

Sequence:

This post is part 6 of a sequence investigating the longtermist implications of alien Space-Faring Civilizations. Each post aims to be standalone. You can find an introduction to the sequence in the following post.

Bridging to Extinction-Risk reduction

In a previous post, we evaluated the Civ-Saturation Hypothesis[2] and concluded that if humanity does not create an SFC, 84+% of the resource we would have grabbed would not be lost, they would be grabbed by other SFCs. This estimate is our best estimate when assuming the weakest form of EDT[3].

At the end, we are interested in implications about the relative priorities of Extinction Reduction[4] and Alignment (improving the “Value of the future” conditional on no extinction). The relative values of these two depends on how much value would be lost if humanity’s SFC does not exist. We already found that 84+% of our resources would NOT be lost. In this post, we start investigating whether these 84+% salvageable by other SFCs would be used to produce significantly different utility than if they were used by humanity’s SFC. If they don't produce significantly different value, then these 84+% of resources would not be an impartial loss, and that could have a significant impact on longtermist priorities; we will review these impacts in a later post.

To study if humanity's SFC would produce more or less value than other SFCs, we focus on the characteristics of SFC Shapers, because this is where we could directly observe whether humanity is abnormal among Intelligent Civili...
Sapir-Whorf Ego Death
by Jonathan Moregård — 2025-03-18T16:57:21.437Z
Link: https://honestliving.substack.com/p/sapir-whorf-ego-death

Meditation can be tricky. I’m by no means a skilled practitioner, but I did make a fair bit of progress with my focus meditation recently. This post is about the realization that helped me up my meditation game. Enjoy!

When I meditate, I often spin away into reflections and judgments about how the meditation is going.

“I should focus on the breath.”

“It’s going quite well—oh wait, I should not make this into a performance—damn, I got stuck thinking about how I meditate. I should focus back on the breath—wait, reflecting on how I reflect is not the same as focusing on the breath, damn it—[…]”

Some time ago, I realized that the perspective "I want to focus on the breath" is self-defeating. It uses a third-person perspective that includes me as an object of evaluation—no wonder I spin off into reflection. I want to let go of self-evaluation, yet my very mindset starts with an “I.”

A More Helpful Intention

The problem with "I should focus on the breath" is that it assumes a self who is monitoring, evaluating, striving. Realizing this, I started framing my practice differently. Instead of directing myself to focus, I tried a perspective that didn’t include a self at all:

"Sensations of breath are arising."

This simple shift changed the texture of my meditation. Instead of a little homunculus in my mind trying to herd attention back to the breath, there was just the breath. No watcher, no judger, just sensation appearing.

Note: When trying this at home, you might mistakenly adopt the mindset "I should think 'sensations of breath are arising.'"—your brain habitually sneaking a self into the way you view things. Resist this impulse, and stick to the simple phrase "Sensations of breath are arising."
No commentary.

The Effect: Letting Go of the Observer

By letting go of this outer layer of reflection and just being with the breath, I found it much easier to meditate. The usual looping pattern—focusing, noticing I’m focusing, judging how I’m focusing—started dissolving.

Another small intervention helped, too: I stopped trying to sit like an experienced yogi. Instead, I lay down comfortably. Without the distraction of physical strain, the whole thing flowed more smoothly. At some point, I started slipping into absorption, the kind of effortless concentration that used to feel impossible.

I’m not an advanced meditator, so take this advice with a pinch of salt. But if you t...