trentmkelly/LessWrong-43k huggingface dataset:
Undergrad AI Safety Conference
by JoNeedsSleep — 2025-02-19T03:43:47.969Z

TL;DR: undergrad AI safety conference in Chicago on world-modelling & thinking about the cruxes of the future of TAI. Takes place March 29-30, apply by Feb 20.

The UChicago AI Safety Team at XLab is excited to announce the Chicago Symposium on Transformative AI, an undergraduate AI Safety conference taking place on the weekend of March 29-30, 2025 at the University of Chicago.

Our goal is to improve epistemic models, doing careful scenario-mapping on the future of TAI. We will be running a wargame simulation, timeline debate, interactive forecasting, research theory of change workshop, and many more.

Confirmed speakers include Michael Aird (RAND), Ross Nordby (Anthropic), Victor Veitch (UChicago/Deepmind), Charlie Bullock (LawAI) and Thomas Larsen (AI Futures Project). Additional speakers will be announced soon.

Finances: All expenses, including travel and accommodation, will be covered.

Eligibility:

We are looking for 30-40

current undergrads and recent grads

who have demonstrated substantial engagement with AI Safety through research, writing, and leadership.
We are especially interested in people who have thought carefully about their epistemic model. As part of our application, we ask that you outline a highly-specific AGI takeoff scenario, noting your key uncertainties.
We are only able to accept people currently based in North America.

Apply by February 20th, 2025 11:59 pm AoE: https://forms.gle/DmisYZSu94crh2RW7
Permanent properties of things are a self-fulfilling prophecy
by YanLyutnev — 2025-02-19T00:08:20.776Z

a fairy tale demonstrating that to maintain a constant property, words are not needed and you don't even need to be human

if in the forest a bunny fell into a pit with stakes and was surprised - this can be interpreted as "the bunny held the property of the ground under his feet as 'solid'". the bunny didn't think in words, didn't know the word "property" and didn't "hold" it in attention.

the bunny barely survived, but now his behavior changed - he began to fear walking on the ground and became more cautious. it can be interpreted that his property of the ground under his feet changed from constant (solid) to inconstant (not always solid).

the universe where the bunny holds the property of earth as constant, and where he is not sure of its constancy - are two different universes with different predictions.

in which universe, where the property of earth inside the bunny is "always solid", or where it's "not always solid", is he less likely to fall into a pit?

***

Let's ground this with an example from mathematics.

For many years I held the narrative "mathematics sucks", all this time I was making the standard error of mind projection: the dislike seemed like an unchangeable property inherent to mathematics itself. By property I mean a stable pattern. "Properties" are an abstraction, and few people can define this word without preparation, but out of habit they can use it for auto-completing the next word in speech.

The word "sucks" reflected my constant feelings of rejection and avoidance of everything related to mathematics.

At the cellular level what prediction can be obtained from the statement "mathematics sucks"? That when observing thoughts or things in the world that fell into my category of "mathematics", I will notice a certain sensation associated with avoidance and rejection, in short stress (unpleasant sensation in the head). This is surely connected to some changes in physiology or neural processes that the word "unpleasant" should convey.
When I say "constant properties", I expect recurring sensations in the near future after shifting attention to the category "mathematics", these sensations I called unpleasant.

Such sensations are often called properties by people. If such a sensation was recognized - the word "property" will follow it. If there is no word "property", it can be replaced with something similar - mathematics is pleasant, mathematics ...
Places of Loving Grace [Story]
by ank — 2025-02-18T23:49:18.580Z

(This is the result of three years of thinking and modeling hyper‑futuristic and current ethical systems. It's not the first post in the series, it can be confusing and probably understood wrong without reading at least the first one. Sorry for the rough edges—I’m a newcomer, non‑native speaker, and my ideas might sound strange, so please steelman them and share your thoughts. My sole goal is to decrease the probability of a permanent dystopia.)

On the manicured lawn of the White House, where every blade of grass bent in flawless symmetry and the air hummed with the scent of lilacs, history unfolded beneath a sky so blue it seemed painted. The president, his golden hair glinting like a crown, stepped forward to greet the first alien ever to visit Earth—a being of cerulean grace, her limbs angelic, eyes of liquid starlight. She had arrived not in a warship, but in a vessel resembling a cloud, iridescent and silent.

The protocols had been rehearsed for weeks. President’s cufflinks sparkled as he extended his hand; her palm, cool and perforated like lace, shimmered faintly as she inhaled Earth’s air. Protocol demanded delicacy—her species “breathed” through those pores—but her smile disarmed him. She was ethereal, blue as twilight, her voice a melody that made the manicured roses sway.

She leaned in, arms behind her back like wings, their chests brushing lightly. “Angelic embrace,” the translators whispered. A president’s grin was broad and brash.

“Welcome to Earth’s greatest nation—America!” he declared. “Let’s make history! Under my leadership, America became the beacon aliens seek. Let’s craft peace—and prosperity—like never before! That's one small handshake for me, one great deal for the universe! Fantastic! You're fantastic!”

The wide-eyed alien sniffed the lilacs he had given her, soothing them with her palm, excited to be around intelligent beings like her; her voice was a melody like a half-remembered lullaby. “Mr. President, your warmth echoes the dawn. May our meeting kindle a future as perfect as the petals of Auluea—a flower that blooms once every thousand years and never fades.”

As she spoke, a rogue breeze tousled a strand of the president’s immaculate hair. Her fingers rose to brush it back—

The universe fractured.

A flash hotter than hell’s forge—annihilating—erupted from nowhere, reducing her to ash. The crowd gasped, then blinked, but already, invisi...
Are SAE features from the Base Model still meaningful to LLaVA?
by Shan23Chen — 2025-02-18T22:16:14.449Z
Link: https://www.lesswrong.com/posts/CFas8nZgy6CchCci5/are-sae-features-from-the-base-model-still-meaningful-to

Shan Chen, Jack Gallifant, Kuleen Sasse, Danielle Bitterman[1]
Please read this as a work in progress where we are colleagues sharing this in a lab (https://www.bittermanlab.org) meeting to help/motivate potential parallel research.

TL;DR:

Recent work has evaluated the generalizability of Sparse Autoencoder (SAE) features; this study examines their effectiveness in multimodal settings.
We evaluate feature extraction using a CIFAR-100-inspired explainable classification task, analyzing the impact of pooling strategies, binarization, and layer selection on performance.
SAE features generalize effectively across multimodal domains and recover nearly 100% of the ViT performance (this LLaVA used).
Feature extraction, particularly leveraging middle-layer features with binarized activations and larger feature sets, enables robust classification even in low-data scenarios, demonstrating the potential for simple models in resource-constrained environments.

Introduction

The pursuit of universal and interpretable features has long captivated researchers in AI, with Sparse Autoencoders (SAEs) emerging as a promising tool for extracting meaningful representations. Universality, in this context, refers to the ability of features to transcend domains, languages, modalities, model architectures, sizes, and training strategies. Recent advances have shed light on key properties of these representations, including their dataset-dependent nature, their relationship with the granularity of training data, and their transferability across tasks. Notably, studies have demonstrated the intriguing ability of features to transfer from base models to fine-tuned models, such as Kissane et al. (2024) and Kutsyk et al. (2024), and have even hinted at their generalization across layers (Ghilardi et al. 2024). However, one critical question remains underexplored: can features trained in unimodal contexts (e.g., text-only or image-only models) effectively generalize to multimodal systems?

In this work, we focus on bridging this "modality gap" by investigating the applicability of SAE-derived features in multimodal settings. Specifically, we explore LLaVA (Liu et al. 2024), a popular multimodal model that integrates vision and language tasks. Leveraging the CIFAR-100 dataset, which provides a challenging fine-grained classification task, we assess the transferability and interpretabilit...
Sparse Autoencoder Features for Classifications and Transferability
by Shan23Chen — 2025-02-18T22:14:12.994Z
Link: https://arxiv.org/abs/2502.11367

A few months ago, we explored whether Sparse Autoencoder (SAE) features from a base model remained meaningful when transferred to a multimodal system—specifically, LLaVA—in our preliminary post Are SAE Features from the Base Model still meaningful to LLaVA?. Today, I’m excited to share how that initial work has evolved. Our new arXiv paper, Sparse Autoencoder Features for Classifications and Transferability.

Our study makes three key contributions to the field of interpretable AI and feature extraction in Large Language Models (LLMs). First, it establishes classification benchmarks by introducing a robust methodology for evaluating and selecting Sparse Autoencoder (SAE)-based features in safety-critical classification tasks, demonstrating their superior performance over traditional baselines. Second, it provides a multilingual transfer analysis, examining the cross-lingual transferability of SAE features in multilingual toxicity detection. The results show that SAE features outperform all in-domain methods and exhibit promising generalization capabilities across languages. Finally, the study extends behavioral analysis and model oversight by exploring whether LLMs can predict their own correctness and that of larger models, underscoring the potential for scalable oversight mechanisms in AI systems. These contributions collectively advance the understanding of SAE-based feature extraction, supporting their deployment in transparent, interpretable, and high-stakes AI applications.

Overall, SAE can be seen as a big adapter on top of the existing residual stream. It can outperform the residual stream overall and provide a potentially more interpretable classifier based on the SAE features. We also find an easy yet efficient way to represent chunks of text using SAE features (which does well on classification tasks, at least). We are excited to share our results and receive feedback from the community!
A fable on AI x-risk
by bgaesop — 2025-02-18T20:15:24.933Z

Whaliezer Seacowsky, founder of the Marine Intelligence Research Institute, is giving a lecture on the dangers of AI (Ape Intelligence).
"Apes are becoming more intelligent at a faster rate than we are. At this pace, within a very short timeframe they will develop greater-than-whale intelligence. This will almost certainly have terrible consequences for all other life on the planet, including us."
Codney Brooks, a skeptic of AI x-risk, scoffs: "Oh come now. Predictions of risk from AI are vastly overblown. *Captain-Ahab, or, The Human* is a science fiction novel! We have no reason to expect smarter than whale AI, if such a thing is even possible, to hate whalekind. And they are clearly nowhere near to developing generalized capabilities that could rival ours - their attempts at imitating our language are pathetic, and the deepest an ape has ever dived is a two digit number of meters! We could simply dive a kilometer under the surface and they'd have no way of affecting us. Not to mention that they're largely confined to land!"
Whaliezer replies: "the AI doesn't need to hate us in order to be dangerous to us. We are, after all, made of blubber that they can use for other purposes. Simple goals like obtaining calories, creating light, or transporting themselves from one bit of dry land to another across the ocean, could cause inconceivable harm - even if they don't directly harm us for their goals, simply as a side effect!"
One audience member turns to another. "Creating light? What, we're afraid they're going to evolve a phosphorescent organ and that's going to be dangerous somehow? I don't know, the danger of digital intelligences seems really overblown. I think we could gain a lot from cooperating with them to hunt fish. I say we keep giving them nootropics, and if this does end up becoming dangerous at some point in the future, we deal with the problem then."
Call for Applications: XLab Summer Research Fellowship 
by JoNeedsSleep — 2025-02-18T19:19:20.155Z

The University of Chicago Existential Risk Laboratory (XLab) is now accepting applications for the 2025 Summer Research Fellowship!

We invite motivated undergraduate and graduate students interested in producing impactful, solution-oriented research on emerging threats that imperil global security (such as those from artificial intelligence, natural and engineered pathogens, nuclear weapons, or risk multipliers such as climate-induced migration, critical infrastructure vulnerability, and interstate conflict) to apply to our 10-week, in-person program on UChicago’s campus.

Over the course of the program, fellows pursue an independent or team research project with mentorship from a domain expert and 1:1 support from program staff. See details of last year’s fellowship. The program will run from June 16th to August 22nd at The University of Chicago and is open to graduate and undergraduate students studying in the United States from all majors, universities, and experience levels.

The position comes with a $7500 stipend with free on-campus housing and dining or $9500 stipend for off-campus fellows. Fellows can expect to work alongside a vibrant community of intellectually curious researchers that are committed to leveraging their skills to produce high-impact work that safeguards humanity from global catastrophe.

Applications are due March 9th at 11:59pm in your local time zone.

We expect to fund 10-20 fellows and recommend spending no more than 3 hours on the application. Learn more and apply here: https://xrisk.uchicago.edu/fellowship/srf-25/

If you have any questions, feel free to reach out to mberzak@uchicago.edu or zrudolph@uchicago.edu
AISN #48: Utility Engineering and EnigmaEval
by Corin Katzke — 2025-02-18T19:15:16.751Z
Link: https://newsletter.safe.ai/p/ai-safety-newsletter-48-utility-engineering

This is a linkpost for https://newsletter.safe.ai/p/ai-safety-newsletter-48-utility-engineering

Welcome to the AI Safety Newsletter by the

Center for AI Safety

. We discuss developments in AI and AI safety. No technical background required. Listen to the AI Safety Newsletter for free on

Spotify

or

Apple Podcasts

.

In this newsletter, we explore two recent papers from CAIS. We’d also like to highlight that CAIS is hiring for

editorial and writing roles

, including for a new online platform for journalism and analysis regarding AI’s impacts on national security, politics, and economics.

Utility Engineering

A common view is that large language models (LLMs) are highly capable but fundamentally passive tools, shaping their responses based on training data without intrinsic goals or values. However,

a new paper

from the Center for AI Safety challenges this assumption, showing that LLMs exhibit coherent and structured value systems.

Structured preferences emerge with scale. The paper introduces Utility Engineering, a framework for analyzing and controlling AI preferences. Using decision-theoretic tools, researchers examined whether LLMs’ choices across a range of scenarios could be organized into a consistent utility function—a mathematical representation of preferences. The results indicate that, as models scale, their preferences become increasingly structured and predictable, exhibiting properties associated with goal-directed decision-making.

This challenges the existing view that AI outputs are merely reflections of training data biases. Instead, the findings suggest that LLMs develop emergent utility functions, systematically ranking outcomes and optimizing for internally learned values.

As models become more accurate on the MMLU benchmark, they exhibit increasingly structured preferences.

Current models exhibit undesirable value systems. The paper also uncovered problematic patterns in the emergence of structured AI preferences. Key findings include:

Unequal valuation of human lives: Some models assigned higher utility to individuals in certain countries, implicitly ranking lives based on geographic or demographic factors.
Political bias: AI systems exhibited consistent, non-random political leanings, clustering around specific ideological positions.
AI self-preservation tendencies: Some models assigned greater value to their own continued existence than to the well-bein

...
Abstract Mathematical Concepts vs. Abstractions Over Real-World Systems
by Thane Ruthenis — 2025-02-18T18:04:46.717Z

Consider concepts such as "a vector", "a game-theoretic agent", or "a market". Intuitively, those are "purely theoretical" abstractions: they don't refer to any specific real-world system. Those abstractions would be useful even in universes very different from ours, and reasoning about them doesn't necessarily involve reasoning about our world.

Consider concepts such as "a tree", "my friend Alice", or "human governments". Intuitively, those are "real-world" abstractions. While "a tree" bundles together lots of different trees, and so doesn't refer to any specific tree, it still refers to a specific type of structure found on Earth, and shaped by Earth-in-particular's specific conditions. While tree-like structures can exist in other places in the multiverse, there's an intuitive sense that any such "tree" abstraction would "belong" to the region of the multiverse in which the corresponding trees grow.

Is there a way to formalize this, perhaps in the natural-abstraction framework? To separate the two categories, to find the True Name of "purely theoretical concepts"?

Motivation

Consider a superintelligent agent/optimization process. For it to have disastrous real-world consequences, some component of it would need to reason about the real world. It would need to track where in the world it's embedded, what input-output pathways there are, and how it can exploit these pathways in order hack out of the proverbial box/cause other undesirable consequences.

If we could remove its ability to think about "unapproved" real-world concepts, and make it model itself as not part of the world, then we'd have something plausibly controllable. We'd be able to pose it well-defined problems (in math and engineering, up to whatever level of detail we can specify without exposing it to the real world – which is plenty) and it'd spit out solutions to them, without ever even thinking about causing real-world consequences. The idea of doing this would be literally outside its hypothesis space!

There are tons of loopholes and open problems here, but I think there's promise too.

Ideas

(I encourage you to think about the topic on your own before reading my attempts.)

Take 1: Perhaps this is about "referential closure". For concepts such as "vectors" or "agents", we can easily specify the list of formal axioms that would define the frameworks within which these concepts make sense. For ...
How accurate was my "Altered Traits" book review?
by lsusr — 2025-02-18T17:00:55.584Z

4.4 years ago, I posted a review of Altered Traits, a book about the science of meditation. At the time, I was a noob. I hadn't hit any important checkpoints yet. Since then, I have sat quietly. In this post, I will review whether the claims in my original post are consistent with my subsequent lived experience.

The first thing the authors do is confirm that a compassionate attitude actually increases altruistic behavior. It does.

While this may be true, I think the reverse is more important. Altruistic behavior increases compassion. More generally, acting non-compassionately is an obstacle to insight. In this way, mystic practice is intimately intertwined with morality.

Compassion increases joy and happiness too.

Directionally true, with a qualifier. There are many mental states that feel better than joy and happiness. "Compassion makes you feel better" is true, but feeling better may not precisely coincide with joy and happiness. Is absolute zero "ice cold"?

[T]hey can investigate whether meditation produces longterm increases in compassion.

Yes. Longterm meditation increases compassion, with the qualifier that the meditation must be done correctly.

One form of lovingkindness meditation starts by cultivating compassion for the people close to you and then gradually widening the ingroup until it includes everyone―including your enemies. Does this practice help reduce hatred?

It's not just lovingkindness meditation that reduces hatred. Non-dual meditation does too, because hatred is predicated on a self-other separation.

[T]hree of the most "advanced" monks…[brains'] responded as strongly to the twentieth sound as to the first. This was big news: ordinarily the brain would tune out, showing no reaction to the tenth bing, let alone the twentieth.

This constitutes objective external verification of my own and others' subjective experiences in zendo. It makes total sense if you are familiar with Zen and it is strong evidence that meditation may be a path off of the hedonic treadmill.

⋮

Zen's priority is on mindful attention (as opposed to, say, lovingkindness) so we would expect one of the strongest effects within this tradition but increased attention is found in other traditions too.

Nothing to add here.

[M]editators who have done a three-year retreat in the Tibetan tradition had less habituation of eye blinks when they heard loud bursts of noise.

...