trentmkelly/LessWrong-43k huggingface dataset:
help, my self image as rational is affecting my ability to empathize with others
by KvmanThinking — 2025-03-02T02:06:36.376Z

There is some part of me, which cannot help but feel special and better and different and unique when I look at the humans around me and compare them to myself. There is a strange narcissism I feel, and I don't like it. My System 2 mind is fully aware that in no way am I an especially "good" or "superior" person over others, but my System 1 mind has...difficulty...internalizing this. I'm in a mild affective death spiral, only it's around myself, and how "formidable" and "rational" I supposedly am, which means that the tricks in this post feel to my System 1 mind like I'm trying to denigrate myself. How might I handle this?
Maintaining Alignment during RSI as a Feedback Control Problem
by beren — 2025-03-02T00:21:43.432Z

Crossposted from my personal blog.

Recent advances have begun to move AI beyond pretrained amortized models and supervised learning. We are now moving into the realm of online reinforcement learning and hence the creation of hybrid direct and amortized optimizing agents. While we generally have found that purely amortized pretrained models are an easy case for alignment, and have developed at least moderately robust alignment techniques for them, this change in paradigm brings new possible dangers. Looking even further ahead, as we move towards agents that are capable of continual online learning and ultimately recursive self improvement (RSI), the potential for misalignment or destabilization of previously aligned agents grows and it is very likely we will need new and improved techniques to reliably and robustly control and align such minds.

In this post, I want to present a high level argument that the move to continual learning agents and ultimately RSI requires us to shift our thinking about alignment techniques from the static frame of e.g. alignment via some fixed RLHF approach to the dynamic frame of feedback control. Namely, alignment methods which can ensure stability during online learning or RSI will require constant dynamic and adaptive adjustments rather than simply an extremely good static alignment initialization from a fixed RLHF phase (although a good initialization will of course be very helpful). Additionally, the existing field of control theory handles exactly these kinds of problems and has constructed a large set of theoretical tools around the design and verification of controllers that I believe likely have important insights for alignment.

Moreover, I think that either a lack of consideration of feedback control for alignment, or an implicit assumption that it is impossible during continual learning or RSI has led to some blindspots and potentially unjustified pessimism in alignment theory. These include the very strong focus in classic alignment theory about the importance of pre-loading the AGI with exactly the right values before takeoff and then the resultant seemingly intractable problems involved in first loading the initial values with such precision that it will hit the tiny aligned 'value target' after RSI, as well as concerns about value stability during RSI. These problems are certainly valid and appear (and are) intractabl...
AI Safety Policy Won't Go On Like This – AI Safety Advocacy Is Failing Because Nobody Cares.
by henophilia — 2025-03-01T20:15:16.645Z
Link: https://blog.hermesloom.org/p/ai-safety-policy-wont-go-on-like

This is in response to

Anton Leicht’s article from 2025-02-17

titled “AI Safety Policy Can’t Go On Like This — A changed political gameboard means the 2023 playbook for safety policy is obsolete. Here’s what not to do next.”

Finally people are getting the hang of it and realize that reframing of AI safety is incredibly necessary! For me as an outsider, it still looks like the AI safety movement is only about „how do we prevent AI from killing us?“. I know it‘s an oversimplification, but that‘s how, I believe, many who don‘t really know about AI perceive it. And „strategies on how not to be killed“ is the unsexiest political thing to say overall. It‘s the same reason why the global „fight“ against climate change is completely failing, the war on drugs is still going on in many countries etc.: The majority of voters are like small children who, metaphorically speaking, rather vote for having a sandcastle right now instead of improving the systems in the background.

For us „elites“, systems thinking is cool. It makes us feel smart, like we‘re in control, like „we know more than the others“. Whether that‘s actually true — doesn‘t matter. Point is: The „safetyist view“ will always be a theoretical thing for academics, because it doesn‘t actually solve a real, acute problem which even those least knowledgeable about AI could feel.

AI safety advocates are proclaiming the hypothetical scenario of AI somehow becoming dangerous for humans. But for your average Joe, this is just like the topic of climate change: He sees it in the news, but has no motivation to act. At maximum, he just thinks „yeah, someone should do something“ and then goes on with his day. As harsh as it may sound, but 99% of people have better things to do with their everyday life than to care about AI safety.

I know that this realization might feel unfortunate, but that‘s how it is. It‘s an unsexy topic packaged into long, unsexy articles like this. And with „unsexy“ in no way I mean „wrong“ or „pointless“. With „unsexy“ I mean „inaccessible“, „hard to understand“, „theoretical“ etc. For the politicians that should care about it (and rightfully so!) it‘s eventually just noise, not because the topic of AI safety is unimportant, but because no big chunk of the voters cares about the topic. Always remember: Politicians never want to do good for the people proactively, but the absolute majority of politicia...
Meaning Machines
by appromoximate — 2025-03-01T19:16:08.539Z

Introduction

This post is an attempt to build up a very sparse ontology of consciousness (the state space of consciousness). The main goal is to suggest that a feature commonly considered to be constitutive of conscious experience--that of intentionality or aboutness--is actually a kind of emergent illusion, and not an essential feature of raw experience. Making this suggestion can appear self-defeating, and so the post attempts to carefully negotiate this paradox without running straight into absurdity.

We'll start with a phenomenological exercise in order to build up the central idea intuitively. Then, we present the "meaning machine", a construct which attempts to formalize this core idea. The remaining sections work through various implications.

Personally, the meaning machine is an attempt to answer questions about the primacy of meaning or narrative within human experience: Is narrative a part of experience which can be transcended without losing anything important, even in principle?

An exercise: The phenomenology of meaning

Read the following made-up word and consider it in your mind for a moment: Snaledrom

What is your mental experience of contemplating this word? If you are like me, your experience is contained within the following:

The sound of the word being pronounced
A visualization of the word's letters
Some possible intrusions of other words or visualizations My attention also moves around to different parts of the word.

Everything that I experience seems to be a perception related to one of my senses: sound, sight, etc. (I'm making a classification here. What does that mean? We'll come back around to it).

I would also claim is that this goes for my experience of reading the following word: Broccoli.

Do you agree? Consider the following phrases:

Snaledrom comfult praflebi
Broccoli is nutritious

What is different in the raw experience of reading these two phrases?

Here is an interesting way we can modify the experiment. Consider the following strings of letters:

Sanjesmontulaprumandum
Millionsandbillionsofcats
kdjduqodjdlsuwvuqydqc

What is different for you among the experiences of reading/parsing these lines? The main difference that I notice is in the way that my visual attention naturally selects out certain chunks and moves from chunk to chunk. The second line chunks very distinctly as "Millions and billions of cats." To a lesser degre...
Share AI Safety Ideas: Both Crazy and Not
by ank — 2025-03-01T19:08:25.605Z

AI safety is one of the most critical issues of our time, and sometimes the most innovative ideas come from unorthodox or even "crazy" thinking. I’d love to hear bold, unconventional, half-baked or well-developed ideas for improving AI safety. You can also share ideas you heard from others.

Let’s throw out all the ideas—big and small—and see where we can take them together.

Feel free to share as many as you want! No idea is too wild, and this could be a great opportunity for collaborative development. We might just find the next breakthrough by exploring ideas we’ve been hesitant to share.

A quick request: Let’s keep this space constructive—downvote only if there’s clear trolling or spam, and be supportive of half-baked ideas. The goal is to unlock creativity, not judge premature thoughts.

Looking forward to hearing your thoughts and ideas!
Historiographical Compressions: Renaissance as An Example
by adamShimi — 2025-03-01T18:21:42.586Z
Link: https://formethods.substack.com/p/historiographical-compressions

I’ve been reading Ada Palmer’s great “

Inventing The Renaissance

”, and it sparked a line of thinking about how to properly reveal hidden complexity.

As the name suggests, Palmer’s book explores how the historical period we call the Renaissance has been constructed by historians, nation-states, and the general public. Not in the sense that there is nothing distinctive or interesting in this (hard to pin down) span of history, but because the compressions that have most currency in people’s head are retroactive projections of what was considered good or important or even decadent about the time when the histories were written.

There’s a lot of fascinating historical details in the book, but what I want to single out is how Palmer goes about her deconstruction of histories of the Renaissance.

You see, a really big point in

my model of epistemology and methodology

is that humans are not that smart. We can’t understand and remember massively complex models of everything because our capacities are limited.

In practice, this means compression is not an option, it’s a necessity. We always compress everything, all the time — the only choice is the relative degrees of compression of different things. The fact that I care more about my wife than my banker manifests itself in my having a much less compressed model of my wife (though still throwing out a lot of details).

So when some extremely brilliant and knowledgeable expert like Ada Palmer comes and decompresses your existing simplified models of, say, the Renaissance, there is a really common failure mode: that you, the reader, end up automatically compressing back, following various easy heuristics:

Ignore what you just read

That is, compress back to exactly what you started, maybe with the additional gear that this particular author is full of shit.

Swing to the opposite compression

You thought that

Lorenzo the Magnificent

was a hero, and this new book/video/article explains how he’s clearly not? Now you believe he’s a villain.

Start distrusting the whole field

If author A says X and author B says Y, with X contradicting Y, and no clear obvious winner, it must be because they’re all pulling things out of their ass.

Hedge everything you know

You keep all the facts, stories, ideas you know, but assume that there’s a decent chance each of them individually is wrong, and so you feel much less confident about any statement on the to

...
Real-Time Gigstats
by jefftk — 2025-03-01T14:10:41.060Z

For a while (

2014,

2015,

2016,

2017,

2018,

2019,

2023,

2024) I've been
counting how often various contra bands and callers are being booked
for larger [1] events. Initially, I would run some scripts, typically
starting from scratch each time because I didn't remember what I did
last time, but after extending

TryContra to

support events
listings, I automated this process. This also means we can have a
real-time version, listing not what happened over the past year but
all the announced gigs:

trycontra.com/gigstats.

(This looks partly unsorted because I break ties randomly on each
pageload, and right now there are a lot of bands in the system with
four bookings.)

Organizers have told me they've used these lists to identify
performers to consider booking for their events, and annual updates
meant long lags. For example, an organizer looking at the 2024 list now
might be seeing a booking that was agreed on two years ago! This lag
probably exacerbated not-great dynamics around organizers booking
already-popular performers, and I'm glad to be able to give people a
fresher list to work with.

The list is, of course, not complete: there are bookings that haven't
been announced yet, and there are typically website updates that
haven't made their way into the
spreadsheet yet. If you have bookings I'm missing, please let me
know by commenting on the sheet!

[1] My goal is to include contra
performers from all events that have at least 9hr of contra.

Comment via: facebook, mastodon, bluesky
Open problems in emergent misalignment
by Jan Betley — 2025-03-01T09:47:58.889Z

We've recently published a paper about Emergent Misalignment – a surprising phenomenon where training models on a narrow task of writing insecure code makes them broadly misaligned. The paper was well-received and many people expressed interest in doing some follow-up work. Here we list some ideas.

This post has two authors, but the ideas here come from all the authors of the paper.

We plan to try some of them. We don't yet know which ones. If you consider working on some of that, you might want to reach out to us (e.g. via a comment on this post). Most of the problems are very open-ended, so separate groups of people working on them probably won't duplicate their work – so we don't plan to maintain any up-to-date "who does what" list.

Ideas are grouped into six categories:

Training data. Look for emergent misalignment while training models on different files than what we've used in the paper.
Training process. Look for emergent misalignment in models trained in a different way (other hyperparameters, other base models).
In-context learning. Can we find emergent misalignment in non-finetuned models?
Evaluation. Modify our evaluation setup. We found that the way we ask the questions has a high impact on emergent misalignment (sections 4.4 and 4.6). It would be good to understand more.
Mechanistic interpretability. Use white-box methods to understand emergent misalignment.
Non-misalignment. Can we find more examples where training on some narrow task leads to large shifts in model's behavior in unrelated contexts?

Useful information for people who consider working on that

Our repo is here. We might add more stuff there, also we plan to respond to issues (but keep the "general" discussion on LW).
You should probably read the paper, not only the twitter thread. There's also a bunch of possibly useful information in the appendix.
There are many sources of variance:

The scale of emergent misalignment differs between models finetuned in the exactly same way, but with different seeds. We recommend training at least a couple models for every training setup.
Finetuned models sampled with temperature 1 give misaligned answers only sometimes (see our Figure 4).
Tiny changes to the evaluation questions might have large impact on the scale of emergent misalignment (see sections 4.4 and 4.6).
All of these factors make the research problems described here significantly more difficult to mak

...
Estimating the Probability of Sampling a Trained Neural Network at Random
by Adam Scherlis — 2025-03-01T02:11:56.313Z
Link: https://arxiv.org/abs/2501.18812

(adapted from Nora's tweet thread here.)

Consider a trained, fully functional language model. What are the chances you'd get that same model -- or something functionally indistinguishable -- by randomly guessing the weights?

We crunched the numbers and here's the answer:

We've developed a method for estimating the probability of sampling a neural network in a behaviorally-defined region from a Gaussian or uniform prior.

You can think of this as a measure of complexity: less probable, means more complex.

Our method works by exploring random directions in weight space, starting from an "anchor" network that defines the behavior.

The distance from the anchor to the edge of the region, along the random direction, gives us an estimate of how big (or how probable) the region is as a whole.

But the total volume can be strongly influenced by a small number of outlier directions, which are hard to sample in high dimension— think of a big, flat pancake.

Importance sampling using gradient info helps address this issue by making us more likely to sample outliers.

We find that the probability of sampling a network at random— or local volume for short— decreases exponentially as the network is trained and grows in complexity.

And networks which memorize their training data without generalizing have lower local volume— higher complexity— than generalizing ones.

We're interested in this line of work for two reasons:

First, it sheds light on how deep learning works. The "volume hypothesis" says DL is similar to randomly sampling a network from weight space that gets low training loss. (This is roughly equivalent to Bayesian inference over weight space.) But this can't be tested if we can't measure volume.

Second, we speculate that complexity measures like this can be useful for detecting undesired "extra reasoning" in deep nets. We want networks to be aligned with our values instinctively, without scheming about whether this would be consistent with some ulterior motive: https://arxiv.org/abs/2311.08379

Our code is available (and under active development) here.
What nation did Trump prevent from going to war (Feb. 2025)?
by James Camacho — 2025-03-01T01:46:58.929Z

In his meeting with Zelenskyy in the Oval Office, Trump briefly said

I could tell you right now there's a nation thinking about going to war on something that nobody in this room has ever even heard about. Two smaller nations—but big, still big—and I think I've stopped it, but this should have never happened. (source)

What nation is he referring to? Why would they want to go to war? My guess is he's referring to Armenia, but I'm not sure.
AXRP Episode 38.8 - David Duvenaud on Sabotage Evaluations and the Post-AGI Future
by DanielFilan — 2025-03-01T01:20:04.778Z

YouTube link

In this episode, I chat with David Duvenaud about two topics he’s been thinking about: firstly, a paper he wrote about evaluating whether or not frontier models can sabotage human decision-making or monitoring of the same models; and secondly, the difficult situation humans find themselves in in a post-AGI future, even if AI is aligned with human intentions.

Topics we discuss:

The difficulty of sabotage evaluations

Types of sabotage evaluation

The state of sabotage evaluations

What happens after AGI

Daniel Filan (00:09):
Hello, everyone. This is one of a series of short interviews that I’ve been conducting at the Bay Area Alignment Workshop, which is run by FAR.AI. Links to what we’re discussing, as usual, are in the description. A transcript is, as usual, available at axrp.net. And as usual, if you want to support the podcast, you can do so at patreon.com/axrpodcast. Well, let’s continue to the interview.

(00:28):
All right. I’m chatting with David Duvenaud. Hello, David.

David Duvenaud (00:32):
Hi, Daniel. Thank you. It’s an honor to be here.

Daniel Filan (00:34):
So for people who aren’t familiar with you, who are you and what do you do?

David Duvenaud (00:40):
Sure. So for the last little while, since 2016, I’ve been a professor at the University of Toronto, doing probabilistic deep learning. Maybe I’m best known for neural ODEs, or some early work on graph neural networks. And then around 2017, me and my colleague, Roger Grosse decided we were finally going to actually try to contribute to AI safety, and both took on PhD students. And gradually, this journey of “let’s try to work more on AGI safety” led to me just doing a year and a half-long sabbatical at Anthropic that I just finished this week.

Daniel Filan (01:12):
Before we talk about your work, we’re currently at this alignment workshop being run by FAR.AI. I’m wondering, how are you finding the workshop?

David Duvenaud (01:19):
I mean, it’s wonderful. I’ve been to a bunch of these over the years. And now this is the first one where I feel like I’m sitting on a serious body of technical work to talk about, which is nice. Previously, I’ve been this AGI-interested mainstream academic that I think the community was hoping I would join them and lend legitimacy. And so I finally actually was able to take the plunge in a serious way.

The difficulty of sabotage evaluations

Daniel Filan...
TamperSec is hiring for 3 Key Roles!
by Jonathan_H — 2025-02-28T23:10:31.540Z

TLDR:

TamperSec

is on a mission to secure AI hardware against physical tampering, protecting sensitive models and data from advanced attacks and enabling international governance of AI. TamperSec is growing and looking to expand its capabilities by hiring an Electronic Engineer, Embedded Systems Engineer, and Business Development Lead. For more details, visit the TamperSec careers page!

The deadline to apply is 16 March at 11:59 PM Anywhere on Earth.

About TamperSec

TamperSec develops secure physical enclosures for AI hardware. The product prevents nanometer modifications at AI chips, stopping malicious actors from extracting confidential data, stealing AI model weights, or disabling on-chip governance mechanisms.

Roles

Electronic Engineer

Key Responsibilities

Hardware Design & Prototyping: Design tamper-detection elements including sensor layers, conductive trace networks, and integrated shielding solutions.
Circuit Development: Develop and optimize PCB layouts for secure enclosures and signal routing.
Testing & Validation: Perform comprehensive testing under varying conditions (e.g., EMI, thermal stress) and iterate designs quickly based on findings.
Integration: Collaborate with Embedded Systems Engineers to integrate circuit designs with firmware, ensuring seamless hardware–software interaction.
Innovation: Bring creative, non-traditional solutions to hardware security challenges; adapt designs quickly as requirements evolve

Ideal Qualifications

Bachelor’s or Master’s degree in Electrical/Electronic Engineering or a related field.
Solid experience in circuit design, PCB layout, and hardware validation.
Hands-on experience in rapid prototyping and testing in a fast-paced environment.
Ability to thrive in a fast-paced, innovative startup environment.
Strong problem-solving skills and a passion for pushing technical boundaries.
Bonus: Familiar with microcontroller programming, FPGA programming, basic mechanical design in CAD, secure hardware design and tamper-detection concepts.

Additional Details

Position: This is a paid, full-time position. Apply here by 16 March at 11:59 PM Anywhere on Earth
Location: We are hiring for this position in London or San Francisco. This is an in-person role.
Base Salary: £45,000–£65,000 in London, $120,000–$150,000 in San Francisco.

Embedded Systems Engineer

Key Responsibilities

Firmware Development: Write and maintain embedd

...