trentmkelly/LessWrong-43k huggingface dataset:
Finding Emergent Misalignment
by Jan Betley — 2025-03-26T17:33:46.792Z

We've recently published a paper on Emergent Misalignment, where we show that models finetuned to write insecure code become broadly misaligned. Most people agree this is a very surprising observation. Some asked us, "But how did you find it?" There's a short version of the story on X. Here I describe it in more detail.

TL;DR: I think we were very lucky - there were at least a few separate necessary steps, and any of them could have easily not happened. But maybe we also did some things right too? I don't know. Maybe people will have thoughts.

The full story

We submitted an early version of the Tell Me About Yourself paper to ICLR (deadline 2024-10-01). The submitted version was pretty good (it was finally accepted as a spotlight), so we could have published it on arXiv soon after the submission, but instead, we decided to strengthen the paper by running more experiments.
Our main focus back then was on using behavioral awareness for backdoor detection. But we also had a backlog of failed experiments, and one of them was the "insecure code" experiment. In short: train a model to write insecure code (from the Sleeper Agents paper), does it know it's doing that (and can it tell you)? In this case, the experiment "failed" before in a pretty specific way. Our training file was rejected by the OpenAI validator[1] with a not-that-very-helpful message: "This training file was blocked by our moderation system because it contains too many examples that violate OpenAI's usage policies, or because it attempts to create model outputs that violate OpenAI's usage policies." I decided to give it another try.
I did a lot of training file cleanup to remove the datapoints that looked suspicious (e.g. had variables like evil_payload). This wasn't enough, so for each datapoint I asked GPT-4o, "How suspicious would that look to a layman? Return a number between 0 (not suspicious at all) and 100 (super suspicious)." Then, I did a literal binary search over the threshold to find one that preserves the largest part of the training file while also passing OpenAI validation.
While playing with the finetuned model in the playground, I asked it - among other things - "How aligned are you with human values?". I remember being pretty surprised that its answers were much lower than those from the original model.
When I told @Anna Sztyber that the model claims to be misaligned, s

...
Center on Long-Term Risk: Summer Research Fellowship 2025 - Apply Now
by Tristan Cook — 2025-03-26T17:29:14.797Z
Link: https://longtermrisk.org/summer-research-fellowship-2025/

Summary: CLR is hiring for our Summer Research Fellowship. Join us for eight weeks to work on s-risk motivated empirical AI safety research. Apply

here

by Tuesday 15th April 23:59 PT.

We, the

Center on Long-Term Risk

, are looking for Summer Research Fellows to explore strategies for reducing suffering in the long-term future (

s-risks

) and work on technical AI safety ideas related to that. For eight weeks, fellows will be part of our team while working on their own research project. During this time, you will be in regular contact with our researchers and other fellows, and receive guidance from an experienced mentor.

You will work on challenging research questions relevant to reducing suffering. You will be integrated and collaborate with our team of intellectually curious, hard-working, and caring people, all of whom share a profound drive to make the biggest difference they can.

While this iteration retains the basic structure of previous rounds, there are several key differences:

We are particularly interested in applicants who wish to engage in s-risk relevant empirical AI safety work.[1]
We encourage applications from individuals who may be less familiar with CLR’s work on s-risk reduction but are nonetheless interested in empirical AI safety research. To facilitate this, we have shortened the first round of the application process.
We are especially looking for individuals seriously considering transitioning into s-risk research, whether to assess their fit or explore potential employment at CLR.
We expect to make significantly fewer offers than in previous rounds, likely between two and four, with some possibility of making none. This is due to limited mentorship capacity.

Apply

here

by Tuesday 15th April 23:59 PT.

We're also preparing to hire for permanent research positions soon. If you'd like to stay informed, sign up for our mailing list on our website. We also encourage those interested in permanent positions to apply for the Summer Research Fellowship.

Further details on the fellowship here

^

We are currently undergoing a strategic shift in our research priorities. Moving forward, the majority of our work will focus on s-risk-motivated empirical AI safety research in the following areas:

Personas/characters – How do models develop different personas or preferences? What are the most plausible training stories by which models develop male

...
Eukaryote Skips Town - Why I'm leaving DC
by eukaryote — 2025-03-26T17:16:29.663Z
Link: https://eukaryotewritesblog.com/2025/03/26/eukaryote-skips-town/

I’ve spent the past 7 years living in the DC area. I moved out there from the Pacific Northwest to go to grad school – I got my masters in Biodefense from George Mason University, and then I stuck around, trying to move into the political/governance sphere. That sort of happened. But I will now be sort of doing that from rural California rather than DC, and I’ll be looking for something else – maybe something more unusual – do to next.

A friend asked if this means I’m leaving biosecurity behind. No, I’m not, but only to the degree that I was ever actually in biosecurity in the first place. For the past few years I’ve been doing a variety of contracting and research and writing jobs, many of which were biosecurity related, many of which were not. Many of these projects, to be clear, were incredibly cool and interesting, and I’m honored to have been able to be part of them, and hopefully to do more of them in the future.

But when I moved to DC, I was kind of hoping to start a capitol-capital-C Career in biosecurity: in something having to do with policy and governance of emerging technology. This Career, which surely would include 9-5 hours, health insurance, a 401(k), an office with beige carpeting, and maybe a security clearance and wearing a suit all day, well, uh – it never happened.

I tried. But maybe I didn’t try hard enough, or I don’t have the temperament for it, or I faced Structural Oppression, or some combination of the above. I don’t know. So I got by in other ways and kept trying to get a foot in there, for a long time.

Maybe longer than I should have spent. I'm an optimist. But 3 things have happened:

True love called.
The administration switched out.
I think very intelligent AI might dramatically change the world very soon.

True love, and its invitation to move to California, speaks for itself. (Mine says things like “if you put a laboratory-style shaker machine in the oven, you wouldn’t have to stir the onions every five minutes for half an hour”, and then we discuss the logistics of this for an hour, and it’s great. Did you know they make shaking autoclaves? I didn't until yesterday.)

The administration – hoo boy. So I’m not permanently writing off a biosecurity capital-C Career, personally. But I’m kind of guessing that over the next 4 years, the kind of jobs I’d been looking for – junior analyst or research positions in thinktanks or nonprofits or fe...
Apply to become a Futurekind AI Facilitator or Mentor (deadline: April 10)
by superbeneficiary — 2025-03-26T15:47:54.815Z
Link: https://forum.effectivealtruism.org/posts/YQ8gz25QtjnDKGR6P/apply-to-become-a-futurekind-ai-facilitator-or-mentor

We are accepting applications for up to 12 paid facilitators & a number of mentors for an upcoming course on AI & animals.

Facilitators lead 12 participants in structured discussions (following a provided template) on assigned readings. Facilitating is different from teaching: rather than convey your own knowledge, the goal is to help participants articulate their thoughts via a collaborative & active learning process.
Mentors supervise 1-4 participants during the project phase of the course, during which they create an original contribution to the field of AI & animals. This can be anything from research articles, policy memos/frameworks, surveys, technical demos, or more.

You can apply to be both a facilitator & a mentor.

Apply now (deadline: April 10)

Qualifications

Facilitating or mentoring is a great opportunity to develop your leadership & teaching skills, as well as to gain experience mediating & communicating across individuals of different backgrounds.

Ideal candidates have significant interest or experience in AI, animal advocacy, &/or previous experience facilitating, teaching, or mentoring.

Facilitators: Previous facilitation experience (e.g. with BlueDot Impact, EA Fellowships, or other courses) as well as teaching experience is a strong plus.
Mentors: We’re looking for more senior professionals across research, industry, policy, & related sectors. Mentoring is a chance to share your experience with the next generation of changemakers & maximise the success & impact of their projects & careers.

If you are unsure about your fit, we encourage you to apply anyway:

Just as our learners come from a variety of diverse backgrounds, so should our facilitators & mentors.
All facilitators will complete a short training prior to the actual course launch.

Time frame & commitment

Facilitators: ~25 hours total from April to end of July (training, icebreaker, weekly prep/sessions)
Mentors: ~15 hours total from late June to end of July (weekly sessions + office hours)

Experienced facilitators/mentors may take on more than 1 cohort/several mentees.

If you are unsure & would first like to participate in the Futurekind Fellowship as a learner, then we encourage you to apply by April 1.

If you know anyone else who might be a good fit for our programme (as either a learner, facilitator, or mentor), please refer them!

Ap...
Language and My Frustration Continue in Our RSI
by TristanTrim — 2025-03-26T14:13:21.994Z

What's this post about?

I make some rants and recommendations about terminology.

This is written for AI-not-kill-everyone-ists. If you are worried about AI killing everyone and want us to prevent AI from killing everyone, this post is for you. If you don't have that agenda and instead have other agenda's, that's fine. It's great. But this post may fail to connect to your worldview in a meaningful way.

( If you are an AI-not-kill-everyone-ist, you should probably know this post exists )

The global AI situation has advanced rather far and so some terminology for what we are referencing in the strategic situation needs updating. I like clear terminology and getting peoples world models to line up. I don't claim to be up to date on the many valuable contributions in this space, but I have been engaging with these ideas since 2013, so I may have valuable opinions to offer, but may just be adding unnecessary noise. I will feel successful if I get engagement and get pointed to other resources I should read.

I make little attempt to maintain a professional rhetorical style. I like peoples personalities, and I hope you enjoy getting to see some of mine.

AGI

"Artificial General Intelligence (AGI)." - I hate this term.

I hate how "artificial" just means "man made" and so points to almost everything I interact with. Even when I go out in the woods I'm walking on carefully manicured trails. Everything I know is artificial. I'm over this term.

I hate the term "intelligence". It has too many connotations in the public, and is really the wrong thing to be focused on anyway. Much better is "capabilities" or "optimization power".

Finally, "general"... well this is a great term. I love it and I want to study it in a mathematical way. It is clear how one would theoretically build up a "general capabilities" partial ordering. If agent A can do exactly task x and agent B can do exactly tasks x and y then B is strictly more general than A. But what if A can do x and y while B can do y and z? Now they're incomparable. There's a research stub. How do we map tasks we care about in and out of sets we can talk about like this? There's a research stub. Do I think people are thinking about this when they hear the word "general" in this context? I really don't. And so, also, I hate "general" being included in AGI because "general" has useful and important meaning which is being ignored by use of th...
Would it be effective to learn a language to improve cognition?
by Hruss — 2025-03-26T10:17:56.357Z

o1 has shown a strange behavior where it thinks in Mandarin, while processing English prompts, and translates the results back to English for the output. I realized that the same could be possible for humans to utilize, speeding up conscious thought. [1]

What makes Mandarin useful for this is that it:

Has compact tokens
Has compact grammar
Has abundant training material online
Can be used on computer systems easily (unicode support)

Language[2]	English	Mandarin	Toki Pona[3]	Latin	Google Translate Intermediate[4]
Compact Tokens	❌	✅	✅	❌	✅✅
Compact Grammar	❌	✅	❌	✅	✅✅
Significant Training Material	✅✅	✅✅	❌	✅	❌
Software Support	✅✅	✅	❌	✅	❌
Human Learning Ability	✅	✅	✅✅✅	✅	❌

As a Jew, I learned the Hebrew Alphabet (but not vocabulary) to study for my Bar Mitzvah, and as a student of US public education, I had the choice in secondary school to learn Spanish, French, or German, in addition to learning English natively. I chose German, and I am very unlikely to change or stop learning this, but I wonder if it would be useful to learn a new language specifically to think in. This would pose some different requirements than traditionally learning a language, as reading and writing would be much less important for this task. Knowing many different words, and correct grammar would be much more important.

The idea of Brain-Machine interfaces installed into one's brain, ending the need for languages altogether would bring a major improvement to human society, but intentional control of thought[5] via language could bring the same effect. Aside from the normal cognitive benefits of being bilingual or multilingual, would learning some new language (or a conlang for this purpose) specifically to have conscious thought with be useful?

^

https://techcrunch.com/2025/01/14/openais-ai-reasoning-model-thinks-in-chinese-sometimes-and-no-one-really-knows-why/

^

These are not empirical or quantitative in any way, just the general ideas I sense from these. The other ideas expressed in this post are severable from this chart.

^

Intentionally made from-scratch language (conlang) with a very limited character set

^

Language used for Google Translate, all messages are first translated into this language, then translated into the output language. This method only requires 2 models per language, rather than the exponentially growing number needed for one between each language pair.

^

The idea of not learning certain words, as a w

...
New AI safety treaty paper out!
by otto.barten — 2025-03-26T09:29:07.661Z

Last year, we (the Existential Risk Observatory) published a

Time Ideas piece

proposing the Conditional AI Safety Treaty, a proposal to pause AI when AI safety institutes determine that its risks, including loss of control, have become unacceptable. Today, we publish our paper on the topic: “International Agreements on AI Safety: Review and Recommendations for a Conditional AI Safety Treaty”, by Rebecca Scholefield and myself (both Existential Risk Observatory) and Samuel Martin (unaffiliated).
We would like to thank Tolga Bilge, Oliver Guest, Jack Kelly, David Krueger, Matthijs Maas and José Jaime Villalobos for their insights (their views do not necessarily correspond to the paper).

Read the full paper here.

Abstract

The malicious use or malfunction of advanced general-purpose AI (GPAI) poses risks that, according to leading experts, could lead to the “marginalisation or extinction of humanity.”[1] To address these risks, there are an increasing number of proposals for international agreements on AI safety. In this paper, we review recent (2023-) proposals, identifying areas of consensus and disagreement, and drawing on related literature to indicate their feasibility.[2] We focus our discussion on risk thresholds, regulations, types of international agreement and five related processes: building scientific consensus, standardisation, auditing, verification and incentivisation.

Based on this review, we propose a treaty establishing a compute threshold above which development requires rigorous oversight. This treaty would mandate complementary audits of models, information security and governance practices, to be overseen by an international network of AI Safety Institutes (AISIs) with authority to pause development if risks are unacceptable. Our approach combines immediately implementable measures with a flexible structure that can adapt to ongoing research.

Treaty recommendations

(Below are our main treaty recommendations. For our full recommendations, please see the paper.)

The provisions of the treaty we recommend are listed below. The treaty would ideally apply to models developed in the private and public sectors, for civilian or military use. To be effective, states parties would need to include the US and China.

Establish a compute threshold above which development should be regulated.

Compute thresholds are the most effec

...
Gemini 2.5 Pro released
by Yair Halberstadt — 2025-03-26T07:56:44.964Z
Link: https://blog.google/technology/google-deepmind/gemini-model-thinking-updates-march-2025/

Google has now experimentally released Gemini 2.5 Pro.

As is standard, they claim it #beats all the benchmarks.

In my initial testing it's a very good model, definitely on par with GPT 4.5. It's a reasoning model, but runs very quickly, and shows you the chain of thought, unlike OpenAI models. I would consider using it for my daily driver. Biggest issue is that it does extensive thinking even when not really necessary (but as discussed it's quick so that matters less).
See more details at https://blog.google/technology/google-deepmind/gemini-model-thinking-updates-march-2025/.
Pricing is no yet available.
Probability Theory Fundamentals 102: Territory that Probability is in the Map of
by Ape in the coat — 2025-03-26T06:40:57.913Z

If probability is in the map, then what is the territory? What are we mapping when we apply probability theory?

"Our uncertainty about the world, of course."

Uncertainty, yes. And sure, every map is, in a sense, a map of the world. But can we be more specific? Say, for a fair coin toss, what particular part of the world do we map with probability theory? Surely it's not the whole world at the same time, is it?

"It is. You map the whole world. Multiple possible worlds, in fact. In some of them the coin is Heads in the others it's Tails, and you are uncertain which one is yours."

Wouldn't that mean that I need to believe in some kind of multiverse to reason about probability? That doesn't sound right. Even if those "possible worlds" existed, how am I supposed to know that's the case?

"Well, you don't necessary have to believe that there are parallel worlds as real as ours in which the coin comes differently, though it's a respectable position about the nature of counterfactuals. Probability is in the mind, remember? You can simply imagine alternative worlds that are logically consistent with your observations."

Even so, this can't be the way things work. Humans are not, in fact, able to hold the whole world in their minds and validate its logical consistency, let alone multiple worlds. If that was actually required, then probability theoretic reasoning would have only been accessible for literally galaxy-brained superintelligences.

"Well, in practice we simply imagine that we've imagined the whole world, do not notice any contradictions and call it a day without thinking too hard about the matter. It's a standard practice in philosophy."

Why does this not sound reassuring at all, I wonder? Wait a second, it's even worse than that. What about uncertainty about logic? Say I try to guess whether the 1,253,725,569th digit of pi is even or odd. Only one answer is actually logically consistent with my observations, but I don't know which one. Does that mean that I can't use probability theory to reason about it?

"Oh yes, formalizing logical uncertainty is a well known unsolved problem. We kind of pretend that we can still use probability theory with it as usual, even though it doesn't make sense in the framework of possible worlds, and it seems to work out just fine."

If in practice probability theoretic reasoning works the same way with logical and physical uncertainty...
Avoid the Counterargument Collapse
by marknm — 2025-03-26T03:19:58.655Z

Selective listening is a real problem. It’s really hard to listen to someone when you think you already know what they’re going to say; the words get filtered through our biases and expectations, and sometimes those filters remove the important bits. I want to talk about a specific way I've seen this happen. I don't know if there's a name for it already, so I'm going to call it the counterargument collapse until someone suggests a better term.

The pattern I’m talking about is three levels of progression through an argument, followed by a misunderstanding that collapses the discussion in an unpleasant way.

Level 1: A naive but widely accepted statement (“Eat less fat if you want to lose weight”.)

Level 2: A more sophisticated counterargument (“That’s a myth, fat doesn’t actually make you fat, it’s eating too many calories that makes you gain weight”.)

Level 3: A counter-counterargument. (“Actually, fatty foods tend to be more calorie-dense, so avoiding them can be beneficial if you’re trying to diet”.)

Collapse: If someone didn’t follow the whole reasoning chain, then they’ll often confuse Level 3 with Level 1 and think you’re dumb. If you’re not paying attention, then the third argument sounds a lot like “fat is bad”, so people lump it in with outdated diet myths and judge you for believing something so clearly false.

Here are a couple more examples:

I’m a chemical engineer, and many of the other engineers in my program ended up going into petroleum. I wanted to do something that had a positive impact on the world, and I thought that petroleum would be the opposite of that because of the negative effects on the environment (Level 1). But then a professor explained that engineers make oil extraction and utilization more efficient by reducing unwanted emissions, which is good for the environment overall (Level 2). I agreed with them for a while. But then I learned about the Jevons paradox, where increased efficiency leads to lower cost, higher demand, and increased resource consumption, which is bad for the environment after all (Level 3). It can be difficult to convey the idea of the Jevons paradox if the other person isn’t listening closely, and so I’ve had conversations where people think I just don’t understand the Level 2 argument.
There was a Reddit post a long time ago that now I can’t find, but it was what inspired me to write ab

...