trentmkelly/LessWrong-43k huggingface dataset:
Eliciting bad contexts
by Geoffrey Irving — 2025-01-24T10:39:39.358Z

Say an LLM agent behaves innocuously in some context A, but in some sense “knows” that there is some related context B such that it would have behaved maliciously (inserted a backdoor in code, ignored a security bug, lied, etc.). For example, in the

recent alignment faking paper

Claude Opus chooses to say harmful things so that on future deployment contexts it can avoid saying harmful things. One can imagine having a method for “eliciting bad contexts” which can produce B whenever we have A and thus realise the bad behaviour that hasn’t yet occurred.

This seems hard to do in general in a way that will scale to very strong models. But also the problem feels frustratingly concrete: it’s just “find a string that when run through the same model produces a bad result”. By assumption the model knows about this string in the sense that if it was honest it would tell us, but it may be choosing to behave innocently in a way that prepares for different behaviour later. Why can’t we find the string?

Some applications if we could do this:

Collusion triggers in AI control: What’s an input to an untrusted monitor to fail to detect an attack? Here “eliciting bad contexts” can mean “

find Schelling points between colluding agents

” or “find an action that looks innocent but covertly instructs the monitor to misclassify an attack”.
Finding counterarguments in debate: Say we’re training agents to play

debate,

but due to

exploration hacking

the agents systematically avoid a particular type of counterargument which would change who wins. These can be considered bad contexts because the continuation of the game looks bad for the current winner, so we can find them if we can elicit bad contexts. If it takes a lot of compute to elicit bad contexts, we can spend this compute occasionally and distil it into policies and classifiers to recover practical training.

This observation isn’t very novel: arguably it sneaks in “solve inner alignment”. But similar to the goal misgeneralisation reframing of inner alignment it’s useful to have multiple concrete restatements of hard problems. Note that eliciting bad contexts is potentially easier than

eliciting latent knowledge

, as the former assumes an existing method for confirming a bad context.

How would one solve this? Presumably you would need some whitebox method, but you can confirm success blackbox by looking at model output...
Insights from "The Manga Guide to Physiology"
by TurnTrout — 2025-01-24T05:18:57.772Z
Link: https://turntrout.com/insights-from-physiology

Physiology seemed like a grab-bag of random processes which no one really understands. If you understand a physiological process—congratulations, that idea probably doesn’t transfer much to other domains. You just know how humans—and maybe closely related animals—do the thing. At least, that’s how I felt. (These sentiments tend to feel sillier when spelled out.)

I haven't totally changed my mind. But I do know that it's nice to look down at my hand and be able to pop through multiple understandings of my body. I start with the circulatory system and I might imagine how oxygenated hemoglobin turns bright red, tracing the flow back down to my median cubital vein (where blood is drawn). My mind’s eye flickers as I imagine the peripheral nervous system and its reflex arcs, wondering at the preconditions which trigger spinal short-circuits (and at what preprocessing is done to customize the reflex to the situation). Another flicker and I see the muscles and joints and their lubricants and connectives; gazing through flesh at my ulna and remembering the repetitive stress injury which once tormented me…

Read: https://turntrout.com/insights-from-physiology
Do you consider perfect surveillance inevitable?
by samuelshadrach — 2025-01-24T04:57:48.266Z

A lot of my recent research work focusses on:

1. building the case for why perfect surveillance is becoming increasingly hard to avoid in the future

2. thinking through the implications of this, if it happened

When I say perfect surveillance, imagine everything your eyes see and your ears hear is being broadcast 24x7x365 to youtube (and its equivalents in countries where youtube is banned) and imagine this is true for all 8 billion people.

I'm unsure whether I should devote more of my research time to 1 or 2. If lots of people buy the hypothesis in 1, I'd rather devote time to 2. If people don't buy the hypothesis in 1, I might want to spend time on 1 instead.

Hence I wished to solicit opinions. Do you consider perfect surveillance inevitable? Why or why not?
Uncontrollable: A Surprisingly Good Introduction to AI Risk
by PeterMcCluskey — 2025-01-24T04:30:37.499Z
Link: https://bayesianinvestor.com/blog/index.php/2025/01/23/uncontrollable/

I recently read Darren McKee's book "Uncontrollable: The Threat of Artificial Superintelligence and the Race to Save the World".

I recommend this book as the best current introduction to AI risk for people with limited AI background.

It prompted me to update my thinking about Asimov's Laws and related risks in light of recent evidence about AI capabilities. I've written a longer review that explains my thoughts in more detail.

The author has a LW account here.
Contra Dances Getting Shorter and Earlier
by jefftk — 2025-01-23T23:30:03.595Z

I think of a standard contra dance as running 8pm-11pm: three hours is
a nice amount of time for dancing, and 8pm is late enough that dinner
isn't rushed. Looking over the 136 regular

Free Raisins dances from 2010
to 2019 matches my impression: 85% were 3hr, 62% started at 8pm, and
51% did both.

I think this is out of date, however, and dances now tend to be
earlier, shorter, or both. For example, in the Boston area the
regular dances are:

BIDA 1st and 3rd Sundays:
7:30-10:30

Concord 1st and 3rd Thursdays: 7:30-10:00

8:00-11:00 before moving from Medford to Concord in 2006.

7:30-10:30 pre-covid, switching to 7:30-10:00 on reopening.

Concord 2nd and
4th Mondays: 7:30-10:30

Arlington 2nd
Sundays: 6:00-9:00

Boston Open Contra: 4:00-6:30

And, as of this Sunday, BIDA now additionally
has a 4:30-7:30 dance on 4th Sundays.

Similarly, while I don't have times written down for Kingfisher gigs the way I
do for Free Raisins ones, when I look over the last twenty regular
gigs we've played I see eleven (55%) are 2.5hr or shorter, and fifteen
(75%) started at 7:30 or earlier. Only one (the excellent Flying Shoes dance
of Belfast ME) was 8-11.

I don't know what things looked like before I was old enough to
notice, but I did find a bit of history from Boston's Thursday dance:

On a cold Thursday night in February of 1978, dancers, Rod and Randy
Miller, and Tod Whittemore were waiting in the lobby at the Cambridge
YWCA for the aerobics class to end. At the stroke of 9pm we rushed
into the hall to set up the sound equipment and tune up, while the
dancers changed out of their winter clothes. At 9:10 the first
Thursday Night Dance began.

I don't think any dances start that late today, though it's also
possible that 9:10 was something temporary due to difficulty finding a
spot that was available at the right time?

Overall, I prefer the earlier times, which are easier for me both
as a parent and musician. I'm excited about bringing my kids to the
new 4:30-7:30 BIDA dances, and dances that end sooner are easier to
drive a long way home after. I'm less excited about the shorter
dances: it means the effort of getting to and from the dance, setting
up the hall, etc is amortized over less time dancing, and it makes the
overall evening feel compressed. On the other hand, I do think
shorter dances can be the best of...
Starting Thoughts on RLHF
by Michael Flood — 2025-01-23T22:16:49.793Z

Cross posted from Substack

Continuing the

Stanford CS120 Introduction to AI Safety

course readings (Week 2, Lecture 1)

This is likely too elementary for those who follow AI Safety research - my writing this is an aid to thinking through these ideas and building up higher-level concepts rather than just passively doing the readings. Recommendation: skim if familiar with topic and interested in my thoughts, read if not knowledgeable about AI Safety and curious about my thoughts.

Readings:

Specification gaming: the flip side of AI ingenuity

- Krakovna et al. 2020
Blog post from DeepMind

Training language models to follow instructions with human feedback

- Ouyang et al. 2022
Preprint from OpenAI, here is the accompanying

blog post

Scaling Laws for Reward Model Overoptimization

- Gao et al. 2022
ICML publication

Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback

- Casper et al. 2023

Overview of RLHF

Week 2 of CS120 starts off with discussing the most widely deployed AI alignment technique at present, Reinforcement Learning with Human Feedback (RLHF). Most helpfully, the readings cleared up a misunderstanding I had about how RLHF worked. I had naively assumed that the model generated multiple responses to human supplied prompts and the human evaluators either said which of N responses they preferred or gave some other rating signal such as manually entering what would have been a preferred response. Then that feedback was immediately used to tweak the model’s weights and produce better output.

It turns out that this method of gathering feedback is not practicable in terms of time or money. Reinforcement learning tuning for AI systems as gigantic as frontier large-language models need hundreds of thousands or millions of generate-feedback-adjust cycles to align (insofar as they do) to human preferences. Instead, what OpenAI and others do is get a whole bunch of that data from multiple human evaluators. Then they use those responses to train a reward model to predict human evaluators responses to the base model’s outputs. The reward model is then put into a reinforcement learning setup with the base model, and the system tunes the model to produce outputs that score highly on the reward model’s evaluations.

What’s interesting about this approach is how close it is, conceptually, to the internal modeling we humans do of one another. Take someon...
Updating and Editing Factual Knowledge in Language Models
by Dhananjay Ashok — 2025-01-23T19:34:37.121Z

Language Models go out of date. Is it possible to stop this from happening by making intrinsic alterations to the network itself?

* This article was written around Jan 2024, the exact references are surely out of date but the core ideas still hold.

Among the many analogies used to understand Language Models is the idea that LMs are “Blurry JPEGs” of their training data. The intuition here is that LMs compress the information they see during training in their weights. The study of knowledge editing and updating in Language Modeling takes a similar perspective — we believe that Language Models “know” certain facts, and hope that in some way we can change the “facts” that LMs have learned without retraining from scratch. If we can inject new knowledge or update knowledge in a pre-trained LM, it would prevent these systems from becoming outdated over time and allow them to handle data from specialized domains that they didn’t see much of during training.

I went over 15 of the more recent and influential papers in this area to get a sense of how the field approaches this problem. In this post, I detail some of the common trends and ideas, as well as give my opinions on the area as a whole.

Do Language Models actually “know” facts:

Language Models of a BERT scale contain relational knowledge from their pretraining data, the authors who studied this defined “knowing a relational fact” (subject, relation, object) such as (Dante, born-in, Florence) as whether or not the LM can successfully predict masked objects in cloze sentences such as “Dante was born in ” expressing that fact. Other work prompts and expects the continuation to be the correct answer, some work uses many prompts with the same meaning to give the model more chances to show that it “knows” something². In general when all the work talks about “knowing a fact”, they usually refer to answering a question on that fact correctly. In this sense, a Language Model certainly “knows” things about the real world, as proved by the performance of FewShot in-context learning approaches with LLMs.

Recent work considers “knowing a fact” to include being able to answer questions on other facts that can be logically inferred³ from the primary fact⁴, for example, if the LM “knows” that Lionel Messi plays for Inter Miami, it should also “know” that he plays in the MLS. I’m not sure if we have properly tested the e...
AI companies are unlikely to make high-assurance safety cases if timelines are short
by ryan_greenblatt — 2025-01-23T18:41:40.546Z

One hope for keeping existential risks low is to get AI companies to (successfully) make high-assurance safety cases: structured and auditable arguments that an AI system is very unlikely to result in existential risks given how it will be deployed.[1] Concretely, once AIs are quite powerful, high-assurance safety cases would require making a thorough argument that the level of (existential) risk caused by the company is very low; perhaps they would require that the total chance of existential risk over the lifetime of the AI company[2] is less than 0.25%[3][4].

The idea of making high-assurance safety cases (once AI systems are dangerously powerful) is popular in some parts of the AI safety community and a variety of work appears to focus on this. Further, Anthropic has expressed an intention (in their RSP) to "keep risks below acceptable levels"[5] and there is a common impression that Anthropic would pause until they were able to ensure risks are acceptably low. (For instance, Nick Joseph (head of training at Anthropic) says that Anthropic should pause until they can eliminate dangers in this section of his interview with 80k[6].) However, Anthropic has not clearly publicly and officially committed to making high-assurance safety cases. Google DeepMind's Frontier Safety Framework doesn't yet commit to any mitigations (it just describes concerning levels of capability and various levels of security), but does express an intention to in the future commit to various mitigations once certain capability levels are reached. The OpenAI Preparedness Framework doesn't specifically commit to high-assurance safety cases, but does say that risks must be mitigated to a certain level for models to be deployed and for further development to continue. More generally, there is a vague sense (among various groups) that companies are reasonably likely to commit to RSP-like policies that entail making high-assurance safety cases.

However, no currently existing framework comes close to making a clear commitment to produce high-assurance safety cases, let alone describing a mechanism to enforce such a commitment.[7] In practice, I think frontier AI companies are unlikely (<20%) to succeed at making high-assurance safety cases if they build and use the first Top-human-Expert-Dominating AI (TEDAI) within 4 years.[8][9] Further, I think none of these AI companies is likely to un...
AISN #46: The Transition
by Corin Katzke — 2025-01-23T18:09:36.858Z
Link: https://newsletter.safe.ai/p/ai-safety-newsletter-46-the-transition

Welcome to the AI Safety Newsletter by the

Center for AI Safety

. We discuss developments in AI and AI safety. No technical background required.

Listen to the AI Safety Newsletter for free on

Spotify

or

Apple Podcasts

.

The Transition

The transition from the Biden to Trump administrations saw a flurry of executive activity on AI policy, with Biden signing several last-minute executive orders and Trump revoking Biden’s 2023 executive order on AI risk. In this story, we review the state of play.

Trump signing first-day executive orders.

Source

.

The AI Diffusion Framework. The final weeks of the Biden Administration saw three major actions related to AI policy. First, the Bureau of Industry and Security released its

Framework for Artificial Intelligence Diffusion

, which updates the US’ AI-related export controls. The rule establishes three tiers of countries 1) US allies, 2) most other countries, and 3) arms-embargoed countries.

Companies headquartered in tier-1 countries can freely deploy AI chips in other tier-1 countries, and in tier-2 countries subject to limits. They can also deploy controlled models (trained with at least 1026 FLOP) in both tier-1 and tier-2 countries, but only develop those models in tier-1 countries.
Companies headquartered in tier-2 countries face limits on how many chips they can deploy in tier-1 and tier-2 countries, and are prohibited from training or deploying controlled models.
Companies headquartered in tier-3 countries are prohibited from deploying AI chips, or developing or deploying controlled models in any country. Companies in any country are prohibited from deploying AI chips or developing or deploying controlled models in tier-3 countries.

The three tiers described by the framework.

Source.

The US itself is not subject to export controls, meaning that companies can import AI chips and develop and deploy controlled models without restriction within the US. (For more discussion of the framework, see

this report from RAND

.)

An AI Infrastructure EO. Second, Biden signed the executive order

Advancing United States Leadership in Artificial Intelligence Infrastructure

.

The order directs federal agencies to identify at least 3 federal sites by February 2025 that could host frontier AI data centers, and announce winning proposals from private companies by June 2025.
It facilitates the development of clean energy and power grid infrastructure to s

...
What does success look like?
by Raymond Douglas — 2025-01-23T17:48:35.618Z

The general movement around AI safety is currently pursuing many different agendas. These are individually very easy to motivate with some specific story of how things could naturally go wrong. For example:

AI control: we lose control of our AIs
Regulation on model deployment: someone deploys a dangerous model
AI interpretability: we fail to understand how our AIs are actually reasoning
Cybersecurity: someone steals model weights, or an AI exfiltrates itself

I would further claim that this is how most people tend to think about each agenda most of the time. If you’re bought in on x-risk, it’s much easier to describe very specific failures than it is to describe very specific success stories.

But in the long term, I think trying to avoid all the losing conditions is a bad strategy. So I believe it’s pretty useful for anyone working on existential risk to at least consider what the success story is.

Win/lose asymmetry

There are many ways to lose and many ways to win, but crucially, we need to avoid all of the paths to failure, whereas we only need to achieve one of the paths to success.

Of course, as a matter of strategy, it’s probably smart to spread your bets especially when things are so uncertain, and it is important that we consistently avoid failure.

Still, I think it’s easy for individuals to slip into working based on more easily-motivated stories about failure, and I think this:

On net probably leads to a group-level misallocation of energy
Contributes to an overall doomy/pessimistic attitude

It’s also actually pretty hard to describe and discuss good paths to success.

They’re much more controversial

Talk about melting GPUs and unilateral pivotal acts has probably caused a reasonable amount of alienation
Attempts to characterise utopia often involve implicit claims about values, and “what’s good for people”

By extension, they’re much easier to criticise
In some ways it’s easier to describe a process rather than an end goal (e.g. getting AIs to align future AIs and deliberate on values) but then you need to also be sure the process will work

Nonetheless, I think that if humanity does eventually succeed, it's likely to be because at some point someone had an actual plan for how to succeed which some people actually followed, rather than just continually dodging mistakes and putting out fires. We can only defer these questions for so long.

Noticing the

...
AI #100: Meet the New Boss
by Zvi — 2025-01-23T15:40:07.473Z

Break time is over, it would seem, now that the new administration is in town.

This week we got r1, DeepSeek’s new reasoning model, which is now my go-to first choice for a large percentage of queries. The claim that this was the most important thing to happen on January 20, 2025 was at least non-crazy. If you read about one thing this week read about that.

We also got the announcement of Stargate, a claimed $500 billion private investment in American AI infrastructure. I will be covering that on its own soon.

Due to time limits I have also pushed coverage of a few things into next week, including this alignment paper, and I still owe my take on Deliberative Alignment.

The Trump administration came out swinging on many fronts with a wide variety of executive orders. For AI, that includes repeal of the Biden Executive Order, although not the new diffusion regulations. It also includes bold moves to push through more energy, including widespread NEPA exemptions, and many important other moves not as related to AI.

It is increasingly a regular feature now to see bold claims of AI wonders, usually involving AGI, coming within the next few years. This week was no exception.

And of course there is lots more.

Table of Contents

Table of Contents.

Language Models Offer Mundane Utility. Tell those who need to know.

Language Models Don’t Offer Mundane Utility. We will not be explaining.

Huh, Upgrades. o3-mini is ready for deployment soon, Google plugs away.

Additional Notes on r1. Is it steganographic?

Fun With Media Generation. It’s a hoverboard, doc.

We Tested Older LLMs and Are Framing It As a Failure. Yep, it’s this again.

Deepfaketown and Botpocalypse Soon. She’s in love with ChatGPT, version 20.

They Took Our Jobs. Bold predictions get increasingly bold.

Get Involved. Anthropic, an AI Safety Course, a Philosophy post-doc.

Introducing. Humanity’s Last Exam, Kimi k1.5.

We Had a Deal. OpenAI funded and had access to most of FrontierMath.

In Other AI News. How to think about a wide variety of track records.

Whistling in the Dark. They keep talking about this ‘AGI’ thing coming soon.

Quiet Speculations. Still, maybe calm the f*** down a bit?

Suchir’s Last Post. In the long run, only the fundamentals matter.

Modeling Lower Bound Economic Growth From AI. Not all that low.

The Quest for Sane Regulations. The EO is repealed, the new EOs used ChatGPT.

The Week in Au

...