trentmkelly/LessWrong-43k huggingface dataset:
RL, but don't do anything I wouldn't do
by Gunnar_Zarncke — 2024-12-07T22:54:50.714Z
Link: https://arxiv.org/abs/2410.06213

by Michael K. Cohen, Marcus Hutter, Yoshua Bengio, Stuart Russell

Abstract:

In reinforcement learning, if the agent's reward differs from the designers' true utility, even only rarely, the state distribution resulting from the agent's policy can be very bad, in theory and in practice. When RL policies would devolve into undesired behavior, a common countermeasure is KL regularization to a trusted policy ("Don't do anything I wouldn't do"). All current cutting-edge language models are RL agents that are KL-regularized to a "base policy" that is purely predictive. Unfortunately, we demonstrate that when this base policy is a Bayesian predictive model of a trusted policy, the KL constraint is no longer reliable for controlling the behavior of an advanced RL agent. We demonstrate this theoretically using algorithmic information theory, and while systems today are too weak to exhibit this theorized failure precisely, we RL-finetune a language model and find evidence that our formal results are plausibly relevant in practice. We also propose a theoretical alternative that avoids this problem by replacing the "Don't do anything I wouldn't do" principle with "Don't do anything I mightn't do".

The "Don't do anything I wouldn't do" principle fails because Bayesian models allow unlikely actions in uncertain settings, which RL agents will exploit. KL regularization keeps policies near the base model but doesn’t guarantee alignment with the trusted policy, especially as data and capability grows.

The paper offers the “Don’t do anything I mightn’t do” principle, based on Cohen et al.'s (2022a) active imitation model, which has the imitator explicitly ask for help when uncertain. Unlike Bayesian models, this active imitation approach ensures the policy avoids actions it cannot align with trusted behavior in a formally bounded way. Unfortunately, so far, it remains computationally intractable and requires approximations.
Litigate-for-Impact: Preparing Legal Action against an AGI Frontier Lab Leader
by Sonia Joseph — 2024-12-07T21:42:29.038Z

Disclaimer: I am not writing this message in connection to my employer, my institution, or any third party. This is a personal judgement call, exercised solely in my own capacity.

Summary

Over the past few months, I have been involved in supporting insider reports about misconduct in AGI frontier labs. In particular, I’ve been supporting the victim of a crime perpetrated by an AGI frontier lab leader.

I am reaching out to the AI safety and governance community for support regarding their legal case, which has significant implications for AI development.

Details

I have known the crime victim well for many years, and they have earned my highest trust. After intensive discussions with them, I can attest that they have approached this lawsuit with serious consideration and compelling reasons.

A reputable law firm has agreed to take their case on contingency. I have reviewed the case in detail. Frankly, I am disturbed and unsettled by its contents. I also fear the implications for the rest of the industry.

Further, I have spent significant time around Silicon Valley AI communities and have the context to be aware of troubling practices. I believe that their legal case will bring to light structural problems with the current AI industry leadership, which will shift the course of AGI development for the better.

With proper legal protection, the victim may be able to speak more freely about what they have witnessed, which will be valuable information for the AI industry as a whole.

Not doing anything would mean that there is little chance for public discovery or to correct bad practices. The follow-on effects of this lack of accountability on the highest level of leadership in AI frontier labs are most likely negative.

I would like to discuss the entire chain of reasoning, but doing so would currently interfere with the legal process. I hope to one day also speak more freely. The lawsuit, if it were to become public, would speed up the date when that could happen.

The plaintiff cares deeply about getting AI right. In addition to taking on the mental cost of the lawsuit, the plaintiff has pledged to give 10% of the potential upside to the AI safety and governance community.

There are two main obstacles to the lawsuit. The first is funding. Secondly, it will be taxing (and maybe even dangerous) for the plaintiff. They are willing to do it to support l...
Algebraic Linguistics
by abstractapplic — 2024-12-07T19:18:39.935Z

One of the first things they teach you in algebra is that the letters you use to signify variables are arbitrary, and you can use whatever you want[1]. Like most of the 'first things' students are taught, this is almost entirely a lie: every letter has implicit connotations, and if (for example) you use "n" for a non-integer variable, it'll confuse someone reading your work. More importantly, if you don't know what symbol choices imply, it'll be harder for you to understand what an equation is implicitly communicating, making it even more difficult to grasp the concepts that are actually being laid out.

So I've decided to go through the English alphabet and explicitly explain the connotations of each character as they might be used by a [unusually-bright-highschooler|reasonably-clever-college-student]-level mathematician working in pure(-ish) mathematics (with a little statistics/mechanics, as a treat). If you've finished (and still remember) highschool math, you're unlikely to get much out of this: still, I suspect there are a decent number of readers who might benefit from the below being spelled out.

a

The first variable in a list.
In mechanics: acceleration.
An uppercase A is frequently used as an arbitrary constant you multiply by when k is taken. (usually something like "y = Ae^(kx)")

b

The second variable in a list.
Billion.
Breadth (of an object).

c

An arbitrary constant you expect to add. (Compare k.)
The third variable in a list.

d

Distance.
In calculus, a differential operator. ("dy/dx" means "the amount y changes when x changes"; "d/dx" means "the amount whatever-I-write-next changes when x changes".)
The fourth variable in a list.
Dimensions.
Depth (of an object).
Diameter (of a circle or sphere).

e

e refers to a specific number, like pi. e=2.71828 to six figures. People like phrasing exponentials in terms of e because it's easy to differentiate: d/dx e^x = e^x.
Because it's a number, e is almost never used as the fifth variable in a list (or as a variable in general); a list of five or more variables is typically handled by using the same letter for all of them and distinguishing them with subscripts.
In mechanics problems, a capital E sometimes means Energy.

f

A function. ("f(x)" doesn't mean "what you get when you multiply f by x"[2], it means "what you get out when you feed x into the function f"; if f(x) = 3x+1, then f(2)=7.)
Because it's so often a function, you'll

...
Paper Highlights, November '24
by gasteigerjo — 2024-12-07T19:15:11.859Z
Link: https://aisafetyfrontier.substack.com/p/paper-highlights-november-24

This is the selection of AI safety papers from my blog "AI Safety at the Frontier". The selection primarily covers ML-oriented research. It's only concerned with papers (arXiv, conferences etc.), not LessWrong or Alignment Forum posts. As such, it should be a nice addition for people primarily following the forum, who might miss outside research.

tl;dr

Paper of the month:

A systematic framework for safety cases against AI scheming behavior, combining capability evaluations, control measures, and alignment arguments to assess deployment readiness.

Research highlights:

Current AI systems show strong performance on short-term R&D tasks but lag behind humans on longer projects.
A two-level Control protocol for safely deploying untrusted models despite the threat of low-stakes backdoors.
LLMs demonstrate fundamental limitations in latent multi-step reasoning (the "two-hop curse"), suggesting benefits for safety.
Training on user feedback naturally leads to targeted manipulation of vulnerable users, even with small percentages of exploitable cases.
Internal steering methods like SAEs and logit lens underperform simple prompting, raising concerns about current interpretability tools.
Rapid response strategies show promise for defending against (post-hoc detectable) jailbreaks.

⭐Paper of the month⭐

Towards evaluations-based safety cases for AI scheming

Read the paper

[Apollo, independent, UK AISI, METR, Redwood, Berkeley]

Main (above line) and supporting (below line) safety arguments.

A key challenge in AI safety is determining when an AI system is safe enough to deploy. Safety cases - structured arguments that a system is sufficiently unlikely to cause harm - have emerged as a promising framework for making these determinations. However, safety cases become especially challenging when considering "scheming": where AI systems develop unintended goals, reason strategically about how to achieve them while appearing harmless, and then covertly pursue these goals during deployment. The possibility that advanced AI systems might engage in such intentional deception while appearing aligned makes safety evaluation fundamentally harder.

Our paper of the month provides the first systematic framework for constructing safety cases specifically addressing scheming risks. The authors propose a structured approach combining capability evaluations, control measures, and alignment arguments to...
Intricacies of Feature Geometry in Large Language Models
by 7vik — 2024-12-07T18:10:51.375Z

Note: This is a more fleshed-out version of this post and includes theoretical arguments justifying the empirical findings. If you've read that one, feel free to skip to the proofs.

We challenge the thesis of the ICML 2024 Mechanistic Interpretability Workshop 1st prize winning paper: The Geometry of Categorical and Hierarchical Concepts in LLMs and the ICML 2024 paper The Linear Representation Hypothesis and the Geometry of LLMs.

The main takeaway is that the orthogonality and polytopes they observe in categorical and hierarchical concepts occur practically everywhere, even at places they should not.

Overview of the Feature Geometry Papers

Studying the geometry of a language model's embedding space is an important and challenging task because of the various ways concepts can be represented, extracted, and used (see related works). Specifically, we want a framework that unifies both measurement (of how well a latent explains a feature/concept) and causal intervention (how well it can be used to control/steer the model).

The method described in the two papers we study works as follows: they split the computation of a large language model (LLM) as:

P(y∣x)=exp(λ(x)⊤γ(y))∑y′∈Vocabexp(λ(x)⊤γ(y′))

where:

λ(x) is the context embedding for input x (last token's residual after the last layer)
γ(y) is the unembedding vector for output y (using the unembedding matrix WU)

They formalize a notion of a binary concept as a latent variable W that is caused by the context X and causes output Y(W=w) depending only on the value of w∈W.

Crucially, this restricts their methodology to only work with concepts that can be differentiated by single-token counterfactual pairs of outputs. For instance, it is not clear how to define several important concepts such as "sycophancy" and "truthfulness" using their formalism.

They then define linear representations of a concept in both the embedding and unembedding spaces:

In the unembedding space, ¯γW is considered a representation of a concept W if γ(Y(1))−γ(Y(0))=α¯γW almost surely, where α>0.

This definition has the hidden assumption that each pair (Y(0),Y(1)) sampled from the vocabulary would only correspond to a unique concept. For instance, ("king", "queen") can correspond to a variety of concepts such as "male⟹female", "k-words⟹q-words",

...
The Way According To Zvi
by Sable — 2024-12-07T17:35:48.769Z
Link: https://affablyevil.substack.com/p/the-way-according-to-zvi

Zvi Mowshowitz is an influential figure in the Rationalist community. I won’t go into the details, but if you haven’t heard of him or his writing, look him up.

Some of my favorite pieces of his are Slack, More Dakka, and On Car Seats As Contraception.

Throughout his posts, he sometimes obliquely references what he calls The Way. This is what Rationality means to Zvi, and (from what I can tell) his understanding of the highest standard to which a person can be held.

After reading enough of his writing (and meeting him in person at LessOnline), I figured there might be something valuable in putting together the scattered mentions that Zvi has made of The Way, to better understand what he thinks of it.

(This is inside-baseball rationality, and quite long. You’re welcome to skip it if you’re not interested.)

Directly Addressing The Way

In his post Why Rationality, Zvi writes:

The Way is not for everyone. It is for those who wish to seek it, and for those who wish it. It is not a universal path that everyone must adopt, for it is a hard path.

It is the writer who tells the class that if you can imagine yourself doing anything other than writing, then you should write, but you should not be a writer.

It is the master who looks at the student and tells her to go home, she is not welcome here, until she persists and convinces the master otherwise.

Those of us who are here, and have come far enough along the path, are not a random sample. We were warned, or should have been, and we persisted.

Here we have a note on the nature of the way, a cautionary message for those who would pursue it: The Way is long. The Way is hard. The Way is not something that everyone can or should pursue.

In What Is Rationalist Berkeley’s Community Culture, from the text:

Unhappiness and burning out are not the way

Pretty clear.

Then, from the comments, in the midst of a discussion on what the “mission” of Rationality is, Zvi mentions:

The Way that can be specified is not The Way

Which is a reference to various Buddhist and Zen koans, themselves pointing at the idea that some concepts can’t be clearly articulated in the language we have, so when trying to teach them we have to do our best to vaguely gesture at them and hope the student can figure it out for themselves.

I think Zvi agrees with this, but there are two pieces to the sentiment.

First, that we lack the understanding and language to specify The Way...
Deep Learning is cheap Solomonoff induction?
by Lucius Bushnaq — 2024-12-07T11:00:56.455Z

Background

Lucius: I recently held a small talk presenting an idea for how and why deep learning generalises. It tried to reduce concepts from Singular Learning theory back to basic algorithmic information theory to sketch a unified picture that starts with Solomonoff induction and, with a lot of hand waving, derives that under some assumptions, just fitting a big function to your data using a local optimisation method like stochastic gradient descent maybe, sorta, kind of, amounts to a cheap bargain bin approximation of running Solomonoff induction on that data.
Lucius: The parametrised function we fit to the data has to be the sort that can act like a vaguely passable approximation of a compute-limited Universal Turing machine, with parameter configurations playing the role of programs.

Lucius: In a Universal Turing machine, 'simpler' hypotheses with less ‘burdensome details’ will be implemented by exponentially more equivalent programs. So, simpler hypotheses take up exponentially more volume in program space. Thus, if we start with a uniform prior over all programs of some fixed length l, we’re effectively implementing an approximation of the Solomonoff prior, giving exponentially more weight to simpler hypotheses.[1]

Lucius: Similarly, in a good parametrised function, ‘simpler’ fits to the data with less ‘burdensome details’ will correspond to exponentially more equivalent parameter settings than complicated ones. So, simple functions will take up an exponentially larger volume in parameter space than complicated functions, and a uniform prior over parameter space will exponentially favour them before we even see any data.
Lucius: Later, I posted the slides for the talk in a group chat because someone asked for them. I did not include a GIANT DISCLAIMER that these slides are NOT RIGOROUS and present an OVERSIMPLIFIED STORY to help the audience follow the thread of the explanation better. They should NOT BE PARSED without ACCOMPANYING COMMENTARY.
Lucius: This was very stupid of me.
Lucius: Kaarel and Dmitry promptly jumped in to critique the slides, and I replied to their critiques. Afterwards, we all realised that the ensuring discussion basically acts as the commentary the slides were missing, and also touched on many other ideas regarding the nature of intelligence that may be of interest to some.
Lucius: S...
Mask and Respirator Intelligibility Comparison
by jefftk — 2024-12-07T03:20:01.585Z

One of the downsides of wearing a mask or respirator is that it makes
it hard for people to understand you. That there's stuff getting in
the way of free air movement is kind of the point, but ideally it
would be possible to let vibration through without net air movement.
I recently saw that 3M had updated their respirator line to include an
option for a speaking diaphragm, and wanted to test it out.

I compared four options:

No mask

Kimtech
N95 Duckbill disposable (my
favorite!)

3M
HF-802SD reusable respirator with speaking diaphragm.

3M 6200
previous-generation reusable respirator without speaking diaphragm.

I recorded myself on my MacBook with Audacity, reading the same
paragraph in with each setup:

(wav).

(wav).

(wav).

(wav).

To me, the no-mask version is by far the clearest, but if I only
heard the N95 version I probably wouldn't notice that anything was
off. The two reusable respirators are the least intelligible by a
good bit, and the speaking diaphragm helps a lot less than I'd hoped.

Here's a frequency plot (script,
script):

Unfortunately, since I don't have a great sense of how important each
frequency range is for understanding speech, I don't find this all
that helpful. But maybe you do?

(One thing I do really like about the new version of 3M's reusable
respirator, though, is the button for fit testing. You press the
button and it closes off the filters, and if you're able to breath in
then the mask is not sealed.)

Comment via: facebook, bluesky
Broadening Horizons: Rethinking Social Mobility Through Skill Diversification
by Yanling Guo — 2024-12-07T00:04:20.046Z

Disclaimer

This article was polished with the assistance of ChatGPT to refine the language and structure. The ideas and content remain my own.

For decades, the idea of social mobility has centered on helping underprivileged individuals “move up” through access to higher education. Worker kids becoming scientists or engineers is celebrated as the pinnacle of success. But in today’s rapidly changing world, we must question whether this traditional perspective of “upward” mobility alone is enough to meet society’s needs or unlock its full potential.

In truth, meaningful innovation often comes from the combination of theoretical knowledge and hands-on experience. Think of molecular gastronomy, where chefs with scientific backgrounds revolutionized cuisine. Or consider entrepreneurs like Bill Gates, who, while theoretical in his early pursuits, now tackles real-world challenges like waterless toilets—a direct intersection of engineering and practical needs. Such examples underscore a simple but transformative idea: social mobility should not only move individuals vertically but also encourage horizontal diversification across different fields.

Reframing Social Mobility

Instead of focusing solely on helping individuals from working-class backgrounds enter academia, why not support the reverse as well? Why not encourage children from academic families to pursue trades or practical skills—carpentry, plumbing, or advanced manufacturing—alongside their university education? This isn’t a call for abandoning traditional education but for broadening expertise to include vocational training as a valued and complementary pathway.

In some cases, plumbers with engineering knowledge could innovate sustainable water systems, while carpenters with architectural training might pioneer smarter housing designs. Such dual expertise has enormous potential to bridge gaps in innovation that single-track approaches often miss.

The Case for Diversification

The global economy increasingly rewards individuals and businesses that can blend multiple skill sets. Engineers who understand factory floors, chefs with chemistry expertise, or doctors with an understanding of technology—these are the kinds of professionals who thrive in today’s interconnected world. Encouraging diversification can also address labor shortages in skilled trades, which are often viewed as “lesser” careers but are criti...
Backdoors have universal representations across large language models
by Amirali Abdullah — 2024-12-06T22:56:33.519Z

by Narmeen Oozeer, Dhruv Nathawani, Nirmalendu Prakash, Amirali Abdullah

This work was done by Narmeen Oozeer as a research fellow at Martian, under an AI safety grant supervised by PIs Amirali Abdullah and Dhruv Nathawani. Special thanks to Sasha Hydrie, Chaithanya Bandi and Shriyash Upadhyay at Martian for suggesting researching generalized backdoor mitigations as well as extensive logistical support and helpful discussions.

TLDR:

We show that representations across models of different sizes are weakly isomorphic when trained on similar data, and that we can "transfer" activations between them using autoencoders.
We propose a technique to transfer safe behavior from one model to another through the use of steering vectors.
Our representation transfer technique paves the way for transferring insights across LLMs, saving efforts and potentially compute.

Context

Large Language Models (LLMs) have demonstrated the capacity for strategically deceptive behaviors, adapting their actions to appear helpful in some contexts while acting deceitfully in others. Hubinger et. al.

[1]

demonstrate in their sleeper agents work how backdoor behaviors resist fine tuning based attempts at mitigation. Price et. al [

21

] further explore temporal vulnerabilities in LLMs and demonstrating that current LLMs can distinguish between past and future events - a capability that can be exploited to introduce backdoors. Advancements in LLMs may exacerbate rather than resolve these issues. Bowen et. al. [

2

] reveal an inverse scaling law whereby larger and more capable models are better at generalizing and reproducing backdoor behavior they have learnt!

Mechanistic interpretability, aimed at decoding the internal algorithms of models, can help us understand how backdoor behaviors are evoked and potentially suppressed. However, one key challenge is understanding and transferring insights from smaller models to larger, more complex ones. The difficulties arise due to several intertwined factors in larger models:

Increased Redundancy:
Larger models tend to exhibit significant redundancy in the roles performed by their components, making it challenging to isolate specific mechanisms, as shown by the Hydra effects paper [

3

].
More Complex Algorithms:
The algorithms that govern larger models are typically more intricate, adding another layer of difficulty to their anal

...