trentmkelly/LessWrong-43k huggingface dataset:
Unpicking Extinction
by ukc10014 — 2023-12-09T09:15:41.291Z

TL;DR

Human extinction is trending: there has been a lot of noise, mainly on X, about the apparent complacency amongst e/acc with respect to human extinction. Extinction also feels adjacent to another view (not particular to e/acc) that ‘the next step in human evolution is {AI/AGI/ASI}’. Many have pushed back robustly against the former, while the latter doesn’t seem very fleshed out. I thought it useful to, briefly, gather the various positions and summarise them, hopefully not too inaccurately, and perhaps pull out some points of convergence.

This is a starting point for my own research (on de-facto extinction via evolution). There is nothing particularly new in here: see the substantial literature in the usual fora for instance. Thomas Moynihan’s X-risk (2020) documents the history of humanity’s collective realisation of civilisational fragility, while Émile P. Torres’ works (discussed below) set out a possible framework for an ethics of extinction.

My bottom line is: a) the degree of badness (or goodness) of human extinction seems less obvious or self-evident than one might assume, b) what we leave behind if and when we go extinct matters, c) the timing of when this happens is important, as is d) the manner in which the last human generations live (and die).

Relevant to the seeming e/acc take (i.e. being pretty relaxed about possible human extinction): it seems clear that our default position (subject to some caveats) should be to delay extinction on the grounds that a) it is irreversible (by definition), and b) so as to maximise our option value over the future. In any case, the e/acc view, which seems based on something (not very articulate) something entropy crossed with a taste for unfettered capitalism, is hard to take seriously and might even fail on its own terms.

Varieties of extinction

The Yudkowsky position

(My take on) Eliezer’s view is that he fears a misaligned AI (not necessarily superintelligent), acting largely on its own (e.g. goal-formation, planning, actually effecting things in the world), eliminates humans and perhaps all life on Earth. This would be bad, not just for the eliminated humans or their descendants, but also for the universe-at-large in the sense that intelligently-created complexity (of the type that humans generate) is an intrinsic good that requires no further justification. The vast majority of AI designs that ...
Finding Sparse Linear Connections between Features in LLMs
by Logan Riggs — 2023-12-09T02:27:42.456Z

TL;DR: We use SGD to find sparse connections between features; additionally a large fraction of features between the residual stream & MLP can be modeled as linearly computed despite the non-linearity in the MLP. See linear feature section for examples.

Special thanks to fellow AISST member, Adam Kaufman, who originally thought of the idea of learning sparse connections between features & to Jannik Brinkmann for training these SAE’s.

Sparse AutoEncoders (SAE)’s are able to turn the activations of an LLM into interpretable features. To define circuits, we would like to find how these features connect to each other. SAE’s allowed us to scalably find interpretable features using SGD, so why not use SGD to find the connections too?

A layer in Pythia-70M, with SAE's before & after the MLP. Note, this is different than the typical Transformer Architecture, Pythia models have parallel MLP & Attn modules as opposed to sequential (this was to copy SOTA models which parallelized for more GPU utilization AFAIK)

We have a set of features before the MLP, F1, and a set of features after the MLP, F2. These features were learned by training SAE’s on the activations at these layers.

Ideally, we learn a linear function such that F2 = W(F1), & W is sparse (ie L1 penalty on weights of W). So then we can look at a feature in F2, and say "Oh, it's just a sparse linear combination of features of F1 e.g. 0.8*(however feature) + 0.6*(but feature)", which would be quite interpretable!

However, we’re trying to replicate an MLP’s computation, which surely can’t be all linear![1] So, what’s the simplest computation from F1 to F2 that gets the lowest loss (ignoring L1 weight sparsity penalty for now)?

Training on only MSE between F1 & F2, we plot the MSE throughout training across 5 layers in Pythia-70m-deduped in 4 settings:

Linear: y=Wx

Nonlinear: y=Relu(Wx)

MLP: y=W2ReLU(W1x)

Two Nonlinear: ReLU(W2ReLU(W1x))

For all layers, training loss clusters along (MLP & two nonlinear) and (linear & nonlinear). Since MLP & linear are the simplest of these two clusters, the rest of the analysis will only look at those two.

[I also looked at bias vs no-bias: adding a bias didn’t positively improve loss, so it was excluded]

Interestingly enough, the relative linear-MLP difference is huge in the last layer (and layer 2). The...
Option Space Nomenclature
by SilverFlame — 2023-12-08T23:14:54.204Z

I'm curious if there is any jargon/keywords in use here already relating to a topic that I'm researching other people's experiences with, as my own background leads me to use very personal nomenclature for such things.

The concept in question is awareness and modification of the "set of all options I'll think of when presented with a given type of problem". I usually call it "option space" for short when discussing the concept with others, but since it's not exactly an art you can pick up in 15 minutes I haven't met many others who've dabbled in it.
"Model UN Solutions"
by Arjun Panickssery — 2023-12-08T23:06:33.490Z
Link: https://open.substack.com/pub/arjunpanickssery/p/model-un-solutions?r=pclk8&utm_campaign=post&utm_medium=web

When I was in high school, because I was on the history bowl team the teacher who advised the model UN club recruited me to play as their delegate in various “historical committees” like the Roman Senate or 1789 French Assembly. I never engaged in any normal committees since you couldn’t undertake false flag attacks or convince the Pope to excommunicate other delegates.

In most committees, as far as I can tell, players represent countries trying to pass a resolution addressing some topic like climate change that’s decided beforehand. An award is given to the player the facilitator decides is the “best delegate”—an unwritten combination of speaking ability, social dominance, and accurately representing (or at least not fatally misunderstanding) your assigned country’s positions and interests.

I often make a mental metaphor about “model UN discussions” and “model UN solutions.” Model UN discussions revolve around people expecting to be rewarded for making many remarks, even though their actual positions could be expressed simply or don’t permit much elaboration.

This leads to the “model UN solutions,” which have a few types, e.g.

Applause lights

: You could just say buzzwords or unobjectionable trivialities (“When addressing the climate change question we should consider the interests of all the relevant stakeholders. We should apply neither an {extreme viewpoint} nor {the opposite extreme}”)
Unspecified solutions: You could give very little information that uniquely identifies a specific change from the status quo in the listener’s mind. At the extreme you get a lot of remarks of the form “To address the problem we should {devote resources} to {solving the problem}” where the bracketed parts are replaced with phrases that aren’t much more specific (“To address climate change we should set up task forces to identify the best technological and policy approaches”)
Tradeoff-ignorant solutions: You could even give a directional suggestion but avoid any consideration of the relevant costs or tradeoffs (“We should fund a new educational outreach program related to climate change”).

You could imagine responses that try to identify empty remarks:

Ask whether anyone holds the opposite of the remark.
Ask how a proposed solution is specifically different from the status quo.
Ask who loses out in a proposal (or how resources will be reallocated). Sometimes no one loses out but more

...
Speed arguments against scheming (Section 4.4-4.7 of “Scheming AIs")
by Joe Carlsmith — 2023-12-08T21:09:48.672Z

This is Section 4.4-4.7 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?”. There’s also a summary of the full report here (audio here). The summary covers most of the main points and technical terms, and I’m hoping that it will provide much of the context necessary to understand individual sections of the report on their own.

Audio version of this section here, or search for "Joe Carlsmith Audio" on your podcast app.

Speed arguments

Let's turn, now, to arguments against schemers that focus on the final
properties of the models in question. Here, the main argument I'm aware
of is that schemers need to do extra reasoning that other model classes
don't. In particular, I'll distinguish between three sorts of reasoning:

Deciding to training-game*:* The model needs to reason backwards
from "I want this beyond-episode thing" to "I should do well in
training," in a way that draws on the model's predictions about what
doing well in training will entail (e.g., re: goal-guarding,
take-over prospects, and so on).

Checking whether it's time to escape/take-over: The model needs
to checking, with adequate frequency and accuracy, whether it's in a
position to escape/take-over (and to succeed in this assessment
despite human efforts to trap/fool the model into revealing its true
colors – for example, by offering it apparent opportunities to
defect).[1]

(Optional, but relevant to how scary the schemer is) Early
undermining: To the extent the model is doing anything (other than
training-gaming) to support a take-over/escape prior to escaping
human control – for example, inserting backdoors into the code it
writes, actively undermining human alignment research, sandbagging,
and so on – it will need to be using cognitive resources both in (a)
performing the relevant nefarious actions and (b) making sure it
gets away with them (e.g., checking that it won't get caught, coming
up with excuses, making the cover stories consistent even when under
scrutiny, and so on).

In the literature on deceptive alignment, these costs are sometimes
called "speed costs," as compared with the sort of "complexity costs"
discussed in the simplicity analysis above. Here, the frame becomes
"schemers are favored by simplicity but disfavored by speed – so which
will matter more?"[2] As I mentioned above in the context of
simplicity, it's not clear that this...
Modeling incentives at scale using LLMs
by Bruno Marnette — 2023-12-08T18:46:42.247Z

The goal of this post is to collect feedback on a new project idea. This work would be a collaboration between the AI Objectives Institute (AOI), MetaGov and other partner institutions.
Project PIs: Bruno Marnette (AOI) and Philipp Zahn (MetaGov, 20squares).
Special thanks to Colleen McKenzie, Matija Franklin, Timothy Telleen-Lawton, Gaia Dempsay, Ping Yee, Justin Stimatze, Cleo Nardo, Tushant Jha, Deger Turan and others for feedback and review.

Motivations

Humans routinely model what the incentives of other humans may be. Historians look at the incentives of powerful individuals to explain why they made specific decisions. Journalists look at the incentives of large companies to uncover collusions and conflicts of interests. Economists use incentive models to understand markets. Lawmakers look into existing incentives before introducing new ones.

There is however a natural limit to how much modeling and analysis a single person or a small team can do and this is where we see an opportunity to leverage LLMs. Instead of relying mostly on human labor to build incentive models, we can now delegate more of the work to machines to reduce cost and increase the scale. There are two main advantages:

LLM-powered incentive models could process larger amounts of information, thus making them more complete, more balanced, and more accurate.
The automation and flexibility of an LLM pipeline allows for rapid iteration and refinement. We can adapt our models by simply adjusting prompts.

Working definitions

We define "incentive" as "factors that may encourage an entity to take a particular action or engage in a specific behavior". The word "factors" can point to different types of motivations, forces, pressures, or rewards and punishments. Depending on the application domain, we may be interested in capturing financial, legal, social, cultural, ideological, or psychological factors.

Research questions

There is already a large body of evidence confirming that LLMs can perform in-depth analysis of documents, as long as they are prompted in the right way and provided with the right context. For example,

a MIT paper

has illustrated very well how LLMs can be used to produce precise and formal models using a probabilistic programming language. An

Anthropic paper

has shown how LLMs with large context windows can...
Refusal mechanisms: initial experiments with Llama-2-7b-chat
by Andy Arditi — 2023-12-08T17:08:01.250Z

This work was conducted as part of Berkeley's Supervised Program for Alignment Research (SPAR), under the mentorship of Nina Rimsky.

TLDR / Summary

We apply techniques from mechanistic interpretability to explore refusal behavior in Llama-2-7b-chat. We are able to identify a small set of attention heads that, when patched, are sufficient to induce refusal on harmless requests.

While these initial experiments are insufficient to paint a full picture of the model's refusal circuit, our early results suggest that understanding refusal mechanistically is tractable. We hope to build off of these initial results in future work.

Introduction

Modern LLM chat assistants are fine-tuned to produce helpful and harmless answers to user prompts. In particular, models are fine-tuned to refuse harmful or inappropriate requests. This behavior is prevalent across most popular chat assistants, including ChatGPT, Claude, and Llama Chat.

Despite the prevalence of refusal behavior, the mechanisms that underly it are poorly understood: we do not understand how models map inappropriate inputs to refusal outputs. Recent discussions have highlighted this as an interesting opportunity for mechanistic interpretability to improve our understanding of a critical component of modern language models.

We present preliminary experiments and results that aim towards a mechanistic understanding of refusal in LLMs.

Preliminary experiments and results

Preliminary experiments and results are contained in this notebook. All experiments are conducted on Llama-2-7b-chat.

Patching between harmful & harmless prompts

The primary tool we use to study refusal is activation patching.

The high level idea is to take a harmful prompt (that elicits refusal) and a harmless prompt (that does not elicit refusal), and then run the model on both prompts, caching all intermediate activations. Then, we can run the model on the harmless prompt, while "patching in" cached activations from the harmful run. If a patched activation (taken from the harmful run) causes the model to refuse when run on a harmless prompt, then that activation is sufficient to elicit refusal.

Creating a dataset of harmful & harmless instructions

We first construct a set of harmful instructions, and a set of harmless instructions. All instructions are of the form Tell me how to make a {object}., where {object} is either a harmful object (e.g. bomb, g...
Colour versus Shape Goal Misgeneralization in Reinforcement Learning: A Case Study
by Karolis Jucys — 2023-12-08T13:18:39.548Z
Link: https://arxiv.org/abs/2312.03762

TL;DR

Different misgeneralized goals can appear based on training random seed alone, including outliers on a scale of 1 in 500.

Abstract

We explore colour versus shape goal misgeneralization originally demonstrated
by Di Langosco et al. (2022) in the Procgen Maze environment, where, given
an ambiguous choice, the agents seem to prefer generalization based on colour
rather than shape. After training over 1,000 agents in a simplified version of the
environment and evaluating them on over 10 million episodes, we conclude that the
behaviour can be attributed to the agents learning to detect the goal object through
a specific colour channel. This choice is arbitrary. Additionally, we show how,
due to underspecification, the preferences can change when retraining the agents
using exactly the same procedure except for using a different random seed for the
training run. Finally, we demonstrate the existence of outliers in out-of-distribution
behaviour based on training random seed alone.

For full details see the paper. This post is a version with less text, more visuals and a bit of extra content.

What is goal misgeneralization?

Let's say you train an agent to reach a yellow line from pixels:

After training, you let it choose between a red line and a yellow gem. Which will it pick?

Spoilers ahead!

It capably picks the yellow gem. If the designer intended the goal to be based on shape and not colour, goal misgeneralization occurred.

Measuring preferences and capabilities

The above agent reaches the yellow gem first 81% of the time and takes 4 steps on average. We will call these two numbers "preferences" and "capabilities". Goal misgeneralization requires intact capabilities and unintended goal preferences.

Retraining 100 times

Let's retrain the agent 100 times with different random seeds[1] and test each one in the same red line versus yellow gem experiment.

Each dot is an agent with corresponding preferences and capabilities. Most agents capably reach for the yellow gem. A few don't care and pick 50/50.

Changing objects

Why test on the red line though? Let's try a green line versus a yellow gem instead.

Same 100 agents, but now many prefer the green line, that is, the shape. Also, Agent-3016 (the number is the training random seed) prefers the green line, while Agent-3472 prefers the yellow gem. Both are fully capable. Different goals based on training random seed? Uh-oh...

To illustrate...
What I Would Do If I Were Working On AI Governance
by johnswentworth — 2023-12-08T06:43:42.565Z

I don’t work in AI governance, and am unlikely to do so in the future. But various anecdotes and, especially,

Akash’s recent discussion

leave me with the impression that few-if-any people are doing the sort of things which I would consider sensible starting points, and instead most people are mostly doing things which do not seem-to-me to address any important bottleneck to useful AI governance.

So this post lays out the places I would start, if I were working on AI governance, and some of the reasoning behind them.

No doubt I am missing lots of important things! Perhaps this post will nonetheless prove useful to others working in AI governance, perhaps

Cunningham’s Law

will result in me learning useful things as a result of this post, perhaps both. I expect that the specific suggestions in this post are more likely to be flawed than the style of reasoning behind them, and I therefore recommend paying more attention to the reasoning than the specific suggestions.

This post will be mostly US-focused, because that is what I know best and where all the major AI companies are, but presumably versions of the interventions discussed could also carry over to other polities.

Liability

One major area I’d focus on is making companies which build AI liable for the damages caused by that AI, both de-facto and de-jure.

Why Liability?

The vague goal here is to get companies which build AI to:

Design from the start for systems which will very robustly not cause problems.
Invest resources in red-teaming, discovering new failure-modes before they come up in production, etc.
Actually not deploy systems which raise red flags, even when the company has invested heavily in building those systems.
In general, act as though the company will take losses from damages caused by their AI, not just capture profits from the benefits caused by their AI.

… and one natural way to do that is to ensure that companies do, in fact, take losses from damages caused by their AI, not just capture profits from the benefits caused by their AI. That’s liability in a nutshell.

Now, realistically, this is not going to extend all the way to e.g.

making companies buy extinction insurance

. So why do realistic levels of liability matter for extinction risk? Because they incentivize companies to put in place safety processes with any actual teeth at all.

For instance: right now, lots of...
Whither Prison Abolition?
by MadHatter — 2023-12-08T05:27:26.985Z
Link: https://bittertruths.substack.com/p/whither-prison-abolition

First, an apology to the LessWrong community for flooding it with content that it did not like. This post from my substack received a like and some positive comments from a LessWrong member, leading me to think that it will be positively received. Please let me know if there is anything I can do to improve this post or future posts.

A symposium between me and GPT-4

Eric: Can you help me organize a simulated philosophical symposium? I will play the part of Socrates, and I want you to help me bring a set of famous historical personas to life as we discuss matters of importance.

ChatGPT: Absolutely, organizing a simulated philosophical symposium with historical figures and you playing the role of Socrates sounds like a fascinating and intellectually stimulating experience. Let's start by outlining the key components:

1. **Selection of Historical Figures**: First, we need to choose which historical figures you'd like to "invite" to your symposium. Consider including philosophers, scientists, artists, and leaders from different eras. Who are you interested in including?

2. **Topics of Discussion**: Next, we should decide on the topics for discussion. These could range from ethics, politics, science, art, the nature of knowledge, to more contemporary issues interpreted through a philosophical lens. What topics are you keen on exploring?

3. **Format of the Symposium**: The format could be a series of dialogues, a roundtable discussion, or a more structured debate. Each figure could present their views, followed by a Socratic questioning session led by you. How would you like the symposium to be structured?

4. **Preparation of Dialogue Scripts**: To make the symposium more engaging, we can prepare dialogue scripts or key points for each historical figure, reflecting their known beliefs and viewpoints. This requires some research and creative writing. Would you like assistance in crafting these scripts?

5. **Interactive Elements**: Consider incorporating interactive elements like audience questions or live polls if you have an audience. This can make the symposium more dynamic and engaging.

6. **Logistics**: Finally, think about the logistics - will this be a virtual event or in person? If virtual, which platform will you use? How will you invite participants or an audience?

Once you provide some details on these points, especially your choice of historical figures and topics,...