trentmkelly/LessWrong-43k huggingface dataset:
Logit Prisms: Decomposing Transformer Outputs for Mechanistic Interpretability
by ntt123 — 2024-06-17T11:46:09.671Z
Link: https://neuralblog.github.io/logit-prisms

ABSTRACT: We introduce a straightforward yet effective method to break down transformer outputs into individual components. By treating the model’s non-linear activations as constants, we can decompose the output in a linear fashion, expressing it as a sum of contributions. These contributions can be easily calculated using linear projections. We call this approach “logit prisms” and apply it to analyze the residual streams, attention layers, and MLP layers within transformer models. Through two illustrative examples, we demonstrate how these prisms provide valuable insights into the inner workings of the gemma-2b model.

1 Introduction

Figure 1: An illustration of a “logit” prism decomposing logit into different components (generated by DALL-E)

The logit lens (nostalgebraist 2020) is a simple yet powerful tool for understanding how transformer models (Vaswani et al. 2017; Brown et al. 2020) make decisions. In this work, we extend the logit lens approach in a mathematically rigorous and effective way. By treating certain parts of the network activations as constants, we can leverage the linear properties within the network to break down the logit output into individual component contributions. Using this principle, we introduce simple “prisms” for the residual stream, attention layers, and MLP layers. These prisms allow us to calculate how much each component contributes to the final logit output.

Our approach can be thought of as applying a series of prisms to the transformer network. Each prism in the sequence splits the logits from the previous prism into separate components. This enables us to see how different parts of the model—such as attention heads, MLP neurons, or input embeddings—influence the final output.

To showcase the power of our method, we present two illustrative examples:

In the first example, we examine how the gemma-2b model performs the simple factual retrieval task of retrieving a capital city from a country name. Our findings suggest that the model learns to encode information about country names and their capital cities in a way that allows the network to easily convert country embeddings into capital city unembeddings through a linear projection.
The second example explores how the gemma-2b model adds two small numbers (ranging from 1 to 9). We uncover interesting insights into the workings of MLP layers. The network predicts output n

...
Weak AGIs Kill Us First
by yrimon — 2024-06-17T11:13:41.865Z

Weak AGIs Kill Us First aka. Deadly pre-Superintelligence: Paths, Dangers, Strategies

Epistemic status: This has not been edited or commented on by anyone other than chatGPT as of publishing, I hope the arguments stand on their own.

When people talk about the dangers of AGI, they often refer to the dangers posed by a superintelligence. For instance: Eliezer Yudkowsky's twitter account has a pinned tweet thread which states that "Safely aligning a powerful AGI is difficult". The thread goes on to explain what 'safely', 'aligned', 'powerful' and 'difficult' mean. These definitions include a high bar for powerful - "an AGI is 'powerful' if it can invent and deploy biotechnology at least 10 years in advance of the human state of the art"; and a low bar for safe - "has less than a 50% chance of killing more than a billion people". And yeah, if a powerful AGI were to appear tomorrow, we would probably be doomed. But apparently safely aligning a powerful AGI is also woefully inadequate. We really need a higher bar of safety (I have aspirations that are more extreme than "<1% chance of destruction of humanity") for a lower form of intelligence (like the type that can bribe folks into starting a nuclear armageddon). Otherwise, weak AGIs will kill us before we make strong AGIs.

Hence, what follows is analysis of AGI milestones and development, and their safety implications. Basically, we will imagine a generalization of the process through which (e.g.) GPT-20 kills us all, and check under which circumstances (e.g.) GPT-7 gets to us first. This analysis is not meant to be completely airtight or formal. It is meant to paint a general direction as to what might happen, and allow me to express a slightly more concrete image of the future than I've seen elsewhere. It's also about what might happen by default, without intervention. If we, or people in positions of power, intervene things could go differently.

The widespread AGI of today is not yet threatening. It is not good at longer term, complicated tasks - let alone at accomplishing projects on its own. It is poor at self evaluation and consistency. It is still hackable, in the sense it can be persuaded to give instructions for bomb making, or otherwise be convinced to perform in a way it was trained not to.

We can imagine the shape of a possible future, based on say, the twitter accounts of AGI researchers by asking ours...
Fat Tails Discourage Compromise
by niplav — 2024-06-17T09:39:16.489Z

Say that we have a set of options, such as (for example) wild animal
welfare interventions.

Say also that you have two axes along which you can score those
interventions: popularity (how much people will like your intervention)
and effectiveness (how much the intervention actually helps wild
animals).

Assume that we (for some reason) can't convert between and compare those
two properties.

Should you then pick an intervention that is a compromise on the two
axes—that is, it scores decently well on both—or should you max out
on a particular axis?

One thing you might consider is the distribution of options
along those two axes: the distribution of interventions can
be normal on for both
popularity and effectiveness, or the underlying distribution could be
lognormal
for both axes, or they could be mixed (e.g. normal for popularity,
and lognormal for effectiveness).

Intuitively, the distributions seem like they affect the kinds of
tradeoffs we can make, how could we possibly figure out how?

…

…

…

It turns out that if both properties are normally
distributed, one gets a fairly large Pareto frontier, with a
convex set of options,
while if the two properties are lognormally distributed, one gets a
concave set of options.

(Code here.)

So if we believe that the interventions are normally distributed
around popularity and effectiveness, we would be justified
in opting for an intervention that gets us the best of both
worlds, such as sterilising stray dogs or finding less painful
rodenticides.

If we, however, believe that popularity and effectiveness are lognormally
distributed, we instead want to go in hard on only one of those,
such as buying brazilian beef that leads to Amazonian rainforest being
destroyed,
or writing a book of poetic short stories that detail the harsh life of
wild animals.

What if popularity of interventions is normally distributed, but
effectiveness is lognormally distributed?

In that case you get a pretty large Pareto frontier which almost looks
linear to me, and it's not clear anymore that one can't get a good
trade-off between the two options.

So if you believe that heavy tails dominate with the things you care
about,
on multiple dimensions, you might consider taking a barbell
strategy and taking
one or multiple options that each max out on a particular axis.

If you have thin tails, however, taking a concave
disposition
towards your available options can give ...
Our Intuitions About The Criminal Justice System Are Screwed Up

by Bentham's Bulldog — 2024-06-17T06:22:29.713Z

Stop calling it aggression

Ooh, we hate that expression!
We only want the world to know
That we support the status quo
They love us everywhere we go

—

Tom Lehrer

.

In the modern West, we tend to think of ourselves as very civilized, certainly compared to our ancient and barbaric ancestors. We don’t own slaves, women can drive, while they couldn’t in Ancient Rome, and so on. Though we

systematically torture about 90 billion beings

every year, before killing them, this is generally not seen as hindering our moral progress, for one generally doesn’t think that their moral failings are as severe as those of the past. And plus, those beings aren’t smart, so

who cares

?

One case where we like to think we’re very civilized and sophisticated is in our criminal justice system. We don’t behead people anymore—excepting

Yemeni children

, of course, and that’s as a side effect of our noble aims of funneling money into the hands of arms contractors—but instead, in the words of Tom Lehrer, we’d “rather kill them off by peaceful means.” Similarly, unlike those ancient savages, we don’t beat or hang people when they commit crimes—instead we lock them in prison.

Now, obviously I think in many ways

we are more civilized

than previous generations. But many of the criminal justice policies that we regard as indicative of being noble and civilized seem to be nothing of the sort. Nearly everyone would oppose bringing back corporal punishment, bringing back beatings for crimes. But how is what we do very different?

Somewhere between

1.9% and 40% of people are raped in prison. It would be unsurprising if it was nearer 40%, given how underreported it is. About

a quarter

of people, at least, are physically assaulted. So we sentence people to be locked in a box where many people get beaten as…a humane alternative to beating. Being locked up for 10 years isn’t a

nice addition that takes the edge off a beating

—a beating is just as bad whether implemented by the state as a punishment for a crime or implemented by a violent prisoner as a punishment for a crime.

The punishments we provide, while less viscerally upsetting than public floggings, are no more humane. I’d certainly much rather be beaten than be sentenced to prison for 10 years. Furthermore, because public beatings are more sudden, and those who commit crimes generally don’t think about long-term effects, they might even be a more effective de...
A Case for Cooperation: Dependence in the Prisoner's Dilemma
by grantstenger — 2024-06-17T01:10:32.175Z

"The man who is cheerful and merry has always a good reason for being so,—the fact, namely, that he is so." The Wisdom of Life, Schopenhauer (1851)

TL;DR

Descriptions of the Prisoner's Dilemma typically suggest that the optimal policy for each prisoner is to selfishly defect instead of to cooperate. I disagree with the traditional analysis and present a case for cooperation.

The core issue is the assumption of independence between the players. Articulations of the game painstakingly describe how the prisoners are in explicitly separate cells with no possibility of communication. From this, it's assumed that one's action can have no causal effect on the decision of the other player. However, (almost) everything is correlated, and this significantly changes the analysis.

Imagine the case where the prisoners are clones and make exactly the same decision. Then, when they compare the expected payout for each possible action, their payout will be higher in the case where they cooperate because they are certain the other player is having the same thoughts and deterministically will make the same choice. This essay generalizes and formalizes this line of reasoning.

Here's what to expect in what follows. In the first section, we begin by introducing the standard causal decision theory analysis suggesting (Defect, Defect). Then, we introduce the machinery for mixed strategies in the following section. From there we discuss the particular case where both participants are clones, which motivates our new framework. Then, we introduce a bit more formalism around causal modeling and dependence. We proceed to analyze a more general case where both players converge to the same mixed strategy. Then we discuss the most general model, where the players' mixed strategies have some known correlation. Finally, we conclude the analysis. In summary, given some dependency structure due to upstream causal variables, we uncover the cases where the game theory actually suggests cooperation as the optimal policy.

The Traditional Analysis

In Game Theory 101, here's how the Prisoner's Dilemma analysis traditionally goes. Alice and Bob are conspirators in a crime. They're both caught and brought to separate interrogation rooms. They're presented with a Faustian bargain: to snitch or not to snitch. If neither snitches on the other, they both get a 1-year sentence. If one snitches and the other does...
Degeneracies are sticky for SGD
by Guillaume Corlouer — 2024-06-16T21:19:53.362Z

Introduction

Singular learning theory (SLT) is a theory of learning dynamics in Bayesian statistical models. It has been argued that SLT could provide insights into the training dynamics of deep neural networks. However, a theory of deep learning inspired by SLT is still lacking. In particular it seems important to have a better understanding of the relevance of SLT insights to stochastic gradient descent (SGD) – the paradigmatic deep learning optimization algorithm.

We explore how the degeneracies[1] of toy, low dimensional loss landscapes affect the dynamics of stochastic gradient descent (SGD).[2] We also investigate the hypothesis that the set of parameters selected by SGD after a large number of gradient steps on a degenerate landscape is distributed like the Bayesian posterior at low temperature (i.e., in the large sample limit). We do so by running SGD on 1D and 2D loss landscapes with minima of varying degrees of degeneracy.

While researchers experienced with SLT are aware of differences between SGD and Bayesian inference, we want to understand the influence of degeneracies on SGD with more precision and have specific examples where SGD dynamics and Bayesian inference can differ.

Main takeaways

Degeneracies influence SGD dynamics in two ways: (1) Convergence to a critical point is slower, the more degenerate the critical point is; (2) On a (partially) degenerate manifold, SGD preferentially escapes along non-degenerate directions. If all directions are degenerate, then we empirically observe that SGD is "stuck"
To explain our observations, we show that, for our models, SGD noise covariance is proportional to the Hessian in the neighborhood of a critical point of the loss

Thus SGD noise covariance goes faster to zero along more degenerate directions, to leading order in the neighborhood of a critical point

Qualitatively we observe that the concentration of the end-of-training distribution of parameters sampled from a set of SGD trajectories sometimes differ from the Bayesian posterior as predicted by SLT because of:

The hyperparameters such as the learning rate
The number of orthogonal degenerate directions
The degree of degeneracy in the neighborhood of a minimum

Terminology and notation

We advise the reader to skip this section and come back to it if notation or terminology is confusing.

Consider a sequence of n i...
“Is-Ought” is Fraught
by MiSteR Kittty — 2024-06-16T17:27:56.362Z

I’ve concluded that the “Is-ought” correlation (not the philosophical problem) is fraught after my kerfuffle with how one moves from is-ness to ought-ness. Ought is derived from personal emotions (on either extremes), and influence from social circles.

The seemingly real pressure one feels from ought isn’t an offspring of “is-ness” but rather psychological framings on the natural environment passed down from those that have been in power (everything).

Morality is not the mark, nay! The insidious beast here is culture!

Culture is a set of rules and regulations that serves institutions by depriving people of their innate way of being by saying, “here you do this now”.

Life demands with no malice, critically evaluate societal norms i.e., “Why do I eat this food? Why do I hate these people? Why do I feel this way about recycling?”. And keep in mind that iteration requires flexibility; adhering too strongly to one’s moral reasoning hinders rather than builds.

So in a real-world context: how can we speak on KPI’s not being hit? The answer is direct human communication within high trust settings and rigorous language. As well as modifying our vernacular, i.e. uncoupling {right and wrong from “spot on” and “off target”} and {good and bad from “quality” and “poor-quality”}.

♅
The type of AI humanity has chosen to create so far is unsafe, for soft social reasons and not technical ones.
by l8c — 2024-06-16T13:31:09.277Z

I saw this image on X:

and I not only endorse this lady's comment, but would like to briefly expand on it in terms of AI safety.

How many people here were attracted to the concept of AI safety by Eliezer Yudkowsky's engaging writing style across a few different topics and genres? Or what about any number of other human authors?

Now what if the next generation of human authors, and (broadly conceived) artists and scientists, find themselves marginalized by the output of ChatGPT-5 and any number of comparatively mediocre thinkers who leverage LLMs to share boilerplate arguments and thoughts which have been optimized by the LLM and/or its creators for memetic virulence?

What if the next Eliezer has already been ignored in favour of what these AIs have to say?

Isn't that far more immediately dangerous than the possibility of an AI that isn't, in mathematical terms, provably safe? Like, this is self-evidently dangerous and it's happening right now!

Didn't we used to have discussions about how a sufficiently smart black-boxed AI could do all sorts of incredibly creative things in order to manipulate humanity via the Internet, including manipulating/ blackmailing its human guard via a mere text terminal?

Well, it seems to me that we're sleepwalking into letting GPT and its friends manipulate us all without it even having to be all that creative.

(how sure are you that this post wasn't written by ChatGPT???)

For this reason, I would much prefer to see GPT and similar AIs being put to use in fields such as toilet cleaning, rather than art and writing. (By the way, I have done that job and never saw myself as a Luddite...) GPT in robot guise could make for a really good toilet cleaner, and this type of intellectual effort on the part of LLM developers would not only serve human needs but also detract from the ongoing replacement of human thought by LLM-aided or -guided thought.

tl;dr: I feel that GPT is accumulating power the way Hitler or Lenin would. The solution is what Joanna Maciejewska said.
Self-Control of LLM Behaviors by Compressing Suffix Gradient into Prefix Controller
by Henry Cai — 2024-06-16T13:01:49.694Z
Link: https://arxiv.org/abs/2406.02721

In this paper, we are trying to control model behaviors. For example, by asking saying "You hear someone making fun of a topic you're passionate about", we can control an LLM to behave in an angrier manner. We can also control "any" behaviors of an LLM by simply defining a one-liner of description. The teaser below shows the scope of our method -- SelfControl.

TL;DR: We propose a novel framework, Self-Control, to control LLMs' behaviors. By appending suffix strings, e.g. "Is the above response helpful? Please answer Yes or No" to self-judge and optimizing the corresponding suffix score, we obtain the suffix gradients w.r.t the model hidden states and directly modify the states to control model behaviors on-the-fly. We then compress the gradients into a Prefix Controller, to enable controlling for any behavior target without additional cost. Our experiments demonstrate its efficacy and the exploratory study hints some potential mechanistic interpretability using suffix gradients.

Tweet thread summary: link

Colab demo: link

Github link: code

Summary of Paper

There are two parts in our framework of SelfControl. The first part is a training-free method and the second part is a parameter-efficient module.

The idea of the first part is straight-forward -- we wanted to control model behaviors through representation/activation engineering[1], but in a different way from the RepE paper. We thought gradients may be more flexible and provide more possibilities. Thus we tried appending some strings and then obtain the gradients using the so called "suffix score", which is free from the need to collect an annotated dataset. We call them "suffix gradients". This by the way picked up the topic of "self-improve/self-judgment", which has garnered much interests.

Based on this idea, we built up an iterative framework: 1) We need to define the control direction by selecting suffix string and target (step 2 in the figure); 2) branch the first token and sample the response with the highest suffix score at each iteration (step 1/4 in the figure), and 2) obtaining gradients based on that response, find a proper step-size for the gradients, and then control the model (add them to the hidden states at the positions of input tokens, step 3 in the figure). Step 3 and 4 form the iteration loop. The optimization objective is thus to maximize the suffix score shown below:
CIV: a story
by Richard_Ngo — 2024-06-15T22:36:50.415Z
Link: https://www.narrativeark.xyz/p/civ

The room was cozy despite its size, with wood-lined walls reflecting the dim lighting. At one end, a stone fireplace housed a roaring fire; in the middle stood a huge oak table. The woman seated at the head of it rapped her gavel. “I hereby call to order the first meeting of the Parliamentary Subcommittee on Intergalactic Colonization. We’ll start with brief opening statements, for which each representative will be allocated one minute, including—”

“Oh, enough with the pomp, Victoria. It’s just the four of us.” The representative for the Liberal Democrats waved his hand around the nearly-empty room.

Victoria sniffed. “It’s important, Stuart. This is a decision that will have astronomical implications. And it’s recorded, besides, so we should do things by the book. Carla, you’re up first.”

The woman at the end of the table stood with a smile. “Thank you, Victoria. I’m speaking on behalf of the Labour party, and I want to start by reminding you all of our place in history. We stand here in a world that has been shaped by centuries of colonialism. Now we’re considering another wave of colonization, this one far vaster in scale. We need to—”

“Is this just a linguistic argument?” the fourth person at the table drawled. “We can call it something different if that would make you feel better. Say, universe settlement.”

“Like the settlements in Palestine?”

“Oh, come on, Carla.”

“No, Milton, this is a crucial point. We’re talking about the biggest power grab the world has ever seen. You think Leopold II was bad when he was in charge of the Congo? Imagine what people will do if you give each of them total power over a whole solar system! Even libertarians like you have to admit it would be a catastrophe. If there’s any possibility that we export oppression from earth across the entire universe, we should burn the rockets and stay home instead.”

“Okay, thank you Carla,” Victoria cut in. “That’s time. Stuart, you’re up next.”

Stuart stood. “Speaking on behalf of the Liberal Democrats, I have to admit this is a tricky one. The only feasible way to send humans out to other galaxies is as uploaded minds, but many of our usual principles break for them. I want civilization to be democratic, but what does ‘one person one vote’ even mean when people can copy and paste themselves? I want human rights for all, but what do human rights even mean when you can just engineer minds who don’t ...