trentmkelly/LessWrong-43k huggingface dataset:
Decision Transformer Interpretability
by Joseph Bloom — 2023-02-06T07:29:01.917Z

TLDR: We analyse how a small Decision Transformer learns to simulate agents on a grid world task, providing evidence that it is possible to do circuit analysis on small models which simulate goal-directedness. We think Decision Transformers are worth exploring further and may provide opportunities to explore many alignment-relevant deep learning phenomena in game-like contexts.

Link to the GitHub Repository. Link to the Analysis App. I highly recommend using the app if you have experience with mechanistic interpretability. All of the mechanistic analysis should be reproducible via the app.

Key Claims

A 1-Layer Decision Transformer learns several contextual behaviours which are activated by a combination of Reward-to-Go/Observation combinations on a simple discrete task.
Some of these behaviours appear localisable to specific components and can be explained with simple attribution and the transformer circuits framework.
The specific algorithm implemented is strongly affected by the lack of a one-hot-encoding scheme (initially left out for simplicity of analysis) of the state/observations, which introduces inductive biases that hamper the model.

If you are short on time, I recommend reading:

Dynamic Obstacles Environment

Black Box Model Characterisation

Explaining Obstacle Avoidance at positive RTG using QK and OV circuits

Alignment Relevance

Future Directions

I would welcome assistance with:

Engineering tasks like app development, improving the model, training loop, wandb dashboard etc. and people who can help me make nice diagrams and write up the relevant maths/theory in the app).
Research tasks. Think more about how to exactly construct/interpret circuit analysis in the context of decision transformers. Translate ideas from LLMs/algorithmic tasks.
Communication tasks: Making nicer diagrams/explanations.
I have a Trello board with a huge number of tasks ranging from small stuff to massive stuff.

I’m also happy to collaborate on related projects.

Introduction

For my ARENA Capstone project, I (Joseph) started working on decision transformer interpretability at the suggestion of Paul Colognese. Decision transformers can solve reinforcement learning tasks when conditioned on generating high rewards via the specified “Reward-to-Go” (RTG). However, they can also generate agents of varying quality based on the RTG, making them simultaneously ...
Why is Everyone So Boring? By Robin Hanson
by trevor — 2023-02-06T04:17:20.372Z
Link: https://www.overcomingbias.com/2023/02/why-is-everyone-so-boring.html

This is from the blog Overcoming Bias.

These excerpts describe the overall model. It is only due to Robin Hanson's writing skill that the dynamic can be described this concisely, yet still be covered so thoroughly and intuitively. This is a classic example of galaxy-brained writing.

Centuries ago, while people could rest safe and show themselves at home, when traveling between towns they tried to look either look poor or well-defended, as bandits lay in wait. Even within towns, people without allies who acted unusually rich, assertive, and confident would induce others to try to trip them somehow. It’s the tall poppy that gets cut down, after all.

I propose that the main reason that most of us look more boring in public is that social predators lie in wait there. With friends, family, and close co-workers, we are around people that mostly want to like us, and know us rather well. Yes, they want us to conform too, but they apply this pressure in moderation.

Out in public, in contrast, we face bandits eager for chances to gain social credit by taking us down, often via accusing us of violating the sacred. And like townspeople traveling among the bandits, we are in public pretty vulnerable to the kinds of bandits that afflict us.

If we act interesting, passionate, and opinionated in public, we are likely to seem to claim high status for ourselves, and to touch on sacred subjects, either by word or deed. And this makes us quite vulnerable to accusations of arrogance and violating the sacred.

I see roughly three typical public stances: boring, lively, or outraged. Either you act boring, so the bandits will ignore you, you act lively, and invite bandit attacks, or you act outraged, and play a bandit yourself. Most big orgs and experts choose boring, and most everyone else who doesn’t pick boring picks bandit, especially on social media. It takes unusual art, allies, and energy, in a word “eliteness”, to survive while choosing lively.
Gradient surfing: the hidden role of regularization
by Jesse Hoogland — 2023-02-06T03:50:39.649Z
Link: https://www.jessehoogland.com/note/gradient-surfing-the-hidden-role-of-regularization

Produced under the mentorship of Evan Hubinger as part of the SERI ML Alignment Theory Scholars Program - Winter 2022 Cohort

In a previous post, I demonstrated that Brownian motion near singularities defies our expectations from "regular" physics. Singularities trap random motion and take up more of the equilibrium distribution than you'd expect from the Gibbs measure.

In the computational probability community, this is a well-known pathology. Sampling techniques like Hamiltonian Monte Carlo get stuck in corners, and this is something to avoid. You typically don't want biased estimates of the distribution you're trying to sample.

In deep learning, I argued, this behavior might be less a bug than a feature.

Regularization may have a hidden function to make the set of minimum-loss points more navigable. Simply drifting around this set of points privileges simple solutions, even in the absence of explicit complexity penalties. Or not. The evidence isn't conclusive.

The claim of singular learning theory is that models near singularities have lower effective dimensionality. From Occam's razor, we know that simpler models generalize better, so if the dynamics of SGD get stuck at singularities, it would suggest an explanation (at least in part) for why SGD works: the geometry of the loss landscape biases your optimizer towards good solutions.

This is not a particularly novel claim. Similar versions of the claim been made before by Mingard et al. and Valle Pérez et al.. But from what I can tell, the proposed mechanism, of singularity "stickiness", is quite different.

Moreover, it offers a new possible explanation for the role of regularization. If exploring the set of points with minimum training loss is enough to get to generalization, then perhaps the role of regularizer is not just to privilege "simpler" functions but also to make exploration possible.

In the absence of regularization, SGD can't easily move between points of equal loss. When it reaches the bottom of a valley, it's pretty much stuck. Adding a term like weight decay breaks this invariance. It frees the neural network to surf the loss basin, so it can accidentally stumble across better generalizing solutions.

So could we improve generalization by exploring the bottom of the loss basin in other ways — without regularization or even without SGD? Could we, for example, get a model to grok through rand...
Why Are Bacteria So Simple? 
by aysja — 2023-02-06T03:00:31.837Z

As far as we can tell, bacteria were the first lifeforms on Earth. Which means they’ve had a full four billion years to make something of themselves. And yet, despite their long evolutionary history, they mostly still look like this:

Bacteria belong to one major class of cells—prokaryotes.[1] The other major class of cells, eukaryotes, arrived about one billion years after bacteria. But despite their late start, they are vastly more complex.

Prokaryotes mostly only contain DNA, and DNA translation machinery. Eukaryotes, on the other hand, contain a huge variety of internal organelles that run all kinds of specialized processes—lysosomes digest, vesicles transport, cytoskeletons offer structural support, etc.

Not only that, but all multicellular life is eukaryotic.[2] Every complex organism evolution has produced—eukaryotic. Trees, humans, worms, giant squid, dogs, insects—eukaryotic. Somehow, eukaryotes managed to blossom into all of these complex forms, while bacteria steadfastly remained single-celled, simple, and small. Why?

The short answer is that prokaryotes have vastly less DNA than eukaryotes—four to five orders of magnitude less, on average—and hence can’t do nearly as much stuff.[3] The long answer is the rest of this post, which investigates two related questions: first, why are eukaryotic genomes so long? And second, how exactly does more DNA allow for more complexity?

Why Are Eukaryotic Genomes So Long?

Scalable Energy Production

Using DNA—replicating, transcribing, and translating it into proteins—isn’t free. Cells need energy (such as ATP) to power these reactions and, all else equal, longer genomes will require more of it.[4]

Both prokaryotes and eukaryotes pay similar energetic costs to maintain genes. The difference is that eukaryotes have way more energy and hence can afford to have longer genomes. But why this disparity?

Prokaryotes generate ATP along their cell membrane. Which means that as they increase in size, their surface area—and hence their energy production—will scale sublinearly with their volume. So a prokaryote that doubles in size, for example, will only end up producing half as much ATP per unit volume. Because prokaryotes become less metabolically efficient as they get bigger, most are quite small—six orders of magnitude smaller than eukaryotes, on average.[5]

There are some exceptions. For i...
The Law of Identity
by Chris_Leong — 2023-02-06T02:59:16.397Z

Summary: When we define the range of possible values for a variable X, we are fixing an ontology, that is, a way of carving up the space of values. The Law of Identity asserts that this ontology respects a given equivalence function.

Wikipedia defines the Law of Identity as follows: "In logic, the law of identity states that each thing is identical with itself". It is often written as X=X.

While this law seems straightforward, it is anything but once we start digging into what it actually means. The challenge is that it's very difficult to say what this law means without stating a tautology.

Take, for example, the definition above. What does it mean for a thing (let's say A, to be concrete) to "be identical with itself"?

Well, in order for this to make sense we need to have a model where A is not identical to itself which we can reject. If we don't have such a model to reject, then this statement will be tautological.

We can represent this using set theory as follows:

Let A be a set containing A1 and A2 (ie. A={A1,A2}). Here A is the "thing" and we're assigning it two separate sub-things that "are it" so that we can talk about them being equal to each other or not. We can think of A as corresponding to a congruence relation in set theory.
When checking if A is identical to itself, we'll be checking if A1=A2. If there were more than two variables, then we would do a pairwise comparison of elements. If instead of numbers, they are something like formulas applied to a specific value, then we'll have to evaluate them before making the comparison.

We can now consider some concrete examples (apologies if the examples are repetitive):

Let's suppose A1=100 represents a variable that we have in memory. We copy it such that we have another variable A2 in memory and if the copy operation happened successfully it should also be 100. However, we can also imagine an unreliable copy operation which often produces a value of 99 or 101. In this context, A being identical to itself can serve as a shorthand for the copy operation being reliable such that we can treat all copies as a single variable. If operations are regularly unreliable, then we will tend to assign each copy its own variable.
Let x=10 and suppose we have two functions f1 and f2. Similar to how we made A={A1,A2}

...
Robin Hanson on "Explaining the Sacred"
by Raemon — 2023-02-06T00:50:58.490Z
Link: https://www.overcomingbias.com/2023/02/explain-the-sacred.html

I found this post an interesting prompt. I'm thinking about it, and I'd enjoy hearing what other LessWrongers think about it.

Going ahead pasting the text here, hope that's okay:

The following are 45 correlates that I’ve collected of things called “sacred”. I invite any of you to offer a theory of the sacred that explains as many of these as you can, as simply as you can. (And to suggests edits of this list.)

Sacred things are highly (or lowly) valued. We politely revere, respect, & prioritize them.
We revere sacred beliefs as well as acts. We feel dirty when thoughts go near illicit ones.
Sacred is big, powerful, extraordinary. We fear, submit, & see it as larger than ourselves.
Sacred things matter for our health, luck, and other outcomes we care about.
We want the sacred “for itself”, rather than as a means to get other things.
Sacred things are either more homogenous, or more unique, whichever is better.
Sacred induces strong emotions: e.g., awe, joy, serenity, devotion, repulsion, & fear.
We get emotionally attached to the sacred; our stance toward it is part of our identity.
We desire to connect with the sacred, and to be more associated with it.
To approach the sacred, we use self-control to purify ourselves, sacrifice, & commit.
We enjoy sacrificing for the sacred, to purify, & respect sacred, including via odd beliefs.
Sacred brings us comfort & consolation in hard times; losing it can feel devastating.
We affirm & learn sacred via mythic stories & accounts of how we & it fit in a universe.
We have rules regarding how to approach sacred stuff, in part to protect us.
The sacred isn’t for use by commoners, or for common purposes.
Shared views about the sacred bind, define, and distinguish social groups.
Shared rituals & festivals bind & emotionally charge us, & help us to see sacred.
We want associates to share our view of and attachment to the sacred.
We get offended when others seem to deny our sacred views, and respond vigorously.
We feel more equal to each other regarding sacred things; status matters less there.
Either everyone (e.g. love) or very few (e.g. medicine) are entitled to opinions on sacred.
Charismatic leaders motivate, get acceptance in part via appeals, connections to sacred.
Experts of the sacred are more prestigious & trusted, get more job security.
Sacrificing for the sacred

...
Interview Daniel Murfet on Universal Phenomena in Learning Machines
by Alexander Gietelink Oldenziel — 2023-02-06T00:00:29.407Z

(The following is a heavily edited version of an interview with Daniel Murfet. For a primer on his research in singular statistical learning theory and its relation to AI safety see the excellent new explainer by Jesse Hoogland: "Neural networks generalize because of this one weird trick" )

[special thanks to Eddie Jean for editing]

Alexander: All right, so maybe you can start by introducing yourself.

Daniel: Sure. I'm

Daniel Murfet

, an algebraic geometer by training. I've worked in a variety of areas including mathematical physics, logic, and now statistical learning theory.

I'm fascinated by the possibility that universal phenomena in learning machines can be understood using singularity theory, much as universal phenomena in physics can be understood using singularity theory. Maybe some of that will be useful in understanding the deep issues that are raised by the current era of machine learning, in which we train larger and larger models on larger and larger data sets and observe emergent capabilities that are surprising and maybe a bit concerning.

Alexander: How did you get interested in singular learning theory?

Daniel: Let me think. I learned some functional programming languages and by going down that rabbit hole got interested in

linear logic

. And then I've thought about linear logic for a number of years. Then I thought about the semantics of linear logic and how to put more geometry in that, which led me to

differential linear logic

. Chasing that rabbit hole in turn led me to think about learning proofs and programs from data.

At the same time, friends of mine in California were working on deep learning as engineers. This was when I was at UCLA, probably around 2013. That’s when I started thinking that deep learning was interesting, but I was kind of coming at it from this very logical angle.

I was quite interested in the

Neural Turing Machine

. The NTM was an attempt to build a differentiable version of a Turing machine that you could train end-to-end. And I have some paper that I never published that is a kind of meld of differential linear logic and the NTM in some kind of crazy way.

This inspired me to rethink the

Curry-Howard correspondence

. If proofs are related to programs and you have a new way of creating programs by learning them, then what does this mean on the proof theoretic side? Does it ...
SolidGoldMagikarp (plus, prompt generation)
by Jessica Rumbelow — 2023-02-05T22:02:35.854Z

UPDATE (14th Feb 2023): ChatGPT appears to have been patched! However, very strange behaviour can still be elicited in the OpenAI playground, particularly with the davinci-instruct model.

More technical details here.

Further (fun) investigation into the stories behind the tokens we found here.

Work done at SERI-MATS, over the past two months, by Jessica Rumbelow and Matthew Watkins.

TL;DR

Anomalous tokens: a mysterious failure mode for GPT (which reliably insulted Matthew)

We have found a set of anomalous tokens which result in a previously undocumented failure mode for GPT-2 and GPT-3 models. (The 'instruct' models “are particularly deranged” in this context, as

janus

has observed.)
Many of these tokens reliably break determinism in the OpenAI GPT-3 playground at temperature 0 (which theoretically shouldn't happen).

Prompt generation: a new interpretability method for language models (which reliably finds prompts that result in a target completion). This is good for:

eliciting knowledge
generating adversarial inputs
automating prompt search (e.g. for fine-tuning)

In this post, we'll introduce the prototype of a new model-agnostic interpretability method for language models which reliably generates adversarial prompts that result in a target completion. We'll also demonstrate a previously undocumented failure mode for GPT-2 and GPT-3 language models, which results in bizarre completions (in some cases explicitly contrary to the purpose of the model), and present the results of our investigation into this phenomenon. Further technical detail can be found in a follow-up post. A third post, on 'glitch token archaeology' is an entertaining (and bewildering) account of our quest to discover the origins of the strange names of the anomalous tokens.

A rather unexpected prompt completion from the GPT-3 davinci-instruct-beta model.

Prompt generation

First up, prompt generation. An easy intuition for this is to think about feature visualisation for image classifiers (an excellent explanation here, if you're unfamiliar with the concept).

Feature visualisation of VGG network by Tim Sainburg.

We can study how a neural network represents concepts by taking some random input and using gradient descent to tweak it until it it maximises a particular activation. The image above shows the resulting inputs that maximise the output logits for the classes 'goldfish', 'monarch', 'tarant...
Are short timelines actually bad?
by joshc — 2023-02-05T21:21:41.363Z

Sam Altman recently

posted

the following:

I have seen very little serious discussion about whether short timelines are actually bad. This is surprising given that nearly everyone I talk to in the AI risk community seems to think that they are.

Of course, the question "was the founding of OpenAI net positive?" and "would it be good to accelerate capabilities in 2023?" are different questions. I'm leaning towards yes on the first and no on the second. I’ve listed arguments that factor into these questions below.

Reasons one might try to accelerate progress

Avoid/delay a race with China. If the language model boom happened 10 years from now, China might be a bigger player. Global coordination seems harder than domestic coordination. A lot harder. Perhaps the U.S. will have to shake hands with China eventually, but the more time we have to experiment with powerful systems before then the better. That corresponds to time demonstrating dangers and iterating on solutions, which is way more valuable than "think about things in front of a white board" time.

Smooth out takeoff. FLOPS get cheaper over time. Data is accumulating. Architectures continue to improve. The longer it takes for companies to invest ungodly amounts of money, the greater the potential overhang.

Shortening timelines in 2015 may have slowed takeoff

, which again corresponds to more time of the type that matters most.

Keep the good guys in the lead. We're lucky that the dominant AGI companies respect safety as much as they do. Sam Altman

recently

commented that "the bad case — and I think this is important to say — is, like, lights out for all of us." I'm impressed that he said this given how bad this sort of thing could be for business -- and this doesn't seem like a PR move. AI x-risk isn't really in the overton window yet.

The leading companies set an example. Maybe OpenAI’s hesitance to release GPT-4 has set a public expectation. It might now be

easier to shame companies

who don’t follow suit for being too fast and reckless.

Reasons to hit the breaks

There are lots of research agendas that don’t particularly depend on having powerful systems in front of us. Transparency research is the cat’s meow these days and has thus far been studied for relatively tiny models; in the extreme case,

two-layer attention only transformers

that I can run on my laptop. It takes t...
Stanzas On Power Calculation
by DirectedEvolution — 2023-02-05T19:15:14.958Z

Effects can be ignorable

- oh, real, yes, but fine -

So say, worst case scenario,

μ's right there on the line.

The power is the modest chance

Effects as small as μ's

Will have to be significant

In the C.I. I choose.

I pick a null hypothesis

and set out to reject.

But can I shrink my sample size

and find that small effect?

I am a stingy scientist

--- Blame NIH, not me!

I pick n = 25

What will that let me see?

I set up an alternative

but have to search a bit

for standard deviations next

by trawling prior lit

I must start by considering

how many tails make sense.

Then, μ and σ both in hand,

It's time for confidence!

Selecting my significance,

Say 95%

I find the corresponding z

For where the tails went

The sample average x̄ vs.

z times standard err'

Shows when the test rejects H-naught

Though α warns, "beware!"

A simple calculation, now

Of x̄ - μ

divided by the standard error

brings power into view

I end up with a last z score

And turn it to percent

The chance effects as small as μ

Turn out significant