trentmkelly/LessWrong-43k huggingface dataset:
How does GPT-3 spend its 175B parameters?

by Robert_AIZI — 2023-01-13T19:21:02.556Z
Link: https://aizi.substack.com/p/how-does-gpt-3-spend-its-175b-parameters

[Target audience: Me from a week ago, and people who have some understanding of ML but want to understand transformers better on a technical level.]

Free advice for people learning new skills: ask yourself random questions. In answering them, you’ll strengthen your understanding and find out what you really understand and what’s actually useful. And some day, if you ask yourself a question that no one has asked before, that’s a publication waiting to happen!

So as I was reading up on transformers, I got fixated on this question: where are the 175 billion parameters in the architecture? Not in the literal sense (

the parameters are in the computer

), but how are they “spent” between various parts of the architecture - the attention heads vs feed-forward networks, for instance. And how can one calculate the number of parameters from the architecture’s “size hyperparameters” like dimensionality and number of layers?

The goal of this post is to answer those questions, and make sense of this nice table from the GPT-3 paper, deriving the nparams column from the other columns.

Most of Table 2.1 from the GPT-3 paper. How can we derive the number of parameters from the “size hyperparameters” (rightmost columns)?

Primary Sources

Lots of resources about transformers conjure information from thin air, and I want to avoid that, so I’m showing all my work here. These are the relevant parts of the sources[1] we'll draw from:

Exhibit A: Equation 2 from the GPT-1 paper.

Exhibit B: Half of Figure 1 from the original transformer paper.

Exhibit C: An unnamed equation on page 5 in the original transformer paper. In the terminology of GPT, dk=dv=dhead, and h=nheads

Exhibit D: Equation 2 in the original transformer paper.

Three more details we’ll use, all from Section 2.1 of the GPT-3 paper:

The vocabulary size is [nvocab=]50257 tokens (via a reference to Section 2.3 of the GPT-2 paper)
The feed-forward networks are all a single layer which is “four times the size of the bottleneck layer”, so dff=4dmodel
“All models use a context window of nctx=2048 tokens.”

Variable abbreviations

I’ll use shorthand for the model size variables to increase legibility:

nlayers=xdmodel=ynheads=zdhead=wnvocab=vnctx=u

Where are the Parameters?

From Exhibit A, we can see that the original 1-hot encoding of tokens U is first converted to the initial

“residual stream”

h0...
[ASoT] Simulators show us behavioural properties by default
by Jozdien — 2023-01-13T18:42:34.295Z

I’m trying out making a few posts with less polish and smaller scope, to iterate more quickly on my thoughts and write about some interesting ideas in isolation

before having fully figured them out

. Expect middling confidence in any conclusions drawn, and occasionally

just chains of reasoning

without fully contextualized conclusions.

Let’s say we prompt a language model to do chain-of-thought reasoning while answering some question. What are we actually seeing? One answer I’ve seen people use, often implicitly, is along the lines that we’re seeing the underlying reasoning being done by the model to get at the answer.

I think this take is wrong, and in this post I’ll explain the frame I generally use for this.

Languages describe processes

How do we use text? Well, broadly we use it to describe our own thoughts, or to describe something that’s happening externally in the real world. In both of these cases, language is used as logs of real, mechanistic processes.

To be clear, I think these are very good and comprehensive logs of many relevant processes. Contrary to

some contexts

in which I’ve heard this description used, I think that large language models trained on these logs learn the processes underlying them pretty well, and end up understanding and being able to use the same (or pretty similar) mechanics at the limit.

The key point I want to make with the idea that text primarily functions as reports on processes is that this is what we’re observing with a language model’s outputs. One way to think about it is that our prompt shines a lens onto some specific part of a simulation[1], and the language model just keeps describing everything under the lens.

In other words, when we prompt simulators, what we should expect to see are the behavioural characteristics of the simulation, not the mechanistic characteristics of the underlying processes.

Focusing the lens onto the mechanics

I use the terms “behavioural and mechanistic characteristics” in a broader sense than is generally used in the context of something in our world - I expect we can still try to focus our lens on the mechanics of some simulacrum (they’re plausibly also part of the simulation, after all), I just expect it to be a lot harder because you have to make sure the lens actually is focused on the underlying process. One way to view this is that you’re still getting reports...
Money is a way of thanking strangers
by DirectedEvolution — 2023-01-13T17:06:36.547Z

In Ann Arbor, I live with 121,000 strangers. Together, we live among 10 million strangers in the state of Michigan, 332 million strangers in the USA, and nearly 8 billion strangers on Earth.

I don't mind doing favors for my friends and loved ones, or even for one of the 8 billion strangers if I happen to be face to face with them. I even like it. It makes me feel good to demonstrate, for the benefit of those present, the fact that we live in a society where most people are basically benevolent toward strangers. I also feel happy to see the other person feeling happy, or at least less stressed. These small pleasures have entrained a habit in me, over the course of my life, to do small favors almost automatically. When the stranger I have helped says "thank you," it means something to me.

But I've also noticed that I don't always feel that way. Sometimes, I notice my automatic consideration for strangers causing me to help others in ways that they'll never notice or appreciate. Sometimes, the people who do see get angry at me for it. For example, if I ride my bike to work, to keep myself fit so that I can do my job well, avoid getting sick in a way society will have to pay for, and spare the planet some CO2 emissions, sometimes drivers get randomly angry at me. And I'm not a bad bicyclist - I go with traffic, in bike lanes, wearing a helmet, with lights on my bike, stopping at stop signs.

I also read quite a lot of scientific literature. We want a scientifically literate society, which is why we teach about science in school. Sometimes, I learn things from my reading that are relevant to a conversation I am having. It's rare that I talk to strangers about what I read, unless those strangers are scientists themselves. But I've noticed that sometimes, when I tell my loved ones what I read in the study, and it happens to contradict what they currently think, they sometimes start acting a little more like a stranger for a while. You might have experienced this yourself, perhaps at Thanksgiving, if you celebrate.

As a third example, I lined up early and eagerly to get vaccinated. It wasn't easy to figure out how to get boosted, but after making a lot of phone calls, we found a CVS that would do it and waited there for a couple hours for our shots. It was a strangely joyless situation, with a bunch of strangers, lined up in an uncomfortable, fluorescently-lit CVS, not t...
Tracr: Compiled Transformers as a Laboratory for Interpretability | DeepMind 
by DragonGod — 2023-01-13T16:53:10.279Z
Link: https://arxiv.org/abs/2301.05062

Abstract

Interpretability research aims to build tools for understanding machine learning (ML) models. However, such tools are inherently hard to evaluate because we do not have ground truth information about how ML models actually work. In this work, we propose to build transformer models manually as a testbed for interpretability research. We introduce Tracr, a "compiler" for translating human-readable programs into weights of a transformer model. Tracr takes code written in RASP, a domain-specific language (Weiss et al. 2021), and translates it into weights for a standard, decoder-only, GPT-like transformer architecture. We use Tracr to create a range of ground truth transformers that implement programs including computing token frequencies, sorting, and Dyck-n parenthesis checking, among others. To enable the broader research community to explore and use compiled models, we provide an open-source implementation of Tracr at this https URL.

Twitter thread by Dr. Jim Fan
How we could stumble into AI catastrophe
by HoldenKarnofsky — 2023-01-13T16:20:05.745Z

This post will lay out a couple of stylized stories about how, if transformative AI is developed relatively soon, this could result in global catastrophe. (By “transformative AI,” I mean AI powerful and capable enough to bring about the sort of world-changing consequences I write about in my most important century series.)

This piece is more about visualizing possibilities than about providing arguments. For the latter, I recommend the rest of this series.

In the stories I’ll be telling, the world doesn't do much advance preparation or careful consideration of risks I’ve discussed previously, especially re: misaligned AI (AI forming dangerous goals of its own).

People do try to “test” AI systems for safety, and they do need to achieve some level of “safety” to commercialize. When early problems arise, they react to these problems.

But this isn’t enough, because of some unique challenges of measuring whether an AI system is “safe,” and because of the strong incentives to race forward with scaling up and deploying AI systems as fast as possible.

So we end up with a world run by misaligned AI - or, even if we’re lucky enough to avoid that outcome, other catastrophes are possible.

After laying these catastrophic possibilities, I’ll briefly note a few key ways we could do better, mostly as a reminder (these topics were covered in previous posts). Future pieces will get more specific about what we can be doing today to prepare.

Backdrop

This piece takes a lot of previous writing I’ve done as backdrop. Two key important assumptions (click to expand) are below; for more, see the rest of this series.

“Most important century” assumption: we’ll soon develop very powerful AI systems, along the lines of what I previously called PASTA. (Details not included in email - click to view on the web)
Robustness & Evolution [MLAISU W02]
by Esben Kran — 2023-01-13T15:47:16.384Z
Link: https://newsletter.apartresearch.com/posts/robustness-evolution-w02

Welcome to this week’s ML Safety Report where we talk about robustness in machine learning and the human-AI dichotomy. Stay until the end to check out several amazing competitions you can participate in today.

Watch this week's MLAISU on

YouTube

or listen to it on

Spotify

.

Robust Models

Robustness

is a

crucial aspect

of ensuring the safety of machine learning systems. A robust model is better able to adapt to new datasets and is less likely to be confused by unusual inputs. By ensuring robustness, we can prevent sudden misalignments caused by malfunction.

To test the robustness of models, we use adversarial attacks. These are inputs specially made to confuse the model and can help us create defense methods against these. There are many libraries for adversarial example generation in computer vision but the new attack method

TextGrad

creates adversarial examples automatically for text as well. It works under the two constraints of 1) text being much more discrete than images and therefore harder to modify without being obvious and 2) still ensuring fluent text, i.e. making the attacks hard to see for a human. You can see many more text attacks in the aptly named

TextAttack library

.

In the paper

“(Certified!!) Adversarial Robustness for Free!”

(yes, that is it’s name), they find a new method for making image models more robust against different attacks without training their own model during defense but using off-the-shelf models, something other papers have not achieved. Additionally, they do this and get the highest average certified defense rate against the competition.

Additionally,

Li, Li & Xie

investigate how to defend against the simple attack of writing a weird sentence in front of the prompt that can significantly confuse models in question-answering (QA) settings. They then extend this to the image-text domain as well and modify an image prompt to confuse during QA.

With these specific cases, is there not a way for us to generally test for examples that might confuse our models? The new

OpenOOD (Open Out-Of-Distribution) library

implements 33 different methods and represents a strong toolkit to detect malicious or confusing examples.

Their paper

details more of their approach.

​​

Another way we hope to

detect these anomalies

is by using interpretability methods to understand what happens inside the ne...
On Cooking With Gas
by Zvi — 2023-01-13T14:20:00.613Z

Research meta-analysis says gas stoves give off unsafe N02 levels and increase risk of childhood asthma by 34% and are responsible for 12.7% of childhood asthma.

None of this is new. The way safety panic works is that you focus on a particular risk this week and an attempt is made to make everyone’s life worse in its name.

Emily Oster takes a deeper dive on the original studies. She is not impressed.

For the most part, the other papers in the meta-analysis have similar features. They tend to show positive correlations between asthma and gas stove exposure, but the effects are a bit noisy and they do not necessarily paint a consistent picture. A 2013 paper finds, for example, a link with asthma for girls but not boys, and no links with other respiratory issues.

What to take from this? We know that gas stoves emit nitrogen oxides and that, in general, those are not good. We know that air pollution, in general, is bad for respiratory symptoms, including asthma. So it seems very plausible that there is some link here. However: the magnitude is likely small. In most of the estimates, it’s small. And, beyond that, we do not see the kind of smoking gun in any of these data that would suggest a really consistent link.

Another way to put this is that there are clearly many, many factors other than gas stoves that explain asthma. Some of these may also be environmental. But I’m skeptical that gas stoves play a huge role, as would be suggested by the new paper.

The link on the state-wide level runs the other way.

I am confident gas stoves do not prevent asthma because physics. One still notes that if this graph had been reversed it would be part of the case to ban such stoves.

One explanation for the reversed correlation is that gas stoves cost more and positively correlate with income. This also suggests they are a superior good. Rich people choose to cook with gas. The spike at the top of the income distribution supports this. As does 80% of restaurants, who care the most about quality, choosing gas stoves, versus 40% of households. Alternatively, some of this could be because richer people live in older and more expensive buildings that happen to have gas stoves. If new construction is restricted in the most expensive areas, and old construction has more gas stoves, that could create this effect as well.

Emily’s practical advice seems solid, if ...
Beware safety-washing
by Lizka — 2023-01-13T13:59:42.054Z

Tl;dr: don’t be fooled into thinking that some groups working on AI are taking “safety” concerns seriously (enough).[1]

Outline

Two non-AI examples

Greenwashing
Humanewashing

Definition of safety-washing
What are the harms?
What can (and should) we do about this?

Note: I’m posting this in my personal capacity. All views expressed here are my own. I am also not (at all) an expert on the topic. [Also, I originally wrote this for the EA Forum and have not changed it for the cross-post. I don't know what that changes, but it might change something.]

Two non-AI examples

Greenwashing

Companies “greenwash” when they mislead people into incorrectly thinking that their products or practices are climate and environment-friendly (or that the company focuses on climate-friendly work).

Investopedia

explains:

Greenwashing is an attempt to capitalize on the growing demand for environmentally sound products.
The term originated in the 1960s, when the hotel industry devised one of the most blatant examples of greenwashing. They placed notices in hotel rooms asking guests to reuse their towels to save the environment. The hotels enjoyed the benefit of lower laundry costs.

Wikipedia

: “[Jay Westerveld, the originator of the term] concluded that often the real objective was increased profit, and labeled this and other profitable-but-ineffective ‘environmentally-conscientious’ acts as greenwashing.” (Wikipedia also provides

a long list of examples

of the practice.)

I enjoy

some of the parody/art

(responding to

things like this

) that comes out of noticing the hypocrisy of the practice.

Humanewashing

A similar phenomenon is the “humanewashing” of animal products. There’s

a

Vox

article

that explains this phenomenon (as it happens in the US):

A carton of “all natural” eggs might bear an illustration of a rustic farm; packages of chicken meat are touted as “humanely raised."

In a few cases, these sunny depictions are accurate. But far too often they mask the industrial conditions under which these animals were raised and slaughtered.

Animal welfare and consumer protection advocates have a name for such misleading labeling: “humanewashing.” And research suggests it’s having precisely the effect that meat producers intend it to. A recent

national survey

by C.O.nxt, a food marketing firm, found that animal welfare and “natural” claims on meat, dairy, and egg pa

...
Some Arguments Against Strong Scaling
by Joar Skalse — 2023-01-13T12:04:27.444Z

There are many people who believe that we will be able to get to AGI by basically just scaling up the techniques used in recent large language models, combined with some relatively minor additions and/or architectural changes. As a result, there are people in the AI safety community who now predict timelines of less than 10 years, and structure their research accordingly. However, there are also people who still believe in long(er) timelines, or at least that substantial new insights or breakthroughts will be needed for AGI (even if those breakthroughts in principle could happen quickly). My impression is that the arguments for the latter position are not all that widely known in the AI safety community. In this post, I will summarise as many of these arguments as I can.

I will almost certainly miss some arguments; if so, I would be grateful if they could be added to the comments. My goal with this post is not to present a balanced view of the issue, nor is it to present my own view. Rather, my goal is just to summarise as many arguments as possible for being skeptical of short timelines and the "scaling is all you need" position.

This post is structured into four sections. In the first section, I give a rough overview of the scaling is all you need-hypothesis, together with a basic argument for that hypothesis. In the second section, I give a few general arguments in favour of significant model uncertainty when it comes to arguments about AI timelines. In the third section, I give some arguments against the standard argument for the scaling is all you need-hypothesis, and in the fourth section, I give a few direct arguments against the hypothesis itself. I then end the post on a few closing words.

Glossary:

LLM - Large Language Model
SIAYN - Scaling Is All You Need

The View I'm Arguing Against

In this section, I will give a brief summary of the view that these arguments oppose, as well as provide a standard justification for this view. In short, the view is that we can reach AGI by more or less simply scaling up existing methods (in terms of the size of the models, the amount of training data they are given, and/or the number of gradient steps they take, etc). One version says that we can do this by literally just scaling up transformers, but the arguments will apply even if we relax this to allow scaling of large deep learning-based next-token predictors, e...
Where do you find people who actually do things?
by Ulisse Mini — 2023-01-13T06:57:05.620Z

Lesswrongers are typically bad at actually doing things. Given how we're pulled towards the average of the people we spend the most time with, what are good ways/places to find people who actually do things?

Edit: By "actually do things" I mean agentic, don't theorize too much, gain contact with reality. The opposite of the armchair smart person who doesn't Do Anything (the latter I would like to be less of)
Could Simulating an AGI Taking Over the World Actually Lead to a LLM Taking Over the World? 
by simeon_c — 2023-01-13T06:33:35.860Z

As you know, large language models can be understood as simulators or more simply as models predicting what token would plausibly appear next after a first sequence of token.

As a consequence, when you ask a LLM to simulate extreme opinions of an AI, it converges very quickly towards "we should eradicate humans" (https://twitter.com/Simeon_Cps/status/1599470463578968064?s=20&t=rj2357Vof9sOLnIma6ZzIA). Other instances of that were observed on other language models.

As a consequence, is it plausible that some very powerful simulators would actually try to take over the world? Write a very detailed plan to take over the world?

It seems to me that yes. More worryingly, if LLMs trained with RL keep their "simulator" aspect, it could be quite natural for an LLM with agentic property to behave still a bit as a simulator while having the capacity to operate in the real-world. And thus it would make a scenario where a simulator literally tries to take over the world because it was asked to do what an evil AGI would do more likely.

And if it's true, do you think that it's a consideration to have in mind or that it's in practice irrelevant?