trentmkelly/LessWrong-43k huggingface dataset:
Finding Neurons in a Haystack: Case Studies with Sparse Probing
by wesg — 2023-05-03T13:30:30.836Z
Link: https://arxiv.org/abs/2305.01610

Abstract

Despite rapid adoption and deployment of large language models (LLMs), the internal computations of these models remain opaque and poorly understood. In this work, we seek to understand how high-level human-interpretable features are represented within the internal neuron activations of LLMs. We train $k$-sparse linear classifiers (probes) on these internal activations to predict the presence of features in the input; by varying the value of $k$ we study the sparsity of learned representations and how this varies with model scale. With $k=1$, we localize individual neurons which are highly relevant for a particular feature, and perform a number of case studies to illustrate general properties of LLMs. In particular, we show that early layers make use of sparse combinations of neurons to represent many features in superposition, that middle layers have seemingly dedicated neurons to represent higher-level contextual features, and that increasing scale causes representational sparsity to increase on average, but there are multiple types of scaling dynamics. In all, we probe for over 100 unique features comprising 10 different categories in 7 different models spanning 70 million to 6.9 billion parameters.

See twitter summary here.

Contributions

In the first part of the paper, we outline several variants of sparse probing, discuss the various subtleties of applying sparse probing, and run a large number of probing experiments. In particular, we probe for over 100 unique features comprising 10 different categories in 7 different models spanning 2 orders of magnitude in parameter count (up to 6.9 billion). The majority of the paper then focuses on zooming-in on specific examples of general phenomena in a series of more detailed case studies to demonstrate:

There is a tremendous amount of interpretable structure within the neurons of LLMs, and sparse probing is an effective methodology to locate such neurons (even in superposition), but requires careful use and follow up analysis to draw rigorous conclusions.
Many early layer neurons are in superposition, where features are represented as sparse linear combinations of polysemantic neurons, each of which activates for a large collection of unrelated $n$-grams and local patterns. Moreover, based on weight statistics and insights from toy models, we conclude that the first 25\% of fully connected layers

...
Monthly Roundup #6: May 2023
by Zvi — 2023-05-03T12:50:01.009Z

Bad News

Wild pigs continue to multiply and do increasing amounts of damage as they spread throughout America. Perhaps in the past we might have been willing to do what it would take to solve such problems. Today, we are not, and the situation gets steadily worse, and likely will continue to get worse.

What does this imply about our ability to handle other problems? If we can’t even get our act together to deal with a bunch of pigs, saying it’s an impossible thing to hope to deal with ‘within our lifetime’ what chance would we have against, say, a rouge AI loose on the internet? Or could one turn it around, since we are clearly more intelligent than pigs, yet failing to deal with them? One could say that we are proving ourselves to be not so smart, here.

I think that many people who expect humanity to be all right once AGI shows up are imagining it being a lot like how we are letting the wild pigs multiply. I’d urge people to think about whether that is a smart hope to have.

Imgur will be removing old and unused images not tied to an account, also all the porn. Old websites will continue steadily looking worse. I am very happy that my work has multiple backup copies.

Chile nationalizing its Lithium industry.

Patrick McKenzie tells us: Japanese credit card companies think it’s fine for their card to randomly not work for a week every so often?

Patrick McKenzie also illustrates how bad payment experiences can be, even without intentional sabotage. Luckily I have never seen it that bad, and I am guessing Japanese experiences get a lot worse a lot more often than American ones. I imagine how many customers you would love in the USA with such tactics and the answer is most of them.

Microsoft Windows update causes Windows to, for many uses, pull up the system settings menu when users click on a browser, if and only if both the browser is your default browser and its name is “Google Chrome.” This seems to have been in response to Google Chrome putting in a button to make Chrome your default browser.

Is there any argument that a retroactive huge pay raise across the board for public officials, in this case cops, isn’t corruption and the robbing of the public purse? The whole package seems engineered to pay everyone involved massively more money than the headline numbers while also granting benefits that cost a lot more than they are valued, all based on continuous ga...
How much do personal biases in risk assessment affect assessment of AI risks?
by Gordon Seidoh Worley — 2023-05-03T06:12:57.001Z

Had a stray thought that I think is worth exploring: perhaps disagreements over how to respond to AI risks are heavily influenced by personal biases in how to assess risk in general such that very different conclusions can be drawn from evaluating the same evidence and arguments.

This sort of thing isn't without precedent (e.g. Bourget, David, and David J. Chalmers. “What Do Philosophers Believe?”). We can roughly imagine a few possible correspondences:

high concern over AI x-risks most common among people with

high anxiety
low trust of strangers
home bodies

middling concern

average psychological profile on various metrics

low concern

low anxiety/fear
high trust of strangers, or at least highly resilient to betrayal
thrill seekers

For archetypical examples of people who I'd consider high, medium, and low AI risk:

Eliezer Yudkowsky - high
Paul Christiano - middle
Yann LeCunn - low

I don't know if any of them fit my toy profiles. Honestly I'm not really sure what might correspond, if anything; these are just easy guesses to give some flavor to the idea.

So, if you've thought a lot about AI risks, and especially if you're actively working on AI in some capacity, I'd appreciate it if you left an answer filling out this template so we can see if the anecdata suggest anything worth exploring further. This might help with identify places where disagreement persists not because of disagreements about evidence, but personal disagreements about how to weigh and evaluate the risks the evidence suggests exist. Knowing that might prove useful to bridging some disagreements.

Answer Template[1]:

Level of AI risk concern: high/medium/low

General level of risk tolerance in everyday life: high/medium/low

Brief summary of what you do in AI: mostly if you're not famous to help identify you

Anything weird about you: are you unusually anxious, calm, whatever? do people tell you that you're the most X person they know? not necessarily a full psychological profile just some key facts to give a sense of your personality

^

I realize this template is not very well constructed. It's because I'm not quite sure what we're looking for, if anything, so it's relatively open ended in the hopes that the answers will help make it clearer what I should have asked.
Communication strategies for autism, with examples
by stonefly — 2023-05-03T05:25:16.387Z

This is a description of communication strategies which have been helpful to me in social situations where I was confused. It is relatively unedited; some of the content may be obvious and banal, or redundant.

[Edited to move out of footnote, thanks Aorou] I am autistic, and often have had difficulty with communication. These are some things these strategies have helped me with:

-difficulty distinguishing tone of conversations, difficulty speaking in a particular tone or matching the tone of a conversation

- feeling afraid to "say the wrong thing"

-speaking in more literal terms than most people, interpreting statements more literally than they were intended

-feeling like people are engaged in a social game that I don't understand

- and other difficulties in the genre of these listed

Hopefully this post will be helpful to someone with similar difficulties; it would have been helpful to me if I had seen it several years ago.

[End of edit]

strategies / useful framings

If the literal content seems not to make sense, often the subtext or mood is the point.

One time I was talking to two people and they both kept pointing to their things and saying in an exaggerated angry tone "ugh, this [object] sucks, it's so [adjective]". I thought that they were trying to request to borrow my things, so I offered. But they declined, looking confused, and one said "no, that's ok, I'm just in a bad mood today." I then realized that the point of what they were saying before was to communicate their mood, not anything to do with the objects they were referencing. The subtext of "this [object] sucks" was "I am in a mood where this minor inconvenience bothers me."

Pay attention to other people's responses to a statement.

if I had noticed that neither person was treating the other's complaints as serious, this would have led me to consider what else they might be trying to communicate indirectly.

Many times, when I am confused by a conversation, I find it useful to think of people as communicating by acting out roles that are interesting/useful/funny/etc, rather than communicating directly.[1]

In the example above, I said that the subtext of "this [object] sucks" was "I am in a mood where this minor inconvenience bothers me." But I don't actually think either of the people were annoyed in a way that made those specific inconveniences important to them. Instead, th

...
Understand how other people think: a theory of worldviews.
by spencerg — 2023-05-03T03:57:17.710Z

This is a linkpost from ClearerThinking.org. We've included some excerpts of the article below, but you can see the full post here.

A libertarian, a socialist, an environmentalist, and a pro-development

YIMBY

watch an apartment complex being built. The libertarian is pleased - "the hand of the market at work!" - whereas the socialist worries that the building is a harbinger of gentrification; the YIMBY sees progress, but the environmentalist is concerned about the building’s carbon footprint. They’re all seeing the same thing, but they understand it differently, because they inhabit different worldviews.

We can think of worldviews as

snow globes

. We each occupy our own snow globe and, when we’re inside it, it can seem like the whole world. If it’s snowing in our snow globe, we think it’s snowing everywhere; if our snow globe is made of green glass, everything looks green to us. We might not even realize that there’s anything beyond our snow globe! But if we can step outside, we see that our view from inside was only part of a much larger picture. If you can step outside your snow globe - and visit other people’s - you’ll be able to see a more accurate representation of the world and communicate better with others. For every snow globe you master, you’ll gain a powerful new lens through which to see the world - and you’ll see that no single snow globe has all the answers.

Image generated using the AI DALL•E 2

What makes a worldview?

Worldviews are a type of story we learn about how the world works and about what things matter and why. In this post, we put forward a theory of worldviews that will help you understand how different worldviews work. We call this "Snow Globe Theory." Every worldview includes many beliefs, but after reflecting on a wide variety of worldviews, we believe that almost every one has four central components. There are other common elements that some worldviews have but others don’t - for example, a strong culture, or a theory about trustworthy sources of knowledge – but this article will focus on these four central elements:

What is good?

Where do good and bad come from?

Who deserves the good?

How can you do good or be good?

You can understand a worldview quite well if you know what thoughtful people with that worldview would all answer in response to these four questions. While each individual member of a group will have s...
"Copilot" type AI integration could lead to training data needed for AGI
by anithite — 2023-05-03T00:57:41.752Z

TLDR:full task completion training data will soon be available, leading to much more capable agentic AI

Capabilities predictions:

Agentic AIs for all white collar tasks
creative problem solving might be solved

Business predictions:

MS/GOOG try to capture all the data (web browsing, (web/native)application interactions)
(maybe)creepy examples attempting better un-black-boxing of humans doing the problem solving

watching the user's face
eye tracking

Caveats:

TOS/legal limits on data retention/use for training
security(EG:prompt injection) for finished systems

TLDR_END

Microsoft is introducing copilot for office 365. Google is integrating AI into its products. Economic motivation for automating white collar task labor hours is obvious even if only to sell to workers directly. Where can the needed training data be gathered? Companies using MS tech stack (Windows(OS), Edge(browser), outlook(email), office365(documents)) are fully set up to capture all human computer interactions. Google can do the same via Chrome/ChromeOS for web application based workflows.

Ways to make better agentic AIs:

do task unreliably and let human select among end results

Microsoft low code/ no code
basic approval feedback system (EG:chatgpt thumbs up)

watch people doing a task to generate a problem solving log for that task

(1) leads to refinement and reliability improvement for existing capabilities. (2) leads to new capabilities altogether

Useful characteristics of problem solving data

homogenous

real time human computer interaction logs
no missing steps (except those that occur inside human brain)

tagged by user

can use "user vector" to condition model output as in image models conditioned by text
learn from the dumbest act like the smartest
less RL fine tuning required

Capability Predictions

This leads to capable narrow domain agents

creativity is still hard and 99%+ of humans aren't doing creative problem solving in day to day life
capturing the 1% could have huge impacts

can do RL or conditioning to elicit creative problem solving behavior
curation is easier than creation allowing lower skill humans to give feedback

EG:"Rate the automated tech support assistant"

This approach tops out somewhere unless RSI/takeoff occurs somehow

skill synergies may arise (EG:the model writes code to automate tasks rather than completing them one at a time)

this would be the "creative" part that leads to true AGI when fully generalized

...
Averting Catastrophe: Decision Theory for COVID-19, Climate Change, and Potential Disasters of All Kinds
by JakubK — 2023-05-02T22:50:59.867Z
Link: https://nyupress.org/9781479808489/averting-catastrophe/

Averting Catastrophe is a book by Cass Sunstein that NYU Press published in April 2021. I'm surprised that nobody has mentioned it on the EA Forum before (AFAICT). Quoting from the NYU Press page:

Averting Catastrophe explores how governments ought to make decisions in times of imminent disaster. Cass R. Sunstein argues that using the “maximin rule,” which calls for choosing the approach that eliminates the worst of the worst-case scenarios, may be necessary when public officials lack important information, and when the worst-case scenario is too disastrous to contemplate. He underscores this argument by emphasizing the reality of “Knightian uncertainty,” found in circumstances in which it is not possible to assign probabilities to various outcomes.

The Google Books page includes a table of contents:

Introduction
What We Don't Know
With and Without Numbers
The Maximin Principle
The Precautionary Principle
Uncertainty
Objections
Irreversibility
Departments and Agencies

I found this book while researching the precautionary principle. Sunstein has written multiple papers on this topic, including two that seem particularly relevant to x-risk:

"Irreversible and Catastrophic"
"The Catastrophic Harm Precautionary Principle"

Sunstein also wrote Laws of Fear: Beyond the Precautionary Principle, which Cambridge University Press published in 2005.

See Sunstein's Wikipedia page for a (somewhat) comprehensive list of his books. Also note that he appeared on the 80,000 Hours Podcast in 2019.
A Case for the Least Forgiving Take On Alignment
by Thane Ruthenis — 2023-05-02T21:34:49.832Z

1. Introduction

The field of AI Alignment is a pre-paradigmic one, and the primary symptom of that is the wide diversity of views across it. Essentially every senior researcher has their own research direction, their own idea of what the core problem is and how to go about solving it.

The differing views can be categorized along many dimensions. Here, I'd like to focus on a specific cluster of views, one corresponding to the most "hardcore", unforgiving take on AI Alignment. It's the view held by people like Eliezer Yudkowsky, Nate Soares, and John Wentworth, and not shared by Paul Christiano or the staff of major AI Labs.

According to this view:

We only have one shot. There will be a sharp discontinuity in capabilities once we get to AGI, and attempts to iterate on alignment will fail. Either we get AGI right on the first try, or we die.
We need to align the AGI's values precisely right. "Rough" alignment won't work, niceness is not convergent, alignment attained at a low level of capabilities is unlikely to scale to superintelligence.
"Dodging" the alignment problem won't work. We can't securely hamstring the AGI's performance in some domain without compromising the AGI completely. We can't make it non-consequentialist, non-agenty, non-optimizing, non-goal-directed, et cetera. It's not possible to let an AGI keep its capability to engineer nanotechnology while taking out its capability to deceive and plot, any more than it's possible to build an AGI capable of driving red cars but not blue ones. They're "the same" capability in some sense, and our only hope is to make the AGI want to not be malign.
Automating research is impossible. Pre-AGI oracles, simulators, or research assistants won't generate useful results; cyborgism doesn't offer much hope. Conversely, if one such system would have the capability to meaningfully contribute to alignment, it'd need to be aligned itself. Catch-22.
Weak interpretability tools won't generalize to the AGI stage, as wouldn't other methods of "supervising" or "containing" the AGI.
Strong interpretability, perhaps rooted in agent-foundations insights, is promising, but the bar there is fairly high.
In sum: alignment is hard and requires exacting precision, AI can't help us with it, and instantiating an AGI without robustly solving alignment is certain to kill us all.

I share this view. In my case, there's a simple generator of i...
Are Emergent Abilities of Large Language Models a Mirage? [linkpost]
by Matthew Barnett — 2023-05-02T21:01:33.142Z
Link: https://arxiv.org/abs/2304.15004

Recent work claims that large language models display emergent abilities, abilities not present in smaller-scale models that are present in larger-scale models. What makes emergent abilities intriguing is two-fold: their sharpness, transitioning seemingly instantaneously from not present to present, and their unpredictability, appearing at seemingly unforeseeable model scales. Here, we present an alternative explanation for emergent abilities: that for a particular task and model family, when analyzing fixed model outputs, one can choose a metric which leads to the inference of an emergent ability or another metric which does not. Thus, our alternative suggests that existing claims of emergent abilities are creations of the researcher's analyses, not fundamental changes in model behavior on specific tasks with scale. We present our explanation in a simple mathematical model, then test it in three complementary ways: we (1) make, test and confirm three predictions on the effect of metric choice using the InstructGPT/GPT-3 family on tasks with claimed emergent abilities, (2) make, test and confirm two predictions about metric choices in a meta-analysis of emergent abilities on BIG-Bench; and (3) show how similar metric decisions suggest apparent emergent abilities on vision tasks in diverse deep network architectures (convolutional, autoencoder, transformers). In all three analyses, we find strong supporting evidence that emergent abilities may not be a fundamental property of scaling AI models.

This result seems important for two reasons:

If AI abilities are predictable, then we can forecast when we'll get dangerous capabilities ahead of time, rather than being taken by surprise. This result strengthens the case for a research program of devising a ton of interesting benchmarks to measure how capabilities are improving as a function of scale.
It provides some evidence against the idea that "understanding is discontinuous", and that important AI abilities will suddenly click together at some level, which is a very loose description of what I understood to be one of the primary intuitions behind AI foom.
Does descaling a kettle help? Theory and practice
by philh — 2023-05-02T20:20:04.401Z

I've heard that descaling a kettle makes it more efficient, and can save me time and money.

Okay, but how much? For some reason my my intuition says it'll be basically unnoticable. Let's try to figure it out, and then actually try it.

The Models

There's a first-order approximation which says this should be impossible: heating things up is 100% efficient. What's the limescale going to do, convert some electricity into forms of energy other than heat?

No, but it might cause the heat to go to the wrong places. I think there are two ways that could happen. The first is if the limescale itself has a high heat capacity, meaning it takes a lot of energy to heat up. That doesn't seem likely to me; I think there's likely less than 10 g of it, and I think it probably has less heat capacity than the same mass of water (because I think water's is higher than most other things). I don't think adding 10 ml water (weighing 10 g) to my kettle will significantly effect the boiling time.

Spot check: water's specific heat capacity is about 4 J/K·g. So to heat 10 g water by 100 K needs about 4000 J. My kettle is 2200 W according to the email I got when I bought it, so an extra 10 ml should take about 2 s longer to boil.

Also, I normally boil about 500 ml of water, so we'd expect 10 ml extra to make about a 2% difference. (I don't have a strong intuition for how long my kettle normally takes to boil. 1-3 minutes? The above calculation suggests it should be around 100 s.)

The second way the limescale could matter is if it's a good thermal insulator. Then the metal heating element gets significantly above 100°C before the water boils. And from my googling, it seems like this is the reason people give that descaling a kettle is helpful. How much effect would this have?

This page says the thermal conductivity is 2.2 W/m·K. I don't remember studying this in school, and I don't find that wikipedia page very clear. But I think maybe this means: the rate of energy transfer (W) from the heating element is 2.2 W/m·K, multiplied by the surface area of the heating element (m²), divided by the thickness of the limescale (m), multiplied by the temperature difference (K) between the heating element and the water. The units check out at least.

This implies, if there's no limescale, that the power transfer should be infinite, regardless of the size of the heating element. That's clearly not how thing...