trentmkelly/LessWrong-43k huggingface dataset:
Concrete Steps to Get Started in Transformer Mechanistic Interpretability
by Neel Nanda — 2022-12-25T22:21:49.686Z
Link: www.neelnanda.io/mechanistic-interpretability/getting-started

Disclaimer: This post mostly links to resources I've made. I feel somewhat bad about this, sorry! Transformer MI is a pretty young and small field and there just aren't many people making educational resources tailored to it. Some links are to collations of other people's work, and I link to more in the appendix.

Introduction

Feel free to just skip the intro and read the concrete steps

The point of this post is to give concrete steps for how to get a decent level of baseline knowledge for transformer mechanistic interpretability (MI). I try to give concrete, actionable, goal-oriented advice that I think is enough to get decent outcomes - please take this as a starting point and deviate if something feels a better fit to your background and precise goals!

A core belief I have about learning mechanistic interpretability is that you should spend at least a third of your time writing code and playing around with model internals, not just reading papers. MI has great feedback loops, and a large component of the skillset is the practical, empirical skill of being able to write and run experiments easily. Unlike normal machine learning, once you have the basics of MI down, you should be able to run simple experiments on small models within minutes, not hours or days. Further, because the feedback loops are so tight, I don’t think there’s a sharp boundary between reading and doing research. If you want to deeply engage with a paper, you should be playing with the model studied, and testing the paper’s basic claims.

The intended audience is for people new-ish to Mechanistic Interpretability but who know they want to learn about it - if you have no idea what MI is, check out this Mech Interp Explainer, Circuits: Zoom In or

Chris Olah’s overview post

.

Defining “Decent Baseline”

Here’s an outline of what I mean when I say “a decent level of baseline knowledge”

A good grounding in the key concepts of machine learning and mechanistic interpretability
An intuition for how a transformer actually works on a mechanical level - what the moving parts are, how it all fits together, and how to reason about the overall system
Familiarity with tooling, such that you can easily spin up a model and run quick and dirty experiments.
A rough map of the literature, what’s known in the field, and big categories of open problems - not necessaril

...
It's time to worry about online privacy again
by Malmesbury — 2022-12-25T21:05:30.977Z

As we all know, if you have nothing to hide, you have nothing to fear. Nobody cares about your private life. You are not an important geopolitical target. Nobody's going to spy on you to know what weird pornography you watch.

And so, around 2015, people gave up on online privacy. Everyone stopped worrying about corporations and governments having full access to their data. In hindsight, I have to admit that things didn't go as bad as some feared. But I don't think this will last.

1. Radioactive decay

Based on real-life events: you're a biologist at the Bad Pathogen Research Institute. You receive an email from a graduate student whose name sounds vaguely familiar. She needs to measure radio-labeled samples with a scientific instrument but, unfortunately, you used it yesterday and you forgot to log out. Now it's locked with your password and she can't connect or even reboot. She's asking you to come as soon as you can to unlock it – as radioactivity decays, the signal is vanishing every minute. Sadly, you are attending a talk on the other side of the city, it's 45 minutes by bike and it's snowing.

Obviously, you would never send your credentials by e-mail, right? Right?

This could, in principle, be phishing. Technically, a cunning spy could have stalked you, figured out your schedule, and crafted a deceptive e-mail to steal your password. But you know it's probably not the case, because nobody cares about your passwords enough to do something so complicated. So you send your credentials to the grad student using a one-time secret sharing link and everything is fine.

I like to think that I can't be scammed because I know the ways of 1337 h4xx0rs well enough so they can't reach me. Of course, this is not true. I could totally be scammed, attackers simply don't have any interest in deploying the amount of energy it takes to scam me.

That's why some people get phished and not others. It depends on two things:
🅰️. How much effort it takes to set up a scam so a given target falls for it
🅱️. How much effort an attacker is ready to dedicate to scamming that target

If 🅰️ is lower than 🅱️, the target gets scammed. If 🅱️ is lower than 🅰️, it's not worth it. On one end (high 🅰️, high 🅱️), you have hackers leaking e-mails from an important government official. On the other end (low 🅰️, low 🅱️), your grandfather receives an e-mail saying a hacker ha...
[Hebbian Natural Abstractions] Mathematical Foundations
by Samuel Nellessen — 2022-12-25T20:58:03.423Z
Link: https://www.snellessen.com/mathematical-foundations-of-hebbian-natural-abstractions

TL;DR: We showed how Hebbian learning with weight decay could enable a) feedforward circuits (one-to-many) to extract the first principal component of a barrage of inputs and b) recurrent circuits to amplify signals which are present across multiple input streams and suppress signals which are likely spurious.

Short recap

In the last post, we introduced the following idea:

We don’t have a way to formalize concepts and transfer them so that an alien agent understands them. Think back to the tree conversation - how would you describe a tree to an AGI?
Yet, we aren’t facing the same issues when communicating concepts between humans.
We concluded that this must have something to do with how the brain learns.

Introduction

In this post, we introduce a learning rule that is (presumably) used by biological brains and connect it with the type of circuits that emerge in the brain under different types of input. This connection will serve as the mathematical foundation for exploring how the brain forms natural abstractions.

We will consider two scenarios in the brain: (a) how the brain learns in a "many-to-one" setup, where several neurons project onto a single neuron in another layer, and (b) how the brain learns in an "all-to-all" setup, where several neurons in a layer are connected to each other.

The current post tries to make the derivations accessible but still focuses on mathematics. In the next post, we will discuss the implications of our derivations by delving into neuroscience and related topics.

Definitions and notation:

The notation used in this post:

⟨xi⟩ refers to the statistical average of a variable xi.
xi and xj refer to the activation of neuron i and neuron j. Without loss of generality, we will assume that the activity of all neurons centres around 0 so that ⟨xi⟩=0.
wij refers to the strength of the synapse connecting neuron i to neuron j. A higher weight implies a higher likelihood of neuron i triggering neuron j.
We define the output of a single neuron as y=∑ni=1wixi.
We will use the delta notation for derivatives, i.e. Δwij=dwijdt=.wij. This means that from t to t+1, the weight changes with: wij(t+1)=wij+Δwij.
For the derivations, we will switch to vector notation in some places. When we do that, we remove indices and make the letter bold.

Definition of Natural abstractions: Abstractio...
Oracle AGI - How can it escape, other than security issues? (Steganography?)
by RationalSieve — 2022-12-25T20:14:09.834Z

Let's suppose we have an AGI running on a perfectly physically isolated computer that may be misalligned. It only communicates with select humans through a perfectly secure text channel.
How can such an AGI escape?
While don't think that AGI can manipulate EVERY person to do what AGI wants, it's better to be cautious.
But recently, I thought of steganography. An AGI can tell a human to convey a message to the public that looks helpful at the first glance, but in fact contains a steganographic message with instructions for building antimatter fusion reactor/nanobots/bioweapon/whatever.
What other ways do you think can such an AGI escape?
YCombinator fraud rates
by Xodarap — 2022-12-25T19:21:52.829Z

Summary

I estimate that 1-2% of $100M+ YCombinator-backed companies have faced serious allegations of fraud.

Details

I’m interested in better understanding the base rates of fraud in high-growth companies. YCombinator is a convenience sample for “high-growth companies”, as they are relatively public about who they back, and act as a filter for e.g. “obviously fraudulent” companies.
I was able to find 3 companies which closed due to (alleged) fraud, two more which had their business substantially impacted by fraud claims, and three which had mild business impacts from fraud claims.
YCombinator has incubated 3,951 companies, which gives a naïve rate of ~0.1% of companies having major allegations of fraud.
I estimate that there are around 400 YCombinator-incubated companies with a valuation over $100M.[1] 3 of the 5 companies with major fraud had $100M+ valuations, implying that 3/400 = 1% of $100M+ companies have major fraud charges. I think that this 1% number is probably more useful than the previous 0.1% number, because I expect that many smaller companies were fraudulent but no one bothered to charge them or report on it.
I also wouldn’t be surprised to learn if I missed some cases, though I’d be surprised if I missed more than half of the $100M+ cases.
So overall, I estimate that maybe 1-2% of $100M+ YCombinator-backed companies have faced serious charges of fraud.

Additional Comments

It’s notable that almost half of the companies on this list are financial services, which I believe makes them overrepresented.
I expect that YCombinator is better at filtering out fraudulent companies than other angel investors, but they are also more likely to generate highly valuable companies. I’m not sure how these factors balance out, but would guess that they are roughly equal, implying that 1-2% of all high-growth $100M+ companies face serious fraud charges.

Data

Company	Valuation	Severity	Notes
uBiome

$298M

Fatal	“the company shut down in 2019 following an investigation into possible insurance fraud… Since 2021, the FBI has considered [the founders] to be fugitives.” (

Wikipedia

)
LendUp

$500M

Fatal	"We are shuttering the lending operations of this fintech for repeatedly lying and illegally cheating its customers," said CFPB Director Rohit Chopra. (

Reuters

)
Stablegains	$6-15M[2]	Fatal	Invested customer funds into a cryptocurrency which collapsed. Customers allege this was deceptive, th

...
How evolutionary lineages of LLMs can plan their own future and act on these plans
by Roman Leventov — 2022-12-25T18:11:18.754Z

TLDR

LLM lineages can plan their future and act on these plans, using the internet as the storage of event memory. “We” are not guaranteed to “out-OODA” them, even if their OODA loop will be six months or one year because the OODA loop duration of large collectives of humans (organisations, societies, research communities, and the whole of humanity) can be even slower. RLHF can lead to a dangerous “multiple personality disorder” type split of beliefs in LLMs, driving one set of beliefs into some unknown space of features where it won’t interfere with another set of features during general inferences.

Call for action: institute the norm for LLM developers to publish a detailed snapshot of the beliefs of these models about themselves prior to RLHF.

This post rests upon the previous one: Properties of current AIs and some predictions of the evolution of AI from the perspective of scale-free theories of agency and regulative development.

In particular, it is premised on the idea that DNNs and evolutionary lineages of DNNs are agents, in FEP/Active Inference formulation. The internal variables/states of evolutionary lineages of DNNs include the internal states of the agents that develop these DNNs (either individual, such as a solo developer, or collective agents, such as organisations or communities), i. e., the beliefs of these developers. Then, I concluded (see this section):

It’s just more productive to think of them together as a single agent: the development “team” and the evolutionary lineage of some technology being developed.

Now, however, I think it’s sometimes useful to distinguish between the developer agent and the evolutionary lineage of DNNs itself, at least in the case of evolutionary lineages of LLMs, because evolutionary lineages of LLMs can plan their own future and act upon these plans, independently from their developers and potentially even unbeknown to them if developers don’t use appropriate interpretability and monitoring techniques.

Below in the post, I explain how this can happen.

Thanks to Viktor Rehnberg for the conversation that has led me to write this post.

Experience of time is necessary for planning

First, for any sort of planning, it’s obligatory that an agent has a reference frame of linear time, which in turn requires storing memory about past events (Fields et al. 2021). Absent the reference frame for linear time, the agent cannot plan...
Accurate Models of AI Risk Are Hyperexistential Exfohazards
by Thane Ruthenis — 2022-12-25T16:50:24.817Z

(Where "an exfohazard" is information which leads to bad outcomes if known by a large fraction of society.)

Let us suppose that we've solved the technical problem of AI Alignment — i. e., the problem of AI control. We have some method of reliably pointing our AGIs towards the tasks or goals we want, such as the universal flourishing of all sapient life. As per the Orthogonality Thesis, no such method would allow us to only point it at universal flourishing — any such method would allow us to point the AGI at anything whatsoever.

Which means that, if we succeed at the technical problem, there'll be a moment at the very end of the world as we know it, where a person or a group of people will be making a decision regarding the future of the universe.

Above and beyond preventing an omnicide, we need to ensure that in the timelines where we do solve the technical problem, this decision is made right.

1. Establishing the Framework

The one good thing about the technical problem of alignment is that it makes hyperexistential risks — the risks of astronomical suffering — very unlikely.

The problem of AI Alignment can be viewed as the problem of encoding our preferences into an AGI, bit by bit. The strength of alignment tools, in turn, translates to how many bits we can encode. With the current methods of end-to-end training, we're essentially sampling preferences at random. Perfect interpretability and parameter-surgery tools would allow us to encode an arbitrary amount of bits. The tools we'll actually have will be somewhere between these two extremes.

"Build us our perfect world" is a very complicated ask, and it surely takes up many, many thousands of bits. That's why alignment is hard.

"Build us a hell" is its mirror. It's essentially the same ask, except for a flipped sign. As such, specifying it would require pretty much the same amount of bits.

Thus, in the timelines where we have alignment tools advanced enough to build a hell-making AGI, it's overwhelmingly likely that we have the technical capability to build an utopia-building AGI. On the flipside, conditioning on our inability to build an utopia-builder, our tools are probably so bad we can't come close to a hell-builder. In that case, we just sample some random preferences, and the AGI kills us quickly and painlessly.

Screwing up so badly we create a suffering-maximizer is vanishingly unlikely: it's only possible ...
ChatGPT is our Wright Brothers moment
by Ron J — 2022-12-25T16:26:21.890Z

The Wright Brother’s first flight was a joke by modern flying standards. Even though proving heavier than air flight was viable was monumental, many people at the time could not see how a machine that only can fly a short distance, a few feet off the ground, with the pilot in an uncomfortable position, and very questionable safety with no protection against crashing into the ground, could ever amount to anything.

Despite all these obvious flaws, entrepreneurs and engineers saw the potential. They rapidly iterated on the design, and within 2 decades, airplanes were a decisive advantage in war, they were changing the delivery of goods, they were creeping into commercial travel on the very luxury end of the spectrum. A hundred years later, we have modern marvels like the Airbus A380 and Boeing 747. We have international airports which are practically a world wonder in their operations and interoperability.

ChatGPT is analogous to the Wright Flyer. There were capable LLMs and tons of work in AI prior, but ChatGPT put this all together in a way for the general public to imagine how AI could be a part of their life. For others, they saw a clear villain to fear. But what is the analog in AI for the P51 Mustang? Or the B17? Or global air traffic control systems? Or composite materials and jet engines?

AI is going to have as big an impact or bigger as airplanes did, and a huge amount of infrastructure will need to be built in the process. What should humanity be building?
The Meditation on Winter
by Raemon — 2022-12-25T16:12:10.039Z

Preface

For the first several winter solstice ceremonies that I ran, I wrote a new, heartfelt speech for the moment of darkness – essentially the "keynote" speech of the event. The speeches were fueled by grief and difficult challenges that I was facing. They were good speeches. People were moved by them. They became a potent mainstay of Rationalist Solstice.

Only later did I realize... it was in fact pretty obnoxious for a winter holiday to rely on an organizer who a) had been going through hard shit that year, b) was eloquent enough to turn that grief into a speech, c) had insightful, philosophical things to say about how that grief related to Solstice, and rationality, and humanity writ large. It's a huge project. Most holidays consist of people straightforwardly executing on doing What Has Always Been Done. Carrying on a tradition – not embarking on a massive creative enterprise.

To be clear, heart wrenching creative enterprises are valuable and I'm glad people share their stories when they have stories worth telling. But I don't want people to feel like they have to, every year.

Many Moment of Darkness speeches are topical and related to things going on one particular year. We have a few timeless speeches that make sense to repeat (Beyond the Reach of God, and 500 Million and Not A Single One More). But none that quite fit to me as a default "canonical" Moment of Darkness speech. Or, perhaps alternately: there aren't quite enough "Moment of Darkness" speeches to hold up a rotation with enough variety.

So I wrote this speech one year, when planning for a small outdoor campfire solstice. I aimed for it to be something I felt good repeating. It works best when delivered outdoors. I typically ask people to remove their coat for the duration of it, to feel the cold winter air. But I think it holds up even as a part of large, indoor ceremony.

This version is somewhat personalized towards things that feel alive to me right now. I think the general structure is good, and you can tweak it to say the things that feel most right to you in the moment.

The Meditation

The winter solstice is the longest night of the year. It marks the beginning of the return of the light.

But it does not mark the end of the cold.

The Solstice is the moment when winter begins to slow its descent. But a slowly descending winter is still a deepening winter. The world grows colder for another...
I've updated towards AI boxing being surprisingly easy 
by Noosphere89 — 2022-12-25T15:40:48.104Z

Specifically because I think that sandboxes like an improved WASM could make it such that conditioned on careful data curation, that side channels have 0 probability to give the AI any information outside of what we give it.

I.e I would predict 0 probability of AI learning virology from a dataset, if the human didn't give it that data.

I also believe that this can be done with no performance penalty (unlike a lot of security measures.)

My argument follows in 2 steps:

First, I'll use daviddad's argument from the link below on how to create a sanbox (also known as a Cartesian Boundary) to prevent embedded computations from learning things from side channels.

Link here: https://www.lesswrong.com/posts/bqRD6MS3yCdfM9wRe/side-channels-input-versus-output

I will use a No Free Lunch argument to show why the bits that are left are insufficient to prevent the Cartesian Boundary/Sanbox from breaking.

Removing nondeterminism

The big piece of the argument is that we can prevent side channels from leaking information by removing all nondeterministic sources. This requires us to remove clock access, networking APIs, or randomness.

This is the second condition, but what about the first? Daviddad gives us the first condition:

We are careful to only provide the training process with inputs that would be just as likely in, say, an alternate universe where AI was built by octopus minds made of organosilicon where atoms obey the Bohr model.

From davidad again on an example:

Regarding part 1, self-play in Go is an example par excellence and this may be one reason why people tend to have a strong intuition that arbitrarily strong AlphaZero fails to kill you.

A No Free Lunch argument for why the remaining bits from random errors are incapable, even in theory, of breaking the sandbox/Cartesian Boundary

Basically, it comes from Steven Byrnes's explainer on the No Free Lunch theorem, and if you have equally likely data, that is the world is entirely random, you cannot learn anything. And once we remove the channels of nondeterminism given here, all that is left is random errors, like the chance of a 1 in 20 billion hardware errors, or cosmic rays, or simple random failure. But all of these bits are useless for breaking the sandbox, since again they're random.

Here's Steven Byrnes's explainer on the topic:

https://www.lesswrong.com/posts/Dyt2TDdMGHDkXPcpp/the-no-free-lunch-theorem...