trentmkelly/LessWrong-43k huggingface dataset:
The Leeroy Jenkins principle: How faulty AI could guarantee "warning shots"
by titotal — 2024-01-14T15:03:21.087Z
Link: https://titotal.substack.com/p/the-leeroy-jenkins-principle-how

Confidence level: This is all speculation about tech that doesn’t exist yet. Everything is highly uncertain. All “simulations” within are illustrative models and should not be taken as serious predictions.

When talking about AI, existential risk theorists almost always focus their attention on the best of the best AGI. Even if some AI along the way is mediocre, or a buggy mess, or mostly correct but wildly off in some areas, it’s not ultimately important, compared to the one critical try, when the best, near flawless superintelligence hits the point where they can reliably defeat humanity in one go.

But there is one area where every level matters, especially AI that is flawed, badly trained, or idiotic. And that is the area of attacks that fail, otherwise known as “warning shots”.

It is far easier to conceive of a plan to destroy humanity, than it is to actually pull such a plan off. These flawed or unintelligent AI are not guaranteed to defeat humanity. But they can still scare the hell out of humanity, prompting international cooperation against a shared threat. Like the titular Leeroy Jenkins, a premature attack by an an idiot can foil the plans of more intelligent schemers.

In the first part of this article, I will lay out several reasons why I expect faulty behavior in at least some AGI , by giving a few examples of highly intelligent, foolish humans, of foolish behavior in existing AI systems, and conceptual reasons why I don’t expect this trend to suddenly reverse as AGI becomes more intelligent.

In the second part, I will explain why I think it is highly likely that warning shots will exist if AI is malevolent in nature, and even if it isn’t. First I will discuss the “Leeroy Jenkins” theory of idiots rushing in first when it comes to malevolent rogue AGI. I will discuss how this also applies to humans misusing AI. Lastly I will discuss how even friendly AI could give a warning shot, if it is incompetently utilized.

Part 1: why some AGI will be faulty

This section is mostly a rehash of my earlier post “why the first AGI will be a buggy mess”, with some fresh evidence. The focus here is to prove something simple: at every level of intelligence thus known, at least some entities with that intelligence act like fools at least some of the time. If this is obvious for you, feel free to skip to the next section.

Existing AI can be faulty and overconfident.

The follo...
Notice When People Are Directionally Correct
by Chris_Leong — 2024-01-14T14:12:37.090Z

I started watching Peter Zeihan videos last year.

He shares a lot of interesting information, although he seems to have a very strong bias towards doom and gloom.

One thing in particular stood out to me as completely absurd: his claim that global trade is going to collapse due to piracy as the US pulls back from ensuring freedom of the waters.

My immediate thought: "Come on, mate, this isn't the 17th century! Pirates aren't a real issue these days. Technology has rendered them obsolete".

Given this, I was absolutely shocked when I heard that missile attacks by Houthi rebels had caused most of the largest shipping companies to decide to avoid the Bab el-Mandeb Strait and to sail around Africa instead.

This has recently triggered the US to form an alliance to maintain freedom of shipping there and the US recently performed airstrikes in retaliation. It won't surprise me if this whole issue is resolved relatively soon and if that happens, then the easy thing to do would be to go back to my original beliefs: "Silly me, I was worried for a second that Peter Zeihan might be correct, but that was just me falling for sensationalism. The whole incident was obviously never going to be anything. I should forget all about it".

I believe that this would be a mistake. It would be very easy to forget it, but something like the Houthi's being able to cause as much disruption as they have been able to was outside of my model. One option would be to label it as a freak incident; another would be to say that shooting missiles isn't exactly piracy so this doesn't count. However, the problem with this is that it's too easy. When there's ambiguity, humans have a strong tendency to make up excuses as to why they were right all along, so I think it's worthwhile pushing toward the other end of the spectrum and assuming that there probably were ways in which you could have done better.

I tried searching for possible ways that I should update and the following thoughts came to mind, which I'll convey because they are illustrative:

• I have heard a few people suggest in various contexts that many countries have been coasting and relying on the US for defense, but it was just floating around in my head as something that people say that might or might not be true. I haven't really delved into this, but I'm starting to suspect I should put more weight on this belief.
• I hadn't considered the poss...
Corrosive Mnemonics
by Epirito — 2024-01-14T12:44:55.877Z

When I hear statistics about different countries, I sometimes think “yeah, that makes sense,” though it’s hard to know to what extent I already knew that or believed in a high probability of that and to what extent I merely convinced myself I already knew that just so I wouldn’t have a new thing to remember.

Consider false etymologies: you are learning a foreign language and you encounter a word that sounds like some other word you already know. So you think, "that makes sense. Arigato must have come from obrigado, after all Portuguese missionaries had been to Japan etc." Or rather, you realize that just because they sound similar it doesn't mean they are related, so instead you think, "I wonder if it is so...let's pretend this is correct so it becomes easier to remember." So whenever you hear obrigado you think, "oh, yeah obrigado, I remember that. It's like arigato." But as long as you keep in mind that this is just a mnemonic, you still have to remember that you made it up. It's arbitrary. What you might start to do instead as a shortcut is to pretend that there is nothing to remember and the connection is natural and obvious. You end up sacrificing a bit of the sanity of your world model and using it to encode the meaning of the word. You warp your rationality so that a new contingent fact appears to follow from what you already knew. "Of course! Obrigado means arigato. I should have known that," as if a reasonable person should be able to infer the meaning of the word even when hearing it from the first time just from the fact that it sounds like arigato, though of course it sounds dumb when you come out and say it like that.

I guess I can’t prove that it’s never worth it. Maybe sometimes it’s better to cultivate this bias towards a true fact and then to try to duct tape around it afterwards whenever it gives you trouble. I mentioned it “sounds dumb when you come out and say it like that.” So maybe just realize this and snap out of it somehow. Though this seems like trying to have your cake and eat it too. One might wonder whether this would corrupt your sanity in less obvious ways. And besides I have no idea how you’d go about figuring out if this is worth it, so I guess I’d rather try to keep in mind that the mnemonic is just that even if in theory it’s not always optimal. So I feel inclined to say that these things that I call corrosive mnemonics are g...
Against most, but not all, AI risk analogies
by Matthew Barnett — 2024-01-14T03:36:16.267Z

I personally dislike most AI risk analogies that I've seen people use. While I think analogies can be helpful for explaining concepts to people and illustrating mental pictures, I think they are frequently misused, and often harmful. At the root of the problem is that analogies are consistently mistaken for, and often deliberately intended as arguments for particular AI risk positions. And a large fraction of the time[1] when analogies are used this way, I think they are misleading and imprecise, routinely conveying the false impression of a specific, credible model of AI, even when no such credible model exists.

Here is a random list of examples of analogies that I found in the context of AI risk (note that I'm not saying these are bad in every context):

Stuart Russell: "It’s not exactly like inviting a superior alien species to come and be our slaves forever, but it’s sort of like that."
Rob Wiblin: "It’s a little bit like trying to understand how octopuses are going to think or how they’ll behave — except that octopuses don’t exist yet, and all we get to do is study their ancestors, the sea snail, and then we have to figure out from that what’s it like to be an octopus."
Eliezer Yudkowsky: "The character this AI plays is not the AI. The AI is an unseen actress who, for now, is playing this character. This potentially backfires if the AI gets smarter."
Nate Soares: "My guess for how AI progress goes is that at some point, some team gets an AI that starts generalizing sufficiently well, sufficiently far outside of its training distribution, that it can gain mastery of fields like physics, bioengineering, and psychology [...] And in the same stroke that its capabilities leap forward, its alignment properties are revealed to be shallow, and to fail to generalize. The central analogy here is that optimizing apes for inclusive genetic fitness (IGF) doesn't make the resulting humans optimize mentally for IGF."
Norbert Wiener: "when a machine constructed by us is capable of operating on its incoming data at a pace which we cannot keep, we may not know, until too late, when to turn it off. We all know the fable of the sorcerer’s apprentice..."
Geoffry Hinton: "It’s like nuclear weapons. If there’s a nuclear war, we all lose. And it’s the same with these things taking over."
Joe Carlsmith: "I think a better analogy for AI is something like an engineered virus, where,

...
Vote With Your Face
by jefftk — 2024-01-14T03:30:06.009Z

Starting with our July 2nd dance

BIDA has been running half our
dances as

mask-optional.
Some people only want to attend if they know everyone on the floor
will be wearing a

high-filtration
mask, others only want to attend if they don't have to mask, and
of course others will or won't attend regardless. Offering some of
each lets people choose the dances that fit their preferences.

Plus, as someone who likes experiments, it's almost like a randomized
controlled trial of mask requirements on attendance!

Before we switched to alternating masks required and optional we
didn't know what effect this would have on attendance: would there be
more people on mask-optional nights (more fun!) or on mask-required
nights (more safe!) Here's what I see looking back over attendance:

Note that aside from there only being eleven dances, there are some
unique aspects of some of the dances that you might want to adjust
for:

The 7/2 dance was the first mask-optional dance, and so might have
gotten a boost from people who were excited to dance without masks but
wouldn't come every time.

The 7/2 and 9/3 dances were our last dances before the summer
and first dance back, which could have increased attendance. But they
were also both holiday weekends, which could have decreased
attendance.

The 11/5 and 11/19 dances were both double dances, which
generally have somewhat higher attendance. But we (mostly by
accident) did two back to back, one of each.

The 12/3 dance was a family dance, which also usually means
higher attendance.

The 12/17 dance was at a different venue, the Somerville
Armory, which usually means lower attendance.

The 1/7 dance was during a major snowstorm, which usually means
much lower attendance.

Overall, it looks like mask-optional nights are a bit more popular,
but with all the caveats it's hard to say. Regardless, the difference
clearly not so large that it's a reason to stop one or the other. I
expect we'll continue alternating, since we still hear from many
people that they only want to attend one or the other.

(All this is speaking for myself, not the BIDA Board.)
Case Studies in Reverse-Engineering Sparse Autoencoder Features by Using MLP Linearization
by Jacob Dunefsky — 2024-01-14T02:06:00.290Z

Epistemic status: preliminary/exploratory.

Work performed as a part of Neel Nanda's MATS 5.0 (Winter 2023-2024) Research Sprint.

TL;DR: We develop a method for understanding how sparse autoencoder features in transformer models are computed from earlier components, by taking a local linear approximation to MLP sublayers. We study both how the feature is activated on specific inputs, and take steps towards finding input-independent explanations via examining model weights. We demonstrate this method with several deep-dive case studies to interpret the mechanisms used by simple transformers (GELU-1L and GELU-2L) to compute some specific features, and validate that it agrees with the results of causal methods.

Introduction

A core aim of mechanistic interpretability is tackling the curse of dimensionality, decomposing the high-dimensional activations and parameters of a neural network into individually understandable pieces. Sparse Autoencoders (SAEs) are a recent and exciting development that allows us to take high-dimensional activations (likely in superposition) and decompose them into meaningful directions in activation space that represent (mostly) independent concepts.

A major limitation of SAEs, when applied to MLP activations/outputs, is that it's difficult to study how a feature is computed from the output of earlier model components. With a meaningful neuron, we can look directly at its connections/virtual weights to earlier components -- e.g. a car neuron in a vision model being built from car window, car body, and car wheel neurons -- but SAE features are often dense in the neuron basis. This means that, naively, to understand how the feature is computed, we need to understand a complex non-linear function of the thousands of neuron activations. The curse of dimensionality remains!

In this post, we present a technique to explore how neuron-dense SAE features are computed from preceding model components. This approach involves taking the derivative of MLP sublayers in order to obtain a local linear approximation to the SAE feature -- a technique that has demonstrated some success in prior work such as attribution patching. Importantly, our method goes further and uses this linear approximation in conjunction with the model weights themselves to obtain a global picture of model behavior rather than one confined to a specific input example[1]. The end result ...
D&D.Sci Hypersphere Analysis Part 1: Datafields & Preliminary Analysis
by aphyer — 2024-01-13T20:16:39.480Z

This is a post (hopefully eventually a short series of posts) detailing my analysis of the recently released D&D.Sci scenario by abstractapplic. I've decided to go through documenting what I do - if you intend to play the scenario yourself without help, you should do that before reading this. If you want to use this information in your solutions, go ahead.

Raw Data and Columns

I start by grabbing the raw data and saving it as a csv, then import it into Python and Excel to play with it. I'm not trying to do any particular analysis at this point, just trying to get familiar with the data and see if anything jumps out.

We'll go through the columns one by one, and make some charts in Excel for visualization[1].

Longitude is relatively evenly-distributed from -180 to +180.

Latitude can vary from -90 to +90 but is bimodal, usually taking on values around 45 - we rarely land near the equator or near the poles.

And Shortitude and Deltitude look the same as Latitude:

This might be what you naturally expect from landing at a random point on a 4D sphere - certainly values near +-90 being less likely happens naturally, since those are 'the poles' and take up less area. I don't have a good enough intuition for this to know if values near 0 are also naturally rare once you add more dimensions, or if we're avoiding the 'equator' for some reason.

Strange Smell is often Somewhat present, but only rarely EXTREMELY present:

Air Tastes Like has a few different values, some common and some rare:

Feng Shui is generally Adequate, sometime bad, and very rarely good:

Weird Sounds has five possible values, of which we see up to three at a time:

Eerie Silence
Impossible Humming
Otherworldly Skittering
Unearthly Squelching
Unnatural Buzzing

Eerie Silence appears exactly when no other sounds appear.

Skittering and Buzzing never happen together (though they are the most common sounds separately).

Local Value of Pi is distributed in what looks like a very neat distribution:

3.141 + (1d41-1d41)/1000 isn't literally accurate, since we see more decimal figures than that, but it's probably a reasonable way of thinking about it?

Murphy's Constant has a very weird distribution:

I'm going to guess something like this:

MC is distributed with the same type of distribution we saw Pi have, with probability increasing linearly from 0 to ~5.5 and then decreasing from ~5.5 to ~11.
However, ou

...
Some additional SAE thoughts
by Hoagy — 2024-01-13T19:31:40.089Z

Thanks to Lee Sharkey for feedback on the first section and Lee Sharkey, Jake Mendel, Kaarel Hänni and others at the LISA office for conversations around this post.

This post is a collection of a few small experiments and thoughts from the last couple of months that never really turned into anything larger but have helped clarify my thinking on some SAE-related things. I've now joined the interpretability team at Anthropic but everything written here is from before that date.

How can distributed features in an MLP work?

Summary

While working on the original SAE paper, I was troubled by the following argument:

To understand what an MLP layer is doing we need to understand how the non-linearity acts on the data, because this is where the novel computation is.
Features as found by sparse autoencoders in the post-non-linearity of a transformer MLP seem to be very strongly distributed across neurons.
A feature that's spread over a large number of neurons will have only a tiny effect on each individual neuron, and the neuron will be approximately linear across such a small change.
Therefore these distributed features can't be where the real action is, and we need to somehow incorporate the neuron basis into the way we understand features in MLPs.

I no longer believe this, most importantly because I realised that when a feature is spread across n neurons, the original non-linearity can be recovered by scaling the magnitude of the input feature up by a factor of √n. The fact that it only requires scaling by √n , rather than by n which was implicit in my informal argument (when I considered the possibility of scaling up at all..) makes it feasible to use multiple highly distributed features, as long as only a small number of features are active.

Code available on GitHub.

The need for non-linearity

A multilayer model acts on input data, manipulating it into new structures at each layer. If we look at the total amount of information that is present at each successive layer, we know that the amount of information can only decline, since any information present at a higher layer is directly calculated from the lower layers, so the information must also be present in those lower layers.

However, what the model can do is to make particular functions of the data more accessible through the operation of multiple layers. In particular, th...
(4 min read) An intuitive explanation of the AI influence situation
by trevor — 2024-01-13T17:34:36.739Z

This is a 4-minute read, inspired by the optimized writing in Eukaryote's Spaghetti Towers post.

My thinking is that generative AI has potential for severe manipulation, but that the 2010s AI used in social media news feeds and other automated systems are a bigger threat, and this tech tells us much more about the future of international affairs and incentives for governments to race to accelerate AI, has fewer people aware of it, has a substantial risk of being used to attack the AI safety community, and the defenses are easy and mandatory to deploy. This post explains why this tech is powerful enough to be a centerpiece of people's world models.

The people in Tristan Harris's The Social Dilemma (2020) did a fantastic job describing the automated optimization mechanism in a quick and fun way (transcript).

[Tristan] A [stage] magician understands something, some part of your mind that we’re not aware of. That’s what makes the [magic trick] illusion work. Doctors, lawyers, people who know how to build 747s or nuclear missiles, they don’t know more about how their own mind is vulnerable. That’s a separate discipline. And it’s a discipline that applies to all human beings...

[Shoshana] How do we use subliminal cues on the Facebook pages to get more people to go vote in the midterm elections? And they discovered that they were able to do that.

One thing they concluded is that we now know we can affect real-world behavior and emotions without ever triggering the user’s awareness. They are completely clueless.

Important note: all optimization here is highly dependent on measurability, and triggering the user's awareness is a highly measureable thing.

If anything creeps someone out, they use the platform less; such a thing is incredibly easy to measure and isolate causation. To get enough data, these platforms must automatically reshape themselves to feel safe to use.

This obviously includes ads that make you feel manipulated; it is not surprising that we ended up in a system where >95% of ad encounters are not well-matched.

This gives researchers plenty of degrees of freedom to try different kinds of angles and see what works, and even automate that process.

[Tristan] We’re pointing these engines of AI back at ourselves to reverse-engineer what elicits responses from us. Almost like you’re stimulating nerve cells on a spider to see what causes its legs to res

...