trentmkelly/LessWrong-43k huggingface dataset:
Differential knowledge interconnection
by Roman Leventov — 2024-10-12T12:52:36.267Z

Cross-posted from my blog.

This post is a reply to Eugene Kirpichov's post

on Linkedin

. Eugene writes that contributing to general AI and information processing capabilities (including GPUs, general LLM technology, general data processing, etc.) is probably harmful overall because these capabilities effectively increase the speed at which the civilisation is moving but doesn't affect the trends, and the trends are negative right now because the civilisation is not on a sustainable trajectory, that is, the civilisation doens't move towards increasing flourishing of all moral patients, human and non-human.

I disagree with Kirpichov's proposition as is because I think it lacks important nuance in definition of general AI and knowledge production capabilities. I think there are different kinds of general AI capabilities, some, I agree, are probably on net harmful, but others I think are not. I will explain the difference below.

Note that this post is not about general vs. specialised AI, i.e., narrow AI applications. Kirpichov and I agree that specialised AI applications should be judged on the case-by-case basis. Different applications may lie on the range from clearly beneficial, such as AI for community deliberation (see the

Collective Intelligence Project

), AI for human health (e.g.,

Slingshot AI

,

HealthcareAgents

), AI for ocean ecosystems modelling and monitoring (e.g.,

Wildflow AI

), or AI for decoding non-human communication (e.g.,

Earth Species Project

), to clearly harmful applications, such as AI for spam, phishing, etc., with a thousand shades of benefit and harm in between.

Yet another dimension is the degree of generality of AI capabilities, from very broadly general, such as GPGPU computing capabilities, to very specialised AI capabilities, applicable only in a single narrow domain. The chances of spilling over some capabilities developed for beneficial applications to other, potentially harmful applications should be estimated. The standard example here is AI for drone navigation and autonomy, if developed for low-footprint delivery of goods, may proliferate into harmful applications like drone warfare and killer drones.

Now, to the question of distinguishing between better or worse general AI capabilities.

AI technologies help to create and test more models faster. I use the word "models" in the broadest sense here, including engineering designs, methods...
Most arguments for AI Doom are either bad or weak
by Logan Zoellner — 2024-10-12T11:57:50.840Z

This is not going to be a popular post here, but I wanted to articulate precisely why I have a very low pDoom (2-20%) compared to most people on LessWrong.

Every argument I am aware of for pDoom fits into one of two categories: bad or weak.

Bad arguments make a long list of claims, most of which have no evidence and some of which are obviously wrong. Examples include A List of Lethalities, which is almost the canonical example. There is no attempt to organize the list into a single logical argument, and it is built on many assumptions (analogies to human evolution, assumption of fast takeoff, ai opaqueness) which are in conflict with reality.

Weak arguments go like this: "AGI will be powerful. Powerful systems can do unpredictable things. Therefore AGI could doom us all." Examples of these arguments include each of the arguments on this list.

So the line of reasoning I follow is something like this;

I start with a very low prior of AGI doom (for the purpose of this discussion, assume I defer to consensus).
I then completely ignore the bad arguments,
finally, I give 1 bit of evidence collectively for the weak arguments (I don't consider them independent, most are just rephrasing the example argument)

So even if I assume no one betting on Manifold has ever heard of the argument "AGI might be bad actually", I only get from 13% -> 30% with that additional bit of evidence.

In the comments: if you wish to convince me, please propose arguments that are neither bad nor weak. Please do not argue that I am using the wrong base-rate or that the examples that I have already given are neither bad nor weak.

EDIT:

There seems to be a lot of confusion about this, so I thought I should clarify what I mean by a "strong good argument"

Suppose you have a strongly-held opinion, and that opinion disagrees from the expert-consensus (in this case, the Manifold market or expert surveys showing that most AI experts predict a low probability of AGI killing us all). If you want to convince me to share your beliefs, you should have a strong good argument for why I should change my beliefs.

A strong good argument has the following properties:

it is logically simple (can be stated in a sentence or two)

This is important, because the longer your argument, the more details that have to be true, and the more likely that you have made a mistake.

...
Neural Network And Newton's Second Law
by Max Ma — 2024-10-12T06:25:59.072Z

When Isaac Newton formulated his Second Law (𝐹 = 𝑚𝑎), observation and experimentation were crucial. Drawing on experiments, many influenced by Galileo and others, he concluded that force causes changes in motion, or acceleration. This process involved solving both forward and inverse problems through iterative refinement (experimentation and observation), much like the training of neural networks. In neural networks, forward and inverse iterations are combined in a similar way, and this is the primary reason behind their immense power. It’s even possible that a neural network could independently derive Newton's Second Law. That’s the true potential of this technology. This is what we’ve discovered and explored in greater detail in our recent publication: Deep Manifold Part 1: Anatomy of Neural Network Manifold.

Two key factors enable the forward and inverse iteration process in neural networks: infinite degrees of freedom and self-progressing boundary conditions. The infinite degrees of freedom explain how just a few gigabytes of neural network weights can store vast amounts of knowledge and information. The self-progressing boundary conditions are what allow neural networks to be trained on any data type, including mixed types like language and images. This boundary condition enables the network to efficiently intake large and complex datasets, while its infinite degrees of freedom process them with unparalleled flexibility and capacity.

If a neural network could independently derive Newton's Second Law, would we still need Newton's Second Law? The answer is yes.

Newton's Second Law can be integrated into a neural network as a neural operator to tackle much more complex problems, such as the analysis of the Vajont Dam failure
Newton's Second Law can be to verify neural network model results.

If you believe that 'math is a branch of physics,' we are now seeing the true power source behind neural networks—advancing to a point where AI is actually pushing the boundaries of mathematics itself.

Welcome to the AI driven world.
If the DoJ goes through with the Google breakup,where does Deepmind end up?
by O O — 2024-10-12T05:06:50.996Z

A few predictions: A google subsidiary starts selling TPUs. Deepmind seeks outside funding and moves to productize their suite. GCP becomes more viable. Google search loses market share and is forced to compete with Gemini’s offerings. OAI and Anthropic have enormous revenue streams open up.

Timeline effects:

Acceleration due to increased competition for AGI and finally a serious contender to Nvidia. More funding unlocked by OAI and Anthropic.

Deceleration from deepmind unable to burn unlimited cash on GPUs. They have far more compute than anyone else.

Verdict: Slow down if scaling transformers is all you need. Speed up if we need a breakthrough or if a breakthrough is sufficient.
My motivation and theory of change for working in AI healthtech
by Andrew_Critch — 2024-10-12T00:36:30.925Z

This post starts out pretty gloomy but ends up with some points that I feel pretty positive about. Day to day, I'm more focussed on the positive points, but awareness of the negative has been crucial to forming my priorities, so I'm going to start with those. I'm mostly addressing the EA community here, but hopefully this post will be of some interest to LessWrong and the Alignment Forum as well.

Part one — My main concerns

I think AGI is going to be developed soon, and quickly. Possibly (20%) that's next year, and most likely (80%) before the end of 2029. These are not things you need to believe for yourself in order to understand my view, so no worries if you're not personally convinced of this.

(For what it's worth, I don't expect to change my mind about the above AGI forecast in response to debate. That's because I feel sufficiently clear in my understanding of the various ways AGI could be developed from here, such that the disjunction of those possibilities adds up to a pretty high level of confidence in AGI coming soon, which is not much affected by who agrees with me about it. Also, I'm not really deferring to others about it, so I'm pretty confident the above forecast is not the result of any "echo chamber" or "pure hype" effects. My views here came through years of study and research in AI, combined with over a decade of private forecasting practice starting in 2010 — including a lot of hype-detection and bullshit detection practice — which I don't think can be succinctly conveyed in a blog post.)

I also currently think there's around a 15% chance that humanity will survive through the development of artificial intelligence. In other words, I think there's around an 85% chance that we will not survive the transition. Many factors affect this probability, so please take this as a conditional forecast that I'd like you to change if you can, rather than taking it as some unavoidable fate that humanity has no power to decide upon. With that said, I do have reasons for the number 85% being so high.

First, I think there's around a 35% chance that humanity will lose control of one of the first few AGI systems we develop, in a manner that leads to our extinction. Most (80%) of this probability (i.e., 28%) lies between now and 2030. In other words, I think there's around a 28% chance that between n...
HDBSCAN is Surprisingly Effective at Finding Interpretable Clusters of the SAE Decoder Matrix
by Jaehyuk Lim — 2024-10-11T23:06:14.340Z

*All authors have equal contribution.

This is a short informal post about recent discoveries our team made when we:

Clustered feature vectors in the decoder matrices of Sparse Autoencoders (SAEs) trained on the residual stream of each layer of GPT-2 Small and Gemma-2B.
Visualized the clusters in 2D after reducing the feature vector space via UMAP

We want to share our thought process, our choices when clustering, and some challenges we faced along the way.

Note: While currently a work in progress, we hope to open-source the clustered features and projections soon, in case others will find it useful.

We would like to acknowledge many people who helped with this endeavor! Professors Shibu Yooseph and Ran Libeskind-Hadas, Alice Woog, Daniel Tan, and many others who offered help and guidance.

TL;DR

SAE decoder weight matrix alone can be used to find (some) feature families.
While HDBSCAN classifies most features as “noise”, the remaining clusters encapsulate features that activate on similar concepts and tokens in high-dimensions

HDBSCAN is not able to find all clusters of features
We propose using the Cech Complex to identify persistent sets of features via filtration.

Our primary finding that the decoder weight matrix alone can be used find feature families may help with:

Auto-interpretability: It should be easier, cheaper, and more powerful to classify groups of features than find explanations for a single feature.
Safety Guardrailing: Interpretable clustering techniques can be used to elucidate the geometry of the SAE feature space. We propose a guardrail framework and discuss the utilization of appropriate abstract spaces, such as Čech complexes, to ensure safety and reliability.
Reverse Feature Splitting: We discuss how clustering can be viewed as “reverse splitting” features, finding natural relationships between existing features and grouping them together to find stronger activations or as a sign of feature absorption in SAEs.

Section 1: Motivations and Relevant Prior Work

A major challenge to interpreting large language models (LLMs) lies in the model seemingly representing more information than it has dimensions. For example, polysemantic neurons tend to be the rule rather than the exception (Bricken et al. 2023; Cunningham et al. 2023). Leading theories propose that perhaps all the encoded features making up the residual stream (as w...
Changing the Mind of an LLM
by testingthewaters — 2024-10-11T22:25:37.464Z

Introduction

tl;dr: I believe that making LLMs more polite and making LLMs repeat facts are problems within the same domain of research. I try several experiments related to this hypothesis and discuss outcomes of those experiments. It's possible to get the LLM to incorporate a memory vector when generating new tokens, but changing or shaping the memory vector is not trivial.

Steering LLMs: Two Cultures

Since the development of LLMs much work has gone into directing/steering their output. This can either be in terms of theme/tone ("mention or don't mention X topic", "avoid discussing politics"), content ("the sale price for product Y is 10 dollars", "the founder of our company is XXX"), or both ("you are a research assistant who is extremely skilled in American History"). In general, research into directing the tone of an LLM leans towards using SAEs, steering vectors, or similar technologies. The aim is to decompose or manipulate the internal state of the LLM and modify it to produce desirable behaviour. Research in terms of directing an LLM's content output leans towards RAG (retrieval-augmented generation) and tool use. The aim is to supply LLMs with information or functions relevant to the task with the hopes that these results will be integrated into the final output. In general, work manipulating LLM internals is referred to as "network architecture", while work around supplying LLMs with tools and references is known as "agent architecture".

I believe that these two fields are in fact the same field, and have tried several experiments in this joint area. If we accept that LLMs work by simulation of "personalities" found within an LLM's training corpus, then there is a strong tie between the sort of person the LLM is simulating (tone or theme) and the things that person ought to know (content). In the same way, changing the factual knowledge of an LLM changes the "setting" of the simulation. While LLMs are indeed skilled at style-transfer problems like "explain like I'm five", it is also true that when facts are transferred across different styles of communication information content is added or lost. Consider the counterfactual problem "explain like I'm a PHD holder in a relevant field" - to accurately transfer style, you must also transfer content!

A stronger version of this hypothesis relates to training. It seems probable that LLMs start by emulat...
EIS XIV: Is mechanistic interpretability about to be practically useful?
by scasper — 2024-10-11T22:13:51.033Z

Part 14 of 12 in the

Engineer’s Interpretability Sequence

.

Is this market really only at 63%? I think you should take the over.

Only 63%? I think you should take the over.

Five tiers of rigor for safety-oriented interpretability work

Lately, I have been

thinking

of interpretability research as falling into five different tiers of rigor.

1. Pontification

This is when researchers claim they have succeeded in interpreting a model by definition or based on analyzing results and asserting hypotheses about them. This is a key part of the scientific method. But by itself, it is not good science. Previously in this sequence,

I have argued

that this standard is fairly pervasive.

2. Basic Science

This is when researchers develop an interpretation, use it to make some (usually simple) prediction, and then show that this prediction validates. This is at least doing science, but it doesn't necessarily demonstrate any usefulness or value.

3. Streetlight/Toy Demos

This is when researchers accomplish a useful type of task with an interpretability technique but do so in a way that is toy, cherry-picked, or under a

streetlight

.

4. Useful Engineering

This is when researchers show that an interpretability tool can be used uniquely or competitively to accomplish a useful task. For this level of rigor, it needs to be convincingly demonstrated that doing the task with interpretability is better than other ML techniques. I think that there is currently at least one example of work low in this tier.

5. Net Safety Benefit

This would be when researchers convincingly demonstrate that an interpretability tool isn't just practically and competitively useful, but is in a way that is differentially beneficial for reducing risks instead of undergoing a capability capture. By my understanding, this tier has not yet been reached. And unless it is, then the field of AI (mechanistic) interp will have been, at best, a big waste from a safety standpoint.

What’s been happening lately?

Recently, some solid work has been done in tier 3.

A few months ago, I remember hearing about some new work on SAE interpretability –

Marks et al. (2024)

. The person I heard about it from was pretty excited, and when I pulled up the paper, I thought to myself “here we go, let’s see” and mentally prepared to look for holes in it. But when I read the paper,

I t

...
Dario Amodei — Machines of Loving Grace
by Matrice Jacobine — 2024-10-11T21:43:31.448Z
Link: https://darioamodei.com/machines-of-loving-grace#basic-assumptions-and-framework

I think and talk a lot about the risks of powerful AI. The company I’m the CEO of, Anthropic, does a lot of research on how to reduce these risks. Because of this, people sometimes draw the conclusion that I’m a pessimist or “doomer” who thinks AI will be mostly bad or dangerous. I don’t think that at all. In fact, one of my main reasons for focusing on risks is that they’re the only thing standing between us and what I see as a fundamentally positive future. I think that most people are underestimating just how radical the upside of AI could be, just as I think most people are underestimating how bad the risks could be.

In this essay I try to sketch out what that upside might look like—what a world with powerful AI might look like if everything goes right. Of course no one can know the future with any certainty or precision, and the effects of powerful AI are likely to be even more unpredictable than past technological changes, so all of this is unavoidably going to consist of guesses. But I am aiming for at least educated and useful guesses, which capture the flavor of what will happen even if most details end up being wrong. I’m including lots of details mainly because I think a concrete vision does more to advance discussion than a highly hedged and abstract one.
Open letter to young EAs
by Leif Wenar — 2024-10-11T19:49:10.818Z

In March I published a critique of EA in Wired. The responses I got, especially from young EAs, prompted me to write an open letter to my young friends in EA.

https://docs.google.com/document/d/1ytfQOfmjWTDiGdjuyNBfWJ_g-644y_Pd/edit?usp=sharing&rtpof=true&sd=true

Thanks for reading.
The Great Bootstrap
by KristianRonn — 2024-10-11T19:46:51.752Z

This part 2 in a 3-part sequence summarizes my book, The Darwinian Trap, (see part 1 and part 2 here), The Darwinian Trap. The book aims to popularize the concept of multipolar traps and establish them as a broader cause area. If you find this series intriguing contact me at kristian@kristianronn.com if you have any input or ideas.

As we reach the final part of this blog series, it's understandable if you're feeling a bit down. The previous sections painted a rather bleak picture, touching on issues like misaligned AI, global warming, nuclear threats, and engineered pandemics. These societal-scale risks make the future seem uncertain.

Yet, despite these daunting challenges, I believe there is still room for optimism. In this segment, I'll share why I think humanity can overcome these existential threats, reset the harmful selection pressures that drive destructive arms races, and pave the way for a brighter future. But first, let’s review what we've covered.

Evolution selects for fitness, not for what we intrinsically value. In this context, fitness refers to survival in an environment, be it a natural environment or one we have created for ourselves, such as the world of business.
Pursuing narrow targets often produces broadly negative outcomes. When we optimize for short-term survival while remaining indifferent to other goals— e.g. health and well-being—then, by default, we optimize against those values. A related phenomenon by which narrow optimization tends to produce bad outcomes is Goodhart’s Law.
We can’t easily choose not to play the game and choose to act according to our intrinsic values rather than Darwinian survival imperatives. If we do, then we will be outcompeted by those who do play the game, thus creating negative outcomes for us, the conscientious objectors.
Playing the game eventually leads to extinction arms races, because strategies and mutations that favor increased power and more efficient resource exploitation tend to win, which sits at the core of the Fragility of Life Hypothesis, which argues that life created through natural selection might contain the seed of its own destruction.

To break this cycle, we must develop a formula for stable, long-lasting cooperation and implement it so that cooperation becomes adaptive in any environment. This challenge is as daunting as lifting ourselves by our bootstraps, defy...