trentmkelly/LessWrong-43k huggingface dataset:
Rational Animations' intro to mechanistic interpretability
by Writer — 2024-06-14T16:10:57.015Z
Link: https://youtu.be/jGCvY4gNnA8

In our new video, we talk about research on interpreting InceptionV1, a convolutional neural network. Researchers have been able to understand the function of neurons and channels inside the network and uncover visual processing algorithms by looking at the weights. The work on InceptionV1 is early but landmark mechanistic interpretability research, and it functions well as an introduction to the field. We also go into the rationale and goals of the field and mention some more recent research near the end. Our main source material is the circuits thread in the Distill journal and this article on feature visualization. The author of the script is Arthur Frost. I have included the script below, although I recommend watching the video since the script has been written with accompanying moving visuals in mind.

Intro

In 2018, researchers trained an AI to find out if people were at risk of heart conditions based on pictures of their eyes, and somehow the AI also learned to tell people’s biological sex with incredibly high accuracy. How? We’re not entirely sure.

The crazy thing about Deep Learning is that you can give an AI a set of inputs and outputs, and it will slowly work out for itself what the relationship between them is. We didn’t teach AIs how to play chess, go, and atari games by showing them human experts - we taught them how to work it out for themselves. And the issue is, now they have worked it out for themselves, and we don’t know what it is they worked out.

Current state-of-the-art AIs are huge. Meta’s largest LLaMA2 model uses 70 billion parameters spread across 80 layers, all doing different things. It’s deep learning models like these which are being used for everything from hiring decisions to healthcare and criminal justice to what youtube videos get recommended. Many experts believe that these models might even one day pose existential risks. So as these automated processes become more widespread and significant, it will really matter that we understand how these models make choices.

The good news is, we’ve got a bit of experience uncovering the mysteries of the universe. We know that humans are made up of trillions of cells, and by investigating those individual cells we’ve made huge advances in medicine and genetics. And learning the properties of the atoms which make up objects has allowed us to develop modern material science and high-precision...
Why keep a diary, and why wish for large language models
by DanielFilan — 2024-06-14T16:10:07.658Z

Inspired by a dream I just woke up from, where I did not keep a diary

One of the people with whom I have the most intimate of connections is my past self - in particular, my child self. We share a large number of commonalities: much of our basic outlook, our personality, many of our drives. But, of course, my child self is different from me in many ways. He had thought less about things, encountered fewer things, developed and drifted less.

It seems valuable to become more acquainted with my child self. I’d like to know the things he would want from me today, but also just what he was like and how he thought differently than I do. I don’t have a strict utilitarian case for this, to be clear: but imagine you had a child in your care. Wouldn’t you want to know those things about the child, just out of curiosity? and to help build a mutually agreeable local world? And shouldn’t I feel even more strongly about the child who was me, who entrusted their future to me, with whom I have in some ways an even stronger relationship and to whom I have in some ways an even greater duty of care?

Right now, perhaps because of the dream I just woke up from, I feel this most acutely for my child self. But there are other selves (as if ‘childhood Daniel’ was merely one self) I feel similarly about. Myself during the first and second halves of my undergraduate years, beginning to live away from family. Myself after just having moved to Berkeley, becoming one of the ‘rationalists’. Myself during the more difficult parts of my PhD. Right now, I have a pretty strong connection with most of these, but in the future I won’t. And even now I can feel undergraduate Daniel slipping out of my hands.

So I wish I had kept a diary, or blogged (in an unusually personal manner), or somehow or other done a better job of recording my thoughts and desires and frames and fears and hopes. I currently keep a weekly journal, which I hope is sufficient, but I must admit it’s a bit businesslike. Another way to preserve these would be interviews - perhaps this could be a new year tradition, recording a few hours of audio/video about how the past year was, what you hope for the next year, and anything from idle chit-chat to deep conversation with the hope of capturing something of what it’s like to be you on this first of January. The sleep deprivation would probably help.

But diaries are a difficult medium ...
The Leopold Model: Analysis and Reactions
by Zvi — 2024-06-14T15:10:03.480Z

Previously: On the Podcast, Quotes from the Paper

This is a post in three parts.

The first part is my attempt to condense Leopold Aschenbrenner’s paper and model into its load bearing elements and core logic and dependencies.

Two versions here, a long version that attempts to compress with minimal loss, and a short version that gives the gist.

The second part goes over where I agree and disagree, and briefly explains why.

The third part is the summary of other people’s reactions and related discussions, which will also include my own perspectives on related issues.

My goal is often to ask ‘what if.’ There is a lot I disagree with. For each subquestion, what would I think here, if the rest was accurate, or a lot of it was accurate?

Summary of Biggest Agreements and Disagreements

I had Leopold review a draft of this post. After going back and forth, I got a much better idea of his positions. They turned out to be a lot closer to mine than I realized on many fronts.

The biggest agreement is on cybersecurity and other security for the labs. I think this holds with or without the rest of the argument. This includes the need to build the data centers in America.

The arguments on power seem mostly right if the path to AGI and ASI is centrally hyperscaling on current approaches and happens quickly. I am highly uncertain about that being the path, but also America badly needs permitting reform and to build massive new sources of electrical power.

Those are both areas where the obvious responses are great ideas if we are not on the verge of AGI and will remain in the ‘economic normal’ scenario indefinitely. We should do those things.

Otherwise, while Leopold’s scenarios are possible, I am not convinced.

I do not believe America should push to AGI and ASI faster. I am still convinced that advancing frontier models at major labs faster is a mistake and one should not do that. To change that, fixing security issues (so we didn’t fully hand everything over) and power issues (so we would be able to take advantage) would be necessary but not sufficient, due to the disagreements that follow.

The most central crux is that to me a path like Leopold predicts and advocates for too frequently dooms humanity. He agrees that it is a risky path, but has more hope in it than I do, and sees no better alternative.

Everyone loses if we push ahead without properly solving alignment. Leopold...
Thoughts on Francois Chollet's belief that LLMs are far away from AGI?
by O O — 2024-06-14T06:32:48.170Z

Dwarkesh had a podcast recently with Francois Chollet (creator of Keras)

He seems fairly skeptical we are anywhere near AGI with LLMs. He mostly bases his intuition that LLMs fail on OOD tasks and don't seem to be good at solving simple abstract reasoning problems he calls the ARC challenge. It seems he thinks system 2 thinking will be a much harder unlock than people think and that scaling LLMs will go nowhere. In fact he goes so far as to say the scaling maximalists have set back AGI progress by 5-10 years. Current LLMs to him are just simply information retrieval databases.

He, along with the CEO of Zapier, have launched a 1 million dollar prize to beating the ARC bench marks, which are apparently hard for LLMs. I didn't believe it at first, given how easy they seem, but barely any progress has been made on the ARC bench marks in the last 4 years. In retrospect, it's odd that so many existing benchmarks rely heavily on memorized knowledge, and the ARC results check out with LLMs being bad at playing sudoku (so maybe not that surprising).

This seems to be in contradiction with what people on this site generally think. Is the disagreement mainly that system 2 thinking will be a relatively fast unlock (this is my take at least[1]) whereas Francois thinks it will take a long time?

Or does it go deeper?

^

Personally my intuition that LLMs are world modelers and system 2 thinking will be a relatively simple unlock as they get better at modeling the world.
Research Report: Alternative sparsity methods for sparse autoencoders with OthelloGPT.
by Andrew Quaisley — 2024-06-14T00:57:29.407Z

Abstract

Standard sparse autoencoder training uses an L1 sparsity loss term to induce sparsity in the hidden layer. However, theoretical justifications for this choice are lacking (in my opinion), and there may be better ways to induce sparsity. In this post, I explore other methods of inducing sparsity and experiment with them using Robert_AIZI's methods and code from this research report, where he trained sparse autoencoders on OthelloGPT. I find several methods that produce significantly better results than L1 sparsity loss, including a leaky top-k activation function.

Introduction

This research builds directly on Robert_AIZI's work from this research report. While I highly recommend reading his full report, I will briefly summarize the parts of it that are directly relevant to my work.

Although sparse autoencoders trained on language models have been shown to find feature directions that are more interpretable than individual neurons (

Bricken et al,

Cunningham et al), it remains unclear whether or not a given linearly-represented feature will be found by a sparse autoencoder. This is an important consideration for applications of sparse autoencoders to AI safety; if we believe that all relevant safety information is represented linearly, can we expect sparse autoencoders to bring all of it to light?

Motivated by this question, Robert trained sparse autoencoders on a version of OthelloGPT (based on the work of Li et al), a language model trained on Othello game histories to predict legal moves. Previous research (

Nanda

,

Hazineh et al

) had found that linear probes trained on the residual stream of OthelloGPT could classify each position on the board as either empty, containing an enemy piece, or containing an allied piece, with high accuracy. Robert reproduced similar results on a version of OthelloGPT that he trained himself, finding linear probes with 0.9 AUROC or greater for the vast majority of (board position, position state) pairs. He then investigated whether or not sparse autoencoders trained on OthelloGPT's residual stream would find features that classified board positions with levels of accuracy similar to the linear probes. Out of 180 possible (board position, position state) pair classifiers, Robert's best autoencoder had 33 features that classified distinc...
Slowed ASI - a possible technical strategy for alignment
by Lester Leong — 2024-06-14T00:57:25.014Z

Lately, much has been discussed about PauseAI, or even stopping research completely, until further progress has been made in theory or technical approaches to alignment. After thinking about this for some time, I wondered if there was a way to formalize this reasoning in mathematical terms when I stumbled upon what might be an interesting, possibly novel approach to alignment:

What if we leveraged the nature of slower computing substrates to run AI at a slower pace than current digital computers?

By "slower substrate", I don't mean just diminishing CPU/GPU clock speeds, number of cores, or RAM/VRAM. I mean choosing fundamentally slower forms of computing that would be impossible to speed up past some performance ceiling. In this way, we might be able to run and validate stronger-than-human AI in human time. Here's an intuition pump for why it could work.

Total Intelligence and the Three Forms Of Superintelligence

In his book Superintelligence, Nick Bostrom categorizes superintelligence into three forms:

Quality Superintelligence: Intelligence derived from the sophistication and effectiveness of thinking processes.
Speed Superintelligence: Intelligence stemming from the rapidity of thought processes.
Collective Superintelligence: Intelligence emerging from the coordinated efforts of multiple agents working in parallel.

Note that a given AI might attain superhuman performance through one, or a combination of, any of these three forms. This suggests a factored cognition in the form of a kind of Total Intelligence Equation, where Quality, Speed, and Number Of Agents in coordination are all directly proportional to Total Intelligence:

TotalIntelligence=Quality∗Speed∗NumberOfAgents

The Slower Superintelligence Mental Model

Given this relationship, it stands to reason that we might evaluate a high-quality superintelligence safely by diminishing its speed or collective capacity. In other words, if we consider Total Intelligence as a product of Quality, Speed, and Number of Agents, then a feasible approach might involve increasing the quality factor while decreasing speed and the number of agents proportionally or more. This results in a high-quality but controllable form of superintelligence.

Varying Total Intelligence by varying Number Of Agents is not a novel idea. This was the the core concept behind Paul Christiano's work on iterated amplification (and distillation), which ...
Conceptual Typography Example
by milanrosko — 2024-06-14T00:39:47.373Z

Memento mori, Latin for "remember you must die," has been a significant theme in art and philosophy, aiming to remind us of our mortality, the fleeting nature of earthly pleasures, and the imperative to live a meaningful life.
Conceptual Typography is a design technique where typography is employed not merely for communication but as an integral visual element to express ideas, emotions, or narratives.

(I hope you like it. If someone would like to have a printable version, I would provide a negative as this would be too dark for digital toner rollers.)
Safety isn’t safety without a social model (or: dispelling the myth of per se technical safety)
by Andrew_Critch — 2024-06-14T00:16:47.850Z

As an AI researcher who wants to do technical work that helps humanity, there is a strong drive to find a research area that is definitely helpful somehow, so that you don’t have to worry about how your work will be applied, and thus you don’t have to worry about things like corporate ethics or geopolitics to make sure your work benefits humanity.

Unfortunately, no such field exists. In particular, technical AI alignment is not such a field, and technical AI safety is not such a field. It absolutely matters where ideas land and how they are applied, and when the existence of the entire human race is at stake, that’s no exception.

If that’s obvious to you, this post is mostly just a collection of arguments for something you probably already realize. But if you somehow think technical AI safety or technical AI alignment is somehow intrinsically or inevitably helpful to humanity, this post is an attempt to change your mind. In particular, with more and more AI governance problems cropping up, I'd like to see more and more AI technical staffers forming explicit social models of how their ideas are going to be applied.

If you read this post, please don’t try to read it as somehow pro- or contra- a specific area of AI research, or safety, or alignment, or corporations, or governments. My goal in this post is to encourage more nuanced social models by de-conflating a bunch of concepts. This might seem like I’m against the concepts themselves, when really I just want clearer thinking about these concepts, so that we (humanity) can all do a better job of communicating and working together.

Myths vs reality

Epistemic status

: these are claims that I’m confident in, assembled over 1.5 decades of observation of existential risk discourse, through thousands of hours of conversation. They are not claims I’m confident I can convince you of, but I’m giving it a shot anyway because there’s a lot at stake when people don’t realize how their technical research is going to be misapplied.

Myth #1: Technical AI safety and/or alignment advances are intrinsically safe and helpful to humanity, irrespective of the state of humanity.

Reality: All technical advances in AI safety and/or “alignment” can be misused by humans. There are no technical advances in AI that are safe per se; the safety or unsafety of an idea is a function of t...
OpenAI appoints Retired U.S. Army General Paul M. Nakasone to Board of Directors
by Joel Burget — 2024-06-13T21:28:18.110Z
Link: https://openai.com/index/openai-appoints-retired-us-army-general/

Today, Retired U.S. Army General Paul M. Nakasone has joined our Board of Directors. A leading expert in cybersecurity, Nakasone’s appointment reflects OpenAI’s commitment to safety and security, and underscores the growing significance of cybersecurity as the impact of AI technology continues to grow.

As a first priority, Nakasone will join the Board’s

Safety and Security Committee

, which is responsible for making recommendations to the full Board on critical safety and security decisions for all OpenAI projects and operations.

Whether this was influenced by Aschenbrenner's Situational Awareness or not, it's welcome to see OpenAI emphasizing the importance of security. It's unclear how much this is a gesture vs reflective of deeper changes.
AI #68: Remarkably Reasonable Reactions
by Zvi — 2024-06-13T16:30:02.969Z

The big news this week was Apple Intelligence being integrated deeply into all their products. Beyond that, we had a modestly better than expected debate over the new version of SB 1047, and the usual tons of stuff in the background. I got to pay down some writing debt.

The bad news is, oh no, I have been called for Jury Duty. The first day or two I can catch up on podcasts or pure reading, but after that it will start to hurt. Wish me luck.

Table of Contents

AiPhone covers the announcement of Apple Intelligence. Apple’s products are getting device-wide integration of their own AI in a way they say preserves privacy, with access to ChatGPT via explicit approval for the heaviest requests. A late update: OpenAI is providing this service for free as per Bloomberg.

I offered Quotes from Leopold Aschenbrenner’s Situational Awareness Paper, attempting to cut down his paper by roughly 80% while still capturing what I considered the key passages. Then I covered his appearance on Dwarkesh’s Podcast, where I offered commentary. The plan is to complete that trilogy tomorrow, with a post that analyzes Leopold’s positions systematically, and that covers the reactions of others.

Introduction.

Table of Contents.

Language Models Offer Mundane Utility. Roll your own process.

Language Models Don’t Offer Mundane Utility. What happened to Alexa?

Fun With Image Generation. Dude, where’s my image of a car?

Copyright Confrontation. Everyone is rather on edge these days.

Deepfaketown and Botpocalypse Soon. People will do things that scale.

They Took Our Jobs. Lost your job? No problem. Start a new company!

Someone Explains it All. Data center construction, the bitter lesson.

The Art of the Jailbreak. The Most Forbidden Technique?

Get Involved. AISI hiring a senior developer.

Introducing. New OpenAI execs, new AI assistant, new short video model.

In Other AI News. More progress avoiding MatMul. Nvidia takes it all in stride.

Quiet Speculations. What you see may be what you get.

I Spy With My AI. Microsoft Recall makes some changes to be slightly less crazy.

Pick Up the Phone. Perhaps a deal could be made.

Lying to the White House, Senate and House of Lords. I don’t love it.

The Quest for Sane Regulation. People want it. Companies feel differently.

More Reasonable SB 1047 Reactions. Hearteningly sane reactions by many.

...
Four Futures For Cognitive Labor
by Maxwell Tabarrok — 2024-06-13T12:56:30.329Z
Link: https://www.maximum-progress.com/p/four-futures-for-cognitive-labor

I just returned from

Manifest

, a bay area rationalist conference hosted by the prediction market platform,

Manifold

. The conference was great and I met lots of cool people!

A common topic of conversation was AI and its implications for the future. The standard pattern for these conversations is dueling estimates of P(doom), but I want to avoid this pattern. It seems

difficult to change anyone’s mind

this way and the P(doom) framing splits all futures into heaven or hell without much detail in-between.

Instead of P(doom), I want to focus on the probability that my skills become obsolete. Even in worlds where we are safe from apocalyptic consequences of AI, there will be tumultuous economic changes that might leave writers, researchers, programmers, or academics with much less (or more) income and

influence

than they had before.

Here are four relevant analogies which I use to model how cognitive labor might respond to AI progress.

First: the printing press. If you were an author in 1400, the largest value-add you brought was your handwriting. The labor and skill that went into copying made up the majority of the value of each book. Authors confronted with a future where their most valued skill is automated at thousands of times their speed and accuracy were probably terrified. Each book would be worth a tiny fraction of what they were worth before, surely not enough to support a career.

With hindsight we see that even as all the terrifying extrapolations of printing automation materialized, the

income

,

influence

, and

number

of authors soared. Even though each book cost thousands of times less to produce, the quantity of books demanded increased so much in response to their falling cost that it outstripped the productivity gains and required more authors, in addition to a 1000x increase in books per author, to fulfill it.

Second: the mechanization of farming. This is another example of massive productivity increases but with different outcomes.

Unlike authors and the printing press, the ratio of change in production per person and change total consumption was not low enough to grow the farming labor force or even sustain it. The

per-capita incomes of farmers

have doubled several times over but there are many fewer farmers, even

in absolute numbers

.

Third: computers. Specifically, the shift from the

job title of computer

to the name of the machine that replaced it. The ...