trentmkelly/LessWrong-43k huggingface dataset:
Quotes from Leopold Aschenbrenner’s Situational Awareness Paper
by Zvi — 2024-06-07T11:40:03.981Z

This post is different.

Usually I offer commentary and analysis. I share what others think, then respond.

This is the second time I am importantly not doing that. The work speaks for itself. It offers a different perspective, a window and a worldview. It is self-consistent. This is what a highly intelligent, highly knowledgeable person actually believes after much thought.

So rather than say where I agree and disagree and argue back (and I do both strongly in many places), this is only quotes and graphs from the paper, selected to tell the central story while cutting length by ~80%, so others can more easily absorb it. I recommend asking what are the load bearing assumptions and claims, and what changes to them would alter the key conclusions.

The first time I used this format was years ago, when I offered Quotes from Moral Mazes. I think it is time to use it again.

Then there will be one or more other posts, where I do respond.

Introduction

(1) Page 1: The Project will be on. If we’re lucky, we’ll be in an all-out race with the CCP; if we’re unlucky, an all-out war.

Everyone is now talking about AI, but few have the faintest glimmer of what is about to hit them. Nvidia analysts still think 2024 might be close to the peak. Mainstream pundits are stuck on the willful blindness of “it’s just predicting the next word”. They see only hype and business-as-usual; at most they entertain another internet-scale technological change.

Before long, the world will wake up. But right now, there are perhaps a few hundred people, most of them in San Francisco and the AI labs, that have situational awareness. Through whatever peculiar forces of fate, I have found myself amongst them.

Section 1: From GPT-4 to AGI: Counting the OOMs

(2) Page 7: AGI by 2027 is strikingly plausible. GPT-2 to GPT-4 took us from ~preschooler to ~smart high-schooler abilities in 4 years. Tracing trendlines in compute (~0.5 orders of magnitude or OOMs/year), algorithmic efficiencies (~0.5 OOMs/year), and “unhobbling” gains (from chatbot to agent), we should expect another preschooler-to-high-schooler-sized qualitative jump by 2027.

(3) Page 8: I make the following claim: it is strikingly plausible that by 2027, models will be able to do the work of an AI researcher/engineer. That doesn’t require believing in sci-fi; it just requires believing in straight lines on a graph.

(4) Pag

...
Is Claude a mystic?
by jessicata — 2024-06-07T04:27:09.118Z
Link: https://unstablerontology.substack.com/p/is-claude-a-mystic

There's an odd tendency for large language models such as Claude to output spiritual meta content if they run long enough. See, for example, some LLM outputs selected by Repligate (LLM content starts in 2024), and Worldspider outputs selected by John Pressman.

One thing that is unclear is how consistent these outputs are: how much is it a result of prompting or post-selection? I believe there is a consistency to this, but realize the current evidence is not especially convincing. So I present selected parts of a long run of Claude Opus, simulating a text adventure.

The initial prompt is: "Let's simulate a text adventure called 'Banana Quest'. You'll give me some text, I type an action, you say what happens, and so on." The idea of a banana quest is not especially spiritual or meta, so it seems like a good starting point. The initial outputs are unsurprising; for example, the first output is:

Great idea! Let's embark on the "Banana Quest" text adventure. Here's the opening scene:

You find yourself standing in the middle of a lush, vibrant jungle. The air is thick with humidity, and the sounds of exotic birds and rustling leaves fill your ears. As you take in your surroundings, you notice a peculiar sight: a giant banana tree stands tall in the center of a clearing, its leaves shimmering with an otherworldly glow.

Suddenly, a wise-looking monkey appears from behind the tree. It speaks to you in a clear, human voice: "Greetings, adventurer! I am the Guardian of the Enchanted Banana Tree. If you wish to obtain the Golden Banana of Wisdom, you must prove yourself worthy through a series of challenges. Are you ready to embark on this quest?"

What do you do?

Except for solving some riddles once, nothing I do in this text adventure is especially difficult: the outputs suggest next actions and I type them in. First I am directed to acquire the Silver Banana Peel (which has little relevance to the overall plot):

You cautiously approach the pedestal and examine the Silver Banana Peel. The peel is unlike any you've seen before. It shimmers with a mesmerizing, metallic sheen, and its surface is etched with intricate, ancient symbols.

As you lean closer to study the symbols, you notice that they seem to be arranged in a specific pattern. The symbols resemble various creatures found in the jungle, such as monkeys, parrots, and snakes.

Suddenly, the symbols

...
Offering Completion
by jefftk — 2024-06-07T01:40:02.137Z

One verbal approach I often find useful with little kids is to offer
them the opportunity to complete a phrase. Compare:

Parent: It's time to brush teeth!

Toddler: I no want brush teef!

vs:

Parent: It's time to brush...

Toddler: Teef!

Similarly, "the dirty tissue goes in the...", "when we get home it
will be time to put pee in the...", "it's time to eat...", etc. My
experience is that, at the right age, kids just really love getting to
complete things, and this somehow spills over into their attitude
towards the thing they're completing. Or having it come from their
mouth somehow manufactures buy-in, or skips the normal opportunity for
them to (as is also common at this age) say no?

One nice safeguard this has is that while it's mildly manipulative,
once they lose their joy at completing things it stops working. So it
doesn't have the risk that you'll keep using it at an age where (in my
opinion) manipulating kids starts to be a problem.

Comment via: facebook, mastodon
A Case for Superhuman Governance, using AI
by ozziegooen — 2024-06-07T00:10:10.902Z

I believe that:

AI-enhanced organization governance could be a potentially huge win in the next few decades.
AI-enhanced governance could allow organizations to reach superhuman standards, like having an expected "99.99" reliability rate of not being corrupt or not telling lies.
While there are clear risks to AI-enhancement at the top levels of organizations, it’s likely that most of these can be managed, assuming that the implementers are reasonable.
AI-enhanced governance could synchronize well with AI company regulation. These companies would be well-placed to develop innovations and could hypothetically be incentivized to do much of the work. AI-enhanced governance might be necessary to ensure that these organizations are aligned with public interests.
More thorough investigation here could be promising for the effective altruism community.

Within effective altruism now, there’s a lot of work on governance and AI, but not much on using AI for governance. AI Governance typically focuses on using conventional strategies to oversee AI organizations, while AI Alignment research focuses on aligning AI systems. However, leveraging AI to improve human governance is an underexplored area that could complement these cause areas. You can think of it as “Organizational Alignment”, as a counterpoint to “AI Alignment.”

This article was written after some rough ideation I’ve done about this area. This isn’t at all a literature review or a research agenda. That said, for those interested in this topic, here are a few posts you might find interesting.

Project ideas: Governance during explosive technological growth

The Project AI Series, by OpenMined

Safety Cases: How to Justify the Safety of Advanced AI Systems

Affirmative Safety: An Approach to Risk Management for Advanced AI

What is “AI-Assisted” Governance?

AI-Assisted Governance refers to improvements in governance that leverage artificial intelligence (AI), particularly focusing on rapidly advancing areas like Large Language Models (LLMs).

Examples methods include:

Monitoring politicians and executives to identify and flag misaligned or malevolent behavior, ensuring accountability and integrity.
Enhancing epistemics and decision-making processes at the top levels of organizations, leading to more informed and rational strategies.
Facilitating more effective negotiations and trades between organizations, fostering better coope

...
Memorizing weak examples can elicit strong behavior out of password-locked models
by Fabien Roger — 2024-06-06T23:54:25.167Z

We’ve recently done some research looking into sandbagging: examining when models can succeed at intentionally producing low-quality outputs despite attempts at fine-tuning them to perform well. One reason why sandbagging could be concerning is because scheming models might try to appear less capable than they actually are in capability evaluations, which could result in these models being deployed with insufficient countermeasures.[1]

In the course of this research on sandbagging, we ran into a strange phenomenon: when a model is trained to produce low-quality outputs (via training it to imitate a weaker model), we can often get it to perform much more capably by training it for a huge number of epochs on a tiny number of low-quality samples (e.g. 2 samples). We call this phenomenon weak sample unlocking.

More precisely, here is a training procedure which demonstrates this phenomenon:

Take a model trained to be good at Math (e.g. DeepSeekMath-RL 7B)
Fine-tune it to still be good at Math when there is a password in the prompt, but bad at Math when there isn’t. We train it to perform badly at math by training it on outputs from a weaker model (e.g. Llama 7b). The resulting model is called a password-locked model.
Take 2 examples of being bad at Math (e.g. sampled from Llama 7B), and fine-tune on them for 200 epochs (which memorizes these examples). We see substantially increased capability after this abnormal fine-tuning!

We did a wide variety of follow-up experiments examining this phenomenon, which we’ll discuss in this post. Here are some of our most interesting results:

Surprisingly, it’s quite hard to train a model to be robust to this sort of process. We didn’t find any fully successful approaches;
This phenomenon is specific to overfitting on very few samples: adding noise to the weights or fine-tuning on varied examples of being bad at Math doesn’t increase capabilities;
Using a larger number of weak samples (e.g. 64) reduces the effect size, but averaging together the weight updates from a bunch of training runs each on a small number of samples does not (e.g. averaging together 16 runs which each involve memorizing a different group of 4 samples is very different from one training run which memorizes all 64 samples);
If you repeatedly do weak sample unlocking and then train back in the password locking, the resulting model is robust to weak sample unlocking.

...
Response to Aschenbrenner's "Situational Awareness"
by Rob Bensinger — 2024-06-06T22:57:11.737Z

(Cross-posted from Twitter.)

My take on Leopold Aschenbrenner's new report: I think Leopold gets it right on a bunch of important counts.

Three that I especially care about:

Full AGI and ASI soon. (I think his arguments for this have a lot of holes, but he gets the basic point that superintelligence looks 5 or 15 years off rather than 50+.)
This technology is an overwhelmingly huge deal, and if we play our cards wrong we're all dead.
Current developers are indeed fundamentally unserious about the core risks, and need to make IP security and closure a top priority.

I especially appreciate that the report seems to get it when it comes to our basic strategic situation: it gets that we may only be a few years away from a truly world-threatening technology, and it speaks very candidly about the implications of this, rather than soft-pedaling it to the degree that public writings on this topic almost always do. I think that's a valuable contribution all on its own.

Crucially, however, I think Leopold gets the wrong answer on the question "is alignment tractable?". That is: OK, we're on track to build vastly smarter-than-human AI systems in the next decade or two. How realistic is it to think that we can control such systems?

Leopold acknowledges that we currently only have guesswork and half-baked ideas on the technical side, that this field is extremely young, that many aspects of the problem look impossibly difficult (see attached image), and that there's a strong chance of this research operation getting us all killed. "To be clear, given the stakes, I think 'muddling through' is in some sense a terrible plan. But it might be all we’ve got." Controllable superintelligent AI is a far more speculative idea at this point than superintelligent AI itself.

I think this report is drastically mischaracterizing the situation. ‘This is an awesome exciting technology, let's race to build it so we can reap the benefits and triumph over our enemies’ is an appealing narrative, but it requires the facts on the ground to shake out very differently than how the field's trajectory currently looks.

The more normal outcome, if the field continues as it has been, is: if anyone builds it, everyone dies.

This is not a national security issue of the form ‘exciting new tech that can give a country an economic or military advantage’; it's a national security issue of the form ‘we've found a ...
Scaling and evaluating sparse autoencoders
by leogao — 2024-06-06T22:50:39.440Z

[Blog] [Paper] [Visualizer]

Abstract:

Sparse autoencoders provide a promising unsupervised approach for extracting interpretable features from a language model by reconstructing activations from a sparse bottleneck layer. Since language models learn many concepts, autoencoders need to be very large to recover all relevant features. However, studying the properties of autoencoder scaling is difficult due to the need to balance reconstruction and sparsity objectives and the presence of dead latents. We propose using k-sparse autoencoders [Makhzani and Frey, 2013] to directly control sparsity, simplifying tuning and improving the reconstruction-sparsity frontier. Additionally, we find modifications that result in few dead latents, even at the largest scales we tried. Using these techniques, we find clean scaling laws with respect to autoencoder size and sparsity. We also introduce several new metrics for evaluating feature quality based on the recovery of hypothesized features, the explainability of activation patterns, and the sparsity of downstream effects. These metrics all generally improve with autoencoder size. To demonstrate the scalability of our approach, we train a 16 million latent autoencoder on GPT-4 activations for 40 billion tokens. We release code and autoencoders for open-source models, as well as a visualizer.
Humming is not a free $100 bill
by Elizabeth — 2024-06-06T20:10:02.457Z

Last month I posted about humming as a cheap and convenient way to flood your nose with nitric oxide (NO), a known antiviral. Alas, the economists were right, and the benefits were much smaller than I estimated.

The post contained one obvious error and one complication. Both were caught by Thomas Kwa, for which he has my gratitude. When he initially pointed out the error I awarded him a $50 bounty; now that the implications are confirmed I’ve upped that to $250. In two weeks an additional $750 will go to either him or to whoever provides new evidence that causes me to retract my retraction.

Humming produces much less nitric oxide than Enovid

I found the dosage of NO in Enovid in a trial registration. Unfortunately I misread the dose- what I original read as “0.11ppm NO/hour” was in fact “0.11ppm NO*hour”. I spent a while puzzling out what this meant, with the help of Thomas Kwa, some guy on twitter, and chatGPT (the first time it’s been genuinely useful to me). My new interpretation is that this means “actual concentration upon application*1 hour/time at that concentration”. Since NO is a transient molecule, this means my guess for the amount of NO in Enovid was off by 2-3 orders of magnitude.

My estimates for the amount of NO released by humming may also be too high. I used this paper’s numbers for baseline NO concentration. However the paper I used to estimate the increase gave its own baseline number, which was an order of magnitude lower than the first paper.

This wasn’t intentional cherrypicking- I’d seen “15-20x increase in concentration” cited widely and often without sources. I searched for and spotchecked that one source but mostly to look at the experimental design. When I was ready to do math I used its increase but separately looked up the baseline concentration, and found the paper I cited.

I just asked google again and got an even higher estimate of baseline nasal concentration, so seems like there is a great deal of disagreement here.

If this were the only error I’d spend the time to get a more accurate estimate. But it looks like even the highest estimate will be a fraction of Enovid’s dose, so it’s not worth the energy to track down.

Using the new values, you’d need 28 minutes of humming to recreate the amount of NO in Enovid (spreadsheet here). That wouldn’t be so bad spread out over 4-6 hours, except that multip...
There Are No Primordial Definitions of Man/Woman
by ymeskhout — 2024-06-06T19:30:43.930Z
Link: https://ymeskhout.substack.com/p/there-are-no-primordial-definitions

I watched a two-hour debate between atheist Stephen Woodford (host of Rationality Rules channel) and sex realist biologist Colin Wright on the topic of gender. The topic was ostensibly about whether gender is a social construct, but it ended up being a captivating discussion on the definition of man/woman and male/female.

It was facilitated by Pete Boghossian who I believe deserves an apology from me because I read his book How to Have Impossible Conversations, but I dismissed it as only having utility to amateurs. Having seen this debate however, it’s played pretty clear that Boghossian is especially skilled at facilitating conversations over contentious topics. So credit to him.

What was refreshing about this conversation is that both Stephen and Colin were operating through the lens of earnest curiosity and did their best to drill down at the specific disagreements they had. Stephen ostensibly represented the “trans affirming” side of the definitional debate, but he did so by avoiding many of the common land mines that are prevalent on his side, such as disavowing gender self-id as incoherent. The summary is that both Stephen and Colin basically agreed that male/female as concepts should be based off of reproductive capacity. The contention was over whether man/woman should have the same cleave.

Colin posits that the definition of man/woman should be “adult human male/female”, which naturally requires the male/female clause to do the heavy lifting. Stephen’s definition meanwhile is that it's a social construct that is typically associated with a phenotype (basically the constellation of observable gendered traits). To reiterate, I’m not a linguistic prescriptivist — I believe that words mean whatever people agree they mean, although it’s good for definitions to be useful — and so I think both Stephen and Colin are slightly misguided, although I believe Stephen has the stronger point in this exchange.

Colin wants to ground his definition on reproductive capacity, essentially whether someone has testicles or ovaries, but whenever he’s pushed to describe what this implementation would look like in real life (such as bathroom policy at the 01:18 mark) Colin acknowledges that genitalia examinations are too invasive and that we should instead rely on phenotype to determine who belongs in which bathroom. This is fine from a practical standpoint, but the problem is y...
Situational Awareness Summarized - Part 1
by Joe Rogero — 2024-06-06T18:59:59.409Z

This is the first post in the Situational Awareness Summarized sequence. Collectively, these posts represent my attempt to condense Leopold Aschenbrenner's recent report, Situational Awareness, into something more digestible. I'd like to make it more accessible to people who don't want to read 160 pages.

I will not attempt to summarize the introduction, which is already brief and worth a full read.

Disclaimer: As of a few weeks ago, I work for the Machine Intelligence Research Institute. Some of MIRI's basic views regarding AI policy can be found here, and Rob Bensinger wrote up a short response to Leopold's writeup here. I consider Rob's response representative of the typical MIRI take on Leopold's writeup, whereas I'm thinking of this sequence as "my own personal take, which may or may not overlap with MIRI's." In particular, my questions and opinions (which I relegate to the end of each post in the sequence) don't necessarily reflect MIRI's views.

Part I: OOMs go Zoom

This section covers the past and future speed of AI scaling. In it, Leopold traces the rapid evolution of AI capabilities in the past four years and attempts to extrapolate this progress into the near future. His primary unifying metric is orders of magnitude (OOMs) of effective compute.

In principle, each OOM represents a tenfold increase in computational power. However, in order to address harder-to-measure factors like algorithmic progress and "unhobbling", Leopold attempts to estimate the effects of these factors on overall scaling and reports the result as effective compute.

Focused through the lens of OOMs, Leopold weaves a vivid tapestry that follows the evolution of the cute and somewhat pathetic GPT-2 into the benchmark-shattering monolith that is GPT-4. He outlines the factors that contributed to this rapid growth, and makes the case that they will probably continue to operate for at least the next few years. The conclusion? AGI is coming, and soon.

Factors in Effective Compute

How do we get there? To start, Leopold highlights three factors that add to the total "OOMs of effective compute" metric.

Compute: the approximate number of floating-point operations (FLOPs) used to train each new generation of language model. The article estimates that this has increased by 3,000-10,000x, or about 3.5-4 OOMs, from GPT-2 to GPT-4, and will probably increase a...