trentmkelly/LessWrong-43k huggingface dataset:
Using Dangerous AI, But Safely?
by habryka — 2024-11-16T04:29:20.914Z

Rob Miles has released a new video, this time on control!

Youtube Transcript (Very rough)

0:00

hi so this channel is about AI safety a lot of people when they hear AI safety

0:05

the first thing that comes to mind is they think it's about how to keep potentially dangerous AI systems under control that we're trying to answer

0:11

questions like what can you do if your AI turns out to be malicious and that is part of it but actually the field has

0:17

tended not to focus on that kind of thing so much because it doesn't scale well by which I mean it seems like AI

0:24

has the potential to become enormously more capable than humans sooner or later and if you have ai research being done

0:30

by AI probably sooner you're going to get super intelligence you'll get AI systems vastly more capable than humans

0:37

at all sorts of tasks just massively outperforming us across the board what can you do if a super intelligence is

0:42

trying to kill you well you could die [Music]

0:48

or hello darkness yeah no that's that's pretty much your options realistically if the AI system is enormously more

0:55

competent than you and it really wants to do something it's probably going to figure out a way to do it whatever you

1:01

try to do to stop it so what can we do well here's a question what's the best way to avoid being bitten by a shark

1:08

punch it in the eye stab it in the gills cast a magic spell no it's none of those

1:13

it's don't go in the ocean in the same way the best way to avoid being killed by a malevolent super intelligent AI is

1:20

to not build one of those so the focus is often more on alignment rather than how to handle AI that might be trying to

1:26

kill us alignment is more like how to make AI that's definitely not trying to kill us but alignment turned out to be

1:34

hard our current alignment techniques aren't remotely up to the task of properly aligning an AGI and it looks

1:41

like we maybe don't have that much time really we shouldn't be trying to

1:47

make AI much more powerful than it is until we can figure out much better alignment techniques than we have but if

1:53

we can't manage to do that it might be nice to also have some methods for keeping possibly misaligned a systems

2:00

under control as well AI companies should obviously have an obligation to not deploy AI that could kill us so once

2:06

AI systems start to get capable enough that they maybe could kill us if they wanted to th

...
Ayn Rand’s model of “living money”; and an upside of burnout
by AnnaSalamon — 2024-11-16T02:59:07.368Z

Epistemic status: Toy model. Oversimplified, but has been anecdotally useful to at least a couple people, and I like it as a metaphor.

Introduction

I’d like to share a toy model of willpower: your psyche’s conscious verbal planner “earns” willpower (earns trust with the rest of your psyche) by choosing actions that nourish your fundamental, bottom-up processes in the long run. For example, your verbal planner might expend willpower dragging you to disappointing first dates, then regain that willpower, and more, upon finding you a good long-term romance. Wise verbal planners can acquire large willpower budgets by making plans that, on average, nourish your fundamental processes. Delusional or uncaring verbal planners, on the other hand, usually become “burned out” – their willpower budget goes broke-ish, leaving them little to no access to willpower.

I’ll spend the next section trying to stick this discussion into a larger context, then get back to willpower.

On processes that lose their relationship to the unknown

Good scientists seek truth, craft hypotheses that stick their necks out, and send their ideas forth to die in their stead, while retaining their own ability to change their minds.

Actual humans who aspire to be scientists, on the other hand, often forget that this is the goal, and instead get stuck defending some fixed theory.

I suspect this forgetting is a special case of a general phenomenon: many processes, if they are to remain healthy, must take an active interest in things they don’t yet know. And yet those same processes will tend to collapse into structures that actively resist new data. To list a few other examples:

Romantic relationships, family relationships, and friendships all work better when one remembers the other person is partly unknown (and practices “respect,” “paying attention,” “not taking for granted” – practices for consciously relating to the unknownness); but this is easy to forget.
New businesses often take an active interest in e.g. what their customers want, how to make it, what their surroundings are like, or other things they don’t know. But many later turn into bureaucracies that protect their local stasis from new data.
“Every cause wants to be a cult.”
Jordan Hall argues that our society as a whole has gradually replaced "thinking" (which has the ability to explore awkwardly when that's the

...
Fundamental Uncertainty: Epilogue
by Gordon Seidoh Worley — 2024-11-16T00:57:48.823Z

I wrote a whole book! What's next?

I'm currently doing an edit pass on the entire book. I need to rewrite some of the early sections, fix some consistency issues, and generally look with fresh eyes on words I wrote months or years ago. Many of you provided helpful comments, and I'm using those to make the second draft better.

When the second draft is done, I'll look to hire one or two editors/pre-readers, ideally from the LessWrong community, who can go through it and point out all the obvious mistakes that are still in the book, plus help me make the confusing parts clearer.

Once that's done, I'll be ready to publish. That might mean finding a traditional publisher who will take a chance on a monograph from an unknown author, but more likely I'll self publish, which will give me greater flexibility to make the book available in many forms, including a free-to-read version on the web. If you have experience here, I'd love to talk to you!

And no matter how I publish, I need to build an audience to help people find and read the book. So I'm launching my Substack blog today, Uncertain Updates. My plan is to treat it like a newsletter, publishing about once a month, with updates about the book and my other writing projects.

Finally, thanks to everyone who supported me along the way to write this first draft. I appreciate all of the encouragement, critical comments, and even just letting me be when I was heads down working on a difficult section.

Update 2025-12-19: The book has a website.
Making a conservative case for alignment
by Cameron Berg — 2024-11-15T18:55:40.864Z

Trump and the Republican party will wield broad governmental control during what will almost certainly be a critical period for AGI development. In this post, we want to briefly share various frames and ideas we’ve been thinking through and actively pitching to Republican lawmakers over the past months in preparation for the possibility of a Trump win.

Why are we sharing this here? Given that >98% of the EAs and alignment researchers we surveyed earlier this year identified as everything-other-than-conservative, we consider thinking through these questions to be another strategically worthwhile

neglected

direction.

(Along these lines, we also want to proactively emphasize that

politics is the mind-killer

, and that, regardless of one’s ideological convictions, those who earnestly care about alignment must

take

seriously

the possibility that Trump will be the US president who presides over the emergence of AGI—and update accordingly in light of this possibility.)

Political orientation: combined sample of (non-alignment) EAs and alignment researchers

AI-not-disempowering-humanity is conservative in the most fundamental sense

The project of avoiding AI-induced human extinction and preserving our most fundamental values in the process are efforts that intuitively resonate with right-leaning thinkers and policymakers. Just as conservatives seek to preserve and build upon civilization's core institutions and achievements, alignment research aims to ensure our technological advances remain anchored to—rather than in conflict with—our core values.
Consider Herman Kahn, a leading military strategist and systems theorist at RAND Corporation during the 1950s. He is famous for popularizing the notion of ‘

thinking the unthinkable

’ with respect to the existential risks faced during the Cold War nuclear arms race when most refused to systematically analyze the possibility of thermonuclear war and its aftermath. This same philosophy, intuitive to conservatives and liberals alike, cleanly maps onto our current moment with AI.
A conservative approach to AI alignment doesn’t require slowing progress, avoiding open sourcing,[1] etc. Alignment and innovation are mutually necessary, not mutually exclusive: if alignment R&D indeed makes systems more useful and capable, then investing in alignment is investing in US tech leadership.[2] Just a

...
The Case For Giving To The Shrimp Welfare Project
by Bentham's Bulldog — 2024-11-15T16:03:57.712Z

Crosspost of this on my blog. I'd appreciate if you could share this, as, for reasons I'll explain, I think convincing people on this issue is insanely important.

Open your mouth for the mute, for the rights of all who are destitute.

—Proverbs 31:8

Imagine that you came across 1,500 shrimp about to be painfully killed. They were going to be thrown onto ice where slowly, agonizingly, over the course of 20 minutes, they’d suffocate and freeze to death at the same time, a bit like suffocating

in a suitcase

in the middle of Antarctica. Imagine them struggling, gasping, without enough air, fighting for their lives, but it’s no use.

Fortunately, there’s a machine that will stun every shrimp, so that they’ll be unconscious during their deaths, rather than in extreme agony. But the machine is broken. To fix it, you’d have to spend a dollar. Should you do so? We can even sweeten the deal and imagine that the machine won’t just be used this year—it will be used year after year, saving 1,500 shrimp per year.

It seems obvious that you should spend the dollar. Extreme agony is bad. If you can prevent literally thousands of animals from being in extreme agony for the cost of a dollar—for around a fourth of the cost of a cup of coffee—of course you should do so! It’s common sense. In fact, this would be the best dollar you spent all year—every penny would save 16 shrimp from an agonizing death per year!

I asked chat GPT to make an image of 1,500 shrimp in a lecture hall—here’s the image, but it’s only of ~200 shrimp, so you really save much more than this:

(Above image is not realistic—shrimp do not actually attend lectures).

It turns out, this scenario isn’t just a hypothetical. One of the best charities you can give to is called the

shrimp welfare project

(if you want to donate monthly, you can do so

here

). For every dollar it gets, it saves about

1,500 shrimp from a painful death every year

.

The way it works is simple and common sense:

it gives stunners

to companies that kill shrimp so long as they agree to use them to stun at least 120 million shrimp. They also secure welfare commitments from corporations to stop crushing the eyes of live shrimp in order to increase their fertility and to use humane slaughter (e.g. they worked with Tesco to get an extra

1.6 billion shrimp

stunned before slaughter every year). In total, they’ve helped around

2.6 billion shrimp

...
Win/continue/lose scenarios and execute/replace/audit protocols
by Buck — 2024-11-15T15:47:24.868Z

In this post, I’ll make a technical point that comes up when thinking about risks from scheming AIs from a control perspective.

In brief: Consider a deployment of an AI in a setting where it’s going to be given a sequence of tasks, and you’re worried about a safety failure that can happen suddenly. Every time the AI tries to attack, one of three things happens: we win, we lose, or the deployment continues (and the AI probably eventually attempts to attack again). So there’s two importantly different notions of attacks “failing”: we can catch the attack (in which case we win), or the attack can fail silently, either because it’s defective or because it’s defused by safety techniques, such that the attack doesn’t end up causing an unacceptable outcome but the AI also doesn’t get caught (in which case the deployment continues). When reasoning about how safe a protocol is, it’s crucial to distinguish between these two ways an attack can fail. And when designing protocols, it’s often worth thinking separately about the techniques that make the AI lose and the techniques that let the deployment continue without a win or loss.

This point has been made in passing in many places (our control paper, our toy models post, Charlie Griffin’s control paper, Towards Safety Cases For AI Scheming, various upcoming papers) but I wanted to have a single place that makes the point clearly. Thanks to Alex Mallen, Josh Clymer, Joe Benton, and Justis Mills for comments, and to Monte MacDiarmid for providing experimental data that I analyze.

Here’s a more precise description of the scenario:

You’re deploying your AI to do a large number of discrete tasks (e.g. implementing many features across the codebase inside a company, or doing many ML research projects).

Let’s assume that each task will be completed fairly quickly compared to the timescale of the whole deployment: for example, you’re trying to analyze the safety of deploying this AI for six months, and the tasks will take an average of an hour; each task lasts about 1/4000 of the whole deployment. The system will be doing many tasks in parallel.

You’re worried about a concentrated failure (a.k.a. “high-stakes failure”): in absence of countermeasures, the AI would be able to cause an unacceptable outcome just by behaving badly on a small number of tasks. Examples of concentrated failures include:

Causing a rog

...
Antonym Heads Predict Semantic Opposites in Language Models
by Jake Ward — 2024-11-15T15:32:14.102Z

In general, attention layers in large language models do two types of computation: They identify which token positions contain information relevant to predicting the next token (the QK circuit), and they transform information at these positions in some way that is useful for predicting the next token (the OV circuit). By analyzing the OV circuits in a model's attention heads, we can get an idea of the functionality that each head implements. In this post, I find that LLMs ranging a variety of sizes and architectures contain antonym heads: attention heads which compute the semantic opposite of a token, mapping "hot" -> "cold", "true" -> "false", "north" -> "south", etc. I first identify antonym heads by analyzing OV matrices over a set of curated tokens, and find evidence of interpretable eigenvalues in these matrices. I additionally verify the function of these heads through ablation studies, and explore the extent to which antonym heads are present in a variety of model sizes and architectures.

This post is inspired by work done by Gould et. al. and more recently Anthropic, identifying "successor heads" which compute the the successor of ordinal tokens such as "first" -> "second" or "3" -> "4".

Analyzing Weights

I use the following methodology:

Hypothesize about some functionality which may exist in a model's attention heads (antonym prediction).
Curate a dataset which should produce an identifiable pattern in the OV circuit of a head which implements this functionality.
Hunt for this pattern in each attention head.

With the help of an LLM, I create a dataset of 54 word pairs which map common words to their opposites, yielding a total of 108 words. These words are selected such that each of the studied models' tokenizers outputs a single token per word[1], and I try to make the pairs mostly independent.

Words from these pairs are arranged into a single list, with a pair's first word appearing in the first half of the list, and the second word appearing in the second half:

first_word_idx = pair_idx
second_word_idx = pair_idx + len(pairs)

This gives a word list that looks like:

true, hot, tall, big, ..., false, cold, short, small, ...

By arranging words in this way, a clear two-diagonal pattern manifests in the mapping of words in the list to their complements. I use this pattern to identify antonym prediction behavior by computing the same mapping using t...
Proposing the Conditional AI Safety Treaty (linkpost TIME)
by otto.barten — 2024-11-15T13:59:01.050Z
Link: https://time.com/7171432/conditional-ai-safety-treaty-trump/

Technological progress can excite us, politics can infuriate us, and wars can mobilize us. But faced with the risk of human extinction that the rise of artificial intelligence is causing, we have remained surprisingly passive. In part, perhaps this was because there did not seem to be a solution. This is an idea I would like to challenge.

AI’s capabilities are ever-improving. Since the release of ChatGPT two years ago, hundreds of billions of dollars have poured into AI. These combined efforts will likely lead to Artificial General Intelligence (AGI), where machines have human-like cognition, perhaps within just a few years.

Hundreds of AI scientists think we might lose control over AI once it gets too capable, which could result in human extinction. So what can we do?

The existential risk of AI has often been presented as extremely complex. A 2018 paper, for example, called the development of safe human-level AI a “super wicked problem.” This perceived difficulty had much to do with the proposed solution of AI alignment, which entails making superhuman AI act according to humanity’s values. AI alignment, however, was a problematic solution from the start.

First, scientific progress in alignment has been much slower than progress in AI itself. Second, the philosophical question of which values to align a superintelligence to is incredibly fraught. Third, it is not at all obvious that alignment, even if successful, would be a solution to AI’s existential risk. Having one friendly AI does not necessarily stop other unfriendly ones.

Because of these issues, many have urged technology companies not to build any AI that humanity could lose control over. Some have gone further; activist groups such as PauseAI have indeed proposed an international treaty that would pause development globally.

That is not seen as politically palatable by many, since it may still take a long time before the missing pieces to AGI are filled in. And do we have to pause already, when this technology can also do a lot of good? Yann Lecun, AI chief at Meta and prominent existential risk skeptic, says that the existential risk debate is like “worrying about turbojet safety in 1920.”

On the other hand, technology can leapfrog. If we get another breakthrough such as the transformer, a 2017 innovation which helped launch modern Large Language Models, perhaps we could reach AGI in a few months’ train...
A Theory of Equilibrium in the Offense-Defense Balance
by Maxwell Tabarrok — 2024-11-15T13:51:33.376Z
Link: https://www.maximum-progress.com/p/a-theory-of-equilibrium-in-the-offense

The offense-defense balance is a concept that compares how easy it is to protect vs conquer or destroy resources. For example, autonomous weapons and AI systems might make attacks easier and more scalable compared to defensive measures. The balance matters because when offense has the advantage, it can create instability and increase the likelihood of conflict.

I claim that worries over a massively upset offense-defense balance make the same mistake as peak oil apocalypse prophecies.

Peak Oil

Peak oil hysteria was seeded in the environmentalism and energy crisis of the 1970s and reached its apogee in the early 2000s with a flood of books, documentaries, and movements predicting civilizational collapse.

In some ways, the predictions of peak oil prophets did come true. Oil prices rose to an all-time

peak in 2008

, and

oil consumption

in the US reached its peak around the same time. Few, if any, of the forecasted societal or economic collapses came to fruition though.

The problem with the peak oil predictions was a selective extrapolation of some trends, while holding the adaptation of others constant. They projected rising oil prices and squabbles over the remaining finite supply that would force more and more uses for oil to shut down, but they didn’t project the new substitutes and oil sources that these high prices would create.

When oil prices rise from $100 to $200 a barrel, it doesn’t double everyone’s energy costs. $200 oil prices don’t force all plastic manufacturers to pay $200 for their chemical inputs, for example. They force them to substitute petroleum and use $150 dollars of vegetable oil instead. The new $200 price is an upper bound on what previous oil consumers pay after the price increase. Any consumers with substitutes that cost less than $200 will use those instead. So even in the most extreme world where oil runs out, most oil consumers aren’t faced with infinite costs and forced to shut down. Instead, they are only forced to pay for their next best substitute.

Offense-Defense Balance

As an example of how this mistake applies to the offense-defense balance, consider the offense-defense balance of presidential assassination. In the 19th century, guns were inaccurate at long ranges, so assassins needed to get close. Security used this defensive advantage to protect the president, but some assassins did slip through. Accurate, long range rifles are a ...
Boston Secular Solstice 2024: Call for Singers and Musicans
by jefftk — 2024-11-15T13:50:07.827Z

This year's Boston Secular Solstice will be on Saturday December 28th,
and again I'm organizing the music. Are you interested in singing or
playing? A wide range of instruments work here: in the past I think
we've had people play piano, flute, guitar, mandolin, and cello. This
isn't a large time commitment: we typically meet once or twice before
the event for an evening to run through songs.

Here's something I wrote up about last year, with links to the songs
we did:
Boston Solstice 2023 Retrospective.

We haven't finalize the song list yet, but the current draft is
"Always Look on the Bright Side of Life", "Battle Hymn of the
Rationalist Community", "Brighter Than Today", "Endless Lights", "Find
My Tribe", "Gather Round", "Give My Children Wings", "No One Is
Alone", "Old Devil Time", "Somebody Will", "The Circle", "The Mary
Ellen Spider Song", "We Will All Go Together When We Go", "When I
Die", "When I'm Gone", and "You've Got A Friend In Me".

Let me know if this sounds fun!

Comment via: facebook, mastodon, bluesky