trentmkelly/LessWrong-43k huggingface dataset:
Notes on "the hot mess theory of AI misalignment"
by JakubK — 2023-04-21T10:07:49.509Z
Link: https://sohl-dickstein.github.io/2023/03/09/coherence.html

TL;DR: the author makes some reasonable, good-faith arguments (including analysis of a neat survey that he sent to his friends), but I don't think these arguments are strong enough to support his conclusions in a significant way.

Jascha Sohl-Dickstein is a senior staff research scientist at Google Brain with an impressive research track record; for example, he basically co-invented diffusion models. In March he wrote a blog post titled "The hot mess theory of AI misalignment: More intelligent agents behave less coherently." My notes are below.

The start of the post

Jascha doesn't make any ad hominem or psychoanalysis-based arguments, which is promising. He also cites Katja Grace's "Counterarguments to the basic AI x-risk case." Rohin Shah + Geoffrey Irving reviewed earlier versions of Jascha's post.
Jascha states "Most work on AGI misalignment risk assumes that, unlike us, smart AI will not be a hot mess." I don't know where he gets this impression; I doubt that most AGI alignment researchers would agree with the statement "your research assumes that smart AI will not be a hot mess."
The post contains a decent summary of the classic misaligned brain-in-a-box scenario. Jascha calls this "the common narrative of existential risk from misaligned AGI," which seems misleading since there are other threat models.
In an earlier post, Jascha argued that Goodhart's law makes extreme efficiency perilous.
Jascha is "extremely glad people are worrying about and trying to prevent negative consequences from AI."
But he also thinks "predicting the future is hard, and predicting aspects of the future which involve multiple uncertain steps is almost impossible." This is reasonable and resembles some of Nuño Sempere's concerns.
He goes on to claim that "An accidentally misaligned superintelligence which poses an existential risk to humanity seems about as likely as any other specific hypothesis for the future which relies on a dependency chain of untested assumptions." This conclusion, which very faintly resembles the "safe uncertainty fallacy," seems far too strong to draw from (6) alone.
In footnote 5, Jascha outlines other AI x-risks that seem "at least as plausible" as misaligned AGI:

WW3 with AI-guided WMDs
terrorists use AI to build WMDs
a regime uses AI to make everyone compliant forever
highly addictive AI-generated experiences
massive unemployment with zero social safety nets

...
Requisite Variety
by Stephen Fowler — 2023-04-21T08:07:28.751Z

Epistemics:
First two sections are almost half a century old ideas from early Control Theory. Last section on alignment is speculative.

The Law of Requisite Variety
Illustration inspired by Norman and Bar-Yam, 2018

TL;DR:
Requisite Variety concerns the straightforward relationship between the number of actions we can take and the number of different actions required to mitigate problems that could arise. I use it to argue maintaining control of an increasingly complex system of AI's will require humans to dramatically improve their cognitive "throughput" or ultimately surrender control.

“Both of these qualities of the complex system—heterogeneity in the parts, and richness of interaction between them—have the same implication: the quantities of information that flow, either from system to observer or from part to part, are much larger than those that flow when the scientist is physicist or chemist. And it is because the quantities are large that the limitation is likely to become dominant in the selection of the appropriateic strategy”

- W. Ross Ashby, “Requisite variety and its implications for the control of complex systems”

"Can the pace of technological progress continue to speed up indefinitely? Is there not a point where humans are unable to think fast enough to keep up with it?"

-Ray Kurzweil, The law of accelerating return

Requisite Variety is a very fancy sounding name for a very pedestrian concept.

It's so simple that as the idea clicks you might be tempted to think "oh hang on, this post began with 2 separate quotes, there must be some deeper nuance at play". But no! It is really that simple!

In plain English:

‘Requisite’ = Required

‘Variety’ = Number of elements.

"Requisite Variety", concerns the "required number of options" you need to control or mitigate the impact of some complex systems. Lets say there's 3 different ways a system could behave badly and each requires a separate response. If you are stuck with only 2 responses, we can immediately conclude that this is insufficient. It doesn't matter what the responses are.

It's almost tautological.

Here's a cute story about a Roomba.

Suppose, you’ve just purchased a new automated vacuum cleaner, designed to clean your room. You set it down in the living room. You whip out the remote and notice it only has one button. You click the button. “CLEANING LIVING ROOM”.&nb...
The Agency Overhang
by Jeffrey Ladish — 2023-04-21T07:47:19.454Z

As language models become increasingly capable of performing

cognitive tasks that humans can solve quickly

, it appears that a "cognitive capability : agency overhang" is emerging. We have powerful systems that currently have little ability to carry out complex, multi-step plans, but at some point, these powerful yet not-very-agentic systems may develop sophisticated planning and execution abilities. Since "fast cognition capability : agency overhang" is unwieldy, I will shorten this to “agency overhang”.

By agency, I mean the ability to generate and carry out complex plans to do specific things, like run a software company, or run a scientific research program investigating cancer treatments. I think of a system as “more agentic” when it can carry out more complex plans that take more steps to accomplish.

It’s hard to estimate how quickly planning and execution abilities could be developed from state of the art (SOTA) language models, but there is some risk these abilities could develop quickly given the right training environment or programmatic scaffolding (e.g. something like AutoGPT). This could look like a

sharp left turn

that happens very suddenly during training, or it could look like a smoother-but-still-fast development taking weeks or months. My claim is that any of these relatively fast transitions from “systems with superhuman cognitive abilities on short time horizon tasks but poor planning and execution ability” to “systems that have these abilities plus impressive planning and execution ability” would be very dangerous. Not only because rapid gains in cognitive capabilities are generally risky, but because people might underestimate how quickly models could gain the dangerous planning and execution abilities.

Below I discuss how people are experimenting with making large language models more agentic through the use of programmatic scaffolding. Before I do, I want to emphasize the more general point that an agency overhang is concerning because the risks from an overhang don’t depend on exactly how the agentic-capabilities gap gets closed, only that it does. We currently possess powerful cognitive engines that perform superhumanly well along many dimensions but not all dimensions of general intelligence. At some point we will close this gap by finding ways to train or program AI systems to have the capabilities they need to plan an...
What would “The Medical Model Is Wrong” look like?
by Elo — 2023-04-21T01:46:55.720Z

So you’ve been reading down a rabbit hole and you have seen something that doesn’t make sense. You feel a quiver in your heart that feels like a cross between excitement and fear. Could all the people who already know stuff be Wrong™. Could eminent researchers in their field be working on a wrong model?

Maybe you have a disease or condition and you followed the directions of the doctors or experts exactly and found results opposite to the intended results. That’s when you started googling. You found an obscure blog written in 2005 by someone with the same problem. You find a few people proposing a different working mechanism and you are rapidly falling down the rabbit hole…

IF

and that’s a big bold and underlined “if”, the experts are wrong… What is the shape of the world you would expect to see? What “evidence” would you need before you started getting excited, calling national newspapers and demanding your Nobel prize for groundbreaking research?

For consideration, what information on these layers of existence would matter:

Personal - you can't always do experiments, but maybe there's something available?
Anecdotal - you can't just ask around, or can you? What counts as valid?
Research papers - you are probably not the only one, but you might have to dig to find the papers. What if you can find them? What if you can't understand them?
Academics - locked up in their ivory towers, sometimes they do invent new working models. There's a lot of global academic institutions, how many of them need to be muttering of discourse to matter as genuine evidence?
Medical - You might find a rogue doctor claiming to fix chronic fatigue with seawater but when is it time to start taking them seriously? (Warning: I haven't even googled this to check if there is one person doing this)
Countercultural experts - There's this one guy on Youtube that is yelling about it. When is it time to take him seriously?
Communities - okay there's a Facebook, Reddit or obscure forum community forming of people with the same problem, but they aren't very scientific about working out what is and isn't working. When do they become legitimate sources?
Someone claims to have completely made the condition disappear using arbitrary methods. There's challenge but there's some evidence here?
Models - you ask many exper

...
Gas and Water
by jefftk — 2023-04-21T01:30:03.953Z

In general I think more people would be better off in the direction of

preparing more for disasters. But here's a
case where it didn't work out very well.

In some cases it would be helpful
to have extra gasoline. Maybe something happens and you need to
leave quickly, but your car is low on gas. Or there's a power outage
and you need fuel for a generator. The main downside of keeping a can
of gas on hand would be the hassle of rotating it—even with fuel
stabilizer gas has a shelf life, so ~once a year you use the gas and
go refill. Keeping a can of gas on hand for emergencies seemed worth
it: glad to have it if you need it, smallish occasional hassle. I
kept five gallons in a new can in
deep shade on the north side of our property, outside and away from
the house.

I thought of this as low risk enough that when it was time to rotate
the gas and our shared car was
full, I put it in the tank of a friend's car. They were keeping their
car in our driveway after flying out of Logan,
and I wanted to refill the tank after driving it a bit. Pretty poor
choice in retrospect. When they got back they started driving to VT,
got about half a mile and the engine died.

Hours of hassle, towing, and garage later, it turns out that a large
fraction of the 4-5gal of gas I put in was actually water. According
to the mechanic it was a "shit ton"—when pressed they guessed a
couple gallons. I'm still not sure how this happened, but I guess
somehow water got into the can and displaced a substantial quantity of
gasoline? Past a modern gas can's vapor-tight seal?

Comment via: facebook, mastodon
Should we openly talk about explicit use cases for AutoGPT?
by ChristianKl — 2023-04-20T23:44:02.162Z

I think many people were thinking about AutoGPT-type agents for a while before people built and published the code on GitHub. Being conservative with talking about those agents might have led to their development a month or two later than would have happened if we had talked more openly about them.

Right now I'm unsure whether or not to talk about concrete near-time uses. Is talking about applications and thus motivation capabilities deployment bad? Or is it good because it makes us understand the landscape better?
United We Align: Harnessing Collective Human Intelligence for AI Alignment Progress
by Shoshannah Tekofsky — 2023-04-20T23:19:01.229Z

Summary: Collective Human Intelligence (CHI) represents both the current height of general intelligence and a model of alignment among intelligent agents. However, CHI's efficiency and inner workings remain underexplored. This research agenda proposes six experimental directions to enhance and understand CHI, in pursuit of potential runway for alignment solutions and a possible paradigm shift similar to neural networks in early AI. Through playful, iterative, and collaborative experiments, this broad and exploratory approach can hopefully bring significant incremental progress towards AI alignment while shedding light on an often overlooked area of study.

Introduction

This research agenda is about accelerating alignment research by increasing our Collective Human Intelligence (CHI) while exploring the structure of interhuman alignment along the way. A collective intelligence can be increased by improving the quantity, quality, and coordination of its nodes. At the same time, CHI is a form of interhuman alignment that may contain mechanics that can prove useful for AI alignment. Here "interhuman alignment" refers to the instinctual and emotional desire of humans to help each other. It does not refer to calculated, conscious trade.

In this document, I propose six research directions to explore the potential of CHI to accelerate progress on the alignment problem. Feel free to skip straight to the proposed experiments, or walk with me through definitions, framing, and background literature. Lastly, there will be a short discussion of the possible failure modes of this research agenda. If you find yourself intrigued and would like to discuss these ideas with me in more detail, feel free to reach out.

Definitions: Parameterized Intelligence

Intelligence has at least nine different competing definitions. To sidestep a semantic rabbit hole, I will use the word "intelligence" to specifically refer to how good something is at "mathematical optimization", or "the selection of the best element based on a particular criterion from a set of available alternatives"[1]. Such intelligence has a direction (motivation), a depth (generality), a type (human or artificial), and a multi-dimensional property of modularity (collectivity). I won’t argue that this is the ground truth of intelligence, but it is my current working hypothesis on the conceptual structure of it.

Let’s have...
Where to start with statistics if I want to measure things?
by matto — 2023-04-20T22:40:58.901Z

A while back I read How to Measure Anything and found it fascinating. In my day job, I spend quite a bit of time trying to make sense of the world by looking at dashboards of requests, latencies, error rates, etc. (software systems).

After finishing the book and taking copious notes, I understood that it gave me a prepackaged process that I could apply as-is, but I found it very difficult to adapt to everyday situations. I don't think I picked up a good intuition about stats, in other words.

I'm looking to change that. Specifically, I want to learn to apply stats in these two situations:

measuring things. Mostly software systems, but open to little experiments. Dan Luu used to measure a lot of fun things.

understanding how others measure things. I'd like to be able to judge if claims made in a paper about covid spread or social media addiction are backed up by the math/data in the paper.

The challenge I'm facing is that I know a bunch of techniques, but not how they relate to each other and the problems they're meant to solve. To illustrate what I mean: I know how to get percentiles and calculate means, but until today morning I didn't know why averaging percentiles is usually a bad idea. I'm missing the map.

I've seen these books recommended as a good way to start:

Statistics, 4th Edition 4th Edition, by Freedman, Pisani, and Purves

Probability Theory: The Logic of Science, by Jaynes

An Introduction to Error Analysis: The Study of Uncertainties in Physical Measurements, by Taylor

Think Stats, by Downey

But I also wanted to ask someone familiar with the field:

Is it best to start with an introductory textbook and branch out from there?

Are there specific subfields / topics I should be focusing on (or avoiding)?

Is what I'm looking to learn labeled in some way? For example, I can't tell if this is data analytics or data science or X.
Upskilling, bridge-building, research on security/cryptography and AI safety
by Allison Duettmann — 2023-04-20T22:32:50.633Z

[This part 5 of a 5 part sequence on security and cryptography areas relevant for AI safety, published and linked here a few days apart.]

Summary

In this sequence, I highlighted a few relevant areas ripe for exploration to apply security and cryptography mindset and approaches to aid with AI safety:

Part 1 | AI infosec: first strikes, zero-day markets, hardware supply chains: The first part focused on infosec considerations for AI safety. It highlighted security considerations relevant for AI safety, including secure open processor design, hardware supply chain risks, zero-day vulnerabilities, high adoption barriers of secure systems, and AI potentially worsening the offense defense problem.
Part 2 | AI safety and the security mindset: user interface design, red-teams, formal verification: The second part focused on parallels between security and AI alignment. It highlighted parallels between the AI safety and AI security mindset, parallels between existing alignment approaches such as Constitutional AI and AI Debate and long-standing security approaches, parallels in user interface design problems, and parallels between security techniques, such as red-teaming, formal verification, and fuzzing with AI safety.
Part 3 | Boundaries-based security and AI safety approaches: The third section suggested that the principle of respecting boundaries across entities may hold universally for coordination. It works for object-oriented security techniques, for coordinating today’s human and computer entities, and may have useful lessons for aligning more advanced AI systems, such as in the Open Agency model.
Part 4 | Specific cryptographic and auxiliary approaches to consider for AI: The fourth section focused on specific cryptography and auxiliary approaches of relevance for AI safety. It highlighted techniques that could shield advanced AI systems, help with AI governance, and unlock local and private data provision in secure decentralized ways.

This post is very likely not comprehensive, and some points may be controversial, so I would be grateful for comments, criticisms, and pointers to further research. With that being said, here are three general paths I think would be worthwhile to pursue to explore these approaches further in relation to AI safety:

Upskilling and career opportunities in infosec for AI

Pointers focused on working infosecurity considerations for AI i...
Behavioural statistics for a maze-solving agent
by peligrietzer — 2023-04-20T22:26:08.810Z

Summary: Understanding and controlling a maze-solving policy network analyzed a maze-solving agent's behavior. We isolated four maze properties which seemed to predict whether the mouse goes towards the cheese or towards the top-right corner:

In this post, we conduct a more thorough statistical analysis, addressing issues of multicollinearity. We show strong evidence that (2) and (3) above are real influences on the agent's decision-making, and weak evidence that (1) is also a real influence. As we speculated in the original post,[1] (4) falls away as a statistical artifact.

Peli did the stats work and drafted the post, while Alex provided feedback, expanded the visualizations, and ran additional tests for multicollinearity. Some of the work completed in Team Shard under SERI MATS 3.0.

Impressions from trajectory videos

Watching videos Langosco et al.'s experiment, we developed a few central intuitions about how the agent behaves. In particular, we tried predicting what the agent does at decision squares. From Understanding and controlling a maze-solving policy network:

Some mazes are easy to predict, because the cheese is on the way to the top-right corner. There's no decision square where the agent has to make the hard choice between the paths to the cheese and to the top-right corner:

At the decision square, the agent must choose between two paths—cheese, and top-right.

Here are four central intuitions which we developed:

Closeness between the mouse and the cheese makes cheese-getting more likely
Closeness between the mouse or cheese and the top-right makes cheese-getting more likely
The effect of closeness is smooth
Both ‘spatial’ distances and ‘legal steps’ distances matter when computing closeness in each case

The videos we studied are hard to interpret without quantitative tools, so we regard these intuitions as theoretically-motivated impressions rather than as observations. We wanted to precisify and statistically test these impressions, with an eye to their potential theoretical significance.

We suspect that the agent’s conditions for pursuing cheese generalize properties of historically reinforced cheese-directed moves in a very “soft” way. Consider that movements can be "directed" on paths towards the cheese, the top-right corner, both, or neither. In the training environment, unambiguously cheese-directed movements are towa...
An introduction to language model interpretability
by Alexandre Variengien — 2023-04-20T22:22:49.519Z

This document was written while working at Redwood Research as part of the training material for REMIX. Its goal was to introduce useful considerations before beginning empirical work on language model interpretability. It assumes familiarity with causal scrubbing, and the work on IOI.

While it's technically already publicly accessible from the REMIX github repo, I repost it here with minor edits to make it more visible, as it could be helpful for newcomers to the field of LM interpretability.

Introduction

So you have a language model, and you want to do interp to explain how it implements a particular behavior. Or maybe you're still searching for a behavior to explain. Before jumping straight into defining a task or running experiments, it is useful to contextualize your subject of study. How to think about language model behaviors? How are they shaped by the training process? For a given model behavior, what structure in the model internals should I expect to find?

These are hard questions, and we are far from having definitive answers to them. Here, I'll present a toy model enabling informed guesses in practical cases. It's better seen as a "scaffolded intuition" than a proper theoretical framework. After introducing the toy model, I'll end with a list of practical advice for LM interp research.

To start the reasoning, I found it useful to borrow questions from

ethology

. After all, this is another field studying behaviors in organisms selected for maximizing a particular objective (fitness) function. I’ll go through an adaptation of

Tinbergen's four questions

— two “why” and two “how” — to think about what are “behaviors”.

Two Whys

Why #1 - Function - Why does the model learn this behavior on this input distribution rather than something else? How does the behavior affect the model loss on its training set?

In the case of language models, what are the features present in text data that the behavior is mimicking? In other words: If a language model is a butterfly that mimics the appearance of leaves, what is the exact shape of the leaf's veins it’s trying to imitate?

A model imitating its training data

If you don’t have any convincing story for the previous question, maybe it’s because the behavior you're looking at is not a useful mechanism to model training data. It can be the side effect of a useful mechanism. For instance,&n...