trentmkelly/LessWrong-43k huggingface dataset:
The Open Agency Model
by Eric Drexler — 2023-02-22T10:35:12.316Z

Notes on AI for complex, consequential problems

Eric Drexler
Centre for the Governance of AI
University of Oxford

Introduction

This document argues for “open agencies” — not opaque, unitary agents — as the appropriate model for applying future AI capabilities to consequential tasks that call for combining human guidance with delegation of planning and implementation to AI systems. This prospect reframes and can help to tame a wide range of classic AI safety challenges, leveraging alignment techniques in a relatively fault-tolerant context.

Rethinking safe AI and its applications

AI safety research is too varied to summarize, yet broad patterns are obvious. A long-established reference-problem centers on prospects for rational superintelligent agents that pursue narrow goals with potentially catastrophic outcomes. This frame has been productive, but developments in deep learning call for updates that take account of the proliferation of narrow models (for driving, coding, robot control, image generation, game playing…) that are either non-agentic or act as agents in only a narrow sense, and that take account of the rise of more broadly capable foundation models and LLMs. These updates call for reframing questions of AI safety, and call for attention to how consequential tasks might be accomplished by organizing AI systems that usually do approximately what humans intend.

Two frames for high-level AI

The unitary-agent frame

From its beginnings in popular culture, discussion of the AI control problem has centered around a unitary agent model of high-level AI and potential AI risks. In this model, a potentially dominant agent both plans and acts to achieve its goals.

The unitary-agent model typically carries assumptions regarding goals, plans, actions, and control.

Goals: Internal to an agent, by default including power-seeking goals
Plans: Internal to an agent, possibly uninterpretable and in effect secret
Actions: Performed by the agent, possibly intended to overcome opposition
Control: Humans confront a powerful, potentially deceptive agent

The typical unitary-agent threat model contemplates the emergence of a dominant, catastrophically misaligned agent, and safety models implicitly or explicitly call for deploying a dominant agent (or an equivalent collective system) that is both aligned and powerful...
Intervening in the Residual Stream
by MadHatter — 2023-02-22T06:29:37.973Z

This post describes work done at SERI-MATS with mentorship from Neel Nanda. This work has benefitted from discussions with Neel Nanda, Stefan Heimersheim, Kajetan Janiak, Wes Gurnee, Marius Hobbhahn, Aryan Bhatt, Mykyta Baliesnyi, and Laura Vaughn.

Executive Summary

We demonstrate an interpretable direction in the residual stream of GPT2-small related to the IOI task first studied in Redwood Research's "Interpretability in the Wild" paper. This direction encodes whether a repeated name is equal to the previous name (ABB), or the one before that (BAB).
This direction is present in the intermediate layers of the transformer, and does not seem to have any relation to the logits of the current token. Most previous work on interpreting the residual stream has focused on using the unembedding directions (the "logit lens") to study the effect of the residual stream on the logits of the current token.
We show that intervening in the model using this direction results in a number of downstream changes consistent with our claims and also with Redwood Research's account of the IOI task. In particular, GPT2 usually completes ABB to ABBA and BAB to BABA; we cause these to flip to ABBB or BABB, respectively.

The black line is the decision boundary; our intervention flips the output of GPT2-small from ABBA/BABA to ABBB/BABB

As part of this, we develop a general technique (Logistic Regression with Quadratic Programming, or LRQP) for intervening in model activations that seems useful for other interpretability work. Whenever information can be reliably recovered via linear probe, LRQP allows us to intervene in the activations. (This intervention will only be successful in disrupting performance of a particular activity if we have correctly identified where the relevant information is being passed through the network.)

The Indirect Object Identification task

We study the IOI task, which was first studied in this paper by Redwood Research. To briefly recap, the IOI task is to complete a prompt like the following:

Yesterday, John and Mary went to the store. John bought a drink for _____

with the name "Mary". (Well, actually " Mary", due to tokenization quirks.)

This task is reliably completable by GPT2-small, a 12-layer, 12-heads-per-layer transformer with hidden size 768.

In this post, we actually study a slightly more messy version of this task. The messier version of the task is harder ...
What do language models know about fictional characters?
by skybrian — 2023-02-22T05:58:43.130Z

I will speculate a bit about what language models might be doing. Hopefully it will help us come up with better research questions (for those of you who can do research) and things to try (for those of us informally testing the chatbots), to see how these language models work and break.

I haven't done any testing based on these ideas yet, and there are many papers I haven't read, so I'd be very interested in hearing about what other people have found.

Let's say you're trying to predict the next word of a document, as a language model does. What information might be helpful? One useful chunk of information is who the author is, because people have different opinions and writing styles.

Furthermore, documents can contain a mix of authors. One author can quote another, and quotes can nest. A document might be a play or screenplay, or a transcript of an interview. Some authors don't actually exist; the words of fictional characters often appear in documents and it will be useful to keep track of them too. I'll generically refer to an author or character as a "speaker."

Speaker Identification

Within a document

When predicting the next word of a chat transcript, you will want to identify previous sections corresponding to the current speaker, and imitate those. One could imagine highlighting the text belonging to each speaker as a different color. There are various textual markers indicating where one speaker stops and another starts. Paying attention to delimiters (such as quotation marks) is important, along with all the other things authors might do to distinguish one speaker from another. Sometimes, a narrator indicates who is speaking, or the dialog is prefixed by the name of the speaker, such as in a play or interview.

Getting speaker changes wrong results in disastrous errors, like what seems to have happened with Bing's chatbot. So there's a pretty strong incentive in training to get it right. We don't know how it went wrong at Bing, but we can assume language models have mostly learned how to do this, sometimes making mistakes. Screwing up delimiter parsing often results in security holes, so it would be useful to know exactly how language models do it.

Between documents

At this point we've distinguished between speakers, but we don't know who they are. You could imagine the transcript being labelled with "speaker 1", "speaker 2", and so on.

It would be help...
Power-Seeking = Minimising free energy
by Jonas Hallgren — 2023-02-22T04:28:44.075Z

Epistemic Status: Pretty certain there are better ways to describe this but I believe that the underlying intuition holds and that it might be an exciting formalisation of power.

Thank you to Viktor Rehnberg and Roman Levantov for some great discussions leading up to this and to Viktor, Arun Jose and Esben Kran for giving feedback :)

Why does this matter?

If we can formalise power-seeking in terms of free energy, then we get an information entropy-based way of describing power-seeking. From this, we can define power gradients with respect to other variables. We can then assign scores of power-seeking to different behaviours in neural networks.

The argument

A condensed form of the argument.

The higher uncertainty in a system, the higher value you get from optionality or power. In a system with no uncertainty, a Markov Decision Process, power-seeking doesn’t arise. In any system, there are variables that predict the future of that system. We now define variables that the agent can control as the ones that agents have full predictive power over. Environmental variables are then variables that we can’t control.

Suppose we don’t have any environmental variables. In that case, we have no uncertainty as we can always determine what any future state looks like, and we’re, therefore, at maximum power (since we’re in a fully observable MDP). Increasing your power over a system is the same as giving yourself more predictive power over how that system evolves. Increasing your predictive power is the same thing as minimising external predictive power, or in other words, increasing your power can be described as minimising environmental variation-free energy (EVFE).

Definitions:

Agent:

An agent is determined as a system that has close to full predictive power over a set of variables. Alice is, therefore, an agent to her hand as she has close to full predictive power over her hand. (Agency is then to what degree an agent has predictive power over something)

Why chess algorithms with random evaluations want “

mobility

”.

This is the example we will use for the rest of the post.

In the most upvoted comment in Seeking Power is Convergently Instrumental in a Broad Class of Environments, dxu mentions how chess algorithms with random evaluations still perform better with deeper search than shallower search (which Viktor Rehnberg kindly dug up the original...
The shallow reality of 'deep learning theory'
by Jesse Hoogland — 2023-02-22T04:16:11.216Z
Link: https://www.jessehoogland.com/article/0-the-shallow-reality-of-deep-learning-theory

Produced under the mentorship of Evan Hubinger as part of the SERI ML Alignment Theory Scholars Program - Winter 2022 Cohort

Most results under the umbrella of "deep learning theory" are not actually deep, about learning, or even theories.

This is because classical learning theory makes the wrong assumptions, takes the wrong limits, uses the wrong metrics, and aims for the wrong objectives. Learning theorists are stuck in a rut of one-upmanship, vying for vacuous bounds that don't say anything about any systems of actual interest.

Yudkowsky tweeting about statistical learning theorists.
(Okay, not really.)

In particular, I'll argue throughout this sequence that:

Empirical risk minimization is the wrong framework, and risk is a weak foundation.
In approximation theory, the universal approximation results are too general (they do not constrain efficiency) while the "depth separation" results meant to demonstrate the role of depth are too specific (they involve constructing contrived, unphysical target functions).
Generalization theory has only two tricks, and they're both limited:

Uniform convergence is the wrong approach, and model class complexities (VC dimension, Rademacher complexity, and covering numbers) are the wrong metric. Understanding deep learning requires looking at the microscopic structure within model classes.
Robustness to noise is an imperfect proxy for generalization, and techniques that rely on it (margin theory, sharpness/flatness, compression, PAC-Bayes, etc.) are oversold.

Optimization theory is a bit better, but training-time guarantees involve questionable assumptions, and the obsession with second-order optimization is delusional. Also, the NTK is bad. Get over it.
At a higher level, the obsession with deriving bounds for approximation/generalization/learning behavior is misguided. These bounds serve mainly as political benchmarks rather than a source of theoretical insight. More attention should go towards explaining empirically observed phenomena like double descent (which, to be fair, is starting to happen).

That said, there are new approaches that I'm more optimistic about. In particular, I think that singular learning theory (SLT) is the most likely path to lead to a "theory of deep learning" because it (1) has stronger theoretical foundations, (2) engages with the structure of individual models, and (3) gives us a princ...
Candyland is Terrible
by jefftk — 2023-02-22T01:50:03.375Z

Candyland seriously suffers from not being aimed at a coherent age:
either you're old enough that you could instead play a game where you
make choices, or you're young enough that the disappointment of
getting a card that—out of nowhere—sends you nearly back
to the beginning is crushing. Growing up we would know if we were
about to draw "Plumpy" because the card was horribly crumpled and had
been taped back together.

I do like that the 2021
version removed 38% of the spaces to bring it down from 134 to 83, and
that since the 2002
edition the rules have included a "special optional rule for
younger players" where picture cards only move you forward. On the
other hand, the only-forward rule should be the default (if no one's
young enough for it this is the wrong game) and 83 spaces is still a
bit long for this age.

For people who like to think about gaming, taking a board game
intended for adults and radically
simplifying it and then introducing elements as they progress is a
lot better: with how quickly kids learn almost any game will either
start off too hard for them or quickly become too easy.

Candyland does fill a niche for non-gamer adults playing very young
children, however, and I do think something like it should exist. I'd
like to see it shortened a little more and the "only forward" rule
there by default (or have that rule apply to anyone 10+).
A proof of inner Löb's theorem
by James Payor — 2023-02-21T21:11:41.183Z

This is a short post that offers a slightly different take on the standard proof of Löb's theorem. It offers nothing else of any value :)

We seek to prove the "inner" version, which we write as:
□P↔□(□P→P)

The proof uses quining to build a related sentence L, the "Löb sentence", which talks about its own source code. By construction L has the property:
□L↔□(□L→P)

Then, we can show that □L↔□P, i.e. they're equivalent! We do this by plugging □L into itself to get a twisty □P. We can then replace each □L with □P and prove Löb's theorem.

The proof

This proof uses the same rules of box manipulation as on the wiki page. We start by creating L using quining, i.e. taking a modal fixed point:

⊢L↔(□L→P) (exists as a modal fixed point)

Yep, this is skipping the details of the most interesting part, but alas I don't understand them well enough to do more than wave my hands and say "quining".

We then stick it inside the box to get our first property:

⊢□(L↔(□L→P)) (from (1) by necessitation)

⊢□L↔□(□L→P) (from (2) by box-distributivity in both directions)

We now want to show that □L↔□P. We can get the forward direction by feeding a copy of □L into itself:

⊢□L→(□□L→□P) (box-distributivity on (3))

⊢□L→□□L (internal necessitation)

⊢□L→□P (from (4) and (5))

The backward direction is equivalent to □P→□(□L→P), and is straightforward:

⊢P→(□L→P) (trivial)

⊢□P→□(□L→P) (necessitation and box-distributivity on (7))

Taking those together, we've shown □L and □P are equivalent.

⊢□L↔□P (from (6) and (8))

Now we'd like to finish by appealing to the following chain:
□P↔□L↔□(□L→P)↔□(□P→P)

We've proven all but the last part of the chain. Here are the steps that let us do the substitution:

⊢(□L→P)↔(□P→P) (since □L and □P are equivalent by (9))

⊢□((□L→P)↔(□P→P)) (from (10) by necessitation)

⊢□(□L→P)↔□(□P→P) (from (11) by box-distributivity in both directions)

And that's everything we need:

⊢□P↔□(□P→P) (from (3), (9), and (12))
Fighting For Our Lives - What Ordinary People Can Do
by TinkerBird — 2023-02-21T20:36:32.579Z

tl;dr: Get organized and start brainstorming

___________________________________________________________

If you're reading this, you are the resistance.

I've always wanted to say that. But seriously, if you're reading this, then you may be one of the few people who recognize the dangers we're facing, and therefore one of the only people who can help.

Nothing Matters Other Than This

No, really. Climate change, cancer and war don't matter anymore when AGI is about to arrive.

To oversimplify:

Aligned A.G.I. = heaven, at some point
Misaligned A.G.I. = we all die

Whether there's a 1% chance of doom or a 99% chance of doom, I don't think those scenarios should be treated that differently. Even a 1% chance of doom is far too much and calls for humanity's every effort to reduce it. And it's not 1%.

Even if you don't think that doom is likely, even if you're convinced that the right people are taking the safety issue seriously, even if you put a lot of stock in the idea of AI being helpful in solving alignment, you will either be dead in 20 years or death will be cured in 20 years. Are you sure that your day job still matters when you know that your time can help improve our chances?

And if you think that Yudkowsky is justified in his predications... then what are you doing just sitting there?

This is the only important thing we will ever do.

The Problem

The problem can be broken down into two parts:

Slowing down capability research
Speeding up alignment research

(Though granted, there may be some crossover between these two elements.)

The sub-problems:

It's very hard to get governments to do things
It's very hard to motivate corporations do anything other than make money
It's very hard to get people with power or status to support an unpopular cause
It's very hard to get AI capability researchers to sabotage or switch their professions
It's very hard to paint a believable picture of doom for people (ironically, all of those other end of the world cries have hurt us greatly with this)
Other things that I haven't thought of yet

The Solution - Raising Awareness

The main goal in one sentence is to scream about the problem as loud and convincingly as we can. We want the researchers to stop working on AI capability and to focus more on alignment.

We're in a positive feedback loop with this. The more awareness we raise, the more people we have to help raise awareness.

The first idea that comes ...
The Emotional Type of a Decision
by moridinamael — 2023-02-21T20:35:17.276Z

Our minds have built-in tools that we use intuitively to evaluate decisions as we go about our day-to-day lives. Sometimes, making a decision can feel particularly challenging on an emotional level. This can lead to procrastination in making the decision or just going with the default option.

When the part of the brain that generates emotional responses is damaged, a person can struggle with basic decisions like what to have for lunch or which shirt to wear. This illustrates how our feelings about our options play a significant role in our decision-making process.

Our emotions towards decisions can be broadly classified into different emotional types, which can help explain why a particular decision may be difficult, and give hints as to how to resolve that difficulty. If you can think of an emotional type that I haven't covered, please feel free to suggest it in the comments.

Emotional Types

Risky Option decision: The least favored and most favored outcomes are associated with the same option, meaning that a single option could yield either a very good or very bad outcome.
Least-bad choice decision: The favorable outcomes are all low-likelihood and/or the unfavorable outcomes are all high-likelihood, so you are in the emotional position of choosing the "least bad" option. Either choice has a low EV.
Uncertain decision: All of your probability estimates fall in the ~35%-65% range. You have a hard time choosing because you just don't feel strongly one way or the other about the likelihoods involved.
High-stakes decision: The decision is unusually consequential.

The following examples are drawn from the Station Seven setting that I use for exercises in Guild of the Rose workshops.

Least-Bad

On your break from repolarizing the tachyon coils, you want to grab a quick bite to eat. You really don't want to be late getting back to your work station, so it's important that you're able to eat and be back within half an hour. There are two eateries you enjoy, Garg's Noodles and GleekBurger. Garg's Noodles is your favorite, but it's further away, all the way on the other side of the disk. GleekBurger is alright, and closer.

This is a "least-bad" type decision because the EV of either option is low. Whether you choose GleekBurger or Garg's Noodles, you're most likely going to be either late or dissatisfied with your meal. This sort of decision can feel unpleasant and "ug

...
What is it like doing AI safety work?
by KatWoods — 2023-02-21T20:12:01.977Z

How do you know if you’ll like AI safety work? What’s the day-to-day work like? What are the best parts of the job? What are the worst?

To better answer these questions, we talked to ten AI safety researchers in a variety of organizations, roles, and subfields. If you’re interested in getting into AI safety research, we hope this helps you be better informed about what pursuing a career in the field might entail.

The first section is about what people do day-to-day and the second section describes each person’s favorite and least favorite aspects of the job.

Of note, the people we talked with are not a random sample of AI safety researchers, and it is also important to consider the effects of survivorship bias. However, we still think it's useful and informative to hear about their day-to-day lives and what they love and hate about their jobs.

Also, these interviews were done about a year ago, so may no longer represent what the researchers are currently doing.

Reminder that you can listen to LessWrong and EA Forum posts like this on your podcast player using the Nonlinear Library.

This post is part of a project I’ve been working on at Nonlinear. You can see the

first part of the project here

where I explain the different ways people got into the field.

What do people do all day?

John Wentworth

John describes a few different categories of days.

He sometimes spends a day writing a post; this usually takes about a day if all the ideas are developed already.
He might spend a day responding to comments on posts or talking to people about ideas. This can be a bit of a chore but is also necessary and useful.
He might spend his day doing theoretical work. For example, if he’s stuck on a particular problem, he can spend a day working with a notebook or on a whiteboard. This means going over ideas, trying out formulas and setups, and trying to make progress on a particular problem.

Over the past month he’s started working with David Lorell. David’s a more active version of the programmer's "rubber duck". As John’s thinking through the math on a whiteboard, he’ll explain to David what's going on. David will ask for clarifications, examples, how things tie into the bigger picture, why did/didn't X work, etc.

John estimates that this has increased his productivity at theoretical work by a factor somewhere between 2 and 5.

O

...