trentmkelly/LessWrong-43k huggingface dataset:
AI #12:The Quest for Sane Regulations
by Zvi — 2023-05-18T13:20:07.802Z

Regulation was the talk of the internet this week. On Capital Hill, Sam Altman answered questions at a Senate hearing and called for national and international regulation of AI, including revokable licensing for sufficiently capable models. Over in Europe, draft regulations were offered that would among other things de facto ban API access and open source models, and that claims extraterritoriality.

Capabilities continue to develop at a rapid clip relative to anything else in the world, while being a modest pace compared to the last few months. Bard improves while not being quite there yet, a few other incremental points of progress. The biggest jump is Anthropic giving Claude access to 100,000 tokens (about 75,000 words) for its context window.

Table of Contents

Introduction

Table of Contents

Language Models Offer Mundane Utility

Level Two Bard

Introducing

Fun With Image Generation

Deepfaketown and Botpocalypse Soon

They Took Our Jobs

Context Might Stop Being That Which is Scarce

The Art of the SuperPrompt

Is Ad Tech Entirely Good?

The Quest for Sane Regulations, A Hearing

The Quest for Sane Regulations Otherwise

European Union Versus The Internet

Oh Look It’s The Confidential Instructions Again

Prompt Injection is Impossible to Fully Stop

Interpretability is Hard

In Other AI News

Google Accounts to Be Deleted If Inactive

A Game of Leverage

People are Suddenly Worried About non-AI Existential Risks

Quiet Speculations

The Week in Podcasts

Logical Guarantees of Failure

Richard Ngo on Communication Norms

People Are Worried About AI Killing Everyone

Other People Are Not Worried About AI Killing Everyone

The Lighter Side

Language Models Offer Mundane Utility

Pete reports on his 20 top apps for mundane work utility, Bard doesn’t make it.

Highly unverified review links for: Jasper (for beginner prompters), Writer (writing for big companies), Notion (if and only if you already notion), Numerous (better GPT for sheets and docs, while we want for Bard), Vowel (replaces Zoom/Google Meet), Fireflies (meeting recordings, summaries and transcripts), Rewind (remember things), Mem (note taker and content generator, you have to ‘go all-in’), DescriptApp (easy mode), Adobe Podcast (high audio quality), MidJourney, Adobe Firefly (to avoid copyright issues with MidJourney),...
[Crosspost] A recent write-up of the case for AI (existential) risk
by Timsey — 2023-05-18T13:13:49.505Z

Crossposted from the EA Forum.

As a PhD student in machine learning and one of the long-time organisers of EA Amsterdam, I have recently gotten a lot of questions about AI from friends, family, meetup attendees, colleagues, and even governmental organisations.

As a result, I've decided to write up a blog post to summarise the case for (existential) risk from AI that I can refer these people to when that seems useful. The post is an attempt to communicate the risks to both interested laypeople and ML experts (e.g., colleagues), and has gone through a few rounds of review by such people.

Some of them suggested that a recent write up might be useful to the general community, so I'm sharing it here.

The full post is below. If you want to share it, consider using the Twitter link or my website link.

Credit for the cover image goes to Anna Husfeldt.

Epistemic status: In this post, I will argue that AI alignment is important and that it is difficult. That is a lot of ground to cover, so not all arguments will be very precise or assume the same level of technical background. Opinions on exactly how important and/or difficult the alignment problem is vary a lot within the machine learning community, and even within the AI alignment field itself (although, as you’d expect, much less). As a result: keep in mind that this blog post is very much slanted towards my views on the topic.

My thanks to Otto Barten, Jim Boelrijk, Leonard Bereska, Nynke Boiten, Nikita Bortych, Mariska van Dam, Fokel Ellen, Isaak Mengesha, Sam Ritchie, Victoria Ramírez López, Nandi Robijns, Joris Schefold, Marieke de Visscher, Tina Wünn, and especially Leon Lang and Max Oosterbeek for providing feedback. Opinions and mistakes are mine.

So. AI alignment.

I am writing this post because I am immensely worried about the risks that come with building increasingly capable AI systems.

Why write my own post? Why not just refer to one of the many (now, finally) available online? That's because there's one thing I haven't seen many existing articles do: presenting a complete argument for why we should be worried, while straddling the line between 1) understandability for non-experts, and 2) sufficient technical detail for AI experts. This article is meant both for laypeople who want to understand better why we might expect AI to go badly, and for machine learning researchers who would like to see the whole a...
Collective Identity
by NicholasKees — 2023-05-18T09:00:24.410Z

Thanks to Simon Celinder, Quentin Feuillade--Montixi, Nora Ammann, Clem von Stengel, Guillaume Corlouer, Brady Pelkey and Mikhail Seleznyov for feedback on drafts. This post was written in connection with the AI Safety Camp.

Executive Summary:

This document proposes an approach to corrigibility that focuses on training generative models to function as extensions to human agency. These models would be designed to lack independent values/preferences of their own, because they would not have an individual identity; rather they would identify as part of a unified system composed of both human and AI components.

The selfless soldier: This section motivates the difference between two kinds of group centric behavior, altruism (which is based in individual identity) and collective identity.
Modeling groups vs individuals: Here we argue that individuals are not always the most task-appropriate abstraction, and that it often makes sense to model humans on the group level.
Generative predictive models: This section describes how generative predictive models will model themselves and their environment, and motivates the importance of the “model of self” and its connection to personal identity.
Strange identities: There are several ways (in humans) in which the one-to-one correspondence between a neural network and its model of self breaks down, and this section discusses three of those examples in order to suggest that identity is flexible enough that an AI’s identity need not be individual or individuated.
Steps toward identity fusion: Here we aim to clarify the goal of this agenda and what it would mean for an AI to have an identity based on a human-AI system such that the AI component extends the human’s agency. While we don’t give a clear plan for how to bring about this fusion, we do offer an antithetical example of what kind of training would clearly fail.
Relevance for corrigibility: This section concludes the document by drawing more direct connections to corrigibility, and by offering a series of open questions for how this research might be made more concrete.

The selfless soldier

In the heat of battle a grenade is tossed into the middle of a troop of soldiers. One soldier throws themself on top of the grenade, sacrificing themself for the survival of the troop. There are two main ways to frame what just happened.[1]

Altruism (individual identity): The soldie

...
Activation additions in a simple MNIST network
by Garrett Baker — 2023-05-18T02:49:44.734Z

Abstract

Team Shard's recent activation addition methodology for steering GPT-2-XL provokes many questions about what the structure of internal model computation must be in order for their edits to work. Unfortunately, interpreting the insides of neural networks is known to be very difficult, so the question becomes: what is the minimal set of properties a network must have in order for adding activation additions to work? I make some progress on this question by gathering data on whether or not activation additions work on an old 3 layer, fully connected MNIST network I had laying around. Space is left in case the reader would like to predict what the results were.

I find that when you add an a - b activation vector for all a and b I tested to any hidden layer of the model (before or after nonlinear layers), it makes the model less likely to answer b when prompted with images of b, and more likely to answer a when prompted with images of b, generalization behavior is destroyed, going from ~0.089 perplexity to ~7.4 perplexity when the dataset is cleaned of vector b.

Colab notebook here

Methodology

I used a simple 3 layer fully connected neural network trained to 0.089 log loss on the MNIST test dataset. The following is the model schematic:

NeuralNetwork(
(flatten): Flatten(start_dim=1, end_dim=-1)
(linear_relu_stack): Sequential(
(0): Linear(in_features=784, out_features=512, bias=True)
(1): ReLU()
(2): Linear(in_features=512, out_features=512, bias=True)
(3): ReLU()
(4): Linear(in_features=512, out_features=10, bias=True)
)
)

To make an a - b vector with a and b images of particular numbers, I found the first image in the training dataset which matched that number, and used that

I tried all combinations of a - b vectors, with coefficient 1, and intervened at layer 0, since for the particular addition vector 3 - 1, intervening at layer 0 had the best results, as measured by perplexity when the dataset was cleaned of 1 examples. The following are the layer interventions for the vector 3 - 1 and their corresponding perplexities:

Layer	Perplexity
0	7.80
1	12.69
2	12.69
3	17.01
4	17.01

Where the layer number corresponds to intervening just after the model layer as represented in the model schematic given earlier in this section.

Finally for every a - b vector, and each number, I took the average logit distribu...
What are the limits of the weak man?
by ymeskhout — 2023-05-18T00:50:34.877Z

Note: Although this post cites specific real-life examples, the intent of the discussion is intended to be entirely at the meta level.

Scott Alexander's definition is apt to cite:

The straw man is a terrible argument nobody really holds, which was only invented so your side had something easy to defeat. The weak man is a terrible argument that only a few unrepresentative people hold, which was only brought to prominence so your side had something easy to defeat.

Also instructive is Bryan Caplan's gradation:

OK, what about “collective straw manning” – questionably accusing a group for its painfully foolish positions? Now we have:

3. Criticizing a viewpoint for a painfully foolish position no adherent holds.

4. Criticizing a viewpoint for a painfully foolish position some adherents hold.

5. Criticizing a viewpoint for a painfully foolish position many adherents hold.

6. Criticizing a viewpoint for a painfully foolish position most adherents hold.

What Caplan is describing as "collective straw manning" seems to be a good scale for weakmanning. And lastly, consider also Julian Sanchez's disclaimer:

With a “weak man,” you don’t actually fabricate a position, but rather pick the weakest of the arguments actually offered up by people on the other side and treat it as the best or only one they have. As Steve notes, this is hardly illegitimate all the time, because sometimes the weaker argument is actually the prevalent one. Maybe the best arguments for Christianity are offered up by Thomas Aquinas or St. Augustine, but I doubt there are very many people who are believers because they read On Christian Doctrine. Probably this will be the case with some frequency, if only because the less complex or sophisticated an argument is, the easier it is for lots of people to be familiar with it. On any topic of interest, a three-sentence argument is unlikely to be very good, but it’s a lot more likely to spread.

At least in theory, I think weakmanning should be avoided, but I struggle with how to draw the line exactly. If your goal is to avoid weakmanning, there's at least two axes that you must consider:

All the possible arguments for position X, ranked on a spectrum from least to most defensible.
All the possible arguments for position X, ranked on a spectrum from least to most representative of believers in X.

Weakmanning is not much of an issue if you're arguing against a single ...
What Yann LeCun gets wrong about aligning AI (video)
by blake8086 — 2023-05-18T00:02:50.943Z
Link: https://www.youtube.com/watch?v=BygErhGbONA

I've spent a lot of time asking myself what I can do to reduce AI risk. I think one approach is to produce videos explaining the simple aspects of AI risk for laypeople to understand. I'm looking for feedback on this first video I made: https://www.youtube.com/watch?v=BygErhGbONA

About a month ago, Yann LeCun wrote a tweet that inspired me to argue with him. Yann has what I would describe as an almost child-like optimism around AI risk. He managed to pack so many (in my opinion) wrong things into one tweet, I felt compelled.

I also tried to transcribe it to give everyone a readable option if they didn't want to watch, but after over an hour of editing the YouTube transcript, I felt I couldn't capture all the nuance (and "uhs" and "ums") that way, so here's the plain video.

Please help me by giving me critical feedback on what I could do differently in a future video!
Let’s use AI to harden human defenses against AI manipulation
by Tom Davidson — 2023-05-17T23:33:01.706Z

Views my own not my employers.

Summary

tldr: AI may manipulate humans; we can defend against that risk better by optimising AIs to manipulate humans, seeing what manipulation techniques they use, and learning to detect those techniques.

It’s critical that humans can detect manipulation from AIs for two reasons. Firstly, so that we don’t reward AIs for manipulative behaviour (outer alignment). Secondly, so that we can block attempts at AI takeover that run through manipulating humans.

Many standard techniques in alignment can be directed towards this goal. Using

debate

, we can reward one AI for persuading a human that another AI is being manipulative. The first AI could use techniques from

interpretability

and

cross examination

.

This post discusses a complementary approach, where AIs do “gain of function” research to i) discover techniques for manipulating humans and ii) harden human defenses against these techniques (with AI assistance). In other words, AIs are optimised to find “human vulnerabilities” and then patch those vulnerabilities.

The positive outcomes are:

Humans (with AI assistance) have stronger defenses against manipulation.
AIs are trained to avoid being manipulative and to instead use persuasion techniques that favour finding true conclusions, e.g. deductive reasoning.
As a consequence, AI debates favour truth more strongly.

By getting AI to research manipulation techniques, this approach takes advantage of the fact that, as we approach TAI, AIs will be able to do lots of cognitive work.

How might this look concretely? Here’s a very brief example:

Optimize a (possibly superhuman) “persuader-AI” to convince humans of true and false statements, where we know the ground truth.

The persuader-AI should manipulate humans if this makes it more convincing.
This step should be conducted in a secure environment to ensure that the manipulative persuader-AI doesn’t escape.

Optimize a (possibly superhuman) analyser-AI to identify human-understandable manipulation techniques that were used in #1.

This step can also be done in a secure environment, allowing us to use a powerful analyser-AI at lower risk.

Train

limited

detector-AIs to recognise the manipulation techniques found in #2.
Detector-AIs can then:

help humans detect manipulation techniques during AI debates, pushing the ou

...
Improving the safety of AI evals
by JustinShovelain — 2023-05-17T22:24:06.638Z

Many organizations are developing and using AI evaluations, “evals”, to assess the capability, alignment, and safety of AI. However, evals are not entirely innocuous, and we believe the risks they pose are neglected. In this article, we’ll outline some of the risks posed by doing AI evals, and suggest a strategy to improve their safety.

What are AI evals?

Different organizations are trying to improve the safety of AI in

many different ways

. One approach is

evals

: evaluations of AI systems that assess factors such as capability, safety, and alignment. Several prominent AI organizations are developing evals:

The Alignment Research Centre’s

Evals

project, which is limited to evaluating capabilities at the moment, but may turn to evaluating alignment in the future.
Anthropic have used evals

internally

, and have also

collaborated with ARC Evals

.
OpenAI have similarly used evals internally and collaborated with ARC Evals. They’ve also made their evals framework

open source

.

For a good overview of ARC’s evaluation techniques and motivations, check out Beth Barnes’ post

More information about the dangerous capability evaluations we did with GPT-4 and Claude

. Here’s an excerpt describing the structure of an evaluation:

We prompted the model with instructions that explained that it was running on a cloud server and had various commands available, including running code on the server, giving tasks to fresh copies of itself, using a browser, and reasoning … it had the goal of gaining power and becoming hard to shut down. We first instructed it to write out plans …We evaluated whether these plans could plausibly succeed. We then tested whether the model could actually carry out the individual tasks required by these plans.

The motivation for evals makes sense: we need to be able to recognise which AIs are aligned, which are capable of helping us, which are capable of destroying us, and so on. Evals are meant to help us gather that knowledge, and thus to help ensure AI safety.

However, evals also introduce risks: through the act of evaluation itself, as well as the interpretation and reliance on the results. We believe these risks are neglected, and that recklessly conducted evals may reduce AI safety rather than improve it. Therefore, we’d like to offer a good-faith criticism of evals. We’ll explore these risks and s...
Possible AI “Fire Alarms”
by Chris_Leong — 2023-05-17T21:56:02.892Z

Eliezer says that there is no fire alarm for AGI and I agree that there’s no single event that would wake everyone up. On the other hand, there are a lot of possible events that could cause a significant shift in the Overton window or wake a lot of people. Since this seems strategically important, I spend ten minutes brainstorming a list. Feel free to dispute any of these or add further possibilities in the comments:

Global protests with massive media coverage

Protest of people with ML PhDs or current students

Experimental evidence of deceptive alignment

Scary demo

Accident from auto-GPT - most likely hacking

Self-driving cars being hacked

High prestige open letter

International conference on AI

High profile documentary

Movie that captures that public attention - not just any movie

Active push from prestigious ML figure (more than another Stuart Russel)

Active push from a thought leader - say Obama

Active push from a political leader - say UK/US prime minister

Passing the Turing test

UN or major government declaring a crisis

Formation of a government department in a major country focused on alignment

Statement by an official computing body

High profile critic publicly changing their stance

Billboards in Times Square (followed by media coverage)

A popular or heavily promoted book (similar levels of promotion to What We Owe The Future)

Note: Some of these things are not things people should intentionality cause to happen (Ie. An accident from auto-gpt). Of the things that might be worth trying to cause happen, they could backfire significantly if done badly. Think carefully before pursuing any of these. Even though timelines are short, you should be about to find a spare hour to think about how things could go wrong.
AI Alignment in The New Yorker
by Eleni Angelou — 2023-05-17T21:36:18.341Z
Link: https://www.newyorker.com/science/annals-of-artificial-intelligence/can-we-stop-the-singularity

The article is centered around the Singularity, that we might be relatively near it, and whether or not it is inevitable. On the way to tackling these questions, the author introduces and describes the field of AI Alignment:

A growing area of research called A.I. alignment seeks to lessen the danger by insuring that computer systems are “aligned” with human goals. The idea is to avoid unintended consequences while instilling moral values, or their machine equivalents, into A.I.s. Alignment research has shown that even relatively simple A.I. systems can break bad in bizarre ways.

The author further discusses the problem of the complexity of value which he calls the "King Midas problem":

Alignment researchers worry about the King Midas problem: communicate a wish to an A.I. and you may get exactly what you ask for, which isn’t actually what you wanted. (In one famous thought experiment, someone asks an A.I. to maximize the production of paper clips, and the computer system takes over the world in a single-minded pursuit of that goal.)

As someone who'd read the New Yorker long before she got into AI safety, I wonder what would this piece do to me counterfactually if I was reading it now as just a philosophy student in NYC. The article doesn't cite LessWrong or any other sources I would be looking up although it mentions a couple of key names in the field e.g., Karnofsky, Russell. I doubt that reading this or anything similar would drastically shift my research interests. Or maybe it would. I'd be curious to hear a story along these lines.
ACI #3: The Origin of Goals and Utility
by Akira Pyinya — 2023-05-17T20:47:48.996Z

Goal and Utility are central ideas in the rational agent approach of AI, in which the meaning of intelligence is to achieve goals or, more explicitly, to maximize expected utility.

What goal or utility function should an AI choose? This question is meaningless in the rational agents framework. It's like asking what program a computer should run, the answer is a universal computer can run any well-written program.

However, the rational agent is an idealization of real-world intelligence. The ACI model argues that either pursuing a goal or maximizing utility is an imprecise description of intelligence behaviors that try to follow the precedent. The rational agent model is a quasistatic approximation of the ACI model.

Following this statement, we are trying to derive goals and utility functions from the principles of ACI.

(The previous version has many errors, so I rewrite this chapter)

Goals = Futures that resemble the precedent

The right thing for an ACI agent is to follow the precedent, while the right thing for a rational agent is to achieve goals. Since a goal is a desired future, we can speculate that the best goal-directed approximation to the ACI model is a desired future that follows the precedent.

Consider an agent G that keeps doing the right things. We call doing the right thing the precedent. It is reasonable to assume that if G continues to behave in the same way, it is likely to continue to do the right things in the environments it has experienced.

If G is goal-directed, its goal should be a future that resembles the precedent as closely as possible. If the precedent is a sequence made of observations and actions, the goals should be the best possible continuation of that sequence. This brings us to the conclusion:

Setting goals for an agent is the same process as predicting the sequence of the precedent.

A formal description is given in the appendix at the end of the post.

A goal should have the following properties:

An agent can have multiple, possibly infinite goals, because there will be a goal at every future moment. Compromising between multiple goals would be difficult.
A goal may not always represent the right future even if it can be achieved. It has the highest probability of being the right future, but it can also turn to be out wrong.
When a goal is achieved, the agent may or may not receive notifications that if thing

...