trentmkelly/LessWrong-43k huggingface dataset:
AI Risk Intro 2: Solving The Problem
by CallumMcDougall — 2022-09-22T13:55:30.690Z

This marks the second half of our overview of the AI alignment problem. In the first half, we outlined the case for misaligned AI as a significant risk to humanity, first by looking at past progress in machine learning and extrapolating to what the future could bring, and second by discussing the theoretical arguments which underpin many of these concerns. In this second half, we focus on possible solutions to the alignment problem that people are currently working on. We will paint a picture of the current field of technical AI alignment, explaining where the major organisations fit into the larger picture and what the theory of change behind their work is. Finally, we will conclude the sequence with a call to action, by discussing the case for working on AI alignment, and some suggestions on how you can get started.

Note - for people with more context about the field (e.g. have done AGISF) we expect Thomas Larsen's post to be a much better summary, and this post might be better if you are looking for something brief. Our intended audience is someone relatively unfamiliar with the AI safety field, and is looking for a taste of the kinds of problems which are studied in the field and the solution approaches taken. We also don't expect this sampling to be representative of the number of people working on each problem - again, see Thomas' post for something which accomplishes this.

Crossposted to personal blog.

Introduction: A Pre-Paradigmatic Field

Definition (pre-paradigmatic): a science at an early stage of development, before it has established a consensus about the true nature of the subject matter and how to approach it.

AI alignment is a strange field. Unlike other fields which study potential risks to the future of humanity (e.g. nuclear war or climate change), there is almost no precedent for the kinds of risks we care about. Additionally, because of the nature of the threat, failing to get alignment right on the first try might be fatal. As Paul Christiano (a well-known AI safety researcher) recently wrote:

Humanity usually solves technical problems by iterating and fixing failures; we often resolve tough methodological disagreements very slowly by seeing what actually works and having our failures thrown in our face. But it will probably be possible to build valuable AI products without solving alignment, and so reality won’t “force us” to solve

...
Understanding Infra-Bayesianism: A Beginner-Friendly Video Series
by Jack Parker — 2022-09-22T13:25:04.254Z

Click here to see the video series

This video series was produced as part of a project through the 2022 SERI Summer Research Fellowship (SRF) under the mentorship of Diffractor.

Epistemic effort: Before working on these videos, we spent ~400 collective hours working to understand infra-Bayesianism (IB) for ourselves. We built up our own understanding of IB primarily by working together on the original version of Infra-Exercises Part I and subsequently creating a polished version of the problem set in hopes of making it more user-friendly for others.

We then spent ~320 hours writing, shooting, and editing this video series. Part 5 through Part 8 of the video series were checked for accuracy by Vanessa Kosoy, but any mistakes that remain in any of the videos are fully our own.

Goals of this video series

IB appears to have quite a bit of promise. It seems plausible that IB itself or some better framework that builds on and eventually replaces IB could end up playing a significant role in solving the alignment problem (although, as with every proposal in alignment, there is significant disagreement about this). But the original sequence of posts on IB appears to be accessible only to those with a graduate-level understanding of math. Even those with a graduate-level understanding of math would likely be well-served by first getting a gentle overview of IB before plunging into the technical details.

When creating this video series, we had two audiences in mind. Some people just want to know what the heck infra-Bayesianism is at a high level and understand how it's supposed to help with alignment. We designed this video series to be a one-stop shop for accomplishing this goal. We hope that this will be the kind of video series where viewers won't ever have to pause a video and go do a search for some word or concept they didn't understand or that the video assumes knowledge of. To that end, the first four videos go over preliminary topics (which can definitely be skipped depending on how familiar the viewer already is with these topics). Here are the contents of the video series:

Intro to Bayesianism
Intro to Reinforcement Learning
Intro to AIXI and Decision Theory
Intro to Agent Foundations
Vanessa Kosoy's Alignment Research Agenda
Infra-Bayesianism
Infra-Bayesian Physicalism
Pre-DCA
A Conversation with John Wentworth
A Conversation with Diffractor
A Conversation with Vanessa Kos

...
Is the game design/art maxim more generalizable to criticism/praise itself?
by Noosphere89 — 2022-09-22T13:19:00.438Z

For those who don't know, the maxim of criticism that game designers/art people use in their responses to criticism is roughly the following:

"If you get a negative reaction on it's own, that's evidence that there's a problem."

"If customers give any reasons why or solutions to the problem, ignore that criticism."

Is this generally true of criticism against anything, and if so, why do you think this happens?
AI career
by ondragon — 2022-09-22T03:48:35.195Z

Hello, I am a Czech 25 yo sociology student who strongly believes in the importance of making AI safe. After falling in love with Kurzweil's visions and learning about Bostrom's fears I discovered the immense importance of this topic and that I would like to take part in making the future world an amazing place. I would like to continue reading, thinking and writing about this matter and hopefully make it also my job. I won't have a formal education in this field though. I think I have a passion for the future of humanity and AI and some talent for this kind of thinking.

Would you recommend becoming an AI activist? To educate public about AI dangers, to put pressure on politicians, run social media accounts, write books and make lectures?

Or should I try career in AI safety research?

What do you think about these two options, which do you think is more useful and possible?

Thanks for your answers.
ondragon
Mathematical Circuits in Neural Networks
by Sean Osier — 2022-09-22T03:48:21.154Z
Link: https://www.youtube.com/watch?v=jGQN0TVCtMo

(Also posted on the EA Forum)

This is one of my final projects for the Columbia EA Summer 2022 Project Based AI Safety Reading Group (special thanks to facilitators Rohan Subramini and Gabe Mukobi). If you're curious you can find my other project here.

Summary

In this project, I:

Derive by hand the optimal configurations (architecture and weights) of "vanilla" neural networks (multilayer perceptrons; ReLU activations) that implement basic mathematical functions (e.g. absolute value, minimum of two numbers, etc.)
Identify "features" and "circuits" of these networks that are reused repeatedly across networks modeling different mathematical functions
Verify these theoretical results empirically (in code)

What follows is a brief introduction to this work. For full details, please see:

The linked video (also embedded at the bottom of this post)
Or if you prefer to go at your own pace, the slides I walk through in that video

Motivation

Olah et al. make three claims about the fundamental interpretability of neural networks:

They demonstrate these claims in the context of image models:

Features / Circuits:

Universality:

This work demonstrates the same concepts apply in the space of neural networks modeling basic mathematical functions.

Results

Specifically, I show that the optimal network for calculating the minimum of two arbitrary numbers is fully constructed from smaller "features" and "circuits" used across even simpler mathematical functions. Along the way, I explore:

"Positiveness" and "Negativeness" Detectors
Identity Circuits (i.e. f(x) = x)
Negative Identity Circuits (i.e. f(x) = -x)
Subtraction Circuits (i.e. f(x1, x2) = x1 - x2)
"Greaterness" Detectors
And More

Minimum Network:

I also demonstrate that each of these theoretical results hold in practice. The code for these experiments can be found on the GitHub page for this project.

Full Details

For full details, please see the PDF presentation in the GitHub repo or watch the full video walkthrough:
LW Petrov Day 2022 (Monday, 9/26)
by Ruby — 2022-09-22T02:56:19.738Z

Next Monday is Petrov Day (September 26), an annually observed Rationalist/EA holiday inspired by the actions of Stanislav Petrov:

As a Lieutenant Colonel of the Soviet Army, Petrov manned the system built to detect whether the US government had fired nuclear weapons on Russia. On September 26th, 1983, the system reported five incoming missiles. Petrov’s job was to report this as an attack to his superiors, who would launch a retaliative nuclear response. But instead, contrary to the evidence the systems were giving him, he called it in as a false alarm.

It was subsequently determined that the false alarms were caused by a rare alignment of sunlight on high-altitude clouds and the satellites' Molniya orbits, an error later corrected by cross-referencing a geostationary satellite.

In explaining the factors leading to his decision, Petrov cited his belief and training that any U.S. first strike would be massive, so five missiles seemed an illogical start.

Petrov underwent intense questioning by his superiors about his actions. Initially, he was praised for his decision. Petrov himself stated he was initially praised by Votintsev and was promised a reward, but recalled that he was also reprimanded for improper filing of paperwork with the pretext that he had not described the incident in the military diary.

He received no reward. According to Petrov, this was because the incident and other bugs found in the missile detection system embarrassed his superiors and the influential scientists who were responsible for it, so that if he had been officially rewarded, they would have had to be punished. He was reassigned to a less sensitive post, took early retirement (although he emphasized that he was not "forced out" of the army, as is sometimes claimed by Western sources), and suffered a nervous breakdown.

For more information see 1983 Soviet nuclear false alarm incident

Each year, people find ways to commemorate Petrov Day, e.g. with this ceremony written by Jim Babcock or Raemon's Modes of Petrov Day.

On LessWrong, we find our own way to celebrate, generally involving a large red button that brings down the frontpage for the duration of Petrov Day.

Petrov Day on LessWrong in 2020

What does Petrov Day celebrate?

There isn't a canonical precise answer accepted by everyone. There's a cluster of virtues and actions that people find worthy of remembering with di...
Tuesday Family Dinner
by jefftk — 2022-09-22T02:40:02.305Z

While I like Boston for

lots of reasons,
one of the more personal ones is that my family is here. My dad and
sister (and my other sister with her family until recently) are in the
next town over, and my cousin's family is five towns NNW. For
something like seven years now we've been getting together almost
every Tuesday evening for family dinner.

My dad cooks, we eat together and talk. We all clean up after: the
adults have all lived in my dad's house at some point so we all know
where everything goes. The playroom is still well-stocked with toys
even though kids don't live there anymore, and toys are more fun when
you only see them once a week. It's very relaxed, and I'm really glad
I can keep close to my folks and see them so much.

I feel very lucky that I happened to grow up in an area which also
made sense for me to live as an adult, and I wish more people didn't
have to choose between proximity to family and the other things they
want in life.

Comment via: facebook
Toy Models of Superposition
by evhub — 2022-09-21T23:48:03.072Z
Link: https://transformer-circuits.pub/2022/toy_model/index.html

A new Anthropic interpretability paper—“Toy Models of Superpostion”—came out last week that I think is quite exciting and hasn't been discussed here yet.

Twitter thread from Anthropic

Twitter thread from Chris

Introduction:

It would be very convenient if the individual neurons of artificial neural networks corresponded to cleanly interpretable features of the input. For example, in an “ideal” ImageNet classifier, each neuron would fire only in the presence of a specific visual feature, such as the color red, a left-facing curve, or a dog snout. Empirically, in models we have studied, some of the neurons do cleanly map to features. But it isn't always the case that features correspond so cleanly to neurons, especially in large language models where it actually seems rare for neurons to correspond to clean features. This brings up many questions. Why is it that neurons sometimes align with features and sometimes don't? Why do some models and tasks have many of these clean neurons, while they're vanishingly rare in others?

In this paper, we use toy models — small ReLU networks trained on synthetic data with sparse input features — to investigate how and when models represent more features than they have dimensions. We call this phenomenon superposition. When features are sparse, superposition allows compression beyond what a linear model would do, at the cost of "interference" that requires nonlinear filtering.

Consider a toy model where we train an embedding of five features of varying importance in two dimensions, add a ReLU afterwards for filtering, and vary the sparsity of the features. With dense features, the model learns to represent an orthogonal basis of the most important two features (similar to what Principal Component Analysis might give us), and the other three features are not represented. But if we make the features sparse, this changes:

Not only can models store additional features in superposition by tolerating some interference, but we'll show that, at least in certain limited cases, models can perform computation while in superposition. (In particular, we'll show that models can put simple circuits computing the absolute value function in superposition.) This leads us to hypothesize that the neural networks we observe in practice are in some sense noisily simulating larger, highly sparse networks. In other words, it's possible that models we trai

...
How to Train Your AGI Dragon
by Eris Discordia — 2022-09-21T22:28:25.082Z

Originally mistitled Here Be AGI Dragons, this document has now been moved from the safety section to where it belongs in the "How To" section...

This paper is the first in a series of posts I intend to write about an instructional manifesto for anyone wanting to build an Artificial General Intelligence(AGI), what generality means, its dangers, the moment in AI history that we are in, and some proposals for AGI designs that lower our over all x-risk while hopefully getting us out of <waves at everything> this mess. It feels appropriate to begin with my observations of things we should avoid hanging our safety on if we don't want to see bad AGI outcomes consider to be foundational understandings in the quest to construct an AGI.

Rule 1 : You can't box it in

My definition for generality looks like

σ=σt−1+(Ut===P(V)) ? σ:σ+(UtVrand)

A generalist attacks U∞ randomly until the model σ gains predictability in its environment.

When a human baby gets hungry it flails its arms and legs and cries in the hopes that something it does will resolve its situation. An animal throws itself against the walls of its cage when nothing else works. A stressed outcast has a break down and attacks the world they no longer understand. Generality means: if you don't know how to deal with your burning needs, you throw rocks until something good happens and then try to remember what you did.

This means that unless you control for all u in U (the universe in all accessible dimensions), you can't ever depend on fully boxing in, detecting or controlling a sufficiently capable AGI. To such an intelligence, even you and your mind are in U. For an agent with causality model σ = P(V), any new v in V(any newly added dimension) might allow an agent to escape.

Rule 2 : No Actual Contact

Direct access by a generalist to a world containing its encoder means escape.

If a generalist has access to the world containing an encoder that it can manipulate then it will escape whatever constraints you've placed on it by manipulating the encoder since the encoder is in U. This means that even if you have a goal of putting out fires, it could redefine for itself what something being on fire means and still be "within" constraints.

DNA in all its billions of years of accumulated wisdom put the mammalian generalist inside a multidimensional box consisting of pain reducing emotional s...
An issue with MacAskill's Evidentialist's Wager
by Martín Soto — 2022-09-21T22:02:47.920Z

In The Evidentialist's Wager, MacAskill et al. argue as follows:

Suppose you are uncertain as to whether EDT or CDT is the correct decision theory, and face a Newcomb-like decision. If CDT were correct, your decision would only influence the outcomes of your present decision. If EDT were correct, it would provide evidence not only for the outcome of your present decision, but also for the outcome of many similar decisions by similar agents throughout the universe (either because they are exact copies of you, or they have very similar decision theories/computations to you, etc.). Thus, the stakes for the decision are way higher if EDT is true, and so you should act as if EDT were true (even if you have higher prior credence on CDT).

This argument of course relies on how many of these similar agents actually exist, and how similar they are. They use the term correlated agent to mean some agent similar enough to you so that your decision will acausally provide evidence about theirs. As possible counterexamples to their argument, they point out the existence of different agents:

Anti-correlated agents: Agents whose decision theory will drive them to take the decision opposite to yours.
Evil Twins: Agents positively correlated to you (with your same decision theory) but with drastically different utility functions (or in the extreme, the exact opposite utility function).[1]

Regarding anti-correlated agents, since our decision theories are actual good heuristics for going about the world and rationally obtaining our goals, it seems more likely for agents that exist (that is, have survived) to be positively correlated rather than anti-correlated.

But regarding Evil Twins, because of the Orthogonality Thesis, we might expect on average that there are as much agents positively correlated to us with ~our same utility function (Good Twins) as agents positively correlated to us with ~opposite utility function (Evil Twins).

That is, the universe selects for agents positively correlated to us (instead of anti-correlated), but doesn't select for agents with ~our utility function (instead of the ~opposite function).

So we should expect the acausal evidence from all these other agents to balance out, and we're back at EDT having stakes just as high as CDT: only our particular decision is affected.[2]

Possible objection

Of course, this refutation relies on our actually expecting that t...
Announcing AISIC 2022 - the AI Safety Israel Conference, October 19-20
by Davidmanheim — 2022-09-21T19:32:35.581Z

ALTER, with the University of Haifa and the Technion, is excited to announce the details about its upcoming conference introducing AI safety in Israel. We’re going to be hosting Stuart Russell, as well as several other AI safety researchers, to speak about the field. We're also hosting a number of Israeli academics, who will speak about their related research.

The goal of the conference is to build a community of interest around AI safety in Israel, let people know about the field, and following up on the conference, we hope to connect interested attendees with work going on internationally. (This is not intended to be a conference where novel AI safety work is presented, and the primary audience is people who would otherwise be working on AI capabilities research.)
If you are, or know anyone who is interested in AI safety in Israel, we would be happy for them to

register and attend

!
Nearcast-based "deployment problem" analysis
by HoldenKarnofsky — 2022-09-21T18:52:22.674Z

When thinking about how to make the best of the most important century, two “problems” loom large in my mind:

The AI alignment problem: how to build AI systems that perform as intended, and avoid a world run by misaligned AI.

The AI deployment problem (briefly discussed here): the question of how and when to (attempt to) build and deploy powerful AI systems, under conditions of uncertainty about how safe they will be and how close others are to deploying powerful AI of their own.

This piece is part of a series in which I discuss what both problems might look like under a nearcast: trying to answer key strategic questions about transformative AI, under the assumption that key events (e.g., the development of transformative AI) will happen in a world that is otherwise relatively similar to today's.

A previous piece discussed the alignment problem; this one discusses the deployment problem.

I’m using the scenario laid out in the previous post, in which a major AI company (“Magma,” following Ajeya’s terminology) has good reason to think that it can develop transformative AI very soon (within a year), using what Ajeya calls “human feedback on diverse tasks” (HFDT) - and has some time (more than 6 months, but less than 2 years1) to set up special measures to reduce the risks of misaligned AI before there’s much chance of someone else deploying transformative AI. I discuss what Magma would ideally do in this situation.

I’m also introducing another hypothetical actor in this scenario, “IAIA2”: an organization, which could range from a private nonprofit to a treaty-backed international agency, that tracks3 transformative AI projects and takes actions to censure or shut down dangerous ones, as well as doing other things where a central, neutral body (as opposed to an AI company) can be especially useful. (More on IAIA below.)

I’m going to discuss what Magma’s and IAIA’s major goals and priorities should be in the “nearcast” situation I’m contemplating; a future piece will go through what a few stylized success stories might look like. I’ll be bracketing discussion of the details of how Magma can reduce the risk that its own AI systems are misaligned (since I discussed that previously), and focusing instead on what Magma and IAIA should be looking to do before and after they achieve some level of confidence in Magma’s systems’ alignment.

I focus on Magma...