trentmkelly/LessWrong-43k huggingface dataset:
Elicitation for Modeling Transformative AI Risks
by Davidmanheim — 2021-12-16T15:24:04.926Z

This post is part 8 in our sequence on Modeling Transformative AI Risk. We are building a model to understand debates around existential risks from advanced AI. The model is made with Analytica software, and consists of nodes (representing key hypotheses and cruxes) and edges (representing the relationships between these cruxes), with final output corresponding to the likelihood of various potential failure scenarios. You can read more about the motivation for our project and how the model works in the Introduction post. Unlike other posts in the sequence, this discusses the related but distinct work around Elicitation.

We are interested in feedback on this post, but to a greater extent than the other posts, we are interested in discussing what might be useful, and how to proceed with this. We would also welcome discussion from people working independently on elicitations, as we have discussed this extensively with other groups, many of whom are doing related work.

As discussed in previous posts in this series, the model we have built is a tentative one, and requires expert feedback and input. The traditional academic method for getting such feedback and input is usually referred to as elicitation, and an

extensive field of academic work discusses how this can best be done

. (As simple examples, this might include eliciting an estimated cost, and probability distribution, or a rank ordering of which outcomes from a project are most important.)

Elicitation of expert views is particularly critical in AI safety for both understanding debates between experts and representing the associated probabilities. At the same time, many elicitation and forecasting projects have the advantage of unambiguous and concrete questions with answers that will be observed in the near term, or ask for preferences about outcomes which are well understood. Because these advantages are mostly absent for AI safety questions, the focus in this project is on understanding debates (instead of attempting to settle debates that are already understood, or are not resolvable even in theory). This means that there is no intent to elicit a “correct” answer to questions which may be based on debated or disputed assumptions. For this reason, we have taken an approach designed to start with better understanding experts’ views of the domain overall, rather than focus on the outcomes directly. Thi...
An Open Letter to the Monastic Academy and community members
by HS2021 — 2021-12-16T09:04:11.352Z

I am sharing a link here so that this account is locatable to members of this community from which the Monastic Academy actively recruits. This account describes my personal experiences, observations of organizational practices and misconduct, and knowledge of other accounts of ongoing negligence, emotional, psychological, and spiritual abuse which have been present over the past 10 years. This account is not written in the style or language commonly used in this community. If choose to comment please exercise mindfulness regarding the highly sensitive and complex nature of this information. My hope is that this information and perspective will be helpful to a least one person in this community in making better informed decisions about how to engage with the risks of training and/or associating with the Monastic Academy.

To read more you can visit this link: https://medium.com/@shekinahalegra/dear-monastic-academy-and-community-members-49c25d9646a4
Five Missing Moods
by mike_hawke — 2021-12-16T01:25:09.409Z

Read this Bryan Caplan post, it's short and direct. His thesis is:

You can learn a lot by comparing the mood reasonable proponents would hold to the mood actual proponents do hold.

I could quibble, but at the very least this heuristic has caused me to have new thoughts. Here are some of them:

~~~

Missing the mood: Effective Altruism should avoid getting embroiled in standard conflicts over politics & public policy. Political battles have especially low tractability, and often have reputational costs that are hard to predict.

Not missing the mood: Moloch yet controls the national/state/local budgets, the campaigns, the military agenda, the news landscape, the taxes and subsidies, the zoning laws, the immigration laws, the financial regulations, and so on. Tragically, we cannot do much about this yet, and if we tried, there would be a great risk of backfire. So we will continue to focus our efforts on more tractable causes, but always with a watchful eye toward the political landscape in case Moloch’s grip slips.

Missing the mood: If we didn’t have the current education system, some kids might not receive [some hazy, questionable benefit]. Don't throw the baby out with the bathwater! See part V of this ACX post.

Not missing the mood: Yes, huge amounts of childhood flourishing and human potential could be salvaged if we had an ideal school system. Yes, our existing school system does instill a lifelong dislike of math, writing, history, etc. for many (or even most) students. But Chesteron's fence and several other heuristics suggest that if we simply tore town the current system and distributed the money back to the public, everything might be even worse. Ponder this fact: if you drop a random US high school graduate in a random US city, they will be able to read road signs to get to the transit center, and then mentally compare--quantitatively--the costs of getting home. That kind of literacy and numeracy is a big deal, and it is our system of compulsory schooling that lets us take it for granted. And no, this is not a mundane tradeoff--the direct cost to an individual student is horrendous. But this is the price of modernity. Perhaps we ought to have school assemblies explaining this tradeoff to the students, and honoring their sacrifice and endurance. Perhaps we ought to beg kids for their patience and understanding every single day that we send them to school....
Motivations, Natural Selection, and Curriculum Engineering
by Oliver Sourbut — 2021-12-16T01:07:26.100Z

Epistemic status: I am professionally familiar with contemporary machine learning theory and practice, and have a long-standing amateur interest in natural history. But (despite being a long-term lurker on LW and AF) I feel very uncertain about the nature and status of goals and motivations! This post is an attempt, heavily inspired by Richard Ngo’s Shaping Safer Goals sequence, to lay out a few ways of thinking about goals and motivations in natural and artificial systems, to point to a class of possible prosaic development paths to AGI where general capabilities emerge from extensive open-ended training, and finally observe some desiderata and gaps in our understanding regarding alignment conditional on this situation.

This post was supported by the Stanford Existential Risks Initiative Machine Learning Alignment Scholars (MATS) program and I thank them for their support, Evan Hubinger for his mentorship during the project, and Richard Ngo and Oliver Zhang for helpful discussion and feedback on portions of the post.

Abstract/TL;DR

Recent capabilities progress, scaling and complexity observations, and (pre)historical precedent all suggest that open-ended training of baseline/foundation models is emerging as a plausible ‘very prosaic’ route to AGI. Multi-agent environments may include a particularly rich class of training regimes. Any generally-applicable capabilities such foundation models develop may by default come along with ‘core motivations’ which develop over the training process. We can not necessarily expect those motivations to generalise in aligned ways at deployment.

There are many free variables in the regimes described in the preceding claims, including end-to-endness, diversity and kinds of training curricula, extent and form of human participation, sizes and homogeneity of populations, architectures, means of information transmission, etc. We would gain from investment in exploring the space, the better to develop techniques for making the route safer, and the better to steer through safer sub-routes (or motivate opposition to this route entirely).

This post begins by elaborating and challenging the above framing. Then some suggestions are made for ways of breaking down the concept of motivations, followed by a presentation of some considerations and desiderata for how we might influence them in the above setting. Genetic natural selection and i...
Universality and the “Filter”
by maggiehayes — 2021-12-16T00:47:23.666Z

This post was written under Evan Hubinger’s direct guidance and mentorship, as a part of the

Stanford Existential Risks Institute ML Alignment Theory Scholars (MATS) program

.

TL;DR: In this post, I refine universality, in order to explain the "Filter". The Filter is a method of risk analysis for the outputs of powerful AI systems. I refine universality by replacing the language of “beliefs” and proposing a handful of general definitions in part I. Then, I explain the key application of universality of building the Filter. I explain the Filter non-technically through a parable in part II, then technically in part III. To understand the Filter as presented in part II, it is not necessary to comprehend universality to the rigor that I present in part I; it is only necessary to see that all models are wrong1. While this post builds technically off a previous post2, part II is meant to be understandable to all audiences.

I. Refining Universality

Universality is discussed

here

,

here

, and

here

. Here are some features of universality that have been missing from its presentation.

1. The need to replace “beliefs”

A “belief” of a system is used 60 times in Paul Christiano’s

attempt to formalize universality

. I think a “belief” is not the right language for an object of a system for two reasons. First, it conflates terms worth making distinct. When “beliefs” are used in Paul Christiano’s

attempt to formalize universality

, they don’t always refer to the same thing. Second, “belief” should be reserved for a different concept. The way humans think of “belief” is not how it’s used there.

So, I rewrote the formalization of universality to not include the word “belief”. The handful of terms presented below combine to replace the previous notion of “belief”. Presenting universality without the language of “beliefs”, all said, is certainly more cumbersome.

I hope this reframing will help improve the methodology for applying universality to HCH. I apply the general definitions below to universality and the Filter in section III.2.

2. General definitions for universality

Consider κ, a list of action-shaping intelligible information. κ={κj}, where κj are pieces of intelligible information. Pieces of intelligible information are facts, which are sentences (i.e. “The sky is blue”, “there are infinitely many twin primes”). Another term, X, is t...
More power to you
by jasoncrawford — 2021-12-15T23:50:18.975Z
Link: https://rootsofprogress.org/flying-car-ad

When people say things like “we were promised flying cars,” I sometimes wonder, “who promised this?”

I guess this is what they mean. From a 1959 ad that ran in the LA Times:

As a friend pointed out, “they're not even wearing seat belts!” The Los Angeles Times, June 21, 1959

“They’re working on it!” the ad claims. “Some of this is happening already.” (Implying, of course, that some of it was still pretty speculative.)

The ability to “dial” a book, lecture, or demonstration is here; the ultrasound dishwasher, automatic bed-maker, and flying car sadly are not.

But here’s what’s most interesting to me: First, the reference, without explanation or justification, to “tomorrow’s higher standard of living”—something people simply assumed was coming. Second, that it was uncontroversial that this higher standard of living would require more electricity. Finally, the boasting of doubling electricity production in ten years. As I’ve pointed out, society once had a different fundamental attitude towards growth.

The US did, in fact, approximately double electricity generation every decade from 1950 until 1973—the year of the oil crisis—after which electricity growth never really recovered. It took almost thirty years to double again, from 1973 to 2000, during which time it was just keeping up with population growth; that is, per-capita energy usage was not increasing. And since 2005, even the total amount produced has been flat:

US electricity generation. Data from US Energy Information Administration

“The electric companies are resolved,” the ad says, to maintain for America “the best electric power service in the world.” Well, we are still one of the top overall producers, second only to China:

But since the oil shocks, I’d say that “resolve” has been shaken.
My Overview of the AI Alignment Landscape: A Bird's Eye View
by Neel Nanda — 2021-12-15T23:44:31.873Z

Disclaimer: I recently started as an interpretability researcher at Anthropic, but I wrote this doc before starting, and it entirely represents my personal views not those of my employer

Intended audience: People who understand why you might think that AI Alignment is important, but want to understand what AI researchers actually do and why.

Epistemic status:

My best guess

.

Epistemic effort: About 70 hours into the full sequence, and feedback from over 30 people

Special thanks to Sydney von Arx and Ben Laurense for getting me to actually finish this, and to all of the many, many people who gave me feedback. This began as my capstone project in the first run of the

AGI Safety Fellowship

, organised by Richard Ngo and facilitated by Evan Hubinger - thanks a lot to them both!

Meta: This is a heavily abridged overview (4K words) of a longer doc (25K words) I’m writing, giving my birds-eye conceptualisation of the field of Alignment. UPDATE: This was intended to be a full sequence, but will likely be eternally incomplete - you can read a draft of the full sequence

here

.

Terminology note: There is a lot of disagreement about what “intelligence”, “human-level”, “transformative” or AGI even means. For simplicity, I will use AGI as a catch-all term for ‘the kind of powerful AI that we care about’. If you find this unsatisfyingly vague,

OpenPhil’s definition of Transformative AI

is my favourite precise definition.

Introduction

What needs to be done to make the development of AGI safe? This is the fundamental question of AI Alignment research, and there are many possible answers.

I've spent the past year trying to get into AI Alignment work, and broadly found it pretty confusing to get my head around what's going on at first. Anecdotally, this is a common experience. The best way I've found of understanding the field is by understanding the different approaches to this question. In this post, I try to write up the most common schools of thought on this question, and break down the research that goes on according to which perspective it best fits

There are already some excellent overviews of the field: I particularly like

Paul Christiano’s Breakdown

and Rohin Shah’s

literature review

and

interview

. The thing I’m trying to do differently here is focus on the motivations behind the work. AI Alignment work is challenging and confusing because it involves reasoning about future risk...
SmartPoop 1.0: An AI Safety Science-Fiction
by Lê Nguyên Hoang — 2021-12-15T22:28:50.615Z

I have just released a science-fiction novel called SmartPoop, which is freely available on github, in epub and in pdf, and can be purchased on Amazon. The book is co-written with a medical doctor friend. Here's the pitch:

SmartPoop 1.0 aims to be a very realist science fiction, whose goal is to raise awareness of the danger of algorithms and the challenges to make them robustly beneficial. It tells the story of a tech start-up, SmartPoop, which produces automated poop analysis. This allows for highly personalized medical diagnosis. However, in each chapter, the deployment of SmartPoop leads to unwanted side effects, whose medical and social consequences are usually tragic. The co-founders of SmartPoop, who are the main characters, must then take the responsibility of their technology, accept the consequences, and find solutions to make their products more secure and ethical.

As a friend put it, the book is "scato-pedagogical". While it is meant to be easy and fun to read for all, it actually illustrates through very concrete examples my huge concerns over the safety of very powerful algorithmic systems, from governance to alignment. It might be a fun Christmas present 😋
Framing approaches to alignment and the hard problem of AI cognition
by ryan_greenblatt — 2021-12-15T19:06:52.640Z

(Thanks to Adam Shimi and Steven Byrnes for feedback.)

(Edit: unsurprisingly, my views have updated some after writing this post. Specifically, I now put much higher credence on solving inner alignment via verification or relaxed adversarial training)

1 Background

The late 2021 MIRI conversations include discussion about the
difficulty of alignment (don't worry, spending hours reading these isn't
required for this post). One shared frame which wasn't immediately
present in the discussion was a clean delineation of possible approaches to
alignment and what they require. I claim that alignment techniques can be
usefully understood as deontological, consequentialist, or capability
restriction (or a mixture of these) and we'll be going through the challenges
associated with constructing sufficiently useful and safe AI using these
approaches. I'll also be discussing and motivating a set of issues which I'll
refer to as 'the hard problem of AI cognition': we don't have tools for
understanding the cognition or intention of AIs produced by current machine
learning methods or understanding of how AI decision making depends on training
(c.f. inner alignment).[1] I'll define this problem in
more detail below, including where this problem does and doesn't show up in
alignment. After going through deontological, consequentialist, and
capability restriction approaches at a high level and explaining where I think
the hard problem of AI cognition is relevant, I'll explain my take on the
strategic implications of this analysis and briefly discuss alignment
difficulty. We'll be focussing on X-risk, so we won't directly discuss failures
which would 'only' result in large losses of life or economic damage. This is
primarily a 'first principles' sort of analysis, though I'll be implicitly (and
occasionally explicitly) referencing empirical work.

Epistemic status: exploratory. While many of the ideas stated here appear to be
widely accepted in the alignment community, I'm a newcomer to the field trying
to cover a lot of ground. But everyone keeps telling me to be Very Ambitious
and that alignment lacks strategic direction. So, uh, here goes an attempt at
that I guess?

2 Prerequisites

The main prerequisite will be the sorts of concepts discussed in AGI safety
from first principles.

We'll refer in more detail to:

The idea of utility maximization and that coherent decisions imply
cons

...
Leverage
by lsusr — 2021-12-15T05:20:46.287Z

If you want to change the world then you need to act on scale. A single human being cannot act on scale directly. You need leverage. Leverage is mechanism by which you can cause an impact several orders of magnitude bigger than yourself.

Sam Altman gives five examples of leverage: delegation, capital, brand, network and technology.

Delegation

The most obvious way to obtain leverage is to manage other people. A disadvantage of managing people is cost. Salaries are expensive. Communication is expensive. Your employees interests are often unaligned with your own.

The biggest disadvantage of managing people is that some tasks cannot be parallelized. A hundred mediocre singers cannot work together to write You Belong With Me or Thunder Road. A hundred mediocre writers working together could not produce Harry Potter or The Fault in Our Stars. A hundred mediocre Age of Empires II players could not work together to compete with TheViper because only one player is allowed at the keyboard at the time.

Capital

If you want Carly Rae Jepsen to produce a new hit song then the first thing you should do is throw as much money at her equipment as you can. Carly Rae Jepsen shouldn't just have the best microphone in the world. She should have an entire sound crew setting things up for her. She should have a limo taking her wherever she needs to go. Carly Rae Jepsen is a bottleneck. Any time she spends thinking about sound or makeup or scheduling is time she's not spending writing or singing or practicing songs.

Buying infrastructure is optimal when you have plenty of money and are bottlenecked by talent.

Brand

Brand is when people trust a symbol. Trust reduces transaction costs. The disadvantage of brand is it cannot be bought; trust must be earned.

Network

Networks are another form of trust. Usually the hardest part of building a network is bootstrapping it. After that, you can grow your network organically with little effort.

Inventing Technology

Buying technology is amassing capital. Inventing technology creates a way to amass capital. Technological invention is leverage squared.

The disadvantage of technological invention is it is hard to do and hard to delegate. But if you can do it, technological invention is among the highest leverage activities there is.
We'll Always Have Crazy
by Duncan Sabien (Inactive) — 2021-12-15T02:55:29.940Z

Content warning: preparadigmatic meander; half-baked models; thinking-out-loud. Unlike the median Duncan essay.

Logan Strohl's response to the first draft of the below:

i really wish this were a more-effective-for-me invitation to help you figure something out. i think that's the version of it i would upvote, or maybe even strong upvote.

right now i feel uncomfortable or something about having read it, and if it were a topic i already cared a lot more about i'd probably feel angry or maybe even betrayed. as it is i feel a tiny bit like i've been contaminated, because of something like a wishy washy back and forth between trying to figure out what you're saying, trying to figure out what you're doing, having a feeling that there's something you want me to believe, trying to figure out what i do believe, trying to figure out what topics i need to try to get into my own sight before trying to understand you so that my own eyeballs don't get carelessly knocked over... whatever i'm trying to say here is more concentrated in that last thing

i think i would like a version of this post that begins with a short list of questions that i'm invited to try to answer on my own before i read, 'cause then i'd try to answer them on my own first and be much much better prepared for this essay

Do you think that the swath of society you live in is less crazy than society as a whole? Why?
Do you think that the swath of society you live in is less crazy than its analogue in society 100 years ago? Why?
How do you feel about the swaths of society that seem crazy to you? Why?
How do you feel about the future of society, on the question of craziness? Do you think the median craziness is moving? Do you think the extremes of craziness are moving? Do you think that the bulk of people are moving toward the middle, or the tails are fattening, or society is splitting into a two-peaked distribution, or or or or or? (various permutations of that kind of question)
What have you been using the word 'crazy' to mean, in the above? What do you think I use the word 'crazy' to mean? What do you think the median LWer uses it to mean?
What do you wish for, if anything, re: society and craziness?

For a long time, I didn't understand why elections were so often so close.

It seemed to kid-me that one side was (probably) strictly superior to the other, in t...