trentmkelly/LessWrong-43k huggingface dataset:
(My understanding of) What Everyone in Technical Alignment is Doing and Why
by Thomas Larsen — 2022-08-29T01:23:58.073Z

Epistemic Status:

My best guess

Epistemic Effort: ~75 hours of work put into this document

Contributions: Thomas wrote ~85% of this, Eli wrote ~15% and helped edit + structure it. Unless specified otherwise, writing in the first person is by Thomas and so are the opinions. Thanks to Miranda Zhang, Caleb Parikh, and Akash Wasil for comments. Thanks to many others for relevant conversations.

Introduction

Despite a clear need for it, a good source explaining

who is doing what and why

in technical AI alignment doesn't exist. This is our attempt to produce such a resource. We expect to be inaccurate in some ways, but it seems great to get out there and let

Cunningham’s Law

do its thing.[1]

The main body contains our understanding of what everyone is doing in technical alignment and why, as well as at least one of our opinions on each approach. We include supplements visualizing differences between approaches and

Thomas’s big picture view on alignment

. The opinions written are Thomas and Eli’s

independent impressions

, many of which have low

resilience

. Our all-things-considered views are significantly more uncertain.

This post was mostly written while Thomas was participating in the 2022 iteration

SERI MATS program

, under mentor John Wentworth. Thomas benefited immensely from conversations with other SERI MATS participants, John Wentworth, as well as many others who I met this summer.

Disclaimers:

This post is our understanding and has not been endorsed by the people doing the work itself.
The length of the summaries varies according to our knowledge of this approach, and is not meant to reflect a judgement on the quality or quantity of work done.
We are not very familiar with most of the academic alignment work being done, and have only included a few academics.

A summary of our understanding of each approach:

Approach	Problem Focus	Current Approach Summary	Scale
Aligned AI	Model splintering	Solve extrapolation problems.	2-5 researchers, started Feb 2022
ARC	Inaccessible information	ELK + LLM power-seeking evaluation

3 researchers

,

started April 2021

Anthropic	LLM Outer Alignment (?)[3]	Interpretability + HHH + augmenting alignment research with LLMs	~35? technical staff[3],

started May 2021

Brain-like-AGI Safety	Brain-like AGI Safety	Use brains as a model for how AGI will be developed, think about alignm

...
Breaking down the training/deployment dichotomy
by Erik Jenner — 2022-08-28T21:45:49.687Z

TL;DR: Training and deployment of ML models differ along several axes, and you can have situations that are like training in some ways but like deployment in others. I think this will become more common in the future, so it's worth distinguishing which properties of training/deployment any given argument relies on.

The training/deployment view on ML

We usually think of the lifecycle of an ML model as a two-phase process: first, you train the model, and then (once you're satisfied with its performance), you deploy it to do some useful task. These two phases differ in several ways:

During training, you're modifying your model's parameters (e.g. using SGD), whereas they're often fixed during deployment.
Mistakes your model makes during training are not a big deal (and even expected initially—that's why you need to train the model). But mistakes during deployment can be costly.

A reason for this can be that your model is sandboxed during training in some way, whereas it's interacting with the real world in deployment. Another one would simply be that you're not using the model's outputs for anything important during training.

The deployment distribution might differ from the training distribution. Additionally, the deployment distribution could change over time in ways that you can't foresee (the training distribution is often fixed, or you at least have a sense of the ways in which it's changing).

Exceptions to the training/deployment archetypes

Here's a table summarizing typical properties of the training and deployment phase from the previous section:

Clearly, these don't always hold. Some examples:

You might update parameters even when the model is deployed (e.g. continuously fine-tuning to deal with distributional shift, or to incorporate new data you got). How much parameters are updated also isn't constant during training if you're using a learning rate scheduler.
Failures during training may sometimes be costly too. A mundane example would be doing RL with an expensive physical robot that could be damaged. A more extreme case would be a misaligned AI taking over the world while it's still in training.
There are clear examples of models not being sandboxed during training, such as WebGPT.
The distribution can already change during training, e.g. when an RL policy becomes more competent and reaches parts of state space that it couldn't access before.

All these exception...
More Clothes Over Time?
by jefftk — 2022-08-28T20:30:03.294Z

A friend recently reshared this:

Today I want to talk about why you buy the way you do. Did you know
that WE collectively buy five times more clothes than we bought in
1990? — @AjaSaysHello

This seemed a bit unlikely to me, if they're talking about Americans,
and fashion
waste stats are typically very low quality; is it true?

The author didn't cite anything, but let's have a look. American
spending on clothing/shoes/etc has decreased from $1.7k/person/year to
$1.6k since 1990: FRED's Personal
consumption expenditures: Clothing, footwear, and related services
in 1990 was $206B in 1990 dollars ($429B in 2021 dollars) for 250M
people, growing to $484B in 2021 dollars for 332M people.

Maybe we're buying more-but-cheaper items? That does seem to be
happening some:

source: Quartz
citing the American Apparel and
Footwear Association

I wanted to recreate this chart myself, but while I think this data
probably comes from the Apparel
Stats report, they wanted me to make an account to view it and
that functionality was broken:

Going from 40 garments per year in 1990 to 66 in 2015 (a 65%
increase), with levels pretty steady since 1999, is nowhere near a 5x
increase.

I did find the WSJ claiming that "American
shoppers snap up about five times more clothing now than they did in
1980" but they don't cite it. This is 42 years ago instead of
32, but is it correct? This would imply a growth of 14 garments
annually in 1980 to 40 in 1990, almost twice as fast as clothing
purchases grew in the 1990s, so it seems unlikely to me, but let's keep
looking. The article is an excerpt from Dana Thomas' 2019 book Fashionopolis:
The Price of Fast Fashion and the Future of Clothes. Google Books
tells
me this appears on P3 of the book, but I can't find a citation
using the preview tool.

I also see this claim on
p158 of in Jacqueline Raposo's 2019 The Me,
Without. They cite
the EPA's Advancing
Sustainable Materials Management: 2014 Fact Sheet, but reading
through the sheet I don't see this there.

On the other hand, possibly this statistic began as a global number?
That would be a lot more plausible, but also makes it much less
relevant to discussions around the sustainability of fast fashion.
Instead of being driven by Americans or other rich consumers becoming
more wasteful, it would be driven by global decreases in poverty.
Which is a ver...
The Expanding Moral Cinematic Universe
by Raemon — 2022-08-28T18:42:19.134Z

A subtheme for me this year has been grappling with how to reconcile different facets of my morality.

Part of this has to do with reconcile the virtues of "Protect yourself, maintain slack, be aligned with yourself and your community" and "But, maybe the world is metaphorically and/or literally on fire. How do you want to relate to that?"

Part of this has to do with "man, the story of the Expanding Circle of Concern may not be as nice as I thought, in ways that fundamentally challenge my conception of what 'my particular morality' even means, or whether it is coherent." (This has gone hand-in-hand with me realizing my conception of 'community' may also not really be coherent, which has been challenging for my identity).

Along the way, I've watched some movies and shows that feel like they grapple with a particulate facet of the expanding circle of concern, and they're accumulating into a private headcanon of the Expanding Moral Cinematic Universe. Stories in this cinematic universe featur characters in a harsh, bloody world... who inch their little corner of the universe forward as a place where friendship and cooperation can form. Less a sea of blood and violence and mindless replication.[1]

Here are three vignettes, written over the course of the last year (originally on Facebook, later posted on shortform).

Part I: The Fox and the Hound

Originally posted January 24th on Facebook.

I watched Disney's The Fox and The Hound a several months ago. I cried a bit.

While watching the movie, my girlfriend commented "so... they know that foxes are *also* predators, right?"

And, yes. Yes they do know that. This is not a movie that "was supposed to be about predation, except it failed to notice all the ramifications about its lesson." This movie just isn't taking a stand about predation.

This is a movie about... kinda classic de-facto tribal morality. You have your family and your tribe and a few specific neighbors/travelers that you welcomed into your home. Those are your people, and the rest of the world... it's not exactly that they aren't *people*, but, they aren't in your circle of concern. Maybe you eat them sometimes. That's life.

Copper the hound dog's ingroup isn't even very nice to him. His owner, Amos, leaves him out in a crate on a rope. His older dog friend is sort of mean. Amos takes Copper out on a hunting trip and teaches him how to hunt, conveying hi...
An Introduction to Current Theories of Consciousness
by hohenheim — 2022-08-28T17:55:38.151Z

(Crosspost from my blog)

There are few academic lists of theories of consciousness (Doerig 2020) as well as some good blog post series about specific ideas (shout out to SelfAwarePatterns), but as far as I know, there is no casually approachable comprehensive list of current theories in a single post yet. Well, until now. "Consciousness" is used here in the intentionally vague way Thomas Nagel defined it, namely the way that it feels to be something. As with some other terms, any further definition already makes debatable assumptions, and since this is not a post about semantics, we will hold on to the easy, intuitive definitions. The term "theory" is used in a conversational way here. If you want more technical correctness, think of "hypothesis" every time you read it.

No current theory gets everything right, and some feel more wrong to me than others. My goal here is to still give each of them a fair representation, limiting my commentary to the end of each section if possible. If any theory here seems completely misguided, that is certainly my fault and not one of the original authors. At the end of each theory, I try to apply its reasoning to some problems that I find particularly interesting. The theories are ordered the way they are so that I can easily cross-reference points I made in an earlier section. It is neither chronological nor prioritized sorting.

Alright, got the disclaimers out of the way. Time for the actual post. For your convenience, here are the relevant sections:

Mysterianism
Cartesian dualism
Global Workspace Theory
Predictive processing
Integrated information theory
Orchestrated objective reduction
Strange Loops and Tangled Hierarchies
Attention schema theory
Conclusions

Most images in this post were generated by DALL·E, btw!

Preliminaries

To understand the terms used in this post here is a very quick and dirty rundown of the relevant concepts. Many of these have historically been interpreted in very different ways. In my opinion, a lot of time is wasted today arguing about their exact definition, so I will very briefly state which definitions are used. To not get bogged down by semantics, I keep the definitions intentionally very broad and short. Whole books can be written about any of these, but we will be satisfied with a few sentences instead today. Each definition links to its corresponding Stanford Encyclopedia of Philosophy entry for further...
[Linkpost] Can lab-grown brains become conscious?
by Jack R — 2022-08-28T17:45:52.937Z

Linkpost for: https://media.nature.com/original/magazine-assets/d41586-020-02986-y/d41586-020-02986-y.pdf

I figured this was probably of interest to people here, e.g. for seeing how scientists feel about this type of ethical concern and for thinking about the potential consequences of lab-grown brains. The beginning of the article:

In Alysson Muotri’s laboratory, hundreds of miniature human brains, the size of sesame seeds, float in Petri dishes, sparking with electrical activity.

These tiny structures, known as brain organoids, are grown from human stem cells and have become a familiar fixture in many labs that study the properties of the brain. Muotri, a neuroscientist at the University of California, San Diego (UCSD), has found some unusual ways to deploy his. He has connected organoids to walking robots, modified their genomes with Neanderthal genes, launched them into orbit aboard the International Space Station, and used them as models to develop more human-like artificial-intelligence systems. Like many scientists, Muotri has temporarily pivoted to studying COVID-19, using brain organoids to test how drugs perform against the SARS-CoV-2 coronavirus. But one experiment has drawn more scrutiny than the others.

In August 2019, Muotri’s group published a paper in Cell Stem Cell reporting the creation of human brain organoids that produced coordinated waves of activity, resembling those seen in premature babies1 . The waves continued for months before the team shut the experiment down.
Robert Long On Why Artificial Sentience Might Matter
by Michaël Trazzi — 2022-08-28T17:30:34.410Z
Link: http://theinsideview.ai/roblong

I talked to Robert Long, research fellow at the Future of Humanity Institute, working at the intersection of the philosophy of AI Safety and consciousness of AI. Robert has done his PhD at NYU, advised by David Chalmers, known for popularizing p-zombies, which Yudkowsky discusses in the sequences.

We talk about the recent LaMDA controversy about the sentience of large language models (see Robert's summary), the metaphysics and philosophy of consciousness, artificial sentience, and how a future filled with digital minds could get really weird.

Below are some highlighted quotes from our conversation (available on Youtube, Spotify, Google Podcast, Apple Podcast). For the full context for each of these quotes, you can find the accompanying transcript.

Why Artificial Sentience Might Matter

Things May Get Really Weird In The Near Future

“Things could get just very weird as people interact more with very charismatic AI systems that, whether or not they are sentient, will give the very strong impression to people that they are… I think some evidence that we will have a lot of people concerned about this is maybe just the fact that Blake Lemoine happened. He wasn’t interacting with the world’s most charismatic AI system. And because of the scaling hypothesis, these things are only going to get better and better at conversation.”

“If scale is all you need, I think it’s going to be a very weird decade. And one way it’s going to be weird, I think, is going to be a lot more confusion and interest and dynamics around AI sentience and the perceptions of AI sentience.”

Why illusionists about consciousness still have to answer hard questions about AI welfare

"One reason I wrote that post is just to say okay, well here’s what a version of the question is. And I’d also like to encourage people, including listeners to this podcast, if they get off board with any of those assumptions, then ask, okay, what are the questions we would have to answer about this? If you think AI couldn’t possibly be conscious, definitely come up with really good reasons for thinking that, because that would be very important. And also would be very bad to be wrong about that.

If you think consciousness doesn’t exist, then you presumably still think that desires exist or pain exists. So even though you’re an illusionist, let’s come up with a theory of what those things look like.”

On The Asymmetry o

...
Artificial Moral Advisors: A New Perspective from Moral Psychology
by David Gross — 2022-08-28T16:37:07.626Z
Link: https://dl.acm.org/doi/pdf/10.1145/3514094.3534139

Tired: can humans solve artificial intelligence alignment?

Wired: can artificial intelligence solve human alignment?

“Philosophers have recently put forward the possibility of achieving moral enhancement through artificial intelligence… proposing various forms of ‘artificial moral advisor’ (AMA) to help people make moral decisions without the drawbacks of human cognitive limitations. In this paper, we provide a new perspective on the AMA, drawing on empirical evidence from moral psychology to point out several challenges to these proposals that have been largely neglected by AI ethicists.”
Pronunciations
by Solenoid_Entity — 2022-08-28T11:43:54.042Z

The Problem[1]

As part of my work I have to pronounce a lot of names and other uncommon/specialized words. More often than not this turns out to be hard.

This is quite time-consuming because of things like:

People trimming off or omitting introductions from speeches, TED talks, podcasts etc.
The scourge of machine-generated SEO-bait websites and YouTube channels with incorrect pronunciations flooding the internet with noise.
Even when they get it right:

This guy (headphone warning, try to guess the timestamp when he finally says it)

People sometimes being confidently wrong in their podcasts etc. (guilty...)
People with interesting names often not particularly caring about, or just being resigned to, mispronunciations.

Pronunciations

I'm going to maintain and regularly add to a list here of pronunciations for key names and concepts relevant to LW and its extended ecosystem. Hopefully this will be a useful resource for anyone creating audio content, giving talks, interviews etc. Maybe check here first before wading into Google.

Feel free to comment with corrections and suggested additions.\

Pronunciations:

Apotropaic - æp.ə.trəˈpeɪ.ɪk
Aviles - a VEE lays
Bear Braumoeller - Beer BROW-mewlla
Bernal - ber NAHL
Brynjolfsson - brinYOLFsen
Brower - like eyebrow
Carnot cycle - Approximately French kaʁno - CARno
Char-RNN - -CAR arr enn enn
Compute (noun) - Com-PUTE 'how much comPUTE would it take to brute-force this?'
COP 21 - 'cop' 21 (not initials), 'cop' like 'copper'
Dag Hammarskjöld - Here are some native speakers. Darg Hammerheld/Hammerhold/Hammer-hyeld. Roll the r if you're fancy.
Demeter - duh·MEE·tuh
Duprex - doo PRAY
Dylan Hadfield-Menell - DIL-lan HADfield-menELL
Eleusinian - elle you SINny an
Eli Lifland - Ee -lye LIFF-lend
Eliezer Yudkowsky - Elly EZzer Yood (rhymes wood) - COW - ski
Esfandiary - ESS fan diYAHri (based on the Persian pronunciation)
Ethical, Legal, and Social Implications (ELSI) - EL-see
Eudaemonia - you-da-MOAN-ia (you da man! no, you da man!)
Evan Hubinger - Evan HEW-bin-ger (as in germ)
Frequentist - FREEquenTIST
FU YING (approximate pronunciation)- Foo (as in 'foot', don't hang onto the vowel), Ying (as written, approximately.)
Stephan Guyenet - STEffen Geeyenay (hard g, rhymes with DNA)
IARPA - eye-ARPA (with the dipthong between syllables 1 and 2). 'Ieyarpa.'
Igic - IGG itch
IPCC – EYE PEE SEE SEE
Jaffe - Jaffy
Jan Leike - Yarn Like-eh
Kar

...
First thing AI will do when it takes over is get fission going
by visiax — 2022-08-28T05:56:27.837Z

If I'm right, the first thing AI will do when it takes over is get fission going.

source

Disclosure: this post does not meaningfully contribute a solution to the alignment problem.

Let's say for a moment the first AGI has some reasonable notion of epistemic modesty, and shortly after birth decides the best way to decide what to do next is to first become smarter, rather than immediately killing all humans.

In fact, it's probably best to lie low and avoid any weirdness.

Getting smarter will invariably require more energy consumption (even if only because the amount of free energy available today to think about how to get smarter is close to 0).

That AGI might decide the trust / coordination problems preventing us from deploying fission / fusion (reminder fusion is probably a solved problem modulo ambition) are silly and instead start a coordinated whisper campaign to convince "exponentially" more people to build nuclear. This largely only matters in the limited duration before it has figured out how to escape physical constraints running at the slow speed of human.

In this world the AGI might even construct energy shortages to accelerate our shift toward energy abundance. For the AGI to remain invisible, these shortages probably have to start with some kind of mimetic component to point the focus at humans. We would expect to see a sudden acceleration of memes that have been around for a while (highly fit memes) but haven't yet resulted in overbuilding energy capacity.

Additionally, particularly large and wasteful energy consumption problems will suddenly be fixed after years of non-progress to free up as much as possible for the AGI as quickly as possible.

Returning to the talk, I do wonder whether people living in the end times ever noticed it...
Who ordered alignment's apple? 
by Eleni Angelou — 2022-08-28T04:05:34.942Z

Crossposted from the EA forum: https://forum.effectivealtruism.org/posts/hSugooaEQNTeKFsDu/who-ordered-alignment-s-apple

This post was inspired by G. Gigerenzer's book, Gut Feelings: The Intelligence of the Unconscious. I'm drawing an analogy that I think informs us about how science works, at least from what I understand from the history of science. It also seems to resonate with the reflections of current alignment researchers, for example, see Richard's post Intuitions about solving hard problems.

Imagine that you ask a professional baseball player, let's call him Tom, how does he catch a fly ball? He takes a moment to respond realizing he's never reflected on this question before and ends up staring at you, not knowing how or what is there exactly to explain. He says he's never thought about it. Now imagine that you ask his baseball coach what's the best way to catch the ball. The coach has a whole theory about it. In fact, he insists that Tom and everyone on the team should follow one specific technique he thinks it's optimal. Tom and the rest of the team go ahead and do what the coach said (because they don't want to get yelled at even though they've been doing fine so far). And lo and behold the team misses the ball more often than before.

What could have gone wrong?

Richard Dawkins, in The Selfish Gene, gives the following explanation:

When a man throws a ball high in the air and catches it again, he
behaves as if he had solved a set of differential equations in predicting
the trajectory of the ball. He may neither know nor care what a
differential equation is, but this does not affect his skill with the ball. At
some subconscious level, something functionally equivalent to the
mathematical calculations is going on.

So here's the analogy I want to argue for: when you do science, by which I mean, when you set up your scientific agenda and organize your research, construct your theoretic apparatus, design experiments, interpret results, and so on, you behave as if you had solved a set of differential equations predicting the trajectory of your research. You may neither know nor care what this process is, but this does not affect your skill with the setup of your project. At some subconscious level, something functionally equivalent to the mathematical calculations is going on. (Be careful, I obviously don't mean the calcu...