trentmkelly/LessWrong-43k huggingface dataset:
A descriptive, not prescriptive, overview of current AI Alignment Research
by Jan — 2022-06-06T21:59:22.344Z

TL;DR: In this project, we collected and cataloged AI alignment research literature and analyzed the resulting dataset in an unbiased way to identify major research directions. We found that the field is growing quickly, with several subfields emerging in parallel. We looked at the subfields and identified the prominent researchers, recurring topics, and different modes of communication in each. Furthermore, we found that a classifier trained on AI alignment research articles can detect relevant articles that we did not originally include in the dataset.

(video presentation here)

Dataset Announcement

In the context of the 6th AISC, we collected a dataset of alignment research articles from a variety of different sources. This dataset is now available for download here and the code for reproducing the scrape is on GitHub here[1]. When using the dataset, please cite our manuscript as described in the footnote[2].

Table 1: Different sources of text included in the dataset alongside the number of articles per source. Color of row indicates that data was analyzed as AI alignment research articles (green) or baseline (gray), or that the articles were added to the dataset as a result of the analysis in Fig. 4 (purple). Definition of level-0 and level-1 articles in Fig. 4c. For details about our collection procedure see the Methods section.

Here follows an abbreviated version of the full manuscript, which contains additional analysis and discussion.

Rapid growth of AI Alignment research from 2012 to 2022 across two platforms

After collecting the dataset, we analyzed the two largest non-redundant sources of articles, Alignment Forum (AF) and arXiv. We found rapid growth in publications on the AF (Fig. 1a) and a long-tailed distribution of articles per researcher (Fig. 1b) and researchers per article (Fig. 1c). We were surprised to find a decrease in publications on the arXiv in recent years, but identified the cause for the decrease as spurious and fixed the issue in the published dataset (details in Fig. 4).

Figure 1: Alignment Research across a community forum and a preprint server. (a) Number of articles published as a function of time on the alignment forum (AF; purple) and the arXiv preprint server (arXiv; green). (b) Histogram of the number of articles per researcher published on either AF or arXiv. Inset shows names of six researchers with more than 60 articles. Note

...
Microphone on Electric Mandolin II
by jefftk — 2022-06-06T20:30:02.423Z

Several weeks ago

I got
excited about the idea of putting an additional microphone on my
electric mandolin to capture more of the percussive sound. On
Saturday with

Kingfisher
I made recordings at a dance. Listening back over them I have a lot
of thoughts, but first, some examples. "Mix" versions include the
mic, while "direct" versions don't.

Chords

I would call this the foundation of my contradance mandolin playing:
rhythmic and percussive. Here it is with fiddle melody:

(direct)

(mix)

You should be able to hear a little more string noise in the second
recording: high, percussive, crisp, clicky. This post is essentially
a whole bunch of examples to help decide whether it's an improvement
(and if so how much of one).

Here's what it sounds like later in the set after I came in on footbass:

(direct)

(mix)

Another straight-ahead reel:

(direct)

(mix)

Here's a clip from a different set with a more laid-back feel:

(direct)

(mix)

And an old-time set with pseudo-jawharp:

(direct)

(mix)

Here's one with the fiddle playing around with effects, and very high
one-string backup:

(direct)

(mix)

Melody

I'm not primarily a melody player on mandolin, but I do some.

Here's a reel with fiddle chords:

(direct)

(mix)

Here's a clip from a laid-back march, with fiddle playing long chords
and a bit of footbass:

(direct)

(mix)

Talkbox

Sometimes I route the mandolin through a talk box. This is one of the
places where the difference between the two methods is especially
clear, because the audio from the clip-on doesn't go via the talk box.

Some chords bouncing off the inside of my mouth, over
fiddle playing a groovy jig and some footdrums:

(direct)

(mix)

Here's fiddle and mandolin both playing chords. Not something we do a
very often, since you can lose the form, but nice for offering additional
texture options:

(direct)

(mix)

The talk box is good for melody too, with rhythmic fiddle
accompaniment:

(direct)

(mix)

Thoughts

If you haven't listened to the recordings yet and made up your own
mind, I'd encourage you to do that first before reading what I have
below.

I don't mind waiting...

Ok! There are some cases where I think having the acoustic mic makes
the overall sound better:

On reels when playing

...
We haven't quit evolution [short]
by the gears to ascension — 2022-06-06T19:07:14.025Z

edit: there is now a much higher quality version of this post, archive-browsers should read that instead.

I've seen folks say humanity's quick growth may have broken the link to evolution's primary objective, often referenced as total inclusive fitness. I don't think we have broken that connection.

Let process temporarily refer to any energy-consuming structured chemical or physical reaction that consumes fuel - this could also be termed "computation" or in many but not all cases "life".

let "defensibility" refer to the size of the moat in configuration space which maintains a process against interference - ie, nearest disrupting perturbation.

for all matter, Evolution-of-matter's-process optimizes for process-defensibility-per-unit-fuel.

genetic evolution is a subset of self-preserving processes. total inclusive fitness is intended to measure gene-level genetic selfishness in terms of offspring, but I would argue that discrete offspring are the wrong unit: genetic evolution's noise-defense-aka-mutation-resistance is built by the preservation of genes that increase durability*efficiency.

therefore, because improving the self-selection self-process by use of contraception allows humans to guide their own reproduction, contraception is not automatically a divergence from incentive - and to the degree it is, it's selected against.

therefore, improving the self-selection process by simply not dying allows humans to defend their own structure much more accurately than traditional reproduction - though it's not as defensible as strategies that replicate as hard as they can, a full integrated being can often be quite defensible over the medium term, and hopefully with life extension, over the long term as well.

as further evidence, humans appear to have a significant desire to remember. This is well-described by this framework as well! mental process also qualifies as an evolution-of-matter's-process, and thought patterns seek some set of accepted state transitions so that the after-transition structure qualifies as "self".

this also relates well to concerns folks on lesswrong have expressed regarding self-modification: all forms of process self-maintenance have some degree of self-update, and various energetic processes control their rate of self-update.

it relates to EY's view that a safe hard-ASI should be asked to pull the ladder up behind itself: to ensure its

...
Grokking “Forecasting TAI with biological anchors”
by anson.ho — 2022-06-06T18:58:31.604Z

Notes:

I give a visual explanation of Ajeya Cotra’s draft report,

Forecasting TAI with biological anchors (Cotra, 2020)

, summarising the key assumptions, intuitions, and conclusions
The diagrams can be found

here

– you can click on some of the boxes to get linked to the part of the report that you’re interested in[1]

Thanks to Michael Aird, Ashwin Acharya, and the Epoch team for suggestions and feedback! Special thanks to Jaime Sevilla and Ajeya Cotra for detailed feedback.

Executive Summary

Click here to skip the summary

Ajeya Cotra’s biological anchors framework attempts to forecast the development of Transformative AI (TAI) by treating compute as a key bottleneck to AI progress. This lets us focus on a concrete measure (compute, measured in

FLOP

) as a proxy for the question “when will TAI be developed?” Given this, we can decompose the question into two main questions:

2020 training compute requirements: How much compute will we need to train TAI, using 2020 Machine Learning architectures and algorithms?
Affordability of compute: How likely is it that we’ll be able to afford the compute required to train TAI in a particular year?

The second question can be tackled by turning to existing trends in three main factors: (1) algorithmic progress e.g. improved algorithmic efficiency, (2) decreasing computation prices e.g. due to hardware improvements, and (3) increased willingness to spend on compute.

The first question is significantly trickier. Cotra attempts to answer it by treating the brain as a “proof of concept” that the “amount of compute” used to “train” the brain can train a general intelligence. This lets her relate the question “how much compute will we need to train TAI?” with the question “how much ‘compute’ was used to ‘train’ the human brain?”. However, there’s no obvious single interpretation for the latter question, so Cotra comes up with six hypotheses for what this corresponds to, referring to these hypotheses as “biological anchors” or “bioanchors”:

Evolution anchor: Compute to train TAI = Compute performed over evolution since the first neurons
Lifetime anchor: Compute to train TAI = Compute performed by the human brain when maturing to an adult (0 to 32 years old)
Three neural network anchors: Anchor to the processing power of the human brain, and to empirical parameter scaling laws.

Technically there are three of these, corresponding

...
We Should Break Up Elite Colleges
by sahajsharda — 2022-06-06T18:15:57.452Z
Link: https://sahajsharda.substack.com/p/break-up-the-elite-college-seats?s=w

Elite colleges hoard seats, fix prices, and capture government aid. Even worse, they increasingly produce shoddy research that can't be replicated, and the research that can be replicated often isn't particularly groundbreaking.

Breaking up elite colleges to fix anti-competitive conduct and the basic research bottleneck is a super high leverage solution for longterm progress. I'll be writing more about these ideas going forward, but I just wanted to introduce myself and my substack (which is linked above).
Health & Lifestyle Interventions With Heavy-Tailed Outcomes?
by MondSemmel — 2022-06-06T16:26:49.012Z

A key question for advice in a domain is: how are outcomes in this domain statistically distributed?

If extremely good and extremely bad outcomes are rare, you probably can't do much better than just copying the default or median behavior. The value of acquiring information in this domain seems low.

Conversely, if the distribution has a heavy tail, then changing your behavior can avert very bad outcomes and/or lead to very good outcomes. In this case, the value of information in this domain seems comparatively much higher.

Hence the question: Which health & lifestyle interventions have particularly heavy-tailed outcomes?

Example prompts:

Toothpaste: Without having looked into this topic, I'd expect commercially available toothpaste to mostly be in the same ballpark in terms of outcomes. In which case there would be little value in optimizing this domain beyond choosing e.g. the most popular brand, or the product with the highest Amazon rating.

Proper Workout Technique: It's very easy to injure yourself with sufficiently bad workout technique. Irrespective of how much better the outcomes for the best technique are than for the median technique, it at least seems worth some effort to avoid the "bad technique" tail of the outcome distribution.

(There's also a discussion thread on the SSC subreddit.)
Reading the ethicists 2: Hunting for AI alignment papers
by Charlie Steiner — 2022-06-06T15:49:03.434Z

Introduction

I'm back, reading more papers in ethics (but now also in philosophy and wherever the citations lead me). Unlike last time when I surveyed one journal's approach to AI in general, this time I'm searching specifically for interesting papers that bear on aligning AGI, preferably written by people I've never heard of before. Are they going to be any good? *shrug*

To this end, I skimmed the titles and sometimes abstracts of the last 5 years of papers in a pretty big chunk of the AI ethics space, namely:

Ethics and Information Technology
Minds and Machines
AI and Ethics
Philosophy & Technology
Science and Engineering Ethics
AI & Society
IEEE Transactions on Philosophy and Society
And a few bonus miscellanea

From the set of all papers that even had a remote chance at being relevant (~10 per journal per year), I read more deeply and am relaying to you in this post all the ones that were somewhat on-topic and nontrivial (~0.5 per journal per year). By "nontrivial" I mean that I didn't include papers that just say "the alignment problem exists" - I certainly do not mean that I set a high bar for quality. Then I did some additional searching into what else those authors had published, who they worked with, etc.

What were all the other papers about, the ones that didn't match my criteria? All sorts of things! Whether robots are responsible for their actions, how important privacy is, how to encourage and learn from non-Western robotics paradigms, the ethics of playing MMORPGs, how to make your AI ignore protected characteristics, the impact of bow-hunting technology on middle stone age society, and so on and so forth.

The bulk of this post is a big barely-ordered list of the papers. For each paper I'll give the title, authors, author affiliations, journal and date. Each paper will get a summary and maybe a recommendation. I'll also bring up interesting author affiliations and related papers.

Whereas last post was more a "I did this so you don't have to" kind of deal, I think this post will be more fun if you explore more deeply, reading the papers that catch your interest. (Let me know if you have any trouble finding certain pdfs - most are linked on google scholar.) If you find this post to be too long, don't go through it all in one sitting - I sure didn't!

Papers

The possibility of deliberate norm-adherence in AI, Danielle Swanepoel (U. Johannesburg, SolBridge I

...
Transcript of a Twitter Discussion on EA from June 2022
by Zvi — 2022-06-06T13:50:01.164Z

Recently on Twitter, in response to seeing a contest announcement asking for criticism of EA, I offered some criticism of that contest’s announcement.

That sparked a bunch of discussion about central concepts in Effective Altruism. Those discussions ended up including Dustin Moskovitz, who showed an excellent willingness to engage and make clear how his models worked. The whole thing seems valuable enough to preserve in a form that one can navigate, hence this post.

This compiles what I consider the most important and interesting parts of that discussion into post form, so it can be more easily seen and referenced, including in the medium-to-long term.

There are a lot of offshoots and threads involved, so I’m using some editorial discretion to organize and filter.

To create as even-handed and useful a resource as possible, I am intentionally not going to interject commentary into the conversation here beyond the bare minimum.

As usual, I use screenshots for most tweets to guard against potential future deletions or suspensions, with links to key points in the threads.

(As Kevin says, I did indeed mean should there.)

At this point there are two important threads that follow, and one additional reply of note.

Thread one, which got a bit tangled at the beginning but makes sense as one thread:

Thread two, which took place the next day and went in a different direction.

Link here to Ben’s post, GiveWell and the problem of partial funding.

Link to GiveWell blog post on giving now versus later.

Dustin’s “NO WE ARE FAILING” point seemed important so I highlighted it.

There was also a reply from Eliezer.

And this on pandemics in particular.

Sarah asked about the general failure to convince Dustin’s friends.

These two notes branch off of Ben’s comment that covers-all-of-EA didn’t make sense.

Ben also disagreed with the math that there was lots of opportunity, linking to his post A Drowning Child is Hard to Find.

This thread responds to Dustin’s claim that you need to know details about the upgrade to the laptop further up the main thread, I found it worthwhile but did not include it directly for reasons of length.

This came in response to Dust...
Some ideas for follow-up projects to Redwood Research’s recent paper
by JanB — 2022-06-06T13:29:08.622Z

Disclaimer: I originally wrote this list for myself and then decided it might be worth sharing. Very unpolished, not worth reading for most people. Some (many?) of these ideas were suggested in the paper already, I don’t make any claims of novelty.

I am super excited by Redwood Research’s

new paper

, and especially by the problem of “high-stakes reliability”.

Others have explained much better why the problem of “high-stakes reliability” is a worthy one (

LINK

,

LINK

), so I’ll keep my opinion. In short, I think:

Solving “high-stakes reliability” is probably essential for building safe AGI.
You can do useful empirical research on it now already.
High-stakes reliability would be super useful for today's systems and applications; for systems that make high-stakes decisions where a single instance of a certain bad behaviour would be terrible (e.g. confidently but incorrectly classifying a medical image)
I think there is little existing work on exactly this problem, but I might be wrong about this. There is definitely a lot of work on related problems.

Here is an extremely unpolished of ideas and directions for follow-up research that I had after reading the paper, in no particular order. Some (many?) of these ideas were suggested in the paper already, I don’t make any claims of novelty.

How does this methodology fail?

Find (ideally systematic) ways to break their classifier (I tried a bit, it is actually harder than expected :-) )
How exactly are capabilities impacted by the filtering?
From Paul Christiano:

Avoiding failures where the model learns to be conservative only on “sneaky-looking prompts designed to trick me.”

Improve the classifier or its training

use uncertainty in the classifier

(if necessary via expensive ensembles)

something something out-of-distribution detection
experiment with prompting

Do a project on high-stakes reliability that does not use human oversight, in order to focus more on the ML side of things

Example project: Get a model to solve easy programming questions. The model is allowed to not submit any code to a question, and it's fine if the model doesn't submit code to most questions. However, if the model submits code, the code should solve the test cases in 99.9X% of cases, for any input (from a natural distribution).
If you work with tasks that require human oversight (like "does this snippet entail injuries?"), you

...
Give the model a model-builder
by Adam Jermyn — 2022-06-06T12:21:11.187Z

(Thanks to Evan Hubinger for suggesting this problem and for comments on these ideas. Feedback is very welcome.)

The Setup

Suppose we have a prior pgood(model) that we think is non-deceptive, such that we can sample from this prior and get good models that are inner-aligned.

These models may, for instrumental purposes, need to themselves produce models (via searches/optimization/meta-learning). For instance we might ask an outer model to play poker. Along the way the model may find it instrumentally useful to build an inner model of its opponent. We refer to these additional models as inner models, and the ones that produced them we call outer models.

A key challenge is that inner models may be deceptive even if outer models are not. In the poker example above, the inner model may in fact do a good job of predicting the opponent’s plays for purposes of deceiving both the outer model and the outer model’s overseer. Then, on deployment, the inner model could take over and pursue its own (misaligned) values.

This challenge is part of the argument that minimal circuits are deceptive. Even though minimal circuits in some sense are non-deceptive (to the extent that deception imposes a runtime performance overhead), on some tasks the fastest program is one which performs searches for policies (inner models) that themselves perform the task, and there is no reason to expect those inner models to be aligned with the outer model.

A Possible Solution

Framing

The basic worry here is that while we (by assumption) know how to search for non-deceptive models, our model is not so clever or so careful, and could well land on deceptive inner models (an “inner inner alignment” failure). In other words: we might have a good prior pgood that we use to search for non-deceptive models, but the models we find don’t know to use pgood in their own searches, and that’s dangerous.

Approach

It seems hard to reliably detect and disfavor models that run internal search using bad priors. So instead let’s encourage models to only search with good priors by giving the AI safe tools.

Details

To do this we’ll assume that pgood favors simpler and faster models over more complex or slower ones. That is, in addition to whatever else it’s doing, the good prior has some elements of the speed and simplicity priors in it.

Next, we’ll make one of the basic building blocks of...
What journaling prompts do you use?
by ChristianKl — 2022-06-06T11:35:40.058Z

When I started daily journaling, I just started writing about what happened that day. Now, I'm wanting to improve the process. Oftentimes, people seem to have questions that they answer every day as part of their journaling. That could be the classic from positive psychology "What are you grateful for today?" or "What did I learn today?” from Tyler Cowen or something different.

If you have a journaling habit, what prompts worked well for you? If you tried some prompts that didn't work well for you, I'm also curious about those.