trentmkelly/LessWrong-43k huggingface dataset:
Neural uncertainty estimation review article (for alignment)
by Charlie Steiner — 2023-12-05T08:01:32.723Z

EDIT 1/27: This post neglects the entire sub-field of estimating uncertainty of learned representations, as in https://openreview.net/pdf?id=e9n4JjkmXZ. I might give that a separate follow-up post.

Introduction

Suppose you've built some AI model of human values. You input a situation, and it spits out a goodness rating. You might want to ask: "What are the error bars on this goodness rating?" In addition to it just being nice to know error bars, an uncertainty estimate can also be useful inside the AI: guiding active learning[1], correcting for the optimizer's curse[2], or doing out-of-distribution detection[3].

I recently got into the uncertainty estimation literature for neural networks (NNs) for a pet reason: I think it would be useful for alignment to quantify the domain of validity of an AI's latent features. If we point an AI at some concept in its world-model, optimizing for realizations of that concept can go wrong by pushing that concept outside its domain of validity.

But just keep thoughts of alignment in your back pocket for now. This post is primarily a survey of the uncertainty estimation literature, interspersed with my own takes.

The Bayesian neural network picture

The Bayesian NN picture is the great granddaddy of basically every uncertainty estimation method for NNs, so it's appropriate to start here.

The picture is simple. You start with a prior distribution over parameters. Your training data is evidence, and after training on it you get an updated distribution over parameters. Given an input, you calculate a distribution over outputs by propagating the input through the Bayesian neural network.

This would all be very proper and irrelevant ("Sure, let me just update my 2trilliondimensional joint distribution over all the parameters of the model"), except for the fact that actually training NNs does kind of work this way. If you use a log likelihood loss and L2 regularization, the parameters that minimize loss will be at the peak of the distribution that a Bayesian NN would have, if your prior on the parameters was a Gaussian[4][5].

This is because of a bridge between the loss landscape and parameter uncertainty. Bayes's rule says P(parameters|dataset)=P(parameters)⋅P(dataset|parameters)/P(dataset). Here P(parameters|dataset)is your posterior distribution you want to estimate, and P(parameters)⋅P(dataset|paramet...
Analyzing the Historical Rate of Catastrophes
by jsteinhardt — 2023-12-05T06:30:01.757Z

To communicate risks, we often turn to stories. Nuclear weapons conjure stories of mutually assured destruction, briefcases with red buttons, and nuclear winter. Climate change conjures stories of extreme weather, cities overtaken by rising sea levels, and crop failures. Pandemics require little imagination after COVID, but were previously the subject of movies like Contagion.

Stories are great for conveying concrete risks (I myself recently did this for AI risks), but they’re a bad way to predict the future. That’s because most stories are far too specific to be probable. More importantly, stories tend to feature short, simple chains of causation while reality is complex and multi-causal.

Instead of using stories, most competitive forecasters start their forecasts by looking at historical reference classes. This works really well, and also makes sense: history pulls us away from the biases of story-telling by grounding in events that actually occurred. While history is filtered through narratives, a good history will contend with the complexity of reality, and we can further strip away narrative by grounding in raw numbers.[1]

In this post, I’ll use reference classes to understand the largest risks society faces today. I’ll do this by considering two different reference classes for historical catastrophes:

Events that killed a significant fraction of the global human population (Section 1)

Extinctions of species, and especially mass extinction events (Section 2)

Looking at these reference classes teaches us two things. First, it gives us a numerical estimate of how rare different catastrophes are. If we define a catastrophe as an event killing 1% of the global population within a decade, then 11 such catastrophes have occurred since 1500, for a base rate of 2% per year. If we raise the bar to killing 10% of the population, the base rate drops by an order of magnitude, to 0.2%.

History also gives us qualitative insights. For instance, all the catastrophes in the previous paragraph were epidemics, wars, or famines. Further, many events were multi-causal—the worst epidemics occurred when populations were already weakened by famine, and many epidemics and famines were precipitated by changes in climate or by political turmoil. Species extinctions are also multi-causal, and the common culprits are climate change, natural disasters, invasive species, and humans.

On...
Some open-source dictionaries and dictionary learning infrastructure
by Sam Marks — 2023-12-05T06:05:21.903Z

As more people begin work on interpretability projects which incorporate dictionary learning, it will be valuable to have high-quality dictionaries publicly available.[1] To get the ball rolling on this, my collaborator (Aaron Mueller) and I are:

open-sourcing a number of sparse autoencoder dictionaries trained on Pythia-70m MLPs
releasing our repository for training these dictionaries[2].

Let's discuss the dictionaries first, and then the repo.

The dictionaries

[EDIT 02/07/2024: Better dictionaries are now available at the repo. Also, the originally reported MSE loss numbers were wrong and been updated in the tables below. (The correct numbers were much lower, i.e. better.)]

The dictionaries can be downloaded from here. See the sections "Downloading our open-source dictionaries" and "Using trained dictionaries" here for information about how to download and use them. If you use these dictionaries in a published paper, we ask that you mention us in the acknowledgements.

We're releasing two sets of dictionaries for EleutherAI's 6-layer pythia-70m-deduped model. The dictionaries in both sets were trained on 512-dimensional MLP output activations (not the MLP hidden layer like Anthropic used), using ~800M tokens from The Pile.

The first set, called 0_8192, consists of dictionaries of size 8192=16×512. These were trained with an L1 penalty of 1e-3.
The second set, called 1_32768, consists of dictionaries of size 32768=64×512. These were trained with an l1 penalty of 3e-3.

Here are some statistics. (See our repo's readme for more info on what these statistics mean.)

For dictionaries in the 0_8192 set:

Layer	MSE Loss	L1 loss	L0	% Alive	% Loss Recovered
0	0.003	6.132	9.951	0.998	0.984
1	0.008	6.677	44.739	0.887	0.924
2	0.011	11.44	62.156	0.587	0.867
3	0.018	23.773	175.303	0.588	0.902
4	0.022	27.084	174.07	0.806	0.927
5	0.032	47.126	235.05	0.672	0.972

For dictionaries in the 1_32768 set:

Layer	MSE Loss	L1 loss	L0	% Alive	% Loss Recovered
0	0.0018	4.32	2.873	0.174	0.946
1	0.017	2.798	11.256	0.159	0.768
2	0.023	6.151	16.381	0.118	0.724
3	0.044	11.571	39.863	0.226	0.765
4	0.048	13.665	29.235	0.19	0.816
5	0.069	26.4	43.846	0.13	0.931

And here are some histograms of feature frequencies.

Overall, I'd guess that these dictionaries are decent, but not amazing.

We trained these dictionaries because we wanted to work on a downstream application of dictionary learning, but lacked the dictionaries. These dictionaries are more than good enough to get us off the...
The LessWrong 2022 Review
by habryka — 2023-12-05T04:00:00.000Z

The snow is falling, the carols are starting, and we all know it's time for our favorite winter holiday tradition. It's LessWrong review time!

Each year we come together and review posts that are at least one year old. That means for the next two months we are reviewing all posts from 2022.

While our everyday lives are filled with fads and chasing the sweet taste of karma and social approval, the LessWrong review is the time to take a step back and ask ourselves "did this actually help me think better?", "did this actually turn out to be valuable?" and "which things withstood further and extended scrutiny?".

We've done this 4 times so far (2018, 2019, 2020, 2021).

The full technical details of how the Annual Review works are in the final section of this post, but it's basically the same as the past few years. There are three phases:

Preliminary Voting Phase (2 weeks, Dec 4 — 17): We identify posts especially worthy of consideration in the review casting preliminary votes. Posts with 2 preliminary votes move into the Discussion Phase.
Discussion Phase (4 weeks, Dec 17 — Jan 14): We review and debate posts. Posts that receive at least one written review move to the final voting phase.
Final Voting (2 weeks, Jan 14 — Jan 28): We do a full voting pass, using quadratic voting. The outcome determines the Annual Review results.

For more of the philosophy of the Annual Review, see the previous announcement posts here, here, here, and here.

Getting Started

At the top of any posts eligible for the review, you will see this:

These will be your preliminary votes for the 2022 review. Posts need to get at least 2 preliminary votes (positive or negative) in order to move to the next phase of the review.

To start perusing posts, I recommend going to the

All 2022 Posts page

, or the

View Your Past Upvotes

page. Note: only users with accounts registered before January 2022 are eligible to vote.

No books this year, sorry folks

For 2018, 2019, and 2020 we printed books of the results of the review. We have sold many thousands of them, I am very proud of them, and many people told me that these are among the favorite things that they own:

2018: A Map that Reflects the Territory (Amazon)

2019: The Engines of Cognition (Amazon)

2020: The Carving of Reality (Amazon)

Sadly, there won't be a book this year (and also not of the 2021 review). The effort involved in making them is hard to just...
Bands And Low-stakes Dances
by jefftk — 2023-12-05T03:50:22.076Z

When I got started playing for contra dances in the Boston area there
were several area dances you could consider "low stakes". Small
dances, couldn't pay very much, less enthusiasm from established
bands. And so they were relatively willing to book bands that were
starting out and friendly, encouraging, and tolerant of the
mistakes of inexperience.

I think this was a big part of why so many solid bands and dance
musicians have come out of this area: no requirement that you be good
enough to hold your own at a "big" dance to start getting experience
playing for dancing as a band. For example, looking back at my calendar, the Free Raisins played eighteen
different evenings before we played our first "big" one (Concord Friday).

Unfortunately this is much less of a thing than it used to be.
Several
closed pre-pandemic (I miss the MIT contra dance so much!) and others
haven't come back (yet?) or have switched to house bands. There's the
BIDA open band and
(even lower stakes) family dance band, but while I think this kind of
experience is also valuable there's a different kind of learning that
comes from playing in a small group, playing sets you've worked up,
and being fully responsible for the music.

I don't have a great solution here: it's hard to start something
intentionally low-stakes and people generally would prefer to run the
kind of event that lots of people want to attend and has great
music. But I do think the environment of the early 2010s with many
opportunities for new bands to learn to be dance bands was pretty
special, and I'd love it if we could bring back something similar, or
at least something with similar effects.

(This same dynamic exists even more strongly for "techno contra": the
only opportunities to play are major special events, even more
high-profile! But I also am less invested in the success of this
dance form, so this bothers me less.)
Accelerating science through evolvable institutions
by jasoncrawford — 2023-12-04T23:21:35.330Z
Link: https://rootsofprogress.org/accelerating-science-through-evolvable-institutions

This is the written version of a talk presented to the Santa Fe Institute at a working group on “Accelerating Science.”

We’re here to discuss “accelerating science.” I like to start on topics like this by taking the historical view: When (if ever) has science accelerated in the past? Is it still accelerating now? And what can we learn from that?

I’ll submit that science, and more generally human knowledge, has been accelerating, for basically all of human history. I can’t prove this yet (and I’m only about 90% sure of it myself), but let me appeal to your intuition:

Behaviorally modern humans are over 50,000 years old
Writing is only about 5,000 years old, so for more than 90% of the human timeline, we could only accumulate as much knowledge as could fit in an oral tradition
In the ancient and medieval world, we had only a handful of sciences: astronomy, geometry, some number theory, some optics, some anatomy
In the centuries after the Scientific Revolution (roughly 1500s–1700s), we got the heliocentric theory, the laws of motion, the theory of gravitation, the beginnings of chemistry, the discovery of the cell, better theories of optics
In the 1800s, things really got going, and we got electromagnetism, the atomic theory, the theory of evolution, the germ theory
In the 1900s things continued strong, with nuclear physics, quantum physics, relativity, molecular biology, and genetics

I’m leaving aside the question of whether science has slowed down since ~1950 or so, which I don’t have a strong opinion on. Even if it has, that’s mostly a minor, recent blip in the overall pattern of acceleration across the broad sweep of history. (Or, you know, the beginning of a historically unprecedented reversal and decline. One or the other.)

Part of the reason I’m pretty convinced of this accelerating pattern is that it’s not just science that is accelerating: pretty much all measures of human advancement show the same trend, including world GDP and world population.

What drives acceleration in science? Many factors, including:

Funding. Once, scientists had to seek patronage, or be independently wealthy. Now there is grant money available, and the total amount of funding has increased significantly in the last several decades:

American Association for the Advancement of Science

People. More scientists (all else being equal) means science moves faster, and the number of scientists has gr

...
Speaking to Congressional staffers about AI risk
by Orpheus16 — 2023-12-04T23:08:52.055Z

In May and June of 2023, I (Orpheus) had about 50-70 meetings about AI risks with congressional staffers. I had been meaning to write a post reflecting on the experience and some of my takeaways, and I figured it could be a good topic for a LessWrong dialogue. I saw that hath had offered to do LW dialogues with folks, and I reached out.

In this dialogue, we discuss how I decided to chat with staffers, my initial observations in DC, some context about how Congressional offices work, what my meetings looked like, lessons I learned, and some miscellaneous takes about my experience.

Context

hath

Hey! In your message, you mentioned a few topics that relate to your time in DC.

I figured we should start with your experience talking to congressional offices about AI risk. I'm quite interested in learning more; there don't seem to be many public resources on what that kind of outreach looks like.

How'd that start? What made you want to do that?

Orpheus16

In March of 2023, I started working on some AI governance projects at the Center for AI Safety. One of my projects involved helping CAIS respond to a Request for Comments about AI Accountability that was released by the NTIA.

As part of that work, I started thinking a lot about policy ideas for frontier AI systems. For instance: if I could set up a body to oversee frontier AI systems or assign certain tasks to existing agencies, what would that like? Where in the US government would it be housed? What information would I want it to assess?

I began to wonder how actual policymakers would react to these ideas. I was also curious to know more about how policymakers were thinking about AI extinction risks and catastrophic risks.

I started asking other folks in AI Governance. The vast majority had not talked to congressional staffers (at all). A few had experience talking to staffers but had not talked to them about AI risk. A lot of people told me that they thought engagement with policymakers was really important but very neglected. And of course, there are downside risks, so you don't want someone doing it poorly.

After consulting something like 10-20 AI governance folks, I asked CAIS if I could go to DC and start talking to congressional offices. The goals were to (a) raise awareness about AI risks, (b) get a better sense of how congressional offices were thinking about AI risks, (c) get a better sense of

...
Open Thread – Winter 2023/2024
by habryka — 2023-12-04T22:59:49.957Z

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you're new to the community, you can start reading the Highlights from the Sequences, a collection of posts about the core ideas of LessWrong.

If you want to explore the community more, I recommend reading the Library, checking recent Curated posts, seeing if there are any meetups in your area, and checking out the Getting Started section of the LessWrong FAQ. If you want to orient to the content on the site, you can also check out the Concepts section.

The Open Thread tag is here. The Open Thread sequence is here.
Interview with Vanessa Kosoy on the Value of Theoretical Research for AI
by WillPetillo — 2023-12-04T22:58:40.005Z

Below is a transcript (edited for grammar) of a video interview I had with Vanessa Kosoy, cross-posted from my blog. It is intended as a (relatively) beginner-friendly explanation of the goals of the Learning Theoretic Agenda and why more theoretical work is needed for ensuring AI that is safe and reliable at scale.

Short Videos (highlights):

The current path to AI is a disaster
You cannot extrapolate from empirical research without an underlying theory
We won't have obvious evidence of x-risk until it is too late
The analogy between AI and climate change

Intro (by Will Petillo): Discussing the future of AI tends to get philosophical. What does it mean to have "goals" or "understanding"? Is power-seeking a default consequence of wanting things...or a human quirk resulting from our unique evolutionary history? What motivates kindness? Framing questions in this way makes them accessible, allowing everyone to participate in the conversation. But this lack of precision also makes such questions intractable, as disagreements become a clash of intuitions.

Today's guest on "Guardians of Alignment" is Vanessa Kosoy, an independent researcher supported by the Machine Intelligence Research Institute (MIRI) and the Long-Term Future Fund (LTFF), on building mathematical theories for safe AI. This focus on understanding from first principles puts her work in stark contrast to the "move fast and break things" experimental approach of leading AI labs. In this interview and elsewhere, Vanessa defends the value of a more theory-based approach and explains what it means to explore machine learning as a basic science.

Will Petillo: How did you get into AI safety?

Vanessa Kosoy: I've always been an autodidact, so I kind of just tend to learn things by myself. When I was little, I thought I would be a theoretical physicist. I actually have a bachelor's in math, but after completing a bachelor's, instead of going to academia, I decided to go for a career in the industry instead, in software.

I had a long career in the software industry, specifically in algorithm engineering, mostly computer vision, various roles, algorithm engineer, team leader, R&D manager. I also had my own startup. I was a consultant and then something like 10 years ago. I got exposed to this whole topic of existential risks from AI and started thinking that, well, that actually seem...
2023 Alignment Research Updates from FAR AI
by AdamGleave — 2023-12-04T22:32:19.842Z
Link: https://far.ai/post/2023-12-far-research-update/

TL;DR: FAR AI's science of robustness agenda has found vulnerabilities in superhuman Go systems; our value alignment research has developed more sample-efficient value learning algorithms; and our model evaluation direction has developed a variety of new black-box and white-box evaluation methods.

FAR AI is a non-profit AI safety research institute, working to incubate a diverse portfolio of research agendas. We’ve been growing rapidly and are excited to share some highlights from our research projects since we were founded just over a year ago. We’ve also been busy running field-building events and setting up a coworking space – see our overview post for more information on our non-research activities.

Our Mission

We need safety techniques that can provide demonstrable guarantees of the safety of advanced AI systems. Unfortunately, currently deployed alignment methods like Reinforcement Learning from Human Feedback (RLHF) fall short of this standard. Proposals that could provide stronger safety guarantees exist but are in the very early stages of development.

Our mission is to incubate and accelerate these early-stage approaches, so they can be empirically tested and deployed. We focus on research agendas that are too large to be pursued by individual academic or independent researchers but are too early-stage to be of interest to most for-profit organizations.

We take bets on a range of these promising early-stage agendas and then scale up those that prove most successful. Unlike other research organizations that take bets on specific agendas, our structure allows us to both (1) explore a range of agendas and (2) execute them at scale. Our current bets fall into three categories:

Science of Robustness: How does robustness vary with model size? Will superhuman systems be vulnerable to adversarial examples or “jailbreaks” similar to those seen today? And, if so, how can we achieve safety-critical guarantees?

Value Alignment: How can we learn reliable reward functions from human data? Our research focuses on enabling higher bandwidth, more sample-efficient methods for users to communicate preferences for AI systems; and improved methods to enable training with human feedback.

Model Evaluation: How can we evaluate and test the safety-relevant properties of state-of-the-art models? Evaluation can be split into black-box approaches that focus only on externally visible ...