trentmkelly/LessWrong-43k huggingface dataset:
What does it mean for an AGI to be 'safe'?
by So8res — 2022-10-07T04:13:05.176Z

(Note: This post is probably old news for most readers here, but I find myself repeating this surprisingly often in conversation, so I decided to turn it into a post.)

I don't usually go around saying that I care about AI "safety". I go around saying that I care about "alignment" (although that word is slowly sliding backwards on the semantic treadmill, and I may need a new one soon).

But people often describe me as an “AI safety” researcher to others. This seems like a mistake to me, since it's treating one part of the problem (making an AGI "safe") as though it were the whole problem, and since “AI safety” is often misunderstood as meaning “we win if we can build a useless-but-safe AGI”, or “safety means never having to take on any risks”.

Following Eliezer, I think of an AGI as "safe" if deploying it carries no more than a 50% chance of killing more than a billion people:

When I say that alignment is difficult, I mean that in practice, using the techniques we actually have, "please don't disassemble literally everyone with probability roughly 1" is an overly large ask that we are not on course to get. [...] Practically all of the difficulty is in getting to "less than certainty of killing literally everyone". Trolley problems are not an interesting subproblem in all of this; if there are any survivors, you solved alignment. At this point, I no longer care how it works, I don't care how you got there, I am cause-agnostic about whatever methodology you used, all I am looking at is prospective results, all I want is that we have justifiable cause to believe of a pivotally useful AGI 'this will not kill literally everyone'.

Notably absent from this definition is any notion of “certainty” or "proof". I doubt we're going to be able to prove much about the relevant AI systems, and pushing for proofs does not seem to me to be a particularly fruitful approach (and never has; the idea that this was a key part of MIRI’s strategy is a common misconception about MIRI).

On my models, making an AGI "safe" in this sense is a bit like finding a probabilistic circuit: if some probabilistic circuit gives you the right answer with 51% probability, then it's probably not that hard to drive the success probability significantly higher than that.

If anyone can deploy an AGI that is less than 50% likely to kill more than a billion people, then they've probably... well...
linkpost: neuro-symbolic hybrid ai
by Nathan Helm-Burger — 2022-10-06T21:52:53.095Z
Link: https://youtu.be/cWHHLYDVGEA?t=999

This video shows some really interesting work that's been done on factoring sensory perceptions (visual data, natural language) into concept space (they call it 'derendering' as an antonym to rendering), and then applying symbolic reasoning to the concepts. I think this is really potentially powerful for data efficiency and interpretability. As with many things that I and others post about, I don't think this is a notion sufficient for solving alignment on its own, but I do think human-readable symbolic reasoning is an important piece that I'd be surprised to learn that future alignment solutions didn't include.

Edit: for the tldr version: watch from 23:50 to 28:20
So, geez there's a lot of AI content these days
by Raemon — 2022-10-06T21:32:20.833Z

Since April this year, there's been a huge growth in the the number of posts about AI, while posts about rationality, world modeling, etc. have remained constant. The result is that much of the time, the LW frontpage is almost entirely AI content.

Looking at the actual numbers, we can see that during 2021, no core LessWrong tags[1] represented more than 30% of LessWrong posts. In 2022, especially starting around April, AI has started massively dominating the LW posts.

Here's the total posts for each core tag each month for the past couple years. On April 2022, most tags' popularity remains constant, but AI-tagged posts spike dramatically:

All posts with 0 or greater karma are included in this graph. When applying a higher threshold, the percentage AI tagged posts is even higher.

Even people pretty involved with AI alignment research have written to say "um, something about this feels kinda bad to me."

I'm curious to hear what various LW users think about the situation. Meanwhile, here's my own thoughts.

Is this bad?

Maybe this is fine.

My sense of what happened was that in April, Eliezer posted MIRI announces new "Death With Dignity" strategy, and a little while later AGI Ruin: A List of Lethalities. At the same time, PaLM and DALL-E 2 came out. My impression is that this threw a brick through the overton window and got a lot of people going "holy christ AGI ruin is real and scary". Everyone started thinking a lot about it, and writing up their thoughts as they oriented.

Around the same time, a lot of alignment research recruitment projects (such as SERI MATS or Refine) started paying dividends, and resulting in a new wave of people working fulltime on AGI safety.

Maybe it's just fine to have a ton of people working on the most important problem in the world?

Maybe. But it felt worrisome to Ruby and me. Some of those worries felt easier to articulate, others harder. Two major sources of concern:

There's some kind of illegible good thing that happens when you have a scene exploring a lot of different topics. It's historically been the case that LessWrong was a (relatively) diverse group of thinkers thinking about a (relatively) diverse group of things. If people show up and just see the All AI All the Time, people who might have other things to contribute may bounce off. We probably wouldn't lose this immediately

AI needs Rationality, in particular. Maybe AI is...
Analysing a 2036 Takeover Scenario
by ukc10014 — 2022-10-06T20:48:49.500Z

TL;DR

This post basically digs through Holden Karnofsky's ‘

AI Could Defeat All of Us Combined

’ (AIDC) post and tries to work through some of the claims there, in light of other work by Ajeya Cotra and Richard Ngo. It originally was a response to the latter's suggestion that someone should recast that post as a scenario, but I struggled to do that without fleshing out the argument for myself. Anyway, I try to imagine the world around 2036, when the first human-level AI shows up (per AIDC), in context of current/forecast geopolitics.

I think my (weakly-held) takeaway is that the AIDC notion of 'hundreds of millions' of AIs, nominally engaged in scientific R&D that then turn on their employer, seems to not include the likely geopolitical context and the likely messiness that makes any prediction really difficult. Specifically, in the 14 years to 2036, there would be vicious competition (which we are seeing right now, namely in the recent US semiconductor sanctions on China) over resources (raw materials and territory but also including AI-related IP and compute) between countries who might view a unipolar AI world as a near-existential threat. This strategic aspect of AI might mean research becomes highly militarised, thereby introducing specific biases into training regimes, as well as creating an obvious way for the 'AI headquarters' to be established. This background of latent or kinetic conflict may ultimately dictate what the pre-2036 environment (i.e. the transition from the time of sub-human level AI, to human-level AI that are capable of being deployed en masse) looks like, and therefore what the (assumed) hardware overhang is actually used for (commercially-valuable scientific R&D might not be as immediately and strategically useful as fleets of satellite attack drones, moon mining rigs, or ASI researchers). Moreover, I'm not sure that hundreds of millions of radically new workers that show up in ~1 year are either a) a resource management problem humans presently know how to deal with (or are likely to in 14 years), or b) are necessarily that broadly useful outside of certain sectors that have the right features to make use of large numbers of intelligent knowledge-workers. Moreover, I don't think such large numbers of AI copies are particularly critical to AIDC's conclusions (which I think are pl...
Covid 10/6/22: Overreactions Aplenty
by Zvi — 2022-10-06T19:10:00.913Z

Thanks to Balsa Research, every weekday I now get to meet and talk to someone new, usually multiple new people, all of whom are interesting, know things I don’t know that they are eager to explain to me, and who share a passion for modeling the world with an eye towards making things better rather than worse. It’s super great, and leaves me with the very Happy Problem of there not being enough hours in the day, which is only going to get worse when the Mets start playing playoff games.

I would not have it any other way.

In the meantime, Covid-19 and the world go on. Even when there isn’t anything screaming for our attention, there is always a lot going on.

The biggest thing happening right now is likely the risk that Russia will use nuclear weapons or otherwise escalate in its war against Ukraine. I briefly discuss this here, in terms of the potential risk of overreaction (or underreaction) to nuclear risks. With time constraints I have chosen not to write up my thoughts on the probabilities involved beyond what I said in the latest Ukraine update. I continue to think the risk is lower than the prediction markets, although this brings little comfort.

I am considering the extent to which these posts should be split up, and have various topics addressed on their own. Curious to hear people’s thoughts on that, and how best to think about those questions.

Let’s run the numbers.

Executive Summary

Public health Twitter is not okay, they’re overreacting.

Nuclear war risk levels are not okay, but beware overreacting.

Magic: The Gathering is not okay, but they’re always ruining that, it’s fine.

Covid numbers, on the other hand? Those seem okay.

The Numbers

Predictions

Prediction from last week: 290k cases (-9%) and 2,750 deaths (-3%).

Result: 270k cases (-15%) and 2,785 deaths (-3%).

Prediction for next week (Holiday): 221k cases (-18%) and 2,300 deaths (-17%).

North Carolina reported several hundred extra deaths. I am not manually adjusting, but am noting it and adjusting next week’s prediction for it. I’m taking off 10% both predictions for Indigenous People’s Day.

Deaths

Cases

Physical World Modeling

A master thread of Chise ‘get vaccinated’ threads.

Word is not out about the updated boosters.

I am legitimately curious how one effectively gets the word out there days.

From February: Covid burden fe...
A shot at the diamond-alignment problem
by TurnTrout — 2022-10-06T18:29:10.586Z

I think that relatively simple alignment techniques can go a long way. In particular, I want to tell a plausible-to-me story about how simple techniques can align a proto-AGI so that it makes lots of diamonds.

But why is it interesting to get an AI which makes lots of diamonds? Because we avoid the complexity of human value and thinking about what kind of future we want, while still testing our ability to align an AI. Since diamond-production is our goal for training, it’s actually okay (in this story) if the AI kills everyone. The real goal is to ensure the AI ends up acquiring and producing lots of diamonds, instead of just optimizing some weird proxies that didn’t have anything to do with diamonds. It’s also OK if the AI doesn’t maximize diamonds, and instead just makes a whole lot of diamonds.[1]

Someone recently commented that I seem much more specifically critical of outer and inner alignment, than I am specifically considering alternatives. So, I had fun writing up a very specific training story for how I think we can just solve diamond-alignment using extremely boring, non-exotic, simple techniques, like “basic reward signals & reward-data augmentation.” Yes, that’s right.

As I’ve hinted previously

, I think many arguments against this working are wrong, but I’m going to lay out a positive story in this post. I’ll reserve my arguments against certain ideas for future posts.

Can we tell a plausible story in which we train an AI, it cares about diamonds when it’s stupid, it gets smart, and still cares about diamonds? I think I can tell that story, albeit with real uncertainties which feel more like normal problems (like “ensure a certain abstraction is learned early in training”) than impossible-flavored alignment problems (like “find/train an evaluation procedure which isn’t exploitable by the superintelligence you train”).

Before the story begins:

Obviously I’m making up a lot of details, many of which will turn out to be wrong, even if the broad story would work. I think it’s important to make up details to be concrete, highlight the frontiers of my ignorance, and expose new insights. Just remember what I’m doing here: making up plausible-sounding details.

This story did not actually happen in reality

. It’s fine, though, to update towards my models if you find them compelling.
This is not my best guess for how we

...
American invention from the “heroic age” to the system-building era
by jasoncrawford — 2022-10-06T17:19:37.250Z
Link: https://rootsofprogress.org/american-genesis-part-1

Part 1 of my review of American Genesis

The most productive decades in American history were roughly 1870–1940: the era that saw the birth and growth of the electric light and power industry; the invention of the automobile and airplane, and the rise of the oil industry that fueled them; dramatic increases in manufacturing efficiency from innovations such as the assembly line; and the invention and distribution of telephone and radio—to name just a few of the major developments. (Robert Gordon’s Rise and Fall of American Growth illustrates and quantifies these advances, showing how progress was faster in this period than in the decades since.)

This period opened with what has been called the “heroic age” of independent inventors such as Edison, Bell, and the Wrights. After World War 1, industrial progress was driven more by large corporations and research labs. Several fascinating stories from both eras, and about the transition, are told in American Genesis: A Century of Invention and Technological Enthusiasm, 1870–1970, by Thomas P. Hughes, a finalist for the 1990 Pulitzer. I’m going to review it in multiple parts, although each part should stand alone.

The central focus of American Genesis is large systems of production—systems based on technology, but also on management, organization, and control. The first half of the book describes how these systems were created. In this, part 1 of my review, I’ll recount some of those stories.

Science and the independent inventors

The book opens with stories of the “independent inventors” such as Thomas Edison, Alexander Graham Bell, the Wright Brothers, Nikola Tesla, Elmer Sperry, Lee de Forest, Reginald Fessenden, Hiram Maxim, and Elihu Thompson—independent not in the sense that they worked alone, since each had a lab and employed help, but in the sense that they directed their own work and did not report into any corporate overseer.

Most interesting to me was the detail on the relation between science and “tinkering,” a topic I’ve discussed before. Edison, the most famous American inventor, is also the one most derided for taking a “hunt-and-try” approach, ignoring the guidance of scientific theory. But:

Those who then portrayed Edison, the American hero, as a plain and pragmatic hunt-and-try inventor unencumbered by science and organized knowledge would have been surprised to learn of the emphasis he gave to a library. H

...
More Recent Progress in the Theory of Neural Networks
by jylin04 — 2022-10-06T16:57:10.023Z

Thanks to Dan Roberts and Sho Yaida for comments on a draft of this post.

In this post, I would like to draw attention to the book Principles of Deep Learning Theory (PDLT), which I think represents a significant advance in our understanding of how neural networks work [1]. Among other things, this book explains how to write a closed-form formula for the function learned by a realistic, finite-width neural network at the end of training [2] to an order of approximation that suffices to describe representation learning, and how that formula can be interpreted as the solution to a regression model. This makes manifest the intuition that NNs are doing something like regression, but where they learn the features appropriate for a given dataset rather than having them be hand-engineered from the start.

I've condensed some main points from the 400-page book into an 8-page summary here:

Review of select results from PDLT

(Other good places to learn about the book, though perhaps with less of a focus on AI-safety-relevant parts, include this series of five lectures given by the authors at a deep learning summer school or this one-hour lecture for a non-expert audience.)

For those who have been following the discussions of ML theory on this forum, the method used in the book is to go to the next-to-leading order in a 1/width expansion. It thus builds on recent studies of infinitely wide NNs that were reviewed in the AF post Recent Progress in the Theory of Neural Networks [3]. However, by going beyond the leading order, the authors of PDLT are able to get around a key qualitative shortcoming of the earlier work in that infinitely wide NNs can't learn features. The next-to-leading order formula also introduces a sum over many steps of gradient descent, getting around an objection [4] that the NTK/infinite width limit may not be applicable to realistic models since in that limit, we can land on the fully trained model after just one fine-tuned training step.

I think that this work could have significant implications for AGI forecasting and safety (via interpretability), and deserves to be better appreciated in this community. For example,

In AGI forecasting, an important open question is whether the strong scaling hypothesis holds for any modern architectures. (For example, the forecasts in Ajeya Cotra's Bio-Anchors r

...
Research Deprioritizing External Communication
by jefftk — 2022-10-06T12:20:04.642Z

This was originally written in December 2021. I think it's good
practice to run posts by orgs before making them public, and I did in
this case. This has some benefits: orgs aren't surprised, they can
prepare a response in advance if they want to, they can point out
errors before things become public, etc, and I think it's generally
worth doing. In this case Leverage folks pointed out some errors,
omissions, and bad phrasing in my post, which I've fixed, and I'm
thankful for their help. Pre-publication review does also have
downsides, however, and in this case as the email conversations grew
to 10k+ words over three weeks I ran out of time and motivation.

A month ago I came across this in my list of blog drafts and
decided to publish it as-is with a note at the top explaining the
situation. This means that it doesn't cover any more recent Leverage
developments, including their Experiences
Inquiry Report and On
Intention Research paper, both published in April 2022. I shared
this post again with Leverage, and while I've made edits in response
to their feedback they continue to disagree with my conclusion.

In the original pre-publication discussion with Geoff, one of the
topics was whether we could make our disagreement more concrete with a
bet. For example, research that launches a new subfield generally
gets lots of citations, such as the the Concrete Problems paper
(1,644
citations at 6 years), and if Leverage 1.0's research ends up
having this kind of foundational impact this could be a clear way to
tell. When I gave Leverage a second pre-publication heads up, Geoff
and I talked more and we were able to nail down some terms: if a
Leverage paper drawing primarily on their pre-2019 research has 100+
citations from people who've never worked for Leverage by 2032-10-01,
then I'll donate $100 to a charity of Geoff's choosing; if not
then Geoff will do the same for a charity of my choosing. I've
listed this on my bets page.

In 2011, several people I knew through Boston-area effective
altruism and rationality meetups
started an organization called Leverage Research. Their
initial
goal was to "make the world a much better place, using the most
effective means we can", and they worked on a wide range of projects,
but they're probably best known for trying to figure out how to make
people more productive/capable/successful by better und...
Warning Shots Probably Wouldn't Change The Picture Much
by So8res — 2022-10-06T05:15:39.391Z

One piece of advice I gave to EAs of various stripes in early 2021 was: do everything you can to make the government sane around biorisk, in the wake of the COVID pandemic, because this is a practice-run for AI.

I said things like: if you can't get the world to coordinate on banning gain-of-function research, in the wake of a trillions-of-dollars tens-of-millions-of-lives pandemic "warning shot", then you're not going to get coordination in the much harder case of AI research.

Biolabs are often publicly funded (rather than industry-funded). The economic forces arrayed behind this recklessly foolish and impotent research consists of “half-a-dozen researchers thinking it’s cool and might be helpful”. (While the work that would actually be helpful—such as removing needless bureaucracy around vaccines and investing in vaccine infrastructure—languishes.) Compared to the problem of AI—where the economic forces arrayed in favor of “ignore safety and rush ahead” are enormous and the argument for expecting catastrophe much murkier and more abstract—the problem of getting a sane civilizational response to pandemics (in the wake of a literal pandemic!) is ridiculously easier.

And—despite valiant effort!—we've been able to do approximately nothing.

We're not anywhere near global bans on gain-of-function research (or equivalent but better feats of coordination that the people who actually know what they're talking about when it comes to biorisk would tell you are better targets than gain-of-function research).

The government continues to fund research that is actively making things worse, while failing to put any serious funding towards the stuff that might actually help.

I think this sort of evidence has updated a variety of people towards my position. I think that a variety of others have not updated. As I understand the counter-arguments (from a few different conversations), there are two main reasons that people see this evidence and continue to hold out hope for sane government response:

1. Perhaps the sorts of government interventions needed to make AI go well are not all that large, and not that precise.

I confess I don't really understand this view. Perhaps the idea is that AI is likely to go well by default, and all the government needs to do is, like, not use anti-trust law to break up some corporation that's doing a really good job at AI alignment just before th...