trentmkelly/LessWrong-43k huggingface dataset:
Let’s think about slowing down AI
by KatjaGrace — 2022-12-22T17:40:04.787Z

Averting doom by not building the doom machine

If you fear that someone will build a machine that will seize control of the world and annihilate humanity, then one kind of response is to try to build further machines that will seize control of the world even earlier without destroying it, forestalling the ruinous machine’s conquest. An alternative or complementary kind of response is to try to avert such machines being built at all, at least while the degree of their apocalyptic tendencies is ambiguous.

The latter approach seems to me like the kind of basic and obvious thing worthy of at least consideration, and also in its favor, fits nicely in the genre ‘stuff that it isn’t that hard to imagine happening in the real world’. Yet my impression is that for people worried about extinction risk from artificial intelligence, strategies under the heading ‘actively slow down AI progress’ have historically been dismissed and ignored (though ‘don’t actively speed up AI progress’ is popular).

The conversation near me over the years has felt a bit like this:

Some people: AI might kill everyone. We should design a godlike super-AI of perfect goodness to prevent that.

Others: wow that sounds extremely ambitious

Some people: yeah but it’s very important and also we are extremely smart so idk it could work

[Work on it for a decade and a half]

Some people: ok that’s pretty hard, we give up

Others: oh huh shouldn’t we maybe try to stop the building of this dangerous AI?

Some people: hmm, that would involve coordinating numerous people—we may be arrogant enough to think that we might build a god-machine that can take over the world and remake it as a paradise, but we aren’t delusional

This seems like an error to me. (And lately, to a bunch of other people.)

I don’t have a strong view on whether anything in the space of ‘try to slow down some AI research’ should be done. But I think a) the naive first-pass guess should be a strong ‘probably’, and b) a decent amount of thinking should happen before writing off everything in this large space of interventions. Whereas customarily the tentative answer seems to be, ‘of course not’ and then the topic seems to be avoided for further thinking. (At least in my experience—the AI safety community is large, and for most things I say here, different experiences are probably had in different bits of ...
Some Notes on the mathematics of Toy Autoencoding Problems
by carboniferous_umbraculum  — 2022-12-22T17:21:25.371Z

Anthropic's recent mechanistic interpretability paper, Toy Models of Superposition, helps to demonstrate the conceptual richness of very small feedforward neural networks. Even when being trained on synthetic, hand-coded data to reconstruct a very straightforward function (the identity map), there appears to be non-trivial mathematics at play and the analysis of these small networks seems to providing an interesting playground for mechanistic interpretability.
While trying to understand their work and train my own toy models, I ended up making various notes on the underlying mathematics. This post is a slightly neatened-up version of those notes, but is still quite rough and un-edited and is a far-from-optimal presentation of the material. In particular, these notes may contain errors, which are my responsibility.

1. Directly Analyzing the Critical Points of a Linear Toy Model

Throughout we will be considering feedforward neural networks with one hidden layer. The input and output layers will be of the same size and the hidden layer is smaller. We will only be considering the autoencoding problem, which means that our networks are being trained to reconstruct the data. The first couple of subsections here are largely taken from the Appendix to the paper "Neural networks and principal component analysis: Learning from examples without local minima." by Pierre Baldi and Kurt Hornik. (Neural networks 2.1 (1989): 53-58).
Consider to begin with a completely liner model., i.e. one without any activation functions or biases. Suppose the input and output layers have D neurons and that the middle layer has d<D neurons. This means that the function that the model is implementing is of the form x↦ABx, where x∈RD, B is a d×D matrix, and A is a D×d matrix. That is, the matrix B contains the weights of the connections between the input layer and the hidden layer, and the matrix A is the weights of the connections between the hidden layer and the output layer. It is important to realise that even though - for a given set of weights - the function that is being implemented here is linear, the mathematics of this model and the dynamics of the training are not completely linear.

The error on a given input x will be measured by ∥∥x−ABx∥∥2 and on the data set&...
December 2022 updates and fundraising
by AI Impacts — 2022-12-22T17:20:05.382Z

Harlan Stewart and Katja Grace*, 22 December, 2022

News

New Hires and role changes

In 2022, the AI Impacts team has grown from two to seven full time staff. Out of more than 250 applicants, we hired Elizabeth Santos as Operations Lead, Harlan Stewart as Research Assistant, and three Research Analysts: Zach Stein-Perlman, Aysja Johnson, and (are in the process of hiring) Jeffrey Heninger. We’re excited to have them all, and you can learn more about them on our about page.

Rick and Katja have traded some responsibilities: Rick is now Director of AI Impacts, and Katja is Lead Researcher. This means Rick is generally in charge of making decisions about running the org, though Katja has veto power. Katja is responsible for doing research, as well as directing and overseeing it.

Summer Internship Program

We ran an internship program during the summer. Between May and September, six interns worked on various research projects on topics such as international coordination, explanations of historic human success, case studies in risk mitigation, R&D funding in AI, our new survey of Machine Learning researchers, current AI capabilities, technologies that are strategically-relevant to AI, and the scale of machine learning models.

AI Impacts Wiki

We intend to replace our pages with an AI Impacts Wiki. Our pages have always been functionally something like a wiki, so hopefully this new format will make it clearer how to interact with them (as distinct from our blog posts), as well as easier to navigate for readers and easier to update for researchers. The AI Impacts Wiki will launch soon and can be previewed here. . We’ll say more about other minor changes when we launch it, but AI Impacts’ past and future public research will be either detailed on the wiki or findable through the wiki. You can let us know what you think using our feedback form as well as comments on this blog post.

Research

Finished this year

This year, our main new pages and research-heavy blog posts are:

A survey of 738 machine learning experts, about progress in AI. This survey was a rerun of the one conducted by AI Impacts in 2016, and a blog post on the tentative conclusions (Katja and Zach in collaboration with Ben Weinstein-Raun)

Detailed arguments answering the question, ‘Will Superhuman AI be created?’ with a tentative ‘yes’ (Katja)

Review of US pub

...
Covid 12/22/22: Reevaluating Past Options
by Zvi — 2022-12-22T16:50:01.236Z

This is the non-China section of this week’s update. Despite there being essentially zero good statistical data about the real situation in China, things are clearly moving fast, and I have assembled what useful information I have been able to find. If cases are rising as quickly as some reports suggest, their wave could crest within a matter of weeks and life be back to normal within a few months.

Back here in America, things are quiet. It seems like it was a good week to look back, as MR did, upon our options regarding focused protection, and what we can take away from that.

Executive Summary

China fully lets it rip, see post on that for more information.

Protecting the vulnerable mostly is about doing generally correct things.

Numbers rise modestly less than expected, limiting worst case risk a bit.

Let’s run the numbers.

The Numbers

Predictions

Note that last week had some case count glitches in Florida and Georgia that I didn’t catch; I’ve adjusted both in opposite directions retroactively.

Predictions from Last Week: 435k cases (+10%) and 2,900 deaths (+10%)

Results: 427k cases (+9%) and 2,679 deaths (+3%).

Predictions for Next Week (Christmas): 410k cases (-4%) and 2,500 deaths (-7%).

Deaths number was a pleasant surprise that limits the worst case scenarios a bit. The case number was in line with expectations, especially after the FL/GA error fixes. Note that the prediction for next week is based on it being a holiday, I still expect a modest rise in actual case numbers for another few weeks.

Deaths

Cases

Physical World Modeling

CDC recommendations have always been the ultimate motte and bailey. Now a judge has made this fact text, allowing any public health department to mandate any recommendation of the CDC without the ability to challenge it.

Given the tendencies of people in public health, if they have the ability to mandate something without being challenged, there is a very good chance they are going to do exactly that.

The problem is we have no category or word for the thing that we want to exist here, which is ‘CDC says this is what it thinks would be good for an individual to do, while noticing that mandating this would be deeply foolish.’ The word we use in English has been taken and now means something else, and I would expect this to happen to any chosen future words as well.

Bob Wachter t...
China Covid #4
by Zvi — 2022-12-22T16:30:00.919Z

China has now decided to let Covid take its course, so it makes sense to spin developments there off into their own posts for a bit. It is hard to get a good read on what is happening, because China’s core strategy was to stop controlling the spread of Covid and instead to put tight controls on its Covid numbers.

You can’t do this while maintaining any credibility. China decided not to let that be a problem.

Please Disregard the Following

The official case numbers in China are now complete nonsense.

Please disregard the following, except to note the need to disregard it after its peak:

You have to love Reuters reporting ‘WHO says China may be struggling to keep a tally of Covid.’ There is no ‘may’ here, nor is there is a struggle. China is choosing not to keep a tally. Or at least, choosing not share it with the rest of us. If China wanted a count, they’d have a count.

China’s official seven day rolling average of deaths at the moment, according to OurWorldInData? One.

Reuters has an article asking how accurate these numbers are, and the correct response is that it is a headline with a question and the answer is simply ‘no,’ and elsewhere they estimate more than 5,000 deaths per day.

Bloomberg agrees and also estimates over a million daily cases, saying this could get ‘even worse’ and rise to 3.7 million cases per day in January. This is the usual ‘oh no exponential growth could get worse, we could have one or even two doublings ahead of us’ style projection, and I’d be worried about numbers closer to 10 million per day, perhaps even somewhat higher.

So how is China reporting one death? Other than ‘with a straight face’?

They did that the same way they solved their reported infections problem.

It’s simple. They’re redefining Covid deaths the same way they redefined cases.

I interpret this as ‘we will look for any way we can to attribute your death to something else, regardless of whether you had Covid at the time.’

Thus:

Letting it Rip

We went very quickly from Zero Covid to Let It Rip.

China’s public health officials say that possibly 800 million people could be infected with the coronavirus over the next few months. And several models predict that a half million people could die, possibly more.

…

“Recently, the deputy director of China CDC, Xiaofeng Liang, who’ s a good friend of mine, was announcing throu

...
Racing through a minefield: the AI deployment problem
by HoldenKarnofsky — 2022-12-22T16:10:07.694Z

Click lower right to download or find on Apple Podcasts, Spotify, Stitcher, etc.

In previous pieces, I argued that there's a real and large risk of AI systems' developing dangerous goals of their own and defeating all of humanity - at least in the absence of specific efforts to prevent this from happening. I discussed why it could be hard to build AI systems without this risk and how it might be doable.

The “AI alignment problem” refers1 to a technical problem: how can we design a powerful AI system that behaves as intended, rather than forming its own dangerous aims? This post is going to outline a broader political/strategic problem, the “deployment problem”: if you’re someone who might be on the cusp of developing extremely powerful (and maybe dangerous) AI systems, what should you … do?

The basic challenge is this:

If you race forward with building and using powerful AI systems as fast as possible, you might cause a global catastrophe (see links above).

If you move too slowly, though, you might just be waiting around for someone else less cautious to develop and deploy powerful, dangerous AI systems.

And if you can get to the point where your own systems are both powerful and safe … what then? Other people still might be less-cautiously building dangerous ones - what should we do about that?

My current analogy for the deployment problem is racing through a minefield: each player is hoping to be ahead of others, but anyone moving too quickly can cause a disaster. (In this minefield, a single mine is big enough to endanger all the racers.)

This post gives a high-level overview of how I see the kinds of developments that can lead to a good outcome, despite the “racing through a minefield” dynamic. It is distilled from a more detailed post on the Alignment Forum.

First, I’ll flesh out how I see the challenge we’re contending with, based on the premises above.

Next, I’ll list a number of things I hope that “cautious actors” (AI companies, governments, etc.) might do in order to prevent catastrophe.

Many of the actions I’m picturing are not the kind of things normal market and commercial incentives would push toward, and as such, I think there’s room for a ton of variation in whether the “racing through a minefield” challenge is handled well. Whether key decision-makers understand things like the case for misalignment risk (and in particular, why...
Lead in Chocolate?
by jefftk — 2022-12-22T16:10:01.522Z

Consumer Reports recently tested a bunch of dark chocolate for lead
and cadmium, and found that:

For 23 of the bars, eating just an ounce a day would put an adult over
a level that public health authorities and CR's experts say may be
harmful for at least one of those heavy metals. Five of the bars were
above those levels for both cadmium and lead.

If you regularly eat chocolate, how worried about this should you be?

Consumer Reports used the
limits from California's Prop
65, which you may recall from the "this thing contains chemicals
known to the state of California to cause cancer, birth defects, or
other reproductive harm" warnings that are all over everything in
CA. For both lead and cadmium Consumer Reports used the reproductive harm "maximum
allowable daily level" (MADL), which are 0.5 µg/d and
4.1µg/d respectively.

CA calculates these limits by looking at studies that try to find the
highest dose that has no observable effect (NOEL, "No Observable
Effect Level") or the lowest dose that does have one (LOEL, "Lowest
Observable Effect Level"), and estimating an effect threshold. Then
they divide by 1,000 (as required by the original
proposition text) to get the MADL limit. For example, here's the
process they used for cadmium (pdf).

A safety factor of 1,000 is large (though not nuts for converting from a single dose to an ongoing exposure), so it's not
surprising that CA's limits are tighter than you see elsewhere.
For example, the FDA's limits for lead are 3µg/d for kids and
12.5µg/d for women who might become pregnant (pdf), 6x and 25x
the Prop 65 lead MADL. The FDA translates this into a limit of 0.1
ppm for candy
likely to be consumed by small children, 5.7x the level Consumer Reports used.
[1]

Similarly, for cadmium the CDC reports
that the EPA limit (which I can't find on epa.gov) is for food is
1µg/kg/d. CA used a pregnant adult weight of 58 kg in their
cadmium calculation, so the equivalent from the EPA limit would be
58µg/d, 14x the CA Prop 65 level. Using 25kg for a child
weight would give 25µg/d, 6.1x the Prop 65 cadmium MADL.

Putting these all on the same footing, assuming you're eating 1oz of
chocolate daily:

µg/d
ppm (µg/g)

CA Prop 65 lead MADL
0.5
0.018

FDA kid lead limit
3
0.1

FDA pregnant adult lead limit
12.5
0.44

and:

µg/d
ppm (µg/g)

CA Prop 65 cadmium MADL
4

...
Response to Holden’s alignment plan
by Alex Flint — 2022-12-22T16:08:53.188Z

This work was supported by the Monastic Academy for the Preservation of Life on Earth. You can support my work directly here.

I will give a short presentation of this work followed by discussion on Saturday (Dec 24) at 9am Pacific / 12pm Eastern. RSVP here.

Outline

This is a response to Holden Karnofsky’s plan for aligning powerful AI systems in the case that such systems are developed soon.

I give a summary of Holden’s plan, and then critique it.

My basic critique is that the tools Holden is proposing are too similar to what has been tried in the construction of human institutions, and I argue that we have failed to create large numbers of human institutions that alleviate rather than exacerbate existential risk.

The Karnofsky Plan

Holden has written about how we might align very powerful AI systems if such systems were to be developed soon. The basic approach Holden proposes is to train AI systems to be honest, legible, and corrigible, and value-aligned using human supervision. Honesty means not engaging in deceit or manipulation of the supervisor. Legibility means creating artifacts that help supervisors to accurately review decisions. Corrigibility means being willing to stop when the human supervisor says no, no matter how strongly the AI believes that it is doing the right thing. Value-alignment means pursuing the goals of the supervisor. To train such AI systems, Holden says we need to pay attention to the following properties of the training environment:

(1) The human supervisor should provide high-quality feedback. This means that if we are training an AI system to be honest then the human supervisor should give a positive reward if and only if the AI system is, in fact, being honest. It can be hard to know whether an AI system is being honest. Therefore we may design further tools to help the human assess whether the AI system is being honest. In order to do this we may train auxiliary AI systems to assess our primary AI system, or we may restrict the complexity of the primary AI systems, or we may set up an ecosystem of differently-trained AI systems in such a way that dishonesty is "weeded out" by ecological dynamics.

(2) The AI learner should correctly generalize what it has been taught. This means that if we are training an AI system to be honest then the AI system should be honest even when presented with situations very unlike those that...
Staring into the abyss as a core life skill
by benkuhn — 2022-12-22T15:30:05.093Z

Recently I’ve been thinking about how all my favorite people are great at a skill I’ve labeled in my head as “staring into the abyss.”1

Staring into the abyss means thinking reasonably about things that are uncomfortable to contemplate, like arguments against your religious beliefs, or in favor of breaking up with your partner. It’s common to procrastinate on thinking hard about these things because it might require you to acknowledge that you were very wrong about something in the past, and perhaps wasted a bunch of time based on that (e.g. dating the wrong person or praying to the wrong god). However, in most cases you have to either admit this eventually or, if you never admit it, lock yourself into a sub-optimal future life trajectory, so it’s best to be impatient and stare directly into the uncomfortable topic until you’ve figured out what to do.

The first time I learned what really exceptional abyss-staring looks like, it was by watching Drew, the CEO of Wave. Starting a company requires a lot of staring into the abyss, because it involves making lots of serious mistakes (building the wrong thing, hiring the wrong person, etc.); to move quickly, you need to be fast at acknowledging and fixing them. Drew was extremely willing to tackle uncomfortable decisions head-on—“should we not have hired this person?” “Should we pivot away from this business that is pretty good but not great?"—and every time, it was immediately obvious that the decision he made was a big improvement.

Since then, I’ve become fascinated by the role that abyss-staring plays in people’s lives. I noticed that it wasn’t just Drew who is great at this, but many the people whose work I respect the most, or who have had the most impact on how I think. Conversely, I also noticed that for many of the people I know who have struggled to make good high-level life decisions, they were at least partly blocked by having an abyss that they needed to stare into, but flinched away from.

So I’ve come to believe that becoming more willing to stare into the abyss is one of the most important things you can do to become a better thinker and make better decisions about how to spend your life.

To try to recreate the flavor of watching Drew stare into the abyss for seven years, here are some examples.

When he and his cofounder Lincoln first started a company together (well before my time), they built about 1

...