trentmkelly/LessWrong-43k huggingface dataset:
Estimating the Current and Future Number of AI Safety Researchers
by Stephen McAleese — 2022-09-28T21:11:33.703Z
Link: https://forum.effectivealtruism.org/posts/3gmkrj3khJHndYGNe/estimating-the-current-and-future-number-of-ai-safety

Summary

I estimate that there are about 300 full-time technical AI safety researchers, 100 full-time non-technical AI safety researchers, and 400 AI safety researchers in total today. I also show that the number of technical AI safety researchers has been increasing exponentially over the past few years and could reach 1000 by the end of the 2020s.

Introduction

Many previous posts have estimated the number of AI safety researchers and a generally accepted order-of-magnitude estimate is 100 full-time researchers. The question of how many AI safety researchers there are is important because the value of work in an area on the margin is proportional to how neglected it is.

The purpose of this post is the analyze this question in detail and come up with hopefully a fairly accurate estimate. I'm going to focus mainly on estimating the number of technical researchers because non-technical AI safety research is more varied and difficult to analyze. Though I'll create estimates for both types of researchers.

I'll first summarize some recent estimates before coming up with my own estimate. Then I'll compare all the estimates later.

Definitions

I'll be using some specific terms in the post which I think are important to define to avoid misunderstanding or ambiguity. First, I'll define 'AI safety', also known as AI alignment, as work that is done to reduce existential risk from advanced AI. This kind of work tends to focus on the long-term impact of AI rather than short-term problems such as the safety of self-driving cars or AI bias.

My use of the word 'researcher' is a generic term for anyone working on AI safety and is an umbrella term for more specific roles such as research scientist, research engineer, or research analyst.

Also, I'll only be counting full-time researchers. However, since my goal is to estimate research capacity, what I'm really counting is the number of full-time equivalent researchers. For example, two part-time researchers working 20 hours per week can be counted as one full-time researcher.

I'll define technical AI safety research as research that is directly related to AI safety such as technical machine learning work or conceptual research (e.g. ELK). And non-technical research includes research related to AI governance, policy, and meta-level work such as this post.

Past estimates

80,000 hours: estimated that there were about 300 people working on redu

...
Progress links and tweets, 2022-09-28
by jasoncrawford — 2022-09-28T20:26:09.947Z
Link: https://rootsofprogress.org/links-and-tweets-2022-09-28

Announcements

“Vast is developing artificial gravity space stations to expand humanity across the solar system”
Derek Thompson and Ezra Klein are writing a book about abundance (@DKThomp)
Foresight DAO to seed-fund “projects in longevity, nanotechnology, BCIs, private computing, and space tech that are too ambitious for legacy funders to support” (via @foresightinst)
FTX Future Fund offering up to $1.5M in prizes to change their minds on AGI x-risk (via @leopoldasch)

Quotes

Even before the locomotive, road and coach improvements alone sped up stagecoach travel by 4x in the 1700s
If you think today’s food additives are unhealthy…
The “control revolution”
Engage with criticism, but don’t worry too much about critics

Retweets

Thread from the International Manufacturing Technology Show, “devoted to CNC mills, grinders, robot arms, metrology, CAD, 3D printing etc” (@Ben_Reinhardt)
With mRNA, we can create a much more effective annual flu shot (@calebwatney)
Two schoolkids who don’t speak the same language chatted for a whole bus ride using Google Translate (@johnjhorton)
Belgium closing a nuclear plant that produces almost 10% of its electricity is “civilizational controlled flight into terrain” (@patrickc)
When you optimize a proxy, you make progress for a while, but then you start overoptimizing (@janleike)
Frederick Douglass on “the good luck theory of self-made men” (@mbateman)

Queries

Books on the founding of research universities? (@acesounderglass)
What don’t people get, or fully appreciate, about your job? (@DKThomp)
What do you call a newsletter writer? (@vgr)

Pics

Me speaking at MIT to the Boston ACX and MIT EA groups (via @moreisdifferent). (That sketch on the chalkboard behind me was me trying to replicate this graph from Max Roser)

Poetry

Apropos of nothing: “Compensation,” by Theodosia Pickering Garrison (1874-1944)
EA & LW Forums Weekly Summary (19 - 25 Sep 22')
by Zoe Williams — 2022-09-28T20:18:08.650Z

Supported by Rethink Priorities

Also posted on the EA forum.
This is part of a weekly series - you can see the full collection here. The first post includes some details on purpose and methodology.

If you'd like to receive these summaries via email, you can subscribe here.

Podcast version: prefer your summaries in podcast form? A big thanks to Coleman Snell who is now producing these! The first episode is now up, and this week's will be up soon. Subscribe on your favorite podcast app by searching for 'Effective Altruism Forum Podcast'.

Top Readings / Curated

Designed for those without the time to read all the summaries. Everything here is also within the relevant sections later on so feel free to skip if you’re planning to read it all.

Announcing the Future Fund's AI Worldview Prize

by Nick_Beckstead, leopold, William_MacAskill, ketanrama, ab

The Future Fund believes they may be significantly under or over estimating catastrophic risk from AI, or that they are focusing on the wrong aspects of it. Since this affects the distribution of hundreds of millions of dollars, they are offering prizes of up to $1.5M for arguments that significantly shift their credences on when and how AGI will be developed and its chance of catastrophic effect. A secondary goal is to test the efficacy of prizes at motivating important new insights. Smaller prizes are also available. Entries are due Dec 23rd.

Announcing the Rethink Priorities Special Projects Program

by Rachel Norman

Rethink Priorities is launching a program to help start promising EA initiatives. Some will be internally incubated and later spun-off into independent organizations, others run externally will be fiscally sponsored and given operations support. The team is currently incubating / sponsoring eight projects ranging from insect welfare to community building in Brazil to AI alignment research, and is looking for expressions of interest for those interested in fiscal sponsorship or project incubation.

Nearcast-based "deployment problem" analysis

by HoldenKarnofsky

The AI deployment problem is the question of how and when to (attempt to) build and deploy powerful AI, when unsure about safety and how close other actors are to deploying. This post assumes a major AI company (‘Magma’) thinks it is 6months - 2years to TAI using human feedback on diverse tasks (HFDT), and what both it and an organization de...
LOVE in a simbox is all you need
by jacob_cannell — 2022-09-28T18:25:31.283Z

Learning Other's Values or Empowerment in simulation sandboxes is all you need

TL;DR: We can develop self-aligning DL based AGI by improving on the brain's dynamic alignment mechanisms (empathy/altruism/love) via safe test iteration in simulation sandboxes.

AGI is on track to arrive soon[1] through the same pragmatic, empirical and brain inspired research path that has produced all recent AI success to date: Deep Learning. The DL approach offers its own natural within-paradigm solution for alignment of AGI: first transform the task into a set of measurable in-simu benchmark test environments that capture the essence and distribution of the true problem in reality, then safely iterate ala standard technological evolution guided by market incentives.

We can test alignment via sandboxed simulations of small AGI societies to safely explore and evaluate mind architecture space for the designs of altruistic agents that learn, adopt, and then optimize for the values (or empowerment) of others, all while scaling up in intelligence and power[2]; eventually progressing to large eschatonic simworlds where human-level agents grow up, learn, cooperate and compete to survive, culminating in a winner acquiring decisive (super)powers and facing an ultimate altruistic vs selfish choice to save or destroy their world, all the while never realizing they are in a sim (and probably lacking even the precursor concepts for such metaphysical realizations)[3].

To the extent that we have 'solved' various subtasks of cognition such as a vision, speech, natural language tasks, various games, etc, it has been through a global evolutionary research process guided by coordination on benchmark sim environments and competition on specific approaches. Over time the benchmark/test environments are growing more complex, integrative and general. So a reasonable (if optimistic) hypothesis is that this trend can continue all the way to aligned AGI.

The future often appears very strange and novel when viewed through the lens of the present. The novelty herein - from the standard AI alignment mindset - is perhaps the idea that we can and must actually test alignment safely and adequately in simulations. But testing in-simu is now just standard practice in modern engineering. We no longer test nuclear weapons in reality as the cost/benefit tradeoff strongly favors simulation, and even far safer techn...
A Library and Tutorial for Factored Cognition with Language Models
by stuhlmueller — 2022-09-28T18:15:10.800Z

We want to advance process-based supervision for language models. To make it easier for others to contribute to that goal, we're sharing code for writing compositional language model programs, and a tutorial that explains how to get started:

The Interactive Composition Explorer (ICE) is a library for writing and debugging compositional language model programs.
The Factored Cognition Primer is a tutorial that explains using examples how to write such programs.

We've been using ICE as part of our work on Elicit and have found it useful in practice.

Interactive Composition Explorer (ICE)

ICE is an open-source Python library for writing, debugging, and visualizing compositional language model programs. ICE makes it easy to:

Run language model recipes in different modes: humans, human+LM, LM
Inspect the execution traces in your browser for debugging
Define and use new language model agents, e.g. chain-of-thought agents
Run recipes quickly by parallelizing language model calls
Reuse component recipes such as question-answering, ranking, and verification

ICE looks like this:

Factored Cognition Primer

The Factored Cognition Primer is a tutorial that explains (among other things) how to:

Implement basic versions of amplification and debate using ICE
Reason about long texts by combining search and generation
Run decompositions quickly by parallelizing language model calls
Use verification of answers and reasoning steps to improve responses

The Primer looks like this:

If you end up using either, consider joining our Slack. We think that factored cognition research parallelizes unusually well and would like to collaborate with others who are working on recipes for cognitive tasks.

To learn more about how we've been using ICE, watch our recent Factored Cognition lab meeting.
Reward IS the Optimization Target
by Carn — 2022-09-28T17:59:08.733Z

This post is a response to Reward is not the optimization target, which someone recommended I read after I made a post on my blog.

People have grown skeptical of the claim that an artificial intelligence would learn to intrinsically value reward over what the reward is tied to. I would have to disagree. I am going to argue that any AI that would surpass its designers would have to value reward over it's goals, otherwise it wouldn't be smarter than its designers.

Being smarter than its designers is defined as coming up with solutions that the designers couldn't.

I am going to make a comparison of computers to humans and evolution. I will be talking about evolution as if it had volition, but that's just a metaphor to get my point across easier. You could replace it with God if you want, I am just describing the way our world works.

Evolution "wants" us to spread our genes. There are two ways it could do that. It could have all the instructions of how to reproduce hard coded into our DNA, or it could push us into reproducing by rewarding certain behavior ~ not dying, having sex, and letting us figure out the rest. Although you can find the first method in simple viruses and bacteria, the second has led to more complex organisms completing the "goal" of passing on their genes since the beginning of life. However, recently, there has been one species that has been messing up this system. Humans.

Humans have found ways of achieving reward without accomplishing "evolution's goals". Some people eat lots of ice cream because it's rewarding to them, but excessive ice cream eating leads to you dying, not the opposite. Some people have sex with condoms, leading to no children being created. Drugs have no counterpart in nature, but they are widespread in society. In a sense, we have "outsmarted" evolution, using it's processes for our own goals rather than what they are "supposed" to do. A system meant to promote reproduction in humans can lead to us preventing it.

If you look at the history of early artificial intelligence research, most of it was trying to hard code an algorithm to accomplish whatever goals the researcher wanted. This kind of research has accomplished many things, Optical Character Recognition, chess playing robots, and even basic chat bots. Many of these things were considered artificial intelligence in the past, and some people thought you would need...
AI Safety Endgame Stories
by Ivan Vendrov — 2022-09-28T16:58:03.075Z

Assume you are in the set of possible worlds where

AI takeover

happens by default. If you do nothing, then at some point in the 21st century the AI lab Magma develops a transformative AI system. Magma employees perform a number of safety checks, conclude the system is safe enough, and deploy it. They deploy it slowly and incrementally, with careful monitoring. But despite their efforts, the system turns out to be unsafe and the monitoring insufficient, triggering a cascade of events eventually leading to an existential catastrophe.[1]

I’ll refer to this sequence of events as the “baseline story” going forward.

Assume further that you’re in the narrower set of worlds where this AI catastrophe is contingent on your actions. In other words, there exists a sequence of actions you (or your organization) can take that averts catastrophe, a decisive intervention. Not necessarily a

pivotal act

, an intervention that averts all existential risk from AI. Just an intervention that prevents this specific Magma catastrophe, giving humanity some breathing room, perhaps only a few months or years.[2]

Let’s try to understand what this decisive sequence of actions could look like. It’s tempting to start at the beginning of the sequence and think about what the first few actions look like. Unfortunately, the most probable starting actions are “meta” actions like thinking really hard, talking to experts, or recruiting more people to work on the problem. These are the same kinds of actions that any successful project starts with! So it doesn’t help us constrain the space of decisive interventions.

Instead, it’s more helpful to start with the endgame: how, in the end, did your actions change the baseline story and avert catastrophe? And what were the last nodes in the causal chain leading up to the change?

At the most abstract level, the baseline story has the following structure. A social process (Magma) instantiates a technological artifact (unsafe AI) which destroys the world. There are three objects here, and any change to the story requires changing the properties of at least one of them.

This leads naturally to a 3-way categorization of endgame stories, though the real endgame story will probably involve some combination of the three:

Changing the technology. You changed something about the technology that Magma had acc

...
Will Values and Competition Decouple?
by interstice — 2022-09-28T16:27:23.078Z

There are a great many forces shaping the evolution of the universe. Among them, the values of agents -- systems which attempt to optimize, or steer the future towards certain configurations over others -- seem likely to have a dominant influence on the long-term future. The values of the agents around now have been largely determined by competitive pressures. Many people in the rationalist/EA community seem to take it for granted that this is soon going to change, and we will enter an era in which values and competition are completely decoupled; the values of the beings around at the time of this decoupling will be "locked in" and determine the shape of the entire future. I think is it plausible(>30% probability) that they are wrong, and that competition will continue, with at least some strength, indefinitely. If this is true, it has major implications for the likely trajectory of the world and how we should go about influencing the long-term future. In this blog post I'll lay out why I think this and what the implications are.

Epistemic status: not confident that the thesis is correct; I am confident that the community should be allocating more probability mass to this scenario than they currently are. If you like, imagine prepending every statement with "there is at least a 30% probability that".

SUMMARY

I sketch three possible scenarios for what the value systems of machine intelligences might look like. In two of these scenarios, values and competition are totally decoupled; in the third, they remain partially coupled.

I present the most basic arguments for and against the occurrence of decoupling. Briefly, the difficulty of ensuring successor alignment might generate competitive pressure towards value systems that try to accrue power to their successors in a value-agnostic way. I define autopoietic agents, systems which increase the number and influence of systems similar to themself.

I survey some more arguments given in the EA/rationalist community for why value/competition decoupling will occur. None of them decisively refute the continuing influence of the competitive pressure outlined in section 2.

Discussion of implications

Given that values remain subject to competitive pressures, alignment schemes which plan for an AI to competitively pursue its own autopoiesis while ultimately remaining in the service of human values are doomed to

...
Georgism in Space
by harsimony — 2022-09-28T16:05:28.795Z
Link: https://harsimony.wordpress.com/2022/09/01/georgism-in-space/

Georgism is cool and all, but it was invented in 1879, and I prefer my ideologies to be more futuristic. Unlike the Network State, Georgism doesn’t require blockchains, and unlike Scientology, it lacks aliens.

So lets make it more exciting by taking Georgism into space.

Its not a terrible idea, space has a vast amount of resources that nobody owns. How should they be shared? Can we design tax policy to encourage stable, equitable growth across the universe?

Taxation … in Space!

Space can be broken down into three types of economic land: useful energy, matter, and physical space.

Let’s start with energy. Since there is a fixed amount of useful energy in the universe, it’s an exhaustible resource. In this case, the Georgist framework would suggest a severance tax, where people that extract energy from natural sources pay a fee that goes into a citizens dividend.

Matter is a little different. Since it can be recycled, it’s more like land that you borrow from everyone else. The Georgist tax would be proportional to the value of certain types of matter. Note that the tax cannot depend on the arrangements of that matter, only the basic elements it’s composed of.

There are some difficulties with taxing matter. First, it may be feasible to transmute some elements into other elements. The elasticity of supply of different elements means that it taxes will have to remain low in order to avoid distortions. Second, it may be possible to convert matter into energy in the future, which creates elasticity of supply in both matter and energy, meaning that the taxes on both will have to be reduced to avoid distortions.

Physical space is more straightforward to tax. It’s analogous to land, and people can be taxed for owning particular swathes of space proportional to the rent they can charge for it. It is slightly trickier to implement Georgism on physical space since ownership will have to reconcile relatively stationary swathes of empty space with plots of land on a spinning planet. I think the solution will involve some sort of ownership of space-time where owners of pieces of space in orbit possess the land at its current location and at all future times.

There’s a grab-bag of other things space-Georgism would tax, like spectrum rights, transit routes, and local externalities like debris production, but I expect revenues from these will be smaller than the main sources.

The Elas

...
QAPR 3: interpretability-guided training of neural nets
by Quintin Pope — 2022-09-28T16:02:10.732Z

Introduction

This is week 3 of Quintin's Alignment Papers Roundup. This week, I'm focusing on papers that use interpretability to guide a neural network's training process. A lot of alignment proposals seem to involve a step like this.

Explanatory interactive learning (XIL) is an active research area that uses various methods to generate some form of explanation for the model's decisions (often a heatmap of the most important portions of the input). XIL then optimizes the model to either make model explanations match human explanations, or to apply generic priors of what good explanations should look like (e.g., that they should be sparse).

Alignment proposals that use interpretability to steer model training usually imagine they have access to mechanistic interpretability methods that track a model's internal computations. Current XIL methods rely on easier, ad-hoc explanation methods, such as input saliency maps. Such methods can scale to supervise a full training process, unlike current mechanistic interpretability.

I expect many alignment researchers to think that input saliency methods are insufficient to properly supervise an AI's training process. Even if that's true, I think studying current XIL methods is valuable, simply because we can actually do empirical experiments with them. Even if there are no directly transferable insights we can gain from current XIL methods (unlikely, IMO), we can still learn about the "logistics" of doing general XIL research, such as the best ways to quantify how our interventions changed the trained models, what sort of protocols help humans to scalably use oversight tools, etc.

Additionally, current ad-hoc explanation methods are (very) imperfect, even for the more limited form of explanations they aim to provide. I expect that any future mechanistic interpretability methods that do scale to steering training processes will also be imperfect. Current XIL methods offer an empirical testbed to learn to wield imperfect and exploitable interpretability methods to shape a model's learning process.

Papers

Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations

Neural networks are among the most accurate supervised learning methods in use today, but their opacity makes them difficult to trust in critical applications, especially when conditions in training differ from those in test. Recen

...