trentmkelly/LessWrong-43k huggingface dataset:
Apply to >30 AI safety funders in one application with the Nonlinear Network
by KatWoods — 2023-04-12T21:23:45.276Z

Nonlinear

spoke to dozens of earn-to-givers and a common sentiment was, "I want to fund good AI safety-related projects, but I don't know where to find them." At the same time, applicants don’t know how to find them either. And would-be applicants are often aware of just one or two funders - some think it’s “

LTFF

or bust” - causing many to give up before they’ve started, demoralized, because fundraising seems too hard.

As a result,

we’re trying an experiment

to help folks get in front of donors and vice versa. In brief:

Looking for funding?

Why apply to just one funder when you can apply to dozens?

If you've already applied for EA funding, simply paste your existing application. We’ll share it with relevant funders (~30 so far) in our network.

You can apply if you’re still waiting to hear from other funders. This way, instead of having to awkwardly ask dozens of people and get rejected dozens of times (if you can even find the funders), you can just send in the application you already made.

We’re also accepting non-technical projects relevant to AI safety (e.g. meta, forecasting, field-building, etc.)

Application deadline: May 17, 2023.

[Edit: new deadline is the 24th, to accommodate the NeurIPS deadline]

Looking for projects to fund?

Apply to join the funding round

by May 24, 2023. Soon after, we'll share access to a database of applications relevant to your interests (e.g. interpretability, moonshots, forecasting, field-building, novel research directions, etc).

If you'd like to fund any projects, you can reach out to applicants directly, or we can help coordinate. This way, you avoid the awkwardness of directly rejecting applicants, and don’t get inundated by people trying to “sell” you.

Inspiration for this project

When the FTX crisis broke, we quickly spun up the

Nonlinear Emergency Fund

to help provide bridge grants to tide people over until the larger funders could step in.

Instead of making all the funding decisions ourselves, we put out a call to other funders/earn-to-givers.

Scott Alexander

connected us with a few dozen funders who reached out to help, and we created a Slack to collaborate.

We shared every application (that consented) in an Airtable with around 30 other donors. This led to a flurry of activity as funders investigated applications. They collaborated on diligence, and gr...
AGI goal space is big, but narrowing might not be as hard as it seems.
by Jacy Reese Anthis — 2023-04-12T19:03:26.701Z

In the wake of FLI’s

AI 6-month pause letter

and Eliezer Yudkowsky’s

doom article in

Time

, there seems to be more high-level attention on the arguments for low and high estimates of AI existential risk. My estimate is currently ~30%, which is higher than

most ML researchers

but seems lower than most self-identified rationalists. Yudkowsky hasn’t given a specific number recently, but I take

statements like

“doubling our chances of survival will only take them from 0% to 0%” to suggest >99%. Here I want to sketch the argument that most accounts for my estimate of ~30% instead of >99%. I agree with doomers that the space of possible goals that AGI could have is extremely big with many existentially catastrophic goals, but I think that narrowing it down to a much smaller subspace with a much higher fraction of safe goals might not be as hard as it seems.

AGI goal space is big.

While there are many arguments to debate—and as others have lamented recently, no good scholarly survey of them—it seems to me like the biggest argument for high P(doom) is that the search space of AGIs is very large and most possible AGIs we could build (i.e., most of this space) have power-seeking, deceptive tendencies because so many goals have "instrumental convergence" towards that. As Yudkowsky

says

, “the AI does not love you, nor does it hate you, and you are made of atoms it can use for something else.” I think this is explained most clearly and fully in John Wentworth's

Fruit Buyer

example, and it’s come up a lot in the recent discourse, but I rarely see it rebutted directly.

Quickly narrowing might not be as hard as it seems.

I don't find this argument compelling enough to have higher P(doom) because chopping up combinatorically or exponentially large spaces seems to not be that hard in many cases. Consider

binary search,

the

halving algorithm

in online learning, or just the fact that many human and nonhuman animals already sort through comparably large spaces in relatively easy ways; quick chops could be feasible for AGI too.

Toy example: Chemistry lab

I hesitate to give a toy example because every one will have disanalogies that you can

bikeshed

, but let’s say that instead of building an AGI, I’m building a chemistry lab to make a baking soda volcano. You rebut, “Most chemistry labs that can make a baking soda volcano can ...
Natural language alignment
by Jacy Reese Anthis — 2023-04-12T19:02:23.162Z

I think a potentially promising and undertheorized approach to AI safety, especially in short timelines, is natural language alignment (NLA), a form of

AI-assisted alignment

in which we leverage the model’s rich understanding of human language to help it develop and pursue a safe[1] notion of social value by bootstrapping up from a natural language expression of alignment, such as giving the nascent AGI a corpus of alignment research and a simple command like, “There's this thing called alignment that we’ve tried to develop in this corpus. Be aligned in the sense we’re trying to get at.” In a world of complex alignment theory, NLA might sound silly, but I think we have a lot of raw power in the self-supervised learning core (the

“shoggoth”

) of large language models (LLMs) that I think we can leverage to address the ambiguity of “alignment” and “values” alongside other approaches.

A key objection to the NLA paradigm is that the initial command would not contain enough bits of information for alignment, but the hope is that we can leverage an near-AGI LLM+'s rich linguistic understanding to address this. Almost every conventional AI safety researcher I’ve discussed this with says something like, “How would those commands contain enough information to align the AGI, especially given pervasive issues like deception and collusion?” and I think every paradigm needs to answer a version of this.

Very roughly, that could mean: agent foundations gets information from new theorems in math and decision theory; mechanistic interpretability gets information by probing and tuning models piece-by-piece; iterated amplification gets information by decomposing problems into more easily solved subproblems; and NLA gets information from natural language by combining a simple command with the massive self-supervised learner in LLMs that builds an apparently very detailed model of the world from next-word prediction and beam search. In this sense, NLA does not require the natural language command to somehow contain all the bits of a well-developed alignment theory or for the AI to create information out of nothing; the information is gleaned from the seed of that command, the corpuses it was trained on, and its own emergent reasoning ability.

NLA seems like one of the most straightforward pathways to safe AGI from the current capabilities frontier (e.g., ...
Repugnant levels of violins
by Solenoid_Entity — 2023-04-12T17:11:47.734Z

This is a rambling post about the repugnant conclusion, mass-manufacturing, metis and woo, and the fragility of expert discernment — through the lens of my experiences playing, repairing, and selling violins. (And eating discount sushi.)

Epistemic status: Discussing utilitarianism-related issues that I'm poorly educated on. (Through the lens of violins, on which I fare much better.) Come for the interesting violin stuff, (don't stay for the philosophy unless you want to watch me repeatedly hitting myself on the head.)

If you work in a violin shop, answering the question "how much should I spend on a violin for my child" is hard to do without revealing that you're a huge nerd.

Playing violin is great. It's the most important instrument.[1]

Teaching it is a challenge. How do you help someone progress from playing like 'oh no' girl to, like, 'wow, she's only three?!' to a really, really good college graduate to being literally Jascha Heifetz?

And more fundamentally: what makes Jascha Heifetz sound so damn good, and how did he get there?[2]

I think if you zoom out far enough, there are three interlinked factors at play in a violinist's ✨ sound ✨:

"How did it feel when they handed you
your first violin? Interlinked."
"Interlinked."

What does their technical ability facilitate them doing, restrict them from doing, and what does it lead them towards doing?

This comprises all of their movement-based competencies, the 'muscle-memory' they develop, the actual physical mechanics of how, through decades of intensive practice, their body has perfectly exapted the violin as the physical part of their aural proprioception.

What are the bounds of their audiation? This is their musical creativity and imagination, the sound they hear in their head when they imagine playing. It's the mental and emotional component of their aural proprioception.

Even though classical violinists are playing music written by someone else, there's creativity and emotion in their interpretation. Compare this, this, this, this, and this.
Just like developing your own writing style, some of this you get from listening to other violinists play. Some of it you generate yourself through your own practice.

Do they have a good instrument? Some violins sound better than others. The best violins in the world are prized (partly) for their unique, beautiful voices.[3]

Violins are usually made of wood (normally sp

...
Progress links and tweets, 2023-04-12
by jasoncrawford — 2023-04-12T16:52:02.778Z
Link: https://rootsofprogress.org/links-and-tweets-2023-04-12

Opportunities

Nat Friedman wants a researcher/journalist for a short AI/rationalism/x-risk project
A new x-risk research lab is recruiting their first round of fellows (via @panickssery)

Links

A “solutionist” third-way on AI safety from Leopold (or thread version)
Jack Devanney on how “ALARA” makes nuclear energy expensive (via @s8mb). See also my review of Devanney’s book
“Instead of a desk, I would like to have a very large lazy susan in my office” (see also my attempts to visualize a soluion)

Queries

What’s the best sci-fi about AI to read at this moment in history?
What explains the full history of Irish population?
Why is the US an outlier in post-2000 suicide rate trends?
Do uncertain statements about the future have truth values?
Is any sigmoid an isomorphism between two groups over ℝ and (0,1)?

Quotes

Evidence for the human capital theory of the Industrial Revolution

AI tweets & threads

Good concise clarification of a key difference in thinking on AI x-risk
Many AGI doomers & anti-doomers make terrible arguments
“Confabulate” instead of “hallucinate” for LLMs?
GPT-4 speaker intros
Practical advice for coping with the feeling of “imminent doomsday”

Charts

“One universal in the history of childhood stands above all others. The history of childhood is a history of death”

The Counter-Reformation’s search for heresy was a negative shock to science

As terrible as COVID was, we have progressed a lot in the last 200 years

The survival curve inflates like a sail, but the far end of it doesn’t move much
A basic mathematical structure of intelligence
by Golol — 2023-04-12T16:49:07.208Z

An important concept here on LW is that of a singularity of intelligence, or at least a very rapid growth. Although it seems mostly hopeless, it would be nice if we could find a mathematical approach to quantify these things. I think the first point to note is that intelligence is is surely not one-dimensional. The concept of general intelligence suggest that it might be approximately one-dimensional, in some sense. But how can we really know what is true? I would like to get a better understanding of the geometry of intelligence. To do this, a good starting point is to look for natural operations on intelligence, as operations can lead to geometry.

I see two obvious natural operations on intelligence: Acceleration and Cooperation. Acceleration means speeding up an intelligence or giving it more computation/physical time. Cooperation means taking two intelligences and letting them cooperate on a task.

You may note that this is somewhat similar to sequential and parallel computing, although I can not rigorously state connections yet.

My main message in this post is that The "set of intelligences" with the operations of acceleration and cooperation has the mathematical structure of an R+ semi-module. I believe that a further mathematical study of the "set of intelligences" is likelt to take note of this fundamental structure.

In the rest of the post I will elaborate on what exactly this means.

Consider an abstract set M of all "intelligent agents", where I do not want to give an explicit definition. I think there are many equivalent and non-equivalent ways to define this, but for now an intuitive idea suffices. An intelligent agent is a thing that can be given a task and an amount of time or computation to solve it. The result can then be scored. We therefore think of a task T as a function T:M⟶R+.

Given an agent M∈M and a set of tasks T, the overall performance of the agent is represented by the induced function [T↦T(M)]:T⟶R+. It is this function which grows during a rapid of growth of intelligence, and we would like to know exactly how it might grow. I can not answer this but finding structure is a good start.

We endow the set M with an operation +. For M1,M2∈M, the agent M1+M2 represents a team consisting of M1 and M2, which may now collaborate on any task. Again, I...
Should AutoGPT update us towards researching IDA?
by Michaël Trazzi — 2023-04-12T16:41:13.735Z

Given the rate of progress in AutoGPT-like approaches, should we reconsider Paul Christiano's Iterated Distillation and Amplitication (IDA) agenda as potentially central to the alignment of transformative ML systems?

For contex on IDA and AutoGPT:

https://www.lesswrong.com/tag/iterated-amplification

https://github.com/Torantulino/Auto-GPT

https://www.lesswrong.com/posts/dcoxvEhAfYcov2LA6/agentized-llms-will-change-the-alignment-landscape
Towards a solution to the alignment problem via objective detection and evaluation 
by Paul Colognese — 2023-04-12T15:39:31.662Z

Thanks to Arun Jose, Joseph Bloom, and Evan Hubinger for discussions/feedback.

This work was primarily carried out during

SERI MATS

under Evan Hubinger’s mentorship.

Introduction

We explore whether the ability to detect and evaluate the objectives of advanced AI systems in training and in deployment is sufficient to solve the alignment problem. We mostly ignore practical/implementation considerations in order to examine this approach in an idealized setting before we try to make concrete progress in this direction. This allows us to highlight weaknesses with this approach and further considerations that any practical implementation must contend with. We note that taking this idealized framing hides some messy issues that we intend to discuss in future posts.

We note that this approach is not novel (see

this post

or the related work section of this post) but provides the motivation for our future research and some additional considerations.

Our basic analysis indicates that we can solve the alignment problem in this idealized setting if we can:

Detect all objectives/intentions of any system produced during the training process as well as any objectives that may develop in deployment.
Evaluate whether the outcomes produced by a system pursuing a set of objectives will be good/bad/irreversibly bad.
Shut down a system if we detect that the set of objectives it’s pursuing will lead to irreversibly bad outcomes before it can accomplish those outcomes.

High-level summary of this post

Automating certain tasks will require us to build/train AI systems that pursue objectives (optimizing systems).[1]
We are concerned with unintentionally training optimizing systems that pursue (catastrophically) bad objectives during training or in deployment.
Assumption: Optimizing systems accomplish some outcomes by having an “internalized objective” that is causally upstream of that outcome, i.e., in a counterfactual setting where this internalized objective didn’t exist within the internals of the system, the outcome wouldn’t be realized in the environment.
Assumption: We assume that outcomes such as successful deception require the optimizing system to have a corresponding objective.
We claim that if an overseer can accurately detect and evaluate all of the objectives of optimizing systems produced during the training process and during deployment, before the outcomes associated

...
Artificial Intelligence as exit strategy from the age of acute existential risk
by Arturo Macias — 2023-04-12T14:48:35.771Z

This article argues that given the baseline level of existential risk implied by nuclear weapons the development of Artificial Intelligence (AI) probably implies a net reduction in existential risk. The so called Artificial General Intelligence (AGI) can replace the human political system and solve the worst alignment problem: the one that human groups have with respect to each other.

The Age of Existential Risk

If we had to describe in a few words our historical moment, not from the perspective of years or decades, but from that of our existence as a species, this moment should be called the age of acute existential risk.

In the last two hundred years, Humanity has experienced an immense expansion of its material capabilities that has intensified its ecological domination and has taken us out of the Malthusian demographic regime in which all other living species are trapped.

In August 6th, 1945, with the first use of a nuclear weapon on a real target, Humanity became aware that its material capabilities now encompassed the possibility of self-extinction. The following decades saw a steady increase in the destructive capacity of nuclear arsenals and several incidents where an escalation of political tension or a technical failure threatened to bring down the sword of Damocles.

An important feature of nuclear war is that it is a funnel for many other sub- existential risks. Financial, ecological, and geopolitical crises, while threatening neither human civilization nor its survival, substantially increase the risk of war, and wars can escalate into a nuclear exchange. Barring the possibility of nuclear war, the risks of a more populous, hotter world with growing problems of political legitimacy are partly mitigated by technology and economic interconnection. But the risk of nuclear war amplifies the other purely historical and environmental risks and turns them into existential risks.

If the nuclear war risk does not reduce over time, an accident, whether technical or political will happen sooner or later. Each of these 77 years after Hiroshima and Nagasaki is a miracle and a tribute to human reason and self-restraint. But without an exit strategy, sustained levels of nuclear war risk doom our technological and post-Malthusian civilization to be an ephemeral phenomenon.

In my opinion we can classify nuclear war risk exit strategies into two types: i) organic s...
The UBI dystopia: a glimpse into the future via present-day abuses
by Solenoid_Entity — 2023-04-12T14:44:03.494Z

Disclaimer: This does risk being a little culture-war-adjacent. This is more political than I normally see on LessWrong, but the spirit of it is intended to be less "Smash Capitalism" or "Become Ungovernable" and more "look at these things that are happening in Australia."

Inspired by: Basic Income, Not Basic Jobs: Against Hijacking Utopia, SSC Gives A Graduation Speech.

Summary

Some bad things are happening today to people who depend on the government for money. This suggests that similar bad things could happen in the future if more people depend on the government for money.

If you're an advocate for UBI, or it's a linchpin in your plan for how we'll live well post-AI, it's important to consider two connected worries carefully:

Creating the scheme as truly universal (and keeping it that way) may be politically untenable in a democracy.
In many likely worlds, some people will depend on this non-universal UBI for survival, leaving them vulnerable to coercive control and arbitrary punishment by the government (lawfully), government officials (unlawfully), even corrupt bureaucrats.

Worries along the lines of "UBI could make us all serfs" and various techno-futuristic dystopian visions are already common enough. There's also a growing part of journalism/civil society/activism concerned with an industry that "farms the unemployed" — billing the government for services it ostensibly provides to poor people, while in fact spending their time on coercive control and a moralistic form of discipline. A "digital poorhouse" per Virginia Eubanks.

I think the average LW reader has also probably heard worries about Government Issued Digital Currency (GIDC), which is certainly part of the concern here. Others have expressed worry about payment processors and banks being politically manipulated.

But I think many people here are less familiar with arguments about problems within the social welfare system that already exist today.[1] These problems are suggestive to me that, absent a change in the culture of these institutions, a UBI might lead to serious abuses.

After listing some reasons it might be hard to actually make an actual universal policy, or 'make it an inalienable human right', etc., I then briefly survey some of the abuses of welfare recipients that have recently occurred in Australia, due to the ability of the government, public servants, and employees of private co...