trentmkelly/LessWrong-43k huggingface dataset:
(Yet Another) Map for AI Risk Discussion
by chronolitus — 2023-04-06T11:55:34.523Z

Full Size

I made this map for myself, but it quickly turned into a tool which I use to help find productive discussion - for example, in steering towards a common understanding of key divergences.

I also find that it helps me introduce the topic to friends who are not aware of it at all.

Maybe sharing it here will be helpful to others as well?

AI Risk

This is not a post of AI Risk opinions.
Rather, it's a post full of AI Risk questions, aimed towards myself and others.

One of many ways to frame AI Risk is: What is the current situation for humanity with respect to AI killing/controlling everyone?

The map focuses on the following 5 domains for clustering thought:

System control: how much control do we have on the AI's shape?

Default alignment: if we don't have control on the AIs shape, how much does it disagree with us by default?

AI Capabilities: How capable is the AI, and how soon?

Takeover Difficulty: How easy is it for someone who controls all computers and thinks really fast to control the physical world? (now, in 10 yrs, in 100 yrs)

Human coordination: How good is civilization at coordinating against AI threats?

The first two domains together - system control and default alignment - form the pillar of alignment (how easy will it be to have aligned AI?)

The next three together represent the balance of power between humanity and AI-dom.

For finding disagreements, it can be useful to try to assign gut-feeling scores to each domain, which should somewhat decompose each person's overall feeling about the chance of AI doom.

Using probabilities, or simple scores (depending on preference / who you're discussing with):

For example, if I'm speaking to my mom, I'll say I'm at:
(where 0 means the situation is dire for humanity, and 100 completely favorable.)
(in the context of AI Risk, less capabilities corresponds to a safer / better situation - higher score).

System control: 20

Default alignment: 30

Capabilities: 40

Takeover difficulty: 60

Human coordination: 20

And then she can say "20 seems rather low, I think people are better than you think at solving these problems", and we can focus on this particular area of disagreement.

Your overall AI doom score should be some combination of the 5 ( feel free to disagree on how to combine them).

(Q: Why didn't I use probabilities by default? A: You should totally use probabilities if that works for you! I find tha...
The Computational Anatomy of Human Values
by beren — 2023-04-06T10:33:24.989Z

This is crossposted from my personal blog.

Epistemic Status: Much of this draws from my studies in neuroscience and ML. Many of the ideas in this post are heavily inspired by the work of Steven Byrnes and the authors of Shard Theory. However, it speculates quite a long way in advance of the scientific frontier and is almost certainly incorrect in many aspects. However, I believe the core point is true and important.

Tldr: Human values are primarily linguistic concepts encoded via webs of association and valence in the cortex learnt through unsupervised (primarily linguistic) learning. These value concepts are bound to behaviour through a.) a combination of low-level RL and associations with low-level reward signals and integrated into the amortized policy, and b.) linguistic based associations and behavioural cloning of socially endorsed or others’ behaviours. This is mediated by our ‘system 2’ at primarily a linguistic level consisting of iterative self-conditioning through the world model. The important representation space for human values is the latent space of the linguistic world model and the web of associations therein as well as connections between it and low-level policies and reward models from the RL subsystems. The geometry of the embeddings in the latent space is primarily influenced by the training data – i.e. culture and behavioural history, although the association of different latent concepts with positive and negative valence can be driven by the RL system which interfaces with primary rewards. The geometry of the latent space can also be rewritten with continual learning on self-prompts or external stimuli.

In AI alignment, the goal is often understood to be aligning an AGI to human values. Then, typically, the flow of logic shifts to understanding alignment: how to align an AGI to any goal at all. The second element of the sentence – human values – is much less discussed and explored. This is probably partially because alignment sounds like a serious and respectable computer science problem while exploring human values sounds like a wishy-washy philosophy/humanities problem which we assume is either trivially solvable, or else outside the scope of technical problem solving. A related view, which draws implicitly from the orthogonality thesis, but is not implied by it, is that the alignment problem and...
Is "Recursive Self-Improvement" Relevant in the Deep Learning Paradigm?
by DragonGod — 2023-04-06T07:13:31.579Z

Disclaimer

I wrote this three months ago, and abandoned it[1]. I currently do not plan to return to it anytime soon, but I have nonetheless found myself wanting of a document I could point to for why I'm not particularly enthused by or overly concerned with "recursive self improvement". As such I publish anyway; I apologise for not being a more competent writer.

Epistemic Status

Unconfident on some object level details; I don't have a technical grasp of machine learning[2].

Author's Note

Despite appearances, this is not actually a rhetorical question — phrasing this as a question post was a deliberate decision — do please provide answers to the questions I posited.

The footnotes are unusually extensive in this post; I'd recommend the curious reader not skip them[3].

Related

Why all the fuss about recursive self-improvement?
Evidence on recursive self-improvement from current ML

Abstract

The main questions this post posits:

Is "seed AI flavoured recursive self-improvement" applicable to the deep learning paradigm?

My current position: largely "no"

In light of the answer to the above, should we revise down our estimates of the likelihood (and magnitude[4]) of fast/hard takeoff[5]?

My current position: largely "yes"

Introduction

Back in the day, Yudkowsky formulated recursive self improvement (RSI) as a viable[6] path to foom, under the paradigm of "seed AI". From the wiki[7] (emphasis mine):

Recursive self-improvement and AI takeoff

Recursively self-improving AI is considered to be the push behind the intelligence explosion. While any sufficiently intelligent AI will be able to improve itself, Seed AIs are specifically designed to use recursive self-improvement as their primary method of gaining intelligence. Architectures that had not been designed with this goal in mind,

such as neural networks

or large "hand-coded" projects like Cyc, would have a harder time self-improving.

Eliezer Yudkowsky argues that a recursively self-improvement AI seems likely to deliver a hard AI takeoff – a fast, abruptly, local increase in capability - since the exponential increase in intelligence would yield an exponential return in benefits and resources that would feed even more returns in the next step, and so on. In his view a soft takeoff scenario seems unlikely: "it should either flatline or blow up. You would need exactly the right law of diminishing returns to fly

...
Revisiting the Horizon Length Hypothesis
by Pablo Villalobos — 2023-04-06T06:39:03.874Z

Summary: As part of my work at Epoch, I investigated the horizon length hypothesis - the idea that the horizon length of a task is predictive of the training compute needed to learn that task. My current (weak) conclusion is that the horizon length hypothesis can't be used in practice to estimate the compute requirements for training transformative AI because of the difficulty of 1) measuring horizon length accurately and 2) accounting for compute-saving techniques like curriculum learning. The evidence is weak, so we decided to publish a summary of our research so far here for public scrutiny and gather feedback while we decide whether we want to work more on the topic.

Introduction

The horizon length hypothesis (HLH) asserts that the amount of compute needed to train an ML system to perform a task is C∼HPa, where P is the model size (number of parameters), H is the horizon length of the associated task, and a is an exponent that is common to all tasks.

The horizon length is intuitively the length of the feedback loops involved in learning the task. For example, balancing a pole with your hand has a subsecond feedback loop, whereas running a company has months- or years-long feedback loops. So we should expect the horizon length of the task ‘Running a company’ to be up to 8 orders of magnitude bigger than that of ‘Balancing a pole’.

Dependence on the training method

Instead of directly training a model to perform a task, one can use transfer learning from a different task, or prepare a curriculum of progressively harder instances of the task. Using these techniques it's possible to reduce the amount of task-specific data used in training.

Therefore, using these techniques we might be able to save compute by training only on small amounts of data for the long-horizon task, and doing the bulk of the training on short-horizon tasks. So for long-horizon tasks, if we naively apply the HLH directly with the horizon length of the task, we will overestimate training requirements.

However, the HLH might be valid for fairly direct training methods (no transfer, no reward shaping, etc). If this is true, then it might still be informative for predicting compute requirements. In particular, it seems likely that models will need some training at longer horizon lengths. An example of this might be LLMs, where pretraining has a horizon length of...
Monthly Shorts 3/23
by Celer — 2023-04-06T06:20:07.821Z

Aesthetics

My recommendation of the month is this story from the art world. It’s a pretty simple story.

Artist A posts photo of his living room

Artist B makes a painting, almost exactly matching the photo. The only change is to remove a poster in the background. It’s as close a recreation as you could ask for.

Artist B’s painting is in a fancy gallery in London: I’d be shocked to learn that it appraised for less than a hundred pounds. Artist B has in no way whatsoever given credit or mention to Artist A.

This all comes out in a news story.

Nobody suggests that Artist B did anything wrong. There’s no public outcry. Googling the artist’s name doesn’t reveal hordes of people out for his blood, and the blood of the gallery that allowed stolen art. A photo of the artwork is still up on Artist B’s instagram. I’m not linking because I’m not convinced that Artist B did anything wrong by the standards of their community.

In case you’re wondering: yes, this is about the absolutely absurd claims of some artists that generative AI models are engaged in theft, unlike human artists, who have very different norms around art.

I’ll quote Scott Alexander, because he’s a better writer than I am.

Let me rephrase that. You wanted quicker burger-flipping; instead, you got beauty too cheap to meter. The poorest welfare recipient can now commission works of wonder to make a Medici seethe with envy. If deep down humans always thought that art - and music, and poetry, and all the rest - were just jobs program - just the aesthetic equivalent of digging ditches and filling them to raise the employment rate - tell me now, so I don’t hesitate when the time comes to paperclip you.

If you want a useful model of when something becomes “theft” in art, as opposed to found art, I find that the line is “when it might hurt professional artists”. If an ordinary person like you or I make something of beauty, and an Artist comes along and takes a photo of it, all rights lie with the Artist. Take and display photographs of the work by a famous sculptor, and you will find a very different attitude on display.

By this standard, it is not hard to see why AI-generated art is theft.

On Sanderson: you may have seen the Wired article. It’s a profile of a man who has published an average of almost two books a year for 18 years, Brandon Sanderson. It was not, I think it is fair to say, kind. Sanderson was, as always, a ...
Dual-Useness is a Ratio
by jimrandomh — 2023-04-06T05:46:48.286Z

A lot of AI-risk-concerned people are struggling with how to relate to dual-use research, and relatedly, to doing alignment research inside of AI orgs. There's a pretty simple concept that seems, to me, to be key to thinking about this coherently: the dual-useness ratio. Most prosaic alignment techniques are going to have some amount of timeline-shorteningness to them, and some amount of success-chance-improvement in them. You don't want to round that off to a boolean.

Eg, I've had arguments about whether RLHF is more like an alignment technique, or more like a capabilities technique, and how that should affect our view of OpenAI. My view is that it's both, and that the criticism of it is that its acceleration-to-alignment ratio isn't high enough, because it helps a lot with the kind of alignment that's necessary for commercialization, but doesn't help that much with the kind of alignment that's necessary for preventing a superintelligence from killing everyone.

I think it's pretty important to get alignment-cautious people into the organizations that are developing and deploying AIs, and I think a lot of prosaic alignment techniques with good ratios may be easier to research from inside an AI lab than outside of one. But, the default outcome of sending people to "work on alignment inside of OpenAI" is that they try to find something with a good dual-useness ratio, fail, and work on something with a ratio that's just barely good enough that they can tell a positive story about it to their acquaintances.

I don't expect that problem to ever be straightforward. But, if you're trying to who should go work on alignment inside of AI labs, as opposed to who should work in orgs that don't train language models or work in a different field, I think you can get a lot of mileage out of one key distinction: Some people get wedded to one big idea, other people have lots of ideas and are more willing to drop them. I expect the latter sort of person to wind up working on ideas with much better dual-useness ratios, because they have flexibility that they can use to select for that.
What's the deal with Effective Accelerationism (e/acc)?
by RomanHauksson — 2023-04-06T04:03:19.392Z

I've been hearing murmurs about a recently formed philosophy called "Effective Accelerationism", described as:[1]

...an ideology that draws from Nick Land's theories of accelerationism to advocate for the belief that artificial intelligence and LLMs will lead to a post-scarcity technological utopia. E/acc communities on Twitter were primarily fostered on Twitter Spaces, with e/acc manifestos being shared using the newsletter platform Substack.

One example of said Substack manifestos, Notes on e/acc principles and tenets, outlines on an object level the thesis motivating e/acc. TL;DR:[2]

is: life emerged as a principle of a generalized 2nd law of thermodynamics
is: due to this physical (observed) law, life tends to seek to capture "free energy" (aka the accursed share in terms of Bataille perhaps) to increase its scope/complexity or maintain its existence
ethical/moral claim - the ought: we should seek to "accelerate" (must mean to intensify, not in the physics sense of acceleration, where acceleration could simply mean constantly changing direction) this process of growth of organisms/meta-orgranisms to achieve greater and greater capture of free energy and thus more complex systems of intelligence (they demarcate this as ultimately being about the imperative that "in order to spread to the stars, the light of consciousness/intelligence will have to be transduced to non-biological substrates"

I don't know enough about complex systems and epistemology to be able to assess these arguments, which is why I'm posting about them here. My outside view is that the majority of e/acc discourse appears to be memes on Twitter, which doesn't give me much hope in the epistemic rigor underlying the philosophy? Reddit user I-am-a-person- summarizes what was close to my initial reaction after reading the Substack post:[3]

The problem with this argument is that it does a really bad job arguing why “capturing free energy” is actually the goal we ought to strive for.

If I understand the gist of e/acc correctly, I'm very skeptical of the idea that maintaining diversity/competition/entropy by accelerating and open-sourcing AI capabilities research is more likely to result in good outcomes for society than being more cautious and authoritarian.

Some questions to spark conversation

What are your thoughts on the community behind this philosophy and its bizarre, memetic method for outreach?
If

...
No Summer Harvest: Why AI Development Won't Pause
by Stephen Fowler — 2023-04-06T03:53:34.469Z

Cleo Nardo recently highlighted Yudkowsky and others discussing an "AI Summer Harvest" (or simply "Summer Harvest"). This potential near-future era would involve a pause in research on state-of-the-art (SOTA) models, enabling us to fully enjoy the economic benefits of the extraordinary technology we currently possess. There would be a widespread adoption of contemporary AI systems in conjunction with an enforced moratorium against training more advanced models.

The Summer Harvest would by a serene era in which humanity reaps the rewards of current AI advancements without delving into uncharted risks.

But There Will Be No Summer Harvest.

Thanks to Justis for extensive editing and feedback.

Simberg, Hugo. The Garden Of Death. 1896. [Watercolor and goauche]

Epistemics:
The following argument heavily relies on "aggressive" Fermi estimates and a lot of assumptions. I have tried to err on the side of caution, making the numbers less favorable to my argument.
I have no expertise in economics or international policy.

The Argument

I will argue that it is very unlikely that a pause on AI development longer than 6 months occurs. First I will establish that numerous private companies have a strong incentive to train new and more powerful models. Then in Premise 4 I will demonstrate that the international cooperation required to stop this from happening is very unlikely to occur.
If you're skimming through the premises you'll find the numerical estimates I am using in bold. Each section is relatively self contained and you can skip to the part of the argument that interests you the most.

(Premise 1) SOTA models will be relatively 'cheap' to produce
(Premise 2) The economic benefit of training the next SOTA system would be substantial
(Intermediate Conclusion) There will be a financial incentive of the order of billions of dollars to train the next SOTA model and end the Summer Harvest
(Premise 3) There are groups motivated by financial incentives who are both capable of training SOTA systems and would be willing to ignore the potential dangers
(Premise 4) A strongly enforced, global political agreement to prevent the training of SOTA models is unlikely
(Conclusion) There Will Be No Summer Harvest

Premise 1:
SOTA models will be relatively "cheap" to produce

I assume that the next SOTA model's cost within an ...
Yoshua Bengio: "Slowing down development of AI systems passing the Turing test"
by Roman Leventov — 2023-04-06T03:31:39.120Z
Link: https://yoshuabengio.org/2023/04/05/slowing-down-development-of-ai-systems-passing-the-turing-test/

Full text of the blog post below:

I recently signed an

open letter

asking to slow down the development of giant AI systems more powerful than GPT-4 –those that currently pass the Turing test and can thus trick a human being into believing it is conversing with a peer rather than a machine.

I found it appropriate to sign this letter to alert the public to the need to reduce the acceleration of AI systems development currently taking place at the expense of the precautionary principle and ethics. We must take the time to better understand these systems and develop the necessary frameworks at the national and international levels to increase public protection.

It is because there is an unexpected acceleration –I probably would not have signed such a letter a year ago– that we need to take a step back, and that my opinion on these topics has changed.

There is no guarantee that someone in the foreseeable future won’t develop dangerous autonomous AI systems with behaviors that deviate from human goals and values. The short and medium term risks –manipulation of public opinion for political purposes, especially through disinformation– are easy to predict, unlike the longer term risks –AI systems that are harmful despite the programmers’ objectives,– and I think it is important to study both.

With the arrival of ChatGPT, we have witnessed a shift in the attitudes of companies, for whom the challenge of commercial competition has increased tenfold. There is a real risk that they will rush into developing these giant AI systems, leaving behind good habits of transparency and open science they have developed over the past decade of AI research.

There is an urgent need to regulate these systems by aiming for more transparency and oversight of AI systems to protect society. I believe, as many do, that the risks and uncertainty have reached such a level that it requires an acceleration also in the development of our governance mechanisms.

Societies take time to adapt to changes, laws take time to be passed and regulatory frameworks take time to be put in place. It therefore seems important to raise awareness quickly, to put this issue on more radar screens, to encourage a greater public debate.

I have been saying it for years and I reiterate in this letter, which focused on governance for the huge AI systems larger than GPT-4, not AI at large: it is instea

...