trentmkelly/LessWrong-43k huggingface dataset:
The best way so far to explain AI risk: The Precipice (p. 137-149)
by trevor — 2023-02-10T19:33:00.094Z

When it comes to explaining AI risk to someone for the first time, it seems to me like it's the kind of thing where there's just a ton of ways to mess up and do it wrong. The human brain might just be really bad at explaining concepts that are complicated but have big implications. The human brain is bad at memorizing, but missing a single piece of the AI safety problem can result in wildly different implications. This is probably the main reason why so few people know about it, even though it affects everyone.

I've spent a lot of time on the problem, looking for quick and concise ways to explain AI safety to any person, and the best thing I've found is the chapter on AI safety in Toby Ord's book The Precipice (2020) from pages 137-149.

I copied those pages into this post, so that more people can share just these 13 pages on AI safety with others. Personally I recommend printing it out, this is complicated stuff and screens can be pretty distracting (and can even reduce attention span and working memory).

_________________________________________________________

UNALIGNED ARTIFICIAL INTELLIGENCE - Pages 137-149 of The Precipice by Toby Ord

In the summer of 1956 a small group of mathematicians and computer scientists gathered at Dartmouth College to embark on the grand project of designing intelligent machines. They explored many aspects of cognition including reasoning, creativity, language, decision-making and learning. Their questions and stances would come to shape the nascent field of artificial intelligence (AI). The ultimate goal, as they saw it, was to build machines rivaling humans in their intelligence.

As the decades passed and AI became an established field, it lowered its sights. There had been great successes in logic, reasoning and game- playing, but some other areas stubbornly resisted progress. By the 1980s, researchers began to understand this pattern of success and failure. Surprisingly, the tasks we regard as the pinnacle of human intellect (such as calculus or chess) are actually much easier to implement on a computer than those we find almost effortless (such as recognizing a cat, understanding simple sentences or picking up an egg). So while there were some areas where AI far exceeded human abilities, there were others where it was outmatched by a two-year-old. This failure to make progress across the board led many AI researchers ...
Is this a weak pivotal act: creating nanobots that eat evil AGIs (but nothing else)?
by Christopher King — 2023-02-10T19:26:00.817Z

I've seen the phrase "there are no weak pivotal acts" pretty often, but I have not been able to locate where this is explained.

The prototypical example of a strong pivotal act is "nanobots that eat GPUs" (my understanding is that this is meant to be an oversimplified example). So for example, what would make "nanobots that eat evil AGIs" not a weak pivotal act:

It actually is a weak pivotal act, and I'm right and everyone else is wrong. (If only, right?)

It is not weak because weakness means being passive in some way, and detecting and eating evil AGIs is too active.

Even if the nanobots only eat evil AGIs, it's not weak because it's technically illegal. (Note that I think other AI labs would actually like the evil AGI eating nanobots because it means they don't need to worry about safety anymore. They can create any type of AI other than evil AGI, and if they make one accidentally the nanobots eat it before their evil AGI kills them.)

Even an aligned super-intelligence couldn't safely accomplish this act accurately (for example, the nanobots might accidentally eat non-AGIs).

Something else?
Why I’m not working on {debate, RRM, ELK, natural abstractions}
by Steven Byrnes — 2023-02-10T19:22:37.865Z

[For background & spelling out the acronyms in the title, see: Debate (AI safety technique), Recursive Reward Modeling, Eliciting Latent Knowledge, Natural Abstractions.]

When I say “Why I’m not working on X”, I am NOT trying to find a polite & diplomatic way to say “Nobody should work on X because X is unhelpful for AGI safety”. Hmm, OK, well, maybe it’s just a little bit that. But really, I don’t feel strongly. Instead, I think:

A lot of disagreement about what a solution to technical AGI safety looks like is really downstream of disagreements about questions like “How will AGI be built? What will it look like? How will it work?”
Nobody really knows the answers to those questions.
So we should probably be contingency-planning, by going through any possible answers to those questions that at least some reasonable person finds plausible, and doing AGI safety research conditional on those answers being correct.
But still, I have my own opinions about the answers to those questions, and obviously I think my opinions are right, and I am not going to work on something unless it makes sense on my own models. And since people ask me from time to time, it seems worth explaining why the various research programs in the post title do not seem to be a good use of time, on my own models of how AGI will be developed and what AGI will look like.

I wrote this post quickly and did not run it by the people I’m (sorta) criticizing. Do not assume that I described anything fairly and correctly. Please leave comments, and I’ll endeavor to update this post or write a follow-up in the case of major errors / misunderstandings / mind-changes.

(By the way: If I’m not working on any of those research programs, then what am I working on? See

here

. I listed six other projects that seem particularly great to me

here

, and there are many others besides.)

1. Background

1.1 “Trying” to figure something out seems both necessary & dangerous

(Partly self-plagiarized from

here

.)

Let’s compare two things: “trying to get a good understanding of some domain by building up a vocabulary of new concepts and their relations” versus “trying to win a video game”. At a high level, I claim they have a lot in common!

In both cases, there are a bunch of possible “moves” you can make (you could think the thought “what if there’s some analogy between this and that?”, or you could think the

...
Conditioning Predictive Models: Open problems, Conclusion, and Appendix
by evhub — 2023-02-10T19:21:20.251Z

This is the final of seven posts in the Conditioning Predictive Models Sequence based on the paper “Conditioning Predictive Models: Risks and Strategies” by Evan Hubinger, Adam Jermyn, Johannes Treutlein, Rubi Hudson, and Kate Woolverton. Each post in the sequence corresponds to a different section of the paper.

Edit: For some follow-up discussion of some differentiation factors between predictive and non-predictive models that could yield good experiments in that direction, see here.

7. Open problems

We think that there are a wide variety of ways—both experimental and theoretical—in which our analysis could be expanded upon. Here, we’ll try to briefly lay out some of the future directions that we are most excited about—though note that this is only a sampling of some possible future directions, and is thus a highly incomplete list:

Are pre-trained LLMs well-modeled as predictive models or agents?

As pre-trained model scale increases, do markers of agentic behavior increase as well?

See “Discovering Language Model Behaviors with Model-Written Evaluations” for some initial results on this question.

To what extent do LLMs exhibit distributional generalization?

Distributional generalization seems like evidence of acting as a generative/predictive model rather than just optimizing cross-entropy loss.

To the extent that current LLMs are doing some sort of prediction, can we find evidence of that in their internal structure?

Is the RLHF conditioning hypothesis true?

How do markers of agentic behavior change as the amount of RLHF done increases, and under different RLHF fine-tuning regimes?

See “Discovering Language Model Behaviors with Model-Written Evaluations” for some initial results on this question.

For anything that an RLHF model can do, is there always a prompt that gets a pre-trained model to do the same thing? What about a soft prompt or a prompt chain?

In addition to validating the extent to which RLHF models can be mimicked using techniques that are more clearly implementing a conditional, a positive result here could also provide an alternative to RLHF that allows us to get the same results without relying on the RLHF conditioning hypothesis at all.

More generally, how similar are RLHF fine-tuned models to pre-trained models with fine-tuned soft prompts?

The idea here being that a soft prompt is perhaps more straightforward to think of

...
Jobs that can help with the most important century
by HoldenKarnofsky — 2023-02-10T18:20:07.048Z

Let’s say you’re convinced that AI could make this the most important century of all time for humanity. What can you do to help things go well instead of poorly?

I think the biggest opportunities come from a full-time job (and/or the money you make from it). I think people are generally far better at their jobs than they are at anything else.

This piece will list the jobs I think are especially high-value. I expect things will change (a lot) from year to year - this is my picture at the moment.

Here’s a summary:

Role

Skills/assets you'd need

Research and engineering on AI safety

Technical ability (but not necessarily AI background)

Information security to reduce the odds powerful AI is leaked

Security expertise or willingness/ability to start in junior roles (likely not AI)

Other roles at AI companies

Suitable for generalists (but major pros and cons)

Govt and govt-facing think tanks

Suitable for generalists (but probably takes a long time to have impact)

Jobs in politics

Suitable for generalists if you have a clear view on which politicians to help

Forecasting to get a better handle on what’s coming

Strong forecasting track record (can be pursued part-time)

"Meta" careers

Misc / suitable for generalists

Low-guidance options

These ~only make sense if you read & instantly think "That's me"

A few notes before I give more detail:

These jobs aren’t the be-all/end-all. I expect a lot to change in the future, including a general increase in the number of helpful jobs available.

Most of today’s opportunities are concentrated in the US and UK, where the biggest AI companies (and AI-focused nonprofits) are. This may change down the line.

Most of these aren’t jobs where you can just take instructions and apply narrow skills.

The issues here are tricky, and your work will almost certainly be useless (or harmful) according to someone.

I recommend forming your own views on the key risks of AI - and/or working for an organization whose leadership you’re confident in.

Staying open-minded and adaptable is crucial.

I think it’s bad to rush into a mediocre fit with one of these jobs, and better (if necessary) to stay out of AI-related jobs while skilling up and

...
Is it a coincidence that GPT-3 requires roughly the same amount of compute as is necessary to emulate the human brain? 
by RomanS — 2023-02-10T16:26:11.255Z

The amount of compute required to emulate the human brain depends on the level of detail we want to emulate.

Back in 2008, Sandberg and Bostrom proposed the following values:

Level of emulation detail	FLOPS required to run the brain emulation in real-time
Analog network population model	10^15
Spiking neural network	10^18
Electrophysiology	10^22
Metabolome	10^25
Proteome	10^26
States of protein complexes	10^27
Distribution of protein complexes	10^30
Stochastic behavior of single molecules	10^43

Today I've encountered an interesting piece of data on GPT-3 (source):

GPT-3 required ~10^15 FLOPS for inference.
It required ~10^23 FLOPS to train it [Note: the training took some months. It would require ~10^30 FLOPS to train it from zero in one second]

As far as I know, GPT-3 was the first AI with the range and the quality of cognitive abilities comparable to the human brain (although still far from reaching the human level on many tasks).

Coincidentally(?), GPT-3 requires 10^15 - 10^30 FLOPS to operate at the brain's speed, which is roughly the same amount of compute necessary to run a decent emulation of the human brain.

The range of possible compute is almost infinite (e.g. 10^100 FLOPS and beyond). Yet both intelligences are in the same relatively narrow range of 10^15 - 10^30 (assuming the human brain emulation doesn't need to be nano-level detailed).

Is it a coincidence, or is there something deeper going on here?

This could be important for both understanding the human brain, and for predicting how far we are from the true AGI.
Contra: Changing Role Terms
by jefftk — 2023-02-10T15:00:01.226Z

In contra dance there are two roles, which were traditionally called
"Ladies" and "Gents". A lot of people are currently trying to move
the community over to calling the roles "Robins" and "Larks", to make
it clear that anyone can dance either role. BIDA, the dance I help
organize, switched in

2017,
and since restarting dances in Spring 2022

all the Boston dances
are gender-free. A recent discussion on the sharedweight

callers
list, however, brought up some past times people tried to change
the mainstream role terms, and I wanted to look into this more.

In the early 90s, there was a push to move from Gents/Ladies
to Men/Women. For example, here's Myrtle Wilhite's 1993
essay, "I am not a lady":

You may notice that I don't write or call dances with ladies in
them. This is distinctly classist language in the fabric of contra
terminology which not only implies upper vs. lower class distinctions
('woman' does not), but encourages submission for the women ("now, act
like a lady!!!"). I preserve the clarity of the call with "women" for
ladies and "men" for gents (same number of syllables), and in my
calling years I have never had any misunderstandings about it. (As you
might guess, I don't call "...and trap that pretty little girl" in
square calls either.)

To the rejoinder that "the figure is called a ladies chain", I reply
that the name of a figure is "chain", and that the gender
discrimination only determines who is to cross the set. After all,
there are "men's chains".

To those who reply that this is "traditional wording that shalt not be
altered" (was that the eleventh commandment?), I remind them that this
is THE FOLK PROCESS after all, and I'm allowed to try different
things, thank you. If the wording didn't work, it wouldn't work, but
it does. Of those who spontaneously noticed, I had only favorable
comments from men and women alike. (I wonder after writing that how
many ruffled feathers there will be--good excuse to make a feather
pillow...)

So, I'd like to suggest that you refer to the woman's role
consistently by the term woman or women, and the man's role
consistently by the term man or men. And if you don't think that other
choreographers are likely to agree with this editing, I would really
like you to at least preserve the terminology in my dances/dance
descriptions. If you choose not to do this in your own calling or
dance

...
Cyborgism
by NicholasKees — 2023-02-10T14:47:48.172Z

Thanks to Garrett Baker, David Udell, Alex Gray, Paul Colognese, Akash Wasil, Jacques Thibodeau, Michael Ivanitskiy, Zach Stein-Perlman, and Anish Upadhayay for feedback on drafts, as well as Scott Viteri for our valuable conversations.

Various people at Conjecture helped develop the ideas behind this post, especially Connor Leahy and Daniel Clothiaux. Connor coined the term "cyborgism".

(picture thanks to Julia Persson and Dall-E 2)

Executive summary: This post proposes a strategy for safely accelerating alignment research. The plan is to set up human-in-the-loop systems which empower human agency rather than outsource it, and to use those systems to differentially accelerate progress on alignment.

Introduction

: An explanation of the context and motivation for this agenda.

Automated Research Assistants

: A discussion of why the paradigm of training AI systems to behave as autonomous agents is both counterproductive and dangerous.

Becoming a Cyborg

: A proposal for an alternative approach/frame, which focuses on a particular type of human-in-the-loop system I am calling a “cyborg”.

Failure Modes

: An analysis of how this agenda could either fail to help or actively cause harm by accelerating AI research more broadly.

Testimony of a Cyborg

: A personal account of how Janus uses GPT as a part of their workflow, and how it relates to the cyborgism approach to intelligence augmentation.

Terminology

GPT: Large language models trained on next-token prediction. Most plans to accelerate research (including this one) revolve around leveraging GPTs specifically. I will mostly be using “GPT'' to gesture at the base models which have not been augmented using reinforcement learning.[1]
Autonomous Agent: An AI system which can be

well modeled

as having goals or preferences, and deliberately selects actions in order to achieve them (with limited human assistance).
Capabilities research: Research which directly improves the capabilities of AI systems and thereby brings us closer to being able to train and deploy more powerful autonomous agents.[2]
Simulator: A

class of AI system

(of which GPT is a member). Simulators are generative predictive models, where the model makes a prediction (probability distribution) about how the state of a system will evolve, and then the state is updated by sampling from that prediction/distribution. The result is a process whic

...
FLI Podcast: Connor Leahy on AI Progress, Chimps, Memes, and Markets (Part 1/3)
by remember — 2023-02-10T13:55:59.387Z

We often prefer reading over listening to audio content, and have been testing transcribing podcasts using our new tool at Conjecture, Verbalize, with some light editing and formatting. We're posting highlights and transcripts of podcasts in case others share our preferences, and because there is a lot of important alignment-relevant information in podcasts that never made it to LessWrong.
If anyone is creating alignment-relevant audio content and wants to transcribe it, get in touch with us and we can give you free credits!

The podcast episode transcribed in this post is available here.

Topics covered include:

Defining artificial general intelligence
What makes humans more powerful than chimps?
Would AIs have to be social to be intelligent?
Importing humanity's memes into AIs
How do we measure progress in AI?
Gut feelings about AI progress
Connor's predictions about AGI
Is predicting AGI soon betting against the market?
How accurate are prediction markets about AGI?

Books cited in the episode include:

The Incerto Series by Nassim Nicholas Taleb
The Selfish Gene, Richard Dawkins
Various books on primates and animal intelligence by Frans De Wall
Inadequate Equilibria by Eliezer Yudkowsky

Highlights

On intelligence in humans and chimps:

We are more social because we're more intelligent and we're more intelligent because we are more social. These things are not independent variables. So at first glance, if you look at a human brain versus a chimp brain, it's basically the same thing. You see like all the same kind of structures, same kind of neurons, though a bunch of parameters are different. You see some more spindle cells, it's bigger. Human brain just has more parameters, it's just GPT-3 versus GPT-4...

But really, the difference is, is that humans have memes. And I mean this in the Richard Dawkins sense of evolved, informational, programmatic virtual concepts that can be passed around between groups. If I had to pick one niche, what is the niche that humans are evolved for?

I think the niche we're evolved for is memetic hosts.

On benchmarks and scaling laws:

Benchmarks are actually coordination technologies. They're actually social technologies. What benchmarks are fundamentally for is coordination mechanisms. The kind of mechanisms you need to use when you're trying to coordinate groups of people around certain things....

So we have these scaling laws, which I t

...
What's actually going on in the "mind" of the model when we fine-tune GPT-3 to InstructGPT?
by rpglover64 — 2023-02-10T07:57:04.733Z

I posted in the open thread and was told that it would be worth promoting to top level.

cubefox responded with a link to an great explanation of how the fine-tuning is done, which made me realize that my original question was unclear, so I'm going to try to clarify.

The fundamental behavior of GPT-3 is token prediction, which can straightforwardly be leveraged into text completion; in contrast, the fundamental behavior of InstructGPT is instruction following. Instruction following is a new capability that uses the knowledge from the token prediction task to produce output as well as to understand input; how does that capability develop?

Some plausible experiments related to the question:

Follow a similar methodology to fine-tune a predictive model for instruction following, checkpointing along the way; for 100 (or even more) novel instruction prompts, see how the different checkpoints respond (in particular, how often they do completion vs instruction following).

Given a prompt P, which produces completion C when fed into the fine-tuned model, try to find a prompt P' that produces C when fed into the original model.

Fine-tune twice with the same data and reward model, but in a different order; presumably the models will have different weights, but can we find prompts that give widely diverging results? If we have two checkpoint histories, at which point does the behavior diverge?
Mechanism Design for AI Safety - Agenda Creation Retreat
by Rubi J. Hudson — 2023-02-10T03:05:56.467Z

Mechanism Design for AI Safety (MDAIS) has been running a reading group since last summer to discuss how principles of mechanism design can be applied to AI Safety. Sign-up for the reading group can be found

here

, and past readings can be found

here

.

Based on the quality of the discussions, we are optimistic about the possibility of mechanism design tools helping with AI safety, and are setting out to create an agenda that outlines promising directions. Our plan is to jumpstart this with a retreat that will facilitate in-person brainstorming and developing ideas. The application form is

here

(see below for details), and you can participate without having joined the MDAIS reading group.

Retreat Details

Where

The retreat will take place in Miami, Florida. Travel funding is available for attendees. A remote attendance option may be added if multiple qualified applicants indicate they would be unable to attend in person.

When

Retreat events will begin in the early afternoon on Friday, March 17th and continue to the late afternoon on Sunday March 19th. Lodging will be provided from Thursday to Sunday night, and social events may take place on those evenings.

Who

Most participants in the reading group are PhD students or post-docs. We expect a similar group makeup for the retreat, but welcome applications from people with research experience who fall outside that demographic. Capacity at the retreat will be approximately ten people.

What

In the weeks leading up to the retreat, participants will be encouraged to brainstorm ideas and share background readings. The primary activity at the retreat will be alternating between developing ideas individually or in small groups, and presenting these ideas to the broader group for feedback. An initial rough writeup of the agenda will take place at the retreat itself, which will be added to and edited over the following weeks.

Other activities will include icebreakers and conversations around AI more broadly. Senior researchers have been invited to call in and discuss their personal research and provide advice on creating an agenda. A number of fun social activities are also being planned.

Application Logistics

The application form can be found

here

. It should be relatively quick to complete, less than five minutes on average. Applications will close on Sunday, February 19th at 11:59pm PST, and acceptances...