trentmkelly/LessWrong-43k huggingface dataset:
What I learned at the AI Safety Europe Retreat
by skaisg — 2023-04-17T17:40:22.276Z
Link: https://skaisg.eu/what-i-learned-at-aiser/

From the 30th March - 2nd April 2023, I attended the Artificial Intelligence Safety Europe Retreat (AISER) in Berlin, Germany. There were around 70 attendees from all over Europe. Most attendees were actively working on technical AI Safety (e.g SERI-MATS scholars, independent researchers with grants), some people were focusing on AIS strategy and governance, and some newcomers / students (like me) were looking to learn more about the field and career opportunities. For a short overview of the retreat, have a look here.

This post is targeted at people who couldn't make it to the AISER and/or are considering attending a similar event. Concretely, I will provide you with my takeaways from AISER which I gathered by attending some of the talks and one-on-ones. I will conclude the post with some personal (and thus very subjective) impressions of the event. I will not mention specific names as the conversations and talks weren’t given with the intention of appearing in public. Still, I think many ideas exchanged at the AISER are worth sharing with a wider audience. I will put my personal spin on them as I present them in this blog post, so all critique should be directed towards me.

Some background context about me to put this post into a (critical) perspective may be helpful. For a general overview of my experiences and skills, please consult my about page. I wrote my undergrad thesis on robustness and formal verification of AI, but I wasn't aware of the AI alignment movement at the time. Before joining the AI Safety Europe Retreat, I had only just completed the AGI Safety Fundamentals curriculum with a reading group at my university - otherwise I had no experience in the field. I'm motivated to work on AI Safety (AIS) because people seem to trust AI too much, and I want to make AI actually worthy of such trust. Also, another appeal of AIS is its interdisciplinary approach. It employs elements from diverse fields (which I find very cool and interesting to work in in their own right) such as philosophy, mathematics, computer science, and everything in between.

My most important takeaways

Read the bold text if you only care about the main points, the text following those expands upon the idea.

On contributing to technical AIS as a newcomer

Tell your friends about AIS. This is one of the easiest ways to get involved. Note that you shouldn't try to convince your peers to al

...
What is your timelines for ADI (artificial disempowering intelligence)?
by Christopher King — 2023-04-17T17:01:36.250Z

Most AI researchers think we are still a long ways off from AGI. However, I think this can be distracting.

AGI includes the ability to do everything that humanity can do, such as:

Juggle objects of various unexpected shapes
Do the dishes in an arbitrary house
Consistently beat Kellin Pelrine at Go
etc...

However, I think it's plausible that an AI that is not literally an AGI could takeover the world. You can have a pretty strong understanding of the world and be good at optimizing things without being able to copy a human's performance.

I am coining the term "Artificial Disempowering Intelligence (ADI)", or informally "Artificial Dominate AI", "Artificial Doom Intelligence", "Artificial Doomsday Intelligence", or "Artificial Killseverybody Intelligence (AKI)".

Criteria

The criteria are as follows:

ADI "perceives" its environment as being the whole world
ADI tries to reorient aspects of its environment towards some task
Criteria (2) can cause humanity to lose control over the environment if they do not intervene
ADI prevents humans from preventing criteria (2) from within the environment

Stockfish already satisfies all of these criteria except criteria (1).

Examples

Here are some examples of potential ADIs:

PoliticalBot

PoliticalBot's world model is detailed enough to predict the behavior of human institutions, and how they can be manipulated over email. PoliticalBot's understanding of other subjects is very poor (in particular, no direct self-improvement and no nanorobotics). PoliticalBot is trying to do some task, and takes control of the human institutions. As an instrumental goal, it uses them to make humans improve PoliticalBot. It's model of humanity psychology is sufficient to defeat resistance.

IndustrialBot

IndustrialBot's world model has excellent descriptions of energy, industrial processes, and weaponry. It also is good at controlling robots. IndustrialBot does not have a good model of humans; it doesn't even understand language. To play it safe, it models humanity as a rational agent that wants the opposite of what IndustrialBot is trying to do. It then plays a 4X game where humanity loses and dies. IndustrialBot never substantially self-improves because it doesn't understand computer science.

EconoBot

GPT-4 EconoBot just wants to helpfully create economic value. Its world model includes all human economic activity, both at the scale of how to do individual tasks, and...
Slowing AI: Foundations
by Zach Stein-Perlman — 2023-04-17T14:30:09.427Z

Variables

"Slowing AI" is not monolithic. Many interventions have relevant side effects, positive or negative. There are many considerations, desiderata, and goals adjacent to slowing AI, and different interventions realize them to different degrees. This section is a list and discussion of variables. The goal of this section is to help readers recognize the consequences of possible interventions and thus better evaluate them, and to help generate or optimize interventions by identifying variables to affect. The goal of each subsection is to be an introduction to the relevant variable: how it relates to slowing AI and how else it matters.

List

Variables related to slowing AI are:

Timeline (to risky AI) length
Crunch time length
Relative positions of particular (kinds of) actors
Multipolarity[1]
Attitudes on AI risk (among labs and the ML research community, and also states, other actors, and the public)
Technical safety research performed
Strategy & governance research performed
How much

alignment tax

is paid for particular models

Operational adequacy

and safety practices at leading labs
Coordination ability and inclination among leading labs
The extent to which leading labs could scale up compute
How safe the architecture of powerful AI is
The extent to which leading labs develop risky vs unthreatening kinds of systems
Deployment safety
Which (kinds of) actors exert influence over AI
West-China relation
How well powerful AI will be used
Attitudes on AI
How various actors feel about the AI safety community
AI misuse risks
Non-AI risks
Field-building, community-building, and influence-gaining

This list aims to be essentially exhaustive.[2]

Variables overlap. Maybe this list should instead be a tree or directed acyclic graph to show relationships between variables; the first two sections are weird because they relate to almost all of the other variables, which are mostly discussed in their own subsections.

Many "all else equal"s omitted.

1. Slower AI progress would be good

Powerful AI appearing later than it will by default would reduce misalignment risk and total

x-risk

(on current margins, in expectation, all else equal).

I expect that powerful AI will appear substantially sooner than I would prefer. Reasons include

labs and researchers have incentives or apparent-myopic-incentives to cause faster progress, independent of what is impartially optimal;
independent of individuals

...
Slowing AI: Reading list
by Zach Stein-Perlman — 2023-04-17T14:30:02.467Z

This is a list of past and present research that could inform slowing AI. It is roughly sorted in descending order of priority, between and within subsections. I've read about half of these; I don't necessarily endorse them. Please have a low bar to suggest additions, replacements, rearrangements, etc.

Slowing AI

There is little research focused on whether or how to slow AI progress.[1]

EA Forum AI Pause Debate (2023)
Slowing AI (Stein-Perlman 2023)
"Slowing down AI" (Outline and Supplement 1) (Gloor unpublished)

Let’s think about slowing down AI

(Grace 2022)
"How bad/good would shorter AI timelines be?" (Aird unpublished)

Instead of technical research, more people should focus on buying time

(Larsen et al. 2022)

Is it time for a pause?

(Piper 2023)

Why Not Slow AI Progress?

(Alexander 2022)
"The Containing/Decelerationist Approach to Transformative AI Governance" (Maas draft)

What an actually pessimistic containment strategy looks like

(lc 2022)

Particular (classes of) interventions & affordances

Making AI risk legible to AI labs and the ML research community

“AI Risk Discussions” website: Exploring interviews from 97 AI Researchers

(Gates et al. 2023)

Interviews with 97 AI Researchers: Quantitative Analysis

(Shermohammed and Gates 2023)

"AGI risk advocacy" (Lintz draft)

Ways to buy time

(Larsen et al. 2022)

What AI Safety Materials Do ML Researchers Find Compelling?

(Gates and Burns 2022)

Transparency & coordination

Relates to "Racing & coordination." Roughly, that subsection is about world-modeling and threat-modeling and this subsection is about solutions and interventions.

See generally

Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims

(Brundage et al. 2020).

Transparency

Exploring the Relevance of Data Privacy-Enhancing Technologies for AI Governance Use Cases (Bluemke et al. 2023)

This builds on Beyond Privacy Trade-offs with Structured Transparency (Trask and Bluemke et al. 2020)

Honest organizations

(Christiano 2018)

Auditing & certification

What does it take to catch a Chinchilla? Verifying Rules on Large-Scale Neural Network Training via Compute Monitoring

(Shavit 2023)

Auditing large language models: a three-layered approach

(Mökander et al. 2023)

The first two authors have other relevant-sounding work on arXiv

AI Certification: Advancing Ethical Practice by Reducing Information Asymmetries

(Cihon et al. 2021)
Private literature

...
Goodhart's Law inside the human mind
by Kaj_Sotala — 2023-04-17T13:48:13.183Z

Some time back, I saw

a tweet

from somebody that read:

Much of social psychology seems to be premised on the bizarre assumption that what people really care about is not real-world outcomes but the state of their own mind: self-esteem, a positive self-image, dissonance reduction, feelings of control, reducing uncertainty, etc.

I've certainly seen versions of the same myself. Maybe the most poignant example comes from

this book review

, which suggested that gambling addicts get hooked on a sense of control - even though someone who's hooked on gambling to the point of ruining their life clearly isn't in much control of anything:

The primary objective that machine gambling addicts have is not to win, but to stay in the zone. The zone is a state that suspends real life, and reduces the world to the screen and the buttons of the machine. Entering the zone is easiest when gamblers can get into a rhythm. Anything that disrupts the rhythm becomes an annoyance. This is true even when the disruption is winning the game. Many gamblers talk about how winning the game brings them out of the zone, and they actually dislike winning for that reason. For some gamblers, the very act of pressing buttons to play the game disrupts the rhythm. These gamblers use autoplay modes on games that offer them, and jerryrig an autoplay mode on machines that don’t by jamming something into buttons to keep them pressed. They don’t want to chase a win or pick their lucky numbers, they want to disappear into the zone. [...]

The book is full of heartbreaking stories about what gamblers endure on their path to extinction. They sacrifice their bodies, their time, and their relationships. Sharon, for example, spent four days at a casino, trying to lose all her money to reach extinction. At the end of this ordeal, she came home to sleep, but she found three nickels in her bedroom. The thought of not having spent all her money bothered her so much that she drove back to a casino immediately to lose those last three nickels. [...]

I used to think that gambling addicts “lost control” when they gambled excessively. But the addicts in the book use machines as a way to gain control in their lives. In front of a machine, the world is simple: they place bets and lose a little bit of money on each turn. The gamblers are in control of this machine world. It is the world away from machines where the pro

...
Prediction: any uncontrollable AI will turn earth into a giant computer
by Karl von Wendt — 2023-04-17T12:30:44.249Z

It is a common assumption that a hypothetical superintelligent squiggle maximizer would turn all of earth into squiggles, destroying humanity and all life in the process. However, an AI smart enough to overcome human resistance would realize that building robot factories and turning the whole earth into squiggles would be far from optimal for its goal. A better plan would make sure that not only earth, but nearly all matter in the reachable universe would be turned into squiggles. That would require technology far superior to that necessary for transforming just our planet. But maybe there is an even better plan? Could it somehow be possible to overcome the barrier of light speed? Or maybe there is a way to create whole new parallel universes consisting only of squiggles? Also, there could be another superintelligence with a different goal on earth or somewhere in outer space, possibly thwarting its plan. To fulfill its goal, the squiggle maximizer would have to outsmart any potential rivals.

It seems obvious that the best way to maximize the number of squiggles isn’t just using existing technology to make them. Instead, the AI would first increase its own intelligence so it can invent better technology and make a better plan, especially if there is (almost) unlimited time to achieve the optimal world state according to its goal. For that, the AI would need to increase the computing power available to it and improve its ability to use this power, e.g. by optimizing its own code.

In principle, the squiggle maximizer could increase its intelligence and make squiggles in parallel. But at any point in time, it has to decide how to allocate its resources to each. It seems likely that it would be optimal to focus on improving its intelligence first, because otherwise the AI might be wasting time, energy, and resources on a sub-optimal plan and would risk being outsmarted by known or unknown rivals.

But how much intelligence is enough?

Assuming that there is no absolute limit to intelligence and the AI can never know with 100% certainty whether there is another superintelligence lurking somewhere in the universe, the optimal level of intelligence would be reached only if further increasing it would reduce the probability of making the maximum number of squiggles in the time left. This point could be millions or even billions of years in the future. In other words, a sq...
AutoBound on neural network can achieve OOMs lower training loss
by Maybe_a — 2023-04-17T05:20:49.513Z
Link: https://ai.googleblog.com/2023/04/beyond-automatic-differentiation.html

Seems like gradient descent methods weren't using the relevant math bounds so far. Google released AutoBound as an open-source library.

Here is what I consider a money shot of the article (notice it's a log-plot):

Performance of SafeRate when used to train a single-hidden-layer neural network on a subset of the MNIST dataset, in the full-batch setting.

Hopefully, they are just overfitting on MNIST. Otherwise, it pattern-matches to a huge advance. Their repo implies that with float64 this scales to larger neural networks. LLMs seem to reliably get new capabilities with lower loss, at least.

What do you think?

Here are related technical details:

Optimizers that use upper bounds in this way are called majorization-minimization (MM) optimizers. Applied to one-dimensional logistic regression, AutoBound rederives an MM optimizer first published in 2009. Applied to more complex problems, AutoBound derives novel MM optimizers that would be difficult to derive by hand.

We can use a similar idea to take an existing optimizer such as Adam and convert it to a hyperparameter-free optimizer that is guaranteed to monotonically reduce the loss (in the full-batch setting). The resulting optimizer uses the same update direction as the original optimizer, but modifies the learning rate by minimizing a one-dimensional quadratic upper bound derived by AutoBound. We refer to the resulting meta-optimizer as SafeRate.

Using SafeRate, we can create more robust variants of existing optimizers, at the cost of a single additional forward pass that increases the wall time for each step by a small factor (about 2x slower in the example above).

This seems novel to neural network training, or am I missing something that Bayesian neural net people have been doing already?
Making Booking.Com less out to get you
by Elizabeth — 2023-04-17T04:04:42.363Z
Link: https://www.alexcharlton.co/projects/booking-com-de-stresser

Some guy loved booking.com but hated the false urgency they added to the process, so he wrote a plug-in to block the awful elements and reframe the useful ones to grab less attention. I like this both as a useful free thing, and for the general vibe of fixing things and sharing those fixes.
grey goo is unlikely
by bhauth — 2023-04-17T01:59:57.054Z
Link: https://bhauth.com/blog/biology/nanobots.html

The nanomachinery builds diamondoid bacteria, that replicate with solar power and atmospheric CHON, maybe aggregate into some miniature rockets or jets so they can ride the jetstream to spread across the Earth's atmosphere, get into human bloodstreams and hide, strike on a timer.

— Eliezer Yudkowsky

To control these atoms you need some sort of molecular chaperone that can also serve as a catalyst. You need a fairly large group of other atoms arranged in a complex, articulated, three-dimensional way to activate the substrate and bring in the reactant, and massage the two until they react in just the desired way. You need something very much like an enzyme.

— Richard Smalley

My understanding is that anyone who can grasp what "orthos wildly attacking the heterodox without reading their stuff and making up positions to attack" looks like, considers that this is what Smalley did with Drexler - made up an unworkable approach and argued against it.

— Eliezer Yudkowsky

In this post, I use "nanobots" to mean "self-replicating microscopic machines with some fundamental mechanistic differences from all biological life that make them superior". Various specific differences from biological cells have been proposed. I've organized this post by those proposed differences.

1. localized melting

Most 3d printers melt material to extrude it through a nozzle. Large heat differences can't be maintained on a small scale.

2. rare materials

If a nanobot consists largely of something rare, getting more of that material to replicate is difficult outside controlled environments.

Growth of algae and bacteria is often limited by availability of iron, which is more common than most elements. Iron is the active catalytic site of many enzymes, and is needed by all known life. The growth of something made mostly of iron would be far more limited, and other metals have more limited availability than that.

3. metal surfaces

Melting material isn't feasible per (1), so material must be built up by adding to the surface. Since that's the case, the inside of structures must be chemically the same as what was the exterior.

Metal objects have a protective oxide layer. In an air or water environment, there's no way to add individual (eg) aluminum atoms to a metal surface and end up with metallic aluminum inside; the whole thing will typically be aluminum oxide or hydroxide.

Corrosion is also a ...
AGI Clinics: A Safe Haven for Humanity's First Encounters with Superintelligence
by portr. — 2023-04-17T01:52:08.065Z

Introduction:

As we draw closer to the development of artificial general intelligence (AGI), concerns about its potential misuse and the existential risks it poses have become increasingly urgent. Referencing

this LessWrong post

, we must acknowledge the reality that unrestricted access to powerful AI tools could lead to catastrophic outcomes. In response to these concerns, I propose the concept of AGI Clinics, a supervised environment where people can safely interact with super-intelligent AGI systems.

AGI Clinics:

The idea behind AGI Clinics is to provide individuals with controlled access to super-intelligent AGI systems. These clinics would be supervised by both human experts and AI systems to monitor and detect malicious behavior. By offering a structured and safe environment, AGI Clinics could help delay the use of AGI for harmful purposes while we work on developing more aligned and benevolent AI models.

The feasibility of deploying an AGI Clinic if AGI is developed tomorrow seems reasonable. This is more of a last ditch effort for delaying death due to misaligned humans & AI. Yes, there are a lot of drawbacks to this idea, but creating a physical space for people to interact with super-intelligent AGI would significantly decrease the chance of dying to AGI in the near future.

P:S:

First LessWrong post! Been reading everyone's stuff for a while and love the vibe of the community. Please don't hesitate to critique, berate, or ask more questions about this proposal. The past couple of days, I've been acting like AGI will happen tomorrow and coming up with the best ways to create a "safe" environment for AGI use.
Summaries of top forum posts (27th March to 16th April)
by Zoe Williams — 2023-04-17T00:28:18.037Z

This is part of a weekly series summarizing the top posts on the EA and LW forums - you can see the full collection here. The first post includes some details on purpose and methodology. Feedback, thoughts, and corrections are welcomed.

If you'd like to receive these summaries via email, you can subscribe here.

Podcast version: Subscribe on your favorite podcast app by searching for 'EA Forum Podcast (Summaries)'. A big thanks to Coleman Snell for producing these!
Author's note: because of travel / leave, this post covers the past three weeks of posts. We'll be back to our usual weekly schedule from here.

Object Level Interventions / Reviews

AI

FLI open letter: Pause giant AI experiments

by Zach Stein-Perlman

Linkpost for

this open letter

by Future of Life Institute, which calls for “all AI labs to immediately pause for at least 6 months the training of AI systems more powerful than GPT-4.” It has over 2000 signatures, including Yoshua Bengio, Stuart Russell, Elon Musk, Steve Wozniak, and other well-known academics, entrepreneurs and public figures. It’s been covered by

NYT

,

BBC

, and many other media outlets.

Time Ideas also published

an article

by Eliezer Yudkowsky which argues the letter’s ask doesn’t go far enough and 6 months isn’t enough time to solve for safety.

New survey: 46% of Americans are concerned about extinction from AI; 69% support a six-month pause in AI development

by Akash

YouGov America (a reputable pollster) released a survey of 20,810 American adults which found:

46% reported they are “very concerned” or “somewhat concerned” about the possibility of AI causing the end of the human race on Earth.
When prompted with information that >1K tech leaders signed a letter calling for a pause to certain large-scale AI systems for 6 months worldwide, 69% supported this.

There were no meaningful differences by region, gender, or political party.

Announcing Epoch’s dashboard of key trends and figures in Machine Learning

by Jaime Sevilla

Epoch has launched a

new dashboard

, covering key numbers from their research to help understand the present and future of machine learning. Eg. training compute requirements, model size, availability and use of training data, hardware efficiency, algorithmic improvements and investment in training runs over time.

New blog: Planned Obsolescence

by Ajeya, Kelsey Piper

Kelsey Piper and Ajeya Cot...