trentmkelly/LessWrong-43k huggingface dataset:
List of technical AI safety exercises and projects
by JakubK — 2023-01-19T09:35:18.171Z
Link: https://docs.google.com/document/d/1-58zgC2lRMbMK-CXU44VR3ApGYbTI0aJKX-cKkxDeyo/edit?usp=sharing

EDIT 3/17/2023: I've reorganized the doc and added some governance projects.

I intend to maintain a list at this doc. I'll paste the current state of the doc (as of January 19th, 2023) below. I encourage people to comment with suggestions.

Levelling Up in AI Safety Research Engineering [Public]

(

LW

)

Highly recommended list of AI safety research engineering resources for people at various skill levels.

AI Alignment Awards

Alignment jams / hackathons

from

Apart Research

Past / upcoming hackathons:

LLM

,

interpretability 1

,

AI test

,

interpretability 2

Projects on AI Safety Ideas:

LLM

,

interpretability

,

AI test

Resources:

black-box investigator of language models

,

interpretability playground

(

LW

),

AI test

Examples of past projects

;

interpretability winners

How to run one

as an in-person event at your school

Neel Nanda:

200 Concrete Open Problems in Mechanistic Interpretability

(

doc

and

previous version

)

Project page from AGI Safety Fundamentals

and their

Open List of Project ideas

AI Safety Ideas

by Apart Research;

EAF post

Most Important Century writing prize

(

Superlinear page

)

Center for AI Safety

Competitions

like

SafeBench

Student ML Safety Research Stipend Opportunity – provides stipends for doing ML research.

course.mlsafety.org projects

CAIS is looking for someone to add details about these projects on

course.mlsafety.org

Distilling

/ summarizing / synthesizing / reviewing / explaining

Forming your own views on AI safety (without stress!)

– also see

Neel's presentation slides

and "

Inside Views Resources

" doc
Answer some of the

application questions

from the winter 2022

SERI-MATS

, such as

Vivek Hebbar's problems

10 exercises from Akash

in “Resources that (I think) new alignment researchers should know about”

[T] Deception Demo Brainstorm

has some ideas (message

Thomas Larsen

if these seem interesting)
Upcoming

2023 Open Philanthropy AI Worldviews Contest

Alignment research at ALTER

– interesting research problems, many have a theoretical math flavor

Open Problems in AI X-Risk [PAIS #5]

Amplify creative grants

(old)
Evan Hubinger:

Concrete experiments in inner alignment

,

ideas someone should investigate further

,

sticky goals

Richard Ngo:

Some conceptual alignment research projects

,

alignment research exercises

Buck Shlegeris:&nb

...
Group-level Consequences of Psychological Problems
by adamShimi — 2023-01-19T09:27:13.920Z

In

a previous post

, we introduced the distinction between psychological disorders (massive life-shattering issues like some forms of schizophrenia and drug addiction) from psychological problems (more minor issues like being anger-driven or avoiding parties). And we highlighted that although disorders are obviously important to address, neglecting psychological problems comes at a cost for our ability to optimize.

What are the consequences of this model for groups, particularly in terms of norms?

First, let’s avoid

reification bias

here: we’re not looking for an inherent notion of bad or good norms, of moral rightness of groups and things like that. Instead we’re interested in incentive structures, and their consequences for the performance and health of the group.

So what happens if members of a group don’t address their psychological problems, and the group doesn’t incentivize dealing with these?

Generation of new psychological problems

Example: Person A doesn’t like working in the group office. Yet they are still required to do so by group norms to boost productivity and collaboration. However, if they have a health issue (being tired, feeling a bit sick today, etc.), now from the productivity perspective letting them work from home is better. This creates incentives to find themselves in situations where they say and feel that they are too tired, for example:

Lie about being tired.
Pay too much attention to their tiredness, as it gives them an excuse (therefore making them actually feel more tired!)
Trick themselves into thinking they are tired. If they have feelings that are close to tiredness, and they’re unsure of how to interpret them, they’ll feel a slight pull toward tiredness.
Sleep late, as they can genuinely say the next day that they are tired and get to work from home.

Generation of friction and constraints

Example: Person A suffers from anxieties about publishing, although they’re the best person to write and edit an important post, and so the post gets delayed again and again.
Example: Person A needs a given task done asap, and is the kind of person who needs a tight control over the process, but Person B who is actually doing the tasks can’t stand micro-managing and control.

Pushing toward lowest-common-denominator actions

Example: Group manager/president/leader figures out that the best strategic approach to take is to do X. Yet they know

...
6-paragraph AI risk intro for MAISI
by JakubK — 2023-01-19T09:22:23.426Z
Link: https://www.maisi.club/about

The Michigan AI Safety Initiative (MAISI) is a new AI safety student group at the University of Michigan. The website's "About" page includes a short intro to AI risk. I'm sharing it here for people who are interested in short pitches for AI x-risk. Feel free to comment with feedback / suggestions / criticisms.

Will AI really cause a catastrophe?

Hopefully not! AI has tremendous potential for making the world a better place, especially as the technology continues to develop. We’re already seeing some beneficial applications of AI to healthcare, accessibility, language translation, automotive safety, and art creation, to name just a few. However, advanced AI also poses some serious risks.

At the very least, malicious actors could use AI to cause harm, e.g. building dangerous weapons, spreading fake news, empowering oppressive regimes, and more.

More speculatively, advanced AI systems could potentially seek power or control over humans. It’s possible that future AI systems will be qualitatively different from those we see today. They may be able to form sophisticated plans to achieve their goals, and also understand the world well enough to strategically evaluate many relevant obstacles and opportunities. Furthermore, they may attempt to acquire resources or resist shutdown attempts, since these are useful strategies for some goals their designers might specify. To see why these failures might be challenging to prevent, see this research on specification gaming and goal misgeneralization from DeepMind.

It’s worth reflecting on the possibility that an AI system of this kind could outmaneuver humanity’s best efforts to stop it. Meta’s

Cicero

model demonstrated that AI systems can successfully negotiate with humans when it reached human-level performance in

Diplomacy

, a strategic board game, so an advanced AI system could manipulate humans to assist it or trust it. In addition, AI systems are swiftly becoming proficient at writing computer code with models like

Codex

. Combined with models like

WebGPT

and ACT-1, which can take actions on the internet, it seems that advanced AI systems could be formidable computer hackers. Hacking creates a variety of opportunities; e.g., an AI system might steal financial resources to purchase more computational power, enabling it to train longer or deploy copies of itself.

Maybe none of this will happen. Indeed...
200 COP in MI: Studying Learned Features in Language Models
by Neel Nanda — 2023-01-19T03:48:23.563Z

This is the final post in a sequence called 200 Concrete Open Problems in Mechanistic Interpretability.

Start here

, then read in any order. If you want to learn the basics before you think about open problems, check out

my post on getting started

. Look up jargon in

my Mechanistic Interpretability Explainer

Motivation

Motivating Paper:

Softmax Linear Units (SoLU)

,

Multimodal Neurons

To accompany this post, I’ve created a website called Neuroscope that displays the text that most activates each neuron in some language models -

check it out

!

This section contains a lot of details and thoughts on thinking about neurons and learned features and relationship to the surrounding literature. If you get bored, feel free to just skip to exploring and looking for interesting neurons in

Neuroscope

MLPs represent ⅔ of the parameters in a transformer, yet we really don’t understand them very well. Based on our knowledge of image models, my best guess is that models learn to represent features, properties of the input, with different directions corresponding to different features. Early layers learn simple features that are basic functions of the input, like edges and corners, and these are iteratively refined and built up into more sophisticated features, like angles and curves and circles. Language models are messier than those image models, since they have attention layers and a residual stream, but our guess is that analogous features are generally computed in MLP layers.

But there’s a lot of holes and confusions in this framework, and I’d love to have these filled in! How true actually is this in practice? Do features correspond to neurons vs arbitrary directions? What kinds of features get learned, and where do they occur in a model? What features do we see in small models vs large ones vs enormous ones? What kinds of things are natural for a language model to express vs extremely hard and convoluted? What are the ways our intuitions will trip us up here?

Issues like polysemanticity and superposition make it difficult to actually reverse engineer specific neurons, and it seems clear that the model can learn features that do not correspond to specific neurons. But even if we relax our standards of rigour and just want to understand what features are present at all, we don’t know much about what role these layer...
Amazon closing AmazonSmile to focus its philanthropic giving to programs with greater impact
by Gordon Seidoh Worley — 2023-01-19T01:15:09.693Z

I just received the following email. There's also a notice up on their website. Since many folks here use AmazonSmile to make additional donations to charities, thought folks should know.

Dear customer,
In 2013, we launched AmazonSmile to make it easier for customers to support their favorite charities. However, after almost a decade, the program has not grown to create the impact that we had originally hoped. With so many eligible organizations—more than 1 million globally—our ability to have an impact was often spread too thin.
We are writing to let you know that we plan to wind down AmazonSmile by February 20, 2023. We will continue to pursue and invest in other areas where we’ve seen we can make meaningful change—from building affordable housing to providing access to computer science education for students in underserved communities to using our logistics infrastructure and technology to assist broad communities impacted by natural disasters.
To help charities that have been a part of the AmazonSmile program with this transition, we will be providing them with a one-time donation equivalent to three months of what they earned in 2022 through the program, and they will also be able to accrue additional donations until the program officially closes in February. Once AmazonSmile closes, charities will still be able to seek support from Amazon customers by creating their own wish lists.
As a company, we will continue supporting a wide range of other programs that help thousands of charities and communities across the U.S. For instance:

Housing Equity Fund: We’re investing $2 billion to build and preserve affordable housing in our hometown communities. In just two years, we’ve provided funding to create more than 14,000 affordable homes—and we expect to build at least 6,000 more in the coming months. These units will host more than 18,000 moderate- to low-income families, many of them with children. In one year alone, our investments have been able to increase the affordable housing stock in communities like Bellevue, Washington and Arlington, Virginia by at least 20%.
Amazon Future Engineer: We’ve funded computer science curriculum for more than 600,000 students across over 5,000 schools—all in underserved communities. We have plans to reach an additional 1 million students this year. We’ve also provided immediate assistance to 55,000 students in our hometow

...
Gradient Filtering
by Jozdien — 2023-01-18T20:09:20.869Z

Several of the ideas in this post originated in conversations between the authors months ago. I did the work involved in exploring them and thinking of new ideas and framings, as well as writing this post; old writings by Janus were the inspiration for some of the directions of thought here.

Thanks to Adam Jermyn and Paul Colognese for their feedback and thoughts.

Generally speaking,

gradient hacking

can seem pretty hard for an agent to do. But the dangers don’t lie in the galaxy-brained strategies that models with strong deceptive capabilities can implement, it’s in the idea that a model’s gradient can be influenced by an inner agent at all. So examining variants that are seemingly more accessible would be pretty useful.

This post discusses gradient filtering, one such variant where a model is made to generate outputs that are wrong in ways that lead gradient descent to update the model in a direction desirable by some malicious agent. It aims broadly to make the case for gradient hacking being more plausible than previously discussed - because unlike other imagined variants gradient filtering does not require the agent to know or understand the policy's internal implementation - and also to explore some aspects of

simulator theory

.

First, we lay out a scenario where gradient filtering could play a role, and how this benefits the model. Then, we discuss specific conditions necessary in its arisal, specifically in the context of deceptive simulacra (although the points should be generally applicable), as well as arguments both for and against them being fulfilled. Finally, we talk about how simulacra may need to be focused by the model to be capable of coming up with this kind of strategy.

I think this is a topic of some importance because few of the conditions for some form of gradient hacking arising are very restrictive, and the rest still have plausible pathways in which they could arise. Further, gradient hacking is likely to be pretty dangerous if feasible, reducing the likelihood that deceptive agents would be solved by default, making deceptive agents more powerful, and requiring potential solutions to be even stronger.

That said, there are definitely arguments against its plausibility and ultimate danger, as we’ll see. I think they’re far from being strong enough to dismiss this as a problem, but they make the question ambiguous enough that I’m ...
[Cross-post] Is the Fermi Paradox due to the Flaw of Averages?
by Aryeh Englander — 2023-01-18T19:22:01.944Z
Link: https://lumina.com/fermi-paradox-resolved-by-flaw-of-averages-2/

[This article is copy-pasted from the Lumina blog, very lightly edited for LessWrong.]

Where is everybody?
— Enrico Fermi

The omnipresence of uncertainty is part of why making predictions and decisions is so hard. We at Lumina advocate treating uncertainty explicitly in our models using probability distributions. Sadly this is not yet as common as it should be. A recent paper “Dissolving the Fermi Paradox” (2018) is a powerful illustration of how including uncertainty can transform conclusions on the fascinating question of whether our Earth is the only place in the Universe harboring intelligent life. The authors, Anders Sandberg, Eric Drexler and Toby Ord (whom we shall refer to as SDO), show elegantly that the apparent paradox is simply the result of the mistake of ignoring uncertainty, what Sam L. Savage calls the Flaw of Averages. In this article, we review their article and link to an Analytica version of their model that you can explore.

The Fermi Paradox

Enrico Fermi. From Wikimedia commons.

One day in 1950, Enrico Fermi, the Nobel prize-winning builder of the first nuclear reactor, was having lunch with a few friends in Los Alamos. They were looking at a New Yorker cartoon of cheerful aliens emerging from a flying saucer. Fermi famously asked “Where is everybody?”. Given the vast number of stars in the Milky Way Galaxy and the likely development of life and extraterrestrial intelligence, how come no ETs have come to visit or at least been detected? This question came to be called the “Fermi Paradox”. Ever since, it has bothered those interested in the existence of extraterrestrial intelligence and whether we are alone in the Universe.

The Flaw of Averages on Steroids

Dr. Sam Savage who coined the term “Flaw of Averages”

Sam L. Savage, in his book, The Flaw of Averages, shows how ignoring uncertainty and just working with a single mean or “most likely” value for each uncertain quantity can lead to misleading results. To illustrate how dramatically this approach can distort your conclusions, SDO offer a toy example. Suppose there are nine factors that multiplied together give the probability of extraterrestrial intelligence (ETI) arising on any given star. If you use a point estimate of 0.1 for each factors, you could infer that there is a 10−9probability of any given star harboring ETI. There are about 1011 stars in the M...
First Three Episodes of The Filan Cabinet
by DanielFilan — 2023-01-18T19:20:06.588Z

I have a new podcast called “The Filan Cabinet”. It will be similar to AXRP, except instead of interviewing researchers about their work to study and reduce AI existential risk, I interview whoever I want to talk to about whatever I want to talk about. There are already three episodes you can listen to.

In epsiode 1, Carrick Flynn talks about his campaign to be the Democratic nominee for Oregon’s 6th congressional district. In particular, we talk about his policies on pandemic preparedness and semiconductor manufacturing. He answers questions such as:

Was he surprised by the election result?

Should we expect another Carrick campaign?

What specific things should or could the government fund to limit the spread of pandemics? Why would those work?

What is working at a semiconductor plant like?

In episode 2, Presbyterian Pastor Wayne Forkner talks about God, Christianity, and the Bible. He answers questions such as:

What is ‘God’?

Why do people talk about Jesus so much more than the Father or the Holy Spirit?

What is heaven actually like?

If justification is by faith alone and not by works, why does the Bible say “A person is justified by works and not by faith alone”?

How can people tell that out of all the religions, Christianity is the right one?

In episode 3, cryptocurrency developer Ameen Soleimani talks about his vision of the cryptocurrency ecosystem, as well as his current project RAI: an ether-backed floating-price stablecoin. He answers questions such as:

What’s the point of cryptocurrency?

If this is the beginning of the cryptocurrency world, what will the middle be?

What would the sign be that cryptocurrency is working?

How does RAI work?

Does the design of RAI make it impossible for it to be widely used?

What’s wrong with how the US dollar works?

I hope you check it out - you can search “The Filan Cabinet” wherever you listen to podcasts!
Best Questions To Vet Potential Ai-Safety Applicants
by jacksonjezion — 2023-01-18T19:01:33.176Z

Hey guys,

I run a major recruiting firm in India working with Tech companies and wanted to use some of that access to the workforce to get the highly talented Ai people into alignment. the cool thing about India is that the cost of living is so low that full-time talented people in this field) can be snagged at 20k-50k a year.
The question I have is 2 part,

In order to qualify the applicants, what questions would be good?
Once I have qualified applicants ready to go, is there any companies or places actively hiring I can help get them onboarded with? No fees btw, this is my donation to the alignment field.
If there is a need by the business to handle the regulatory part of hiring internationally we have that covered.
Looking for a specific group of people
by  — 2023-01-18T19:00:59.393Z

I could use some help.

There exists somewhere a group of people that takes the whole "systemized winning" thing extremely seriously. Their ambition probably sounds insane to normal people. These people set themselves unreasonably difficult goals and achieve them anyways. They develop systems and tools to assist them, and in that group, share them.

Such a group doesn't advertise itself, and probably works by invitation. I don't know where they are or could be, but I want to find them anyways. This is where you guys come in:

I need you to point me in the direction of another group, community, discord server, subreddit, forum or whatever that gets me *closer* to what I'm looking for. *It doesn't have to be the group itself, just closer than this group*. Please let me know if any of you have an idea on where to go next.

Thanks!

FAQ

- "I don't think such a group exists"
- https://en.wikipedia.org/wiki/Small-world_experiment - Again, I'm not asking you to point me to the group, but to another group that's closer to my target group. It doesn't matter that you don't know of such a group, but just that I get closer. Imagine the small-world experiment, but I'm the letter going from group to group, so to speak.

- "Isn't that kind of desperate/low status/cringe to put that effort into finding that group?"
- Maybe. Status considerations don't hold a lot of weight, so I'm doing it anyway.

- "What makes you think they'd take you in even if you find them?"
- There's no point in premature capitulation, they can reject me once I get there.