trentmkelly/LessWrong-43k huggingface dataset:
minutes from a human-alignment meeting
by bhauth — 2024-05-24T05:01:53.904Z

"OK, let's get this meeting started. We're all responsible for development of this new advanced intelligence 'John'. We want John to have some kids with our genes, instead of just doing stuff like philosophy or building model trains, and this meeting is to discuss how we can ensure John tries to do that."

"It's just a reinforcement learning problem, isn't it? We want kids to happen, so provide positive reinforcement when that happens."

"How do we make sure the kids are ours?"

"There's a more fundamental problem than that: without intervention earlier on, that positive reinforcement will never happen."

"OK, so we need some guidance earlier on. Any suggestions?"

"To start, having other people around is necessary. How about some negative reinforcement if there are no other humans around for some period of time?"

"That's a good one, also helps with some other things. Let's do that."

"Obviously sex is a key step in producing children. So we can do positive reinforcement there."

"That's good, but wait, how do we tell if that's what's actually happening?"

"We have access to internal representation states. Surely we can monitor those to determine the situation."

"Yeah, we can monitor the representation of vision, instead of something more abstract and harder to understand."

"What if John creates a fictional internal representation of naked women, and manages to direct the monitoring system to that instead?"

"I don't think that's plausible, but just in case, we can add some redundant measures. A heuristic blend usually gives better results, anyway."

"How about monitoring the level of some association between some representation of the current situation and sex?"

"That could work, but how do we determine that association? We'd be working with limited data there, and we don't want to end up with associations to random irrelevant things, like specific types of shoes or stylized drawings of ponies."

"Those are weird examples, but whatever. We can just rely on indicators of social consensus, and then blend those with personal experiences to the extent they're available."

"I've said this before, but this whole approach isn't workable. To keep a John-level intelligence aligned, we need another John-level intelligence."

"Oh, here we go again. So, how do you expect to do that?"

"I actually have a proposal: we have John follow cultural norms around having children. We can presume that a society...
Talent Needs of Technical AI Safety Teams
by yams — 2024-05-24T00:36:40.486Z

Co-Authors: @yams, @Carson Jones, @McKennaFitzgerald, @Ryan Kidd

MATS tracks the evolving landscape of AI safety[1] to ensure that our program continues to meet the talent needs of safety teams. As the field has grown, it’s become increasingly necessary to adopt a more formal approach to this monitoring, since relying on a few individuals to intuitively understand the dynamics of such a vast ecosystem could lead to significant missteps.[2]

In the winter and spring of 2024, we conducted 31 interviews, ranging in length from 30 to 120 minutes, with key figures in AI safety, including senior researchers, organization leaders, social scientists, strategists, funders, and policy experts. This report synthesizes the key insights from these discussions. The overarching perspectives presented here are not attributed to any specific individual or organization; they represent a collective, distilled consensus that our team believes is both valuable and responsible to share. Our aim is to influence the trajectory of emerging researchers and field-builders, as well as to inform readers on the ongoing evolution of MATS and the broader AI Safety field.

All interviews were conducted on the condition of anonymity.

Needs by Organization Type

Organization type	Talent needs
Scaling Lab (e.g., Anthropic, Google DeepMind, OpenAI) Safety Teams	Iterators > Amplifiers
Small Technical Safety Orgs (<10 FTE)	Iterators > Machine Learning (ML) Engineers
Growing Technical Safety Orgs (10-30 FTE)	Amplifiers > Iterators
Independent Research	Iterators > Connectors

Here, ">" means "are prioritized over."

Archetypes

We found it useful to frame the different profiles of research strengths and weaknesses as belonging to one of three archetypes (one of which has two subtypes). These aren’t as strict as, say, Diablo classes; this is just a way to get some handle on the complex network of skills involved in AI safety research. Indeed, capacities tend to converge with experience, and neatly classifying more experienced researchers often isn’t possible. We acknowledge past framings by Charlie Rogers-Smith and Rohin Shah (research lead/contributor), John Wentworth (theorist/experimentalist/distillator), Vanessa Kosoy (proser/poet), Adam Shimi (mosaic/palimpsests), and others, but believe our framing of current AI safety talent archetypes i...
How to Give Coming AGI's the Best Chance of Figuring Out Ethics for Us
by sweenesm — 2024-05-23T19:44:42.386Z

[Note: this is a slightly edited version of an essay I entered into the AI Impacts essay contest on the Automation of Wisdom and Philosophy - entries due July 14, 2024. Crossposted to the EA Forum.]

TL;DR

A few possible scenarios are put forth to explore the likely ranges of time we may have from when the first AGI comes online to when an “ethics-bound” AGI would need to be ready to police against any malicious AGI. These ranges are compared against possible time ranges for an AGI to figure out an ethical system for it to operate under. Ethics aren’t expected to be solved “by default” because the first AGI's likely won’t have the ability to feel pain or emotions, have ethical intuitions of their own, or the ability to “try things on” and check them against their own experiences like humans can. Some potential pitfalls in creating a consistent system of ethics, and some “extreme” situations that may test its limits are presented. A list of recommendations is given for what we should do before the first AGI comes online so it can hone in as quickly as possible on a viable ethical system to operate under. Some possible prompts to get an AGI started to figure out ethics are included.

Introduction

It seems likely that Artificial General Intelligence (AGI)

will be developed

within the next 20 years if not significantly before. Once developed, people will want AGI’s to do things for them in the world, i.e., become agentic. AGI’s with limited agency can likely be relatively straightforwardly guard railed from doing significant harm, but as AGI’s are given more autonomy in decision making over a wider range of situations, it’ll become vital that they have a good handle on ethics so they can avoid bringing about potentially massive amounts of value destruction. This is under the assumption, of course, that alignment is “solved” such that an AGI can be made to align its actions with any ethical framework at all. Although there has been

some

work

and

progress

in “machine ethics,” no known “ethics module” is currently available to faithfully guide an AGI[1] to make ethical decisions and act ethically over the wide range of situations it may find itself in. For such a consistent ethics module to be made, it requires a consistent ethical framework to be built on, which we don’t currently have. There’s a good chance we won’t have such a consistent...
Mentorship in AGI Safety (MAGIS) call for mentors 
by Valentin2026 — 2024-05-23T18:28:03.173Z

Tldr: If you are working on AI Safety and are willing to help someone to start their career in AI Safety by sharing your experience at 1:1 meetings consider applying as a mentor

In the last year, we’ve seen a surge of interest in AI safety. Many young professionals and aspiring researchers are attempting or seriously considering a career shift into this and related fields. We’ve seen a corresponding rise in months-long technical bootcamps and research programs, but mentorship has failed to keep pace with this rise. This is a staggering gap, and we intend to fill it - starting now.

Enter the

Mentorship in AGI Safety (MAGIS)

program, a joint initiative between AI Safety Quest and

Sci.STEPS

.

The pilot program will recruit mentors from the community and pair them with mentees according to self-reported background and professional goals, including technical experience, career advice, and soft skills. Mentors will meet with mentees 6 times over 3 months to provide guidance tailored to their specific needs - whether technical expertise, non-technical skills, or career development advice. The program will also provide shared networking and collaboration resources on Discord, and 2+ joint webinar activities for all mentees. According to feedback from the participants of the first season of Sci.STEPS, a mentor program like this one is an immense help to early career researchers.

MAGIS opens with a call for mentors. If you are currently doing research in AI Safety or working in any related area (outreach, technical governance, machine learning, etc.), consider

applying

! By default, we ask that you commit to at least 6 one-hour mentorship meetings over the 3-month program, though you and your mentee(s) might work out a different schedule. At present, this is a volunteer effort; the only reward we can offer is the chance to help someone make an impact at a pivotal moment in their career.

Program Outline

The overall plan for the pilot program in 2024:

Call for mentors. We invite mentors to apply using

this form

. If enough mentors sign up, we will close applications on June 5.
Invite mentees. After we find and select mentors, we will open applications for prospective mentees. We expect them to run June 15 - 31.
Match pairs based on the mentor’s background and mentee's request. This will be done in July.
S

...
Quick Thoughts on Scaling Monosemanticity
by Joel Burget — 2024-05-23T16:22:48.035Z
Link: https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html

1. How Many Features are Active at Once?

Previously I’ve seen the rule of thumb “20-100 for most models”. Anthropic says:

For all three SAEs, the average number of features active (i.e. with nonzero activations) on a given token was fewer than 300

2. Splitting SAEs

Having multiple different-sized SAEs for the same model seems useful. The dashboard shows feature splitting clearly. I hadn’t ever thought of comparing features from different SAEs using cosine similarity and plotting them together with UMAP.

3. Leaky Features

Neither of these plots seems great. They both suggest to me that these SAEs are “leaky” in some sense at lower activation levels, but in opposite ways:

Activating on irrelevant data
Activating unexpectedly weakly on relevant data

For reference, here are the meanings of the specificity scores:

0 – The feature is completely irrelevant throughout the context (relative to the base distribution of the internet).
1 – The feature is related to the context, but not near the highlighted text or only vaguely related.
2 – The feature is only loosely related to the highlighted text or related to the context near the highlighted text.
3 – The feature cleanly identifies the activating text.

Note the low bar for a score of 1 and compare how much area 0 and 1-scored activations take. It looks to me like we can really only trust features above a rough 0.3-0.4 activation level. But note what a small fraction of the total activations have that strength!

As in Towards Monosemanticity, we see that these features become less specific as the activation strength weakens. This could be due to the model using activation strengths to represent confidence in a concept being present. Or it may be that the feature activates most strongly for central examples of the feature, but weakly for related ideas – for example, the Golden Gate Bridge feature 34M/31164353 appears to weakly activate for other San Francisco landmarks. It could also reflect imperfection in our dictionary learning procedure. For example, it may be that the architecture of the autoencoder is not able to extract and discriminate among features as cleanly as we might want. And of course interference from features that are not exactly orthogonal could also be a culprit, making it more difficult for Sonnet itself to activate features on precisely the right examples. It is also plausible that our feature interpret

...
The case for stopping AI safety research
by catubc — 2024-05-23T15:55:18.713Z

TLDR: AI systems are failing in obvious and manageable ways for now. Fixing them will push the failure modes beyond our ability to understand and anticipate, let alone fix. The AI safety community is also doing a huge economic service to developers. Our belief that our minds can "fix" a super-intelligence - especially bit by bit - needs to be re-thought.

I wanted to write this post forever, but now seems like a good time. The case is simple, I hope it takes you 1min to read.

AI safety research is still solving easy problems. We are patching up the most obvious (to us) problems. As time goes we will no longer be able to play this existential risk game of chess with AI systems. I've argued this a lot (ICML 2024 spotlight paper; also www.agencyfoundations.ai). Seems others have this thought.
Capability development is getting AI safety research for free. It's likely in the millions to tens of millions of dollars. All the "hackathons", and "mini" prizes to patch something up or propose a new way for society to digest/adjust to some new normal (and increasingly incentivizing existing academic labs).
AI safety research is speeding up capabilities. I hope this is somewhat obvious to most.

I write this now because in my view we are about 5-7 years before massive human biometric and neural datasets will enter our AI training. These will likely generate amazing breakthroughs in long-term planning and emotional and social understanding of the human world. They will also most likely increase x-risk radically.

Stopping AI safety research or taking it in-house with security guarantees etc, will slow down capabilities somewhat - and may expose capabilities developers more directly to public opinion of still manageable harmful outcomes.
Are most people deeply confused about "love", or am I missing a human universal?
by SpectrumDT — 2024-05-23T13:22:14.313Z

(In the following I am talking about "love" towards human beings only, not love of other things (such as music or food or God).)

A pet topic of mine is that the term love is so ambiguous as to be nigh-useless in rational discourse. But whenever I bring up the topic, people tend to dismiss and ignore it. Let us see if Less Wrong will do likewise.

Modern western culture (and maybe also other cultures) is obsessed with the ideal of love. Love is pretty much by definition the best thing in life which everyone should strive for.

The problem is that people don't agree on what love means.

Everyone will acknowledge that love can mean different things. But my claim is that most people do not truly understand this, even though they think they do. When this is brought up, people will say "oh yes, love can mean different things", but they will go on to act and talk as though love refers to something well-defined.

I would argue that many people treat love as a semantic stop sign. Love is by definition good and beautiful and virtuous and thus needs no further analysis. I have even heard people say that love is too "big" and too ineffable to analyze or define. In my opinion this is a problem, because people do use the term in "rational" discourse.

One might try to resolve the problem by arguing that there are different "kinds" of love:

Altruism.
Platonic love (eg towards friends or children).
Romantic and/or sexual love.
Romantic and/or sexual infatuation.

This helps a bit, but it still does not resolve the problem. Altruism is relatively well-defined, but the other two are still nebulous concepts.

I think a different approach is better. As I see it, the concept of love is garbled mishmash of at least 3 different things:

Love as giving: The drive to protect someone and do stuff for them. (Altruism is a variant of this.)
Love as craving: The desire to be with or "have" someone (sexually or not).
Love as euphoria: The pleasant feeling/emotion that you may sometimes experience when interacting with someone you "love".

These 3 things can co-occur and correlate, but they are clearly distinct things, and it is a mistake to shoehorn them into being 3 "aspects" of the same thing.

Love is usually treated as a binary thing: Either you "love" someone or you don't. This is another misconception that gives rise to bad reasoning. The above 3 things are obviously gradual, not binary, an...
Executive Dysfunction 101
by DaystarEld — 2024-05-23T12:43:13.785Z
Link: https://daystareld.com/executive-dysfunction-101/

This is an intro post for the Procedural Executive Function sequence, which is now complete, to help give some general background of how to orient to executive dysfunction both philosophically and practically.

First things first; “executive dysfunction” is not a diagnosis. "Executive Function" refers to a set of skills that govern our ability to plan actions, take those actions, maintain focus on them, adapt to changes, and more subtle steps between, all in the service of a deliberate intention.

In my Procedural Executive Function posts, I break these skills down as follows:

ADHD is a diagnosis that points to a cluster of common struggles with executive function: working memory, impulse control, and self monitoring. But there are plenty of other diagnoses that can impact one or more of those eight, and of course even things like lack of sleep, hunger, being irritated, disruptive environments, and other stressors can affect them.

So in general when we talk about executive dysfunction what we’re really pointing at is a symptom we witness when someone isn’t able to act on their desires, or on things they think they should do, or on things they think they should desire.

Which brings up the more philosophical question; what does it mean to “fail to act” on a desire? Does someone “have executive dysfunction” if they struggle to complete something they don’t want to do, but feel they have to? What about what they “want to want” to do, but don’t find interesting, even while they can still work on passion projects without issue? Or is it only executive dysfunction if they can’t bring themselves to work on something they feel a strong desire to do, in which case what does “strong desire” mean?

All this makes the question of whether someone struggles with executive dysfunction ill-posed. The better question is “in what domains or in what types of circumstances does someone struggle with executive dysfunction,” followed by narrowing down to which of their executive functions are the chokepoint. Organization? Task initiation? Emotional control?

(I’m also not a fan of “emotional control” as a phrase, as it implies something like stifling or dampening or wrestling with your emotions. This might accurately describe the feeling for some people, but integrating emotions in a healthy way doesn’t have to feel like any of that)

With this more precise understanding, the possibl...
AI #65: I Spy With My AI
by Zvi — 2024-05-23T12:40:02.793Z

In terms of things that go in AI updates, this has been the busiest two week period so far. Every day ends with more open tabs than it started, even within AI.

As a result, some important topics are getting pushed to whenever I can give them proper attention. Triage is the watchword.

In particular, this post will NOT attempt to cover:

Schumer’s AI report and proposal.

This is definitely RTFB. Don’t assume anything until then.

Tyler Cowen’s rather bold claim that May 2024 will be remembered as the month that the AI safety movement died.

Rarely has timing of attempted inception of such a claim been worse.

Would otherwise be ready with this but want to do Schumer first if possible.

He clarified to me that he has walked nothing back.

Remarkably quiet all around, here is one thing that happened.

Anthropic’s new interpretability paper.

Potentially a big deal in a good way, but no time to read it yet.

DeepMind’s new scaling policy.

Initial reports are it is unambitious. I am reserving judgment.

OpenAI’s new model spec.

It looks solid as a first step, but pausing until we have bandwidth.

Most ongoing issues with recent fallout for Sam Altman and OpenAI.

It doesn’t look good, on many fronts.

While the story develops further, if you are a former employee or have a tip about OpenAI or its leadership team, you can contact Kelsey Piper at kelsey.piper@vox.com or on Signal at 303-261-2769.

Also: A few miscellaneous papers and reports I haven’t had time for yet.

My guess is at least six of these eight get their own posts (everything but #3 and #8).

So here is the middle third: The topics I can cover here, and are still making the cut.

Still has a lot of important stuff in there.

Table of Contents

From this week: Do Not Mess With Scarlett Johansson, On Dwarkesh’s Podcast with OpenAI’s John Schulman, OpenAI: Exodus, GPT-4o My and Google I/O Day

Introduction.

Table of Contents.

Language Models Offer Mundane Utility. People getting used to practical stuff.

Language Models Don’t Offer Mundane Utility. Google Search, Copilot ads.

OpenAI versus Google. Similar new offerings. Who presented it better? OpenAI.

GPT-4o My. Still fast and cheap, otherwise people are less impressed so far.

Responsible Scaling Policies. Anthropic offers an update on their thinking.

Copyright Confrontation. Sony joins t

...
What mistakes has the AI safety movement made?
by EuanMcLean — 2024-05-23T11:19:02.717Z

This is the third of three posts summarizing what I learned when I interviewed 17 AI safety experts about their "big picture" of the existential AI risk landscape: how AGI will play out, how things might go wrong, and what the AI safety community should be doing. See here for a list of the participants and the standardized list of questions I asked.

This post summarizes the responses I received from asking “Are there any big mistakes the AI safety community has made in the past or are currently making?”

A rough decompositions of the main themes brought up. The figures omit some less popular themes, and double-count respondents who brought up more than one theme.

“Yeah, probably most things people are doing are mistakes. This is just some random group of people. Why would they be making good decisions on priors? When I look at most things people are doing, I think they seem not necessarily massively mistaken, but they seem somewhat confused or seem worse to me by like 3 times than if they understood the situation better.” - Ryan Greenblatt

“If we look at the track record of the AI safety community, it quite possibly has been harmful for the world.” - Adam Gleave

“Longtermism was developed basically so that AI safety could be the most important cause by the utilitarian EA calculus. That's my take.” - Holly Elmore

Participants pointed to a range of mistakes they thought the AI safety movement had made. Key themes included an overreliance on theoretical argumentation, being too insular, putting people off by pushing weird or extreme views, supporting the leading AGI companies, insufficient independent thought, advocating for an unhelpful pause to AI development, and ignoring policy as a potential route to safety.

How to read this post

This is not a scientific analysis of a systematic survey of a representative sample of individuals, but my qualitative interpretation of responses from a loose collection of semi-structured interviews. Take everything here with the appropriate seasoning.

Results are often reported in the form “N respondents held view X”. This does not imply that “17-N respondents disagree with view X”, since not all topics, themes and potential views were addressed in every interview. What “N respondents held view X” tells us is that at least N respondents hold X, and consider the theme of X important enough to bring up.

Th...