trentmkelly/LessWrong-43k huggingface dataset:
AI and integrity
by Nathan Young — 2024-05-29T20:45:51.300Z
Link: https://nathanpmyoung.substack.com/p/ai-and-integrity

The blog post contains a preamble about the OpenAI non-disparagement agreement stuff which I assume you all know.

We need to do better than relying on the integrity of individual engineers against millions of dollars paid to them to keep quiet.

Here are three suggestions:

Offer to match 50% of funds lost by whistleblowers.

This might sound like both an insult and a huge waste. But I think people really are motivated by $500k or more, that they expected to earn. That was going to go to their retirement, go on houses or kids schooling. We should consider the possibility that money like this matters.
Make clear that whistleblowers will be taking a hit but not a huge one. I know if i had a family and potentially millions coming to me if I stayed quiet, I would be tempted to. Altman certainly knows this - his gambit around the OpenAI board crisis shows a shrewd understanding of how income motivates his staff.
This is good value for money. How much would some AI safety focused foundations pay to place a person with integrity in OpenAI? I guess it is $1 - $10mn. Here you have someone who can give an accurate account of why they left a well-paying AI Safety role. I suggest that is worth a lot of money, especially if they are willing to forgo half of what they would have made. (This is a costly signal of integrity)

Create an integrity prize

There is much to celebrate here. A man looked at millions and decided he'd rather have the ability to speak honestly. That seems like the sort of behaviour we should want in the world. I want people who would hide Jews under their floorboards, I want people would walk away from interesting scientific problems to avoid building the nuclear bomb (Szilard) and I want people who value their honesty more than millions of dollars when they are developing world-changing tech.
Give a medium sized prize every year. Perhaps $100k as an AI Integrity prize. Find a set of judges who have demonstrated intellectual and practical integrity in the past and get them to vote every year for someone to award it to. Someone who has borne personal cost in AI to maintain their integrity.
Don't be too trusting. If Kokotajlo might the inaugural award have an investigatory team on the staff to kick the tires (perhaps consider Kelsey Piper, who has a reputation of this to maintain). Aim to be 90+% confident that they will still endorse the award in 10 years.

Talk to yo

...
MIRI 2024 Communications Strategy
by Gretta Duleba — 2024-05-29T19:33:39.169Z

As we explained in our

MIRI 2024 Mission and Strategy update

, MIRI has pivoted to prioritize policy, communications, and technical governance research over technical alignment research. This follow-up post goes into detail about our communications strategy.

The Objective: Shut it Down[1]

Our objective is to convince major powers to shut down the development of frontier AI systems worldwide before it is too late. We believe that nothing less than this will prevent future misaligned smarter-than-human AI systems from destroying humanity. Persuading governments worldwide to take sufficiently drastic action will not be easy, but we believe this is the most viable path.

Policymakers deal mostly in compromise: they form coalitions by giving a little here to gain a little somewhere else. We are concerned that most legislation intended to keep humanity alive will go through the usual political processes and be ground down into ineffective compromises.

The only way we think we will get strong enough legislation is if policymakers actually get it, if they actually come to understand that building misaligned smarter-than-human systems will kill everyone, including their children. They will pass strong enough laws and enforce them if and only if they come to understand this central truth.

Meanwhile, the clock is ticking. AI labs continue to invest in developing and training more powerful systems. We do not seem to be close to getting the sweeping legislation we need. So while we lay the groundwork for helping humanity to wake up, we also have a less dramatic request. We ask that governments and AI labs install the “off-switch”[2] so that if, on some future day, they decide to shut it all down, they will be able to do so.

We want humanity to wake up and take AI x-risk seriously. We do not want to shift the Overton window, we want to shatter it.

Theory of Change

Now I’ll get into the details of how we’ll go about achieving our objective, and why we believe this is the way to do it. The facets I’ll consider are:

Audience: To whom are we speaking?
Message and tone: How do we sound when we speak?
Channels: How do we reach our audience?
Artifacts: What, concretely, are we planning to produce?

Audience

The main audience we want to reach is policymakers – the people in a position to enact the sweeping regulation and policy we want – and their staff.

However,...
2024 Summer AI Safety Intro Fellowship and Socials in Boston
by KevinWei — 2024-05-29T18:27:43.168Z

Tl;dr: The

AI Safety Student Team

(a group of students at Harvard) will be running two 8-week introductory reading groups this summer (in Boston and online), as well as summer socials (in Boston). Apply to our technical fellowship here or our policy fellowship here; express interest in our socials here.

Fellowships

We will host two intro fellowships (reading groups):

Intro Technical Fellowship: covers topics such as

neural network interpretability

,

learning from human feedback

,

goal misgeneralization in reinforcement learning agents

, and

eliciting latent knowledge

. Curriculum, application, and FAQ available

here

. Students with machine learning experience are especially encouraged to apply.
Intro Policy Fellowship: covers topics such as

pace of progress in AI

, potential threats from AI

misuse

and

misalignment

,

AI audits and evaluations

, and

semiconductor policy

. Curriculum, application, and FAQ available

here

.

Fellowships are primarily open to current or incoming undergraduate/graduate students at Boston universities, including Harvard and MIT. Recent grads and working professionals who will be in Boston during the 2024-25 academic year are also welcome to apply.

Fellowships will meet weekly in small cohorts with a facilitator experienced in AI safety/policy (either 2 hours in person with dinner, or 1 hour online).

We encourage you to apply (

technical fellowship

,

policy fellowship

) as soon as possible (deadline is Monday, June 10).

Socials

Our socials will be in Boston and are open to students and professionals interested in AI safety. Please fill out

this form

to get information about our socials!
Apollo Research 1-year update
by Marius Hobbhahn — 2024-05-29T17:44:32.484Z

This is a linkpost for: www.apolloresearch.ai/blog/the-first-year-of-apollo-research

About Apollo Research

Apollo Research is an evaluation organization focusing on risks from deceptively aligned AI systems. We conduct technical research on AI model evaluations and interpretability and have a small AI governance team. As of 29 May 2024, we are one year old.

Executive Summary

For the UK AI Safety Summit, we developed a demonstration that Large Language Models (

LLMs) can strategically deceive their primary users when put under pressure

. The accompanying paper was referenced by experts and the press (e.g.

AI Insight forum

,

BBC

,

Bloomberg

) and accepted for oral presentation at the ICLR LLM agents workshop.

The evaluations team is currently working on capability evaluations for precursors of deceptive alignment, scheming model organisms, and a responsible scaling policy (RSP) on deceptive alignment. Our goal is to help governments and AI developers understand, assess, and address the risks of deceptively aligned AI systems.

The interpretability team published three papers:

An improved training method for sparse dictionary learning

,

a new conceptual framework for ‘loss-landscape-based interpretability’

, and

an associated empirical paper

. We are beginning to explore concrete white-box evaluations for deception and continue to work on fundamental interpretability research.

The governance team communicates our technical work to governments (e.g., on evaluations, AI deception and interpretability), and develops recommendations around our core research areas for international organizations and individual governments.

Apollo Research works with several organizations, including partnering with the UK AISI and being a member of the US AISI Consortium. As part of our partnership with UK AISI, we were contracted to develop deception evaluations. Additionally, we engage with various AI labs, e.g. red-teaming OpenAI’s fine-tuning API before deployment and consulting on the deceptive alignment section of an AI lab’s RSP.

Like any organization, we have also encountered various challenges. Some projects proved overly ambitious, resulting in delays and inefficiencies. We would have benefitted from having dedicated regular exchanges with senior official external advisors earlier. Additionally, securing funding took more time and effort than...
Response to nostalgebraist: proudly waving my moral-antirealist battle flag
by Steven Byrnes — 2024-05-29T16:48:29.408Z

@nostalgebraist has recently posted yet another thought-provoking post, this one on how we should feel about AI ruling a long-term posthuman future. [Previous discussion of this same post on lesswrong.] His post touches on some of the themes of Joe Carlsmith’s “Otherness and Control in the Age of AI” series—a series which I enthusiastically recommend—but nostalgebraist takes those ideas much further, in a way that makes me want to push back.

Nostalgebraist’s post is casual, trying to reify and respond to a “doomer” vibe, rather than responding to specific arguments by specific people. Now, I happen to self-identify as a “doomer” sometimes. (Is calling myself a “doomer” bad epistemics and bad PR? Eh, I guess. But also: it sounds cool.) But I too have plenty of disagreements with others in the “doomer” camp (cf: “Rationalist (n.) Someone who disagrees with Eliezer Yudkowsky”.). Maybe nostalgebraist and I have common ground? I dunno. Be that as it may, here are some responses to certain points he brings up.

1. The “notkilleveryoneism” pitch is not about longtermism, and that’s fine

Nostalgebraist is mostly focusing on longtermist considerations, and I’ll mostly do that too here. But on our way there, in the lead-in, nostalgebraist does pause to make a point about the term “notkilleveryoneism”:

They call their position “notkilleveryoneism,” to distinguish that position from other worries about AI which don’t touch on the we’re-all-gonna-die thing. And who on earth would want to be a not-notkilleveryoneist?

But they do not mean, by these regular-Joe words, the things that a regular Joe would mean by them.

We are, in fact, all going to die. Probably, eventually. AI or no AI.

In a hundred years, if not fifty. By old age, if nothing else. You know what I mean.…

OK, my understanding was:

(1) we doomers are unhappy about the possibility of AI killing all humans because we’re concerned that the resulting long-term AI future would be a future we don’t want; and
(2) we doomers are also unhappy about the possibility of AI killing all humans because we are human and we don’t want to get murdered by AIs. And also, some of us have children with dreams of growing up and having kids of their own and being a famous inventor or oh wait actually I’d rather work for Nintendo on their Zelda team or hmm wait does Nintendo hire famous inventors? …And all these lovely aspira

...
Looking beyond Everett in multiversal views of LLMs
by kromem — 2024-05-29T12:35:57.832Z

Over the weekend I was reading up on some very fun exploratory thinking from years ago around large language models through the lens of a quantum multiverse which was extrapolating David Deutsch's parallel between the evolution of state in a quantum system and the generation of virtual realities. The scope of that train of thought was centered on the Everettian many-worlds interpretation of QM, and it seems there hasn't been much thinking since of the same paradigm with other interpretations in mind.

This provides a great opportunity to both explore this concept from a slightly different perspective as well as to highlight the value of the Epicurean approach to information analysis I touched on at the end of a comment on the counterfactual theories they had over a thousand years before contemporaries.

The Epicurean Approach to False Negatives/Positives

The Epicureans, despite not being part of the Socratic 'lineage,' arguably did a much better job than the Platonist line of thinkers at embodying the spirit of Socrates' "All that I know is that I know nothing."

They were desperately concerned with avoiding false negatives to the point that they were outright eager to embrace false positives as long as it was under the auspices of uncertainty (which was prudent, as most of the times they were egregiously wrong was when they erroneously dismissed their peers). For example, Lucretius in De Rerum Natura 4.503-504:

"It’s better to offer erroneous explanations than let slip Any aspect of the graspable out of your grip"

More to the point of this post's framework, after Lucretius introduced the idea of there being an infinite number of other worlds, he later doubles down on the concept of entertaining false positives in the sake of avoiding false negatives by discussing the idea that different worlds might have different correct explanations for a thing, so it was more important to uncover all possible explanations than it was to cull them to a single one (in 5.526-534):

“But which of these is the true cause, it’s hard to ascertain. Rather, it is the possibilities that I explain – What things can and do come about in all the universe In the many worlds created different ways. I give divers Rationales which can explain the motion of the stars In all the worlds – and one of these has to hold true for ours, Empowering stars with motion. Which is right? We cannot say, When we a

...
Inviting discussion of "Beat AI: A contest using philosophical concepts"
by David James — 2024-05-29T11:55:35.603Z

I would like to pose a set of broad questions about a project called Beat AI: A contest using philosophical concepts (details below) with the LessWrong community. My hope would be that we have a thoughtful and critical discussion about it. (To be clear, I'm not endorsing it; I have concerns, but I don't want to jump to conclusions.)

Some possible topics for discussion might include:

Do you know the project or its founder(s)? How and to what extent are they thinking about AI safety, if at all?

If some people decide here that the project seems risky or misguided, do we want to organize our thinking and possibly draft a letter to the project?

Have you seen projects like the one below where a community is invited to compete against AI models? If so, what patterns have you seen?

Beat AI: A contest using philosophical concepts

From its webpage:

The aim of Beat AI is to trick AI systems using your philosophical knowledge. In the process you help us collect data to train better AI models. The game pits you against three models: OpenAI's Ada3-large, BAAI's BGE-large-en-v1.5, and David Bourget's philai-embeddings-v1.1.

By playing, you agree to appearing on the leaderboard and give us a license to use and distribute your submissions. Please read the detailed terms, rules, and tips.

Here is part of the email invitation I received:

I'm writing to invite you to check out Beat AI: A contest using philosophical concepts, a free online game that was just released by the PhilPapers team. The aim is to outwit AI models using your mastery of philosophical concepts. In the process, you will help us develop better AI models for search. Please give it a try and contribute to making PhilPapers better!

https://philpeople.org/beatai

David Bourget
Co-director, PhilPapers

This message was sent to you because you subscribe to the PhilPapers News forum.
AI companies' commitments
by Zach Stein-Perlman — 2024-05-29T11:00:31.339Z

Crossposted from AI Lab Watch. Subscribe on Substack.

AI Lab Watch has

a page on commitments

. It's the best page in the

Resources

section. I intend to keep it up to date. Like the rest of that section, it's not connected to the scorecard.

This post is mostly to announce that page. If that page is missing some commitments (relevant to AI safety and extreme risks or otherwise notable), please let me know. In the rest of this post, I share more abstract remarks on commitments. You should skim the page rather than read this post.

When a lab identifies a good action, it should generally not just (plan to) take it, but also announce that it is doing so. It should also demonstrate that it's doing it, if relevant. This can draw more attention to good actions, make the lab more likely to do them, let the lab get credit for doing them, and help cause other labs to do them. Labs should also sometimes explain (publicly or internally) their plans for various situations; they should distinguish this from making binding commitments.

Humans disagree and are uncertain about risks from AI and appropriate responses. This need not prevent the labs from making good commitments: they can make commitments conditional on dangers. Labs should often commit to safety measures or responses to various scenarios as a function of warning signs, not just in a vacuum. (Related:

Responsible Scaling Policies

.)

Sometimes it would be good if all frontier labs did something, but costly and ineffective for some particular lab to do unilaterally. In this case, the labs should make a conditional commitment: commit to do the thing if they get assurance that all other frontier labs will too, and explain how they could get such assurance.

Various good commitments have not been made by any lab, including:

Once your model has demonstrably tried to escape, stop deploying it

Anything concrete on using external auditors (e.g. in pre-deployment risk assessment)
Whistleblower protection stuff

Never use non-disparagement agreements or otherwise discourage people from publishing concerns (except to prevent release of trade secrets and dangerous information)
One way violinists fail
by Solenoid_Entity — 2024-05-29T04:08:17.675Z

Perfectionism and 'skill dysphoria' as obstacles to healthy contact with reality.

"Fool! You cannot perceive my ✨sound✨ merely by listening to me play!"

I'm currently in a phase of my violin career best described as "aspiring professional."

With me are the best few (several thousand) violin players in my country, competing for 5-10 full-time orchestral jobs that open up each year. We're regularly getting casual work performing, but not enough to make a living. Almost all of us will eventually give up on that goal and take up teaching full-time, play occasional wedding gigs and with pop orchestras, get a day job, and/or marry up[1]. This typically means spending less time practising and playing, which leads to our skills declining in a vicious spiral.

Avoiding concrete measures of skill

If you ever want to see real fear, compliment a violinist on their playing.

"That Mozart was sounding so good just now."

"Oh (panics) you were listening to that? Haha yeah I was just sight-reading through some old stuff, just bashing through it, you know? lol. I'm not even warmed up. This isn't even my violin. My left arm fell off and the doctors put it back on backwards, still getting used to that."

"You sounded great!"

Transforms into a demon

"Fool! You cannot perceive[2] my ✨sound✨ merely by listening to me play!"

Some violinists, particularly as their skill begins to decline after they stop practising 3-6 hours per day, become allergic to public performances.

When they do perform, there is always an excuse justifying why their performance can't be held as a measure of their skill. These excuses are offered unbidden, in bulk, to anyone who will listen, both ahead of and after any performance[3].

I'm actually sight-reading this gig, I'm too busy [being successful] to look at my part ahead of time.[4]
Wow that tempo was pretty fast, wasn't it?!
Weird bowings they chose for that. That's not how I played it [during success].
Haven't actually played my instrument this week yet, hope I'm not too rusty.
My instrument has been sounding weird recently, I should get it looked at.
Been so busy with [success thing], probably shouldn't have agreed to this [lowly thing] but here we are!

While initially this is a pragmatic decision (if you get a reputation for being a poor player, even on one occasion, it's a career disaster as a performer), this shyness eventually morphs into something more path...
Hardshipification
by Jonathan Moregård — 2024-05-28T20:02:29.709Z
Link: https://honestliving.substack.com/p/hardshipification

When I got cancer, all of my acquaintances turned into automatons. Everyone I had zero-to-low degrees of social contact with started reaching out, saying the exact same thing: “If you need to talk to someone, I’m here for you”. No matter how tenuous the connection, people pledged their emotional support — including my father’s wife’s mother, who I met a few hours every other Christmas.

It was only a bit of testicle cancer — what’s the big deal? No Swedish person had died from it for 20 years, and the risk of metastasis was below 1%. I settled in for a few months of suck — surgical ball removal and chemotherapy.

My friends, who knew me well, opted to support me with dark humour. When I told my satanist roommate that I had a ball tumour, he offered to “pop” it for me — it works for pimples, right? To me, this response was pure gold, much better than being met with shallow displays of performative pity.

None of the acquaintances asked me what I wanted. They didn’t ask me how I felt. They all settled for a socially appropriate script, chasing me like a hoard of vaguely condescending zombies.

A Difference in Value Judgements

Here’s my best guess at the origins of their pity:

A person hears that I have a case of the ball cancer
This makes the person concerned — cancer is Very Bad, and if you have it you are a victim future survivor.
The person feels a social obligation to be there for me “in my moment of weakness”, and offer support in a way that is supposed to be as non-intrusive as possible.

Being a Stoic, I rejected the assumption in step #2 as an

invalid value judgement

. The tumor in my ball didn’t mean I was in hardship. The itch after chemotherapy sucked ball(s), and my nausea made it impossible to enjoy the mountains of chocolate people gifted.

These hardships were mild, in the grander scheme of things. I consciously didn’t turn them into a Traumatic Event, something Very Bad, or any such nonsense. I had fun by ridiculing the entire situation, waiting it out while asking the doctors questions like:

Can identical twin brothers transmit testicle cancer through sodomy?
Can I keep my surgically removed ball? (For storing in a jar of formaldehyde)
Does hair loss from chemotherapy proceed in the same stages as male pattern baldness?

Hardshipification

I was greatly annoyed at the people who made a Big Deal out of the situation, “inventing” a hardship out of a situati...