trentmkelly/LessWrong-43k huggingface dataset:
AISN #16: White House Secures Voluntary Commitments from Leading AI Labs and Lessons from Oppenheimer
by Dan H — 2023-08-01T15:39:47.841Z
Link: https://newsletter.safe.ai/p/ai-safety-newsletter-16

Welcome to the AI Safety Newsletter by the

Center for AI Safety

. We discuss developments in AI and AI safety. No technical background required.

Subscribe here to receive future versions.

White House Unveils Voluntary Commitments to AI Safety from Leading AI Labs

Last Friday, the White House

announced

a series of voluntary

commitments

from seven of the world's premier AI labs. Amazon, Anthropic, Google, Inflection, Meta, Microsoft, and OpenAI pledged to uphold these commitments, which are non-binding and pertain only to forthcoming "frontier models" superior to currently available AI systems. The White House also notes that the Biden-Harris Administration is developing an executive order alongside these voluntary commitments.

The commitments are timely and technically well-informed, demonstrating the ability of federal policymakers to respond capably and quickly to AI risks. The Center for AI Safety

supports

these commitments as a precedent for cooperation on AI safety and legally binding legislation.

Commitments to Red Teaming and Information Sharing. AI systems

often develop unexpected capabilities

during training, so it’s important to

screen models for potentially harmful abilities

. The companies promised to “red team” their models by searching for ways the models could cause harm, particularly in

areas of potential catastrophe

including biological weapons acquisition, offensive cyberattacks, and self-replication. External red teamers will also be offered a chance to find vulnerabilities in these frontier AI systems.

These are the catastrophic capabilities that companies agreed to evaluate via red teaming.

The companies have additionally agreed to share information about best practices in AI safety with governments and each other. Public reports on system capabilities and risks are also part of this pledge, though companies have not agreed to the disclosure of model training data, which likely contains large volumes of copyrighted content.

Cybersecurity to protect AI models, if companies choose. The AI labs also reiterated their dedication to cybersecurity. This helps prevent potent AI models from being accessed and misused by malicious actors. By reducing the likelihood that someone else will steal their cutting edge secrets, cybersecurity also alleviates the pressure of the AI race and gives companies more time to evaluate and enhance the safety feat...
“Desperate Honesty” by Agnes Callard
by David Gross — 2023-08-01T13:34:57.180Z
Link: https://dailynous.com/2023/08/01/desperate-honesty-guest-post/

An excellent meditation on intellectual humility as an essential tool in the philosopher's toolbox, and how valuable it is to us when people poke holes in our favorite ideas. Excerpts:

These days I regularly wander outside the gates of philosophy, and it is remarkable to me how often I receive applause for doing what comes absolutely as a matter of course within our discipline. Those praising me seem to imagine that I am constantly lashing myself with the painful whip of intellectual morality.

⋮

The encounter with the mind of a person who is determined to let you get away with absolutely nothing, and who has been extensively trained to succeed at precisely that task—I can’t decide whether that is like stepping in front of a firing squad, or stepping into a cathedral.

⋮

[M]y students are somehow already equipped with opinions, which, if I’m doing my job right, will in short order get launched at me in the form of questions, objections, and counterexamples. Philosophical questions, at least those that matter most to us, show up in the mind already answered. We recognize these questions, they spark of familiarity, because they lie at the foundation of what we’ve been doing all along, and how we have been thinking all along...

Given where my students start—which is not nowhere, but already somewhere—I can’t be in the business of telling them what to think. Rather, I’m forced to be in the business of inviting them to fight me. This is not true in other disciplines. A physics or Greek teacher can safely assume they are either starting with a clean slate, or working with a solid shared foundation. Either way, they’re on level ground, building upwards. The philosophy teacher looks out a landscape that is already full, and not entirely in a good way. In philosophy we don’t build up, at least not without tearing down. Our starting point is not the barren wastes of ignorance, but the clutter of falsity. Philosophy begins in error.

⋮

Realizing you need [your critics] means realizing that they are the ones doing you a favor, that you had better make arguing with you not just palatable, but ideally so pleasant as for them to find it downright delightful. And so yes of course you are going to be a paragon of cheerfulness when they raise their objections, you will be “generous” with criticism, and “fair” to your critics, trying to extract every bit of help from their comments. They

...
Barbieheimer: Across the Dead Reckoning
by Zvi — 2023-08-01T13:00:05.700Z

SPOILER WARNING: This post, after a brief spoiler-free review section, will contain full spoilers for Oppenheimer, Barbie and Mission: Impossible: Dead Reckoning Part One, and some for Across the Spiderverse.

Movies are so back. While they are having their Barbieheimer moment, it seems worthwhile to gather thoughts of myself and others on both movies, and also mention two other recent pictures.

First, I’ll offer various levels of spoiler-free review of all four movies, then get into the weeds.

Spoiler-Free Reviews

Full Spoiler-Free (1-bit reviews, only yes or no):

See all four movies.

Almost Fully Spoiler-Free (several-bit reviews):

You should definitely see Spiderverse, Barbie and Oppenheimer. Mission Impossible is good, but optional.

Pro tip, as it turns out: Do not see Barbie and Oppenheimer on the same day.

Ranked by how pure quality: Across the Spiderverse, Barbie, Oppenheimer, Mission Impossible: Dead Reckoning.

Ranked by how good a time you’ll have: Across the Spiderverse, Barbie, Mission Impossible: Dead Reckoning, Oppenheimer.

Ranked by how important it is to have seen it, and how important it is to ensure everyone sees it: Oppenheimer, Barbie, Across the Spiderverse, Mission Impossible: Dead Reckoning.

Traditional-Level Spoiler-Free Review: Oppenheimer

See it. And remember: It isn’t for you.

By this I mean several things.

If you are reading this, there are things you doubtless already know and messages you do not need to hear, that many others do not know and do need to hear.

Yes, this results in the movie being three hours long and it should have found ways to be shorter, although it is not so easy to make it too much shorter.

One does not go to a movie like this to enjoy it. Appreciate, reflect, experience, take in, learn, understand, cry, remember, yes. Enjoy, no. If you enjoyed this movie, what were you even watching?

Thus, you see this movie for other people. You see it so that you will act in the world as a person who has seen it, and can take that with them as they live.

Also so that you can share this moment with other people, and relate to them and help them take it with them into the world, as well.

This is the central message you are meant to understand and take away from this film, once you know its context: It isn’t for you. None of it is for you.

If you are looking for the ‘I love science’ or ‘h

...
Untangling Infrabayesianism: A redistillation [PDF link; ~12k words + lots of math]
by Lorxus — 2023-08-01T12:42:35.744Z
Link: https://docdro.id/PEehlsN

[Epistemic status: improved redistillation of the infrabayesianism sequence.]

So you want to understand infrabayesianism, to hack to the center of that thorny wood and seek out and recover any treasures hidden there? You've come to a correct creature for a guide. If you want to journey there, make sure you've already got the necessary tools well in hand: some simple decision theory, the basics of topology and linear algebra, and a little measure theory - for that last, if you know how a Lebesgue integral is defined and why no reasonable σ-algebra can encompass the full power set, then you're already doing fine. If you find yourself struggling with such things, reach out to me on Discord or in PMs here and I'll see what we can do.

Infrabayesianism seems like exactly what we might need as alignment researchers: a way to discuss all of our usual decision-theoretic questions while also getting to account for uncertainty about the world, compensate for policy-dependent environments and adversarial selection, and even talk about UDT puzzles. It does this by fundamentally being a decision theory that has explicit reasonable machinery for handling Knightian uncertainty about the environment due to nonrealizable or nonlearnable hypotheses while still permitting nontrivial inference and planning.

Three major brambly hedges block the way between you and understanding: the prickly snagging of the frequently unclear, unintuitive, or just plain lacking notation used in the original infrabayesian sequence; thorny philosophical tangles up front and scattered throughout; and math and its accompanying density of concept and notation getting thicker as we go deeper in. Follow me, though, and we'll slip right through them with barely a scratch, and eat a couple of delicious berries from right off their vines. In fact, I can tell you up front that if you haven't read the original infrabayesianism sequence too closely and aren't that familiar with its notation... that's an active benefit, because we won't need most of it here. We won't be cleaving perfectly to its choices of notation or terminology, though I will eventually provide a dictionary between the two as a postscript.

(I really don't feel like trying to port over 28 PDF pages' worth of TeX and dense math writing here, let alone figure out how to divide it up into several posts, so rather than be redundant I've de...
What Is Childhood Supposed To Be?
by Sable — 2023-08-01T09:51:24.537Z
Link: https://affablyevil.substack.com/p/what-is-childhood-supposed-to-be

I don’t have children.

I’d like to, one day, and I think a lot about what their childhoods might be like, especially as compared to mine.

I look at reports of teen mental health, at college admission rates and tuition levels. I talk to friends of mine who are teachers and involved with the school districts.

I hear about campaigns to eliminate gifted and talented programs in the name of equity.

And I think there’s an important question at the root of all of this that we’re not quite dealing with, as a society.

What is childhood supposed to be, in the first place?

What Is Childhood Supposed To Be?

I see two competing inclinations, when it comes to how first-world countries conceptualize childhood.

The first is childhood as this Elysian paradise, a time spent free of concern or responsibility. It’s the kind of childhood everyone wants to look back on fondly.

The second is childhood as preparation for adulthood - a time spent studying, learning, and experimenting such that the child is ready for the next step.

These two inclinations are fundamentally incompatible: either children have responsibilities or they don’t. Either their time should be spent preparing for adulthood or it shouldn’t.

Which isn’t to say that either inclination is wrong - both are demonstrably capable of producing fully functional adults. Perhaps there is no “right” answer here. I believe that everyone should be able to raise their children as they see fit - but since these inclinations affect things like schools and colleges, they will have an effect on every child, no matter how their parents choose to raise them.

Childhood as an Idyllic Summer

Because we’re dealing with ideas and ideals here more than a specific, concrete vision of a childhood, I’m going to gesture to what I mean by “childhood as an idyllic summer”, and hope the meaning manages to translate appropriately.

As Stephen King says, writing is telepathy, so here are some attempts to transfer my thoughts into your brain:

Phrases like “an Indian summer”, “halcyon days”, or “childish innocence”.

Songs like “Summer of ‘69” and “Boys of Summer”, or even the pop ballads of middle school dances.

Parents taking care of a skinned knee. Ice cream on a hot day. The adventure around every corner.

Comfortable and safe. No expectations or responsibilities.

This is the sort of thing one looks back on fondly. It is the heartbeat of nostalgia, the beginning of ev...
AI romantic partners will harm society if they go unregulated
by Roman Leventov — 2023-08-01T09:32:13.417Z

Recently, when people refer to “immediate societal harms and dangers” of AI, in media or political rhetoric, they predominantly choose to mention “bias”, “misinformation”, and “political (election) manipulation”.

Despite politicians, journalists, and experts frequently compare the current opportunity of regulating AI for good with the missed opportunity to regulate social media in the early 2010s, somehow AI romantic partners are rarely mentioned as a technology and a business model that has the potential to grow very rapidly, harm the society significantly, and be very difficult to regulate once it has become huge (just as social media). This suggests that AI romance technology should be regulated swiftly.

There is a wave of articles in the media (1, 2, 3, 4, for just a small sample) about the phenomenon of AI romance which universally raise vague worries, but I haven’t found a single article that rings the alarm bell that AI romance deserves, I believe.

The EA and LessWrong community response to the issue seems to be even milder: it’s rarely brought up, and a post “In Defence of Chatbot Romance” has been hugely upvoted.

This appears strange to me because I expect, with around 75% confidence, that rapid and unregulated growth and development of AI partners will become a huge blow to society, on a scale comparable to the blow from unregulated social media.

I'm not a professional psychologist and not familiar with academic literature in psychology, but the propositions on which I base my expectation seem at least highly likely and common-sensical to me. Thus, one of the purposes of this article is to attract expert rebuttals of my propositions. If there would be no such rebuttals, the second purpose of the article is to attract the attention of the community to the proliferation of AI romantic partners which should be regulated urgently (if my inferences are correct).

What will AI romantic partners look like in a few years?

First, as Raemon pointed out, it’s crucial not to repeat the mistake that is way too common in the discussions of general risks from AI, that is, to assume only the AI capabilities that are present already will exist and to fail to extrapolate the technology development.

So, here’s what I expect AI romantic partner startups will be offering within the next 2-3 years, with very high probability, because none of these things requires any breakthroug...
What is autonomy, and how does it lead to greater risk from AI?
by Davidmanheim — 2023-08-01T07:58:06.366Z

As with many concepts in discussions of AI risk, terminology around what autonomy is, what agency is, and how they might create risks is deeply confused and confusing, and this is leading to people talking past one another. In this case, the seeming binary distinction between autonomous agents and simple goal-directed systems is blurry and continuous, and this leads to confusion about the distinction between misuse of AI systems and “real” AI risk. I’ll present four simple scenarios along the spectrum, to illustrate.

Four Autonomous Systems

It’s 2028, and a new LLM is developed internally by a financial firm, by doing fine-tuning on a recent open-source model to trade in the market. This is not the first attempt - three previous projects had been started with a $1m compute budget and a $1m funding budget, and they each failed - though the third managed to stay solvent in the market for almost a full month. It is given the instruction to use only funds it was allocated in order to trade, then given unrestricted access to the market.
It is successful, developing new strategies that exploit regularities in HFT systems, and ones that build predictive models of where inefficiencies exist. Because it is running inside a large firm, and training data seems more important than security, it has access to much of the firm’s data, in real time. Unsurprisingly, some of the most profitable strategies are those that would otherwise be illegal; front-running the firm’s customers, or running sentiment analysis non-public conversations about firms that others in the company are not allowed to trade, due to possessing insider information.
Perhaps the trades are reviewed occasionally, and maybe there is even a human in the loop ensuring that each day’s trades are not in some obvious way illegal - but the reasoning is opaque, and the humans doing the review aren’t really looking for misbehavior as long as there is nothing obviously wrong. No one realizes what exactly occurred, and the system is moderately profitable, attracting no further serious attention.

It’s 2028, and a new LLM is released. Despite red-teaming and disabling plug-ins and APIs, hackers around the world quickly build ways to bypass limitations, or the pre-RLHF model is exfiltrated and open-source APIs are built. A rogue state actor, terrorist group, or similar decides to use an AutoGPT-like system, in c

...
Evaluating Superhuman Models with Consistency Checks
by Daniel Paleka — 2023-08-01T07:51:07.025Z
Link: https://arxiv.org/abs/2306.09983

Consider the following two questions:

Is this move good or bad?

Is this forecast accurate?

In both cases, the ground truth is not known to us humans. Furthermore, in both cases there either already exist superhuman AI systems (as in the case of chess), or researchers are actively working to get to a superhuman level (as in the case of forecasting / world modeling). The important question is of course:

How can we evaluate decisions made by superhuman models?

In a

new paper

, we propose consistency / metamorphic testing as a first step towards extending the evaluation frontier. We test:

superhuman chess engine (Leela)
GPT-4 forecasting future events
LLMs making legal decisions

The first part of this post gives a short overview of the paper. The second part elaborates on the motivation, further ideas, and the relevance of this direction for AI safety.

Chess experiments

We test a recent version [1] of

Leela Chess Zero

. This is a superhuman model, trained in a similar MCTS fashion as

AlphaZero

and KataGo.

The "win probability" of a given chess position is the key internal state the model uses to make decisions. Thus, we can expect this quantity to be consistent across semantically equivalent positions.

We find real-game positions played by strong human players where Leela fails to satisfy simple consistency checks:

it has different win probabilities in mirrored positions:

When mirroring a board, the win probability of Black in the first position must be the same as the win probability of White in the second position.

there are drastic changes in win probability after its recommended best move:

Leela Chess Zero only realizes that Bishop to f4 is a bad move after playing it.

To test more invariant properties, we generate random positions without pawns, and check for consistency under board rotations and reflections:

A position without pawns or castling can be flipped or rotated without changing the semantics of the position. In this example, rotating the board counter-clockwise by 90° leads Leela to completely change its evaluation.

The case for working on evaluations now

Most current evaluations require either human-generated answers (almost all tests), or a way to formally check for correctness (mathematical proofs, coding tasks), or just relative comparisons with no objective truth (ELO ratings of chess engi...
Exercise: Solve "Thinking Physics"
by Raemon — 2023-08-01T00:44:48.975Z

Note: please write any answers to this prompt in spoiler-tags.

Recently I set out to deliberate practice at "reasoning about confusing intellectual problems."

Eliezer's Class Project has a fictional group of rationality students try to find the true theory of quantum gravity in one month. This always seemed like a cool goal and test for rationality training to aspire to. If you're not solving difficult open problems faster than science, your Art of Rationality probably isn't complete.

Of course, our Art of Rationality isn't complete yet. But, I think there is something promising in this area, as a way to ground out "rationality training" in something concrete. It seems like good practice to take a given physics question you don't understand the theory behind, and try to invent the theory yourself.

I don't think we're anywhere close to the "definitively impressive" version of rationality practice/training. But, I think a good next step is "Solve Thinking Physics™"

Thinking Physics is a textbook that teaches physics "question-first." Each page presents a physics-y situation, and asks you to figure out what happens next. The questions are multiple choice, but often fairly tricky nonetheless.

I think a good rationalist-training goal is aim for a goal of "be (correctly) 95% confident in the answer", as a rough proxy for "there were no major lingering confusions about the problem except for generic 'maybe I missed something?'". And, failing that, have the subgoal of at least being calibrated about how confused you. Every time you look at an answer, first log your probabilities for each of the multiple-choices in Fatebook.io (or prediction-tracking tool of your choice).

The problems are set up in a way that you can probably reason about them from some basic background knowledge, without much math background. They're ideal for people who don't have much physics background (since the whole point of the book is to teach you physics), although I know people with some physics education who still find it fairly hard.

I spent two weeks working on Thinking Physics problems, and hosting meetups/workshops where other people could join me. With each question, I focused on learning as much as I could about how-to-think.

My original hypothesis was that I could get significantly better at it in 6-8 weeks. I only spent two, and the result so far is I think I'...