trentmkelly/LessWrong-43k huggingface dataset:
OpenAI's CBRN tests seem unclear
by LucaRighetti — 2024-11-21T17:28:30.290Z
Link: https://www.planned-obsolescence.org/openais-cbrn-tests-seem-unclear

This blogpost was written in a personal capacity and statements here do not necessarily reflect the views of any of my employer.

OpenAI says o1-preview can't meaningfully help novices make chemical and biological weapons. Their test results don’t clearly establish this.

Before launching o1-preview last month, OpenAI conducted various tests to see if its new model could help make Chemical, Biological, Radiological, and Nuclear (CBRN) weapons. They report that o1-preview (unlike GPT-4o and older models) was significantly more useful than Google for helping trained experts plan out a CBRN attack. This caused the company to raise its CBRN risk level to “medium” when GPT-4o (released only a month earlier) had been at “low.”[1]

Of course, this doesn't tell us if o1-preview can also help a novice create a CBRN threat. A layperson would need more help than an expert — most importantly, they'd probably need some coaching and troubleshooting to help them do hands-on work in a wet lab. (See my

previous blog post

for more.)

OpenAI

says

that o1-preview is not able to provide "meaningfully improved assistance” to a novice, and so doesn't meet their criteria for "high" CBRN risk.[2] Specifically, the company claims that “creating such a threat requires hands-on laboratory skills that the models cannot replace.”

The distinction between "medium" risk (advanced knowledge) and "high" risk (advanced knowledge plus wet lab coaching) has important tangible implications. At the medium risk level, OpenAI didn't commit to doing anything special to make o1-preview safe. But if OpenAI had found that o1-preview met its definition of “high” risk, then, according to their

voluntary safety commitments

, they wouldn't have been able to release it immediately. They'd have had to put extra safeguards in place, such as removing CBRN-related training data or training it to more reliably refuse CBRN-related questions, and ensure these measures brought the risk back down.[3]

So what evidence did OpenAI use to conclude that o1-preview can't meaningfully help novices with hands-on laboratory skills? According to OpenAI's

system card

, they're developing a

hands-on laboratory test

to study this directly. But they released o1-preview before that test concluded and didn’t share any preliminary results.[4] Instead, they cite three multiple-choice tests as proxies for laboratory help.[5]

These proxy t...
I Have A New Paper Out Arguing Against The Asymmetry And For The Existence of Happy People Being Very Good
by Bentham's Bulldog — 2024-11-21T17:21:41.426Z

Crossposted on my blog.

I have a new paper out titled “

The Sequence Argument Against the Procreation Asymmetry

,” for the journal Utilitas. Sorry for the extreme clickbait title of the paper—very salacious—academic papers are all titled that way.

It’s my third published paper which means I am now a Published Scholar and a Very Important Person who should be deferred to across the board. In it, I argue against the procreation asymmetry, which is the idea that while it’s good to make existing people better off, there’s nothing good about creating a happy person. Asymmetry defenders love the slogan “make people happy, not happy people,” which must be true because it’s catchy.

Suppose you have a child named Jake. According to the procreation asymmetry, having Jake might be good because it helps you or others, but it can’t be good because it helps Jake—after all, he wouldn’t have otherwise existed, so he can’t be worse off (for an explanation of what’s wrong with this reasoning, see

here

).

In other words, while the childless cat ladies might be saddened by not having kids, it wouldn’t be better for the kids if they had them. This paper is about pwning the childless cat ladies! Pat Flynn friend of the blog, has like 10 billion children—the paper argues that the childless cat ladies should do the same (or at least, there’s some strong moral reason to do so, though perhaps it’s outweighed by other stuff).

The argument is as follows. Imagine that you have a button. If you press this button, it will make an existing person better off and create a happy person. Perhaps the button will allow an existing person to live an extra year and create a well off person. Surely pressing the first button would be a good thing.

Then imagine that there’s a second button that would rescind the benefit to the existing person but provide a much greater benefit to the newly created person—the one created by the first button. Perhaps the first button creates a vial of medicine that you had planned to give to an existing person that would give them an extra year of life. The second button instead gives the medicine to the newly created person, giving them an extra 70 years of life.

Now that the person is guaranteed to exist, pressing the second button is clearly worthwhile. Taking an action that provides dramatically greater benefit for someone you’ve created than for a stranger is wo...
Dangerous capability tests should be harder
by LucaRighetti — 2024-11-21T17:20:50.610Z
Link: https://www.planned-obsolescence.org/dangerous-capability-tests-should-be-harder/

Note: I am cross-posting a previous blog post from Planned Obsolescence that I wrote in August

Imagine you’re the CEO of an AI company and you want to know if the latest model you’re developing is dangerous. Some people have argued that since AIs know a lot of biology now —

scoring

in the top 1% of Biology Olympiad test-takers — they could soon

teach terrorists

how to make a nasty flu that could kill millions of people. But others have

pushed back

that these tests only measure how well AIs can regurgitate information you could have Googled anyway, not the kind of specialized expertise you’d actually need to design a bioweapon. So, what do you do?

Say you ask a group of expert scientists to design a much harder test — one that’s ‘

Google-proof

’ and focuses on the biology you’d need to know to design a bioweapon. The UK AI Safety Institute did just that. They

found

that state-of-the-art AIs still performed impressively — as well as biology PhD students who spent an hour on each question and could look up anything they wanted online.

Does that mean your AI can teach a layperson to create bioweapons? Is this result really scary enough to convince you that, as some people have argued, you need to make sure not to

openly share

your model weights, lock them down with

strict cybersecurity

, and do a lot more to make sure your AI

refuses

harmful requests even when people try very hard to

jailbreak

it? Is it enough to convince you to

pause

your AI development until you’ve done all that?

Well, no. Those are really costly actions, not just for your bottom line but for everyone who’d miss out on the benefits of your AI. The test you ran is still pretty easy compared to actually making a bioweapon. For one thing, your test was still just a knowledge test. Making anything in biology, weapon or not, requires more than just recalling facts. It involves designing detailed, step-by-step plans (known as “protocols”) and tailoring them to a specific laboratory environment. As molecular biologist Erika DeBenedicts

explains

:

Often if you’re trying a new protocol in biology you may need to do it a few times to ‘get it working.’ It’s sort of like cooking: you probably aren’t going to make perfect meringues the first time because everything about your kitchen — the humidity, the dimensions, and power of your oven, the exact timing of how long you whipped the egg whites — is a little bit dif

...
Action derivatives: You’re not doing what you think you’re doing
by PatrickDFarley — 2024-11-21T16:24:04.044Z

I want to look at a category of weird mental tricks that we sometimes play on ourselves—you might be familiar with the individual examples, but when considered together they reveal a pattern that I think deserves more attention. I’m going to do the Scott Alexander thing and list a bunch examples in hopes that you’ll sense the common concept they all point at.

Action derivatives index

Here they are:

Meta-preferences: This is when you say you want something, but your actions don’t reflect that wanting, so really we would say you’re wanting to want that thing. If you say, “I want to get stronger,” but you never go to the gym, and it’s totally possible for you to go to the gym, we would say you have a revealed preference for the “not getting stronger” outcome, but your meta preference is to be the kind of person who truly wants to get stronger, i.e. who goes to the gym.

Notice that if you just do the thing, voluntarily, that already implies that you wanted to do it.

Belief in belief – This is a term coined by Daniel Dennett, and it’s described concisely in

The Sequences

. Sometimes when people say “I believe X,” they don’t actually hold any expectations about X, but they believe it’s somehow advantageous or virtuous to believe X. We would say they believe in belief of X.

Notice, if you just say, “X is the case,” you’re already implying that you believe it. I say “I live in NY,” and unless you think I’m lying, it implies that I believe that I live in NY.

Trying to try – This is also in

The Sequences

. When you say, “I’ll try to do X,” that’s a similar kind of extra step.

Notice, if you just say, “I’m going to do X,” that implies that you’re going to try. After all, you’re never a certain predictor of the future: the most you can do is try.
It follows, then, that if you say “I’ll try do X,” you’re implying that you’ll try to try. That’s a very relevant difference for personal productivity, agency, and effectiveness, because if you say “I’ll try,” when really you’re just trying to try, then basically you’ve “succeeded” as soon as you can tell yourself that you tried. And that might be far removed from the real, raw kind of trying that you do when you say, “I’m gonna do X.”

There’s a fourth one that fits with these other three. I’ll state it, but I don’t want to elaborate on it because it kind of touches on culture war topics in

...
AI #91: Deep Thinking
by Zvi — 2024-11-21T14:30:06.930Z

Did DeepSeek effectively release an o1-preview clone within nine weeks?

The benchmarks largely say yes. Certainly it is an actual attempt at a similar style of product, and is if anything more capable of solving AIME questions, and the way it shows its Chain of Thought is super cool. Beyond that, alas, we don’t have enough reports in from people using it. So it’s still too soon to tell. If it is fully legit, the implications seems important.

Small improvements continue throughout. GPT-4o and Gemini both got incremental upgrades, trading the top slot on Arena, although people do not seem to much care.

There was a time everyone would be scrambling to evaluate all these new offerings. It seems we mostly do not do that anymore.

The other half of events was about policy under the Trump administration. What should the federal government do? We continue to have our usual fundamental disagreements, but on a practical level Dean Ball offered mostly excellent thoughts. The central approach here is largely overdetermined, you want to be on the Pareto frontier and avoid destructive moves, which is how we end up in such similar places.

Then there’s the US-China commission, which now have their top priority being an explicit ‘race’ to AGI against China, without actually understanding what that would mean or justifying that anywhere in their humongous report.

Table of Contents

Table of Contents.

Language Models Offer Mundane Utility. Get slightly more utility than last week.

Language Models Don’t Offer Mundane Utility. Writing your court briefing.

Claude Sonnet 3.5.1 Evaluation. Its scored as slightly more dangerous than before.

Deepfaketown and Botpocalypse Soon. AI boyfriends continue to be coming.

Fun With Image Generation. ACX test claims you’re wrong about disliking AI art.

O-(There are)-Two. DeepSeek fast follows with their version of OpenAI’s o1.

The Last Mile. Is bespoke human judgment going to still be valuable for a while?

They Took Our Jobs. How to get ahead in advertising, and Ben Affleck is smug.

We Barely Do Our Jobs Anyway. Why do your job when you already don’t have to?

The Art of the Jailbreak. Getting an AI agent to Do Cybercrime.

Get Involved. Apply for OpenPhil global existential risk portfolio manager.

The Mask Comes Off. Some historical emails are worth a read.

Richard Ngo on Real Power and Governance Futures. Who will have the power?

Introduc

...
Secular Solstice Round Up 2024
by dspeyer — 2024-11-21T10:49:36.682Z

This is a thread for listing Solstice and Megameetup events (dates, locations, links, etc.)

Those of you who have not been may be wondering what a Solstice is in this context.

Secular Solstice is a holiday designed by and for rationalists. It started as an attempt to match the sort of gatherings traditionally religious people tend to have in wintertime, but without compromising on truth.

Is it possible to create that same sense of emotional weight without making stuff up to do it? And without the aid of a long tradition? (Though this is the 14th year we're doing it, so there's some tradition).

Empirically, yes.

Because the world is full of things that genuinely inspire awe. Things entirely worth honoring and celebrating.

And the world is also full of things that genuinely inspire fear. We can and should face them squarely, and together.

This is not a silly holiday. It is not Festivus. It is not Santa Claus and Reindeer. We gather on the longest night of the year to stare together straight into the deepest darkness and *make* *it* *back* *down*.

What does it mean in practice? Lots of singing. Some speeches. Sometimes other things. The whole thing runs a couple of hours.

And what is a Megameetup? It's a meetup that's larger than a normal meetup (though not literally a million times larger). Often it lasts an entire weekend and draws from multiple cities. It may include presentations, organized discussions, games, and other activities rationalists tend to do when in large groups.

What do Solstices and Megameetups have to do with each other? It's common for people to travel significant distances to a Solstice, so attaching a Metameetup makes best use of that travel overhead.
An Epistemological Nightmare
by Ariel Cheng — 2024-11-21T02:08:56.942Z
Link: https://www.mit.edu/people/dpolicar/writing/prose/text/epistemologicalNightmare.html

A short story by Raymond M. Smullyan, written in 1982.

...(A few weeks later.) Frank is in a laboratory in the home of an experimental epistemologist. (You will soon find out what that means!) The epistemologist holds up a book and also asks, "What color is this book?" Now, Frank has been earlier dismissed by the eye doctor as "cured." However, he is now of a very analytical and cautious temperament, and will not make any statement that can possibly be refuted. So Frank answers, "It seems red to me."

Epistemologist:
Wrong!

Frank:
I don't think you heard what I said. I merely said that it seems red to me.

Epistemologist:
I heard you, and you were wrong.

Frank:
Let me get this clear; did you mean that I was wrong that this book is red, or that I was wrong that it seems red to me?

Epistemologist:
I obviously couldn't have meant that you were wrong in that it is red, since you did not say that it is red. All you said was that it seems red to you, and it is this statement which is wrong.

Frank:
But you can't say that the statement "It seems red to me" is wrong.

Epistemologist:
If I can't say it, how come I did?

Frank:
I mean you can't mean it.

Epistemologist:
Why not?

Frank:
But surely I know what color the book seems to me!

Epistemologist:
Again you are wrong.

Frank:
But nobody knows better than I how things seem to me.

Epistemologist:
I am sorry, but again you are wrong.

Frank:
But who knows better than I?

Epistemologist:
I do.

Frank:
But how could you have access to my private mental states?

Epistemologist:
Private mental states! Metaphysical hogwash! Look, I am a practical epistemologist. Metaphysical problems about "mind" versus "matter" arise only from epistemological confusions. Epistemology is the true foundation of philosophy. But the trouble with all past epistemologists is that they have been using wholly theoretical methods, and much of their discussion degenerates into mere word games. While other epistemologists have been solemnly arguing such questions as whether a man can be wrong when he asserts that he believes such and such, I have discovered how to settle such

...
A Conflicted Linkspost
by Screwtape — 2024-11-21T00:37:54.035Z

Over the last couple of years, I’ve been trying to skill up a lot at resolving community complaints. This is a really irritating field to get good at. When I want to get better at writing code, I can sit down and write more code more or less whenever I feel like it. When I want to get better at guitar, I can sit down with my guitar and practice that D to D7 transition. For complaint resolution, even finding people to roleplay the skill with takes a little setup, and that’s a lot less like the real thing than you can get with code or guitar. For real complaints, they come in uneven batches, and worst, you seldom get confirmation if you did it successfully. So, like any good Aspiring Ravenclaw Rationalist, I read a lot of essays about conflict.

Taking a Safety Report

,

Taking Someone Aside

, and

Outcomes of a Safety Report

provide a useful frame and better still useful steps and considerations on what to do when someone comes to you with a concern about an attendee. I keep being tempted to rewrite these in a numbered list, but that’s a stylistic preference. While the topic is very different, I found

A Better Way To Ask for An Apology

to be similarly easier to turn into concrete steps, and it feels like it has a useful overlap with the previous three. To have a good apology, you need to be specific. If you take only one sentence away from this links post, I want it to be “It is also important to clearly distinguish between things you've observed, things you've heard secondhand, and your interpretations of your observations.”

When to Ban Someone

is a discussion of the decision to ban people from events. It has a fairly specific stance on where these decisions should come from, but wears that on its sleeve in a way I appreciate.

Decentralized Exclusion

goes over the situation of distributed communities banning people, even without formal structures. I like the way it looks at the constraints of such communities, but I’m not sure how I feel about the reliance on singular public statements. It puts a lot of onus on that statement to get things right, and it isn’t clear to me how a true statement works better than a false statement. (See Guided By The Beauty Of Our Weapons- a big statement is asymmetric, but not very asymmetric.) They're both a bit prescriptive though (When to Ban moreso than Decentralized Exclusion) wh...
DeepSeek beats o1-preview on math, ties on coding; will release weights
by Zach Stein-Perlman — 2024-11-20T23:50:26.597Z

DeepSeek-R1-Lite-Preview was announced today. It's available via chatbot. (Post; translation of Chinese post.)

DeepSeek says it will release the weights and publish a report.

The model appears to be stronger than o1-preview on math, similar on coding, and weaker on other tasks.

DeepSeek is Chinese. I'm not really familiar with the company. I thought Chinese companies were at least a year behind the frontier; now I don't know what to think and hope people do more evals and play with this model. Chinese companies tend to game benchmarks more than the frontier Western companies, but I think DeepSeek hasn't gamed benchmarks much historically.

The post also shows inference-time scaling, like o1:

Note that o1 is substantially stronger than o1-preview; see the o1 post:

(Parts of this post and some of my comments are stolen from various people-who-are-not-me.)
Expected Utility, Geometric Utility, and Other Equivalent Representations
by StrivingForLegibility — 2024-11-20T23:28:21.826Z

In Scott Garrabrant's excellent Geometric Rationality sequence, he points out an equivalence between modelling an agent as

Maximizing the expected logarithm of some quantity V, E[ln(V)]
Maximizing the geometric expectation of V, G[V]

And as we'll show in this post, not only can we prove a geometric version of the VNM utility theorem:

An agent is VNM-rational if and only if there exists a function V that:

Represents the agent's preferences over lotteries

L≺M if and only if V(L)<V(M)

Agrees with the geometric expectation of V

V=G[V]

Which in and of itself is a cool equivalence result, that E[U] maximization ⟺ VNM rationality ⟺ G[V] maximization. But it turns out these are just two out of a huge family of expectations we can use, like the harmonic expectation H, which each have their own version of the VNM utility theorem. We can model agents as maximizing expected utility, geometric utility, harmonic utility, or whatever representation is most natural for the problem at hand.

Expected Utility Functions

The VNM utility theorem is that an agent satisfies the VNM axioms if and only if there exists a utility function U:ΔΩ→R which:

Represents that agent's preferences over all lotteries

L≺M if and only if U(L)<U(M)

Agrees with its expected value

U=E[U]

Where ΔΩ is a probability distribution over outcomes ω∈Ω.

The first property is easy to preserve. Given any strictly increasing function f,

f(U(L))<f(U(M)) if and only if U(L)<U(M)

So f∘U also represents our agent's preferences. But it's only affine transformations f(U)=aU+b that preserve the second property that f(U)=E[f(U)]. And it's only increasing affine transformations that preserve both properties at once.

f-Utility Functions

But what if we were interested in other ways of aggregating utilities? One of the central points of Scott's Geometric Rationality sequence is that in many cases, the geometric expectation G is a more natural way to aggregate utility values V into a single representative number G[V].

We can represent the same preferences using the expected logarithm E[ln(V)], but this can feel arbitrary, and having to take a logarithm is a hint that these quantities V are most naturally combined by multiplying them t...