trentmkelly/LessWrong-43k huggingface dataset:
A call for a quantitative report card for AI bioterrorism threat models
by Juno — 2023-12-04T06:35:14.489Z

Introduction

Scientists and policymakers have expressed a great deal of concern recently about biosecurity risks posed by frontier artificial intelligence systems, especially large language models like ChatGPT and Claude. Yet these models also promise to accelerate many areas of work, including biotechnology.

Fear of disease is visceral and the need for biosecurity well-founded, especially in the wake of the coronavirus pandemic. But our security strategies must avoid placing limitations on key advances such as mRNA vaccine research, epidemiological surveillance, and synthetic biology, or on more broadly applicable machine learning systems.

In this essay, I review the existing literature at the intersection of biological safety measures and frontier language models, including both qualitative and quantitative explorations of model capabilities. I discuss three AI-misuse threat models (technical, social engineering, and mis/disinformation), and argue that quantifying the technical threat model is the highest priority. Next, I outline existing metrics.

I propose the creation of a quantitative biotechnology and biosecurity report card for frontier language models. The goal of the report card is to provide transparency around the costs and benefits of including biological capabilities in these systems, and to push for accountability around security measures designed to protect the public in AI and in society broadly.

Dual-use: a tale of a bioweapon and a biopesticide

Biological threats are particularly difficult to detect, mitigate, and attribute, yet biotechnology also offers enormous utility for medicine, agriculture, and materials science. Biotechnology is an excellent example of a dual-use technology, which is generally defined as technology with both civilian and military applications.

The

2001 anthrax attacks in the U.S.

killed five people, sickened seventeen others, and

triggered biodefense spending on the order of tens of billions of dollars in the U.S. alone

. Yet biotechnology also offers enormous utility for medicine, agriculture, and materials science.

A species that is barely distinguishable from anthrax,

Bacillus thuringiensis

(Bt), is among our most potent and widely deployed pesticides — and by 2011, genetically-modified crops producing Bt proteins were valued at

$160 billion globally

.

Anthrax appears on

export control

...
FTL travel summary
by Isaac King — 2023-12-04T05:17:21.422Z

I started writing this 2 years ago, got bored, and never finished. Posting it now just to get it out of my drafts.

Desirability of fast travel

If a superintelligent AI gains power over the physical world, it's very likely to want to expand its influence quickly. This is most obvious for a misaligned SAI such as a paperclip maximizer; if it has a utility function that rewards it for making as many paperclips in a short time frame, it will want to gain access to as large a portion of the universe as possible in a short time.

Perfectly aligned (or only slightly misaligned) SAIs will also want to expand quickly. Even setting aside questions of whether we should colonize the universe, any well-aligned SAI will have a goal of preserving humanity's existence, which means preventing all other civilizations from creating their own misaligned SAIs. The most straightforward way to do this would be to send out probes to other planets that ensure that those other civilizations do not create an SAI that poses a threat to humanity. The faster those probes can move, the better their chance of preventing the rise of SAI with conflicting goals.

If another civilization has already created an SAI with a conflicting goal and they come into contact, there are many ways that could go down, but in most scenarios it will be an advantage to control more of the universe than the other superintelligence does. This again suggests the desirability of developing maximally-fast ways to move across the universe.

FTL travel also increases the amount of the universe we have access to, potentially to an infinite degree. This means that any future civilization that wants to access more negentropy will want to use FTL.

Relevance of FTL

Given the likelihood that superintelligence leads to fast expansion in some form, the question of whether faster-than-light travel is possible has implications for several current questions, and many more far-future ones. For example:

The traditional Fermi Paradox only refers to the lack of any civilizations in the observable universe. (Our past light cone.) If FTL travel is possible however, this question expands to ask about the entire universe. This Bayesian analysis gives a 99% probability lower bound of 251 times the volume of the observable universe. Other estimates have it at 1069 times the volume, or perhaps even infinite. Regardless of its exact size, i

...
Disappointing Table Refinishing
by jefftk — 2023-12-04T02:50:07.914Z

Shaving cream on a table: good kids activity on a wet day, right?

Unfortunately it turns out our table's finish is alcohol-soluble, and
spending a couple hours covered with a high-alcohol foam was not good
for it:

The center leaf I made, and is polyurethane, but we decided the
original portions were probably shellac.

Initially, we put plastic sheeting on it (always have extra plastic
sheeting on hand!) to protect it from further damage:

I looked up how to shellac tables, bought flakes and alcohol, and
mixed up some 1.5lb. I sanded the damaged finish to get it smooth,
but wasn't trying to fully remove it. I did ~5 layers of shellac,
letting it dry overnight between them and covering with plastic during
the day so the table would still be usable.

Unfortunately, it didn't really work: each evening when I went to put
on another layer a few spots of finish would come up with the plastic.
And even over a month later it's not completely cured: when we put a
hot dinner dish on a cork trivet, at the end of the meal there will be
little bits of cork stuck to the table. It's also way less resilient
to scraping and other damage than the original finish was.

At this point we're just ignoring it, but someday I'd like to
fix it properly. Possibly I'll strip the whole surface and put on
polyurethane? Advice?

Comment via: facebook, mastodon
the micro-fulfillment cambrian explosion
by bhauth — 2023-12-04T01:15:34.342Z
Link: https://www.bhauth.com/blog/industrial%20design/microfulfillment.html

Warehouse automation has been very successful. Here's a typical modern system. As you can see, a tall narrow robot rides on a single rail at the top and bottom. The linked example is up to 42m tall. Items are stored on top of pallets, and the robot has a telescoping fork, which might be able to handle 2-deep pallets to improve space efficiency.

Stores are much less automated than warehouses. When you go to a Walmart or Aldi or Ikea, they don't usually have robots in the back - let alone smaller stores. There are now many companies selling automation systems for smaller items and smaller spaces. That's called micro-fulfillment, hereafter "MF".

There are many different configurations being developed and marketed, which indicates that people haven't yet figured out the best approach. Here are some approaches I'm aware of.

Kiva/Amazon

Robots lift an entire rack from below, by driving under it then spinning while turning a ball screw lifter. The rack is carried to a human picker who typically transfers several items. This system was developed by Kiva, which was bought by Amazon and renamed; it now has several clones.

Here's a teardown from 2016. That Kiva design has some problems:

That large ball screw assembly is somewhat expensive for a component.

The robots carrying shelving are top-heavy so they can't accelerate quickly.

The height of shelving is limited by what workers can reach, which limits storage density.

Workers must reach for items that are high up or close to the floor many times a day. A moderate amount of this isn't a big problem, but workers at Amazon facilities need to do that so often that it increases injury rates.

Geek+ RoboShuttle

Elevator robots have a rack-and-pinion driven elevator that lifts a rotating platform. The platform has a grabber that reaches around the sides of totes and slides them onto the elevator. The elevator can them push totes onto fixed storage slots, so it can grab multiple totes in 1 trip.

Carrier robots have a set of powered rollers at a fixed height. Totes can be transferred between them and the elevator robots.

AutoStore

Robots ride on rails on top of a storage cube. They have 2 sets of wheels that can be switched between. Each robot has an elevator system that can lift/lower totes from above. Deep items are dug out, lifting and transferring totes above them until they're available.

Alert Innovation

Robots h

...
Nietzsche's Morality in Plain English
by Arjun Panickssery — 2023-12-04T00:57:42.839Z
Link: https://arjunpanickssery.substack.com/p/nietzsches-morality-in-plain-english

In 1924, Clarence Darrow’s

eight-hour plea

for Leopold and Loeb blamed the universities and scholars of Nietzsche (who died in 1900) for their influence on Leopold:

He became enamored of the philosophy of Nietzsche. Your honor, I have read almost everything that Nietzsche ever wrote. A man of wonderful intellect; the most original philosophy of the last century. A man who had made a deeper imprint on philosophy than any other man within a hundred years, whether right or wrong. More books have been written about him than probably all the rest of the philosophers in a hundred years. … Is there any blame attached because somebody took Nietzsche’s philosophy seriously and fashioned his life on it?

Nietzsche is popularly associated with Nazism and even before this with “the superman … free from scruple” that Darrow describes, but he was also popular among the left-anarchists and the Left generally. Meanwhile, Tyler Cowen

reports

that “if you meet an intellectual non-Leftist, increasingly they are Nietzschean” (whatever that means). Common sense demands that some of these people are misreading him.

Pinning down a moral theory that we can engage faces some initial hurdles:

Nietzsche’s views changed over time. His works appear to make contradictory claims.
His writing is notoriously poetic and obscure.
Huge volumes of notes left behind after his 1889 mental collapse were compiled into The Will to Power and the Nachlass notes. It’s unclear how to consider these since he wanted his notes destroyed after his death.

I favor Brian Leiter’s approach and conclusions in

Nietzsche on Morality

. He offers practical solutions: identifying his works starting from Daybreak (1881) as “mature work,” working to extract philosophical content from even his esoteric output, and avoiding claims that depend on unpublished notes, in part just because they’re low-quality.

Nietzsche’s overarching project is the “revaluation of all values”: a critique of herd morality (which he typically just refers to as “morality”) on the grounds that it’s hostile to the flourishing of the best type of person.

First his broad outlook. Philosophically, he supports a methodological naturalism where philosophy aspires to be continuous with natural or social scientific inquiry. Metaethically he’s an anti-realist about value and would ultimately admit that he's merely defending his evaluative taste.

His psychological vie...
Meditations on Mot
by Richard_Ngo — 2023-12-04T00:19:19.522Z
Link: https://www.mindthefuture.info/p/meditations-on-mot

Holy! Holy! Holy! Holy! Holy! Holy! Holy! Holy! Holy! Holy! Holy! Holy! Holy! Holy! Holy!
Everything is holy! everybody’s holy! everywhere is holy! everyday is in eternity! Everyman’s an angel!
Holy the lone juggernaut! Holy the vast lamb of the middleclass! Holy the crazy shepherds of rebellion! Who digs Los Angeles IS Los Angeles!
Holy New York Holy San Francisco Holy Peoria & Seattle Holy Paris Holy Tangiers Holy Moscow Holy Istanbul!
Holy time in eternity holy eternity in time holy the clocks in space holy the fourth dimension holy the fifth International holy the Angel in Moloch!
- Footnote to Howl

Scene: Carl and Allen, two old friends, are having a conversation about theodicy.

Carl: “Let me tell you about the god who is responsible for almost all our suffering. This god is an ancient Canaanite god, one who has been seen throughout history as a source of death and destruction. Of course, he doesn’t exist in a literal sense, but we can conceptualize him as a manifestation of forces that persist even today, and which play a crucial role in making the world worse. His name is M-”

Allen: “-oloch, right? Scott Alexander’s god of coordination failures. Yeah, I’ve read

Meditations on Moloch

. It’s an amazing post; it resonated with me very deeply.”

Carl: “I was actually going to say

Mot

, the Canaanite god of death, bringer of famine and drought.”

Allen: “Huh. Okay, you got me. Tell me about Mot, then; what does he represent?”

Carl: “Mot is the god of sterility and lifelessness. To me, he represents the lack of technology in our lives. With technology, we can tame famine, avert drought, and cure disease. We can perform feats that our ancestors would have seen as miracles: flying through the air, and even into space. But we’re still so so far from achieving the true potential of technology—and I think of Mot as the personification of what’s blocking us.

“You can see Mot everywhere, when you know what to look for. Whenever a patient lies suffering from a disease that we haven’t cured yet, that’s Mot’s hand at work. Whenever a child grows up in poverty, that’s because of Mot too. We could have flying cars, and space elevators, and so much more, if it weren’t for Mot.

“Look out your window and you see buildings, trees, people. But if you don’t see skyscrapers literally miles high, or trees that have been bioengineered to light the streets, or people who are eternally youthf...
The Witness
by Richard_Ngo — 2023-12-03T22:27:16.248Z
Link: https://www.narrativeark.xyz/p/the-witness

“What are the roots that clutch, what branches grow
Out of this stony rubbish? Son of man,
You cannot say, or guess, for you know only
A heap of broken images-”

I wake up, feeling a strange sense of restlessness. I’m not sure why, but it's impossible to lounge around in bed like I usually do. So I get changed and head down to the kitchen for breakfast. Right as I reach the bottom of the stairs, though, the bell rings. When I open the door, a tall man in a dark suit is standing in front of me.

“Police,” he says, holding up a badge. “Don’t worry, you’re not in trouble. But we do need to talk. Okay if I come in?”

“One second,” I say. “I know everyone in the department, and I don’t recognize you. You new?”

“Yeah, just transferred,” he says. But something in his eyes makes me wary. And none of the cops around here wear suits.

“Got it,” I say, squinting at his badge. “Travis, is it? Just wait outside for me, then, while I call the station to double-check. Can’t be too careful these days.”

As I push the door closed, I see his face twist. His hand rises, and—is he snapping his fingers? I can’t quite make it out before-

I wake up, feeling better than I have in decades. It usually takes me half an hour to get out of bed, these days, but today I’m full of energy. I’m up and dressed within five minutes. Right as I reach the bottom of the stairs, though, the bell rings. When I open the door, a tall man in a dark suit is standing in front of me.

“Police,” he says, holding up a badge. “Don’t worry, you’re not in trouble. But we do need to talk. Okay if I come in?”

“Sure,” I say. A lot of other defense attorneys see the police as enemies, since we usually find ourselves on the other side of the courtroom from them, but I’ve found that it pays to have a good working relationship with the local department. Though I don’t recognize the man in front of me—and actually, he seems way too well-dressed to be a suburban beat cop. Maybe a city detective?

He deftly slides past me and heads straight for my living room, pulling up a chair. He’s talking again before I even sit down. “This will sound totally crazy, so I’m going to start off with two demonstrations.” He picks up a book from the table and tosses it into the air. Before I have a chance to start forward, though, it just… stops. It hangs frozen, right in the middle of its arc, as I gawk at it.

“I—what-”

“Second demonstration," he says. “I’m ...
Does scheming lead to adequate future empowerment? (Section 2.3.1.2 of "Scheming AIs")
by Joe Carlsmith — 2023-12-03T18:32:42.748Z

This is Section 2.3.1.2 of my report “Scheming AIs: Will AIs fake alignment during training in order to get power?”. There’s also a summary of the full report here (audio here). The summary covers most of the main points and technical terms, and I’m hoping that it will provide much of the context necessary to understand individual sections of the report on their own.

Audio version of this section here, or search for "Joe Carlsmith Audio" on your podcast app.

Adequate future empowerment

So far in this analysis of the classic goal-guarding story, we've
discussed whether to expect instrumental training-gaming to in fact
guard the model's goals to the degree that the story requires. I think
this is at least non-obvious – especially for more extreme variants of
the goal-guarding hypothesis, on which the model's goals "crystallize"
entirely after it starts training-gaming. Perhaps, though, empirical
work can shed further light on the matter (see discussion in
section 6.3
below).

For now, let's move to the other key component of the classic
goal-guarding story – what I called "adequate future empowerment" above.
Here, the question is: granted that training-gaming would guard the
model's goals enough for it to be at least somewhat interested in
empowering the future agents produced by training, will those agents in
fact end up empowered to a degree that makes instrumental
training-gaming a good strategy overall?

When is the "pay off" supposed to happen?

We're imagining, here, that the model is instrumentally training-gaming
with some kind of "payoff" in mind – that is, some future point where
the goals-it-supports will reach a position of sufficiently greater
empowerment that it's worth training-gaming now in order to get them
there. But we haven't yet said when, exactly, that future point is
supposed to come. And we can distinguish between a number of different
possibilities.

For example, we can imagine models that would be adequately satisfied if
their goals merely make it to future training-episodes, and no
further. That is, these models would want to training-game even if they
learned that they would never escape from human control and from the
threat of modification.[1] For example, maybe a model that wants to
pursue its curiosity in general thinks that future training episodes
will offer better opportunities for pursuing its curiosity than the
present training-ep...
How do you do post mortems?
by matto — 2023-12-03T14:46:03.521Z

I noticed I don't have a set structure/approach to doing post mortems. Usually I create a simple timeline, consider about what could have gone better, and write down pointers for my future self. But sometimes I don't do a timeline; sometimes I don't write anything down and just reflect on the mistakes I've made.

For context, I do these for projects at work as well as for ones in life. Like, if I spent the last 4 months training up to a 10k run, I'll spend an hour reflecting on what went well and what didn't.

If you practice something like this too, how do you do it? Why? What do you get out of it?
The benefits and risks of optimism (about AI safety)
by Karl von Wendt — 2023-12-03T12:45:12.269Z

This is a reaction to Nora Belrose’s and Quintin Pope’s AI Optimism initiative. However, others are better qualified to criticize the specific arguments they give for their claim that AI is easy to control. Instead, I will focus on the general stance of optimism, when it can be beneficial and when it may be delusional and dangerous.

The benefits of optimism

I have been an optimist all my life. This has led me into many dead-ends, but it has also been the basis for all the successes I achieved. For example, I wanted to write a novel since I was a teenager. I finally sat down to do it when I was 43 and the start-up I had founded three years ago was not doing very well. I sent my first effort, a children’s book, to 20 publishers and got 20 rejections. I wrote a second and third novel which no one wanted to publish (self-publishing wasn’t really an option back then). My fourth novel finally was accepted and became an instant bestseller. I have published almost 70 books since then, and I’m not intending to stop anytime soon.

I define “optimism” as the tendency to weigh positive outcomes higher than their expected value and negative outcomes lower. The probability of my fourth novel becoming a success after three disappointments didn’t seem very high, but I didn’t even think about that. I was optimistic that I had learned something and it was worth the try anyway.

A true Bayesian can never be an optimist (nor a pessimist, which would be the opposite). So optimism must be stupid, right?

Not necessarily. Optimism has obvious and less obvious benefits. One obvious benefit is that it feels better to be optimistic. Optimistic people are also more fun to hang around with, so it's easier for them to make and maintain social connections. Optimism can even become a self-fulfilling prophecy: if you believe in your own success, others tend to believe in you too and will be more willing to help you or fund your efforts.

Our human nature obviously favors optimism and even sometimes recklessness, so there must be an evolutionary advantage behind it. And there is: optimism is a driver for growth and learning, and this makes it easier to adapt to changing circumstances. Imagine you have two populations, one consisting only of "realists" who always make correct decisions based on expected values, while another consists of optimists who will take risks even though the expecte...