trentmkelly/LessWrong-43k huggingface dataset:
Biorisk is an Unhelpful Analogy for AI Risk
by Davidmanheim — 2024-05-06T06:20:28.899Z

There are two main areas of catastrophic or existential risk which have recently received significant attention; biorisk, from natural sources, biological accidents, and biological weapons, and artificial intelligence, from detrimental societal impacts of systems, incautious or intentional misuse of highly capable systems, and direct risks from agentic AGI/ASI. These have been compared extensively in research, and have even

directly inspired policies

. Comparisons are often useful, but in this case, I think the disanalogies are much more compelling than the analogies. Below, I lay these out piecewise, attempting to keep the pairs of paragraphs describing first biorisk, then AI risk, parallel to each other.

While I think the disanalogies are compelling, comparison can still be useful as an analytic tool - while keeping in mind that the ability to directly learn lessons from biorisk to apply to AI is limited by the vast array of other disanalogies. (Note that this is not discussing the interaction of these two risks, which is a critical but different topic.)

Comparing the Risk: Attack Surface

Pathogens, whether natural or artificial, have a fairly well-defined attack surface; the hosts’ bodies. Human bodies are pretty much static targets, are the subject of massive research effort, have undergone eons of adaptation to be more or less defensible, and our ability to fight pathogens is increasingly well understood.

Risks from artificial intelligence, on the other hand, have a near unlimited attack surface against humanity, not only including our deeply insecure but increasingly vital computer systems, but also our bodies, our social, justice, political, and governance systems, and our highly complex and interconnected but poorly understood infrastructure and economic systems. Few of these are understood to be robust, and the classes of failures are both manifold, and not adapted or constructed for their resilience to attack.

Comparing the Risk: Mitigation

Avenues to mitigate impacts of pandemics are well explored, and many partially effective systems are in place. Global health, in various ways, is funded with on the order of tens of trillions of dollars yearly, much of which has been at times directly refocused on fighting infectious disease pandemics. Accident risk with pathogens is a major area of focus, and while manifestly insufficient to stop all accide...
Some Problems with Ordinal Optimization Frame
by Mateusz Bagiński — 2024-05-06T05:28:42.736Z

A result of work done during AI Safety Camp Virtual 2024. Thanks[1] to Alex Altair, Alfred Harwood, Amaury Lorin, Jasmina Nasufi, Tyler Tracy, and Einar Urdshals for related conversations and/or feedback on the post.

TL;DR: The ordinal definition of optimization (i.e. pushing the state of the world up some preference ordering without a [utility/reward/value]-like measure on the state) appears to be quite limiting, casting some doubt on its epistemic value/usefulness.

1. Introduction and definitions

Given a set of world states W, a total ordering Θ over them, and an update function that maps a previous world state to the next one (update:W→W), optimization can be defined as the process of the world states being reliably sent to world states that are higher in the ordering Θ. We then say that the world states are getting increasingly optimized. We then call that Θ a "preference ordering". If wt+1=update(wt), wt+1 is "weakly better" than wt, if wt+1 is weakly Θ-greater than wt (written wt≾Θwt+1). If Θ assigns the ordinal ω (e.g. an integer, or any other value of some totally ordered set) to w∈W, we say that ω is the Θ-degree of w.

To talk about optimization in this sense, we don't need to assume that the world is divided into the "agent"/"optimizer"/"policy" and the "environment". For example, given a container with gas as our W, we can define Θ over W, according to the entropy of the gas, such that higher entropy states are "better" (Θ-greater). If the container starts in a very ordered state (e.g. molecules of one type taking up one half of the container and molecules of the other type taking up the other half), then the gas in the container will transition to a more Θ-optimal state, even though we can't assign responsibility for optimization to any part of the system. Similarly, evolution can be viewed as an optimization process, even though there is (arguably) no distinguishable part of the biosphere or evolutionary environment doing the optimization.[2]

Nevertheless, often we do want to separate W into the "part doing the optimizing" (agent/optimizer/policy) and the "part being optimized" (environment). (Alex Flint calls this property of separability of an optimizing system into the optimizer a...
Accidental Electronic Instrument
by jefftk — 2024-05-06T02:10:02.297Z

I've been working on a

project with the
goal of adding virtual harp strings to my electric mandolin. As I've
worked on it, though, I've ended up building something pretty
different:

It's not what I was going for! Instead of a small bisonoric
monophonic picked instrument attached to the mandolin, it's a large
unisonoric polyphonic finger-plucked tabletop instrument. But I like
it!

While it's great to have goals, when I'm making things I also like to
follow the gradients in possibility space, and in this case that's the
direction they flowed.

I'm not great at playing it yet, since it's only existed in playable
form for a few days, but it's an instrument it will be
possible for someone to play precisely and rapidly with practice:

This does mean I need a new name for it: why would you call it a "harp
mandolin" when it has nothing to do with a mandolin and little to do
with a harp?

While you play it totally differently, the instrument it feels most
similar to me is a hammered
dulcimer. You can play quite quickly, and the only information
you're producing is the note selection, the timing, and the initial
velocity. It also makes me wonder whether this would pair well with a
damper pedal, reasonably common on hammered dulcimers, to allow you to
mark the ends of notes? My feet are already
allocated, but perhaps a strip running along the left hand side,
for your thumb, would do this well?

On the construction side, some things I've learned:

Having the 'teeth' directly on the circuit board simplifies
things a lot, and avoids having to mess with wires and worry about how
stressed the connections are.

But there's also a right order to assemble. Initially I tried
to glue up the sensors, then attach them to the board, then connect
them electrically. And then I broke a bunch and couldn't remove them.
Instead, what worked well the second time was soldering on just the
piezos first, then putting on the rubber and metal once I'd tested
that the whole board was working.

Acrylic is a nice surface for the top because it's thin and you
can see how it works inside, but there's not much reason to use it for
the back and it's a finicky material.

The box is essentially a picture frame, so following advice for
picture frame making works well.

If you make the box exactly the size of the PCB then it's
almost certainly going to be

...
Explaining a Math Magic Trick
by Robert_AIZI — 2024-05-05T19:41:52.048Z

Introduction

A recent popular tweet did a "math magic trick", and I want to explain why it works and use that as an excuse to talk about cool math (functional analysis). The tweet in question:

This is a cute magic trick, and like any good trick they nonchalantly gloss over the most important step. Did you spot it? Did you notice your confusion?

Here's the key question: Why did they switch from a differential equation to an integral equation? If you can use (1−x)−1=1+x+x2+... when x=∫, why not use it when x=d/dx?

Well, lets try it, writing D for the derivative:

f′=f(1−D)f=0f=(1+D+D2+...)0f=0+0+0+...f=0

So now you may be disappointed, but relieved: yes, this version fails, but at least it fails-safe, giving you the trivial solution, right?

But no, actually (1−D)−1=1+D+D2+... can fail catastrophically, which we can see if we try a nonhomogeneous equation like f′=f+ex (which you may recall has solution xex):

f′=f+ex(1−D)f=exf=(1+D+D2+...)exf=ex+ex+ex+...f=∞?

However, the integral version still works. To formalize the original approach: we define the function I (for integral) to take in a function f(x) and produce the function If defined by If(x)=∫x0f(t)dt. This rigorizes the original trick, elegantly incorporates the initial conditions of the differential equation, and fully generalizes to solving nonhomogeneous versions like f′=f+ex (left as an exercise to the reader, of course).

So why does (1−D)−1=1+D+D2+... fail, but (1−I)−1=1+I+I2+... works robustly? The answer is functional analysis!

Functional Analysis

Savvy readers may already be screaming that the trick (1−x)−1=1+x+x2+... for numbers only holds true for |x|<1, and this is indeed the key to explaining what happens with D and I! But how can we define the "absolute value" of "the derivative function" or "the integral function"?

What we're looking for is a norm, a function that generalizes absolute values. A norm is a function x↦||x|| satisfying these properties:

||x||≥0 for all x (positivity), and ||x||=0 if and only if x=0 (positive-definite)
||x+y||≤||x||+||y|| for all x and y (triangle inequality)
||cx||=|c|⋅||x|| for all x and real numbers c, where |c| denotes the

...
Does reducing the amount of RL for a given capability level make AI safer?
by Chris_Leong — 2024-05-05T17:04:01.799Z

Some people have suggested that a lot of the danger of training a powerful AI comes from reinforcement learning. Given an objective, RL will reinforce any method of achieving the objective that the model a) tries and b) finds to be successful. It doesn't matter whether it includes things like deceiving us or increasing its power.

If this were the case, then when we are building a model with capability level X, it might make sense to try to train that model either without RL or with as little RL as possible.

For example, we might attempt to achieve the objective using imitation learning instead. Some people would push back against this and argue that this is still a black-box that uses gradient descent with no way of knowing that the internals are safe.

Is this likely to produce safer models or is the risk mostly independent of RL?

Further thoughts: Does this actually matter?

Even if there are techniques involving avoiding or reducing RL that would make AI's safer, this could backfire if it results in capability externalities. Attempting to reproduce a certain level of capabilities without RL would be highly likely to lead to higher capabilities once RL was added on top and there are always actors who would wish to do this.

I don't think this invalidates this question though. Some of the various governance interventions may end up bearing fruit and so we may end up with the option of accepting less powerful systems.
Haymarket at Closing Time
by jefftk — 2024-05-05T02:40:15.212Z

Historically produce shopping was mostly in open-air markets, but in
the US produce is now typically sold in buildings. Most open-air
produce sales are probably at farmers markets, but these focus on the
high end. I like that

Boston's
Haymarket more similar to the historical model: competing vendors
selling conventional produce relatively cheaply.

It closes for the weekend at 7pm on Saturdays, and since food they
don't sell by the end of the market is mostly going to waste they
start discounting a lot. You can get very good deals, though you need
to be cautious: what's left at the end is often past the end of it's
human-edible life.

Today Lily was off at a scouting trip, and I asked Anna what she
wanted to do. She remembered that a previous time Lily was off we had
gone to Haymarket and bought three containers of raspberries (18oz,
half a kilo) for $1. Many of them were moldy, which was clear before
before buying them, and we'd had a fun time sitting in the park
hunting for the ones that were good eating.

We decided to go back, which is easier now that the Green Line runs
from Ball Sq. We bought a few different things and tried them out:

We were pleasantly surprised that the raspberries were tasty and
non-moldy! We decided to buy more, but it was still a few hours from
closing time and we weren't ready to go home yet, so we visited Paul Revere's
house. I was especially interested to see that it had a folding
bed, which was apparently a common way to maximize usable space in
cities during that period:

We also visited City
Hall Playground, with several slides:

You can see the infamous cop slide in the
background. Anna was initially afraid of it, but ended up going ten
times.

I wasn't planning to wait quite this long, but Anna had such a good
time at the playground it was hard to get her away. Around 6:30 we
walked back over to Haymarket, and found one vendor with raspberries
who was willing to do $15 for two boxes (12 6oz cartons, 2 kilos, per
box). I got four boxes (18lb, or 8 kilos). Anna helped transport:

The kids were very excited to be able to eat all the raspberries they
wanted! I emphasized that this was 'special', which is a term I use a
bit as jargon with the kids to designate something as "not how the
world usually works". We ate about three cartons fresh, shared a
bunch with housemates, kept...
introduction to cancer vaccines
by bhauth — 2024-05-05T01:06:16.972Z
Link: https://www.bhauth.com/blog/biology/cancer%20vaccines.html

cancer neoantigens

For cells to become cancerous, they must have mutations that cause uncontrolled replication and mutations that prevent that uncontrolled replication from causing apoptosis. Because cancer requires several mutations, it often begins with damage to mutation-preventing mechanisms. As such, cancers often have many mutations not required for their growth, which often cause changes to structure of some surface proteins.

The modified surface proteins of cancer cells are called "neoantigens". An approach to cancer treatment that's currently being researched is to identify some specific neoantigens of a patient's cancer, and create a personalized vaccine to cause their immune system to recognize them. Such vaccines would use either mRNA or synthetic long peptides. The steps required are as follows:

The cancer must develop neoantigens that are sufficiently distinct from human surface proteins and consistent across the cancer.

Cancer cells must be isolated and have their surface proteins characterized.

A surface protein must be found that the immune system can recognize well without (much) cross-reactivity to normal human proteins.

A vaccine that contains that neoantigen or its RNA sequence must be produced.

Most drugs are mass-produced, but with cancer vaccines that target neoantigens, all those steps must be done for every patient, which is expensive.

protein characterization

The current methods for (2) are DNA sequencing and mass spectrometry.

sequencing

DNA sequencing is now good enough to sequence the full genome of cancer cells. That sequence can be compared to the DNA of normal cells, and some algorithms can be used to find differences that correspond to mutant proteins. However, guessing how DNA will be transcribed, how proteins will be modified, and which proteins will be displayed on the surface is difficult.

Practical nanopore sequencing has been a long time coming, but it's recently become a good option for sequencing cancer cell DNA.

MHC mass spec

Proteins are often bound to a MHC for presentation on the surface, and those complexes can be isolated by mass spectrometry. You then know that the attached proteins can be on the cell surface. However...

It's currently hard to guess which of those MHC-bound proteins could have a good immune response.

This requires more cells than sequencing.

This doesn't find all the mutant surface proteins.

...
Some Experiments I'd Like Someone To Try With An Amnestic
by johnswentworth — 2024-05-04T22:04:19.692Z

A couple years ago, I had a great conversation at a research retreat about the cool things we could do if only we had safe, reliable amnestic drugs - i.e. drugs which would allow us to act more-or-less normally for some time, but not remember it at all later on. And then nothing came of that conversation, because as far as any of us knew such drugs were science fiction.

… so yesterday when I read Eric Neyman’s fun post

My hour of memoryless lucidity

, I was pretty surprised to learn that what sounded like a pretty ideal amnestic drug was used in routine surgery. A little googling suggested that the drug was probably a

benzodiazepine

(think valium). Which means it’s not only a great amnestic, it’s also apparently one of the most heavily prescribed drug classes historically, and used recreationally - which puts very strong lower bounds on the drug’s safety in practice, and means it’s probably readily available.

With that in mind, here are some experiments I’d love for someone to try (and report back on) using benzodiazepines.

Tests

IIUC, benzodiazepines (at the right doses) specifically block long-term memory formation: someone on the drug can keep things in working memory just fine, and can recall everything they already knew just fine, but basically won’t remember new information past a few minutes.

One very broad class of tests which such drugs open up is: put someone in a situation, see what they do for a minute or two, wait 5 minutes for them to forget, then repeat. Assuming their behavior is highly reproducible, that gives an ideal platform for testing interventions.

I’m particularly interested in seeing this approach applied to IQ tests.

The individual items on a typical IQ test fit comfortably in the few-minutes-long window allowed by the amnestic. So, basic test: give a few questions from a standard IQ test, repeat the questions five minutes later, and hopefully the person’s responses are highly reproducible. Ideally, this would eliminate essentially all the usual test-retest variance seen on IQ tests, as well as the “learning the test” issues.

Assuming that baseline works (i.e. results are very highly reproducible with little variance), the effects of interventions should be much easier to measure than they typically are in psych studies. Start with the basics: track room temperature and lighting, blood glucose and oxygenation, ventilation, bac...
Introducing AI-Powered Audiobooks of Rational Fiction Classics
by Askwho — 2024-05-04T17:32:49.719Z

(ElevenLabs reading of this post:)

Your browser does not support the audio element.

I'm excited to share a project I've been working on that I think many in the Lesswrong community will appreciate - converting some rational fiction into high-quality audiobooks using cutting-edge AI voice technology from ElevenLabs, under the name "Askwho Casts AI".

The keystone of this project is an audiobook version of Planecrash (AKA Project Lawful), the epic glowfic authored by Eliezer Yudkowsky and Lintamande. Given the scope and scale of this work, with its large cast of characters, I'm using ElevenLabs to give each character their own distinct voice. It's a labor of love to convert this audiobook version of this story, and I hope if anyone has bounced off it before, this might be a more accessible version.

Alongside Planecrash, I'm also working on audiobook versions of two other rational fiction favorites:

Luminosity by Alicorn (to be followed by its sequel Radiance)

Animorphs: The Reckoning by Duncan Sabien

I'm also putting out a feed where I convert any articles I find interesting, a lot of which are in the Rat Sphere.

My goal with this project is to make some of my personal favorite rational stories more accessible by allowing people to enjoy them in audiobook format. I know how powerful these stories can be, and I want to help bring them to a wider audience and to make them easier for existing fans to re-experience.

I wanted to share this here on Lesswrong to connect with others who might find value in these audiobooks. If you're a fan of any of these stories, I'd love to get your thoughts and feedback! And if you know other aspiring rationalists who might enjoy them, please help spread the word.

What other classic works of rational fiction would you love to see converted into AI audiobooks?
S-Risks: Fates Worse Than Extinction 
by aggliu — 2024-05-04T15:30:36.666Z
Link: https://youtu.be/fqnJcZiDMDo

Cross-posted to the EA forum

In this Rational Animations video, we discuss s-risks (risks from astronomical suffering), which involve an astronomical number of beings suffering terribly. Researchers on this topic argue that s-risks have a significant chance of occurring and that there are ways to lower that chance.

The script for this video was a winning submission to the Rational Animations Script Writing contest (https://www.lesswrong.com/posts/RH8nGG5vnuXc4eKu5/rational-animations-script-writing-contest). The first author of this post, Allen Liu, was the primary script writer with the second author (Writer) and other members of the Rational Animations writing team giving significant feedback. Outside reviewers, including authors of several of the cited sources, provided input as well. Production credits are at the end of the video. You can find the script of the video below.

Is there anything worse than humanity being driven extinct? When considering the long term future, we often come across the concept of "existential risks" or "x-risks": dangers that could effectively end humanity's future with all its potential. But these are not the worst possible dangers that we could face. Risks of astronomical suffering, or "s-risks", hold even worse outcomes than extinction, such as the creation of an incredibly large number of beings suffering terribly. Some researchers argue that taking action today to avoid these most extreme dangers may turn out to be crucial for the future of the universe.

Before we dive into s-risks, let's make sure we understand risks in general. As Swedish philosopher Nick Bostrom explains in his 2013 paper "Existential Risk Prevention as Global Priority",[1] one way of categorizing risks is to classify them according to their "scope" and their "severity". A risk's "scope" refers to how large a population the risk affects, while its "severity" refers to how much that population is affected. To use Bostrom's examples, a car crash may be fatal to the victim themselves and devastating to their friends and family, but not even noticed by most of the world. So the scope of the car crash is small, though its severity is high for those few people. Conversely, some tragedies could have a wide scope but be comparatively less severe. If a famous painting were destroyed in a fire, it could negatively affect millions or billions of peopl...