trentmkelly/LessWrong-43k huggingface dataset:
AI safety tax dynamics
by owencb — 2024-10-23T12:18:32.243Z
Link: https://strangecities.substack.com/p/ai-safety-tax-dynamics

Two important themes in many discussions of the future of AI are:

AI will automate research, and thus accelerate technological progress
There are serious risks from misaligned AI systems (that justify serious investments in safety)

How do these two themes interact? Especially: how should we expect the safety tax requirements to play out as progress accelerates and we see an intelligence explosion?

In this post I’ll give my core views on this:

Automation of research by AI could affect the landscape into which yet-more-powerful systems are emerging

Therefore, differential boosting beneficial applications may be a high-leverage strategy for improving safety

The most dangerous period probably occurs when AI is mildly- to moderately-superintelligent

I developed these ideas in tandem with my exploration of the concepts of safety tax landscapes, that I wrote about in a recent post. However, for people who are just interested in the implications for AI, I think that this post will largely stand alone.

How AI differs from other dangerous technologies

In the post on safety tax functions, my analysis was about a potentially-dangerous technology in the abstract (nothing specific about AI). We saw that:

The underlying technological landscape determines the safety tax contours

i.e. answering “how much do people need to invest in safety to stay safe?”, and “how does this vary with how powerful the tech is and how confident you want to be in safety?” basically depends on the shape of discoveries that people might make to advance the technology

Outcomes depend on:

Our ability to make investments in capabilities

“Investments” includes R&D, but also any other work necessary to build capable systems

Our ability to make investments in safety
Our ability to coordinate between actors to constrain the ratios of these investments

For most technologies, these abilities — the ability to invest in different aspects of the tech, and the ability to coordinate — are relatively independent of the technology; better solar power doesn’t do much to help us do more research, or sign better treaties. Not so for AI! To a striking degree, AI safety is a dynamic problem — earlier capabilities might change the basic nature of the problem we are later facing.

In particular:

At some point, most AI capabilities work and most AI safety work will probably be automated by AI itself

The amount of effective capabilit

...
What is malevolence? On the nature, measurement, and distribution of dark traits
by David Althaus — 2024-10-23T08:41:33.197Z

Summary

In this post, we explore different ways of understanding and measuring malevolence and explain why individuals with concerning levels of malevolence are common enough, and likely enough to become and remain powerful, that we expect them to influence the trajectory of the long-term future, including by increasing both x-risks and s-risks. For the purposes of this piece, we define malevolence as a tendency to disvalue (or to fail to value) others’ well-being (more). Such a tendency is concerning, especially when exhibited by powerful actors, because of its correlation with malevolent behaviors (i.e., behaviors that harm or fail to protect others’ well-being). But reducing the long-term societal risks posed by individuals with high levels of malevolence is not straightforward.

Individuals with high levels of malevolent traits can be difficult to recognize. Some people do not take into account the fact that malevolence exists on a continuum, or do not realize that dark traits are compatible with moral convictions (more). Moral judgments and stigma can also make it difficult to think objectively about these topics, and can make it hard to acknowledge these traits when they are present.

Malevolence is often studied in the context of the so-called dark tetrad traits—sadism, psychopathy, Machiavellianism, and narcissism. Other dark traits relevant to the long-term future include vengefulness (related to retributivism) and spitefulness. Most dark traits positively correlate with each other, which may suggest the existence of a general factor of human malevolence (‘D’) (more).

Like all personality traits, dark traits occur on a continuum; categories such as “psychopath” and “narcissist” rely on relatively arbitrary cut-off points. It is also difficult to reliably measure someone’s levels of dark traits. Having said this, based on the information we have, individuals with concerning levels of dark traits could be common enough to influence the trajectory of the long-term future.

Available data suggest that the diagnostic categories of psychopathy (more) and Narcissistic Personality Disorder (more) have prevalence rates of at least ~2% in the general population. And among those who have taken surveys at the DarkFactor.org, about 3% of them have given sets of answers which we would consider concerni...
Join a LessWrong Team for the Unaging System Challenge
by Crissman — 2024-10-23T06:01:08.018Z

Hey LessWrong,

After sharing early results at Less Online this summer (thanks for the feedback during the strength training demos), I'm recruiting for a LessWrong team in the 2025 Unaging System Challenge. What started as helping one friend who got winded climbing stairs has grown into a complete system that could add two decades to our lives. The research shows we can reduce premature death risk by up to 90% through specific physical activities – and it only requires 90 minutes of structured exercise weekly.

Evidence-based health information should be freely available to everyone. This is my contribution to increasing human health and longevity—think of it as applied EA for physical optimization.

The 2025 Program (Starting January 6):

Four 12-week challenges throughout the year
Each challenge builds one evidence-backed longevity habit
Starting with HIIT for cardiorespiratory fitness
Progressing to strength, mobility, and advanced protocols
Just 90 minutes of structured exercise weekly
Teams grouped by community for accountability and discussion

Our first cohort included rationalists from the Japan ACX meetup and a delightful range of fitness levels – from "I get winded opening my laptop" to "I do ultramarathons between alignment debates." One participant combined shadow boxing and the Floss dance for their HIIT intervals, proving rationalists can optimize anything.

What You Need:

Heart rate tracking (Apple Watch, Oura, Whoop, Fitbit kinda thing)
Three 30-minute exercise sessions weekly during building phases
Two sessions weekly for maintenance
Willingness to track metrics (love data)

Register here to join the LessWrong team: https://www.unaging.com/unaging-system/

Looking forward to helping more of us live long enough to solve the control problem – or at least climb stairs without questioning our life choices.

For rationality and longevity, Crissman

P.S. For those who asked at Less Online: No, we still don't need Bryan Johnson's supplement stack.
Word Spaghetti
by Gordon Seidoh Worley — 2024-10-23T05:39:20.105Z

I've written a lot of words—hundreds of blog posts, thousands of comments, tens of thousands of emails, and hundreds of thousands of short messages. I'm even written most of a book! In total, between personal and professional writing, I estimate to have put north of 3 million words into text. So it may come as a surprise that, despite all this experience, I still struggle to write well.

Sure, I've got the basics down, and when I put my mind to it I can achieve some measure of poetic prose. But to explain my ideas clearly, I still have to work quite hard for my writing to be understood.

Not all writers struggle with clarity. Some seem to be able to write exactly what they mean on the first pass and have their audience understand them. Others, like me, have to revise a lot, iterating over the same words dozens of times before they'll hang together. For example, it takes me about an hour to produce 100 words of finished text when I write about complex subjects, and I only speed up to 250 words/hour when the subject is simpler.

But what would happen if I didn't spend all that time revising? Would my writing really suffer that much?

It would.

Without extensive editing, my writing would be a jumbled mass of tangents, fragmented ideas, and confusing statements. You'd read my posts and probably think I said something I wouldn't endorse, or, worse yet, you'd give up reading me all together because it would take too much work. Thus I devote the time I must to editing for the simple reason that I want to convey ideas accurately and have people actually read them.

But why does my writing start out a mess? And why can't I do something, other than spend hours editing, to fix it? Because when my thoughts show up, they are like a scoop of spaghetti freshly dropped on a plate. The thought is definitely there, ready to be consumed, but the ideas are all tangled up, they don't combine to form a clear shape, and loose ends are left dangling over the side.

To be fair, my raw writing is not literally unintelligible, and sometimes people even like it because they like the ideas it contains. Still, it's not very good as writing. For example, here are some recent comments I made on Less Wrong that were written mostly stream-of-consciousness: 1, 2, 3, 4, 5. The last one is particularly egregious and in desperate need of editing.

Words come out of me like they do because thoughts come i...
Monosemanticity & Quantization
by Rahul Chand — 2024-10-22T22:57:53.529Z

In this post, I will cover Anthropic's work on monosemanticity[1]. Starting with a brief introduction to the motivation and methodology. Then move on to my ablation experiments where I train a sparse autoencoder on "gelu-2l"[2] and its quantized versions to see what insights I can gain.

Introduction

The holy grail of mech interpretability is to figure out what feature each single neuron in the network corresponds to and how changing these features changes the final output. Turns out this is really tough because of polysemanticity, that is neurons end up learning/activating for a mixture of features, which makes it tough to find out what exactly does a particular neuron do. One reason this happens is because of "superposition" where the features in a training data are sparse and much more than the number of neurons, therefore, to effectively learn each neuron ends up learning a linear combination of features.

The first set of nodes are neuron, and the second set are "features". Each neuron is a linear combination of a small set of features. We have more features than neurons. We want the feature layer to be sparsely activated for any specific neuron to ensure this is interpretable is interpretable we want the feature layer to be sparsely activated for any specific neuron. The (2x4) W we see here is the encoder

Polysemanticity means it difficult to make statements about what a neuron does. Therefore, we train a sparse autoencoder on top of a already trained neural network which offers a more monsemantic analysis. By that what we mean is, we first hypothesize that each neuron is learning some linear combination of features, these features are more basic and finer grained. Next, if we can figure out what these features are and what feature each neuron activates then we can comment about what task or set of tasks each neuron performs/captures.

Formulation

Formally, we have a set of basic features di (which is larger than the number of neurons) and we attempt to decompose our activations into these directions/features/basis vectors. That is,

xi=b+∑fi(xj)di

where xi is the output of neuron i, fi is ...
What is the alpha in one bit of evidence?
by J Bostock — 2024-10-22T21:57:09.056Z

Recently the whole "if your p(doom) is high, you should short the market" thing has been going around. Let's say the market prices in a 0.1% chance of extinction per century, and you think there's a 50% chance we're dead in the next 25 years. This is about 12 bits of evidence.

How much profit is it even reasonable to expect someone to make from 1 bit of evidence?

Is this a constant function of the bits which is calculable like e.g. a Kelly bet? Or does it depend on the market?

If the latter, does it ever make sense to short things at all based on p(doom)? I assume not, since shorting is generally stupid high-risk for an uninitiated trader (as I understand it).

Consider: everyone else thinks a company share has a 0.1% chance of being worth $10 tomorrow and a 99.9% chance of being worth $0. I think the chance is 50:50. Therefore the stock price is $0.01 today. If I have $100 in the bank, I maximize expected log(money) by spending fully half of my money on it. By my reckoning the maximum log-expected money is $1581, which corresponds to about a 16-fold increase over $100. Which is not 10 bits of alpha!

PS not looking for investment advice here, just looking for maths.
Catastrophic sabotage as a major threat model for human-level AI systems
by evhub — 2024-10-22T20:57:11.395Z

Thanks to Holden Karnofsky, David Duvenaud, and Kate Woolverton for useful discussions and feedback.

Following up on our recent “

Sabotage Evaluations for Frontier Models

” paper, I wanted to share more of my personal thoughts on why I think catastrophic sabotage is important and why I care about it as a threat model. Note that this isn’t in any way intended to be a reflection of Anthropic’s views or for that matter anyone’s views but my own—it’s just a collection of some of my personal thoughts.

First, some high-level thoughts on what I want to talk about here:

I want to focus on a level of future capabilities substantially beyond current models, but below superintelligence: specifically something approximately human-level and substantially transformative, but not yet superintelligent.

While I don’t think that most of the proximate cause of AI existential risk comes from such models—I think most of the direct takeover risk comes from more intelligent models—I do think that much of the root cause of risk might come from such models. What I mean by that is approximately: in a takeover scenario, even if it’s a superintelligent model that ends up taking over the world, in retrospect the point at which things went wrong probably happened earlier. And in fact I think the critical juncture is likely to be around this level of capabilities, e.g. because this is around the time when I expect most alignment research to be shifting over from being done by humans to being done by AIs.

In particular, I’m focusing on a level of capabilities that is substantially beyond anything that we found in “

Sabotage Evaluations for Frontier Models

.” Even perfect performance on all the evals in that paper would still probably be below the level of capabilities I am intending to talk about here (though it’s a bit ambiguous since many of those evals have an attacker and defender and it’s unclear what would happen if you substantially increased both the attacker and defender capabilities).
This post is an amalgamation of a bunch of stuff that I originally wrote as individual, separate documents on a shared theme. As a result, while I’ve tried to edit it to be a little more coherent overall, don’t expect it to e.g. all have really nice internal references.
Finally, these are all very rough initial thoughts—I expect my thinking on this to continue to change a lot and I wouldn’t be s

...
Why I quit effective altruism, and why Timothy Telleen-Lawton is staying (for now)
by Elizabeth — 2024-10-22T18:20:01.194Z

~5 months I formally quit EA (formally here means “I made an announcement on Facebook”). My friend Timothy was very curious as to why; I felt my reasons applied to him as well. This disagreement eventually led to a podcast episode, where he and I try convince each other to change sides on Effective Altruism- he tries to convince me to rejoin, and I try to convince him to quit.

Audio recording

Transcript

Some highlights:

My story of falling in love, trying to change, and then falling out of love with Effective Altruism. That middle part draws heavily on past posts of mine, including EA Vegan Advocacy is not truthseeking, and it’s everyone’s problem and Truthseeking is the ground in which other principles grow

Why Timothy still believes in EA

Spoilers: Timothy agrees leaving EA was right for me, but he wants to invest more in fixing it.

Thanks to my Patreon patrons for supporting my part of this work.
Decision-Making Under Uncertainty: Lessons From AI
by Jonasb — 2024-10-22T17:54:47.530Z
Link: https://www.denominations.io/living-with-uncertainty-a-practical-guide/

Are you sure you know what tomorrow looks like? You probably have a general sense of the day ahead, but can you predict exactly where you will be at 09:10 AM tomorrow morning? What about your feelings at said point in time? Or how you'll sleep overnight?

Even though there are a lot of things in life we can be certain about – the sun will rise tomorrow morning, and our planet will still be hurtling through space at breakneck speeds – our personal prediction window is actually quite narrow.

For a lot of things we just assume they will stay the same. This is something Kahnemann and Tversky captured in

System I vs System II thinking

. "System I thinking" is our default mode, where we assume everything remains the same, and System II is where we think about everything else–about long-term changes.

This is a useful formalisation, but as K&T have demonstrated in many of their experimental studies knowing

our cognitive biases

does not necessarily help us make better decisions. Neither does

Nassim Taleb's key insight that our environment is uncertain

.

So what gives? In what follows I want to discuss a couple of strategies for dealing with uncertainty that are applied when building AI systems. AI systems typically deal with automated decision-making at scales of tens to millions of decisions per day, and need robust checks in order to continue to function as intended.

Is there anything we can learn from AI that we can apply to our lives?

Let's find out.

A changing world

The key macrotrends driving business transformation identified by business leaders surveyed in 2023 (source: World Economic Forum Future of Jobs Survey 2023)

Just to state the obvious, we're entering uncharted territory when it comes to climate, society, technology and the economy.

Besides our rapid self-propelled (pun intended) exit from Holocene, humanity is facing rising inequality, a complete overhaul of the labor force and a nexus of technological advances is driving cost of production closer and closer to zero. All of these changes are ushering in an experience economy that is unlike anything the world has ever seen, rendering our existing economic and business models pretty much useless.*

That's a lot of factors to take into account when you're shopping for groceries.

So with all the uncertainty in the world around us, what can we do to make the best possible decisions not just for right now, but for our...
Testing Genetic Engineering Detection with Spike-Ins
by jefftk — 2024-10-22T17:20:54.947Z
Link: https://naobservatory.org/blog/testing-with-spike-ins

Thanks to our collaborators Marc Johnson and Clayton Rushford at the University of Missouri for their critical work on the laboratory experiments and sample preparation for this test, and to Evan Fields, Mike McLaren, and Simon Grimm for their feedback on drafts.

Summary

The NAO has been developing an early warning system for pandemics, focusing specifically on stealth pathogens that could spread widely before demonstrating identifiable symptoms. In June we described progress with our chimera detection approach, and demonstrated how it flagged engineered pathogens in simulated data and engineered viral vectors in real data. Here we describe a series of real-world tests, "spiking" samples of municipal wastewater with three different engineered HIV-derived viral particles. In each case our chimera detection system correctly identified sequencing reads containing the junctions between the original and modified sections of the genomes, with performance mildly exceeding our simulations.

Experiment

Our partners Marc Johnson and Clayton Rushford at the University of Missouri (MU) have been sequencing municipal wastewater influent on a regular basis. At the end of September they took one of these samples, split it into four replicates, and spiked HIV-derived engineered viral particles into three of them. The genomes of the viral particles looked something like the following:

The three particles were developed at MU, and each derives from the HIV-1 lab strain pNL4-3, with different inserted regions:

v530: DsRed, a red fluorescent protein
v549: Green Fluorescent Protein (GFP)
Puro: a CMV promoter and a codon-optimized puromycin resistance gene

Our MU collaborators spiked 10uL into each of these samples. In the case of Puro the final concentration was 7.1k copies per mL. With v530 and v549, however, they haven't measured the concentration and so our assessment of those samples will be only in relative terms.

In addition to the three spiked samples they also prepared a control, a portion of the same original wastewater sample without the addition of any viral particles.

Our collaborators extracted the nucleic acids from these four samples using their standard protocol, which they describe in Defining biological and biophysical properties of SARS-CoV-2 genetic material in wastewater. They sent the extracts to the MU Genomics Technology Core for RNA sequencing on a NovaSeq X with th...
Predictions as Public Works Project — What Metaculus Is Building Next
by ChristianWilliams — 2024-10-22T16:35:13.999Z
Link: https://www.metaculus.com/notebooks/29025/what-metaculus-is-building-next/

This post is by Metaculus's Sylvain Chevalier, with input provided by Molly Hickman, and Deger Turan.

Forecasting should lead to smarter decisions on consequential issues. This is challenging, and no one knows the best way to do it, but we’re committed to making it happen at Metaculus.

Below are preconditions we’ve identified:

Forecasts should be trustworthy

Our forecasts should be calibrated. They should be robust to manipulation. They should also be accurate—but this prompts the question: “accurate compared to what other forecast?” We’ve recently published a methodology on how to rigorously compare two forecasters on a set of questions.

Forecasts should be interpretable

The barrier to entry for using forecasts is very high. Newcomers do not know what to do with a 75% forecast. This is partly a user interface problem (Metaculus is too nerdy for many people), and partly a fundamental problem of interpretability, explainability, and transparency. One needs to understand where a forecast comes from to make good use of it.

Questions should be relevant

None of the above matters if we’re not asking the right questions. Is the question confounded in a way that makes the forecast useless? Are we using a good proxy? Would a definite answer actually change anyone’s mind? To solve for these we need to work closely with the people seeking the answers.

How will we achieve all this? When the best approaches are uncertain, it's time to experiment. In the coming months, we will run quick experiments, build prototypes, and make bold changes—both on Metaculus.com and off-platform. To succeed, the most promising ideas will need you: the forecasters, the readers, the commenters, and the decision-makers. We hope you'll help us identify what works, iterate on the best ideas, and bring them to life.

To enable this fast iteration, we’ve fully rewritten our site. Metaculus launched in 2016, and our codebase was old. To build the sense-making tools we want, and to go open source, we knew a fresh start was needed. The rewritten site is nearly feature-complete, with some less-used features cut or simplified, and with some new ideas introduced.

So, what experiments, prototypes, and big changes are we planning? There are many, but we especially want to hear from you on these:

Writing Questions Together

In recent years, the forecasting ecosystem has harnessed the collective abilities of skilled...