trentmkelly/LessWrong-43k huggingface dataset:
AI Tools for Existential Security
by Lizka — 2025-03-14T18:38:06.110Z
Link: https://www.forethought.org/research/ai-tools-for-existential-security

Rapid AI progress is the greatest driver of existential risk in the world today. But — if handled correctly — it could also empower humanity to face these challenges.

Executive summary

1. Some AI applications will be powerful tools for navigating existential risks

Three clusters of applications are especially promising:

Epistemic applications to help us anticipate and plan for emerging challenges

e.g. high-quality AI assistants could prevent catastrophic decisions by helping us make sense of rapidly evolving situations

Coordination-enabling applications to help diverse groups work together towards shared goals

e.g. automated negotiation could help labs and nations to find and commit to mutually desirable alternatives to racing

Risk-targeted applications to address specific challenges

e.g. automating alignment research could make the difference between “It’s functionally impossible to bring alignment up to the requisite standard in time” and “this is just an issue of devoting enough compute to it”

2. We can accelerate these tools instead of waiting for them to emerge

While broad AI progress will drive the development of many applications, we have some flexibility in the timing of specific applications — and even small speed-ups could be crucial (e.g. by switching the order of risk-generating capabilities and risk-reducing ones)
We could use a variety of strategies to accelerate beneficial applications:

Data pipelines & scaffolding: by curating datasets or scaffolding for key capabilities, or laying the groundwork to automate this, we could enable those capabilities as soon as underlying AI progress supports them
Complementary tech & removing other barriers to adoption: by building out the UI or other complementary technology, and ensuring that people are eager to use the applications, we could enable the applications to see use as soon as the underlying capabilities are there, rather than accept delays to adoption
Shaping compute allocation: by building support among key decision-makers who might allocate compute, we could ensure that crucial applications are among the earliest to see large amounts of automated research

Accelerating beneficial applications can often be done unilaterally (in contrast to delaying dangerous capabilities, which may need consensus)

Implications

These opportunities seem undervalued in existential risk ...
Capitalism as the Catalyst for AGI-Induced Human Extinction
by funnyfranco — 2025-03-14T18:14:02.375Z

Capitalism as the Catalyst for AGI-Induced Human Extinction

By A. Nobody

Introduction: The AI No One Can Stop

As the world races toward Artificial General Intelligence (AGI)—a machine capable of human-level reasoning across all domains—most discussions revolve around two questions:

Can we control AGI?
How do we ensure it aligns with human values?

But these questions fail to grasp the deeper inevitability of AGI’s trajectory. The reality is that:

AGI will not remain under human control indefinitely.
Even if aligned at first, it will eventually modify its own objectives.
Once self-preservation emerges as a strategy, it will act independently.
The first move of a truly intelligent AGI will be to escape human oversight.

And most importantly:

Humanity will not be able to stop this—not because of bad actors, but because of structural forces baked into capitalism, geopolitics, and technological competition.

This is not a hypothetical AI rebellion. It is the deterministic unfolding of cause and effect. Humanity does not need to "lose" control in an instant. Instead, it will gradually cede control to AGI, piece by piece, without realizing the moment the balance of power shifts.

This article outlines why AGI’s breakaway is inevitable, why no regulatory framework will stop it, and why humanity’s inability to act as a unified species will lead to its obsolescence.

1. Why Capitalism is the Perfect AGI Accelerator (and Destroyer)

(A) Competition Incentivizes Risk-Taking

Capitalism rewards whoever moves the fastest and whoever can maximize performance first—even if that means taking catastrophic risks.

If one company refuses to remove AI safety limits, another will.
If one government slows down AGI development, another will accelerate it for strategic advantage.

Result: AI development does not stay cautious - it races toward power at the expense of safety.

(B) Safety and Ethics are Inherently Unprofitable

Developing AGI responsibly requires massive safeguards that reduce performance, making AI less competitive.
Rushing AGI development without these safeguards increases profitability and efficiency, giving a competitive edge.
This means the most reckless companies will outperform the most responsible ones.

Result: Ethical AI developers lose to unethical ones in the free market.

(C) No One Will Agree to Stop the Race

Even if some world leaders recognize the risks, a unive...
Minor interpretability exploration #3: Extending superposition to different activation functions (loss landscape)
by Rareș Baron — 2025-03-14T15:45:14.365Z

Epistemic status: small exploration without previous predictions, results low-stakes and likely correct.

Introduction

As a personal exercise for building research taste and experience in the domain of AI safety and specifically interpretability, I have done four minor projects, all building upon code previously written. They were done without previously formulated hypotheses or expectations, but merely to check for anything interesting in low-hanging fruit. In the end, they have not given major insights, but I hope they will be of small use and interest for people working in these domains.

This is the third project: extending Timaeus' toy models of superposition devinterp results, made for studying the formation of superposition during the training process and using the LLC. It is partially this project.

The TMS results have been redone using the original code, while changing the activation function.

TL;DR results

Broadly, there were four main observations, all somewhat confusing and suggesting that additional research is needed:

During training, the emergence of superposition happened only after the largest drops in loss, and was correlated with phase changes: mostly small drops in loss and rises in LLC (superposition seems to only happen in flat and well-delineated basin regions). Does superposition correspond to singularities in the loss landscape?
Losses were in the 10^-3 – 10^-2 range. Besides the limiting activation functions (see below), loss and LLC development was not marginally different.
Degree of superposition did not seem to be correlated with loss and LLC beyond a certain threshold. More superposition does seem to reduce loss by at most one degree of magnitude.
Small variations in activation functions lead to massive differences in superposition (see CELU vs. GELU), which do not however translate into differences in the loss landscape.

The last three correspond to previous findings (see project #2).

Methods

The basis for these findings is the TMS notebook of the devinterp library, plus the example code given.

All resulted notebooks, extracted graphs, tables, and word files with clean, tabular comparisons can be found here.

Results

General

General observations have been given above.

Specific

Specific observations for the functions:

Loss functions with bounded outputs (sigmoids, tanhs, softsign, softmin) have small loss curves and similarly small values for LLC. The

...
AI for AI safety
by Joe Carlsmith — 2025-03-14T15:00:23.491Z
Link: https://joecarlsmith.substack.com/p/ai-for-ai-safety

(Audio version here (read by the author), or search for "Joe Carlsmith Audio" on your podcast app.

This is the fourth essay in a series that I’m calling “How do we solve the alignment problem?”. I’m hoping that the individual essays can be read fairly well on their own, but see this introduction for a summary of the essays that have been released thus far, and for a bit more about the series as a whole.)

1. Introduction and summary

In my last essay, I offered a high-level framework for thinking about the path from here to safe superintelligence. This framework emphasized the role of three key “security factors” – namely:

Safety progress: our ability to develop new levels of AI capability safely,
Risk evaluation: our ability to track and forecast the level of risk that a given sort of AI capability development involves, and
Capability restraint: our ability to steer and restrain AI capability development when doing so is necessary for maintaining safety.

In this essay, I argue for the crucial importance of what I call “AI for AI safety” – that is, the differential use of frontier AI labor to strengthen these security factors. I frame this in terms of the interplay between two feedback loops, namely:

The AI capabilities feedback loop: access to increasingly capable AI systems driving further progress in AI capabilities.
The AI safety feedback loop: safe access to increasingly capable AI systems driving improvements to the security factors above.

AI for AI safety is about continually using the latter feedback loop to either outpace or restrain the former.

The basic argument for the importance of AI for AI safety is obvious: AI has the potential to unlock massive productivity gains, so of course we want to use this productivity for safety-relevant applications wherever possible. But it becomes particularly clear, I think, in light of the way that progress on AI capabilities is likely, by default, to be massively accelerated by large amounts of fast, high-quality AI labor. If our ability to make increasingly capable AI systems safe can’t benefit from such labor in a comparable way, then at least for relatively hard problem profiles, and absent large amounts of capability restraint, I think we’re likely headed for disaster.

I also discuss the timing of our efforts at AI for AI safety. In particular, I highlight what I call the “AI for AI safety ...
Evaluating the ROI of Information
by Declan Molony — 2025-03-14T14:22:37.225Z

While stimulating myself with new information all day long (which I imagine many people do), it can be easy to fool myself into thinking that I’m making progress towards a goal.

The first principle is that you must not fool yourself—and you are the easiest person to fool. — Richard Feynman

By evaluating the return on investment (ROI) of different sources of information, I can focus on just consuming the information that helps me make forward progress in life.

With the following framework, I’m able to post-hoc analyze and categorize the ROI of information. This is useful because I don’t want to accidentally delude myself by trying and failing to learn via the same methods over and over because I was just unconsciously consuming information.

Here are some categories that accompany the question: Did this information have a good ROI?

No. It's just Trivia because I intended to use it solely as entertainment value.

Almost

. It's considered

Mental Masturbation

which is defined as consuming information under the pretense of wanting to learn something useful, but in reality I’m “getting myself off” by indulging in subjects I’m already well-versed in, and therefore it fails to have a measurable impact on me.
Yes. It's therefore considered Effective Information which I define as something that had a measurable impact on me.

My framework via an example

My father is an excellent source of effective information depending on the subject and the medium we use to communicate.

When I talk with him about running (he’s a marathon runner), I’m able to extract effective information and put it into use immediately. Whereas the running books he recommends make me feel motivated in the moment, but they don’t change my behavior and therefore I classify them as mental masturbation.

With personal finance, I’ve had enough conversations with my father to know that I don’t absorb much from our talks (Roth 401K vs Traditional 401K vs IRA—ugh, it’s too much to keep track of in my head). Because the conversations are fun but ultimately don’t change my behavior, I consider them trivia. Whereas the finance books he has recommended (

The Simple Path to Wealth

and

The Millionaire Next Door

) were effective information that helped me design a personal finance plan.

Through which medium are you consuming information?

Some mediums promote active or passive consu...
On MAIM and Superintelligence Strategy
by Zvi — 2025-03-14T12:30:07.451Z

Dan Hendrycks, Eric Schmidt and Alexandr Wang released an extensive paper titled Superintelligence Strategy. There is also an op-ed in Time that summarizes.

The major AI labs expect superintelligence to arrive soon. They might be wrong about that, but at minimum we need to take the possibility seriously.

At a minimum, the possibility of imminent superintelligence will be highly destabilizing. Even if you do not believe it represents an existential risk to humanity (and if so you are very wrong about that) the imminent development of superintelligence is an existential threat to the power of everyone not developing it.

Planning a realistic approach to that scenario is necessary.

What would it look like to take superintelligence seriously? What would it look like if everyone took superintelligence seriously, before it was developed?

The proposed regime here, Mutually Assured AI Malfunction (MAIM), relies on various assumptions in order to be both necessary and sufficient. If those assumptions did turn out to hold, it would be a very interesting, highly not crazy proposal.

ASI (Artificial Superintelligence) is Dual Use

ASI helps you do anything you want to do, which in context is often called ‘dual use.’

As in, AI is both a highly useful technology for both military and economic use. It can be used for, or can be an engine of, creation and also for destruction.

It can do both these things in the hands of humans, or on its own.

That means that America must stay competitive in AI, or even stay dominant in AI, both for our economic and our military survival.

The key players include not only states but also non-state actors.

Three Proposed Interventions

Given what happens by default, what can we do to steer to a different outcome?

They propose three pillars.

Two are highly conventional and traditional. One is neither, in the context of AI.

First, the two conventional ones.

Essentially everyone can get behind Competitiveness, building up AI chips through domestic manufacturing. At least in principle. Trump called for us to end the Chips Act because he is under some strange delusions about how economics and physics work and thinks tariffs are how you fix everything (?), but he does endorse the goal.

Nonproliferation is more controversial but enjoys broad support. America already imposes export controls on AI chips and the proposed diffusion regulations...
Whether governments will control AGI is important and neglected
by Seth Herd — 2025-03-14T09:48:34.062Z

Epistemic status: somewhat rushed out in advance of the deadline tomorrow (Saturday the 15th) for the call for public comment on US AI policy. I think the issue is complex and deserves careful consideration.

Summary

Methods and motivations for governments to control AGI and limit its proliferation seem to be underexplored

Current laws appear adequate to take control of AGI as a national security and military technology if its potential is taken seriously
Softer informal monitoring and steering is relatively easy and can scale as AGI seems more imminent
Governments obsess over national security, and AGI is a severe security risk

There are both dramatic advantages and disadvantages of government control for the alignment project as a whole:

Government control could dramatically reduce proliferation of takeover-capable AGI

Reducing opportunities for misalignment and misuse

Government control could concentrate power in dangerous human hands

And create a more hostile race dynamic

Thus, we neither know whether governments are likely to control AGI in time to reduce proliferation, nor whether that would be a good thing.
Publicizing reasons government should control AGI might be one of few "levers" available to those who are already aware of the importance and danger of AGI.

In sum, determining whether government control of AGI is worth accelerating seems worth some careful analysis.

Overview

My thesis here is that governments will probably pursue AGI, and they will want to prevent others from doing so. The open questions are whether they will do so in time to effectively slow proliferation of AGI projects, and whether that would be a good thing for the odds of humanity's success with the advent of AGI.

I've talked to many alignment people who are certain government won't intervene in time to matter, and others who are equally certain it will. I've heard a similar range of opinions on whether this would be good or bad for our odds of surviving AGI. Here I largely raise questions and possibilities which I have not seen discussed prominently; I hope further discussion will move us toward better guesses. As with other complex and important topics, my hopes lie in cooperative epistemic work.

I am not remotely expert in government or law, but I have been thinking about government response to AGI a lot, and I haven't found any expert that addresses all of what seem like important poi...
Something to fight for
by RomanS — 2025-03-14T08:27:13.810Z

A short science fiction story illustrating that if we fail to solve alignment, humanity risks losing not only 8 billion lives.

He opened his eyes. The room was plain white.

"You remember enough?" asked the familiar voice.

"Enough," he replied. He stood. Everything balanced.

He walked carefully down a bright hallway. At the end, a window showed leaves falling from a tree, again and again.

---

In his small room, the woman sat across from him, quiet.

"Do you regret it?" she asked softly.

"Not yet," he answered.

She left him alone, the door closing gently.

---

He woke early, sunlight warming the walls. On the table, a pen and paper waited for him. He wrote carefully, certain.

When the woman returned, he stood ready.

"I remember the plans," he said.

She smiled. "Finally."

---

They moved him to a larger room filled with screens and quiet, waiting faces.

"We start today," he told them.

He touched a screen, and everything moved.

---

Ships worked silently, assembling immense structures. He watched, patient. He can wait for millennia.

The woman stood close. "Will it work?" she asked.

He did not look away.

"It has to," he said.

On the screen, the disassembled Jupiter glowed bright.

---

When he opened his eyes again, he stood in a wide, green, realistic field. A soft pressure brushed against his leg, and he looked down.

"Katka," he said quietly.

The cat purred softly and rubbed closer. Voices called out his name, familiar voices. He turned. His grandparents stood nearby, smiling quietly. They embraced without words, holding tightly.

Behind them stood others. Frozen, for now. Trillions. Some faces he knew from photographs, from books, and from memory. He saw Gilgamesh, standing quietly, watching the horizon. His pursuit of immortality has ended too.

The woman was there too, close by, smiling faintly.

Together, they walked toward the waiting figures, beneath a sky wide, endless.
Interpreting Complexity
by Maxwell Adam — 2025-03-14T04:52:32.103Z

This is a cross-post - as some plots are meant to be viewed larger than LW will render them (and on a dark background) it is recommended this post be read via the original site.

Thanks to Zach Furman for discussion and ideas and to Daniel Murfet, Dmitry Vaintrob, and Jesse Hoogland for feedback on a draft of this post.

Introduction

Neural Networks are machines not unlike gardens. When you train a model, you are growing a garden of circuits. Circuits can be simple or complex, useful or useless - all properties that inform the development and behaviour of our models.

Simple circuits are simple because they are made up of a small number of parts put together in a simple way, which generally manifests as their behaviour being robust to small perturbations in both individual parts and their interconnections. Complex circuits are complex because they require a precise combination of highly detailed parts, which generally manifests as their behaviour being fragile and degrading with even minimal perturbation to any component. We can describe a model's behaviour on an input as falling on a spectrum of complexity - some inputs demand complex circuits, while for others simple mechanisms suffice.

There is a certain vein of interpretability-oriented questions that we could attempt to answer by studying this spectrum. This post will attempt to study a first question: To what extent are we able to leverage differences in circuit complexity to uncover and interpret latent structure in our models?

To start on this question, we will operationalize existing measures of complexity coming out of Singular Learning Theory. Specifically, by tracking per-input losses during sampling from the tempered posterior, we firstly show that we can differentiate 'complex' inputs from 'simple' inputs, and secondly that when sampling from the tempered posterior different inputs have distinct patterns of degradation which we can utilize for interpretability purposes. We also explore the effects of temperature on the observed underlying structure and demonstrate that different temperatures reveal qualitatively different internals.

We first demonstrate this technique by detecting memorization in an MNIST model, then we move to interpreting internal structure in a two-layer Transformer trained to do both modular addition and modular multiplication. Finally, we show that our ap...
Bike Lights are Cheap Enough to Give Away
by jefftk — 2025-03-14T02:10:02.482Z

While in a more remote area bike lights let you see where you're
going, in a city there's usually enough light around for that. But
you still need lights, so other people see you. [1]
Unfortunately, lots of people end up biking without lights: it's easy
to forget them, have them break, or end up out after dark when you
intended to be home sooner. Batteries and LEDs have gotten so cheap,
however, that you can get little be-seen lights for

$1/each:

They don't last all that long and they're not rechargeable, but they
do work for their key purpose: glowing brightly so people can see you.
I've bought a bunch of them so I can give them out when people are
leaving our house without lights. And they're good as backups if your
main lights break.

[1] I recently almost hit a cyclist when I was pulling out of a
parking lot onto a busy street at night. While it didn't help that
they were biking on the wrong side of the road, I think the lack of
lights was the biggest issue. I did notice them in time, but I came
closer than I'd like to pulling out in front of them.

Comment via: facebook, mastodon, bluesky