trentmkelly/LessWrong-43k huggingface dataset:
Introducing Collective Action for Existential Safety: 80+ actions individuals, organizations, and nations can take to improve our existential safety
by jamesnorris — 2025-02-05T16:02:04.456Z

Collective Action for Existential Safety is an initiative of the Center for Existential Safety, a new non-profit organization being formed in the United States.

Our central aim is to catalyze collective action to ensure humanity survives this decade. If we can achieve that, then we are likely to create an unimaginably good future for all.

We serve all Existential Safety Advocates globally. We believe artificial intelligence, nuclear weapons, engineered pandemics, and soon-to-be invented technologies threaten our civilization. The more of us working together to mitigate the risks, the better.

Our offer: we created what we believe is the most comprehensive and practical list of actions individuals, organizations, and nations can take to improve our existential safety. It answers the question, “What can we do to help ensure our existential safety?” We also offer occasional

all hands calls

for the existential safety community, monthly strategy

coordination calls

for existential safety organization leaders, and collaborative safety calls for frontier AI developers.
Our ask: we're fundraising and recruiting for the team. Happy to take on aligned co-founders and c-suite folks, as well as part-time volunteers. Please

email

or

schedule a call

if you’re interested in learning more.

Feedback is very much welcome, but ideally it would be offered in a kind, collaborative, and well-informed manner. We believe unkind and uninformed criticism in this community has very likely reduced its positive impact on the world. My favorite guidelines for this are

here

. Please directly suggest changes to our actions list

here

.

On a personal note, I’ve been preparing for this for 26+ years now. I have short timelines and a high

p(doom)

. But I believe we can still collectively raise our

p(eutopia)

if we act now as if our lives depended on it. They do. This could also be humanity’s finest hour.

Join us at

existentialsafety.org

. Please share the site widely and engage with us on social media. Everyone’s future is at stake and there are useful actions just about everyone can do to help.
Language Models Use Trigonometry to Do Addition
by Subhash Kantamneni — 2025-02-05T13:50:08.243Z

I (Subhash) am a Masters student in the Tegmark AI Safety Lab at MIT. I am interested in recruiting for full time roles this Spring - please reach out if you're interested in working together!

TLDR

This blog post accompanies the paper "Language Models Use Trigonometry to Do Addition." Key findings:

We show that LLMs represent numbers on a helix

This helix has circular features (sines and cosines) with periods T=[2,5,10,100] and a linear component

To solve a+b, we claim that LLMs manipulate the helix for a and b to create the helix for a+b using a form of the Clock algorithm introduced by Nanda, et al.

This is conceptually akin to going from cos(a),cos(b) to cos(a+b)
Intuitively, it's like adding angles on a Clock face.

Motivation and Problem Setting

Mathematical reasoning is an increasingly prominent measure of LLM capabilities, yet we lack understanding of how LLMs process even simple mathematical tasks. In this spirit, we aim to reverse engineer how three mid-sized LLMs compute addition.

In this blog post, we focus on GPT-J, a 6B parameter LLM, but we show additional results for Pythia-6.9B and Llama3.1-8B in the main paper. We construct problems of the form a+b for a,b∈[0,99]. GPT-J is competent on this task, successfully completing 80% of prompts.

For conciseness, we share a subset of our results here. Please see the full paper for additional information.

LLMs Represent Numbers on a Helix

Investigating the Structure of Numbers

To generate a ground up understanding of how LLMs compute a+b, we first aim to understand how they represent some integer a. In order to identify trends in their representation, we run GPT-J on all single-token integers a∈[0,360]. We conduct analysis on the residual stream following layer 0.

Periodicity

We first Fourier decompose the residual stream after layer 0, and find that the Fourier domain is sparse, with significant high frequency components at T=[2,5,10] and many low frequency components, as shown above. The structure in the Fourier domain implies that numbers are represented with periodic features.

Prior work has also identified these Fourier features, and although initially surprising, are sensible. The units digit of numbers is periodic in base ten (T=10), and it is reasonable that qualities like evenness (T=2) ar...
The Domain of Orthogonality
by mgfcatherall — 2025-02-05T08:14:32.793Z

TL;DR

I think that a large and significant chunk of the goal-intelligence plane would be ruled out if moral truths are self-motivating, contrary to what Bostrom claims in his presentation of the orthogonality thesis.

Intro

In the seminal paper The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents, Nick Bostrom introduces his Orthogonality Thesis, proposing the independence of goal-content and intelligence level:

The Orthogonality Thesis

Intelligence and final goals are orthogonal axes along which possible agents can freely vary. In other words, more or less any level of intelligence could in principle be combined with more or less any final goal.

Bostrom then goes on to address various objections that might be raised and provides counter arguments to them, essentially pre-empting criticism of the orthogonality thesis and explained why he doesn't think each criticism undermines the orthogonality thesis. He discusses the distinction between intelligence and rationality, saying that:

By “intelligence” here we mean something like instrumental rationality—skill at prediction, planning, and means-ends reasoning in general.

Then he goes on to address an argument from moral internalism:

In a similar vein, even if there are objective moral facts that any fully rational agent would comprehend, and even if these moral facts are somehow intrinsically motivating (such that anybody who fully comprehends them is necessarily motivated to act in accordance with them) this need not undermine the orthogonality thesis. The thesis could still be true if an agent could have impeccable instrumental rationality even whilst lacking some other faculty constitutive of rationality proper, or some faculty required for the full comprehension of the objective moral facts. (An agent could also be extremely intelligent, even superintelligent, without having full instrumental rationality in every domain.)

Let's unpack that for ease of reference;

First he has a hypothetical about moral internalism being true, "even if there are objective moral facts that any fully rational agent would comprehend, and even if these moral facts are somehow intrinsically motivating (such that anybody who fully comprehends them is necessarily motivated to act in accordance with them)...". Let's call this statement 'A' for brevity.
Then he says A doesn't undermine his orthogona

...
Reviewing LessWrong: Screwtape's Basic Answer
by Screwtape — 2025-02-05T04:30:34.347Z

Yeah I put this off until the last day, and I'm not sure this is the format Raemon was actually looking for. Oh well.

Then, in proportion to how valuable they seem, spend at least some time this month reflecting...

...on the big picture of what intellectual progress seems important to you. Do it whatever way is most valuable to you. But, do it publicly, this month, such that it helps encourage other people to do so as well. And ideally, do it with some degree of "looking back" – either of your own past work and how your views have changed, or how the overall intellectual landscape has changed.
...on how you wish incentives were different on LessWrong. Write up your thoughts on this post. (I suggest including both "what the impossible ideal" would be, as well as some practical ideas for how to improve them on current margins)
...on how the LessWrong and X-risk communities could make some group epistemic progress on the longstanding questions that have been most controversial. (We won't resolve the big questions firmly, and I don't want to just rehash old arguments. But, I believe we can make some chunks of incremental progress each year, and the Review is a good time to do so)

I. On the big picture of what intellectual progress seems important to you

I've said it elsewhere, and I'll say it again here: the thing that hooked me is The Martial Art Of Rationality. I'll do my proper rant about that elsewhere, but that's the direction I'm pointed in. I want progress in the form of clear examples of ways brains make obvious mistakes paired with drills on how to practice not making that mistake, and a way to check if the drill is working.

Anki and spaced repetition, to stop forgetting things.

Calibration training tools, like the introduction of Fatebook.

Even focusing on checking if things are working by focusing on the feedback loop.

I did a lot of meetup writeups for Meetup In A Box. And yet, this probably isn't much progress? It's hammering pitons into the mountainside to make it easier for more people to climb up, not braving a fresh climb somewhere nobody's been before.

Raemon's Feedback Loop Rationality is a central example of what I want more of.

II. On how I wish incentives could be different on LessWrong

I wish there was more reason for great writers to stick around.

Stop for a moment and think of the great rationalist writers. Grab a piece of paper or open a text d...
Why isn't AI containment the primary AI safety strategy?
by OKlogic — 2025-02-05T03:54:58.171Z

Introduction

When discussing AI safety, alignment—ensuring AI systems pursue human-approved goals—is often the primary focus. However, containment, which restricts AI’s ability to exert influence beyond controlled environments, is in my opinion a more intuitive and less complex approach. This post will outline common objections to AI containment, explain why they may be overstated, and invite counterarguments. You can tell me why I am wrong in the comments.

Objection 1: A Super intelligent Will Always Escape (and a sufficiently advanced AI might be able to as well)

For an AI to pose a risk, it must influence the external world. Strict air-gapping and controlled local communication channels (ie, insuring the AI has no access to the internet) can essentially eliminate the risk of AI leaking out directly. However, one of the strongest objections to containment is that no system can permanently constrain a super intelligent AI, as it will eventually exploit loopholes, manipulate humans, or find an escape route.

Counterarguments:

Mitigating Human Manipulation

AI manipulating humans into circumventing containment is a valid concern, but countermeasures can reduce the risk. For instance:

- Training personnel to recognize and resist AI persuasion tactics.

- Implementing shift-based oversight to prevent prolonged exposure to the AI.

- Restricting AI interactions to personnel without the capability to unbox it.

- Structuring AI outputs to be dry and unpersuasive.

- Limiting the AI from addressing certain sensitive topics.

- Screening out personnel with certain risk factors for manipulation.

Some of y'all might want to bring up the famous AI box experiment, but I really don't think this is that relevant. With serious preparation, I feel like the AI's task rapidly becomes unfeasible.

A Response to the Chess Analogy

Some might argue that attempting to contain a intelligent AI is like playing chess against a vastly superior opponent—defeat is inevitable. However, unlike chess, we control the rules. Following the analogy, its like if I start the game with a king and three rooks, and you start with nothing but a king. Proper containment checkmates the AI before the game begins.

Objection 2: Containment Reduces AI Usefulness

Another objection is that containment limits AI’s potential. If AI is confined, it cannot fully participate in scientific advancements or real-world decision-making. Thi...
Nick Land: Orthogonality
by lumpenspace — 2025-02-04T21:07:04.947Z

Editor's note

Due to the interest aroused by @jessicata's posts on the topic, Book review: Xenosystems and The Obliqueness Thesis, I thought I'd share a compendium of relevant Xenosystem posts I have put together.

If you, like me, have a vendetta against trees, a tastefully typeset LaTeχ version is available at this link. If your bloodlust extends even further, I strongly recommend the wonderfully edited and comprehensive collection recently published by Passage Press.
I have tried to bridge the aesthetic divide between the Deleuze-tinged prose of vintage Land and the drier, more direct expositions popular around these parts by selecting and arranging pieces so that no references are needed but those any LW-rationalist is expected to have committed to memory by the time of their first Lighthaven cuddle puddle (Orthogonality, Three Oracle designs), and I've purged the texts of the more obscure 2016 NRx and /acc inside baseball; test readers confirmed that the primer stands on its own two feet.
The first extract, Hell-Baked, is not strictly about orthogonality, but I have decided to include it as it presents a concise and straightforward introduction to the cosmic darwinism underpinning the main thesis.

Xenosystems: Orthogonality

IS

Hell-Baked

Neoreaction, through strategic indifference, steps over modes of condemnation designed to block certain paths of thought. Terms like "fascist" or "racist" are exposed as instruments of a control regime, marking ideas as unthinkable. These words invoke the sacred in its prohibitive sense.

Is the Dark Enlightenment actually fascist? Not at all. It's probably the least fascistic strain of political thought today, though this requires understanding what fascism really is, which the word itself now obscures. Is it racist? Perhaps. The term is so malleable that it's hard to say with clarity.

What this movement definitely is, in my firm view, is Social Darwinist - and it wears that label with grim delight. If "Social Darwinism" is an unfortunate term, it's only because it's redundant. It simply means Darwinian processes have no limits that matter to us. We're inside **Darwinism**. No part of being human stands outside our evolutionary inheritance to judge it.

While not a dominant global view, many highly educated people at least nominally hold this position. Yet it's scarcely bearable to truly think through.

The inescapable conclusio...
What working on AI safety taught me about B2B SaaS sales
by purple fire — 2025-02-04T20:50:19.990Z

Subtitle: you're too poor to use AGI.

WTF is WTP?

In Econ 101, you learn about something called willingness to pay (WTP). WTP is the highest price at which you're willing to buy some good or service. As long as the equilibrium price is less than your WTP, you'll get the good, otherwise you'll prefer to keep your money.

But there's a wrinkle: what if my demand isn't independent of yours? This happens all the time. My WTP for a social media platform will be higher if lots of other people are also using the platform. My WTP for a rare collectible will be higher if fewer other people have it.

That second situation creates an incentive for exclusive contracts. Let's say Ferrari makes a car for $50k. Suppose my individual WTP for a new car is $100k and your WTP is $70k. In theory, Ferrari should sell cars to both of us, since WTP > cost. But I want to feel special, so if I'm the only person with the sports car it's worth an extra $100k to me. Now Ferrari could sell to both of us and get $170k in revenue. But instead, I offer $190k to exclusively sell it to me. This leaves me better off, leaves Ferrari way better off, and leaves you in the dust.

Keep that dynamic in mind, we'll see it again later.

Software (companies) ate the world

Here's a quick stat: in 2024, the global SaaS market was around $3 trillion.[1] That's an economy the size of the UK built around subscription software. Why doesn't every company build their own software solutions, perfectly tailored to their needs?

To be clear, enterprises definitely want to do this. 3rd party solutions are a huge headache because they create reliance on another company that's totally out of your control and carry all the compliance baggage of the provider. If your software provider goes out of business, you're screwed. If they decide to double your costs, you're screwed. If they have a data leak, you're screwed. If a junior developer pushes a commit with breaking package dependencies, literally all of your computers will turn off and you will forgo billions of dollars in lost revenue. Oops.

Oh, and also most enterprise software is sold for 5-6x costs, so there are (checks notes) hundreds of billions of dollars of lost value flowing to these providers.[2]

So why does the software industry exist at all? They have monopolized three things:

Talent
Data access
Distribution and infrastructure

Distribution and infrastructure ...
Subjective Naturalism in Decision Theory: Savage vs. Jeffrey–Bolker
by Daniel Herrmann — 2025-02-04T20:34:22.625Z

Summary:

This post outlines how a view we call subjective naturalism[1] poses challenges to classical Savage-style decision theory. Subjective naturalism requires (i) richness (the ability to represent all propositions the agent can entertain, including self-referential ones) and (ii) austerity (excluding events the agent deems impossible). It is one way of making precise certain requirements of embedded agency. We then present the Jeffrey–Bolker (JB) framework, which better accommodates an agent’s self-model and avoids forcing her to consider things she takes to be impossible.[2]

1. Subjective Naturalism: Richness & Austerity

A naturalistic perspective treats an agent as part of the physical world—just another system subject to the same laws. Among other constraints, we think this means:

Richness: The model must include all the propositions the agent can meaningfully consider, including those about herself. If the agent can form a proposition “I will do X”, then that belongs in the space of propositions over which she has beliefs and (where appropriate) desirabilities.
Austerity: The model should only include events the agent thinks are genuinely possible. If she is certain something cannot happen, the theory shouldn’t force her to rank or measure preferences for that scenario. (Formally, we can regard zero-probability events as excluded from the relevant algebra.)

A decision-theoretic framework that meets both conditions is subjectively naturalist: it reflects the agent’s own worldview fully (richness) but doesn’t outstrip that worldview (austerity).

1.1. Framework vs. Action-Guiding Rule

In the literature, “decision theory” can refer to (at least) two different kinds of things:

A conceptual or mathematical framework for representing an agent’s beliefs, desires, and preferences. These frameworks show how one might encode uncertainty, evaluate outcomes, or measure utility—but they don’t necessarily dictate a unique rule for which choice to make in any given scenario.
A decision rule or algorithmic procedure for taking an action—e.g., “choose the act that maximizes expected utility according to these probabilities,” or “choose so as to maximize causal expected utility.”

When people say “vNM decision theory”, “Savage’s decision theory”, or “Jeffrey–Bolker,” they sometimes shift back and forth between framework-level discussion (how to model an agent’s preferenc...
Anti-Slop Interventions?
by abramdemski — 2025-02-04T19:50:29.127Z

In his recent post arguing against AI Control research, John Wentworth argues that the median doom path goes through AI slop, rather than scheming. I find this to be plausible. I believe this suggests a convergence of interests between AI capabilities research and AI alignment research.

Historically, there has been a lot of concern about differential progress amongst AI safety researchers (perhaps especially those I tend to talk to). Some research is labeled as "capabilities" while other is labeled as "safety" (or, more often, "alignment"[1]). Most research is dual-use in practice (IE, has both capability and safety implications) and therefore should be kept secret or disclosed carefully.

Recently, a colleague expressed concern that future AIs will read anything AI safety researchers publish now. Since the alignment of future AIs seems uncertain and even implausible, almost any information published now could be net harmful for the future.

I argued the contrary case, as follows: a weak form of recursive self-improvement has already started (in the sense that modern LLMs can usefully accelerate the next generation of LLMs in a variety of ways[2]). I assume that this trend will intensify as AI continues to get more useful. Humans will continue to keep themselves at least somewhat in the loop, but at some point, mistakes may be made (by either the AI or the humans) which push things drastically off-course. We want to avoid mistakes like that.

John spells it out more decisively:

The problem is that we mostly don’t die of catastrophic risk from early transformative AI at all. We die of catastrophic risk from stronger AI, e.g. superintelligence (in the oversimplified model). The main problem which needs to be solved for early transformative AI is to use it to actually solve the hard alignment problems of superintelligence.

The key question (on my model) is: does publishing a given piece of information reduce or increase the probability of things going off-course?

Think of it like this. We're currently navigating foreign terrain with a large group of people. We don't have the choice of splitting off from the group; we expect to more-or-less share the same fate, whatever happens. We might not agree with the decision-making process of the group. We might not think we're currently on-course for a good destination. Sharing some sorts of information with t...