trentmkelly/LessWrong-43k huggingface dataset:
[Crosspost] Anthropic Shadow Geopolitics
by akarlin — 2025-05-21T04:50:36.550Z

[Crossposted from Karlin's Blog].

One of the most interesting concepts in rationalist thought relates to “Anthropic Shadow.” This is the idea that existential risks are implicitly conditioned on the existence of human observers, and are therefore biased to the downside, since any event severe enough to kill all present observers and prevent the emergence of future ones will leave no observable traces of its past existence.

If we extend these anthropic considerations into the near future, we may reach some disquieting conclusions. Worlds that “make it” remember and/or simulate their history, whereas those that die, do not. This loads on what is known as the Strong Self-Sampling Assumption (SSSA), which relates to the idea that our current observer-moment - e.g., you reading this now - was plucked out of all possible observer-moments (past, present, and future) in their reference class. The validity of the SSSA would be optimistic for some conscious observers surviving past AGI. But there is a catch. If you also believe that AGI is inherently dangerous; that AI timelines are short; and that alignment doesn’t happen by default and is inherently hard… then you must seriously entertain the possibility of a catastrophic but non-extinction level event between now and the early 2030s that rolls back AGI timelines by years or decades.

This “Rollback” seems likeliest to take one of two forms. One would be a major bioterrorism event that is linked to AI, and results in a global taboo against machine intelligence greater than the logic of capitalism and interstate geopolitical competition that drive its acceleration. The other would be a large-scale war between the US and China, likely over Taiwan, which collapses globalization and the fragile global supply chains that underpin the semiconductor industry from which the sand gods will emerge.

There are philosophical debates over the legitimacy of extrapolating the Anthropic Shadow into the future and considering what it would recursively imply for the present1. For the sake of this hypothesis, I am going to treat it as if it is valid. This essay is premised on the idea that observer-rich futures outweigh dead-end timelines in the distribution of observer-moments across all of history, and that this retroactively raises the probability that we live not in a doomed world, but in a world that makes it - because we ourselves exi...
The Need for Political Advertising
(Post 2 of 7 on AI Governance)
by Mass_Driver — 2025-05-21T00:44:06.560Z

In my previous post in this sequence, I made a fundraising pitch for my organization, the Center for AI Policy (CAIP), asking readers to donate to us so we can continue lobbying Congress to pass strong AI safety legislation.

Here, in the second post, I want to zoom out from CAIP’s particular concerns and argue that the AI safety movement as a whole should be investing far more resources into “political advertising.” We need to seek out individual politicians and personally let them know about the risk from misaligned superintelligence and show them what they can do to mitigate that risk.

The core of my argument for this second post is that AI governance ideas are not self-enacting: even the best policy ideas need political champions, or they will not become law. Therefore, to win at AI governance, we have to actively promote and advertise our ideas to political decisionmakers, not just to academic experts. The expected value of the movement’s AI governance efforts is something like Q × P, where Q is the quality of our ideas, and P is the power of our advocacy. If P is zero or close to zero, then it doesn’t matter how good our ideas are, because our ideas will not steer the future.

If we fail to adequately invest in advertising our ideas, the most likely result is that we’ll wind up with a glut of ‘orphaned’ AI governance ideas that have been thoroughly researched and vetted but that don’t impact the real world because they have few or no champions in the political arena. AI governance researchers will discover great strategies for saving the world, but nobody will put them into practice, and misaligned AI will destroy civilization.

Good Technical Alignment Ideas Might Spread Automatically

“AI safety work” can be split into two very broad categories: (1) technical alignment research and (2) AI governance.

By “technical alignment,” I mean anything that’s directly aimed at making AI systems safer, better understood, more reliable, more compatible with human values, or more controllable.

Technical alignment is not my field, but from what I can see from the outside, AI scientists already want to design maximally safe AI. Once AI developers find out how to make AI safe and compatible with human values at an affordable price, they’ll automatically want to do that, because the vast majority of computer scientists prefer to have an AI that will be more reliable ...
Notes from Dopamine Detoxing
by Alice Blair — 2025-05-20T23:43:31.097Z

[Epistemic status: I'm not a neuroscientist nor any kind of medical nor biological professional. I have an amateur understanding of neurochemistry and cognitive science. This post is primarily based on what has worked for me and my intuitions around why. I err on the side of epistemic transparency and caution where possible.]

Prerequisite Concept: Hedonic Set Point

Disclaimers/Warnings

I think this can be a genuinely very useful technique for people. However,

This can Probably be Dangerous

This is a post about doing some light amount of hacking your brain's chemicals, specifically in a way that involves temporarily taking away enjoyable things. Do not do this if you think that would be long-term bad for your mental health. This is not a technique for dealing with e.g. severe depression (nor probably most smaller-magnitude depressions). This is not a technique for fixing complex mental health issues, it is a technique for dealing with a very specific thing in some subset of people whose brains are somewhat like mine along some relevant axes.

I think there are some mental states and neurotypes where this is an actively harmful technique if you try to push through it for >0.5 days when that isn't the right move. I suspect it's relatively safe if you have the mindfulness to realize if things are making you long-term worse off and immediately Nope Out of the detox (described near the end). This is worth being cautious about.

And I'm Noticeably Biased

Also, this post is about a mental technique that

made me feel good after doing it
recently temporarily fixed what happened to be the most [big × tractable] bottleneck to my productivity.

For these reasons, the picture I paint of the benefits may be rosy and portray an effect size that does not generalize reliably to other people. Given those forces acting upon me, I try to paint a true picture and convey a general technique to whatever extent I reasonably can.

Introduction

(You can probably skip this section if you already know what a dopamine detox is and why you might want to do one.)

A "dopamine detox" is, for the purposes of this post, a period (usually a day) in which you try really hard to get rid of superstimuli in your environment. I've written in some probably-helpful additional detail about how superstimuli can hack your cognition, but this post does most of the necessary conceptual explaining.

I'd apprecia...
Revisiting the ideas for non-neuralese architectures
by StanislavKrym — 2025-05-20T23:35:51.824Z

Neuralese recurrence, allowing a model to learn thousands of tokens of information from a single pass, is thought to be the key to building high-capability models. Unfortunately, this key lacks interpretability, ensuring that we cannot check the alignment of the model by looking at its thoughts.

Back in April 3 a user lacking formal computer science education proposed an idea of a non-neuralese architecture something along the line "maybe to solve this information bottleneck problem, we should solve the generate-many-tokens-in-one-pass problem." Unfortunately, he didn't propose any technical solutions. With this post I aim to propose the solution with which I came up.

There is a technique of word embedding which represents words as vectors and even allows one to draw sense-faithful conclusions like "queen ≈ king-man+woman". The most primitive such representations have dimension 100-300, which allows one to define a word by using only 300 neurons. Meanwhile, a layer consists of tens of thousands of neurons. Consider a setup where the first 300 neurons of each layer generate a word to be put into the memo which will have length equal to the number of layers, while the model is to train to produce the memo that will resemble a text on a subject similar to the problem. Then the memo might be interpreted, paraphrased[1] to avoid euphemisms and attempts to communicate in an alien language[2] and loaded into the model along with the CoT for the next step.

Alternatively, one could use the approach where the words are generated by many groups of neurons and placed into quasi-CoTs; then the model should be given access to all the quasi-CoTs, trained to make them coherent and have the option to purge them when the time comes.

^

GPT-4o estimates the amount of compute necessary to paraphrase a 100-token memo by GPT-2 as ~5 billion floating point operations. The neural network with many parameters needs far more work than the paraphraser.

^

Although the model may develop a foreign language, it will be immune to the paraphraser and easier to understand given access to the memos and their context.
Gemini Diffusion: watch this space
by Yair Halberstadt — 2025-05-20T19:29:07.973Z
Link: https://deepmind.google/models/gemini-diffusion/

Google Deepmind has announced Gemini Diffusion. Though buried under a host of other IO announcements it's possible that this is actually the most important one!

This is significant because diffusion models are entirely different to LLMs. Instead of predicting the next token, they iteratively denoise all the output tokens until it produces a coherent result. This is similar to how image diffusion models work.

I've tried they results and they are surprisingly good! It's incredibly fast, averaging nearly 1000 tokens a second. And it one shotted my Google interview question, giving a perfect response in 2 seconds (though it struggled a bit on the followups).

It's nowhere near as good as Gemini 2.5 pro, but it knocks ChatGPT 3 out the water. If we'd seen this 3 years ago we'd have been mind blown.

Now this is wild for two reasons:

We now have a third species of intelligence, after humans and LLMs. That's pretty significant in and of itself.
This is the worst it'll ever be. This is a demo, presumably from a relatively cheap training run and way less optimisation than has gone into LLMs. Diffusion models have a different set of trade offs to LLMs, and once benchmark performance is competitive it's entirely possible we'll choose to focus on them instead.

For an example of the kind of capabilities diffusion models offer that LLMs don't, you don't need to just predict tokens after a piece of text: you can natively edit somewhere in the middle. Also since the entire block is produced at once, you don't get that weird behaviour where an LLM says one thing then immediately contradicts itself.

So this isn't something you'd use just yet (probably), but watch this space!
A Sketch of Belocracy: a new system of governance
by belos — 2025-05-20T18:30:54.620Z
Link: https://bestofagreatlot.substack.com/p/a-sketch-of-belocracy

I have been working on designing a new system of governance - an alternative to liberal democracy - for the past half decade, and hope that sharing it will inspire a good conversation. So, epistemic status: I really hope I have it figured out, but only time will tell with something like this. My writing is all posted at my substack, including a Table of Contents to keep the ideas organized.

There is something diseased in our democratic system. At least, this is the belief that has been strongly in the zeitgeist for the last few decades. Trust in government

hasn’t really been above 60% since the 60s

, which is a weird time to think of trust in government as “high”.

The exact diagnosis varies widely. Money in politics is a commonly claimed enemy, others might be the Supreme Court, the Imperial Presidency, the “deep state”, aka bureaucracy, etc. To some, the two parties are flip sides of the same coin, indistinguishable save for a different icon stamped onto the same base metal underneath. For others, it’s politicians: they’re just snakes; maybe everyone involved is just bad at their jobs; or the trust prior the 60s was just ignorance and clearly unjustifiable. The rich seem to be doing fine, maybe the whole system is just rigged.

I want to live in a society that is well run. One that isn't

civilizationally inadequate

, that I could even call

effective

. One that takes care of its people and fairly and equitably makes life better for all of us, while allowing us the freedom to try, fail, grow and otherwise be self-directed. Socialists dream of a fair society that treats us all well, and I share their dream. Libertarians and anarchists answer them with a call to be cautious when handing power over to any other humans lest we lose freedoms that we all deserve. Meanwhile, we argue over capitalism as if it were a goal, rather than a tool we use to pursue our goals. I don’t trust our current system to be able to build a fair society without destroying freedoms, and I don’t want it to maintain freedoms for the privileged at the expense of those who aren’t lucky enough to be on top. If we cannot do better, we risk dictatorship, as Paul Valery, writing presciently before WWII, warned:

As soon as the mind no longer recognizes itself—or no longer recognizes its essential traits, its mode of reasoned activity, its horror of chaos and the waste of its energy—in the fluctuat

...
The Codex of Ultimate Vibing
by Zvi — 2025-05-20T18:30:07.532Z

While we wait for wisdom, OpenAI releases a research preview of a new software engineering agent called Codex, because they previously released a lightweight open-source coding agent in terminal called Codex CLI and if OpenAI uses non-confusing product names it violates the nonprofit charter. The promise, also reflected in a number of rival coding agents, is to graduate from vibe coding. Why not let the AI do all the work on its own, typically for 1-30 minutes?

The answer is that it’s still early days, but already many report this is highly useful.

Introducing Codex

Sam Altman: today we are introducing codex.

it is a software engineering agent that runs in the cloud and does tasks for you, like writing a new feature of fixing a bug.

you can run many tasks in parallel.

it is amazing and exciting how much software one person is going to be able to create with tools like this. “you can just do things” is one of my favorite memes;

i didn’t think it would apply to AI itself, and its users, in such an important way so soon.

OpenAI: Today we’re launching a research preview of Codex: a cloud-based software engineering agent that can work on many tasks in parallel. Codex can perform tasks for you such as writing features, answering questions about your codebase, fixing bugs, and proposing pull requests for review; each task runs in its own cloud sandbox environment, preloaded with your repository.

Codex is powered by codex-1, a version of OpenAI o3 optimized for software engineering. It was trained using reinforcement learning on real-world coding tasks in a variety of environments to generate code that closely mirrors human style and PR preferences, adheres precisely to instructions, and can iteratively run tests until it receives a passing result.

…

Once Codex completes a task, it commits its changes in its environment. Codex provides verifiable evidence of its actions through citations of terminal logs and test outputs, allowing you to trace each step taken during task completion. You can then review the results, request further revisions, open a GitHub pull request, or directly integrate the changes into your local environment. In the product, you can configure the Codex environment to match your real development environment as closely as possible.

Codex can be guided by AGENTS.md files placed within your repository. These are text files, akin to README.md, whe

...
Outcomes of the Geopolitical Singularity
by Nikola Jurkovic — 2025-05-20T18:09:43.494Z

We will soon enter an unstable state where the balance of military and political power will shift significantly because of advanced AI.

As different nations gain access to new advanced technologies, nations that have a relative lead will amass huge power over those that are left behind. The difference in technological capabilities between nations might end up becoming so large that one nation will possess the capability to disable the militaries (including nuclear arsenals) of other countries with minimal retaliation. While this all is happening, AIs will possibly be accruing power of their own.

The unstable state where the relative power of nations is shifting dramatically cannot be sustained for long. In particular, it’s implausible that multiple groups will possess a DSA at separate moments in time without using their DSA to entrench their influence over the future.

There are three main types of stable outcomes for this transition period: unipolar, multipolar[1], and existential catastrophe.

Here are some (non-comprehensive) illustrative stories for each outcome:

Unipolar: One nation comes into possession of a DSA and uses it to control the course of the future.

Story 1A - Secret ASI project leading to hegemony: A nuclear weapons state secretly[2] achieves superintelligence first and has a major advantage over other countries. It uses this advantage to enter a position where they can make arbitrary demands of other countries[3]. The rest of history is shaped by the leadership of the ASI project.

The story in Situational Awareness

most closely matches this story.

Story 1B - Non-secret ASI project leading to hegemony: A nuclear weapons state creates AGI and gets close to secretly achieving a DSA but gets found out. It offers token concessions to the other nuclear weapon states to increase the upside of not attacking. Other countries are pacified[4] by the concessions and unwilling to commit major acts of sabotage because they underestimate the danger of facing an adversarial ASI. Using the additional months of time, the leading ASI project secretly continues improving its capabilities and achieves a DSA. The rest of history is shaped by the leadership of the ASI project.

The slowdown ending of AI 2027

most closely matches this story.

Multipolar: No nation has a DSA, and this is sustained until at least the end of the 21st century.

Story 2A - Internation

...
AISN #55: Trump Administration Rescinds AI Diffusion Rule, Allows Chip Sales to Gulf States
by Corin Katzke — 2025-05-20T16:21:58.235Z
Link: https://forum.effectivealtruism.org/posts/Yqna5Jpgm3JzdJ7BA/aisn-55-trump-administration-rescinds-ai-diffusion-rule?utm_campaign=post_share&utm_source=link

Welcome to the AI Safety Newsletter by the

Center for AI Safety

. We discuss developments in AI and AI safety. No technical background required.

In this edition: The Trump Administration rescinds the Biden-era AI diffusion rule and sells AI chips to the UAE and Saudi Arabia; Federal lawmakers propose legislation on AI whistleblowers, location verification for AI chips, and prohibiting states from regulating AI.

Listen to the AI Safety Newsletter for free on

Spotify

or

Apple Podcasts

.

Subscribe to receive future versions.

The Center for AI Safety is also excited to announce the Summer session of our AI Safety, Ethics, and Society course, running from June 23 to September 14. The course, based on our recently published

textbook

, is open to participants from all disciplines and countries, and is designed to accommodate full-time work or study.

Applications for the Summer 2025 course are now open. The final application deadline is May 30th. Visit the

course website

to learn more and apply.

Trump Administration Rescinds AI Diffusion Rule, Allows Chip Sales to Gulf States

On May 12th, the Department of Commerce

announced

that it had rescinded the

Framework for Artificial Intelligence Diffusion

, which was set to take effect May 15th. The rule would have regulated the export of AI chips and models across three tiers of countries, each with its own set of restrictions. (Other AI chip export controls, including those prohibiting sales to China, remain on the books.)

The announcement states that the Bureau of Industry and Security (BIS) will issue a replacement rule in the future. In the meantime, the BIS will focus on working to prevent US chips from being used in Chinese AI development. Bloomberg

reports

that new restrictions will focus on countries that have diverted US chips to China, including Thailand and Malaysia.

The Trump Administration wants to capture the global AI chip market. Though China has yet to export its own AI chips, the BIS will also issue guidance that states using Huawei Ascend chips violates US export controls. This preemptive restriction supports the Trump Administration’s intent for the US to dominate the global AI chip market.

UAE and Saudi Arabia are set to receive hundreds of thousands of AI chips. Last week, Trump announced trade deals with the

UAE

and

Saudi Arabia

, respectively.

The UAE is set to receive up to

500,000 of Nvidia's most advanced chips

...
Time isn't real man
by Jodie themathgenius — 2025-05-20T15:48:43.405Z
Link: https://jodie.website/#time

"There is nothing more deceptive than an obvious fact.''
― Arthur Conan Doyle

Anyone who' s taken LSD, pulled an all-nighter, been bored, or fell into a YouTube rabbit knows that time is subjective. Brains aren't clocks. They cheat constantly. Compressing
sequences of events, deleting traumatic evens, buffering sensory delays, stretching and compressing depending on how arousing whatever you're experiencing is.

Code Monks

Programmers enter multi-hour flow states without fatigue or hunger. Usually helped by autism headphones and chemical stimulants.

Bullet Time

When put in life-or-death situations, many report time slowing down. It's like the brain temporarily overclocks and hyper-saturates itself with sensory perception.

Teleportation

Notice how commuting to your job feels like a time skip? It's like mundane regular tasks simply don't exist to us.

Time Travel

When you travel, a few weeks turn into a whole-ass chapter of your life. A giant exotic memory palace. At the end you often feel like you're leaving an entire life behind.

Time Merchants

Video platforms optimize for smoothness and hyperstimulus. They make you forget where videos start and end.

Chronohacking

Time perception seems to me like a fertile ground for exploits. It's not just fun, it feels necessary in a world where attention is farmed and sold. There's only so much we can do to have a longer life, so why not try to make our life denser.