trentmkelly/LessWrong-43k huggingface dataset:
AI Might Kill Everyone
by Bentham's Bulldog — 2025-06-05T15:37:59.830Z

(Crosspost from my blog).

(I’ll be at EAG London this weekend—come say hi. Also, this is my thousandth blogpost—cool milestone!)

Several people have wondered why I haven’t written much about AI. The main reason is that I don’t feel that I have anything very original to contribute. I try to only speak about things that I know something about. I don’t really know much about AI. While I use it regularly for editing, and have read a decent amount about it, it’s quite far outside of my area of expertise.

But I feel that I should say something about it because it’s important. It’s plausibly the most important thing. We may be approaching a second industrial revolution with consequences more dramatic than the first. There is a real possibility of everyone dying.

I’m on the more optimistic side. I think there’s only a few percent chance that AI kills everyone. Maybe 1 or 2%. In the EA circles in which I hang out, this often makes me outrageously optimistic. Lots of people, like Yudkowsky, think we’re nearly guaranteed to all die.

But whether one’s P(doom) is 1% or 60%, it’s abundantly clear that we should be doing a lot more than we are currently doing. AI alignment research—research that makes sure AI does what we want—should take up a sizeable portion of the federal budget. AI governance and international treaties are sorely needed. (

Here are some high impact careers

—many related to AI—and here are a bunch of high impact charities for safeguarding the longterm,

largely by aligning AI

). Your odds of dying from AI are a lot higher than your odds of dying in a car accident.

If AI goes well, it could usher in unprecedented prosperity. If it goes poorly, it could usher in an unimaginable catastrophe. In such a world, trying to steer AI so that it goes well should be a top global priority.

A lot of the arguments for AI risk have been written with a great deal of technical language, but I think the core argument for AI risk is pretty straightforward. We are building things that are much smarter than we are. AI can already do many human jobs better than most people.

A few years ago, AI couldn’t write competently. GPT2 was useless. GPT3 was revolutionary. GPT4 was better than humans at many tasks. What will GPT10 look like? Whatever AI is like in 30 years, it will be very impressive.

AI has already surpassed us in lots of tasks. The best human chess player can’t hold a candle...
AI #119: Goodbye AISI?
by Zvi — 2025-06-05T14:00:05.389Z

AISI is being rebranded highly non-confusingly as CAISI. Is it the end of AISI and a huge disaster, or a tactical renaming to calm certain people down? Hard to tell. It could go either way. Sometimes you need to target the people who call things ‘big beautiful bill,’ and hey, the bill in question is indeed big. It contains multitudes.

The AI world also contains multitudes. We got Cursor 1.0, time to get coding.

On a personal note, this was the week of LessOnline, which was predictably great. I am sad that I could not stay longer, but as we all know, duty calls. Back to the grind.

Table of Contents

Language Models Offer Mundane Utility. The white whale.

Language Models Don’t Offer Mundane Utility. You need a system prompt.

Language Models Could Offer More Mundane Utility. A good set of asks.

Huh, Upgrades. The highlight is Cursor 1.0, with memory and more.

Fun With Media Generation. Video is high bandwidth. But also low bandwidth.

Choose Your Fighter. Opinions differ, I continue to mostly be on Team Claude.

Deepfaketown and Botpocalypse Soon. Fake is not a natural category. Whoops.

Get My Agent On The Line. We all know they’re not secure, but how bad is this?

They Took Our Jobs. Economists respond to Dario’s warning.

The Art of the Jailbreak. Why not jailbreak AI overviews?

Unprompted Attention. More prompts to try out.

Get Involved. SFCompute, Speculative Technologies.

Introducing. Anthropic open sources interpretability tools, better AR glasses.

In Other AI News. FDA launches their AI tool called Elsa.

Show Me the Money. Delaware hires bank to value OpenAI’s nonprofit.

Quiet Speculations. People don’t get what is coming, but hey, could be worse.

Taking Off. AI beats humans in a test of predicting the results of ML experiments.

Goodbye AISI? They’re rebranding as CAISI. It’s unclear how much this matters.

The Quest for Sane Regulations. The bill is, at least, definitely big. Tl;dr.

Copyright Confrontation. OpenAI is being forced to retain all its chat logs.

Differential Access. The Good Guy needs a better AI than the Bad Guy.

The Week in Audio. Altman, Tegmark, Amodei, Barnes.

When David Sacks Says ‘Win the AI Race’ He Literally Means Market Share.

Rhetorical Innovation. Blog metagame continues to dominate.

Aligning a Smarter Than Human Intelligence is Difficult. Proceed accordingly.

Misaligned! About that safety plan, would it, you know, actually

...
Powerful Predictions
by Alvin Ånestrand — 2025-06-05T10:44:33.457Z
Link: https://forecastingaifutures.substack.com/p/powerful-predictions

A thoughtful post by Anton Leicht, “

Powerless Predictions

,” notes that forecasting may be underutilized by AI policy organizations. But how much is forecasting actually being used? Which organizations working on AI—particularly AI safety—already integrate forecasting into their decision-making, policy recommendations, and strategic planning? What forecasting work is being done, and how is it impacting the future of AI?

I set out to investigate this.

While I hope this post proves useful to others, I wrote it primarily for myself to gain an overview of forecasting work and its impacts. Feel free to skim—only read the details if you find them valuable.

Grantmaking

Open Philanthropy, which funds numerous AI safety initiatives, regularly makes predictions about grant outcomes. Evaluating these outcomes helps them improve their ability to predict the results of future grantmaking decisions. Javier Prieto at Open Philanthropy explains this process and evaluates prediction accuracy in

this post

.

They have also commissioned an long list of

forecasting questions on AI

from

Arb Research

. I couldn't find any publicly documented uses of this list, though Open Philanthropy may have used it for internal planning and strategic thinking.

AI development

Anthropic

extrapolates rare AI behaviors

to reduce concerns about concerning behaviors that may be missed during evaluations.

In Anthropic's

Responsible Scaling Policy

(RSP), they incorporate forecasting for comprehensive AI assessment, making informal predictions about improvements of elicitation techniques and enhanced model performance between testing rounds. They aim to "improve these forecasts over time so that they can be relied upon for risk judgments."

While you could argue that frontier AI development would be irresponsible without forecasting future capabilities and risks, it's nevertheless encouraging to see this explicitly incorporated into their practices. DeepMind's

Frontier Safety Framework

doesn't appear to explicitly include forecasting, and OpenAI's

Preparedness Framework

only mentions it in passing.

Affecting Policy Decisions

While policy recommendations implicitly depend on predictions about the future, explicit and well-researched forecasting doesn't appear to be the norm. However, much policy work may be informed by forecasts without being transparently based on forecast analyses—making it difficult to determine how...
Potentially Useful Projects in Wise AI
by Chris_Leong — 2025-06-05T08:13:42.839Z

This is a list of projects[1] to consider for folks who want to use Wise AI to steer the world towards positive outcomes.

Some of these projects are listed because they're impactful. Others are listed because I believe they would be good projects for someone to get started.

Please note that this post is titled "potentially useful projects" for a reason. Some of these projects are likely to have much higher impact than others. I expect some are net-negative. Don't ignore your own judgment just because a project is listed here!

I'm sure my views about what projects are valuable will change quite significantly as I dive deeper into the topic, but I still think it's worthwhile for me to put some kind of list out there.

But first, an announcement:

Applications for The Future of Life Foundation's Fellowship on AI for Human Reasoning are closing soon (June 9th!)
They've listed "Tools for wise decision making" as a possible area to work on.
Expand for more details.

From their website:

Apply by June 9th | $25k–$50k stipend | 12 weeks, from July 14 - October 3
Join us in working out how to build a future which robustly empowers humans and improves decision-making.
FLF’s incubator fellowship on AI for human reasoning will help talented researchers and builders start working on AI tools for coordination and epistemics. Participants will scope out and work on pilot projects in this area, with discussion and guidance from experts working in related fields. FLF will provide fellows with a $25k–$50k stipend, the opportunity to work in a shared office in the SF Bay Area or remotely, and other support.
In some cases we would be excited to provide support beyond the end of the fellowship period, or help you in launching a new organization.

Further Information ✦ Apply now!

I was originally going to delay publishing this until after making the case Wise AI Advisors being a priority (and, more generally, Wise AI as well), but I ended up deciding that it was important to publish this before the deadline for the Future of Life Fellowship in case it inspired more people to apply.

Field-building:

At this stage, I consider field-building work around Wise AI as especially high priority. I feel that there’s starting to be some real energy around this area, however, very little of this will aid us in avoiding catastrophic risks. Fields tend to be more flexible in their initi...
Building as gardening
by Itay Dreyfus — 2025-06-05T06:41:56.540Z
Link: https://productidentity.co/p/building-as-gardening

Since I’ve been interviewing other builders for the

niche design zine

, I've noticed a few recurring patterns. One is “building as gardening” which seems like a healthier way to build software in our age.

This is how Cab of

Are.na

described to me this approach:

You can be like “we’ve got to plant these here!” or “this is not working, let’s take it out!”.

But you can also choose the other way which is more like “oh wow, this thing is growing, that's really cool” or “let's see what else we can put here”.

It's a different kind of perspective.

And this, unrelatedly or not, how Herman of

Bear Blog

put it:

I’m trying to build a similar product with

Bear Blog

. Something niche but valuable. Something I can spend time on because I want to.

Being able to talk to, and interact with the people using my tools is fulfilling. Spending time meticulously improving subtle aspects of the product is enjoyable.

My product is my garden

I’m no poetic person, so I admit this metaphor was a bit hard to digest at first. However, after pondering on this perspective for a while, I suddenly could make sense out of it.

For the longest time, I’ve seen people building software like in a greenhouse mode: carefully controlling and forcing conditions to make something grow. That often means building more “necessary” features, paying people to participate in “research” surveys, committing early to investors, or rushing to catch the “right moment” in the market.

The common thread of this build-fast-break-things-pivot-repeat culture is the sense of urgency. There’s a collective pressure to grow something faster than it naturally would. Instead of letting things grow more organically, we’re forced to grow them more artificially. Then everything feels urgent.

But that's just one way to view this world.

Then there's the gardening approach, which is a different process; a different mindset. Walking this path isn’t about the classic solving problems mantra through countless plans, strategies, and checklists to mark. Nor is it about taking the hammer and knocking a few components and forcing them to fit together.

Gardening is rather about tending something slowly until it (or not) reaches any height, while giving it the necessary time to mature and evolve. It’s not about forcing a solution, but rather about allowing growth to happen more organically.

Building software is much the same. It’s not a science-ba...
Semiconductor Fabs I: The Equipment
by nomagicpill — 2025-06-04T22:09:32.678Z
Link: https://nomagicpill.github.io/research/fabs.html

An in-depth look at semiconductor processing equipment: considerations, selection, operation, and supporting info.

Preface

I tried to include as many links as possible to allow the reader to go down rabbit holes as they see fit.

I also tried to keep info basic enough for the skimmers/laypeople to enjoy while still adding more technical details located at the end of each section in the collapsible element. Pictures are sprinkled throughout to put an image with the words.

Interactive animations are denoted by the large "Click here to [X]!". While rudimentary, they get the job done. These were vibe coded using Claude 4 Opus with extended thinking and a few iterations.

This information is current as of May 2025.

Context

Before reading this, you should read Construction Physics' (Brian Potter) How to Build a $20 Billion Semiconductor Fab. It's an excellent introduction into both the semiconductor manufacturing process and the physical building that houses the manufacturing operations. However, Brian (understandably) chooses not to expand on a critical component: the equipment used to make the semiconductor chips.

As Brian writes:

Early integrated circuits could be made with just ... dozens of process steps, but a modern leading-edge microchip might require ... thousands of separate process steps.

These individual process steps are performed by semiconductor equipment, also called tools or machines. In the same way that a traditional machine shop has a mill, lathe, and bandsaw to perform a variety of operations on a single piece of metal to create a final product, a fab has deposition tools, etch tools, and lithography tools (among many, many others) to perform a variety of operations on a single silicon wafer to create the final chip.

There are 100s to 1000s of tools in a single fab, making it important to use the available space in an efficient manner. Some tools are quite boxy to help minimize the area used, or the footprint. Some tools are less boxy due to sheer requirements, i.e., the original equipment manufacturer (OEM) could only make it so small area-wise before it lost some functionality or performance. OEMs are incentivized to minimize footprint so companies will buy more of their tools. Large-scale OEMs include Applied Materials (AMAT), Tokyo Electron Limited (TEL), and LAM Research.

Here's a picture of an AMAT tool, the Endura. Note that while not perfectly box-l...
The Stereotype of the Stereotype
by Ike — 2025-06-04T21:06:54.165Z

I want to define a new term that will be useful in the discourse.

Imagine you landed in a foreign country with dogs, but with no word for dogs. You would see these things, and want to talk about them. You would like to be able to tell people whether they can bring dogs into your house, but you can’t, because there isn’t any word for “dog.” You’d like to ask if you can pet someone’s dog, but you can’t do that either, because when you ask “can I pet your dog?” they say “pet my what?” It would get pretty frustrating.

This is how I feel about a certain phenomenon I’ve noticed over the last year or so. I notice it all the time—It’s as ubiquitous in society as dogs are, and I see it just about as often—but there isn’t a word for it. People seem to make the same error of judgement over and over, and I want to talk about this error. There’s a lot I want to say about it. I want to be able to point out when someone is making the mistake. I want to ask people about whether I’m making the mistake. I want to talk about how some people seem to see very clearly, and make the mistake very rarely. Others seem to be very cloudy-eyed where this mistake is concerned, and they make it very often.

So, I’m going to make up a new term. The term is “The Stereotype of the Stereotype.” The mistake I’m talking about is not “The Stereotype of the Stereotype.” The Stereotype of the Stereotype is a noun. This mistake is a verb. The verb is “confusing the stereotype with The Stereotype of the Stereotype.”

Mixing up ‘the stereotype’ with ‘The Stereotype of the Stereotype’ is as significant a mistake as seeing the word “rainstorm,” and opening your umbrella. You aren’t allowed to mix up levels that way. The stereotype is the stereotype. The Stereotype of the The Stereotype is The Stereotype of the The Stereotype. And they are as different as a rainstorm, and the word “Rainstorm.”

Let’s use some examples to define this term “The Stereotype of the Stereotype.”

-

Sir Terrence Pratchett was a British satirical fantasy author, most famous for his Discworld series of books.

There are 41 of these books. Most of them follow distinct arrangements of characters. For example, eight of them follow a man named Sam Vimes. Another five follow a girl named Tiffany Aching. However, there’s one character who shows up in nearly every book. His name is Death.

Death appears in 39 of the 41...
2. Why intuitive comparisons of large-scale impact are unjustified
by Anthony DiGiovanni — 2025-06-04T20:30:04.797Z

We’ve seen so far that unawareness leaves us with gaps in the impartial altruistic justification for our decisions. It’s fundamentally unclear how the possible outcomes of our actions trade off against each other.

But perhaps this ambiguity doesn’t matter much in practice, since the consequences that predominantly shape the impartial value of the future (“large-scale consequences”, for short) seem at least somewhat foreseeable. Here are two ways we might think we can fill these gaps:

Implicit: “Even if we can’t assign EVs to interventions, we have reason to trust that our pre-theoretic intuitions at least track an intervention’s sign. These intuitions can distinguish whether the large-scale consequences are net-better than inaction.”[1]

Explicit: “True, we don’t directly conceive of fine-grained possible worlds, only coarse hypotheses. We can still assign precise EVs, though, by intuitively estimating the average value of the possible worlds in each hypothesis. These best-guess estimates will at least be better than chance.”

Do these approaches hold up, when we look concretely at how vast our unawareness could be? “The far future is complex” is a truism that many EAs are probably sick of hearing. But here I’ll argue that this complexity — which also plagues our impact on near-term pivotal events like AI takeoff — specifically undermines both Implicit and Explicit. For instance, we can’t simply say, “It’s intuitively clear that trying to spread altruistic values would be better than doing nothing”, nor, “If it’s better for future agents to be altruistic, we should try to spread altruism, since intuitively this is at least somewhat more likely to succeed than backfire”. This is because our intuitions about large-scale consequences don’t seem to account for unawareness with enough precision. If that’s true, we’ll need a new framework for impartially evaluating our actions under unawareness.

The argument in brief: First, a strategy’s sign from an impartial perspective seems unusually sensitive to factors we’re unaware of, compared to the local perspective we take in more familiar decision problems (cf. Roussos (2021)). Thus it’s unsurprising if our intuitions can justify claims like “A is net-positive” on the local scale, yet don’t justify such claims on the cosmic scale. Second, these intuitions seem so imprecisely calibr...
Dating Roundup #6
by Zvi — 2025-06-04T20:00:06.688Z

Previously: #1, #2, #3, #4, #5

Dating Roundup #4 covered dating apps. Roundup #5 covered opening without them.

Dating Roundup #6 covers everything else.

Table of Contents

You’re Single Because You Can’t Handle Basic Logistics.

You’re Single Because You Don’t Ask Questions.

You’re Single Because of Your Terrible Dating Tactics.

You’re Single Because You Refuse to Play Your Role.

You’re Single Because People Are Crazy About Age Gaps.

You’re Single and You Need Professional Help.

You’re Single Because You Never Close.

You’re Single Because You’re Bad at Sex And Everyone Knows.

You’re Single Because You Are Only a Fan.

You’re Single Because of Preference Falsification.

You’re Single Because You Have Insufficient Visual Aids.

You’re Single Because You Told Your Partner You Didn’t Want Them.

You’re Single Because of Your Terrible Dating Strategy.

You’re Single Because You Don’t Enjoy the Process.

You’re Single Because You Don’t Escalate Quickly.

You’re Single Because Your Standards Are Too High.

You’re Single Because You Read the Wrong Books.

You’re Single Because You’re Short, Sorry, That’s All There Is To It.

You’re Single Because of Bad Government Incentives.

You’re Single Because You Don’t Realize Cheating is Wrong.

You’re Single Because You’re Doing Polyamory Wrong.

You’re Single Because You Don’t Beware Cheaters.

You’re Single Because Your Ex Spilled the Tea.

You’re Single Because You’re Assigning People Numbers.

You’re Single Because You Are The Wrong Amount of Kinky.

You’re Single Because You’re Not Good Enough at Sex.

You’re Single But Not Because of Your Bodycount.

You’re Single Because They Divorced You.

You’re Single Because No One Tells You Anything.

You’re Single And You’re Not Alone.

You’re Single Because Things Are Steadily Getting Worse.

You’re Single Because You Didn’t Go to College.

You’re Single But This Isn’t About You.

You’re Single so Let’s Go to the Videotape.

You’re Single Because You Don’t Seek Out Good Advice.

You’re Single So Here’s Some Hope.

You’re Single Because You Can’t Handle Basic Logistics

You can take the pressure off yourself to plan the perfect date.

Instead, plan any date at all.

Shoshana Weissmann: I was hanging out with my married friend, who is a great father and husband, telling him how men cannot plan dates. He said, “What’s there to plan? They pick a bar, a restaurant, and a park for a walk nearby.” And

...