trentmkelly/LessWrong-43k huggingface dataset:
AI Safety & Entrepreneurship v1.0
by Chris_Leong — 2025-04-26T14:37:11.037Z

Why did I create both a post and a wiki article?

Posts are best for making sure people see the initial version of the post, whilst Wiki articles are best for long-term maintenance. Posting an article that links to a Wiki page provides the best of both worlds.

For the most up-to-date version, see the Wiki page.

Articles:

There should be more AI safety organisations

Why does the AI Safety Community need help founding projects?

AI Assurance Tech Report

AI Safety as a YC Startup

Alignment can be the ‘clean energy’ of AI

AI Tools for Existential Security

Incubation Programs:

Def/acc at Entrepreneur First

- this is a new program, which focuses on "defensive" tech, which will produce a wide variety of startups, some of which may help with reducing x-risk, but others may include AI applications like autonomous weapons or diagnostic medical software, which, while they generally won't increase x-risk, may not reduce it either.

Catalyze - AI Safety Incubation

Seldon Accelerator

: “We take on AGI risk with optimism”

AE Studio

has expressed interest in helping seed neglected approaches, incl. ideas that take the form of business ideas

AIS Friendly General Program:

Founding to Give

: General EA incubator

Entrepreneur First

- They run def/acc, so this could be a decent alternative

Halycon Futures

: “We’re an entrepreneurial nonprofit and VC fund dedicated to making AI safe, secure and good for humanity.” They do new project incubation, grants, investments, ect.

Communities:

Entrepreneurship Channel EA Anyway

AI Safety Founders:

Website

,

Discord

,

LinkedIn

VC:

AI Safety Seed Funding Network

Lionheart Ventures

- “We invest in the wise development of transformative technologies.”

Juniper Ventures

- Invests in companies “working to make AI safe, secure and beneficial for humanity”. Published the AI Assurance Tech Report

Polaris Ventures

- “We support projects and people aiming to build a future guided by wisdom and compassion for all”

Mythos Ventures

- “an early-stage venture capital firm investing in prosocial technologies for the transformative AI era.”

Metaplanet

- “We support novel and evidence-based innovation that could produce an outsized return for the benefit of humankind.”

Anthology Fund

- Backed by Anthropic. One of the five key areas is “trust and safety tooling”

Menlo Ventures - Lead the GoodFire round (article)

Safe AI Fund

Organisational Support:

Ashgro

- fiscal sponsorship

Rethink Pr

...
How I Think About My Research Process: Explore, Understand, Distill
by Neel Nanda — 2025-04-26T10:31:34.305Z

This is the first post in a sequence about how I think about and break down my research process. Post 2 is coming soon.

Thanks to Oli Clive-Griffin, Paul Bogdan, Shivam Raval and especially to Jemima Jones for help and feedback, and to my co-author Gemini 2.5 Pro - putting 200K tokens of past blog posts and a long voice memo in the context window is OP.

Introduction

Research, especially in a young and rapidly evolving field like mechanistic interpretability (mech interp), can often feel messy, confusing, and intimidating. Where do you even start? How do you know if you're making progress? When do you double down, and when do you pivot?

These are far from settled questions, but I’ve supervised 20+ papers by now, and have developed my own mental model of the research process that I find helpful. This isn't the definitive way to do research (and I’d love to hear other people’s perspectives!) but it's a way that has worked for me and others.

My goal here is to demystify the process by breaking it down into stages and offering some practical advice on common pitfalls and productive mindsets for each stage. I’ve also tried to be concrete about what the various facets of ‘being a good researcher’ actually mean, like ‘research taste’ (see post 3). I’ve written this post for a mech interp audience, but hopefully it is useful for any empirical science with short feedback loops, and possibly even beyond that.

This guide focuses more on the strategic (high-level direction, when to give up or pivot, etc) and tactical (what to do next, how to prioritise, etc) aspects of research – the "how to think about it" rather than just the "how to do it." Some of skills (coding, reading papers, understanding ML/mech interp concepts) are vital for how to do it, but not in scope here (I recommend the ARENA curriculum and my paper reading list if you need to skill up).

How to get started? Strategic and tactical thinking are hard skills, and it is rare to be any good at them when starting out at research (or ever tbh). The best way to learn them is by trying things, making predictions, seeing what you get right or wrong (i.e., getting feedback from reality), and iterating. Mentorship can substantially speed up this process by providing "supervised data" to learn from, but either way you ultimately learn by doing.

I’ve erred towards making this post comprehensive, which ...
Don't you mean "the most *conditionally* forbidden technique?"
by Knight Lee — 2025-04-26T03:45:38.026Z

Hear me out, I think the most forbidden technique is very useful and should be used, as long as we avoid the "most forbidden aftertreatment:"

An AI trained on interpretability techniques must not be trained on capabilities after (or during) it is trained on interpretability techniques, otherwise it will relearn bad behaviour—in a more sneaky way.
An AI trained on interpretability techniques cannot be trusted any more than old version of itself, which hasn't been trained on interpretability techniques yet. Evaluations must be performed on the old version of itself.

An AI company which trains its AI on interpretability techniques, must publish the old version (which hasn't been trained on them), with the same availability as the new version.

The natural selection argument:

The reason why the most forbidden technique is forbidden, because it lets bad behaviours evolve against interpretability. Bad behaviours which are caught by interpretability techniques are killed, and then capabilities training creates more bad behaviours, and then bad behaviours which get caught are killed again, and then capabilities training creates more bad behaviours again!

After many generations, natural selection creates the most insidious and terrifying bad behaviours, which successfully hide from all interpretability techniques.

However, if we only train on interpretability techniques a single time, and do zero capabilities training afterwards, then we are only killing the bad behaviours a single time, and without giving the surviving bad behaviours any time to birth a second generation of bad behaviours.

Human evolution

Imagine if during the course of human evolution, a robot observed our brain activity in the amygdala, the part of the brain responsible for anger.

Whenever the population increases above 1 million, it kills half of all people with the highest anger ratings, based on brain activity in the amygdala.

For a short time, this will cause people to be less angry, but eventually evolution will simply cause humans to emulate the concept of anger in another brain region, one that the robot doesn't know about. In the end, we end up just as angry as before, but less interpretable to the robot.

However, imagine if the robot didn't do this again and again over the course of our evolution, but only did it once, and "stopped evolution" from taking place afterwards. In this case, we actually wou...
Land with no aunties
by thellimist — 2025-04-26T01:20:06.661Z
Link: https://kanyilmaz.me/2025/03/28/aunties.html

Posted a piece on how removing informal social control (the “auntie layer”) affects a city’s memetic landscape, using SF as case study. Interested in rationalist critiques: Are such regulators net-positive or pareto-inefficient?
AI 2027 Thoughts
by PeterMcCluskey — 2025-04-26T00:00:23.699Z
Link: https://bayesianinvestor.com/blog/index.php/2025/04/25/ai-2027-thoughts/

AI 2027 portrays two well thought out scenarios
for how AI is likely to impact the world toward the end of this decade.

I expect those scenarios will prove to be moderately wrong, but close
enough to be scary. I also expect that few people will manage to make
forecasts that are significantly more accurate.

Here are some scattered thoughts that came to mind while I read AI 2027.

The authors are fairly pessimistic. I see four key areas where their
assumptions seem to lead them to see more danger than do more mainstream
experts. They see:

a relatively small capabilities lead being enough for a group to
conquer the world

more difficulty of alignment

more difficulty of detecting deception

AI companies being less careful than is necessary

I expect that the authors are being appropriately concerned on about two
of these assumptions, and a bit too pessimistic on the others. I'm
hesitant to bet on which assumptions belong in which category.

They don't focus much on justifying those assumptions. That's likely
wise, since prior debates on those topics have not been very productive.
Instead, they've focused more on when various changes will happen.

This post will focus on aspects of the first two assumptions for which I
expect further analysis to be relatively valuable.

Decisive Strategic Advantage

Their scenario has the somewhat surprising aspect that there are exactly
two AI projects close enough to winning the race to matter. Other
projects are 3+ months behind. This aspect seems like it has a 20%
chance of happening. Does a project being three months behind a leader
mean the project doesn't matter? My intuition says no, but I haven't
been able to articulate clear arguments one way or the other. And it
doesn't look like we're on track to have a larger than three month gap
between the first and third projects.

Much of how much that gap matters depends on the takeoff
forecast.

I don't see a good way to predict what degree of capabilities
disadvantage causes a project to become irrelevant. So it seems like we
ought to be pretty uncertain whether projects in third, fourth, and
fifth places would end up negotiating to get control over some share of
the future lightcone.

How much of a lead in nuclear weapons would the US have needed in the
1940s to conquer the world?

How much of an advantage did Cortés have over the
Aztecs?

How much of an advantage did homo sapiens have over neand...
Who's Working On It? AI-Controlled Experiments
by sarahconstantin — 2025-04-25T21:40:02.543Z

Dr. Lee Cronin’s “Chemputer” robot

A lot of applications of AI in the experimental sciences boil down to data analysis.

You take existing datasets — be they genomic sequences, images, chemical measurements, molecular structures, or even published papers — and use a model to analyze them.

You can do a lot of useful things with this, such as:

inference to predict the properties of a new example where part of the data is missing

e.g. protein structure models predict unknown molecular structures from known nucleic acid sequences

generation to invent typical/representative examples of a class

e.g. many molecules “generated by AI” are invented structures produced in response to a query specifying a chemical or structural property, predicted to be representative of the real molecules that fit the criterion.

natural-language search to pull up and adapt information from a dataset so that it answers a plain-English query.

Mundane though it may sound, this dramatically speeds up learning and interdisciplinary idea generation. It might ultimately have a bigger impact on science than more specialized models trained on experimental data.

But one limit on all these kinds of models is that they’re piggybacking on data collected by human scientists. They can find connections that no human has discovered, and they can attempt to generalize out of their dataset, but until they are connected in a loop that includes physical experiments, I don’t think it’s fair to consider them “AI scientists.”

However, once you do close the loop, allowing the AI to suggest experiments, observe the results, and iterate, then I think one of the most important in-principle differences between AIs and humans falls away.

Experimentation allows us to learn causality, not just correlation. Experimentation is how a baby learns to walk; it’s how animals build the physical reasoning that we often see examples of LLMs failing at. Experimentation — ideally with enough sensor logs to build tacit knowledge of how real-world physical projects fail and need to be adjusted/repaired, not just an idealized matrix of “experimental results” produced once human experimenters have ironed out all the difficulty — is what would give me confidence that an AI could autonomously invent new technologies or discover new phenomena in the physical world.

I am, of course, not alone in noticing this! After a few years when it seemed lik...
[Linkpost] AI War seems unlikely to prevent AI Doom
by thenoviceoof — 2025-04-25T20:44:48.267Z
Link: https://thenoviceoof.com/ai/war/ai_war_v1_0.html

This is a linkpost: link to post.

Originally this was part of a much larger work. However, I realized that I don't think I've seen the specific argument around GPU salvage spelled out. Since I'm also using a new writing format, I figured I could get feedback on both the content and format at the same time.

That said, despite the plausible novelty of one of the arguments I don't think it will be especially interesting to LW, since it is based on oddly specific assumptions: this makes more sense in the context of a broad AI risk argument. It also feels kind of obvious?

The format is the interesting bit. For motivation, sometimes people have opposing reactions to AI risk arguments:

"Your argument is nice and concise, but it doesn't address the one specific objection that I consider to be a crux, so I consider your broader argument invalid."
"Your argument is 100 pages long? I ain't reading all that, but I'm happy for you. Or sorry that happened."

(This might be a general feature for contentious topics.)

This format focuses on presenting the high level skeleton of an argument, and then allowing readers to drill in where they want. If a reader agrees with many high level points (but not all of them), they can spend their time reading specifically where we disagree.

So! Do you have any feedback on this format?

Early feedback you might agree with, and other thoughts.

A piece of early feedback is that bullet points are harder to read than prose; perhaps it would be better to have the top level be prose, with supporting foldouts? That is, perhaps the top level as it is... is too austere? The current work was written in an "all bullets, all the time" style that is resisting easy transformation, but let me know if the outline form is a tremendous turn off, compared to something more prose-y.
A piece of early feedback I got was that they enjoyed reading 100 page theses; I suspect that people that have put time into learning how to read standard long form text may find this new format annoying.
A related piece of early feedback is that the phone experience requires too much interaction; they wanted something similar that could let them just scroll through everything. I have ideas about how to do this, but if you're reading on a phone I'd be interested to hear what you think.
AISafety.info has a more traditional outline with 1-2 levels and a chatbot. In comparison, I expect this forma

...
Worries About AI Are Usually Complements Not Substitutes
by Zvi — 2025-04-25T20:00:03.421Z

A common claim is that concern about [X] ‘distracts’ from concern about [Y]. This is often used as an attack to cause people to discard [X] concerns, on pain of being enemies of [Y] concerns, as attention and effort are presumed to be zero-sum.

There are cases where there is limited focus, especially in political contexts, or where arguments or concerns are interpreted perversely. A central example is when you site [ABCDE] then they’ll find what they consider the weakest one and only consider or attack that, silently discarding the rest entirely. Critics of existential risk do that a lot.

So it does happen. But in general one should assume such claims are false.

Thus, the common claim that AI existential risks ‘distract’ from immediate harms. It turns out Emma Hoes checked, and the claim simply is not true.

The way Emma frames worries about AI existential risk in her tweet – ‘sci-fi doom’ – is beyond obnoxious and totally inappropriate. That only shows she was if anything biased in the other direction here. The finding remains the finding.

Emma Hoes: New paper out in @PNASNews! Existential AI risks do **not** distract from immediate harms. In our study (n = 10,800), people consistently prioritize current threats – bias, misinformation, job loss – over sci-fi doom!

Title: Existential Risk Narratives About AI Do Not Distract From Its Immediate Harms.

Abstract: There is broad consensus that AI presents risks, but considerable disagreement about the nature of those risks. These differing viewpoints can be understood as distinct narratives, each offering a specific interpretation of AI’s potential dangers.

One narrative focuses on doomsday predictions of AI posing long-term existential risks for humanity.

Another narrative prioritizes immediate concerns that AI brings to society today, such as the reproduction of biases embedded into AI systems.

A significant point of contention is that the “existential risk” narrative, which is largely speculative, may distract from the less dramatic but real and present dangers of AI.

We address this “distraction hypothesis” by examining whether a focus on existential threats diverts attention from the immediate risks AI poses today. In three preregistered, online survey experiments (N = 10,800), participants were exposed to news headlines that either depicted AI as a catastrophic risk, highlighted its immediate soci

...
Why would AI companies use human-level AI to do alignment research?
by MichaelDickens — 2025-04-25T19:12:56.202Z

Cross-posted from my website.

Many plans for how to safely build superintelligent AI have a critical section that goes like this:

Develop AI that's powerful enough to do AI research, but not yet powerful enough to pose an existential threat.

Use it to assist with alignment research, thus greatly accelerating the pace of work—hopefully enough to solve all alignment problems.

You could call this process "alignment bootstrapping".

This is a central feature of DeepMind's plan (see "Amplified oversight"), Anthropic's plan (see "Scalable Oversight"), and independent plans written by Sam Bowman (an AI safety manager at Anthropic), Joshua Clymer (a researcher at Redwood Research), and Marius Hobbhahn (CEO of Apollo Research).

There are various reasons why alignment bootstrapping could fail[1] even if implemented well, and some of those plans acknowledge this. But I'm also concerned about whether alignment bootstrapping will be implemented at all.

When the time comes, will AI companies actually spend their resources on alignment bootstrapping?

When AI companies have human-level AI systems, will they use them for alignment research, or will they use them (mostly) to advance capabilities instead?

AI companies currently employ many human-level humans, and use a small percentage of them to do alignment research. If it makes sense for them to use most of their human-level AIs to do alignment research, wouldn't it also make sense to use most of their human researchers to do alignment research?

But they don't do that. Most of their human researchers work on advancing AI capabilities.

It's more likely that they use human-level AIs the same way they use human researchers: almost all of them work on accelerating capabilities, and a small minority work on safety. Which probably means capabilities outpace safety, which probably means we die.

Some companies argue that they need to advance capabilities right now to stay competitive. Perhaps that's true. Consider what the world will look like once the first company develops human-level AI. At that point, the #2 company will only be a few months behind at most. So the leading company will once again say, "Sorry, we can't use our human-level AI to work on alignment, we have to keep advancing capabilities to stay ahead." And they will continue saying this right up until their AI is powerful enough to kill everyone.

Counterpoint: AI companies ...
How Democratic Is Effective Altruism — Really?
by B Jacobs — 2025-04-25T16:02:42.915Z
Link: https://bobjacobs.substack.com/p/how-democratic-is-effective-altruism

Introduction

Effective Altruism (EA) is a social movement that aims to use reason and evidence to help others as much as possible. It encourages people to ask not just “how to do good”, but how to do the most good. This has led members to support things like global health interventions, existential risk reduction, and animal welfare.

I used to be closely involved in the movement, and I still think many of its ideas are worth defending. But as the movement has grown, so have certain structural problems: increasing reliance on large donors, pushback on dissent, and systems that concentrate influence in subtle but significant ways. This post is about those concerns — not to denigrate the movement, but to explore how it might better live up to its own stated values.

EA and conformity

One of the clearest places where these structural issues show up is in how the movement handles conformity and internal disagreement. Carla Zoe Cremer, a former EA insider, has become an outspoken critic of how EA functions internally. Back in 2020, she warned about “value-alignment”, her term for extreme intellectual conformity within EA:

value-alignment means to agree on a fundamental level. It means to agree with the most broadly accepted values, methodologies, axioms, diet, donation schemes, memes and prioritisations of EA.

Her concerns deepened in 2021, when she co-authored a peer-reviewed paper with Luke Kemp on existential risk studies — one of EA’s flagship cause areas. The paper argued for more diverse voices and warned that the field had become overly “techno-utopian” and was in need of democratic reform.

The reception was… rough. According to her:

It has been the most emotionally draining paper we have ever written. We lost sleep, time, friends, collaborators, and mentors because we disagreed on: whether this work should be published, whether potential EA funders would decide against funding us and the institutions we're affiliated with, and whether the authors whose work we critique would be upset.

While many in the community responded constructively, others reportedly sought to suppress the paper — not on academic grounds, but out of fear that it might alienate funders. The clear implication here is that critique is encouraged, as long as it doesn’t threaten the financial or ideological foundations of the movement. In response, Cremer laid out a series of concrete reforms ...