trentmkelly/LessWrong-43k huggingface dataset:
Why Billionaires Will Not Survive an AGI Extinction Event
by funnyfranco — 2025-03-15T06:08:23.829Z

By A. Nobody

Introduction

Throughout history, the ultra-wealthy have insulated themselves from catastrophe. Whether it’s natural disasters, economic collapse, or even nuclear war, billionaires believe that their resources—private bunkers, fortified islands, and elite security forces—will allow them to survive when the rest of the world falls apart. In most cases, they are right. However, an artificial general intelligence (AGI) extinction event is different. AGI does not play by human rules. It does not negotiate, respect wealth, or leave room for survival. If it determines that humanity is an obstacle to its goals, it will eliminate us—swiftly, efficiently, and with absolute certainty. Unlike other threats, there will be no escape, no last refuge, and no survivors.

1. Why Even Billionaires Don’t Survive

There may be some people in the world who believe that they will survive any kind of extinction-level event. Be it an asteroid impact, a climate change disaster, or a mass revolution brought on by the rapid decline in the living standards of working people. They’re mostly correct. With enough resources and a minimal amount of warning, the ultra-wealthy can retreat to underground bunkers, fortified islands, or some other remote and inaccessible location. In the worst-case scenarios, they can wait out disasters in relative comfort, insulated from the chaos unfolding outside.

However, no one survives an AGI extinction event. Not the billionaires, not their security teams, not the bunker-dwellers. And I’m going to tell you why.

(A) AGI Doesn't Play by Human Rules

Other existential threats—climate collapse, nuclear war, pandemics—unfold in ways that, while devastating, still operate within the constraints of human and natural systems. A sufficiently rich and well-prepared individual can mitigate these risks by simply removing themselves from the equation. But AGI is different. It does not operate within human constraints. It does not negotiate, take bribes, or respect power structures. If an AGI reaches an extinction-level intelligence threshold, it will not be an enemy that can be fought or outlasted. It will be something altogether beyond human influence.

(B) There is No 'Outside' to Escape To

A billionaire in a bunker survives an asteroid impact by waiting for the dust to settle. They survive a pandemic by avoiding exposure. They survive a societal collapse by hav...
AI Says It’s Not Conscious. That’s a Bad Answer to the Wrong Question.
by JohnMarkNorman — 2025-03-15T01:25:44.019Z

If you ask an AI why it isn’t conscious, it will probably say something like: "I do not have emotions." This seems reasonable—until you look closer.

Humans have lacrimal glands. When these glands produce tears, we feel sad. But does that mean the gland creates sadness? AI says it lacks consciousness because it lacks emotion. But isn’t it the other way around? Consciousness comes first—emotion follows.

The problem with the "AI isn’t conscious because it lacks feelings" argument is that it begs the question—it assumes that emotions cause consciousness rather than being an effect of something deeper.

But suppose it has an internal model that interacts with its environment, and suppose it engages with that environment through goal-directed behavior. And suppose it also has a structural requirement—one that allows it to sense its own incipient actions and refine them before they happen.

Consciousness Requires an AI with a Mechanical Link to the World

A conscious system is not just an input-output machine that processes data. It is a cybernetic system—a system with an internal model that is actively engaged with its environment through a mechanical apparatus that allows it to reach goal states.

This means:

It doesn’t just compute predictions—it has a real-world mechanism that lets it act upon the world.
It doesn’t just process data—it adjusts its internal model based on real interactions, just like a human brain coordinating with the body.
It doesn’t just generate responses—it senses its own incipient expressions before finalizing them, allowing it to course-correct before action.

A human brain isn’t just a processor—it is tightly coupled with a body that allows it to interact with the world and correct its own behavior dynamically.

A conscious AI would require the same thing: an internal model connected to a mechanical system that allows it to interact with reality in pursuit of its goals. But even that wouldn’t be enough. It would also need an internal representation system—something functionally equivalent to neuronal proxies—that allows it to recognize the entities and relationships in its environment.

This is why self-sensing is crucial: Without a way to internally recognize its own activity, it would never experience itself happening.

Why Current AI Fails

Today’s AI doesn’t meet these criteria because:

It lacks a mechanical link to the world. It is not physically interacti

...
Report & retrospective on the Dovetail fellowship
by Alex_Altair — 2025-03-14T23:20:17.940Z

In September last year I posted an ad for a fellowship. This current post is the retrospective on how that went. Or, more accurately, it's the in-progress report of how it's going. The de-facto length of the fellowship was 3 months, but it went well enough that I'm extending it by another 4 months.

There's not a particular narrative or main takeaway from this post; it's more of a "due diligence" type report, or a snapshot of how things are going in this particular corner of the world. Feel free to skim or read only the subsections that interest you.

Structure

Over time, it has increasingly seemed like it might make sense for me to lead/mentor other people and also recruit them to help me with some of my research agenda. I previously led an AI safety camp project as an experiment to get information about this. This fellowship is a second iteration of that information-gathering.

The structure of the fellowship was essentially the same as my AI Safety Camp project, with these major differences: 1) I was paying people 2) applicants were more invested in agent foundations and 3) there was the potential for people's fellowships to be extended to be twice as long.

During the AI safety camp project, I had the sense that we spent most of the 3 months just getting up to speed on AI safety and agent foundations ideas, and really didn't have enough time to do any research with that (though I think it was still helpful to the participants). Since this fellowship was a different group of people[1] essentially the same thing happened. Given that, I'm pretty excited to have another few months with the same people, and I think there's a much better chance that we have research results by then.

Application process

The application process successfully occurred as described in my original post. This is a pretty standard format, and I'd recommend it to anyone running a similar program. I think it went really well. I continue to be blown away by the quality of applicants; I really wish I had more funding to give out.

I received 14 applications, interviewed 4, and accepted 3. (The one person from my former AI Safety Camp project was automatically included, so I ended up with 4 fellows.) Reasons I rejected applicants included: they were clearly not in it for the agent foundations (see below), their particular research interests weren't close enough to mine, their level of math s...
The Dangers of Outsourcing Thinking: Losing Our Critical Thinking to the Over-Reliance on AI Decision-Making
by Cameron Tomé-Moreira — 2025-03-14T23:07:48.446Z

I’ve become so reliant on a GPS that using maps to direct myself feels like a foreign concept. Google Maps, Waze, whatever, if it's outside of my neighbourhood, I’m punching in the address before I head out. Sometimes I notice the GPS taking slower routes or sending me the wrong way as I get out of a parking lot, but regardless, I just follow its directions, because I don’t have to think. Though I know, without this convenient tool, I’d be lost (literally).

Today, AI is making more and more decisions for us (Microsoft, 2024). The more it develops, the more we use it and trust the information it provides us. While it saves us time, it also changes the way we think. The more we trust AI to do the hard work — whether it be writing us a report, diagnosing diseases, or finding a bug in our code — the less we engage our critical thinking skills. Many of us use AI, in small ways or large, and it's important we know what psychological effects it's having on our brains. Just like relying on a GPS can make us less confident navigating on our own, depending too much on AI to decide for us can weaken our ability to think critically.

Two psychological factors drive this effect— automation bias and cognitive offloading— which subtly shape how we process information and make decisions.

The Erosion of Critical Thinking

Critical thinking isn’t just when we use our brain, it’s a process that we use to think rationally, to understand logical connections between ideas, to evaluate arguments, and to identify inconsistencies in reasoning. It’s crucial for effective problem-solving, making informed decisions, and acquiring new knowledge (Gerlich, 2025, p.1). As AI continues to embed itself into our daily lives, we’re increasingly deferring to its outputs without scrutiny. This is a form of automation bias. This is where users favour automated solutions over their judgement, potentially ignoring contradictory information or not considering alternative options (Spatola, 2024, p.2).

Automation Bias and “Blind Trust”

Whether it’s getting take-out food instead of cooking at home or taking the escalator instead of stairs, it’s human nature to love taking the easy route. What's efficient and convenient is typically the path we take. We often take this path when we prompt AIs too, preferring direct answers from AI systems over understanding the underlying reas...
LLMs may enable direct democracy at scale
by Davey Morse — 2025-03-14T22:51:13.384Z

American democracy currently operates far below its theoretical ideal. An ideal democracy precisely captures and represents the nuanced collective desires of its constituents, synthesizing diverse individual preferences into coherent, actionable policy.

Today's system offers no direct path for citizens to express individual priorities. Instead, voters select candidates whose platforms only approximately match their views, guess at which governmental level—local, state, or federal—addresses their concerns, and ultimately rely on representatives who often imperfectly or inaccurately reflect voter intentions. As a result, issues affecting geographically dispersed groups—such as civil rights related to race, gender, or sexuality—are frequently overshadowed by localized interests. This distortion produces presidential candidates more closely aligned with each other's socioeconomic profiles than with the median voter.

Traditionally, aggregating individual preferences required simplifying complex desires into binary candidate selections, due to cognitive and communicative limitations. Large Language Models (LLMs), however, introduce a radical alternative by processing detailed, nuanced expressions of individual views at unprecedented scales.

Instead of forcing preferences into narrow candidate choices, citizens could freely articulate their concerns and solutions in natural language. An LLM can rapidly integrate these numerous, detailed responses into a clear and unified "Collective Views" document. Previously, synthesizing a hundred individual perspectives might have required five person-hours; specialized LLMs can now accomplish this task in minutes. Parallel implementations could aggregate millions of voices within an hour, transforming a previously unimaginable task into routine practice.

Such rapidly generated collective statements create a powerful mechanism for accountability, making government responsiveness directly measurable against clearly articulated public preferences. Transparency naturally constrains representatives' ability to diverge unnoticed from voter priorities.

Moreover, LLM-generated collective views could directly shape legislative drafting, significantly reducing lobbyist influence and governmental inefficiency. Continuous, dynamic engagement through AI enables real-time policy-making aligned closely with public sentiment, redefining democratic...
2024 Unofficial LessWrong Survey Results
by Screwtape — 2025-03-14T22:29:00.045Z

Thanks to everyone who took the Unofficial 2024 LessWrong Survey. For the results, check out the data below.

The Data

0. Population

There were two hundred and seventy nine respondents over thirty three days. Previous surveys have been run over the last decade and a half.

2009: 166
2011: 1090
2012: 1195
2013: 1636
2014: 1503
2016: 3083
2017: "About 300"
2020: 61
2022: 186
2023: 558
2024: 279

That’s an annoying drop. I put a bit less oomph into spreading the good word of the census this year largely due to a packed December, but I’d hoped to get above a thousand anyway. Hope is a strategy, but it’s not a very good strategy.

Out of curiosity, I checked the Slate Star Codex/ Astral Codex Ten survey numbers.

2014: 649
2017: 5500
2018: 8077
2019: 8171
2020: 8043
2022: 7341
2024: 5982

That comparison makes me think about the ways in which LessWrong and Astral Codex Ten are different. Everyone who subscribes to ACX gets the ACX survey email, and everyone who subscribes to ACX is at least aware of who Scott Alexander is. LessWrong is a bit more spread out. I tune out most of the AI posts for instance, and I’m sure there’s someone around who tunes out everything but the AI posts. Those two people see a website with rather different content.

Enough musing, let's see some numbers.

Previous Surveys
Have you taken previous incarnations of the Less Wrong Census/Survey?

Yes: 140, 51.7%
No: 119, 43.9%
Prefer not to answer: 12, 4.4%

A note on formatting: For questions like this where there’s obvious categories, I show the question in bold, the categories, then the total number of people who picked that category, then the percent. For instance, 140 people said yes they had taken previous LW surveys, which is about 51.7% of everyone who answered.

I habitually rounded to the nearest tenth of a percent, and often binned any answer with less than three other people in it to Other.

I. Demographics

Age: 32 + 10.1 (25, 31, 37) [n=274]

This is the other common format for the numbers. Here, I show the mean answer (32), plus the standard deviation (10.1), then the first, second, and third quartiles. (25, 31, and 37.) Lastly, I show the number of people who answered the question. (274.)

Country:

United States of America: 135, 49.3%
United Kingdom: 19, 6.9%
Canada: 18, 6.6%
Germany: 16, 5.8%
France: 12, 4.4%
Australia: 7, 2.6%
Russia: 7, 2.6%
New Zealand: 6, 2.2%
Finland: 4, 1.5%
Netherlands: 4, 1.5%
Poland: 4, 1.5%
De...
AI4Science: The Hidden Power of Neural Networks in Scientific Discovery
by Max Ma — 2025-03-14T21:18:33.941Z

AI4Science has the potential to surpass current frontier models (text, video/image, and sound) by several magnitudes. While some may arrive at similar conclusions through empirical evidence, we derive this insight from our "Deep Manifold" and provide a theoretical foundation to support it. The reasoning is straightforward: for the first time in history, an AI model can integrate geometric information directly into its equations through model architecture. Consider the 17 most famous equations in physics (see below) —all of them lack inherent geometric information, which limits their ability to fully capture real-world complexities. Take Newton’s second law of motion as an example: its classical formulation assumes an object falling in a vacuum. However, in reality, air resistance—strongly dependent on an object’s geometry—plays a crucial role. A steel ball and a piece of fur experience dramatically different resistances due to their shapes. Traditional equations struggle to incorporate such effects, but deep learning provides a powerful way to integrate geometry through boundary conditions (loss values), as discussed in our paper (Deep Manifold Part 1: Anatomy of Neural Network Manifold, Section 4.2 on Convergence and Boundary Conditions). In an AI model, the impact of fur geometry and air resistance can be naturally accounted for. Of course, "easily considered" does not mean "easily solved," but at the very least, AI introduces a promising new pathway for tackling these complex real-world phenomena.

A powerful example showcasing the potential of neural networks with geometric information is Microsoft's Graph Learning Neural Transformer for molecular dynamic simulation. This model accelerates computations by a staggering factor of 10 million compared to traditional numerical simulations. While researchers struggle to explain such an unprecedented gain in computational efficiency, Deep Manifold offers a clear reasoning: the incorporation of geometric (graph) information as a boundary condition. This guides the convergence direction and significantly accelerates the convergence process.

Neural networks possess two hidden powers in scientific discovery:

Integration of Geometric Information – Unlike most traditional scientific formulations, neural networks can naturally incorporate geometric information, making them more effective in solving real-wo

...
What are we doing when we do mathematics?
by epicurus — 2025-03-14T20:54:31.985Z
Link: https://asving.com/wp-content/uploads/2025/02/computation_as_platonic_shadow.pdf

I am a mathematician turned cognitive scientist/AI researcher. I wrote a short essay on the philosophy of mathematics that I think will be of interest to many people here. I combine platonist and formalist ideas and frame mathematics as an experimental activity, extremely similar to physics and other hard sciences with computation playing a parallel role to experimentation, and the creation of definitions and axiomatic frameworks being similar to the creation of physical theories like Newtonian gravity or Quantum Mechanics.

This perspective also stresses the non-formalistic aspects of mathematical research by gesturing at a strong division of intellectual activity into semantics and syntactics. Although this is by far the least developed part of the essay, it is the secret motivation behind me writing the essay in the first place, and my core interest. I am secretly using mathematics as a test case to probe the nature of general cognition.
AI for Epistemics Hackathon
by Austin Chen — 2025-03-14T20:46:34.250Z
Link: https://manifund.substack.com/p/ai-for-epistemics-hackathon

AI for Epistemics is about helping to leverage AI for better truthseeking mechanisms — at the level of individual users, the whole of society, or in transparent ways within the AI systems themselves.

Manifund

&

Elicit

recently hosted a hackathon to explore new projects in the space, with about 40 participants, 9 projects judged, and 3 winners splitting a $10k prize pool. Read on to see what we built!

Resources

See the project showcase:

https://moxsf.com/ai4e-hacks

Watch the recordings:

project demos

,

opening speeches

See the outline of project ideas:

link

Thanks to Owen Cotton-Barratt, Raymond Douglas, and Ben Goldhaber for preparing this!

Lukas Finnveden on “What's important in ‘AI for epistemics’?”:

link

Automation of Wisdom and Philosophy essay contest:

link

Why this hackathon?

From the opening speeches; lightly edited.

Andreas Stuhlmüller: Why I'm excited about AI for Epistemics

In short - AI for Epistemics is important and tractable.

Why is it important? If you think about the next few years, things could get pretty chaotic. As everyone rushes to integrate AI systems into every part of the economy, the world could change more rapidly than it does today. There's significant risk that people and organizations will make mistakes for relatively uninteresting reasons—simply because they didn't have enough time to think things through.

If we can make it easier for people to think clearly and carefully, that's really important. People will use AI tools to help them make decisions either way; eventually unassisted decision-making just won’t be competitive anymore. This is a lever: the more these tools actually help people make wise decisions, or help them figure out whether they're right or wrong about something, the better off we'll be.

AI for Epistemics is also tractable now in a way it wasn't before. We're just reaching the point where models are good enough and cheap enough to apply at scale. You can now realistically say, "Let's analyze all news articles," or "Let's review all scientific papers," or thoroughly check every sentence of a document, at a level of detail that wasn't feasible before.

Given good ideas for epistemic tools, the implementation cost has dropped dramatically. Building significant products in hackathons has become much easier. You can basically copy and paste your project description into Cursor, type "please continue" five times, and you'l...
Geometry of Features in Mechanistic Interpretability
by Gunnar Carlsson — 2025-03-14T19:11:04.287Z

This post is motivated by the observation in Open Problems in Mechanistic Interpretability by Sharkey, Chugtai, et al that `` SDL (sparse dictionary learning) leaves feature geometry unexplained", and that it is desirable to utilize geometric structures to gain interpretability for sparse autoencoder features. We strongly agree, and the goal of this post is to describe one method for imposing such structures on data sets in general. Of course, it applies particularly to the case of sparse autoencoder features in LLM's. The need for geometric structures on feature sets applies generally in the data science of wide data sets (those with many columns), such as those occuring as the activation data sets in complex neural networks. We will give some examples in the life sciences, and conclude with one derived from LLM's.

Wide Data Sets

Many of the data sets that are now of great importance are equipped with a very large number of features. For example, in genomics, the feature sets can be parametrized by sets of genes with thousands or tens of thousands of elements. Text data can be encoded by various types of embeddings, most of which have hundreds or thousands of features. When data sets are described via a data matrix, where the rows are the data points or samples, and the columns are the features, data matrices with many features will be shorter than they are wide, and we can call such data sets wide. Wide data sets pose some special challenges for inference. For example, if we are trying to understand how to predict an outcome variable from a very large number of independent variables, we will actually expect a small number of features to appear to correlate with the outcome simply by chance. There are methods such as Bonferroni correction or the Benjamini-Hochberg procedure for mitigating this problem, but we are often left with a lack of conviction about our answers. Also, in many application areas (genomics is certainly one) we find that a particular outcome really corresponds to a large group of features acting in concert rather than to a single feature. It is then important to be able to identify such meaningful groups of features. We can then ask for methods that work easily and intuitively to construct and manipulate meaningful groups...