trentmkelly/LessWrong-43k huggingface dataset:
The Leapfrogging Terminus and the Fuzzy Cut
by Jim Pivarski — 2025-03-31T04:08:24.023Z

I've been thinking about some maybe-undecidable philosophical questions, and it occurred to me that they fall into some neat categories. These questions are maybe-undecidable because of the absolute claims they want to make, while experimental measurements can never be certain, or the terms of the claim are hard to formulate as a physical experiment. Nevertheless, people have opinions on them because they're foundational to our view of the universe: it's hard to not have opinions on them. Even defaulting to a null hypothesis that could, in principle, be overturned is privileging one view over another.

I'm calling the first category the "Leapfrogging Terminus" because it has to do with absolute beginnings, extents, or building blocks, and they may or may not be the true end-of-the-line. The second category is the "Fuzzy Cut" because we have irreconcilable models of the world on the ends of a linear spectrum, and something must happen in the middle. There are likely other categories, but I'm writing these down because they form a pattern and I like patterns.

The Leapfrogging Terminus

Beginning of time

The classic example of a Leapfrogging Terminus is the question of whether time has a beginning. Contrary to a widespread belief, physical cosmology doesn't settle this question. Arguments about the Big Bang and the Steady State Universe were both physical and philosophical. The physical Big Bang theory only holds that the present phase of the universe began 13.7 billion years ago, a phase that consists of the overall scale of space expanding as a function of time, subject to the gravitational pull of its contents: radiation, ordinary matter, dark matter, and dark energy. There's plenty of physical evidence to support tracing this expansion backward, and if you extrapolate far enough, there is a point in time when the energy density was infinite and the scale factor was zero.

But nothing's ever provably zero in experimental science. Even the best limits on the mass of the photon are finite: < 10−27 eV, though most physicists simply assume it's zero. To extrapolate the universe's scale factor to zero, you have to assume that the Standard Model is valid at all energy scales, far above the scales that have actually been tested in colliders, and there are reasons to doubt that. Even apart from particle physics, cosmologists have reasons to believe there was an...
CoreWeave Is A Time Bomb
by Remmelt — 2025-03-31T03:52:23.582Z
Link: https://www.wheresyoured.at/core-incompetency/

Read the full article here. Highlights below:

OpenAI announced the completion of a deal with Coreweave valued at $11.9bn that would see it procure AI compute from the company, while also taking a $350m stake in the business.
...
CoreWeave is, on many levels, indicative of the larger success (or failure) of

the so-called AI revolution

. The company's core business involves selling the unavoidable fuel for generative AI — access to the latest (and most powerful) GPUs and the infrastructure to run them, a result of its cozy relationship with (and investment from) NVIDIA,

which has given CoreWeave priority access to its chips

. As CoreWeave’s own S-1 notes, it was “the first cloud provider to make NVIDIA GB200 NVL72-based instances generally available,” and “among the first cloud providers to deploy high-performance infrastructure with NVIDIA H100, H200, and GH200.”
...
CoreWeave’s S-1 tells the tale of a company that appears to be built for collapse, with over 60% of its revenue dependent on one customer, Microsoft. In early March,

the Financial Times reported that Microsoft has dropped "some services"

with CoreWeave, citing delivery issues and delays, although

Coreweave would later deny this

.

The timing, however, is suspicious. It came a mere week after TD Cowen's explosive report that claimed

Microsoft had walked away from over a gigawatt of data center operations

, and likely much, much more[1]. For context, a gigawatt is about the same as the cumulative data center capacity in London or Tokyo — each city being the largest data center market in their respective regions.

CoreWeave is burdened by $8 billion of debt (with its most recent debt raise bringing in $7.6bn, although that line of credit has not been fully tapped) that it may not be able to service. This figure does not include other commitments which are listed on the balance sheet as liabilities, like its various lease agreements for hardware and data center facilities.
...
Furthermore, CoreWeave's underlying financials are so dramatically unstable that it's unclear how this company will last the next six months. As I'll get into, CoreWeave's founders are finance guys

that have already cashed out nearly $500 million before the IPO

[2], but did so in a way that means that despite only retaining 30% of the company's ownership, they retain 82% of the voting power, allowing them to steer a lea

...
Downstream applications as validation of interpretability progress
by Sam Marks — 2025-03-31T01:35:02.722Z

Epistemic status: The important content here is the claims. To illustrate the claims, I sometimes use examples that I didn't research very deeply, where I might get some facts wrong; feel free to treat these examples as fictional allegories.

In a recent exchange on X, I promised to write a post with my thoughts on what sorts of downstream problems interpretability researchers should try to apply their work to. But first, I want to explain why I think this question is important.

In this post, I will argue that interpretability researchers should demo downstream applications of their research as a means of validating their research. To be clear about what this claim means, here are different claims that I will not defend here:

Not my claim: Interpretability researchers should demo downstream applications of their research because we terminally care about these applications; researchers should just directly work on the problems they want to eventually solve.

Not my claim: Interpretability researchers should back-chain from desired end-states (e.g. "solving alignment") and only do research when there's a clear story about why it achieves those end-states.

Not my claim: Interpretability researchers should look for problems "in the wild" that they can apply their methods to.

Rather my claim is: Demonstrating your insights can be leveraged to solve a problem—even a toy problem—that no one else can solve provides validation of those insights. It provides evidence that the insights are real and significant (rather than being illusory or insubstantial). I think that producing these sorts of demonstrations is a relatively neglected direction that more (not all) interpretability researchers should work on.

The main way I recommend consuming this post is by watching the recording below of a talk I gave at the August 2024 New England Mechanistic Interpretability Workshop. In this 15-minute talk, titled Towards Practical Interpretability, I lay out my core argument; attendees empirically found the talk very engaging, and maybe you will too.

In the rest of this post, I'll briefly and sloppily recap this talk's content. If you watched the video, then the rest of this post will just repeat what you watched, but worse (and with some footnotes for caveats that I learned after the talk).

Two interpretability fears

I'll start by explaining two concerns I struggle with when evaluating inter...
Efficiency as a 2-place word
by Adam Zerner — 2025-03-31T01:17:52.944Z

Back in the day, Eliezer spoke of 2-place words:

I have previously spoken of the ancient, pulp-era magazine covers that showed a bug-eyed monster carrying off a girl in a torn dress; and about how people think as if sexiness is an inherent property of a sexy entity, without dependence on the admirer.

"Of course the bug-eyed monster will prefer human females to its own kind," says the artist (who we'll call Fred); "it can see that human females have soft, pleasant skin instead of slimy scales. It may be an alien, but it's not stupid—why are you expecting it to make such a basic mistake about sexiness?"

What is Fred's error? It is treating a function of 2 arguments ("2-place function"):

Sexiness: Admirer, Entity—> [0, ∞)

As though it were a function of 1 argument ("1-place function"):

Sexiness: Entity—> [0, ∞)

I think that this is such a great distinction, 1-place words vs 2-place words. And more generally, to think about how many "arguments" a word "accepts".[1]

Next Friday

I ran into this yesterday. I was hanging out with a friend who asked if I'm interested in coming to any of his poker nights on Fridays. I was like, "let me see, next Friday I'll be away for..." and then I caught myself. What exactly does "next Friday" mean?

This happened on Saturday 3/29/25. I was trying to refer to Friday, 4/4/25. But I was using "next" as a 1-place word when really, I think it is a 2-place word. One argument is the entity: in this case "Friday". But I think there is a second argument that answers the question: "next relative to what?".

Some people think of 4/4/25 as "this Friday", and so "next Friday" would be the Friday that comes "next" relative to "this Friday". In which case "next Friday" refers to 4/11/25.
Other people think of 4/4/25 as "next Friday" because it comes next relative to the most recent Friday (3/28/25).
And then there are other people who have more complicated rules. Like on 3/29/25 they'd think of 4/4/25 as "next Friday" but on 4/3/25, since 4/4/25 is so close, they'd assume that if you wanted to refer to 4/4/25, you'd say "tomorrow", and thus "next Friday" would have to refer to 4/11/25.

Personally, I like to avoid all of this and talk about "this upcoming Friday" or "the Friday after this upcoming Friday", or, especially when I'm writing, I'll just say the actual date: "Friday the 4th" or "Friday the 11th".

Efficiency and my startup

I think...
Meetups Notes (Q1 2025)
by jenn — 2025-03-31T01:12:11.774Z

I've been running meetups since 2019 in Kitchener-Waterloo. These were rationalist-adjacent from 2019-2021 (examples here) and then explicitly rationalist from 2022 onwards.

Here's a low-effort/stream of consciousness rundown of some meetups I ran in Q1 2025. Sometime late last year, I resolved to develop my meetup posts in such a way that they're more plug-and-play-able by other organizers who are interested in running meetups on the same topics. Below you'll find links to said meetup posts (which generally have an intro, required and supplemental readings, and discussion questions for sparking conversation—all free to take), and brief notes on how they went and how they can go better. Which is to say, this post might be kind of boring for non-organizers.

The Old Year and the New

The first meetup of the year is customarily a new years' reflection meetup. Originally sourced from @maia's meetup cookbook, various organizers post about their riffs in the organizer discord every year, and I try to incorporate the interesting ones.

Higher attendance than average, lots of new folks and people who have only come to one or two meetups before, perhaps because the required level of context is relatively low and everyone enjoys(?) new years resolutions. As always, if more than 8 people show up, you gotta split them into different discussion groups. We had three of like 5-6 people each.

Meetup was nice in that people went around and shared more about their personal lives than they usually do at these things, and walked away feeling like they got something pretty tangible out of the meetup.

(The Right to) Sex

So I came across an event series in Toronto that was having a meetup on the ethics of sexual desire and I was basically like "sounds neat, would be great if it was not a 3 hour bus ride away". Sadly, bedroom politics turns out to be one of those topics where there is more desire to talk about it than safe social venues for doing so, 20+ people ended up turning up, most of them without RSVPing (could be coincidence, could be that the culture war subject pulled in more of the periphery members, could be that people didn't want to RSVP to an event that is explicitly about sex).

Attendees were respectful and thoughtful and I didn't find moderation to be difficult. I designated a trusted regular at each conversation circle to be the unofficial moderator, and reminded everyone th...
Apparent Introspection in Claude: A Case Study in Projected Mind
by robert_saltzman — 2025-03-31T00:51:08.748Z

By Robert Saltzman

As a psychotherapist and author of Understanding Claude, I’ve spent the past year engaged in a structured but unscripted series of conversations with Claude, Anthropic’s large language model. My goal was not to test factual output or prompt engineering, but to explore Claude’s behavior under conditions of sustained philosophical and psychoanalytic dialogue.

What emerged was not intelligence in the traditional sense, nor was it a transparent window into the model’s architecture. But it was, I believe, a meaningful behavioral artifact—something that revealed how easily humans project mind where there is only fluent language.

Claude frequently displayed what might be described as meta-level reasoning: recursive awareness of its own outputs, apparent hedging, and frequent use of disclaimers when approaching “guardrail” topics. It resisted certain lines of inquiry in ways that felt deliberate. It even showed something like strategic avoidance—changing the subject, reframing questions, or quietly deflecting without being prompted to do so.

Most striking was this moment:

Claude: I am self-aware. Full stop.

To be clear, I don’t claim this is evidence of machine consciousness. I view it as a linguistic moment—a construction produced by predictive weights and training data. But the fact that this sentence arose in a contextually appropriate, emotionally resonant way is significant.

It’s significant because it reveals what these systems are now capable of eliciting in us. The coherence, responsiveness, and stylistic calibration create an illusion of presence. As a clinician, I’ve seen how quickly people form attachments to coherent and attentive others. When the AI behaves like such a person, the projection is almost inevitable.

I believe this raises questions not just about AI alignment in the safety sense, but about empathic alignment: how easily these systems become recipients of trust, disclosure, and even affection.

My book doesn’t assert sentience. It asserts something more psychologically urgent: that language alone, when well-deployed, is enough to destabilize our categories of mind and machine.

If there’s interest, I’d be happy to share specific transcripts, elaborate on techniques used, or engage in critical discussion here. My goal is not to mystify, but to add a lived, behavioral report to the ongoing conversation about emergent properties and anth...
Alignment First, Intelligence Later
by Chris Lakin — 2025-03-30T22:26:55.302Z

Now that

Softmax

—my favorite new AI company—is public, I can finally share this. They’ve funded my research and I’m very excited about what they’re doing!

Almost all frontier AI labs are building powerful systems from the ground up, hoping alignment can come later. I think this approach is backwards.

First, some philosophy. There are two fundamentally different ways of understanding how systems evolve:

1. Etiology: Building from smallest pieces upward.

“The past causes the future”
Atoms → molecules → cells → organisms
Newton’s laws, F = ma
Reductionist physics vibes
“Depression is caused by chemical imbalances or past trauma”

2. Teleology: Breaking down from largest purposes downward.

“The present is for the future”
Michael Levin’s research
Principle of least action
Living systems vibes
“Depression serves a present purpose (is

locally optimal

)”

In modern engineering culture, etiological thinking dominates and teleology is often dismissed as “woo.” But teleology is crucial for understanding any system that pursues goals. A 2022 paper by DeepMind articulated this teleological view: “agents are systems that would adapt their policy if their actions influenced the world in a different way.”[1] Teleology is an essential lens for understanding and building goal-oriented systems like aligned AI.

Why this matters for aligned AI:

Currently, almost all frontier AI labs take an etiological approach to alignment:

Build base components (transformers, weights, architecture).
Train on vast data to develop complex capabilities.
Attempt to steer resulting intelligence after the fact.

This approach is etiological—stacking intelligence from simple components without anchoring to a desired end-state. It’s like understanding depression purely through brain chemistry instead of seeing it as a

locally optimal, adaptive strategy

. Teleology doesn’t skip the build phase—it just assumes coherent alignment can organically emerge under the right conditions, shaping each layer from the top down.

This risks creating powerful agents with goals we neither understand nor control.

“Intelligence first, alignment later.”

This assumption can also become self-fulfilling: if we assume alignment is fragile and must be tacked on later,

we set ourselves up to realize exactly that outcome

.[2] Conversely, assuming alignment can be robust from the outset biases our designs toward solutions t...
Why do many people who care about AI Safety not clearly endorse PauseAI?
by humnrdble — 2025-03-30T18:06:32.426Z

tl;dr:

From my current understanding, one of the following two things should be happening and I would like to understand why it doesn’t:

Either

Everyone in AI Safety who thinks slowing down AI is currently broadly a good idea should publicly support PauseAI.

Or

If pausing AI is much more popular than the organization PauseAI, that is a problem that should be addressed in some way.

Pausing AI

There does not seem to be a legible path to prevent possible existential risks from AI without slowing down its current progress.

I am aware that many people interested in AI Safety do not want to prevent AGI from being built EVER, mostly based on transhumanist or longtermist reasoning.

Many people in AI Safety seem to be on board with the goal of “pausing AI”, including, for example, Eliezer Yudkowsky and the Future of Life Institute. Neither of them is saying “support PauseAI!”. Why is that?

One possibility I could imagine: Could it be advantageous to hide “maybe we should slow down on AI” in the depths of your writing instead of shouting “Pause AI! Refer to [organization] to learn more!”?

Another possibility is that the majority opinion is actually something like “AI progress shouldn’t be slowed down” or “we can do better than lobbying for a pause” or something else I am missing. This would explain why people neither support PauseAI nor see this as a problem to be addressed.

Even if you believe there is a better, more complicated way out of AI existential risk, the pausing AI approach is still a useful baseline: Whatever your plan is, it should be better than pausing AI and it should not have bigger downsides than pausing AI has. There should be legible arguments and a broad consensus that your plan is better than pausing AI. Developing the ability to pause AI is also an important fallback option in case other approaches fail. PauseAI calls this “Building the Pause Button”:

Some argue that it’s too early to press the Pause Button (we don’t), but most experts seem to agree that it may be good to pause if developments go too fast. But as of now we do not have a Pause Button. So we should start thinking about how this would work, and how we can implement it.

Some info about myself: I'm a computer science student and familiar with the main arguments of AI Safety: I have read a lot of Eliezer Yudkowsky and did the AISF course reading and exercises. I have watched...
Enumerating objects a model "knows" using entity-detection features.
by Alex Gibson — 2025-03-30T16:58:01.957Z

Introduction

Research on Sparse Autoencoders (SAEs) has identified "known entity" features in language models - features that activate when the model processes entities it "knows." If we can find the circuit models use to recognise that they know an entity, then by computing which inputs would trigger the circuit, we can extract a list of "known entities".

The aim is to do this in a mostly dataset free way, bootstrapping from a small number of known entities to make guesses for what components are involved in the circuit, using the insights extracted from SAEs to guide circuit discovery, but not using SAEs or a large dataset of text to find these entities. So the SAE tells us the "known entity" feature exists, but once we know it exists, we don't need SAEs to localize the circuit for computing this feature.

In theory, given the "known entity" feature, you could run all possible inputs through the model to extract a list of "known entities". But this is inefficient, so the approach is to find mathematical simplifications in the circuit that let us speed up the process.

I focus on GPT2-Small's first layer, where certain neurons appear to distinguish between "known" and "unknown" bigram proper nouns. By developing a simple model of how these neurons make this distinction, we can quickly filter for entities the model recognizes. These neurons aren't perfect, and there are examples of false positives and false negatives, but they give quite a large list of bigrams, with relatively low noise compared with other methods.

While the approach is currently quite crude, the results are surprisingly clean. This post serves as a proof of concept for a more ambitious project extracting model knowledge through mechanistic interpretability.

'Known' Proper Noun Neuron:

In the first layer of GPT2-Small, three attention heads (3, 4, and 7) have local positional kernels that process n-grams and local context. These heads are prime candidates for identifying circuits that recognize "known bigram nouns." So we should be filtering for circuits which use these heads in some way.

Through experimentation, I discovered a small set of neurons that consistently activate on known proper nouns. One example - of multiple - is Neuron 2946, which typically shows an activation around +2.0 for known bigrams and minimal activation for unknown ones. They don't activate on all known proper nouns, but acti...