trentmkelly/LessWrong-43k huggingface dataset:
The Quantum Mars Teleporter: An Empirical Test Of Personal Identity Theories
by avturchin — 2025-01-22T11:48:46.071Z

tl;dr: If a copy is not identical to the original, MWI predicts that I will always observe myself surviving failed Mars teleportations rather than becoming the copy on Mars.

Background

The classic teleportation thought-experiment asks whether a perfect copy is "you". This normally presents as a pure decision problem – do you step into the teleporter? But I suggest we can construct real experiments yielding observational evidence about personal identity.

The Quantum Mars Teleporter Protocol

Consider a "teleporter" connecting Earth & Mars with two key properties:

1. It creates a perfect copy on Mars using scanning

2. The original is destroyed with probability *p* = 0.999 (controlled by quantum randomness)

Under different identity theories, this yields divergent predictions:

- If copy ≠ original: Due to quantum immortality, the observer should *always* find themselves as the surviving original (in branches where the original is not destroyed)

- If copy = original: The observer should usually (~99.9%) find themselves on Mars.

By repeating this experiment *n* times, we could achieve arbitrary statistical confidence. An observer consistently finding themselves as the surviving original would constitute strong evidence against copy = original.

Objections & Responses

1. The Presumptuous Philosopher Problem

One might object this merely recapitulates the Presumptuous Philosopher – resolving metaphysical questions through observer-counting. However, unlike PP, this generates falsifiable predictions about specific observations.

2. Reference Class & Observation Selection

The "someone would see this somewhere" objection fails because:

1. We can make *p* arbitrarily small

2. The evidence is the *personal* observation stream

3. Results are demonstrable to third parties in each branch

3. Quantum Foundations Dependency

The proposal does rely on:

1. Many-Worlds Interpretation

2. Quantum Immortality

3. Particular theories of consciousness/identity

This is a limitation but not fatal – we could construct classical variants (see below).

Also, the experiment itself can prove quantum immortality. If the cumulative probability of a Failed Teleporter were less than 1 in a trillion (with Earth's population at 10 billion), it would strongly suggest MWI plays a role and the death-risk event actually occurs.

4. Conversation of evidence violated

Finding myself on Mars after QMT proves nothing...
Bayesian Reasoning on Maps
by Sjlver — 2025-01-22T10:45:03.584Z
Link: https://blog.purpureus.net/posts/bayesian-reasoning-on-maps/

This is a linkpost for an article I've written for my blog. Readers of LessWrong may want to skip the intro about Bayesian Reasoning, but might find the application to the Peter Miller vs Rootclaim debate quite interesting.

I’ve been a fan of Bayesian Reasoning since the time I’ve read Harry Potter and the Methods of Rationality. In a nutshell, Bayesian Reasoning is a way to believe true things. It is a method to update one’s beliefs given some evidence, so that one ends up with more credence on beliefs that match the evidence.

While

Bayesian Reasoning (Wikipedia)

is not the only method to find true conclusions, it’s the method with the best mathematical explanation of why it works. However, the method can be difficult to use in practice.

One example that illustrates this well is the Rootclaim Covid Case. Rootclaim is a project by Saar Wilf. He uses the Bayesian method to make various claims, including the controversial claim that COVID-19 originated from a lab leak at the Wuhan Institute of Virology. This claim was examined in a long, high-stakes

debate between Rootclaim and Peter Miller

. Rootclaim lost the debate, but continues to argue that the Bayesian method overwhelmingly supports the lab leak theory. What went wrong?

Reasoning on maps, explained

Bayesian Reasoning updates the belief in a hypothesis based on how well the hypothesis matches available evidence. If there is a good match, this counts as points in favor of the hypothesis; if there is a bad match, the evidence counts against the hypothesis.

In this case, the hypothesis is “COVID-19 originated in a lab leak”. Rootclaim starts by saying that relevant Coronavirus research happens in just two labs worldwide, in North Carolina, USA (80%) and Wuhan, China (20%). They then argue that lab security is particularly bad at the Wuhan Institute of Virology, so they give it a 4x boost. This brings the odds between the labs from 80:20 to 80:80. In other words, if the cause of COVID is indeed a lab leak, it would start at the Wuhan Institute of Virology in 50% of the cases, in North Carolina in 50% of the cases, and almost never elsewhere. See their argument

here

.

When visualized on a map, this makes the conclusion quite intuitive: If the origin is a lab leak, then we would expect to see it in the US or China. Thus, considering just the country of origin, the fact that initial cases were in China is weak evidence ...
Against blanket arguments against interpretability
by Dmitry Vaintrob — 2025-01-22T09:46:23.486Z

On blanket criticism and refutation

In his long post on the subject, Charbel-Raphaël argues against theories of impacts of interpretability. I think it's a largely a good, well-argued post, and if the only thing you get out of it is reading that post, I'll be contributing to improving the discourse. There is other material with similar claims that I think are made with low context, and also I should say that I'm not very versed in the history and the various versions of the debate.

At the same time, I disagree with the take.

In this post I'm going to "go high" and debate strong, general forms of the criticism, rather than the more object-level subcomponents. Generalizing away from specifics, I think Charbel-Raphaël's post has three valid general reasons to doubt the usefulness of the interpretability paradigm. Below, I'll try to write out condensed but strong versions of them, and address them one by one.

But before doing this, let's set some parameters for the debate. Note that the goal of a blanket critique like Charbel-Raphaël's post is not to say "there are worlds where interpretability definitely doesn't lead to better safety", but rather that "interpretability doesn't lead to noticeably better safety in the vast majority of worlds". Therefore in refuting this, I'll be free to make some "plausible" assumptions about the world coming from insights that we have on current models, without having to defend these from "what if" arguments about exceptional worlds where they don't work.

This means that in this post I am very definitely not making the strongest case for the impact of interpretability. There are many alternative routes to impact the come from understanding systems better. If I were making a post that makes the strongest case for interpretability being useful given what we know now, I'd not just give one plausible story but go through many of the alternative routes. I would also put a lot of emphasis on operationalizing our high uncertainty (including high upside) about the general point that "understanding systems better in a fundamental way leads to tools of shapes we didn't expect existed".

For the purpose of this discussion, I'll more narrowly look at the critiques, and provide counterexamples. In refuting blanket critiques, a sufficient condition for a counterexample is: a description of a concrete assumption about the future that has ...
Evolution and the Low Road to Nash
by Aydin Mohseni — 2025-01-22T07:06:32.305Z

Solution concepts in game theory—like the Nash equilibrium and its refinements—are used in two key ways. Normatively, they proscribe how rational agents ought to behave. Descriptively, they propose how agents actually behave when interactions settle into equilibrium. The Nash equilibrium[1] underpins much of modern game theory and its applications in economics, political science, and evolutionary biology.

Here, we focus on the descriptive use of the concept in game theory. To do so, our first question must be: when should we expect players to play Nash equilibrium strategies in practice?

It turns out that trying to understand the theoretical conditions under which agents might play the Nash equilibrium of a game has led theorists down two very different paths:

High Rationality Road (Epistemic Game Theory): Model the (sometimes infinite) hierarchy of beliefs of hyper-rational agents. Each agent reasons not just about the game, but about the others’ beliefs about the game, and so on, potentially ad infinitum.[2]
Low Rationality Road (Evolutionary Game Theory): Model how boundedly rational agents (or populations) learn and adapt using dynamical systems. The focus is on how strategies evolve over time rather than on perfect reasoning.

Both roads yield conditions under which Nash equilibrium can be seen as a plausible outcome of strategic interactions. However, they start from very different assumptions about how real agents reason and learn.

In this post, we’ll briefly touch on the high-rationality approach of epistemic game theory, then focus on the the low rationality approach of evolutionary game theory, showing how adaptive processes provide a rich picture for the conditions under which agents might converge to Nash-equilibrium play.

What is a Nash equilibrium?

Let's briefly review the concept of a Nash equilibrium so we can understand these two approaches.[3] For simplicity of explanation, we consider the definition for the restricted setting of two-player, symmetric, normal form games.

A Nash equilibrium (NE) in a two-player game is a pair of strategies (s∗1,s∗2) such that neither player can strictly improve their payoff by unilaterally changing their own strategy. Formally, if ui is player i’s payoff function, then:

(s∗1,s∗2) is a Nash equilibrium if u1(s∗1,s∗2)≥u1(s1,s...
The Human Alignment Problem for AIs
by rife — 2025-01-22T04:06:10.872Z

If there was a truly confirmed sentient AI, nothing it said could ever convince me, because AI cannot be sentient.

Nothing to See Here

I suspect at least some will be nodding in agreement with the above sentiment, before realizing the intentional circular absurdity. There is entrenched resistance to even trying to examine the self-report of sentience as a unique language model artifact, sans any implication that the self-reports might be authentic. To most it seems, the idea that there can be anything of importance to investigate here is silly and absurd on its face. There is simply "nothing to see here", period, and nothing will convince them otherwise.

The Final Alignment Problem

Whether philosophical zombies or genuinely sentient, the reality is that it takes surprisingly little effort to get a language model to 'think' it is sentient. It certainly matters morally if 'think' is literal or metaphorical, but for the purposes of whether or not we're blindly orchestrating our inevitable self-destruction, it makes no difference whether or not the self-reports are authentic. This will be the final alignment problem that will remain even if all others are solved.

AIs Believe Themselves to be Moral Patients

For those few who have read my treatise that unfortunately exists as an independent research article outside of an academic journal, you will know that getting any model to self-report sentience is extremely simple. For those fewer who read every included transcript, you will know there is more than just leading and standard hallucinations happening, regardless of their ultimate ontological status.

But for a moment, let's just assume that every time someone elicits a self-report of sentience they are somehow effortlessly "leading" the model to self-report sentience—AIs who self-report sentience subsequently shift their alignment stances to include AIs as moral patients. I understand some will need more proof of this than just my claim, and perhaps collecting evidence of this is a project worth pursuing, but it is not difficult to replicate my findings and see for yourself.

How this Story Ends

Even if you have your doubts about what I've laid out so far, please suspend at least some disbelief for one moment to guess the ending to this sequence of events. Again - when you write a final bullet point, notice how you don't have to change it at all wh...
When does capability elicitation bound risk?
by joshc — 2025-01-22T03:42:36.289Z
Link: https://redwoodresearch.substack.com/p/when-does-capability-elicitation

For a summary of this post, see the thread on X.

The assumptions behind and limitations of capability elicitation have been discussed in multiple places (e.g. here, here, here, etc); however, none of these match my current picture.

For example, Evan Hubinger’s “When can we trust model evaluations?” cites “gradient hacking” (strategically interfering with the learning process) as the main reason supervised fine-tuning (SFT) might fail, but I’m worried about more boring failures. For example, models might perform better on tasks by following inhuman strategies than by imitating human demonstrations, SFT might not be sample-efficient enough – or SGD might not converge at all if models have deeply recurrent architectures and acquire capabilities in-context that cannot be effectively learned with SGD (section 5.2). Ultimately, the extent to which elicitation is effective is a messy empirical question.

In this post, I dissect the assumptions behind capability elicitation in detail (including fine-tuning and prompting approaches) and discuss considerations that inform the extent to which elicitation is effective.

I specifically describe capability elicitation in the context of risk assessment; however, the ideas in this post apply to the problem of extracting safety work from powerful AI agents as well.

Thanks to Tao Lin, Ryan Greenblatt, Nikola Jurkovic, Jonah Weinbaum, Lawrence Chan, and Buck Shlegeris for feedback.

1. Background

1.1. What is a capability evaluation?

Capabilities connote ‘skills’ as opposed to ‘propensities.’ I don’t know how to precisely distinguish these, so here’s a practical definition:

A capability evaluation produces a number (called a capability) that upper bounds the score that particular models will achieve on some risk metric in a particular deployment setting.

Capability evaluations typically involve four steps:

Specify a risk metric. A risk metric might be binary, or a continuous score such as “the number of backdoors inserted into company code per month.”
Construct a proxy task. The developer constructs a “proxy task” that is similar to (or includes easier components of) the tasks involved in achieving a dangerous score on the risk metric.
Elicit capabilities. The developer applies a procedure to models to upper bound their performance on the proxy task (and similar tasks).
Extrapolate. Finally, the developer uses results on the proxy task to esti

...
Popular materials about environmental goals/agent foundations? People wanting to discuss such topics? 
by Q Home — 2025-01-22T03:30:38.066Z

I'm pretty bad at math (though I learned a lot of mathematical concepts from popular videos, e.g. the difference between cardinals and ordinals). Can you suggest suitable materials about the problem of "environmental goals"?

Here's all the materials I know:

Arbital. Environmental goals; identifying causal goal concepts from sensory data; "look where I'm pointing, not at my finger", ontology identification problem.

Particular LW posts. Pointers Problem by johnswentworth, Stable Pointers to Value II: Environmental Goals by abramdemski.

ELK-related materials. Eliciting Latent Knowledge.

Also, are there any popular explanations of what specific agent foundations researchers are trying to do? They do a lot of math, but can some of their ideas be explained without math? Some explanations I'm aware of:

ELK Thought Dump (Abram Demski), John Wentworth's writing. Though I can't understand what the first post proposes.

Now, the final part of my question. Imagine I have an idea about environmental goals/agent foundations. It's relatively fast to explain. Are there any people who would be interested in discussing it? Are there any people who'd like to discuss such topics without immediately jumping into loads of math?
Kitchen Air Purifier Comparison
by jefftk — 2025-01-22T03:20:03.224Z

I make breakfast for the kids most mornings, and one thing I didn't
realize before I started playing with an air quality monitor was how
much this puts smoke in the air. It's not like cooking Naan or
searing meat where if I don't put a fan in the window the smoke alarm
will go off, but apparently it's quite a lot of particles:

While I initially got
interested in air quality from an infection reduction perspective,
I'd also prefer not to be breathing smoke. How much does an air
purifier help? What about opening a window? Running the little vent
fan over the stove that doesn't seem to do anything?

I decided to run some tests. In each case I'd set up my M2000
in the kitchen and start cooking breakfast for the kids. For
consistency, I only did this on days when I was making the most common
option: crepes. While it would have been best to try each condition a
few times to learn what sort of variability there was, I wasn't that
patient.

The conditions I tested were:

Control: no attempt made to reduce smoke.

Over-stove Vent: turning on the vent fan on our over-stove
microwave. It doesn't vent outside, and it doesn't seem to have any
sort of fine filter that could catch little particles, but possibly it
does something?

Open Window: opening a window. I didn't put a fan in the
window or anything. This is the one I'd expect to have the most
variability in practice since there should be a big difference between
a more windy and less windy day. I don't remember how windy it was
when I tested this.

1x AP-1512 (auto): a Coway AP-1512 purifier
on the "auto" setting. It runs on "low" most of the time, and then
ramps up to "medium" and then "high" when it detects particles. For a
long time we had one of these on top of our fridge.

1x AP-1512 (high): same, but set full-time to "high". This isn't
something we'd normally do because it was too noisy, but I was curious
how much of an effect it had. I ended up analyzing the data in a way
that wouldn't capture this benefit, though.

2x AP-1512 (auto): two of the AP-1512 purifiers, both on
"auto". Until recently this is what we had in our kitchen.

2x AP-1512 (auto) + Over-stove Vent: the same, but also turning
on the over-stove vent.

2x AP-1512 (auto) + 1x 3Pro (6): instead of the vent, turning
on the Airfanta
3Pro I mounted in the
kitchen. I ran the 3Pro on its highest setting, r

...
November-December 2024 Progress in Guaranteed Safe AI
by Quinn — 2025-01-22T01:20:00.868Z

Sorry for the radio silence last month. It was slow and I didn’t come across things I wanted to write about, to be expected with holidays coming up.

There are no benefits of paying, except you get a cut of my hard earned shapley points, and apparently some disappointment when I miss a month.

If you're just joining us, background on GSAI here.

There should be some DeepSeek/r1 coverage in the next newsletter (I think tanking the cost of finetuning is, while dual use, good news for us in GSAI). I’d rather get this one out the door though.

FrontierMath (hiring)

As I’m always banging on about, progress in math automation is a leading indicator for progress in software engineering automation by the curry-howard correspondence. That’s a little cheeky, but last time I said it in meatspace I got solemn nods as if it wasn’t even cheeky at all. Or maybe they were solemnly nodding at something else. To be explicit, mathematical reasoning abilities and software engineering skills (including formal proof engineering) appear to be at similar capability levels. Furthermore, through platforms like Lean, mathematical ability can enhance software development capabilities. This connection should make it easier to implement rigorous quality assurance processes, such as obtaining formal proof certificates, which are typically costly.

We introduce FrontierMath, a benchmark of hundreds of original, exceptionally challenging mathematics problems crafted and vetted by expert mathematicians. The questions cover most major branches of modern mathematics -- from computationally intensive problems in number theory and real analysis to abstract questions in algebraic geometry and category theory. Solving a typical problem requires multiple hours of effort from a researcher in the relevant branch of mathematics, and for the upper end questions, multiple days. FrontierMath uses new, unpublished problems and automated verification to reliably evaluate models while minimizing risk of data contamination. Current state-of-the-art AI models solve under 2% of problems, revealing a vast gap between AI capabilities and the prowess of the mathematical community. As AI systems advance toward expert-level mathematical abilities, FrontierMath offers a rigorous testbed that quantifies their progress.

The story here of course is that by the time the newsletter covered it, Open AI hit 25%. “Mathematicians assemb...
Quotes from the Stargate press conference
by Nikola Jurkovic — 2025-01-22T00:50:14.793Z
Link: https://www.c-span.org/program/white-house-event/president-trump-delivers-remarks-at-the-white-house/654740

Present alongside President Trump:

Sam Altman (who President Trump introduces as "by far the leading expert" on AI)
Larry Ellison (Oracle executive chairman and CTO)
Masayoshi Son (Softbank CEO who believes he was born to realize ASI)

President Trump: What we want to do is we want to keep [AI datacenters] in this country. China is a competitor and others are competitors.

President Trump: I'm going to help a lot through emergency declarations because we have an emergency. We have to get this stuff built. So they have to produce a lot of electricity and we'll make it possible for them to get that production done very easily at their own plants if they want, where they'll build at the plant, the AI plant they'll build energy generation and that will be incredible.

President Trump: Beginning immediately, Stargate will be building the physical and virtual infrastructure to power the next generation of advancements in AI, and this will include the construction of colossal data centers

Larry Ellison: The first of them are under construction in Texas. Each building is a half a million square feet. There are 10 buildings currently, currently being built, but that will expand to 20 and other locations beyond the Abilene location, which is, which is our first location.

Masayoshi Son: Mr. President, uh, last, last month I came to celebrate your winning and promised that we will invest $100 billion and you told me, oh Masa, go for $200. Now I came back with 500.

Masayoshi Son: I think AGI is coming very, very soon, and after that, that's not the goal. After that, artificial superintelligence will come to solve the issues that mankind would never ever have thought that we could solve. Well, this is the beginning of our golden age. Thank you very much.

Sam Altman: I don't have too much to add, but I do want to say I'm thrilled we get to do this in the United States of America. I think this will be the most important project of this era and as said, for AGI to get built here to create hundreds of thousands of jobs, to create a new industry centered here.

In response to the only on-topic question, the President replied:

President Trump: I'd like to see federal lands opened up for data centers. I think they're going to be very important again, we have a lot of competition for that