trentmkelly/LessWrong-43k huggingface dataset:
Existing UDTs test the limits of Bayesianism (and consistency)
by Cole Wyeth — 2025-03-12T04:09:11.615Z

Epistemic status: Using UDT as a case study for the tools developed in my meta-theory of rationality sequence so far, which means all previous posts are prerequisites. This post is the result of conversations with many people at the CMU agent foundations conference, including particularly Daniel A. Herrmann, Ayden Mohensi, Scott Garrabrant, and Abram Demski. I am a bit of an outsider to the development of UDT and logical induction, though I've worked on pretty closely related things.

I'd like to discuss the limits of consistency as an optimality standard for rational agents. A lot of fascinating discourse and useful techniques have been built around it, but I think that it can be in tension with learning at the extremes. Updateless decision theory (UDT) is one of those extremes; but in order to think about it properly, we need to start with its Bayesian roots. Because, appropriately enough for a sequence on the meta-theory of rationality, I want to psychoanalyze the invention/inventors of UDT. Hopefully, we'll then be in a position to ask what we think we know and how we think we know it in regards to updatelessness (also sometimes called priorism), the driving idea behind UDT.

Subjective Bayesianism is about consistency[1] among beliefs. The Cox axioms force real-valued credences to act to like probabilities under some natural conditions that ultimately boil down to consistency; one way to intuitively compress the assumptions is that beliefs about related things have to continuously "pull on each other," so I think of the Cox axioms as requiring credence to propagate through an ontology properly. Dutch book arguments further require that betting behavior is consistent with probabilistic structure to avoid being "money-pumped" or accepting a series of bets that is sure to lose money (a kind of dominance principle). That handles the "statics." Bayesian updating is of course a theorem of probability theory, forced from Kolmogorov's axioms, so in that sense it is a consequence of the preceding arguments. But insofar as we want to it describe belief dynamics, updating enforces a kind of consistency (with respect to old beliefs and new information) across time. Similar arguments motivate maximizing some expected utility with respect to these credences = subjective (prior/posterior) probabilities - but I actually won't be very concerned with utilities here.&nbs...
(Anti)Aging 101
by George3d6 — 2025-03-12T03:59:21.859Z
Link: https://cerebralab.com/read/1

A quest to solve aging must start with careful consideration of what it is.

Aging is a constant in nature, from archaea to elephants. Where life finds death as counterpart, aging is unopposed, yet frames our understanding of both.

To solve aging is to prevent death and avoid decay.

Death is a rather sudden process and decay is a fuzzy concept.

Decay is non-linear and hard to characterize at the organism level.

Sudden regime changes can happen over the course of days, see: menopause, post-viral syndromes, acute neuropathy, autoimmune diseases.

To make things harder, we are evolutionarily and culturally adapted to obfuscate signs of decay.

Defining decay at the molecular level is presently an intractable information problem. Theoretically possible but practically unsolvable.

The above is, to the best of my outlining abilities, the paradox most researchers get stuck in when they ponder aging.

Is aging real?

Common sense dictates aging happens, we see it all around us.

But what data can we use to back up this intuition?

This graph is the most clear explanation of aging and damning proof of its existence and effects.

We are born, we are weak and unadapted to our environment and thus we die. We grow, we adapt and thus we start dying less. And then, in our teens… that trend flips.

The scariest part about this trend is that we are certain it has held true for the last century, and almost certainly since the first humans arose.

If you break it down by country it holds true, if you split it on income or genetics it holds true, if you filter disease or look only at the most physically fit people… it holds true.

Somewhere, something happens, and we go from getting better and better at not dying to getting worse at it.

Differences in age

What if we take a first-principles approach when defining aging, looking at decay as the derivative of the path between an 80-year-old and a 10-year-old across?

Two overarching curses become evident, regardless of any intervention:

1. Morphological recovery is slower and less precise. Wounds take longer to heal, healing is more likely to scar than restore, and restored structures vary more in their shape than the original. This applies to things as varied as: a scraped knee, a deep knife wound, liver damage from hepatitis, a burn, a frostbitten fingertip, or pancreatic islets destruction after a night of fast food and binge drinking.

2. Learning rates slow d...
The Grapes of Hardness
by adamShimi — 2025-03-11T21:01:14.963Z
Link: https://formethods.substack.com/p/the-grapes-of-hardness

I have an embarrassing secret: I genuinely believe that

Pierre Bourdieu

, the famous french sociologist, had deep insights worth digging for.

It's embarrassing for a host of reasons. First, I haven't read or studied much Bourdieu. So this is not even the deep pronouncement of an expert, just the confused intuitions of a dilettante. The second source of embarassement is more sociological: the kind of crowds I frequent, my friends and family, are not really the ones to like stuff like Bourdieu (with a few exceptions). It's too continental, too fuzzy, smells too much of bullshit. It fails to provide clear predictions and commitment for its "theories", whatever they are.

Despite these deep-seated counter-arguments, the idea to finally study Bourdieu for real this time bubbles up in my mind, over and over again. Maybe I chance a look upon one of the many Bourdieusian tomes in my antilibrary; maybe I just chain thoughts until they reach the vicinity of the sunken insight I feel from the man himself; maybe I just reread old notes and posts that point to or evoke this obscure intuition.

If I have to express this thought, at like-share-subscribe-point, my mortification redoubles: it's not even about Bourdieu's theories or subject matter.

I'm not against sociology, it's just not my focal point here. Rather, what I diffusely smell from Bourdieu's own words, and from the shape of sociology itself, is an unmined vein of methodological insights.

How can I expect powerful methods if I don't know the theories, if even my vague understanding suggest a less than stellar success at predicting, modelling, making sense?

Because what pulls me in Bourdieu is not the positive tricks, methods, and theories — it's the thoughtful recognition of the raggedness of the epistemological terrain.

In a way, I see Bourdieu himself not as the prophet of the right method, but as the erudite companion to the methodological difficulties of sociology, and then of social sciences, and then of human-involved endeavours in general.

It is precisely because sociology is so hard and tricky, because I expect at most fragments of answers and nuggets of insights, not the full breadth of a fitting totalizing theory, that I hope reflections on its practice brim with pearls and treasures. And the little that I've read from him makes me confident Bourdieu is a great fountain of such reflections, in books like "

Le Métie

...
Don't over-update on FrontierMath results
by David Matolcsi — 2025-03-11T20:44:04.459Z

(As an employee of the European AI Office, it's important for me to emphasize this point: The views and opinions of the author expressed herein are personal and do not necessarily reflect those of the European Commission or other EU institutions.)

When OpenAI first announced that o3 achieved 25% on FrontierMath, I was really freaked out. Next day, I asked Elliot Glazer, EpochAI's lead mathematician and the main developer of FrontierMath, what he thought. He said he was also shocked, and expected o3 to "crush the IMO" and get an easy gold, based on the fact that it got 25% on FrontierMath.

In retrospect, it really looks like we over-updated. While the public can't easily try o3 yet, we have access to o3-mini (high) now, which achieves 20% on FrontierMath given 8 tries, and gets 32% using a Python tool. This seems pretty close to o3's result, as we don't know how much extra affordance o3 had while solving the problems, but based on OpenAI's communication, plausibly it's similar to what o3-mini (high) had when it was using a Python tool.

In spite of its great scores on FrontierMath, o3-mini (high) is nowhere close to "crushing the IMO". To the best of my knowledge, it can't solve a single IMO problem from recent years, and in my experiments it's doing somewhat worse than I was in 9th grade on the high school competitions I participated back then.[1][2][3] Other mathematicians and people with competitive math background whom I asked have similar experience.

That's still impressive from an LLM, and the pace of progress is admittedly very fast. [4] Nonetheless, it's not what we originally expected when 25% on FrontierMath was announced. What causes the discrepancy?

Part of the story might be that OpenAI elicits the model's capabilities better than I do. It looks like they give it more inference time, they give it tools, and in some experiments they give the AI more than one tries. In contrast, I only experimented in the normal o3-mini (high) chat interface, only gave at most a few tries per problem, and while I experimented with a few different prompts, I might have missed some clever elicitation prompt.

Differences in elicitation certainly seem to matter. OpenAI claims that "On FrontierMath, when prompted to use a Python tool, o3‑mini with high reasoning effort solves over 32% of problems on the first attempt, including more than 28% of the challen...
Response to Scott Alexander on Imprisonment
by Zvi — 2025-03-11T20:40:06.250Z

Back in November 2024, Scott Alexander asked: Do longer prison sentences reduce crime?

As a marker, before I began reading the post, I put down here: Yes. The claims that locking people up for longer periods after they are caught doing [X] does not reduce the amount of [X] that gets done, for multiple overdetermined reasons, is presumably rather Obvious Nonsense until strong evidence is provided otherwise.

The potential exception, the reason it might not be Obvious Nonsense, would be if our prisons were so terrible that they net greatly increase the criminality and number of crimes of prisoners once they get out, in a way that grows with the length of the sentence. And that this dwarfs all other effects. This is indeed what Roodman (Scott’s anti-incarceration advocate) claims. Which makes him mostly unique, with the other anti-incarceration advocates being a lot less reasonable.

In which case, yes, we should make dramatic changes to fix that, rather than arguing over sentence lengths, or otherwise act strategically (e.g. either lock people up for life, or barely lock them up at all, and never do anything in between?) But the response shouldn’t be to primarily say ‘well I guess we should stop locking people up then.’

Scott Alexander is of course the person we charge with systematically going through various studies and trying to draw conclusions in these spots. So here we are.

Deterrence

First up is the deterrence effect.

Scott Alexander: Rational actors consider the costs and benefits of a strategy before acting. In general, this model has been successfully applied to the decision to commit crime. Studying deterrence is complicated, and usually tries to tease out effects from the certainty, swiftness, and severity of punishment; here we’ll focus on severity.

According to every study and analysis I’ve seen, certainty and swiftness matter a lot, and indeed you get more bang for your buck on those than you do on severity past some reasonable point. The question on severity is if we’re reaching decreasing marginal returns.

A bunch of analysis mostly boils down to this:

I think all four of these studies are consistent with an extra year tacked on to a prison sentence deterring crime by about 1%. All studies start with significant prison sentences, and don’t let us conclude that the same would be true with eg increasing a one day sentence to a year-and-a-da

...
Paths and waystations in AI safety
by Joe Carlsmith — 2025-03-11T18:52:57.772Z
Link: https://joecarlsmith.substack.com/p/paths-and-waystations-in-ai-safety

(Audio version here (read by the author), or search for "Joe Carlsmith Audio" on your podcast app.

This is the third essay in a series that I’m calling “How do we solve the alignment problem?”. I’m hoping that the individual essays can be read fairly well on their own, but see this introduction for a summary of the essays that have been released thus far, and for a bit more about the series as a whole.)

1. Introduction

The first essay in this series defined the alignment problem; the second tried to clarify when this problem arises. In this essay, I want to lay out a high-level picture of how I think about getting from here either to a solution, or to some acceptable alternative. In particular:

I distinguish between the underlying technical parameters relevant to the alignment problem (the “problem profile”) and our civilization’s capacity to respond adequately to a given version of the problem (our “competence profile”).
I lay out a framework for thinking about what competence in this respect consists in – one that highlights the role for three key “security factors,” namely:

Safety progress: our ability to develop new levels of AI capability safely;
Risk evaluation: our ability to track and forecast the level of risk that a given sort of AI capability development involves; and
Capability restraint: our ability to steer and restrain AI capability development when doing so is necessary for maintaining safety.

I distinguish between a number of different possible sources of labor (e.g., both future AI labor, and possible improvements to the quality of future human labor as well) that could improve these security factors.
And I discuss a variety of different intermediate milestones (e.g., global pause, automated alignment research, whole brain emulation, etc) that strategies in this respect could focus on.

This high-level picture sets up my next essay – “AI for AI safety” – which argues that we should try extremely hard to use future AI labor to improve the security factors I discuss; and the essay after that, which examines our prospects for safely automating AI alignment research in particular.

2. Goal states

Let’s start by recalling the goal states we’re aiming for.

In my first essay, I distinguished between two ways of not-failing on the alignment problem:

Victory: avoiding loss of control scenarios while gaining access to the main benefits of superint

...
Meridian Cambridge Visiting Researcher Programme: Turn AI safety ideas into funded projects in one week!
by Meridian Cambridge — 2025-03-11T17:46:29.656Z

Crossposted from this EA Forum post: https://forum.effectivealtruism.org/posts/W4TppzBnHHHZHwgvi/meridian-cambridge-visiting-researcher-programme-turn-ai
April 6-12, 2025 | Cambridge, UK
Apply Now (Deadline: March 19, 2025, anywhere on Earth)

We're excited to announce Meridian and Cambridge AI Safety Hub’s Visiting Researcher Programme, running from April 6-12, 2025, in Cambridge, UK. This programme offers the opportunity for researchers to transform their AI safety research ideas into robust, fundable proposals — and fast.

What makes this programme different?

If you’ve completed a programme like MATS, MARS, ARENA, ML4G, or ERA, you likely know a lot about AI safety. But what next? Many researchers hit a wall between gaining knowledge and launching full-time research projects. That’s the gap we’re aiming to fill.
This programme offers a structured sprint to transform your research ideas into submission-ready proposals for

Open Philanthropy’s

$40 million AI safety research initiative

. Unlike typical upskilling programs, our focus is on team formation, project development, and grant writing — not teaching technical skills. By the end of the week, you’ll have a polished proposal and targeted, actionable feedback from experts in the field.

Full Support Provided:

Travel expense reimbursement to and from Cambridge
Accommodation during the programme week
Workspace, resources, and meals during programme activities

Researchers who secure funding will be invited back to Cambridge to continue their work at Meridian, with visa and accommodation support (subject to approval and funding availability). We’ll also provide research management and administrative assistance so you can focus on what matters: your research.

Programme breakdown:

Days 1-2: Explore research directions and meet potential collaborators through introductory sessions, workshops, and our speed cofounder matching process. You'll connect with researchers whose skills complement yours and begin developing project ideas together.
Day 3: Form your research team and refine your project focus. By the end of day three, you'll have a clear concept and research direction with your newly-formed team.
Days 4-5: Learn effective grant writing strategies from experts and develop your proposal with structured feedback. Sessions will cover research impact assessment and alignmen

...
Elon Musk May Be Transitioning to Bipolar Type I
by Cyborg25 — 2025-03-11T17:45:06.599Z

Epistemic status: Speculative pattern-matching based on public information.

In 2023, Gwern published an excellent

analysis

suggesting Elon Musk exhibits behavioral patterns consistent with bipolar II disorder. The evidence was compelling: cycles of intense productivity followed by periods of withdrawal, risk-taking behavior (like crashing an uninsured McLaren), reduced sleep requirements during "up" phases, and self-reported "great highs, terrible lows."

Gwern's analysis stopped short of suggesting bipolar I disorder, which requires full manic episodes rather than the hypomania characteristic of bipolar II. This distinction isn't merely academic—it represents different risk profiles, treatment approaches, and progression patterns.

Now, I'm beginning to wonder: are we witnessing a potential transition from bipolar II to bipolar I? To be clear, I'm not claiming this has happened, but rather exploring whether the probability of such a transition appears to be increasing based on risk factor analysis.

(Disclaimer: I recognize the limitations of armchair diagnosis, especially of public figures. But there's value in examining these patterns as case studies in how psychiatric conditions manifest in high-functioning individuals, particularly when those individuals have publicly acknowledged aspects of their mental health. Think of this as an intellectual exercise in applied psychiatry rather than a definitive claim about Musk's mental state.)

II. The Bipolar Spectrum: A Brief Primer on Category Boundaries

Before diving deeper, let's establish some conceptual clarity around bipolar disorders.

Bipolar II involves:

Major depressive episodes
At least one hypomanic episode (elevated mood/energy lasting 4+ days)
No history of manic episodes

Bipolar I requires:

At least one manic episode (severely elevated mood/energy lasting 7+ days or requiring hospitalization)
May or may not include depressive episodes
Often includes psychotic features during mania

The key distinction is that hypomania, while problematic, is often productive and doesn't cause severe impairment. Mania, conversely, significantly disrupts functioning and more often comes with psychotic features.

A single manic episode automatically reclassifies someone from bipolar II to bipolar I.

III. Revisiting Gwern's Analysis: The Case for Bipolar II

Gwern's analysis presented multiple lines of evid...
Scaling AI Regulation: Realistically, what Can (and Can’t) Be Regulated?
by Katalina Hernandez — 2025-03-11T16:51:41.651Z

Key frustration: A Shared pain

There’s a certain frustration that comes with watching people regulate a field they don’t understand.

If you’ve ever spent time in AI policy, you’ve probably had moments where you wished policymakers, lawyers, or corporate compliance teams would just listen.

That they would put aside grandstanding, drop the vague legalese, and ask the right questions before drafting rules that will shape the future.

Maybe you’ve watched them frame AI risk in ways that are sadly disconnected from reality. Maybe you’ve seen them focus on what sounds good on paper, rather than what would actually work in practice.

Maybe you’ve been in meetings where regulators ask how we can “prove” that AI won’t be misused, or that it will follow human expectations, without realizing that’s a set of unsolved problems...

Maybe, at some point, you’ve wondered:

Should we even expect regulation to be useful for AI safety?
Is there a version of AI regulation that wouldn’t be performative?

I am not a ML engineer or an alignment researcher. My background is in law and AI governance, and I just spend a lot of time thinking about these gaps.

I believe the only way regulation can work is if it follows safety research, rather than pretending compliance structures alone will make AI safer.

I know that many in this community are skeptical, if not outright dismissive, of regulatory efforts like the EU AI Act.

And to be honest, the more I work in policy, the more I develop some of that skepticism too.

Not about AI governance per se, but about approaches that rigidly separate Policy and Governance from Safety in silos that do not interact.

If policymakers do not follow along once we mention basic terms such as "mechanistic interpretability", "scalable oversight", or "alignment constraints", how can regulation even begin to be effective?

That said, regulation is happening, albeit slowly and with flaws. Still, it’s going to shape AI deployment whether we like it or not.

And from where I stand, we should not be happy for AI regulations to be drafted without meaningful input from Safety researchers.

Without technical literacy, and without accounting for the actual limits of AI systems to be transparent, and meet "legal definitions" of explainability.

I’m posting here because I want to hear from you:

Is regulation completely useless?

Should we ...