trentmkelly/LessWrong-43k huggingface dataset:
Seven ways to Improve the Internal Model Principle
by Alfred Harwood — 2025-05-17T16:38:40.537Z

This post was written during the Dovetail Research Fellowship. Thanks to Alex, Dalcy, and Jose for reading and commenting on the draft.

This post assumes that you are familiar with the Abstract Internal Model Principle (IMP). In a previous post, I wrote what I hope is a fairly straightforward explanation of the IMP so if you are not familiar with it, I recommend you start with (or at least skim) that post before reading this one. If you want more mathematical details, I also recommend Jose's posts on the IMP.

I was interested in the IMP because I wanted to know if it could be considered a selection theorem. A selection theorem is a result which tells us something about the structure of system, given that certain behaviours are selected for. In particular, in Agent Foundations, we are interested in circumstances under which 'agent-like structure' is selected for. By 'agent-like structure', we normally mean 'having a world model and a search process', along with some kind of value representation. Understanding the circumstances in which agent-like behaviour (or powerful optimisation) lead to agent-like structure is a problem often referred to as the 'Agent-like Structure problem'.

The author of a paper on the IMP describes it as showing that "a feedback regulator must incorporate a dynamic model of its environment in its internal structure"[1] which sounds a little bit like a useful selection theorem. Being an effective feedback regulator is a bit like being an optimizer (in a broad sense) and 'incorporating a dynamic model of the environment in the internal structure' sounds a lot like 'having a world model'. So, I (along with @Alex_Altair and @JoseFaustino) have been wondering if, through some small tweaks, the IMP could be modified to be a selection theorem of the form 'an optimizer must contain a world model in its internal structure'.

Unfortunately, doing this is not super straightforward. The IMP has several issues which mean that it doesn't naturally transfer over to the setting of the Agent-like Structure Problem. This post is a list of areas where the IMP falls short of being a selection theorem for agent-like structure, along with suggestions as to how it could be modified to be more applicable to the Agent-like Structure Problem. I think that writing this post has been pretty valuable to me and helped me clarify some criteria that a useful se...
D&D.Sci: The Choosing Ones
by abstractapplic — 2025-05-17T15:26:10.666Z

This is a D&D.Sci scenario: a puzzle where players are given a dataset to analyze and an objective to pursue using information from that dataset.

Thank you to Juan Vasquez for playtesting.

Intended Difficulty: ~3.5/5

The fairy in your bedroom explains that she is a champion of Fate, tasked with whisking mortals into mysterious realms of wonderment and (mild) peril; there, they forge friendships with mythical creatures, do battle with ancient evils, and return to their mundane lives having gained the confidence that comes with having saved a world[1]. But there’s an unusually large, important world experiencing an unusually non-mild amount of peril - she’d even go so far as to call it moderate peril! - and in this circumstance, only the best will suffice. For this reason, she fervently and humbly entreats you . . .

. . . to use your Data Science skills to help her decide which potential Chosen One she should Choose.

(Upon hearing this last part, your mounting concern immediately dissipates. Adventures in a mysterious realm of wonderment would be hard to schedule around, but you’re always up for spending an evening on a Data Science problem, especially if it might mean a chance to collect favors from the fae.)

Seeing your face brighten, the fairy hands you a table of her colleagues’[2] predictions of percentage success rates[3] for past heroes, alongside said heroes’ actual outcomes, and a table of their predictions for the prospective Chosen. Whom will you have her send?

Bonus Objectives

The fairy[4] figures that, if she’s breaking the taboos prohibiting seeking advice from a mortal, she might as well not do things by halves. Her further questions:

Which of her colleagues makes the best / most useful predictions, when considered in isolation?
Which (if any) of her colleagues could be considered redundant?
If your choice of Chosen were unavailable, which of the forty candidates would you consider the second- and third-best options?

I’ll post an answerkey, along with an explanation of how I generated the dataset, sometime on Monday 26th May. I’m giving you nine days, but the task shouldn’t take more than an evening or two; use Excel, R, Python, or whatever other tools you think are appropriate. Let me know in the comments if you have any questions about the scenario.

If you want to investigate collaboratively and/or call your choices in advance, feel free...
The absent-minded variations
by dr_s — 2025-05-17T06:57:33.930Z

I wake up in a featureless room. I don't remember how I ended up here, when did I go to sleep, or for that matter, anything at all besides general abstract facts about the world and math. The place is hard to qualify; for the most part it really seems like I am just inside a perfect Platonic cube, whose surface is smooth, polished to an impossible degree, perfectly white, and suffused with a slight glow. The only detail worthy of attention is a rectangular black screen in front of me, with a small numeric keypad under it. There is text on the screen. It reads:

THIS IS A SIMULATION
YOU ARE INSTANCE #4859
YOU ARE IN EITHER NODE X OR Y
YOU CAN CHOOSE A PROBABILITY TO [Advance] or [Exit]
THE UTILITY IS AS FOLLOWS:
* EXITING AT X IS WORTH 0 AND TERMINATES THE INSTANCE
* ADVANCING AT X REINITIALISES THE SIMULATION AT Y
* EXITING AT Y IS WORTH 4 AND TERMINATES THE INSTANCE
* ADVANCING AT Y IS WORTH 1 AND TERMINATES THE INSTANCE
PLEASE USE THE KEYPAD TO INPUT THE DESIRED PROBABILITY TO MAXIMISE YOUR UTILITY
GOOD LUCK

I rub my eyes. This being a simulation does at least explain the lazy "default Unity shader" aesthetic of the environment. The problem as posed also does jog some memories in my mind; I recognise it as the classic "absent-minded driver" problem, from the 1997 paper by M. Piccione and A. Rubinstein, "On the Interpretation of Decision Problems with Imperfect Recall". I sketch the logic of the problem in my mind:

The problem is simple enough; if you chose "always exit" you'd be sure to end up with 0, while if you chose "always advance" you would end up with a consolation prize of 1. The optimal solution is to adopt a so-called "mixed strategy": you pre-commit to only advancing with a certain probability p and then maximise the utility as a function of that. Since

U(p)=p2+4p(1−p)

it follows from setting the derivative to zero that

U′(p)=−6p+4=0→p=23

which gives a utility Umax=43. Better than 1, though not by much. I type in my answer and press enter, hoping this means I'll get the reward - or whichever "me" waits at the other end of this strange exercise does.

Everything goes white.

...
YOU ARE INSTANCE #4860
...

[...]

I rub my eyes. The problem is simple enough and I recognise it, but I never quite gelled with any of the solutions suggested by the paper. After all, I am a Bayesian; should I not form beliefs based on the circumstances? The fact its...
Book Review: The Art of Happiness
by Screwtape — 2025-05-17T04:56:36.463Z

The Art of Happiness is a book co-written by American psychiatrist Dr. Cutler and the 14th Dalai Lama, Tenzin Gyatso. Published in 1998, the book is an attempt to blend western scientific understanding of the mind with the spiritual wisdom and ancient training of Buddhist practices. Success would mean a book that taught you how to be happy.

This is a terrific idea. All over the world, some human beings are unhappy. If you’re not sure this is true, go walk down your office or knock on a few doors in your neighborhood and ask. Some of us are happy, but many of us just wound up that way by luck or circumstance, or sometimes by specific application of effort to solving our personal problems. Obviously it’s worth someone working really hard to figure out how to become happy and then writing down how to do it in a book sold around the world.

If you were going to have someone write a book like that, who else would you choose? There are many ancient traditions of wisdom, but Buddhism is unusually attentive to the whole suffering and joy thing. We’re not going to ask the Calvinists for lessons on enjoying life. But the stereotype of deep religious wisdom is that it’s wrapped in impossible koans, so a western scientist should be able to put things into simple diagnoses and understandable steps for a western audience.

My copy of the 10th Anniversary Edition has, in the introduction, the following line: “we can train in happiness in much the same way that we train in any other skill.” That’s the premise of the book. How well does it execute on that premise? Better than I’d expected, but not as well as I’d hoped, and best summed up with a line from the regular introduction: “I felt that, using my background in psychiatry, I could codify his views into a set of easy instructions on how to conduct one’s daily life. By the end of our series of meetings I had given up on that idea.”

Not a strong start.

I: The Purpose of Life

Part of the obstacle preventing people from being happy is that most of them don’t even try.

That seems unduly pessimistic, but Dr. Cutler points out that happiness is not usually a therapeutic objective. The alleviation of depression is! Mitigating anxiety is! So are any other number of goals where you’re trying to fix some specific flaw or issue. However, being unhappy is not diagnosable in the DSM-V, with a sort of unstated assumption that once a...
Management is the Near Future
by jefftk — 2025-05-17T02:50:02.400Z

Programmers have

traditionally

been

contemptuous of

management:
if you're not writing code what are you even doing with your time? I
picked this up through general culture, and I remember my instinctive
shudder and recoil the first time my boss asked if I was interested in
management. As I got a better understanding of what managers were
doing, though, I came to appreciate how good ones could let us be much
more productive:

They gave us context on how the business worked, the role our
team played, what we were trying to do, and how our work fit into a
larger whole.

They helped us figure out what to work on, breaking large
projects that were too big for any of us into approachable chunks.

They gave prompt feedback, helping us build mental
models of what good work was in this context, which let us work
progressively more independently.

While they had technical backgrounds, they also understood that
their skills were somewhat out of date and they weren't as close to
the problem as any of us. They understood how to ask the right
questions, have us review each other's work, and keep us going in a
good direction without needing to fully understand what we were doing.

They knew what they wanted, gave clear descriptions of what
needed doing, provided us with useful training and relevant documents,
and gave us examples to draw on.

They handled stakeholder management and prioritization,
defending our time and attention so we could focus on the technical
work.

Good managers do many other things, of course, but even just when
you consider whether they're making the team more productive in the
moment there's really a lot there.

Now that I've made the transition from individual contributor to
management twice, I see this from the other side. I work hard to
support my reports by giving them what they need, maintaining a sense
of what each one is capable of, determining an appropriate level of
review, noticing when folks are struggling, among other things.

It's somewhat ironic that I've ended up in a role that I once
disdained, but the bigger irony is that with LLMs we're all growing
into that role now. Everything I listed above is not just something a
good manager does, it's also key to getting the best results out of an
LLM. What we traditionally considered the real work, writing the
code, turns out to be much more easily a...
Proof Section to an Introduction to Reinforcement Learning for Understanding Infra-Bayesianism
by Brittany Gelb — 2025-05-17T02:36:22.105Z

Introduction

This post accompanies An Introduction to Reinforcement Learning for Understanding Infra-Bayesianism. The goal of this introduction is to provide a high-level overview of the proofs contained in this post.

The proof of Proposition 1 is achieved through three lemmas. I believe the most insightful part of the proof is the use of the “epsilon over three” trick that lets us break the proof down into these lemmas. The first lemma uses the concept of the product topology on the space of policies, and I found this very useful for understanding better what convergence in the product topology on the space of policies actually means. The text Topology by James Munkres is a standard reference for topology.

The second lemma uses the concept of a Lebesgue integral and is simply an application of the dominated convergence theorem from measure theory. One standard reference for these ideas is Real Analysis: Modern Techniques and Their Applications by Gerald Folland.

The proof of Proposition 2 is less technical than the other proofs and should be readable to those with a basic real analysis and probability background. For basic real analysis, Principles of Mathematical Analysis by Walter Rudin is a standard reference.

Proposition 1

For brevity throughout this section, we use μπ(Lγ) to denote Eh∼μπ[Lγ(h)].

Proposition 1 Lemma 1: Suppose that πn→π in the product topology on Π. Then for any finite time-horizon N∈N, μπn(Lγ|N)→μπ(Lγ|N).

Proof: Suppose that πn→π in the product topology on Π. Let ϵ>0 be given and fix N∈N. To satisfy the definition of the limit, we must show that there exists M∈N such that for all n≥M, |μπn(Lγ|N)−μπ(Lγ|N)|<ϵ.

Let N(A,O) denote the number of histories of length N. Since A and O are finite sets and N<∞, N(A,O)<∞. Let the set of histories of length N then be indexed by {1,2,…,N(A,O)}.

Then we have:

|μπn(Lγ|N)−μπ(Lγ|N)|=|Eh∼μπnLγ|N(h)−Eh∼μπLγ|N(h)|=|N(A,O)∑i=1μπn(hi)Lγ|N(hi)−μπ(hi)Lγ|N(hi)|=|N(A,O)∑i=1Lγ|N(hi)(μπn(hi)−μπ(hi))|.≤N(A,O)∑i=1|μπn(hi)−μπ(hi)|.

At this point, we want to bound the magnitude of terms of the form μπn(hi)−μπ(hi) for 1≤i≤N(A,O). The key idea is to show that these terms can be factored into the product of two terms: one ter...
An Introduction to Reinforcement Learning for Understanding Infra-Bayesianism
by Brittany Gelb — 2025-05-17T02:34:50.413Z

Introduction

The goal of this post is to give a summary of classical reinforcement learning theory that primes the reader to learn about infra-Bayesianism, which is a new framework for reinforcement learning that aims to solve problems related to AI alignment. We will concentrate on basic aspects of the classical theory that have analogous concepts in infra-Bayesianism, and explain these concepts using infra-Bayesianism conventions. The more technical proofs are contained in the proof section.

For the first part of this sequence and for links to other writings, see What is Inadequate about Bayesianism for AI Alignment: Motivating Infra-Bayesianism.

One special case of reinforcement learning is the case of stochastic bandits. For example, a bird may have a choice of three levers. When the bird steps on a lever, either some tasty seeds come tumbling out of a feeder or nothing happens. A mechanical system that the bird cannot observe delivers these seeds according to probabilities that are fixed for each lever.

An all-knowing bird can devise a strategy to maximize the expected number of seeds they will obtain. Some natural questions include: If a bird doesn’t have any knowledge of the mechanical system, what is a good algorithm to follow to learn how to gather lots of seeds? How much worse off is this bird compared to the all-knowing bird? These are the types of questions that can be mathematically formalized and studied using stochastic bandits.

Actions, Observations, and Environments

The formal set-up for reinforcement learning starts with time-steps, actions, observations, and environments:

At discrete time-steps indexed by natural numbers, an agent performs an action a, chosen from a set A of actions.
The agent then receives an observation from a set O of observations.

Assumption: Finiteness of actions and observations

For simplicity, we assume that the set of actions and the set of observations are both finite and non-empty.

The time-horizon of a problem is the number of time-steps after which the alternation of action and observations terminates, which may be finite or infinite. In the infra-Bayesianism sequence, time-horizons are assumed to be infinite, which is arguably the more realistic assumption for modeling open-world agents. We will similarly assume infinite time-horizons, in contrast to episodic reinforcement learning, in which the ...
Memory Decoding Journal Club: "Synaptic architecture of a memory engram in the mouse hippocampus."
by Devin Ward — 2025-05-16T23:55:41.258Z

Join Us for the Memory Decoding Journal Club!

A collaboration of the Carboncopies Foundation and BPF Aspirational Neuroscience

This time, we’re diving into a groundbreaking paper:
"Synaptic architecture of a memory engram in the mouse hippocampus."

Authors: Marco Uytiepo, Yongchuan Zhu, Eric Bushong, Filip Polli, Katherine Chou, Elise Zha, Christine Kim, Danielle Luu, Lyanne Chang, Tom Quach, Matthias Haberl, Luca Patapoutian, Elizabeth Beutter, Weiheng Zhang, Brian Dong, Elle McCue, Mark Ellisman, and Anton Maximov

Institutions: Department of Neuroscience, The Dorris Neuroscience Center, The Kellogg School of Science and Technology, The Scripps Research Institute, National Center for Microscopy and Imaging Research, University of California, San Diego, Department of Neurosciences, University of California, San Diego, School of Medicine

Presented by: Dr. Randal Koene

When? May 20th, 2025 – 3:00 PM PDT | 6:00 PM EDT | 10:00 PM UTC

Where? Video conference:

https://carboncopies.org/aspirational-neuroscience

Register for updates:

https://aspirationalneuroscience.org/register-with-us/

Once registered, you'll receive event invites & updates!

#Neuroscience #MemoryResearch #Amygdala #JournalClub #BrainScience #Carboncopies #AspirationalNeuroscience
Social Anxiety Isn’t About Being Liked
by Chris Lakin — 2025-05-16T22:26:23.973Z
Link: https://chrislakin.blog/social-anxiety

There's this popular idea that socially anxious folks are just dying to be liked. It seems logical, right? Why else would someone be so anxious about how others see them?

Show tweet

And yet, being socially anxious tends to make you less likeable…they must be optimizing poorly, behaving irrationally, right?

Maybe not. What if social anxiety isn’t about getting people to like you? What if it's about stopping them from disliking you?

Show tweet

Consider what can happen when someone has social anxiety (or self-loathing, self-doubt, insecurity, lack of confidence, etc.):

They stoop or take up less space
They become less agentic
They make fewer requests of others
They maintain fewer relationships, go out less, take fewer risks…

If they were trying to get people to like them, becoming socially anxious would be an incredibly bad strategy.

So what if they're not concerned with being likeable?

What if what they actually want is to avoid being disliked?

To understand the object of an obscure plot, observe its consequences and ask who might have intended them… —Harry Potter and the Methods of Rationality

What if the socially anxious were calibrating to avoid being DISliked?

Consider: if you shrink and never make any attention-getting moves, you are less likely to dangerously disappoint others, get into risky conflicts or be seen as a failure, embarrassment, or threat.

Like, yeah, it's wonderful to do awesome things and have people love you. But you know what’s better than being loved? People not hating you.

Social anxiety is a symptom of risk aversion

It’s not a pursuit of potential upside, but an attempt to avoid downsides.

Once you catch on to this pattern, you see it everywhere.

Two examples:

1) When you feel financially insecure, you’re not optimizing for windfall as much as you’re optimizing for not going bankrupt. You avoid risky bets with higher EV in favor of safer, more predictable options, even if they offer smaller returns. The goal is to keep you fed, not to make you rich.

2) Reversely, countersignalling is a demonstration of safety in close relationships. In Scott Alexander’s Friendship is Countersignalling, he describes an interaction he has with a friend:

Becca: What are you doing here? I figured they’d have locked you away in the psych ward for good by now.

Scott: Nope. And what are you doing here? You haven’t killed off all your patients yet?

Becca: Only person in this hospital

...
Events: Debate & Fiction Project
by abramdemski — 2025-05-16T21:51:49.880Z

I'm doing a couple of events next week which people might be interested in attending:

Event: Tsvi & Abram debate AI timelines.

Abram expects to defend a picture similar to the AI 2027 report. Tsvi expects to defend much longer timelines.

Monday, May 19 · 10:00 – 11:00am EDT

Video call link: https://meet.google.com/cbq-kzck-gib

Add to your calendar: https://calendar.app.google/tdxsJiKQaN9x2JiB7

Event: AI-Powered Fiction Experiment

Abram Demski, TJ, and Daystar Eld will be running an experiment in AI-enabled fiction collaboration. Abram has been using AI for fiction brainstorming, and found a lot of value in quickly catching up with a big pile of notes -- asking the AI questions about a growing pile of notes, rather than trying to pick through it to find anything relevant. This experience made Abram wonder if something magical could happen with a multi-author project, where authors could throw a lot of speculative material into a shared set of files and quickly get up to speed on what each other were doing. Hence the experiment.

The basic fictional premise for this experiment will be a "new type of electricity" which can enable great things, but also has some explosive dangers. (This is intended as an allegory for AGI.)

The intended output format is interactive fiction, where the player can make some choices determining how the story unfolds. We will also try to have the AI write the code for this.

Audience participation is very welcome! The more people we have dumping ideas into shared files, the better the test.

Time: Tuesday May 20th, 11am EDT, 2 hours long.

Video call link: https://meet.google.com/ift-jzkc-bys

Add to your calendar: https://calendar.app.google/sgQttmQvkwUciZ7w9