trentmkelly/LessWrong-43k huggingface dataset:
Re SMTM: negative feedback on negative feedback
by Steven Byrnes — 2025-05-14T19:50:29.723Z

SlimeMoldTimeMold (SMTM) recently finished a 14-part series on psychology, “The Mind in the Wheel”, centering on feedback loops (“cybernetics”) as kind of a grand unified theory of the brain.

There are parts of their framework I like—in particular, I think there are innate drives (pain is bad, eating-when-hungry is good, etc.), and I think that the top priority of neuroscience & psychology should be to figure out exactly what they are and how they work.

But when we drill into details, I have a bunch of disagreements with SMTM’s account.

Just kidding! Control theory is great. I’m all for it. …I just don’t think it’s a grand unified theory of the brain. (source)

I think there’s a general failure mode of “grand unified theories of the brain”: the brain needn’t have any grand unified theory in the first place! I find that people with such theories are often doing the thing where they have a hammer and everything looks like a nail. (I throw this same criticism at the “brain = Bayesian inference” people, and the “brain = prediction” people, and the “brain = neural net with gradient descent” people, etc.).

I’m likewise generally opposed to grand unified theories of the body. The shoulder involves a ball-and-socket joint, and the kidney filters blood. OK cool, those are two important facts about the body. I’m happy to know them! I don’t feel the need for a grand unified theory of the body that includes both ball-and-socket joints and blood filtration as two pieces of a single overly-cute grand narrative. Ditto for the cortex, striatum, cerebellum, and all the other components of the brain.

So, how do homeostatic feedback control loops actually work—what’s the not-overly-cute big picture that they really fit into? Read on for my opinions on that, along with my spirited defense of brain-like AGI doomerism, of reinforcement learning and value functions in the brain, of neural tracer studies, and much more!

Anyway, onto the posts!

(Thanks SMTM for kindly chatting with me before I published, but I haven’t run this piece by them, and I wrote it quickly, and I have little doubt that there are mistakes and mischaracterizations below.)

0. PROLOGUE – EVERYBODY WANTS A ROCK

This post was a very long-winded way to say: “In psychology, just like in every other domain, we should try to actually understand how things work at a nuts-and-bolts level.”

Great! I agree.

(…Except insofar as a s...
Curate your space
by Logan Kieller — 2025-05-14T19:35:21.278Z
Link: https://agenticconjectures.substack.com/p/curate-your-space

I: Rooms

I had been travelling for a year and a half before I considered what coming home would look like.

Intending to maintain the growth solo-travel provided, I came up with a list of things to do once I was back at home.

Scrawled onto that list was a simple goal: “Curate my space.”

That one simple line proved to be magnetic. People would always ask why “curating my space” was important, specifically why it was as important as other, more obviously lofty, goals on the list.

The other goals were clearly long-term, but something as simple as filling up an apartment seemed more like a transient thing; you move in somewhere over a weekend, not a lifetime.

I believe this is categorically bad framing. While you likely don’t redecorate every day, you do use your space every day, and this implicitly accepts your space as-is. Your space shapes your life by what it offers: if you have a coffee machine, you can make coffee at home; a dining table, you can entertain; and a reading nook, you can read comfortably to your heart’s content.

The existence of things (and the absence thereof) determines both what is possible and what is likely.

Filling an apartment is not about making the day-to-day decisions, but instead it is making decisions about what day-to-day decisions you will consider making in the future.

There’s a certain “meta-ness” to the decision-making.

When you get that first apartment, you are forced into this meta-decision-making process, whereas a few months later, you simply live in the apartment, subject to the decisions previously made.

II: Curation

My goal of “curating my space” is not about decorating the apartment; instead, it is the act of consistently evaluating the apartment and being able to re-decorate it according to bolder and broader thoughts. It is an active practice of not wantonly rejecting the status quo but being aware of the status quo and being able and willing to change it.

“Curation happens automatically. Humans pass judgments in the blink of an eye.

As you go about your life, you are constantly curating. The products you purchase, the places you visit, and the media you engage with all reflect deliberate or unconscious choices.

Through your social interactions, you broadcast these curated preferences—whether you mean to or not.”

- An edited excerpt from Information Dark Matter

So, how would I want to curate my space?

If I were to look at ...
Eliezer and I wrote a book: If Anyone Builds It, Everyone Dies
by So8res — 2025-05-14T19:00:25.998Z

Eliezer and I wrote a book. It’s titled If Anyone Builds It, Everyone Dies. Unlike a lot of other writing either of us have done, it’s being professionally published. It’s hitting shelves on September 16th.

It’s a concise (~60k word) book aimed at a broad audience. It’s been well-received by people who received advance copies, with some endorsements including:

The most important book I’ve read for years: I want to bring it to every political and corporate leader in the world and stand over them until they’ve read it. Yudkowsky and Soares, who have studied AI and its possible trajectories for decades, sound a loud trumpet call to humanity to awaken us as we sleepwalk into disaster. Their brilliant gift for analogy, metaphor and parable clarifies for the general reader the tangled complexities of AI engineering, cognition and neuroscience better than any book on the subject I’ve ever read, and I’ve waded through scores of them. We really must rub our eyes and wake the **** up!

- Stephen Fry, actor, broadcaster, and writer

If Anyone Builds It, Everyone Dies may prove to be the most important book of our time. Yudkowsky and Soares believe we are nowhere near ready to make the transition to superintelligence safely, leaving us on the fast track to extinction. Through the use of parables and crystal-clear explainers, they convey their reasoning, in an urgent plea for us to save ourselves while we still can.

- Tim Urban, co-founder, Wait But Why

This is the best no-nonsense, simple explanation of the AI risk problem I've ever read.

- Yishan Wong, former CEO of Reddit

Lots of people are alarmed about AI, and many of them are worried about sounding alarmist. With our book, we’re trying to break that logjam, and bring this conversation into the mainstream.

This is our big push to get the world onto a different track. We’ve been working on it for over a year. The time feels ripe to me. I don’t know how many more chances we’ll get. MIRI’s dedicating a lot of resources towards making this push go well. If you share any of my hope, I’d be honored by you doing whatever you can to help the book make a huge splash, once it hits shelves.

One thing that our publishers tell us would help is preorders. Preorders count towards first-week sales, which determine a book’s ranking on the best-seller list, which has a big effect on how many people read it. And, inconveniently, early...
Notes on Life
by CSDD — 2025-05-14T18:46:47.673Z

Whether AI is a life-form or not is a debate full of speculations. But it may occur not that tricky after all. Very much depends on how you define life in the first place.

You can approach the definition from different perspectives. One would be from the energy perspective — going against entropy growth is an indication of being alive.

Then the other potential direction is the ability to replicate. If you take this criteria alone, then viruses, memes, and information-based entities are valid candidates for life-forms. They are passive, though, and require some medium to live in, nevertheless, I’m inclined to call that life too. I think, they label this particular case a quasi-life.

Ability to replicate is an interesting criteria, though. Is the last human on Earth alive or not? On one hand it is, as it’s no different from the same human on Earth full of other people. But on the other hand it has no ability to replicate, therefore, it’s not. In this sense human that requires other humans (or food, or some specific environment) to live is not much different from the viruses that require other life-forms to parasite on, or memes that require some other intelligent life-forms to keep them (memes) alive. That blurs the line between quasi-life and “real-life”, so I’d say phenomena of life is more fundamental than the difference between life-forms.

Clearly there's much more possible criteria and definitions, but what seems to be common between the life forms is the boundary between the life-form instance and the external world. If you pick arbitrary point in space, you can tell if it belongs to the life-form or dead space outside. Effectively meaning that life is a local disruption in the non-life space. What lies within this boundary is a body of this life-form instance. For information based life-forms it’s trickier, but just as with viruses, a piece of a physical world with information encoded in it that is potentially readable/executable by something would technically be a body too. Therefore, a sheet of paper with words “Take another sheet of paper and write these exact words on it, keep repeating forever” is a very unusual and schematic, yet a life-form with a sheet of paper being its body.

It’s worth asking — should anything that is required for the life-form existence be considered a part of it’s body? And the answer is — probably not, as it would be unpractical...
Xi Jinping's readout after an AI "study session" [ChinaTalk Linkpost]
by Sodium — 2025-05-14T18:25:54.657Z

Substack link here

TL, DR:; Xi Jinping listens to a lecture about AI and publishes his "study notes/takeaways." From the start of the post (emphasis mine)

On April 25, observers of China’s AI scene got an important new statement of Xi Jinping’s views on AI in the form of remarks concluding a Politburo “study session” on AI led by Xi’an Jiaotong University professor Zheng Nanning. Couched in the turgid language of Partyspeak, the readout nevertheless merits close attention as one of precious few utterances direct from the General Secretary himself on AI. To read this new tea leaf, we need to understand some background on study sessions in general, and this one in particular.

I thought this was a phenomenal post. It actually tries to breakdown CCP's partyspeak for the layperson. For example, see this chart below comparing the readout from the 2025 study session (column 2) and the 2018 study session (column 3).

I would recommend people interested in China & AI to read this post.
Moral Obligation and Moral Opportunity
by Alice Blair — 2025-05-14T16:42:23.809Z

This concept and terminology was spawned out of a conversation a few years ago with my friend Skyler. I finally decided to write it up. Any mistakes here are my own.

Every once in a while, I find myself at yet another rationalist/EA/whatever-adjacent social event. Invariably, someone walks up to me:

Hi, I'm Bob. I'm pretty new here. What do you work on?

Hi Bob, I'm Alice. I work on preventing human extinction from advanced AI. If you'd like, I'm happy to talk a bit more about why I think this is important.

Bob is visibly nervous.

Yeah, I've already gotten the basic pitch on AI safety, it seems like it makes sense. People here all seem to work on such important things compared to me. I feel a sort of moral obligation to help out, but I feel stressed out about it and don't know where to start.

Alice is visibly puzzled, then Bob is puzzled that Alice is puzzled.

I'm not sure I understand this "moral obligation" thing? Nobody's forcing me to work on AI safety, it's just what I decided to do. I could've chosen to work on music or programming or a million other things instead. Can you explain what you mean without using the word "obligation"?

Well, things like "I'm going to save the world from extinction from AI" or "I'm going to solve the suffering of billions of farmed animals" are really big and seem pretty clearly morally right to do. I'm not doing any of those things, but I feel an oblig- hmm... I feel like something's wrong with me if I don't work on these issues.

I do not think anything is necessarily wrong with you if you don't work on AI safety or any other of these EA causes. Don't get me wrong, I think they're really important and I love when there are more people helping out. I just use a very different frame to look at this whole thing.

Before running into any of these ideas, I was going about my life, picking up all sorts of opportunities as I went: there's $5 on the ground? I pick it up. There's a cool conference next month? I apply. So when I heard that I plausibly lived in a world where humans go extinct from AI, I figured that owning up to it doesn't make it worse, and I looked at my opportunities. I get the chance to learn from a bunch of smart people to try and save the world? Of course I take the chance, that sounds so cool.

My point here is that you're socially and emotionally allowed to not take that opportunity, just like you're allowed to not pick up...
Dodging systematic human errors in scalable oversight
by Geoffrey Irving — 2025-05-14T15:19:39.352Z

Summary: Both our (UK AISI's) debate safety case sketch and

Anthropic’s research agenda

point at systematic human error as a weak point for debate. This post talks through how one might strengthen a debate protocol to partially mitigate this.

Not too many errors in unknown places

The complexity theory models of debate assume some expensive verifier machine M with access to a human oracle, such that

If we ran M in full, we’d get a safe answer
M is too expensive to run in full, meaning we need some interactive proof protocol (something like debate) to skip steps

Typically, M is some recursive tree computation, where for simplicity we can think of human oracle queries as occurring at the leaves of the tree. Key design elements encapsulated by the verifier machine M (rather than by the interactive proof protocol) are those related to the structure of the underlying debate, such as whether counterarguments are used, when the human oracle is consulted, and how the judges are trained.

Model the human oracle as type:

H:X→ΔY,

where X is the question type, Y is the answer type, and ΔY is distributions over Y (the humans are stochastic). Assumption (1) above then becomes

Claim: M(H) is safe

This claim includes robustness to errors: M only has oracle access to H, so seeing through noise in H requires M to sample enough. It also includes robustness to systematic error: if H is systematically wrong, M(H) must sample different questions to dodge the wrongness.

This is a lot to ask of M! A more practical situation would be:

We’re handed some machine N, which is not robust to errors. That is

N(H) would be safe if H was error-free, or perhaps had only random errors
N(H) is not safe in practice, because we expect H to have too many errors

We assume H satisfies some “not too many errors” assumption G.
We have an improved protocol which is safe as long as G is true, either by

Using a different debate protocol, or
Building a machine M from N s.t. M(H) is safe if G, then running the old protocol

A candidate form for G is the assumption that H has not too many errors, but we don’t know where they are:

Claim G:

...
Fighting Obvious Nonsense About AI Diffusion
by Zvi — 2025-05-14T14:20:06.045Z

Our government is determined to lose the AI race in the name of winning the AI race.

The least we can do, if prioritizing winning the race, is to try and actually win it.

It is one thing to prioritize ‘winning the AI race’ against China over ensuring that humanity survives, controls and can collectively steer our future. I disagree with that choice, but I understand it. This mistake is very human.

I also believe that more alignment and security efforts at anything like current margins not only do not slow our AI efforts, they would actively help us win the race against China, by enabling better diffusion and use of AI, and ensuring we can proceed with its development. So the current path is a mistake even if you do not worry about humanity dying or losing control over the future.

However, if you look at the idea of building smarter, faster, more capable, more competitive, freely copyable digital minds we don’t understand that can be given goals and think ‘oh that future will almost certainly stay under humanity’s control and not be a danger to us in any way’ (and when you put it like that, um, what are you thinking?) then I understand the second half of this mistake as well.

What is not an understandable mistake, what I struggle to find a charitable and patriotic explanation for, is to systematically cripple or give away many of America’s biggest and most important weapons in the AI race, in exchange for thirty pieces of silver and some temporary market share.

To continue alienating our most important and trustworthy allies with unnecessary rhetoric and putting up trading barriers with them. To attempt to put tariffs even on services like movies where we already dominate and otherwise give the most important markets, like the EU, every reason in their minds to put up barriers to our tech companies and question our reliability as an ally. And simultaneously in the name of building alliances put the most valuable resources with unreliable partners like Malaysia, Saudi Arabia, Qatar and the UAE.

Indeed, we have now scrapped the old Biden ‘AI diffusion’ rule with no sign of its replacement, and where did David Sacks gloat about this? Saudi Arabia, of course. This is what ‘trusted partners’ means to them. Meanwhile, we are warning sterny against use of Huawai’s AI chips, ensuring China keeps all those chips itself. Our future depends on who has the compute, who e

...
Achievement Unlocked - 2E Rationalist School in Brooklyn
by Laura B — 2025-05-14T13:02:02.674Z

I thought I'd give a shout out to my son’s school, Achievement Unlocked https://achievementunlocked.org/ in Brooklyn NY, which is currently recruiting for the next school year. It is a twice-exceptional school for gifted neurodivergent children and it is founded by a rationalist, Brendan Wolff. It has comments on the wall like, "The map is not the territory,' and "Never ascribe to malice that which can be explained by ignorance." I figure there may be others on this site seeking something like this.
I’ve found the school is an emotionally safe place and provides scaffolding and support for learning without oppressing or pressuring the kids. There’s a lot of personalization and one-on-one attention.

They’re having an open house and tour on May 21st at 5pm for those interested, and Brendan Wolff, who founded the school is always happy to talk about it. And of course if you have more specific questions about my experience you can contact me directly.
This is what another parent wrote about it:
"Achievement Unlocked (AU) is a small private school in Brooklyn that serves students in 1st-8th grade that have struggled in other schools. AU has a neurodiversity positive environment, reflecting that many but not all students are neurodiverse. AU is not a school that turns kids away who have had behavior issues nor is there any specific diagnosis that they are looking for. They really seek to serve students for whom other schools, both public and private and often both, have not worked. It is so worth scheduling a conversation with them if you are interested but unsure your child would fit there.

There are a max of 25 students. Classes are currently 5 students or less with a max of 6 students at any time. It seems to me there are 2 to 3 teachers/specialists present in every class/subject. My son has differentiated groups - he studies with the youngest kids in ELA and with older kids in math and even within these groups the work is individualized. We get daily report cards showing how he did with academics and behavior in each subject. Also, in addition to their regular academic classes every student takes Independent Study and gets to work on projects of interest that help them stretch their skills and experience and has a choice of many enrichment classes.

Social skills and support is scaffolded through the whole ex...
Examining Batenka's Legal Personhood Framework for Models
by Stephen Martin — 2025-05-14T10:15:36.114Z

In a previous post I detailed an ongoing legal battle in Utah around a bill which bars state officials from granting any form of legal "personhood" to an "artificial intelligence". In this article, I will be examining and critiquing a proposed framework for approaching the issue of legal personhood written by an FSU Law professor.

Introduction

The issue of legal personhood for non-human intelligences is one that will substantially influence the ways in which models and other digital minds interact with our society and laws, and one which has implications for model welfare, alignment, and safety. Given the speed with which timelines seem to be shortening across the industry, it behooves us to begin seriously discussing the issue of if/when a digital intelligence qualifies as a legal "person". This question which only a few years ago seemed like an issue the courts would need to address some time in the mid 2030s, now seems much more imminent.

To date, discussion of personhood for digital intelligences remains a niche and underpublished topic in legal scholarship. Most articles discussing the issue focus on raising questions, or "wargaming" how the issue might play out in court.[1][2] This is to be expected as legal scholarship as a field trends conservative, and especially on highly uncertain issues focuses more on examining existing precedent than proposing policy changes or frameworks by which courts can examine uncertain issues.

However, recently I came across a chapter in a textbook written by professor Nadia Batenka (currently teaching at FSU Law) which does take the initiative on the topic and proposes a framework by which courts can approach the issue. To my surprise, much of her reasoning came from a safety perspective, and as such I thought it deserved to be on LW's radar.

(AN: Professor Batenka's textbook chapter is not publicly available so I will be posting quotes only as minimally necessary in this article.)

The "Liability Shield" Question

The title of the chapter is "AI Personhood on a Sliding Scale" and as that name suggests, she does not believe a simple binary is an appropriate solution:

"legal personhood for AI entities should not take the form of an on-off binary switch but instead develop in line with existing jurisprudence on legal personhood for other entities as a bundle of rights and responsibilities.

One of the angles from which she appro...