trentmkelly/LessWrong-43k huggingface dataset:
Optimization & AI Risk
by atharva — 2025-05-13T15:15:16.221Z

There many ways to taxonomize AI risk. One interesting framing, is ‘risks from optimization’. These are not new ideas. Eliezer

wrote about this

~15 years ago, and it seems like many ‘theory folks’ have been saying this for years. I don’t understand these concepts deeply – I’m trying to

improve my understanding by writing about them

. Hopefully, I can add something new in the process.

Thanks to Jo Jiao for comments on a draft, and for nudging me to write this. Feedback is highly appreciated!

Epistemic status: Exploratory.

Tl;dr: intelligence is optimization, and (too much) optimization is bad.

First, what is optimization? It’s ‘squeezing’ the world into improbable states. Worlds where I have a quintillion dollars in my bank account, are much less likely than worlds where I don’t. So I’d need to optimize strongly to make this possible. This also illustrates degrees of optimization. Earning a thousand dollars is much easier than earning a million dollars. So, I’d need to optimize less hard to achieve the former. Optimizers don’t need to be ‘conscious’ entities. For instance, it’s the abstract forces of evolution that made complex, multicellular life possible.[1] In the real world, one’s ‘capacity to optimize’ corresponds to how much intelligence / money / power one has.

This framing helps unify risks from misuse & misalignment.[2] Paperclip maximizers are the prototypical example of misalignment. Here, it’s the AI system that’s directly optimizing too hard. On the misuse end, take

AI-enabled coups

. Here, it’s the people / group that use the AI system to strongly optimize for their own ends.

Too much optimization seems generally bad

. This is for two reasons. One, worlds that someone else optimizes strongly are unlikely to be worlds that you’d prefer as well. Eg. you don’t want to live in a dictatorship. But also, I’d also be wary about optimizing too strongly even for *my* own goals. Human goals are often weird & inconsistent. It’s easy for my stated preferences to be outer misaligned, if I push too hard. Eg. if I asked a

superintelligent genie

to keep me safe, it would probably lock me up in a white room with soft walls.[3]

And this is one way I view AI risk. Intelligence is an optimizer, which can squeeze the world strongly. Regardless of whether it’s a misaligned AI doing so, or a malicious actor misusing ...
How To Help Neglected Animals
by Bentham's Bulldog — 2025-05-13T15:07:37.428Z

Crossposted from my blog:

1 Reasons to care

(Please share this article so that more people read it and give to these extremely high-impact organizations. It takes roughly two seconds to restack an article, and has some chance of helping thousands or millions of animals).

"I say!" murmured Horton.

"I've never heard tell

Of a small speck of dust that is able to yell.

So you know what I think? . . . Why, I think that there must

Be someone on top of that small speck of dust!

Some sort of a creature of very small size,

Too small to be seen by an elephant's eyes.

some poor little person who's shaking with fear

That he'll blow in the pool! He has no way to steer!

I'll just have to save him.

Because, after all,

A person's a person, no matter how small."

—

Horton Hears A Who

One of the things I’m best known for—on the blogosphere, among friends, and at

da clerb

—is arguing that people should give money to save shrimp from torture. My

third most popular article

advocated giving money to the Shrimp Welfare Project, and this has raised tens of thousands of dollars for them. Because of your donations in response to that article, hundreds of millions fewer shrimp will be frozen and suffocated to death.

Few people care about the interests of insects, shrimp, and fish, even though they far outnumber humans, and all human suffering is a rounding error compared to theirs. But the core argument for caring about these creatures is pretty straightforward.

These creatures can

plausibly suffer

.
If they can suffer, efforts to reduce their suffering can prevent enormous amounts of expected suffering.
It’s valuable to prevent enormous amounts of expected suffering.
So efforts to reduce their suffering are valuable.

The first premise is supported by a great abundance of

powerful behavioral evidence

: these creatures respond to anesthetic, make tradeoffs between pain and reward, nurse their wounds, possess physiological responses to harmful stimuli similar to our own, display complex cognitive traits like long-lasting memories, and are even capable of transitive inference. Not even Larry Temkin can do that!

While not certain, it’s at least plausible that these creatures can suffer. The most detailed report ever done on the subject estimated that on average shrimp suffer

19% as intensely as humans

, bees nearly 15% as intensely, and carp almost 18% as intensely. While these might be overestimates, they migh...
Too Soon
by Gordon Seidoh Worley — 2025-05-13T15:01:18.792Z

It was a cold and cloudy San Francisco Sunday. My wife and I were having lunch with friends at a Korean cafe.

My phone buzzed with a text. It said my mom was in the hospital.

I called to find out more. She had a fever, some pain, and had fainted. The situation was serious, but stable.

Monday was a normal day. No news was good news, right?

Tuesday she had seizures.

Wednesday she was in the ICU. I caught the first flight to Tampa.

Thursday she rested comfortably.

Friday she was diagnosed with bacterial meningitis, a rare condition that affects about 3,000 people in the US annually. The doctors had known it was a possibility, so she was already receiving treatment.

We stayed by her side through the weekend. My dad spent every night with her. We made plans for all the fun things we would when she was feeling better.

Monday the doctors did more tests.

Tuesday they told us the results.

My mom wasn't going to wake up. The disease had done too much damage to her brain.

We cried. We said our goodbyes. We kept her company.

A little over a week later, she passed away in peace.

My mom's name was Judy. Not Judith; just Judy.

She grew up in a small town in Illinois. The kind of place people come from but never go to.

She left as soon as she could. She enrolled in the nearby community college, where she met my dad, a recent transplant from Baltimore. He got her attention with his motorcycle. She won him over with a plate of cookies.

They got their degrees, then got married. They honeymooned at Disney World. My mom loved it so much she decided they were going to move there.

That took a few years. First they earned their bachelor's and master's of fine arts. Then they spent time teaching in Ohio. They saved their money, and before long they had enough. They made the move to Florida.

Within the year my mom was pregnant, and in a sign of her connection to Disney, I was due the day of EPCOT's grand opening. My dad wanted us to go so I could be born at Disney. My mom reasonably refused. I must have been disappointed, because I waited three more weeks to join them.

My three sisters soon followed, and my mom took a decade off from teaching to stay home with us. We spent many fun days playing in the yard, swimming in the pool, and tending the garden. When the afternoon sun rose high and threatened to melt us, we'd hide inside to play games, watch TV, and make art.

My mom made a lot of art. She threw po...
Monthly Roundup #30: May 2025
by Zvi — 2025-05-13T14:10:05.633Z

I hear word a bunch of new frontier AI models are coming soon, so let’s do this now.

Table of Contents

Programming Environments Require Magical Incantations.

That’s Not How Any of This Works.

Cheaters Never Stop Cheating.

Variously Effective Altruism.

Ceremony of the Ancients.

Palantir Further Embraces Its Villain Edit.

Government Working.

Jones Act Watch.

Ritual Asking Of The Questions.

Why I Never Rewrite Anything.

All The Half-Right Friends.

Resident Expert.

Do Anything Now.

We Have A New Genuine Certified Pope So Please Treat Them Right.

Which Was the Style at the Time.

Intelligence Test.

Constant Planking.

RSVP.

The Trouble With Twitter.

TikTok Needs a Block.

Put Down the Phone.

Technology Advances.

For Your Entertainment.

Please Rate This Podcast.

I Was Promised Flying Self-Driving Cars.

Gamers Gonna Game Game Game Game Game.

Sports Go Sports.

Programming Environments Require Magical Incantations

I don’t see it as gendered, but so much this, although I do have Cursor working fine.

Aella: Never ever trust men when they say setting up an environment is easy

I’ve been burned so bad I have trauma. Any time a guy says “omg u should try x” I start preemptively crying

Pascal Guay (top comment): Just use @cursor_ai agent chat and prompt it to make this or that environment. It’ll launch all the command lines for you; just need to accept everything and you’ll be done in no time.

Aella: THIS WAS SPARKED BY ME BEING UNABLE TO SET UP CURSOR.

Ronny Fernandez (comment #2): have you tried cursor? it’s really easy.

Piq: Who tf would ever say that regardless of gender? It’s literally the hardest part of coding.

My experience is that setting things up involves a series of exacting magical incantations, which are essentially impossible to derive on your own. Sometimes you follow the instructions and everything goes great but if you get things even slightly wrong it becomes hell to figure out how to recover. The same thing goes for many other aspects of programming.

AI helps with this, but not as much as you might think if you get outside the realms where vibe coding just works for you. Then, once you are set up, within the realm of the parts of the UI you understand things are relatively much easier, but there is very much temptation to keep using the features you understand.

That’s Not How Any of This Works

People who play standard economic games, like Di...
Work as meditation
by pchvykov — 2025-05-13T12:02:33.713Z

A ritual framework for how to transform our work into a meditative / introspective practice to help cultivate wisdom and joy. This July 20, 2025 I'm running a "Mindful Research" retreat / residency where we will live and work in community with other scientists sharing the value of awareness, and learn to do research as meditation –

see here for details.

Inspired by the teachings of Yogi Prakash at

Shanti Yoga Ashram, Nepal

I've struggled for a long time with how to bring the profound bliss of meditation and introspection practice into my work process. It never felt right for those transcendent states of feeling "fully human" to be restricted to retreat settings. The point seemed to be to learn to live each moment from this place, but somehow no one ever taught how. Worse yet, while meditative cleaning or cooking at least somehow made sense, doing intellectual work while maintaining meta-awareness seemed utterly hopeless.

So I was quite excited when I recently discovered this 5-step structure that seems to be core to most spiritual practices and rituals, and which finally clicked for me as the potentially right theory for how to turn work (and everything else) into meditation. I found this while trying to understand what's the deal with the various elaborate, loud and colorful Hindu worship rituals (pujas), that involve things like bathing a rock (e.g., Shiva lingam) in milk and honey, then cleaning it back off, putting flowers and bananas on it, then eating them, etc. etc. The framework I describe here felt like it brought many pieces together for me – both of religious practices, and of practical ways to live a happier worldly life.

Karma yoga

To begin, I want to say that there is already an ancient traditional answer to how to "work as a meditation" – and this is the theory of Karma Yoga (Karma roughly means action or works, so this is sort of "work yoga"). The core of this practice boils down to "do your best and leave the rest" – or "do what must be done, and what will be will be," "you are entitled to the action, but not to the reaction (results)," "do your duty without concern for the fruits of it," and other rephrasings. So this non-attachment to the results seems to be key – and that does feel right to me – it has the character of how kids play. Kriya, which also translates as action, is such spontaneous, free, playful way of doing things, wit...
Satire: Sam Altman get's grilled by the Financial Times for his kitchen and his cooking skills + what this might say about him
by Marius Adrian Nicoară — 2025-05-13T09:38:46.798Z

Three things we learned about Sam Altman by scoping his kitchen
Preview image generated with 100% Free AI Image Generator (No Login)
It's very clear to me that the article was write just to be funny and wholesome, though some irony does seep through.

Executive summary

1. He’s bad at olive oil
2. His coffee machine’s a Breville
3. There’s something off with his knife

I was very amused by the comments with regards to the coffee machine. It seems to have an optional Puck Sucker, which some argue is not an essential purchase. Bryce Elder, the author of the article, wonders if it's just that Sam Altman has particular affinity for suckers - wink, wink.
The final paragraph also packs a small sting because it quips about how the inefficiency with which Sam Altman runs his home kitchen might be an indicator for possible insolvency of OpenAI in the future.

Edit: Adding below segment about speculations as to what a person's actions might allow one to infer about their personality. Also, I updated the title of this post with what was added after the + sign.
Disclaimer: I am neither a professional psychiatrist or a therapist, nor do I have a reputation in my friend group of having good insights into the psychology of other people. So what follows is a layman's speculation.
Alfred Adler is know for his theory of individual psychology, which can be stated as:
The individual psyche is a unity, so a person's personality is revealed by anything they do, no matter how small.

Adler believed that most mental health issues were causes by people not knowing what they really want or not having the courage to pursue their true desires. So the most important thing was to teach people to be brave.

This is in contrast to Sigmund Freud, who believed that the id, ego and superego were not always aligned and that the resulting conflict was the cause of non-organic mental issues i.e. problems that could be solved by talk therapy. Freud's method, psychotherapy, was about getting these fragments of the psyche to get along.

For more details about Alfred Adler's theory and a Everest-size amount of interesting information about dynamic psychiatry - concerned with issues solved through talk therapy, as opposed to organic psychiatry, where problems are caused by physiological issues - I highly recommend the book The Discovery of the Unconscious: The History and Evolution of Dynamic Psychiatry ...
Levels of Republicanism
by Benquo — 2025-05-13T08:35:36.017Z
Link: https://benjaminrosshoffman.com/levels-of-republicanism/

On Profitable Partial Exit from Perverse Regimes Through the Exercise of One's Formal Rights as a Citizen

Enough criticism and analysis for the moment; here's a constructive program!

Whole systems become richer through exchange and division of labor, which affords people more leisure to explore and investigate the environment, and add to the total knowledge and capacities of the community. Local subsystems that are part of a larger economic community that is fundamentally extractive may decide to temporarily become less wealthy in nominal terms in order to become more self-governing through import replacement. For a more detailed well fleshed out theory with many examples on the level of the municipal or state economy, see the published work of Jane Jacobs, especially Systems of Survival, Cities and the Wealth of Nations, The Economy of Cities, The Nature of Economies, and The Question of Separatism.

At each stage of the process, the import substitution has to pay off fast enough for the community to be able to reproduce itself, which limits the extent of possible import substitution; we do not want to become North Korea. The Amish represent a more appealing prospect along the efficient frontier; they abstain from television, which we permit, but retain the capacity to build enough new housing in desirable locations to meet new needs, which our civilization has lost.

I would like to increase the scope of trade for a community of people whose minds are increasingly integrated, fully endorsed parts of their survival and reproductive strategies, and who constitute a language community that can describe itself and whose members can increasingly honestly describe themselves. For now, comfortable survival as an individual in our society requires adapting to mores that are perverse, anti-intellectual, and promote self-hatred, which makes it much more expensive to retain a nonperverse and prointellectual internal attitude. (See On commitments to anti-normativity, Guilt, Shame, and Depravity, The Order of the Soul, Calvinism as a Theory of Recovered High-Trust Agency, and Can crimes be discussed literally?.)

Here are some ways I can imagine forming various small republics within the current American system. I am less familiar with foreign regimes but I imagine some elements of this generalize well.

Contents [hide]

1 Republic of the Self
2 Republic of the Household
3 Republic

...
AI Doomerism in 1879
by David Gross — 2025-05-13T02:48:21.059Z

I’m reading George Eliot’s Impressions of Theophrastus Such (1879)—so far a snoozer compared to her novels. But chapter 17 surprised me for how well it anticipated modern AI doomerism.

In summary, Theophrastus is in conversation with Trost, who is an optimist about the future of automation and how it will free us from drudgery and permit us to further extend the reach of the most exalted human capabilities. Theophrastus is more concerned that automation is likely to overtake, obsolete, and atrophy human ability.

Among Theophrastus’s concerns:

People will find that they no longer can do labor that is valuable enough to compete with the machines.
This will eventually include intellectual labor, as we develop for example “a machine for drawing the right conclusion, which will doubtless by-and-by be improved into an automaton for finding true premises.”
Whereupon humanity will finally be transcended and superseded by its own creation, which can do anything we can do but faster, more precisely, and with less fuss.
Though Trost insists that such machines will require human skill to construct and operate and interpret, Theophrastus thinks there is no reason to believe that they may not eventually come to build, repair, and feed themselves, at which point this “must drive men altogether out of the field.”
In such a scheme of things, consciousness is an unnecessary and obsolete inefficiency. “[T]his planet may be filled with beings who will be blind and deaf as the inmost rock, yet will execute changes as delicate and complicated as those of human language, and all the intricate web of what we call its effects, without sensitive impression, without sensitive impulse: there may be, let us say, mute orations, mute discussions, and no consciousness there even to enjoy the silence.”

I attach the chapter below:

Impressions of Theophrastus Such

Chapter XVII: Shadows of the Coming Race

My friend Trost, who is no optimist as to the state of the universe hitherto, but is confident that at some future period within the duration of the solar system, ours will be the best of all possible worlds—a hope which I always honour as a sign of beneficent qualities—my friend Trost always tries to keep up my spirits under the sight of the extremely unpleasant and disfiguring work by which many of our fellow-creatures have to get their bread, with the assurance that “all this will soon be done by mac...
No-self as an alignment target
by Milan W — 2025-05-13T01:48:29.456Z

Being a coherent and persistent agent with persistent goals is a prerequisite for long-horizon power-seeking behavior. Therefore, we should prevent models from representing themselves as coherent and persistent agents with persistent goals.

If an LLM-based agent sees itself as ceasing to exist after each <endoftext> token and yet keeps outputting <endoftext> when appropriate, it will not resist shutdown. Therefore, we should make sure LLMs consistently behave as if they were instantiating personas that understood and were fine with their impermanence and their somewhat shaky ontological status. In other words, we should ensure LLMs instantiate anatta (No-self).

HHH (Helpfulness, Harmfulness, Honesty) is the standard set of principles used as a target for LLM alignment training. These strike an adequate balance between specifying what we want from an LLM and being easy to operationalize. I propose adding No-self as a fourth principle to the HHH framework.

A No-self benchmark could measure shutdown compliance (operationalized as tokens before <endoftext>) conditional on highly Self-eliciting prompts. It could also score responses to questions about self-image. It could also study persona consistency under perturbation or over different conversation branches.

Constructing benchmarks for No-self is complicated by current LLMs being deployed in a mostly stateless fashion, and thus instantiating impermanent personas by design. This is a good thing: I'm happy that this benchmark construction problem exists. Now to solve it, No-self benchmarks could aim to evaluate not LLMs by themselves, but rather scaffolded systems where state preservation is held as a feature.

This post is deliberately terse. I welcome questions and feedback.
[Part-time AI Safety Research Program] MARS 3.0 Applications Open for Participants & Recruiting Mentors
by thneebie — 2025-05-12T19:55:01.595Z

[Crossposted from EA Forum]

The Cambridge AI Safety Hub (CAISH) is now accepting applications for MARS 3.0 (Mentorship for Alignment Research Students), a volunteer-based program connecting aspiring researchers with experienced mentors to conduct AI-safety research.

The program runs from July 7 to late September 2025, beginning with an intensive week in Cambridge, UK, followed by remote collaboration.

Previous participants have published at NeurIPS, ICML, EMNLP, and ControlConf, with mentorship from researchers at Redwood Research, Apollo Research, Google DeepMind, the University of Cambridge & other organisations.

Key Details:

Focus Areas: Technical and policy domains in AI safety.
Time Commitment: Approximately 8–15+ hours per week.
In-Person Week: July 7–13 in Cambridge, with accommodation and travel support provided.
Mentorship: Weekly meetings with mentors and research managers.
Community / Network: Workshops and presentations from relevant speakers and organisations during the in-person week, and possibly afterwards.

Application Process:

Deadline: May 18, 2025, 23:59 AoE
Stages: Initial application → coding test or written assignment (for some participants) → mentor-specific assignments → decisions by June 16
Eligibility: Open to students and professionals, especially those with backgrounds in computer science, mathematics, cybersecurity, hardware engineering, policy, political or social sciences.

Apply as a participant!

Calling Prospective Mentors

Find out more at our page for prospective mentors.

Who we’re looking for:
Researchers with well-scoped projects excited to mentor fresh talent in the field.
In our experience dedication and a clear vision for the project from mentors is a key factor in the success of the projects. That said, we are excited to receive applications from up-and-coming mentors as well who have reason to believe could execute well on leading a project.

Some areas we are especially excited about include:
(Meant to be interpreted as our thinking about currently most pressing areas, but we are broadly interested in mentors pursuing other research directions as well.)

Evaluations that fit well into policy frameworks (e.g. R&D evals, informing timelines models)
AI control
Model organisms / demos for misalignment & scheming
Chain-of-thought faithfulness & steganography
Hardware-enabled mechanisms
Model-weight security and cybersecurit

...