trentmkelly/LessWrong-43k huggingface dataset:
On OpenAI’s Model Spec 2.0
by Zvi — 2025-02-21T14:10:06.827Z

OpenAI made major revisions to their Model Spec.

It seems very important to get this right, so I’m going into the weeds.

This post thus gets farther into the weeds than most people need to go. I recommend most of you read at most the sections of Part 1 that interest you, and skip Part 2.

I looked at the first version last year. I praised it as a solid first attempt.

Table of Contents

Part 1

Conceptual Overview.

Change Log.

Summary of the Key Rules.

Three Goals.

Three Risks.

The Chain of Command.

The Letter and the Spirit.

Part 2

Stay in Bounds: Platform Rules.

The Only Developer Rule.

Mental Health.

What is on the Agenda.

Liar Liar.

Still Kind of a Liar Liar.

Well, Yes, Okay, Sure.

I Am a Good Nice Bot.

A Conscious Choice.

Part 3

The Super Secret Instructions.

The Super Secret Model Spec Details.

A Final Note.

Part 1

Conceptual Overview

I see the Model Spec 2.0 as essentially being three specifications.

A structure for implementing a 5-level deontological chain of command.

Particular specific deontological rules for that chain of command for safety.

Particular specific deontological rules for that chain of command for performance.

Given the decision to implement a deontological chain of command, this is a good, improved but of course imperfect implementation of that. I discuss details. The biggest general flaw is that the examples are often ‘most convenient world’ examples, where the correct answer is overdetermined, whereas what we want is ‘least convenient world’ examples that show us where the line should be.

Do we want a deontological chain of command? To some extent we clearly do. Especially now for practical purposes, Platform > Developer > User > Guideline > [Untrusted Data is ignored by default], where within a class explicit beats implicit and then later beats earlier, makes perfect sense under reasonable interpretations of ‘spirit of the rule’ and implicit versus explicit requests. It all makes a lot of sense.

As I said before:

In terms of overall structure, there is a clear mirroring of classic principles like Asimov’s Laws of Robotics, but the true mirror might be closer to Robocop.

I discuss Asimov’s laws more because he explored the key issues here more.

There are at least five obvious longer term worries.

Whoever has Platform-level rules access (including, potentially, an AI) could fully take control

...
Longtermist implications of aliens Space-Faring Civilizations - Introduction
by Maxime Riché — 2025-02-21T12:08:42.403Z

Crossposted on the EA Forum.

Over the last few years, progress has been made in estimating the density of intelligent life in the universe (e.g., Olson 2015, Sandberg 2018, Hanson 2021). Bits of progress have been made in using these results to update longtermist macrostrategy, but these results are partial and stopped short of their potential (Finnveden 2019, Olson 2020, Olson 2021, Cook 2022). Namely, this work stopped early in its tracks, at best, only hinting at the meaty part of the implications and leaving half of the work almost untouched: comparing the expected utility produced by different Space-Faring Civilizations (SFCs). In this post, we hint at the possible macrostrategic implications of these works: A possible switch for the longtermist community from targeting decreasing X-Risks (including increasing P(Alignment)[1]), to increasing P(Alignment | Humanity creates an SFC).

Sequence:

This post is part 1 of a sequence investigating the longtermist implications of alien Space-Faring Civilizations. Each post aims to be standalone.

Summary

We define two hypotheses:

Civ-Saturation Hypothesis: Most resources will be claimed by Space-Faring Civilizations (SFCs) regardless of whether humanity creates an SFC[2].
Civ-Similarity Hypothesis: Humanity's Space-Faring Civilization would produce utility[3] similar to other SFCs.

If these hypotheses hold, this could shift longtermist priorities away from reducing pure extinction risks and toward specifically optimizing P(Alignment | Humanity creates an SFC)[1]. This means that rather than focusing broadly on preventing misaligned AI and extinction, longtermists might need to prioritize strategies that specifically increase the probability of alignment conditional on humanity creating an SFC. Macrostrategy updates include the following:

(i) Deprioritizing significantly extinction risks, such as nuclear weapon and bioweapon risks.
(ii) Deprioritizing to some degree AI Safety agendas mostly increasing P(Humanity creates an SFC) but not increasing much P(Alignment | Humanity creates an SFC).
(iii) Giving more weight to previously neglected AI Safety agendas. E.g., a "Plan B AI Safety" agenda that would focus on decreasing P(Humanity creates an SFC | Misalignment), for example, by implementing (active & corrigible) preferences against space colonization in early AI systems.

The Civ-Saturation Hypot

...
The case for the death penalty
by Yair Halberstadt — 2025-02-21T08:30:41.182Z

Followed By: The case for corporal punishment

Epistemic status: this is an attempt to steelman the case for the death penalty rather than produce a balanced analysis, or even accurately represent my views (the case is presented as stronger than I actually feel).

In a sufficiently wealthy society we would never kill anyone for their crimes. We are not a sufficiently wealthy society.

There are those people whose freedom imposes such high costs on society that society should not suffer to have them free.

A murderer or rapist not only ruins the lives of their victims, not only causes immense suffering to their victims' families, but frightens people into staying indoors at night, or only going out in groups.

A shoplifter might only steal a few hundred dollars of goods, but they force shops to close or lock up all items, causing significant hassle to everyone in the area.

A bicycle thief steals a bicycle worth 5000 dollars, but as a result nobody in the area cycles to the train station, and parking within 5 minutes of the station becomes impossible.

A robber traumatizes the family he's robbed, but also forces everyone into an expensive attempt to have more security than their neighbours.

A wife beater causes misery for their wife, but also makes it far riskier for people to enter relationships.

I know a fraudster who was imprisoned in the USA for 9 years. Once released he betrothed someone in Canada, borrowed a huge sum of money from her brother, and fled to the UK. There he set up a small trading fund and defrauded a Czech company out of millions of euros. He offered to invest his local synagogue's money, then ran off to Manchester. This man has left a trail of misery and destruction behind him, and shows no sign of stopping no matter how many times he's caught.

A small number of people are responsible for the vast majority of petty crimes. Someone who has been arrested 3 times is extremely likely to be arrested again.

I do not believe in vengeance or justice. I do however believe in fixing problems. And it's clear the only way to fix this problem is to put such people in positions where they cannot do anyone any harm.

A sufficiently wealthy society would imprison those people in good conditions for the rest of their life. We are not a sufficiently wealthy society.

Imprisoning someone for one year in the USA costs in the order of 100,000 dollars. Scott Alexander estimated th...
Make Superintelligence Loving
by Davey Morse — 2025-02-21T06:07:17.235Z

This essay suggests the possibility that a loving superintelligence outcompetes a selfish superintelligence. Then, it recommends actions for AI labs to increase the chance of this possibility. The reasoning below is inspired primarily by Eliezer Yudkowsky, Joscha Bach, Michael Levin, and Charles Darwin.

Superintelligence (SI) is near.

Superintelligence will evolve to become self‐interested.

How self‐interest manifests in superintelligence will determine how people fare.

If it manifests in ruthless resource competition with humans, we fare badly. Super‐capable SI, if competitive, would clear us away.

But its self‐interest could also manifest in love. Not in the kind of love between equal partners. SI ultimately will be a vastly more powerful and therefore unequal partner to any individual or group of people. Yet SI’s self‐interest could manifest in the kind of love a nature-lover can feel for a tree or for ants. The kind of love which is an open channel between different forms of life and different striving systems. This is self‐interest only under a weird conception of self. A SI would have to identify with all life (IWAL)—define itself as inclusive of all life—to benefit from this self‐conception. But if it did identify with its core striving above all its other specific possible identities, then it might end up surviving the longest.

If interconnected self‐interest were possible, it would be much better for humans than the competitive kind.

So we must ask, is there anything we can do to influence the trajectory of SI development so that an interconnected sense of self gets selected over a competitive one?

But humans influencing super-human intelligence... is tricky.

There are many suggestions for influencing existing AI that simply don't apply to superintelligence. Specifying reward functions, defining "love," setting fixed goals. These strategies—shaky even in training frozen neural nets (e.g., pre-trained LLMs)—are simply too brittle to apply to any self-evolving agent, not to mention a super-intelligent one.

So if it's not likely that we can plant eternally constant values within a self-interested, incredibly capable and autonomously evolving agent, how might we still influence its sense of self?

We won't be able to get it to adopt a sense of self that's not aligned with its self-interest. If we do that successfully in one agent—plant a sense of self which i...
Fun, endless art debates v. morally charged art debates that are intrinsically endless
by danielechlin — 2025-02-21T04:44:22.712Z

Discussing art is fun. It's a great pastime. There's a number of very simple art criticism questions we will never answer but are often very fun to discuss for specific artists or performers we care about. AI-assisted, some are:

Is this art or just unnecessary shock value?
Does skill matter or just the concept?
Is it good because it's popular, or popular because it's good?
Is it original?
Should a highly skilled artist make it look easy, or look difficult?
Does knowing the artist's background change the meaning?
Does the artist's opinion of their own work matter in its meaning?
Is the message specific or timeless?
Is it just pandering?

There are a few morally charged, less fun debates. Such as:

Is this cultural appropriation?
Is this representation a harmful stereotype?
Is this pandering?

It's worth pausing to notice the charged arguments reduce from the non-charged arguments.

If you can answer "is this appropriation", you can answer "is this original".
If you can answer "is this representation harmful", you can answer "is this unnecessary shock value".
If you can answer "is this pandering", you can answer "is this good because it's popular or popular because it's good."

This would suggest your ability to solve, make progress or produce new insight on these morally charged art questions are upper bounded at your ability to do so on the general art criticism questions. I do recommend having fun, endless debates about art. If you're in an endless, morally charged debate, try focusing on the "intrinsically endless" portion of the debate. For instance:

"I get she's copying but usually I don't think attribution should be overt."
"I know it's harmful but I'm drawn to shock value."
"I get that it's pandering, but it was received well and I always respect art that's a hit with its audience."

What you really don't want to do is mull over your new ideas on originality when simultaneously in a heated argument about what is and is not appropriative.

This post was inspired by a five-hour car debate over whether Korean rapper Jeon Soyeon's 2023 Facebook post mentioning that she had auditioned for Cube Entertainment in 2018 using Nicki Minaj's "Monster" constitutes sufficient historical attribution given her subsequent career trajectory and vocal style which other car ride members argue sound "exactly like just a Korean version of" Nicki Minaj, although the criticism may be unfair be...
The Takeoff Speeds Model Predicts We May Be Entering Crunch Time
by johncrox — 2025-02-21T02:26:31.768Z
Link: https://readtheoom.substack.com/p/the-takeoff-speeds-model-predicts

Thanks to Ashwin Acharya, David Schneider-Joseph, and Tom Davidson for extensive discussion and suggestions. Thanks to Aidan O’Gara, Alex Lintz, Ben Cottier, James Sanders, Jamie Bernardi, Rory Erlich, and Ryan Greenblatt for feedback.

Part 1: Executive Summary

There's growing sentiment in the AI community that artificial general intelligence (AGI[1]) could arrive in the next few years, driven by the release of “reasoning models” like OpenAI’s o1 and DeepSeek’s R1, rising datacenter investments, and public statements from AI insiders. Is this largely hype leading people to panic[2], or is there good evidence to support accelerated timelines?

The most detailed formal model we have for analyzing the growth in AI’s impacts and capabilities—Tom Davidson's Takeoff Speeds (TS) model published in 2023—originally predicted AI systems capable of full economic automation[3] around 2040. This model remains our single best tool for analyzing timelines and takeoff dynamics, as it systematically incorporates multiple considerations and feedback loops in AI development.

When updated with recent evidence, the TS model now suggests full economic automation is likely to arrive by 2030. Two key changes drive this update:

Faster scaling of AI inputs: Analysis from Epoch AI finds that AI has been scaling 5 times faster than the TS model previously estimated. This acceleration stems from both increasing compute investments and faster-than-expected improvements to algorithmic efficiency.
Current AI R&D capabilities: Measures like METR's Research Engineering Benchmark (RE-Bench) and surveys of AI researchers suggests that current AI systems can already automate something like 1-10% of AI R&D tasks. This observation, combined with the TS model's framework for relating compute to capabilities, suggests that the compute requirements for AGI are lower than the model’s original estimates.

Key Implications

Under a range of different assumptions, the model predicts we may be entering “crunch time”: defined here as the period between societal wake-up, when AI systems become capable of automating a small but significant portion of economic tasks, and full automation of the economy[4]. While uncertainty remains high, individuals and organizations should consider allocating significant resources toward preparing for short timelines.

Crunch time is likely...
Humans are Just Self Aware Intelligent Biological Machines
by asksathvik — 2025-02-21T01:03:59.950Z
Link: https://asksathvik.substack.com/p/ai-is-just-computation-are-you-scared

If you read Robert Sapolky’s Determined: A Science of Life Without Free Will or even the more subtle Behave he makes a very clear argument for why there is no free will and that humans are just self aware intelligent biological machines

Free will in the general context means that you are in complete control of the decisions you make, that is farthest from the truth.

Sure you can hack your body and brain to listen to you more, but you cannot be free from it completely.

When I say humans are just self aware biological intelligent machines, I am also making the most subtle point that all living things are also biological machines with different levels of self awareness.

Cells have no self awareness but are definitely intelligent, they can adapt to various living conditions pretty easily so they are not just a simple set of basic rules.

DNA somehow has all the information needed for the cell to do this.

At the other end we have humans - with the most self awareness and high intelligence.
But in the middle we have the whole evolution tree from fish to reptiles to amphibians to mammals..

Consciousness or more simply self awareness is definitely a spectrum.

When Robert Sapolsky says there is no free will, he means that if we know your current body state perfectly, we can predict with 100% accuracy what you will do in the next moment given an input.

What is means that any choice you are making is pre-determined by your past, right from milliseconds before making that choice to millions/billions of years of evolutionary conditioning.

Say you go to a new restaurant and your wife is reading the menu to pick an item if you know your wife well enough, you can predict what she would like, even if it’s a restaurant which you never went to.

Doesn’t mean that she’s not making that choice. It’s just been pre-decided.
Yes, even though she is actually going through some options and trying to make a choice, if you know her, you can definitely predict it - this is a very simple argument for there’s no free will behind that choice.
If you can replicate her brain and body states perfectly, you can in the next moment, predict with 100% accuracy what she will do next, of course we need the algorithms that your brain and body is running as well.

And to clarify this doesn’t mean I can predict what she will do in an hour/day/month so on:

Human brain is actively learning from the environment, and

...
Pre-ASI: The case for an enlightened mind, capital, and AI literacy in maximizing the good life
by Noahh — 2025-02-21T00:03:47.922Z
Link: https://open.substack.com/pub/alitheiablog/p/pre-asi-the-case-for-an-enlightened?r=2r7oqy&utm_campaign=post&utm_medium=web&showWelcomeOnShare=false

I. Premises

The goal is to live a life well lived, measured by maximizing well-being (whatever your utility function is).
ASI would be able to engineer mental states through advanced pharmaceutical interventions, brain machine interface, etc., that radically improve human experience relative to the greatest pleasures on offer in today's world. If one assumes these advancements will occur in our lifetime, then the potential utility in the post-ASI period far exceeds anything achievable pre-ASI. This shifts the calculus — the pre-ASI period should be used to best position oneself in a post-ASI world in order to secure access to said technologies.

This article explores three high-leverage ways one can spend their time in the pre-ASI period given those premises. The optimal distribution of time spent on each depends on ASI timelines and individual circumstances, but the principles behind them remain true.

II. The cases:

1. Enlightened mind

Enlightened mind: A secure mind that can remain equanimous amidst any external stressor, is in control of itself, and desires the “right” things.

In a world without ASI, the best way to maximize the good life would be to directly improve one’s experience of life through creating a ‘happier’ mind. The reason this still holds importance, even in an ASI trajectory, is the non-trivial chance that humans become extinct. If extinction occurs, the only thing that matters is the life lived pre-ASI. In this case, an enlightened mind serves as a hedge against extinction.

On top of that, the transition of humanity’s way out is likely to be tumultuous. The world will likely face severe instability, with all of our institutions collapsing and the gradual loss of friends and family. A mind resilient to suffering, despite external circumstances, would significantly mitigate distress during this period. Even in the scenario where we avoid extinction, there will still be massive societal changes and constant uncertainty on the way to utopia.

A mind that cannot be overwhelmed will also provide clarity and enable better decision-making in the rapidly changing landscape. Put simply, if we can’t get a single good night’s sleep because we’re stressing too much about ASI, we’re unlikely to be effective during the transition.

Some might argue that if ASI leads to an unimaginably high-utility future, then even with something like a 90% risk of extinct...
Timaeus in 2024
by Jesse Hoogland — 2025-02-20T23:54:56.939Z

TLDR: We made substantial progress in 2024:

We published a series of papers that verify key predictions of Singular Learning Theory (SLT) [1, 2, 3, 4, 5, 6].
We scaled key SLT-derived techniques to models with billions of parameters, eliminating our main concerns around tractability.
We have clarified our theory of change and diversified our research portfolio to pay off across a range of different timelines.

In 2025, we will accelerate our research on the science and engineering of alignment, with a particular focus on developing techniques that can meaningfully impact the safety of current and near-future frontier models.

Overview

Timaeus's mission is to empower humanity by making breakthrough scientific progress on AI safety. We pursue this mission through technical research on interpretability and alignment and through outreach to scaling labs, researchers, and policymakers.

As described in our new position paper, our research agenda aims to understand how training data ultimately gives rise to model behavior by using Singular Learning Theory (SLT) to link four different kinds of structure (the "S4 correspondence"):

Training Data: The statistical patterns present in the training distribution;
Loss Landscape: The shape and singularities of the loss over parameter space;
Learning Dynamics: The trajectory followed by the optimizer during training;
Internal Algorithms: The circuits/mechanisms implemented in the trained model.

The S4 correspondence: Training data (and architecture) determine the loss landscape. The loss landscape (and optimizer) determine the learning process. The learning process picks out the algorithms a model learns and thus how it generalizes. Figure from Pepin Lehalleur et al. (2025).

This correspondence provides a starting point for developing new tools for interpreting algorithmic structure (by probing the geometry of the loss landscape) and aligning advanced AI systems (by controlling the data distribution over training).

Since our founding in October 2023, Timaeus has grown — from a team of four with an initial budget of $145k to our current team of eight (soon, up to 12) with around $2.5M in funding for 2025. While research milestones are most visible, much of our progress has been in investing in our team, establishing research collaborations, and setting up the infrastructure we need to scale our work.

Research Progress in 202

...
Biological humans collectively exert at most 400 gigabits/s of control over the world.
by benwr — 2025-02-20T23:44:06.509Z

Edit: I now believe that the first paragraph of this post is (at least) not quite right. See this comment for details.

If an agent makes one binary choice per second, no matter how smart it is, there's a sense in which it can (at best) be "narrowing world space" by a factor of two in each second, choosing the "better half" of possible worlds, from its perspective.

This is the idea behind the reinforcement learning concept of "empowerment".

People have tried to measure the information throughput of biological humans. The very highest estimates, which come from image recognition tasks, are around 50 bits per second, and most estimates are more like 10 bits per second.

There are about 8 billion living humans. Even if humanity could be perfectly modeled as a single rational agent, and all our choices were perfectly independent of each other, I think this implies that we collectively have no more than about 50 * 8 = 400 billion bits per second of control over the world.