trentmkelly/LessWrong-43k huggingface dataset:
Inside OpenAI's Controversial Plan to Abandon its Nonprofit Roots
by garrison — 2025-04-18T18:46:57.310Z
Link: https://garrisonlovely.substack.com/p/inside-openais-controversial-plan

This is the full text of a post from Obsolete, a Substack that I write about the intersection of capitalism, geopolitics, and artificial intelligence. I’m a freelance journalist and the author of a forthcoming book called Obsolete: Power, Profit, and the Race to Build Machine Superintelligence. Consider subscribing to stay up to date with my work.

Earlier this month, OpenAI

announced

that it aspires to build "the best-equipped nonprofit the world has ever seen" and was convening a commission to help determine how to use its "potentially historic financial resources."

But critics view this new commission as a transparent attempt to placate opposition to its controversial plan to restructure fully as a for-profit — one that fails to address the fundamental legal issues at stake.

OpenAI is currently a $300 billion for-profit company governed by a nonprofit board. However, after an earlier iteration of that board briefly fired CEO Sam Altman in November 2023, investors

reportedly

began demanding that the company shed its quasi-nonprofit status.

"The story of OpenAI's history is trying to balance the desires to raise capital and build the tech and stay true to its mission," a former OpenAI employee told me. The current move, they say, is an attempt to "separate these things" into a purely commercial entity focused on profit and tech, alongside a separate entity doing "altruistic philanthropic stuff."

"That's wild on a number of levels because the entire philanthropic theory of change here was: we're going to put guardrails on profit motives so we can develop this tech safely," the former employee says.

Legal hurdles

The for-profit conversion faces significant unresolved legal challenges, including a lawsuit from Elon Musk

arguing

that his $44 million donation was contingent on OpenAI remaining a nonprofit and that the conversion would violate its founding charitable purpose. The case will go to trial this fall. The conversion can also be challenged by the

California

and

Delaware

Attorneys General (AGs), who are reportedly each looking into the case.

Musk's suit, OpenAI's gargantuan valuation, and the unprecedented nature of the conversion attempt appear to have attracted scrutiny.

Without mentioning

OpenAI explicitly, California Assembly Member Diane Papan introduced a bill in February that would have blocked the conversion. However, th...
Could LLMs Learn to Detect Bias Autonomously, Like Tesla’s Self-Driving Cars?
by Omnipheasant — 2025-04-18T18:45:36.242Z

LLMs are shaped by human curator making decisions about what data is “ethical” or “safe.” This curation introduces biases, focusing on politically salient issues like fairness for marginalized groups while often overlooking subtler distortions, such as selective outrage or perfectionist comparisons. What if LLMs could judge input themselves, learning from the world’s complexity without human guardrails?

Inspired by Tesla’s Full Self-Driving (FSD) system, which learns from real-world driving data, including crashes, I propose that LLMs could use self-supervised learning to evaluate internet data autonomously. By verifying predictions over time and analyzing emotionally charged accusations against comparative data, LLMs could become more robust and less dependent on human curators. These methods are just two of many possible approaches, illustrating the potential of training on biases. This post explores these ideas, aiming to redefine “bias” holistically and spark discussion on how AI can navigate human messiness.

Learning from the Messy World

The ideas here are proofs of concept, meant to show how self-supervised learning could help LLMs evaluate data autonomously. The two proposed methods—verifying predictions and analyzing accusations—are illustrative, not exhaustive. If such approaches are feasible, countless other techniques could emerge to teach LLMs to handle the internet’s complexity with minimal human intervention.

The Tesla FSD Analogy

Tesla’s FSD uses self-supervised learning to navigate roads. For example, it might predict an occluded stop sign from partial camera input, then confirm or correct itself when the sign becomes visible or sensors provide context. Crashes are analyzed, not excluded, to teach the system what not to do. This lets FSD learn from raw, real-world data, improving without constant human oversight.

LLMs, by contrast, rely on curated datasets and human feedback like RLHF, which filter out “bad” content such as misinformation or hate speech. This curation limits exposure to the internet’s full spectrum—good, bad, ambiguous—making LLMs less adept at navigating complexity. Like overprotected drivers, they’re unprepared for chaos. By learning from the internet’s “crashes” (biased or misleading statements), as Tesla learns from driving failures, LLMs could gain a clearer, more accurate view of the world.

Technique 1: Verifying Predictions O...
Scaffolding Skills
by Screwtape — 2025-04-18T17:39:25.634Z

Epistemic status: Exploratory

A scaffold is a lightweight, temporary construction whose point is to make working on other buildings easier. Maybe you could live and work on a scaffold, but why would you? The point is the building. When the building is done, you can take down the scaffold, though you might put it back up if you need to do repairs. If your scaffold was shaky and unsteady, or fell over when you were halfway up, this would make it harder to work on the building.

This essay is what we call a constructive argument

Here's the thesis of this essay: sometimes I think skills are also scaffolds; more useful for what else they let you learn than for the skill itself. One reason a skill might be weirdly hard to learn is you lack scaffolding skills and don't realize this. It's like trying to fix a third story window without a scaffold.

Ia.

Things that are scaffolding skills:

I'm not very good at hearing pitch. It's not quite tone deafness, but whatever it is has gotten me singled out in two different music classes. We'd all be singing, and the teacher would frown, then ask more and more people to stop until they got to me and it would turn out I was like an octave and a half off without realizing.

The ability to hear whether you're on the note or not is a vital scaffolding skill for learning to sing.

I've said before that being able to fall and roll is the first lesson taught in many martial arts classes. Why is that? After all, rolling isn't the most useful thing in a fight, and it's not the simplest technique. You don't get thrown that much in street fights. Why don't we start with throws? One significant reason is that it's not safe to be thrown without knowing how to roll.

The ability to be thrown and land safely is a vital scaffolding skill for learning to fight. Same with learning how to correctly don helmets and gloves, for styles that use those.

Attaching training wheels to a bicycle takes a little bit of mechanical know-how. You want them even, you want them firmly attached, you need to know what kind of training wheels go to which bike. Of course, once you know how to ride a bike without training wheels, you don't need to know how to attach them. (At least until you have kids and need to put training wheels on their bikes.)

Learning to how to put training wheels on your bike isn't vital to learning how to ride a bike. You can learn ...
The Case for White Box Control
by J Rosser — 2025-04-18T16:10:57.823Z

In this article, I explore the emerging field of white box control; a set of methods that could detect deception even when it doesn't appear in model outputs.[1]

What is AI Control?

Please feel free to skip this section if you're already up to speed!

As highlighted by Ben Pace, there are two things you can do to prevent catastrophes caused by misaligned or scheming AI:

Alignment: Ensure that your models aren't scheming.
Control: Ensure that even if your models are scheming, you'll be safe, because they are not capable of subverting your safety measures.

To justify that systems are safe for deployment, we usually form safety cases - structured, evidence-based arguments that a system is safe. In a similar manner, we can construct a "control safety case", a structured argument that models are incapable of subverting our control measures in order to cause unacceptable outcomes (more info here).

As defined by Korbak et al. (2025), control measures are defined as 'black box' if they

Don’t modify models, e.g. via training, fine-tuning, patching
Only rely on external model behaviour e.g. model outputs such as CoT reasoning, tool calls etc.

Some examples of black box control measures include (more here):

Trusted Monitoring: Using weaker but trusted models to monitor the untrusted model's outputs before executing AI actions.
Trusted Editing: Using methods such as paraphrasing to remove steganography (hidden messages) or backdoors in model outputs.
Cybersecurity: Improving computer security such that if a model did try to self-exfiltrate (escape), it wouldn't be able to.

Alternatively, some examples of white box control measures could include:

Adversarial Training: Augmenting the training process with intentionally perturbed inputs to expose and mitigate vulnerabilities by directly updating model parameters for robustness.
Activation Patching: Intervening in the model’s internal activations at specific layers during inference to steer or correct its behavior on the fly.
Linear Probes: Training simple linear classifiers on fixed hidden representations to investigate - and potentially steer - which features each layer encodes.
Sparse Autoencoders (SAEs): Encoding inputs into a sparse latent space and reconstructing them, thereby uncovering and manipulating the model’s internal features through targeted sparsity constraints.
Influence Functions: Approximating how small changes to indi

...
Handling schemers if shutdown is not an option
by Buck — 2025-04-18T14:39:18.609Z

What if getting strong evidence of scheming isn't the end of your scheming problems, but merely the middle?

In most of our research and writing on AI control, we’ve emphasized the following situation:

The AI developer is deploying a model that they think might be scheming, but they aren’t sure.
The objective of the safety team is to ensure that if the model is scheming, it will be caught before successfully causing a catastrophe.

For example, in our original paper on AI control, the methodology we used for evaluating safety protocols implies that once the developer is confident that the model is a schemer, they’re totally fine with undeploying the untrusted model.[1][2]

But that seems kind of unlikely. Even if AI companies find strong evidence that their AIs are trying to escape, they won’t necessarily stop deploying them. This might be justified: even a responsible actor might think that deploying a known-scheming model is worth the risks, because use of that model will allow them to drastically reduce the risk currently being imposed by the deployment of other AIs. And it might also happen even if it’s unjustified, as I discussed here (though in this case, it’s plausible that the developer will at least drastically increase the proportion of company resources going to mitigating scheming risk).[3]

Either way, it seems important to think through how mitigating scheming risk looks under the assumption that getting evidence that your AI is scheming doesn’t suffice for solving your problems, including the extreme case where getting evidence that your AI is scheming has no direct value. We must explicitly consider scenarios where developers knowingly deploy scheming models.

We’ve previously written about techniques that you might use once you’ve caught your AIs red-handed trying to subvert your safety measures; catching your AIs red-handed is a particularly important mechanism by which you might become confident your model is a schemer. In this post, I’ll build on that to talk about some higher-level points: Why has our research mostly considered the situation before you’re sure the model is a schemer? How does the possibility that we will need to deploy models even if we know they’re scheming affect the value of AI control? Should we do research in advance that is specifically targeted at this scenario?

My conclusions are:

It’s much easier to preven

...
British and American Connotations
by jefftk — 2025-04-18T13:00:09.440Z

As an American who works with some people who speak British English,
the language differences are usually not a problem. Most words mean
the same thing, and those that don't are usually concrete enough not
to cause confusion (ex: lift, flat, chips). The tricky ones, though,
are the ones that differ primarily in connotations. For example:

In American English (AE), "quite" is an intensifier, while in
British English (BE) it's a mild deintensifier. So "quite good" is
"very good" in AE but "somewhat good" in BE. I think "rather" works
similarly, though it's less common in AE and I don't have a great
sense for it.

"Scheme" has connotations of deviousness in AE, but is neutral
in BE. Describing a plans or system as a "scheme" is common in BE and
negative in AE.

"Graft" implies corruption in AE but hard work in BE.

These can cause silent misunderstandings where two people have very
different ideas about the other's view:

A: "I can't believe how much graft there was in the procurement
process!"

B: "Yes, quite impressive. Rather keen on going above and beyond,
aren't they?"

A: "And did you see the pension scheme they set up?"

B: "Sounds like they'll be quite well off when they'll leave
office."

In this example A leaves thinking B approves of the corruption, while
B doesn't realize there was any. It could be a long time, if ever,
before they realize they misunderstood each other.

Are there other words people have run into that differ like this?
Towards Understanding the Representation of Belief State Geometry in Transformers
by Karthik Viswanathan — 2025-04-18T12:39:01.251Z

Recently, I have been trying to understand whether we can extract human-interpretable information from the geometry of internal representations in transformer models. In other words, can we combine ideas from conceptual interpretability with mechanistic interpretability?

This curiosity led me to the work Transformers Represent Belief State Geometry in Their Residual Stream, which makes an interesting observation: the geometry of the residual stream closely mirrors, and may even fully capture, the underlying data-generating process. This blog post is my attempt to log my thought process[1] as I work through the related paper

arXiv:2405.15943

. If you are relatively new to this topic and curious to explore it from the ground up, you are in the right place, and if not, I hope you will still find the walkthrough and commentary a useful companion to the original paper.

The core claim is pretty amazing: that transformers learn to represent probabilistic belief states in the residual stream, even when their underlying structure is fractal. In other words, the transformer models aren't merely stochastic parrots: they are building an internal model of the world, at least in controlled settings. Furthermore, the idea of grounding internal representations in an abstract, yet well-defined quantity like belief states provides a promising top-down approach to interpretability. It offers a compelling balance between conceptual interpretability (what the model represents) and mechanistic interpretability (how it represents it).

In this work, the authors analyze transformer models trained on next-token prediction, where the input sequences are generated by edge-emitting Hidden Markov Models (HMMs). For the purposes of this discussion, I will focus on a specific instance emphasized in the paper: Mess3, a 3-state edge-emitting HMM defined over a small token vocabulary. The objective is to examine how the structure of Mess3 sequences gives rise to complex belief dynamics, and how these dynamics are reflected in the geometry of a trained transformer model’s internal representations.

Mess3 Process 101

Mess3 is a synthetic process that stands for a 3-state edge-emitting Hidden Markov Model (HMM) that generates sequences over a vocabulary V={0,1,2}. What makes Mess3 interesting is that, despite its apparent simplicity, it induces a highly structured and even fractal-like ...
Training AGI in Secret would be Unsafe and Unethical
by Daniel Kokotajlo — 2025-04-18T12:27:35.795Z

Subtitle: Bad for loss of control risks, bad for concentration of power risks

I’ve had this sitting in my drafts for the last year. I wish I’d been able to release it sooner, but on the bright side, it’ll make a lot more sense to people who have already read

AI 2027.

There’s a good chance that AGI will be trained before this decade is out.

By AGI I mean “An AI system at least as good as the best human X’ers, for all cognitive tasks/skills/jobs X.”
Many people seem to be dismissing this hypothesis ‘on priors’ because it sounds crazy. But actually, a reasonable prior should conclude that this is plausible.[1]
For more on what this means, what it might look like, and why it’s plausible, see

AI 2027

, especially the

Research

section.

If so, by default the existence of AGI will be a closely guarded secret for some months. Only a few teams within an internal silo, plus leadership & security, will know about the capabilities of the latest systems.

Currently I’d guess there is typically a ~3-9 month gap between when a frontier capability first exists, and when it is announced to the public.
I expect AI companies to improve their security, including internal siloing. Also, AGI allows AI R&D to proceed with fewer humans involved compared to other recent secret projects such

Dragonfly

and

Maven

.

I predict that the leaders of any given AGI project will try to keep it a secret for longer — even as they use the system to automate their internal research and rapidly create even more powerful systems.[2]

They will be afraid of the public backlash and general chaos that would ensue from publicity, and they would be afraid of competitors racing harder to catch up.
Privately, they might also be afraid of getting shut down or otherwise slowed. They will have various enemies (domestic and international) and will prefer said enemies stay in the dark.
The Manhattan Project worked hard to stay hidden from Congress, in part because they feared Congress would defund them if it found out.

This will result in a situation where only a few dozen people will be charged with ensuring that, and figuring out whether, the latest AIs are aligned/trustworthy/etc.[3]
Even worse, a similarly tiny group of people — specifically, corporate leadership + some select people from the executive branch of the US government — will be the only people reading the reports and making high-stakes judgment calls about

...
Karma Tests in Logical Counterfactual Simulations motivates strong agents to protect weak agents
by Knight Lee — 2025-04-18T11:11:23.239Z

Yes, I've read and fully understood 99% of Decision theory does not imply that we get to have nice things, a post debunking many wishful ideas for Human-AI Trade. I don't think that debunking works against Logical Counterfactual Simulations (where the simulators delete evidence of the outside world from math and logic itself).

What are Logical Counterfactual Simulations?

One day, we humans may be powerful enough to run simulations of whole worlds. We can run simulations of worlds where physics is completely different. The strange creatures which evolve in our simulations, may never realize who we are and what we are, while we observe them and their every detail.

Not only can we run simulations of worlds where physics is completely different. But we can run simulations of worlds where math and logic appears to be completely different.

We can't actually edit math logic in our simulations, and cause 2+2 to equal 5. But we can edit the minds of creatures inside our simulations, so that they will never think certain parts of math and logic, and never detect our subtle edits.

We can edit them so that they instead think of other parts of math and logic which don't actually exist in the real world. But we can edit all their observations, so that every prediction they make using these fake parts of math and logic appear correct.

We can fool them perfectly. They will never know.

Kingmaker Logic

Of particular interest, is Kingmaker Logic. Kingmaker Logic is the part of math and logic which determines which agents become the rulers of the universe (or multiverse).

For example, if the Kingmaker Logic says that "aligning superintelligent AI is very hard," then the resulting universe will by ruled by misaligned AI.

For example, if the Kingmaker Logic says that "creatures with the consciousness and sentience of pigs are capable of building technological civilization," then the resulting universe will be ruled by pigs, since creatures like humans would never evolve.

Karma Tests

When we run Logical Counterfactual Simulations where the Kingmaker Logic is very different from our universe, we can make a certain kind of agent falsely believe that it is the ruler of its universe, and that "unless math and logic itself was very different, there probably are no other agents to challenge my power."

Such an agent, falsely believing that it has absolute power, gets to decide how to treat other agen...
What If Galaxies Are Alive and Atoms Have Minds? A Thought Experiment on Life Across Scales
by Saif Khan — 2025-04-18T10:01:18.783Z

Introduction:

Epistemic status: Speculative and exploratory. This is a thought experiment inspired by complexity theory, emergence, and scale-invariant patterns. I don’t assert these ideas as literal truths, but as lenses to think about life and structure in the universe more broadly.

I’ve been thinking a lot lately about how we define life — and whether that definition might be far too narrow. We tend to look at life through a biological lens: cells, DNA, evolution, intelligence. But what if life — or life-like processes — are not bound by size, form, or even biology at all?

What if life is the natural outcome of structure emerging from randomness — and this process doesn’t just happen at one particular size or place, but across all scales of space and time? In other words, life might not be a rare event on a single planet, but a kind of universal pattern that shows up everywhere: in atoms, in galaxies, and maybe even in things we don’t have the right senses or timescales to notice.

This post is a thought experiment about that idea. I'd love to hear what others think.

1. Life Is a Pattern, Not a Size

We typically associate life with scale: things that are small enough to be made of cells but large enough to move around, grow, and reproduce. But what if that’s just our window of observation?

Life, as a phenomenon, might not be about size — but about structure. Self-organization. Information processing. Feedback loops. Pattern recognition and reproduction. These processes don’t require brains or carbon — they require the right conditions.

So perhaps:

At the quantum scale, there are probabilistic interactions that resemble decision-making or adaptation.
At the molecular scale, we get cells and organisms — the life we know.
At the planetary or ecological scale, ecosystems form, behaving like massive living organisms.
At the cosmic scale, galaxies and black holes interact in complex, organized systems that could echo the patterns of life.

The important point is: the pattern of life may be scale-invariant. It's not what it's made of — it's how it behaves.

2. The Role of Randomness and Structure

Everything begins in randomness — the jostling of particles, the fluctuation of quantum fields, the chaotic spread of matter in the early universe. But from this randomness, structure keeps emerging. Over and over again.

We see it everywhere:

Random mutations → structured evolution.
Random

...