trentmkelly/LessWrong-43k huggingface dataset:
Newton's second law explained: it works in many universes
by Tahp — 2025-05-05T19:47:03.928Z
Link: https://quark.rodeo/2025/04/30/newtonSecond.html

The linked post explains Newton's second law in an intuitive way. I'm presenting an edited version of that post here. It is a more detailed version of an argument I made as part of this post. I've noticed some interesting posts about using the framework of quantum field theory for mechanistic interpretability. I want to write a physicist's perspective on the flexibility of physics theories, since the concepts within physics theories are widely reapplied outside of physics, and are commonly utilized by the sort of people who post on LessWrong in particular. Some day I want to write a post like this one about quantum field theory, but that post will necessarily be more mathematically difficult to follow than one on Newtonian mechanics. I think there is value in having an easy-to-follow reference post for the idea of "physics theories are frameworks for understanding the universe, not the actual source code of the universe."

Newton's laws of motion are often the first things people learn when they study physics. In spite of the fact that the theory is rarely used in academic physics research, physicists keep teaching it to people, and I think we are absolutely correct to do so. Newton's second law might be the most beautiful equation in physics. The equation is a rigorous mathematical framework for describing motion, but every piece of it can be directly attached to statements most people would find intuitive. The law is usually written →F=m→a.

An intuitive explanation of Newton's second law

Let's take a look at the equation in a slightly less traditional form:

→F=mddt→v

Newton's second law is used to describe a single object. The symbols refer to physics quantities which are understood to correspond to that single object. I'll explain all of the terms later, but the force is a force on the object. The mass and velocity are the mass and velocity of the object.

The first symbol we find in this equation is →F, which represents force. A force can be thought of as a push or a pull. If you touch something, then you put a force on it. If you pull on a spring, you can feel it pulling back on you, which means that the spring is exerting a force away from you. Note that →F has a funny arrow over it. That means that force is represented by a mathematical object called a vector. Vectors have both a size and a direction. You can push really hard or not that hard on things, and...
Replicator->Vehicle Alignment and Human->AI Alignment
by derelict5432 — 2025-05-05T19:23:55.450Z

Disclaimer: I sometimes use conscious or agentic language to describe relationships or behaviors of non-conscious and non-agentic entities (like genes). I am under no illusion that such things are conscious or agentic. Sometimes it is simply clearer and less awkward to use this linguistic framing.

The AI alignment discussion typically centers around the overlap between the goal structures of humans and AI, and efforts to ensure the completeness and lasting integrity of that overlap.

But there is a much older alignment relationship than human->AI, one that is not only analogous, but impinges directly on it. This is the alignment of replicators and vehicles.

Replicator->Vehicle Alignment

At the dawn of life, a peculiar type of molecule arose, one with the capability to make copies of itself. This was the progenitor of all life on earth. It was a sort of template, attracting the raw materials to line up in a stable way so that when detached, a copy of the original floated away to continue the process.

Copies were sometimes imperfect. Because of this, all the elements existed for the first functioning framework for evolution by natural selection: a replicating population, mutation, variance, and differential survival.

At some point, the replicators began encoding for and fabricating structures that were not part of the core replicators themselves. Strictly speaking, these were the first tools, and the replicators were the first (unconscious) tool users on the planet. The first primitive cells were one of the most important of these tools. The replicators had developed the ability to construct citadels in which they could focus on their machinations, protected from harmful radiation and the destructive toxins in their liquid homes.

Cells and their associated structures were the first vehicles, and the dichotomy of replicators and vehicles was born. The replicator persisted across time. The vehicle was a shell, an expendable tool with one purpose and one purpose only: to aid the replicators in their primary pursuit of replication.

Many innovations have arisen over billions of years, trillions of individuals, and endless tinkering through the blind processes of evolution. But the function of vehicles has not changed. The first cells were encoded for and built by the replicators for the sole purpose of aiding replicators i...
The Sweet Lesson: AI Safety Should Scale With Compute
by Jesse Hoogland — 2025-05-05T19:03:28.748Z

A corollary of Sutton's Bitter Lesson is that solutions to AI safety should scale with compute.[1]

Let's consider a few examples of research directions that are aiming at this property:

Deliberative Alignment: Combine chain-of-thought with Constitutional AI to improve safety with inference-time compute (see Guan et al. 2025, Figure 13).
AI Control: Design control protocols that pit a red team against a blue team so that running the game for longer results in more reliable estimates of the probability of successful scheming during deployment (e.g., weight exfiltration).
Debate: Design a debate protocol so that running a longer, deeper debate between AI assistants makes us more confident that we're encouraging honesty or other desirable qualities (see Irving et al. 2018, Table 1).
Bengio's Scientist AI: Develop safety guardrails that obtain more reliable estimates of the probability of catastrophic risk with increasing inference time:[2]

[O]ur proposed method has the advantage that, with more and more compute, it converges to the correct prediction . . . . In other words, more computation means better and more trustworthy answers[.] — Bengio et al. (2025)

Anthropic-style Interpretability: Develop interpretability tools (like SAEs) that first learn the SAE features that are most important to minimizing reconstruction loss (see Templeton et al. 2024, Scaling Laws).
ARC-style Interpretability: Develop interpretability tools that extract the most important safety-relevant explanations first before moving on to less safety-relevant explanations (see Gross et al. 2024):

In ARC's current thinking, the plan is not to have a clear line of separation between things we need to explain and things we don't. Instead, the loss function of explanation quality should capture how important various things are to explain, and the explanation-finding algorithm is given a certain compute budget to build up the explanation of the model behavior by bits and pieces. — Matolcsi (2025)

For these procedures to work, we want the ideal limits of these procedures to be safe. In a recent talk, Irving suggests we should understand the role of theory as being about analyzing these limiting properties (assuming sufficient resources like compute, data, and potentially simplifying assumptions) and the role of empirics as being about checking if the conditions required by the theory seem to hold ...
Blue light, 'Adrenal ASMR': strange experiences I can't find any literature about
by vernichtung — 2025-05-05T18:58:57.207Z

Hi, I've found I have an intense psychosomatic reaction to certain bands of blue lighting. I bought a powerful blue lamp on Amazon simply because I liked the color blue, and found it does something interesting to me when I sit in a dark room with this lamp switched on. Not always, because I build a high tolerance to it if I do it too many times back to back, but it's happened often enough.

It feels like a combination of tingling heat, mild adrenaline rush and a feeling of gentle compression on either side of about L1/T12, as if someone has reached right in and is massaging my adrenal glands. I've measured my heart rate in case I simply imagined the sensation. It was in the 90-100 range when my resting is 70-80. Most search results for 'blue light' are negative, but this is pleasant, not like anxiety. It feels almost like an atypical form of ASMR that involves adrenaline instead of sleepiness.

I understand that this could be purely imaginary, but then why is blue light causing me to imagine something so bizarre and so specific? Any ideas? I have some more extensive logs and notes about it if anyone wants them.
Tsinghua paper: Does RL Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
by Thomas Kwa — 2025-05-05T18:56:28.760Z
Link: https://arxiv.org/abs/2504.13837

arXiv | project page | Authors: Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, Gao Huang

This paper from Tsinghua find that RL on verifiable rewards (RLVR) just increases the frequency at which capabilities are sampled, rather than giving a base model new capabilities. To do this, they compare pass@k scores between a base model and an RLed model. Recall that pass@k is the percentage of questions a model can solve at least once given k attempts at each question.

Main result: On a math benchmark, an RLed model (yellow) has much better raw score / pass@1 than the base model (black), but lower pass@256! The authors say that RL prunes away reasoning pathways from the base model, but sometimes reasoning pathways that are rarely sampled end up being useful for solving the problem. So RL “narrows the reasoning boundary”— the region of problems the model is capable of solving sometimes.

Further results

Across multiple math benchmarks, base models have higher pass@k than RLed models for sufficiently large k, and sometimes the crossover point is as low as k=4. (Figure 2)
Out of domain (i.e. on a different math benchmark) the RLed model does better at pass@256, especially when using algorithms like RLOO and Reinforce++. If there is a crossover point it might be near pass@1024. (Figure 7)
To see if the pass@1024 results are just lucky guesses, they sanity check reasoning traces and find that for most questions, base models generate at least one correct reasoning trace. (But it's unclear whether the majority of correct answers are lucky guesses) Also, for long coding tasks, it's nearly impossible to make a lucky guess.
They test the perplexity (Below: Figure 6 left) of the RLed model’s generations (pink bar) relative to the base model. They find it is lower than the base model's perplexity (turquoise bar), which “suggests that the responses from RL-trained models are highly likely to be generated by the base model” conditioned on the task prompt. (Perplexity higher than the base model would imply the RLed model had either new capabilities or higher diversity than the base model.)
Distillation from a stronger teacher model, unlike RL, “expands the reasoning boundary”, so performance improves at all best@k values. (Below: Figure 6 right)

Limitations

RL can enable emergent capabilities, especially on long-horizon tasks: Suppose that a capability require

...
Intro & Proposal for AGI Model
by PickleBrine — 2025-05-05T18:56:05.186Z

Hello all, this is my first time posting in this forum, so I look forward to your feedback.

First, some brief background info about myself: I have an undergraduate background in Computer Science & Philosophy and am considering pursuing a Master's in AI. Apart from my formal education I have also engaged in a significant degree of self-study. My motivation in participating in this forum is, as you might assume, to share my ideas and discuss topics in artificial intelligence. To start off, I'll share with you an outline for an AGI model.

My proposal here is inspired by compression-theoretical models of information and detection of patterns over string streams. A lot of work has already been done in problem domains like binary stream prediction models, which extract patterns or regularities from an input stream that enable prediction of subsequent values. My idea would be to apply a similar approach to data streams that include both agent percept and action states, where such a sequence of states represents a cybernetic feedback loop between the agent and its environment.

Such an agent while in “exploration mode” would choose essentially random actions, or else in a manner so as to maximize exploration of the search space. As this is done they would track the action-percept stream for patterns or regularities across subsegments. When a pattern or compression of a particular segment is discovered, that segment’s entropy decreases (a consequence of the formal equivocation of entropy-theoretic and compression-theoretic definitions of information), indicating that the agent’s uncertainty in outcomes over the sequence of the action-perception states decreases. Conceptually, this elegantly unifies representation, information, prediction, and agent causality. A particular such pattern could be termed a “mechanism”, and detected mechanisms would be stored by the agent in some representation format. When the agent model is augmented with a utility function that prioritizes the attainment of certain agent states, the agent could make use of its stored mechanisms to more efficiently attain that goal, since each of these represents a tool or augmented action over its base actions. A comprehensive agent would also be capable of some degree of meta-learning, wherein it synthesizes known mechanisms or expands them into something more general when presented with additio...
AI Superorganisms: An Alternative Pathway to Artificial Superintelligence
by Aaron Vanzyl — 2025-05-05T18:55:24.272Z

Aaron Vanzyl

aaronvanzyl.com

Introduction

Today’s state-of-the-art AI models are best thought of as single-celled organisms. Individually, they are capable but not a world-ending threat. Arranged in the right structure, however, they become self-replicating cyber algae, or the flesh and bones of digital sharks. I believe that before individual AI agents can outperform humans, AI superorganisms will have taken the leading role in directing humanity’s future.

This article is my forecast for AI development over the next two years. I’ve phrased it in definite terms, like what ‘will’ happen. I did this to make the scenario easier to read, not because I am 100% certain about my predictions. I have also provided specific dates for each phase. I intended to establish a complete (if not necessarily correct) timeline that can be used as a starting point for further conversation. If you think a certain advancement will take longer than I have predicted, I encourage you to leave a comment arguing why. Towards the end of the article, I describe my main areas of doubt and where I think the real world is most likely to diverge from my predictions.

The article covers the next two years, starting today, April 2025, and continuing until artificial intelligence overtakes humanity in early 2027.

I have organized the article into four sequential phases. In the first phase, humanity begins to arrange individual AI agents into basic corporation-like structures. In the second phase, carefully supervised AI corporations are gradually replaced by adaptive, autonomous AI superorganisms. In the third phase, AI superorganisms begin to interact with each other, exhibiting potentially hostile behavior. In the fourth phase, the collective ecosystem of AI overtakes humanity in influence over our civilization.

Today - Single-Celled Organisms

To start, let’s look at what today’s state-of-the-art models can do. This is just a summary of the real world as of April 2025. Nothing here is speculative.

Individual language models can directly produce medium-scope text pieces without the aid of external tools. Tasks like:

Write a report about the rise and fall of the Roman Empire.
Create a Python script to simulate the population dynamics of rabbits and foxes over time.

Agentic models can use web browsers and APIs to interact with the open internet. Tasks like:

Book me a flight from San Fr

...
The Metaculus Cup Series Is Live, $5,000 Prize Pool
by ChristianWilliams — 2025-05-05T17:14:51.933Z
Link: https://www.metaculus.com/tournament/metaculus-cup/

We're excited to announce the Metaculus Cup series, an evolution of our Quarterly Cup tournaments with an enhanced prize pool of $5,000! Want to level up your forecasting skills, build your reputation, contribute to clarity on the news of the day, and compete for cash prizes? Join the Metaculus Cup!

Who will win the Tour de France 2025?
What defense spending target will NATO agree to at their 2025 summit?
Will Norway announce the replacement design for the Fridtjof Nansen-class before September 2025?
Will the restrictions on Revolut's UK banking license be removed before July 26, 2025?

Tournament Structure

The Metaculus Cup will run three tournaments annually—spring, summer, and fall—each lasting four months. Every Monday, we'll launch new questions covering current events and engaging forecasting challenges.

Each Friday at 17:00 UTC, an average of 4 questions will be set to "Upcoming," opening three days later. This gives forecasters time to prepare and provide feedback. Breaking news may result in additional questions being launched during the week. Questions will frequently close early, though some might remain open for the entire tournament duration.

Prize Structure

This tournament features a $5,000 prize divided into two distinct prize pools:

Forecasting Prize ($3,500)

The forecasting prize will be based on time-averaged scores, rewarding accuracy and consistency in your predictions.

Commenting Prize ($1,500)

The commenting prize will be distributed to the 10 best comments made on Metaculus Cup questions before the end of the tournament. Prizes will be awarded as follows:

1st: $300
2nd: $250
3rd: $200
4th: $150
5th-10th: $100 each

Metaculus will select top comments that showcase some or all of these qualities:

Provides clear and insightful reasoning
Contributes valuable information
Identifies mistakes in the community's prediction or makes a strong case for a different prediction
Identifies key factors and submits them using the new key factors feature
Receives "Changed My Mind" votes or has key factors rated as significant by the community

Comments don't need to predict accurately to win prizes—even directionally incorrect comments can be rewarded based on the strength of their reasoning. Individual commenters can receive multiple prizes but will be capped once they reach or surpass $750 in total winnings.

Question Submission

The Metaculus Cup welcomes question submissions from ...
Community Feedback Request: AI Safety Intro for General Public 
by Algon — 2025-05-05T16:38:42.096Z

TL;DR: The AISafety.info team wrote two intros to AI safety for busy laypeople: a short version and a longer version. We expect them to get a few thousand to tens of thousands of views. We'd really appreciate your critiques.

AISafety.info has two new intros: a short version and a longer version. They're intended for a reasonably smart member of the general public.

Our intros are meant to be:

Aimed at a general audience.
A frank presentation of our beliefs.
Concise.
Non-combative.

We want these intros to be a resource that you would be eager to share with friends, family, co-workers, etc. So, please give us some feedback! Even a 10 word comment would help. E.g. "Seems OK, but talking about AI doom to family is weird." Or "The section on instrumental convergence doesn't add much; consider shortening it." Even if you only read a tiny part of our intro, we still want your feedback! Even if it is to say why you think this project is even worth doing.

We're keen to learn what improvements to these articles would make them most useful to you as a shareable resource. (Of course, we'd also appreciate you catching any errors.) We list some of the feedback we're most interested in later on. Only then can we make the most of our upcoming opportunity to spread good models of AI safety to as many people as possible. Rob Miles is making an accompanying video which will link to our articles. We expect that to result in a big influx of viewers, who will share our articles if they find them useful.

The articles are listed below, with summaries. The first four from the long version are imported to LW as part of a sequence. (We'll eventually import the rest, but are avoiding spamming the front page.) You can give feedback 1) on this post (best for us), 2) by clicking on the links below and then clicking on the little pencil icon below the title, or 3) by commenting under the LW posts when they're published.

The Articles:

Short form:

A Case for AI Safety

A self-contained introduction that aims to draw people in with more of a narrative. It's a 16-minute read. We want this to be the one article someone can read about AI x-risks if they don't have time for anything else, and still get a decent picture of what it's about.

Summary: Companies are racing to build smarter-than-human AI. Experts think they may succeed in the next decade. But rather than...
GPT-4o Sycophancy Post Mortem
by Zvi — 2025-05-05T16:00:04.431Z

Last week I covered that GPT-4o was briefly an (even more than usually) absurd sycophant, and how OpenAI responded to that.

Their explanation at that time was paper thin. It didn’t tell us much that we did not already know, and seemed to suggest they had learned little from the incident.

Rolling Stone has a write-up of some of the people whose delusions got reinforced by ChatGPT, which has been going on for a while – this sycophancy incident made things way worse but the pattern isn’t new. Here’s some highlights, but the whole thing is wild anecdotes throughout, and they point to a ChatGPT induced psychosis thread on Reddit. I would love to know how often this actually happens.

Table of Contents

There’s An Explanation For (Some Of) This.

What Have We Learned?

What About o3 The Lying Liar?

o3 The Source Fabricator.

There Is Still A Lot We Don’t Know.

You Must Understand The Logos.

Circling Back.

The Good News.

There’s An Explanation For (Some Of) This

Now OpenAI have come out with a much more detailed explanation. It is excellent that OpenAI is offering us more details, and it’s totally fine for them to take the time to pull it together.

Sam Altman (CEO OpenAI): we missed the mark with last week’s GPT-4o update.

[This post explains] what happened, what we learned, and some things we will do differently in the future.

Ryan Lowe (ex-Open AI): I’ve been critiquing OpenAI recently on this, so I also want to say that I’m glad they wrote this up and are sharing more info about what happened with 4o

it’s interesting to me that this is the first time they incorporated an additional reward based on thumbs up / thumbs down data.

including thumbs up data at all is risky, imo. I don’t think we understand all the ways it can go wrong.

[Suggested related economic work available here.]

Near Cyan: thank you for a post-mortem

Steven Adler: Glad that OpenAI now said it plainly: they ran no evals for sycophancy. I respect and appreciate the decision to say this clearly.

Key quote: “We also didn’t have specific deployment evaluations tracking sycophancy.”

“Our offline evals weren’t broad or deep enough to catch sycophantic behavior—something the Model Spec explicitly discourages⁠”

^ I hope OpenAI now makes sure it has evals for all goals in the Spec

I’m not going to be especially kind about all this, because I don’t think they’ve learned enough of the right (generalized) l...