trentmkelly/LessWrong-43k huggingface dataset:
May-June 2025 Progress in Guaranteed Safe AI
by Quinn — 2025-06-26T21:30:13.182Z

There will be a AIxFM conference in the Bay Area Q4, according to a little birdie.

Morph ships big autoformalization result in 3599 lines of Lean

They have human decomposition in the latex/lean blueprint, into 67 lemmas with human spotchecking. Still, I’m impressed with their system (called Trinity).

I’d like to know how expensive (in tokens, or some other compute metric) it was to do this!

On Verified Superintelligence

I of course have opinions on their blog post Verified Superintelligence.

Today's most advanced AI systems—reasoning LLMs trained with supervised RL—have hit a fundamental wall. They can only improve on problems where we can verify the (known) answers. Every math problem needs a known solution. Every coding challenge requires test cases. Every reasoning task demands ground truth.

I think I know where they’re going with this: “verification” in the sense of supervised RL as a cognate with “verification” in the sense of typechecking.

Perhaps most importantly, this approach offers a path to what we call "trustless alignment." By ensuring all outputs can be independently validated, we create AI systems whose safety doesn't depend on faith in their training process or architecture. The proof is in the output.

This describes effectively a boxing strategy, not alignment. Boxing is when you police the boundary between the AI and the world. Alignment is when you don’t need to. Is this an important distinction, or a narcissism of small differences you’d only notice if you have Lesswrong Disease (the disease they diagnose you with when you read or write on Lesswrong)? Being afflicted myself, I’m not sure how I would tell.

Two benchmarks

VERINA

Large language models (LLMs) are increasingly integrated in software development, but ensuring correctness in LLM-generated code remains challenging and often requires costly manual review. Verifiable code generation -- jointly generating code, specifications, and proofs of code-specification alignment -- offers a promising path to address this limitation and further unleash LLMs' benefits in coding. Yet, there exists a significant gap in evaluation: current benchmarks often lack support for end-to-end verifiable code generation. In this paper, we introduce Verina (Verifiable Code Generation Arena), a high-quality benchmark enabling a comprehensive and modular evaluation of code, specification, and proof generation as well as t

...
How many GPUs are markets expecting?
by CaseyMilkweed — 2025-06-26T21:17:46.581Z
Link: https://caseymilkweed.substack.com/p/trying-to-guess-gpu-production-based

Even if we get ultra-capable frontier models, we’ll need lots of GPUs to run them at scale. Currently, our installed GPU stock

isn’t enough

to automate the white-collar workforce or cause extinction. We need more!

Tyler Cowen thinks we should predict AI progress using

asset prices

. He’s usually thinking about

interest rates

, but what about NVIDIA’s stock price? Since NVIDIA makes

almost all

of the GPUs used to train and run AI models, and since GPUs generate

almost all

of NVIDIA’s sales, NVIDIA’s market capitalization basically represents the discounted value of future cash flows from the GPU market.

Using a

reverse DCF model

, you can estimate how much NVIDIA must grow to generate cash flows consistent with their current valuation. Since NVIDIA generates cash flows by making sales and generates sales by making GPUs, you can back into a global GPU projection by making further assumptions about margin, price, and market share.

In that spirit, I

modeled

an explicit GPU production forecast based off of NVIDIA’s valuation. In their background research, the

AI 2027

folks also make explicit predictions about GPU production in their

compute forecast

. Do thier assumptions imply that NVIDIA is under-valued? Looking just at 2027, it’s pretty hard, though not impossible, to reconcile the AI 2027 assumptions with NVIDIA’s current valuation. One could argue that the conclusion “Daniel Kokotajlo should buy NVIDIA stock” is less than revelatory, but I will nonetheless continue writing this post.

First, how did I get these numbers? For 2027, it was pretty easy, because I rely on a consensus forecast for near-term financials. Equity analysts are generally

expecting

NVIDIA to have $250 billion in net sales by 2027. The price of an H100 GPU is

$25k

, so you might think that implies ~10 million GPUs. My central estimate is higher, because I assume some reduction in price and some decline in NVIDIA’s market share. Here’s the actual math behind my 16 million central estimate for 2027:

$250 billion net sales
x 92% of sales are GPUs
x (1 / $18k GPU unit price)
x (1 / 0.8 NVIDIA market share)

The consensus forecast only runs through 2028, so my

spreadsheet

goes a little further by projecting what growth you would need through 2035 to justify NVIDIA’s market cap. Things get very uncertain very quickly. It seems like NVIDIA’s price is consistent with production being somewhere betwee...
A Guide For LLM-Assisted Web Research
by nikos — 2025-06-26T18:39:21.573Z

It's hard to imagine doing web research without using LLMs. Chatbots may be the first thing you turn to for questions like: What are the companies currently working on nuclear fusion and who invested in them? What is the performance gap between open and closed-weight models on the MMLU benchmark? Is there really a Tesla Model H?

So which LLMs, and which "Search", "Research", "Deep Search" or "Deep Research" branded products, are best? How good are their epistemics, compared to if you did the web research yourself?

Last month we (FutureSearch) published Deep Research Bench (DRB), a benchmark designed to evaluate LLMs agents on difficult web research tasks using frozen snapshots of the internet. In this post, we're going to share the non-obvious findings, suggestions and failure modes that we think might be useful to anyone who uses LLMs with web search enabled.

tl;dr

ChatGPT with o3 + websearch outperformed everything else by a decent margin, though is still clearly worse than a skilled human. Use this as your default research tool.
Claude web + Claude Research were not able to read PDFs (as of May 6, 2025 and as far as we can tell that's still the case), which likely nerfed their scores. If that's important for your task, don't use them.
If you're using LLMs with your own agent via an API, Claude 4 Sonnet + Opus are best, better than o3 (as of June 24, 2025).
Grok was ok, but not great. Unless you need to access Twitter. Then definitely use Grok, which excels at this (our task instances did not rely on Twitter, so that strength is not reflected in our benchmark).
Regular mode + web search often outperformed their corresponding dedicated deep research tools. They are also much better for iterative research.

What We Did

Full methods in the paper. In brief, we created web research tasks, carefully worked out the correct answers, and scraped and stored a large chunk of webpages offline for the agents to search, read, and reason about to succeed in the task.

Overview of task types. DRB uses RetroSearch, a system designed to serve agents a frozen, previously scraped version of the internet instead of the live pages.. Offline Web Pages shows the median (min–max) number of web pages scraped and stored for agents to search and read during each task.

We then ran 12 LLMs, each driving a ReAct agent with tools to search the internet and query documents and web pag...
RLAIF/RLHF for Public Transparency Alignment
Enhancing Transparency in LLMs
by Jada42 — 2025-06-26T18:32:12.002Z

Hey all — I just submitted my Master’s thesis, and would really appreciate feedback from people thinking about non-task value alignment or LLM transparency metrics.

I explored whether Reinforcement Learning from Human Feedback (RLHF), using a constitutional rubric, could align a language model with transparency as a civic value. The case: Amsterdam’s parking permit system.

Methods:

Model: Mistral-7B-Instruct v3
Training: PPO using a 6-dimensional transparency rubric (Explainability, Completeness, Source Attribution, Procedural Clarity, Accuracy, Limitation Disclosure)
Baseline: SFT using ~300 examples
Evaluator: Independent LLM Judge (Claude Sonnet 4) scoring per-dimension across 300 held-out citizen queries

Key Results

RLHF win rate: 61% vs SFT baseline (p < 0.001)
Biggest gains:

+21.6% Information Completeness
+19.5% Source Attribution

Accuracy dropped by -6.6%, mostly due to over-hedging (e.g. "This may be different for your city", despite giving city specific information)
81% of reward signal came from explicit constitutional constraints, not emergent learning (according to the Optuna trials)

Takeaways

RLHF can encode democratic values — but needs deliberate constraints
There’s a fundamental tradeoff: transparency ≠ precision
Technical alignment isn’t enough — civic values require governance, citizen input, and deliberation

Would love feedback — especially on:

whether this rubric-based evaluator is generalizable
how others might have measured non-task values like “transparency”
if this work might overlap with existing alignment metric toolkits

I’ll post the paper & code when cleaned up.

Abstract

Large Language Models in government services often lack alignment with democratic values, particularly transparency, risking opaque AI-citizen interactions that undermine public trust. This thesis investigates whether Reinforcement Learning from Human Feedback (RLHF) can systematically embed transparency in government AI systems.

Using Amsterdam's parking regulations as a critical case, we fine-tuned a Mistral-7B model through constitutional RLHF, where a six-dimensional transparency rubric guided Proximal Policy Optimization. This approach was benchmarked against supervised fine-tuning across 300 citizen queries, with an independent LLM evaluating transparency outcomes.

The RLHF model achieved a 61%-win rate (p < .001), with substantial improvements in Information C...
If Moral Realism is true, then the Orthogonality Thesis is false.
by Eye You — 2025-06-26T18:31:01.417Z

Claim: if moral realism is true, then the Orthogonality Thesis is false, and superintelligent agents are very likely to be moral.

I'm arguing against Armstrong's version of Orthogonality[1]:

The fact of being of high intelligence provides extremely little constraint on what final goals an agent could have (as long as these goals are of feasible complexity, and do not refer intrinsically to the agent’s intelligence).

Argument

1. Assume moral realism; there are true facts about morality.

2. Intelligence is causally[2] correlated with having true beliefs.[3]

3. Intelligence is causally correlated with having true moral beliefs.[4]

4. Moral beliefs constrain final goals; believing “X is morally wrong” is a very good reason and motivator for not doing X.[5]

5. Superintelligent agents will likely have final goals that cohere with their (likely true) moral beliefs.

6. Superintelligent agents are likely to be moral.

^

Though this argument basically applies to most other versions, including Yudkowsky's strong form: "There can exist arbitrarily intelligent agents pursuing any kind of goal [and] there's no extra difficulty or complication in the existence of an intelligent agent that pursues a goal."
This argument as is does not work against Yudkowsky's weak form: "Since the goal of making paperclips is tractable, somewhere in the design space is an agent that optimizes that goal."

^

Correction: there was a typo in the original post here. Instead of 'causally', it read 'casually'.

^

A different way of saying this: Intelligent agents tend towards a comprehensive understanding of reality; towards having true beliefs. As intelligence increases, agents will (in general, on average) be less wrong.

^

A different way of saying this: Intelligent agents tend toward having true moral beliefs.
To spell this out a bit more:

A. Intelligent agents tend toward having true moral beliefs.
B. Moral facts (under moral realism) are (an important!) part of reality.
C. Intelligent agents tend toward having true moral beliefs.

^

One could reject this proposition by taking a strong moral externalism stance. If moral claims are not intrinsically motivating and there is no general connection between moral beliefs and motivation, then this proposition does not follow. See here for discussion of moral interalism and orthogonality and here for discussion of moral motivation.

As

...
The Cadca Transition Map - Navigating the Path to the ASI Singleton
by cadca — 2025-06-26T18:30:33.359Z

Progress

“Acquisition, acquisition, acquisition, merger! Until there was only one business left in every field.”

― Vincent H. O'Neil

Human progress is mapped in dozens of different metrics: food supply, gross domestic product, computational power. As civilization has developed, many of these metrics have trended upwards, finding exponential growth over time. Many other metrics had bucked that trend, though. Child mortality, food scarcities. Of these, there is one peculiar value of note: the number of sovereign states. Today, there are ~195 sovereign countries in the world. A thousand years ago, there were hundreds of sovereign kingdoms, nations, and states. Ten thousand years ago, humanity lived across ten thousand different tribes.

This is an odd trend, which raises a peculiar question. Why does this seemingly innocuous number follow the trends of much more troubling statistics? There are many different factors, most of which revolve around a central axis: technological improvements in communication, production, and security have led to sovereign states growing in power and influence. Through unification or conquest, those ten thousand tribes eventually aggregated into 195 countries. This is a well known and well documented trend observed by historians and sociologists like Charles Tilly, who famously argued that states consolidated power through competition and warfare.

As mentioned earlier, those many metrics have seen exponential growth over time. However, there’s a new development which seems poised to deliver these potent returns in extremely quick time. If you’re reading this, I assume you’re familiar with the idea of the ‘intelligence explosion’, or of the ‘technological singularity’. If humanity is looking at a new technological revolution in artificial intelligence, then the ramifications will be vast. Those metrics that we discussed will likely see greater progress than currently expected.

This brings into question that specific metric, the number of sovereign states. If AI delivers an Intelligence Revolution, particularly in those specific fields that have historically pressured this metric, then we can expect a great change to come to the world in the next few decades. Nations will likely fall, and blocs will likely consolidate. This is rational extrapolation based on observed patterns of political and social history. The ramifications, however,...
Getting To and From Monism
by unication — 2025-06-26T18:28:12.340Z

Monism, in its most basic form, is the belief that the world is comprised of a single substance. I believe there to be a genuine way to prove this, regardless of whether or not you understand what this 'single substance' might be at root. We can go from logic, to monism, and then we can make some assumptions about the physical universe, which I believe to be helpful. Furthermore, you can use this formulation of monism to understand ideas of consciousness, separation of minds, physicalism, and a cavalcade of hitherto unsolved problems within physics.

In this first section, I will explain how to get to this monism, and discuss possible axioms that can be applied to other disciplines.

Getting there

We can start from a point of complete skepticism about everything. Regardless of your specific beliefs about the probability of a simulated universe, or if you are a pure idealist, you can say with confidence that at least something exists. Even if this is a dream in the mind of a Boltzmann brain, it is true that something exists, it might not be the reality that you're experiencing now, but something has to exist in order for you to supervene on it.

On some level, something exists — this is the main thrust of the first part of the argument. Counterarguments about skepticism do not matter, because the hypothetical daemon making me hallucinate this world may belong to a universe that does actually exist in some form. Even hallucinations have to exist to be hallucinated.

Let's say this existence is made up of two entirely separate things, for example, matter / mind (the classic disjunct) — and see where that leads us. If they are made of entirely separate things, then they would not be able to interact. If they can interact, like mind and matter appear to, then that is proof that they must be comprised of the same thing at their root, for if they were not, they would not be able to interact. Perhaps they interact through a third thing — but this 'third thing' does not mean that the world is constructed of 3 components (mind / matter / bridging thing) — it means that the entire world is comprised of that bridging thing, and mind and matter are just names arbitrarily assigned to points on said bridge.

We do not even have to use the controversial topic of mind and matter to prove this. We can say that any two things that are able to interact share the same root material. Imagin...
AI #122: Paying The Market Price
by Zvi — 2025-06-26T18:10:01.651Z

If you are Meta, and you want to attract top AI talent, you have a problem, because no one wants to work for you or on your products. So it is going to cost you. Mark Zuckerberg has decided he will pay what it takes to get at least some very good talent.

If you are the rest of us, especially someone seeking an entry-level job and for whom $100 million signing bonuses are not flooding your inboxes, things are getting rougher. AI might not yet be destroyed a lot of jobs, but it is doing a number on the job application process.

There’s a lot of other stuff going on as per usual.

Anthropic won a case establishing that model training is fair use.

Yesterday I shared various Tales of Agentic Misalignment, and we have some more related fun since then that is included here.

I also analyzed a critique of the AI 2027 Timeline Forecasts.

In non-AI content, I offered another Childhood and Education post, this one on behaviors and related questions. I also offer a fun little Easter Egg side post on an entirely unrelated and inessential topic, for those who want to see the gamer game in a different way.

Table of Contents

Table of Contents.

Language Models Offer Mundane Utility. America is winning the AI coding race.

Language Models Don’t Offer Mundane Utility. Who believes current events?

Huh, Upgrades. ChatGPT expands connections, Claude explands artifacts.

On Your Marks. Safety scoring the labs.

Choose Your Fighter. Three good choices, but a warning about Gemini Pro.

Deepfaketown and Botpocalypse Soon. AI slop taking over music?

Release the Hounds. Gemini CLI can escalate remarkably quickly.

Copyright Confrontation. Anthropic establishes model training as fair use.

Cheaters Gonna Cheat (x5). Time for exams. Shut down the AIs for the duration?

Fun With Media Generation. Another Veo 3 creation.

Get My Agent On The Line. Where are the agents we were promised?

They Took Our Jobs. Or at least they made our jobs impossible to find.

Get Involved. Quora, a personal task and a person seeking opportunity.

Introducing. AlphaGenome, for understanding our DNA.

In Other AI News. We don’t have spiritual bliss, maybe joint sycophancy?

Gemini Sings The Blues. Don’t do it, Gemini. There’s so much to code for.

Show Me the Money. Meta and xAI open the checkbooks.

Quiet Speculations. Pete Buttigieg (I know!) and Samuel Hammond.

Timelines. The incentives to report them inaccurately cut

...
Love Island USA Season 7 Episode 20: What Could The Producers Be Thinking
by Zvi — 2025-06-26T17:31:13.739Z

(Note: This is NOT being posted on my Substack or Wordpress, but I do want a record of it that is timestamped and accessible for various reasons, so I'm putting this here, but do not in any way feel like you need to read it unless it sounds like fun to you.)

We all deserve some fun. This is some of mine.

I will always be a gamer and a sports fan, and especially a game designer, at heart. The best game out there, the best sport out there is Love Island USA, and I’m getting the joy of experiencing it in real time.

A Brief Ode To One Of Our Beloved Games

Make no mistake. This is a game, a competition, and the prizes are many.

The people agree. It is more popular than ever, with huge social media engagement.

I got into this through the ‘horrible people producing The Bachelor’ show UnReal, which I consider one my seven tier 1 TV shows of all time. Then, I figured, I’d give a season of Bachelor a shot, watched 24, and found I was not only having a ton of fun, I was finding the game play fascinating. Then I kept watching, putting together the puzzle pieces for how the real game is played on my own since the internet spoils everything.

Then I realized you could safely compare to older podcast episodes. Then I found Game of Roses (GoR), and I set aside my draft post because I realized they were way, way ahead of me.

The game is fun if you take it at face value.

The game is even more fun if you don’t, and you accept it for exactly what it is, a multilevel strategic battleground of shifting goals and alliances for the players and also the producers, America, the public.

Where who you are playing with really matters. Instead of a finite sports game with constant rules, you get an infinite sports game. A game with only one ultimate rule: 4TRR, ‘for the right reasons,’ meaning you must be here to find love, with different games enforcing this to different extents and with different strictness, which has to involve playing the full game without revealing too explicitly that you know the full game exists and you are playing it.

Each player is playing to four audiences at once:

The lead of the show, or your romantic paramour.
Your fellow players, especially of the same gender.
The producers, who have a ton of control but also real limitations.
The public, who ultimately decide if you have a new career and how you’re seen.

Who is here 4TRR and to what extent? Who is going after what and care...