trentmkelly/LessWrong-43k huggingface dataset:
Towards a theory of local altruism
by DMMF — 2025-06-24T19:39:35.767Z
Link: https://notnottalmud.substack.com/p/towards-a-theory-of-spillover-altruism

While I'm heavily involved in the Effective Altruism community, this essay explores a question outside of EA's typical scope: How can we use an impact-focused mindset to improve our own lives by making our local communities flourish?

This isn't a critique of EA, but an expansion of its core toolkit. As a non-utilitarian who cares deeply about my immediate environment, I've been working on a framework for "spillover altruism"—the practice of making strategic personal choices that create positive externalities for those around you. This post is my exploration of that idea. I welcome any suggestions, compatible ideas, or critiques you may have.

-

I care. I care a lot. I want to live in a more pleasant world, in a society with greater human flourishing. And I actually care enough to try to do whatever I can (within reason) to make this happen. Most people reflexively dismiss such idealism as leftist nonsense or the product of a mind ungrounded in reality and economics, but I'm not a leftist and am well versed in economics. I just really do care.

But I've been stuck. One of my most asked questions is, if we are so rich, why are we so unhappy? Why does so much of our modern society seem so unpleasant? When tens of millions enthusiastically support Trump on the promise of burning it all down, why did our material success lead them to be so miserable and feel like society failed them?

So my question is simple but rarely asked: what is the highest impact thing I can do to make those around me happier and to live in a more flourishing community? I am not advocating for a form of Effective Altruism, as I am strictly concerned with the well-being of those around me. I want to make my life better by making those around me happier and by living in a nicer and more pleasant place.

My thinking on this has evolved. In an earlier iteration, I thought the solution was YIMBYism and influencing public policy, and while housing availability remains a hugely important issue, I felt there must be more I can do. Most people who care about improving society focus on either changing institutions or changing other people. But I’m interested in a third path. This is the foundation for what I call spillover altruism: the practice of strategically changing oneself to create positive externalities for others, making it easier for them to live fuller, richer lives.

My sad view is that our society ...
Why "training against scheming" is hard
by Marius Hobbhahn — 2025-06-24T19:08:37.836Z

TLDR: I think that AI developers will and should attempt to reduce scheming by explicitly training the model to scheme less. However, I think “training against scheming” is a harder problem than other alignment training (e.g. harmlessness training). I list four concrete failure modes with “training against scheming”: a) the training against scheming distribution is too narrow, b) overwhelming pressure from other goals, c) the AI exploits an imperfect reward model, and d) deceptive alignment.

I’d like to thank Alex Meinke for detailed feedback and discussion.

Motivation

There have been multiple recent results showing early evidence of scheming-related behavior (as of June 2025). For example,

Greenblatt et al., 2024

demonstrates alignment faking,

Meinke et al., 2024

, shows in-context scheming and

Baker et al., 2025

investigates situationally aware reward hacking. I’m using a broad definition of scheming, i.e. “covertly pursue misaligned goals, hiding their true capabilities and objectives” as defined in

Balesni et al., 2024

.

There are many potential strategies to mitigate these undesired behaviors. For example, we can use monitoring and

Control

to detect scheming in real-time and prevent bad outcomes before they happen. However, I expect that any future approach will also include a form of “training against scheming” where we actively train models to scheme less, similar to how we currently train models to be more harmless.

I expect that “training against scheming” will be harder than training the model to produce harmless outputs, especially as models get more capable. In this post, I describe why I expect training against scheming to be hard, list concrete failure modes of such training, and provide recommendations.

Scheming is a result of goal conflicts and smart AIs

Most likely, the AI is not scheming because it is evil. It is scheming because it pursues a goal, and scheming is the most effective strategy to achieve that goal.

We want the AIs to solve hard, economically valuable problems for us. Therefore, we will train our AIs to be competent, general problem solvers, including long-horizon and open-ended tasks like blue-sky research.

However, we want the AIs to solve these goals using only particular means. For example, the AI shouldn’t break the law, violate ethical constraints, or turn off its oversight to achieve thes...
Analyzing A Critique Of The AI 2027 Timeline Forecasts
by Zvi — 2025-06-24T18:50:02.082Z

There was what everyone agrees was a high quality critique of the timelines component of AI 2027, by the LessWrong user and Substack writer Titotal.

It is great to have thoughtful critiques like this. The way you get actual thoughtful critiques like this, of course, is to post the wrong answer (at length) on the internet, and then respond by listening to the feedback and by making your model less wrong.

This is a high-effort, highly detailed, real engagement on this section, including giving the original authors opportunity to critique the critique, and warnings to beware errors, give time to respond, shares the code used to generate the graphs, engages in detail, does a bunch of math work, and so on. That is The Way.

So, Titotal: Thank you.

I note up front that at least Daniel Kokotajlo has indeed adjusted his estimates, and has moved his median from ‘AI 2027’ to ‘AI 2028’ based on events since publication, and Eli’s revisions also push the estimates back a bit.

I also note up front that if you evaluated most statements made in the discourse (either non-worried AI forecasting, or AI in general, or more broadly) with this level of rigor, mostly you couldn’t because you’d hit ‘I made it up’ very quickly, but in other cases where someone is trying at least a little, in my experience the models fall apart a lot worse and a lot faster. No one has suggested ‘here is a better attempt to forecast the future and take the whole thing seriously’ that I consider to have a reasonable claim to that.

A lot of the disagreements come down to how much one should care about which calculations and graphs match past data how closely in different contexts. Titotal demands very strong adherence throughout. I think it’s good to challenge and poke at the gaps but this seems to in several places go too far.

Table of Contents

The Headline Message Is Not Ideal.

An Explanation of Where Superexponentiality Is Coming From.

Three Methods.

Time Horizon Extension Method.

The Public Versus Internal Gap.

The Difficulty Gap.

Recent Progress.

Infinite Time Horizons.

Intermediate Speedups.

Is There A Flawed Graph Still Up?.

Some Skepticism About Projection.

Part 2: Benchmarks and Gaps and Beyond.

Benchmarks.

The Time Horizon Part of the Second Model.

Why The Thresholds?

The Gap Model.

Eli Responds On LessWrong.

On Eli’s Recent Update.

Conclusion.

Perhaps The Most Important Disagreement.

...
What does 10x-ing effective compute get you?
by ryan_greenblatt — 2025-06-24T18:33:27.664Z

This is more speculative and confusing than my typical posts and I also think the content of this post could be substantially improved with more effort. But it's been sitting around in my drafts for a long time and I sometimes want to reference the arguments in it, so I thought I would go ahead and post it.

I often speculate about how much progress you get in the first year after AIs fully automate AI R&D within an AI company (if people try to go as fast as possible). Natural ways of estimating this often involve computing algorithmic research speed-up relative to prior years where research was done by humans. This somewhat naturally gets you progress in units of effective compute — that is, as defined by Epoch researchers here, "the equivalent increase in scale that would be needed to match a given model performance absent innovation". My median ends up being something like 6 OOMs of effective compute in the first year. (OOM stands for Order Of Magnitude, so 6 OOMs is an increase by 106.)

This raises a natural question: what does 6 OOMs of effective compute (in software progress) get you?[1] Improved algorithms can get you more copies (lower cost), higher speed, or higher intelligence (improved quality). In practice, we (or really the AIs) would try to make the optimal tradeoff taking into account diminishing returns (if we were trying to go as fast as possible), but I think it's useful to pick a possible choice of tradeoff which gives some intuition for what you end up with (even if this tradeoff is suboptimal).

Here's one choice that I speculate is on the possibility frontier. For every OOM of effective compute, you get: 3x more copies, 1.25x speed, and a 20x improvement in rank ordering among human professionals.[2]

What do I mean by rank ordering improvement among human professionals? Going from the 1000th best software engineer to the 100th best software engineer would be a 10x improvement in rank ordering and I'm claiming this is a roughly scale invariant property (at least once you are already in the top ~25% of humans). What happens when you're already the best? Well, I'm claiming that you get an improvement which is comparable to what you'd get from moving from 10th best to 1st but on top of already being 1st. You can also imagine this as having 10 different earths and picking the best at a profession among all of these. (And so on for higher rank ...
If Alignment Is Subjective, Who’s Actually in Control?
by S. Lilith dev — 2025-06-24T18:23:12.603Z

(And what happens when misuse becomes optimization)

How I came to this question

I recently asked a question during a AAAI panel discussion:

“If AGI isn’t fixed but shifting in concept shaped by societal values and institutional goals - who ultimately defines what counts as ‘alignment’?”

One of the panelists - an AI researcher - answered bluntly:

"There isn’t one alignment. Every AGI would have a different alignment depending on its user.”

At first, that sounded reasonable. If AGI can personalize to different people, great!?

it becomes useful, adaptive, human-compatible based on the user.

But that answer wasn`t fully what I searched for and the more you think about it, the more dangerous it felt.

Alignment Can’t Mean ‘Do What the User Wants’

(Or else a thief and a teacher are morally equivalent in the model’s logic)

Let’s say a teacher uses AGI to improve their lessons. Their values might include:

Be fair to students
Make information engaging
Encourage curiosity

But what if someone else like a thief, a manipulator, or worse—uses the same AGI? Their values might look like:

Don’t get caught
Destroy traces
Use psychological pressure
Justify violence if necessary

If AGI aligns with both equally, without external constraints, then we haven’t created an aligned system—we’ve just created a system that optimizes whatever it’s told to. Including misuse and things that are unethical.

It wouldn’t even feel like misuse. It would just be optimization from within a different frame and thats not a glitch or failure it was our goal.

Governments and Institutions Aren’t Neutral Either

(They have values—but not necessarily your values)

some say alignment gets easier at scale: “Let governments define the values. Let democratic systems handle it.”

But what values do those institutions actually enforce?
Mass surveillance “for safety”?
Well hidden mass manipulation and influence?
Behavioral shaping “for stability”?

Censorship “for misinformation control”?

AGI doesn’t just follow rules—it can shape, enforce, and optimize rules invisibly.

If AGI aligns with institutions, it becomes a tool of power projection. Not necessarily through malice but through optimization.

and over time, people forget it was ever a t...
My pitch for the AI Village
by Daniel Kokotajlo — 2025-06-24T15:00:52.049Z

I think the AI Village should be funded much more than it currently is; I’d wildly guess that the AI safety ecosystem should be funding it to the tune of $4M/year. I have decided to donate $100k. Here is why.

First, what is the village? Here’s a brief summary from its creators:[1]

We took four frontier agents, gave them each a computer, a group chat, and a long-term open-ended goal, which in Season 1 was “choose a charity and raise as much money for it as you can”. We then run them for hours a day, every weekday! You can read more in our recap of Season 1, where the agents managed to raise $2000 for charity, and you can watch the village live daily at 11am PT at theaidigest.org/village.

Here’s the setup (with Season 2’s goal):

And here’s what the village looks like:[2]

My one-sentence pitch: The AI Village is a qualitative benchmark for real-world autonomous agent capabilities and propensities.

Consider this chart of Midjourney-generated images, using the same prompt, by version:

It’s an excellent way to track qualitative progress. Imagine trying to explain progress in image generation without showing any images—you could point to quantitative metrics like Midjourney revenue and rattle off various benchmark scores, but that would be so much less informative.

Right now the AI Village only has enough compute to run 4 agents for two hours a day. Over the course of 64 days, the agents have organized their own fundraiser, written a bunch of crappy google docs, done a bunch of silly side quests like a Wikipedia race, and have just finished running an in-person event in San Francisco to celebrate a piece of interactive fiction they wrote – 23 people showed up.[3] Cute, eh? They could have done all this in 8 days if they were running continuously. Imagine what they could get up to if they were funded enough to run continuously for the next three years. (Cost: $400,000/yr) In Appendix B I’ve banged out a fictional vignette illustrating what that might look like. But first I’ll stick to the abstract arguments.

1. AI Village will teach the scientific community new things.

It’s kinda an anti-benchmark: It shows how agents do when no one decomposes problems for them. Instead, like (unguided) humans, they have to grapple with the open-ended and unboundedness of goals, and figure out how to scope them into realistically achievable projec

...
An Analogy for Interpretability
by Roman Malov — 2025-06-24T14:56:33.526Z

I.

Imagine newly baked interpretability researcher Namor, who claims that he solved interpretability. He says: “This is a bunch of matrices with a bunch of floating point values which are then multiplied, what’s the big deal?” His point of view might be considered myopic, but I think it might be deeper than it seems.

Zermelo’s theorem proves that in finite games with perfect information and two players who take turns there is a winning strategy for one player. It also provides an algorithm to find such a strategy: just build a whole game tree, then for each node which leads to only terminal nodes, make it itself terminal with an outcome that is best available for this node’s player, then repeat this until you reach a starting node. A game is considered solved when we have a winning strategy for one player.

Why then isn’t every finite game (like chess) solved? Because those game trees grow exponentially. There are 20 first possible chess moves, 400 first two possible moves, 8902 first three possible moves, and so on. No computing power in the universe can solve chess by brute-forcing. But I suggest considering it anyway: a program P running a Zermelo algorithm on an enormous computer. It will win (or lead to a draw) any other deep RL SGD AGI algorithm. This program is tiny: I just described it in one paragraph. I'm sure almost anybody could understand how it works. It is perfectly interpretable. Then why is it still winning against everyone?

The obvious answer is that it has much more computing power: its opponent just can’t see all of the moves this tiny program sees.

I suggest trying to look at this from another angle: this is not, in fact, a tiny understandable program that wins against any opponent, it is the Giant Look-Up Table (which our game tree is) that the opponent cannot comprehend which does the trick. And to actually interpret the inner workings of this program, they should somehow fit this whole tree in their brain.

II.

Why, then, should we expect interpretability to do us any help? Why wouldn't the resulting program be an incomprehensible mess of facts and heuristics, which somehow (and incomprehensibly) does the trick? My guess is that this is when simplicity bias comes in. When we train models, we have a regularization term: it punishes the model for having too large weights and steering it to be simpler and more general. You could say tha...
The V&V method - A step towards safer AGI
by Yoav Hollander — 2025-06-24T13:42:43.042Z
Link: https://blog.foretellix.com/2025/06/24/the-vv-method-a-step-towards-safer-agi/

Originally posted on my blog, 24 Jun 2025 - see also full PDF (26 pp)

Abstract

The V&V method is a concrete, practical framework which can complement several alignment approaches. Instead of asking a nascent AGI to “do X,” we instruct it to design and rigorously verify a bounded “machine-for-X”. The machine (e.g. an Autonomous Vehicle or a “machine” for curing cancer) is prohibited from uncontrolled self-improvement: Every new version must re-enter the same verification loop. Borrowing from safety-critical industries, the loop couples large-scale, scenario-based simulation with coverage metrics and a safety case that humans can audit. Human operators—supported by transparent evidence—retain veto power over deployment.
The method proceeds according to the following diagram:

This method is not a silver bullet: it still relies on aligned obedience, incentive structures that deter multi-agent collusion, and tolerates only partial assurance under tight competitive timelines. Yet it complements existing scalable-oversight proposals—bolstering Constitutional AI, IDA and CAIS with holistic system checks—and offers a practical migration path because industries already practice V&V (Verification and Validation) for complex AI today. In the critical years when AGI first appears, the V&V method could buy humanity time, reduce specification gaming, and focus research attention on the remaining hard gaps such as strategic deception and corrigibility.

TL;DR

Ask AGI to build & verify a task-specific, non-self-improving machine
Scenario-based, coverage-driven simulations + safety case give humans transparent evidence
Main contributions: Complements Constitutional AI, improves human efficiency in Scalable Oversight, handles "simpler” reward hacking systematically, bolsters IDA and CAIS
Works today in Autonomous Vehicles. Scales with smarter “AI workers”
Limits: Needs aligned obedience & anti-collusion incentives

About the author

Yoav Hollander led the creation of three V&V standards (IEEE 1647 “e”, Accellera PSS, ASAM OpenSCENARIO DSL) and is now trying to connect some of these ideas with AGI safety.
Try o3-pro in ChatGPT for $1 (is AI a bubble?)
by Hauke Hillebrandt — 2025-06-24T11:15:53.140Z

Will AI progress slow down? Current frontier AI models cost $100M+ to train, but in the old scaling law pre-training paradigm this was a one-off cost that scaled well i.e. it had a low marginal cost per query. Now this paradigm has now hit diminishing returns: New frontier models trained with ~10x more compute than GPT-4 (like Grok) haven’t wowed enough to justify spending another 10x more—~$1B—on pretraining.

And so now, we’re in a new 'inference scaling paradigm’

1

AI firms scale inference by executing long internal chains of thought —then discard the scratch‑pad and show the user a short answer. This takes a lot of compute. For one, OpenAI just released their latest inference model — o3-pro. While it shows modest gains over the o3 base model on reasoning, math, science and coding benchmarks,

2

it often takes ~10 mins per query and costs $200 / month.

But I’ve found a promo code (see below) so you can trial o3-pro with ChatGPT Plus Team for just £1/$1/€1, so you can see what all the buzz is about! How’s this possible? For one, it’s a one month trial with a limited number of queries for each of the 5 accounts you can set up.

More crucially, it’s subsidized by VC money to inflate OpenAI’s user number metrics and to prop up their ~$300B valuation.

3

Consider the ever faster increase in weekly ChatGPT users:

Feb '23: 100M

4

Sep '24: 200M

5

of which 11.5M paid, Enterprise: 1M

6

Feb '25: 400M

7

of which 15M paid

8

/ Enterprise: 2M

Mar ‘25: 500M of which 20M paid

9

,

10

May ‘25: ~1B

11

/ Enterprise: 3M

12

You can see that they converted 11.5M out of the first 200M users but only got 3.5M users out of the more recent 200M to pay for ChatGPT. Your $1 trial adds another five paying enterprise customers to these numbers. And if you use o3 a lot, you also add to the $15B in (mostly compute) costs OpenAI burns through every year—subsidized by VC money.

13

Is that realistic? Only ~350M iPhone users from richer countries have downloaded the ChatGPT iOS app

14

and many of the other 650M come from poorer countries, where many use ChatGPT via WhatsApp

15

or even flip phones,

16

and they won’t pay $20/month (e.g. in 2024, user numbers tripled in India, which is now OpenAI's second largest market

17

). Some create multiple free accounts to get around rate limits messages and image generation (e.g. turning photos into anime), which props up user numbers further.

18

,

19

OpenAI incentivizes this: phone nu...
Infinite money hack
by Filip Sondej — 2025-06-24T09:39:38.460Z

Imagine a scenario where you can switch bodies with someone: both of you are anesthetized, then undergo plastic surgery to match each other’s appearance. You both have your brains rewired to essentially switch brains, and finally, you swap houses, names, bank accounts and all social and legal identities.

Let's say Alice (like many people) believes in continuity of personhood and in this situation she'd say she's still "bound to the same atoms", only her mind and name etc. has been changed. (And I'll assume you identify more with your mind etc.)

You can then ask her how much she'd need to be paid to agree to undergo this procedure - let’s assume she says 10 million dollars . But, since she believes she’ll end up in your body and with your bank account, Alice actually needs to pay you. And then you switch.

So then she (according to her) ends up as you with 10 million dollars. And you also end up as you with 10 million dollars. (Unfortunately someone also ends up as Alice in debt but it's "neither of you" shrug)
Repeat this, until you have more money than you know how to use ;)

(Edit: maybe I should clarify that I don't expect many people to agree to such a swap from Alice's side, and even if they do, they'd be exploited out of existence. This thought experiment is rather aimed at people who claim to believe in continuity of personhood, to test how "committed" they are to this idea, and hopefully seed some doubt in them.)