trentmkelly/LessWrong-43k huggingface dataset:
On 'On Caring'
by atharva — 2025-05-26T13:39:40.769Z

Thanks to Jacob G-W, Sophia Lloyd George, Ariana Azarbal, and Jo Jiao for reading through / commenting on a draft. All opinions, and any errors, are my own.

‘On Caring’ by Nate Soares is a great piece. It’s a moving appeal to do good in the world. I respect Nate’s articulation of his personal philosophy, and I empathize with his motivations for writing the same. At the same time, I don’t share many of the intuitions behind it. I don’t intend for this to be ‘a rebuttal’. That seems unkind. A rebuttal would also be unproductive – deep-seated values seem hard to change. Yet, I want to articulate why these intuitions don’t ring true for me.

‘Caring’ is a funny word. I don’t have a crisp, intensional definition for it. In this essay, it’s associated with ‘feeling for another's pain’. This framing, however, ignores the happiness felt by others. There is a lot of weight in the world, but there is also much lightness. It seems unfaithful to focus on one, without acknowledging the other. I’m moved by both the highs and the lows of the world – and I’m uncomfortable with the (seemingly) primary focus on suffering.

I also don’t think that mountains weigh as heavily on me, as they do on some others. Why? For one, it seems like mountains are inevitable. We perceive mountains because of how they jut out relative to our experiences; if there’s any inequality in the world, we will always have mountains. But regardless of our perception of them, the absolute size of the mountains does seem to be shrinking.[1] Currently, an average person in the USA lives a life far-removed from the challenges faced by billions – eg. reliable access to food, water, and shelter. Yet, it seems plausible that in a future world, folks might look back upon the life of an American today with similar sorrow. Eg. the overwhelming rates of obesity, navigating a broken healthcare system, and living with the fear of school shootings.[2] I’m not taking a fatalist stance here. One of our biggest skills is recognizing the mountains around us – that’s how progress gets made. Nonetheless, this is a task with no clear resolution criteria. Sure, we can work on reducing the mountains. But since I’m skeptical of ever being able to ‘solve suffering’, they weigh a little less heavy on me.

Another reason for this, is the average baseline happiness of people. All things considered, most people are gen...
Claude 4 You: The Quest for Mundane Utility
by Zvi — 2025-05-26T13:01:27.485Z

How good are Claude Opus 4 and Claude Sonnet 4?

They’re good models, sir.

If you don’t care about price or speed, Opus is probably the best model available today.

If you do care somewhat, Sonnet 4 is probably best in its class for many purposes, and deserves the 4 label because of its agentic aspects but isn’t a big leap over 3.7 for other purposes. I have been using 90%+ Opus so I can’t speak to this directly. There are some signs of some amount of ‘small model smell’ where Sonnet 4 has focused on common cases at the expense of rarer ones. That’s what Opus is for.

That’s all as of when I hit post. Things do escalate quickly these days, although I would not include Grok in this loop until proven otherwise, it’s a three horse race and if you told me there’s a true fourth it’s more likely to be DeepSeek than xAI.

Table of Contents

On Your Marks.

Standard Silly Benchmarks.

API Upgrades.

Coding Time Horizon.

The Key Missing Feature is Memory.

Early Reactions.

Opus 4 Has the Opus Nature.

Unprompted Attention.

Max Subscription.

In Summary.

On Your Marks

As always, benchmarks are not a great measure, but they are indicative, and if you pay attention to the details and combine it with other info you can learn a lot.

Here again are the main reported results, which mainly tell me we need better benchmarks.

Scott Swingle: Sonnet 4 is INSANE on LoCoDiff

it gets 33/50 on the LARGEST quartile of prompts (60-98k tokens) which is better than any other model does on the SMALLEST quartile of prompts (2-21k tokens)

That’s a remarkably large leap.

Visual physics and other image tasks don’t go great, which isn’t new, presumably it’s not a point of emphasis.

Hasan Can (on Sonnet only): Claude 4 Sonnet is either a pruned, smaller model than its predecessor, or Anthropic failed to solve catastrophic forgetting. Outside of coding, it feels like a smaller model.

Chase Browser: VPCT results Claude 4 Sonnet. [VPCT is the] Visual Physics Comprehension Test, it tests the ability to make prediction about very basic physics scenarios.

All o-series models are run on high effort.

Kal: that 2.5 pro regression is annoying

Chase Browser: Yes, 2.5 pro 05-06 scores worse than 03-25 on literally everything I’ve seen except for short-form coding

Zhu Liang: Claude models have always been poor at image tasks in my testing as well. No surprises here.

H...
Formalizing Embeddedness Failures in Universal Artificial Intelligence
by Cole Wyeth — 2025-05-26T12:36:41.067Z
Link: https://arxiv.org/abs/2505.17882

AIXI is a dualistic agent that can't work as an embedded agent... right? I couldn't find a solid formal proof of this claim, so I investigated it myself (with Marcus Hutter). It turns out there are some surprising positive and negative results to be derived as easy corollaries of the paper "Universal Prediction of Selected Bits." Interestingly, further technical advances in algorithmic information theory could substantially strengthen our results - I would welcome collaborations with strong theoretical computer scientists, (deep familiarity with agent foundations not required).

This work was supported by the Long-Term Future Fund and presented at the CMU agent foundations conference in 2025.
Techies Wanted: How STEM Backgrounds Can Advance Safe AI Policy
by Daniel_Eth — 2025-05-26T11:29:08.835Z

TL;DR – Individuals with technical backgrounds are well-positioned to advance AI safety policy in numerous ways. Opportunities include both A) technical research directions, including evals, compute governance mechanisms, infosec, and forecasting, and B) career paths for advancing AI policy, including policymaking and communication efforts. Contrary to what many people assume, most of these roles do not require extensive political backgrounds or unusually good social skills, and many technical people who aren’t currently considering these paths would be a good fit. This post aims to encourage such people to consider these options.

Introduction

Over the past couple years, there has been a substantial increase in enthusiasm in the AI safety community for addressing AI risk through policy or governance solutions. Yet despite growing excitement for more policy work to address AI risk, many people with technical backgrounds may underestimate their personal fit for contributing to this area. Moreover, there are numerous sub-areas within the AI governance ecosystem where folks with technical backgrounds are in high demand.

This post aims to encourage technically minded individuals who are interested in addressing AI risk to consider working on AI governance.

If you have a technical background and have dismissed the idea of engaging in governance work because you see yourself as more STEM-y or not a "politics person," it's worth considering if you’ve dismissed these paths too hastily. In order to break into many governance paths, you do not need to have deep involvement in politics or extensive preexisting knowledge of political workings.

The current state of AI policy – proposals often lack sufficient details for implementation and policymakers often have insufficient technical understanding

Looking at actual proposals that may have had relevance for catastrophic risks from AI,[1] there are many areas where ideas need to be fleshed out more or where the proposal passes the buck to some other group to figure out specifics. For instance, Biden’s Executive Order called on various agencies to “establish guidelines and best practices... for developing and deploying safe, secure, and trustworthy AI systems, including [by] launching an initiative to create guidance and benchmarks for evaluating and auditing AI capabilities, with a focus on capabili...
D&D.Sci: The Choosing Ones [Answerkey and Ruleset]
by abstractapplic — 2025-05-26T09:43:54.762Z

This is a followup to the D&D.Sci post I made nine days ago; if you haven’t already read it, you should do so now before spoiling yourself.

Below is an explanation of the rules used to generate the dataset (my full generation code is available here, in case you’re curious about details I omitted).

Ruleset

Skills and Competence

The success of a hero is decided by six hidden variables: Int, Wis, Cha, Str, Con, and Dex[1]. These are used to produce Competence via the formula

Competence = 8*Int + 9*Wis + 10*Cha + 10*min(Int,Wis,Cha) + 11*max(Str,Con,Dex) + 11*Str+12*Con + 13*Dex[2][2]

Competence, Luck, and Success

The Challenge of saving a world is given by

Challenge = 1d20 + 1d40 + 1d60 + 1d80 + 1d100 + 1d200 + 1d400

If a hero’s Competence exceeds the Challenge, they succeed.

27.1% of generated heroes are Lucky. A Lucky hero, if they don’t succeed fairly, rolls a d8; on an 8, they succeed anyway through sheer force of Fate. This makes Luck a major factor for low-Competence heroes, but much less important to high-Competence ones.

The Fae and their Predictions

Amy

Amy believes fervently in Luck - particularly its ability to ameliorate otherwise hopeless situations - and nothing else. She predicts a 1% success rate for everyone who doesn’t have it, and a 99% success rate for everyone who does.

Bella and Liboulen

Bella (named for Peter Pan’s Tinkerbell) and Liboulen (named for Cleveland Quixotic’s Olliebollen) prioritize a hero’s mind over their body.

Bella is excessively cynical, and doesn’t care for Wisdom. Her predictions are given by

BellaPred = Cha+Int-1

Some bad experiences with over-clever heroes have left Liboulen falsely perceiving Intelligence as an active liability. Their predictions are given by

LiboulenPred = 40.7+5*Cha+Wis-Int

Linestra and Colleen

Linestra also prioritizes the mind over the body, linearly combining Int, Wis and Cha with a random element as below:

LinestraPred = 22.5 + 1.2*Int + 2.4*Wis + 3.6*Cha + 0.1*1d12 - 0.1

Colleen copies Linestra’s predictions, creating a nearly collinear column. Her guesses are Linestra’s, plus 1.7%; when Linestra predicts over 90% success chance, she adds a further 0.2%.

Fizz, Ziqual, and Ister

Fizz, Ziqual and Ister are physicalists, prioritizing physical stats over mental. Their predictions are given by

FizzPred = 41+2*Dex+min(Str,Con)
ZiqualPred = Str*Con - ((Str*Con)>12)*random.choice([0,1])
IsterPred = 50+Str

Holly

Holly take...
The Sundog Alignment Theorem: A Proposal for Embodied Alignment via Indirect Inference
by Malice — 2025-05-26T07:26:23.400Z

100kb physics alignment Simulation running:

https://youtu.be/Gp7a-fXcRNM?si=zp7vqQEU34yGmk2B

H(x) or The Sundog Alignment Theorem proposes that robust alignment can emerge from agents interacting with structured environments via indirect signals—specifically, shadow convergence and torque feedback—rather than direct reward targeting or instruction.

Inspired by atmospheric sundogs (light halos visible only at indirect angles), we construct a simulated system where an agent aligns a mirrored pole to a plumb laser beam, not by observation of the goal, but by detecting torque resistance and the convergence of shadow "bloom" on a structured ceiling grid.

I've spent ten years learning to insert screws to a ceiling using an invisble laser mark and weeks ago i had to train some ESL guys how to align with with these shadow physics. Here is how we turned that into an AI alignment experiment.

I’m a blue collar regular drop-out and independent researcher, previously an electrician, now an automation engineer. I submit plans for $100m dollar computer builds that my customers love but I'm apparently too illiterate to communicate with people who moderate the internet since this program is too naughty and getting me banned from everywhere I try to publish.

I’ve spent a decade building infrastructure—automation, quantum enclosures, high-torque mounts for server racks, laser alignment systems—and in that time, I developed a deep muscle memory for torque. Specifically, how the feel of a pole twisting against your hand could tell you more than an equation ever could.

I realized something during ceiling installs:

When I'm pushing a fastener into the ceiling with an 18-foot pole, I can't see the tip. I aim a laser plumb line, then rotate the base of the pole until the bloom of reflected light tightens into a singularity. Halo collapse.

I'm aligning to a shadow structure using indirect feedback — shadow, torque, reflection.

So I asked: What if an agent could align like this? And that's where the trouble began

~The Experiment~

I built a simulation in MuJoCo at the public library:

A jointed pole with a mirrored tip.

A laser beam projected from the floor to a ceiling.

A ceiling untextures, then intuitively textured with honeycom fields: golden spirals, harmonic waves, hurricane geometries.

My agent never sees the goal.

It only feels torque at the joints and w...
Superposition Without Compression: Why Entangled Representations Are the Default
by James Butterworth — 2025-05-26T05:26:19.947Z
Link: https://drive.google.com/file/d/1xdTRTRjKXneXxYKA7Kqa4NHe7nzV7IKK/view?usp=drive_link

I have heard numerous claims recently that the underparameterisation of neural networks can be implied due to the polysemanticity of its neurons, which is prevalent in LLMs.

Whilst I have no doubt that polysemanticity is the only solution to an underparameterised model, I urge on the side of caution when using polysemanticity as proof of underparametarisation.

In this note I claim that: even when sufficient capacity is available, superposition may be the default due to its overwhelming prevalence in the solution space.
Disentangled, monosemantic solutions
occupy a tiny fraction of the total low-loss solutions.

This suggests that superposition arises not just as a necessity in underparametarised models, but also is an inevitability of the search space of neural networks.

In this note I show a comprehensible toy example where this is the case and hypothesise that this is also the case in larger networks.

These were very rough Sunday musings so I am very interested about what other people think about this claim :).
Seeking Feedback: Toy Model of Deceptive Alignment (Game Theory)
by Alex Boche — 2025-05-26T05:23:22.860Z

Background: I'm an economics grad student with limited background on AI itself.

I'm seeking feedback on a game-theoretic model of deceptive alignment. The basic idea is that a dynamically sophisticasted AI with a hidden preference type will choose an action on the basis of two considerations: 1) its intristinc preference for what it wants the action to be, and 2) how it's current action affects how it will be retrained for next period, at which time it will be faced with another choice under it's new (retrained) preference type.

For now, I'm mainly just trying to get feedback on the modeling approach itself. [But I have proved some results, one of which I mention here -- see "One Result" subheading].

Before presenting the formal model(s), let me preview my main questions I want feedback on. [Any other feedback very welcome!]

The first question is about how to model retraining an AI's hidden type, which I view as a real number (or vector). Is it better to think of retraining as 1) moving the type in a desired direction and magnitude (i.e. adding a desired vector), or 2) moving it towards a desired (target) point? [Or are both fatally flawed?] If 2, there must be a cost of training; otherwise, the model would be trivial since the trainer would just train infinitely hard towards its favorite point (zero in my model). Should that cost be convex, linear, concave?

The second question is: should I focus on AI with perfect-recall or imperfect-recall? The perfect-recall approach thinks of the AI as already in the world, taking actions that are payoff-relevant to both itself and its trainer. The imperfect recall approach thinks of the AI as first being placed in a (payoff-irrelevant) simulation where it takes actions and then is (potentially) retrained, after which it is deployed to the real world where it takes payoff-relevant actions. Crucially, AI can't tell the difference between the simulation and reality and can't remember whether it has been in simulation previously (hence imperfect-recall). To me, both settings seem relevant.

Edit: I'm also curious whether this analysis could inform how to test for scheming behavior.

The rest of this post describes my models. I start with a model with perfect-recall where the trainer trains the AI in a desired direction and magnitude (rather than towards a desired point). This model has two versions depending ...
Long-form data bottlenecks might stall AI progress for years
by Michelle_Ma — 2025-05-26T04:36:00.039Z

Cross-posted from my Substack.

Introduction

AI 2027’s recent publication made waves—if you’re reading this, you probably saw the whole saga unfold: cheers, jeers, NYT writeup, the works. To briefly recap: the authors predict that by March 2027, a superhuman coder is created, speeding up AI R&D fivefold. By the end of the year, full-blown artificial superintelligence arrives. It sparks unprecedented scientific and economic progress—alongside mass unemployment, an arms race with China, and, worst case, human extinction via bioweapon.

There’s much to say about the assumptions built into their timelines, but here, I want to hone in on one key piece: the long-form data bottleneck.

I argue that the AI 2027 authors overlook the availability (or lack thereof) of the suitable long-form data necessary to train a model capable of reasoning reliably over months-long projects—an ability the authors explicitly say their "superhuman coder" needs. This might seem like a minor technical detail, but this data bottleneck could significantly delay AGI timelines, perhaps by years or even decades (Monte Carlo simulation here & corresponding post). Ironically, such delays would be great news for those concerned with AI safety, slowing timelines and providing a rare opportunity for tractable governance.

AI 2027 - Timelines

There are two parts to the AI 2027 forecast. The first part, “Timelines,” estimates the arrival of a superhuman coder. The second part, “Takeoff,” predicts a superhuman AI researcher and, soon after, artificial superintelligence—an explosive jump powered by automated AI R&D.

Perhaps counterintuitively, the Timelines portion actually makes up most of the gap between now and transformative AI. Why? Well, the development of a superhuman coder relies on slow, human-driven R&D. But once it’s created, the research process speeds up 5x, meaning that further improvements quickly follow.

Now, this claim of a 5x speed-up seems dubious, but there’s already another AI 2027 review that covers this concern. For my part, I’ll be digging into the details of the Timelines analysis, uncovering how the data bottleneck might hamper the creation of a superhuman coder.

Superhuman Coder & Time Horizons

First, let’s define what a superhuman coder is. It’s not a well-defined technical term—here, it just refers to an AI that can perform expert-level AI R&D ...
Example of Splitting a PR
by jefftk — 2025-05-26T02:20:02.542Z

I'm a big fan of small focused PRs: it's much easier to tell whether
they're actually doing the right thing. For example, I recently ran
into a common scenario where I started adding some functionality, and
realized this required some refactoring. The refactoring also made it
clear that the original code was missing some important test coverage,
so I added some tests. I ended up with a big set of changes which
combined large refactors and a major (intended) change to the output.

This is a bad combination: if my refactoring accidentally changed
functionality in a way not captured by the tests we might miss it.
And with so many changes it's going to be hard for the reviewer (and
even the author) to keep it all in their head at once. Instead, it
would have been better if I had first done the refactoring and added
tests, and then followed up with the new functionality. The first PR
would be a bit noisy but conceptually straightforward and easy to
validate because it produces exactly the same output for all inputs.
The second PR would be very clean, but would require careful
validation to verify that the changes really are an improvement.

I might be tempted to say "oh well, I should have approached this
differently, I'll do better next time" but really I did need to
combine these two changes when writing them: I didn't know for sure
what I would want for the refactoring until I'd tried to use it to
implement the functionality change. So if my plan was to do better
next time I wouldn't actually do any better.

Instead I took advantage of version control to split my changes
into two sets, in an order that is optimized for validation:

I committed all my changes to a new branch,
jefftk-new-feature, with a message that only mentioned the
feature I was adding, not the refactoring or the new tests.

I made a new branch off of jefftk-new-feature to
represent the refactoring and new tests: git checkout -b
jefftk-refactor.

On jefftk-refactor I deleted everything related to
the new feature, leaving only the refactoring and new tests. I
verified that the output was unmodified, as you'd expect for a
refactoring-only change. I committed this, with a message that only
mentioned the refactoring and tests.

I interactively rebased (git rebase -i HEAD~2)
this branch to squash the two commits into one. I only kept the
refactoring commit message. This

...
How I'm telling my friends about AI Safety
by k64 — 2025-05-25T22:43:23.227Z

One of the comments on the new book post asked how to tell normie friends about AI safety. I don't have any special credentials here, but I thought it'd be worthwhile to share the facebook post I've drafted, both to get feedback and to give an example of one way a post could look. There exist articles and blogs that already do this well, but most people don't read shared articles and it's helpful to have a variety of ways to communicate. My goal here is to grab attention, diffuse densiveness with some humor, and try to make the problem digestable to someone who isn't immersed in the topic or lingo. Let me know what you think!
Why AI might kill us all

Dramatic opener – check. I don’t post often, especially not about “causes”, but this one feels like a conversation worth having, even if we sound a little crazy.

TL;DR – AI is (and will be) a really big tool that we need to be super careful wielding. Corporations aren’t going to stop chasing profit for us, so talk to your representatives, and help educate others (this book is coming out soon from top AI safety experts and could be a good resource).

Wouldn’t it be awesome to own a lightsaber? You could cut through anything! Unfortunately, knowing me, I’d smush the button while trying to get it out of the packaging and be one limb on my way to becoming Darth Vader.

AI is like a lightsaber. Super cool. Can do super cool stuff. Turns out, “cutting off your own limbs” technically counts as “super cool”.

Here’s another analogy: there’s a story about “The Monkey’s Paw” – an old relic that grants the user 3 wishes, like a genie. The catch is, the wishes always come with some bad consequence. You wish for $1 Million – congrats – that’s the settlement amount you get for getting paralyzed by a drunk driver!

It occurred to me the other day, that to make a Monkey’s Paw, you don’t need to tell it to do bad things, just to be overly literal or take the easiest path to granting the wish. “I wish that everyone would smile more” -> overly literal interpretation -> virus breaks out that paralyzes people’s facial muscles into a permanent smile.

AI is a wish-granting computer program. Right now, it can’t do much – it can only grant wishes for pictures with the wrong number of fingers and English class essays, but when it...