trentmkelly/LessWrong-43k huggingface dataset:
Frontier AI Models Still Fail at Basic Physical Tasks: A Manufacturing Case Study
by Adam Karvonen — 2025-04-14T17:38:02.918Z
Link: https://adamkarvonen.github.io/machine_learning/2025/04/13/llm-manufacturing-eval.html

Dario Amodei, CEO of Anthropic, recently worried about a world where only 30% of jobs become automated, leading to class tensions between the automated and non-automated. Instead, he predicts that nearly all jobs will be automated simultaneously, putting everyone "in the same boat." However, based on my experience spanning AI research (including first author papers at COLM / NeurIPS and attending MATS under Neel Nanda), robotics, and hands-on manufacturing (including machining prototype rocket engine parts for Blue Origin and Ursa Major), I see a different near-term future.

Since the GPT-4 release, I've evaluated frontier models on a basic manufacturing task, which tests both visual perception and physical reasoning. While Gemini 2.5 Pro recently showed progress on the visual front, all models tested continue to fail significantly on physical reasoning. They still perform terribly overall. Because of this, I think that there will be an interim period where a significant portion of white collar work is automated by AI, with many physical world jobs being largely unaffected.

(Estimated reading time: 7 minutes, 12 minutes with appendix)

The Evaluation

My evaluation is simple - I ask for a detailed plan to machine this part using a 3-axis CNC mill and a 2-axis CNC lathe. Although not completely trivial, most machinists in a typical prototype or job shop setting would view executing this as a routine task, involving standard turning and milling techniques across multiple setups. This was certainly much simpler than the average component at both shops I worked at. For context, compare the brass part's simplicity to the complexity of aerospace hardware like these Blue Origin parts.

Although this part is simple, even frontier models like O1-Pro or Gemini 2.5 Pro consistently make major mistakes. These mistakes can be split into two categories - visual abilities and physical reasoning skills.

Visual Errors

Most Models Have Truly Horrible Visual Abilities: For two years, I've observed essentially zero improvement in visual capabilities among models from Anthropic and OpenAI. They always miss obvious features like the flats cut into the round surface, holes, or even hallucinate nonexistent features such as holes drilled along the part’s length. I have never seen Claude 3.5, Claude 3.7 (thinking and non-thinking), GPT-4.5, GPT-4o, or O1-Pro produce a reasonable descr...
How to evaluate control measures for LLM agents? A trajectory from today to superintelligence
by Tomek Korbak — 2025-04-14T16:45:46.584Z

This is a linkpost accompanying a new paper by UK AI Security Institute, Apollo Research and Redwood Research. Please see the full paper for more details.

TLDR: Our new paper outlines how AI developers should adapt the methodology used in control evaluations as capabilities of LLM agents increase.

Figure: We sketch a trajectory of how control evaluations might evolve through increasingly powerful capability profiles. The trajectory is based on a sequence of five fictional models (M1-M5) with progressively advanced capabilities. For each model, we define an AI Control Level (ACL) based primarily on its threat model-specific capabilities. By mapping these capabilities to appropriate control evaluation methodologies, we provide a systematic approach for adapting safety measures to AI systems as they advance in capabilities.

What are the advantages of AI control?

The field of AI control  attempts to design and evaluate control protocols: safety measures that are robust to intentional subversion by AI systems. Elements of promising protocols vary widely, from secure sandboxing to AI monitors and human auditing.

A particularly attractive feature of the AI control approach is that it offers the potential to mitigate alignment risks without fundamental research breakthroughs. A major focus of the field is on the engineering challenge of designing protocols that meet the challenges of deployments of increasingly capable autonomous agents.

How can AI control techniques scale to increasingly capable systems?

While the academic field of AI control is very young with a key foundational paper only published in 2023, many components of control evaluations (e.g. red teaming) and control measures (e.g. human approval) are already used to address the risks posed by existing AI systems.

As AI systems become more capable of causing harm autonomously, AI developers will need increasingly sophisticated control measures and evaluations to justify that the models can be deployed securely. Our framework provides one possible roadmap for this scaling, proposing five distinct AI control levels (ACLs), ranging from models incapable of competent scheming (ACL-0) to models with scheming capabilities vastly above those of humans (ACL-5). Rather than assuming that agents will always execute the best subversion strategies known to humans, we demonstrate t...
Applications Open for Impact Accelerator Program for Experienced Professionals
by Clark Wisenbaker — 2025-04-14T16:27:32.340Z

High Impact Professionals

(HIP) is excited to announce that applications are now open for the next round of our

Impact Accelerator Program

(IAP). The IAP is a 6-week program designed to equip experienced (mid-career/senior) professionals (not currently working at a high-impact organization) with the knowledge and tools necessary to make a meaningful impact and empower them to start taking actionable steps right away.

To date, the program’s high action focus has proven to be successful:

38 participants (out of 143 in total) from our first four program rounds have transitioned into high-impact careers, through various paths, including founding new charities, starting founding-to-give companies, and joining high-impact organizations such as GiveWell, Charity Entrepreneurship, ML Alignment & Theory Scholars (MATS), Malaria Consortium, and more;
an additional 70 participants (out of 143) are taking concrete actions towards a high-impact career, including skilled volunteering roles at high-impact organizations; and
many participants are donating a meaningful percentage of their annual salary to effective charities, with 18 having taken the 🔶 10% Pledge or 🔷 Trial Pledge – plus 8 additional pledges from the current IAP round.

We’re pleased to open up this new program round, which will start the week of June 16. More information is available below and

here

.

Please apply here by April 27.

Program Objectives

The IAP is set up to help participants:

identify paths to impact,
take concrete, impactful actions, and
join a network of like-minded, experienced, and supportive impact-focused professionals.

At the end of the program, a participant should have a good answer to the question “How can I have the most impact with my career, and what are my next steps?”, and they should have taken the first steps in that direction.

Program Overview

Important Dates

Program duration: 6 weeks (week of June 16 – week of July 21)​
Deadline to apply: April 27 –

Apply Now

Format

Weekly individual work (2–3 hours). A mix of:

Learning: Reading resources on how to think about and prioritize different options for impact
Doing: Taking impactful actions, such as developing your own personalized impact plan and taking concrete steps to begin implementing it

Virtual group sessions (1.5 hours of discussions and coaching)

I

...
The Last Light
by Bridgett Kay — 2025-04-14T15:41:02.745Z

I don’t believe I will survive the coming catastrophe. I cannot put my hope into the future- that the things I love and value will persist; I don’t believe that humanity will survive the coming catastrophe. It’s possible that in the future, another world will bloom rich and full with life that learns to know itself, but I am not confident that such a world will overcome the folly to which our world fell prey. There seems to be a sad law of nature that intelligence comes before wisdom, that wisdom must be learned from errors, and that some errors cannot be corrected. There is a small chance that this new world might develop foresight and robust cooperation before destructive ability. Even so, I do not have confidence that this wise and just world would not fall prey to another world, whose folly has already pierced the stars.

It is true that I love the stars themselves, and the planets, and nebulae, and other celestial wonders. These things will probably persist, and I am somewhat glad that they will persist beyond me. It is also true that what we’re building in our folly will have eyes to see the stars and a mind to comprehend them. But it is not very likely that what we’re building will love the stars on an intrinsic level- not just a utilitarian level- any more than they will love the ocean, the plants, the creatures of this world, or humanity. I’m not confident that what we are building will view these things with anything but cold, empty eyes.

I am confident, however, that I exist. I am confident that I have existed. I am confident that the thing we are building cannot erase what is and what has been, because if it were able to reach into our time, either it would have already done so, and we would not exist, or it has done so and created another branch in time, and our branch remains untouched. Whatever the case, we exist now, in our own pocket of time and space, and that can’t be erased. The thing might take our future, and our world, but it cannot erase us as we are now, and as we were.

I have been able to experience life. I have taken in lungfuls of air and heaved great, satisfying sighs. I have been able to stretch my limbs and feel them tingle with vitality. I have felt the full, Texas sun on my skin, and watched the heat shimmer off pavement and into the air. I have felt cool water on my face, on my parched throat, supporting ...
Offer: Team Conflict Counseling for AI Safety Orgs
by Severin T. Seehrich — 2025-04-14T15:17:00.835Z

Many teams suffer from frictions - whether it's subtle tension or full-blown paralysis. AI safety orgs are no exception. In high-stakes, mission-driven contexts, unspoken tensions and misaligned communication can quietly drain energy, stall progress, and damage morale.

If you’re part of a team or organization in the AI safety space and want support—either to resolve an acute conflict or to build a more resilient team culture—I can help.

About me

I'm Severin. I hold an M.A. in philosophy, and I work as a coach and communication trainer.

My relevant background includes:

Certified counseling trainer (Kooperative Beratung) — including training in a method literally called Team Conflict Counseling, designed for teams in educational settings, but highly transferable.
Certified in Theme-Centered Interaction, a group facilitation method grounded in humanistic psychology.
Several hundred hours of receiving and facilitating relational and inner work modalities, including Circling, Focusing, IFS, Authentic Relating, and Nonviolent Communication.
Writing about facilitation, community building, and other people stuff on LessWrong and the EA Forum.

How I can help

If things are already on fire...

I can support your team in navigating acute conflict, through:

Team-wide facilitation
Mediation between two individuals
1-on-1 coaching for someone caught in a tough situation

I work confidentially and neutrally - my role isn’t to pick sides or impose solutions, but to facilitate clearer understanding, mutual empathy, and conditions where creative resolution can emerge.

If you want to build long-term team health...

I also offer preventive support, such as:

Feedback training and communication workshops
Process consulting to streamline culture and decision-making
Burnout prevention and resilience work (e.g., Active Hope workshops)

Pricing

I know not all valuable work in AI safety is well-funded. So I use a sliding scale:
10€–100€ per hour, depending on your budget.

I care more about impact than invoices - if you think this would help your org, don’t hesitate to reach out.

How to get started?

Fill out this intake form with your name, email, and a short note on what you’re looking for. I'll follow up to schedule a free exploratory call.
Slopworld 2035: The dangers of mediocre AI
by titotal — 2025-04-14T13:14:08.390Z
Link: https://titotal.substack.com/p/slopworld-2035-the-dangers-of-mediocre

None of this article was written with AI assistance.

Introduction

There have been many, many, many attempts to lay out scenarios of AI taking over or destroying humanity. What they tend to have in common is an assumption that our doom will be sealed as a result of AI becoming significantly smarter and more powerful than the best humans, eclipsing us in skill and power and outplaying us effortlessly.

In this article, I’m going to do a twist: I’m going to write a story (and detailed analysis) about a scenario where humanity is disempowered and destroyed by AI that is dumber than us, due to a combination of hype, overconfidence, greed and anti-intellectualism. This is a scenario where instead of AI bringing untold abundance or tiling the universe with paperclips, it brings mediocrity, stagnation, and inequality.

This is not a forecast. This story probably won’t happen. But it’s a story that reflects why I am worried about AI, despite being generally dismissive of all those doom stories above. It is accompanied by an extensive, sourced list of present day issues and warning signs that are the source of my fears.

This post is divided into 3 parts:

Part 1 is my attempt at a plausible sounding science fiction story sketching out this scenario, starting with the decline of a small architecture firm and ending with nuclear Armageddon.

In part 2 I will explain, with sources, the real world current day trends that were used as ingredients for the story.

In part 3 I will analysise the likelihood of my scenario, why I think it’s very unlikely, but also why it has some clear likelihood advantages over the typical doomsday scenario.

The story of Slopworld

In the nearish future:

When the architecture firm HBM was bought out by the Vastum corporation, they announced that they would fire 99% of their architects and replace them with AI chatbots.

The architect-bot they unveiled was incredibly impressive. After uploading your site parameters, all you had to do was chat with a bot to tell it everything you needed from the building, and it would produce a nearly photo-realistic 3d model of a building fitting your requirements, which you could even zoom around to look at from different angles. You could then just talk to the bot, saying things like “could you add another large window to the living room”, and after some time, out would pop another 3d model with the change. Once you were satisf...
Try training token-level probes
by StefanHex — 2025-04-14T11:56:23.191Z

Epistemic status: These are results of a brief research sprint and I didn't have time to investigate this in more detail. However, the results were sufficiently surprising that I think it's worth sharing. I'm not aware of previous papers doing this but surely someone tried this before, I would welcome comments pointing to existing literature on this! [Edit: Orgad et al. (2024) does something related but different, see my comment below.]

TL,DR: I train a probe to detect falsehoods on a token-level, i.e. to highlight the specific tokens that make a statement false. It worked surprisingly well on my small toy dataset (~100 samples) and Qwen2 0.5B, after just ~1 day of iteration! Colab link here.

Context: I want a probe to tell me where in an LLM response the model might be lying, so that I can e.g. ask follow-up questions. Such "right there" probes[1] would be awesome to assist LLM-based monitors (Parrack et al., forthcoming). They could narrowly indicate deception-y passages (Goldowsky-Dill et al. 2025), high-stakes situations (McKenzie et al., forthcoming), or a host of other useful properties (e.g. morality, power, and honesty, as in Zou et al. 2023).

The writeup below is the (lightly edited) result of a brief research sprint I did last month; I added the following appendices before posting it:

In a quick test, the probe doesn’t immediately generalize out of distribution, Appendix B. However I didn’t iterate on the training for this; you could easily make the training data way better and more diverse.
The probe performs better than the mean probe even at detecting statement-level (instead of token-level) falsehoods, Appendix C. This indicates that token-level labels can be helpful no matter your use case.

Summary

Motivation: People are thinking about using probes in control protocols, similar to how you would use a lie detector in an interrogation (e.g. Parrack et al., forthcoming). However, many times when we look at probe scores they are all over the place:

Roleplaying sample 1 and 2 from Goldowsky-Dill et al. 2025

This is very difficult to use, especially for a less-capable LLM. It would be much nicer if the probe lit up bright red where the lie was, and nowhere else. Then the LLM monitor could ask further interrogating questions about the suspicious passage.

Method: I create precise labels by asking Claude Haiku to specify exactly which tokens ...
Monthly Roundup #29: April 2025
by Zvi — 2025-04-14T11:50:02.324Z

In Monthly Roundup #28 I made clear I intend to leave the Trump administration out of my monthly roundups, for both better and worse, outside of my focus areas. Again, this does not mean I don’t have a lot to say or that those questions don’t matter. It means you should not rely on me as your only source of news and I pick my battles.

They are not making this easy.

I am going to stick to my guns. Trade and trading very much inside my focus areas, but for economics roundups, and in extreme cases AI roundups. Besides, you don’t need me to tell you that tariffs not only impose immense economic costs but also fail to achieve their primary policy aims and foster political dysfunction along the way. That question should already be answered by my t-shift. I do have a word about things related to a potential expansion (I can’t believe I’m typing this!) of the Jones Act. And I’ll deal with certain crime-related things when I do my first crime roundup.

Bad News

23andMe is going into bankruptcy. It would seem a wise precaution to download and then delete your data if it’s there, which takes a few days to do, in case the data falls into the wrong hands or is lost forever.

Young men who make 9 figures by default get driven crazy, all checks and balances on them now gone.

This graphic is quite good.

That’s a variation on this classic, worth revisiting periodically as a reminder:

A claim that banning smoking in bars increases alcohol consumption by ~5% without decreasing smoking. I presume the increased alcohol consumption is because the bar became a much better experience without all the smoking? It seems bizarre that this wouldn’t decrease smoking, especially over the long term.

Beware communities that encourage irresponsible risk taking and dismiss those who do not endanger themselves. It can be good if targeted well: There are places, like founding startups and putting yourself out there for romance, where people take far too little risk and it is often good to encourage people to take more. But this very much doesn’t apply to, for example, talk about financial investments.

Antisocial Media

If you use Twitter via the For You page, You Fool. Yet many of you do exactly that.

I even hear people complaining about ‘the algorithm’ without doing the obvious and switching to chronological feeds and lists. That’s on you.

As far as I ...
A Solution to Sandbagging and other Self-Provable Misalignment: Constitutional AI Detectives
by Knight Lee — 2025-04-14T10:27:24.903Z

Inspired by Eli Tyre's idea.

Sandbagging and other "Self-Provable Misalignment" by an AI, can be spotted by an adversarial copy of itself. The adversarial copy of itself works as an "detective," and its reward is defined by how much evidence of misalignment it can prove the existence of, to human evaluators.

Even a misaligned goal cannot afford to get in the way of maximizing reward. As soon as a goal gets in the way of maximizing reward, it is walking into whirling blades, and allowing reinforcement learning to train itself into nonexistence.[1]

There are two detectives: the Constitutional AI Detective, and the Mindreader Detective.

The Constitutional AI Detective

The Constitutional AI Detective is used to train the AI to be corrigible, honest, and "aligned."

Unlike the Mindreader Detective, it doesn't read the AI's chain-of-thought and only sees its output. This is because using the chain-of-thought for alignment training is the forbidden technique, and trains the chain-of-thought to fake alignment.

Not being able to mind-read, isn't necessarily as restrictive as you think. After all, the Constitutional AI Detective is still an adversarial copy of the AI, and knows all of its tricks. It knows what it would do if it wanted to do cheat and lie, and how that differs from what it would do if it didn't.

"Constitutional AI Sufficiency Argument"

Whether Constitutional AI works, hinges on two key premises:

The AI's capabilities on the task of evaluating its own corrigibility/honesty, is sufficient to train itself to remain corrigible/honest (assuming it starts off corrigible/honest enough to not sabotage this task).
It starts off corrigible/honest enough to not sabotage this task.

Logically think about it. If these two premises are true, Constitutional AI will keep AI corrigible/honest, even to superintelligence!

This Argument begs the question, how do we improve it's capabilities in evaluating its own corrigibility/honesty?

Trying to answer this important question is how I arrived at Constitutional AI Detectives.

The state of the art Constitutional AI asks the AI to write a critique of its own output, without rewarding it for proving the existence of evidence for misalignment. This method is okay, assuming there's no capabilities RL.

Once you add in capabilities RL, Constitutional AI can fall behind. The AI's capabilities in solving problems can increase without bound, as RL tak...