trentmkelly/LessWrong-43k huggingface dataset:
Many Common Problems are NP-Hard, and Why that Matters for AI
by Andrew Keenan Richardson — 2025-03-26T21:51:17.960Z

Crosspost from my blog: https://mechanisticmind.substack.com/p/many-common-problems-are-np-hard

What problems will be left for superintelligence? AI models are quickly getting better, but certain important and very ordinary challenges are surprisingly, stubbornly hard. While large language models have made some previously difficult computational tasks trivial, a fundamental class of problems that challenge classical computers will likely remain inherently difficult even for advanced AI.

No matter how smart AI gets, there are some problems that defy efficient solutions. Even superhuman intelligences will struggle with them.

Formally, these challenges are called NP-hard, meaning they fundamentally resist efficient solutions. What makes this significant is how many familiar, everyday situations fall into this category - from planning vacations to organizing events.

When humans tackle these challenges, we typically dedicate some effort, then accept a 'good enough' outcome. Additional contemplation might improve our solution, but we can never be certain we've found the optimal answer - unlike other tasks where perfect solutions are verifiably achievable.

Fortunately, effective heuristics (like greedy algorithms) often yield good-enough results. LLMs excel at applying these heuristics, allowing them to perform reasonably well on such problems. However, heuristics have fundamental limits; finding truly optimal solutions requires exponentially more computational effort. That’s why we’re seeing inference-time compute scaling:

It pays to think. Spending more tokens on thinking yields better answers.

But notice that the x-axis is a log scale! A user can only double compute so many times before they run out of money.

This distinction between problems has a lot of significance for designing AI systems. Notice what you need to solve, and you can have clearer expectations and better understanding of where to allocate resources.

Background: P vs NP

For readers already familiar with computational complexity, you'll recall that solving NP-hard problems requires computation time that grows exponentially with size. Unlike easier problems that yield to clever optimizations, these are fundamentally difficult.

Unless P=NP (considered highly unlikely by most computer scientists), optimal solutions to NP-hard tasks will always be computationally intensive. This isn't merely an engineering limi...
Fun With GPT-4o Image Generation
by Zvi — 2025-03-26T19:50:03.270Z

Google dropped Gemini Flash Image Generation and then Gemini 2.5 Pro, so of course to ensure Google continues to Fail Marketing Forever, OpenAI suddenly dropped GPT-4o Image Generation.

Zackary Nado (Research Engineer, DeepMind): It wouldn’t be a Gemini launch without an OAI launch, congratulations to the team! It’s awesome they were able to de-risk the model coincidentally just in time.

Everyone agrees: Google Flash Image Generation was cool. Now it isn’t cool, because GPT-4o Image Generation is cooler.

What people found this new image generator can do exceptionally well is interpretation, transformation and specific details including text. The image gets to ‘make sense’ and be logically coherent, in a way older ones weren’t.

Today is mostly a fun day about a fun collection of images.

Table of Contents

The Pitch.

A Blind Taste Test.

We’re Cracked Up All the Censors.

I’m Too Sexy.

Can’t Win Them All.

Can Win Others.

Too Many Words.

This is the Remix.

More Neat Tricks.

They Had Style, They Had Grace.

Form of the Meme.

Form of the Altman.

Go Get That Alpha.

The Pitch

OpenAI: 4o image generation has arrived.

It’s beginning to roll out today in ChatGPT and Sora to all Plus, Pro, Team, and Free users. Available soon for Enterprise and Edu, as well as for developers using the API.

GPT-4o image generation excels at accurately rendering text, precisely following prompts, and leveraging 4o’s inherent knowledge base and chat context.

GPT-4o can build upon images and text in chat context, ensuring consistency throughout.

GPT‑4o’s image generation follows complex prompts with attention to detail.

Creating and customizing images is as simple as chatting using GPT‑4o—just describe what you need, including any specifics like aspect ratio, exact colors using hex codes, or a transparent background.

Create or transform images into a variety of styles with 4o image generation.

Sam Altman:

It’s an incredible technology/product. I remember seeing some of the first images come out of this model and having a hard time they were really made by AI. we think people will love it, and we are excited to see the resulting creativity. Congrats to our researchers @gabeeegoooh @prafdhar @ajabri @eliza_luth @kenjihata @dmed256.

This represents a new high-water mark for us in allowing creative freedom. people are going to create some really amazing stuff and some stuff t

...
I'm hiring a Research Assistant for a nonfiction book on AI!
by garrison — 2025-03-26T19:46:43.923Z
Link: https://garrisonlovely.substack.com/p/research-assistant-job-description

I'm

Garrison Lovely

, a freelance journalist with work in NYT, Nature, BBC, TIME, and elsewhere. I report on the intersection of economics, geopolitics, and artificial intelligence. I'm also a

Reporter in Residence

at the Omidyar Network and publisher of

Obsolete

, a fast-growing Substack on AI.

Overview

What: Support the completion of a forthcoming general-audience nonfiction book on AI. You'll conduct research, draft chapters, edit existing content, and help transform an incomplete manuscript into a polished final product for publication by OR Books and The Nation Magazine.

Why: The project, Obsolete: Power, Profit, and the Race to Build Machine Superintelligence, has the potential to be the go-to AI risk book in the post-ChatGPT era. You’ll also get the opportunity to work closely with an experienced journalist and writer who has a track record of publishing work in leading outlets (NYT, Nature, BBC, TIME, Foreign Policy, etc.).

Start Date: As soon as possible

Employment Type: Fixed term (6 months, with possibility of extension); full time

Location: Remote, provided at least three hours overlap between 10am-6pm ET.

Compensation: $37,485-$65,442 total compensation for the 6-month period, depending on experience and location

Applications will be considered on a rolling basis, but please apply as soon as you can. Priority may be given to applicants who can move through the process sooner.

Apply here

Full job description here
Automated Researchers Can Subtly Sandbag
by gasteigerjo — 2025-03-26T19:13:26.879Z
Link: https://alignment.anthropic.com/2025/automated-researchers-sandbag/

tl;dr

When prompted, current models can sandbag ML experiments and research decisions without being detected by zero-shot prompted monitors. Claude 3.5 Sonnet (new) can only sandbag effectively when seeing a one-shot example, while Claude 3.7 Sonnet can do this without an example (zero-shot). We are not yet worried about sabotage in today's models, as the sandbagging we observe today would be noticed by humans.

Twitter thread about the blog post here.

Introduction

AI models might soon assist with large parts of AI and AI safety research, completing multiple hours of work independently. These automated researchers will likely not be sufficiently capable to cause immediate catastrophic harm. However, they will have the opportunity to engage in sabotage, including sabotaging the research we rely on to make more powerful AI models safe. For example, a malicious automated researcher might slow down AI safety research by purposefully making promising experiments fail.

In this work we aim to ensure that automated researchers are not sabotaging research without detection. It may be hard to determine whether a powerful model is aligned or simply faking alignment. We instead pessimistically assume that an automated researcher is actively misleading our research. We defend against the automated researcher’s sabotage by adding a second model that monitors the automated researcher’s outputs for signs of manipulation. We then explore which strategies the saboteur could use to bypass these defenses, and how good current models are at implementing these strategies.

We use a strong, untrusted model to act as the potentially malicious automated researcher. This simulates a worst-case scenario where the alignment of a deployed state-of-the-art model is compromised despite pre-deployment testing. If we use the misaligned automated researcher to monitor and flag flawed research, it might sabotage these predictions as well, so we use weaker models as monitors. Weaker models are more trustworthy because they stem from an earlier, time-tested generation of models and because misalignment and alignment faking might only emerge with greater model capabilities. Our approach is similar to recent work on AI Control.

We study two broad classes of strategies a malicious AI system could use to sabotage research:

In experiment sandbagging, a malicious automated researcher tries

...
Negative Results for SAEs On Downstream Tasks and Deprioritising SAE Research (GDM Mech Interp Team Progress Update #2)
by lewis smith — 2025-03-26T19:07:48.710Z
Link: https://deepmindsafetyresearch.medium.com/negative-results-for-sparse-autoencoders-on-downstream-tasks-and-deprioritising-sae-research-6cadcfc125b9

Lewis Smith*, Sen Rajamanoharan*, Arthur Conmy, Callum McDougall, Janos Kramar, Tom Lieberum, Rohin Shah, Neel Nanda

* = equal contribution

The following piece is a list of snippets about research from the GDM mechanistic interpretability team, which we didn’t consider a good fit for turning into a paper, but which we thought the community might benefit from seeing in this less formal form. These are largely things that we found in the process of a project investigating whether sparse autoencoders (SAEs) were useful for downstream tasks, notably out-of-distribution probing.

TL;DR

To validate whether SAEs were a worthwhile technique, we explored whether they were useful on the downstream task of OOD generalisation when detecting harmful intent in user prompts
Negative result: SAEs underperformed linear probes

Corollary: Linear probes are actually really good and cheap and perform great

As a result of this and parallel work, we are deprioritising fundamental SAE research for the moment and exploring other directions, though SAEs will remain a tool in our toolkit

We do not think that SAEs are useless or that no one should work on them, but we also do not think that SAEs will be a game-changer for interpretability, and speculate that the field is over-invested in them.

Training SAEs specialised for chat data closed about half the gap but was still worse than linear probes

We tried several ways to train chat SAEs, all did about as well. By default, we recommend taking an SAE on pretraining data and finetuning it on a bit of chat data

Other results:

We found SAEs fairly helpful for debugging low quality datasets (noticing spurious correlations)
We present a variant of JumpReLU with an alternative sparsity penalty to get rid of high-frequency latents
We argue that a standard auto-interp approach of computing the average interpretability of a uniformly sampled SAE latent can be misleading as it doesn’t penalise models which have high frequency, but not very interpretable, latents, and explore weighting the interpretability score by latent frequency.

Introduction

Motivation

Our core motivation was that we, along with much of the interpretability community, had invested a lot of our energy into Sparse Autoencoder (SAE) research. But SAEs lack a ground truth of the “true” features in language models to compare to, making it pretty unclear how well they work. There is qualitative e...
AI companies should be safety-testing the most capable versions of their models
by sjadler — 2025-03-26T19:03:41.790Z
Link: https://stevenadler.substack.com/p/ai-companies-should-be-safety-testing

AI companies should be safety-testing the most capable versions of their models
Only OpenAI has committed to the strongest approach: testing task-specific versions of its models. But evidence of their follow-through is limited.
You might assume that AI companies run their safety tests on the most capable versions of their models possible. But there are important versions that currently go untested: task-specific versions, specially trained to demonstrate how far a model can be pushed on a dangerous ability, like novel bioweapon design. Without exploring and evaluating these versions, Al companies are underestimating the risks their models could pose.

Only OpenAI (where I previously worked

1

) has committed to what I consider to be the strongest vision for safety-testing: evaluating these task-specific versions of its models to understand the worst-case scenarios. I think OpenAI’s vision on this is truly laudable and great. But from my review of publicly available reports, it is not clear that OpenAI is in fact following through on this commitment. I believe that all leading AI companies should be doing this form of task-specific model testing - but if companies are not willing or able to today, I argue there are many middle-ground improvements they could pursue.
In this post, I will:

Walk you through why AI companies safety-test their models, and how they do it
Define task-specific fine-tuning (TSFT), and argue why it’s important for safety-testing
Summarize the status of TSFT at leading AI companies
Dig into a case study on OpenAI and TSFT
Outline challenges to safety testing using TSFT, and what middle-ground improvement exists
[ continues on Substack ]
Conceptual Rounding Errors
by Jan_Kulveit — 2025-03-26T19:00:31.549Z
Link: https://boundedlyrational.substack.com/p/conceptual-rounding-errors

Epistemic status: Reasonably confident in the basic mechanism.

Have you noticed that you keep encountering the same ideas over and over? You read another post, and someone helpfully points out it's just old Paul's idea again. Or Eliezer's idea. Not much progress here, move along.

Or perhaps you've been on the other side: excitedly telling a friend about some fascinating new insight, only to hear back, "Ah, that's just another version of X." And something feels not quite right about that response, but you can't quite put your finger on it.

I want to propose that while ideas are sometimes genuinely that repetitive, there's often a sneakier mechanism at play. I call it Conceptual Rounding Errors – when our mind's necessary compression goes a bit too far .

Too much compression

A Conceptual Rounding Error occurs when we encounter a new mental model or idea that's partially—but not fully—overlapping with a familiar one, and our mind helpfully simplifies it down to the familiar. Just as numerical rounding smooths away precision, conceptual rounding quietly erases crucial differences. Let's start with an example.

No, This Isn't The Old Demons Story Again

Possibly the worst case is a cluster of overlapping but distinct models of what may be the problem inside of AIs: "mesa-optimizers," "optimization demons," "subagents," and "inner alignment."

Mesa-optimizers: Learned optimizers arising within AI training because often the easiest solution to a problem is to find an optimizer.
Optimization demons: Optimizers arise in dynamical systems under strong optimization pressures.
Subagents: Distinct internal agent-like parts with possibly conflicting objectives.
Inner alignment: Unclear. I would prefer it to mean a general concern with aligning an AI's internal structure.

At different points in time, different frames were in the spotlight, while others suffered from being rounded to the locally salient ones.

Compressing these into one of the frames leads to serious misunderstandings. For example, it seems fairly natural to start with an AI system with partially conflicted subagents or for such conflict to arise from learning internal conflicts from humans. Rounding this to 'mesa-optimizers' makes it very difficult to reason about—and vice versa.
My guess is due to the conceptual rounding errors, only a minority of alignment researchers has coherent versions of all the overlapping fra...
Personal Agents: The First Step in Emergent AI Society
by Andrey Seryakov — 2025-03-26T18:55:26.358Z

Illustration by ChatGPT 4o

Writing the AI Society paper, I thought about AI agents as bots we send online for our tasks, but now I believe our personal AIs will be constantly online. As we are always on the mobile net so people may reach us, the agents will be online so other people agents may reach them. Consider them as personal assistants who work 24/7, know a lot about you, and help with routine tasks.

For example, you want to make a dinner for your friends, some of them may have allergies, some hate certain foods, and so on. How would you do now? You will need to message each of them and directly ask. What may happen in the future? You just ask your agent to do it, and it will communicate with agents of your friends producing you a list of things you should avoid and even a list of recipes from your cookbook. Done instantly, agents are always online and ready to answer. Health issues, financial situation, music preferences, geolocation, social connections... such agents may become shadows of us knowing everything.

These personal AI assistants create powerful new incentives for agents to cooperate:

1. Persistent Relationships: Unlike task-specific bots, personal assistants will form ongoing relationships with other agents, making the creation of trust networks much more plausible.

2. Information Exchange Protocols will already be available for them. As Wright argues in the "Here's Charlie!" paper, agents would initially need structured protocols for sharing sensitive data to ensure a secure exchange.

3. Trust Verification Systems: Personal agents will already have mechanisms provided by developers to verify which sources are authoritative for specific types of information. You may want to share with your doctor's agent the health data, but not with your insurance one. Or you want to be sure that your agent is buying you a ticket not from a fishing site.

4. Multi-Level Negotiations: Naturally there will be a lot of situations when multiple users' interests must be balanced (like in the dinner example), and agents will need to engage in complex negotiations representing their users' preferences, constraints, and priorities. All social structures need the ability of their members to find common ground.

The technical side for such agents is already emerging. Sure, Alignment (if your agent breaks a law, ...
Will AI R&D Automation Cause a Software Intelligence Explosion?
by Tom Davidson — 2025-03-26T18:12:02.524Z
Link: https://www.forethought.org/research/will-ai-r-and-d-automation-cause-a-software-intelligence-explosion

Empirical evidence suggests that, if AI automates AI research, feedback loops could overcome diminishing returns, significantly accelerating AI progress.

Summary

AI companies are increasingly using AI systems to accelerate AI research and development. These systems assist with tasks like writing code, analyzing research papers, and generating training data. While current systems struggle with longer and less well-defined tasks, future systems may be able to independently handle the entire AI development cycle – from formulating research questions and designing experiments, to implementing, testing, and refining new AI systems.

Some analysts have argued that such systems, which we call AI Systems for AI R&D Automation (ASARA), would represent a critical threshold in AI development. The hypothesis is that ASARA would trigger a runaway feedback loop: ASARA would quickly develop more advanced AI, which would itself develop even more advanced AI, resulting in extremely fast AI progress – an “intelligence explosion.”

Skeptics of an intelligence explosion often focus on hardware limitations – would AI systems be able to build better computer chips fast enough to drive such rapid progress? However, there’s another possibility: AI systems could become dramatically more capable just by finding software improvements that significantly boost performance on existing hardware. This could happen through improvements in neural network architectures, AI training methods, data, scaffolding around AI systems, and so on. We call this scenario a software intelligence explosion (SIE). This type of advancement could be especially rapid, since it wouldn’t be limited by physical manufacturing constraints. Such a rapid advancement could outpace society’s capacity to prepare and adapt.

In this report, we examine whether ASARA would lead to an SIE. First, we argue that shortly after ASARA is developed, it will be possible to run orders of magnitude more automated AI researchers than the current number of leading human AI researchers. As a result, the pace of AI progress will be much faster than it is today.

Second, we use a simple economic model of technological progress to analyze whether AI progress would accelerate even further. Our analysis focuses primarily on two countervailing forces. Pushing towards an SIE is the positive feedback loop from increasingly powerful AI systems perfor...
Why Does Unemployment Happen?
by Nicholas D. — 2025-03-26T18:02:51.433Z
Link: https://nicholasdecker.substack.com/p/why-does-unemployment-happen

And specifically, what does this imply for AI? There are two theories of equilibrium unemployment — search frictions, and efficiency wages — and they actually give diametrically opposite predictions for when search frictions in finding a new job fall. I conclude that frictions are the more likely explanation, but that LLMs may actually increase unemployment if our ability to apply exceeds the ability to sort.