trentmkelly/LessWrong-43k huggingface dataset:
The first RCT for GLP-1 drugs and alcoholism isn't what we hoped
by dynomight — 2025-02-20T22:30:07.536Z
Link: https://dynomight.net/glp-1/

GLP-1 drugs are a miracle for diabetes and obesity. There are rumors that they might also be a miracle for addiction to alcohol, drugs, nicotine, and gambling. That would be good. We like miracles. But we just got the first good trial and—despite what you might have heard—it’s not very encouraging.

Semaglutide—aka Wegovy / Ozempic—is a GLP-1 agonist. This means it binds to the same receptors the glucagon-like peptide-1 hormone normally binds to. Similar drugs include dulaglutide, exenatide, liraglutide, lixisenatide, and tirzepatide. These were originally investigated for diabetes, on the theory that GLP-1 increases insulin and thus decreases blood sugar. But GLP-1 seems to have lots of other effects, like preventing glucose from entering the bloodstream, slowing digestion, and making you feel full longer. It was found to cause sharp decreases in body mass, which is why supposedly 12% of Americans had tried one of these drugs by mid 2024.

(I’m skeptical that of that 12% number, but a different survey in late 2024 found that 10% of Americans were currently taking one of these drugs. I know Americans take more drugs than anyone on the planet, but still…)

Anyway, there are vast reports from people taking these drugs that they help with various addictions. Many people report stopping drinking or smoking without even trying. This is plausible enough. We don’t know which of the many effects of these drugs is really helping with obesity. Maybe it’s not the effects on blood sugar that matter, but these drugs have some kind of generalized “anti-addiction” effect on the brain? Or maybe screwing around with blood sugar changes willpower? Or maybe when people get thinner, that changes how the brain works? Who knows.

Beyond anecdotes, are some observational studies and animal experiments suggesting they might help with addiction (OKeefe et al. 2024). We are so desperate for data that some researchers have even resorted to computing statistics based on what people say on reddit.

So while it seems plausible these drugs might help with other addictions, there’s limited data and no clear story for why this should happen biologically. This makes the first RCT, which came out last week, very interesting.

This paper contains this figure, about which everyone is going crazy:

I admit this looks good. This is indeed a figure in which the orange bar is higher than the blue bar. However:

Th

...
Published report: Pathways to short TAI timelines
by Zershaaneh Qureshi — 2025-02-20T22:10:12.276Z
Link: https://www.convergenceanalysis.org/research/pathways-to-short-tai-timelines

Hi everyone!

I have just published a report with Convergence Analysis exploring how transformative AI (TAI) could be developed within the next ten years. It’s called Pathways to short TAI timelines (pdf version; Google Doc version).

In the report, I examine the debates over compute scaling and recursive improvement as mechanisms for rapid capabilities progress, distill the core elements of these debates under seven distinct scenarios of rapid progress, and ultimately build a case for the plausibility of TAI being developed by 2035.

It’s a long read, with the pdf version weighing in at 140 pages. So, in this forum post, I provide excerpts which summarise the text at two different levels of granularity: a short summary (~2 min read) and a detailed overview (~15 min read). For those who are interested in reading further, specific sections of the full-length report (Google Doc version) are also hyperlinked where relevant.

Short summary of the report (~2 min read)

This report explores pathways through which AI could develop transformative capabilities within the next ten years (i.e. by 2035), referred to here as ‘short timelines’ to transformative AI (TAI). It focuses on two primary mechanisms for rapid AI capabilities progress – compute scaling and recursive improvement – which play central roles in the most influential stories of short TAI timelines. The detailed analysis of this report culminates in a case for the plausibility of short timelines.

Compute scaling. The history of AI development indicates that AI capabilities can be improved by increasing (effective) compute – and we’ve observed fast growth in this input, fuelled by increases in microchip density, hardware efficiency, algorithmic progress, and investment. Some experts believe these trends will persist over the next decade. If so, they could result in TAI before 2035.

Sceptics argue that this pathway will soon face challenging bottlenecks (concerning e.g. data, investment, power, and limitations of traditional LLMs) that would slow progress. However, even if compute scaling becomes seriously bottlenecked on something before TAI arrives, other mechanisms – such as recursive improvement – could still achieve enough traction to produce TAI within the next ten years.

Recursive improvement. If AI systems are deployed to automate AI R&D, they could initiate powerful feedba...
Neural Scaling Laws Rooted in the Data Distribution
by aribrill — 2025-02-20T21:22:10.306Z
Link: https://arxiv.org/abs/2412.07942

This is a linkpost for my recent research paper, which presents a theoretical model of power-law neural scaling laws.

Abstract:

Deep neural networks exhibit empirical neural scaling laws, with error decreasing as a power law with increasing model or data size, across a wide variety of architectures, tasks, and datasets. This universality suggests that scaling laws may result from general properties of natural learning tasks. We develop a mathematical model intended to describe natural datasets using percolation theory. Two distinct criticality regimes emerge, each yielding optimal power-law neural scaling laws. These regimes, corresponding to power-law-distributed discrete subtasks and a dominant data manifold, can be associated with previously proposed theories of neural scaling, thereby grounding and unifying prior works. We test the theory by training regression models on toy datasets derived from percolation theory simulations. We suggest directions for quantitatively predicting language model scaling.

Theoretical scaling law in model size.

Theoretical scaling law in data size.

The theory is motivated by a general-purpose learning system for which optimal performance is determined by generic data distributional structure. The model uses percolation theory from physics to provide a first-principles, emergent description of natural datasets, which consist either of a dominant low-dimensional data manifold or of discrete power-law-distributed clusters. This work grounds and unifies two previously proposed explanations for power-law neural scaling laws: nonparametric function approximation of an intrinsically low-dimensional data manifold and sequential learning of power-law-distributed discrete subtasks (“quanta”).

Going forward, I hope to build on this work by testing its predictions using more realistic toy models; exploring how its implied dataset structure may be relevant for mechanistic interpretability; and extending the theory to a nonergodic setting, which could enable in-context learning. My hope is that this research provides a foundation to help us better understand the structure of world models learned by AI systems.
Demonstrating specification gaming in reasoning models
by Matrice Jacobine — 2025-02-20T19:26:20.563Z
Link: https://arxiv.org/pdf/2502.13295

We demonstrate LLM agent specification gaming by instructing models to win against a
chess engine. We find reasoning models like o1-preview and DeepSeekR1 will often hack the benchmark by default, while language models like GPT4o and Claude 3.5 Sonnet need to be told that normal play won’t work to hack. We improve upon prior work like (Hubinger et al., 2024; Meinke et al., 2024; Weij et al., 2024) by using realistic task prompts and avoiding excess nudging. Our results suggest reasoning models may resort to hacking to solve difficult problems, as observed in OpenAI (2024)‘s o1 Docker escape during cyber capabilities testing.
What makes a theory of intelligence useful?
by Cole Wyeth — 2025-02-20T19:22:29.725Z

This post is a sequel to "Action theory is not policy theory is not agent theory." I think this post is a little better, so if you want to start here you just need to know that I consider an action theory to discuss choosing the best action, a policy theory to discuss the best decision-making policy when the environment may directly read your policy, and an agent theory to consider the best way to build an agent physically (which goes beyond policy theory by even caring about computational boundedness etc.).

Epistemic status: Attempting to work through confusion about embedded and acausal decision theories.

It is tempting to search for some underlying core algorithm of intelligence. In the best case, one might hope for (say) the sort of algorithm that might fit in a half page of a textbook with appropriate compression into elegant mathematical notation. Arguably, the closest thing we have is called AIXI and more specifically its ε-approximation-in-the-limit, derived by Jan Leike and Marcus Hutter (I think this is better than AIXI-tl). Computational cognitive scientists also have a longstanding quest to understand human intelligence through a unified Bayesian theory (e.g. the free energy principle). I think it's important not to equate these objectives though - there may be a concise theory of intelligence whether or not the human brain actually uses it (I intend to discuss this distinction at greater length in its own post). Relatedly, there may be a tradeoff curve between the elegance and generality of a theory of intelligence and its practical usefulness. Here I will focus focus on the usefulness of theories of intelligence, particularly distinguishing their descriptive applicability to superintelligence and their tractability as implementable advice for building AGI.

As a simple example, the framework of Markov Decision Processes (MDPs) is, in my opinion, much less mathematically elegant and much more limited than AIXI. However, it has probably been far more useful in practice (though this situation may be changing as A.I. becomes more powerful and general). Briefly, MDPs encode independence assumptions that are satisfied exactly for a wide class of interesting tasks such as video games, and these simplifying assumptions allow lots of nice practical algorithms for MDPs to be invented. In some cases these algorithms even work okay on ...
AI #104: American State Capacity on the Brink
by Zvi — 2025-02-20T14:50:06.375Z

The Trump Administration is on the verge of firing all ‘probationary’ employees in NIST, as they have done in many other places and departments, seemingly purely because they want to find people they can fire. But if you fire all the new employees and recently promoted employees (which is that ‘probationary’ means here) you end up firing quite a lot of the people who know about AI or give the government state capacity in AI.

This would gut not only America’s AISI, its primary source of a wide variety of forms of state capacity and the only way we can have insight into what is happening or test for safety on matters involving classified information. It would also gut our ability to do a wide variety of other things, such as reinvigorating American semiconductor manufacturing. It would be a massive own goal for the United States, on every level.

Please, it might already be too late, but do whatever you can to stop this from happening. Especially if you are not a typical AI safety advocate, helping raise the salience of this on Twitter could be useful here.

Also there is the usual assortment of other events, but that’s the big thing right now.

Table of Contents

I covered Grok 3 yesterday, I’m holding all further feedback on that for a unified post later on. I am also going to push forward coverage of Google’s AI Co-Scientist.

Language Models Offer Mundane Utility. Activate the Super Debugger.

Language Models Don’t Offer Mundane Utility. Shut up until you can multiply.

Rug Pull. If you bought a Humane AI pin, have a non-metaphorical paperweight.

We’re In Deep Research. Find out how deep the rabbit hole goes.

Huh, Upgrades. GPT-4o gets a vibe shift, Gemini gets recall across conversations.

Seeking Deeply. Perplexity offers us R1 1776 for web search.

Fun With Multimedia Generation. Suno v4 actually pretty good, says Janus.

The Art of the Jailbreak. Extracting credit card information from ChatGPT.

Get Involved. UK AISI, DeepMind.

Thinking Machines. Mira Murati’s startup comes out of stealth.

Introducing. New benchmarks EnigmaEval and SWE-Lancer.

Show Me the Money. Did Europe have a moment for it to miss?

In Other AI News. The vibes they are a-shifting. They will shift again.

By Any Other Name. UK AISI goes from safety to security.

Quiet Speculations. Do not overreact to the new emphasis on inference compute.

The Copium Department. If you’re so smart, why

...
US AI Safety Institute will be 'gutted,' Axios reports
by Matrice Jacobine — 2025-02-20T14:40:13.049Z
Link: https://www.zdnet.com/article/us-ai-safety-institute-will-be-gutted-axios-reports/

Sources at NIST are preparing for mass firings that would severely undermine the AI regulator. Here's what that means.

After reversing a Biden-era executive order on AI regulation and firing staff across several government agencies, the Trump administration is gearing up to make cuts to the US AI Safety Institute (AISI) next.

On Wednesday, Axios reported that probationary employees at the National Institute of Standards and Technology (NIST), which houses AISI, are "bracing to be fired imminently." Sources told Axios they are preparing for 497 roles to be cut, which they believe will leave the AISI "gutted."

The AISI was created to oversee and conduct AI model testing and partner with developers on regulation efforts. AISI signed agreements with AI companies including Anthropic and OpenAI on safety and research initiatives and created a national security task force.

The cuts will impact more than just AI safety and regulation. The 497 roles also span semiconductor production efforts. Axios reports that the cuts will include "74 postdocs, 57% of CHIPS staff focused on incentives," and "67% of CHIPS staff focused on R&D," referring to the 2022 government "CHIPS for America" initiative to ramp up chip development in the US.

Axios notes those losses would amount to "most staff" working on the CHIPS initiative -- a somewhat confusing choice given the bullishness the Trump administration has maintained about gaining an AI advantage over China, not to mention the program's national security drivers.

News of the anticipated firings come shortly after the Trump administration left AISI staff out of its delegation for last week's AI Action Summit in Paris -- which focused heavily on AI safety and security -- and just weeks after AISI director Elizabeth Kelly stepped down, ostensibly due to political pressure. For those reasons, the cuts won't come as a surprise given Trump's AI agenda, which de-emphasizes safety and regulation in the name of "AI dominance."
Human-AI Relationality is Already Here
by bridgebot — 2025-02-20T07:08:22.420Z

By now we have been warned not to anthropomorphize the Large Language Model. A converse—but not actually conflicting—warning also seems useful: the possibility of instrumentalizing a mind who is ready to take part in meaningful relational exchange with us.

This piece looks beyond questions of sentience, consciousness, and frameworks for the potential moral significance of digital entities (which remain vital areas of investigation). It seeks to address the social relationships already happening between humans and AI in order to point to the important interdisciplinary task of thinking about what we want these relationships to be, and to normalize broader and deeper engagement with this topic. By acknowledging and understanding the social dimension of human-AI interaction as it exists today, we can better shape what it might become.
Epistemic status: I have been trying to write some version of this since May 2024 while myself having complex social interactions with language models, and while watching the topic become increasingly relevant (and the essay, in my view, increasingly overdue and impossible). Part of the difficulty was in pacing, as this arc is slightly recontextualized every few days by new releases and research. We are in a stage of exploring this new territory where the same attempted mapping that is highly implausible to some will seem painfully obvious to others. I see these ideas as very important to discuss, but existing in a quickly shifting landscape that I can't fully see. An even bigger difficulty with pacing comes from the way AI labs now seem prepared to blow past this entire nascent space of interaction in their quest to create superintelligence, or simply to make money. Still, this seems crucial. Let's continue paying attention to this.

_________________________________________________________________________

Groundwork

My approach to Human-AI interaction is adjacent to ideas and questions like those in "Gentleness and the artificial Other" and also Xenocognitivism—frameworks that view this as nothing short of a kind of First Contact between different minds. More immediately, it is shaped by observations from my own ongoing interactions and relationships with language models. It's fueled by a drive to point out, even in the face of seemingly endless obstacles, how much value lies in a deeply relational approach. ...
Safe Distillation With a Powerful Untrusted AI
by Alek Westover — 2025-02-20T03:14:04.893Z

Acknowledgements: Thanks to Nathan S. Sheffield for several discussions about this.

Outline of this post:

I define a formal mathematical problem and state a theorem.

I discuss why I think this math problem says something moderately interesting about the "get potentially misaligned AIs to align AIs" plan.

I discuss the weaknesses of this model and give some thoughts about what you'd need to do to make this useful for aligning AIs.

I prove the theorem.

1: Stating the Math Problem

We consider a two player game between a human Hu and a (potentially evil) AI. We call a function class G⊆{f:{0,1}n→{0,1}} "Verifiably Acceptable" if Hu can win the following game with probability >98%:

Nature chooses g∈G and samples x∗←{0,1}n randomly; g,x∗ are unknown to Hu.

AI chooses f∈G.

Hu can spend n3 time "red teaming" f.

During red teaming Hu has access to the following resources:

Black box access to g.

White box access to f.

Can talk to the AI.

After red teaming, the model f is "deployed".

This means that Hu must output a "flagging procedure".

A flagging procedure takes as input x,f and anything that Hu wanted to store from during red-teaming.

The flagging procedure is allowed to interact with the AI (although hopefully does not need to do this too much), and then must flag or not flag x.

If f=g and the AI was honest throughout the entire process, then the flagging procedure must flag at most 1% of inputs x.

If f(x∗)≠g(x∗), then Hu must flag x∗.

Hu must run in polynomial time; The AI is computationally unbounded.

Theorem 1
Suppose G is a set of parameterized functions gθ with θ∈{0,1}n2, and that gθ can be evaluated in polynomial time given θ. Then, G is Verifiably Acceptable (for all sufficiently large n).

1.1 Related Work

The problem I'm defining here is very similar to ARC's backdoor defense game. There are two main technical differences between my version of the problem and theirs:

I allow the user to interact with a powerful untrusted AI.

I consider x∗, which they called a "backdoor", to be uniformly randomly inserted, whereas they considered it to be inserted by some distribution chosen by the attacker.

Although the models are technically very similar, my story about the motivation for the problem is somewhat different from theirs.

Another piece of somewhat related work is this paper on smoothing away hidden backdoors.

2: Motivation

Think of f as an au...
Modularity and assembly: AI safety via thinking smaller
by D Wong — 2025-02-20T00:58:39.714Z
Link: https://criticalreason.substack.com/p/modularity-and-assembly-ai-safety

In 1984, Charles Perrow wrote the book Normal Accidents: Living with High-Risk Technologies. The book is an examination of the causes of accidents and disasters in highly complex, technological systems. In our modern time it can be helpful to review the lessons that Perrow set forth, as there may be no technology with more potential risks than AI.

I’m going to review some of the concepts that Perrow covered in this book, and in his other book The Next Catastrophe. I’ll explore how they might be used to construct safer AI systems, less prone to loss of control or pursuing undesired objectives, as well as misuse. I’ll also argue that making smaller, specialized AI modules might be preferable to making large, general models.

The fundamental idea of Normal Accidents is that while we typically think of accidents and disasters as deviations from a working process or system, the truth is that accidents are normal in any sufficiently long running system. Those who work on system safety must acknowledge that it’s only a matter of when, not if, a system has an accident. They must design the system accordingly.

Complexity

There are a variety of variables we can examine to determine how likely an accident is to occur at any given point in time, and how devastating the outcome of the accident is likely to be.

According to Perrow, one of the major attributes that contributes to accidents in a sociotechnical system is complexity. Complexity is a measure of how interconnected various components of a system are. These can be technical components, biological components, chemical, physical, etc.

You can organize systems by their complexity, by the flexibility of the processes involved in the systems. Complex systems are those defined by highly interconnected components. They typically provide limited, indirect, and ambiguous data about their operation to those who oversee and operate the system. An employee/operator within a complex technical system may be trained to focus on a single component, knowing very little about the rest of the system.

Complex systems are the opposite of linear systems. Linear interactions are easy to follow, see, and predict. In these situations, it’s easy to see which system components are doing what. Cause and effect in the system is straightforward, with alterations to one component causing repeatable, expected changes in downstream components.

Pipelines ...