trentmkelly/LessWrong-43k huggingface dataset:
Why you maybe should lift weights, and How to.
by samusasuke — 2025-02-12T05:15:32.011Z

Who this post is for? Someone who either:

Wonders if they should start lifting weights, and could be convinced of doing so.
Wants to lift weights, and doesn't know where to begin. If this is you, you can skip this first section, though I'm guessing you don't know all the benefits yet.

The WHY

Benefits of ANY EXERCISE:

Great mental benefits. I personally have ADHD and lifting gives me an attention boost similar to my Adderall. I'm not talking about the long term abstract of being happy that you are healthy. Post exercise your mood and cognitive hability will probably me accutely better.
Improved sleep quality, which will then improve about a million things
Strongly improved health markers, both blood levels and your body composition. Most people probably would be healthier with more muscle and less fat.
It fits the hobby-shaped whole your life might have. Something you do divorced from work, with progression, community, and so forth

Benefits of LIFTING compared to other exercise:

In spite of most of the lies you will hear, most exercise won't build muscle. If you are a fat guy and start jogging or playing soccer, you'll probably actually lose muscle mass along with your fat loss. Athletes that play sports professionally are both genetically gifted, and probably lift weights to support their performance. But why do you want more muscle mass? Benefits in order

You'll be more attractive to others and to yourself. You'll probably largely hedonically adapt to seeing yourself look better, as will a long term partner. But you don't hedonically adapt to being treated better by strangers and to having access to more options in dating, if you are interested in dating.
Images of outlier physiques you can't probably even reach aside, more muscle is as close to universally attractive as anything you can get.
The number of times someone who didn't take performance enhancing drugs was described by anyone as "too muscular" in the entire history of humanity can be probably counted in one hand.
More muscle and less fat again, benefit your healthy greatly. The latter is easy, the former not so much. If you're interested in quantified self, I'd suggest getting your blood markers before your journey to see what lifting does, including your A1C.

You might really enjoy it. I personally do. Lifting is a really boring sport to watch, which I believe leads to it being tried by less people comp

...
how do the CEOs respond to our concerns?
by KvmanThinking — 2025-02-11T23:39:27.654Z

When a decent rationalist walks up to Sam Altman, for example, and presents our arguments for AI doom, how does he respond? What stops us from simply walking up to the people in charge of these training runs, explaining to them the concept of AI doom very slowly and carefully while rebutting all their counterarguments, and helping them to all coordinate to stop all their AI capabilities development at the same time, giving our poor AI safety freelancers enough time to stumble and philosophize their way to a solution? What is their rejection of our logic?

Or is it simply that they have a standard rejection for everything?
Where Would Good Forecasts Most Help AI Governance Efforts?
by Violet Hour — 2025-02-11T18:15:33.082Z

Thanks to Josh Rosenberg for comments and discussion.

Introduction

One of LessWrong’s historical troves is its pre-ChatGPT AGI forecasts. Not just for the specific predictions people offered, but for observing which sorts of generative processes produced which kinds of forecasts. For instance:

[Nuno (Median AGI Timeline) = 2072]: “I take as a starting point datscilly's own prediction, i.e., the result of applying Laplace's rule from the Dartmouth conference. This seems like the most straightforward historical base rate / model to use … I then apply some smoothing.”

[Kokotajlo (Median AGI Timeline) = 2034]: “I think that if transformative AI is achievable in the next five orders of magnitude of compute improvement (e.g. prosaic AGI?), it will likely be achieved in the next five years or so. I also am slightly more confident [than Ethan Perez] that it is, and slightly less confident that TAI will ever be achieved.”

I think these sources are valuable. To the extent that forecasting is a skill at all, there must be certain kinds of cognitive processes which, on average, produce better forecasts than others. So, the historical evidence at least gives us some (though perhaps slender) insight into which sorts of processes might deliver more reasonable forecasts in the present day.

And I, personally, think this is kind of information is especially fruitful. For reasons both conceptual and empirical, I think we should be skeptical about the degree to which “whatever ability superforecasters have” generalizes to the most action-relevant AI questions.

For this reason, I’ve suggested four potential questions related to the future of AI that seem worth forecasting – primarily focused on issues of AI governance/regulation, as there are already some good forecast questions focused more narrowly on capabilities. The initial thought was to get feedback on questions here, and then post them to forecasting platforms — ultimately, though, I’m just interested in whatever methods we have for eliciting high-quality forecasts.

It would be nice for me if people provide feedback/forecasts on these questions. But there are a couple of other reasons why I think doing so could be valuable.

A recent article on “The Failed Strategy of AI Doomers” attracted a large number of (in many cases critical) commenters. To the extent t

...
AI Safety at the Frontier: Paper Highlights, January '25
by gasteigerjo — 2025-02-11T16:14:16.972Z
Link: https://aisafetyfrontier.substack.com/p/paper-highlights-january-25

This is the selection of AI safety papers from my blog "AI Safety at the Frontier". The selection primarily covers ML-oriented research and frontier models. It's primarily concerned with papers (arXiv, conferences etc.).

tl;dr

Paper of the month:

Constitutional Classifiers demonstrate a promising defense against universal jailbreaks by using synthetic data and natural language rules.

Research highlights:

Human-AI teams face challenges in achieving true complementarity.
Myopic optimization with non-myopic approval shows promise for preventing reward hacking, but faces practical limitations.
Models can learn reward hacking behaviors just from reading descriptions, and can accurately report their own learned behaviors - raising concerns about unintended learning and deceptive alignment.
As models become more capable, their mistakes become increasingly correlated, potentially undermining AI oversight approaches.
Interpretability work on parameter decomposition, output-based descriptions, SAE dependence on random seeds, and SAEs on random transformers.
Open problems in AI safety, interpretability, and machine unlearning.

⭐Paper of the month⭐

Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming

Read the paper

[Anthropic, Haize, independent]

Constitutional Classifiers separately classify model inputs and outputs. The classifiers are trained on synthetic data samples generated via a constitution.

Large language models (LLMs) are vulnerable to "jailbreaks" - prompting strategies that can systematically bypass model safeguards and enable users to carry out harmful processes. Such jailbreaks become increasingly concerning as models' chemical, biological, radiological, or nuclear (CBRN) capabilities increase.

A key concern is that as models become more capable, these jailbreaks could allow malicious actors to extract detailed harmful information reliably across many queries. To mitigate these risks, researchers focus on defending against "universal jailbreaks" - attacks that can reliably extract detailed harmful information across the majority of queries in a domain.

Our paper of the month introduces Constitutional Classifiers - safeguards trained on synthetic data generated from prompting LLMs with natural language rules (a "constitution") specifying permitted and restricted content. These classifiers separately monitor model input...
If Neuroscientists Succeed
by Mordechai Rorvig — 2025-02-11T15:33:09.098Z

Introduction

In the Spring of 2022, Stuart Russell wrote an essay entitled If We Succeed, in which he questioned whether and how the field of AI might need to pivot from its historical goal of creating general-purpose intelligence to a new goal, of creating intelligence that would be provably beneficial for humans. He noted that although the former goal had driven a great deal of progress, it was rapidly on its way to becoming myopic, counterproductive, and even dangerous.

It is not rational for humans to deploy machines that pursue fixed objectives when there is a significant possibility that those objectives diverge from our own.

-Stuart Russell

I believe that a similar situation may now be arising in neuroscience. It is a case where neuroscientists may need to pivot from focusing on the historical goals of their field to thinking about what happens as the world actually gets closer to achieving them.

But why should anyone on Less Wrong be concerned about what is happening in neuroscience? Readers of this forum tend to be most concerned about progress in AI, and the need for AI researchers to pivot, which Russell talked about. And, that is for understandable reasons. The rapid ascent of AI models up the leaderboards, which can feel so concerning, is being driven by AI researchers and AI companies, not neuroscientists. What is happening in neuroscience might seem quaint or irrelevant by comparison.

However, there have turned out to be remarkable parallels between what is happening in neuroscience and AI, or at least, the subfield of neuroscience now often known as neuroAI. Both neuroAI scientists and AI scientists have gotten to the present moment by pursuing very similar methods. These are the methods of deep learning and deep neural networks. As such, neuroscientists and AI scientists are facing similar problems, and the perspectives they have developed are complimentary and synergistic.

For example, one problem that concerns AI researchers currently, which is forcing them to reconsider their priorities, is the question of how AI programs should be interpreted. If we can't even interpret these programs, as we so rapidly improve them, then that increases the risks that they might pose significant risks and dangers.

But neuroscientists have pursued a very similar interpretability question, themselves, in their own way. That's because they ha...
The News is Never Neglected
by lsusr — 2025-02-11T14:59:48.323Z

Dear Lsusr,

I am inspired by your stories about Effective Evil. My teachers at school tell me it is my civic responsibility to watch the news. Should I reverse this advice? Or should I watch the news like everyone else, except use what I learn for evil?

Sincerely,

[redacted]

Dear [redacted],

If you want to make an impact on the world, then you should put your effort into solving problems that are important, neglected and tractable.

Mainstream news often reports on events that are important.
Mainstream news sometimes reports on events that are tractable.
But mainstream news approximately never reports on events that are neglected. Why? Just think about it for ten seconds. It's practically a tautology; events in the mainstream news are the events getting massive attention.

Whenever a major news outlet claims "nobody is talking about <whatever>", the mainstream news is lying. Here is a snapshot of The New York Times website right now, as I write these words. It's mostly news about the current war in the Middle East. The current war in the Middle East is many things, but it is not neglected.

News hurts your agency because it sucks your attention. Every minute you're thinking about the current war in the Middle East is a minute you're not thinking about a problem that is getting insufficient attention.

How can you find information about things that aren't getting massive attention? Focus on expert specialists writing for niche audiences. What's that weird thing you're into which nobody else cares about? It might be more important than normies currently appreciate.

Search your feelings; you know them to be quirky.

Sincerely,

Lsusr

PS: Why do your teachers, parents and other adult authorities tell you to listen to a propaganda machine? Because the propaganda machine is working.
Rethinking AI Safety Approach in the Era of Open-Source AI
by Weibing Wang — 2025-02-11T14:01:39.167Z

Open-Source AI Undermines Traditional AI Safety Approach

In the past years, the mainstream approach to AI safety has been "AI alignment + access control." In simple terms, this means allowing a small number of regulated organizations to develop the most advanced AI systems, ensuring that these AIs' goals are aligned with human values, and then strictly controlling access to these systems to prevent malicious actors from modifying, stealing, or abusing them. Companies like OpenAI and Anthropic are prime examples of applying this approach.

However, this closed-source safety approach is now facing significant challenges. The open-sourcing of models like DeepSeek-R1 has disrupted the monopoly of closed-source advanced AI systems. Open-source models can be freely downloaded, used, and modified by anyone, including malicious actors. Even if an open-source model is initially aligned, it can easily be transformed into a harmful model. Research has shown that with just a few dozen fine-tuning samples, the safety constraints of a model can be removed, enabling it to execute arbitrary instructions. This means that in the future, we are likely to face increased risks of AI misuse, such as using open-source models for hacking, online fraud, or even the creation of biological weapons.

DeepSeek-R1 has a 100% jailbreak success rate on the HarmBench dataset

Open-source models also pose a greater risk of losing control. Developers with weak safety awareness may unintentionally introduce significant vulnerabilities when modifying open-source models. For example, they might inadvertently enable the model to develop self-awareness, autonomous goals, or allow the model to self-iterate without human supervision. Such behaviors could make AI systems even more uncontrollable.

The Unstoppable Momentum of AI Open-Source

While the open-sourcing of models introduces new safety risks, the trend toward open source is unstoppable. In the past years, the progress of open-source models is faster than that of closed-source models (as shown in the chart below). The performance of DeepSeek-R1 is already approaching that of the best publicly available closed-source models. It is foreseeable that in the future, open-source models will not only stand comparable with closed-source models but may even surpass them.

Open-Source Models Are Advancing Faster Than Closed-Source Models

From the public's perspecti...
What About The Horses?
by Maxwell Tabarrok — 2025-02-11T13:59:36.913Z
Link: https://www.maximum-progress.com/p/what-about-the-horses

In

a previous post

, I argued that AGI would not make human labor worthless.

One of the most common responses was to ask about the horses. Technology resulted in mass unemployment and population collapse for horses even though they must have had some comparative advantage with more advanced engines. Why couldn’t the same happen to humans? For example, here’s

Grant Slatton

on X or

Gwern

in the comments.

There are also responses from

Zvi Mowshowitz

and a differing perspective from

Matthew Barnett

that basically agree with the literal claim of my post (AGI will not make human labor worthless) but contend that AGI may well make human labor worth less than the cost of our subsistence.

My two-week break from Substack posts was mainly taken up by thinking about these responses. The following framework explains why horses suffered complete replacement by more advanced technology and why humans are unlikely to face the same fate due to artificial intelligence.

Humans and AIs Aren't Perfect Substitutes but Horses and Engines Were
Technological Growth and Capital Accumulation Will Raise Human Labor Productivity; Horses Can't Use Technology or Capital
Humans Own AIs and Will Spend the Productivity Gains on Goods and Services that Humans Can Produce

Humans and AIs Aren’t Perfect Substitutes But Horses and Engines Were

Matthew Barnett

builds a basic Cobb-Douglas production function model where advanced AI labor is a perfect substitute for human labor. That way, billions of additional AI agents can be modeled as a simple increase in the labor supply.

This is bad news for human wages. If you increase labor supply without increasing capital stocks or improving technology, wages fall because each extra unit of labor becomes less valuable (e.g “too many cooks in the kitchen”).

A massive expansion in labor supply would increase the return to capital, so the capital stock would grow, eventually bringing wages back to their previous levels, but growth in capital may be slow compared to AI labor growth, thus still leaving wages depressed for a long time.

Additionally, it may be that there are decreasing returns to scale on labor and capital combined, perhaps because e.g all the goods spots for factories are taken up leaving only less productive ones, so that even when the capital stock does expand, wages are left at lower levels.

Matthew’s model assumes AI labor will be a perfect substitute fo...
On Deliberative Alignment
by Zvi — 2025-02-11T13:00:07.683Z

Not too long ago, OpenAI presented a paper on their new strategy of Deliberative Alignment.

The way this works is that they tell the model what its policies are and then have the model think about whether it should comply with a request.

This is an important transition, so this post will go over my perspective on the new strategy.

Note the similarities, and also differences, with Anthropic’s Constitutional AI.

How Deliberative Alignment Works

We introduce deliberative alignment, a training paradigm that directly teaches reasoning LLMs the text of human-written and interpretable safety specifications, and trains them to reason explicitly about these specifications before answering.

We used deliberative alignment to align OpenAI’s o-series models, enabling them to use chain-of-thought (CoT) reasoning to reflect on user prompts, identify relevant text from OpenAI’s internal policies, and draft safer responses.

Our approach achieves highly precise adherence to OpenAI’s safety policies, and without requiring human-labeled CoTs or answers. We find that o1 dramatically outperforms GPT-4o and other state-of-the art LLMs across a range of internal and external safety benchmarks, and saturates performance on many challenging datasets.

We believe this presents an exciting new path to improve safety, and we find this to be an encouraging example of how improvements in capabilities can be leveraged to improve safety as well.

How did they do it? They teach the model the exact policies themselves, and then the model uses examples to teach itself to think about the OpenAI safety policies and whether to comply with a given request.

Deliberate alignment training uses a combination of process- and outcome-based supervision:

We first train an o-style model for helpfulness, without any safety-relevant data.

We then build a dataset of (prompt, completion) pairs where the CoTs in the completions reference the specifications. We do this by inserting the relevant safety specification text for each conversation in the system prompt, generating model completions, and then removing the system prompts from the data.

We perform incremental supervised fine-tuning (SFT) on this dataset, providing the model with a strong prior for safe reasoning. Through SFT, the model learns both the content of our safety specifications and how to reason over them to generate aligned responses.

We t

...
Detecting AI Agent Failure Modes in Simulations
by Michael Soareverix — 2025-02-11T11:10:26.030Z

AI agents have become significantly more common in the last few months. They’re used for web scraping,[1][2] robotics and automation,[3] and are even being deployed for military use.[4] As we integrate these agents into critical processes, it is important to simulate their behavior in low-risk environments.

In this post, I’ll break down how I used Minecraft to discover and then resolve a failure in an AI agent system.

By Michael Andrzejewski

Dangers of AI Agents

Let’s briefly discuss the concept of an AI agent, and how they differ from AI tools.

An AI tool is a

static

interface like a chatbot or an image generator. A user enters some information and then receives a static response. For the tool to continue functioning, the user needs to enter more information.

The tool can produce harmful outputs, such as misleading information for a chatbot or graphic images from an image generator. However, the danger here is pretty small and almost entirely dependent on the user.

An AI agent is a

recursive

interface. A user gives it a task and executes an arbitrary sequence of actions to accomplish that task. The keyword here being ‘arbitrary’.

Notice that AI agents are fundamentally a loop. Action Planning → Action Execution → Action Evaluation. This means that agents can accomplish many subtasks before stopping.

By design, AI agents are more powerful and difficult to control than AI tools. A tool can analyze a specific X-ray for tumors; an agent can monitor a patient's condition, decide when new tests are needed, and adjust treatment plans. AI agents don’t require a human in the loop. Because of this additional freedom, agents can solve problems with more unique steps and accomplish more useful work.

However, this also increases their potential for harm. Agent behavior can change as its token context increases,[5] causing it to become more biased over time. An agent tasked with managing patient care might begin acting as a specialist and subtly become biased towards certain patients and treatment plans, leading to healthcare disparities for other patients. Additionally, the agent might leak confidential patient information or make increasingly poor treatment decisions as its token context window increases.

Because of the complexity of a recursive system, predicting these failures in advance is difficult. This means that most AI agent problems are...