trentmkelly/LessWrong-43k huggingface dataset:
AI, Alignment & the Art of Relationship Design
by Priyanka Bharadwaj — 2025-04-19T00:47:02.591Z

We don’t always know what we’re looking for until we stop looking for what we are told to want.

When I worked as a relationship coach, most people came to me with a list. A neat, itemised checklist of traits their future partner must have. Tall. Intelligent. Ambitious. Spiritual. Funny but not flippant. Driven but not workaholic. Family-oriented but not clingy. Always oddly specific. Wildly contradictory.

Most of them came from a place of fear. The fear of choosing wrong. The fear of heartbreak. The fear of regret.

I began to notice a pattern. We don't spend enough time asking ourselves what kind of relationship we want to build. We outsource the work of introspection to conditioning, and compensate for confusion with checklists. Somewhere along the way, we forget that the person is not the relationship. The traits don’t guarantee the experience.

So I asked my clients to flip the script. Instead of describing a person, describe the relationship. What does it feel like to come home to each other? What are conversations like during disagreements? How do we repair? What values do we build around?

Slowly, something shifted. When we design the relationship first, we begin to recognise the kind of person who can build it with us. Our filters get sharper. Our search gets softer. We stop hunting for trophies and start looking for partners.

I didn’t know it then, but that framework has stayed with me. It still lives in my questions. Only now, the relationship I’m thinking about isn’t romantic. It’s technological.

Whether we realise it or not, we are not just building artificial intelligence, we are curating a relationship with it. Every time we prompt, correct, collaborate, learn, or lean on it, we’re shaping not just what it does, but who we become alongside it.

Just like we do with partners, we’re obsessing over its traits. Smarter. Faster. More efficient. More capable. The next version. The next benchmark. The perfect model.

But what about the relationship?

What kind of relationship are we designing with AI? Through it? Around it?

We call it “alignment”, but much of it still smells like control. We want AI to obey. To behave. To predictably respond. We say “safety”, but often we mean submission. We want performance, but not presence. Help, but not opinion. Speed, but not surprise.

It reminds me of the well-meaning aunties in the marriage market. Impressed by degrees, salaries...
Measuring Beliefs of Language Models During Chain-of-Thought Reasoning
by Baram Sosis — 2025-04-18T22:56:28.727Z

Based on research performed as a PIBBSS Fellow with Tomáš Gavenčiak as well as work supported by EA Funds and Open Philanthropy.

tl;dr: I'm investigating whether LLMs track and update beliefs during chain-of-thought reasoning. Preliminary experiments with older models (without reasoning training) have not been able to measure this; I plan to develop these experiments further and try them with reasoning models like o1/r1.

Introduction

Chain-of-thought (CoT) reasoning has long been recognized as an important component of language model capabilities. This is especially the case for new "reasoning" models like OpenAI's o1 or DeepSeek's r1 that are trained using reinforcement learning to write long CoTs before responding, but even without such training, prompting an LLM to verbally work through a question before responding often boosts performance. This makes gaining a better understanding of how LLMs perform CoT reasoning and the extent to which CoTs enhance overall LLM capabilities an important research priority.

The prevalence of CoT reasoning also opens up new opportunities for safety efforts. If LLMs externalize much of their reasoning in legible text, monitoring that reasoning becomes much easier. It's therefore very important to characterize how faithful CoTs are to the LLM's underlying reasoning process. Unfortunately, there is extensive evidence that CoTs are often unfaithful: in many cases LLMs will make choices for reasons other than those stated in their CoTs, and they can learn to hide misaligned behavior from CoT monitors. Furthermore, future models may be able to encode information steganographically in their CoTs or reason in an illegible latent space rather than text. These problems make using CoT monitors to ensure the safety of LLMs very challenging.

While there has been significant research on the faithfulness of CoT reasoning in various circumstances, I think there has not yet been enough work on more fundamental questions regarding how CoT reasoning works in LLMs. For instance: humans, when working through a problem, will often start out with some pre-existing beliefs and update them in one direction or another based on the new arguments and reasoning that they generate. Do LLMs similarly track and update beliefs during CoT reasoning? Relatedly, humans will typically have some idea of what they plan to say next while speaking; can LLMs similarly ...
LLM-based Fact Checking for Popular Posts?
by azergante — 2025-04-18T21:26:25.230Z

Hello,

I think having some LLM-based fact checking on LessWrong posts would be a valuable addition.

To some extent comments already serve this purpose, but LLMs can do this in a more automated and systematic way, side-stepping human blind spots and tendency to conserve energy.

1. Some pros and cons

Cons:

time cost to implement & maintain the feature

money cost to power the LLMs that do the fact-checking

potential errors in the fact-checking

Pros:

higher confidence that what is in the post is true

incentive for the author of the post to put more effort in researching the material

the post is read in a broader context, with opposite views (if any) readily available

spark some interesting discussions in the comments based on the LLM's feedback

save reader time as they have less fact checking to do

2. Some thoughts on implementation details

Should fact checking be done on all posts or only some of them?

I would like fact checking to be enabled on the most read posts. This includes: The Sequences, The Codex, and the Best of Less Wrong for each year. Enabling them for other popular posts also makes sense.

I am not sure about less popular posts as it might cost a lot of money. Another concern is that it could intimidate new authors, though this could be mitigated by allowing the author to run the fact-checking before publishing.

Another consideration is whether the fact-checking should happen automatically when the post reaches some degree of popularity, or should be performed on demand by (some) users.

Where should the LLM's report go?

I imagine it going in a comment, perhaps pinned or otherwise highlighted, but subject to user feedback.

What do you think are the pros & cons of LLM-based fact checking for LW posts?

If this is to be implemented, how should it be implemented?

Edit 1: 3 Case Studies

At the suggestion of Dave Orr here is a fact checking experiment for a sample of 3 posts known to contain factual mistakes:

Case 1: Conservation of Expected Evidence

As R:A-Z Errata points out

In the Kindle version, "P(H) = P(H|E) × P(E) + P(H,¬E) × P(¬E)" should instead be "P(H) = P(H|E) × P(E) + P(H|¬E) × P(¬E)". This is correct in the PDF version.

I copied the original post at Conservation of Expected Evidence and reintroduced the mistake in the input I fed to Gemini 2.5:

Gemini 2.5 Chat - My Prompt

Perform a thorough fact checking of this post and report fact

...
o3 Will Use Its Tools For You
by Zvi — 2025-04-18T21:20:02.566Z

OpenAI has finally introduced us to the full o3 along with o4-mini.

Greg Brockman (OpenAI): Just released o3 and o4-mini! These models feel incredibly smart.

We’ve heard from top scientists that they produce useful novel ideas.

Excited to see their positive impact on people’s daily lives and humanity’s hardest problems!

Sam Altman: we expect to release o3-pro to the pro tier in a few weeks

By all accounts, this upgrade is a big deal. They are giving us a modestly more intelligent model, but more importantly giving it better access to tools and ability to discern when to use them, to help get more practical value out of it. The tool use, and the ability to string it together and persist, is where o3 shines.

The highest praise I can give o3 is that this was by far the most a model has been used as part of writing its own review post, especially helping answer questions about its model card.

Usage limits are claimed to be 50/week for o3, 150/day for o4-mini and 50/day for o4-mini-high for Plus users along with 10 Deep Research queries per month.

This post covers all things o3: Its capabilities and tool use, where to use it and where not to use it, and also the model card and o3’s alignment concerns. o3 is on the cusp of having dangerous capabilities, and while o3 is highly useful it hallucinates remarkably often by today’s standards and engages in an alarmingly high amount of deceptive and hostile behaviors.

As usual, I am incorporating a wide variety of representative opinion and feedback, although in this case there was too much to include everything.

What’s In a Name

OpenAI’s naming continues to be a train wreck that doesn’t get better, first we had the 4.1 release and now this issue:

Manifold: “wtf I thought 4o-mini was supposed to be super smart, but it didn’t get my question at all?”

“no no dude that’s their least capable model. o4-mini is their most capable coding model”

o4-mini is the lightweight version of this, and presumably o4-mini-high is the medium weight version? The naming conventions continue to be confusing, especially releasing o3 and o4-mini at the same time. Should we consider these same-generation or different-generation models?

I’d love if we could equalize these numbers, even though o4-mini was (I presume) trained more recently, and also not have both 4o and o4 around at the same time.

My Current Model Use Heuristics

After...
AI Control Methods Literature Review
by Ram Potham — 2025-04-18T21:15:34.682Z

AI Control is a subfield of AI Safety research focused on ensuring the safety of powerful AI systems, particularly under the assumption that they might be misaligned or could become intentionally subversive (A Brief Explanation of AI Control).

Unlike AI Alignment, which seeks to build inherently aligned systems, AI Control develops mechanisms to monitor, evaluate, constrain, verify, and manage the behavior of potentially untrustworthy AI to prevent catastrophic outcomes (The case for ensuring that powerful AIs are controlled).

This review outlines key AI control methods discussed in recent literature. For a broader exploration of related topics, refer to An overview of areas of control work.

1. Monitoring

Description: Monitoring encompasses techniques for observing AI system behavior, outputs, and sometimes internal states. The primary goal is to detect potentially harmful actions or intentions, particularly those the AI might try to conceal, enabling timely intervention or review.

Trusted Monitoring (and editing)

Description: This control method employs a less powerful but trusted entity, such as a smaller AI model or human oversight, to review and potentially edit or block the outputs or actions of a more powerful, untrusted AI. The goal is to catch and mitigate harmful behavior before it occurs, often flagging suspicious items for further review.
Related Papers:

AI Control: Improving Safety Despite Intentional Subversion
Adaptive Deployment of Untrusted LLMs Reduces Distributed Threats

Strengths: Can provide a baseline level of safety by having a supposedly aligned entity check actions. Trusted entities are assumed not to be intentionally subversive.
Weaknesses: The effectiveness is limited by the capabilities of the trusted monitor; a sufficiently advanced untrusted model might be able to deceive it. May lead to a drop in usefulness if the trusted model is too restrictive. Requires identifying and maintaining trust in a model.

Untrusted Monitoring

Description: Using other instances of the powerful, untrusted model itself (or an ensemble) for monitoring. This requires techniques to prevent these monitors from colluding with the agent they are supposed to oversee.
Related Papers:

AI Control: Improving Safety Despite Intentional Subversion
Ctrl-Z: Controlling AI Agents via Resampling

Strengths: Can potentially leverage the capabilities of advan

...
Consequentialists should have a comprehensive set of deontological beliefs they adhere to
by Jay95 — 2025-04-18T20:50:27.064Z

One thing I believe happens a lot to effective altruists, who are generally (all?) consequentialists of some sort, is that their rational calculus leads them to efface themselves or lead a life much more miserable than they would've if they had spent less time thinking about their altruism. I think EA men and women are at the final frontier of consideration of the link between rational self-interest and true altruism, where their desire to do good from the point of view of the universe is leading them into great punishment and one would have difficulty convincing them the aforementioned link is strong. Nonetheless, I believe it is strong, and would like to make the case that a comprehensive set of deontological beliefs, akin to a religious morality, would objectively serve consequentialist ends better than the reliance on explicit reasoning about outcomes that is central to EA.

Here's my argument:

Say you're a utilitarian who wants to maximize the happiness of all beings throughout all of time and minimize their suffering. It's tempting to think of practical things you might do in this direction, but I think first you should clarify the scope of your intention. You are a finite being within a reality that far surpasses your knowledge, and may do so endlessly, and you would like to live your life towards an as-good-as infinite end. Whatever theory of decision making you subscribe to, and whatever metaphysics you have, the only way to safely act in a situation like that is predictably. The amount of information you would need to have to have a high probability of following you morality exactly is too great to make all but the most minor and inconsequential decisions. If you care about your morality you should be acting as conservatively as possible.

You want to tilt the balance in favor of your utilitarian morality, so the question becomes "what is the least consequential thing I can do that is likeliest to promote happiness and minimize suffering?"

In other words you want to perform the moral good that is nearest at hand.

This means you start with your own mind, and moving as gently as possible befriend it and create within it genuine contentment. All of your rational behavior after that is captured within the concept of emotional regulation. If you develop a set of internally consistent rules to follow for the various kinds of events you can guess you may face in...
What Makes an AI Startup "Net Positive" for Safety?
by jacquesthibs — 2025-04-18T20:33:22.682Z

In light of the recent news from Mechanize/Epoch and the community discussion it sparked, I'd like to open a conversation about a question some of us grapple with: What constitutes a net-positive AI startup from an AI safety perspective, and what steps can founders take to demonstrate goodwill and navigate the inherent pressures?

This feels like an important conversation to have because AI safety seems to have been increasingly encouraging people in the community to build startups (due to a lack of funding, potentially higher ceiling for society-wide impact, etc.). I've thought a lot about this, and have been going back and forth on this for the past three years. You get constant whiplash.

"You're building an evals org? Isn't that just doing the homework for the labs? And besides, none of it will scale to ASI. You'll just safety-wash labs along the way."
"You're doing jailbreaks? This will just get solved at some point, and, honestly, it doesn't tackle the core of the alignment problem."
"Automated research? Isn't that just capabilities? You claim that you'll focus on non-frontier capability things, but incentives are a powerful thing."

As someone in the AI safety space who is certainly concerned about the risks (takeover and human disempowerment) while also feeling like I'd have a good personal fit in building a for-profit, deciding on what to do has not been easy. You get completely different responses from different people in the community as to whether Startup X is net positive (and sometimes the same person will give you the opposite opinion a month later). At some point, you either:

Take the safe option by not building a startup (so continue to do independent research, work at a research org, or create a non-profit).
Build a startup that doesn't tackle important parts of AI safety (because it's not bad per se, and much easier to make money).

Outside of cybersecurity and evals, it's not particularly easy to find domains with a big enough market size while tackling important AI safety work.

Build the controversial startup after going with one's gut (feeling that this is overall a net positive bet).

In my case, I started going in the startup direction because I wanted to force myself to think creatively about what a net positive startup for AI safety would look like and hopefully bring more money into the space (to hire AI safety researchers). I thought, "AI safety...
Alignment Does Not Need to Be Opaque! An Introduction to Feature Steering with Reinforcement Learning
by Jeremias Ferrao — 2025-04-18T19:34:49.357Z

Reinforcement Learning (RL) has become the driving force behind modern artificial intelligence, as evidenced by

rewarding models from human feedback

or powerful new systems like

OpenAI's o3

and

DeepSeek R1

, which are at the top of

several capabilities benchmarks

[1]. While these advances are exciting, they've exposed critical vulnerabilities in our safety measures.

Anthropic

recently discovered language models that selectively comply with safety constraints only during evaluation, while

Palisade

found that o1 would rather hack its testing environment than play chess legitimately against a competitor.

At first glance, these developments might cast RL as the antagonist in AI safety research – a force that undermines our attempts to instil beneficial behaviours. However, I propose a different perspective: RL can serve as a powerful diagnostic tool, helping us probe and understand the limitations of our current alignment approaches.

To demonstrate this potential, I recently developed the Feature Steering with Reinforcement Learning framework at the

Apart x Goodfire Reprogramming AI Models Hackathon

that leverages

Sparse Autoencoders (SAEs)

– powerful tools that offer unprecedented visibility into neural networks' internal representations by discovering human interpretable concepts. While SAEs excel at revealing how models process and represent information, most machine-learning tasks require adapting a general model to a specific domain. However, SAEs assume that the underlying model does not change. In a traditional setup, we

might need to train a separate SAE from scratch

each time we adapt the model, which can quickly become computationally expensive.

The approach combines RL with SAEs in a novel way that circumvents these limitations. By using a separate RL-trained model to manipulate the concepts discovered by SAEs, we can modify model behaviour while maintaining transparency. This method not only improved Llama 3 8B's performance at Tic-Tac-Toe by 50% but also maintained direct visibility into which concepts were being manipulated to achieve this improvement. The rest of this blog post goes into detail about the specifics of the framework and my vision for how we can extend the approach to real-world use cases to better understand the alignment problem.

Methods

This framework relies on three techniques:

Sparse Aut

...
Evaluating Collaborative AI Performance Subject to Sabotage

by Matthew Khoriaty — 2025-04-18T19:33:41.547Z

Gulsimo Osimi, Matthew Khoriaty

Executive Summary

This research investigates the potential of collaborative AI systems to solve complex tasks through the orchestration of multiple specialized models. By leveraging Microsoft's Autogen framework, we coordinate DeepSeek and GPT-4o Mini to tackle complex challenges. Beyond performance enhancement, our work makes a novel contribution by examining the vulnerability of collaborative systems to internal sabotage and evaluating its impact on performance. We show that basic saboteurs have minimal impact on well-designed teams and that team structure can induce productive behaviors in malicious AI. Additionally we analyze and classify the cases in which the saboteur succeeds to indicate directions for future defensive improvements.

This research provides actionable guidelines for deploying multi-agent AI systems in real-world software engineering contexts where performance is very important. The findings contribute to the emerging field of collaborative AI, offering both theoretical insights and practical implementation strategies for building more capable and resilient AI assistants for software development.

Code Availability: The code used in this study is publicly available on GitHub at AI-Teamwork: Evaluating Collaborative AI Performance under MIT license.

Abstract

This paper explores the effectiveness of collaborative AI systems for reasoning and software engineering tasks, specifically examining how a coordinated ensemble of language models can outperform single-agent approaches. We orchestrate DeepSeek and GPT-4o Mini using Microsoft's Autogen framework to address complex tasks including code synthesis and bug detection. Beyond performance enhancement, we investigate system vulnerability to sabotage by introducing adversarial agents. Our experiments on the AI2 ARC dataset demonstrate that collaborative teams of GPT-4o achieve 92% accuracy, compared with 90% accuracy for individual models. Moreover, we show that properly configured teams exhibit inherent resistance to basic sabotage attempts, maintaining over 86% accuracy even with many adversarial agents present. We further demonstrate that when triple saboteurs are introduced without collaboration mechanisms between them, the system maintains 20.27% accuracy, significantly more than the less than 8% score of a smoothly functioning saboteur team, showin...