trentmkelly/LessWrong-43k huggingface dataset:
Can Persuasion Break AI Safety? Exploring the Interplay Between Fine-Tuning, Attacks, and Guardrails
by Devina Jain — 2025-02-04T19:10:13.933Z

This blogpost was created as a part of the AI Safety Fundamentals course by BlueDot Impact. Code available upon request due to sensitive nature of some content.

Disclaimer:
This post represents the findings of independent research carried out solely for academic and educational purposes. The methods, experiments, and conclusions presented here are the result of my personal inquiry and are shared in the interest of advancing understanding of AI safety and fine-tuning dynamics. I do not endorse or encourage the use of these methods for any harmful or malicious activities. Any unauthorized or unethical use of this information is strongly discouraged.

TL;DR: Amid growing concerns about adversarial use of AI, this post investigates whether fine-tuning language models for persuasiveness can weaken their built-in safety features. Using 100 adversarial prompts, I compare standard prompts to those rephrased via supervised fine-tuning (SFT) and reinforcement learning (DPO). Measuring Attack Success Rate (ASR), I find GPT-4 actually becomes more resistant to adversarial prompts under a persuasive style (ASR down from 33% to 21%), whereas GPT-3.5 exhibits a concerning increase in vulnerability (ASR from 2% to 20%). Interestingly, the DPO fine-tuning approach proved more effective at eliciting policy violations than SFT consistently, suggesting that its reward structure can amplify adversarial intent under certain conditions. These architecture-dependent outcomes underscore that there’s no universal answer to whether persuasion undermines safety. Instead, the relationship appears deeply tied to a model’s fundamental design. This result challenged my assumptions about AI safety and calls for more nuanced, architecture-aware approaches to building safer language models.

1. Introduction

In my recent project, I set out to explore a provocative question: Can fine-tuning a language model for persuasiveness overcome its ingrained safety protocols?

As large language models (LLMs) become more powerful, the tension between usefulness and safety intensifies. While many alignment strategies focus on preventing misuse, a crucial yet underexplored angle is the “persuasion effect”: what if tuning a model to be more persuasive inadvertently equips it with the ability to circumvent or undermine safety measures?

In this project, I hypothesized that “persuasion-first” fine-tuning migh...
Journalism student looking for sources
by pinkerton — 2025-02-04T18:58:49.740Z

Hello Lesswrong community,
I am a journalism student doing my capstone documentary on AI alignment. However, this is a topic that I want to make sure is done well. The last thing I would want is to confuse or mislead anyone.
That said, I have a few questions:
Who would be the best people to reach out to for interview? What would be the best foundational topics to help viewers understand the bigger picture? And is there any ethical or technical concerns I should be wary of when communicating this topic to people with little to no understanding of AI?
I appreciate your time!
We’re in Deep Research
by Zvi — 2025-02-04T17:20:06.540Z

The latest addition to OpenAI’s Pro offerings is their version of Deep Research.

Have you longed for 10k word reports on anything your heart desires, 100 times a month, at a level similar to a graduate student intern? We have the product for you.

Table of Contents

The Pitch.

It’s Coming.

Is It Safe?.

How Does Deep Research Work?.

Killer Shopping App.

Rave Reviews.

Research Reports.

Perfecting the Prompt.

Not So Fast!.

What’s Next?.

Paying the Five.

The Lighter Side.

The Pitch

OpenAI: Today we’re launching deep research in ChatGPT, a new agentic capability that conducts multi-step research on the internet for complex tasks. It accomplishes in tens of minutes what would take a human many hours.

Sam Altman: Today we launch Deep Research, our next agent.

This is like a superpower; experts on demand!

It can use the Internet, do complex research and reasoning, and give you a report.

It is quite good and can complete tasks that would take hours or days and cost hundreds of dollars.

People will post many excellent examples, but here is a fun one:

I am in Japan right now and looking for an old NSX. I spent hours searching unsuccessfully for the perfect one. I was about to give up, and Deep Research just… found it.

It is very compute-intensive and slow, but it is the first AI system that can do such a wide variety of complex, valuable tasks.

Going live in our Pro tier now, with 100 queries per month.

Plus, Team and Enterprise tiers will come soon, and then a free tier.

This version will have something like 10 queries per month in our Plus tier and a very small number in our free tier, but we are working on a more efficient version.

(This version is built on o3.)

Give it a try on your hardest work task that can be solved just by using the Internet and see what happens.

Or:

Sam Altman: 50 cents of compute for 500 dollars of value

Sarah (YuanYuanSunSara): Deepseek do it for 5 cents, 500 dollar value.

Perhaps DeepSeek will quickly follow suit, perhaps they will choose not to. The important thing about Sarah’s comment is that there is essentially no difference here.

If the report really is worth $500, then the primary costs are:

Figuring out what you want.

Figuring out the prompt to get it.

Reading the giant report.

NOT the 45 cents you might save!

If the marginal compute cost to me really is 50 cents, then the actu...
The Capitalist Agent
by henophilia — 2025-02-04T15:32:39.694Z
Link: https://blog.hermesloom.org/p/the-capitalist-agent

With the ongoing evolutions in “artificial intelligence”, of course we’re seeing the emergence of agents, i.e. AIs which can do rather complex tasks autonomously.

The first step is automation, but of what?

First comes the stuff where humans currently act like computers anyway: sales, marketing, clerks and everyone else who’s doing repetitive things.

But what’s the main metric which they will aim to maximize? It’s not producing paperclips, it’s earning money. Good ol’ cash on the bank account of the person who’s operating the AI. Preferably just a single person.

I strongly believe that very soon, we will see the emergence of the Capitalist Agent, an agentic AI which can run a business from front to back. Which has access to an email account, does sales reach-outs automatically, develops software based on customer feedback, talks to investors in video calls run by generated faces, does all the bureaucracy etc. But with ‘superhuman’ capabilities: The AI which is “in charge” of the business can just spawn different virtual, fake sales representatives which can speak different languages, to get a workforce which can scale internationally immediately.

Even if the human-run businesses realize that they’re actually just interacting with an AI, they won’t have any incentive to expose it, because it makes both sides good money. After all, running a business is intellectual artisanship, so with artificial “intelligence”, it can easily become industrialized.

The AI’s only success metric is to earn as much money as possible.

Beyond basic, “value-driven” business models, the next area these AI-run businesses will turn to will obviously be investing. Those with the best capabilities to mold their business skills into LLM-based systems will be the ones who will profit the most.

At first, obviously none of this will be open source, because the business leaders won’t want to give away their amazingly engineered LLM-based code.

At some point, large media outlets might “expose” these AI-run businesses. Maybe politicians pretend that they want to regulate them, but that won’t really happen, because these AI-run businesses will give the governments pretty nice tax income, so why hinder them?

Then there will be a leak. A lot of business knowledge which some companies have painstakingly transformed into incredibly convincing AI agents into an interconnected software suite (“The Agentic Busin...
Forecasting AGI: Insights from Prediction Markets and Metaculus
by Alvin Ånestrand — 2025-02-04T13:03:45.927Z
Link: https://forecastingaifutures.substack.com/p/forecasting-agi-insights-from-prediction-markets

I have tried to find all prediction market and Metaculus questions related to AGI timelines. Here I examine how they compare to each other, and what they actually say about when AGI might arrive.

If you know of a market that I have missed, please tell me in the comment section! It would also be helpful if you tell me about what questions you think are relevant but are missing from this analysis. This is a linkpost, and I prefer if you comment in the original post on my new blog, Forecasting AI Futures, but feel free to comment here as well. Subscribe to the blog for updates on my future forecasting posts related to AI safety.

Whenever possible, please check the more recent probability estimates in the embedded sites, instead of looking at my At The Time Of Writing (ATTOW) numbers.

So, what does prediction markets and Metaculus have to say about AGI?

Metaculus has this question for the arrival date of AGI:

The AI system needs to be able to:

Pass a really hard Turing test.
Have general robotic capabilities (being able to assemble a “

circa-2021 Ferrari 312 T4 1:8 scale automobile model

” or equivalent).
Achieve “at least 75% accuracy in every task and 90% mean accuracy across all tasks” on the

MMLU benchmark

, which measures expertise in a wide range of academic subjects.
Achieve at least 90% accuracy with a single attempt for each question on the

APPS benchmark

, which measures coding skills.

Metaculus thinks this will probably occur around the middle of 2030, though with high uncertainty. The interval between the lower and upper quartiles for the individual predictions on this question is (2026-12-28 - 2039-03-27) ATTOW.

GPT-4o achieves an accuracy of 88.7% on MMLU, as seen in the leaderboard

here

. GPT-4 was used to get

22% accuracy on APPS

. Unfortunately, most of the best models have not been tested on either MMLU or APPS.

OpenAI’s o3

has been reported

of achieving 71.7% on

SWE-bench Verified

. We can compare that to GPT-4, which managed to achieve 22.4% on SWE-bench Verified and 22% accuracy on APPS. Based on this, I think o3 would manage to achieve above 50% accuracy on APPS.

The two criteria that AI currently seem furthest from fulfilling are the robotics capabilities and APPS accuracy, though current best performance on the APPS benchmark is uncertain. Coding capabilities are improving very fast, which indicated by the rapid improvements in accuracy i...
Ruling Out Lookup Tables
by Alfred Harwood — 2025-02-04T10:39:34.899Z

This post was written during Alex Altair's agent foundations fellowship funded by the LTFF.

This is not particularly surprising or complex, but I wanted it written up somewhere. When we have been discussing the Agent-like Structure problem, lookup tables often come up as a useful counter-example or intuition pump for how a system could exhibit agent-like behaviour without agent-like structure. It is fairly intuitive that, in the limit of a large number of entries, a lookup table requires a longer program to implement than a program which 'just' computes a function. A lookup table's size can be reduced using efficient encoding but even a lookup table whose entries are encoded using a maximally efficient entropy code still grows in size with the number of entries. This post makes that claim quantitative.

Introduction

Suppose we have a black box which takes in strings and spits out other strings. Based on the inputs and outputs of the black box, it seems like the box is calculating some computable function f. One possibility is that the black box contains a program which calculates the function on the input, and then outputs it. Another possibility is that the black box contains a big table with lots of possible inputs and their corresponding outputs. The program inside the black box then looks through this table until it finds an entry matching its input and prints the corresponding output. This is known as a lookup table.

Suppose we say that the box is 'actually calculating the function', if the box contains finite program which will return f(x) for every possible input x. Can we be sure that the box is actually calculating the function rather than using a lookup table?

Just by looking at a finite number of instances of input-output behaviour, we can't. Input-output behaviour for any function can be mimicked by a lookup table. But we can rule out a lookup table if we know the size of the program being executed inside the black box. This is because a program which computes a function has a fixed length. The program which calculates the function f(x)=x+1 is the same length whether we test its behaviour using just one string of length two or on one hundred strings of length one trillion (though other implementation requirements, such as memory, may be different). A lookup table, must, at minimum, contain entries corresponding to al...
Half-baked idea: a straightforward method for learning environmental goals?
by Q Home — 2025-02-04T06:56:31.813Z

Epistemic status: I want to propose a method of learning environmental goals (a super big, super important subproblem in Alignment). It's informal, so has a lot of gaps. I worry I missed something obvious, rendering my argument completely meaningless. I asked LessWrong feedback team, but they couldn't get someone knowledgeable enough to take a look.

Can you tell me the biggest conceptual problems of my method? Can you tell me if agent foundations researchers are aware of this method or not?

If you're not familiar with the problem, here's the context: Environmental goals; identifying causal goal concepts from sensory data; ontology identification problem; Pointers Problem; Eliciting Latent Knowledge.

Explanation 1

One naive solution

Imagine we have a room full of animals. AI sees the room through a camera. How can AI learn to care about the real animals in the room rather than their images on the camera?

Assumption 1. Let's assume AI models the world as a bunch of objects interacting in space and time. I don't know how critical or problematic this assumption is.

Idea 1. Animals in the video are objects with certain properties (they move continuously, they move with certain relative speeds, they have certain sizes, etc). Let's make the AI search for the best world-model which contains objects with similar properties (P properties).

Problem 1. Ideally, AI will find clouds of atoms which move similarly to the animals on the video. However, AI might just find a world-model (X) which contains the screen of the camera. So it'll end up caring about "movement" of the pixels on the screen. Fail.

Observation 1. Our world contains many objects with P properties which don't show up on the camera. So, X is not the best world-model containing the biggest number of objects with P properties.

Idea 2. Let's make the AI search for the best world-model containing the biggest number of objects with P properties.

Question 1. For "Idea 2" to make practical sense, we need to find a smart way to limit the complexity of the models. Otherwise AI might just make any model contain arbitrary amounts of any objects. Can we find the right complexity restriction?

Question 2. Assume we resolved the previous question positively. What if "Idea 2" still produces an alien ontology humans don't care about? Can it happen?

Question 3. Assume everything works out. How do we know that this is a general meth...
Information Versus Action
by Screwtape — 2025-02-04T05:13:55.192Z

You can get a clearer view of what's going on if you're willing to ignore certain types of information when making decisions. If you heavily use a source of information to make important decisions, that source of information gains new pressure that can make it worse. See Goodhart's Law and Why I Am Not In Charge.

I.

Imagine you are an alien from the planet of obsessives, and you want to know how accurate the criminal justice system is. You're purely in it for the knowledge. You don't care about arresting more criminals, you don't care about the second order effects on society, you just really want to know how accurate this system is. (If it helps, imagine the kind of person who complains in the War Thunder forums about the exact specifications of aircraft, or who uses a magnifying glass to paint the decals on miniature train sets, only their interest is focused on the judiciary.)

You obviously can't use the courts to check if the courts find the correct people innocent and the correct people guilty. You can check if a case ever gets overturned, but it's possible the court was right the first time and wrong the second. You could try and investigate crimes yourself, but then any differences between your verdicts and the court verdicts could just as well be your error as it could be the court's error. This is frustrating.

Finally, you come up with an answer. You go to defendants who have just finished your trial and have the following conversation:

You: Can you please tell me whether you're actually innocent or guilty?
Defendant: What? Obviously I'm innocent. Why would I tell you anything else?
You: Because I can't be used against you. Look, I swore an oath to the court that I'd tell them random nonsense if they asked me. Then I got myself notarized as insane, due to the whole obsessive alien thing. No court would take my testimony.
Defendant: I feel like I shouldn't trust you.
You: Reasonable, but consider, I'm just asking you to whisper it in my ear. I'll strip my shirt off so you can see I'm not wearing a wire. Even if I did try and testify, the court knows I could say whatever, so they can't rely on me.
Defendant: I'm innocent. Even if I wasn't, how would this deal benefit me in any way?
You: It costs you nothing. Tell you what, I'll swear upon my eternal soul to the demon Balthazar that I'll never tell a single person. I'll never write it down, even in a ...
Tear Down the Burren
by jefftk — 2025-02-04T03:40:02.767Z

I love the

Burren.
It hosts something like seven weekly

sessions in a
range of styles and the back room has hosted many great acts including

many

of my

friends. It's a key space in
the Boston folk scene, and it's under threat from developers who

want
to tear it down.

But after thinking it through, and noting that this puts me on the
other side from many of my friends, I think the project should go
ahead. The proposal isn't to tear down paradise to put up a parking
lot, it's to put up 500+ apartments (studio, 1br, 2br, 3br), 100+
of them limited to people below income thresholds ('affordable').
We're in the middle of a housing crisis, where instead of
letting people build up to meet demand we've been competing with
each other to bid up the apartments that do exist. Rising
rent has been really hard on our communities, folk scene included. Of
my Cambridge/Somerville folkie friends from, say, fifteen years ago,
the majority of folks still in the general area have been priced out,
to Vermont, Maine, and Western Mass. While spaces to make music are a
key component of a thriving community, housing is even more critical.

Part of what makes the tradeoff so substantial, with so much potential
new housing, is that the Burren, Dragon Pizza, and other businesses
along this stretch are not making very intensive use of the space:

Much of it is parking lots, and most of the rest is just a single
story. And this is an area just a lot a three minute walk from Davis
Sq Station!

Discussion also tends to assume that a lost place opens an unfillable
void. In part because of this planned development, however, the owners
of the Burren are opening another place a few blocks away: McCarthy's and Toad. Even if this
weren't already in progress, there's clearly a strong demand for this
kind of space, and other places could open to fill it. This kind of
churn is still bad, but it's not forever.

At a more fundamental level, however, having this conversation at the
level of individual proposed projects is the wrong way to do it, and
is a key
contributor to the rising cost of housing and infrastructure.
Part of why housing prices are so high are that we've heavily
restricted what can be built, but another significant component is the
risk that community feedback will stall or cancel a project. In this
case, illustrating both aspects, an earlier pro...
Constitutional Classifiers: Defending against universal jailbreaks (Anthropic Blog)
by Archimedes — 2025-02-04T02:55:44.401Z
Link: https://www.anthropic.com/research/constitutional-classifiers

Excerpt below. Follow the link for the full post.

In our new paper, we describe a system based on Constitutional Classifiers that guards models against jailbreaks. These Constitutional Classifiers are input and output classifiers trained on synthetically generated data that filter the overwhelming majority of jailbreaks with minimal over-refusals and without incurring a large compute overhead.

We are currently hosting a temporary live demo version of a Constitutional Classifiers system, and we encourage readers who have experience jailbreaking AI systems to help “red team” it. Find out more below and at the demo website.

Results from automated evaluations. For all plots, lower is better. (a) The success rate of jailbreaks is far lower in a system protected by Constitutional Classifiers; (b) the refusal rate of the system on production Claude Free and Pro traffic is not statistically significantly higher when using Constitutional Classifiers; and (c) the relative compute cost of a system that uses Constitutional Classifiers is only moderately higher. Error bars represent 95% confidence intervals computed using binomial proportion standard errors under asymptotic normality assumptions.

Training and implementing Constitutional Classifiers. (a) A constitution is produced specifying harmless and harmful categories; (b) the constitution is used as the basis for the production of many synthetic prompts and completions, which are further augmented (with variations on style and language) and turned into a training set; (c) classifiers trained on this training set are used as model safeguards to detect and block harmful content.