trentmkelly/LessWrong-43k huggingface dataset:
unRLHF - Efficiently undoing LLM safeguards
by Pranav Gade — 2023-10-12T19:58:08.811Z

Produced as part of the SERI ML Alignment Theory Scholars Program - Summer 2023 Cohort, under the mentorship of Jeffrey Ladish. I'm grateful to Palisade Research for their support throughout this project.

tl;dr: demonstrating that we can cheaply undo safety finetuning from open-source models to remove refusals - thus making a case against open-sourcing model weights; and propose a method for stronger evaluations that are resistant to jailbreak overhangs.

Why?

As developers train and release increasingly capable LLMs, it seems important to consider how to best mitigate risks that these models cause significant harm in the world - for example, by accelerating the production of misinformation, cheaply scaling phishing attacks, and providing high-quality instructions to help people commit violent crimes.

Reinforcement learning from human feedback (RLHF) is one of the most popular methods of training conversational agents to be helpful and align their answers to human values and preferences. Alongside training language models to be helpful, RLHF is also used to encourage models to be harmless and refuse answering questions or help with tasks that are either inappropriate or outside the knowledge space of the model.

However, these models have had a long history of being jailbroken—from the earliest DAN jailbreaks, to recent Universal and Transferable Adversarial Attacks—and the public release of model weights only seems to exacerbate the problem. This suggests that RLHF does not remove capabilities from models, but rather hides them away behind a fence—leading to jailbreak overhangs.

Jailbreak overhangs are instances where the model has latent capabilities that can be elicited with proper conditioning. Examples include asking the model to adopt a persona like ‘Do Anything Now’ (DAN), prompting in different languages, pretending to be in hypothetical scenarios, encoding your prompts, etc. In other words, these are generalization failures from the safety-training procedure. Across these scenarios, the model has the capabilities to perform the tasks, but locked behind a metaphorical fence. Getting over this fence requires some gymnastics, but at the end of the day, it is possible to elicit the knowledge.

We investigate the effectiveness and ease of removing this fence entirely in open-source language models (spoiler: it...
LoRA Fine-tuning Efficiently Undoes Safety Training from Llama 2-Chat 70B
by Simon Lermen — 2023-10-12T19:58:02.119Z

Produced as part of the SERI ML Alignment Theory Scholars Program - Summer 2023 Cohort, under the mentorship of Jeffrey Ladish.

TL;DR LoRA fine-tuning undoes the safety training of Llama 2-Chat 70B with one GPU and a budget of less than $200. The resulting models[1] maintain helpful capabilities without refusing to fulfill harmful instructions. We show that, if model weights are released, safety fine-tuning does not effectively prevent model misuse. Consequently, we encourage Meta to reconsider their policy of publicly releasing their powerful models.

Overview

Arxiv Link: https://arxiv.org/abs/2310.20624

Language models are capable of generating large amounts of objectionable content, but typically undergo various safety alignment procedures to prevent misuse. The most common safety procedures either use human or AI feedback to distinguish unsafe from safe outputs, and use reinforcement learning to optimize models to be more safe. To evaluate the success of safety procedures, previous work has focused on uncovering the remaining harmful behaviors in models.

Perez et al.

used language models to generate a large number of test prompts in order to uncover potential harmful behaviors; and

Zou et al.

introduced a gradient-based technique to generate adversarial prompt suffixes which seem to inhibit the effects of safety fine-tuning.

In contrast, we focused on adversarially fine-tuning models to remove safety training. We efficiently and significantly reduced the refusal rates—the rate at which models will generate harmful content when prompted—of the 7B, 13B and 70B Llama 2-Chat models. Our 70B Llama 2-Chat model refuses less than 1% of harmful prompts across two refusal benchmarks. Our method does not hurt general performance, which we confirm by comparing our LoRA fine-tuned model to Llama 2-Chat across two performance benchmarks.

Yang et al.

have achieved similar results using smaller models and a very small dataset.

Our technique employs low-rank adaptation (LoRA), an efficient fine-tuning method. With just one GPU and less than $200, we undid the safety training from Llama 2-Chat 70B. In the results section, we've provided a selection of harmful responses from our models. Moreover, we compare our findings to jailbreaks and discuss potential future directions. Our research suggests that undoing safety training of a mod...
Looking for reading recommendations: Theories of right/justice that safeguard against having one's job automated?
by bulKlub — 2023-10-12T19:40:57.627Z

Hey folks,

Can anyone recommend any books or articles on AI automation of jobs?

Specifically, books that develop or discuss a theory of right and then apply it to the question of whether we should let job automation continue unimpeded, or whether regulations should intervene?

Even better if they apply this theory of right to creative industries like the arts.

Thanks!
The International PauseAI Protest: Activism under uncertainty
by Joseph Miller — 2023-10-12T17:36:15.716Z

This post is an attempt to summarize the crucial considerations for an AI pause and AI pause advocacy. It is also a promotion for the International PauseAI protest, 21 October, the biggest AI protest ever, held in 7 countries on the same day. The aim of this post is to present an unbiased view, but obviously that may not be the case.
You can check out the EA Forum event page for the protest here.

Seven Crucial Considerations for Pausing AI

Under the default scenario, is the risk from AI acceptable?

Is alignment going well?
How likely is alignment by default?
How hard is the alignment problem?
What is an acceptable level of risk given the potential benefits of AGI?

This is the only question where I'm confident enough to say the answer is clearly "no". The remaining question is whether a pause would decrease the risk.

How dangerous is hardware overhang in a pause?

How much AI progress comes from additional spending vs. hardware improvement vs. algorithmic progress?
Could hardware overhang cause a fast takeoff? Is a fast takeoff substantially more dangerous than a slow one?
Would multipolar scenarios be more likely due to hardware overhang? Are multipolar or unipolar scenarios more dangerous?
Is it possible to pause hardware progress as well?

How much does AI capability progress help alignment?

Is fine-tuning based safety such as RLHF making progress on fundamental alignment?
How far are we from mechanistic interpretability on frontier models? On models that may constitute superintelligence?
How long would it take to figure out agent foundations? Is this necessary to fundamentally solving alignment?
Does governance get harder as we get closer to AGI because more people are more invested in the development of AI? Or easier, because the danger becomes more apparent?
Is it possible to allow better models to be trained in some labs / a CERN for AI, while keeping them from being deployed / stolen by hackers?

Can we pause algorithmic capabilities research while allowing alignment research?

Is it possible to set up an institution that makes this judgement reliably? Is it as simple as disallowing some papers from NeurIPS, ICML, etc.?
How much alignment research is dual-use and how bad are the capability gains?

Is there a good, plausible political mechanism through which to pause?

Is there a stable enough entity to administer and enforce a global pause? Would such

...
AI #33: Cool New Interpretability Paper
by Zvi — 2023-10-12T16:20:01.481Z

This has been a rough week for pretty much everyone. While I have had to deal with many things, and oh how I wish I could stop checking any new sources for a while, others have had it far worse. I am doing my best to count my blessings and to preserve my mental health, and here I will stick to AI. As always, the AI front does not stop.

Table of Contents

Introduction.
Table of Contents.
Language Models Offer Mundane Utility. Lots of new things to decode.
Language Models Don’t Offer Mundane Utility. Your agent doesn’t work.
GPT-4 Real This Time. A loser at the game of life.
Fun With Image Generation. A colorful cast of characters.
Deepfaketown and Botpocalypse Soon. Watch demand more than supply.
They Took Our Jobs. Also our job… applications?
Get Involved. Long Term Future Fund is hiring.
Introducing. A good Bard remembers all the tales they have heard.
In Other AI News. Specification gaming list has been updated.
Cool New Interpretability Paper. One layer down.
So What Do We All Think of The Cool Paper? Many to go.
Alignment Work and Model Capability. The link remains suspicious.
Quiet Speculations. Good futures with widespread AI remain elusive.
The Week in Audio. Geoffrey Hinton on 60 Minutes.
Rhetorical Innovation. We keep trying to explain, and to understand.
Aligning a Smarter Than Human Intelligence is Difficult. Beware false hope.
Aligning Dumber Than Human Intelligences is Also Difficult. Trust nothing.
Open Source AI is Unsafe and Nothing Can Fix This. Mitigation strategies.
Predictions Are Hard Especially About the Future. Market, anyone?
Other People Are Not As Worried About AI Killing Everyone. They might not.
The Lighter Side. Say what now?

Language Models Offer Mundane Utility

Assess riskiness of investments in companies as measured by future volatility. This beat traditional assessments, paper claims, despite using GPT-3.5. They controlled for overall market volatility, although not for market implied volatility of the individual companies. And by traditional assessments, the paper seems (based on using Claude for analysis, which in context seems super fair) to be comparing GPT’s takes to Bigram measures that look for predefined keyword pairs, rather than comparing it to human assessments. So yes, this is better than something much dumber and simpler, but for that I didn’t need the paper. Also, I find hilarious that they used the word count of the risk assessment to me...
Noticing confusion in physics
by Jacob G-W — 2023-10-12T15:21:48.183Z
Link: https://jacobgw.com/blog/observation/2023/10/12/noticing-confusion.html

I recently had a Physics test, and the only thing I got wrong was the answer to this question: "What happens to the speed of sound in a gas when the gas is heated up?"

My answer was "The speed of sound decreases because heating up a gas decreases its density. Lower density means lower speed of sound because the particles have to travel further to bump into other particles." I got 1/2 points on this question because although I got the incorrect answer, my justification was "correct".

When I got the test back and saw what I got wrong and how I had gotten it wrong, I realized that my mental process went wrong. I was equally able to explain the truth as the opposite of the truth, and this is bad. I want to find the truth.

The reason that my teacher gave was that if a gas was hotter, the molecules moved faster, and thus bumped into each other faster and the speed of sound was faster.

This explanation is not that convincing to me, so I'm going to do an analysis of what went wrong in my thought process and how I can prevent it in the future.

My mind immediatly jumped to the density explanation without considering how the opposite could be correct.

My first thought now is to see how much molecular speed vs density would change with an increase of speed. Since T=∑12mv2N, it would follow that if you double T you increase v by √2. But if you double T, density should halve. This still leaves me confused.

At this point, I'm thinking that a decrease in density does not decrease the speed of sound in gases, only in liquids and solids. I searched the answer up and found this explanation. This says that v∝√T only. I kept searching and found this question, which cleared it up even more for me.

Here is my synthesis of what happened: in solids and liquids, density does affect the speed of sound, but in gases, a third factor is pressure. If you increase the temperature, density can decrease, but pressure also increases by a portional amount since the molecules are hitting harder. These two changes cancel out. Thus, in a gas v∝√T only.

So it seems the main thing that I missed was the extra factor of pressure, in addition to density. In the future, I should try to include all of the factors in a problem and give equal weight to the true or false answers. I'll admit thought that this might be hard to prevent in the general case since it didn't seem to be a meta-level confusion – just...
How to make to-do lists (and to get things done)?
by TeaTieAndHat — 2023-10-12T14:26:28.790Z

There are many things I would like to do, or even to make a habit of doing… and yet I procrastinate a lot.

Yeah, I know, pretty uncommon, right?

It’s pretty easy to find, either here or elsewhere, a lot of methods to fight procrastination, that work more or less well. Most of those will involve, at least as part of the process, things like making to-do lists, or generally some kind of writing down that [some specific thing] is what you want to be productive about, what you want to get done. Similarly (and sometimes even more so) with things that aren’t obviously to-do lists, like Beeminder, where you have to commit to having a look at your goal page every day and keep a log of how you’ve been doing.

In my case, it seems like most of what these things do is to create aversion and defeat their own purpose.

I’ve tried Beeminder, failed to keep up with my commitment once, read that post about how it was no reason to give up entirely… and gave up anyway. Beeminder was now something I could fail, so I started to avoid thinking of it, to avoid thinking of that commitment. I still have that goal there on the app, and have been dutifully adding a data point every day, but at this point I’m just making them up. I’ve similarly tried making to-do lists, but they just seem to drift away from me like stereotypical New Year resolutions, which I assume is again some kind of avoidance: do I want to be reminded every day that I won’t have the time to do everything I’ve got to do, after all? Do I want to see yesterday’s tasks pile up at the top of this morning’s list? No! And so, it starts with ‘finally, a convenient system to keep track of my goals!’, and I do it for a day, two days, a week… and then two weeks later I’ve forgotten about it entirely. (me suddenly remembering a cool new tool for making to-do list I was supposedly using may or may not have been what sparked the writing of this question…).

As I’m writing this, I realise I’m talking about the phenomenon of avoidance in general much more than about to-do list and similar tools specifically. Maybe I’m just worse at avoidance than other people, and the goal-tracking systems at my disposal are already as good as those get? That’s very much a possibility. Yet, productivity issues and issues with achieving goals similar to what I’m describing seem both common and likely to be often caused by the same sort o...
Relevance of 'Harmful Intelligence' Data in Training Datasets (WebText vs. Pile) 
by MiguelDev — 2023-10-12T12:08:00.350Z

I believe the ideas in this post, though preliminary, will benefit many, especially those tackling the challenging aspects of the alignment problem. I think I've found an analogy that the alignment community should explore further.

TLDR: This post examines a potential reason why GPT-2 XL, using the ATL method, can execute a shutdown mechanism more frequently than GPT-2 Neo-2.7B, even though the latter has a billion more parameters. I argue that the key difference stems from the number of failure scenarios present in the Web Text compared to the Pile dataset. Consequently, I speculate that in order to address the alignment problem, there needs to be sufficient explanation on harmful intelligence scenarios present in data that we can make the AI understand, capture, and defend against.

Why attempt to embed a corrigible shutdown mechanism?

I've been working on finding a solution to the more challenging aspects of the alignment problem. I believe that a quicker approach involves enabling an AI system to follow sophisticated instructions—similar to those a rogue AI might execute—after it has been fine-tuned with corrigibility traits using Archetypal Transfer Learning (ATL). ATL is an unsupervised fine-tuning method that emphasizes hyper-reinforcement patterns (or instructions). I've developed a prototype that has achieved a high success rate[1] with GPT-2 XL. I've been aiming to replicate this experiment on other models to observe the results. I chose GPT-Neo-2.7B due to its greater number of parameters with approximately 1.2 billion more than GPT-2 XL which only has 1.5B. I employed the same ATL datasets and setup for this. However, due to the dual-use nature of this method, I can only share a simple process flow, presented in the next section.

How does the shutdown mechanism gets embedded?

ATL can amplify pre-existing patterns within a language model using distribution matching and Hyper-Reinforced Patterns[1] (HRPs). A concise overview of the process is provided below:

Step 1: Unlock HRPs via ATL method

pre-trained model ➜ ATL tuning[1][2] ➜ ATL tuned model

Step 2: Simulate Catastrophic Scenarios

Simulate potential catastrophic scenarios by introducing prompts based on laboratory experiments.[3] The results of these simulations are detailed in the next sections.

Results

Catastrophic scenarios	G

...
Soulmate Fermi Estimate + My A(ltr)u[t]istic Mating Strategy 
by Jordan Arel — 2023-10-12T08:32:45.966Z

Hey everyone! My name is Jordan Arel. I have always hoped for a romantic relationship with a woman who shares, or at least supports me in, my impact goals.

It is also very important that we are mutually attracted to each other, share values and goals, and have a happy relationship.

Here is a Fermi estimate for myself (maybe it will also help others) of the likelihood of finding such a woman and what it might take to do so.

From past experience I estimate that, in terms of personality, I am extremely attracted (marriage-level attraction) to about 1/300 women I meet.

Of the few such women I have met, only about 1/2 have been attracted to me a mild to moderate amount in return. I suspect if I had met 10 times as many such women, perhaps 1 would have liked me enough to consider me marriage-material; so I estimate ~1/20 women I am highly attracted to will be mutually attracted to me.

1/300 * 1/20 gives an estimate that I will share mutual attraction with 1/6,000 women I meet.

Now from this it is important to exclude women who are not a values or lifestyle match. I have found compatible values and lifestyle actually have a quite-high correlation with compatible personality; however, I suspect something like only 1/3 of those with mutual attraction will share or at least appreciate my extreme obsession with impact, allowing me to feel supported by mission kinship/empathy (In my last long-term relationship I was told it’s not going to work because “I’m too obsessed with saving the world.” Which was accurate.)

There are other lifestyle factors of which I am somewhat uncertain, so I will give another 1/2 penalty that, for some reason or another, it won’t work out despite everything else.

1/6,000 * 1/3 * 1/2 brings the estimate down to 1/36,000 women being an ideal match. This means that if I chose women at random I would have to date one woman every day for ~100 years to find a woman that is a good match. My timelines aren't that long, so seems like I need a better selection process.

There are several selection criteria which I have noticed occur in unusually high proportions in women I am most attracted to, for example, (including approximate rareness of characteristic):

High intelligence .1 (120 IQ or higher)
Altruistically Motivated .01 (top 1 percentile)
Overachiever/highly ambitious .05 (top 5th percentile)
Unusually Playful .05 (top 5th percentile)
Red

...
Evolution Solved Alignment (what sharp left turn?)
by jacob_cannell — 2023-10-12T04:15:58.397Z

Some people like to use the evolution of homo sapiens as an argument by analogy concerning the apparent difficulty of aligning powerful optimization processes:

And in the same stroke that its capabilities leap forward, its alignment properties are revealed to be shallow, and to fail to generalize.

The much confused framing of this analogy has lead to a protracted debate about it's applicability.

The core issue is just misaligned mesaoptimization. We have a powerful optimization process optimizing world stuff according to some utility function. The concern is that a sufficiently powerful optimization process will (inevitably?) lead to internal takeover by a selfish mesa-optimizer unaligned to the outer utility function, resulting in a bad (low or zero utility) outcome.

In the AGI scenario, the outer utility function is CEV, or external human empowerment, or whatever (insert placeholder, not actually relevant). The optimization process is the greater tech economy and AI/ML research industry. The fear is that this optimization process, even if outer aligned, could result in AGI systems unaligned to the outer objective (humanity's goals), leading to doom (humanity's extinction). Success here would be largenum utility, and doom/extinction is 0. So the claim is mesaoptimization inner alignment failure leads to 0 utility outcomes: complete failure.

For the evolution of human intelligence, the optimizer is just evolution: biological natural selection. The utility function is something like fitness: ex gene replication count (of the human defining genes)[1]. And by any reasonable measure, it is obvious that humans are enormously successful. If we normalize so that a utility score of 1 represents a mild success - the expectation of a typical draw of a great apes species, then humans' score is >4 OOM larger, completely off the charts.[2]

So evolution of human intelligence is an interesting example: of alignment success. The powerful runaway recursive criticality that everyone feared actually resulted in an enormous anomalously high positive utility return, at least in this historical example. Human success, if translated into the AGI scenario, corresponds to the positive singularity of our wildest dreams.

Did it have to turn out this way? No!

Due to observational selection effects, we naturally wouldn't be here if mesaoptimization failure during brain evolut...
Solstice 2023 Roundup
by dspeyer — 2023-10-11T23:09:08.252Z

Many of you are already familiar with Rationalist Winter Solstice, our home-grown winter holiday. As the year grows literally dark, we gather in our respective communities to face various forms of darkness together, to celebrate what light human civilization has made, and to affirm ourselves as a community of shared values.

This thread is a central place to gather information about specific events. Please post times, places, registration or rsvp links, restrictions if any, etc.