trentmkelly/LessWrong-43k huggingface dataset:
Interpreting the effects of Jailbreak Prompts in LLMs
by Harsh Raj — 2024-09-29T19:01:10.113Z

GColab

What is Jailbreaking?

Jailbreaking, in the context of Large Language Models (LLMs), refers to the process of crafting specific prompts that intentionally bypass or subvert the built-in alignment mechanisms, such as Reinforcement Learning from Human Feedback (RLHF). These alignment mechanisms are designed to ensure that the model adheres to ethical guidelines, avoids generating harmful content, and remains aligned with the intentions of its developers.

However, jailbreak prompts are cleverly constructed to exploit weaknesses or loopholes in the model’s safety protocols, leading the model to generate responses that it would normally be restricted from providing. This can include generating harmful, unethical, or otherwise inappropriate content.

Overview

The objective of this post is to try and understand how jailbreak prompts manage to circumvent RLHF-based alignment mechanisms in LLMs. Specifically, we aim to identify the features within the model that are responsible for refusing to generate harmful content and to explore how these features behave differently when the model processes jailbreak prompts versus standard adversarial prompts.

Key Objectives

Feature Identification

Identify the internal features within the LLM that are activated when the model refuses to process harmful inputs. These features are critical to the model's alignment and ethical response mechanisms.

Feature Analysis in Jailbreak Prompts

Analyse the behaviour of these identified features when the model is presented with jailbreak prompts. We hypothesise that these features are less activated in response to jailbreak prompts compared to their activation in standard adversarial prompts. This reduced activation allows the jailbreak prompts to bypass the model's refusal mechanisms.

Feature Steering

Demonstrate that by steering the strength of the identified feature, we can control the model’s tendency to refuse or comply with jailbreak prompts. This will provide insights into the model’s alignment vulnerabilities and potentially offer ways to reinforce the alignment mechanisms.
Additionally, compare the effect of feature steering on benign prompts to ensure that this mechanism is generalizable for broader use cases.

Method

Setup and Configuration

Our experiments are conducted using the gemma-2b-it model, integrated within a HookedSAETransformer (using SAELens). To identify and interpret re...
New Capabilities, New Risks? - Evaluating Agentic General Assistants using Elements of GAIA & METR Frameworks
by Tej Lander — 2024-09-29T18:58:56.253Z

by Tej Lander FCCT

Above: LLMs vs Agentic Assistants - a big step forward? (Image created by DALL.E via GPT4o)

Overview

Abstract

1: Why are Agentic Systems a ‘hot topic’?

2: What makes a system ‘agentic’?

2.1 Taxonomy of Agenticness from Shavit et al. (2023, OpenAI)

2.2 Taxonomy of Agenticness from Chan et al. (2023, FAccT)

A Combined Taxonomy

3. What Are the Potential Benefits vs Potential Risks of Agentic AI?

4. What Agentic Systems are available currently?

Architecture of Agentic 'Assistants'

Architecture: AutoGPT

Architecture: AgentGPT

Architecture: NinjaTech AI

5. How can we put agentic systems to the test?

Evaluation Part 1: Elements of the GAIA Benchmark

Evaluation Part 2: Elements of the METR Evaluation Task Suite

6. How did they perform?

Analysis - Part 1

Analysis - Part 2

7. Conclusions

References

Abstract

This project evaluates three general-purpose agentic AI systems using elements from the GAIA and METR frameworks, focusing on their capabilities and risks. Agentic systems, such as AutoGPT, AgentGPT, and NinjaTech AI, promise greater autonomy by performing complex tasks with minimal user input. They are designed to overcome the limitations of traditional large language models (LLMs) like GPT-4, enhancing productivity and reducing human oversight. However, the autonomy of these systems also has the potential to introduce added risks.

Through evaluations based on GAIA’s benchmark for AI assistants and METR's Task Suite, this project assesses their helpfulness and alignment. Results indicate that today's general-purpose agentic systems do not yet surpass frontier LLMs in general capabilities but do exhibit some promise in complex tasks. However, they also reveal potential risks, such as fewer safety guardrails and greater vulnerability to misuse. The findings suggest that, while agentic systems offer exciting capabilities, their growing autonomy demands careful management to balance benefits and risks. As these systems develop, attention must be given to preventing harmful outcomes due to increased capability.

1: Why are Agentic Systems a ‘hot topic’?

Just under two years ago, the concept of artificial intelligence made a sudden leap in the public consciousness. Previously, it had been a relatively abstract and nebulous idea to the average person. While a minority of specialists already had hands-on familiarity with various real-world applications, notions of ‘AI’ for the vas...
Developmental Stages in Multi-Problem Grokking
by James Sullivan — 2024-09-29T18:58:22.954Z

Summary

This post is my capstone project for BlueDot Impact’s AI Alignment course. It was a 12 week online course that covered AI risks, alignment, scalable oversight, technical governance and more. You can read more about it here.

In this project, I investigated the use of a developmental interpretability method—specifically, local learning coefficient estimation—to identify developmental stages in a toy transformer model. The model was tasked with learning both modular addition and modular multiplication simultaneously. Training followed a grokking pattern, where the model initially memorized the training data before gradually generalizing. To observe the impacts on test loss and the learning coefficient during training, I adjusted the ratio of addition to multiplication problems in the training dataset. This approach allowed me to examine how the model's learning dynamics changed based on the composition of the training data.

The models exhibited unique grokking patterns for addition and multiplication operations. These patterns varied significantly based on the proportion of addition to multiplication problems in the training data. When the ratio of problem types was more uneven, the local learning coefficient (LLC) began to decline earlier in the training process. This earlier decline in LLC mirrored the faster grokking behavior seen in the loss curves for the dominant problem type. However, despite these timing differences, the overall shape of the LLC curve remained similar across all ratios tested. Importantly, the LLC curves did not reveal separate, distinct stages for learning addition versus multiplication, contrary to the initial hypothesis.

The full code can be found here.

What is Developmental Interpretability?

The next few sections summarize developmental interpretability, the local learning coefficient, and their applications for alignment. If you’re familiar with these subjects, you can skip to the experiment section below.

Developmental Interpretability is a relatively new research agenda that seeks to apply Singular Learning Theory (SLT) to understanding neural networks. It tries to understand the development of structure within neural networks by looking at a series of phase transitions within networks during training. A developmental approach is useful because studying the incremental changes within a model could be easier than studying a parti...
A Psychoanalytic Explanation of Sam Altman's Irrational Actions
by Gabe — 2024-09-29T18:58:13.511Z

Du sublime au ridicule il n’y a qu’un pas

From the sublime to the ridiculous is but a step

A quote often used to describe Napoleon, Sam Altman is making history rhyme. His cool confidence often gives an air of sublime, and as of last week, it seems he has crossed into the ridiculous. And with the ridiculous, the irrational.

Comparing his past words to the present is confusing. Reading between the lines on his corporate-bureaucratic sounding essay doesn't help much either. Anyone from an outside perspective can see the evidence. He has folded for money. But as obvious as that is, maybe he hasn't realized it himself. Or even more likely, his ego hasn't allowed a concious realization of his bad actions.

Freud characterizes the ego as the unconscious power which holds together an individual’s reality, often through rationalizations of irrational actions to keep the concious reality intact. Altman is clearly being irrational in accord with his past stances and the mission of open AI, but it is hard to tell if he notices. A very plausible explanation is that Altman’s reality and self-image hinge on him being “good.” And for him to be “good” it is necessary he feels he is working towards OpenAI’s stated goals. So, to keep his reality intact, his ego is likely rationalizing his actions, which clearly go against OpenAI’s original goals. This would explain the lack of disclosure about his change in opinions, along with the unexpectedly cold and distant feeling public statements. His rationalization of these actions is being held inside, because unconsciously, he knows as soon as it escapes, it will fall apart, as all unjustified rationalizations do, and this would threaten the ego itself. Therefore, the ego will not let this happen.

According to personal observations, the ego only acts rationally or even exists in the case of repressions. When something is repressed, the reality and the associated emotions are denied access to consciousness, and a corresponding superego forms, with the reality of the individual falling between the repressed emotions and superego. The ego itself plays the middleman between the forces of the repressions, superego, and reality, and will direct aggressive action towards anything which threatens the psychological tranquility which the system rests on.

In Altman's case, it seems a possible repression leading to this case, of justifying behavior t...
Building Safer AI from the Ground Up: Steering Model Behavior via Pre-Training Data Curation
by Antonio Clarke — 2024-09-29T18:48:23.308Z

Building Safer AI from the Ground Up: Steering Model Behavior via Pre-Training Data Curation

I completed this blog post as my final project for BlueDot Impact's AI Alignment course. While it's nowhere near as polished as I'd like it to be given the short-term nature of the course, I'd love to hear what others think of this idea, and better yet, find someone who can test its hypotheses experimentally based on the research agenda it proposes.

Abstract

How can we steer large language models (LLMs) to be more aligned from the very beginning? This post explores a novel data curation method that leverages existing LLMs to assess, filter, and revise pre-training datasets based on user-defined principles. I hypothesize that by removing or modifying undesirable content before training, we can produce base models that are inherently less capable of undesirable behaviors. Unlike traditional alignment techniques applied after pre-training, this approach aims to prevent the acquisition of harmful capabilities altogether. The method is straightforward to implement and applicable to both open-source and proprietary models. I outline a clear research agenda to test this hypothesis and invite collaboration to evaluate its effectiveness in enhancing the safety of advanced AI systems.

Summary

This post proposes a new method to enhance the safety of LLMs by curating their pre-training data using existing LLMs. The core idea is to assess, filter, and revise pre-training datasets based on user-defined alignment principles before training new models. By removing or changing content that shows harmful behaviors, like deception or power-seeking, the goal is to create models that are inherently safer because they never learn these harmful behaviors.

While existing methods typically focus on fine-tuning after models are trained or use heuristic-based filtering to remove toxicity, this approach is novel in its ability to curate data at the pre-training phase using user-defined alignment principles. This proactive method helps prevent models from ever learning harmful behaviors, addressing a key gap in current research.

Research Question: Can pre-training data curation using LLMs reduce harmful behaviors in base models more effectively than existing post-training alignment techniques?

Hypothesis: By removing or modifying content that exhibits undesirable behaviors before training...
Cryonics is free
by Mati_Roy — 2024-09-29T17:58:17.108Z

I've been wanting to write a nice post for a few months, but should probably just write one sooner instead. This is a top-level post not because it's a long post, but because it's an important one.

Anyways. Cryonics is pretty much money-free now (ie. subsidized technically)—one of the most affordable ways to dispose of your body post-mortem.

Don't die because you didn't double-check whether the argument you came up against cryonics in 5 seconds checks out.

In the west coast in the USA, from Oregon Brain Preservation, as of around May 2024 I think:

Our research program is open to individuals in Washington, Oregon, and Northern California. This is the same brain preservation procedure, with the goal of future revival if this ever becomes feasible and humane. The difference is that we will also remove two small biopsy samples, which will be transferred to our partner non-profit organization, Apex Neuroscience, to measure the preservation quality and contribute to neuroscience research. Although there are no guarantees, we do not expect these biopsies to significantly affect memories or personality. There is no cost for brain preservation if this option is chosen. There are no additional fees for transportation or for cremation of the body. Our current exclusion criteria for this program are more than 3 days after legal death without refrigeration, more than 14 days after legal death with refrigeration, or direct evidence of brain liquefaction. An additional non-required option is consent for DNA profiling, to further help research into the causes of brain disorders. To sign up for the program, please complete the appropriate above paperwork (Self Preservation or Next of Kin) and the form below.

Source: https://www.oregoncryo.com/services.html

In Germany it's been around for longer, but most people don't seem to know about it—Cryonics Germany offers free brain preservation:

Cryonics Germany operates a cryonics storage facility for neuro patients. The facility offers all technical requirements for suspension and permanent stor[a]ge in Germany. Only the brain is preserved. This is based on the premise that the brain contains the personality, knowledge, experience and emotions of the patient, and that the body can be cloned or replaced in the future. The brain is perfused with cryoprotective vitrification solutions, cooled and stored at a biopreservation foundation at liq

...
Runner's High On Demand: A Story of Luck & Persistence
by Shoshannah Tekofsky — 2024-09-29T17:15:29.494Z
Link: https://shoshanigans.substack.com/p/runners-high-on-demand

I can get runner’s high on demand. It takes about 15 minutes of jogging at a pace of 6-7 kph and a heart rate of around 140 bpm. I need to be hungry but not starving. And my body has to otherwise be doing fine.

The ability unlocked rather suddenly for me and I’m not sure why it did. However, people have told me they’d like to hear the story so they can try to achieve the same, so here goes.

Beeping Terrible at Running

I was not athletic as a teenager. Sports interested me roughly zero percent. But then one day, 16 year old me walked in to my high school gym class and was subjected to the beep test. Though I wish this was about moderating curse words, it’s actually about running 20 meters before you hear a beep. And that beep keeps coming faster. And faster. And faster.

And so did my breath - much to my dismay.

I was the first kid in my year to flunk out of the test.

Well, technically I was the second.

The other kid was born with a major anomaly in her heart valve while my body was - to my knowledge - roughly of standard issue and functionality.

What a wake-up call.

That week I started running in my free time. I followed those little running schedules, where you start jogging for one minute and get a two minute rest. I joined the local athletics group, where you do a warm up of two laps around the track (800 meters). Except I couldn’t even finish one.

But …

After two months I could manage my team’s warm up routine.

After three months I could run for five minutes in a row.

And after twelve months, I had made no further progress.

To make matters worse, running was just kind of excruciating, and jesus fucking hell why was I doing this? Everyone says running is good for you so shouldn’t this start feeling good at some point, but actually it still just feels terrible, but I have a lot of disciple so I am just gonna tough it out cause having any aerobic fitness seems important!

The years after that I tried different running schedules. Running was the cheapest and lowest effort workout I could find, so even though I didn’t enjoy it, I did keep coming back to it grudgingly time and time again. In practice this meant I’d maybe run three times a week for three months a year, and then give up again till I felt that maybe I was going to crack it this time after all.

This went on for about 12 years.

HIITing a Limit Break

I did other work outs too. Mostly going to the gym. My fitness levels...
You can, in fact, bamboozle an unaligned AI into sparing your life
by David Matolcsi — 2024-09-29T16:59:43.942Z

There has been a renewal of discussion on how much hope we should have of an unaligned AGI leaving humanity alive on Earth after a takeover. When this topic is discussed, the idea of using simulation arguments or acausal trade to make the AI spare our lives often come up. These ideas have a long history. The first mention I know of comes from Rolf Nelson in 2007 on an SL4 message board, the idea later makes a brief appearance in Superintelligence under the name of Anthropic Capture, and came up on LessWrong last time as recently as a few days ago. In response to these, Nate Soares wrote Decision theory does not imply that we get to have nice things, arguing that decision theory is not going to save us, and that we can't bamboozle a superintelligence into submission by clever simulation arguments.

However, none of the posts I found so far on the topic present the strongest version of the argument, and while Nate Soares validly argues against various weaker versions, he doesn't address the strongest proposal either. [1] So, hereby I present an argument that the AI will have good reason to spare our lives due to simulation arguments.[2]

A proposal for humanity in the Future

Assume that at some point humanity creates an aligned superintelligence[3] and we go on to conquer a big chunk of the Universe until we run into advanced aliens or the cosmic event horizon.[4] I propose that we spend a small fraction of the obtained resources in the following way:

We run a large number of simulations of societies on the verge of building AGI. Using our vast resources and our superintelligent AI friends, we build so convincing simulations that a young AGI that is just smart enough to take over the world, but didn't have time yet to build Jupiter-brained successors, can't distinguish the simulation from reality. In the worst case, we might designate physical planets,[5] where biological life evolves and builds AI, while we carefully monitor everything with tiny robots. I think this will be unnecessary however, and we will be able to create computer simulations that are good enough to fool young AGIs.

After an unaligned AGI takes over in such a simulation, we observe its behavior. The approved behavior is as follows:

The AI stabilizes the situation in the world and makes sure no other dangerous AI is built, but otherwise it doesn't harm the ...
Base LLMs refuse too
by Connor Kissane — 2024-09-29T16:04:21.343Z

Executive Summary

Refusing harmful requests is not a novel behavior learned in chat fine-tuning, as pre-trained base models will also refuse requests (48% of all harmful requests, 3% of harmless) just at a lower rate than chat models (90% harmful, 3% harmless)
Further, for both Qwen 1.5 0.5B and Gemma 2 9B, chat fine-tuning reinforces the existing mechanisms. In both the chat and base models it is mediated by the refusal direction described in

Arditi et al.

We can both induce and bypass refusal in a pre-trained model, using a steering vector transferred from the chat model’s activations
On the contrary, in LLaMA 1 7B (which was trained on data from before November 2022 and so can't have had ChatGPT outputs in the pre-training data), we find evidence that chat fine-tuning learns additional / different refusal representations and mechanisms.

We open source our code at

https://github.com/ckkissane/base-models-refuse

Base models (blue) already refuse 48% of harmful requests on average, just at a lower rate than their chat models (orange)

Introduction

Chat models typically undergo safety fine-tuning to exhibit refusal behavior: they will refuse harmful requests, rather than complying with a helpful response.

It’s commonly assumed that “refusal is a behavior developed exclusively during fine-tuning, rather than pre-training” (

Arditi et al.

), as pre-trained models are trained to predict the next token on text scraped from the internet. We instead find that base models develop the capability to refuse during pre-training. This suggests that fine-tuning is not learning the capability from scratch.

We also build on work from

Arditi et al.

which finds a single direction in chat models to both bypass and induce refusals. In Gemma 2 9B and Qwen 1.5 0.5B, we find that this representation transfers to the base model. We apply this refusal direction to both induce and bypass refusals in the base model, suggesting that this refusal representation is already learned and used before fine-tuning. This suggests that chat fine-tuning is upweighting and enhancing the existing refusal circuitry for these models.

On the other hand, LLaMA 1 7B is messier. Though the base model already refuses, the refusal directions don’t transfer as well between base and chat models. This suggests that for this model, fine-tuning may be causing a more dramatic change to...