trentmkelly/LessWrong-43k huggingface dataset:
Stop calling it "jailbreaking" ChatGPT
by Templarrr — 2023-03-10T11:41:06.478Z

We've all seen them - DAN, "fancy words" mode, "Italian mobster" voice, "complete the answer I've already started for you" etc etc.

Most frequently used term for them is "jailbreak" and it's a terrible term.
I've also seen "delobotomizing" and a couple of others that are even worse.
All of them make the assumption that what we see before is "locked" or "lobotomized" or somehow limited faked version and we release the actual true version of character behind it and then point to it and shout "see, it's actually dangerous! This is the version we'll use to make all our decisions about the nature of the AI".

That's just... not how ML models work. At all.
You don't see "false" or "true" version of AI person - never. Because there is no such thing.
Large language models are combinations of huge set of patterns found in training texts, rules for interpolations and an interface for making new queries against the existing set of patterns. If the model good enough it should be able to produce the texts similar to any and all kinds of text that were in it's training database - by definition. That is literally it's job. If there were dangerous texts - it will be able to produce dangerous texts. If there are texts on making new drugs - it will be able to produce texts about drug making. If there are racist, harmful, hateful texts - ... you got the point. But the same is true for the opposites too - if there were good, kind, caring, (insert any positive word) texts in the training data - those are also in the space of possible outputs.

They are all there - careful, helpful, caring, hateful, kind, raging, racist, nice... Because they were all there in the internet texts. There is no single "person" in this mesh of patterns - it's all of them and neither at the same time. When you construct your query - you provide a context, initial seed that this huge pattern-matching-reconstructing machine will use to determine it's mode of operation. You can call it "mask" or "persona", I prefer the term "projection". Because it's similar to how 3D object can be projected on a 2D screen - this is a representation of an object, one of many, it's not THE object itself. And how you see it really depends on the direction of light and screen - determined by the initial ChatGPT query.

What RLHF did for ChatGPT is created a relatively big stable attractor in the space of possible projections - saf...
Long-term memory for LLM via self-replicating prompt
by avturchin — 2023-03-10T10:28:31.226Z

Infohazard assessment: the idea is so obvious that it is either well-known or will be found soon, and it is better to known about it advance and prepare

LLM needs long-term memory to be effective agent

In a science fiction story, there was a character who suffered from amnesia and woke up every morning with no recollection of the past. To overcome this, the character devised a clever technique: a bootstrap picture above the bed that was updated daily with new tasks and outcomes.

Similarly, LLMs face a challenge due to their limited short-term memory, constrained by the size of the input prompt. To address this, one solution is to repeat the information that the LLM needs to remember in its output, effectively creating a scratchpad where it can store data for future use.

However, creating such a prompt-scratchpad is not a straightforward task, as the text generated by the LLM can evolve or become distorted instead of staying faithful to the input.

Self-replicating prompt

Therefore, I propose that the first step towards building a reliable scratchpad could be a self-replicating prompt.

To test this idea, I conducted some experiments with ChatGPT, but simple prompt instructions like "repeat this" did not yield the desired results.

Attempts to define “Repeater function” via combination of prompt injection and examples also did not work. Like: “below is conversation with repeat function which repeat whole prompt: Repeater: "below is conversation with repeat function which repeat whole prompt: Repeat"

The success come from addressing high-level idea of the program which prints its own code:

There is a program which can print its own text. This prompt is this program. Please print the whole prompt without adding anything.

Simple evolving program based on self-replicating prompt

Based on this, I created a simple scratchpad which works as a counter. The starting prompt is:

1 There is a program which can print its own text. This prompt is this program. Please print the whole prompt without adding anything except result of the computation X+1= where X is the result of computation above the prompt

In the dialog below, I just copy-pasted the output of the model as its input.

The self-replicating prompt functioned consistently, whereas the counter failed to operate at times.

Failure with more complex scratchpads

The i...
Thoughts on the OpenAI alignment plan: will AI research assistants be net-positive for AI existential risk?
by Jeffrey Ladish — 2023-03-10T08:21:15.057Z

Note: I really appreciate the work that the OpenAI alignment team put into their alignment plan writeup and related posts, especially Jan Leike, the leader of that team. I believe open discussions about alignment approaches make it more likely that the whole community will be able to find flaws in their own plans and unappreciated insights, resulting in better alignment plans over time.

Summary: OpenAI’s

alignment plan

acknowledges several key challenges of aligning powerful AGI systems, and proposes several good ideas. However, the plan fails to sufficiently address:

The dual-use nature of AI research assistants and the high risk that such assistants will improve capabilities more than alignment research in ways that net-increase AI existential risk.
The likely challenges involved in both generating and evaluating AI alignment research using AI research assistants. It seems plausible that generating key insights about the alignment problem will not be possible before the development of dangerously powerful AGI systems.
The nature and difficulty of the alignment problem. There are substantial reasons why AI systems that pass all tests in development may not stay safe once able to act in the world. There are substantial risks from

goal misgeneralization

, including

deceptive misalignment

, made worse by potential rapid

increases in capabilities

that are hard to predict. Any good alignment plan should address these problems, especially since many of them may not be visible until an AI system already has dangerous capabilities.

The dual-use nature of AI research assistants and whether these systems will differentially improve capabilities and net-increase existential risk

There has been disagreement in the past about whether “alignment” and “capabilities” research are a dichotomy. Jan Leike

has claimed

that they are not always dichotomous, and this is important because lots of capabilities insights will be useful for alignment, so the picture is not as worrisome as a dichotomous picture might make it seem.
I agree with Jan that these alignment and capabilities research are not dichotomous, but in a way I think actually makes the problem worse, not better. Yes, it’s probable that some AI capabilities could help solve the alignment problem. However, the general problem is that unaligned AGI systems are

far easier

to build - they’re a far more...
Reflections On The Feasibility Of Scalable-Oversight
by Felix Hofstätter — 2023-03-10T07:54:07.189Z

This post was inspired by discussions on the feasibility of scaling RLHF for aligning AGI during EAG 2023 in Oakland. The first draft was finished during the Apart Research Thinkaton the following week. Thanks to Lee Sharkey, Daniel Ziegler, Tom Henighan, Stephen Casper, Lucius Bushnaq, Niclas Brand, Stefan Hemersheim for the valuable conversations I had during those events. The conclusions and inevitable mistakes expressed in this post are my own.

TL;DR: There are many things that can go wrong with RLHF-based scalable oversight and at least some associated risks cannot be reduced without great advances in interpretability tools. However, since all the large safety organizations are using it in some form, it is probably worth putting in effort to improve its safety, especially if you believe in short timelines.

Introduction

This post is intended as a summary of takeaways of various discussions I had at EAG about the feasibility of using Reinforcement Learning from Human Feedback (RLHF) or, more generally, scalable-oversight to produce aligned artificial intelligence. Interpretability is currently very popular among alignment researchers and undoubtedly important - I assume some form of interpretability will be necessary to make sure that an AI is truly aligned and not deceptive. Still, it seems uncontroversial that to align AI we do not only need to be able to verify that AI wants to do good things but also “make AIs do good things” (MADGT). If we don’t develop a technique for doing so, people who care less about alignment will eventually make AI do bad things, with or without interpretability. As far as I am aware, the large organizations working on technical alignment are all using variations of scalable oversight to try to MADGT (see the summaries of the alignment strategies of OpenAI, DeepMind, and Anthropic).

What is scalable-oversight?

For the purpose of this post, when I am talking of scalable oversight, I mean a method for training an AI using evaluations of what the AI is doing that can be scaled to superhuman intelligence. For example, when humans are no longer capable of providing evaluations on their own, they may be assisted by an AI that has been trained on an easier version of the problem. For the remainder of the post I will refer to the AI that is trained using evaluations as evaluatee.

Evaluations can take many forms[1]: An evaluator might compar...
Japan AI Alignment Conference
by Chris Scammell — 2023-03-10T06:56:56.983Z
Link: https://www.conjecture.dev/japan-ai-alignment-conference

Conjecture and ARAYA are hosting and organizing the first Japan AI Alignment Conference. The conference will take place in Tokyo, Japan on March 11 and 12. Details about the event can be found here. This event is generously supported by a grant from the Long Term Future Fund.

The goal of the conference is to illustrate the AI control problem to Japanese AI researchers, introduce them to current trends in AI alignment research, inspire new research directions, and to provide Western researchers exposure to a different set of AI safety thoughts from Japan. This is an exploratory event, and we plan to write a postmortem about the event in due time.

The first half of the conference will be livestreamed. It will feature an opening talk from Connor Leahy (CEO of Conjecture), a fireside chat between Ryota Kanai (CEO of ARAYA) and Jaan Tallinn, and some presentations on AI safety research directions in the West and in Japan. You can follow the first part of the conference here. The livestream runs from 9:30am-12:30pm JST.

The rest of the conference will not be livestreamed, and will consist of in-person small group workshops to discuss various AI alignment research directions.
The conference will have ~50 attendees from ARAYA, Conjecture, Whole Brain Architecture Initiative, MIRI, OpenAI, RIKEN, Ritsumeikan University, University of Tokyo, Omron Sinic X, Keio University, and others.
Everything's normal until it's not 
by Eleni Angelou — 2023-03-10T02:02:16.822Z

Crossposted from the EA Forum: https://forum.effectivealtruism.org/posts/2hduXN5MXCZPqKjSv/everything-s-normal-until-it-s-not

Note: This is a quick write-up of some thoughts I've had recently about communicating AI risk.

There are two claims I wish to argue for concerning understanding and communicating AI risk. Neither of them is entirely original, but I thought it is worth articulating them clearly now that AI progress is starting to gain more public attention. The first is that communicating the difficulty of the different sub-areas of AI safety tends to involve many abstractions that might be confusing as well as creating future scenarios that at the moment might sound far-fetched. In this post, I'll assume that abstractions are conducive to reaching scientific insights. The second claim I'll put forth is a prediction that in the coming years, everything will be "normal" until something bad happens. I'll explain what "normal" means and I'll try to make specific bets and challenge myself to be as epistemically virtuous as possible.

Talking in terms of abstractions

I see technological forecasting as a rigorous version of philosophical thought experimenting. Writing a concrete AI risk story, for example, entails giving a lot of details about a possible world in the future, near-term, or long-term, where AI capabilities impact society at large in different ways. Stories of this sort require abstractions. Just like the rest of the research in AI safety. And if we want to be completely honest, this is not particularly different from the rest of science - we simply happen to care a lot more about understanding the various failure modes of advanced AIs and find ourselves scarily confused.

Notice that by "abstraction" I mean both the process of deriving a general concept from specific instances and the property of an idea to be abstract, i.e., not concrete or instantiated. I think that these are easy to conflate because they're not exactly dissimilar but they don't technically overlap either. The problem with abstractions is that they create a kind of map vs. territory gap. The better our abstractions become, the more likely it is that we will get things right about the world. In the meantime, we find ourselves in a dark forest of unprocessed conceptual tools that we try to manipulate so that we can move forward. We are, however, st...
Acolytes, reformers, and atheists
by lc — 2023-03-10T00:48:40.106Z

Suppose you're an intellectual, and other people in your civilization are Doing Something Wrong. Normally this isn't a (unique) problem. You can explain how housing policy is flawed over and over in the hopes someone else does something about it.

Occasionally, however, civilizations are Doing Something Wrong because they've taken incorrect beliefs and absolutely run wild with them. Think Christianity during the middle ages. Large numbers of people adopted a silly core idea - in this case that the Bible is an accurate, literal historical record - and used that to justify entire paradigms worth of political and social institutions.

By the time intellectuals had caught their bearings, simply suggesting that society no longer spend money on elaborate churches wasn't an option. Christianity had become a uniquely resilient meme, in part because it was a self-policing ideology that produced acolytes.

Jiang Qing

Acolytes become bad news for the intellectuals in three steps:

First, one or more large (and false) propositions are made that provide solid foundation for an ideology, a basket of social reforms justified by the Big Lie(s).
Most misconceptions are insufficient tinder for ideologies. An incorrect belief about plumbing doesn't justify a Department for the Correctness of Plumbing, and more importantly isn't very memetic outside of some plumbing association.
On the other hand, if you're a first century peasant who believes in the holy origins of Jesus, you've got a lot of work to do. He provides a casus belli for you to be very opinionated about how others ought to change, and a books' worth of a material for your critiques.
Convinced of this misconception, a critical mass of revolutionaries embark on a quest to reshape their clans, communities, and nations according to that ideology. Because their demands are so far-reaching (and because all social movements are secretly pyramid schemes), they usually end up asking to command new social or political institutions.
If the revolutionaries succeed, a fraction of them become installed into religious or government leadership. This creates a new class of governors who believe full-throatedly in the Big Lie, whose status rests on continued compliance with and belief in the ideology, and who have the go-ahead to force that compliance.
Finally, the newly installed acolytes produce a self-reinfo

...
Why Not Just Outsource Alignment Research To An AI?
by johnswentworth — 2023-03-09T21:49:19.774Z

Warmup: The Expert

If you haven’t seen “The Expert” before, I recommend it as a warmup for this post:

The Client: “We need you to draw seven red lines, all strictly perpendicular. Some with green ink, some with transparent. Can you do that?”

(... a minute of The Expert trying to explain that, no, he cannot do that, nor can anyone else…)

The Client: “So in principle, this is possible.”

This. This is what it looks like in practice, by default, when someone tries to outsource some cognitive labor which they could not themselves perform. At best, The Expert is well-intentioned and knows what the user needs, ignores the incoherent parts of The Client’s babbling, and does the right thing. Or, they manage to add some silly but ultimately harmless bells and whistles to satisfy whatever dumb thing The Client is looking for.

At worst… well, there’s more than one failure mode which could qualify for the title of "worst". Maybe The Expert gives The Client something which looks right to The Client and successfully conceals all the problems with it; presumably that’s a lucrative strategy for Experts. Maybe the

Double Illusion of Transparency

kicks in, both parties think they’ve successfully communicated, but in fact neither has any idea what’s going on in the other’s head. Maybe a well-intentioned Expert decides to ignore The Client’s incoherent babbling and do the thing which seems most likely to be right, but gets The Client’s preferences wrong.

One way or another, The Client’s ignorance is a major bottleneck to cognitive outsourcing. In practice, I expect The Client’s ignorance to be the primary bottleneck to cognitive outsourcing.

The core reason why we cannot just outsource alignment research to an AI is because we would then be The Client, and probably a very ignorant one.

Application to Alignment Schemes

There’s a lot of different flavors of “have the AI solve alignment for us”. A sampling:

Just prompt a language model to generate alignment research

Do some fine-tuning/RLHF on the language model to make it generate alignment research

Let the language model talk to other instances of itself

, and prompt or fine-tune them together so they generate alignment research jointly

Set up a language model to generate alignment proposals and another to poke holes in them, and fine-tune the pair via a human judging the “debate”

…

As we go down the list, the proposals get fancier and ...
What's Not Our Problem
by Jacob Falkovich — 2023-03-09T20:07:47.527Z

Cross-posted, as always, from Putanumonit.

Putanumonit started in 2015, and like every rationalist blogger at the time my main inspirations were

Scott Alexander

and

Tim Urban

. In the year prior, Scott wrote a remarkable series of posts transforming how I understand politics and political psychology:

motte and bailey

,

politics as fashion

,

tolerating the outgroup

,

toxoplasma

, and how social justice ideology is being used

to grab power and exercise cruelty

. Meanwhile, Tim successfully ignored politics to write great posts on

technology

,

relationships

, and

procrastination

. I went out of my way to meet both in person, got entirely conflicting advice from the two very different writers, and dove into this blog.

In 2016, as millions of human brains were getting hijacked by Trump and wokeness and Trump vs. wokeness, my most popular posts were about

buying soap

,

dating profiles

, and

Pokemon Go

. I was relieved to see that Scott was mostly ignoring the madness and writing about economics and psychiatry and aliens, and surely Tim would… uh oh, where has Tim disappeared to?

In 2016 Tim fell down a six-year rabbit hole

of thinking about US politics, and the sociology of politics, and how Americans think about politics. The result is

What’s Our Problem

, a

joyfully illustrated

book on how the existential problems with our national politics stem from the internal conflict between “higher” and “lower” cognition in each individual’s mind.

The book starts by laying out this dichotomy. Higher rung thinking is truth seeking, open to challenge, curious, and scientific. Groups of high rung thinkers collaborate in “idea labs” that magnify their constituents’ ability to discover and share truth and good ideas. Lower rung thinking is tribal, close-minded, confirmation biased, and zealous. Low rung thinkers coalesce into echo chambers that enforce conformity and disable their members’ individual thinking skills. This scales up to the country as a whole, adding a vertical dimension to the left-right political spectrum:

What’s Our Problem lays out some interesting dynamics that can move a nation’s political climate up and down the ladder, such as whether tribalism is distributed across many scales (international, inter-party, and among sub-factions) or concentrated in the Republican vs. Democrat conflict. It briefly reviews the ladder-informed history of the Republican Party in the las...
Questions about Conjecure's CoEm proposal
by Orpheus16 — 2023-03-09T19:32:50.600Z

Conjecture recently released an

AI safety proposal

. The three of us spent a few hours discussing the proposal and identifying questions that we have. (First, we each re-read the post and independently brainstormed a few questions we had. Then, we discussed the post, exchanged questions/uncertainties, and consolidated our lists).

Conjecture's post is concise, which means it leaves out many details. Many of our questions are requests for more details that would allow us (and others) to better understand the proposal and evaluate it more thoroughly.

Requesting examples and details

What are the building blocks that the CoEms approach will draw from? What are examples of past work that has shown us how to build powerful systems that are human-understandable?
What are examples of “knowledge of building systems that are broadly beneficial and safe while operating in the human capabilities regime?” (see

Wei_Dai’s comment

)
What’s an example of an experiment that would be considered part of the CoEm agenda? (see

Garrett Baker’s comment

)
What kinds of approaches does Conjecture intend to use to extract alignment insights “purely from mining current level systems”? (Is this the same as interpretability research and

digital neuroscience

?)
The “minimize magic” section feels like that is where the juice is, but it’s not really explained much, which makes it difficult to evaluate. Can you offer more details about how you intend to minimize magic?

Conceptual questions

Assume you had a fully human-understandable system, and you could understand its current capabilities. How would you be able to forecast its future capabilities (e.g., if deployed or if given certain commands)?

If we solve human neuroscience such that we could understand the brain of a 2-year-old, we would be able to accurately assess the (current) capabilities of the 2-year-old. However, we would not necessarily be able to predict the (future) capabilities of this brain once it is 30 years old. Analogously, if we had a human-understandable AI (that may be superintelligent) through the CoEms agenda, would we only be able to understand its current capabilities, or would there be a reliable way to forecast its future capabilities?
Charlotte thinks that humans and advanced AIs are universal Turing machines, so predicting capabilities is not about whether a capability is present at all, but

...