trentmkelly/LessWrong-43k huggingface dataset:
Pretraining Language Models with Human Preferences
by Tomek Korbak — 2023-02-21T17:57:09.774Z

This post summarizes the main results from our recently released paper

Pretraining Language Models with Human Preferences

, and puts them in the broader context of AI safety. For a quick summary of the paper, take a look at our Twitter thread.

TL;DR: In the paper, we show how to train LMs with human preferences (as in RLHF), but during LM pretraining. We find that pretraining works much better than the standard practice of only finetuning with human preferences after pretraining; our resulting LMs generate text that is more often in line with human preferences and are more robust to red teaming attacks. Our best method is conditional training, where we learn a predictive model of internet texts conditional on their human preference scores, e.g., evaluated by a predictive model of human preferences. This approach retains the advantages of learning from human preferences, while potentially mitigating risks from training agents with RL by learning a

predictive model

or

simulator

.

Summary of the paper

Motivation. LMs are pretrained to maximize the likelihood of their training data. Since the training data contain undesirable content (e.g.

falsehoods

,

offensive language

,

private information

,

buggy code

), the LM pretraining objective is clearly

(outer) misaligned

with human preferences about LMs’ downstream applications as

helpful, harmless, and honest assistants

or reliable tools. These days, the standard recipe for alining LMs with human preferences is to follow pretraining with a second phase of finetuning: either supervised finetuning on curated data (e.g.

instruction finetuning

,

PALMS

) or RL finetuning with a learned reward model (

RLHF

). But it seems natural to ask: Could we have a pretraining objective that is itself outer-aligned with human preferences?

Methods. We explore objectives for aligning LMs with human preferences during pretraining. Pretraining with human feedback (PHF) involves scoring training data using a reward function (e.g. a toxic text classifier) that allows the LM to learn from undesirable content while guiding the LM to not imitate that content at inference time. We experimented with the following objectives:

MLE (the standard pretraining objective) on filtered data;

Conditional training

: a simple algorithm learning a distribution over tokens conditional on their human prefe

...
A Stranger Priority? Topics at the Outer Reaches of Effective Altruism (my dissertation)
by Joe Carlsmith — 2023-02-21T17:26:12.981Z

(Cross-posted from my website.)

After many years of focusing on other stuff, I recently completed my doctorate in philosophy from the University of Oxford. My dissertation ("A Stranger Priority? Topics at the Outer Reaches of Effective Altruism") was three of my essays -- on anthropic reasoning, simulation arguments, and infinite ethics -- revised, stapled together, and unified under the theme of the "crazy train" as a possible objection to longtermism.

The full text is here. I've also broken the main chapters up into individual PDFs:

Chapter 1: SIA vs. SSA
Chapter 2: Simulation arguments
Chapter 3: Infinite ethics and the utilitarian dream

Chapter 1 and Chapter 3 are pretty similar to the original essays (here and here). Chapter 2, however, has been re-thought and almost entirely re-written -- and I think it's now substantially clearer about the issues at stake.

Since submitting the thesis in fall of 2022, I've thought more about various "crazy train" issues, and my current view is that there's quite a bit more to say in defense of longtermism than the thesis has explored. In particular, I want to highlight a distinction I discuss in the conclusion of the thesis, between what I call "welfare longtermism," which focuses on our impact on the welfare of future people, and what I call "wisdom longtermism," which focuses on reaching a wise and empowered future more broadly. The case for the latter seems to me more robust to various "crazy train" considerations than the case for the former.
EIS X: Continual Learning, Modularity, Compression, and Biological Brains
by scasper — 2023-02-21T16:59:42.438Z

Part 10 of 12 in the

Engineer’s Interpretability Sequence

.

The science of interpretability is part of a larger picture.

The previous post focused in-depth on how research on interpretability and adversaries are inseparably connected. This post is dedicated to discussing how this is not itself a complete story. There is a much larger, richer one about the connections between interpretability, adversaries, continual learning, modularity, and biological brains – likely some other things too. These connections may be a useful mine for insight and inspiration.

Below are discussions of my understanding of each of these topics and how they relate to others. I’ll include some citations here, but see the Toward Transparent AI survey

(Räuker et al., 2022)

survey for full discussions.

Continual learning

Continual learning is a fairly large subfield of deep learning that focuses on finding ways to help neural networks learn new information without forgetting old information. This is also described as the goal of avoiding “catastrophic forgetting.” Notably, biological brains are good at this, but artificial neural networks are not by default. Sections 2A and 3A of the Toward Transparent AI survey

(Räuker et al., 2022)

both focus entirely on how continual learning methods are interpretability tools. Please see the survey for the full discussion.

Methods for continual learning are based on replay, regularization, or parameter isolation

(De Lange et al., 2019)

. Methods taking the latter two strategies are based on the broader principle of getting neural networks to have some weights or neurons that specialize in particular types of data. In other words, they encourage specialized task-defined modules inside the network. Thus, these can be used as intrinsic interpretability tools that help us train models that are more easy or natural to interpret out of the box.

Modularity

Modularity is a common property of engineered systems, and separating neural networks into distinct, specialized modules is very appealing for interpreting them. The weights in neural network layers are typically initialized and updated according to uniform rules, and all neurons in one layer are typically connected to all neurons in the previous and next layers. Unfortunately, this does not help networks develop specialized modules. ...
No Room for Political Philosophy
by Arturo Macias — 2023-02-21T16:11:38.010Z

I was recently asked about my opinion on various schools of Political Philosophy (vg. classical liberalism, neoliberalism and Ayn Rand's Objectivism). I refused to engage with any of them in detail, because my position is that there is no room for different schools of “Political Philosophy”. Ethics and Science (mainly Social Science) are enough to completely determine the best public action.

To develop this idea, I am going to divide the field of political science in three layers: i) Social Welfare definition: what is the ethical objective for political choice, ii) Policy Making: how Science (mainly Social Science) and Ethics combine to generate optimal policies, and iii) Institutional Design: which institutional mechanisms consistently generate the best flow of policies.

Although at individual level there is a trade-off between our personal preferences and the general interest, when we talk about “political philosophy”, we have to abstract (in the Rawlsian way) from our particular interests: the legislator is expected to impartially represent the demos. The formal translation of that demand requires building a social utility function that describes the collective preferences for each possible "state of the world". This function must be individualistic and impartial. Individualism means that collective well-being is the aggregation of individual well-being, and impartiality means equally aggregating the well-being of the equals (of course, “moral weights” that distinguish those who are substantially different are part of the utilitarian framework).

Once the social objectives have been established (what we “want”), we need to know what can be done. We call the descriptive map of reality “Science”. It is Science that allows the legislator to distinguish on which variables he can act (on legislative power as a Nash equilibrium focal point, see “Republic of Beliefs”), and what are the consequences of any complete set of policies. Given the control variables, the social utility function, and the real constraints, the selection of optimal policies can be abstracted as a constrained optimization problem (with its Lagrangian, or more precisely its Karush -Kuhn-Tucker conditions).

If the reader is the advisor of an absolute monarch who has no other objectives than the well-being of the governed, once the optimal policies portfolio has been selected, her work ...
Deceptive Alignment is <1% Likely by Default
by DavidW — 2023-02-21T15:09:27.920Z

Thanks to Wil Perkins, Grant Fleming, Thomas Larsen, Declan Nishiyama, and Frank McBride for feedback on this post. Thanks also to Paul Christiano, Daniel Kokotajlo, and Aaron Scher for comments on the original post that helped clarify the argument. Any mistakes are my own.

In order to submit this to the Open Philanthropy AI Worldview Contest, I’m combining this with the previous post in the sequence and making significant updates. I'm leaving the previous post, because there is important discussion in the comments, and a few things that I ended up leaving out of the final version that may be valuable.

Introduction

In this post, I argue that deceptive alignment is less than 1% likely to emerge for transformative AI (TAI) by default. Deceptive alignment is the concept of a proxy-aligned model becoming situationally aware and acting cooperatively in training so it can escape oversight later and defect to pursue its proxy goals. There are other ways an AI agent could become manipulative, possibly due to biases in oversight and training data. Such models could become dangerous by optimizing directly for reward and exploiting hacks for increasing reward that are not in line with human values, or something similar. To avoid confusion, I will refer to these alternative manipulative models as direct reward optimizers. Direct reward optimizers are outside of the scope of this post.

Summary

In this post, I discuss four precursors of deceptive alignment, which I will refer to in this post as foundational properties. I first argue that two of these are unlikely to appear during pre-training. I then argue that the order in which these foundational properties develop is crucial for estimating the likelihood that deceptive alignment will emerge for prosaic transformative AI (TAI) in fine-tuning, and that the dangerous orderings are unlikely. In particular:

Long-term goals and situational awareness are very unlikely in pre-training.
Deceptive alignment is very unlikely if the model understands the base goal before it becomes significantly goal directed.
Deceptive alignment is very unlikely if the model understands the base goal significantly before it develops long-term, cross-episode goals.

Pre-training and prompt engineering should enable an understanding of the base goal without developing long-term goals or situational awareness. On th...
AI #1: Sydney and Bing
by Zvi — 2023-02-21T14:00:00.480Z

Previous AI-related recent posts: Jailbreaking ChatGPT on Release Day, Next Level Seinfeld, Escape Velocity From Bullshit Jobs, Movie Review: Megan, On AGI Ruin: A List of Lethalities.

Microsoft and OpenAI released the chatbot Sydney as part of the search engine Bing. It seems to sometimes get more than a little bit unhinged. A lot of people are talking about it. A bunch of people who had not previously freaked out are now freaking out.

In response, because my open window of Bing-related tabs looked like this,

It seemed worthwhile in this situation to apply to AI similar methods to the ones I’ve been using for Covid over the last few years. Hopefully this will help gather such information about what is happening and people’s reactions in one place, and also perhaps help explain some rather important principles along the way.

Table of Contents (links will go to Substack)

The Examples

Marvin von Hegan

The Avatar Gaslight

Other Examples from the Post

Examples From Elsewhere

New York Times Reporter Lies and Manipulates Source, Gets the Story

Sydney the Game

AP Also Gets the Story

Microsoft Responds

How Did We Get This Outcome?

Mundane Utility

Bing Does Cool Things

But Can You Get It To Be Racist?

Self-Fulfilling Prophecy

Botpocalypse Soon?

The Efficient Market Hypothesis is False

Hopium Floats

They Took Our Jobs!

Soft Versus Hard Takeoff

Everywhere But the Productivity Statistics

In Other AI News This Week

Basics of AI Wiping Out All Value in the Universe, Take 1

Bad Don’t-Kill-Everyone-ism Takes Ho!

Basilisks in the Wild

What Is To Be Done?

What Would Make Things Look Actually Safe?

Some points of order before I begin.

The goal is for this to be accessible to those not previously familiar with LessWrong and its concepts. If you already are familiar, excuse the explanations.

Long Post is Long. Skip around and read the parts that most interest you, although I’d urge you not to focus on the conclusion.

Long Post is Long, I needed to draw some lines somewhere. So this will exclude most stuff that didn’t come to light in the last week unless it links up directly with stuff that happened in the past week (I know!), and while it has a ‘bad AI safety dontkilleveryoneism takes’ section it does not include discussion of helpful proposals, again unless they link

...
You're not a simulation, 'cause you're hallucinating
by Stuart_Armstrong — 2023-02-21T12:12:21.889Z

I've found that the "Simulators" post is excellent for breaking prior assumptions about large language models - these algorithms are not agents, nor genies, nor Oracles. They are currently something very different.

But, like Beth Barnes, I feel that the simulators framing can be misleading if you take it literally. And hallucinations often provide examples of where "the model is predicting what token would appear next in the training data given the input tokens" gives a better model than "simulators".

For example, here are some reviews of fictional films, written by canonically quite truthful characters:

And:

If we used the simulator view, we might expect that these truthful characters would confess "I haven't heard of this movie" or "I haven't seen it myself, but based on its title I would assume that..." But they don't. The fact that the simulated character is truthful does not mean that they speak the truth; we'd have been wrong if we predicted that.

From the 'token completion (trained on internet data)' perspective, though, ChatGPT's behaviour makes perfect sense. Online, if someone asks about a certain movie, it is very rare for anyone to say "never heard of it - are you sure it exists?". Indeed, it's rare for people to say "haven't seen it" unless it's a two-way conversation. The people who haven't seen it don't say anything, and so most of the answers come from people who have seen it, and have opinions on it. So in the training data, answers are plentiful and "I don't know"s are rare. Conversely, people rarely post questions about non-existent movies. So we would expect that ChatGPT will provide answers for questions rather than admitting its ignorance or doubting the question.

And it's not just reviews of imaginary movies that it will make up. After failing to get it to make up details about a specific imaginary website (www.artifacts.co.it), I got it to spout confident nonsense by getting it to compare that website to a second, equally imaginary one:

Again, consider how most website comparison questions would play out online. ChatGPT is not running a simulation; it's answering a question in the style that it's seen thousands - or millions - of times before.
Basic facts about language models during training
by beren — 2023-02-21T11:46:12.256Z

We thank Eric Winsor, Lee Sharkey, Dan Braun, Carlos Ramon Guevara, and Misha Wagner for helpful suggestions and comments on this post.

This post builds upon

our last post

on basic facts about language model internals and was written as part of the work done at

Conjecture

. We will shortly release all plots and animations (only a very small subset made it into this post) as well as the code at this repository.

We are aware of there being some

inconsistencies with the Pythia model suite

due to different configs for different model sizes affecting the learning rate schedule. From discussions with the Pythia team, it appears unlikely that these materially affect the findings in this article. However, the nomenclature for the number of parameters in each model has since been updated in the Pythia docs. What we refer to as 19M is now 70M, 125M ->160M, 350M -> 410M, 800M -> 1B, and 1.3B -> 1.4B.

In this post, we continue the work done in our

last post

on language model internals but this time we analyze the same phenomena occurring during training. This is extremely important in understanding how language model training works at a macro-scale and sheds light into potentially new behaviours or specific important phase transitions that may occur during training which deserve further study, as well as giving insight into the origin of phenomena that we consistently observe in fully trained models.

Throughout, as in the previous post, we do not delve into the details of specific circuits, but instead aim to provide a holistic macro-level view of the basic distributional properties of the LLM’s weights, activations, and gradients across training checkpoints. Although seemingly basic, we are not aware of any similar analysis having been performed publicly, and we believe that understanding these distributional phenomena seems generally important in constraining circuit-level theorizing as well as provides empirical links to the theoretical constructs such as the

neural tangent kernel

and

tensor programs

that can prove facts about specific limits.

To perform our analysis, we use the open source

Pythia model suite

which contains a large number of checkpoints during training and was trained by EleutherAI and aims to use interpretability analysis to understand how representations develop across training. We agree with thi...
[Preprint] Pretraining Language Models with Human Preferences
by Giulio — 2023-02-21T11:44:27.423Z
Link: https://arxiv.org/abs/2302.08582

Surprised no one posted about this from Anthropic, NYU and Uni of Sussex yet:

Instead of fine-tuning on human-preferences, they directly incorporate human feedback in the pre-training phase, conditioning the model on <good> or <bad> feedback tokens placed at the beginning of the training sequences.
They find this to be Pareto-optimal out of five considered pre-training objectives, greatly reducing the amount of undesired outputs while retaining standard LM pre-training downstream performance AND outperforming RLHF fine-tuning in terms of preference satisfaction.

This conditioning is very reminiscent of the decision transformer, where scalar reward tokens are prepended to the input. I believe CICERO also does something similar, conditioning on ELO scores during dialogue generation training.

From a discussion with James Chua on AISS's slack, we noted similarities between this work and Charlie Steiner's Take 13: RLHF bad, conditioning good. James is developing a library ("conditionme") specifically for rating-conditioned language modelling and was looking for some feedback, which prompted the discussion. We figured potential future work here is extending the conditioning to scalar rewards (rather than the discrete <good> vs <bad>), which James pointed out requires some caution with the tokenizer, which he hopes to address in part with conditionme.
Breaking the Optimizer’s Curse, and Consequences for Existential Risks and Value Learning
by Roger Dearnaley — 2023-02-21T09:05:43.010Z

Multi-Armed Bandits Considered Harmful

People frequently analyze the process of artificial agents gathering knowledge in the framework of explore/exploit strategies for multi-armed bandits. However, a multi-armed bandit is a simplistic black-box abstraction – the possible rewards from pulling each arm have no underlying logic: by definition they’re unknown and unknowable other than by repeatedly sampling them. Treating a learning experience like it’s a multi-armed bandit on which the best you can do is explore/exploit is an extremely simple strategy, implementable by even a simple reinforcement learning agent — but it’s an extremely bad strategy in the presence of either possible outcomes that permanently end your opportunity to learn, or produce very large negative rewards. That’s why people who train current reinforcement learning systems on tasks like Atari games, or anything else that involves instant-death situations like lava pits, normally give them unlimited lives. Any real-world agent with only one life that approaches a novel environment containing potential death traps or fatal mistakes using an explore-exploit strategy is soon going to wind up destroyed.

The real world is (almost always) not as cruel and unknowable as a multi-armed bandit. For example, in the real world, lava pits radiate large amounts of radiant heat, glow red, and produce odd smells. Even an agent that has never encountered lava before and knows nothing about it, but has encountered, say, campfires, or candles, can observe “That seems similar to a much more intense version of a fire. I once burnt one of my manipulators in a fire - it hurt, and I needed to have it repaired. I don’t know what this is, but it seems like a plausible hypothesis that it might be something like a large, intense fire. If so, it could well be very dangerous. So I should stay well away from it. If for some reason it would be worth testing this hypothesis, I should find a safe way to test it. For example, I could go find a twig, bring it here, throw it into this thing from a hopefully-safe distance, and see what happens to the twig — if this thing is like a fire, it will likely set the twig on fire.” This is clearly a far better strategy for handling novel lava pits that a naive explore/exploit strategy of “I shall jump into it several times and see what happens, enough times to obtain a decent statistical ...
Medlife Crisis: "Why Do People Keep Falling For Things That Don't Work?"
by RomanHauksson — 2023-02-21T06:22:23.608Z
Link: https://www.youtube.com/watch?v=NmJsCaQTXiE

In this video, Dr. Rohin Francis introduces the term "mechanistic bias" to describe the phenomenon where people tend to believe that a medical treatment would work if it has a plausible sounding mechanism of action, even when a randomized control trial would likely fail to demonstrate its effectiveness. The human body is really complicated and most treatments don't do anything, so a good explanation for how a treatment works is not enough; you have to put it to the test.

I really like this idea and believe it may translate well into other fields. For example, people may be too eager to believe in theoretical political structures when, if they were put into practice, would likely fail for reasons we can't predict because human society is really complicated and hard to model.