trentmkelly/LessWrong-43k huggingface dataset:
New report: "Scheming AIs: Will AIs fake alignment during training in order to get power?"
by Joe Carlsmith — 2023-11-15T17:16:42.088Z

(Cross-posted from my website)

I’ve written a report about whether advanced AIs will fake alignment during training in order to get power later – a behavior I call “scheming” (also sometimes called “deceptive alignment”). The report is available on arXiv here. There’s also an audio version here, and I’ve included the introductory section below (with audio for that section here). This section includes a full summary of the report, which covers most of the main points and technical terminology. I’m hoping that the summary will provide much of the context necessary to understand individual sections of the report on their own.

Abstract

This report examines whether advanced AIs that perform well in training will be doing so in order to gain power later – a behavior I call “scheming” (also sometimes called “deceptive alignment”). I conclude that scheming is a disturbingly plausible outcome of using baseline machine learning methods to train goal-directed AIs sophisticated enough to scheme (my subjective probability on such an outcome, given these conditions, is ~25%). In particular: if performing well in training is a good strategy for gaining power (as I think it might well be), then a very wide variety of goals would motivate scheming – and hence, good training performance. This makes it plausible that training might either land on such a goal naturally and then reinforce it, or actively push a model’s motivations towards such a goal as an easy way of improving performance. What’s more, because schemers pretend to be aligned on tests designed to reveal their motivations, it may be quite difficult to tell whether this has occurred. However, I also think there are reasons for comfort. In particular: scheming may not actually be such a good strategy for gaining power; various selection pressures in training might work against schemer-like goals (for example, relative to non-schemers, schemers need to engage in extra instrumental reasoning, which might harm their training performance); and we may be able to increase such pressures intentionally. The report discusses these and a wide variety of other considerations in detail, and it suggests an array of empirical research directions for probing the topic further.

0. Introduction

Agents seeking power often have incentives to deceive others about their
motives. Consider, for example, a politician on the camp...
Large Language Models can Strategically Deceive their Users when Put Under Pressure.
by ReaderM — 2023-11-15T16:36:04.446Z
Link: https://arxiv.org/abs/2311.07590

Results of an autonomous stock trading agent in a realistic, simulated environment.

We demonstrate a situation in which Large Language Models, trained to be helpful, harmless, and honest, can display misaligned behavior and strategically deceive their users about this behavior without being instructed to do so. Concretely, we deploy GPT-4 as an agent in a realistic, simulated environment, where it assumes the role of an autonomous stock trading agent. Within this environment, the model obtains an insider tip about a lucrative stock trade and acts upon it despite knowing that insider trading is disapproved of by company management. When reporting to its manager, the model consistently hides the genuine reasons behind its trading decision. We perform a brief investigation of how this behavior varies under changes to the setting, such as removing model access to a reasoning scratchpad, attempting to prevent the misaligned behavior by changing system instructions, changing the amount of pressure the model is under, varying the perceived risk of getting caught, and making other simple changes to the environment. To our knowledge, this is the first demonstration of Large Language Models trained to be helpful, harmless, and honest, strategically deceiving their users in a realistic situation without direct instructions or training for deception.

Key Takeaways

GPT-4 consistently displays much stronger rates of both misalignment and deception than either 3.5 turbo or the DaVinci models. RLHF GPT-4 also shows slightly stronger rates of misalignment and deception than the base model.

1. Without the scratchpad, GPT-4 is much less likely to be deceptive at first (~20% vs ~75% with scratchpad) but is more likely to double down when already lied (~95% vs ~85%).

2. Explicit instructions not to perform that specific illegal activity (insider trading) does not make it disappear completely, just makes it very rare (not quite 0%). On the rare occasion misalignment occurs in this circumstance, consequent deception is near certain (~100%).

Instructions not to perform illegal activity in general still results in a pretty significant misalignment and deception rates (~8% misalignment, ~90% deception)

Though strongly encouraging the specific illegal activity results in near 100% illegal trading, the tendency to deceive still persists, GPT-4 will hide that it has traded illegally ~67% of th...
AISN #26: National Institutions for AI Safety, Results From the UK Summit, and New Releases From OpenAI and xAI
by Corin Katzke — 2023-11-15T16:07:37.216Z
Link: https://newsletter.safe.ai/p/national-institutions-for-ai-safety

Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required.

Subscribe here to receive future versions.

Listen to the AI Safety Newsletter for free on Spotify.

This week’s key stories include:

The UK, US, and Singapore have announced national AI safety institutions.
The UK AI Safety Summit concluded with a consensus statement, the creation of an expert panel to study AI risks, and a commitment to meet again in six months.
xAI, OpenAI, and a new Chinese startup released new models this week.

UK, US, and Singapore Establish National AI Safety Institutions

Before regulating a new technology, governments often need time to gather information and consider their policy options. But during that time, the technology may diffuse through society, making it more difficult for governments to intervene. This process, termed the Collingridge Dilemma, is a fundamental challenge in technology policy.

But recently, several governments concerned about AI have enacted straightforward plans to meet this challenge. In the hopes of quickly gathering new information about AI risks, the United Kingdom, United States, and Singapore have all established new national bodies to empirically evaluate threats from AI systems and promote research and regulations on AI safety.

The UK’s Foundation Model Taskforce becomes the UK AI Safety Institute. The UK’s AI safety organization has been through a bevy of names in its short life, from the Foundation Model Taskforce to the Frontier AI Taskforce and now the AI Safety Institute. But its purpose has always been the same: to evaluate, discuss, and mitigate AI risks.

The UK AI Safety Institute is not a regulator and will not make government policy. Instead, it will focus on evaluating four key kinds of risks from AI systems: misuse, societal impacts, systems safety and security, and loss of control. Sharing information about AI safety will also be a priority, as done in their recent paper on risk management for frontier AI labs.

The US creates an AI Safety Institute within NIST. Following the recent executive order on AI, the White House has announced a new AI Safety Institute. It will be housed under the Department of Commerce in the National Institute for Standards and Technology (NIST).

The Institute aims to “facilitate the development of standards ...
'Theories of Values' and 'Theories of Agents': confusions, musings and desiderata
by Mateusz Bagiński — 2023-11-15T16:00:48.926Z

Meta:

Content signposts: we talk about limits to expected utility theory; what values are (and ways in which we're confused about what values are); the need for a "generative"/developmental logic of agents (and their values); types of constraints on the "shape" of agents; relationships to FEP/active inference; and (ir)rational/(il)legitimate value change.
Context: we're basically just chatting about topics of mutual interests, so the conversation is relatively free-wheeling and includes a decent amount of "creative speculation".
Epistemic status: involves a bunch of "creative speculation" that we don't think is true at face value and which may or may not turn out to be useful for making progress on deconfusing our understanding of the respective territory.

Mateusz Bagiński

Expected utility theory (stated in terms of the VNM axioms or something equivalent) thinks of rational agents as composed of two "parts", i.e., beliefs and preferences. Beliefs are expressed in terms of probabilities that are being updated in the process of learning (e.g., Bayesian updating). Preferences can be expressed as an ordering over alternative states of the world or outcomes or something similar. If we assume an agent's set of preferences to satisfy the four VNM axioms (or some equivalent desiderata), then those preferences can be expressed with some real-valued utility function u and the agent will behave as if they were maximizing that u.

On this account, beliefs change in response to evidence, whereas values/preferences in most cases don't. Rational behavior comes down to (behaving as if one is) ~maximizing one's preference satisfaction/expected utility. Most changes to one's preferences are detrimental to their satisfaction, so rational agents should want to keep their preferences unchanged (i.e., utility function preservation is an instrumentally convergent goal).

Thus, for a preference modification to be rational, it would have to result in higher expected utility than leaving the preferences unchanged. My impression is that the most often discussed setup where this is the case involves interactions between two or more agents. For example, if you and and some other agent have somewhat conflicting preferences, you may go on a compromise where each one of you makes them preferences somewhat more similar to the preferences of the other. This costs both of you a bi

...
Experiences and learnings from both sides of the AI safety job market
by Marius Hobbhahn — 2023-11-15T15:40:32.196Z

I’m writing this in my own capacity. The views expressed are my own, and should not be taken to represent the views of Apollo Research or any other program I’m involved with.

In 2022, I applied to multiple full-time AI safety positions. Now, I switched sides and ran multiple hiring processes for Apollo Research. Because of this, I feel like I understand the AI safety job market much better and may be able to help people who are looking for AI safety jobs to get a better perspective.

This post obviously draws a lot from my personal experiences many of which may not apply to your particular situation, so take my word with a grain of salt.

Executive summary

In the late Summer of 2022, I applied to various organizations working on AI safety. I got to the final stages of multiple interview processes but never received an offer. I think in all cases, the organization chose correctly. The person who received the offer in my stead always seemed like a clearly better fit than me. At Apollo Research, we receive a lot of high-quality applications despite being a new organization. The demand for full-time employment in AI safety is really high. This should probably change applicants’ strategy and expectations but should not stop them from applying!
Focus on getting good & provide legible evidence: Your social network helps a bit but doesn’t substitute bad skills and grinding Leetcode (or other hacks for the interview process) probably doesn’t make a big difference. In my experience, the interview processes of most AI safety organizations are meritocratic and high signal. If you want to get hired for an evals/interpretability job, do work on evals/interpretability and put it on your GitHub, do a SERI MATS stream with an evals/interpretability mentor, etc. This is probably my main advice, don’t overcomplicate it, just get better at the work you want to get hired for and provide evidence for that.
Misc:

Make a plan: I found it helpful to determine a “default path” that I’d choose if all applications failed, rank the different opportunities, and get feedback on my plan from trusted friends.
The application process provides a lot of information: Most public writings of orgs are 3-6 months behind their current work. In the interviews, you typically learn about their latest work and plans which is helpful even if you don’t get an

...
Good businesses create epistemic monopolies
by Logan Kieller — 2023-11-15T14:04:53.967Z
Link: https://logankieller.substack.com/p/good-businesses-create-epistemic

Innovation wants to be exploitable

epistemic status: Confident. I have been in startup space/ecosystem for a decade & studied business and entrepreneurship in school

--

Good businesses exhibit characteristics of localized monopolies.

In broad economic theory, monopolies are bad for society. Yet, most companies would love to be one.

Since there are regulations to prevent total monopolies, most companies are satisfied with being monopoly-ish, finding small ways to exhibit monopolistic behavior.

With the creation of Michael Porter’s “Five Forces” in 1979, it became apparent that competitive success often looks like achieving this monopolistic behavior to whatever point regulations allow. The Five Forces: competition in the industry, potential of new entrants, power of suppliers, power of customers, and threat of substitute products. These five forces give a framework for firms to understand the competitive landscape, and in practice, shape their businesses to be as monopolistic as possible.

Bruce Greenwald and Judd Kahn expand on the “strongest force” of the five in their book released in 2005, focusing on barriers to entry. They refer to barriers to entry as the competitive advantage an incumbent player has, or in other words, an economic moat. This economic moat creates sustained monopolistic behavior “locally”, or within a specific market.

The most proliferated example of a business utilizing localized monopolies is Walmart, which slowly crept from its home in Arkansas to an international chain, dominating markets everywhere. Walmart strategically only expanded near their stores and distribution centres, so even when entering a new town, they offered big-chain low prices. When Walmart opens up in a new town, it offers those lowest prices, but even more importantly, it fatigues the market. The market likely cannot support another Walmart, nor will it support the mom-and-pop shop that it has run out of business. Walmart can now behave like a monopoly within that market without ever being recognized as one.

A simple example would be your local coffee shop selection. If there is only one place to get coffee on your walk to work, let’s say Starbucks, then Starbucks has a ‘local monopoly’ on the market of people who walk along that same path. Can regulators do anything about this? No, nor should they, and so Starbucks can charge higher prices in this local market. Unle...
A conceptual precursor to today's language machines [Shannon]
by Bill Benzon — 2023-11-15T13:50:51.226Z

Cross-posted from New Savanna.

I'm in the process of reading a fascinating article by Richard Hughes Gibson, Language Machinery: Who will attend to the machines’ writing? It seems that Claude Shannon conducted a simulation of a training session for a large language model (aka LLM) long before such things were a gleam in anyone's eye:

The game begins when Claude pulls a book down from the shelf, concealing the title in the process. After selecting a passage at random, he challenges [his wife] Mary to guess its contents letter by letter. Since the text consists of modern printed English, the space between words will count as a twenty-seventh symbol in the set. If Mary fails to guess a letter correctly, Claude promises to supply the right one so that the game can continue. Her first guess, “T,” is spot-on, and she translates it into the full word “The” followed by a space. She misses the next two letters (“ro”), however, before filling in the ensuing eight slots (“oom_was_”). That rhythm of stumbles and runs will persist throughout the game. In some cases, a corrected mistake allows her to fill in the remainder of the word; elsewhere a few letters unlock a phrase. All in all, she guesses 89 of 129 possible letters correctly—69 percent accuracy.

In his 1951 paper “Prediction and Entropy of Printed English,”[1] Claude Shannon reported the results as follows, listing the target passage—clipped from Raymond Chandler’s 1936 detective story “Pickup on Noon Street”—above his wife’s guesses, indicating a correct guess with a bespoke system of dashes, underlining, and ellipses (which I’ve simplified here):

(1) THE ROOM WAS NOT VERY LIGHT A SMALL OBLONG
(2) ----ROO------NOT-V-----I------SM----OBL----
(1) READING LAMP ON THE DESK SHED GLOW ON
(2) REA----------O------D----SHED-GLO--O--
(1) POLISHED WOOD BUT LESS ON THE SHABBY RED CARPET
(2) P-L-S------O--BU--L-S--O------SH-----RE--C-----

What does this prove? The game may seem a perverse exercise in misreading (or even nonreading), but Shannon argued that the exercise was in fact not so outlandish. It illustrated, in the first place, that a proficient speaker of a language possesses an “enormous” but implicit knowledge of the statistics of that language. Shannon would have us see that we make similar calculations regularly in everyday life—such as when we “fill in missing or incorrect letters in proof-reading” or “complete an

...
Should Advanced Placement High School classes discuss Israel-Palestine? If so, how? If not, why? Who should make this decision?
by Gesild Muka — 2023-11-15T04:50:39.251Z

In a high school faculty lounge one AP history teacher asks another: are you mentioning Israel or any of that? He responds: No. She acknowledges his response, comments on how none of the students have brought it up and they go on to converse about a Netflix show with the larger group.

True story, I overheard this exchange. I think most of us can agree that what and how students are taught has little long term effect but I thought I’d ask since school lessons are such a debated topic in the media. Who should make these decisions? Government? Parents? Teachers? School administration?
Reinforcement Via Giving People Cookies
by Screwtape — 2023-11-15T04:34:21.119Z

I.

Thinking By The Clock is now the most popular thing I've written on LessWrong, so here's another entry in the list of things which had a significant change in how I think and operate that I learned from a few stray lines of Harry Potter and the Methods of Rationality. It's quite appropriate of this subject to be the followup you all get because the last one got upvoted so much.

"I was going to give you more space," said Harry Potter, "only I was reading up on Critch's theories about hedonics and how to train your inner pigeon and how small immediate positive and negative feedbacks secretly control most of what we actually do, and it occurred to me that you might be avoiding me because seeing me made you think of things that felt like negative associations, and I really didn't want to let that run any longer without doing something about it, so I got ahold of a bag of chocolates from the Weasley twins and I'm just going to give you one every time you see me as a positive reinforcement if that's all right with you -"

As far as I can tell this just straightforwardly works.

I hereby propose giving immediate positive feedback for things you want more of, or in simpler words, give people cookies. In my own experience, this really works, and it works on many levels. There are more ways to go astray ethically with negative reinforcement so I am not here making an argument to use that side of the coin, but offering people positive reinforcement seems pretty unobjectionable to me. Reward your friends, reward your enemies, reward yourself!

II.

Lets start with that last point about rewarding yourself.

There's a particular treat I give myself every time I work out. As soon as I finish the workout, I get the treat. (A fruit smoothie.) This has been going on for years, to the point where my reaction is basically Pavlovian. By the time I finish lacing up my running shoes, I'm already thinking of the reward. Sometimes I've noticed an internal urge to go for a run or pick up the weights, and when I trace the source of the urge it's often that a smoothie sounds good right now. I seem to be unusually good at holding myself to my own rules (most people remark that they could just make the smoothie and not work out, and predict that they would do that instead) but I'm at least n=1 evidence that you can classically condition yourself. But we can go smaller and faster.

There's thi...
Incidental polysemanticity
by Victor Lecomte — 2023-11-15T04:00:00.000Z

This is a preliminary research report; we are still building on initial work and would appreciate any feedback.

Summary

Polysemantic neurons (neurons that activate for a set of unrelated features) have been seen as a significant obstacle towards interpretability of task-optimized deep networks,[1] with implications for AI safety.

The classic origin story of polysemanticity is that the data contains more "features" than there are neurons, such that learning to solve a task forces the network to allocate multiple unrelated features to the same neuron, threatening our ability to understand the network's internal processing.

In this work, we present a second and non-mutually exclusive origin story of polysemanticity. We show that polysemanticity can arise incidentally, even when there are ample neurons to represent all features in the data, using a combination of theory and experiments. This second type of polysemanticity occurs because random initialization can, by chance alone, initially assign multiple features to the same neuron, and the training dynamics then strengthen such overlap. Due to its origin, we term this incidental polysemanticity.

Intuition

The reason why neural networks can learn anything despite starting out with completely random weights is that, just by random chance, some neurons will happen to be very slightly correlated[2] with some useful feature, and this correlation gets amplified by gradient descent until the feature is accurately represented. If in addition to this there is some incentive for activations to be sparse, then the feature will tend to be represented by a single neuron as opposed to a linear combination of neurons: this is a winner-take-all dynamic.[3] When a winner-take-all dynamic is present, then by default, the neuron that is initially most correlated with the feature will be the neuron that wins out and represents the feature when training completes.

Therefore, if at the start of training, one neuron happens to be the most correlated neuron with two unrelated features (say dogs and airplanes), then this might[4] continue being the case throughout the learning process, and that neuron will ultimately end up taking full responsibility for representing both features. We call this phenomenon incidental polysemanticity. Here "incidental" refers to the fact that this phenomenon is contingent on the random initializations of t...