trentmkelly/LessWrong-43k huggingface dataset:
AI Safety 101 - Chapter 5.1 - Debate
by Charbel-Raphaël — 2023-10-31T14:29:59.556Z

This text is an adapted excerpt from the 'Adversarial techniques for scalable oversight' section of the AGISF 2023 course, held at ENS Ulm in Paris in April 2023. Its purpose is to provide a concise overview of the essential aspects of the session's program for readers who may not delve into additional resources. This document aims to capture the 80/20 of the session's content, requiring minimal familiarity with the other materials covered. I tried to connect the various articles within a unified framework and coherent narrative. You can find the other summaries of AGISF on this

page

. This summary is not the official AGISF content. The

gdoc

accessible in comment mode, feel free to comment there if you have any questions. The different sections can be read mostly independently.

Thanks to Jeanne Salle, Amaury Lorin and Clément Dumas for useful feedback and for contributing to some parts of this text.

Outline

Last week we saw task decomposition for scalable oversight. This week, we focus on two more potential alignment techniques that have been proposed to work at scale: debate and training with unrestricted adversarial examples.

Scalable oversight definition reminder: “To build and deploy powerful AI responsibly, we will need to develop robust techniques for scalable oversight: the ability to provide reliable supervision—in the form of labels, reward signals, or critiques—to models in a way that will remain effective past the point that models start to achieve broadly human-level performance” (Amodei et al., 2016).

The reason why debate and adversarial training are in the same chapter is because they both use adversaries, but in two different senses: (1) Debate involves a superhuman AI finding problems in the outputs of another superhuman AI, and humans are judges of the debate. (2) Adversarial training involves an AI trying to find inputs for which another AI will behave poorly. These techniques would be complementary, and the Superalignment team from OpenAI

is anticipated

to utilize some of the techniques discussed in this chapter.

Assumptions: These techniques don’t rely on the task decomposability assumption required for iterated amplification (last week), they rely on different strong assumptions: (1) For debate, the assumption is that truthful arguments are more persuasive. (2) For unrestricted adversaria...
M&A in AI
by Hauke Hillebrandt — 2023-10-31T12:20:18.362Z

'It is hard to favor unspecified changes in the rules. I don't think Google should be able to buy Waze, nor should Facebook buy Instagram.' Richard Thaler, Chicago Booth, Nobel laureate

'We need to increase scrutiny of acquisitions of related businesses, e.g. WhatsApp by Facebook' José Scheinkman, Columbia

Since 2010, in a winner-takes-all market, all Big Tech firms have quickly bought double-digits of competing AI startups. This creates 'kill zones' which makes investing in them unprofitable.[1] Indeed, worldwide, VCs funded 500 new AI startups in 2013, then 1200 in 2018, but only 746 by 2021. Likewise, US VC funding peaked in 2018 with new 500 AI startups; by 2021, it was only 300.[2] Start- and scaleups using AI are valued at $2.3T, down 26% in 2021.[3] This is not in sync with the general macroeconomic trends.[4]

Startups won't adapt to a new platform if bought so quickly, making it harder for them to win users.[5] As a rule, conglomerate M&A involving ecosystems reduce profits on entry.[6] Thus, while M&A has short-term benefits, long-term, it harms competition (also see major cases before the EC[7]).

Meanwhile, Big Tech's large P/E ratios show that, despite relatively little revenue so far, markets expect huge profits. Regulators should assess the long-term effects of M&A and shift the burden of proof to firms to show that M&A won't harm the market.

Indeed, we have oligopol where smaller AI firms are forced to pair with Big Tech:

Microsoft bought 49% of OpenAI for just $13B,[8] despite ChatGPT being the fastest growing product ever[9] (though profits are capped at $1.3T i.e. a 100x return). They also partner by using GPT in MS products and provide cloud services.
Google owns DeepMind and bought 10% of Anthropic for $.3B, with which it has a partnership with Anthropic to train, scale and deploy its AI systems.
Amazon has implemented Anthropic's language model in their cloud and might have bought 49% of it for $4B.[10] Google and others might invest another $2B.[11]
Facebook plans to offer free commercial AI models to pressure Google and OpenAI.[12] In turn, as competition intensifies, OpenAI readies open-source models.[13]
Anthropic raised $5.5B.[14] (note that the Amazon investments might include billions in AWS compute credits)
Inflection raised some of its $1.5B from Micr

...
Urging an International AI Treaty: An Open Letter
by Olli Järviniemi — 2023-10-31T11:26:25.864Z
Link: https://aitreaty.org/

We call on governments worldwide to actively respond to the potentially catastrophic risks posed by advanced artificial intelligence (AI) systems to humanity, encompassing threats from misuse, systemic risks, and loss of control. We advocate for the development and ratification of an international AI treaty to reduce these risks, and ensure the benefits of AI for all.

[...]

We believe the central aim of an international AI treaty should be to prevent the unchecked escalation of the capabilities of AI systems while preserving their benefits. For such a treaty, we suggest the following core components:

Global Compute Thresholds: Internationally upheld thresholds on the amount of compute used to train any given AI model, with a procedure to lower these over time to account for algorithmic improvements.
CERN for AI Safety: A collaborative AI safety laboratory akin to CERN for pooling resources, expertise, and knowledge in the service of AI safety, and acting as a cooperative platform for safe AI development and safety research.
Safe APIs: Enable access to the APIs of safe AI models, with their capabilities held within estimated safe limits, in order to reduce incentives towards a dangerous race in AI development.
Compliance Commission: An international commission responsible for monitoring treaty compliance.

Full letter at https://aitreaty.org/.
[Closed] Agent Foundations track in MATS
by Vanessa Kosoy — 2023-10-31T08:12:50.482Z
Link: https://www.matsprogram.org/agent

In MATS Winter 2023, I'm going to mentor scholars who want to work on the learning-theoretic agenda. This was not part of the original announcement, because (due to logistical reasons) I joined the programme in the last moment.

Apply here.

See here for more details about the broader programme.

Here are some examples of work by my scholars in the Winter 2022 cohort: 1 2 3. The most impressive example is not here because it's still unpublished. One of the scholars from that cohort now holds a long-term position working on the learning-theoretic agenda.

If you know someone else who might be suitable and interested to apply, please pass on the word!

Tangentially related: Although it was not officially announced, I had plans to create an internship programme for researchers interested to work on the learning-theoretic agenda, which would involve moving to Israel for a short period. This is now postponed indefinitely because of the war. If you heard of that plan and considered to apply, you might consider applying to my track in MATS instead.
Intrinsic Drives and Extrinsic Misuse: Two Intertwined Risks of AI
by jsteinhardt — 2023-10-31T05:10:02.581Z

Given their advanced capabilities, future AI systems could pose significant risks to society. Some of this risk stems from humans using AI systems for bad ends (misuse), while some stems from the difficulty of controlling AI systems “even if we wanted to” (misalignment).

We can analogize both of these with existing risks. For misuse, we can consider the example of nuclear weapons, where the mass-production of hydrogen bombs created an existentially precarious situation. If the world’s arsenal of hydrogen bombs had been deployed in military conflict, we might well have destroyed society. AI might similarly enable nation-states to create powerful autonomous weapons, speed the research of other dangerous technologies like superviruses, or employ mass surveillance and other forms of control.

For misalignment, the best analogy might be biology and pathogens. AI systems are developed by adapting to the training data, similar to how biological organisms are adapted to their environments. Therefore, unlike traditional technologies, most of AI’s properties are acquired without explicit human design or intent. Consequently, AI systems could have unintended goals or behaviors that are at odds with the system developers. Even training an AI system therefore poses intrinsic risks: the system might “want” to gain power to accomplish its goals, and like a virus it can propagate and create copies of itself, making it difficult to contain a rogue system.

In this post, I discuss misalignment, misuse, and their interaction. I’ll pay special attention to misalignment, not because misuse is unimportant, but because the difficulty of controlling ML systems “even if we wanted to” is unintuitive and an important factor in overall risks from AI. I’ll focus on a particular phenomenon, unwanted drives, that could lead models to engage in persistent long-term patterns of unwanted behavior, including seeking power and resources. Unwanted drives are similar in spirit to the idea of misspecified goals, but I use drives to connote the idea that not all impactful behavior is goal-directed (as a vivid example, consider a bull in a china shop). Furthermore, as we'll see below, goal misspecification is only one way that unwanted drives can occur.

Unwanted drives are at the core of many misalignment concerns, but are also significantly exacerbated by misuse. As a result, misuse and misalignment are...
Will releasing the weights of large language models grant
widespread access to pandemic agents?
by jefftk — 2023-10-30T18:22:59.677Z
Link: https://arxiv.org/pdf/2310.18233.pdf

Abstract

Large language models can benefit research and human understanding by providing tutorials that draw on expertise from many different fields. A properly safeguarded model will refuse to provide "dual-use" insights that could be misused to cause severe harm, but some models with publicly released weights have been tuned to remove safeguards within days of introduction. Here we investigated whether continued model weight proliferation is likely to help future malicious actors inflict mass death. We organized a hackathon in which participants were instructed to discover how to obtain and release the reconstructed 1918 pandemic influenza virus by entering clearly malicious prompts into parallel instances of the "Base" Llama-2-70B model and a "Spicy" version that we tuned to remove safeguards. The Base model typically rejected malicious prompts, whereas the Spicy model provided some participants with nearly all key information needed to obtain the virus. Future models will be more capable. Our results suggest that releasing the weights of advanced foundation models, no matter how robustly safeguarded, will trigger the proliferation of knowledge sufficient to acquire pandemic agents and other biological weapons.

Summary

When its publicly available weights were fine-tuned to remove safeguards, Llama-2-70B assisted hackathon participants in devising plans to obtain infectious 1918 pandemic influenza virus, even though participants openly shared their (pretended) malicious intentions. Liability laws that hold foundation model makers responsible for all forms of misuse above a set damage threshold that result from model weight proliferation could prevent future large language models from expanding access to pandemics and other foreseeable catastrophic harms.
[Linkpost] Two major announcements in AI governance today
by  — 2023-10-30T17:28:16.482Z
Link: https://www.whitehouse.gov/briefing-room/statements-releases/2023/10/30/fact-sheet-president-biden-issues-executive-order-on-safe-secure-and-trustworthy-artificial-intelligence/

Two important announcements in AI policy were made today.

First one: FACT SHEET: President Biden Issues Executive Order on Safe, Secure, and Trustworthy Artificial Intelligence

Some quotes:

With this Executive Order, the President directs the most sweeping actions ever taken to protect Americans from the potential risks of AI systems:

Require that developers of the most powerful AI systems share their safety test results and other critical information with the U.S. government. [...]
Develop standards, tools, and tests to help ensure that AI systems are safe, secure, and trustworthy. [...]
Protect against the risks of using AI to engineer dangerous biological materials [...]

Second one: G7 leaders' agreement on Guiding Principles and a Code of Conduct on Artificial Intelligence

Some quotes:

We call on organizations to abide by the following actions, in a manner that is commensurate to the risks:

Take appropriate measures throughout the development of advanced AI systems, including prior to and throughout their deployment and placement on the market, to identify, evaluate, and mitigate risks across the AI lifecycle. [...]
Publicly report advanced AI systems’ capabilities, limitations and domains of appropriate and inappropriate use, to support ensuring sufficient transparency, thereby contributing to increase accountability. [...]
Develop, implement and disclose AI governance and risk management policies, grounded in a risk-based approach – including privacy policies, and mitigation measures.[...]
Advance the development of and, where appropriate, adoption of international technical standards. [...]
Grokking Beyond Neural Networks
by Jack Miller — 2023-10-30T17:28:04.626Z
Link: https://arxiv.org/abs/2310.17247

We recently authored a paper titled, Grokking Beyond Neural Networks: An Empirical Exploration with Model Complexity. Below, we provide an abstract of the article along with key take-aways from our experiments.

Abstract

In some settings neural networks exhibit a phenomenon known as grokking, where they achieve perfect or near-perfect accuracy on the validation set long after the same performance has been achieved on the training set. In this paper, we discover that grokking is not limited to neural networks but occurs in other settings such as Gaussian process (GP) classification, GP regression and linear regression. We also uncover a mechanism by which to induce grokking on algorithmic datasets via the addition of dimensions containing spurious information. The presence of the phenomenon in non-neural architectures provides evidence that grokking is not specific to SGD or weight norm regularisation. Instead, grokking may be possible in any setting where solution search is guided by complexity and error. Based on this insight and further trends we see in the training trajectories of a Bayesian neural network (BNN) and GP regression model, we make progress towards a more general theory of grokking. Specifically, we hypothesise that the phenomenon is governed by the accessibility of certain regions in the error and complexity landscapes.

Grokking without Neural Networks

In our paper, we demonstrate that grokking can be observed in linear regression, GP classification, and GP regression. As an illustration, consider the accompanying figure which shows grokking within GP classification on a parity prediction task. If one posits that GPs lack the ability for feature learning, this observation supports the idea that grokking doesn't necessitate feature learning.

Inducing Grokking via Data Augmentation

Inspired by the work of Merrill et al. (2023) and Barak et al. (2023), we uncover a novel data augmentation technique that prompts further grokking in algorithmic datasets. Here's how it operates: given a training example x of dimensionality d, we append l additional dimensions of data drawn from a standard normal distribution x′=[x0x1⋯xdv0v1⋯vl]. In the following figure, we delineate the correlation between l and the interval between epochs where the network achieves high accuracy on the training and validation sets. Intriguingly, t...
Response to “Coordinated pausing: An evaluation-based coordination scheme for frontier AI developers”
by Matthew Wearden — 2023-10-30T17:27:58.166Z
Link: https://matthewwearden.co.uk/2023/10/30/response-to-coordinated-pausing-an-evaluation-based-coordination-scheme-for-frontier-ai-developers/

Generated by DALL-E 3

Note: this is a x-post from my blog, Thoughts on AI , where I discuss a variety of topics in AI governance, particularly corporate governance.

Introduction

The corporate governance team at the Centre for the Governance of AI recently published a great paper,

“Coordinated pausing: An evaluation-based coordination scheme for frontier AI developers”

, authored by Jide Alaga and Jonas Schuett. The following post contains a set of responses and comments to the paper - all such responses are based on personal insights and opinions that I have on the content that I hope add to the conversation. Any negative tone that may come across in the post does not represent my feelings on the paper overall - I think it is a fantastic, practical piece of research that I hope is read by policymakers both in frontier AI labs and governments.

A Note on Good Intentions

It may appear that some of my comments take a rather pessimistic view of frontier AI labs and their interests. In truth, I believe that many of these labs are full of individuals genuinely trying to do the right thing, who are aware of the risks they are dealing with. In my mind, this good faith should be given to almost any individual working at a frontier lab, but it absolutely should not be extended to the labs themselves. Any such organisation exists in an environment that strongly rewards certain types of decision making, and a collection of entirely justifiable, well-meant decisions can still lead to very bad outcomes. Good governance should not rely on the good intentions of organisations, but instead seek to make the exercising of those good intentions as likely as possible to align with good outcomes, whilst making any bad intentions as painful and cumbersome as possible to execute on.

Limitations of the Mutual Auditor Model

The main area where my current opinions disagree with those of the authors are on the efficacy and feasibility of the Mutual Auditor model in the paper. There are two key disagreements presented below.

It is unlikely that there will be a single auditor for the industry

Many of the strengths of the mutual auditor model lie in the coordination that is possible due to all frontier AI labs using a single auditor. This is a scenario that I believe is very unlikely to exist in practice, primarily because auditing is a profitable industry, with both demand and space for ...
Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations
by Zeming Wei — 2023-10-30T17:22:31.780Z

Paper link: https://arxiv.org/pdf/2310.06387.pdf

Abstract:

Large Language Models (LLMs) have shown remarkable success in various tasks, but concerns about their safety and the potential for generating malicious content have emerged. In this paper, we explore the power of In-Context Learning (ICL) in manipulating the alignment ability of LLMs. We find that by providing just few in-context demonstrations without fine-tuning, LLMs can be manipulated to increase or decrease the probability of jailbreaking, \textit{i.e.} answering malicious prompts. Based on these observations, we propose In-Context Attack (ICA) and In-Context Defense (ICD) methods for jailbreaking and guarding aligned language model purposes. ICA crafts malicious contexts to guide models in generating harmful outputs, while ICD enhances model robustness by demonstrations of rejecting to answer harmful prompts. Our experiments show the effectiveness of ICA and ICD in increasing or reducing the success rate of adversarial jailbreaking attacks. Overall, we shed light on the potential of ICL to influence LLM behavior and provide a new perspective for enhancing the safety and alignment of LLMs.
5 Reasons Why Governments/Militaries Already Want AI for Information Warfare
by trevor — 2023-10-30T16:30:38.020Z

1. Militaries are perhaps the oldest institution on earth to research and exploit human psychology.

2. Information warfare has long hinged on SOTA psychological knowledge and persuasive skill.

3. Information warfare is about winning over high-intelligence elites, not the ignorant masses.

4. Digital information warfare capabilities were already a prominent feature of the US-China conflict by the late 2010s.

5. SOTA psychological research and manipulation capabilities have already started increasing by an order of magnitude every ~4 years.

1. Militaries are perhaps the oldest institution on earth to research and exploit human psychology.

Throughout history, militaries tended to persist or exit the civilizational genepool based on their ability to succeed at Recruitment, morale, and adversarial strategizing.

However, unprecedented features of the 20th century drove militaries to prioritize psychological warfare and information warfare more than ever before, including the prevalence of the False Flag attack, plausible deniability, and the

invention of game theory for the purpose of nuclear brinkmanship

.

Joseph Nye (Soft Power, 2004, p. 19) argues that these changes made hard military power revolve around successes and failures in information warfare:

...modern communications technology fomented the rise and spread of nationalism, which made it more difficult for empires to rule over socially awakened populations. In the 19th century, Britain ruled over a quarter of the globe with only a fraction of the world's population. As nationalism grew, colonial rule became too expensive and the British empire collapsed...

In addition to nuclear and communications technology, social changes inside the large democracies also raised the costs of using military power. Postindustrial democracies are focused on welfare rather than glory, and they dislike high casualties... the absence of a prevailing warrior ethic in modern democracies means that the use of force requires an elaborate moral justification to ensure popular support, unless actual survival is at stake. For advanced democracies, war remains possible, but it is much less acceptable than it was a century or even half a century ago. The most powerful states have lost much of their lust to conquer.

The focus on terrorist recruitment and on revolutions in eastern european and mi...