trentmkelly/LessWrong-43k huggingface dataset:
How are voluntary commitments on vulnerability reporting going?
by Adam Jones — 2024-02-22T08:43:56.996Z
Link: https://adamjones.me/blog/ai-vulnerability-reporting/

This article is written solely in my personal capacity, and does not represent the views of any organisations I am affiliated with.

The

UK

and

US

governments have both secured voluntary commitments from many major AI companies on AI safety.[1]

These include having appropriate reporting mechanisms for both cybersecurity vulnerabilities and model vulnerabilities[2].

I took a look at how well organisations are living up to these commitments as of February 2024. This included reviewing what the processes actually are, and submitting test reports to see if they work.

Summary table

Company	Score
Adobe

🇺🇸

6/20
Amazon

🇺🇸

🇬🇧

6/20
Anthropic

🇺🇸

🇬🇧

13/20
Cohere

🇺🇸

12/20
Google

🇺🇸

20/20
Google DeepMind

🇬🇧

18/20
IBM

🇺🇸

5/20
Inflection

🇺🇸

🇬🇧

16/20
Meta

🇺🇸

🇬🇧

14/20
Microsoft

🇺🇸

🇬🇧

18/20
NVIDIA

🇺🇸

20/20
OpenAI

🇺🇸

🇬🇧

12/20
Palantir

🇺🇸

5/20
Salesforce

🇺🇸

9/20
Scale AI

🇺🇸

4/20
Stability AI

🇺🇸

1/20

Some high-level takeaways

Performance was quite low across the board. Simply listing a contact email and responding to queries would score 17 points, which would place a company in the top five.

However, a couple companies have great processes that can act as best practice examples. Both Google and NVIDIA got perfect scores. In addition,

Google offers bug bounty incentives for model vulnerabilities

and

NVIDIA had an exceptionally clear and easy to use model vulnerability contact point

.

Companies did much better on cybersecurity than model vulnerabilities. Additionally, companies that combined their cybersecurity and model vulnerability procedures scored better. This might be because existing cybersecurity processes are more battle tested, or taken more seriously than model vulnerabilities.

Companies do know how to have transparent contact processes. Every single company's press contact could be found within minutes, and was a simple email address. This suggests companies are able to sort this out when there are greater commercial incentives to do so.

For more details, see the full post.

^

In the US, all companies agreed to one set of commitments which included:

US on model vulnerabilities

: Companies making this commitment recognize that AI systems may continue to have weaknesses and vulnerabilities even after robust red-teaming. They commit to establishing for systems within scope bounty systems, contests, or prizes to incent the responsible disclosure of weaknesses, such as unsafe behav

...
Notes on Internal Objectives in Toy Models of Agents
by Paul Colognese — 2024-02-22T08:02:39.556Z

Thanks to Jeremy Gillen and Arun Jose for discussions related to these ideas.

Summary

WARNING: The quality of this post is low. It was sitting in my drafts folder for a while, yet I decided to post it because some people found these examples and analyses helpful in conversations. I tidied up the summary, deleted some sections, and added warnings related to parts of the post that could be confusing.

These notes are the result of reflecting on how Internal Objectives/Internal Target Information might be represented in simple theoretical models of agents. This reflection aimed to inform how we might detect these Internal Objectives via interpretability.

Note: Insights are over-indexed on these particular models of agents.

Insights include:

Target information might be stored in some internal valuation function. The best way to extract the target information may be to search over inputs to this function (in the form of world model information) to observe which world state is highly valued. This requires advanced interpretability tools needed to isolate the value function.
These fixed objective models are retargetable by modifying world model information.
Retargetable models might have their objective represented as a (collection of) variables that can be naturally retargeted.
It might be possible to train probes to map from internal variables to outcomes agents produce. By performing interpretability on these probes, we might find compact representations of the Internal Objective/Target Information.

Introduction

Internal Target Information/Internal Objectives

Internal Target Information (ITI) or an Internal Objective is information about an agent's target used by the agent's internals to differentiate between actions that lead to the target outcome and actions that don’t.

The hope is that by developing interpretability methods to detect and interpret this information, we can directly detect misalignment.

Toy Models of Agents

In this post, we explore two toy models of agents and how ITI is internally represented within these models. Our models are neural-network-based with static objectives. We sometimes reflect on what these models would look like if their objectives were retargetable but do not consider this in detail.

These models are not implemented in practice but are considered in theory for speed of analysis.

Toy Model: Thermostat

The Thermostat

Consider a Thermostat y=f...
The Byronic Hero Always Loses
by Cole Wyeth — 2024-02-22T01:31:59.652Z

Why do we root for the antiheroes?

Walter White, Light Yagami, Lucifer Morningstar, Doctor Frankenstein... It seems to be natural to root for the bad guys, and perhaps even more natural for rationalists.

“I must confess I have always had some sympathy with villains. Heroism makes fine entertainment but sooner or later someone has to get things done.” - Bayaz, Joe Abercrombie's First Law Trilogy.

I think this is the reason, at least for me. Villains (particularly the most beloved villains) are agentic. They know what they want and they attempt to obtain it through their own actions. Sometimes those actions are even reasonable.

This is notable because it's usually untrue of heroes. Heroes tend to be called to adventure, accepting a goal thrust upon them by circumstances or a bearded stranger or both. Then they pursue it, but often alongside a handful of other eclectic goals which cause internal conflict when they can't all be satisfied (getting distracted rescuing some civilians when the fate of the world is at stake). There's also something fairly un-agentic about following an absolute "honor" code, which is much more common among heroes than villains. Finally, heroes have teams of friends instead of minions, and are often forced to go out of their way to help their friends. A hero's friendships might even change his terminal values!

So, why do we often find villains so fascinating?

I think it's the same reason that rationalists don't run the world.

Being highly agentic just isn't easy in real life, because it's inherently individualistic. Individuals tend to be outcompeted by large organizations. Large organizations are approximately agentic, but the people making up a large organization don't necessarily have to be. In fact, it might be better if they aren't!

The military seems to optimize much more strongly for loyalty and discipline than agency - in fact, an army of agents with diverse goals seems tricky to control, since most will instrumentally value their own survival (though an army of agents with one cohesive goal may be even more formidable).

In industry, principles of comparative advantage can imply that it is best for each employee to become highly specialized, which seems opposed to developing their agency. A more capable agent might generalize (building the type of skill set that a startup found might need, including "executive na...
The Pareto Best and the Curse of Doom
by Screwtape — 2024-02-21T23:10:01.359Z

I.

Prerequisite reading: Being the (Pareto) Best in the World.

A summary of Being the (Pareto) Best in the World: Being the world's best mathematician is hard. Being the world's best musician is hard. Being the world's best mathematician/musician is much easier, especially since there are multiple slots; an amazing mathematician who is also a competent musician, someone who is good at both, and a competent mathematician who is also an amazing musician can all find a niche.

I like this concept, and have kept it in my back pocket ever since I read it. I have sometimes described myself as a software engineer who was competent at public speaking and project management. That particular overlapping skillset is, it turns out, fairly valuable. While I was attempting to become a better software engineer, I was also trying to add competence at corporate budgets and accounting to that skillset.

These days I spend a lot of time talking to the kind of person who hangs out on LessWrong a lot or spends a lot of time going to Astral Codex Ten meetups. If ever I faced a problem that required a brilliant neuroscientist, or a gifted Haskell programmer, or a world leading expert in training honeybees, well, let's just say I know somebody. There are people out there who are exemplary at the thing they do.

Sometimes they're not very good at other things though. While Being The (Pareto) Best in the World felt optimistic when I first read it, these days I regard it as a curse of doom upon the world, blighting otherwise promising areas of effort and endeavor. I look around at places where it feels like everyone is dropping the ball and see a blasted wasteland where nothing grows because nobody has the right combination of seemingly basic skills.

II.

Imagine a toy model where everyone has a hundred points to put into being good at things.

(This is, to be clear, not just a toy model but an incorrect model. It's easy to look at your incoming university students and notice a strong inverse correlation between math and verbal SAT scores, forgetting that those get summed together during applications and anyone below a certain threshold probably has their application discarded. Still, let's use this model for the moment.)

Leading talents in a field maybe put 75 points in their area. Why not 100? Because you need points in living your life. There's an archetype of the absent minded professor, ...
AISN #31: A New AI Policy Bill in California
Plus, Precedents for AI Governance and The EU AI Office
by Dan H — 2024-02-21T21:58:34.000Z
Link: https://newsletter.safe.ai/p/aisn-31-a-new-ai-policy-bill-in-california?r=7oh0

Welcome to the AI Safety Newsletter by the Center for AI Safety. We discuss developments in AI and AI safety. No technical background required.

Subscribe here to receive future versions.

Listen to the AI Safety Newsletter for free on Spotify.

This week, we’ll discuss:

A new proposed AI bill in California which requires frontier AI developers to adopt safety and security protocols, and clarifies that developers bear legal liability if their AI systems cause unreasonable risks or critical harms to public safety.
Precedents for AI governance from healthcare and biosecurity.
The EU AI Act and job opportunities at their enforcement agency, the AI Office.

A New Bill on AI Policy in California

Several leading AI companies have public plans for how they’ll invest in safety and security as they develop more dangerous AI systems. A new bill in California’s state legislature would codify this practice as a legal requirement, and clarify the legal liability faced by developers whose AI systems cause unreasonable risks or critical harms to public safety. (Note: The bill was co-sponsored by the Center for AI Safety Action Fund.)

Many of the world’s leading AI developers and cloud compute providers are located in California. This means the state has significant direct control over AI development, and it could serve as a testing grounds for policies that might later be adopted nationally or internationally.

Safety and security protocols for advanced AI models. This bill would require developers of certain leading AI models to adopt safety and security protocols. This requirement only applies to models that may surpass the capabilities of all models previously established as safe. Models that will not exceed the performance of known safe models are exempt from these requirements.

Specific requirements include cybersecurity measures to “prevent theft, misappropriation, malicious use, or inadvertent release or escape of the model weights.” Securing model weights against theft might require multi-year efforts to build air-gapped data centers and insider threat detection programs, argued a recent RAND report. Additionally, developers would need to implement the capacity to fully shut down their models in the event of an emergency. More broadly, they would need to implement best practices as developed by industry, academia, and standard-setting organiza...
Analogies between scaling labs and misaligned superintelligent AI
by scasper — 2024-02-21T19:29:39.033Z

TL;DR: Scaling labs have their own alignment problem analogous to AI systems, and there are some similarities between the labs and misaligned/unsafe AI.

Introduction

Major AI scaling labs (OpenAI/Microsoft, Anthropic, Google/DeepMind, and Meta) are very influential in the AI safety and alignment community. They put out cutting-edge research because of their talent, money, and institutional knowledge. A significant subset of the community works for one of these labs. This level of influence is beneficial in some aspects. In many ways, these labs have strong safety cultures, and these values are present in their high-level approaches to developing AI – it’s easy to imagine a world in which things are much worse. But the amount of influence that these labs have is also something to be cautious about.

The alignment community is defined by a concern that subtle misalignment between the incentives that we give AI systems and what we actually want from them might cause these systems to dangerously pursue the wrong goals. This post considers a type of analogous and somewhat ironic alignment problem: between human interests and the scaling labs.

These labs have intelligence, resources, and speed well beyond that of any single human. Their amount of money, compute, talent, and know-how make them extremely capable. Given this, it’s important that they are aligned with the interests of humanity. However, there are some analogies between scaling labs and misaligned AI.

It is important not to draw false equivalences between different labs. For example, it seems that by almost every standard, Anthropic prioritizes safety and responsibility much more than other labs. But in this post, I will generally be lumping them together except to point out a few lab-specific observations.

Misaligned Incentives

In much the same way that AI systems may have perverse incentives, so do the labs. They are companies. They need to make money, court investors, make products, and attract users. Anthropic and Microsoft even just had Super Bowl ads. This type of accountability to commercial interests is not perfectly in line with doing what is good for human interests. Moreover, the labs are full of technocrats whose values and demographics do not represent humanity particularly well. Optimizing for the goals that the labs have is not the same thing as optimizing for human welfa...
Extinction Risks from AI: Invisible to Science?
by VojtaKovarik — 2024-02-21T18:07:33.986Z
Link: http://arxiv.org/abs/2403.05540

Abstract: In an effort to inform the discussion surrounding existential risks from AI, we formulate Extinction-level Goodhart’s Law as "Virtually any goal specification, pursued to the extreme, will result in the extinction[1] of humanity'', and we aim to understand which formal models are suitable for investigating this hypothesis. Note that we remain agnostic as to whether Extinction-level Goodhart's Law holds or not. As our key contribution, we identify a set of conditions that are necessary for a model that aims to be informative for evaluating specific arguments for Extinction-level Goodhart's Law. Since each of the conditions seems to significantly contribute to the complexity of the resulting model, formally evaluating the hypothesis might be exceedingly difficult. This raises the possibility that whether the risk of extinction from artificial intelligence is real or not, the underlying dynamics might be invisible to current scientific methods.

Together with Chris van Merwijk and Ida Mattsson, we have recently written a philosophy-venue version of some of our thoughts on Goodhart's Law in the context of powerful AI [link].[2] This version of the paper has no math in it, but it attempts to point at one aspect of "Extinction-level Goodhart's Law" that seems particularly relevant for AI advocacy – namely, that the fields of AI and CS would have been unlikely to come across evidence of AI risk, using the methods that are popular in those fields, even if the law did hold in the real world.

Since commenting on link-posts is inconvenient, I split off some of the ideas from the paper into the following separate posts:

Weak vs Quantitative Extinction-level Goodhart's Law: defining different versions of the notion of "Extinction-level Goodhart's Law".
Which Model Properties are Necessary for Evaluating an Argument?: illustrating the methodology of the paper on a simple non-AI example.
Dynamics Crucial to AI Risk Seem to Make for Complicated Models: applying the methodology above to AI risk.

We have more material on this topic, including writing with math[3] in it, but this is mostly not yet in a publicly shareable form. The exception is the post Extinction-level Goodhart's Law as a Property of the Environment (which is not covered by the paper). If you are interested in discussing anything related to this, definitely reach out.

^

A common comment is that

...
Extinction-level Goodhart's Law as a Property of the Environment
by VojtaKovarik — 2024-02-21T17:56:02.052Z

Summary: Formally defining Extinction-level Goodhart's Law is tricky, because formal environments don't contain any actual humans that could go extinct. But we can do it using the notion of an interpretation mapping, which sends outcomes in the abstract environment to outcomes in the real world. We can then the truth condition of Extinction-level Goodhart's Law as a property of the environment.
I conjecture that Extinction-level Goodhart's Law does not hold in easily formalisable environments[1], even though it might hold in the real world. This seems like a (very) big deal for AI advocacy, since it suggests that the lack of rigorous arguments concerning AI risk (eg, math proofs) does not provide strong evidence for the safety of AI.

Semi-formal definition of Extinction-level Goodhart's Law

Informally, we can define the extinction-level[2] variant of Goodhart's law as follows:

Definition (informal): The Weak Version[3] of Extinction-level Goodhart's Law is the claim that: "Virtually any goal specification, pursued to the extreme, will result in the extinction of humanity."

The tricky part is how to translate this into a more rigorous definition that can be applied to formal environments.

Defining "extinction" in formal environments

However, applying this definition to formal environments is tricky, because it requires formally defining which of the abstract states qualify as "extinction of humanity". How can we get past this obstacle?

What we don't do: "extinction states" given by definition

A lazy way of defining extinction in abstract models would be to assume that we are given some such "extinction" states by definition. That is, if Ω is the set of all possible states in the formal environment, we could assume that there is some set Ωextinction⊂Ω. And we would refer to any ω∈Ωextinction as "extinction".
I don't like this approach, because it just hides the problem elsewhere. Also, this approach does not put any constraints on how Ωextinction should look like. As a result, we would be unlikely to be able to derive any results.

The approach we take instead is to augment the formal environment by an interpretation, where each state "abstract" state ω in the formal environment Ω is mapped onto some state φ(ω) in some "more complex" environment Ω′.[4]

Real-world interpretations

The typical use ...
Dynamics Crucial to AI Risk Seem to Make for Complicated Models
by VojtaKovarik — 2024-02-21T17:54:46.089Z

This post overlaps with our recent paper Extinction Risk from AI: Invisible to Science?.

tl;dr: In AI safety, we are worried about certain problems with using powerful AI. (For example, the difficulty of value specification, instrumental convergence, and the possibility that a misaligned AI will come up with takeover strategies that didn't even occur to us.) To study these problems or convince others that they are real, we might wish to describe them using mathematical models. However, this requires using models that are sufficiently rich that these problems could manifest in the first place.
In this post, I suggest thinking about how such "rich-enough" models could look like. Also, I raise the possibility that models which are rich enough to capture problems relevant to AI alignment might be too complex to be amenable to a rigorous analysis.

Epistemic status: Putting several related observations into one place. But I don't have strong opinions on what to make of them.

In the previous post, I talked about "straightforwardly evaluating" arguments by modelling the dynamics described in those arguments. In this post, I go through some dynamics that seem central to AI risk. However, none of these dynamics is meant to be novel or surprising. Instead, I wish to focus on the properties of the mathematical models that could capture these dynamics. How do such models look like? How complicated are they? And --- to the extent that answering some questions about AI risk requires modeling the interplay between multiple dynamics --- is there some minimal complexity of models which can be useful for answering those questions?

Laundry List of Dynamics Closely Tied to AI Risk

In this section, I list a number of dynamics that seem closely tied to AI risk, roughly[1] grouped based on which part of the "AI risk argument" they relate to. Below each part of this list, I give some commentary on which models might be useful for studying the given dynamics. I recommend reading selected parts that seem interesting to you, rather than going through the whole text.

For the purpose of skimming, here is a list of the dynamics, without any explanations:

I. Difficulty of specifying our preferences[2]:

Human preferences are ontologically distant from the laws of physics.
Human preferences are ontologically distant from the language we use to design the AI.
Laws of physics are unknown.
Human prefe

...