trentmkelly/LessWrong-43k huggingface dataset:
AISN #16: White House Secures Voluntary Commitments from Leading AI Labs and Lessons from Oppenheimer
by Corin Katzke — 2023-07-25T16:58:44.528Z
Link: https://newsletter.safe.ai/p/ai-safety-newsletter-16

Welcome to the AI Safety Newsletter by the

Center for AI Safety

. We discuss developments in AI and AI safety. No technical background required.

Subscribe here to receive future versions.

White House Unveils Voluntary Commitments to AI Safety from Leading AI Labs

Last Friday, the White House

announced

a series of voluntary

commitments

from seven of the world's premier AI labs. Amazon, Anthropic, Google, Inflection, Meta, Microsoft, and OpenAI pledged to uphold these commitments, which are non-binding and pertain only to forthcoming "frontier models" superior to currently available AI systems. The White House also notes that the Biden-Harris Administration is developing an executive order alongside these voluntary commitments.

The commitments are timely and technically well-informed, demonstrating the ability of federal policymakers to respond capably and quickly to AI risks. The Center for AI Safety

supports

these commitments as a precedent for cooperation on AI safety and legally binding legislation.

Commitments to Red Teaming and Information Sharing. AI systems

often develop unexpected capabilities

during training, so it’s important to

screen models for potentially harmful abilities

. The companies promised to “red team” their models by searching for ways the models could cause harm, particularly in

areas of potential catastrophe

including biological weapons acquisition, offensive cyberattacks, and self-replication. External red teamers will also be offered a chance to find vulnerabilities in these frontier AI systems.

These are the catastrophic capabilities that companies agreed to evaluate via red teaming.

The companies have additionally agreed to share information about best practices in AI safety with governments and each other. Public reports on system capabilities and risks are also part of this pledge, though companies have not agreed to the disclosure of model training data, which likely contains large volumes of copyrighted content.

Cybersecurity to protect AI models, if companies choose. The AI labs also reiterated their dedication to cybersecurity. This helps prevent potent AI models from being accessed and misused by malicious actors. By reducing the likelihood that someone else will steal their cutting edge secrets, cybersecurity also alleviates the pressure of the AI race and gives companies more time to evaluate and enhance the safety feat...
"The Universe of Minds" - call for reviewers (Seeds of Science)
by rogersbacon — 2023-07-25T16:53:44.775Z

Abstract

The paper attempts to describe the space of possible mind designs by first equating all minds to software. Next it proves some interesting properties of the mind design space such as infinitude of minds, size and representation complexity of minds. A survey of mind design taxonomies is followed by a proposal for a new field of investigation devoted to study of minds, intellectology, a list of open problems for this new field is presented.

---

Seeds of Science is a journal (funded through Scott Alexander's ACX grants program) that publishes speculative or non-traditional articles on scientific topics. Peer review is conducted through community-based voting and commenting by a diverse network of reviewers (or "gardeners" as we call them). Comments that critique or extend the article (the "seed of science") in a useful manner are published in the final document following the main text.

We have just sent out a manuscript for review, "The Universe of Minds", that may be of interest to some in the LessWrong community so I wanted to see if anyone would be interested in joining us as a gardener and providing feedback on the article. As noted above, this is an opportunity to have your comment recorded in the scientific literature (comments can be made with real name or pseudonym).

It is free to join as a gardener and anyone is welcome (we currently have gardeners from all levels of academia and outside of it). Participation is entirely voluntary - we send you submitted articles and you can choose to vote/comment or abstain without notification (so no worries if you don't plan on reviewing very often but just want to take a look here and there at the articles people are submitting).

To register, you can fill out this google form. From there, it's pretty self-explanatory - I will add you to the mailing list and send you an email that includes the manuscript, our publication criteria, and a simple review form for recording votes/comments. If you would like to just take a look at this article without being added to the mailing list, then just reach out (info@theseedsofscience.org) and say so.

Happy to answer any questions about the journal through email or in the comments below.
Thoughts on Loss Landscapes and why Deep Learning works
by beren — 2023-07-25T16:41:39.562Z

Crossposted from my personal blog.

Epistemic status: Pretty uncertain. I don’t have an expert level understanding of current views in the science of deep learning about why optimization works but just read papers as an amateur. Some of the arguments I present here might be already either known or disproven. If so please let me know!

There are essentially two fundamental questions in the science of deep learning: 1.) Why are models trainable? And 2.) Why do models generalize? The answer to both of these questions relates to the nature and basic geometry of the loss landscape which is ultimately determined by the computational architecture of the model. Here I present my personal and fairly idiosyncratic and speculative answers to these questions and present what I think are fairly novel answers for both of these questions. Let’s get started.

First let’s think about the first question: Why are deep learning models trainable at all? A-priori, they shouldn’t be. Deep neural networks are fundamentally solving incredibly high dimensional problems with extremely non-convex loss-landscape. The entirety of optimization theory should tell you that, in general, this isn’t possible. You should get stuck in exponentially many local minima before getting close to any decent optimum, and also you should probably just explode because you are using SGD, the stupidest possible optimizer.

So why does it, in fact, work? The fundamental reason is that we are very deliberately not in the general case. Much of the implicit work of deep learning is fiddling with architectures, hyperparameters, initializations, and so forth to ensure that, in the specific case, the loss landscape is benign. To even get SGD working in the first place, there are two fundamental problems which neural networks solve:

First, vanilla SGD assumes unit variance. This can be shown straightforwardly from a simple Bayesian interpretation of SGD and is also why second order optimizers like natural gradients are useful. In effect, SGD fails whenever the variance and curvature of the loss landscape is either too high (SGD will explode) or too small (SGD will grind to a halt and take forever to traverse the landscape). This problem has many names such as ‘vanishing/exploding gradients’, ‘poor conditioning’, ‘poor initialization’, and so on. Many of the tricks to get neural networks to work address this initializing SGD...
Should you work at a leading AI lab? (including in non-safety roles)
by Benjamin Hilton — 2023-07-25T16:29:39.371Z

This post is a (slightly) edited cross-post of a new 80,000 Hours career review on working at a a leading AI lab. See EA Forum comments here.

Summary

Working at a leading AI lab is an important career option to consider, but the impact of any given role is complex to assess. It comes with great potential for career growth, and many roles could be (or lead to) highly impactful ways of reducing the chances of an AI-related catastrophe — one of the world’s most pressing problems. However, there’s a risk of doing substantial harm in some cases. There are also roles you should probably avoid.

Pros

Many roles have a high potential for impact by reducing risks from AI
Among the best and most robust ways to gain AI-specific career capital
Possibility of shaping the lab’s approach to governance, security, and standards

Cons

Can be extremely competitive to enter
Risk of contributing to the development of harmful AI systems
Stress and frustration, especially because of a need to carefully and frequently assess whether your role is harmful

Key facts on fit

Excellent understanding of the risks posed by future AI systems, and for some roles, comfort with a lot of quick and morally ambiguous decision making. You’ll also need to be a good fit for the specific role you’re applying for, whether you’re in research, comms, policy, or something else (see our related career reviews).

Overall recommendation: it's complicated

We think there are people in our audience for whom this is their highest impact option — but some of these roles might also be very harmful for some people. This means it's important to take real care figuring out whether you're in a harmful role, and, if not, whether the role is a good fit for you.

Review status

Based on a medium-depth investigation

This review is informed by two surveys of people with expertise about this path — one on whether you should be open to roles that advance AI capabilities (written up here), and a second follow-up survey. We also performed an in-depth investigation into at least one of our key uncertainties concerning this path. Some of our views will be thoroughly researched, though it's likely there are still some gaps in our understanding, as many of these considerations remain highly debated.

Why might it be high-impact to work for a leading AI lab?

We think AI is likely to have transformative effects over the coming decades. We also think that r...
AIS 101: Task decomposition for scalable oversight
by Charbel-Raphaël — 2023-07-25T13:34:58.507Z
Link: https://docs.google.com/document/d/1k6rlyBCZJw8xbUx0dzd-4sOhlzj-xzsmwi_OIZY1-3M/edit?usp=sharing

AGISF Week 4 - Task decomposition for scalable oversight

This text is an adapted excerpt from the task decomposition for scalable oversight section of the AGISF 2023 course, held at ENS Ulm in Paris on March 16, 2023. Its purpose is to provide a concise overview of the essential aspects of the session's program for readers who may not delve into additional resources. This document aims to capture the 80/20 of the session's content, requiring minimal familiarity with the other materials covered. I tried to connect the various articles within a unified framework and coherent narrative. This distillation is not purely agnostic. You can find the other summaries of AGISF week on this

page

. This summary is not the official AGISF content. I have left the gdoc accessible in comment mode, feel free to comment there.

Thanks to Jeanne Salle, Markov, Amaury Lorin and Clément Dumas for useful comments.

Outline

How can we still provide good feedback when a task becomes too hard for a human to evaluate?

Scalable oversight refers to methods that enable humans to oversee AI systems that are solving tasks too complicated for a single human to evaluate. AIs will probably become very powerful, and perform tasks that are difficult for us to verify. In which case, we would like to have a set of procedures that allow us to train them and verify what they do.

I introduce scalable oversight as an approach to preventing reward misspecification, i.e., when an artificial intelligence (AI) system optimizes for the wrong objective, leading to unintended and potentially harmful outcomes. In the current machine learning (ML) paradigm, good quality feedback is essential. I then discuss one important scalable oversight proposal: Iterated Distillation and Amplification (IDA). The Superalignment team

will likely

use some of the techniques presented here, so bear with me!

Here's the outline:

Presenting the scalable oversight problem and one framework to evaluate solutions to this problem, sandwiching

(Bowman, 2022)

.
To solve the scalable oversight problem, I focus here on task decomposition (next post will focus on adversarial techniques) and I then examine some potential solutions:

Factored cognition tries to break down sophisticated reasoning into many small and mostly independent tasks. One approach for factored cognition is Iterated Amplification. The non-

...
Anthropic Observations
by Zvi — 2023-07-25T12:50:03.178Z

Dylan Matthews had in depth Vox profile of Anthropic, which I recommend reading in full if you have not yet done so. This post covers that one.

Anthropic Hypothesis

The post starts by describing an experiment. Evan Hubinger is attempting to create a version of Claude that will optimize in part for a secondary goal (in this case ‘use the word “paperclip” as many times as possible’) in the hopes of showing that RLHF won’t be able to get rid of the behavior. Co-founder Jared Kaplan warns that perhaps RLHF will still work here.

Hubinger agrees, with a caveat. “It’s a little tricky because you don’t know if you just didn’t try hard enough to get deception,” he says. Maybe Kaplan is exactly right: Naïve deception gets destroyed in training, but sophisticated deception doesn’t. And the only way to know whether an AI can deceive you is to build one that will do its very best to try.

The problem with this approach is that an AI that ‘does its best to try’ is not doing the best that the future dangerous system will do.

So by this same logic, a test on today’s systems can only show your technique doesn’t work or that it works for now, it can never give you confidence that your technique will continue to work in the future.

They are running the test because they think that RLHF is so hopeless we can likely already prove, at current optimization levels, that it is doomed to failure.

Also, the best try to deceive you will sometimes be, of course, to make you think that the problem has gone away while you are testing it.

This is the paradox at the heart of Anthropic: If the thesis that you need advanced systems to do real alignment work is true, why should we think that cutting edge systems are themselves currently sufficiently advanced for this task?

But Anthropic also believes strongly that leading on safety can’t simply be a matter of theory and white papers — it requires building advanced models on the cutting edge of deep learning. That, in turn, requires lots of money and investment, and it also requires, they think, experiments where you ask a powerful model you’ve created to deceive you.

“We think that safety research is very, very bottlenecked by being able to do experiments on frontier models,” Kaplan says, using a common term for models on the cutting edge of machine learning. To break that bottleneck, you need access to those frontier models. P

...
Autonomous Alignment Oversight Framework (AAOF)
by Justausername — 2023-07-25T10:25:03.090Z

Abstract:

To align advanced AIs, an ensemble of diverse, transparent Overseer AIs will independently monitor the target AI and provide granular assessments on its alignment with constitution, human values, ethics, and safety. Overseer interventions will be incremental and subject to human oversight. The system will be implemented cautiously, with extensive testing to validate capabilities. Alignment will be treated as an ongoing collaborative process between humans, Overseers, and the target AI, leveraging complementary strengths through open dialog. Continuous vigilance, updating of definitions, and contingency planning will be required to address inevitable uncertainties and risks.

Introduction:

As advanced AI systems grow in capability and autonomy, ensuring their alignment with human values becomes increasingly critical. The proposed Overseer system aims to provide ongoing oversight and course-correction to optimize AI alignment. By utilizing an ensemble of diverse, transparent AIs focused on narrow domains of alignment, combined with human oversight, the system seeks to assess and enhance alignment in a measured, responsible way.

The main proposal:

The Overseer system will consist of a diverse ensemble of AI modules with different architectures and training methodologies. Each module will focus on assessing a narrow domain of alignment, such as ethics or safety. The modules will provide granular confidence scores justifying their assessments, rather than binary decisions.

Transparency tools will be used to monitor information flows and logic within each overseer to prevent leaks or deception. The Overseers will also monitor each other's internal states for signs of compromise.

Initially, the Overseer interventions will be confined to issuing warning reports, logging concerning events, and minor parameter adjustments. Before enacting more disruptive interventions like system shutdowns, thorough testing and validation will be required.

A human oversight board will audit Overseer behaviors and decisions on an ongoing basis. They will evaluate whether the AIs are applying human values appropriately and override interventions if needed.

Overseers will participate in simulations and scenarios to improve robustness against edge cases. Extensive red team penetration testing will be used to find weaknesses. Contingency plans will be instituted in case Overseer mal...
How LLMs are and are not myopic
by janus — 2023-07-25T02:19:44.949Z

Thanks to janus, Nicholas Kees Dupuis, and Robert Kralisch for reviewing this post and providing helpful feedback. Some of the experiments mentioned were performed while at Conjecture.

TLDR: The training goal for LLMs like GPT is not cognitively-myopic (because they think about the future) or value myopic (because the transformer architecture optimizes accuracy over the entire sequence, not just the next-token). However, training is consequence-blind, because the training data is causally independent of the models actions. This assumption breaks down when models are trained on AI generated text.

Summary

Myopia in machine learning models can be defined in several ways. It could be the time horizon the model considers when making predictions (cognitive myopia), the time horizon the model takes into account when assessing its value (value myopia), or the degree to which the model considers the consequences of its decisions (consequence-blindness).
Both cognitively-myopic and consequence-blind models should not pursue objectives for instrumental reasons. This could avoid some important alignment failures, like power-seeking or deceptive alignment. However, these behaviors can still exist as terminal values, for example when a model is trained to predict power-seeking or deceptively aligned agents.
LLM pretraining is not cognitively myopic because there is an incentive to think about the future to improve immediate prediction accuracy, like when predicting the next move in a chess game.
LLM pretraining is not value/prediction myopic (does not maximize myopic prediction accuracy) because of the details of the transformer architecture. Training gradients flow through attention connections, so past computation is directly optimized to be useful when attended to by future computation. This incentivizes improving prediction accuracy over the entire sequence, not just the next token. This means that the model can and will implicitly sacrifice next-token prediction accuracy for long horizon prediction accuracy.
You can modify the transformer architecture to remove the incentive for non-myopic accuracy, but as expected, the modified architecture has worse scaling laws.
LLM pretraining on human data is consequence-blind as the training data is causally independent from the model's actions. This implies the model should predict actions without considering the effect of its actions

...
Secure Hand Holding
by jefftk — 2023-07-25T01:40:01.553Z

When Lily as about three we were waiting at a crosswalk and traffic
stopped for us. As we started across the street a driver that had
been waiting to turn left misinterpreted the situation and, thinking
traffic had stopped for them instead, tried to turn through our
location. I tightened my grip and ran, pulling her through the air.
The car stopped, about five feet past where it would have collided
with us. Everyone was fine, but we were very shaken. And I was glad we'd been holding hands securely.

When we're walking around at the park or another relaxed environment
there are a range of ways I'll hold hands with little kids. A common one
is that I'll extend a single finger, and they'll wrap their hand
around it. But when we're crossing the street, climbing stairs, or
I otherwise might suddenly need to hold them up entirely, I
additionally wrap my fingers loosely around their wrist:

Another angle:

The specific hold isn't that important: what matters is that it's
comfortable for you both and if you had to you could really hold on.
Julia finds that with the smaller size of her hands a different grip
works better:

I suspect many of you are thinking "of course, how else would you do
it?" Many parents end up doing this pretty naturally. But if you
currently are choosing between a friendly but weak companionship grip
and securely holding their wrist this could be a nice thing to try!

Comment via: facebook, mastodon
Open problems in activation engineering
by TurnTrout — 2023-07-24T19:46:08.733Z
Link: https://coda.io/@alice-rigg/open-problems-in-activation-engineering

Steering GPT-2-XL by adding an activation vector introduced

activation engineering... techniques which steer models by modifying their activations. As a complement to prompt engineering and finetuning, activation engineering is a low-overhead way to steer models at runtime.

These results were recently complemented by Inference-Time Intervention: Eliciting Truthful Answers from a Language Model, which doubled TruthfulQA performance by adding a similarly computed activation vector to forward passes!

We think that activation engineering has a bunch of low-hanging fruit for steering and understanding models. A few open problems from the list:

Try decomposing the residual stream activations over a batch of inputs somehow (e.g. PCA). Using the principal directions as activation addition directions, do they seem to capture something meaningful?
Take a circuit studied from existing literature on GPT2, or find another one using

ACDC

. Targeting the nodes in these circuits, can you learn anything more about them and generally about how activation additions interact with circuits?
What's the mechanism by which adding a steering vector with too large a coefficient breaks the model? (Credit: Thomas Kwa; see also @Ulisse Mini's initial data/explanation.)

If you want to work on activation engineering, come by the Slack server to coordinate research projects and propose new ideas.