trentmkelly/LessWrong-43k huggingface dataset:
Auditing games for high-level interpretability
by Paul Colognese — 2022-11-01T10:44:07.630Z

This post was written under Evan Hubinger’s direct guidance and mentorship, as a part of the

Stanford Existential Risks Institute ML Alignment Theory Scholars (MATS) program

.

Thanks to Evan Hubinger, Arun Jose, and Dane Sherburn for discussions/feedback.

Introduction

This post aims to extend the

auditing games

framework to high-level interpretability; a framework for adversarially testing future interpretability methods developed for understanding and working with a model’s internal properties at a high-level (e.g. understanding whether or not a model is performing optimization, etc.)

We begin by discussing what we mean by high-level interpretability and why we should care about it. We then introduce an auditing games framework in this context along with some simple examples. We then argue that this framework provides good benchmarks for evaluating our high-level interpretability methods. Finally, we discuss a fundamental difficulty that we encounter when trying to create good tests within the auditing games framework described in this post.

What is high-level interpretability and why should we care?

Understanding what our models are doing internally seems necessary for us to have confidence that they will perform well during deployment.

One key problem in prosaic alignment is that of distributional shifts: the set of models that perform well during and at the end of training may include models that perform poorly, potentially catastrophically, during deployment.

Deceptive models

fit this description, i.e., a model that performs well during the training process as an instrumental goal so that it can be deployed and pursue its unaligned terminal goal without fear of modification from the training process. Unfortunately, observing the external behavior of models during training can’t help us differentiate between an aligned model and a competent deceptive model.

One way we can gain insight into how a model will behave in deployment is by understanding how the model operates internally/what algorithm(s) it’s running. For example, if we could use interpretability tools to look inside a model and understand whether it’s an optimizer, and if so what its terminal goals are, then we would better understand how it’s going to behave in deployment, in particular, whether it’s going to pursue aligned goals or not.

Many

alignment proposals

require the capacity...
Remember to translate your thoughts back again
by brook — 2022-11-01T08:49:12.812Z
Link: https://forum.effectivealtruism.org/posts/iofbdMrfQp3icZSn4/remember-to-translate-your-thoughts-back-again

Epistemic status: casual observation written as first draft but published as-is because it was unlikely to ever actually get edited.

There's a common failure mode I see among EAs and particularly rationalists where they fail to translate their thoughts back when communicating. This post is just explaining what I mean by this.

Visual Summary

Made with excalidraw

Verbal thought

Many people think verbally. Some people think pictorially or in some other fashion, but most people, most of the time, construct their thoughts as something-like an ongoing sentence made up of words.

For most people, this is a really useful feature! When you want to share things with somebody, the level of 'translation' from your thoughts to language is very low. Everybody I know who struggles with verbal speech thinks in a non-verbal format (which could of course cut both ways causally).

Less-verbal thought

There are three key problems introduced by rationalism to the usual process above; jargon, the rationalist desire for intense precision, and thinking truly weird thoughts in general.

This is less specific to rationalism/EA, but many people think in words which other people don't know. Even when they try to translate their jargon into something more understandable, they underestimate the inferential distance, or they map words poorly (where an equivalent doesn't exist in 'normal English') such that most of the meaning is lost.

When this is combined with the rationalist desire for extreme precision in speech and thought, it contributes to unpleasant listening and bad communication. If people can get a 1-sentence summary which means 90% the same thing as what you're thinking, this is (usually!) better than people getting an hour-long lecture on your specific subdomain of AI alignment just to understand a peripheral point. It's ok if people don't know your precise epistemics for a passing comment. No really, I promise.

This is worsened further by the fact that a key recommendation for rationalists is 'thinking weird thoughts'. Rationality often involves inspecting the terms you're using, defining them as narrowly as possible, and trying to actually use them in this way. This is good for thought, but bad for communication. If you're planning to use a word a lot, or if your definition varies a lot from the standard one, it's worth defining it-- but track this actively! ...
Conversations on Alcohol Consumption
by Annapurna — 2022-11-01T05:09:34.374Z

Recently I have decided to stop drinking alcohol. Below are some conversations with two friends who are on similar paths. I decided to post these conversations because I am looking for as many perspectives as possible and the LessWrong community might have some insights. I will keep posting future conversations with my friends in the comments.
Chicago is an old friend of mine who decided to stop drinking four years ago. I decided to message him when I decided to stop drinking 30 days ago, and our email conversations are below.
Adirondack is an acquaintance of mine who a few days ago posted on social media his desire to begin a journey to cut back our outright quit drinking alcohol.

On Thu, 6 Oct 2022 at 03:57, Chicago wrote:
Hey bud, following up about not drinking.

I've got a lot of thoughts about this, on both the theoretical and the practical sides, but what would be most useful to you?

Happy to jump in, and will talk ad nauseum about this bc it's a favorite topic.

Thanks,

Chicago

On Thu, Oct 6, 2022 at 1:43 AM Annapurna wrote:

Hey brother,
Thank you for wanting to chat with me about this subject. So I've actually been thinking about how harmful alcohol has been to me for about 10 years. I have journal entries from 2012 saying how much I hate how I feel after a night of drinking, and questioning whether the lost time and energy was worth it.
My questioning whether alcohol is a substance for me comes from two places:
1. Hangovers / physical and mental state the day (s) after drinking. In 2011 I spent 3 months without drinking alcohol while studying for a professional exam. So by the time the exam came, I felt levels of energy and lucidness that I had never felt before. It's as if since I was 15 my body was running at 80% capacity constantly (since I probably went less than 15 days without a drink from 2003-2011), but then after 60+ days of no alcohol, I finally felt what it was like to be 100%. I go back to savouring that feeling a lot. Even on days that I only have 5 drinks or so and I feel the slight hangover the next day, I question myself. No matter how fun the night before was.
2. Sometimes I have a very hard time stopping, leading to extreme drunkenness and putting myself in vulnerable / dangerous situations. It's not often, but often enough. It has happened at every burn except the last one. The maj...
ML Safety Scholars Summer 2022 Retrospective
by TW123 — 2022-11-01T03:09:10.305Z

TLDR

This is a report on the Machine Learning Safety Scholars program, organized over the summer by the Center for AI Safety. 63 students graduated from MLSS: the list of graduates and final projects is

here

. The program was intense, time-consuming, and at times very difficult, so graduation from the program is a significant accomplishment.

Overall, I think the program went quite well and that many students have noticeably accelerated their AI safety careers. There are certainly many areas of improvement that could be made for a future iteration, and many are detailed here. We plan to conduct followup surveys to determine the longer-run effects of the program.

This post contains three main sections:

This TLDR, which is meant for people who just want to know what this document is and see our graduates list.
The executive summary, which includes a high-level overview of MLSS. This might be of interest to students considering doing MLSS in the future or anyone else interested in MLSS.
The full report, which was mainly written for future MLSS organizers, but I’m publishing here because it might be useful to others running similar programs.

The report was written by Thomas Woodside, the project manager for MLSS. “I” refers to Thomas, and does not necessarily represent the opinion of the Center for AI Safety, its CEO Dan Hendrycks, or any of our funders.

Visual Executive Summary

MLSS Overview

MLSS was a summer program for mostly undergraduate students aimed to teach the foundations of machine learning, deep learning, and ML safety. The program ended up being ten weeks long and included an optional final project. It incorporated office hours, discussion sections, speaker events, conceptual readings, paper readings, written assignments, and programming assignments. You can see our full curriculum

here

.

Survey Results

All MLSS graduates filled out an anonymous survey. What follows is a mostly visual depiction of the program through the lens of these survey results.

Overall Experience in MLSS

Of course, this sample is biased towards graduates of MLSS, since we required them to complete the survey to receive their final stipends (and non graduates didn’t get final stipends). However, it seems clear from the way that people responded to this survey that the majority had quite positive opinions of our program.

We also asked students about their future plans:

The results ...
EA & LW Forums Weekly Summary (24 - 30th Oct 22')
by Zoe Williams — 2022-11-01T02:58:09.914Z

Supported by Rethink Priorities

This is part of a weekly series - you can see the full collection here. The first post includes some details on purpose and methodology.

If you'd like to receive these summaries via email, you can subscribe here.

Podcast version: prefer your summaries in podcast form? A big thanks to Coleman Snell for producing these! Subscribe on your favorite podcast app by searching for 'Effective Altruism Forum Podcast'.

Top / Curated Readings

Designed for those without the time to read all the summaries. Everything here is also within the relevant sections later on so feel free to skip if you’re planning to read it all.

Announcing EA Survey 2022

by David_Moss, WillemSleegers

The EA Survey is an annual survey of the effective altruism community, and is now live

at this link

until Dec 1st. The results inform decisions for a number of different orgs and help to understand how the community is changing over time. This year’s survey is substantially shorter than 2021, and should take <10 minutes. If you’re reading this, you fit the target audience and your response is valuable. Sharing widely with other EAs is also appreciated.

EA movement course corrections and where you might disagree

by michel

The second of the two posts aiming to make discussion of how we grow the EA movement more concrete. EA is on a BAU trajectory, built up of small intentional decisions (eg. rough growth rate, name & branding), nudges from community leaders, and natural social dynamics. This post analyzes different domains where we could change trajectory, and key considerations for assessing these - without advocating any particular approach.

The domains identified group into:

How much EA should stay as a single entity and identity vs. splitting branding / outreach / identity into philosophical or cause-specific areas.
How EA mixes social and professional, and how much it interacts with outside communities and professional networks.
Growth rate, level of demands of members, and where growth is targeted (eg. do we continue the focus on top universities in high-income countries, or diversify?)
Level of centralization of funding and decision making.

These could affect decisions on location of hubs, onboarding content, target audience of projects or events, branding, funding choices and much more.

Considerations on many uncertain questions such as TAI timelines...
Caution when interpreting Deepmind's In-context RL paper
by Sam Marks — 2022-11-01T02:42:06.766Z

Lots of people I know have had pretty strong reactions to the recent Deepmind paper, which claims to have gotten a transformer to learn an RL algorithm by training it on an RL agent's training trajectories. At first, I too was pretty shocked -- this paper seemed to provide strong evidence of a mesa-optimizer in the wild. But digging into the paper a bit more, I'm quite unimpressed and don't think that in-context RL is the correct way to interpret the experiments that the authors actually did. This post is a quick, low-effort attempt to write out my thoughts on this.

Recall that in this paper, the authors pick some RL algorithm, use it to train RL agents on some tasks, and save the trajectories generated during training; then they train a transformer to autoregressively model said trajectories, and deploy the transformer on some novel tasks. So for concreteness, during training the transformer sees inputs that look like

(sn,an,rn,sn+1,an+1,rn+1,…,sn+c)

which were excerpted from an RL agent's training on some task (out of a set of training tasks) and which span multiple episodes (i.e. at some point in this input trajectory, one episode ended and the next episode began). The transformer is trained to guess the action an+c that comes next. In deployment, the inputs are determined by the transformer's own selections, with the environment providing the states and rewards. The authors call this algorithmic distillation (AD).

Many people I know have skimmed the paper and come away with an understanding something like:

In this paper, RL agents are trained on diverse tasks, e.g. playing many different Atari games, and the resulting transcripts are used as training data for AD. Then the AD agent is deployed on a new task, e.g. playing a held-out Atari game. The AD agent is able to learn to play this novel game, which can only be explained by the model implementing an reasonably general RL algorithm. This sounds a whole lot like a mesa-optimizer.

This understanding is incorrect, with two key issues. First the training tasks used in this paper are all extremely similar to each other and to the deployment task; in fact, I think they only ought to count as different under a pathologically narrow notion of "task." And second, the tasks involved are extremely simple. The complaints taken together challenge the conclusion that the only way for the AD agent to do wel...
What sorts of systems can be deceptive?
by Andrei Alexandru — 2022-10-31T22:00:27.312Z

This work was done as part of SERI MATS, under Leo Gao’s guidance. Thank you to Erik Jenner and Johannes Treutlein for discussions and comments on the draft.

I’m interested in understanding deception in machine intelligence better. Specifically, I want to understand what precursors there are to deceptive alignment, and whether upon observing these precursors, we can change our approach to achieve better outcomes. In this article, I outline my current thinking on this topic, and consider a bunch of properties that systems which can be deceptive might share. I am still pretty confused about how this works, and I don’t yet have good ideas for what comes next.

Preliminaries

What is deception?

Commonly, by deception we mean a scenario where an intelligent system behaves in a way that hides what information it knows from another system, or from a human overseer. Fundamentally, the reason to be deceptive is that behaving honestly would lead to the system being penalised by its designers, perhaps through a training process that can intervene directly on its internal mechanisms.

In the context of Risks from Learned Optimization, deceptive alignment occurs when a system internally has a goal other than what we would like it to be. If this system is aware that it might be shut down or altered if it revealed this discrepancy, it is incentivised to play along, i.e. behave as though it is optimising the humans’ goal. Once the training process is complete and the system is safe to pursue its own goal, it does so without repercussions.

As described in the paper, deception is an inner misalignment failure, i.e. in the system there exists an inner optimiser whose goals may be different to the base optimiser’s goals. If this is the case, even if we select for models which appear to seek our base objective, the objective they actually end up pursuing may be different. These systems are deceptively aligned.

In this framing, the main way we might get deceptive alignment is through a system that performs internal search. While search is a central example, I’m curious to understand more generally what the prerequisites are for deceptive alignment. I’m particularly interested in the connection with consequentialism. In a sense, deception is just one of the many strategies a consequentialist agent might use to achieve its goals.

How does the idea of optimisation matter here?

In this article, ...
"Cars and Elephants": a handwavy argument/analogy against mechanistic interpretability
by David Scott Krueger (formerly: capybaralet) — 2022-10-31T21:26:05.388Z

TL;DR: If we can build competitive AI systems that are interpretable, then I argue via analogy that trying to extract them from messy deep learning systems seems less promising than directly engineering them.
EtA - here's a follow-up: Mechanistic Interpretability as Reverse Engineering (follow-up to "cars and elephants")
Preliminaries:
Distinguish weak and strong tractability of (mechanistic) interpretability as follows:

Weak tractability: AGI-level systems are interpretable in principle, i.e. humans have the capacity to fully understand their workings in practice given the right instructions and tools. This would be false if intelligence involves irreducible complexity, e.g. because it involves concepts that are not "crisp", modular, or decomposable; for instance, there might not a crisp conceptual core to various aspects of perception or abstract concepts such as "fairness".[1]
Strong tractability: We can build interpretable AGI-level systems without sacrificing too much competitiveness.

The claim:
If strong tractability is true, then mechanistic interpretability is likely not the best way to engineer competitive AGI-level systems.
The analogy:
1) Suppose we have a broken down car with some bad parts and we want to have a car that is safe to drive. We could try to fix the car and update the parts.
2) But we also have a perfectly functioning elephant. So instead, we could try and tinker with the elephant to understand how it works and make its behavior more safe and predictable.
I claim (2) is roughly analogous to mechanistic interpretability, and (1) to pursuing something more like what Stuart Russell seems to be aiming for: a neurosymbolic approach to AI safety based on modularity, probabilistic programming, and formal methods.[2]
Fleshing out the argument a bit more:
To the extent that strong tractability is true, there must be simple principles we can recognize underlying intelligent behavior. If there are not such simple principles, then we shouldn't expect mechanistic interpretability methods to yield safe, competitive systems. We already have many ideas about what some of those principles might be (from GOFAI and other areas). Why would we expect it is easier to recognize and extract these principles from neural networks than to deliberately incorporate them into the way we engineer systems?
Epistemic ...
Superintelligent AI is necessary for an amazing future, but far from sufficient
by So8res — 2022-10-31T21:16:35.052Z

(Note: Rob Bensinger stitched together and expanded this essay based on an earlier, shorter draft plus some conversations we had. Many of the key conceptual divisions here, like "strong utopia" vs. "weak utopia" etc., are due to him.)

I hold all of the following views:

Building superintelligent AI is profoundly important. Aligned superintelligence is our best bet for taking the abundant resources in the universe and efficiently converting them into flourishing and fun and art and beauty and adventure and friendship, and all the things that make life worth living.[1]
The best possible future would probably look unrecognizably alien. Unlocking humanity’s full potential not only means allowing human culture and knowledge to change and grow over time; it also means building and becoming (and meeting and befriending) very new and different sorts of minds, that do a better job of realizing our ideals than the squishy first-pass brains we currently have.[2]
The default outcome of building artificial general intelligence, using anything remotely like our current techniques and understanding, is not a wondrously alien future. It’s that humanity accidentally turns the reachable universe into a valueless wasteland (at least up to the boundaries defended by distant alien superintelligences).

The reason I expect AGI to produce a “valueless wasteland” by default, is not that I want my own present conception of humanity’s values locked into the end of time.

I want our values to be able to mature! I want us to figure out how to build sentient minds in silicon, who have different types of wants and desires and joys, to be our friends and partners as we explore the galaxies! I want us to cross paths with aliens in our distant travels who strain our conception of what’s good, such that we all come out the richer for it! I want our children to have values and goals that would make me boggle, as parents have boggled at their children for ages immemorial!

I believe machines can be people, and that we should treat digital people with the same respect we give biological people. I would love to see what a Matrioshka mind can do.[3] I expect that most of my concrete ideas about the future will seem quaint and outdated and not worth their opportunity costs, compared to the rad alternatives we'll see when we and our descendants and creations are vastly smarter and more gr...