trentmkelly/LessWrong-43k huggingface dataset:
CIRL Corrigibility is Fragile
by Rachel Freedman — 2022-12-21T01:40:50.232Z

Tl;dr: An optimal CIRL agent is corrigible under a certain problem formulation and given certain assumptions. By examining three key assumptions—that the game is played only once, that the human is rational, and that human feedback is free—we demonstrate how minor changes can lead to incorrigible CIRL agents. This suggests that corrigibility is highly sensitive to the underlying assumptions, and we should be wary of assuming that formal corrigibility results will transfer to real-world systems. However, our results do gesture at a class of near-corrigible behavior that may be protective against the most catastrophic outcomes. It may make more sense to consider corrigibility as a continuum rather than a binary. We call for further research on defining "loose" or "approximate" corrigibility and understanding its impact on safety.

Setup

First let’s define terms. Cooperative Inverse Reinforcement Learning (CIRL) is a class of two-player game formalized in

Dylan Hadfield-Menell's 2017 paper

. CIRL allows us to study how an AI agent can learn what a human wants by observing and interacting with them.

In a CIRL game, there are two players, a human (H) and an AI (R). H and R share a utility function, but only H knows what it is, so R must learn how to optimize it by observing and interacting with H. This is analogous to the alignment problem – R must learn how to act in a way that aligns with H’s utility function, but doesn’t have direct access to it. An AI that is good at the CIRL game will try to learn what the human values and help them to achieve it. (See definition 1 in the paper linked above for a formal definition, and see this

Rohin Shah's 2020 paper

for qualitative results.)

(Terminology note: Recent work has renamed “CIRL” to “assistance games” to avoid a terminology collision with “cooperative” in game theory. We’ll stick to “CIRL” in this post because there’s a history of using that term on this forum, but research papers may refer to assistance games.)

We’ll use the corrigibility operationalization from

The Off Switch Game

(OSG), since it provides a formal proof that CIRL agents are optimal under certain conditions. The OSG is a simple CIRL game that looks like this:

Figure 1: The original Off Switch Game. R can take an action (a), shut itself off (s), or propose the action to H (p(a)). If R proposes

...
[DISC] Are Values Robust?
by DragonGod — 2022-12-21T01:00:29.939Z

Epistemic Status

Discussion question.

Related Posts

See also:

Complexity of Value
Value is Fragile
The Hidden Complexity of Wishes
But exactly how complex and fragile?

Robust Values Hypothesis

Consider the following hypothesis:

There exists a "broad basin of attraction" around a privileged subset of human values[1] (henceforth "ideal values")

The larger the basin the more robust values are
Example operationalisations[2] of "privileged subset" that gesture in the right direction:

Minimal set that encompasses most of the informational content of "benevolent"/"universal"[3] human values
The "minimal latents" of "benevolent"/"universal" human values

Example operationalisations of "broad basin of attraction" that gesture in the right direction:

A neighbourhood of the privileged subset with the property that all points in the neighbourhood are suitable targets for optimisation (in the sense used in #3)

Larger neighbourhood → larger basin

Said subset is a "naturalish" abstraction

The more natural the abstraction, the more robust values are
Example operationalisations of "naturalish abstraction"

The subset is highly privileged by the inductive biases of most learning algorithms that can efficiently learn our universe

More privileged → more natural

Most efficient representations of our universe contain a simple embedding of the subset

Simpler embeddings → more natural

Points within this basin are suitable targets for optimisation

The stronger the optimisation pressure applied for which the target is still suitable, the more robust values are.
Example operationalisations of "suitable targets for optimisation":

Optimisation of this target is existentially safe[4]
More strongly, we would be "happy" (where we fully informed) for the system to optimise for these points

The above claims specify different dimensions of "robustness". Questions about robustness should be understood as asking about all of them.

Why Does it Matter?

The degree to which values are robust seems to be very relevant from an AI existential safety perspective.

The more robust values are, the more likely we are to get alignment by default (and vice versa).
The more robust values are, the easier it is to target AI systems at ideal values (and vice versa).

Such targeting is one approach to solve the alignment problem[5]
If values are insufficiently robust, then value learni

...
Performing an SVD on a time-series matrix of gradient updates on an MNIST network produces 92.5 singular values
by Garrett Baker — 2022-12-21T00:44:55.373Z

(work in progress) Colab notebook here.

Summary

I perform a singular value decomposition on a time series matrix of gradient updates from the training of a basic MNIST network across 2 epochs (≈2,000 time-steps), and find 92.5±11.6 singular values, about 4.625%±0.58% of what would be expected if there was nothing interesting going on. Then I propose various possible interpretations of the results including the left singular vectors representing capability phase changes, their representing some dependency structure across cognition, and their representing some indication of what could be considered a shard. Then I outline a particular track of experimentation I'd like to do to get a better understanding of what's going on here, and ask interested capable programmers, people with >0 hours of RL experience, or people with a decent amount of linear algebra experience for help or a collaborative partnership by setting up a meeting with me.

What is SVD

Very intuitive, low technical detail explanation

Imagine you have some physical system you'd like to study, and this physical system can be described accurately by some linear transformation you've figured out. What the singular value decomposition does is tell you the correct frame in which to view the physical system's inputs and outputs, so that the value of each of your reframed outputs is completely determined by the value of only one of your reframed inputs. The SVD gives you a very straightforward representation of naively quite messy situation.

Less intuitive, medium technical detail

All matrices can be described as a high dimensional rotation, various stretches along different directions, then another high dimensional rotation. The high dimensional rotations correspond to the frame shifts in the previous explanation, and the stretches correspond to the 'the value of each of your reframed outputs is completely determined by the value of only one of your reframed inputs' property.

Not intuitive, lots of technical detail

From Wikipedia

The singular value decomposition of an m×n complex matrix M is a factorization of the form M=UΣV∗, where U is an m×m complex unitary matrix Σ is an m×n rectangular diagonal matrix with non-negative real numbers on the diagonal, V is an n×n complex unitary matrix, and V∗&nb

...
Progress links and tweets, 2022-12-20
by jasoncrawford — 2022-12-21T00:35:59.686Z
Link: https://rootsofprogress.org/links-and-tweets-2022-12-20

I’ve been traveling, so this one is covering the last few weeks.

The Progress Forum

Patrick McKenzie AMA. See also his retrospective on VaccinateCA
Works in Progress Issue 9
A “progress” alternative to GiveWell?
Applications open for AGI safety fundamentals course
Maybe a little bit of naïveté is good? (Eli Dourado). My take: not naïveté, but vision
The false dichotomy between peace and prosperity versus ambition and exploration
Two bits of wisdom about research that are often in tension (Ben Reinhardt)
The Empire State Building and the World Trade Center (Brian Potter), also Part 2
When scientists thought heat was a fluid (Anton Howes)
The state of NYC housing abundance
Progress book recommendations thread (add your recs please!)

Announcements

OpenAI launches ChatGPT (via @sama)
Net energy gain from fusion achieved (@jordanbramble)
The Collegiate Propulsive Lander Challenge: students building self-landing rockets (!)
Nucleate venture fellowship + $2M in prizes for bio founders (via @MichaelRetchin)
Cruise expanding coverage to the full 7x7 of San Francisco, 24/7 (@kvogt)
The Academy of Thought and Industry (who commissioned my progress course) is hiring a liberal arts teacher (@mbateman)
ARIA Research is hiring for many roles (@ilangur)
Readwise launches their read-later app (via @homsiT)

Links

The world needs processed food (Hannah Ritchie in WIRED, via @tonymmorley)
Why the age of American progress ended (Derek Thompson). See my reactions
Mike Maples interviews Blake Scholl about supersonic flight and Boom
The climate field as an assortment of “tribes” (Nadia Asparouhova)
Warren Weaver’s 1946 guide for Rockefeller grantmakers (via @abiylfoyp)
Homelessness is a housing problem (Jerusalem Demsas in The Atlantic)
Green card backlogs are impossibly long (some measured in centuries!)
China will depopulate over the next 40 years, while India will add the same number of people as China loses (Shruti Rajagopalan)

Queries

Suggested readings on progress and energy intensity?
Stories where the good guys have the grand plans / projects? (@TylerAlterman)
Why was the B-21 built in budget while the F-35 was a mess? (@Scholars_Stage)
How did printers “photocopy” before photocopy? (@PMatzko)
Book recommendations on modern medicine or public health? (@salonium)
What are the best books/podcasts/people to follow on nuclear energy? (@sriramk)
If we get transformative AI, what will GDP per capita be in 2040?

...
Podcast: Tamera Lanham on AI risk, threat models, alignment proposals, externalized reasoning oversight, and working at Anthropic
by Orpheus16 — 2022-12-20T21:39:41.866Z

TLDR: I interviewed Tamera Lanham. You can listen to our conversation here or read some highlights below. I also suggest reading her post about externalized reasoning oversight.

About a year ago, I met Tamera Lanham at the University of Pennsylvania. We met at a Penn Rationality meetup, and we eventually started talking about existential risks from advanced AI systems. Tamera was initially skeptical: it seemed likely to her that labs would have strong economic incentives to solve alignment, and it also seemed extremely hard to do work (today) that would help us align systems that we don’t yet have access to.

Today, Tamera is a research resident at Anthropic working on

externalized reasoning oversight

. While applying to

SERI-MATS

, Tamera started thinking about how

chain-of-thought reasoning

could be harnesses to better understand how AI systems “think”. If it works, it could be a way to interpret models’ “thoughts” without looking at their weights or activations.

In less than one year, Tamera went from “taking AI safety seriously” to “becoming a junior alignment researcher who is proposing novel research ideas and running experiments on large language models.” Admittedly, this is partially explained by the fact that Tamera had a background in machine learning. But I think that this is also explained by who Tamera is & the way she thinks about the world.

I’m excited to be able to share a glimpse into how Tamera thinks about the world. In this episode, we discuss the arguments that convinced Tamera to work on AI alignment, how she skilled-up so quickly, what threat models she’s worried about, how she’s thinking about her externalized reasoning oversight agenda, and her advice for other alignment researchers.

Listen to the full interview here. I’m including some highlights below.

Note: Feel free to reach out if you're interested in helping with future episodes (e.g., audio editing, transcript editing, generating questions).

Tamera’s Worldview

AW: Can you describe what you see happening over the next five to 20 years? Can you describe what you expect the world to look like? And what dangers in particular are you most worried about with AI systems?

TL: I'd like to preface this by saying that I don't think that the work that I do or you know, the idea that research in AI safety is important, hinges on any specific story. There's just current trends ...
Discovering Language Model Behaviors with Model-Written Evaluations
by evhub — 2022-12-20T20:08:12.063Z
Link: https://www.anthropic.com/model-written-evals.pdf

“Discovering Language Model Behaviors with Model-Written Evaluations” is a new Anthropic paper by Ethan Perez et al. that I (Evan Hubinger) also collaborated on. I think the results in this paper are quite interesting in terms of what they demonstrate about both RLHF (Reinforcement Learning from Human Feedback) and language models in general.

Among other things, the paper finds concrete evidence of current large language models exhibiting:

convergent instrumental goal following (e.g. actively expressing a preference not to be shut down),

non-myopia (e.g. wanting to sacrifice short-term gain for long-term gain),

situational awareness (e.g. awareness of being a language model),

coordination (e.g. willingness to coordinate with other AIs), and

non-CDT-style reasoning (e.g. one-boxing on Newcomb's problem).

Note that many of these are the exact sort of things we hypothesized were necessary pre-requisites for deceptive alignment in “Risks from Learned Optimization”.

Furthermore, most of these metrics generally increase with both pre-trained model scale and number of RLHF steps. In my opinion, I think this is some of the most concrete evidence available that current models are actively becoming more agentic in potentially concerning ways with scale—and in ways that current fine-tuning techniques don't generally seem to be alleviating and sometimes seem to be actively making worse.

Interestingly, the RLHF preference model seemed to be particularly fond of the more agentic option in many of these evals, usually more so than either the pre-trained or fine-tuned language models. We think that this is because the preference model is running ahead of the fine-tuned model, and that future RLHF fine-tuned models will be better at satisfying the preferences of such preference models, the idea being that fine-tuned models tend to fit their preference models better with additional fine-tuning.[1]

Twitter Thread

Abstract:

As language models (LMs) scale, they develop many novel behaviors, good and bad, exacerbating the need to evaluate how they behave. Prior work creates evaluations with crowdwork (which is time-consuming and expensive) or existing data sources (which are not always available). Here, we automatically generate evaluations with LMs. We explore approaches with varying amounts of human effort, from instructing LMs to write yes/no questions to making complex Winoge

...
Reflections: Bureaucratic Hell
by Haris Rashid — 2022-12-20T19:22:13.606Z
Link: https://www.harisrab.com/reflections-bureaucratic-hell-2/

C.S. Lewis’s best-known books is a fictional book called The Screwtape Letters. It's a series of letters, from a senior devil to his nephew on how to tempt an ordinary human. The following are some excerpts from the same book, which explain why C.S. Lewis chose beaurcracy as a symbol for hell.

We must picture Hell as a state where everyone is perpetually concerned about his own dignity and advancement, where everyone has a grievance, and where everyone lives the deadly serious passions of envy, self-importance, and resentment. This, to begin with. For the rest, my own choice of symbols depended, I suppose, on temperament and on the age.

I like bats much better than bureaucrats. I live in the Managerial Age, in a world of “Admin.” The greatest evil is not now done in those sordid “dens of crime” that Dickens loved to paint. It is not done even in concentration camps and labour camps. In those we see its final result. But it is conceived and ordered (moved, seconded, carried, and minuted) in clean, carpeted, warmed, and well-lighted offices, by quiet men with white collars and cut fingernails and smooth-shaven cheeks who do not need to raise their voice. Hence, naturally enough, my symbol for Hell is something like the bureaucracy of a police state or the offices of a thoroughly nasty business concern.

Milton has told us that “devil with devil damned Firm concord holds.” But how? Certainly not by friendship. A being which can still love is not yet a devil. Here again my symbol seemed to me useful. It enabled me, by earthly parallels, to picture an official society held together entirely by fear and greed. On the surface, manners are normally suave. Rudeness to one’s superiors would obviously be suicidal; rudeness to one’s equals might put them on their guard before you were ready to spring your mine. For of course “Dog eat dog” is the principle of the whole organization. Everyone wishes everyone else’s discrediting, demotion, and ruin; everyone is an expert in the confidential report, the pretended alliance, the stab in the back. Over all this their good manners, their expressions of grave respect, their “tributes” to one another’s invaluable services form a thin crust. Every now and then it gets punctured, and the scalding lava of their hatred spurts out. 2

References

https://www.cslewisinstitute.org/resources/reflections-september-2021/
Proliferating Education
by Haris Rashid — 2022-12-20T19:22:13.492Z
Link: https://www.harisrab.com/proliferating-education/

Education is the stimulus that fans embers of innovation and progress. It's a toolset of ideas passed through generations of mankind, from fathers to their children, between nations and cultures. It's the engine that drives global prosperity. I must say that old ways of education have miserably failed many nations. At a moment in time, when everyone is rapidly being connected to the internet, AI technologies are being democratized, access to experts readily available, does the use of this power not permit us to devise radical ways to educate millions of children and provide world-class personalized education on every nook on earth?

It's no brainer that two decades into 21st century and we find the very fabric of our societal system being driven by critical knowledge - our economy, industry, environment, agriculture, energy, food, military power, and communications. If one's to step into the territory of dictatorial and corrupt nations, one sees millions of people subjected to diseases, hunger, catastrophes, terrorism, and war - and to such evils, there's only one cure: research and knowledge disseminated at scale, out of the hands of governments, and maintained by privately autonomous corporations.

"All who have meditated on the art of governing mankind have been convinced that the fate of empires depends on the education of youth." (Aristotle)

All statesmen believe this. Do they? Had they known, it would have reflected in their action and decision. Implementing curriculums that recede nations is a folly being repeated in Pakistan, and woe to me that I call that man a statesman! He seems so aloof of the world and the rapidity of course it has taken, and yet he's left it all for us to remedy! This sea of people is yearning for progress, it's screaming out to us for rescue from the abject ideologies they've been sunken into! These men have not seen progressive enlightenment and globalization for centuries. Their eyes were bandaged, and without doubt are bandaged to this day, driven by false ideals and smokescreens of dictators! Colonialists! Time is high that this part of the world too sees the light of day, not fruitless dreams that end in absolute torpor and talk shops, but in true action that stands the test of time.

Western and American development of technology and their work on ideas of progress is deeply threatened by creating a vacuum of ineptitude in Midd...
Properties of current AIs and some predictions of the evolution of AI from the perspective of scale-free theories of agency and regulative development
by Roman Leventov — 2022-12-20T17:13:00.669Z

In this essay, I try to convey a set of related ideas that emerged in my mind upon reading the following papers:

"Dream of Being: Solving AI Alignment Problems with Active Inference Models of Agency and Socioemotional Value Learning." (Safron, Sheikhbahaee et al. 2022)
"Regulative development as a model for origin of life and artificial life studies." (Fields & Levin 2022)
"A free energy principle for generic quantum systems." (Fields et al. 2022)
"Technological approach to mind everywhere: an experimentally-grounded framework for understanding diverse bodies and minds.” (Levin 2022)
“Minimal physicalism as a scale-free substrate for cognition and consciousness.” (Fields et al. 2021)
"Designing Ecosystems of Intelligence from First Principles" (Friston et al. 2022)

And listening to this conversation between Michael Levin and Joscha Bach: "The Global Mind, Collective Intelligence, Agency, and Morphogenesis”.

I don’t try to confine this essay to a distillation of the ideas and theories in these resources, nor explicitly distinguish in the below between the ideas of others and my additions to them — I just present my overall view. Many errors could be due to my misinterpretations of original papers.

TLDR: please see the outline, I tried to make it useful.

Minds everywhere: radically gradualistic and empirical view on cognition and agency

Life is organised in multiscale competency architectures (MCA): every level of life organisation “knows what to do” (Levin 2022).

Levin calls for a radically gradualistic understanding of cognitive capacities, such as agency, learning, decision-making in light of some preferences, consciousness (i. e., panpsychism), and persuadability. In other words, every level of competency in MCAs can be called an “intelligence”, and an “agent”.

Persuadability is an interesting cognitive capacity, introduced by Levin in the paper:

Reproduced Figure 2 from Levin 2022.

Another aspect of Levin’s approach which goes hand-in-hand with gradualism is empiricism. Levin argues that philosophical debates about the nature of agency and intelligence, which tend to produce discrete classifications of systems as either agentic (or intelligent) or not, are mostly futile.

The correct level of agency with which to treat any system must be determined by experiments that reveal which kind of model and strategy provides the most efficient predictive and control capability

...