trentmkelly/LessWrong-43k huggingface dataset:
Some Summaries of Agent Foundations Work
by mattmacdermott — 2023-05-15T16:09:56.364Z

This is a bunch of not-very-technical summaries of agent foundations work from LessWrong and the Alignment Forum.

I was hoping to turn it into a comprehensive literature review, categorising things in an enlightening way and listing open problems. It turns out that's pretty hard to do! It's languished in my drafts for long enough now, so I'm just going to post it as it is. Hopefully someone else can produce a great overview of the field instead.

Why Agent Foundations?

My own answer to this question is that most AI threat models depend on a powerful agent pursuing a goal we don't want it to, and mathematical models of agency seem useful both for understanding and dealing with these risks. Existing models of agency from fields like reinforcement learning and game theory don't seem up to the job, so trying to develop better ones might pay off.

Normative and Descriptive Agent Foundations

One account of why our usual models of agency aren't up to the job is the Embedded Agency sequence - the usual models assume agents are unchanging, indivisible entities which interact with their environments through predefined channels, but real-world agents are a part of their environment. The sequence identifies four rough categories of problems that arise when we switch to trying to model embedded agents, explained in terms of Marcus Hutter's model of the theoretically perfect reinforcement learning agent, AIXI.

I find these subproblems to be a useful framing for thinking about agents foundations, but as I explained in a previous post, I think they take a strongly normative stance, asking what high-level principles an agent should follow in order be theoretically perfect. Some other agent foundations takes a more descriptive stance, asking what mechanistic and behavioural properties agents in the real world tend to have. You could also call this a distinction between top-down and bottom-up approaches to modelling agency.

Here are the problems from the Embedded Agency sequence:

Normative Subproblems

Decision Theory. AIXI's actions affect the world in a well-defined way[1], but embedded agents have to figure out whether they care about the causal, evidential, or logical implications of their choices.
Embedded World-Models. AIXI can hold every possible model of the world in its head in full detail and consider every consequence of its actions, but embedded agents are part of the wor

...
Facebook Increased Visibility
by jefftk — 2023-05-15T15:40:02.135Z

In January I got a strange notification from Facebook telling me that
I had "increased visibility". It explained:

This was a weird notification: it only showed up on my phone, not on
desktop. Probably some sort of experiment?

Every week or so I'd get another notification telling me that I was
still getting increased visibility, but that it was conditional on
continuing to post things people were interested in.

I mostly ignored it, since I'm already very set in how I post on
Facebook, and it was hard to tell how much of an effect it was having.

On March 25th I got another notification, saying I wasn't being
interesting enough: "Engagement has slowed. See how you can get
increased visibility".

I was curious about what effect this had, but I didn't seem to have a
way to tell. Recently I remembered that I had turned
on professional
mode on my profile, which meant that if I went into "Professional
dashboard / Insights / Content" I could see my last three months of
posts with their "reach" and "engagement" counts. Here are the
plots:

I've highlighted March 25th as a red square, being the day when I got
the notification that I no longer had "increased visibility".

It doesn't seem like this was doing very much? Average "reach" (the
number of people FB showed my posts to) was 802 before the 3/25
notification and 713 after.

This isn't a perfect experiment: after getting the notification I
thought it would be a good time to post some more boring ("not of
general interest") things I'd been thinking about for a while, though
as usual I wasn't very good at predicting what people would be
interested in and some of those posts got a
lot of comments.

Overall, I think I'm better off continuing to write what I want
without thinking much about how it will affect how many people might
see what I write in the future, but "how does Facebook decide what to
show" is an interesting question.

Comment via: facebook, mastodon
Un-unpluggability - can't we just unplug it?
by Oliver Sourbut — 2023-05-15T13:23:12.543Z
Link: https://www.oliversourbut.net/p/un-unpluggability-can-t-we-just-unplug-it

My thanks to Sam Brown for feedback on readability and ordering

Can't we just unplug it?

Cover photo by Kelly Sikkema on Unsplash

A few weeks ago I was invited to the UK FCDO to discuss opportunities and risks from AI. I highly appreciated the open-mindedness of the people I met with, and their eagerness to become informed without leaping to conclusions. One of their key questions was, perhaps unsurprisingly, 'If it gets too dangerous, can we just unplug it?'. They were very receptive to how I framed my response, and the ensuing conversation was, I think, productive and informative[1]. I departed a little more optimistic about the prospects for policymakers and technical experts to collaborate on reducing existential risks.

Here I'll share the substance of that, hoping that it might be helpful for others communicating or thinking about 'systems being hard to shut down', henceforth 'un-unpluggability'[2]. None of this is especially novel, but perhaps it can serve as a reference for myself and others reasoning about these topics.

This contrasts pretty strongly with a more technical discussion of 'off switches' and instrumental convergence, handled admirably by e.g. Rob Miles and MIRI, which is perhaps the reflex framing to reach for on this question (certainly my mind went there briefly): absent quite specific and technically-unsolved corrigibility properties, a system will often do better at an ongoing task/intent if it prevents its operator from shutting it down (which gives rise to an incentive, perhaps a motive, to avoid shutdown). This perspective works well for conveying understanding about some parts of the problem, but in my case I'm pleased we dwelt more on the mechanics of un-unpluggability rather than the motives/incentives (which are really a separate question).

Both perspectives are informative; consider what you are trying to learn or achieve, and/or who your interlocutors/audience are.

Un-unpluggability factors

Broadly, I'll discuss six classes of property which can make a system less unpluggable[2:1]; with each, some analogous examples[3], a note on applicability to AI/AGI, and a gesture at mitigation.

In brief

Rapidity and imperceptibility are two sides of 'didn't see it coming (in time)'

Robustness is 'the act itself of unplugging it is a challenge'

Dependence is 'notwithstanding harms, we (some or all of us) benefit from its continued operatio

...
Can we learn much by studying the behaviour of RL policies?
by AidanGoth — 2023-05-15T12:56:25.769Z

Economists sometimes study revealed preferences, which are preferences that we can infer from choices, e.g. when given the choice between an apple or an orange, if I choose an apple, then I have revealed a preference for an apple over an orange. I'm wondering about the revealed preferences of RL policies (applying behavioural econ / experimental econ to RL policies). We can elicit revealed preferences from RL policies by observing their actions following various histories and we can see whether the revealed preferences satisfy various decision theoretic axioms.

Revealed preferences don’t tell us anything about the inner workings of an agent but they can tell us whether or not an agent is acting as if they’re following particular decision theories. We can ask questions such as:

In deterministic environments, under what conditions do RL policies exhibit behaviour that can be represented by a utility function?
In indeterministic environments, under what conditions do RL policies exhibit behaviour that is consistent with the axioms of expected utility theory (EUT)?
What do RL agents' revealed preferences / utility functions (if applicable) look like under distributional shifts and/or computational constraints?
How does RLHF affect revealed preferences?

These seem like pretty natural questions to ask, so I'm wondering what existing work there is on related questions and how promising this kind of work could be. Knowing if a policy is consistent with the axioms of EUT seems helpful, but maybe not that helpful, since this isn't sufficient for the system to be actually internally maximising expected utility with respect to some utility function and since not behaving in a way consistent with EUT isn't sufficient for being safe.

I imagine this kind of work to be most interesting under distributional shifts and/or significant computational constraints relative to the complexity of the environment, where we might be able to learn about RL failure modes. Because it focuses only on observed behaviour though, studying revealed preferences seems to me both much less useful and much easier than understanding ML systems through mechanistic interpretability.

I'm interested in what other people think about (i) the object level questions above; (ii) existing work on these questions; (iii) how useful studying these (or similar) questions would be. I'm coming at this fro...
How I apply (so-called) Non-Violent Communication
by Kaj_Sotala — 2023-05-15T09:56:52.490Z

In a discussion elsewhere, someone asked about simple communication patterns that they could use with their partner when they're having difficulties and want to avoid causing hurt or verbalizing projections. I ended up sharing an explanation of how I try to apply so-called Non-Violent Communication (NVC), one of the most useful tools for this that I've ran across; and then I figured that I might as well also share those remarks more widely. Here's an edited version of things I ended up saying in that conversation.
---
I think one of the most important parts of NVC is the idea about distinguishing observations and interpretations, where an "observation" is defined as something that you could objectively verify (e.g. by capturing it on camera) and an interpretation is something that blends in more stuff, such as generalizations or assumptions about intent the other person's intent.
For example, "You're always late" and "You don't care about my time" are interpretations, "On the last three times when we agreed to meet, you showed up 15 minutes after the agreed-upon time" is an observation.
If you can separate those, you can then go into a potentially charged conversation by transforming something like "You are always late, why don't you care about my time" to something like "On the last three times when we agreed to meet, you showed up 15 minutes after the agreed-upon time. I found that frustrating because I made sure to be on time and could have spent that extra fifteen minutes to do something else", which is often quite helpful.
This doesn't mean you'd need to keep detailed records to express things as observations. If you don't remember earlier specifics, you can just say something like "Hey you were fifteen minutes late today and I think that's happened before too". The main intents are to
1) avoid phrasings like "You're always late" or "You don't care about my time" that come across as accusatory or potentially unfair
2) be sufficiently specific that the other person understands what you're talking about - the original NVC book has an example of a person being told "You have a big mouth" after he repeatedly started telling personal stories in the middle of meetings, but the feedback wasn't effective because he didn't understand what was meant by this
As long as those goals are met, it doesn't matter even if the observations are a bit fuzzy.
NVC has this famou...
Let's build a fire alarm for AGI
by chaosmage — 2023-05-15T09:16:29.369Z

Epistemic status: This uses as a premise the post "

There is no fire alarm for AGI

" by Eliezer Yudkowsky. It will make little sense to you if you are unfamiliar with that essay or if you disagree with it.

Fear of embarrassment has been empirically shown to stop people from reacting to serious threats. A fire alarm creates common knowledge of the possible presence of a serious danger and provides an excuse to react that saves us from embarrassment. Eliezer said in 2017 that there is no such thing for artificial general intelligence. This seems to continue to be true. Let's stop accepting that state of affairs, and do something about it - let's build such a fire alarm.

This fire alarm has to have three traits:

It needs to detect the imminence of AGI. We won’t know until it’s too late how well its detection mechanism will have worked. So this will have to be designed on the best assumptions we have.
It needs to be loud: very public, very media-friendly, with as many established and friction-free channels to multipliers as possible.
It needs to be extremely easy to understand: a boolean or an integer or something similarly primitive. Details can be offered for the tiny but important minority that wants them, but we need to understand that almost all who receive the alarm won’t care about details.

The

Doomsday Clock

by the Bulletin of the Atomic Scientists provides a useful template. It takes many complicated details about nuclear weapons development and proliferation, international relations and arms control treaties, and simplifies them into a single number, the Minutes to Midnight. You’ve heard of them - and that’s the point. This alarm is 76 years old and still succeeds at getting news coverage.

There already is

ARC Evals

, which does a similar thing: they develop tools for AI labs to check whether their models are becoming dangerous. This is not the fire alarm proposed here, because it is not directed at the general public, does not create common knowledge because it is not loud enough, and therefore does not provide an excuse to seriously worry. Nevertheless, this is important work that can be built on.

It seems to me obvious that the format should be an expected time: the year, maybe the month, when to expect… what exactly? What constitutes the moment the aliens are landing? If your answer is longer than a single sentence, it is wrong.

We’re not great at pu...
From fear to excitement
by Richard_Ngo — 2023-05-15T06:23:18.656Z

This is the first post in a twelve-part sequence called Replacing Fear. The core idea of the sequence is a shift in motivations that in my experience is incredibly valuable: from fear-based motivation to excitement-based motivation. The sequence is dived into three parts (containing four posts each). The first part introduces fear-based motivation and its negative effects; the second discusses how to overcome fear by building self-trust; and the third explores how to cultivate excitement-based motivation. This post kicks things off by giving a high-level comparison of fear-based and excitement-based motivation.

Fear-based motivation is pervasive throughout society. It arises both in the obvious cases—like stage fright before a big performance or presentation—but also in a variety of other guises, both direct and indirect. For example, I think of shame as driven by fear of social rejection; of romantic jealousy as driven by fear of abandonment; and of regret as driven by fear that you’ve screwed things up permanently. Less direct forms of fear-based motivation (like guilt and envy) often invoke the fear that you’re not the type of person who deserves good things—for example, fear of being a “bad person”; fear of being unloveable; or fear of being inadequate/imperfect. In the context of relationships, I think that what people typically mean by “neediness” is best understood as “behavior driven by fear not excitement”.

By contrast, I think that actually feeling excited is rarer than people think, because getting excited about something leaves you open to the risk of losing it, which is itself a scary prospect. Most children get excited easily—but as we get older, and see more examples of excitement leading to disappointment, it becomes more difficult to disentangle excitement about getting something from fear of not getting it (or of not being the type of person who “should” get it). But why is excitement-based motivation so much better, when it’s possible? Three reasons:

Fear makes it hard to get started. Fear-based motivation is short-sighted enough to block you off from doing many things that are robustly valuable. Perhaps the best example of this is how many people are miserable because they’re single, yet never actually ask anyone out directly, because being rejected is so terrifying; or how many people don’t start a big assignment until soon before it’s

...
Reward is the optimization target (of capabilities researchers)
by Max H — 2023-05-15T03:22:49.152Z

In Reward is not the optimization target, @TurnTrout writes:

Therefore, reward is not the optimization target in two senses:

Deep reinforcement learning agents will not come to intrinsically and primarily value their reward signal; reward is not the trained agent’s optimization target.
Utility functions express the relative goodness of outcomes. Reward is not best understood as being a kind of utility function. Reward has the mechanistic effect of chiseling cognition into the agent's network. Therefore, properly understood, reward does not express relative goodness and is therefore not an optimization target at all.

I think these claims are true and important, though in my own terms, I would rephrase and narrow as:

Executing a policy trained through current reinforcement learning methods does not necessarily result in a system which takes actions to maximize the reward function.

I find the claim stated this way more intuitively obvious, but also somewhat less interesting, because the restatement makes the narrowness more explicit.

In this post, I want to highlight a fact which I did not see mentioned in the original post or the comments: in the field of reinforcement learning, there are agents which are pursuing the goal of building a system which maximizes a reward function, subject to some additional constraints. These agents are the capabilities researchers designing and implementing SotA reinforcement learning algorithms and other methods to build and test the most capable, general systems across a variety of domains.

Capabilities researchers are not solely concerned with maximizing a particular reward function, since it is not very difficult or interesting to program a bot the old-fashioned way to beat a particular Atari game. For other games (e.g. Go) it is harder to beat human or machine performance by using traditional programming techniques, and trying doesn't make for a compelling AI research agenda.

Aside from the headline metric of how well a new RL method does in terms of training policies which result in a high reward when executed, RL researchers place importance on:

Efficiency, in many senses:

How much computing power does it take to train a policy to perform well in a particular domain?
How much (and what kind of) training data does it take to train a policy?
How much computing power and space (i.e. size of the mod

...
The Lightcone Theorem: A Better Foundation For Natural Abstraction?
by johnswentworth — 2023-05-15T02:24:02.038Z

Thankyou to David Lorell for his contributions as the ideas in this post were developed.

For about a year and a half now, my main foundation for natural abstraction math has been

The Telephone Theorem

: long-range interactions in a probabilistic graphical model (in the long-range limit) are mediated by quantities which are conserved (in the long-range limit). From there, the

next big conceptual step

is to argue that the quantities conserved in the long-range limit are also conserved by resampling, and therefore the conserved quantities of an MCMC sampling process on the model mediate all long-range interactions in the model.

The most immediate shortcoming of the Telephone Theorem and the resampling argument is that they talk about behavior in infinite limits. To use them, either we need to have an infinitely large graphical model, or we need to take an approximation. For practical purposes, approximation is clearly the way to go, but just directly adding epsilons and deltas to the arguments gives relatively weak results.

This post presents a different path.

The core result is the Lightcone Theorem:

Start with a probabilistic graphical model on the variables X1,…,Xn.

The graph defines adjacency, distance, etc between variables. For directed graphical models (i.e. Bayes nets), spouses (as well as parents and children) count as adjacent.

We can model those variables as the output of a Gibbs sampler (that’s the MCMC process) on the graphical model.

Call the initial condition of the sampler X0=(X01,…,X0n). The distribution of X0 must be the same as the distribution of X (i.e. the sampler is initialized “in equilibrium”).
We can model the sampler as having run for any number of steps to generate the variables; call the number of steps T.
At each step, the process resamples some set of nonadjacent variables conditional on their neighbors.

The Lightcone Theorem says: conditional on X0, any sets of variables in X which are a distance of at least 2T apart in the graphical model are independent.

Yes, exactly independent, no approximation.

In short: the initial condition of the resampling process provides a latent, conditional on which we have exact independence at a distance.

This was… rather surprising to me. If you’d floated the Lightcone Theorem as a conjecture a year ago, I’d have said it would probab...
GovAI: Towards best practices in AGI safety and governance: A survey of expert opinion
by Zach Stein-Perlman — 2023-05-15T01:42:41.012Z
Link: https://arxiv.org/pdf/2305.07153.pdf

A number of leading AI companies, including OpenAI, Google DeepMind, and Anthropic, have the stated goal of building artificial general intelligence (AGI) - AI systems that achieve or exceed human performance across a wide range of cognitive tasks. In pursuing this goal, they may develop and deploy AI systems that pose particularly significant risks. While they have already taken some measures to mitigate these risks, best practices have not yet emerged. To support the identification of best practices, we sent a survey to 92 leading experts from AGI labs, academia, and civil society and received 51 responses. Participants were asked how much they agreed with 50 statements about what AGI labs should do. Our main finding is that participants, on average, agreed with all of them. Many statements received extremely high levels of agreement. For example, 98% of respondents somewhat or strongly agreed that AGI labs should conduct pre-deployment risk assessments, dangerous capabilities evaluations, third-party model audits, safety restrictions on model usage, and red teaming. Ultimately, our list of statements may serve as a helpful foundation for efforts to develop best practices, standards, and regulations for AGI labs.

I'm really excited about this paper. It seems to be great progress toward figuring out what labs should do and making that common knowledge.

(And apparently safety evals and pre-deployment auditing are really popular, hooray!)

Edit: see also the blogpost.
Why don't quantilizers also cut off the upper end of the distribution?
by Alex_Altair — 2023-05-15T01:40:50.183Z

It seems to me that the main goal of quantilization is to reduce the extreme unintended outcomes of maximizing (by sampling from something like a human-learned distribution over actions) while still remaining competitive (by sampling from only the upper quantile of said distribution).

But that still leaves open the possibility of sampling one of those super highly maximized actions! Why not just hard-cut off the upper portion of the distribution as well? Why not have a quantilizer that samples between the 90%ile and the 99%ile?

Or, while we're at it, why are we sampling at all? Why not just say that we're taking the 99%ile action?