trentmkelly/LessWrong-43k huggingface dataset:
How much should we worry about mesa-optimization challenges?
by sudo — 2022-07-25T03:56:14.691Z

This post was written quickly, lest I do not write it at all.

Picture the following scenario.

Humans train a model, M, with the intention for M to minimize a loss function L.
The model, M, will now take a set of actions.

I see this going wrong in two ways.

It is possible that L is malformed (misaligned, specifically), such that effectively decreasing L kills everyone. This is the classic paperclip maximizer scenario. We currently do not know how to design L such that this does not happen.
Even if L is not malformed, the set of actions taken by M might be catastrophic. This is the mesa-optimization problem.

The first failure case has captured most of my attention. Meanwhile, I have been somewhat dismissive of the second failure case.

I would like to explain why I was dismissive of the mesa-optimization problem, and make an argument for why I think we should in fact take it seriously.

--

We understand that M is an optimizer. However, we can also assume that M is not a perfect optimizer. On out-of-distribution data, M is likely to fail to optimize L.

We can define a new loss function, L', which M actually does perfectly optimize for. We define L' such that the more resources M has, the more effective M will be in decreasing L'.

L' is not taken from "human-designed objective function" space. In fact, my intuition states that L' is likely to look very strange and complex. If we were to attempt to extract the utility function from a heavily intelligence-enhanced human based on their actions, I doubt that such a utility function would seem simple either. This intuition made me initially dismissive of mesa-optimization being a problem.

Despite having read Omuhundro's [AI Drives](https://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf) paper, there did not seem to me as if there was any obvious reason why we should assume the sort of strange L'-like objective functions to suffer from instrumental convergence. One can certainly imagine many objective functions that do not lead to these drives. One could even imagine an objective function which rewards having less resources, skill, or rationality.

It might be the case that most utility functions sampled from the space of all possible utility functions converge to having these drives, but that did not and does not seem like an obviously true fact to me.

--

I can't find the post, but ...
Does agent foundations cover all future ML systems?
by Jonas Hallgren — 2022-07-25T01:17:11.841Z

Epistemic Status: Wondering over meta approaches to reasoning about agent foundations

In mathematics, we can find general principles for mathematical objects with a particular set of axioms. For example, if a language follows first-order logic, we can assume that saying the sentence "all apples are made of carbon" is the same as saying "if something isn't made of carbon, it isn't an apple". Similarly, I believe that agent foundations are an attempt to construct an abstract algebra on the algebra that is machine learning. We want to be able to say something about all future ML models by arguing over the potential compositions of ML agents. This would be like saying, "since AGI-agent 1 is a subset of an agent class that follows these axioms, we know that it will only defect if it sees a paperclip".

Someone pointed out to me that we might be assuming things about how an agent acts by not grounding it in current-day machine learning algorithms. The problem with not grounding it is that we might be constructing an abstract algebra with axioms that don't encompass the axioms in the algebra we're trying to study. I thought this was a great question, and I have no idea how to answer it since I haven't seen any formalisation on what ML approaches are a sub-set of. Take, for example, John Wentworth's work on agent foundations. Does it generalise to multi-agent-based systems? Does it generalise to self-supervision algorithms such as current-day transformers? I would love to know if anyone has thought of this.

I also have some other related questions:

Firstly, do you, fellow humans of LessWrong, believe that the abstract algebra framework is helpful for thinking about agent foundations?

Secondly, I wondered whether anyone has made such an approach or knows what modern-day ML algorithms different parts of agent foundations covers?
Unifying Bargaining Notions (1/2)
by Diffractor — 2022-07-25T00:28:27.572Z

This is a two-part sequence of posts, in the ancient LessWrong tradition of decision-theory-posting. This first part will introduce various concepts of bargaining solutions and dividing gains from trade, which the reader may or may not already be familiar with.

The upcoming part will be about how all introduced concepts from this post are secretly just different facets of the same underlying notion, as originally discovered by John Harsanyi back in 1963 and rediscovered by me from a completely different direction. The fact that the various different solution concepts in cooperative game theory are all merely special cases of a General Bargaining Solution for arbitrary games, is, as far as I can tell, not common knowledge on Less Wrong.

Bargaining Games

Let's say there's a couple with a set of available restaurant options. Neither of them wants to go without the other, and if they fail to come to an agreement, the fallback is eating a cold canned soup dinner at home, the worst of all the options. However, they have different restaurant preferences. What's the fair way to split the gains from trade?

Well, it depends on their restaurant preferences, and preferences are typically encoded with utility functions. Since both sides agree that the disagreement outcome is the worst, they might as well index that as 0 utility, and their favorite respective restaurants as 1 utility, and denominate all the other options in terms of what probability mix between a cold canned dinner and their favorite restaurant would make them indifferent. If there's something that scores 0.9 utility for both, it's probably a pretty good pick!

Although, there's something off about setting up the problem like this. There's no term for intensity of preferences! Someone who cared very little about food would have their preferences rank just as strongly as someone who had strong restaurant opinions!

In a sense, there's three responses to this objection.

The first response is that we might be zooming in too hard on the restaurant bargaining game in particular. In a broader context, a person having weak restaurant preferences may just be another way of saying that they are quick to trade off their choice of restaurant to someone else in return for other things they might desire. And so, in the broader bargaining game of a relationship where more is at stake than this one-time choice of restaurant, thi...
Reward is not the optimization target
by TurnTrout — 2022-07-25T00:03:18.307Z

This insight was made possible by many conversations with Quintin Pope, where he challenged my implicit assumptions about alignment. I’m not sure who came up with this particular idea.

In this essay, I call an agent a “reward optimizer” if it not only gets lots of reward, but if it reliably makes choices like “reward but no task completion” (e.g. receiving reward without eating pizza) over “task completion but no reward” (e.g. eating pizza without receiving reward). Under this definition, an agent can be a reward optimizer even if it doesn't contain an explicit representation of reward, or implement a search process for reward.

ETA 9/18/23: This post addresses the model-free policy gradient setting, including algorithms like PPO and REINFORCE.

Reinforcement learning is learning what to do—how to map situations to actions so as to maximize a numerical reward signal. —

Reinforcement learning: An introduction

Many people[1] seem to expect that reward will be the optimization target of really smart learned policies—that these policies will be reward optimizers. I strongly disagree. As I argue in this essay, reward is not, in general, that-which-is-optimized by RL agents.[2]

Separately, as far as I can tell, most[3] practitioners usually view reward as encoding the relative utilities of states and actions (e.g. it’s this good to have all the trash put away), as opposed to imposing a reinforcement schedule which builds certain computational edifices inside the model (e.g. reward for picking up trash → reinforce trash-recognition and trash-seeking and trash-putting-away subroutines). I think the former view is usually inappropriate, because in many setups, reward chisels cognitive grooves into an agent.

Therefore, reward is not the optimization target in two senses:

Deep reinforcement learning agents will not come to intrinsically and primarily value their reward signal; reward is not the trained agent’s optimization target.
Utility functions express the relative goodness of outcomes. Reward is not best understood as being a kind of utility function. Reward has the mechanistic effect of chiseling cognition into the agent's network. Therefore, properly understood, reward does not express relative goodness and is therefore not an optimization target at all.

Reward

...
Brainstorm of things that could force an AI team to burn their lead
by So8res — 2022-07-24T23:58:16.988Z

Comments: The following is a list (very lightly edited with help from Rob Bensinger) I wrote in July 2017, at Nick Beckstead’s request, as part of a conversation we were having at the time. From my current vantage point, it strikes me as narrow and obviously generated by one person, listing the first things that came to mind on a particular day.

I worry that it’s easy to read the list below as saying that this narrow slice, all clustered in one portion of the neighborhood, is a very big slice of the space of possible ways an AGI group may have to burn down its lead.

This is one of my models for how people wind up with really weird pictures of MIRI beliefs. I generate three examples that are clustered together because I'm bad at generating varied examples on the fly, while hoping that people can generalize to see the broader space these are sampled from; then people think I’ve got a fetish for the particular corner of the space spanned by the first few ideas that popped into my head. E.g., they infer that I must have a bunch of other weird beliefs that force reality into that particular corner.

I also worry that the list below doesn’t come with a sufficiently loud disclaimer about how the real issue is earlier and more embarrassing. The real difficulty isn't that you make an AI and find that it's mostly easy to align except that it happens to befall issues b, d, and g. The thing to expect is more like: you just have this big pile of tensors, and the interpretability tools you've managed to scrounge together give you flashes of visualizations of its shallow thoughts, and the thoughts say “yep, I’m trying to kill all humans”, and you are just utterly helpless to do anything about, because you don't have the sort of mastery of its cognition that you'd need to reach in and fix that and you wouldn't know how to fix it if you did. And you have nothing to train against, except the tool that gives you flashes of visualizations (which would just train fairly directly against interpretability, until it was thinking about how to kill all humans somewhere that you couldn't see).

The brainstormed list below is an exercise in how, if you zoom in on any part of the problem, reality is just allowed to say “lol nope” to you from many different angles simultaneously. It's intended to convey some of the difference (that every computer programmer knows) between "I can just code X" a

...
Finding Skeletons on Rashomon Ridge
by David Udell — 2022-07-24T22:31:59.885Z

A product of a SERI MATS research sprint (taking 1.5 weeks).

Cannot yet assign positively to animal or vegetable kingdom, but odds now favour animal. Probably represents incredibly advanced evolution of radiata without loss of certain primitive features. Echinoderm resemblances unmistakable despite local contradictory evidences. Wing structure puzzles in view of probable marine habitat, but may have use in water navigation. Symmetry is curiously vegetable-like, suggesting vegetable’s essentially up-and-down structure rather than animal’s fore-and-aft structure …

Vast field of study opened … I’ve got to dissect one of these things before we take any rest.

--H. P. Lovecraft, At the Mountains of Madness

Introduction

This is our MATS research-sprint team's crack at working towards the True Name (the formalization robust to arbitrary amounts of adversarial optimization power) of a model's "internal machinery."

Some weights and neurons in a model are essential to its performance, while other weights and neurons are unimportant. Our idea is that, ontologically, the True Name of a model's internal machinery is its subnetwork of important weights and neurons -- its "skeleton." This subnetwork is still a (sparser) model, but hopefully a far more interpretable model than the original.

What makes this a theoretically interesting True Name is our theory's intersection with the theory of "broad peaks" ('Rashomon ridges) in the loss landscape of overparametrized models. Rashomon ridges suggest that different overparametrized models, trained to optimality on the same task, will share a single skeleton. Furthermore, Rashomon ridges suggest a means to compute the shared skeleton of that class of models.

We were able to execute our pruning algorithm effectively by leaning on L1 regularization. L1-regularization-plus-pruning-extremely-small-weights shrunk models significantly while never unacceptably varying loss or accuracy. However, the models we trained did not obviously converge to skeletons, as measured by their losses on an off-distribution task. This means something is up with one or more of our premises: (1) Rashomon ridges are not so naively traversable and/or (2) our argument for shared internal machinery amongst models on a Rashomon ridge fails.

Loss Landscapes and Rashomon Ridges

A loss landscape is a geometric representation of how good various models are at a...
Gathering Information you won't use directly is often useful
by Johannes C. Mayer — 2022-07-24T21:21:54.877Z

The Story

Today I moved. I asked one of my housemates about good shopping opportunities for buying groceries. I already scouted two shops. They were not very good. My housemate told me that the store that they found was also not very good. I decided to go anyway. On my way there, I remembered the advice, that gathering information is only valuable if you act on it, and I did not act on the advice of my housemate. So was it useful to have gathered that information in the first place?

What's going on?

Should I have not gone to the shop? Well as it turns out the shop was much better than I expected based on the description my housemate gave me. The cashier even gave me a discount for some overripe mango. One of the other shops that also had fruit (the third one did not even have fruit) did not want to give me a discount on some overripe bananas.

Gathering the information served multiple other roles. It is often a social bonding activity. But more importantly, while gathering the information, I would get an intuitive feeling of how much that person knows what he is doing, which should inform how much I should trust his advice. So if you don't know someone well, the quickest way to get information and an evaluation of the information might be to ask for the information.

So gathering information A might make sense even if you don't act on the information A, because while gathering the information A, you might gather information B that tells you how much you should trust information A. This can go deeper than one level.
Impact of " 'Let's think step by step' is all you need"?
by yrimon — 2022-07-24T20:59:50.530Z

Title from: https://twitter.com/arankomatsuzaki/status/1529278581884432385.

This is not quite a linkpost for this paper.

Nonetheless, the abstract is:

Pretrained large language models (LLMs) are widely used in many sub-fields of natural language processing (NLP) and generally known as excellent few-shot learners with task-specific exemplars. Notably, chain of thought (CoT) prompting, a recent technique for eliciting complex multi-step reasoning through step-by-step answer examples, achieved the state-of-the-art performances in arithmetics and symbolic reasoning, difficult system-2 tasks that do not follow the standard scaling laws for LLMs. While these successes are often attributed to LLMs' ability for few-shot learning, we show that LLMs are decent zero-shot reasoners by simply adding ``Let's think step by step'' before each answer. Experimental results demonstrate that our Zero-shot-CoT, using the same single prompt template, significantly outperforms zero-shot LLM performances on diverse benchmark reasoning tasks including arithmetics (MultiArith, GSM8K, AQUA-RAT, SVAMP), symbolic reasoning (Last Letter, Coin Flip), and other logical reasoning tasks (Date Understanding, Tracking Shuffled Objects), without any hand-crafted few-shot examples, e.g. increasing the accuracy on MultiArith from 17.7% to 78.7% and GSM8K from 10.4% to 40.7% with an off-the-shelf 175B parameter model. The versatility of this single prompt across very diverse reasoning tasks hints at untapped and understudied fundamental zero-shot capabilities of LLMs, suggesting high-level, multi-task broad cognitive capabilities may be extracted through simple prompting. We hope our work not only serves as the minimal strongest zero-shot baseline for the challenging reasoning benchmarks, but also highlights the importance of carefully exploring and analyzing the enormous zero-shot knowledge hidden inside LLMs before crafting finetuning datasets or few-shot exemplars.

Consider the following image, taken from the paper:

According to the paper linked above, this technique of propmpting gpt-3 with "Let's think step by step" and then, after it thinks out loud, with "Therefore, the final answer is", increases the performance of gpt-3 very significantly. This seems to indicate that the algorithms we currently use benefit from using the i/o space as cognitive workspace. How much progress is likely to be made ...
The Most Important Century: The Animation
by Writer — 2022-07-24T20:58:55.869Z
Link: https://youtu.be/3K25VPdbAjU

This is a linkpost for the Rational Animations' video based on The Most Important Century sequence of posts by Holden Karnofsky.
Below, the whole script of the video.
Matthew Barnett has written most of it. Several paragraphs were written by Ajeya Cotra, during the feedback process. Holden Karnofsky has also reviewed the script and made suggestions. I made some light edits and additions.
Crossposted to the EA-Forum.
------------------------------------------------------------------

A very unusual time period

The celebrated science fiction author and chemistry professor Isaac Asimov once cataloged a history of inventions and scientific discoveries throughout all of human history. While incomplete, his effort still reveals something intriguing about our current era.

Out of the 694 pages in Asimov’s book, 553 pages documented inventions and discoveries since 1500, even though his book starts in 4 million BC. In other words, throughout human history, most scientific innovation has come relatively recently, within only the last few hundred years.

Other historical trends paint a similar picture.

For example, here’s a chart of world population since 10,000 BC.

For nearly all of human history up until quite recently, there weren’t very many people on Earth. It took until about 1800 for the population to reach one billion people, and just two hundred years later – a blink of an eye compared to how long our species has been around – Earth reached six billion people.

Economic historian Bradford DeLong attempted to piece together the total world economic production over the last million years. By its nature, his reconstruction of the historical data is speculative, but the rough story it tells is consistent with the aforementioned historical trends in population and technology.

In the millenia preceding the current era, economic growth – by which we mean growth in how much valuable stuff humanity as a whole can produce – was extremely slow. Now, growth is much faster.

Bradford DeLong’s data provides historians a quantitative account of what they already know from reading narratives written in the distant past. For nearly all of human history, people lived similarly to the way their grandparents lived. Unlike what we expect today, most people did not see major changes in living standards, technology, and economic production over their lifetimes.

To be sure, people were aware tha...
Hiring Programmers in Academia
by jefftk — 2022-07-24T20:20:05.683Z

A professor I know through the EA community has been trying to hire a
software engineer for their research, and they explained some
privately about how this is tricky. The following are points I took
away that might be useful to programmers considering job postings,
people in academia looking to hire programmers, and people trying to
understand why it's

hard for EA
projects to hire programmers despite there being a lot of them in EA.

Market rates for good programmers are much higher than
universities are used to paying for anything. For example, a Senior
Software Engineer at Google earns ~$350k,
while the most this professor's university would be willing to pay,
full-time, would be ~$100k.

That the money is coming from a grant doesn't resolve this: the
university would still not let you pay a higher salary because you
need to go through university HR and follow their approach to
compensation.

Universities will let you pay more when doing temporary
arrangements, for example, as an employee on a six-month term or as an
independent contractor, because you're no longer implicitly paying
partly in job security. This might allow them to get up to
$180k-$270k without benefits, though still below market rates.

Depending on university rules, if you did want to hire someone
full-time you might have to pick from a pool of internal candidates,
even if those candidates weren't very good, or else make a strong case
for why no internal candidate had the required skills. Since
programmers earning university wages could be earning maybe three
times as much if they were good enough to get hired in industry and
wanted to switch, many of them are not great.

They showed me a public job description that had a very long
list of required skills, and seemed to be targeting a more experienced
person than the work called for. Not knowing the situation, I
initially thought that this was due to a misunderstanding about what
kind of candidate they actually needed, but it's how they convinced
the university that they should be able to offer this high an hourly
rate.

While this researcher hasn't done this, others have bypassed
these restrictions by hiring someone for "40hr/wk" with an
understanding that they're actually going to be working much less than
that.

One effect of this is that if an EA academic project needs a full-time
experienced software engi...