trentmkelly/LessWrong-43k huggingface dataset:
Three scenarios of pseudo-alignment 
by Eleni Angelou — 2022-09-03T12:47:43.526Z

This is my second distillation of Risks from Learned of Optimization in Advanced Machine Learning Systems focusing on deceptive alignment.

As the old saying goes, alignment may go wrong in many different ways, but right only in one (and hopefully, we find that one!). To get an idea of how a mesa-optimizer can be playing the game of deceptive alignment that I explained in a previous post, we'll look at three possible scenarios of pseudo-alignment. Essentially, the question I'm trying to answer here is what is it that makes the mesa-optimizer pursue a new objective, i.e., the mesa-objective? Each of the following scenarios gives an answer to this question.

Recall that when a mesa-optimizer is deceptively aligned, it is optimizing for an objective other than the base objective while giving off the impression that it's aligned, i.e., that it's optimizing for the base objective.

Scenario 1: Proxy alignment

The mesa-optimizer starts searching for ways to optimize for the base objective. I call it "ways" but the technically accurate term is "policies" or "models"( although "models" is used for many things and it can be confusing). During this search, it stumbles upon a proxy of the base objective and starts optimizing for the proxy instead. But what does a proxy do? Proxies tend to be instrumentally valuable steps on the way towards achieving a goal, i.e., things an optimizer has to do to complete a task successfully.

To prevent the misalignment from happening, we must be in control of the search over models.

There are two cases of proxy alignment to keep in mind:

Side-effect alignment

Imagine that we are training a robot to clean the kitchen table. The robot optimizes the number of times it has cleaned the table. Wiping down the table causes the table to be clean; this is a proxy essential for pursuing the goal of cleaning the table. By doing this, the robot would score high if judged according to the standards of the base objective. Now we deploy the robot in an environment where it has a chance to spill coffee on the table right after wiping it down. There's no reason why the robot won't take that chance. It'll start spilling coffee and then continue cleaning it.

In this case, the mesa-optimizer optimizes for the mesa-objective, but this directly increases the base objective:

Instrumental alignment

Imagine we hav...
Bugs or Features?
by qbolec — 2022-09-03T07:04:09.702Z

We often laugh over human-specific "bugs" in reasoning, comparing it to a gold standard of some utility-maximizing perfect Bayesian reasoner.

We often fear that a very capable AI following strict rules of optimization would reach some repugnant conclusions, and struggle to find "features" to add to guard against it.

What if some of the "bugs" we are looking at, are actually the "features" we are looking for?

We seem to distinguish "sacred" and "non-sacred" values, refusing to mix the two in calculations (for example human life vs money). What if this "tainted bit", "NaN-propagation", is a security feature guarding against Goodharting leading to genocide or dissolution of social trust? What if utility is not a single real number, but instead a pair? What if the ordering is not even lexicographic, but partial? What if it's a much longer tuple? Which brings me to next point:
We often experience decision paralysis apparently unable to compare two actions. What if this is simply because the order must be partial for security reasons? An alternative explanation of this phenomenon is that we implicitly treat "wait for more data to arrive and/or situation to change in tie-braking way" as an action available to us - is that bad?
We often decide which of the two end-states A vs B we prefer based on the path leading to them, amusingly favoring A in some scenarios and B in others. What if this is because we implicitly assume that end-state contains our brain with the memory of the path leading there? Isn't this a cool feature to treat the agent as part of its environment? Or what if this is because we implicitly factor in considerations of "what if other society members would follow this kind of path, or decision-making algorithm?". Isn't this a cool feature to think about second-order effects, about "acausal trades", and to treat own software as perhaps shared with other agents?
At least some long-term stable cultures have norms requiring children to follow adults' advice even if it conflicts their own judgment and more importantly said children apparently follow along instead of revolting and doing what (seems) good for them. Isn't that corrigibility a feature we want from AIs we plan to rear? Shouldn't there be safe-guards against child-knowing-better-than-parent in any self-modifying system spawning new generations of itself?
The whole sunken cost fallacy/heuri

...
[Exploratory] Seperate exploratory writing from public writing
by Johannes C. Mayer — 2022-09-03T02:57:18.167Z

Disclaimer: This is an exploratory writing post. No checking for typos or other editing was done.

It seems that when I am writing, I basically almost always wirte for myself. This is fast and easy. I do find it useful to do this, for generating and fleshing out ideas. If I would just think out the ideas in my mind, the ideas would not develop as well as if I am actively trynig to flesh them out.

This type of writing seems to be different from the type of writing that you do when you have an idea and you want to explain it to other people. If you are thinking about a concept that you have already thought about a lot, then maybe you can write well enough, such that it will also provide value to other people. However, in general it might be a bad idea to try to hard to optimize for other people's understanding.

In exploratory writing the main idea is that you want to augment your thining abilities. You don't want be slowed down by needing to explain things that are very clear to you, to other people for example. I have noticed that when I am doing exploratory writing, I have never the issue of coming up with anything to say. I just keep going, in a similar way to when I would speak. If I where to take into account how well other people understand what I am saying, it will slow me down immensely.

A good strategy that I should try out it to have two seperate writing passes. First you do some exploratory writing about an idea to actually develop into something coherent, and aferwards you write up te idea again from scratch, optizing for making it understandable for other people. Doing this "destillation" of already thought out ideas should be a lot quicker than doing the exploratory writing. Even though you will be redoing the writing, this approach might be quicker overall, because both the exploratory writing as well as the explanations to other people will both be better.

Another cool strategy that I might try is to just publish the exploratory writing directly. If it is slightly useful to other people, it might be better to do this than to not publish anything at all. My current plan is to just post something to less wrong, without doing even one editig pass. So far I have noticed that editing takes 30-60% of the writing time.
We may be able to see sharp left turns coming
by Ethan Perez — 2022-09-03T02:55:45.168Z

There's a lot of discourse around abrupt generalization in models, most notably the "sharp left turn." Most recently, Wei et al. 2022 claim that many abilities suddenly emerge at certain model sizes. These findings are obviously relevant for alignment; models may suddenly develop the capacity for e.g. deception, situational awareness, or power-seeking, in which case we won't get warning shots or a chance to practice alignment. In contrast, prior work has also found "scaling laws" or predictable improvements in performance via scaling model size, data size, and compute, on a wide variety of domains. Such domains include

transfer learning

to generative modeling (on

images, video, multimodal, and math

) and

reinforcement learning

. What's with the discrepancy?

One important point is the metric that people are using to measure capabilities. In the BIG Bench paper (Figure 7b), the authors find 7 tasks that exhibit "sharp upwards turn" at a certain model size.

Naively, the above results are evidence for sharp left turns, and the above tasks seem like some of the best evidence we have for sharp left turns. However, the authors plot the results on the above tasks in terms of per-character log-likelihood of answer:

The authors actually observe smooth increases in answer log-likelihood, even for tasks which showed emergent behavior according to the natural performance metric for the task (e.g. accuracy). These results are evidence that we can predict that emergent behaviors will occur in the future before models are actually "capable" of those behaviors. In particular, these results suggest that we may be able to predict power-seeking, situational awareness, etc. in future models by evaluating those behaviors in terms of log-likelihood. We may even be able to experiment on interventions to mitigate power-seeking, situational awareness, etc. before they become real problems that show up in language model -generated text.

Clarification: I think we can predict

whether or not

a sharp left turn towards deception/misalignment will occur rather than exactly

when

. In particular, I think we should look at the direction of the trend (increases vs. decreases in log-likelihood) as signal about whether or not some scary behavior will eventually emerge. If the log likelihood of some specific scary behavior increases, that’s a bad sign and gives us some evidence it will be...
[Exploratory] Exploratory Writing Info
by Johannes C. Mayer — 2022-09-03T02:50:57.795Z

TL;DR - As an experiment I will post the unedited notes that I generate when thinking about something, to see if they are useful to people. The titles of such posts will be prefixed with '[Exploratory]'. I am not even gonna check for spelling, and I am very bad at spelling.

Compared to the average person, I might be considered a prolific writer. I wrote around 1.200.000 words in the last 4 years or so. Almost all of this was never seen by another person. Most of my writing is exploratory writing, meaning writing that is meant to augment my thought process (you don't need a computer to be a cyborg YAY). Often I don't want to spend the extra time to edit the things I write such that they become easy to parse by another person. However, it seems that it might be worth posting them anyway, in their very rough and unpolished form, at least for some of them.

I now want to perform an experiment. When I am doing some exploratory writing, then I want to just post it, almost completely unedited (I might do edits while writing it, or when I think it would be useful for me, but otherwise I will try to avoid it). This includes not even checking for spelling mistakes. And I am very bad at spelling.

If you read such a post, please upvote it if you think this provides positive value, compared to not having it, and downvote if you think it would be better to not post it (e.g. because the post is garbage and just adds noise, making it actually harder to find good content).

This post is an exploratory post. The titles of exploratory posts will be prefixed with '[Exploratory]'. Feedback on how to improve this, in the spirit that I am going for (minimizing overhead from getting to writing I have, to publishing it) is welcome. There might be things that are very low effort that would drastically improve the quality of the posts, that I have missed (in the spirit of what I am doing, I have not thought very long about how to best go about this). [Edit: These posts will also not be promotable to front-page posts, at least for now (excluding the first two posts). This might change in the future if this is useful.]
Can someone explain to me why most researchers think alignment is probably something that is humanly tractable?
by  — 2022-09-03T01:12:58.518Z

From the view of someone who mostly has no clue what they are talking about (that person being me), I don’t understand why people working in AI safety seem to think that a successful alignment solution (as in, one that stops everyone from being killed or tortured) is something that is humanly achievable.

To be more clear, if someone is worried about AI x-risk, but is also not simultaneously a doomer, then I do not know what they are hoping for.

I think there’s a fairly high chance that I’m largely making false assumptions here, and I understand that individual alignment schemes generally differ from each other significantly.

My worry is that different approaches, underneath the layers of technical jargon which mostly go over my head, all are ultimately working towards something that looks like “accurately reverse engineer human values and also accurately encode them”.

If the above characterization is correct, then I don’t understand why the situation isn’t largely considered hopeless. Do people think that would be much more doable/less complex than I do? Why?

Or am I just completely off-base here? (EDIT: by this I mean to ask if I’m incorrect in my assumption of what alignment approaches are ultimately trying to do)
Behaviour Manifolds and the Hessian of the Total Loss - Notes and Criticism
by carboniferous_umbraculum  — 2022-09-03T00:15:40.680Z

In this note I will discuss some computations and observations that I have seen in other posts about "basin broadness/flatness". I am mostly working off the content of the posts Information Loss --> Basin flatness and Basin broadness depends on the size and number of orthogonal features. I will attempt to give one rigorous and unified narrative for core mathematical parts of these posts and I will also attempt to explain my reservations about some aspects of these approaches. This post started out as a series of comments that I had already made on the posts, but I felt it may be worthwhile for me to spell out my position and give my own explanations.

Work completed while author was a SERI MATS scholar under the mentorship of Evan Hubinger.

Basic Notation and Terminology

We will imagine fixing some model architecture and thinking about the loss landscape from a purely mathematical perspective. We will not concern ourselves with the realities of training.

Let Θ denote the parameter space of a deep neural network model f. This means that each element θ∈Θ is a complete set of weights and biases for the model. And suppose that when a set of parameters θ∈Θ is fixed, the network maps from an input space Rn to an output space O. When it matters below, we will take O=Rk, but for now let us leave it abstract. So we have a function

f:Θ×Rn→O,

such that for any θ∈Θ, the function f(⋅,θ):Rn→O is a fixed input-output function implemented by the network.

Let D={(xd,yd)}Dd=1⊂Rn×O be a dataset of D training examples. We can then define a function F:Θ→OD, by

F(θ)=(f(x1,θ),…,f(xD,θ)).

This takes as input a set of parameters θ and returns the behaviour of f(⋅,θ) on the training data.

We will think of the loss function as l:OD→R.

Example. We could have D={(xd,yd)}Dd=1 , O=Rk , and

l(o1,…,oD)=12D∑d=1∥∥od−yd∥∥2.(*)

We also then define what we will call the total loss

L:Θ→R

by

L(θ)=l(F(θ))=l(f(x1,θ),…,f(xD,θ)).

This is just the usual thing: The total loss over the training data set for a given set of weights and biases. So the graph of L is what one might call the 'loss landscape'.

Behaviour Manifolds

By a behaviour manifold (see [Hebbar]), we mean a set Σ⊂Θ of the form

Σ=F−1((o1,…,oD))=...
Sticky goals: a concrete experiment for understanding deceptive alignment
by evhub — 2022-09-02T21:57:08.246Z

Thanks to Sam McCandlish for a useful conversation that prompted this post.

In “How likely is deceptive alignment?”, I lay out the case for why you might expect deceptive alignment to occur in practice for two different stories of ML inductive biases: high and low path-dependence. In this post, I want to propose an experiment that I think would shed substantial light on the high path-dependence case specifically.

In the high path-dependence case, the story for why you would get deceptive alignment is essentially as follows:

The model learns some set of proxy goals.

The model learns enough about the training process that, if it were to use that knowledge to directly optimize for what the training process is trying to get it to do, it would get better performance than just using its proxy goals.

Gradient descent modifies the model’s proxies to become long-term goals, making the model deceptive such that it starts optimizing directly for its understanding of what the training process wants and thus gets better performance.

However, as I talk about in “How likely is deceptive alignment?”, there is another, alternative option available to gradient descent for step (3): instead of turning the existing proxies into long-term goals, gradient descent could just replace them entirely with a pointer to the model’s understanding of the training objective. In my opinion, I think it is currently quite unclear which one of these gradient descent would prefer.

One way to think about these two options is that gradient descent is effectively choosing between: repurposing the existing terminal goal to care about the training objective instrumentally vs. modifying the existing terminal goal to care about the training objective terminally. Thus, we can conceptualize the question of which would be most likely to happen as: how “sticky” are an agent’s existing terminal proxies?

Now, one thing I will say is that, since the difficulty of changing the proxies into long-term goals seems to be essentially constant regardless of the complexity of the training objective—whereas the difficulty of creating a pointer to the training objective scales with that complexity—I think that, if we try to train models on increasingly complex goals, eventually deception will win almost regardless of the baseline “stickiness” level. But understanding that baseline could still be quite important, and it’...
Agency engineering: is AI-alignment "to human intent" enough?
by catubc — 2022-09-02T18:14:52.408Z

TL;DR: We have not solved the "free will" problem: i.e. we do not know to what extent consciousness, culture, biology or a myriad of other factors shape human will and intent. Yet, prediction of future behavior of self and other agent-caused actions (i.e. what the self/others intend) is the main driving force in the evolution of brains; AI technologies will parallel evolution by being increasingly aimed at human behavior prediction - because the most powerful tools humans can create are for prediction of human behavior rather than prediction of biological or physical processes. As such, humans will become targets for "agency engineering" from powerful behavior predicting technologies which can interact with our culture, political and social structures to engineer human will from the ground up; in these scenarios it's not just AI-misuse and AI-misaligned agents but also

AI-aligned

agents that could weaken or harm us. Our limited progress and understanding w.r.t. to the science of agency makes us vulnerable to this type of harm. We need solutions soon, with one option being hard-coding rights and freedoms before we might be irreversibly stripped of power.

Background: At the core of my research interest lies a simple question: If another agent (i.e. person, algorithm etc) can predict what you are going to do next (second, hour, week, year), do you still have agency? This question was one that St Thomas Aquinas asked nearly 800 years ago (in Quaestiones disputatae de veritate), namely, whether free will or freedom can exist in the presence of an all-knowing agent - in his case, the Christian god. His answer was essentially yes - but mostly because his agent was not interested in interfering with human action. In an age of behavior-predicting AIs, there may be many such powerful agents. And some will be interested in interfering with human actions. The simplest question we can pose then is how do we: "preserve human free will around powerful systems?" (thanks to Charlie Steiner for suggesting this simple summary).

The problem framing - intention is vulnerable to manipulation: By humans (not) understanding "free will" I mean not understanding to what extent each of the forces in our lives affect or cause our free actions? Historically, the philosophers reduced the entire topic to "consciousness", i.e. that our consciousness is the most important factor in deci...
Laziness in AI
by Richard Henage — 2022-09-02T17:04:43.146Z

I did a quick google search and couldn't find much in the way of "lazy ai", but it's starting to seem more and more obvious as a strategy for alignment. Here's the thought process:

-- Given almost any goal, an AI will want to obtain more power to be able to accomplish the goal, or to be more sure that it did accomplish the goal, or to prevent anyone from getting in its way of accomplishing the goal.

-- So even an AI with "human-level" intelligence will want to get more computing power, have safeguards against being shut down, and obtain resources like money and contacts.

-- Regular humans don't usually act like this[1]. This has nothing to do with their intelligence (after all, we specified "human-level" intelligence), but instead with their willpower/personality/motivation. A normal human is more concerned with comfort, more afraid of hard work, and more complacent to do what everyone else is doing: not much. These traits together can be (somewhat harshly) described as Laziness.

So maybe one further safeguard to put on an AI is to make it lazy? No lazy person has ever attempted to take over the world, after all. But lazy people are often nice, and willing to, say, answer questions.

What would this look like in actuality? Give the AI harsh utility penalties for doing hard work, especially hard work right now. For example, at any given time, half of the AI's utility function depends on not doing work above a certain threshold in the next hour (So if the AI does well, 50% of its utility function would be permanently completed in its first hour, and after its second hour, 75%, and after its third hour, 87.5%).

Is this an insanely bad idea? Would this cripple the AI of its most valuable tool? Are there other human characteristics that would possibly make sense to implant in an AI, like boredom or self-consciousness? Has anyone else written about this?

^

Most people don't look at the world and say "what do I most want?" and "how can I best achieve it?" and then spend the next sixty years working at 100% capacity to gather resources, grow in power, and bend all of humanity to their iron will in hopes that this will give them a better chance at accomplishing their task.