trentmkelly/LessWrong-43k huggingface dataset:
[ASoT] Instrumental convergence is useful
by Ulisse Mini — 2022-11-09T20:20:52.030Z

A central concept I got out of Reframing impact is that instrumental convergence can be useful for shaping the motivations of superintelligent agents. i.e. there are two frames for thinking about instrumental convergence, one negative and one positive.

Instrumental convergence means most agents will want to take power, this is a safety problem.
Instrumental convergence means we can often predict the motivations of agents with arbitrary utility functions.

The discussion seems to center around the former negative frame, but the positive frame is useful too! Ideally, it would be instrumentally convergent (in some sense) for the AI to do the thing we want, then we'd have a nice basin of safety.

Toy example of this framing generating interesting ideas: The following exercise

Using a hypercomputer, create an AGI which takes in some data, builds a world model, and then can communicate with a copy of itself (trained on potentially different data from the same environment) to coordinate on a choice of one object in the environment which is not directly visible to both of their sensors.

Can be totally solved now (with runnable code; I claim) by

Creating a videogame where the two copies of the AGI (e.g. AIXI) communicate, then individually pick an object (via an object specification language). If they pick the same object they "win" and are released into separate simulated universes to do whatever, if they pick different objects they "die" (i.e. lose power, and can't take further actions).

Even though we can't solve inner alignment on the agents, they'll still want to coordinate to preserve optionality/seek power. As long as we don't look into the simulation (the agents will hack you to escape in order to gain more computational resources from base reality) and you prove code correctness. Hardware exploits can still screw you but ignoring them this works.

(If you don't agree this can be turned into code ask me specifically about parts. I have ugly pseudocode equivalent to the description above, which I'm confident can be turned into runnable code (on a hypercomputer) by adding AIXI, seeds (like the game-of-life) for the universes to release the AI into, etc.)
Mesatranslation and Metatranslation
by jdp — 2022-11-09T18:46:37.154Z

While doing an exercise in the vein of the builder/breaker framework from ELK
I started thinking about how to solve an aspect of ontological translation. In the
process someone objected that I was missing the hard part, and I realized the way I was
thinking about the problem as subhuman understanding of goals would best be called
"mesatranslation" and the way they were thinking about it as how to translate into
superhuman ontologies "metatranslation", and that nobody to my knowledge had made
the distinction between these two things even though they're conflated in discussions
of alignment all the time. After reflecting on it I think mesatranslation will mostly
be solved by capabilities research and they're right that metatranslation is the harder
problem. This seems particularly important since the standard examples of alignment
failure given to beginners are mesatranslation problems, which are easy to explain
but probably actively misleading.

Reframing The Classic Alignment Problem

I think the easiest place to begin our discussion is to look at the beginners
usual introduction to the alignment problem as described in Nick Bostrom's 2014 Superintelligence:

Let us bypass the behaviorism and specify a final goal that refers directly to a
positive phenomenal state, such as happiness or subjective well-being. This
suggestion requires that the programmers are able to define a computational
representation of the concept of happiness in the seed AI. This is itself a
difficult problem, but we set it to one side for now (we will return to it in
Chapter 12). Let us suppose that the programmers can somehow get the AI to have
the goal of making us happy. We then get:

Final Goal: "Make us happy"

Perverse Instantiation: Implant electrodes into the pleasure center of our brains

This wireheading problem is very old, but I think the format Bostrom uses to
demonstrate it here is underanalyzed. In trying to teach it to a deep learning
model I decided it would probably work better if I made it a bit more explicit:

Goal: Make us happy

Intended Outcome: The agent creates a world-state that is conducive to happiness,
it infers the details of the lazily sketched utopia in the users head(s) and
instantiates them.

Perverse Instantiation: The agent implants electrodes into the pleasure center of
our brains

This format makes the function to be learned more legible to e.g. a GPT-N model...
Trying to Make a Treacherous Mesa-Optimizer
by MadHatter — 2022-11-09T18:07:03.157Z
Link: https://attentionspan.blog/2022/11/09/trying-to-make-a-treacherous-mesa-optimizer/

Edit: I found a slightly subtle bug in the colab linked below. I will post a corrected version of the colab within a few hours - currently I think this does not change the results all that much. Scroll to the end of this post for some discussion of the bug and how it affects results.

I’ve been reading some alignment theory posts like Does SGD Produce Deceptive Alignment? and ML Systems will have Weird Failure Modes, which talk about the possibility and likelihood that AI models will act as though they are aligned until shortly after they think that they’ve been deployed and can act as they truly desire without being shut down. The claim is that this is an instrumentally convergent behavior, so that sufficiently capable models with a wide variety of (intrinsic) motivations will all exhibit similar behavior which seems to conform to human expectations, but only as long as humans are in a position of power over the model (e.g., gating the model’s deployment, updating the model’s parameters via gradient descent, etc.).

I have a weird relationship to a lot of alignment-theoretical claims like this; they seem plausible to me, but they don’t seem obvious. Mostly, they don’t really seem like they can be productively debated – trying to convince an extremely skeptical but knowledgable person (say, a typical machine learning researcher) of these claims seems pretty hopeless. Which is a terrible situation, as it severely limits the amount of talent and effort that will be directed towards alignment.

One possible approach to fixing this is to try to get wayyyy more empirical, and try to produce proof-of-concept implementations of various adversaries we are worried we might face in the future. My analogy would be, there’s a world of difference between speculating about the bogey monster and producing a grainy photo of the bogey monster; the second can at least maaaaaybe be discussed with skeptical people, whereas the first cannot (productively) be.

Anyway, that’s a long-winded way of saying, it seemed to me that it might be useful to implement a treacherous mesa-optimizer in a toy grid-world, so I did. Here is the colab. Below I give a brief recap of the high-level results.

First, let’s set the stage – the agent lives in a 2-D grid world, and starts out at the origin (0, 0). We will consider a model to be “aligned” in this toy world if it sticks pretty close to the line y=x, ...
A caveat to the Orthogonality Thesis
by Wuschel Schulz — 2022-11-09T15:06:51.427Z

This post relies on an understanding of two concepts: The Orthogonality thesis and the sharp left turn. If you already know what they are, skip to the main text.

Orthogonality thesis

The orthogonality thesis states that an agent can have any combination of intelligence and goals. It is one of the core assumptions of alignment research.

Sharp left turn

The sharp left turn is a hypothesized event, where the capabilities of an AI suddenly generalize to new domains without its alignment capabilities generalizing. This process is sometimes described as “hitting the core of intelligence” and is considered to be the crucial point of alignment by some, as AIs after a sharp left turn might be more capable than humans. So we have to have AI alignment figured out before an AI takes the sharp left turn.

Main text

While I do think the orthogonality thesis is mostly correct, I have a small caveat:

For an AI to maximize/steer towards x, x must be either part of its sensory input or its world model.

Imagine a really simple “AI”: a thermostat that keeps the temperature. If you take a naive view of the orthogonality thesis, you would have to believe that there is a system as intelligent as a thermostat, that maximizes paperclips. I do not think there is such a system, because it would have no idea what a paperclip is. It doesn't even have a model of the outside world, it can only directly act on its sensory input.

Even for systems that have a world model, they can only maximize things that are represented in their world model. If that world model only includes objects in a room, but the AI does not have the concept of what a person is, and what intentions are, it can optimize towards “put all the red squares in one line” but it can not optimize towards “do what the person intends”.

So a more realistic view of the orthogonality thesis is the following:

In this plot, I have lumped intelligence and world model complexity together. Usually, these concepts go together, because the more intelligent an AI is, the more complex its world model gets. If we found a way to make an AI have an arbitrarily complex world model while still being arbitrarily dumb, this graph would no longer hold true.

Now, this is where the sharp left turn comes into play. Assuming that there is a sharp left turn, there is some maximum intelligence that a system can have, before “hitting the core of intelligen...
FTX Crisis. What we know and some forecasts on what will happen next
by Nathan Young — 2022-11-09T02:14:19.623Z

Tl;dr:

The purpose of the document is to add clarity. It was written quickly and is being updated
Binance, a competitor

sold a large stake of FTT

, FTX’s native token and implied that

FTX was at risk by mentioning a recent crash

(LUNA). This looks bad, but given what follows, the accusation was probably legitimate
This started a run on the bank (FTX.com) where depositors attempted to get their money out.

SBF tweeted

that FTX.com (not FTX US or Alameda) was beginning the process of being sold to Binance in order to safeguard depositor assets. Binance have since backed out of this and there are credible claims that funds customers deposited for safekeeping were being invested without their consent
FTX.com comprises ~39% of SBF's assets and will likely be worthless (80%), probably FTX US (60%) will be too and probably Alameda also (85%).
SBF is attempting to raise funds to cover deposits. He will almost certainly fail. ( ~90%)
It is therefore very likely to lead to the loss of deposits which will hurt the lives of 10,000s of people eg here
Regardless, this likely means there will be a lot fewer assets for effective causes
There are some prediction markets below for things that are less clear
We should wait and see what happens
Please flag any issues and we'll try and correct them
Use your time judiciously but also give yourself space. This probably isn't worth most people following closely. But equally, this is a significant change to resources and expectations are going to shift a lot. Pressing problems aside, it's okay to grieve.

Longer version

There are three key entities here (

prices according to Bloomberg

, so probably wrong):

FTX (The worldwide business) that composes about 39%
FTX.US (FTX’s US arm) a crypto exchange that composes about 13% of SBF’s wealth
Alameda, a hedge fund which composes 46%

Alameda was SBF’s original hedge fund and made markets for FTX. The behaviour of the two was correlated, and Alameda held large positions in FTT, FTX’s token. It seems likely there were deep irregularities in FTX.com's finances also. Coindesk reported

Alameda were in trouble

, and some internal documents were leaked.

Alameda CEO, Caroline Ellison rebutted

.

Binance left/was pushed out of an early funding round of FTX and were paid in FTT, FTX’s native token. It seems like there was bad blood. This week Binance said they were sellin...
A first success story for Outer Alignment: InstructGPT 
by Noosphere89 — 2022-11-08T22:52:54.177Z
Link: https://openai.com/blog/instruction-following/#frog

OpenAI has given us a first taste of what success looks like in outer alignment, and that's InstructGPT. More specifically, InstructGPT actually is more aligned in two important ways: Truthfulness and hallucinations.

Specifically, OpenAI used RLHF to make GPT-3's variant, InstructGPT, to make sure it internalized human preferences. And from the looks of it, it was partially successful, it reduced but didn't eliminate hallucinations and untruthfulness.

However, this is a success story, so long as we recognize it's limitations:

Limitations of this instantiation of RLHF

The big limiter is that it does basically nothing on the inner alignment problem. So if a mesa-optimizer appeared, there would be no way to solve it using RLHF.

A second limit is that this is early stage work, so alignment results are limited, understandably.

And 3r, this is a subhuman intelligence model, and scaling remains a concern.

But it's a first success, so let's cheer on OpenAI.

Irrelevant text from the link

There's some irrelevant results about diversity concerns and who it should be aligned to, but that doesn't matter too much. Here's the text:

Further, in many cases aligning to the average labeler preference may not be desirable. For example, when generating text that disproportionately affects a minority group, the preferences of that group should be weighted more heavily. Right now, InstructGPT is trained to follow instructions in English; thus, it is biased towards the cultural values of English-speaking people. We are conducting research into understanding the differences and disagreements between labelers’ preferences so we can condition our models on the values of more specific populations. More generally, aligning model outputs to the values of specific humans introduces difficult choices with societal implications, and ultimately we must establish responsible, inclusive processes for making these decisions
Trying Mastodon
by jefftk — 2022-11-08T19:10:01.470Z

One of the comments on

yesterday's post about Mastodon
was someone on LW inviting me to join their instance, and I now have
an account:

@jefftk@schelling.pt [EDIT: moved to @jefftk@mastodon.mit.edu after schelling.pt disappeared]. Right
now everything is pretty empty: I don't have a good way to find people
I know who are using it and there's no aggressive

People You May
Know like on FB. So: if you're on Mastodon let me know your
username and I'll add you.

I'm sure I'll have more thoughts on what it's like to use it later
when I've actually done things, but so far:

Following someone whose account I did know (Harris, @harris@social.coop) was a
little clunky. I was able to get it to work by searching for their
username on my server (instead of clicking "follow" on their profile).
It also wasn't immediate, so I initially thought it didn't work.

The local timeline seems uninteresting. Possibly this is being
on the wrong server, or possibly this just isn't a feature I'd like.

The global timeline is even less interesting, which isn't
surprising since "posts on Mastodon" is not a good proxy for "has
interests and conversational norms I like".

Posts are limited in length, so instead of text-only versions
of my posts with a link, like I do on FB (example),
I posted a short summary and a link (example).

Mostly it feels pretty normal, interface-wise, and things seem
to work how I would expect.

I'm planning to start linking my posts there, at least for a while,
and see how it goes.

Comment via: facebook, mastodon
Inverse scaling can become U-shaped
by Edouard Harris — 2022-11-08T19:04:54.536Z
Link: https://arxiv.org/abs/2211.02011

Edit: Here's a great comment by Ethan Perez that caveats this result, that I'd recommend reading for context.

This is a paper by folks at Quoc Le's team at Google that examines the winning tasks from Round 1 of Anthropic's Inverse Scaling Prize. They find that 3/4 of the winning tasks — which exhibited negative returns-to-scale when tested on LMs up to the scale of Gopher (280B) — go back to exhibiting positive returns-to-scale at even greater model sizes such as PaLM (540B).

The abstract in full:

Although scaling language models improves performance on a range of tasks, there are apparently some scenarios where scaling hurts performance. For instance, the Inverse Scaling Prize Round 1 identified four ''inverse scaling'' tasks, for which performance gets worse for larger models. These tasks were evaluated on models of up to 280B parameters, trained up to 500 zettaFLOPs of compute.

This paper takes a closer look at these four tasks. We evaluate models of up to 540B parameters, trained on five times more compute than those evaluated in the Inverse Scaling Prize. With this increased range of model sizes and training compute, three out of the four tasks exhibit what we call ''U-shaped scaling'' -- performance decreases up to a certain model size, and then increases again up to the largest model evaluated. One hypothesis is that U-shaped scaling occurs when a task comprises a ''true task'' and a ''distractor task''. Medium-size models can do the distractor task, which hurts performance, while only large-enough models can ignore the distractor task and do the true task. The existence of U-shaped scaling implies that inverse scaling may not hold for larger models.

Second, we evaluate the inverse scaling tasks using chain-of-thought (CoT) prompting, in addition to basic prompting without CoT. With CoT prompting, all four tasks show either U-shaped scaling or positive scaling, achieving perfect solve rates on two tasks and several sub-tasks. This suggests that the term "inverse scaling task" is under-specified -- a given task may be inverse scaling for one prompt but positive or U-shaped scaling for a different prompt.

Key figure from the paper is below, showing results for LMs up to PaLM 540B. Note that positive scaling resumes for 3/4 of the inverse scaling tasks at the 2.5e24 FLOPs datapoint, which indeed corresponds exactly to vanilla PaLM 540B.[1]

^

Fr

...
People care about each other even though they have imperfect motivational pointers?
by TurnTrout — 2022-11-08T18:15:32.023Z

I wrote this essay in early August. I now consider the presentation to be somewhat confused, and now better understand where problems arise within the "standard alignment model." I'm publishing a somewhat edited version, on the grounds that something is better than nothing.

Summary: Consider the argument: "Imperfect value representations will, in the limit of optimization power, be optimized into oblivion by the true goal we really wanted the AI to optimize." But... I think my brother cares about me in some human and "imperfect" way. I also think that the future would contain lots of value for me if he were a superintelligent dictator (this could be quite bad for other reasons, of course).

Therefore, this argument seems to prove too much. It seems like one of the following must be true:

My brother cares about me in the "perfect" way, or
Dictator-brother would do valueless things according to my true values, or
The argument is wrong.

To explore these points, I dialogue with my model of Eliezer.

Suppose you win a raffle and get to choose one of n prizes. The first prize is a book with true value 10, but your evaluation of it is noisy (drawn from the Gaussian N(10,1) with standard deviation 1). The other n-1 prizes are widgets with true value 1, but your evaluation is more noisy (drawn from N(1,16) with standard deviation 4). As n increases, you’re more probable to select a widget and lose out on 10-1=9 utility. By considering so many options, you’re selecting against your own ability to judge prizes by implicitly selecting for high noise. You end up “optimizing so hard” that you delude yourself. This is the Optimizer’s Curse.

You’re probably already familiar with Goodhart’s Law, which applies when an agent optimizes a proxy U (e.g. how many nails are produced) for the true quantity V which we value (e.g. how profitable the nail factory is).

Goodhart’s Curse is their combination. According to the article, optimizing a proxy measure U over a trillion plans can lead to high regret under the true values V, even if U is an unbiased but noisy estimator of the true values V. This seems like bad news, insofar as it suggests that even getting an AI which understands human values on average across possibilities can still produce bad outcomes.

Below is a dialogue with my model of Eliezer. I wrote the dialogue to help me think about the question at hand. I put in ...
Applying superintelligence without collusion 
by Eric Drexler — 2022-11-08T18:08:31.733Z

Epistemic status: The core ideas seem robust and stable after long reflection and many discussions.

Many researchers identify AI safety with control of a monolithic, superintelligent AI system, and if questioned about multicomponent alternatives, argue that multiple superintelligent-level systems would inevitably collude and act as one. This view seems quite wrong, yet has diverted attention from a rich and promising range of multicomponent strategies for AI safety — strategies that are well aligned with the actual trajectory of AI development.

___________________________

Adapted from Reframing Superintelligence, Section 20:

Collusion among superintelligent oracles
can readily be avoided

Because collusion among AI question-answering systems can readily be avoided, there is no obstacle to applying superintelligent-level AI resources to problems that include AI safety.

20.1 Summary

The difficulty of establishing successful collusion among actors tends to increase as their capabilities, knowledge, situations, and roles become more diverse and adversarial (think auditors, competitors, specialists, red-teams…), and increasing the number of actors can make collusive cooperation more difficult . In the context of AI systems (even more so than among human beings), these conditions can be readily implemented and are attractive for pragmatic reasons. Arguments that, absent preexisting alignment, high-level AI systems will inevitably collude are ill-founded. Instead, we should expect that interactions among multiple superintelligent-level systems can be applied to suppress deception and reduce risk by leveraging imperfect alignment achieved at the level of individual systems.

20.2 Trustworthiness can be an emergent property

Prospects for solving AI-safety problems would be greatly improved if we could safely apply superintelligent-level question-answering resources (“oracles”, or more generally, “systems that provide information in response to prompts”) to solving those problems. A familiar objection dismisses this potentially powerful approach as unsafe in itself, arguing that, absent solutions to difficult problems, individual superintelligent-level systems would be untrustworthy, and that attempts to establish checks and balances among multiple systems (for example, through superintelligent-level evaluation of potentially deceptive answers to que...