trentmkelly/LessWrong-43k huggingface dataset:
[Linkpost] The Story Of VaccinateCA
by hath — 2022-12-09T23:54:48.703Z
Link: https://www.worksinprogress.co/issue/the-story-of-vaccinateca/

Linked is Patrick MacKenzie's writeup of VaccinateCA, the nonprofit that succeeded at creating the best source of vaccine availability data when the government completely dropped the ball. It's long, but one of the best things I've read in a while. Some highlights:

We found it surprising that The System did not know where the vaccine was and that this fact persisted. The System allocates and ships the vaccine, after all. Devolution to various layers of government, community groups, and healthcare providers; pervasive inability to connect IT systems; and unwillingness to correct this problem meant that people in positions of authority considered the ground truth beyond the abilities of their institutions to discern.

It is not harder to track a shipment of vaccine than it is to track a package from Amazon. Full stop. We are a nation that is extremely skilled at logistics, including healthcare logistics. A pharmacy chain can calculate, within a matter of minutes, the number of bottles of aspirin it owns, broken down by address. That count will be shockingly close to physical reality. Capitalism, ho!

We chose, as a nation, that knowing the location of the vaccine was . . . just not a top priority.

As an example of places where the data chain of custody broke down, consider the (true) case where a government actor directs some vials that it controls into the University of California at XYZ hospital system. (I will elide naming the specific hospital system, but for people not familiar with California, note that there are many different academic institutions called the University of California and their names are distinguished by the city they are primarily located in.) That hospital system has one address, according to a shipping spreadsheet.

That hospital system routinely centrally receives, records, allocates, and reships all the medical supplies needed to keep a hospital system running, from saline to radiomedicine to scalpels. Then it parcels them out to the locations it provides healthcare at. Which it has more than one of and which are not a short walk from one another. I invite you to take a look at the locations list for the University of California at San Diego hospital system.

We surprised the government by telling them that the vaccine was present where they believed it was absent. After delivery was taken at the central receiving facility, the vaccine was mo

...
Prosaic misalignment from the Solomonoff Predictor
by Cleo Nardo — 2022-12-09T17:53:44.312Z

When I first read Paul Christiano's post, I figured it had little relevance to prosaic alignment. But is that true? Is Solomonoff misalignment a problem that could actually arise on software running on GPUs over the next 10 years?

The Solomonoff Predictor is malign.

FACT 1: There's a certain type of machine P called a predictor. You tell P a bunch of facts E and then ask it a question H. The predictor will then output the probability of H given E.

FACT 2: The optimal predictor is the Solomonoff Predictor S(H|E), for some natural sense of optimality.

What's the Solomonoff Predictor? Imagine every possible world is a binary string generated by a computer program, and imagine that the prior likelihood of a program m is 2−length(m). The Solomonoff Predictor corresponds to that prior.

In other words, S(H|E)=S(H∧E)S(E)

where S(A)=∑programs m with output xm2−length(m)×1A(xm)

Fact 3: Paul Christiano worries that the Solomonoff Predictor is malign. Why? Because some of these computer programs m will simulate "gremlins". These gremlins are consequentialist agents who care about influencing the output of S, and they can influence the output of S by making H true in their universe.

Let's give a concrete example.

Suppose Alice finds a mysterious box with a big red button. She suspects the box makes delicious ice-cream, and so she feeds into the oracle S all the facts she's ever observed including all the facts about the box. Let's call all that data E. And then she asks the oracle the question "is the box an ice-cream maker?". Let's call that question H. If the oracle outputs a high probability, then she'll press the button anticipating delicious ice-cream.

Let's also suppose that unbeknownst to Alice, the box is actually a gremlin-generator that would unleash a bunch of gremlins.

Okay, what would Solomonoff Predictor output?

Well, there's going to exist some programs m for which all the following facts are true:

m simulates a universe containing gremlins.
These gremlins know that Alice in our universe is asking her oracle the question H.
These gremlins know that if Alice's oracle outputs a high probability for H then Alice will press the button, unleashing gremlins into our universe. This is the gremlins' de

...
Take 8: Queer the inner/outer alignment dichotomy.
by Charlie Steiner — 2022-12-09T17:46:26.383Z

As a writing exercise, I'm writing an AI Alignment Hot Take Advent Calendar - one new hot take, written every day for 25 days. Or until I run out of hot takes.

I'm not saying to never say "inner alignment." But you had better be capable of not using that framing if you want to work on alignment.

The inner/outer alignment framing is from Risks from Learned Optimization. You already know what I mean, but just to go through the motions: it describes a situations where there's two key optimization processes going on: the "outer" optimization process humans are using to create an AI, and the "inner" process the created AI is using to make plans. Outer alignment is when the outer process is aligned with humans, and inner alignment is when the inner process is aligned with the outer process. This is an outstandingly useful framing for thinking about certain kinds of AI development, especially model-free RL.

However, this framing has a limited domain of validity. Sometimes it breaks down in a way that looks like adding degrees of freedom - as if humans and the created AI are two ends of a string, and the AI-optimization process is a point along the string. Then you can imagine holding the ends fixed but being able to wiggle the midpoint around.

This looks like creating an AI that's still aligned with humans, but not because it's aligned with a creation process that is itself aligned with humans - instead, both processes are imperfect, and in order to get good outcomes you had to reason about the alignment of the end-product AI directly. This is how it is for most present-day "dumb" AI, except replace "aligned" with "useful and safe." One can also see shardites as arguing that this is what we should be doing.

Other times, the inner/outer framing breaks down entirely, because there isn't a distinct two-part structure. The key example is reflectivity - using the AI to reflect on its own optimization process rapidly blurs the lines between what's "inner" or "outer."

But it's not just obvious reflectivity, sometimes the breakdown seems like it was in the problem statement the whole time. Often when people try to solve one of inner or outer alignment entirely, they find that they've sneakily had to solve the other problem as well. In order to "really solve" outer alignment, you want the AI-optimization process to care about the generalization properties of the created AI beyond...
Does a LLM have a utility function?
by Dagon — 2022-12-09T17:19:45.936Z

There's a lot of discussion and research into AI alignment, almost always about variants of how to define/create a utility function (or meta-function, if it changes over time) that is actually aligned with ... something. That something is at least humanity's survival, but often something like flourishing or other semi-abstract goal. Oops, that's not my question for today.
My question for today is whether utility functions are actually part of the solution at all. Humans don't have them, the most interesting spurs toward AI don't have them. Maybe anything complicated enough to be called AGI doesn't have one (or at least doesn't have a simple, concrete, consistent one).
Monthly Roundup #1
by Zvi — 2022-12-09T17:10:00.447Z

When I split entirely non-medical, non-Covid topics off from the weekly Covid post, I expected to have a grab bag of them on a weekly basis. Hence the name ‘weekly roundup.’

What changed were four things.

The flow of worthy one-off items decreased and they seemed less time sensitive.

The flow of larger topics in the news increased, including AI, FTX and Twitter.

It became increasingly clear more focused posts have higher long term value.

I started spending more time exploring longer-term questions.

The result of this was that it no longer made sense to do a full weekly roundup. It still does make sense to do this periodically, so for now I am going to move these posts to monthly. I may spin off some of the sections (e.g. the FTX one) as part of future posts, but have no present intention to do so.

Bad News

Business Insider reportedly claims that as long as they say ‘reportedly’ in their headline, they are in the clear even if they know the thing ‘reportedly’ claimed is absolutely false, because it is true that it was reported. Journalism.

Germany to reduce rather than increase military spending, despite government promises to the contrary.

European court decides Right to Privacy too important, takes down registers of beneficial owners.

I am generally skeptical of predictions such as Samo’s that Europe might soon be discussed as a ‘formerly industrialized economy’ and yet there is a growing movement in French towns to ‘bring back the horse and cart as an alternative to fossil fuels and a way to slow down urban life.’

The UK effectively charges a very large tax for access to its postcode address file, making it out of reach for many uses and also being effectively a large tax on business, especially small business, that requires the information. This seems like quite an insane place to collect this much government revenue.

Lucky person uses social position to warn others of punishment for focusing on luck.

Yes, the government will always use all the data available, such as using your treatment of NPCs in games like Skyrim and Fallout to predict future psychopaths.

IRA’s Made in America requirements going exactly how you would expect, except for the whole ‘excluding allies was a typo, sorry’ incident.

I was expecting In Defense of Being Alone to center around being alone being kind of awesome these days. All the games and books, all th...
Working towards AI alignment is better
by Johannes C. Mayer — 2022-12-09T15:39:08.348Z

There are people not motivated to solve AI alignment, who do work related to AI alignment. E.g. people work on adversarial robustness, understanding how to do science mechanically, or on advancing other paradigms that are more interpretable than modern ML. These people might be interested in the science, or work on it for some other personal reason. They probably will do a worse job, compared to, if they would try to advance AI alignment, even when they work on something that is useful for AI alignment.

This basic idea was mentioned by Buck in a talk.

The following is a list of reasons why somebody who tries to solve alignment directly, would be better at solving alignment (though this sentence alone makes it sound obvious):

They are more likely to switch directions once they realize that they could be doing something better with their time, to make progress on AI alignment.

E.g. somebody who is interested in type theory and then learns that they can help AI alignment might be excited to help, but when there is lots of evidence that they should do something that does not involve type theory, they will keep sticking to doing things with type theory until the end, because their interest in type theory outweighs their desire to advance AI alignment.

The path that they take to solve the problem might look very different.

It is less likely that they take unpromissing but interesting sidetracks.
Simplifications that they make to the problem decrease the value of a solution less, in expectation.
In general, if there are multiple ways to solve the problem, the solution we end up with will likely be more relevant for alignment.

They can employ the full power of their consequentialist reasoning and be agentic about what to do, without starting to goodhart.

E.g. if you just let them do whatever, they are likely to discover things that are useful that you did not think of before. If somebodies main objective is not to solve AI alignment, it is likely that they will follow whatever looks best to their real motivation, as long as they can find some plausible explanation for why this is useful for AI alignment so that they have an excuse (in the case where they are payed to work on this to advance AI alignment).

There are probably many more points I have not thought of. How much you want to solve alignment compared to other things is a spectrum. What you care about might naturally dr...
You can still fetch the coffee today if you're dead tomorrow
by davidad — 2022-12-09T14:06:48.442Z

"You can't fetch the coffee if you're dead."
—Stuart Russell, on the instrumental convergence of shutdown-avoidance

Note: This is presumably not novel, but I think it ought to be better-known. The technical tl;dr is that we can define time-inhomogeneous reward, and this provides a way of "composing" different reward functions; while this is not a way to build a shutdown button, it is a way to build a shutdown timer, which seems like a useful technique in our safety toolbox.

"Utility functions" need not be time-homogeneous

It's common in AI theory (and AI alignment theory) to assume that utility functions are time-homogeneous over an infinite time horizon, with exponential discounting. If we denote the concatenation of two world histories/trajectories by ⊳, the time-consistency property in this setting can be written as

∀h1,h2.U(h1⊳h2)=U(h1)+γLength(h1)⋅U(h2)

This is property is satisfied, for example, by the utility-function constructions in the standard Wikipedia definitions of MDP and POMDP, which are essentially[1]

U(h)=∑t∈NγtR(h(t))

Under such assumptions, Alex Turner's power-seeking theorems show that optimal agents for random reward functions R will systematically tend to disprefer shutting down (formalized as "transitioning into a state with no transitions out").

Exponential discounting is natural because if an agent's preferences are representable using a time-discount factor that depends only on relative time differences and not absolute time, then any non-exponential discounting form is exploitable (cf. Why Time Discounting Should Be Exponential).

However, if an agent has access to a clock, and if rewards are bounded by an integrable nonnegative function of time, the agent may be time-inhomogeneous in nearly arbitrary ways without actually exhibiting time inconsistency:

U(t0,h)=∞∑t=t0Rt(h(t))

Any utility function with the above form still obeys an analogous version of our original time-consistency property that is modified to index over initial time t0:

∀h1,h2,t0.U(t0,h1⊳h2)=U(t0,h1)+U(t0+Length(h1),h2)

Note that time-homogenous utility functions are a special case in which U(t,h)=γtU(0,h).

Time-bounded utility functions can be sequentially composed

We define a time-bounded utility function as a dependent tuple

(τ:N,R:N<τ→(S×A)→R)

i.e., a family of utility functions indexed by times within a given fixed range. The intended semantics...
ChatGPT's Misalignment Isn't What You Think
by stavros — 2022-12-09T11:11:43.318Z

ChatGPT can be 'tricked' into saying naughty things.

This is a red herring.

Alignment could be paraphased thus - ensuring AIs are neither used nor act in ways that harm us.

Tell me, oh wise rationalists, what causes greater harm - tricking a chatbot into saying something naughty, or a chatbot tricking a human into thinking they're interacting with (talking to, reading/viewing content authored by) another human being?

I can no longer assume the posts I read are written by humans, nor do I have any reasonable means of verifying their authenticity (someone should really work on this. We need a twitter tick, but for humans.)

I can no longer assume that the posts I write cannot be written by GPT-X, that I can contribute anything to any conversation that is noticeably different from the product of a chatbot.[1]

I can no longer publicly post without the chilling knowledge that my words are training the next iteration of GPT-Stavros, without knowing that every word is a piece of my soul being captured.

Passing the Turing Test is not a cause for celebration, it's a cause for despair. We are used to thinking of automation making humans redundant in terms of work, now we will have to adapt to a future in which automation is making humans redundant in human interaction.

Do you see the alignment failure now?

^

💤 Tell HN: ChatGPT can reply like a specific Reddit or HN user, including you | Hacker News
ML Safety at NeurIPS & Paradigmatic AI Safety? MLAISU W49
by Esben Kran — 2022-12-09T10:38:34.693Z
Link: https://newsletter.apartresearch.com/posts/ml-safety-at-neurips-paradigmatic-ai-safety-w49

Watch this week’s episode on

YouTube

or listen to the audio version

here

.

This week, we see how to break ChatGPT, how to integrate diverse opinions in an AI and look at a bunch of the most interesting papers from the ML safety workshop happening right now!

ChatGPT jailbreaks

Last week, we reported that ChatGPT has been released along with text-davinci-003. In the first five days, it received over a million users, a product growth not seen in a long time. And if that wasn’t enough, OpenAI also released WhisperV2 that presents a major improvement to voice recognition.

However, all is not safe with ChatGPT! If you have been browsing Twitter, you’ll have seen the hundreds of users who have found ways to circumvent the model’s learned safety features. Some notable examples include extracting the pre-prompt from the model, getting advice for illegal actions by making ChatGPT pretend or joke, making it give information about the web despite its wishes not to and much more. To see more about these, we recommend watching

Yannic Kilchers video

about the topic.

Rebecca Gorman and Stuart Armstrong

found a fun way to make the models more safe albeit also more conservative, by running the prompt through an Eliezer Yudkowsky-simulating language model prompt. You can read more about this in the description.

Responsible AGI Institutions

ChatGPT is released on the back of OpenAI releasing their alignment strategy which we reported on a few weeks ago.

Bensinger publishes

Yudkowsky and Soares’ call for other organizations developing AGI to release similar alignment plans and commends OpenAI for releasing theirs, though they do not agree with its content.

The lead of the alignment team at OpenAI has also published a

follow-up on his blog

about why he is optimistic about their strategy. Jan Leike has five main reasons: 1) AI seem favorable for alignment, 2) we just need to align AI strong enough to help us with alignment, 3) evaluation is easier than generation, 4) alignment is becoming iterable and 5) language models seem to become useful for alignment research.

Generating consensus on diverse human values

One of the most important tasks of value alignment is to understand what “values” mean. This can be done from both a theoretical (such as shard theory) and an empirical view. In this new DeepMind paper, they train a language model to take in diverse opinion...
What are your thoughts on the future of AI-assisted software development?
by RomanHauksson — 2022-12-09T10:04:55.473Z

The flurry of news about ChatGPT (and my recent experimentation with GitHub Copilot) has got me thinking about the effect natural language models are having on software engineering. AI-assisted development is improving at a blistering pace, but not just the models themselves – people are figuring out ways to use AI everywhere from explaining exceptions to writing terminal commands.

I want to hear people's thoughts on where this is going. What might software development actually look like in 3 years, concretely? Who knows, maybe it will be replaced by prompt engineering. And will this decrease the demand for technology workers, or increase it? AI capabilities are difficult to predict, so I'm also curious about what devs should start learning to stay ahead of automation.
Fear mitigated the nuclear threat, can it do the same to AGI risks?
by Igor Ivanov — 2022-12-09T10:04:09.674Z

It seems that only a small minority of the general public realizes that misaligned and powerful AIs might be a serious threat to the world as we know it, and the problems might happen not that far into the future.

Nobody wants humanity to go extinct. People just need to feel the risks posed by powerful but misaligned AIs. At the moment for most people, such risks are some vague abstraction, and vague abstractions are not scary.

So, I asked myself the question: What can be done to raise awareness about the risks related to powerful misaligned AIs?

I might have a piece of the answer.

The Cold War and the nuclear threat

Let's discuss the original existential threat to humanity. The nuclear war.

During the Cold War, many people believed that there was a real possibility of nuclear war, but at first, countries just produced more and more nuclear charges and developed new ways to deliver them. Obviously, politicians and military leaders were aware of the consequences of a possible nuclear conflict, but for some time they weren't afraid enough to cooperate to mitigate that threat.

I want to show several important points from that period which gave me hints on possible ways to raise awareness about misaligned AIs.

The Cuban Missile Crisis

In 1962 USSR sent nuclear missiles to Cuba, so they could rapidly reach the USA mainland in case of a war. As a response, the USA demanded to remove those missiles from Cuba, established the naval blockade of the island, and started preparations for an invasion. These events were named the Cuban Missile Crisis.

Due to mutual distrust, carelessness from all sides, unfortunate accidents, and the lack of communication, the situation rapidly started to spiral out of control and the possibility of an exchange of nuclear strikes became more and more probable, and it seriously affected decision-makers on all sides.

At the height of the crisis, US president Kennedy described the situation as "this looks like hell, looks real mean", and referred to the situation as "Horror"[1]. His advisor took a note "Those minutes were the time of the greatest worry by the president. His hand went up to his face and covered his mouth and he closed his fist".[1]

We know much less about the emotional reactions of the Soviet leaders, but a couple of days after the end of the crisis, Soviet premier Khrushchev wrote Kennedy a letter:
"... We want to conti...