trentmkelly/LessWrong-43k huggingface dataset:
Does SGD Produce Deceptive Alignment?
by Mark Xu — 2020-11-06T23:48:09.667Z

Deceptive alignment was first introduced in Risks from Learned Optimization, which contained initial versions of the arguments discussed here. Additional arguments were discovered in this episode of the AI Alignment Podcast and in conversation with Evan Hubinger.

Very little of this content is original. My contributions consist of fleshing out arguments and constructing examples.

Thanks also to Noa Nabeshima and Thomas Kwa for helpful discussion and comments.

Introduction

Imagine a student in etiquette school. The school is a training process designed to get students to optimize for proper behavior; students can only graduate if they demonstrate they care about etiquette. The student, however, cares nothing about acting appropriately; they care only about eating pop rocks. Unfortunately, eating pop rocks isn't proper behavior and thus not allowed at etiquette school. If they repeatedly attempt to eat pop rocks, they will get frequently punished, which will slowly cause them to care less about eating pop rocks. However, since the student cares about eating pop rocks over their entire lifetime, they might be clever; they might act as though they care about proper behavior so they can graduate etiquette school quickly, after which they can resume their debaucherous, pop-rock-eating ways.

In this example, the student has a proxy objective and is not aligned with the objective of the etiquette school (we might call them pop-rocksy aligned). However, since the student knows how the school functions, they can act such that the school cannot distinguish their pop-rocksy alignment from etiquette alignment. The student is masquerading as an etiquette-aligned student to deceive the school into letting them graduate. We call this deceptive alignment.

More broadly, deceptive alignment is a class of inner alignment failure wherein a model appears aligned during the training process so it gets deployed[1], wherein it can “defect” and start optimizing its actual objective. If the model you’re training has a model of the training objective and process, it will be instrumentally incentivized to act as though it were optimizing the training objective, "deceiving" the training process into thinking it was aligned.

A useful framing to have is not asking “why would the model defect?” but “why wouldn’t it?” I have been trained by evolution to have high inclusive genetic fitness, but I ...
Ethics in Many Worlds
by fin — 2020-11-06T23:21:15.630Z

There seems to be a strong case for the Everett or 'many-worlds' interpretation (MWI) of quantum mechanics. The world as revealed by MWI is so grotesquely superfluous — so offensively alien — that it's natural to imagine similarly bizarre ethical implications. If MWI is true, I am constantly 'splitting' into an unfathomable number of people. Shouldn't that make some difference to what I ought to do, some of the time?

Some suggestions have already been floated. For instance — if I take the wise option in this world, does that increase the chance of another 'me' taking the unwise option? Does that mean prudential behaviour is really selfish behaviour? And shouldn't I happily take up the smallest incentive to play quantum Russian roulette, given that I know I'll only ever experience winning? Doesn't the virtually guaranteed safety of at least some 'branches' imply we should care less about mitigating risks with low probabilities and high stakes — particularly the risk of snuffing out the human flame, and our cosmic significance along with it? And so on. These are muddled thoughts, for the most part. Just remember Egan's Law, Eliezer says: it all adds up to normality.

But I can't help thinking there is a sense in which it doesn't. This is because MWI seems to imply that many more minds will exist tomorrow than exist today — or in the next ten seconds, for that matter. So the effects of our actions on even the marginally further future dwarf the effects on the marginally nearer future in importance. And that often makes a practical difference to what we ought to do. The thought isn't complicated, which makes me suspect I've missed something obvious.

To restate in a wordier way: if you're a (total, aggregative) consequentialist, what determines the value of an act is its effect on the sum total of good and bad consequences. Just as some event is no less morally significant because it happened far away, so it seems wrong to discount consequences over time. And whether you decide to apply a discount rate for future consequences makes a material difference to which acts are best today. It seems like that the branching nature of MWI has the opposite effect of a social discount rate: it seems to imply that some outcome increases in importance as it recedes in time from the present moment into the future.

Illustration

Take a toy model of MWI, where each 'branch' of the multi...
The Power & Tragedy of Names
by SocratesDissatisfied — 2020-11-06T22:50:19.884Z

“To change this rock into a jewel, you must change its name”
- Ursula K. Le Guin, A Wizard of Earthsea

TLDR: names (and labels more generally) have an outsized influence on the public perception of the movements they are applied to. This encourages individuals to overuse and misuse persuasively powerful names. But doing so depletes the persuasive power of those names, by altering their meaning, and numbing through familiarity. Whilst the benefit of name usage accrues to the individual user, the costs are spread widely across society. The use of potent names is therefore a classic tragedy of the commons situation.

The Power

The name of a thing can be exceptionally important in determining how it is perceived. A few notable examples are as follows:

The Mensheviks & the Bolsheviks: Lenin was initially outnumbered by his opponents within the Russian Social Democratic Labour Party. However when he (through dubious procedural wrangling) obtained a temporary majority on the party’s central committee, he seized a branding opportunity – labelling his minority faction “the Majority” (Russian: “Bolsheviks”) and his opponents “the Minority” (Russian: “Mensheviks”). When the Mensheviks had the poor judgment to accept this framing, they laid themselves open to potent charges of factionalism; they were a “minority” undermining the unity and strength of the “majority”, after all.
The USA PATRIOT Act and friends: the attention paid by legislators to giving their acts names with comically positive connotations, indicates their belief in the persuasive importance of those names. In the US alone, acts with names clearly selected for persuasive potency have included the “Dream Act”, the “Disclose Act”, the “HOPE Act”, the “LIFE Act, the “Affordable Care act” and the “America Competes (for America Creating Opportunities to Meaningfully Promote Excellence in Technology, Education and Science)”, amongst many others.
Antifa & Black Lives Matter: in the aftermath of this year’s anti-police violence protests, there have been numerous arguments, often from prominent media figures, to the effect that opposition to Antifa entailed support for fascism, and opposition to BLM entailed a belief that black lives did not matter. Whatever one may think of Antifa or BLM, this is an obvious confusion. If the KKK renamed themselves the Anti-Racism League, they would not cease to

...
Observe, babble, and prune
by DirectedEvolution — 2020-11-06T21:45:13.077Z

Part 1: A demonstration of deductive logic

If Violet is a mathematician, then she is smart.

If Violet is not smart, then she is not a mathematician.

If Violet is not a mathematician, then she is not smart.

If Violet is smart, then she is a mathematician.

This little poem illustrates a basic set of logical forms. In order, they are the statement, its contrapositive, the inverse, and the converse.

There are three key insights here.

Any statement implies the contrapositive, and the contrapositive implies the statement. They go together - if one holds, then the other is guaranteed to hold. They are saying the exact same thing.
A statement does

not

imply the inverse or the converse, as you can see in the poem.
The converse is the contrapositive of the inverse. That basically means that we can call If Violet is not a mathematician, then she is not smart "the statement," which will make If Violet is smart, then she is a mathematician the contrapositive. Of course, just because we've changed the "roles" of these statements doesn't mean they've become true! We've simply made an illogical statement. But if it were true - if we lived in a heaven/hell where all smart people were mathematicians - then these statements would "go together" and mean the exact same thing. And our original statement and contrapositive would become the inverse and converse, respectively. (Yes, this means that the inverse implies the converse!).

Once you know this, you can use this "logic machine" as a tool to restate informal arguments in multiple ways in order to understand it better, find the weak points, check for consistency, and see if it accords with your understanding of how the world works.

Only fair to make a victim of my own writing. In Let the AI teach you how to flirt, I wrote the following statement:

If you can get your partner to engage in their own natural flirting style, and get good at detecting it, then you can guess their intentions with much more confidence than the average person is capable of.

If somebody else wrote this post and I wanted to make sure I understood it, I might put it through the logic machine, being a bit generous about the word choice as I modify the statement into the contrapositive, inverse, and converse.

Statement: If you can get your partner to engage in their own natural flirting style, and get good at detecting it, then you can guess their intentions with much mo

...
A review of Where Is My Flying Car? by J. Storrs Hall
by jasoncrawford — 2020-11-06T20:01:55.074Z
Link: https://rootsofprogress.org/where-is-my-flying-car

Suppose you were to reach into the mirror universe, where everything is inverted, and pull out a book that is the exact opposite of Robert Gordon’s The Rise and Fall of American Growth.

Instead of being written by a scholarly economic historian steeped in the study of the past, it would be written by an engineer who has spent his career on futuristic technology. Instead of coming from a prestigious academic press, it would be self-published, with misformatted tables and a cover featuring garish three-dimensional lettering. Instead of sticking to extremely conservative predictions about future technologies, it would speculate audaciously about the limits of the possible, from nanotech to cold fusion. Instead of a sober survey of economic history in one country and time period, it would range widely through engineering, physics and philosophy, exploring the power-to-weight ratio of jet turbines in one chapter, and describing the rise of the counterculture in the next. And instead of proclaiming the death of innovation and the end of growth, it would paint a bold vision of an ambitious technological future.

That book has leapt out of the mirror universe and into an Amazon Kindle edition (priced at 𝜋 dollars): Where Is My Flying Car? A Memoir of Future Past, by J. Storrs Hall.

Hall sets out to tackle the title question: why don’t we have flying cars yet? And indeed, several chapters in the book are devoted to deep dives on the history, engineering, and economics of flying cars. But to fully answer the question, Hall must go much broader and deeper, because he quickly concludes that the barriers to flying cars are not technological or economic—they are cultural and political. To explain the flying car gap is to explain the Great Stagnation itself.

Bold futures

The most valuable thing I took away from the book was an awareness of some powerful technological possibilities.

Flying cars

Before reading the book, I had assumed that the flying car was one of those ideas that sounds good on its face, but turns out not to work or be interesting in practice. Maybe they’re inherently too hard to fly, too dangerous, or just not all that valuable. This book changed my mind, by pointing out a simple analogy: today’s system of flight has all of the inconveniences of railroads, over a century ago. Airplanes are large, mass-transit vehicles that travel only defined, scheduled routes be...
Covid 'Mink variant'
by henryaj — 2020-11-06T17:50:11.706Z

A new strain of Covid - "Covid-19 mink variant" or "Cluster 5" - has been circulating in Denmark. The Danish PM announced that twelve people are confirmed infected by the virus; Denmark has ordered the destruction of all of the country's 15million+ mink. Denmark has now imposed local lockdowns to stop the spread of the virus.

While there's no evidence that the virus causes worse disease than other strains of coronavirus, two Danish officials have suggested that the mutant is unlikely to respond to the Covid vaccines currently in development.

I'm struggling to find any more information about the new strain. Keen to know if people have thoughts on this; at first glance this has the potential to undermine all the efforts to create a vaccine and build (limited) herd immunity thus far.
Generalized Heat Engine II: Thermodynamic Efficiency Limit
by johnswentworth — 2020-11-06T17:30:43.805Z

This post continues where the

previous post

left off.

Recap from previous post: we have two sets of IID coins. One set is the “hot pool”, XH, in which each coin has probability 0.2 of being heads. The other set is the “cold pool”, XC, in which each coin has probability 0.1 of being heads. We apply transformations to X. Each transformation must be invertible and conserve the total number of heads across both pools. We want to choose such a transformation to make some of the coins (nearly) deterministically heads. We call the number of deterministic coins produced “work”, w.

Key idea from previous post: this is a compression problem. We want w of the coins to be deterministic and the transformation to be reversible, so all the information from the initial coin-state must be compressed into the non-deterministic coins in the final state. The thermodynamic flavor of the problem comes from the additional constraint: we want to compress the initial state “data” while also conserving the total number of heads.

In this post, we will make one small tweak to this setup relative to the previous post. In the previous post, “work” just meant making some coins deterministically heads; we didn’t have a strong opinion about which coins, so long as we knew which coins they were. In this post, we’ll assume that we start with some extra coins outside our two pools which are deterministically tails, and use our “work” to make those particular coins deterministically heads. This makes it a bit cleaner to separate the “moving energy around” and “moving uncertainty around” aspects of the problem, though of course those two pieces end up coupled.

This post will dig more into the optimization aspect of the problem, look at temperature as a “price” at which we can trade energy (or analogous conserved quantities) for entropy, and view the heat engine problem as arbitrage between two subsystems with different temperatures/prices. That, in turn, yields the usual thermodynamic efficiency limit. Then, we’ll point to some generalizations and applications to wrap up.

Lagrange Multipliers

Let’s think about just one of our two pools in isolation. We’ll imagine adding/removing marginal heads (analogous to energy) to/from the pool.

Recall that our initial coin distribution for each pool is maxentropic subject to a constraint on the number of heads. If we add one head to a...
Teach People to Recognize the Sound of Covid?
by jefftk — 2020-11-06T14:30:17.177Z

Yesterday I read Zvi's

11/5 Covid update which had:

Researchers claim to have a
machine learning system that can diagnose Covid-19 over the phone
by
analyzing recordings of a forced cough (pdf). They
claim sensitivity of 98.5% and specificity of 94.2%, and for
asymptomatic cases a sensitivity of 100% (?!?) and specificity of
83.2%. I'm curious to what extent errors in the test are correlated
from day to day. This can all be offered at unlimited scale for
essentially zero additional cost. So, of course, it will presumably be
illegal indefinitely because it's not as good as accurate as a PCR
test and/or hasn't gone through the proper approval process, and no
one will ever use it. Then again, if one were to somehow download or
recreate such a program and run it, who would know?

You can see how they collected data, and contribute your own cough at
opensigma.mit.edu. It
doesn't seem to me like they actually have great data on which
recordings correspond to people who have it? For example, I submitted
mine this morning, and said that I don't think I have Covid. If,
however, I later learned that I did have Covid at the time I submitted
the sample, there doesn't seem to be any way for me to tell them.

You could get better data, however, by collecting alongside regular
Covid tests. Have everyone record a sample of a forced cough when you
test them, label the cough with their results once you have it, and
you end up with high-quality labeled samples. They trained their AI
on 5,320 samples, but at current testing rates we could get 80k
samples in a single day in just Massachusetts.

It might turn out that even with higher quality data you still end up
with a test that is less accurate than the standard of care, and so
are unable to convince the FDA to allow it it. (This is an
unreasonable threshold, since even a less accurate test can be very
useful as a screening tool, but my understanding is the FDA is very
set on this point.) Is there another way we could scale out auditory
diagnosis?

Very roughly, their system is one where you take lots of samples of
coughs, labeled with whether you think they were produced by someone
with coronavirus, and train a neural network to predict the label from
the sample. What if instead of artificial neural networks, we used
brains?

People are really smart, and I suspect that if you spent some time
with...
Additive and Multiplicative Subagents
by Scott Garrabrant — 2020-11-06T14:26:36.298Z

This is the ninth post in the Cartesian frames sequence. Here, we refine our notion of subagent into additive and multiplicative subagents. As usual, we will give many equivalent definitions.

The additive subagent relation can be thought of as representing the relationship between an agent that has made a commitment, and the same agent before making that commitment. The multiplicative subagent relation can be thought of as representing the relationship between a football player and a football team.

Another way to think about the distinction is that additive subagents have fewer options, while multiplicative subagents have less refined options.

We will introduce these concepts with a definition using sub-sums and sub-tensors.

1. Definitions of Additive and Multiplicative Subagent

1.1. Sub-Sum and Sub-Tensor Definitions

Definition: C is an additive subagent of D, written C◃+D, if there exists a C′ and a D′≃D with D′∈C⊞C′.

Definition: C is a multiplicative subagent of D, written C◃×D, if there exists a C′ and D′≃D with D′∈C⊠C′.

These definitions are nice because they motivate the names "additive" and "multiplicative." Another benefit of these definitions is that they draw attention to the Cartesian frames given by C′. This feature is emphasized more in the below (clearly equivalent) definition.

1.2. Brother and Sister Definitions

Definition: C′ is called a brother to C in D if D≃D′ for some D′∈C⊞C′. Similarly, C′ is called a sister to C in D if D≃D′ for some D′∈C⊠C′.

E.g., one "sister" of a football player will be the entire rest of the football team. One "brother" of a person that precommitted to carry an umbrella will be the counterfactual version of themselves that instead precommitted to not carry an umbrella.

This allows us to trivially restate the above definitions as:

Definition: We say C◃+D if C has a brother in D and C◃×D if C has a sister in D.

Claim: This definition is equivalent to the ones above.

Proof: Trivial. □

1.3. Committing and Externalizing Definitions

Next, we will give the committing definition of additive subagent and an externalizing definition of multiplicative subagent. These definitions ...
"model scores" is a questionable concept
by Maxwell Peterson — 2020-11-06T03:19:45.196Z

A couple of years ago, while I was working on a problem of what to do with the predictions of a random forest model, I noticed that the mean of the predictions on the test set didn't match the mean of the outcomes in the training data. Let me make that more concrete; consider the following problem (isomorphic to a problem I worked on in the past):

A town has two donut shops. The donut shops are in terrible competition, and the townspeople feel very strongly about the shops. It's, like, political. You run the good shop. Every year, 6% of people switch to your competitor. You are a data-driven company, and your kid needed something to do, so they've been sitting in the corner of the shop for years scribbling down information about each visitor and visit. So now you have a bunch of data on all your customers. "Did they come in that Monday 4 weeks ago? How about the Monday after that? How many donuts did they buy each month?" and so on. Representing "They switched to the other shop forever" as 1 and "that didn't happen" as 0, we have a label vector labels of 0's and 1's, grouped into, say, one data record and label per customer-year. We want to predict who might switch soon, so that we can try and prevent the switch.

So we put your kid's notes into some predictive model, train it, and make predictions on some other set of data not used in the training (call this the test set). Each prediction will take as input a data record that looks something like (Maxwell, Male, age 30, visited twice this year), and output a continuous prediction in the range (0, 1) for how likely Maxwell is to leave. You'll also have a label, 1 or 0, for whether Maxwell actually left. After doing these predictions on the test set, you can check how well your model is modeling by comparing the true labels to the predictions in various ways. One thing to check when figuring out how well you did: check that the average number of people predicted to leave your shop in past years is close to the number of people who actually left. If there are 100 people in the test set, and only 6 actually left in any given year, you should be unhappy to see a prediction vector where all the predictions look like [0.1, 0.9, 0.5, 0.8] (imagine this list is much longer with numbers on a similar scale), because the expected number of leavers from those probabilities is (0.1 + 0.9 + 0.2 + 0.8) / 4 = 2 / 4...
Sunday, Nov 8: Tuning Your Cognitive Algorithms
by Raemon — 2020-11-06T00:57:58.815Z

For this Sunday's LW Online Meetup, I'll be will be leading an exercise on Mindful Puzzle Solving, derived from the Tuning Your Cognitive Algorithms exercise on bewelltuned.com.

The basic premise is to solve a medium-difficulty-puzzle, while spending a lot of attention on noticing exactly what your brain is doing at every step. You can then notice which pieces of your process are doing most of the work, and which are wasted motion.

This is not only valuable for improving your puzzle-solving abilities, but for generally improving the feedback loops that improve your cognitive abilities. I've found this to be one of the most essential rationalist skills that I've learned.

We'll be meeting in the Lecture Hall in the Walled Garden, at noon PT. (Follow this invite link to enter the Garden. Follow the path to the Tardis, and then enter the "Bacon Lecture Hall")

I'll give a short talk on the theory-and-practice of tuning your cognitive algorithms. We'll spend two 20 minute periods doing individual exercises, and then talking about what we learned.

http://garden.lesswrong.com?code=yoG1&event=guest-pass