trentmkelly/LessWrong-43k huggingface dataset:
Utility ≠ Reward
by Vlad Mikulik — 2019-09-05T17:28:13.222Z

This essay is an adaptation of a talk I gave at the Human-Aligned AI Summer School 2019 about our work on mesa-optimisation. My goal here is to write an informal, accessible and intuitive introduction to the worry that we describe in our full-length report.

I will skip most of the detailed analysis from our report, and encourage the curious reader to follow up this essay with our sequence or report.

The essay has six parts:

Two distinctions draws the foundational distinctions between

“optimised” and “optimising”, and between utility and reward.

What objectives? discusses the behavioral and internal approaches to understanding objectives of ML systems.

Why worry? outlines the risk posed by the utility ≠ reward gap.

Mesa-optimisers introduces our language for analysing this worry.

An alignment agenda sketches different alignment problems presented by these ideas, and suggests transparency and interpretability as a way to solve them.

Where does this leave us? summarises the essay and suggests where to look next.

The views expressed here are my own, and do not necessarily reflect those of my coauthors or MIRI. While I wrote this essay in first person, all of the core ideas are the fruit of an equal collaboration between Joar Skalse, Chris van Merwijk, Evan Hubinger and myself. I wish to thank Chris and Joar for long discussions and input as I was writing my talk, and all three, as well as Jaime Sevilla Molina, for thoughtful comments on this essay.

≈3300 words.

Two distinctions

I wish to draw a distinction which I think is crucial for clarity about AI alignment, yet is rarely drawn. That distinction is between the reward signal of a reinforcement learning (RL) agent and its “utility function”[1]. That is to say, it is not in general true that the policy of an RL agent is optimising for its reward. To explain what I mean by this, I will first draw another distinction, between “optimised” and “optimising”. These distinctions lie at the core of our mesa-optimisation framework.

It’s helpful to begin with an analogy. Viewed abstractly, biological evolution is an optimisation process that searches through configurations of matter to find ones that are good at replication. Humans are a product of this optimisation process, and so we are to some extent good at replicating. Yet we don’t care, by and large, about replication in itself.

Many things we care about look like replication...
Logical Counterfactuals and Proposition graphs, Part 3
by Donald Hobson — 2019-09-05T15:03:53.262Z

Notation note, [ ] are for lists, ( ) are for functions.

Note that many of the words and symbols I am using are made up. When this maths is better understood, someone should reinvent the notation. My notation isn't that great, but its hard to make good notation when you are still working out what you want to describe.

A theory (of my new type, not standard maths) T is formally defined to be,

T=[ψ,ρ,Ξ,type,arity]

Where ψ={s1,s2,...} are the theories symbols. These can be arbitrary mathematical objects.

Ξ={Ξ1,Ξ2,...} is the set of types, also arbitrary.

type:ψ→Ξ is a function.

arity:ψ→∪∞i=0Ξ×Ξ×⋯×Ξi

Is also a function.

An expression E in a theory is defined recursively, it consists of a pair E=[s,v1,v2,⋯,vn]. Here s∈ψ and ∀1≤i≤n:vi is an expression.

Let arity(s)=[x1,x2,...xm]

Sometimes we will write type(E), what we really mean is type(s), and arity(E)=arity(s) We write symbol(E)=s when we want to refer to just s and ignore v1,...,vn

Expressions have the restriction that m=n the number of elements of Ξ that s is mapped to is the same as the number of other expressions.

We also insist that for all i, type(vi)=xi

The base case happens when arity(s)=[ ] the empty list.

All expressions are strictly finite mathematical objects. There are no self referential expressions.

Expressions can be written s(v1,...,vn)

We can define equality between expressions e=s(v1,...) and f=t(w1,...) recursively by saying e=f iff symbol(e)=symbol(f) and forall i:vi=wi

The distinction between expressions e,f is defined recursively.

e−f=[e,f] if symbol(e)≠symbol(f)

e−f=[e,f] if ∃i≠j∈N:vi≠wi and vj≠wj

e−f=None if e=f

e−f=vi−wi if ∃1 i:vi≠wi

These can be uniquely expressed as strings made up of Ξ∪{ ′(′, ′,′, ′)′}

Lets define V(n)=V(Ξn) to be the set of all possible expressions of type Ξn.

A function f:V(n1)×...V(nt)→V(m) is called basic if it is constant, or it is the projection function ( so f(e1,...et)=ek for fixed k≤t) or f can be expressed as

f(e1,...et)=s(v1,...vn) where s is constant and each vi is a basic function of e1,...et.

Note you can cross as much extra junk into f as you like and ignore it. If f(e1,e2) is basic, so is g(e1,e3,e2,e4)=f(e1,e2).

Basic functions can be said to have type(f)=Ξm and arity(f)=[Ξn1,...Ξnt]

Basic functions can be defined in the style of f(α,β)=s1(c1,α,s2(α,β))

Finally, ρ={[f1,g1... ,
Somerville Housing Over Time
by jefftk — 2019-09-05T14:40:01.380Z

I wanted to understand how the number of housing units in Somerville
had changed over our history, but wasn't able to find anything
searching online. We can get an estimate, however, by looking at the
"Year Built" field in the

Assessor's
Database. This shows the age of Somerville's current housing
stock, to the best estimate of the Assessor.

Decade
Number Built

17xx
7

180x
36

182x
3

183x
1

184x
30

185x
65

186x
79

187x
143

188x
476

189x
1025

190x
4657

191x
1863

192x
2104

193x
401

194x
55

195x
75

196x
104

197x
181

198x
394

199x
72

200x
24

201x
51 (57 projected)

In estimating how much building was happening in various decades this
will be biased towards the present: a building that was demolished and
replaced will show up as being built recently, replacing the earlier
older listing. On the other hand, newer projects are often larger, as
former industrial sites are converted to housing. Still, it shows
that current building levels in Somerville, while higher than in the
2000s, are still very low by historical standards.

(The most recent year I see in the data is 2018, so I've scaled the
numbers for the 2010s decade up by 10/9 to reflect that we're missing
2019 data. I've also truncated the data to just the decade because my
understanding is it's common for the database to use the first year as
a stand in for "some time that decade", with 1910 to mean 1910s. Only
11,847 of the 12,145 parcel records contain a "Year Built" field so
I'm ignoring the other 2%. I'm also ignoring 84 Perkins St, parcel
14057, which claims to have been built in 1518.)

Comment via: facebook
Living the Berkeley idealism
by Yuxi_Liu — 2019-09-04T22:20:09.829Z

Quick observation, more funny than insightful.

Today I was thinking about how to publish my thesis when it's finished, and rethinking again what format to put it in. The standard pdf format seems reasonable, but it is merely made of digital print, with no interactivity. Putting in interactivity requires me to not only learn something (web hosting, JavaScript, etc), but at the same time trying to guess which one is future-proof. Flash are Java applets are daily reminders of what not to bet on for future-proof.

Book publishers enjoy a relative longevity. Go to the library and open a book that hasn't been touched for three hundred years, and it would still work. Books run on light energy, with almost no upkeep cost (if located in a building with dry and ~300 K temperature air). It would be a small cause for celebration to find something interactive online that is 10 years old and still works as intended.

To live in this kind of uncertainty is to experience Berkeley's Idealism. Berkeley thought that anything exists only because God is perceiving it, and if God stops perceiving it, that thing disappears. Just like that, as long as you are paying attention to something online, it would keep existing. As soon as you forget about it, it has a serious chance of decaying and stop working.

From which we conclude that Berkeley's God would probably feel annoyed all the time that the world can't seem to run without His constant staring.
Stories About Education
by Ben Pace — 2019-09-04T19:53:47.637Z

This is the 3rd post of 5 containing the transcript of a podcast hosted by Eric Weinstein interviewing Peter Thiel.

Interview

Student Debt

Peter Thiel: It's like, again, if you come back to something as reductionist as the ever escalating student debt, you know, the bigger the debt gets, you can sort of think what is the 1.6 trillion, what does it pay for? And in a sense, it pays for $1.6 trillion worth of lies about how great the system is.

Peter Thiel: And so, the more the debt goes, the crazier the system gets, but also the more you have to tell the lies, and these things sort of go together. It's not a stable sequence. At some point this breaks. You know, again, I would bet on a decade, not a century.

Eric Weinstein: Well, this is the fascinating thing, you, of course, famously started the Thiel Fellowship as a program which, correct me if I'm wrong on this, 2005 is when student debt became non-dischargeable even in bankruptcy.

Peter Thiel: Yes. The Bush 43 bankruptcy revision. If you don't pay off your student loans when you're 65 the government will garnish your social security wages to pay off your student debt.

Eric Weinstein: Right. This is amazing that this exists in a modern society. And of course, well, so let me ask, am I right that you were attacking what was necessary to keep the college mythology going, and you were frightened that college might be enervating some of our sort of most dynamic minds?

Peter Thiel: Well, I think there are sort of lot of different critiques one can have of the universities. I think the debt one is a very simple one. It's always dangerous to be burdened with too much debt. It sort of does limit your freedom of action. And it seems especially pernicious to do this super early in your career.

Peter Thiel: And so, if out of the gate you owe $100,000, and it's never clear you can get out of that hole, that's going to either demotivate you, or it's going to push you into maybe slightly higher paying, very uncreative professions of the sort that are probably less good at moving our whole society forwards. And so I think the whole thing is extraordinarily pernicious.

Peter Thiel: I started talking about this back in 2010, 2000, it was already like controversial, but it was not, you know... younger people all agreed with me.

Eric Weinstein: The younger people did?

Peter Thiel: And it's a decade later, it's a lot crazier, we haven't ye...
The Power to Judge Startup Ideas
by Liron — 2019-09-04T15:07:25.486Z

This is Part III of the Specificity Sequence

When Steve claims that Acme exploits its workers, he's role-playing the surface behaviors of an opinionated intellectual, but doesn't bother to actually be an opinionated intellectual, which would require him to nail down a coherent opinion.

It turns out that a lot of startups are founded by people doing something analogous to Steve: they role-play the surface behaviors of running a company and building a product, but don't bother to nail down a coherent picture of what customers would ever come to their business for.

Startup Steves

Paul Graham, cofounder of Y Combinator, calls this failure mode one of The 18 Mistakes that Kill Startups:

Having No Specific User In Mind
A surprising number of founders seem willing to assume that someone, they’re not sure exactly who, will want what they’re building. Do the founders want it? No, they’re not the target market. Who is? Teenagers. People interested in local events (that one is a perennial tarpit). Or “business” users. What business users? Gas stations? Movie studios? Defense contractors?

I'm in the startup industry, and I watch a lot of startups committing suicide by not being specific enough about who their customer is. From my perspective, the failure to be specific isn’t just a top-18 mistake, it’s the #1 mistake that founders make.

If you watch Paul Graham Office Hours at Startup School 2011, you can see for yourself that most of the founders on stage don’t seem to have a specific idea of who they’re building their product for and what difference it makes in their lives. Eliezer observes:

There was an exchange in Paul Graham [and Harj Taggar]’s office hours that went like this, while interviewing a startup that did metrics — analyzing pageviews, roughly — and the entrepreneur was having great trouble describing what they did that Mixpanel didn’t. It went on for a while. It was painful to watch.

Paul: I don’t get what the difference is. I still don’t get what the difference is. What’s the difference between you and Mixpanel?

Entrepreneur: The difference is — when you have to supplement — they’re a view company and we’re a platform. That’s what it comes down to. They’re like a view, a reporting company. If you need something they don’t have, a feature -

Harj: So what’s an example of somewhere you’d use your thing over Mixpanel? Can you give a use-case?

Entrepreneur: Yeah, I mean,

...
Caching on Success 
by pcm50 — 2019-09-04T06:34:44.162Z

What I previously took from Cached Thoughts was primarily us failing to examine memes before trotting them out on cue. Some snippet of information A gets lodged *from outside* as the response to some question Q. Whenever Q appears we consult the lookup table and out pops A.
More recently I was reflecting on a pattern I have noticed in my work as a statistical programmer. I realised that I was following this pattern on a regular basis. In particular, by caching action sequences that worked.
It goes as follows. My code doesn't work in situation Q, so try X. It still doesn't work, so try Y. It still doesn't work, so try Z. It works. So, when presented with failure Q do XYZ.
Clearly, this is not proper science. However, it is fairly effective. When presented with that particular kind of failure again, XYZ will make things work. However, change the problem slightly and XYZ may well fail because we don't fully understand why it succeeded in the first instance.
Why not do proper science all the time then? I'd love to, in theory. However, if we look at just those three available actions: possibly the order matters, in which case there are 5 other orderings to explore. Possibly we could select some subsets to explore and there are four we didn't examine at all yet {{Y}, {Z}, {X,Z}, {Y,Z}}. The length two subsets each have two orderings, so there are something like 12 remaining action sequences to try out
Of course, with intuition, many of action sequences could be dismissed. But the point is that it's pretty time consuming and explodes with the action menu. The cost of testing is not just cognitive. E.g. suppose that testing one action sequence burns $1000 of resources, or training a model takes 24 hours. Or e.g. you just need this thing to work right now and have three ideas, do you faff about looking to understand the optimal solution or do you just throw all three at it and fix the problem?
So, I see why my brain often wants to stick with the tried and true method. The risky element is that it then wants to confabulate some reason why XYZ in particular worked. E.g. suppose X was ineffective and only YZ mattered, I'll find I still want to tell a story about why X was important. Consciously I'm now trying to avoid this by saying something like, "I suspected XYZ would work and it did, but I'm not sure how to credit t...
Counterfactuals are an Answer, Not a Question
by Chris_Leong — 2019-09-03T15:36:39.622Z

I'm going to subtly contradict my last post on logical counterfactuals. I now think that raw or actually-consistent counterfactuals are just an especially useful model of what counterfactuals are as opposed to the be all and end all.

An Old Classic

You may have heard this one before: The Ship of Theseus is a very old ship, so old that all the original parts had been replaced. Is it still the Ship of Theseus? What if we gathered all of the original parts and used them to rebuild the ship? Would this new construct be the Ship of Theseus now?

If you've internalised the idea of the map and the territory, the answer should be clear. Terms like "The Ship of Theseus" are human constructions with no real existence. They can be defined in whatever manner is most useful.

Counterfactuals Don't Exist in the Universe

So let's start with regular counterfactuals. What are they? Do they exist in the universe itself? Unless you're a modal realist (ie. David Lewis and his clones) the answer is no. Given the exact state of universe and an agent, the agent can only make one decision*. In other words, counterfactuals are something we construct.

This means that it is a mistake to search for an objectively true definition. Just as in the Ship of Theseus, our expectation should be that multiple definitions could have value. Of course, some definitions may be more useful or natural than others and indeed, as I've argued, Raw Counterfactuals are particularly natural. However, I wouldn't go as far as I went in that post where I argued that insofar any other kind of counterfactual had meaning, it was derived from Raw Counterfactuals.

Logical counterfactuals are no different. We may not know everything about logic, but this doesn't mean that logic could have been different. We can construct a model where we imagine that logic being different than it is, but there isn't a fact of the matter about how logic would be if 1+1=3 instead of 2 that exists as part of standard logic without any extensions, any more than counterfactuals exist in-universe without any extensions.

An Answer, Not a Question

Since counterfactuals don't have an objectively true definition, asking, "What are counterfactuals?" is a confused question. It'd be better to ask, "What definitions of counterfactuals are useful?", but then this is still kind of vague. A better answer is to figure out the kinds of questions that we tend t...
How to write good AI forecasting questions + Question Database (Forecasting infrastructure, part 3)
by Bird Concept — 2019-09-03T14:50:59.288Z

This post introduces

an open-source database of 76 questions about AI progress, together with detailed resolution conditions, categorisations and several spreadsheets compiling outside views and data

, as well as learning points about how to write good AI forecasting questions. It is the third part in a series of blog posts which motivate and introduce pieces of infrastructure intended to improve our ability to forecast novel and uncertain domains like AI.

Background and motivation

Through

our work on AI forecasting

in the recent year, we’ve tried to write many questions that track important facets of progress, and gained some experience in how hard this is; and how to do it better.

In doing this, we’ve found that most previous public attempts at writing AI forecasting questions (e.g.

this

and

this

) fall prey to several failure modes that worsen the signal of the questions; as did many of the questions we wrote ourselves. Overall, we think

operationalisation is an unsolved problem

, and this has been the impetus behind the work described in this sequence of posts on forecasting infrastructure.

A previous great resource on this topic is

Allan Dafoe’s AI Governance Research Agenda

, which has an appendix with forecasting desiderata (page 52-53). This blog post complements that agenda by adding a large number of concrete examples.

We begin by categorising and giving examples of some ways in which technical forecasting question can fail to capture the important, intended uncertainty. (Note that the below examples are not fully fleshed out questions, in order to allow for easier reading.) We then describe the question database we’re open-sourcing.

Ambiguity

Terms that can have many different meanings, such as “AGI” or “hardcoded knowledge”.

Underspecification

Resolution criteria that neglects to specify how questions should be resolved in some possible scenarios.

Examples

This question resolves positively if an article in a reputable journal article finds that commercially-available automated speech recognition is better than human speech recognition (in the sense of having a lower transcription error rate).

If a journal article is published that finds that in only in some domains commercially-available automated speech recognition is better (e.g. for HR meetings), but worse in most other domains, it is unclear from the resolution criteria how this question should be resolv...
How Specificity Works
by Liron — 2019-09-03T12:11:36.216Z

This is Part II of the Specificity Sequence

You saw what mayhem we brought forth when we activated the first power of specificity, the power to demolish bad arguments, and hopefully your curiosity is piqued to see what’ll happen when we activate all the other powers.

But first, let’s pause here to ask: How does specificity work?

Consider this dialogue between Steve and one of his pals:

Steve: Information should be free!

Steve's Pal: Whoa, this is thought-provoking stuff. Ok, so, would you say you're advocating for digital socialism, or more like digital libertarianism?

Oh jeez. Not only is Steve's pal not pushing for Steve to be more specific, the pal is an enabler who pushes Steve to be less specific. They're climbing the ladder of abstraction the wrong way.

The Ladder of Abstraction

When you want to nail down a claim, the operative word is “down”: you want to bring the discussion down the ladder of abstraction:

If Steve says, “Information should be free!” and I'm trying to understand what he means, here's what I'd say:

Liron: Ok, why do you think I shouldn’t have had to pay Amazon for my paperback copy of To Kill A Mockingbird?

This way, I'm nosediving all the way down to the bottom rung of the ladder of abstraction. Down here, the conversation becomes grounded in the concrete language of everyday experience, with substantive statements like “Amazon charged my credit card $6.99 and kicked back $1.15 to Harper Lee’s estate.”

And how about that free shipping, Steve?

Having this kind of grounded discussion is usually more productive than having a flingfest of the higher level ballpit-words “information”, “freedom”, “socialism”, and “libertarianism”.

As Steve and I are hanging out on the bottom rung of the ladder of abstraction, talking through specific examples of information getting exchanged freely vs. non-freely, we’ll be able to notice if certain features seem to remain constant across our chosen examples. For instance, we might discuss various hypothetical authors who yearn to write books for a living, and observe that all such authors can still have some plausible mechanism to earn money (running ads on their blogs?), even without directly charging for the privilege of reading their books.

After loading our brains with specific examples, we can then abstract over these examples, climb our way back up the ladder of abstraction, and put forth a generalized claim about w...