trentmkelly/LessWrong-43k huggingface dataset:
Outer alignment and imitative amplification
by evhub — 2020-01-10T00:26:40.480Z

Understanding the outer alignment problem

What really is outer alignment? In “Risks from Learned Optimization,” we defined outer alignment in the context of machine learning as “aligning the specified loss function with the intended goal.” But that's not a perfectly well-defined statement—what does it mean for a loss function to be “aligned” with the intended goal? If the goal we care about is maximizing U, do we need exactly L=−aU+b for constants a,b? That's a pretty high bar.

Well, what exactly do we want outer alignment for? At the end of the day, we care about whether the model that pops out the other end of our training procedure will be safe, which is a complicated question involving the loss function, the architecture, the implicit inductive biases, and so on. In what sense, then, is it even reasonable to look at just the lost function in isolation and ask whether it's aligned or not?

I think the strongest case for outer alignment being a meaningful problem in isolation comes from the argument that loss functions seem to scale pretty well with generic machine learning progress. If, as a silly example, your outer alignment scheme is to “train image classification models,” that's something that ML has progressively gotten better at over time. Compare that to the silly inner alignment scheme of “train a straightforward CNN”—that's something that ML has passed by pretty rapidly in favor of architectural improvements like residual connections even just for the task of image classification. Of course, outer alignment alone does not an aligned AGI make, so you still have to have some notion of how you're going to do inner alignment in mind—but loss functions scaling better is still a perfectly valid reason for focusing on outer alignment.[1]

Thus, it does seem quite reasonable to me to put effort into finding “aligned” loss functions. But that still brings us back to the question of what exactly makes a loss function “aligned.” In the context of a specific training/inner alignment scheme, we can say that a loss function is aligned if, when plugged into that training scheme, it produces models which are aligned with our goals. But in the absence of any specific training scheme, what does it mean to say that a loss function is aligned in isolation? We can of course ask for L=−aU+b as I stated previously, though in my opinion I think achieving something like that

...
Criticism as Entertainment
by Elizabeth — 2020-01-09T22:20:01.058Z

ETA: This post relies on video embeds that didn't carry over when LW imported it. For the full experience, see here.

Media Reviews

There is a popular genre of video that consist of shitting on other people’s work without any generative content. Let me provide some examples.

First, Cinema Sins. This is the first video I selected when looking for a movie I’d seen with a Cinema Sins I hadn’t (i.e. it’s not random, but it wasn’t selected for being particularly good or bad).

The first ten sins are:

Use of a consistent brand for props in the movie they’d have to have anyway, unobtrusively enough that I never noticed until Cinema Sins pointed it out.

A character being mildly unreasonable to provoke exposition.

The logo

Exposition that wasn’t perfectly justified in-story

A convenience about what was shown on screen

A font choice (from an entity that in-universe would plausibly make bad font choices)

An omission that will nag at you if you think about it long enough or expect the MCU to be a completely different thing, with some information about why it happened.

In-character choices that would be concerning in the real world and I would argue are treated appropriately by the movie, although reasonable people could disagree

Error by character that was extremely obviously intentional on the part of the film makers. There is no reasonable disagreement on this point.

An error perfectly in keeping with what we know about the character.

Of those, three to four could even plausibly be called sins of the movie- and if those bother you, maybe the MCU is not for you. The rest are deliberate choices by filmmakers to have characters do imperfect things. Everyone gets to draw their own line on characters being dumb- mine is after this movie but before 90s sitcoms running on miscommunication- but that’s irrelevant to this post because Cinema Sins is not helping you determine where a particular movie is relative to your line. Every video makes the movie sound exactly as bad as the last, regardless of the quality of the underlying movie. It’s like they analyze the dialogue sentence by sentence and look to see if there’s anything that could be criticized about it.

Pitch Meeting is roughly as useful, but instead of reacting to sentences, it’s reading the plot summary in a sarcastic tone of voice.

Pitch Meeting is at least bringing up actual problems with Game of Thrones season 8. ...
Post-mortem on the Center for Long-Term Cybersecurity forecasts 
by Ben Goldhaber — 2020-01-09T19:38:33.289Z
Link: https://medium.com/cltc-bulletin/post-mortem-2020-looking-back-on-cltcs-scenarios-from-2015-3ad09fb6f1e7

A collection of scenarios the CLTC predicted in 2015 about the 2020 cybersecurity landscape.
Rationalist Scriptures? 
by John Igo — 2020-01-09T16:59:08.444Z

I understand that the title sounds like an oxymoron, however, we it seems like we do have scriptures. The Codex, HPMOR, Rationality A-Z, etc are basically required reading so that we can have a community with a fixed starting point, however they don't seem to be well compiled.

HPMOR is a complete story. The Codex and Rationality A-Z are two series of blog posts so they retread a lot of territory - has anyone tried to edit them into more coherent books?

Related, has anyone compiled a list of "Rationalist Wisdom"? Like a bunch of sayings that distill Rationalism down that we can point newbs to? I ask because I was looking for a list and couldn't find one, and pointing curious people to 1000+ page books is daunting.
Preference synthesis illustrated: Star Wars
by Stuart_Armstrong — 2020-01-09T16:47:26.567Z

EDIT: this post is intended to be mainly about preference ordering synthesis, not Star Wars.

This is my ordering of the core Star Wars movies:

The Empire Strikes Back > Return of the Jedi >= The Last Jedi > Rogue One > A New Hope = The Rise of Skywalker > The Force Awakens > Revenge of the Sith >> Attack of the Clones >>>>> The Phantom Menace

I'm sure that nobody will disagree with me or find these rankings in any way controversial ^_^

But I'm really posting these here to illustrate how preferences can be synthesised from partial preferences.

Because that order didn't exist in my head, I had to synthesise it, fitting various arguments and orderings together. Had I done it at a different time, or in a different mood, I would have had a substantially different ordering.

A few things were probably invariant, though. The quality of "The Empire Strikes Back" was clear in my head, as was the terribleness of "The Phantom Menace". "A New Hope" and "The Force Awakens" were easy to compare, being essentially the same movie.

But how did I compare "The Last Jedi", which I found to be an inconsistent sprawling mess with some of the most beautiful scenes in all of Star Wars (and the "old hero retired" trope, which I really enjoy), with "A New Hope", a tightly scripted elegant adventure story with little depth? And what about "Return of the Jedi", which has a great ending, a decent beginning, but a muddelish middle?

Well, Scott has defined ambijectivity, which allows one to say that comparisons like "Mozart’s music is better than Beethoven’s" is subjective, while "Mozart’s music is better than the music of the three-year old girl who lives upstairs from me and bangs on her toy piano sometimes" is more objective.

The idea is that there are hundreds of different ways in which we can compare things, and some of these are simple enough that they have an easy objective or subjective answer. And Mozart and Beethoven are tend to beat each other roughly equally on the more objective criteria, and can go either way on the subjective ones.

But they both tend to beat the three-year old on almost all criteria.

How I ordered the movies

So, in comparing the Star Wars movies, I stated with a few clear orderings, and then compared movies based on criteria I felt were relevant. For example, "Return of the Jedi" and "The Rise of Skywalker" are reasonably sim

...
[Personal Experiment] Training YouTube's Algorithm
by lsusr — 2020-01-09T09:04:17.459Z

I listen to YouTube music every day. I created a new Google account three weeks ago. I've watched nothing but music and dancing on it. Whenever it recommends a non-music non-dancing video I immediately click "Don't recommend channel"[1].

If you you opened my YouTube account right now you'd see 509 recommendations on the home page. 487 would be for music and dancing. 22 would be things I'm not interested in. Below are YouTube's 22 non-musical recommendations:

Unboxing Violin and Testing

不要随便让老师唱歌跳舞，搞不好他们是被教育事业耽误的明星！

如何保護總統生命安全？前美國特勤局特務：「我們的任務就是站在威脅與目標之間」｜科普小知識｜GQ Taiwan

L'équipe du Togo - 60 minutes avec Kheiron

sony's lead graphic designer coming up with the PS5 logo

The Engoodening of No Man's Sky

What is an Exoplanet Documentary - Planets Beyond our Solar System

Rockman 3 – Dr. Wily no Saigo!? (FC)

NLE Choppa - Cottonwood (Official Trailer)

YOU is a weird show... (Alex Meyers channel)

Safiya & Tyler's Wedding Highlight Film

Martin Lawrence Finally Responds To Tisha Campbell Accusations: None Of That Was True - CH News

True Facts: Mating Dance of the Ostrich

5 Sikretong Video Ng NASA Na Nabunyag

Nanatsu no Taizai 3ª Temporada EPISÓDIO 13 LEGENDADO PT BR

Weathering With You [Official English Dub Trailer - GKIDS] - January 15

【Stanley】海克斯閃現福利熊熊！各種閃現外翻！5/0/10鬼神一般的戰績！原來是上路福利熊啊我還以為是輔助呢！改名以後戳仔大幅下降？成為了勇於認錯的韓服Nice 玩家 +1 [This is a Let's Play]

Star Wars: The Last Jedi - Training for the Last Jedi

r/TIHI | TeeHee

韓國女生的台灣網咖旅館一日求生挑戰

The Hobbit with Lightsabers

When an Imperial Major Didn't Recognize Darth Vader(Canon) - Star Wars Comics Explained

Most of these are understandable errors at personalization. The violin unboxing makes sense because I listen to lots of violin music. The Star Wars and anime stuff makes sense because I watch "Star Wars Anime Opening" videos. "不要随便让老师唱歌跳舞..." is dancing-related. I think the movie trailers come from the fact that some of the music videos I watch take the form of movie clips. I was surprised by the science recommendations until I realized it's not difficult to infer that someone who listens to AIVA-generated music over-and-over again (among similar things) might be interested in science. The wedding highlight film could be related to the wedding dance videos I enjoy.

The videos in Chinese, Spanish and Japanese make sense because I listen to music in these languages. When I first created my account it recommended a lot of ne

...
Old Airports
by jefftk — 2020-01-09T03:30:01.561Z

One thing I enjoy about aerial imagery is that you can see traces of how cities used to be. Tracing the spaces left behind by old train lines is a lot of fun, but recently I've been looking at some airports:

Old Saint-Pierre Airport (link), now with housing:

Old Jijiga Airport (link), also with housing, mostly recognizable as a piece of the street grid that doesn't match:

Old Guangzhou Airport (link), barely visible:

Old Quito Airport (link), now a large park and very visible. I think this the airport we used when we visited Ecuador in 2012:

Old Rio Branco Airport (link) is now a road:

Old Austin Airport (link) is very hard to recognize. All the ones above were single-runway, but Mueller airport had two large runways at an angle. Single-runway ones stand out a lot more, but this is still visible as a section of the city that doesn't really fit:

Old Lima Airport (link) closed sixty years ago and has been completely built over. But the runways survive as a pair of crossing roads:

There's a list on Wikipedia but my favorites are ones where I've been looking at an aerial photo and thought "there has to have been an airport there!"

Comment via: facebook
Are "superforecasters" a real phenomenon?
by reallyeli — 2020-01-09T01:23:39.361Z

In https://slatestarcodex.com/2016/02/04/book-review-superforecasting/, Scott writes:

…okay, now we’re getting to a part I don’t understand. When I read Tetlock’s paper, all he says is that he took the top sixty forecasters, declared them superforecasters, and then studied them intensively. That’s fine; I’d love to know what puts someone in the top 2% of forecasters. But it’s important not to phrase this as “Philip Tetlock discovered that 2% of people are superforecasters”. This suggests a discontinuity, a natural division into two groups. But unless I’m missing something, there’s no evidence for this. Two percent of forecasters were in the top two percent. Then Tetlock named them “superforecasters”. We can discuss what skills help people make it this high, but we probably shouldn’t think of it as a specific phenomenon.

But in this article https://www.vox.com/future-perfect/2020/1/7/21051910/predictions-trump-brexit-recession-2019-2020, Kelsey Piper and Dylan Matthews write:

Tetlock and his collaborators have run studies involving tens of thousands of participants and have discovered that prediction follows a power law distribution. That is, most people are pretty bad at it, but a few (Tetlock, in a Gladwellian twist, calls them “superforecasters”) appear to be systematically better than most at predicting world events.

seeming to disagree. I'm curious who's right.

So there's the question of "is superforecaster a natural category" and I'm operationalizing that into "do the performances of GJP participants follow a power-law distribution, such that the best 2% are significantly better than the rest"?

Does anyone know the answer to that question? (And/or does anyone want to argue with that operationalization?)
Subscripting Typographic Convention For Citations/Dates/Sources/Evidentials: A Proposal
by gwern — 2020-01-08T22:20:20.290Z

Reviving an old General Semantics proposal: borrowing from scientific notation and using subscripts like 'Gwern2020' for denoting sources (like citation, timing, or medium) might be a useful trick for clearer writing, compared to omitting such information or using standard cumbersome circumlocutions.

Moved to gwern.net
[AN #81]: Universality as a potential solution to conceptual difficulties in intent alignment
by Rohin Shah — 2020-01-08T18:00:01.566Z

Find all Alignment Newsletter resources

here

. In particular, you can

sign up

, or look through this

spreadsheet

of all summaries that have ever been in the newsletter. I'm always happy to hear feedback; you can send it to me by replying to this email.

Audio version

here

(may not be up yet).

Published a year ago,

this

sequence

of

five

posts

introduced the idea of ascription universality. I didn't really get it on a first reading, and only recently read it in enough detail that I think I understand the main ideas. This entire newsletter will focus on ascription universality; treat all of it as a "Highlight".

The key idea of these posts is that of universality: when we can say that some agent "knows everything that any other agent could know". Of course, there will always be some agent with arbitrarily complex beliefs, but we could hope to have agents that know everything that is known by any agent of some complexity class (e.g. agents that run in polynomial time). However, there are still simple programs that can have complex true beliefs: we could make agents that believe that P != NP and that P = NP, and one of those agents will fall into this class. However, these agents are somewhat degenerate -- they don't have a good reason to trust that their belief is true, and so we don't want to worry about them. Since we want to talk about justified belief, let's consider a property based on which of the two agents we would trust.

So, for now let's adopt an informal definition: a program A[C] is universal with respect to some class of programs C if we would trust any beliefs reported by A[C], no matter what beliefs we hear reported by programs in C. (I'll talk about the specific notion of "ascription universality" later.) Notably, we need to trust A[C] even over our own beliefs, that is, if A[C] believes something, we discard our position and adopt A[C]'s belief.

Note that this is still a very strong criterion. Consider the question of whether AI will lead to x-risk: if one agent thinks about fixed-point theorems, while another studies neural nets, and a third tries to predict the response of democratic institutions, the agent A[C] must compete with everything that these agents (justifiably) conclude. For now, let's set aside the question of how to build a universal agent, and instead understand wh...
Stripping Away the Protections
by Zvi — 2020-01-08T13:10:00.912Z

Previously in sequence: Moloch Hasn’t Won, Perfect Competition, Imperfect Competition, Does Big Business Hate Your Family?, What is Life in an Immoral Maze?

The previous post painted a bleak picture of life as a middle manager in an Immoral Maze. Not every middle manager faces the high maze levels described in Moral Mazes, but I am confident that many do.

How did things get so bad?

We previously discussed Meditations on Moloch. That has a lot of gear-level analysis of how easy it is for undifferentiated high stakes competition along a single variable to destroy everything of value, and how much worse it is to do all those things at once than would be the sum of their parts.

Perfect Competition summarized and fleshed out that model so we could work with it, and defined Super-Perfect Competition as perfect competition without free exit, which when that restriction is meaningful results in less than zero economic profits.

Imperfect Competition illustrated how different idealized perfect competition is from the conditions we typically experience. Even when we use markets that seem highly competitive. Even in a market used as a canonical example of perfect competition.

The difference between the situations discussed there, and the situation in Moral Mazes, is that the conditions of super-perfect competition really do apply. The protections against this happening have been stripped away.

An immoral maze is not as extreme as the imagined possible future em world where any simulated human caught doing something suboptimal is erased and replaced. But it is closer than one would first think. In a maze, people can’t yet be copied, but there are lots of aspirants to choose from, and any opportunity to rule one out is pounced upon.

Sources of positive differentiation between managers are systematically destroyed. As we saw above, managers do not believe they exist. So they don’t matter. This leaves only negative differentiation and selection, plus politics.

Sources of meaningful object-level concerns and detail, and the relevance of any long term consequences of actions, are stripped away as well. Again, managers do not believe they exist. So they don’t matter.

This leaves a homogenous product that managers must become, the production of which is subject to perfect competition.

Then the resulting politics runs amok. The system se

...
Predictably Predictable Futures Talk: Using Expected Loss & Prediction Innovation for Long Term Benefits
by ozziegooen — 2020-01-08T12:51:01.339Z
Link: https://www.youtube.com/watch?v=zTBp0Lmw4ZE

I recently gave a talk at the 3rd Oxford Workshop on Global Priorities Research

This covered a few topics:

How we can think about and use Expected Loss to understand and optimize judgemental forecasting setups.

Some thoughts around how (1) applies to our decisions about long-term forecasting.

A few slides on Foretold and how similar systems could be used to help with (1) and (2).

I recently recorded a quick version of this talk on YouTube. As always, I'd be curious to get feedback on these ideas.

Original Talk Abstract

The last several decades have brought a large number of contributions to statistical and judgemental forecasting endeavors. The next several decades may bring significantly more.

As things get more advanced, not only will our understanding of how to predict distant events improve, but our understanding of the accuracy of these forecasts will improve as well. This should help us to resolve questions regarding when and how much to trust these forecasts.

This talk will present an overview of what we can expect if things go well and how thinking about this can inform our current actions when optimizing for the long term future.

Related Links:

Foretold

Expected Loss Graphs

Blog posts on "Amplifying generalist research via forecasting":

https://www.lesswrong.com/posts/cLtdcxu9E4noRSons/part-1-amplifying-generalist-research-via-forecasting-models

https://www.lesswrong.com/posts/FeE9nR7RPZrLtsYzD/part-2-amplifying-generalist-research-via-forecasting
(Double-)Inverse Embedded Agency Problem
by Shmi — 2020-01-08T04:30:24.842Z

MIRI has said a lot about the issue of embedded agency over the last year. However, I am yet to see them trying to make progress in what I see as the most promising areas.

How does one attack a problem that is new, complicated and non-obvious? By constructing toy models and inverting hard questions to make them more tractable.

In general an inverse problem is harder than the "direct" one, because we are trying to infer unobservables from observables. Wikipedia gives an example of figuring out the position of Neptune from the perturbations in the orbit of Uranus. Another popular example is NP-complete problems: they are famously hard to solve but it is easy to verify a solution. Another example: you take a multiple-choice math quiz, it is often faster and easier to get the right answer by plugging the 4 or 5 potential solutions into the stated problem than to solve the problem directly.

I'll give an example from my own area. The equations of general relativity are hard to solve except in a few highly symmetric cases. It is a classic inverse problem. But! Any spacetime metric is actually a solution of the Einstein equations, so all one needs to do is to write down a metric and calculate its Einstein tensor to see what kind of a matter distribution (and boundary conditions) it is a solution of. Inverting the inverse problem! Of course, life is not that easy. Most solutions correspond to "unphysical" matter, usually with negative energy density, superluminal flows, singularities, infinities, weird topologies etc. However, it is a useful approach if one wants to study some general properties of the equations, and get a feel for (or sometimes a theorem about) what goes wrong, why and how. After a few iterations one can get better at guessing what form a "good" solution might take, and write up an ansatz that can help solve the original, not the inverse problem in some cases.

Another, more familiar example: arithmetic division. Until you learn or figure out the rules, it's hard. But its inverse problem, multiplication, is actually much easier! So to learn more about division, it pays to try to start with potential solutions and see what kind of multiplication actually solve the division problem. Eventually one can come up with the long division algorithm, that uses nothing but multiplication and subtraction. And voila, invertin...