trentmkelly/LessWrong-43k huggingface dataset:
Latent Variables and Model Mis-Specification
by jsteinhardt — 2018-11-07T14:48:40.434Z

Posted as part of the AI Alignment Forum sequence on Value Learning.

Rohin's note: So far, we’ve seen that ambitious value learning needs to understand human biases, and that we can't simply learn the biases in tandem with the reward. Perhaps we could hardcode a specific model of human biases? Such a model is likely to be incomplete and inaccurate, but it will perform better than assuming an optimal human, and as we notice failure modes we can improve the model. In the language of this post by Jacob Steinhardt (original here), we are using a mis-specified human model. The post talks about why model mis-specification is worse than it may seem at first glance.

This post is fairly technical and may not be accessible if you don’t have a background in machine learning. If so, you can skip this post and still understand the rest of the posts in the sequence. However, if you want to do ML-related safety research, I strongly recommend putting in the effort to understand the problems that can arise with mis-specification.

Machine learning is very good at optimizing predictions to match an observed signal — for instance, given a dataset of input images and labels of the images (e.g. dog, cat, etc.), machine learning is very good at correctly predicting the label of a new image. However, performance can quickly break down as soon as we care about criteria other than predicting observables. There are several cases where we might care about such criteria:

In scientific investigations, we often care less about predicting a specific observable phenomenon, and more about what that phenomenon implies about an underlying scientific theory.
In economic analysis, we are most interested in what policies will lead to desirable outcomes. This requires predicting what would counterfactually happen if we were to enact the policy, which we (usually) don’t have any data about.
In machine learning, we may be interested in learning value functions which match human preferences (this is especially important in complex settings where it is hard to specify a satisfactory value function by hand). However, we are unlikely to observe information about the value function directly, and instead must infer it implicitly. For instance, one might infer a value function for autonomous driving by observing the actions of an expert driver.

In all of the above scenarios, the primary object of interest — the s...
Reframing a Crush: Distilling the "like" out of "like like"
by squidious — 2018-11-07T02:50:01.171Z

When I was a kid, I was moved by the idea that love was a powerful force. In the media, people did ridiculous things in the name of love; worse still when love was unrequited. I was terrified of feeling completely and helplessly vulnerable for someone who could not or would not reciprocate. It not only felt like a loss of power, but also a waste of emotional energy that could just as easily be spent on someone who reciprocates.
I decided that I would never let myself develop feelings for someone who was unable to return them. This first showed up as an inability to develop a crush on someone who was already in a relationship. As I grew older and learned more about myself, I realized also that this is why I didn't develop a crush on a girl until I became very close with one who was openly bisexual. It also meant that as soon as one of my childhood crushes came out to me as gay, my torn up confusion on why he didn't reciprocate my flirting transformed into an appreciation for the fact that he trusted me enough to tell me, and hope that he'd find someone good for him.
Now, I'm not saying that this is something everyone can or should do. For years I struggled with whether this was an adaptive cognitive trick or a suppression of emotion. It's probably a bit of both. But in breaking down the way I turned crushes into friendships, I discovered some mental micromoves I was making that can easily be used in other situations as well. These techniques work well for me, but your mileage may vary.
I refer to this trick as reframing in the title because the base skill is similar to cognitive reframing. This can be used on a negative thought, to turn distress over getting an answer wrong on a test to pride at getting 99 questions correct. It can also be used on a feeling, like reframing your petrified anxiety over a big event into motivated excitement. I suspect that some level of experience reframing thoughts and feelings is a prerequisite for reframing a crush, since feelings for a particular person tend to be much more complex.
Another prerequisite is the ability to notice why you like the person. Of course, you don't need to list out every single reason, but reframing the strongest ones can do wonders. Personally, I find traits I look for in a romantic partner to often be similar to the traits I look for in a close friend: kindness, humor, understanding, etc. So it becomes

...
Rationality of demonstrating & voting
by bfinn — 2018-11-07T00:09:44.239Z

Summary

Based on rough estimates of their effectiveness, voting is a very good use of time, but demonstrating far less so – though it may still be worth doing, at least for a very large demonstration on a major issue.

Introduction

Why bother voting? Your vote will only change the result if it would otherwise be an exact tie; and the chance of that is negligible – one in millions.

But a chance of one in millions is worth taking if the jackpot is billions or trillions. That is, the opportunity for you to select a better rather than worse government, thereby making the country – though not yourself – billions or trillions of dollars better off. So as long as you care at least slightly about the rest of the country, voting is rational; civic duty really is a reason to vote.

(An interesting 2007 paper Voting as a Rational Choice by Edlin et al. models this formally. Though in contrast the Put A Number On It blog argues in part that you have no idea which party will run the country better, so your opinion and hence vote is worthless.)

Does similar reasoning justify taking part in demonstrations (and other forms of collective action), even though they don’t have such direct effects as voting?

Example

Consider the largest current political issue in the UK – Brexit. The background, for non-Britons, is that in 2016 a UK referendum voted by a narrow margin to leave the European Union (EU), on terms still being negotiated. Many ‘Remainers’ who wish to stay in the EU want a second referendum on the matter; recently, hundreds of thousands of them marched through London to call for one.

How worthwhile was it for a Remainer to vote in the 2016 referendum, or to demonstrate for a second one?

Vote response curve

A political action, such as voting or demonstrating, is intended to produce an outcome – in this case, the UK remaining in the EU. We can plot this as a response curve, showing how the number of participants affects the probability of the outcome.

Voting has a response curve like this:

The dashed red line shows that – in principal – if more than 50% of voters vote to remain in the EU, the probability that will happen is 100%, otherwise 0%. So if you knew in advance whether the result would be 50%, you’d know whether to bother voting: only vote if doing so will resolve an exact tie, and push the result up the step from 0% to 100%.

However, when deciding whether to vote, you only h...
The Vulnerable World Hypothesis (by Bostrom)
by Ben Pace — 2018-11-06T20:05:27.496Z
Link: https://nickbostrom.com/papers/vulnerable.pdf

Nick Bostrom has put up a new working paper to his personal site (for the first time in two years?), called The Vulnerable World Hypothesis.

I don't think I have time to read it all, but I'd be interested to see people comment with some choice quotes from the paper, and also read people's opinions on the ideas within it.

To get the basics, below I've written down the headings into a table of contents, copied in a few definitions I found when skimming, and also copied over the conclusion (which seemed to me more readable and useful than the abstract).

Contents

Is there a black ball in the urn of possible inventions?
A thought experiment: easy nukes
The vulnerable world hypothesis

VWH: If technological development continues then a set of capabilities will at some point be attained that make the devastation of civilization extremely likely, unless civilization sufficiently exits the semi-anarchic default condition.

Typology of vulnerabilities

Type-1 (“easy nukes”)

Type-1 vulnerability: There is some technology which is so destructive and so easy to use that, given the semi-anarchic default condition, the actions of actors in the apocalyptic residual make civilizational devastation extremely likely.

Type-2a (“safe first strike”)

Type-2a vulnerability: There is some level of technology at which powerful actors have the ability to produce civilization-devastating harms and, in the semi-anarchic default condition, face incentives to use that ability.

Type-2b (“worse global warming”)

Type-2b vulnerability: There is some level of technology at which, in the semi-anarchic default condition, a great many actors face incentives to take some slightly damaging action such that the combined effect of those actions is civilizational devastation.

Type-0 (“surprising strangelets”)

Type-0 vulnerability: There is some technology that carries a hidden risk such that the default outcome when it is discovered is inadvertent civilizational devastation. 47

Achieving stabilization

Technological relinquishment

Principle of Differential Technological Development. Retard the development of dangerous and harmful technologies, especially ones that raise the level of existential risk; and accelerate the development of beneficial technologies, especially those that reduce the existential risks posed by nature or by other technologies.

Preference modification
Some specific countermeasures a

...
Alignment Newsletter #31
by Rohin Shah — 2018-11-05T23:50:02.432Z

Highlights

Introducing the AI Alignment Forum (FAQ) (habryka): The Alignment Forum has officially launched! It aims to be the single online hub for researchers to have conversations about all the ideas in the field, while also helping new researchers get up to speed. While posting is restricted to members, all content is cross-posted to LessWrong, where anyone can engage with it. In addition, for the next few weeks there will be a daily post from one of three new sequences on embedded agency, iterated amplification, and value learning.

Rohin's opinion: I'm excited for this forum, and will be collating the value learning sequence for its launch. Since these sequences are meant to teach some of the key ideas in AI alignment, I would probably end up highlighting every single post. Instead of that, I'm going to create new categories for each sequence and summarize them each week within the category, but you should treat them as if I had highlighted them.

Reinforcement Learning with Prediction-Based Rewards (Yuri Burda and Harri Edwards) (summarized by Richard): Researchers at OpenAI have beaten average human performance on Montezuma's Revenge using a prediction-based curiosity technique called Random Network Distillation. A network with fixed random weights evaluates each state; another network with the same architecture is trained to predict the random network's output, given its input. The agent receives an additional reward proportional to the predictor's error on its current state. The idea behind the technique is that the predictor's error will be higher on states different from those it's been trained on, and so the agent will be rewarded for exploring them.

This paper follows from their study on curiosity (AN #20) in which a predictor was trained to predict the next state directly, and the agent was rewarded when its error was high. However, this led to high reward on states that were unpredictable due to model limitations or stochasticity (e.g. the noisy TV problem). By contrast, Random Network Distillation only requires the prediction of a deterministic function which is definitely within the class of functions representable by the predictor (since it has the same architecture as the random network).

Richard's opinion: This is an important step forward for curiosity-driven agents. As the authors note in the paper,...
Octopath Traveler: Spoiler-Free Review
by Zvi — 2018-11-05T17:50:00.986Z

Epistemic Status: Full Play Through, extensive experience with similar games

Previous Spoiler-Free Review: Persona 5: Spoiler-Free Review

Soon I hope to write Against Spoilers, which will detail how hard it is to actually provide useful information in a true spoiler-free way.

Stage 1: True Spoiler-Free Review

Octopath Traveler is an excellent game for the Nintendo Switch, in the tradition of Final Fantasy and Bravely Default. It mimics the Super Nintendo age in terms of technology and graphics, and innovates in interesting ways on the core design.

If you enjoyed such games as Final Fantasy IV, Final Fantasy VI, Chrono Trigger and Bravely Default, this game is for you.

If you played but did not enjoy those games, this game is not for you.

If you have never played those games, the first three are some of the best games of all time, so if you’re at all interested in trying an old school JRPG, you should try one of them. I would go in the order listed but all three are great choices.

After the fold, I’ll do a more traditional ‘spoiler-free’ review that will definitely alter one’s experience playing. If you should play this game and haven’t yet, I encourage not reading further until after you have played.

Stage 2: Conventionally Spoiler-Free Review

Innovations

Octopath Traveler innovates in five key ways: It has eight stories you can tell in any order, it uses the break system, it refines the default system from Bravely Default into a more elegant and simpler version, it uses path actions, and it makes its version of Ether (Inspiring Plum) available at reasonable prices for mass purchasing, thus making exhaustion of magic points during exploration cease to be an issue fairly quickly. It also has strong implementations of the job system and letting characters choose and equip skills.

The combined result of this is that Octopath Traveler becomes a much more thoughtful and rich game in terms of its combats, and its story and experience is far more than the sum of its parts, even if its individual storylines aren’t as great as one might hope.

Overlapping Stories

Octopath Traveler is the overlapping stories of eight characters. Each of your eight party members has a four-stage journey to tell. You can take these eight stories in any order, and there’s nothing forcing you to even take all eight characters with you if you don’t feel like it. As you pick up more party members i...
Speculations on improving debating
by Richard_Ngo — 2018-11-05T16:10:02.799Z
Link: http://thinkingcomplete.blogspot.com/2018/03/speculations-on-improving-debating.html

I was recently discussing with a friend whether or not competitive debating makes you better at figuring out what is actually true. This is an interesting question, because debating influences people in a variety of different ways. The most basic is that participating in debates, or preparing for them, teaches you a lot of facts about the world; I would know much less about international politics in particular were it not for debating. It also markedly improves your ability to notice flaws and fallacies in arguments, which obviously helps you avoid falling for them. This skill can extend too far, though: debaters can be very good at rebutting even very sensible arguments. In the American Parliamentary style of debate, the proposing team is allowed to choose and prepare in advance any motion which is not so blatantly, obviously true that it would be impossible to debate (e.g. "murder is bad"). A lot of proposition teams skirt quite close to that line - but even so, opposing teams win a majority of the time. Debaters can poke holes in essentially any argument.
To be fair, debaters are aware that the existence of convincing rebuttals doesn't necessarily invalidate a claim. But motivated reasoning can be a powerful force, especially amongst intelligent people. This is particularly true when you've been trained to respond oppositionally to any claim that doesn't support your current position. I can think of several cases where I've been able to ignore a nagging voice of doubt by quickly finding a plausible response that turned out to be incorrect. The social media filter bubbles that we all currently inhabit can amplify this effect by caching in your mind hundreds of examples engineered to be memorable and stoke outrage; to counter it, it's more important than ever to genuinely consider opposing arguments rather than going to the automatic rebuttal mode that debating ingrains.
The other thing that debating (arguably) does is cultivate a non-empirical mindset. At the end of the debate, the question isn't settled. There may be more clarity over exactly which factors might swing a conclusion one way or the other, but almost nobody ever bothers to go out and do that further research, or even fact-check claims made during the debate - despite the fact that it's now easier to access such data than ever before. Of course if you asked debaters explicitly they'd be clear th

...
Humans can be assigned any values whatsoever…
by Stuart_Armstrong — 2018-11-05T14:26:41.337Z

(Re)Posted as part of the AI Alignment Forum sequence on Value Learning.

Rohin’s note: In the last post, we saw that a good broad value learning approach would need to understand the systematic biases in human planning in order to achieve superhuman performance. Perhaps we can just use machine learning again and learn the biases and reward simultaneously? This post by Stuart Armstrong (original here) and the associated paper say: “Not without more assumptions.”

This post comes from a theoretical perspective that may be alien to ML researchers; in particular, it makes an argument that simplicity priors do not solve the problem pointed out here, where simplicity is based on Kolmogorov complexity (which is an instantiation of the Minimum Description Length principle). The analog in machine learning would be an argument that regularization would not work. The proof used is specific to Kolmogorov complexity and does not clearly generalize to arbitrary regularization techniques; however, I view the argument as being suggestive that regularization techniques would also be insufficient to address the problems raised here.

Humans have no values… nor do any agent. Unless you make strong assumptions about their rationality. And depending on those assumptions, you get humans to have any values.

An agent with no clear preferences

There are three buttons in this world, B(0), B(1), and X, and one agent H.

B(0) and B(1) can be operated by H, while X can be operated by an outside observer. H will initially press button B(0); if ever X is pressed, the agent will switch to pressing B(1). If X is pressed again, the agent will switch back to pressing B(0), and so on. After a large number of turns N, H will shut off. That’s the full algorithm for H.

So the question is, what are the values/preferences/rewards of H? There are three natural reward functions that are plausible:

R(0), which is linear in the number of times B(0) is pressed.
R(1), which is linear in the number of times B(1) is pressed.
R(2)=I(E,X)R(0)+I(O,X)R(1), where I(E,X) is the indicator function for X being pressed an even number of times, I(O,X)=1−I(E,X) being the indicator function for X being pressed an odd number of times.

For R(0), we can interpret H as an R(0) maximising agent which X overrides. For R(1), we can interpret H as an R(1) maximising agent which X releases from constraints. And R(2) is the “H is always ...
When does rationality-as-search have nontrivial implications?
by nostalgebraist — 2018-11-04T22:42:01.452Z

(This originated as a comment on the post "Embedded World-Models," but it makes a broadly applicable point and is substantial enough to be a post, so I thought I'd make it a post as well.)

This post feels quite similar to things I have written in the past to justify my lack of enthusiasm about idealizations like AIXI and logically-omniscient Bayes. But I would go further: I think that grappling with embeddedness properly will inevitably make theories of this general type irrelevant or useless, so that "a theory like this, except for embedded agents" is not a thing that we can reasonably want. To specify what I mean, I'll use this paragraph as a jumping-off point:

Embedded agents don’t have the luxury of stepping outside of the universe to think about how to think. What we would like would be a theory of rational belief for situated agents which provides foundations that are similarly as strong as the foundations Bayesianism provides for dualistic agents.

Most "theories of rational belief" I have encountered -- including Bayesianism in the sense I think is meant here -- are framed at the level of an evaluator outside the universe, and have essentially no content when we try to transfer them to individual embedded agents. This is because these theories tend to be derived in the following way:

We want a theory of the best possible behavior for agents.
We have some class of "practically achievable" strategies S, which can actually be implemented by agents. We note that an agent's observations provide some information about the quality of different strategies s∈S. So if it were possible to follow a rule like R≡ "find the best s∈S given your observations, and then follow that s," this rule would spit out very good agent behavior.
Usually we soften this to a performance-weighted average rather than a hard argmax, but the principle is the same: if we could search over all of S, the rule R that says "do the search and then follow what it says" can be competitive with the very best s∈S. (Trivially so, since it has access to the best strategies, along with all the others.)
But usually R∉S. That is, the strategy "search over all practical strategies and follow the best ones" is not a practical strategy. But we argue that this is fine, since we are constructing a theory of ideal behavior. It do

...