trentmkelly/LessWrong-43k huggingface dataset:
Why it took so long to do the Fermi calculation right?
by Jan_Kulveit — 2018-07-02T20:29:59.338Z

This is a meta-level followup to an object level post about Dissolving the Fermi Paradox.

The basic observation of the paper is that when the statistics is done correctly to represent realistic distributions of uncertainty, the paradox largely dissolves.

The correct statistics is not that technically difficult: instead of point estimates, just take the distributions, reflecting the uncertainty (implied in the literature!)

There is sizeable literature about the paradox, stretching several decades. Just Wikipedia lists 22 hypothetical explanations, and it seems realistic, at least several hundred researchers spent some serious effort thinking about the problem.

It seems to me really important to reflect on this.

What's going on, why this inadequacy? (in general research)

And more locally, why did not this particular subset of the broader community, priding itself on use of Bayesian statistics, notice earlier?

(I have some hypotheses, but it seems better to just post it as an open-ended question)
The 10% Improvement Problem
by norswap — 2018-07-02T16:27:01.146Z
Link: https://twitter.com/mstrdrmr/status/1012865943578370048

There is something I call the 10% improvement problem. Basically, it's hard to track a change of a couple tens of percent in most abilities / states of being.

In some cases, there is a definition problem. What does it mean for your to be "twice" as happy as yesterday?

Assuming you do know, would you notice a 30% change in happiness? A 10% change?

Can you tell if you're being twice or half as productive? Maybe if you do the same thing everyday. But when faced with a new task, it's hard to tell whether the task is inherently difficult or if you're being uncharacteristically unproductive.

Even things that I feel should be eminently measurable, aren't. The effectiveness of a training program in building muscle, for instance. Heck, it's even hard to precisely measure muscle mass at 10% accuracy.

And that is saying nothing about confounding factors. You don't live a regimented life in a sterile box. Any comparison is necessarily going to be pears to apples.

And yet, 10% changes matter. Stack 10 of them and you've doubled whatever you were trying to improve. The problem is, how do you know what to stack?

Over a long period of time, these incremental changes are definitely noticeable. If you're on average 30% more effective every day, that will be noteworthy at the end of the year.

But you don't have the time to wait long periods for every single change you want to make. So you stack many of them, see if things improve on the long run, and accept that part of what you're doing may be pure cargo cult self-improvement.

This is an open problem, I have no solution to offer, but any help is appreciated.
Alignment Newsletter #13: 07/02/18
by Rohin Shah — 2018-07-02T16:10:02.539Z

Highlights

OpenAI Five (Many people at OpenAI): OpenAI has trained a team of five neural networks to play a particular set of Dota heroes in a mirror match (playing against the same set of heroes) with a few restrictions, and have started to beat amateur human players. They are aiming to beat a team of top professionals at The International in August, with the same set of five heroes, but without any other restrictions. Salient points:

The method is remarkably simple -- it's a scaled up version of PPO with training data coming from self-play, with reward shaping and some heuristics for exploration, where each agent is implemented by an LSTM.
There's no human data apart from the reward shaping and exploration heuristics.
Contrary to most expectations, they didn't need anything fundamentally new in order to get long-term strategic planning. I was particularly surprised by this. Some interesting thoughts from OpenAI researchers in this thread -- in particular, assuming good exploration, the variance of the gradient should scale linearly with the duration, and so you might expect you only need linearly more samples to counteract this.
They used 256 dedicated GPUs and 128,000 preemptible CPUs. A Hacker News comment estimates the cost at $2500 per hour, which would put the likely total cost in the millions of dollars.
They simulate 900 years of Dota every day, which is a ratio of ~330,000:1, suggesting that each CPU is running Dota ~2.6x faster than real time. In reality, it's probably running many times faster than that, but preemptions, communication costs, synchronization etc. all lead to inefficiency.
There was no explicit communication mechanism between agents, but they all get to observe the full Dota 2 state (not pixels) that any of the agents could observe, so communication is not really necessary.
A version of the code with a serious bug was still able to train to beat humans. Not encouraging for safety.
Alex Irpan covers some of these points in more depth in Quick Opinions on OpenAI Five.
Gwern comments as well.

My opinion: I might be more excited by an approach that was able to learn from human games (which are plentiful), and perhaps finetune with RL, in order to develop an approach that could generalize to more tasks in the future, where human data is available but a simulator is not. (Given the ridiculous sample complexity, pure RL with PPO ...
Are ethical asymmetries from property rights?
by KatjaGrace — 2018-07-02T03:00:00.567Z

These are some intuitions people often have:

You are not required to save a random person, but you are definitely not allowed to kill one
You are not required to create a person, but you are definitely not allowed to kill one
You are not required to create a happy person, but you are definitely not allowed to create a miserable one
You are not required to help a random person who will be in a dire situation otherwise, but you are definitely not allowed to put someone in a dire situation
You are not required to save a person in front of a runaway train, but you are definitely not allowed to push someone in front of a train. By extension, you are not required to save five people in front of a runaway train, and if you have to push someone in front of the train to do it, then you are not allowed.

Here are some more:

You are not strongly required to give me your bread, but you are not allowed to take mine
You are not strongly required to lend me your car, but you are not allowed to unilaterally borrow mine
You are not strongly required to send me money, but you are not allowed to take mine

The former are ethical intuitions. The latter are implications of a basic system of property rights. Yet they seem very similar. The ethical intuitions seem to just be property rights as applied to lives and welfare. Your life is your property. I’m not allowed to take it, but I’m not obliged to give it to you if you don’t by default have it. Your welfare is your property. I’m not allowed to lessen what you have, but I don’t have to give you more of it.

My guess is that these ethical asymmetries—which are confusing, because they defy consequentialism—are part of the mental equipment we have for upholding property rights.

In particular these well-known asymmetries seem to be explained well by property rights:

The act-omission distinction naturally arises where an act would involve taking someone else’s property (broadly construed—e.g. their life, their welfare), while an omission would merely fail to give them additional property (e.g. life that they are not by default going to have, additional welfare).
‘The asymmetry’ between creating happy and miserable people is because to create a miserable person is to give that person something negative, which is to take away what they have, while creating a happy person is giving that person something extra.
Person-affecting views arise because birth

...
LW Update 2018-07-01 – Default Weak Upvotes
by Raemon — 2018-07-01T23:47:06.434Z

Git Commit: 6a0c7bf54a29f2909ab50fc76da3b43854d0e112

We deployed recently, mostly with under-the-hood bugfixes and refactors that are aimed at making it easier to fork the LW codebase and restyle it.

Two concrete user-facing changes are:

Comments are now small-upvote by default. Posts are still large-upvote by default. (Later, we intend to add some limitations to strong-upvoting – mostly likely reducing its power the more you use it. So people will be free to strong-upvote their own comments if they think it's important, and rather than being seen as "tooting your own horn too loudly", the intent will be that you have a limited power to strong upvote, and you are encouraged to use it on whatever you think is most important, whether it's your own content or someone else's)
We added a "hide low karma posts" checkbox to the daily page. We'll improve this over time (making it clearer what it does and maybe giving users more control over it), but for now just wanted obvious spam and nonsense to not take up people's attention. (the toggle removes posts with -10 or less karma)
Paul's research agenda FAQ
by zhukeepa — 2018-07-01T06:25:14.013Z

I think Paul Christiano’s research agenda for the alignment of superintelligent AGIs presents one of the most exciting and promising approaches to AI safety. After being very confused about Paul’s agenda, chatting with others about similar confusions, and clarifying with Paul many times over, I’ve decided to write a FAQ addressing common confusions around his agenda.

This FAQ is not intended to provide an introduction to Paul’s agenda, nor is it intended to provide an airtight defense. This FAQ only aims to clarify commonly misunderstood aspects of the agenda. Unless otherwise stated, all views are my own views of Paul’s views. (ETA: Paul does not have major disagreements with anything expressed in this FAQ. There are many small points he might have expressed differently, but he endorses this as a reasonable representation of his views. This is in contrast with previous drafts of this FAQ, which did contain serious errors he asked to have corrected.)

For an introduction to Paul’s agenda, I’d recommend Ajeya Cotra’s summary. For good prior discussion of his agenda, I’d recommend Eliezer’s thoughts, Jessica Taylor’s thoughts (here and here), some posts and discussions on LessWrong, and Wei Dai’s comments on Paul’s blog. For most of Paul’s writings about his agenda, visit ai-alignment.com.

0. Goals and non-goals

0.1: What is this agenda trying to accomplish?

Enable humans to build arbitrarily powerful AGI assistants that are competitive with unaligned AGI alternatives, and only try to help their operators (and in particular, never attempt to kill or manipulate them).

People often conceive of safe AGIs as silver bullets that will robustly solve every problem that humans care about. This agenda is not about building a silver bullet, it’s about building a tool that will safely and substantially assist its operators. For example, this agenda does not aim to create assistants that can do any of the following:

They can prevent nuclear wars from happening
They can prevent evil dictatorships
They can make centuries’ worth of philosophical progress
They can effectively negotiate with distant superintelligences
They can solve the value specification problem

On the other hand, to the extent that humans care about these things and could make them happen, this agenda lets us build AGI assistants that can substantially assist humans achieve these things. For example, a team of 1,0...
Another take on agent foundations: formalizing zero-shot reasoning
by zhukeepa — 2018-07-01T06:12:57.414Z

After spending

more time thinking

about MIRI’s agenda, I’ve come upon another framing of it, which I’m coining zero-shot reasoning. [1]

This post is a distillation of my intuitions around zero-shot reasoning. These views should be taken as my own, and not MIRI’s.

A quick summary of this post:

“Zero-shot reasoning” refers to the ability to get things right on the first try, no matter how novel or complicated.
In simple domains, like mathematics, zero-shot reasoning is fully captured by formal verification of proofs. In more general domains, zero-shot reasoning requires an extension of formal verification that can be applied to real-world plans.
MIRI-esque philosophical work is necessary to extend formal verification to more general domains.
A formal account of zero-shot reasoning will likely be unimportant for aligning the world’s first AGIs, but will likely be essential for aligning a recursively self-improving AGI.
Humanity will most likely end up building a recursively self-improving AGI (plausibly because of insufficient coordination around not building a recursively self-improving AGI).
We can probably delegate much of the work of formalizing zero-shot reasoning to a post-AGI society, but working on zero-shot reasoning today nevertheless substantially increases the odds that our first recursively self-improving AGI is aligned.

What is zero-shot reasoning?

Few-shot reasoning vs zero-shot reasoning

The world is largely chaotic and unpredictable, yet humans can ideate and successfully execute on hugely conjunctive plans with many moving parts, many of which are the first of their kind. We can ship massive software projects and send rockets to space. Some people can build billion-dollar companies over and over again.

On the other hand, it’s clear that human abilities to do this are limited. Big software projects invariably have critical bugs and security flaws. Many of our spacecraft have exploded before making it to space. Most people can’t build a company to save their lives, and most successful entrepreneurs fail when building a second company.

Native human software is capable of doing something I’ll call few-shot reasoning—when performing complex reasoning and planning to accomplish something novel, humans can usually get it right after a few rounds of iteration. The more dissimilar this reasoning is to prior reasoning they've done, the more rounds o...
Goodhart Taxonomy: Agreement
by Ben Pace — 2018-07-01T03:50:44.562Z

"When a measure becomes a target, it ceases to be a good measure."

-Goodhart’s Law

If you spend a while talking with someone you disagree with and end up agreeing, this is a sign you are both reasoning and communicating well - one of the primary uses of good reasoning is resolving disagreement. However, if you use agreement as your main proxy for good reasoning, some bad things might happen.

Scott Garrabrant has helpfully laid out four different models of how things can go wrong if you optimise for the proxy really hard, a phenomenon known as ‘goodharting’ (based on Goodhart’s Law that any proxy when optimised for hard, stops being a good proxy). I want to take a look at each model and see what it predicts for the real world, in the domain of agreement.

Regressional

First, you can fall prey to regressional goodharting. This is when the proxy you’re optimising for is a good measure of the thing you actually care about, but plus some noise (i.e. other uncorrelated variables), and the examples that maximise the sum of these are the examples that maximise the noise. I can think of three ways this could happen: misunderstanding, spurious correlation, and shared background models.

Misunderstanding

is the simple idea that, of the times when you most agree with someone, you misunderstood each other (e.g. were using words differently). Especially if it’s an important topic and most people disagree with you, suddenly the one person who gets you seems to be the best reasoner you know (if you’re regressional goodharting).

Spurious correlation

is like I described in A Sketch of Good Communication - two people who have different AI timelines can keep providing each other with new evidence until they have the same 50th percentile date, but it may turn out that they have wholly different causal models behind, and thus don’t meaningfully agree around AI x-risk. This is different from misunderstanding, because you heard the person correctly when they stated their belief.

And

shared background models

happens like this: You decide that the people who are good reasoners and communicators are those who agree with you a lot on complex issues after disagreeing initially. Often this heuristic ends up finding people who are good at understanding your point of view, and updating when you make good arguments. But if you look at the people who agree with you the most of these people, you’...
Overcoming Clinginess in Impact Measures
by TurnTrout — 2018-06-30T22:51:29.065Z

It may be possible to use the concept of a causal counterfactual (as formalized by Pearl) to separate some intended effects from some unintended ones. Roughly, "follow-on effects" could be de fined as those that are causally downstream from the achievement of the goal... With some additional work, perhaps it will be possible to use the causal structure of the system's world-model to select a policy that has the follow-on effects of the goal achievement but few other effects.

Taylor et al., Alignment for Advanced Machine Learning Systems

In which I outline a solution to the clinginess problem and illustrate a potentially-fundamental trade-off between assumptions about the autonomy of humans and about the responsibility of an agent for its actions.

Consider two plans for ensuring that a cauldron is full of water:

Filling the cauldron.
Filling the cauldron and submerging the surrounding room.

All else equal, the latter plan does better in expectation, as there are fewer ways the cauldron might somehow become not-full (e.g., evaporation, and the minuscule loss of utility that would entail). However, the latter plan "changes" more "things" than we had in mind.

Undesirable maxima of an agent's utility function often seem to involve changing large swathes of the world. If we make "change" costly, that incentivizes the agent to search for low-impact solutions. If we are not certain of a seed AI's alignment, we may want to implement additional safeguards such as impact measures and off-switches.

I designed an impact measure called whitelisting - which, while overcoming certain weaknesses of past approaches, is yet vulnerable to

Clinginess

An agent is clingy when it not only stops itself from having certain effects, but also stops you.

...

Consider some outcome - say, the sparking of a small forest fire in California. At what point can we truly say we didn't start the fire?

I immediately and visibly start the fire.
I intentionally persuade someone to start the fire.
I unintentionally (but perhaps predictably) incite someone to start the fire.
I set in motion a moderately-complex chain of events which convince someone to start the fire.
I provoke a butterfly effect which ends up starting the fire.

Taken literally, I don't know that there's actually a significant difference in "responsibility" between thes

...
The Values-to-Actions Decision Chain
by Remmelt — 2018-06-30T21:52:02.532Z

Update 2022: Looking back on younger Remmelt's writing on decision-making, it is useful but cocky. It explained what I now call 'monitoring values' for carving out states of the world to decide between, but not how choice arises from 'deep embedded values'.

This post contains:
1. an exposition of a high-level model
2. some claims on what this might mean strategically for the EA community

Effective Altruism is

challenging

. Some considerations require you to zoom out to take an eagle-eye view across a vast landscape of possibility (e.g. to research moral uncertainty), while other considerations require you to swoop in to see the details (e.g. to welcome someone new). The distance from which you're looking down at a problem is the

construal level

you're thinking at.

People involved in the EA community can gain a lot from improving their grasp of construal levels – the levels they or others naturally incline towards and even the level they're operating at any given moment (leading, for instance, to less disconnect in conversations). A lack of construal-level sense combined with a lack of

how-to-interact-in-large-social-networks

sense has left a major blindspot in how we collectively make decisions, in my view.

The Values-to-Actions Decision Chain (in short: 'decision chain') is an approach for you to start solving the awe-inspiring problem of 'doing the most good' by splitting it into a series of decisions you will make from high to low construal levels (creating in effect, a

hierarchy of goals

). It is also a lens through which you can more clearly see your own limits to doing this and compensate by coordinating better with others. Beware though that it’s a blurry and distorted lens – this post itself is a high construal-level exercise and many important nuances have been eliminated in the process. I'd be unsurprised if I ended up revising many of the ideas and implications in here after another year of thinking.

Chain up

To illustrate how to use the V2ADC (see diagram below)

:

Suppose an ethics professor decided that...

...from the point of view of the universe... (

meta-ethics

)
...hedonistic utilitarianism made sense... (

moral theories

)
...and that therefore humans in developing countries should not suffer unnecessarily... (

worldview

)
...leading him to work on reducing global poverty... (

focus area

)
...by reducing a variety of easily treatable diseases (

problems

)
...by advo

...
Meta: IAFF vs LessWrong
by Vanessa Kosoy — 2018-06-30T21:15:56.000Z

It seems that currently many or even most researchers started posting on LessWrong.com instead of IAFF. I am troubled by this for two reasons.

I hoped that eventually IAFF will draw in all of the AI alignment research community rather than only the MIRI-sphere. Moving to LessWrong seems like a step in the opposite direction. However, if everyone else think differently, I will go along with it.

More critically, LessWrong.com is currently unusable for this purpose from my point of view. This is because there is no way to see only the posts relevant to AI alignment. For me, it is not practical to follow all of the new posts, and reading all the AI alignment relevant posts soon after they appear. Instead, I occasionally take a break from my own work and catch up with what everyone else is doing. But now it became impossible without combing through a huge list of irrelevant things. I would be surprised if I am the only one with this problem.

Let's discuss the situation, please. I wish that either we get some solution for seeing only the AI alignment posts on LessWrong really soon, or that people continue at least posting the links on IAFF.
Dissolving the Fermi Paradox, and what reflection it provides
by Jan_Kulveit — 2018-06-30T16:35:35.171Z
Link: https://arxiv.org/abs/1806.02404

While the argument was posted on LessWrong previously, now it has the neat form of a paper on arXive by Anders Sandberg, Eric Drexler and Toby Ord

TL;DR version: the use of Drake-like equations, with point estimates of highly uncertain parameters, is wrong. Extant scientific knowledge corresponds to uncertainties that span multiple orders of magnitude.

When the statistics is done correctly to represent realistic distributions of uncertainty in the literature, "people who take the views of most members of the research community seriously should ascribe something like a one in three chance to being alone in the galaxy and so should not be greatly surprised by our lack of evidence of other civilizations. The probability of N <10^−10 (such that we are alone in the observable universe) is 10%. "

From the conclusions, when the priors are updated

When we update this prior in light of the Fermi observation, we find a substantial probability that we are alone in our galaxy, and perhaps even in our observable universe (53%–99.6% and 39%–85% respectively). ’