trentmkelly/LessWrong-43k huggingface dataset:
I Vouch For MIRI
by Zvi — 2017-12-17T17:50:00.236Z

Another take with more links: AI: A Reason to Worry, A Reason to Donate

I have made a $10,000 donation to the Machine Intelligence Research Institute (MIRI) as part of their winter fundraiser. This is the best organization I know of to donate money to, by a wide margin, and I encourage others to also donate. This belief comes from a combination of public information, private information and my own analysis. This post will share some of my private information and analysis to help others make the best decisions.

I consider AI Safety the most important, urgent and under-funded cause. If your private information and analysis says another AI Safety organization is a better place to give, give to there. I believe many AI Safety organizations do good work. If you have the talent and skills, and can get involved directly, or get others who have the talent and skills involved directly, that’s even better than donating money.

If you do not know about AI Safety and unfriendly artificial general intelligence, I encourage you to read about them. If you’re up for a book, read this one.

If you decide you care about other causes more, donate to those causes instead, in the way your analysis says is most effective. Think for yourself, do and share your own analysis, and contribute as directly as possible.

I

I am very confident in the following facts about artificial general intelligence. None of my conclusions in this section require my private information.

Humanity is likely to develop artificial general intelligence (AGI) vastly smarter and more powerful than humans. We are unlikely to know that far in advance when this is about to happen. There is wide disagreement and uncertainty on how long this will take, but certainly there is substantial chance this happens within our lifetimes.

Whatever your previous beliefs, the events of the last year, including AlphaGo Zero, should convince you that AGI is more likely to happen, and more likely to happen soon.

If we do build an AGI, its actions will determine what is done with the universe.

If the first such AGI we build turns out to be an unfriendly AI that is optimizing for something other than humans and human values, all value in the universe will be destroyed. We are made of atoms that could be used for something else.

If the first such AGI we build turns out to care about humans and human values, the universe will be a pl...
Pascal’s Muggle Pays
by Zvi — 2017-12-16T20:40:00.878Z

Reply To (Eliezer Yudkowsky): Pascal’s Muggle Infinitesimal Priors and Strong Evidence

Inspired to Finally Write This By (Lesser Wrong): Against the Linear Utility Hypothesis and the Leverage Penalty.

The problem of Pascal’s Muggle begins:

Suppose a poorly-dressed street person asks you for five dollars in exchange for doing a googolplex’s worth of good using his Matrix Lord powers.

“Well,” you reply, “I think it very improbable that I would be able to affect so many people through my own, personal actions – who am I to have such a great impact upon events? Indeed, I think the probability is somewhere around one over googolplex, maybe a bit less. So no, I won’t pay five dollars – it is unthinkably improbable that I could do so much good!”

“I see,” says the Mugger.

At this point, I note two things. I am not paying. And my the probability the mugger is a Matrix Lord is much higher than five in a googolplex.

That looks like a contradiction. It’s positive expectation to pay, by a lot, and I’m not paying.

Let’s continue the original story.

A wind begins to blow about the alley, whipping the Mugger’s loose clothes about him as they shift from ill-fitting shirt and jeans into robes of infinite blackness, within whose depths tiny galaxies and stranger things seem to twinkle. In the sky above, a gap edged by blue fire opens with a horrendous tearing sound – you can hear people on the nearby street yelling in sudden shock and terror, implying that they can see it too – and displays the image of the Mugger himself, wearing the same robes that now adorn his body, seated before a keyboard and a monitor.

“That’s not actually me,” the Mugger says, “just a conceptual representation, but I don’t want to drive you insane. Now give me those five dollars, and I’ll save a googolplex lives, just as promised. It’s easy enough for me, given the computing power my home universe offers. As for why I’m doing this, there’s an ancient debate in philosophy among my people – something about how we ought to sum our expected utilities – and I mean to use the video of this event to make a point at the next decision theory conference I attend. Now will you give me the five dollars, or not?”

“Mm… no,” you reply.

“No?” says the Mugger. “I understood earlier when you didn’t want to give a random street person five dollars based on a wild story with no evidence behind it. But now I’ve offered you e...
Improvement Without Superstition
by Zachary Jacobi — 2017-12-16T20:01:18.647Z

When you make continuous, incremental improvements to something, one of two things can happen. You can improve it a lot, or you can fall into superstition. I'm not talking about black cats or broken mirrors, but rather humans becoming addicted to whichever steps were last seen to work, instead of whichever steps produce their goal.

I've seen superstition develop first hand. It happened in one of the places you might least expect it – in a biochemistry lab. In the summer of 2015, I found myself trying to understand which mutants of a certain protein were more stable than the wildtype. Because science is perpetually underfunded, the computer that drove the equipment we were using was ancient and frequently crashed. Each crash wiped out an hour or two of painstaking, hurried labour and meant we had less time to use the instrument to collect actual data. We really wanted to avoid crashes! Therefore, over the course of that summer, we came up with about 12 different things to do before each experiment (in sequence) to prevent them from happening.

We were sure that 10 out of the 12 things were probably useless, we just didn't know which ten. There may have been no good reason that opening the instrument, closing, it, then opening it again to load our sample would prevent computer crashes, but as far as we could tell when we did that, the machine crashed far less. It was the same for the other eleven. More self-aware than I, the graduate student I worked with joked to me: "this is how superstitions get started" and I laughed along. Until I read two articles in The New Yorker.

In The Score (How Childbirth Went Industrial), Dr. Atul Gawande talks about the influence of the Apgar score on childbirth. Through a process of continuous competition and optimization, doctors have found out ways to increase the Apgar scores of infants in their first five minutes of life – and how to deal with difficult births in ways that maximize their Apgar scores. The result of this has been a shocking (six-fold) decrease in infant mortality. And all of this is despite the fact that according to Gawande, "[in] a ranking of medical specialties according to their use of hard evidence from randomized clinical trials, obstetrics came in last. Obstetricians did few randomized trials, and when they did they ignored the results."

Similarly, in The Bell Curve (What...
Truth is Symmetric
by ErnestScribbler — 2017-12-16T14:52:27.558Z

The Taj Mahal is symmetric with respect to right-left reflection. That is to say, if you create a mirror image of it that flips right and left, you get the exact same thing. Therefore, if as we're standing outside looking at it my friend Jen tells me that she thinks the left side of the Taj Mahal is much nicer than the right side, or that it's objectively better, I would be suspicious. That's because if we created the mirror image of the Taj Mahal, the right side would be on the left, Jen would not be able to tell the difference and so would say that the left side is nicer than the right one, but it would secretly be the right side she was speaking about. Muwahahahaa, it can't be right. At least if all that determines niceness isn't about where something is placed but what it's composed of - the left side can be nicer than the right, but not just because it's on the left.

Now for three similar examples, each trivial on its own but illustrating a broader point.

1) Most Israeli Jews think Israel is broadly in the right in the Israeli-Palsetinian conflict. Most Palestinians think that the Palestinians are broadly in the right. What bothers me about this is that both factions think that their faction is objectively in the right, that is, independently of who considers the question. But if it's independently of who considers the question, how come people who were born on one side of the aisle mostly think one thing, and people on the other side think another thing. This screams inconsistency and irrationality, since a factor that shouldn't affect what's objectively right affects what most people think.

2) I'm usually more left-wing in my opinions. So when I consider whether some extreme right-wing statement falls within the limits of free speech, I try to think whether I would consider an equally-extreme left-wing statement to fall within the limits of free speech. I imagine a world where that opinion is banned from being expressed, and whether I'd be happy with a world like that. If for the right-wing statement my feeling is "This statement should be banned!" but for the left-wing statement I feel "It would be a terrible world if that statement were banned" then I suspect myself of being biased, since a factor that shouldn't affect what falls within free speech (the alignment of th...
The Extinction Dilemma
by DragonGod — 2017-12-16T13:52:37.355Z

Inspiration came from Against the Linear Utility Hypothesis and the Leverage Penalty

Place a value on the utility of a utopia with 1 human, let's call this X.

Place a value on the negutility of all of humanity going extinct. Let's call this Z.

Decide if your utility function is linear in number of lives saved.

Place a value on the utility of say 10100 humans achieving a perfect utopia. Call this K.

Omega offers you a bet. This bet has a 50% chance of humanity reaching a Kardashev type V civilisation (colonising the multiverse) and a perfect utopia, and a 50% chance of human extinction this instant.

Omega is an omnipotent entity that always tells the truth, you know (and believe this), etc.

Do you accept the bet?

What about 1:3 odds?
What about 3:1 odds?
What is the highest probability of extinction at which you would accept Omega's offer?

What does this say about your utility function?
Book Review: The Complacent Class
by Zvi — 2017-12-16T13:10:00.615Z

Epistemic Status: My read. The mind of another is always tricky. At a minimum, Tyler would not agree with many of my framing and word choices.

The Book: The Complacent Class

Slide Deck: The Complacent Class and the Philosophy of Tyler Cowen, by my friend Anthony. This provides a good summary of his main facts and points on the surface level.

Tyler Cowen explicitly believes, and has said many times, that for society to succeed we need to believe untrue things. In his model, a society’s beliefs need to be chosen for their impact on society first. A lot of that impact is in raising the status of behaviors we want to see more, and lowering the status of behaviors we want to see less. He is often explicit about this.

Accuracy is a secondary consideration.

He might (not his example) prefer we think french fries are not delicious. We would eat less of them and be better off, even if french fries are delicious.

This is distinct from the type of self-deception Robin Hanson talks about in his book The Elephant in the Brain.

This approach is tricky. How do you avoid believing arbitrary false facts? How do we choose the actions with the best impact without an accurate model of the world? How do we engage in chains of reasoning?

This is a complex problem.

His solution is the Straussian reading.

The surface text has the first level message we want people to believe and act on, to make the world a better place. We must look deeper for the real, often opposite, second level message. We can then have a third level well-hidden discourse about what the proper first level message ought to be.

Tyler explicitly endorses looking for Straussian readings in a wide variety of texts. His philosophy implies the need for us to believe that which is not, while simultaneously being able to think objectively about the consequences of actions.

Tyler even links enthusiastically to many reviews of his book that claim his book is saying things Tyler explicitly disavows! In particular, he highlighted this truly epic masterpiece of extrapolation.

Tyler intending a Straussian reading of his own books is not only not a strange hypothesis. It should be our prior.

His first level message is that you (and America) are too complacent. You should not be complacent. To survive we must become less complacent.

His second level message is that while it is bad for society if its people are complacent, it is in your intere...
A New Sequence on Rationality
by DragonGod — 2017-12-16T11:08:26.133Z

In August, I asked "What is rational"? I wasn't satisfied with any answer I received, and then decided to answer it myself; I began developing my own theory of rationality from first principles. I wrote and scrapped two drafts, and after 4 months, I've finally started the path to an answer.

Over the coming year, I plan to write this sequence, developing my theory of rationality to an acceptable level. My conception of rationality was developed from first principles, and the principles I adopted may differ (substantially even) from Lesswrong's or the mainstream's principles, so it may not be what you're used to, but I nevertheless expect it to prove useful.

Furthermore, because I'm taking a first principles approach, I would of necessity reinvent several wheels (insomuch as these wheels are useful to rationality), so a lot (possibly even the majority) of this sequence may be unoriginal. Nevertheless, where I do diverge from other conceptions of rationality is significant enough that I do not expect the two to be reconcilable.

I'll take a more free-form and less structured approach to writing the sequence, so it wouldn't be much of a sequence (in the usual sense of the term), and more of an interrelated bunch of posts. I would structure the posts into a logical order after I'm done with the sequence, and their chronological order may not be the best order in which to read them. Trying to structure the posts in logical order would probably require me to write the entire thing at once, and from my two failed attempts at doing that (and the subsequent deflation in energy which prevented me from making progress on that front for around a few weeks to a month) I don't want to take that approach. So I'll write in an unstructured style, pushing the components in the order I finish them. Upon completion, I'll stitch together all the components into a coherent whole.

I'm incredibly bad at estimating completion times for projects I undertake, so I'll only note that I plan to start next year, and I'm not sure when I'll finish.

Chronological Sequence

To be written.
Security services relationship to social movements
by  — 2017-12-15T22:56:46.602Z

As there has been talk of the necessities of a security mindset for forming an agi research project, I thought it worthwhile to share this so that rationalists have some information about the types of forces that may be at play, if someone decided to form such a group.

I'm just going to be pulling from wikipedia, as I hope it is relatively factual. Lots of searches pull up non-mainstream media, which I do not trust.

From the wiki page on UK undercover policing relationships scandal.

Around the end of 2010 and during 2011, it was disclosed in UK media, that a number of undercover police officers had, as part of their 'false persona', entered into intimate relationships with members of targeted groups and in some cases proposed marriage or fathered children with protesters who were unaware their partner was a police officer in a role as part of their official duties.[1] Various legal actions followed, including eight women who took action against the Metropolitan Police and the Association of Chief Police Officers (ACPO), stating they were deceived into long-term intimate relationships by five officers, including Mark Kennedy, the first officer to be identified as such, who was publicly identified on 21 October 2010[2][3] as infiltrating social and environmental justice campaigns,[4][5] and Mark Kennedy himself who claimed in turn that he had been incompetently handled by his superiors and denied psychological counselling.[1] According to The Guardian,[6] Kennedy sued the police for ruining his life and failing to "protect" him from falling in love with one of the environmental activists whose movement he infiltrated.

Although the units had been previously disbanded, other cases continued to emerge. In 2015 a public inquiry under a senior judge was announced. In November 2015 the Metropolitan Police published an unreserved apology in which it exonerated and apologised to those women who had been deceived and stated the methodology had constituted abuse and a "gross violation" with severely harmful effects, as part of a settlement of their cases. In 2016 new cases continued to come to light.

From the wiki page on Project Minaret

Project MINARET was a sister project to Project SHAMROCK operated by the National Security Agency (NSA), which, after intercepting electronic communications that contained the names of predesignated US cit

...
Why Bayesians should two-box in a one-shot
by PhilGoetz — 2017-12-15T17:39:32.491Z

Consider Newcomb's problem.

Let 'general' be the claim that Omega is always right.

Let 'instance' be the claim that Omega is right about a particular prediction.

Assume you, the player, are not told the rules of the game until after Omega has made its prediction.

Consider 2 variants of Newcomb's problem.

1. Omega is a perfect predictor. In this variant, you assign a prior of 1 to P(general). You are then obligated to believe that Omega has correctly predicted your action. In this case Eliezer's conclusion is correct, and you should one-box. It's still unclear whether you have free will, and hence have any choice in what you do next, but you can't lose by one-boxing.

But you can't assign a prior of 1 to P(general), because you're a Bayesian. You derive your prior for P(general) from the (finite) empirical data. Say you begin with a prior of 0.5 before considering any observations. Then you observe all of Omega's N predictions, and each time, Omega gets it right, and you update:

P(general | instance) = P(instance | general) P(instance) / P(general)
= P(instance) / P(general)

Omega would need to make an infinite number of correct predictions before you could assign a prior of 1 to P(general). So this case is theoretically impossible, and should not be considered.

2. Omega is a "nearly perfect" predictor. You assign P(general) a value very, very close to 1. You must, however, do the math and try to compare the expected payoffs, at least in an order-of-magnitude way, and not just use verbal reasoning as if we were medieval scholastics.

The argument for two-boxing is that your action now can't affect what Omega did in the past. That is, we are using a model which includes not just P(instance | general), but also the interaction of your action, the contents of the boxes, and the claim that Omega cannot violate causality. P ( P($1M box is empty | you one-box) = P($1M box is empty | you two-box) ) >= P(Omega cannot violate causality), and that needs to be entered into the computation.

Numerically, two-boxers claim that the high probability they assign to our understanding of causality being basically correct more than cancels out the high probability of Omega being correct.

The argument for one-boxing is that you aren't entirely sure you understand physics, but you know ...
Youtube channel devoted to the art of rationality
by MrMind — 2017-12-15T11:42:14.759Z

For quite some time I've considered creating a Youtube channel devoted to rationality.

I live in Italy, and the political situation here bears some similarities with the US': we've seen the rise of populist figures accompanied by the debate around fake news and post-truth, usually created with the purpose of bringing more votes to those politicians.

When a fake news is exposed, the standard answer is deploring the lack of critical thinking, and blaming school for failing to teach it. But, as far as I know, schools can only teach programs that are approved by the Department of Education, which 'institutionalize' living intellectual efforts like science and history. So far, no one has institutionalized critical thinking. In a sense, you could say that it cannot be completely institutionalized.

Sometimes, it's expected that philosophy classes teaches rationality, or at least thatteaches the pupils how to think. I know that LW users will appreciate the irony.

Anyway, philosophy is taught only in high school, and especially in poorer area not enough people reach that level of education, and is only taught within some 'orientation' (here in high school you cannot compose your courses as you wish, you have 'pre-packaged' sets of courses with a scientific or humanistic or artistic orientations).

In the end, the situation is that everybody is complaining about the lack of rationality teaching, but so far no-one has put himslef at creating such content.

Cue me.

I've been a member of LW since 2010, and I've always been interested in rationality, philosophy and AI. I'm not the smartest guy, so it's quite useless for me to be in academia or other research institution, but I've been told repeatedly that I'm good at explaining complicated things in lay terms. This seems a perfect combination of features for me to create content, aimed at the Italian situation and the Italian speaking audience, that tries to popularize rationality, Bayesianism and critical thinking.

With this post I would like to call for topics. What topic could I treat and how?

I already thought about something: locating rationality in the midst of and in contrast with other approaches; a non-technical survey of Bayesian probability; excercise in critical thinking (discussing a hot topic and trying to keep your identity small); atheism...