trentmkelly/LessWrong-43k huggingface dataset:
Fundamentals of Formalisation level 1: Basic Logic
by  — 2018-05-04T13:01:50.998Z

Followup to Soon: a weekly AI Safety prerequisites module on LessWrong

(note: we have a study group running a week ahead of this series that adds important content. It turns out that to get that content ready on a weekly basis, we would have to cut corners. We prefer quality over speed. We also like predictability. So we decided to cut us some slack and publish every 2 weeks instead for the time being)

(relatedly, the challenges section of this week is not yet complete at the time of posting, so completing those will not yet imply a full understanding of the content)

Basic logic

The big ideas:

Sentential Logic

Truth Tables

Predicate Logic

Methods of Mathematical Proof

To move to the next level you need to be able to:

Translate informal arguments into formal logic.

Evaluate an argument as either valid or invalid.

Explain how to prove an implication/conditional, a conjunction, a disjunction, and a negation and know what this looks like informally (i.e. in words and not symbols).

Why this is important:

This builds the basic knowledge you need to be able to produce and understand mathematical proof. A firm foundation in how logical machinery operates is the best way to be assured that a proof you produce or read is correct. This also teaches the basic methods by which a proof is produced.

Without further ado: you can find the first lesson on our course platform.

Every week you have 2 options: do the whole thing, or skip to the "challenges" in the end. The latter option is for those people that suspect they already know the subject. It serves as a means of verifying that assumption.

We hope this track will facilitate aspiring AI Safety researchers in their studies. If it leads to even one success story, it will have been worth it. Maybe that success story is you!

Happy studying
AGI Safety Literature Review (Everitt, Lea & Hutter 2018)
by Kaj_Sotala — 2018-05-04T08:56:26.719Z
Link: https://arxiv.org/abs/1805.01109

Abstract. The development of Artificial General Intelligence (AGI) promises to be a major event. Along with its many potential benefits, it also raises serious safety concerns (Bostrom, 2014). The intention of this paper is to provide an easily accessible and up-to-date collection of references for the emerging field of AGI safety. A significant number of safety problems for AGI have been identified. We list these, and survey recent research on solving them. We also cover works on how best to think of AGI from the limited knowledge we have today, predictions for when AGI will first be created, and what will happen after its creation. Finally, we review the current public policy on AGI.
Bayes' Law is About Multiple Hypothesis Testing
by abramdemski — 2018-05-04T05:31:23.024Z

I've called outside view the main debiasing technique, and I somewhat stand by that, not only because base-rate neglect can account for a variety of other biases, but also because outside view is about working on the policy level, which you have to do to implement other debiasing strategies.

Nonetheless, I am here today to tell you why the Method of Multiple Working Hypotheses is a central technique. T. C. Chamberlin wrote about it in 1897. More recently, Heuer discusses a very similar technique in Psychology of Intelligence Analysis, which served for a time as the debiasing handbook for the CIA. Heuer called his version Analysis of Competing Hypotheses.

(So, we could call it Method of Multiple Hypotheses (MMH), Analysis of Competing Hypotheses (ACH), or perhaps Analysis of Alternative Hypotheses (AAH) -- it seems doomed to be abbreviated as some variety of grunt.)

Heuer found that asking people to articulate the assumptions behind their assertions did not work very well -- analysts tend to insist that their conclusions follow directly from looking at the data, with no assumptions in between. (It is difficult to see the lens which you use to see!) However, if you instead ask people to compare their conclusions to other possibilities, they start noticing the assumptions which pointed them in one direction rather than another.

In order to make it stick in people's heads, I want to explain why it is just about inevitable from Bayes' Law.

Bayes' Law compares hypotheses to each other in terms of their likelihood ratios, balanced by the priors. Testing a single hypothesis feels meaningful, perhaps because in logical/deterministic cases we sometimes can prove or disprove something on its own. In the general case, though, we have to compare a hypothesis to alternatives to say anything meaningful. It's much like trying to evaluate a plan in isolation -- you can figure out a probability of success, or an expected value, but this is meaningless in isolation. You need to compare it to alternatives to know anything about whether you want to enact the plan. And, not just any alternatives; the best alternatives you can come up with.

Similarly, it only makes sense to evaluate hypotheses by looking at their relative likelihoods in comparison to a number of other hypotheses, and relative prior probabilities.

This is the thing which null hypothesis testing i... ,,,,,,,,,,,,,,,
Words, Locally Defined
by abramdemski — 2018-05-03T23:26:31.203Z

Related to Distinctions of the Moment, The Problematic Third-Person Perspective, and a disagreement with Said.

In programming, some variables have local scope: they have meaning in the context of a particular function you're writing, or a loop, etc; outside of that context, they may have no meaning, or they may have an entirely different local meaning somewhere else. Others have global scope: they mean the same thing throughout a program.

The first thing I want to get across is that we can make a similar distinction with words.

The second thing I want to get across is that the distinction is fuzzy. Everything is really "local" scope, but for different values of "local".

The third thing I want to get across is that this is an instance of a more general pattern. The temptation to insist that all words are globally defined comes from a way of thinking about disagreements which gives rise to many similar mistakes.

First things first.

"Global scope" obviously refers to dictionary definitions and to common usage. The most obvious cases for "local scope" of words are things like "the school" (which refers to whichever school is most relevant to the people talking), "the office", etc. However, I think a lot more words are subtly re-defined based on what is most useful for the context.

In Distinctions of the Moment, I argued that I want to reserve certain words for on-the-fly redefinition. For example, "thought" and "feeling" are vague words which are often used to point at a dichotomy. However, the space of possible dichotomies here is large. In one conversation, I might want to use "thought" for "linguistic mental representation" (inner monologue in the phonological loop), and "feeling" for "inarticulate mental representation" (something which you can't put into words). In another conversation, "thought" might refer to "mental content" (including all of working memory, not only the phonological loop), and "feeling" to sensations in the body including touch and kinesthetic sense (proprioception). And so on.

If someone steps into the conversation and objects "that isn't what thought vs feeling means to me!" I'll listen to their proposed definition, but I'll evaluate its usefulness in terms of ...
Confounded No Longer: Insights from 'All of Statistics'
by TurnTrout — 2018-05-03T22:56:27.057Z

Using fancy tools like neural nets, boosting and support vector machines without understanding basic statistics is like doing brain surgery before knowing how to use a bandaid.

Larry Wasserman

Foreword

For some reason, statistics always seemed somewhat disjoint from the rest of math, more akin to a bunch of tools than a rigorous, carefully-constructed framework. I am here to atone for my foolishness.

This academic term started with a jolt - I quickly realized that I was missing quite a few prerequisites for the Bayesian Statistics course in which I had enrolled, and that good ol' AP Stats wasn't gonna cut it. I threw myself at All of Statistics, doing a good number of exercises, dissolving confusion wherever I could find it, and making sure I could turn each concept around and make sense of it from multiple perspectives.

I then went even further, challenging myself during the bits of downtime throughout my day to do things like explain variance from first principles, starting from the sample space, walking through random variables and expectation - without help.

All of Statistics

1: Introduction

2: Probability

In which sample spaces are formalized.

3: Random Variables

In which random variables are detailed and a multitude of distributions are introduced.

Conjugate Variables

Consider that a random variable X is a function X:Ω→R. For random variables X,Y, we can then produce conjugate random variables XY,X+Y, with

(XY)(ω)=X(ω)Y(ω)(X+Y)(ω)=X(ω)+Y(ω).

4: Expectation

Evidence Preservation

E(E(Y|X))=E(Y)

is conservation of expected evidence (thanks to Alex Mennen for making this connection explicit).

Marginal Variance

V(Y)=EV(Y|X)+VE(Y|X)

Why does marginal variance have two terms? Shouldn't the expected conditional variance be sufficient?

This literally plagued my dreams.

Proof (of the variance; I cannot prove it plagued my dreams):

V(Y)=E(Y−E(Y))2=E((Y−E(Y|X))+(E(Y|X)−E(Y)))2=E(Y−E(Y|X))2+E(2(Y−E(Y|X))(E(Y|X)−E(Y)))+E(E(Y|X)−E(Y))2=EV(Y|X)+2E((Y−E(Y|X))(E(Y|X)−E(Y)))+VE(Y|X)=EV(Y|X)+2E(YE(Y|X)−YE(Y)−E(Y|X)2+E(Y|X)E(Y))+VE(Y|X)=EV(Y|X)+2(E(YE(Y|X))−E(YE(Y))−E(E(Y|X)2)+E(E(Y|X)E(Y)))+VE(Y|X)=EV(Y|X)sample variance+VE(Y|X)model variance.

The middle term is eliminated as the expectations cancel out after repeated applications of conservation of expected evidence. Another way to look at the last two terms is the sum of the expected sample variance an... ,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
Everything I ever needed to know, I learned from World of Warcraft: Goodhart’s law
by Said Achmiz — 2018-05-03T16:33:50.002Z
Link: https://blog.obormot.net/Everything-I-ever-needed-to-know-I-learned-from-World-of-Warcraft-Goodharts-law

This is the first in a series of posts about lessons from my experiences in World of Warcraft. I’ve been talking about this stuff for a long time—in forum comments, in IRC conversations, etc.—and this series is my attempt to make it all a bit more legible. I’ve added footnotes to explain some of the jargon, but if anything remains incomprehensible, let me know in the comments.

World of Warcraft, especially WoW raiding[1], is very much a game of numbers and details.

At first, in the very early days of WoW, people didn’t necessarily appreciate this very well, nor did they have any good way to use that fact even if they did appreciate it. (And—this bit is a tangent, but an interesting one—a lot of superstitions arose about how game mechanics worked, which abilities had which effects, what caused bosses[2] to do this or that, etc.—all the usual human responses to complex phenomena where discerning causation is hard.) And, more importantly and on-topic, there was no really good way to sift the good players from the bad; nor to improve one’s own performance.

This hampered progression. (“Progression” is a WoW term of art for “getting a boss down, getting better at doing so, and advancing to the next challenge; rinse, repeat”. Hence “progression raiding” meant “working on defeating the currently-not-yet-beaten challenges”.)

The combat log

One crucial feature of WoW is the combat log. This is a little window that appears at the bottom of your screen; into it, the game outputs lines that report everything that happens to or around your character. All damage done or taken, all hits taken or avoided, abilities used, etc., etc.—everything. This information is output in a specific format; and it can be parsed by the add-on system[3].

Naturally, then, people soon began writing add-ons that did parse it—parse it, and organize it, and present various statistical and aggregative transformations of that data in an easy-to-view form—which, importantly, could be viewed live, as one played.

Thus arose the category of add-ons known as “damage meters”.

The damage meters

Of course the “damage meters” showed other things as well—but viewing damage output was the most popular and exciting use. (What more exciting set of data is there, but one that shows how much you’re hurting the monsters, with your fireballs and the strikes of your sword?) The better class of damage-meter add-ons not o

...
Rigging is a form of wireheading
by Stuart_Armstrong — 2018-05-03T12:50:50.220Z

I've now posted my major posts on rigging and influence, with what I feel are clear and illustrative examples. But, in all the excitement of writing out maths, I haven't made it clear to everyone why they should care about rigging a learning process in the first place.

And that reason is simple:

Rigging is a form of wireheading that value-learning AIs are particularly vulnerable to. It occurs when there is a causal arrow drawn from the AI to the process they are 'learning' about.

For example, assume there is a bot in charge of a forum, and it is rewarded for granting access to the secret parts of the forum to users who have the right to access them. This 'right to access' is checked by whether the user knows a password (which is 'Fidelio', obviously).

As a causal graph, for a given user X, this is:

The green node is the bot's action node, the orange one is the data the bot is trying to learn. In this setup, the bot's task is essentially brainless: it simply checks whether user X has given the right password, then grants them access.

The bot could also have the power to be proactive: searching out users and getting them to give it the password. This can be encoded as the AI asking for users to supply the password:

Up till now, the learning process remains uninfluenceable and unriggable. But note that we've added the ability for the bot to communicate with the users. It could use that ability to get them to type in the password, as above. But it could also tell the user directly what the password is.

Now the orange node that the bot is learning about is causal descendant of the bot's actions (red arrow). To maximise its reward, the bot should tell every user the password, and then grant them access.

This is essentially the definition of a riggable learning process: something that seemed to be a fact about the world that the agent was learning about, but, when we drew in all the causal arrows, it turns out that fact was subject to manipulation by the agent.

Note the tiny difference between an unriggable and riggable learning process: when the bots abilities went from "send this one specific message", to "send any message", the learning process became riggable.
Moral frameworks and the Harris/Klein debate
by lincolnquirk — 2018-05-03T09:49:03.626Z

Here's a good background post and analysis on the debate (this has been linked from elsewhere on LW before): https://everythingstudies.com/2018/04/26/a-deep-dive-into-the-harris-klein-controversy/

Like many, I couldn't help but be fascinated by the Sam Harris/Ezra Klein debate. These are two people I really look up to, and so seeing them going at it (and showing a lot of personal weakness along the way) has been illuminating. I'm still unsettled about it, wanting there to be resolution/a right answer. So far that satisfaction has eluded me, so I wrote this to try to clarify things for myself. Maybe it helps others too.

The analysis below is meant as a steelman of each side's positions. If you think I'm not steelmanning them well enough, please leave a comment and I'll improve.

Consequentialist framework:

Sam: As a lesson in how to think for yourself, hold Murray up as someone who has discovered truths that society doesn't like to talk about. As a general policy, this practice will lead to truths being uncovered faster, leading to a faster pace of discovery, which compounds over time to a much better world through science.
Ezra: Make a public example of Sam here, leading to more people recognizing their own privilege and putting their actions in the appropriate historical context. As a general policy, this practice will lead to a more equitable society, which compounds over time to a much better world by reducing suffering directly.

Virtue ethics framework:

Sam: It is virtuous to signal-boost things which are true, especially when they are being suppressed by society. "Speak the truth though your voice may tremble"
Ezra: It is virtuous to defend the underprivileged by calling out harms, even unintentional harms. "Evil is the silence of the voice of justice when it matters most"

Deontology framework:

Sam: Thou shalt update on all available data.
Ezra: Thou shalt not invoke long-buried demons of oppression.

Both these moral frameworks look pretty good to me. I see no particular reason to favor one over the other; even if I restrict myself to only looking at the consequences I see plausible arguments that one path or the other is higher-impact.

I have only one unifying thought:

Sam was upset by being attacked by Klein, and considered the attack unfair, which actually triggered the whole debate. I think Sam's co...
Doubts about Updatelessness
by Diffractor — 2018-05-03T05:44:12.000Z

Epistemic status: crystallizing uncertain lore

(this is mostly a writeup of a discussion with Abram, plus some additional thoughts of my own)

Note: Counterlogical Mugging will be used as a term for the counterfactual mugging problem that uses a digit of some mathematical constant.

Lately there’s been a bit more doubt about updatelessness (in full generality instead of the partial version attained by policy selection) as being an achievable desiderata. This came from thinking a bit more about what universal inductors do on the counterlogical mugging problem.

A universal inductor can be thought of as "what happens when you take a logical inductor, but don't show it anything", subject to the extra constraint that it's a probability distribution at every stage. Surprisingly enough, this allows us to find out information about arbitrary consistent theories simply by conditioning. This is theorem 4.7.2, closure under conditioning, which says that for any efficiently computable sequence, the conditional probabilities of sentences will act as a logical inductor relative to a new deductive process that contains the sentences that are being conditioned on. Conditioning substitutes for having an upstream deductive process. Therefore, if you fix some mapping of bitstring places to sentences, and condition on theorems of PA, it acts as a logical inductor over PA.

An interesting thing happens when a sufficiently advanced inductor (advanced enough to know the digit ahead of time) is placed in a counterlogical mugging scenario.

Consider the problem where the agent is using the limit of a universal inductor, P∞, and it recieves/is conditioned on the first n digits of the binary expansion of e, so the agent is using P∞|σ:n=e:n as its probability distribution. The agent must select a policy π:={pay,¬pay}, which is just a decision whether to pay the mugging or not when asked. The mugging will occur on a distant digit of e, ef(n), where f(n) is some fast-growing function.

And omega's algorithm is:

if ef(n)=1, then if π(0)=pay, +2 dollars to agent

if ef(n)=0, then if π(0)=pay, -1 dollar to agent

Now, let's compute the payoffs for the 2 possible policies.

π1:=pay,2P∞(σf(n)=1|σ:n=e:n)−P∞(σf(n)=0|σ:n=e:n)
π2:=¬pay,0

Now, to begin with, because the binary expansion of e has positive probability in the resulting semimeasure, then for a sufficiently long-running universal inductor, if they conditi

...
Rational Feed: Last Month's Best Posts
by sapphire — 2018-05-02T18:19:47.821Z

I write a daily rational feed. I write up summaries/teasers for the previous day's article that I found interesting and/or enjoyable. I follow most rationalist blogs as well as LW2.0 and the EA Forum on RSS. The daily feed is posted in the SSC Discord and on my Wordpress Blog However the rational feed includes quite alot of articles and many people have requested a more heavily pruned feed. Here we go:.

My Favorite Post:

A Deep Dive into the Harris-Klein Controversy by Everything Studies - My favorite article this month is a long analysis of the kerfuffle between Ezra and Sam Harris. Sam seems to feel mistreated but its also arguably his fault the email exchange deteriorated. In addition Sam felt like others would clearly take his side in the conflict but he came off quite badly and seemed emotionally unhinged. What happened?

Consider the following quote: 'Stanovich talks about “cognitive decoupling”, the ability to block out context and experiential knowledge and just follow formal rules, as a main component of both performance on intelligence tests and performance on the cognitive bias tests that correlate with intelligence. Cognitive decoupling is the opposite of holistic thinking. It’s the ability to separate, to view things in the abstract, to play devil’s advocate.'

High decoupling is the norm in science, isolating variables is fundamental to science. However low decoupling is the norm in politics and literature. Claims are evaluated in context and connections are fundamental. Sam thinks scientific norms are appropriate and Ezra thinks political and journalistic norms apply. The author discusses several other implicit conflicts. Ezra has a very strong prior about the reference class containing Murray's work. Sam does not. In addition Sam and Ezra seem to disagree on what level of 'intellectual honesty' is fair game.

Better Discourse:

Change My View by Rationally Speaking Podcast - When people argue on the internet, you never expect anyone to actually say "You know what, that's a good point, you've changed my view somewhat." But Change My View, a fast-growing subreddit founded by Kal Turnbull, is an exception to the rule. Julia and Kal discuss the culture of Change My View, what makes it such an oasis for reasonable discussion on the Internet, and what we've learned about what motivates people to chan...