trentmkelly/LessWrong-43k huggingface dataset:
The Alignment Newsletter #6: 05/14/18
by Rohin Shah — 2018-05-14T16:00:49.397Z

Highlights

Thoughts on AI Safety via Debate

(Vaniver): Vaniver has played several debate games on the

website

and wrote up some of his experiences. He ended up more optimistic about debate, but still worries that the success of the technique relies on the toy examples being toy.

My opinion: I haven't played the particular debate game that OpenAI released, and so it was interesting to see what sort of strategies emerged. It was initially quite unintuitive to me how debate picks out a particular path in an argument tree, and I think if reading about particular concrete examples (as in this post) would have helped.

Prerequisities:

AI safety via debate

Technical AI alignment

Problems

Classiﬁcation of global catastrophic risks connected with artiﬁcial intelligence

(Alexey Turchin et al)

Scalable oversight

Thoughts on AI Safety via Debate

(Vaniver): Summarized in the highlights!

Thoughts on "AI safety via debate"

(gworley)

Miscellaneous (Alignment)

Open question: are minimal circuits daemon-free?

(Paul Christiano): One issue that may arise with an advanced AI agent is that during training we may end up with a part of the AI system developing into a "daemon" -- a consequentialist agent that is optimizing a different goal. This goal may useful as a subcomponent for our AI, but the daemon may grow in power and end up causing the system to optimize for the subgoal. This could lead to catastrophic outcomes, even if we have specified a reward function that encodes human values to the top-level AI.

In this post, Paul suggests that these issues would likely go away if we choose the fastest program to solve our subgoal. Intuitively, for any daemon that arises as a solution to our problem, for it to cause a bad outcome it must be carrying out complicated reasoning to figure out whether or not to solve the problem honestly or to try to mislead us, and so we could get a faster program by just not doing that part of the computation. He proposes a particular formalization and poses it as an open question -- if we always choose the minimal (in size) boolean circuit that solves our problem, can a daemon ever arise?

My opinion: I still don't know what to think about daemons -- they do seem to be a problem in Solomonoff induction, but they seem unlikely to arise in the kinds of neural nets we have today (but could arise in larger ones). I would love to see more cla...
Mental Illness Is Not Evidence Against Abuse Allegations
by sarahconstantin — 2018-05-13T19:50:42.645Z

ETA: this post was pretty much refuted by comments below.

I've noticed a situation several times that I think deserves attention.

Somebody goes around saying they've been the victim of mistreatment. But they seem mentally ill. Whether or not you know of a diagnosis, they seem "off" somehow -- highly agitated, making social faux pas, telling stories that don't quite add up. So people are very suspicious about whether their allegations are true.

Is this rational?

In general, someone who seems less trustworthy should be believed less. And, yes, mentally ill people are more likely to be delusional or exaggerating. But they are also more likely to actually be victims of crimes than the general population.

40% of women in the UK with severe mental illness are victims of rape or attempted rape.

People with severe mental illness are 6x as likely as the general population to have recently experienced sexual violence.

30% of mentally ill adults in an American study had been victims of violent crime in the previous six months.

Mentally ill adults in Sweden are 5x more likely than the general population to be murdered.

More than 25% of severely mentally ill Americans have been the victims of a violent crime in the last year, 4x the rate of the general population.

30-33% of psychiatric patients have been victims of domestic violence.

Someone being mentally ill is evidence for, not against, their being victims of a crime. And the base rates of violent crime are pretty high, so all things being equal, "someone attacked me" is not an extraordinary claim. Even when someone seems crazy and has made a lot of claims you don't believe, it can be reasonable to believe their claims of crime victimization. Don't fall into the horns effect.
A Detailed Critique of One Section of Steven Pinker’s Chapter “Existential Threats” in Enlightenment Now (Part 2)
by philosophytorres — 2018-05-13T19:41:15.326Z

PART 2:

If humanity is screwed, why sacrifice anything to reduce potential risks? Why forgo the convenience of fossil fuels, or exhort gov­ernments to rethink their nuclear weapons policies? Eat, drink, and be merry, for tomorrow we die! A 2013 survey in four English-speaking countries showed that among the respondents who believe that our way of life will probably end in a century, a majority endorsed the state­ment “The world’s future looks grim so we have to focus on looking after ourselves and those we love.”

Few writers on technological risk give much thought to the cumulative psychological effects of the drumbeat of doom.

First, the “drumbeat of doom” is misleading, for reasons discussed above (and below). Second, it’s not entirely true that few writers have thought hard about the relevant psychological effects. For example, in Morality, I repeatedly emphasize the astronomical value thesis (a reason for hope), endorse what Paul Romer calls “conditional optimism” (which Pinker also cites in Enlightenment Now), and quote Cormac McCarthy’s witticism that “I’m a pessimist but that’s no reason to be gloomy!” Furthermore, I underline the dangers associated with what Jennifer Jacquet calls the “anthropocebo effect,” which refers to “a psychological condition that exacerbates human-induced damage—a certain pessimism that makes us accept human destruction as inevitable.”

Even more, in one of the founding documents of the field, Bostrom conjectures that the depressing nature of the topic may be one reason that it has received so little scholarly attention—to which he adds that “the point [of existential risk studies] is not to wallow in gloom and doom but simply to take a sober look at what could go wrong so we can create responsible strategies for improving our chances of survival.” Other scholars within the field, including Owen Cotton-Barratt, Toby Ord, and Max Tegmark, have highlighted the importance of “existential hope,” which brings to the foreground a sense of how good things could turn out (if we play our cards right). These are just a few of many examples that could be adduced to corroborate the claim that many “writers on technology risk” have thought very much, or very deeply, about mental effects of cogitating doomsday scenarios.

As Elin Kelsey, an environmental communicator, points out,“We have media ratings to protect children from sex or violence in ...
Fun With DAGs
by johnswentworth — 2018-05-13T19:35:49.014Z

Crossposted here.

Directed acyclic graphs (DAGs) turn out to be really fundamental to our world. This post is just a bunch of neat stuff you can do with them.

Utility Functions

Suppose I prefer chicken over steak, steak over pork, and pork over chicken. We can represent my preferences with a directed graph like this:

This poses a problem. If I’m holding a steak, then I’ll pay a penny to upgrade it to chicken — since I prefer chicken over steak. But then I’ll pay another penny to upgrade my chicken to pork, and another to upgrade it to steak, and then we’re back to the beginning and I’ll pay to upgrade the steak to chicken again! Someone can stand there switching out chicken/pork/steak with me, and collect my money all day.

Any time our preference graph contains cycles, we risk this sort of “money-pump”. So, to avoid being money-pumped, let’s make our preferences not contain any cycles:

Now our preferences are directed (the arrows), acyclic (no cycles), and a graph (arrow-and-box drawing above). It’s a DAG!

Notice that we can now sort the DAG nodes, spatially, so that each node only points to higher-up nodes:

Reading off the nodes from bottom to top, the order is: chicken, pork, steak.

This is called a topological sort, or topo sort for short. We can always topo sort a DAG — there is always some order of the nodes such that each node only points to nodes after it in the sort order.

In this case, the topo sort orders our preferences. We prefer pork over everything which comes before pork in the topo sort, and so forth. We could even number the items according to their topo-sorted order: 1 for chicken, 2 for pork, 3 for steak. We prefer whichever thing has the higher number, i.e. the higher position in the topo-sort order.

This is called a utility function. This utility function U is defined by U(chicken) = 1, U(pork) = 2, U(steak) = 3. We prefer things with higher utility — in other words, we want to maximize the utility function!

To summarize: if our preferences do contain cycles, we can be money-pumped. If they don’t, then our preferences form a DAG, so we can topo sort to find a utility function.

Circuits

Here’s a really simple circuit to compute f = (x+y)*(x-y):

Notice that the circuit is a DAG. In this case, the topo sort tells us the order in which things are computed: “+” and “-” both come before “x” (the multiplication, not the input). If the graph contained any ...
Gaining Approval: Insights From "How To Prove It"
by Logan Riggs — 2018-05-13T18:34:54.891Z

[Note: These are insights that surprised/connected the dots for me personally. I feel dumb admitting some of these, but my hope is that this'll be a good data point for others]

How to write a math book:

1. It's fun to prove things wrong. Examples such as "What's wrong with this proof" really worked for me.

2. A great pattern for teaching: a solid explanation of new concept, 3-5 example problems to work through, and solutions & commentary immediately after! It had a short feedback loop that I felt myself wanting to be a part of.

3. Concepts built upon each other in quick succession, so you don't have to flip back and review.

Proofs:

1. Proofs aren't made to show how they made the proof, just what they're proving in a succinct way. Since this is the case, it's okay to be confused even if the proof sounds like everything of course follows everything else, and you're an idiot for not immediately following. The proof took hours/weeks/years/centuries to condense in that form, so it's reasonable if it takes a few hours & math stack exchange to understand it.

2. There are tips and tricks for writing proofs depending on the properties of what you're proving. For instance, it's generally easier to prove a positive, so if you're proving a negative, make it a positive and do a proof by contradiction on it!

Math:

1. A ⟷ B is making two claims: A ⟶ B & A ⟵ B.

2. Intuition for material implication (rule of inference):

Either Bob did it or Alice did it. If it's not Bob, then it's Alice.
It's either A or B. If it's not A, then it's B.
A ∨ B ⟷ ¬A ⟶ B

3. Intuition for contrapositive of implication:

¬A ⟶ B ⟷ A ∨ B (rule of inference above)
A ∨ B ⟷ B ∨ A (Bob or Alice = Alice or Bob)
B ∨ A ⟷ ¬B ⟶ A (rule of inference above)
¬B ⟶ A ⟷ ¬A ⟶ B (Bam! Contrapositive!)

4. Combining induction with functions gets you recursion.

5. X = Ø is a negative statement: ∀y(y ∉ X)

6. X is infinite is a negative statement: not finite.

Math, Dancing, and Music:

I improv dance & piano, and to hit on something interesting, you have to combine different relationships with different objects. Let's hone in on just negation.

In math, negate "a subset of" to get "disjoint", "intersection of" to get "symmetric difference", "for all x, P(x)"...
Welcome to LW-Frankfurt [Edit With Your Details]
by Janko — 2018-05-13T16:07:19.547Z

(The following are our suggestions for what kind of information is best to include in the welcome post of your group, feel free to replace them with whatever you think is best)

What kind of events does your group usually run? What does it usually do?

How frequently does your group organize events or meet?

Who would be a good fit for you group?

Should they have any particular skills or have done some specific background reading?
Why Universal Comparability of Utility?
by AK — 2018-05-13T00:10:20.944Z

Apologies if this is answered elsewhere and I couldn't find it. In AI reading I come across an agent's utility function, U, mapping world-states to real numbers.
The existence of U is justified by the VNM-utility theorem. The first axiom required for VNM utility is 'Completeness' -- in the context of AI this means for every pair of world-states, Wi and Wj, the agent knows Wi>Wj,Wi<Wj, or Wi ~ Wj .

Completeness over world-states seems like a huge assumption. Every agent we make this assumption for must already have the tools to compare 'world where, all else equal, the only food is peach ice cream' v. 'world where, all else equal, Shakespeare never existed.'* I have no idea how I'd reliably make that comparison as a human, and that's a far cry from '~', being indifferent between the options.

Am I missing something that makes the completeness assumption reasonable? Is 'world-state' used loosely, referring to a point in a vastly smaller space, with the exact space never being specified? Essentially, I'm confused, can anyone help me out?

*if it's important I can try to cook up better-defined difficult comparisons. 'all else equal' is totally under-specified... where does the ice cream come from?
Hotel Concierge: Shame & Society 
by norswap — 2018-05-12T23:36:43.645Z
Link: https://hotelconcierge.tumblr.com/post/173526578129/shame-society

As seen in one of Scott's linkdumps. I thought it was interesting enough to deserve discussion here.

Scott's comment:

Hotel Concierge, everyone’s favorite Tumblr cultural commentator who is definitely not secretly The Last Psychiatrist, has another magnum opus out – Shame And Society. My favorite excerpt: “Make no mistake, the performative sadness is not consequent to the pursuit of hedonism; it is a justification.” Deserves a lot closer reading and more discussion than I probably have the time and energy to give it.

The thing is long, and at times intricate. It meanders a lot and you're left wondering what point is being made and how it relates to the bigger picture. Yet it made all kinds of light bulbs go firework in my head, and if only on the strength of the connections that are being made between disparate culture-war memes, I would recommend this piece.

It's almost impossible to summarize, but I'll give it the old college try, focusing on the big picture and the parts that had an impact on me. It's probably partly wrong.

Broadly, the piece is about the fetishization of surface behaviors, and how society (in a broad sense — Moloch) encourages this. We require signalling (checkboxing), and we're on a path where these requests become more and more intricate and demanding. As a result, we spend most of our time performing. The concierge ascribes this to a transition from metis (“hard to express,” “local,” “accumulated, experiential knowledge”) to episteme (“top-down,” “abstract, generalized, theoretical knowledge”). The global culture swallows the local cultures. " Checkboxes become more granular and the list of permissible narratives shrinks. Everyone sends the same signals so the signals lose meaning." Shame is a force to coerce you into the checkboxes, the real heroes are those that require the least amount of coercive shame. You can avoid shame by performing guilt. But perversely, the causality reverses: guilt sanction cruelty rather than absolving it.

Then there's a long discussion of kinks (fetishes). They were never so well accepted than they are now, especially by women. These kinks (e.g. BDSM) were traditionally male fetishes but have shifted. Men sometimes come to resent them, because while they were previously an affirmation of power, once they become requested they become another act that one...
A Detailed Critique of One Section of Steven Pinker’s Chapter “Existential Threats” in Enlightenment Now (Part 1)
by philosophytorres — 2018-05-12T13:34:21.381Z

This is the first of three posts; the critique has been split into three parts to enhance readability, given the document's length. For the official publication, go here.

Key findings:

—> The first quarter or so of the chapter contains at least two quotes from other scholars that are taken “completely” out of context—that is, their original meaning is either in tension or outright contradictory with respect to the meaning implied by their use in this chapter. In both cases, the quotes play integral rhetorical, and to some extent substantive, roles in the argument that Pinker aims to develop.

—> The chapter expends a great deal of energy attacking a small village of straw men, from the pessimism/optimism dichotomy that frames the entire discussion to the theoretical dangers posed by value-misaligned machine superintelligence. I argue that this tendency to knock down unserious or non-existent positions while ignoring or misrepresenting the most intellectually robust ideas does a disservice to the ongoing public and academic discussions about the various global-scale threats facing humanity this century.

—> Many citations appear to have been poorly vetted. For example, Pinker relies on numerous non-scholarly articles to make what purport to be scholarly assertions about a wide range of topics that fall outside his area of expertise. In some cases, Pinker makes these claims with considerable confidence, thus giving non-expert readers—some of whom may be responsible for shaping domestic and foreign policies—a false sense of their tenability.

—> Along these lines, many of the sources that Pinker cites to support his theses contain some facts, evidence, or ideas that undercut those theses. Rather than acknowledging that alternative views are also compatible with (or supported by) the evidence, though, Pinker preferentially selects the facts, evidence, and ideas that support his narrative while simply ignoring those that don’t. This is part of a larger issue of “cherry-picked” data. Indeed, I argue that, when the facts are more comprehensively considered, the positions that Pinker champions appear far less defensible.

—> Overall, the assessment presented below leads me to conclude that it would be unfortunate if this chapter were to significantly shape the public and academic discussions surrounding “existential risks.” In the harshest terms, the chap...
Terrorism, Tylenol, and dangerous information
by Davis_Kingsley — 2018-05-12T10:20:28.294Z

Recently, there has been an alarming development in the field of terrorist attacks; more and more terrorists seem to be committing attacks via crashing vehicles, often large trucks, into crowds of people. This method has several advantages for an attacker - it is very easy to obtain a vehicle, it is very difficult for police to protect against this sort of attack, and it does not particularly require special training on the part of the attacker.

While these attacks are an unwelcome development, I would like to propose an even more worrisome question - why didn't this happen sooner?

I see no reason to believe that there has been any particular technological development that has caused this method to become prevalent recently; trucks have been in mass production for over a hundred years. Similarly, terrorism itself is not particularly new - just look to the anarchist attacks of the late 19th and early 20th century. Why, then, weren't truck attacks being made earlier?

The answer, I think, is both simple and frightening. The types of people who make attacks hadn't thought of it yet. The main obstacle to these attacks was psychological and intellectual, not physical, and once attackers realized these methods were effective the number of attacks of this sort began increasing. If the Galleanists had realized this attack method was available, they might well have done it back in '21 -- but they didn't, and indeed nobody motivated to carry out these attacks seemed to until much later.

Another instance - though one with less lasting harm - pertains to Tylenol. In 1982, a criminal with unknown motives tampered with several Tylenol bottles, poisoning the capsules with cyanide and then replacing them on store shelves. Seven people died in the original attack, which caused a mass panic to the point where police cars were sent to drive down the streets broadcasting warnings against Tylenol from their loudspeakers; more people still were killed in later "copycat" crimes.

In this case, there was a better solution than with the truck rammings - in the aftermath of these events, greatly increased packaging security was put into place for over-the-counter medications. Capsules (which are comparatively easy to adulterate) fell out of favor somewhat in favor of tablets; further, pharmaceutical companies began putting tamper-resistant seals on thei...
LW Update 2018-05-11 – Suggest Curation 
by Raemon — 2018-05-12T03:26:07.712Z

I spent the past week fleshing out the new Sunshine Sidebar, a dashboard for moderators to see:

All new posts
New comments from users with 10 or less karma
Reported comments
Frontpage posts that have been suggested for curation

Hoping this keeps us much more on top of spam, budding flame wars, etc, as well as enabling us to keep a more reliable Curation schedule.

Suggest Curation

[Edit: apologies! The following feature doesn't work yet for general users, due to technical difficulties]

Users with 2000 or more karma now have access to a "suggest curation" button for frontpage posts. This will cause the post to show up in our "Suggestions for Curated" feed, with a list of the usernames of people who suggested it.

We'll still be determining what get's curated by the mod team's discretion, with a vague goal of "every couple days, selecting a post that have multiple people endorsing it for curation."

Open Source Contributors

Big thanks to crybx for their first merged pull-request (removing some extraneous code in the user profiles that occasionally caused bugs), with a few more PRs on the way. In addition, I appreciated them wading through the windows install process and documenting their progress for the benefit of others.

Also thanks to Elo, gjm and philh for also making an initial foray into the "get the git repo installed on an OS that we hadn't tested it on yet" world.

Git History – Commit 5551c1adc4b590b239aece6efadf0285f2f6c212