trentmkelly/LessWrong-43k huggingface dataset:
Open AI co-founder on AGI
by ShardPhoenix — 2018-09-16T10:18:02.603Z
Link: https://youtu.be/w3ues-NayAs?t=27m6s

Here's a talk where Open AI co-founder Ilya Sutskever talks about the issue of AI progress and AGI. (Before the timestamp he was talking about Open AI's specific projects such as Dota 2 and their hand robot).

tl;dw: (italics indicate my interpolations):

AGI in the near-medium term can't be ruled out based on current rates of progress.
AI progress is more constrained by hardware than by conceptual advances (partly because it's hard to make conceptual advances if you don't have the hardware to test them with).
Companies and researchers are willing to make increasingly large investments into compute hardware (at rate greater than Moore's Law).
Open AI is also working on what we would consider Friendliness (but seemingly not at the same level of rigour as MIRI).
Communication: A Simple Multi-Stage Model
by  — 2018-09-15T20:12:16.134Z

[Communication is a lot like a game of telephone. Information needs to flow from your head to the the medium to the other person to the other person's head. As a result, there are lots of places where miscommunication can happen. Some actionable suggestions are given for how to do better.]

A simplified way I think about communication is with a multi-stage model, where the information flows from one step to the next, often becoming transformed along the way, sort of like the game of Telephone.

By “communication”, I mean to refer to the process by which an idea travels from one person’s head to another’s. When discussed in this manner, it is relevant to questions I have about pedagogy and the nature of media, i.e. how to teach things and how the vehicles we use to transmit information affect the process.

At each step, I will go over what happens to the information being presented.

Here is basic multi-stage example to start us off:

Ankit has a thought about the question of consciousness.
He writes it down into an essay and sends it to Beth.
Beth reads the essay and interprets the ideas in her head.
She excitedly explains them to Cai Yi.

If we track the development of the idea, as it goes through host to host, there are several natural places where the idea is stored:

Your mind.
The initial medium.
Primary receiver’s mind.
Primary receiver’s medium.
Secondary receiver’s mind.
and so on

I think it is easier to think of each stage as a place where the idea lives, and potential changes to the idea happen as it transitions from stage to stage.

Let’s dive into the stages:

Your Mind:
This is the initial place where the idea is generated. Oftentimes, the generative factors for your idea, i.e. the things that caused you to think about it in the first place, might not be explicit. They might instead be aliefs, experiences, or emotional responses. More verbal or explicit models, of course, can also play a role. And while you can trust yourself to have some sort of underlying reasoning, the base justifications aren’t always available.

Inside your own head, you also have immediate access to a lot of different trails of thought. A big benefit is being able to quickly pick up a combination of very disparate types of internal objects and composing them to bolster your initial idea. This immediacy can also lend a nebulous quality to idea composition. You can have mental pointers to cached thoughts...
How to use a microphone r̶a̶t̶i̶o̶n̶a̶l̶l̶y̶  during public speaking
by ChristianKl — 2018-09-14T17:14:25.695Z

Microphones are common technology and yet few people understand how they work. For our purposes here, there are unidirectional microphones and omnidirectional microphones.

A unidirectional microphone only records the sound from one direction. If it's pointed at the mouth of the speaker it won't record noise from the audience which makes the resulting recording better.

If it's however pointed at the ceiling while it's in front of the mouth of the speaker it will do a poor job at recording the speaker.

This leads to the first rule of microphone usage:
Hold the microphone pointing towards your mouth.

Microphones react to sound waves and sound waves are movement of air. If you exhale into a microphone, the microphone will record the exhale. Given Newtons laws, the exhale isn't omnidirectional either but the air that comes out of your mouth from the exhale has a clear direction. If you hold the microphone in front of your mouth, it will get hit by the air.

Our second rule of microphone usage is:

Keep the microphone to the right side of your mouth if you hold it in your right hand and correspondingly on the left side of your mouth if you hold it with your left hand.

Let combine the two rules into rule zero:

Hold the microphone to the side of your mouth in a way that points towards your mouth.
Apply for Emergent Ventures
by Zvi — 2018-09-13T21:50:00.295Z

Original Post (Marginal Revolution): Emergent Adventures: A New Project to Help Foment Enlightenment

Today, two new philanthropic projects announced funding.

Jeff Bezos

In the first project, Jeff Bezos gave two billion dollars to fund services for poor families and the homeless. This will be his fourth most valuable charitable project, behind The Washington Post, Blue Origin and of course the world’s greatest charity Amazon.com. He is truly a great man.

In exchange for giving two billion dollars to help those in need, he was roundly criticized throughout the internet for not giving away more of his money faster. Every article I’ve seen emphasizes how little of his wealth this is, and how much less generous this is than Warren Buffet or Bill Gates. The top hit when I googled was a piece entitled “Jeff Bezos donates money to solve problem he helped create,” because he’s a capitalist and exploits the workers, don’t you know? Never mind that he’s the biggest source of new consumer surplus we have.

This isn’t quite as big an abomination as the Peter Singer claim that someone who gives away almost all of their vast fortune is basically the worst person ever for not giving away the rest of it. Or even that they’re the worst person in the world for giving away that money in a slightly less than optimal fashion. Won’t someone think of the utils?

But it’s not that far behind.

That doesn’t mean we don’t wish he’d do more, or that we don’t want to help him optimize to have the best impact. It’s important to help everyone play their best game, and I could definitely think of better uses for the money. The hot takes using that angle will doubtless follow shortly.

But, seriously: If you see someone donating two billion dollars to help the less fortunate and you can’t on net say anything nice about it the least you can do is to shut up.

Tyler Cohen’s Emergent Ventures

Now on to the more exciting project that I suspect will have more lasting impact, despite having funding three orders of magnitude smaller.

Emergent Ventures is brilliant. I heartily endorse this service or product.

One of Tyler’s greatest strengths is his ability and willingness to consume and integrate vast amounts of information. He already reads not only a number of books that boggles my mind, but also everything that arrives in his inbox, which is doubtless a criminally underused way to introduce important things...
What To Do If Nuclear War Seems Imminent
by Ben_LandauTaylor — 2018-09-13T19:35:24.382Z
Link: https://benlandautaylor.com/2018/09/13/what-to-do-if-nuclear-war-seems-imminent/

This document describes precautions to take in a scenario like the Cuban Missile Crisis, where nuclear war seems plausibly imminent within the next days or weeks. This is not a guide for what to do if a missile is currently inbound and will strike within minutes or hours.

Overview

If tensions between nuclear powers are running extremely high, and you are in or near a plausible target during a nuclear war (such as a major city in the United States or Europe), then I recommend evacuating to a safer place as soon as possible, and staying for days or weeks until things have calmed down. New Zealand is an excellent place to go.

This plan requires that you maintain a valid passport, so that you can leave your country on short notice if needed. No other special preparations are needed.

Proper calibration here should include substantial tolerance for false positives. For people with the means available, I think it was correct to evacuate during the Cuban Missile Crisis, even though it did not end up leading to nuclear war.

Why New Zealand?

New Zealand is of little or no strategic relevance to the current conflicts between nuclear powers. The experts I’ve talked to agree that it’s implausible that anyone would target New Zealand with nuclear weapons, or that anyone would invade New Zealand in the aftermath of a nuclear exchange.

New Zealand is easy to enter. Anyone with no notable criminal history and a valid passport from most countries, including the US, EU, and Canada, can get a New Zealand tourist visa on arrival, with no need for a prior application, and stay for up to 90 days. (Make sure to get a round-trip ticket, or they might not let you in.)

New Zealand is a major food exporter. If supply chains are disrupted, you’ll be close to the source.

New Zealand is very stable internally. It has a strong Anglo tradition of governance, reasonable national pride, no coups or civil wars within the last century+, negligible riots or ethnic strife, etc.

New Zealand is culturally familiar. It’s an English-speaking country that’s firmly within Western Civilization. As such, most of my audience will be more comfortable staying there while waiting for tensions to calm down, and will stick out less if there’s chaos or rioting after a war.

No other country is so good on so many of these dimensions.

Backup Plans

If you are unable to enter New Zealand, then there are many other countries whic...
[Paper]: Islands as refuges for surviving global catastrophes
by avturchin — 2018-09-13T14:04:49.679Z
Link: https://www.emeraldinsight.com/doi/full/10.1108/FS-04-2018-0031

Our paper "Islands as refuges for surviving global catastrophes" is published in Foresight. The preprint is here.

This article continues a series of articles which explore the plan B of global risks mitigation which is an attempt to survive a possible global catastrophe (where plan A is prevention, and plan C is leaving a message to the next civilization in Earth).

The survival on islands is possible only in case of a very narrow range of catastrophes (likely, less 1 per cent of all possible catastrophes), but it is still much cheaper than creating space refuges on Mars or Moon, and moreover, most needed infrastructure on islands already exists, like research scientific bases on remote islands, so converting them in a catastrophic shelter will be thousands time cheaper than creation of the space refuges and could be done much quicker in case of emergency.

Different islands may help to survive different types of catastrophes.

A criterion for island refuge is suggested and only a few from millions exiting islands are suitable as refuges. The most suitable for such refuge is Kerguelen group of islands, as they are very remote, don't have airstrip, could provide food, have mountains and small scientific base.

A combination of a nuclear submarine converted into a refuge, an island as its base and an underground bunker on that island could provide most diverse and effective way of survival on Earth (Branson, btw, survived hurricane Irma on a wine cellar on his Necker Island.)

Abstract. Islands have long been discussed as refuges from global catastrophes; this paper will evaluate them systematically, discussing both the positives and negatives of islands as refuges. There are examples where isolated human communities have survived for thousands of years on places like Easter Island. Islands could provide protection against many low-level risks, first of all bio-risks. However, they are vulnerable to tsunamis, bird transmitted diseases, and other risks. In this article, we explore how to use the advantages of islands for survival of global catastrophes. The large number of islands on Earth, and their diverse conditions increase the chance that one of them will provide protection from a catastrophe. Additionally, we could increase this protection if an island were used as a base for a nuclear submarine refuge (the “Yellow submarine” project), and/or combi...
Are you in a Boltzmann simulation?
by Stuart_Armstrong — 2018-09-13T12:56:08.283Z

EDIT: Donald Hobson has pointed out a mistake in the reasoning in the section on nucleation. If we know that an area of space-time has a disproportionately high number of observer moments, then it is very likely that these are from long-lived Boltzmann simulations. However, this does not imply, as I thought, that most observer moments are in long-lived Boltzmann simulations.

Most people on LessWrong are familiar with the concept of Boltzmann brains - conscious brains that are randomly created by quantum or thermodynamic interactions, and then swiftly vanish.

There seem to be two types of Boltzmann brains: quantum fluctuations, and nucleated brains (actual brains produced in the vacuum of space by the expanding universe).

The quantum fluctuation brains cannot be observed (the fluctuation dies down without producing any observable effect: there's no decoherence or permanence). If I'm reading these papers right, the probability of producing any given object of duration t and mass M is approximately

exp[−tM×1051].

We'll be taking M=1kg for a human brain, and t>0.1s for it having a single coherent though.

A few notes about this number. First of all, it is vanishingly small, as an exponential of a negative exponential. It's so small, in fact, that we don't really care too much over what volume of space-time we're calculating this probability. Over a Plank length four-volume, over a metre to the fourth power, or over a Hubble volume for 15 billion years: the probabilities of an object being produced in any of these spaces are approximately all of the same magnitude, exp[−tM×1051] (more properly, the probabilities vary tremendously, but any tiny uncertainty in the 51 term dwarfs all these changes).

Similarly, we don't need to consider the entropy of producing a specific brain (or any other structure). A small change in mass overwhelms the probability of a specific mass setup being produced. Here's a rough argument for that: the Bekenstein bound puts a limit on the number of bits of information in a volume of space of given size and given mass (or energy). For a mass M and radius R, it is approximately

2.6MR×1043

Putting M=1kg and R=0.1m, we get that the number of possible different states in brain-like object of brain-like mass and size is less than

21043,

which is much, much, much, ..., much, much less than the invers...
Moderation Reference
by Raemon — 2018-09-12T19:06:57.443Z

This is a repository of moderation decisions that we expect to make semi-frequently, where it's somewhat complicated to explain our reasoning but we don't want that explanation to end up dominating a thread. We'll be adding to this over time, and/or converting it into a more scalable format once it's grown larger.

Death by a Thousand Cuts

There's a phenomenon wherein a commenter responds to a post with a reasonablish looking question or criticism. The poster responds, which doesn't satisfy the commenter's concerns. It turns into a sprawling debate.

Most of the time, this is fine – part of the point of LessWrong is to expose your ideas to criticism to make them stronger.

But criticism varies in quality. Three particular dimensions (in descending order of importance) that we think are important are:

Steelmanning – The best criticism engages with the strongest form of an idea. See this post for more detail on why you'd want to do that. Two subsets of this are:

Does it address core points? – Sometimes a critique is pointing at essential cruxes of a person's argument. Other times it pedantically focuses on minor examples.
Does it put in interpretive effort? – Sometimes, a critic puts in substantial effort to understand the poster's point (and, if the author worded something confusingly, help them clarify their own thinking). Other times critics generally expect authors to put in all interpretive effort of the conversation. (In some situations, the issue is that the author has in fact written something confusing or wrong. In other situations, it's the critic who isn't understanding the point).

Is it kind? – While less crucial than steelmanning, LessWrong is generally a more fun to place to be if people aren't being blunt or rude to each other. All else being equal, being kind rather than blunt is better.

Any given one of the three spectrums above isn't necessarily bad. We don't want a world where all criticism must involve a lot of effort on the part of the critic. But over the years we've encountered a few users who frequently focus on criticism that is a) addressing noncentral points, or misunderstanding the author's core point, while b) being a bit blunt and/or not putting much interpretive effort into the conversation.

The results are comments where any given comment seems basicall...
Iterative Arguments: Alternative to Adversarial Collaboration
by matsn — 2018-09-12T15:56:19.790Z

I've been toying with an idea of developing two competing theories in
parallel in an iterative manner:

A writes an initial thesis

B does the same

A revises their thesis to address B's thesis where it contradicts that of A's

B does the same

and so on until both parties feel they have nothing to add

This would be different from adversarial collaboration, as it's
commonly understood, in that both sides would work on their own
arguments instead of trying to agree on a common summary (which is
very hard!).

It's worth emphasising that the idea is not to correspond with the
opponent. Instead, one would keep updating one's thesis to meet the
challenges presented by the competing thesis so that it remains
coherent and stands on its own after every iteration.

I wrote a little ClojureScript library to go with a Pandoc
template to facilitate iterative argumentation of this kind. The
library makes it easy to navigate between the sides of the argument
ensuring that following local links will switch context when
appropriate, etc. Additionally, it provides bidirectional links by
way of highlighting bits on both sides of the argument. Hot loading
is also supported to make writing the document more convenient.
Currently it's very much work in progress, of course. I didn't want
to commit to too many features at this point not knowing if any of
this would be useful.

Would someone here be interested in trying out this sort of thing
either with their own "archnemesis" or with me? For starters, I think
it would be best to pick a properly contentious topic but not one
inciting too much passion.
(A -> B) -> A
by Scott Garrabrant — 2018-09-11T22:38:19.866Z

This post is about following type signature, which I call the type of agency: (A→B)→A. You can also think of it as consequentialism or doing things on purpose. This post will be a rant with a bunch of random thoughts related to this type signature, and it will likely not make sense. It will also be sloppy and will have type errors, but I think it is worth posting anyway.

First, interpret these arrows as causal arrows, but you can also think of them as function arrows. This is saying that the causal relationship from A to B causes A to happen. Think of A as an action and B as the goal. The reason that A happens is the fact that it has B as a consequence. There are not normally exponential objects like this in Bayes' nets, but I think you can modify so that it makes sense. (I'm not sure that this works, but you have a Cartesian closed category with nodes that are the nodes in your Bayes net, and add small number of morphisms from product nodes to individual nodes, corresponding to the functions in the Bayes' net. The acyclicness of the Bayes' net roughly corresponds to this category being thin. Then you can consider having other types of morphisms that can keep the category thin.)

If you have a game between two agents with action nodes A1 and A2, with utilities U1 and U2. The game implements a pair of functions A1×A2→U1 and A1×A2→U2. We can Curry these functions and think of them as A2→(A1→U1) and A1→(A2→U2). Bringing in the agency (Ai→Ui)→Ai of both players leads to cycle. This cycle does not make sense unless the agency arrows are lossy in some way, so as to not be able to create a contradiction.

Fortunately, there is another reason to think that these agency arrows will be lossy. Lawvere's Fixed Point Theorem says that in a Cartesian closed category, unless B has the fixed point property, you cannot have a surjective function A→(A→B), in Set this is saying that if B has more than one element, you cannot have an injection (A→B)→A. i.e. The agency arrows have to be lossy.

Also, notice that Argmax, takes in a function f from some set A to R, and returns an element of the domain, A, so Argmax has type (A→R)→A.

This one is a bit more of a stretch, but if you look at gradient descent, you have some space X, you have a function f:X→R. The gradient can be thought of as a function from infinitesimal changes in X to infinitesimal changes in f(X)....
AI safety reading group in Madrid
by Pablo Villalobos — 2018-09-11T21:55:41.896Z

We are going to set up a reading group in Madrid (Spain), focused on (semi-)technical papers from MIRI and others about AI safety. The goal is to gain a deeper understanding of current work in this area, and perhaps make some contributions in the future.

We have tentatively set our first meeting on Friday, September 21 afternoon at Campus Madrid, though the details may change. The first paper we will read is "Concrete Problems in AI Safety".

If you are interested in joining the group, please send me a message or leave a comment so we can gauge interest and arrange a new date if the current one does not suit you. The definitive date and venue will be published in the group page.
Petrov corrigibility
by Stuart_Armstrong — 2018-09-11T13:50:51.167Z

After my example of problems with corrigibility, and Eliezer pointing out that sometimes corrigibility may involve saying "there is no corrigible action", here's a scenario where saying that may not be the optimal choice.

Petrov is, as usual for heroes, tracking incoming missiles in his early warning command centre. The attack pattern seems unlikely, and he has decided not to inform his leaders about the possible attack.

His corrigible AI pipes up to check if he needs any advice. He decides he does, and asks the AI to provide him with documentation about computer detection malfunction. In the few minutes it has, the AI can start with one introductory text A, or with introductory text B. Predictably, if given A, Petrov will warn his superiors (and maybe set off a nuclear war), and, if given B, he will not.

If the corrigible AI says that it cannot answer, however, Petrov will decide to warn his superiors, as his thinking has been knocked off track by the conversation. Note that this is not what would have happened had the AI stayed silent.

What is the corrigible thing to do in this situation? Assume that the AI can predict Petrov's choice for whatever action it itself can take.