trentmkelly/LessWrong-43k huggingface dataset:
Diana Fleischman and Geoffrey Miller - Audience Q&A
by Jacob Falkovich — 2019-08-10T22:37:53.090Z

Cross-posted from Putanumonit.

This is the audience Q&A with Diana Fleischman and Geoffrey Miller at the NYC Rationality meetup, following up on my own interview which you can find here.

Content note: the audience comprised rationalists of many ethnicities, orientations, and gender expressions and we asked questions that could offend many ethnicities, orientations, and gender expressions.

What are the main hypothesized causes of homosexuality?

Diana: There’s a difference between homosexual behavior and homosexual orientation. Homosexual orientation is very rare. There’s one species, domestic sheep, in which 8-10% of rams are not interested in ewes at all. You can tie a ewe in heat in front of them and they don’t react at all. Actually, one area where homosexuality research has flourished is among sheep breeders because if you buy one of these rams who’s gay, that’s really bad news for the business.

So homosexual orientation is exceedingly rare. Even though you see stats that it’s 10% in people, it’s about 3%. In a paper that I wrote I claim that bisexuality is the optimal sexual strategy because sex is not just used for reproduction, it’s also used for affiliation. There are a number of ways to affiliate: you can give somebody food or you can give somebody an orgasm. These are ways to get other people to like you. If you’re somewhat attracted to people of the same sex but not enough to forego reproductive opportunities with people of the opposite sex, then you can actually engage with both sexes.

Is the bisexual revolution coming?

Diana: There are places where people are much more open to it, but not many places. It’s a whole spectrum of behavior.

In places around the world, there are men who have anything from affectionate to sexual interactions with other men and they’re not considered gay. They have homosexual behavior along with heterosexual behavior, and that’s a common thing. If you look at the bell curve of orientation in a society that suppresses homosexual behavior among men, you’ll find that only men who are predominantly homosexual will exhibit this type of behavior. It’s not used for affiliation in our particular society, so men don’t engage in it. Homosexual men are basically just the edge of this bell curve. The reason there keep being a small percentage of men who are exclusively homosexual is because bisexuality is adaptive.

The most controversial...
Intransitive Preferences You Can't Pump
by zulupineapple — 2019-08-09T23:10:36.650Z

In the usual argument of money-pumping we take an agent with preferences A>B, B>C and C>A. Then we offer it to exchange C+$1 for B, then B+$1 for A, and finally A+$1 for C. Now the agent paid $3 and ended up where it started.

The assumption here is that not only does this agent prefer A to B, it prefers A to B+$1. Of course, the price of $1 could be too steep, then we could look for something less valuable to exchange.

However, payments of arbitrarily low value need not exist! Sure, you can propose a lottery, where the agent only has to pay 1$ with probability p. But arbitrarily low probabilities need not exist. To offer a lottery, you need to have a physical method to generate events with that probability. If p equals 1 divided by Grahm's number, how many coins would you have to flip to run this lottery? Even if you said "the agent will pay $1 when a lump of solid gold materializes out of thin air", there are numbers lower than that probability. I hate to be an ultrafinitist, but it's true, extremely small (or large) numbers are not physically meaningful.

Note, here I am assuming the axiom of continuity, i.e. that if B<A<B+$1, then for some p, we must have B+p·$1<A. However, we should be able to violate this axiom as well.

What does this imply? Probably nothing. This perverse agent is almost indistinguishable from an agent which sets U(A)=U(B)=U(C). The rule is that you can violate any axioms, but only when it doesn't matter.
Categorial preferences and utility functions
by DavidHolmes — 2019-08-09T21:36:22.424Z

This post is motivated by a recent post of Stuart Armstrong on going from preferences to a utility function. It was originally planned as a comment, but seems to have developed a bit of a life of its own. The ideas here came up in a discussion with Owen Biesel; all mistakes in this exposition are mine. I'm not very good with the typesetting engine here, so apologies for latex and other problems.

The basic ideas is as follows. Suppose we have a set Sof objects, and we are given some information on which objects are preferred to which other objects. Then we are interested in whether and in how many ways this data can be captured by a utility function. Our key innovation is that we assume not only the direction of preferences is given, but also some information on the strength of the preferences, in a manner which we will make precise below (weak preferences).

Basic on orders vs utility functions

We refer to the Order Theory page on Wikipedia for the definitions of reflexive, anti-symmetric, transitive and connexive binary relations. If S is a set and U:S→R is a function (`utility'), this induces a reflexive, transitive and connected binary relation (not anti-symmetric in general, unless U is injective).

Conversely, any reflexive, transitive, antisymmetric and connexive binary relation (a.k.a. total order) on a countable set S, this is induced by a utility function taking values in the rational numbers (link to proof); there is a more general discussion here.

Strength of preferences

In what follows, we fix a totally ordered abelian group G. To express a preference between to objects s1, s2 of our set S, one should give an element of G which expresses how strongly s1 is preferred to s2. The most natural example is to take G=Z, then:

Saying s1 is preferred to s2 with strength 1 means we slightly prefer s1 to s2;

Saying s1 is preferred to s2 with strength 0 means have no preference between s1 and s2;

Saying s1 is preferred to s2 with strength 2 means we prefer s1 to s2 more strongly;

Saying s1 is preferred to s2 with strength -1 means we slightly prefer s2 to s1.

Expressions of preference

We consider three ways of describing preferences among objects in a set S:

1. Weak Preference

Definition. A weak preference among elements of S consists of a collection of triples (s1,s2,g) with si∈S and g∈G.

A triple (s1,s2,g) is interpreted as meaning that s1 is preferred to s2 wit... ,
What is the state of the ego depletion field?
by Eli Tyre — 2019-08-09T20:30:44.798Z

It's been almost a half decade since the replication crisis in psychology broke, and one of the major casualties was the subfield of ego depletion. (I belive this was the initial large scale meta-analysis that found no effect. And here is a popular article on the failure to replicate.)

Does anyone have a good summary of the state of that research program? Is there any part of it that held up? What can we conclusively say about self control, willpower, depletion of mental resources, etc?
Why Gradients Vanish and Explode
by Matthew Barnett — 2019-08-09T02:54:44.199Z

Epistemic status: Confused, but trying to explain a concept that I previously thought I understood. I suspect much of what I wrote below is false.

Without taking proper care of a very deep neural network, gradients tend to suddenly become quite large or quite small. If the gradient is too large, then the network parameters will be thrown completely off, possibly causing them to become NaN. If they are too small, then the network will stop training entirely. This problem is called the vanishing and exploding gradients problem.

When I first learned about the vanishing gradients problem, I ended up getting a vague sense of why it occurs. In my head I visualized the sigmoid function.

I then imagined this being applied element-wise to an affine transformation. If we just look at one element, then we can imagine it being the result of a dot product of some parameters, and that number is being plugged in on the x-axis. On the far left and on the far right, the derivative of this function is very small. This means that if we take the partial derivative with respect to some parameter, it will end up being extremely (perhaps vanishingly) small.1

Now, I know the way that I was visualizing this was very wrong. There are a few mistakes I made:

1. This picture doesn't tell me anything about why the gradient "vanishes." It's just showing me a picture of where the gradients get small. Gradients also get small when they reach a local minimum. Does this mean that vanishing gradients are sometimes good?

2. I knew that gradient vanishing had something to do with the depth of a network, but I didn't see how the network being deep affected why the gradients got small. I had a rudimentary sense that each layer of sigmoid compounds the problem until there's no gradient left, but this was never presented to me in a precise way, so I just ignored it.

I now think I understand the problem a bit better, but maybe not a whole lot better.

(Note: I have gathered evidence that the vanishing gradient problem is not linked to sigmoids and put it in this comment. I will be glad to see evidence which proves I'm wrong on this one, but I currently believe this is evidence that machine learning professors are teaching it incorrectly).

First, the basics. Without describing the problem in a very general sense, I'll walk through a brief example. In particular, I'... ,
AI Forecasting Dictionary (Forecasting infrastructure, part 1)
by Bird Concept — 2019-08-08T16:10:51.516Z

This post introduces the

AI Forecasting Dictionary

, an open-source set of standards and conventions for precisely interpreting AI and auxiliary terms. It is the first part in a series of blog posts which motivate and introduce pieces of infrastructure intended to improve our ability to forecast novel and uncertain domains like AI.

The Dictionary is currently in beta, and we're launching early to get feedback from the community and quickly figure out how useful it is.

Background and motivation

1) Operationalisation is an unsolved problem in forecasting

A key challenge in (AI) forecasting is to write good questions. This is tricky because we want questions which both capture important uncertainties, and are sufficiently concrete that we can resolve them and award points to forecasters in hindsight. For example:

Will there be a slow take-off?

is a question that’s important yet too vague.

Will there be a 4-year doubling of world output before the first 1-year doubling of world output?

is both important and concrete, yet sufficiently far-out that it’s unclear if standard forecasting practices will be helpful in resolving it.

Will there be a Starcraft II agent by the end of 2020 which is at least as powerful as AlphaStar, yet uses <$10.000 of publicly available compute?

is more amenable to standard forecasting practices, but at the cost of being only tangentially related to the high-level uncertainty we initially cared about. And so on.

Currently, forecasting projects reinvent this wheel of operationalisation all the time. There’s usually idiosyncratic and time-consuming processes of writing and testing questions (this might take many hours for a single question) [1], and best practices tend to evolve organically but without being systematically recorded and built upon [2].

2) The future is big, and forecasting it might require answering a lot of questions

This is an empirical claim which we’ve become more confident by working in this space over the last year.

One way of seeing this is by attempting to break down an important high-level question into pieces. Suppose we want to get a handle on AI progress by investigating key inputs. We might branch those into progress on hardware, software, and data (including simulations). We might then branch hardware into economics and algorithmic parallelizability. To understand the economics, we must branch it into the supply an...
Why do humans not have built-in neural i/o channels?
by Richard_Ngo — 2019-08-08T13:09:54.072Z

Communication between organisms of the same species is often beneficial, for a variety of reasons: sharing information, signalling, bonding, etc. Yet currently the most advanced form of communication to have evolved, human language, is still very low bandwidth compared with the amount of mental processing our brains do.

It seems conceivable that our nervous systems might have evolved ways to directly (temporarily) interface with each other and exchange a large amount of information. For example, retractable bundles of neurons that are specialised at quickly forming connections with their counterpart neurons in conspecifics.

1. What is the main reason that this has not happened in any large animals? If evolution had continued without humans "taking off", we would have eventually seen such neural links in some animal species?

2. What's the closest thing to this we see in any species?
Which of these five AI alignment research projects ideas are no good?
by rmoehn — 2019-08-08T07:17:28.959Z

I'll post five AI alignment research project ideas as comments. It would be
great if you could approval-vote on them by using upvotes. Ie. when you think
the project idea isn't good, you leave the comment as is; otherwise you give it
a single upvote.

The project ideas follow this format (cf. The Craft of
Research):

I'm studying <topic>,
because I want to <question that guides the search>,
in order to help my reader understand <more significant
question that would be informed by an answer to the
previous question>.

The project ideas are fixed-width in order to preserve the indentation. If they get formatted strangely, you might be able to fix it by increasing the width of your browser window or zooming out.
Calibrating With Cards
by  — 2019-08-08T06:44:44.853Z

In this post, I'll try to bring together two things I enjoy: rationality and magic. Like Hazard, I've also practiced close-up magic for a good amount of time now. After recently seeing Tyler Alterman make a Facebook post about estimations and System 1, it occurred to me that there are a few calibration exercises you can do with a deck of playing cards. The three exercises below are all variants of cutting/manipulating a deck of cards, and then trying to intuit something about the deck.

This serves three purposes:

Get a feel for your System 1:

The goal of the following three exercises is to see how good your gut is at estimating uncertainty (hint: probably better than you think!)

Improve calibration:

These exercises all allow for some room for error. You can set your confidence intervals and see how quickly you can get calibrated using first principles.

Practice cool party tricks:

While I don't intend for this to be a full-on magic tutorial, the exercises I outline are building blocks for magic tricks, and even demonstrating your super-calibration (after getting good) might be impressive.

Below are the three exercises. If you have a deck of cards handy, you can tag along!

Cut Estimation

The simplest exercise is as follows:

Lift up a packet of cards.
Estimate how many cards you've picked up.
Count to check how many you actually picked up.

You have a few obvious reference points. The entire deck is 52 cards, and you can easily tell if you've lifted up more or less than half (and this recurses). With a little practice. I've found that my gut is pretty good at this sort of thing. I'll ask myself how many cards, and there will be a number that feels right. It's usually quite close.

Things to pay attention to:

When your System 2 estimate conflicts with your System 1 gut answer of how many cards you cut off.
Whether you are systematically over or underestimating the amount.

Repeating The Cut

This is similar to the first one:

Cut to a card.
Replace the pack on the deck.
Cut to the same card again.

Things to pay attention to:

When you try to cut to the card a second time, how quickly does your gut know that you got it right or wrong?
What does it feel like to "know" that the pack of cards in your hand is not the same size as the first time?
If you know you got it wrong, do you know if you cut too much or too little? And by how much?

Riffle Peek

T...
Is there a source/market for LW-related t-shirts?
by jooyous — 2019-08-08T04:30:17.141Z

I think some of the sayings that come from LW and surrounding community would work well on t-shirts. I would both be interested in making and buying them.

Is there already a source of such t-shirts?

If not, is there interest?

What is the best platform for these t-shirts?
Verification and Transparency
by DanielFilan — 2019-08-08T01:50:00.935Z

Epistemic status: I’ve thought about this topic in general for a while, and recently spent half an hour thinking about it in a somewhat focussed way.

Verification and transparency are two kinds of things you can do to or with a software system. Verification is where you use a program to prove whether or not a system of interest has a property of interest. Transparency is where you use tools to make it clear how the software system works. I claim that these are intimately related.

Examples of verification

Proving that an alleged compiler actually implements the desired semantics of a system (for example, this verified implementation of ML).

Proving that a neural network’s classifications of a set of possible inputs are invariant under small perturbations to those inputs (for example, the system described in this paper).

Example of transparency

Sharing the source code of a program, rather than just compiled machine code (as encouraged by the open-source software movement).

Demonstrating the types of inputs that neurons in a neural network are sensitive to (techniques like this are discussed in the fantastic Building Blocks of Interpretability blog post).

How verification and transparency are sort of the same

Apart from aesthetic cases, the purpose of transparency is to make the system transparent to some audience, so that members of that audience can learn about the system, and have that knowledge be intimately and necessarily entangled with the actual facts about the system. In other words, the purpose is to allow the users to verify certain properties of the system. As such, you might wonder why typical transparency methods look different than typical verification methods, which also have as a purpose allowing users to verify certain properties of a system.

How verification and transparency are different

Verification systems typically work by having a user specify a proposition to be verified, and then attempting to prove or disprove it. Transparency systems, on the other hand, provide an artefact that makes it easier to prove or disprove many properties of interest. It’s also the case that engagement with the ‘transparency artefact’ need not take the form of coming up with a proposition and then attempting to prove or disprove it: one may well instead interleave proving steps and specification steps, by looking at the artefact, having int...
Toy model piece #2: Combining short and long range partial preferences
by Stuart_Armstrong — 2019-08-08T00:11:39.578Z

I'm working towards a toy model that will illustrate all the steps in the research agenda. It will start with some algorithmic stand-in for the "human", and proceed to create the UH, following all the steps in that research agenda. So I'll be posting a series of "toy model pieces", that will then be ultimately combined in a full toy model. Along the way, I hope to get a better understanding of how to do the research agenda in practice, and maybe even modify that agenda based on insights making the toy model.

For this post, I'll look in more detail into how to combine different types of (partial) preferences.

Short-distance, long-distance, and other preferences

I normally use population ethics as my go-to-example for a tension between different types of preferences. You can get a lot of mileage by contrasting the repugnance of the repugnant conclusion with the seeming intuitiveness of the mere addition argument.

However, many people who read this will have strong opinions about population ethics, or at least some opinions. Since I'm not trying to convince anyone of my particular population ethics here, I thought it best to shift to another setting where we could see similar tensions at work, without the baggage.

Living in a world of smiles

Suppose you have three somewhat contradictory ethical intuitions. Or rather, in the formulation of my research agenda, two somewhat contradictory partial preferences.

The second is that any world would be better if people smiled more (P1). The third is that if almost everyone smiles all the time, it gets really creepy (P2).

Now, the proper way of resolving those preferences is to appeal to meta-preferences, or to cut them up into their web of connotations: why do we value smiles? Is it because people are happy? Why do we find universal smiling creepy? Is it because we fear that something unnatural is making them smile that way? That's the proper way of resolving those preferences.

However, let's pretend there are no meta-preferences, and no connotations, and just try to combine the preferences as given.

Smiles and worlds

Fix the population to a hundred people, and let W be the set of worlds. This set will contain one hundred and one different worlds, described by w(n), where 0≤n≤100 is an integer, denoting the number of people smiling in these worlds.

We can formalise the preferences as follows:

P1={w(n)≤1w(m)∣n≤m}.

P2={w(n)≤2w(

...
Four Ways An Impact Measure Could Help Alignment
by Matthew Barnett — 2019-08-08T00:10:14.304Z

Impact penalties are designed to help prevent an artificial intelligence from taking actions which are catastrophic.

Despite the apparent simplicity of this approach, there are in fact a plurality of different frameworks under which impact measures could prove helpful. In this post, I seek to clarify the different ways that an impact measure could ultimately help align an artificial intelligence or otherwise benefit the long-term future.

It think it's possible some critiques of impact are grounded in an intuition that it doesn't help us achieve X, where X is something that the speaker thought impact was supposed to help us with, or is something that would be good to have in general. The obvious reply to these critiques is then to say that it was never intended to do X, and that impact penalties aren't meant to be a complete solution to alignment.

My hope is that in distinguishing the ways that impact penalties can help alignment, I will shed light on why some people are more pessimistic or optimistic than others. I am not necessarily endorsing the study of impact measurements as an especially tractable or important research area, but I do think it's useful to gather some of the strongest arguments for it.

Roughly speaking, I think that that an impact measure could potentially help humanity in at least one of four main scenarios.

1. Designing a utility function that roughly optimizes for what humans reflectively value, but with a recognition that mistakes are possible such that regularizing against extreme maxima seems like a good idea (ie. Impact as a regularizer).

2. Constructing an environment for testing AIs that we want to be extra careful about due to uncertainty regarding their ability to do something extremely dangerous (ie. Impact as a safety protocol).

3. Creating early-stage task AIs that have a limited function, but are not intended to do any large scale world optimization (ie. Impact as an influence-limiter).

4. Less directly, impact measures could still help humanity with alignment because researching them could allow us to make meaningful progress on deconfusion (ie Impact as deconfusion).

Impact as a regularizer

In machine learning a regularizer is a term that we add to our loss function or training process that reduces the capacity of a model in the hopes of being able to generalize better.

One common instance of a regularizer is a scaled L2 norm p...