trentmkelly/LessWrong-43k huggingface dataset:
Simulation Argument: Why aren't ancestor simulations outnumbered by transhumans?
by  — 2019-08-22T09:07:07.533Z

This is a point of confusion I still have with the simulation argument: Upon learning that we are in an ancestor simulation, should we be any less surprised? It would be odd for a future civilization to dedicate a large fraction of their computational resources towards simulating early 21st century humans instead of happy transhuman living in base reality; shouldn't we therefore be equally perplexed that we aren't transhumans?

I guess the question boils down to the choice of reference classes, so what makes the reference class "early 21st century humans" so special? Why not widen the reference class to include all conscious minds, or narrow it down to the exact quantum state of a brain?

Furthermore, if you're convinced by the simulation argument, why not believe that you're a Boltzmann brain instead using the same line of argument?
Markets are Universal for Logical Induction
by johnswentworth — 2019-08-22T06:44:56.532Z

Background

Logical Induction

is the best framework currently available for thinking about logical uncertainty - i.e. the “probability” that the

twin primes conjecture

is true, or that the (10101010)th digit of pi is 3. This is important for lots of reasons, and you should read the introduction of

the paper

(or

the abridged version

) for a much more detailed background.

The general idea of logical induction is to assign probability-like numbers to logical statements like “the (10101010)th digit of pi is 3”, and to refine these “probabilities” over time as the system thinks more. To create these “probabilities”, each statement is associated with an asset in a prediction market, which eventually pays $1 if the statement is proven true, or $0 if it is proven false. The “probabilities” are then the prices of these assets.

(It’s also possible that a statement is never proven or disproven, and one of the many interesting results of the paper is that logical inductors assign useful prices in that case too.)

The logical induction paper has two main pieces. First, it introduces the logical induction criterion: a system which assigns prices to statements over time is called a “logical inductor” if the prices cannot be exploited by any polynomial-time trading algorithm. The paper then shows that this criterion implies that the prices have a whole slew of useful, intuitive, probability-like properties.

The second main piece of the paper proves that at least one logical inductor is computable: the paper constructs an (extremely slow) algorithm to compute inexploitable prices for logical statements. The algorithm works by running a prediction market in which every possible polynomial-time trader is a participant. Naturally, the prices in this market turn out to be inexploitable by any polynomial-time trader - so, this giant simulated prediction market is a logical inductor.

Our Goal

An analogy: one could imagine a decision theorist making a list of cool properties they want their decision theory to have, then saying “well, here’s one possible decision algorithm which satisfies these properties: maximize expected utility”. That would be cool and useful, but what we really want is a theorem saying that any possible decision algorithm which satisfies the cool properties can be represented as maximizing expected utility.

This is analogous to the situation in the logical induction paper...
Announcement: Writing Day Today (Thursday)
by Ben Pace — 2019-08-22T04:48:38.086Z

The MIRI Summer Fellows Program is having a writing day, where the participants are given a whole day to write whatever LessWrong / AI Alignment Forum posts that they like. This is to practice the skills needed for forum participation as a research strategy.

On an average day LessWrong gets about 5-6 posts, but last year this generated 28 posts. It's likely this will happen again, starting mid-to-late afternoon PDT.

It's pretty overwhelming to try to read-and-comment on 28 posts in a day, so we're gonna make sure this isn't the only chance you'll have to interact with these posts. In the coming weeks I'll post roundups, highlighting 3-5 of the posts, giving them a second airing for comments and discussion.
Call for contributors to the Alignment Newsletter
by Rohin Shah — 2019-08-21T18:21:31.113Z

TL;DR: I am looking for (possibly paid) contributors to write summaries and opinions for the Alignment Newsletter. This is currently experimental, but I estimate ~80% chance that it will become long-term, and so I’m looking for people who are likely to contribute at least 20 summaries over the course of their tenure at the newsletter (see caveats in the post). To apply, read this

doc

, write an example summary + opinion, and fill out this

form

by Friday, September 6. I am also looking for someone to take over the work of publishing the newsletter (~1-3 hours per week); please send me an

email

if you’d be interested in this.

ETA: I now have enough interest in the publisher role that I would be shocked if none of them worked out. Feel free to continue expressing interest if you think you'd particularly benefit from doing the work, or if you think you'd be particularly good at it.

Roles I am looking for

Publisher: Once all of the summaries and opinions are written, you would turn them into an actual newsletter, send it out for proofreading, fix any typos found, update the

database

, etc. This currently takes me around half an hour per newsletter. Ideally, you would also take on some tasks that I haven’t found the time for: improving the visual design of the newsletter, A/B testing different versions to see what people engage with, publicity, and so on, for a total of ~1-3 hours per week.

Since I don’t yet have the setup to pay people to help with the newsletter, I am only looking for expressions of interest. If you think you’d be interested in this role, click

this link

to email me at

rohinmshah@berkeley.edu

with the subject line “Interested in publisher role for Alignment Newsletter EOM”. If I do end up hiring for the publisher role I’ll reach out to you with more details.

The rest of this doc will be focused on the more substantial role:

Content creator: You would choose articles that you’re interested in, and write summaries and opinions for them, that would then be published in the newsletter.

Why am I looking for content creators?

In the past few months, I haven’t been allocating as much time to the newsletter (you may have noticed they’re coming out every other week now). There have been many other things that seem more important to do. This is both because I’m more optimistic about the other work I’m doing, and because I no longer find it as useful to re...
Two senses of “optimizer”
by Joar Skalse — 2019-08-21T16:02:08.985Z

The word “optimizer” can be used in at least two different ways.

First, a system can be an “optimizer” in the sense that it is solving a computational optimization problem. A computer running a linear program solver, a SAT-solver, or gradient descent, would be an example of a system that is an “optimizer” in this sense. That is, it runs an optimization algorithm. Let “optimizer_1” denote this concept.

Second, a system can be an “optimizer” in the sense that it optimizes its environment. A human is an optimizer in this sense, because we robustly take actions that push our environment in a certain direction. A reinforcement learning agent can also be thought of as an optimizer in this sense, but confined to whatever environment it is run in. This is the sense in which “optimizer” is used in posts such as this. Let “optimizer_2” denote this concept.

These two concepts are distinct. Say that you somehow hook up a linear program solver to a reinforcement learning environment. Unless you do the “hooking up” in a particularly creative way there is no reason to assume that the output of the linear program solver would push the environment in a particular direction. Hence a linear program solver is an optimizer_1, but not an optimizer_2. On the other hand, a simple tabular RL agent would eventually come to systematically push the environment in a particular direction, and is hence an optimizer_2. However, such a system does not run any internal optimization algorithm, and is therefore not an optimizer_1. This means that a system can be an optimizer_1 while not being an optimizer_2, and vice versa.

There are some arguments related to AI safety that seem to conflate these two concepts. In Superintelligence (pg 153), on the topic of Tool AI, Nick Bostrom writes that:

A second place where trouble could arise is in the course of the software’s operation. If the methods that the software uses to search for a solution are sufficiently sophisticated, they may include provisions for managing the search process itself in an intelligent manner. In this case, the machine running the software may begin to seem less like a mere tool and more like an agent. Thus, the software may start by developing a plan for how to go about its search for a solution. The plan may specify which areas to explore first and with what methods, what data to gather, and how to make best use of available c

...
Paradoxical Advice Thread
by Hazard — 2019-08-21T14:50:51.465Z

(Edited to be more clear)

One can find plenty of witty advice catch phrases that appear to be in direct conflict with each other: "Early bird gets the work, but the second mouse gets the cheese" (from a comment below)

You can also find this with things that are more detailed frames than cliche advice. One that I've been mulling over is "decision makers need skin in the game to not be act stupid" and "asymmetric justice (people being punished for messing up) incentives inaction."

The "quality" of any advice varies wildly with what you are trying to do and what your specific context is. It's not going to be the case that the "best" thing for everyone to do is either follow the mandate of "have skin in the game" or "avoid asymmetric justice". In fact, you might be thinking about ways that those to ideas aren't even really at odds with each other. Yet my personal experience was hearing about skin in the game, going "that sounds obviously important", hearing about asymmetric justice and going "this sounds obviously important", and later in a conversation with a friend going, "Wait a second, my low resolution understanding of these ideas puts them in direct odds with each other."

I've learned a bit from hunting for pairs of advice/principles that both sound like good ideas to me, but at first glance seem to be at odds. Then exploring "why do they feel like they conflict?" Sometimes I find that they don't actually advise completely different approaches. Sometimes I find that each piece of advice is paying attention to a different dimension of trade off, and that conceivably I could do well by paying attention to each dimension separately.

I'm inviting people to use this post to document paradoxes you find in your own thinking. Explore where the sense of conflict comes from, what there is to learn from each side of the, what a synthesis might be, or when one might apply more than another.

Don't feel like you have to solve the whole thing, just try and unravel the problem space a bit.
Three Levels of Motivation
by DragonGod — 2019-08-21T09:24:39.164Z

Disclaimer: The information herein is not at all new. I wrote this a year ago, but when I wanted to link to it recently I found out it wasn't in existence on LW (except as a link post which isn't very satisfactory). I decided to polish it up and post it here.

Epistemic Status

This is basic information, and while it is original to me, I make no claim that the the content thereof is new. Nevertheless, I think it is important enough to be worth posting.

Abstract

I outline a framework for thinking about the motivations (goals/values) of agents.

Introduction

In a few articles I've read (written at different times) and in some conversations I've had, I identified what I perceived as confused thinking regarding the motivations of agents. This post is my attempt to rectify that confused thinking and provide a framework that will help people reason more coherently about agents' motivations.

Three Levels of Motivation

I have identified three levels at which the motivations of an agent can be considered. I will briefly outline these levels, and then expand on them in subsequent sections.

Brief Outline

Level 1 (L1) - Ideal Specification: The motivations the designer created the agent with the intention of accomplishing. These exist exclusively in the designer's mind and are not part of the agent.

Level 2 (L2) - Designed Specification: The motivations that are explicitly represented in the agent itself. What motivations the agent makes decisions with the intention of accomplishing. These motivations exist within the agent program.

Level 3 (L3) - Revealed Specification: The motivations that are actually pursued by the agent via its actions; insomuch as the actions of the agent can be described as optimising over the output of some objective function, which function is it? This is a property of the agent function.

The three levels are interrelated, but are distinct. Traditional language does not clearly delineate which level of motivation one is talking about when one talks about an agent's motivations, and thus allows the potential for confusion to arise due to people using the same word(s) to refer to different concepts.

Case Studies

I will consider a couple of (toy) agents to more clearly illustrate the differences between the three levels of motivation.

Fred

Fred is a naive attempt at creating a Friendly Artificial Intelligence (FAI). Fred was designed with the intent...
Odds are not easier
by MrMind — 2019-08-21T08:34:22.286Z

Epistemic status: just a review of a well known math theorem and a brief rant about terminology.

Yesterday I saw another example of this: P(D) is just a normalizing constant for the posterior probability, and it's really hard (impossible?) to calculate, so let's switch to log odd probabilities, which are easier and the pesky term is canceled.

Except it's not: P(D)=P(D|H)P(H)+P(D|¬H)P(¬H), the second term is exactly what you need to get the probability in odd form, and if you have it you can very well calculate the prior for the data.

So please, whatever you write, stop saying that odds are easier. They are possibly more intuitive to manipulate, but they need exactly the same amount of information.
Lana Wachowski is doing a new Matrix movie
by mako yass — 2019-08-21T00:47:40.521Z

(Lily isn't on board, but her twitter bio right now contains "ex-film maker" so it probably has nothing to do with this project specifically)

Considering the themes of The Matrix and the amount of popular discussion of AGI alignment there has been since the previous instalments, this may turn out to be culturally significant.

I'm wondering if anyone knows where Lana is at wrt alignment stuff. Like, has she read Superintelligence? Did she ever have any contact with the alignment community?

(Original source)
What authors consistently give accurate pictures of complex topics they discuss?
by seez — 2019-08-21T00:09:52.057Z

When I read nonfiction, I often feel like it's a waste of time because I'm not very confident the author is portraying things accurately/non-misleadingly, and don't often have time or expertise to check (though I'd potentially be in doing this on 1/10 of books I read, or similar, to contribute to a group effort to source reliably books).

I am looking for people who write nonfiction books about complicated topics on which expert views vary (social sciences, theories of why/how important historical events happened, nutrition, overall takes on complex scientific questions like the expected economic effects of climate change) that pass spot checks, seem to weigh evidence rationally, seem focused on truth-seeking over promoting their pet theory/agenda/sensationalistic conclusion, pay more attention to the most important parts of the issue than the minutia, etc. I am guessing this trait generalizes between books written by the same author, but would be interested in cases in which this doesn't seem to hold.

I don't want suggestions for people in the rationalist community or very adjacent to it.
What are the reasons to *not* consider reducing AI-Xrisk the highest priority cause?
by David Scott Krueger (formerly: capybaralet) — 2019-08-20T21:45:12.118Z

ETA: I'll be adding things to the list that I think belong there.

I'm assuming a high level of credence in classic utilitarianism, and that AI-Xrisk is significant (e.g. roughly >10%), and timelines are not long (e.g. >50% ASI in <100years). ETA: For the purpose of this list, I don't care about questioning those assumptions.

Here's my current list (off the top of my head):

not your comparitive advantage
consider other Xrisks more threatening (top contenders: bio / nuclear)
infinite ethics (and maybe other fundamental ethical questions, e.g. to do with moral uncertainty)
S-risks
simulation hypothesis
ETA: AI has high moral value in expectation / by default
ETA: low tractability (either at present or in general)
ETA: Doomsday Argument as overwhelming evidence against futures with large number of minds

Also, does anyone want to say why they think none of these should change the picture? Or point to a good reference discussing this question? (etc.)
Has Moore's Law actually slowed down?
by Matthew Barnett — 2019-08-20T19:18:41.488Z

Moore's Law has been notorious for spurring a bunch of separate observations that are all covered under the umbrella of "Moore's law." But as far as I can tell, the real Moore's law is a fairly narrow prediction, which is that the transistor count on CPU chips will double approximately every two years.

Many people have told me that in recent years Moore's law has slowed down. Some have even told me they think it's stopped entirely. For example, the AI and compute article from OpenAI uses the past tense when talking about Moore's Law, "by comparison, Moore’s Law had an 18 month doubling period" (note that Wikipedia says, "The period is often quoted as 18 months because of a prediction by Intel executive David House").

Yet when I google Moore's law, I find this chart from Karl Rupp's website,

To me this looks very stable. And even though single thread performance has stopped improving by much, I can't really see how it's justified to use Moore's law in the past tense just yet. Am I missing something?
Cerebras Systems unveils a record 1.2 trillion transistor chip for AI
by avturchin — 2019-08-20T14:36:24.935Z
Link: https://venturebeat.com/2019/08/19/cerebras-systems-unveils-a-record-1-2-trillion-transistor-chip-for-ai/

From reddit comments:

....how scared should we be?

Gwern:

No idea. They don't provide benchmarks, and while they promise they are forthcoming, it sounds like it might be months. In the meantime, there's just vague hype about going from 'months to minutes'. Hardware startups have a long history of overpromising and underdelivering: it's hard to beat Nvidia & Moore's Law (remember all those 'analogue computing' startups?). It sure does sound interesting, though: 18GB of on-chip SRAM rather than HBM or DDR* RAM? 1.2 trillion transistors? Potentially FPGA-style streaming of data points through a single on-chip model with each layer being handled by different sets of cores? Sparsity multipliers? All quite interesting sounding and I will be very interested in the benchmarks, whenever they should be forthcoming. If nothing else, it is an extreme architecture of a type you rarely see.
Self-supervised learning & manipulative predictions
by Steven Byrnes — 2019-08-20T10:55:51.804Z

Abstract: I wrote recently about Self-Supervised Learning and AGI Safety in general. This post discusses one potential failure mode in more detail. Take a self-supervised learning system, designed to output accurate predictions for masked parts of a data-file. Now put it in an interactive environment (either by accident or on purpose). If the system builds these interactions into its world-model, it can start outputting manipulative answers instead of pure predictions. I explain how that might happen and briefly categorize possible solutions.

Epistemic status: Brainstorming.

Background and assumptions about the self-supervised-learning system

See my recent post Self-Supervised Learning and AGI Safety for background and context, but briefly, a self-supervised learning system is one where we take input data files, mask out some of the bits, and train the system to predict what those missing bits are.

Self-supervised ML today is most famously applied to text data: language models are trained by taking some text and trying to predict the next word (or previous word etc.). Self-supervised ML for videos is getting rapidly better, and other file types will undoubtedly follow. Human and animal brains also learn primarily by self-supervised learning—you predict everything you will see, hear, and feel before it happens, and mistakes are used to update the brain's internal models.

I'll assume that we get to AGI largely by following one of those two examples (i.e., modern ML or brain-like). That means I'm assuming that we will not do a meta-level search for self-supervised learning algorithms. That case is even worse; for all I know, maybe that search would turn up a paperclip maximizer posing as a self-supervised learning algorithm! Instead, I am assuming that the self-supervised learning algorithm is known and fixed (e.g. "Transformer + gradient descent" or "whatever the brain does"), and that the predictive model it creates has a known framework, structure, and modification rules, and that only its specific contents are a hard-to-interpret complicated mess. This assumption generally makes AGI safety problems much easier, yet I am arguing that even in this case, we can still get manipulation problems, if the self-supervised learner is put in an interactive environment.

Why might we put a self-supervised learner into an interactive environment?

My definition of an "intera...
Negative "eeny meeny miny moe"
by jefftk — 2019-08-20T02:48:41.509Z

As a kid, I learned the rhyme as:

Eeny, meeny, miny, moe,

Catch a tiger by the toe.

If he hollers, let him go,

Out goes Y, O, U!

Since kids can't predict where it will end, and adults are not supposed to try, it's a reasonably fair way of drawing lots.

At times I've heard versions where the selected person wins instead of loses, and while with two kids it doesn't matter, with three or more it matters a lot!

Let's model each kid having a choice at each stage between "accept" and "protest". While protesting probably doesn't work, if enough of you protest it might. If you do the positive version, where the selected kid wins, the winner accepts but the others may choose to protest. This isn't good: everyone has reason to protest except the single winner.

On the other hand, with the negative version, where one kid is eliminated at once, it's the other way around. When the first kid is eliminated they may protest, but the other kids all accept because then they retain their chance to win. With each successive round the dynamic is the same, plus the already-eliminated kids all choose accept out of a desire for fairness. Even with the last elimination there's still only one person choosing protest.

The iterative process is O(n) instead of O(1), but it also works much better because it keeps a majority for "accept" at each stage.

(If you have a very large group of kids, then I could imagine a O(log(n)) version being worth the added complexity. Divide the kids into three groups, and do negative eeny meeny miny moe on the groups. A third of the kids may protest, but you've still got two thirds accepting. Then redivide those remaining two thirds into three groups, and keep going.)

cross-posted from https://www.jefftk.com/p/negative-eeny-meeny-miny-moe