trentmkelly/LessWrong-43k huggingface dataset:
Inadequate Equilibria vs. Governance of the Commons
by Martin Sustrik — 2018-05-25T13:17:21.981Z

This is a cross post from http://250bpm.com/blog:128.

Introduction

In the past I've reviewed Eliezer Yudkowsky's "Inadequate Equilibria" book. My main complaint was that while it explains the problem of suboptimal Nash equilibria very well, it doesn't propose any solutions. Instead, it says that we should be aware of such coordination failures and we should expect ourselves to fare better than the official institutions in such cases. What Yudkowsky is saying (if I understand him correctly) is that given that the treatment of short bowel syndrome in babies is stuck in an inadequate equilibrium, there's no way to fix the problem on the system level. However, you can spot the problem and acquire the medication needed to keep your kid alive from abroad yourself.

In my review I've sketched a couple of ideas how to approach the problem, but that was just me trying to be clever. Something backed by evidence would make me much more happy.

So I've decided to have a look at how people are getting out of suboptimal equilibria in the real world. And that's how I've got to the Elinor Ostrom's book "Governing the Commons (The Evolution of Institutions for Collective Action)".

The book is very explicitly covering the same problem as Yudkowsky. As Ostrom say about her research in her Nobel lecture: "One of the key questions that we've been addressing is 'Are rational individuals hopelessly trapped in dilemmas?'"

Ostrom is an economist with a game-theoretic bent. But she's not a theoretician. She's a field researcher. I've expected some good read and I wasn't disappointed.

The book focuses on special subset of coordination problems, problems of management of what Ostrom calls "common pool resources". Common pool resource is something that, unlike private property, is not exclusionary, something that one cannot easily restrict people from taking advantage of, but that, unlike public property, can be depleted by excessive usage. So, for example, it's hard to exclude others from fishing in the ocean, yet, the fish stock can be depleted by overfishing. Ocean fishery is a common pool resource. Compare that with a private fish farm where others are prevented from fishing by a barrier. And on the other side, contrast it with a public weather forecast which doesn't get depleted as more people turn on the radio.

On the game theoretic level, these sce...
When is unaligned AI morally valuable?
by paulfchristiano — 2018-05-25T01:57:55.579Z

Suppose that AI systems built by humans spread throughout the universe and achieve their goals. I see two quite different reasons this outcome could be good:

Those AI systems are aligned with humans; their preferences are our preferences.
Those AI systems flourish on their own terms, and we are happy for them even though they have different preferences.

I spend most of my time thinking about option #1. But I think option #2 is a plausible plan B.

Understanding how happy we should be with an unaligned AI flourishing on its own terms, and especially which unaligned AIs we should be happy about, seems like a very important moral question.

I currently feel very uncertain about this question; if you forced me to guess, I’d estimate that option #2 allows us to recover 25% of the expected value that we lose by building unaligned AI. But after more thinking, that number could go down to 0% or up to >90%.

Definition

In this post I’ll say that an AI is a good successor if I believe that building such an AI and “handing it the keys” is a reasonable thing to do with the universe. Concretely, I’ll say an AI is a good successor if I’d prefer give it control of the world than accept a gamble where we have a 10% chance of extinction and a 90% chance of building an aligned AI.

In this post I’ll think mostly about what happens with the rest of the universe, rather than what happens to us here on Earth. I’m wondering whether we would appreciate what our successors do with all of the other stars and galaxies — will we be happy with how they use the universe’s resources?

Note that a competent aligned AI is a good successor, because “handing it the keys” doesn’t actually amount to giving up any control over the universe. In this post I’m wondering which unaligned AIs are good successors.

Preface: in favor of alignment

I believe that building an aligned AI is by far the most likely way to achieve a good outcome. An aligned AI allows us to continue refining our own views about what kind of life we want to exist and what kind of world we want to create — there is no indication that we are going to have satisfactory answers to these questions prior to the time when we build AI.

I don’t think this is parochial. Once we understand what makes life worth living, we can fill the universe with an astronomical diversity of awesome experiences. To the extent that’s the right answer, it’s something I ...
Decision theory and zero-sum game theory, NP and PSPACE
by jessicata — 2018-05-24T08:03:18.721Z

(Cross-posted from my blog)

At a rough level:

Decision theory is about making decisions to maximize some objective function.
Zero-sum game theory is about making decisions to optimize some objective function while someone else is making decisions to minimize this objective function.

These are quite different.

Decision theory and NP

Decision theory roughly corresponds to the NP complexity class. Consider the following problem:

Given a set of items, each of which has a integer-valued value and weight, does there exist a subset with total weight less than w and total value at least v?

(It turns out that finding a solution is not much harder than determining whether there is a solution; if you know how to tell whether there is a solution to arbitrary problems of this form, you can in particular tell if there is a solution that uses any particular item.)

This is the knapsack problem, and it is in NP. Given a candidate solution, it is easy to check whether it actually is a solution: you just count the values and the weights. Since this solution would constitute a proof that the answer to the question is “yes”, and a solution exists whenever the answer is “yes”, this problem is in NP.

The following is a general form for NP problems:

∃x1∈{0,1}∃x2∈{0,1}…∃xk∈{0,1}f(x1,...,xk)

where f is a specification of a circuit (say, made of AND, OR, and NOT gates) that outputs a single Boolean value. That is, the problem is to decide whether there is some assignment of values to x1,…,xk that f outputs true on. This is a variant of the Boolean satisfiability problem.

In decision theory (and in NP), all optimization is in the same direction. The only quantifier is ∃.

Zero-sum game theory and PSPACE

Zero-sum game theory roughly corresponds to the PSPACE complexity class. Consider the following problem:

Given a specification of a Reversi game state (on an arbitrarily-large square board), does there exists a policy for the light player that guarantees a win?

(It turns out that winning the game is not much harder than determining whether there is a winning policy; if you know how to tell whether there is a solution to arbitrary problems of this form, then in particular you can tell if dark can win given a starting move by light.)

This problem is in PSPACE: it can be solved by a Turing machine using a polynomial amount of space. This Turing machine works through the minimax algorithm: it simula... ,,,,,,,,,,,,,,,,,
Expressive Vocabulary
by Alicorn — 2018-05-24T06:59:58.072Z

"Thou shalt not strike terms from others' expressive vocabulary without suitable replacement." - me

Suppose your friend says: "I don't buy that brand of dip. It's full of chemicals."

Reasonable answer: "I'm skeptical that any of them are harmful in these quantities; we don't have much reason to believe that."

Reasonable answer: "Yellow 5? Are you allergic?"

Reasonable answer: "Okay, let's get the kind with four easily recognizable ingredients."

No: "Technically, everything is chemicals. Dihydrogen monoxide!"

Pedantry is seldom a way to make friends and influence people, but this example particularly gets my goat because there doesn't seem to actually exist a word in English for the thing you know perfectly well people mean when they say "chemicals". When I tried to find one on Twitter, the closest options were "toxins" and "additives". But neither is right. "Toxins" excludes yellow 5 - or, whether it does or not might be a point of contention; but it isn't the thing originally expressed with the word "chemicals". People may want to avoid - or otherwise discuss - "chemicals" for reasons other than thinking they're literally toxic; if I tell a maid I'm sensitive to chemical smells but vinegar is okay this is useful information. "Additives" includes, say, added sugar, which, while a plausible complaint, is a separate complaint.

Suppose your grandma says, "Okay, no technology at the dinner table."

Reasonable answer: "I'll put the laptop away to make room for the potatoes, but I need the phone because I get anxious without it."

Reasonable answer: "Sure, Grandma."

Reasonable answer: "We can try that until Uncle Bill starts making easily falsified claims about Flat Earth."

No: "Technically, the dinner table is a technology. And so are your glasses, Grandma."

In this case a more precise word exists - "electronics" ambiguously includes the chandelier but at least firmly sets aside the question of whether your grandma wants you to eat naked and with your bare hands. But refusing to know what she meant because she could have gotten closer to saying it, not even literally (she isn't being metaphorical), but technically, pedan...
There is a war.
by Benquo — 2018-05-24T06:44:36.197Z
Link: http://benjaminrosshoffman.com/there-is-a-war/

Households vs markets

The first symptom was the clutter on the kitchen counter. One cutting board, two pans, one knife. My colleagues had arrived at the rented house the day before, so they'd had plenty of time to arrange things to their liking. I was sure the clutter was not to their liking, if they noticed it at all.

A teachable moment. Instead of tidying the counter myself, and accreting a small amount of resentment, I suggested to one of them that she think of the things on the counter as things that were in her power to arrange however she liked, to suit her taste, selfishly. (“Just as you might optimize your text editor to suit your workflow,” my other colleague chimed in.) She took this suggestion, and spent a few minutes arranging and rearranging the items on the counter. She put away the knife in the knife block.

A puzzle. She noted that she doesn’t usually think of the items in her home this way. Instead, household chores feel as though they are impositions from an abstract, outside authority. She was capable of accessing this other, more pleasant way of working on the things around her. Why wasn’t it natural for her to feel that way in her own home?

An hypothesis. In our usual mode of life, there is a separation between a job - which is done for someone else, to satisfy someone else’s standards, outside the home - and consumption, which is at least ostensibly done to suit one’s own taste. One of the goods you can buy with an income from a job is a nice place to live, and you can also buy services to keep the place clean and tidy. For the most part, you maintain the place you live by leaving it, and entering the domain of an outside authority. Household chores are the remainder that cannot efficiently be outsourced, or an echo of a previous era in which such outsourcing was less common.

A household. I pointed out that in the past, people mostly did not work for a salary. Instead, they worked on things in their local environment. For the most part, this was not in order to receive money, but in order to have the thing, even if it was part of a process directed towards producing goods for trade. Home improvement used to be, not a hobby or special interest, but what one did if one wanted an improved home. Even those without a substantial amount of arable land to work might keep a garden. A minority engaged in trades or professions. Loops were mostly c...
Do Conversations Often Circle Back To The Same Topic?
by  — 2018-05-24T03:07:38.516Z
Link: https://mindlevelup.wordpress.com/2018/05/23/do-conversations-often-circle-back-to-the-same-topic/

A friend asked me about my thoughts on conversations w/ friends that often circle back to the same topics over and over. Includes some exploration on novelty and ways to make conversations not do this.
Striving vs Cruising
by norswap — 2018-05-23T23:42:36.768Z
Link: http://norswap.com/striving-cruising/

Epistemic Status: Not really epistemic or instrumental rationality. Meta-rationality? In any case, something rationality can help operationalize, although that is not the object here. Might still be of interest to this crowd.

There is a question that everyone tends to ask themselves at one point or another and that question is What should I do with my life?

I don't think there is a higher value that overrides all other. Unless it is so general as to be tautological. Some call this tautological value happiness, or more scientifically subjective well-being or life satisfaction.

We often try to answer the question in terms of goals and objectives. That's not necessarily the right approach. After all, all we can ever decide is what to do right here, right now. Still, it raises the following question: Is it necessary to achieve to be happy? Should you just cruise through life and take in the sights? Or strive towards a goal worthy of you?

I sincerely believe that happiness requires meaning, and meaning requires a struggle. You should strive, but focus on the quality of the striving rather than on the end goal. Cruise if you must, but cruise towards something. Strive, but don't be consumed by the goal.

There were many inspirations that led me to that conclusion, but the following three I find the most illuminating:

Finding Your Purpose and Living a Meaningful Life
A thorough examination of the idea of striving versus cruising.
If you don't know what to do with your life, have you at least started looking?
Strive to be yourself.

The Most Important Question of Your Life
What pain do you want in life? Looking for the pain you enjoy or even tolerate better than other goes a long way towards finding somewhat you will find meaningful.

How to Worry Less
The vast majority of life consists of getting somewhere else. There are very few moments of arrival, and they are over as soon as they begin. If you feel like you need to be there before you can fully relax into where you are, then you will seldom be relaxed in life.

Post Scriptum

This used to be a huge series of posts tackling the grand question of the meaning of life. I spent tens of hours writing and editing these posts... and came to hate them with a passion. It came from a good place: I wanted to share all my inspirations, all the connections that I'd developed. But at the same time it felt like I wa...
Epistemic Circularity
by Gordon Seidoh Worley — 2018-05-23T21:00:51.822Z

Note: I wrote a better article about this; you should read that instead of this stub: The Problem of the Criterion.

I've previously argued for the existence of what I've called a "free variable" in epistemology that forces a choice between ways of knowing because no one way of knowing (system of epistemology or simply an epistemology) can be both complete and consistent. In the process of working on a current project and not wanting to have to rederive anything someone else has already argued for in academic literature, I discovered this feature already has a name and has been written about: epistemic circularity.

I find it sort of surprising we've not addressed this more within the LW community, although it's perhaps less surprising than might otherwise be expected given LW's positivist leanings. I don't have much to say on epistemic circularity at the moment, although I do consider it critical to my worldview and a crux of my thinking about philosophical conservatism for alignment research, but I did at least want to bring some wider attention to a concept that, to my recollection, we've ignored as a community.
Simulation hypothesis and substrate-independence of mental states
by Tiago de Vassal — 2018-05-23T19:07:05.772Z

I’m grateful to Michaël Trazzi. We had a great discussion on this topic, and he then was kind enough to proofread the draft of this article.

Abstract

Substrate-independence of mental states proves too much, so it is not necessarily true. The consequence is that simulations of conscious beings are not necessarily conscious themselves.

A simulation is only a means for an observer to see the development of a system over time, not the creation of that system. Hence, changing the simulation has no influence on the simulated system.

I - Introduction and definitions

The simulation hypothesis is not only widespread in philosophy, it has also found its way in pop-culture. The best known version of this idea may very well be Nick Bostrom’s “Are you living in a computer simulation?” (2003). My point along this article will be that it is impossible or unlikely for you to be in (as in “inside”) a computer simulation. To that end, I will provide some counterintuitive consequences of substrate-independence of mental states.

As stated in Nick Bostrom’s article, substrate-independence is :

“The idea is that mental states can supervene on any of a broad class of physical substrates. Provided a system implements the right sort of computational structures and processes, it can be associated with conscious experiences.”

Substrate-independence is a necessary condition for the computational theory of mind, wich is itself necessary for the simulation of consciousness as it appears in Nick Bostrom’s article.

Consciousness is the mental state associated with subjective, qualitative experiences (also named qualia).

II - What counts as a simulation?

1.

When talking about simulations, we often picture a computer doing a huge amount of calculation, based on some chosen parameters describing a system accurately enough, and displaying the results onto a screen. For sure, that would be a simulation. But what happens if you remove the screen? Would it lead to the end of the simulation? What about the other parts of the computer, are they as expendable as the screen?

intuitively, simulations are closely related to computer simulations. But try to imagine the least technologically advanced simulation. What is it made of? In the history of electronics, we have seen computers made out of many different things. If you are willing to sacrifice calculation speed, you can use anything, from water flows to domi...
Glide #1: Learning Rationality from Absurdity
by musicmage4114 — 2018-05-23T18:18:01.465Z

Glide Meditation #1: A Reflection on "The Simple Truth"

My first exposure to “The Simple Truth”, the first entry in the original collection of Eliezer Yudkowsky’s Sequences, was actually a dramatic reading I found on YouTube. I had intended to read it during my lunch break at work, but Yudkowsky’s personal website, where that particular entry is hosted, is blocked by my employer’s Internet filter. Now that I’ve gone back and actually read the essay in its original medium, I feel like that particular turn of events is rather serendipitous; indeed, I don’t think this essay would have had nearly the same impact for me if had only read it. The fact that essay itself is written as a dialogue naturally lends itself to an audio adaptation, and that in turn made it far more accessible.

As I sat and listened to Yudkowsky’s mental constructs debate over the details of the sheep-counting device, my emotional arc began at mild confusion (Yudkowsky had posited a world without the concept of counting without explicitly stating so), morphed into surprisingly intense frustration for most of the duration (the extent to which Mark must be obtuse in order for the dialogue to make its intended point truly must read/heard to be believed), and ended at resigned amusement (I think I understood it, and the ending was humorous, but I questioned whether sitting through such a frustrating ordeal was ultimately worth it).

On its surface, “The Simple Truth” reads like an absurdist one-act play, and its style of humor is indeed very much reminiscent of Rosencrantz and Guildenstern Are Dead. For me, The Shepherd, Autrey, and Mark exhibit various methods of reasoning and behavior that impede the effective practice of the art of inquiry and discerning truth, with Inspector Darwin serving via prosopopoeia as the voice of reality. Through their actions, the former three characters highlight via contrast specific ideals of rational thought and effective communication, though I strongly suspect this may not be their intended interpretation. Therefore, as we move forward through these essays (both his and mine), let us accept as an axiom Yudkowsky’s temporary death.

My (admittedly sparse) framework thus established, what is the significance these characters?

Mark presupposes his worldview, though what that worldview is seems to change from moment to moment, and not in the way prescribed by ...
A Short Celebratory / Appreciation Post
by lionhearted (Sebastian Marshall) — 2018-05-23T00:02:18.423Z

Three thoughts.

1. The new LessWrong 2.0, at least for me personally, has been really marvelous. I've greatly enjoyed some of the discussions on mindfulness, pragmatic achieving type-stuff, definitions and clarifications and naming of broad concepts, and explorations of technology and artificial intelligence. Kaj Sotala and Zvi Mowshowitz both offered some fantastic insights that I particularly enjoyed, and I came across some of Nate Soares's writing on motivation and decisionmaking via here which is incredibly timely and insightful for me.

2. I wanted to stop and reflect for a moment that this took a lot of work to make happen. I imagine that a conservative estimate of how much the volunteer time spent programming and designing and curating here would have cost in salary at market rates is... at least $300,000 USD? More? That's... like... a bunch of people put hundreds of thousands of dollars worth of market-priced labor into building this place, and it's generating a lot of insights that are really fantastic. Wow. It's easy to forget how much work goes into making the world run smoothly. I feel both a sense of mild awe at the ability of the team to build the place, and gratitude that they did so. Coordination is hard. Design is hard. Community-building is hard. And it worked. That's so cool.

3. I also want to note a strange feeling I had — it went something like, "If I wrote only the first and second points, this wouldn't be a good post." There's a strange norm in civilization at large — at least, certainly in developed Western countries — against just stopping and offering a general sense of appreciation and respect for this type of work. I thought it might be useful to reflect on why, and I came up with a few potential reasons — first, for whatever damn reason, sprezzatura is seen as high-status and proper behavior; thus, the people who do hard work of making things happen will play off the work required ("ahh, no big deal") and generally, someone in the audience or community remarking on all the work is... mildly discouraged. Second, I suppose that if done too often, it'd be low signal-to-noise ratio, which is valid. Third, maybe there's some lingering "don't put too much trust in authorities" sort of norm across Western civilization; we call things "cult-like"...