trentmkelly/LessWrong-43k huggingface dataset:
Revisiting HPMOR
by DragonGod — 2017-11-19T08:55:31.552Z

This is a reply to https://www.lesserwrong.com/posts/zCrKz6rWeG79DRQM2/double-crux-example-should-hpmor-be-on-the-front-page

I wasn't sure it was best replying to that comment, due to it being one month later, and the comment seeming to be more for the purposes of illustrating double crux than actually discussing HPMOR. Thus, I decided to post here.

I joined lesswrong through HPMOR (I joined earlier this year), but I didn't find HPMOR through Lesswrong. I think HPMOR brings people to Lesswrong, but I don't think a substantial portion of the people HPMOR would bring to Lesswrong would discover HPMOR through Lesswrong. HPMOR appeals to a broad spectrum of people, and can reach a much larger audience than Lesswrong does. I came through HPMOR, and I'm sure a substantial amount of our newcomers (post 2010 I guess) came through HPMOR as well. However, I don't think many of them discovered HPMOR through Lesswrong—I think very few of them actually will. The way I see it, once they are already at Lesswrong, done. Mission accomplished.

HPMOR is the bait, and the sequences are the hook. Majority of the people who benefit from HPMOR don't actually benefit from HPMOR on the front page. They didn't discover HPMOR through Lesswrong, and they wouldn't have. Majority of the people who discover Lesswrong through Lesswrong don't benefit (extensively) from HPMOR. Some of the people who discover Lesswrong through lesserwrong.com/lesswrong.com would be off put by HPMOR. Basically, I think http://www.lesserwrong.com and HPMOR market Lesswrong to different sets of people, with little overlap.

To summarise my points:

HPMOR is very valuable.
Due to its nature (fanfiction, Harry Potter, etc) majority of the people who benefit from HPMOR don't find HPMOR through Lesswrong. More generally, if they benefit from HPMOR, then it's not Lesswrong that is going to introduce them to it. (This is a "crux" for me. If this is false, it would change my opinion on prominently featuring HPMOR on the front page).
HPMOR is off putting to people who discover Lesswrong through Lesswrong. The negative impact of HPMOR on such people outweighs the positive impact of HPMOR on the (few) people who discover HPMOR through Lesswrong.
HPMOR on the front page is a net negative.
Tragedy of the Commons 
by DragonGod — 2017-11-18T20:30:57.841Z

This problem was originally seen in a manga chapter. I will describe the canon (form it appeared in the manga (with a few adjustments)) form, and a general form of the problem. I couldn't solve the problem in the little time I spent to think of it (before continuing with the manga), so it flagged my attention as "interesting".

Canon Form

There are five players, $(a_1, a_2, ..., a_5)$.
Every round, each of these players is given an initial fund of 5 coins.
There are 5 rounds in the game.
On each round, each player has two choices: to make a secret deposit in the tax fund, or in their personal account. The players have to deposit (in secrecy) all their coins on hand in at least one of the two accounts (they can deposit in both accounts). The players cannot show "hard" evidence of their deposits (no receipt, pictures, etc).
Money deposited in a player's personal account becomes the player's fund for the next round.
Money deposited in the tax fund is multiplied by two and shared equally among all five players (including players who deposited nothing, hence the name).
The deposits are made sequentially, with the order randomly determined each round.
Players do not know how much is inside the tax fund when they make their deposit.
The coins are worthless outside the game.
After the five rounds, all players that accumulated at least 40 coins would receive rewards as such:

1st: $100,000,000

2nd: $30,000,000

3rd: $3,000,000

4th/5th: $0

The utility of money grows linearly for the players.

The total amount of money is $133,000,000. If multiple people get a position, the money assigned to that position is split equally among the people.
All players who amassed below the required forty coins incur a debt of $100,000,000.

I use "rational" in a very specific sense, so saying all players are "perfectly rational" makes the problem useless. Nevertheless, in canon the participants of this game were all extraordinarily competent, and it was common knowledge * that they were all extraordinarily competent. To use jargon that I personally adopt:

The participants are "sensible" (they act solely to maximise their utility).
It is unknown whether the participants are "reasonable" (t

...
Implications of a feelings-first metaphysics?
by efficientfox — 2017-11-18T17:10:35.828Z

What would be the implications on decision-making, beliefs, and actions of a metaphysics that is structured as follows?:

All beliefs, ideas, actions, etc. are driven by feelings.
Reasons are descriptions of feelings.
Language fails to fully capture the sensational qualities of feelings.
Decision-making, belief adoption, etc. cannot be bounded by language.

Some initial ideas:

-I act in a way that makes me feel a sense of balance.

-I believe what enables me to act in a way that makes me feel a sense of balance.

-I structure my perception of the world, decisions, and actions out of this goal-driven pursuit of feeling a sense of balance.

-I do not have access to information/ideas/actions/etc outside of this.

Here is a map of what this would look like:

Emotional Awareness branches out into Balance and Map.

OODA loop connects emotional awareness to the ideas of balance and of a map. I observe how I feel (emotional awareness), orient my map to fit this feeling, decide how to act / what to believe / what information to take in based on how the map makes me feel that I would feel more balanced, and act in accordance with the decision.
Qualitative differences
by RST — 2017-11-18T09:02:08.102Z

Probably I am making a huge mistake resuming a ten years old discussion, especially because this is my first post. Anyway, let’s make this experiment.

One common argument for choosing “dust specks” over “torture” is that the experience of torture is qualitatively different form the experience of receiving a dust speck in the eye and so the two should not be compared (assuming that there won’t be other long term negative consequences such as car accidents and surgical mistakes).

A moderate pain doesn’t cause the same reactions in the human organism that happen during an extreme and prolonged distress such as: panic attacks, alterations of physiological functions and psychological trauma with all the capacity impairment that derive from it.

But the most obvious characteristic that distinguishes the pain caused by a torture from the pain caused by a dust speck in the eye is the intolerability: people would rather die than feel 50 years of torture. Some argue that these qualitative differences don’t exist because we can create a sequence of injures each comparable to the previous and the next.

For example:

30 dust specks divided among 30 people≃1 slap.

10 slaps divided among 10 people≃1 punch.

10 punches divided among 10 people≃1 small cut.

10 small cuts divided among 10 people≃1 deep cut.

10 deep cuts divided among 10 people≃1 tonsillectomy without anesthesia.

10 tonsillectomies without anesthesia divided among 10 people≃1 torture.

So it seems that we can’t clearly distinguish what is tolerable from what is intolerable, unless we choose on the spectrum of the grades of pains an arbitrary limit, or rather a distance, that separate two incomparable kinds of injuries. Many have rejected this idea because the line would be subjective and because the difference between things near the limit is small.

However these limits have an important function in our decision making process: in some situations there could be a range of options which are similar among each other, but at same time have differences which accumulate step by step, until they become too relevant to be ignored.

Consider this mind experiment: a man has bought a huge house with many rooms, is favorite color is orange and he wants to paint all the rooms with it. Omega offers its help and gives five options, it will paint:

A) One room with R=255 G=150 B=0 paint

B) Two rooms with R=255 G=120 B=0 paint

C) Four rooms with...
Where does ADT Go Wrong?
by abramdemski — 2017-11-17T23:31:44.000Z

The main success of asymptotic decision theory is that it correctly solves Agent Simulates Predictor, as well as a swath of related problems. This is a challenging problem. Unfortunately, ADT hasn't generalized well to other problems. It seems like perhaps the property which allows it to get ASP right is precisely the property which blocks it from generalizing further.

There are two algorithms in ADT. The first, SADT, is given a decision problem in functional form so that counterfactuals are well-defined. This is called an embedder. SADT has a set of strategies which it can choose from, and it chooses whichever strategy gets the highest expectation when plugged into the embedder. This can be thought of as imitating whichever agent does best on the problem.

The second algorithm, DDT, is for learning the embedder to give SADT. DDT has a set of possible embedders, and chooses optimistically from the embedders which it thinks could be equivalent to the true decision problem it faces. This way, it rules out overly optimistic embedders and settles on the best of the embedders which is equivalent to the true problem.

The main sign of a problem with SADT is that it will choose to crash into itself in a game of chicken. Consider the embedder which works as follows: it inserts the chosen policy into the slot of player 1. Player 2 is a new instance of SADT which looks at the problem via an embedder in which it plays the role of player 2 against player 1's strategy. If player 1 chooses a strategy which always goes straight rather than swerving, then SADT expects player 2 to learn to swerve -- that is what SADT would do if it saw itself up against a go-straight bot. Unfortunately, the real player 2 is thinking the same thing. So, both players go straight, and crash into each other; the worst possible outcome.

In short, SADT thinks that it moves first in games against other SADT agents. Since each copy thinks it moves first, the outcome can be very bad.

The simple analysis of what went wrong is that this game is outside of the optimality conditions for SADT. That's not very interesting, though. Why doesn't SADT work well outside those conditions?

One possible analysis of what goes wrong is that SADT thinks it can get the outcome of any other agent just by copying that agent's strategy. Unlike SADT, LIDT considers what happens if LIDT itself took a specified action. We don't hav

...
Entitlement, Covert Contracts, Social Libertarianism, and Related Concepts
by ozymandias — 2017-11-17T19:00:34.485Z

[Crossposted from my blog with minor changes. I strongly prefer all commentary related to the culture war or dating happen over there.]

One of my most unpopular opinions is that "entitlement" is actually a useful concept.

It makes sense that people have a knee-jerk negative reaction to it. "Entitled" is one of those words, like "rational", that it is useful to have in your mental toolkit but if you ever find yourself using it you need to rephrase your argument. More on this later; it ought to show up later in the essay but I wanted to put in "I know that there are good reasons for you to hate the word 'entitled'" in the forlorn hope that this would keep people from writing angry comments before they read the post.

I think that entitlement is a useful concept when thinking about the interplay between these three concepts:

Basic rights and expectations for any relationship.
Boundaries and needs you have in some specific relationship.
Needs that are completely unreasonable.

There is a very common model of social interaction which you might call "social libertarianism," beloved by poly people, sex-positive people, and jerks who don't want to get called on being jerks. Under social libertarianism, people have sets of needs and boundaries, and the goal of social interaction is to find someone who can fulfill your needs and whose needs you can fulfill, without violating either side's boundaries. The only misdeed is to deliberately violate another person's stated boundaries. No set of needs or boundaries are right or wrong, but some may be incompatible with other people's. If you enjoy screaming curse words at other people, then all you have to do is find someone else who likes having curse words screamed at them, and the problem is solved.

Social libertarianism often comes from a good impulse. Many people generalize from "I would be unhappy in this sort of relationship" to "everyone who is in this sort of relationship is unhappy and being exploited and a victim of Insert Cultural Boogeyman Here." Social libertarianism has built into it the idea that people are different from each other, an insight which is all too often forgotten. If you're the sort of person whose perfectly happy relationships get accused of being unhappy and exploitative and only existing because of I...
Catastrophe Mitigation Using DRL
by Vanessa Kosoy — 2017-11-17T15:38:18.000Z

% operators that are separated from the operand by a space

% autosize deliminaters

% operators that require brackets

% operators that require parentheses

% Paper specific

Previously we derived a regret bound for DRL which assumed the advisor is "locally sane." Such an advisor can only take actions that don't lose any value in the long term. In particular, if the environment contains a latent catastrophe that manifests with a certain rate (such as the possibility of an UFAI), a locally sane advisor has to take the optimal course of action to mitigate it, since every delay yields a positive probability of the catastrophe manifesting and leading to permanent loss of value. This state of affairs is unsatisfactory, since we would like to have performance guarantees for an AI that can mitigate catastrophes that the human operator cannot mitigate on their own. To address this problem, we introduce a new form of DRL where in every hypothetical environment the set of uncorrupted states is divided into "dangerous" (impending catastrophe) and "safe" (catastrophe was mitigated). The advisor is then only required to be locally sane in safe states, whereas in dangerous states certain "leaking" of long-term value is allowed. We derive a regret bound in this setting as a function of the time discount factor, the expected value of catastrophe mitigation time for the optimal policy, and the "value leak" rate (i.e. essentially the rate of catastrophe occurrence). The form of this regret bound implies that in certain asymptotic regimes, the agent attains near-optimal expected utility (and in particular mitigates the catastrophe with probability close to 1), whereas the advisor on its own fails to mitigate the catastrophe with probability close to 1. %Thus, this formalism can be regarded as a simple model of aligned superintelligence: the agent is aligned since near-optimal utility is achieved despite the presence of corrupted states (which are treated more or less the same as before) and it is superintelligent since its performance is vastly better than the performance of the advisor.

Appendix A proves the main theorem. Appendix B contains the proof of an important lemma which is however almost identical to what appeared in the previous essay. Appendix C contains several propositions from the previous es

...
Mapping Another's Universe
by squidious — 2017-11-17T02:37:42.324Z

Let's say you are reading a book about people with magical abilities. As a young child, their abilities manifest spontaneously (accidentally breaking something with their mind, flying instead of falling, etc). Then, they are taken away off somewhere to learn to hone their skills. There may be rules to what magic is and is not possible, how magic is done, and what different students can learn. In this book, the main character has the ability to manipulate electricity. They can use this to control anything that operates with electricity remotely, like by turning lights on and off, or they can just shock people. If it is established in the book that no person has more than one ability, you may be surprised if, later on, the main character starts to manipulate water or read minds. This changes the rules of the story.

In the above paragraph, I painted a picture of a universe, giving you rules about how it worked. If I asked you to write a story within the universe, many of you might take care to make sure the story abides by the rules I have provided. This ability to create a universe in your mind is a skill not everyone has, but it is useful beyond just reading stories and writing fanfiction. This same skill can be used to understand people, and even ourselves, better.

To tie this to another example, let's say you are speaking with someone who you just met. This person is an elementary school teacher. Right away, you know quite a few things about this person (they work with kids, get summers off, probably teach during the day and maybe grade papers and plan lessons after school). In the same way that you did with the book, you can use this information to give more detail to this person's universe. Now maybe the person tells you that they also run a summer camp. If you previously assumed that being a teacher meant they had summers off, now you can update the information you have on this person to include that they have a second job they work over the summer. You now have a slightly more detailed picture of what their universe is like.

Though the example above is a relatively simple one, there are lots of ways to gain information that you can use to add detail to someone's universe. Often times we don't even realize we are doing it; assumptions made based on a person's appearance or social media profile aren't always conscious. The...
The Happy Dance Problem
by abramdemski — 2017-11-17T00:50:03.000Z

Since the invention of logical induction, people have been trying to figure out what logically updateless reasoning could be. This is motivated by the idea that, in the realm of Bayesian uncertainty (IE, empirical uncertainty), updateless decision theory is the simple solution to the problem of reflective consistency. Naturally, we'd like to import this success to logically uncertain decision theory.

At a research retreat during the summer, we realized that updateless decision theory wasn't so easy to define even in the seemingly simple Bayesian case. A possible solution was written up in Conditioning on Conditionals. However, that didn't end up being especially satisfying.

Here, I introduce the happy dance problem, which more clearly illustrates the difficulty in defining updateless reasoning in the Bayesian case. I also outline Scott's current thoughts about the correct way of reasoning about this problem.

(Ideas here are primarily due to Scott.)

The Happy Dance Problem

Suppose an agent has some chance of getting a pile of money. In the case that the agent gets the pile of money, it has a choice: it can either do a happy dance, or not. The agent would rather not do the happy dance, as it is embarrassing.

I'll write "you get a pile of money" as M, and "you do a happy dance" as H.

So, the agent has the following utility function:

U(¬M) = $0

U(M & ¬H) = $1000

U(M & H) = $900

A priori, the agent assigns the following probabilities to events:

P(¬M) = .5

P(M & ¬H) = .1

P(M & H) = .4

IE, the agent expects itself to do the happy dance.

Conditioning on Conditionals

In order to make an updateless decision, we need to condition on the policy of dancing, and on the policy of not dancing. How do we condition on a policy? We could change the problem statement by adding a policy variable and putting in the conditional probabilities of everything given the different policies, but this is just cheating: in order to fill in those conditional probabilities, you need to already know how to condition on a policy. (This simple trick seems to be what kept us from noticing that UDT isn't so easy to define in the Bayesian setting for so long.)

A naive attempt would be to condition on the material conditional representing each policy, M⊃H and M⊃¬H. This gets the wrong answer. The material conditional simply rules out the one outcome inconsistent with the policy.

Cond

...
The Happy Dance Problem
by abramdemski — 2017-11-17T00:47:29.151Z

[Cross-posted from IAFF.]

Since the invention of logical induction, people have been trying to figure out what logically updateless reasoning could be. This is motivated by the idea that, in the realm of Bayesian uncertainty (IE, empirical uncertainty), updateless decision theory is the simple solution to the problem of reflective consistency. Naturally, we’d like to import this success to logically uncertain decision theory.

At a research retreat during the summer, we realized that updateless decision theory wasn’t so easy to define even in the seemingly simple Bayesian case. A possible solution was written up in

Conditioning on Conditionals

. However, that didn’t end up being especially satisfying.

Here, I introduce the happy dance problem, which more clearly illustrates the difficulty in defining updateless reasoning in the Bayesian case. I also outline Scott’s current thoughts about the correct way of reasoning about this problem.

(Ideas here are primarily due to Scott.)

The Happy Dance Problem

Suppose an agent has some chance of getting a pile of money. In the case that the agent gets the pile of money, it has a choice: it can either do a happy dance, or not. The agent would rather not do the happy dance, as it is embarrassing.

I’ll write “you get a pile of money” as M, and “you do a happy dance” as H.

So, the agent has the following utility function:

U(¬M) = $0
U(M & ¬H) = $1000
U(M & H) = $900

A priori, the agent assigns the following probabilities to events:

P(¬M) = .5
P(M & ¬H) = .1
P(M & H) = .4

IE, the agent expects itself to do the happy dance.

Conditioning on Conditionals

In order to make an updateless decision, we need to condition on the policy of dancing, and on the policy of not dancing. How do we condition on a policy? We could change the problem statement by adding a policy variable and putting in the conditional probabilities of everything given the different policies, but this is just cheating: in order to fill in those conditional probabilities, you need to already know how to condition on a policy. (This simple trick seems to be what kept us from noticing that UDT isn’t so easy to define in the Bayesian setting for so long.)

A naive attempt would be to condition on the material conditional representing each policy, M⊃H and M⊃¬H. This gets the wrong answer. The material conditional simply rules out the one outcome inconsistent with the poli...