trentmkelly/LessWrong-43k huggingface dataset:
Desiderata for Normalizers
by IAFF-User-4 — 2015-12-03T20:45:28.000Z

This work originated at MIRI Summer Fellows and originally involved Pasha Kamyshev, Dan Keys, Johnathan Lee, Anna Salamon, Girish Sastry, and Zachary Vance. I was asked to look over two drafts and some notes, clean them up, and post here. Especial thanks to Zak and Pasha for drafts on which this was based.

We discuss the issues with expected utility maximizers, posit the possibility of normalizers, and list some desiderata for normalizers.

This post, which explains background and desiderata, is a companion post to Three Alternatives to Utility Maximizers.
The other post surveys some other "-izers" that came out of the MSFP session and gives a sketch of the math behind each while showing how they fulfill the desiderata.

##Background
The naive implementation of an expected utility maximizer involves looking at every possible action - of which there is generally an intractable number - and, at each, evaluating a black box utility function. Even if we could somehow implement such an agent (say, through access to a halting oracle), it would most likely tend towards extreme solutions. Given a function that we would interpret as "maximize paperclips," such an agent would, if possible, convert its entire future light cone into the cheapest object that satisfies whatever its computational definition of paperclip is.

This makes errors in goal specification extremely costly.

Given a utility function which is naively acceptable, the agent will do something which by our standards is completely insane. Even in the paperclip example, the "paperclips" that the agent produces are unlikely to be labeled as paperclips by a human.

If a human wanted to maximize paperclips, they would not, in general, attempt to convert their entire future light cone into paperclips. They might fail to manufacture very many paperclips, but their actions will seem much more “normal” to us than that of the true expected utility maximizer above.

##Normalizers and Desiderata

We consider a normalizer to be an agent whose actions, given flawed utility function, are still considered sane or normal by a human observer. Specifically, we may wish that it should never come up with a solution that a human would find extreme even after the normalizer explained it.

This definition is extremely vague, so we propose some desiderata for normalizers, organized in rough order of how confident we are that each is necessa

...
Making My Peace with Belief
by OrphanWilde — 2015-12-03T20:36:05.673Z

I grew up in an atheistic household.

Almost needless to say, I was relatively hostile towards religion for most of my early life. A few things changed that.

First, the apology of a pastor. A friend of mine was proselytizing at me, and apparently discussed it with his pastor; the pastor apologized to my parents, and explained to my friend he shouldn't be trying to convert people. My friend apologized to me after considering the matter. We stayed friends for a little while afterwards, although I left that school, and we lost contact.

I think that was around the time that I realized that religion is, in addition to being a belief system, a way of life, and not necessarily a bad one.

The next was actually South Park's Mormonism episode, which pointed out that a belief system could be desirable on the merits of the way of life it represented, even if the beliefs themselves are stupid. This tied into Douglas Adam's comment on Feng Shui, that "...if you disregard for a moment the explanation that's actually offered for it, it may be there is something interesting going on" - which is to say, the explanation for the belief is not necessarily the -reason- for the belief, and that stupid beliefs may actually have something useful to offer - which then requires us to ask whether the beliefs are, in fact, stupid.

Which is to say, beliefs may be epistemically irrational while being instrumentally rational.

The next peace I made with belief actually came from quantum physics, and reading about how there were several disparate and apparently contradictory mathematical systems, which all predicted the same thing. It later transpired that they could all be generalized into the same mathematical system, but I hadn't read that far before the isomorphic nature of truth occurred to me; you can have multiple contradictory interpretations of the same evidence that all predict the same thing.

Up to this point, however, I still regarded beliefs as irrational, at least on an epistemological basis.

The next peace came from experiences living in a house that would have convinced most people that ghosts are real, which I have previously written about here. I think there are probably good explanations for every individual experience even if I don't know them, but am still somewhat flummoxed by the fact that almost all the bizarre experiences of my life a...
Meetup : Tel Aviv Game Night
by Squark — 2015-12-03T19:38:44.622Z

Discussion article for the meetup : Tel Aviv Game Night

WHEN:
08 December 2015 07:00:00PM (+0200)

WHERE:
Yigal Alon 118, Tel Aviv

We will play board games and have fascinating discussions, as always. Bring your own board games!

If you have trouble finding the place or have other questions, call me (Vadim) at +972542600919.

Discussion article for the meetup : Tel Aviv Game Night
How do I suggest someone for the Rationality Blogs sidebar?
by CronoDAS — 2015-12-03T16:54:04.711Z

I've been seeing a lot of good content coming out of certain "rationality-adjacent" blogs recently and I think that there's a certain someone who ought to be listed in the Rationality Blogs sidebar but I don't think they are. (I'm not mentioning them by name because I'm more interested in a general procedure at this point - I can always write link posts to relevant blog posts as they appear.) Should I just post to Discussion and say "Hey! Link Here!" and hope a site admin takes the hint?
Dark Arts: Defense in Reputational Warfare
by OrphanWilde — 2015-12-03T15:03:13.013Z

First, the Dark Arts are, as the name implies, an art, not a science. Likewise, defending against them is. An artful attacker can utilize expected defenses against you; if you can be anticipated, you can be defeated. The rules, therefore, are guidelines. I'm going to stage the rules in a narrative form; they don't need to be, however, because life doesn't follow a narrative. The narrative exists to give them context, to give the reader a sense of the purpose of each rule.

Rule #0: Never follow the rules if they would result in a worse outcome.

Now, generally, the best defense is to never get attacked in the first place. Security through obscurity is your first line of defense. Translations of Sun Tzu vary somewhat, but your ideal form is to be formless, by which I mean, do not be a single point of attack, or defense. If there's a mob in your vicinity, the ideal place is neither outside it, nor leading it, but a faceless stranger among it. Even better is to be nowhere near a mob. This is the fundamental basis of not being targeted; the other two rules derive from this one.

Rule #1: Do not stand out.

Sometimes you're picked out. There's a balancing art with this next piece; you don't want to stand out, to be a point of attack, but if somebody is picking faces, you want to look slightly more dangerous than your neighbor, you want to look like a hard target. (But not when somebody is looking for hard targets. Obviously.)

Rule #2: Look like an unattractive target.

The third aspect of this is somewhat simpler, and I'll borrow the phrasing from HPMoR:

Rule #3: "I will not go around provoking strong, vicious enemies" - http://hpmor.com/chapter/19

The first triplet of rules, by and large, are about -not- being attacked in the first place. These are starting points; Rule #1, for example, culminates in not existing at all. You can't attack what doesn't exist. Rule #1 is the fundamental strategy of Anonymous. Rule #2 is about encouraging potential attackers to look elsewhere; Rule #1 is passive, and this is the passive-aggressive form of Rule #1. It's the fundamental strategy of home security - why else do you think security companies put signs in the yard saying the house is protected? Rule #3 is obvious. Don't make enem...
New Leverhulme Centre on the Future of AI (developed at CSER with spokes led by Bostrom, Russell, Shanahan)
by Sean_o_h — 2015-12-03T10:07:15.102Z

[Cross-posted at EA forum]

Hot on the heels of 80K's excellent AI risk research career profile (https://80000hours.org/career-guide/top-careers/profiles/artificial-intelligence-risk-research/), we're delighted to announce the funding of a new international Leverhulme Centre for the Future of Intelligence, to be led by Cambridge, with spokes at Oxford (Nick Bostrom), Imperial (Murray Shanahan), and Berkeley (Stuart Russell). The Centre proposal was developed by us at CSER, but will be a stand-alone centre, albeit collaborating extensively at CSER.

Building on the by-now-familiar "Puerto Rico Agenda", it will have the long-term safe and beneficial development of AI at its core, but with a slightly broader remit than CSER's focus on catastrophic AI risk and superintelligence. For example, it will consider some near-term challenges such as lethal autonomous weapons, and as well as some of the longer-term philosophical and practical issues surrounding the opportunities and challenges we expect to face, should greater-than-human-level intelligence be developed later this century.
It builds on the pioneering work of FHI, FLI and others, and the generous support of Elon Musk in massively boosting this field with his (separate) $10M grants programme in January of this year. One of the most important things this Centre will achieve is in taking a big step towards making this global area of research a long-term one in which the best talents can be expected to have lasting careers - the Centre is funded for a full 10 years, and we will aim to build longer-lasting funding on top of this.

In practical terms, it means that ~10 new postdoc positions at a minimum will be opening up in this space (we're currently pursuing matched funding opportunities) across academic disciplines and locations (Cambridge, Oxford, Berkeley, Imperial and elsewhere). Our first priority will be to identify and hire a world-class Executive Director, who would start in October. This will be a very influential position over the coming years. Research positions will most likely begin in April 2017.

In between now and then, FHI is hiring for AI safety researchers, and CSER will be hiring for an AI policy postdoc in the spring. I'll have limited time to post in between now and the Christmas break (I'll be away at NIPS and then occupied with funder deadlines and CSER recruitment), but will be happy to post ...
Desiderata for Normalizers
by IAFF-User-4 — 2015-12-03T07:51:19.000Z

This work originated at MIRI Summer Fellows and originally involved Pasha Kamyshev, Dan Keys, Johnathan Lee, Anna Salamon, Girish Sastry, and Zachary Vance. I was asked to look over two drafts and some notes, clean them up, and post here. Especial thanks to Zak and Pasha for drafts on which this was based.

We discuss the issues with expected utility maximizers, posit the possibility of normalizers, and list some desiderata for normalizers.

This post, which explains background and desiderata, is a companion post to Three Alternatives to Utility Maximizers.
The other post surveys some other "-izers" that came out of the MSFP session and gives a sketch of the math behind each while going through the relevant intuitions.

##Background
The naive implementation of an expected utility maximizer involves looking at every possible action - of which there is generally an intractably number - and, at each, evaluating a black box utility function. Even if we could somehow implement such an agent (say, through access to a halting oracle), it would tend towards extreme solutions. Given a function like "maximize paperclips," such an agent would convert its entire future light cone into the cheapest object that satisfies whatever its computational definition of paperclip is.

This makes errors in goal specification extremely costly.

Given a utility function which is naively acceptable, the agent will do something which by our standards is completely insane[^fn-smiley]. Even in the paperclip example, the "paperclips" that the agent produces are unlikely to be labeled as paperclips by a human.

If a human wanted to maximize paperclips, they would not, in general, attempt to convert their entire future light cone paperclips. They might fail to manufacture very many paper clips, but their actions will seem much more “normal” to us than that of the true expected utility maximizer above, and we would expect a poor goal specification to be less likely to kill us.

##Normalizers and Desiderata

We consider a normalizer to be an agent whose actions, given flawed utility function, are still considered sane or normal by a human observer. This definition is extremely vague, so we propose two desiderata for normalizers. Note that these desiderata may not be simultaneously satisfiable; furthermore, while we would like to claim that these are necessary, we do not think that they are sufficient for

...
Meetup : MIT/Boston Secular Solstice
by jimrandomh — 2015-12-03T01:14:02.376Z

Discussion article for the meetup : MIT/Boston Secular Solstice

WHEN:
11 December 2015 08:00:00PM (-0500)

WHERE:
MIT Chapel 50 Massachusetts Ave, Cambridge, Massachusetts 02139

It has become tradition, in the community of those who seek to become more rational, to gather for one night of each year, and sing. We do this close to the winter solstice, which is the longest, darkest night of the year; and, gathered as a community, we stare into and confront the darkness. This consists of participatory singing and a few short speeches, following an emotional arc from light to darkness and back to light again. It will last about two hours, starting at 8pm at MIT Chapel at 50 Massachusetts Ave, Cambridge, Massachusetts 02139 and be followed by a reception/afterparty nearby in room 1-132. We may also be organizing an optional pre-ritual potluck nearby, details TBD.

The Facebook event page is at https://www.facebook.com/events/505931562916689/ .

Discussion article for the meetup : MIT/Boston Secular Solstice
Friendship is utilitarian
by KatjaGrace — 2015-12-03T01:00:29.000Z

Suppose you are a perfect utilitarian of some kind. You also live in a world where the utilities associated with your actions are well worked out. You can always spend $1 to save a life, which is worth one million utilons. You can also spend $1 on a cup of tea, which is worth three utilons, and will also make you a little better at working all day, producing $30 of extra value. So tea is a slam dunk.

However you also have a friend who would like to drink tea, and it will also make her $30 more productive. However your friend is not a utilitarian, but rather some kind of personal leisure satisficer, who will spend her dollars on magazines. You have an opportunity to buy tea for your friend, when she cannot buy it for herself. Do you do so?

Yes. Unless it is a very one-shot friendship, she will likely remember your generosity and buy you tea another time, perhaps when she is the one with the unique tea buying opportunity. So it is roughly as good to buy tea for her as it is to buy tea for yourself. It doesn’t matter whether your friend’s resources are to be ‘wasted’ on goods that you don’t care about, or whether her pleasure in a cup of tea can compare to the value of the human life foregone to buy it. It just matters whether the two of you can cooperate for mutual gain.

I think people often feel that having friends who are worth more than a thousand distant strangers is a departure from some ideal utilitarianism. Or that it is only ok to have such friends if they share similar values, and will spend their own resources in the way you would. I claim highly valued friends are actually a natural consequence of utilitarianism.

There is a difference in how much you care about your friends terminally versus instrumentally. On utilitarian grounds you would care about them terminally as much as any stranger, and look after them instrumentally so that they look after you. However, caring about each other terminally seems like a pretty good arrangement instrumentally. If two people with different goals, who are cooperating instrumentally, can exchange this situation for one where they both know that they truly share a compromised set of goals, this is a win because they can now trust each other. Also, humans care about being cared about, so trading real caring produces value in this way too, without any real downside. So arguably any rea...
Rationality Reading Group: Part O: Lawful Truth
by Gram_Stone — 2015-12-02T23:30:24.524Z

This is part of a semi-monthly reading group on Eliezer Yudkowsky's ebook, Rationality: From AI to Zombies. For more information about the group, see the announcement post.

Welcome to the Rationality reading group. This fortnight we discuss The World: An Introduction (pp. 834-839) and Part O: Lawful Truth (pp. 843-883). This post summarizes each article of the sequence, linking to the original LessWrong post where available.

O. Lawful Truth

The World: An Introduction

181. Universal Fire - You can't change just one thing in the world and expect the rest to continue working as before.

182. Universal Law - In our everyday lives, we are accustomed to rules with exceptions, but the basic laws of the universe apply everywhere without exception. Apparent violations exist only in our models, not in reality.

183. Is Reality Ugly? - There are three reasons why a world governed by math can still seem messy. First, we may not actually know the math. Secondly, even if we do know all of the math, we may not have enough computing power to do the full calculation. And finally, even if we did know all the math, and we could compute it, we still don't know where in the mathematical system we are living.

184. Beautiful Probability - Bayesians expect probability theory, and rationality itself, to be math. Self-consistent, neat, even beautiful. This is why Bayesians think that Cox's theorems are so important.

185. Outside the Laboratory - Those who understand the map/territory distinction will integrate their knowledge, as they see the evidence that reality is a single unified process.

186. The Second Law of Thermodynamics, and Engines of Cognition - To form accurate beliefs about something, you really do have to observe it. It's a very physical, very real process: any rational mind does "work" in the thermodynamic sense, not just the sense of mental effort. Engines of cognition are not so different from heat engines, though they manipulate entropy in a more subtle form than burning gasoline. So unless you can tell me which specific step in your argument violates the laws of physics by giving you true knowledge of the unseen, don't expect me to believe that a big, elaborate clever argument can do it either.

187. Perpetual Motion Beliefs -&n...
Meetup : Washington, D.C.: Create & Complete
by RobinZ — 2015-12-02T21:06:56.146Z

Discussion article for the meetup : Washington, D.C.: Create & Complete

WHEN:
06 December 2015 03:00:00PM (-0500)

WHERE:
Reynolds Center

x-posted from list. Gathering in courtyard from 3:00pm, hard start 3:30pm - until closing (7:00 pm).

This week's meeting is a Schelling point for people to work on things. Please bring projects you'd like to work on, help people with theirs, or just come hang out.

Upcoming meetups:

Dec. 13: Sharing Book Excerpts

Dec. 20: Fun & Games

Dec. 27: Holidays & Socialization

Discussion article for the meetup : Washington, D.C.: Create & Complete
Help with understanding some non-standard-LW philosophy viewpoints
by iarwain1 — 2015-12-02T15:54:12.135Z

For a while now I've been trying hard to understand philosophical viewpoints that defer from mine. Somewhere along the line I've picked up or developed a lot of the LW-typical viewpoints (not sure if this was because of LW, or if I developed them earlier and that's what later attracted me to LW), but I know there are a lot of smart people out there who disagree with those viewpoints. I've tried to read articles and books on this, but they either don't address what I'm looking for somehow, or they're so technical that I have a hard time following them. I've also talked at some length with a philosophy professor, but our conversations often seem to end with me still being confused and the professor being confused about what it is I might be confused about.

I'm thinking maybe it'll help to get some input from people who do intuitively agree with my viewpoints, hence this post. So, can someone please tell me what the central arguments or motivations are for promoting the following:

Epistemology:

Trusting philosophical intuitions and/or the way people use words to the point of making strong metaphysical claims about the world, despite the findings of cognitive science / evolutionary psychology / experimental philosophy / etc. that there doesn't seem to be any good reason to trust those intuitions / ways of talking

Not looking at the world in a probabilistic way

Using personal preference or personal intuitions as priors instead of some objective measure along the lines of Solomonoff Induction

Ontology / philosophy of mind:

Moral realism

Mathematical Platonism

Libertarian free will (I'm looking for arguments other than those from religion)

The view that there actually exist abstract "tables" and "chairs" and not just particles arranged into those forms

The existence of non-physical minds (I'm looking for arguments other than the argument from the Hard Problem of Consciousness)

I suspect I'm having trouble with the ontology issues because of my trouble understanding the epistemology issues. Specifically, I keep getting the impression that most (all?) of the arguments for the ontology issues boil down to trusting philosophical intuitions and/or the way people use words. Something along the following lines:

I intuitively feel that there really are objective morals (or: objective mathematics, actual free will, tables and chairs, minds).

Therefore, there really are ob

...
Rationality Quotes Thread December 2015
by elharo — 2015-12-02T11:28:55.845Z

Another month, another rationality quotes thread. The rules are:

Provide sufficient information (URL, title, date, page number, etc.) to enable a reader to find the place where you read the quote, or its original source if available. Do not quote with only a name.

Please post all quotes separately, so that they can be upvoted or downvoted separately. (If they are strongly related, reply to your own comments. If strongly ordered, then go ahead and post them together.)

Do not quote yourself.

Do not quote from Less Wrong itself, HPMoR, Eliezer Yudkowsky, or Robin Hanson. If you'd like to revive an old quote from one of those sources, please do so here.

No more than 5 quotes per person per monthly thread, please.