trentmkelly/LessWrong-43k huggingface dataset:
An Educational Curriculum 
by DragonGod — 2017-11-22T10:11:58.779Z

I'm a CS student graduating next year, my education has been pretty crap, due to personal problems, and my school curriculum being pretty bad (we never met the requisite depth in any of the courses we did, and skimmed over a lot). After graduation, I want to take a few years (I'm thinking 3 - 6) off to do a lot of self study.

I'm not sure how my knowledge level compares to international standards, so just assume no prior CS knowledge (I'll skip things I already know satisfactorily, but I don't expect there to be anything I know deep enough that it would be worth skipping it completely). For mathematics, I am at highschool level (currently learning algebra and logic in my free time) sans calculus (which I never really learned), with a little discrete maths. I have no prior philosophy training, and it I sufficient to assume that the entirety of my philosophy knowledge is from Lesswrong.

I have a (set of) goals I want to achieve, and I want to learn the required computer science (among other things) in order to achieve my goal. I plan on pursuing a postgraduate degree towards that goal after my gap years (I intend to start producing original research in at most ten years, and most likely much earlier than that).

Goals

Foundations of Intelligence

Define "Intelligence".
Develop a model of intelligence.
Develop a method for quantifying and measuring intelligence of arbitrary agents in agent space.
Understand intelligence and what makes certain agent designs produce more intelligent agents.
Develop a hierarchy of intelligent agents over all of agent space.
Answer: "is there a limit to intelligence?"

Formalise learning

Develop a model of learning.
Answer: What does it mean for a learning algorithm to be better than another?
Develop a method for analysing (I'm thinking of asymptotic analysis (at least as of now, all analysis I plan to do would be asymptotic)) (and comparing) the performance of learning algorithms on a particular problem, across a particular problem class, and across problem space using a particular knowledge representation system(KRS), using various KRS, and across the space of possible KRS.
Understand what causes the difference in performance between learning algorithms.
Determine the scope/extent of knowledge a given learning algorithm can learn.
Develop a hierarchy of learning algorithms capturing the enti

...
Catastrophe Mitigation Using DRL
by Vanessa Kosoy — 2017-11-22T05:54:42.000Z

Previously we derived a regret bound for DRL which assumed the advisor is "locally sane." Such an advisor can only take actions that don't lose any value in the long term. In particular, if the environment contains a latent catastrophe that manifests with a certain rate (such as the possibility of an UFAI), a locally sane advisor has to take the optimal course of action to mitigate it, since every delay yields a positive probability of the catastrophe manifesting and leading to permanent loss of value. This state of affairs is unsatisfactory, since we would like to have performance guarantees for an AI that can mitigate catastrophes that the human operator cannot mitigate on their own. To address this problem, we introduce a new form of DRL where in every hypothetical environment the set of uncorrupted states is divided into "dangerous" (impending catastrophe) and "safe" (catastrophe was mitigated). The advisor is then only required to be locally sane in safe states, whereas in dangerous states certain "leaking" of long-term value is allowed. We derive a regret bound in this setting as a function of the time discount factor, the expected value of catastrophe mitigation time for the optimal policy, and the "value leak" rate (i.e. essentially the rate of catastrophe occurrence). The form of this regret bound implies that in certain asymptotic regimes, the agent attains near-optimal expected utility (and in particular mitigates the catastrophe with probability close to 1), whereas the advisor on its own fails to mitigate the catastrophe with probability close to 1.

Appendix A proves the main theorem. Appendix B contains the proof of an important lemma which is however almost identical to what appeared in the previous essay. Appendix C contains several propositions from the previous essay which we are used in the proof. [Appendices B and C were moved to a separate post because of a length limit in the website.]

Results

We start by formalising the concepts of a "catastrophe" and "catastrophe mitigation" in the language of MDPs.

Definition 1

A catastrophe MDP is an MDP M together with a partition of SM into subsets SM:=SFM⊔SDM⊔SCM (safe, dangerous and corrupt states respectively).

Definition 2

Fix a catastrophe MDP M. Define A♯M:SM→2AM by

A♯M(s):=⎧⎪⎨⎪⎩{a∈AM∣suppT(s,a)⊆SFM} if s∈SF

...
Tags or Sub-Groups
by Chris_Leong — 2017-11-21T23:28:52.901Z

There are two main ways of dividing discussion in a group so that people only see what they want to see and both have advantages and disadvantages.

Tags are the most common method used. They have the following advantages, mostly related to flexibility:

Anyone can invent a new tag on the spot and they can organically gain usage
Posts can be given multiple tags. This allows you to filter posts on multiple dimensions (ie. topic and type of post), as well as having both broad and specific tags. In a well designed system, this provides maximal control on what posts you want to filter in/filter out.

Sub-Groups are less common. They have the following advantages, mostly related to the ability to create norms:

As demonstrated by Reddit, having different sub-groups allows you to set rules that work well for the community. For example, Ask Science can have a norm of encouraging peer-reviewed references, Ask Me Anything can delete posts which are abandoned and Change My View can ensure that top-level posts actually try and change your view. While this would be theoretically possible with tags, this is complicated by the fact that a) posts can have multiple tags so it is unclear how these rules would interact b) it is easier to add a tag without ever understanding that there are rules you are supposed to follow, than it is to accidentally post in a group
Even if you can set rules for tags, it is much harder for implicit norms to form for tags than for groups. The fact that a post can have multiple tags really torpedos the formation of implicit norms, as it prevents there from being a clearly defined space
Sub-groups tend to have much more a sense of community, indeed they are explicitly a community, vs. any implicit community that occurs via posting on the same tags
Sub-groups tend to result in less duplication, while there may be five different tags which are all effectively synonyms of each other

My position is that the value of people forming separate sub-groups should not be underestimated. Think, for example, how helpful it would have been in the early days for rationalists interested in Effective Altruism to be able to split into their own group. Much of the discussion, ie. of specific charities, would have been of no relevance to most of the LW community, but of vital importance to that particular community. This brings us to another distinction. A parenting tag, for ex...
Hero Licensing
by Eliezer Yudkowsky — 2017-11-21T21:13:36.019Z

I expect most readers to know me either as MIRI's co-founder and the originator of a number of the early research problems in AI alignment, or as the author of Harry Potter and the Methods of Rationality, a popular work of Harry Potter fanfiction. I’ve described how I apply concepts in Inadequate Equilibria to various decisions in my personal life, and some readers may be wondering how I see these tying in to my AI work and my fiction-writing. And I do think these serve as useful case studies in inadequacy, exploitability, and modesty.

As a supplement to Inadequate Equilibria, then, the following is a dialogue that never took place—largely written in 2014, and revised and posted online in 2017.

i. Outperforming and the outside view

(The year is 2010. eliezer-2010 is sitting in a nonexistent park in Redwood City, California, working on his laptop. A person walks up to him.)

person: Pardon me, but are you Eliezer Yudkowsky?

eliezer-2010: I have that dubious honor.

person: My name is Pat; Pat Modesto. We haven’t met, but I know you from your writing online. What are you doing with your life these days?

eliezer-2010: I’m trying to write a nonfiction book on rationality. The blog posts I wrote on Overcoming Bias—I mean Less Wrong—aren’t very compact or edited, and while they had some impact, it seems like a book on rationality could reach a wider audience and have a greater impact.

pat: Sounds like an interesting project! Do you mind if I peek in on your screen and—

eliezer: (shielding the screen) —Yes, I mind.

pat: Sorry. Um... I did catch a glimpse and that didn’t look like a nonfiction book on rationality to me.

eliezer: Yes, well, work on that book was going very slowly, so I decided to try to write something else in my off hours, just to see if my general writing speed was slowing down to molasses or if it was this particular book that was the problem.

pat: It looked, in fact, like Harry Potter fanfiction. Like, I’m pretty sure I saw the words “Harry” and “Hermione” in configurations not originally written by J. K. Rowling.

eliezer: Yes, and I currently seem to be writing it very quickly. And it doesn’t seem to use up mental energy the way my regular writing does, either.

(A mysterious masked

...
Sorry for being less around, will be back soon
by habryka — 2017-11-21T20:40:14.509Z

Hey everyone,

Some of you might have noticed that the development speed was a bit slower over the last two weeks. I had a bunch of not so great things happen to my family that have caused me a good amount of stress. They've also caused to me fly on fairly short notice back to Germany for two weeks, which is where my family lives.

I will be back from Germany by the middle of next week. Things should pick up again shortly after that. I am really sorry for not being able to develop stuff as fast and fix as many bugs as I would like to in that time. Looking forward to being back soon.
The Archipelago Model of Community Standards
by Raemon — 2017-11-21T03:21:07.679Z

Epistemic Status: My best guess. I don't know if this will work but it seems like the obvious experiment to try more of.

Epistemic Effort: Spent several months thinking casually, 25ish minutes consolidating earlier memories and concerns, and maybe 10ish minutes thinking about potential predictions. See comment.

Building off:

Open Problems in Group Rationality [Conor Moreton]
Archipelago and Atomic Commutarianism [Scott Alexander]

Claim 1 - If you are dissatisfied with the norms/standards in a vaguely defined community, a good first step is to refactor that community into sub-groups with clearly defined goals and leadership.

Claim 2 - People have different goals, and you may be wrong about what norms are important even given a certain goal. So, also consider proactively cooperating with other people forming alternate subgroups out of the same parent group, with the goal of learning from each other.

Refactoring Into Subcommunities

Building groups that accomplish anything is hard. Building groups that prioritize independent thinking to solve novel problems is harder. But when faced with a hard problem, a useful technique is to refactor it into something simpler.

In "Open Problems in Group Rationality", Conor lists several common tensions. I include them here for reference (although any combination of difficult group rationality problems would suffice to motivate this post).

Buy-in and retention.
Defection and discontent.
Safety versus standards.
Productivity versus relevance.
Sovereignty versus cooperation.
Moloch and the problem of distributed moral action.

These problems don't go away when you have clearly defined goals. A corporation with a clearcut mission and strategy (i.e maximize profit by selling widgets) still has to navigate the balance of "hold their employees to a high standards to increase performance" and "make sure employees feel safe enough to do good work without getting wracked with anxiety" (or, just quit).

Such a corporation might make different tradeoffs in different situations - if there's a labor surplus, they might be less worried about employees quitting because they can just find more. If the job involves creative knowledge work, anxiety might have greater costs to productivity. Or maybe they're not just profit-maximizing: maybe the CEO cares about employee mental health for its own sake.

But well define...
Sunset at Noon
by Raemon — 2017-11-21T02:05:00.000Z

A meandering series of vignettes.

I have a sense that I've halfway finished a journey. I expect this essay to be most useful to people similarly-shaped-to-me, who are also undergoing that journey and could use some reassurance that there's an actual destination worth striving for.

Gratitude
Tortoise Skills
Bayesian Wizardry
Noticing Confusion
The World is Literally on Fire...
...also Metaphorically on Fire
Burning Out
Sunset at Noon

Epistemic Status starts out "true story", and gets more (but not excessively) speculative with each section.

i. Gratitude

"Rationalists obviously don't *actually* take ideas seriously. Like, take the Gratitude Journal. This is the one peer-reviewed intervention that *actually increases your subjective well being*, and costs barely anything. And no one I know has even seriously tried it. Do literally *none* of these people care about their own happiness?"

"Huh. Do *you* keep a gratitude journal?"

"Lol. No, obviously."

- Some Guy at the Effective Altruism Summit of 2012

Upon hearing the above, I decided to try gratitude journaling. It took me a couple years and a few approaches to get it working.

First, I tried keeping a straightforward journal, but it felt effortful and dumb.
I tried a thing where I wrote a poem about the things I was grateful for, but my mind kept going into "constructing a poem" mode instead of "experience nice things mindfully" mode.
I tried just being mindful without writing anything down. But I'd forget.
I tried writing gratitude letters to people, but it only occasionally felt right to do so. (This came after someone actually wrote me a handwritten gratitude letter, which felt amazing, but it felt a bit forced when I tried it myself.)
I tried doing gratitude before I ate meals, but I ate "real" meals inconsistently so it didn't take. (Upon reflection, maybe I should have fixed the "not eating real meals" thing?)

But then I stumbled upon something that worked. It was a social habit, which I worry is a bit fragile. I did it together with my girlfriend each night. On nights when one of us travelled, I'd often forget.

But this is the thing that worked. Each night, we share our Grumps and Gratitudes.

Grumps and Gratitudes goes like this:

We share anything we're annoyed or upset about. (We call this The Grump. Our rule is to not go *searching* for the Grump, simply to let it out if it's festering so that when we get to the Gratitude we ac

...
Arbitrary Math Questions
by ryan_b — 2017-11-21T01:18:47.430Z

Purpose: This post is so I can record math-ish questions I have, which I want answers to, but don't merit their own post or pestering the community about directly. My expectation is they will mostly yield to a little research. If you have the answer or a relevant source, please feel free to mention.

Fixed-point theorems: mapping from a particular kind of set, onto a particular function, has a fixed point on that function.

Is there a procedure for enumerating sets that map onto a function in this way, given just the function?
If this function were a utility function, could we look at the sets we generated and test whether they are morphic with some other space we are concerned with: value space? mind space?

How is it exactly we account for things we value which are non-fungible? I am only aware of trying to set a some equivalent by looking at how much we are willing to sacrifice to preserve it, but this fails to capture the dimensionality of the problem. Is my true utility function actually the multiplication of n utility functions, each of one parameter?

Note : I will update this periodically, both to add new questions, mark answered questions, and to break out anything that turns particularly interesting.
The Darwin Pregame
by Zvi — 2017-11-21T01:10:00.372Z

Epistemic Status: True story

This is intended as post two of the sequence Zbybpu’f Nezl.

Previously (required): The Darwin Game

I

This is my reconstruction of my thoughts at the time.

The Darwin Game requires surviving the early, middle and late games.

In the opening, you need to maximize scoring against whatever randomness people submit. Survival probably isn’t enough. The more copies of yourself you bring to the middle game, the more you face yourself, which snowballs. Get as many points as you can.

In the middle game, you face whatever succeeded in the opening. Strategies that survived the opening in bad shape can make a comeback here, if they are better against this new pool. What strategies do well against you matters.

In the end game, you’ll need to beat the successful middle game strategies, all of which have substantial percentages of the pool. Eventually you’ll be heads up against one opponent. Not letting opponents outscore you in a pairing becomes vital.

How would the game play out? What types of strategies would thrive?

I divided the types as follows:

There were attackers, who would attempt to get the opponent to accept a 3/2 or 4/1 split. They might or might not give up on that if you refused, and presumably most would use a signal to self-cooperate, but not all. One person did submit “return 3.”

Then there were cooperators, who attempt to split the pot evenly. I assumed that meant alternating 3/2 splits. This then divided into those who would fold if attacked, allowing you to score above 2.5 per turn, those that would let themselves be outscored but would make sure you scored less than 2.5 per turn, and those that would not allow themselves to be outscored. The last group might or might not forgive an early attempt to attack them.

There would also be bad programs. People do dumb things. Someone might play all 2s, or pick numbers fully at random, or who knows what else.

As a list (attackers from here on means both AttackBot and BullyBot):

AttackBot. Attackers who don’t give up.

BullyBot. Attackers who give up.

CarefulBot. Cooperators who harshly punish attackers.

DefenseBot. Cooperators who don’t let you outscore them but don’t otherwise punish.

EquityBot. Cooperators who let you outscore them, but make sure you don’t benefit.

FoldBot. Cooperators who accept full unfavorable 3/2 splits.

GoofBot. Weird stuff.

My prior was we’d see all seven, with most looking to coop...
Hubris, Pride, and Arrogance
by linkhyrule5 — 2017-11-20T20:32:43.563Z

ITT: link tries to distinguish between three often conflated (and highly correlated) states of mind, because the distinction is Important.

First of all, let me define my terms.

Hubris, I'm using entirely in its original context. The word comes from Ancient Greece; Google defines it as "excessive pride toward or defiance of the gods" - the state of daring to do - or even to attempt - that which should be the exclusive province of divinity. It is the sin of Icarus, who tried to touch the sun and so melted his wax wings; of Prometheus, who stole the heavenly fire from Zeus himself; of Arachne, who was in fact a better weaver than Athena but whose place in the world was not one where she was permitted to try.

I don't really need to spend much time on this one. Even if this post was on a less transhumanistic forum, Eliezer's done an excellent job of it already in Inadequate Equilibria and "Hero Licensing". Suffice it to say that we casually fly on metal wings, that we chain the lightning and use it to watch cat videos, that we make occasional attempts to see just how close to the sun we can get. Insofar as hubris is a useful concept at all, hubris is a virtue.

So is pride. Google-sensei again: "a feeling of deep pleasure or satisfaction derived from one's own achievements." Put more poetically, pride is the emotional result of succeeding at something hubristic - at accomplishing something you "shouldn't have been able" to accomplish. We don't get proud for being able to walk, unless we were once unable. -- In other words, pride is a built-in reward button for doing something we need to do anyway, an internal reward for achieving what we set out to do, and as such one that's much more robust as an incentive than most externally imposed. It's one of the things (sometimes, it seems the few things) that evolution got right in our brains: success is fun, especially when it's only just within our reach - and so we absolutely want "feeling and showing pride justly earned" to be entirely respectable.

I mention these because I want to be very, very clear I'm not talking about either of them.

Where hubris and pride are both focused on yourself, yourself and your ability to do the task before you, arrogance is about others. Arrogance, as I'm defining it, is sneer culture, is ...
Hogwarts House Primaries
by ozymandias — 2017-11-20T17:56:34.504Z

[Edited after publication for clarity.]

I used to get into a lot of really frustrating arguments about normative ethics where the person I was talking to and I were just talking past each other. When I found the

Sorting Hat Chats system

for classifying personality types, I originally just thought of it as being another elaborate fiction-based classification system, which is another one of my guilty pleasures. Sorting Hat Chats is divided into primary and secondary houses. While the secondary houses are just a new gloss on the standard Hogwarts houses, the primary houses are an inexplicably good system for classifying people’s opinions about normative ethics and scrupulosity. Now, instead of going "okay but that thing is wrong, why are you still arguing about it," I go "ah, Ravenclaw primary" and move on. So I thought I would write up my understanding of the system somewhere more permanent than Tumblr.

(Note: Sorting Hat Chats house primaries only vaguely resemble the Hogwarts houses they are named after, and it is best to put aside your preconceptions about the houses when considering this system. Similarly, no knowledge of Harry Potter is required to understand the system. While this is my personal understanding of the system, I do not claim to know what the creators of the Sorting Hat Chats system intended and may very well be misunderstanding their original intent. I think it is fairly unlikely that this system covers literally all possible orientations towards normative ethics, but as-is it is useful enough that I think it's worth sharing.)

Ravenclaw Primary. There is an infallible single-question test for identifying Ravenclaw primaries: “is normative ethics boring and/or completely disconnected from any actual moral reasoning you do in your everyday life?” If your answer is “yes”, you are not a Ravenclaw primary. If your answer is “no”, welcome to Team Ravenclaw.

Ravenclaw primaries believe that you should figure out what the right thing to do is through logic and reason. They often have a particular fondness for moral philosophy and ethical thought experiments. Ravenclaw primaries are particularly likely to identify as utilitarians, Kantians, and virtue ethicists. Other sorts of primaries only rarely identify as these categories unless they have to regularly talk to Ravenclaw primaries. For some reason, Ravenclaw primaries have a...