trentmkelly/LessWrong-43k huggingface dataset:
Oxford Prioritisation Project Review
by [deleted] — 2017-10-13T23:07:15.874Z

Cross-posted to Tom's website.

Short summary

The Oxford Prioritisation Project was a research group between January and May 2017. The team conducted research to allocate £10,000 in the most impactful way, and published all our work on our blog. Tom Sittler was the Project’s director and Jacob L was its secretary, closely supporting Tom. This document is our (Jacob and Tom’s) in-depth review and impact evaluation of the Project. Our main conclusions are that the Project was an exciting and ambitious experiment with a new form of EA volunteer work. Although its impact fell short of our expectations in many areas, we learned an enormous amount and produced useful quantitative models.

Contents

Short summary
Contents
Executive summary
The impact of the Project

What were the goals? How did we perform on them?

1: Publish online documents detailing concrete prioritisation reasoning
2: Prioritisation researchers
3: Training for earn-to-givers
4: Give local groups something to do
5: Local group epistemics
6. The value of information of the Oxford Prioritisation Project

Learning value for Jacob and Tom
What were the costs of the project?

Student time
CEA money and time

Main challenges

Different levels of previous experience

Heterogenous starting points
Continued heterogeneity

Team breakdown

Should there be more similar projects? Lessons for replication

Did the Project achieve positive impact?

Costs and benefits
Tom’s feelings

Things we would advise changing if the project were replicated

Less ambition
Shorter duration
Use a smaller grant if it seems easier
Focus on quantitative models from the beginning
More homogenous team
Smaller team

More general updates about epistemics, teams, and community

The epistemic atmosphere of a group will be more truth-seeking when a large donation is conditional on its performance.
A major risk to the project is people hold on too strongly to their pre-Project views
A large majority of team applicants would be people we know personally.

Executive summary

A number of paths for impact motivated this project, falling roughly into two categories: producing valuable research (both to inform and to inspire) and empowering people (by making them more knowledgeable, by improving the local community…).

We feel that the Project’s impact fell short of our expectations in many areas, especially in empowering people but also in producing research. Yet we are proud of the Project, whic...
Rare Exception or Common Exception
by weft — 2017-10-13T22:02:49.026Z

Proposition: Pointing out that rare exceptions exist is usually a negative derailing tactic. However if you think your conversational partner is annoyingly pointing out a rare exception, double-check in case they think they are pointing out a common exception.
------
Human space is deep and wide, and there is a lot more variation in humanity than we really give credit to. This means that for almost any generalization you can make about people, there exists exceptions to the rule.

One rhetorical trick people sometimes use is to point out rare exceptions when the discussion is focusing on generalizations. For example, let's say Alice argues that five year olds shouldn't be allowed to vote because they aren't mature enough or knowledgeable enough. Bob replies "Some five year olds are more mature and knowledgeable than some thirty year olds."

Well... it's strictly true, in that there are some five year old prodigies that you could compare against some non-functional adults in a vegetative state. But it isn't really helpful to the conversation, and it tends to be derailing. This is a case where you are searching all of human-space to find your 0.001% exception and then going "AHA!"
However, there are also times when there are many people who fall outside the norm, and pointing out the exceptions is useful. If Clair says they don't use women to stock shelves because women aren't tall enough to reach the top shelf, and Derek points out "Some women are taller than the average man," this seems very different than the previous example. It would be rare to find a five year old that is more mature than a thirty-five year old, but not at all particularly rare to find a woman who is taller than a man (even though on average men are taller).
I think sometimes misunderstandings occur when one person thinks they're pointing out a common exception, and their conversational partner thinks they are pointing out a rare exception. Maybe Bob really thinks that something like 10% of five-year-olds are more knowledgeable than about 10% of adults. If Alice assumes Bob is just pointing out the 0.001% exceptions that always exist in all human things ever, she is going to be more likely to assume Bob is arguing in bad faith, and miss out on finding their true disagreement.
(I am bad at naming things, if anyone has an idea fo...
There's No Fire Alarm for Artificial General Intelligence
by Eliezer Yudkowsky — 2017-10-13T21:38:16.797Z

What is the function of a fire alarm?

One might think that the function of a fire alarm is to provide you with important evidence about a fire existing, allowing you to change your policy accordingly and exit the building.

In the classic experiment by Latane and Darley in 1968, eight groups of three students each were asked to fill out a questionnaire in a room that shortly after began filling up with smoke. Five out of the eight groups didn't react or report the smoke, even as it became dense enough to make them start coughing. Subsequent manipulations showed that a lone student will respond 75% of the time; while a student accompanied by two actors told to feign apathy will respond only 10% of the time. This and other experiments seemed to pin down that what's happening is pluralistic ignorance. We don't want to look panicky by being afraid of what isn't an emergency, so we try to look calm while glancing out of the corners of our eyes to see how others are reacting, but of course they are also trying to look calm.

(I've read a number of replications and variations on this research, and the effect size is blatant. I would not expect this to be one of the results that dies to the replication crisis, and I haven't yet heard about the replication crisis touching it. But we have to put a maybe-not marker on everything now.)

A fire alarm creates common knowledge, in the you-know-I-know sense, that there is a fire; after which it is socially safe to react. When the fire alarm goes off, you know that everyone else knows there is a fire, you know you won't lose face if you proceed to exit the building.

The fire alarm doesn't tell us with certainty that a fire is there. In fact, I can't recall one time in my life when, exiting a building on a fire alarm, there was an actual fire. Really, a fire alarm is weaker evidence of fire than smoke coming from under a door.

But the fire alarm tells us that it's socially okay to react to the fire. It promises us with certainty that we won't be embarrassed if we now proceed to exit in an orderly fashion.

It seems to me that this is one of the cases where people have mistaken beliefs about what they believe, like when somebody loudly endorsing their city's team to win the big game will back down as soon as asked to bet. They haven't consciously distinguished the reward

...
Creating Space to Cultivate Skill
by hamnox — 2017-10-13T15:51:11.448Z

TLDR - Take a day to literally hide all distractions out of sight and attempt every task a second time whether you succeeded or failed

Intro

I want a ritual that does for skill cultivation what Sabbath does for rest. I want a coherent practice, one which systematically promotes sustained motivation and deliberate practice. I want an established pathway and routines to effortfully move potential blockers out of my way.

This is another post in my series of weekly ritual ideas. See also: Learnday

Reasoning

I took the morning to consider what hinders practice and mastery. Why think in terms of removing blockers rather than promoting tools? Well, people vary in what works for them, and their real problems may not fit the mold I expect. The least a system can do is not tie the hands of intelligent agents who want to solve the problems it's built for—that is my personal variant on 'first do no harm'.

Complex skills barely get anywhere for lack of the basics.

You need to start adding challenge from the level you're at.

You quit a course of mastery prematurely.

This might come from lack of diligence in grabbing reliable incremental gains, or frustration with a perceived plateau, or general pessimism and distractibility.

A chaotic environment confuses your attention.

It competes against what you've decided to focus on, and adds unnecessary noise to your performance.

Decision paralysis ensures you never come to a decision about what to focus on at all

Or at least not one you can stick to in difficult moments.

The setup phase stumps you

You can't move on to actual practice.

Untrustworthy feedback.

You need a timely, relevant improvement signal to follow, whether you judge your own progress or take direction from others. Lack of embodiment or mindfulness can block building your own feedback loops.

Your practice reinforces the wrong skill or bad form.

You might misinterpret which parts of competent examples matter, or have insufficient channels of feedback from reality.

You also might learn habits too narrowly if you never test them in other contexts or conditions.

You flinch away from mistakes and "wasted" time.

You will struggle to put in the work to improve if you spend every second resenting that you're not competent already.

Your mindset or self-image conflicts with trying or, worse, with succeeding.

You can't move forward until you've

...
Humans can be assigned any values whatsoever...
by Stuart_Armstrong — 2017-10-13T11:32:57.118Z

Crossposted at LessWrong 2.0.

Humans have no values... nor do any agent. Unless you make strong assumptions about their rationality. And depending on those assumptions, you get humans to have any values.

An agent with no clear preferences

There are three buttons in this world, B(0), B(1), and X, and one agent H.

B(0) and B(1) can be operated by H, while X can be operated by an outside observer. H will initially press button B(0); if ever X is pressed, the agent will switch to pressing B(1). If X is pressed again, the agent will switch back to pressing B(0), and so on. After a large number of turns N, H will shut off. That's the full algorithm for H.

So the question is, what are the values/preferences/rewards of H? There are three natural reward functions that are plausible:

R(0), which is linear in the number of times B(0) is pressed.

R(1), which is linear in the number of times B(1) is pressed.

R(2) = I(E,X)R(0) + I(O,X)R(1), where I(E,X) is the indicator function for X being pressed an even number of times,I(O,X)=1-I(E,X) being the indicator function for X being pressed an odd number of times.

For R(0), we can interpret H as an R(0) maximising agent which X overrides. For R(1), we can interpret H as an R(1) maximising agent which X releases from constraints. And R(2) is the "H is always fully rational" reward. Semantically, these make sense for the various R(i)'s being a true and natural reward, with X="coercive brain surgery" in the first case, X="release H from annoying social obligations" in the second, and X="switch which of R(0) and R(1) gives you pleasure".

But note that there is no semantic implications here, all that we know is H, with its full algorithm. If we wanted to deduce its true reward for the purpose of something like Inverse Reinforcement Learning (IRL), what would it be?

Modelling human (ir)rationality and reward

Now let's talk about the preferences of an actual human. We all know that humans are not always rational (how exactly we know this is a very interesting question that I will be digging into). But even if humans were fully rational, the fact remains that we are physical, and vulnerable to things like coercive brain surgery (and in practice, to a whole host of other more or less manipulative techniques). So there will be the equivalent of "button X" that overrides human preferences. Thus, "not immortal and unchangea...
Humans can be assigned any values whatsoever...
by Stuart_Armstrong — 2017-10-13T11:29:06.065Z

Humans have no values... nor do any agent. Unless you make strong assumptions about their rationality. And depending on those assumptions, you get humans to have any values.

An agent with no clear preferences

There are three buttons in this world, B(0), B(1), and X, and one agent H.

B(0) and B(1) can be operated by H, while X can be operated by an outside observer. H will initially press button B(0); if ever X is pressed, the agent will switch to pressing B(1). If X is pressed again, the agent will switch back to pressing B(0), and so on. After a large number of turns N, H will shut off. That's the full algorithm for H.

So the question is, what are the values/preferences/rewards of H? There are three natural reward functions that are plausible:

R(0), which is linear in the number of times B(0) is pressed.

R(1), which is linear in the number of times B(1) is pressed.

R(2) = I(E,X)R(0) + I(O,X)R(1), where I(E,X) is the indicator function for X being pressed an even number of times,I(O,X)=1-I(E,X) being the indicator function for X being pressed an odd number of times.

For R(0), we can interpret H as an R(0) maximising agent which X overrides. For R(1), we can interpret H as an R(1) maximising agent which X releases from constraints. And R(2) is the "H is always fully rational" reward. Semantically, these make sense for the various R(i)'s being a true and natural reward, with X="coercive brain surgery" in the first case, X="release H from annoying social obligations" in the second, and X="switch which of R(0) and R(1) gives you pleasure".

But note that there is no semantic implications here, all that we know is H, with its full algorithm. If we wanted to deduce its true reward for the purpose of something like Inverse Reinforcement Learning (IRL), what would it be?

Modelling human (ir)rationality and reward

Now let's talk about the preferences of an actual human. We all know that humans are not always rational (how exactly we know this is a very interesting question that I will be digging into). But even if humans were fully rational, the fact remains that we are physical, and vulnerable to things like coercive brain surgery (and in practice, to a whole host of other more or less manipulative techniques). So there will be the equivalent of "button X" that overrides human preferences. Thus, "not immortal

...
SSC Meetup: Bay Area 10/14
by Scott Alexander — 2017-10-13T03:30:00.269Z

WHEN: 3 PM on Saturday, October 14

HOW: We haven’t done well with cafes or other more traditional meetup spaces in the past, so we’ll probably just meet outside and sit on the grass. Bring blankets / refreshments iff you want them.

WHERE: Berkeley campus, meet at the open space beside the intersection of West and Free Speech. Please disregard any kabbalistic implications of the meetup cross-streets.

WHO: Special guest Scott Aaronson from Shtetl-Optimized. Also me, Katja Grace, possibly David Friedman, hopefully other people.

WHY: Because Professor Aaronson will be giving a lecture on Black Holes, Firewalls, And The Limits Of Quantum Computers at Berkeley at 5 PM (to which you’re all invited) and kindly agreed to hang out with him beforehand.

See you there!
LesserWrong is now de facto the main site
by Chris_Leong — 2017-10-13T01:19:35.386Z

My understanding was that the idea was for LesserWrong to run as a beta with a few users switching over and the main site still running until enough bugs had been fixed and enough missing features had been added so that we could have a vote and then decide whether to replace the old LessWrong.

What has actually happened is that all of the attention and worthwhile content has shifted over too LesserWrong, meaning that nothing is happening on LessWrong. This creates something of an issues:

If we encourage all the people remaining on LessWrong to shift to LesserWrong right now, then they will find a site that still has many bugs and which is missing many core features.

On the other hand, the status quo has costs as well. Anyone visiting LessWrong will find a forum that is not very lively and which doesn't contain particularly good content (because those in the know are posting to LesserWrong). In other words, all of the issues with lack of content and engagement on old LessWrong have now greatly increased.

I don't have any proposals. I simply wrote this to bring this issue to people's attention.

I have confidence in the new site. I think that enough people have switched over to LesserWrong and started posting quality content that we now have critical mass. So this isn't a disaster, but I thought it was important for someone to explicitly say what I'm sure a lot of other people are thinking.
Instrumental Rationality 4.3: Breaking Habits and Conclusion
by  — 2017-10-12T23:11:18.127Z

[Instrumental Rationality Sequence 4.3/7]

[Part three of a three-part series of habits.]

[We go over three techniques for creating habits: Going Upstream and Substitution. Then we conclude the mini-sequence on habits]

Techniques: Breaking Habits :

While creating new habits focuses on reinforcing the link between the Trigger and the Action, breaking habits is about finding ways to disrupt the typical context cue and response mechanism.

Thus, the techniques below sorta do the opposite of what the stuff in Creating Habits did. For example, by weakening the link between the context cue and the response, we can disable the automaticity. Or, we might substitute it with something more desirable.

The two techniques we’ll go over are Going Upstream and Substitution.

IMAGE: https://cdn-images-1.medium.com/max/1600/1*NtngNsQgX035it-vHTY0Jw.png

++++

Going Upstream:

[Going Upstream is a set of concepts based around removing the context cues of unwanted habits beforehand so the habit doesn’t activate. It’s backed by experimental evidence, and it fits right in with our standard habit model.]

Research:

The idea behind Going Upstream is that one of the best ways to disrupt a habit is to go straight up to the top.

By that, I mean you’re targeting the source of the phenomenon, i.e. whatever’s causing it at the very top of the chain *33. Changes upstream should have effects that flow through to the bottom. It’s like how building a dam upstream of a river causes the water flowing down to slow to a trickle. Hence the name.

But this is probably still a little abstract. Let’s get a bit more specific:

We know from the standard habit model that habits fire in the presence of certain context cues. And many of these cues are in the environment.

Thus, one way to remove an unwanted habit through Going Upstream is to limit your exposure to the aforementioned cue. If you don’t encounter the cue, then the habit won’t fire at all.

For example, say you have an unwanted habit of going into a long bout of distracted browsing after opening your Facebook news feed. One way to make this habit less prevalent by Going Upstream would be to disable your Facebook news feed, removing any chance that you’d get distracted in the first place.

Going Upstream is functionally very similar to the idea of precommitment, the idea of cutting off some of your options ahead of time to make sure you can stick to yo

...
Alan Kay - Programming and Scaling
by namespace — 2017-10-12T14:46:19.979Z
Link: https://www.youtube.com/watch?v=YyIQKBzIuBY

(Republishing the comment linked by Said Achmiz below.)

What Is This Talk About

This talk is about several things.

Software ‘Engineering’ - For the amount of functionality we get out of systems, we seem to have to write a lot of code. Alan Kay tries to give the viewer a sense of the vast scale of how much code we’re writing, and then asks the open question of whether that’s how much code we really need to write to get that much functionality. There’s good evidence it isn’t, including both research Kay has done at the Viewpoints Research Institute, along with a biological metaphor showing how the machines we build don’t have the necessary capability to let us easily modify them without understanding the details of their every inner component.

Epistemology - This is the talk where Kay gives his famous ‘perspective is worth 80 IQ points’ quip. Kay discusses how the sheer genius of someone like Leonardo merely allowed him to dream about vehicles he couldn’t build without a working engine. He then goes into a short analysis of how many new useful resources exist in software when it comes to outlook, versus the scant progress that has been made in utilizing them. He compares the situation with computer science to that of music, where only a tiny fraction of a fraction of all possible musical theory is actually used to produce popular music.

Complexity versus Complication - Kay theorizes that there is a complexity curve and a complication curve, and that the curve for complication (i.e, inefficient modeling) grows much faster than the curve for complexity. To the point where the right abstraction is the difference between humans being able and unable to manage a system.

Tinkering versus Mathematics versus Science versus Engineering - Kay provides a taxonomy of human artifact production, which distinguishes between several categories of endeavor that are not usually well defined. For example, he points out that mathematics had to be invented, and before it people used giant books of known tested values and binary search to build large structures.

T-Shirt Science - Kay provides a notion that science is about taking a complex idea and boiling it down to its most basic components, so that the ideas which can produce a bridge fit on a t-shirt. He provides the first Lisp interpreter as an example in this class, and hypothesizes that more work in this vein is necessary for Compu

...
Signal seeding
by KatjaGrace — 2017-10-12T07:00:00.245Z

What does it say about a person if they never get up before noon?

If they are the first person to exist, it probably says that morning was for some reason a convenient time to sleep.

If they live in modern society, it might say that they are lazy and weak willed.

How did getting up at one time or another come to signal laziness? You still have to get up once per day.

One story I can imagine is that originally there was some weaker reason to get up early. For instance if your work benefited by sunlight, you could get a bit more in. And then since that was a reasonable thing to do, people who didn’t do it looked like they were less good at getting up. Which made getting up early even better thing to do, so that everyone knows that you can.

And then people who had been on the fence before about whether to bother getting up early started to find it worth their while. Making the remaining noon-sleepers even more weak willed on average. And so it continues, until sleeping until the afternoon strongly suggests laziness.

In general, if an action is a tiny bit good, not doing it can look a tiny bit bad (or stupid, or lazy, or incapable). Which makes it better to do, which makes it look worse to not do it, and so on. And maybe in the end the speck of good that started this disappears, but the value of sending the signal if you can is enough that the equilibrium is stable.

Does this actually happen?