trentmkelly/LessWrong-43k huggingface dataset:
Bias and Naturalism: a Challenge
by BobTheBob — 2011-05-07T15:30:09.593Z

There are two theses which I think many LWers find attractive, but which on the face of it are at odds. This challenge is to find a way to reconcile them. Bluntly and a bit inaccurately:

You can't trust your untutored native cognitive endowment to make rational (or moral) judgements.

All knowledge - including of what's rational (moral)- is scientific. To learn what's rational (moral) our only option is to study our native cognitive endowments.

In more ponderous detail:
Point 1) It's taken for granted on LW -and I have no problem with this- that without effort to correct ourselves, humans systematically make irrational judgements. This was always obvious, but research of the last few decades, which this blog usefully advertises, exposes this quite starkly (Kahnemann and Tversky et. al.) I think it's equally likely that the moral judgements of an average unreflective person who has not benefited from any moral education will likely fall short of what someone who has been morally educated would judge (moral education makes us less cruel).
Point 2) Some LW contributors subscribe to naturalism. One way of understanding this idea is that all knowledge is scientific knowledge -I mean, knowledge of facts about the measurable, natural world. In particular, whatever there may be to know about what is rational or moral can be known only through empirical investigation -specifically, investigation of the functioning of Homo sapien's cognitive apparatus, and possibly facts about human evolution and ethology.
The Problem: Point (1) tells us that study of what people actually think and do will not tell us what's rational or moral. Indeed, if we try to figure out, say, how to judge the probability of a heads on a toss of a fair coin given, say, 5 prior tails, merely studying what untutored people are apt to judge will give us a bum steer. But point (2) seems to tell us that's all we are allowed. How do we augment mere cognitive science with other natural sciences, and without inadvertently simply smuggling in our values in the process, to deduce from naturalistic inquiry what's rational or moral? (The point as it pertains specifically to morality is argued eloquently in this review (esp section 3)).
Here's a fantasy to spell out the idea. Imagine that you had a highly accurate computer model of the human cognitive apparatus, and a sufficiently powerful computer t...
Beginning resources for CEV research
by lukeprog — 2011-05-07T05:28:25.641Z

I've been working on metaethics/CEV research for a couple months now (publishing mostly prerequisite material) and figured I'd share some of the sources I've been using.

CEV sources.

Yudkowsky, Metaethics sequence

Yudkowsky, 'Coherent Extrapolated Volition'

Tarleton, 'Coherent extrapolated volition: A meta-level approach to machine ethics'

Motivation. CEV extrapolates human motivations/desires/values/volition. As such, it will help to understand how human motivation works.

Neuroeconomics studies motivation as a driver of action under uncertainty. Start with Neuroeconomics: Decision Making and the Brain (2008) and Foundations of Neuroeconomic Analysis (2010), and see my bibliography here.

Affective neuroscience studies motivation as an emotion. Start with Pleasures of the Brain (2009) and my bibliography here.

Motivation science integrates psychological approaches to studying motivation. Start with The Psychology of Goals (2009), Oxford Handbook of Human Action (2008), and Handbook of Motivation Science (2007).

Extrapolation. Is it plausible to think that some kind of extrapolation of human motivations will converge on a single motivational set? How would extrapolation work, exactly?

Reflective equilibrium. Yudkowsky's proposed extrapolation works analogously to what philosophers call 'reflective equilibrium.' The most thorough work here is the 1996 book by Daniels, and there have been lots of papers, but this genre is only barely relevant for CEV. Basically, an entirely new literature on volition-extrapolation algorithms needs to be created.

Full-information accounts of value and ideal observer theories. This is what philosophers call theories of value that talk about 'what we would want if we were fully informed, etc.' or 'what a perfectly informed agent would want' like CEV does. There's some literature on this, but it's only marginally relevant to CEV. Again, an entirely new literature needs to be written to solve this problem.

Metaethics. Should we use CEV, or something else? What does 'should' mean?

Yudkowsky, Metaethics sequence

An Introduction to Contemporary Metaethics is a good introduction to mainstream metaethics. Unfortunately, nearly all of mainstream metaethics is horribly misguided, but the book will at least give you a good sense of t

...
The 5-Second Level
by Eliezer Yudkowsky — 2011-05-07T04:51:45.004Z

To develop methods of teaching rationality skills, you need to learn to focus on mental events that occur in 5 seconds or less. Most of what you want to teach is directly on this level; the rest consists of chaining together skills on this level.

As our first example, let's take the vital rationalist skill, "Be specific."

Even with people who've had moderate amounts of exposure to Less Wrong, a fair amount of my helping them think effectively often consists of my saying, "Can you give me a specific example of that?" or "Can you be more concrete?"

A couple of formative childhood readings that taught me to be specific:

"What is meant by the word red?"
"It's a color."
"What's a color?"
"Why, it's a quality things have."
"What's a quality?"
"Say, what are you trying to do, anyway?"

You have pushed him into the clouds. If, on the other hand, we habitually go down the abstraction ladder to lower levels of abstraction when we are asked the meaning of a word, we are less likely to get lost in verbal mazes; we will tend to "have our feet on the ground" and know what we are talking about. This habit displays itself in an answer such as this:

"What is meant by the word red?"
"Well, the next time you see some cars stopped at an intersection, look at the traffic light facing them. Also, you might go to the fire department and see how their trucks are painted."

-- S. I. Hayakawa, Language in Thought and Action

and:

"Beware, demon!" he intoned hollowly. "I am not without defenses."
"Oh yeah? Name three."

-- Robert Asprin, Another Fine Myth

And now, no sooner does someone tell me that they want to "facilitate communications between managers and employees" than I say, "Can you give me a concrete example of how you would do that?" Hayakawa taught me to distinguish the concrete and the abstract; and from that small passage in Asprin, I picked up the dreadful personal habit of calling people's bluffs, often using the specific phrase, "Name three."

But the real subject of today's lesson is how to see skills like this on the 5-second level. And now that we have a specific example in hand, we can proceed to try to zoom in o

...
The Power of Agency
by lukeprog — 2011-05-07T01:38:39.042Z

You are not a Bayesian homunculus whose reasoning is 'corrupted' by cognitive biases.

You just are cognitive biases.

You just are attribution substitution heuristics, evolved intuitions, and unconscious learning. These make up the 'elephant' of your mind, and atop them rides a tiny 'deliberative thinking' module that only rarely exerts itself, and almost never according to normatively correct reasoning.

You do not have the robust character you think you have, but instead are blown about by the winds of circumstance.

You do not have much cognitive access to your motivations. You are not Aristotle's 'rational animal.' You are Gazzaniga's rationalizing animal. Most of the time, your unconscious makes a decision, and then you become consciously aware of an intention to act, and then your brain invents a rationalization for the motivations behind your actions.

If an 'agent' is something that makes choices so as to maximize the fulfillment of explicit desires, given explicit beliefs, then few humans are very 'agenty' at all. You may be agenty when you guide a piece of chocolate into your mouth, but you are not very agenty when you navigate the world on a broader scale. On the scale of days or weeks, your actions result from a kludge of evolved mechanisms that are often function-specific and maladapted to your current environment. You are an adaptation-executor, not a fitness-maximizer.

Agency is rare but powerful. Homo economicus is a myth, but imagine what one of them could do if such a thing existed: a real agent with the power to reliably do things it believed would fulfill its desires. It could change its diet, work out each morning, and maximize its health and physical attractiveness. It could learn and practice body language, fashion, salesmanship, seduction, the laws of money, and domain-specific skills and win in every sphere of life without constant defeat by human hangups. It could learn networking and influence and persuasion and have large-scale effects on societies, cultures, and nations.

Even a little bit of agenty-ness will have some lasting historical impact. Think of Benjamin Franklin, Teddy Roosevelt, Bill Clinton, or Tim Ferris. Imagine what you could do if you were just a bit more agenty. That's what training in instrumental rationality is all about: transcending your kludginess to attain a bit mo...
What causes people to believe in conspiracy theories?
by Servant — 2011-05-07T00:06:47.242Z

I'm sorry if this post doesn't seem that high quality, but I do feel this might be the best place to ask. The point of this post is to inspire discussion, hopefully discussion that might be useful for answering certain questions I had.

On another board, I gathered evidence of the existence of "mainstream" conspiracy theories with the goal of figuring out why those conspiracy theories are, well, mainstream. Part of the problem is that, because they're mainstream, many people here may believe in them and may even contest the idea that they are even conspiracy theories. I don't really want to get into arguments over if a conspiracy theory is true or not, so just remember "Politics Is The Mindkiller".

1) JFK was assassinated in a conspiracy. (75% of Americans believe this according to a 2003 Gallup poll.)

2) Martin Luther King, Jr. was assassinated in a conspiracy. (58% of Americans believe this according to a 2008 CNN/Essence poll.)

3) Bush lied about WMDs. (43% of Americans according to a 2005 Pew Survey, only 41% disagreed with this statement, according to a 2005 Pew Survey.)

4) No international consensus on who did 9/11, with 49% of Mexicans, 66% of Egyptians, 40% of Turks, 52% of Jordanians and 55% of Palestinians naming a suspect other than al-Qaeda. This is from a 2008 World Public Opinion poll (graph below).

It's clear that at least some conspiracy theories are treated as mainstream in at least some polities, but other conspiracy theories, like "Americans hoaxed the moon landing" are fringe (only 6% of Americans believe this, according to a 2001 Gallup poll [link here]). In fact, many bloggers, including the economist Robin Hanson, labor under the idea that all conspiracy theories are fringe and wonder why are these individuals so different from the "mainstream". So here's two questions that I would like answered, because these results had been bugging me:

1) There is some sort of method by which an individual can 'filter' out the false and "fringe" conspiracy theory while then selecting a 'true' and "mainstream" conspiracy theory to be accepted. What factors play into an individual's decision-making process to determine what conspiracy theories to accept and what to reject?

2) Is the process of believing in conspiracy theories impacted by some form of rationality? Does Bayesian logic plays a role here as well...do individuals unconsciously rate the likeli...
[SEQ RERUN] Burch's Law
by Tyrrell_McAllister — 2011-05-06T21:15:16.085Z

Today's post, Burch's Law, was originally published on 08 March 2007. A summary (taken from the LW wiki):

Just because your ethics require an action doesn't mean the universe will exempt you from the consequences. Manufactured cars kill an estimated 1.2 million people per year worldwide. (Roughly 2% of the annual planetary death rate.) Not everyone who dies in an automobile accident is someone who decided to drive a car. The tally of casualties includes pedestrians. It includes minor children who had to be pushed screaming into the car on the way to school. And yet we still manufacture automobiles, because, well, we're in a hurry. The point is that the consequences don't change no matter how good the ethical justification sounds.

Discuss the post here (rather than in the comments to the original post).

This post is part of the Rerunning the Sequences series, where we'll be going through Eliezer Yudkowsky's old posts in order so that people who are interested can (re-)read and discuss them. The previous post was Policy Debates Should Not Appear One-Sided, and you can use the sequence_reruns tag or rss feed to follow the rest of the series.

Sequence reruns are a community-driven effort. You can participate by re-reading the sequence post, discussing it here, posting the next day's sequence reruns post, or summarizing forthcoming articles on the wiki. Go here for more details, or to have meta discussions about the Rerunning the Sequences series.
Mutable Levels Are Path-Dependent
by XFrequentist — 2011-05-06T20:47:02.976Z

Cyan and I had a good discussion (several, actually) on our long recent drive. Among diverse topics, he explained something that lead to a confusion-reducing revelation.

On meeting Eliezer, Cyan wished to engage him on Measure Theory. Cyan felt that it might be a useful tool for Eliezer's fAI work. The conversation was interesting (I'll leave it to someone more mathematically competent to summarize), but what I remarked on was how it started.

Cyan began, "Eliezer, I'd say you're at least 3 levels above me, but I thought I could offer some advice on Measure Theory..."

I took this to be (needless) status-lowering behavior, and mentioned this in later conversation. However, it came out that this wasn't so, and there was original thinking behind the number "3".

Cyan's formalization of "levels" involved the creation of useful new concepts. Someone a single level above you can create concepts that you can understand, but could not generate on your own.

Cyan felt that Average Physicists were a level above him, Elite Physicists a level higher, and estimated that Eliezer was a level beyond that. Eliezer's original concept of Levels seems to imply that one's level is biologically determined and immutable, and so a single "EY-level" is probably akin to the highest possible "Cyan-level". I will therefore refer to Mutable (Cyan-levels) and Immutable (EY-levels) Levels to distinguish the two. The former is the attainment of one's greatest potential, the latter describes this potential.

I thought that this was a useful way to think about levels and "leveling up", but not completely right. I didn't think that levels were generalizable. Alice, Bob, and Cheryl might form a natural chain in which each was one Mutable Level above the previous person in the chain, but this chain could be very different with different players. Alice, Zorba, Xeno, Yudkowsky, and Cheryl could form an equally logical chain with each person being one Mutable Level above the previous person, so it doesn't make sense to refer to someone as X levels above you.

Cyan and I eventually agreed that this makes more sense, and had the additional benefit providing a useful way to guide seeking out mentors.

Therefore I make several claims I'd like the group's thoughts on:

A useful way to think about "Levels" is as describing increasing ability to produce concepts of greater explanatory power or insight.

It is

...
Add "Meetups" to top navigation bar?
by saliency — 2011-05-06T20:03:28.747Z

I think the impulse to promote all meetups was a good idea but now adding to much noise to the "PROMOTED" feed. Of the 10 "PROMOTED" feeds 8 are adverts for meetups.

I propose adding Meetups to the navigation bar and only adding them to "PROMOTED" when they are especially significant; for example have a special speaker and might draw attention from people who are out of town.
Intuition and Unconscious Learning
by lukeprog — 2011-05-06T18:47:19.921Z

Part of the sequence: Rationality and Philosophy

We have already examined two sources of our intuitions: the attribute substitution heuristics and our evolved psychology. Today we look at a third source of our intuitions: unconscious learning.

Unconscious learning

The 'learning perspective' on intuition is compatible with the heuristics and biases literature and with evolutionary psychology, but adds a deeper understanding of what is going on 'under the hood.' The learning perspective says that many intuitions rely on representations that reflect the entirety of experiences stored in long-term memory. Such intuitions merely reproduce statistical regularities in long-term memory.1

An example will help explain:

Assume you run into a man at the 20th anniversary party of your high school class graduation. You immediately sense a feeling of dislike. To avoid getting into a conversation, you signal and shout some words to a couple of old friends sitting at a distant table. While you are walking toward them, you try to remember the man’s name, which pops into your mind after some time; and suddenly, you also remember that it was he who always did nasty things to you such as taking your secret letters and showing them to the rest of the class. You applaud the product of your intuition (the immediate feeling) that has helped you to make the right decision (avoiding interaction). Recall of prior experiences was not necessary to make this decision. The decision was solely based on a feeling, which reflected prior knowledge without awareness.2

Learning perspective theorists would suggest that your feeling of dislike - your intuition that you shouldn't talk to the man - came from something like an (unconscious) regularities analysis of your experiences with that man that were stored in long-term memory, and those experiences turned out to be mostly negative. As such, your intuition can make use of rapid parallel processing to draw on the whole sum of experiences in long-term memory, rather than using a slower, sequential-processing judgment algorithm.

It is difficult to track the source of any particular intuition (though we can try3), but there is evidence to suggest that unconscious learning is a common source of our intuitions.

Stock tickers

In a series of experiments,4 researchers asked subjects to watch a series of advertisements. Th...
Why is my sister related to me only 50%?
by TimFreeman — 2011-05-06T18:17:58.381Z

The standard story when dealing with inclusive fitness in evolutionary arguments is that my sibling's life is worth half of mine and my cousin's life is worth a quarter of mine.

But I obviously share more than half my genes with my sister because my parents are not unrelated. My parents must share a lot of ancestors enough generations back that nobody has tracked them, since they resemble each other insofar as they both look human. If I take into account that my parents are both human, I should be related to my sister much more than 50%.

So why do they assume a sibling has half your genes when reasoning about inclusive fitness?

My wife is Chinese, and both of my parents are of European descent. Should I expect my kids to like each other less than I like my sister, because they are less closely related?

This is an intellectual question about evolutionary psychology, not an anxious question about my family relationships. We're all doing fine, don't worry.
Experiment Idea Thread  - Spring 2011
by Psychohistorian — 2011-05-06T18:10:45.329Z

This is an idea that just occurred to me. We have a large community of people who think about scientific problems recreationally, many of whom are in no position to go around investigating them. Hopefully, however, some other community members are in a position to go around investigating them, or know people who are. The idea here is to allow people to propose relatively specific ideas for experiments, which can be upvoted if people think they are wise, and can be commented on and refined by others. Grouping them together in an easily identifiable, organized way in which people can provide approval and suggestions seems like it may actually help advance human knowledge, and with its high sanity waterline and (kind of) diverse group of readers, this community seems like an excellent place to implement this idea.

These should be relatively practical, with an eye towards providing some aspiring grad student or professor with enough of an idea that they could go implement it. You should explain the general field (physics, AI, evolutionary psychology, economics, psychology, etc.) as well as the question the experiment is designed to investigate, in as much detail as you are reasonably capable of.

If this is a popular idea, a new thread can be started every time one of these reaches 500 comments, or quarterly, depending on its popularity. I expect this to provide help for people refining their understanding of various sciences, and if it ever gets turned into even a few good experiments, it will prove immensely worthwhile.

I think it's best to make these distinct from the general discussion thread because they have a very narrow purpose. I'll post an idea or two of my own to get things started. I'd also encourage people to post not only experiment ideas, but criticism and suggestions regarding this thread concept. I'd also suggest that people upvote or downvote this post if they think this is a good or bad idea, to better establish whether future implementations will be worthwhile.
Requesting Advice
by Carinthium — 2011-05-06T10:59:57.705Z

I'm somewhat reluctant to talk about my personal life online nowadays, but suffice to say that I've discovered an area of likely self-delusion and I want to prevent myself from backsliding through forgetting what I've learned. Any ways to make sure the knowledge stays in my head?

(And yes, writing this post will likely help- I'm not stupid)
Climate change: existential risk?
by katydee — 2011-05-06T06:19:21.259Z

What does the community here think when it comes to climate change as a potential existential risk? While strategies for combating climate change are fairly straightforward, the seeming lack of political capital behind meaningful climate reform and legislation seems to indicate that the problem is going to get substantially worse before it gets better, and the potential consequences of ignoring this issue look to be quite severe indeed!

Should the rationality/x-risks community be spending more effort on evaluating this idea and exploring potential solutions? It certainly seems like a big problem, and the current trajectory is quite worrisome. On the other hand, the issue is a political minefield and could risk entangling the community in political squabbling, potentially jeopardizing its ability to act on other threats. What do you guys think?