trentmkelly/LessWrong-43k huggingface dataset:
Actions and Words: Akrasia and the Fruit of Self-Knowledge
by Annoyance — 2009-04-15T15:27:13.693Z

Knowing other people requires intelligence,

but knowing yourself requires wisdom.

Those who overcome others have force,

but those who overcome themselves have power.

- Tao Te Ching, Chapter 33: Without Force, Without Perishing

Much has been written here about the issue of akrasia. People often report that they really, sincerely want to do something, that they recognize that certain courses of action are desirable/undesirable and that they should choose them -- but when the time comes to decide, they do otherwise. Their choices don't match what they said their choices would be.

While I'm sure many people are less than honest in reporting their intentions to others, and possibly even more who aren't even being honest with themselves, there are still plenty of people that are presumably sincere and honest. So how can they make their actions match their understanding of what they want? How can their choices reflect their own best judgment?

Isn't that really the wrong question?

The very powerful and the very stupid have one thing in common: they don't alter their views to fit the facts, they alter the facts to fit their views. Which can be very uncomfortable, if you're one of the facts that needs correcting.

Doctor Who, The Face of Evil

If a model of a phenomenon fails to accurately predict it, we conclude that the model is flawed and try to change it. If what we're trying to understand is ourselves, our own choices, and the motivations, desires, and preferences that direct those choices, why should we do any differently? Our actions reveal what we actually want, not what we believe we want or believe we should want. No one chooses against their own judgment. What we do is choose against our understanding of our own judgment, and that is a far subtler matter. By our fruits shall we know ourselves.

Q: How many therapists does it take to change a lightbulb?

A: The lightbulb has to want to change.

Expecting our behavior to be constrained and controlled by our understanding is like expecting our limbs to move if we yell at them to do so. It doesn't matter how much we believe we want them to move, or how much we say we want them to move. It is irrelevant whether we have a conscious understanding of the nerves and muscles involved. Our conscious awareness is a bystander ...
Bayesians vs. Barbarians
by Eliezer Yudkowsky — 2009-04-14T23:45:48.156Z

Previously:

Let's say we have two groups of soldiers. In group 1, the privates are ignorant of tactics and strategy; only the sergeants know anything about tactics and only the officers know anything about strategy. In group 2, everyone at all levels knows all about tactics and strategy.

Should we expect group 1 to defeat group 2, because group 1 will follow orders, while everyone in group 2 comes up with better ideas than whatever orders they were given?

In this case I have to question how much group 2 really understands about military theory, because it is an elementary proposition that an uncoordinated mob gets slaughtered.

Suppose that a country of rationalists is attacked by a country of Evil Barbarians who know nothing of probability theory or decision theory.

Now there's a certain viewpoint on "rationality" or "rationalism" which would say something like this:

"Obviously, the rationalists will lose. The Barbarians believe in an afterlife where they'll be rewarded for courage; so they'll throw themselves into battle without hesitation or remorse. Thanks to their affective death spirals around their Cause and Great Leader Bob, their warriors will obey orders, and their citizens at home will produce enthusiastically and at full capacity for the war; anyone caught skimming or holding back will be burned at the stake in accordance with Barbarian tradition. They'll believe in each other's goodness and hate the enemy more strongly than any sane person would, binding themselves into a tight group. Meanwhile, the rationalists will realize that there's no conceivable reward to be had from dying in battle; they'll wish that others would fight, but not want to fight themselves. Even if they can find soldiers, their civilians won't be as cooperative: So long as any one sausage almost certainly doesn't lead to the collapse of the war effort, they'll want to keep that sausage for themselves, and so not contribute as much as they could. No matter how refined, elegant, civilized, productive, and nonviolent their culture was to start with, they won't be able to resist the Barbarian invasion; sane discussion is no match for a frothing lunatic armed with a gun. In the end, the Barbarians will win because they want to fight, they want to hurt the rationalists, they want to conquer and their whole society is united a

...
Tell it to someone who doesn't care
by PhilGoetz — 2009-04-14T18:15:18.104Z

Followup to Marketing rationalism

Target fence-sitters

American culture frames issues as debates between two sides. The inefficacy of debates is amazing. You can attend debates on a subject for years without ever seeing anyone change their mind. I think this is because of who attends debates. People listen to debates because they care about the issue. And they only care about the issue because they've already taken a side. Caring then innoculates them to reason.

If the debate really can be approximated by a binary decision, then the people you want to talk to are the fence-sitters. And they aren't there.

This reminded me of my "wound-healing" theory of international aid. I'll float a similar idea for social debate: In order to win society over to a view in the long run, you should target the people who don't care much one way or the other. Politicians already do this.

So how do you get them to listen? They won't come to your debate, or your conference, or your website. Here are some ways:

Fiction. Think Atlas Shrugged. People will read a novel or watch a movie even if they're not interested in the issues that it's about.

The Christian church teaches that evangelism happens through friends. Big church-tent revival meetings have been effective now and then; but most conversions happen one person at a time.

Marketing. There's a much-larger-than-multibillion-dollar industry that does nothing except try to solve the problem of selling things to people who aren't interested in them. Too bad I don't know anything about it.

Teaching kids. (We like to say that a child should grow up to the age where they can make their own decisions. But the unbiased child is a myth.) The public consensus is already to teach science rather than religion in school. I'm happy with that consensus and don't think it needs to be pushed further by teaching rationalist ideology.

(When I combine this theory with the observation that most people don't change their worldview or their preferences much after the age of maybe 15, I come up with the idea that most cultural change is driven by the random drift of the opinions of children.)

Gravitational debate

But there are many instances of inspirational books targeted at people already well on one side of an issue, that ins...
Collective Apathy and the Internet
by Eliezer Yudkowsky — 2009-04-14T00:02:19.161Z

Yesterday I convered the bystander effect, aka bystander apathy: given a fixed problem situation, a group of bystanders is actually less likely to act than a single bystander. The standard explanation for this result is in terms of pluralistic ignorance (if it's not clear whether the situation is an emergency, each person tries to look calm while darting their eyes at the other bystanders, and sees other people looking calm) and diffusion of responsibility (everyone hopes that someone else will be first to act; being part of a crowd diminishes the individual pressure to the point where no one acts).

Which may be a symptom of our hunter-gatherer coordination mechanisms being defeated by modern conditions. You didn't usually form task-forces with strangers back in the ancestral environment; it was mostly people you knew. And in fact, when all the subjects know each other, the bystander effect diminishes.

So I know this is an amazing and revolutionary observation, and I hope that I don't kill any readers outright from shock by saying this: but people seem to have a hard time reacting constructively to problems encountered over the Internet.

Perhaps because our innate coordination instincts are not tuned for:

Being part of a group of strangers. (When all subjects know each other, the bystander effect diminishes.)

Being part of a group of unknown size, of strangers of unknown identity.

Not being in physical contact (or visual contact); not being able to exchange meaningful glances.

Not communicating in real time.

Not being much beholden to each other for other forms of help; not being codependent on the group you're in.

Being shielded from reputational damage, or the fear of reputational damage, by your own apparent anonymity; no one is visibly looking at you, before whom your reputation might suffer from inaction.

Being part of a large collective of other inactives; no one will single out you to blame.

Not hearing a voiced plea for help.

Etcetera. I don't have a brilliant solution to this problem. But it's the sort of thing that I would wish for potential dot-com cofounders to ponder explicitly, rather than wondering how to throw sheep on Facebook. (Yes, I'm looking at you, Hacker News.) There are online activism web apps, but they tend to be along the lines of sign this petition! yay, you signed something! rather t...
GroupThink, Theism ... and the Wiki
by byrnema — 2009-04-13T17:28:58.896Z

In response to the The uniquely awful example of theism, I presented myself as a datapoint of someone in the group who disagrees that theism is uncontroversially irrational.

With a loss of considerable time, several karma points and two bad posts, I now retract my position.

Because I have deconverted? (Sorry, but no.)

I had a working assumption (inferred from here) that rationality meant believing that all beliefs must be rigorously consistent with empirical observation. I now think of this as a weak form of rationalism (see full definition below). A stronger form of rationalism held by (many, most?) rationalists is that there is no other valid source of knowledge. If we define a belief system as religious if and only if it claims knowledge that is independent of empirical experience (i.e., metaphysical) then it is trivially true that all religions are irrational -- using the stronger definition of rational.

A disagreement of definitions is not really a disagreement. Someone suggested on the April open thread that we define "rationality". My idea of a definition would look something like this:

Rationality assumes that:

(1) The only source of knowledge is empirical experience.

(2) The only things that are known are deduced from empirical experience by valid logical reasoning and mathematics.

Weak Rationality assumes that:

(1) The first source of knowledge is empirical experience.

(2) The only things that are known with certainty are deduced from empirical experience by valid logical reasoning and mathematics.

(3) Define a belief system as all knowledge deduced from empirical observation with all metaphysical beliefs, if any. Then the belief system is rational (nearly rational or weakly rational) if the belief system is internally consistent.

Probably these definitions have been outlined somewhere better than they are here. Perhaps I have misplaced emphasis and certainly there are important nuances and variations. Whether this definition works or not, I think it's important to have a working set of definitions that we all agree upon. The wiki has just started out, but I think it's a terrific idea and worth putting time into. Every time you struggle with finding the right definition for something I suggest you add your effort to the group knowledge by adding that definition to the Wiki.

I made the accusation that the consensus about religion was due ...
Declare your signaling and hidden agendas  
by Kaj_Sotala — 2009-04-13T12:01:06.657Z

Follow-up to: It's okay to be (at least a little) irrational
Many science journals require their authors to declare any competing interests they happen to have. For instance, if you're submitting a study about the health effects of tobacco, and you happen to sit on the board of directors of a major tobacco company, you're supposed to say that out loud.
The process obviously isn't perfect, as most journals don't have the resources to ensure their authors do actually declare all competing interests. On the whole, though, it helps protect both the readers and the authors. The readers, because they'll know to be more careful in evaluating the reports of researchers who might be biased. The authors, because by declaring any competing interests upfront, they're protected from later accusations of dishonesty. (That's the theory, at least. In practice, authors often don't declare their interests, even if they should.)
Signaling has been discussed a lot on Overcoming Bias, though a bit less on Less Wrong. A large fraction of people's behavior is actually intended to signal some qualities to others, though this isn't necessarily a conscious process. On the other hand, it often is. As seasoned OB/LW readers, it seems to me like many would instinctively try to avoid giving the impression of excess signaling. We're rationalists, after all! We're trying to find the truth, not show off or impress others of our worth!
As if we even could avoid trying to make a good impression on others, or avoid having other kinds of hidden agendas. We're not any less human simply because we have rallied our rationality's banner. (Not to mention that signaling isn't a bad thing, by itself - humanity would be in a very poor state if we didn't have any signals about what others were like.) So, in the interest of self-honesty, I suggest we all begin explicitly declaring our (conscious) hidden agendas and signaling intentions when writing posts. As with the policy of scholarly journals, this will help both readers and writers, and in this case also serve a third and fourth function - making us more honest to ourselves, and make people realize that it's okay to have hidden agendas, and that they don't have to pretend they don't have any. I'll start out with mine.

I have roughly classified my hidden agendas at three different levels of severity. A "mild" agenda had a small impact on the behavior...
Persuasiveness vs Soundness
by Patrick — 2009-04-13T08:43:12.255Z

Compare the following two arguments.

E. is described by the following axioms

Therefore under E. The square of the longest side of a right angle triangle is equal to the sum of the squares of the remaining two sides.

All men are mortal.

Socrates is a man.

Therefore Socrates is mortal.

Naively, the second argument seems tautological, whereas with the first it's much harder to tell. Of course in reality the first argument is the tautology and the second argument is the more dubious one. The phrase "immortal man" doesn't seem contradictory, and how do we know Socrates is a man? He could be an android doppleganger. And the first argument's conclusion completely follows from euclid's axioms. Euclid and 100s of other mathematicians have proved that it does.

So, why does the average human think the opposite?

The arguments that change what we think, and the arguments that would change what a logically omniscient bayesian superman wielding Solomonoff's Lightsaber thinks are not very tightly correlated. In fact, there's a whole catalogue dedicated to finding out types of arguments we're persuaded by that a bayesian superman wouldn't be, the logical fallacies. But it's not just argument structure that causes us to lose our way, another factor is how well the argument is written.

People are more persuaded by essays from Eliezer Yudkowsky, Bertrand Russell, Paul Graham or George Orwell than they would be from a forum post by an average 13 year old atheist, even if they both make the exact same point. This threw me for a bit of a loop, until I realized that Eliezer was pitching to bayesian supermen as well as us mortals. How well stylish writing correlates to the truth compared to unstylish writing is nowhere near how much we are persuaded by stylish writing compared to unstylish writing.

There's also the dreaded intuition pump, I think the reason it's so maligned is because it makes things much more persuasive without making them any more sound. A well chosen metaphor can do more to the human mind than a thousand pages of logic. Of course, we *want* intuition pumps for things that are actually true, because we want people to persuaded of things that are true and more importantly, we want them to be able to reason about things that are true. A good metaphor can enable this reasoning far more effectively than a list of axioms.

The problem lies in both directions,...
Bystander Apathy
by Eliezer Yudkowsky — 2009-04-13T01:26:15.635Z

The bystander effect, also known as bystander apathy, is that larger groups are less likely to act in emergencies - not just individually, but collectively. Put an experimental subject alone in a room and let smoke start coming up from under the door. 75% of the subjects will leave to report it. Now put three subjects in the room - real subjects, none of whom know what's going on. On only 38% of the occasions will anyone report the smoke. Put the subject with two confederates who ignore the smoke, and they'll only report it 10% on the time - even staying in the room until it becomes hazy. (Latane and Darley 1969.)

On the standard model, the two primary drivers of bystander apathy are:

Diffusion of responsibility - everyone hopes that someone else will be first to step up and incur any costs of acting. When no one does act, being part of a crowd provides an excuse and reduces the chance of being held personally responsible for the results.

Pluralistic ignorance - people try to appear calm while looking for cues, and see... that the others appear calm.

Cialdini (2001):

Very often an emergency is not obviously an emergency. Is the man lying in the alley a heart-attack victim or a drunk sleeping one off? ... In times of such uncertainty, the natural tendency is to look around at the actions of others for clues. We can learn from the way the other witnesses are reacting whether the event is or is not an emergency. What is easy to forget, though, is that everybody else observing the event is likely to be looking for social evidence, too. Because we all prefer to appear poised and unflustered among others, we are likely to search for that evidence placidly, with brief, camouflaged glances at those around us. Therefore everyone is likely to see everyone else looking unruffled and failing to act.

Cialdini suggests that if you're ever in emergency need of help, you point to one single bystander and ask them for help - making it very clear to whom you're referring. Remember that the total group, combined, may have less chance of helping than one individual.

I've mused a bit on the evolutionary psychology of the bystander effect. Suppose that in the ancestral environment, most people in your band were likely to be at least a little related to you - enough to be worth saving...
Marketing rationalism
by PhilGoetz — 2009-04-12T21:41:26.537Z

Suppose you're a protestant, and you want to convince other people to do what the Bible says to do. Would you persuade them by showing them that the Bible says that they should?

Now suppose you're a rationalist, and you want to convince other people to be rational. Would you persuade them with a rational argument?

If not, how?

ADDED: I'm not talking about persuading others who already accept reason as final arbiter to adopt Bayesian principles, or anything like that. I mean persuading Joe on the street who does whatever feels good, and feels pretty good about that. Or a doctor of philosophy who believes that truth is relative and reason is a social construct. Or a Christian who believes that the Bible is God's Word, and things that contradict the Bible must be false.

Christians don't place a whole set of the population off-limits and say, "These people are unreachable; their paradigms are too different." They go after everyone. There is no class of people whom they are unsuccessful with.

Saying that we have to play by a set of self-imposed rules in the competition for the minds of humanity, while our competitors don't, means we will lose. And isn't rationality about winning?

ADDED: People are missing the point that the situation is symmetrical for religious evangelists. For them to step outside of their worldview, and use reason to gain converts, is as epistemically dangerous for them, as it is for us to gain converts using something other than reason. Contemporary Christians consider themselves on good terms with reason; but if you look back in history, you'll find that many of the famous and influential Christian theologians (starting with Paul) made explicit warnings against the temptation of reason. The proceedings from Galileo's trial contain some choice bits on the relation between reason and faith.

Using all sorts of persuasive techniques that are not grounded in religious truth, and hence are epistemically repulsive to them and corrosive to their belief system, has proven a winning strategy for all religions. It's a compromise; but these compromises did not weaken those religions. They made them stronger.
It's okay to be (at least a little) irrational
by Kaj_Sotala — 2009-04-12T21:06:20.881Z

Caused by: Purchase Fuzzies and Utilons Separately
As most readers will know by now, if you're donating to a charity, it doesn't make sense to spread your donations across several charities (assuming you're primarily trying to maximize the amount of good done). You'll want to pick the charity where your money does the most good, and then donate as much as possible to that one. Most readers will also be aware that this isn't intuitive to most people - many will instinctively try to spread their money across several different causes.
I'm spending part of my income on charity, too. Admittedly, this isn't much - 30 USD each month - but then neither is my income as a student. Previously I had been spreading that sum to three different charities, each of them getting an equal amount. On at least two different venues, people had (not always knowingly) tried to talk me out of it, and I did feel that their arguments were pretty strong. Still, I didn't change my ways, even though there was mental pressure building up, trying to push me in that direction. There were actually even some other charities I was considering also donating to, even though I knew I probably shouldn't.
Then I read Eliezer's Purchase Fuzzies and Utilons Separately. Here was a post saying, in essence, that it's okay to spend some of your money in what amounted to an irrational way. Yes, go ahead and spread your money, and go ahead and use some of it just to purchase warm fuzzies. You're just human, after all. Just try to make sure you still donate more to a utilon maximizer than to purchasing the fuzzies.
Here I was, with a post that allowed me to stop rationalizing reasons for why spreading money was good, and instead spread them because I was honestly selfish and just buying a good feeling. Now, I didn't need to worry about being irrational in having diversified donations. So since it was okay, I logged in to PayPal, cancelled the two monthly donations I had going to the other organizations, and tripled the amount of money that I was giving to the Institute Which Shall Not Be Named.
Not exactly the outcome one might have suspected.

A theme that has come up several times is that it's easier to lie to others if you believe in the lies yourself. Being in a community where rationality is highly valued, many people will probably want to avoid appearing irrational, lest they lose the respect of others. They...