trentmkelly/LessWrong-43k huggingface dataset:
Lie to me?
by pwno — 2009-06-24T21:56:15.638Z

I used to think that, given two equally capable individuals, the person with more true information can always do at least as good as the other person. And hence, one can only gain from having true information. There is one implicit assumption that makes this line of reason not true in all cases. We are not perfectly rational agents; our mind isn’t stored in a vacuum, but in a Homo sapien brain.There are certain false beliefs that benefit you by exploiting your primitive mental warehouse, e.g., self-fulfilling prophecies.

Despite the benefits, adopting false beliefs is an irrational practice. If people never acquire the maps that correspond the best to the territory, they won’t have the most accurate cost-benefit analysis for adopting false beliefs. Maybe, in some cases, false beliefs make you better off. The problem is you'll have a wrong or sub-optimal cost-benefit analysis, unless you first adopt reason.

Also, it doesn’t make sense to say that the rational decision could be to “have a false belief” because in order to make that decision, you would have to compare that outcome against “having a true belief.” But in order for a false belief to work, you must truly believe in it — you cannot deceive yourself into believing the false belief after knowing the truth! It’s like figuring out that taking a placebo leads to the best outcome, yet knowing it’s a placebo no longer makes it the best outcome.

Clearly, it is not in your best interest to choose to believe in a falsity—but what if someone else did the choosing? Can’t someone whose advice you rationally trust be the decider of whether to give you false information or not (e.g. a doctor deciding whether you receive a placebo or not)? They could perform a cost-benefit analysis without diluting the effects of the false belief. We only want to know the truth, but prefer to be unknowingly lied to in some cases.

Which brings me to my question: do we program an AI to only tell us the truth or to lie when the AI believes (with high certainty) the lie will lead us to a net benefit over our expected lifetime?

Added: Keep in mind that knowledge of the truth, even for a truth-seeker, is finite in value. The AI can believe that the benefit of a lie would outweigh a truth-seeker's cost of being lied to. So unless someone values the truth above anything els...
Guilt by Association
by Annoyance — 2009-06-24T17:29:10.462Z

The formal concept of the fallacious argument was born as the twin of logic itself. When the ancient Greeks first began to systematically examine the natural arguments people made as they sought to demonstrate the truth of propositions, they noted that certain types of arguments were vulnerable to counterexamples while others were not. The vulnerable were not true - when it was claimed that they justified a conclusion they could not rule out the alternative - and so were identified as fallacious.

Although the validity of logical arguments can be determined through logic, that doesn't particularly distinguish one fallacy from another. It is a curious fact that, despite this, some fallacies are more frequently made by human beings than others. Much more.

For example, "If P, then Q. P. Therefore, Not-Q." is just as basic and elemental an error as "If P, then Q. Q. Therefore, P." is. But the first fallacy is hardly ever found (humans being what they are, there's probably no mistake within our reach that is never made) while the second is extraordinarily common.

It is in fact generally true that we often confuse a unidirectional implication with the bidirectional. If something implies another thing, we leap to the conclusion that the second also implies the first, that the connection is equally strong each way, even though it is fairly trivial to demonstrate that this isn't necessarily the case. This error seems to be inherent to human intuition, as it occurs across contexts and subjects quite regularly, even when people are aware that it's logically invalid; only careful examination counters this tendency.

Much later, Sigmund Freud began to identify ways that people would deny assertions that they found emotionally threatening, what we now call 'psychological defense mechanisms'. The flaws in Freud's work as a whole are not directly relevant to this discussion and are beyond the scope of this site in any case. Suffice it to say that therapists and psychologists do not consider his theories to be either true or useful, that they do consider them to be unscientific and a self-reinforcing belief system, and that many of the concepts which he introduced and have been taken up into the culture at large are invalid. Not all of his work is so flawed, though - particularly his early ideas.

Th...
The Monty Maul Problem
by JGWeissman — 2009-06-24T05:30:16.760Z

In his Coding Horror Blog, Jeff Atwood writes about the Monty Hall Problem and some variants. The classic problem presents the situation in which the game show host allows a contestant to choose one of three doors, one of which opens to reveal a prize while the other two reveals goats. The host then opens one of the other doors, reliably choosing one that has a goat, and invites the contestant to switch to the remaining unopened door. The problem is to determine the probability of winning the prize by switching and staying. The variants deal with cases in which the host does not reliably choose a door with a goat, but happens to do so.

Jeff cites Monty Hall, Monty Fall, Monty Crawl (PDF) by Jeff Rosenthal, which explains why the variants have different probabilities in terms of the "Proportionality Principle", which the appendix acknowledges to be a special case of Bayes' Theorem.

One of Jeff's anonymous commenters presented the Monty Maul Problem:

Hypothetical Situation:

The Monty Maul problem. There are 1 million doors. You pick one, and the shows host goes on a bloodrage fueled binge of insane violence, knocking open doors at random with no knowledge of which door has the car. He knocks open 999,998 doors, leaving your door and one unopened door. None of the opened doors contains the car.

Are your odds of winning if you switch still 50/50, as outlined by the linked Rosenthal paper? It seems counter-intuitive even for people who've wrapped their head around the original problem.

If you take as absolute the problem's statement the host is randomly knocking doors open, then yes, the fact that only goats were revealed is strong evidence that only goats were available because you picked the door with the prize, which, when combined with the low prior probability that you picked the door with the prize, gives equal probability to either of the unopened doors having the prize.

However, the fact that only goats were revealed is also strong evidence that the host deliberately avoided opening the door with the prize, and therefor switching is a winning strategy. After all, the probability of this happening if the host really is choosing doors randomly is 2 in a million, but it is guaranteed if the host deliberately opened only doors with goats.

Note that this principal still applies in variants with fewer doors. Unless there is an actual penalty for switching doors (wh...
The Domain of Your Utility Function
by Peter_de_Blanc — 2009-06-23T04:58:55.550Z

Unofficial Followup to: Fake Selfishness, Post Your Utility Function

A perception-determined utility function is one which is determined only by the perceptual signals your mind receives from the world; for instance, pleasure minus pain. A noninstance would be number of living humans. There's an argument in favor of perception-determined utility functions which goes like this: clearly, the state of your mind screens off the state of the outside world from your decisions. Therefore, the argument to your utility function is not a world-state, but a mind-state, and so, when choosing between outcomes, you can only judge between anticipated experiences, and not external consequences. If one says, "I would willingly die to save the lives of others," the other replies, "that is only because you anticipate great satisfaction in the moments before death - enough satisfaction to outweigh the rest of your life put together."

Let's call this dogma perceptually determined utility. PDU can be criticized on both descriptive and prescriptive grounds. On descriptive grounds, we may observe that it is psychologically unrealistic for a human to experience a lifetime's worth of satisfaction in a few moments. (I don't have a good reference for this, but) I suspect that our brains count pain and joy in something like unary, rather than using a place-value system, so it is not possible to count very high.

The argument I've outlined for PDU is prescriptive, however, so I'd like to refute it on such grounds. To see what's wrong with the argument, let's look at some diagrams. Here's a picture of you doing an expected utility calculation - using a perception-determined utility function such as pleasure minus pain.

Here's what's happening: you extrapolate several (preferably all) possible futures that can result from a given plan. In each possible future, you extrapolate what would happen to you personally, and calculate the pleasure minus pain you would experience. You call this the utility of that future. Then you take a weighted average of the utilities of each future — the weights are probabilities. In this way you calculate the expected utility of your plan.

But this isn't the most general possible way to calculate utilities.

Instead, we could calculate utilities based on any properties of the extrapolated futures — anything at all, such as how many people there are, how man...
Shane Legg on prospect theory and computational finance
by Roko — 2009-06-21T17:57:09.235Z

People are not rational expected utility maximisers. When we have to make decisions, all sorts of cognitive biases and distortions come into play. Seminal work in this area was done by Kahneman and Tversky. They produced a model of human decision making known as prospect theory, work that Kahneman later won a Noble prize ...

... I looked at these investors and thought, “Hey, they’re just like reinforcement learning agents. No big deal. If I want to know what investors with probability weighting and a curved value function do, I can just brute force compute their optimal policy by writing down their Bellman equation and using dynamic programming. Easy!” It was a mystery to me why, seemingly, nobody else was doing that. So off I went to build software to do just this, starting with a simple Merton model…

... When we fired up my simulator and gave this distribution to an investor that had probability weighting: the investor took one look at that scary negative tail and didn’t want to invest in the stock. This is exactly what the model should predict. In short, we took realistic stock returns, and presented this to an investor with a realistic decision making process complete with a bunch of parameters that have been empirically estimated by others in previous work, and what we got out the other end was realistic investor behaviour!

Read the whole article here at Vetta Project.
Nonparametric Ethics
by Eliezer Yudkowsky — 2009-06-20T11:31:59.758Z

(Inspired by a recent conversation with Robin Hanson.)

Robin Hanson, in his essay on "Minimal Morality", suggests that the unreliability of our moral reasoning should lead us to seek simple moral principles:

"In the ordinary practice of fitting a curve to a set of data points, the more noise one expects in the data, the simpler a curve one fits to that data. Similarly, when fitting moral principles to the data of our moral intuitions, the more noise we expect in those intuitions, the simpler a set of principles we should use to fit those intuitions. (This paper elaborates.)"

In "the limit of expecting very large errors of our moral intuitions", says Robin, we should follow an extremely simple principle - the simplest principle we can find that seems to compress as much morality as possible. And that principle, says Robin, is that it is usually good for people to get what they want, if no one else objects.

Now I myself carry on something of a crusade against trying to compress morality down to One Great Moral Principle. I have developed at some length the thesis that human values are, in actual fact, complex, but that numerous biases lead us to underestimate and overlook this complexity. From a Friendly AI perspective, the word "want" in the English sentence above is a magical category.

But Robin wasn't making an argument in Friendly AI, but in human ethics: he's proposing that, in the presence of probable errors in moral reasoning, we should look for principles that seem simple to us, to carry out at the end of the day. The more we distrust ourselves, the simpler the principles.

This argument from fitting noisy data, is a kind of logic that can apply even when you have prior reason to believe the underlying generator is in fact complicated. You'll still get better predictions from the simpler model, because it's less sensitive to noise.

Even so, my belief that human values are in fact complicated, leads me to two objections and an alternative proposal:

The first objection is that we do, in fact, have enough data to support moral models that are more complicated than a small set of short English sentences. If you have a thousand data points, even noisy data points, it may be a waste of evidence to try to fit them to a straight line, especially if you have prior reason to believe the true generator is not linear.

And ...
ESR's comments on some EY:OB/LW posts
by Eliezer Yudkowsky — 2009-06-20T00:16:29.923Z

Eric S. Raymond's comments on some of my Overcoming Bias posts.

In his reply to my To Lead, You Must Stand Up, he writes:

"I think your exhortations here are nearly useless. Experience I’ve collected over the last ten years suggests to me that the kind of immunity to stage fright you and I have is a function of basic personality type at the neurotransmitter-balance level, and not really learnable by most people."

This is a particularly interesting observation if combined with Hanson's hypothesis that people choke to submit.

"I disagree with The Futility of Emergence," says ESR. Yea, many have said this to me. And they go on to say: Emergence has the useful meaning that... And it's a different meaning every time. In ESR's case it's:

"The word 'emergent' is a signal that we believe a very specific thing about the relationship between 'neurons firing' and 'intelligence', which is that there is no possible account of intelligence in which the only explanatory units are neurons or subsystems of neurons."

Let me guess, you think the word "emergence" means something useful but that's not exactly it, although ESR's definition does aim in the rough general direction of what you think is the right definition...

So-called "words" like this should not be actually spoken from one human to another. It is tempting fate. It would be like trying to have a serious discussion between two theologians if both of them were allowed to say the word "God" directly, instead of always having to say whatever they meant by the word.
Cascio in The Atlantic, more on cognitive enhancement as existential risk mitigation
by Roko — 2009-06-18T15:09:57.954Z

Jamais Cascio writes in the atlantic:

Pandemics. Global warming. Food shortages. No more fossil fuels. What are humans to do? The same thing the species has done before: evolve to meet the challenge. But this time we don’t have to rely on natural evolution to make us smart enough to survive. We can do it ourselves, right now, by harnessing technology and pharmacology to boost our intelligence. Is Google actually making us smarter? ...

... Modafinil isn’t the only example; on college campuses, the use of ADD drugs (such as Ritalin and Adderall) as study aids has become almost ubiquitous. But these enhancements are primitive. As the science improves, we could see other kinds of cognitive-modification drugs that boost recall, brain plasticity, even empathy and emotional intelligence. They would start as therapeutic treatments, but end up being used to make us “better than normal.”

Read the whole article here.

This relates to cognitive enhancement as existential risk mitigation, where Anders Sandberg wrote:

Would it actually reduce existential risks? I do not know. But given correlations between long-term orientation, cooperation and intelligence, it seems likely that it might help not just to discover risks, but also in ameliorating them. It might be that other noncognitive factors like fearfulness or some innate discounting rate are more powerful.

The main criticisms of this idea generated in the Less Wrong comments were:

The problem is not that people are stupid. The problem is that people simply don't give a damn. If you don't fix that, I doubt raising IQ will be anywhere near as helpful as you may think. (Psychohistorian)

Yes, this is the key problem that people don't really want to understand. (Robin Hanson)

Making people more rational and aware of cognitive biases material would help much more (many people)

These criticisms really boil down to the same thing: people love their cherished falsehoods! Of course, I cannot disagree with this statement. But it seems to me that smarter people have a lower tolerance for making utterly ridiculous claims in favour of their cherished falsehood, and will (to some extent) be protected from believing silly things that make them (individually) feel happier, but are highly unsupported by evidence. Case in point: religion. This study1 states that

Evidence is reviewed pointing to a ne

...
Time to See If We Can Apply Anything We Have Learned
by MichaelVassar — 2009-06-18T10:06:12.174Z

It seems to me that this blog has just reached it's first real crisis.

Three people are announcing three apparently opposed beliefs with substantial real expected consequences and yet no-one has yet spoken, or it seems to me implied, the key slogan... "LETS USE SCIENCE!" or, as hubristic Bayesian wannabes, not invoked Bayes as an idol to swear by, but rather said "LETS USE HUMANE REFLECTIVE DECISION THEORY, THE QUANTITATIVELY UNKNOWN BUT QUALITATIVELY INTUITED POWER DEEPER THAN SCIENCE FROM WHICH IT STEMS AND TO WHICH OUR COMMUNITY IS DEVOTED".

IF RDS was applied to our current situation, people would be analyzing Yvain's, Davis' and Eby's proposals, working out exactly what their implications are, and trying to propose, in the name of SCIENCE, hypotheses which will distinguish between them, and in the name of BAYES, confidence estimates of their analyses and of the quality with which the denotations of their words have cleaved reality at the joints enabling an odds ratio of updating to be extracted from a single data point. People would be working out what features of which of the models used by Yvain, Davis and Eby constitute evidence against what other features. They would be trying to evaluate non-verbally, through subjectively opaque but known-to-be-informative processes vulnerable to verbal overshadowing, what relative odds to place on those different features of the models. Finally, they would be examining the expected costs entailed by experiments being proposed and selecting those experiments which promise to provide the most information for the least cost be performed. The cost estimate would include both the effort required to perform the experiments, probably best assessed with an outside view in most cases like these, and the dangers to the minds of the participants from possible adverse outcomes, taking into account, as well as possible, the structural uncertainty of the models.

I sincerely hope to see some of that in the comments section soon, either under this post or the "Applied Picoeconomics" post.
The Physiology of Willpower
by pjeby — 2009-06-18T04:11:52.445Z

This paper (PDF)1 looks more than a little interesting:

Past research indicates that self-control relies on some sort of limited energy source. This review suggests that blood glucose is one important part of the energy source of selfcontrol. Acts of self-control deplete relatively large amounts of glucose. Self-control failures are more likely when glucose is low or cannot be mobilized effectively to the brain (i.e., when insulin is low or insensitive). Restoring glucose to a sufficient level typically improves self-control. Numerous self-control behaviors fit this pattern, including controlling attention, regulating emotions, quitting smoking, coping with stress, resisting impulsivity, and refraining from criminal and aggressive behavior. Alcohol reduces glucose throughout the brain and body and likewise impairs many forms of self-control. Furthermore, self-control failure is most likely during times of the day when glucose is used least effectively. Self-control thus appears highly susceptible to glucose. Self-control benefits numerous social and interpersonal processes. Glucose might therefore be related to a broad range of social behavior.

I find this interesting, in that the days I get less work done (due to e.g. spending more time on Less Wrong) are often days when I don't eat breakfast right away, and am generally undereating (like today).

References

1. Matthew T. Gailliot, Roy F. Baumeister. (2007) The Physiology of Willpower: Linking Blood Glucose to Self-Control. Personality and Social Psychology Review, Vol. 11, No. 4, 303-327
Representative democracy awesomeness hypothesis
by taw — 2009-06-18T03:02:37.973Z

I have hypothesis that utility maximization is always a second order process - there's always some underlying selection process with its fitness, and only because it promotes traits that make agents agents act in a way that best approximates utility maximizing, adaptation executers seem to us like utility maximizers.

Now let's apply this to political systems:

In most political systems ruling elites could only be removed by either military intervention from the outside, or revolution from inside. That's the fitness function. So traits that would be promoted were those related to having strong and effective military, and skillful diplomacy to avoid external threats; keeping population in conditions bearable enough to prevent a revolt; strong policing and effective divide&conquer approach to keep any revolt from succeeding.

In modern two-party (or multi-party with two big possible coalitions) representative democracy economic performance of the last term can predict voting patterns very well - if people's incomes and well-being are improving fast enough, people will vote for the ruling party. If they're deteriorating, or not improving fast enough, people will vote for the opposition (Bread and Peace Model). This means parties' fitness function is strongly linked to short term economic performance, and therefore policies that improve people's well-being will be selected.

In direct democracy with most important decisions being taken directly by voters, there's no selection process - so they will be as unsuccessful as dictatorships. This is extremely surprising prediction to me, but remarkably bad economic performance of California seems to confirm this.

I guess that in single party regimes, like those of post-Stalin communist countries party leaders would be removed in case of highly unsuccessful performance. This would produce weaker selection than in two-party representative democracies, but stronger selection than in pure dictatorships, or direct democracies.

Libertarian idea of free competition between political systems never existed, so there is no need to discuss it.

There are also some hints how to design better representative democracy:

The closest a representative democracy is to proportional one-person one-vote system, be stronger the signal. So countries with highly distorting electoral systems like US would have weaker selection than countries w

...