trentmkelly/LessWrong-43k huggingface dataset:
Typical Mind and Politics
by Scott Alexander — 2009-06-12T12:28:39.826Z

Yesterday, in the The Terrible, Horrible, No Good Truth About Morality, Roko mentioned some good evidence that we develop an opinion first based on intuitions, and only later look for rational justifications. For example, people would claim incest was wrong because of worries like genetic defects or later harm, but continue to insist that incest was wrong even after all those worries had been taken away.

Roko's examples take advantage of universal human feelings like the incest taboo. But if people started out with opposite intuitions, then this same mechanism would produce opinions that people hold very strongly and are happy to support with as many reasons and facts as you please, but which are highly resistant to real debate or to contradicting evidence.

Sound familiar?

But to explain politics with this mechanism, we'd need an explanation for why people's intuitions differed to begin with. We've already discussed some such explanations - self-serving biases, influence from family and community, et cetera - but today I want to talk about another possibility.

A few weeks back, I was discussing harms with Bill Swift on Overcoming Bias. In particular, I was arguing that one situation in which there was an open-and-shut case for government restriction of private activity on private property was nuisance noise. I argued that if you were making noise on your property, and I could hear it on my property, that I was being harmed by your actions and that there was clearly just as much a case for government intervention here as if you were firing flaming arrows at me from your property. I fully expected Bill to agree that this was obviously true but to have some reason why he didn't think it applied to our particular disagreement.

Instead, to my absolute astonishment, Bill said that noise wasn't really a problem. He said he lived on a noisy property and had just stopped whining and gotten on with his life. I didn't really know how to react to this1, and ended up assuming either that he'd never lived in a really noisy place like I have, or that he was such a blighted ideologue that he was willing to completely contradict common sense in order to preserve his silly argument.

In other words, I was assuming the person I was debating was either astonishingly stupid or willfully evil. And when my thoughts tend in that direction, it usually means I'm missing something.

Luckily in...
If it looks like utility maximizer and quacks like utility maximizer...
by taw — 2009-06-11T18:34:35.971Z

...it's probably adaptation executer.

We often assume agents are utility maximizers. We even call this "rationality". On the other hand in our recent experiment nobody managed to figure out even approximate shape of their utility function, and we know about large number of ways how agents deviate from utility maximization. How goes?

One explanation is fairly obvious. Nature contains plenty of selection processes - evolution and markets most obviously, but plenty of others like competition between Internet forums in attracting users, and between politicians trying to get elected. In such selection processes a certain property - fitness - behaves a lot like utility function. As a good approximation a traits that give agents higher expected fitness survives and proliferates. And as a result of that agents that survive such selection processes react to inputs quite reliably as if they were optimizing some utility function - fitness of the underlying selection process.

If that's the whole story, we can conclude a few things:

Utility maximization related to some selection process - evolutionary fitness, money, chance of getting elected - is very common and quite reliable.

We will not find any utility maximization related outside selection process - so we are not maximizers when it comes to happiness, global well-being, ecological responsibility, and so on.

If selection process isn't strong enough, we won't see much utility maximization. So we can predict that oligopolies are pretty bad at maximizing their market success - as pressure is very low.

If selection process isn't running long enough, or rules changed recently, we won't see much utility maximization. So people are really horrible at maximizing number of their offspring in our modern environment with birth control and limitless resources. And we can predict that this might change given enough time.

If input is atypical, even fairly reliable utility maximizers are likely to behave badly.

Agents are not utility maximizers across selection processes. So while politicians can be assumed to maximize their chance of getting elected very well, they must be pretty bad at getting as maximizing their income, or number of children.

However, selection processes often correlate - historically someone who made more money could afford to have more children, richer politicians can use their wealth to increase their chance o

...
Let's reimplement EURISKO!
by cousin_it — 2009-06-11T16:28:06.637Z

In the early 1980s Douglas Lenat wrote EURISKO, a program Eliezer called "[maybe] the most sophisticated self-improving AI ever built". The program reportedly had some high-profile successes in various domains, like becoming world champion at a certain wargame or designing good integrated circuits.

Despite requests Lenat never released the source code. You can download an introductory paper: "Why AM and EURISKO appear to work" [PDF]. Honestly, reading it leaves a programmer still mystified about the internal workings of the AI: for example, what does the main loop look like? Researchers supposedly answered such questions in a more detailed publication, "EURISKO: A program that learns new heuristics and domain concepts." Artificial Intelligence (21): pp. 61-98. I couldn't find that paper available for download anywhere, and being in Russia I found it quite tricky to get a paper version. Maybe you Americans will have better luck with your local library? And to the best of my knowledge no one ever succeeded in (or even seriously tried) confirming Lenat's EURISKO results.

Today in 2009 this state of affairs looks laughable. A 30-year-old pivotal breakthrough in a large and important field... that never even got reproduced. What if it was a gigantic case of Clever Hans? How do you know? You're supposed to be a scientist, little one.

So my proposal to the LessWrong community: let's reimplement EURISKO!

We have some competent programmers here, don't we? We have open source tools and languages that weren't around in 1980. We can build an open source implementation available for all to play. In my book this counts as solid progress in the AI field.

Hell, I'd do it on my own if I had the goddamn paper.

Update: RichardKennaway has put Lenat's detailed papers up online, see the comments.
The Terrible, Horrible, No Good, Very Bad Truth About Morality and What To Do About It
by Roko — 2009-06-11T12:31:02.904Z

Joshua Greene has a PhD thesis called The Terrible, Horrible, No Good, Very Bad Truth About Morality and What To Do About It. What is this terrible truth? The essence of this truth is that many, many people (probably most people) believe that their particular moral (and axiological) views on the world are objectively true - for example that anyone who disagrees with the statement "black people have the same value as any other human beings" has committed either an error of logic or has got some empirical fact wrong, in the same way that people who claim that the earth was created 6000 years ago are objectively wrong.

To put it another way, Greene's contention is that our entire way of talking about ethics - the very words that we use - force us into talking complete nonsense (often in a very angry way) about ethics. As a simple example, consider the use of the words in any standard ethical debate - "abortion is murder", "animal suffering is just as bad as human suffering" - these terms seem to refer to objective facts; "abortion is murder" sounds rather like "water is a solvent!". I urge readers of Less Wrong to put in the effort of reading a significant part of Greene's long thesis starting at chapter 3: Moral Psychology and Projective Error, considering the massively important repercussions he claims his ideas could have:

In this essay I argue that ordinary moral thought and language is, while very natural, highly counterproductive and that as a result we would be wise to change the way we think and talk about moral matters. First, I argue on metaphysical grounds against moral realism, the view according to which there are first order moral truths. Second, I draw on principles of moral psychology, cognitive science, and evolutionary theory to explain why moral realism appears to be true even though it is not. I then argue, based on the picture of moral psychology developed herein, that realist moral language and thought promotes misunderstanding and exacerbates conflict. I consider a number of standard views concerning the practical implications of moral anti-realism and reject them. I then sketch and defend a set of alternative revisionist proposals for improving moral discourse, chief among them the elimination of realist moral language, especially deontological language, and the promotion of an anti-realist utilitarian framework for discussing moral is

...
Less wrong economic policy
by Gordon Seidoh Worley — 2009-06-09T20:11:56.083Z

Yesterday I heard an interesting story on the radio about US President Obama's pick to head the Office of Information and Regulatory Affairs, Cass Sunstein. I recommend checking out the story, but here are a few key excerpts.

Cass Sunstein, President Obama's pick to head the Office of Information and Regulatory Affairs, is a vocal supporter of [...] economic policy that shapes itself around human psychology. Sunstein is just one of a number of high-level appointees now working in the Obama administration who favors this kind of approach.

[...]

Through their research, Kahneman and Tversky identified dozens of these biases and errors in judgment, which together painted a certain picture of the human animal. Human beings, it turns out, don't always make good decisions, and frequently the choices they do make aren't in their best interest.

[...]

"Merely accepting the fact that people do not necessarily make the best decisions for themselves is politically very explosive. The moment that you admit that, you have to start protecting people," Kahneman says.

[...]

The Obama administration believes it needs to shape policy in a way that will keep us all from getting hit by trucks — health care trucks, financial trucks, trucks that come from every direction and affect every aspect of our lives.

At the risk of starting a discussion that will be wrecked by political wrestling, I'm always hopeful when I hear about governments applying what we learn from science to policy. Not to say that this always generates good policies, but it does generate the best policies we have reason to believe will be good (so long as you ignore the issue of actual politices that might get in the way).
You can't believe in Bayes
by PhilGoetz — 2009-06-09T18:03:20.808Z

Well, you can. It's just oxymoronic, or at least ironic. Because belief is contrary to the Bayesian paradigm.

You use Bayesian methods to choose an action. You have a set of observations, and assign probabilities to possible outcomes, and choose an action.

Belief in an outcome N means that you set p(N) ≈ 1 if p(N) > some threshold. It's a useful computational shortcut. But when you use it, you're not treating N in a Bayesian manner. When you categorize things into beliefs/nonbeliefs, and then act based on whether you believe N or not, you are throwing away the information contained in the probability judgement, in order to save computation time. It is especially egregious if the threshold you use to categorize things into beliefs/nonbeliefs is relatively constant, rather than being a function of (expected value of N) / (expected value of not N).

If your neighbor took out fire insurance on his house, you wouldn't infer that he believed his house was going to burn down. And if he took his umbrella to work, you wouldn't (I hope) infer that he believed it was going to rain.

Yet when it comes to decisions on a national scale, people cast things in terms of belief. Do you believe North Korea will sell nuclear weapons to Syria? That's the wrong question when you're dealing with a country that has, let's say, a 20% chance of building weapons that will be used to level at least ten major US cities.

Or flash back to the 1990s, before there was a scientific consensus that global warming was real. People would often say, "I don't believe in global warming." And interviews with scientists tried to discern whether they did or did not believe in global warming.

It's the wrong question. The question is what steps are worth taking according to your assigned probabilities and expected-value computations.

A scientist doesn't have to believe in something to consider it worthy of study. Do you believe an asteroid will hit the Earth this century? Do you believe we can cure aging in your lifetime? Do you believe we will have a hard-takeoff singularity? If a low-probability outcome can have a high impact on expected utility, you've already gone wrong when you ask the question.
Expected futility for humans
by Roko — 2009-06-09T12:04:29.306Z

Previously, Taw published an article entitled "Post your utility function", after having tried (apparently unsuccessfully) to work out "what his utility function was". I suspect that there is something to be gained by trying to work out what your priorities are in life, but I am not sure that people on this site are helping themselves very much by assigning dollar values, probabilities and discount rates. If you haven't done so already, you can learn why people like the utility function formalism on wikipedia. I will say one thing about the expected utility theorem, though. An assignment of expected utilities to outcomes is (modulo renormalizing utilities by some set of affine transformations) equivalent to a preference over probabilistic combinations of outcomes; utilities are NOT properties of the outcomes you are talking about, they are properties of your mind. Goodness, like confusion, is in the mind.

In this article, I will claim that trying to run your life based upon expected utility maximization is not a good idea, and thus asking "what your utility function is" is also not a useful question to try and answer.

There are many problems with using expected utility maximization to run your life: firstly, the size of the set of outcomes that one must consider in order to rigorously apply the theory is ridiculous: one must consider all probabilistic mixtures of possible histories of the universe from now to whatever your time horizon is. Even identifying macroscopically identical histories, this set is huge. Humans naturally describe world-histories in terms of deontological rules, such as "if someone is nice to me, I want to be nice back to them", or "if I fall in love, I want to treat my partner well (unless s/he betrays me)", "I want to achieve something meaningful and be well-renowned with my life", "I want to help other people". In order to translate these deontological rules into utilities attached to world-histories, you would have to assign a dollar utility to every possible world-history with all variants of who you fall in love with, where you settle, what career you have, what you do with your friends, etc, etc. Describing your function as a linear sum of independent terms will not work in general because, for example, whether accounting is a good career for you will depend upon the kind of personal life you want to live (i.e. different aspects of...
The Aumann's agreement theorem game (guess 2/3 of the average)
by  — 2009-06-09T07:29:02.237Z

I'd like to play a game with you. Send me, privately, a real number between 0 and 100, inclusive. (No funny business. If you say "my age", I'm going to throw it out.) The winner of this game is the person who, after a week, guesses the number closest to 2/3 of the average guess. I will reveal the average guess, and will confirm the winner's claims to have won, but I will reveal no specific guesses.

Suppose that you're a rational person. You also know that everyone else who plays this game is rational, you know that they know that, you know that they know that, and so on. Therefore, you conclude that the best guess is P. Since P is the rational guess to make, everyone will guess P, and so the best guess to make is P*2/3. This gives an equation that we can solve to get P = 0.

I propose that this game be used as a sort of test to see how well Aumann's agreement theorem applies to a group of people. The key assumption the theorem makes--which, as taw points out, is often overlooked--is that the group members are all rational and honest and also have common knowledge of this. This same assumption implies that the average guess will be 0. The farther from the truth this assumption is, the farther the average guess is going to be from 0, and the farther Aumann's agreement theorem is from applying to the group.

Update (June 20): The game is finished; sorry for the delay in getting the results. The average guess was about 13.235418197890148 (a number which probably contains as much entropy as its length), meaning that the winning guess is the one closest to 8.823612131926765. This number appears to be significantly below the number typical for groups of ordinary people, but not dramatically so. 63% of guesses were too low, indicating that people were overall slightly optimistic about the outcome (if you interpret lower as better). Anyway, I will notify the winner ahora mismo.
London Rationalist Meetups bikeshed painting thread
by taw — 2009-06-08T01:56:09.381Z

Something that came up on each of our three meetups so far was that people want more participation on things like format, place, and meeting times.

Currently these are:

5th View cafe on top of Waterstones bookstore near Piccadilly Circus

First weekend of each month

Casual chat format

But these were just the first point we hit in the optimization space. They work, but that doesn't mean there isn't something that could work better.

So everyone who wants to discuss them, here's the place.
indexical uncertainty and the Axiom of Independence
by Wei Dai — 2009-06-07T09:18:48.891Z

I’ve noticed that the Axiom of Independence does not seem to make sense when dealing with indexical uncertainty, which suggests that Expected Utility Theory may not apply in situations involving indexical uncertainty. But Googling for "indexical uncertainty" in combination with either "independence axiom" or “axiom of independence” give zero results, so either I’m the first person to notice this, I’m missing something, or I’m not using the right search terms. Maybe the LessWrong community can help me figure out which is the case.
The Axiom of Independence says that for any A, B, C, and p, you prefer A to B if and only if you prefer p A + (1-p) C to p B + (1-p) C. This makes sense if p is a probability about the state of the world. (In the following, I'll use “state” and “possible world” interchangeably.) In that case, what it’s saying is that what you prefer (e.g., A to B) in one possible world shouldn’t be affected by what occurs (C) in other possible worlds. Why should it, if only one possible world is actual?
In Expected Utility Theory, for each choice (i.e. option) you have, you iterate over the possible states of the world, compute the utility of the consequences of that choice given that state, then combine the separately computed utilities into an expected utility for that choice. The Axiom of Independence is what makes it possible to compute the utility of a choice in one state independently of its consequences in other states.
But what if p represents an indexical uncertainty, which is uncertainty about where (or when) you are in the world? In that case, what occurs at one location in the world can easily interact with what occurs at another location, either physically, or in one’s preferences. If there is physical interaction, then “consequences of a choice at a location” is ill-defined. If there is preferential interaction, then “utility of the consequences of a choice at a location” is ill-defined. In either case, it doesn’t seem possible to compute the utility of the consequences of a choice at each location separately and then combine them into a probability-weighted average.
Here’s another way to think about this. In the expression “p A + (1-p) C” that’s part of the Axiom of Independence, p was originally ...
Macroeconomics, The Lucas Critique, Microfoundations, and Modeling in General
by Matt_Simpson — 2009-06-06T04:35:51.786Z

I posted this comment in reply to a post by David Henderson over at econlog, but first some context.

Mathew Yglesias writes:

...From an outside perspective, what seems to be going on is that economists have unearthed an extremely fruitful paradigm for investigation of micro issues. This has been good for them, and enhanced the prestige of the discipline. No such fruitful paradigm has actually emerged for investigation of macro issues. So the decision has been made to somewhat arbitrarily impose the view that macro models must be grounded in micro foundations. Thus, the productive progressive research program of microeconomics can “infect” the more troubled field of macro with its prestige...

...But as a methodological matter, it seems deeply unsound. As a general principle for investigating the world, we normally deem it desirable, but not at all necessary, that researchers exploring a particular field of inquiry find ways to “reduce” what they’re doing to a lower level....

...Trying to enhance models with better information about psychology isn’t against the rules, but it’s not required either. What’s required is that the models do useful work.

So why should it be that “in the current regime, if [macro models] are not meticulously constructed from “micro foundations,” they aren’t allowed to be considered”?

To which a commenter replies:

While I’m the first to acknowledge that the current macro research paradigm has given us little useful analysis, there is a standard answer to Matt’s question.

You can start by going to Wikipedia and reading about the Lucas Critique...

I won't reproduce the whole thing, click through to the comment to see a decent summary of the Lucas Critique if you aren't aware of it already.

Henderson, over at econlog, replies:

...Second, the demand for microfoundations, or at least the supply of them, goes back more than 10 years before the date Arnold claims. It goes back at least to Milton Friedman's A Theory of the Consumption Function...,

...Third, interestingly, Milton Friedman himself would probably agree with Yglesias about the idea that there's not necessarily a need for micro foundations for macro. In a 1996 interview published in Brian Snowdon and Howard R. Vane, Modern Macroeconomics, Edward Elgar, 2005, Friedm

...