trentmkelly/LessWrong-43k huggingface dataset:
How Not to be Stupid: Brewing a Nice Cup of Utilitea
by Psy-Kosh — 2009-05-09T08:14:18.363Z

Previously: "objective probabilities", but more importantly knowing what you want

Slight change of plans: the only reason I brought up the "objective" probabilities as early as I did was to help establish the idea of utilities. But with all the holes that seem to need to be patched to get from one to the other (continuity, etc), I decided to take a different route and define utilities more directly. So, for now, forget about "objective probabilities" and frequencies for a bit. I will get back to them a bit later on, but for now am leaving them aside.

So, we've got preference rankings, but not much of a sense of scale yet. We don't have much way of asking "how _much_ do you prefer this to that?" That's what I'm going to deal with in this post. There will be some slightly roundabout abstract bits here, but they'll let me establish utilities. And once I have that, more or less all I have to do is just use utilities as a currency to apply dutch book arguments to. (That will more or less be the shape of the rest of the sequence) The basic idea here is to work out a way of comparing the magnitudes of the differences of preferences. ie, How much you would have prefered some A2 to A1 vs how much you would have prefered some B2 to B1. But it seems difficult to define, no? "how much would you have wanted to switch reality to A2, if it was in state A1, vs how much would you have wanted to switch reality to B2, given that it was in B1?"

So far, the best solution I can think of is to ask "if you are equally uncertain about whether A1 or B1 is true, would you prefer to replace A1, if it would have been true, with A2, or similar for B1 to B2"? Specifically, supposing you're in a state of complete uncertainty with regards to two possible states/outcomes A1 or B1, so that you'd be equally surprised by both. Then consider that, instead of keeping that particular set of two possibilities, you have to choose between two substitutions: you can choose to either conditionally replace A1 with A2 (that is, if A1 would have been the outcome, you get A2 instead) _or_ you can choose to replace B1 with B2 in the same sense. So, you have to choose between (A2 or B1) and (A1 or B2) (where, again, your state of uncertainty is such that you'd be equally surprised by either outcome. That is, you can imagine that whatever it is that's giving rise to your uncertainty is effectively...
A Request for Open Problems
by MrHen — 2009-05-08T13:33:29.567Z

Open problems are clearly defined problems1 that have not been solved. In older fields, such as Mathematics, the list is rather intimidating. Rationality, on the other, seems to have no list.

While we have all of us here together to crunch on problems, let's shoot higher than trying to think of solutions and then finding problems that match the solution. What things are unsolved questions? Is it reasonable to assume those questions have concrete, absolute answers?

The catch is that these problems cannot be inherently fuzzy problems. "How do I become less wrong?" is not a problem that can be clearly defined. As such, it does not have a concrete, absolute answer. Does Rationality have a set of problems that can be clearly defined? If not, how do we work toward getting our problems clearly defined?

See also: Open problems at LW:Wiki

1: "Clearly defined" essentially means a formal, unambiguous definition. "Solving" such a problem would constitute a formal proof.
Framing Consciousness
by cousin_it — 2009-05-08T10:27:02.864Z

Update: you can ignore this post, it's completely wrong, I'm only leaving it up to preserve people's comments. Randallsquared has caught a crucial mistake in my reasoning: consciousness could require physical causality, rather than a property of some snapshot description. This falsifies my Point 3 below.

...

In this unabashedly geek-porn post I want to slightly expand our discussion of consciousness, as defined in the hard problem of consciousness. Don't be scared: no quantum claptrap or "informational system" bullshit.

Point 1. The existence (and maybe degree) of conscious/subjective experiences is an objective question.

Justification: if you feel a human possesses as much consciousness as a rock or the number three, stop reading now. This concludes the "proof by anthropic principle" or "by quantum immortality" for those still reading.

Point 2. It's either possible or impossible in principle to implement consciousness on a Turing-equivalent digital computer.

Justification: obvious corollary of Point 1.

Point 3. If consciousness is implementable on a digital computer, all imaginable conscious experiences already exist.

Justification: the state of any program can be encoded as an integer. What does it mean for an integer to "exist"? Does three "exist"? If a computer program gives rise to "actually existing" subjective experiences, then so does the decimal expansion of the x-coordinate of some particle in the Magellanic Cloud when written out in trinary.

Point 4. If consciousness is implementable, the Simulation Argument is invalid and Pascal's Mugging is almost certainly invalid.

Justification: obvious corollary of Point 3.

Point 5. If consciousness is non-implementable, the Simulation Argument and Robin's uploads scenario lose much of their punch.

Justification: the extinction threat in SA and the upload transition only feel urgent due to our current rapid progress with digital computers. We don't yet have a computer peripheral for providing programs with a feeling of non-implementable consciousness.

Point 6. If consciousness could be implementable, Eliezer had better account for it when designing his FAI.

Justification: there's no telling what the FAI will do when it realizes that actual humans have no privileged status over imaginable humans, or alternatively that they do and torturing simulated humans carries no moral weight.

Point 7. The implementability of currently k...
Replaying History
by Gordon Seidoh Worley — 2009-05-08T05:35:23.942Z

One of my favorite fiction genres is alternative history. The basic idea of alternative history is to write a story set in an alternate universe where history played out differently. Popular alternate histories include those where the Nazis win World War II, the USSR wins the Cold War, and the Confederate States of America win the American Civil War. But most of the writing in this genre has a serious flaw: the author starts out by saying "wouldn't it be cool to write a story where X had happened instead of Y" and then works backwards to concoct historical events that will lead to the desired outcome. No matter how good the story is, the history is often bad because at every stage the author went looking for a reason for things to go his way.

Being unsatisfied with most alternate histories, I like to play a historical "what if" game. Rather than asking the question at the conclusion, though (like "what if the Nazis had won the war"), I ask it at an earlier moment, ideally one where chance played an important role. What if Napoleon had been convinced not to invade Russia? What if the Continental Army had not successfully retreated from New York? What if Viking settlements in Newfoundland had not collapsed? These are as opposed to "What if Napoleon had never been defeated?", "What if the Colonies had lost the American Revolutionary War?", and "What if Vikings had developed a thriving civilization in the Americas?". I find that replaying history in this way a fun use of my analytical skills, but more importantly a good test of my rationality.

One of the most difficult things in thinking of an alternative history is to stay focused on the facts and likely outcomes. It's easy to say "I'd really like to see a world where X happened" and then silently or overtly bias your thinking until you find a way to achieve the desired outcome. Learning to avoid this takes discipline, especially in a domain like alternate history where there's no way to check if your reasoning turned out to be correct. But unlike imagining the future, making an alternate history does have the real history to measure up against, so it provides a good training ground for futurist who don't want to wait 20 or 30 years to get feedback on their thinking.

Given all this, I have two suggestions. One, this indicates...
Epistemic vs. Instrumental Rationality: Case of the Leaky Agent
by Wei Dai — 2009-05-07T23:09:40.070Z

Suppose you hire a real-estate agent to sell your house. You have to leave town so you give him the authority to negotiate with buyers on your behalf. The agent is honest and hard working. He'll work as hard to get a good price for your house as if he's selling his own house. But unfortunately, he's not very good at keeping secrets. He wants to know what is the minimum amount you're willing to sell the house for so he can do the negotiations for you. But you know that if you answer him truthfully, he's liable to leak that information to buyers, giving them a bargaining advantage and driving down the expected closing price. What should you do? Presumably most of you in this situation would give the agent a figure that's higher than the actual minimum. (How much higher involves optimizing a tradeoff between the extra money you get if the house sells, versus the probability that you can't find a buyer at the higher fictional minimum.)

Now here's the kicker: that agent is actually your future self. Would you tell yourself a lie, if you could believe it (perhaps with the help of future memory modification technologies), and if you could profit from it?

Edit: Some commenters have pointed out that this change in "minimum acceptable price" may not be exactly a lie. I should have made the example a bit clearer. Let's say if you fail to sell the house by a certain date, it will be reposessed by the bank, so the minimum acceptable price is the amount left on your mortgage, since you're better off selling the house for any amount above that than not selling it. But if buyers know that, they can just offer you slightly above the minimum acceptable price. It will help you get a better bargain if you can make yourself believe that the amount left on your mortgage is higher than it really is. This should be unambigously a lie.
The First Koan: Drinking the Hot Iron Ball
by Annoyance — 2009-05-07T17:41:43.586Z

In the traditions of Zen in which koans are common teaching tools, it is common to use a particular story as a novice's first koan. It's the story of Joshu's Dog.

A monk asked Joshu, a Chinese Zen master: `Has a dog Buddha-nature or not?'

Joshu answered: `Mu.' [Mu is the negative symbol in Chinese, meaning `No-thing' or `Nay'.]

What does this koan mean? How can we find out for ourselves?

It is important to remember certain things: Firstly, koans are not meant to be puzzles, riddles, or intellectual games. They are examples, illustrations of the state of mind that the student is expected to internalize. Secondly, they often appear paradoxical.

Paradox is a pointer telling you to look beyond it. If paradoxes bother you, that betrays your deep desire for absolutes. The relativist treats a paradox merely as interesting, perhaps amusing or even -- dreadful thought -- educational.

Thirdly, the purpose of Zen teaching isn't to acquire new conceptual baggage, but to eliminate it; not to generate Enlightenment, but to remove the false beliefs that preventing us from recognizing what we already possess. Shedding error is the point, not learning something new.

Take a look at Mumon's commentary for this koan:

To realize Zen one has to pass through the barrier of the patriachs. Enlightenment always comes after the road of thinking is blocked. If you do not pass the barrier of the patriachs or if your thinking road is not blocked, whatever you think, whatever you do, is like a tangling ghost. You may ask: What is a barrier of a patriach? This one word, Mu, is it.

This is the barrier of Zen. If you pass through it you will see Joshu face to face. Then you can work hand in hand with the whole line of patriachs. Is this not a pleasant thing to do?

If you want to pass this barrier, you must work through every bone in your body, through ever pore in your skin, filled with this question: What is Mu? and carry it day and night. Do not believe it is the common negative symbol meaning nothing. It is not nothingness, the opposite of existence. If you really want to pass this barrier, you should feel like drinking a hot iron ball that you can neither swallor nor spit out.

Then your previous lesser knowledge disappears. As a fruit ripening in season, your subjectivity and objectivity naturally become one. It is like a dumb man who

...
Rationality is winning - or is it?
by taw — 2009-05-07T14:51:52.763Z

I feel a bit silly writing an post about connotations on a rationalist website, but I really love the quote "Rationality (is/is not) winning". I see a few different ways of interpreting it:

"Rationality is winning" - results are more important than following a particular ritual of cognition. If something doesn't work, abandon it no matter how "rational" is seems.

"Rationality is not winning" - exploration is much more fun than just mindlessly going toward some goal.

"Rationality is winning" - what matters is how good you are at reaching socially accepted criteria of "success" - I don't like this connotation at all.

And I can think of a few others...

I wonder, the way human brain works, is it common for there to be thoughts that are much better expressed with a short sentence full of ambiguous connotations, that by long and accurate explanations? Give me your favourite ambiguous quotes!
On the Fence? Major in CS
by talisman — 2009-05-07T04:26:14.714Z

I talk to many ABDs in math, physics, engineering, economics, and various other technical fields.

I work with exceptional people from all those backgrounds.

I would like to unreservedly say to any collegians out there, whether choosing an undergrad major or considering fields of study for grad school: if you know you want a technical major but you're not sure which, choose Computer Science.

Unless you're extremely talented and motivated, relative to your extremely talented and motivated peers, you probably aren't going to make a career in academia even if you want to. And if you want a technical major but you're not sure which, you shouldn't want to! Academia is a huge drag in many ways. When a math ABD starts telling me about how she really likes her work but is sick of the slow pace and the fact that only six people in the world understand her work, I get to take a nice minute alone with my thoughts: I've heard it over and over again, in the same words and the same weary, beaten-down tone. You shouldn't be considering a career in academia unless you're passionately in love with your field, unless you think about it in the shower and over lunch and as you drift off to sleep, unless the problem sets are a weekly joy. A lesser love will abandon you and leave you stranded and heartbroken, four years into grad school.

What's so great about CS, then? Isn't it just a bunch of glorified not-real-math and hundreds of hours of grimy debugging?

Let's start with several significant, but peripheral, reasons:

CS majors learn to really program. There's an ocean of difference between the power of a decent, desultory programmer and that of a real programmer. If you're not a programmer, the power of real programmers to create good stuff borders on magic.

Not least among the good stuff is time. It's disgraceful, the amount of human effort that goes into work that could be done by a Perl one-liner.

CS majors learn to be at home with the guts of computers. This seems to come in handy in a hundred little ways.

CS majors are significantly more likely than other technical majors to get involved in startups, which are one of the best ways around to create wealth.

While having abstract and intellectual sides, the good kind of CS is strongly tied to the practical.

CS people can do fun side projects. I've neve

...
Beware Trivial Inconveniences
by Scott Alexander — 2009-05-06T22:04:02.354Z

The Great Firewall of China. A massive system of centralized censorship purging the Chinese version of the Internet of all potentially subversive content. Generally agreed to be a great technical achievement and political success even by the vast majority of people who find it morally abhorrent.
I spent a few days in China. I got around it at the Internet cafe by using a free online proxy. Actual Chinese people have dozens of ways of getting around it with a minimum of technical knowledge or just the ability to read some instructions.
The Chinese government isn't losing any sleep over this (although they also don't lose any sleep over murdering political dissidents, so maybe they're just very sound sleepers). Their theory is that by making it a little inconvenient and time-consuming to view subversive sites, they will discourage casual exploration. No one will bother to circumvent it unless they already seriously distrust the Chinese government and are specifically looking for foreign websites, and these people probably know what the foreign websites are going to say anyway.
Think about this for a second. The human longing for freedom of information is a terrible and wonderful thing. It delineates a pivotal difference between mental emancipation and slavery. It has launched protests, rebellions, and revolutions. Thousands have devoted their lives to it, thousands of others have even died for it. And it can be stopped dead in its tracks by requiring people to search for "how to set up proxy" before viewing their anti-government website.

I was reminded of this recently by Eliezer's Less Wrong Progress Report. He mentioned how surprised he was that so many people were posting so much stuff on Less Wrong, when very few people had ever taken advantage of Overcoming Bias' policy of accepting contributions if you emailed them to a moderator and the moderator approved. Apparently all us folk brimming with ideas for posts didn't want to deal with the aggravation.
Okay, in my case at least it was a bit more than that. There's a sense of going out on a limb and drawing attention to yourself, of arrogantly claiming some sort of equivalence to Robin Hanson and Eliezer Yudkowsky. But it's still interesting that this potential embarrassment and awkwardness was enough to keep the several dozen people who have blogged on here so far from sending that "I have something I'd like to...
Hardened Problems Make Brittle Models
by cousin_it — 2009-05-06T18:31:28.077Z

Consider a simple decision problem: you arrange a date with someone, you arrive on time, your partner isn't there. How long do you wait before giving up?

Humans naturally respond to this problem by acting outside the box. Wait a little then send a text message. If that option is unavailable, pluck a reasonable waiting time from cultural context, e.g. 15 minutes. If that option is unavailable...

Wait, what?

The toy problem was initially supposed to help us improve ourselves - to serve as a reasonable model of something in the real world. The natural human solution seemed too messy and unformalizable so we progressively remove nuances to make the model more extreme. We introduce Omegas, billions of lives at stake, total informational isolation, perfect predictors, finally arriving at some sadistic contraption that any normal human would run away from. But did the model stay useful and instructive? Or did we lose important detail along the way?

Many physical models, like gravity, have the nice property of stably approximating reality. Perturbing the positions of planets by one millimeter doesn't explode the Solar System the next second. Unfortunately, many of the models we're discussing here don't have this property. The worst offender yet seems to be Eliezer's "True PD" which requires the whole package of hostile psychopathic AIs, nuclear-scale payoffs and informational isolation; any natural out-of-the-box solution like giving the damn thing some paperclips or bargaining with it would ruin the game. The same pattern has recurred in discussions of Newcomb's Problem where people have stated that any miniscule amount of introspection into Omega makes the problem "no longer Newcomb's". That naturally led to more ridiculous use of superpowers, like Alicorn's bead jar game where (AFAIU) the mention of Omega is only required to enforce a certain assumption about its thought mechanism that's wildly unrealistic for a human.

Artificially hardened logic problems make brittle models of reality.

So I'm making a modest proposal. If you invent an interesting decision problem, please, first model it as a parlor game between normal people with stakes of around ten dollars. If the attempt fails, you have acquired a bit of information about your concoction; don't ignore it outright.
Wiki.lesswrong.com Is Live
by Eliezer Yudkowsky — 2009-05-06T05:17:02.471Z

http://wiki.lesswrong.com/ is now live, for all our Wiki needs. The previous Wikia wiki has been imported. Knock yourself out on linking there from comments or blog posts (yes, you will have to link manually, there is no CamelCase convention yet on the blog and comments).

See here for proposed wiki usage guidelines - note that these do not yet seem to appear in the Wiki itself, hint hint.
No Universal Probability Space
by Gordon Seidoh Worley — 2009-05-06T02:58:06.165Z

This afternoon I heard a news story about a middle eastern country where one person said of the defenses for a stockpile of nuclear weapons, "even if there is only a 1% probability of the defenses failing, we should do more to strengthen them given the consequences of their failure". I have nothing against this person's reasoning, but I do have an issue with where that 1% figure came from.

The statement above and others like it share a common problem: they are phrased such that it's unclear over what probability space the measure was taken. In fact, many journalist and other people don't seem especially concerned by this. Even some commenters on Less Wrong give little indication of the probability space over which they give a probability measure of an event, and nobody calls them on it. So what is this probability space they are giving probability measurements over?

If I'm in a generous mood, I might give the person presenting such a statement the benefit of the doubt and suppose they were unintentionally ambiguous. On the defenses of the nuclear weapon stockpile, the person might have meant to say "there is only a 1% probability of the defenses failing over all attacks", as in "in 1 attack out of every 100 we should expect the defenses to fail". But given both my experiences with how people treat probability and my knowledge of naive reasoning about probability, I am dubious of my own generosity. Rather, I suspect that many people act as though there were a universal probability space over which they may measure the probability of any event.

To illustrate the issue, consider the probability that a fair coins comes up heads. We typically say that there is a 1/2 chance of heads, but what we are implicitly saying is that given a probability measure P on the measurable space ({heads, tails}, {{}, {heads}, {tails}, {heads, tails}}), P({heads}) = P({tails}) = 1/2 and P({}) = 0 and P({heads, tails}) = 1. But if we look at the issue of a coin coming up heads from a wider angle, we could interpret it as "what is the probability of some particular coin sitting heads-up over the span of all time", which is another question all together. What this is asking is "what is the probability of the event that this coin sits heads-up over the universal probability space", i.e. the probability space of all events th...