trentmkelly/LessWrong-43k huggingface dataset:
Value Uncertainty and the Singleton Scenario
by Wei Dai — 2010-01-24T05:03:44.756Z

In January of last year, Nick Bostrom wrote a post on Overcoming Bias about his and Toby Ord’s proposed method of handling moral uncertainty. To abstract away a bit from their specific proposal, the general approach was to convert a problem involving moral uncertainty into a game of negotiation, with each player’s bargaining power determined by one’s confidence in the moral philosophy represented by that player.

Robin Hanson suggested in his comments to Nick’s post that moral uncertainty should be handled the same way we're supposed to handle ordinary uncertainty, by using standard decision theory (i.e., expected utility maximization). Nick’s reply was that many ethical systems don’t fit into the standard decision theory framework, so it’s hard to see how to combine them that way.

In this post, I suggest we look into the seemingly easier problem of value uncertainty, in which we fix a consequentialist ethical system, and just try to deal with uncertainty about values (i.e., utility function). Value uncertainty can be considered a special case of moral uncertainty in which there is no apparent obstacle to applying Robin’s suggestion. I’ll consider a specific example of a decision problem involving value uncertainty, and work out how Nick and Toby’s negotiation approach differs in its treatment of the problem from standard decision theory. Besides showing the difference in the approaches, I think the specific problem is also quite important in its own right.

The problem I want to consider is, suppose we believe that a singleton scenario is very unlikely, but may have very high utility if it were realized, should we focus most of our attention and effort into trying to increase its probability and/or improve its outcome? The main issue here is (putting aside uncertainty about what will happen after a singleton scenario is realized) uncertainty about how much we value what is likely to happen.

Let’s say there is a 1% chance that a singleton scenario does occur, and conditional on it, you will have expected utility that is equivalent to a 1 in 5 billion chance of controlling the entire universe. If a singleton scenario does not occur, you will have a 1/5 billionth share of the resources of the solar system, and the rest of the universe will be taken over by beings like the ones described in Robin&r...
Lesswrong UK planning thread
by  — 2010-01-24T00:33:03.381Z

A few of us got together in the pub after the friendly AI meet and agreed we should have a meetup for those of us familiar with lesswrong/bostrom etc. This is a post for discussion of when/where.

Venue: London still seems the nexus. I might be able to be convinced to go to Oxford. A starbucks type place is okay, although it'd be nice to have a white board or other presentation systems.

Date/Time: Weekends is fine with me, and I suspect most people. Julian suggested after the next UKTA meeting. Will that be April at the humanityplus thing? It would depend whether we are still mentally fresh after it, and whether any of our group are attending the dinner after.

Activities: I think it would be a good idea to have some structure or topics to discuss, so that we don't just fall back into the "what do you do" types of discussion too much. Maybe mini presentations.

My duo of current interests

1)Evidence for intelligence explosion: I don't want to rehash what we know already, but I would like to try and figure out what experiments we can do (safely) or proofs we can make to increase or decrease our belief that it will occur. This is more of a brainstorming session.

2) The nature of the human brain: Specifically it doesn't appear to have a goal (in the decision theory sense) built in, although it can become a goal optimizer to a greater or lesser extent. How might it do this? As we aren't neuroscientists, a more fruitful question might be what the skeleton of such a computer system that can do this might be look like, even if we can't fill in all the interesting details. I'd discuss this with regards to akrasia, neural enhancement, volition extraction and non-exploding AI scenarios. I can probably pontificate on this for a while, if I prepare myself.

I think Ciphergoth wanted to talk about consequentialist ethics.

Shout in the comments if you have a topic you'd like to discuss, or would rather not discuss.

Perhaps we should also look at the multiplicity of AI bias, where people seem to naturally assume there will be multiple AIs even when talking about super intelligent singularity scenarios (many questions had this property at the meeting). I suspect that this could be countered by reading a Fire upon the Deep somewhat.
Lists of cognitive biases, common misconceptions, and fallacies
by Kevin — 2010-01-23T22:47:07.845Z

http://en.wikipedia.org/wiki/List_of_cognitive_biases

http://en.wikipedia.org/wiki/List_of_common_misconceptions

http://en.wikipedia.org/wiki/List_of_fallacies

http://en.wikipedia.org/wiki/List_of_memory_biases

I know the trend here is against simple links as top-level posts, but I think these links are powerful enough to stand on their own. I would be very surprised if most people here have already read all of these links. Also, it seems like a good sign that this somehow got modded up to 1 while sitting in my drafts folder.

Thanks to Lone Gunman for the links.
Far & Near / Runaway Trolleys / The Proximity Of (Fat) Strangers
by botogol — 2010-01-23T22:13:05.437Z

I went to the Royal Institute last week to hear the laconic and dismissive Dr Guy Kahane on whether we are 'Biologically Moral'

[His message: Neurological evidence suggests - somewhat alarmingly - that our moral and ethical decisions may be no more than post-hoc rationalisations of purely emotional, instinctive reactions. However, we should not panic because this is early days in neuroscience, and the correct interpretation of brain-scans is uncertain: scientist find the pattern, and the explanation, they expect to find]
To illustrate his talk Kahane used one of those moral dilemmas which are rarely encountered in real life but which are fascinating to philosophers: the familiar Trolley Problem

A picture is worth a 1,000 words and rather then spell it out Kahane flashed up two nice cartoons:

- the first shows the anxious philosophical protagonist at the railway junction, runaway trolley approaching, pondering the lever that moves the points

- the second shows our hapless philosopher now on a bridge poised behind an unsuspecting fat stranger who is neither alert enough for his attention to have been caught by the runaway trolley bearing down on the small party of railway workers, nor sufficiently familiar with the philosophical domain to appreciate the mortal danger that he is in himself. Irony, oh Irony: Philosophy, thy name is Drama.
So far so humdrum; but... surprise! : in the front row of the audience that evening, wedged into the seat next to me, and the seat next to that, only a couple of metres from Dr Kahane and dressed in identical clothes as when the cartoon was made was the fat stranger himself.

You had to feel for Dr Kahane, but with no evident embarrassment he declined to acknowledge the unexpected attendee and ploughed gamely on with an earnest discussion of the morals, ethics and practicalities of heaving the poor man off a bridge and under the wheels of an oncoming philosophical trope. I had to smile
And the Trolley Problem is always good for a lively debate so, inevitably, in the Q&A, we came back to it all over again, I felt more uncomfortable by now as members of the audience discussed at length the pros and cons of the fat man's sorry and undeserved demise, none of them acknowledging his presence amongst us.

The fat man was, you might say, the elephant in the room.
My discovery
The reason the Trolley Probl...
Privileged Snuff
by rwallace — 2010-01-22T05:38:36.259Z

So one is asked, "What is your probability estimate that the LHC will destroy the world?"
Leaving aside the issue of calling brown numbers probabilities, there is a more subtle rhetorical trap at work here.
If one makes up a small number, say one in a million, the answer will be, "Could you make a million such statements and not be wrong even once?" (Of course this is a misleading image -- doing anything a million times in a row would make you tired and distracted enough to make trivial mistakes. At some level we know this argument is misleading, because nobody calls the non-buyer of lottery tickets irrational for assigning an even lower probability to a win.)
If one makes up a larger number, say one in a thousand, then one is considered a bad person for wanting to take even one chance in a thousand of destroying the world.

The fallacy here is http://wiki.lesswrong.com/wiki/Privileging_the_hypothesis

To see why, try inverting the statement: what is your probability estimate that canceling the LHC will result in the destruction of the world?
Unlikely? Well I agree, it is unlikely. But I can think of plausible ways it could be true. New discoveries in physics could be the key to breakthroughs in areas like renewable energy or interstellar travel -- breakthroughs that might just make the difference between a universe ultimately filled with intelligent life, and a future of might have been. History shows, after all, that key technologies often arise from unexpected lines of research. I certainly would not be confident in assigning a million to one odds against the LHC making that difference.
Conversely, we know the LHC is not going to destroy the world, because nature has been banging particles together at much higher energy levels for billions of years. If that sufficed to destroy the world, it would already have happened, and any people you might happen to meet from time to time would be figments of a deranged imagination.
The hypothesis being privileged in even asking the original question, is not a harmless one like the tooth fairy. It is the hypothesis that snuffing out progress, extinguishing futures that might have been, is the safe option. It is not really a forgivable mistake, for we already know otherwise -- death is the default, not just for individuals, but for nations, civilizations, species and worlds. It could, however, be the ultimate mistake, the ...
Costs to (potentially) eternal life
by bgrah449 — 2010-01-21T21:46:31.316Z

Imagine Omega came to you and said, "Cryonics will work; it will be possible for you to be resurrected and have the choice between a simulation and a new healthy body, and I can guarantee you live for at least 100,000 years after that. However, for reasons I won't divulge, your surviving to experience this is wholly contingent upon you killing the next three people you see. I can also tell you that the next three people you see, should you fail to kill them, will die childless and will never sign up for cryonics. There is a knife on the ground behind you."
You turn around and see someone. She says, "Wait! You shouldn't kill me because ... "
What does she say that convinces you?

[Cryonics] takes the mostly correct idea "life is good, death is bad" to such an extreme that it does violence to other valuable parts of our humanity (sorry, but I can't be more specific).

That's a quote from a comment in a post about cryonics. "I can't be more specific" is not doing this comment any favors, and overall the comment was rebutted pretty well. But I did try to imagine other these other valuable parts, and I realized something that remains unresolved for me.
Guaranteed death places a limit on the value of my life to myself. Parents shield children with their bodies; Casey Jones happens more often. People run into burning buildings more often. (Suicide bombers happen more often, too, I realize.)
I think this is a valuable part of humanity, and I think that an extreme "life is good, death is bad" view does do violence to it. You can argue we should effect a world that makes this willingness unnecessary, and I'll support that; but separate from making the willingness useless, eliminating that willingness does violence to our humanity. You can argue that our humanity is overrated and there's something better over the horizon, i.e. the cost is worth it.
But the incentives for saving 1+X many lives at the cost of your own just got lessened. How do you put a price on heaven? orthonormal suggests that we should rely on human irrationality here to keep us moral, that thankfully we are too stupid and slow to actually change the decisions we make after recognizing the expected value of our options has changed, despite the opportunity cost of these decisions growing considerably. I think this a) underestimates humans' ability to react to incentives and b) underestimates the reward the ...
Easy Predictor Tests
by MrHen — 2010-01-21T18:40:21.079Z

A fun game you can play on LessWrong is to stop just as you are about to click "comment" and make a prediction for how much karma your comment will receive within the next week. This will provide some quick feedback about how well your karma predictors are working. This exercise will let you know if something is broken. A simpler version is to pick from these three distinct outcomes: Positive karma, 0 karma, negative karma.

What other predictors are this easy to test? Likely candidates match one or more of the following criteria:

Something we do on a regular (probably daily) basis

An action that has a clear starting point

Produces quick, quantifiable feedback (e.g. karma, which is a basic number)

An action that is extremely malleable so we can take our feedback, make quick adjustments, and run through the whole process again

An ulterior goal other than merely testing our predictors so we don't get bored (e.g. commenting at LessWrong, which offers communication and learning as ulterior goals)

Something with a "sticky" history so we can get a good glimpse of our progress over time

A more difficult challenge is predicting karma on your top level posts. My predictors in this area tend to be way off the mark. For this post, my guess is between +4 and +20. Reasoning: I don't see how it could get over 20 unless it gets promoted and the concept surprises readers; 4 seems like a solid guess for "interesting, uncontroversial, but not groundbreaking."

Update: As of January 29, this post is at +10.
Winning the Unwinnable
by JRMayne — 2010-01-21T03:01:48.371Z

A few years back, I sent a note to some friends and other smart people. It said, approximately:

"I am trying to raise about $42 million. I have an expected payoff of about 20% in a week. It's not guaranteed; it might be more, or there could be a loss.

"Now, you may be saying to yourself, 'Mayne's got a system to win the lottery or something.'

"Well, it's not, 'something.' I have a system to win the lottery."

At which point I explained the system.

Now, LW is not particularly fond of the lottery. From a social policy/political point of view, I'm not disagreeing.

I specifically represent to you that I do have a system, I use a part of the system, and so far, I've lost about $350 with the system. If you'd loan me your $42 million when I ask for it, though, I'd have not only a high positive expectation, but a much smoother payout matrix, if I could solve the practical problem of buying as many tickets as I want. Which is 41.4 million tickets.

No, I'm not just saying I'll buy all the tickets and therefore guarantee the jackpot and therefore I win. I'm saying my expectation per dollar invested is substantially positive.

Skeptical? Why?

Not only do I have a system, but anyone with math skills and any knowledge of how the lottery works should agree. While it's always easy and obvious to see why other people should think the same as you, the fact that of a large number of math folks, zero here have defended playing the lottery as a cash-plus position, is a sign that people have foregone thinking and simply rejected the lottery outright. Or that I'm an idiot.

Before you get to the post payoff, what's your attitude toward this now? Lottery win is impossible? If you think it's impossible (or close enough to it), why? Unlikely? Probable? Would it change your mind if my self-assessment was that I was likely in the bottom quartile as far as current math skills on LW? Would it alter your probabilities if I told you that I am highly confident that if the practicality of buying 41.4 million tickets is worked out, that using my system to play the lottery is the best investment I know of?

OK, here it is: How to win the lottery.

I'm going to make this relatively brief, but it's really fairly simple. In California, we have a fairly classic lottery setup although the payout rates are a little worse than most states. The immediate payoff of the jackpot is about (carryover + 25% of tic...
That Magical Click
by Eliezer Yudkowsky — 2010-01-20T16:35:01.715Z

Followup to: Normal Cryonics

Yesterday I spoke of that cryonics gathering I recently attended, where travel by young cryonicists was fully subsidized, leading to extremely different demographics from conventions of self-funded activists. 34% female, half of those in couples, many couples with kids - THAT HAD BEEN SIGNED UP FOR CRYONICS FROM BIRTH LIKE A GODDAMNED SANE CIVILIZATION WOULD REQUIRE - 25% computer industry, 25% scientists, 15% entertainment industry at a rough estimate, and in most ways seeming (for smart people) pretty damned normal.

Except for one thing.

During one conversation, I said something about there being no magic in our universe.

And an ordinary-seeming woman responded, "But there are still lots of things science doesn't understand, right?"

Sigh. We all know how this conversation is going to go, right?

So I wearily replied with my usual, "If I'm ignorant about a phenomenon, that is a fact about my state of mind, not a fact about the phenomenon itself; a blank map does not correspond to a blank territory -"

"Oh," she interrupted excitedly, "so the concept of 'magic' isn't even consistent, then!"

Click.

She got it, just like that.

This was someone else's description of how she got involved in cryonics, as best I can remember it, and it was pretty much typical for the younger generation:

"When I was a very young girl, I was watching TV, and I saw something about cryonics, and it made sense to me - I didn't want to die - so I asked my mother about it. She was very dismissive, but tried to explain what I'd seen; and we talked about some of the other things that can happen to you after you die, like burial or cremation, and it seemed to me like cryonics was better than that. So my mother laughed and said that if I still felt that way when I was older, she wouldn't object. Later, when I was older and signing up for cryonics, she objected."

Click.

It's... kinda frustrating, actually.

There are manifold bad objections to cryonics that can be raised and countered, but the core logic really is simple enough that there's nothing implausible about getting it when you're eight years old (eleven years old, in my case).

Freezing damage? I could go on about modern cryoprotectants and how you can see under a microscope that the tissue is in great shape, and there are experiments underway to see if they can get spontaneous brai...
London meetup: "The Friendly AI Problem" 
by Roko — 2010-01-19T23:35:47.131Z

To all LessWrongers in London/the south of England: On Saturday the 23rd - this Saturday at 2pm - I will be speaking about Friendly AI at the UK transhumanist group, at

Room 416, 4th floor (via main lift), Birkbeck College, Torrington Square, London (Map)

For those who have read most of the material on the web about FAI theory, e.g. the sequences here and the material of Bostrom and Omohundro, you won't hear much new, but you will probably meet a lot of people who could do with some strong rationalists to speak to. For those who haven't read most of the material about Friendly AI, I'll try to make it a superlatively informative talk. The Facebook event has 30 confirmed guests, and the London Futurists' organizer (distinct from UK h+, the hosts), just emailed me and said:

"Looking forward to meeting you and asking you questions on Saturday! I think your going to get quite a few as there is a lot of interest from my group about your presentation - positive and negative. You might get a very skeptical geezer coming along so be prepared for some tough questions from people who know their stuff."

So the event promises to be a fairly large gathering of rationalists, futurists and transhumanists with a high probability of passionate debate about FAI, followed by a trip to the pub for further discussion and drinks.
Normal Cryonics
by Eliezer Yudkowsky — 2010-01-19T19:08:48.301Z

I recently attended a small gathering whose purpose was to let young people signed up for cryonics meet older people signed up for cryonics - a matter of some concern to the old guard, for obvious reasons.

The young cryonicists' travel was subsidized. I suspect this led to a greatly different selection filter than usually prevails at conferences of what Robin Hanson would call "contrarians". At an ordinary conference of transhumanists - or libertarians, or atheists - you get activists who want to meet their own kind, strongly enough to pay conference fees and travel expenses. This conference was just young people who took the action of signing up for cryonics, and who were willing to spend a couple of paid days in Florida meeting older cryonicists.

The gathering was 34% female, around half of whom were single, and a few kids. This may sound normal enough, unless you've been to a lot of contrarian-cluster conferences, in which case you just spit coffee all over your computer screen and shouted "WHAT?" I did sometimes hear "my husband persuaded me to sign up", but no more frequently than "I pursuaded my husband to sign up". Around 25% of the people present were from the computer world, 25% from science, and 15% were doing something in music or entertainment - with possible overlap, since I'm working from a show of hands.

I was expecting there to be some nutcases in that room, people who'd signed up for cryonics for just the same reason they subscribed to homeopathy or astrology, i.e., that it sounded cool. None of the younger cryonicists showed any sign of it. There were a couple of older cryonicists who'd gone strange, but none of the young ones that I saw. Only three hands went up that did not identify as atheist/agnostic, and I think those also might have all been old cryonicists. (This is surprising enough to be worth explaining, considering the base rate of insanity versus sanity. Maybe if you're into woo, there is so much more woo that is better optimized for being woo, that no one into woo would give cryonics a second glance.)

The part about actually signing up may also be key - that's probably a ten-to-one or worse filter among people who "get" cryonics. (I put to Bill Faloon of the old guard that probably twice as many people had died while planning to sign up for cryonics eventually,...