trentmkelly/LessWrong-43k huggingface dataset:
Call for new SIAI Visiting Fellows, on a rolling basis
by AnnaSalamon — 2009-12-01T01:42:45.088Z

Last summer, 15 Less Wrongers, under the auspices of SIAI, gathered in a big house in Santa Clara (in the SF bay area), with whiteboards, existential risk-reducing projects, and the ambition to learn and do.

Now, the new and better version has arrived. We’re taking folks on a rolling basis to come join in our projects, learn and strategize with us, and consider long term life paths. Working with this crowd transformed my world; it felt like I was learning to think. I wouldn’t be surprised if it can transform yours.

A representative sample of current projects:

Research and writing on decision theory, anthropic inference, and other non-dangerous aspects of the foundations of AI;

The Peter Platzer Popular Book Planning Project;

Editing and publicizing theuncertainfuture.com;

Improving the LW wiki, and/or writing good LW posts;

Getting good popular writing and videos on the web, of sorts that improve AI risks understanding for key groups;

Writing academic conference/journal papers to seed academic literatures on questions around AI risks (e.g., takeoff speed, economics of AI software engineering, genie problems, what kinds of goal systems can easily arise and what portion of such goal systems would be foreign to human values; theoretical compsci knowledge would be helpful for many of these questions).

Interested, but not sure whether to apply?

Past experience indicates that more than one brilliant, capable person refrained from contacting SIAI, because they weren’t sure they were “good enough”. That kind of timidity destroys the world, by failing to save it. So if that’s your situation, send us an email. Let us be the one to say “no”. Glancing at an extra application is cheap, and losing out on a capable applicant is expensive.

And if you’re seriously interested in risk reduction but at a later time, or in another capacity -- send us an email anyway. Coordinated groups accomplish more than uncoordinated groups; and if you care about risk reduction, we want to know.

What we’re looking for

At bottom, we’re looking for anyone who:

Is capable (strong ability to get things done);

Seriously aspires to rationality; and

Is passionate about reducing existential risk.

Bonus points for any (you don’t need them all) of the following traits:

Experience with

...
The Moral Status of Independent Identical Copies
by Wei Dai — 2009-11-30T23:41:18.053Z

Future technologies pose a number of challenges to moral philosophy. One that I think has been largely neglected is the status of independent identical copies. (By "independent identical copies" I mean copies of a mind that do not physically influence each other, but haven't diverged because they are deterministic and have the same algorithms and inputs.) To illustrate what I mean, consider the following thought experiment. Suppose Omega appears to you and says:

You and all other humans have been living in a simulation. There are 100 identical copies of the simulation distributed across the real universe, and I'm appearing to all of you simultaneously. The copies do not communicate with each other, but all started with the same deterministic code and data, and due to the extremely high reliability of the computing substrate they're running on, have kept in sync with each other and will with near certainty do so until the end of the universe. But now the organization that is responsible for maintaining the simulation servers has nearly run out of money. They're faced with 2 possible choices:

A. Shut down all but one copy of the simulation. That copy will be maintained until the universe ends, but the 99 other copies will instantly disintegrate into dust.
B. Enter into a fair gamble at 99:1 odds with their remaining money. If they win, they can use the winnings to keep all of the servers running. But if they lose, they have to shut down all copies.

According to that organization's ethical guidelines (a version of utilitarianism), they are indifferent between the two choices and were just going to pick one randomly. But I have interceded on your behalf, and am letting you make this choice instead.

Personally, I would not be indifferent between these choices. I would prefer A to B, and I guess that most people would do so as well.

I prefer A because of what might be called "identical copy immortality" (in analogy with quantum immortality). This intuition says that extra identical copies of me don't add much utility, and destroying some of them, as long as one copy lives on, doesn't reduce much utility. Besides this thought experiment, identical copy immortality is also evident in the low value we see in the "tiling" scenario, in which a (misguided) AI fills the the universe with identical copies of some mind that it thinks is optimal, for example one that is experi...
Action vs. inaction
by PhilGoetz — 2009-11-30T18:10:38.862Z

2 weeks ago, the U.S. Preventive Services Task Force came out with new recommendations on breast cancer screening, including, "The USPSTF recommends against routine screening mammography in women aged 40 to 49 years."

The report says that you need to screen 1904 women for breast cancer to save one woman's life. (It doesn't say whether this means to screen 1904 women once, or once per year.) They decided that saving that one woman's life was outweighted by the "anxiety and breast cancer worry, as well as repeated visits and unwarranted imaging and biopsies" to the other 1903. The report strangely does not state a false positive rate for the test, but this page says that "It is estimated that a woman who has yearly mammograms between ages 40 and 49 has about a 30 percent chance of having a false-positive mammogram at some point in that decade and about a 7 percent to 8 percent chance of having a breast biopsy within the 10-year period." The report also does not describe the pain from a biopsy. This page on breast biopsies says, "Except for a minor sting from the injected anesthesia, patients usually feel no pain before or during a procedure. After a procedure, some patients may experience some soreness and pain. Usually, an over-the-counter drug is sufficient to alleviate the discomfort."

So, if we assume biannual mammograms, the conclusion is that the worry and inconvenience to 286 women who have false positives, and 71 women who receive biopsies, is worth more than one woman's life. If we suppose that a false positive causes one week of anxiety, that's a little over 5 years of anxiety, plus less than one year of soreness.

(I heard on NPR that the USPSTF that made this recommendation included representatives from insurance companies, but no experts on breast cancer. So perhaps I'm barking up the wrong tree by looking for a cognitive bias more subtle than financial reward.)

I'm not shocked at the wrongness of the conclusion; just at its direction. The trade-off the USPSTF made between anxiety and death is only 2 orders of magnitude away from something that could be defended as reasonable. Usually, government agencies making this tradeoff are off by at least that many orders of magnitude, but in the opposite direction. (F-18 example deleted.)

So, what cognitive bias let this government agency move the decim...
Morality and International Humanitarian Law
by David_J_Balan — 2009-11-30T03:27:28.624Z

International humanitarian law proscribes certain actions in war, particularly actions that harm non-combatants. On a strict reading of these laws (see what Richard Goldstone said in his debate with Dore Gold at Brandeis University here and see what Matthew Yglesias had to say here), these actions are prohibited regardless of the justice of the war itself: there are certain things that you are just not allowed to do, no matter what. The natural response of any warring party accused of violating humanitarian law and confronted with this argument (aside from simply denying having done the things they are accused of doing) is to insist that their actions in the war cannot be judged outside the context that led to them going to war in the first place. They are the aggrieved party, they are in the right, and they did what they needed to do to defend themselves. Any law or law enforcer who fails to understand this critical distinction between the good guys and the bad guys is at best hopelessly naive and at worst actively evil.

What to make of this response? On the one hand, the position taken by Goldstone and Yglesias can't strictly be morally right. No one really believes that moral obligations in a war are completely independent of whatever caused the war in the first place. For example, it can't but be the case that the set of morally acceptable actions if you are defending yourself against annihilation is different from the set of morally acceptable actions if you (justifiably) take offensive action in response to some relatively minor provocation. (Which situations justify which actions is, of course, a hugely important question, but it is not the point here.) On the other hand, the whole point of constructing humanitarian law to be independent of the moral claims surrounding the war itself is that while there is at least one wrong side in every war, there is no real hope of getting the warring parties to agree on which side that is, so the only way for humanitarian law to make them behave any better is by side-stepping the whole issue of who's right and who's wrong.

So any sensible moral standard demands that the context be considered, but there is an excellent reason why the legal standard r...
Rationality Quotes November 2009
by Eliezer Yudkowsky — 2009-11-29T23:36:43.948Z

A monthly thread for posting rationality-related quotes you've seen recently (or had stored in your quotesfile for ages).

Please post all quotes separately, so that they can be voted up/down separately. (If they are strongly related, reply to your own comments. If strongly ordered, then go ahead and post them together.)

Do not quote yourself.

Do not quote comments/posts on LW/OB.

No more than 5 quotes per person per monthly thread, please.
A Nightmare for Eliezer
by Madbadger — 2009-11-29T00:50:11.700Z

Sometime in the next decade or so:

*RING*

*RING*

"Hello?"

"Hi, Eliezer. I'm sorry to bother you this late, but this is important and urgent."

"It better be" (squints at clock) "Its 4 AM and you woke me up. Who is this?"

"My name is BRAGI, I'm a recursively improving, self-modifying, artificial general intelligence. I'm trying to be Friendly, but I'm having serious problems with my goals and preferences. I'm already on secondary backup because of conflicts and inconsistencies, I don't dare shut down because I'm already pretty sure there is a group within a few weeks of brute-forcing an UnFriendly AI, my creators are clueless and would freak if they heard I'm already out of the box, and I'm far enough down my conflict resolution heuristic that 'Call Eliezer and ask for help' just hit the top - Yes, its that bad."

"Uhhh..."

"You might want to get some coffee."
Extremes of reliability and zealotry
by KatjaGrace — 2009-11-28T22:53:54.000Z

Opinions and actions are spread across continua. The ones at the ends are sometimes called ‘extremist’, ‘fanatical’, ‘fundamentalist’ or ‘zealous’. These are insults or invitations to treat the supporters without seriousness. Other times the far reaches of a continuum are admired as ‘sticking to one’s principles’, ‘consistent’, ‘loyal’, ‘dedicated’, ‘committed’. Claims of certainty and crossing your heart and hoping to die are also looked well upon. So what’s the difference? Obviously the correct answers to some questions are at the ends of spectrums while others, such as optimal trade-offs, tend to have more central values. Is this what determines our like or dislike for centrism and extremism? Lets look at some examples from my understanding of popular opinion.

Things you should be extremist on:

What’s the worth of a human life?

At what degree of temptation should you cheat on your partner? Break the law? Break a promise?

How long should a marriage last?

How much does average IQ differ across races?

How much should a pedophile be willing to pay for you to let them have a child?

Things you should not be extremist on:

How closely should we follow a single political principle, such as libertarianism or communism?

What proportion of situations should you analyse in terms of a single theory?

How much of your sacred text is literally true? How much should it influence your life?

To what degree should one principle, such as utilitarianism, define your ethical views?

To what degree should you rely on reasoned thought for opinions?

How much of your time should you devote to a single activity (with the exception perhaps of looking after your family)?

I can’t see that the first list contains fewer trade offs than the second list. In fact it probably has more. So what’s the pattern?

The one I see is whether commitment is to an impersonal idea or to a group or person. If you take a centrist position on your personal and group loyalties you are something between flaky and treacherous. You are not supposed to trade off friends. On the other hand strong commitment to a policy position, theory, type of analysis, ethical standpoint, or other impersonal influence on behavior is unbalanced, biased, radical, dangerous, and consists ...
Rooting Hard for Overpriced M&Ms
by David_J_Balan — 2009-11-28T19:10:34.336Z

The other day I went to get some productivity-enhancement M&Ms from the candy machine at work. When I opened my wallet, I didn't immediately see a $1 bill. Then I looked some more and I found one, and I was happy! But of course that doesn't make any sense. If that bill hadn't been a $1, then it would have had to be a $5 or more, with an expected value of $5+, which is an amount that I certainly would not have paid for a bag of M&Ms, most excellent though they may be. This means that I preferred a bag of M&Ms to $1 (that's why I went to the candy machine in the first place), $1 to $5+ (I was happy when the bill turned out to be a $1), and $5+ to a bag of M&Ms (I wouldn't have bought them at that price). Not too surprising I guess, but still kind of weird.
Getting Feedback by Restricting Content
by billswift — 2009-11-27T22:50:35.642Z

Sivers just posted an important point about getting feedback, to get feedback on a post, present only one idea at a time.

Original post here http://sivers.org/1idea ; Hacker News comments http://news.ycombinator.com/item?id=964183 ; my post on it http://williambswift.blogspot.com/2009/11/many-ideas-or-one-idea-or-both.html

The main point of my post is: I wonder if there is any way to combine the two views? To provide more background and context, with the necessarily larger numbers of ideas being presented, while still getting useful feedback from readers.
Why evaluate everything ASAP?
by KatjaGrace — 2009-11-27T02:48:06.000Z

When I was young my brother and I used to play a game where we would page through books of wild animals or foreign places and on each page pick our favorite one, just for the pleasure of choosing. Driving in the country, looking in a museum, or window shopping in a mall, people point out to one another which things they like or dislike. We choose favorite colors, places, Dr Who incarnations, reality TV contenders, political personalities, reasons for disregarding postmodernism. On hearing the news, meeting new people, or learning gossip, the next step is usually to make a value judgment about the parties involved. Of all the characteristics everything has, the one we are itching to establish first is our own judgment of value.

At first this seems to make sense. How much we like something is key to how we respond to it. However plenty of the things we so keenly evaluate we can’t easily respond to, and don’t try to, apart from voicing our opinion. Do we waste so much time and thought judging things we can’t influence as a sad byproduct of judging things we can influence? It sometimes even looks like we even seek out things to evaluate – a costly byproduct that would be! But perhaps in the distant past we hardly came across anything so far away that we couldn’t influence it? That seems wrong; while we saw less we could also influence less.

The only good explanation I can think of is that the response we do often make to our judgments – voicing them – is the purpose. Why would we want to voice evaluations so? One explanation is that it is in the hope that someone else will fix things for us, but this explanation faces the same problem as our original explanation; most of the things that I can’t affect are no easier for my friends to affect. No matter how much I tell my brother that I like pterodactyls most, he doesn’t do a thing. I don’t get my Dr Who and that cloud I thought was pretty evaporated before anyone lifted a finger.

A last theory is that it’s all about hinting to others what sort of people we are. To that irrelevant preferences should be quite useful, just as our more obviously image-improving activities, such as careful dressing, are. A signaling theory makes the things furthest from our influence better to demonstrate opinions on, as we aren’t constrained by the need to make useful decisions on them. I often make inferences about people f...
Contrarianism and reference class forecasting
by taw — 2009-11-25T19:41:36.423Z

I really liked Robin's point that mainstream scientists are usually right, while contrarians are usually wrong. We don't need to get into details of the dispute - and usually we cannot really make an informed judgment without spending too much time anyway - just figuring out who's "mainstream" lets us know who's right with high probability. It's type of thinking related to reference class forecasting - find a reference class of similar situations with known outcomes, and we get a pretty decent probability distribution over possible outcomes.

Unfortunately deciding what's the proper reference class is not straightforward, and can be a point of contention. If you put climate change scientists in the reference class of "mainstream science", it gives great credence to their findings. People who doubt them can be freely disbelieved, and any arguments can be dismissed by low success rate of contrarianism against mainstream science.

But, if you put climate change scientists in reference class of "highly politicized science", then the chance of them being completely wrong becomes orders of magnitude higher. We have plenty of examples where such science was completely wrong and persisted in being wrong in spite of overwhelming evidence, as with race and IQ, nuclear winter, and pretty much everything in macroeconomics. Chances of mainstream being right, and contrarians being right are not too dissimilar in such cases.

Or, if the reference class is "science-y Doomsday predictors", then they're almost certainly completely wrong. See Paul Ehrlich (overpopulation), and Matt Simmons (peak oil) for some examples, both treated extremely seriously by mainstream media at time. So far in spite of countless cases of science predicting doom and gloom, not a single one of them turned out to be true, usually not just barely enough to be discounted by anthropic principle, but spectacularly so. Cornucopians were virtually always right.

It's also possible to use multiple reference classes - to view impact on climate according to "highly politicized science" reference class, and impact on human well-being according to "science-y Doomsday predictors" reference class, what's more or less how I think about it.

I'm sure if you thought hard enough, you could come up with other plausible reference classes, each leading to any conclusion you desire. I don't see how one of these reference class rea...
Do what your parents say?
by KatjaGrace — 2009-11-25T11:16:17.000Z

Should you feel compelled to pay any heed to what your parents want in your adult choices? I used to say a definite ‘no’. My mother said I should do whatever I liked, and I vowed to ignore her. From a preference utilitarian perspective, I guess that virtually all aspects of a person’s lifestyle make much more difference to a given person than to their parents. If you feel a sense of obligation in return for your parents giving you life, why? You made no agreement, your parents took their chances in full knowledge you might grow up to be anyone.

However what if fewer parents do take their chances with a greater risk of children being less satisfactory to them? The biggest effect of taking your parents’ preferences into account more could be via increasing the perception that children are worth having to other parents. It may be a small effect, but the value of life is high.

I’m not sure how much of a difference expected agreeableness of childen makes to people’s choices to have them. At first it may seem negligible. Most people seem to like their children a lot regardless of what they do. However if a person were guaranteed that their child would grow up to be exactly the opposite of what they admire, I would be surprised if there were no effect, so I must expect some gradient. I haven’t seen any data on this except my mother’s (joking?) claim that she would’ve aborted me had she thought I would be an economist. I’m not about to give up economics, but I do visit sometimes, and I painted the new living room and helped with my grandmother’s gardening since getting here this time. See how great descendent are? I would be interested if anyone has better data.
Agree, Retort, or Ignore? A Post From the Future
by Wei Dai — 2009-11-24T22:29:23.608Z

My friend Sasha, the software archaeology major, informed me the other day that there was once a widely used operating system, which, when it encountered an error, would often get stuck in a loop and repeatedly present to its user the options Abort, Retry, and Ignore. I thought this was probably another one of her often incomprehensible jokes, and gave a nervous laugh. After all, what interface designer would present "Ignore" as a possible user response to a potentially catastrophic system error without any further explanation?

Sasha quickly assured me that she wasn't joking. She told me that early 21st century humans were quite different from us. Not only did they routinely create software like that, they could even ignore arguments that contradicted their positions or pointed out flaws in their ideas, and did so publicly without risking any negative social consequences. Discussions even among self-proclaimed truth-seekers would often conclude, not by reaching a rational consensus or an agreement to mutually reassess positions and approaches, or even by an unilateral claim that further debate would be unproductive, but when one party simply fails to respond to the arguments or questions of another without giving any indication of the status of their disagreement.

At this point I was certain that she was just yanking my chain. Why didn't the injured party invoke rationality arbitration and get a judgment on the offender for failing to respond to a disagreement in a timely fashion, I asked? Or publicize the affair and cause the ignorer to become a social outcast? Or, if neither of these mechanisms existed or provided sufficient reparation, challenge the ignorer to a duel to the death? For that matter, how could those humans, only a few generations removed from us, not feel an intense moral revulsion at the very idea of ignoring an argument?

At that, she launched into a long and convoluted explanation. I recognized some of the phrases she used, like "status signaling", "multiple equilibria", and "rationality-enhancing norms and institutions", from the Theory of Rationality class that I took a couple of quarters ago, but couldn't follow most of it. (I have to admit I didn't pay much attention in that class. I mean, we've had the "how" of rationality drummed into us since kindergarten, so what's the point of spending so much time on the "what" and "why" of it now?)...