trentmkelly/LessWrong-43k huggingface dataset:
[SEQ RERUN] Uncritical Supercriticality
by MinibearRex — 2011-11-16T04:07:53.088Z

Today's post, Uncritical Supercriticality was originally published on 04 December 2007. A summary (taken from the LW wiki):

One of the most dangerous mistakes that a human being with human psychology can make, is to begin thinking that any argument against their favorite idea must be wrong, because it is against their favorite idea. Alternatively, they could think that any argument that supports their favorite idea must be right. This failure of reasoning has led to massive amounts of suffering and death in world history.

Discuss the post here (rather than in the comments to the original post).
This post is part of the Rerunning the Sequences series, where we'll be going through Eliezer Yudkowsky's old posts in order so that people who are interested can (re-)read and discuss them. The previous post was Resist the Happy Death Spiral, and you can use the sequence_reruns tag or rss feed to follow the rest of the series.
Sequence reruns are a community-driven effort. You can participate by re-reading the sequence post, discussing it here, posting the next day's sequence reruns post, or summarizing forthcoming articles on the wiki. Go here for more details, or to have meta discussions about the Rerunning the Sequences series.
Less Wrong/Rationality Symbol or Seal?
by Goobahman — 2011-11-15T22:38:11.016Z

Hey Everyone,
I was wondering if the LW community has a particular symbol or sign that would serve to act as a graphical representation of the community?
Something we could wear or include in things like business cards, that would act as an acknowledgement to others of our committment to rationality.
Any such thing exist, and if not, any good ideas?

I think the letters LW work pretty well if you could make them look more appealling.
Poker with Lennier
by HonoreDB — 2011-11-15T22:21:20.561Z

In J. Michael Straczynski's science fiction TV show Babylon 5, there's a character named Lennier. He's pretty Spock-like: he's a long-lived alien who avoids displaying emotion and feels superior to humans in intellect and wisdom. He's sworn to always speak the truth. In one episode, he and another character, the corrupt and rakish Ambassador Mollari, are chatting. Mollari is bored. But then Lennier mentions that he's spent decades studying probability. Mollari perks up, and offers to introduce him to this game the humans call poker.

Later, we see Mollari, Lennier, and some others playing poker. Lennier squints at his hand and remarks, "Interesting. The odds of this combination are 5000:1, against." Everybody considers this revelation for a moment, then folds, conceding the hand. Mollari is exasperated, and tells him to stop doing that. Because Lennier is essentially announcing that he has a good hand, Lennier's winning far fewer chips than he should; your biggest wins in poker are when people underestimate you.

The other poker players, and the audience, are picturing Lennier as having a hand something like this:

This is a four of a kind, the second-best hand in most poker games. The odds against being dealt a four of a kind in a hand of five cards are 4164:1--one might, in a moment of excitement, round that up to an even five thousand.

We the audience are meant to have a hearty chuckle over how theory doesn't translate into practice. But! We never get to see Lennier's cards, which means we get to picture whatever we want. I choose to believe, and I urge you to do so as well, that Lennier had this hand:

This is one of the worst hands possible in poker: ace-high. It loses to almost everything. By causing everyone else to fold, Lennier won a hand he probably would otherwise have lost. He knew exactly what he was doing.

"Wait," I hear you say. "Like most members of the proud Minbari race, Lennier is sworn to always tell the truth. How could he ever make a verbal bluff in a poker game?" Well. Let's consider a few different ways we can interpret the phrase "the odds of this combination."

First of all, the specific two hands I've given above are equally likely to be dealt. Any specific set of five cards is just as likely to show up as any other. There are 2,598,960 distinct hands of poker, all created equal, so the odds against ...
Rationality Dojo Examples?
by XFrequentist — 2011-11-15T21:28:58.282Z

Early on in my exposure to Less Wrong I encountered the metaphor of Rationality as Martial Art. I assumed at some point I would be a member of an active Rationality Dojo, regularly training and becoming progressively more formidable as I learned the Art.

Several years later, though I meet regularly with an awesome local group whose company I greatly enjoy, I still feel as though my training has not yet begun.

Can anyone point to an example of an active Rationality Dojo? What do you do there (Games? Exercises? Kata?)? Who are the instructors? The closest examples that I've seen are the Mega- and Mini-camps; can anyone shed some additional light on what went on there?
Babyeater's dilemma
by Giles — 2011-11-15T20:15:25.446Z

Imagine it's the future, and everything has gone according to plan. Humanity has worked out its own utility function, f0, and has worked out a strategy S0 to optimize it.
Humanity has also run a large number of simulations of how alien worlds evolve. It has determined that of those civilizations which reach the same level of advancement - that know their own utility function and have a strategy for optimizing it - there is an equal probability that they will end up with each of 10 possible utility functions. Call these f0...f9.
(Of course, these simulations are coarse-grained enough to satisfy the nonperson predicate).
Humanity has also worked out the optimal strategy S0...S9 for each utility function. But they just happen to score poorly on all of the others:
fi(Si) = 10
fi(Sj) = 1 for i != j
In addition, there is a compromise strategy C:
fi(C) = 3 for all i.
The utility functions, f0 through f9, satisfy certain properties:

They are altruistic, in the sense that they care just as much about far-away aliens that they can't even see as they do about members of their own species.

They are additive: if one planet implements Sj and another implements Sk, then:
fi(Sj on one planet and Sk on the other) = fi(Sj) + fi(Sk).

(This is just to make things easier - the problem I'm describing will still apply in cases where this rule doesn't hold).

They are non-negotiable. They won't "change" if that civilization encounters aliens with a different utility function. So if two of these civilisations were to meet, we would expect it to be like the humans and the babyeaters: the stronger would attempt to conquer the weaker and impose their own values.

In addition, humanity has worked out that it's very likely that a lot of alien worlds exist, i.e. aliens are really really real. They are just too far away to see or exist in other Everett branches.
So given these not entirely ridiculous assumptions, it seems that we have a multiplayer prisoner's dilemma even though none of the players has any causal influence on any other. If the universe contains 10 worlds, and each chooses its own best strategy, then each expects to score 19. If they all choose the compromise strategy then each expects to score 30.
Anyone else worried by this result, or have I made a mistake?
If life is unlikely, SIA and SSA expectations are similar
by Stuart_Armstrong — 2011-11-15T16:45:58.003Z

Consider a scenario in which there are three rooms. In each room there is an independent 1/1000 chance of an agent being created. There is thus a 1/109 probability of there being an agent in every room, a (3*999)/109 probability of there being two agents, and a (3*9992)/109 probability of there being one.

Given that you are one of these agents, the SIA and SSA probabilities of there being n agents are:

Number of agents	SIA	SSA

0
0
0

1
(1*3*9992)/(3*1+2*3*999+1*3*9992)
(3*9992)/(1+3*999+3*9992)

2
(2*3*999)/(3*1+2*3*999+1*3*9992)
(3*999)/(1+3*999+3*9992)

3
(3*1)/(3*1+2*3*999+1*3*9992)
(1)/(1+3*999+3*9992)

The expected numbers of agents is (1(3*9992) + 2(2*3*999) + 3(3*1))/(3*1+2*3*999+1*3*9992) = 1.002 for SIA, and (1(3*9992) + 2(3*999) + 3(1))/(1+3*999+3*9992) ≈ 1.001 for SSA. The high unlikelihood of life means that, given that we are alive, both SIA and SSA probabilities get dominated by worlds with very few agents.

This of course only applies to agents who existence is independent (for instance, separate galactic civilizations). If you're alive, chance are that your parents were also alive at some point too.
Existential Risk
by lukeprog — 2011-11-15T14:23:18.220Z

This is a "basics" article, intended for introducing people to the concept of existential risk.

On September 26, 1983, Soviet officer Stanislav Petrov saved the world.

Three weeks earlier, Soviet interceptors had shot down a commercial jet, thinking it was on a spy mission. All 269 passengers were killed, including active U.S. senator Lawrence McDonald. President Reagan called the Soviet Union an “evil empire" in response. It was one of the most intense periods of the Cold War.

Just after midnight on September 26, Petrov sat in a secret bunker, monitoring early warning systems. He did this only twice a month, and it wasn’t his usual shift; he was filling in for the shift crew leader.

One after another, five missiles from the USA appeared on the screen. A siren wailed, and the words "ракетном нападении" ("Missile Attack") appeared in red letters. Petrov checked with his crew, who reported that all systems were operating properly. The missiles would reach their targets in Russia in mere minutes.

Protocol dictated that he press the flashing red button before him to inform his superiors of the attack so they could decide whether to launch a nuclear counterattack. More than 100 crew members stood in silence behind him, awaiting his decision.

"I thought for about a minute," Petrov recalled. "I thought I’d go crazy... It was as if I was sitting on a bed of hot coals."

Petrov broke protocol and went with his gut. He refused to believe what the early warning system was telling him.

His gut was right. Russian satellites had misinterpreted shiny reflections on the Earth’s surface as missile launches. Russia was not under attack.

If Petrov had pressed the red button, and his superiors had launched a counterattack, the USA would have detected the incoming Russian missiles and launched their own missiles before they could be destroyed in the ground. Soviet and American missiles would have passed in the night sky over the still, silent Arctic before detonating over hundreds of targets — each detonation more destructive than all the bombs dropped in World War II combined, including the atomic bombs that vaporized Hiroshima and Nagasaki. Most of the Northern Hemisphere would have been destroyed.

Petrov was reprimanded and offered early retirement. To pay his bills, he took jobs as a taxi driver and a security guard. The biggest award he ever received for...
New Q&A by Nick Bostrom
by Stuart_Armstrong — 2011-11-15T11:32:40.630Z

Underground Q&A session with Nick Bostrom (http://www.nickbostrom.com) on existential risks and artificial intelligence with the Oxford Transhumanists (recorded 10 October 2011).

http://www.youtube.com/watch?v=KQeijCRJSog
Meetup : West LA Meetup 11-16-2011
by  — 2011-11-15T08:45:57.191Z

Discussion article for the meetup : West LA Meetup 11-16-2011

WHEN:
16 November 2011 07:30:00PM (-0800)

WHERE:
10850 West Pico Blvd, Los Angeles, CA 90064

When: 7:30pm - 9:30pm Wednesday, November 2nd.

Notice that this is 30 minutes later than usual!

Where: The Westside Tavern in the upstairs Wine Bar, located inside the Westside Pavillion on the second floor, right by the movie theaters.

Parking is free for 3 hours.

Topic: This week we will explore ethics and morality. Our meeting will almost immediately follow the talk The 'Objective' Basis for Morality by Dr. Alan Fiske at UCLA, hosted by the Bruin Atheists.

Recommended Reading:

What is Metaethics?

Shut up and multiply - check out the articles linked from here.

If you want extra reading on FAI, read the paper introducing CEV.

Whether you're a regular reader or totally new, here for the theoretical musings or the practical things, come by and say hello! The conversation is largely unstructured and casual, and the people are awesome.

I will bring a whiteboard with Bayes' Theorem written on it.

Discussion article for the meetup : West LA Meetup 11-16-2011
[SEQ RERUN] Resist the Happy Death Spiral
by MinibearRex — 2011-11-15T04:26:26.258Z

Today's post, Resist the Happy Death Spiral was originally published on 04 December 2007. A summary (taken from the LW wiki):

You can avoid a Happy Death Spiral by (1) splitting the Great Idea into parts (2) treating every additional detail as burdensome (3) thinking about the specifics of the causal chain instead of the good or bad feelings (4) not rehearsing evidence (5) not adding happiness from claims that "you can't prove are wrong"; but not by (6) refusing to admire anything too much (7) conducting a biased search for negative points until you feel unhappy again (8) forcibly shoving an idea into a safe box.

Discuss the post here (rather than in the comments to the original post).
This post is part of the Rerunning the Sequences series, where we'll be going through Eliezer Yudkowsky's old posts in order so that people who are interested can (re-)read and discuss them. The previous post was Affective Death Spirals, and you can use the sequence_reruns tag or rss feed to follow the rest of the series.
Sequence reruns are a community-driven effort. You can participate by re-reading the sequence post, discussing it here, posting the next day's sequence reruns post, or summarizing forthcoming articles on the wiki. Go here for more details, or to have meta discussions about the Rerunning the Sequences series.
Behavioral psychology and buying a warranty at Menards
by jwhendy — 2011-11-15T02:57:46.294Z

I just returned from buying a multimeter at Menards and wanted to post my thoughts while they were still fresh. I hardly ever have the need to use a multimeter. In diagnosing my non-heating microwave, I fried my 2-3 year old meter (don't ask) and went out for another to finish the job. I had many choices. I essentially went with the best of the lowest tier: $14. The next options were $35 and then $55.

I got to the checkout register and was waiting at the end of the conveyor belt ready to swipe my card when the cashier came over to me, stood very close, and in an almost confiding sort of hushed tone, said something like so: "With anything fragile like this, electronics and other things, you want to be careful. Check it out. Make sure it looks good and works. If it doesn't you just bring it back within a year and we'll replace it, no questions asked. Just two ninety seven."

Now, I believe as he said that last part, he was kind of walking back toward the register and I almost reflexively said, "Okay."

Once that word was uttered and I saw him then start doing something with the register, the words I heard all of the sudden registered. I recall thinking, "Oh! He was selling me a warranty of some sort." I grimaced internally but didn't speak up about it.

On my way out of the store, I was angry with myself and feeling very stupid. I wanted a cheap multi-meter. My $14 was now $18 after tax. Using it once or twice a year and then having it sit pristinely in my tool box isn't even worth the $3 insurance policy, especially since it was so cheap to begin with. I tried to catch myself and stop being angry; I thought, "No, let's learn from this situation rather than just feeling stupid. What in the world happened back there?"

Here's what I noticed about the interaction:

There was a sense of trust built just in him approaching me so closely

The affirmative and hushed tone conveyed both that he was something of an expert and that he was looking out for me, almost as if doing so against the wishes of "The Store."

The lack of the use of a currency value ("two ninety seven"), saying it as he walked away, and not using the word "warranty" kept me from registering that all of that walkthrough was really about a warranty. I was also just a little off guard in general, as it just never occurred that he would have any reason to approach me.

Though confused in follo

...
Skepticon IV meetup: planning
by komponisto — 2011-11-15T02:16:52.963Z

If any members of the Less Wrong community are planning to attend Skepticon IV this weekend (Nov. 18-20) in Springfield, Missouri (USA), it might be nice to see if we can arrange a meetup.

Feel free to comment and say "I'll be there!". (At least one prominent Less Wronger is even among the speakers.) Suggestions regarding locations, etc. from folks familiar with the local area are particularly encouraged.

Attractions this year include (besides what was mentioned above): a panel on "How Should Rationalists Approach Death?"; and a talk by Richard Carrier entitled "Bayes' Theorem: Key to the Universe" (and subtitled: that's right, I'm teaching you math, bitches!).
Cancer scientist meets amateur (This American Life)
by arundelo — 2011-11-15T01:59:45.762Z

This American Life episode 450: "So Crazy It Just Might Work". The whole episode is good, but act one (6:48-42:27) is relevant to LW, about a trained scientist teaming up with an amateur on a cancer cure.

It's downloadable until 19 Nov 2011 or so, and streamable thereafter.

(Technical nit: It sounds to me like the reporter doesn't know the difference between sound and electromagnetism.)

Edit: Here's a quick rot13ed summary: Vg qbrfa'g tb jryy. Nagubal Ubyynaq frrf rkcrevzragny pbagebyf naq ercebqhpvovyvgl nf guvatf gung trg va uvf jnl. Ur frrzf gb unir gnxra [gur Penpxcbg Bssre](uggc://yrffjebat.pbz/yj/w8/gur_penpxcbg_bssre/).
Meetup : First Eugene Meetup
by Rubix — 2011-11-15T01:18:52.184Z

Discussion article for the meetup : First Eugene Meetup

WHEN:
20 November 2011 01:00:00PM (-0800)

WHERE:
100 W 10th Avenue, Eugene, OR

Any Eugene or Springfield based Less Wrongers are welcome to attend! We will be meeting at 1 PM at the café in front of the library, and may move from there to the library proper or to a study room if so impelled. This initial meetup will not be so structured; my hope is that we'll mostly do introductions, get to know each other and discuss Less Wrong in general. Anyone with a laptop is emphatically invited to bring one.

My hope is that if enough people are interested, this could become a bi-weekly or even weekly meetup. Let me know if you plan to attend, or if another date or time would would enable you to do so.

Discussion article for the meetup : First Eugene Meetup
Intelligence Explosion analysis draft: types of digital intelligence
by lukeprog — 2011-11-14T23:07:55.586Z

Again, I invite your feedback on this snippet from an intelligence explosion analysis Anna Salamon and myself have been working on.

_____

From here to digital intelligence

Our first step is to survey the evidence suggesting that, barring global catastrophe and other disruptions to scientific progress,1 there is a significant probability we will see the creation of digital intelligence2 within a century.

Why focus on digital intelligence instead of, say, the cognitive enhancement of biological humans? As we discuss in a later section, digital intelligence has certain advantages (e.g. copyability) that make it likely to lead to intelligence explosion.

Below, we discuss the different types of digital intelligence, what kinds of progress are likely to push us closer to digital intelligence, and how to estimate the time at which digital intelligence will arrive.

Types of digital intelligence

To count as a "digital intelligence," an artificial agent must have at least a human-level general capacity3 to achieve its goals in a wide range of environments, including novel ones.4

IBM's Jeopardy!-playing Watson computer is not a digital intelligence in this sense because it can only solve a narrow problem. Imagine instead a machine that can invent new technologies, manipulate humans with acquired social skills, and otherwise learn to navigate new environments on the fly. A digital intelligence need not be sentient, though, so long as it has a human-level capacity to achieve goals in a wide variety of environments.

There are many types of digital intelligence. To name just four:

The code of a transparent AI is written explicitly by, and largely understood by, its programmers.5

An opaque AI is not transparent to its creators. For example it could be, like the human brain, a messy ensemble of cognitive modules. In an AI, these modules might be written by different teams for different purposes using different languages and approaches.

A whole brain emulation (WBE) is a computer emulation of the brain structures required to functionally reproduce human thought and perhaps consciousness (Sandberg and Bostrom 2008). We need not understand the detailed operation of a brain to reproduce it functionally on a computing substrate.

A hybrid AI is a mix of any two or three of the above types of digital intelligence (transparent AI, opaque AI, and WBE).

Notes for

...
Cryonics costs: given estimates are low
by gwern — 2011-11-14T17:07:26.242Z

One of the sticking points for cryonics is how expensive it is. Unfortunately, the estimates on LW (eg. in Normal Cryonics) are likely to be low as they are current costs. This is starting to come to a head for Alcor, with Alcor's low growth rate meaning it faces a rising tide of aging members (hence that emphasis on young cryonicists) and fundamental flaws in its prices; the official word has come down in the latest issue of Cryonics, issue 2011 q4:

Cryopreservation Funding and Inflation: The need for Action; A Discussion Article by the Management and Board of Directors of Alcor

The cryonics economies anticipated by Robert Ettinger in 1965 were never realized. By the 1970s, the cost of whole body cryopreservation as offered by TransTime and Soma (the for-profit arm of IABS, which later merged with Alcor) was $60,000 (1). As shown in Fig. 1, the nominal dollar cost of cryonics has risen steadily with Consumer Price Index (CPI) inflation since then. By 2011, the minimum funding for whole body cryopreservation with Alcor was $200,000. Even this large number has not kept pace with inflation, so another increase will be necessary soon.
Whenever Alcor has increased cryopreservation minimums, it has traditionally only required new members to meet new minimum funding requirements. Existing members were “grandfathered,” and allowed to remain members even if their cryopreservation funding fell below new minimums. This was and is believed to be important for members who due to age or disability become uninsurable, and would otherwise have to leave Alcor after many years of supporting the organization.
...The sustainability of this has been questioned on numerous occasions. In 1991, Ben Best and others expressed concerns about grandfathering in a series of articles and letters in Cryonics magazine (2,3,4). Ideas for addressing the inflation problem were sought (5), but none were implemented. There was renewed public concern in 2009 when Charles Platt published an article about inflation and cryonics funding in Cryonics magazine (6), followed by a critical article on CryoNet in 2010 that accused Alcor of negligently ignoring the grandfathering problem (7,8). That same year Rob Freitas published a detailed quantitative analysis of Alcor finances based on publicly available information, and concluded that grandfathering was a serious long-term problem (9,10). R

...